Optimizing a rules engine at BriefCatch
A 69% reduction in failures on unseen holdout data — and a regression-testing capability built to keep it that way.
“The CodeEvolver team helped us make substantial, measurable improvements to several highly complex rules systems. Just as important, they helped us expand our eval-driven development and regression-testing capabilities.”
Software that helps lawyers write and cite with confidence
BriefCatch provides legal-writing and authority-review software that helps lawyers improve prose, review citations, and identify potential problems in the use of legal authority.
BriefCatch relied on two systems to parse legal documents and deliver reliable writing suggestions: a complex rules engine and an optional AI layer. CodeEvolver was engaged to address known failure cases in both the rules engine and the AI layer, and to extend coverage to new-scope items.
Rules engines are auditable — and very hard to improve
Rules engines are predictable, explainable, and auditable. They also can deliver highly reliable output if you can train them on enough test cases.
However, improving a rules engine involves a mountain of complexity. There are thousands of rules for parsing and editing legal documents, and changing a single rule can cascade into unexpected downstream consequences. Even with AI coding tools, expanding coverage one rule at a time is slow, risky, and hard to verify without good data.
Four phases, each one measured before it shipped
Established the real baseline
Before changing anything, we systematically tested the existing system against a large dataset to establish an objective baseline and identify the highest-impact opportunities for improvement.
Ran the CodeEvolver engine
It wrote hundreds of new rules and updated the AI system prompts, organizing the updates into 21 legal writing topics. Two new rules engine layers were added alongside the existing one to handle the new scope items.
Reviewed every change
Our AI engineers reviewed, tested, and refined the code changes before merging them through a pull request, so nothing reached production without multiple sign-offs.
Validated on holdout data
We reran the new system on a holdout dataset that the system had never seen before, confirming the gains were real and not memorized.
We reduced the failure rate by over 3×
On the holdout dataset, the new system exceeded our most aggressive targets: we reduced the failure rate by over 3x! This represents an 88% score on a deliberately challenging set of known edge cases. That is, this was not a measurement of performance on ordinary documents; it was designed to test the kinds of examples most likely to expose system weaknesses.
Fixing failure cases directly improves customer trust and creates a positive impact on BriefCatch’s business.
Eval-driven engineering is what makes it possible
In AI application development, eval-driven engineering is the technique of creating testing data sets to evaluate the performance of your system. Without evals, you cannot achieve reliability with a complex rules engine. In the legal field, this means turning to open-source legal databases (since customer data cannot be accessed).
Three components make it work:
- Building training and evaluation data from open-source legal databases
- Iteratively improving with CodeEvolver
- Wiring comprehensive tests to detect regressions when they happen
As part of the engagement, our team created 18,000 new tests for BriefCatch to detect regressions, and CodeEvolver made it easy to learn from a subset of that data practically overnight. In addition to creating comprehensive tests, our team carefully reviewed every proposed change before merging and deploying into production.
The outcome was more than a pull request. Alongside the updated rules engine, BriefCatch now has a robust evaluation and regression-testing capability that lets them reliably verify the strength of future improvements.
The takeaway for complex rules systems
To improve AI reliability you first need to establish a baseline figure for the current system. Then improvement becomes something you can continuously run, review, and repeat with the CodeEvolver self-improvement engine.
Thinking about improving your AI system?
We help companies achieve reliable AI output wherever mistakes are costly (think legal, healthcare, finance, etc.). Please reach out if you or someone you know needs support.