Introduction
CodeEvolver is a self-improvement engine for AI reliability. It reflects on the errors in your AI application, and makes changes to the prompts and code to improve your system. Every time it makes a change, it measures the performance on an evaluation set.
Improvement occurs across three levers:
- Prompts: optimized instructions, in-context learning, etc.
- Code: Context pipelines, reasoning steps, AI modules, sub-agents, tool calls, etc.
- Models: Selection of AI model (Excluded from this analysis)
How it works
At the core of CodeEvolver's self improvement engine is a coding agent that makes changes to the prompts and code in your application.
More than just a continuous, long-running coding agent, CodeEvolver combines our proprietary artificial selection algorithm with skills, memory, deep observability, and various reflection agents.
CodeEvolver builds on previous research-backed evolutionary optimizers like GEPA and AlphaEvolve. It also contains its own meta reflection agent for analyzing its shortcomings and proposing improvements for making itself stronger over time.
Experimental setup
To test CodeEvolver, we picked AI agent benchmarks that deal with retrieval and question answering: HoVer and PhantomWiki.
Each of our benchmark experiments uses a third-party program written by researchers as the baseline. We also extract the relevant benchmark dataset and split it into training, validation, and testing sets. Importantly, the test set is only used at the very end to measure the holdout set results.
We run CodeEvolver with an instruction set that combines the constraints for the optimization and any ideas for improvement. These ideas span high-level architecture ideas, retrieval strategies, etc. The instructions used are located here.
Performance
CodeEvolver shows successful improvement on every benchmark we've tried.
On question answering and retrieval benchmarks, we've seen:
- 85% relative gain on HoVer's validation set and 80% on PhantomWiki's
- 51% - 58% relative gain on holdout test sets
Results
HoVer
The HoVer program started as a three-hop retrieval pipeline that scored about 0.47 F1 on the validation set. Over the run it rebuilt the pipeline into an entity-targeted multi-hop retriever: it replaced the original summarize-then-query architecture with a module that extracts specific named entities that it still needs to retrieve. It then added query deduplication and a conditional fifth hop that fires only when a genuinely new query is found. Deeper searches were used on the final hops (in terms of pages retrieved per search), and the prompt was fine-tuned to include false-coverage checks, disambiguation, and rules for parent companies, locations, and named works. The final program scored about 0.87 F1 on the validation set.
The starting and final HoVer program can be found here and here.
Along the way, CodeEvolver attempted several other approaches: entity-name query targeting, programmatic query-retry and deduplication, up-front planning of all hop targets, and an LM-based reranker that selected the best 21 documents from a larger pool. Interestingly, the heavier "let the model decide everything" approaches did not perform as well as the lighter, code-level ones — the LM-based document reranker and the up-front "plan all the hops in advance" module both regressed sharply (the reranker alone cost roughly 7% on the evaluation set), whereas the incremental structural changes (round-robin interleaving with a conditional fifth hop and asymmetric retrieval depth) delivered the steady gains. The full iteration-by-iteration log is available in the HoVer optimization memory.
PhantomWiki
CodeEvolver started from a single DSPy ReAct agent with a bare "question → answer" signature and a ColBERT retrieval, scoring roughly 0.38 F1 at baseline. It ended with a question-type-routed, hybrid two-phase architecture: a deterministic Python classifier sorts each question into one of four types (entity, attribute, count-pivot, count-answer); "count-pivot" questions run a two-phase pipeline where a Phase 1 entity-finder locates every pivot entity and a Phase 2 counter then computes a count independently for each one — eliminating the "enumerate-then-collapse" failure where the agent found many entities but reported only a single count — while the other question types use an enriched single-phase agent. It also added a pre-built exact-match date-of-birth index (an O(1) lookup returning every person born on a given date, bypassing ColBERT's recall ceiling), a hobby/occupation anchor index, and a set of targeted reasoning rules in the signatures (reciprocal cousinhood, hop-by-hop traversal, multi-anchor expansion). The best program reached about 0.69 F1 on the validation set, an 80% relative gain over the baseline.
The starting and final PhantomWiki program can be found here and here.
Along the way, CodeEvolver attempted several approaches: enriching the prompt with breadth-first enumeration guidance and kinship-term definitions, adding scratchpad "workspace" tools the agent could record findings to, broad attribute fan-out instructions, and several ways of widening or randomly sampling the entities handed to the counting phase. Interestingly, the prompt-only attempts to push the agent to enumerate more — the workspace scratchpad tools, the broad fan-out instructions, and random pivot sampling — did not perform as well as the structural code changes: the exact-match date index and the hybrid two-phase architecture produced the two largest single-step gains of the entire run, while several of the enumeration-prompt experiments improved the training subsample but regressed on the validation set. The full iteration-by-iteration log is available in the PhantomWiki optimization memory.