Why manual prompting isn't enough
Prompt engineering is a tedious, messy cycle. Teams write long system prompts, manually inspect traces, and iterate for months until they are happy enough with the outcomes. Over time, that cycle continues as new and improved AI models are incorporated and as engineers add complexity to the system.
In our own work, we have repeatedly seen teams spend months refining prompt wording, only to discover that the best solution required changing the context pipelines, modules, and tools. Prompt iteration should be performed in the context of evolving the system as a whole.
To solve the manual nature of prompting, prompt optimization offers an automated process of improving large language model prompts using evaluation datasets and iterative search algorithms. Instead of hand-editing instructions, engineers define a reward metric and allow optimizers to search for stronger prompt variants.
To prompt in the context of an evolving system, system-level evolution provides the strongest impact. At CodeEvolver, we specialize in optimizing the entire system. Before discussing system-level evolution, it helps to examine how modern prompt optimizers operate.
What is prompt optimization in practice?
Prompt optimization treats prompts as parameters in a measurable search space. In practice, this requires three components: a task specification, an evaluation dataset containing either ground-truth labels or judged outputs, and a modular LLM program that executes the prompts.
Given these elements, an optimizer generates candidate prompt variants, evaluates them against the dataset, and iteratively improves performance.
This objective is evaluated under operational constraints such as model behavior, token cost, and structural requirements of the program. Framed this way, prompts shift from static strings to tunable components within a larger AI system, refined through structured search rather than manual experimentation.
GEPA: Genetic Pareto optimization for prompts
The state-of-the-art prompt optimizer is GEPA (Genetic Pareto Optimization), introduced by Agrawal et al. (2025).
GEPA applies evolutionary search to prompt improvement. Candidate prompts are generated and mutated, each variant evaluated across a dataset, weak performers eliminated, and stronger variants preserved. Instead of collapsing to a single best prompt, GEPA maintains a Pareto frontier of strong variants across examples. This prevents premature convergence and preserves diversity across heterogeneous data distributions, since different prompt variants may perform better on different subsets of the dataset.
In reported benchmarks, GEPA achieved roughly a 20% improvement on multi-hop QA tasks relative to baseline prompts. These improvements occur without fine-tuning model weights; the gains come entirely from structured prompt search.
We ran our own prompt optimization experiments and saw GEPA improve prediction accuracy from 91% to 96% in a news fact-checking application. We've also run prompt optimization experiments in tandem with many of our CodeEvolver experiments, which can be found here.
What makes this powerful in production is compounding. A three percent lift in one module does not sound dramatic. Across five modules, it changes system behavior in ways that are very noticeable.
GEPA has also demonstrated advantages over earlier optimizers such as MIPRO V2, particularly in settings where example-level tradeoffs and diversity matter. For teams evaluating prompt optimizer options today, GEPA-based systems have become a reference point.
When can you use prompt optimization?
There are two types of data sets: those with ground truth and those without. Datasets with ground truth "gold" labels work really well for prompt optimization, because ground truth serves as a reliable reward function. Thus, the first instance where you should use prompt optimization is when you have an objective metric, like ground truth.
However, many tasks are much more qualitative. What do you do when you do not have ground truth?
Meta's Dual Prompt Optimizer, part of Prompt-Ops, addresses this by using comparative judgment instead of an explicit objective function. Two prompt variants are generated and run on identical inputs. A language model judge evaluates the outputs and selects the preferable one. That comparison becomes the optimization signal.
This approach enables prompt optimization for system prompts, agent reasoning strategies, researcher context tuning, and claim extraction modules. The method is bounded by the quality and bias of the judging model, and issues such as reward hacking and drift remain practical concerns. Even so, comparative optimization expands prompt search into domains that previously required manual experimentation.
Competitive landscape: prompt optimizers today
Teams exploring prompt optimization tools today typically encounter three broad categories.
The first category includes prompt optimizers that operate over multiple AI modules. DSPy provides an AI framework that replaces prompt writing with prompt optimization, including built-in support for GEPA-based optimization. Opik offers a flexible prompt optimization layer that integrates across AI frameworks.
CodeEvolver extends prompt optimization further, evolving not only prompt text but also context pipelines, tool integrations, and graph structure within the AI system. The distinction here is architectural: traditional prompt optimizers assume a fixed system graph, whereas CodeEvolver treats the graph itself as part of the optimization space.
The second category consists of single-module prompt optimizers. Tools such as Microsoft Agent Lightning, Meta Prompt-Ops (Dual Prompt Optimizer), and research systems like AutoPrompt refine isolated prompts rather than full systems. They are useful for targeted improvements but do not optimize multi-agent or multi-module architectures.
The third category includes prompt playground environments such as those offered by OpenAI, Anthropic, and LangSmith. These platforms allow manual experimentation and side-by-side comparisons, but they do not perform deep search-space exploration across datasets or modules.
As AI systems become more modular and agentic, these distinctions stop being academic. They shape how your system scales.
From prompt optimization to workflow evolution
Prompt optimization produces measurable gains in accuracy and robustness. In real systems, though, performance is often constrained by retrieval configuration, tool schemas, code logic, and workflow topology.
Optimizing prompts alone cannot fix structural bottlenecks elsewhere in the system. The issue isn't the wording of the prompt. It's the retrieval layer, the tool interface, or the way modules pass context between each other. What looked like a prompt problem was really a systems problem.
CodeEvolver combines prompt and code optimization in one process, mimicking what a real AI engineer would do if they were to spend enough time inspecting every failure in their system. A coding agent reflects on the failures of a system, proposes changes to the prompts or the code, and executes improvements over many iterations.
For systems that require reliability, optimization must extend beyond individual prompts to the full workflow. CodeEvolver was built to optimize against all levers in an AI system, including prompts, context pipelines, reasoning steps, AI modules, tools and more. Find out more at codeevolver.ai.