There are infinite combinations of potential prompts.
The prompt engineering loop is familiar to anyone who has shipped an LLM feature. Look at a failing trace. Form a theory. Edit the prompt. Rerun. Feel hopeful. Find a new failure. Repeat. Months pass and the prompt grows to a thousand words. Nobody remembers which sentence fixed what.
Three reasons the loop fails
First, sample size. A human inspects a handful of traces per session; conclusions drawn from ten examples are folklore, not engineering. The change that fixes your ten traces regresses fifty you didn't look at — and without an eval suite, you'll discover that in production.
Second, search width. A person tries one idea at a time, serially, and anchors on their last theory. The space of useful changes — instructions, few-shot examples, decomposition, retrieval parameters, reasoning structure — is combinatorially large. Manual search explores a sliver of it.
Third, the moving target. Every model release shifts what the optimal prompt looks like. The months of tuning you did against last year's model are partially obsolete the day you upgrade. Manual tuning is a recurring cost that slows your entire product development efforts down.
What to do instead
Make the loop the job of a prompt optimizer. The recipe: an evaluation dataset that represents your real inputs; a reward metric your team agrees on; and an engine that proposes candidate changes, scores each one against the data, and keeps only the winners. Run until improvements plateau, then validate on data the engine never saw.
CodeEvolver optimizes not just the prompts but the system as well. In our benchmark experiments, our optimizer found 80% and 85% relative gains on validation sets — including changes a human prompt-tuner would likely never try, because they weren't prompt changes at all. One example win on one benchmark came from removing a summarization step and switching to direct passage extraction. No amount of wordsmithing would have found that.
The inspect-tweak-rerun loop isn't wrong because the people doing it lack skill. It's wrong because it takes significant time that is better spent on defining what good looks like in the eval data. Once that data is defined, the data itself defines your optimal prompt by way of your optimizer.