Not every task has a single objective ground truth you can label. Many domains have multiple good answers. As long as good output follows some objective criteria or logical pattern, a language model can be used to score output. Often this is done through an evaluation rubric or through a combination of LLM judge frameworks (e.g., faithfulness, relevance, brevity).
When good output is easy to recognize, a language model judge works very well. However, assuming intelligence is fixed, the system can only be as smart as its teacher. Not to worry, there are many improvements to be made to a system that don't require intelligence gains; these include process, reasoning steps, resolving system errors, and more.
Exploiting the cracks
One downside to language model judges is that anything that scores is something an optimizer will learn to exploit.
A human tuning prompts probes the judge's weaknesses occasionally and by accident. An optimization engine running hundreds of candidates probes them systematically. If the judge rewards length, candidates get longer. If it's swayed by confident phrasing, candidates get confident. The optimizer isn't cheating — it's doing exactly what was asked, which is the problem. The reward function is part of the system's spec, and under optimization pressure, some gaps can be found.
Hardening the judge
What we've found works: decompose the rubric into specific, separately-scored criteria (faithfulness to source, coverage of required points, format compliance) rather than one holistic quality score — narrow questions are harder to game than vibes. Anchor as much as possible to checkable facts: citations that resolve, claims entailed by the source, schema fields present. Look for biases and address them (e.g., randomize presentation order if there is a first option bias).
Mix your signal types! A reward function that's part judge, but grounded in example ground truth labels is far harder to game than any single component.
Audit the winners
Final discipline: when optimization ends, our team reads the winning candidates' outputs before anything ships. If the improvement is real, the reading is quick confirmation. If the optimizer found a judge exploit, this is where it surfaces — visibly, in a diff, before production. That review is built into our delivery: every engagement ends with a PR a human reviews, not an automatic deploy.