Golden Sets vs Model Judges in CI Evaluation

Two ways to put a number on a non-deterministic system

A golden set is a fixed list of cases, each with a human-agreed correct answer, scored mechanically: exact match, structural match, or a checked predicate. The scoring code has no opinion. OpenAI’s simple-evals harness is this shape — fixed prompts, fixed answers, a published run configuration.

A model judge is a second model, given a rubric and the output of the system under test, asked to score it. No expected answer is written in advance; the rubric stands in for one.

Both emit a number your CI can compare against last week’s. They fail in different places, and the difference decides which one you are allowed to block a release on. The broader mechanics of getting any repeatable signal out of a stochastic system are covered in CI evaluation of non-deterministic AI systems; this page is about the choice between the two scoring methods.

The number itself

A golden set gives a stable, auditable number. Same inputs, same expected outputs, same comparison — so when the score moves, the only thing that can have moved is the system under test. That property is what makes the number usable in an argument. A disputed regression is settled by opening the file and reading the row that flipped, which is a conversation that ends in minutes rather than a debate about whether the measurement is trustworthy.

What it costs is human effort, twice. Someone writes the cases and the expected answers up front, and someone keeps them current as the product changes. The second cost is the one teams underestimate: prompts get rewritten, tool schemas gain a field, a formatting convention changes, and expected outputs that were right in March are wrong in June. A stale golden set does not announce itself — it just starts failing for reasons unrelated to quality, and the usual response is to loosen the assertions until it goes quiet.

A model judge scales to any volume of cases. There is no per-case authoring cost, so the same rubric can score fifty regression cases or every request that hit production yesterday. The price is structural: you have introduced a second non-deterministic system into the measurement path, and that system also needs validating. Two things can now regress and only one of them is the product. When the score drops, the first question is no longer “what broke?” but “which of the two models moved?”

	Golden set	Model judge
Number is	stable, auditable	non-deterministic, needs its own validation
Cost	human effort to build and to keep current	scales to any case volume
Comparable over time	while the set is unchanged	only if the judge version is pinned and recorded
Known biases	none in mechanical scoring	position and verbosity, must be controlled for

Comparability across time

This is where judges quietly break a metric. Judge scores drift when the judging model is upgraded, so a score is only comparable across time if the judge version is pinned and recorded alongside it. Without that, a step change in the chart is indistinguishable from a step change in the product — and the judge upgrade usually happens on the vendor’s schedule, not yours.

The fix is unglamorous: store the judge’s model identifier and the rubric’s hash in the same row as the score, treat a judge change as a baseline reset, and re-score the historical cases with the new judge before you trust a comparison across the boundary. If your stack does routing between models, the judge has to be exempt from it. A router that silently moves the judge to a different model has changed your ruler without changing any code you reviewed.

A golden set has the same requirement in a milder form. Editing rows resets the baseline too — but the edit is a diff in your repository, reviewed by a person, not a vendor announcement you might have missed.

Known biases

Judges show position bias and verbosity bias: they prefer the option shown first and the longer answer. These are documented in the MT-Bench and Chatbot Arena study of LLM-as-a-judge, which also reports agreement above 80% between a strong judge model and human preferences on that benchmark — the case for judges and the case for controlling them come from the same paper.

Both biases have to be controlled for rather than hoped away. Position bias is handled by shuffling: score each pair in both orders and keep the result only when the two agree, or randomise order per case so the effect averages out instead of accumulating in one direction. Verbosity bias is handled by length normalisation, or by a rubric that scores specific claims rather than overall impression. Skip these and you will reliably measure a system that got wordier as a system that got better.

A golden set has no equivalent failure. A string or structural comparison has no preference for the longer answer and no memory of what it saw first.

The judge also needs its own validation before you believe any of its output, which is a second evaluation project on top of the first. Hugging Face’s walkthrough of building and checking an LLM judge sets it out: write the rubric, score a set of human-labelled examples, and measure agreement between judge and human before the judge is allowed near a real number.

They are complementary, not alternative

That validation step is the connection between the two options. Measuring judge-human agreement requires human-labelled cases — which is exactly what a golden set is. A small golden set calibrates the judge, and the judge extends that calibration over the cases the golden set does not cover.

So the mature setup is not one or the other. It is a human-labelled set small enough to keep current, used both as a blocking gate and as the calibration data for the judge, plus a judge running over the long tail — production samples, generated variations, the cases nobody had time to label. When the judge is upgraded, the golden set is what tells you whether the new judge still agrees with humans, and by how much it shifted.

The judge’s own quality depends on the model behind it, which is a live cost-versus-capability decision — the same one covered in frontier versus mid-tier model selection. A judge is one of the roles where the cheaper model tends to disappoint, because bias control and rubric adherence are precisely what smaller models do worst.

Which to pick when

Pick a golden set alone if your outputs have a checkable right answer — extraction, classification, which tool got called, whether the JSON validates — and you need a number that blocks a merge. Mechanical scoring is the only option here that produces a result you can defend in a postmortem without first defending the measurement.

Pick a model judge alone if outputs are open-ended, you have no labelled data yet, and you need coverage over real traffic this week rather than next quarter. Accept the number as a trend, not a gate: pin the judge version, shuffle for position, normalise for length, and do not page anyone on a single-run movement.

Pick both for anything you will still be running in six months, and split their jobs: the golden set is the blocking gate in CI, the judge is a non-blocking trend over volume. Build the golden set first even if the judge is what you want, because without it you have no way to know whether the judge is measuring quality or measuring length.

Pick neither yet if you cannot name what a failing case looks like. Both methods encode a definition of correct; neither will invent one for you.