The newest model lost, and only the eval knew

There's a reflex in this work I've learned to distrust: a newer model drops, bigger numbers in the name, so you reach for it. I had that reflex with PermitCheck, my Toronto permit-readiness checker, which reads a property's drawings and verifies them against the zoning that actually applies. The hardest part is the reading: pulling the applicant's proposed numbers off a dense, busy architectural sheet. So I lined up a range of vision models to do that one job and measured instead of guessing.

The newest, biggest model lost. An older, cheaper one read those drawings more accurately, by a clear margin, and cost less per check. I wouldn't have believed it from the changelog. I believed it because I had a small set of real outcomes to grade against, and the grades weren't close.

What a golden set is

A golden set is a small collection of real cases where you already know the right answer, assembled for your exact task. Mine is real decided applications: packages that went through the City and came back with the City's own determination of what was wrong with them. That determination is the answer key. For each one I know which dimensions were non-compliant and by how much, so I can ask any version of the system "did you catch what the City caught" and get a number back instead of a feeling.

Seventeen packages, hand-labeled once, was enough to turn "this feels better" into "this is six points better or it isn't." The cases don't have to be many, but they have to be real and the labels have to be right, because a wrong label corrupts every score you read after it.

1

Collect real decided cases

Packages that went through the City and came back with its own determination of what was wrong with them.

2

Hand-label the answer key

For each case, which dimensions were non-compliant and by how much. Labeled once, and the labels have to be right.

3

Score every change against the same cases

A model swap or a prompt tweak: each one gets a number back instead of a feeling.

4

Read which cases moved

55 to 62 on the same cases, and you can see which ones.

The surprises it surfaced

Once you can measure, the system starts telling you things you didn't want to hear. The model choice was the first surprise: the newest one lost.

| Extraction model | Read rate | Correct rate | Cost | |---|---|---|---| | gemini-3-flash-preview (newest) | 77% | 50% | higher | | gemini-2.5-flash (older) | 81% | 62% | lower |

Same eval, same 17 hand-labeled packages, isolated to the extraction step. The older model reads the drawings better, gets more values right, and costs less. The default is now measured, not assumed.

The second was where the difficulty lived. I'd assumed my misses came from missing rules, the system not knowing some limit. The eval said no: I had the rules and the limits were right.

The misses were almost all reading, the model failing to pull a small number off a crowded page. The bottleneck had moved from "we don't know the rule" to "can we read the drawing," which is a far better problem to have and a completely different one to fund.

The third was the most useful. The eval flagged a batch of confident failures the City hadn't flagged. My first instinct was that the eval was wrong. It wasn't.

The tool was failing an existing lot for being narrower than the zone's minimum, and an existing undersized lot is grandfathered, not a violation. The eval had found a real bug in my rules by noticing my answers disagreed with reality on cases where reality was known. I'd never have found it by reading the code.

Why the eval is load-bearing

Every other improvement I made this season, like running the read more than once and reconciling the passes, or resolving each property's binding limit instead of a generic default, was only legible because the eval scored it. Same for teaching the tool to refuse when it couldn't confirm a value. Without the eval, "I improved the extraction" is a claim. With it, it's 55 to 62 on the same cases, and you can see exactly which cases moved.

These systems also drift silently. A prompt tweak or a model bump can make the thing worse, and nothing in the running product will tell you. Users won't file a ticket that says "your tool is 7% less right than last week." Only a measurement notices, and only if you built it before you needed it.

Build the small set of real cases before the first improvement, not after the first regression.