The Gap Left in the Draft is a wall made of words — a rule with a gap. The Duct Tape Doesn’t Care is a wall made of pins — parts that don’t fit. This one is a wall made of a ruler you stopped re-checking. Nobody tampered with it. Nobody drove through a gap. The world it was calibrated on just moved underneath it, and the ruler kept reporting confident numbers the whole time.
That’s a third, distinct failure mode: not “the rule had a gap” or “the parts don’t fit,” but “the ruler was calibrated on the old world and nobody re-checked it against the new one.” ● real means the case is reported. ◐ mine means the framing is the lab’s, not a measurement.
A benchmark exists to grade AI agents against real professional work — human evaluators compare an agent’s output to what an actual freelancer delivered, on real jobs: 3D mockups, floor plans, video ads. Grading this way is accurate, but slow and expensive at scale. So the operators build a second, automated judge — an AI grading AI output — and calibrate it against older, weaker models, where its scores lined up well with the human raters.
Then two much stronger models arrive at once. Run through the same automated judge, their scores come back inflated — one nearly 3×, another roughly 2.5× — versus what human evaluators actually gave the same output. The instrument didn’t break loudly. It kept producing confident-looking numbers. It just stopped tracking the thing it was built to track, the moment the world it was calibrated on changed underneath it. That’s the tell: you don’t distrust a measurement because someone gamed it. You distrust it because you never checked whether the thing it was built to measure is still the thing it’s measuring.
Same three buckets as every RVP lab — Resources, Variables, Parameters — pointed at the instrument this time, not the rule.
| The ruler that stopped measuring | |
|---|---|
| Parameter | The human-graded score — the actual ground truth. Slow and expensive to produce, but real. |
| Variable | Whether the cheap automated proxy still tracks that ground truth as the thing being measured — model capability — shifts under it. |
| Resource | The cost tradeoff: pay humans to grade every case, or trust a fast automated stand-in that can silently decouple from reality. |
Drag the two sliders and watch the dot. The trap is the top-right corner: a judge can look just as trustworthy at high drift as at zero drift, because nothing about its own output changes when it goes stale. ◐ mine — the placement is the lab’s read, not a measurement.
Your move: the instructor hands you a third instrument — a school test, a credit model, a fitness tracker, any proxy for something real. Place it on these same two axes and defend, in one paragraph, whether it’s due for a fresh check against ground truth. There is no right coordinate. There is only a placement you can argue for.
This is the constraint type The Gap Left in the Draft and The Duct Tape Doesn’t Care don’t cover: an instrument that was honest when it was built and stopped being honest without anyone changing it. Nobody tampered with the judge. Nobody exploited a gap in it the way The Gap Left in the Draft’s teams exploited a gap in statute. The world underneath it just moved faster than anyone re-validated the tool against. The only way to catch it is periodically re-checking against the expensive, slow ground truth — which is exactly the discipline the benchmark’s own operators showed by catching it and reporting it honestly instead of quietly switching to the cheaper number.
The benchmark, the human-vs-automated-judge discrepancy, and the specific inflation factors (~3× and ~2.5× for the two strong models) are ● real — drawn from the reported case, kept general on the identifying details. The “ruler stopped measuring the thing” framing and the confidence-vs-drift axes are ◐ mine — a teaching synthesis, not a measurement. Model names stay unveiled per the standing rule; the benchmark operator is kept general. Spot something off? Email User Zero — corrections get acknowledged right here.