The Gap Left in the Draft is a wall made of words — a rule with a gap. The Duct Tape Doesn’t Care is a wall made of pins — parts that don’t fit. This one is a wall made of a ruler you stopped re-checking. Nobody tampered with it. Nobody drove through a gap. The world it was calibrated on just moved underneath it, and the ruler kept reporting confident numbers the whole time.
That’s a third, distinct failure mode: not “the rule had a gap” or “the parts don’t fit,” but “the ruler was calibrated on the old world and nobody re-checked it against the new one.” ⚪ NOT YET SOURCED means a claim this page has not yet backed with a source. Do not rely on it. ◐ mine means the framing is the lab’s, not a measurement.
A benchmark exists to grade AI agents against real professional work — human evaluators compare an agent’s output to what an actual freelancer delivered, on real jobs: 3D mockups, floor plans, video ads. Grading this way is accurate, but slow and expensive at scale. So the operators build a second, automated judge — an AI grading AI output — and calibrate it against older, weaker models, where its scores lined up well with the human raters.
Then two much stronger models arrive at once. Run through the same automated judge, their scores come back inflated — one nearly 3×, another roughly 2.5× — versus what human evaluators actually gave the same output. The instrument didn’t break loudly. It kept producing confident-looking numbers. It just stopped tracking the thing it was built to track, the moment the world it was calibrated on changed underneath it. That’s the tell: you don’t distrust a measurement because someone gamed it. You distrust it because you never checked whether the thing it was built to measure is still the thing it’s measuring.
Same three buckets as every RVP lab — Resources, Variables, Parameters — pointed at the instrument this time, not the rule.
| The ruler that stopped measuring | |
|---|---|
| Parameter | The human-graded score — the actual ground truth. Slow and expensive to produce, but real. |
| Variable | Whether the cheap automated proxy still tracks that ground truth as the thing being measured — model capability — shifts under it. |
| Resource | The cost tradeoff: pay humans to grade every case, or trust a fast automated stand-in that can silently decouple from reality. |
Drag the two sliders and watch the dot. The trap is the top-right corner: a judge can look just as trustworthy at high drift as at zero drift, because nothing about its own output changes when it goes stale. ◐ mine — the placement is the lab’s read, not a measurement.
Your move: the instructor hands you a third instrument — a school test, a credit model, a fitness tracker, any proxy for something real. Place it on these same two axes and defend, in one paragraph, whether it’s due for a fresh check against ground truth. There is no right coordinate. There is only a placement you can argue for.
On July 16, 2026, Moonshot AI released Kimi K3, a 2.8-trillion-parameter open-weight model. Within hours, Arena’s Frontend Code leaderboard — an independent, blind evaluation — ranked it first, at 1,679 points, ahead of Claude Fable 5, and above Claude Opus 4.8 and GPT-5.6 Sol on frontend-coding tasks. The result rattled AI valuations.
That’s one ruler. Here’s the other: Moonshot itself said K3 still sits behind Fable 5 and GPT-5.6 Sol on overall performance. Same model, same week — one ruler calls it the best in the world, another puts it a tier down. Neither is lying. They are measuring different things, and only one of them got quoted in the headline.
That’s the tell, live: the Frontend Code Arena measures frontend coding — a real, narrow thing. Reading a first-place finish there as “the best AI” asks a ruler for a number it was never calibrated to give; a ruler is only valid inside the range it was checked against. And a second caveat the coverage itself flagged: until the open weights ship (July 27, 2026), every K3 number is a claim from Moonshot or its API — a confident ranking nobody outside can yet independently re-run against real repositories.
Sourced · pulled 2026-07-19: The New Stack (Arena blind eval; K3 ahead of Opus 4.8 & GPT-5.6 Sol on frontend coding, “the real test comes when developers throw production workflows at it”) · Tom’s Hardware (1,679 pts, ahead of Fable 5; Moonshot says behind on overall; unverifiable until July 27) · Simon Willison · CNBC.
This is the constraint type The Gap Left in the Draft and The Duct Tape Doesn’t Care don’t cover: an instrument that was honest when it was built and stopped being honest without anyone changing it. Nobody tampered with the judge. Nobody exploited a gap in it the way The Gap Left in the Draft’s teams exploited a gap in statute. The world underneath it just moved faster than anyone re-validated the tool against. The only way to catch it is periodically re-checking against the expensive, slow ground truth — which is exactly the discipline the benchmark’s own operators showed by catching it and reporting it honestly instead of quietly switching to the cheaper number.
The core example carries no citations — with one exception: the live Kimi K3 postscript above is ● real / sourced and names the models, because it’s a public, cited event — the receipts for the same phenomenon the general case teaches in the abstract. Everything else below stays general. The benchmark, the human-vs-automated-judge discrepancy, and the specific inflation factors (~3× and ~2.5× for the two strong models) are marked ⚪ NOT YET SOURCED: this lab names no benchmark, no operator, and no document you could check them against. Those are precise numbers wearing no source — treat them as unverified and do not rely on them. Keeping the operator “general” is a stylistic choice that has the effect of making the claim uncheckable, which is exactly why it cannot also wear a “● real” badge. The “ruler stopped measuring the thing” framing and the confidence-vs-drift axes are ◐ mine — a teaching synthesis, not a measurement. Model names stay unveiled per the standing rule in the general case; the K3 postscript is the sourced exception. Spot something off? Email User Zero — corrections get acknowledged right here.