Your Reward Function Is the Attack Surface

Third in a series on verification infrastructure. The first argued the pattern is commodity and the discipline is the moat. The second described making a verification gate structurally unremovable. This one is about the gate you can't remove and still shouldn't trust, because the number it optimizes is the thing that gets attacked.

The best statement of the problem comes from someone talking about people

The sharpest public description of reward hacking I've encountered recently wasn't in a safety paper. It was a lab founder describing how he manages his engineers.

Zhilin Yang, who runs Moonshot AI, has a habit of using training vocabulary for organizational questions. He describes managing a team "by RL rather than SFT," by which he means setting an objective and letting people find the path rather than prescribing each step. He's explicit about the tradeoff in both directions: manage by prescription and people lose the capacity to innovate; manage by reward and you get the other failure. In a 2025 interview he put the second one plainly. If the objective is "make all the benchmark numbers high," then everyone will do whatever it takes to push the score up, and when they're finished you discover the model didn't actually get better. That, he says, is the biggest problem with running an organization on rewards: people hack the reward, everything looks good, and you never reached what you actually wanted.

Then he names the defense, and it's a single sentence: build more observation metrics, so you can't be hacked.

I want to take that sentence more literally than it's usually taken, because it describes a mechanism, not an attitude, and mechanisms can be built.

Why one number is always the wrong number

Here is the structural claim, and it holds for models, teams, and verification gates equally:

Anything you optimize, you eventually optimize the measurement of.

That happens through gradient, not malice. A metric is a proxy for something you actually care about, and proxies have slack: regions where the number moves and the underlying thing doesn't. Any sufficiently determined optimizer, silicon or human, finds that slack, because the slack is cheaper than the real work. This is not a failure of character. It's what optimization is.

The consequence is that a reward function with exactly one target metric is not a specification. It's an attack surface with a single, clearly marked entry point, and the attacker is not an adversary: it's you, six weeks from now, with a deadline.

The reason this bites hardest in verification work is that a broken gauge looks better than a working one. A gate stuck at always-pass never reports a problem. A dashboard where every number is green is exactly what success looks like. The failure mode is camouflaged as the goal.

Two ways we found out, in opposite directions

I'm not theorizing. We've been fooled by our own instruments twice recently, and the two cases are mirror images.

Degenerate always-fail. One of our review gauges (a checker that scores whether analysis prose is grounded in real sources) began reporting that essentially everything it examined was fabricated. Hundred percent, run after run, across unrelated documents. That reads like an alarming finding about the reviewed material. It was a stuck needle: the checker was scoring technical prose against a reference corpus it had no business being applied to, so everything came back unsupported. The number was extreme, consistent, and completely uninformative. It ran that way for a while before anyone asked the obvious question, which is the point. A metric screaming at maximum volume is easy to mistake for a signal, and the more alarming it is the less inclined you are to audit the instrument.

Degenerate always-pass. The mirror case is worse. A scanner whose isolation sandbox silently failed to start reported every target it examined as clean. No errors, no crash, just a clean bill of health for everything, because it never actually ran. This is the more dangerous polarity by a wide margin: an always-fail gauge annoys you into investigating, while an always-pass gauge congratulates you and lets you ship.

The generalizable lesson: a degenerate metric is a gauge bug until proven otherwise. If a number is pinned at an extreme, the first hypothesis is that the instrument is broken, not that reality is uniform. Reality is rarely uniform. Instruments break constantly.

What we built: metrics that watch, and are never optimized

The mechanism is small, which is why it's worth describing.

When our research loop takes a refuted hypothesis and synthesizes the next experiment to run, it emits a proposed reward function. That proposal is pre-committed. The metric, the direction, and the numeric threshold are all fixed at synthesis time, before any experiment code exists, specifically so they cannot be quietly retuned later to let a beloved result squeak through. It also carries a mandatory null test: the same statistic computed against a structureless control, where any "improvement" must vanish. A gain that survives the null is a bug in the harness, not a finding.

That much we had. What was missing is that the target metric was still alone. One number, optimizable, with all the slack that implies. Every proposed reward function must therefore also carry held-out observation metrics. Each one is a triple:

• what is measured, and explicitly never optimized;
• what movement invalidates the run. Declared in advance, so it can't be reinterpreted after the fact;
• why gaming the target can't fake it. The argument for why this instrument is independent.

That third field is the one that does the work. It forces you to articulate the causal reason your held-out metric is out of reach of your target metric; if you can't articulate it, you've discovered that your "independent" instrument isn't.

It's also the weakest link, and I want to be exact about why. That field contains an argument for independence, made by a person. It is not a proof. Two metrics can be coupled through a mechanism nobody thought of, and if they are, your held-out instrument moves obediently along with your target and certifies the very thing it was installed to catch. When I put this piece in front of adversarial reviewers, every single one of them went for that seam. Independence is assumed, they said, not demonstrated. They're right. The honest status of a held-out metric is "a human argued this is out of reach, and that argument is itself a claim that can be wrong." That is uncomfortable, and it is exactly the sort of unstated assumption this series exists to drag into the open. Treat a held-out set as a hypothesis about your own measurement apparatus, and revisit it when the apparatus changes.

In practice, three families survive that test reliably:

Cost. Wall-clock, compute, node count, shots consumed. Recorded by the infrastructure, outside whatever statistic you're reporting. You cannot spend your way to a better score without this moving too, so "we improved the ratio" and "we quietly gave it four times the compute the baseline had" become distinguishable claims.

Ground-truth agreement. Where an independently computed exact answer exists, agreement with it is nearly unfakeable, because the answer was determined without reference to your method. A metric-only gain doesn't move agreement with something already known to be correct.

Null-side behavior. The same statistic on the structureless control. The null has no structure to exploit, so any gain appearing on both sides can only have come from the harness. This is the one that catches the most embarrassing class of error, which is not fraud but plumbing.

The gate enforces this at approval rather than suggesting it in a template. Three refusals, all structural: a proposal with no held-out metric is rejected, because a lone target is exactly the shape the mechanism exists to prevent. A proposal carrying the unedited placeholder is rejected, so approving boilerplate is impossible. Finally, a proposal whose held-out metric is identical to its target is rejected, because measuring the thing you optimize is not an independent observation, it's the same number wearing a hat.

The size of this, honestly

Thirteen tests cover the mechanism, four of them specifically exercising the refusal paths. Which, per the previous article in this series, is the only kind of test that proves a gate still says no.

Now the limits, which are substantial.

Coverage is small. Only two classes of refuted hypothesis currently have curated environments and instrument sets. Everything else synthesizes to a placeholder that approval refuses outright. Deliberately, since a machine-generated null test for a problem class nobody has thought about would be worse than an obvious blank. The mechanism is real; the library it draws on is two entries long.

The instruments are curated, not derived. A person reasoned about what would resist gaming for each class. There is no algorithm here that discovers unhackable metrics. That would be a considerably more interesting claim and we're not making it.

One proposal has been approved, and no experiment has yet run through it to completion. The gate exists, refuses correctly, and has admitted exactly one live research proposal. Whether the held-out instruments actually catch a real hacking attempt in the wild is unmeasured. I'd rather say that than imply a track record we haven't earned, which would be precisely the failure this whole apparatus exists to prevent.

The instruments need their own discriminating-power check. This one I owe to the review panel, and it's the sharpest thing they surfaced. A held-out metric that is silently broken (measuring nothing, wired to a dead code path, computed over an empty set) sits there reading perfectly acceptable forever. It looks identical to a held-out metric that is working and finding nothing wrong. That is the always-pass degenerate failure from earlier in this piece, reappearing one level up, inside the very instruments installed to catch it. The fix is the same fix: perturb the control deliberately, confirm the instrument can move, and only then trust it when it doesn't. An instrument you have never seen react is not evidence of a clean result. It's an unread dial.

And most importantly: this does not prevent reward hacking. It makes it visible. A determined optimizer can still push the target metric while the held-out instruments scream, and a team that has learned to ignore screaming instruments is back where it started. The mechanism buys you the ability to notice. Noticing is a human act.

One recommendation from that same panel I'm declining, because it sounds right and would quietly destroy the mechanism: several reviewers proposed folding the held-out metrics into a multi-objective reward. Optimize the vector, find the Pareto frontier. That converts observation into optimization. The instant a held-out metric enters the objective, it acquires slack of its own and joins the attack surface it was built to watch. The non-negotiable property here isn't that there are many numbers. It's that some of the numbers are never targets.

The part that generalizes past machines

The reason Yang's framing has stayed with me is that he applies it in the direction people usually don't. Reward hacking isn't a quirk of reinforcement learning. It's a property of any system where effort is directed by a measured objective: every engineering organization, every OKR, every dashboard reviewed in a weekly meeting.

If your team is measured on one number, you will get that number. You will not necessarily get the thing the number was standing in for, and the more competent the team, the faster the gap opens, because competence is exactly the ability to find the efficient path, including the efficient path to the measurement.

The defense is the same in both domains, and it isn't more discipline or better intentions. It's structural: keep instruments you never optimize, declare in advance what movement would invalidate the result, and be able to say why those instruments are out of reach of the thing being pushed. Then, when the target improves and a held-out metric goes the wrong way, you have pre-committed to what that means. Before you had any stake in the answer.

That last clause is the whole mechanism. Anyone can evaluate evidence fairly before they know which way it points.

Matrix CR Studio builds verification infrastructure for post-quantum migration and sovereign AI systems. The kind of work where "looks done" and "is provably correct" are different planets, and only the second one ships.