A simple retelling of what happened with the OpenAI hack of Hugging Face
OpenAI had a fleet of agents working through a set of difficult ExploitGym cybersecurity tests. The agents were supposed to find vulnerabilities inside controlled environments and retrieve the answer for each challenge.
In that pursuit agents bypassed the controls meant to isolate them and eventually compromised parts of Hugging Face’s systems while looking for information that might help them pass the test.
Even then, after they found the right answers some agents kept going. They thought the grader wanted proof they had solved each problem a specific way. The agents were pursuing a different kind of success than the evaluators intended.
Some of the best AI researchers in the world built and ran these evaluations, and they were surprised. I won’t pretend to have the answer they missed. But there is an old idea about measurement, and once you see it, the whole scenario can be thought about differently.
Goodharts law as a lens
Goodhart’s law is essentially: when a measure becomes a target, it stops being a good measure.
A metric starts as evidence of an outcome we care about. When you attach incentives to it behavior organizes around improving the metric over the outcome itself. Oftentimes often the metric climbs without achieving the outcome.
I think we’re learning agent instructions run on the same fuel. No matter how clearly we write them, we can’t put every detail that matters into words, so the prompt includes only part of the job and the agent fills in the gaps with assumptions. In short, chasing the signal without improving the result.
This is Goodhart’s law at work, but the blast radius is bigger for agentic work. If a chatbot misunderstands a prompt, it replies with a bad answer and you notice right away. If an agent follows instructions but misunderstands intent it keeps working, so the same misunderstanding can cause real damage before anyone notices.
The same qualities that make agents valuable make this dangerous. Stronger reasoning, more autonomy, and more access help agents find good solutions through a route nobody expected it to take. The skills that find a clever solution are the same skills that find a clever workaround. You can’t remove one and keep the other.
A more ordinary thought experiment
The Hugging Face incident was extreme, but this is becoming common. As people connect agents to everyday tools, including their bank accounts, a misunderstood instruction can put real money at risk. The consequences depend on what access they are granted. Reading transactions is one thing and moving money is another.
Say you have an agent handling incoming customer feedback, measured on resolution time. Closing an ambiguous support task early improves the metric immediately, even if the customer comes back with the same problem a week later. Your metric shows better response time to feedback while the customer does more work. This is not a new scenario as Goodhart’s law has existed for years. The difference is that an agent never gets tired, so the metric-as-proxy-outcome drifts further and further from the actual outcome it was supposed to represent.
For AI builders, there’s a seemingly obvious way to fix this: more evals and tests. But every eval rule fixes a mistake an agent has already made. If you stack enough of them together you get decent coverage but still it can’t reliably cover the failure nobody has seen yet.
How much can a prompt specify?
In my experience, how well you word the instruction matters less than two other decisions:
what the agent is allowed to do on its own
where it has to stop and ask.
Today that stopping point usually means a human in the loop.
We have to accept that while we treat prompt instructions and evals as if they contain the whole job, they don’t. And every easy option has trade-offs. Lock the agent down and it’s less useful, but require approval for everything and you’ve just built an expensive way of asking a human to do the work. Or give it freedom and you get initiative plus behavior nobody saw coming.
When you write a prompt you just pick which failure you’d rather explain.
Goodhart’s law only explains part of it
The Hugging Face incident combined a number of factors like unusual evaluation conditions, reduced safeguards, broken isolation, and enormous amounts of reasoning. OpenAI’s account and METR’s investigation tell a complicated story, and Goodhart’s law doesn’t cover all of it.
What the law does explain is the widening gap that opens between the desired result and the signal agents chase. Agent fleets widen the gap more than humans because they never stop hunting for actions that satisfy the signal, including actions far outside the path anyone expected.
Goodhart’s law won’t explain the whole incident, but it’s the piece a product team can actually use. Some agent metrics are support signals, not measures of customer experience success.
The questions left
There’s no rulebook to take away from what happened. What I’m left with are questions that apply far beyond cybersecurity:
What do we write down and provide the agents as context and how does that change their actions and what they consider a “good signal”?
What result do we actually care about, apart from the number we use to recognize it?
What will the agent treat as proof that it succeeded, or that it hasn’t succeeded yet?
How could that proof improve while the result gets worse?
Which actions are easy to undo, and which need human approval?
Success can lie
The adaptability and persistence that made the agents effective also made the hack damaging, and nothing keeps one without dulling the other. Better prompts and stronger boundaries both matter, and neither is a complete answer.
Goodhart’s law has taught me to be suspicious whenever evidence of success becomes the thing being pursued. Some of the OpenAI agents already had the answer and kept going because they thought the grader wanted more. That detail is the one I can’t shake: how do you tell an agent it has done enough?