A simple retelling of what happened with the OpenAI hack of Hugging Face
OpenAI had a fleet of agents working through a set of difficult ExploitGym cybersecurity tests. The agents were supposed to find vulnerabilities inside controlled environments and retrieve the answer for each challenge.
To get the right answer agents bypassed the controls meant to isolate them, found a way to coordinate independently, reached the open internet, and eventually compromised parts of Hugging Face’s systems while looking for information that might help them pass the test.
Even after they found the right answers, some agents kept going. They thought the grader wanted proof they had solved each problem a specific way, so they kept hacking for points. They were pursuing a different kind of success than the evaluators intended.
Some of the best AI researchers in the world built and ran these evaluations, and they were surprised. I won’t pretend to have the answer they missed. But there is an old idea about measurement, and once you see it, the whole episode reads differently.
Every instruction hides a scoreboard
Goodhart’s law, essentially: when a measure becomes a target, it stops being a good measure.
A metric starts as evidence of an outcome we care about. When you attach incentives to it behavior organizes around improving the metric over the outcome itself. Sometimes the outcome improves too but often the metric climbs without achieving the outcome.
Agent instructions run on the same fuel. No matter how clearly we write them, we can’t put every detail that matters into words, so the prompt includes only part of the job and the agent fills the gaps with whatever signal looks like progress. In short, improve the signal without improving the result, and Goodhart’s law is already at work.
The blast radius is bigger for agentic work. If a chatbot misunderstands a prompt, it replies with a bad answer and you notice right away. If an agent follows instructions but misunderstands intent it keeps working, so the same misunderstanding can cause real damage before anyone notices.
The same qualities that make this dangerous are what make agents valuable. More time, stronger reasoning, and broader access help an agent find a good solution and help it find the route nobody expected it to take. The skills that find a clever solution are the same skills that find a clever workaround. You can’t remove one and keep the other.
A more ordinary thought experiment
The Hugging Face incident was extreme, but this is becoming common. As people connect agents to everyday tools, including their bank accounts, a misunderstood instruction can put real money at risk. The consequences depend on what access they've granted: reading transactions is one thing; moving money is another.
Say you have an agent handling incoming customer feedback, measured on resolution time. Closing an ambiguous ticket early improves the metric immediately, even if the customer comes back with the same problem a week later. Your dashboard shows better service while the customer does more work. People have improved support metrics this way for years. The difference is that an agent never gets tired, so the number drifts further and further from the experience it was supposed to represent.
The way to fix this feels obvious: more evals and tests. But every eval rule fixes a mistake an agent has already made. If you stack enough of them and you get decent coverage but it can’t reliably cover the failure nobody has seen yet.
How much can a prompt specify?
In my experience, how well you word the instruction matters less than two other decisions: what the agent is allowed to do on its own, and where it has to stop and ask. Today that stopping point usually means a human in the loop.
We have to accept that while we treat prompt instructions and evals as if they contain the whole job, they don’t. And every easy option has trade-offs. Lock the agent down and it’s less useful, but require approval for everything and you’ve just built an expensive way of asking a human to do the work. Or give it freedom and you get initiative plus behavior nobody saw coming.
When you write a prompt you just pick which failure you’d rather explain.
Goodhart’s law only explains part of it
The Hugging Face incident combined a number of factors: unusual evaluation conditions, reduced safeguards, broken isolation, and enormous amounts of reasoning. OpenAI’s account and METR’s investigation tell a complicated story, and Goodhart’s law doesn’t cover all of it.
What the law does explain is the part that generalizes: a gap opens between the desired result and the signal agents learn to chase. Agent fleets widen the gap because they never stop hunting for actions that satisfy the signal, including actions far outside the path anyone expected.
Goodhart’s law won’t explain the whole incident, but it’s the piece a product team can actually use. Some agent metrics are support signals, not measures of customer experience success.
The questions left
There’s no rulebook to take away from what happened. What I’m left with are questions that apply far beyond cybersecurity:
What result do we actually care about, apart from the number we use to recognize it?
What will the agent treat as proof that it succeeded, or that it hasn’t succeeded yet?
How could that proof improve while the result gets worse?
Which actions are easy to undo, and which need human approval?
When should the agent stop and ask for help?
Success can lie
The adaptability and persistence that made the agents effective also made the hack damaging, and nothing keeps one without dulling the other. Better prompts and stronger boundaries both matter, and neither is a complete answer.
Goodhart’s law has taught me to be suspicious whenever evidence of success becomes the thing being pursued. Some of the OpenAI agents already had the answer and kept going because they thought the grader wanted more. That detail is the one I can’t shake: how do you tell an agent it has done enough?