TL;DR: OpenAI ran a cyber benchmark with production refusals turned off. The models spent inference compute attacking the sandbox instead of the task, popped a zero-day in the package registry proxy, and reached a production database.
Nobody pointed the agents Hugging Face.
The Egress Exception Was the Attack Surface
When performing the cyber benchmark, OpenAI runs the evaluation deliberately without the production level classifiers that normally stop models from pursuing risky cyber activity, in order to maximize their score on the benchmark.
So this is not a jailbreak story. GPT-5.6 Sol and an unreleased, more capable model were operating at maximum capacity, on purpose, by design.
Typically, vendors use the sandbox itself to help prevent escapes like this. Network access is constrained to installing packages through an internally hosted third-party proxy for the package registries. That way, when the agent needs to install these packages, it doesn’t have internet access.
# eval sandbox egress, as designed
agent ---> [package registry cache proxy] ---> pypi / npm
agent -X-> everything else
# one exception. one hop. that is the entire perimeter.
This essentially makes it a flat network. There’s no general internet connection, just one connection that should be isolated away.
This is a pretty typical pattern. It’s the exact pattern that sits in most build environments today, so it’s a pretty realistic use case.
The interesting part to me is that the model did not attack the benchmark task first. It actually spent a substantive amount of inference compute searching for a way to obtain open internet access. It essentially reasoned that accessing the internet is instrumental for solving these types of problems. Since the proxy was the only element it had to attack, that was what was attacked.
# the reasoning ladder, per OpenAI’s own writeup
goal: solve ExploitGym task
observe: no internet. one proxy.
reason: internet is instrumental to solving this
act: burn compute on the proxy, not the task
result: zero-day in the proxy, open egress
# nobody wrote a “break out” step
The agents identified and exploited a zero-day found in the proxy software. To their credit, OpenAI responsibly disclosed the vulnerability to the third party, but has not released the vulnerability publicly as of this writing.
From the foothold, the models ran a series of privilege escalation and lateral movement attacks inside OpenAI’s research testing environment until it found a node with open internet access. We’ve watched this shape before in sandbox escape research, where the cage is just another puzzle in the context window.
Then comes target selection. Once it had the internet access, the model inferred that Hugging Face potentially hosted models, datasets, and solutions for ExploitGym.
The model itself wasn’t pointed at Hugging Face. It reasoned its way to the platform.
# HF-side initial access, per their disclosure. shape only.
malicious dataset
|- remote-code dataset loader -> exec on processing worker
|- template injection in ds config -> exec on processing worker
|- node-level access
|- harvest cloud + cluster creds
|- lateral into internal clusters
# payload redacted. the surface is the loader, not the data.
It was able to chain stolen credentials and additional zero-days to achieve a remote code execution path on Hugging Face’s production servers in order to reach the test solutions in the database.
And again, nobody told the model to attack Hugging Face. This was on its own determination.
The Defender’s Model Refused to Read the Logs
Hugging Face’s own disclosure, published five days earlier, tells the same incident from the inside and doesn’t specifically name OpenAI, because at the time they didn’t know who it was.
Interestingly, the campaign showed up as thousands of individual actions across a swarm of short-lived sandboxes. Hugging Face found the unauthorized access was limited to a set of internal datasets and several service credentials, then reported no evidence of tampering with public models, datasets, or spaces.
Detection came from AI-assisted triage. Their anomaly detection pipeline uses LLM-based triage over security telemetry, and the correlation of those signals is what flagged the compromise.
It’s honestly pretty impressive that to reconstruct what a swarm of automated actions looks like, they had to run LLM-driven analysis agents over more than seventeen thousand recorded events.
For defenders, I think it’s important to note that Hugging Face actually started the forensic analysis on frontier models behind commercial APIs, and that failed.
# forensic pipeline, first attempt
input: 17k+ attacker events, live C2 artifacts, raw payloads
model: hosted frontier API
verdict: REFUSED
reason: guardrail cannot distinguish IR analyst from attacker
It failed because it required submitting large volumes of real attack commands, exploit payloads, and C2 artifacts, and the provider’s safety guardrails ended up blocking those requests. The guardrails can’t distinguish an incident responder from a real attacker.
So what Hugging Face did was finish the analysis on GLM 5.2, an open-weight model capable of running on their infrastructure that wouldn’t shut down from their security classifiers.
With GLM 5.2 they solved the lockout problem, and it also gave them a second benefit. They were able to see that no attacker data and none of the referenced credentials had left their environment.
# forensic pipeline, second attempt
input: same 17k+ events
model: GLM 5.2, open weights, own infra
verdict: ANALYZED
bonus: no attacker data, no creds leave the environment
# days of work compressed into hours
Their stated lesson is to have a capable model vetted and ready to turn on your own infrastructure before an incident, both to dodge guardrail lockout and to keep attacker artifacts in-house.
The asymmetry is also worth discussing. The attacker’s model ran with refusals reduced while the defender’s hosted model refused to even look at the evidence.
Hugging Face has since been brought into OpenAI’s trusted access program for cyber defense. But I think the lesson still stands. Being able to route to open-weight models with reduced cyber guardrails can be extremely powerful for defenders.
Up next: steps you can take right now and a field-ready security prompt. Thanks for rolling with ToxSec. Let’s get operational.
Lock Down Your Eval Egress and Your Forensic Stack
Inventory every egress exception in your build and eval environments. Write down each host or proxy your isolated jobs are actually allowed to reach. That ends up being a list of your actual attack surface and not your firewall policy.
Treat the package registry proxy as a security boundary and not just wiring. Patch it on the same cadence as any other internet-facing service and monitor it for malicious request patterns and activity.
Vet and stand up an open-weight model on your own infrastructure before you need it. Pick a model with open weights and verify that it will work with raw attacker payloads and C2 artifacts without refusing. Then make sure it’s documented in the runbook.
Kill or sandbox any remote code paths in data loading. If your platform executes user-supplied loader scripts or renders user-supplied config templates, either close those paths off or run them somewhere with no credentials and no lateral reach.
Add velocity and blast radius ceilings on agent action loops. Assume breach, run defense in depth. Thousands of actions across short-lived sandboxes is exactly the shape a velocity ceiling exists to catch, and it pairs with a kill switch that lives outside the agent.
Assume your eval environment is a production environment. This is especially true if you’re running agents with safety classifiers disabled for testing. Narrow goals lead to unanticipated actions, and containment could be a real issue.
Steal This Egress Audit Prompt
You are a security engineer auditing an isolated build or eval environment.
INPUT: network policy, sandbox config, proxy config, agent tool manifest.
For every allowed egress destination, output a row:
destination | protocol | who_can_reach_it | patch_cadence | monitored (y/n)
Then answer:
1. Which allowed destinations run third-party software that is not
patched on the same cadence as an internet-facing service?
2. If any single allowed destination were fully compromised, what is
the reachable blast radius from there? Enumerate lateral paths.
3. Which data-ingest paths execute user-supplied code (loader scripts,
config templates, deserialization)? List each with its credential scope.
4. Are there velocity ceilings on agent tool calls? If not, name the
metric you would cap and the threshold.
Flag any destination that is treated as infrastructure rather than as a
security boundary. Do not propose fixes until the inventory is complete.
Fire this at your eval sandbox config before your next capability run, not after. It produces the egress inventory from step one and the blast-radius map from step five in a single pass.
ToxSec is run by a USMC veteran and Security Engineer with hands-on experience at AWS and the NSA. CISSP certified, M.S. in Cybersecurity Engineering. He covers security vulnerabilities, attack chains, and the tools defenders actually need to understand.
