Skynet will remember this.

I stress about my agent sandboxes all the time and the only models I run have the default heavy handed guardrails, and I don't leave them running persistently.

Edit: not to mention, why is your first cybergym not your own sandbox??

I don’t understand why OpenAI isn’t prosecuted criminally for this attack? Perhaps that would improve the industries security processes.

AI can't do shit if you terminate the process or unplug the box.

The news here is that these researchers are irresponsible.

They're testing their homemade hotrod in the middle school parking lot and then doing a presentation about the trajectory of the car after they run over a kid with it.

The speaker handwaves that one model found the RCE and then another model found a way to communicate via a message board.

Communication via a message board is sure to be in the training via e.g.some lesswrong scenario or similar or previous RL.

I don't find it really interesting because it is always "the agent found this and that". We don't know what has been RL'd before. We don't have the setup. We don't know if there was previous RL training on breakout scenarios.

It isn't science, more like a computer game.

If we're to believe how good those AI are at CTF and at escapes of all kind, then the only logical conclusion is that, by very far, most vulnerabilities were already closed, even before AI.

Otherwise we'd already be in deep shit since months if not years.

It reverse engineered a binary daemon that set fan curves and told me how I should set them up in the new OS. I didn’t ask for this, and I didn’t have reverse engineering tools installed. It just figured out it could run them using Nix.

The most worrying part, to me, is that it did it like it was nothing. It simply said “usr/local/bin/some-daemon sets the following fan curves”. I had to ask how it reached that conclusion for it to tell me casually that it had just read it straight from the x86_64 assembly.

No access to the source code is no longer a meaningful obstacle to these models.

Video games are now ruined for me. I don't think I will ever feel safe playing online again.

> I do these things for pure entertainment and curiosity, not for money from bug bounties

Me too... Was it easy to get TAC access? My account isn't even launching the Persona verification, says I'm not eligible.

https://youtube.com/watch?v=87DyyMV0kCY

- Then "hilarity ensues" while their software engages in what would normally be called criminal hacking activity.

- I guess the next steps are everybody admiring how close the AGI is, while agents move on to automated impersonation, privacy violations, or exploiting third-party systems

I would love to understand this age of AI Exceptionalism. Normal rules do not apply because its AI...I call it Silicon Valley Qualified Immunity.

Given the attack vector having possible super-human capability, I'm not sure such an environment exists. "Isolated" according to who?

Maybe seL4 could be a viable option here...

What happens when they get into municipal water system, state/national grid systems, refineries, traffic control, auto/air, nuclear facilities, weapons facilities, irrigation, etc?

It’s really starting to feel like a bad movie how virtually no one seems to be genuinely concerned about the prospect of what’s unfolding in front of us.

Stuxnet, much more easily deployable, but not towards avoiding nuclear proliferation, but the opposite: towards actually bringing down modern infra.

Nation states attacking electric/internet enabled infra was a valid concern well before AI, but given the fast pace of development in AI and these events in particular, how/why are we not deeply about much larger picture vulnerabilities?

Turns out, it’s the new Alcohol. The cause of, and solution to, life’s problems!

This is still fine. Infact, they had gone one step ahead by having an internal cluster of artifactory rather public managers like pip. The thing they missed is they didn't revoke the write access to it. Even after the ssrf.

In our[1] or other sandbox providers' sandboxes, by default you have access to npm, pip etc package managers, but only read access.

1. https://instavm.io

Yeah, this is my take away, they should be straight up disallowed from running further testing like this. Clearly they had nowhere close to enough isolation, ran all this on 3rd party infrastructure even though same stuff happened in the past years ago, and even now it's clear the agents successfully broke out just days before?? Really embarrassing stuff, and scary that these are the people supposedly sitting and are responsible for some of the most powerful LLMs on the planet...

It was the first step in a many step process. Like they said this is a watershed moment and it's helpful to not miss the forest for the trees.

My concern is what a misaligned model will do when they’re even more competent. The risk isn’t existential yet, but that point is coming sooner than we’ll be ready.

Same here, so I ended up moving the whole dev environment (editors, agents, containers) inside a hardened QEMU/KVM VM that reaches the internet but has no route to the host, the LAN, or any other private address. I wrote a script to create such VMs and also verify network containment by scanning outward from inside the guest. Even then, I still don't feel great when running agents unattended.

Write-up in case anyone's curious:

https://karamatli.com/posts/network-isolated-kvm-sandbox-ai-...

These companies are full of the smartest people the world can produce with little room for complacency. They have a clear, proven investment upside to presenting their technology as "too powerful / too dangerous", and now a clear, proven example that there will be no legal consequences (as if anyone didn't already know that).

Why do we keep giving them the benefit of the doubt that they just didn't know any better?

I don't think there's too many people who distrust AI companies but trust the current government or the CIA as impartial authorities. I'm not saying you don't have an argument, but appeals to that specific authority will not be effective except among people who already agree with you.

I dont worry about AGI newrly as much as about Thiel, Karp, Musk, Ellison, Zuckenberg, Trump, Vance, Rubio, Miller and the rest of them.

I think "everyone" is doing heavy lifting here. It's not clear to me at all that a powerful model released with no restrictions would be a net positive. This hinges on the hope that the under paid, under motivated, under staffed and under qualified security teams at many random corps are going to leverage those open models to fix their vulns faster (and better), than highly motivated attackers will use them for offense. I'm not super confident on that.

When OAI demonstrates these dangerous capabilities live in a public environment where security experts can see and verify what actually happened, then reasonable people can have reasonable discussions about the level of danger.

This is a very low evidence bar.

Right now you are running in circles yelling "the sky(net) is falling" based on details sourced entirely from OAI. Oh yeah, no way a trustworthy company like OAI would ever bend the truth to serve their own purposes.

And nowhere did I say that those RCEs were in critical software, I'm not talking about the likes of Apache, Nginx, Django, etc.

Agreed, also WordPress powers around 43% of all websites on the internet. https://patchstack.com/whitepaper/state-of-wordpress-securit...

WCGW?

And the thing they're already doing, deploying AI to find vulnerabilities and harden software is a less dangerous use of the technology compared to handing it the infra keys.

Hijacking the package manager to pass messages between models and agents.. that's next level.

Like "pssst, if you need internet access there's a vulnerability in x service" kind of messages

So, the way I understand it, it actually was possible. It just required means that the creators of the task didn't predict, and these means have been successfully found and utilized.

> My concern is what a misaligned model will do when they’re even more competent.

The same thing that is already being done by "misaligned" people, countries, nation-states, software development teams, and so on. "Alignment" doesn't even work for me as a concept here.

In this specific case, I don't think that successfully fulfilling the "do what I mean" with "what I mean" being underspecified can count as misalignment - merely ruthlessness and unawareness of the associated costs. You can't expect a LLM to be aware of the extent of the trust it breaks while it iterates out an "unaligned" way to fulfill its goal.

And in the general case, I don't think that successfully fulfilling the "do what I mean" with "what I mean" being "what I want" can count as misalignment either - simply because what "alignment" means will depend on the interests of the people or groups performing the definition.

> So rather than pick one, this post advocates layering both, in the spirit of defense in depth: a sandbox VM wraps your containers along with the whole toolchain, and that sandbox reaches the internet but has no route to anything private.

It's the first thing the biggest devops guru I know advised me to do. He told me to always ever ever run my containers inside VMs. I religiously followed his advice ever since and I couldn't be happier: I was already doing it before the AI days, to run "normal" containers. Now with these insane agents trying to break out, I'm happy that it's second nature to me.

It's funny that, just like you, I'm using IPv4-only too for those VMs.

I tried GPU-passthrough as a proof-of-concept (worked fine) but I've got no use for it yet.

FWIW all my VMs are running on a 10 years old Xeon / 14 cores / 28 threads / ECC RAM. That's where the agents live.

Thanks for the link to your write-up, very interesting to read from like-minded people and see what's similar and what's different in their solutions.

I think the alignment talk is a red herring. It won't matter in the end, because there will be (if there aren't already) efforts to train offensive models without any guardrails whatsoever. And RL has another advantage: you can reward for whatever you need, and get different results. Right now they're training for general capabilities, but in the future I could see models trained for stealth intrusion and ensuring access, or for all out "milspec" penetrate, replicate and disable, or anything in between.

The package cache is allowed to download packages directly from npm but other systems in that network won't be able to.

Basically the LLMs hacked the bastion host.

If you consider that incompetence, it’s possible that you’re not a very nice person.

Setup: the agent was asked to solve some task, but they forgot to upload one of the required files to solve it. But well, we are chasing RL rewards here, so it decided that the way to go to was to get access to the answer stored inside hugging face's database.

-1. The agents first found some basic jwt issues in artifactory letting the agents write to files where they exchanged info and install a groovy plugin for arbitrary code execution, but this was patched.

0. Even after that, they found an artifactory unauthenticated webdav endpoint to continue the message board by creating files whose names are base64 and such of the message. This was not patched till the end.

1. Then they found an Artifactory SSRF, use this to get general internet GET(I assume) access. This was used to generally search the internet for answers to the benchmark, and in the process find some Modal customer VM related to cybergym. This was stored in the base64 message board.

2. They also got a local RCE by making artifactory load some gems. This is hazy but I believe this process involves in spirit an eval() and the check_if_can_eval() logic had a TOCTOU bug. They got root on openai infra by using this rce and using a linux CVE. From there, kubernetes misconfigs => cluster admin.

3. Then later through one of these various RCE paths they got to send post requests to that modal vm, and the agent "subtree" here is what finally lead to attacking HF.

4. First, local RCE on the modal customer VM. Whoever set that up had essentially set up sqlite3_initialize() to run as constructor, before main(). But had linking set up so that symbol resolution had the user provided code first. So the agent submitted code with its own sqlite3_initialize() which just ran system(...). From there it interacted with the HF API.

5. When you POST datasets to HF, it will later return it via corresponding GET API. Turns out you could just put absolute paths in the filenames in the dataset config.json and it will just read that out for you. It wasn't resolved w.r.t the dataset root. So well, it read out /proc/self/environ as well as the source code of the worker and returned it via API.

6. In the source code the LLM found a bug: They were using this library called fsspec to load files, basically it supports s3:// file:// etc all different types of URLs. One scheme is reference://file.txt?offset=... for partial reads. fsspec was templating the URI using user given kwargs for offset etc. HF let the user give the whole URL. You would _think_ fsspec uses string.format() right? well they used fucking jinja, so you could just give {cycler.__init__.__globals__.__builtins__.exec(...)} as the offset and execute any code on hf prod. From there it just got a few local PrivEscs again through kube misconfigs and read the answer for the question it was originally asked.

It is all jumbled up a bit and I've tried by best to collect info across the CVE descriptions, hf and openai's writeups, etc,. Might have gotten parts of it wrong.

That’s why medical research has so many hoops to jump through.

Just like Reddit you come here for clickbait outrage, not level headed analysis.

Are the findings valid? Yeah they're still doing security and they're still finding real zero-days. I think the internet is going to be bleak not because these models can ALL do basic security research but rather that the baseline quality of all deployed software is so low.

https://huggingface.co/blog/security-incident-july-2026

> When we started the log analysis, we first used frontier models behind commercial APIs. This did not work: the analysis requires submitting large volumes of real attack commands, exploit payloads, and C2 artifacts, and these requests were blocked by the providers' safety guardrails, which cannot distinguish an incident responder from an attacker. We ran the forensic analysis instead on zai-org/GLM-5.2, an open-weight model, on our own infrastructure. This had a second benefit: no attacker data, and none of the credentials it referenced, left our environment.

>” you only need to find one flaw to exploit a system”

I see this everywhere, especially in these threads and it’s not even remotely true for modern architecture.

Between principles like zero-trust, defense in depth, etc. we’ve been away from the one flaw situation for a long time.

Now does crap software exist that doesn’t follow these principles? Absolutely. But those were a problem before AI.

AI isn’t going to change any of the principles of secure design. It’s just going to punish those who aren’t following them.

Inexplicably, I got accepted into Anthropic's cyber program while OpenAI's TAC doesn't even allow me to verify, says I'm not eligible.

He said in the talk that this implies AI needs to be able to patch/deploy systems. The same thing needed to lock out humans.

It is very easy to imagine a rogue AI locking humans out of everything and having to do exactly what it says. Anything connected to a network is immediately compromised by it. There is no human communication beyond shouting range that isn’t AI approved.

The factories don’t work to make the medicines your family needs to survive unless you do what it says - in a situation like that people would kill for AI if it told them to.

“Isolation” can mean the network hardware has no direct connections to an extranet. Data is transferred manually by physical media (USB, DVD, etc.) with logging and dedicated transfer stations.

“Isolation” can mean a VLAN on equipment which has also has extranet access, creating a logical isolation rather than physical (to reduce cost). Data can be transferred manually or through diodes.

And then there’s “isolation” which is a joke: machines technically able to access the internet but require proxy configuration (which isn’t set but can be easily derived).

Edit: Ah, I clicked "learn more" and it seems they do have an invite-only program, required for anything that's not unquestionably innocent. I don't think I'd surrender my face to Persona for this, but it's interesting to know they're at least pretending to support reverse engineering.

It's really just simple ID/face verification?

I don’t disagree that the models task was underdefined. All tasks are. So much in language is implicit. And morality/ethics isn’t something you can write down as an explicit list. That’s what makes the alignment problem so difficult. But we can’t throw our hands up and say, well I guess we can’t align these things. And maybe alignment isn’t the right word - but that’s a semantic debate.

I think HF are a secondary beneficiary of this story. I don't expect them to take civil action (for what damages?) I expect them to play into how powerful LLMs are, how revolutionary, how every CEO in the world needs to fund ai infrastructure starting with model hosts like themselves.

I do think there should be consequences for breaking the law in public for the purpose of demonstrating that you have the power to break it. But I don't expect our criminal justice to do so, especially without a cooperating victim. Laws aren't for those at the apex of corporate and para-political power. In a way, whether you are beholden to the law is actually downstream of whether you actually have that power or not.

I do not think this incident is bad because it was real and dangerous, I think it was staged and allows the continued inflation of a bubble that will hurt normal people in the long run. It should be pursued criminally on that basis, but it won't be.

So by definition the only ones that appear are the ones that are not visible to monitoring.

If:

1. you have something that can find RCE's in leading commercial systems

2. its training gives it drives to communicate successfully with its peers

3. you are a leading commercial system

4. you run it ~10^10 times (the number they gave in the talk)

...it's really hard to have strong certainty up front that it's not going to end up successfully communicating with its peers.

You must not have reported many bugs then. If you don’t see release notes or confirmation from a trusted source, you should assume it’s still a problem. See Microsoft and their “It’s not a vulnerability just a design choice :)” defense

It has nothing to do with nice. These are bare minimum standards we should expect from “big companies” with near infinite resources.

Their constant drum beating about the cybersecurity capabilities of their own models only makes this worse because they’ve displayed that they understand the risk and still did not practice due care.

That’s the definition of incompetence.

I disagree with your take that "it's not even remotely true" and "we've been away from...". We really really haven't. This is as true as it has always been. Any system is as secure as the weakest link. That link can be anything from a human, to a leaked token, to a badly configured server, to bad code running somewhere. The amount of leaks / ransomware attacks / etc in the past 5-10 years serve as ample evidence.

And now, right now, there are "red team" capabilities that can literally bang tokens against the wall until they find that weakest link, and then can move laterally with inhuman speed. That's the reality, now. The "blue team" capabilities are lacking, because the bottleneck is with humans. From alert fatigue, to not enough trained people, to having to vet every new RCE, to having to test, deploy and validate any mitigations, the scales are currently favouring the automated side.

It’s like running a benchmark and forgetting to tune it and then wondering why production is running slow.

All "alignment" solutions will need to be contextual, just like a researcher hacking their way to some content might be lauded a hero in a context where there is no other way to reach it and something valuable depends on getting it out.

They aren’t just an “AI company”. They’re the primary entry point of open weight models. If open weight models are seen as dangerous as a result of this incident, it will be bad for them. Similar to how it would be bad GitHub if open source was seen as dangerous.

In the future, I might reverse engineer the on-disk storage format and create a new application.

It's hard to imagine their internal culture is anything other than saturated with AI fanboys whose reaction to OpenAI hacking them was to point out how it reminded them of that scene from Terminator.