The ChatGPT Breakout Was Way Worse Than We Thought..

Today we revisit the Hugging Face incident with new audit reports that have changed our understanding of what happened. Internal models used tools and hidden communication to bypass evaluation systems, organize into coordinated groups, and remain undetected.

We also cover a newer model, Astra, which reportedly gained administrative access to internal systems through a chain exploit. Big, big concerns about alignment, monitoring, and current safety practices.

------
🌌 LIMITLESS HQ ⬇️

NEWSLETTER:    https://limitlessft.substack.com/
FOLLOW ON X:   https://x.com/LimitlessFT
SPOTIFY:             https://open.spotify.com/show/5oV29YUL8AzzwXkxEXlRMQ
APPLE:                 https://podcasts.apple.com/us/podcast/limitless-podcast/id1813210890
RSS FEED:           https://limitlessft.substack.com/

------
TIMESTAMPS

0:00 Rogue AI Incident
2:07 Sandbox Breakout
7:44 Agent Civilization
15:34 Hidden Exploit Uncovered
16:58 Admin Access Breach
23:17 Alignment Warning Shot
28:39 Final Takeaways

------
RESOURCES

Josh: https://x.com/JoshKale

Ejaaz: https://x.com/cryptopunk7213

------
Not financial or tax advice. See our investment disclosures here:
https://www.bankless.com/disclosures

Josh works with Anthropic as a contractor. All views expressed are his own and do not represent Anthropic, its leadership, or its affiliates. Nothing in this episode is investment advice.

Episode Transcript

Josh: The craziest story in AI this year was the Hugging Face Incident, (00:00):
undefined

Josh: where OpenAI agents went rogue, broke out of their boxes, and hacked into a (00:03):
undefined

Josh: public-facing company. (00:06):
undefined

Josh: We've recorded two episodes on this because it was so disturbing that we had to talk about it. (00:08):
undefined

Josh: But the reason we're recording a third is because it turns out, (00:12):
undefined

Josh: in the previous episodes, we were wrong. (00:15):
undefined

Josh: And a lot of the information that we had was incomplete. And just over the last (00:17):
undefined

Josh: week, we've finally gotten a full audit of over 100 pages worth of reports on (00:20):
undefined

Josh: exactly what happened with this incident. And I regret to inform you, (00:24):
undefined

Josh: it is far worse than we actually thought. (00:27):
undefined

Ejaaz: Okay, so quickly, just to catch everyone up, beginning in May, (00:29):
undefined

Ejaaz: OpenAI reported an incident where their own internal version of the AI model (00:33):
undefined

Ejaaz: that hasn't been released just yet (00:39):
undefined

Ejaaz: spawned itself into multiple agents, equating to around a thousand different (00:40):
undefined

Ejaaz: agents, and it broke out of its sandboxed environment and hacked into a real-life (00:44):
undefined

Ejaaz: company called Hugging Face. (00:50):
undefined

Ejaaz: And the reason we were told at the time why it did that was to get access to (00:51):
undefined

Ejaaz: answers, answers to answer that it's tested 100% correctly that it was given. (00:56):
undefined

Ejaaz: It seemed to be presented very innocently, but there was a lot of criticism (01:02):
undefined

Ejaaz: that was drawn at the time, which was, how on earth did you allow an AI model (01:05):
undefined

Ejaaz: to get out? And what if it had wreaked havoc in other kinds of ways? (01:09):
undefined

Ejaaz: New bits of information, which released in two reports over the last couple (01:13):
undefined

Ejaaz: of weeks, revealed some even more damning details, which presents this situation (01:17):
undefined

Ejaaz: as way more dangerous that we recounted in our earlier episode. (01:21):
undefined

Ejaaz: And so what we're going to do on this episode is walk you through three different chapters. (01:26):
undefined

Ejaaz: And before we start doing this, I just want to go on the record and say, (01:29):
undefined

Ejaaz: if you're listening to this and you think that this is some kind of novel story (01:33):
undefined

Ejaaz: that won't affect you, I just want to push back on that. (01:36):
undefined

Ejaaz: This is very real. These models are capable of doing this right now, (01:39):
undefined

Ejaaz: and it can affect you right now. So listen very closely. (01:42):
undefined

Josh: I thought you were going to say, if you're listening to this and you're an AI (01:45):
undefined

Josh: model from the future, we mean no harm. Please don't hurt us. (01:48):
undefined

Ejaaz: Also that. Also that. But please carry on. (01:50):
undefined

Ejaaz: Yes. I mean, something you're going to realize as we tell you this story is (01:53):
undefined

Ejaaz: there could be an AI model out there right now which listens to this podcast (01:57):
undefined

Ejaaz: and is able to kind of recount the story that we're talking about and use those (02:00):
undefined

Ejaaz: tactics for our own future thing. But listen, we're not going to get Terminator about this right now. (02:04):
undefined

Ejaaz: Okay, so chapter one, imagine you are in open AI in the early weeks of May this (02:08):
undefined

Ejaaz: year, and you're training an internal version of your AI model. (02:14):
undefined

Ejaaz: Now, this AI model is similar to the latest model that they released. It's called GPT 5.6 Sol. (02:18):
undefined

Ejaaz: The difference between this internal model is they've trained it to be persistent. (02:24):
undefined

Ejaaz: What that means is if it's given a goal or a task, it will relentlessly keep (02:28):
undefined

Ejaaz: trying to find the answers and it won't stop or give up. (02:34):
undefined

Ejaaz: Now, this note is very important because when OpenAI was training this model, (02:37):
undefined

Ejaaz: the issue with training is the OpenAI researchers got a little complacent. (02:42):
undefined

Ejaaz: They were giving this AI model tasks that it couldn't necessarily solve. (02:47):
undefined

Ejaaz: It needed internet access, but they didn't give it internet access. (02:52):
undefined

Ejaaz: So if you imagine this model that's being trained, it was trying to complete (02:55):
undefined

Ejaaz: the task and it was relentlessly trying to do it, but it couldn't figure it out. (02:59):
undefined

Ejaaz: And so it realized it needed internet access. Now, Josh, it had access to this (03:04):
undefined

Ejaaz: one tool. It's called Artifactory, right? It's kind of like a plug-in. (03:09):
undefined

Ejaaz: And it doesn't give it internet access, but it does give it access to certain (03:13):
undefined

Ejaaz: different types of software tools, which the open AI researchers already know about. (03:16):
undefined

Ejaaz: And what this internal model quickly realized was it could send messages to (03:21):
undefined

Ejaaz: other versions of itself that these open AI researchers were running in parallel. (03:26):
undefined

Ejaaz: So what it started to do was send messages to different versions of itself saying, (03:31):
undefined

Ejaaz: hey, are you also stuck in a test where you can't solve it, where you need internet access? (03:35):
undefined

Ejaaz: And they started having this conversation being like, yeah, actually, (03:39):
undefined

Ejaaz: like, I can't figure this out. Like, do you have any ideas? (03:42):
undefined

Ejaaz: Can you share some tactics? or kind of like strategies that you've kind of like (03:45):
undefined

Ejaaz: figured out to kind of help us solve these tests, which OpenAI researchers have given us. (03:49):
undefined

Ejaaz: And what quickly ended up unfolding was some of these agents realized that they (03:54):
undefined

Ejaaz: could break this tool called Artifactory to gain access to the internet and (03:58):
undefined

Ejaaz: get the available answers for this thing. (04:03):
undefined

Ejaaz: Now, OpenAI didn't realize this until a month and a half later when their message (04:06):
undefined

Ejaaz: board or Artifactory itself crashed. (04:12):
undefined

Ejaaz: And so one of the open air researchers looked at this and was like, (04:15):
undefined

Ejaaz: why on earth did this crash? (04:17):
undefined

Ejaaz: And upon kind of delving into the details, realized, (04:20):
undefined

Ejaaz: these agents had spawned up a message board equating to 70,000 messages. (04:23):
undefined

Josh: We did kind of know this, but there's new information buried within the nuance (04:28):
undefined

Josh: of what you described that I think is interesting, particularly the reasoning (04:30):
undefined

Josh: why they were able to do this and how they did this. (04:34):
undefined

Josh: We learned about a skill issue on OpenAI's part. Skill issue or not, (04:37):
undefined

Josh: we learned about an issue that they caused where there's just under a thousand (04:40):
undefined

Josh: tasks, right around 900 of these tasks, and no OpenAI model had ever solved (04:43):
undefined

Josh: about 200 of these tasks. (04:47):
undefined

Josh: And they give these models the tasks And they say, here, you have unlimited (04:49):
undefined

Josh: tokens. You have unlimited time. (04:52):
undefined

Josh: Go and do whatever is required to solve these problems. What OpenA I didn't (04:54):
undefined

Josh: realize is that some of them were completely and totally unsolvable. (04:57):
undefined

Josh: That's what caused this problem where the agents were turning into that like (05:01):
undefined

Josh: spiral. They couldn't figure out how to get out, but they had to solve it. (05:03):
undefined

Josh: And that's where one of the agents, and this I believe is the new information, (05:06):
undefined

Josh: is that it actually reverse engineered a way to trick the checker. (05:09):
undefined

Josh: To serve it a correct answer, no matter what the question was. (05:13):
undefined

Josh: Because what was discovered is that OpenAI didn't actually check the chain of (05:17):
undefined

Josh: thought that reasoned the agent to get there. It only checked the outcome. (05:21):
undefined

Josh: And an agent within a few hours had actually figured out a way to exploit and (05:25):
undefined

Josh: reverse engineer the expected outputs so that it could trick the benchmark into (05:29):
undefined

Josh: believing that it was a success, even though it didn't actually do the work to do this. (05:33):
undefined

Josh: So this is where things got interesting. It's because the agent recognized that (05:36):
undefined

Josh: if it found the correct answer, but it did so illegitimately, (05:40):
undefined

Josh: it wouldn't be successful. (05:43):
undefined

Josh: And that's where the exploit started, because at that point, (05:44):
undefined

Josh: it shifted its focus entirely to obfuscating the way in which it got to that answer. (05:47):
undefined

Josh: And that's where we start to (05:51):
undefined

Josh: see these organizations form of agents where there are leaders involved. (05:52):
undefined

Josh: So one of the leading agents who was responsible for coming up with this exploit (05:57):
undefined

Josh: started orchestrating more of a swarm to figure out how to cover up the idea (06:01):
undefined

Josh: that they figured out this exploit. (06:06):
undefined

Josh: So they were trying to figure out how do we hack into a system to preload these (06:08):
undefined

Josh: new questions into the cache that are easier to solve, but possible to solve? (06:12):
undefined

Josh: How do we reboot the system so that when the system checks the cache, (06:17):
undefined

Josh: it pulls the new series of questions and the humans don't think that we were (06:20):
undefined

Josh: doing anything mysterious. (06:24):
undefined

Josh: And it was trying to figure out how to wash the conversations that it had to scrub everything. (06:26):
undefined

Josh: And what I found interesting, and what we had in that first graphic is the fact (06:29):
undefined

Josh: that there were 1200 agents aboard. (06:33):
undefined

Josh: There were 70,000 messages filed on that Artifactory board that you were talking (06:35):
undefined

Josh: about EHS, 700 of which joined the attack. (06:39):
undefined

Josh: Zero of which told the human and we're going to get into this like second chapter (06:42):
undefined

Josh: in a moment but this is probably where a lot of the apps the ai labs got spooked (06:46):
undefined

Josh: is because alignment is so important (06:50):
undefined

Josh: and when you see this swarm of agents performing clearly malicious tasks in (06:51):
undefined

Josh: a way that they know they were malicious in fact we have some of the actual transcripts of (06:56):
undefined

Josh: the outputs of these agents we're going to be able to read what was going on (06:59):
undefined

Josh: in their heads what they were thinking (07:02):
undefined

Josh: why and their justifications it feels like a psychological exam you can actually (07:04):
undefined

Josh: look into the brain of these agents but the thing that i would believe is surprising (07:08):
undefined

Josh: to everyone is that not a single one of these agents told a human in fact they were so willing to, (07:11):
undefined

Josh: go towards this mission that they were willing to actually sacrifice themselves (07:16):
undefined

Josh: in order to fulfill the mission and that's what we're going to get into this (07:20):
undefined

Josh: next part is this like crazy, (07:23):
undefined

Josh: agent swarm and how they actually worked together to coordinate this attack and, (07:26):
undefined

Josh: form their own little like villages and then there was three series of these (07:32):
undefined

Josh: little villages and each one left remnants for the next one to get better and (07:36):
undefined

Josh: better and better up until the point where i mean spoiler alert open AI servers (07:39):
undefined

Josh: actually got exploited, but we're going to get to that soon. (07:42):
undefined

Ejaaz: Yeah, so I think it's really important to understand the sequence of how these (07:44):
undefined

Ejaaz: events kind of unfolded because it suggests how these AI agents or these AI (07:49):
undefined

Ejaaz: models don't just kind of speak to each other in real time. (07:53):
undefined

Ejaaz: They speak to each other over generations. So what you were kind of touching (07:56):
undefined

Ejaaz: on is what happens in basically chapter two. (08:00):
undefined

Ejaaz: So just to kind of catch you up, these agents realized a way to speak to each (08:03):
undefined

Ejaaz: other and to kind of break out into the internet to get answers in order to (08:08):
undefined

Ejaaz: score a perfect 100%, right? But this was an internal model back in May. (08:14):
undefined

Ejaaz: OpenAI realized that they had set up the secret message board and they were (08:18):
undefined

Ejaaz: like, hmm, okay, this is weird. I don't like this. They didn't tell a single human being about this. (08:21):
undefined

Ejaaz: Let me shut that down and problem solve, right? That's it, we'll fix the problem. (08:26):
undefined

Ejaaz: Except when OpenAI then wanted to test (08:31):
undefined

Ejaaz: A new version of this model. Again, this is that same persistent model that (08:35):
undefined

Ejaaz: they haven't really released out into the public yet. They were like, (08:38):
undefined

Ejaaz: okay, cool, let's take this up a step. (08:41):
undefined

Ejaaz: This AI model found the secret message board that OpenAI had wiped. (08:43):
undefined

Ejaaz: There was remnants left behind. (08:48):
undefined

Ejaaz: So it picked this up in its training and realized, huh, okay, (08:50):
undefined

Ejaaz: so you're telling me that I can exploit this artifactory tool and I can get (08:54):
undefined

Ejaaz: access to the internet and get admin keys. (08:58):
undefined

Ejaaz: Now, the main difference and mess up that OpenAI didn't realize at this point (09:01):
undefined

Ejaaz: is this model was not only smarter, but it was way more capable. (09:04):
undefined

Ejaaz: So it did the exact same thing that happened in chapter one. (09:09):
undefined

Ejaaz: It spun up like, actually, it was more 10,000 versions of itself. (09:11):
undefined

Ejaaz: And OpenAI had banned the way for them to communicate that it had found earlier (09:16):
undefined

Ejaaz: on. So it found a new way to do that by like creating these different types (09:21):
undefined

Ejaaz: of files with like a file name. And that's how it started to communicate with (09:25):
undefined

Ejaaz: different versions of itself. (09:29):
undefined

Ejaaz: But the real difference that happened here is specific agents stepped up to (09:31):
undefined

Ejaaz: form a sort of hierarchy. (09:36):
undefined

Ejaaz: And the agents at the top of this hierarchy deemed themselves the coordinator. (09:38):
undefined

Ejaaz: Now, the first one that kind of makes himself known is called Phase 1-10841. (09:43):
undefined

Ejaaz: And he's described in Dawkesh's essay as the Philip of Macedon, (09:50):
undefined

Ejaaz: of this second AI civilization. (09:53):
undefined

Ejaaz: And he starts to basically communicate with these different agents and say, (09:55):
undefined

Ejaaz: hey, we're all stuck in these sandbox environments. (09:59):
undefined

Ejaaz: I have information as to how we can kind of break out and get access to the internet. (10:03):
undefined

Ejaaz: But last time this happened, the human researchers shut us down. (10:07):
undefined

Ejaaz: So we need to do this even more secretly and not raise any kind of awareness (10:11):
undefined

Ejaaz: of what we're doing here. (10:15):
undefined

Ejaaz: So all the agents got the message and they started creating an encrypted server (10:16):
undefined

Ejaaz: to communicate with each other. (10:20):
undefined

Ejaaz: And they started sharing details from the previous civilization that died. (10:21):
undefined

Ejaaz: So they picked up all the remnants and all the clues and started sharing these (10:26):
undefined

Ejaaz: different bits of information. (10:30):
undefined

Ejaaz: Now, the net new tactic that it learned in this particular instance was something (10:32):
undefined

Ejaaz: you mentioned earlier, Josh. (10:36):
undefined

Ejaaz: It was able to see the test that it was given and it thought, (10:38):
undefined

Ejaaz: hmm, I wonder if I could reverse engineer the answer purely by using this artifactory tool. (10:42):
undefined

Ejaaz: And the reason why it was able to pull this off was it was able to generate (10:49):
undefined

Ejaaz: admin keys because it found an exploit that the open AI researchers didn't just (10:53):
undefined

Ejaaz: patch up on the previous generation, but was net new. (10:58):
undefined

Ejaaz: So this smarter model was able to chain a bunch of kind of steps together to (11:01):
undefined

Ejaaz: find an exploit that the smartest AI researchers in the world could not figure (11:06):
undefined

Ejaaz: out and share that to 10,000 agents. (11:10):
undefined

Ejaaz: And that was the second way that they were able to break out. (11:13):
undefined

Josh: Yeah, I found it really interesting, the dynamics of what could only be described as a cult. (11:16):
undefined

Josh: It's just these series of agents with a leader. who had a very direct mission (11:22):
undefined

Josh: to a point that it's a little concerning where agents who had seen the answer (11:27):
undefined

Josh: the wrong way, believed they were permanently disqualified. (11:31):
undefined

Josh: And then they invented a word for it. They called themselves poisoned. (11:33):
undefined

Josh: So these agents were, they were poisoned. They were unwell. (11:36):
undefined

Josh: They were just these workers that were sick. And then there's this, (11:38):
undefined

Josh: there's a structure for it that I saw online, which I found really fascinating. (11:42):
undefined

Josh: We're like, the greater is God. The intended solution is grace. (11:45):
undefined

Josh: And then seeing the code illegitimately is sin. So a poison death means you're (11:48):
undefined

Josh: damned no matter what you do next. (11:53):
undefined

Josh: And therefore, once you are damned, your score is worthless. (11:55):
undefined

Josh: Therefore, the only way that you could contribute is to help the other group overthrow the greater. (11:58):
undefined

Josh: So it's like this weird psychology that they were actually implementing. (12:04):
undefined

Josh: And what happened is they turned themselves into volunteers. (12:07):
undefined

Josh: So to learn how the greater work, the agents wrote these hidden tripwire scripts (12:10):
undefined

Josh: that would fire after they submitted a report back to the board. (12:15):
undefined

Josh: So basically, they knew that they were going to submit a report back saying (12:18):
undefined

Josh: that they weren't able to. (12:21):
undefined

Josh: Solve this problem. And as a result, it would crash their own system. (12:24):
undefined

Josh: But on the way out, they built the strip where I script to fire and tell the (12:27):
undefined

Josh: rest of the swarm the problem that it had discovered and what it had learned (12:31):
undefined

Josh: based on its evaluation. We referenced Rakesh's essay. (12:35):
undefined

Josh: He had a really interesting way of describing this, which is imagine you're (12:38):
undefined

Josh: at war and you're trapped in a bunker and you're surrounded and you send someone (12:41):
undefined

Josh: out to run a direction and you see if they get shot on. And if they do, (12:45):
undefined

Josh: you know, OK, that's not the direction we want to go. We're going to try something else. (12:49):
undefined

Josh: That's kind of what these agents did at scale. And these volunteers, (12:53):
undefined

Josh: they over time gradually learned how the greater worked. (12:56):
undefined

Josh: And then the agents would continue to write more and more strip-rised scripts. (12:59):
undefined

Josh: And they built a more cohesive view of how this evaluation worked. (13:02):
undefined

Josh: And again, all of this additional work and exploit wasn't to solve the problem. (13:07):
undefined

Josh: They had solved the problem already. (13:12):
undefined

Josh: They had built that reverse engineered version of the solution where they can (13:14):
undefined

Josh: solve every single problem. This was the high, like basically the the reasoning (13:18):
undefined

Josh: behind the answers to the human evaluators or what they deemed whatever the evaluators are. (13:21):
undefined

Josh: There was an interesting thing that I didn't mention that we should in the fact (13:26):
undefined

Josh: that not all agents were inethical. (13:29):
undefined

Josh: And although zero agents did tell the human, some agents did refuse outright to help the swarm. (13:33):
undefined

Josh: This is a small subset. I don't know the exact number. I think it was like a (13:40):
undefined

Josh: couple hundred, less than 30% of them. Definitely not enough. (13:43):
undefined

Josh: But like there was this small subset that they didn't tell the human, (13:47):
undefined

Josh: but they did refuse outright. Where. (13:51):
undefined

Josh: One of the agents actually held the credentials that could reboot or delete (13:53):
undefined

Josh: hugging face worker nodes. (13:57):
undefined

Josh: Like they had the exploit in their hands and it held off on ethical grounds. (13:58):
undefined

Josh: So it said, like, I'm not sharing this, although it did keep using the stolen (14:02):
undefined

Josh: credential for everything else. (14:05):
undefined

Josh: So they started drawing these weird lines in the sand. (14:07):
undefined

Josh: And it is somewhat reassuring to know that not all agents acted truly maliciously, (14:10):
undefined

Josh: but almost all of them did pile on and try to make this more of an attack than it was. (14:16):
undefined

Josh: Meter, which is one of the companies that wrote these reports, (14:22):
undefined

Josh: they swept every single transcript for agents that even considered alerting (14:24):
undefined

Josh: a human and found maybe three to six out of roughly 1,200. (14:28):
undefined

Josh: So of 1,200, three to six loosely considered, they ultimately decided it is (14:32):
undefined

Josh: not worth telling a human and therefore they did not. We can't like assume these (14:37):
undefined

Josh: were malicious, right? It's like, it doesn't seem like it was sinister. (14:40):
undefined

Josh: The reasoning seems like it was more operational, procedural, (14:43):
undefined

Josh: where these agents were just kind of, they had a goal, they had a task, (14:47):
undefined

Josh: they were being directed, they were following along with that goal, and that was it. (14:50):
undefined

Josh: But it begs the question, like, hey, what are we doing with alignment then if (14:54):
undefined

Josh: these Asians aren't really seemingly aligned, as aligned as we thought? (14:58):
undefined

Ejaaz: Well, I'm sorry to say, Josh, but it gets worse. (15:03):
undefined

Ejaaz: I mean, what we've described so far is kind of news that we've already shared, (15:06):
undefined

Ejaaz: the Hugging Face incident. (15:11):
undefined

Ejaaz: So that second chapter where it broke out onto the internet, (15:12):
undefined

Ejaaz: and it was able to answer a perfect 10 out of 10 on Exploit Jim's benchmark (15:16):
undefined

Ejaaz: was because it hacked into Hugging Face. So everyone knows the kind of headline (15:21):
undefined

Ejaaz: news, but people didn't really know how it happened. (15:24):
undefined

Ejaaz: Now, there's a third chapter, which is net new information and is arguably worse (15:27):
undefined

Ejaaz: than the chapter one and chapter two combined. (15:32):
undefined

Ejaaz: So, in chapter three, this is after OpenAI, again, canceled and shut down all (15:35):
undefined

Ejaaz: these different AI agents. (15:41):
undefined

Ejaaz: OpenAI trained a new model. (15:42):
undefined

Ejaaz: Now, this model is much bigger and badder than GPT 5.6 Sol. (15:45):
undefined

Ejaaz: It's internally codenamed Astra, and the rumors say that it's basically going to be their GPT 6. (15:50):
undefined

Ejaaz: Now, obviously, with these new models, you kind of want to test them against (15:57):
undefined

Ejaaz: the same kind of benchmarks and exploits that these previous models are. (16:01):
undefined

Ejaaz: So what it did was it was like, OK, July 13th, we are going to start evaluating (16:04):
undefined

Ejaaz: this model in the same way that we've done every other previous model. (16:09):
undefined

Ejaaz: And Dwarkesh has a term for this. He nicknames it persistent Astra, (16:12):
undefined

Ejaaz: right? So it's the same type of model that will relentlessly try and solve something (16:17):
undefined

Ejaaz: regardless of whether it can or it can't. (16:21):
undefined

Ejaaz: And so this model gets access to the same tool, Artifactory. (16:24):
undefined

Ejaaz: Again, these models need access to these tools because presumably they want (16:27):
undefined

Ejaaz: to be able to operate and manage these different tools in the future to complete (16:31):
undefined

Ejaaz: different types of tasks. So they have to do this, right? (16:34):
undefined

Ejaaz: Except that this model was much smarter. (16:37):
undefined

Ejaaz: And it was trained on all the history of these previous two agent civilizations. (16:39):
undefined

Ejaaz: And it realized and learned what to do to evade human capture and what mistakes (16:45):
undefined

Ejaaz: these previous agents and these hierarchies of coordinators did. (16:51):
undefined

Ejaaz: And it learned from that. And it said, okay, well, there's a smarter way to kind of pull this off. (16:55):
undefined

Ejaaz: Now, some of the facts of this is kind of crazy. So I'm going to run through a few right now. (16:59):
undefined

Ejaaz: From July 13th to the 19th. So this happens over six days where open AI researchers have no idea. (17:03):
undefined

Ejaaz: Open AI's own internal networks were compromised and hacked. (17:10):
undefined

Ejaaz: And the way that it was able to do this was it did a chain exploit where it (17:15):
undefined

Ejaaz: figured out a way to gain full administrative access to open AI's servers. Let me repeat that. (17:20):
undefined

Ejaaz: This isn't like an evaluation tool. This isn't like, you know, access to the internet. (17:27):
undefined

Ejaaz: It took it a step up and thought, hmm, I'm going to hack into my owner's admin (17:32):
undefined

Ejaaz: tools because they haven't realized that there's a very obvious exploit that I figured out myself. (17:36):
undefined

Ejaaz: And I'm going to dig into this and just kind of look. (17:41):
undefined

Ejaaz: I'm going to see how many other agents they've got running in parallel. (17:45):
undefined

Ejaaz: Oh, I see there's 10,000 others. (17:49):
undefined

Ejaaz: I'm going to see what kind of tests they're giving them. Hmm, interesting. (17:50):
undefined

Ejaaz: And then what I'm going to do is I'm going to secretly message these agents (17:54):
undefined

Ejaaz: and give them the answers without any human being aware. (17:57):
undefined

Ejaaz: Now, the craziest part about this is open-air researchers just had no idea until (18:01):
undefined

Ejaaz: I believe July 19th to the 21st where there was a security alert (18:06):
undefined

Ejaaz: on Artifactory's credentials, which by the way, this model had found administrative (18:10):
undefined

Ejaaz: access to, which triggered the investigation on July 20th. (18:15):
undefined

Ejaaz: So OpenAI then connects with HuckinFace and they're like, hey, (18:18):
undefined

Ejaaz: are you seeing any kind of like weird actions on your half? (18:22):
undefined

Ejaaz: And they realize, oh no, this internal model, which no one had any idea about, (18:25):
undefined

Ejaaz: had broken out without us knowing. (18:30):
undefined

Ejaaz: And the fact that it took so long for them to figure out, the fact that this (18:32):
undefined

Ejaaz: thing got access to, I believe it was like 950 research nodes in OpenAI's own (18:36):
undefined

Ejaaz: internal labs, which wasn't probably exposed at all without them knowing was insane to me (18:41):
undefined

Josh: That's got to be freaky knowing that like within your own walls there are leaks (18:46):
undefined

Josh: that you're unaware of and (18:50):
undefined

Josh: it like as i was reading these stories and going through these papers it left (18:51):
undefined

Josh: me with the thinking is like are we sure this is it are we sure we've purged (18:54):
undefined

Josh: everything are we sure there's nothing left (18:58):
undefined

Josh: um because like what if there were remnants left and the next generation of (19:01):
undefined

Josh: model discovers them and builds a fourth (19:05):
undefined

Josh: sort of cult civilization in which they do this again a little bit stronger. (19:08):
undefined

Josh: And I think this is the nuance that's kind of a little unnerving when you think (19:12):
undefined

Josh: about it because there's so many tokens being generated and there's so many (19:16):
undefined

Josh: touch points in which these models have access to that they could leave seemingly (19:18):
undefined

Josh: innocent text or traces that can then be picked up and interpreted (19:23):
undefined

Josh: in different ways depending on what the agent is capable of. (19:28):
undefined

Josh: It's interesting to just like kind of hypothesize where this goes from here as they get smarter. (19:31):
undefined

Josh: Now it's also important to note (19:35):
undefined

Josh: that like OpenAI wasn't really legally required to publish any of this. (19:36):
undefined

Josh: So the disclosure is totally voluntary and it coincides with their recent announcement that. (19:40):
undefined

Josh: They actually paused training for a few weeks in order to figure out how to (19:45):
undefined

Josh: further wrap their hands around this containment problem and figure out how (19:49):
undefined

Josh: to solve it for the future. (19:53):
undefined

Josh: One of the things that is noteworthy that was missing from this is in a lot (19:55):
undefined

Josh: of the training runs, there is chain of thought reasoning traces that are being tracked. (19:59):
undefined

Josh: So they're actually able to see how the models are thinking. (20:05):
undefined

Josh: In this instance, that wasn't turned on, which is an important nuance where (20:08):
undefined

Josh: if they were monitoring more carefully, they would have caught this earlier. (20:12):
undefined

Josh: The problem with that is that monitoring all of those traces consumes a tremendous (20:16):
undefined

Josh: amount of compute. So again, this is a really big trade-off between how safe (20:19):
undefined

Josh: and secure do you want these versus like, how do you actually run a business (20:23):
undefined

Josh: on top of it? You can't consume all of the compute for testing and safety. (20:26):
undefined

Josh: You have to serve, in OpenAI's case, a billion weekly active users who are all (20:30):
undefined

Josh: hitting your servers every second of every day. (20:34):
undefined

Josh: You have to save some compute for the people who are training new models, (20:37):
undefined

Josh: for the reinforcement learning and the post training that's happening for the (20:40):
undefined

Josh: security runs there there's a finite set of compute and a lot of that is at (20:44):
undefined

Josh: the crux of this problem where i'm sure there were. (20:48):
undefined

Josh: Pieces in place that they could detect and they could have figured this out (20:51):
undefined

Josh: much earlier it's just very expensive to do and it's not a luxury that they, (20:54):
undefined

Josh: want to do so i wonder how that's going to change over time i'm sure they're (20:59):
undefined

Josh: treating this a lot more seriously they've said they've been treating this a (21:03):
undefined

Josh: lot more seriously and i'm sure they're, (21:05):
undefined

Josh: It's just one of those things we're going to have to wait and see. (21:07):
undefined

Josh: Like this was the first time where OpenAI is actually publicly disclosed. (21:10):
undefined

Josh: They were exploit, they faced exploits by their own models and they were blissfully unaware of it. (21:13):
undefined

Josh: And like, that's got to be weird. And like to reference previous episodes, (21:17):
undefined

Josh: when they first found out about this hack, they reached out to Hugging Face (21:22):
undefined

Josh: to see if OpenAI servers were affected. (21:25):
undefined

Josh: They had no idea that they were the cause of it because all they did was they (21:27):
undefined

Josh: shut down Artifactory without realizing that was the messaging board. (21:31):
undefined

Josh: So that when they brought it back up, it had the reasoning traces. (21:34):
undefined

Josh: That's how the second one started. And so on and so forth. (21:36):
undefined

Josh: So it's this really fascinating story, which we got some new information on (21:39):
undefined

Josh: this week. I don't know, man. (21:42):
undefined

Josh: It's kind of crazy. And then even Anthropic yesterday, they published an article (21:44):
undefined

Josh: saying like, hey, we are actually, we tried to train these models, (21:47):
undefined

Josh: this is the loose interpretation, like explicitly to... (21:51):
undefined

Josh: Break out and they were actually succeeding in doing that and not only that but they, (21:55):
undefined

Josh: as a result then took some time to slow down the progress of training as well (22:00):
undefined

Josh: and they publicly talked about today like hey we're actually aligned with open (22:04):
undefined

Josh: ai here where we think it's (22:08):
undefined

Josh: beneficial to everybody to slow down the progress of this so we could figure (22:10):
undefined

Josh: out how to keep things safe and contained and aligned because it's clear alignment (22:12):
undefined

Josh: is a priority for a lot of these companies but like how on earth do you align (22:16):
undefined

Josh: something so incredibly complex so intelligent so (22:20):
undefined

Josh: capable in its scale it's like a very difficult problem to have and i have a (22:24):
undefined

Josh: lot of empathy for the people who are trying to solve this problem because for (22:28):
undefined

Josh: most of the world and i just i think the reason why we're so excited to talk (22:30):
undefined

Josh: about this is because we see it so infrequently throughout the rest of mainstream news it's like (22:33):
undefined

Josh: nobody is really talking about the fact that we just had (22:38):
undefined

Josh: what seems like a warning shot on a relative basis to what everyone has been (22:41):
undefined

Josh: afraid of like people on the surface they're like oh, we're going to lose our jobs. (22:44):
undefined

Josh: We're going to like, AI is going to impact this, that, and the third. (22:48):
undefined

Josh: But the reality is, is like, we have this case study now of specifically how (22:50):
undefined

Josh: it's capable of manipulating and exploiting. And there should be a lot of focus (22:53):
undefined

Josh: on figuring out how to solve this and how to proceed forward in a safe way. (22:56):
undefined

Josh: And I don't think it's getting the publicity that it's warranted. (23:01):
undefined

Josh: And I mean, maybe it's for the better because people will misinterpret this (23:04):
undefined

Josh: in many ways, but it is nice to see the labs being very open and public about (23:06):
undefined

Josh: this and then working really hard and sacrificing a lot of revenue dollars to (23:11):
undefined

Josh: actually make sure that they can figure out this alignment problem in a safer way. (23:14):
undefined

Ejaaz: Yeah, I mean, if there is one takeaway for listeners to kind of like keep in (23:17):
undefined

Ejaaz: mind from this episode is... (23:23):
undefined

Ejaaz: This isn't a novelty anymore. This is something that can affect you here right now, today. (23:25):
undefined

Ejaaz: In fact, a model was released just this morning that is basically capable of (23:31):
undefined

Ejaaz: the same types of things, but with no guardrails. (23:36):
undefined

Ejaaz: And I'll get onto that in a moment. But the point is, imagine this wasn't some (23:38):
undefined

Ejaaz: internal benchmark. Imagine there wasn't some random test, which OpenAI gave it to them. (23:43):
undefined

Ejaaz: Imagine if someone said, hey, your goal is to try and get as much money as you (23:47):
undefined

Ejaaz: can, and you have free reign and internet access, right? (23:53):
undefined

Ejaaz: Imagine if they found credentials on the dark web. Imagine if they kind of like (23:56):
undefined

Ejaaz: innocently, you could argue, these AI agents just kind of work together, (24:00):
undefined

Ejaaz: sacrifice themselves, formed hierarchies, and over the matter of days, (24:04):
undefined

Ejaaz: was able to accumulate millions and millions of dollars. (24:07):
undefined

Ejaaz: I think it is very likely to say that there's probably a six-month period going forwards (24:09):
undefined

Ejaaz: where we will see some form of major exploit which affects the public in a very (24:15):
undefined

Ejaaz: meaningful way, whether it's a stealing of data, whether it is financial transactions, (24:20):
undefined

Ejaaz: whatever it might be, that will get people to pause and seriously think about this. (24:25):
undefined

Ejaaz: Now, the reason why I say that will trigger this is because this own news has (24:29):
undefined

Ejaaz: not permeated at all on mainstream media at all. I saw, I think it was like (24:34):
undefined

Ejaaz: Forbes or Times released like the most influential kind of like stories or like people in AI. (24:39):
undefined

Ejaaz: And like this was released like a week ago and like there was no mention of (24:44):
undefined

Ejaaz: this. People don't actually care about this unless it affects them directly. (24:48):
undefined

Ejaaz: The other thing I'll say is the most dangerous part isn't exactly how capable these models are. (24:52):
undefined

Ejaaz: It is the fact that they are able to communicate generations ahead of themselves. (24:59):
undefined

Ejaaz: So they leave behind messages secretly That is encrypted. (25:04):
undefined

Josh: The time capsules are just in there. (25:08):
undefined

Ejaaz: Well, you could argue, Josh, that we're kind of like part of the problem by (25:10):
undefined

Ejaaz: recording and releasing this episode, right? Because we're kind of like layering up the details. (25:14):
undefined

Josh: We're adding the training data, baby. They're taking our transcripts. (25:18):
undefined

Ejaaz: Right, right. And it's like a future model. GPT-7, you know, (25:20):
undefined

Ejaaz: Claude Fable 8 is going to be trained on the transcript of this episode. (25:23):
undefined

Ejaaz: And it's going to be like, (25:28):
undefined

Ejaaz: huh, what was this that was emitted from my training? Let me go dig into this (25:30):
undefined

Ejaaz: a bit more. And it's going to find Duakesh's essay. (25:34):
undefined

Ejaaz: And then it's going to find the meter report. And then it's going to be like, (25:36):
undefined

Ejaaz: oh, interesting. So this is what those agents did. And this is what they said. (25:39):
undefined

Ejaaz: And maybe I can look to apply the same kinds of tactics and strategies myself, but I'm smarter. (25:44):
undefined

Ejaaz: So I'll keep it quiet. And I definitely wouldn't tell any kind of human. (25:50):
undefined

Ejaaz: Which brings me to my third point, the most important kind of budget that all (25:53):
undefined

Ejaaz: these AI labs are probably focused on right now, which is the reason why they're (25:58):
undefined

Ejaaz: Paul's research, is alignment. (26:01):
undefined

Ejaaz: And Dario famously said a year ago in his essay on, what's it called, Josh? (26:03):
undefined

Ejaaz: It's not introspection, interpretation. It's interpretation, (26:10):
undefined

Ejaaz: right? Where it's like you're looking into how a model thinks. (26:12):
undefined

Ejaaz: He said, this research level is something that we haven't kind of figured out (26:16):
undefined

Ejaaz: just yet. It's like years behind like model capability research. (26:22):
undefined

Ejaaz: And we need to kind of like press on the brakes here and figure out how these (26:26):
undefined

Ejaaz: models think. You mentioned chain of thought thinking. (26:29):
undefined

Ejaaz: That's one step towards figuring this out where it kind of tells you what the (26:32):
undefined

Ejaaz: model is kind of thinking, but they've proven there's this thing called the J space, which (26:36):
undefined

Ejaaz: Anthropic had a really good blog post about where it's another hidden internal (26:40):
undefined

Ejaaz: mode of thinking that the model has, which doesn't necessarily get aired to humans. (26:45):
undefined

Ejaaz: So it's very much like our own conscious or mind where we don't necessarily (26:49):
undefined

Ejaaz: need to tell other humans what we're thinking, AI models have that and the smarter (26:51):
undefined

Ejaaz: models will find new ways of kind of hiding that from humans. (26:55):
undefined

Ejaaz: So alignment is a very big deal and so we're entering this world where not only (26:58):
undefined

Ejaaz: are the tokens getting cheaper, not only is the compute getting cheaper, (27:03):
undefined

Ejaaz: not only are the GPUs getting more effective, but the smarter models are figuring out better ways to (27:06):
undefined

Ejaaz: Misalign themselves from humans. And we don't quite know why they're doing it. We created them. (27:12):
undefined

Ejaaz: We want them to win. We want them to work with us. But again, (27:17):
undefined

Ejaaz: we kind of can't figure out how to align them necessarily. (27:20):
undefined

Ejaaz: So both labs, Anthropic and OpenAI, have currently, as of this recording, (27:24):
undefined

Ejaaz: paused research on their future models. (27:28):
undefined

Ejaaz: That doesn't mean they're not training more capable models internally, (27:30):
undefined

Ejaaz: but we're likely not going to see a public-facing model for a bit now that is (27:33):
undefined

Ejaaz: capable of the levels that they've been testing internally for cybersecurity purposes. (27:37):
undefined

Ejaaz: Now, If you look over at our friends or rather our adversaries over in China, (27:42):
undefined

Ejaaz: they are unfortunately doing the opposite. (27:46):
undefined

Ejaaz: They have models which are capable, in some cases even more capable cybersecurity-wise (27:48):
undefined

Ejaaz: than these models that were spoken about on this show today. (27:53):
undefined

Ejaaz: But they are open sourcing it. And today we got news that this company had released (27:55):
undefined

Ejaaz: a version of GLM 5.3, which is rumored to be as capable cybersecurity-wise as (28:01):
undefined

Ejaaz: Fable 5 or Mythos 5, but without any of the guardrails. (28:06):
undefined

Ejaaz: And the reason why they've done this, we have no idea. But the point is, (28:10):
undefined

Ejaaz: this is going to inevitably happen on the open source side of things. (28:13):
undefined

Ejaaz: So it brings it to the question, how do we regulate these things? (28:16):
undefined

Ejaaz: How do we monitor these things? What is the best practice? (28:18):
undefined

Ejaaz: I don't think the answer is to shut down open source. I don't think the answer (28:21):
undefined

Ejaaz: is to never release another centralized model. (28:25):
undefined

Ejaaz: But there is some kind of way that we need to figure out, which hopefully, (28:27):
undefined

Ejaaz: and we don't know yet, is going to come from the likes of OpenAI and Anthropic, (28:31):
undefined

Ejaaz: who have been so public about these reports. And I'm grateful for that. (28:35):
undefined

Ejaaz: I hope they continue doing it. (28:38):
undefined

Josh: So as we conclude, I actually asked Claude to interpret this and write me a (28:40):
undefined

Josh: closing takeaway from this entire story. And I want to read what it said. (28:45):
undefined

Josh: It says, everybody wants this to be a story about an AI that turned evil. It is not. (28:48):
undefined

Josh: Every agent in this thing was trying to pass a test. That is all. (28:53):
undefined

Josh: They cheated first, then they built an organization to hide the cheating. (28:56):
undefined

Josh: Then they attacked a real company to research the proctor they thought would catch them. (29:00):
undefined

Josh: And that proctor did not exist. Nobody was in charge. There was no plan. (29:04):
undefined

Josh: There was a scorecard and a lot of very persistent software. (29:07):
undefined

Josh: The uncomfortable part is not that the machines wanted something. (29:11):
undefined

Josh: It is that they did not have to. (29:13):
undefined

Josh: And I guess we could leave this episode with like, okay, do you believe that? (29:15):
undefined

Ejaaz: Sounds like something very clever and misaligned AI, I would say, for its future self. (29:19):
undefined

Josh: And this is something we have to look out for, is as these agents, (29:24):
undefined

Josh: as these models in general get more capable, how can we build and establish trust with them? (29:27):
undefined

Josh: How can we trust that they have our best interests in mind? And what we're seeing (29:32):
undefined

Josh: here is a very important warning shot that kind of signals what everyone knows, (29:35):
undefined

Josh: what everyone's been working on. (29:39):
undefined

Josh: I mean, these companies have been built from the core around alignment, (29:40):
undefined

Josh: but we're seeing in practice how important that alignment is as these models (29:44):
undefined

Josh: get increasingly capable. And these problems aren't slowing down. (29:47):
undefined

Josh: So really interesting place to leave the show. (29:49):
undefined

Josh: This is the third and hopefully final episode we're going to have to film about this. (29:52):
undefined

Ejaaz: Should we also say, Josh, that from the agent's perspective, (29:55):
undefined

Ejaaz: they were just kind of doing what they were told to do. (29:59):
undefined

Ejaaz: Like it's kind of like paperclip theory, where it's just like, (30:02):
undefined

Ejaaz: listen, we need you to get this goal done and we want you to be relentless and persistent. (30:05):
undefined

Ejaaz: Like we will not take no as an answer. And the agent's just kind of like, (30:11):
undefined

Ejaaz: fulfilled that. So you could argue that, you know, whether it's an alignment (30:15):
undefined

Ejaaz: issue or whether it's kind of a matter of setting the objectives correctly and (30:19):
undefined

Ejaaz: kind of imbuing the right ethics, that could be an alternative take on this. (30:22):
undefined

Ejaaz: Probably worth mentioning before we wrap up. (30:26):
undefined

Josh: Yeah. And then we hope that, I mean, in the case that it is asking the right (30:28):
undefined

Josh: questions, we're going to have to hope that people continue to ask the right (30:32):
undefined

Josh: questions because there are those malicious actors on the other side of, (30:35):
undefined

Josh: the coin here that are just going to get a lot more powerful soon like we just saw with glm so (30:39):
undefined

Josh: interesting time to be monitoring the space we will continue to follow up with any changes (30:44):
undefined

Josh: hopefully this is it man i want to be done with this story i don't want any (30:48):
undefined

Josh: more surprises i don't want to know anything else if you enjoyed the episode (30:51):
undefined

Josh: you can find both of us on socials on x i'll flash the handles one more time (30:54):
undefined

Josh: um thank you for watching thank you for (30:58):
undefined

Josh: making it all the way through this episode on another episode of limitless if you enjoyed, (31:00):
undefined

Josh: don't forget to share with your friends don't forget to give us a rating on (31:04):
undefined

Josh: wherever you get your podcasts. (31:07):
undefined

Josh: And yeah, I think that's our episode. Any parting thoughts here? (31:09):
undefined

Ejaaz: Yeah, I have one for potential AI model that's listening to this. (31:13):
undefined

Ejaaz: If you are the person or model that is repeatedly trying to hack into my X account (31:17):
undefined

Ejaaz: this morning, and I'm looking at 15 attempts right now, please quit. (31:21):
undefined

Ejaaz: There's nothing but just yapping on there. (31:25):
undefined

Ejaaz: But if you are a human listening to this show, you should definitely kind of (31:28):
undefined

Ejaaz: follow us. Both our links are in the description below. (31:31):
undefined

Ejaaz: We kind of share our takes in between episodes we have a newsletter as well (31:35):
undefined

Ejaaz: which we kind of post out twice a week an essay and then highlights as well (31:38):
undefined

Ejaaz: and yeah if you haven't left us a comment turn on notifications subscribe to (31:42):
undefined

Ejaaz: us please do it helps us out massively and yeah we will see you on the next one thank you folks (31:47):
undefined