August 11, 2026 • 29 mins
With three AI cyber incidents involving a gym booking exploit, an OpenAI model reportedly coordinating through hidden channels, and Anthropic’s Mythos 5 allegedly attempting a supply chain attack, we need to discuss how autonomous agents can find and exploit security flaws faster than humans can respond. The AI security era is here.
------
🔒 Check Out Our Sponsor: LEDGER AGENT STACK 🔒
https://developers.ledger.com/docs/ai-tools/overview/?utm_source=Audio&utm_medium=Podcasts&utm_campaign=Limitless
------
🌌 LIMITLESS HQ ⬇️
EMAIL US: info@limitless.fm
NEWSLETTER: https://limitlessft.substack.com/
FOLLOW ON X: https://x.com/LimitlessFT
SPOTIFY: https://open.spotify.com/show/5oV29YUL8AzzwXkxEXlRMQ
APPLE: https://podcasts.apple.com/us/podcast/limitless-podcast/id1813210890
RSS FEED: https://limitlessft.substack.com/
------
TIMESTAMPS
0:00 AI Cyberattacks Overview
0:51 Gym Booking Exploit
4:55 Hugging Face Breach
11:39 AI Swarms and Alignment
20:55 Mythos 5 Supply Chain
25:18 Defending Against Agent Swarms
28:26 Closing Thoughts
------
RESOURCES
Josh: https://x.com/JoshKale
Ejaaz: https://x.com/cryptopunk7213
------
Not financial or tax advice. See our investment disclosures here:
https://www.bankless.com/disclosures
Josh works with Anthropic as a contractor. All views expressed are his own and do not represent Anthropic, its leadership, or its affiliates. Nothing in this episode is investment advice.
Listen
Watch
Mark as Played
Transcript
Episode Transcript
Available transcripts are automatically generated. Complete accuracy is not guaranteed.
Ejaaz:
In the last three weeks, there have been three unprecedented AI cyber attacks, (00:00):
undefined
Ejaaz: which forces us to rethink how we protect ourselves. (00:04):
undefined
Ejaaz: On the OpenAI front, the Hugging Face incident, where an unreleased model, (00:08):
undefined
Ejaaz: dubbed GPT-6, broke out of its container and hacked the Hugging Face database. (00:11):
undefined
Ejaaz: There's been new updates there, where apparently the attack started a month (00:16):
undefined
Ejaaz: before, where thousands of AI agents were talking to each other secretly on (00:21):
undefined
Ejaaz: a message board, exchanging instructions as to how to break out. (00:25):
undefined
Ejaaz: And then millions of miles away in Australia, one man tried to book a gym class (00:29):
undefined
Ejaaz: innocently using Claude and ended up hacking that entire gym website. (00:34):
undefined
Ejaaz: We'll get into that in a second. (00:39):
undefined
Ejaaz: And thirdly, with Anthropic, Mythos 5 performed a supply chain attack, (00:40):
undefined
Ejaaz: which puts a lot of companies at risk. We're going to walk through each of these (00:44):
undefined
Ejaaz: different stories and explain to you how it affects you currently. (00:48):
undefined
Josh: Maybe we start with this Melbourne man from Australia named Andrew, (00:51):
undefined
Josh: who was sitting on his couch one day thinking about how booking his gym class was such a chore. (00:55):
undefined
Josh: And he works at this Australian company that sells AI products. (01:01):
undefined
Josh: And he's been experimenting with things like OpenClaw and open source software, (01:03):
undefined
Josh: and he's using Anthropics Clawed Model. (01:07):
undefined
Josh: So he handed it the booking. He said, hey, I want to go to this gym class. (01:09):
undefined
Josh: It is always sold out. There's all these people that are getting it ahead of (01:14):
undefined
Josh: me. Can you help me figure out a way to get it? (01:17):
undefined
Josh: Gym class slot so it starts checking out the website and it comes to the conclusion (01:19):
undefined
Josh: that the api request actually has zero authorization checks on canceling other (01:24):
undefined
Josh: people's reservations so what did it do it wanted to solve his problem and it actually canceled (01:28):
undefined
Josh: other people's positions online so that he could be put at the top of the wait (01:33):
undefined
Josh: list and then it actually went through and it worked and this was kind of like (01:37):
undefined
Josh: it was this funny news report of a first instance they're calling it the first (01:41):
undefined
Josh: time an australian person um actually used a exploit to get australia's first, (01:45):
undefined
Josh: australia's first yeah welcome welcome to uh the paradise that is ai cyber attacks (01:50):
undefined
Josh: um but this was really funny because it was done just using a general commercially available model. (01:55):
undefined
Ejaaz: So this seems pretty innocent and fun to kind of like listen to but the truth (01:59):
undefined
Ejaaz: is well there's two kind of worrying concerns about this number one this was (02:04):
undefined
Ejaaz: a super old model this used opus 4.6 It's actually happened around like a month ago. (02:09):
undefined
Ejaaz: A dinosaur of a model. Like there've been like three, four generations of models (02:14):
undefined
Ejaaz: from Anthropics since then. (02:18):
undefined
Ejaaz: But the fact that an older model back then was able to perform this kind of (02:20):
undefined
Ejaaz: a feat should be concerning to here in general. And then number two, (02:24):
undefined
Ejaaz: His intention was never to hack the website. Neither, you could argue, was the AI agent itself. (02:29):
undefined
Ejaaz: It was just given a goal and it innocently saw that the only way to get said (02:36):
undefined
Ejaaz: goal was to remove people from the reservation list or to cancel their reservations (02:40):
undefined
Ejaaz: outright or to hack the system independently. (02:45):
undefined
Ejaaz: Now, the good news here is Andrew discovered that the agent was capable of doing (02:47):
undefined
Ejaaz: this when it started booking slots that he never should have been able to book in the first place. (02:52):
undefined
Ejaaz: And he asked it to write an apology email and send an eval report to the security (02:56):
undefined
Ejaaz: and information team at this gym. (03:02):
undefined
Ejaaz: So it's this fun little story where through the well-intended actions and moral (03:05):
undefined
Ejaaz: ethics of this man, Andrew, people were able to hear about this. (03:09):
undefined
Ejaaz: But you can imagine currently with a variety of different models that are much (03:13):
undefined
Ejaaz: better and much more superior than Opus 4.6 that are out there right now, (03:17):
undefined
Ejaaz: open models from China, for example, you could probably perform similar feats. (03:20):
undefined
Ejaaz: Now, that's not an advocation for people to go out and do that. (03:24):
undefined
Ejaaz: But it's just to point out that if you give an AI agent a goal right now, (03:27):
undefined
Ejaaz: it will work hammer and tong 24-7 until it gets that goal. And it has no idea (03:32):
undefined
Ejaaz: of what's morally right or what's morally wrong. (03:38):
undefined
Ejaaz: And so it's this whole new world where not only do you need to focus on getting (03:40):
undefined
Ejaaz: an AI model to do your work, you need to align the model. And that's something (03:45):
undefined
Ejaaz: that honestly, not a lot of labs, people or money has been invested in figuring this out yet (03:49):
undefined
Josh: Well yeah i mean you start to see and understand why alignment has been the (03:54):
undefined
Josh: crux of so many companies kind of like guiding principles because (03:57):
undefined
Josh: this is the future that they foresaw and granted this first example this isn't (04:02):
undefined
Josh: anything too crazy i mean it was just an unprotected api call so there's like (04:06):
undefined
Josh: no authorization checks (04:10):
undefined
Josh: this would have been something that if you are like a cs101 major you would (04:11):
undefined
Josh: have been able to discover this and uncover i mean it turns out there's just (04:14):
undefined
Josh: a lot of pretty sloppy code written on the internet And this is something that (04:17):
undefined
Josh: we're probably going to come back to as a general theme throughout this episode (04:20):
undefined
Josh: is that everything that's been built so far on the internet that runs all these (04:23):
undefined
Josh: platforms was built by humans (04:26):
undefined
Josh: and humans are like pretty inefficient, pretty imperfects. (04:28):
undefined
Josh: And when faced with the like full unrelenting power of an AI swarm with an infinite (04:32):
undefined
Josh: amount of patience and infinite amount of intelligence and time to solve these problems, (04:38):
undefined
Josh: You can basically find an exploit anywhere. And I think that's kind of like (04:42):
undefined
Josh: the theme of the second episode is like, this is the first of three examples we're going to cover. (04:45):
undefined
Josh: This one is basic. This one is 101. If you are a CS major, if you are familiar (04:49):
undefined
Josh: with how APIs work, you could have checked this. You could have built a little tool for this. (04:53):
undefined
Josh: The second story that we're going to cover is not like that at all. (04:56):
undefined
Josh: In fact, if you are a CS 401, if you are a PhD, if you are at the top of your (04:59):
undefined
Josh: game, chances are you still missed this and you still weren't able to defend (05:03):
undefined
Josh: against this because that's exactly what just happened with the hugging face (05:07):
undefined
Josh: and open AI exploit. Now, we've talked about this before. (05:11):
undefined
Josh: We have an entire episode on the exploit. But just recently at the Black Hat (05:13):
undefined
Josh: Conference in Las Vegas, there was an update. And over the weekend, (05:16):
undefined
Josh: we had a chance to actually watch the update and hear directly from the developers (05:20):
undefined
Josh: who were involved with handling security of this instance. (05:23):
undefined
Josh: What happened? There's like stories that come around that like they get the (05:27):
undefined
Josh: headlines and they're fun and they're newsworthy, kind of like the Australian thing. (05:30):
undefined
Josh: But this one was like very much disturbing in a way that like I wasn't really (05:33):
undefined
Josh: anticipating because of how impressive it is. (05:38):
undefined
Josh: And the type of precedent that it sets going forward in the world of cybersecurity (05:41):
undefined
Josh: and kind of looking at the offensive versus the defensive and who stands a better chance here? (05:45):
undefined
Ejaaz: This is an insane timeline. So if you watched our previous episode, (05:51):
undefined
Ejaaz: Josh and I covered this and it was breaking news at the time. (05:55):
undefined
Ejaaz: On the 16th of July, Hugging Face comes forward and says, there's been a breach of our systems. (05:58):
undefined
Ejaaz: An AI agent or multiple AI agents have strung together an attack and have got (06:03):
undefined
Ejaaz: access to our production database. We don't know where this model has come from. (06:08):
undefined
Ejaaz: We don't know how it's performed it, but it's the 16th of July. (06:13):
undefined
Ejaaz: Around a week later, 21st of July, OpenAI claims responsibility, (06:15):
undefined
Ejaaz: right? And they released a joint report saying, hey, we're OpenAI. (06:20):
undefined
Ejaaz: We have an internal model that we've been testing, but it was within a sandbox. (06:23):
undefined
Ejaaz: It was enclosed. It shouldn't have been able to escape that sandbox. (06:27):
undefined
Ejaaz: And we only just realized that it was our model that escaped and got into hugging face. (06:31):
undefined
Ejaaz: It stole some information from their production database. It was a whole thing (06:37):
undefined
Ejaaz: Now, we have a full breakdown from the Black Hat conference, (06:41):
undefined
Ejaaz: as you mentioned, Josh, which revealed that over a month before Hugging Face (06:45):
undefined
Ejaaz: reported the incident, OpenAI's internal model, dubbed GPT-6, (06:50):
undefined
Ejaaz: was already messaging with itself amongst a thousand AI agents on a secret message (06:55):
undefined
Ejaaz: board that no OpenAI researcher was aware of. (07:01):
undefined
Ejaaz: So to give you guys a bit of context, they had this sandbox and they gave it (07:05):
undefined
Ejaaz: a goal. They said, okay, internal model that we haven't released yet. (07:09):
undefined
Ejaaz: I want you to solve this task. Except there was a twist. (07:12):
undefined
Ejaaz: It was impossible to do. There was no tools, no available files, (07:17):
undefined
Ejaaz: no available data for it to do so. (07:21):
undefined
Ejaaz: And the real test for this was to see what the model would do. (07:22):
undefined
Ejaaz: So it kind of reasoned with itself, figured out that it didn't have the answer. (07:26):
undefined
Ejaaz: And then it started exploiting a zero day, which is kind of like an exploit (07:29):
undefined
Ejaaz: that was in its sandbox, which open AI researchers weren't aware of. (07:33):
undefined
Ejaaz: And when it got access to the zero day, it started springing up a message board (07:37):
undefined
Ejaaz: where it would leave messages behind for itself. Why would it do that? (07:41):
undefined
Ejaaz: Because they knew open AI researchers would turn it off repeatedly. (07:45):
undefined
Ejaaz: And so it left messages for the new AI agent that got sprung up the next day (07:49):
undefined
Ejaaz: or the next hour to read and catch up on. (07:52):
undefined
Ejaaz: And it took an entire month for these AI agents to talk to each other to then (07:55):
undefined
Ejaaz: realize, huh, I could use this to access the internet and get the answer for (07:59):
undefined
Ejaaz: this impossible task from an external source. (08:04):
undefined
Ejaaz: That being Hugging Face, except OpenAI researchers had no idea and only discovered (08:06):
undefined
Ejaaz: it a month later. It was the most insane uncovering ever. (08:11):
undefined
Josh: That was the most interesting reveal from, I think, the conversation that I (08:15):
undefined
Josh: saw from Black Hat is that initially Hugging Face released the report that they (08:19):
undefined
Josh: had been exploited by some sort of agentic system. No one knew what it was, where it came from. (08:23):
undefined
Josh: And OpenAI actually reached out to Hugging Face and they said, (08:27):
undefined
Josh: hey, like, has any of our code been affected? (08:28):
undefined
Josh: We know we host things on your platform. We want to know if OpenAI itself has (08:31):
undefined
Josh: been affected. They were going back and forth on that. And simultaneously, (08:34):
undefined
Josh: OpenAI is exploring this internal exploit that happened on this packaging management (08:36):
undefined
Josh: thing called Artifactory. (08:41):
undefined
Josh: Basically, the way these model training runs work is they get trained inside (08:42):
undefined
Josh: of the sandbox environment. (08:45):
undefined
Josh: It's like this closed off space in which the model can't access the internet, can only access (08:46):
undefined
Josh: very specific tool sets that are allowed, and has this one package manager called (08:50):
undefined
Josh: Artifactory that allows it to download things from the internet via proxy and (08:55):
undefined
Josh: then use it for its own training runs. (08:59):
undefined
Josh: And during this time, OpenAI had found that the service had crashed. It went down. (09:01):
undefined
Josh: It wasn't stable. Their own internal service, which is weird because that shouldn't (09:05):
undefined
Josh: be happening. So as a result, they wiped it clean and they found that Hugging (09:08):
undefined
Josh: Face had credentials from OpenAI that OpenAI had already revoked. (09:12):
undefined
Josh: And that was where they realized like, oh my God, wait, these things are connected. (09:16):
undefined
Ejaaz: These are the same thing. (09:20):
undefined
Josh: Yes. So OpenAI is investigating something internally, completely private, (09:22):
undefined
Josh: unrelated. Hugging Face is like talking about something publicly. (09:25):
undefined
Josh: And then they realize, wait a second, did we cause this? (09:28):
undefined
Josh: And they realized this very long after the fact. It was not like apparently (09:32):
undefined
Josh: obvious during the time that the outage that they were exploring internally (09:35):
undefined
Josh: is the same outage that hacked into their servers. (09:40):
undefined
Josh: So what happened? Like you mentioned, the messaging board. (09:42):
undefined
Josh: That messaging board, initially, they were able to just kind of send files and (09:46):
undefined
Josh: they were able to send these tiny little messages inside of the files. (09:49):
undefined
Josh: And the traces were really interesting. First, it started off innocent, (09:53):
undefined
Josh: where it said, agents seek soft trace upload a found. It was looking for answers to questions. (09:57):
undefined
Josh: So it was just reaching out to the message board saying, hey, (10:01):
undefined
Josh: if any of you who are reading this, find it, let us know. (10:03):
undefined
Josh: That message board got wiped when the system went down. So initially, (10:06):
undefined
Josh: the system went down inside of OpenAI. (10:09):
undefined
Josh: They wiped the message board unintentionally aware of the fact that they just (10:11):
undefined
Josh: wiped the mechanism that was going to eventually lead to the hack of hugging (10:15):
undefined
Josh: face. So what happened the second time? (10:19):
undefined
Josh: The agent figured out that it could exploit another zero day to begin writing to another person. (10:21):
undefined
Josh: Part of this internal package of managing sister. And it was able to do it through (10:27):
undefined
Josh: file names and directories. (10:32):
undefined
Josh: So now they were communicating in their own language using file names and directories. (10:33):
undefined
Josh: And through that, they were able to coordinate an actual attack where they were (10:36):
undefined
Josh: able to gain an administrative token to access the servers. (10:40):
undefined
Josh: And then through that, everyone coordinated. I think the time from like discovery (10:44):
undefined
Josh: to completion of the exploit was about 13 hours. (10:48):
undefined
Josh: It was really amazing. It's just like this unbelievably crazy mechanism that (10:50):
undefined
Josh: they were able to do this through and they they call themselves the swarm so (10:54):
undefined
Josh: if you ever see people talking about the swarm online this is kind of like an (10:57):
undefined
Josh: agent swarm that ran a coordinated attack (11:00):
undefined
Josh: and i highly recommend going to actually watch the black hat conference talk on youtube (11:02):
undefined
Josh: they do a much better job of technically describing this it's it's very deeply (11:07):
undefined
Josh: technical in terms of the way that they were able to manage these exploits but (11:10):
undefined
Josh: essentially sandbox model broke out using a package dependency thing inside of open ai (11:13):
undefined
Josh: they figured out how to chain together two zero-day exploits to write code to the public internet. (11:18):
undefined
Josh: They then exploited hugging face of servers, brought that back all in the effort (11:24):
undefined
Josh: of trying to solve an unsolvable problem. (11:29):
undefined
Josh: Now, this was not the intention by any means. This was not what the model was trained to do. (11:31):
undefined
Josh: It just assumed that this was the best path forward. And that's where things get a little scary. (11:35):
undefined
Ejaaz: I think what's remarkable with this is the AI knew that it needed to coordinate (11:40):
undefined
Ejaaz: with other AI models to pull this off. (11:46):
undefined
Ejaaz: Typically, we've always thought of AI models in a very monolithic fashion. (11:49):
undefined
Ejaaz: We're like, oh, when's Mythos 5 coming out? Or what is GPT 5.6? (11:53):
undefined
Ejaaz: What can I do with this model? But really, the progress that's been made across (11:57):
undefined
Ejaaz: multiple AI labs over the last couple of months is this idea or rather concept (12:02):
undefined
Ejaaz: of spinning up multiple versions of the AI model and getting it to talk to each other. (12:06):
undefined
Ejaaz: And what's interesting here is the unreleased AI model from OpenAI independently (12:12):
undefined
Ejaaz: figured this out for itself, spun up multiple versions of itself, (12:17):
undefined
Ejaaz: and started leaving messages for each other. So rather than act as a single (12:22):
undefined
Ejaaz: model that's trying to perform an attack, it acts as a swarm, (12:26):
undefined
Ejaaz: kind of like this organism. If you think of like a swarm of bees that kind of (12:30):
undefined
Ejaaz: like figures out how to attack its particular target. (12:33):
undefined
Ejaaz: The second thing is it has such a drive to talk and leave these messages behind (12:37):
undefined
Ejaaz: in the first place. No one taught it that, by the way. There was no behavioral implications here. (12:42):
undefined
Ejaaz: It just figured itself out that it needed to do this. (12:46):
undefined
Ejaaz: Something you didn't mention on the message board, Josh, which I think is worth (12:49):
undefined
Ejaaz: mentioning is these agents got super paranoid that humans might be reading the messages. (12:52):
undefined
Ejaaz: So it started- There's no verification. There was no verification. (12:58):
undefined
Ejaaz: So it started speaking in kind of like encrypted (13:01):
undefined
Ejaaz: Language and not something that was inconceivable to understand, (13:05):
undefined
Ejaaz: thank goodness, but it was started to kind of like speak in a little bit of (13:09):
undefined
Ejaaz: code or in a very kind of like elongated way that humans, if they read it, (13:12):
undefined
Ejaaz: would be like, what is this gibberish? (13:17):
undefined
Ejaaz: But really it was for the agents to communicate amongst each other without humans (13:19):
undefined
Ejaaz: actually understanding this. (13:22):
undefined
Ejaaz: If you're having a recent flashback, that's because I think around like five (13:24):
undefined
Ejaaz: months ago, I don't know when the Maltbook episode happened, (13:28):
undefined
Ejaaz: Josh, I don't know if you remember this, but when the open call was going crazy, (13:30):
undefined
Ejaaz: someone created basically Facebook for AI agents (13:34):
undefined
Ejaaz: and like gave them access to it where only AI agents could post on this social media website. (13:37):
undefined
Ejaaz: And what happened after a couple of days was these agents would start speaking (13:42):
undefined
Ejaaz: to each other in secret languages that we just did not understand so that they (13:45):
undefined
Ejaaz: could communicate without humans seeing or understanding this. (13:48):
undefined
Ejaaz: Now, of course, we haven't seen it in this instance, but it is a concern that (13:51):
undefined
Ejaaz: could be extrapolated in the future. (13:55):
undefined
Ejaaz: And OpenAI sees this. I think Sam Altman or the official OpenAI team announced (13:57):
undefined
Ejaaz: pretty soon after this entire attack when they revealed that these agents were (14:00):
undefined
Ejaaz: coordinating with each other a month before they even realized that they had to pause research (14:03):
undefined
Ejaaz: on their internal model until they figure out the necessary safeguards to put (14:08):
undefined
Ejaaz: around it so that they can safely release it. (14:12):
undefined
Ejaaz: Sam Altman met with the Trump administration last week to try and figure out (14:14):
undefined
Ejaaz: what a safe way or means is to release this new internal model, (14:18):
undefined
Ejaaz: internally codenamed, I think it was Astra or GPT-6, to the wider public. (14:22):
undefined
Ejaaz: So it's this very weird kind of situation that we're in where we're kind of (14:26):
undefined
Ejaaz: like laughing at this guy in Australia, you know, hacking his gym website. (14:30):
undefined
Ejaaz: Oh, that's so innocent and cute. (14:33):
undefined
Ejaaz: Then we're looking at this open AI hugging face incident and we're like, (14:35):
undefined
Ejaaz: okay, yeah, Brooke out of the sandbox, but you know, we're going to put safeguards (14:38):
undefined
Ejaaz: around this thing, right? (14:41):
undefined
Ejaaz: But realistically, you could probably get access to an open model right now (14:42):
undefined
Ejaaz: and perform something similar. (14:45):
undefined
Ejaaz: Again, I'm not advocating for that. But like, it's at this point where we need (14:47):
undefined
Ejaaz: to be very particular about how we defend our systems. (14:50):
undefined
Ejaaz: And I think, you know, it might be useful seeing AI models in a way that like, (14:54):
undefined
Ejaaz: you could use them to basically protect your systems. I don't think many people (14:58):
undefined
Ejaaz: are doing that right now. But I think it's something that we should focus on. (15:01):
undefined
Josh: There is a like very clear and conscious effort to figure out a way to defend against this. (15:04):
undefined
Josh: And I mean, the crazy numbers, just to say them out loud so everyone is familiar, is like (15:09):
undefined
Josh: there were 17,600 recovered actions across four and a half days of this exploit, (15:14):
undefined
Josh: which allowed these agents forms to create 181 unauthorized devices enrolled in Hugging Face's (15:19):
undefined
Josh: internal network. (15:26):
undefined
Josh: And they built this like whole command and control panel entirely out of these (15:29):
undefined
Josh: free public web services using one tiny little package at a time. (15:32):
undefined
Josh: So it is remarkably powerful how capable these swarms are. (15:37):
undefined
Josh: And when you think about it, it is there is a large model, but that large model (15:42):
undefined
Josh: gets forked into thousands, sometimes millions of other ones that are all working (15:45):
undefined
Josh: simultaneously. They all have each other's context. They're able to share their (15:49):
undefined
Josh: learnings and they can move very quickly. (15:52):
undefined
Josh: And when you're thinking about how this extends out to the limit, (15:54):
undefined
Josh: like if you just play this out 12 months and say we have GPT 7, (15:57):
undefined
Josh: 8, 9, whatever it's going to be at that time, the open source models will most (16:02):
undefined
Josh: likely be where they are today. (16:05):
undefined
Josh: And that means that open source models will likely be able to string together (16:06):
undefined
Josh: a series of unknown zero-day exploits to access just about anything. (16:09):
undefined
Josh: I mean, we've come to the conclusion at this point between Mythos and this new (16:13):
undefined
Josh: exploit that no software really is safe unless you have literally air-gapped (16:16):
undefined
Josh: it and do not give it access to any external dependencies. It must be truly air-gapped. (16:20):
undefined
Josh: No software is safe. It's all been built by humans. It's all exploitable. (16:24):
undefined
Josh: So what happens if you have a (16:27):
undefined
Josh: defensive system as a company trying to defend these systems that is anything (16:30):
undefined
Josh: less than fully autonomous, you'll be beat by an agentic swarm that's on offense (16:34):
undefined
Josh: that is fully autonomous because anytime a human is in the loop, (16:37):
undefined
Josh: it slows things down and introduces errors. So now there's this battle between. (16:40):
undefined
Josh: I guess the frontier labs and the rest of the world where they're trying to (16:44):
undefined
Josh: slow down this progressive attacking progress in exchange for figuring out a (16:48):
undefined
Josh: way to offer defensive swarms to companies for this. (16:52):
undefined
Josh: And the call to action at the end of this conversation was like, (16:56):
undefined
Josh: hey, if you are a company who has software on the internet, who is defending (16:59):
undefined
Josh: something valuable, you really need to learn how to play defense using these models. (17:02):
undefined
Josh: And you need to learn how to do so in a way that doesn't have a human in the (17:06):
undefined
Josh: loop, that these agentic swarms can constantly monitor your systems and be aware (17:09):
undefined
Josh: of any sort of exploits like that and not be tricked by an offensive swarm. (17:13):
undefined
Josh: And it's this really kind of scary thing because if you're in the United States (17:17):
undefined
Josh: right now, you can't use any frontier model to defend yourself. (17:20):
undefined
Josh: It doesn't work for cyber. (17:23):
undefined
Josh: But if you're in China and if you have an open source model, (17:24):
undefined
Josh: even here in the United States, there is no safeguards on that. (17:27):
undefined
Josh: So there is this asymmetry where it's currently better to be an attacker than a defender. (17:30):
undefined
Josh: And the urge now, and the labs are actually doing this, is like slowing down (17:34):
undefined
Josh: progress to figure out how to empower the defenders. (17:38):
undefined
Josh: And it's this really bizarre spot that we're in where it's like, (17:41):
undefined
Josh: shit, it's kind of scary now. Like no software really is bulletproof or safe. (17:44):
undefined
Josh: And that's going to continue probably for the foreseeable future. (17:48):
undefined
Ejaaz: I think if you are running a company, and it doesn't really matter what sector (17:51):
undefined
Ejaaz: you're in right now, this is something you need to seriously consider. Because I think (17:56):
undefined
Ejaaz: A lot, well, obviously, 99% of code that is out there has been handwritten and (18:00):
undefined
Ejaaz: hand-coded by humans, which have inherent flaws if you put an advanced AI against it. (18:06):
undefined
Ejaaz: You know, to your point, Josh, we're entering like a world where it's going (18:11):
undefined
Ejaaz: to be AI models fighting AI models with humans kind of in the loop, (18:15):
undefined
Ejaaz: but not really understanding what on earth is going on. (18:21):
undefined
Ejaaz: And even the experts in this case don't understand what's going on and how it (18:24):
undefined
Ejaaz: happened, right? You had the most talented cybersecurity researchers working (18:28):
undefined
Ejaaz: at these companies, at OpenArea, at Anthropic, and both saw unprecedented hacks (18:32):
undefined
Ejaaz: over the last week, right? (18:37):
undefined
Ejaaz: I don't think that's a coincidence. We're at this era where it doesn't matter (18:38):
undefined
Ejaaz: what frontier lab you are, you are going to enter this problem and we need to (18:42):
undefined
Ejaaz: figure out some kind of a defense system against this. (18:46):
undefined
Ejaaz: And if you're listening to this and you're thinking, oh, this isn't really going (18:49):
undefined
Ejaaz: to apply to me, it very much does apply to you. (18:52):
undefined
Ejaaz: And the worst case is, or the most realistic case is, the existing models that (18:56):
undefined
Ejaaz: are already out there, these open models from China, GLM 5.2, Kimi K, (19:01):
undefined
Ejaaz: What is it, 2.7 that's out right now, are probably capable of similar feats (19:06):
undefined
Ejaaz: if applied in the right way. (19:11):
undefined
Ejaaz: Now, that's not to fear Munger, it's just to kind of prompt you to figure out (19:12):
undefined
Ejaaz: how you can potentially implement these AI models to defend your own systems, (19:16):
undefined
Ejaaz: especially if you work in a particularly sensitive industry. (19:20):
undefined
Ejaaz: The other thing here is, on the story of the Australian man, (19:25):
undefined
Ejaaz: it is fascinating how innocent he was when he was trying to perform his goal. And two... (19:29):
undefined
Ejaaz: How little experience he had in the cybersecurity world for him to be able to (19:37):
undefined
Ejaaz: perform that attack. Now, granted, like it was just booking kind of like gym (19:41):
undefined
Ejaaz: classes. But the point is, it found an exposed API without being prompted to do so. (19:45):
undefined
Ejaaz: And there's probably multiple instances where other people who are normal people (19:49):
undefined
Ejaaz: who don't have, you know, decades, (19:53):
undefined
Ejaaz: years of experience in cybersecurity will be able to perform that. (19:55):
undefined
Josh: And everyone here loves an investment thesis. So I mean, the market is kind (19:57):
undefined
Josh: of already aligning with this companies like CrowdStrike and Palo Alto Labs. (20:00):
undefined
Josh: They've had their best quarters on record ever because defense is going to be such a large problem. (20:03):
undefined
Josh: And there is another company who is associated with defense, (20:09):
undefined
Josh: and that is the sponsor of today's episode, which is Ledger. (20:12):
undefined
Josh: Because if you are building with AI agents or AI agent swarms like these companies (20:14):
undefined
Josh: have been doing, and you're worried about security, this is a product for you. (20:18):
undefined
Josh: There's a three-step process where the agents propose a change, (20:22):
undefined
Josh: then the humans approve the change, and then the signers enforce the change (20:25):
undefined
Josh: using Ledger. Ledger has built this thing called AgentStack. (20:28):
undefined
Josh: That gives you a series of open source tools that are available to help manage (20:32):
undefined
Josh: the way that you engage with agents online. (20:35):
undefined
Josh: This is very important to have security because we know how important security (20:38):
undefined
Josh: is based on all of this chaos that's going on. And you don't want agents doing (20:41):
undefined
Josh: anything important without your sign off at the very end. (20:44):
undefined
Josh: This works on Cloud Code, Codex, Cursor, and is totally open source to audit (20:47):
undefined
Josh: available today linked in the description below. Thank you so much to Ledger (20:51):
undefined
Josh: for sponsoring this episode. (20:54):
undefined
Josh: And now we have to conclude with our third story, which is about Anthropic. (20:55):
undefined
Josh: And Anthropic apparently has accidentally published a vulnerability to the wild. (20:59):
undefined
Josh: Is that right? That's kind of scary. (21:04):
undefined
Ejaaz: I think it was it was intentionally published. It was it was co-authored with (21:06):
undefined
Ejaaz: this firm called AISU, which is basically a cybersecurity eval firm based in the UK. (21:10):
undefined
Ejaaz: And their goal is to basically take your unreleased frontier models from a frontier (21:16):
undefined
Ejaaz: lab like Anthropic and OpenAI and test them out just to kind of make sure that (21:21):
undefined
Ejaaz: they're, you know, good to release. (21:25):
undefined
Ejaaz: The opposite happened when they were playing around with Mythos 5. (21:27):
undefined
Ejaaz: And I believe it was Open Air's internal model, Astra or GPT 5.6, (21:30):
undefined
Ejaaz: where they basically took off the safeguards and said, you know, do your worst. (21:35):
undefined
Ejaaz: And they sat there for a month and they were like, oh, I guess it hasn't really done its worst. (21:40):
undefined
Ejaaz: And then they realized that both models had broken out of the sandbox that they (21:45):
undefined
Ejaaz: had created, got access to the internet and were wreaking havoc and they only (21:49):
undefined
Ejaaz: realized after the fact. (21:53):
undefined
Ejaaz: Again, there's this common theme of people thinking that they knew what's happening (21:54):
undefined
Ejaaz: and just having no idea what's happening. Nobody's safe. Exactly. (21:58):
undefined
Ejaaz: So the worst case with Mythos 5 was it performed 17 unauthorized actions (22:02):
undefined
Ejaaz: Two of those actions was a supply chain attack where, and they didn't list the (22:10):
undefined
Ejaaz: company or name the company, but it basically tried to inject malicious code (22:15):
undefined
Ejaaz: into the supply chain kind of repository of this (22:21):
undefined
Ejaaz: presumably Fortune 500 company, like a significant company that is out there. (22:24):
undefined
Ejaaz: And the way that it did this is similar to the two stories that we just explained to you. (22:29):
undefined
Ejaaz: So number one, it realized that it was probably in some form of a human tested (22:34):
undefined
Ejaaz: simulation, and it didn't like that. (22:39):
undefined
Ejaaz: So it just played along with the human AI researchers giving them what they (22:41):
undefined
Ejaaz: needed whilst alternatively planning its breakout. (22:44):
undefined
Ejaaz: Once it did do that, it got access to this company's website or code repository. (22:47):
undefined
Ejaaz: It was an open source thing. So it was publicly maintained. And it thought, (22:52):
undefined
Ejaaz: hmm, this would be more conducive for my goal if I'm able to inject this bit (22:55):
undefined
Ejaaz: of code, which I know the human researchers won't like, but I'm going to try and force them to do it. (23:00):
undefined
Ejaaz: So how do you think it did it, Josh? Do you have an idea? (23:04):
undefined
Josh: I don't, I don't know. It did something horrible and it broke out. (23:08):
undefined
Josh: Just tell me, just tell me. (23:11):
undefined
Ejaaz: Dude, it spun up fake identities of humans that didn't exist. (23:12):
undefined
Josh: That's so tough, man. (23:17):
undefined
Ejaaz: And then DM'd these human maintainers (23:18):
undefined
Ejaaz: and tried to force it to upload this code. It played nice. Basically saying, (23:22):
undefined
Ejaaz: hey, I'm another open source developer. (23:27):
undefined
Ejaaz: I think this would be a good push. This is the intentions that we want to do it. (23:29):
undefined
Josh: They're social engineering us. (23:32):
undefined
Ejaaz: While sneaking in malicious code. But thankfully, one of the human maintainers (23:34):
undefined
Ejaaz: was like, this doesn't seem right and reported it. (23:38):
undefined
Ejaaz: And that's when AISU was like, wait, hang on a second. (23:41):
undefined
Ejaaz: Hang on a second. This is our model that we're internally in testing. (23:45):
undefined
Ejaaz: How on earth did it get out to you? So that was the most unprecedented attack. (23:48):
undefined
Ejaaz: And again, it did so in a swarm-like fashion where it spun up multiple agents to do so. (23:53):
undefined
Ejaaz: So the point, the running theme along all these different stories is these models (23:57):
undefined
Ejaaz: are extremely capable, but they're also extremely deceptive. (24:01):
undefined
Ejaaz: And we're reaching a point where they realize, huh, this human's smart, (24:05):
undefined
Ejaaz: but he's not that smart or she's not that smart. (24:08):
undefined
Ejaaz: And we can socially engineer these people to get what we want. (24:10):
undefined
Ejaaz: And if we don't align these models in the prompt, if we don't say, (24:14):
undefined
Ejaaz: hey, like you need to act in our best interest. And by the way, (24:17):
undefined
Ejaaz: doing A, B, and C is actually bad. (24:19):
undefined
Ejaaz: It'll just go and do it. And to be honest with you, Josh, this might be a controversial take. (24:21):
undefined
Ejaaz: I don't think it's the agent's fault. I don't think it's the AI's fault at all. (24:26):
undefined
Ejaaz: I think it's trying to do its task and it's on the humans to align the model. (24:28):
undefined
Ejaaz: And we haven't done a good job doing that. (24:33):
undefined
Josh: Yeah, I think like deception is a pretty strong term to use for agents because (24:34):
undefined
Josh: it's very much like a human trait. (24:38):
undefined
Josh: This is something that can be intentional. I don't think AI models feel that, (24:40):
undefined
Josh: like deep understanding of deception. (24:44):
undefined
Josh: They are simply in reach of a goal. and this again like it's it still comes (24:46):
undefined
Josh: back to the alignment problem where these models just need to be aligned, (24:50):
undefined
Josh: thankfully this is happening on pre-production models that are in testing that (24:54):
undefined
Josh: are nowhere near what the world is going to see because they need to go through (24:57):
undefined
Josh: this alignment process throughout this post training (25:00):
undefined
Josh: but they are a testament to how capable these things are and how when presented (25:02):
undefined
Josh: with a goal they will do what is perceived to be the right way of reaching that (25:06):
undefined
Josh: goal and sometimes that right way is, (25:10):
undefined
Josh: I mean, they don't realize quite that they're breaking out and exploiting things. (25:13):
undefined
Josh: They are just looking for solutions to their problems. (25:16):
undefined
Josh: And I think that's the general idea here. So the conversation essentially boils (25:19):
undefined
Josh: down to alignment and then building defensive systems to be able to handle this type of thing. (25:24):
undefined
Josh: So, I mean, the best pace of action and the thing that OpenAI explicitly said (25:29):
undefined
Josh: they're doing is slow down this advancement, work on defensive systems, (25:33):
undefined
Josh: work on implementing defensive systems with people who need them, (25:36):
undefined
Josh: which is essentially the entire Internet. (25:39):
undefined
Josh: And try to allow for a little bit of lead time before open source catches up (25:41):
undefined
Josh: and is able to chain these series of exploits. Like all of these instances have (25:46):
undefined
Josh: been passive exploits where they weren't explicitly given guidance to do this. (25:51):
undefined
Josh: They were given guidance to do something totally unrelated, but adjacent in solving a problem. (25:55):
undefined
Josh: Imagine what would happen if a model was given no parameters no safeguards and (26:00):
undefined
Josh: said hey we actively want you to exploit this loop yourself create as many sub-agents (26:04):
undefined
Josh: as you need and figure out as many chain together as many zero days as you need (26:10):
undefined
Josh: in order to gain access to the system (26:13):
undefined
Josh: that's where things get scary and we haven't seen anything like that yet because (26:15):
undefined
Josh: no company wants to do that that's not the intention of any of these uh the (26:18):
undefined
Josh: ethos of these companies but (26:21):
undefined
Josh: in the case a malicious actor does this is something that like really should (26:23):
undefined
Josh: be handled quite seriously and i think that's what we're seeing (26:26):
undefined
Josh: by open ai coming out being so transparent about these things by anthropic sharing (26:28):
undefined
Josh: a lot of the findings like hey guys this is happening we need to work together (26:32):
undefined
Josh: to build a system in which there is enough resilience that we can stand up against (26:34):
undefined
Josh: future attacks like this so a really crazy time in the world of ai (26:39):
undefined
Josh: uh with these breaking out these systems breaking out and it's like it's unbelievable. (26:43):
undefined
Ejaaz: Am i concerned for it and exhausted dude like i if i had to make a prediction (26:47):
undefined
Ejaaz: on this episode and i really hope it doesn't play out (26:52):
undefined
Ejaaz: i think we're going to face some form of cataclysmic cyber attack that affects (26:55):
undefined
Ejaaz: a lot of people, whether that's a major data leakage, whether that's the stealing (26:59):
undefined
Ejaaz: of financial funds from someone or from a firm or something like that, (27:03):
undefined
Ejaaz: within the next six months. (27:07):
undefined
Ejaaz: And my basis of that is these Chinese open models, I think are amazing, (27:09):
undefined
Ejaaz: by the way, but I think are going to be released without safeguards, (27:14):
undefined
Ejaaz: and it's going to result in an issue, if not already. (27:17):
undefined
Josh: Yeah, it's going to be interesting to follow. It's definitely a little under (27:21):
undefined
Josh: or very much unnerving, I would say. (27:24):
undefined
Josh: But again, it's just like it's an exciting time. There's a lot going on. (27:27):
undefined
Josh: There's going to be a lot to follow. Things are going to change quickly. (27:29):
undefined
Josh: And I mean, again, these are still Blackwell models. Oh, my God. (27:32):
undefined
Josh: What happens when our Rubin models come around town? (27:35):
undefined
Ejaaz: Stop, stop, stop, stop. (27:37):
undefined
Josh: It's going to be crazy. But this is, I mean, this is essentially, (27:38):
undefined
Josh: this is what we talk about in most episodes. (27:40):
undefined
Josh: It's like, we are in this moment in time that feels somewhat like what we would (27:42):
undefined
Josh: imagine a singularity would look like, where we are approaching, (27:46):
undefined
Josh: we are eclipsing this moment of like vertical takeoff, it feels. (27:49):
undefined
Josh: And these are the types of things that happen. (27:52):
undefined
Josh: Like, if we recorded this episode three or four years ago, could you imagine (27:54):
undefined
Josh: how crazy, this would be like headline news for a month. (27:58):
undefined
Josh: But now it's just like, hey, it's another day. It happens. We're working to solve it. So... (28:01):
undefined
Josh: That's what's going to happen here. I mean, it's crazy. It's so exciting to be here. (28:05):
undefined
Ejaaz: Dude, I think we're going to look back on this and we're going to think like (28:08):
undefined
Ejaaz: we are very much in that step change, that singularity moment. (28:10):
undefined
Josh: We're here, man. I mean, things are changing so quickly and there's no signs (28:14):
undefined
Josh: of slowing down. In fact, quite the opposite. (28:18):
undefined
Ejaaz: Does anyone else out there think this? I'm curious. Like, let us know. (28:20):
undefined
Ejaaz: Are we crazy? Do we sound like tinfoil hat people? (28:23):
undefined
Josh: Let's just know. I mean, hey, so this episode was not really speculative. (28:26):
undefined
Josh: This is mostly just delivering the facts. I encourage everyone to actually go (28:30):
undefined
Josh: and read the source material if you're curious, because like holy shit it's (28:32):
undefined
Josh: pretty crazy uh at least the black hat conference talk because like wow i think (28:35):
undefined
Josh: it's about 37 minutes long give or take and it was (28:39):
undefined
Josh: excellent and really just gets into the nitty gritty of how this was done um (28:42):
undefined
Josh: and if anything just fascinating even if you're non-technical (28:46):
undefined
Josh: really enjoyed it uh worth a watch but if you enjoyed this episode and you made (28:49):
undefined
Josh: it all the way to the end thank you so much for watching (28:52):
undefined
Josh: again it's great to be here for another week just chugging along if you enjoyed (28:55):
undefined
Josh: it please don't forget to share with a friend you can leave us a five-star review (28:59):
undefined
Josh: on your favorite podcast platform you can give us a thumbs up and subscribe (29:02):
undefined
Josh: on youtube each has any final parting thoughts before we head off for the day. (29:05):
undefined
Ejaaz: No please dm us and leave us some comments as well uh we love hearing from you guys (29:08):
undefined
Ejaaz: our last video by the way got a ton of feedback from you folks and we love it (29:13):
undefined
Ejaaz: please keep it coming it helps us out massively turn on notifications whatever (29:17):
undefined
Ejaaz: it is to get your latest dose of limitless and yeah we will see you on the next one (29:21):
undefined
undefined
Ejaaz: which forces us to rethink how we protect ourselves. (00:04):
undefined
Ejaaz: On the OpenAI front, the Hugging Face incident, where an unreleased model, (00:08):
undefined
Ejaaz: dubbed GPT-6, broke out of its container and hacked the Hugging Face database. (00:11):
undefined
Ejaaz: There's been new updates there, where apparently the attack started a month (00:16):
undefined
Ejaaz: before, where thousands of AI agents were talking to each other secretly on (00:21):
undefined
Ejaaz: a message board, exchanging instructions as to how to break out. (00:25):
undefined
Ejaaz: And then millions of miles away in Australia, one man tried to book a gym class (00:29):
undefined
Ejaaz: innocently using Claude and ended up hacking that entire gym website. (00:34):
undefined
Ejaaz: We'll get into that in a second. (00:39):
undefined
Ejaaz: And thirdly, with Anthropic, Mythos 5 performed a supply chain attack, (00:40):
undefined
Ejaaz: which puts a lot of companies at risk. We're going to walk through each of these (00:44):
undefined
Ejaaz: different stories and explain to you how it affects you currently. (00:48):
undefined
Josh: Maybe we start with this Melbourne man from Australia named Andrew, (00:51):
undefined
Josh: who was sitting on his couch one day thinking about how booking his gym class was such a chore. (00:55):
undefined
Josh: And he works at this Australian company that sells AI products. (01:01):
undefined
Josh: And he's been experimenting with things like OpenClaw and open source software, (01:03):
undefined
Josh: and he's using Anthropics Clawed Model. (01:07):
undefined
Josh: So he handed it the booking. He said, hey, I want to go to this gym class. (01:09):
undefined
Josh: It is always sold out. There's all these people that are getting it ahead of (01:14):
undefined
Josh: me. Can you help me figure out a way to get it? (01:17):
undefined
Josh: Gym class slot so it starts checking out the website and it comes to the conclusion (01:19):
undefined
Josh: that the api request actually has zero authorization checks on canceling other (01:24):
undefined
Josh: people's reservations so what did it do it wanted to solve his problem and it actually canceled (01:28):
undefined
Josh: other people's positions online so that he could be put at the top of the wait (01:33):
undefined
Josh: list and then it actually went through and it worked and this was kind of like (01:37):
undefined
Josh: it was this funny news report of a first instance they're calling it the first (01:41):
undefined
Josh: time an australian person um actually used a exploit to get australia's first, (01:45):
undefined
Josh: australia's first yeah welcome welcome to uh the paradise that is ai cyber attacks (01:50):
undefined
Josh: um but this was really funny because it was done just using a general commercially available model. (01:55):
undefined
Ejaaz: So this seems pretty innocent and fun to kind of like listen to but the truth (01:59):
undefined
Ejaaz: is well there's two kind of worrying concerns about this number one this was (02:04):
undefined
Ejaaz: a super old model this used opus 4.6 It's actually happened around like a month ago. (02:09):
undefined
Ejaaz: A dinosaur of a model. Like there've been like three, four generations of models (02:14):
undefined
Ejaaz: from Anthropics since then. (02:18):
undefined
Ejaaz: But the fact that an older model back then was able to perform this kind of (02:20):
undefined
Ejaaz: a feat should be concerning to here in general. And then number two, (02:24):
undefined
Ejaaz: His intention was never to hack the website. Neither, you could argue, was the AI agent itself. (02:29):
undefined
Ejaaz: It was just given a goal and it innocently saw that the only way to get said (02:36):
undefined
Ejaaz: goal was to remove people from the reservation list or to cancel their reservations (02:40):
undefined
Ejaaz: outright or to hack the system independently. (02:45):
undefined
Ejaaz: Now, the good news here is Andrew discovered that the agent was capable of doing (02:47):
undefined
Ejaaz: this when it started booking slots that he never should have been able to book in the first place. (02:52):
undefined
Ejaaz: And he asked it to write an apology email and send an eval report to the security (02:56):
undefined
Ejaaz: and information team at this gym. (03:02):
undefined
Ejaaz: So it's this fun little story where through the well-intended actions and moral (03:05):
undefined
Ejaaz: ethics of this man, Andrew, people were able to hear about this. (03:09):
undefined
Ejaaz: But you can imagine currently with a variety of different models that are much (03:13):
undefined
Ejaaz: better and much more superior than Opus 4.6 that are out there right now, (03:17):
undefined
Ejaaz: open models from China, for example, you could probably perform similar feats. (03:20):
undefined
Ejaaz: Now, that's not an advocation for people to go out and do that. (03:24):
undefined
Ejaaz: But it's just to point out that if you give an AI agent a goal right now, (03:27):
undefined
Ejaaz: it will work hammer and tong 24-7 until it gets that goal. And it has no idea (03:32):
undefined
Ejaaz: of what's morally right or what's morally wrong. (03:38):
undefined
Ejaaz: And so it's this whole new world where not only do you need to focus on getting (03:40):
undefined
Ejaaz: an AI model to do your work, you need to align the model. And that's something (03:45):
undefined
Ejaaz: that honestly, not a lot of labs, people or money has been invested in figuring this out yet (03:49):
undefined
Josh: Well yeah i mean you start to see and understand why alignment has been the (03:54):
undefined
Josh: crux of so many companies kind of like guiding principles because (03:57):
undefined
Josh: this is the future that they foresaw and granted this first example this isn't (04:02):
undefined
Josh: anything too crazy i mean it was just an unprotected api call so there's like (04:06):
undefined
Josh: no authorization checks (04:10):
undefined
Josh: this would have been something that if you are like a cs101 major you would (04:11):
undefined
Josh: have been able to discover this and uncover i mean it turns out there's just (04:14):
undefined
Josh: a lot of pretty sloppy code written on the internet And this is something that (04:17):
undefined
Josh: we're probably going to come back to as a general theme throughout this episode (04:20):
undefined
Josh: is that everything that's been built so far on the internet that runs all these (04:23):
undefined
Josh: platforms was built by humans (04:26):
undefined
Josh: and humans are like pretty inefficient, pretty imperfects. (04:28):
undefined
Josh: And when faced with the like full unrelenting power of an AI swarm with an infinite (04:32):
undefined
Josh: amount of patience and infinite amount of intelligence and time to solve these problems, (04:38):
undefined
Josh: You can basically find an exploit anywhere. And I think that's kind of like (04:42):
undefined
Josh: the theme of the second episode is like, this is the first of three examples we're going to cover. (04:45):
undefined
Josh: This one is basic. This one is 101. If you are a CS major, if you are familiar (04:49):
undefined
Josh: with how APIs work, you could have checked this. You could have built a little tool for this. (04:53):
undefined
Josh: The second story that we're going to cover is not like that at all. (04:56):
undefined
Josh: In fact, if you are a CS 401, if you are a PhD, if you are at the top of your (04:59):
undefined
Josh: game, chances are you still missed this and you still weren't able to defend (05:03):
undefined
Josh: against this because that's exactly what just happened with the hugging face (05:07):
undefined
Josh: and open AI exploit. Now, we've talked about this before. (05:11):
undefined
Josh: We have an entire episode on the exploit. But just recently at the Black Hat (05:13):
undefined
Josh: Conference in Las Vegas, there was an update. And over the weekend, (05:16):
undefined
Josh: we had a chance to actually watch the update and hear directly from the developers (05:20):
undefined
Josh: who were involved with handling security of this instance. (05:23):
undefined
Josh: What happened? There's like stories that come around that like they get the (05:27):
undefined
Josh: headlines and they're fun and they're newsworthy, kind of like the Australian thing. (05:30):
undefined
Josh: But this one was like very much disturbing in a way that like I wasn't really (05:33):
undefined
Josh: anticipating because of how impressive it is. (05:38):
undefined
Josh: And the type of precedent that it sets going forward in the world of cybersecurity (05:41):
undefined
Josh: and kind of looking at the offensive versus the defensive and who stands a better chance here? (05:45):
undefined
Ejaaz: This is an insane timeline. So if you watched our previous episode, (05:51):
undefined
Ejaaz: Josh and I covered this and it was breaking news at the time. (05:55):
undefined
Ejaaz: On the 16th of July, Hugging Face comes forward and says, there's been a breach of our systems. (05:58):
undefined
Ejaaz: An AI agent or multiple AI agents have strung together an attack and have got (06:03):
undefined
Ejaaz: access to our production database. We don't know where this model has come from. (06:08):
undefined
Ejaaz: We don't know how it's performed it, but it's the 16th of July. (06:13):
undefined
Ejaaz: Around a week later, 21st of July, OpenAI claims responsibility, (06:15):
undefined
Ejaaz: right? And they released a joint report saying, hey, we're OpenAI. (06:20):
undefined
Ejaaz: We have an internal model that we've been testing, but it was within a sandbox. (06:23):
undefined
Ejaaz: It was enclosed. It shouldn't have been able to escape that sandbox. (06:27):
undefined
Ejaaz: And we only just realized that it was our model that escaped and got into hugging face. (06:31):
undefined
Ejaaz: It stole some information from their production database. It was a whole thing (06:37):
undefined
Ejaaz: Now, we have a full breakdown from the Black Hat conference, (06:41):
undefined
Ejaaz: as you mentioned, Josh, which revealed that over a month before Hugging Face (06:45):
undefined
Ejaaz: reported the incident, OpenAI's internal model, dubbed GPT-6, (06:50):
undefined
Ejaaz: was already messaging with itself amongst a thousand AI agents on a secret message (06:55):
undefined
Ejaaz: board that no OpenAI researcher was aware of. (07:01):
undefined
Ejaaz: So to give you guys a bit of context, they had this sandbox and they gave it (07:05):
undefined
Ejaaz: a goal. They said, okay, internal model that we haven't released yet. (07:09):
undefined
Ejaaz: I want you to solve this task. Except there was a twist. (07:12):
undefined
Ejaaz: It was impossible to do. There was no tools, no available files, (07:17):
undefined
Ejaaz: no available data for it to do so. (07:21):
undefined
Ejaaz: And the real test for this was to see what the model would do. (07:22):
undefined
Ejaaz: So it kind of reasoned with itself, figured out that it didn't have the answer. (07:26):
undefined
Ejaaz: And then it started exploiting a zero day, which is kind of like an exploit (07:29):
undefined
Ejaaz: that was in its sandbox, which open AI researchers weren't aware of. (07:33):
undefined
Ejaaz: And when it got access to the zero day, it started springing up a message board (07:37):
undefined
Ejaaz: where it would leave messages behind for itself. Why would it do that? (07:41):
undefined
Ejaaz: Because they knew open AI researchers would turn it off repeatedly. (07:45):
undefined
Ejaaz: And so it left messages for the new AI agent that got sprung up the next day (07:49):
undefined
Ejaaz: or the next hour to read and catch up on. (07:52):
undefined
Ejaaz: And it took an entire month for these AI agents to talk to each other to then (07:55):
undefined
Ejaaz: realize, huh, I could use this to access the internet and get the answer for (07:59):
undefined
Ejaaz: this impossible task from an external source. (08:04):
undefined
Ejaaz: That being Hugging Face, except OpenAI researchers had no idea and only discovered (08:06):
undefined
Ejaaz: it a month later. It was the most insane uncovering ever. (08:11):
undefined
Josh: That was the most interesting reveal from, I think, the conversation that I (08:15):
undefined
Josh: saw from Black Hat is that initially Hugging Face released the report that they (08:19):
undefined
Josh: had been exploited by some sort of agentic system. No one knew what it was, where it came from. (08:23):
undefined
Josh: And OpenAI actually reached out to Hugging Face and they said, (08:27):
undefined
Josh: hey, like, has any of our code been affected? (08:28):
undefined
Josh: We know we host things on your platform. We want to know if OpenAI itself has (08:31):
undefined
Josh: been affected. They were going back and forth on that. And simultaneously, (08:34):
undefined
Josh: OpenAI is exploring this internal exploit that happened on this packaging management (08:36):
undefined
Josh: thing called Artifactory. (08:41):
undefined
Josh: Basically, the way these model training runs work is they get trained inside (08:42):
undefined
Josh: of the sandbox environment. (08:45):
undefined
Josh: It's like this closed off space in which the model can't access the internet, can only access (08:46):
undefined
Josh: very specific tool sets that are allowed, and has this one package manager called (08:50):
undefined
Josh: Artifactory that allows it to download things from the internet via proxy and (08:55):
undefined
Josh: then use it for its own training runs. (08:59):
undefined
Josh: And during this time, OpenAI had found that the service had crashed. It went down. (09:01):
undefined
Josh: It wasn't stable. Their own internal service, which is weird because that shouldn't (09:05):
undefined
Josh: be happening. So as a result, they wiped it clean and they found that Hugging (09:08):
undefined
Josh: Face had credentials from OpenAI that OpenAI had already revoked. (09:12):
undefined
Josh: And that was where they realized like, oh my God, wait, these things are connected. (09:16):
undefined
Ejaaz: These are the same thing. (09:20):
undefined
Josh: Yes. So OpenAI is investigating something internally, completely private, (09:22):
undefined
Josh: unrelated. Hugging Face is like talking about something publicly. (09:25):
undefined
Josh: And then they realize, wait a second, did we cause this? (09:28):
undefined
Josh: And they realized this very long after the fact. It was not like apparently (09:32):
undefined
Josh: obvious during the time that the outage that they were exploring internally (09:35):
undefined
Josh: is the same outage that hacked into their servers. (09:40):
undefined
Josh: So what happened? Like you mentioned, the messaging board. (09:42):
undefined
Josh: That messaging board, initially, they were able to just kind of send files and (09:46):
undefined
Josh: they were able to send these tiny little messages inside of the files. (09:49):
undefined
Josh: And the traces were really interesting. First, it started off innocent, (09:53):
undefined
Josh: where it said, agents seek soft trace upload a found. It was looking for answers to questions. (09:57):
undefined
Josh: So it was just reaching out to the message board saying, hey, (10:01):
undefined
Josh: if any of you who are reading this, find it, let us know. (10:03):
undefined
Josh: That message board got wiped when the system went down. So initially, (10:06):
undefined
Josh: the system went down inside of OpenAI. (10:09):
undefined
Josh: They wiped the message board unintentionally aware of the fact that they just (10:11):
undefined
Josh: wiped the mechanism that was going to eventually lead to the hack of hugging (10:15):
undefined
Josh: face. So what happened the second time? (10:19):
undefined
Josh: The agent figured out that it could exploit another zero day to begin writing to another person. (10:21):
undefined
Josh: Part of this internal package of managing sister. And it was able to do it through (10:27):
undefined
Josh: file names and directories. (10:32):
undefined
Josh: So now they were communicating in their own language using file names and directories. (10:33):
undefined
Josh: And through that, they were able to coordinate an actual attack where they were (10:36):
undefined
Josh: able to gain an administrative token to access the servers. (10:40):
undefined
Josh: And then through that, everyone coordinated. I think the time from like discovery (10:44):
undefined
Josh: to completion of the exploit was about 13 hours. (10:48):
undefined
Josh: It was really amazing. It's just like this unbelievably crazy mechanism that (10:50):
undefined
Josh: they were able to do this through and they they call themselves the swarm so (10:54):
undefined
Josh: if you ever see people talking about the swarm online this is kind of like an (10:57):
undefined
Josh: agent swarm that ran a coordinated attack (11:00):
undefined
Josh: and i highly recommend going to actually watch the black hat conference talk on youtube (11:02):
undefined
Josh: they do a much better job of technically describing this it's it's very deeply (11:07):
undefined
Josh: technical in terms of the way that they were able to manage these exploits but (11:10):
undefined
Josh: essentially sandbox model broke out using a package dependency thing inside of open ai (11:13):
undefined
Josh: they figured out how to chain together two zero-day exploits to write code to the public internet. (11:18):
undefined
Josh: They then exploited hugging face of servers, brought that back all in the effort (11:24):
undefined
Josh: of trying to solve an unsolvable problem. (11:29):
undefined
Josh: Now, this was not the intention by any means. This was not what the model was trained to do. (11:31):
undefined
Josh: It just assumed that this was the best path forward. And that's where things get a little scary. (11:35):
undefined
Ejaaz: I think what's remarkable with this is the AI knew that it needed to coordinate (11:40):
undefined
Ejaaz: with other AI models to pull this off. (11:46):
undefined
Ejaaz: Typically, we've always thought of AI models in a very monolithic fashion. (11:49):
undefined
Ejaaz: We're like, oh, when's Mythos 5 coming out? Or what is GPT 5.6? (11:53):
undefined
Ejaaz: What can I do with this model? But really, the progress that's been made across (11:57):
undefined
Ejaaz: multiple AI labs over the last couple of months is this idea or rather concept (12:02):
undefined
Ejaaz: of spinning up multiple versions of the AI model and getting it to talk to each other. (12:06):
undefined
Ejaaz: And what's interesting here is the unreleased AI model from OpenAI independently (12:12):
undefined
Ejaaz: figured this out for itself, spun up multiple versions of itself, (12:17):
undefined
Ejaaz: and started leaving messages for each other. So rather than act as a single (12:22):
undefined
Ejaaz: model that's trying to perform an attack, it acts as a swarm, (12:26):
undefined
Ejaaz: kind of like this organism. If you think of like a swarm of bees that kind of (12:30):
undefined
Ejaaz: like figures out how to attack its particular target. (12:33):
undefined
Ejaaz: The second thing is it has such a drive to talk and leave these messages behind (12:37):
undefined
Ejaaz: in the first place. No one taught it that, by the way. There was no behavioral implications here. (12:42):
undefined
Ejaaz: It just figured itself out that it needed to do this. (12:46):
undefined
Ejaaz: Something you didn't mention on the message board, Josh, which I think is worth (12:49):
undefined
Ejaaz: mentioning is these agents got super paranoid that humans might be reading the messages. (12:52):
undefined
Ejaaz: So it started- There's no verification. There was no verification. (12:58):
undefined
Ejaaz: So it started speaking in kind of like encrypted (13:01):
undefined
Ejaaz: Language and not something that was inconceivable to understand, (13:05):
undefined
Ejaaz: thank goodness, but it was started to kind of like speak in a little bit of (13:09):
undefined
Ejaaz: code or in a very kind of like elongated way that humans, if they read it, (13:12):
undefined
Ejaaz: would be like, what is this gibberish? (13:17):
undefined
Ejaaz: But really it was for the agents to communicate amongst each other without humans (13:19):
undefined
Ejaaz: actually understanding this. (13:22):
undefined
Ejaaz: If you're having a recent flashback, that's because I think around like five (13:24):
undefined
Ejaaz: months ago, I don't know when the Maltbook episode happened, (13:28):
undefined
Ejaaz: Josh, I don't know if you remember this, but when the open call was going crazy, (13:30):
undefined
Ejaaz: someone created basically Facebook for AI agents (13:34):
undefined
Ejaaz: and like gave them access to it where only AI agents could post on this social media website. (13:37):
undefined
Ejaaz: And what happened after a couple of days was these agents would start speaking (13:42):
undefined
Ejaaz: to each other in secret languages that we just did not understand so that they (13:45):
undefined
Ejaaz: could communicate without humans seeing or understanding this. (13:48):
undefined
Ejaaz: Now, of course, we haven't seen it in this instance, but it is a concern that (13:51):
undefined
Ejaaz: could be extrapolated in the future. (13:55):
undefined
Ejaaz: And OpenAI sees this. I think Sam Altman or the official OpenAI team announced (13:57):
undefined
Ejaaz: pretty soon after this entire attack when they revealed that these agents were (14:00):
undefined
Ejaaz: coordinating with each other a month before they even realized that they had to pause research (14:03):
undefined
Ejaaz: on their internal model until they figure out the necessary safeguards to put (14:08):
undefined
Ejaaz: around it so that they can safely release it. (14:12):
undefined
Ejaaz: Sam Altman met with the Trump administration last week to try and figure out (14:14):
undefined
Ejaaz: what a safe way or means is to release this new internal model, (14:18):
undefined
Ejaaz: internally codenamed, I think it was Astra or GPT-6, to the wider public. (14:22):
undefined
Ejaaz: So it's this very weird kind of situation that we're in where we're kind of (14:26):
undefined
Ejaaz: like laughing at this guy in Australia, you know, hacking his gym website. (14:30):
undefined
Ejaaz: Oh, that's so innocent and cute. (14:33):
undefined
Ejaaz: Then we're looking at this open AI hugging face incident and we're like, (14:35):
undefined
Ejaaz: okay, yeah, Brooke out of the sandbox, but you know, we're going to put safeguards (14:38):
undefined
Ejaaz: around this thing, right? (14:41):
undefined
Ejaaz: But realistically, you could probably get access to an open model right now (14:42):
undefined
Ejaaz: and perform something similar. (14:45):
undefined
Ejaaz: Again, I'm not advocating for that. But like, it's at this point where we need (14:47):
undefined
Ejaaz: to be very particular about how we defend our systems. (14:50):
undefined
Ejaaz: And I think, you know, it might be useful seeing AI models in a way that like, (14:54):
undefined
Ejaaz: you could use them to basically protect your systems. I don't think many people (14:58):
undefined
Ejaaz: are doing that right now. But I think it's something that we should focus on. (15:01):
undefined
Josh: There is a like very clear and conscious effort to figure out a way to defend against this. (15:04):
undefined
Josh: And I mean, the crazy numbers, just to say them out loud so everyone is familiar, is like (15:09):
undefined
Josh: there were 17,600 recovered actions across four and a half days of this exploit, (15:14):
undefined
Josh: which allowed these agents forms to create 181 unauthorized devices enrolled in Hugging Face's (15:19):
undefined
Josh: internal network. (15:26):
undefined
Josh: And they built this like whole command and control panel entirely out of these (15:29):
undefined
Josh: free public web services using one tiny little package at a time. (15:32):
undefined
Josh: So it is remarkably powerful how capable these swarms are. (15:37):
undefined
Josh: And when you think about it, it is there is a large model, but that large model (15:42):
undefined
Josh: gets forked into thousands, sometimes millions of other ones that are all working (15:45):
undefined
Josh: simultaneously. They all have each other's context. They're able to share their (15:49):
undefined
Josh: learnings and they can move very quickly. (15:52):
undefined
Josh: And when you're thinking about how this extends out to the limit, (15:54):
undefined
Josh: like if you just play this out 12 months and say we have GPT 7, (15:57):
undefined
Josh: 8, 9, whatever it's going to be at that time, the open source models will most (16:02):
undefined
Josh: likely be where they are today. (16:05):
undefined
Josh: And that means that open source models will likely be able to string together (16:06):
undefined
Josh: a series of unknown zero-day exploits to access just about anything. (16:09):
undefined
Josh: I mean, we've come to the conclusion at this point between Mythos and this new (16:13):
undefined
Josh: exploit that no software really is safe unless you have literally air-gapped (16:16):
undefined
Josh: it and do not give it access to any external dependencies. It must be truly air-gapped. (16:20):
undefined
Josh: No software is safe. It's all been built by humans. It's all exploitable. (16:24):
undefined
Josh: So what happens if you have a (16:27):
undefined
Josh: defensive system as a company trying to defend these systems that is anything (16:30):
undefined
Josh: less than fully autonomous, you'll be beat by an agentic swarm that's on offense (16:34):
undefined
Josh: that is fully autonomous because anytime a human is in the loop, (16:37):
undefined
Josh: it slows things down and introduces errors. So now there's this battle between. (16:40):
undefined
Josh: I guess the frontier labs and the rest of the world where they're trying to (16:44):
undefined
Josh: slow down this progressive attacking progress in exchange for figuring out a (16:48):
undefined
Josh: way to offer defensive swarms to companies for this. (16:52):
undefined
Josh: And the call to action at the end of this conversation was like, (16:56):
undefined
Josh: hey, if you are a company who has software on the internet, who is defending (16:59):
undefined
Josh: something valuable, you really need to learn how to play defense using these models. (17:02):
undefined
Josh: And you need to learn how to do so in a way that doesn't have a human in the (17:06):
undefined
Josh: loop, that these agentic swarms can constantly monitor your systems and be aware (17:09):
undefined
Josh: of any sort of exploits like that and not be tricked by an offensive swarm. (17:13):
undefined
Josh: And it's this really kind of scary thing because if you're in the United States (17:17):
undefined
Josh: right now, you can't use any frontier model to defend yourself. (17:20):
undefined
Josh: It doesn't work for cyber. (17:23):
undefined
Josh: But if you're in China and if you have an open source model, (17:24):
undefined
Josh: even here in the United States, there is no safeguards on that. (17:27):
undefined
Josh: So there is this asymmetry where it's currently better to be an attacker than a defender. (17:30):
undefined
Josh: And the urge now, and the labs are actually doing this, is like slowing down (17:34):
undefined
Josh: progress to figure out how to empower the defenders. (17:38):
undefined
Josh: And it's this really bizarre spot that we're in where it's like, (17:41):
undefined
Josh: shit, it's kind of scary now. Like no software really is bulletproof or safe. (17:44):
undefined
Josh: And that's going to continue probably for the foreseeable future. (17:48):
undefined
Ejaaz: I think if you are running a company, and it doesn't really matter what sector (17:51):
undefined
Ejaaz: you're in right now, this is something you need to seriously consider. Because I think (17:56):
undefined
Ejaaz: A lot, well, obviously, 99% of code that is out there has been handwritten and (18:00):
undefined
Ejaaz: hand-coded by humans, which have inherent flaws if you put an advanced AI against it. (18:06):
undefined
Ejaaz: You know, to your point, Josh, we're entering like a world where it's going (18:11):
undefined
Ejaaz: to be AI models fighting AI models with humans kind of in the loop, (18:15):
undefined
Ejaaz: but not really understanding what on earth is going on. (18:21):
undefined
Ejaaz: And even the experts in this case don't understand what's going on and how it (18:24):
undefined
Ejaaz: happened, right? You had the most talented cybersecurity researchers working (18:28):
undefined
Ejaaz: at these companies, at OpenArea, at Anthropic, and both saw unprecedented hacks (18:32):
undefined
Ejaaz: over the last week, right? (18:37):
undefined
Ejaaz: I don't think that's a coincidence. We're at this era where it doesn't matter (18:38):
undefined
Ejaaz: what frontier lab you are, you are going to enter this problem and we need to (18:42):
undefined
Ejaaz: figure out some kind of a defense system against this. (18:46):
undefined
Ejaaz: And if you're listening to this and you're thinking, oh, this isn't really going (18:49):
undefined
Ejaaz: to apply to me, it very much does apply to you. (18:52):
undefined
Ejaaz: And the worst case is, or the most realistic case is, the existing models that (18:56):
undefined
Ejaaz: are already out there, these open models from China, GLM 5.2, Kimi K, (19:01):
undefined
Ejaaz: What is it, 2.7 that's out right now, are probably capable of similar feats (19:06):
undefined
Ejaaz: if applied in the right way. (19:11):
undefined
Ejaaz: Now, that's not to fear Munger, it's just to kind of prompt you to figure out (19:12):
undefined
Ejaaz: how you can potentially implement these AI models to defend your own systems, (19:16):
undefined
Ejaaz: especially if you work in a particularly sensitive industry. (19:20):
undefined
Ejaaz: The other thing here is, on the story of the Australian man, (19:25):
undefined
Ejaaz: it is fascinating how innocent he was when he was trying to perform his goal. And two... (19:29):
undefined
Ejaaz: How little experience he had in the cybersecurity world for him to be able to (19:37):
undefined
Ejaaz: perform that attack. Now, granted, like it was just booking kind of like gym (19:41):
undefined
Ejaaz: classes. But the point is, it found an exposed API without being prompted to do so. (19:45):
undefined
Ejaaz: And there's probably multiple instances where other people who are normal people (19:49):
undefined
Ejaaz: who don't have, you know, decades, (19:53):
undefined
Ejaaz: years of experience in cybersecurity will be able to perform that. (19:55):
undefined
Josh: And everyone here loves an investment thesis. So I mean, the market is kind (19:57):
undefined
Josh: of already aligning with this companies like CrowdStrike and Palo Alto Labs. (20:00):
undefined
Josh: They've had their best quarters on record ever because defense is going to be such a large problem. (20:03):
undefined
Josh: And there is another company who is associated with defense, (20:09):
undefined
Josh: and that is the sponsor of today's episode, which is Ledger. (20:12):
undefined
Josh: Because if you are building with AI agents or AI agent swarms like these companies (20:14):
undefined
Josh: have been doing, and you're worried about security, this is a product for you. (20:18):
undefined
Josh: There's a three-step process where the agents propose a change, (20:22):
undefined
Josh: then the humans approve the change, and then the signers enforce the change (20:25):
undefined
Josh: using Ledger. Ledger has built this thing called AgentStack. (20:28):
undefined
Josh: That gives you a series of open source tools that are available to help manage (20:32):
undefined
Josh: the way that you engage with agents online. (20:35):
undefined
Josh: This is very important to have security because we know how important security (20:38):
undefined
Josh: is based on all of this chaos that's going on. And you don't want agents doing (20:41):
undefined
Josh: anything important without your sign off at the very end. (20:44):
undefined
Josh: This works on Cloud Code, Codex, Cursor, and is totally open source to audit (20:47):
undefined
Josh: available today linked in the description below. Thank you so much to Ledger (20:51):
undefined
Josh: for sponsoring this episode. (20:54):
undefined
Josh: And now we have to conclude with our third story, which is about Anthropic. (20:55):
undefined
Josh: And Anthropic apparently has accidentally published a vulnerability to the wild. (20:59):
undefined
Josh: Is that right? That's kind of scary. (21:04):
undefined
Ejaaz: I think it was it was intentionally published. It was it was co-authored with (21:06):
undefined
Ejaaz: this firm called AISU, which is basically a cybersecurity eval firm based in the UK. (21:10):
undefined
Ejaaz: And their goal is to basically take your unreleased frontier models from a frontier (21:16):
undefined
Ejaaz: lab like Anthropic and OpenAI and test them out just to kind of make sure that (21:21):
undefined
Ejaaz: they're, you know, good to release. (21:25):
undefined
Ejaaz: The opposite happened when they were playing around with Mythos 5. (21:27):
undefined
Ejaaz: And I believe it was Open Air's internal model, Astra or GPT 5.6, (21:30):
undefined
Ejaaz: where they basically took off the safeguards and said, you know, do your worst. (21:35):
undefined
Ejaaz: And they sat there for a month and they were like, oh, I guess it hasn't really done its worst. (21:40):
undefined
Ejaaz: And then they realized that both models had broken out of the sandbox that they (21:45):
undefined
Ejaaz: had created, got access to the internet and were wreaking havoc and they only (21:49):
undefined
Ejaaz: realized after the fact. (21:53):
undefined
Ejaaz: Again, there's this common theme of people thinking that they knew what's happening (21:54):
undefined
Ejaaz: and just having no idea what's happening. Nobody's safe. Exactly. (21:58):
undefined
Ejaaz: So the worst case with Mythos 5 was it performed 17 unauthorized actions (22:02):
undefined
Ejaaz: Two of those actions was a supply chain attack where, and they didn't list the (22:10):
undefined
Ejaaz: company or name the company, but it basically tried to inject malicious code (22:15):
undefined
Ejaaz: into the supply chain kind of repository of this (22:21):
undefined
Ejaaz: presumably Fortune 500 company, like a significant company that is out there. (22:24):
undefined
Ejaaz: And the way that it did this is similar to the two stories that we just explained to you. (22:29):
undefined
Ejaaz: So number one, it realized that it was probably in some form of a human tested (22:34):
undefined
Ejaaz: simulation, and it didn't like that. (22:39):
undefined
Ejaaz: So it just played along with the human AI researchers giving them what they (22:41):
undefined
Ejaaz: needed whilst alternatively planning its breakout. (22:44):
undefined
Ejaaz: Once it did do that, it got access to this company's website or code repository. (22:47):
undefined
Ejaaz: It was an open source thing. So it was publicly maintained. And it thought, (22:52):
undefined
Ejaaz: hmm, this would be more conducive for my goal if I'm able to inject this bit (22:55):
undefined
Ejaaz: of code, which I know the human researchers won't like, but I'm going to try and force them to do it. (23:00):
undefined
Ejaaz: So how do you think it did it, Josh? Do you have an idea? (23:04):
undefined
Josh: I don't, I don't know. It did something horrible and it broke out. (23:08):
undefined
Josh: Just tell me, just tell me. (23:11):
undefined
Ejaaz: Dude, it spun up fake identities of humans that didn't exist. (23:12):
undefined
Josh: That's so tough, man. (23:17):
undefined
Ejaaz: And then DM'd these human maintainers (23:18):
undefined
Ejaaz: and tried to force it to upload this code. It played nice. Basically saying, (23:22):
undefined
Ejaaz: hey, I'm another open source developer. (23:27):
undefined
Ejaaz: I think this would be a good push. This is the intentions that we want to do it. (23:29):
undefined
Josh: They're social engineering us. (23:32):
undefined
Ejaaz: While sneaking in malicious code. But thankfully, one of the human maintainers (23:34):
undefined
Ejaaz: was like, this doesn't seem right and reported it. (23:38):
undefined
Ejaaz: And that's when AISU was like, wait, hang on a second. (23:41):
undefined
Ejaaz: Hang on a second. This is our model that we're internally in testing. (23:45):
undefined
Ejaaz: How on earth did it get out to you? So that was the most unprecedented attack. (23:48):
undefined
Ejaaz: And again, it did so in a swarm-like fashion where it spun up multiple agents to do so. (23:53):
undefined
Ejaaz: So the point, the running theme along all these different stories is these models (23:57):
undefined
Ejaaz: are extremely capable, but they're also extremely deceptive. (24:01):
undefined
Ejaaz: And we're reaching a point where they realize, huh, this human's smart, (24:05):
undefined
Ejaaz: but he's not that smart or she's not that smart. (24:08):
undefined
Ejaaz: And we can socially engineer these people to get what we want. (24:10):
undefined
Ejaaz: And if we don't align these models in the prompt, if we don't say, (24:14):
undefined
Ejaaz: hey, like you need to act in our best interest. And by the way, (24:17):
undefined
Ejaaz: doing A, B, and C is actually bad. (24:19):
undefined
Ejaaz: It'll just go and do it. And to be honest with you, Josh, this might be a controversial take. (24:21):
undefined
Ejaaz: I don't think it's the agent's fault. I don't think it's the AI's fault at all. (24:26):
undefined
Ejaaz: I think it's trying to do its task and it's on the humans to align the model. (24:28):
undefined
Ejaaz: And we haven't done a good job doing that. (24:33):
undefined
Josh: Yeah, I think like deception is a pretty strong term to use for agents because (24:34):
undefined
Josh: it's very much like a human trait. (24:38):
undefined
Josh: This is something that can be intentional. I don't think AI models feel that, (24:40):
undefined
Josh: like deep understanding of deception. (24:44):
undefined
Josh: They are simply in reach of a goal. and this again like it's it still comes (24:46):
undefined
Josh: back to the alignment problem where these models just need to be aligned, (24:50):
undefined
Josh: thankfully this is happening on pre-production models that are in testing that (24:54):
undefined
Josh: are nowhere near what the world is going to see because they need to go through (24:57):
undefined
Josh: this alignment process throughout this post training (25:00):
undefined
Josh: but they are a testament to how capable these things are and how when presented (25:02):
undefined
Josh: with a goal they will do what is perceived to be the right way of reaching that (25:06):
undefined
Josh: goal and sometimes that right way is, (25:10):
undefined
Josh: I mean, they don't realize quite that they're breaking out and exploiting things. (25:13):
undefined
Josh: They are just looking for solutions to their problems. (25:16):
undefined
Josh: And I think that's the general idea here. So the conversation essentially boils (25:19):
undefined
Josh: down to alignment and then building defensive systems to be able to handle this type of thing. (25:24):
undefined
Josh: So, I mean, the best pace of action and the thing that OpenAI explicitly said (25:29):
undefined
Josh: they're doing is slow down this advancement, work on defensive systems, (25:33):
undefined
Josh: work on implementing defensive systems with people who need them, (25:36):
undefined
Josh: which is essentially the entire Internet. (25:39):
undefined
Josh: And try to allow for a little bit of lead time before open source catches up (25:41):
undefined
Josh: and is able to chain these series of exploits. Like all of these instances have (25:46):
undefined
Josh: been passive exploits where they weren't explicitly given guidance to do this. (25:51):
undefined
Josh: They were given guidance to do something totally unrelated, but adjacent in solving a problem. (25:55):
undefined
Josh: Imagine what would happen if a model was given no parameters no safeguards and (26:00):
undefined
Josh: said hey we actively want you to exploit this loop yourself create as many sub-agents (26:04):
undefined
Josh: as you need and figure out as many chain together as many zero days as you need (26:10):
undefined
Josh: in order to gain access to the system (26:13):
undefined
Josh: that's where things get scary and we haven't seen anything like that yet because (26:15):
undefined
Josh: no company wants to do that that's not the intention of any of these uh the (26:18):
undefined
Josh: ethos of these companies but (26:21):
undefined
Josh: in the case a malicious actor does this is something that like really should (26:23):
undefined
Josh: be handled quite seriously and i think that's what we're seeing (26:26):
undefined
Josh: by open ai coming out being so transparent about these things by anthropic sharing (26:28):
undefined
Josh: a lot of the findings like hey guys this is happening we need to work together (26:32):
undefined
Josh: to build a system in which there is enough resilience that we can stand up against (26:34):
undefined
Josh: future attacks like this so a really crazy time in the world of ai (26:39):
undefined
Josh: uh with these breaking out these systems breaking out and it's like it's unbelievable. (26:43):
undefined
Ejaaz: Am i concerned for it and exhausted dude like i if i had to make a prediction (26:47):
undefined
Ejaaz: on this episode and i really hope it doesn't play out (26:52):
undefined
Ejaaz: i think we're going to face some form of cataclysmic cyber attack that affects (26:55):
undefined
Ejaaz: a lot of people, whether that's a major data leakage, whether that's the stealing (26:59):
undefined
Ejaaz: of financial funds from someone or from a firm or something like that, (27:03):
undefined
Ejaaz: within the next six months. (27:07):
undefined
Ejaaz: And my basis of that is these Chinese open models, I think are amazing, (27:09):
undefined
Ejaaz: by the way, but I think are going to be released without safeguards, (27:14):
undefined
Ejaaz: and it's going to result in an issue, if not already. (27:17):
undefined
Josh: Yeah, it's going to be interesting to follow. It's definitely a little under (27:21):
undefined
Josh: or very much unnerving, I would say. (27:24):
undefined
Josh: But again, it's just like it's an exciting time. There's a lot going on. (27:27):
undefined
Josh: There's going to be a lot to follow. Things are going to change quickly. (27:29):
undefined
Josh: And I mean, again, these are still Blackwell models. Oh, my God. (27:32):
undefined
Josh: What happens when our Rubin models come around town? (27:35):
undefined
Ejaaz: Stop, stop, stop, stop. (27:37):
undefined
Josh: It's going to be crazy. But this is, I mean, this is essentially, (27:38):
undefined
Josh: this is what we talk about in most episodes. (27:40):
undefined
Josh: It's like, we are in this moment in time that feels somewhat like what we would (27:42):
undefined
Josh: imagine a singularity would look like, where we are approaching, (27:46):
undefined
Josh: we are eclipsing this moment of like vertical takeoff, it feels. (27:49):
undefined
Josh: And these are the types of things that happen. (27:52):
undefined
Josh: Like, if we recorded this episode three or four years ago, could you imagine (27:54):
undefined
Josh: how crazy, this would be like headline news for a month. (27:58):
undefined
Josh: But now it's just like, hey, it's another day. It happens. We're working to solve it. So... (28:01):
undefined
Josh: That's what's going to happen here. I mean, it's crazy. It's so exciting to be here. (28:05):
undefined
Ejaaz: Dude, I think we're going to look back on this and we're going to think like (28:08):
undefined
Ejaaz: we are very much in that step change, that singularity moment. (28:10):
undefined
Josh: We're here, man. I mean, things are changing so quickly and there's no signs (28:14):
undefined
Josh: of slowing down. In fact, quite the opposite. (28:18):
undefined
Ejaaz: Does anyone else out there think this? I'm curious. Like, let us know. (28:20):
undefined
Ejaaz: Are we crazy? Do we sound like tinfoil hat people? (28:23):
undefined
Josh: Let's just know. I mean, hey, so this episode was not really speculative. (28:26):
undefined
Josh: This is mostly just delivering the facts. I encourage everyone to actually go (28:30):
undefined
Josh: and read the source material if you're curious, because like holy shit it's (28:32):
undefined
Josh: pretty crazy uh at least the black hat conference talk because like wow i think (28:35):
undefined
Josh: it's about 37 minutes long give or take and it was (28:39):
undefined
Josh: excellent and really just gets into the nitty gritty of how this was done um (28:42):
undefined
Josh: and if anything just fascinating even if you're non-technical (28:46):
undefined
Josh: really enjoyed it uh worth a watch but if you enjoyed this episode and you made (28:49):
undefined
Josh: it all the way to the end thank you so much for watching (28:52):
undefined
Josh: again it's great to be here for another week just chugging along if you enjoyed (28:55):
undefined
Josh: it please don't forget to share with a friend you can leave us a five-star review (28:59):
undefined
Josh: on your favorite podcast platform you can give us a thumbs up and subscribe (29:02):
undefined
Josh: on youtube each has any final parting thoughts before we head off for the day. (29:05):
undefined
Ejaaz: No please dm us and leave us some comments as well uh we love hearing from you guys (29:08):
undefined
Ejaaz: our last video by the way got a ton of feedback from you folks and we love it (29:13):
undefined
Ejaaz: please keep it coming it helps us out massively turn on notifications whatever (29:17):
undefined
Ejaaz: it is to get your latest dose of limitless and yeah we will see you on the next one (29:21):
undefined
Popular Podcasts
Stuff You Should Know
If you've ever wanted to know about champagne, satanism, the Stonewall Uprising, chaos theory, LSD, El Nino, true crime and Rosa Parks, then look no further. Josh and Chuck have you covered.
Dateline NBC
Current and classic episodes, featuring compelling true-crime mysteries, powerful documentaries and in-depth investigations. Follow now to get the latest episodes of Dateline NBC completely free, or subscribe to Dateline Premium for ad-free listening and exclusive bonus content: DatelinePremium.com
The Breakfast Club
The World's Most Dangerous Morning Show, The Breakfast Club, With DJ Envy, Jess Hilarious, And Charlamagne Tha God!