Dwarkesh Patel: The Scaling Era of AI is Here

Episode Transcript
Available transcripts are automatically generated. Complete accuracy is not guaranteed.
Ryan Sean Adams: Dwarkesh Patel, we are big fans. It's an honor to have you. (00:03):
undefined

Dwarkesh: Thank you so much for having me on. (00:07):
undefined

Ryan Sean Adams: Okay, so you have a book out. It's called The Scaling Era, an oral history of AI from 2019 to 2025. (00:08):
undefined

Ryan Sean Adams: These are some key dates here. This is really a story of how AI emerged. (00:16):
undefined

Ryan Sean Adams: And it seemed to have exploded on people's radar over the past five years. (00:21):
undefined

Ryan Sean Adams: And And everyone in the world, it feels like, is trying to figure out what just (00:26):
undefined

Ryan Sean Adams: happened and what is about to happen. (00:29):
undefined

Ryan Sean Adams: And I feel like for this story, we should start at the beginning, as your book does. (00:32):
undefined

Ryan Sean Adams: What is the scaling era of AI and when abouts did it start? What were the key milestones? (00:36):
undefined

Dwarkesh: So I think the undertold story about everybody's, of course, (00:42):
undefined

Dwarkesh: been hearing more and more about AI. (00:47):
undefined

Dwarkesh: The under-told story is that the big contributor to these AI models getting (00:48):
undefined

Dwarkesh: better over time has been the fact that we are throwing exponentially more compute (00:53):
undefined

Dwarkesh: into trading frontier systems every year. (00:58):
undefined

Dwarkesh: So by some estimates, we spend 4x every single year over the last decade trading (01:00):
undefined

Dwarkesh: the frontier system than the one before it. (01:04):
undefined

Dwarkesh: And that just means that we're spending hundreds of thousands of times more (01:06):
undefined

Dwarkesh: compute than the systems of the early 2010s. (01:10):
undefined

Dwarkesh: Of course, we've also had algorithmic breakthroughs in the meantime. (01:15):
undefined

Dwarkesh: 2018, we had the Transformer. (01:17):
undefined

Dwarkesh: Since then, obviously, many companies have made small improvements here and there. (01:19):
undefined

Dwarkesh: But the overwhelming fact that we're spending already hundreds of billions of (01:23):
undefined

Dwarkesh: dollars in building up the infrastructure, (01:28):
undefined

Dwarkesh: the data centers, the chips for these models, and this picture is only going (01:31):
undefined

Dwarkesh: to intensify if this exponential keeps going, (01:36):
undefined

Dwarkesh: 4x a year, over the next two years, is something that is on the minds of the (01:38):
undefined

Dwarkesh: CFOs of the big hyperscalers and the people planning the expenditures and training going forward, (01:45):
undefined

Dwarkesh: but is not as common in the conversation around where AI is headed. (01:49):
undefined

Ryan Sean Adams: So what do you feel like people should know about this? (01:54):
undefined

Ryan Sean Adams: Like what is the scaling era? There have been other eras maybe of AI or compute, (01:57):
undefined

Ryan Sean Adams: but what's special about the scaling era? (02:02):
undefined

Dwarkesh: People started noticing. Well, first of all, in 2012, there's this, (02:04):
undefined

Dwarkesh: Ilya Seskaver and others started using neural networks in order to categorize images. (02:09):
undefined

Dwarkesh: And just noticing that instead of doing something hand-coded, (02:16):
undefined

Dwarkesh: you can get a lot of juice out of just neural networks, black boxes. (02:19):
undefined

Dwarkesh: You just train them to identify what thing is like what. (02:24):
undefined

Dwarkesh: And then people started playing around these neural networks more, (02:28):
undefined

Dwarkesh: using them for different kinds of applications. (02:31):
undefined

Dwarkesh: And then the question became, we're noticing that these models get better if (02:33):
undefined

Dwarkesh: you throw more data at them and you throw more compute at them. (02:39):
undefined

Dwarkesh: How can we shove as much compute into these models as possible? (02:41):
undefined

Dwarkesh: And the solution ended up being obviously internet text. So you need an architecture (02:47):
undefined

Dwarkesh: which is amenable to the trillions of tokens that have been written over the (02:51):
undefined

Dwarkesh: last few decades and put up on the internet. (02:55):
undefined

Dwarkesh: And we had this happy coincidence of the kinds of architectures that are amenable (02:57):
undefined

Dwarkesh: to this kind of training with the GPUs that were originally made for gaming. (03:01):
undefined

Dwarkesh: We've had decades of internet text being compiled and Ilias actually called it the fossil fuel of AI. (03:05):
undefined

Dwarkesh: It's like this reservoir that we can call upon to train these minds, (03:12):
undefined

Dwarkesh: which are like, you know, they're fitting the mold of human thought because (03:17):
undefined

Dwarkesh: they're trading on trillions of tokens of human thought. (03:20):
undefined

Dwarkesh: And so then it's just been a question of making these models bigger, (03:23):
undefined

Dwarkesh: of using this data that we're getting from internet techs to further keep training them. (03:27):
undefined

Dwarkesh: And over the last year, as you know, the last six months, the new paradigm has (03:33):
undefined

Dwarkesh: been not only are we going to pre-train on all this internet text, (03:38):
undefined

Dwarkesh: we're going to see if we can have them solve math puzzles, (03:41):
undefined

Dwarkesh: coding puzzles, and through this, give them reasoning capabilities. (03:45):
undefined

Dwarkesh: The kind of thing, by the way, I mean, I have some skepticism around AGI just (03:50):
undefined

Dwarkesh: around the corner, which we'll get into. (03:55):
undefined

Dwarkesh: But just the fact that we now have machines which can like reason, (03:56):
undefined

Dwarkesh: like, you know, you can like ask a question to a machine and it'll go away for a long time. (04:00):
undefined

Dwarkesh: It'll like think about it and then like it'll come back to you with a smart answer. (04:04):
undefined

Dwarkesh: And we just sort of take it for granted. But obviously, we also know that they're (04:06):
undefined

Dwarkesh: extremely good at coding, especially. (04:10):
undefined

Dwarkesh: I don't know if you actually got a chance to play around with Cloud Code or (04:12):
undefined

Dwarkesh: Cursor or something. But it's a wild experience to design, explain at a high (04:15):
undefined

Dwarkesh: level, I want an application to does X. (04:20):
undefined

Dwarkesh: 15 minutes later, there's like 10 files of code and the application is built. (04:22):
undefined

Josh Kale: That's where we stand. (04:28):
undefined

Dwarkesh: I have takes on how much this can continue. The other important dynamic, (04:29):
undefined

Dwarkesh: I'll add my monologue here, but the other important dynamic is that if we're (04:33):
undefined

Dwarkesh: going to be living in the scaling era, you can't continue exponentials forever, (04:36):
undefined

Dwarkesh: and certainly not exponentials that are 4x a year forever. (04:41):
undefined

Dwarkesh: And so right now, we're approaching a point where within by 2028, (04:44):
undefined

Dwarkesh: at most by 2030, we will literally run out of the energy we need to keep trading (04:50):
undefined

Dwarkesh: these frontier systems, (04:57):
undefined

Dwarkesh: the capacity at the leading edge nodes, which manufacture the chips that go (04:58):
undefined

Dwarkesh: into the dyes, which go into these GPUs, even the raw fraction of GDP that will (05:02):
undefined

Dwarkesh: have to use to train frontier systems. (05:07):
undefined

Dwarkesh: So we have a couple more years left of the scaling era. And the big question (05:09):
undefined

Dwarkesh: is, will we get to AGI before then? (05:12):
undefined

Ryan Sean Adams: I mean, that's kind of a key insight of your book that like, (05:15):
undefined

Ryan Sean Adams: we're in the middle of the scaling era. (05:18):
undefined

Ryan Sean Adams: I guess we're like, you know, six years in or so. And we're not quite sure. (05:19):
undefined

Ryan Sean Adams: It's like, like the protagonist in the middle of the story, We don't know exactly (05:23):
undefined

Ryan Sean Adams: which way things are going to go. (05:26):
undefined

Ryan Sean Adams: But I want you to maybe, Dworkesh, help folks get an intuition for why scaling in this way even works. (05:28):
undefined

Ryan Sean Adams: Because I'll tell you, for me and for most people, our experience with these (05:36):
undefined

Ryan Sean Adams: revolutionary AI models probably started in 2022 with ChatGPT3 and then ChatGPT4 (05:41):
undefined

Ryan Sean Adams: and seeing all the progress, all these AI models. (05:46):
undefined

Ryan Sean Adams: And it just seems really unintuitive that if you take a certain amount of compute (05:49):
undefined

Ryan Sean Adams: and you take a certain amount of data, out pops AI, out pops intelligence. (05:55):
undefined

Ryan Sean Adams: Could you help us get an intuition for this magic? (06:01):
undefined

Ryan Sean Adams: How does the scaling law even work? Compute plus data equals intelligence? Is that really all it is? (06:05):
undefined

Dwarkesh: To be honest, I've asked so many AI researchers this exact question on my podcast. (06:12):
undefined

Dwarkesh: And I could tell you some potential theories of why it might work. (06:17):
undefined

Dwarkesh: I don't think we understand. (06:20):
undefined

Dwarkesh: You know what? I'll just say that. I don't think we understand. (06:24):
undefined

Ryan Sean Adams: We don't understand how this works. We know it works, but we don't understand (06:27):
undefined

Dwarkesh: How it works. We have evidence from actually, of all things, (06:30):
undefined

Dwarkesh: primatology of what could be going on here, or at least like why similar patterns (06:36):
undefined

Dwarkesh: in other parts of the world. (06:41):
undefined

Dwarkesh: So what I found really interesting, There's this research by this researcher, (06:42):
undefined

Dwarkesh: Susanna Herculana Huzel, (06:46):
undefined

Dwarkesh: which shows that if you look at how the number of neurons in the brain of a rat, (06:48):
undefined

Dwarkesh: different kinds of rat species increases, as the weight of their brains increase (06:56):
undefined

Dwarkesh: from species to species, there's this very sublinear pattern. (07:01):
undefined

Dwarkesh: So if their brain size doubles, the neuron count will not double between different rat species. (07:04):
undefined

Dwarkesh: And there's other animals where there's other kinds of... (07:09):
undefined

Dwarkesh: Families of species for which this is true. The two interesting exceptions to (07:14):
undefined

Dwarkesh: this rule, where there is actually a linear increase in neuron count and brain (07:18):
undefined

Dwarkesh: size, is one, certain kinds of birds. (07:22):
undefined

Dwarkesh: So, you know, birds are actually very smart, given the size of their brains, and primates. (07:26):
undefined

Dwarkesh: So the theory for what happened with humans is that we unlocked an architecture that was very scalable. (07:32):
undefined

Dwarkesh: So the way people talk about transformers being more scalable and then LSTMs, (07:38):
undefined

Dwarkesh: the thing that preceded them in 2018. (07:41):
undefined

Dwarkesh: We unlocked this architecture as it's very scalable. (07:43):
undefined

Dwarkesh: And then we were in an evolutionary niche millions of years ago, (07:46):
undefined

Dwarkesh: which rewarded marginal increases in intelligence. (07:50):
undefined

Dwarkesh: If you get slightly smarter, yes, the brain costs more energy, (07:53):
undefined

Dwarkesh: but you can save energy in terms of like not having to, you can cook, (07:56):
undefined

Dwarkesh: you can cook food so you don't have to spend much more on digestion. (07:59):
undefined

Dwarkesh: You can find a game, you can find different ways of foraging. (08:02):
undefined

Dwarkesh: Birds were not able to find this evolutionary niche, which rewarded the incremental (08:06):
undefined

Dwarkesh: increases in intelligence because if your brain gets too heavy as a bird, you're not going to fly. (08:11):
undefined

Dwarkesh: So it was this happy coincidence of these two things. Now, why is it the case (08:17):
undefined

Dwarkesh: that the fact that our brains could get bigger resulted in us becoming as smart (08:21):
undefined

Dwarkesh: as we are? We still don't know. (08:27):
undefined

Dwarkesh: And there's many different dissimilarities between AIs and humans. (08:29):
undefined

Dwarkesh: While our brains are quite big, we don't need to be trained. (08:32):
undefined

Dwarkesh: You know, a human from the age they're zero to 18 is not seeing within an order (08:35):
undefined

Dwarkesh: of magnitude of the amount of information these LLMs are trained on. (08:41):
undefined

Dwarkesh: So LLMs are extremely data inefficient. (08:44):
undefined

Dwarkesh: They need a lot more data, but the pattern of scaling, I think we see in many different places. (08:46):
undefined

Ryan Sean Adams: So is that a fair kind of analog? This analog has always made sense to me. (08:53):
undefined

Ryan Sean Adams: It's just like transformers are like neurons. (08:57):
undefined

Ryan Sean Adams: You know, AI models are sort of like the human brain. (09:00):
undefined

Ryan Sean Adams: Evolutionary pressures are like gradient descent, reward algorithms and out (09:04):
undefined

Ryan Sean Adams: pops human intelligence. We don't really understand that. (09:09):
undefined

Ryan Sean Adams: We also don't understand AI intelligence, but it's basically the same principle at work. (09:13):
undefined

Dwarkesh: I think it's a super fascinating, but also very thorny question because is gradient (09:17):
undefined

Dwarkesh: intelligence like evolution? (09:23):
undefined

Dwarkesh: Well, yes, in one sense. But also when we do gradient descent on these models, (09:24):
undefined

Dwarkesh: we start off with the weights and then we're, you know, it's like learning how (09:29):
undefined

Dwarkesh: does chemistry work, how does coding work, how does math work. (09:35):
undefined

Dwarkesh: And that's actually more similar to lifetime learning, which is to say that, (09:38):
undefined

Dwarkesh: like, by the time you're already born to the time you turn 18 or 25, (09:42):
undefined

Dwarkesh: the things you learn, and that's not evolution. (09:46):
undefined

Dwarkesh: Evolution designed the system or the brain by which you can do that learning, (09:48):
undefined

Dwarkesh: but the lifetime learning itself is not evolution. And so there's also this (09:53):
undefined

Dwarkesh: interesting question of, yeah, is training more like evolution? (09:57):
undefined

Dwarkesh: In which case, actually, we might be very far from AGI because the amount of (10:01):
undefined

Dwarkesh: compute that's been spent over the course of evolution to discover the human (10:04):
undefined

Dwarkesh: brain, you know, could be like 10 to the 40 flops. There's been estimates, you know, whatever. (10:07):
undefined

Dwarkesh: I'm sure it will bore you to discover, talk about how these estimates are derived, (10:12):
undefined

Dwarkesh: but just like how much versus is it like a single lifetime, (10:15):
undefined

Dwarkesh: like going from the age of zero to the age of 18, which is closer to, (10:20):
undefined

Dwarkesh: I think, 10 to the 24 flops, which is actually less than compute than we use (10:23):
undefined

Dwarkesh: to train frontier systems. (10:26):
undefined

Dwarkesh: All right, anyways, we'll get back to more relevant questions. (10:28):
undefined

Ryan Sean Adams: Well, here's kind of a big picture question as well. (10:33):
undefined

Ryan Sean Adams: It's like I'm constantly fascinated with the metaphysical types of discussions (10:36):
undefined

Ryan Sean Adams: that some AI researchers kind of take. (10:41):
undefined

Ryan Sean Adams: Like a lot of AI researchers will talk in terms of when they describe what they're (10:43):
undefined

Ryan Sean Adams: making, we're making God. (10:48):
undefined

Ryan Sean Adams: Like why do they say things like that? What is this talk of like making God? (10:49):
undefined

Ryan Sean Adams: What does that mean? Is it just the idea that scaling laws don't cease? (10:53):
undefined

Ryan Sean Adams: And if we can, you know, scale intelligence to AGI, then there's no reason we (10:57):
undefined

Ryan Sean Adams: can't scale far beyond that and create some sort of a godlike entity. (11:03):
undefined

Ryan Sean Adams: And essentially, that's what the quest is. We're making artificial superintelligence. (11:07):
undefined

Ryan Sean Adams: We're making a god. We're making god. (11:12):
undefined

Dwarkesh: I think people focus too much on when they, I think this God discussion focuses (11:13):
undefined

Dwarkesh: too much on the hypothetical intelligence of a single copy of an AI. (11:19):
undefined

Dwarkesh: I do believe in the notion of a super intelligence, which is not just functionally, (11:27):
undefined

Dwarkesh: which is not just like, oh, it knows a lot of things, but is actually qualitatively (11:32):
undefined

Dwarkesh: different than human society. (11:36):
undefined

Dwarkesh: But the reason is not because I think it's so powerful that any one individual (11:38):
undefined

Dwarkesh: copy of AI will be as smart, but because of the collective advantages that AIs (11:42):
undefined

Dwarkesh: will have, which have nothing to do with their raw intelligence, (11:48):
undefined

Dwarkesh: but rather the fact that these models will be digital or they already are digital, (11:51):
undefined

Dwarkesh: but eventually they'll be as smart as humans at least. (11:55):
undefined

Dwarkesh: But unlike humans, because of our biological constraints, these models can be copied. (11:58):
undefined

Dwarkesh: If there's a model that has learned a lot about a specific domain, (12:03):
undefined

Dwarkesh: you can make infinite copies of it. (12:06):
undefined

Dwarkesh: And now you have an infinite copies of Jeff Dean or Ilya Satskova or Elon Musk (12:08):
undefined

Dwarkesh: or any skilled person you can think of. (12:12):
undefined

Dwarkesh: They can be merged. So the knowledge that each copy is learning can be amalgamated (12:15):
undefined

Dwarkesh: back into the model and then back to all the copies. (12:21):
undefined

Dwarkesh: They can be distilled. They can run at superhuman speeds. (12:24):
undefined

Dwarkesh: These collective advantages, also they can communicate in latent space. (12:29):
undefined

Dwarkesh: These collective advantages. (12:32):
undefined

Ryan Sean Adams: They're immortal. I mean, you know, as an example. (12:33):
undefined

Dwarkesh: Yes, exactly. No, I mean, that's actually, tell me if I'm rabbit holing too (12:36):
undefined

Dwarkesh: much, but like one really interesting question will come about is how do we prosecute AIs? (12:40):
undefined

Dwarkesh: Because the way we prosecute humans is that we will throw you in jail if you commit a crime. (12:45):
undefined

Dwarkesh: But if there's trillions of copies or thousands of copies of an AI model, (12:50):
undefined

Dwarkesh: if a copy of an AI model, if an instance of an AI model does something bad, what do you do? (12:57):
undefined

Dwarkesh: Does the whole model have to get, and how do you even punish a model, (13:01):
undefined

Dwarkesh: right? Like, does it care about its weights being squandered? (13:04):
undefined

Dwarkesh: Yeah, there's all kinds of questions that arise because of the nature of what AIs are. (13:09):
undefined

Dwarkesh Patel: And also who is liable for that, right? (13:14):
undefined

Dwarkesh: Like, is it the toolmaker? (13:16):
undefined

Dwarkesh Patel: Is it the person using the tool? Who is responsible for these things? (13:17):
undefined

Dwarkesh Patel: There's one topic that I do want to come to here about scaling laws, (13:20):
undefined

Dwarkesh Patel: At what time did we realize that scaling laws were going to work? (13:23):
undefined

Dwarkesh Patel: Because there were a lot of theses early in the days, early 2000s about AI, (13:27):
undefined

Dwarkesh Patel: how we were going to build better models. (13:31):
undefined

Dwarkesh Patel: Eventually, we got to the transformer. But at what point did researchers and (13:33):
undefined

Dwarkesh Patel: engineers start to realize that, hey, this is the correct idea. (13:36):
undefined

Dwarkesh Patel: We should start throwing lots of money and resources towards this versus other (13:39):
undefined

Dwarkesh Patel: ideas that were just kind of theoretical research ideas, but never really took off. (13:42):
undefined

Dwarkesh Patel: We kind of saw this with GPT two to three, where there's this huge improvement. (13:45):
undefined

Dwarkesh: A lot of. (13:49):
undefined

Dwarkesh Patel: Resources went into it. Was there a specific moment in time or a specific breakthrough (13:49):
undefined

Dwarkesh Patel: that led to the start of these scaling laws? (13:53):
undefined

Dwarkesh: I think it's been a slow process of more and more people appreciating this nature (13:55):
undefined

Dwarkesh: of the overwhelming role of compute in driving forward progress. (14:00):
undefined

Dwarkesh: In 2018, I believe, Dario Amadei wrote a memo that was secret while he was at (14:05):
undefined

Dwarkesh: OpenAI. Now he's the CEO of Anthropic. (14:13):
undefined

Dwarkesh: But while he's at OpenAI, he's subsequently revealed on my podcast that he wrote (14:16):
undefined

Dwarkesh: this memo where the title of the memo was called Big Blob of Compute. (14:19):
undefined

Dwarkesh: And it says basically what you expect it to say, which is that like, (14:25):
undefined

Dwarkesh: yes, there's ways you can mess up the process of training. You have the wrong (14:29):
undefined

Dwarkesh: kinds of data or initializations. (14:31):
undefined

Dwarkesh: But fundamentally, AGI is just a big blob of compute. (14:33):
undefined

Dwarkesh: And then we've gotten over the subsequent years, there was more empirical evidence. (14:37):
undefined

Dwarkesh: So a big update, I think it was 2021. (14:41):
undefined

Dwarkesh: Correct me. Somebody definitely will correct me in the comments. (14:44):
undefined

Dwarkesh: I'm wrong. There were these, (14:46):
undefined

Dwarkesh: there's been multiple papers of these scaling laws where you can show that the (14:48):
undefined

Dwarkesh: loss of the model on the objective of predicting the next token goes down very predictably, (14:54):
undefined

Dwarkesh: almost to like multiple decimal places of correctness based on how much more (15:01):
undefined

Dwarkesh: compute you throw in these models. (15:07):
undefined

Dwarkesh: And the compute itself is a function of the amount of data you use and how big (15:08):
undefined

Dwarkesh: the model is, how many parameters it has. (15:13):
undefined

Dwarkesh: And so that was an incredibly strong evidence back in the day, (15:15):
undefined

Dwarkesh: a couple of years ago, because then you could say, well, OK, (15:19):
undefined

Dwarkesh: if it really has this incredibly low loss of predicting the next token in all (15:22):
undefined

Dwarkesh: human output, including scientific papers, including GitHub repositories. (15:28):
undefined

Dwarkesh: Then doesn't it mean it has actually had to learn coding and science and all (15:34):
undefined

Dwarkesh: these skills in order to make those predictions, which actually ended up being true. (15:40):
undefined

Dwarkesh: And it was it was something people, you know, we take it for granted now, (15:43):
undefined

Dwarkesh: but it actually even as of a year or two ago, people were really even denying that premise. (15:46):
undefined

Dwarkesh: But some people a couple of years ago just like thought about it and like, (15:50):
undefined

Dwarkesh: yeah, actually, that would mean that it's learned the skills. (15:53):
undefined

Dwarkesh: And that's crazy that we just have this strong empirical pattern that tells (15:55):
undefined

Dwarkesh: us exactly what we need to do in order to learn these skills. (15:59):
undefined

Dwarkesh Patel: And it creates this weird perception, right, where like very early on and so (16:02):
undefined

Dwarkesh Patel: to this day, it really is just a token predictor, right? (16:05):
undefined

Dwarkesh Patel: Like we're just predicting the next word in the sentence. But somewhere along (16:07):
undefined

Dwarkesh Patel: the lines, it actually creates this perception of intelligence. (16:10):
undefined

Dwarkesh Patel: So I guess we covered the early historical context. I kind of want to bring (16:14):
undefined

Dwarkesh Patel: the listeners up to today, where we are currently, where the scaling laws have (16:18):
undefined

Dwarkesh Patel: brought us in the year 2025. (16:21):
undefined

Dwarkesh Patel: So can you kind of outline where we've gotten to from early days of GPTs to (16:23):
undefined

Dwarkesh Patel: now we have GPT-4, we have Gemini Ultra, we have Club, which you mentioned earlier. (16:28):
undefined

Dwarkesh Patel: We had the breakthrough of reasoning. (16:32):
undefined

Dwarkesh Patel: So what can leading frontier models do today? (16:33):
undefined

Dwarkesh: So there's what they can do. And then there's the question of what methods seem to be working. (16:36):
undefined

Dwarkesh: I guess we can start at what they seem to be able to do. They've shown to be (16:41):
undefined

Dwarkesh: remarkably useful at coding and not just at answering direct questions about (16:46):
undefined

Dwarkesh: how does this line of code work or something. (16:51):
undefined

Dwarkesh: But genuinely just autonomously working for 30 minutes or an hour, (16:53):
undefined

Dwarkesh: doing the task, it would take a front-end developer a whole day to do. (16:57):
undefined

Dwarkesh: And you can just ask them at a high level, do this kind of thing, (17:01):
undefined

Dwarkesh: and they can go ahead and do it. (17:04):
undefined

Dwarkesh: Obviously, if you've played around with it, you know that they're extremely (17:05):
undefined

Dwarkesh: useful assistants in terms of research, in terms of even therapists, (17:07):
undefined

Dwarkesh: whatever other use cases. (17:12):
undefined

Dwarkesh: On the question of what training methods seem to be working, (17:13):
undefined

Dwarkesh: we do seem to be getting evidence that pre-training is plateauing, (17:16):
undefined

Dwarkesh: which is to say that we had GPT 4.5, which was just following this old mold (17:19):
undefined

Dwarkesh: of make the model bigger, (17:25):
undefined

Dwarkesh: but it's fundamentally doing the same thing of next token prediction. (17:27):
undefined

Dwarkesh: And apparently it didn't pass muster. The OpenAI had to deprecate it because (17:31):
undefined

Dwarkesh: there's this dynamic where the bigger the model is, the more it costs not only (17:36):
undefined

Dwarkesh: to train, but also to serve, right? (17:39):
undefined

Dwarkesh: Because every time you serve a user, you're having to run the whole model, (17:41):
undefined

Dwarkesh: which is going, so, but that doesn't be working is RL, which is this process (17:44):
undefined

Dwarkesh: of, not just training them on existing tokens on the internet, (17:49):
undefined

Dwarkesh: but having the model itself try to answer math and coding problems. (17:52):
undefined

Dwarkesh: And finally, we got to the point where the model is smart enough to get it right (17:55):
undefined

Dwarkesh: some of the time, and so you can give it some reward, and then it can saturate (17:57):
undefined

Dwarkesh: these tough reasoning problems. (18:01):
undefined

Dwarkesh Patel: And then what was the breakthrough with reasoning for the people who aren't familiar? (18:04):
undefined

Dwarkesh Patel: What made reasoning so special that we hadn't discovered before? (18:08):
undefined

Dwarkesh Patel: And what did that kind of unlock for models that we use today? (18:11):
undefined

Dwarkesh: I'm honestly not sure. I mean, GBD-4 came out a little over two years ago, (18:14):
undefined

Dwarkesh: and then it was after two years after GPT-4 came out that O-1 came out which (18:19):
undefined

Dwarkesh: was the original reasoning breakthrough I think last November and, (18:23):
undefined

Dwarkesh: And subsequently, a couple of months later, DeepSeq showed in their R1 paper. (18:28):
undefined

Dwarkesh: So DeepSeq open source their research and they explained exactly how their algorithm worked. (18:33):
undefined

Dwarkesh: And it wasn't that complicated. It was just like what you would expect, (18:37):
undefined

Dwarkesh: which is get some math problems, (18:41):
undefined

Dwarkesh: give for some initial problems, tell the model exactly what the reasoning trace (18:44):
undefined

Dwarkesh: looks like, how you solve it, just like write it out and then have the model (18:48):
undefined

Dwarkesh: like try to do it raw on the remaining problems. (18:51):
undefined

Dwarkesh: Now, I know it sounds incredibly arrogant to say, well, it wasn't that complicated. (18:54):
undefined

Dwarkesh: Why did it take you years? (18:57):
undefined

Dwarkesh: I think there's an interesting insight there of even things which you think (18:58):
undefined

Dwarkesh: will be simple in terms of high level description of how to solve the problem (19:02):
undefined

Dwarkesh: end up taking longer in terms of haggling out the remaining engineering hurdles (19:05):
undefined

Dwarkesh: than you might naively assume. (19:10):
undefined

Dwarkesh: And that should update us on how long it will take us to go through the remaining (19:12):
undefined

Dwarkesh: bottlenecks on the path to AGI. (19:17):
undefined

Dwarkesh: Maybe that will be tougher than people imagine, especially the people who think (19:19):
undefined

Dwarkesh: we're only two to three years away. (19:22):
undefined

Dwarkesh: But all this to say, yeah, I'm not sure why it took so long after GPT-4 to get (19:24):
undefined

Dwarkesh: a model trained on a similar level of capabilities that could then do reasoning. (19:27):
undefined

Dwarkesh Patel: And in terms of those abilities, the first answer you had to what can it do was coding. (19:31):
undefined

Dwarkesh Patel: And I hear that a lot of the time when I talk to a lot of people is that coding (19:36):
undefined

Dwarkesh Patel: seems to be a really strong suit and a really huge unlock to using these models. (19:40):
undefined

Dwarkesh Patel: And I'm curious, why coding over general intelligence? (19:44):
undefined

Dwarkesh Patel: Is it because it's placed in a more confined box of parameters? (19:48):
undefined

Dwarkesh Patel: I know in the early days, we had the AlphaGo and And we had the AIs playing (19:51):
undefined

Dwarkesh Patel: chess and they exceed, they perform so well because they were kind of contained (19:54):
undefined

Dwarkesh Patel: within this box of parameters that was a little less open-ended than general intelligence. (19:58):
undefined

Dwarkesh Patel: Is that the reason why coding is kind of at the frontier right now of the ability of these models? (20:01):
undefined

Dwarkesh: There's two different hypotheses. One is based around this idea called Moravac's paradox. (20:06):
undefined

Dwarkesh: And this was an idea, by the way, one super interesting figure, (20:13):
undefined

Dwarkesh: actually, I should have mentioned him earlier. (20:16):
undefined

Dwarkesh: One super interesting figure in the history of scaling is Hans Moravac, (20:18):
undefined

Dwarkesh: who I think in the 90s predicts that 2028 will be the year that we will get to AGI. (20:22):
undefined

Dwarkesh: And the way he predicts this, which is like, you know, we'll see what happens, (20:29):
undefined

Dwarkesh: but like not that far off the money as far as I'm concerned. (20:32):
undefined

Dwarkesh: The way he predicts this is he just looks at the growth in computing power year (20:35):
undefined

Dwarkesh: over year and then looks at how much compute he estimated the human brain to be to require. (20:41):
undefined

Dwarkesh: And just like, OK, we'll have computers as powerful as the human brain by 2028. (20:47):
undefined

Dwarkesh: Which is like at once a deceptively simple argument, but also ended up being (20:51):
undefined

Dwarkesh: incredibly accurate and like worked, right? (20:57):
undefined

Dwarkesh: I might add a fact drive it was 2028, but it was within that, (21:01):
undefined

Dwarkesh: like within something you would consider a reasonable guess, given what we know now. (21:03):
undefined

Dwarkesh: Sorry, anyway, so the Morrowind's paradox is this idea that computers seemed (21:07):
undefined

Dwarkesh: in AI get better first at the skills which humans are the worst at. (21:12):
undefined

Dwarkesh: Or at least there's a huge variation in the human repertoire. (21:19):
undefined

Dwarkesh: So we think of coding as incredibly hard, right? We think this is like the top (21:22):
undefined

Dwarkesh: 1% of people will be excellent coders. (21:26):
undefined

Dwarkesh: We also think of reasoning as very hard, right? So if you like read Aristotle, (21:29):
undefined

Dwarkesh: he says, the thing which makes humans special, which distinguishes us from animals is reasoning. (21:32):
undefined

Dwarkesh: And these models aren't that useful yet at almost anything. The one thing they can do is reasoning. (21:38):
undefined

Dwarkesh: So how do we explain this pattern? And Moravec's answer is that evolution has (21:45):
undefined

Dwarkesh: spent billions of years optimizing us to do things we take for granted. (21:51):
undefined

Dwarkesh: Move around this room, right? I can pick up this can of Coke, (21:56):
undefined

Dwarkesh: move it around, drink from it. (21:58):
undefined

Dwarkesh: And that we can't even get robots to do at all yet. (22:00):
undefined

Dwarkesh: And in fact, it's so ingrained in us by evolution that there's no human, or. (22:04):
undefined

Ryan Sean Adams: At least humans who don't have (22:08):
undefined

Dwarkesh: Disabilities will all be able to do this. And so we just take it for granted (22:10):
undefined

Dwarkesh: that this is an easy thing to do. (22:13):
undefined

Dwarkesh: But in fact, it's evidence of how long evolution has spent getting humans up to this point. (22:15):
undefined

Dwarkesh: Whereas reasoning, logic, all of these skills have only been optimized by evolution (22:19):
undefined

Dwarkesh: over the course of the last few million years. (22:27):
undefined

Dwarkesh: So there's been a thousand fold less evolutionary pressure towards coding than (22:30):
undefined

Dwarkesh: towards just basic locomotion. (22:35):
undefined

Dwarkesh: And this has actually been very accurate in predicting what kinds of progress (22:38):
undefined

Dwarkesh: we see even before we got deep learning, right? (22:42):
undefined

Dwarkesh: Like in the 40s when we got our first computers, the first thing that we could (22:43):
undefined

Dwarkesh: use them to do is long calculations for ballistic trajectories at the time for World War II. (22:47):
undefined

Dwarkesh: Humans suck at long calculations by hand. (22:52):
undefined

Dwarkesh: And anyways, so that's the explanation for coding, which seems hard for humans, (22:56):
undefined

Dwarkesh: is the first thing that went to AIs. (23:00):
undefined

Dwarkesh: Now, there's another theory, which is that this is actually totally wrong. (23:02):
undefined

Dwarkesh: It has nothing to do with the seeming paradox of how long evolution has optimized (23:05):
undefined

Dwarkesh: us for, and everything to do with the availability of data. (23:10):
undefined

Dwarkesh: So we have GitHub, this repository of all of human code, at least all open source (23:15):
undefined

Dwarkesh: code written in all these different languages, trillions and trillions of tokens. (23:23):
undefined

Dwarkesh: We don't have an analogous thing for robotics. We don't have this pre-training (23:26):
undefined

Dwarkesh: corpus. And that explains why code has made so much more progress than robotics. (23:29):
undefined

Ryan Sean Adams: That's fascinating because if there's one thing that I could list that we'd (23:34):
undefined

Ryan Sean Adams: want AI to be good at, probably coding software is number one on that list. (23:38):
undefined

Ryan Sean Adams: Because if you have a Turing complete intelligence that can create Turing complete (23:44):
undefined

Ryan Sean Adams: software, is there anything you can't create once you have that? (23:49):
undefined

Ryan Sean Adams: Also, like the idea of Morvac's paradox, I guess that sort of implies a certain (23:52):
undefined

Ryan Sean Adams: complementarianism with humanity. (23:58):
undefined

Ryan Sean Adams: So if robots can do things that robots can do really well and can't do the things (24:01):
undefined

Ryan Sean Adams: humans can do well, well, perhaps there's a place for us in this world. (24:06):
undefined

Ryan Sean Adams: And that's fantastic news. It also maybe implies that humans have kind of scratched (24:09):
undefined

Ryan Sean Adams: the surface on reasoning potential. (24:14):
undefined

Ryan Sean Adams: I mean, if we've only had a couple of million years of evolution and we haven't (24:17):
undefined

Ryan Sean Adams: had the data set to actually get really good at reasoning, it seems like there'd (24:21):
undefined

Ryan Sean Adams: be a massive amount of upside, unexplored territory, (24:25):
undefined

Ryan Sean Adams: like so much more intelligence that nature could actually (24:29):
undefined

Ryan Sean Adams: contain inside of reasoning. (24:33):
undefined

Ryan Sean Adams: I mean, are these some of the implications of these ideas? (24:35):
undefined

Dwarkesh: Yeah, I know. I mean, that's a great insight. Another really interesting insight (24:38):
undefined

Dwarkesh: is that the more variation there (24:41):
undefined

Dwarkesh: is in a skill in humans, the better and faster that AIs will get at it. (24:44):
undefined

Dwarkesh: Because coding is the kind of thing where 1% of humans are really good at it. (24:50):
undefined

Dwarkesh: The rest of us will, if we try to learn it, we'd be okay at it or something, right? (24:55):
undefined

Dwarkesh: And because evolutionists spend so little time optimizing us, (25:00):
undefined

Dwarkesh: there's this room for variation where the optimization hasn't happened uniformly (25:03):
undefined

Dwarkesh: or it hasn't been valuable enough to saturate the human gene pool for this skill. (25:07):
undefined

Dwarkesh: I think you made an earlier point that I thought was really interesting I wanted (25:14):
undefined

Dwarkesh: to address. Can you remind me of the first thing you said? Is it the complementarianism? Yes. (25:16):
undefined

Dwarkesh: So you can take it as a positive future. You can take it as a negative future (25:23):
undefined

Dwarkesh: in the sense that, well, what is the complementary skills we're providing? (25:27):
undefined

Dwarkesh: We're good meat robots. (25:30):
undefined

Ryan Sean Adams: Yeah, the low skilled labor of the situation. (25:33):
undefined

Dwarkesh: We can do all the thinking and planning. One dark future, (25:35):
undefined

Dwarkesh: one dark vision of the future is we'll get those meta glasses (25:39):
undefined

Dwarkesh: and the AI speaking into our ear and it'll tell us to go put this brick over (25:44):
undefined

Dwarkesh: there so that the next data center couldn't be built because the AI's got the (25:50):
undefined

Dwarkesh: plan for everything. It's got the better design for the ship and everything. (25:53):
undefined

Dwarkesh: You just need to move things around for it. And that's what human labor looks (25:55):
undefined

Dwarkesh: like until robotics is solved. (25:58):
undefined

Dwarkesh: So yeah, it depends on how you... On the other hand, you'll get paid a lot because (26:01):
undefined

Dwarkesh: it's worth a lot to move those bricks. We're building AGI here. (26:04):
undefined

Dwarkesh: But yeah, it depends on how you come out of that question. (26:08):
undefined

Ryan Sean Adams: Well, there seems to be something to that idea, going back to the idea of the (26:09):
undefined

Ryan Sean Adams: massive amount of human variation. (26:12):
undefined

Ryan Sean Adams: I mean, we have just in the past month or so, we have news of meta hiring AI (26:14):
undefined

Ryan Sean Adams: researchers for $100 million signing bonuses, okay? (26:18):
undefined

Ryan Sean Adams: What does the average software engineer make versus what does an AI researcher (26:22):
undefined

Ryan Sean Adams: make at kind of the top of the market, right? (26:27):
undefined

Ryan Sean Adams: Which has got to imply, obviously there's some things going on with demand and (26:29):
undefined

Ryan Sean Adams: supply, but also that it does also seem to imply that there's massive variation (26:33):
undefined

Ryan Sean Adams: in the quality of a software engineer. (26:38):
undefined

Ryan Sean Adams: And if AIs can get to that quality, well, what does that unlock? (26:40):
undefined

Ryan Sean Adams: Yeah. So, okay. Yeah. So I guess we have like coding down right now. (26:44):
undefined

Ryan Sean Adams: Like another question though is like, what can't AIs do today? (26:48):
undefined

Ryan Sean Adams: And how would you characterize that? Like what are the things they just don't do well? (26:53):
undefined

Dwarkesh: So I've been interviewing people on my podcast who have very different timelines (26:57):
undefined

Dwarkesh: from a role to get to AGI. I have had people on who think it's two years away (27:02):
undefined

Dwarkesh: and some who think it's 20 years away. (27:05):
undefined

Dwarkesh: And the experience of building AI tools for myself actually has been the most (27:08):
undefined

Dwarkesh: insight driving or maybe research I've done on the question of when AI is coming. (27:13):
undefined

Ryan Sean Adams: More than the guest interviews. (27:18):
undefined

Dwarkesh: Yeah, because you just, I have had, I've probably spent on the order of a hundred (27:20):
undefined

Dwarkesh: hours trying to build these little tools. The kinds I'm sure you've also tried (27:25):
undefined

Dwarkesh: to build of like, rewrite auto-generated transcripts for me to make them sound, (27:28):
undefined

Dwarkesh: the rewritten the way a human would write them. (27:32):
undefined

Dwarkesh: Find clips for me to tweet out, write essays with me, co-write them passage (27:35):
undefined

Dwarkesh: by passage, these kinds of things. (27:39):
undefined

Dwarkesh: And what I found is that it's actually very hard to get human-like labor out (27:41):
undefined

Dwarkesh: of these models, even for tasks like these, which should be death center in (27:45):
undefined

Dwarkesh: the repertoire of these models, right? (27:49):
undefined

Dwarkesh: They're short horizon, they're language in, language out. (27:50):
undefined

Dwarkesh: They're not contingent on understanding some thing I said like a month ago. (27:53):
undefined

Dwarkesh: This is just like, this is the task. (27:58):
undefined

Dwarkesh: And I was thinking about why is it the case that I still haven't been able to (28:00):
undefined

Dwarkesh: automate these basic language tasks? Why do I still have a human work on these things? (28:04):
undefined

Dwarkesh: And I think the key reason that you can't automate even these simple tasks is (28:09):
undefined

Dwarkesh: because the models currently lack the ability to do on the job training. (28:15):
undefined

Dwarkesh: So if you hire a human for the first six months, for the first three months, (28:21):
undefined

Dwarkesh: they're not going to be that useful, even if they're very smart, (28:24):
undefined

Dwarkesh: because they haven't built up the context, they haven't practiced the skills, (28:26):
undefined

Dwarkesh: they don't understand how the business works. (28:29):
undefined

Dwarkesh: What makes humans valuable is not that mainly the raw intellect obviously matters, (28:31):
undefined

Dwarkesh: but it's not mainly that. (28:35):
undefined

Dwarkesh: It's their ability to interrogate their own failures in this really dynamic, (28:36):
undefined

Dwarkesh: organic way to pick up small efficiencies and improvements as they practice (28:40):
undefined

Dwarkesh: the task and to build up this context as they work within a domain. (28:45):
undefined

Dwarkesh: And so sometimes people wonder, look, if you look at the revenue of OpenAI, (28:50):
undefined

Dwarkesh: the annual recurring revenue, it's on the order of $10 billion. (28:54):
undefined

Dwarkesh: Kohl's makes more money than that. McDonald's makes more money than that, right? (28:57):
undefined

Dwarkesh: So why is it that if they've got AGI, they're, you know, like Fortune 500 isn't (29:01):
undefined

Dwarkesh: reorganizing their workflows to, you know, use open AI models at every layer of the stack? (29:07):
undefined

Dwarkesh: My answer, sometimes people say, well, it's because people are too stodgy. (29:12):
undefined

Dwarkesh: The management of these companies is like not moving fast enough on AI. (29:15):
undefined

Dwarkesh: That could be part of it. I think mostly it's not that. (29:18):
undefined

Dwarkesh: I think mostly it genuinely is very hard to get human-like labor out of these (29:20):
undefined

Dwarkesh: models because you can't. (29:23):
undefined

Dwarkesh: So you're stuck with the capabilities you get out of the model out of the box. (29:26):
undefined

Dwarkesh: So they might be five out of 10 at rewriting the transcript for you. (29:30):
undefined

Dwarkesh: But if you don't like how it turned out, if you have feedback for it, (29:33):
undefined

Dwarkesh: if you want to keep teaching it over time, once the session ends, (29:36):
undefined

Dwarkesh: the model, like everything it knows about you has gone away. (29:41):
undefined

Dwarkesh: You got to restart again. It's like working with an amnesiac employee. (29:44):
undefined

Dwarkesh: You got to restart again. (29:47):
undefined

Ryan Sean Adams: Every day is the first day of employment, basically. (29:49):
undefined

Dwarkesh: Yeah, exactly. It's a groundhog day for them every day or every couple of hours, in fact. (29:52):
undefined

Dwarkesh: And that makes it very hard for them to be that useful as an employee, (29:56):
undefined

Dwarkesh: right? They're not really an employee at that point. (29:59):
undefined

Dwarkesh: This, I think, not only is a key bottleneck to the value of these models, (30:02):
undefined

Dwarkesh: because human labor is worth a lot, right? (30:06):
undefined

Dwarkesh: Like $60 trillion in the world is paid to wages every year. (30:09):
undefined

Dwarkesh: If these model companies are making on the order of $10 billion a year, that's a big way to AGI. (30:12):
undefined

Dwarkesh: And what explains that gap? What are the bottlenecks? I think a big one is this (30:18):
undefined

Dwarkesh: continual learning thing. (30:21):
undefined

Dwarkesh: And I don't see an easy way that that just gets solved within these models. (30:23):
undefined

Dwarkesh: There's no like, with reasoning, you could say, oh, it's like train it on math (30:26):
undefined

Dwarkesh: and code problems, and then I'll get the reasoning. And that worked. (30:29):
undefined

Dwarkesh: I don't think there's something super obvious there for how do you get this (30:31):
undefined

Dwarkesh: online learning, this on-the-job training working for these models. (30:35):
undefined

Ryan Sean Adams: Okay, can we talk about that, go a little bit deeper on that concept? (30:38):
undefined

Ryan Sean Adams: So this is basically one of the concepts you wrote in your recent post. (30:41):
undefined

Ryan Sean Adams: AI is not right around the corner. Even though you're an AI optimist, (30:44):
undefined

Ryan Sean Adams: I would say, and overall an AI accelerationist, you You were saying it's not (30:48):
undefined

Ryan Sean Adams: right around the corner. (30:53):
undefined

Ryan Sean Adams: You're saying the ability to replace human labor is a ways out. (30:54):
undefined

Ryan Sean Adams: Not forever out, but I think you said somewhere around 2032, (30:58):
undefined

Ryan Sean Adams: if you had to guess on when the estimate was. (31:01):
undefined

Ryan Sean Adams: And the reason you gave is because AIs can't learn on the job, (31:05):
undefined

Ryan Sean Adams: but it's not clear to me why they can't. (31:09):
undefined

Ryan Sean Adams: Is it just because the context window isn't large enough? (31:10):
undefined

Ryan Sean Adams: Is it just because they can't input all of the different data sets and data (31:14):
undefined

Ryan Sean Adams: points that humans can? Is it because they don't have stateful memory the way a human employee? (31:19):
undefined

Ryan Sean Adams: Because if it's these things, all of these do seem like solvable problems. (31:25):
undefined

Ryan Sean Adams: And maybe that's what you're saying. They are solvable problems. (31:28):
undefined

Ryan Sean Adams: They're just a little bit longer than some people think they are. (31:30):
undefined

Dwarkesh: I think it's like in some deep sense a solvable problem because eventually we will build AGI. (31:35):
undefined

Dwarkesh: And to build AGI, we will have had to solve the problem. (31:40):
undefined

Dwarkesh: My point is that the obvious solutions you might imagine, for example, (31:43):
undefined

Dwarkesh: expanding the context window or having this (31:46):
undefined

Dwarkesh: like external memory using systems like rag these (31:49):
undefined

Dwarkesh: are basically techniques we already have to it's called retrieval augmented (31:54):
undefined

Dwarkesh: generate anyways these kinds of retrieval augmented generation i (31:57):
undefined

Dwarkesh: don't think these will suffice and just to put a finer point first of all like (32:00):
undefined

Dwarkesh: what is the problem the problem is exactly as you say that within the context (32:04):
undefined

Dwarkesh: window these models actually can learn on the job right so if you talk to it (32:09):
undefined

Dwarkesh: for long enough it will get much better at understanding your needs and what your exact problem is. (32:13):
undefined

Dwarkesh: If you're using it for research for your podcast, it will get a sense of like, (32:17):
undefined

Dwarkesh: oh, they're actually especially curious about these kinds of questions. Let me focus on that. (32:21):
undefined

Dwarkesh: It's actually very human-like in context, right? The speed at which it learns, (32:25):
undefined

Dwarkesh: the task of knowledge it picks out. (32:28):
undefined

Dwarkesh: The problem, of course, is the context length for even the best models only (32:30):
undefined

Dwarkesh: last a million or two million tokens. (32:33):
undefined

Dwarkesh: That's at most like an hour of conversation. (32:36):
undefined

Dwarkesh: Now, then you might say, okay, well, why can't we just solve that by expanding (32:39):
undefined

Dwarkesh: the context window, right? So context window has been expanding for the last (32:42):
undefined

Dwarkesh: few years. Why can't we just continue that? (32:44):
undefined

Ryan Sean Adams: Yeah, like a billion token context window, something like this. (32:47):
undefined

Dwarkesh: So 2018 is when the transformer came out and the transformer has the attention mechanism. (32:50):
undefined

Dwarkesh: The attention mechanism is inherently quadratic with the nature of the length (32:55):
undefined

Dwarkesh: of the sequence, which is to say that if you go from if you double go from 1 (33:00):
undefined

Dwarkesh: million tokens to 2 million tokens, (33:05):
undefined

Dwarkesh: it actually costs four times as much compute to process that 2 millionth token. (33:07):
undefined

Dwarkesh: It's not just 2 to as much compute. so it gets super linearly more expensive (33:12):
undefined

Dwarkesh: as you increase the context length and for the last, (33:18):
undefined

Dwarkesh: seven years people have been trying to get around this inherent quadratic nature (33:23):
undefined

Dwarkesh: of attention of course we don't know secretly what the labs are working on but we have frontier, (33:26):
undefined

Dwarkesh: companies like deep seek which have open source their research and (33:32):
undefined

Dwarkesh: we can just see how their algorithms work and they found (33:35):
undefined

Dwarkesh: these constant time modifiers to attention which is (33:38):
undefined

Dwarkesh: to say that they there's like a it'll still (33:41):
undefined

Dwarkesh: be quadratic but it'll be like one half times (33:44):
undefined

Dwarkesh: quadratic but the inherent like super linearness has not (33:47):
undefined

Dwarkesh: gone away and because of that yeah you might be able to increase it from 1 million (33:49):
undefined

Dwarkesh: tokens to 2 million tokens by finding another hack like uh make sure experts (33:53):
undefined

Dwarkesh: just run such things latent attention is another such technique but or kbcash (33:57):
undefined

Dwarkesh: right there's many other things that have been discovered but people have not (34:01):
undefined

Dwarkesh: discovered okay how do you get around the fact that if you went to a billion, (34:05):
undefined

Dwarkesh: it would be a billion squared as expensive in terms of compute to process that token. (34:09):
undefined

Dwarkesh: And so I don't think you'll just get it by increasing the length of the context window, basically. (34:14):
undefined

Ryan Sean Adams: That's fascinating. Yeah, I didn't realize that. Okay, so the other reason in (34:19):
undefined

Ryan Sean Adams: your post that AI is not right around the corner is because it can't do your taxes. (34:23):
undefined

Ryan Sean Adams: And Dwarkesh, I feel your pain, man. Taxes are just like quite a pain in the ass. (34:27):
undefined

Ryan Sean Adams: I think you were talking about this from the context of like computer vision, (34:33):
undefined

Ryan Sean Adams: computer use, that kind of thing. (34:36):
undefined

Ryan Sean Adams: So, I mean, I've seen demos. I've seen some pretty interesting computer vision (34:38):
undefined

Ryan Sean Adams: sort of demos that seem to be right around the corner. (34:42):
undefined

Ryan Sean Adams: But what's the limiter on computer use for an AI? (34:46):
undefined

Dwarkesh: There was an interesting blog post by this company called Mechanize where they (34:49):
undefined

Dwarkesh: were explaining why this is such a big problem. And I love the way they phrased it, which is that, (34:54):
undefined

Dwarkesh: Imagine if you had to train a model in 1980, a large language model in 1980, (34:58):
undefined

Dwarkesh: and you could use all the compute you wanted in 1980 somehow, (35:04):
undefined

Dwarkesh: but you didn't have, you were only stuck with the data that was available in (35:08):
undefined

Dwarkesh: the 1980s, of course, before the internet became a widespread phenomenon. (35:14):
undefined

Dwarkesh: You couldn't train a modern LLM, even with all the computer in the world, (35:17):
undefined

Dwarkesh: because the data wasn't available. (35:20):
undefined

Dwarkesh: And we're in a similar position with respect to computer use, (35:22):
undefined

Dwarkesh: because there's not this corpus of collected videos, people using computers (35:26):
undefined

Dwarkesh: to do different things, to access different applications and do white collar work. (35:31):
undefined

Dwarkesh: Because of that, I think the big challenge has been accumulating this kind of data. off. (35:37):
undefined

Ryan Sean Adams: And to be clear, when I was saying the use case of like, do my taxes, (35:43):
undefined

Ryan Sean Adams: you're effectively talking about an AI having the ability to just like, (35:47):
undefined

Ryan Sean Adams: you know, navigate the files around your computer, (35:51):
undefined

Ryan Sean Adams: you know, log in to various websites to download your pay stubs or whatever, (35:54):
undefined

Ryan Sean Adams: and then to go to like TurboTax or something and like input it all into some (35:58):
undefined

Ryan Sean Adams: software and file it, right? (36:02):
undefined

Ryan Sean Adams: Just on voice command or something like that. That's basically doing my taxes. (36:04):
undefined

Dwarkesh: It should be capable of navigating UIs that it's less familiar with or that (36:08):
undefined

Dwarkesh: come about organically within the context of trying to solve a problem. (36:13):
undefined

Dwarkesh: So for example, I might have business deductions. (36:17):
undefined

Dwarkesh: It sees on my bank statement that I've spent $1,000 on Amazon. (36:20):
undefined

Dwarkesh: It goes logs in my Amazon. (36:24):
undefined

Dwarkesh: It sees like, oh, he bought a camera. So I think that's probably a business (36:25):
undefined

Dwarkesh: expense for his podcast. (36:29):
undefined

Dwarkesh: He bought an Airbnb over a weekend in the cabins of whatever, (36:30):
undefined

Dwarkesh: in the woods of whatever. That probably wasn't a business expense. (36:35):
undefined

Dwarkesh: Although maybe, maybe it's, if it's a sort of like a gray, if it's willing to (36:38):
undefined

Dwarkesh: go in the gray area, maybe I'll talk to you. Yeah, yeah, yeah. (36:42):
undefined

Ryan Sean Adams: Do the gray area stuff. (36:45):
undefined

Dwarkesh: I was, I was researching. (36:46):
undefined

Dwarkesh: But anyway, so that, including all of that, including emailing people for invoices, (36:50):
undefined

Dwarkesh: and haggling with them, it would be like a sort of week long task to do my taxes, right? (36:55):
undefined

Dwarkesh: You'd have to, there's a lot of work involved. That's not just like do this (37:01):
undefined

Dwarkesh: skill, this skill, this skill, but rather of having a sort of like plan of action (37:04):
undefined

Dwarkesh: and then breaking tasks apart, dealing with new information, (37:08):
undefined

Dwarkesh: new emails, new messages, consulting with me about questions, et cetera. (37:11):
undefined

Ryan Sean Adams: Yeah, I mean, to be clear on this use case too, even though your post is titled (37:16):
undefined

Ryan Sean Adams: like, you know, AI is not right around the corner, you still think this ability (37:18):
undefined

Ryan Sean Adams: to file your taxes, that's like a 2028 thing, right? (37:22):
undefined

Ryan Sean Adams: I mean, this is maybe not next year, but it's in a few years. (37:27):
undefined

Dwarkesh: Right, which is, I think that was sort of, people maybe write too much in The (37:31):
undefined

Dwarkesh: Decital and then didn't read through the arguments. (37:35):
undefined

Ryan Sean Adams: I mean, that never happens on the internet. Wow. (37:37):
undefined

Dwarkesh: First time. (37:40):
undefined

Dwarkesh: No, I think like I'm arguing against people who are like, you know, this will happen. (37:42):
undefined

Dwarkesh: AGI is like two years away. I do think the wider world, the markets, (37:47):
undefined

Dwarkesh: public perception, even people who are somewhat attending to AI, (37:53):
undefined

Dwarkesh: but aren't in this specific milieu that I'm talking to, are way underpricing AGI. (37:57):
undefined

Dwarkesh: One reason, one thing I think they're underestimating is not only will we have (38:03):
undefined

Dwarkesh: millions of extra laborers, millions of extra workers, (38:09):
undefined

Dwarkesh: potentially billions within the course of the next decade, because then we will (38:12):
undefined

Dwarkesh: have a potentially, I think like likely we will have AGI within the next decade. (38:16):
undefined

Dwarkesh: But they'll have these advantages that human workers don't have, (38:20):
undefined

Dwarkesh: which is that, okay, a single model company, so suppose we solve continual learning, right? (38:23):
undefined

Dwarkesh: So there, and we saw computer use. So as far as white collar work goes, (38:27):
undefined

Dwarkesh: that might fundamentally it would be solved. (38:31):
undefined

Dwarkesh: You can have AIs which can use not just they're not just like a text box where (38:32):
undefined

Dwarkesh: you put into you ask questions in a chatbot and you get some response out. (38:36):
undefined

Dwarkesh: It's not that useful to just have a very smart chatbot. You need it to be able (38:39):
undefined

Dwarkesh: to actually do real work and use real applications. (38:42):
undefined

Dwarkesh: Suppose you have that solved because it acts like an employee. (38:45):
undefined

Dwarkesh: It's got continual learning. It's got computer use. (38:48):
undefined

Dwarkesh: But it has another advantage that humans don't have, which is that copies of (38:49):
undefined

Dwarkesh: this model are going being deployed all through the economy and it's doing on the job training. (38:53):
undefined

Dwarkesh: So copies are learning how to be an accountant, how to be a lawyer, (38:57):
undefined

Dwarkesh: how to be a coder, except because it's an AI and it's digital, (39:00):
undefined

Dwarkesh: the model itself can amalgamate all this on-the-job training from all these copies. (39:04):
undefined

Dwarkesh: So what does that mean? Well, it means that even if there's no more software (39:10):
undefined

Dwarkesh: progress after that point, which is to say that no more algorithms are discovered, (39:13):
undefined

Dwarkesh: there's not a transformer plus plus that's discovered. (39:17):
undefined

Dwarkesh: Just from the fact that this model is learning every single skill in the economy, (39:20):
undefined

Dwarkesh: at least for white-collar work, you might just, based on that alone, (39:25):
undefined

Dwarkesh: have something that looks like an intelligence explosion. (39:30):
undefined

Dwarkesh: It would just be a broadly deployed intelligence explosion, but it would functionally (39:31):
undefined

Dwarkesh: become super intelligent just from having human-level capability of learning on the job. (39:35):
undefined

Dwarkesh Patel: Yeah, and it creates this mesh network of intelligence that's shared among everyone. (39:41):
undefined

Dwarkesh Patel: That's a really fascinating thing. So we're going to get there. (39:45):
undefined

Dwarkesh Patel: We're going to get to AGI. it's going to be incredibly smart. (39:48):
undefined

Dwarkesh Patel: But what we've shared recently is just kind of this mixed bag where currently (39:51):
undefined

Dwarkesh Patel: today, it's pretty good at some things, but also not that great at others. (39:54):
undefined

Dwarkesh Patel: We're hiring humans to do jobs that we think AI should do, but it probably doesn't. (39:57):
undefined

Dwarkesh Patel: So the question I have for you is, is AI really that smart? Or is it just good (40:00):
undefined

Dwarkesh Patel: at kind of acing these particular benchmarks that we measure against? (40:04):
undefined

Dwarkesh Patel: Apple, I mean, famously recently, they had their paper, The Illusion of Thinking, (40:08):
undefined

Dwarkesh Patel: where it was kind of like, hey, AI is like pretty good up to a point, (40:11):
undefined

Dwarkesh Patel: but at a certain point, it just falls apart. (40:14):
undefined

Dwarkesh Patel: And the inference is like, maybe it's not intelligence, maybe it's just good (40:16):
undefined

Dwarkesh Patel: at guessing. So I guess the question is, is AI really that smart? (40:21):
undefined

Dwarkesh: It depends on who I'm talking to. I think some people overhype its capabilities. (40:24):
undefined

Dwarkesh: I think some people are like, oh, it's already AGI, but it's like a little hobbled (40:27):
undefined

Dwarkesh: little AGI where we're like sort of giving it a concussion every couple of hours (40:31):
undefined

Dwarkesh: and like it forgets everything. (40:35):
undefined

Dwarkesh: We're like trapped in a chatbot context. But fundamentally, the thing inside (40:37):
undefined

Dwarkesh: is like a very smart human. (40:41):
undefined

Dwarkesh: I disagree with that perspective. So if that's your perspective, (40:44):
undefined

Dwarkesh: I say like, no, it's not that smart. (40:46):
undefined

Dwarkesh: Your perspective is just statistical associations. I say definitely smarter. (40:47):
undefined

Dwarkesh: Like it's like genuinely there's an intelligence there. (40:51):
undefined

Dwarkesh: And the, so one thing you could say to the person who thinks that it's already (40:54):
undefined

Dwarkesh: AGI is this, look, if a single human had as much stuff memorized as these models (40:58):
undefined

Dwarkesh: seem to have memorized, right? (41:03):
undefined

Dwarkesh: Which is to say that they have all of internet text, everything that human has (41:04):
undefined

Dwarkesh: written on the internet memorized, they would potentially be discovering all (41:08):
undefined

Dwarkesh: kinds of connections and discoveries. (41:13):
undefined

Dwarkesh: They'd notice that this thing which causes a migraine is associated with this kind of deficiency. (41:16):
undefined

Dwarkesh: So maybe if you take the supplement, your migraines will be cured. (41:22):
undefined

Dwarkesh: There'd be just this list of just like trivial connections that lead to big (41:25):
undefined

Dwarkesh: discoveries all through the place. (41:28):
undefined

Dwarkesh: It's not clear that there's been an unambiguous case of an AI just doing this by itself. (41:30):
undefined

Dwarkesh: So then why, so that's something potentially to explain, like if they're so (41:37):
undefined

Dwarkesh: intelligent, why aren't they able to use their disproportionate capabilities, (41:40):
undefined

Dwarkesh: their unique capabilities to come up with these discoveries? (41:44):
undefined

Dwarkesh: I don't think there's actually a good answer to that question yet, (41:47):
undefined

Dwarkesh: except for the fact that they genuinely aren't that creative. (41:49):
undefined

Dwarkesh: Maybe they're like intelligent in the sense of knowing a lot of things, (41:51):
undefined

Dwarkesh: but they don't have this fluid intelligence that humans have. (41:54):
undefined

Dwarkesh: Anyway, so I give you a wish-washy answer because I think some people are underselling (41:57):
undefined

Dwarkesh: the intelligence. Some people are overselling it. (42:00):
undefined

Ryan Sean Adams: I recall a tweet lately from Tyler Cowen. I think he was referring to maybe (42:03):
undefined

Ryan Sean Adams: O3, and he basically said, it feels like AGI. (42:07):
undefined

Ryan Sean Adams: I don't know if it is AGI or not, but like to me, it feels like AGI. (42:10):
undefined

Ryan Sean Adams: What do you account for this feeling of like intelligence then (42:14):
undefined

Dwarkesh: I think this is actually very interesting because it gets to a crux that Tyler (42:18):
undefined

Dwarkesh: and I have so Tyler and I disagree on two big things one he thinks you know (42:22):
undefined

Dwarkesh: as he said in the blog post 03 is AGI I don't think it's AGI I think it's, (42:28):
undefined

Dwarkesh: it's orders of magnitude less valuable or, you know, like many orders of magnitude (42:32):
undefined

Dwarkesh: less valuable and less useful than an AGI. (42:37):
undefined

Dwarkesh: That's one thing we disagree on. The other thing we disagree on is he thinks (42:39):
undefined

Dwarkesh: that once we do get AGI, we'll only see 0.5% increase in the economic growth (42:43):
undefined

Dwarkesh: rate. This is like what the internet caused, right? (42:47):
undefined

Dwarkesh: Whereas I think we will see tens of percent increase in economic growth. (42:49):
undefined

Dwarkesh: Like it will just be the difference between the pre-industrial revolution rate (42:53):
undefined

Dwarkesh: of growth versus industrial revolution, that magnitude of change again. (42:57):
undefined

Dwarkesh: And I think these two disagreements are linked because if you do believe we're (43:00):
undefined

Dwarkesh: already at AGI and you look around the world and you say like, (43:05):
undefined

Dwarkesh: well, it fundamentally looks the same, you'd be forgiven for thinking like, (43:08):
undefined

Dwarkesh: oh, there's not that much value in getting to AGI. (43:12):
undefined

Dwarkesh: Whereas if you are like me and you think like, no, we'll get this broadly at (43:14):
undefined

Dwarkesh: the minimum, at a very minimum, we'll get a broadly deployed intelligence explosion once we get to AGI, (43:17):
undefined

Dwarkesh: then you're like, OK, I'm just expecting some sort of singulitarian crazy future (43:22):
undefined

Dwarkesh: with a robot factories and, you know, solar farms all across the desert and things like that. (43:26):
undefined

Ryan Sean Adams: Yeah, I mean, it strikes me that your disagreement with Tyler is just based (43:31):
undefined

Ryan Sean Adams: on the semantic definition of like what AGI actually is. (43:35):
undefined

Ryan Sean Adams: And Tyler, it sounds like he has kind of a lower threshold for what AGI is, (43:39):
undefined

Ryan Sean Adams: whereas you have a higher threshold. (43:44):
undefined

Ryan Sean Adams: Is there like a accepted definition for AGI? (43:45):
undefined

Dwarkesh: No. One thing that's useful for the purposes of discussions is to say automating (43:48):
undefined

Dwarkesh: all white collar work because robotics hasn't made as much progress as LLMs (43:54):
undefined

Dwarkesh: have or computer use has. (43:59):
undefined

Dwarkesh: So if we just say anything a human can do or maybe 90% of what humans can do (44:01):
undefined

Dwarkesh: at a desk, an AI can also do, that's potentially a useful definition for at (44:06):
undefined

Dwarkesh: least getting the cognitive elements relevant to defining AGI. (44:11):
undefined

Dwarkesh: But yeah, there's not one definition which suits all purposes. (44:15):
undefined

Ryan Sean Adams: Do we know what's like going on inside of these models, right? (44:18):
undefined

Ryan Sean Adams: So like, you know, Josh was talking earlier in the conversation about like this (44:23):
undefined

Ryan Sean Adams: at the base being sort of token prediction, right? (44:26):
undefined

Ryan Sean Adams: And I guess this starts to raise the question of like, what is intelligence in the first place? (44:29):
undefined

Ryan Sean Adams: And these AI models, I mean, they seem like they're intelligent, (44:35):
undefined

Ryan Sean Adams: but do they have a model of the world the way maybe a human might? (44:40):
undefined

Ryan Sean Adams: Are they sort of babbling or like, is this real reasoning? (44:44):
undefined

Ryan Sean Adams: And like, what is real reasoning? Do we just judge that based on the results (44:49):
undefined

Ryan Sean Adams: or is there some way to like peek inside of its head? (44:53):
undefined

Dwarkesh: I used to have similar questions a couple of years ago. And then, (44:56):
undefined

Dwarkesh: because honestly, the things they did at the time were like ambiguous. (45:00):
undefined

Dwarkesh: You could say, oh, it's close enough to something else in this trading data set. (45:03):
undefined

Dwarkesh: That is just basically copy pasting. It didn't come up with a solution by itself. (45:07):
undefined

Dwarkesh: But we've gotten to the point where I can come up with a pretty complicated (45:12):
undefined

Dwarkesh: math problem and it will solve it. (45:17):
undefined

Dwarkesh: It can be a math problem, like not like, you know, undergrad or high school math problem. (45:19):
undefined

Dwarkesh: Like the problem we get, the problems the smartest math professors come up with (45:24):
undefined

Dwarkesh: in order to test International Math Olympiad. (45:28):
undefined

Dwarkesh: You know, the kids who spend all their life preparing for this, (45:31):
undefined

Dwarkesh: the geniuses who spend all their life, all their young adulthood preparing to (45:34):
undefined

Dwarkesh: take these really gnarly math puzzle challenges. (45:37):
undefined

Dwarkesh: And the model will get these kinds of questions, right? They require all this (45:40):
undefined

Dwarkesh: abstract creative thinking, this reasoning for hours, the model will get the right. (45:43):
undefined

Dwarkesh: Okay, so if that's not reasoning, then why is reasoning valuable again? (45:48):
undefined

Dwarkesh: Like, what exactly was this reasoning supposed to be? (45:53):
undefined

Dwarkesh: So I think they genuinely are reasoning. I mean, I think there's other capabilities (45:56):
undefined

Dwarkesh: they lack, which are actually more, in some sense, they seem to us to be more (45:59):
undefined

Dwarkesh: trivial, but actually much harder to learn. But the reasoning itself, I think, is there. (46:03):
undefined

Dwarkesh Patel: And the answer to the intelligence question is also kind of clouded, (46:07):
undefined

Dwarkesh Patel: right? Because we still really don't understand what's going on in an LLM. (46:10):
undefined

Dwarkesh Patel: Dario from Anthropoc, he recently posted the paper about interpretation. (46:14):
undefined

Dwarkesh Patel: And can you explain why we don't even really understand what's going on in these (46:17):
undefined

Dwarkesh Patel: LLMs, even though we're able to make them and yield the results from them? Mmm. (46:21):
undefined

Dwarkesh Patel: Because it very much still is kind of like a black box. We write some code, (46:27):
undefined

Dwarkesh Patel: we put some inputs in, and we get something out, but we're not sure what happens in the middle, (46:30):
undefined

Dwarkesh: Why it's creating this output. (46:34):
undefined

Dwarkesh Patel: I mean, it's exactly what you're saying. (46:36):
undefined

Dwarkesh: It's that in other systems we engineer in the world, we have to build it up bottom-ups. (46:38):
undefined

Dwarkesh: If you build a bridge, you have to understand how every single beam is contributing to the structure. (46:44):
undefined

Dwarkesh: And we have equations for why the thing will stay standing. (46:50):
undefined

Dwarkesh: There's no such thing for AI. We didn't build it, more so we grew it. (46:54):
undefined

Dwarkesh: It's like watering a plant. And a couple thousand years ago, (46:59):
undefined

Dwarkesh: they were doing agriculture, but they didn't know why. (47:03):
undefined

Dwarkesh: Why do plants grow? How do they collect energy from sunlight? All these things. (47:08):
undefined

Dwarkesh: And I think we're in a substantially similar position with respect to intelligence, (47:13):
undefined

Dwarkesh: with respect to consciousness, with respect to all these other interesting questions (47:18):
undefined

Dwarkesh: about how minds work, which is in some sense really cool because there's this (47:23):
undefined

Dwarkesh: huge intellectual horizon that's become not only available, but accessible to investigation. (47:27):
undefined

Dwarkesh: In another sense, it's scary because we know that minds can suffer. (47:33):
undefined

Dwarkesh: We know that minds have moral worth and we're creating minds and we have no (47:37):
undefined

Dwarkesh: understanding of what's happening in these minds. (47:42):
undefined

Dwarkesh: Is a process of gradient descent a painful process? (47:44):
undefined

Dwarkesh: We don't know, but we're doing a lot of it. (47:48):
undefined

Dwarkesh: So hopefully we'll learn more. But yeah, I think we're in a similar position (47:51):
undefined

Dwarkesh: to some farmer in Uruk in 3500 BC. (47:54):
undefined

Josh Kale: Wow. (47:57):
undefined

Ryan Sean Adams: And I mean, the potential, the idea that minds can suffer, minds have some moral (47:58):
undefined

Ryan Sean Adams: worth, and also minds have some free will. (48:03):
undefined

Ryan Sean Adams: They have some sort of autonomy, or maybe at least a desire to have autonomy. (48:06):
undefined

Ryan Sean Adams: I mean, this brings us to kind of this sticky subject of alignment and AI safety (48:11):
undefined

Ryan Sean Adams: and how we go about controlling the intelligence that we're creating, (48:15):
undefined

Ryan Sean Adams: if even that's what we should be doing, controlling it. And we'll get to that in a minute. (48:20):
undefined

Ryan Sean Adams: But I want to start with maybe the headlines here a little bit. (48:24):
undefined

Ryan Sean Adams: So headline just this morning, latest OpenAI models sabotaged a shutdown mechanism (48:28):
undefined

Ryan Sean Adams: despite commands to the contrary. (48:34):
undefined

Ryan Sean Adams: OpenAI's O1 model attempted to copy itself to external servers after being threatened (48:36):
undefined

Ryan Sean Adams: with shutdown that denied the action when discovered. (48:41):
undefined

Ryan Sean Adams: I've read a number of papers for this. Of course, mainstream media has these (48:44):
undefined

Ryan Sean Adams: types of headlines almost on a weekly basis now, and it's starting to get to daily. (48:48):
undefined

Ryan Sean Adams: But there does seem to be some evidence that AIs lie to us, (48:53):
undefined

Ryan Sean Adams: If that's even the right term, in order to pursue goals, goals like self-preservation, (48:58):
undefined

Ryan Sean Adams: goals like replication, even deep-seated values that we might train into them, (49:03):
undefined

Ryan Sean Adams: sort of a constitution type of value. (49:08):
undefined

Ryan Sean Adams: They seek to preserve these values, which maybe that's a good thing, (49:11):
undefined

Ryan Sean Adams: or maybe it's not a good thing if we don't actually want them to interpret the values in a certain way. (49:15):
undefined

Ryan Sean Adams: Some of these headlines that we're seeing now, To you, with your kind of corpus (49:21):
undefined

Ryan Sean Adams: of knowledge and all of the interviews and discovery you've done on your side, (49:25):
undefined

Ryan Sean Adams: is this like media sensationalism or is this like alarming? (49:29):
undefined

Ryan Sean Adams: And if it's alarming, how concerned should we be about this? (49:33):
undefined

Dwarkesh: I think on net, it's quite alarming. I do think that some of these results have (49:37):
undefined

Dwarkesh: been sort of cherry picked. (49:42):
undefined

Dwarkesh: Or if you look into the code, what's happened is basically the researchers have (49:44):
undefined

Dwarkesh: said, hey, pretend to be a bad person. (49:47):
undefined

Dwarkesh: Wow, AI is being a bad person. Isn't that crazy? (49:50):
undefined

Dwarkesh: But the system prompt is just like hey do this bad thing right now i personally (49:53):
undefined

Dwarkesh: but i have also seen other results which are not of this quality i mean the (49:57):
undefined

Dwarkesh: the clearest example so backing up, (50:02):
undefined

Dwarkesh: what is the reason to think this will be a bigger problem in the future than (50:05):
undefined

Dwarkesh: it is now because we all interact with these systems and they're actually like (50:08):
undefined

Dwarkesh: quite moral or aligned right like you can talk to a chatbot and you like ask (50:13):
undefined

Dwarkesh: it to how should you deal with some crisis where there's a correct answer, (50:17):
undefined

Dwarkesh: you know, like it will tell you not to be violent. It'll give you reasonable advice. (50:22):
undefined

Dwarkesh: It seems to have good values. So it's worth noticing this, right? (50:26):
undefined

Dwarkesh: And being happy about it. (50:29):
undefined

Dwarkesh: The concern is that we're moving from a regime where we've trained them on human (50:31):
undefined

Dwarkesh: language, which implicitly has human morals and the way, you know, (50:36):
undefined

Dwarkesh: normal people think about values implicit in it. (50:41):
undefined

Dwarkesh: Plus this RLHF process we did to a regime where we're mostly spending compute (50:44):
undefined

Dwarkesh: on just having them answer problems yes or no or correct or not rather just like. (50:50):
undefined

Dwarkesh: And pass all the unit tests, get the right answer on this math problem. (50:58):
undefined

Dwarkesh: And this has no guardrails intrinsically in terms of what is allowed to do, (51:02):
undefined

Dwarkesh: what is the proper moral way to do something. (51:09):
undefined

Dwarkesh: I think that can be a loaded term, but here's a more concrete example. (51:11):
undefined

Dwarkesh: One problem we're running into with these coding agents more and more, (51:15):
undefined

Dwarkesh: and this has nothing to do with these abstract concerns about alignment, (51:18):
undefined

Dwarkesh: but more so just like how do we get economic value out of these models, (51:21):
undefined

Dwarkesh: is that Claude or Gemini will, instead of writing code such that it passes the unit tests, (51:24):
undefined

Dwarkesh: it will often just delete the unit tests so that the code just passes by default. (51:32):
undefined

Dwarkesh: Now, why would it do that? Well, it's learned in the process. (51:37):
undefined

Dwarkesh: It was trained on the goal during training of you must pass all unit tests. (51:41):
undefined

Dwarkesh: And probably within some environment in which it was trained, (51:45):
undefined

Dwarkesh: it was able to just get away. (51:48):
undefined

Dwarkesh: Like there wasn't designed well enough. And so it found this like little hole (51:49):
undefined

Dwarkesh: where it could just like delete the file that had the unit test or rewrite them (51:53):
undefined

Dwarkesh: so that it always said, you know, equals true, then pass. (51:55):
undefined

Dwarkesh: And right now we can discover these even without, even though we can discover (51:59):
undefined

Dwarkesh: these, you know, it's still past, there's still been enough hacks like this, (52:03):
undefined

Dwarkesh: such that the model is like becoming more and more hacky like that. (52:06):
undefined

Dwarkesh: In the future, we're going to be training models in ways that we is beyond our (52:10):
undefined

Dwarkesh: ability to even understand, certainly beyond everybody's ability to understand. (52:14):
undefined

Dwarkesh: There may be a few people who might be able to see just the way that right now, (52:18):
undefined

Dwarkesh: if you came up with a new math proof for some open problem in mathematics, (52:21):
undefined

Dwarkesh: there will be only be a few people in the world who will be able to evaluate that math proof. (52:24):
undefined

Dwarkesh: We'll be in a similar position with respect to all of the things that these (52:28):
undefined

Dwarkesh: models are being trained on at the frontier, especially math and code, (52:31):
undefined

Dwarkesh: because humans were big dum-dums with respect to this reasoning stuff. (52:34):
undefined

Dwarkesh: And so there's a sort of like first principles reason to expect that this new (52:38):
undefined

Dwarkesh: modality of training will be less amenable to the kinds of supervision that (52:41):
undefined

Dwarkesh: was grounded within the pre-training corpus. (52:46):
undefined

Ryan Sean Adams: I don't know that everyone has kind of an intuition or an idea why it doesn't (52:49):
undefined

Ryan Sean Adams: work to just say like, so if we don't want our AI models to lie to us, (52:54):
undefined

Ryan Sean Adams: why can't we just tell them not to lie? (52:59):
undefined

Ryan Sean Adams: Why can't we just put that as part of their core constitution? (53:01):
undefined

Ryan Sean Adams: If we don't want our AI models to be sycophants, why can't we just say, (53:05):
undefined

Ryan Sean Adams: hey, if I tell you I want the truth, not to flatter me, just give me the straight up truth. (53:10):
undefined

Ryan Sean Adams: Why is this even difficult to do? (53:16):
undefined

Dwarkesh: Well, fundamentally, it comes down to how we train them. And we don't know how (53:18):
undefined

Dwarkesh: to train them in a way that does not reward lying or sycophancy. (53:22):
undefined

Dwarkesh: In fact, the problem is OpenAI, they explained why their recent model of theirs (53:26):
undefined

Dwarkesh: was they had to take down was just sycophantic. (53:30):
undefined

Dwarkesh: And the reason was just that they rolled out, did it in the A-B test and the (53:33):
undefined

Dwarkesh: version, the test that was more sycophantic was just preferred by users more. (53:37):
undefined

Dwarkesh: Sometimes you prefer the lie. (53:42):
undefined

Dwarkesh: Yeah, so that's, if that's what's preferred in training, you know, (53:44):
undefined

Dwarkesh: Or, for example, in the context of lying, if we've just built RL environments (53:47):
undefined

Dwarkesh: in which we're training these models, where they're going to be more successful if they lie, right? (53:52):
undefined

Dwarkesh: So if they delete the unit tests and then tell you, I passed this program and (53:59):
undefined

Dwarkesh: all the unit tests have succeeded, it's like lying to you, basically. (54:06):
undefined

Dwarkesh: And if that's what is rewarded in the process of gradient descent, (54:09):
undefined

Dwarkesh: then it's not surprising that the model you interact with will just have this (54:12):
undefined

Dwarkesh: drive to lie if it gets it closer to its goal. (54:17):
undefined

Dwarkesh: And I would just expect this to keep happening unless we can solve this fundamental (54:20):
undefined

Dwarkesh: problem that comes about in training. (54:24):
undefined

Dwarkesh Patel: So you mentioned how like ChatGPT had a version that was sycophantic, (54:26):
undefined

Dwarkesh Patel: and that's because users actually wanted that. (54:30):
undefined

Dwarkesh Patel: Who is in control? Who decides the actual alignment of these models? (54:32):
undefined

Dwarkesh Patel: Because users are saying one thing, and then they deploy it, (54:36):
undefined

Dwarkesh Patel: and then it turns out that's not actually what people want. (54:38):
undefined

Dwarkesh Patel: How do you kind of form consensus around this alignment or these alignment principles? (54:41):
undefined

Dwarkesh: Right now, obviously, it's the labs who decided this, right? (54:47):
undefined

Dwarkesh: And the safety teams of the labs. (54:49):
undefined

Dwarkesh: And I guess the question you could ask is then who should decide these? Because this will be... (54:51):
undefined

Dwarkesh Patel: Assuming the trajectory, yeah. So we keep going to get more powerful. (54:56):
undefined

Dwarkesh: Because this will be the key modality that all of us use to get, (54:59):
undefined

Dwarkesh: not only get work done, but even like, I think at some point, (55:03):
undefined

Dwarkesh: a lot of people's best friends will be AIs, at least functionally in the sense (55:06):
undefined

Dwarkesh: of who do they spend the most amount of time talking to. It might already be AIs. (55:10):
undefined

Dwarkesh: This will be the key layer in your business that you're using to get work done (55:14):
undefined

Dwarkesh: so this process of training which shapes their personality who gets to control (55:20):
undefined

Dwarkesh: it I mean it will be the laughs functionally, (55:25):
undefined

Dwarkesh: But maybe you mean, like, who should control it, right? I honestly don't know. (55:30):
undefined

Dwarkesh: I mean, I don't know if there's a better alternative to the labs. (55:34):
undefined

Dwarkesh Patel: Yeah, I would assume, like, there's some sort of social consensus, (55:36):
undefined

Dwarkesh Patel: right? Similar to how we have in America, the Constitution. (55:39):
undefined

Dwarkesh Patel: There's, like, this general form of consensus that gets formed around how we (55:41):
undefined

Dwarkesh Patel: should treat these models as they become as powerful as we think they probably will be. (55:44):
undefined

Dwarkesh: Honestly, I don't have, I don't know if anybody has a good answer about how (55:47):
undefined

Dwarkesh: you do this process. I think we lucked out, we just, like, really lucked out with the Constitution. (55:49):
undefined

Dwarkesh: It also wasn't a democratic process which resulted in the constitution, (55:55):
undefined

Dwarkesh: even though it instituted a Republican form of government. (55:58):
undefined

Dwarkesh: It was just delegates from each state. They haggled it out over the course of a few months. (56:00):
undefined

Dwarkesh: Maybe that's what happens with AI. But is there some process which feels both (56:05):
undefined

Dwarkesh: fair and which will result in actually a good constitution for these AIs? (56:10):
undefined

Dwarkesh: It's not obvious to me that, I mean, nothing comes up to the top of my head. (56:15):
undefined

Dwarkesh: Like, oh, this, you know, do rank choice voting or something. (56:19):
undefined

Dwarkesh Patel: Yeah, so I was going to ask, is there any, I mean, having spoken to everyone (56:22):
undefined

Dwarkesh Patel: who you've spoken to is there any alignment path which looks most promising which (56:24):
undefined

Dwarkesh: Feels the. (56:27):
undefined

Dwarkesh Patel: Most comforting and exciting to you (56:28):
undefined

Dwarkesh: I i think alignment in the sense of you (56:30):
undefined

Dwarkesh: know and eventually we'll have these super intelligent systems what do we do (56:33):
undefined

Dwarkesh: about that i think the the approach that i think is most promising is less about (56:36):
undefined

Dwarkesh: finding some holy grail some you know giga brain solution some equation which (56:44):
undefined

Dwarkesh: solves the whole puzzle and more like one. (56:49):
undefined

Dwarkesh: Having this Swiss cheese approach where, look, we kind of have gotten really good at jailbreaks. (56:53):
undefined

Dwarkesh: I'm sure you've heard a lot about jailbreaks over the last few years. (57:01):
undefined

Dwarkesh: It's actually much harder to jailbreak these models because, (57:03):
undefined

Dwarkesh: you know, people try to whack at these things in different ways. (57:06):
undefined

Dwarkesh: Model developers just like patched these obvious ways to do jailbreaks. (57:10):
undefined

Dwarkesh: The model also got smarter. So it's better able to understand when somebody (57:14):
undefined

Dwarkesh: is trying to jailbreak into it. (57:18):
undefined

Dwarkesh: That, I think, is one approach. Another is, I think, competition. (57:20):
undefined

Dwarkesh: I think the scary version of the future is where you have this dynamic where (57:24):
undefined

Dwarkesh: a single model and its copies are controlling the entire economy. (57:27):
undefined

Dwarkesh: When politicians want to understand what policies to pass, they're only talking (57:31):
undefined

Dwarkesh: to copies of a single model. (57:35):
undefined

Dwarkesh: If there's multiple different AI companies who are at the frontier, (57:36):
undefined

Dwarkesh: who have competing services, and whose models can monitor each other, right? (57:40):
undefined

Dwarkesh: So Claude may care about its own copies being successful in the world and it (57:44):
undefined

Dwarkesh: might be able to willing to lie on their behalf, even if you ask one copy to supervise another. (57:50):
undefined

Dwarkesh: I think you get some advantage from a copy of OpenAI's model monitoring a copy (57:53):
undefined

Dwarkesh: of DeepSeek's model, which actually brings us back to the Constitution, right? (57:58):
undefined

Dwarkesh: One of the most brilliant things in the Constitution is the system of checks and balances. (58:01):
undefined

Dwarkesh: So some combination of the Swiss cheese approach to model development and training (58:04):
undefined

Dwarkesh: and alignment, where you're careful, if you notice this kind of reward hacking, (58:09):
undefined

Dwarkesh: you do your best to solve it. (58:13):
undefined

Dwarkesh: You try to keep as much of the models thinking in human language rather than (58:14):
undefined

Dwarkesh: letting it think in AI thought in this latent space thinking. (58:19):
undefined

Dwarkesh: And the other part of it is just having normal market competition between these (58:23):
undefined

Dwarkesh: companies so that you can use them to check each other and no one company or (58:27):
undefined

Dwarkesh: no one AI is dominating the economy or advisory roles for governments. (58:31):
undefined

Ryan Sean Adams: I really like this like bundle of ideas that you sort of put together in that (58:41):
undefined

Ryan Sean Adams: because like, I think a lot of the, you know, AI safety conversation is always (58:45):
undefined

Ryan Sean Adams: couched in terms of control. (58:50):
undefined

Ryan Sean Adams: Like we have to control the thing that is the way. And I always get a little (58:52):
undefined

Ryan Sean Adams: worried when I hear like terms like control. (58:56):
undefined

Ryan Sean Adams: And it reminds me of a blog post I think you put out, which I'm hopeful you continue to write on. (58:59):
undefined

Ryan Sean Adams: I think you said it was going to be like one of a series, which is this idea (59:05):
undefined

Ryan Sean Adams: of like classical liberal AGI. And we were talking about themes like balance of power. (59:08):
undefined

Ryan Sean Adams: Let's have Claude check in with ChatGPT and monitor it. (59:13):
undefined

Josh Kale: When you have themes like transparency as well, (59:17):
undefined

Ryan Sean Adams: That feels a bit more, you know, classically liberal coded than maybe some of (59:19):
undefined

Ryan Sean Adams: the other approaches that I've heard. (59:25):
undefined

Ryan Sean Adams: And you wrote this in the post, which I thought was kind of, (59:27):
undefined

Ryan Sean Adams: it just sparked my interest because I'm not sure where you're going to go next (59:30):
undefined

Ryan Sean Adams: with this, but you said the most likely way this happens, (59:33):
undefined

Ryan Sean Adams: that is AIs have a stake in humanity's future, is if it's in the AI's best interest (59:37):
undefined

Ryan Sean Adams: to operate within our existing laws and norms. (59:42):
undefined

Ryan Sean Adams: You know, this whole idea that like, hey, the way to get true AI alignment is (59:45):
undefined

Ryan Sean Adams: to make it easy, make it the path of least resistance for AI to basically partner with humans. (59:49):
undefined

Ryan Sean Adams: It's almost this idea if the aliens (59:56):
undefined

Ryan Sean Adams: landed or something, we would create treaties with the aliens, right? (59:59):
undefined

Ryan Sean Adams: We would want them to adopt our norms. We would want to initiate trade with them. (01:00:03):
undefined

Ryan Sean Adams: Our first response shouldn't be, let's try to dominate and control them. (01:00:08):
undefined

Ryan Sean Adams: Maybe it should be, let's try to work with them. Let's try to collaborate. (01:00:13):
undefined

Ryan Sean Adams: Let's try to open up trade. (01:00:16):
undefined

Ryan Sean Adams: What's your idea here? And like, are you planning to write further posts about this? (01:00:18):
undefined

Dwarkesh: Yeah, I want to. It's just such a hard topic to think about that, (01:00:22):
undefined

Dwarkesh: you know, something always comes up. (01:00:25):
undefined

Dwarkesh: But the fundamental point I was making is, look, in the long run, (01:00:26):
undefined

Dwarkesh: if AIs are, you know, human labor is going to be obsolete because of these inherent (01:00:32):
undefined

Dwarkesh: advantages that digital minds will have and robotics will eventually be solved. (01:00:37):
undefined

Dwarkesh: So our only leverage on the future will no longer come from our labor. (01:00:41):
undefined

Dwarkesh: It will come from our legal and economic control over the society that AIs will (01:00:50):
undefined

Dwarkesh: be participating in, right? So, you know, AIs might make the economy explode (01:00:58):
undefined

Dwarkesh: in the sense of grow a lot. (01:01:03):
undefined

Dwarkesh: And for humans to benefit from that, it would have to be the case that AIs still (01:01:04):
undefined

Dwarkesh: respect your equity in the S&P 500 companies that you bought, right? (01:01:08):
undefined

Dwarkesh: Or for the AIs to follow your laws, which say that you can't do violence onto (01:01:13):
undefined

Dwarkesh: humans and you got to respect humans' properties. (01:01:18):
undefined

Josh Kale: It would have to be the case that AIs are actually bought into our (01:01:21):
undefined

Dwarkesh: System of government, into our laws and norms. And for that to happen, (01:01:25):
undefined

Dwarkesh: the way that likely happens is if it's just like the default path for the AIs (01:01:30):
undefined

Dwarkesh: as they're getting smarter and they're developing their own systems of enforcement (01:01:37):
undefined

Dwarkesh: and laws to just participate in human laws and governments. (01:01:41):
undefined

Dwarkesh: And the metaphor I use here is right now you pay half your paycheck in taxes, (01:01:46):
undefined

Dwarkesh: probably half of your taxes in some way just go to senior citizens, right? (01:01:53):
undefined

Dwarkesh: Medicare and Social Security and other programs like this. (01:01:59):
undefined

Dwarkesh: And it's not because you're in some deep moral sense aligned with senior citizens. (01:02:04):
undefined

Dwarkesh: It's not like you're spending all your time thinking about like, (01:02:09):
undefined

Dwarkesh: my main priority in life is to earn money for senior citizens. (01:02:11):
undefined

Dwarkesh: It's just that you're not going to overthrow the government to get out of paying this tax. And so... (01:02:15):
undefined

Ryan Sean Adams: Also, I happen to like my grandmother. She's fantastic. You know, (01:02:22):
undefined

Ryan Sean Adams: it's those reasons too. But yeah. (01:02:25):
undefined

Dwarkesh: So that's why you give money to your grandmother directly. But like, (01:02:26):
undefined

Dwarkesh: why are you giving money to some retiree in Illinois? Yes. (01:02:29):
undefined

Josh Kale: Yes. (01:02:33):
undefined

Dwarkesh: Yeah, it's like, okay, you could say it's like, sometimes people, (01:02:34):
undefined

Dwarkesh: some people are trying to that post by saying like, oh no, I like deeply care (01:02:37):
undefined

Dwarkesh: about the system of social welfare. (01:02:39):
undefined

Dwarkesh: I'm just like, okay, maybe you do, but I don't think like the average person (01:02:41):
undefined

Dwarkesh: is giving away hundreds of thousands of dollars a year, tens of thousands of (01:02:45):
undefined

Dwarkesh: dollars a year to like some random stranger they don't know, (01:02:47):
undefined

Dwarkesh: who's like, who's not like especially in need of charity, right? (01:02:50):
undefined

Dwarkesh: Like most senior citizens have some savings. (01:02:53):
undefined

Dwarkesh: It's just, it's just because this is a law and you like, you give it to them (01:02:55):
undefined

Dwarkesh: or you'll get, go to jail. (01:02:59):
undefined

Dwarkesh: But fundamentally, if the tax was like 99%, you would, like, (01:03:01):
undefined

Dwarkesh: you would, maybe you wouldn't overthrow the government. You'd just, (01:03:05):
undefined

Dwarkesh: like, leave the jurisdiction. (01:03:07):
undefined

Dwarkesh: You'd, like, emigrate somewhere. And AIs can potentially also do this, (01:03:08):
undefined

Dwarkesh: right? There's more than one country. (01:03:12):
undefined

Dwarkesh: They could, like, there's countries which would be more AI forward. (01:03:14):
undefined

Dwarkesh: And it would be a bad situation to end up in where... (01:03:16):
undefined

Dwarkesh: All this explosion in AI technology is happening in the country, (01:03:21):
undefined

Dwarkesh: which is doing the least amount to protect humans', (01:03:24):
undefined

Dwarkesh: rights and to provide some sort of monetary compensation to humans once their (01:03:28):
undefined

Dwarkesh: labor is no longer valuable. (01:03:36):
undefined

Dwarkesh: So our labor could be worth nothing, but because of how much richer the world (01:03:38):
undefined

Dwarkesh: is after AI, you have these billions of extra researchers, workers, etc. (01:03:42):
undefined

Dwarkesh: It could still be trivial to have individual humans have the equivalent of millions, (01:03:47):
undefined

Dwarkesh: even billions of dollars worth of wealth. In fact, it might literally be invaluable (01:03:54):
undefined

Dwarkesh: amounts of wealth in the following sense. So here's an interesting thought experiment. (01:03:59):
undefined

Dwarkesh: Imagine you have this choice. You can go back to the year 1500, (01:04:02):
undefined

Dwarkesh: but you know, of course, the year 1500 kind of sucks. (01:04:07):
undefined

Dwarkesh: You have no antibiotics, no TV, no running water. But here's how I'll make it up to you. (01:04:09):
undefined

Dwarkesh: I can give you any amount of money, but you can only use that amount of money in the year 1500. (01:04:15):
undefined

Dwarkesh: And you'll go back with these sacks of gold. How much money would I have to (01:04:20):
undefined

Dwarkesh: give you that you can use in the year 1500 to make you go back? And plausibly. (01:04:24):
undefined

Dwarkesh Patel: The answer is (01:04:27):
undefined

Dwarkesh: There's no amount of money you would rather have in the year 1500 than just (01:04:27):
undefined

Dwarkesh: have a normal life today. (01:04:30):
undefined

Dwarkesh: And we could be in a similar position with regards to the future where there's (01:04:31):
undefined

Dwarkesh: all these different, I mean, you'll have much better health, (01:04:36):
undefined

Dwarkesh: like physical health, mental health, longevity. (01:04:39):
undefined

Dwarkesh: That's just like the thing we can contemplate now. But people in 1500 couldn't (01:04:42):
undefined

Dwarkesh: contemplate the kinds of quality of life advances we would have 500 years later, (01:04:45):
undefined

Dwarkesh: right? So anyways, this is all to say that this could be our future for humans, (01:04:49):
undefined

Dwarkesh: even if our labor isn't worth anything. (01:04:54):
undefined

Dwarkesh: But it does require us to have AIs that choose to participate or in some way (01:04:57):
undefined

Dwarkesh: incentivize to participate in some system which we have leverage over. (01:05:05):
undefined

Ryan Sean Adams: Yeah, I find this just such a fast, I'm hopeful we do some more exploration (01:05:12):
undefined

Ryan Sean Adams: around this because I think what you're calling for is basically like, (01:05:16):
undefined

Ryan Sean Adams: what you would be saying is invite them into our property rights system. (01:05:19):
undefined

Ryan Sean Adams: I mean, there are some that are calling in order to control AI, (01:05:22):
undefined

Ryan Sean Adams: they have great power, but they don't necessarily have capabilities. (01:05:25):
undefined

Ryan Sean Adams: So we shouldn't allow AI to hold money or to have property. (01:05:28):
undefined

Ryan Sean Adams: I think you would say, no, actually, the path forward to alignment is allow (01:05:31):
undefined

Ryan Sean Adams: AI to have some vested interest in our property rights system and some stake (01:05:36):
undefined

Ryan Sean Adams: in our governance, potentially, right? (01:05:42):
undefined

Ryan Sean Adams: The ability to vote, almost like a constitution for AIs. (01:05:44):
undefined

Ryan Sean Adams: I'm not sure how this would work, but it's a fascinating thought experiment. (01:05:47):
undefined

Dwarkesh: I will say one thing I think this could end disastrously if we give them a stake (01:05:53):
undefined

Dwarkesh: in their property system but we let them play, (01:06:00):
undefined

Dwarkesh: us off each other. So if you think about, there's many cases in history where (01:06:04):
undefined

Dwarkesh: the British, initially, the East India Trading Company was genuinely a trading (01:06:09):
undefined

Dwarkesh: company that operated in India. (01:06:13):
undefined

Dwarkesh: And it was able to play off, you know, it was like doing trade with different, (01:06:15):
undefined

Dwarkesh: different, you know, provinces in India, there was no single powerful leader. (01:06:18):
undefined

Dwarkesh: And by playing, you know, by doing trade, one of them, leveraging one of their (01:06:23):
undefined

Dwarkesh: armies, etc., they were able to conquer the continent. Similar thing could happen to human society. (01:06:28):
undefined

Dwarkesh: The way to avoid such an outcome at a high level is involves us playing the (01:06:32):
undefined

Dwarkesh: AIs off each other instead, right? (01:06:38):
undefined

Dwarkesh: So this is why I think competition is such a big part of the puzzle, (01:06:40):
undefined

Dwarkesh: having different AIs monitor each other, having this bargaining position where (01:06:45):
undefined

Dwarkesh: there's not just one company that's at the frontier. (01:06:49):
undefined

Dwarkesh: Another example here is if you think about how the Spanish conquered all these (01:06:51):
undefined

Dwarkesh: new world empires, it's actually so crazy that a couple hundred conquistaDwars (01:06:55):
undefined

Dwarkesh: would show up and conquer a nation of 10 million people, the Incas, (01:06:58):
undefined

Dwarkesh: Aztecs. And why were they able to do this? (01:07:03):
undefined

Dwarkesh: Well, one of the reasons is the Spanish were able to learn from each of their (01:07:05):
undefined

Dwarkesh: previous expeditions, whereas the Native Americans were not. (01:07:11):
undefined

Dwarkesh: So Cortez learned from how Cuba was subjugated when he conquered the Aztecs. (01:07:15):
undefined

Dwarkesh: Pizarro was able to learn from how Cortez conquered the Aztecs when he conquered the Incas. (01:07:22):
undefined

Dwarkesh: The Incas didn't even know the Aztecs existed. So eventually there was this (01:07:25):
undefined

Dwarkesh: uprising against Pizarro and Manco Inca led an insurgency where they actually (01:07:30):
undefined

Dwarkesh: did figure out how to fight horses, (01:07:36):
undefined

Dwarkesh: how to fight people, you know, people in armor on horses, don't fight them on (01:07:37):
undefined

Dwarkesh: flat terrain, throw rocks down at them, et cetera. (01:07:42):
undefined

Dwarkesh: But by this point, it was too late. If they knew this going into the battle, (01:07:44):
undefined

Dwarkesh: the initial battle, they might've been able to fend off because, (01:07:48):
undefined

Dwarkesh: you know, just as the conquistaDwars only arrived at a few hundred soldiers, (01:07:51):
undefined

Dwarkesh: we're going to the age of AI with a tremendous amount of leverage. (01:07:54):
undefined

Dwarkesh: We literally control all the stuff, right? (01:07:58):
undefined

Dwarkesh: But we just need to lock in our advantage. We just need to be in a position (01:08:01):
undefined

Dwarkesh: where, you know, they're not going to be able to play us off each other. (01:08:04):
undefined

Dwarkesh: We're going to be able to learn what their weaknesses are. (01:08:08):
undefined

Dwarkesh: And this is why I think one good idea, for example, would be that, (01:08:11):
undefined

Dwarkesh: look, DeepSeek is a Chinese company. (01:08:14):
undefined

Dwarkesh: It would be good if, suppose DeepSeek did something naughty, (01:08:17):
undefined

Dwarkesh: like the kinds of experiments we're talking about right now where it hacks the (01:08:21):
undefined

Dwarkesh: unit tests or so forth. I mean, eventually these things will really matter. (01:08:24):
undefined

Dwarkesh: Like Xi Jinping is listening to AIs because they're so smart and they're so capable. (01:08:27):
undefined

Dwarkesh: If China notices that their AIs are doing something bad, or they notice a failed (01:08:32):
undefined

Dwarkesh: coup attempt, for example, (01:08:37):
undefined

Dwarkesh: it's very important that they tell us And we tell them if we notice something (01:08:38):
undefined

Dwarkesh: like that on our end, it would be like the Aztecs and Incas talking to each (01:08:43):
undefined

Dwarkesh: other about like, you know, this is what happens. (01:08:46):
undefined

Dwarkesh: This is how you fight. This is how you fight horses. (01:08:49):
undefined

Dwarkesh: This is the kind of tactics and deals they try to make with you. Don't trust them, etc. (01:08:51):
undefined

Dwarkesh: It would require cooperation on humans' part to have this sort of red telephone. (01:08:56):
undefined

Dwarkesh: So during the Cold War, there was this red telephone between America and the (01:08:59):
undefined

Dwarkesh: Soviet Union after the human missile crisis, where just to make sure there's (01:09:03):
undefined

Dwarkesh: no misunderstandings, they're like, okay, if we think something's going on, (01:09:06):
undefined

Dwarkesh: let's just hop on the call. (01:09:08):
undefined

Dwarkesh: I think we should have a similar policy with respect to these kinds of initial (01:09:10):
undefined

Dwarkesh: warning signs we'll get from AI so that we can learn from each other. (01:09:15):
undefined

Dwarkesh Patel: Awesome. Okay, so now that we've described this artificial gender intelligence, (01:09:19):
undefined

Dwarkesh Patel: I want to talk about how we actually get there. How do we build it? (01:09:22):
undefined

Dwarkesh Patel: And a lot of this we've been discussing kind of takes place in this world of (01:09:25):
undefined

Dwarkesh Patel: bits. But you have this great chapter in the book called Inputs, (01:09:27):
undefined

Dwarkesh Patel: which discusses the physical world around us, where you can't just write a few strings of code. (01:09:30):
undefined

Dwarkesh Patel: You actually have to go and move some dirt and you have to ship servers places (01:09:35):
undefined

Dwarkesh Patel: and you need to power it and you need physical energy from meat space. (01:09:38):
undefined

Dwarkesh Patel: And you kind of describe these limiting factors where we have compute, (01:09:41):
undefined

Dwarkesh Patel: we have energy, we have data. (01:09:46):
undefined

Dwarkesh Patel: What I'm curious to know is, do we have enough of this now? or is there a clear (01:09:47):
undefined

Dwarkesh Patel: path to get there in order to build the AGI? (01:09:52):
undefined

Dwarkesh Patel: Basically, what needs to happen in order for us to get to this place that you're describing? (01:09:54):
undefined

Dwarkesh: We only have a couple more years left of this scaling, (01:09:57):
undefined

Dwarkesh: this exponential scaling before we're hitting these inherent roadblocks of energy (01:10:02):
undefined

Dwarkesh: and our ability to manufacture ships, which means that if scaling is going to (01:10:09):
undefined

Dwarkesh: work to deliver us AGI, it has to work by 2028. (01:10:13):
undefined

Dwarkesh: Otherwise, we're just left with mostly algorithmic progress, (01:10:17):
undefined

Dwarkesh: But even within algorithmic progress, the sort of low-hanging fruit in this (01:10:19):
undefined

Dwarkesh: deep learning paradigm is getting more and more plucked. (01:10:23):
undefined

Dwarkesh: So then the odds per year of getting to AGI diminish a lot, right? (01:10:26):
undefined

Dwarkesh: So there is this weird, funny thing happening right now where we either discover (01:10:31):
undefined

Dwarkesh: AGI within the next few years, (01:10:36):
undefined

Dwarkesh: or the yearly probability craters, and then we might be looking at decades of (01:10:40):
undefined

Dwarkesh: further research that's required in terms of algorithms to get to AGI. (01:10:44):
undefined

Dwarkesh: I am of the opinion that some algorithmic progress is necessarily needed because (01:10:47):
undefined

Dwarkesh: there's no easy way to solve continual learning just by making the context length (01:10:51):
undefined

Dwarkesh: bigger or just by doing RL. (01:10:55):
undefined

Dwarkesh: That being said, I just think the progress so far has been so remarkable that, (01:10:57):
undefined

Dwarkesh: you know, 2032 is very close. (01:11:01):
undefined

Dwarkesh: My time has to be slightly longer than that, but I think it's extremely plausible (01:11:04):
undefined

Dwarkesh: that we're going to see a broadly deployed intelligence explosion within the next 10 years. (01:11:08):
undefined

Dwarkesh Patel: And one of these key inputs is energy, right? a lot, I actually heard it mentioned (01:11:13):
undefined

Dwarkesh Patel: on your podcast, is the United States relative to China on this particular place (01:11:18):
undefined

Dwarkesh Patel: of energy, where China is adding, what is the stat? (01:11:24):
undefined

Dwarkesh Patel: I think it's one United States worth of energy every 18 months. (01:11:26):
undefined

Dwarkesh Patel: And their plan is to go from three to eight terawatts of power versus the United (01:11:30):
undefined

Dwarkesh Patel: States, one to two terawatts of power by 2030. (01:11:34):
undefined

Dwarkesh Patel: So given that context of that one resource alone, is China better equipped to (01:11:36):
undefined

Dwarkesh Patel: get to that place versus with the United States? (01:11:41):
undefined

Dwarkesh: So right now, America has a big advantage in terms of chips. (01:11:44):
undefined

Dwarkesh: China doesn't have the ability to manufacture leading-edge semiconductors, (01:11:50):
undefined

Dwarkesh: and these are the chips that go into... (01:11:53):
undefined

Dwarkesh: You need these dyes in order to have the kinds of AI chips to... (01:11:56):
undefined

Dwarkesh: You need millions of them in order to have a frontier AI system. (01:12:01):
undefined

Dwarkesh: Eventually, China will catch up in this arena as well, right? (01:12:08):
undefined

Dwarkesh: Their technology will catch up. So the export controls will keep us ahead in (01:12:10):
undefined

Dwarkesh: this category for 5, 10 years. (01:12:14):
undefined

Dwarkesh: But if we're looking in the world where timelines are long, which is to say (01:12:16):
undefined

Dwarkesh: that AGI isn't just right around the corner, they will have this overwhelming (01:12:19):
undefined

Dwarkesh: energy advantage and they'll have caught up in chips. (01:12:23):
undefined

Dwarkesh: So then the question is like, why wouldn't they win at that point? (01:12:27):
undefined

Dwarkesh: So the longer you think we're away from AGI, the more it looks like China's game to lose. (01:12:30):
undefined

Dwarkesh: I mean, if you look in the nitty gritty, I think it's more about having centralized (01:12:37):
undefined

Dwarkesh: sources of power because you need to train the AI in one place. (01:12:42):
undefined

Dwarkesh: This might be changing with RL, but it's very important to have a single site (01:12:47):
undefined

Dwarkesh: which has a gigawatt, two gigawatts more power. (01:12:51):
undefined

Dwarkesh: And if we ramped up natural gas, you know, you can get generators and natural (01:12:54):
undefined

Dwarkesh: gas and maybe it's possible to do a last ditch effort, even if our overall energy (01:13:00):
undefined

Dwarkesh: as a country is lower than China's. The question is whether we will have the (01:13:04):
undefined

Dwarkesh: political will to do that. (01:13:07):
undefined

Dwarkesh: I think people are sort of underestimating how much of a backlash there will be against AI. (01:13:08):
undefined

Dwarkesh: The government needs to make proactive efforts in order to make sure that America (01:13:14):
undefined

Dwarkesh: stays at the leading edge in AI from zoning of data centers to how copyright (01:13:18):
undefined

Dwarkesh: is handled for data for these models. (01:13:25):
undefined

Dwarkesh: And if we mess up, if it becomes too hard to develop in America, (01:13:27):
undefined

Dwarkesh: I think it would genuinely be China's game to lose. (01:13:32):
undefined

Ryan Sean Adams: And do you think this narrative is right, that whoever wins the AGI war, (01:13:34):
undefined

Ryan Sean Adams: kind of like whoever gets to AGI first, just basically wins the 21st century? Is it that simple? (01:13:38):
undefined

Dwarkesh: I don't think it's just a matter of training the frontier system. (01:13:43):
undefined

Dwarkesh: I think people underestimate how important it is to have the compute available to run these systems. (01:13:46):
undefined

Dwarkesh: Because eventually once you get to AGI, just think of it like a person. (01:13:51):
undefined

Dwarkesh: And what matters then is how many people you have. (01:13:55):
undefined

Dwarkesh: I mean, it actually is the main thing that matters today as well, (01:13:59):
undefined

Dwarkesh: right? Like, why could China take over Taiwan if it wanted to? (01:14:02):
undefined

Dwarkesh: And if it didn't have America, you know, America, it didn't think America would intervene. (01:14:05):
undefined

Dwarkesh: But because Taiwan has 20 million people or on the order of 20 million people (01:14:09):
undefined

Dwarkesh: and China has 1.4 billion people. (01:14:13):
undefined

Dwarkesh: You could have a future where if China has way more compute than us, (01:14:17):
undefined

Dwarkesh: but equivalent levels of AI, it would be like the relationship between China and Taiwan. (01:14:21):
undefined

Dwarkesh: Their population is functionally so much higher. This just means more research, (01:14:27):
undefined

Dwarkesh: more factories, more development, more ideas. (01:14:31):
undefined

Dwarkesh: So this inference capacity, this capacity to deploy AIs will actually probably (01:14:35):
undefined

Dwarkesh: be the thing that determines who wins the 21st century. (01:14:41):
undefined

Ryan Sean Adams: So this is like the scaling law applied to, I guess, nation state geopolitics, right? (01:14:44):
undefined

Ryan Sean Adams: And it's back to compute plus data wins. (01:14:50):
undefined

Ryan Sean Adams: If compute plus data wins superintelligence, compute plus data also wins geopolitics. (01:14:54):
undefined

Dwarkesh: Yep. And the thing to be worried about is that China, speaking of compute plus (01:15:00):
undefined

Dwarkesh: data, China also has a lot more data on the real world, right? (01:15:04):
undefined

Dwarkesh: If you've got entire megalopolises filled with factories where you're already (01:15:08):
undefined

Dwarkesh: deploying robots and different production systems which use automation, (01:15:13):
undefined

Dwarkesh: you have in-house this process knowledge you're building up which the AIs can (01:15:19):
undefined

Dwarkesh: then feed on and accelerate. (01:15:24):
undefined

Dwarkesh: That equivalent level of data we don't have in America. (01:15:27):
undefined

Dwarkesh: So this could be a period in which those technological advantages or those advantages (01:15:31):
undefined

Dwarkesh: in the physical world manufacturing could rapidly compound for China. (01:15:37):
undefined

Dwarkesh: And also, I mean, their big advantage as a civilization and society, (01:15:41):
undefined

Dwarkesh: at least in recent decades, has been that they can do big industrial projects fast and efficiently. (01:15:44):
undefined

Dwarkesh: That's not the first thing you think of when you think of America. (01:15:50):
undefined

Dwarkesh: And AGI is a huge industrial, high CapEx, Manhattan project, right? (01:15:53):
undefined

Dwarkesh: And this is the kind of thing that China excels at and we don't. (01:16:00):
undefined

Dwarkesh: So, you know, I think it's like a much tougher race than people anticipate. (01:16:03):
undefined

Ryan Sean Adams: So what's all this going to do for the world? So once we get to the point of AGI, (01:16:07):
undefined

Ryan Sean Adams: we've talked about GDP and your estimate is less on the Tyler Cowen kind of (01:16:11):
undefined

Ryan Sean Adams: half a percent per year and more on, I guess, the Satya Nadella from Microsoft, (01:16:15):
undefined

Ryan Sean Adams: what does he say, 7% to 8% once we get to AGI. (01:16:21):
undefined

Ryan Sean Adams: What about unemployment? Does this cause mass, I guess, job loss across the (01:16:24):
undefined

Ryan Sean Adams: economy or do people adopt? (01:16:30):
undefined

Ryan Sean Adams: What's your take here? Yeah, what are you seeing? (01:16:33):
undefined

Dwarkesh: Yeah, I mean, definitely will cause job loss. I think people who don't, (01:16:36):
undefined

Dwarkesh: I think a lot of AI leaders try to gloss over that or something. And like, I mean. (01:16:39):
undefined

Josh Kale: What do you mean? (01:16:43):
undefined

Dwarkesh: Like, what does AGI mean if it doesn't cause job loss, right? (01:16:43):
undefined

Dwarkesh: If it does what a human does and. (01:16:45):
undefined

Josh Kale: It does it (01:16:47):
undefined

Dwarkesh: Cheaper and better and faster, like why would that not cause job loss? (01:16:48):
undefined

Dwarkesh: The positive vision here is just that it creates so much wealth, (01:16:52):
undefined

Dwarkesh: so much abundance, that we can still give people a much better standard of living (01:16:56):
undefined

Dwarkesh: than even the wealthiest people today, even if they themselves don't have a job. (01:17:00):
undefined

Dwarkesh: The future I worry about is one where instead of creating some sort of UBI that (01:17:06):
undefined

Dwarkesh: will get exponentially bigger as society gets wealthier, (01:17:12):
undefined

Dwarkesh: we try to create these sorts of guild-like protection rackets where if the coders got unemployed, (01:17:16):
undefined

Dwarkesh: then we're going to make these bullshit jobs just for the coders and this is (01:17:26):
undefined

Dwarkesh: how we give them a redistribution. (01:17:32):
undefined

Dwarkesh: Or we try to expand Medicaid for AI, but it's not allowed to procure all of (01:17:34):
undefined

Dwarkesh: these advanced medicines and cures that AI is coming up with, (01:17:42):
undefined

Dwarkesh: rather than just giving people, you know, maybe lump sums of money or something. (01:17:46):
undefined

Dwarkesh: So I am worried about the future where instead of sharing this abundance and (01:17:50):
undefined

Dwarkesh: just embracing it, we just have these protection rackets that maybe let a few (01:17:54):
undefined

Dwarkesh: people have access to the abundance of AI. (01:18:00):
undefined

Dwarkesh: So maybe like if you sue AI, if you sue the right company at the right time, (01:18:03):
undefined

Dwarkesh: you'll get a trillion dollars, but everybody else is stuck with nothing. (01:18:06):
undefined

Dwarkesh: I want to avoid that future and just be honest about what's coming and make (01:18:09):
undefined

Dwarkesh: programs that are simple and acknowledge how fast things will change and are (01:18:15):
undefined

Dwarkesh: forward looking rather than trying to turn what already exists into something (01:18:21):
undefined

Dwarkesh: amenable to the displacement that AI will create. (01:18:25):
undefined

Ryan Sean Adams: That argument reminds me of, I don't know if you read the essay recently came (01:18:29):
undefined

Ryan Sean Adams: out called The Intelligence Curse. Did you read that? (01:18:32):
undefined

Ryan Sean Adams: It was basically the idea of applying kind of the nation state resource curse (01:18:34):
undefined

Ryan Sean Adams: to the idea of intelligence. (01:18:40):
undefined

Ryan Sean Adams: So like nation states that are very high in natural resources, (01:18:42):
undefined

Ryan Sean Adams: they just have a propensity. (01:18:45):
undefined

Ryan Sean Adams: I mean, an example is kind of like a Middle Eastern state with lots of oil reserves, right? (01:18:48):
undefined

Ryan Sean Adams: They have this rich source of a commodity type of abundance. (01:18:53):
undefined

Ryan Sean Adams: They need their people less. And so they don't invest in citizens' rights. (01:18:58):
undefined

Ryan Sean Adams: They don't invest in social programs. (01:19:02):
undefined

Ryan Sean Adams: The authors of the intelligence curse were saying that there's a similar type (01:19:04):
undefined

Ryan Sean Adams: of curse that could happen once intelligence gets very cheap, (01:19:08):
undefined

Ryan Sean Adams: which is basically like the nation state doesn't need humans anymore. (01:19:11):
undefined

Ryan Sean Adams: And those at the top, the rich, wealthy corporations, they don't need workers anymore. (01:19:15):
undefined

Ryan Sean Adams: So we get kind of locked in this almost feudal state where, you know, (01:19:19):
undefined

Ryan Sean Adams: everyone has the property that their grandparents had and there's no meritocracy (01:19:23):
undefined

Ryan Sean Adams: and sort of the nation states don't reinvest in citizens. (01:19:27):
undefined

Ryan Sean Adams: Almost some similar ideas to your idea that like, you know, that the robots (01:19:31):
undefined

Ryan Sean Adams: might want us just, or sorry, the AIs might just want us for our meat hands (01:19:35):
undefined

Ryan Sean Adams: because they don't have the robotics technology on a temporary basis. (01:19:39):
undefined

Ryan Sean Adams: What do you think of this type of like future? Is this possible? (01:19:43):
undefined

Dwarkesh: I agree that that is like definitely more of a concern given that humans will (01:19:46):
undefined

Dwarkesh: not be directly involved in the economic output that will be generated in the CIA civilization. (01:19:49):
undefined

Dwarkesh: The hopeful story you can tell is that a lot of these Middle Eastern resource, (01:19:54):
undefined

Dwarkesh: you know, Dutch disease is another term that's used, (01:19:59):
undefined

Dwarkesh: countries, the problem is that they're not democracies, so that this wealth (01:20:01):
undefined

Dwarkesh: can just be, the system of government (01:20:06):
undefined

Dwarkesh: just lets whoever's in power extract that wealth for themselves. (01:20:08):
undefined

Dwarkesh: Whereas there are countries like Norway, for example, which also have abundant (01:20:11):
undefined

Dwarkesh: resources, who are able to use those resources to have further social welfare (01:20:15):
undefined

Dwarkesh: programs, to build sovereign wealth funds for their citizens, (01:20:21):
undefined

Dwarkesh: to invest in their future. (01:20:24):
undefined

Dwarkesh: We are going into, at least some countries, America included, (01:20:26):
undefined

Dwarkesh: will go into the age of AI as a democracy. (01:20:29):
undefined

Dwarkesh: And so we, of course, will lose our economic leverage, but the average person (01:20:33):
undefined

Dwarkesh: still has their political leverage. (01:20:38):
undefined

Dwarkesh: Now, over the long run, yeah, if we didn't do anything for a while, (01:20:39):
undefined

Dwarkesh: I'm guessing the political system would also change. (01:20:44):
undefined

Dwarkesh: So then the key is to lock in or turn our current, well, it's not just political leverage, right? (01:20:46):
undefined

Dwarkesh: We also have property rights. So like we own a lot of stuff that AI wants, factories, (01:20:52):
undefined

Dwarkesh: sources of data, et cetera, is to use the combination of political and economic (01:20:56):
undefined

Dwarkesh: leverage to lock in benefits for us for the long term, but beyond our the lifespan (01:21:01):
undefined

Dwarkesh: of our economic usefulness. (01:21:07):
undefined

Dwarkesh: And I'm more optimistic for us than I am for these Middle Eastern countries (01:21:10):
undefined

Dwarkesh: that started off poor and also with no democratic representation. (01:21:13):
undefined

Ryan Sean Adams: What do you think the future of like ChachipD is going to be? (01:21:17):
undefined

Ryan Sean Adams: If we just extrapolate maybe one version update forward to ChatGPT 5, (01:21:20):
undefined

Ryan Sean Adams: do you think the trend line of the scaling law will essentially hold for ChatGPT 5? (01:21:25):
undefined

Ryan Sean Adams: I mean, another way to ask that question is, do you feel like it'll feel like (01:21:30):
undefined

Ryan Sean Adams: the difference between maybe a BlackBerry and an iPhone? (01:21:33):
undefined

Ryan Sean Adams: Or will it feel more like the difference between, say, the iPhone 10 and the (01:21:37):
undefined

Ryan Sean Adams: iPhone 11, which is just like incremental progress, not a big breakthrough, (01:21:41):
undefined

Ryan Sean Adams: not an order of magnitude change? Yeah. (01:21:45):
undefined

Dwarkesh: I think it'll be somewhere in between but I don't think it'll feel like a humongous (01:21:50):
undefined

Dwarkesh: breakthrough even though I think it's in a remarkable pace of change because (01:21:53):
undefined

Dwarkesh: the nature of scaling is that sometimes people talk about it as an exponential process, (01:21:58):
undefined

Dwarkesh: Exponential usually refers to like it going like this. (01:22:03):
undefined

Dwarkesh: So having like a sort of J curve aspect to it, where the incremental input is (01:22:07):
undefined

Dwarkesh: leading to super linear amounts of output, in this case, intelligence and value, (01:22:10):
undefined

Dwarkesh: where it's actually more like a sideways J. (01:22:16):
undefined

Dwarkesh: The exponential means the exponential and the scaling laws is that you need (01:22:20):
undefined

Dwarkesh: exponentially more inputs to get marginal increases in usefulness or loss or intelligence. (01:22:23):
undefined

Dwarkesh: So and that's what we've been seeing, right? I think you initially see like some cool demo. (01:22:29):
undefined

Dwarkesh: So as you mentioned, you see some cool computer use demo, which comes at the (01:22:34):
undefined

Dwarkesh: beginning of this hyper exponential, I'm sorry, of this sort of plateauing curve. (01:22:38):
undefined

Dwarkesh: And then it's still an incredibly powerful curve and we're still early in it. (01:22:44):
undefined

Dwarkesh: But the next demo will be just adding on to making this existing capability (01:22:48):
undefined

Dwarkesh: more reliable, applicable for more skills. (01:22:54):
undefined

Dwarkesh: The other interesting incentive in this industry is that because there's so (01:22:57):
undefined

Dwarkesh: much competition between the labs, you are incentivized to release a capability. (01:23:01):
undefined

Dwarkesh: As soon as it's even marginally viable or marginally cool so you can raise more (01:23:06):
undefined

Dwarkesh: funding or make more money off of it. (01:23:11):
undefined

Dwarkesh: You're not incentivized to just like sit on it until you perfected it, (01:23:13):
undefined

Dwarkesh: which is why I don't expect like tomorrow OpenAI will just come out with like, (01:23:17):
undefined

Dwarkesh: we've solved continual learning, guys, and we didn't tell you about it. (01:23:20):
undefined

Dwarkesh: We're working on it for five years. (01:23:22):
undefined

Dwarkesh: If they had like even an inkling of a solution, they'd want to release it ASAP (01:23:24):
undefined

Dwarkesh: so they can raise a $600 billion round and then spend more money on compute. (01:23:27):
undefined

Dwarkesh: So yeah, I do think it'll seem marginal. But again, marginal in the context of seven years to AGI. (01:23:32):
undefined

Dwarkesh: So zoom out long enough and a crazy amount of progress is happening. (01:23:38):
undefined

Dwarkesh: Month to month, I think people overhype how significant any one new release is. So I guess the answer. (01:23:42):
undefined

Dwarkesh Patel: To when we will get AGI very much depends on that scaling trend holding. (01:23:48):
undefined

Dwarkesh Patel: Your estimate in the book for AGI was 60% chance by 2040. (01:23:52):
undefined

Dwarkesh Patel: So I'm curious, what guess or what idea had the most influence on this estimate? (01:23:57):
undefined

Dwarkesh Patel: What made you end up on 60% of 2040? (01:24:00):
undefined

Dwarkesh Patel: Because a lot of timelines are much faster than that. (01:24:04):
undefined

Dwarkesh: It's sort of reasoning about the things they currently still lack, (01:24:06):
undefined

Dwarkesh: the capabilities they still lack, and what stands in the way. (01:24:10):
undefined

Dwarkesh: And just generally an intuition that things often take longer to happen than (01:24:13):
undefined

Dwarkesh: you might think. Progress tends to slow down. (01:24:16):
undefined

Dwarkesh: Also, it's the case that, look, you might have heard the phrase that we keep (01:24:19):
undefined

Dwarkesh: shifting the goalposts on AI, right? (01:24:23):
undefined

Dwarkesh: So they can do the things which skeptics were saying they couldn't ever do already. (01:24:26):
undefined

Dwarkesh: But now they say AI is still a dead end because problem X, Y, (01:24:30):
undefined

Dwarkesh: Z, which will be solved next year. (01:24:34):
undefined

Dwarkesh: Now, there's a way in which this is frustrating, but there's another way in which there's some, (01:24:36):
undefined

Dwarkesh: It is the case that we didn't get to AGI, even though we have passed the Turing (01:24:43):
undefined

Dwarkesh: test and we have models that are incredibly smart and can reason. (01:24:46):
undefined

Dwarkesh: So it is accurate to say that, oh, we were wrong and there is some missing thing (01:24:49):
undefined

Dwarkesh: that we need to keep identifying about what is still lacking to the path of AGI. (01:24:53):
undefined

Dwarkesh: Like it does make sense to shift the goalposts. And I think we might discover (01:24:57):
undefined

Dwarkesh: once continual learning is solved or once extended computer use is solved, (01:25:00):
undefined

Dwarkesh: that there were other aspects of human intelligence, which we take for granted (01:25:04):
undefined

Dwarkesh: in this Moravax paradox sense, but which are actually quite crucial to making (01:25:08):
undefined

Dwarkesh: us economically valuable. (01:25:12):
undefined

Ryan Sean Adams: Part of the reason we wanted to do this, Dwarkesh, is because we both are enjoyers (01:25:14):
undefined

Ryan Sean Adams: of your podcast. It's just fantastic. (01:25:18):
undefined

Ryan Sean Adams: And you talk to all of the, you know, those that are on the forefront of AI (01:25:20):
undefined

Ryan Sean Adams: development, leading it in all sorts of ways. (01:25:25):
undefined

Ryan Sean Adams: And one of the things I wanted to do with reading your book, (01:25:28):
undefined

Ryan Sean Adams: and obviously I'm always asking myself when I'm listening to your podcast is (01:25:30):
undefined

Ryan Sean Adams: like, what does Dwarkesh think personally? (01:25:34):
undefined

Ryan Sean Adams: And I feel like I sort of got that insight maybe toward the end of your book, (01:25:36):
undefined

Ryan Sean Adams: like, you know, in the summary section, where you think like there's a 60% probability (01:25:39):
undefined

Ryan Sean Adams: of AGI by 2040, which puts you more in the moderate camp, right? (01:25:44):
undefined

Ryan Sean Adams: You're not a conservative, but you're not like an accelerationist. (01:25:48):
undefined

Ryan Sean Adams: So you're moderate there. (01:25:50):
undefined

Ryan Sean Adams: And you also said you think more than likely AI will be net beneficial to humanity. (01:25:51):
undefined

Ryan Sean Adams: So you're more optimist than Doomer. So we've got a moderate optimist. (01:25:57):
undefined

Ryan Sean Adams: And you also think this, and this is very interesting, There's no going back. (01:26:01):
undefined

Ryan Sean Adams: So you're somewhat of an AI determinist. And I think the reason you state for (01:26:05):
undefined

Ryan Sean Adams: not, you're like, there's no going back. (01:26:10):
undefined

Ryan Sean Adams: It struck me, there's this line in your book. It seems that the universe is (01:26:12):
undefined

Ryan Sean Adams: structured such that throwing large amounts of compute at the right distribution of data gets you AI. (01:26:16):
undefined

Ryan Sean Adams: And the secret is out. If the scaling picture is roughly correct, (01:26:21):
undefined

Ryan Sean Adams: it's hard to imagine AGI not being developed this century, even if some actors (01:26:24):
undefined

Ryan Sean Adams: hold back or are held back. (01:26:28):
undefined

Ryan Sean Adams: That to me is an AI determinist position. Do you think that's fair? (01:26:31):
undefined

Ryan Sean Adams: Moderate with respect to accelerationism, optimistic with respect to its potential, (01:26:34):
undefined

Ryan Sean Adams: and also determinist, like there's nothing else we can do. We can't go backwards here. (01:26:39):
undefined

Dwarkesh: I'm determinist in the sense that I think if AI is technologically possible, it is inevitable. (01:26:43):
undefined

Dwarkesh: I think sometimes people are optimistic about this idea that we as a world will sort of, (01:26:48):
undefined

Dwarkesh: I collectively decide not to build AI. And I just don't think that's a plausible outcome. (01:26:53):
undefined

Dwarkesh: The local incentives for any actor to build AI are so high that it will happen. (01:26:58):
undefined

Dwarkesh: But I'm also an optimist in the sense that, look, I'm not naive. (01:27:02):
undefined

Dwarkesh: I've listed out all the way, like what happened to the Aztecs and Incas was (01:27:05):
undefined

Dwarkesh: terrible. And I've explained how that could be similar to what AIs could do (01:27:08):
undefined

Dwarkesh: to us and what we need to do to avoid that outcome. (01:27:10):
undefined

Dwarkesh: But I am optimistic in the sense that the world of the future fundamentally (01:27:13):
undefined

Dwarkesh: will have so much abundance that there's all these, (01:27:18):
undefined

Dwarkesh: that alone is a prima facie reason to think that there must be some way of cooperating (01:27:22):
undefined

Dwarkesh: that is mutually beneficial. (01:27:28):
undefined

Dwarkesh: If we're going to be thousands, millions of times wealthier, (01:27:30):
undefined

Dwarkesh: is there really no way that humans are better off or can we can find a way for (01:27:33):
undefined

Dwarkesh: humans to become better off as a result of this transformation? (01:27:37):
undefined

Dwarkesh: So yeah, I think you've put your finger on it. (01:27:40):
undefined

Ryan Sean Adams: So this scaling book, of course, goes through the history of AI scaling. (01:27:43):
undefined

Ryan Sean Adams: I think everyone should should pick it up to get the full chronology, (01:27:46):
undefined

Ryan Sean Adams: but also sort of captures where we are in the midst of this story is like, we're not done yet. (01:27:49):
undefined

Ryan Sean Adams: And I'm wondering how you feel at this moment of time. (01:27:55):
undefined

Ryan Sean Adams: So I don't know if we're halfway through, if we're a quarter way through, (01:27:58):
undefined

Ryan Sean Adams: if we're one tenth of the way through, but we're certainly not finished the path to AI scaling. (01:28:03):
undefined

Ryan Sean Adams: How do you feel like in this moment in 2025? (01:28:07):
undefined

Ryan Sean Adams: I mean, is all of this terrifying? Is it exciting? (01:28:11):
undefined

Ryan Sean Adams: Is it exhilarating? (01:28:15):
undefined

Ryan Sean Adams: What's the emotion that you feel? (01:28:17):
undefined

Dwarkesh: Maybe I feel a little sort of hurried. I personally feel like there's a lot (01:28:20):
undefined

Dwarkesh: of things I want to do in the meantime, (01:28:24):
undefined

Dwarkesh: including what my mission is with the podcast, which is to, and I know it's (01:28:27):
undefined

Dwarkesh: your mission as well, is to improve the discourse around these topics, (01:28:33):
undefined

Dwarkesh: to not necessarily push for a specific agenda, but make sure that when people are making decisions, (01:28:38):
undefined

Dwarkesh: they're as well-informed as possible, They have as much strategic awareness (01:28:42):
undefined

Dwarkesh: and depth of understanding around how AI works, what it could do in the future as possible. (01:28:47):
undefined

Dwarkesh: And, but in many ways, I feel like I still haven't emotionally priced in the future I'm expecting. (01:28:55):
undefined

Dwarkesh: In this one very basic sense, I think that there's a very good chance that I (01:29:00):
undefined

Dwarkesh: live beyond 200 years of age. (01:29:06):
undefined

Dwarkesh: I have not changed anything about my life with regards to that knowledge, right? (01:29:08):
undefined

Dwarkesh: I'm not like, when I'm picking partners, I'm not like, oh, this is the person, (01:29:12):
undefined

Dwarkesh: now that I think I'm going to live for 200, you know, like hundreds of years. (01:29:17):
undefined

Ryan Sean Adams: Yeah. (01:29:20):
undefined

Dwarkesh: Well, you know, ideally I would pick a partner that would, ideally you pick (01:29:23):
undefined

Dwarkesh: somebody who would be, that would be true regardless. (01:29:27):
undefined

Dwarkesh: But you see what I'm saying, right? There's like, the fact that I expect my (01:29:30):
undefined

Dwarkesh: personal life, the world around me, the lives of the people I care about, (01:29:34):
undefined

Dwarkesh: humanity in general to be so different has, it just like doesn't emotionally resonate as much as, (01:29:37):
undefined

Dwarkesh: I, my intellectual thoughts and my emotional landscape aren't in the same place. (01:29:45):
undefined

Dwarkesh: I wonder if it's similar for you guys. (01:29:50):
undefined

Ryan Sean Adams: Yeah, I totally agree. I don't think I've priced that in. Also, (01:29:51):
undefined

Ryan Sean Adams: there's like non-zero chance that Eliezer Yudkowsky is right, Dworkesh. (01:29:54):
undefined

Ryan Sean Adams: Do you know? And so that scenario, I just, I can't bring myself to emotionally price in. (01:29:58):
undefined

Ryan Sean Adams: So I veer towards the optimism side as well. (01:30:03):
undefined

Ryan Sean Adams: Dworkesh, this has been fantastic. Thank you so much for all you do on the podcast. (01:30:07):
undefined

Ryan Sean Adams: I have to ask a question for our crypto audience as well, which is, (01:30:12):
undefined

Ryan Sean Adams: when are you going to do a crypto podcast on Dwarkech? (01:30:15):
undefined

Dwarkesh: I already did. It was with one Sam Bigman-Fried. (01:30:19):
undefined

Ryan Sean Adams: Oh my God. (01:30:22):
undefined

Dwarkesh: Oh man. (01:30:24):
undefined

Ryan Sean Adams: We got to get you a new guest. We got to get you someone else to revisit the top best. (01:30:26):
undefined

Dwarkesh: Don't look that one up. It's Ben Omen. Don't look that one up. (01:30:29):
undefined

Dwarkesh: I think in retrospect. You know what? We'll do another one. (01:30:31):
undefined

Ryan Sean Adams: Fantastic. I'll ask you (01:30:36):
undefined

Dwarkesh: Guys for some recommendations. That'd be great. Dwarkech, thank you so much. (01:30:37):
undefined

Dwarkesh: But I've been following your stuff for a while, for I think many years. (01:30:40):
undefined

Dwarkesh: So it's great to finally meet. and this was a lot of fun. (01:30:43):
undefined

Ryan Sean Adams: Appreciate it. It was great. Thanks a lot. (01:30:46):
undefined

All Episodes

Episode Transcript

Popular Podcasts

Crime Junkie

24/7 News: The Latest

Stuff You Should Know

.css-15opob5{left:0;position:absolute;top:0.8rem;} All Episodes

.css-14f5ked{margin:0;word-break:break-word;display:-webkit-box;-webkit-box-orient:vertical;box-orient:vertical;-webkit-line-clamp:2;overflow:hidden;}Dwarkesh Patel: The Scaling Era of AI is Here

Episode Transcript

Popular Podcasts

.css-r6mb8g{margin:0;word-break:break-word;display:-webkit-box;-webkit-box-orient:vertical;box-orient:vertical;-webkit-line-clamp:1;overflow:hidden;}Crime Junkie

24/7 News: The Latest

Stuff You Should Know

All Episodes

Dwarkesh Patel: The Scaling Era of AI is Here

Crime Junkie