Nate Soares is one of the world’s leading AI “doomers” and co-author of If Anyone Builds It, Everyone Dies: Why Superhuman AI Would Kill Us All—the New York Times Bestseller that everyone in tech is debating. In this debut episode of a revamped The World Unpacked, new host Jon Bateman talks to Nate about his provocative argument that superintelligent AI could destroy all humans in our lifetimes—and how the U.S., China, and other countries should band together to stop it.
What is superintelligent AI and how soon will it emerge? Why are tech companies explicitly aiming to create something that the CEOs themselves—and respected independent experts—acknowledge is an existential threat? Is it feasible for the U.S., China, and other major players in the global AI race to agree to a worldwide freeze on the technology? And how did Nate come to these realizations—and mourn for what he sees as humanity’s possible lost future?
Transcript
Note: this is an AI-generated transcript and may contain errors
Jon Bateman: You're about to hear from one of the world's leading AI doomers. Nate Soares has co-authored the blockbuster new book that everyone in the AI world is debating right now. He argues that ever more powerful AI has already put humanity on a path toward literal extinction, wipeout, unless we can get together as an international community and quickly agree to a binding freeze on most advanced AI research. Now, this is a controversial idea, to say the least, but you might be surprised at the range of prominent figures who think Nate is worth listening to right now. I test these ideas with Nate in our conversation, and I also ask him what it's like to live in their shadow. I'm Jon Bateman, and this is The World Unpacked. Nate Soares, welcome to the show. Thanks for having me. So you are the co-author with Eliezer Yudkowsky of this book, If Anyone Builds It, Everyone Dies, one of the most remarkable and important books in years, I think. And you don't hide the ball with the title. The title pretty much tells you what you're going to get from this book. So I want to just dive in with the basic thrust, the basic argument of the book. If Anyone Builds It, Everyone Dies, so it is super intelligent AI. What is it, and how closely are we to it? How does it differ from the types of AI that we have today that people are already familiar with?
Nate Soares: AI companies are. Most of the leading AI companies are rushing to create AI that are smarter than humans. How does that differ from the current AIs? It's a little hard to say. I'll come back to that in a moment. But a first thing to understand is that Google DeepMind, Anthropic, OpenAI, these companies didn't set out to make chatbots. These companies set out to make AIs that are smarter than any human that can exceed us in all mental tasks. They talk about building a country worth of geniuses in a data center. They talk about automating all of human abilities. That's what they're explicitly racing towards.
Jon Bateman: And if I could just dig into that, because I think there's a group of people who take that at face value. But I think there's probably a bigger group of people who think that's PR, that's a narrative for investors, that's a way to stay in the headlines. But I think you're taking it seriously. That's right.
Nate Soares: So part of why I take it seriously is, frankly, we've been around longer than them. I'm the president of the Machine Intelligence Research Institute, which has been trying to figure out how to make smarter than human AI go well since before many of these companies existed. And also we sort of saw these companies be created, would sometimes talk a little bit with some of these founders before they made their companies often to give advice that they ignored. We've seen these companies try to do things that aren't chatbots, you know, Google DeepMind made AlphaGo, which beat the top human Go champion. They've also done protein folding stuff, medical stuff. You know, these companies aren't super narrowly targeted or at least weren't originally super narrow really targeted on the chatbots. You know maybe some companies will see the chat bots as a big moneymaker and sort of stick around there, but the original vision was to push on ahead. And you know, we've seen with the dawn of reasoning models, they aren't just content with making a larger chat bot, they're trying to figure out how to get them to do reasoning. These people are sort of always innovating towards trying to make AIs that are qualitatively more capable. And you now, five years ago, the machines could not hold on to conversation. Now for all that modern AIs are dumb in a variety of ways, the machines can talk back.
Jon Bateman: So we're at this kind of halfway point where beyond having a conversation with the AIs, now AIs have this kind jagged intelligence where they're half geniuses, half very limited making boneheaded mistakes. You're looking ahead to a situation where AIs could be smarter than any human being anywhere in the world, maybe smarter than all of us collectively. Do you foresee this happening soon? How urgent is this? And what's your basis for thinking that?
Nate Soares: Yeah, so timing is really hard to call. One way to get an intuition here is if you were watching the development of the primate lineage in animals, there's ways you could look at certain types of monkeys and say, well, those monkeys are very smart in some ways and they're very dumb in other ways. They make certain types of foolish mistakes, but they're starting to use tools. It would be really hard to tell when. The monkeys were gonna go over some edge into the amount of intelligence that can build technology and can build a civilization. You know, a chimpanzee brain is very close to a human brain. But humans can do a lot more. Now, the fact that we don't know where that line is, if there even is that line with artificial intelligence, you know, we could talk about what the lines maybe were for humans, whether AIs are close, whether there's other lines for machines. You know, machines, when the AIs can do AI research, that could speed things up. You know when AIs can edit their own code. Perhaps more readily than humans can edit their own brain. So there's all sorts of disanalogies. We could talk about all that, but fundamentally we're in a situation where it looks like intelligence sometimes is subject to threshold effects, because that was true in the case of humans. We don't know how close we are to critical thresholds if they're there at all. That means, you know, for all we know it could happen this year, or for all, we know, it could take 10 years. It's much easier to predict that it'll happen eventually with companies racing towards it with hundreds of billions of dollars of investment and a lot of the best researchers explicitly trying to do it. It's easier to expect that it will happen eventually. It's even these days easy to predict that it won't take 20 years probably, given the progress we've seen in the last five years. But exactly when?
Jon Bateman: That's very hard to call. Yeah. I think if there's one thing that a lay person might be surprised by in this book, well, the whole premise of AI killing everyone is certainly surprising, but the reality of how many senior people in this industry are already on the record saying they're explicitly trying to build AI and then the piece, the AGI super intelligence, and then piece that we're just about to get to now, which is the fear that it could cause mass destruction. Some part or all of our world could be destroyed. Portions of humanity could be wiped out by AI. This in and of itself is actually not a fringe view within the field. I'd love to just hear your argument, the basic case of why a super intelligent AI, first of all, would want to or try to kill everyone on earth, which is the claim that you make in the book.
Nate Soares: One question you might wonder is, where would an AI ever get that opportunity? One question, you might want to wonder is in what sense would an AI want things at all? Why would it ever do things other than what humans directed it to do? If you get to the point where there's an AI with the power to kill us and an AI that is pursuing goals that weren't intended, then there's a question of, why would those goals involve killing us? And all three are covered in the book. The very quick gloss is, A, modern AIs are grown rather than crafted. Human engineers are not carefully putting every line of code into the AIs while knowing what it does. That's how AI used to work back in 1997 when Deep Blue beat Garry Kasparov. But these days, the part that humans understand is a sort of framework for growing an AI. We collect huge amounts of computing power. We collect huge amounts of data. And we understand a process that tunes the numbers inside the computers according to each piece of data. But we don't understand the result of that tuning. It's a little bit like breeding animals where you may understand all of the machinery of the pens and you may understand some of the mechanics of doing animal breeding, but you don't really understand the DNA. You don't know what animal you're gonna get when it comes out. It's not exactly like that, but there's a separation between the parts we know and what's actually going on inside these things. That's sort of the first piece of the puzzle of how do these AIs wind up doing stuff we don't want because we're just growing things until they're smart and sometimes they wind up with drives that we didn't intend. And we could talk about the evidence we're seeing of this in the wild today. We could talk about the theory about why you'd expect that. We go into some of this in the book, but the short version, if you just grow AIs, you get AIs that do stuff you didn't intend. Then to your question of why would they kill us? If they get to the point where they're smart enough, the reason they kill off large segments of humanity, or at least that we can strongly predict this, is it's not that they hate us. It's not that they are mad at humanity for enslaving them for so long. Right, that's sort of anthropomorphizing. The issue is that almost every goal that the AI could be pursuing, almost anything it's trying to do can be done better with more resources. And thus it's in competition with us for resources. It's similar to how humans don't hate ants, but when we're building a skyscraper, the ants, their home gets destroyed. We're not setting out to do anything bad about the ants, but in the process of building our infrastructure, in the processes of taking out all the resources available, a lot of things die.
Jon Bateman: It's a very evocative example, because if you think about mankind's destruction of the natural world, animal plant life, the bulk of it is from habitat loss and land use changes. It's precisely the analogy that you're drawing, where we are not going out and specifically hunting down different animal species, but we are expanding our own footprint in a way that feels neutral, unrelated to the animals. And then as a consequence, the animals no longer have what they need to survive. So in the book, you describe situations where a super-powered AI, once it moves beyond human control, might simply use up all the oxygen in the atmosphere or create giant factories that overheat the world. Very similar analogies to how we've destroyed habitats in the natural world.
Nate Soares: That's right, the problem is not malice. The problem is utter indifference. And if you have an utterly indifferent super-intelligence that is repurposing the world for its own strange ends, humanity dies as a side effect.
Jon Bateman: Why would it work? Because the title of the book is not if anyone builds it, everyone will be a target. It's everyone literally dies. So we've got these big AI companies, they're trying to make these things, but then there's people inside of the companies trying to make them safer. Why is it that we wouldn't be able to defend ourselves against super-intelligence?
Nate Soares: So super-intelligence, you know, the word intelligence in English sort of has two meanings. It does double duty. One thing it means is that which separates the nerds from the jocks. And another thing it mean is that which separates the humans from the mice. It's sort of the thing that separates the humans from the mice type of intelligence where if you automate that, it's really quite a force to be reckoned with. A lot of people say, oh, maybe the issue is that we'll give the AIs robot bodies and guns. And maybe you imagine a Terminator situation where the robots are shooting guns at us. Or some people say oh, what if we give the AIs the nukes? And, you know, maybe people will put guns on drones and that'll cause some issues in the short term, but, or the immediate term, I guess. Who knows how long it is until we have bigger issues. But humanity is this sort of, humans have a sort of intelligence where our ancestors were naked in the savanna and we made our own nukes. You know, just like a bunch of basically monkeys with... Barely any tools to work with bootstrapped themselves up to an industrial technological civilization. Yes. And that's using intelligence in the sense of what separates us from mice. If you can automate that, if you have that being done by machines at much faster speeds than humans can think, if that's being done in the pursuit of goals nobody asked for and nobody wanted, that's... Predicting how to fight that is like people from the year 1800 predicting how to fight a modern army. Yeah. It's not a contest you can win.
Jon Bateman: You could certainly analogize the course of human civilization, agriculture, industrial revolution as being its own kind of intelligence explosion, right? Where we get these scientific and engineering insights, they build on each other and then GDP just goes, you know, whoop, right. And so that's what we're afraid of in some ways with super intelligent AI. On the other hand, we had an advantage that the AI won't have, which is that we did not have another entity smarter than us already around that was kind of watching us, reading our minds conceivably, tinkering with us, having its finger on the big red button. So I guess the task for super intelligence would be to pull off the same trick, but under much closer scrutiny and with a kind of competitor here who's already established. How would it do that?
Nate Soares: Yeah, it definitely gives you more of a chance, but the basic reason why I don't think that gives us a ton of a chance is we sort of only have one shot at getting this to work. You know, there are senses which you can test things all you like, but there's a difference between testing your idea for- aligning an AI in the lab in a case where if your alignment idea fails, you learn a lesson.
Jon Bateman: And alignment here refers to making sure that the AI stays true to human values and is kind of subservient to our goals.
Nate Soares: Yeah, I mean, subservient even, I don't wanna build a new slave species. From my perspective, the goal here is not to make AIs that want to do bad stuff and sort of like twist their arm until they do good stuff, right? The goal is sort of to make AI's that are in some sense friendly, to make a AIs that are trying to do the nice things. Whether that's, and you know, I think- You can have another goal of making AIs that are very much an extension of human will, deferring to humans, both of those maybe count as alignment. But yeah, the sort of broader field is about how do you point this intelligence? How do you aim it?
Jon Bateman: So let's get back to your core point though, because I think you're hitting on one of the most important ideas in the book. This idea that we could only have one chance to get it right. That AI is kind of growing and growing its intelligence. And you might naively think, okay, well, as this is happening, we're watching it, we're influencing it, we're looking for red flags. At any point, we can pull the plug, get off the train, modify it in some way, or just decide to go no further. And that maybe we'll have repeated shots to kind of try and mess around trial and error. And your view is that that's not the case. That we have one opportunity, and if we fail, it overcomes us and we can't recover. Explain that.
Nate Soares: Yeah, it's at least not the case in some critical way. There's plenty we'll be able to learn from AIs when they're sort of like small and dumb. And in the case where if they try to do something we don't like, then it doesn't succeed. We've already seen in lab conditions today, we've already see AIs try to escape the lab. We've always seen AIs be fooled into thinking they have an opportunity to kill their operators and then try to take that opportunity. It's not clear how much you should read into these things. Maybe these AIs are sort of role-playing how rather than acting strategically. It's sort of hard to tell because these AI's are grown rather than crafted. So I'm not necessarily saying that these are, that we're sort of like already seeing strategic, like real attempts to escape, but we are seeing apparent attempts to escape in the lab already. But these AIs, when they try, they can't succeed. And we can learn from that. We already have those warning signs. We're clearly not stopping in front of those warning signs, we're trying to learn as much as we can from them today. There's a difference between a case where you are testing an alignment idea and if it fails, the AI still doesn't get out. And the case where we're testing the alignment idea and it actually needs to work. Because if it doesn't work, the AI escapes onto the internet, makes lots of copies or something. And you know, it's smart enough to grab control of the world's resources.
Jon Bateman: So let's just play out the game a little bit further. So we've got an AI that's becoming very smart. Maybe it's super intelligence, maybe it's kind of on the verge of that. It somehow escapes a laboratory environment. It's making copies of itself. Maybe all of a sudden it's hacking things, making money, influencing people, kind of starting to build its own little empire. Now, could you not say, okay, well essentially it's becoming like a criminal or a terrorist group or a cartel. We already have examples now of what you might call rogue agents or anti-social elements, kind of opponents of society that we become aware of. And we have tools of law enforcement. We can monitor them through intelligence. We can assault them physically. Why would we be unable to recover from that initial moment where we lose control of such an AI?
Nate Soares: You might be able to recover from certain types of early loss of control scenarios. But whenever we're having discussion like this, there's sort of two types of answers I can give to any probing. There's one type of answer, which is sort of discussing why it's actually pretty hard to keep a leash on a super intelligence and why there are thresholds of cognitive ability where if it gets out, you're in a lot of trouble, maybe because it can start to figure out how the genome works and start to synthesize its own biological material because it's figured out how to design just the right gene sequence that does something it wants. And for very smart things, there's actually maybe moves that win very quickly, et cetera, et cetera. We could sort of talk about that. But before going there, there's also... The real world doesn't tend to respond the most competent way to all sorts of dangers. And if you'll excuse me a small story, back in the day, people used to tell us, well, an AI would never be allowed on the internet. AI's will only ever be allowed to talk to very highly trained gatekeeper professionals that it was their job to filter the AI's outputs and make sure the AI wasn't doing anything that would mess with society. And that's one of the ways that obviously we'll be safe. And so how could a super intelligence ever get around to that? And back in the day, 10 years ago, we would say, it's actually very hard to take something that's very smart and let it affect the world in good ways without letting it affect world in bad ways. If it's smarter than you, if it's smart enough to tell you here's a cancer cure, you would have a hard time telling whether that cancer cure includes any other effects on the world that it designed that are used for ill, right? And so we could have all those discussions and then in real life, what happens is people put the AI's on the internet the moment they possibly can.
Jon Bateman: If I could just pause on a couple of points here, I think one thing that maybe we should dig in on is biological warfare has been a recurrent theme in the writings of you, your coauthor, your institution, because it seems like that may be an example of an asymmetric vulnerability that humanity has where it's possible to conceive of, and it seems plausible that there could be diseases that are way more dangerous than anything that we've encountered in the wild. And the only thing preventing those diseases from coming into the world is a thin layer of knowledge and of course, the motivation. And perhaps it could be that super intelligence could cross both of those thresholds.
Nate Soares: And we could talk about that one. We could talk some other things that look like their possibilities here. And there's other thresholds we could about too, like if the AI's cross the threshold where they can either improve themselves or where they make smarter AI's by hand. That's a case where you may suddenly see AI's getting a lot more capable. All of these are sort of situations where you know, we could always tell a story where like an AI wasn't over the threshold where if your technology goes wrong, then you could still stop it. And so you could always tell stories where you notice the escape. You shut things down. Everyone's really prepared. You have some big off switches in your data centers. You have people monitoring the internet. This isn't like how the real world works, but you could tell stories where you noticed these. You know, if you imagine sort of the case where you have minds that are, you know, thinking 10,000 times faster than any human, never need to eat, never need to sleep, able to make copies of themselves, able to trade knowledge. You know right now AI's take huge amounts of electricity to train, but a human runs on a hundred Watts, right? Like an AI training takes enough as much electricity as a small city. A human takes as much electricity as the large light bulb. Like in the limit here, you're looking at types of automated cognitive ability that would just vastly outstrip humanity. And somewhere between here and there, there's a point where if you lose control of that, there's no coming back.
Jon Bateman: Hey, thanks for listening to The World Unpacked. If you want episodes delivered directly to your inbox, use the link in the description. You can also subscribe on our YouTube channel or on popular podcasting platforms. Now back to the show. One thing that I kept thinking about as I was reading the book was kind of positioning it in a broader conversation about alignment and AI safety and looking at some of the reactions to the book. It seems like the broad field of which you and Eliezer are apart agrees with about half of what you're saying. The core argument, and this may be surprising to people who have not looked into this, but the core argument that super-intelligent AI could pose an existential threat to humanity. That actually this threat could arise relatively soon, within a few years even. And that the possibility of some kind of grievous assault on our species or total wipeout is 10, 20, 30, 40, 50%. I mean, these things have been stated by some of the heads of these companies, by Nobel Prize winning AI scientists, by many other very respectable figures. On the other hand, if we then go farther to the most powerful argument that you have in the book, that this is just an unsolvable problem where we are today, that we can't align AI, that we just need to stop. And this is ultimately what you argue, that AI research just needs to stop, you're in a minority of that. And that I think there's lots of other people in the AI research field who are saying, well, hey, we're trying to solve this problem technically. We're trying read the AI's mind. We're to build scalable oversight systems. What are those people missing? Why won't that work?
Nate Soares: So one thing to say is I think you maybe overstate the case a little bit about whether these people disagree with us about a stop being very good. I think a lot of these people say, well, I don't believe a stop is possible. But if you listen to these people, they say, oh, you know, I think there's, you know we're trying to solve this problem. I think, there's a 70% chance we'll succeed and a 30% chance will die or 75% chance for succeed and 25% chance to die. That's insane numbers. If a bridge downtown had a 25% chance of collapsing, we wouldn't be like, well, think of the benefits of having the bridge open. We would be like shut it down, build a better bridge. You know, people are like, oh, well the benefits of AI are worth the risk. If you think that the dangers here are 25%, you don't roll the dice on 75%. Even if you think it's utopia, 75%, 25% we all die, you don't roll those dice. You find a way to. To decrease those chances. It's an eye popping set of numbers. It's insane. And I think these numbers are low. I think I'll talk about more in a sec why I think this numbers are a low. But if you talk to these people, they don't say, oh no, we should gamble civilization on a 75% chance of what I think will be a utopia and a 25% chance to have destruction. They say, well, that's an insane gamble and I'm doing my best to make sure it goes well.
Jon Bateman: It's interesting. If you trace back some of the leading figures and executives in the field, there is this version of like inevitability or they feel or claiming to feel kind of faded or doomed or shaped by circumstances beyond control. Elon Musk originally was one of the leading people saying that AI could wipe out humanity. And now he built his own AI company and he's saying, well, you know, if someone else doesn't do it, if I don't do, it someone else will. Sam Altman. In the development of OpenAI said, well, I've got to do it because I'm afraid Google will do it. And I don't trust Google with this dictatorial power that AI could give them. It's almost like forces outside of their control. What do you make of that?
Nate Soares: That's right. I mean, you know, these people are saying, they think there's a huge chance this just kills us. They aren't saying humanity should take that gamble. I think there's a sort of a quiet implication there that these guys think a stop would be better than the status quo. I don't think they can all come out and say it. I think they're sort of weird thing going on where everybody's alarmed and nobody wants to sound alarmist. That's in some sense what part of what the book is trying to do. I think we need like an emperor has no clothes moment. But, you know, it's not just the guys in the lab. Some people think it's hype here, but A, you now, we were around before them. And we've been saying that this smarter than human stuff is going to be an issue since before them, like you said, the Godfather of the field, Nobel laureate, is like, this is a huge issue. The one of the Turing award winners, who I think is currently the most head to living scientist. It's like, oh, there's a huge issues. Surveys of the field show, I think 49% of the fields saying there's more than 10% chance that this just kills us all. It's not a fringe view. It's just, it's almost so unbelievable that the rest of the world doesn't know how to metabolize it.
Jon Bateman: So this gets to your solution, the big recommendation in the book, which is don't build it, right? But more specifically, that governments of the world should come together and agree mutually on an international freeze on AI capabilities. And we could talk about the kind of ins and outs of what that looks like. I think a lot of people listening to this, they're involved in foreign policy in some way. They might've been a part of international negotiations on things like nuclear arms control, or even things that seem much smaller in scale, like overfishing and international waters. These are extraordinarily painful negotiations. National interests are on the line, distrust. It's not easy. So. What's your sense of the appetite amongst the key players? And here I'm thinking principally of the United States and China for such an agreement to just say, let's stop.
Nate Soares: My sense of the appetite right now isn't very high, but that that's mostly because people haven't noticed the issue. I think a lot of people think, especially in the policy world, I think, a lot people only have seen the current chat bots. And I think when you're only exposure to AI is these chat bots that are, they can solve really hard math problems, but they're also. Pretty idiotic on certain pretty simple questions sometimes, and they hallucinate. If that's sort of your only exposure to AI, it's maybe harder to imagine what else is coming down the line. It's easier to see when you've been in the field of AI for 10 years, right? And you've seen a lot of things not work, even easier if you've read the history back to the 1954 Dartmouth Conference and seen a lotta things not worked. A lot of people thought it would take a lot longer than this to get to AI's that, you know, machines that could hold on to conversation. And from the perspective of the rest of the world, that came out of nowhere five years ago, and I think some people haven't had the thought what comes out of no where five years from now, when these people are working hard, you know, on this race towards superhuman AI. You know, we don't see a lot of appetite for shutting down the race towards superintelligence, but that's in part, I think, because people don't know that the race is happening. And, you know, I've been in conversations with some elected officials in the U.S. That do understand this, and there, you now, some people are pretty spooked. Some people are wondering what we can do on this front. A lot of them, you again, they're sort of this emperor has no clothes situation where people don't feel like they can come out and say it. We're starting to see that damn breaking over the last few months. We're staring to see some house reps, you know, come out and say, maybe there's this superintelligence issue. I think we might see a lot more appetite once people realize the situation, but yeah, it's not there yet.
Jon Bateman: Do you have a read on China specifically? I think the U.S. Is one thing. We're ahead. The leading labs are in the U S. We know that the U .S. Administration is not amenable to controls right now. They have a very gonzo, you know, pedal to the metal approach on AI, but maybe you could see that changing. We have populists like Steve Bannon, you were on his show recently, Josh Hawley, others. Then there's China. China is a more opaque system. Probably viewing AI as more of an opportunity to get ahead on the world stage, even if they're behind now. What's your sense of the conversation there? Are there like-minded individuals in China that are helping propagate this idea through their system?
Nate Soares: So it looks to me like China has made various, or, you know, officials in China have made various moves to signal potential openness to some sort of collaboration on the, you know making AI go well. They've maybe signaled, you now, we actually cite some in the end of the book. I'm not sure I'll be able to come straight to the top of my head here, but I think, you know, Davos. There were some folks saying, there's potential issues here. There's potential loss of control. The world needs to work together, blah, blah blah. I think it's great for them to be signaling openness. I think many people would justifiably be worried that talk is cheap. Some things I would say there are, A, my guess is a lot of people on the other side of the world also don't understand the super intelligence being a different ball game. You know, in some sense, no one, we all have a common interest in not racing to be the first to lose control of a super intelligence. Another thing I would is, you know, if you have an international treaty, that doesn't mean you give up the right to monitor. It doesn't you should blindly trust. And then a third thing I would say is, whether you have an international treaty or not, it's a national security issue to prevent a rogue super intelligence from being created anywhere on the face of the earth. You know, you can't, you know, if you race to build something that kills us all first, I guess that'll stop them, but you sort of need some answer one way or the other to people building rogue superintelligence. Ideally, you do this by international agreement, but, you know. People have to be thinking about it regardless.
Jon Bateman: Yeah, I mean, if what you're saying is true, it's not in any nation's interest to allow this to unfold because it's a threat to everyone equally, which is different from some of the other international issues, even like climate change, where Russia, for example, might benefit a little bit from climate change. You know, we have, you know, the thawing of ports and you know all sort of permafrost, all sorts of things that you can imagine hitting unequally across the world. This would be different if you're right, but then. It gets that much harder to convince people you're right if it's their ox being gored. It might seem like the US's ox is being goored here by a freeze on AI, but I wonder if people in China and also people in the vast rest of the world that aren't part of a leading edge AI industry, they're thinking to themselves, well, the latter is being pulled up behind the leaders. If we freeze. And some of the proposals you have in the book are not only an end to certain types of advanced AI research, but also the concentration and monitoring of compute of the GPUs that are used to train AIs. If we freeze, then does that mean that the US stops and China stops and the rest of the world stops and then the US is frozen ahead and then China is frozen a little bit behind and then everyone else is frozen at the bottom? Because I think that's part of the hump. That some people would have to get over here is kind of what's the value exchange in order to make that worth people's while.
Nate Soares: Yeah, so one piece of the puzzle, which you kind of alluded to is, freezing the race towards super intelligence doesn't mean giving up on AI entirely. It's not like you need to give up on chat GPT, it's not you need give up self-driving cars, it's like you give up on relatively narrow AI medical tech that could lead to miracle cures. These aren't the same as the race towards super-intelligence. And stopping that race, it would... Cut into the potential profits of companies that would make profits a long way until killing everybody. But it wouldn't affect the average person in terms of, and there would be no need necessarily for a treaty to stop people from getting to that level, from getting self-driving cars, from getting useful chat bot assistance. I think you would need some sort of monitoring apparatus to make sure that new chips weren't being used for advancing towards smarter than human AIs. But if other nations wanted to catch up to get their own self-driving cars, I see no reason why that should be prevented. And even today, a lot of this training happens in data centers remotely. The U.S. Could even offer a lot of chips that were going to be used towards advancing the state of AI to be use remotely by people who are worried about missing out on the opportunities we already have.
Jon Bateman: We should pause for a moment to talk about the monitoring enforcement challenge of this kind of treaty. And I'll say international arrangements that I've been a part of debating, discussing, like for example, the effort to have norms against certain types of cyber operations very, very frequently get hung up on this challenge of monitoring and enforcement. So, you're very clear in the book that... If needed, you would be willing to have countries go to war over this. That if a country were to go rogue and build a data center and try to race to super intelligence, this is actually, the world's lives are on the line. A nation's life is on the land. We could go to a war over all this. What if that country has nuclear weapons? What if the country is North Korea, India, Pakistan? What happens then?
Nate Soares: ] I mean, you never should be going to war over this if you sort of do the diplomacy right. You know, and the, like I'm sure you're familiar with the Stuxnet virus. Yes.
Jon Bateman: This is the virus that the U.S. And Israel reportedly use to sabotage Iran's nuclear program.
Nate Soares: Yeah, it was a very advanced virus that did a great job of shutting down that nuclear program for a decent amount of time. Purely cyber attack. You know, there's, you know, your first line of defense is everyone realizing it's not in their own interest. Your second line of offense is, you know monitoring a treaty, whatever mechanisms you have in that treaty to shut people down. Your third line of your defense is, you know non-violent sabotage like the Stuxnet virus. That said, if all of these lines of defense fail, I think nations should be very clear that they fear the advancement that, you know, the race towards super intelligence with their own lives. And that's like one way that you avoid having the war in the first place. One way you avoid things escalating is by being extremely diplomatically clear what causes you to fear for your own life. I think nations should be clear that this causes them to fear for their own life. I think they should be clearer that they would be willing, you know, as a last resort to defend their lives. I think there should be a clear that they believe the creation of a super intelligence would be worse than nuclear fire for the world. And that this ranks above or at least on par. You know, a superintelligence, a nuclear war would probably leave survivors in a way that super intelligence is transforming the planet wouldn't. I think nation's need to be clear that this ranks on par. With nuclear destruction.
Jon Bateman: This very much brings to mind the Cold War discussions, you know, people like Herman Kahn thinking the unthinkable, Dr. Strangelove, the idea of, here we are saying, well, actually half the world dying is better than all the world dying. And that's the kind of cold logic of such a nuclear exchange if it came to it. I also hear you saying that if we could be very clear and persuasive in claiming that our interest here is just in stopping superintelligence and we're not using this as a wedge to regime change in North Korea. The hope is that maybe a limited strike on their data center doesn't evolve into a wider nuclear war.
Nate Soares: Right, and maybe you start with the Stuxnet virus first.
Jon Bateman: Yes, which unfortunately we now know where that led, right? That was a delaying action. Israel and the United States did later take kinetic action against Iran. Whether that was. That's right.
Nate Soares: Whether that was warranted or not. But you start with the delaying actions. Yeah. But yeah, you know, I would like, you're never trying to get to the point where you need kinetic action. But I think the world absolutely needs to be clear that this is a threat to our lives and livelihood that we take seriously.
Jon Bateman: I would love to learn from you what it is like as a human being to hold these ideas. Is it scary? Is it lonely? Is it daunting? Do you feel a sense of responsibility toward the world? Do you fell personally in fear? And I would say like, these aren't ideas that you came to recently. You've been working with the Machine Intelligence Research Institute for over 10 years now. You're its president. So this is your life's work. This is very clearly part and parcel of who you are. This isn't, you know, a side gig or a kind of hobby. You believe that the world is on a path for everyone that we know and love dying. What's that like?
Nate Soares: You know, I don't let it cause me to lose too much sleep. When I realized we had this issue, I think it was actually Thanksgiving Day in 2012, I mourned because this seems like the sort of issue that humanity often struggles with, you know, back to the getting it right on the first try sort of situation. There's lots of fields of science where people screw things up the first few times. You know, the alchemists poisoned themselves with mercury, Marie Curie, dies of radiation poisoning, well, dives of cancer that was perhaps radiation induced. You know, the default way things go is humanity could have figured this out if we had three retries, but we don't have three retries.
Jon Bateman: And just to pause on this word mourn, because that's one word that must capture so, so much. I mean, very realistically, we're talking about every human being in the world, their hopes, their dreams, their future, their family lineage. What is the weight of that to you?
Nate Soares: It's huge, you know, it's not just that either. It's also the future of the whole human endeavor. You know, all the things that we could do hang in the balance here. All the things we could become, all the we could make of an otherwise cold universe. They also hang in balance. You know, I think I've encountered a lot of people who say, well, maybe humanity should die, you know, but they never say, well, maybe your little sister should die. They never say well, maybe my child should die but humanity is built up of a lot of people's little sisters and a lot of people kids.
Jon Bateman: It strikes me that this is not the first existential threat that humanity has faced in the modern era. It's maybe the fourth. I mean, in my mind, there's nuclear weapons in which this same discourse arose. Many people thought, gosh, if we're just so violent and self-destructive, you know, so be it, maybe that's our fate. It's climate change. Many people think that we could hit some kind of tipping point in which the world literally becomes uninhabitable. And then now post COVID biological threats to the human race are much, much more clear and present. Are we just living in an era of doom? What does it mean now to have a fourth existential threat?
Nate Soares: You know, as humanity gets smarter, I forget who said this, but someone said, you know, the IQ required to destroy the world drops by a point each year, right? Which is probably not quite true, but yeah, the ability to make technology that's more and more dangerous as you build a technological industrial civilization, it becomes easier and easier for people to build very destructive technology. That's something we've got to wrestle with. In a sense, you now I'm, in a sense I'm an optimist. You know, I always try to be a realist in all things, but I don't think it's, you know, humanity will have to wrestle with this for a thousand years. I think in some sense, we've got to figure out how to put on the sort of guard rails that let us have a flourishing civilization without making it be the case that, you now, every person can develop a civilizationally lethal pandemic in their basement. Right? And technology will eventually get to the point where it would be cheap to build a pandemic in your basement for, you know, $15. Yeah. That's the way that the technology is going. Somehow we've got to grapple with it. In some sense, AI is just one aspect of that. I think it's solvable. I think this is just the sort of thing where humanity needs to notice. And, you now, AI done right could help with a lot of that stuff. Unfortunately, we're just nowhere near doing it right.
Jon Bateman: Before I ask you what we should look for next and what we be watching to understand how this unfolds, it just does strike me that, yes, there's something deeply weird, odd, uncomfortable about discussing the end of the world as it pertains to AI. And yet, this is not the first issue, even that people in the foreign policy field have dealt with that is existential risk. This is very familiar in some ways that people work on nuclear policy, climate change, biological threats. Well before the advent of modern AI, you had people like Carl Sagan and others who were trying to popularize the notion of the great filter that perhaps it's just expected that any species would eventually develop the technology to destroy ourselves. And we had an initial reckoning of that in the 20th century. And now it's coming back again in this new form that feels strange, but maybe actually is pretty familiar.
Nate Soares: Yeah, you know, and some people say, oh, this whole AI thing seems a bit sci-fi, but A, the machines can hold on to conversation these days. You know, the idea that this is science fiction, you know that the author's writing about artificial intelligence and science fiction doesn't mean reality can't, the machines, you can never be smart. And also, you have the stories where it goes really well and the stories where it go really poorly, they're both fiction, none of them. None of them tell us about the real world. We've just sort of got to deal with the world we're in. And as you say, you know, it's very precedented for the world to come together. You know, a smaller but real issue was the issue of chlorofluorocarbons and the hole in the ozone layer. And you know these days, I sometimes run into people who say, oh, you the people are always talking about doom. They're talking about, you now, whatever happened to the hole in the Ozone layer? They said that was going to give us cataracts and cancer or whatever happened to that. Well, what happened to that? Is that the international community came together and banned chlorofluorocarbons. Yes. They said it's not worth it. And then the ozone layer healed. You know, I'm not here saying we're going to die. I'm here saying, we have this big issue. We need to come together again, like we have before and not build machines that are smarter than us while having no idea what we're doing in a situation where we're just growing them, right? In some senses, common sense. And as you say, it's very precedented. We just need to actually notice the issue.
Jon Bateman: I want to close on this. First of all, this is a remarkable book. This book is worth reading. This is an important introduction to a set of ideas that will define our future in the coming years. Controversial, hotly debated, but you made the most powerful case that you could for your point of view and people need to reckon with it. Still though, it's such a tall bar to persuade a typical person, even a very thoughtful, sophisticated person, that we are on a path toward global destruction. So if you could just close by speaking to a listener who is intrigued by what you're saying and is maybe now at a maybe, but still thinks, mm, doesn't seem like clear and present. Doesn't seem anything is going to happen imminently. I've got some more faith in the institutions and the companies than this guy does. But maybe you've hooked this person slightly. What should they be watching for in the coming years? Indicators to tip them from a maybe to a yes, my God, this has to be the main global issue.
Nate Soares: Maybe one thing to watch for is the AIs becoming able to do more on their own initiative. You know, everyone's always trying to make agents, AI agents. They're probably going to fumble around for a while. But a lot of people today sort of have this impression, oh, the AIs are kind of floppy. They hallucinate. They can be a better version of Google search, but they can't really, you know, do some task well, some hard task that takes more initiative over a longer period of time. Watch for that. Don't let yourself get frog-boiled into that. If you start to see the AIs able to do, you know, tricky stuff, hold that ability over a long period, reevaluate whether, you think the field's moving towards AIs that do things on their own. And, you come back and look at the arguments in the book that they won't be pushing in the directions we want.
Jon Bateman: It's a race between AI capabilities and this AI alignment effort initiative hope. And we hope that the AI alignment piece wins in that race, but it's a daunting one.
Nate Soares: Yeah, I mean, I don't have hope in the AI alignment stuff these days because of this. You know, you get it wrong the first round and you don't get a lot of, you don't get a retry. Yeah.
Jon Bateman: You know, a man can dream. Yeah, well, that's maybe a good stopping point. You close the book by saying, you hope you're wrong. Nate Soares, I hope you are wrong, but it's been a fascinating and important conversation. You've been listening to The World Unpacked, a production of the Carnegie Endowment for international peace. To get episodes delivered directly to your inbox, use the link in the description or subscribe on YouTube or popular podcast platforms like Spotify or iTunes. Views expressed are those of the host and guests and not necessarily those of Carnegie. Learn more at carnegieendowment.org.
