How dangerous is superintelligence? Connor Leahy, a former open-source LLM builder and current AI safety advocate, argues that the threat is existential.  

On The World Unpacked, Connor Leahy joins Jon Bateman to outline why he sees superintelligence as a national security emergency and how a deterrence-based “trust but verify” approach, modeled on nuclear arms control, could be the solution to runaway AI. 

Transcript

Note: this is an AI-generated transcript and may contain errors

Connor Leahy: There cannot be freedom or liberty in a world where there are institution systems that fully outclass us in every single way. The same way that chimps in a zoo don't really have freedom. Obviously, we must pursue economically and militarily viable AI systems. This is absolutely necessary for our economic and military competitiveness. We do not want to fall behind. We cannot fall behind, this is very, very important for our future. Superintelligence will not give us this advantage. Superintelligence is not a tool. Not a weapon. It's an adversary. If America built superintelligence, it will not allow America to achieve its objectives. It means America will cease to exist.

Jon Bateman: Connor Leahy, welcome to The World Unpacked. Thanks so much for having me. I've been looking forward to this conversation because you are one of the few people actually doing what a lot of others out there are hoping or waiting for, which is you are trying to build a political movement to regulate AI, specifically super intelligence. So in this conversation, I'm gonna have you take our audience inside some of your private discussions with congresspeople and others on Capitol Hill. We're going to talk about how an individual can participate in this movement because that's actually something that you are seeking to accomplish. But first, I thought we could tour through your background a little bit because it's unusual. You actually started off building large language models of your own. Why did you get into that?

Connor Leahy: So I guess you could kind of take my background into kind of like three chapters, so to speak. You know, as the first, like the early days, you know, where I first kind of gave some variety and then where I led a group called Luther AI. We were a very early open source AI movement that built some of the very first open source AI systems, not just open weight, open source, which even today is quite rare. And the reason I got into this, well, it's kind of a pretty funny story that would be none of the details we can get into another time maybe, but ultimately, It was because when I saw GPT-2, which is kind of the ancestor, you know, great ancestor of chat GPT, I saw the thing that I had basically been worried about since I was a teenager. When I was teenager, it was kind of I guess the prelude to the whole story, I was thinking, you know very hard about like, how can I help the world? How can I fix problems? Like there's a bunch of problems in the world, that sucks, I gotta go fix them. How do I solve all these problems? So I thought, well, maybe I could be a doctor. I could try to cure cancer or something, or I could to solve climate change or peace and whatever. I'll figure something out. But then I thought hold on a second, I'm just one guy. Maybe if I dedicate my whole career, I could cure maybe one disease. But if I could instead figure out how intelligence works, well then I can automate science and then I could can cure all the diseases. Great, I'll go do that.

Jon Bateman: It's pretty ambitious for a teenager, if I may say so.

Connor Leahy: I mean, you know, in the way teenagers are ambitious, right? Like I was just like, well.

Jon Bateman: I'm going to cure all disease. How hard can it be, you know? And so just to say on large language models specifically, because most people out there who aren't part of the technical community became aware of these when ChatGPT was launched and in those heady days in early 2023 when concerns about AI risk, but also hopes about its potential, really started to surge. What did you see in GPT-2 and in the systems that you started to build on your own that gave you a sense of where this could be headed.

Connor Leahy: So fundamentally, what I realized after thinking about this more for years and reading much more about this is one of the things I realized is, wait, if you have something that's so smart that it can cure cancer, cure all diseases, something that all of our greatest scientists have spent literally millennia trying to do, that's a very powerful thing. That's a dangerous thing, how do you control something like that? So this became the problem I dedicated my career to solving, is how do we control something that is that powerful so we can actually get a good outcome from it? But at the time, I thought this was still pretty far away. AI at the, time was pretty very brittle is like the way I would describe it. You could have an AI that can like play chess or play Go or maybe could do a specific image recognition task and they were pretty good at this and this was economically useful and so on but they were very brittle. They could do one thing and one thing only. You know, if you try to get them to do a second thing, everything would break. You'd have to have a bunch of PhD students high on Adderall spending many, many weeks trying to like tweak it, gather more data, et cetera. And GPT was different. What I saw in GPT-2 was the thing that was missing to what I thought would lead to super intelligence, which is general purpose pattern learning. It had the ability that you can just throw data at it. You can just stuff at it and you could just give it more computing power. You can give more memory and it would learn more and more complex patterns all by itself. So like, you know, the smallest models maybe could barely spell a couple of words. You gave them a bit more computing power, now they could write whole sentences. You give them a big more computing power now they can write whole paragraphs. You give the more computing power now that paragraphs start being more accurate. And this is out a human telling them.

Jon Bateman: How to do each of these things. So flash forward into today's world, we actually have advanced large language models that are solving mathematical problems that have been unsolved for a century. We've had people win the Nobel Prize off of scientific breakthroughs in chemistry, biology, physics that are part of large language models. So you were ahead of all of this in your thinking, but you also understood the dark side, which is what took you into this AI safety world. You then had your own company, Conjecture, which did technical AI safety work. Now, at some point along the line, I guess you pivoted from thinking that AI safety, that the constraints on these systems is a technical problem that a technically-minded person like you, when you were young, could solve through engineering, to maybe an unsolvable problem at a certain scale, uh, something that- actually we need society as a whole, legislatures to come in and just prevent certain things from being built. How did you shift from seeing this as a technical solvable problem to a problem for society and really politics?

Connor Leahy: Think that's the right way to see it. Fundamentally what I realize is that the problem we're facing with AI is not a technical problem. It is a political problem. It is the question of what experiments get run on the general population? Who gets to decide what level of risk of death you, your family, et cetera, gets to expose to? Who decides what kind of technology is widely distributed, including to both good and bad actors? Who decides questions. Fundamentally, if you know, for example, if it was legal for every person in the world to own nuclear weapons, and there was zero restrictions on this, we would be in a very unsafe world right now. You know, even if maybe most people wouldn't want to hurt someone, it will happen. And there would be accidents, you know. There would be deliberate attacks. It would be a very dangerous world to live in in this circumstance. And I realized that with AI, in a sense, it was very similar. It's that it is so much harder to build AI that is good, whatever that even means, than it is to build a AI that's powerful. So by default, what you get are things that are powerful, things that can gain a power, that can solve goals, that can achieve objectives, and so on. You don't get things that act in your interest, necessarily.

Jon Bateman: So the who decides question, that's very profound. And really, right now, the actual factual answer is people in private companies are deciding with virtually no democratic oversight. But I want to hold on that just for a second, because you might say a mainstream view within AI safety and the problem that engineers in this field are working on is there is a belief by some. That you can build ever more advanced AI systems, even a super intelligent system that's way more intelligent than any human being and do it safely. And we get there bit by bit through empirical experimentation. There's hordes of people at places like Anthropic, OpenAI and outside research institutes doing that. I think you think that won't work past a certain point of AI capability. Walk us through why that's the case.

Connor Leahy: I think there's two answers, questions of why wouldn't it work and why would it be bad even if it worked? Fundamentally, the first question, it's worth reframing what these people are actually saying. So let's take a step back. When we talk about superintelligence, we're talking about an AI system or billions of AI systems, and I don't think it'll be a single one, right? We're talking billions of copies of AI system that can out-compete humanity. That means they can outcompete you economically, they can now compete you militarily, politically, technologically. Everything else. All things equal, if you are up against billions of things that can out-compete you in every relevant domain, you get out-competed. And you get, you know, to and below the margin. So that, and this is how humanity gets out-completed to extinction, natural selection, all things equal. So this is a very dangerous direction to go in. So when these people talk about, they will create an aligned superintelligence or whatever that means. What they're basically saying is, they're going to create a one world government that will control your entire private lives and all other private people's private lives, all the governments, all of the militaries. It can build super weapons. It can, you know, build any kind of new technology and it will only do good things that are good for you and will do this using software with no bugs on the first try. That's what they're saying.

Jon Bateman: To slightly push back on that. I mean, there are people, Robert Wright, for example, who has written about an end game for AI being a world government. I would say, though, that the leaders of the AI labs, Daria Amadeus, Sam Altman, that's not what they say they're driving toward. They actually tend to couch. Their company's strategies in terms of a national strategy for the United States and a kind of democratic world within the kind of Westphalian state structure. But I guess you think that these systems would become so powerful that political institutions like states would just fade away.

Connor Leahy: Obviously. If you have something that can outcompete you at every level, again, it's about power. There are no property rights in a world where there are super intelligences. A super intelligence can take anything from you at any point in time. No government can defend against a super intelligence. No military, no police can police a super-intelligence. And once a super intelligent exists, we are at complete mercy of this thing. And this brings us to the second half of the previous question is that, would this even be good? Now, I'm personally not a big believer of freedom. You know, and liberty and a pursuit of happiness. And there cannot be freedom or liberty in a world where there are, you know creatures, beings, institutions, systems that fully outclass us in every single way. The same way that, you now chimps in a zoo don't really have freedom. Now we can say maybe the chimps have all the food they want, but they don't have freedom, they're not enfranchised. Chimps don't get to vote. So obviously if we're in a word where there our systems. That can outclass humans to this level. Even if they weren't to destroy us, well, we would still be disenfranchised.

Jon Bateman: You have no power. Yeah, it does seem like there's a real trade-off between those who think that superintelligence itself can be governed and constrained are attempting to build the technical and institutional means to do that. But then those means themselves would be a way to surveil and control other people's use of AI. And if AI becomes so powerful that AI is just how we do everything, that does potentially become a means of. Controlling, surveilling how everyone does everything. So it is kind of a pick your poison problem, right? Either the AI outcompetes us or the efforts to constrain ever more powerful AI itself becomes a very troubling force. So you have a solution to that. You are the US executive director of an organization called Control AI. What is your theory for how to stop this?

Connor Leahy: The first and most important thing that we have to do, the most acute thing is we have to prohibit the development of super intelligence. Fundamentally, if we get into the world where super intelligence exists, it's too late. Humanity has abdicated its ability to control the future and the future will be decided by machines. We should simply not get into this situation in the first place. To do this, our general recommendation of a policy is kind of two parts and then international part. On the domestic side, there's two parts. One prohibit the development. The same way it's illegal to build a nuclear bomb. It should be illegal to built a super intelligence. It should illegal to have a program to an attempt to build super intelligence if Microsoft had a nuclear weapons program. Now this is controversial, but we should shut that down. I have very controversial opinion I have here. And then on the second side, there is we must monitor and restrict the precursors to super intelligence We cannot wait until a super intelligence exists and we evaluate it or some nonsense like this. Because obviously super intelligence is a fully autonomous system that can escape, it can get out, it can be manipulated, it can, there's no containing something like this, it's like a child trying to contain an adult, it's just like, it doesn't really work.

Jon Bateman: So this is where I think it becomes crucial to maybe spend a moment defining superintelligence and explaining the difference between those sorts of systems which you want to prohibit and what other systems you would allow to continue to be developed.

Connor Leahy: The way we define superintelligence generally in legal text is autonomous systems that plausibly can circumvent, undermine, or destroy the U.S. United States government and or its security forces. So this would be a system that could be, that could not be policed. That is so powerful, it could destroy the United States military, it could evade its law enforcement, it would break our ability as a nation to be a nation.

Jon Bateman: And is that a question of just the system's pure capability or of how it's deployed? I mean, do you understand what I'm saying by that? Yeah, maybe you could imagine a very, very powerful system that in theory, if we were to give it the keys to the kingdom, right? I mean in Terminator 2, they gave Skynet access to the nuclear weapons. So somebody might say, well, just don't do that. And then maybe the same system wouldn't pose the same risks or Or maybe you could pose the question from the other direction and say, maybe a set of less powerful systems, if given the keys to the kingdom, could pose those threats. So how do you see that, like, the system level versus just how it's deployed?

Connor Leahy: In practice, deployment doesn't matter for superintelligence, because it can get into anything. It can get in to anything. It can win any economic game. It can run companies. It can make money. It can hire people. It can one political campaigns. It can develop new technologies. How would it get access to nuclear weapons? Does it need those?

Jon Bateman: Well, you just said I could do it, I couldn't get access to anything.

Connor Leahy: Well, to be clear, I can. I can tell you how. Yeah. It goes to North Korea, tells to them, hey, I will make you win the next war because I've already got all the plans drafted up. In return, give me those codes. That's how I get access to nuclear weapons. Yeah. To be clear there are many ways to get access nuclear weapons if you are smarter than all humans put together. For a human it's very hard. Well, it's not impossible. You know, again, North Koreans did build nuclear weapons so definitely I think it can be done. I think its very hard, but it's totally doable.

Jon Bateman: You're listening to The World Unpacked with John Bateman, where we dive deep into pressing global issues and make sense of the big forces shaping our world. Now, have you learned anything so far that intrigued or surprised you? Let me know in the comments, or just give us a like. And if you wanna hear more of my conversations with the world's most informed and interesting people, you can subscribe right here. Now, back to the show. Yeah, I guess I'm playing with this example because it does raise some interesting questions. I mean, like, it's not physically impossible for a person to build nuclear weapons. So part of the way that we enforce that law is through intelligence gathering, law enforcement informants. We try to track the supply chain. Right. So maybe one might hope that similar tools could be used to detect. Efforts by a powerful AI to take extreme measures against humanity. What do you think?

Connor Leahy: It would definitely be plausible and, you know, it would be interesting to build such systems, but again, there's a fundamental trade-off. This works as long as it's not smarter than you. If it's already, there is another very important thing we should talk about very briefly, which is recursive self-improvement. So when we're talking about AI systems like this, one of the ways that most experts believe this goes out of control is through a process that's called recursive, self- improvement. This is the idea that once you have one AI that's as good as a top AI engineer, Well, now you have a million of them. And of course, all of them can develop a better AI. And now you a million better AIs. And they can of course develop an even better AI and so on and so forth. So the difference can change very quickly.

Jon Bateman: Yeah, and also people should know that those in the AI industry at the frontier of capabilities, their strategy is to achieve recursive self-improvement. And they use various formulations to describe this. But generally, people at leading labs say, we're in the beginning stages of this now.

Connor Leahy: Your assessment. That's correct. Many experts believe, and I think this includes Nobel Prize-winning scientists, people at these labs and so on, and myself believe that it is very plausible that we will see superintelligence within the next one, two, three years. If that's the case...

Jon Bateman: Than any legislation that you're seeking to pass. First of all, it needs to be passed very quickly, and then needs to bite very quickly. Talk me through the timelines here, because the more I think about it, that's a real crucible. I mean, you said earlier, in very stark terms, that once superintelligence exists, kind of game over, right? I mean maybe humanity wouldn't be eliminated right away, but we would no longer be in control. Of what happened. We would no longer have the tools to influence the course of events, and then later this could lead to dramatic lethal threats to our species. So you think one to three years is the time that we have? I'd say 30%, maybe 50. Okay. How are we doing on passing enforceable legislation in that time?

Connor Leahy: Far from maximally bad? Say more about that. The way I like to think about politics is usually that everything is impossible until it very suddenly isn't. And we've had a lot of very suddenly not lately. So the way I think the good timeline looks is not that things bumble along at the pace they have for the last five years. It's that very suddenly people, they have preference cascades, people start acting, people pressuring the Congress, people petting Congress, people are like, what the hell is going on here? National security, you know, is like, what the hell is going on here? And then things can go very quickly. We don't need a bill. It doesn't have to be a bill, like, hypothetically, you know, President Trump could look Xi Jinping directly in the eye, they shake hands and like, yep, we're going to make sure this doesn't happen. You know, doesn't have to Be formal. What has to happen fundamentally is much more pragmatic than that. It's like, it's very pragmatic. And that, you know, let's let's say tomorrow, there's a genie that allows us to pass any bill everywhere in the world. So we passed the ban ASI forever in all countries bill. What happens? Well, in one week, it's violated in spirit, in two weeks, it is violated in letter. No one cares. What matters is enforcement. What matters, is execution. What matters as law enforcement, much more so than this specific bill or that specific bill. If we need to get some legislation, we need get a specific bill, we can do that. What we need is people to be bought in. We need willingness, as one might say in military context. And what we have found over and over is that the main bottleneck to willingness is actually understanding. The number one, you know, I'm sure we'll talk about this a bit more, but the number one experience I have when I talk to members of Congress or members of the national security establishment is just that they have not been briefed on this before. Yeah. They just didn't know. They didn't that people thought it was one to three years away, that it would have these severe consequences, and that it's such a severe national security threat. And then they start asking the question, Okay, what can we do? How fast can we act? Can we get these things?

Jon Bateman: So let's walk through that. This is a fascinating question, because for those out there that are troubled by this AI existential risk concern, superintelligence, and by the way, there's probably a much larger group of people out there who have other concerns about AI that they're just waiting for Congress or some kind of legislature and power to take hold of, whether it's child protection, copyright, job loss, privacy, data centers, I could go on for minutes on this. So, you're actually talking to members of Congress, and you're going to them and saying, let's do something. When you walk inside that room, I guess, what's the first thing that you say to try to see someone's attention? And what's first question that they ask?

Connor Leahy: Question. I have a pretty standard pitch, pretty standard explanation that I usually give to people and it usually starts with two pieces of information. The first piece of information I try to convey to people is that superintelligence and AI are not the same thing. Superintelligence is a small subset of AI.

Jon Bateman: Which, by the way, tell that to President Trump. As people may know, superintelligence has had its own understanding, which you've described at an extreme level of AI capability. Recently, Donald Trump decided that he wanted to just rebrand AI itself as superintelligens.

Connor Leahy: Yeah, that's right. And I'm happy to change to another term or like, but this is super intelligence actually is also relatively new term back in the day, we call this AGI. And even before that, we called it ultra intelligence. So like, there's been like, multiple changes of this term over the years. And now there's another rebrandom process, we can call it what you want. The big thing. Now, this is the really thing I care about the big thing, the thing that can outcompete humanity, whether we call super intelligence, ultra intelligence, AGI, whatever. That's the thing I care. We're using the word superintelligence because it's kind of, you know, was until recently the norm. So for the sake of this conversation, we'll just keep using it. But I don't really care what we call it.

Jon Bateman: So there's a difference between that and AI.

Connor Leahy: AI is an incredibly broad set of technologies, you know, includes everything from, you know, protein folding to game playing to image recognition to like, this is a very, very broad technology. So what we're absolutely what I am absolutely not against is AI. I am not against AI by any means. I think there are many, many very economically and militarily important applications of AI that we should pursue and are very important to a prospering country and happiness and stability and so on. Not that there aren't other problems and other regulatory burdens that come up with such new technologies. But they're different from what I'm talking about here when it comes to superintelligence. Superintelligence is a national security emergency. This is a natural security threat. This is not a commercial question. Now, if the question, for example, child harm, I think is a very important problem, but it is a commercial. You know what I mean? It's like a product liability question. It's a consumer protection question. This is NOT a consumer-protection question. This is, you know, a citizen's lives protection problem. This is a military problem. And so we control our very single issue in this regard. I'm not, we fully agree that there are other very important problems to add, but we only focus on this one specific problem. Okay. And this one, specific problem, the only solution is to not build it. We currently have no technological, political, or other way of handling something this powerful. There's no way. So we must not build, we must prevent its creation both domestically and abroad. We must create, we generally recommend is the creation of a trust but verify regime backed by credible deterrence. You know, it's a national security issue and also become a deterrence problem. It becomes a question of how can we prevent any actor from creating such systems? And you know, there's many levels to deterrence, right? There's, you know there's, you know restricting the resources necessary to, you know produce such systems in the first place. There is credible threats to that of escalatory ladder to prevent. Furthering down that direction. And it's, of course, a technical verification question. How do you verify that people are not pursuing this technology? And this is usually where I was telling you about what I tell Congress people. This is the second half where I usually talk about. The first half, I explained what is superintelligence. I'm not talking about AI. I'm only here to talk about this very narrow subset. And the second thing I actually explained to them, which I think is very important and we haven't touched on yet, is that AI is not like normal software. It is grown, not written. This is a thing that's very confusing to people. You asked me earlier what I saw when I saw GPT-2. Well, shortly after that, I built my own GPT2 because I was a hacker at heart. I was computer hacker at hard and scientist and the best way I knew to understand something and I thought this was the most important thing to understand is to build it. So I built by own, trying to understand it. What I found is that I could not understand it and no one could because this is not lines of code. Normal software is lines of the code to tell the computer exactly what to do step by step. This is not how AI works. AI is what's called neural networks. And neural networks are more like grown. They take in information, they learn from their environment, and they kind of grow. And what comes out the other end of this neural network is kind of imagined in like billions and billions and billions of numbers. And if you execute those numbers on your computer, you get chat GPT.

Jon Bateman: Fundamentally, the brain is composed of individual neurons. Those neurons are connected through synaptic links, and our thoughts emerge from the communications among and the connections between neurons. We know this, but we actually can't, with today's science, look at these neurons and synapses, and from that, derive what a person is thinking. And that's... The exact same thing with LLMs, right? Now, I should say, people are working on this, a so-called mechanistic interpretability. Lately, there's actually been a bit of excitement around some new discoveries that maybe suggest that, while we're not there yet, in principle, perhaps it could be possible in the future to actually look inside that LLM brain and tell precisely what it's thinking. Do you put any stock in that?

Connor Leahy: Well, that recent result, I think, is unbelievably overhyped and not nearly as interesting as people think it is. But absolutely do I think this is a scientific problem. I think also neuroscience is a science problem. I think we can decode the human brain. I think there's no fundamental reason we can't do this. I just think it's really, really, hard and it's going to take decades to do it. And the same thing is here with AI. I'm not saying we can understand AI someday. If our greatest mathematicians, scientists, philosophers spent three generations. You know, working on mechanistic interpretability, on AI control, on AI oversight, on, you know, all this kind of stuff. Yeah, I think we could do a lot.

Jon Bateman: That's just not happening. And a tiny fraction of the resources going toward this type of research versus advancing the capability. So it's not a fair fight. One side is drastically outracing the other. So OK, so you walk into the office of a member of Congress. Now, in my experience, there's kind of two pitches, two types of pitches that you can make to a member Congress. The one pitch, and I think this is what I'm hearing from you, is I've got a good and important idea. It's true. It's right. It's good. That's actually the hard way to pitch someone, unfortunately. Hate to say it, but I think the easy way to pick someone is the other way, is to say, I've got something that's in your political self-interest. And if you don't do this, you'll regret it later. You'll lose an election, you lose votes, donations, you'll lose your position on a valuable committee. Do you have a case for political self-interest? Or do you actually just try to convince people that they personally will die in a few years if they don't ask?

Connor Leahy: Well, I have two answers to this question. The first, a bit of a baby snarky answer is, today, ABC put out a high quality polling showing that over 60% of Americans thinks there's at least a medium chance that AI is going to lead to extinction across both parties. So, simple answer, the data is in our favor. There's even polling that polls even higher on this regard.

Jon Bateman: And it's funny to me, because I live in this world. I live and breathe these ideas. I was actually shocked when my dad visited one time, and having read none of this literature, he just said, full stop, yeah, I think there's about a 50-50 chance that we might all be killed with AI. So that is out there. I think it's a low-salience issue, though, right? Like, if you were to ask voters to rank how they choose a candidate, I mean, I'd be shocked if any more than a handful are ranking on this issue.

Connor Leahy: So funny fact, we do have data on this, where you're correct, it is low saliency, but it's derivative. How fast it is rising in the ranking per month is the highest of any issue. Really? Yep. Okay. So I agree. It is currently not yet a top 10 issue by any ways, but the number of steps it is going up the ranking every month these polls get run is higher than any other

Jon Bateman: So, so do you then say to that member, maybe not in 2028, maybe not in 2030, but in 2032 or in 2034, if we're still around then, this will be a top three issue in your district and you will want to say that you got ahead of it? Just how do you make the case that this really is going to matter as much or more than other things that keep congressmen and congresswomen up at night?

Connor Leahy: So I gave you the snarky answer, but I can give you the true answer, which is I think the more interesting answer, which is that truth is an asymmetric weapon. Fundamentally, it is much easier to convince people if you're telling the truth. And I have found it, so another thing is when people work with me, they sometimes learn something about me, which is that I am a frustratingly simple-minded, in a sense, where a lot of how I do strategy and persuasion and so on is just to try the stupidest possible thing first. Just first of all, just do the dumbest thing you can think of, do that until it fails. So a lot of what we did at Control.ai is like, hey, let's just tell everyone the truth and just see where it fails, you know, let just see, you maybe no one cares, they only ask us the polls or whatever. And I mean, I don't know how to say it, but it hasn't failed yet. It is our ability to get people to like care about or like get upset or interest in this issue. Just from explaining to them the fact is much higher than anything we've ever tried about like odds, your political galaxy brain, strategy, interest, whatever. Like we've tried this kind of stuff and it just didn't work that well compared to just giving them the raw facts of here's what experts believe, here's was going on, here's going on. I'm open to these kinds of political games if it makes sense, but so far, I basically can persuade people this problem's a problem as fast as I can tell them about it. Like, I have just, like... That's fascinating. I... This is not the case for all issues, to be clear. Like, there are many issues for which this is not true. But for reasons we could talk about, we can argue about, on this specific issue, I have not just run into this problem, empirically. You know, maybe in like 20% of the cases, sure. Yeah, no, I...

Jon Bateman: That's hopeful. That's inspiring. I guess, I don't know. I once had a hairstylist who was a 9-11 conspiracy truther. So this is during the George Floyd W. Bush era. And she just told me flat out, our current serving president conspired to kill 3 million Americans. Now, I talked to her about it. And I thought, OK, she definitely believes this. But does she believe it in such a thorough going way? That she's acting the way one should if you fully absorb and internalize it, right? Like, if I were me, I'd be fleeing the country, right. So I guess then it's one thing to kind of sit down with someone and they listen to you and they say, you know what, I get it. I never thought about it before, but I don't have a counter to what you're saying. I am now worried about it. Does that lead to those people then acting? With the same level of urgency that you and your colleagues have to write and pass legislation.

Connor Leahy: The way I tend to think about things is that there are four steps. There's attention, there's information, there is persuasion, and then there's action. And all of these are actually quite different and you can't skip a step. If you start with action, nothing happens. If you star with persuasion nothing happens, first you need attention. People need to like even register and they have 30 minutes slot on their calendar to think or talk about this issue. Otherwise nothing happens if you're not on their calendar, they are not thinking about it. So that's the very first thing. And like, let me tell you. Most people do not have a 30 minute slot in their calendar to think about existentialists. They just don't like, it sounds stupid, right? But like, literally they do not have 30 minutes a week to think about this. So us just like, you know, shoving our way into their inbox and getting them a 30-minute meeting to think about this is the first step to do anything. Nothing can happen until 30 minutes have been spent thinking about this topic. Then there's information. Most people just do not have accurate information about this topics. Most politicians still have not been given a single high quality briefing on this topic. Industry is actively trying to make the information as hard to access as possible. They're trying to fud, they're trying spread fear, uncertainty, and doubt. They're tying to mislead, they are trying to scare, and so on. So It is an adversarial environment. When it comes to persuasion, my experience is just isn't that hard. No one wants this. No one want AI to replace everyone. This isn't like a partisan issue or anything. Like I talked to Democrats, I talked Republicans, I talked, you know, MAGA, I talk to a progressive and everyone's like, hell no, we don't want that. Like I didn't know this was happening. That seems really bad. What can we do? And then there's action. And so action I think is its own category of things in the sense that action is not just a function of belief. This is, I think if you think about, oh, if they really believe that they would act, I think this is a bad model of how things work in the real world. Like, this is an important component of it. Until you have some buy-in, action is impossible. But then you also have to see, like, what other constraints are constraining their actions? You know, what can they do with their party? What can they, you know, regarding the president? What can we do regarding Congress, et cetera, et cetera. And so the way we like to think about how we approach actions is kind of like a escalating ladder of actions of increasing importance. Publicness and goodness. So even the smallest possible actions that we think are impactful are those that are public, that have some small impact and that are good. Class example is, have a politician say on Twitter, hey, I'm kind of worried about this. Seems like a small thing, but it's something you can build off of.

Speaker 3: Uh-huh.

Connor Leahy: You can say, hey, look at all your colleagues who have recently said on Twitter this is a problem. And the next one I convinced becomes even easier, it compounds. So then maybe we have like a couple of people who are now interested and they say, oh, 10 other members of my party are also concerned about this issue. Maybe we can create a bill.

Jon Bateman: Yeah, I remember last year, I think, there was this guy on Twitter who would keep this rolling list of all the members of Congress who have ever said anything about AI and existential risk and super intelligence. And after it got up to a couple of dozen, I was like, well, yeah, that really is something. I haven't seen that list in a while because I think it's grown so large, right? You have actually captured, you and people of your ilk have actually capture a significant amount of mind share there. Yes. And

Connor Leahy: And this is how this kind of process works fundamentally. There is a thing where democracy moves slow, sometimes too slow. But in a sense, this is also on purpose. To do something large in democracy, you should have to put in a lot of work. You should have convinced a lot people, because otherwise it's a dictatorship. If one person could just unilaterally change everything, that would be really bad. That's very anti-democratic. So in a way, yes, it is slow and it's frustrating, but that's what it means to Democratic. We do in fact have to, we've now briefed here in the US over 200 offices in Congress, I've personally talked to dozens of politicians face to face on this issue, and still most of the ones I meet have not gotten the 30 minute briefing before they met me. So there's just still a lot of work due. So the thing I'm concerned about is less could we get there, the question is how fast can we get? This is the thing that I think concerns me is timelines. I am I am not concerned about persuasion. When it comes to actions, I'm like, we are already getting more and more actions. Bernie Sanders and Greg Kassar recently introduced a bill which bans ASI. It's not going to pass probably that specific bill, but it's the first of many. Now we're already talking to maybe Republican versions of such bills and now the ideas are in the air. Now people are talking about it. And so I think if we had, I don't know, three years, five years, I think we'd make it. Like I think the normal democratic process could work here. And when it comes to less than three years, it's so possible through the normal democratic process.

Jon Bateman: Process, but it is trickier. So in my experience talking to senators, members of Congress, particularly about a new issue, you explain the pitch, you explain the idea, and then one of the first things they want to know is where are other powerful people on this issue? What does my president say? What does my party leadership say? Often part of the question is, and this is not like a dirty question, but like where is industry on this, right? So, Where is industry on this? We talked before about how fundamentally the big AI companies are trying to do recursive self-improvement. I think I would say they're trying to create super intelligence. Some of them use that term, actually. Yep, on the website. And then also, if I were them, I would be thinking, well, even if I'd be willing to stop short of the kind of super intelligence that Connor's worried about. His bill is probably going to stumble me in some way. I want to at least get close or closer than I am now. And if his bill passes, like, it'll create a lot of ambiguity about how far I could go. I guess, how do you overcome that or do you see your coalition potentially including people within industry or is this going to just have to be adversarial? Is this just a pill that we need to kind of force them to swallow or do have allies within this world.

Connor Leahy: I mean, fundamentally, I think one of the sins of, you know, people engaged in AI safety is that they've been extremely naive about how adversarial politics is, just like truly deeply naive, just completely taken for a ride by companies and by lobbyists, and so just completely taking for a right. This is extremely adversarial. This is unbelievably adversarial, they are fighting us tooth and nail, no dirty trick is off the books. This is like big tobacco level is the kind of stuff we're fighting.

Jon Bateman: Give us a picture of that.

Connor Leahy: So the main thing is fear, uncertainty, and doubt. So this is the big tobacco playbooks. But I mentioned big tobacco for a reason. So big tobacco is famous for having mastered a specific playbook for lobbying that is often called fear, uncertainty, doubt, or even the big-tobacco playbook. The way it works is, let's say, you're currently unregulated and some new harm seems to be plausible. You might even know it's true, as the tobacco companies did all the way back in the 1950s, the new smoking cost cancer. Already knew this from their internal tests, but that was of course bad for business. So what did they do? Now, they didn't try to convince you that smoking was 100% safe, because that's too silly. You know, like you would know that's cartoonish, like that's obviously not true. So instead, what they said was, well, of course, we would never want to harm our customers, but let's wait, not act too hastily. Let's wait until the data is in. Actually, we will fund You know, let's have a committee, maybe. You know we have many committees to evaluate the evaluations of the, you know, considerations of the committees. The main goal is to stall for time. So when people wonder, like, what are these companies? Oh, they say they're for regulation, they say, they're against it, and then they do this, they do that. It's so confusing. Yeah, that's the point.

Jon Bateman: Yeah, so one response to this is surveying the actual legislative action that I see. And obviously, like you said, there's the Bernie Sanders and Greg Kassar bill. So I know there was a bill in the UK that was introduced that you all had something to do with. So there is now this emerging set of discussions around banning superintelligence. But up until now, I think the biggest focus of AI safety legislative action. Has been probably what you would just call transparency. So California passed a path-breaking bill on AI safety. It was essentially a transparency bill. Put out your plan. Tell us about incidents as they arise. Create a channel for whistleblowing if there's kind of skullduggery happening within the company. We saw similar bills in New York, Illinois. And then again, there's still a lot of focus on transparency, like let's put kind of auditors inside companies to kind of tell us what's going on. You're going much farther than that and actually saying we're just going to stop you from doing certain things. Actually, criminal liability. I think I read that the UK bill, you could have a life sentence. In jail for attempting to build ASI. Is that right?

Connor Leahy: What's the current sentence for attempting to build nuclear bombs?

Jon Bateman: I don't know.

Connor Leahy: Well, it should probably be on that magnitude of order, is my answer, which would be within a factor of 10 of whatever the punishment is for attempting to build a nuclear bomb. Well, nuclear bombs actually kill less people than superintelligence, so we can argue about that. Fair enough. But I think the more interesting question is, or rather an important point that I think has to be raised here, is that most of the so-called AI safety world is controlled opposition. Again, people are very naive about how adversarialism. This is very common in like pharma or big oil Although like this is super common, like patient advocacy organizations, there are tons of them that are just like fake or just like funded by pharma. There's not a conspiracy, it's like well known. This is also the case for like many like environmentalist organizations, many like, you know, like it's just a very common thing.

Jon Bateman: Maybe I can offer a number of fine men. You can push back on that. OK, so one kind of common industry tactic is just pure astroturfing, right? Like, you're on TV, you see an ad, seemingly in favor of whatever the industry wants. And then at the bottom, it says paid for by concerned citizens for such and such. But there really are no concerned citizens, right. It's a post office box in a strip mall somewhere that's purely funded with industry money. I think the people that are against you in the AI safety community, I might call them optimists, maybe you would call them naive optimists who actually believe that there are technical solutions, that we have time, that working inside or with companies, there's enough good faith there to make a difference. I mean, would you disagree with that?

Connor Leahy: I don't like psychoanalyzing individual people. I think this isn't productive. I'm sure they're nice people who love their families and love their dogs. What I care about is their actions. Let me give you a concrete example. The number one largest funder in the AI safety world is a group called Coefficient Giving. They used to be called Open Philanthropy. Their longtime CEO is named Holden Kronofsky. He's married to the president of Anthropic.

Jon Bateman: Here. So he's not currently part of Coefficient Giving, but you are right about that. I'll just say to the audience, you're watching a Carnegie Endowment for International Peace podcast. Coeufficient Giving has funded Carnegie in the past. Actually, Holden was a visiting fellow here. He's a nice guy.

Connor Leahy: Him before. He's a very nice guy, very smart guy. I don't want to psychoanalyze him here, I'm just stating a publicly verifiable fact.

Jon Bateman: It's certainly fair to say that there's a very interconnected web of relationships, financial and ideological interests among the people building AI and among the people seeking to create safety through these kind of optimistic pathways, right? The ones that are actually seeing this as more of an engineering problem. There's a bit of a revolving door, right, where, you know, people might spend time working at one of the leading labs. Then spend time working at a safety advocacy organization or a safety technical organization. Some of the safety groups are funded by money that came from the AI industry or maybe work directly for those companies. So there's definitely a web of interlocking interests, to put it nicely. I'm trying to think how we got onto this. I guess the question that I was driving at is, could industry be part of your coalition? I think your answer is no. Does, if you get what you want, does that actually mean? We essentially stop building more advanced general purpose artificial intelligence than we have today because one of the questions I had looking at your bills is you want to kind of cut off this RSI pathway. People now say that we're kind of in the foothills of RSI now. Are we already at a level of capability that should be stopped?

Connor Leahy: I think we're very close. We might have already crossed it. I don't know. I don't think we will know until it's too late, in a sense, which is why we have to be conservative. If we wait until we're 100% sure and we're like, it's already way too late. This brings us back to the original question. The original question is who gets to make these choices? When you asked me should we not build next generation or should we build this generation, my answer is I shouldn't get to decide this. Neither should any individual CEO. This should be a democratic, you know, national security level question that we decide as a country, as a civilization, and hopefully also in a consultancy with other international powers and their interests and our interests, and we find a way forward for humanity. I shouldn't get to make these choices. You shouldn't make this choice. And sure as hell these companies shouldn't to make this choices.

Jon Bateman: And one of the strangest things is the only thing that's stopping us, us meaning society as a whole through our public institutions, from making those choices is just doing it, right? Yep. The government has the tools through Congress. I actually want to circle back to something you said earlier that maybe we actually don't even need legislation. Are you referring to something like the export control tool that President Trump used to block for a time one of anthropic models? Is that kind of what you're thinking of? For example.

Connor Leahy: For example, I'm just trying to like differentiate between politics and real politic, you know, like in we want these things to overlap as much as possible. But like, if you need to get something done, you can get something done, is what I'm saying. Like if you really need to get something done, like the military needs to get something done. We can figure out ways to do it, you know, overseen by our constitution. And, but like we have many levers. There hasn't have to be, so for example, a good counter example or like negative example of this is the Nuclear Non-Proliferation Treaty. This is a great example of a thing that sounds great on paper and it's absolutely useless because it has a provision in it that it only comes into effect after 44 specific country sign. So a single holdout means nothing ever happens.

Jon Bateman: Well, okay, but this treaty is in effect, but but are you saying that we wouldn't want to pursue a similar model?

Connor Leahy: We wouldn't want to pursue a similar model. What I'm saying is, there are many options. It's not the question of, do we do a treaty, or do we a multilateral agreement, or do a bill, or do... There are many option, is what I'm hinting at here. I'm not saying it has to be this. I'm just saying, look, we can do it this way, we can it that way, we could do it that this way. There's actually many ways to solve this problem. Fundamentally, we have many national security tools to address national security problems, and we should use whatever tool is most suited to this problem. I personally think passing a bill would be a really nice one. I think it's a good tool for national security, but not the only one we have.

Jon Bateman: So there's export controls that Trump has used. Some people questioned whether that was a lawful use of those tools in practice. Interestingly, the companies didn't even push back. I think they wanted to stay on Trump's good side. And I also just think, in general, courts heavily defer to the executive on any credible claim of national security. There's other laws like the International Economic Emergencies Power Act, which is kind of super law that presidents use for all sorts of national-security issues. OK, so there's many paths toward actually implementing a ban on super intelligence. I'd say also there's may paths toward building political power. So I'm curious for, first of all, you guys are not only meeting with members of Congress. You have done some mass mobilization through letter writing campaigns and empowering individuals to be part of this conversation to talk to their members and others. What have you learned from that experience? How easy is it to activate members of the general public? Is that a tool that can work on the timelines that we have?

Connor Leahy: I think absolutely so, and we've learned a lot of just how possible it is and how much, again, I keep the same lesson over and over. If there's one story of control, just tell the truth, just tell people. They don't even need convincing. So much, whenever I talk to anyone, I talk my mom about this issue. I remember my grandma when she was on the way out, she wasn't doing well or whatever, but she really wanted to know what I was doing. And I explained, well, you know, I'm working on AI, you know might replace humans. I remember she really looked at me. She's like, it's no good, no good. You stop that. I'm like, yes, grandma. Like there's a thing where there's a deep intuition of just like, if you build something smarter than you, that you don't control. Yeah, that's not a smart thing to.

Jon Bateman: Yeah, it's funny because for a number of years, people in the AI safety community were hesitant to broach this subject publicly because of how weird it sounds. It's often accused of being sci-fi. And I think it is. It does sound weird in a way. It is sci-fI in a waY. We're generally familiar with this trope from sci-Fi, but also it's deeply familiar too. I mean, you mentioned the genie in the bottle, right? The monkey's paw. I mean, stories from literature and kind of the world's canon about Elon Musk referred at one point to, we're summoning a demon. I mean, drawing upon powers beyond our control and seeking to ride a tiger and somehow even though the tiger could eat us, we'll be able to ride it, right? I mean this is, it's actually a very deeply familiar intuition.

Connor Leahy: Exactly. This is the number one thing I feel the most gaslit of my entire life, is how much the AI industry tried to make me think that normal people didn't have good intuitions about this, that they couldn't understand this issue. And now, the truth is, I have an even more cynical interpretation here. I think you're not being adversarial enough. I think the industry tried to make people think you can't talk about it. I think it was much worse than this. I think there was a deliberate effort by groups, including, say, coefficient, giving, and surrounding cluster. Try to prevent people like me to talk to the general public and be like, oh, no, no. Don't do that. Don't. Do that. It'll be really harmful. It'll cause all these harms and blah, blah, and, you know, and there are many other people that believe versions of this for a long time. I think most of these people have not changed their mind since then, but there was a very deliberate effort by industry. Like don't tell the public about this. Don't tell a government about this because they're too stupid. They don't understand they're going to, they're gonna do something stupid. You know, they are going to panic and that's not good. We, the smart people, we have to handle it. You know we have. Like, all the way back to the early 2000s, this was a common topic being discussed on these forums of like, should the public be informed or not? And the answer usually was no, we have to do it ourselves, we the smart people.

Jon Bateman: There are other ways that one might build political power around this issue. One classic way would be through electioneering, right, involvement in campaigns. So I don't know if your organization is doing anything along these lines, okay, it looks like no. But others out there, actually people who have made money off of the AI industry have become major political donors, including helping funding campaigns like those of people like Alex Boraz in New York. On a kind of AI safety platform. So that's kind of one thing that you can do, right? You can sort of use volunteerism and money to directly intervene in the political process to kind of support candidates or parties who are on the right side of this issue. What do you make of that as a tactic? Is the time ripe for something like that? Are there candidates out there with those platforms and ideas that could be supported?

Connor Leahy: Well, my personal opinion on the constitutionality and whether we should be allowing such activity is too long for this podcast to contain. But generally, I would say, yes, I think it makes sense. I think is a thing that should be done. I think that is a normal part of how we do politics in the United States, for better or for worse. This is a very normal part our political process. If you are an individual person who is willing to donate to a local candidate, this is a really cheap way, actually, to influence politics in pure dollar terms. So this is a thing a lot of people don't know. Direct donations matter drastically to candidates, much more than you think. You can meet these people face to face for a couple thousand dollars and just tell them about what you care about. This is just a thing that can be done. So it's a very effective thing. We often are told we should be doing this. We should be donating candidates just so we can just meet them, tell them this issue, put it on the radar, get them in this direction. This is a very, very effective tactic. We currently don't do it. For various logistical and legal reasons, we think it's currently too complex, but it will be something that I would be open to in the future.

Jon Bateman: Yeah, and classically in the political ecosystem, you've just got different groups with different legal statuses and different political characters who are doing different parts of this tapestry. So and then another part of this tapestry is kind of classic protest techniques, civil disobedience, direct action. We've actually seen people chain themselves to the doors of AI labs or go on hunger strikes. Is that an effective tactic?

Connor Leahy: I fully support people's constitutional right to freedom of speech, but I do not think it's an effective tactic. I don't think in our current world it's a very effective tactic, I don't think it has been effective. It mostly just serves to kind of increase the temperature and increase chaos, while we currently need the opposite. The solution to the problem of AI is law. That is the solution. The solution is not chaos. It's not breaking things, it's law. It's sensible, ordered application of law. And so what we need right now, even so it can feel like we should be, oh, we should to be panicking. Oh, we shouldn't be screaming or whatever. This does not solve the problem. What we need is calm, deliberate action. You know, it should be appropriate levels of action, which is a lot higher than what we're currently doing. But I support people's right to freedom of expression and freedom of assembly. It's not something that we currently think is very effective.

Jon Bateman: We have to get back to this, but China question, right? And I'm sure this has come up in your conversations with politicians and other leaders. If America stops its progress in terms of AI capabilities, China won't, this is the story, right, and then they will catch up with us and then race ahead and they will use these powerful tools, which we won't have, to outcompete us economically and militarily. How often do you hear this response, and what is your countertribute?

Connor Leahy: I heard almost every meeting I have, and I think it is the correct question. Thankfully, people are starting to think properly adversarial towards China. I think a lot of people are still not thinking adversarially enough towards China, I think a lot people are doing very too friendly and too naive when it comes to China, that we are in fact in a very severe competition and that we have to take competition very seriously if we want to continue to have the West as we have it. That being said... There's a very easy answer to this question. And the most important prerequisite for this answer is just to separate superintelligence from AI. Obviously, we must pursue economically and militarily viable AI systems aggressively. This is absolutely necessary for our economic and military competitiveness. We do not want to fall behind. We cannot fall behind, this is very, very important for our future. Superintelligence will not give us this advantage. Superintellgence is not a tool, it's not a weapon, it's an adversary. If America built superintelligence. It will not allow America to achieve its objectives. It means America will cease to exist.

Jon Bateman: So the idea that super intelligence is not a tool to combat an adversary is itself an adversary. I think that's just the key idea here for a national security audience, right? You did say earlier that in your view, we're already close to the line of super intelligence or like the indicators that will like, almost immediately lead towards super intelligence. And so then I think if you are able to achieve your policy goals, We will. Basically stop it around where we are. So then you still do have this problem of that would enable China to catch up to where we are, like in the best case, where we actually have a trustworthy, verifiable agreement with them. Is that acceptable? Or kind of, what do you do with that?

Connor Leahy: Well, the first problem is they have already caught up because all of our data centers are penetrated. Every single system in the United States has been penetrated, these companies fall. So this is a fun fact. In America, we have a very sophisticated set of laws regarding weapons contractors and defense manufacturers, where like Alaki Martin has to follow a lot of laws with background checks and access controls and none of these AI companies follow these laws. So they're all compromised. Every single AI company in the United States is compromised. They hire Chinese citizens often to run their data centers. Like the level of clown show that goes on in the cyber security, I know people who like build these data centers and who work at these companies and so on, is hard to describe. Now imagine if Lockheed Martin was building our like hypersonics with this level of security. This would be unacceptable.

Jon Bateman: Well, OK, there's a lot I want to pull apart there. We don't have time for all of it. Maybe I would just kind of skip to the finish line with this thought and just sort of say something like, it isn't the case that China has the equivalent of American models today with whatever espionage access that it has, right? So if you're right that every AI company and data center is kind of just fully, deeply compromised, obviously there's kind of some translational gap there because China doesn't have the equivalent of the best.

Connor Leahy: The military does. The military has, I am very, very confident in this, the military has literal copies of our best models. Like I'm very confident of this. Not their private companies, their military intelligence, yes, absolutely. So what I'm saying here is not, this is not the end of the story. The only thing I'm seeing here is like, these companies have been lying to the American people and to the military about how this is how we do an arms race. Look, I'm down for an arms raise if we're actually doing an arms rise, right? Like if we're doing hypersonics or we're doing stealth bombers and we want to do an actual arms race, makes sense to me. This is not what's happening right now. Because part of an arms race means your arms have to not fall into the enemy's hands. Otherwise you're not gaining any advantage. And this is what's happened right now, if all of these companies were working in skiffed levels, data facilities, if they had all background checks on every single employee, they're using all the levels access controls to build weapons grade material, I'm open to this. This is is not an arms-race. Because this is not how we build arms. So what I wanted to spell here is this myth that this is, you know, obviously it's an arms race to a bad thing. We already established that with super intelligence, but it's also not even an arms raise. It's just these companies building whatever the hell they want and having it all stolen by China anyways. So I think that's why I say people are not being adversarial enough. Like I think we should be much more paranoid than we currently are. I think should be more paranoid, we should more concerned. So if we want to beat China, this is the not the way. This is just not possible. I am very open to us pursuing non-super intelligent AI systems to gain battlefield and other advantages against China. I think this is a very good thing for the United States military to pursue, but that is not what these companies are doing. Maybe Booz Allen Hamilton is doing it. I don't know. I don't know what they're working on, right? But I can promise you it'll open an anthropic or not.

Jon Bateman: What's the Chinese equivalent of control AI and the movement that you're building here? What's this status within China? We talked before about attention, information, persuasion, action, like this whole pipeline that you need to build in order to create just the political and policy reality to stop superintelligence. Where is China in that pipeline? And what's your level of confidence that, if we got there... They'll get there.

Connor Leahy: So I'm not Chinese. I don't speak Chinese. I've never been to China, so I don't want to say I'm an expert here by any means. All of this is second, third-hand information and so on. My understanding is that these issues are still very nascent, you know, in a very early stage in China. These haven't deeply percolated, and they're only slowly coming to a head. But one thing is that, look, if there's one thing I know about the CCP, is that they like control. And superintelligence will take away their control. And that is not something I think they will want to do or they will wanna tolerate. From my perspective, the way to get this information to China, the best effective way I have, is just for us to take it seriously. All I promise you that now their military generals are starting to get briefings because the Americans are freaking out about it. If the Americans start caring about this issue, the Chinese will know about it, that's the way I approach this problem.

Jon Bateman: I think there's something to that. There's also the problem that just as in Washington, actions in Beijing or Shenzhen are viewed with great suspicion and always kind of interpreted through the most negative possible lens, because we view the other side as untrustworthy and nefarious and they're kind of contrary to our interests, we're mutually hostile. So in Beijing, actions by Washington. And Silicon Valley are also viewed through a very negative adversarial lens. So if Trump were to go to Xi Jinping and say, hey, let's agree to stop together, there would also be people in China, very powerful people saying, this is a trick. This is a way for them to lock in their advantages to hold us down. America won't stop. They're trying to get us to stop. I guess some of this comes back to the verification question. Do you have a way to verify internally within America and externally with a country like China that superintelligence isn't being built?

Connor Leahy: So these techniques are still in their infancy for the most part because they just have not been funded. I mean, fun fact, a lot of the technology to verify the nuclear test ban treaty wasn't invented until after the treaty. It takes time to develop technologies like this. We have to throw $10 billion at DARPA and the NSA to figure this problem out. Sometimes you meet these coefficient giving funded two man startups working on verification technology. I'm like, dude, that's the NSA's problem. You don't have a security clearance, right? The NSA should be spending $10 million on this problem. So. Are there plausible mechanisms? Yes, there's actually a lot of plausible things we can do here. Fun fact, it is actually harder to build a modern GPU than a nuclear bomb. Nuclear bombs, we could build them back in the 1940s. We sure as hell couldn't build a GPU. Much, much simpler technology. Much simpler technology than a GPU is. A GPU is a much, much more complex, finicky piece of technology with a much tighter supply chain in many sense than nuclear weapons are. If we hadn't built the regime, like after the 1940's, it was widely expected that every country would have nukes. It's made out of rocks. Like uranium is everywhere, right? Like how could you constrain that? It's all, like there's all these uranium mines, these third world countries. There'll be no way to constrain it. It's, we're doomed. That was a much worse situation actually than we are in right now. Then the number of facilities that can produce say a top end GPU are, you know, like two companies maybe. And you know most of them are Western allies and you know they require you know hundreds of inputs materials. They require massive amounts of energy, of infrastructure, specialized talent.

Jon Bateman: There are many angles. In other words, this tight and complex supply chain creates opportunities for surveillance and control choke points, essentially. And then I guess these criminal penalties are probably going to do a lot of work, right? I mean, like 20 to life is it gives you something to think about if you're at a frontier lab in the US or China.

Connor Leahy: Yeah, I think there's a fundamental thing is like, look, no one's gonna risk going to jail for Mark Zuckerberg, you know? Like if we tell you like, hey, like no one is gonna do a nuclear weapon program for Microsoft, right? Like no one would risk that. No engineer, no sane scientist would do that. If we just make clear, hey, that's not acceptable. To be clear, if nuclear weapons were legal, oh, they would love to build nuclear weapons. You kidding me? These people would love to do that, it would be cool as hell. So obviously the reason they don't do that is because it's illegal. So, you know, in a good world, we don't throw anybody in prison ever, right? Like, that's the beautiful thing about deterrence. The beautiful thing about deterrents is that we do deterrence, right. Well, you don't have to, you know, you never actually have to fire a gun. You know, you never have to actually do anything, you know, as long as you actually back it up properly. Like, this is the good outcome. The good outcome is we're just like, look, guys, go home, go work on, I don't know, fusion reactors or something, you know, go work on something else or, you know, work on non-super intelligent AI, go work on. You know, there's many things you can work on and make a ton of money on and build great companies on there's Plenty of wonderful things for all these smart people to be working on just don't build the thing that we can't control and that will kill us Like I think it's a pretty fair offer

Jon Bateman: Connor, you've given us a wonderful tour of your own background and of a lot of complex technical and policy issues, even international policy. I wanna end where I end a lot of conversations with guests, which is to the person out there who agrees with you, you've sold them, right? They're worried about super intelligence now. They think that we should ban it. They think it's possible to ban it, what do you think they should do? And also what mixture personally of hope and fear you have for achieving this mission in the time that we have.

Connor Leahy: The first and simplest thing anyone can do is go to controlai.org, contract your representatives. It makes a difference. We do live in a democracy. It takes five minutes. Give them a call, send them an email. It makes the difference about this issue if you care about this issues. If you want to do more of this issue, we also are organizing citizen groups where you can meet other people who care about the issue or apply for a job we're hiring right now. We need some top talent to make sure that we can continue this mission. And when it comes to my personal interview on this... It's actually kind of interesting, if you ask me this question, at this point in time, because it's kind of like an interesting point in this regard is that my whole life, I've been basically like, we're not gonna make it. Like, it's like almost, I'm gonna do everything I can, but like, let's be honest, like the chances are very, very slim that we're gonna make. And that has actually kind of changed in like a weird way, in a sense that I am more confident than I have ever been in my life, that if we have three to five years, we're going to make it, but unfortunately I'm also the most confident I've ever been that we don't have three years.

Jon Bateman: Fascinating. Conor Leahy, thanks for joining us today. Thank you. You've been listening to The World Unpacked, a production of the Carnegie Endowment for International Peace. To get episodes delivered directly to your inbox, use the link in the description, or subscribe on YouTube or popular podcast platforms like Spotify or iTunes. Views expressed are those of the host and guests, and not necessarily those of Carnegie. Learn more at carnegieendowment.org.