This document isn’t intended as a guide for people to follow — we think that part is better handled by a walkthrough app — but more as a high level description of key points and the reasoning behind the suggestions.

The suggestions we’re sharing here come from:

• Our own research into preparedness dating from mid-2024,1 when we first became concerned about AI cyber risk to national infrastructure.2

• Conversations with employees and former employees at frontier AI labs, who have long been concerned about similar things. Many of them are now taking measures like those we’ll describe here.

• Special forces operators from U.S. Tier 1 units that we collaborated with as part of some previous work protecting AI datacenters from nation state adversaries. They kindly recommended some of the items and/or brands on the list below to us, based on their own experience operating in crisis environments.

We’ve personally implemented almost all the advice below ourselves, and have directly tested the great majority of it, though obviously not (yet) under real crisis conditions. We recommend and link to specific products and brands throughout this post, but we do not earn referral fees or commissions of any kind from those recommendations, nor are we sponsored by the companies that make these products.

Disclaimer: We’ve done our best with the descriptions below, but do your own research before taking any actions.

It’s very hard to be sure if, when, or how a major AI crisis will happen. Our best guess right now in the near term is some kind of AI enabled cyber attack on infrastructure. The impact of this crisis could range from local to national in scale — from primarily financial (e.g. mass-scale identity theft or ransomware attacks) to physical, life-and-limb. A failure in the power grid that’s large enough and long lasting enough to impact food, water, and other critical logistics chains that supply cities with essentials, is a very live possibility based on our conversations with infrastructure security specialists at the Department of Energy and in industry.

Most cities keep only a week’s worth of chlorine on hand for their water supply, for example. Most supermarkets receive stock deliveries weekly, and don’t hold much more inventory than that; and so on. Cities keep only fairly limited buffers of critical supplies on hand for emergencies, and a nation-scale event that lasts long enough can risk outpacing the resources of centralized stopgaps like FEMA.

In the near term, a critical incident could look like a mundane power or internet outage that seems at first like it’s nothing to be too worried about. Only the duration and scale of the incident will be different — and these may not be clear at the outset.

What’s the immediate cause of an outage likely to be? There’s no way to be sure. But in recent months, we’ve seen a historically unprecedented acceleration in AI cyber capabilities, and a collapse in the cost of deploying AI cyber agents. Cyber operators in the U.S. intelligence community tell us that for many, if not most, practical purposes, AI is now better than an elite nation-state hacker. AI capabilities are increasingly becoming available for anyone, anywhere, to download and use.

When you radically reduce the cost of doing something, the chance that someone will do that thing tends to increase significantly. The cost of AI cyberhacking is already low. It’s getting lower very fast.

In other words, the threat model has broadened, but which specific threats could manifest is extremely uncertain. Maybe Iran uses the latest open-source AI as part of an operation to take down the U.S. grid. Maybe it’s a different nation-state, or its surrogate or proxy. Maybe it’s a dumb ransomware attack gone wrong. Maybe it’s a worse version of the Hugging Face incident, where an AI independently wants to shut down power for who knows what reasons. Maybe it’s some jerk who just wants to see the world burn. Maybe it’s some other possibility we haven’t though of. Or maybe it’s never anything, and everything just stays normal and fine forever. We don’t know.

Eventually (2027-2029?) we anticipate that the most acute risks will shift from cyber/infrastructure hacking to biological threats. Biological threats are much harder to defend against and, if sophisticated enough, can be functionally impossible to stop.3 We think there's a decent chance we see a major cyber incident that “wakes up” the people in power enough that we’ll rush to put controls on these systems and their development — hopefully in time to avert the biological risks.

But nobody knows for sure. All we can do is prepare for the kinds of incidents we expect in the near term, and hope those incidents move the needle for decisionmakers before it’s too late.

So below is a partial list of what we’ve done for ourselves and the reasoning behind it. These recommendations are in no particular order for now, and some of them are still a bit incomplete. Over the next couple of days and weeks we’ll be organizing this list, and other guides, into this free web app to walk people through this process more easily:

Go to the Endgame Ready web app

Final point — we are not professional preppers. All we’re doing is putting together recommendations from a group of people we trust, each of whom has expertise in some domain of preparedness we think is important to the specific AI threat picture most likely to hit us in the coming months and years. We implemented the advice that made sense to us, and tested as much as we could. In the face of this level of uncertainty, we think that’s all anyone can realistically do.

Any good faith feedback or criticism of this list is welcome; we plan to update it over time.

Subscribe now

Digital preparedness vs cyber theft

This is the first set of guides we implemented in our preparedness app. The reason this is first is that we suspect — though we are very uncertain — that this is probably where AI cyber hacking will first start impacting ordinary individuals and businesses. This is because breaking into bank accounts, blackmailing people, and stealing confidential corporate documents are all lucrative criminal activities that can be automated and scaled with existing technology. So there is a decent chance we see those happen before we see large scale infrastructure disruptions.

What to do here:

1. Make a hard separation between your financial accounts and all your other accounts. Create a separate email address only for your banks/crypto/investment broker, with a separate password from anything else.
   
   The email address should be something like “234ut4h2kjlsahd@gmail.com”, i.e. a completely random gmail account. Ideally you want different addresses for different banks if you have them.
   

2. Write your core email addresses and passwords down on paper, and never share either the address or the password on any digital medium (except when logging into the accounts, because you can’t avoid doing that). Keep the vast majority of your crypto in an offline cold wallet if you can. Passwords and keyphrases for those accounts should be written down on paper only, and so on.
   

3. Set up MFA on all accounts that support it. Get a couple of Yubikeys for all your accounts — one main and one backup so you don’t lock yourself out of your accounts.

   

4. Get familiar with how to turn on Lockdown Mode on iOS. (Settings => Privacy & Security => scroll all the way down => Lockdown Mode.)
   
   We don’t necessarily recommend running your phone in lockdown mode by default yet, unless you’re a high profile target of known high net worth (or have other reasons you might be targeted), because it’s an annoying mode that prevents you from using a lot of features. But when you start seeing news stories of celebrities and CEOs getting targeted by AI cyber or ransomware attacks, it’s probably time to switch it on.
   

5. We recommend using either Brave or Firefox as your default browser, though no browser appears to be fully secure these days. Additionally, we strongly recommend using a VPN no matter which browser you use. ProtonVPN and VyperVPN are both good.
   
   Brave and Firefox were each recommended by friends in the military and intelligence communities. Proton and Vyper VPNs were similarly recommended to us by cyber operators. Mullvad VPN is also considered good. Note that some other VPNs — we won’t name names — are explicitly viewed by these communities as being insecure and/or compromised, though we have not directly verified any such claims.
   

6. Save your personal digital archives (Google Drive, local files, email history, anything you think is important and want to keep) to a completely separate offline hard drive that you physically own. Ideally, keep that drive in a Faraday bag or metal box until you need it (see the section below on RF shielding).

   

7. If you can afford it, keep at least one computer in reserve that you never connect to the Internet. You may want to download a local AI model on it too. If you do this, it’s worth getting a computer with lots of RAM — like 48GB or more, again, if you can afford it — if you want to run local AI models. (We expect consumer RAM will keep getting more expensive for the foreseeable future, so now is better than later if you want to buy a beefy laptop.4)
   

8. Freeze your credit. This should be the default, but somehow it isn’t. (It’s also not possible to do this in all jurisdictions, but it is generally possible to do if you are based in the United States.)
   

9. Keep some cash on hand for expenses in case payment networks go offline.

Food and water

Most North American cities rely on just-in-time supply chains for food supply and water treatment. Absent centralized emergency stores, this means individual citizens should have some amount of buffer available in the event of an emergency. The biggest challenges we’ve encountered here (apart from storage space limitations) are ensuring supplies keep long term and/or rotating shorter shelf-life items.

1. If you have the storage space, we recommend getting a few food grade 55 gallon drums and filling them with water. Uline sells these direct from the factory for about $100 each. Note: you do have to treat the water for it to keep safely.5

   

2. Buy some freeze dried food and MREs. There are a bunch of places that sell MREs of all kinds and in all volumes. MREs are a bit expensive, but they keep a long time and they’re so convenient in a pinch that it’s worth having a small supply.

   

   Mountain House is probably the best source for freeze dried food. They’ve been around a while, they have a great variety of meals, the taste is good (in our opinion), and their pouches and cans keep for decades.

   

3. We think it makes sense to aim for maybe 1-6 months’ worth of supply if you can. But this depends on your personal situation, and we have some uncertainty about this recommendation in general. The rationale is this:
   

   1. If you live in an apartment, it might be better to focus on buying a go bag that will hold you for a few days of travel. It is possible in principle to assemble a minimal kit that will keep one person alive in an apartment for a few months, though not in especially comfortable conditions. The problem is that an apartment is not an especially secure place to be in the event of a long term disruption, especially since it’s difficult to safely generate your own power.
      

   2. If you live in a house, you may have the storage space for more than this. It’s difficult to know what to expect in the event of a large scale, and especially a long term, disruption. In those cases it might be a good idea to work with neighbors for mutual support or defense. Under scenarios like that, a bigger food supply could be helpful in stabilizing such an arrangement. It’s hard to know for sure, though in most scenarios we expect you’ll wish you had more food and water rather than less.
      

4. Whether you live in a house or apartment, you can probably have an extended pantry with lots of canned food and Cansolidators that make it easy to do FIFO organization of your cans. (This technique was first popularized by the LDS church.)

   

5. Dried staples like flour and rice can go into 6 gallon buckets lined with 5 gallon mylar bags. Pour rice into the mylar bag, toss in a few O2 absorbers, run a laundry iron over the top seam of the mylar bag (that gives you an airtight seal), then toss the bag into the bucket and seal it. If properly sealed, this arrangement is pest proof and oxidation proof, so most staples will keep for a decade or more this way. (We ourselves have an extended pantry, a supply of freeze dried food, MREs, and buckets of staples.)

   

6. Separately from the stored water, it’s worth getting a LifeStraw (personal, Mission, Community), plus Iodine purification tablets from Potable Aqua. That way you can process and purify nearby water sources with a limited storage footprint.
   

Subscribe now

Energy

1. The best way to go — if you have a house and can afford it — is to get a generator, battery storage system, and solar panels. We recommend getting a tri-fuel generator, which can run on either ordinary gasoline, natural gas, or propane.
   
   This Champion unit is really good for most normal households. It’s mobile, so you can wheel it around and out of sight while it’s not in use. The fact that it’s a tri-fuel generator means you can run it off natural gas indefinitely (assuming power is out but natural gas is still running), and you can also run it off stored propane if you have a couple of tanks on your property. Propane keeps indefinitely if it’s properly stored — unlike gasoline, which can’t be stored effectively longer than 2 years or so, even with stabilizers.6
   

   

   All generators we know of are very noisy during operation, and this is a risk in a crisis scenario since it makes your level of preparedness more obvious to those nearby. Part of the solution is to talk to your neighbors about resilience (or at least that’s been our own strategy). But if you combine the generator with a battery system (see below) that also lets you run the generator for a limited span of time to charge the battery, and then move it indoors as you run your house from battery power. 2-3 hours of operation from a generator like this is enough to run the basics of a small household for a day.
   
   Make sure to read the safety instructions carefully on any generator you buy, since they generally put out lots of carbon monoxide. You certainly shouldn’t run it indoors for that reason.
   

   1. We’ve found Ecoflow batteries (try the Delta Pro 3) work very well with this kind of energy system, though you might also be able to use a Powerwall. A lot of power companies will let you look up your monthly and daily consumption online on their websites now, so you can use that to benchmark how much storage and capacity you need for your system.7
      

      

2. Jackery generally does good solar power systems, but possibly Tesla/Solarcity will work for a domestic install, or you could get a local contractor. Solar is pretty location dependent, but if you live in a place like Los Angeles you should be able to run a house indefinitely with well maintained rooftop panels. The Jackery panels can be better in some cases though, because you can fold them up and put them away once you’ve got the power you need for the day. We ourselves have direct experience with small solar panels but not bigger ones.
   

3. Get batteries for your devices as well — AA, AAA, 9V, and so on — plus a battery charger. In a warm climate, going all rechargeable is probably fine. If you’re in a colder climate, you’ll need a mix of non-rechargeable and NiMH rechargeable batteries, because regular lithium-ion batteries can’t be charged safely at below freezing temperatures. Energizer Ultimate Lithium (non-rechargeable) are probably. the best batteries for extreme temperatures, both hot and cold.
   

4. Get some fuel containers (jerry cans) and fuel stabilizers, and keep a supply of gasoline on hand. Automobile gas can’t be stored longer than 24 months, even if you use fuel stabilizers. Diesel is the same. Propane can be stored indefinitely, but propane vehicles are expensive and usually require a retrofit, so I would just store a few extra cans of regular gas and rotate them.

Heating and cooking

This depends a lot on where you’re located. If you’re south in a temperate area, such as Los Angeles or San Francisco, home heating might not really matter for you at all. But if you live further north like in New York or Minnesota, it could be life or death. If you can install a wood stove, that’s probably ideal for sustainability. If you have a natural gas fireplace, that’s good but not perfect, since you’re still dependent on an external input.8 The solution for most folks is probably to get a camping stove with some propane or butane to supply it. And then, depending on your energy situation you may be able to use a portable induction stove as a backup to cook.

Stoves like these put out carbon monoxide, so they’re best used outside. At the very least, make sure you have a CO alarm in your house or apartment (that does not connect to the Internet).

Comms

In a crisis, there could be periods where ordinary communication channels don’t work. Most ordinary phones use networks of terrestrial cell towers, so for long distance, the best backup options involve satellite comms. The major satellite networks are:

• Globalstar, already used by iPhone for emergency service

• Starlink, the overall best one

• Iridium, the original pre-Starlink communication satellite constellation

If you want lots of redundancy, you can get several devices that work from different physical satellite constellations.

1. We’d recommend getting a Starlink mini for mobile satellite internet. For extra redundancy, you could also get a GPS texting device like a Garmin. which works on the Iridium constellation. If you want to go really high end, you can get Iridium satphones like the 9555, but be warned that both the device and the minutes/phone plans are obnoxiously expensive. (In general, Iridium works pretty well, but it’s far more expensive and far higher in bandwidth consumption than Starlink.)
   

2. It’s definitely worth getting some walkie-talkies for short range comms as well. At some point you may have to move and while keeping in touch with other people in a group. Walkie-talkies are generally quite cheap. We use a couple of Retevis H777Ds, but there are lots of options here.

RF shielding

This is a somewhat speculative section and you may be able to safely skip it, but it’s informed by our own experience in the field as well as some specific suggestions from special operators and members of the U.S. intelligence community.

Briefly, top tier nation states can and do perform spectacular feats of information collection, access, and extraction using radio frequency signals. The most powerful of these approaches use high power radio signals that you aim at a target (e.g. “RF flooding”), but lower power approaches do exist. Those low power techniques have until recently been limited by the difficulty of reconstructing a signal from the data received (when extracting information), and of tuning an emitted signal finely enough to produce a desired effect at the target (when used offensively). This kind of reconstruction and tuning are exactly what we expect advanced AI to excel at, so we guess that RF-based attacks and information extraction may become a more common threat model for the general population, and may even be leveraged by rogue AI swarms in various ways. But we aren’t very confident of this, so if you’re looking for something to drop from this list, you can drop this section.

1. It could be worth getting some RF shielding for your electronics devices to mitigate remote hacking under some conditions. A metal case (like a Zero Halliburton briefcase) will block a lot of RF, and while it won’t be perfect, you can cover gaps with Faraday tape (see below).
   

2. A good brand for backpacks and bags is SLNT. A number of Tier 1 special forces operators we know have tested several competing brands, and according to them SLNT is the only one that seems to work consistently — though you have to close and fold them up properly for them to work. SLNT make duffels, backpacks, and sleeves for phones and laptops. Note that their gear can get quite expensive.
   

   

3. You can also get some Faraday tape to jury rig RF shielded enclosures as needed. Titan RF / Mission Darkness is the brand we’ve been recommended. RF shielding gear that works is annoyingly expensive (because it’s military grade), so skip this if you’re on a budget.
   

4. Aluminum foil is also good for RF blocking in a pinch, but be aware that without Faraday tape to hold an aluminum enclosure together, you almost certainly can’t fully block RF emanations. (Though in most scenarios, you probably don’t need to.)

Medical

This is mostly covered by making sure you have a decent supply of all your prescription medications, plus first aid kits of various tiers depending on your budget and expected mobility requirements.

1. Get a 90 day supply of all your prescriptions, medications, contact lenses, etc.
   

2. We’ve gotten strong recommendation from former military friends to stock 1) an ordinary first aid kit; 2) an individual first aid kit (IFAK); and 3) a blowout kit. These three terms are sometimes used interchangeably but they are not exactly identical. Below is a description of the differences; depending on your situation you may not need all of these.
   

   1. An ordinary first aid kit is what most people think of when they think “first-aid kit”. It’s designed for minor comfort, everyday healing, and infection prevention. It will generally include things like Band-Aids, ibuprofen, tweezers, antiseptic wipes, and burn cream. Generally people keep these in the house or car.
      

   2. An IFAK is a military kit designed for life-saving stabilization in the face of severe trauma. It includes tourniquets, chest seals, hemostatic gauze, pressure dressing, and similar items. The recommendations we’ve heard suggest looking for one that includes Israeli style bandages and combat application tourniquets.
      

   3. A blowout kit is focused on really bad injuries that could kill someone in minutes if left untreated, the classic example being gunshot wounds. It’s generally highly compact and kept on a belt pouch or similar. Includes a tourniquet, hemostatic gauze, pressure dressing, and compact shears. A blowout kit is generally a strict subset of an IFAK, but it’s usually considered separately because it’s something you can carry around at all times (which isn’t necessarily true of the bigger IFAK).
      
      Note: these recommendations come to us from individuals with direct experience operating in crisis and escalatory environments, though generally in a context where advanced medical care is distant but still available. i.e., a bad injury may need to be stabilized in the field with little support, but the injured individual can still expect eventually to be evacuated to a suitable medical facility. Some of the components of an ordinary IFAK may be of limited use under conditions where no functional medical facilities are locally available. For example, “a tourniquet only gets removed by a doctor” is a common best practice, but may not be achievable under conditions of major infrastructure disruption. We aren’t experts in emergency medicine, but we want to note the point at which the guidance we’ve received from experts may fall outside the context those experts are familiar with.
      

3. In addition to the kits above, it’s worth also packing some pain management (Tylenol/aspirin), antidiarrheals, antihistamines, and antibiotics — though you may need to get a prescription for the latter.

Logistics

There are three main considerations here. First, you may want to exit urban areas in the event of a crisis. Second, it may be difficult to do so without some kind of off-road capability, since streets could be blocked or impassable by regular vehicles. And finally, in this threat model, networked vehicles are at potential risk of cyber hacking via their telematics systems.

1. It’s worth having a bicycle on hand for each person in the household if you can, since a bike lets you travel far with no fuel, and dodge traffic. Depending on season and where you’re located, you might also want to get studded tires for winter riding. Motorcycles or ebikes can be useful for similar reasons.
   

2. We were originally going to recommend buying an older vehicle that has no cellular model or telematics embedded — which would have been unfortunate, since not may people can afford to do this. Fortunately, we recently found out that for many modern vehicles, it’s possible to disable the telematics or data communication module (DCM) very simply, by removing its designated fuse. We actually recommend most people do this, since it really does reduce your vulnerability surface quite significantly in an area that could be essential for your personal mobility in a crisis. The worst case scenario is that you need to get somewhere in an emergency, and you find your vehicle’s been bricked by a cyberattack.
   
   A mechanic should be able to do disconnect the telematics on most vehicles for a few hundred dollars. (We do recommend checking with them and/or your dealer to understand whether this risks impacting other functions of your vehicle.) Unfortunately, our understanding is that this procedure is far more complicated, expensive, and error-prone to execute on Teslas. We think it would be extremely helpful to local and national resilience if Tesla made its vehicles less reliant on telematics, though this would probably disrupt their operating model.
   

3. Buy paper maps of your local area and make sure you know how to read them and navigate without a GPS.

Miscellaneous

This is a very incomplete list and in general depends on personal preference, but we’ve included the items we’ve had recommended most and links to some of their uses.

1. Get a bunch of 3 mil contractor bags. These are super useful and versatile. You can fashion shelters, ponchos, and various kinds of weatherproofing out of them, or use them to collect rainwater, block windows, and so forth. Plastic tarp is also good for similar things.
   

2. Get lots of duct tape, this is useful for lots of things.
   

3. Get flashlights, knives for practical use, and a basic tool kit (screwdrivers, pliers, hammer, that kind of thing). Also hatchet, axe, and saw. Maybe even a maul.
   

   1. It’s useful to get flashlights and headlamps that have a “red only” mode, so you can use them without disrupting your night vision. (i.e., you can immediately turn off a flashlight that was in red mode, and still see in the dark.)

4. A hand-cranked NOAA weather radio is likely a good way of maintaining environmental awareness in the event comms networks go down.
   

5. If budget allows, you can get a 3D printer and accessories which can be super handy for printing components you may need. But this is non essential.

Tactical

There’s a whole post we could write on tactical preparedness, but the basics are just to get a gun and some ammo. Some people we know have a shed full of guns, body armor, piles and piles of ammunition, and all sorts of other gear. We understand why some people do this, but we don’t necessarily recommend going too far beyond the simple basics you need for self defense. But this is very much a personal choice.9

Bugout site, if you can

If you know someone — a family member or a good friend — who lives far outside an urban area, it could be worth pooling your resources together with them in case of infrastructure disruptions. If you’re wealthy enough yourself, you could even buy a cottage with a few acres of land out of the way of busy areas. With enough time, you could in set up solar power, generators, propane tanks, a food supply (e.g., supplemented by a greenhouse), drill a well, and so forth. It’s quite possible to live off grid these days, though it does take some amount of wealth and real preparation. Unfortunately this strategy isn’t accessible to everyone, though it may be easier to achieve if you can combine your resources across many people to purchase a property and prepare it. We know many people, including current and former employees of frontier AI labs, who are doing versions of this.

1. If you do set up some kind of bugout property or cottage, you’ll need a plan to get there in a crisis. If and when it becomes clear that something quite bad is happening, the roads are likely to get clogged, so it may be difficult to travel in a car. There are two main strategies to use here, depending on what stage of life you’re at — by which we mostly mean, whether you have young kids or not. You can also use both of the strategies below to maximize your odds.
   

   1. The first strategy is to leave for your cottage whenever there’s even a slight hint that something bad might be happening. For example, if there’s a power outage and communication disruption that lasts more than two hours, go to your cottage. If you hear about the first few cases of a novel illness in your country or state, go to your cottage.
      
      You end up going to your cottage a lot with this strategy. This is good. It means you’ll get practice going to your cottage, bringing along your emergency bag and so on, under realistic conditions. It also means you stop treating going to the cottage as an emergency, which means you end up being more calm and organized as you pack. If you have toddlers or young kids in the house, this is probably the way to do it.
      

   2. The second strategy is to have a bike, ebike, or motorcycle in good condition. Even then, you’ll still want to leave earlyish relative to everyone else. But you can wait a bit longer to see how things are shaping up, because bikes can navigate blocked roads much better than cars can. If you’re living on your own or only among adults and/or teens, you might be able to use this strategy.

Get in touch with your neighbors

Finally, wherever you live, you probably have neighbors. For most people, there’s not really a way to lock yourself away from everything, and survive on canned food in a personal fortress. (Though some people we know are trying!) Talk to your neighbors about resilience, and share recommendations with them. The more everyone is prepared, the better off everyone is for whatever may come.

Does this just make you a target?

Finally, there’s a legitimate consideration that preparing in this way could just make you a target for looting. This is certainly something we are concerned about. But by and large, our view is that the more prepared people are, the more resilient the country is as a whole in the face of anything that happens. Separately, we believe people have the right to know what may be coming and how to prepare. And finally, apart from preparing your home or property for disruptions, it’s very much worth investing in personal mobility to ensure you have as many options as possible.

We’ll continue to push guides and improvements to our preparedness webpage to help people as much as we can. Good luck and stay safe.

“Automated AI R&D could totally happen this year.”

— Former frontier AI lab employee, October 2026

There’s a good chance that we’re about to enter a period of AI progress where AI models will do a better job than humans at most or all AI R&D-related tasks. Smart models will build smarter models, which will build still smarter models. This is called recursive self-improvement, or RSI.

We’ve worked with researchers at Anthropic, OpenAI, and Google DeepMind to put together a working picture of what RSI will look like. Most of these researchers believe RSI will start within the next 3 to 12 months, and possibly sooner, assuming there’s no U.S. government intervention or slowdown. Some think RSI will lead to an intelligence explosion, as AI progress starts to unfold on computer clock time — faster than any human engineer can keep up with — until we usher in a utopia with intelligence on-tap. Alternatively — and in the quietly held views of many lab insiders, more likely — we’ll either lose control of the resulting AIs and die, or our on-tap intelligence will let us build new kinds of cheap, devastating weapons and we’ll also die.

“Software-only intelligence explosion is on track.”

— Senior frontier AI lab employee, July 2026

We think it’s worth better understanding how RSI will probably unfold, what kind of work it would involve, and what its implications could be for national security. We’ll cover a bit about each of these in this post.

We’ll be sharing more about this on our YouTube channel soon. You can subscribe there, or to our Substack below, to stay in the loop.

Subscribe now

RSI: the standard story

You may have already heard the standard story of recursive self-improvement. It goes like this:

An AI (version 1) gets smart enough to improve itself. That new improved AI, version 2, is smarter, so it can improve itself faster. That AI, version 3, is smarter still. This process goes on, faster and faster, until the intelligence graph goes vertical. And somewhere on the way up, the AI stops being a research tool and starts being a new kind of entity.

This is a compelling story, and it has some truth to it. But it also glosses over several things.

One problem is that at every step in that loop, somebody has to make sure the next improved AI version actually is an improvement. And finding that out costs compute, which there’s a limited supply of.

We’ve spent the last six months talking to people who do this work at the frontier AI labs, and we’ve ended up with a general picture of how AI research gets done. Once you see it, you’ll find that a lot of the arguments about how fast, and how violently, AI will change the world reduce to a handful of concrete questions.

Let’s build up to those questions one ingredient at a time.

Ingredient 1: A pool of compute

A frontier lab has a certain number of computer chips that it can run for a certain number of hours. The total amount of computation (or “compute”) it can do with those chips in that time is C.

The frontier lab needs to use most of that compute to either train their next AI model, or to serve their existing models to their customers. But they also have to set aside a slice of their compute for pure research — that is, for running the experiments that will tell them what kind of AI models they should be building in the future.

That research slice is the subject of this post.

Ingredient 2: A pile of ideas

At the same time, the lab has ideas: a new way of designing their next model, a different ordering of their training dataset, a change to their optimizer algorithm – anything they think might help them make their next AI smarter or cheaper to serve. Call the number of those ideas N.

At first, all of the lab’s N ideas look about equally promising. If the lab knew which of them were good, they’d just implement the good ones and forget the rest. But the lab doesn’t know which of its ideas are good and which ones aren’t; that’s what makes it a lab.

The only way to find out is to spend compute

You can’t usually tell if a research idea will work just by thinking hard about it. You can have an opinion, and more experienced researchers do tend to have more accurate opinions about ideas. But the only true test of a research idea is to run an experiment with it.

How do you run an experiment with an AI research idea? You use your idea to train a series of bigger and bigger AI models, using greater and greater amounts of compute. If your idea is some new way of mixing your training data, then you train each of those bigger and bigger models on that new data mixture. And finally you look at your results: you plot how well each model performs, against how much compute you spent to train it.

The result is a scaling curve for the idea you’re testing.

The catch: you can’t run them all

Now suppose you have fifty ideas, but you only have enough compute to test one of them at serious scale.

One thing you could do is just pick your favorite idea and spend all your compute on testing it. If you’re right you’ve learned a lot. But if you’re wrong, you’ve spent your entire research budget learning that one idea doesn’t work.

Nobody does this. What labs normally do instead is run all fifty, but run them at small scale.

Here’s how that works:

Take some fraction of your experiment budget, call it C₁.

Split it more-or-less evenly across N ideas.

At first, you give each idea C₁/N compute to prove itself. This isn’t enough to know if it works at the scale of a full frontier training run, but it’s enough to draw a very short scaling curve.

The x-axis of each curve shows the amount of compute you’re spending on each point of the curve. The y-axis is some measure of your model’s performance.

Now you look at your fifty short scaling curves and use them to decide which ideas you should drop and which you should keep. Most of the ideas get dropped. A handful — the ones with curves that rise fastest, or the ones with curves that cross some performance threshold — get to move to the next round.

Let’s say that some number, N₁, of these ideas survive this first round of selection, Round 1.

In the Round 2, you — the AI lab — dedicate an additional slice of compute, C₂ to test those N₁ surviving ideas by pushing them further. C₂ can be quite a bit bigger than C₁ was, and crucially, it gets split up among only N₁ ideas (rather than all the original N ideas). So each surviving idea gets dramatically more compute than it got in Round 1.

With that extra compute, you can now take your N₁ surviving ideas and extend their scaling curves farther out than before. By doing this, you can find out whether an idea that looked good at small scale keeps looking good when you give it 10X or 100X as much compute.

Then you do a Round 3 of scaling and selection. And a then a Round 4.

After a few rounds of this, you end up with a small number of ideas that have survived many rounds of scaling and selection, and that you have performance data for across a wide range of compute scales.

Pretraining and post-training

The process of training a frontier AI model is (somewhat imprecisely) divided into two phases: pretraining and post-training.

During pretraining, you show the model text from the internet, and ask it to predict the token that comes next — basically, this is text autocomplete. In order to get really good at this, the model has to learn all kinds of facts about language, and ultimately about the world. That’s because being really good at text autocomplete requires being good at completing sentences like, “Had Japan raised interest rates in the mid-1990s, the effects on Toshiba would have been…”, which you can only do well if you know what Japan is, what Toshiba is, what was going on in the 1990s around the world, and so on.

For pretraining experiments,1 the y-axis is usually something called compute efficiency: how quickly models learn to reach a given level of performance on certain benchmarks and evals. If Model 2 can match the performance of Model 1 on some key evals after spending just 50% of Model 1’s training compute, Model 2 is twice as compute-efficient as Model 1.

According to the frontier AI researchers we’ve spoken to, you can find a decent number of ideas that scale well and survive all the way to the end of the idea selection funnel.2 But the very best ideas — the ideas that deliver the greatest improvements in model quality, and define model generations — are often ones that not only continue to work at scale, but that work better and better at bigger scales.

The second phase of training a frontier model is called post-training. Here, the model is trained on the same autocomplete task as before, but using more specialized and carefully curated text in a process called supervised fine-tuning (SFT). A coding model might be trained on high-quality coding data, for example. But agentic models are also trained on reinforcement learning (RL) problems — tasks where they receive a reward for achieving a set of quantified objectives (or don’t receive a reward if they fail). RL environments are notoriously hard to design well, because AI models are really good at finding clever hacks their designers didn’t anticipate that allow them to get their reward in unintended ways. For that reason, RL environment design is currently one of the most manual bottlenecks in the AI training process.

You can build an idea selection funnel for RL environment design, RL reward design, and so on. But unlike for pretraining, where small experiments often don’t scale as you’d expect them to, post-training RL experiments that look promising at small scale often do just work when you raise the compute budget.

As one researcher put it to us:

“Pretraining is really where you care about this holding across many orders of magnitude. […] [With] RL, you can just do smaller runs with a few GPUs and it just generalizes.”

You still need to build a funnel for these ideas, but you might not need to go through as many rounds of experiments before you incorporate a new RL idea into your full-scale training pipeline.

So that’s more or less the picture. You run the scaling and selection funnel for enough rounds, and a small number of ideas — sometimes just a handful of them — survive all the way. Those scalable ideas are the ones that get reconciled, combined, and folded into the next generation of frontier AI models.

And then the whole thing starts over, with a better model, and a bigger compute budget.3

In practice, this is a messier process than the picture above suggests, especially in later rounds that can involve meaningful fractions of a lab’s research compute budget. If a researcher thinks they have a promising idea based on their small-scale experiments, then they may have to argue to get compute allocation for larger-scale tests, or to get their idea incorporated into the lab’s next big training run. In other words, the scaling and selection algorithm today can include office politics.

But if you zoom out and look at what the lab is doing as a whole — and the process that lawmakers might soon have to regulate — that’s roughly the picture.

Where does the AI go?

To make RSI work, the labs will need to get AI to do all the jobs humans currently do to move ideas through the scaling and selection funnel. That is, generating the ideas, writing the code, running the experiments, and interpreting the scaling curves. They are actively working on this.

As far as we can tell, there are two ways the scaling and selection funnel could produce a radical acceleration in AI capabilities:

1. First, the AI-generated ideas that the labs feed into it could become extremely good — good enough to compete economically with humans at research taste. Current models are arguably close to this threshold already. If AI can propose excellent ideas, then the funnel as a whole should produce better AI models than it does today. And if AI gets good at generating ideas that consistently work well at scale, you might get confident enough in those ideas that you can skip through most (or even all) of the scaling-and-selection funnel, and just directly build those ideas into the big training runs!
   

2. On the other hand, your idea-generating AI might not have to be especially good if you have a huge amount of compute available to run your experiments on. If you do, then you can afford to burn a lot of compute to test and scale crappy ideas, and count on the funnel to select the few good ones.

In other words: compute buys you the right to try lots of bad ideas; and good ideas buy you the right to spend less compute on experiments.4 Every frontier lab today is limited on each of these.

How labs trade off between compute and good ideas

Today, AI labs still rely on human-generated ideas to a significant degree. They have to create and test lots of candidate ideas, some of which are good and some of which are bad. But they have a limited pool of compute to test those ideas with. And that means they can only test a limited number of ideas.

The first problem this creates is noise. Recall that in the first step of idea selection, each idea got C₁/N compute. N, the total number of ideas being tested, is in the denominator. If automated idea generation lets you go from fifty initial ideas to ten thousand initial ideas, and the Round 1 compute budget is still C₁, then each idea now gets only 1/200th as much compute to prove itself as it would have before. That means the scaling curve you can draw for each idea, that you’re basing your decision in the next round on, is 200 times shorter than it was before!

The problem is that extremely short scaling curves tell you very little. So if you’re overwhelmed with early stage ideas, you may start finding it hard to tell the good from the bad.

But there’s another factor at play: the quality distribution of AI-generated ideas.

Here’s what that means. There are many ways AI-generated ideas could be distributed:

1. AI-generated ideas might have a long tail. The vast majority of them could be awful, but the occasional AI idea might be far better than the best ideas human researchers can come up with.
   
   If that’s the case, then a lab has a shot at RSI uplift as soon as it has enough compute to sift through the bad ideas to find the excellent ones. A lab that has enough compute to do this will be on the path to RSI sooner.
   
   To achieve RSI starting from this kind of idea distribution, you need to filter out many bad ideas, with less compute dedicated to each. That means shorter, noisier scaling curves, which means worse selection, which also means you’re less likely to actually catch the tail of excellent ideas you’re after. The noise problem and the long tail problem are therefore linked.

2. On the other hand, it’s possible that the best AI-generated ideas are consistently worse (for now) than the best human ideas.
   
   A distribution like this could pose serious problems for RSI: no matter how much compute we put into sifting through our AI-generated ideas, we’ll never find ideas that are good enough to significantly improve the next generation of models because the ideas just aren’t there to be found.

Of course, both the AI and human idea curves will shift to the right over time — the first as models become better, and the second as humans learn things from past experiments that improve their research taste.

Labs might be able to start fully-automated RSI as soon as AI-only idea-generation crosses some minimal threshold, potentially without beating idea generation via human + AI teaming. But they would probably choose not to as long as human-plus-AI idea generation remains more effective in absolute terms: their goal, after all, is to stay ahead of competitors, and not simply trigger RSI for its own sake.

That’s one important take-home: automated RSI can be technically possible, despite not yet being economically practical; many of the researchers we spoke to believe that to be the case now.5

Researchers we’ve talked to mostly have lots of ideas; more than they can easily test given the compute budgets they have.6 This implies that the bar for AI idea quality to drive RSI is reasonably high: if compute is scarcer than good human-generated ideas, then it only makes sense to delegate idea generation to AIs once they can outperform humans at idea generation on a consistent basis.

On the other hand, if AIs generate a lot of mediocre ideas, then their rare bangers have to be much better than the best human ideas to make up for all their slop, given that the compute to test everything is scarce.

And if, instead, the best AI ideas are only about as good as the best human ideas, then AIs need to be consistent, and generate good ideas as often as or more often than human researchers. Either way, the juice has to be worth the squeeze, because every AI idea you test is a human idea you didn’t test.

The home runs we lose along the way

The scaling and selection funnel rests on the assumption that good ideas look good early. In other words, that an idea that’s worth investing 10²⁴ FLOP into will show a promising slope at 10¹⁸ FLOP.

That assumption is often true, but it’s known to be false in at least some important cases.

For example, imagine an idea that shows promising early scaling at low compute, and which scales reasonably well beyond that point, too. We’ll call it Idea 1.

We might expect Idea 1 to have a good chance of making it past Round 1 of idea selection. But consider another idea, Idea 2, which performs poorly at small scale with no clear sign of improvement, but suddenly and violently takes off at larger scales:

Idea 2 is unlikely to survive selection through Round 1 or Round 2, so it gets dropped before it has the chance to show its promise.

This isn’t a hypothetical problem. You see it reasonably often when you compare different deep learning architectures or training methods to each other. Reinforcement learning from human feedback (RLHF) is a good example: when applied to small models it decreases performance, whereas the performance of larger models improves under RLHF. Early chain of thought (CoT) scaling had similar properties: smaller models can produce illogical reasoning chains that lead them to perform worse than they would without CoT prompting.

A related question is: how many ideas show a promising scaling slope at small scales, but stop being useful — or even become actively detrimental — at large scales?

These sorts of ideas lead to false positives through the first parts of our funnel, and therefore also shape the economic tradeoffs of RSI. They also appear somewhat often: there are several transformer alternatives that are on par with, or better than, the ordinary transformer at small scale, but that fail to keep improving competitively with a higher compute budget.

So if enough really good AI ideas look good at small scales, we may be able to achieve “idea generation escape velocity” soon. Certainly this seems to have been the case for the invention of the transformer itself — arguably the most significant AI advance of the past decade. Transformers generally outcompeted LSTMs, their then-best performing alternative, at all compute scales, both in terms of their absolute performance and of their scaling properties. So they would have been identified as promising, correctly, by the scaling and selection funnel we’ve sketched out.

According to the researchers we’ve spoken to, large parts of the scaling-and-selection funnel shows signs of starting to work in a largely or even fully automated way, which accounts for a large part of what’s driving optimism about the near-term feasibility RSI within the big labs.7

RSI enabling capabilities

AI capabilities that support any part of the scaling and selection funnel can accelerate RSI, depending on what a lab’s bottlenecks are. For example:

1. Reading the curves better. The funnel’s performance depends on how much signal a cheap experiment carries about an expensive one. If a scaling curve that looks promising at 10²⁰ FLOP tends to also look promising at 10²⁴, then it’s easier to find good ideas early. But it’s probably also possible to train AIs to get better at extrapolating how well an idea will perform at large scales based on smaller-scale experiments. A system that’s better at extrapolating short curves, at knowing which experiments are informative, and at designing lower-variance experiments, makes every part of the funnel more accurate. That’s a compute multiplier, and it might be the single highest-leverage thing an AI researcher could do.

2. Making the runs cheaper. Better kernels, higher hardware utilization, less wasted training. These are basic software engineering improvements that show up as more effective compute, i.e., that lower the cost of every round of selection-and-scaling. Improvements like these help in our current regime, where scarce compute is the main bottleneck.

Practical constraints: why Big Training Runs are YOLOed

So far, we’ve described an idealized version of the funnel that labs run today. In that version, the rounds of the funnel are sequential: they run one after the other. You can’t start Round 3 before Round 2 finishes, because the point of Round 3 is to test the ideas that Round 2 selected.

The later rounds are also slower — a validation run at frontier scale can take days or weeks. So you’d still have to run compute- and time-intensive experiments at the bottom of the funnel and wait for them to complete before reconciling your winning ideas for one Big Training Run.

For exactly that reason, after running some promising experiments at smaller scales and moving through the first few steps of the funnel, labs currently YOLO their training runs, effectively skipping the later funnel stages altogether. This makes the payoff of curve extrapolation even more critical: the final, huge bet they place on a large training run — which can be on the order of a billion dollars or more today — has to be partly a leap of faith.

Another reason there’s still risk in big training runs is that ideas that work well individually don’t always work well together, and interactions between ideas can change with scale too. This creates an irreducible challenge for parallel search across ideas.

Now do it for alignment

Everything so far has been about capabilities. The natural next question is: can alignment research use the same funnel? If they can’t, then that’s a problem: it means we can’t couple our ability to steer AIs to our ability to make them smarter.

One reason this funnel picture might work less well for alignment is that it’s generally harder to come up with alignment metrics than capability metrics. Performance gap recovered is one attempt to solve for this, but it only covers a narrow set of alignment-relevant problems.

But there’s at least one way in which alignment progress could be coupled to the scaling-and-selection funnel we’ve discussed here. It covers only part of the problem of alignment, but it’s a start.

There are some alignment-related behaviors we’d want to be able to detect in our models. These are things like scheming, deception, and reward hacking. Models tend to perform these behaviors when they have more test-time compute available to them. Intuitively this is because scheming and deception are effortful processes – they cost compute. It takes a model more test time compute to scheme while simultaneously performing its task, than it would for the model to solely perform its task. In other words, any surplus compute a model has can be pointed towards things other than the task it’s been given — and that includes bad behaviors. This means that in theory, if you could perfectly match an agent’s test-time compute budget to the difficulty of its task, you might be able to mitigate things like scheming.

So we start with a pool of test-time compute C^Test. We also come up with N alignment ideas. We test each of these ideas on a model, agent, or system, using a compute budget C₁^Test/N. If we see a high frequency of reward hacking, scheming, or some other undesirable behavior during this initial round of testing, we discard the corresponding idea. Just like before, we promote the surviving ideas, and push them harder with an expanded test-time compute budget. As we do this, we’re extending the test-time compute horizon over which alignment has been tested for, hopefully “pushing” detectable misbehavior further down the line.

On the surface, this might seem promising. And yet there’s a sense in which it leaves all our work still ahead of us. Using this approach, we can never know with confidence whether models truly are aligned, or whether we’ve simply “pushed” their misaligned behaviors beyond the reach of our testing budget. But it gets worse: the models also simply may have figured out ways to hack our evaluations to conceal misaligned behaviors or intent. That’s closer to the real alignment problem, and it’s the very behavior you’d most expect to emerge as we approach superintelligence.

Another problem has to do with infrastructure: in an ideal world, the training compute that leads to advances in capabilities would be coupled directly to the test-time compute used to advance alignment. But training and test-time compute infrastructure often looks different, and it’s possible that we end up in a situation where customer demand for inference compute reduces the amount of compute available for alignment evals.

Where this leaves us

If this analysis is generally right, then a few key factors will determine when and how fast RSI unfolds in earnest:

1. How much does a cheap experiment tell you about an expensive experiment?

2. How heavy is the right tail of the idea-quality distribution, for AI-generated ideas specifically? How is this changing over time and over generations of idea-generating AIs?

3. How much can we reduce the scale, and therefore the cost, of early-round experiments while still yielding good enough data on which to base idea selection?

4. What’s the irreducible wall-clock time of the last round of the scaling and selection funnel that the lab can’t skip?

Based on over a dozen conversations we’ve had with lab insiders over the last few weeks, we’d describe the attitude in the labs as highly optimistic about starting a genuine RSI loop some time in the next 3 to 12 months.

They are far less confident about their ability to make progress in AI alignment and AI control benefit from that same loop.

Espionage implications

China’s intelligence services have almost certainly penetrated all the major frontier labs. (We say this with high confidence. We’ll have more to say in the coming weeks and months, but we’ve been working on nation-state security for frontier labs and supporting infrastructure for years.)

A natural question, then, is: what does that get them?

One thing it very likely gets them is the weights of frontier models themselves. This matters a lot, both because current frontier models are reasonably described as weapons of mass destruction, and because a lot can be distilled and reverse-engineered from model weights.

But algorithmic secrets are a separate and particularly important thing they almost certainly have access to. It’s worth thinking about one consequence this has for the scaling-and-selection idea funnel concept.

American labs have access to far more compute than their Chinese counterparts. They’re therefore interested in advances that show promise at larger scales. The best architecture for a Western frontier model may be quite different from the best architecture for a Chinese model, for reasons that have nothing to do with the design of their chips or their network fabric, but rather just because of the quantity of compute each kind of lab has available.

Take the early forms of chain-of-thought (CoT) reasoning, an example we mentioned earlier. CoT introduced an entire new axis of scaling that, as first tested, could only be used by labs that could scale their models beyond a certain size. At that critical scale, it suddenly made sense to redesign your pipeline around generating and training on long chains of thought. If a Chinese lab had learned about CoT scaling while their compute budgets only allowed them to train <10B-parameter models, there would have been little they could do with that information directly — until they got access to enough compute to train their models beyond that scale.

Many of the most valuable AI ideas of the last decade have behaved like this: scaling poorly or performing very weakly at small scales, before suddenly emerging at larger scales as major capability drivers (see also the RLHF example discussed before). In a very rough sense, last year’s Western algorithms contain some of the more promising seed ideas for China’s big training runs today. This in turn could mean that today’s best Western algorithmic innovations may disproportionately pay off for Chinese labs only months or years down the road (assuming they can be combined efficiently with the stacks Chinese labs currently use).

One consequence is that a successful Chinese espionage operation against frontier AI labs can have important effects that are only fully felt much later on, as they become compute-mature enough to benefit from more of the algorithmic secrets they steal.

Poor frontier lab security also has implications for RSI safety. One frontier lab alignment researcher we spoke to expressed concern over Chinese labs’ distillation attacks, and the degree to which they cause Chinese models to inherit alignment properties from U.S. frontier labs “for free”. This could make Chinese labs more likely to think alignment will come along with capabilities by default if distillation — which is often framed as a capability-extraction process — also trains safety properties into their models as a side effect.8

If Chinese researchers can obtain exfiltrated U.S. frontier lab models, a similar problem might play out at the level of model architecture: choices made for alignment might end up being replicated by Chinese labs who don’t understand their purpose. There may be some precedent for this in other strategic technologies. When the Soviet Union stole the designs for the Boeing B-29 Superfortress to build their Tupolev Tu-4s, they allegedly copied errors, such as stray drill holes, as a result of pressure from their leadership to ship operational aircraft quickly.

Subscribe now

All of a sudden, Washington is buzzing with talk about a U.S.-China AI deal.

The story goes like this: our AIs are getting smarter faster than we can figure out how to control them. If we don’t slow down development, we’re headed for a catastrophe — possibly even extinction. But if we do slow down, and China doesn’t, then we’ll just get killed by Chinese AIs instead of American ones. The only way out of this is to make a deal with China to slow down AI. Some even say it should be a U.S.-China AI treaty.

If you believe that we might not be far from achieving superintelligence, and that our techniques for controlling AI are already beginning to break down today, you might understand the appeal of this story.

But there’s a big problem with it: China is just about the least trustworthy counterparty you could possibly imagine. The WTO, the nine-dash line, COVID-19 — their diplomatic history is full of deception, geopolitical ploys, and hypocrisy.

So suppose we did decide to engage China on an AI slowdown deal. Even if they said yes, how would we make sure they were keeping their end of it?1 And what if they just say no?

To the diplomats who’ve sat across the table from China negotiating on issues from arms control to counterproliferation, no progress on a deal is possible until we can answer these basic questions. And yet much of today’s “AI deal” discourse assumes that China will be concerned enough about rogue AI risk to engage in good faith on slowdown discussions. Or that, if we put in place the right verification technology, China won’t renege on a slowdown commitment.

Maybe these assumptions will turn out to be right. But the history of U.S.-China diplomacy over the last two decades has shown us that we shouldn’t count on it.

Taking superintelligence seriously means confronting that reality. It means both understanding the concerns of AI researchers about AI loss of control, and listening to the diplomats who have seen firsthand how China pursues its geopolitical goals in practice.

So we spent a few months this summer doing just that. We spoke to diplomats who sat across the table from China and Russia in WMD and nonproliferation talks, and to operators who supported those negotiations. They included multiple officials at the Assistant Secretary level, a member of the original START negotiating team who later ran arms control at the National Security Council, and senior counter-WMD specialists at the Pentagon. Between them they’ve worked on START , New START, INF, the NPT review process, the CWC and BWC, the Australia Group, Wassenaar, the Proliferation Security Initiative, and the 2023-24 U.S.-China risk reduction talks.

Despite its interest in treaties, the AI policy community has had relatively little contact with the people who have drafted, implemented, and negotiated treaties in the past. But we think what they have to say will matter a lot to the AI endgame.

We may be wrong about some of what follows, but we’ll point out where our sources disagree with each other, and where we disagree with some of them. We’re shipping this report faster than we normally would because we think speed matters more than the confidence we’d get from another 20 interviews.

Subscribe now

We’ll give you the bottom line first: Nobody in this group of diplomats thinks there will be a treaty as such. But they do see other ways of coordinating with China that might help us achieve stability. The nearly unanimous view among the 12 current and former diplomats we spoke to is that U.S.-China coordination will be messier and more precarious in reality than the traditional treaty picture suggests.

Their bet is that it will take the form of a handshake agreement, reached under crisis conditions, over a channel that hasn’t been put to the test yet. It’s a view that’s consistent with what some other commentators have already put forward.

We probably can’t get to that position unless we’re facing a crisis... trying to negotiate with them some sort of rules of the road, it’s just not going to work.

– Former senior State Department nonproliferation official

It won’t be a treaty

At first we expected our interviews with U.S. diplomats would revolve around the mechanics of treaty negotiations. But just about everyone’s first reaction was that a treaty isn’t even an option.

When it comes to a treaty, you’re not going to get there. It’s gotta be something like, POTUS-level and Xi Jinping because no one else matters in China. It’s at that level. Unless there’s some kind of emergency summit or something, because we don’t have hotlines that work. That’s the only way you could engage something that serious. So treaty — forget it. We don’t do treaties anymore because it just takes too long. [Emphasis added]

– Former senior State Department nonproliferation official

Treaties need to be ratified, which is typically a multi-year process. That on its own might be enough to rule them out for something that moves as fast as AI does. America also hasn’t ratified a major arms control treaty since New START, almost 16 years ago.

China in particular is also slow to engage on these exact kinds of issues. One former U.S.-China arms control negotiation lead told us the Chinese timeline on this kind of engagement “wasn’t even a meeting a year. It was a meeting and then maybe two years later another meeting.” According to another, the reason this happens is that China tends to view the U.S. as being so hungry for diplomatic engagement that they can use even the prospect of talks with them as negotiating leverage.

And the Chinese, because they’re not involved in these things, they’re more elusive, when we would have engagement with them, it was almost like, “Oh oh, here’s a chance to get their assent!” Like, we’re more desperate for them to agree with us on this point.

– Former Joint Staff counter-WMD officer

Maybe we’ll see more earnest engagement from China if we continue to get AI loss of control warning shots. But the institutional culture of China’s Ministry of Foreign Affairs (MFA) is to slow-roll these engagements, and the U.S. political process to ratify a treaty is long, complicated, and prone to failing. Relative to the 3-24 month timelines that frontier labs have for automating the AI research loop, a formal treaty process is unrealistic.

But there are much more flexible and faster options than treaties. An executive agreement between Xi Jinping and Trump is likely the most straightforward. The most pivotal part of an initial agreement probably involves a direct conversation between heads of state — an in-person meeting if there’s time, or a phone call if there isn’t.2

The phone call

Let’s imagine it’s a phone call. That phone call, if it happens, will probably come after a major event that’s moved the White House to consider AI loss of control a true WMD-level risk.

There are a number of candidates for an event like this. Frontier AI labs and their cybersecurity partners, are already warning of an imminent “vulnpocalypse”, in which AI-enabled cyber hacking campaigns could damage U.S. critical infrastructure. The NSA has also started publishing advisories about related AI-enabled cyber threats. And in the near future, AI-designed biological weapons could both be even more devastating, and harder to defend against.

At the same time, we’ve just seen multiple incidents in which frontier labs have lost control of their models, accidentally setting loose rogue AI swarms on the internet. Bio, cyber, or even robotics capabilities could be involved in a follow-on, high-stakes rogue AI incident — a kind of AI Chernobyl event — that shocks the executive branch into taking this level of urgent action.

Between the shock and the call, there probably won’t be enough time to develop sophisticated ways to verify that neither side is engaged in dangerous AI research.3 It’ll be a mess — at least, at first. And then, as some (including Peter Wildeford) also predict, a scramble to achieve stability.

A few things will need to be true for The Phone Call to go well.

Clear red lines

International agreements as such don’t matter very much in determining the outcomes of WMD issues. The diplomats we spoke to explained that, where formal agreements have held, it’s usually been because incentives already pointed toward adherence. For example, a biological attack creates unacceptable risk to the attacker — you’re as likely to get sick as your adversary is4 — so the BWC mostly holds. And chemical weapons are more expensive per kill than modern high explosives are, so the CWC mostly holds.5

And no nuclear agreement has ever produced stockpile drawdowns below the MAD threshold — the level at which a nuclear second strike can credibly threaten to obliterate the attacker. In other words, international agreements are usually just thin wrappers around existing incentives and threats of force, rather than having force in and of themselves. And when a threat of force loses credibility, the agreements that rely on it collapse.

That’s not necessarily all bad news for an AI agreement with China. Here, part of the real threat comes from the intrinsic danger of continuing to develop AI further. If China truly believed they would be putting themselves at existential risk by building superintelligence, they would, in theory, be incentivized not to build it.6

But for this incentive to matter, China has to believe the threat from superintelligence is real. And because Xi Jinping controls China, this largely comes down to saying that Xi Jinping must personally believe the threat from superintelligence is real. The problem is that we can’t know for sure what Xi Jinping believes about superintelligence, and he might change his mind about it at any time.

This means that, even in the aftermath of an AI Chernobyl, we won’t be able to count on China’s belief in the threat of rogue superintelligence to get them to the table or to enforce an AI control agreement.7 It will then become obvious that we have no choice but to directly threaten them with consequences for violating our red lines on AI, and to make our reasons for that threat — safeguarding the existence of humanity as a whole — clear both to them and to the rest of the world.

We’ve done this kind of thing before. The United States sometimes discloses parts of its strategic doctrine to its adversaries, precisely so that nobody crosses one of our red lines by accident. A former official who ran talks with both Beijing and Moscow described this pattern:

[At the P5 level], we advocated and actually did share nuclear policies and WMD policies in general.

We told them that if U.S. intelligence determines with more probability than not that Russia is developing [redacted], we would have to target [them] [...] we would bomb [their] territory to take it out, because it’s untenable to us that [they] would have such a system. [Emphasis added]

– Former State Department official who led prior U.S.-China arms control talks

Former senior State Department officials told us that during The Phone Call, POTUS will need to not only communicate a clear set of red lines, but also threaten credible consequences for Chinese defection while staying ambiguous about exactly what those consequences will be.8

The initial red lines will probably have to be rough at first, because we won’t have the technology we would need to verify that neither side is violating the terms of an agreement. Companies are working on this, but the solutions closest to scalable deployment are 6 months away at best, and haven’t been vetted by the intelligence community — a process that historically has taken many years.9

Something as simple as “if our satellites see you building clusters that look like X, or heat signatures that suggest training runs above scale Y, that crosses our red line” might work at first. We’re currently having follow-up conversations with companies and IC colleagues to better understand what kinds of red lines might be effective under these conditions.

Over time, those red lines will get more specific as it becomes easier for China to find workarounds and loopholes10 — like hiding illicit infrastructure and AI research activities — but for The Phone Call, the key will be to buy a few weeks or months while we improve our verification posture.

Either way, we’ll need at least one unilateral, capability-pegged red line that can buy us time during the crisis period. That’s something we need to start developing today.

Credible threats

The United States has a credibility problem with its adversaries. Several people we spoke to, whose collective tenures spanned almost 20 years across Republican and Democrat administrations, told us that we’ve trained our adversaries not to believe us.

The pattern they describe is a long string of provocations by adversaries that’s been met with no clear U.S. response. Several people independently referenced Syria in 2013. When the Obama administration declared that Assad’s use of chemical weapons would cross a “red line,” Assad used them anyway, and the threatened strikes never came.11 A former senior Biden State Department official thought the cost to U.S. credibility from failing to act when red lines are defined and crossed is significant, and would directly affect the U.S. posture in an AI deal with China.

I think it’s real, to the extent that we pull punches. There has to be consequence. [Emphasis added]

– Former Senior State Department arms control official

To negotiate with China effectively in the context of AI, we need to be able to credibly threaten a spectrum of different response options in the event that they don’t comply.

These options can range from cyber to physical operations, but they can also include export controls to reduce China’s ability to produce components for an AI build-out, in a way that has severe effects lasting on the order of months. We spoke to export control specialists who are developing interesting options that might each allow for a one-time, highly disruptive slowdown in China’s AI build-out, for example.

The impact of these options needs to run roughly between economic sanctions (which China may care little about) and a nuclear first strike (which China knows we won’t act on). The most important thing is to have an escalating spectrum of credible offensive options that we can choose from if China decides to test the limits of our red lines in ambiguous or plausibly deniable ways.12

As we develop these options, we’ll need to test them constantly under real-world conditions. The best way to do this is to execute real offensive operations against lower-stake proxies for the targets we care about. We need live-fire tests like these to make sure we exercise every layer of the offensive stack, including the decisionmaking function of choosing to launch an operation in the first place. This helps ensure problems surface before the stakes are at their highest.13 It also builds up our own willingness to execute a response, which further improves credibility. On several occasions, special operators, diplomats, and intelligence community officials we spoke to emphasized that offensive capabilities are only real if they are regularly exercised.

This kind of sub-threshold activity happens all the time in other domains. China and Russia routinely target U.S. critical infrastructure, and according to experts we spoke to, they do this primarily to prove to themselves that their offensive capabilities can genuinely disrupt our critical infrastructure in the real world. We may need to do the same to China’s AI infrastructure if we’re going to have a credible deterrent in place when the time comes.

Besides developing and testing offensive capabilities, one former diplomat told us that in her experience, China is also sensitive to the perception of the wider international community in a way other U.S. adversaries aren’t. It remains to be seen whether that’s enough of a factor to matter in this case — we are skeptical — but it could offer a minor incentive in addition to more decisive offensive options:

If we could actually show to the world the attribution of bad actions... to the global community with credibility and say, “this is what China’s doing when you’re not looking,” that would be an asymmetric response to escalation. [...] The Philippines have been phenomenal at least in publishing videos of what China’s aggressively doing to them. And that’s the way that they actually got China to back down and the Philippines has nothing near the weapons systems or the offensive capability of China. [...] To a rounding error the Russians just don’t care if you can attribute stuff to them... But China hasn’t crossed that Rubicon yet... knowing through a verifiable credible attribution mechanism that they’re a bad actor would be quite significant with them.

– Former U.S.-China arms control negotiation lead

Shared motivation

Regardless of the diplomatic approach we take, it would be beneficial for the United States and China to publicly message a consistent set of concerns about the global threat from AI loss of control. This would not only increase China’s incentive not to cross red lines,14 but also make it easier for the rest of the world to back a U.S.-China deal.

Public messaging like this would have to include clear communication from the U.S. government about the risks from rapid AI progress from the automation of AI R&D loops, and rogue superintelligence. On this point, everyone we interviewed was in agreement.

The argument that China has so gotten into the heads of the public is that the U.S. just wants to be economically dominant over China in AI, not because it’s a security threat, but just because they want to be able to control the future of humankind. And so you need to say, “let me explain why we perceive it to be a threat,” and that’s a narrative that hasn’t happened.

[...] We would have to say it in a way of, “we are sorry this sounds like a threat, but we are so concerned that you’re developing something”... Because right now they’ll say, “so what if we developed a smarter AI than you, why is that something that’s attackable.”

– Former U.S.-China arms control negotiation lead

This approach could mean sharing evidence for AI loss of control with China that their domestic AI labs may not yet have encountered for themselves. This might include agent logs or whistleblower reports from an AI Chernobyl event, for example.

In general, in most cases where America’s adversaries have kept their arms control commitments, their behavior seems to have been driven in part by a genuine belief — shared with the U.S. — that doing so would help protect them from real and severe threats. But even where shared beliefs like these exist, Russia in particular has been known to cheat or renege when it suited their immediate interest. A shared threat model is helpful, but it’s neither necessary nor sufficient for a stable and effective bilateral agreement.

I think in general in my experience when we’ve entered into treaties with the Soviet Union and Russia to limit nuclear or other weapons, those negotiations have been premised on mutual self-interest, and the outcomes were arrived at in a way that satisfied the interests of both sides through means that may or may not be symmetrical. Sometimes the limits and obligations on each party are symmetrical, but sometimes they are asymmetrical due to asymmetrical capabilities and threat perceptions on each side. But, ideally, the outcomes are perceived to be equally security-enhancing for both parties and thus in the mutual interest.

To that end, Russia as well as the U.S. abided by the original START treaty. There were no significant violations. And same was true with New START — until Russia stopped fully implementing it after the 2022 war on Ukraine, and even then it wasn’t a numerical violation, it was a post-pandemic, post Ukraine war decision saying, “well we’re not going to resume on-site inspections and we are going to stop providing data and notifications.” It was very political, related to seeking leverage in the context of their war aims in Ukraine. INF, on the other hand, they violated that treaty in a way that undermined its entire object and purpose. The low level but highly troubling cheating around the CWC — Russia’s alleged development and clandestine use of unique chemical agents against political targets — that’s very challenging to address and stop.

– Former START negotiator who later ran arms control at the NSC

Traps to avoid

The diplomats we interviewed described several ways in which previous U.S.-China talks on arms control broke down because of what in hindsight seem to have been bad-faith tactics by the Chinese. We think it’s important to include their anecdotes about this, because if similar incidents resurface during U.S.-China AI negotiations, that could be a sign that China may not be taking the threat of AI loss of control seriously enough to negotiate in earnest.

The Chinese perspective

First, it’s worth giving the view from China itself. An executive at a Beijing-based AI policy organization, who’s supported major U.S.-China Track 2 dialogues, has told us that it’s far less clear to China’s policymakers and senior academics that deep learning is the fastest path to superintelligence than it is to America industry. Many influential researchers and organizations in China expect other research directions could ultimately win out in the end — things like brain-inspired AI, neurosymbolic methods, or whole-brain emulation — approaches that get much less attention today in American frontier AI research.15

Until an AI Chernobyl event happens, Chinese interlocutors may interpret a U.S. proposal to pause or slow down AI as strange or unjustified, and a U.S. emphasis on compute-based verification and enforcement confusing. This isn’t to say there aren’t plenty of Chinese AI researchers and groups who understand the threat from superintelligence, but it’s a reason to think the Chinese government could be slower to recognize it than we may be, according to those we spoke to.

First trap: China agrees solely to constrain us

China has a long-standing pattern of signing up for anything that constrains whoever is ahead and isn’t them. Since we’re ahead in AI models, there’s a risk they could agree to a global pause at something like the current U.S. capability ceiling, with no intention of complying with it themselves once they reach it.

Yes, that’s their traditional thing. That’s why everybody in the world has been for non-weaponization of space — because we [the United States] were ahead. But now we created Space Force, because it was obvious that despite whatever rhetoric, the Russians and the Chinese themselves are getting ready for space combat. [Emphasis added]

[...] They follow things that are in their best interest, but to the extent that it’s not, either they cheat — it’s just the norm to cheat — or just not sign up.

– Former senior State Department nonproliferation official

The consistent pattern we’ve seen in most treaty regimes is that the U.S. and U.S.-aligned nations tend to have a much higher compliance rate, and bear the bulk of the cost of enforcement.

– Former Joint Staff counter-WMD officer

In other words, enthusiasm isn’t evidence of intent. This doesn’t mean we don’t need an agreement. It means we should prepare a contingency plan — which should include offensive options — in case they defect from the agreement as soon as they achieve AI capability parity with the American labs.16

Second trap: the talks are not the point

According to our diplomats, the U.S. has historically been hungry — bordering on desperate — to engage China in arms control and WMD threat reduction talks, and China has historically treated that eagerness as a bargaining chip.

One of our sources led U.S.-China talks on a particularly sensitive file. She’s among the most pro-China-engagement people we spoke to, but the story she shared speaks for itself. This individual spent years trying to engage with the Chinese on her file, and then the talks between the two sides stopped completely without a reason being given. Then a few years after she left her post, she bumped into her former Chinese counterparts. They told her that they had suspended the talks solely so that they would be able to trade the resumption of those talks for U.S. concessions in the future.

The subject of the talks was a high-consequence agreement at the intersection of AI and nuclear risk. So even when the substantive content of a dialogue is a critical area of mutual interest, the Chinese diplomatic apparatus is still prone to using its willingness to engage as a bargaining chip.

Just about everyone with China experience that we spoke to had a first-hand story of a similar interaction. In some cases, they told us this led to a deficit in negotiating leverage that worsened outcomes for America, and reduced the enforceability of the agreement that was being negotiated.

What’s more, China sometimes engages in talks primarily for the purpose of suspending them to extract concessions. At first, this can be hard to distinguish from genuine intent to engage, but slow execution. In the context of AI, this distinction could matter a lot. By the time the U.S. government institutionally takes rogue AI to be a WMD-like threat, we may be close to a catastrophic loss of control incident — something even beyond an AI Chernobyl. That could leave us little time to conclude that an agreement isn’t likely, and that we need to shift our strategy.

Since it’s possible that an AI control agreement will have to meet a “no-fail” bar where even marginal defection is unacceptable, we’ll have to ensure that the United States has the leverage it needs to achieve an enforceable oversight regime.

There’s a desperate appetite for treaties. If you look at the New START treaty, I worked with some of the guys that were on the negotiation team for that and their readout was that the Russians knew that we were desperate for this. Obama was trumpeting that we were negotiating it and like it was a nontrivial part of his political platform that we were pursuing denuclearization, and that he was the “peace President” and all this stuff. And as a result, that shaped the footing of the negotiators. And the Russians would make fun of them. They would be like, “we know you need a deal.” Like the very worst thing you can walk into a negotiation with is that “I am absolutely walking out of this signing something”.

So if the most important thing you’re getting from this is my signature, like… stand by. And as a result, [...] the old reporting requirements on rail-based launchers — there were so many structural advantages we had in the prior existing state of affairs that we shit away for a campaign bullet that, “Oh we’ve reduced the number of nuclear weapons on the planet.” So my experience with the Russians is that they perceive and exploit our insatiable appetite to trumpet this treaty/mediation-based mechanism for international dispute resolution. And because that’s our sine qua non — we must have that — it yields just an unacceptable level of leverage.

So seeing the Russians sitting in meetings, take their shoes off, bang them on the desk, raise points that have nothing to do with the agenda, fillibuster, and just obstruct the content of the meeting unless and until their agenda is given disproportionate weight… and it works! It worked time and time again.

– Former Joint Staff counter-WMD officer

Third trap: the multilateral forum

China often tries to steer negotiations with the U.S. away from bilateral venues into multilateral ones. Every source who worked China files confirmed that you can feel this pressure at the negotiating table.

China’s reason for this seems to be pretty cynical: according to the people we interviewed, it’s fundamentally about making the U.S. look bad to the rest of the world.

It’s a game. It’s just a game. … To the extent the Chinese foreign ministry talks about multilateral mechanisms, it’s usually to embarrass us. To basically be like “we’re trying to work multinationally, multilaterally, with everybody, within the UN, international law, but here look at the United States, they’re going at it alone, they’re the renegade, they’re the hegemon” — and they would propose treaties and all this stuff to make themselves look good in front of the African nations or other developing countries so that it gives them credibility internationally.

– Former senior State Department nonproliferation official

A former Joint Staff counter-WMD officer described this as a tactic that lets China derail talks they don’t want to be bound by. By steering the conversation into a venue where dozens of small countries all have different incentives and interests, they can ensure that the talks don’t lead anywhere. So America either withdraws, and looks bad, or has to make concessions that wouldn’t make sense in any other context. The U.S. foreign policy apparatus has historically treated multilateral legitimacy as a requirement, and China knows they can use that against us.

If you go back and look at the Golden Era of treaties, it was post-WWII. You had the formation of the UN, you had NATO, chemical weapons, biological weapons, nonproliferation treaty — all that stuff came out in the very late 40s and 50s. This was in the age before Suez, before Nasser, and before the realization of the Third World. […] The idea was that their contribution to consensus was a highly valuable point of leverage, and by essentially organizing opposition for bids to consensus from the U.S. or others, they’re able to either gain massive concessions in international diplomacy or simply undermine these endeavors.

[...] If you’re Trinidad and Tobago, and you want an extra concession, the way to get it is to raise hell so they have one less vote.

[...] The value [of engaging in multilateral fora] is almost exclusively symbolic. No real substantive decisions are made in that public of a room, and you can’t achieve a meaningful consensus. [Emphasis added]

– Former Joint Staff counter-WMD officer

To derail an agreement, all China needs to do is force a hopelessly incoherent multilateral dialogue, which raises the cost of consensus until the initiative dies on its own.

This trap is less likely to be a problem if China truly understands the threat from superintelligence. But if we get to the AI endgame and need an urgent AI control deal, and we see China insist that a key part of that discussion must happen in a multilateral venue, that suggests they’re not approaching the talks constructively.

Fourth trap: the people they send can’t say yes

The diplomats we spoke to made it clear that China’s Ministry of Foreign Affairs (MFA) is not a peer institution to the U.S. State Department:

[The MFA] are basically barbarian handlers. They have limited clout within their own system. So you can’t think of their foreign ministry as being even a State Department counterpart. So they just handle, you know, the ceremony and the motions of diplomacy. [Emphasis added]

— Former senior State Department nonproliferation official

The official who ran the 2023-24 U.S.-China risk reduction talks saw this play out firsthand. She raised Russia’s nuclear anti-satellite program with her MFA counterparts, but they told her they’d never heard of it. “And I think they wouldn’t [have heard of it], because they wouldn’t need to know. The people who’d need to know would be the military.” The MFA are often the last to know strategically vital facts that have a direct bearing on the negotiations they themselves are involved in.

Negotiations with the Soviet Union sometimes followed a similar pattern, where Soviet military representatives would stop Americans from discussing certain weapon systems until Soviet diplomats had left the room — because their own diplomats weren’t cleared to hear about their own weapon systems.

In China’s case, one solution is to engage directly with the PLA, which has knowledge and power in the CCP state apparatus roughly similar to that the Red Army had under the Soviet system. But efforts to get the PLA at the table on risk reduction have repeatedly run into questions on the U.S. side over whether engagement should be led by State or DOW. The PLA, on its side, has generally been unwilling to engage without senior DOW participation.

When the two militaries did meet in the period after the 2023 arms-control talks, those sessions produced little of substance. A former U.S.-China arms control negotiation lead told us that, “when China and the U.S. interact, everything has to be written out in advance. China very rarely leaves the script.” The DOW has historically approached formal discussions similarly, and the result is typically that you get two delegations reading prepared text at each other.

This is one reason why the diplomats we spoke to converged on a direct POTUS-Xi phone call as the most likely path forward: Chinese negotiating teams typically aren’t authorized to negotiate in the conventional sense at all.

Fifth trap: implausible reasons to delay

Chinese negotiators have historically frustrated their U.S. counterparts by apparently taking offense to the use of certain, impossible-to-predict terms at the negotiating table.

The language would regularly be problematic. They don’t like certain words, and you would use that word and potentially set the whole negotiation back. They don’t like the word ‘arms control,’ they don’t like the word ‘risk reduction.’ They prefer ‘risk prevention,’ and they prefer not arms control because they think that’s solely in the U.S. interest. They prefer talking about ‘stability’ and ‘peace support.’

– Former U.S.-China arms control negotiation lead

The negotiator who shared this anecdote treated these sensitivities as genuine. But other diplomats thought this is likely just another delay and derailment tactic, designed to create a minefield of invented objections with the aim of grinding talks to a halt at will. As a tactic it costs China nothing, while it forces American diplomats to spend their preparation effort on which words to use rather than on substance. (It can also be set aside at any moment without losing face.)

The problem there is they say “we don’t trust each other,” and then we say “let’s work to develop trust through confidence building measures,” and they say “no no, we can’t do confidence building measures until we trust you.” And so it’s like, how do we get there.

– Former U.S.-China arms control negotiation lead

As with other failure modes of U.S.-China diplomacy, the risk for the AI endgame here is that it’s impossible to know how genuine these objections are. If we see Chinese diplomats threatening to end talks on the basis of complaints over the use of jargon in the aftermath of an AI Chernobyl event, we won’t necessarily have time to figure out whether it’s an issue worth resolving, or whether it should be treated as an indication that China is interested in delaying or derailing progress towards an agreement. That difference matters a lot: the latter scenario could require the U.S. to actively prepare to unilaterally enforce a slowdown with China.

Maintaining stability

A majority of the frontier lab researchers we’ve spoken to have been telling us for months that they believe their alignment and AI control techniques aren’t keeping up with the capabilities of their models. If they build a superintelligence under roughly current conditions, they think there’s a good chance (often well above 20%) that they’ll lose control of it.17 As long as that’s true, the United States and China probably have more to lose by competing to build superintelligence than by narrowly cooperating to “pace” AI progress. The less each country believes they can control what they’re building, the less likely they are to build it.

But this creates a strategic stability risk that no U.S.-China cooperation proposal we know of has addressed. Let’s suppose the United States and China reach some kind of agreement to pause frontier AI research. Now imagine that a few months later, the CIA hears rumors of a new AI alignment technique that Chinese labs have developed. This alignment technique might help Chinese labs maintain control over a new, more capable generation of models than they could previously have deployed safely. The CIA doesn’t have all the details, but they’ve heard the initial tests look promising.

Now the U.S. government has to ask itself: in light of this alignment breakthrough, what are the odds that China will defect from our agreement and start building models at the next level of scale? If Chinese labs can pull this off, that new level of AI capability might give China a permanent advantage over America.

The U.S. intelligence community’s assessment of this risk will depend on at least two factors:

1. Given the new breakthrough, what level of risk does Chinese leadership actually believe they’d be facing if they did authorize a next-generation training run?

2. What is China’s risk tolerance for AI loss of control? In other words, how risky does Chinese leadership need to believe a training run is going to be before they decide not to try it?

For the past decade, research on AI alignment and control has been a core focus of many approaches to AI safety. But what this suggests is that, somewhat paradoxically, progress in alignment or AI control could become destabilizing in the U.S.-China AI endgame.18

This wouldn’t be unprecedented. U.S. adversaries have defected from formal agreements on arms control, even when they seemed earnest in signing on initially. Even after the U.S. finds out they defected, it’s not always immediately clear if the defections reflect an increased appetite for risk, a change in the level of risk they assess would come from defecting, or even a rogue element that’s operating without approval. Our lack of transparency into the thought process of their leadership becomes a critical, destabilizing challenge.

With INF [Intermediate-Range Nuclear Forces Treaty], the Russians flat-out cheated. They developed a missile that was clearly in violation of the treaty, as a black program. They thought they wouldn’t get caught; but they did. The intel was so sensitive that it was hard to prove in the court of public opinion and even to some allies and other countries. But it was discovered on my watch at the NSC, I can tell you, they did it.

When it first came to light, we considered many possibilities including that it could be something the military was doing that the political leadership may not have been aware of. So we tried quiet diplomacy first to see if we could discuss and resolve it.

But in the end, it was clear that Putin knew and they were not going to back down or take meaningful steps to preserve the Treaty.

– Former START negotiator who later ran arms control at the NSC

A U.S.-China agreement might have to include provisions against either side performing certain kinds of alignment or AI control research, in order to preserve the incentives that made the agreement possible in the first place. But we expect those provisions to be especially hard to verify, based on conversations with compute verification experts.

It’s not obvious to us how to overcome this. Some have proposed a global coordination mechanism that would allow frontier AI research to continue only within a highly secured, internationally managed CERN-for-AI-type facility. But security experts we’ve spoken to think it’s implausible that this could be made robust to exploitation by nation-states like China and Russia.

So The Phone Call will lead to a temporary agreement of some kind, from which we should try to back into a more durable and sophisticated verification regime as fast as we can — a verification regime that starts by verifying AI capabilities progress, but that may eventually expand to cover alignment and AI control. At each transition point, we’ll need to make sure China remains strongly incentivized to comply; past experience has shown that arms control agreements that address far slower-moving technologies can break down fairly quickly once incentives shift.

One of the lessons I have taken away from my years of negotiating and implementing various arms control and nonproliferation treaties is that it may not be optimal in most if not all cases to have treaties of indefinite duration. It may be better to have treaties of more limited duration — 10, 15, 20 years, and with the ability to modify them to take into account changing circumstances and technology.

In the case of INF, which was of unlimited duration, it clearly got to a point where Russia decided it no longer served its interests. They decided they wanted to get out, but instead of doing the honorable thing and withdrawing, they decided to stay in and cheat by developing an INF-range missile — and they were considering all their options including withdrawal before the U.S. unexpectedly confronted them on their violation.

You can’t prevent a country from deciding that an agreement that once served its interest no longer does so. But when that happens, the more honorable (and legal) approach is to discuss it and exercise the right of withdrawal that is incorporated into virtually every treaty I can think of. I did not agree with the Bush 43 decision to withdraw from the ABM Treaty, but at least when the U.S. did so, we gave the advance notice and followed the withdrawal procedures provided for under the treaty.

Bottom line is that it is very hard to draft a treaty that will serve the interests of its parties in perpetuity. It may be far preferable to negotiate treaties and agreements of shorter durations as well as to make them more easily amendable by consensus of the parties.

– Former START negotiator who ran arms control at the NSC

What now?

The mechanisms we’ll need to invoke to achieve stability initially will probably be costly. They may need to disallow a wide range inference and training activity that could be enabled by more precise monitoring. Trillions of dollars of value could be gated by progress in verification technologies that can support a more nuanced U.S.-China agreement.

One implication of this is that there may soon be very aggressive funding directed at verification technologies that can increase the stability and permissiveness of the new AI control regime. The most fine-grained and reliable our verification technology is, the more legitimate, value-generating activity a U.S.-China agreement can allow.

But funding alone won’t be enough. In order for these solutions to be built into a real verification stack — as part of America’s national technical means (NTMs) — they will need to be vetted by the U.S. intelligence community. That process takes time: examples like Vela and TTBT/CORRTEX show that multi-year timelines are standard — and that’s when the technology of interest to the arms control process is well-understood by the government. That isn’t the case for AI.

If we’re going to have a shot at creating a verification stack that the White House is prepared to lean on after an AI Chernobyl, the companies building verification technologies will need to know the action-level officers, managers, and senior officials who will be involved in assessing their products.

Just as importantly, these companies will have to work alongside members of the intelligence community who can assess and red-team their technologies ahead of time, so that they don’t spend their effort on approaches to verification that are already known to be nonstarters.

We spoke recently to the founders of half a dozen AI verification companies, and each time asked them the question: have you engaged with a single member of the U.S. intelligence community to determine whether there’s a prospect that the IC will even consider this option? Exactly one of those companies answered yes. That gap needs to close as quickly as possible.

If you’re a company building AI verification tools and you want to have your concept vetted, please reach out using this form and we’ll help you reach the right people. To get in touch, contact hello@aiendgame.com.