Why we wrote this
All of a sudden, Washington is buzzing with talk about a U.S.-China AI deal.
The story goes like this: our AIs are getting smarter faster than we can figure out how to control them. If we don’t slow down development, we’re headed for a catastrophe — possibly even extinction. But if we do slow down, and China doesn’t, then we’ll just get killed by Chinese AIs instead of American ones. The only way out of this is to make a deal with China to slow down AI. Some even say it should be a U.S.-China AI treaty.
If you believe that we might not be far from achieving superintelligence, and that our techniques for controlling AI are already beginning to break down today, you might understand the appeal of this story.
But there’s a big problem with it: China is just about the least trustworthy counterparty you could possibly imagine. The WTO, the nine-dash line, COVID-19 — their diplomatic history is full of deception, geopolitical ploys, and hypocrisy.
So suppose we did decide to engage China on an AI slowdown deal. Even if they said yes, how would we make sure they were keeping their end of it?1 And what if they just say no?
To the diplomats who’ve sat across the table from China negotiating on issues from arms control to counterproliferation, no progress on a deal is possible until we can answer these basic questions. And yet much of today’s “AI deal” discourse assumes that China will be concerned enough about rogue AI risk to engage in good faith on slowdown discussions. Or that, if we put in place the right verification technology, China won’t renege on a slowdown commitment.
Maybe these assumptions will turn out to be right. But the history of U.S.-China diplomacy over the last two decades has shown us that we shouldn’t count on it.
Taking superintelligence seriously means confronting that reality. It means both understanding the concerns of AI researchers about AI loss of control, and listening to the diplomats who have seen firsthand how China pursues its geopolitical goals in practice.
So we spent a few months this summer doing just that. We spoke to diplomats who sat across the table from China and Russia in WMD and nonproliferation talks, and to operators who supported those negotiations. They included multiple officials at the Assistant Secretary level, a member of the original START negotiating team who later ran arms control at the National Security Council, and senior counter-WMD specialists at the Pentagon. Between them they’ve worked on START , New START, INF, the NPT review process, the CWC and BWC, the Australia Group, Wassenaar, the Proliferation Security Initiative, and the 2023-24 U.S.-China risk reduction talks.
Despite its interest in treaties, the AI policy community has had relatively little contact with the people who have drafted, implemented, and negotiated treaties in the past. But we think what they have to say will matter a lot to the AI endgame.
We may be wrong about some of what follows, but we’ll point out where our sources disagree with each other, and where we disagree with some of them. We’re shipping this report faster than we normally would because we think speed matters more than the confidence we’d get from another 20 interviews.
We’ll give you the bottom line first: Nobody in this group of diplomats thinks there will be a treaty as such. But they do see other ways of coordinating with China that might help us achieve stability. The nearly unanimous view among the 12 current and former diplomats we spoke to is that U.S.-China coordination will be messier and more precarious in reality than the traditional treaty picture suggests.
Their bet is that it will take the form of a handshake agreement, reached under crisis conditions, over a channel that hasn’t been put to the test yet. It’s a view that’s consistent with what some other commentators have already put forward.
We probably can’t get to that position unless we’re facing a crisis... trying to negotiate with them some sort of rules of the road, it’s just not going to work.
– Former senior State Department nonproliferation official
It won’t be a treaty
At first we expected our interviews with U.S. diplomats would revolve around the mechanics of treaty negotiations. But just about everyone’s first reaction was that a treaty isn’t even an option.
When it comes to a treaty, you’re not going to get there. It’s gotta be something like, POTUS-level and Xi Jinping because no one else matters in China. It’s at that level. Unless there’s some kind of emergency summit or something, because we don’t have hotlines that work. That’s the only way you could engage something that serious. So treaty — forget it. We don’t do treaties anymore because it just takes too long. [Emphasis added]
– Former senior State Department nonproliferation official
Treaties need to be ratified, which is typically a multi-year process. That on its own might be enough to rule them out for something that moves as fast as AI does. America also hasn’t ratified a major arms control treaty since New START, almost 16 years ago.
China in particular is also slow to engage on these exact kinds of issues. One former U.S.-China arms control negotiation lead told us the Chinese timeline on this kind of engagement “wasn’t even a meeting a year. It was a meeting and then maybe two years later another meeting.” According to another, the reason this happens is that China tends to view the U.S. as being so hungry for diplomatic engagement that they can use even the prospect of talks with them as negotiating leverage.
And the Chinese, because they’re not involved in these things, they’re more elusive, when we would have engagement with them, it was almost like, “Oh oh, here’s a chance to get their assent!” Like, we’re more desperate for them to agree with us on this point.
– Former Joint Staff counter-WMD officer
Maybe we’ll see more earnest engagement from China if we continue to get AI loss of control warning shots. But the institutional culture of China’s Ministry of Foreign Affairs (MFA) is to slow-roll these engagements, and the U.S. political process to ratify a treaty is long, complicated, and prone to failing. Relative to the 3-24 month timelines that frontier labs have for automating the AI research loop, a formal treaty process is unrealistic.
But there are much more flexible and faster options than treaties. An executive agreement between Xi Jinping and Trump is likely the most straightforward. The most pivotal part of an initial agreement probably involves a direct conversation between heads of state — an in-person meeting if there’s time, or a phone call if there isn’t.2
The phone call
Let’s imagine it’s a phone call. That phone call, if it happens, will probably come after a major event that’s moved the White House to consider AI loss of control a true WMD-level risk.
There are a number of candidates for an event like this. Frontier AI labs and their cybersecurity partners, are already warning of an imminent “vulnpocalypse”, in which AI-enabled cyber hacking campaigns could damage U.S. critical infrastructure. The NSA has also started publishing advisories about related AI-enabled cyber threats. And in the near future, AI-designed biological weapons could both be even more devastating, and harder to defend against.
At the same time, we’ve just seen multiple incidents in which frontier labs have lost control of their models, accidentally setting loose rogue AI swarms on the internet. Bio, cyber, or even robotics capabilities could be involved in a follow-on, high-stakes rogue AI incident — a kind of AI Chernobyl event — that shocks the executive branch into taking this level of urgent action.
Between the shock and the call, there probably won’t be enough time to develop sophisticated ways to verify that neither side is engaged in dangerous AI research.3 It’ll be a mess — at least, at first. And then, as some (including Peter Wildeford) also predict, a scramble to achieve stability.
A few things will need to be true for The Phone Call to go well.
Clear red lines
International agreements as such don’t matter very much in determining the outcomes of WMD issues. The diplomats we spoke to explained that, where formal agreements have held, it’s usually been because incentives already pointed toward adherence. For example, a biological attack creates unacceptable risk to the attacker — you’re as likely to get sick as your adversary is4 — so the BWC mostly holds. And chemical weapons are more expensive per kill than modern high explosives are, so the CWC mostly holds.5
And no nuclear agreement has ever produced stockpile drawdowns below the MAD threshold — the level at which a nuclear second strike can credibly threaten to obliterate the attacker. In other words, international agreements are usually just thin wrappers around existing incentives and threats of force, rather than having force in and of themselves. And when a threat of force loses credibility, the agreements that rely on it collapse.
That’s not necessarily all bad news for an AI agreement with China. Here, part of the real threat comes from the intrinsic danger of continuing to develop AI further. If China truly believed they would be putting themselves at existential risk by building superintelligence, they would, in theory, be incentivized not to build it.6
But for this incentive to matter, China has to believe the threat from superintelligence is real. And because Xi Jinping controls China, this largely comes down to saying that Xi Jinping must personally believe the threat from superintelligence is real. The problem is that we can’t know for sure what Xi Jinping believes about superintelligence, and he might change his mind about it at any time.
This means that, even in the aftermath of an AI Chernobyl, we won’t be able to count on China’s belief in the threat of rogue superintelligence to get them to the table or to enforce an AI control agreement.7 It will then become obvious that we have no choice but to directly threaten them with consequences for violating our red lines on AI, and to make our reasons for that threat — safeguarding the existence of humanity as a whole — clear both to them and to the rest of the world.
We’ve done this kind of thing before. The United States sometimes discloses parts of its strategic doctrine to its adversaries, precisely so that nobody crosses one of our red lines by accident. A former official who ran talks with both Beijing and Moscow described this pattern:
[At the P5 level], we advocated and actually did share nuclear policies and WMD policies in general.
We told them that if U.S. intelligence determines with more probability than not that Russia is developing [redacted], we would have to target [them] [...] we would bomb [their] territory to take it out, because it’s untenable to us that [they] would have such a system. [Emphasis added]
– Former State Department official who led prior U.S.-China arms control talks
Former senior State Department officials told us that during The Phone Call, POTUS will need to not only communicate a clear set of red lines, but also threaten credible consequences for Chinese defection while staying ambiguous about exactly what those consequences will be.8
The initial red lines will probably have to be rough at first, because we won’t have the technology we would need to verify that neither side is violating the terms of an agreement. Companies are working on this, but the solutions closest to scalable deployment are 6 months away at best, and haven’t been vetted by the intelligence community — a process that historically has taken many years.9
Something as simple as “if our satellites see you building clusters that look like X, or heat signatures that suggest training runs above scale Y, that crosses our red line” might work at first. We’re currently having follow-up conversations with companies and IC colleagues to better understand what kinds of red lines might be effective under these conditions.
Over time, those red lines will get more specific as it becomes easier for China to find workarounds and loopholes10 — like hiding illicit infrastructure and AI research activities — but for The Phone Call, the key will be to buy a few weeks or months while we improve our verification posture.
Either way, we’ll need at least one unilateral, capability-pegged red line that can buy us time during the crisis period. That’s something we need to start developing today.
Credible threats
The United States has a credibility problem with its adversaries. Several people we spoke to, whose collective tenures spanned almost 20 years across Republican and Democrat administrations, told us that we’ve trained our adversaries not to believe us.
The pattern they describe is a long string of provocations by adversaries that’s been met with no clear U.S. response. Several people independently referenced Syria in 2013. When the Obama administration declared that Assad’s use of chemical weapons would cross a “red line,” Assad used them anyway, and the threatened strikes never came.11 A former senior Biden State Department official thought the cost to U.S. credibility from failing to act when red lines are defined and crossed is significant, and would directly affect the U.S. posture in an AI deal with China.
I think it’s real, to the extent that we pull punches. There has to be consequence. [Emphasis added]
– Former Senior State Department arms control official
To negotiate with China effectively in the context of AI, we need to be able to credibly threaten a spectrum of different response options in the event that they don’t comply.
These options can range from cyber to physical operations, but they can also include export controls to reduce China’s ability to produce components for an AI build-out, in a way that has severe effects lasting on the order of months. We spoke to export control specialists who are developing interesting options that might each allow for a one-time, highly disruptive slowdown in China’s AI build-out, for example.
The impact of these options needs to run roughly between economic sanctions (which China may care little about) and a nuclear first strike (which China knows we won’t act on). The most important thing is to have an escalating spectrum of credible offensive options that we can choose from if China decides to test the limits of our red lines in ambiguous or plausibly deniable ways.12
As we develop these options, we’ll need to test them constantly under real-world conditions. The best way to do this is to execute real offensive operations against lower-stake proxies for the targets we care about. We need live-fire tests like these to make sure we exercise every layer of the offensive stack, including the decisionmaking function of choosing to launch an operation in the first place. This helps ensure problems surface before the stakes are at their highest.13 It also builds up our own willingness to execute a response, which further improves credibility. On several occasions, special operators, diplomats, and intelligence community officials we spoke to emphasized that offensive capabilities are only real if they are regularly exercised.
This kind of sub-threshold activity happens all the time in other domains. China and Russia routinely target U.S. critical infrastructure, and according to experts we spoke to, they do this primarily to prove to themselves that their offensive capabilities can genuinely disrupt our critical infrastructure in the real world. We may need to do the same to China’s AI infrastructure if we’re going to have a credible deterrent in place when the time comes.
Besides developing and testing offensive capabilities, one former diplomat told us that in her experience, China is also sensitive to the perception of the wider international community in a way other U.S. adversaries aren’t. It remains to be seen whether that’s enough of a factor to matter in this case — we are skeptical — but it could offer a minor incentive in addition to more decisive offensive options:
If we could actually show to the world the attribution of bad actions... to the global community with credibility and say, “this is what China’s doing when you’re not looking,” that would be an asymmetric response to escalation. [...] The Philippines have been phenomenal at least in publishing videos of what China’s aggressively doing to them. And that’s the way that they actually got China to back down and the Philippines has nothing near the weapons systems or the offensive capability of China. [...] To a rounding error the Russians just don’t care if you can attribute stuff to them... But China hasn’t crossed that Rubicon yet... knowing through a verifiable credible attribution mechanism that they’re a bad actor would be quite significant with them.
– Former U.S.-China arms control negotiation lead
Shared motivation
Regardless of the diplomatic approach we take, it would be beneficial for the United States and China to publicly message a consistent set of concerns about the global threat from AI loss of control. This would not only increase China’s incentive not to cross red lines,14 but also make it easier for the rest of the world to back a U.S.-China deal.
Public messaging like this would have to include clear communication from the U.S. government about the risks from rapid AI progress from the automation of AI R&D loops, and rogue superintelligence. On this point, everyone we interviewed was in agreement.
The argument that China has so gotten into the heads of the public is that the U.S. just wants to be economically dominant over China in AI, not because it’s a security threat, but just because they want to be able to control the future of humankind. And so you need to say, “let me explain why we perceive it to be a threat,” and that’s a narrative that hasn’t happened.
[...] We would have to say it in a way of, “we are sorry this sounds like a threat, but we are so concerned that you’re developing something”... Because right now they’ll say, “so what if we developed a smarter AI than you, why is that something that’s attackable.”
– Former U.S.-China arms control negotiation lead
This approach could mean sharing evidence for AI loss of control with China that their domestic AI labs may not yet have encountered for themselves. This might include agent logs or whistleblower reports from an AI Chernobyl event, for example.
In general, in most cases where America’s adversaries have kept their arms control commitments, their behavior seems to have been driven in part by a genuine belief — shared with the U.S. — that doing so would help protect them from real and severe threats. But even where shared beliefs like these exist, Russia in particular has been known to cheat or renege when it suited their immediate interest. A shared threat model is helpful, but it’s neither necessary nor sufficient for a stable and effective bilateral agreement.
I think in general in my experience when we’ve entered into treaties with the Soviet Union and Russia to limit nuclear or other weapons, those negotiations have been premised on mutual self-interest, and the outcomes were arrived at in a way that satisfied the interests of both sides through means that may or may not be symmetrical. Sometimes the limits and obligations on each party are symmetrical, but sometimes they are asymmetrical due to asymmetrical capabilities and threat perceptions on each side. But, ideally, the outcomes are perceived to be equally security-enhancing for both parties and thus in the mutual interest.
To that end, Russia as well as the U.S. abided by the original START treaty. There were no significant violations. And same was true with New START — until Russia stopped fully implementing it after the 2022 war on Ukraine, and even then it wasn’t a numerical violation, it was a post-pandemic, post Ukraine war decision saying, “well we’re not going to resume on-site inspections and we are going to stop providing data and notifications.” It was very political, related to seeking leverage in the context of their war aims in Ukraine. INF, on the other hand, they violated that treaty in a way that undermined its entire object and purpose. The low level but highly troubling cheating around the CWC — Russia’s alleged development and clandestine use of unique chemical agents against political targets — that’s very challenging to address and stop.
– Former START negotiator who later ran arms control at the NSC
Traps to avoid
The diplomats we interviewed described several ways in which previous U.S.-China talks on arms control broke down because of what in hindsight seem to have been bad-faith tactics by the Chinese. We think it’s important to include their anecdotes about this, because if similar incidents resurface during U.S.-China AI negotiations, that could be a sign that China may not be taking the threat of AI loss of control seriously enough to negotiate in earnest.
The Chinese perspective
First, it’s worth giving the view from China itself. An executive at a Beijing-based AI policy organization, who’s supported major U.S.-China Track 2 dialogues, has told us that it’s far less clear to China’s policymakers and senior academics that deep learning is the fastest path to superintelligence than it is to America industry. Many influential researchers and organizations in China expect other research directions could ultimately win out in the end — things like brain-inspired AI, neurosymbolic methods, or whole-brain emulation — approaches that get much less attention today in American frontier AI research.15
Until an AI Chernobyl event happens, Chinese interlocutors may interpret a U.S. proposal to pause or slow down AI as strange or unjustified, and a U.S. emphasis on compute-based verification and enforcement confusing. This isn’t to say there aren’t plenty of Chinese AI researchers and groups who understand the threat from superintelligence, but it’s a reason to think the Chinese government could be slower to recognize it than we may be, according to those we spoke to.
First trap: China agrees solely to constrain us
China has a long-standing pattern of signing up for anything that constrains whoever is ahead and isn’t them. Since we’re ahead in AI models, there’s a risk they could agree to a global pause at something like the current U.S. capability ceiling, with no intention of complying with it themselves once they reach it.
Yes, that’s their traditional thing. That’s why everybody in the world has been for non-weaponization of space — because we [the United States] were ahead. But now we created Space Force, because it was obvious that despite whatever rhetoric, the Russians and the Chinese themselves are getting ready for space combat. [Emphasis added]
[...] They follow things that are in their best interest, but to the extent that it’s not, either they cheat — it’s just the norm to cheat — or just not sign up.– Former senior State Department nonproliferation official
The consistent pattern we’ve seen in most treaty regimes is that the U.S. and U.S.-aligned nations tend to have a much higher compliance rate, and bear the bulk of the cost of enforcement.
– Former Joint Staff counter-WMD officer
In other words, enthusiasm isn’t evidence of intent. This doesn’t mean we don’t need an agreement. It means we should prepare a contingency plan — which should include offensive options — in case they defect from the agreement as soon as they achieve AI capability parity with the American labs.16
Second trap: the talks are not the point
According to our diplomats, the U.S. has historically been hungry — bordering on desperate — to engage China in arms control and WMD threat reduction talks, and China has historically treated that eagerness as a bargaining chip.
One of our sources led U.S.-China talks on a particularly sensitive file. She’s among the most pro-China-engagement people we spoke to, but the story she shared speaks for itself. This individual spent years trying to engage with the Chinese on her file, and then the talks between the two sides stopped completely without a reason being given. Then a few years after she left her post, she bumped into her former Chinese counterparts. They told her that they had suspended the talks solely so that they would be able to trade the resumption of those talks for U.S. concessions in the future.
The subject of the talks was a high-consequence agreement at the intersection of AI and nuclear risk. So even when the substantive content of a dialogue is a critical area of mutual interest, the Chinese diplomatic apparatus is still prone to using its willingness to engage as a bargaining chip.
Just about everyone with China experience that we spoke to had a first-hand story of a similar interaction. In some cases, they told us this led to a deficit in negotiating leverage that worsened outcomes for America, and reduced the enforceability of the agreement that was being negotiated.
What’s more, China sometimes engages in talks primarily for the purpose of suspending them to extract concessions. At first, this can be hard to distinguish from genuine intent to engage, but slow execution. In the context of AI, this distinction could matter a lot. By the time the U.S. government institutionally takes rogue AI to be a WMD-like threat, we may be close to a catastrophic loss of control incident — something even beyond an AI Chernobyl. That could leave us little time to conclude that an agreement isn’t likely, and that we need to shift our strategy.
Since it’s possible that an AI control agreement will have to meet a “no-fail” bar where even marginal defection is unacceptable, we’ll have to ensure that the United States has the leverage it needs to achieve an enforceable oversight regime.
There’s a desperate appetite for treaties. If you look at the New START treaty, I worked with some of the guys that were on the negotiation team for that and their readout was that the Russians knew that we were desperate for this. Obama was trumpeting that we were negotiating it and like it was a nontrivial part of his political platform that we were pursuing denuclearization, and that he was the “peace President” and all this stuff. And as a result, that shaped the footing of the negotiators. And the Russians would make fun of them. They would be like, “we know you need a deal.” Like the very worst thing you can walk into a negotiation with is that “I am absolutely walking out of this signing something”.
So if the most important thing you’re getting from this is my signature, like… stand by. And as a result, [...] the old reporting requirements on rail-based launchers — there were so many structural advantages we had in the prior existing state of affairs that we shit away for a campaign bullet that, “Oh we’ve reduced the number of nuclear weapons on the planet.” So my experience with the Russians is that they perceive and exploit our insatiable appetite to trumpet this treaty/mediation-based mechanism for international dispute resolution. And because that’s our sine qua non — we must have that — it yields just an unacceptable level of leverage.
So seeing the Russians sitting in meetings, take their shoes off, bang them on the desk, raise points that have nothing to do with the agenda, fillibuster, and just obstruct the content of the meeting unless and until their agenda is given disproportionate weight… and it works! It worked time and time again.– Former Joint Staff counter-WMD officer
Third trap: the multilateral forum
China often tries to steer negotiations with the U.S. away from bilateral venues into multilateral ones. Every source who worked China files confirmed that you can feel this pressure at the negotiating table.
China’s reason for this seems to be pretty cynical: according to the people we interviewed, it’s fundamentally about making the U.S. look bad to the rest of the world.
It’s a game. It’s just a game. … To the extent the Chinese foreign ministry talks about multilateral mechanisms, it’s usually to embarrass us. To basically be like “we’re trying to work multinationally, multilaterally, with everybody, within the UN, international law, but here look at the United States, they’re going at it alone, they’re the renegade, they’re the hegemon” — and they would propose treaties and all this stuff to make themselves look good in front of the African nations or other developing countries so that it gives them credibility internationally.
– Former senior State Department nonproliferation official
A former Joint Staff counter-WMD officer described this as a tactic that lets China derail talks they don’t want to be bound by. By steering the conversation into a venue where dozens of small countries all have different incentives and interests, they can ensure that the talks don’t lead anywhere. So America either withdraws, and looks bad, or has to make concessions that wouldn’t make sense in any other context. The U.S. foreign policy apparatus has historically treated multilateral legitimacy as a requirement, and China knows they can use that against us.
If you go back and look at the Golden Era of treaties, it was post-WWII. You had the formation of the UN, you had NATO, chemical weapons, biological weapons, nonproliferation treaty — all that stuff came out in the very late 40s and 50s. This was in the age before Suez, before Nasser, and before the realization of the Third World. […] The idea was that their contribution to consensus was a highly valuable point of leverage, and by essentially organizing opposition for bids to consensus from the U.S. or others, they’re able to either gain massive concessions in international diplomacy or simply undermine these endeavors.
[...] If you’re Trinidad and Tobago, and you want an extra concession, the way to get it is to raise hell so they have one less vote.
[...] The value [of engaging in multilateral fora] is almost exclusively symbolic. No real substantive decisions are made in that public of a room, and you can’t achieve a meaningful consensus. [Emphasis added]– Former Joint Staff counter-WMD officer
To derail an agreement, all China needs to do is force a hopelessly incoherent multilateral dialogue, which raises the cost of consensus until the initiative dies on its own.
This trap is less likely to be a problem if China truly understands the threat from superintelligence. But if we get to the AI endgame and need an urgent AI control deal, and we see China insist that a key part of that discussion must happen in a multilateral venue, that suggests they’re not approaching the talks constructively.
Fourth trap: the people they send can’t say yes
The diplomats we spoke to made it clear that China’s Ministry of Foreign Affairs (MFA) is not a peer institution to the U.S. State Department:
[The MFA] are basically barbarian handlers. They have limited clout within their own system. So you can’t think of their foreign ministry as being even a State Department counterpart. So they just handle, you know, the ceremony and the motions of diplomacy. [Emphasis added]
— Former senior State Department nonproliferation official
The official who ran the 2023-24 U.S.-China risk reduction talks saw this play out firsthand. She raised Russia’s nuclear anti-satellite program with her MFA counterparts, but they told her they’d never heard of it. “And I think they wouldn’t [have heard of it], because they wouldn’t need to know. The people who’d need to know would be the military.” The MFA are often the last to know strategically vital facts that have a direct bearing on the negotiations they themselves are involved in.
Negotiations with the Soviet Union sometimes followed a similar pattern, where Soviet military representatives would stop Americans from discussing certain weapon systems until Soviet diplomats had left the room — because their own diplomats weren’t cleared to hear about their own weapon systems.
In China’s case, one solution is to engage directly with the PLA, which has knowledge and power in the CCP state apparatus roughly similar to that the Red Army had under the Soviet system. But efforts to get the PLA at the table on risk reduction have repeatedly run into questions on the U.S. side over whether engagement should be led by State or DOW. The PLA, on its side, has generally been unwilling to engage without senior DOW participation.
When the two militaries did meet in the period after the 2023 arms-control talks, those sessions produced little of substance. A former U.S.-China arms control negotiation lead told us that, “when China and the U.S. interact, everything has to be written out in advance. China very rarely leaves the script.” The DOW has historically approached formal discussions similarly, and the result is typically that you get two delegations reading prepared text at each other.
This is one reason why the diplomats we spoke to converged on a direct POTUS-Xi phone call as the most likely path forward: Chinese negotiating teams typically aren’t authorized to negotiate in the conventional sense at all.
Fifth trap: implausible reasons to delay
Chinese negotiators have historically frustrated their U.S. counterparts by apparently taking offense to the use of certain, impossible-to-predict terms at the negotiating table.
The language would regularly be problematic. They don’t like certain words, and you would use that word and potentially set the whole negotiation back. They don’t like the word ‘arms control,’ they don’t like the word ‘risk reduction.’ They prefer ‘risk prevention,’ and they prefer not arms control because they think that’s solely in the U.S. interest. They prefer talking about ‘stability’ and ‘peace support.’
– Former U.S.-China arms control negotiation lead
The negotiator who shared this anecdote treated these sensitivities as genuine. But other diplomats thought this is likely just another delay and derailment tactic, designed to create a minefield of invented objections with the aim of grinding talks to a halt at will. As a tactic it costs China nothing, while it forces American diplomats to spend their preparation effort on which words to use rather than on substance. (It can also be set aside at any moment without losing face.)
The problem there is they say “we don’t trust each other,” and then we say “let’s work to develop trust through confidence building measures,” and they say “no no, we can’t do confidence building measures until we trust you.” And so it’s like, how do we get there.
– Former U.S.-China arms control negotiation lead
As with other failure modes of U.S.-China diplomacy, the risk for the AI endgame here is that it’s impossible to know how genuine these objections are. If we see Chinese diplomats threatening to end talks on the basis of complaints over the use of jargon in the aftermath of an AI Chernobyl event, we won’t necessarily have time to figure out whether it’s an issue worth resolving, or whether it should be treated as an indication that China is interested in delaying or derailing progress towards an agreement. That difference matters a lot: the latter scenario could require the U.S. to actively prepare to unilaterally enforce a slowdown with China.
Maintaining stability
A majority of the frontier lab researchers we’ve spoken to have been telling us for months that they believe their alignment and AI control techniques aren’t keeping up with the capabilities of their models. If they build a superintelligence under roughly current conditions, they think there’s a good chance (often well above 20%) that they’ll lose control of it.17 As long as that’s true, the United States and China probably have more to lose by competing to build superintelligence than by narrowly cooperating to “pace” AI progress. The less each country believes they can control what they’re building, the less likely they are to build it.
But this creates a strategic stability risk that no U.S.-China cooperation proposal we know of has addressed. Let’s suppose the United States and China reach some kind of agreement to pause frontier AI research. Now imagine that a few months later, the CIA hears rumors of a new AI alignment technique that Chinese labs have developed. This alignment technique might help Chinese labs maintain control over a new, more capable generation of models than they could previously have deployed safely. The CIA doesn’t have all the details, but they’ve heard the initial tests look promising.
Now the U.S. government has to ask itself: in light of this alignment breakthrough, what are the odds that China will defect from our agreement and start building models at the next level of scale? If Chinese labs can pull this off, that new level of AI capability might give China a permanent advantage over America.
The U.S. intelligence community’s assessment of this risk will depend on at least two factors:
Given the new breakthrough, what level of risk does Chinese leadership actually believe they’d be facing if they did authorize a next-generation training run?
What is China’s risk tolerance for AI loss of control? In other words, how risky does Chinese leadership need to believe a training run is going to be before they decide not to try it?
For the past decade, research on AI alignment and control has been a core focus of many approaches to AI safety. But what this suggests is that, somewhat paradoxically, progress in alignment or AI control could become destabilizing in the U.S.-China AI endgame.18
This wouldn’t be unprecedented. U.S. adversaries have defected from formal agreements on arms control, even when they seemed earnest in signing on initially. Even after the U.S. finds out they defected, it’s not always immediately clear if the defections reflect an increased appetite for risk, a change in the level of risk they assess would come from defecting, or even a rogue element that’s operating without approval. Our lack of transparency into the thought process of their leadership becomes a critical, destabilizing challenge.
With INF [Intermediate-Range Nuclear Forces Treaty], the Russians flat-out cheated. They developed a missile that was clearly in violation of the treaty, as a black program. They thought they wouldn’t get caught; but they did. The intel was so sensitive that it was hard to prove in the court of public opinion and even to some allies and other countries. But it was discovered on my watch at the NSC, I can tell you, they did it.
When it first came to light, we considered many possibilities including that it could be something the military was doing that the political leadership may not have been aware of. So we tried quiet diplomacy first to see if we could discuss and resolve it.
But in the end, it was clear that Putin knew and they were not going to back down or take meaningful steps to preserve the Treaty.
– Former START negotiator who later ran arms control at the NSC
A U.S.-China agreement might have to include provisions against either side performing certain kinds of alignment or AI control research, in order to preserve the incentives that made the agreement possible in the first place. But we expect those provisions to be especially hard to verify, based on conversations with compute verification experts.
It’s not obvious to us how to overcome this. Some have proposed a global coordination mechanism that would allow frontier AI research to continue only within a highly secured, internationally managed CERN-for-AI-type facility. But security experts we’ve spoken to think it’s implausible that this could be made robust to exploitation by nation-states like China and Russia.
So The Phone Call will lead to a temporary agreement of some kind, from which we should try to back into a more durable and sophisticated verification regime as fast as we can — a verification regime that starts by verifying AI capabilities progress, but that may eventually expand to cover alignment and AI control. At each transition point, we’ll need to make sure China remains strongly incentivized to comply; past experience has shown that arms control agreements that address far slower-moving technologies can break down fairly quickly once incentives shift.
One of the lessons I have taken away from my years of negotiating and implementing various arms control and nonproliferation treaties is that it may not be optimal in most if not all cases to have treaties of indefinite duration. It may be better to have treaties of more limited duration — 10, 15, 20 years, and with the ability to modify them to take into account changing circumstances and technology.
In the case of INF, which was of unlimited duration, it clearly got to a point where Russia decided it no longer served its interests. They decided they wanted to get out, but instead of doing the honorable thing and withdrawing, they decided to stay in and cheat by developing an INF-range missile — and they were considering all their options including withdrawal before the U.S. unexpectedly confronted them on their violation.
You can’t prevent a country from deciding that an agreement that once served its interest no longer does so. But when that happens, the more honorable (and legal) approach is to discuss it and exercise the right of withdrawal that is incorporated into virtually every treaty I can think of. I did not agree with the Bush 43 decision to withdraw from the ABM Treaty, but at least when the U.S. did so, we gave the advance notice and followed the withdrawal procedures provided for under the treaty.
Bottom line is that it is very hard to draft a treaty that will serve the interests of its parties in perpetuity. It may be far preferable to negotiate treaties and agreements of shorter durations as well as to make them more easily amendable by consensus of the parties.– Former START negotiator who ran arms control at the NSC
What now?
The mechanisms we’ll need to invoke to achieve stability initially will probably be costly. They may need to disallow a wide range inference and training activity that could be enabled by more precise monitoring. Trillions of dollars of value could be gated by progress in verification technologies that can support a more nuanced U.S.-China agreement.
One implication of this is that there may soon be very aggressive funding directed at verification technologies that can increase the stability and permissiveness of the new AI control regime. The most fine-grained and reliable our verification technology is, the more legitimate, value-generating activity a U.S.-China agreement can allow.
But funding alone won’t be enough. In order for these solutions to be built into a real verification stack — as part of America’s national technical means (NTMs) — they will need to be vetted by the U.S. intelligence community. That process takes time: examples like Vela and TTBT/CORRTEX show that multi-year timelines are standard — and that’s when the technology of interest to the arms control process is well-understood by the government. That isn’t the case for AI.
If we’re going to have a shot at creating a verification stack that the White House is prepared to lean on after an AI Chernobyl, the companies building verification technologies will need to know the action-level officers, managers, and senior officials who will be involved in assessing their products.
Just as importantly, these companies will have to work alongside members of the intelligence community who can assess and red-team their technologies ahead of time, so that they don’t spend their effort on approaches to verification that are already known to be nonstarters.
We spoke recently to the founders of half a dozen AI verification companies, and each time asked them the question: have you engaged with a single member of the U.S. intelligence community to determine whether there’s a prospect that the IC will even consider this option? Exactly one of those companies answered yes. That gap needs to close as quickly as possible.
If you’re a company building AI verification tools and you want to have your concept vetted, please reach out using this form and we’ll help you reach the right people. To get in touch, contact hello@aiendgame.com.
We don’t just mean how will we tell if China is keeping to their end of a deal, but also how will we compel them to do so. There’s already a small but growing ecosystem of startups and research organizations building verification tools to let us detect whether China is keeping their end of an AI deal. But detection is only half of compliance. The other half of compliance is enforcement — i.e., even if we detect that China is violating a deal, how would we make them stop?
This phone call, we expect, will be on a completely different level of urgency and priority from the U.S.-China engagements on AI that have taken place so far. This will be a conversation in which we expect POTUS will put forward concrete requirements for, at the very least, a mutual AI slowdown, monitored by verification mechanisms and backed by threats of consequence for violations. Despite the recent increase in dialogue between Washington and Beijing on this topic, a phone call like this one is still far outside expectations in most DC circles.
Even though there are some good teams working on different parts of this verification stack today, the solutions aren’t comprehensive and may not be ready in time.
Though it helps if you can position your high risk research activities in your adversary’s territory, so that their citizens get sick before yours do if an accident happens.
More exactly, chemical weapons are cheaper for top tier militaries to defend against than high explosives are, so top tier militaries that expect to fight one another would rather focus their military supply chains on high explosives and ban chemical weapons. And, because they’re top tier militaries, they can threaten use of force on lower tier militaries that violate the ban.
This is actually similar to the basis of the real-world BWC, which mostly holds because today’s bioweapons aren’t controllable by the attacker either.
We’ll refer to a mutual agreement to either slow down or pause AI development as an AI control agreement (or deal), by analogy with an arms control agreement.
If we’re too specific about consequences in our threats, then we’re telling our adversary exactly what to expect depending on how exactly they choose to defect. This lets them preemptively harden what they expect us to attack, which in turn lets them systematically test how serious we are about our red lines, while controlling their exposure to our retaliation. (Either because they’ve hardened themselves against our threatened attack, or because they’ve assessed that they’re willing to absorb the damage we’d cause them if we carried through our threatened attack.)
And even those timelines applied to agreements about technologies (like nuclear weapon systems) that were already broadly understood by the U.S. intelligence community, since those technologies either originated or were refined in large part by the U.S. government itself. AI is different: the government has limited existing expertise and would have to ramp up capacity aggressively before it could confidently vet AI verification solutions.
And as verification infrastructure becomes more precise and reliable. Though we should mention that some experts don’t believe verification controls that rely on observable red lines will stay reliable in practice over the long term. One datacenter industry veteran — and cofounder of a multi-billion dollar neocloud — observed that even China’s existing set of strategies for obscuring ownership, distributing compute, and routing capacity through affiliates could already be sophisticated enough to make verification functionally impractical. We’ll have more to say about this in a few weeks.
During the rest of the Obama administration, that is. They did come later under Trump.
These options need to be credible in two senses. First, they need to credibly promise significant damage to China if they are used. But second, China needs to find it credible that we will use them. If we have great options that nobody believes we’ll use, then we can’t credibly threaten to use those options. Our team has been developing some ideas here, and will be sharing some parts of our work on this — to the extent it’s appropriate — in the near future.
It also reduces the risk that we’ll reach for an offensive option at the critical moment, only to find it doesn’t work as well as we thought it did.
To a limited extent
This is in part because China has less compute than we do.
Assuming the agreement allows for parity, which it arguably shouldn’t.
“Above 10%” is a number many have converged on publicly, but in private conversations it’s clear that many have estimates significantly higher than this.
We’ll explore this idea more in a future post.
It may indeed be messy, but let's try anyway.
Existential safety is the quintessential transpartisan issue. Humanity itself is at risk from AI.
Let's show Trump and Xi that there is a win-win way forward that protects and benefits all of humanity at the same time: https://TrumpXiDeal.ai.
Peace through shared prosperity is the safer, smarter path we could collectively choose to take.