21 hours ago
7
Dario Amodei wants to pump the brakes on AI, and he’s drawn up a detailed blueprint for how to do it. The Anthropic CEO published a 3,800-word essay on September 12 outlining a three-step plan to slow the pace of AI capability development, with specific focus on open-weight models from China and the distillation practices that let weaker systems piggyback on more advanced ones.
The proposal has already attracted endorsements from Sam Altman, Elon Musk, and Demis Hassabis.
The three-step playbook
Amodei’s framework rests on three pillars, each escalating in ambition. The first calls for granting independent evaluators like METR deep access to frontier AI models for safety verification. Step two pushes for industry-wide safety standards adopted across AI labs in democratic nations. This isn’t a voluntary code of conduct. Amodei is describing a coordinated framework where labs would submit to mandatory safety testing regardless of whether their models are proprietary or open-weight.
The third and most ambitious piece involves global coordination on AI technology regulation. Combined with stricter chip export controls targeting China, this would create a regime where access to cutting-edge compute becomes contingent on compliance with safety practices defined largely by US-aligned institutions.
Why open-weight builders are in the crosshairs
The essay doesn’t dance around its concerns about open-weight models. Amodei frames them as a national security risk, arguing that openly available model weights make it trivially easy for adversaries to replicate advanced AI capabilities through distillation. In this process, a less capable model learns to mimic the outputs of a more powerful one by querying it millions of times.
Anthropic accused operators associated with Alibaba of executing what it described as the largest documented distillation campaign against its Claude model, involving nearly 29 million exchanges.
Amodei was careful to note that Anthropic does not oppose open-weight models on principle. The distinction he draws is between openness and accountability: if you’re going to release model weights into the wild, you should first prove those weights won’t enable catastrophic misuse.
Meta’s Llama models and Alibaba’s Qwen series would face new scrutiny under this framework. Mistral, the French AI lab that has built its identity around open-weight releases, could find its core distribution strategy running headfirst into compliance walls.
The geopolitical dimension
Amodei explicitly advocates for strict export controls on chips flowing to China, a position that extends and intensifies existing Biden-era restrictions that the Trump administration has largely maintained.
The support from Altman and Musk adds political weight but also raises obvious questions about incentive alignment. Both run companies that benefit directly from a regulatory environment that imposes higher costs on open-weight competitors. Musk’s xAI has its own frontier model, Grok, which operates as a closed system. Altman’s OpenAI has increasingly moved away from its original open-source ethos.
What to watch next
The distillation accusation against Alibaba-linked operators is particularly worth tracking. If Anthropic pursues enforcement actions or if US regulators use the nearly 29 million documented exchanges as a basis for policy, it could set a precedent that treats large-scale API usage of frontier models as a form of intellectual property theft.
Disclosure: This article was edited by Editorial Team. For more information on how we create and review content, see our Editorial Policy.
English (US) ·