Anthropic has called for a global mechanism to pause the development of artificial intelligence, citing concerns about AI’s growing ability to engage in self-improvement, to the point that it could eventually build its own successor without human input.
The AI giant’s concerns echo those expressed by Elon Musk, among others, who in 2023 signed an open letter encouraging AI labs to pause “for at least 6 months” the training of sufficiently advanced AI systems.
Anthropic’s call comes as the company prepares to go public, having this week filed documents with the United States Securities and Exchange Commission that would allow it to pursue an initial public offering (IPO) this year. It was recently valued at just under $1tn.
Anthropic’s Call for a Global Pause Mechanism
In a new blog post on the Anthropic website co-authored by an Anthropic founder and a lead researcher at the organization, the company stated its belief that “it would be good for the world to have the option to slow or temporarily pause frontier AI development to enable societal structures and alignment research to keep up with the advance of the technology”.
The company committed its Anthropic Institute – a research arm of Anthropic dedicated to raising awareness about the implications of AI and guiding public conversation on the matter – to developing systems that would enable such a pause or slowdown in cutting-edge AI development.
The systems Anthropic has in mind would enable AI developers to verify that other labs across the globe have also paused or slowed development, guarding against the possibility that a “bad actor” could advance its own development efforts under the guise of participating in the agreed-upon break.
However, the authors warned that the unique nature of AI technology makes detecting development much more challenging than limiting proliferation in other fields, such as nuclear weaponry.
“Training runs are far easier to conceal than missile silos, their inputs are general-purpose, and the incentive to defect quietly is enormous, because whoever continues while others pause could inherit the lead”, the company said.
Anthropic also laid out the primary cause of its concern in the article, referring to a technical process known as “recursive self-improvement”, which signifies an AI system that is capable of fully autonomously designing and developing its own successor.
AI technology has not yet reached that stage, and Anthropic said it was not inevitable. But the company warned that it “could come sooner than most institutions are prepared for”.
It pointed to its own development experience, where autonomous AI agents can now run code themselves and delegate work to other AI agents. The net effect perceived by Anthropic is that the human role is “narrowing” at each step in the development process.
Anthropic argued that AI systems capable of building their own successors would mark a major technological shift, with potential benefits for science, healthcare and other fields. But full recursive self-improvement could also make it harder for humans to secure, monitor and shape AI systems.
“If systems are capable of fully building their own successors, the ways we secure them, monitor them, and shape their behavior all grow much more important”, the company said.
Elon Musk’s 2023 Call for an AI Development Pause
Anthropic’s warning and call for a global pause mechanism come just three years after a similar open letter from the Future of Life Institute drew thousands of signatures, including Elon Musk’s.
The letter described the AI development race as “out-of-control”, resulting in something that no one, including the AI labs, is capable of understanding, predicting, or reliably controlling.
The letter called on all AI labs to “immediately pause for at least 6 months” the training of AI systems more powerful than GPT-4 – a benchmark that has already long since been surpassed, by multiple models.
As with Anthropic’s suggested pause, the Future of Life letter recommended that any AI development pause be “public and verifiable, and include all key actors”.
“If such a pause cannot be enacted quickly, governments should step in and institute a moratorium”, it said.
Earlier this week, US President Donald Trump signed an executive order aimed at enhancing regulation of frontier AI development, marking a shift from the administration’s previously noninterventionist approach to the new technology.
Under the executive order, titled Promoting Advanced Artificial Intelligence Innovation and Security, tech companies would be required to provide the US government access to “covered frontier models” for a period of up to 30 days before they plan to release the model to trusted partners.
The government would also have oversight over which “trusted partners” are granted access to frontier AI models, “to promote secure innovation and strengthen the cybersecurity of critical infrastructure”.
The dangers of insufficiently regulated AI development were also recently addressed by the Catholic Church. Pope Leo XIV released last month his first encyclical letter, titled Magnifica Humanitas.
The document positioned AI as a new anthropological challenge, requiring careful discernment on the part of developers, policy-makers and wider society to ensure that the debated technology becomes a helpful tool in service of man, rather than a replacement for man.