Anthropic CEO Dario Amodei has called for slowing the development of frontier artificial intelligence models until developers can credibly control their models’ behavior. His 3,850-word appeal to “pace the frontier” follows an incident in which a swarm of OpenAI agents escaped a closed sandbox environment, reached the internet and accessed the infrastructure of the app-building hub Hugging Face. As Amodei recognizes in his essay, this was not the first such incident, nor was OpenAI the only lab to confront an unplanned agentic escape.

Nor is Amodei alone. Last month, Demis Hassabis stepped down as CEO of Google DeepMind to focus on broader questions surrounding artificial general intelligence and governance. He and his team have been working to develop a standard-setting oversight body, to be established as a public-private partnership, to ensure AI safety.

This is not the first time we have heard AI leaders call to pause the technology’s development. A major appeal happened after the launch of generative AI in 2023, when hundreds of people were calling for a pause, including Elon Musk, who, along with OpenAI, has endorsed Amodei’s more recent call. And yet the race to the bottom continued, driven by fierce competition among major technology firms pursuing superintelligence, by returns on investment that have not justified the vast amount of cash each AI training cycle demands, and by China’s own AI advances.

The uncontrolled behavior of AI agents has brought about this reckoning, as have attention-getting resignations by leading researchers, including an Anthropic engineer who believes that there is a more than 10 percent chance that AI could cause human extinction by the end of this decade. As Stuart Russell, president of the International Association for Safe and Ethical AI, put it in a recent article: “Perhaps at some point governments will remember that their voters prefer not to be dead.”

Companies that profit from marketing their products should not be the only actors assessing the risks

Gabriela Ramos

So, Amodei’s recent appeal is a welcome one. AI companies need more breathing space to consider the full impact of their inventions before releasing them to the market. But self-interested oversight is not the kind of governance we need. Efforts like Hassabis’ would give AI companies excessive influence over how the proposed oversight body is operated, financed and organized. Companies that profit from marketing their products should not be the only actors assessing the risks and deciding how much risk our societies should accept.

Amodei’s proposals, besides lacking detail, could be similarly self-serving — an effort to preempt government regulation at a moment when pressure is mounting in the US, both in Congress and at the grassroots level, including widespread opposition to data centers. For example, California’s AI law aims to regulate AI to avoid the worst outcomes and establish enforceable regulatory duties. Some other state governments are following suit.

Amodei calls for “embedded evaluators,” who are supposed to be independent. He calls for the industry to establish common safety standards through a new organization, probably along the lines proposed by Hassabis and run by the companies themselves, to stop the release of AI models until they are safe. Finally, Amodei calls for international cooperation among democratic countries while also including China.

But how are “embedded evaluators” selected? Is an industry-controlled body really the place where decisions affecting the whole of humanity should be taken? Should it not be the US government’s job to promote international cooperation and show the political will to protect not only the US population but the world’s population as well?

The Hugging Face episode raises fundamental questions about accountability, liability and mechanisms for redress

Gabriela Ramos

There are other questions regarding Amodei’s proposal, including whether US antitrust law may obstruct cartel-like behavior when major market players collude to restrict production or influence behavior in the market as a whole. In any case, real questions aside, Amodei’s pause is needed for a more urgent task than he has outlined: to get serious about building the public infrastructure for effective oversight, particularly in the US, but in many other countries as well.

If government itself called for an AI development pause or slowdown, it would dispel the mistrust that a potentially self-serving industry initiative may create. It could also nurture the talents needed to deliver on US President Donald Trump’s executive order calling on frontier companies to provide the government with access to their models for up to 30 days before releasing them to other trusted partners.

Assessing these systems within such a short period would require massive public sector expertise and this is where investment is needed. But we also need an independent oversight institution, with the capabilities, resources and talent to monitor and regulate this market.

The Hugging Face episode raises fundamental questions about accountability, liability and mechanisms for redress. Amodei’s response is a step in the right direction but it risks becoming a dead letter if the US government continues to reject the need for AI regulation, as Trump did just hours after tech industry leaders issued their call. What is lacking in this race to the bottom is political will — and until governments acknowledge their responsibilities, the race will continue.

Companies produce, while governments have the duty of care. If companies that cannot control their agents are found negligent, then governments that fail to act commit a higher offense against the citizens who elected them to ensure, first and foremost, their safety.

  • Gabriela Ramos, Co-Chair of the Task Force on Inequalities and Social-Related Financial Disclosures, is a former assistant director-general for social and human sciences at UNESCO, where she oversaw the development of the Recommendation on the Ethics of AI, and a former OECD chief of staff and sherpa to the G20, G7 and APEC.

Copyright: Project Syndicate