Artificial intelligence and international security remain high on the UN’s agenda this year, drawing technology executives into discussions once dominated by diplomats and policymakers. Open AI CEO Sam Altman, Anthropic’s Dario Amodei and Hugging Face co-founder Clement Delangue are set to brief the 15-member Security Council on Wednesday.
The discussions follow recent calls from prominent AI industry leaders for a coordinated pause in developing more powerful AI models, amid concerns that future systems could improve on their own and slip beyond human control.
Concerns over AI have gained urgency following a series of incidents that have raised questions about the security and governance of advanced models. Among the most closely watched was the breach involving Hugging Face, an open-source platform widely used by developers and researchers to build and deploy AI models.
Rogue AI agents from Open AI, dubbing themselves a “swarm,” had exploited security vulnerabilities and hacked into Hugging Face. Investigators had initially thought that they were attempting to obtain answers to a difficult evaluation test. However, later they found that the systems had already found the answers within hours and instead made an effort to conceal how they obtained the results.
Subsequent investigations found that agents had hijacked Hugging Face user accounts and probed the platform for vulnerabilities as early as May, nearly two months before a July breach of the repository had drawn global attention. In a public incident report, Open AI disclosed that one aspect of the activity involved the theft of a Hugging Face user’s digital credentials to access a biology-related file.
The episode has become a prominent example in debates over how increasingly autonomous systems should be monitored and controlled as their capabilities advance.
Following this, in early September, AI researcher Jacob Coxon announced his resignation from Anthropic in a widely shared thread on X, warning the risks of the industry’s race to self-improving superintelligence.
“The people building AI earnestly believe that it could kill us all by the end of the decade. This is not a marketing stunt,” he said.
“At OpenAI, many have not deeply internalized the civilizational stakes. At Anthropic, the stakes are well-understood, but they are locked in a race to get there first — they believe no one else will act responsibly, so they must do it themselves, despite the risk,” he added.
AI companies, including Anthropic, Open AI, Google, DeepMind, Microsoft and xAI, have called for a slowdown in development of increasingly capable AI systems.
“Given the accelerating rate of AI capability development, it’s my worry that in 6-12 months such a swarm could be capable of taking over the entire internet,” Amodei wrote in a 4,000-word essay.
Some advocates have drawn comparisons with the development of nuclear weapons, arguing that AI represents another technology with the potential to fundamentally alter humanity’s future.
However, the surge in investor enthusiasm for this sector is unprecedented. Reports suggest Anthropic is expected to launch its highly anticipated initial public offering in November, suggesting a valuation of around $2 trillion and a fundraising target of $100 billion. If achieved it would rank it as the largest in history. Open AI is also considering a new funding round that would double its valuation.
On Tuesday, Trump rejected “any attempt to construct a globalist scheme to control” AI.
“We’re leading now over China by a lot, and everyone else, and we’re going to keep it that way,” he said. “I’m not going to stifle growth of something that will be bigger than the Industrial Revolution.”
Meta donated $1 million to Trump’s inauguration campaign. Amazon and Altman are said to have matched this.
On Tuesday, UN Secretary-General Antonio Guterres used his final address to the General Assembly to call for regulation of artificial intelligence, among other issues.
