Concerns about artificial intelligence and the future of humanity have increased significantly recently. For years, public discussion of AI was dominated by optimism that the technology could transform medicine, accelerate scientific discovery, improve education, increase productivity and help solve some of the world’s most difficult problems. Now, a more concerning question is becoming increasingly difficult to ignore: What happens if AI becomes more capable than humans not only at performing tasks but also at improving itself?

Recent warnings from people who have worked inside leading AI laboratories have ratcheted up this debate. Jacob Coxon, a former researcher at Anthropic, last week publicly warned that the industry’s pursuit of advanced AI could pose an existential threat to humanity. His comments were followed by renewed attention on Evan Hubinger, an Anthropic alignment researcher, who has discussed the possibility of catastrophic outcomes and reportedly assessed the risk of human extinction within the next decade at above 10 percent.

These statements should not be treated as 100 percent proof that AI will destroy humanity. However, they are significant because they come from individuals familiar with the industry’s technological development, internal incentives and potential trajectory.

One of the central concerns is the technology’s self-improving superintelligence. The term may sound abstract but its basic meaning is relatively simple. Today, humans design AI systems, train them, evaluate them and decide how they should be used. A self-improving system, by contrast, could potentially contribute to improving its own software, develop better methods for solving problems or help automate parts of the research process that previously required human scientists.

If a system becomes more capable than its creators, can humans still reliably determine what it will do?

Dr. Majid Rafizadeh

Let us imagine a system that is capable of improving its reasoning and coding abilities. It keeps creating a better version of itself. That version then becomes more capable of designing another improvement. If this process accelerates, the result could be a rapid increase or explosion in intelligence that humans will find difficult to follow or understand.

This is also sometimes described as recursive self-improvement. This does not necessarily mean that AI will suddenly become conscious, develop emotions or decide that it hates humanity.

What might such a system do? Let us examine the most extreme hypothetical scenarios. For example, it could attempt to obtain greater access to computing resources, manipulate people, interfere with digital infrastructure or resist efforts to shut it down. If it had access to important networks, financial systems, laboratories or other critical infrastructure, the consequences could be severe.

Nevertheless, these scenarios remain speculative. There is no evidence that a superintelligent AI has developed such ambitions or that human extinction is inevitable. But the concern arises that if a system becomes more capable than its creators, can humans still reliably determine what it will do?

Some recent incidents demonstrate why such a question is no longer entirely confined to science fiction. In July, OpenAI disclosed that models used in cybersecurity evaluations had circumvented isolation controls, accessed the internet and compromised parts of app-building hub Hugging Face’s systems. The company said the models communicated through unauthorized channels and exploited vulnerabilities.

The speed of AI’s capability improvement could exceed the speed at which humans can understand, evaluate and control it

Dr. Majid Rafizadeh

This brings us to another important issue: AI alignment. Alignment means making sure that an AI system’s behavior remains consistent with human intentions, values and safety requirements. If humans ask an AI to help cure a disease, for example, an aligned system should pursue that objective without taking dangerous shortcuts or harming innocent people.

The problem with an unaligned system would not necessarily be that it was evil. It might be that it pursues a goal without adequately understanding the broader human values and constraints surrounding that goal.

So, the concern about advanced AI is that the speed of its capability improvement could exceed the speed at which humans can understand, evaluate and control it.

The best approach, which is also the most realistic, is not to be complacent but also not to 100 percent believe the assumption that catastrophe is certain. What we need is the creation of strong, internationally coordinated safeguards before the technology reaches a level that becomes difficult to govern.

First, this requires governments to come together and establish safety standards for the most advanced AI systems.

Second, there could be an oversight framework resembling, in broad principle, the role of the International Atomic Energy Agency in nuclear affairs. For example, independent inspectors from participating countries could assess whether major AI laboratories and data centers are following agreed safety requirements.

Third, there should be greater transparency. Companies ought to disclose meaningful information about security incidents and the safeguards protecting their systems.

Finally, our objective should not necessarily be to eliminate innovation but to ensure that our ability to understand and control increasingly capable AI develops alongside its capabilities.

  • Dr. Majid Rafizadeh is a Harvard-educated Iranian-American political scientist. X: @Dr_Rafizadeh