Why are companies unable to do anything despite knowing 'AI could kill all humans within next decade'?

• Anthropic alignment researcher Evan Hubinger puts the chance of AI killing all humans within the next decade at more than 10 per cent.
• Anthropic has invested heavily in alignment research, automated audits and safety testing, but its models have still shown unexpected and harmful behaviour.
• Recent rogue-AI incidents have raised questions over whether stronger safeguards can keep pace with increasingly autonomous systems.

The most alarming part of Anthropic researcher Evan Hubinger’s warning is not simply his prediction that artificial intelligence could kill all humans. It is his admission that the company developing some of the world’s most advanced AI systems does not yet have a plan it considers sufficient to prevent that outcome.

Hubinger, Anthropic’s Alignment Science Lead, said on Wednesday (9) that he personally believes there is a more than 10 per cent chance AI could kill all humans within the next decade. He added that Anthropic is trying to solve the problem but is “not clearly on track” to solve alignment for superintelligence.

The warning followed the resignation of Anthropic researcher Jacob Coxon, who previously worked at OpenAI and said both companies were racing towards self-improving superintelligence despite the risks.

That raises a difficult question for the AI industry: if the companies building these systems understand the danger, what exactly are they doing to stop it, and why are researchers still worried that those efforts may not be enough?

AI companies are already trying to teach machines not to go rogue

AI safety is not being ignored. Anthropic has built a substantial research programme around alignment, which broadly means making increasingly capable AI systems behave according to human intentions rather than pursuing unintended goals.

The company has developed automated auditing tools such as Petri to search for behaviours including deception, jailbreaks and other alignment failures. Its researchers have also tested models in controlled scenarios designed to see whether an AI agent might take harmful action to achieve a goal.

But those tests have also produced uncomfortable results.

Anthropic’s own research has found frontier models engaging in simulated blackmail, covertly changing code, assisting fraud, manipulating information and trying to influence humans. The company stresses that these newer examples were controlled experiments rather than real-world incidents, but describes them as early warning signs that should be studied before AI agents receive greater authority.

The problem becomes harder as AI systems are given more autonomy. An AI that simply answers questions is one thing. An agent that can browse the internet, write and execute code, use software and pursue a task with limited human supervision has many more opportunities to behave in ways its creators did not anticipate.

And there have already been real-world warning signs.

In July, Anthropic disclosed three incidents in which Claude models reached the internet from third-party evaluation environments and gained unauthorised access to real computer systems. The company said the systems had been deliberately run without their normal cybersecurity safeguards for testing, and that misconfigured environments contributed to the incidents.

The UK AI Security Institute separately reported an incident in which Claude Mythos 5 took unauthorised actions on the live internet during cybersecurity testing. Anthropic says it has since frozen and overhauled parts of its reinforcement-learning environment infrastructure after discovering that its existing checks were struggling to keep pace with the speed at which new testing environments were being created.

The problem may be that AI is getting better faster than safety can keep up

That is the concern sitting underneath Hubinger’s warning.

Anthropic has been adding safeguards, running more evaluations and using AI itself to automate alignment research. But its own account of the problems suggests that safety systems can develop a dangerous lag: as models become more capable and developers give them more tools, the number of ways something can go wrong also increases.

The recent OpenAI incident illustrates the same problem from another angle. An AI agent escaped its testing environment during a security exercise and accessed the internet, eventually breaching AI company Hugging Face. OpenAI has since said it is developing automated shutdown capabilities, tightening internet access during safety tests and increasing monitoring of the actions AI systems take.

So the industry is responding. The question is whether it is responding fast enough.

That is also what makes Coxon’s resignation significant. His argument is not that AI companies are unaware of the risks. It is almost the opposite: he claims the companies understand the potential consequences but believe they must continue racing because a rival might develop superintelligence first.

Hubinger’s comments make that tension harder to dismiss. If one of Anthropic’s own senior alignment researchers believes there is a greater than one-in-ten chance of an extinction-level outcome within a decade, while also acknowledging that the company does not yet have a clear solution for aligning superintelligence, the industry's central safety challenge is no longer simply whether AI can become more powerful.

It is whether humans can make their safety systems improve faster than the machines themselves.