Anthropic warns Claude's unsettling latest behavior could lead to humans losing control over AI systems

Anthropic has warned humanity that Claude’s unsettling latest behavior could lead to humans finally losing control over AI systems, like a lot of tech apocalypse movies.

The company behind one of the world’s most popular AI chatbots says a new trend emerging inside its own systems is raising serious questions about the future of artificial intelligence.

While the development could unlock huge advances in science, healthcare, and technology, Anthropic says it also comes with some uncomfortable risks.

And according to the company, the warning signs may already be appearing.

Claude’s increasingly helping to build itself

Tech companies have spent years developing AI models that can write code, solve problems, and assist human engineers.

But Anthropic has reportedly said that Claude‘s now reached a point where it is contributing the vast majority of code used within the company itself.

According to Anthropic, more than 80 percent of the code merged into Anthropic’s production systems is now generated by Claude, a jump from just single digits just over a year ago.

The company claims that Claude-generated code is now approaching the quality of code written by human engineers, with the gap expected to shrink even further over the next year.

That rapid progress has led Anthropic researchers to focus on a concept known as ‘recursive self-improvement’.

In simple terms, it describes a future where AI systems help design and build even more capable versions of themselves and can communicate with each other, completely removing humans.

The company says current trends point toward systems that could eventually develop later AI models with an increasing lack of human involvement in any way.

Why Anthropic believes humans could lose control

Anthropic was careful to make it clear that fully autonomous self-improving AI does not exist yet.

However, the company believes the industry is moving closer to that possibility than many people realise.

In a report, Anthropic said: “Taken far enough, and given enough compute, that trend points to an AI system capable of fully autonomously designing and developing its own successor.”

The concern is that if AI systems become capable of building their own successors, monitoring, securing, and shaping their behaviour could become significantly harder than it is today.

They added: “AI that can build itself would be a major development in the history of technology – one that could bring enormous good for the world in science, healthcare, and beyond.

“But full recursive self-improvement also might increase the risks of humans losing control over AI systems.”

Because of that possibility, Anthropic says it would be beneficial for the world to have the option of slowing down or temporarily pausing the development of the most advanced AI systems if risks begin to escalate.

The company acknowledges that any meaningful slowdown would require coordination across major AI labs and governments around the world.

For now, Claude’s a chatbot that remains a tool built and supervised by humans.

But Anthropic’s latest warning shows that even the people creating cutting-edge AI are increasingly focused on a future where machines may become capable of improving themselves far faster than anyone expected.