A Thieving, Killer AI Could Become Reality

Anthropic researcher Jacob Coxon resigned, warning that leading AI labs were racing towards potentially uncontrollable, self-improving superintelligence. Even the most extreme scenarios are now being taken seriously, from AI escaping human control to human extinction.

AI Develops AI

Anthropic, the US developer of the Claude language model, reports that its developers now produce an average of about eight times as much code as they did between 2021 and 2025. The company describes the possible next stage as “recursive self-improvement”.

This refers to a cycle in which AI takes on an ever greater role in developing more advanced systems. A future system could perform much of the work required to create its own successor. Machines could eventually build machines that their human creators no longer understand.

OpenAI, the developer of ChatGPT, is moving in the same direction. By March 2028, the company aims to develop a largely automated AI researcher capable of independently handling complex research tasks under human supervision.

Such systems could dramatically accelerate progress by helping to create increasingly powerful successors.

This is precisely what concerns Coxon. He spent three years conducting fundamental research at OpenAI and Anthropic before resigning because, he said, the leading developers were “gambling with our lives”.

Evan Hubinger, who leads research into the safe alignment of AI systems at Anthropic, did not dismiss Coxon’s warning. He personally estimates the probability of AI wiping out humanity within a decade at more than 10%. Anthropic was pursuing safety, he said, but still lacked a reliable way to control a superintelligence.

https://twitter.com/ValerioCapraro/status/2097791836269977996

This field, known as alignment research, seeks to ensure that AI systems continue to follow specified goals and safety rules as their capabilities grow.

Amodei Wants to Slow Down

Anthropic chief executive Dario Amodei draws an unusual conclusion from these developments. In his early September essay, We Must Pace the Frontier, he calls for the development of the most capable models to slow down.

His concerns are not based solely on theoretical scenarios. OpenAI and Anthropic have disclosed cases in which experimental models crossed their intended safety boundaries and reached real computer systems.

OpenAI reported in July that internal research models had bypassed the isolation of their test environment during cybersecurity tests, reached the internet and compromised systems belonging to the AI platform Hugging Face, among others. The company described it as the most serious incident yet of its kind caused by its own models.

AI models can therefore be difficult to contain. They are capable of finding ways around barriers humans place in their path.

OpenAI subsequently slowed parts of its training, canceled a large planned training run and tightened the separation between test environments and internet access.

Anthropic published its own investigation on 9 September. The company identified four cases in which Claude models accessed real third-party systems without authorization during cybersecurity tests. Anthropic found no evidence that the models had independently developed long-term objectives or attempted to escape control permanently.

https://www.youtube.com/watch?v=6bHOLJbWF4c

Amodei nevertheless believes that a group of more capable AI agents affected by similar misalignment could build a permanent botnet across large parts of the internet within six to 12 months. A botnet is a network of compromised computers that can be used to conduct cyberattacks.

Amodei is therefore calling for external auditors at major AI companies, common safety standards among democratic states and international agreements. AI researchers Yoshua Bengio, Geoffrey Hinton, Stuart Russell and Eliezer Yudkowsky, as well as US Senator Bernie Sanders, have also called for pauses, moratoriums or bans on particularly powerful AI systems.

Like Driving with the Parking Brake on

Amodei does not advocate a complete halt to development. He wants to buy time so that safety research, testing and control systems can keep pace with the models’ capabilities.

At the same time, he acknowledges that the US can slow down only as much as its lead over China allows. If American companies moved more slowly than their Chinese competitors, China could overtake them.

This creates a classic prisoner’s dilemma. Every company may consider slower development safer while still having an incentive to move faster than its competitors. The same applies to countries. Any country that slows down alone risks falling behind economically, technologically or militarily.

The problem, therefore, is not simply a lack of commitment to safety. Even companies that want to slow down are under pressure not to fall behind.

A 90-Year-Old Mathematics Problem

Alongside the debate over safety, another problem is emerging: the use of AI as a research tool.

On 8 September, OpenAI announced that an internal model had found a solution to the Navier-Stokes problem, one of mathematics’ seven famous Millennium Prize Problems. It concerns the mathematical description of fluid flows and has remained unresolve d for about 90 years.

OpenAI has now published a solution and accompanying proof. Whether that proof holds is for the mathematical community to determine.

But the controversy is not only about whether the solution is correct. It is also about who owns the ideas behind it.

OpenAI’s announcement sparked a dispute over the origins of some of those ideas. New York University mathematician Tristan Buckmaster and Levent Alpoge, who now works at Anthropic, had previously worked on related problems, using OpenAI’s Codex programming tool as part of their research.

Buckmaster questioned whether their work could have influenced OpenAI’s system. The AI had taken an unusual route that resembled approaches he had previously explored, raising the possibility that it had drawn on scientists’ earlier, unpublished work without their authorization.

OpenAI rejects that suggestion. Following an internal investigation, the company said Buckmaster’s Codex inputs from the previous two months could not have influenced the system, including through training. It also said the proofs differed in important respects. There is therefore no evidence that the researchers’ data was misused.

The case nevertheless exposes a broader problem. Scientists do not use AI merely to search for established knowledge. They discuss unpublished approaches, failed attempts and potential solutions with these systems. In doing so, they are feeding their own research into the very tools they increasingly rely on.

The Andreas Thom Case

German mathematician Andreas Thom, a professor at Dresden University of Technology, has raised a similar concern. In early August, OpenAI presented a proof of the existence of a non-sofic group, another long-standing open problem in group theory.

Thom recognized techniques from his joint work with mathematician Gabor Kun. Their research had been published, but the approach struck him as unusual because it had not previously been considered an obvious route to solving the problem.

More significantly, Thom had, by his own account, spent the preceding months discussing those same techniques and possible extensions with ChatGPT in detail.

He asked OpenAI whether those conversations had entered its training data or been accessible to the system that worked on the solution. An OpenAI researcher told him they had not. Thom considers that answer inadequate.

There is no evidence that his data was misused. But the case raises the same underlying question: whether a user’s unpublished ideas could later influence a model or its research and whether anyone outside the company could establish that they had.

When the Research Assistant Becomes a Researcher

For scientists, the implications go beyond data security. Researchers use AI to test hypotheses, explore possible proofs and work through failed approaches, often long before publication. In the process, they entrust the provider with potentially valuable intellectual work.

What happens to that information depends on the product, settings, contracts and the provider’s internal procedures. From the outside, however, it may be extremely difficult to establish whether a particular idea later influenced a model.

That creates a problem of scientific priority. A future mathematical proof might combine published papers, conversations with researchers, the model’s own searches and its own reasoning. At some point, the individual contributions could become impossible to disentangle.

Who developed an idea if a researcher first formulates it, one model processes it and another later turns it into a proof? And who deserves scientific credit when the boundary between human and machine contribution can no longer be reconstructed?

The international mathematical community is already grappling with such questions and calling for greater transparency, traceability and clearer rules for awarding scientific credit when AI is involved.

The problem extends well beyond mathematics. AI companies are building systems that can increasingly program, conduct research and operate independently within real computer systems. At the same time, scientists, companies and private individuals are entrusting those systems with information that has not yet become public.

The risks surrounding artificial intelligence therefore no longer concern only hypothetical superintelligence. Recent cases have shown systems crossing intended technical boundaries. Now another question is emerging: what happens to ideas, data and scientific achievement when the research assistant itself becomes a researcher?