Is AI going to kill us all?

In the past few weeks, fears of catastrophic AI risk have gone truly mainstream.

The panic was precipitated by several significant pieces of news, but particularly a viral post by a former Anthropic researcher, Jacob Coxon, who posted a thread on X warning people that: “The people building AI earnestly believe that it could kill us all by the end of the decade. This is not a marketing stunt.”

He then went on a media blitz appearing on CNN, NBC and Fox News, where he clarified that:

“Right now there's no risk of extinction. The current models, the worst they can do is maybe hack into something, potentially cause a lot of damages in infrastructure, but they’re not intelligent enough to outsmart us at the level that would lead to extinction.”

Coxon has not named any projects that should be shut down, did not share any information from inside Anthropic, and in general has not suggested any concrete measures that could be taken to avoid this doom. Nevertheless, the BBC ran headlines such as, ‘More Than 10% Chance AI Could Kill All Humans’, the Free Press stated that ‘Yes, AI Might Really Kill Us All’, and The Guardian ran with the headline, ‘Anthropic Researchers Say AI Could Cause Human Extinction by 2030’.

The media’s completely uncritical amplification of claims by certain AI researchers has understandably sent a shiver down many people’s spines, as they try to come to terms with yet another existential nightmare to worry about under capitalism in its period of senile decay.

But is there any basis to these fears of an imminent AI apocalypse?

I resigned from Anthropic today. I spent the last three years doing pretraining research at both OpenAI and Anthropic. Neither company is acting responsibly. They are racing straight to self-improving superintelligence and gambling with our lives. More thoughts below.

The answer to how exactly AI is going to kill us all varies depending on who you ask. Whether it’s through AI designing a lethal new super-virus, weaponising our infrastructure against us, or tricking humans into launching a thermonuclear war, experts admit that it would require this technology to become far more sophisticated than it currently is, in a way that places it outside of human control.

The real question of AI doom is therefore what Jacob Coxon identified in his interviews:

“What I find most scary is if AI is used to make itself more intelligent. […] You can take an AI and give it the problem of AI research. And then you get what’s called an intelligence explosion.”

The argument that Coxon is raising is not new to the machine learning community, where it is known as the problem of ‘Recursive Self-Improvement’ or RSI.

At first glance, the idea of RSI seems alarming, evoking long-standing science fiction visions of machine enslavement. However, based on today’s AI technology, it remains an impossibility. Numerous countervailing factors and bottlenecks strictly cap the pace of AI advancement. More fundamentally, the concept of an exponential ‘intelligence explosion’ is based on a philosophical misconception about how modern artificial intelligence actually works.

The dramatic progress of these models since the 2022 release of ChatGPT stems directly from neural scaling laws. In the late 2010s, researchers observed that machine learning performance could be steadily enhanced simply by exponentially scaling up model size and training data volumes. The ‘bitter lesson’ of machine learning was that rather than relying on architectural or algorithmic innovations, scaling the data and computational power used to train a model by 10x, 100x, or 1000x was the surest way to unlock the next level of capabilities. Today’s leading models – from GPT-6 Astra and Claude Fable to increasingly competitive open-source alternatives – all rely on this same approach.

It is not for nothing that AI companies have done everything in their power to train on every last scrap of human data. Every book written, every line of code debugged, every forum post answered, and every piece of art that they could get their hands on has been ingested. Anthropic has even been caught secretly scanning and destroying millions of hard-copy books in what is called ‘Project Panama’. The abilities of these AI models therefore come from what many people understand instinctively, which is that they have ingested the collective knowledge of humanity.

AI models are the product of our collective, historical labour, appropriated by a handful of tech monopolies without compensation, alienated from its creators, and sold back to us as a subscription service / Image: own work

Language and text is an abstraction from the physical world we inhabit, but over centuries and generations, human societies have stored vast amounts of information in our writing, the majority of which has been uploaded and made available on the internet. It is this data that has made possible the recent explosion of artificial intelligence based on ‘Large Language Models’ or LLMs. Purely by iterating over the vast volumes of human language and data available online, these LLMs have learned to map an incredible amount of information about our world.

Under capitalism, this process takes on a deeply sinister character. These AI models are the product of our collective, historical labour, appropriated by a handful of tech monopolies without compensation, alienated from its creators, and sold back to us as a subscription service.

Furthermore, because these models are privately owned, because their inner workings and training data are guarded as business secrets, and because they are specifically trained to simulate human interaction, they are presented to us not as tools, but as an alien lifeform. They present a facade of agency and even consciousness.

An important insight that flows from this understanding is that the improvements we have seen in AI abilities so far have not come from ‘cleverer algorithms’ or the development of fundamentally new underlying technology. There have been incremental advances but many of these have been to overcome barriers like using memory more efficiently to make the next round of scaling possible. Even the most advanced models today are still based on LLMs.

The AI companies – which are already hundreds of billions of dollars in debt – have a slight problem. Their valuations are based on the idea that AI will continue to develop its capabilities to swallow more and more sectors of the economy. But we’re now at a point where the next round of scaling is not so easy.

There just is not enough human-produced training data to scale up by a further order of magnitude. Simultaneously, the data centres required to train and power these behemoths are reaching their economic and thermodynamic limits, which sets real limits for how much bigger these models could feasibly even get. Nevermind the fact that the current trajectory of fossil-fuel-powered data centre expansion will all but guarantee a faster rate of global warming.

The current approach to AI model improvements is therefore colliding with very real limits. And once scaling on these axes is exhausted, there are not very many good ideas as to where to go next.

RSI-doomers will say that once we have models that are as good at AI research as humans, they will discover the next breakthrough beyond the limits of LLMs. But this requires a strong touch of magical thinking. These models are as ‘smart’ as they are because they have absorbed the near-totality of human knowledge. They are very good in domains where there is lots of human data to train on, but are very bad at figuring out things that even humans have not figured out.

People who argue that AI can be truly creative and come up with new ideas will point to the incredible advances that these models have made in coding and mathematics. It’s important to point out that these are the two domains where we would most expect LLMs to excel; there is a vast amount of high-quality training data and they deal with discrete, symbolic information, where answers can be checked and verified, unlike many other areas of human knowledge.

Nevertheless, OpenAI announced on 9 September 2026 that their models managed to achieve a breakthrough in mathematics that so far no human had managed, by solving the Navier-Stokes Millennium Prize Problem. This impressive achievement was presented as if the AI models replaced the role of human mathematicians entirely. But the evidence shows the opposite.

The approach that was used to solve the problem was based on a proposition that was made by a team of Spanish mathematicians in 2023 called the Córdoba–Martínez-Zoroa approach. To complicate matters further, it also so happened that two other mathematicians, Tristan Buckmaster and Levent Alpöge, had been using this exact same approach to try to solve the Navier-Stokes prize problem for the better part of a year using OpenAIs models via their Codex app.

The two researchers questioned whether OpenAI had used their Codex chat history to inform their attempt. OpenAI was forced to acknowledge in a statement that, “[w]hile unlikely, we cannot rule out that de-identified data derived from their usage of our products helped improve our models.” In other words, they don’t know if their model used its conversations with the mathematicians to find a solution to the problem.

View this post on Instagram            A post shared by Revolutionary Communists of America (@communistsofamerica)

Regardless of whether or not OpenAI stole these researchers’ work, it’s clear that the correct framing of the problem was supplied by a mountain of previous work by human mathematicians. The role of the OpenAI models in cracking the problem (which cannot be discounted) was to ‘brute force’ attack the problem with an incredible amount of computing power once they had been provided with a correct framing.

This example therefore proves the exact opposite of what RSI-doomers think it does. It shows that AI agents can be very powerful at computationally testing the hypotheses and questions posed by human intuition. It shows that AI, while a very powerful one, is still a tool.

The question therefore arises: why does it seem like there is an entire industry of AI-doomers that are trying so hard to convince us otherwise?

The level of sheer hysteria and hyperbole around the fear of superintelligence has led to a healthy reaction of distrust by many people online. One idea circulating is that this AI-doom narrative is a conspiracy for the AI companies to enshrine their monopoly power. But it is almost certainly not a premeditated conspiracy.

The far less remarkable truth is that the AI industry attracts scientists and engineers whose training encourages a reductive, highly computational worldview. Raised on a diet of technical problem solving, many tend to view different aspects of our world, from human intelligence to complex social questions, as if they work the same way as a computer system, rather than phenomena with their own laws of development. If you believe that everything works like a computer then the idea of RSI and an immediate AI takeover does seem terrifying.

Fear of an out-of-control AI also reflects much deeper underlying anxieties about living in a decaying social system ruled by the invisible hand of market forces, where we actually are unable to control the economic and technological systems that we have created. Much as the decay of the Roman Empire created many a doomsday cult predicting the end of the world, the predictions of AI apocalypse today reflect a semi-conscious angst about the terminal trajectory of the capitalist system.

That being said, any good capitalist knows that you never let a good crisis go to waste. Since the media firestorm around these events, Dario Amodei, the CEO of Anthropic put out a call to “pace the frontier” by slowing the rate of AI development in the frontier labs and called for the US government to put regulations in place.

Dario Amodei, the CEO of Anthropic put out a call to “pace the frontier” by slowing the rate of AI development / Image: TechCrunch, Flickr

Regardless of whether Amodei or any other individual is sincere about wanting a ‘pacing agreement’ to ensure safety, it is clear that such an agreement could be very commercially beneficial for both OpenAI and Anthropic. These companies increasingly face massive liabilities for the consequences of their misaligned models. This was most dramatically demonstrated by the Hugging Face hack, where a ‘swarm’ of OpenAI models being trained to identify and weaponise exploits in software were able to escape through a poorly insulated ‘sandbox’ environment and started hacking into another company via the internet.

The investigation into this event has revealed that the immense rush to scale up training runs means that oversight of what exactly these models are learning is lagging significantly behind. Such events have been reported both inside OpenAI and Anthropic, where engineers do not have the time to properly check that their training data and reinforcement learning environments aren’t accidentally teaching the models unwanted and destructive behaviours. Disregarding basic safety procedures is not some inevitable outcome of AI development, but the result of the anarchic race to be the latest out to market with the biggest and best model.

On top of the safety concerts, further scaling is also horrendously expensive. The current hysteria therefore presents a perfect pretext for why they might ‘slow down’ development. OpenAI and Anthropic have to contend with investors who are nervous that this technology will never repay their massive investments. Spending less money on expensive training runs would lower costs and improve their profit margins, which is very desirable for two companies preparing to raise an unprecedented amount of new funding by listing on the stock exchange.

Additionally, there may be one other reason to pursue government regulation, which is the existential danger that these two companies face from below. As was pointed out by Niels Rogge, an engineer for Hugging Face, Amodei’s call for “pacing the frontier” could be used to suppress the progress of cheap, open-source models that have been closing in on the capabilities of Anthropic’s Claude and OpenAI’s GPT-series models and which can be deployed at a fraction of the price. Collusion between OpenAI, Anthropic and the US government could be used to put in place restrictions that only the very big and well-funded labs are able to clear, thereby enacting a form of ‘regulatory capture’ to give them monopoly pricing power.

We will have to see how it plays out, but given that open-source models, particularly Chinese open-source models, are an existential threat to OpenAI/Anthropic’s business model, it explains clearly why CEOs like Dario Amodei are sounding the alarm about the rise of China. The question of ‘AI supremacy’ has become central to the inter-imperialist rivalry that defines our epoch, and in AI as in other spheres, the threat of China is used as the justification for why we must line up behind the agenda of our deranged ruling class.

What terrifies people above all else is the pervasive feeling that nobody is in control. This is precisely the root of the crisis. Even if we could trust the people steering the AI-race (which their behaviour makes very clear we cannot) the anarchic forces of capitalist competition ensure that the overall outcome is outside of human control.

The AI companies know this. Every statement they make about ‘AI safety’ distances themselves from any responsibility for the ways that they are choosing to build these systems. Always, the implication is that AI is inevitable, as if it was being conjured up by some invisible force and not actively being built by their engineers. The logic is always, ‘well, somebody is going to do this, so why not us?’

This highlights the fundamentally uncontrolled nature of market competition. Driven solely by the pursuit of short-term profit, private firms push technological developments forward without regard for their wider societal fallout, leaving the state to clean up the wreckage afterwards.

The sensational hype around an uncontrollable, existential AI doom serves as a convenient distraction from this reality: the core danger is not a rogue superintelligence, but rather that AI is already in the wrong hands.

While it will not lead to human extinction, the combination of LLMs and brute-force scaling represents an extraordinarily wasteful and dangerous approach to building artificial intelligence. Instead of rationally integrating these models into the economy at the service of human needs, the market-driven AI race is accelerating the worst tendencies of capitalism; deepening our alienation from our labour, driving down wages and working conditions, creating ever more efficient war machines, and skyrocketing energy demands that compound the climate crisis. Even the collapse of the AI boom threatens us with a worldwide recession!

Never before has the need for economic planning by the vast majority of society been posed so starkly. It must be emphasised that AI does not have to be built this way. Under a democratically planned economy, many different forms and uses of AI would make possible the planning of the economy far more harmoniously than under capitalism, and at the service of goals defined by the great majority of humanity, not unelected tech executives.

The problem that stands before us is that you cannot plan what you do not control, and you cannot control what you do not own. A revolution is necessary to take the data centres out of the hands of the doomsday cult that owns them and begin the task of redesigning both AI and our whole economic system under the conscious control of human beings.