The Real AI Misalignment Problem Is Capitalism

This September, some rather technical issues regarding artificial intelligence, or rather their upshot, made headlines. The reason lay in two resignations from companies leading the development of frontier AI models, namely David Robinson, OpenAI’s Safety Officer, and Jacob Coxon, a developer at Anthropic. Each resigned because they no longer believed the companies were taking seriously the risks that frontier models’ recent “advances” pose to the survival of humanity. So, not just what most of us would call a “technical issue.”

Those fears were not hype. They were based on careful argument: a September 6 blog by Jakub Pachocki, OpenAI’s chief scientist, serves as an example. It’s worth looking more closely.

One fear the blog expressed was that frontier models are on the brink of “recursive self-improvement” (or RSI): the ability to take autonomous control over their future refinement. This could mean that, very quickly, humans lose all control over where advanced AI’s development is headed, even if it involves concentrations of computational power that by themselves pose a threat to human life.

Another fear — perhaps more alarming because the danger is already baked into our future, whatever decisions about future development are made now — is that Big Tech companies are fast losing the ability even to evaluate whether their current models (without RSI) are in fact aligned to the values they were designed to serve or aligned to completely different values that don’t serve human interests at all.

The reason for this second fear sounds technical. Until now, as Pachocki explains, OpenAI relied on assessing a model’s so-called chain-of-thought (CoT) to assess what that model was doing and with what values it was attempting to align. But, as Pachocki notes in chilling language: “Unfortunately our evaluations indicate [that] our ability to rely on chain-of-thought monitoring is progressively diminishing.” Why? Because out on the “frontier,” AI models are getting ever better at hiding what they are actually doing and the values that actually guide their processing, or at least are getting better at not revealing them in their CoT.

Luckily, in the case of OpenAI’s latest model, researchers at the UK’s AI Safety Institute were able to find alternative evidence of how the model was going rogue. But the issue goes far beyond technicalities. In simple terms, OpenAI has fashioned a machine that is now smart enough to undermine our belief in anything it tells us, because, as Pachocki puts it, AI is getting better at “manipulating its own reasoning process.”

In any reasonable world, that should be reason enough not just to halt further AI advances (that is, the rush to RSI) but to stop their current development, because we can no longer reliably align them to what humans need and want. Can it be scientific or even business “progress” when, as Pachocki coolly says, “progress in generalizable alignment may not sufficiently outstrip [technological] progress in general model intelligence”? And yet Pachocki’s blog did not call for a halt in AI development. He merely proposed that RSI should be “paced,” using words echoed by Dario Amodei of Anthropic. The reason given was economic competitiveness: RSI, Pachocki wrote, “is the only way to remain at the AI frontier.”

Admittedly, OpenAI concluded three weeks later that its GPT-6.1 Astra model, which had been slated for release as the public debate raged, should be pulled for now. Rather than just sighing with relief (and hoping for similar judgement calls when the pressure is even greater, say, on the eve of Anthropic’s mooted $2 trillion flotation), it’s more prudent to ask: What is the disease of which the reasoning of OpenAI’s chief scientist is the symptom?

The disease is capitalism or, more precisely, the inevitability of a profound misalignment with the broad goals of human life — including the basic goal of continued existence — when extraordinarily powerful computation is developed almost exclusively to serve economic ends: corporate profit, national economic competition, and abstract efficiency. This is the deeper misalignment problem that Big Tech can’t openly talk about: the conflict between a technological will to power always aligned to optimizing profit and a tolerable life for humans. Fail to confront this conflict as societies, and we guarantee that AI will become predatory on humanity’s future.

This is only the most dramatic way in which today’s commercially driven AI is becoming predatory on human life. Three other ways are also crucial.

The first is the most obvious: AI is predatory on human life by taking human-produced content (almost all of it) as the inputs to train its models. This data grab was clear almost from the start, and legal disputes with the cultural industries over copyright and compensation continue. Meanwhile, other forms of data grab have emerged: the exploitation of Global South workers to train AI’s early functioning and, most recently, the enticing of professionals (once privileged but now precarious) into literally showing specialist AI how, for example, medical reasoning works. The latter work can pay well until the AI has been trained, in which case the human expertise expended becomes redundant.

AI is predatory in a second, more subtle way: by shaping our habits of thought and imagination as we get better at prompting AI and start to reframe our questions about the world in forms that predictive text machines (that is, AI) are suited to answer.

And there is a third way in which, over time, predatory commercial AI threatens to capture the human mind: by changing our understanding of what intelligence is. We are progressively overvaluing the outputs of AI’s disembodied intelligence while progressively undervaluing our own embodied intelligence — and that of other sentient animals too. Behind this third form of capture lies a deep philosophical error going back to René Descartes, so that resisting the corporate capture of the human mind will require us to forge a different philosophical language from the one currently fashionable in Big Tech.

For now, though, the challenge is more immediate, and it absolutely cannot be left to those few brave insiders in Big Tech corporations willing to sacrifice their salaries to reveal the dangerous compromises their employers are making.

It is for workers in every workplace, and in every network of freelance workers, to start to assert their capacity to say no to commercially imposed AI. After all, the reason why AI is being forced on society so fast and with so little consultation is because, without widespread and deep adoption of their products, Big Tech corporations will never recoup their huge up-front investment and start making reliable profits.

Start seriously resisting commercial AI’s adoption and a different AI future becomes possible, as Evgeny Morozov has argued: a future where communities, rather than rejecting AI outright, start getting together to decide the social goals for which they might want to use AI, on what terms, when, and where. Clearly, for workers to claim such power to determine the direction of AI adoption, some radical changes will also be needed in how work institutions currently function.

But maybe realizing that is the start of a larger recognition of the unlivable contradiction that capitalism currently forces AI to inhabit. For sure, without the boldness to imagine radical alternatives, the outlook for the “AI era” is bleak.

This article originally appeared in Jacobin, a democratic socialist magazine publishing long-form essays and analysis on politics, economics, and culture. Subscribe to the print edition for $20 a year.