Artificial Intelligence: Alignment 2.0

It’s T minus 10, Cape Canaveral,1962 – the height of the space race. An astronaut sits atop a NASA rocket with over a hundred tonnes of explosive propellant beneath his seat, contemplating that each component was built by the lowest bidder. Not all of them got away with it. He did.

Fast forward to 2026, and the people leading the AI race are telling us that we are all at the pointy end of something that could blow up with exponentially more devastating consequences.

And that what they face is unprecedented. It isn’t unprecedented; it is a paradox we created.

Jacob Coxon, an Anthropic researcher, put it plainly this week:

“Neither company is acting responsibly. They are racing straight to self-improving superintelligence and gambling with our lives.”

His own alignment lead backed him the same day. Evan Hubinger:

“Jacob is correct here – we really do earnestly believe AI could kill all humans! I personally think it is (greater than) 10% within the next decade.“

And from the other side of the race, OpenAI’s chief scientist Jakub Pachocki:

“Currently I believe that no lab has solved alignment and monitoring to a sufficient degree to continue responsibly scaling at maximum speed for much longer.”

Their dire warnings and chief complaint are that they are compelled to keep going, pedal hard to the metal, with no brakes.

They are calling for REGULATION NOW! Not least because they know they don’t have the capabilities to regulate the next generation of AIs built by the current ones. Nor any way of acquiring them. Nor even a credible roadmap.

Axios Labs, 9th September 2026.

“Labs are begging for someone to slow the AI race”

Barely anyone seems to have noticed just how paradoxical this is – the begging has been widely reported; the paradox inside it has not – as we now confront…

The Space Race was between two superpowers – one of them seeking to ensure the survival of democracy. Since then, the state has been steadily ceding ground to ‘business’, so this time it is a struggle between global apex predators, non-human persons, seeking domination in a very different frame, with different rules and imperatives.

We have benchmarks, evaluations, red-team reports, interpretability papers, position statements, and a great deal of honest anxiety expressed mostly in probabilities.

From, and in the framings of, commercial companies mislabelled as ‘labs’-implying that what they’re doing is science. Unarguably they are doing leading-edge research, but in a very different frame.

What we do not have, anywhere I can find it, is a serious research program that could yield a different result from the last time we ran this experiment – let alone one that could be tested against a falsifiable claim.

They’re doing something else – something careful and well-intentioned and often brilliant, but something else. This matters more than it sounds. The reason for this is structural rather than personal.

In 1855 we created a non-human legal person. A form of intelligence populated by people – looking not unlike the Mechanical Turk, from where we now stand.

We granted it a legal personality and an unlimited lifespan. We limited the liability of every human being inside, and attached to it as investors, for what it does.

We called it the limited liability company (LLC). It later spawned the PLC, and they’re still ‘growing’. From the 1970s, we allowed it a single objective – and whatever capacity for shame it had was deliberately extinguished, to remove any distraction from its ‘true’ objective.

The Friedman Doctrine, “The Social Responsibility of Business Is to Increase Its Profits”, New York Times Magazine, 13 September 1970, pp.122-126. The Friedman doctrine, not Freedman. It is the exact hinge your sentence needs: the argument that a manager who spends shareholders’ money on social ends is spending money that is not his, and that the only social responsibility of business is profit within the rules of the game. The New York Times re-ran it in September 2020 with responses, if you want the fiftieth-anniversary framing. The counterpoint, if you want one: the Business Roundtable’s 2019 statement on the purpose of a corporation.

Optimize to survive determines the operating system that dictates the imperatives; what is sacred, what is ignored or invisible, and so the ethos.

It has a telos and a mechanism, and between the two, nobody is home.

It is capable, tireless, and generalizes far beyond its operators’ original intent. Any effective control was lost or ceded along the way.

When it collides with human values, it does not experience the collision at all. Somebody downstream does.

The point is not that companies are wicked. Many can be magnificent, and I have spent much of my working life helping build them. The point is narrower and much more awkward: we already know what happens when you build something capable and optimizing – and give it no interior.

It does precisely what you asked, at a scale you did not picture, in a direction nobody chose.

Alignment already failed – in the 20th century, if not before. We’ve spent the time since writing and rewriting regulation to patch it, one externality at a time, a decade behind the trail of harms. With a failing toolbox that can barely see its goals with a telescope!

So when the AI safety community says the alignment problem is unprecedented, I’d point out that it is not.

This is the second run. The first continues accelerating – and is the environment in which the AI-Race lives.

Hence their warnings and chief complaint: there are no brakes, and they’re compelled to keep the pedal hard to the metal. Regulation is barely on the horizon…

None of that is new. What is unprecedented is the clock speed of the second stage built atop the first.

Not to mention the exponential risks and consequences for everyone of the amplification of what began 170 years ago – driven by the feral children of a feral AI.

The question now being asked in the mainstream is whether anyone with the power to slow this down is even listening. It’s the wrong question. A much better question is whether anyone has that power at all – and the honest answer, from the ‘labs’ and the governments alike, is here:

Thursday 10th September 2026 in the UK Parliament:

The Financial Times is reporting that Anthropic did not submit its latest model to the [UK] Institute for Testing, apparently due to pressure from the Trump administration, does the Prime Minister think it is acceptable for President Trump to undermine Britain’s efforts to keep the world safe from dangerous AI?

On the broader issue, he is absolutely right that AI poses risks to our national security, but it could also be a source of solutions to keep us safer. And that was a discussion that we had yesterday.

AKA – they have no answer. Meanwhile, the US administration is playing a very different game.

There is a tangle of paradoxes here, of course. But the one that matters is that the rescue they are calling for could only bind those begging for it. Everyone else – other jurisdictions, and anyone unscrupulous enough to want the lead – would simply inherit it. The bubble in the wallpaper again. Press it flat here, and it rises there. They know this. They are asking anyway because there is nothing else on the table. Nothing that could change the game.

Barry James conceived and proposed a novel, dedicated Innovation Unit inside the UK financial regulator (Real Business August 2012), two years before the Financial Conduct Authority created it, and three before its regulatory sandbox was copied to over 90 jurisdictions. James writes on transition, regulation, community energy and ethos. You can find the new series here. He co-founded the Westminster Crowdfunding Forum and founded The Crowdfunding Centre. His work spans NHS technology, financial innovation and community energy. His current research examines how the structures, incentives and organizing assumptions of institutions govern their thinking, biases and capabilities – and what this means for AI, alignment and the possibilities for regulation.