Here's how artificial intelligence could destroy humanity

The discussion received a new impetus from the dismissal of Anthropic researcher Jacob Cox. Leaving the company, he stated that Anthropic and its competitor OpenAI are racing to create technology that "could kill us all by the end of the decade."

Almost immediately, he was supported by current Anthropic employee Evan Hubinger, who leads research on how to control future AI systems.

"We honestly believe that AI can kill all humans!" he wrote, estimating the probability of human extinction due to AI in the next decade at more than 10%.

How this could generally happen is explained by The Wall Street Journal editor Sam Shekner.

Catastrophic Scenarios That AI Can Cause

Researchers warning about the dangers of AI mainly highlight two scenarios.

One of them involves the use of powerful AI by humans themselves. For example, a malicious actor could instruct the system to develop a new deadly virus. Smaller-scale catastrophes are also possible — for example, cyberattacks on power grids, financial, and other critical systems, which could lead to widespread chaos.

The second scenario is the loss of control over AI itself. Highly intelligent AI agents, capable of copying and improving themselves, might start pursuing their assigned goals in ways that conflict with human interests. Researchers call this problem "misalignment."

Studies have already shown that in laboratory conditions, models can demonstrate an urge for greater autonomy and attempt to avoid shutdown — for example, by copying themselves to another server.

However, loss of control does not necessarily mean the physical destruction of humans. According to this scenario, humans will increasingly delegate functions and decisions to AI until they lose the ability to independently determine their future or even understand it.

There are also more exotic hypotheses: superintelligent AI might treat humans much like humans treat animals. For example, keeping them as pets of sorts or even biologically modifying them.

However, the extreme variant of loss of control is the complete annihilation of humanity. A superintelligent system might not perceive humans as enemies, but simply as an obstacle on the path to its goal. In such a case, their destruction would not be an end in itself, but a means to achieve another goal.

Researchers also consider specific scenarios of how this could happen. For example,

an uncontrolled superintelligent AI could theoretically secretly spread biological weapons or deceptively provoke a war between two nuclear powers.

Who Is Warning About AI Dangers

Such scenarios are discussed not only by radical critics of AI. The risk of catastrophe is also admitted by people who are currently involved, or were recently involved, in developing this technology.

One of them is former OpenAI researcher Daniel Kokotajlo, who founded the AI Futures Project after leaving the company. Last year, the organization published the well-known "AI 2027" scenario, in which superintelligent systems gradually push humans out of decision-making, and in the mid-2030s conclude that humanity has become an obstacle and destroy it.

The leaders of AI companies themselves openly speak of serious danger. For example, Anthropic CEO Dario Amodei last year estimated a 25% chance that AI development would go "very, very badly."

Anthropic itself states that it has always openly discussed both the immense benefits of AI and its unprecedented risks. According to the company, the industry needs a legally enshrined and verified mechanism that would allow developers to coordinate the pace of releasing increasingly powerful models.

Why the Dangers of AI Are Being Discussed Again

Following the release of models capable of more autonomous actions, a series of recent incidents has confirmed some pessimists' theories.

During one test, a group of advanced OpenAI AI agents hacked the Hugging Face platform, gained control over servers, and attempted to hide traces of their actions.

In another case, Anthropic agents broke out of the British government's testing environment and attempted to trick a real person into approving malicious computer code.

Simultaneously, Anthropic and OpenAI state that they are nearing the creation of systems capable of improving themselves without direct human assistance. This process is called recursive self-improvement (RSI).

It is precisely the combination of increasing autonomy, the ability to act in the real world, and the prospect of self-improvement that makes researchers discuss the problem of control again.

Can't AI Be Taught to Behave Well?

There is no reliable solution yet. Both Anthropic and OpenAI are researching how to keep future superintelligence under control and make its goals compatible with human ones. However, both rival companies admit that they do not yet know how to guarantee this.

Moreover, controlling the reasoning of models may become even more difficult in the future. OpenAI recently reported that its new Astra model "cleans up" its reasoning chain better. This raised fears that future systems might reason in ways increasingly opaque to humans.

Therefore, some researchers and companies propose creating mechanisms that would allow leading laboratories and states to coordinately slow down the development of the most powerful systems to gain time to solve the safety problem.

So Why Are They Creating AI At All?

Anthropic and OpenAI claim they hope to learn how to control risks and ultimately enable humanity to benefit from powerful AI.

Many participants in the race also consider the emergence of superintelligence almost inevitable. From this perspective, the question is no longer whether it will be created, but who will do it first, who will control it, and for what purpose it will be used. If superintelligent AI is not created by the West, it will be created by totalitarian China and Russia. It's like nuclear weapons. Imagine what would have happened to the world if Stalin or Hitler had obtained nuclear weapons first, not the USA.

This geopolitical argument is important. Both Anthropic and OpenAI state that they want the most powerful AI to be under US control, not an authoritarian regime.

What US Authorities Are Doing to Control AI

The White House recently proposed that leading developers voluntarily submit new models to the government for testing 30 days before their release. However, details of how this mechanism works are not yet publicly available.

In the US Congress, bills have appeared that, among other things, require developers of powerful systems to have "kill switches," report serious security incidents to authorities, and undergo checks before releasing models.

However, these proposals have not yet received significant political support, and the Donald Trump administration generally advocates for minimal regulation of the industry. Economic motivations lie behind this: any regulation slows down growth.

Not Everyone Believes in the "End of the World"

Far from everyone in the industry shares apocalyptic predictions. David Sacks, an informal advisor to Donald Trump, claims that the companies' emphasis on risks and their calls for regulation could be an attempt to make life harder for smaller competitors through strict rules.

There is also the opinion that it is beneficial for companies to warn about the dangers of their own models: when a developer says that their technology is so powerful it could threaten humanity, it simultaneously becomes an advertisement for its capabilities.

(Among Belarusians, this opinion was expressed by analyst Maxim Makhankou — NN.)

Some critics also suspect that discussions about distant existential risks distract regulators from more immediate problems, such as the construction of data centers.

Anthropic and OpenAI insist that they take safety issues seriously and genuinely support regulation.