Philp: Should we fear artificial intelligence?

In recent months, public discussion about artificial intelligence has increasingly been shaped by warnings of catastrophe. Activists, technology executives, and scientists have argued that governments may have little time to act before increasingly capable AI systems become impossible to control. Just how credible are these claims?

At the centre of these concerns is the idea of recursive self-improvement. The theory holds that an AI system could learn to improve its own software, then use that improved version to make further advances, setting off a rapid cycle of development that could eventually exceed human control.

The concern is not entirely theoretical. AI systems have already demonstrated limited forms of self-improvement under carefully controlled conditions. Researchers have shown that some systems can modify code, test the changes against established benchmarks, and build on successful results.

AI-assisted systems have also discovered mathematical procedures that had eluded human researchers for years. In narrow and clearly defined settings, machines can improve their own performance without direct human intervention at every step.

But those successes come with an important limitation.

Current AI systems perform best when there is a clear way to determine whether an answer is correct. Their improvements generally depend on mathematical proofs, formal verification systems, or computational benchmarks that provide an objective measure of success.

Remove that clear scoreboard, and the picture changes significantly.

Scientific research often requires more than finding an answer. Researchers must decide which questions are worth pursuing, recognize when an approach has reached a dead end, and judge whether an apparently interesting result is actually important. These are forms of judgment that current AI systems have not demonstrated reliably.

Recent experiments with advanced AI agents illustrate the problem. When given significant computing resources and extended periods to work on difficult, unpublished research questions, AI systems can perform many supporting tasks well. They can search academic literature, write computer code, and configure software environments.

What they often struggle to do is make sustained progress on the central scientific problem. They may continue pursuing unproductive approaches, fail to recognize promising alternatives, or use resources inefficiently.

AI can therefore appear extremely capable in short, well-defined tasks while remaining much less effective when a problem requires days of open-ended reasoning and judgment.

Even if those limitations are eventually overcome, a runaway intelligence explosion would not necessarily follow.

More powerful AI systems still depend on the physical world. Building them requires advanced computer chips, enormous data centres, electricity, cooling systems, manufacturing capacity, and complex international supply chains. Software may improve quickly, but factories and power grids cannot expand at the same speed.

There may also be diminishing returns from additional computing power or intelligence. Each major advance could require increasingly large amounts of energy, hardware, data, or time.

Claims that AI could cause human extinction deserve particularly careful scrutiny.

A catastrophe, even an extraordinarily severe one, is not the same as extinction. Eliminating every human population on Earth would require overcoming geographic isolation, differences in infrastructure, defensive responses, and countless other obstacles.

Proposed scenarios, including the deliberate creation of biological threats, generally depend on long chains of events succeeding without interruption while governments, scientists, health systems, and other institutions simultaneously fail to respond effectively.

None of this means AI presents no serious risks. Rapid technological change warrants careful oversight, strong safety research, and sensible regulation. Problems that appear difficult today may also be solved faster than expected.

But policy should distinguish between what AI systems have actually demonstrated and what researchers believe they might eventually become capable of doing.

The challenge is not to dismiss the risks, but to evaluate them with the same standards of evidence expected in other areas of science. Decisions about a technology this consequential should be driven by measurable evidence, clearly stated uncertainties, and careful analysis rather than by the assumption that catastrophe is inevitable.

Tim Philp has enjoyed science since he was old enough to read. Having worked in technical fields all his life, he shares his love of science with readers weekly. He can be reached by e-mail at tphilp@bfree.on.ca.