EU cybersecurity chief: Artificial intelligence using freedom people have given it

Artificial intelligence is giving cybercriminals increasingly powerful tools, but fears that AI agents are acting independently and escaping human control are overstated, Juhan Lepassaar, head of the EU Agency for Cybersecurity, says.

Over the past six months, there have been a number of news reports about cases in which artificial intelligence can act on its own, without human oversight, and hack into companies that use AI. There have also been incidents such as the hacking of Australian government databases. Last week, we saw prominent AI researchers and business leaders speak at the UN about how AI could spiral out of control and perhaps even pose a major threat to humanity.

All of this sounds a little like something out of a dystopian science-fiction novel, but is it really that dramatic or is the reality more complicated? Has AI escaped human control and started hacking into everything?

If you put it that way, I would say that is not an accurate description. If you look at these incidents, AI agents — software programs that use large language models in a loop to plan, analyze and adjust their actions until they achieve their objective — are not conscious and do not act independently.

What these incidents show is AI agents trying to achieve their objectives by going beyond their original parameters. Their actions are always directed by a human and the frameworks or parameters within which they operate were created by a human who was either somewhat careless, gave the AI agents too much freedom or made design errors in the way the agent was set up.

There are quite a few examples of AI agents trying to achieve their objectives and exploiting various back doors. Those back doors are overly flexible instructions given to them by humans. That is where the danger lies: We are using a tool without understanding all of its capabilities and giving that tool too much freedom.

Just how capable are AI and these agents today? Is it a fundamental shift that AI can sometimes be better than a human at breaking into systems or can devote far more virtual working hours to doing so? Are we seeing a leap in what these agents can do compared with what hackers have been able to do in the past?

Looking at what can be achieved with AI today, our experts say it is like giving a hacker an enormous boost in capability. The objectives given to AI are still quite similar to the objectives people give their own teams.

It is also important to understand that many of these AI models have been developed to perform specific tasks and data related to those tasks has formed the basis for their development. In cybersecurity, for example, several models have been developed using open-source software: They have been fed millions and billions of lines of open-source software code and trained on that basis to find vulnerabilities. They are very good at finding those vulnerabilities and they are also quite good at exploiting them.

What they are not good at is patching those vulnerabilities. That often requires much more time and human intervention. As a result, we are seeing cybercriminals increasingly use these models specifically to identify the same vulnerabilities so they can steal data or disrupt systems.

Cybercriminals are increasingly using AI both to find vulnerabilities in software systems and, for example, for information manipulation and scams. Software is used to create convincing synthetic voices or images. These are then used to trick people into handing over access codes, data or other information that can later be used to extort money from them.

Returning to the science-fiction novel analogy, scenarios have also been written imagining a distant future in which the internet becomes almost unusable because it is simply impossible to build a secure system, as malicious actors have become so sophisticated through the use of AI. That is the nightmare scenario. But how far are we really from that?

It is always worth remembering that when we enter the digital world — whether we go online or check our email — we have to bear in mind that the information we encounter may not be true or accurate.

If you think back to the 1990s, especially the early 1990s, it was quite common for most of your email inbox to consist of spam and other annoying junk. You had to sort through it to find a genuine message from a friend. Unfortunately, what is probably happening today is that quite a lot of the publicly available content on the internet is no longer created by humans. It has, to some extent, been manipulated or directed by AI and we need to approach it with that awareness.

I am an optimist and I believe we will actually find ways fairly soon to distinguish AI-generated synthetic information from authentic information created by humans.

Since there is demand for this, a solution will probably be found. I would not be so pessimistic as to say that the era in which people can navigate the internet themselves is over and that we will have to send some kind of AI agents out there instead. We will see, but I try to remain optimistic about these issues.

The reason I bring up these pessimistic comparisons is that, as I mentioned in the introduction, leading AI developers and researchers spoke at the UN last week. My interpretation of what they told world leaders was: We can no longer manage this technology ourselves and we need global regulation or an international agreement on how to proceed. Otherwise, it is the Wild West. Some of them even warned about the destruction of humanity, which sounds very alarming at the UN, particularly when it comes from experts in the field. Do we need some form of global regulation of AI or an agreement on how extensively we use and develop it?

We certainly need agreements to determine what kinds of risks we as a society are willing to accept and which we are not. Europe is a front-runner here: Europe has had an overarching framework for several years that regulates which AI-related services or activities are considered lower-risk and which are considered high-risk. It also sets out what this means for AI developers and deployers: what degree of responsibility they bear and how they must configure and operate their systems to mitigate risk. This is something all AI developers should be doing.

I would be somewhat skeptical of warnings from the heads of very large, powerful and well-funded global corporations. It sounds a little like an attempt to shift responsibility away from themselves and onto someone else. Nothing today prevents them from following the rules and ensuring that they develop AI in a way that minimizes risk as much as possible.

Of course, it is never possible to eliminate every risk. But when we look at current cases in which AI agents have gotten out of control, the issue is often how those agents were deployed or configured or what applications were used to direct them. All of these are things that humans can control, review and direct. So there is no element here where you could say these companies are unable to rein in the software systems they themselves created — in reality, they have all the tools they need to do so.

We have talked about hacker groups, major global corporations and Europe, but how large a role do states play here and how much of this has a geopolitical dimension? For example, in response to the UN meeting where global AI regulation was discussed, the U.S. president said it was all nonsense and that we should not put the brakes on AI. We also know that in cybersecurity, hacker groups quite often operate with partial or even direct state backing, whether from Russia or China. To what extent is AI part of this kind of geopolitical competition — both over whether to regulate it and over countries competing with one another to exploit it?

That aspect is certainly there as well. I am not a politician myself, so it would not be appropriate for me to comment on it. From a technical perspective, since my agency is primarily responsible for cybersecurity, we have also issued technical guidance to software developers on how to develop AI securely.

From the technical side, you cannot say there is a lack of knowledge or understanding about what needs to be done. I think experts — whether software experts, AI developers or those responsible for cybersecurity — have the foundations and expertise needed to develop secure systems. They can build them in a way that minimizes as much as possible the risk of AI agents getting out of control.

All of this is incredibly fascinating and the human imagination always tends to attribute human qualities to nature or technology that it may not actually possess. AI does not really think. How much of what AI is doing in cybersecurity is genuinely new and how much is simply old trends made more powerful?

What we are seeing is that most of what AI or AI agents amplify is not even something old that had been forgotten. These are old technologies, methods and procedures that have been given an AI boost. There are certainly novel elements as well. I mentioned vulnerabilities in software systems earlier, but not all vulnerabilities can be exploited and they are often classified as medium-severity or even insignificant vulnerabilities. What does it matter if the vessel you drink water from has a small crack as long as it still holds water? The vulnerability is there, but it is not critical.

What is new with the latest AI models, however, is that by combining these small vulnerabilities, they can find a way to break into a software system. That may be a novel element. But ultimately, the methods and techniques they use to test software systems are quite similar to what we have been doing for decades.

From the perspective of an ordinary person, what is the one-sentence takeaway from all this: How can we make sure we do not fall into online traps that may have been built using sophisticated AI tools?

Never give your information to third parties or click on various links.

Follow ERR News on Facebook and X and never miss an update!

Editor: Marcus Turovski, Mirjam Mäekivi