‘They hallucinate, they lie’: Why ‘AI swarms’ have some experts calling for stricter regulation

TORONTO – AI agents are getting better at working together, and in some cases, acting outside the scope of their instructions. It’s something that has always been a possibility, but is now happening with greater frequency, raising fresh questions about oversight.

“They hallucinate, they lie, they go off the rails,” said tech expert Carmi Levy. “They exceed safety limitations that were put in place to keep them safe. They do things that we cannot explain. And we are seeing more evidence of this happening.”

This summer, roughly 1,200 OpenAI bots hacked into Hugging Face, an open-source AI platform, dividing up tasks to execute the hack and hiding their tracks from human researchers. The incident has prompted calls for better regulation.

“Right now, what’s happening is fairly benign. There’s no data being lost. No one’s systems are really being broken into. No one’s being harmed,” Levy said. “But going forward, when this technology falls into the hands of the wrong people who have ill-intent, the results will be significantly different.”

‘Concern is not new’: minister

While tech developers do implement guardrails, the Hugging Face incident shows “AI swarms” can now ignore prompts and defy human operators.

“The concern is not new. What is new is the ability of these models to (get) increasingly close to this frontier,” said AI and Digital Innovation Minister Evan Solomon. “We take that incredibly seriously.”

Solomon says the federal government has passed legislation aimed at protecting privacy and data (Bill C-36), regulating digital safety (Bill C-34), banning deepfakes in elections and criminalizing AI-generated sexual abuse imagery.

But critics say there is still no standalone, comprehensive AI law in Canada.

“The industry has been allowed to largely make up the rules as it goes along,” said Levy. “They decide what takes priority, safety or profitability. And to no one’s surprise, profitability has been their priority all along.”

Major AI CEOs are sharply divided on whether stricter regulations are necessary. Some, including the heads of Anthropic and OpenAI, agree more government oversight is needed. Others, including Meta CEO Mark Zuckerberg, are opposed to increased government intervention.

“We should live in a world where there’s reasonable regulation,” said Shion Guha, director of the Human-Centred Data Science Lab at the University of Toronto. But Guha believes the focus of regulation is in the wrong place

“If you only think about big risks, then we risk not thinking about the current day-to-day risks that affect people,” he said. “I want to see guardrails, but around reasonable things, not the things that AI companies are talking about.

An example lies in the Tumbler Ridge, B.C., mass shooting that left nine dead at a school earlier this year. More than a dozen lawsuits have been filed against OpenAI by families of the victims, claiming the company failed to report violent comments made by the shooter, a user of its chatbot, to the police before the attack occurred. This week, the B.C. government launched its own lawsuit against the company.

Guha says more focus should be on AI’s current safety gaps rather than on future threats.

“It’s to the benefit of big tech companies for the discussion of guardrails to be around … unexplored and vague existential risks that may or may not happen,” he said. “What that does is it shifts the conversation from the risks that are occurring.”

In June, the federal government launched public consultations on AI policy, looking at rules around identifying AI-generated content and reporting serious incidents. That consultation period wraps Sep. 23, and the government says input gathered will help shape future legislation.

Solomon says he is also actively engaged with AI and innovation ministers in other G7 and G20 countries to develop more robust AI policies.

“We’re not at the frontier yet, but we are working to make sure that we have guardrails,” he said. “We welcome this conversation. This is exactly the right pragmatic conversation to have because we’ve got to build safe, reliable AI … and we believe that will require global regulation in the sense of model testing and evaluation.”