A team of researchers at the University of Toronto has discovered a new class of cyberthreat that gives hackers more power and reach at far less cost. It can be built with free AI models. Every online device is a potential target. And current cyber defences are not yet ready for it.

The researchers, who released their work June 2, are believed to be the first to show that publicly accessible AI models can be used to power a worm that adapts its strategy as it spreads from one device to the next. It can seize control of an entire network and hijack computing power to allow hackers to launch sophisticated attacks at virtually no cost.

Conducted in a secure digital lab walled off from the outside world, the research shows that highly skilled hackers don’t need cutting-edge AI or deep pockets to unleash malware capable of learning, calculating and pivoting in real time – exploiting known vulnerabilities in each device as it proliferates across a system.

The findings raise profound concerns about the security of our interconnected world – from financial systems to hospitals to the networks underpinning critical services. 

“It was imperative for us to understand this threat in a controlled, academic setting before bad actors figured it out for themselves,” says Nicolas Papernot, who authored the research alongside members of his CleverHans Lab located at U of T and the Vector Institute, where he is a Canada CIFAR (Canadian Institute for Advanced Research) AI Chair.  

Papernot – who is also an associate professor of computer engineering in U of T’s Faculty of Applied Science & Engineering and computer science in the Faculty of Arts & Science – added that the research was shared only after careful scrutiny to remove any information that could aid threat actors, noting it is well understood that such are efforts are underway behind closed doors. He says he felt compelled to go public as early as possible to give researchers, policymakers and the general public a chance to protect themselves against an emerging threat that stretches from everyday laptops to HVAC systems and the energy grid.

Before publishing, the researchers shared their findings with national science, security and defence bodies and sought advice on how to responsibly release the information. 

“The reason we are doing this research is to ensure the security of the digital ecosystem we all rely on – to keep people safe. This finding catapults us into a new era of cybersecurity,” says Papernot, a faculty affiliate at U of T’s Schwartz Reisman Institute for Technology and Society, which focuses on ensuring AI is responsible, inclusive and beneficial for everyone. 

“By understanding the risks, we are now positioned to develop the countermeasures needed to detect and defend against threats like this.” 

Underestimated threats

One of the world’s leading cybersecurity experts, Papernot has made it his lab’s mission to anticipate the security concerns that matter most – even the ones the cybersecurity community isn’t paying attention to yet.

The rise of the most powerful AI models like Anthropic’s Claude Mythos has sparked widespread alarm over their unprecedented capacity to unearth hidden security flaws, even as big-tech players maintain tight controls to prevent misuse.

Papernot’s team, however, was interested in the potential misuse of smaller, relatively simple models that anyone can download and modify for free. While valuable for researchers and developers, these “open-weight” AI models can be stripped of their safety guardrails and, with enough technical knowledge, manipulated to do harm.

This risk is often downplayed on the assumption that these models lack the power to do real damage. So, Papernot’s team decided to put that assumption to the test in a safe, academic setting.

Building a prototype

A worm is a digital invader that crawls through a network, copying itself onto every device it touches – no clicks required and without users’ knowledge. If it takes root, it can wreak havoc across an entire system. Traditionally, this type of attack follows a fixed script programmed by a human. If it hits a defence it wasn’t programmed to crack, it fails. Cybersecurity experts know this and have built protections to contain such threats.

For their AI-powered version, Papernot’s team built a proof-of-concept prototype in a secure, closed system, taking extensive precautions. Their experiments emulate the capabilities of an AI-driven worm in a simulation of dozens of interconnected devices, including laptops, printers and cameras. 

The researchers’ work showed that open-weight AI models could be used to engineer a far more sophisticated threat – one that can scope out each target, tailor its attacks and take over a machine before cloning itself onto the next one. The worm also gathers information as it moves deeper into a network, with every breach revealing passwords and weak points that can unlock another machine. And because it adapts, no single defence can stop it.

The worm extends its reach at its victims’ expense. Once it embeds itself in a machine, the AI worm siphons processing power to fuel its reasoning and launch the next attack. This stolen compute propels its spread, essentially eliminating the cost of each new infection. 

“Hackers have typically had to prioritize the most high-value targets because time and computing resources were limited,” Papernot says. “But now, once a worm is launched, the cost would drop to nearly zero.”

Unlike prior research on a worm that spreads itself through AI applications, the researchers’ prototype represents a threat that can operate outside AI systems to attack the underlying software, putting a much wider range of devices at risk.

“Every device connected to the internet – laptops, cameras, smart thermostats and everything else – becomes a potential target, if not for the data it holds, then as a foothold to attack more valuable targets.”

A new era of cyberthreat

While the research demonstrates that AI worms don’t require expensive models or computing power, building one still demands technical expertise. Even so, Papernot suspects that the window for defences is rapidly closing – and that the cybersecurity world isn’t ready for what is coming.

Unlike the powerful, heavily safeguarded Mythos, the prototype does not root out unknown weaknesses. But in an uncontrolled setting, the worm could gain internet access and scan and exploit warning notices about newly discovered vulnerabilities, outpacing the software patches meant to stop them. 

Some of these can be fixed with software updates. But others are human errors such as weak passwords and sloppy IT setups that can’t be solved by pushing out a patch. That means a hacker doesn’t need the most advanced AI models to cause unprecedented damage.  

“In an interconnected world, no system is immune to this threat,” Papernot says. “Sharing these findings is the first step in galvanizing researchers, industry leaders and policymakers to take action – and quickly.” 

Every device is a potential source of information for the next attack, so locking down your own makes the whole network tougher to crack. Papernot urges IT professionals to shore up any security settings that could leave their systems exposed. Users need to do their part, too.

“Everyone has a role to play in keeping us safe,” Papernot says.

That means practising good security hygiene: Keep your devices patched and up to date. Use strong passwords. Enable multifactor authentication. 

“We can no longer afford to hit ‘ignore’ on software updates,” he says. “Every door you close is one less way in, so it’s worth taking a few minutes to reboot.”

PhD students Jonas Guan, left, Nick (Hengrui) Jia, centre, and U of T Associate Professor Nicolas Papernot (photo by Nick Iwanyshyn)

Disclosure for defence

For Papernot, publishing the findings is itself an act of defence that academic research is uniquely positioned to mount.

He points to the precedent set by University Professor Emeritus Geoffrey Hinton, who won a Nobel Prize for his role in ushering in the AI revolution. “Geoffrey has been vocal about the role academic research plays in shaping decision-making when it comes to regulating AI. This type of collective mobilization by academia, industry and governments is exactly what we need to address this new threat we have identified here with AI-driven computer worms.”  

It is a well-established practice in cybersecurity research to build proof-of-concept prototypes in controlled environments to better understand emerging threats and evaluate defences against them. Conducting such studies in an academic setting ensures that the research remains independent, upholds ethical and safety standards and is open to review and scrutiny, ultimately benefiting the broader community.

Papernot credits his co-authors and collaborators Jonas Guan, Tom Blanchard, Hanna Foerster, Hengrui Jia and Gabriel Huang for helping bring this threat to light.

His lab is already hard at work developing countermeasures. And he says U of T is the perfect place to do it. “U of T brings the deep AI expertise, multi-disciplinary talent, safe research environment, infrastructure and institutional scale crucial to solving big problems like this,” he says. “And the solutions to this problem will involve the increased availability of open-source AI models of all sizes and transparency from the companies creating the most powerful models.” 

“We’re ready to work with the rest of the world to find solutions and build a safer future.”