Despite certain improvements in accuracy, large language models still hallucinate a lot. So much so, in fact, that cybersecurity researchers warn that criminals can easily weaponize delusional AI outputs to spread malware throughout the internet.
According to SecurityWeek, the attack works by exploiting a persistent flaw in AI coding assistants. Basically, when these tools recommend third-party software packages, there’s a strong possibility that they include names of ones that don’t actually exist.
Astonishingly, cybersecurity researchers at Tel Aviv University and Intuit found that this scenario can be exploited in common AI coding tools ranging from Cursor to Microsoft’s Copilot, at rates of anywhere form 85 to 100 percent, depending on the specifics of the engineering task.
The attack, called “adversarial hallucination squatting,” or “hallusquatting,” takes advantage of this fact. To run it, attackers can simply identify hallucinated package names that they know AI coding assistants will reference, register them as real repositories, and stuff malware inside. That malicious package then lies in wait for the near-guarantee than an AI assistant will access it and clone it into its owner’s machine.
