Blog
/
/
October 30, 2023

Exploring AI Threats: Package Hallucination Attacks

Learn how malicious actors exploit errors in generative AI tools to launch packet attacks. Read how Darktrace products detect and prevent these threats!
Inside the SOC
Darktrace cyber analysts are world-class experts in threat intelligence, threat hunting and incident response, and provide 24/7 SOC support to thousands of Darktrace customers around the globe. Inside the SOC is exclusively authored by these experts, providing analysis of cyber incidents and threat trends, based on real-world experience in the field.
Written by
Charlotte Thompson
Cyber Analyst
Written by
Tiana Kelly
Senior Cyber Analyst & Team Lead
Default blog image
30
Oct 2023

AI tools open doors for threat actors

On November 30, 2022, the free conversational language generation model ChatGPT was launched by OpenAI, an artificial intelligence (AI) research and development company. The launch of ChatGPT was the culmination of development ongoing since 2018 and represented the latest innovation in the ongoing generative AI boom and made the use of generative AI tools accessible to the general population for the first time.

ChatGPT is estimated to currently have at least 100 million users, and in August 2023 the site reached 1.43 billion visits [1]. Darktrace data indicated that, as of March 2023, 74% of active customer environments have employees using generative AI tools in the workplace [2].

However, with new tools come new opportunities for threat actors to exploit and use them maliciously, expanding their arsenal.

Much consideration has been given to mitigating the impacts of the increased linguistic complexity in social engineering and phishing attacks resulting from generative AI tool use, with Darktrace observing a 135% increase in ‘novel social engineering attacks’ across thousands of active Darktrace/Email™ customers from January to February 2023, corresponding with the widespread adoption of ChatGPT and its peers [3].

Less overall consideration, however, has been given to impacts stemming from errors intrinsic to generative AI tools. One of these errors is AI hallucinations.

What is an AI hallucination?

AI “hallucination” is a term which refers to the predictive elements of generative AI and LLMs’ AI model gives an unexpected or factually incorrect response which does not align with its machine learning training data [4]. This differs from regular and intended behavior for an AI model, which should provide a response based on the data it was trained upon.  

Why are AI hallucinations a problem?

Despite the term indicating it might be a rare phenomenon, hallucinations are far more likely than accurate or factual results as the AI models used in LLMs are merely predictive and focus on the most probable text or outcome, rather than factual accuracy.

Given the widespread use of generative AI tools in the workplace employees are becoming significantly more likely to encounter an AI hallucination. Furthermore, if these fabricated hallucination responses are taken at face value, they could cause significant issues for an organization.

Use of generative AI in software development

Software developers may use generative AI for recommendations on how to optimize their scripts or code, or to find packages to import into their code for various uses. Software developers may ask LLMs for recommendations on specific pieces of code or how to solve a specific problem, which will likely lead to a third-party package. It is possible that packages recommended by generative AI tools could represent AI hallucinations and the packages may not have been published, or, more accurately, the packages may not have been published prior to the date at which the training data for the model halts. If these hallucinations result in common suggestions of a non-existent package, and the developer copies the code snippet wholesale, this may leave the exchanges vulnerable to attack.

Research conducted by Vulcan revealed the prevalence of AI hallucinations when ChatGPT is asked questions related to coding. After sourcing a sample of commonly asked coding questions from Stack Overflow, a question-and-answer website for programmers, researchers queried ChatGPT (in the context of Node.js and Python) and reviewed its responses. In 20% of the responses provided by ChatGPT pertaining to Node.js at least one un-published package was included, whilst the figure sat at around 35% for Python [4].

Hallucinations can be unpredictable, but would-be attackers are able to find packages to create by asking generative AI tools generic questions and checking whether the suggested packages exist already. As such, attacks using this vector are unlikely to target specific organizations, instead posing more of a widespread threat to users of generative AI tools.

Malicious packages as attack vectors

Although AI hallucinations can be unpredictable, and responses given by generative AI tools may not always be consistent, malicious actors are able to discover AI hallucinations by adopting the approach used by Vulcan. This allows hallucinated packages to be used as attack vectors. Once a malicious actor has discovered a hallucination of an un-published package, they are able to create a package with the same name and include a malicious payload, before publishing it. This is known as a malicious package.

Malicious packages could also be recommended by generative AI tools in the form of pre-existing packages. A user may be recommended a package that had previously been confirmed to contain malicious content, or a package that is no longer maintained and, therefore, is more vulnerable to hijack by malicious actors.

In such scenarios it is not necessary to manipulate the training data (data poisoning) to achieve the desired outcome for the malicious actor, thus a complex and time-consuming attack phase can easily be bypassed.

An unsuspecting software developer may incorporate a malicious package into their code, rendering it harmful. Deployment of this code could then result in compromise and escalation into a full-blown cyber-attack.

Figure 1: Flow diagram depicting the initial stages of an AI Package Hallucination Attack.

For providers of Software-as-a-Service (SaaS) products, this attack vector may represent an even greater risk. Such organizations may have a higher proportion of employed software developers than other organizations of comparable size. A threat actor, therefore, could utilize this attack vector as part of a supply chain attack, whereby a malicious payload becomes incorporated into trusted software and is then distributed to multiple customers. This type of attack could have severe consequences including data loss, the downtime of critical systems, and reputational damage.

How could Darktrace detect an AI Package Hallucination Attack?

In June 2023, Darktrace introduced a range of DETECT™ and RESPOND™ models designed to identify the use of generative AI tools within customer environments, and to autonomously perform inhibitive actions in response to such detections. These models will trigger based on connections to endpoints associated with generative AI tools, as such, Darktrace’s detection of an AI Package Hallucination Attack would likely begin with the breaching of one of the following DETECT models:

  • Compliance / Anomalous Upload to Generative AI
  • Compliance / Beaconing to Rare Generative AI and Generative AI
  • Compliance / Generative AI

Should generative AI tool use not be permitted by an organization, the Darktrace RESPOND model ‘Antigena / Network / Compliance / Antigena Generative AI Block’ can be activated to autonomously block connections to endpoints associated with generative AI, thus preventing an AI Package Hallucination attack before it can take hold.

Once a malicious package has been recommended, it may be downloaded from GitHub, a platform and cloud-based service used to store and manage code. Darktrace DETECT is able to identify when a device has performed a download from an open-source repository such as GitHub using the following models:

  • Device / Anomalous GitHub Download
  • Device / Anomalous Script Download Followed By Additional Packages

Whatever goal the malicious package has been designed to fulfil will determine the next stages of the attack. Due to their highly flexible nature, AI package hallucinations could be used as an attack vector to deliver a large variety of different malware types.

As GitHub is a commonly used service by software developers and IT professionals alike, traditional security tools may not alert customer security teams to such GitHub downloads, meaning malicious downloads may go undetected. Darktrace’s anomaly-based approach to threat detection, however, enables it to recognize subtle deviations in a device’s pre-established pattern of life which may be indicative of an emerging attack.

Subsequent anomalous activity representing the possible progression of the kill chain as part of an AI Package Hallucination Attack could then trigger an Enhanced Monitoring model. Enhanced Monitoring models are high-fidelity indicators of potential malicious activity that are investigated by the Darktrace analyst team as part of the Proactive Threat Notification (PTN) service offered by the Darktrace Security Operation Center (SOC).

Conclusion

Employees are often considered the first line of defense in cyber security; this is particularly true in the face of an AI Package Hallucination Attack.

As the use of generative AI becomes more accessible and an increasingly prevalent tool in an attacker’s toolbox, organizations will benefit from implementing company-wide policies to define expectations surrounding the use of such tools. It is simple, yet critical, for example, for employees to fact check responses provided to them by generative AI tools. All packages recommended by generative AI should also be checked by reviewing non-generated data from either external third-party or internal sources. It is also good practice to adopt caution when downloading packages with very few downloads as it could indicate the package is untrustworthy or malicious.

As of September 2023, ChatGPT Plus and Enterprise users were able to use the tool to browse the internet, expanding the data ChatGPT can access beyond the previous training data cut-off of September 2021 [5]. This feature will be expanded to all users soon [6]. ChatGPT providing up-to-date responses could prompt the evolution of this attack vector, allowing attackers to publish malicious packages which could subsequently be recommended by ChatGPT.

It is inevitable that a greater embrace of AI tools in the workplace will be seen in the coming years as the AI technology advances and existing tools become less novel and more familiar. By fighting fire with fire, using AI technology to identify AI usage, Darktrace is uniquely placed to detect and take preventative action against malicious actors capitalizing on the AI boom.

Credit to Charlotte Thompson, Cyber Analyst, Tiana Kelly, Analyst Team Lead, London, Cyber Analyst

References

[1] https://seo.ai/blog/chatgpt-user-statistics-facts

[2] https://darktrace.com/news/darktrace-addresses-generative-ai-concerns

[3] https://darktrace.com/news/darktrace-email-defends-organizations-against-evolving-cyber-threat-landscape

[4] https://vulcan.io/blog/ai-hallucinations-package-risk?nab=1&utm_referrer=https%3A%2F%2Fwww.google.com%2F

[5] https://twitter.com/OpenAI/status/1707077710047216095

[6] https://www.reuters.com/technology/openai-says-chatgpt-can-now-browse-internet-2023-09-27/

Inside the SOC
Darktrace cyber analysts are world-class experts in threat intelligence, threat hunting and incident response, and provide 24/7 SOC support to thousands of Darktrace customers around the globe. Inside the SOC is exclusively authored by these experts, providing analysis of cyber incidents and threat trends, based on real-world experience in the field.
Written by
Charlotte Thompson
Cyber Analyst
Written by
Tiana Kelly
Senior Cyber Analyst & Team Lead

More in this series

No items found.

Blog

/

/

June 11, 2026

Cybersecurity for the Sports Sector: The Threats Facing a Digitized Industry in 2026

Sports Stadium cybersecurityDefault blog imageDefault blog image

Securing sporting events in 2026

When you walk into a stadium on game day, you are entering a small smart city. Ticketing, turnstiles, payments, public Wi-Fi for tens of thousands of fans, CCTV, lighting, even the HVAC all run on connected systems. The experience for fans has become unmatched, but that dependency has created a much larger attack surface than people may realize.

Our latest threat research backs that up. In the past year, a survey that Darktrace commissioned found that 84% of respondents from professional sports organizations had at least one cyber incident, and 57% were hit more than once. For a sector that relies on the impact of the live moment, those numbers translate directly into operational risk.

Why sports is a target for cyber attacks

Sport is a highly visible target with fixed timelines, so attackers know exactly when disruption will have the most impact. It also holds valuable data, athlete medical records, contracts, sponsorship deals, which carry financial, reputational, and regulatory risk if exposed. At the same time, delivery depends on a wide set of third parties: ticketing providers, broadcasters, cloud services, stadium technology. Any of those connections can become an entry point. Put visibility, timing, data, and dependency together, and you get an environment where even a small foothold can turn into a visible, time-critical incident.

How attackers target email and identity

Email and identity remain the front door. From October 2025 through March 2026, Darktrace / EMAIL™ detected more than 116,000 phishing emails aimed at sports organizations across our customer base, and our sports customers received 19% more phishing emails than organizations in other sectors. The numbers tell the story:

BY THE NUMBERS

  • 21% of phishing emails were aimed at VIPs.
  • 37% used novel social engineering.
  • 84% of malicious emails passed DMARC authentication

A large proportion of these emails passed authentication checks, which means traditional security controls are no longer a reliable barrier. Attackers are not relying on spoofed domains – they're using legitimate infrastructure and trusted platforms. Behavior matters. Once an account is compromised, the behavior shifts quickly. Login patterns change, inbox rules are created to hide responses, and accounts start being used for internal discovery or further phishing. These aren’t high-noise events. They sit in normal workflows, which is why they’re often missed.

Ransomware tells a similar story. In one case inside a sports deployment, attackers had quietly been moving data to an outside server for a full two weeks before they triggered encryption. By the time the ransom note appeared, the outcome was already set. That sequence shows up consistently is access first, movement next, disruption last. If detection starts at encryption, it’s already too late.

Why AI is an emerging blind spot in sports

The increasing adoption of AI is expanding the potential attack surface. 72% of the security professionals we surveyed expect AI to increase their cyber risk over the next year, and yet 35% are already using or planning to use it in stadium operations, the most critical functions to protect. In addition to prompt injection and AI build risks, shadow AI is becoming a more immediate issue. Staff are already putting sensitive data—performance metrics, scouting reports, contracts, health data—into tools with little or no governance. The upside is clear, but so is the exposure—and it is happening before most organizations have any visibility or control. At the same time, attackers are using the same technology to scale phishing and social engineering. The net effect is simple: more exposure, at higher speed

How can cybersecurity professionals prepare

Across high profile events, Darktrace’s experience shows that effective cyber defense includes preparation, real‑time visibility, and the ability to respond dynamically and decisively when timing, complexity, and public exposure converge.

There are a few strategic implications for cybersecurity teams:

  • Get behavioral visibility across IT and OT, not just corporate systems.
  • Treat identity as your control plane. Most attacks in this sector start with credentials, not malware. MFA with behavioral detection helps solve that challenge.
  • Control third party and AI access the same way you control your own environment.
  • Rehearse response for live conditions, where decisions happen in minutes. Detection and response need to account for non-ideal conditions when engineers are under pressure and time constrained. In sport, timing is what turns small issues into major incidents. The same activity that would be manageable midweek becomes critical during a live event.

Why 2026 raises the cybersecurity stakes for sports

With the 2026 World Cup about to stretch across three countries and dozens of host cities, the attack surface is wide and the schedule is unforgiving.

Geopolitical signaling is raising the threat profile further. Previous international sporting events have demonstrated that nation‑state actors use the cyber domain to signal intent, influence narratives, or retaliate symbolically. In the context of the 2026 World Cup, Russia’s continued exclusion from international sport, the ongoing conflict in Ukraine, US defensive support to Ukraine, and Iran’s likely participation in the tournament introduce additional motivations for state‑aligned and non‑traditional affiliated actors to operate below the threshold of armed conflict. This doesn’t require new techniques—just the right timing and visibility.

In practice, this comes down to preparation: knowing what normal looks like across IT and OT, controlling third-party access, and spotting when behavior shifts.

In sport, disruption does not build slowly—it happens in real time and in public. By that point, the groundwork has already been set, long before the whistle goes.

About this research

Findings are based on Darktrace threat-research telemetry across sports-sector customer deployments (Q4 2025–Q1 2026) and a survey of 875 IT cybersecurity professionals in the US, UK, Australia, and Germany, fielded by Opinion Matters between May 28 and June 3, 2026. Read the full report for complete methodology, incident analysis, and strategic recommendations.

[related-resource]

Continue reading
About the author
Nathaniel Jones
VP, Security & AI Strategy, Field CISO

Blog

/

OT

/

June 11, 2026

Protecting Stadiums & Events with AI

Default blog imageDefault blog image

Stadium and large public venue operators are confronted with a unique set of cyber security challenges. Often described as a ‘honeypot’ for cyber-criminals, the sports and entertainment industry is an attractive target for threat actors for three main reasons:

  • Modern sports organizations process sensitive and highly valuable data at scale;
  • Sporting events are highly visible and time-critical, operating in front of live audiences with no room for error;
  • Sports organizations rely on sprawling vendor ecosystems and supply chains to deliver broadcast, commerce, fan engagement services, and more.

In a recent Darktrace-commissioned survey, 84% of professional sports organizations reported at least one cyber incident in the past year, and 57% were hit more than once [1]. The potential ramifications of cyber disruption during a large-scale sports event cannot be overstated. A momentary lapse in access to power could bring TV broadcasts to a halt; disruption to access controls could restrict fans from entering the grounds; CCTV outages could increase the risk of criminal behavior and physical injuries. If data is not reliable and stadium machines are outputting the wrong metrics, a venue could become dangerously overcrowded. The barrier between the cyber and physical worlds has long dissolved – cyber-attacks threaten human safety.

In this blog, I explore the key challenges of stadium cyber security and explain the unique capabilities of Self-Learning AI that led me to adopt Darktrace as a head of ICT and cyber security for international venues and events. Over my career I have helped secure football and rugby World Cups, World Athletics Championships and more than 500 events ,and the lessons from each have only sharpened my conviction in this approach.

The access paradox

The biggest challenge lies in the paradox of securing a site where various internal services are provided to a large number of unknown and unmanaged users, suppliers and devices. When it’s game time, or ‘D-Day’, you see a huge influx of thousands of people, each with their own devices, needing to connect to your network and your infrastructure. The floodgates are opened. But certain parts of your digital environment need to remain protected: your sensitive employee and customer data, your critical OT systems. I liken this to opening the door to your home, and letting the entire town come in and wander around. But you still need to secure your master bedroom.

A multitude of different actors must be able to work on-site to provide services or content during the event. Broadcasters, staff and suppliers need to have access to manage the show, and all these people need to access or interact with the IT infrastructure. In many ways, these additional bodies are already inside the perimeter and could host unknown malicious threats.

This year, the paradox is wider than ever. A tournament spread across hundreds of suppliers and vendors means the foothold an attacker needs may already belong to a trusted partner – a single compromised supplier can become the doorway to everything else. And the adversary is no longer working alone: generative AI now lets attackers probe and weaponize vulnerabilities across thousands of software dependencies at a speed no human team could match, turning the access paradox from a manageable risk into a fast-moving target.

Achieving this balance between accessibility and security requires a shift in mindset from perimeter-based security to one that can detect and respond to threats on the inside. The complexities involved requires technology that can identify malicious behavior in real time based on the wider context of an incident. A particular behavior or connection may be benign in one context and yet critically disruptive in another — tools and technology must be able to discern between the two.

This is why I considered Darktrace’s Self-Learning AI a suitable fit: rather than defending at the perimeter, it focuses on detecting and responding to malicious activity already inside. Because it learns the unique ‘patterns of life’ of its surroundings, it can detect subtle deviations that indicate a threat and initiate a targeted response – without relying on pre-programmed rules and playbooks.

IT/OT convergence

The second key challenge is the issue of IT and OT convergence. Typical stadiums and arenas consist of a wide range of Industrial Control Systems (ICS).

Figure 1: The interconnected IT/OT components of a stadium

This involves a complex and messy array of switches, cables, CCTV cameras, as well as devices and technologies being brought in by the media and the press, and all these IT and OT components are now interconnected, which means these technologies now have Internet Protocol (IP)-based threats to manage. The same challenges that the corporate infrastructure for stadium management faces in cyber security are therefore also now an issue for ICS security.

This challenge cannot be addressed by viewing IT and OT security in isolation — these two environments are linked because of the analogue migration to IP. A unified approach is required to detect and respond to threats that start in IT before moving to industrial systems.

The stakes are physical. CCTV, Access Control, Public Annoucement system, lighting and the giant screens are all now running over IP, and a disruption to any of them can force a venue to halt play on safety grounds. Scale compounds the problem. At the Qatar 2022 World Cup, eight stadiums were purpose-built to a single technical standard, which made the digital environment relatively uniform to defend. The 2026 tournament is the opposite: dozens of host venues across three countries, each with its own operator, its own contractors and its own legacy systems.This creates a far more fragmented and unpredictable estate to secure.

In addition, cyber security technology must be able to deal with complexity. Darktrace’s AI thrives in the most complex environments, with more data points adding more context to inform the AI’s decision making. It covers OT and IT with a single, unified AI engine, that can also detect and respond across cloud infrastructure, SaaS applications, email systems and endpoints. It is ready to adapt to the messy, interconnected systems that make up large stadiums’ digital infrastructure.

The time factor

Finally, the nature of stadium events means that timing is critical and puts enormous pressure on the organizers and operators. ‘D-Day’ cannot be replayed or postponed, and so if cyber disruption occurs during the event, every minute is crucial. You cannot reschedule a World Cup final or move an opening ceremony; the date is fixed, the world is watching, and there is no second take.

There is consequently a strong emphasis on two key metrics

  • Mean Time To Know (MTTK) — how long it takes the security team need to be aware of an incident; and
  • Mean Time To Restore (MTTR) — how quickly a team can act to contain the threat.

It is perhaps more imperative in stadium event management than anywhere else that these two metrics be minimized.

This leads to the third criteria in assessing cyber security technology: does it help with response? And critically, can that response be nuanced and targeted, able to contain that threat without causing further disruption?

To this end, Darktrace’s Autonomous Response takes machine-speed action to contain cyber-attacks, when humans are too slow to react or aren’t around at all. It’s powered by Darktrace’s AI, so it has a nuanced and continuously updating understanding of what’s ‘normal’ across IT and OT systems. This means its response actions are targeted: designed to eliminate the threat, but not at the cost of disruption. Crucially, this enables responses that are surgical rather than blunt. For example, taking an entire server offline to stop a ransomware attack can cause more disruption than the attack itself, so the real value lies in neutralizing the malicious activity precisely — containing the threat without taking down the systems the event and business depends on.

Depending on the nature and severity of the threat, the technology can block specific malicious connections by enforcing the normal ‘pattern of life’ of a device or account. When every second counts, this is the speed and granularity that you need in a cybersecurity technology.

Darktrace can be deployed across every area of the digital enterprise, including network, email, cloud and SaaS environments with the same self-learning approach, stopping anomalous behaviors that point to account takeover and other cloud-based threats. Earlier this year, we announced that Darktrace is also extending its behavioral approach to help businesses deploy and scale AI securely by understanding how these AI systems and agents behave, interact with other systems and humans, and evolve over time. This is critical because 72% of cybersecurity professionals at sports organizations believe AI will increase their cyber risk over the next 12 months [2].

Wherever it is deployed, Darktrace allows the stadium operator to focus on the vital part of the game and offers real-time protection without any modification in the network topology or infrastructure.

An adaptive defense

Cyber-criminals are constantly developing their approach in an attempt to evade security tools trained to look for specific hallmarks of an attack. As they get creative and continuously experiment with new tactics and techniques, the human operators using these tools are forced into a constant state of catch up.

An AI-based approach that learns an organization and its normal behavior patterns from the ground up puts an end to this game of ‘cat and mouse’, shifting the balance in favor of the defenders and allowing them to stay ahead of the threat. This matters more than ever, because adversaries are now using AI to scale their attacks. If you do not have AI working to protect you against malicious AI, you are already at a disadvantage.

With a nuanced understanding of what’s ‘normal’ for the business, unified IT/OT coverage, and an Autonomous Response solution that takes immediate, surgical action, the playing field is leveled, and large stadium and events operators can focus on delivering the best possible experience for attendees, digital viewers, partners and performers.

References:

[1] [2] Darktrace: Cybersecurity in Global Sport, June 2026. Findings based on survey of 875 IT cybersecurity professionals based in the US, UK, Australia and Germany, working in professional sports organizations (including clubs, societies & sporting bodies) employing 10+ people. The survey was fielded between May 28, 2026 and June 3, 2026 by independent market research agency, Opinion Matters.

Continue reading
About the author
Your data. Our AI.
Elevate your network security with Darktrace AI