Blog
/
AI
/
October 30, 2023

Exploring AI Threats: Package Hallucination Attacks

Learn how malicious actors exploit errors in generative AI tools to launch packet attacks. Read how Darktrace products detect and prevent these threats!
Inside the SOC
Darktrace cyber analysts are world-class experts in threat intelligence, threat hunting and incident response, and provide 24/7 SOC support to thousands of Darktrace customers around the globe. Inside the SOC is exclusively authored by these experts, providing analysis of cyber incidents and threat trends, based on real-world experience in the field.
Written by
Charlotte Thompson
Cyber Analyst
Written by
Tiana Kelly
Senior Cyber Analyst & Team Lead
Default blog image
30
Oct 2023

AI tools open doors for threat actors

On November 30, 2022, the free conversational language generation model ChatGPT was launched by OpenAI, an artificial intelligence (AI) research and development company. The launch of ChatGPT was the culmination of development ongoing since 2018 and represented the latest innovation in the ongoing generative AI boom and made the use of generative AI tools accessible to the general population for the first time.

ChatGPT is estimated to currently have at least 100 million users, and in August 2023 the site reached 1.43 billion visits [1]. Darktrace data indicated that, as of March 2023, 74% of active customer environments have employees using generative AI tools in the workplace [2].

However, with new tools come new opportunities for threat actors to exploit and use them maliciously, expanding their arsenal.

Much consideration has been given to mitigating the impacts of the increased linguistic complexity in social engineering and phishing attacks resulting from generative AI tool use, with Darktrace observing a 135% increase in ‘novel social engineering attacks’ across thousands of active Darktrace/Email™ customers from January to February 2023, corresponding with the widespread adoption of ChatGPT and its peers [3].

Less overall consideration, however, has been given to impacts stemming from errors intrinsic to generative AI tools. One of these errors is AI hallucinations.

What is an AI hallucination?

AI “hallucination” is a term which refers to the predictive elements of generative AI and LLMs’ AI model gives an unexpected or factually incorrect response which does not align with its machine learning training data [4]. This differs from regular and intended behavior for an AI model, which should provide a response based on the data it was trained upon.  

Why are AI hallucinations a problem?

Despite the term indicating it might be a rare phenomenon, hallucinations are far more likely than accurate or factual results as the AI models used in LLMs are merely predictive and focus on the most probable text or outcome, rather than factual accuracy.

Given the widespread use of generative AI tools in the workplace employees are becoming significantly more likely to encounter an AI hallucination. Furthermore, if these fabricated hallucination responses are taken at face value, they could cause significant issues for an organization.

Use of generative AI in software development

Software developers may use generative AI for recommendations on how to optimize their scripts or code, or to find packages to import into their code for various uses. Software developers may ask LLMs for recommendations on specific pieces of code or how to solve a specific problem, which will likely lead to a third-party package. It is possible that packages recommended by generative AI tools could represent AI hallucinations and the packages may not have been published, or, more accurately, the packages may not have been published prior to the date at which the training data for the model halts. If these hallucinations result in common suggestions of a non-existent package, and the developer copies the code snippet wholesale, this may leave the exchanges vulnerable to attack.

Research conducted by Vulcan revealed the prevalence of AI hallucinations when ChatGPT is asked questions related to coding. After sourcing a sample of commonly asked coding questions from Stack Overflow, a question-and-answer website for programmers, researchers queried ChatGPT (in the context of Node.js and Python) and reviewed its responses. In 20% of the responses provided by ChatGPT pertaining to Node.js at least one un-published package was included, whilst the figure sat at around 35% for Python [4].

Hallucinations can be unpredictable, but would-be attackers are able to find packages to create by asking generative AI tools generic questions and checking whether the suggested packages exist already. As such, attacks using this vector are unlikely to target specific organizations, instead posing more of a widespread threat to users of generative AI tools.

Malicious packages as attack vectors

Although AI hallucinations can be unpredictable, and responses given by generative AI tools may not always be consistent, malicious actors are able to discover AI hallucinations by adopting the approach used by Vulcan. This allows hallucinated packages to be used as attack vectors. Once a malicious actor has discovered a hallucination of an un-published package, they are able to create a package with the same name and include a malicious payload, before publishing it. This is known as a malicious package.

Malicious packages could also be recommended by generative AI tools in the form of pre-existing packages. A user may be recommended a package that had previously been confirmed to contain malicious content, or a package that is no longer maintained and, therefore, is more vulnerable to hijack by malicious actors.

In such scenarios it is not necessary to manipulate the training data (data poisoning) to achieve the desired outcome for the malicious actor, thus a complex and time-consuming attack phase can easily be bypassed.

An unsuspecting software developer may incorporate a malicious package into their code, rendering it harmful. Deployment of this code could then result in compromise and escalation into a full-blown cyber-attack.

Figure 1: Flow diagram depicting the initial stages of an AI Package Hallucination Attack.

For providers of Software-as-a-Service (SaaS) products, this attack vector may represent an even greater risk. Such organizations may have a higher proportion of employed software developers than other organizations of comparable size. A threat actor, therefore, could utilize this attack vector as part of a supply chain attack, whereby a malicious payload becomes incorporated into trusted software and is then distributed to multiple customers. This type of attack could have severe consequences including data loss, the downtime of critical systems, and reputational damage.

How could Darktrace detect an AI Package Hallucination Attack?

In June 2023, Darktrace introduced a range of DETECT™ and RESPOND™ models designed to identify the use of generative AI tools within customer environments, and to autonomously perform inhibitive actions in response to such detections. These models will trigger based on connections to endpoints associated with generative AI tools, as such, Darktrace’s detection of an AI Package Hallucination Attack would likely begin with the breaching of one of the following DETECT models:

  • Compliance / Anomalous Upload to Generative AI
  • Compliance / Beaconing to Rare Generative AI and Generative AI
  • Compliance / Generative AI

Should generative AI tool use not be permitted by an organization, the Darktrace RESPOND model ‘Antigena / Network / Compliance / Antigena Generative AI Block’ can be activated to autonomously block connections to endpoints associated with generative AI, thus preventing an AI Package Hallucination attack before it can take hold.

Once a malicious package has been recommended, it may be downloaded from GitHub, a platform and cloud-based service used to store and manage code. Darktrace DETECT is able to identify when a device has performed a download from an open-source repository such as GitHub using the following models:

  • Device / Anomalous GitHub Download
  • Device / Anomalous Script Download Followed By Additional Packages

Whatever goal the malicious package has been designed to fulfil will determine the next stages of the attack. Due to their highly flexible nature, AI package hallucinations could be used as an attack vector to deliver a large variety of different malware types.

As GitHub is a commonly used service by software developers and IT professionals alike, traditional security tools may not alert customer security teams to such GitHub downloads, meaning malicious downloads may go undetected. Darktrace’s anomaly-based approach to threat detection, however, enables it to recognize subtle deviations in a device’s pre-established pattern of life which may be indicative of an emerging attack.

Subsequent anomalous activity representing the possible progression of the kill chain as part of an AI Package Hallucination Attack could then trigger an Enhanced Monitoring model. Enhanced Monitoring models are high-fidelity indicators of potential malicious activity that are investigated by the Darktrace analyst team as part of the Proactive Threat Notification (PTN) service offered by the Darktrace Security Operation Center (SOC).

Conclusion

Employees are often considered the first line of defense in cyber security; this is particularly true in the face of an AI Package Hallucination Attack.

As the use of generative AI becomes more accessible and an increasingly prevalent tool in an attacker’s toolbox, organizations will benefit from implementing company-wide policies to define expectations surrounding the use of such tools. It is simple, yet critical, for example, for employees to fact check responses provided to them by generative AI tools. All packages recommended by generative AI should also be checked by reviewing non-generated data from either external third-party or internal sources. It is also good practice to adopt caution when downloading packages with very few downloads as it could indicate the package is untrustworthy or malicious.

As of September 2023, ChatGPT Plus and Enterprise users were able to use the tool to browse the internet, expanding the data ChatGPT can access beyond the previous training data cut-off of September 2021 [5]. This feature will be expanded to all users soon [6]. ChatGPT providing up-to-date responses could prompt the evolution of this attack vector, allowing attackers to publish malicious packages which could subsequently be recommended by ChatGPT.

It is inevitable that a greater embrace of AI tools in the workplace will be seen in the coming years as the AI technology advances and existing tools become less novel and more familiar. By fighting fire with fire, using AI technology to identify AI usage, Darktrace is uniquely placed to detect and take preventative action against malicious actors capitalizing on the AI boom.

Credit to Charlotte Thompson, Cyber Analyst, Tiana Kelly, Analyst Team Lead, London, Cyber Analyst

References

[1] https://seo.ai/blog/chatgpt-user-statistics-facts

[2] https://darktrace.com/news/darktrace-addresses-generative-ai-concerns

[3] https://darktrace.com/news/darktrace-email-defends-organizations-against-evolving-cyber-threat-landscape

[4] https://vulcan.io/blog/ai-hallucinations-package-risk?nab=1&utm_referrer=https%3A%2F%2Fwww.google.com%2F

[5] https://twitter.com/OpenAI/status/1707077710047216095

[6] https://www.reuters.com/technology/openai-says-chatgpt-can-now-browse-internet-2023-09-27/

Inside the SOC
Darktrace cyber analysts are world-class experts in threat intelligence, threat hunting and incident response, and provide 24/7 SOC support to thousands of Darktrace customers around the globe. Inside the SOC is exclusively authored by these experts, providing analysis of cyber incidents and threat trends, based on real-world experience in the field.
Written by
Charlotte Thompson
Cyber Analyst
Written by
Tiana Kelly
Senior Cyber Analyst & Team Lead

More in this series

No items found.

Blog

/

Network

/

November 17, 2026

GTA VI Cyber Threats: Fake downloads, ransomware and account hijacking

Default blog imageDefault blog image

Key findings

  • Darktrace detected GTA VI-themed suspicious activity across organizations in multiple sectors ahead of the game’s release, showing how malicious gaming content can reach corporate environments outside the gaming industry.
  • Twelve Darktrace customer environments connected to gta6demo[.]asia, a domain that public threat research has associated with a fake GTA VI demo used to distribute the Vidar infostealer. Seven of the twelve communicated with infrastructure associated with AS56322 ServerAstra Kft as part of this connectivity.

·       Darktrace investigated ransomware activity at two European games companies. One incident involved DireWolf ransomware alongside reported data exfiltration. A separate Qilin incident included Active Directory reconnaissance, new administrative credentials, possible lateral movement and substantial cloud-storage transfers before ransomware deployment.

  • Another incident on a gaming organization involved a SaaS account hijack in which a compromised account authenticated from unusual IP addresses, created a new privileged account and subsequently sent a high volume of suspected malicious email. An administrative account later authenticated from rare endpoints.

GTA VI gives criminals an audience

Grand Theft Auto VI arrives on November 19 with the kind of attention few entertainment releases command [1]. Before launch, that interest is already being exploited by sites offering supposed demos and downloads.

During September and October 2026, Darktrace detected devices across multiple customer environments connecting to GTA VI-themed sites associated with suspicious activity. The organizations involved were spread across EMEA and the Americas and included Manufacturing, Finance, and Public administration and defense sectors.

Darktrace has also investigated multiple incidents affecting gaming organizations including ransomware and a SaaS account hijack. Those investigations show a separate side of the threat: attackers operating inside corporate infrastructure and accounts, sometimes well before the eventual impact becomes apparent.

Following a fake GTA VI download

On October 7, 2026, Darktrace detected a device making suspicious DNS requests for gta6-play[.]ru, an unofficial Russian-language GTA VI site that had also been flagged as likely malicious in open-source intelligence (OSINT).

‍

Figure 1: A screenshot of the suspicious endpoint gta6-play[.]ru .

‍

Figure 2: A screenshot of a “Download Studio” endpoint led to on gta6-play[.]ru.

‍

The site advertised a GTA VI download. The site led to a separate “Download Studio” page offering a Windows executable. Analysis found the file was packaged using InstallMate, a commercial installation framework, with an embedded .tsuarch section containing a compressed tiz3 archive.


Further examination identified functionality designed to inspect the environment in which the installer was running. This included checking registry entries for installed software and disk devices, enumerating network adapters, inspecting MAC addresses, and searching for running processes and window titles.


The installer also searched for processes associated with virtualization, debugging, and network analysis, including VMware and VirtualBox components, alongside debugging and monitoring utilities such as OllyDbg, WinDbg, Process Hacker, Process Monitor, Wireshark and Fiddler.

The presence of those checks did not prove that the executable was malicious. The installer contained functionality capable of extracting embedded files, but the investigation did not establish that it was being used to deploy a malicious payload. No definitive malicious payload or command-and-control infrastructure was identified. The software appeared to be InstallRex, which is classified by several security vendors as a potentially unwanted program.

That is still a useful finding; A user looking for GTA VI content could move from an unofficial gaming site to an executable that inspected the local environment, including checking for software commonly used by security researchers.

A second GTA VI lure appeared across twelve organizations

During September 2026, Darktrace alerted on HTTP connections to gta6demo[.]asia across twelve customer environments. Seven of the twelve communicated with infrastructure associated with AS56322 ServerAstra Kft as part of this connectivity.

‍

Figure 3: An example model alert generated due to a suspicious connection to gta6demo[.]asia.

‍

In a case affecting a UK-based customer, Darktrace’s Real-Time AI Analyst further investigated connections to gta6demo[.]asia and generated an incident due to the highly suspicious nature of the behavior.

‍

Figure 4: A Real-Time AI Analyst incident highlighting the suspicious activity for further investigation

‍

Public threat research has associated gta6demo[.]asia with a campaign advertising a fake GTA VI demo. Users clicking "Play Now" were reportedly directed towards an executable containing the Vidar infostealer [2].

There is a practical difference between this activity and an attack against a games company. The users encountering GTA VI-themed infrastructure can work anywhere. Darktrace saw the activity across several sectors outside Gaming, including Finance, Manufacturing, and Public administration and defense.

A major game release can therefore create security exposure well beyond its publisher. A search for a demo or unofficial download on a corporate device is enough to bring gaming-themed infrastructure into an enterprise environment.

Ransomware operators inside the gaming industry

The attacks Darktrace observed against Gaming companies themselves show different patterns.

In August 2026, Darktrace threat research identified possible malicious activity at a European game developer. Ransomware was ultimately deployed and data was exfiltrated, with the DireWolf ransomware group claiming responsibility.

Darktrace observed suspicious file downloads and beaconing from several devices to rare endpoints alongside Monero cryptocurrency mining, Active Directory and network reconnaissance, unusual administrative RDP connections, activity associated with data staging and HTTP-based exfiltration.

Much of the intrusion was therefore visible before ransomware appeared; the environment was already showing evidence of ingress tooling, external command-and-control communications (C2), reconnaissance, unusual administrative access and data movement.

A separate investigation in December 2025 provides a closer view of this pre-ransomware activity. A European games company was targeted by Qilin ransomware with Darktrace observing scanning and LDAP searches involving userAccountControl, which can expose information useful for identifying accounts and Service Principal Names within Active Directory.

New administrative credentials then appeared on a Domain Controller and the Domain Controller also generated scanning, anomalous RPC, SMB, and SSH activity, and NetShareEnum activity through srvsvc. Elsewhere in the environment, long Remote Desktop sessions and SMB enumeration appeared consistent with lateral movement.

Several devices also transferred data to cloud storage services including WebDrive and Amazon Web Services (AWS). In some cases, large downloads preceded subsequent uploads. Darktrace observed the transfer pattern, although the available telemetry does not establish that every transfer represented attacker-controlled exfiltration.

The activity preceding the Qilin deployment included Active Directory reconnaissance, privileged credential use, network and share enumeration, unusual remote access and significant data movement. Each appeared before ransomware became the defining feature of the incident.

SaaS hijacking shifts the attacker evidence

A separate investigation in May 2026 involved the hijacking of a SaaS account at a Gaming organization.

The compromised account authenticated from several IP addresses that were unusual for the organization. It then performed suspicious SaaS group administration, including creating a new privileged account. The account subsequently sent a high volume of emails assessed as likely malicious spam and potentially intended to compromise additional users.

The following day, an administrative SaaS account authenticated from rare endpoints, raising the possibility that further privileged access had been obtained.

The activity presented a different detection problem from the ransomware cases. Authentication from rare infrastructure, administrative changes, creation of a privileged account and subsequent email activity provided the strongest indications that a legitimate identity had been compromised.

For a Gaming company using hosted platforms for development, communication and business operations, a compromised account with the right permissions can provide considerable access without requiring the attacker to reproduce a conventional network intrusion.

Detecting the intrusion before impact

The ransomware investigations show why encryption should not be the first point at which an incident comes into focus. Darktrace observed LDAP and Active Directory-related reconnaissance, new privileged credentials, unusual Domain Controller activity, remote sessions, network enumeration and abnormal data movement before ransomware deployment.

The SaaS case produced different evidence: authentication from rare infrastructure, unexpected administrative changes and activity performed through a compromised account. GTA VI-themed activity required another approach again, following users through unofficial sites and redirects and examining the software offered for download.

GTA VI has put gaming back into the headlines ahead of its November release. The security activity around it is already visible beyond the games industry, while ransomware and account-compromise investigations show attackers operating inside gaming companies through very different routes.

For defenders, those routes leave evidence before the incident reaches its most damaging stage. The practical work is in recognizing those changes early enough to investigate them.

Credit to Gemma Stubberfield (Cyber Analyst), Ahmed Gardezi (Senior Cyber Analyst), Tara Gould (Malware Research Lead), Nathaniel Jones (SVP, Global Threat Intelligence)

References:

1.        https://www.rockstargames.com/VI

2.     https://www.malwarebytes.com/blog/threat-intel/2026/08/fake-gta-6-extended-look-and-demo-sites-deliver-an-infostealer

‍

Continue reading
About the author
Gemma Stubberfield
Cyber Analyst

Blog

/

AI

/

November 17, 2026

How Microsoft and Darktrace Secure the AI-Powered Enterprise

Default blog imageDefault blog image

When legitimate activity creates business risk

AI is making information more accessible and enabling autonomous agents to act across enterprise systems. But legitimate access can still result in sensitive-data exposure, unintended actions, or activity that deviates from business expectations.

Microsoft's evolving security strategy recognizes this shift, emphasizing continuous risk assessment, stronger identity protection, and security that adapts as AI systems operate. For enterprises, the challenge is increasingly about recognizing when trusted activity introduces risk, even when existing controls are functioning as intended.

Where global intelligence meets deep business context

Microsoft brings global threat intelligence and comprehensive security capabilities, helping organizations protect identities, data, applications, and infrastructure. Its integrated approach connects security signals, governance, detection, and response across increasingly complex environments.

Darktrace complements these strengths with deep, organization-specific behavioral understanding. Powered by Adaptive AI, Darktrace continuously learns the patterns and relationships unique to each business, developing a Unique Behavioral Profile for each organization that helps distinguish meaningful threats from legitimate activity. By connecting identity, activity, context, and intent, it adds another dimension to security intelligence.

Together, Microsoft and Darktrace combine global intelligence with deeper knowledge of how each customer operates. Through supported integrations, this behavioral context enriches existing Microsoft security workflows, enabling more informed investigations and precise response.

This complementary approach extends across three interconnected areas of enterprise security- AI, People and Infrastructure. From understanding how AI and agents operate to protecting trusted communications and the infrastructure businesses depend on, Darktrace strengthens Microsoft's security capabilities with behavioral insight specific to each organization.

Understanding AI beyond permissions and policies

As AI moves into business-critical operations, security teams need to understand more than which applications and agents are approved. They need visibility into how AI is used, what information it accesses, and whether its actions remain aligned with business intent.

Microsoft provides extensive AI development, governance, and security capabilities. Darktrace / SECURE AI complements these with behavioral analysis across Shadow AI, prompts, agent identities and actions, and development risk. Policy Manager helps evaluate activity against organizational AI policies, while the announced integration with Microsoft Agent 365 is designed to enrich agent risk assessments with organization-specific behavioral signals. Together, these capabilities help organizations scale AI adoption with greater confidence.

Recognizing risk within trusted relationships

Sophisticated threats increasingly exploit established business relationships, familiar communications, and legitimate accounts. A message from a trusted supplier or an action taken through an authorized identity may appear routine until considered against the organization's normal behavior.

Darktrace / EMAIL complements Microsoft's native email, identity, and data protection capabilities by understanding how people and organizations communicate. This enables more precise detection of targeted phishing, business email compromise, account takeover, and data loss, supported by proportionate response. Integrations such as Unified Quarantine within Microsoft Defender for Office 365 bring this additional protection into existing security workflows.

Seeing the bigger picture behind an attack across domains

Threats rarely unfold within a single security domain. Suspicious identity activity, unexpected cloud connections, and unusual network behavior may appear unrelated until the relationships between them reveal a developing attack.

Darktrace / HYBRID NETWORK complements Microsoft's security capabilities with organization-specific behavioral detection across network, cloud, identity, and operational technology. Real-Time AI Analyst, including Cyber AI Analyst, connects related activity into investigated incidents that can be delivered into Microsoft Sentinel, helping security teams accelerate investigations and support targeted containment across the hybrid enterprise.

Turning security intelligence into business confidence

As AI becomes more deeply integrated into business operations, organizations need security that understands not only what technology can access, but how it behaves within their unique environment.

Microsoft and Darktrace bring complementary strengths to that challenge. By connecting global threat intelligence and integrated security capabilities with deep behavioral understanding, the partnership helps organizations recognize emerging risks, respond with greater precision, and strengthen their existing security investments.

The opportunity is to turn security intelligence into meaningful business context, enabling organizations to innovate and operate AI with confidence.

Darktrace named a 2026 Microsoft Partner of the Year Finalist for ISV Innovation

Darktrace has been named a finalist for the 2026 Microsoft Partner of the Year Award in the ISV Innovation category.  

This recognition reflects our collaboration with Microsoft to bring business context to enterprise security. By connecting Microsoft's global threat intelligence and integrated security capabilities with Darktrace's deep behavioral understanding, we help organizations recognize emerging risks and respond with greater precision across AI, People and Infrastructure.

Thank you to Microsoft, our customers, and our teams for their continued collaboration.

Meet Darktrace at Microsoft Ignite and the Microsoft Frontier Tour

Explore how Darktrace and Microsoft are bringing behavioral security to AI, people, and infrastructure.

Join Darktrace at Microsoft Ignite in San Francisco, November 17–19, or connect with our experts as we join the Microsoft Frontier Tour across 21 cities worldwide.

Meet Darktrace at Microsoft Ignite

Connect with Darktrace at the Microsoft Frontier Tour

[related-resource]

Continue reading
About the author
Your data. Our AI.
Elevate your network security with Darktrace AI