Blog
/
AI
/
October 30, 2023

Exploring AI Threats: Package Hallucination Attacks

Learn how malicious actors exploit errors in generative AI tools to launch packet attacks. Read how Darktrace products detect and prevent these threats!
Inside the SOC
Darktrace cyber analysts are world-class experts in threat intelligence, threat hunting and incident response, and provide 24/7 SOC support to thousands of Darktrace customers around the globe. Inside the SOC is exclusively authored by these experts, providing analysis of cyber incidents and threat trends, based on real-world experience in the field.
Written by
Charlotte Thompson
Cyber Analyst
Written by
Tiana Kelly
Deputy Team Lead, London & Cyber Analyst
Default blog imageDefault blog imageDefault blog imageDefault blog imageDefault blog imageDefault blog image
30
Oct 2023

AI tools open doors for threat actors

On November 30, 2022, the free conversational language generation model ChatGPT was launched by OpenAI, an artificial intelligence (AI) research and development company. The launch of ChatGPT was the culmination of development ongoing since 2018 and represented the latest innovation in the ongoing generative AI boom and made the use of generative AI tools accessible to the general population for the first time.

ChatGPT is estimated to currently have at least 100 million users, and in August 2023 the site reached 1.43 billion visits [1]. Darktrace data indicated that, as of March 2023, 74% of active customer environments have employees using generative AI tools in the workplace [2].

However, with new tools come new opportunities for threat actors to exploit and use them maliciously, expanding their arsenal.

Much consideration has been given to mitigating the impacts of the increased linguistic complexity in social engineering and phishing attacks resulting from generative AI tool use, with Darktrace observing a 135% increase in ‘novel social engineering attacks’ across thousands of active Darktrace/Email™ customers from January to February 2023, corresponding with the widespread adoption of ChatGPT and its peers [3].

Less overall consideration, however, has been given to impacts stemming from errors intrinsic to generative AI tools. One of these errors is AI hallucinations.

What is an AI hallucination?

AI “hallucination” is a term which refers to the predictive elements of generative AI and LLMs’ AI model gives an unexpected or factually incorrect response which does not align with its machine learning training data [4]. This differs from regular and intended behavior for an AI model, which should provide a response based on the data it was trained upon.  

Why are AI hallucinations a problem?

Despite the term indicating it might be a rare phenomenon, hallucinations are far more likely than accurate or factual results as the AI models used in LLMs are merely predictive and focus on the most probable text or outcome, rather than factual accuracy.

Given the widespread use of generative AI tools in the workplace employees are becoming significantly more likely to encounter an AI hallucination. Furthermore, if these fabricated hallucination responses are taken at face value, they could cause significant issues for an organization.

Use of generative AI in software development

Software developers may use generative AI for recommendations on how to optimize their scripts or code, or to find packages to import into their code for various uses. Software developers may ask LLMs for recommendations on specific pieces of code or how to solve a specific problem, which will likely lead to a third-party package. It is possible that packages recommended by generative AI tools could represent AI hallucinations and the packages may not have been published, or, more accurately, the packages may not have been published prior to the date at which the training data for the model halts. If these hallucinations result in common suggestions of a non-existent package, and the developer copies the code snippet wholesale, this may leave the exchanges vulnerable to attack.

Research conducted by Vulcan revealed the prevalence of AI hallucinations when ChatGPT is asked questions related to coding. After sourcing a sample of commonly asked coding questions from Stack Overflow, a question-and-answer website for programmers, researchers queried ChatGPT (in the context of Node.js and Python) and reviewed its responses. In 20% of the responses provided by ChatGPT pertaining to Node.js at least one un-published package was included, whilst the figure sat at around 35% for Python [4].

Hallucinations can be unpredictable, but would-be attackers are able to find packages to create by asking generative AI tools generic questions and checking whether the suggested packages exist already. As such, attacks using this vector are unlikely to target specific organizations, instead posing more of a widespread threat to users of generative AI tools.

Malicious packages as attack vectors

Although AI hallucinations can be unpredictable, and responses given by generative AI tools may not always be consistent, malicious actors are able to discover AI hallucinations by adopting the approach used by Vulcan. This allows hallucinated packages to be used as attack vectors. Once a malicious actor has discovered a hallucination of an un-published package, they are able to create a package with the same name and include a malicious payload, before publishing it. This is known as a malicious package.

Malicious packages could also be recommended by generative AI tools in the form of pre-existing packages. A user may be recommended a package that had previously been confirmed to contain malicious content, or a package that is no longer maintained and, therefore, is more vulnerable to hijack by malicious actors.

In such scenarios it is not necessary to manipulate the training data (data poisoning) to achieve the desired outcome for the malicious actor, thus a complex and time-consuming attack phase can easily be bypassed.

An unsuspecting software developer may incorporate a malicious package into their code, rendering it harmful. Deployment of this code could then result in compromise and escalation into a full-blown cyber-attack.

Figure 1: Flow diagram depicting the initial stages of an AI Package Hallucination Attack.

For providers of Software-as-a-Service (SaaS) products, this attack vector may represent an even greater risk. Such organizations may have a higher proportion of employed software developers than other organizations of comparable size. A threat actor, therefore, could utilize this attack vector as part of a supply chain attack, whereby a malicious payload becomes incorporated into trusted software and is then distributed to multiple customers. This type of attack could have severe consequences including data loss, the downtime of critical systems, and reputational damage.

How could Darktrace detect an AI Package Hallucination Attack?

In June 2023, Darktrace introduced a range of DETECT™ and RESPOND™ models designed to identify the use of generative AI tools within customer environments, and to autonomously perform inhibitive actions in response to such detections. These models will trigger based on connections to endpoints associated with generative AI tools, as such, Darktrace’s detection of an AI Package Hallucination Attack would likely begin with the breaching of one of the following DETECT models:

  • Compliance / Anomalous Upload to Generative AI
  • Compliance / Beaconing to Rare Generative AI and Generative AI
  • Compliance / Generative AI

Should generative AI tool use not be permitted by an organization, the Darktrace RESPOND model ‘Antigena / Network / Compliance / Antigena Generative AI Block’ can be activated to autonomously block connections to endpoints associated with generative AI, thus preventing an AI Package Hallucination attack before it can take hold.

Once a malicious package has been recommended, it may be downloaded from GitHub, a platform and cloud-based service used to store and manage code. Darktrace DETECT is able to identify when a device has performed a download from an open-source repository such as GitHub using the following models:

  • Device / Anomalous GitHub Download
  • Device / Anomalous Script Download Followed By Additional Packages

Whatever goal the malicious package has been designed to fulfil will determine the next stages of the attack. Due to their highly flexible nature, AI package hallucinations could be used as an attack vector to deliver a large variety of different malware types.

As GitHub is a commonly used service by software developers and IT professionals alike, traditional security tools may not alert customer security teams to such GitHub downloads, meaning malicious downloads may go undetected. Darktrace’s anomaly-based approach to threat detection, however, enables it to recognize subtle deviations in a device’s pre-established pattern of life which may be indicative of an emerging attack.

Subsequent anomalous activity representing the possible progression of the kill chain as part of an AI Package Hallucination Attack could then trigger an Enhanced Monitoring model. Enhanced Monitoring models are high-fidelity indicators of potential malicious activity that are investigated by the Darktrace analyst team as part of the Proactive Threat Notification (PTN) service offered by the Darktrace Security Operation Center (SOC).

Conclusion

Employees are often considered the first line of defense in cyber security; this is particularly true in the face of an AI Package Hallucination Attack.

As the use of generative AI becomes more accessible and an increasingly prevalent tool in an attacker’s toolbox, organizations will benefit from implementing company-wide policies to define expectations surrounding the use of such tools. It is simple, yet critical, for example, for employees to fact check responses provided to them by generative AI tools. All packages recommended by generative AI should also be checked by reviewing non-generated data from either external third-party or internal sources. It is also good practice to adopt caution when downloading packages with very few downloads as it could indicate the package is untrustworthy or malicious.

As of September 2023, ChatGPT Plus and Enterprise users were able to use the tool to browse the internet, expanding the data ChatGPT can access beyond the previous training data cut-off of September 2021 [5]. This feature will be expanded to all users soon [6]. ChatGPT providing up-to-date responses could prompt the evolution of this attack vector, allowing attackers to publish malicious packages which could subsequently be recommended by ChatGPT.

It is inevitable that a greater embrace of AI tools in the workplace will be seen in the coming years as the AI technology advances and existing tools become less novel and more familiar. By fighting fire with fire, using AI technology to identify AI usage, Darktrace is uniquely placed to detect and take preventative action against malicious actors capitalizing on the AI boom.

Credit to Charlotte Thompson, Cyber Analyst, Tiana Kelly, Analyst Team Lead, London, Cyber Analyst

References

[1] https://seo.ai/blog/chatgpt-user-statistics-facts

[2] https://darktrace.com/news/darktrace-addresses-generative-ai-concerns

[3] https://darktrace.com/news/darktrace-email-defends-organizations-against-evolving-cyber-threat-landscape

[4] https://vulcan.io/blog/ai-hallucinations-package-risk?nab=1&utm_referrer=https%3A%2F%2Fwww.google.com%2F

[5] https://twitter.com/OpenAI/status/1707077710047216095

[6] https://www.reuters.com/technology/openai-says-chatgpt-can-now-browse-internet-2023-09-27/

Inside the SOC
Darktrace cyber analysts are world-class experts in threat intelligence, threat hunting and incident response, and provide 24/7 SOC support to thousands of Darktrace customers around the globe. Inside the SOC is exclusively authored by these experts, providing analysis of cyber incidents and threat trends, based on real-world experience in the field.
Written by
Charlotte Thompson
Cyber Analyst
Written by
Tiana Kelly
Deputy Team Lead, London & Cyber Analyst

More in this series

No items found.

Blog

/

Network

/

June 27, 2025

Patch and Persist: Darktrace’s Detection of Blind Eagle (APT-C-36)

login on laptop dual factor authenticationDefault blog imageDefault blog image

What is Blind Eagle?

Since 2018, APT-C-36, also known as Blind Eagle, has been observed performing cyber-attacks targeting various sectors across multiple countries in Latin America, with a particular focus on Colombian organizations.

Blind Eagle characteristically targets government institutions, financial organizations, and critical infrastructure [1][2].

Attacks carried out by Blind Eagle actors typically start with a phishing email and the group have been observed utilizing various Remote Access Trojans (RAT) variants, which often have in-built methods for hiding command-and-control (C2) traffic from detection [3].

What we know about Blind Eagle from a recent campaign

Since November 2024, Blind Eagle actors have been conducting an ongoing campaign targeting Colombian organizations [1].

In this campaign, threat actors have been observed using phishing emails to deliver malicious URL links to targeted recipients, similar to the way threat actors have previously been observed exploiting CVE-2024-43451, a vulnerability in Microsoft Windows that allows the disclosure of a user’s NTLMv2 password hash upon minimal interaction with a malicious file [4].

Despite Microsoft patching this vulnerability in November 2024 [1][4], Blind Eagle actors have continued to exploit the minimal interaction mechanism, though no longer with the intent of harvesting NTLMv2 password hashes. Instead, phishing emails are sent to targets containing a malicious URL which, when clicked, initiates the download of a malicious file. This file is then triggered by minimal user interaction.

Clicking on the file triggers a WebDAV request, with a connection being made over HTTP port 80 using the user agent ‘Microsoft-WebDAV-MiniRedir/10.0.19044’. WebDAV is a transmission protocol which allows files or complete directories to be made available through the internet, and to be transmitted to devices [5]. The next stage payload is then downloaded via another WebDAV request and malware is executed on the target device.

Attackers are notified when a recipient downloads the malicious files they send, providing an insight into potential targets [1].

Darktrace’s coverage of Blind Eagle

In late February 2025, Darktrace observed activity assessed with medium confidence to be  associated with Blind Eagle on the network of a customer in Colombia.

Within a period of just five hours, Darktrace / NETWORK detected a device being redirected through a rare external location, downloading multiple executable files, and ultimately exfiltrating data from the customer’s environment.

Since the customer did not have Darktrace’s Autonomous Response capability enabled on their network, no actions were taken to contain the compromise, allowing it to escalate until the customer’s security team responded to the alerts provided by Darktrace.

Darktrace observed a device on the customer’s network being directed over HTTP to a rare external IP, namely 62[.]60[.]226[.]112, which had never previously been seen in this customer’s environment and was geolocated in Germany. Multiple open-source intelligence (OSINT) providers have since linked this endpoint with phishing and malware campaigns [9].

The device then proceeded to download the executable file hxxp://62[.]60[.]226[.]112/file/3601_2042.exe.

Darktrace’s detection of the affected device connecting to an unusual location based in Germany.
Figure 1: Darktrace’s detection of the affected device connecting to an unusual location based in Germany.
Darktrace’s detection of the affected device downloading an executable file from the suspicious endpoint.
Figure 2: Darktrace’s detection of the affected device downloading an executable file from the suspicious endpoint.

The device was then observed making unusual connections to the rare endpoint 21ene.ip-ddns[.]com and performing unusual external data activity.

This dynamic DNS endpoint allows a device to access an endpoint using a domain name in place of a changing IP address. Dynamic DNS services ensure the DNS record of a domain name is automatically updated when the IP address changes. As such, malicious actors can use these services and endpoints to dynamically establish connections to C2 infrastructure [6].

Further investigation into this dynamic endpoint using OSINT revealed multiple associations with previous likely Blind Eagle compromises, as well as Remcos malware, a RAT commonly deployed via phishing campaigns [7][8][10].

Darktrace’s detection of the affected device connecting to the suspicious dynamic DNS endpoint, 21ene.ip-ddns[.]com.
Figure 3: Darktrace’s detection of the affected device connecting to the suspicious dynamic DNS endpoint, 21ene.ip-ddns[.]com.

Shortly after this, Darktrace observed the user agent ‘Microsoft-WebDAV-MiniRedir/10.0.19045’, indicating usage of the aforementioned transmission protocol WebDAV. The device was subsequently observed connected to an endpoint associated with Github and downloading data, suggesting that the device was retrieving a malicious tool or payload. The device then began to communicate to the malicious endpoint diciembrenotasenclub[.]longmusic[.]com over the new TCP port 1512 [11].

Around this time, the device was also observed uploading data to the endpoints 21ene.ip-ddns[.]com and diciembrenotasenclub[.]longmusic[.]com, with transfers of 60 MiB and 5.6 MiB observed respectively.

Figure 4: UI graph showing external data transfer activity.

This chain of activity triggered an Enhanced Monitoring model alert in Darktrace / NETWORK. These high-priority model alerts are designed to trigger in response to higher fidelity indicators of compromise (IoCs), suggesting that a device is performing activity consistent with a compromise.

 Darktrace’s detection of initial attack chain activity.
Figure 5: Darktrace’s detection of initial attack chain activity.

A second Enhanced Monitoring model was also triggered by this device following the download of the aforementioned executable file (hxxp://62[.]60[.]226[.]112/file/3601_2042.exe) and the observed increase in C2 activity.

Following this activity, Darktrace continued to observe the device beaconing to the 21ene.ip-ddns[.]com endpoint.

Darktrace’s Cyber AI Analyst was able to correlate each of the individual detections involved in this compromise, identifying them as part of a broader incident that encompassed C2 connectivity, suspicious downloads, and external data transfers.

Cyber AI Analyst’s investigation into the activity observed on the affected device.
Figure 6: Cyber AI Analyst’s investigation into the activity observed on the affected device.
Figure 7: Cyber AI Analyst’s detection of the affected device’s broader connectivity throughout the course of the attack.

As the affected customer did not have Darktrace’s Autonomous Response configured at the time, the attack was able to progress unabated. Had Darktrace been properly enabled, it would have been able to take a number of actions to halt the escalation of the attack.

For example, the unusual beaconing connections and the download of an unexpected file from an uncommon location would have been shut down by blocking the device from making external connections to the relevant destinations.

Conclusion

The persistence of Blind Eagle and ability to adapt its tactics, even after patches were released, and the speed at which the group were able to continue using pre-established TTPs highlights that timely vulnerability management and patch application, while essential, is not a standalone defense.

Organizations must adopt security solutions that use anomaly-based detection to identify emerging and adapting threats by recognizing deviations in user or device behavior that may indicate malicious activity. Complementing this with an autonomous decision maker that can identify, connect, and contain compromise-like activity is crucial for safeguarding organizational networks against constantly evolving and sophisticated threat actors.

Credit to Charlotte Thompson (Senior Cyber Analyst), Eugene Chua (Principal Cyber Analyst) and Ryan Traill (Analyst Content Lead)

Appendices

IoCs

IoC – Type - Confidence
Microsoft-WebDAV-MiniRedir/10.0.19045 – User Agent

62[.]60[.]226[.]112 – IP – Medium Confidence

hxxp://62[.]60[.]226[.]112/file/3601_2042.exe – Payload Download – Medium Confidence

21ene.ip-ddns[.]com – Dynamic DNS Endpoint – Medium Confidence

diciembrenotasenclub[.]longmusic[.]com  - Hostname – Medium Confidence

Darktrace’s model alert coverage

Anomalous File / Suspicious HTTP Redirect
Anomalous File / EXE from Rare External Location
Anomalous File / Multiple EXE from Rare External Location
Anomalous Server Activity / Outgoing from Server
Unusual Activity / Unusual External Data to New Endpoint
Device / Anomalous Github Download
Anomalous Connection / Multiple Connections to New External TCP Port
Device / Initial Attack Chain Activity
Anomalous Server Activity / Rare External from Server
Compromise / Suspicious File and C2
Compromise / Fast Beaconing to DGA
Compromise / Large Number of Suspicious Failed Connections
Device / Large Number of Model Alert

Mitre Attack Mapping:

Tactic – Technique – Technique Name

Initial Access - T1189 – Drive-by Compromise
Initial Access - T1190 – Exploit Public-Facing Application
Initial Access ICS - T0862 – Supply Chain Compromise
Initial Access ICS - T0865 – Spearphishing Attachment
Initial Access ICS - T0817 - Drive-by Compromise
Resource Development - T1588.001 – Malware
Lateral Movement ICS - T0843 – Program Download
Command and Control - T1105 - Ingress Tool Transfer
Command and Control - T1095 – Non-Application Layer Protocol
Command and Control - T1571 – Non-Standard Port
Command and Control - T1568.002 – Domain Generation Algorithms
Command and Control ICS - T0869 – Standard Application Layer Protocol
Evasion ICS - T0849 – Masquerading
Exfiltration - T1041 – Exfiltration Over C2 Channel
Exfiltration - T1567.002 – Exfiltration to Cloud Storage

References

1)    https://research.checkpoint.com/2025/blind-eagle-and-justice-for-all/

2)    https://assets.kpmg.com/content/dam/kpmgsites/in/pdf/2025/04/kpmg-ctip-blind-eagle-01-apr-2025.pdf.coredownload.inline.pdf

3)    https://www.checkpoint.com/cyber-hub/threat-prevention/what-is-remote-access-trojan/#:~:text=They%20might%20be%20attached%20to,remote%20access%20or%20system%20administration

4)    https://msrc.microsoft.com/update-guide/vulnerability/CVE-2024-43451

5)    https://www.ionos.co.uk/digitalguide/server/know-how/webdav/

6)    https://vercara.digicert.com/resources/dynamic-dns-resolution-as-an-obfuscation-technique

7)    https://threatfox.abuse.ch/ioc/1437795

8)    https://www.checkpoint.com/cyber-hub/threat-prevention/what-is-malware/remcos-malware/

9)    https://www.virustotal.com/gui/url/b3189db6ddc578005cb6986f86e9680e7f71fe69f87f9498fa77ed7b1285e268

10) https://www.virustotal.com/gui/domain/21ene.ip-ddns.com

11) https://www.virustotal.com/gui/domain/diciembrenotasenclub.longmusic.com/community

Continue reading
About the author
Charlotte Thompson
Cyber Analyst

Blog

/

Email

/

June 18, 2025

Darktrace Collaborates with Microsoft: Unifying Email Security with a Shared Vision

Default blog imageDefault blog image

In today’s threat landscape, email remains the most targeted vector for cyberattacks. Organizations require not only multi-layered defenses but also advanced, integrated systems that work collaboratively to proactively mitigate threats before they cause damage

That’s why we’re proud to announce a new integration between Darktrace / EMAIL and Microsoft Defender for Office 365, delivering a Unified Quarantine experience that empowers security teams with seamless visibility, control, and response across both platforms.

This announcement builds on a strong and growing collaboration. In 2024, Darktrace was honored as Microsoft UK Partner of the Year and recognized as a Security Trailblazer at the annual Microsoft Security 20/20 Awards, a testament to our shared commitment to innovation and customer-centric security.

A Shared Mission: Stopping Threats at Machine Speed

This integration is more than a technical milestone,as it’s a reflection of a shared mission: to protect organizations from both known and unknown threats, with efficiency, accuracy, and transparency.

  • Microsoft Defender for Office 365 delivers a comprehensive security framework that safeguards Microsoft 365 email and collaboration workloads leveraging advanced AI, global threat intelligence and information on known attack infrastructure.
  • Darktrace / EMAIL complements this with Self-Learning AI that understands the unique communication patterns within each organization, detecting subtle anomalies that evade traditional detection methods.

Together, we’re delivering multi-layered, adaptive protection that’s greater than the sum of its parts.

“Our integration with Microsoft gives security teams the tools they need to act faster and more precisely to detect and respond to threats,” said Jill Popelka, CEO of Darktrace. “Together, we’re strengthening defenses where it matters most to our customers: at the inbox.”

Unified Quarantine: One View, Total Clarity

The new Unified Quarantine experience gives customers a single pane of glass to view and manage email threatsregardless of which product took action. This means:

  • Faster investigations with consolidated visibility
  • Clear attribution of actions and outcomes across both platforms
  • Streamlined workflows for security teams managing complex environments

“This integration is a testament to the power of combining Microsoft’s global threat intelligence with Darktrace’s unique ability to understand the ‘self’ of an organization,” said Jack Stockdale, CTO of Darktrace. “Together, we’re delivering a new standard in proactive, adaptive email security.”

A New Era of Collaborative Cyber Defense

This collaboration represents a broader shift in cybersecurity: from siloed tools to integrated ecosystems. As attackers become more sophisticated, defenders must move faster, smarter, and in unison.

Through this integration, Darktrace and Microsoft establish a new standard for collaboration between native and third-party security solutions, enhancing not only threat detection but also comprehensive understanding and proactive measures against threats.

We’re excited to bring this innovation to our customers and continue building a future where AI and human expertise collaborate to secure the enterprise.

[related-resource]

Continue reading
About the author
Carlos Gray
Senior Product Marketing Manager, Email
Your data. Our AI.
Elevate your network security with Darktrace AI