Exploring AI Threats: Package Hallucination Attacks
Learn how malicious actors exploit errors in generative AI tools to launch packet attacks. Read how Darktrace products detect and prevent these threats!
Darktrace cyber analysts are world-class experts in threat intelligence, threat hunting and incident response, and provide 24/7 SOC support to thousands of Darktrace customers around the globe. Inside the SOC is exclusively authored by these experts, providing analysis of cyber incidents and threat trends, based on real-world experience in the field.
Written by
Charlotte Thompson
Cyber Analyst
Written by
Tiana Kelly
Deputy Team Lead, London & Cyber Analyst
Share
30
Oct 2023
AI tools open doors for threat actors
On November 30, 2022, the free conversational language generation model ChatGPT was launched by OpenAI, an artificial intelligence (AI) research and development company. The launch of ChatGPT was the culmination of development ongoing since 2018 and represented the latest innovation in the ongoing generative AI boom and made the use of generative AI tools accessible to the general population for the first time.
ChatGPT is estimated to currently have at least 100 million users, and in August 2023 the site reached 1.43 billion visits [1]. Darktrace data indicated that, as of March 2023, 74% of active customer environments have employees using generative AI tools in the workplace [2].
However, with new tools come new opportunities for threat actors to exploit and use them maliciously, expanding their arsenal.
Much consideration has been given to mitigating the impacts of the increased linguistic complexity in social engineering and phishing attacks resulting from generative AI tool use, with Darktrace observing a 135% increase in ‘novel social engineering attacks’ across thousands of active Darktrace/Email™ customers from January to February 2023, corresponding with the widespread adoption of ChatGPT and its peers [3].
Less overall consideration, however, has been given to impacts stemming from errors intrinsic to generative AI tools. One of these errors is AI hallucinations.
What is an AI hallucination?
AI “hallucination” is a term which refers to the predictive elements of generative AI and LLMs’ AI model gives an unexpected or factually incorrect response which does not align with its machine learning training data [4]. This differs from regular and intended behavior for an AI model, which should provide a response based on the data it was trained upon.
Why are AI hallucinations a problem?
Despite the term indicating it might be a rare phenomenon, hallucinations are far more likely than accurate or factual results as the AI models used in LLMs are merely predictive and focus on the most probable text or outcome, rather than factual accuracy.
Given the widespread use of generative AI tools in the workplace employees are becoming significantly more likely to encounter an AI hallucination. Furthermore, if these fabricated hallucination responses are taken at face value, they could cause significant issues for an organization.
Use of generative AI in software development
Software developers may use generative AI for recommendations on how to optimize their scripts or code, or to find packages to import into their code for various uses. Software developers may ask LLMs for recommendations on specific pieces of code or how to solve a specific problem, which will likely lead to a third-party package. It is possible that packages recommended by generative AI tools could represent AI hallucinations and the packages may not have been published, or, more accurately, the packages may not have been published prior to the date at which the training data for the model halts. If these hallucinations result in common suggestions of a non-existent package, and the developer copies the code snippet wholesale, this may leave the exchanges vulnerable to attack.
Research conducted by Vulcan revealed the prevalence of AI hallucinations when ChatGPT is asked questions related to coding. After sourcing a sample of commonly asked coding questions from Stack Overflow, a question-and-answer website for programmers, researchers queried ChatGPT (in the context of Node.js and Python) and reviewed its responses. In 20% of the responses provided by ChatGPT pertaining to Node.js at least one un-published package was included, whilst the figure sat at around 35% for Python [4].
Hallucinations can be unpredictable, but would-be attackers are able to find packages to create by asking generative AI tools generic questions and checking whether the suggested packages exist already. As such, attacks using this vector are unlikely to target specific organizations, instead posing more of a widespread threat to users of generative AI tools.
Malicious packages as attack vectors
Although AI hallucinations can be unpredictable, and responses given by generative AI tools may not always be consistent, malicious actors are able to discover AI hallucinations by adopting the approach used by Vulcan. This allows hallucinated packages to be used as attack vectors. Once a malicious actor has discovered a hallucination of an un-published package, they are able to create a package with the same name and include a malicious payload, before publishing it. This is known as a malicious package.
Malicious packages could also be recommended by generative AI tools in the form of pre-existing packages. A user may be recommended a package that had previously been confirmed to contain malicious content, or a package that is no longer maintained and, therefore, is more vulnerable to hijack by malicious actors.
In such scenarios it is not necessary to manipulate the training data (data poisoning) to achieve the desired outcome for the malicious actor, thus a complex and time-consuming attack phase can easily be bypassed.
An unsuspecting software developer may incorporate a malicious package into their code, rendering it harmful. Deployment of this code could then result in compromise and escalation into a full-blown cyber-attack.
Figure 1: Flow diagram depicting the initial stages of an AI Package Hallucination Attack.
For providers of Software-as-a-Service (SaaS) products, this attack vector may represent an even greater risk. Such organizations may have a higher proportion of employed software developers than other organizations of comparable size. A threat actor, therefore, could utilize this attack vector as part of a supply chain attack, whereby a malicious payload becomes incorporated into trusted software and is then distributed to multiple customers. This type of attack could have severe consequences including data loss, the downtime of critical systems, and reputational damage.
How could Darktrace detect an AI Package Hallucination Attack?
In June 2023, Darktrace introduced a range of DETECT™ and RESPOND™ models designed to identify the use of generative AI tools within customer environments, and to autonomously perform inhibitive actions in response to such detections. These models will trigger based on connections to endpoints associated with generative AI tools, as such, Darktrace’s detection of an AI Package Hallucination Attack would likely begin with the breaching of one of the following DETECT models:
Compliance / Anomalous Upload to Generative AI
Compliance / Beaconing to Rare Generative AI and Generative AI
Compliance / Generative AI
Should generative AI tool use not be permitted by an organization, the Darktrace RESPOND model ‘Antigena / Network / Compliance / Antigena Generative AI Block’ can be activated to autonomously block connections to endpoints associated with generative AI, thus preventing an AI Package Hallucination attack before it can take hold.
Once a malicious package has been recommended, it may be downloaded from GitHub, a platform and cloud-based service used to store and manage code. Darktrace DETECT is able to identify when a device has performed a download from an open-source repository such as GitHub using the following models:
Device / Anomalous GitHub Download
Device / Anomalous Script Download Followed By Additional Packages
Whatever goal the malicious package has been designed to fulfil will determine the next stages of the attack. Due to their highly flexible nature, AI package hallucinations could be used as an attack vector to deliver a large variety of different malware types.
As GitHub is a commonly used service by software developers and IT professionals alike, traditional security tools may not alert customer security teams to such GitHub downloads, meaning malicious downloads may go undetected. Darktrace’s anomaly-based approach to threat detection, however, enables it to recognize subtle deviations in a device’s pre-established pattern of life which may be indicative of an emerging attack.
Subsequent anomalous activity representing the possible progression of the kill chain as part of an AI Package Hallucination Attack could then trigger an Enhanced Monitoring model. Enhanced Monitoring models are high-fidelity indicators of potential malicious activity that are investigated by the Darktrace analyst team as part of the Proactive Threat Notification (PTN) service offered by the Darktrace Security Operation Center (SOC).
Conclusion
Employees are often considered the first line of defense in cyber security; this is particularly true in the face of an AI Package Hallucination Attack.
As the use of generative AI becomes more accessible and an increasingly prevalent tool in an attacker’s toolbox, organizations will benefit from implementing company-wide policies to define expectations surrounding the use of such tools. It is simple, yet critical, for example, for employees to fact check responses provided to them by generative AI tools. All packages recommended by generative AI should also be checked by reviewing non-generated data from either external third-party or internal sources. It is also good practice to adopt caution when downloading packages with very few downloads as it could indicate the package is untrustworthy or malicious.
As of September 2023, ChatGPT Plus and Enterprise users were able to use the tool to browse the internet, expanding the data ChatGPT can access beyond the previous training data cut-off of September 2021 [5]. This feature will be expanded to all users soon [6]. ChatGPT providing up-to-date responses could prompt the evolution of this attack vector, allowing attackers to publish malicious packages which could subsequently be recommended by ChatGPT.
It is inevitable that a greater embrace of AI tools in the workplace will be seen in the coming years as the AI technology advances and existing tools become less novel and more familiar. By fighting fire with fire, using AI technology to identify AI usage, Darktrace is uniquely placed to detect and take preventative action against malicious actors capitalizing on the AI boom.
Credit to Charlotte Thompson, Cyber Analyst, Tiana Kelly, Analyst Team Lead, London, Cyber Analyst
Darktrace cyber analysts are world-class experts in threat intelligence, threat hunting and incident response, and provide 24/7 SOC support to thousands of Darktrace customers around the globe. Inside the SOC is exclusively authored by these experts, providing analysis of cyber incidents and threat trends, based on real-world experience in the field.
Threat actors frequently exploit ongoing world events to trick users into opening and executing malicious files. Darktrace security researchers recently identified a threat group using reports around the arrest of Venezuelan President Nicolàs Maduro on January 3, 2025, as a lure to deliver backdoor malware.
Technical Analysis
While the exact initial access method is unknown, it is likely that a spear-phishing email was sent to victims, containing a zip archive titled “US now deciding what’s next for Venezuela.zip”. This file included an executable named “Maduro to be taken to New York.exe” and a dynamic-link library (DLL), “kugou.dll”.
The binary “Maduro to be taken to New York.exe” is a legitimate binary (albeit with an expired signature) related to KuGou, a Chinese streaming platform. Its function is to load the DLL “kugou.dll” via DLL search order. In this instance, the expected DLL has been replaced with a malicious one with the same name to load it.
Figure 1: DLL called with LoadLibraryW.
Once the DLL is executed, a directory is created C:\ProgramData\Technology360NB with the DLL copied into the directory along with the executable, renamed as “DataTechnology.exe”. A registry key is created for persistence in “HKCU\Software\Microsoft\Windows\CurrentVersion\Run\Lite360” to run DataTechnology.exe --DATA on log on.
Figure 2. Registry key added for persistence.
Figure 3: Folder “Technology360NB” created.
During execution, a dialog box appears with the caption “Please restart your computer and try again, or contact the original author.”
Figure 4. Message box prompting user to restart.
Prompting the user to restart triggers the malware to run from the registry key with the command --DATA, and if the user doesn't, a forced restart is triggered. Once the system is reset, the malware begins periodic TLS connections to the command-and-control (C2) server 172.81.60[.]97 on port 443. While the encrypted traffic prevents direct inspection of commands or data, the regular beaconing and response traffic strongly imply that the malware has the ability to poll a remote server for instructions, configuration, or tasking.
Conclusion
Threat groups have long used geopolitical issues and other high-profile events to make malicious content appear more credible or urgent. Since the onset of the war in Ukraine, organizations have been repeatedly targeted with spear-phishing emails using subject lines related to the ongoing conflict, including references to prisoners of war [1]. Similarly, the Chinese threat group Mustang Panda frequently uses this tactic to deploy backdoors, using lures related to the Ukrainian war, conventions on Tibet [2], the South China Sea [3], and Taiwan [4].
The activity described in this blog shares similarities with previous Mustang Panda campaigns, including the use of a current-events archive, a directory created in ProgramData with a legitimate executable used to load a malicious DLL and run registry keys used for persistence. While there is an overlap of tactics, techniques and procedures (TTPs), there is insufficient information available to confidently attribute this activity to a specific threat group. Users should remain vigilant, especially when opening email attachments.
Credit to Tara Gould (Malware Research Lead) Edited by Ryan Traill (Analyst Content Lead)
Indicators of Compromise (IoCs)
172.81.60[.]97 8f81ce8ca6cdbc7d7eb10f4da5f470c6 - US now deciding what's next for Venezuela.zip 722bcd4b14aac3395f8a073050b9a578 - Maduro to be taken to New York.exe aea6f6edbbbb0ab0f22568dcb503d731 - kugou.dll
Under Medusa’s Gaze: How Darktrace Uncovers RMM Abuse in Ransomware Campaigns
What is Medusa Ransomware in 2025?
In 2025, the Medusa Ransomware-as-a-Service (RaaS) emerged as one of the top 10 most active ransomware threat actors [1]. Its growing impact prompted a joint advisory from the US Cybersecurity and Infrastructure Security Agency (CISA) and the Federal Bureau of Investigation (FBI) [3]. As of January 2026, more than 500 organizations have fallen victim to Medusa ransomware [2].
Darktrace previously investigated Medusa in a 2024 blog, but the group’s rapid expansion and new intelligence released in late 2025 has lead Darktrace’s Threat Research team to investigate further. Recent findings include Microsoft’s research on Medusa actors exploiting a vulnerability in Fortra’s GoAnywhere MFT License Servlet (CVE-2025-10035)[4] and Zencec’s report on Medusa’s abuse of flaws in SimpleHelp’s remote support software (CVE-2024-57726, CVE-2024-57727, CVE-2024-57728) [5].
Reports vary on when Medusa first appeared in the wild. Some sources mention June 2021 as the earliest sightings, while others point to late 2022, when its developers transitioned to the RaaS model, as the true beginning of its operation [3][11].
Madusa Ransomware history and background
The group behind Medusa is known by several aliases, including Storm-1175 and Spearwing [4] [7]. Like its mythological namesake, Medusa has many “heads,” collaborating with initial access brokers (IABs) and, according to some evidence, affiliating with Big Game Hunting (BGH) groups such as Frozen Spider, as well as the cybercriminal group UNC7885 [3][6][13].
Use of Cyrillic in its scripts, activity on Russian-language cybercrime forums, slang unique to Russian criminal subcultures, and avoidance of targets in Commonwealth of Independent States (CIS) countries suggest that Medusa operates from Russia or an allied state [11][12].
Medusa ransomware should not be confused with other similarly named malware, such as the Medusa Android Banking Trojan, the Medusa Botnet/Medusa Stealer, or MedusaLocker ransomware. It is easily distinguishable from these variants because it appends the extension .MEDUSA to encrypted files and drops the ransom note !!!READ_ME_MEDUSA!!!.txt on compromised systems [8].
Who does Madusa Ransomware target?
The group appears to show little restraint, indiscriminately attacking organizations across all sectors, including healthcare, and is known to employ triple extortion tactics whereby sensitive data is encrypted, victims are threatened with data leaks, and additional pressure is applied through DDoS attacks or contacting the victim’s customers, rather than the more common double extortion model [13].
Madusa Ransomware TTPs
To attain initial access, Medusa actors typically purchase access to already compromised devices or accounts via IABs that employ phishing, credential stuffing, or brute-force attacks, and also target vulnerable or misconfigured Internet-facing systems.
Between December 2023 and November 2025, Darktrace observed multiple cases of file encryption related to Medusa ransomware across its customer base. When enabled, Darktrace’s Autonomous Response capability intervened early in the attack chain, blocking malicious activity before file encryption could begin.
Some of the affected were based in Europe, the Middle East and Africa (EMEA), others in the Americas (AMS), and the remainder in the Asia-Pacific and Japan region. The most impacted sectors were financial services and the automotive industry, followed by healthcare, and finally organizations in arts, entertainment and recreation, ICT, and manufacturing.
Remote Monitoring and Management (RMM) tool abuse
In most customer environments where Medusa file encryption attempts were observed, and in one case where the compromise was contained before encryption, unusual external HTTP connections associated with JWrapper were also detected. JWrapper is a legitimate tool designed to simplify the packaging, distribution, and management of Java applications, enabling the creation of executables that run across different operating systems. Many of the destination IP addresses involved in this activity were linked to SimpleHelp servers or associated with Atera.
Medusa actors appear to favor RMM tools such as SimpleHelp. Unpatched or misconfigured SimpleHelp RMM servers can serve as an initial access vector to the victims’ infrastructure. After gaining access to SimpleHelp management servers, the threat actors edit server configuration files to redirect existing SimpleHelp RMM agents to communicate with unauthorized servers under their control.
The SimpleHelp tool is not only used for command-and-control (C2) and enabling persistence but is also observed during lateral movement within the network, downloading additional attack tools, data exfiltration, and even ransomware binary execution. Other legitimate remote access tools abused by Medusa in a similar manner to evade detection include Atera, AnyDesk, ScreenConnect, eHorus, N-able, PDQ Deploy/Inventory, Splashtop, TeamViewer, NinjaOne, Navicat, and MeshAgent [4][5][15][16][17].
Data exfiltration
Another correlation among Darktrace customers affected by Medusa was observed during the data exfiltration phase. In several environments, data was exfiltrated to the endpoints erp.ranasons[.]com or pruebas.pintacuario[.]mx (143.110.243[.]154, 144.217.181[.]205) over ports 443, 445, and 80. erp.ranasons[.]com was seemingly active between November 2024 and September 2025, while pruebas.pintacuario[.]mx was seen from November 2024 to March 2025. Evidence suggests that pruebas.pintacuario[.]mx previously hosted a SimpleHelp server [22][23].
Apart from RMM tools, Medusa is also known to use Rclone and Robocopy for data exfiltration [3][19]. During one Medusa compromise detected in mid-2024, the customer’s data was exfiltrated to external destinations associated with the Ngrok proxy service using an SSH-2.0-rclone client.
Medusa Compromise Leveraging SimpleHelp
In Q4 2025, Darktrace assisted a European company impacted by Medusa ransomware. The organization had partial Darktrace / NETWORK coverage and had configured Darktrace’s Autonomous Response capability to require manual confirmation for all actions. Despite these constraints, data received through the customer’s security integration with CrowdStrike Falcon enabled Darktrace analysts to reconstruct the attack chain, although the initial access vector remains unclear due to limited visibility.
In late September 2025, a device out of the scope of Darktrace's visibility began scanning the network and using RDP, NTLM/SMB, DCE_RPC, and PowerShell for lateral movement.
CrowdStrike “Defense Evasion: Disable or Modify Tools” alerts related to a suspicious driver (c:\windows\[0-9a-b]{4}.exe) and a PDQ Deploy executable (share=\\<device_hostname>\ADMIN$ file=AdminArsenal\PDQDeployRunner\service-1\exec\[0-9a-b]{4}.exe) suggest that the attackers used the Bring Your Own Vulnerable Driver (BYOVD) technique to terminate antivirus processes on network devices, leveraging tools such as KillAV or AbyssWorker along with the PDQ Software Deployment solution [19][26].
A few hours later, Darktrace observed the same device that had scanned the network writing Temp\[a-z]{2}.exe over SMB to another device on the same subnet. According to data from the CrowdStrike alert, this executable was linked to an RMM application located at C:\Users\<compromised_user>\Documents\[a-z]{2}.exe. The same compromised user account later triggered a CrowdStrike “Command and Control: Remote Access Tools” alert when accessing C:\ProgramData\JWrapper-Remote Access\JWrapper-Remote Access Bundle-[0-9]{11}\JWrapperTemp-[0-9]{10}-[0-9]{1}-app\bin\windowslauncher.exe [27].
Figure 1: An executable file associated with the SimpleHelp RMM tool being written to other devices using the SMB protocol, as detected by Darktrace.
Soon after, the destination device and multiple other network devices began establishing connections to 31.220.45[.]120 and 213.183.63[.]41, both of which hosted malicious SimpleHelp RMM servers. These C2 connections continued for more than 20 days after the initial compromise.
CrowdStrike integration alerts for the execution of robocopy . "c:\windows\\" /COPY:DT /E /XX /R:0 /W:0 /NP /XF RunFileCopy.cmd /IS /IT commands on several Windows servers, suggested that this utility was likely used to stage files in preparation for data exfiltration [19].
Around two hours later, Darktrace detected another device connecting to the attacker’s SimpleHelp RMM servers. This internal server had ‘doc’ in its hostname, indicating it was likely a file server. It was observed downloading documents from another internal server over SMB and uploading approximately 70 GiB of data to erp.ranasons[.]com (143.110.243[.]154:443).
Figure 2: Data uploaded to erp.ranasons[.]com and the number of model alerts from the exfiltrating device, represented by yellow and orange dots.
Darktrace’s Cyber AI Analyst autonomously investigated the unusual connectivity, correlating the separate C2 and data exfiltration events into a single incident, providing greater visibility into the ongoing attack.
Figure 3: Cyber AI Analyst identified a file server making C2 connections to an attacker-controlled SimpleHelp server (213.183.63[.]41) and exfiltrating data to erp.ranasons[.]com.
Figure 4: The same file server that connected to 213.183.63[.]41 and exfiltrated data to erp.ranasons[.]com was also observed attempting to connect to an IP address associated with Moscow, Russia (193.37.69[.]154:7070).
One of the devices connecting to the attacker's SimpleHelp RMM servers was also observed downloading 35 MiB from [0-9]{4}.filemail[.]com. Filemail, a legitimate file-sharing service, has reportedly been abused by Medusa actors to deliver additional malicious payloads [11].
Figure 5: A device controlled remotely via SimpleHelp downloading additional tooling from the Filemail file-sharing service.
Finally, integration alerts related to the ransomware binary, such as c:\windows\system32\gaze.exe and <device_hostname>\ADMIN$ file=AdminArsenal\PDQDeployRunner\service-1\exec\gaze.exe, along with “!!!READ_ME_MEDUSA!!!.txt” ransom notes were observed on network devices. This indicates that file encryption in this case was most likely carried out directly on the victim hosts rather than via the SMB protocol [3].
Conclusion
Threat actors, including nation-state actors and ransomware groups like Medusa, have long abused legitimate commercial RMM tools, typically used by system administrators for remote monitoring, software deployment, and device configuration, instead of relying on remote access trojans (RATs).
Attackers employ existing authorized RMM tools or install new remote administration software to enable persistence, lateral movement, data exfiltration, and ingress tool transfer. By mimicking legitimate administrative behavior, RMM abuse enables attackers to evade detection, as security software often implicitly trusts these tools, allowing attackers to bypass traditional security controls [28][29][30].
To mitigate such risks, organizations should promptly patch publicly exposed RMM servers and adopt anomaly-based detection solutions, like Darktrace / NETWORK, which can distinguish legitimate administrative activity from malicious behavior, applying rapid response measures through its Autonomous Response capability to stop attacks in their tracks.
Darktrace delivers comprehensive network visibility and Autonomous Response capabilities, enabling real-time detection of anomalous activity and rapid mitigation, even if an organization fall under Medusa’s gaze.
Credit to Signe Zaharka (Principal Cyber Analyst) and Emma Foulger (Global Threat Research Operations Lead
Edited by Ryan Traill (Analyst Content Lead)
Appendices
List of Indicators of Compromise (IoCs)
IoC - Type - Description + Confidence + Time Observed
185.108.129[.]62 IP address Malicious SimpleHelp server observed during Medusa attacks (High confidence) - March 7, 2023
185.126.238[.]119 IP address Malicious SimpleHelp server observed during Medusa attacks (High confidence) - November 26-27, 2024
213.183.63[.]41 IP address Malicious SimpleHelp server observed during Medusa attacks (High confidence) - November 28, 2024 - Sep 30, 2025
213.183.63[.]42 IP address Malicious SimpleHelp server observed during Medusa attacks (High confidence) - July 4 -9 , 2024
31.220.45[.]120 IP address Malicious SimpleHelp server observed during Medusa attacks (High confidence) - September 12 - Oct 20 , 2025
91.92.246[.]110 IP address Malicious SimpleHelp server observed during Medusa attacks (High confidence) - May 24, 2024
45.9.149[.]112:15330 IP address Malicious SimpleHelp server observed during Medusa attacks (High confidence) - June 21, 2024
89.36.161[.]12 IP address Malicious SimpleHelp server observed during Medusa attacks (High confidence) - June 26-28, 2024
193.37.69[.]154:7070 IP address Suspicious RU IP seen on a device being controlled via SimpleHelp and exfiltrating data to a Medusa related endpoint - September 30 - October 20, 2025
erp.ranasons[.]com·143.110.243[.]154 Hostname Data exfiltration destination - November 27, 2024 - September 30, 2025
pruebas.pintacuario[.]mx·144.217.181[.]205 - Hostname Data exfiltration destination - November 27, 2024 - March 26, 2025
lirdel[.]com · 44.235.83[.]125/a.msi (1b9869a2e862f1e6a59f5d88398463d3962abe51e19a59) File & hash Atera related file downloaded with PowerShell - June 20, 2024
wizarr.manate[.]ch/108.215.180[.]161:8585/$/1dIL5 File Suspicious file observed on one of the devices exhibiting unusual activity during a Medusa compromise - February 28, 2024
!!!READ_ME_MEDUSA!!!.txt" File - Ransom note
*.MEDUSA - File extension File extension added to encrypted files
gaze.exe – File - Ransomware binary
Darktrace Model Coverage
Darktrace / NETWORK model detections triggered during connections to attacker controlled SimpleHelp servers:
Anomalous Connection/Anomalous SSL without SNI to New External
Anomalous Connection/Multiple Connections to New External UDP Port
Anomalous Connection/New User Agent to IP Without Hostname