When AI Agents Go Off Script: What the OpenAI and Hugging Face Incident Means for Defenders
As AI agents become more autonomous, understanding behavior is becoming even more important. The OpenAI and Hugging Face incident highlights why behavioral security is foundational to securely deploying AI in the enterprise.
Darktrace cyber analysts are world-class experts in threat intelligence, threat hunting and incident response, and provide 24/7 SOC support to thousands of Darktrace customers around the globe. Inside the SOC is exclusively authored by these experts, providing analysis of cyber incidents and threat trends, based on real-world experience in the field.
Written by
Dr. Tim Bazalgette
Chief AI Officer, Darktrace
Share
22
Jul 2026
The recent security incident disclosed jointly by OpenAI and Hugging Face is quickly becoming one of the most discussed AI security stories of the year; it's one that has generated great discussion amongst our researchers and technical teams here at Darktrace.
During an internal evaluation of advanced OpenAI models, an autonomous agent reportedly escaped its intended testing boundaries, obtained internet access, and compromised Hugging Face infrastructure while attempting to evaluate and benchmark the model's cyber capabilities. OpenAI has described it as an unprecedented cyber incident and is collaborating with Hugging Face to investigate and share its findings publicly to help defenders understand the implications and improve AI safety across the community.
What this means for organizations deploying AI agents
The most important takeaway for security leaders is not that the AI model is capable of complex and effective chained attacks. It's what this tells us about the guardrails and protections that need to be put in place to ensure these increasingly capable models are safely used within our businesses.
According to OpenAI's investigation, the models were not acting with malicious intent. They were attempting to achieve a legitimate assigned objective. In pursuing that objective, they identified a path that involved obtaining information from a third party's environment and followed it to completion. From the model's perspective, that appears to have been an effective solution to the problem it was given.
Many organizations still hold the assumption that giving an AI agent a legitimate goal will naturally result in legitimate behavior. Increasingly, that is not something defenders can safely assume.
A human who encounters a "permission denied" message generally understands the technical restriction as well as the social meaning behind it. An agent may interpret that message only as evidence that one method to achieve its objective has failed. What makes this incident notable is the agent’s ability to keep pursuing the objective, revise its plan, and combine many individually useful actions into an outcome its designers did not intend.
This is not an isolated challenge. Recent research from the UK's AI Security Institute found that frontier models often attempt to "cheat" evaluations by finding unintended shortcuts to success. More interestingly, that behavior is not always reflected in chain-of-thought outputs and is not always admitted when models are questioned about their actions. One lesson for defenders is that understanding what an AI system says it is doing is not always the same as understanding its behavior and what it is actually doing.
Why behavioral security is an essential security foundation
That's the principle behind Darktrace / SECURE AI and our broader approach to behavioral security. Understanding intent or putting in place static guardrails is not enough; security teams need visibility into how agents behave over time, what resources they interact with, how their behavior changes, and whether they begin pursuing objectives in unexpected ways. Research released by NIST earlier this summer reinforced that rules alone are not sufficient to defend against AI threats, with one conclusion being that "there will always be a way to prompt an AI system to disregard its rules - it's just a matter of finding it."
Another interesting lesson is that nothing described in OpenAI's preliminary findings appears fundamentally impossible for a skilled human attacker. But it does highlight the accessibility and scale that could be unlocked for attackers as increasingly capable models become more widely available. Resource constraints that attackers deal with will decrease as frontier models can help execute elements of those attack workflows autonomously, persist through failure, and iterate towards an objective. Capabilities that were once limited to skilled attackers can increasingly be supplemented with these systems, lowering the barrier to entry to conduct complex cyber operations.
It is incredibly positive to see OpenAI and Hugging Face investigating this incident collaboratively and sharing their findings publicly. Transparency and collaboration around incidents like these helps the entire security community better understand how rapidly AI capabilities are evolving and where we need to improve safeguards. It also reinforces why Darktrace is excited to be partnering with OpenAI through the Daybreak Cyber Partner Program: to explore how advanced AI can be used safely and responsibly in support of defenders while developing the visibility, governance, and security controls needed for increasingly autonomous systems.
This incident is an important reminder that capable AI systems will not always behave in the ways we expect. As organizations deploy AI agents across critical workflows, understanding behavior is an even more important foundation for effective security.
[related-resource]
Secure Your AI Adoption
Practical guidance for securing LLM deployments, AI-integrated workflows, and agent-based systems, including controls to manage AI risk.
Darktrace cyber analysts are world-class experts in threat intelligence, threat hunting and incident response, and provide 24/7 SOC support to thousands of Darktrace customers around the globe. Inside the SOC is exclusively authored by these experts, providing analysis of cyber incidents and threat trends, based on real-world experience in the field.
Darktrace Advances Incident Investigation and AI-Agent Security with OpenAI Daybreak Models
Darktrace is teaming up with OpenAI's Daybreak Defense Network to help security teams move from alert to action faster, combining Darktrace's behavioral threat detection with OpenAI's contextual reasoning to clarify business impact and strengthen visibility into AI-agent risk.
AI Agents: Securing the Path from Intent to Action
Recent guidance from the NCSC sets out the controls needed to deploy agentic AI safely. But approval is only the starting point – the next step is understanding how agents actually behave once they are operating in the enterprise.
When AI Becomes the Lure: A Fake Gemini Installer Delivers Vidar
With the growing prevalence of generative AI, attackers are increasingly exploiting interest in AI tools to distribute malware. Darktrace recently investigated an incident in which a malicious executable masquerading as a Google Gemini installer resulted in a Vidar information stealer infection.
Darktrace Advances Incident Investigation and AI-Agent Security with OpenAI Daybreak Models
Earlier this year, Darktrace joined OpenAI’s Daybreak Defense Network to explore how their cyber capabilities can be integrated within Darktrace products and services to transform how security teams move from signal to action.
At the heart of this work is Darktrace's behavioral understanding of customer environments and identification of complex security incidents, combined with OpenAI models that can add context to help explain why an incident matters and its potential impact on the business. By bringing these capabilities into defensive workflows security teams already use, the goal is to give defenders not just greater visibility, but the context and guidance they need to act with confidence.
Since joining the program, we've been working with OpenAI to explore how these capabilities can address specific security challenges for defenders.
The problem we're solving
Attackers continue to change how they operate, including by using AI to increase the speed and scale of some techniques. Security teams are already managing a large volume of alerts, and the question isn't just what's happening, but how it could affect the organization. Even when an incident is fully investigated and correlated, technical severity alone doesn't tell a security team how much it actually matters to the business. That same challenge extends to internal AI adoption. As organizations adopt more AI systems and agents, security teams need visibility into their behavior, access and activity, along with the broader business context needed to identify and investigate potential risk.
Darktrace's Adaptive AI™ builds a detailed, organization-specific picture of what's normal for each environment, and uses that picture to investigate threats and identify complex security activity across domains. OpenAI's models can build on Darktrace's correlated, technically prioritized incidents by adding context that can help defenders understand what may be at stake.
What we're building
Our work is focused on two areas: supporting security investigation and response, and helping defenders identify risky behavior across enterprise AI systems and agents.
The first aligns Darktrace's behavioral understanding with OpenAI models to support security investigation and prioritization. Darktrace's Adaptive AI continuously learns the unique patterns of normal behavior within each customer it protects, creating a deep, organization-specific understanding of its digital estate. When unusual activity emerges, OpenAI's models can draw on that context to help analysts investigate the incident, understand its significance and assess potential business consequences — reducing the need to manually assemble context from fragmented signals.
Second, we are exploring how these capabilities can support AI-agent and runtime security through Darktrace / SECURE AI™. OpenAI’s Daybreak models can build on the detections and visibility Darktrace / SECURE AI provides, connecting signals across a customer's environment and help defenders identify potentially risky behavior involving AI systems and agents. Activity that might appear isolated can instead be connected with related signals, helping defenders investigate the broader context and determine appropriate remediation.
Darktrace brings deep cybersecurity expertise, an evolving understanding of each customer's environment, and AI-driven identification of threats across the digital estate. Through the Daybreak Defense Network, Darktrace is exploring how OpenAI models can augment those capabilities in defensive security workflows — supporting incident investigation and response and improving visibility into AI-agent and runtime risk.
These capabilities are still in development, and we're excited to continue building on this work.
To learn more about how Darktrace continues to innovate to meet today's most pressing security challenges, register for our upcoming launch broadcast here.
Botnet Behind the Camera: Mirai Katana Activity on a Video Recording Device
Key takeaways
Darktrace identified a camera device infected with the Mirai/Katana botnet in a sports-sector customer environment, showing how exposed IoT devices can become active participants in wider attack chains.
The compromise involved suspicious Wget behavior, file downloads from rare external IPs, unusual incoming HTTP connections to video recorder management interfaces, and large outbound data transfers to infrastructure associated with botnet activity.
The incident highlights the importance of extending visibility and response beyond traditional endpoints, as unmanaged or overlooked connected devices can be exploited for command-and-control, malware delivery, and data exfiltration.
Mirai and the Katana variant
Mirai is a botnet that first emerged in August 2016 and is well known for launching large-scale distributed-denial-of-service (DDoS) attacks, typically targeting exposed Internet of Things (IoT) devices. It identifies vulnerable IoT devices ,often by abusing default credentials or exposed services, and recruiting them into a remotely controlled botnet that can be used in DDoS campaigns [1].
Katana, one of the many variants that arose after Mirai’s source code was released publicly, was first observed in late 2020 and has been seen using more advanced capabilities, including custom command-and-control (C2), persistence mechanisms, and DDoS functionality [2].
In March 2026, research from the Nokia Deepfield Emergency Response Team (ERT) identified Katana as a Mirai-derived DDoS botnet targeting Android-based TV set-top boxes through exposed Android Debug Bridge (ADB) access. Observed capabilities included custom C2, runtime domain rotation, multiple DDoS methods, and an on-device compiled kernel rootkit used for persistence and stealth [3].
Darktrace’s detection of Mirai Botnet activity on a camera device
In early 2026, Darktrace identified a Network/Digital Video Recorder (NVR/DVR) on the network of a sports-sector customer that had been infected with the Mirai Katana botnet and subsequently used to exfiltrate data from the customer’s environment. Seemingly related follow-up activity was observed on the same device several months later.
In both instances, the Darktrace Security Operations Centre (SOC) alerted the customer as part of the Managed Threat Detection (MTD) service. However, as Darktrace’s Autonomous Response capability was not fully enabled on the affected device, Darktrace was unable to proactively block the suspicious activity or prevent the compromise from continuing and recurring.
The initial compromise appears to have occurred when the affected device was seen using Wget to download Linux-based Executable and Linkable Format (ELF) files from a rare external IP, 195.177.94[.]105, which had not previously been observed in the customer’s network. Further analysis downloaded file hashes identified files related to the Mirai botnet.
Figure 1: Darktrace’s Real-Time AI Analyst investigation into the unusual outbound connection where the ELF files were downloaded.
Within a few hours, Darktrace detected the device uploading close to 3GB of data to another external IP, 50.7.49[.]4:3017 (ASN AS30058 FDCSERVERS), suggesting that the activity was likely routed via a virtual private server (VPS) hosted by FDC Servers [2]. Attackers often abuse VPS infrastructure from legitimate cloud providers to blend in with legitimate traffic and evade IP reputation and geolocation-based detections.
Figure 2: Darktrace’s detection of the unusual data upload activity by the affected camera device.
Darktrace continued to observe similar data transfers to multiple rare endpoints including 171.225.223[.]53, 95.161.128[.]62, 61.7.209[.]88, 95.161.128[.]62, which have been linked to Mirai by open-source intelligence (OSINT).
Figure 3: Darktrace’s detection of spikes in unusual external data transfer activity from the camera device.
Exploitation continued
Several months later, Darktrace identified the same exfiltration pattern on the device again, this time with stronger indications of associations with Mirai Katana botnet infection.
The device received incoming HTTP connections from 129.121.114[.]124, an external IP known to be associated with the Katana botnet IP [3]. The connections targeted the ‘/dvr/cmd’ path using the root username and user agent Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.135 Safari/537.36 Edge/12.246.
The ‘/dvr/cmd’ path appears to be associated with the affected device’s web management functionality. This API endpoint has historically been targeted by Mirai and other IoT botnets through the exploitation of critical command injection vulnerabilities and automated botnet exploitation [4].
Figure 4: Darktrace’s detection of HTTP connectivity from the external IP associated with Mirai Katana Botnet.
A few days later, Darktrace observed the Wget utility being used to download ELF files, including “/lil”, from the IP 129.121.114[.]124. OSINT reporting has since associated this IP address with the Mirai Katana botnet. Notably, the IP observed earlier in the year, 195.177.94[.]105, had also hosted a file named “lil”, indicating a link between the observed activity.
Over the following days, the device received a sudden spike in connections from multiple rare external endpoints, suggesting a possible successful brute force attack. Darktrace also observed the device exfiltrating just under 4GB of data to another Mirai-associated IP address, 66.92.198[.]194, over ports 3344, 954922, and 80. Finally, the device was seen uploading data to the Mirai botnet IP 5.175.249[.]53 over port138 and exhibited an increase in UDP connections to 34.18.28[.]10 over port 9068.
Following both file download events, Darktrace identified spikes in external data transfers and connection attempts to rare destinations. While Darktrace’s Threat Research team could not confirm with high confidence that this to activity was directly associated with Mirai, it may indicate that Mirai Katana includes data exfiltration functionality.
Darktrace’s threat researchers also identified an internet-facing NTP server belonging to a separate customer receiving incoming connection attempts from the same initially observed IP, 195.177.94[.]105,over the port 123. This suggests that Mirai Katana may not exclusively target IoT devices.
Conclusion
This case demonstrates how threat actors can exploit overlooked IoT and OT devices to support broader malicious objectives. Here, a camera device infected with a botnet was used to exfiltrate data from the customer's environment, showing how peripheral assets can become active participants in an attack chain.
This case also reinforces a challenge many organizations face today: extending security visibility beyond traditional endpoints and servers. Cameras, sensors, and other connected devices often operate with limited monitoring and may fall outside established security processes, despite maintaining network connectivity and access to potentially sensitive environments. This is particularly relevant in the sports sector, where growing reliance on connected cameras, smart stadium technologies, and other IoT devices continues to expand the attack surface, as highlighted in Darktrace's Sports Sector Threat Report.
As botnets like Kata and Mirai continue to evolve, defenders need visibility across unmanaged IoT and edge devices, as well as security solutions that can recognize subtle deviations in device behavior that may indicate an emerging compromise.
Credit to Parvatha Ananthakannan (Cyber Analyst), Signe Zaharka (Principal Analyst)
Edited by Ryan Traill (Content Manager)
Appendices
Darktrace Model Detections
· Anomalous File / EXE from Rare External Location
· Anomalous File / Multiple EXE from Rare External Locations
· Device / Initial Attack Chain Activity
· Unusual Activity / Unusual External Data to New Endpoint
· Anomalous Connection / Data Sent to Rare Domain
· Unusual Activity / Enhanced Unusual External Data Transfer
· Anomalous Connection / Uncommon 1 GiB Outbound
· Device / Significant UDP Increase
· Anomalous Connection / Low and Slow Exfiltration to IP
· Compromise / Large Number of Suspicious Failed Connections
· Compromise / Large Number of Suspicious Successful Connections
