Blog
/
Identity
/
April 29, 2025

MFA Under Attack: AiTM Phishing Kits Abusing Legitimate Services

Tycoon 2FA uses AiTM phishing and legitimate services to bypass MFA. Darktrace AI stopped it, read the blog to learn how Self-Learning AI detects sophisticated threats.
Inside the SOC
Darktrace cyber analysts are world-class experts in threat intelligence, threat hunting and incident response, and provide 24/7 SOC support to thousands of Darktrace customers around the globe. Inside the SOC is exclusively authored by these experts, providing analysis of cyber incidents and threat trends, based on real-world experience in the field.
Written by
Alexandra Sentenac
Cyber Analyst
fingerprintDefault blog image
29
Apr 2025

In late 2024 and early 2025, the Darktrace Security Operations Center (SOC) investigated alerts regarding separate cases of Software-as-a-Service (SaaS) account compromises on two customer environments that presented several similarities, suggesting they were part of a wider phishing campaign.

This campaign was found to leverage the project collaboration and note-taking application, Milanote, and the Tycoon 2FA phishing kit.

Legitimate services abused

As highlighted in Darktrace's 2024 Annual Threat Report [1], threat actors are abusing legitimate services, like Milanote, in their phishing campaigns. By leveraging these trusted platforms and domains, malicious actors can bypass traditional security measures, making their phishing emails appear benign and increasing the likelihood of successful attacks.

Darktrace categorizes these senders and platforms as free content senders. These services allow users to send emails containing custom content (e.g., files) from fully validated, fixed service address belonging to legitimate corporations. Although some of these services permit full body and subject customization by attackers, the structure of these emails is generally consistent, making it challenging to differentiate between legitimate and malicious emails.

What is Tycoon 2FA?

Tycoon 2FA is an Adversary-in-the-Middle (AitM) phishing kit, first seen in August 2023 and distributed via the Phishing-as-a-Service (PhaaS) model [2]. It targets multi-factor authentication (MFA) by intercepting credentials and MFA tokens during authentication on fake Microsoft or Google login pages. The attacker captures session cookies after MFA is completed, allowing them to replay the session and access the user account, even if credentials are reset. The rise in MFA use has increased the popularity of AitM phishing kits like Tycoon 2FA and Mamba 2FA, another AiTM phishing kit investigated by Darktrace.

Initial access via phishing email

At the beginning of 2025, Darktrace observed phishing emails leveraging Milanote being sent to multiple internal recipients in an organization. In this attack, the same email was sent to 19 different users, all of which were held by Darktrace.

The subject line of the emails mentioned both a legitimate internal user of the company, the company name, as well as a Milanote board regarding a “new agreement” in German. It is a common social engineering technique to mention urgent matters, such as unpaid invoices, expired passwords, or awaiting voicemails, in the subject line to prompt immediate action from the user. However, this tactic is now widely covered in phishing awareness training, making users more suspicious of such emails. In this case, while the subject mentioned a “new agreement,” likely raising the recipient’s curiosity, the tone remained professional and not overly alarming. Additionally, the mention of a colleague and the standardized language typical of free content sender emails further helped dispel concerns regarding the email.

These emails were sent by the legitimate address support@milanote[.]com and referenced "Milanote" in the personal field of the header but originated from the freemail address “ahnermatternk.ef.od.13@gmail[.]com”. Darktrace / EMAIL recognized that none of the recipients had previously received a file share email from Milanote, making this sender unfamiliar in the customer's email environment

The emails contained several benign links to legitimate Milanote endpoints (including an unsubscribe link) which were not flagged by Darktrace. However, they also included a malicious link designed to direct recipients to a pre-filled credential harvesting page hosted on Milanote, prompting them to register for an account. Despite not blocking the legitimate Milanote links in the same email, Darktrace locked the malicious link, preventing users from visiting the credential harvester.

Credential harvesting page sent to recipients, as seen in. sandbox environment.
Figure 1: Credential harvesting page sent to recipients, as seen in. sandbox environment.

Around one minute later, one recipient received a legitimate email from Milanote confirming their successful account registration, indicating they had accessed the phishing page. This email had a lower anomaly score and was not flagged by Darktrace / EMAIL because, unlike the first email, it did not contain any suspicious links and was a genuine account registration notification. Similarly, in the malicious Milanote email, only the link leading to the phishing page was blocked, while the benign and legitimate Milanote links remained accessible, demonstrating Darktrace’s precise and targeted actioning.

A legitimate and a malicious Milanote email received by one recipient.
Figure 2: A legitimate and a malicious Milanote email received by one recipient.

Around the same time, Darktrace / NETWORK observed the same user’s device making DNS query for the domain name “lrn.ialeahed[.]com” , which has been flagged as a Tycoon 2FA domain [2], suggesting the use of this phishing platform.

Once the user had entered their details in the credential harvester, it is likely that they were presented a document hosted on Milanote that contained the final payload link – likely hidden behind text instructing users to access a “new agreement” document.

External research indicates that the user was likely directed to a Cloudflare Turnstile challenge meant to reroute unwanted traffic, such as automated security scripts and penetration testing tools [2] [3]. After these checks and other background processes are completed, the user is directed to the final landing page. In this case, it was likely a fake login prompt hosted on the attacker’s server, where the user is asked to authenticate to their account using MFA. By burrowing malicious links and files in this manner, threat actors can evade analysis by traditional security email gateways, effectively bypassing their protection.

Darktrace’s analysis of the structure and word content of the phishing emails resulted in an 82% probability score that the email was malicious, and the email further received a 67% phishing inducement score, representing how closely the structure and word content of the emails compared to typical phishing emails.

All these unusual elements triggered multiple alerts in Darktrace / EMAIL, focusing on two main suspicious aspects: a new, unknown sender with no prior correspondence with the recipients or the environment, and the inclusion of a link to a previously unseen file storage solution.

Milanote phishing email as seen within Darktrace / EMAIL.
Figure 3: Milanote phishing email as seen within Darktrace / EMAIL.

After detecting the fifth email, the “Sender Surge” model alert was triggered in Darktrace / EMAIL due to a significant number of recipients being emailed by this new suspicious sender in a short period. These recipients were from various departments across the customer’s organization, including sales, marketing, purchasing, and production. Darktrace / EMAIL determined that the emails were sent to a highly unusual group of internal recipients, further raising doubts about the business legitimacy.

Darktrace / EMAIL suggested actions to contain the attack by holding all Milanote phishing emails back from recipient’s inboxes, except for the detailed email with locked links. However, autonomous actions were not enabled at the time, allowing the initial email to reach recipients' inboxes, providing a brief window for interaction. Unfortunately, during this window, one recipient clicked on the Milanote payload link, leading to the compromise of their account.

SaaS account takeover

About three minutes after the malicious Milanote email was received, Darktrace / IDENTITY detected an unusual login to the email recipient’s SaaS account. The SaaS actor was observed accessing files from their usual location in Germany, while simultaneously, a 100% rare login occurred from a location in the US that had never been seen in the customer’s environment before. This login was also flagged as suspicious by Microsoft 365, triggering a 'Conditional Access Policy' that required MFA authentication, which was successfully completed.

Tycoon 2FA adnimistration panel login page dated from October 2023 [3].
Figure 4: Tycoon 2FA adnimistration panel login page dated from October 2023 [3].

Despite the successful authentication, Darktrace / IDENTITY recognized that the login from this unusual location, coupled with simultaneous activity in another geographically distant location, were highly suspicious. Darktrace went on to observe MFA-validated logins from three separate US-based IP addresses: 89.185.80[.]19, 5.181.3[.]68, and 38.242.7[.]252. Most of the malicious activity was performed from the latter, which is associated with the Hide My Ass (HMA) VPN network [5].

Darktrace’s detection of the suspicious login from the US while the legitimate user was logged in from Germany.
Figure 5: Darktrace’s detection of the suspicious login from the US while the legitimate user was logged in from Germany.
Darktrace’s detection of the suspicious login following successful MFA authentication.
Figure 6: Darktrace’s detection of the suspicious login following successful MFA authentication.

Following this, the malicious actor accessed the user’s inbox and created a new mailbox rule named “GTH” that deleted any incoming email containing the string “milanote” in the subject line or body. Rules like this are a common technique used by attackers to leverage compromised accounts for launching phishing campaigns and concealing replies to phishing emails that might raise suspicions among legitimate account holders. Using legitimate, albeit compromised, accounts to send additional phishing emails enhances the apparent legitimacy of the malicious emails. This tactic has been reported as being used by Tycoon 2FA attackers [4].

The attacker accessed over 140 emails within the legitimate user’s inbox, including both the inbox and the “Sent Items” folder. Notably, the attacker accessed five emails in the “Sent Items” folder and modified their attachments. These emails were mainly related to invoices, suggesting the threat actor may have been looking to hijack those email threads to send fake invoices or replicate previous invoice emails.

Darktrace’s Cyber AI AnalystTM launched autonomous investigations into the individual events surrounding this suspicious activity. It connected these separate events into a single, broad account takeover incident, providing the customer with a clearer view of the ongoing compromise.

Cyber AI Analyst’s detection of unusual SaaS account activities in a single incident.
Figure 7: Cyber AI Analyst’s detection of unusual SaaS account activities in a single incident.
Cyber AI Analyst investigation of suspicious activities performed by the attacker.
Figure 8: Cyber AI Analyst investigation of suspicious activities performed by the attacker.

Darktrace's response

Within three minutes of the first unusual login alert, Darktrace’s Autonomous Response intervened, disabling the compromised user account for two hours.

As the impacted customer was subscribed to the Managed Threat Detection Service, Darktrace’s SOC team investigated the activity further and promptly alerted the customer’s security team. With the user’s account still disabled by Autonomous Response, the attack was contained, allowing the customer’s security team valuable time to investigate and remediate. Within ten minutes of receiving the alert from Darktrace’s SOC, they reset the user’s password, closed all active SaaS sessions, and deleted the malicious email rule. Darktrace’s SOC further supported the customer through the Security Operations Service Support service by providing information about the data accessed and identifying any other affected users.

Autonomous Response actions carried out by Darktrace / IDENTITY to contain the malicious activity
Figure 9: Autonomous Response actions carried out by Darktrace / IDENTITY to contain the malicious activity.

A wider Milanote phishing campaign?

Around a month before this compromise activity, Darktrace alerted another customer to similar activities involving two compromised user accounts. These accounts created new inbox rules named “GFH” and “GVB” to delete all incoming emails containing the string “milanote” in their subject line and/or body.

The phishing emails that led to the compromise of these user accounts were similar to the ones discussed above. Specifically, these emails were sent via the Milanote platform and referenced a “new agreement” (in Spanish) being shared by a colleague. Additionally, the payload link included in the phishing emails showed the same UserPrincipalName (UPN) attribute (i.e., click?upn=u001.qLX9yCzR), which has been seen in other Milanote phishing emails leveraging Tycoon 2FA reported by OSINT sources [6]. Interestingly, in some cases, the email also referenced a “new agreement” in Portuguese, indicating a global campaign.

Based on the similarities in the rule’s naming convention and action, as well as the similarities in the phishing email subjects, it is likely that these were part of the same campaign leveraging Milanote and Tycoon 2FA to compromise user accounts. Since its introduction, the Tycoon 2FA phishing kit has undergone several enhancements to increase its stealth and obfuscation methods, making it harder for security tools to detect. For example, the latest versions contain special source code to obstruct web page analysis by defenders, prevent users from copying meaningful text from the phishing webpages, and disable the right-click menu to prevent offline analysis [4].

Conclusion

Threat actors are continually employing new methods to bypass security detection tools and measures. As highlighted in this blog, even robust security mechanisms like MFA can be compromised using AitM phishing kits. The misuse of legitimate services such as Milanote for malicious purposes can help attackers evade traditional email security solutions by blurring the distinction between legitimate and malicious content.

This is why security tools based on anomaly detection are crucial for defending against such attacks. However, user awareness is equally important. Delays in processing can impact the speed of response, making it essential for users to be informed about these threats.

[related-resource]

Appendices

References

[1] https://www.darktrace.com/resources/annual-threat-report-2024

[2] https://www.validin.com/blog/tycoon_2fa_analyzing_and_hunting_phishing-as-a-service_domains

[3] https://blog.sekoia.io/tycoon-2fa-an-in-depth-analysis-of-the-latest-version-of-the-aitm-phishing-kit/#h-iocs-amp-technical-details

[4] https://blog.barracuda.com/2025/01/22/threat-spotlight-tycoon-2fa-phishing-kit

[5] https://spur.us/context/38.242.7.252    

[6] https://any.run/report/5ef1ac94e4c6c1dc35579321c206453aea80d414108f9f77abd2e2b03ffbd658/be5351d9-53c0-470b-8708-ee2e29300e70

Indicators of Compromise (IoCs)

IoC         Type      Description + Probability

89.185.80[.]19 - IP Address - Malicious login

5.181.3[.]68 - IP Address -Malicious login

38.242.7[.]252 - IP Address - Malicious login and new email inbox rule creation -  Hide My Ass VPN

lrn.ialeahed[.]com – Hostname - Likely Tycoon 2FA domain

Darktrace Model Detections

Email alerts

Platforms / Free Content Sender + High Sender Surge

Platforms / Free Content Sender + Sender Surge

Platforms / Free Content Sender + Unknown Initiator

Platforms / Free Content Sender

Platforms / Free Content Sender + First Time Recipient

Unusual / New Sender Surge

Unusual / Sender Surge

Antigena Anomaly / High Antigena Anomaly

Association / Unknown Sender

History / New Sender

Link / High Rarity Link to File Storage

Link/ Link To File Storage

Link / Link to File Storage + Unknown Sender

Link / Low Link Association

Platforms / Free Content Sender + First Time Initiator

Platforms / Free Content Sender + Unknown Initiator + Freemail

Platforms / Free Content Sender Link

Unusual / Anomalous Association

Unusual / Unlikely Recipient Association

IDENTITY

SaaS / Access / Unusual External Source for SaaS Credential Use

SaaS / Compromise / Login from Rare High Risk Endpoint

SaaS / Access / M365 High Risk Level Login

SaaS / Compromise / Login From Rare Endpoint While User Is Active

SaaS / Access / MailItemsAccessed from Rare Endpoint

SaaS / Unusual Activity / Multiple Unusual SaaS Activities

SaaS / Unusual Activity / Multiple Unusual External Sources For SaaS Credential

SaaS / Compliance / Anomalous New Email Rule

SaaS / Compromise / Unusual Login and New Email Rule

SaaS / Compromise / SaaS Anomaly Following Anomalous Login

Antigena / SaaS / Antigena Suspicious SaaS Activity Block

Antigena / SaaS / Antigena Enhanced Monitoring from SaaS User Block

Antigena / SaaS / Antigena Unusual Activity Block

Antigena / SaaS / Antigena Suspicious SaaS and Email Activity Block

Cyber AI Analyst Incident

Possible Hijack of Office365 Account

MITRE ATT&CK Mapping

Tactic – Technique

DEFENSE EVASION, PERSISTENCE, PRIVILEGE ESCALATION, INITIAL ACCESS - Cloud Accounts

INITIAL ACCESS - Phishing

CREDENTIAL ACCESS - Steal Web Session Cookie

PERSISTENCE - Account Manipulation

PERSISTENCE - Outlook Rules

RESOURCE DEVELOPMENT - Email Accounts

RESOURCE DEVELOPMENT - Compromise Accounts

Experts breakdown of Identity Security in the Cloud Era

This white paper discusses the current identity threat landscape and how defenders can adopt new tools to better secure their users and data.

Inside the SOC
Darktrace cyber analysts are world-class experts in threat intelligence, threat hunting and incident response, and provide 24/7 SOC support to thousands of Darktrace customers around the globe. Inside the SOC is exclusively authored by these experts, providing analysis of cyber incidents and threat trends, based on real-world experience in the field.
Written by
Alexandra Sentenac
Cyber Analyst

More in this series

No items found.

Blog

/

Email

/

July 24, 2026

Darktrace / EMAIL Expands Behavioral Defense Across Email and Collaboration Workflows

Default blog imageDefault blog image

Email and collaboration tools do more than carry messages. They are where organizations approve payments, share sensitive data, reset credentials, and make thousands of everyday decisions. Increasingly, they are interfaces through which humans direct AI agents in their daily activity. Email, Slack and Teams are high volume, rich with sensitive data, and an easy place to hide malicious activity.

The opportunity isn’t lost on bad actors. Darktrace / EMAIL detected more than 32 million high-confidence phishing emails globally in 2025, and 70% of those messages passed DMARC authentication.  Phishing is increasingly difficult to detect and familiar trust signals alone are not enough. People and security teams need to understand how a message fits the normal behavior of the sender, recipient, and organization. They also need to correlate activity across platforms to spot threats that span multiple channels.

To effectively secure against today’s evolved threats, security teams need to act at two levels: they need to help each employee make a safer decision ‘in the moment’, and they need to understand the wider patterns that may expose the business to risk.

Darktrace is introducing four new capabilities in Darktrace / EMAIL to address both challenges. The new features explain suspicious content more clearly to end users, strengthen the capabilities of Darktrace / Adaptive Human Defense with richer guidance, let organizations define their own patterns for detecting sensitive data in messages, and give security teams a process-level view of risk across email and collaboration workflows.

Darktrace / EMAIL Inbox Analysis highlights risky content within your emails

A warning is more useful when it explains what the user should look at. To help do that, we’ve expanded Darktrace / EMAIL’s Inbox Analysis Add-In to highlight potentially dangerous content within the body of emails that Darktrace / EMAIL flags as potentially suspicious or high risk.  

The add-in can highlight language designed to create urgency, financial references, requests for payment, suspicious links, and content that is unusual for the sender. Each highlighted element includes a pop up that explains why it may be suspicious. Instead of asking an employee to accept a verdict without context, the analysis helps them examine the message and make a more informed decision.

Enhanced Just-In-Time Training Banners in Darktrace / Adaptive Human Defense

Enhanced Just-In-Time Training Banners build on the same principle. The banners now include a contextual header, actionable advice, and specific detection context. This gives employees more useful guidance at the point of risk without adding unnecessary information or cognitive load.

Together, the capabilities help turn a warning into a short learning moment. Employees can see what looks unusual, understand what action to take, and build their judgment.

Custom Sensitive Data Detection in Darktrace / EMAIL - Data Loss Prevention

Sensitive data is different for every business. Standard categories such as payment card details or government identifiers matter, but organizations also have their own customer codes, project names, research formats, account structures, and internal identifiers.

Custom Sensitive Data Detection in Darktrace / EMAIL - Data Loss Prevention allows administrators to write custom expressions for the data their organization needs to protect. Matched content can trigger existing model actions and data loss prevention (DLP) workflows, extending Darktrace's DLP capabilities.

This extends data loss detection beyond a fixed library of common data types. Security teams can apply controls to information that is sensitive in the context of their own organization and adapt those controls as the business changes.

Introducing Email and Collaboration Workflow Risk Posture Dashboards

Some of the most important risks are not isolated events. They are repeated ways of working that create an opening for error, misuse, or attack. For example, a payment request may be one suspicious message, but a recurring approval workflow that relies on weak verification is a business process risk.

The new Email and Collaboration Workflow Risk Posture Dashboard analyzes email and collaboration data across Email, Microsoft Teams, Slack and Zoom to provide a process-level view of risk in the organization. These may include financial authorization workflows, sensitive data sharing patterns, and activity that could expose credentials.

The dashboard brings these patterns into a view and provides actionable recommendations. This helps security teams determine where to investigate or strengthen controls, where ownership needs to be clarified, and where the business may need to change a risky process. It gives CISOs a clearer view of how human and communication risk is embedded in everyday operations, not only where individual alerts occur.

Behavior connects the individual decision to the wider risk

These capabilities build on Darktrace’s unique behavioral approach to security. We use Adaptive AI to learn how people and AI normally behave within an organization, creating the context needed to recognize when activity changes.

Within the Darktrace Behavioral Defense Platform, Darktrace / EMAIL helps protect people against phishing, account takeover, data exfiltration, and human risk across email and collaboration tools. The new capabilities extend that protection in both directions. They give employees clearer context for the decision in front of them, while giving security leaders a broader view of the workflows and behavior that create risk across the organization.

The result is not simply more alerts. It is a better understanding of why something is risky, what action to take, and where the organization can reduce risk before a familiar process becomes an easy route for an attacker.

[related-resource]

Continue reading
About the author
Carlos Gray
Senior Product Marketing Manager, Email

Blog

/

/

July 24, 2026

When Guardrails Break: Why Securing AI Requires Behavioral Detection and Autonomous Containment

behavioral detectionDefault blog imageDefault blog image

Bottom line up front: Governance, guardrails, identity controls, and secure development are necessary to secure AI, but they are not sufficient. AI systems are probabilistic, adaptive, and non-deterministic. Therefore, organizations need two critical layers of security:

  1. Behavioral-based detection that can identify when AI begins to act outside its intended purpose; and  
  2. Surgical, explainable autonomous containment that can stop risky activity before it causes material damage.  

That capability depends on multiple specialized AI models working together, not one LLM making every decision.

Organizations are embedding AI into development, business operations, and security workflows faster than most security programs can adapt. The risk is no longer limited to the model. It extends across prompts, data, identities, agents, memory, APIs, tools, permissions, and the trust relationships connecting them.

In my recent blog, Securing AI: Analysis of the Complete Security Stack with Governance and Controls, I outlined a defense-in-depth strategy spanning governance, identity, data security, secure development, runtime detection, autonomous containment, and recovery. The most urgent requirement across that architecture is the ability to understand how AI behaves in practice and contain it when that behavior becomes risky.  

Why non-deterministic systems require behavioral-based detection

Traditional controls remain foundational. Organizations need least privilege, strong identity controls, secure-by-design architecture, data governance, AI inventories, guardrails, testing, and clear boundaries on autonomy.

But deterministic controls, which assume predictable and repeatable behavior, cannot fully secure non-deterministic systems, where the same input may not always produce the same outcome.

AI agents can interpret the same instruction differently, chain individually authorized actions into an unsafe outcome, or pursue a legitimate goal through a method the organization did not anticipate. One of the most recent examples of this is the incident that OpenAI and Hugging Face jointly disclosed, where an autonomous agent escaped its intended testing boundaries and compromised Hugging Face infrastructure.  

An agent may have permission to access data and invoke a tool, but that does not mean every use of that access is appropriate. It is not enough to know whether an action is allowed. Organizations need to know whether it makes sense.

  • Is this normal for this agent?  
  • Is it acting within its intended purpose?  
  • Is it accessing unusual data, invoking an unexpected tool, or beginning to drift?  
  • Do a series of ordinary-looking actions become risky when viewed together?

Behavioral-based detection specific to an environment or organization with an understanding of context and risk enables provides the needed detection engineering for AI systems. It learns normal activity across people, systems, data, devices, and AI agents, then identifies deviations and evaluates their risk, intent, and context. This enables detection of misuse, abuse, compromise, manipulation, and unintended behavior even when no known attack signature or explicit policy violation exists.

Why accuracy is the foundation for SOC optimization

AI will only improve the SOC if it produces accurate, explainable, and actionable outcomes.

If analysts must manually validate every AI-generated finding because they cannot understand the evidence or confidence behind it, automation has not reduced workload. It has moved the workload. False positives increase fatigue. False negatives cause the most risk and damage to organizations. Inaccurate autonomous actions can disrupt critical operations.

Accuracy is therefore more than a model-performance metric. It is the prerequisite for analyst trust, SOC optimization, and safe autonomous response.

That accuracy is unlikely to come from one model.

Generative AI is valuable for natural-language analysis, summarization, and human interaction. But an LLM should not be the sole analytical engine for behavioral-based detection, investigation, risk assessment, and containment. Interpretability and consistency are required for high-consequence security decisions.

A stronger architecture uses multiple specialized AI systems collaboratively:  

  • Behavioral models can establish normal activity.  
  • Unsupervised learning can identify novel anomalies.  
  • Graph analysis can evaluate relationships among agents, identities, systems, and tools.  
  • Other models can correlate events, investigate competing hypotheses, and assess risk.  
  • Semantic models can analyze language where behavior-based language analysis is needed but this can be used in tandem with vector embeddings, graph neural networks, and a variety of other AI systems.

Each model contributes a different analytical perspective. Their outputs can corroborate one another, improving accuracy and creating a more reliable basis for response. The objective is not one model operating as an oracle. It is layered, adaptive intelligence designed to produce decisions the SOC can understand and trust.

Autonomous containment is required to secure autonomous systems

Many SOCs remain hesitant to trust LLM-based agents with autonomous containment. That concern is reasonable. A poorly selected response can isolate the wrong asset, stop a critical workflow, block a legitimate identity, or create more operational damage than the original incident.

But relying exclusively on human response is also not viable.

AI systems can operate at machine speed. They can expose sensitive data, execute workflows, modify records, call tools, or propagate actions across connected systems before an analyst can investigate and intervene. The behavior may be unintentional, the result of an agent optimizing toward a goal, or caused by misuse, compromise, prompt injection, or offensive AI.

Intent affects the investigation. It does not change the need to stop the damage.

Organizations need autonomous response, but it must be surgical and explainable. The objective is not to shut down an entire agent, user, application, or business process whenever an anomaly occurs. It is to interrupt the specific risky behavior: block an unusual connection, constrain a tool call, stop an abnormal data transfer, or temporarily limit an agent when it is performing anomalous, risky activity.  

That buys humans time. It stops the spread, limits damage, and allows the SOC to investigate without unnecessarily disrupting the business.

Layered, Adaptive AI provides a path forward

Darktrace has spent more than a decade researching and operationalizing layered, behavioral, Adaptive AI that learns a specific organization rather than relying only on historic attacks or predefined signatures.

The approach is designed to understand normal behavior, identify anomalous activity, assess its risk, correlate related events, autonomously investigate, and, when necessary, apply targeted containment while normal operations continue.

That sequence matters. Autonomous response cannot simply be added to the end of an LLM workflow. Trusted containment depends on broad visibility, continuous behavioral understanding, multiple analytical techniques, risk and context evaluation, autonomous investigation, explainability, and precise response actions.

This represents a more responsible model for security autonomy: not automation for its own sake, but controlled autonomy built to improve security outcomes and protect business operations.

Security must enable AI adoption

The answer for security teams is not to block AI. Organizations are adopting it to improve productivity, accelerate development, and create new business value.

But innovation without behavioral detection and autonomous containment is not sustainable.

Organizations should continue investing in governance, identity, least privilege, data security, secure MLOps, guardrails, testing, evaluation, validation, verification, kill switches, rollback, and forensic readiness. At the same time, they cannot wait for every governance program to mature before addressing runtime risk.

Behavioral-based detection and autonomous containment provide an immediate layer of resilience. They allow organizations to detect exploitation and risky AI behavior they did not anticipate, contain it at machine speed, and preserve human control over broader remediation.

The future of AI security will not be defined by a single model making every decision. It will be defined by multiple specialized AI systems working collaboratively, with sufficient accuracy, transparency, and context to support trusted autonomous action.

Surgical, explainable autonomous containment is no longer a future capability. It is a requirement for scaling AI securely today.

Learn how to build a defense-in-depth strategy for securing AI at scale in our talk at Black Hat on August 5 at 3:15 PM.  

[related-resource]

Continue reading
About the author
Your data. Our AI.
Elevate your network security with Darktrace AI