Blog
/
Email
/
December 31, 2024

Defending AITM Phishing and Mamba Attacks

Analyze the challenges posed by AITM phishing threats and Mamba 2FA, and discover how to safeguard your systems effectively.
Inside the SOC
Darktrace cyber analysts are world-class experts in threat intelligence, threat hunting and incident response, and provide 24/7 SOC support to thousands of Darktrace customers around the globe. Inside the SOC is exclusively authored by these experts, providing analysis of cyber incidents and threat trends, based on real-world experience in the field.
Written by
Patrick Anjos
Senior Cyber Analyst
Default blog image
31
Dec 2024

What are Adversary-in-the-Middle (AiTM) phishing kits?

Phishing-as-a-Service (PhaaS) platforms have significantly lowered the barriers to entry for cybercriminals, enabling a new wave of sophisticated phishing attacks. Among the most concerning developments in this landscape is the emergence of Adversary-in-the-Middle (AiTM) phishing kits, which enhance traditional phishing tactics by allowing attackers to intercept and manipulate communications in real-time. The PhaaS marketplace offers a wide variety of innovative capabilities, with basic services starting around USD 120 and more advanced services costing around USD 250 monthly [1].

These AiTM kits are designed to create convincing decoy pages that mimic legitimate login interfaces, often pre-filling user information to increase credibility. By acting as a man-in-the-middle, attackers can harvest sensitive data such as usernames, passwords, and even multi-factor authentication (MFA) tokens without raising immediate suspicion. This capability not only makes AiTM attacks more effective but also poses a significant challenge for cybersecurity defenses [2].

Mamba 2FA is one such example of a PhaaS strain with AiTM capabilities that has emerged as a significant threat to users of Microsoft 365 and other enterprise systems. Discovered in May 2024, Mamba 2FA employs advanced AiTM tactics to bypass MFA, making it particularly dangerous for organizations relying on these security measures.

What is Mamba 2FA?

Phishing Mechanism

Mamba 2FA employs highly convincing phishing pages that closely mimic legitimate Microsoft services like OneDrive and SharePoint. These phishing URLs are crafted with a specific structure, incorporating Base64-encoded parameters. This technique allows attackers to tailor the phishing experience to the targeted organization, making the deception more effective. If an invalid parameter is detected, users are redirected to a benign error page, which helps evade automated detection systems [5].

Figure 1: Phishing page mimicking the Microsoft OneDrive service.

Real-Time Communication

A standout feature of Mamba 2FA is its use of the Socket.IO JavaScript library. This library facilitates real-time communication between the phishing page and the attackers' backend servers. As users input sensitive information, such as usernames, passwords, and MFA tokens on the phishing site, this data is immediately relayed to the attackers, enabling swift unauthorized access [5].

Multi-Factor Authentication Bypass

Mamba 2FA specifically targets MFA methods that are not resistant to phishing, such as one-time passwords (OTPs) and push notifications. When a user enters their MFA token, it is captured in real-time by the attackers, who can then use it to access the victim's account immediately. This capability significantly undermines traditional security measures that rely on MFA for account protection.

Infrastructure and Distribution

The platform's infrastructure consists of two main components: link domains and relay servers. Link domains handle initial phishing attempts, while relay servers are responsible for stealing credentials and completing login processes on behalf of the attacker. The relay servers are designed to mask their IP addresses by using proxy services, making it more difficult for security systems to block them [3].

Evasion Techniques

To evade detection by security tools, Mamba 2FA employs several strategies:

  • Sandbox Detection: The platform can detect if it is being analyzed in a sandbox environment and will redirect users to harmless pages like Google’s 404 error page.
  • Dynamic URL Generation: The URLs used in phishing attempts are frequently rotated and often short-lived to avoid being blacklisted by security solutions.
  • HTML Attachments: Phishing emails often include HTML attachments that appear benign but contain hidden JavaScript that redirects users to the phishing page [5].

Darktrace’s Coverage of Mamba 2FA

Starting in July 2024, the Darktrace Threat Research team detected a sudden rise in Microsoft 365 customer accounts logging in from unusual external sources. These accounts were accessed from an anomalous endpoint, 2607:5500:3000:fea[::]2, and exhibited unusual behaviors upon logging into Software-as-a-Service (SaaS) accounts. This activity strongly correlates with a phishing campaign using Mamba 2FA, first documented in late June 2024 and tracked as Mamba 2FA by Sekoia [2][3].

Darktrace / IDENTITY  was able to identify the initial stages of the Mamba 2FA campaign by correlating subtle anomalies, such as unusual SaaS login locations. Using AI based on peer group analysis, it detected unusual behavior associated with these attacks. By leveraging Autonomous Response actions, Darktrace was able to neutralize these threats in every instance of the campaign detected.

On July 23, a SaaS user was observed logging in from a rare ASN and IP address, 2607:5500:3000:fea::2, originating from the US and successfully passed through MFA authentication.

Figure 2: Model Alert Event Log showing Darktrace’s detection of a SaaS user mailbox logging in from an unusual source it correlates with Mamba 2FA relay server.

Almost an hour later, the SaaS user was observed logging in from another suspicious IP address, 45.133.172[.]86, linked to ASN AS174 COGENT-174. This IP, originating from the UK, successfully passed through MFA validation.

Following this unusual access, the SaaS user was notably observed reading emails and files that could contain sensitive payment and contract information. This behavior suggests that the attacker may have been leveraging contextual information about the target to craft further malicious phishing emails or fraudulent invoices. Subsequently, the user was detected creating a new mailbox rule titled 'fdsdf'. This rule was configured to redirect emails from a specific domain to the 'Deleted Items' folder and automatically mark them as read.

Implications of Unusual Email Rules

Such unusual email rule configurations are a common tactic employed by attackers. They often use these rules to automatically forward emails containing sensitive keywords—such as "invoice”, "payment", or "confidential"—to an external address. Additionally, these rules help conceal malicious activities, keeping them hidden from the target and allowing the attacker to operate undetected.

Figure 3: The model alert “SaaS / Compliance / Anomalous New Email Rule,” pertaining to the unusual email rule created by the SaaS user named ‘fdsdf’.

Blocking the action

A few minutes later, the SaaS user from the unusual IP address 45.133.172[.]86 was observed attempting to send an email with the subject “RE: Payments.” Subsequently, Darktrace detected the user engaging in activities that could potentially establish persistence in the compromised account, such as registering a new authenticator app. Recognizing this sequence of anomalous behaviors, Darktrace implemented an Autonomous Response inhibitor, disabling the SaaS user for two hours. This action effectively contained potential malicious activities, such as the distribution of phishing emails and fraudulent invoices, and gave the customer’s security team the necessary time to conduct a thorough investigation and implement appropriate security measures.

Figure 4: Device Event Log displaying Darktrace’s Autonomous Response taking action by blocking the SaaS account.
Figure 5: Darktrace / IDENTITY highlighting the 16 model alerts that triggered during the observed compromise.

In another example from mid-July, similar activities related to the campaign were observed on another customer network. A SaaS user was initially detected logging in from the unusual external endpoint 2607:5500:3000:fea[::]2.

Figure 6: The SaaS / Compromise / SaaS Anomaly Following Anomalous Login model alert was triggered by an unusual login from a suspicious IP address linked to Mamba 2FA.

A few minutes later, in the same manner as demonstrated in the previous case, the actor was observed logging in from another rare endpoint, 102.68.111[.]240. However, this time it was from a source IP located in Lagos, Nigeria, which no other user on the network had been observed connecting from. Once logged in, the SaaS user updated the settings to "User registered Authenticator App with Notification and Code," a possible attempt to maintain persistence in the SaaS account.

Figure 7: Darktrace / IDENTITY highlighted the regular locations for the SaaS user. The rarity scores associated with the Mamba 2FA IP location and another IP located in Nigeria were classified as having very low regularity scores for this user.

Based on unusual patterns of user behavior, a Cyber AI Analyst Incident was also generated, detailing all potential account hijacking activities. Darktrace also applied an Autonomous Response action, disabling the user for over five hours. This swift action was crucial in preventing further unauthorized access, potential data breaches and further implications.

Figure 8: Cyber AI Analyst Incident detailing the unusual activities related to the SaaS account hijacking.

Since the customer had subscribed to Darktrace Security Operations Centre (SOC) services, Darktrace analysts conducted an additional human investigation confirming the account compromise.

How Darktrace Combats Phishing Threats

The initial entry point for Mamba 2FA account compromises primarily involves phishing campaigns using HTML attachments and deceptive links. These phishing attempts are designed to mimic legitimate Microsoft services, such as OneDrive and SharePoint, making them appear authentic to unsuspecting users. Darktrace / EMAIL leverages multiple capabilities to analyze email content for known indicators of phishing. This includes looking for suspicious URLs, unusual attachments (like HTML files with embedded JavaScript), and signs of social engineering tactics commonly used in phishing campaigns like Mamba 2FA. With these capabilities, Darktrace successfully detected Mamba 2FA phishing emails in networks where this tool is integrated into the security layers, consequently preventing further implications and account hijacks of their users.

Mamba 2FA URL Structure and Domain Names

The URL structure used in Mamba 2FA phishing attempts is specifically designed to facilitate the capture of user credentials and MFA tokens while evading detection. These phishing URLs typically follow a pattern that incorporates Base64-encoded parameters, which play a crucial role in the operation of the phishing kit.

The URLs associated with Mamba 2FA phishing pages generally follow this structure [6]:

https://{domain}/{m,n,o}/?{Base64 string}

Below are some potential Mamba 2FA phishing emails, with the Base64 strings already decoded, that were classified as certain threats by Darktrace / EMAIL. This classification was based on identifying multiple suspicious characteristics, such as HTML attachments containing JavaScript code, emails from senders with no previous association with the recipients, analysis of redirect links, among others. These emails were autonomously blocked from being delivered to users' inboxes.

Figure 9: Darktrace / EMAIL highlighted a possible phishing email from Mamba 2FA, which was classified as a 100% anomaly.
Figure 10: Darktrace / EMAIL highlighted a URL that resembles the characteristics associated with Mamba 2FA.

Conclusion

The rise of PhaaS platforms and the advent of AiTM phishing kits represent a concerning evolution in cyber threats, pushing the boundaries of traditional phishing tactics and exposing significant vulnerabilities in current cybersecurity defenses. The ability of these attacks to effortlessly bypass traditional security measures like MFA underscores the need for more sophisticated, adaptive strategies to combat these evolving threats.

By identifying and responding to anomalous activities within Microsoft 365 accounts, Darktrace not only highlights the importance of comprehensive monitoring but also sets a new standard for proactive threat detection. Furthermore, the autonomous threat response capabilities and the exceptional proficiency of Darktrace / EMAIL in intercepting and neutralizing sophisticated phishing attacks illustrate a robust defense mechanism that can effectively safeguard users and maintain the integrity of digital ecosystems.

Credit to Patrick Anjos (Senior Cyber Analyst) and Nahisha Nobregas (Senior Cyber Analyst)

Get the latest insights on emerging cyber threats

Attackers are adapting, are you ready? This report explores the latest trends shaping the cybersecurity landscape and what defenders need to know in 2025.

  • Identity-based attacks: How attackers are bypassing traditional defenses
  • Zero-day exploitation: The rise of previously unknown vulnerabilities
  • AI-driven threats: How adversaries are leveraging AI to outmaneuver security controls

Stay ahead of evolving threats with expert analysis from Darktrace. Download the report here.

Appendices

Darktrace Model Detections

  • SaaS / Access / M365 High Risk Level Login
  • SaaS / Access / Unusual External Source for SaaS Credential Use
  • SaaS / Compromise / Login From Rare Endpoint While User Is Active
  • SaaS / Compliance / M365 Security Information Modified
  • SaaS / Compromise / Unusual Login and New Email Rule
  • SaaS / Email Nexus / Suspicious Internal Exchange Activity
  • SaaS / Compliance / Anomalous New Email Rule
  • SaaS / Email Nexus / Possible Outbound Email Spam
  • SaaS / Compromise / Unusual Login and Account Update
  • SaaS / Compromise / SaaS Anomaly Following Anomalous Login
  • SaaS / Compliance / M365 Security Information Modified
  • SaaS / Compromise / Login From Rare Endpoint While User Is Active
  • SaaS / Compromise / Unusual Login, Sent Mail, Deleted Sent
  • SaaS / Unusual Activity / Multiple Unusual SaaS Activities
  • SaaS / Email Nexus / Unusual Login Location Following Link to File Storage
  • SaaS / Unusual Activity / Multiple Unusual External Sources For SaaS Credential
  • IaaS / Compliance / Uncommon Azure External User Invite
  • SaaS / Compliance / M365 External User Added to Group
  • SaaS / Access / M365 High Risk Level Login
  • SaaS / Compliance / M365 Security Information Modified
  • SaaS/ Unusual Activity / Unusual MFA Auth and SaaS Activity
  • SaaS / Compromise / Unusual Login and Account Update

Cyber AI Analyst Incidents:

  • Possible Hijack of Office365 Account
  • Possible Hijack of AzureActiveDirectory Account
  • Possible Unsecured Office365 Resource

List of Indicators of Compromise (IoCs)

IoC       Type    Description + Confidence

2607:5500:3000:fea[::]2 - IPv6 - Possible Mamba 2FA relay server

2607:5500:3000:1cab:[:]2 - IPv6 - Possible Mamba 2FA relay server

References

1.     https://securityaffairs.com/136953/cyber-crime/caffeine-phishing-platform.html

2.     https://any.run/cybersecurity-blog/analysis-of-the-phishing-campaign/

3.     https://www.bleepingcomputer.com/news/security/new-mamba-2fa-bypass-service-targets-microsoft-365-accounts/

4.     https://cyberinsider.com/microsoft-365-accounts-targeted-by-new-mamba-2fa-aitm-phishing-threat/

5.     https://blog.sekoia.io/mamba-2fa-a-new-contender-in-the-aitm-phishing-ecosystem/

MITRE ATT&CK Mapping

Tactic – Technique

DEFENSE EVASION, PERSISTENCE, PRIVILEGE ESCALATION, INITIAL ACCESS - Cloud Accounts

DISCOVERY - Cloud Service Dashboard

RESOURCE DEVELOPMENT - Compromise Accounts

CREDENTIAL ACCESS - Steal Web Session Cookie

PERSISTENCE - Account Manipulation

PERSISTENCE - Outlook Rules

RESOURCE DEVELOPMENT - Email Accounts

INITIAL ACCESS - Phishing

Inside the SOC
Darktrace cyber analysts are world-class experts in threat intelligence, threat hunting and incident response, and provide 24/7 SOC support to thousands of Darktrace customers around the globe. Inside the SOC is exclusively authored by these experts, providing analysis of cyber incidents and threat trends, based on real-world experience in the field.
Written by
Patrick Anjos
Senior Cyber Analyst

More in this series

No items found.

Blog

/

/

September 22, 2026

Darktrace / SECURE AI: Extending Behavioral Security for the Age of Agentic AI

Default blog imageDefault blog image

AI is moving faster than the controls built to secure it

Over 80% of Darktrace’s customers now use generative AI services, and the average organization interacted with five different AI provides in August 20261. That level of adoption reflects how quickly AI has become embedded in day-to-day business operations. But as adoption accelerates, so do the opportunities for new forms of exposure that often operate outside traditional security controls. Prompts can expose sensitive information, an over-permissioned agent can turn a routine task into a serious security concern, and an employee using an unsanctioned AI tool to get work done can quietly introduce risk long before the security team is aware of its existence.

Securing AI isn’t a nice-to-have. It's an urgent, board-level requirement for any business that wants AI adoption to be an advantage rather than a liability.

Why traditional security controls fall short

The challenge isn’t a lack of AI security tools, it’s that most security solutions weren’t built for the way AI behaves.  

AI systems are adaptive and, by nature, unpredictable. A prompt can be entirely benign in one context and a serious risk in another. An agent's permissions can look reasonable in isolation and dangerous the moment they're combined with what that agent is actually doing. Rule-based controls, built to catch known signatures and static policy violations, simply aren't designed to catch this kind of subtlety. They might be able to tell you what happened, but they can’t tell you if it mattered.  

Effective AI security requires something different: a deep understanding of what normal looks like across every human, system, and agent in an environment, so the earliest signs of drift, misuse, or compromise stand out as they emerge.

Behavioral understanding is our foundation

Darktrace isn’t reinventing itself to secure AI. We’re extending what we already do. For over a decade, Darktrace has been built on a single premise – that every organization has its own unique, evolving way of operating, shaped by how its systems, users, and devices behave. Understanding that is the only reliable way to catch what rules and signatures miss. Our Adaptive AI™ continuously learns the distinct behaviors, relationships, and operational patterns of every enterprise it protects, building a Unique Behavioral Profile that no other vendor can replicate. It's how we've spent years interpreting ambiguity, uncovering subtle intent, and spotting drift before it becomes dangerous across networks, cloud, identities, email, OT, and endpoints. AI is a new domain, but behavioral security isn't a new discipline for us.

Introducing Darktrace / SECURE AI

Darktrace / SECURE AI helps organizations embrace AI innovation without losing control of how it is used. Delivered through the Darktrace Behavioral Defense Platform™, it provides visibility into AI activity across employees, agents, and AI systems, helping security teams understand how AI is being used throughout their business.

Darktrace then applies behavioral understanding to that activity, adding the context needed to distinguish routine usage from genuine risk. By understanding the relationships between users, agents, systems, and data, / SECURE AI helps organizations move beyond simply observing AI usage to understanding risk, governing behavior against policy, and enabling AI adoption with confidence. The result is greater oversight, stronger governance, and the ability to confidently accelerate AI innovation without creating unmanaged risk.

Securing your AI ecosystem

AI risk doesn't originate from a single source, and effective AI security can't focus on just one layer of the problem. Organizations need visibility, understanding, and governance across the entire AI ecosystem, from the prompts users submit and the agents they create, to the development environments where AI is built and the unsanctioned tools operating outside approved channels. Here's how Darktrace / SECURE AI helps organizations secure each of these areas.

Shadow AI management

Darktrace / SECURE AI helps security teams discover unsanctioned AI services and track usage trends over time using Darktrace telemetry and supported SASE integrations such as Microsoft Entra Global Secure Access. It also extends visibility to Model Context Protocol (MCP) usage, helping teams understand where AI tools and agents are connecting to external services. Through Darktrace platform integrations, organizations can block unsanctioned AI usage or quarantine affected devices when needed, helping them reduce exposure and guide users toward approved options.

AI prompt analysis

Prompts reveal what users are asking AI to do, what information they are sharing, and the outcomes they are trying to produce. Darktrace / SECURE AI provides visibility into prompts, sessions, and responses across supported platforms, including Microsoft Copilot, Copilot Studio, ChatGPT Enterprise, Claude, Salesforce, and AWS Bedrock2. Behavioral analysis detects activity such as attempted jailbreaks, sensitive data exposure, and potential indirect prompt injection, while risk scoring helps analysts prioritize the sessions that need attention. Policy Manager maps organizational AI policies against prompt activity and surfaces potential violations, helping teams govern AI use with direct evidence instead of relying on fragmented logs or inferred intent.

AI agent identities and actions

AI agents have their own permissions, roles, relationships, and access to systems and data. Darktrace / SECURE AI brings agent identities together with the users interacting with them, giving security teams visibility into access, sessions, and connections across supported environments. A real-time audit trail helps teams continuously evaluate whether an agent’s activity remains aligned with its intended purpose, so they can identify excessive access or behavioral drift before it becomes a security issue.

AI agent development risk management

Many AI risks are introduced during development, when agents are created, permissions are assigned, and connections to data sources are established. Darktrace / SECURE AI provides visibility across both low-code and high-code environments. In platforms such as AWS Bedrock, teams can examine AI architecture and the agent identities involved in development and deployment. In low-code environments, such as Copilot Studio, they can observe agents as they are created, monitor whether behavior stays aligned with their intended purpose, and connect prompt activity during development with behavior in production. This helps organizations address misconfigurations and excessive permissions before they reach production.

The promise of AI, secured

The question isn't whether to adopt AI, it's whether security teams can enable secure adoption that allows every business to benefit from AI’s potential.

Darktrace / SECURE AI is built to solve this problem – extending a decade of behavioral understanding to the newest, fastest-moving part of the enterprise, using the same principles which already protect people and hybrid infrastructure.

Darktrace / SECURE AI is generally available now. Discover the product, or get a demo today.

[related-resource]

[1] Based on aggregated Darktrace product telemetry across a fleet of ~8,200 observed deployments, measuring generative AI service usage across accounts monitored in August 2026. The 80%+ figure reflects the share of monitored accounts with any generative AI service usage during the same period.

[2] Microsoft, Copilot, ChatGPT, Claude, Salesforce, and Amazon Bedrock are trademarks of their respective owners.

Continue reading
About the author
Brittany Woodsmall
Product Marketing Manager, AI

Blog

/

/

September 17, 2026

The Problem of Re-defining Human Value in the Agentic Age

Default blog imageDefault blog image

Newsfeeds are constantly informing us about the rapid escalation of agentic AI systems. These systems move far beyond simple machine-based computations, and  the focused, defined and bounded assistance that most AI systems started out as.

The next evolution of AI will harness the agentic properties of orchestration, automation, and heightened value-chains in IT, taking on the burden of workflow management, not just workflow delivery.

In nearly all of these instances, promises are made such as ‘this will free up human time’ or ‘this will allow people to focus on higher-order strategy’. However that message is delivered, one thing is clear: in ceding the orchestration and management of work to increasingly sophisticated AI agents and agentic systems, human value will be elevated to a particular and specific layer: the ability to judge its outputs.  As AI takes on more tasks, humans should be able to focus on a higher level of governance; making sure the decisions that AI offers us are ethical, responsible and worthwhile.  

But there are two major problems with that approach.

This blog discusses the problem of how Agentic systems are re-shaping how we review information, where we fit in, and when we make decisions.  It also discusses the problem of how the increased flow of confident, generated information affects the way we make judgement.  This blog considers how human judgement needs to adapt, and how a behavioral defense approach – using techniques pioneered by Darktrace – can help us do that.

The challenge of knowing where human judgement belongs

As we confer more automated decision-making to agentic systems, it might look increasingly less like ‘granting permissions’, and more like ‘surrendering authority’.  

The judgement layer for AI-generated work is not a fixed boundary. We have become used to the idea of a ‘human in the loop’ (HITL) and, historically, relationships between humans and IT systems were reasonably clear and bounded.  Computer and software systems were programmed to carry out certain tasks or automated functions, and humans could control the gates and decision points where actions were undertaken. Even across highly complex computational workflows, human interaction was a controllable node within the process; we were able to configure and regulate. But in the agentic age, where that human interaction sits, and what it can influence shifts every time AI systems are granted autonomy.  

This leads us to the first problem: if humans are moving themselves (or are being moved) into the ‘judgement’ part of the value chain, exactly where and when do we exercise that judgement?  

Humans are no longer the sole shepherds of computer-based or software-controlled outputs.  We are at times at least one step further (and slower) behind the new agentic shepherds.  We might also be blind to what they are doing.  Not only might we be removed and blind to the actions of our AI shepherds, but with the challenge of unknown, unapproved AI systems operating beyond our control, humans might not even know that our work is being shepherded by an AI at all.  Simply put, with the advent of greater levels of autonomy and orchestration, humans are at risk of not even knowing where to apply our newly-extended powers of strategic judgement.

Shadow AI – the use of unapproved AI systems or processes – is a growing threat to the role of effective governance and oversight. Shadow AI isn't just the AI you can't see. Its the AI you already know about being used in an unapproved way. The ability to generate effective oversight of the AI systems you use (or that are used on your behalf) will be increasingly important to ensure that human judgement in the AI value chain is effective, and deliberately placed.

The problem of what makes good judgement

The second problem lies in how flawed human judgement can be.  Humans are historically, notoriously, and, sometimes dangerously, unreliable when it comes to exercising judgement.  Humans are prone to the worst kinds of bias, the seduction of malign influence, and the sometimes-overwhelming urge to succeed. AI has long had a known flaw of operating with sycophancy, providing outputs that tend to agree with or flatter the human user.  But as AI grows ever more effective, there is a risk of both hyper-enablement (where humans increasingly and knowingly enable AI despite potential harm), as well as the greater risk of suggestion. Both of these aspects could skew the newly-elevated input of human judgement.

Imagine a highly competent AI system that has just orchestrated and managed a dizzying array of processes and workflows.  The AI is designed to present the human decision-maker with recommendations; based on analysis, comparison and other programmed factors.  This is where the human judgement layer is enabled.  But what if that judgement is summarily diffused by an AI-based recommendation that emulates the decision, provides plausible but unattractive alternatives, then suggests (or, worse, directs) the human end-user to take a particular course of action.

The risk here is that you are given a recommendation, tailored to your preferences (which the AI has learned, or which you have divulged), and which appears to make perfect sense.  It appears to be a well-weighted recommendation, with sound arguments that tap into our inherent biases or inclinations so that a specific decision-path is followed. With the growth of agentic systems specifically designed to match user profiles (from Cowork agents to ‘digital twin’ models), the likelihood of agentic influence could badly skew human judgement or, at the least, devalue the proposition that humans are taking a higher-layer of strategic control over AI-based decisions.

If AI convincingly recommends something that may be problematic, it can be difficult to discern both accurate data, and the context required to make the right judgement.

Given the two problems described above, the job of exercising valuable human judgement in the agentic age can draw down to these two questions:

  • When should humans intervene in the agentic process?  
  • How can we make the best possible judgement calls?

What humans contribute that AI cannot

For all the flaws that make human judgement unreliable, people have the edge over even the most sophisticated and powerful AI systems when it comes to issues such as ethics and social context.  An AI system can, with startling granularity, rank the value of adopting a new business proposal: offering predictive metrics on costs, returns, market value, time-to-deliver operations, conformance with legal registers, etc.  But it can’t tell if the business proposal is ethically sound, or if the business venture will potentially affect groups outside of the analyzed proposal. It can’t tell you if the CEO has a ‘bad feeling’ about this effort.  It can’t tell you if this is the right thing to do.  

The ability to add social context, balance complex interpersonal dynamics, understand nuance, and to go beyond what seems economically reasonable is where human judgement can add value.  

Human judgement is difficult to encapsulate in metrics. And the way we train our development may need to adapt too. Rather than building up a gradual, experiential knowledge base, we should think about training the skill of judgement itself; especially for an agentic age.

How behavioral security strengthens AI governance

If this all feels like a vicious circle (‘I need AI help to make good judgements’ / ‘AI can twist what I need to judge’) it needn’t be. The key to this is having a defense-in-depth approach, with tools that can actually help.

This is precisely where behavioral security becomes important. The complex and nuanced way that humans exercise judgement is often rooted in our ability to recognize behavior that doesn't look right. We may not always be able to articulate it immediately, but we can often identify when an action, recommendation, or outcome feels inconsistent with the context around it. As AI systems take on more responsibility across the decision chain, preserving that ability to recognize meaningful deviations becomes increasingly important.

Darktrace’s / SECURE AI is designed to do exactly that. It applies behavioral security to AI ecosystems, helping organizations understand how people, AI tools, identities, and agents interact across the business. By learning the patterns of normal AI usage and surfacing activity that deviates from those patterns, it provides security teams with the context needed to investigate risk, understand unusual behavior, and make informed governance decisions. Rather than relying solely on predefined rules or assumptions, this behavioral understanding helps organizations distinguish between expected AI activity and behavior that warrants closer scrutiny.

This matters because we are already in an era of information overload. If humans are expected to elevate their value through strategic judgement, the ability to do this without being overwhelmed by data (good or bad) will be critical.  

We need the ability to discern when we're being misled by AI, and whether our judgement calls are being made on the basis of accurate, contextual information. Darktrace / SECURE AI provides that additional layer of defensive security for activity we cannot easily see. Whether it is suspected Shadow AI or skewed recommendations, the net result is a protected organization, where users can more effectively use AI to make positive judgements.

For those where that judgement is a critical skill (both individuals, as well as those working in security teams), improving our metacognition - the ability to understand information in a broader context - will supercharge the value of human judgement. When those judgements are grounded in context rather than assumptions we have better information to make sound decisions.

Conclusion

Human judgement is a skill that is honed over time and experience.  Darktrace’s / SECURE AI employs the same principles, but at machine-speed. Rather than influencing or directing, Darktrace / SECURE AI offers AI-enabled assurance; providing human-based judgement with the right context to make a balanced decision.  

What we judge can be valued by the legitimacy of its outputs. For AI, those outputs are valued on the speed and accuracy of the information provided.  Increasingly for humans, the value of our outputs will be based on the validity of our judgement, and how we justify our decisions in ways that engineer confidence.  

Humans often know more than we can express, while AI is prone to expressing more than it truly understands. Humans can bridge the context AI often fails to appreciate. When that judgement is supported by relevant, impartial AI systems, this is the future space where good AI governance will be exercised.

Discover Darktrace / SECURE AI.

[related-resource]

Continue reading
About the author
Jason Lusted
AI Governance Advisor
Your data. Our AI.
Elevate your network security with Darktrace AI