Blog
/
Identity
/
April 29, 2025

MFA Under Attack: AiTM Phishing Kits Abusing Legitimate Services

Tycoon 2FA uses AiTM phishing and legitimate services to bypass MFA. Darktrace AI stopped it, read the blog to learn how Self-Learning AI detects sophisticated threats.
Inside the SOC
Darktrace cyber analysts are world-class experts in threat intelligence, threat hunting and incident response, and provide 24/7 SOC support to thousands of Darktrace customers around the globe. Inside the SOC is exclusively authored by these experts, providing analysis of cyber incidents and threat trends, based on real-world experience in the field.
Written by
Alexandra Sentenac
Cyber Analyst
fingerprintDefault blog image
29
Apr 2025

In late 2024 and early 2025, the Darktrace Security Operations Center (SOC) investigated alerts regarding separate cases of Software-as-a-Service (SaaS) account compromises on two customer environments that presented several similarities, suggesting they were part of a wider phishing campaign.

This campaign was found to leverage the project collaboration and note-taking application, Milanote, and the Tycoon 2FA phishing kit.

Legitimate services abused

As highlighted in Darktrace's 2024 Annual Threat Report [1], threat actors are abusing legitimate services, like Milanote, in their phishing campaigns. By leveraging these trusted platforms and domains, malicious actors can bypass traditional security measures, making their phishing emails appear benign and increasing the likelihood of successful attacks.

Darktrace categorizes these senders and platforms as free content senders. These services allow users to send emails containing custom content (e.g., files) from fully validated, fixed service address belonging to legitimate corporations. Although some of these services permit full body and subject customization by attackers, the structure of these emails is generally consistent, making it challenging to differentiate between legitimate and malicious emails.

What is Tycoon 2FA?

Tycoon 2FA is an Adversary-in-the-Middle (AitM) phishing kit, first seen in August 2023 and distributed via the Phishing-as-a-Service (PhaaS) model [2]. It targets multi-factor authentication (MFA) by intercepting credentials and MFA tokens during authentication on fake Microsoft or Google login pages. The attacker captures session cookies after MFA is completed, allowing them to replay the session and access the user account, even if credentials are reset. The rise in MFA use has increased the popularity of AitM phishing kits like Tycoon 2FA and Mamba 2FA, another AiTM phishing kit investigated by Darktrace.

Initial access via phishing email

At the beginning of 2025, Darktrace observed phishing emails leveraging Milanote being sent to multiple internal recipients in an organization. In this attack, the same email was sent to 19 different users, all of which were held by Darktrace.

The subject line of the emails mentioned both a legitimate internal user of the company, the company name, as well as a Milanote board regarding a “new agreement” in German. It is a common social engineering technique to mention urgent matters, such as unpaid invoices, expired passwords, or awaiting voicemails, in the subject line to prompt immediate action from the user. However, this tactic is now widely covered in phishing awareness training, making users more suspicious of such emails. In this case, while the subject mentioned a “new agreement,” likely raising the recipient’s curiosity, the tone remained professional and not overly alarming. Additionally, the mention of a colleague and the standardized language typical of free content sender emails further helped dispel concerns regarding the email.

These emails were sent by the legitimate address support@milanote[.]com and referenced "Milanote" in the personal field of the header but originated from the freemail address “ahnermatternk.ef.od.13@gmail[.]com”. Darktrace / EMAIL recognized that none of the recipients had previously received a file share email from Milanote, making this sender unfamiliar in the customer's email environment

The emails contained several benign links to legitimate Milanote endpoints (including an unsubscribe link) which were not flagged by Darktrace. However, they also included a malicious link designed to direct recipients to a pre-filled credential harvesting page hosted on Milanote, prompting them to register for an account. Despite not blocking the legitimate Milanote links in the same email, Darktrace locked the malicious link, preventing users from visiting the credential harvester.

Credential harvesting page sent to recipients, as seen in. sandbox environment.
Figure 1: Credential harvesting page sent to recipients, as seen in. sandbox environment.

Around one minute later, one recipient received a legitimate email from Milanote confirming their successful account registration, indicating they had accessed the phishing page. This email had a lower anomaly score and was not flagged by Darktrace / EMAIL because, unlike the first email, it did not contain any suspicious links and was a genuine account registration notification. Similarly, in the malicious Milanote email, only the link leading to the phishing page was blocked, while the benign and legitimate Milanote links remained accessible, demonstrating Darktrace’s precise and targeted actioning.

A legitimate and a malicious Milanote email received by one recipient.
Figure 2: A legitimate and a malicious Milanote email received by one recipient.

Around the same time, Darktrace / NETWORK observed the same user’s device making DNS query for the domain name “lrn.ialeahed[.]com” , which has been flagged as a Tycoon 2FA domain [2], suggesting the use of this phishing platform.

Once the user had entered their details in the credential harvester, it is likely that they were presented a document hosted on Milanote that contained the final payload link – likely hidden behind text instructing users to access a “new agreement” document.

External research indicates that the user was likely directed to a Cloudflare Turnstile challenge meant to reroute unwanted traffic, such as automated security scripts and penetration testing tools [2] [3]. After these checks and other background processes are completed, the user is directed to the final landing page. In this case, it was likely a fake login prompt hosted on the attacker’s server, where the user is asked to authenticate to their account using MFA. By burrowing malicious links and files in this manner, threat actors can evade analysis by traditional security email gateways, effectively bypassing their protection.

Darktrace’s analysis of the structure and word content of the phishing emails resulted in an 82% probability score that the email was malicious, and the email further received a 67% phishing inducement score, representing how closely the structure and word content of the emails compared to typical phishing emails.

All these unusual elements triggered multiple alerts in Darktrace / EMAIL, focusing on two main suspicious aspects: a new, unknown sender with no prior correspondence with the recipients or the environment, and the inclusion of a link to a previously unseen file storage solution.

Milanote phishing email as seen within Darktrace / EMAIL.
Figure 3: Milanote phishing email as seen within Darktrace / EMAIL.

After detecting the fifth email, the “Sender Surge” model alert was triggered in Darktrace / EMAIL due to a significant number of recipients being emailed by this new suspicious sender in a short period. These recipients were from various departments across the customer’s organization, including sales, marketing, purchasing, and production. Darktrace / EMAIL determined that the emails were sent to a highly unusual group of internal recipients, further raising doubts about the business legitimacy.

Darktrace / EMAIL suggested actions to contain the attack by holding all Milanote phishing emails back from recipient’s inboxes, except for the detailed email with locked links. However, autonomous actions were not enabled at the time, allowing the initial email to reach recipients' inboxes, providing a brief window for interaction. Unfortunately, during this window, one recipient clicked on the Milanote payload link, leading to the compromise of their account.

SaaS account takeover

About three minutes after the malicious Milanote email was received, Darktrace / IDENTITY detected an unusual login to the email recipient’s SaaS account. The SaaS actor was observed accessing files from their usual location in Germany, while simultaneously, a 100% rare login occurred from a location in the US that had never been seen in the customer’s environment before. This login was also flagged as suspicious by Microsoft 365, triggering a 'Conditional Access Policy' that required MFA authentication, which was successfully completed.

Tycoon 2FA adnimistration panel login page dated from October 2023 [3].
Figure 4: Tycoon 2FA adnimistration panel login page dated from October 2023 [3].

Despite the successful authentication, Darktrace / IDENTITY recognized that the login from this unusual location, coupled with simultaneous activity in another geographically distant location, were highly suspicious. Darktrace went on to observe MFA-validated logins from three separate US-based IP addresses: 89.185.80[.]19, 5.181.3[.]68, and 38.242.7[.]252. Most of the malicious activity was performed from the latter, which is associated with the Hide My Ass (HMA) VPN network [5].

Darktrace’s detection of the suspicious login from the US while the legitimate user was logged in from Germany.
Figure 5: Darktrace’s detection of the suspicious login from the US while the legitimate user was logged in from Germany.
Darktrace’s detection of the suspicious login following successful MFA authentication.
Figure 6: Darktrace’s detection of the suspicious login following successful MFA authentication.

Following this, the malicious actor accessed the user’s inbox and created a new mailbox rule named “GTH” that deleted any incoming email containing the string “milanote” in the subject line or body. Rules like this are a common technique used by attackers to leverage compromised accounts for launching phishing campaigns and concealing replies to phishing emails that might raise suspicions among legitimate account holders. Using legitimate, albeit compromised, accounts to send additional phishing emails enhances the apparent legitimacy of the malicious emails. This tactic has been reported as being used by Tycoon 2FA attackers [4].

The attacker accessed over 140 emails within the legitimate user’s inbox, including both the inbox and the “Sent Items” folder. Notably, the attacker accessed five emails in the “Sent Items” folder and modified their attachments. These emails were mainly related to invoices, suggesting the threat actor may have been looking to hijack those email threads to send fake invoices or replicate previous invoice emails.

Darktrace’s Cyber AI AnalystTM launched autonomous investigations into the individual events surrounding this suspicious activity. It connected these separate events into a single, broad account takeover incident, providing the customer with a clearer view of the ongoing compromise.

Cyber AI Analyst’s detection of unusual SaaS account activities in a single incident.
Figure 7: Cyber AI Analyst’s detection of unusual SaaS account activities in a single incident.
Cyber AI Analyst investigation of suspicious activities performed by the attacker.
Figure 8: Cyber AI Analyst investigation of suspicious activities performed by the attacker.

Darktrace's response

Within three minutes of the first unusual login alert, Darktrace’s Autonomous Response intervened, disabling the compromised user account for two hours.

As the impacted customer was subscribed to the Managed Threat Detection Service, Darktrace’s SOC team investigated the activity further and promptly alerted the customer’s security team. With the user’s account still disabled by Autonomous Response, the attack was contained, allowing the customer’s security team valuable time to investigate and remediate. Within ten minutes of receiving the alert from Darktrace’s SOC, they reset the user’s password, closed all active SaaS sessions, and deleted the malicious email rule. Darktrace’s SOC further supported the customer through the Security Operations Service Support service by providing information about the data accessed and identifying any other affected users.

Autonomous Response actions carried out by Darktrace / IDENTITY to contain the malicious activity
Figure 9: Autonomous Response actions carried out by Darktrace / IDENTITY to contain the malicious activity.

A wider Milanote phishing campaign?

Around a month before this compromise activity, Darktrace alerted another customer to similar activities involving two compromised user accounts. These accounts created new inbox rules named “GFH” and “GVB” to delete all incoming emails containing the string “milanote” in their subject line and/or body.

The phishing emails that led to the compromise of these user accounts were similar to the ones discussed above. Specifically, these emails were sent via the Milanote platform and referenced a “new agreement” (in Spanish) being shared by a colleague. Additionally, the payload link included in the phishing emails showed the same UserPrincipalName (UPN) attribute (i.e., click?upn=u001.qLX9yCzR), which has been seen in other Milanote phishing emails leveraging Tycoon 2FA reported by OSINT sources [6]. Interestingly, in some cases, the email also referenced a “new agreement” in Portuguese, indicating a global campaign.

Based on the similarities in the rule’s naming convention and action, as well as the similarities in the phishing email subjects, it is likely that these were part of the same campaign leveraging Milanote and Tycoon 2FA to compromise user accounts. Since its introduction, the Tycoon 2FA phishing kit has undergone several enhancements to increase its stealth and obfuscation methods, making it harder for security tools to detect. For example, the latest versions contain special source code to obstruct web page analysis by defenders, prevent users from copying meaningful text from the phishing webpages, and disable the right-click menu to prevent offline analysis [4].

Conclusion

Threat actors are continually employing new methods to bypass security detection tools and measures. As highlighted in this blog, even robust security mechanisms like MFA can be compromised using AitM phishing kits. The misuse of legitimate services such as Milanote for malicious purposes can help attackers evade traditional email security solutions by blurring the distinction between legitimate and malicious content.

This is why security tools based on anomaly detection are crucial for defending against such attacks. However, user awareness is equally important. Delays in processing can impact the speed of response, making it essential for users to be informed about these threats.

[related-resource]

Appendices

References

[1] https://www.darktrace.com/resources/annual-threat-report-2024

[2] https://www.validin.com/blog/tycoon_2fa_analyzing_and_hunting_phishing-as-a-service_domains

[3] https://blog.sekoia.io/tycoon-2fa-an-in-depth-analysis-of-the-latest-version-of-the-aitm-phishing-kit/#h-iocs-amp-technical-details

[4] https://blog.barracuda.com/2025/01/22/threat-spotlight-tycoon-2fa-phishing-kit

[5] https://spur.us/context/38.242.7.252    

[6] https://any.run/report/5ef1ac94e4c6c1dc35579321c206453aea80d414108f9f77abd2e2b03ffbd658/be5351d9-53c0-470b-8708-ee2e29300e70

Indicators of Compromise (IoCs)

IoC         Type      Description + Probability

89.185.80[.]19 - IP Address - Malicious login

5.181.3[.]68 - IP Address -Malicious login

38.242.7[.]252 - IP Address - Malicious login and new email inbox rule creation -  Hide My Ass VPN

lrn.ialeahed[.]com – Hostname - Likely Tycoon 2FA domain

Darktrace Model Detections

Email alerts

Platforms / Free Content Sender + High Sender Surge

Platforms / Free Content Sender + Sender Surge

Platforms / Free Content Sender + Unknown Initiator

Platforms / Free Content Sender

Platforms / Free Content Sender + First Time Recipient

Unusual / New Sender Surge

Unusual / Sender Surge

Antigena Anomaly / High Antigena Anomaly

Association / Unknown Sender

History / New Sender

Link / High Rarity Link to File Storage

Link/ Link To File Storage

Link / Link to File Storage + Unknown Sender

Link / Low Link Association

Platforms / Free Content Sender + First Time Initiator

Platforms / Free Content Sender + Unknown Initiator + Freemail

Platforms / Free Content Sender Link

Unusual / Anomalous Association

Unusual / Unlikely Recipient Association

IDENTITY

SaaS / Access / Unusual External Source for SaaS Credential Use

SaaS / Compromise / Login from Rare High Risk Endpoint

SaaS / Access / M365 High Risk Level Login

SaaS / Compromise / Login From Rare Endpoint While User Is Active

SaaS / Access / MailItemsAccessed from Rare Endpoint

SaaS / Unusual Activity / Multiple Unusual SaaS Activities

SaaS / Unusual Activity / Multiple Unusual External Sources For SaaS Credential

SaaS / Compliance / Anomalous New Email Rule

SaaS / Compromise / Unusual Login and New Email Rule

SaaS / Compromise / SaaS Anomaly Following Anomalous Login

Antigena / SaaS / Antigena Suspicious SaaS Activity Block

Antigena / SaaS / Antigena Enhanced Monitoring from SaaS User Block

Antigena / SaaS / Antigena Unusual Activity Block

Antigena / SaaS / Antigena Suspicious SaaS and Email Activity Block

Cyber AI Analyst Incident

Possible Hijack of Office365 Account

MITRE ATT&CK Mapping

Tactic – Technique

DEFENSE EVASION, PERSISTENCE, PRIVILEGE ESCALATION, INITIAL ACCESS - Cloud Accounts

INITIAL ACCESS - Phishing

CREDENTIAL ACCESS - Steal Web Session Cookie

PERSISTENCE - Account Manipulation

PERSISTENCE - Outlook Rules

RESOURCE DEVELOPMENT - Email Accounts

RESOURCE DEVELOPMENT - Compromise Accounts

Experts breakdown of Identity Security in the Cloud Era

This white paper discusses the current identity threat landscape and how defenders can adopt new tools to better secure their users and data.

Inside the SOC
Darktrace cyber analysts are world-class experts in threat intelligence, threat hunting and incident response, and provide 24/7 SOC support to thousands of Darktrace customers around the globe. Inside the SOC is exclusively authored by these experts, providing analysis of cyber incidents and threat trends, based on real-world experience in the field.
Written by
Alexandra Sentenac
Cyber Analyst

More in this series

No items found.

Blog

/

AI

/

May 28, 2026

From Efficiency to Exposure: How AI Adoption Is Creating Unseen Vulnerabilities on the Factory Floor

AI in manufacturingDefault blog imageDefault blog image

How AI agents impact the manufacturing industry

Security teams and IT personnel across the manufacturing industry are under constant pressure to protect production, maintain uptime, and safeguard critical assets but the rise of AI is bringing huge new opportunities alongside new cyber risks. Across manufacturing, AI is embedded into workflows, decision-making, and increasingly, autonomous AI agents are acting on behalf of employees and systems.  

Agentic systems are powerful because they can act independently, but that same autonomy also creates cyber and operational risk. Agents have extensive permissions and are capable of carrying out complex tasks, making decisions, and interacting with tools or external systems with little to no human intervention.

Unlike traditional AI models that perform predefined tasks, AI agents use advanced techniques to mimic human decision-making processes, dynamically adapting to new challenges, making decision and taking action based on their own judgement. They look like employees operationally but lack judgment, ethics, or fear of consequences like humans do. This means they can be easily manipulated by cybercriminals, and an AI agent embedded across an OT network creates threats that extend well beyond data exposure. For example, at BMW, AI identifies faults in welding processes as they occur. At its Spartanburg plant, AI monitors the weld of 300-400 metal studs onto every SUV frame to detect misplaced or faulty studs and correct them instantly. Corruption of BMW’s AI system could lead to catastrophic quality control errors.

Adopting agentic AI systems across manufacturing raises some concerns across security teams. New data from our State of AI Cybersecurity survey shows that 78% of manufacturing security professionals are worried about employee use of AI agents – their top concern. That’s followed by employee use of generative AI tools like CoPilot and ChatGPT, a worry for 76% of security professionals at manufacturing organizations. As these tools gain more access to business data and processes, and more autonomy within organizations, security teams, who today have minimal visibility of agent activity in their environments, increasingly have sensitive data exposure (a worry for 60%) and accidental policy and regulatory violations (59%) on their minds.

External AI-powered threats are evolving just as quickly

The same capabilities transforming manufacturing are also reshaping cyberattacks.

AI is enabling attackers to automate reconnaissance, refine targeting, and adapt in real time. What once required time and manual effort can now be executed continuously and at scale. Manufacturers are already seeing the impact. According to manufacturing security professionals we surveyed, 76% are already being impacted by AI-powered threats and 90% see AI increasing the success of social engineering attacks.

And the techniques themselves are evolving. Concerns across the manufacturing sector show growing anxiety about the range of AI-powered attack routes, most pressingly of adaptive malware that evolves in real-time – a prospect half (49%) of manufacturing security professionals we surveyed are worried by, a full 9% more than the average across industries. AI adaptive malware is followed by:

  • Automated vulnerability scanning and exploit chaining (48%) which has become even more pressing as Anthropic’s new Mythos AI Model supercharges vulnerability discovery
  • Hyper-personalized phishing campaigns (46%), which remain a mainstay in hackers’ arsenals, and AI has amplified their effectiveness by making phishing emails more convincing and harder to detect.

This is not just an increase in volume, it is a shift toward threats that evolve as they unfold - often faster than static defenses can respond.

Despite rising awareness, many manufacturers are not yet equipped to manage this shift. More than half (51%) say they are not adequately prepared for AI-driven threats, and only 37% have formal policies governing AI deployment.  

Securing AI through visibility, context, and guardrails

Addressing this challenge does not require manufacturers to slow innovation. It requires a different approach to security, one that can operate at the same speed and scale as AI. Three specific priorities are emerging for manufacturers looking to take advantage of the power of AI.

Visibility is foundational.  

Organizations need to understand where AI is being used, what it can access, and how it behaves across both IT and OT environments. Without that, risk cannot be measured or managed. It is no surprise that Darktrace’s research found that 91% of manufacturing security professionals said that they need to understand how AI makes decisions before trusting it. This is even more critical in operational settings where disruption has safety, environmental, financial, and reputational impacts.

Context is what turns visibility into action.  

In environments shaped by AI, normal behavior is constantly shifting. Detecting threats requires a behavioral approach; understanding patterns of life across the organization and identifying subtle deviations in real time – a step change in organizations’ traditional approach to security and risk management.

Guardrails ensure that agency does not become exposure  

As AI systems take on greater responsibility, organizations need clear boundaries around what they can do and when they can act independently. These controls must be embedded into systems themselves, not applied after the fact.  

Securing AI Agents Across Manufacturing IT and OT

The rise of agentic AI is transforming manufacturing - powering next-generation operations while reshaping the security landscape. This is not just an increase in threats, but a shift to autonomous systems, continuously evolving behaviors, and risks moving at machine speed. For organizations trying to grapple with the challenge of enabling AI while managing the risk, visibility, context and guardrails should be foundational.

Darktrace helps manufacturers build secure AI approaches by making those foundations possible. It provides visibility and real-time detection and response to unusual activity across IT and OT environments and allows organizations to understand AI activity from the prompts employees use and the agents they build to how those agents are behaving across the environment. For manufacturers scaling AI, this delivers a foundation for innovation without sacrificing control.

Continue reading
About the author
Oakley Cox
Director of Product

Blog

/

AI

/

May 28, 2026

How to Evaluate AI Vendors: 5 Key categories for AI Adoption

Default blog imageDefault blog image

Understanding the AI buyers’ market

AI adoption has become a central topic of discussion in boardrooms, drawing growing interest from business leaders. Ultimately, organizations hope that an investment in AI technology will have tremendous returns. However, the process of buying an AI solution is not as straight forward as it appears on the surface.  

While business leaders may be eager to improve productivity across their operations, practitioners responsible for evaluating and selecting AI solutions may not always have the visibility or technical understanding needed to make the right decisions for their business. What is typically marketed as a holistic solution to their most critical problems is usually followed by uncertainty when AI tools are finally operationalized in real environments.

This guide is intended to support security leaders who are under growing pressure to adopt AI tools while navigating complex terminology, vendor claims, and increasingly crowded buying cycles. Ultimately, the goal is to help organizations evaluate and adopt AI in a safe, effective, and well-governed way. To support this, we’ve structured the evaluation framework across five key categories:

  1. Governance, safety, and data controls
  1. Data gathering and training
  1. Model and technique choice
  1. Performance and accuracy validation    
  1. Interpretability, adjustability, and transparency    

What buying AI looks like in cybersecurity

While investing in AI can bring immense benefits to your security team, first-time buyers of AI cybersecurity solutions may not know where to start. They will have to determine the type of tool they want, know the options available, and evaluate vendors. Research and understanding are critical to ensure purchases are worth the investment.  

With acceleration in AI adoption, accompanied by the recent boom in agentic AI and autonomous agents, CISOs must look “beneath the hood" of these tools to understand how they work, how they are governed, and to ensure the system is secure and compliant with internal policies.

Challenges in the AI buyers’ marketplace  

The AI security software market is buzzing with hype and flashy promises, which, understandably, needs to be addressed with due diligence. Potential buyers, especially in the cybersecurity space, are hesitant when it comes to allowing AI autonomous capabilities across their workflows, and a lack of vendor transparency can exacerbate those feelings.  

Reinforcing this sentiment, research from this year's Darktrace’s State of AI Cybersecurity report shows where confidence and hesitancy emerge amongst potential buyers. On the one hand, security professionals agree that they have good visibility into the logic and reasoning processes their AI solutions use. However, they lack the explainability and trust to allow AI to take independent remedial action.

  • 89% say they have good visibility into the reasoning behind the outputs generated by AI solutions
  • 92% say they need to understand how a defensive AI tool makes decisions before they can trust it
  • Only 14% say they allow AI to act independently, performing autonomous actions without human approval
  • 74% say they are limiting the autonomy of AI taking action in their SOC until explainability improves

Given the desire for trust and explainability we are seeing from buyers, it's important for them to be equipped with the right questions to ask vendors during an assessment or POV of AI tools in order to demystify marketing hype from real operational outcomes.

Below is a list of categories in which buyers can assess AI vendors or AI Service Providers (AISPs) to help reach safe adoption and maximize their ROI.  

5 categories of AI vendor assessment

Darktrace groups these AI-related questions into 5 categories: governance, data and training, model and technique choice, performance validation, and interpretability and adjustability. By asking questions regarding each of these 5 categories, buyers can gain a deeper understanding of how an AISP’s systems work and whether they suit their business requirements.

Governance, safety, and data controls

Governance of AI systems is critical for all AISPs. Whether their platform is based around a single model, or is a more complex, composite AI solution, strong governance is essential to ensure the system is safe, robust, and reliable.

A simple question you could ask is:

What AI governance policies and frameworks do you follow, and/or certifications do you currently maintain?

For more questions you can ask vendors, download the full guide here.

Darktrace is certified to the ISO/IEC 42001 standard, the world’s first AI Management System (AIMS) standard. ISO/IEC 42001 addresses the unique ethical and technical challenges AI poses by setting out a structured way to manage risks such as transparency, accuracy, and misuse. This includes a commitment to ethical AI development, and effective management and monitoring of AI systems both prior to and continually after release.

Data gathering and training

Accurate, meaningful, and unbiased data gathering is the first important step in producing any AI system. An AI model trained using inaccurate, unbalanced, or poor-quality training data will fail to perform optimally.

To alleviate concerns regarding training data quality, a question you could ask is:

What steps do you take to prevent bias in your AI models and training data?

For more questions, download the full guide here.

AISPs should be able to provide information about the steps taken, workflows followed, and auditing performed to reduce AI bias where appropriate. While it’s sometimes impossible to fully remove bias from an AI model, appropriate actions should be taken to mitigate or reduce bias where relevant.

Model and technique choice

Different AI techniques are optimal for different tasks. For example, research from Gartner suggests that relying on a single “one-size-fits-all" model can lead to data gaps, especially in highly specialized domains.

To achieve more accurate and robust AI solutions, AI leaders should move beyond using just one model or technique, embrace composite AI practices, and adopt a holistic AI system perspective.

A straightforward question you could ask is simply:

What type(s) of AI model(s) do you utilize in your solution?

For more questions, download the full guide here.

While specific detailed information about custom systems used by AISPs is likely proprietary, buyers should expect vendors to be able to provide an overview of the broad techniques used. This will allow you as a buyer to determine if the type of model is appropriate for your use case.

Performance and accuracy validation  

Testing and evaluation of performance is essential for all AI systems. Performance analysis should be performed both before release and continually after release to identify potential data or model drift.  

A question you could ask to understand an AISPs testing workflow is:

How do you audit, test, evaluate, verify, and validate your AI model outputs?

For more questions, download the full guide here.

Testing workflows will likely vary depending on the type of model – measurements relevant to one system may not always be relevant to others. Assessment of systems should also extend beyond these standard accuracy and robustness tests, and should also feature physical performance, such as latency and resource consumption.  

Interpretability, adjustability, and transparency  

AI systems are typically a black box, simply providing an output without an explanation of how that output was attained. Interpretability and transparency are critical to ensure that both SOC teams and end-users trust the outputs of a system to be accurate and meaningful.

A question you could ask is:

How do you promote a trust relationship between human analysts and AI outputs?

For more questions, download the full guide here.

In the context of cybersecurity, trust and interpretability are even more essential. This is particularly relevant for generative AI-based systems (including most AI Agents), where the risk of hallucination can reduce trust in responses.

Cybersecurity systems often need to perform autonomous actions to block incoming threats – an email filtering system may hold potentially dangerous emails; a firewall may block malicious inbound connections. If SOC teams can’t trust these systems to perform accurately, these systems may be limited or disabled, critically reducing their defensive power.

Darktrace as an AI-native cybersecurity vendor

Darktrace has been building and applying AI in cybersecurity for over a decade, developing its capabilities alongside an increasingly complex and fast‑moving threat landscape. This experience has resulted in a mature, multi-layered approach to AI, which continuously learns the normal patterns of each organization to understand behavior, interpret context, and identify meaningful deviations — without relying on predefined rules or known attack signatures. Over time, this has enabled a proven behavioral understanding that helps uncover subtle signals of risk that may otherwise be missed.

With the backing of our ISO/IEC 42001 certification, stakeholders, customers, and partners can be confident that Darktrace is responsibly, ethically, and safely developing its AI systems, and managing the use of AI in day-to-day operations in a compliant and secure manner.  

Explore the principles behind Darktrace’s responsible AI approach, informed by collaboration with global experts in academia and governments, detailing how accountability, explainability, and continuous validation are built into its cybersecurity technology.

How Darktrace secures AI systems

Darktrace now brings these capabilities to monitor and respond to risk generated from AI systems across organizations with Darktrace / SECURE AI. This solution analyzes how prompts, agents, and systems are used within the context of each organization, bringing every AI interaction into a single view. This unique approach helps teams understand intent, assess risk, protect sensitive data, and enforce policy across both human and AI agent activity.

Stay up to date

Sign up for the Secure AI Readiness Program here: This gives you exclusive access to the latest news on the latest AI threats, updates on emerging approaches shaping AI security, and insights into the latest innovations, including Darktrace’s ongoing work in this area.

Ready to talk with a Darktrace expert on securing AI? Register here to receive practical guidance on the AI risks that matter most to your business, paired with clarity on where to focus first across governance, visibility, risk reduction, and long-term readiness.  

Further Reading on AI in cybersecurity

When deciding to invest in an AI solution, it’s important to understand what this means for you and your organization. The questions presented here are only a starting point in understanding an AI solution and whether it is appropriate for your use case.  

Gain deeper knowledge on applications of AI in cybersecurity and Darktrace’s multi-layered AI in the AI Arsenal White Paper.

[related-resource]

Continue reading
About the author
Jamie Bali
Technical Author (AI) Developer
Your data. Our AI.
Elevate your network security with Darktrace AI