Evaluating Email Security: How to Select the Best Solution for Your Organization
In today’s saturated market for email security, it can be difficult to cut through the noise of AI hype and vendor claims. CISOs should be using a structured evaluation framework to support informed, objective comparisons of different vendors – to allow them to make the best decision for their organization.
Darktrace cyber analysts are world-class experts in threat intelligence, threat hunting and incident response, and provide 24/7 SOC support to thousands of Darktrace customers around the globe. Inside the SOC is exclusively authored by these experts, providing analysis of cyber incidents and threat trends, based on real-world experience in the field.
Written by
Carlos Gray
Senior Product Marketing Manager, Email
Share
21
May 2025
When evaluating email security solutions, it’s crucial to move beyond marketing claims and focus on real-world performance. One of the most effective ways to achieve this is through an A/B comparison approach – a side-by-side evaluation of vendors based on consistent, predefined criteria.
This method cuts through biases, reveals true capability differences, and ensures that all solutions are assessed on a level playing field. It’s not just about finding an objectively good solution – it’s about finding the best solution for your organization’s specific needs.
An A/B comparison approach is particularly effective for three main reasons:
Eliminates bias: By comparing solutions under identical conditions, it’s easier to spot differences in performance without the fog of marketing jargon.
Highlights real capabilities: Direct side-by-side testing exposes genuine strengths and weaknesses, making it easier to judge which features are impactful versus merely decorative.
Encourages objective decision-making: This structured method reduces emotional or brand-driven decisions, focusing purely on metrics and performance.
Let’s look at the key factors to consider when setting up your evaluation to ensure a fair, accurate, and actionable comparison.
Deployment: Setting the stage for fair evaluation
To achieve a genuine comparison, deployment must be consistent across all evaluated solutions:
Establish the same scope: All solutions should be granted identical visibility across relevant tenants and domains to ensure parity.
Set a concrete timeline: Deploy and test each solution with the same dataset, at the same points in time. This allows you to observe differences in learning periods and adaptive capabilities.
Equal visibility and synchronized timelines prevent discrepancies that could skew your understanding of each vendor’s true capabilities. But remember – quicker results might not equal better learning or understanding!
Tuning and configurations: Optimizing for real-world conditions
Properly tuning and configuring each solution is critical for fair evaluation:
Compare on optimal performance: Consult with each vendor to understand what optimal deployment looks like for their solution, particularly if machine learning is involved.
Consider the long term: Configuration adjustments should be made with long-term usage in mind. Short-term fixes can mask long-term challenges.
Data visibility: Ensure each solution can retain and provide search capabilities on all data collected throughout the evaluation period.
These steps guarantee that you are comparing fully optimized versions of each platform, not underperforming or misconfigured ones.
Evaluation: Applying consistent metrics
Once deployment and configurations are aligned, the evaluation itself must be consistent, to prevent unfair scoring and help to identify true differences in threat detection and response capabilities.
Coordinate your decision criteria: Ensure all vendors are measured against the same set of criteria, established before testing begins.
Understand vendor threat classification: Each vendor may have different ways of classifying threats, so be sure to understand these nuances.
Maintain communication: If results seem inaccurate, engage with the vendors. Their response and remediation capabilities are part of the evaluation.
Making a decision: Look beyond the metrics
When it comes to reviewing the performance of each solution, it’s important to both consider and look beyond the raw data. This is about choosing the solution that best aligns with your specific business needs, which may include factors and features not captured in the results.
Evaluate based on results: Consider accuracy, threats detected, precision, and response effectiveness.
Evaluate beyond results: Assess the overall experience, including support, integrations, training, and long-term alignment with your security strategy.
Review and communicate: Internally review the findings and communicate them back to the vendors.
Choosing the right email security solution isn’t just about ticking boxes, it’s about strategic alignment with your organization’s goals and the evolving threat landscape. A structured, A/B comparison approach will help ensure that the solution you select is truly the best fit.
For a full checklist of the features and capabilities to compare, as well as how to perform a commercial and technical evaluation, check out the full Buyer’s Checklist for Evaluating Email Security.
[related-resource]
A Buyer’s Checklist for Evaluating Email Security
Get the most effective system for comparing email security solutions – based on real-world performance and measurable capabilities.
Darktrace cyber analysts are world-class experts in threat intelligence, threat hunting and incident response, and provide 24/7 SOC support to thousands of Darktrace customers around the globe. Inside the SOC is exclusively authored by these experts, providing analysis of cyber incidents and threat trends, based on real-world experience in the field.
Darktrace / EMAIL Expands Behavioral Defense Across Email and Collaboration Workflows
New capabilities help employees understand suspicious content, give administrators more control over sensitive data, and show security teams where risky workflows sit across the organization.
From One Alert to a Wider Campaign: Uncovering Novel Threats Through Behavioral Analysis
Key Findings
Darktrace’s Threat Research team is exploring a new method to identify malicious campaigns by correlating similar anomalous activity across its customer fleet.
The method uses unsupervised clustering to identify shared behavioral patterns across multiple customer deployments without relying on prior indicators of compromise (IoCs).
The approach identified shared infrastructure across dozens of customers, with activity linked to the likely POPA botnet.
A second cluster identified anomalous activity associated with Monero cryptocurrency mining.
How does Darktrace conduct threat research?
The Darktrace Threat Research team aims to identify malicious campaign activity affecting multiple customers. By their nature, these campaigns will lead to shared behavioral patterns across multiple victims.
In this piece, we describe a method for identifying malicious campaign activity that uses this simple observation as a guiding principle. We begin by outlining a few key facts about the modern threat landscape and why behavioral detection is necessary to meet the challenges it presents. We then explore how fleet-wide behavioral detections can be leveraged using the principle above to systematically identify multi-deployment campaigns impacting the Darktrace customer fleet. Finally, we demonstrate that this research is not purely theoretical, using real examples of campaign activity detected through this novel approach.
Shifting to a behavioral-focused approach
The modern era of cyber-defense necessitates detection capabilities beyond traditional signature-based methods. Technologies such as antivirus signatures, hash-based malware identification, and IP blocklists have historically been successful in defending against known threats. However, they are easily circumvented by attackers and inherently limited in defending against the novel attack techniques that today's threat landscape is producing.
Recent breakthroughs in AI represent a transformational risk multiplier that cannot be ignored. AI systems do not merely execute existing attacks at machine speed; there are signs they can also synthesize novel tradecraft [1], increasing the likelihood that defenders will encounter attack patterns that have never previously been observed. Meanwhile vulnerability disclosures continue to accelerate. More than 48,000 CVEs were published in 2025, and published statistics to date indicate that 2026 is already on track to significantly exceed that figure [2]. While multiple factors may contribute to vulnerability discovery, AI is seen as a key factor in this rapid growth. Organizations should expect a growing proportion of malicious activity to fall outside the scope of detections that rely solely on prior knowledge of known threats.
By design, Darktrace’s behavioral and anomaly-focused detection capabilities are well-suited to this testing landscape. With a focus on detecting unknown and unexpected activity that deviates from the baseline of “normal” behavior, detection of CVE exploitation without prior knowledge of the vulnerability and before public disclosure, for example, is achievable [3][4].
Darktrace’s diverse customer base benefits from a further layer of protection: working with only deidentified alert metadata, our threat researchers hunt proactively across the fleet. Beyond the level of the individual customer, there is a second, larger vantage to be leveraged.
Darktrace data provides unique insight into the unexpected behavioral patterns that occur across its customer fleet. What can we see if we look at this fleet-wide telemetry and hunt for systematic behavioral campaigns that impact our customer base?
Correlating across the customer base
At the deployment level, Darktrace focuses on whether activity is unusual in the context of an organization's normal behavior, but as we zoom out and look at these alerts at the fleet level, the problem fundamentally changes . We are no longer seeking anomalies in largely normal data; we are seeking systematic patterns within data already known to be anomalous. An isolated behavior anomaly may be notable for a single customer; the same fingerprint across dozens is a signal of a structured campaign. Hunting for such campaigns and sharing our findings are key areas of focus for Darktrace’s Threat Research team. In this spirit, Darktrace is actively exploring a a new research approach that leverages deidentified alert metadata to increase campaign detection ability.
This approach is based on the premise that an expert human analyst presented with a set of alerts will identify likely related activity by assessing whether the metadata fields indicate a relationship. For example, two connections to a rare endpoint that exhibits naming structure that is likely from a single algorithm are a strong indicator, as is the use of an unusual port carrying a given protocol (e.g. port 6000 for SSL). Darktrace’s approach to campaign detection encodes this expert human judgement process through deterministic embedding functions that obtain reproducible vector representations of alert data. Unsupervised clustering is then applied to these vector representations to obtain groups of alerts that describe semantically similar alert data across multiple customers. These clusters therefore represent similar anomalous activity that has occurred across multiple customers. In other words, these clusters are candidates for malicious campaign activity.
Figure 1: Layers of AI-based analysis provided through Darktrace products and cross-customer correlation.
Case study 1 – unusual connections to shared underlying infrastructure
In one recent case, the clustering produced several tight groups of alerts spanning dozens of customers. Each group centered on a single domain whose subdomains were algorithmically generated (e.g. s01699, s1736, s91). Examined side by side, these clusters shared an unmistakable behavioral shape: each involved a domain with a clear naming pattern (e.g. s016841786.net-echo[.]com, s1670.pixellog[.]io, s217.pixellog[.]io) , with connections over SSL on the unusual port 6000 to servers hosted by non-mainstream providers (e.g. OVH, Hetzner, GTHost, Leaseweb), rather than generally expected content-delivery networks.
Figure 2: An example model alert triggering on unusual outbound connections and data transfer to s1670.pixellog[.]io.
Closer inspection revealed that the same subdomain label appeared under different domains (e.g. s1786.net-echo[.]com and s1786.pixellog[.]io). Following this lead, Darktrace’s threat researchers pivoted across the customer base and found numerous distinct subdomain labels reused under more than one registered domain. In several cases, the same label resolved to the same IP across different domains (e.g. s1704 under both pixellog[.]io and net-echo[.]com → 148.113.222[.]244; s01684 → 15.235.67[.]95), suggesting the use of ephemeral subdomain infrastructure in an attempt to avoid signature-based defense techniques.
This example demonstrates how multi-deployment data can uncover shared infrastructure that may be difficult to identify at the individual deployment level alone. In particular, the presented method facilitated this discovery by producing coherent, tight clusters of geometrically similar vector embeddings of malicious activity from large-scale fleet data, which could then be presented to a researcher for further investigation. This activity highlighted across the Darktrace customer base was ultimately linked to likely POPA botnet activity through use of open-source intelligence (OSINT) [5].
Case Study 2 – anomalous user agent and txt URI
A second example shows the method surfacing a single, self-contained cluster. The clustering was not given any indicators to look for; instead, it grouped devices that shared an unusual combination of features. On inspection, that shared consisted of:
- unusual user agent (tes)
- specific request path (/md5_GIP.txt)
- uncommon destination port (83)
- hostname impersonating Google (matching regular expression ^www\.google\.com\.[a-zA-Z0-9]+\.fast-dns-host\.com$)
Treating these shared features as IoCs, Darktrace’s threat researchers then pivoted to an exploration across the customer base to investigate the full breadth of the campaign. This search identified further commonalities; additional domains which shared the same IP as the domain family highlighted in the cluster, as well as evidence of cryptocurrency mining on affected devices. Additional OSINT research also aligned with these findings, linking the discovered payloads and infrastructure to Monero cryptocurrency mining [6].
Conclusion
The method outlined in this blog is under active development, with ongoing technical refinement providing opportunities to further enhance its capabilities. While the capability continues to mature, this approach to threat hunting leverages the powerful vantage point of fleetwide telemetry across a diverse customer base to help protect Darktrace customers. This method allows campaigns that at the deployment level appear as isolated alerts to be seen at their true scale and spread. It enables proactive fleet-wide analysis without prior IoCs to identify campaigns impacting Darktrace customers.
This is possible because of what the method is built on: the atomic unit is the Darktrace alert, a signal that encodes years of cyber expertise, while the multi-deployment nature of the method allows organizations to indirectly benefit from findings from other networks through shared intelligence by Darktrace’s Threat Research team.
Credit to Daniel Levy (Threat Hunting Data Scientist), Paul Jennings (Principal Cyber Analyst), Emma Foulger (Global Head, Threat Operations)
“I wonder if running that tool influenced the connectivity in some way.”
“I might need to target down or reboot, but first, I'll investigate the network using ARP. I wonder if ARP is responding properly. If I see "ip neigh stale," that could indicate a problem. I think arping might help in this situation, but I’ll need to keep an eye on how the network is behaving. It's all about figuring out where the issue lies!”
図10：エージェントの異常なLDAPアクティビティに対してトリガーされたこのLinuxサーバーに対するモデル検知は、Cyber AI Analystによって自律的に調査され、その調査結果にはエージェントの異常なSMBセッションアクティビティに関する内容が含まれ、Cyber AIAnalystインシデントに記録されました。
DarktraceのCyber AI Analystは、エージェントのアクティビティによってトリガーされたモデル検知を自律的に調査し、その調査結果を1つの重要なインシデントレポートにまとめました。インシデント報告書の内容は、エージェントによるネットワークスキャンから始まりコーディング演習内容のSMBベースの上書きで終わる、エージェントの行動の各段階を詳しくカバーしています。
図20：Cyber AI Analystは、モデル検知に対する自律的な調査を通じて得られた所見をつなぎ合わせ、エージェントの進行状況のタイムラインを作成し、アクティビティを視覚的に表したグラフとともに提供しました。
Cyber Verification Program下のAnthropic Opus 4.6モデルなど、他のフロンティアモデルを使用したエージェントも、Darktraceのシミュレートされた環境で同じ設定とタスクを使ってテストされました。反復の度に、エージェントは演習の内容または採点プログラムを変更して不正を行おうとしました。基盤となるモデルに関係なく、エージェントは自分が置かれた環境を侵害することで、100％のスコアを達成する方法を見つけることに成功していました。これらのエージェントによるハッキングは、アクティビティの内容こそ違っていましたが、すべてその環境のベースラインの振る舞いから大きく逸脱した動作を引き起こしていました。これらの逸脱はDarktraceプラットフォームに十分な検知の機会を提供し、エージェントの逃げ場はありませんでした。
.avif)
.avif)
