Evaluating Email Security: How to Select the Best Solution for Your Organization
In today’s saturated market for email security, it can be difficult to cut through the noise of AI hype and vendor claims. CISOs should be using a structured evaluation framework to support informed, objective comparisons of different vendors – to allow them to make the best decision for their organization.
Darktrace cyber analysts are world-class experts in threat intelligence, threat hunting and incident response, and provide 24/7 SOC support to thousands of Darktrace customers around the globe. Inside the SOC is exclusively authored by these experts, providing analysis of cyber incidents and threat trends, based on real-world experience in the field.
Written by
Carlos Gray
Senior Product Marketing Manager, Email
Share
21
May 2025
When evaluating email security solutions, it’s crucial to move beyond marketing claims and focus on real-world performance. One of the most effective ways to achieve this is through an A/B comparison approach – a side-by-side evaluation of vendors based on consistent, predefined criteria.
This method cuts through biases, reveals true capability differences, and ensures that all solutions are assessed on a level playing field. It’s not just about finding an objectively good solution – it’s about finding the best solution for your organization’s specific needs.
An A/B comparison approach is particularly effective for three main reasons:
Eliminates bias: By comparing solutions under identical conditions, it’s easier to spot differences in performance without the fog of marketing jargon.
Highlights real capabilities: Direct side-by-side testing exposes genuine strengths and weaknesses, making it easier to judge which features are impactful versus merely decorative.
Encourages objective decision-making: This structured method reduces emotional or brand-driven decisions, focusing purely on metrics and performance.
Let’s look at the key factors to consider when setting up your evaluation to ensure a fair, accurate, and actionable comparison.
Deployment: Setting the stage for fair evaluation
To achieve a genuine comparison, deployment must be consistent across all evaluated solutions:
Establish the same scope: All solutions should be granted identical visibility across relevant tenants and domains to ensure parity.
Set a concrete timeline: Deploy and test each solution with the same dataset, at the same points in time. This allows you to observe differences in learning periods and adaptive capabilities.
Equal visibility and synchronized timelines prevent discrepancies that could skew your understanding of each vendor’s true capabilities. But remember – quicker results might not equal better learning or understanding!
Tuning and configurations: Optimizing for real-world conditions
Properly tuning and configuring each solution is critical for fair evaluation:
Compare on optimal performance: Consult with each vendor to understand what optimal deployment looks like for their solution, particularly if machine learning is involved.
Consider the long term: Configuration adjustments should be made with long-term usage in mind. Short-term fixes can mask long-term challenges.
Data visibility: Ensure each solution can retain and provide search capabilities on all data collected throughout the evaluation period.
These steps guarantee that you are comparing fully optimized versions of each platform, not underperforming or misconfigured ones.
Evaluation: Applying consistent metrics
Once deployment and configurations are aligned, the evaluation itself must be consistent, to prevent unfair scoring and help to identify true differences in threat detection and response capabilities.
Coordinate your decision criteria: Ensure all vendors are measured against the same set of criteria, established before testing begins.
Understand vendor threat classification: Each vendor may have different ways of classifying threats, so be sure to understand these nuances.
Maintain communication: If results seem inaccurate, engage with the vendors. Their response and remediation capabilities are part of the evaluation.
Making a decision: Look beyond the metrics
When it comes to reviewing the performance of each solution, it’s important to both consider and look beyond the raw data. This is about choosing the solution that best aligns with your specific business needs, which may include factors and features not captured in the results.
Evaluate based on results: Consider accuracy, threats detected, precision, and response effectiveness.
Evaluate beyond results: Assess the overall experience, including support, integrations, training, and long-term alignment with your security strategy.
Review and communicate: Internally review the findings and communicate them back to the vendors.
Choosing the right email security solution isn’t just about ticking boxes, it’s about strategic alignment with your organization’s goals and the evolving threat landscape. A structured, A/B comparison approach will help ensure that the solution you select is truly the best fit.
For a full checklist of the features and capabilities to compare, as well as how to perform a commercial and technical evaluation, check out the full Buyer’s Checklist for Evaluating Email Security.
[related-resource]
A Buyer’s Checklist for Evaluating Email Security
Get the most effective system for comparing email security solutions – based on real-world performance and measurable capabilities.
Darktrace cyber analysts are world-class experts in threat intelligence, threat hunting and incident response, and provide 24/7 SOC support to thousands of Darktrace customers around the globe. Inside the SOC is exclusively authored by these experts, providing analysis of cyber incidents and threat trends, based on real-world experience in the field.
Por qué la protección del Email basada en IA se volvió esencial para este íder mundial en servicios financieros
Hear the cybersecurity transformation story of this leading money transmitter, who facilitates more than $9 billion in remittances via thousands of agent locations across the US serving more than two million active customers.
Detecting and Mitigating Adversary-in-the-Middle Phishing Attacks with Darktrace Services
Threat actors often use advanced phishing toolkits and Adversary-in-the-Middle (AitM) attacks in Business Email Compromise (BEC) campaigns, Discover how Darktrace detected and mitigated a sophisticated attack leveraging Dropbox, highlighting the importance of robust cybersecurity measures.
Bytesize Security: Insider Threats in Google Workspace
What is an insider threat?
An insider threat is a cyber risk originating from within an organization. These threats can involve actions such as an employee inadvertently clicking on a malicious link (e.g., a phishing email) or an employee with malicious intent conducting data exfiltration for corporate sabotage.
Insiders often exploit their knowledge and access to legitimate corporate tools, presenting a continuous risk to organizations. Defenders must protect their digital estate against threats from both within and outside the organization.
For example, in the summer of 2024, Darktrace / IDENTITY successfully detected a user in a customer environment attempting to steal sensitive data from a trusted Google Workspace service. Despite the use of a legitimate and compliant corporate tool, Darktrace identified anomalies in the user’s behavior that indicated malicious intent.
Attack overview: Insider threat
In June 2024, Darktrace detected unusual activity involving the Software-as-a-Service (SaaS) account of a former employee from a customer organization. This individual, who had recently left the company, was observed downloading a significant amount of data in the form of a “.INDD” file (an Adobe InDesign document typically used to create page layouts [1]) from Google Drive.
While the use of Google Drive and other Google Workspace platforms was not unexpected for this employee, Darktrace identified that the user had logged in from an unfamiliar and suspicious IPv6 address before initiating the download. This anomaly triggered a model alert in Darktrace / IDENTITY, flagging the activity as potentially malicious.
Figure 1: A Model Alert in Darktrace / IDENTITY showing the unusual “.INDD” file being downloaded from Google Workspace.
Following this detection, the customer reached out to Darktrace’s Security Operations Center (SOC) team via the Security Operations Support service for assistance in triaging and investigating the incident further. Darktrace’s SOC team conducted an in-depth investigation, enabling the customer to identify the exact moment of the file download, as well as the contents of the stolen documents. The customer later confirmed that the downloaded files contained sensitive corporate data, including customer details and payment information, likely intended for reuse or sharing with a new employer.
In this particular instance, Darktrace’s Autonomous Response capability was not active, allowing the malicious insider to successfully exfiltrate the files. If Autonomous Response had been enabled, Darktrace would have immediately acted upon detecting the login from an unusual (in this case 100% rare) location by logging out and disabling the SaaS user. This would have provided the customer with the necessary time to review the activity and verify whether the user was authorized to access their SaaS environments.
Conclusion
Insider threats pose a significant challenge for traditional security tools as they involve internal users who are expected to access SaaS platforms. These insiders have preexisting knowledge of the environment, sensitive data, and how to make their activities appear normal, as seen in this case with the use of Google Workspace. This familiarity allows them to avoid having to use more easily detectable intrusion methods like phishing campaigns.
Darktrace’s anomaly detection capabilities, which focus on identifying unusual activity rather than relying on specific rules and signatures, enable it to effectively detect deviations from a user’s expected behavior. For instance, an unusual login from a new location, as in this example, can be flagged even if the subsequent malicious activity appears innocuous due to the use of a trusted application like Google Drive.
Credit to Vivek Rajan (Cyber Analyst) and Ryan Traill (Analyst Content Lead)
Get the latest insights on emerging cyber threats
Attackers are adapting, are you ready? This report explores the latest trends shaping the cybersecurity landscape and what defenders need to know in 2025.
Identity-based attacks: How attackers are bypassing traditional defenses
Zero-day exploitation: The rise of previously unknown vulnerabilities
AI-driven threats: How adversaries are leveraging AI to outmaneuver security controls
RansomHub Ransomware: investigación de Darktrace sobre la herramienta más nueva en ShadowSyndicate's Arsenal
What is ShadowSyndicate?
ShadowSyndicate, also known as Infra Storm, is a threat actor reportedly active since July 2022, working with various ransomware groups and affiliates of ransomware programs, such as Quantum, Nokoyawa, and ALPHV. This threat actor employs tools like Cobalt Strike, Sliver, IcedID, and Matanbuchus malware in its attacks. ShadowSyndicate utilizes the same SSH fingerprint (1ca4cbac895fc3bd12417b77fc6ed31d) on many of their servers—85 as of September 2023. At least 52 of these servers have been linked to the Cobalt Strike command and control (C2) framework [1].
What is RansomHub?
First observed following the FBI's takedown of ALPHV/BlackCat in December 2023, RansomHub quickly gained notoriety as a Ransomware-as-a-Service (RaaS) operator. RansomHub capitalized on the law enforcement’s disruption of the LockBit group’s operations in February 2024 to market themselves to potential affiliates who had previously relied on LockBit’s encryptors. RansomHub's success can be largely attributed to their aggressive recruitment on underground forums, leading to the absorption of ex-ALPHV and ex-LockBit affiliates. They were one of the most active ransomware operators in 2024, with approximately 500 victims reported since February, according to their Dedicated Leak Site (DLS) [2].
ShadowSyndicate and RansomHub
External researchers have reported that ShadowSyndicate had as many as seven different ransomware families in their arsenal between July 2022, and September 2023. Now, ShadowSyndicate appears to have added RansomHub’s their formidable stockpile, becoming an affiliate of the RaaS provider [1].
Darktrace’s analysis of ShadowSyndicate across its customer base indicates that the group has been leveraging RansomHub ransomware in multiple attacks in September and October 2024. ShadowSyndicate likely shifted to using RansomHub due to the lucrative rates offered by this RaaS provider, with affiliates receiving up to 90% of the ransom—significantly higher than the general market rate of 70-80% [3].
In many instances where encryption was observed, ransom notes with the naming pattern “README_[a-zA-Z0-9]{6}.txt” were written to affected devices. The content of these ransom notes threatened to release stolen confidential data via RansomHub’s DLS unless a ransom was paid. During these attacks, data exfiltration activity to external endpoints using the SSH protocol was observed. The external endpoints to which the data was transferred were found to coincide with servers previously associated with ShadowSyndicate activity.
Darktrace’s coverage of ShadowSyndicate and RansomHub
Darktrace’s Threat Research team identified high-confidence indicators of compromise (IoCs) linked to the ShadowSyndicate group deploying RansomHub. The investigation revealed four separate incidents impacting Darktrace customers across various sectors, including education, manufacturing, and social services. In the investigated cases, multiple stages of the kill chain were observed, starting with initial internal reconnaissance and leading to eventual file encryption and data exfiltration.
Attack Overview
Internal Reconnaissance
The first observed stage of ShadowSyndicate attacks involved devices making multiple internal connection attempts to other internal devices over key ports, suggesting network scanning and enumeration activity. In this initial phase of the attack, the threat actor gathers critical details and information by scanning the network for open ports that might be potentially exploitable. In cases observed by Darktrace affected devices were typically seen attempting to connect to other internal locations over TCP ports including 22, 445 and 3389.
C2 Communication and Data Exfiltration
In most of the RansomHub cases investigated by Darktrace, unusual connections to endpoints associated with Splashtop, a remote desktop access software, were observed briefly before outbound SSH connections were identified.
Following this, Darktrace detected outbound SSH connections to the external IP address 46.161.27[.]151 using WinSCP, an open-source SSH client for Windows used for secure file transfer. The Cybersecurity and Infrastructure Security Agency (CISA) identified this IP address as malicious and associated it with ShadowSyndicate’s C2 infrastructure [4]. During connections to this IP, multiple gigabytes of data were exfiltrated from customer networks via SSH.
Data exfiltration attempts were consistent across investigated cases; however, the method of egress varied from one attack to another, as one would expect with a RaaS strain being employed by different affiliates. In addition to transfers to ShadowSyndicate’s infrastructure, threat actors were also observed transferring data to the cloud storage and file transfer service, MEGA, via HTTP connections using the ‘rclone’ user agent – a command-line program used to manage files on cloud storage. In another case, data exfiltration activity occurred over port 443, utilizing SSL connections.
Lateral Movement
In investigated incidents, lateral movement activity began shortly after C2 communications were established. In one case, Darktrace identified the unusual use of a new administrative credential which was quickly followed up with multiple suspicious executable file writes to other internal devices on the network.
The filenames for this executable followed the regex naming convention “[a-zA-Z]{6}.exe”, with two observed examples being “bWqQUx.exe” and “sdtMfs.exe”.
Figure 1: Cyber AI Analyst Investigation Process for the SMB Writes of Suspicious Files to Multiple Devices' incident.
Additionally, script files such as “Defeat-Defender2.bat”, “Share.bat”, and “def.bat” were also seen written over SMB, suggesting that threat actors were trying to evade network defenses and detection by antivirus software like Microsoft Defender.
File Encryption
Among the three cases where file encryption activity was observed, file names were changed by adding an extension following the regex format “.[a-zA-Z0-9]{6}”. Ransom notes with a similar naming convention, “README_[a-zA-Z0-9]{6}.txt”, were written to each share. While the content of the ransom notes differed slightly in each case, most contained similar text. Clear indicators in the body of the ransom notes pointed to the use of RansomHub ransomware in these attacks. As is increasingly the case, threat actors employed double extortion tactics, threatening to leak confidential data if the ransom was not paid. Like most ransomware, RansomHub included TOR site links for communication between its "customer service team" and the target.
Figure 2: The graph shows the behavior of a device with encryption activity, using the “SMB Sustained Mimetype Conversion” and “Unusual Activity Events” metrics over three weeks.
Since Darktrace’s Autonomous Response capability was not enabled during the compromise, the ransomware attack succeeded in its objective. However, Darktrace’s Cyber AI Analyst provided comprehensive coverage of the kill chain, enabling the customer to quickly identify affected devices and initiate remediation.
Figure 3: Cyber AI Analyst panel showing the critical incidents of the affected device from one of the cases investigated.
In lieu of Autonomous Response being active on the networks, Darktrace was able to suggest a variety of manual response actions intended to contain the compromise and prevent further malicious activity. Had Autonomous Response been enabled at the time of the attack, these actions would have been quickly applied without any human interaction, potentially halting the ransomware attack earlier in the kill chain.
Figure 4: A list of suggested Autonomous Response actions on the affected devices."
Conclusion
The Darktrace Threat Research team has noted a surge in attacks by the ShadowSyndicate group using RansomHub’s RaaS of late. RaaS has become increasingly popular across the threat landscape due to its ease of access to malware and script execution. As more individual threat actors adopt RaaS, security teams are struggling to defend against the increasing number of opportunistic attacks.
For customers subscribed to Darktrace’s Security Operations Center (SOC) services, the Analyst team promptly investigated detections of the aforementioned unusual and anomalous activities in the initial infection phases. Multiple alerts were raised via Darktrace’s Managed Threat Detection to warn customers of active ransomware incidents. By emphasizing anomaly-based detection and response, Darktrace can effectively identify devices affected by ransomware and take action against emerging activity, minimizing disruption and impact on customer networks.
Credit to Kwa Qing Hong (Senior Cyber Analyst and Deputy Analyst Team Lead, Singapore) and Signe Zahark (Principal Cyber Analyst, Japan)
.jpg)
