Evaluating Email Security: How to Select the Best Solution for Your Organization
In today’s saturated market for email security, it can be difficult to cut through the noise of AI hype and vendor claims. CISOs should be using a structured evaluation framework to support informed, objective comparisons of different vendors – to allow them to make the best decision for their organization.
Darktrace cyber analysts are world-class experts in threat intelligence, threat hunting and incident response, and provide 24/7 SOC support to thousands of Darktrace customers around the globe. Inside the SOC is exclusively authored by these experts, providing analysis of cyber incidents and threat trends, based on real-world experience in the field.
Written by
Carlos Gray
Senior Product Marketing Manager, Email
Share
21
May 2025
When evaluating email security solutions, it’s crucial to move beyond marketing claims and focus on real-world performance. One of the most effective ways to achieve this is through an A/B comparison approach – a side-by-side evaluation of vendors based on consistent, predefined criteria.
This method cuts through biases, reveals true capability differences, and ensures that all solutions are assessed on a level playing field. It’s not just about finding an objectively good solution – it’s about finding the best solution for your organization’s specific needs.
An A/B comparison approach is particularly effective for three main reasons:
Eliminates bias: By comparing solutions under identical conditions, it’s easier to spot differences in performance without the fog of marketing jargon.
Highlights real capabilities: Direct side-by-side testing exposes genuine strengths and weaknesses, making it easier to judge which features are impactful versus merely decorative.
Encourages objective decision-making: This structured method reduces emotional or brand-driven decisions, focusing purely on metrics and performance.
Let’s look at the key factors to consider when setting up your evaluation to ensure a fair, accurate, and actionable comparison.
Deployment: Setting the stage for fair evaluation
To achieve a genuine comparison, deployment must be consistent across all evaluated solutions:
Establish the same scope: All solutions should be granted identical visibility across relevant tenants and domains to ensure parity.
Set a concrete timeline: Deploy and test each solution with the same dataset, at the same points in time. This allows you to observe differences in learning periods and adaptive capabilities.
Equal visibility and synchronized timelines prevent discrepancies that could skew your understanding of each vendor’s true capabilities. But remember – quicker results might not equal better learning or understanding!
Tuning and configurations: Optimizing for real-world conditions
Properly tuning and configuring each solution is critical for fair evaluation:
Compare on optimal performance: Consult with each vendor to understand what optimal deployment looks like for their solution, particularly if machine learning is involved.
Consider the long term: Configuration adjustments should be made with long-term usage in mind. Short-term fixes can mask long-term challenges.
Data visibility: Ensure each solution can retain and provide search capabilities on all data collected throughout the evaluation period.
These steps guarantee that you are comparing fully optimized versions of each platform, not underperforming or misconfigured ones.
Evaluation: Applying consistent metrics
Once deployment and configurations are aligned, the evaluation itself must be consistent, to prevent unfair scoring and help to identify true differences in threat detection and response capabilities.
Coordinate your decision criteria: Ensure all vendors are measured against the same set of criteria, established before testing begins.
Understand vendor threat classification: Each vendor may have different ways of classifying threats, so be sure to understand these nuances.
Maintain communication: If results seem inaccurate, engage with the vendors. Their response and remediation capabilities are part of the evaluation.
Making a decision: Look beyond the metrics
When it comes to reviewing the performance of each solution, it’s important to both consider and look beyond the raw data. This is about choosing the solution that best aligns with your specific business needs, which may include factors and features not captured in the results.
Evaluate based on results: Consider accuracy, threats detected, precision, and response effectiveness.
Evaluate beyond results: Assess the overall experience, including support, integrations, training, and long-term alignment with your security strategy.
Review and communicate: Internally review the findings and communicate them back to the vendors.
Choosing the right email security solution isn’t just about ticking boxes, it’s about strategic alignment with your organization’s goals and the evolving threat landscape. A structured, A/B comparison approach will help ensure that the solution you select is truly the best fit.
For a full checklist of the features and capabilities to compare, as well as how to perform a commercial and technical evaluation, check out the full Buyer’s Checklist for Evaluating Email Security.
[related-resource]
A Buyer’s Checklist for Evaluating Email Security
Get the most effective system for comparing email security solutions – based on real-world performance and measurable capabilities.
Darktrace cyber analysts are world-class experts in threat intelligence, threat hunting and incident response, and provide 24/7 SOC support to thousands of Darktrace customers around the globe. Inside the SOC is exclusively authored by these experts, providing analysis of cyber incidents and threat trends, based on real-world experience in the field.
Journey of a Threat: How Multi-Layered AI Works in Darktrace / EMAIL
Follow a malicious email as it moves through Darktrace / EMAIL’s multi-layered AI system, from raw data to final decision. Each layer works together to detect threats, understand intent, and take autonomous action.
How email-delivered prompt injection attacks can target enterprise AI – and why it matters
Prompt injection is a newly emerging threat, with only a handful of confirmed victims so far – targeting how AI systems use data rather than exploiting traditional software vulnerabilities. As agentic AI becomes embedded across enterprise environments, attackers may attempt to manipulate these systems through hidden instructions in everyday email content.
Darktrace Unites Human Behavior and Threat Detection Across Email, Slack, Teams, and Zoom
Introducing the adaptive era of email security: a unified platform that connects personalized coaching, collaboration tools, and user behavior into a self-improving defense system.
Hola VPN Abuse: From Proxy Traffic to Malware and Cryptomining
Introduction
In enterprise environments, non-compliant software traffic can introduce unexpected exposure by creating unmanaged paths for outbound connectivity. Hola VPN is a notable example because of its peer-to-peer design, which can effectively turn user devices into routing or exit nodes for other parties’ traffic, shifting the risk profile from that of a traditional virtual private network (VPN) to something closer to a distributed proxy.
As a result, the appearance of Hola-related activity, whether from prior installation or unintended background connections, should be treated with caution. Such activity may provide a foothold for malicious behavior, including lateral movement or command-and-control communication.
This blog explores how Hola-associated activity appeared as part of broader patterns of suspicious behavior observed across the Darktrace customer base.
The campaign
In February and March 2026, Darktrace observed similar anomalous activity across multiple customer environments, with affected devices showing consistent behavioral patterns. These included connections to multiple *.hola[.]org endpoints using Hola-related user agents, suggesting interaction with Hola infrastructure rather than isolated or incidental traffic.
Following these connections, affected customer environments showed downloads of suspicious executable files from rare external endpoints 188.241.219[.]55 and 184.241.218[.]111. Both endpoints have been flagged as potentially malicious by open-source intelligence (OSINT) [1][2].
These downloads were conducted using consistent user agents across impacted customers, specifically ‘Hola svc_js_win32/1.249.408’ and ‘Hola svc_js_win32/1.251.389’, suggesting a possible association with Hola-related activity.
Notably, this pattern aligns with recent reporting that, in some cases, Hola distributed an undeclared executable component, me[.]exe, which was later assessed to be a likely Monero-mining binary introduced via a compromised delivery pipeline [3].
Case Study 1
Darktrace first observed a new device on January 19, 2026, within a customer environment based in the Europe, Middle East, and Africa (EMEA) region. On the same day it appeared on the network, the device communicated with multiple pieces of Hola VPN-linked infrastructure before downloading a binary from a hola[.]org subdomain.
Figure 1: Cyber AI Analyst investigation highlighting Hola VPN service activity potentially associated with subsequent HTTP command-and-control (C2) connections.
Subsequent Darktrace telemetry revealed a recurring pattern of activity from the day the device was first observed through to March 4, 2026. During this period, the device repeatedly issued HTTP GET requests to the URI /bwfile?size=1048576, each returning a 200 OK response, indicating successful file retrieval.
This behavior was accompanied by a POST request to /bwfile, followed by an additional GET request for a significantly larger file at /bwfile?size=26214400, suggesting a deliberate and structured file transfer pattern.
Notably, the binary download activity was not tied to a single static host. Instead, it was observed across multiple URLs that changed over time while remaining within the same hola[.]org domain. This pattern suggests the use of rotating or distributed delivery infrastructure rather than a fixed endpoint.
Figure 2: Variation in URLs over time within the same hola[.]org domain, indicating the use of dynamically changing endpoints.
Across these events, the activity was consistently associated with the user agent Hola svc_js_win32/1.249.408, further linking the traffic to Hola-related service components. Amid these persistent and unusual connections, on February 22, Darktrace observed the device connecting to 188.241.219[.]55/proxy-peer-windows-amd64[.]exe, resulting in the download of an executable file.
Figure 3: File transfer event showing the download of an executable from the rare external endpoint 188.241.219[.]55.
Based on its file hash, the downloaded file was assessed as a likely Trojan downloader [4], with import hash (imphash) values showing similarities to samples linked to Vidar, Rhadamanthys, and Stealc according to OSINT [5]. Overall, this sequence of activity suggests that Hola-related connectivity may have been leveraged as part of a broader malware delivery chain.
Darktrace’s Autonomous Response
Due to the highly unusual activity observed, Darktrace Autonomous Response was triggered by the device’s behavior. However, as the customer deployment was configured in “Human Confirmation” mode, manual approval was required before any action could be taken.
Had the deployment been set to “Fully Autonomous” mode, Darktrace would have automatically:
Blocked connections to the associated ports and external endpoints
Prevented all outgoing network connections from the device
Enforced the device’s established ‘pattern of life’, allowing normal activity to continue while restricting any anomalous behavior
Figure 4: Example of a Darktrace Autonomous Response model highlighting the action that would have been taken, demonstrating how the system identifies anomalous behavior and applies targeted containment measures to restrict suspicious network activity.
Case Study 2
While the first case focused on anomalous activity from a newly observed device, Darktrace also identified cases in which devices had already been communicating with Hola-related endpoints prior to the suspected campaign. This may suggest pre-existing Hola usage within the environment, potentially increasing exposure and creating an avenue for subsequent suspicious activity.
One case involved three devices within a customer network based in the Americas (AMS). In this instance, a different payload was identified: me[.]exe, a potentially malicious cryptocurrency miner also referred to as HolaMonitorService[.]exe [6][7]. The downloads were observed from infrastructure similar to that seen in Case 1, including an IP address within the same 188.241.0.0/16 subnet.
Connections to *.hola[.]org, alongside the use of potential Hola-related user agents consistent with those in Case 1, were also identified, further suggesting a link between the observed activity and Hola-associated infrastructure.
Darktrace observed activity indicative of unusual VPN usage on the first affected device on February 2, followed by telemetry suggesting potential Tor usage. This was later followed by the download of me[.]exe on March 10 from 188.241.218[.]111. Notably, this device was the earliest among the three within the deployment to exhibit the presence of the suspicious executable.
Figure 5: Cyber AI Analyst detection highlighting the download of a suspicious executable from a similar external endpoint in a separate deployment.
On March 5, 2026, the second affected device exhibited a slightly different progression, initiating connections to http-test1[.]hola[.]org using the user agent ‘hola_get’. This activity was followed by the download of me[.]exe from the same endpoint on March 13, consistent with the broader pattern of Hola-related downloads observed across the environment.
Figure 6: Example of Hola VPN-related connectivity observed on the network prior to the suspected campaign, indicating pre-existing usage that may have contributed to subsequent activity.
The final affected device within this customer’s network demonstrated a more limited but related pattern, also downloading me[.]exe on March 17 using the same ‘hola_get’ user agent.
While the earlier Hola VPN usage observed across the deployment may not have been directly related to the suspected malware campaign, it may nonetheless have contributed to reduced visibility. The presence of pre-existing Hola-related traffic could have obscured malicious activity, making it more difficult to distinguish legitimate usage from attacker-driven behavior and, in turn, hindering the timely identification of the emerging compromise.
Darktrace’s Autonomous Response
For this deployment, the customer had their Autonomous Response capability configured in “Fully Autonomous” mode, allowing Darktrace to take action without human intervention. As a result, the system was able to autonomously disrupt the activity as soon as relevant events were identified through model detections.
Figure 7: Darktrace Autonomous Response actions taken against suspicious activity linked to Hola VPN.
Suspected cryptomining activity
As previously noted, some of the observed executable payloads appear to be linked to cryptomining malware. Across a subset of affected customer environments, this assessment was further supported by subsequent device activity consistent with Monero mining. Affected devices established follow-on connections to multiple external endpoints aligned with known mining infrastructure, indicating post-download execution.
Considering the broader sequence of activity, this pattern may point to a wider form of abuse in which legitimate VPN-related traffic is used to mask or facilitate malicious behavior following compromise.
On several devices, the download of executable files, including a newly observed peer[.]exe, was followed by alerts indicative of cryptocurrency mining activity. Mining-related credentials such as ‘x’ were observed using the Minergate protocol to communicate with endpoints within the 89.125.255.0/24 subnet and 188.241.218[.]111, the same endpoint involved in earlier download activity. Additional credentials appeared to reflect device-specific CPU identifiers, for example ‘12th Gen Intel(R) Core (TM) i5-1235U’.
Observed mining methods included login, submit, and job, consistent with active participation in a pool-based mining workflow rather than passive or incidental contact. The login method indicates that the host authenticated to the mining service as a worker, job reflects the assignment of computational tasks, and submit shows completed work being returned to the pool [8]. This sequence suggests that affected devices were actively contributing processing resources as part of an unauthorized distributed mining operation.
The presence of unauthorized cryptominers can lead to degraded system performance and reduced device stability. Beyond the immediate resource impact, such activity often serves as an indicator of a broader compromise rather than an isolated issue. This may increase the risk of further malware deployment, persistence mechanisms, and lateral movement, particularly in environments where the initial intrusion has not been fully contained.
Conclusion
Across affected environments, detections such as unusual VPN usage, connections to Hola infrastructure, anomalous HTTP activity, suspicious file downloads, and subsequent cryptomining behavior were linked into a single, evolving incident narrative. This aggregation provided a clearer view of attack progression, enabling security teams to understand not just isolated alerts, but the full sequence of compromise from initial contact through to post-exploitation.
Ultimately, these activities show that the risk posed by non-compliant software such as Hola VPN can extend far beyond simple policy violations. What began as traffic to Hola-related infrastructure was, in multiple cases, followed by behavior suggesting deliberate misuse, including suspicious executable downloads using Hola-related user agents and, in some instances, evidence of active cryptomining. These were not isolated anomalies, but elements of a broader pattern in which seemingly benign proxy or VPN-related communications may have created a pathway for malicious delivery and unauthorized resource exploitation.
The significance of this activity lies not only in the downloads or mining, but in what it reveals about an attacker’s ability to blend malicious operations into traffic associated with software that may already have a foothold in the environment. When unapproved software operates within an enterprise, it can reduce visibility, blur the distinction between legitimate and malicious traffic, and create opportunities to extend compromise in ways that are persistent and difficult to detect. Darktrace’s anomaly-based approach enables these behavioral distinctions to be identified, regardless of whether the device is new or long established within the network.
Credit to Min Kim (Associate Principal Analyst), Priya Thapa (Senior Cyber Analyst) Edited by Ryan Traill (Content Manager)