Blog
/
AI
/
January 30, 2025

Reimagining Your SOC: Overcoming Alert Fatigue with AI-Led Investigations  

Reimagining your SOC Part 2/3: This blog explores how the challenges facing the modern SOC can be addressed by transforming the investigation process, unlocking efficiency and scalability in SOC operations with AI.
Inside the SOC
Darktrace cyber analysts are world-class experts in threat intelligence, threat hunting and incident response, and provide 24/7 SOC support to thousands of Darktrace customers around the globe. Inside the SOC is exclusively authored by these experts, providing analysis of cyber incidents and threat trends, based on real-world experience in the field.
Written by
Brittany Woodsmall
Product Marketing Manager, AI
Default blog imageDefault blog imageDefault blog imageDefault blog imageDefault blog imageDefault blog image
30
Jan 2025

The efficiency of a Security Operations Center (SOC) hinges on its ability to detect, analyze and respond to threats effectively. With advancements in AI and automation, key early SOC team metrics such as Mean Time to Detect (MTTD) have seen significant improvements:

  • 96% of defenders believing AI-powered solutions significantly boost the speed and efficiency of prevention, detection, response, and recovery.
  • Organizations leveraging AI and automation can shorten their breach lifecycle by an average of 108 days compared to those without these technologies.

While tool advances have improved performance and effectiveness in the detection phase, this has not been as beneficial to the next step of the process where initial alerts are investigated further to determine their relevance and how they relate to other activities. This is often measured with the metric Mean Time to Analysis (MTTA), although some SOC teams operate a two-level process with teams for initial triage to filter out more obviously uninteresting alerts and for more detailed analysis of the remainder. SOC teams continue to grapple with alert fatigue, overwhelmed analysts, and inefficient triage processes, preventing them from achieving the operational efficiency necessary for a high-performing SOC.

Addressing this core inefficiency requires extending AI's capabilities beyond detection to streamline and optimize the following investigative workflows that underpin effective analysis.

Challenges with SOC alert investigation

Detecting cyber threats is only the beginning of a much broader challenge of SOC efficiency. The real bottleneck often lies in the investigation process.

Detection tools and techniques have evolved significantly with the use of machine learning methods, improving early threat detection. However, after a detection pops up, human analysts still typically step in to evaluate the alert, gather context, and determine whether it’s a true threat or a false alarm and why. If it is a threat, further investigation must be performed to understand the full scope of what may be a much larger problem. This phase, measured by the mean time to analysis, is critical for swift incident response.

Challenges with manual alert investigation:

  • Too many alerts
  • Alerts lack context
  • Cognitive load sits with analysts
  • Insufficient talent in the industry
  • Fierce competition for experienced analysts

For many organizations, investigation is where the struggle of efficiency intensifies. Analysts face overwhelming volumes of alerts, a lack of consolidated context, and the mental strain of juggling multiple systems. With a worldwide shortage of 4 million experienced level two and three SOC analysts, the cognitive burden placed on teams is immense, often leading to alert fatigue and missed threats.

Even with advanced systems in place not all potential detections are investigated. In many cases, only a quarter of initial alerts are triaged (or analyzed). However, the issue runs deeper. Triaging occurs after detection engineering and alert tuning, which often disable many alerts that could potentially reveal true threats but are not accurate enough to justify the time and effort of the security team. This means some potential threats slip through unnoticed.

Understanding alerts in the SOC: Stopping cyber incidents is hard

Let’s take a look at the cyber-attack lifecycle and the steps involved in detecting and stopping an attack:

First we need a trace of an attack…

The attack will produce some sort of digital trace. Novel attacks, insider threats, and attacker techniques such as living-off-the-land can make attacker activities extremely hard to distinguish.

A detection is created…

Then we have to detect the trace, for example some beaconing to a rare domain. Initial detection alerts being raised underpin the MTTD (mean time to detection). Reducing this initial unseen duration is where we have seen significant improvement with modern threat detection tools.

When it comes to threat detection, the possibilities are vast. Your initial lead could come from anything: an alert about unusual network activity, a potential known malware detection, or an odd email. Once that lead comes in, it’s up to your security team to investigate further and determine if this is this a legitimate threat or a false alarm and what the context is behind the alert.

Investigation begins…

It doesn’t just stop at a detection. Typically, humans also need to look at the alert, investigate, understand, analyze, and conclude whether this is a genuine threat that needs a response. We normally measure this as MTTA (mean time to analyze).

Conducting the investigation effectively requires a high degree of skill and efficiency, as every second counts in mitigating potential damage. Security teams must analyze the available data, correlate it across multiple sources, and piece together the timeline of events to understand the full scope of the incident. This process involves navigating through vast amounts of information, identifying patterns, and discerning relevant details. All while managing the pressure of minimizing downtime and preventing further escalation.

Containment begins…

Once we confirm something as a threat, and the human team determines a response is required and understand the scope, we need to contain the incident. That's normally the MTTC (mean time to containment) and can be further split into immediate and more permanent measures.

For more about how AI-led solutions can help in the containment stage read here: Autonomous Response: Streamlining Cybersecurity and Business Operations

The challenge is not only in 1) detecting threats quickly, but also 2) triaging and investigating them rapidly and with precision, and 3) prioritizing the most critical findings to avoid missed opportunities. Effective investigation demands a combination of advanced tools, robust workflows, and the expertise to interpret and act on the insights they generate. Without these, organizations risk delaying critical containment and response efforts, leaving them vulnerable to greater impacts.

While there are further steps (remediation, and of course complete recovery) here we will focus on investigation.

Developing an AI analyst: How Darktrace replicates human investigation

Darktrace has been working on understanding the investigative process of a skilled analyst since 2017. By conducting internal research between Darktrace expert SOC analysts and machine learning engineers, we developed a formalized understanding of investigative processes. This understanding formed the basis of a multi-layered AI system that systematically investigates data, taking advantage of the speed and breadth afforded by machine systems.

With this research we found that the investigative process often revolves around iterating three key steps: hypothesis creation, data collection, and results evaluation.

All these details are crucial for an analyst to determine the nature of a potential threat. Similarly, they are integral components of our Cyber AI Analyst which is an integral component across our product suite. In doing so, Darktrace has been able to replicate the human-driven approach to investigating alerts using machine learning speed and scale.

Here’s how it works:

  • When an initial or third-party alert is triggered, the Cyber AI Analyst initiates a forensic investigation by building multiple hypotheses and gathering relevant data to confirm or refute the nature of suspicious activity, iterating as necessary, and continuously refining the original hypothesis as new data emerges throughout the investigation.
  • Using a combination of machine learning including supervised and unsupervised methods, NLP and graph theory to assess activity, this investigation engine conducts a deep analysis with incidents raised to the human team only when the behavior is deemed sufficiently concerning.
  • After classification, the incident information is organized and processed to generate the analysis summary, including the most important descriptive details, and priority classification, ensuring that critical alerts are prioritized for further action by the human-analyst team.
  • If the alert is deemed unimportant, the complete analysis process is made available to the human team so that they can see what investigation was performed and why this conclusion was drawn.
Darktrace cyber ai analyst workflow, how it works

To illustrate this via example, if a laptop is beaconing to a rare domain, the Cyber AI Analyst would create hypotheses including whether this could be command and control traffic, data exfiltration, or something else. The AI analyst then collects data, analyzes it, makes decisions, iterates, and ultimately raises a new high-level incident alert describing and detailing its findings for human analysts to review and follow up.

Learn more about Darktrace's Cyber AI Analyst

  • Cost savings: Equivalent to adding up to 30 full-time Level 2 analysts without increasing headcount
  • Minimize business risk: Takes on the busy work from human analysts and elevates a team’s overall decision making
  • Improve security outcomes: Identifies subtle, sophisticated threats through holistic investigations

Unlocking an efficient SOC

To create a mature and proactive SOC, addressing the inefficiencies in the alert investigation process is essential. By extending AI's capabilities beyond detection, SOC teams can streamline and optimize investigative workflows, reducing alert fatigue and enhancing analyst efficiency.

This holistic approach not only improves Mean Time to Analysis (MTTA) but also ensures that SOCs are well-equipped to handle the evolving threat landscape. Embracing AI augmentation and automation in every phase of threat management will pave the way for a more resilient and proactive security posture, ultimately leading to a high-performing SOC that can effectively safeguard organizational assets.

Every relevant alert is investigated

The Cyber AI Analyst is not a generative AI system, or an XDR or SEIM aggregator that simply prompts you on what to do next. It uses a multi-layered combination of many different specialized AI methods to investigate every relevant alert from across your enterprise, native, 3rd party, and manual triggers, operating at machine speed and scale. This also positively affects detection engineering and alert tuning, because it does not suffer from fatigue when presented with low accuracy but potentially valuable alerts.

Retain and improve analyst skills

Transferring most analysis processes to AI systems can risk team skills if they don't maintain or build them and if the AI doesn't explain its process. This can reduce the ability to challenge or build on AI results and cause issues if the AI is unavailable. The Cyber AI Analyst, by revealing its investigation process, data gathering, and decisions, promotes and improves these skills. Its deep understanding of cyber incidents can be used for skill training and incident response practice by simulating incidents for security teams to handle.

Create time for cyber risk reduction

Human cybersecurity professionals excel in areas that require critical thinking, strategic planning, and nuanced decision-making. With alert fatigue minimized and investigations streamlined, your analysts can avoid the tedious data collection and analysis stages and instead focus on critical decision-making tasks such as implementing recovery actions and performing threat hunting.

Stay tuned for part 3/3

Part 3/3 in the Reimagine your SOC series explores the preventative security solutions market and effective risk management strategies.

Coming soon!

Inside the SOC
Darktrace cyber analysts are world-class experts in threat intelligence, threat hunting and incident response, and provide 24/7 SOC support to thousands of Darktrace customers around the globe. Inside the SOC is exclusively authored by these experts, providing analysis of cyber incidents and threat trends, based on real-world experience in the field.
Written by
Brittany Woodsmall
Product Marketing Manager, AI

More in this series

No items found.

Blog

/

Cloud

/

March 5, 2026

Inside Cloud Compromise: Investigating Attacker Activity with Darktrace / Forensic Acquisition & Investigation

Forensic Acquisition and investigationDefault blog imageDefault blog image

Investigating cloud attacks with Darktrace/ Forensic Acquisition & Investigation

Darktrace / Forensic Acquisition & Investigation™ is the industry’s first truly automated forensic solution purpose-built for the cloud. This blog will demonstrate how an investigation can be carried out against a compromised cloud server in minutes, rather than hours or days.

The compromised server investigated in this case originates from Darktrace’s Cloudypots system, a global honeypot network designed to observe adversary activity in real time across a wide range of cloud services. Whenever an attacker successfully compromises one of these honeypots, a forensic copy of the virtual server's disk is preserved for later analysis. Using Forensic Acquisition & Investigation, analysts can then investigate further and obtain detailed insights into the compromise including complete attacker timelines and root cause analysis.

Forensic Acquisition & Investigation supports importing artifacts from a variety of sources, including EC2 instances, ECS, S3 buckets, and more. The Cloudypots system produces a raw disk image whenever an attack is detected and stores it in an S3 bucket. This allows the image to be directly imported into Forensic Acquisition & Investigation using the S3 bucket import option.

As Forensic Acquisition & Investigation runs cloud-natively, no additional configuration is required to add a specific S3 bucket. Analysts can browse and acquire forensic assets from any bucket that the configured IAM role is permitted to access. Operators can also add additional IAM credentials, including those from other cloud providers, to extend access across multiple cloud accounts and environments.

Figure 1: Forensic Acquisition & Investigation import screen.

Forensic Acquisition & Investigation then retrieves a copy of the file and automatically begins running the analysis pipeline on the artifact. This pipeline performs a full forensic analysis of the disk and builds a timeline of the activity that took place on the compromised asset. By leveraging Forensic Acquisition & Investigation’s cloud-native analysis system, this process condenses hour of manual work into just minutes.

Successful import of a forensic artifact and initiation of the analysis pipeline.
Figure 2: Successful import of a forensic artifact and initiation of the analysis pipeline.

Once processing is complete, the preserved artifact is visible in the Evidence tab, along with a summary of key information obtained during analysis, such as the compromised asset’s hostname, operating system, cloud provider, and key event count.

The Evidence overview showing the acquired disk image.
Figure 3: The Evidence overview showing the acquired disk image.

Clicking on the “Key events” field in the listing opens the timeline view, automatically filtered to show system- generated alarms.

The timeline provides a chronological record of every event that occurred on the system, derived from multiple sources, including:

  • Parsed log files such as the systemd journal, audit logs, application specific logs, and others.
  • Parsed history files such as .bash_history, allowing executed commands to be shown on the timeline.
  • File-specific events, such as files being created, accessed, modified, or executables being run, etc.

This approach allows timestamped information and events from multiple sources to be aggregated and parsed into a single, concise view, greatly simplifying the data review process.

Alarms are created for specific timeline events that match either a built-in system rule, curated by Darktrace’s Threat Research team or an operator-defined rule  created at the project level. These alarms help quickly filter out noise and highlight on events of interest, such as the creation of a file containing known malware, access to sensitive files like Amazon Web Service (AWS) credentials, suspicious arguments or commands, and more.

 The timeline view filtered to alarm_severity: “1” OR alarm_severity: “3”, showing only events that matched an alarm rule.
Figure 4: The timeline view filtered to alarm_severity: “1” OR alarm_severity: “3”, showing only events that matched an alarm rule.

In this case, several alarms were generated for suspicious Base64 arguments being passed to Selenium. Examining the event data, it appears the attacker spawned a Selenium Grid session with the following payload:

"request.payload": "[Capabilities {browserName: chrome, goog:chromeOptions: {args: [-cimport base64;exec(base64...], binary: /usr/bin/python3, extensions: []}, pageLoadStrategy: normal}]"

This is a common attack vector for Selenium Grid. The chromeOptions object is intended to specify arguments for how Google Chrome should be launched; however, in this case the attacker has abused the binary field to execute the Python3 binary instead of Chrome. Combined with the option to specify command-line arguments, the attacker can use Python3’s -c option to execute arbitrary Python code, in this instance, decoding and executing a Base64 payload.

Selenium’s logs truncate the Arguments field automatically, so an alternate method is required to retrieve the full payload. To do this, the search bar can be used to find all events that occurred around the same time as this flagged event.

Pivoting off the previous event by filtering the timeline to events within the same window using timestamp: [“2026-02-18T09:09:00Z” TO “2026-02-18T09:12:00Z”].
Figure 5: Pivoting off the previous event by filtering the timeline to events within the same window using timestamp: [“2026-02-18T09:09:00Z” TO “2026-02-18T09:12:00Z”].

Scrolling through the search results, an entry from Java’s systemd journal can be identified. This log contains the full, unaltered payload. GCHQ’s CyberChef can then be used to decode the Base64 data into the attacker’s script, which will ultimately be executed.

Decoding the attacker’s payload in CyberChef.
Figure 6: Decoding the attacker’s payload in CyberChef.

In this instance, the malware was identified as a variant of a campaign that has been previously documented in depth by Darktrace.

Investigating Perfctl Malware

This campaign deploys a malware sample known as ‘perfctl to the compromised host. The script executed by the attacker downloads a Go binary named “promocioni.php” from 200[.]4.115.1. Its functionality is consistent with previously documented perfctl samples, with only minor changes such as updated filenames and a new command-and-control (C2) domain.

Perfctl is a stealthy malware that has several systems designed  to evade detection. The main binary is packed with UPX, with the header intentionally tampered with to prevent unpacking using regular tools. The binary also avoids executing any malicious code if it detects debugging or tracing activity, or if artifacts left by earlier stages are missing.

To further aid its evasive capabilities, perfctl features a usermode rootkit using an LD preload. This causes dynamically linked executables to load perfctl’s rootkit payload before other system modules, allowing it to override functions, such as intercepting calls to list files and hiding output from the returned list. Perfctl uses this to hide its own files, as well as other files like the ld.so.preload file, preventing users from identifying that a rootkit is present in the first place.

This also makes it difficult to dynamically analyze, as even analysts aware of the rootkit will struggle to get around it due to its aggressiveness in hiding its components. A useful trick is to use the busybox-static utilities, which are statically linked and therefore immune to LD preloading.

Perfctl will attempt to use sudo to escalate its permissions to root if the user it was executed as has the required privileges. Failing this, it will attempt to exploit the vulnerability CVE-2021-4034.

Ultimately, perfctl will attempt to establish a C2 link via Tor and spawn an XMRig miner to mine the Monero cryptocurrency. The traffic to the mining pool is encapsulated within Tor to limit network detection of the mining traffic.

Darktrace’s Cloudypots system has observed 1,959 infections of the perfctl campaign across its honeypot network in the past year, making it one of the most aggressive campaigns seen by Darktrace.

Key takeaways

This blog has shown how Darktrace / Forensic Acquisition & Investigation equips defenders in the face of a real-world attacker campaign. By using this solution, organizations can acquire forensic evidence and investigate intrusions across multiple cloud resources and providers, enabling defenders to see the full picture of an intrusion on day one. Forensic Acquisition & Investigation’s patented data-processing system takes advantage of the cloud’s scale to rapidly process large amounts of data, allowing triage to take minutes, not hours.

Darktrace / Forensic Acquisition & Investigation is available as Software-as-a-Service (SaaS) but can also be deployed on-premises as a virtual application or natively in the cloud, providing flexibility between convenience and data sovereignty to suit any use case.

Support for acquiring traditional compute instances like EC2, as well as more exotic and newly targeted platforms such as ECS and Lambda, ensures that attacks taking advantage of Living-off-the-Cloud (LOTC) strategies can be triaged quickly and easily as part of incident response. As attackers continue to develop new techniques, the ability to investigate how they use cloud services to persist and pivot throughout an environment is just as important to triage as a single compromised EC2 instance.

Credit to Nathaniel Bill (Malware Research Engineer)

Continue reading
About the author
Nathaniel Bill
Malware Research Engineer

Blog

/

AI

/

March 2, 2026

What the Darktrace Annual Threat Report 2026 Means for Security Leaders

Image of the Earth from spaceDefault blog imageDefault blog image

The challenge for today’s CISOs

At the broadest level, the defining characteristic of cybersecurity in 2026 is the sheer pace of change shaping the environments we protect. Organizations are operating in ecosystems that are larger, more interconnected, and more automated than ever before – spanning cloud platforms, distributed identities, AI-driven systems, and continuous digital workflows.  

The velocity of this expansion has outstripped the slower, predictable patterns security teams once relied on. What used to be a stable backdrop is now a living, shifting landscape where technology, risk, and business operations evolve simultaneously. From this vantage point, the central challenge for security leaders isn’t reacting to individual threats, but maintaining strategic control and clarity as the entire environment accelerates around them.

Strategic takeaways from the Annual Threat Report

The Darktrace Annual Threat Report 2026 reinforces a reality every CISO feels: the center of gravity isn’t the perimeter, vulnerability management, or malware, but trust abused via identity. For example, our analysis found that nearly 70% of incidents in the Americas region begin with stolen or misused accounts, reflecting the global shift toward identity‑led intrusions.

Mass adoption of AI agents, cloud-native applications, and machine decision-making means CISOs now oversee systems that act on their own. This creates an entirely new responsibility: ensuring those systems remain safe, predictable, and aligned to business intent, even under adversarial pressure.

Attackers increasingly exploit trust boundaries, not firewalls – leveraging cloud entitlements, SaaS identity transitions, supply-chain connectivity, and automation frameworks. The rise of non-human identities intensifies this: credentials, tokens, and agent permissions now form the backbone of operational risk.

Boards are now evaluating CISOs on business continuity, operational recovery, and whether AI systems and cloud workloads can fail safely without cascading or causing catastrophic impact.

In this environment, detection accuracy, autonomous response, and blast radius minimization matter far more than traditional control coverage or policy checklists.

Every organization will face setbacks; resilience is measured by how quickly security teams can rise, respond, and resume momentum. In 2026, success will belong to those that adapt fastest.

Managing business security in the age of AI

CISO accountability in 2026 has expanded far beyond controls and tooling. Whether we asked for it or not, we now own outcomes tied to business resilience, AI trust, cloud assurance, and continuous availability. The role is less about certainty and more about recovering control in an environment that keeps accelerating.

Every major 2026 initiative – AI agents, third-party risk, cloud, or comms protection – connects to a single board-level question: Are we still in control as complexity and automation scale faster than humans?

Attackers are not just getting more sophisticated; they are becoming more automated. AI changes the economics of attack, lowering cost and increasing speed. That asymmetry is what CISOs are being measured against.

CISOs are no longer evaluated on tool coverage, but on the ability to assure outcomes – trust in AI adoption, resilience across cloud and identity, and being able to respond to unknown and unforeseen threats.

Boards are now explicitly asking whether we can defend against AI-driven threats. No one can predict every new behavior – survival depends on detecting malicious deviations from normal fast and responding autonomously.  

Agents introduce decision-making at machine speed. Governance, CI/CD scanning, posture management, red teaming, and runtime detection are no longer differentiators but the baseline.

Cloud security is no longer architectural, it is operational. Identity, control planes, and SaaS exposure now sit firmly with the CISO.

AI-speed threats already reshaping security in 2026

We’re already seeing clear examples of how quickly the threat landscape has shifted in 2026. Darktrace’s work on React2Shell exposed just how unforgiving the new tempo is: a honeypot stood up with an exposed React was hit in under two minutes. There was no recon phase, no gradual probing – just immediate, automated exploitation the moment the code appeared publicly. Exposure now equals compromise unless defenses can detect, interpret, and act at machine speed. Traditional operational rhythms simply don’t map to this reality.

We’re also facing the first wave of AI-authored malware, where LLMs generate code that mutates on demand. This removes the historic friction from the attacker side: no skill barrier, no time cost, no limit on iteration. Malware families can regenerate themselves, shift structure, and evade static controls without a human operator behind the keyboard. This forces CISOs to treat adversarial automation as a core operational risk and ensure that autonomous systems inside the business remain predictable under pressure.

The CVE-2026-1731 BeyondTrust exploitation wave reinforced the same pattern. The gap between disclosure and active, global exploitation compressed into hours. Automated scanning, automated payload deployment, coordinated exploitation campaigns, all spinning up faster than most organizations can push an emergency patch through change control. The vulnerability-to-exploit window has effectively collapsed, making runtime visibility, anomaly detection, and autonomous containment far more consequential than patching speed alone.

These cases aren’t edge scenarios; they represent the emerging norm. Complexity and automation have outpaced human-scale processes, and attackers are weaponizing that asymmetry.  

The real differentiator for CISOs in 2026 is less about knowing everything and more about knowing immediately when something shifts – and having systems that can respond at the same speed.

[related-resource]

Continue reading
About the author
Mike Beck
Global CISO
Your data. Our AI.
Elevate your network security with Darktrace AI