Blog
/
/
March 4, 2019

The VR Goldilocks Problem and Value of Continued Recognition

Security and Operations Teams face challenges when it comes to visibility and recognition. Learn more about how we find a solution to the problems!
Inside the SOC
Darktrace cyber analysts are world-class experts in threat intelligence, threat hunting and incident response, and provide 24/7 SOC support to thousands of Darktrace customers around the globe. Inside the SOC is exclusively authored by these experts, providing analysis of cyber incidents and threat trends, based on real-world experience in the field.
Written by
Max Heinemeyer
Global Field CISO
Default blog imageDefault blog imageDefault blog imageDefault blog imageDefault blog imageDefault blog image
04
Mar 2019

First, some context about VR

Security Operations teams face two fundamental challenges when it comes to 'finding bad'.

The first is gaining and maintaining appropriate visibility into what is happening in our environments. Visibility is provided through data (e.g. telemetry, logs). The trinity of data sources for visibility concern accounts/credentials, devices, and network traffic.

The second challenge is getting good recognition within the scope of what is visible. Recognition is fundamentally about what alerting and workflows you can implement and automate in response to activity that is suspicious or malicious.

Visibility and Recognition each have their own different associated issues.

Visibility is a problem about what is and can be generated and either read as telemetry, or logged and stored locally, or shipped to a central platform. The timelines and completeness of what visibility you have can depend on factors such as how much data you can or can't store locally on devices that generate data - and for how long; what your data pipeline and data platform look like (e.g. if you are trying to centralise data for analysis); or the capability of host software agents you have to process certain information locally.

The constraints on visibility sets the bar for factors like coverage, timelines and completeness of what recognition you can achieve. Without visibility, we cannot recognize at all. With limited visibility, what we can recognize may not have much value. With the right visibility, we can still fail to recognise the right things. And with too much recognition, we can quickly overload our senses.

A good example of a technology that offers the opportunity to solve these challenges at the network layer is Darktrace. Their technology provides visibility, from a network traffic perspective, into data that concerns devices and the accounts/credentials associated with them. They then provide recognition on top of this by using Machine Learning (ML) models for anomaly detection. Their models alert on a wide range of activities that may be indicative of threat activity, (e.g. malware execution and command and control, a technical exploit, data exfiltration and so on).

The major advantage they provide, compared to traditional Intrusion Detection Systems (IDS) and other vendors who also use ML for network anomaly detection, is that you can a) adjust the sensitivity of their algorithms and b) build your own recognition for particular patterns of interest. For example, if you want to monitor what connections are made to one or two servers, you can set up alerts for any change to expected patterns. This means you can create and adjust custom recognition based on your enterprise context and tune it easily in response to how context changes over time.

The Goldilocks VR Matrix

Below is what we call the VR Goldilocks Matrix at PBX Group Security. We use it to assess technology, measure our own capability and processes, and ask ourselves hard questions about where we need to focus to get the most value from our budget, (or make cuts / shift investment) if we need to.

In the squares are some examples of what you (maybe) should think about doing if you find yourself there.

Important questions to ask about VR

One of the things about Visibility and Recognition is that it’s not a given they are ‘always on’. Sometimes there are failure modes for visibility (causing a downstream issue with recognition). And sometimes there are failure modes or conditions under which you WANT to pause recognition.

The key questions you must have answers to about this include:

  • Under what conditions might I lose visibility?
  • How would I know if I have?
  • Is that loss a blind spot (i.e. data is lost for a given time period)…
  • …or is it 'a temporal delay’ (e.g. a connection fails and data is batched for moving from A to B but that doesn’t happen for a few hours)?
  • What are the recognitions that might be impacted by either of the above?
  • What is my expectation for the SLA on those recognitions from ‘cause of alert’ to ‘response workflow’?
  • Under what conditions would I be willing to pause recognition, change the workflow for what happens upon recognition, or stop it all together?
  • What is the stacked ranked list of ‘must, should, could’ for all recognition and why?

Alerts. Alerts everywhere.

More often than not, Security Operations teams suffer the costs of wasted time due to noisy alerts from certain data sources. As a consequence, it's more difficult for them to single out malicious behavior as suspicious or benign. The number of alerts that are generated due to out of the box SIEM platform configurations for sources like Web Proxies and Domain Controllers are often excessive, and the cost to tune those rules can also be unpalatable. Therefore, rather than trying to tune alerts, teams might make a call to switch them off until someone can get around to figuring out a better way. There’s no use having hypothetical recognition, but no workflow to act on what is generate (other than compliance).

This is where technologies that use ML can help. There are two basic approaches...

One is to avoid alerting until multiple conditions are met that indicate a high probability of threat activity. In this scenario, rather than alerting on the 1st, 2nd, 3rd and 4th ‘suspicious activities’, you wait until you have a critical mass of indicators, and then you generate one high fidelity alert that has a much greater weighting to be malicious. This requires both a high level of precision and accuracy in alerting, and naturally some trade off in the time that can pass before an alert for malicious activity is generated.

The other is to alert on ‘suspicious actives 1-4' and let an analyst or automated process decide if this merits further investigation. This approach sacrifices accuracy for precision, but provides rapid context on whether one, or multiple, conditions are met that push the machine(s) up the priority list in the triage queue. To solve for the lower level of accuracy, this approach can make decisions about how long to sustain alerting. For example, if a host triggers multiple anomaly detection models, rather than continue to send alerts (and risk the SOC deciding to turn them off), the technology can pause alerts after a certain threshold. If a machine has not been quarantined or taken off the network after 10 highly suspicious behaviors are flagged, there is a reasonable assumption that the analyst will have dug into these and found the activity is legitimate.

Punchline 1: the value of Continued Recognition even when 'not malicious'

The topic of paused detections was raised after a recent joint exercise between PBX Group Security and Darktrace in testing Darktrace’s recognition. After a machine being used by the PBX Red Team breached multiple high priority models on Darktrace, the technology stopped alerting on further activity. This was because the initial alerts would have been severe enough to trigger a SOC workflow. This approach is designed to solve the problem of alert overload on a machine that is behaving anomalously but is not in fact malicious. Rather than having the SOC turn off alerts for that machine (which could later be used maliciously), the alerts are paused.

One of the outcomes of the test was that the PBX Detect team advised they would still want those alerts to exist for context to see what else the machine does (i.e. to understand its pattern of life). Now, rather than pausing alerts, Darktrace is surfacing this to customers to show where a rule is being paused and create an option to continue seeing alerts for a machine that has breached multiple models.

Which leads us on to our next point…

Punchline 2: the need for Atomic Tests for detection

Both Darktrace and Photobox Security are big believers in Atomic Red Team testing, which involves ‘unit tests’ that repeatedly (or at a certain frequency) test a detection using code. Unit tests automate the work of Red Teams when they discovery control strengths (which you want to monitor continuously for uptime) or control gaps (which you want to monitor for when they are closed). You could design atomic tests to launch a series of particular attacks / threat actor actions from one machine in a chained event. Or you could launch different discreet actions from different machines, each of which has no prior context for doing bad stuff. This allows you to scale the sample size for testing what recognition you have (either through ML or more traditional SIEM alerting). Doing this also means you don't have to ask Red Teams to repeat the same tests again, allowing them to focus on different threat paths to achieve objectives.

Mitre Att&ck is an invaluable framework for this. Many vendors are now aligning to Att&ck to show what they can recognize relating to attack TTPs (Tools, Tactics and Procedures). This enables security teams to map what TTPs are relevant to them (e.g. by using threat intel about the campaigns of threat actor groups that are targeting them). Atomic Red Team tests can then be used to assure that expected detections are operational or find gaps that need closing.

If you miss detections, then you know you need to optimise the recognition you have. If you get too many recognitions outside of the atomic test conditions, you either have to accept a high false positive rate because of the nature of the network, or you can tune your detection sensitivity. The opportunities to do this with technology based on ML and anomaly detection are significant, because you can quickly see for new attack types what a unit test tells you about your current detections and that coverage you think you have is 'as expected'.

Punchline 3: collaboration for the win

Using well-structured Red Team exercises can help your organisation and your technology partners learn new things about how we can collectively find and halt evil. They can also help defenders learn more about good assumptions to build into ML models, as well as covering edge cases where alerts have 'business intelligence' value vs ‘finding bad’.

If you want to understand the categorisations of ways that your populations of machines act over time, there is no better way to do it than through anomaly detection and feeding alerts into a system that supports SOC operations as well as knowledge management (e.g. a graph database).

Working like this means that we also help get the most out of the visibility and recognition we have. Security solutions can be of huge help to Network and Operations teams for troubleshooting or answering questions about network architecture. Often, it’s just a shift in perspective that unlocks cross-functional value from investments in security tech and process. Understanding that recognition doesn’t stop with security is another great example of where technologies that let you build your own logic into recognition can make a huge difference above protecting the bottom line, to adding top line value.

Inside the SOC
Darktrace cyber analysts are world-class experts in threat intelligence, threat hunting and incident response, and provide 24/7 SOC support to thousands of Darktrace customers around the globe. Inside the SOC is exclusively authored by these experts, providing analysis of cyber incidents and threat trends, based on real-world experience in the field.
Written by
Max Heinemeyer
Global Field CISO

More in this series

No items found.

Blog

/

Cloud

/

June 12, 2025

Breaking Silos: Why Unified Security is Critical in Hybrid World

laptop with statistics on itDefault blog imageDefault blog image

Hybrid environments demand end-to-end visibility to stop modern attacks

Hybrid environments are a dominant trend in enterprise technology, but they continue to present unique issues to the defenders tasked with securing them. By 2026, Gartner predicts that 75% of organizations will adopt hybrid cloud strategies [1]. At the same time, only 23% of organizations report full visibility across cloud environments [2].

That means a strong majority of organizations do not have comprehensive visibility across both their on-premises and cloud networks. As a result, organizations are facing major challenges in achieving visibility and security in hybrid environments. These silos and fragmented security postures become a major problem when considering how attacks can move between different domains, exploiting the gaps.

For example, an attack may start with a phishing email, leading to the compromise of a cloud-based application identity and then moving between the cloud and network to exfiltrate data. Some attack types inherently involve multiple domains, like lateral movement and supply chain attacks, which target both on-premises and cloud networks.

Given this, unified visibility is essential for security teams to reduce blind spots and detect threats across the entire attack surface.

Risks of fragmented visibility

Silos arise due to separate teams and tools managing on-premises and cloud environments. Many teams have a hand in cloud security, with some common ones including security, infrastructure, DevOps, compliance, and end users, and these teams can all use different tools. This fragmentation increases the likelihood of inconsistent policies, duplicate alerts, and missed threats. And that’s just within the cloud, not even considering the additional defenses involved with network security.

Without a unified security strategy, gaps between these infrastructures and the teams which manage them can leave organizations vulnerable to cyber-attacks. The lack of visibility between on-premises and cloud environments contributes to missed threats and delayed incident response. In fact, breaches involving stolen or compromised credentials take an average of 292 to identify and contain [3]. That’s almost ten months.

The risk of fragmented visibility runs especially high as companies undergo cloud migrations. As organizations transition to cloud environments, they still have much of their data in on-premises networks, meaning that maintaining visibility across both on-premises and cloud environments is essential for securing critical assets and ensuring seamless operations.

Unified visibility is the solution

Unified visibility is achieved by having a single-pane-of-glass view to monitor both on-premises and cloud environments. This type of view brings many benefits, including streamlined detection, faster response times, and reduced complexity.

This can only be accomplished through integrations or interactions between the teams and tools involved with both on-premises security and cloud security.

AI-driven platforms, like Darktrace, are especially well equipped to enable the real-time monitoring and insights needed to sustain unified visibility. This is because they can handle the large amounts of data and data types.

Darktrace accomplishes this by plugging into an organization’s infrastructure so the AI can ingest and analyze data and its interactions within the environment to form an understanding of the organization’s normal behavior, right down to the granular details of specific users and devices. The system continually revises its understanding about what is normal based on evolving evidence.

This dynamic understanding of normal means that the AI engine can identify, with a high degree of precision, events or behaviors that are both anomalous and unlikely to be benign. This helps reduce noise while surfacing real threats, across cloud and on-prem environments without manual tuning.

In this way, given its versatile AI-based, platform approach, Darktrace empowers security teams with real-time monitoring and insights across both the network and cloud.

Unified visibility in the modern threat landscape

As part of the Darktrace ActiveAI Security Platform™, Darktrace / CLOUD works continuously across public, private, hybrid, and multi-cloud deployments. With real-time Cloud Asset Enumeration and Dynamic Architecture Modeling, Darktrace / CLOUD generates up-to-date architecture diagrams, giving SecOps and DevOps teams a unified view of cloud infrastructures.

It is always on the lookout for changes, driven by user and service activity. For example, unusual user activity can significantly raise the asset’s score, prompting Darktrace’s AI to update its architectural view and keep a living record of the cloud’s ever-changing landscape, providing near real-time insights into what’s happening.

This continuous architectural awareness ensures that security teams have a real-time understanding of cloud behavior and not just a static snapshot.

Darktrace / CLOUD’s unified view of AWS and Azure cloud posture and compliance over time.
Figure 1. Darktrace / CLOUD’s unified view of AWS and Azure cloud posture and compliance over time.

With this dynamic cloud visibility and monitoring, Darktrace / CLOUD can help unify and secure environments.

Real world example: Remote access supply chain attacks

Sectop Remote Access Trojan (RAT) malware, also known as ‘ArchClient2,’ is a .NET RAT that contains information stealing capabilities and allows threat actors to monitor and control targeted computers. It is commonly distributed through drive-by downloads of illegitimate software via malvertizing.

Darktrace has been able to detect and respond to Sectop RAT attacks using unified visibility and platform-wide coverage. In one such example, Darktrace observed one device making various suspicious connections to unusual endpoints, likely in an attempt to receive C2 information, perform beaconing activity, and exfiltrate data to the cloud.

This type of supply chain attack can jump from the network to the cloud, so a unified view of both environments helps shorten detection and response times, therefore mitigating potential impact. Darktrace’s ability to detect these cross-domain behaviors stems from its AI-driven, platform-native visibility.

Conclusion

Organizations need unified visibility to secure complex, hybrid environments effectively against threats and attacks. To achieve this type of comprehensive visibility, the gaps between legacy security tools across on-premises and cloud networks can be bridged with platform tools that use AI to boost data analysis for highly accurate behavioral prediction and anomaly detection.

Read more about the latest trends in cloud security in the blog “Protecting Your Hybrid Cloud: The Future of Cloud Security in 2025 and Beyond.”

References:

1. Gartner, May 22, 2023, “10 Strategic Data and Analytics Predictions Through 2028

2. Cloud Security Alliance, February 14, 2024, “Cloud Security Alliance Survey Finds 77% of Respondents Feel Unprepared to Deal with Security Threats

3. IBM, “Cost of a Data Breach Report 2024

Continue reading
About the author
Pallavi Singh
Product Marketing Manager, OT Security & Compliance

Blog

/

OT

/

June 11, 2025

Proactive OT security: Lessons on supply chain risk management from a rogue Raspberry Pi

man working on computerDefault blog imageDefault blog image

Understanding supply chain risk in manufacturing

For industries running Industrial Control Systems (ICS) such as manufacturing and fast-moving consumer goods (FMCG), complex supply chains mean that disruption to one weak node can have serious impacts to the entire ecosystem. However, supply chain risk does not always originate from outside an organization’s ICS network.  

The implicit trust placed on software or shared services for maintenance within an ICS can be considered a type of insider threat [1], where defenders also need to look ‘from within’ to protect against supply chain risk. Attackers have frequently mobilised this form of insider threat:

  • Many ICS and SCADA systems were compromised during the 2014 Havex Watering Hole attack, where via operators’ implicit trust in the trojanized versions of legitimate applications, on legitimate but compromised websites [2].
  • In 2018, the world’s largest manufacturer of semiconductors and processers shut down production for three days after a supplier installed tainted software that spread to over 10,000 machines in the manufacturer’s network [3].
  • During the 2020 SolarWinds supply chain attack, attackers compromised a version of Orion software that was deployed from SolarWinds’ own servers during a software update to thousands of customers, including tech manufacturing companies such as Intel and Nvidia [4].

Traditional approaches to ICS security have focused on defending against everything from outside the castle walls, or outside of the ICS network. As ICS attacks become more sophisticated, defenders must not solely rely on static perimeter defenses and prevention. 

A critical part of active defense is understanding the ICS environment and how it operates, including all possible attack paths to the ICS including network connections, remote access points, the movement of data across zones and conduits and access from mobile devices. For instance, original equipment manufacturers (OEMs) and vendors often install remote access software or third-party equipment in ICS networks to facilitate legitimate maintenance and support activities, which can unintentionally expand the ICS’ attack surface.  

This blog describes an example of the convergence between supply chain risk and insider risk, when a vendor left a Raspberry Pi device in a manufacturing customer’s ICS network without the customer’s knowledge.

Case study: Using unsupervised machine learning to detect pre-existing security issues

Raspberry Pi devices are commonly used in SCADA environments as low-cost, remotely accessible data collectors [5][6][7]. They are often paired with Industrial Internet of Things (IIoT) for monitoring and tracking [8]. However, these devices also represent a security risk because their small physical size and time-consuming nature of physical inspection makes them easy to overlook. This poses a security risk, as these devices have previously been used to carry out USB-based attacks or to emulate Ethernet-over-USB connections to exfiltrate sensitive data [8][9].

In this incident, a Darktrace customer was unaware that their supplier had installed a Raspberry Pi device on their ICS network. Crucially, the installation occurred prior to Darktrace’s deployment on the customer’s network. 

For other anomaly detection tools, this order of events meant that this third-party device would likely have been treated as part of the customer’s existing infrastructure. However, after Darktrace was deployed, it analyzed the metadata from the encrypted HTTPS and DNS connections that the Raspberry Pi made to ‘call home’ to the supplier and determined that these connections were  unusual compared to the rest of the devices in the network, even in the absence of any malicious indicators of compromise (IoCs).  

Darktrace triggered the following alerts for this unusual activity that consequently notified the customer to the pre-existing threat of an unmanaged device already present in their network:

  • Compromise / Sustained SSL or HTTP Increase
  • Compromise / Agent Beacon (Short Period)
  • Compromise / Agent Beacon (Medium Period)
  • Compromise / Agent Beacon (Long Period)
  • Tags / New Raspberry Pi Device
  • Device / DNS Requests to Unusual Server
  • Device / Anomaly Indicators / Spike in Connections to Rare Endpoint Indicator
Darktrace’s External Sites Summary showing the rarity of the external endpoint that the Raspberry Pi device ‘called home’ to and the model alerts triggered.  
Figure 1: Darktrace’s External Sites Summary showing the rarity of the external endpoint that the Raspberry Pi device ‘called home’ to and the model alerts triggered.  

Darktrace’s Cyber AI Analyst launched an autonomous investigation into the activity, correlating related events into a broader incident and generating a report outlining the potential threat along with supporting technical details.

Darktrace’s anomaly-based detection meant that the Raspberry Pi device did not need to be observed performing clearly malicious behavior to alert the customer to the security risk, and neither can defenders afford to wait for such escalation.

Why is this significant?

In 2021 a similar attack took place. Aiming to poison a Florida water treatment facility, attackers leveraged a TeamViewer instance that had been dormant on the system for six months, effectively allowing the attacker to ‘live off the land’ [10].  

The Raspberry Pi device in this incident also remained outside the purview of the customer’s security team at first. It could have been leveraged by a persistent attacker to pivot within the internal network and communicate externally.

A proactive approach to active defense that seeks to minimize and continuously monitor the attack surface and network is crucial.  

The growing interest in manufacturing from attackers and policymakers

Significant motivations for targeting the manufacturing sector and increasing regulatory demands make the convergence of supply chain risk, insider risk, and the prevalence of stealthy living-off-the-land techniques particularly relevant to this sector.

Manufacturing is consistently targeted by cybercriminals [11], and the sector’s ‘just-in-time’ model grants attackers the opportunity for high levels of disruption. Furthermore, under NIS 2, manufacturing and some food and beverage processing entities are now designated as ‘important’ entities. This means stricter incident reporting requirements within 24 hours of detection, and enhanced security requirements such as the implementation of zero trust and network segmentation policies, as well as measures to improve supply chain resilience [12][13][14].

How can Darktrace help?

Ultimately, Darktrace successfully assisted a manufacturing organization in detecting a potentially disruptive 'near-miss' within their OT environment, even in the absence of traditional IoCs.  Through passive asset identification techniques and continuous network monitoring, the customer improved their understanding of their network and supply chain risk.  

While the swift detection of the rogue device allowed the threat to be identified before it could escalate, the customer could have reduced their time to respond by using Darktrace’s built-in response capabilities, had Darktrace’s Autonomous Response capability been enabled.  Darktrace’s Autonomous Response can be configured to target specific connections on a rogue device either automatically upon detection or following manual approval from the security team, to stop it communicating with other devices in the network while allowing other approved devices to continue operating. Furthermore, the exportable report generated by Cyber AI Analyst helps security teams to meet NIS 2’s enhanced reporting requirements.  

Sophisticated ICS attacks often leverage insider access to perform in-depth reconnaissance for the development of tailored malware capabilities.  This case study and high-profile ICS attacks highlight the importance of mitigating supply chain risk in a similar way to insider risk.  As ICS networks adapt to the introduction of IIoT, remote working and the increased convergence between IT and OT, it is important to ensure the approach to secure against these threats is compatible with the dynamic nature of the network.  

Credit to Nicole Wong (Principal Cyber Analyst), Matthew Redrup (Senior Analyst and ANZ Team Lead)

[related-resource]

Appendices

MITRE ATT&CK Mapping

  • Infrastructure / New Raspberry Pi Device - INITIAL ACCESS - T1200 Hardware Additions
  • Device / DNS Requests to Unusual Server - CREDENTIAL ACCESS, COLLECTION - T1557 Man-in-the-Middle
  • Compromise / Agent Beacon - COMMAND AND CONTROL - T1071.001 Web Protocols

References

[1] https://www.cisa.gov/topics/physical-security/insider-threat-mitigation/defining-insider-threats

[2] https://www.trendmicro.com/vinfo/gb/threat-encyclopedia/web-attack/139/havex-targets-industrial-control-systems

[3]https://thehackernews.com/2018/08/tsmc-wannacry-ransomware-attack.html

[4] https://www.theverge.com/2020/12/21/22194183/intel-nvidia-cisco-government-infected-solarwinds-hack

[5] https://www.centreon.com/monitoring-ot-with-raspberry-pi-and-centreon/

[6] https://ieeexplore.ieee.org/document/9107689

[7] https://www.linkedin.com/pulse/webicc-scada-integration-industrial-raspberry-pi-devices-mryff

[8] https://www.rowse.co.uk/blog/post/how-is-the-raspberry-pi-used-in-the-iiot

[9] https://sepiocyber.com/resources/whitepapers/raspberry-pi-a-friend-or-foe/#:~:text=Initially%20designed%20for%20ethical%20purposes,as%20cyberattacks%20and%20unauthorized%20access

[10] https://edition.cnn.com/2021/02/10/us/florida-water-poison-cyber/index.html

[11] https://www.mxdusa.org/2025/02/13/top-cyber-threats-in-manufacturing/

[12] https://www.shoosmiths.com/insights/articles/nis2-what-manufacturers-and-distributors-need-to-know-about-europes-new-cybersecurity-regime

[13] https://www.goodaccess.com/blog/nis2-require-zero-trust-essential-security-measure#zero-trust-nis2-compliance

[14] https://logisticsviewpoints.com/2024/11/06/the-impact-of-nis-2-regulations-on-manufacturing-supply-chains/

Continue reading
About the author
Nicole Wong
Cyber Security Analyst
Your data. Our AI.
Elevate your network security with Darktrace AI