Technology
Products
Blogs
Resources
Company
English
What’s New
Technology
Products
Blogs
Resources
Company

Darktrace Cyber AI Research Centre

The cyber security industry has a history of reacting to attacks detecting only techniques it’s witnessed before. But attackers continue to innovate, combining new methods in new ways to create novel approaches, and most recently, to expose dormant software vulnerabilities – and adversaries show no signs of slowing down.

What would it take to tip the scales in favor of the defender?

“Research unlocks the unknowns; it also helps shed light on what we are collectively up against.”
Jack Stockdale OBE, CTO

Based in Cambridge, UK, the Darktrace Cyber AI Research Centre focuses on how both adversaries and defenders may apply AI to the ever-growing challenge of escalating cyber-attacks. Comprised of mathematicians, scientists, and AI experts, the Darktrace Cyber AI Research Centre has produced breakthroughs that have organically grown our knowledge of the security ecosystem.

More than 100 patents protected or pending for AI and machine learning concepts have evidenced new paths forward and today, our researchers are actively exploring new, innovative approaches in areas including defensive and adversarial AI, deepfakes, AI in social engineering, natural language processing, graph theory, and self-healing systems. Below you can read a selection of our research abstracts stemming from projects undertaken by expert members of our research team.

“[The Centre] saw growth of 69.2% in related patent applications in the three months ending November, compared to the same period [the previous year] — the highest percentage growth out of all companies tracked.”
GlobalData

Read our new Darktrace Discourse paper, authored by one of our researchers based at the Centre. The author has seven years’ experience in the automation of complex cyber-centric processes with specialization in the offensive domain, and holds a PhD in Astrophysics from the University of Cambridge.

Prevent: Security through Adversity

Read our attack path modeling research paper, which outlines how the fusion of Self-Learning AI and Graph Theory can yield cross-domain, dynamic and risk-prioritized attack pathways within a cyber environment.

Attack Path Modeling Research
48% headcount increase in 2021
24 doctorates,
59 master’s degrees

Research titles and abstracts

A multi-platform approach to autonomous response

John Boyer
How autonomous response technology works seamlessly across different environments, providing a solution for organizations using multiple cloud environments and SaaS applications.

Modern companies have large numbers of employees, who use multiple SaaS accounts to access multiple services, from multiple devices, in multiple time zones.

As these companies continue to expand, and continue to embrace cloud computing, it is becoming exceedingly difficult to stop cyber-attacks before significant damage is done. Companies can no longer rely on network-monitoring tools to direct human intervention and must embrace a different approach.

Darktrace Antigena is an autonomous-response platform that can act from anywhere inside an organization’s network infrastructure: on user devices, network devices, SaaS accounts, and on email messages.

However, for an autonomous-response platform to be effective, it needs to understand the various aliases and behaviors that represent a single “user”.

Antigena is underpinned by “context-gathering” technologies in Darktrace SaaS and Cloud Modules that retrieve data about users’ devices, roles, or departments, among other contextual markers, and associate disparate entities.

By developing this “meta” identity, it is possible to identify a threat in one SaaS service and autonomously respond in the same service, another service, or in many. In response to an unusual login attempt on an IDaaS provider, Antigena can temporarily shut down the user’s Zoom account — to prevent the attacker performing social engineering — and disable the user’s Google Workspace account — to prevent data exfiltration.

This approach is well suited to a dynamic and expanding problem and can stop cyber-attacks before they have done significant damage. Although Antigena will have to include new services as they emerge, such as Zero Trust and endpoint, the approach of understanding the user’s behavior and responding to a threat in all relevant technologies will not need to change.

Identification of services associated with data uploads for analysis of exfiltration

Dr Tim Bazalgette
How the intelligent analysis of connection properties can help assess whether specific uploads are associated with malicious exfiltration.

Malicious actors sometimes exfiltrate data to a single endpoint, but more frequently upload data to multiple endpoints. They may do so to obscure the exfiltration and avoid detection, or because legitimate services, such as cloud storage, use multiple endpoints for scalability.

Consequently, determining whether an external data upload is unusual by looking for previous uploads to the same endpoint is not sufficient, because external services may have additional endpoints that have been observed but not associated with the service. Therefore, the service itself needs to be characterized.

We can characterize a service by finding data-upload connections with common properties, such as identical or similar hostnames, identical JA3 client hashes, or identical ASNs. By identifying the dominant transfer endpoints involved in a single upload event, remaining external connections made by the device can be successively restricted based on various properties of those in the dominant group, until a plausible characterization of an external service is made.

We can then search for previous connections and uploads to the same service using these properties rather than specific hostnames or endpoints and associate observed endpoints with services, even if the hostnames are different.

This prevents legitimate uploads to commonly-used services from being identified as possible cases of exfiltration, even if these uploads are not always to the same endpoint, while also enabling precise characterization of malicious exfiltration patterns.

Identification of cryptomining credentials and their use in differentiating between insider threats and wide-spread malware

Dr Andrew Woodford
How global analysis of cryptomining credentials can identify malicious criminal organizations leveraging corporate computing power for financial gain.

Most cryptocurrencies work via a blockchain secured using a proof-of-work model. In the proof-of-work model, it takes significant computational effort to produce the next block in the blockchain, but whoever produces the next block first is rewarded with some of the cryptocurrency associated with that blockchain.

Cryptocurrency mining is the race to produce the next block in a cryptocurrency’s blockchain and earn the reward. To do this, individual miners group together to distribute the effort of producing the next block and share the reward if one of the group is the first to do so. To track who deserves part of the reward and how big their part should be, miners send credential information to the pool when they register.

Our research aimed to identify mining credentials and use them to differentiate insider threats from wide-spread malware. We extracted user credentials from network traffic using Deep Packet Inspection and our knowledge of crypto-mining communication protocols, such as getblocktemplate or stratum.

We found that credentials usually include a cryptocurrency-wallet or email address. Sometimes, email addresses tell us something about the person or group responsible for the mining device.

We also compared each credential to all others seen across all mining traffic. We found that some credentials appear in a range of unrelated compromises, which suggests they are likely associated with widely-spread malware. By contrast, credentials that appear only once are more likely to result from an insider compromise. As we can now track observed mining credentials, we can also detect when new credentials are used.

Robust identification of ransomware encryption over SMB

Dr Tim Bazalgette
How the high-level analysis of SMB operations, files, and properties can identify ransomware infections.

Ransomware encrypts all files accessible to an infected device, including those on SMB network shares.

As a result, repeated patterns can be observed in network traffic, such as the read of a file, followed by a write of the same file with a new extension, typically accompanied by a MIME-type change given the encryption.

However, SMB is a notoriously chatty protocol, and for backwards-compatibility reasons, offers a wide range of commands for accomplishing the same task. For instance, a write may be accomplished by WRITE, WRITE_RAW, WRITE_ANDX, WRITE_AND_CLOSE, etc. The chattiness of the protocol may also mean that the write of a specific file may be distributed across multiple commands, not all of which may come in an expected order.

The ransomware may also encrypt the file in different ways (e.g. by first renaming, then encrypting, or vice versa), which often makes it difficult to reliably identify possible steps of encryption on a file-by-file basis.

Consequently, instead of trying to identify ransomware on an operation-by-operation basis, it is more fruitful to analyze higher level activity, beginning with the end goal of the ransomware: encryption.

This may be reflected by a number of higher-level actions on files, including MIME-type changes, and the addition of suspicious extensions. By taking all filenames observed in the SMB session and sorting them alphabetically, we obtain a list of pairs of similar filenames. These pairs can then be sorted by the time at which they were first observed, and the resulting pair represents a potential encryption step.

The two files can be compared to reveal MIME-type and extension changes, among other things. We can then analyze the general statistics of these properties for the entire session, and robustly identify ransomware encryption, in a manner that is not affected by the low-level complications of the SMB protocol.

Matryoshka Bloom filters quickly and efficiently determine rarity

Dr Matthew Ferguson
A novel data structure for the efficient storage and retrieval of statistics on how common strings are in specific environments.

We consider the rarity of certain strings when analyzing the behavior of devices and users. The strings might be hostnames or domain names in HTTP requests, email addresses, or executable names. We can determine their rarity by observing network traffic, making lists of known entities over time, and comparing new observations to old.

Matryoshka Bloom filters form the basis of a new method to determine rarity by finding out how often a string appears in these lists. A Bloom filter is a method of probabilistically determining if a specific string is a member of a set. A Bloom filter contains an array of bits and computes several hashes for each string to produce a list of indices that point to elements of this array. If these elements are all ones, it is highly probable that the string appears in the set. If a single bit is set to zero, the string does not appear in the set. The Bloom filter uses much less memory than a hash set would use to determine set membership, at the cost of a small but non-zero probability of a false positive. Choosing an appropriate set of filter parameters (such as array size and number of hashes) can reduce this cost.

However, a standard Bloom filter can only give a binary answer to the question of membership. By structuring the popularity data as a sequence of nested sets, each of which is a subset of the next (like Matryoshka dolls), we can construct a series of Bloom filters. These filters are able to give each string a rarity score when queried (between zero and the number of “popularity” sets available). This structure increases the fault tolerance of the filters and reduces the scope for false positives. Additionally, the small size of the filters allows us to distribute the information needed to perform these lookups among devices on a distributed deployment. This reduces bandwidth requirements to keep our rarity data up-to-date.

Securing tenant data in the public cloud

Alex Smith
Secure practices to ensure corporate data sitting within public servers such as AWS is sufficiently isolated, in the event of a data breach targeting these public cloud providers.

If all customer requests were sent to a multi-tenant service hosted on the same instance, and that instance was compromised, the resulting security incident would be extensive, and it would affect all tenants.

When using a public cloud provider and a Software as a Service (SaaS) solution with multiple tenants, data flow needs to be secure. As the cloud provider and the consumer share responsibility for security, consumers must take care not to use public-cloud services in a way that undermines data segregation or otherwise reduces our overall security.

Taking Amazon Web Services (AWS) and the Darktrace for Endpoint product as an example, each tenant receives its own container (hosted in ECS Fargate) to ensure absolute separation of data. The container has a DNS entry that is aliased to an AWS Application Load Balancer, which allows data to flow directly to an isolated, customer-specific container and service for processing. Using ECS Fargate decreases costs (as customers pay for a small fraction of container runtime) but ensures that a system breach would be isolated to a single tenant.

This technology has two important benefits:

  1. Potential compromises would be confined to a single tenant.
  2. The transient nature of this type of immutable infrastructure means that attackers must pursue a moving target.

Of course, there are many other security measures in place to prevent the compromise of an instance (container in this case) in the first place, and this technology is just one measure that Darktrace takes to protect customers from this unlikely but very serious situation.

Analysis of email structure to detect malicious intent

Dr Antony Lawson
Using AI to determine whether an email is trying to induce the reader to do something that would lead to compromise, such as clicking on a link or opening a file.

Malicious emails usually try to induce the recipient to take a particular action. For example, extortion emails may try to force the recipient to make a cryptocurrency payment, and phishing emails may tempt the recipient to click a link that points to a malicious payload or a fake login screen.

Looking for such emails by watching for specific content, addresses or domains is a poor strategy because these features change over time. The content might change to capitalize on a contemporary topic, such as the COVID-19 pandemic, and they are sent from different addresses to circumvent blocklists.

We have created a classifier that analyzes incoming email and assigns a score in four categories of inducement: extortion, solicitation, phishing, and other spam.

This is achieved by analyzing email structure and non-specific content. Structure variables include, among other things, the number of sentences, average sentence length, average paragraph length, and the number of characters that come before the first link. Content variables include the number and density of hyperlinks, references to currency, HTML tags, non-standard punctuation, etc.

Some language processing also identifies words and phrases associated with each category and contributes to the four scores.

This approach deals with new content and new email addresses better than many other approaches and is demonstrably effective. For example, this classifier has identified many phishing emails related to COVID-19 despite being trained on data from before the pandemic. One email encouraged employees to log in and contribute to their company’s COVID relief fund.

Additionally, a profile of typical behavior can be developed for each sender by tracking these scores over time. By comparing inducement scores of a new email to the scores of previous emails (using both probabilistic and substantiality metrics) we can obtain an “inducement shift” score. We can pass this score to an autonomous-response platform as a supplement to the inducement score, because a large shift might indicate an account takeover and deserves a more robust response.

Programmatically monitoring disparate SaaS environments

Jacob Araiza
Optimizing the use, visibility and security of environments that span multiple complex cloud and SaaS environments.

Security blind spots have developed outside of the local enterprise network as organizations embrace cloud applications and enterprise networks become more distributed. SaaS applications are one such blind spot, because data on user interactions with SaaS applications may contain critical security insights that are not accessible to IT security teams.

Where these data are accessible they come from a number of third-party providers, but each provider focuses on a different aspect of the data, has their own approach to security and auditing, and has a different API.

Darktrace SaaS Modules aim to illuminate this blind spot by retrieving and combining disparate data.

As an example, Salesforce make audit data on interactions with system records (such as create, delete) available from an API on a per-object basis. The dataset updates in near real time, though it captures only a small sample of user behavior.

Salesforce also creates chunked event logs of certain file interactions (such as downloads), but only makes them available for download at large intervals. So, to get a fuller picture of Salesforce data it is necessary to retrospectively associate these events with the time-series data on record interactions.

SaaS Modules use intelligent hashing to detect duplication and combine metrics from many events into a single notice that better represents a user’s actions. Then, they assign metrics to each notice using an extractor framework to provide normalized data, which can be analyzed by Darktrace’s machine-learning algorithms when combined with existing local-network data.

Combining data from SaaS applications with data from the local network provides a data set that is richer than the sum of its parts. It enables detection of malicious activity within individual SaaS applications, and of activity that crosses from one SaaS application to another, or from a SaaS application to a local network.

Anomalous behaviors within SaaS applications can be detected, irrespective of login location, to give security operators better insight into emerging threats more quickly.

Identifying compromises with graphs of high-precision security metadata

Dr Tim Bazalgette and Dr Dickon Humphrey
Using directed graphs of lower-level security events to reliably discover complex security incidents.

Security alerts are frequently oriented around a single entity or device, identifying the specific suspicious activity observed and its source. However, major compromises are rarely limited to a single entity, and instead spread through the network, exhibiting a wide variety of techniques and possible indicators of compromise.

Piecing together this activity to identify its scope and any additional properties, such as a possible patient zero or any C2 endpoints, is typically the work of a human analyst. Various tools can represent specific events — such as network connections, SaaS actions, or DNS queries — in a graph format, which can help with this analysis, but it remains a difficult task to analyze these low-level events and determine which are likely to be associated with the compromise.

However, systems such as AI Analyst produce more precise, higher-level alerts for security events. For instance, rather than identifying that a device connected to three suspicious endpoints 27 times and data was transferred, AI analyst might produce a single exfiltration event highlighting the source device and all destinations.

If these higher-level insights are represented on a directed graph, then each edge is significantly more succinct and can be treated with much higher confidence. Possible compromises can then be identified through analysis of this graph, with key properties such as patient zeros and C2 domains being straightforward to read from a subgraph associated with the compromise.

This allows for the automated detection of compromises, and the automated determination of their full scope, without human attention.

Structural construct detection as a variable indicator of compromise

Stephen Pickman
Detecting email spoofing attempts by identifying unusual patterns in HTML and CSS.

Modern emails are formatted using HTML and CSS, allowing the writer to apply structure, layout, and branding that would not be possible using plain text. Many companies issue style guides and templates for external or official communications that prescribe fonts, text size and other aspects of layout.

Use of HTML/CSS introduces complexity to the email space, and with it the potential for exploitation. For example, malicious emails often use legitimate CSS styling, or user-invisible text to conceal a malicious payload and mimic a known company layout.

We have found that legitimate email communication broadly falls into three categories:

  1. Internal communication using plain text or simple HTML (interpersonal communications)
  2. External communication using complex HTML with recurring structures and styles (such as email signatures)
  3. External communication using complex HTML with full set templates and styles (such as newsletters and announcements)

The categories are characterized by features, such as the frequency of CSS appearance, frequency of HTML node appearance, and HTML tree depth. A classifier can use these categories to further direct feature extraction and tracking. The ability to quantify the complexity and style of a HTML document, and to track changes over time or against a model, allows the detection of anomalous and potentially malicious email communications.

This approach has been incorporated into the Darktrace Antigena Email product and contributes to detecting account takeovers and behavioral anomalies.

Gaussian constructions on surfaces for anomalous event identification

Dr Tim Bazalgette and Dr Dickon Humphrey
Determining anomalies by creating probability density functions on different surfaces corresponding to the properties of interest.

Anomalous events are often an early sign of a security compromise, and the nature of these anomalies can take multiple forms. For example, actions in AWS environments are associated with a broad range of metadata — in addition to the action itself, properties including the user, user agent, source IP address, ASN, and timestamp. Ideally, then, we should be able to use similar events in the past to identify potentially anomalous events, such as connections from an unusual location, or at an unusual time.

However, there may be only a small number of prior events associated with particular combinations of metadata. For instance, a given user and user agent might only be associated with a handful of AWS actions in the past month. Moreover, identifying typical behavior requires an understanding of cycles and other patterns in the data involved. It is useful to understand, for example, whether an action normally occurs at a certain hour of the day, or a certain day of the week.

These inherent relations can be investigated when they are represented on an appropriate surface. Trivially, geolocational data can be mapped to a sphere, but timing data may also be mapped to other surfaces such as a circle or a torus, depending on how many kinds of cyclicity it is necessary to represent.

A Gaussian (or its equivalent for the surface, e.g. a Kent distribution for a sphere, or a von Mises distribution for a circle) can then be constructed at each past observed event. Combining them produces an effective probability density function for events on the surface. New events can then be assessed for their anomalousness based on this function.

This technique allows for the generation of spatial and temporal heatmaps for events given any number of past observations and can rapidly identify anomalous events even with minimal data. Darktrace uses this technique to highlight activity in the map view of the Threat Visualizer’s SaaS console and is also used throughout AI Analyst.

Detecting and preventing misdirected emails with correspondence semantics

Ben Akrill
Identifying if the recipient of a particular email appears correct in the wider context of user activity.

Data breaches do not always start with malicious actors, they are sometimes the result of human error. Misdirecting an email is a risk for every employee and every organization. Detecting and preventing misdirected emails has the potential to stop accidental personal data breaches before they happen.

Detecting misdirected emails involves more than checking whether the sender and the recipient are known correspondents. Either through a simple mistake, or misplaced trust in the address field autocomplete, it is easy to misdirect an email and send it to a previous correspondent. Accurate classification must also consider the content of the message.

A practicable solution must also overcome two challenges. First, the training set for confirmed positive matches is very small. Misdirected emails are both uncommon and embarrassing, making classifier supervision challenging.

Second, response time is critical to functionality in this case. Intervention must happen before the email leaves the sender’s outbox. However, maintaining an acceptable response time is problematic because tracking correspondence within networks of varying size requires dynamic scaling.

We have made a classifier that tracks semantics shared over a network of correspondents.The classifier detects anomalous messages by comparing message content to that of previous communications between sender and recipient. It uses unsupervised learning to detect anomalies without relying on reported and confirmed breaches and produces anomaly scores that are weighted against previous correspondence from the entire network. Complexity is constant with the number of correspondents, but memory scaling is sublinear.

The classifier responds quickly enough that Antigena (Darktrace’s Autonomous Response technology) will be able to warn the sender of possible misdirection and ask for confirmation. It might also be used to suggest more likely recipients, by finding similar addresses with a lower anomaly score, to which Antigena can divert the message with a single click.

Analyzing network activity to detect compromised devices sending spam emails

Dr Andrés Curto Martín
Distinguishing between genuine and illegitimate email traffic from network servers.

Some malware exploits its victims by sending large numbers of spam emails. This behavior is also observed on devices controlled by malicious external actors, and sometimes by unauthorized internal actors using corporate devices for their own interests. If corporate devices persistently send spam, the corporate domain or public IP addresses may eventually be included in known spam lists, e.g. SpamHaus, interfering with corporate email or even causing reputational damage.

It is difficult to detect compromised devices sending spam because network connections related to spam are usually short-lived and can occur randomly, and because legitimate corporate email campaigns or peaks of activity in email servers can be misinterpreted.

We use a supervised machine-learning method to automatically analyze outgoing messages and determine whether they are legitimate or spam. The method analyzes the frequency of outgoing messages from a given device, several properties of outgoing messages, and other variables related to the principle of locality.

For example, a server performing legitimate activity tends to connect to a predictable number of endpoints, and an anomalous increase might indicate that the device has been compromised. The method also analyzes email subject fields to look for suspicious terminology, and the list of recipients to look for large numbers of recipients, and non-corporate or generic free email services.

Preliminary results demonstrate highly accurate detection of compromised devices sending spam, computational efficiency, and the ability to detect outgoing spam almost immediately.

Using graph theory to identify critical nodes within computer networks

Darktrace researcher, PhD
How graph theory can be used to map out cross-domain, realistic, and risk-assessed attack paths across an entire digital enterprise

In an already under-resourced cyber security industry, demand for talent is currently much greater than supply. While understaffed and under-resourced blue teams try to defend increasingly large networks, the red teams that might have the insight to direct the resource allocation are infrequently used because red-team exercises are expensive and non-exhaustive. The result is a blue team that becomes decreasingly effective over time but periodically (and non-exhaustively) corrected by expensive insights from external red teams.

One way to overcome these problems is to model attack paths in real-time. That way, blue teams would have continual insight and may continuously adapt their approach to defending the most critical network assets without the need for expensive external input. In short, the solution is to automate an internal red team.

Our method constructs two weighted graphs to show pair-wise relations between network entities that might be compromised, such as devices and user accounts.

First, a graph is drawn with directed edge weights representing the estimated probability of rapid lateral movement from the source to the destination entity. For example, if a device has well-established communication pathways to a server with a high CVSS score, then the edge weight will be closer to one. Edges also consider intrinsic mechanisms that enhance security, such as multi-factor authentication, endpoint-protection agents, or even just a more security-aware user.

Then, to form a second graph, objective importance scores are either manually or automatically seeded and propagated through the graph via edges weighted according to shared access or trust relationships. For example, if the CEO of an organization has access to a file shared with only one other employee—some of the importance associated with the CEO is propagated to this other user.

If the CEO has access to a file that many other users can access, the importance of the CEO is diluted amongst the many users, suggesting that this file is not especially important. When available, the graph also includes email communication patterns.

We use these graphs to simulate the compromise of all potential network entry points—including any human with access to the internet, as well as externally-facing infrastructure. The simulation yields impact scores that correlate to path lengths to high-importance nodes. The scores can be modulated according to how exposed an entry point is to an outsider.

This results in a dynamic list of network nodes, ordered by the potential damage to the organization if compromised at the current time. The paths to these nodes were also highlighted, allowing the blue team to remediate accordingly. Compared to traditional red team exercises, this method is continuous, rigorous, and cost-effective.

Darktrace researchers

Our research team comprises experts in a wide range of subjects, some of whose specialisms are described in more detail below.

PhD in Mathematics
University of York
Specializes in mathematical physics, anomaly detection in complex data to improve efficiency, and optimization and stability in data processing.
PhD in Mathematics
University of Bath
Specializes in applications of AI to endpoint and cloud environments and network traffic analysis.
PhD in Cosmology
University of Cantabria
Specializes in developing novel advanced statistical techniques in data analysis and software engineering for supercomputers.
PhD in Theoretical Linguistics
University of Cambridge
Specializes in algorithmic modeling of cognitive representations and acquisition of natural language syntax.
Master’s in Engineering
University of Oxford
Specializes in applications of machine learning techniques in bioengineering, applying neural networks to extract structures in medical imagery, and investigating novel phishing techniques.
Master’s in Physics
University of Cambridge
Specializes in the application of nuclear magnetic resonance to manufacturing processes, and the application of machine learning to galaxy clustering.
PhD in Statistics
Durham University
Specializes in non-parametric modeling, statistical and optimization problems, dimension reduction and classification problems.
Master’s in Mathematics
Durham University
Specializes in mathematical biology and mathematical modeling, numerical analysis and adversarial attack simulations.
PhD in Biochemistry
University of Cambridge
Specializes in neuroscience and behaviour modeling, machine learning to identify novel threats, and automated healing of computer systems following a digital compromise.
PhD in Mathematics and Physics
University of Cambridge
Specializes in applications of machine learning and statistical analysis, clustering techniques and the use of neural networks for the classification of categorical data.
PhD in Astrophysics
University of Cambridge
Specializes in general automation of complex, typically human-driven processes with current focus on adversarially-oriented preventative security.
Bachelor of Science
Stanford University
Specializes in electrical engineering including analog circuit design and organic semiconductors.