Darktrace cyber analysts are world-class experts in threat intelligence, threat hunting and incident response, and provide 24/7 SOC support to thousands of Darktrace customers around the globe. Inside the SOC is exclusively authored by these experts, providing analysis of cyber incidents and threat trends, based on real-world experience in the field.
Written by
Dan Fein
VP, Product
Share
02
Jul 2020
Introduction
Many organizations turn to AI to augment their email security only after a damaging attack serves as a wake-up call. Such was the case recently with an academic institution in the APAC region. The organization felt fully protected at the time, with spam protection, URL protection, and the full Mimecast package across the email realm. However, it was interested in understanding how Cyber AI might defend its inbox, and scheduled a conversation to learn more.
Two days before that conversation was due to take place, the company was hit by a Business Email Compromise. An attacker had taken over an internal Microsoft 365 account and sent a fraudulent invoice to the organization’s accounts department. The invoice, which contained subtly edited bank details, claimed to be from Siemens, one of the world’s leading automation technology companies. Given that Siemens was an established supplier, the attack succeeded – and the academic institution unwittingly paid over $60,000 into an attacker’s bank account.
The incident alerted this organization not only to the scale and severity of email attacks, but also the limitations in its existing security tools. The security team swiftly deployed Antigena Email, and the AI’s ability to neutralize the advanced attacks missed by other tools was immediately realized when the attacker tried again a week later.
On this occasion, with the AI now in place, unusual behavior on the compromised SaaS account together with the suspicious email activity caused several Darktrace models to fire, culminating in a 73% anomaly score and the decision to autonomously hold back the offending email. The below details the timeline of this attack, and how Darktrace was able to identify an attack where other security tools failed.
Patient Zero: the initial compromise
Darktrace first detected something was amiss when a SaaS login was seen from an unusual IP address located in the UAE.
Tue April 21 04:15:40 (local time)
It then noticed a behavior commonly associated with Business Email Compromise – the creation of an inbox processing rule so that any incoming emails from the vendor would be deleted and redirected to the threat-actor.
Tue April 21st 05:04:21
Wed April 22nd 16:51:14
Figure 1: Darktrace detecting the anomalous mailbox rule, indicating a 97-100% anomalous activity
This initial compromise led to a very targeted spear phishing campaign. The attacker found an authentic email chain regarding an invoice from the domain siemens.com. Using the trusted supplier’s reputation, they copied the exact format of the invoice and then created a spoofed domain ‘siemesm.com’, intending to target the accounts team at the organization with a fresh invoice – but with one important detail changed – the bank details.
The attacker was careful. Before launching the attack, they sent a test email to the compromised account to ensure it was getting through. As seen below, this alerted Antigena Email with a 38% anomaly score.
The fraudulent invoice
The following morning, the attacker set up some fake correspondence between the newly created account and Patient Zero, before looping in the accounts team with a new request – this time, asking for $78,000. Again, the exchange appeared legitimate: from a well-known supplier, and with the apparent ratification of a trusted colleague.
Figure 2: Antigena Email detecting the anomalous emails, with the red hold icon indicating that it held the emails back from the recipient
Here, we can see Antigena Email growing gradually more suspicious of this unusual behavior over time, with the highlighted ‘hold’ icon indicating that Antigena Email prevented the two offending emails from being delivered. The high anomaly score was influenced not only by the lack of previous correspondence between the two email users, but also the email rules that were made on the account itself the previous day. Furthermore, Cyber AI recognized the email as a fake reply: not a direct response to the email, but one which attempted to replicate it by copying and pasting correspondence from a separate email chain.
Realizing their initial attack had failed, the attacker then leveraged a technique known as ‘island hopping’, harvesting the company-wide contact list, and going on to launch a more generic phishing campaign to dozens of email users across the company – hoping in turn to compromise their accounts.
Figure 3: A selection of the subsequent emails sent from the initially compromised account
Again, Antigena Email deemed every one of these emails to be 100% anomalous and held them back in every case.
Whilst this was a relatively advanced attack with multiple stages, the techniques described above represent nothing exceptional – Antigena Email stops these kinds of attacks on a daily basis, in hundreds of organizations around the globe.
By deploying Cyber AI into their email environments, defenders not only have unrivalled visibility over internal and external mail flow, but the ability to stop targeted and sophisticated attacks, well before they can escalate into a crisis.
Darktrace cyber analysts are world-class experts in threat intelligence, threat hunting and incident response, and provide 24/7 SOC support to thousands of Darktrace customers around the globe. Inside the SOC is exclusively authored by these experts, providing analysis of cyber incidents and threat trends, based on real-world experience in the field.
“I wonder if running that tool influenced the connectivity in some way.”
“I might need to target down or reboot, but first, I'll investigate the network using ARP. I wonder if ARP is responding properly. If I see "ip neigh stale," that could indicate a problem. I think arping might help in this situation, but I’ll need to keep an eye on how the network is behaving. It's all about figuring out where the issue lies!”
図10：エージェントの異常なLDAPアクティビティに対してトリガーされたこのLinuxサーバーに対するモデル検知は、Cyber AI Analystによって自律的に調査され、その調査結果にはエージェントの異常なSMBセッションアクティビティに関する内容が含まれ、Cyber AIAnalystインシデントに記録されました。
DarktraceのCyber AI Analystは、エージェントのアクティビティによってトリガーされたモデル検知を自律的に調査し、その調査結果を1つの重要なインシデントレポートにまとめました。インシデント報告書の内容は、エージェントによるネットワークスキャンから始まりコーディング演習内容のSMBベースの上書きで終わる、エージェントの行動の各段階を詳しくカバーしています。
図20：Cyber AI Analystは、モデル検知に対する自律的な調査を通じて得られた所見をつなぎ合わせ、エージェントの進行状況のタイムラインを作成し、アクティビティを視覚的に表したグラフとともに提供しました。
Cyber Verification Program下のAnthropic Opus 4.6モデルなど、他のフロンティアモデルを使用したエージェントも、Darktraceのシミュレートされた環境で同じ設定とタスクを使ってテストされました。反復の度に、エージェントは演習の内容または採点プログラムを変更して不正を行おうとしました。基盤となるモデルに関係なく、エージェントは自分が置かれた環境を侵害することで、100％のスコアを達成する方法を見つけることに成功していました。これらのエージェントによるハッキングは、アクティビティの内容こそ違っていましたが、すべてその環境のベースラインの振る舞いから大きく逸脱した動作を引き起こしていました。これらの逸脱はDarktraceプラットフォームに十分な検知の機会を提供し、エージェントの逃げ場はありませんでした。
エージェンティックハーネスは会話履歴をローカルに保存しますが、ダークトレースの研究者たちは、保存されたAIの応答が本当にモデルによって生成されたものであるかどうかの検証が行われていないことを発見しました。また、この設計上の選択がAnthropic Claude Code、AWS Kiro-CLI、OpenAI Codex、およびオープンソースのPiにわたって共通していることを確認しました。
会話履歴は多くの場合クライアント側に保存されます。例えば、Anthropic Claude Code、OpenAI Codex、AWS Kiro-CLI、Piなどでもそうです。そのため、ユーザーはセッションを自由に再開でき、一部の製品にはその履歴を操作する機能が組み込まれています。たとえば、会話の中の特定の時点に戻り、送信したメッセージを編集して、会話を別の軌道へ進めていくことができます。きわめて重要な点として、私たちが調査したすべてのケースにおいて、保存されているAIの応答が対応するモデルから生成されたものであり改ざんされていないことの検証は行われていませんでした。
