ブログ
/
Cloud
/
November 19, 2025

生成AIの保護: Darktrace / CLOUDでAmazon Bedrockのリスクを管理する

Amazon Bedrockのような生成AIサービスは、アクセス、可視性、データ露出に関連した新たなリスクをもたらしつつあります。 本稿では、Darktrace / CLOUDがBedrockおよびSageMaker環境において、コンフィギュレーションに対する深い可視性、権限の分析、設定ミスの検知、挙動の異常の検知により、これらのインシデントを防ぐのにどう役立つかを解説します。
Inside the SOC
Darktrace cyber analysts are world-class experts in threat intelligence, threat hunting and incident response, and provide 24/7 SOC support to thousands of Darktrace customers around the globe. Inside the SOC is exclusively authored by these experts, providing analysis of cyber incidents and threat trends, based on real-world experience in the field.
Written by
Adam Stevens
Senior Director of Product, Cloud | Darktrace
Default blog image
19
Nov 2025

企業内生成AIのセキュリティリスクと課題

生成AIとAmazon Bedrockのようなマネージド型基盤モデルプラットフォームは、組織がインテリジェントなアプリケーションを構築し、展開する方法を大きく変化させています。チャットボットから要約ツールまで、Bedrockは基盤モデルを企業のデータとサービスに接続することにより、迅速なエージェント開発を可能にします。しかしこの柔軟性にはさまざまなセキュリティ課題が伴い、特に可視性、アクセス管理、そして意図しないデータ露出に関連したリスクがあります。

組織が生成AIの業務への導入を急ぐ中で、従来型のセキュリティコントロールは対応に遅れが目立ちます。Bedrockのエージェント、モデル、ガードレール、そしてベースとなるAWSサービスからなる多層的アーキテクチャは、標準的なポスチャ管理ツールでは想定されていなかった新たなブラインドスポットを作り出しています。可視性のギャップにより、エージェントがどのデータセットにアクセスできるのか、あるいはモデルの出力が機密性の高い情報を露出させる可能性がないかを知ることが難しくなります。その一方で、開発者はセキュリティチームがIAM権限を確認したり、ガードレールを検証したりできるよりも速いペースで進むことが多く、リスクの拡大につながる設定のミスが起こりがちです。AWSのような共有責任モデルにおいては、この複雑性によってオーナーシップの境界があいまいになる可能性があり、セキュリティチームにとってAIシステムが組織のデータとどのように相互動作しているかについて、情報を継続的かつ自動的に得られることがきわめて重要になります。

Darktrace / CLOUDはBedrock環境に対して包括的な可視性およびポスチャ管理を提供し、エージェントとナレッジベースを自動的に検知し積極的にスキャンすることにより、テクノロジーの拡大とイノベーションのペースを落とすことなく、AIインフラの保護に貢献します。

現実のシナリオ:行き過ぎたアクセス

たとえば、会社のナレッジベースを使用しビジネス上の質問にスタッフがすばやく回答できるようにするためのBedrockエージェントを展開しているとします。エージェントはAmazon S3に格納されている文書を参照するナレッジベースに接続され、APIを介して社内のサービスへのアクセス権を与えられています。

システムを早期に稼働させようと、開発者はエージェントに幅広い実行権限を持つロールを割り当てました。このロールは複数のS3バケットに対するアクセス権を付与されており、バケットの1つには機密性の顧客情報が含まれていました。この過剰な権限付与は悪意によるものではありませんでした。IAMポリシー作成の複雑性と、どのバケットに機密性の高いデータが含まれているかを特定するのが難しかったことが原因です。

チームはエージェントが意図した文書だけを使用すると思っていました。しかし、従業員がどのようにエージェントとやりとりするか、あるいはエージェントがどのようにデータを処理する可能性があるかについては十分に検討がされませんでした。  

ある従業員が顧客の四半期のアクティビティについていつものように質問をしたところ、エージェントは規制対象データを含む情報を出力し、適切なアクセス権を持たない人に開示してしまいました。

これはプロンプトインジェクションやモデルの不正操作が行われたケースではありません。エージェントは単に指示に従い、アクセスを許可されているリソースを使用したにすぎません。この開示はIAMポリシーに適合していましたが、まったく意図とは異なる結果となりました。

Darktrace / CLOUDによってこれらのリスクがどう防止されるか

Darktrace / CLOUDはBedrockおよびSageMaker環境に対して多層的な可視性とインテリジェントな分析能力を提供することで、意図しないデータ露出のようなシナリオを回避することができます。それぞれの機能は次のように使用されます:

コンフィギュレーションレベルの可視性

Bedrock環境にはしばしば複数のコンポーネント、たとえばエージェント、ガードレール、基盤モデルが含まれ、それぞれがコンフィギュレーションを持っています。Darktrace / CLOUDはこれらのコンフィギュレーションをインデックス化し、チームは次が可能になります:

  1. 展開されたエージェントを検査しそれらが承認されたデータソースにのみ接続されていることを確認する。
  2. 評価ジョブのセットアップおよびそれらのAmazon S3データセットへのリンクを追跡し、機密性の高い情報を露出させる可能性のある隠れたデータフローを明らかにする。
  3. すべてのAIコンポーネントに対する認識を維持し、見落としたアセットからリスクが発生する可能性を縮小する。

Bedrock、SageMakerおよびその他のAWSサービス全体のコンフィギュレーションデータを一元的に管理することでDarktrace / CLOUDはAIアセットの可視性に対する信頼できる唯一の情報源を提供します。チームは各コンポーネントがどのように設定されているか、および社内のセキュリティポリシーに合致しているかどうかを即座に確認することができます。これにより当て推量を排除し、監査を加速し、設定の不整合がデータ露出リスクを生むのを防止することができます。

 Agents for bedrock relationship views.
図1:Bedrockとエージェントの関係

アーキテクチャの認識

複雑なAI環境ではコンポーネント間の相互動作を理解するのが難しいことがあります。Darktrace / CLOUDはリアルタイムのアーキテクチャダイアグラムを作成することにより:

  1. エージェント、モデル、データセット間の関係を可視化します。  
  1. 相互接続されたサービス間の意図しないデータアクセス経路やリスクの伝播を特定します。

これにより、セキュリティチームは脆弱さが露出につながる前にそれらを発見することができます。これらの関係を動的に可視化することにより、Darktrace / CLOUDはプロアクティブなリスク管理を可能にし、アーキテクチャのドリフト、冗長なデータ接続、あるいは監視されていないエージェントを、攻撃者が悪用したり偶発的な誤使用が起こる前に発見することができます。これにより調査にかかる時間を短縮するとともに、AIワークロード全体のコンプライアンスへの自信を高めることができます。

Figure 2: Full Bedrock agent architecture including lambda and IAM permission mapping
図2:lambdaおよびIAM権限マッピングを含むBedrockエージェントアーキテクチャ全体図

アクセスおよび権限の分析

IAM権限はBedrockを含むあらゆるAWSサービスに適用されます。Bedrockエージェントが他のワークロードに対して広範に定義されたIAMロールを引き受けるとき、しばしば過剰な権限を継承します。最小権限のコントロールを厳密に行っていなければ、エージェントは必要なものよりも格段に多くのデータやサービスにアクセスできる可能性があり、防げるはずだったセキュリティ露出を作り出してしまいます。Darktrace / CLOUDは:

  1. 実行ロールおよびユーザー権限をレビューして過剰な権限を特定します。
  2. 権限昇格や承認されていないAPIアクションを可能にする可能性のある異常ににフラグを立てます。

これによりエージェントが最小権限の原則の枠内で運用されるようにし、アタックサーフェスを縮小することができます。リスクの高いロールを特定することに加えて、Darktrace / CLOUDは通常のアクセスのパターンを継続的に学習し、権限が悪用されたり、拡大されたりした場合にリアルタイムに識別することができます。セキュリティチームは、アクションがなぜ異常なのか、およびそれが接続されているアセットにどう影響する可能性があるのかについてのコンテキストを理解し、推奨された具体的な対策を取ることにより、生産性を維持しつつ露出を最小化することができます。

設定のミスの検知

設定ミスはクラウドセキュリティインシデントの主要な原因の1つです。Darktrace / CLOUDは以下を自動的に検知します:

  1. 機密性の高いトレーニングデータが含まれているかもしれない、公開アクセス可能なS3バケット
  2. 不適切なまたは機密情報を含む出力を許可する可能性のある、Bedrock環境のガードレール不足  
  3. 暗号化の欠如、直接インターネットアクセス、モデルへのrootアクセスなどその他の問題  

これらのリスクを早期に明らかにすることにより、チームはこれらが悪用可能になる前に修正を行うことができます。Darktrace / CLOUDは人手で行っていたレビューのプロセスを、自動化された、継続的なチェックに変え、発見までの時間を短縮するとともに、小さな見落としが大規模なインシデントにエスカレートするのを防止することができます。このような自動的な確認により、組織はAIシステムのコンプライアンスを維持し、安全を組み込んだ設計を維持しつつ、自信を持ってイノベーションを進めることができます。

Configuration data for Anthropic foundation model
 図3:Anthropic基盤モデルのコンフィギュレーションデータ

ビヘイビアベースの異常検知

コンフィギュレーションが正しい場合にも、その動作が脅威の発生の兆候を示すことがあります。AWS CloudTrailを使用して、Darktrace / CLOUDは:

  1. エージェントが予期しないデータセットをクエリーしているなど、通常と異なるデータアクセスのパターンを監視します。
  2. モデル汚染攻撃の試みかもしれない異常なトレーニングジョブの起動を検知します。

こうしたリアルタイムのビヘイビア分析により、組織は疑わしいアクティビティにすばやく対応することができます。それぞれのBedrockコンポーネントの"正常な”動作を継続的に学習することにより、Darktrace / CLOUDは正式な侵害インジケーターが発生する前に、脅威を示すものかもしれない微妙な変化を検知することができます。その結果、より早期の検知、調査の工数の削減、そしてAI駆動のワークロードが意図通りに機能することを継続的に保証することができます。

まとめ

生成AIはビジネスを変革するさまざまな機能を提供しますが、イノベーションと共に変化しつづける複雑なリスクも伴います。Amazon Bedrockのようなサービスの柔軟性は新たな効率化や理解を可能にしますが、正しい利用であっても意図せずに機密性の高いデータを露出させたり、セキュリティコントロールをすり抜けてしまう場合があります。多くの組織がAIの大規模な導入を進めるなかで、開発を遅らせることなくこれらの環境を包括的に監視し保護する能力はきわめて重要になってきます。

コンフィギュレーションに対する深い可視性、アーキテクチャの理解、権限と動作の分析、そしてリアルタイムの脅威検知を組み合わせることにより、DarktraceはBedrockやSageMaker等のAIツールに対する継続的な保証をセキュリティチームに提供します。組織は適応型のインテリジェントな保護によりAIシステムが管理されているという安心感を持ってイノベーションを続けることができます。

[related-resource]

企業内のAIを防御する方法についてさらに知る

組織を新たなアタックサーフェスに露出させることなく、AIによるイノベーションを安全に実現するための方法とは?ホワイトペーパーをお読みになり、AIが原因となる各種リスクを見つけ出す方法をご確認ください。

Inside the SOC
Darktrace cyber analysts are world-class experts in threat intelligence, threat hunting and incident response, and provide 24/7 SOC support to thousands of Darktrace customers around the globe. Inside the SOC is exclusively authored by these experts, providing analysis of cyber incidents and threat trends, based on real-world experience in the field.
Written by
Adam Stevens
Senior Director of Product, Cloud | Darktrace

More in this series

No items found.

Blog

/

AI

/

July 17, 2026

AI Is Taking on Stadium Operations. How Can Security Teams Keep it Protected?

Default blog imageDefault blog image

How to Secure AI in Stadium Operations

Key takeaways

  • AI is entering high-impact stadium functions such as access control, crowd management, ticketing, facilities, and surveillance.  
  • Shadow AI and third-party AI use can create risks that stadium security teams cannot readily see.  
  • Security teams must understand not only which AI systems exist, but also what they can access and what actions they can take.  
  • Live-event resilience requires continuous monitoring and response across AI, IT, OT, identities, and third parties.

Modern stadiums are infrastructure unlike any other. I’ve written before on event day sparking stadiums into life with shops and food stands, transport hubs, vast telecommunications infrastructure, field-side technology and beyond, acting as one super-sized, connected ecosystem. Stadiums’ scale and complexity make them some of the toughest environments in cybersecurity. Now, we’re adding AI to those operations and bringing a new dimension of risk.

The benefits of AI in stadium operations are easy to see. It can help stadium operators move fans safely through crowded gates, forecast demand at concession stands, support biometric entry, identify suspicious behavior on CCTV, and manage heating and ventilation. Used well, it can make live events safer, faster, and more efficient.

But it also changes the security model.

In Darktrace’s recent research into the threat landscape surrounding sports, we asked cybersecurity professionals protecting professional sports organizations where in their footprint a cyber compromise would have the greatest impact. The area they named most, highlighted by 34% of the professionals we spoke to, was stadium operations. At the same time, 35% said their organizations are already using AI in stadium operations, or plan to do so in the next 12 months.

Security teams are no longer just protecting traditional IT systems around a stadium. They are increasingly being asked to protect AI systems that are operating in the stadium’s most fundamental functions.

Approved AI vs. shadow AI in stadium operations

There is a clear difference between AI a stadium’s security team knows about and AI it does not.

Approved AI is the AI that has been reviewed, tested, and integrated into the venue’s operating environment. It may support CCTV analytics, access control, facility management, ticketing, logistics, broadcast operations, or anti-piracy monitoring. It should have clear ownership, access controls, logging, vendor review, and data protection rules. That does not make it risk-free, but it allows security teams to institute proper governance.

Shadow AI is different. It is the unapproved use of AI tools by employees, contractors, or suppliers. It often starts with good intent. Someone wants to work faster. A staff member pastes internal information into a public AI tool to draft a briefing. A developer uses an AI assistant to debug ticketing code. A supplier connects an AI scheduling tool to delivery routes. A designer uploads unreleased venue plans or sponsor material to generate a mockup.

None of those actions may feel like a security decision to the person doing them. But each one can move sensitive operational data into an environment the stadium does not control, creating hidden risk.

The approved AI stack may be visible to security teams. The shadow AI stack often is not.

Why game day increases AI cybersecurity risk

In a typical enterprise environment, a security team may have hours to investigate a strange login or an unexpected connection to a third-party service. Within a stadium, the moment an incident is likely to occur is also the moment when teams are at their most stretched and the incident can have the greatest repercussions: game day.

If an AI system used for crowd management behaves unexpectedly, the issue is not only technical. It may affect physical movement inside the venue.

If a supplier tool is sending operational data to an unapproved AI platform, the issue is not only data governance. It may expose delivery routes, restricted access schedules, or staffing plans.

The most dangerous scenario is not always a loud, dramatic attack but a hidden dependency that no one has mapped such as a vendor adding an AI feature through a software update or a staff workflow using an unapproved tool.

By the time the venue is live, those hidden connections can become operational risk.

The supply chain is part of the stadium attack surface

Any major sporting event is made by its supply chain and partnerships: catering firms, transport providers, broadcast systems, facilities teams. Every piece is necessary and each creates a security channel. The risk of supply chain compromise has been well established for some time and has been the source of some of the most high-profile breaches we’ve seen. The data breach at MSG Entertainment, owner of Madison Square Garden, that was widely reported in March, originated in a breach of Oracle’s E-Business Suite, used in MSG Entertainment’s back-office systems, while the 2018 Olympic Destroyer attack on the Pyeongchang Winter Olympics reportedly began with the compromise of the main IT service provider for the Games. The addition of AI is heightening the risk.

A stadium can have strict rules for its own AI systems, but its vendors may be using separate tools. Some may use AI to manage staffing, delivery windows, inventory, or customer communications. Others may not realize that AI features have been added into software they already use.

This is one of the hardest parts of securing AI in stadium operations. The risk does not always come from a tool the venue selected. It may come from a tool a supplier selected or a feature the supplier did not know had been turned on.

Security teams need to treat vendor AI the same way they treat vendor access. They need to know what suppliers can connect to, what data they can see, what tools they use, and whether those tools introduce new routes for data exposure or lateral movement.

A third-party AI tool does not need deep access to create risk. Sometimes it only needs the right operational detail at the wrong time.

Four questions for securing AI in stadium operations

As AI becomes part of stadium operations, security teams need to move beyond basic approval lists. There are four questions they need to ask:

1. Where is AI being used?

This includes obvious tools, such as computer vision, access control, ticketing, logistics, and facility management. But it also includes less visible AI inside SaaS platforms, vendor tools, browser extensions, developer workflows, smart building systems, and collaboration tools.

2. What can the AI access?

Can it see incident logs, staffing plans, ticketing data, video feeds, building controls, fan information, credentials, or supplier systems? Can it only analyze information, or can it also trigger actions?

3. What can the AI do?

AI agents are not just passive tools. Some can call APIs, update records, generate instructions, trigger workflows, or act with the permissions of a user or service account. In a stadium, that distinction is critical. There is a big difference between an AI system that recommends an action and one that can take an action.

4. What does normal look like?

In your security architecture, static rules will not be enough. AI use changes quickly: tools appear inside existing platforms, vendors add new services, and staff find workarounds when they are under pressure. Security teams need to understand normal behavior across people, identities, devices, networks, cloud services, suppliers, and AI tools so they can spot when something changes.

That is especially important in live-event environments, where small anomalies can matter. A connection to an unapproved AI service may be harmless in one context and serious in another, and an AI agent taking action at 3 a.m. may be expected during setup but suspicious during a match. Context is what turns raw activity into useful security insight. It’s also what enables rapid response. Your own AI-based security systems can respond to threats at machine speed if they can build the live context to know action needs to be taken.

AI can make stadiums safer, but only if it is secured

AI has a real role to play in stadium operations. It can help teams detect crowd pressure earlier, reduce bottlenecks, manage facilities more efficiently, improve the fan experience, and support event teams during high-pressure moments.

The answer is not to slow all AI adoption. That's not the goal. The answer is to make AI visible, governed, and secure before it becomes part of match-day operations.

For stadium operators and event organizers, that means mapping AI use across the venue and supplier ecosystem. It means understanding what each AI system can access and what actions it can take. It means giving staff approved tools that meet their needs, rather than leaving them to find workarounds. It means writing AI use into vendor contracts and audits. And it means monitoring behavior across the full environment, not only the systems that are easiest to see. A stadium cannot secure what it cannot see.

When AI becomes part of how a stadium moves people, controls access, manages facilities, supports suppliers, and protects media rights, it stops being a side project. It becomes part of the event infrastructure.

Event infrastructure must be thoroughly prepared before venue gates open and sustained with the operational resilience required to support a secure, seamless, and reliable event experience.

How Darktrace helps secure AI in stadium operations

Darktrace brings more than a decade of behavioral AI expertise, built on an enterprise‑wide platform designed to operate in complex, ambiguous environments. We protect the large-scale integrated IT and OT environments that underpin stadium operations from the 2022 FIFA World Cup in Qatar, to Formula 1 Grand Prixes around the world and stadiums across the USA.

Other cybersecurity technologies try to predict each new attack based on historical attacks. The problem is that AI operates like humans do. Every action introduces new information that changes how AI behaves, making it unpredictable in nature. Historical attack tactics are now only a small part of the equation, forcing vendors to retrofit unproven acquisitions to secure AI.  

Darktrace is fundamentally different. Our Adaptive AI continuously learns how your people and AI behave, building an understanding of your organization so it can detect and respond autonomously when behavior deviates. Our Behavioral Defense Platform secures your AI, people, and infrastructure as you onboard new workflows, agents, and applications, enabling your AI transformation at scale.

As AI changes what organizations can do, Darktrace helps them move forward with confidence. We give the security teams defending the people and technology within stadium infrastructure the understanding, visibility, and autonomous action they need to protect new technologies as they are integrated into operations, so their organizations drive the progress that will define the AI era.

[related-resource]

Continue reading
About the author

Blog

/

AI

/

July 15, 2026

Security After Signatures: Operating in a World of Pre‑CVE Disclosure Exploitation, Collapsed Trust Boundaries, and Autonomous Systems

Default blog imageDefault blog image

Three shifts have reshaped what it means to defend an enterprise securely.  

First, exploitation often begins before defenders have a Common Vulnerabilities and Exposures (CVE) identifier, a security advisory, or an entry in the Cybersecurity and Infrastructure Security Agency's (CISA) Known Exploited Vulnerabilities (KEV) catalog.

Secondly, the trust boundary has moved beyond the network edge into identities, tokens, APIs, and Software-as-a-Service (SaaS) workflows.  

Third, an increasing share of business activity is executed through automation, integrations, and AI agent-like systems that can act faster than teams can verify intent.  

If your security model still relies on detecting known bad artefacts, triaging isolated alerts, and waiting for confirmation before acting, you are already behind the threat.  

This is not a failure of security teams; it’s a failure of the operating model to keep pace with how the environment has changed.

A SOC built around alerts and signatures assumes that malicious activity will eventually surface as an event. In real incidents, however, the decisive evidence is rarely a single event. Instead, it is a chain of individually explainable actions that only appears malicious once you connect the dots across identity, non-human identity, cloud, email, SaaS, operational technology (OT), and network telemetry.

The defenders succeeding today observe behaviors, link them into sequences, understand what those sequences mean, and contain impact before the full story unfolds. That is the operating model the current threat environment demands.  

Exploitation before disclosure

The first shift is the straightforward: the time to exploit has dropped to nearly zero.  

In one example, Darktrace observed a sequence of subtle but strategically significant anomalies within a customer environment that later aligned with exploitation of CVE‑2025‑0994 in Trimble Cityworks by likely Chinese-nexus threat actors. Behavioral indicators were visible at least 18 days before public disclosure, with related anomalies emerging 40 to 50 days earlier during the intrusion window.  

This case illustrates a familiar pattern: clusters of weak‑signal anomalies combing to form an actionable picture of intrusion long before a CVE is published. Such activity reflects long‑horizon, option‑preserving operator models often associated with mature state‑linked activity.  

Figure 1: Darktrace’s detection of malicious exploitation of CVE 2025-0994, later tied to Chinese-nexus threat actors targeting critical national infrastructure (CNI) in the US, weeks before public disclosure.

Throughout 2025 and 2026, Darktrace has continued to observe the value of anomaly-based detections across a range of incidents.

CVE CVE Public Disclosure Date Darktrace Detection Date Days Between Detection of Exploitation and CVE Public Disclosure
CVE 2025 0994
(Trimble City Works)
2025-02-06 2025-01-19 18 Days
CVE 2025-24183
(Apache)
2025-03-10 2025-02-18 20 days
CVE 2025-10035
(Fortra GoAnywhere)
2025-09-18 2025-09-11 7 days

Identity is the real control plane

The second shift is that identity has replaced perimeter as the primary control plane. As Darktrace’s Annual Threat Report 2026 illustrated, identity remains the main challenge in defending against modern intrusions. A clear example is the Adversary-in-the-Middle (AiTM) case published by Darktrace in December 2025. A phishing email led to the compromise of an Office 365 account. Session hijacking bypassed multi-factor authentication (MFA), and the compromised account was used for follow-on phishing and persistence activities including the creation of malicious email rules.  

Every step in that sequence mattered. A successful login alone does not prove legitimacy. An inbox rule, on its own, may not appear catastrophic. Mail activity, viewed in isolation, may seem operationally normal. But the behavioral chain tells a different story: credential theft, token abuse, persistence, and onward compromise through a trusted identity.  

This is why the question is no longer “Did the user authenticate successfully”. The more important question is, “Does this identity action make sense right now, in this context, given what came before it?” The AiTM case shows how identity can be compromised. In practice, however, attacks rarely remained confined to identity alone.  

In another Darktrace case, a compromised SaaS account triggered activity across the email, SaaS, and network layers, including inbox rule changes, phishing propagation, and connections to suspicious infrastructure. Viewed in isolation, none of these events were decisive. Together, however,  they formed a behavioral sequence that revealed the intrusion, with the full attack story automatically correlated and surfaced to defenders by Darktrace’s Cyber AI Analyst.  

Figure 2: Cyber AI Analyst correlated and appended additional events to the incident, including other users who connected to the suspicious redirect link after outbound phishing emails were sent.

AI accelerates the threat  

The third shift is the one many teams still underestimate: trusted tooling, integrations, and AI agent-like systems can create actions that appear legitimate but are strategically dangerous.  

The shift becomes clearer when examining how governments are now framing AI risk. In 2026, guidance published by CISA, UK’s National Cyber Security Centre (NCSC) and Five Eyes partners warned that agentic systems expand attack surfaces, accumulate privilege, and can behave in ways that are difficult to predict or explain [1]. The advice is simple: assume unexpected behavior and design controls around it.  

The real risk is not AI usage. It is unknown autonomy: systems with credentials, data access, and action paths that can execute workflow steps without sufficient behavioral validation, traceability, or human oversight. Darktrace’s Model Context Protocol (MCP) risk analysis provides a useful framework for understanding this challenge. Over-privileged agents, content injection, and tool abuse become high-consequence risks when connected systems can dynamically retrieve data, execute actions, and communicate externally.  

Whether security teams like it or not, AI is already in the enterprise. It will help drive innovation, but it will also be abused, whether accidentally or maliciously. In each of the cases below, AI either scaled the attacker, built the tooling, or existed within the environment as something to exploit or misuse.

1. AI as an Attack Multiplier

In one campaign targeting Mexican government entities, a single operator used commercial AI platforms to generate exploits, automate reconnaissance, and process large volumes of data, compressing work that would traditionally have required an entire team into a single workflow [2].  

Darktrace is also observing this trend further down the stack. In one case, Darktrace identified AI-generated malware exploiting React2Shell, where an attacker used a Large Language Model (LLM) to produce working exploit code and deploy it at scale.  

[darktrace.com], [darktrace.com]

2. AI as an Attack Surface

Attempted AI exploitation is now appearing within customer environments. In one case involving an automation technology manufacturer, a compromised LLM proxy was seemingly used as a stepping stone to access additional AI services. When that attempt failed, the attacker pivoted to cryptomining.

What is clear is that the AI layer has already become an asset worth probing, exploiting, and pivoting through. It is also clear that defenders benefit from rapidly understanding how these activities connect. In this case, Cyber AI Analyst automatically pieced together the intrusion, while Darktrace’s Managed Threat Detection service alerted to the customer, enabling the activity to be contained before it could progress further.

Figure 3: Cyber AI Analyst's investigation into a compromised LLM proxy that was abused for cryptomining activity.

AI as a trusted but dangerous actor

This does not require a cinematic vision of “rogue AI.” The Salesloft incident provides a more grounded example, where AI and automation operate with legitimate access but served malicious intent. In that case, attackers abused compromised OAuth tokens associated with the Drift AI chat agent to export significant volumes of data from Salesforce environments.  

The activity resembled legitimate API usage and relied on trusted SaaS integrations rather than malware or other obvious signs of intrusion. That is precisely the challenge. Traditional security controls are good at detecting forced entry, but far less effective when a trusted application integration behaves in a way that is technically permitted yet operationally harmful.  

In these scenarios, the security challenge shifts from validating access to validating behavior.

This is what that looks like in practice: AI-linked identities executing legitimate actions that require behavioral validation rather than access validation.

Figure 4: Darktrace / SECURE AI highlights anomalous activity across AI identities, surfacing critical behavior that requires validation and containment.

Early observations from Darktrace / SECURE AI deployments reinforce this reality. Across Darktrace's observed fleet, AI service connections per deployment increased 13% during the first half of 2026, reaching over 16 million connections overall. The typical organisation now interacts with seven different AI providers, evidence that AI is no longer operating at the edges of the enterprise. It is increasingly woven into day-to-day business activity.

The most common risks are not compromised models or advanced AI attacks. Instead, they stem from employees and business functions exposing sensitive information through entirely legitimate-looking interactions. Darktrace has observed repeated submission of personally identifiable information (PII), tax information, identification documents, and medical data into LLM prompts, alongside widespread use of unsanctioned (shadow) AI services and growing AI activity from mobile devices.  

For defenders, the challenge is increasingly one of context: understanding when legitimate business use crosses into material risk, while preserving privacy and user trust.

Conclusion

Across all three shifts, the pattern is the same: behavior precedes understanding. Security teams are not losing because adversaries have become invisible. An increasingly outdated security model assumes that malicious activity will reveal itself cleanly and early. It no longer does.  

In 2026 and beyond, defenders win by understanding behavioral sequences, continuously validating trust, and acting before certainty becomes hindsight. That is security after signatures. That is security in the AI era.

Credit to: Daniel Levy, Threat Hunting Data Scientist

Edited by: Ryan Traill, Content Manager

References

[1] https://www.cyber.gov.au/business-government/secure-design/artificial-intelligence/careful-adoption-of-agentic-ai-services  

[2]https://www.latimes.com/business/story/2026-02-26/hacker-used-anthropics-claude-ai-to-steal-mexican-government-data

Continue reading
About the author
Nathaniel Jones
VP, Security & AI Strategy, Field CISO
あなたのデータ × DarktraceのAI
唯一無二のDarktrace AIで、ネットワークセキュリティを次の次元へ