Update:
Following the initial publication of this blog detailing exploitation campaigns utilizing the recently disclosed vulnerability, Darktrace analysts expanded the scope of the threat research investigation to identify potential earlier, pre-CVE disclosure, exploitation of CVE 2024-3400. While the majority of PAN-OS exploitation activity seen in the Darktrace customer base occurred after the public release of the CVE, Darktrace did also see tooling activity likely related to CVE-2024-3400 exploitation prior to the vulnerability's disclosure. Unlike the post-CVE-release exploitation activity, which largely reflected indiscriminate, opportunistic targeting of unpatched systems, these pre-CVE release activities likely represented selective targeting by more calculated actors.
Between March 26 and 28, Darktrace identified two Palo Alto firewall devices within the network of a public sector customer making HTTP GET requests utilizing both cURL and wget user agents, versions of which were seen in later compromise activity in April. The devices requested multiple shell script files (.sh) from rare external IP addresses. These IPs are likely associated with an operational relay box (ORB) network[1]. The connections also occurred without a specified hostname lookup, suggesting the IPs were hardcoded into process code or already cached through unexpected running processes. One of the destination IPs was later confirmed by Palo Alto Network’s Unit 42 as associated with exploitation of the PAN-OS vulnerability[2]. This observed activity closely resembles post-exploitation activity seen on affected firewall devices in mid-April. However, unlike the more disruptive and noisier follow-on exploitation activity seen in post-CVE-release incidents, the pre-CVE-release case observed by Darktrace appears to have been much more discreet, likely due to the relevant threat actor's desire to remain undetected.
--
Introduction
Perimeter devices such as firewalls, virtual private networks (VPNs), and intrusion prevention systems (IPS), have long been the target of adversarial actors attempting to gain access to internal networks. However, recent publications and public service announcements by leading public institutions underscore the increased emphasis threat actors are putting on leveraging such products to initiate compromises.
A blog post by the UK National Cyber Security Center (NCSC) released in early 2024 notes that as improvements are made in the detection of phishing email payloads, threat actors have again begun re-focusing efforts to exploiting network edge devices, many of which are not secure by design, as a means of breach initiation.[i] As such, it comes as no surprise that new Common Vulnerabilities and Exposures (CVEs) are constantly discovered that exploit such internet-exposed systems.
Darktrace analysts frequently observe the impacts of such CVEs first through their investigations via Darktrace’s Security Operations Center (SOC). Beginning in April 2024, Darktrace’s SOC began handling alerts and customer requests for potential incidents involving Palo Alto Networks firewall devices. Just days prior, external researchers publicly disclosed what would later be classified as PAN-OS CVE-2024-3400, a form of remote command execution vulnerability that affects several versions of Palo Alto Networks’ firewall operating system (PAN-OS), namely PAN-OS 11.1, 11.0 and 10.2. At the time, multiple Darktrace customers were unaware of the recently announced vulnerability.
The increase in observed SOC activity for Palo Alto firewall devices, coupled with the public announcement of the new CVE prompted Darktrace researchers to look for evidence of PAN-OS exploitation on customer networks. Researchers also focused on documenting post-exploitation activity from threat actors leveraging the recently disclosed vulnerability.
As such, this blog highlights the network-based behaviors involved in the CVE-2024-3400 attack chains investigated by Darktrace’s SOC and Threat Research teams. Moreover, this investigation also provides a deeper insight into the post-compromise activities of threat actors leveraging the novel CVE. Such insights will not only prove relevant for cybersecurity teams looking to inhibit compromises in this specific instance, but also highlights general patterns of behavior by threat actors utilizing such CVEs to target internet-facing systems.
CVE-2024-3400
In mid-April 2024, the Darktrace SOC observed an uptick in activity involving recurring patterns of malicious activity from Palo Alto firewall appliances. In response to this trend, Darktrace initiated a Threat Research investigation into such activity to try and identify common factors and indicators across seemingly parallel events. Shortly before the Threat Research team opened their investigation, external researchers provided public details of CVE-2024-3400, a form of remote command execution vulnerability in the GlobalProtect feature on Palo Alto Network firewall devices running PAN-OS versions: 10.2, 11.0, and 11.1.[ii]
In their proof of concept, security researchers at watchTowr demonstrated how an attacker can pass session ID (SESSID) values to these PAN-OS devices to request files that do not exist. In response, the system creates a zero-byte file with root privileges with the same name.[iii] Log data is passed on devices running telemetry services to external servers through command line functionality.[iv] Given this functionality, external actors could then request non-existent files in the SESSID containing command parameters which then be interpreted by the command line functionality.[v] Although researchers first believed the exploit could only be used against devices running telemetry services, this was later discovered to be untrue.[vi]
As details of CVE-2024-3400 began to surface, Darktrace’s Threat Research analysts quickly identified distinct overlaps in the observed activity on specific customer deployments and the post-exploitation behavior reported by external researchers. Given the parallels, Darktrace correlated the patterns of activity observed by the SOC team to exploitation of the newly discovered vulnerability in PAN-OS firewall appliances.
Campaign Analysis
Between the April and May 2024, Darktrace identified four main themes of post-exploitation activity involving Palo Alto Network firewall devices likely targeted via CVE-2024-3400: exploitation validation, shell command and tool retrieval, configuration data exfiltration, and ongoing command and control through encrypted channels and application protocols.
1. Exploit Validation and Further Vulnerability Enumeration
Many of the investigated attack chains began with malicious actors using out-of-band application security testing (OAST) services such as Interactsh to validate exploits against Palo Alto firewall appliances. This exploit validation activity typically resulted in devices attempting to contact unusual external endpoints (namely, subdomains of ‘oast[.]pro’, ‘oast[.]live’, ‘oast[.]site’, ‘oast[.]online’, ‘oast[.]fun’, ‘oast[.]me’, and ‘g3n[.]in’) associated with OAST services such as Interactsh. These services can be used by developers to inspect and debug internet traffic, but also have been easily abused by threat actors.
While attempted connections to OAST services do not alone indicate CVE-2024-3400 exploitation, the prevalence of such activities in observed Palo Alto firewall attack chains suggests widespread usage of these OAST services to validate initial access methods and possibly further enumerate systems for additional vulnerabilities.
2. Command and Payload Transmission
The most common feature across analyzed incidents was HTTP GET requests for shell scripts and Linux executable files (ELF) from external IPs associated with exploitation of the CVE. These HTTP requests were frequently initiated using the utilities, cURL and wget. On nearly every device likely targeted by threat actors leveraging the CVE, Darktrace analysts highlighted the retrieval of shell scripts that either featured enumeration commands, the removal of evidence of compromise activity, or commands to retrieve and start binaries on the destination device.
a) Shell Script Retrieval
Investigated devices commonly performed HTTP GET requests to retrieve shell command scripts. Despite this commonality, there was some degree of variety amongst the retrieved payloads and their affiliation with certain command tools. Several distinct types of shell commands and files were identified during the analyzed breaches. For example, some firewall devices were seen requesting .txt files associated with both Sliver C2, whose malicious use has previously been investigated by Darktrace, and Cobalt Strike. The target URIs of devices’ HTTP requests for these files included, “36shr.txt”, “2.txt”, “bin.txt”, and “data.txt”.
More interestingly, though, was the frequency with which analyzed systems requested bash scripts from rare external IP addresses, sometimes over non-standard ports for the HTTP protocol. These bash scripts would feature commands usually for the recipient system to check for certain existing files and or running processes. If the file did not exist, the system would then use cURL or wget to obtain content from external sites, change the permissions of the file, and then execute, sending output to dev/null as a means of likely defense evasion. In some scripts, the system would first make a new folder, and change directories prior to acquiring external content. Additionally, some samples highlighted multiple attempts at enumeration of the host system.
Not every retrieved file that was not explicitly a binary featured bash scripts. Model alerts on some deployments also included file masquerading attempts by threat actors, whereby the Palo Alto firewall device would request content with a misleading extension in the URI. In one such instance, the requested URI, and HTTP response header suggests the returned content is an image/png, but the actual body response featured configuration parameters for a new daemon service to be run on the system.
Bash scripts analyzed across customer deployments also mirrored those identified by external security teams. External researchers previously reported on a series of identifiable shell commands in some cases of CVE-2024-3400 exploitation analyzed by their teams. Commands frequently involved a persistence mechanism they later labeled as the “UPSTYLE” backdoor.[vii] This python-based program operates by reading commands hidden in error logs generated by 404 requests to the compromised server. The backdoor interprets the requests and writes the output to CSS files on the device. In many cases, Darktrace’s Threat Research team noted clear parallels between shell commands retrieved via HTTP GET request with those directly involving UPSTYLE. There were also matches with some URI patterns identified with the backdoor and requests observed on Darktrace deployments.
The presence of these UPSTYLE-related shell commands in response to Palo Alto firewall devices’ HTTP requests provides further evidence for initial exploitation of the CVE. Many bash scripts in examined cases interacted with folders and files likely related to CVE-2024-3400 exploitation. These scripts frequently sought to delete contents of certain folders, such as “/opt/panlogs/tmp/device_telemetry/minute/*” where evidence of exploitation would likely reside. Moreover, recursive removal and copy commands were frequently seen targeting CSS files within the GlobalProtect folder, already noted as the vulnerable element within PAN-OS versions. This evidence is further corroborated by host-based forensic analysis conducted by external researchers.[viii]
b) Executable File Retrieval
Typically, following command processing, compromised Palo Alto firewall devices proceeded to make web requests for several unusual and potentially malicious files. Many such executables would be retrieved via processed scripts. While there a fair amount of variety in specific executables and binaries obtained, overall, these executables involved either further command tooling such as Sliver C2 or Cobalt Strike payloads, or unknown executables. Affected systems would also employ uncommon ports for HTTP connections, in a likely attempt to evade detection. Extensions featured within the URI, when visible, frequently noted ‘.elf’ (Linux executable) or ‘.exe’ payloads. While most derived hashes did not feature identifiable open-source intelligence (OSINT) details, some samples did have external information tying the sample to specific malware. For example, one such investigation featured a compromised system requesting a file with a hash identified as the Spark malware (backdoor) while another investigated case included a host requesting a known crypto-miner.
3. Configuration Data Exfiltration and Unusual HTTP POST Activity
During Darktrace’s investigations, there were also several instances of sensitive data exfiltration from PAN-OS firewall devices. Specifically, targeted systems were observed making HTTP POST requests via destination port 80 to rare external endpoints that OSINT sources associate with CVE-2024-3400 exploitation and activity. PCAP analysis of such HTTP requests revealed that they often contained sensitive configuration details of the targeted Palo Alto firewall devices, including the IP address, default gateway, domain, users, superusers, and password hashes, to name only a few. Threat actors frequently utilized Target URIs such as “/upload” in their HTTP POST requests of this multi-part boundary form data. Again, the User-Agent headers of these HTTP requests largely involved versions of cURL, typically 7.6.1, and wget.
4. Ongoing C2 and Miscellaneous Activity
Lastly, a smaller number of affected Palo Alto firewall devices were seen engaging in repeated beaconing and/or C2 communication via both encrypted and unencrypted protocols during and following the initial series of kill chain events. Such encrypted channels typically involved protocols such as TLS/SSL and SSH. This activity likely represented ongoing communication of targeted systems with attacker infrastructure. Model alerts typically highlighted unusual levels of repeated external connectivity to rare external IP addresses over varying lengths of time. In some investigated incidents, beaconing activity consisted of hundreds of thousands of connections over several days.
Some beaconing activity appears to have involved the use of the WebSocket protocol, as indicated by the appearance of “/ws” URIs and validated within packet captures. Such connections were then upgraded to an encrypted connection.
While not directly visible in all the deployments, some investigations also yielded evidence of attempts at further post-exploitation activity. For example, a handful of the analyzed binaries that were downloaded by examined devices had OSINT information suggesting a relation to crypto-mining malware strains. However, crypto-mining activity was not directly observed at this time. Furthermore, several devices also triggered model alerts relating to brute-forcing activity via several authentication protocols (namely, Keberos and RADIUS) during the time of compromise. This brute-force activity likely represented attempts to move laterally from the affected firewall system to deeper parts of the network.
Conclusion
Between April and late May 2024, Darktrace’s SOC and Threat Research teams identified several instances of likely PAN-OS CVE-2024-3400 exploitation across the Darktrace customer base. The subsequent investigation yielded four major themes that categorize the observed network-based post-exploitation activity. These major themes were exploit validation activity, retrieval of binaries and shell scripts, data exfiltration via HTTP POST activity, and ongoing C2 communication with rare external endpoints. The insights shared in this article will hopefully contribute to the ongoing discussion within the cybersecurity community about how to handle the likely continued exploitation of this vulnerability. Moreover, this article may also help cybersecurity professionals better respond to future exploitation of not only Palo Alto PAN-OS firewall devices, but also of edge devices more broadly.
Threat actors will continue to discover and leverage new CVEs impacting edge infrastructure. Since it is not yet known which CVEs threat actors will exploit next, relying on rules and signatures for the detection of exploitation of such CVEs is not a viable approach. Darktrace’s anomaly-based approach to threat detection, however, is well positioned to robustly adapt to threat actors’ changing methods, since although threat actors can change the CVEs they exploit, they cannot change the fact that their exploitation of CVEs results in highly unusual patterns of activity.
Credit to Adam Potter, Cyber Analyst, Sam Lister, Senior Cyber Analyst
Appendices
Pre-CVE-Release IoCs
38.54[.]104[.]14/3.sh
154.223[.]16[.]34/1.sh
154.223[.]16[.]34/co.sh
38.54[.]104[.]14/
Indicators of Compromise
Indicator – Type – Description
94.131.120[.]80 IP C2 Endpoint
94.131.120[.]80:53/?src=[REDACTED]=hour=root URL C2/Exfiltration Endpoint
134.213.29[.]14/?src=[REDACTED]min=root URL C2/Exfiltration Endpoint
134.213.29[.]14/grep[.]mips64 URL Payload
134.213.29[.]14/grep[.]x86_64 URL Payload
134.213.29[.]14/?deer URL Payload
134.213.29[.]14/?host=IDS URL Payload
134.213.29[.]14/ldr[.]sh URL Payload
91ebcea4e6d34fd6e22f99713eaf67571b51ab01 SHA1 File Hash Payload
185.243.115[.]250/snmpd2[.]elf URL Payload
23.163.0[.]111/com URL Payload
80.92.205[.]239/upload URL C2/Exfiltration Endpoint
194.36.171[.]43/upload URL C2/Exfiltration Endpoint
update.gl-protect[.]com Hostname C2 Endpoint
update.gl-protect[.]com:63869/snmpgp URL Payload
146.70.87[.]237 IP address C2 Endpoint
146.70.87[.]237:63867/snmpdd URL Payload
393c41b3ceab4beecf365285e8bdf0546f41efad SHA1 File Hash Payload
138.68.44[.]59/app/r URL Payload
138.68.44[.]59/app/clientr URL Payload
138.68.44[.]59/manage URL Payload
72.5.43[.]90/patch URL Payload
217.69.3[.]218 IP C2 Endpoint
5e8387c24b75c778c920f8aa38e4d3882cc6d306 SHA1 File Hash Payload
217.69.3[.]218/snmpd[.]elf URL Payload
958f13da6ccf98fcaa270a6e24f83b1a4832938a SHA1 File Hash Payload
6708dc41b15b892279af2947f143af95fb9efe6e SHA1 File Hash Payload
dc50c0de7f24baf03d4f4c6fdf6c366d2fcfbe6c SHA1 File Hash Payload
109.120.178[.]253:10000/data[.]txt URL Payload
109.120.178[.]253:10000/bin[.]txt URL Payload
bc9dc2e42654e2179210d98f77822723740a5ba6 SHA1 File Hash Payload
109.120.178[.]253:10000/123 URL Payload
65283921da4e8b5eabb926e60ca9ad3d087e67fa SHA1 File Hash Payload
img.dxyjg[.]com/6hiryXjZN0Mx[.]sh URL Payload
149.56.18[.]189/IC4nzNvf7w/2[.]txt URL Payload
228d05fd92ec4d19659d71693198564ae6f6b117 SHA1 File Hash Payload
54b892b8fdab7c07e1e123340d800e7ed0386600 SHA1 File Hash Payload
165.232.121[.]217/rules URL Payload
165.232.121[.]217/app/request URL Payload
938faec77ebdac758587bba999e470785253edaf SHA1 File Hash Payload
165.232.121[.]217/app/request63 URL Payload
165.232.121[.]217:4443/termite/165.232.121[.]217 URL Payload
92.118.112[.]60/snmpd2[.]elf URL Payload
2a90d481a7134d66e8b7886cdfe98d9c1264a386 SHA1 File Hash Payload
92.118.112[.]60/36shr[.]txt URL Payload
d6a33673cedb12811dde03a705e1302464d8227f SHA1 File Hash Payload
c712712a563fe09fa525dfc01ce13564e3d98d67 SHA1 File Hash Payload
091b3b33e0d1b55852167c3069afcdb0af5e5e79 SHA1 File Hash Payload
5eebf7518325e6d3a0fd7da2c53e7d229d7b74b6 SHA1 File Hash Payload
183be7a0c958f5ed4816c781a2d7d5aa8a0bca9f SHA1 File Hash Payload
e7d2f1224546b17d805617d02ade91a9a20e783e SHA1 File Hash Payload
e6137a15df66054e4c97e1f4b8181798985b480d SHA1 File Hash Payload
95.164.7[.]33:53/sea[.]png URL Payload
95.164.7[.]33/rules URL Payload
95.164.7[.]33:53/lb64 URL Payload
c2bc9a7657bea17792048902ccf2d77a2f50d2d7 SHA1 File Hash Payload
923369bbb86b9a9ccf42ba6f0d022b1cd4f33e9d SHA1 File Hash Payload
52972a971a05b842c6b90c581b5c697f740cb5b9 SHA1 File Hash Payload
95d45b455cf62186c272c03d6253fef65227f63a SHA1 File Hash Payload
322ec0942cef33b4c55e5e939407cd02e295973e SHA1 File Hash Payload
6335e08873b4ca3d0eac1ea265f89a9ef29023f2 SHA1 File Hash Payload
134.213.29[.]14 IP C2 Endpoint
185.243.115[.]250 IP C2 Endpoint
80.92.205[.]239 IP C2 Endpoint
194.36.171[.]43 IP C2 Endpoint
92.118.112[.]60 IP C2 Endpoint
109.120.178[.]253 IP C2 Endpoint
23.163.0[.]111 IP C2 Endpoint
72.5.43[.]90 IP C2 Endpoint
165.232.121[.]217 IP C2 Endpoint
8.210.242[.]112 IP C2 Endpoint
149.56.18[.]189 IP C2 Endpoint
95.164.7[.]33 IP C2 Endpoint
138.68.44[.]59 IP C2 Endpoint
Img[.]dxyjg[.]com Hostname C2 Endpoint
Darktrace Model Alert Coverage
· Anomalous Connection / New User Agent to IP Without Hostname
· Device / New User Agent (triggered by pre-CVE-release activity)
· Anomalous File / Script from Rare External Location (triggered by pre-CVE-release activity)
· Anomalous File / Masqueraded File Transfer
· Anomalous File / EXE from Rare External Location
· Anomalous File / Multiple EXE from Rare External Locations
· Anomalous File / Script and EXE from Rare External
· Anomalous File / Suspicious Octet Stream Download
· Anomalous File / Numeric File Download
· Anomalous Connection / Application Protocol on Uncommon Port
· Anomalous Connection / Posting HTTP to IP Without Hostname
· Anomalous Connection / Multiple Failed Connections to Rare Endpoint
· Anomalous Connection / Suspicious Self-Signed SSL
· Anomalous Connection / Anomalous SSL without SNI to New External
· Anomalous Connection / Multiple Connections to New External TCP Port
· Anomalous Connection / Rare External SSL Self-Signed
· Anomalous Server Activity / Outgoing from Server
· Anomalous Server Activity / Rare External from Server
· Compromise / SSH Beacon
· Compromise / Beacon for 4 Days
· Compromise / Sustained TCP Beaconing Activity To Rare Endpoint
· Compromise / High Priority Tunnelling to Bin Services
· Compromise / Sustained SSL or HTTP Increase
· Compromise / Connection to Suspicious SSL Server
· Compromise / Suspicious File and C2
· Compromise / Large Number of Suspicious Successful Connections
· Compromise / Slow Beaconing Activity To External Rare
· Compromise / HTTP Beaconing to New Endpoint
· Compromise / SSL or HTTP Beacon
· Compromise / Suspicious HTTP and Anomalous Activity
· Compromise / Beacon to Young Endpoint
· Compromise / High Volume of Connections with Beacon Score
· Compromise / Suspicious Beaconing Behaviour
· Compliance / SSH to Rare External Destination
· Compromise / HTTP Beaconing to Rare Destination
· Compromise / Beaconing Activity To External Rare
· Device / Initial Breach Chain Compromise
· Device / Multiple C2 Model Breaches
MITRE ATTACK Mapping
Tactic – Technique
Initial Access T1190 – Exploiting Public-Facing Application
Execution T1059.004 – Command and Scripting Interpreter: Unix Shell
Persistence T1543.002 – Create or Modify System Processes: Systemd Service
Defense Evasion T1070.004 – Indicator Removal: File Deletion
Credential Access T1110.001 – Brute Force: Password Guessing
Discovery T1083 – File and System Discovery
T1057 – Process Discovery
Collection T1005 – Data From Local System
Command and Control
T1071.001 – Application Layer Protocol: Web Protocols
T1573.002 – Encrypted Channel: Asymmetric Cryptography
T1571 – Non-Standard Port
T1105 – Ingress Tool Transfer
Exfiltration
T1041 – Exfiltration over C2 Protocol
T1048.002 - Exfiltration Over Alternative Protocol: Exfiltration Over Asymmetric Encrypted Non-C2 Protocol
References
[1] https://cloud.google.com/blog/topics/threat-intelligence/china-nexus-espionage-orb-networks
[2] https://unit42.paloaltonetworks.com/cve-2024-3400/
[i] https://www.ncsc.gov.uk/blog-post/products-on-your-perimeter
[ii] https://security.paloaltonetworks.com/CVE-2024-3400
[iii] https://labs.watchtowr.com/palo-alto-putting-the-protecc-in-globalprotect-cve-2024-3400/
[iv] https://labs.watchtowr.com/palo-alto-putting-the-protecc-in-globalprotect-cve-2024-3400/
[v] https://labs.watchtowr.com/palo-alto-putting-the-protecc-in-globalprotect-cve-2024-3400/
[vi] https://security.paloaltonetworks.com/CVE-2024-3400
[vii] https://www.volexity.com/blog/2024/04/12/zero-day-exploitation-of-unauthenticated-remote-code-execution-vulnerability-in-globalprotect-cve-2024-3400/
[viii] https://www.volexity.com/blog/2024/05/15/detecting-compromise-of-cve-2024-3400-on-palo-alto-networks-globalprotect-devices/
.jpeg)
