Digitizing the Dark: Cyber-attacks against power grids threaten modernity itself

Andrew Tsonchev, Director of Technology | Wednesday July 31, 2019

Among all historical discoveries, none has transformed civilization quite like electricity. From the alarm clock that wakes you up in the morning to the lights you flip off before falling asleep, the modern world has largely been made possible by electric power — a fact we tend only to reflect on with annoyance when our phones run out of battery.

However, the days of taking for granted our greatest discovery may well be nearing an end. As international conflict migrates to the digital domain, state-sponsored cyber-criminals are increasingly targeting energy grids, with the intention of causing outages that could bring victimized regions to a screeching halt. And ironically, the more advanced our illuminated world of electronics becomes, the more proficient these cyber-attacks will be at sending society back to the Dark Ages.

The light bulb goes off

On December 23, 2015, at the Prykarpattyaoblenergo power plant in Western Ukraine, a worker noticed his computer cursor quietly flitting across the screen of its own accord.

Unbeknownst to all but a select few criminals, the worker was, in fact, witnessing the dawn of a new era of cyber warfare. For the next several minutes, the cursor systematically clicked open one circuit breaker after another, leaving more than 230,000 Ukrainians without power. The worker could only watch as the cursor then logged him out of the control panel, changed his password, and shut down the backup generator at the plant itself.

As the first documented outage precipitated by a cyber-attack, the incident provoked speculation from the global intelligence community that nation-state actors had been involved, particularly given the sophisticated tactics in question. Indeed, blackouts that plunge entire cities — or even entire countries — in darkness are a devastating tactic in the geopolitical chess game. Unlike direct acts of war, online onslaughts are difficult to trace, shielding those responsible from the international backlash that accompanies military aggression. And with rival economies racing to invent the next transformative application of electricity, it stands to reason that adversaries would attempt to win that race by literally turning off the other’s lights.

Since the watershed Ukraine attack, the possibility of a similar strike has been a top-of-mind concern for governments around the globe. In March 2018, both American and European utilities were hit by a large-scale attack that could have “shut power plants off at will” if so desired, but which seemed intended instead for surveillance and intimidation purposes. While such attacks may originate in cyberspace, any escalation beyond mere warning shots would have dramatic consequences in the real world.

Smart meters, smarter criminals

Power distribution grids are sprawling, complex environments, controlled by digital systems, and composed of a vast array of substations, relays, control rooms, and smart meters. Between legacy equipment running decades-old software and new IIoT devices designed without rudimentary security controls, these bespoke networks are ripe with zero-day vulnerabilities. Moreover, because conventional cyber defenses are designed only to spot known threats facing traditional IT, they are blind to novel attacks that target such unique machines.

Among all of these machines, smart meters — which communicate electricity consumption back to the supplier — are notoriously easy to hack. And although most grids are designed to avoid this possibility, the rapid adoption of such smart meters presents a possible gateway for threat-actors seeking to access a power grid’s control system. In fact, disabling individual smart meters could be sufficient to sabotage the entire grid, even without hijacking that control system itself. Just a 1% change in electricity demand could prompt a grid to shut down in order to avoid damage, meaning that it might not take many compromised meters to reach the breaking point.

More alarming still, a large and sudden enough change in electricity demand could create a surge that inflicts serious physical damage and produces enduring blackouts. Smart energy expert Nick Hunn asserts that, in this case, “the task of repairing the grid and restoring reliable, universal supply can take years.”

Empowering the power plant

Catching suspicious activity on an energy grid requires a nuanced and evolving understanding of how the grid typically functions. Only this understanding of normalcy for each particular environment — comprised of millions of ever-changing online connections — can reveal the subtle anomalies that accompany all cyber-attacks, whether or not they’ve been seen before.

The first step is visibility: knowing what’s happening across these highly distributed networks in real time. The most effective way to do this is to monitor the network traffic generated by the control systems, as OT machines themselves rarely support security agent software. Fortunately, in most power grid architectures, these machines communicate with a central SCADA server, which can therefore provide visibility over much of the grid. However, traffic from the control system is not sufficient to see the total picture, since remote substations can be directly compromised by physical access or serve as termination points for a web of smart meters. To achieve total oversight, dedicated monitoring probes can be deployed into key remote locations.

Once you get down to this level — monitoring the bespoke and often antiquated systems inside substations — you have firmly left the world of commodity IT behind. Rather than dealing with standard Windows systems and protocols, you are now facing a jungle of custom systems and proprietary protocols, an environment that off-the-shelf security solutions are not designed to handle.

The only way to make sense of these environments is to avoid predefining what they look like, instead using artificial intelligence that self-learns to differentiate between normal and abnormal behavior for each power grid while ‘on the job’. Vendor- and protocol-agnostic, such self-learning tools are singularly capable of detecting threats against both outdated machines and new IIoT devices. And with power plants and energy grids fast becoming the next theater of cyber warfare, the switch to AI security cannot come soon enough.

To learn more about how self-learning AI tools defend power grids and critical infrastructure, check out our white paper: Cyber Security for Industrial Control Systems: A New Approach.

Andrew Tsonchev

Andrew is a technical expert on cyber security and advises Darktrace’s strategic customers on advanced threat defense, AI and autonomous response. He has a background in threat analysis and research, and holds a first-class degree in physics from Oxford University and a first-class degree in philosophy from King’s College London. His comments on cyber security and the threat to critical national infrastructure have been reported in international media, including CNBC and the BBC World.

Shining a light on Shamoon 3: What cyber AI revealed about the data-wiping malware

Max Heinemeyer, Director of Threat Hunting | Wednesday July 10, 2019

Responsible for some of the “most damaging cyber-attacks in history” since 2012, the Shamoon malware wipes compromised hard drives and overwrites key system processes, intending to render infected machines unusable. During a trial period in the network of a global company, Darktrace observed a Shamoon-powered cyber-attack on December 10, 2018 — when several Middle Eastern firms were impacted by a new variant of the malware.

While there has been detailed reporting on the malware files and wiper modules that these latest Shamoon attacks employed, the complete cyber kill chain involved remains poorly understood, while the intrusions that led to the malware’s eventual “detonation” last December has not received nearly as much coverage. As a consequence, this blog post will focus on the insights that Darktrace’s cyber AI generated regarding (a) the activity of the infected devices during the “detonation” and (b) the indicators of compromise that most likely represent lateral movement activity during the weeks prior.

A high-level overview of major events leading up to the detonation on December 10th.

In the following, we will dive into that timeline more deeply in reverse chronological order, going back in time to trace the origins of the attack. Let’s begin with zero hour.

December 10: 42 devices “detonate”

A bird's-eye perspective of how Darktrace identified the alerts in December 2018.

What immediately strikes the analyst’s eye is the fact that a large accumulation of alerts, indicated by the red rectangle above, took place on December 10, followed by complete network silence over the subsequent four days.

These highlighted alerts represent Darktrace’s detection of unusual network scans on remote port 445 that were conducted by 42 infected devices. These devices proceeded to scan more machines — none of which were among those already infected. Such behavior indicates that the compromised devices started scanning and were wiped independently from each other, instead of conducting worming-style activity during the detonation of the malware. The initial scanning device started its scan at 12:56 p.m. UTC, while the last scanning device started its scan at 2:07 p.m. UTC.

Not only was this activity readily apparent from the bird’s-eye perspective shown above, the detonating devices also created the highest-priority Darktrace alerts over a several day period: “Device / Network Scan” and “Device / Expanded Network Scan”:

Moreover, when investigating “Devices — Overall Score,” the detonating devices rank as the most critical assets for the time period December 8–11:

Darktrace AI generated all of the above alerts because they represented significant anomalies from the normal ‘pattern of life’ that the AI had learned for each user and device on the company’s network. Crucially, none of the alerts were the product of predefined ‘rules and signatures’ — the mechanism that conventional security tools rely on to detect cyber-threats. Rather, the AI revealed the activity because the scans were unusual for the devices given their precise nature and timing, demonstrating the necessity of the such a nuanced approach in catching elusive threats like Shamoon. Of further importance is that the company’s network consists of around 15,000 devices, meaning that a rules-based approach without the ability to prioritize the most serious threats would have drowned out the Shamoon alerts in noise.

Now that we’ve seen how cyber AI sounded the alarms during the detonation itself, let’s investigate the various indicators of suspicious lateral movement that precipitated the events of December 10. Most of this activity happened in brief bursts, each of which could have been spotted and remediated if Darktrace had been closely monitored.

November 19: Unusual Remote Powershell Usage (WinRM)

One such burst of unusual activity occurred on November 19, when Darktrace detected 14 devices — desktops and servers alike — that all successfully used the WinRM protocol. None of these devices had previously used WinRM, which is also unusual for the organization’s environment as a whole. Conversely, Remote PowerShell is quite often abused in intrusions during lateral movement. The devices involved did not classify as traditional administrative devices, making their use of WinRM even more suspicious.

Note the clustering of the WinRM activity as indicated by the timestamp on the left.

October 29–31: Scanning, Unusual PsExec & RDP Brute Forcing

Another burst of likely lateral movement occurred between October 29 and 31, when two servers were seen using PsExec in an unusual fashion. No PsExec activity had been observed in the network before or after these detections, prompting Darktrace to flag the behavior. One of the servers conducted an ICMP Ping sweep shortly before the lateral movement. Not only did both servers start using PsExec on the same day, they also used SMBv1 — which, again, was very unusual for the network.

Most legitimate administrative activity involving PsExec these days uses SMBv2. The graphic below shows several Darktrace alerts on one of the involved servers — take note of the chronology of detections at the bottom of the graphic. This clearly reads like an attacker’s diary: ICMP scan, SMBv1 usage, and unusual PsExec usage, followed by new remote service controls. This server was among the top five highest ranking devices during the analyzed time period and was easy to identify.

Following the PsExec use, the servers also started an anomalous amount of remote services via the srvsvc and svcctl pipes over SMB. They did so by starting services on remote devices with which they usually did not communicate — using SMBv1, of course. Some of the attempted communication failed due to access violation and access permission errors. Both are often seen during malicious lateral movement.

Additional context around the SMBv1 and remote srvsvc pipe activity. Note the access failure.

Thanks to Darktrace’s deep packet inspection, we can see exactly what happened on the application layer. Darktrace highlights any unusual or new activity in italics below the connections — we can easily see that the SMB activity is not only unusual because of SMBv1 being used, but also because this server had never used this type of SMB activity remotely to those particular destinations before. We can also observe remote access to the winreg pipe — likely indicating more lateral movement and persistence mechanisms being established.

The other server conducted some targeted address scanning on the network on October 29, employing typical lateral movement ports 135, 139 and 445:

Another device was observed to conduct RDP brute forcing on October 29 around the same time as the above address scan. The desktop made an unusual amount of RDP connections to another internal server.

A clear plateau in increased internal connections (blue) can be seen. Every colored dot on top represents an RDP brute force detection. This was again a clear-cut detection not drowned in other noise — these were the only RDP brute force detections for a several-month monitoring time window.

October 9–11: Unusual Credential Usage

Darktrace identifies the unusual use of credentials — for instance, if administrative credentials are used on client device on which they are not commonly used. This might indicate lateral movement where service accounts or local admin accounts have been compromised.

Darktrace identified another cluster of activity that is likely representing lateral movement, this time involving unusual credential usage. Between October 9 and 11, Darktrace identified 17 cases of new administrative credentials being used on client devices. While new administrative credentials were being used from time to time on devices as part of normal administrative activity, this strong clustering of unusual admin credential usage was outstanding. Additionally, Darktrace also identified the source of some of the credentials being used as unusual.

Conclusion

Having observed a live Shamoon infection within Darktrace, there are a few key takeaways. While the actual detonation on December 10 was automated, the intrusion that built up to it was most likely manual. The fact that all detonating devices started their malicious activity roughly at the same time — without scanning each other — indicates that the payload went off based on a trigger like a scheduled task. This is in line with other reporting on Shamoon 3.

In the weeks leading up to December 10, there were various significant signs of lateral movement that occurred in disparate bursts — indicating a ‘low-and-slow’ manual intrusion.

The adversaries used classic lateral movement techniques like RDP brute forcing, PsExec, WinRM usage, and the abuse of stolen administrative credentials.

While the organization in question had a robust security posture, an attacker only needs to exploit one vulnerability to bring down an entire system. During the lifecycle of the attack, the Darktrace Enterprise Immune System identified the threatening activity in real time and provided numerous suggested actions that could have prevented the Shamoon attack at various stages. However, human action was not taken, while the organization had yet to activate Antigena, Darktrace’s autonomous response solution, which could have acted in the security team’s stead.

Despite having limited scope during the trial period, the Enterprise Immune System was able to detect the lateral movement and detonation of the payload, which was indicative of the malicious Shamoon virus activity. A junior analyst could have easily identified the activity, as high-severity alerts were consistently generated, and the likely infected devices were at the top of the suspicious devices list.

Darktrace Antigena would have prevented the movement responsible for the spread of the virus, while also sending high-severity alerts to the security team to investigate the activity. Even the scanning on port 445 from the detonating devices would have been shut down, as it presented a significant deviation from the known behavior of all scanning devices, which would have further limited the virus’s spread, and ultimately, spared the company and its devices from attack.

Max Heinemeyer

Max is a cyber security expert with over nine years’ experience in the field, specializing in network monitoring and offensive security. At Darktrace, Max works with strategic customers to help them investigate and respond to threats, as well as overseeing the cyber security analyst team in the Cambridge UK headquarters. Prior to his current role, Max led the Threat and Vulnerability Management department for Hewlett-Packard in Central Europe. In this role he worked as a white hat hacker, leading penetration tests and red team engagements. He was also part of the German Chaos Computer Club when he was still living in Germany. Max holds a MSc from the University of Duisburg-Essen and a BSc from the Cooperative State University Stuttgart in International Business Information Systems.