Engineering-Grade Security For Water Systems

By Rees Machtemes

An overview of the risk landscape for water utilities, along with a new set of strategies to quell the threat.

In March, the U.S. EPA ordered states to add cybersecurity criteria into their regular audits of water utilities. This is not a surprise, as cyberattacks with physical consequences have been more than doubling annually since 2019. This is bad news, and we will almost certainly see additional regulations within the next few years. There is good news as well — the engineering profession has powerful and often-neglected tools for addressing cyber risks. These tools are unique to operational technology (OT) and industrial environments — tools and approaches that are not even mentioned in cybersecurity standards such as the NIST Cybersecurity Framework or the IEC 62443 series. Better yet, the protections provided by engineering-grade solutions are deterministic and predictable, even in the face of the most powerful attacks. By deploying engineering-grade protections today, water utilities can get ahead of the cybersecurity problem, rather than constantly chasing new regulations.

New Orders From The EPA

In their 13-page order to states, the EPA argues that securing public water systems’ (PWS) operations against cyberattack is essential to the supply of safe drinking water.¹ The order came with a 100- page guidance document designed to find and fix “significant deficiencies.”² The guidance document is structured as a checklist of cyber-specific audit criteria, organized into eight major categories: account security, device security, data security, governance and training, vulnerability management, supply chain and third-party services, incident response and recovery, and “other.” These criteria mirror the most important functions in the five pillars of the NIST Cybersecurity Framework. In short, the guidance is very sensible and covers the basics expected of any modern IT or OT cybersecurity program.

An Increasing OT Threat Environment

The deteriorating threat environment is what motivates this increased focus on cybersecurity in sanitary audits. Today’s ransomware criminals are opportunistic, target anyone with money, and employ increasingly sophisticated attack strategies. In the last several years, ransomware criminals were responsible for most of the attacks that impaired critical infrastructures and other industrial operations. The 2023 Threat Report,³ a joint initiative by ICSStrive and Waterfall Security Solutions, examines cyberattacks with physical consequences to industrial operations and critical infrastructures, including water and wastewater systems. The report includes a comprehensive list of all such attacks since 2010, including links to public reports and sources that can be used to verify details of the attacks. The report concludes that most attacks with physical consequences are criminal ransomware, with hacktivist threat actors behind most of the rest. Figure 1 shows that the number of attacks with physical consequences on industrial targets have more than doubled every year since 2019.

Figure 1. Attacks on OT with physical consequences

These include attacks on water systems. In June 2022, during the peak of monsoon season, Goa’s Water Resources Department had 15 flood monitoring systems rendered inoperable by ransomware. Without a budget to restore the monitoring systems, it was decided to operate without visibility into the watershed management program, impacting local water treatment. In December 2022, Empresas Publicas Medellín suffered a BlackCat/ALHPV ransomware attack. While affecting IT and billing systems primarily, the attack had the side effect of cutting off water for 28,000 customers on prepaid water plans. More recently, in April 2023, Galil Sewage Corp. became a victim of a GhostSec hacktivist attack, where internet-connected pump controllers were defaced and disabled.

In addition to these consequential attacks, there have been many near misses — attacks that could have been consequential if the circumstances had been slightly different. In August 2022, a botched Cl0p-gang ransomware attack on Thames Water fell apart when attackers attempted to extort the utility and threatened to shut down operations. In fact, they had breached South Staffordshire Water’s network, another utility elsewhere in the UK. In July 2022, a ransomware attack hit the Narragansett Bay Commission in Rhode Island. While a spokesperson for the commission claimed there was no disruption to wastewater collection and treatment, the organization had in fact paid a $250K ransom, and further details were not made public. Finally, who can forget the February 2021 attack at Oldsmar, FL, where an operator observed a remote user attempting to increase lye levels well beyond safe levels.

Standards Change And Evolve

As the OT threat environment continues to deteriorate, it is only a matter of time before new regulations are enacted to deal with increasing threats to water and wastewater. Figure 2 illustrates a timeline of new and recent cybersecurity directives, putting the EPA directive to PWS in perspective.

Figure 2. A timeline of recent government cybersecurity regulations

After the Colonial Pipeline incident, where a gasoline pipeline went down for six days, the Transportation Safety Administration (TSA) issued security directive 2021-02, subsequently revised twice to the current TSA 2021-02C version. Like the EPA guidance, the pipeline directive talks mostly about conventional approaches to cybersecurity but had some new approaches as well. What’s new is language about securing the critical boundary between IT and OT networks. The directive states that the goal of a pipeline operator is to prevent disruption of physical operations, even if the IT network is crippled. The directive requires that, during incident response activities, the IT network must be completely separated from the OT network, so that pipeline operations can continue “at necessary capacity” while the IT network is restored. This means there cannot be any OT dependencies on IT networks or services. In particular, the directive calls out “trust relationships” as particularly dangerous dependencies that must either be eliminated or, if they cannot, must be documented to the TSA with a detailed plan as to how to achieve the “necessary capacity” goal despite the trust relationship before and during an incident.

More recently, the U.S. National Cybersecurity Strategy was released. It contains five pillars, the first of which is to defend critical infrastructure. The introduction to the strategy makes two important points. One explains that cyberattack tools, techniques, and procedures (TTP), that were once the sole domain of nation states with nearly unlimited resources, are now available for purchase to any attacker with money. Ransomware criminals have lots of money. These powerful nation-grade attack tools are now a pervasive threat, targeting anyone with money.

A second point in the national strategy’s introduction is an ambitious goal: A single person’s momentary lapse in judgment, such as clicking on an insecure link or inserting a compromised USB drive, should not have national security consequences or impact critical infrastructure. This is ambitious, but by deploying engineering-grade protections now, utilities can meet this goal. It is possible for utilities to engineer themselves out of the cybersecurity problem, rather than constantly chase new regulations.

Engineering-Grade Protections

While the engineering profession has managed risks to safety, the public, and the environment for a long time, managing cyber risk is new to the profession. In June 2022, the U.S. Department of Energy (DOE) published the Cyber-Informed Engineering (CIE) Strategy, a strategy that is now being broadened to include all critical infrastructures, including water treatment and distribution systems. A key goal of the strategy is to develop a body of engineering knowledge for managing cyber risk, and that body of knowledge has two key components. The first and predictable part is to document cybersecurity practices relevant to protecting industrial control systems, so that engineers can apply these practices more consistently. The second is to identify process, automation, and network design elements that are unique to the engineering profession and can be used to address cyber risk — design elements that are not cybersecurity controls and so do not exist in conventional cybersecurity approaches, including the NIST CSF and the IEC 62443.

One such design element is network engineering, including the techniques documented in the book Secure Operations Technology (SEC-OT).⁴ Network engineering is focused on preventing cyber-sabotage attacks from reaching networks whose worst-case consequences of compromise are unacceptable. The most common example of network engineering is the unidirectional gateway deployed at the IT/OT consequence boundary — the boundary between the IT network whose worst-case consequences of compromise are generally acceptable business consequences and the OT network whose worst-case consequences of compromise are unacceptable threats to public safety. The gateways permit OT data to flow into IT networks to enable business automation for efficient operations and physically prevent any cyber-sabotage attacks from flowing back into protected OT networks.

A second example of an engineering-grade design element is manual operations as a fallback during cyber emergencies. Being able to conduct manual operations will take away a big lever — the threat of shutdown — from cyber-threat actors like criminal ransomware gangs. This is a change of mindset, as many engineering teams prided themselves on eliminating manual operations a decade ago in the name of efficiency and cost savings. Operating a water treatment or distribution system manually, while disconnected from IT during a cyber emergency, is a powerful tool for reducing cyber risk to public safety.

A third example is the use of analog safety systems, as documented in the text Security PHA Review for Consequence-Based Cybersecurity (SPR).⁵ SPR is adapted from the well-known OSHA process hazard analysis (PHA) methodology, widely used for mitigating risks to safety in industrial processes.

For example, if your life depends on an automated steam boiler not exploding, would you prefer that the boiler be engineered with redundant mechanical over-pressure release valves or a longer password on the boiler’s programmable logic controller? Security PHA review is a way to identify cyber risks to safety and recommends the deployment of un-hackable analog safety devices as backups to digital safety systems.

All these examples are unique to the engineering domain. Where is a manual fallback, or over-pressure valve, or unidirectional gateway in the NIST CSF or IEC 6443? These are powerful tools to reduce cyber risk to critical infrastructures, in addition to the cybersecurity tools to which we have long become accustomed.

Another security engineering risk-mitigation strategy that has been proposed for the water sector is to ensure that water treatment systems can be run manually in the event of a cyberattack on the operational network. It also might present a problem to smaller utilities utilizing pre-built treatment skids that are pre-engineered, or that have a process with too quick a transit time or requiring sensitive and highly accurate computer control that a human would not be able to manually perform. It also requires additional operator training and regular practice drills to ensure that manual operations would be successful in an emergency.

The Fundamental Difference

Public water systems are fundamental to society and public safety. They must be protected differently, because the consequences of a cyberattack on water system automation networks are more serious than the same attack targeting less consequential IT networks. Cyberattacks impairing water system operations have become very real and are increasing exponentially. Cyber-informed engineering is a new approach to designing safer, more resilient water systems. One big advantage of these engineering tools is that, rather than chasing the new government regulations that will inevitably arise in the years ahead, these tools enable water utilities to get ahead of the cyber problem with efficient and long-lasting engineering solutions.

References:

1. Memorandum: Addressing PWS Cybersecurity in Sanitary Surveys or an Alternate Process (pdf); United States Environmental Protection Agency; March 3, 2023; https://www.epa.gov/system/files/documents/2023-03/Addressing PWS Cybersecurity in Sanitary Surveys Memo_March 2023.pdf
2. Guide: Guidance on Evaluating Cybersecurity During Public Water System Sanitary Surveys (pdf); United States Environmental Protection Agency; March 3, 2023; https://www.epa.gov/system/files/documents/2023-03/230228_Cyber SS Guidance_508c.pdf
3. Report: 2023 Threat Report (pdf); ICSStrive & Waterfall Security Solutions; May 8, 2023; https://waterfall-security.com/scada-security/whitepapers/2023-threat-report/
4. Book: Secure Operations Technology; Andrew Ginter; Abterra Technologies Inc; 2018; https://waterfall-security.com/sec-ot
5. Book: Security PHA Review for Consequence-Based Cybersecurity; Edward Marszal and Jim McGlone; International Society of Automation (ISA); 2019

About The Author

Rees Machtemes is a professional engineer with 15 years of hands-on experience with both IT and OT systems and a director of industrial security at Waterfall Security Solutions. He has designed power generation and transmission substations, automated a food and beverage plant, audited and tested private and government telecom solutions, and supported IT data centers and OT hardware vendors. This experience has led him to champion cyber-safe systems design and architecture. An obsessive tinkerer and problem-solver, he’s often found next to a soldering station, mechanic’s toolbox, or stack of UNIX servers. He holds a B.Sc. in electrical engineering from the University of Alberta.