Cybersecurity Hygiene Is Effective Preventive Maintenance

By Jacques Brados and Laurie Kusmaul

Organizations can enhance cybersecurity and minimize cyberattacks by improving basic cybersecurity hygiene.

Cyberattacks are escalating in the real world. Recent events raise concerns about loss of fuel, interruption of electricity, alteration of chemicals in drinking water, and impairment of other critical infrastructure. These concerns are not new to industrial control system (ICS) defenders, responders to incidents, and system assessors. Numerous thwarted and successful incidents occur, but don’t always make the news.

Cyberattacks repeatedly occur using weak entry points with malevolent software (malware). In some cases, the malware has been around for a year or more. Unfortunately, the attacks continue to result in ransom demands, loss of critical data, and increased concern or fear. Improving basic cybersecurity hygiene can help utility owners and operators enhance security and minimize cyberattacks.

Help Is Available
The Department of Homeland Security has an agency dedicated to critical infrastructure, called the Cybersecurity and Infrastructure Security Agency (CISA at cisa.gov). CISA can help federal, state, local, tribal, and territorial governments, as well as public and private sector critical infrastructure organizations. CISA services are generally free, and the agency offers a no-cost vulnerability scanning (pen testing) service and other no-cost assessments for critical infrastructure organizations. More information can be found at https://www.cisa.gov/cyber-resource-hub.

CISA has a dedicated ICS group called the Industrial Control System – Cyber Emergency Response Team (ICS-CERT) that offers free training, onsite assessments of the ICS, vulnerabilities of ICS vendors, ICS incident response, and several other services at https://us-cert.cisa.gov/.

A list of basic activities to reduce the probability of a ransomware incident is available from CISA.¹ Following are some key takeaways from that list, which follows National Institute of Standards and Technology (NIST) guidelines, along with author recommendations based on many years of experience.

Make Backups — And Restore From Those Backups
This may seem like a mammoth task for already-stressed and often-overworked information technology (IT) and ICS teams. But restoring from backup is the most important thing an organization can do to recover from various threats, including ransomware attacks, without feeling forced to pay the ransom. It’s also important to know that backups are good and restoration will work. Examples of backup issues include unreliable tape media, inclusion of a Structured Query Language database that is not captured in the backup set because the file was open, and the existence of malware prior to backup.

Backups may be stored online or offline. Operational backups are kept online and used to restore an accidentally deleted file more quickly than offline backups. While convenient and accessible, these backups will get encrypted during a ransomware attack. Security backups are kept offline and used to restore data after a significant incident.

Various scenarios exist for restoring data from security backups, and any restoration exercise should include documenting success and potential issues.

• A full restore from backup involves pulling hardware from inventory, restoring from offline backup to a machine on a disconnected test bench environment, and scanning for viruses. Some data likely will need to be entered manually to fill the gap in time from when the offline backup occurred.
• Restoration from golden images entails the use of computers or virtual environments that are normally powered down. They are regularly updated, data is synchronized, and then they are powered back down. This provides a standard baseline environment to quickly use in production when an incident occurs.
• A full restore from installation media and data, which is typically a last resort, is a factory reset on the compromised hardware. Reinstallation of firmware (the software stored on a computer to make it run), the operating system, and applications must occur before data can be restored. Consider documenting the location of the installation media in the incident response plan.
• For an ICS, it may be necessary to restore programmable logic controllers (PLCs) and industrial network switches along with the servers and computers. Pulling the parts together from inventory to build a PLC on a test bench, reloading the program, and checking for faults can increase cybersecurity and peace of mind.

Look For IT Vulnerabilities
It’s important to look for weakness in the defense systems and commit to a scheduled exercise. IT and ICS owners should examine all hardware and software configurations. Have some fun thinking like a cyberattacker: How could someone with your knowledge threaten or harm your organization even without your username, password, and keys?

Consider hiring an “ethical hacker” to help detect such threats. White box, nondestructive penetration testing has proven beneficial for many owners. Someone from the client organization provides advance approval of the planned tests and looks over the shoulder of the tester. Often, the tester gets access to the point of typing the command “delete all,” but does not enter the command. In this way, the client can see how a cyberattacker would be successful. From an IT perspective, penetration or pen testing remotely may be just as successful for internet-facing devices.

Look For ICS Vulnerabilities
The ICS requires passive and respectful testing because it is more sensitive to scanning, and resulting problems can be more severe and complex to recover from. Active penetration test scans performed by IT professionals not trained in the operation of the ICS have stopped water pumps and tripped electrical generators, causing outages.

Regularly Patch And Update Software And Operating Systems To The Latest Available Version
There are armies looking for vulnerabilities in software to exploit. Microsoft and other software companies have dedicated teams looking for vulnerabilities and patching their codes. These vulnerabilities are broadcast widely along with the fix, but it is each organization’s responsibility to apply patches rapidly.

On the IT side, this means updating to the most recent version of finance, billing, inventory, customer service, marketing, work order, and other software packages. Owners should upgrade to the most recent version of the operating system and patch frequently. Currently, most organizations use Microsoft Windows as their operating systems. Every six months, Microsoft provides a significant update to supported operating systems. Each client software package needs to be tested on the new operating system to identify potential problems. A planned list of tests might include such functions as timesheets, payroll, and year-end payroll. Testers might create a fictitious asset and a work order, make entries, and close the work order. They should also try to enter erroneous data such as trying to work 25 hours in a day, incorrectly typing a password, and entering negative water consumption.

Microsoft releases smaller security patches and bug fixes on the second Tuesday of every month. These come with rankings of Critical, Important, Moderate, and Low to help administrators determine which patches/fixes to apply. Consider releasing the critical ones without testing. If a server or application fails, it can be restored from backup.

On the ICS side, applying updates can prove more complicated. There are well-documented occurrences where Microsoft updates did not work with ICS software. Facility operators’ screens showing process parameters such as water quality have turned blue and displayed error text (referred to as the “blue screen of death”). In such cases, plants may not continue to operate in automatic mode, and operators may need to switch to manual control until the ICS support team can respond.

Industrial control systems are the most critical and have the highest impact if downtime occurs. When ICS support team members are asked why they don’t take the time to test patches offline, the general response is that ICS and IT systems are typically not connected. This frequently proves to be a fallacy; the ICS is often found to be connected and possessing outbound access to the internet in most author-conducted evaluations. It’s crucial to take the time to test and install operating system and ICS software patches.

Know Where The ICS And IT Systems Are Connected
One of the recommendations from the Colonial Pipeline response team was to verify that the ICS can function without the IT system. Consider asking your IT/ICS managers these questions:

• Where are the points of connection between the ICS and the IT system?
• Are these IT/ICS link cables clearly labeled?
• Is there clear guidance for when to disconnect the IT/ICS link cables?
• What happens to the ICS when all IT/ICS link cables are disconnected?
• Has disconnecting the IT/ICS link cables been tested in the past 12 months?
• Where are the results documented, and when is the next test scheduled?

Change Default Passwords, For The Love Of Bits And Bytes!
The most popular way for a cyberattacker to gain unauthorized access is to send a bad link or infected file via email. An employee or external contractor who unknowingly clicks on a link or attachment delivered this way releases some form of malware. Intruders who gain access to the network this way typically use default or easily guessable passwords to create problems. Establishing multifactor authentication and changing default passwords to strong passwords or passphrases are strong deterrents.

No-Computer Wednesdays
For a few hours every Wednesday, a water utility turns off the ICS computer monitors at a treatment plant. The computers are still on in case an unexpected event occurs. The operators run the plant and distribution system in manual mode because the utility director wants to be sure plant employees know how to work systems beyond watching computer monitors. This practice, which is part of the emergency preparedness and disaster recovery plan, ensures that the instrumentation and PLC programs function as expected without operator intervention.

There are other great ways to test the ICS midweek:

• Periodically recover an ICS server from a backup.
• Disconnect the ICS and IT system for a few hours to confirm network segregation.
• Change passwords to devices that only support local passwords.
• Coordinate with the local emergency manager to simulate a fire in the data center or a chemical building. Frequent exercises with operators, ICS support, IT support, managers, fire, and police are good cyberattack defenses.

No-Change Fridays
In the ICS world, patches and changes are done in the middle of the week, with everyone watching for adverse effects. Changes are never done on Fridays or over the weekend. The last day of the work week is for:

• Documenting what occurred during the week by updating and closing work orders.
• Policy and procedure review and updating.
• Reviewing vulnerabilities published in the IT world at Multi State – Information Sharing and Analysis Center (MS-ISAC), the NIST National Vulnerability Database (https://nvd.nist.gov/), or other locations.
• Reviewing vulnerabilities published in the ICS world at https://us-cert.cisa.gov/ics/advisories.
• Reaching out to a neighboring utility to discuss what others are seeing and doing.
• Planning the evolution for the following Wednesday.

Fresh Perspective
A new set of eyes can be helpful for assessing any system, whether they belong to an internal auditor or a trusted consultant. Someone who knows your industry, but does not know your organization or specific unit, can provide the right perspective and ask hard questions to help you. Better to have friends rather than enemies find vulnerabilities.

It Comes Down To Healthy Practices
Consider personal hygiene and preventive maintenance. Brushing your teeth, bathing, eating right, exercising, paying attention to potential threats around you, and visiting a doctor and dentist require time and money. But healthy practices decrease your chances of an emergency trip to the hospital. Investing now in cyber hygiene can yield cyber benefits when you least expect it.

References:

1. https://www.cisa.gov/sites/default/files/publications/CISA_MS-ISAC_Ransomware Guide_S508C.pdf

About The Authors

Jacques Brados, CISSP, is the water cybersecurity national practice lead for Black & Veatch. He has secured industrial control systems and IT systems for 21 years as a consultant and as a municipal manager. He has also supported American Water Works Association cybersecurity risk-assessment tool development.

 

Laurie Kusmaul is a control system programmer and cybersecurity professional with Black & Veatch. A solutions-oriented industrial IT/ OT professional, she has designed, implemented, and integrated cost-effective, high-performance technical solutions for many organizations, including those seeking help with water and wastewater treatment and distribution needs.