5 Critical Questions Raised by Water Treatment Facility HackIncident Highlights the Need to Enhance OT Security
As the investigation into the hacking of a water treatment facility in Florida continues, cybersecurity experts say the incident points to the urgent need to enhance operational technology security.
"The lesson to be learned here is that public sector organizations that provide critical services - which are largely unregulated for security - need some scrutiny on their IT and OT security," says Mike Hamilton, the CISO of CI Security, who formerly worked at the Department of Homeland Security.
The Friday attack against the water treatment facility in Oldsmar, Florida, is being investigated by the local Pinellas County Sheriff's Office, which has notified the FBI and U.S. Secret Service.
A hacker remotely accessed a system that controls the chemicals that are added to the water to make it safe to drink and increased the level of sodium hydroxide - lye - from 100 parts per million to 11,100 parts per million, the local sheriff says. But the hack was thwarted within minutes, with officials restricting remote access.
Lye is used to reduce the acidity of water to make it more alkaline. But too much sodium hydroxide can be deadly. For example, in higher concentrations, it's used in drain cleaner to dissolve organic matter.
Oldsmar Mayor Eric Seidel said the water treatment facility had controls in place that would have prevented the tainted water from leaving the plant. For example, the change to the lye levels would not have affected the water for 24 to 36 hours, and the system has alarms that would have been triggered by the increased lye levels.
Nevertheless, the incident points to the major risks involved in remotely accessing systems and the urgent need to ensure OT security.
Here are five key questions the Florida incident raises:
1. Was Use of Remote Access Appropriate?
The hacker apparently used TeamViewer to gain remote access to the water treatment facility's network.
TeamViewer has long been an attractive target for attackers, because the software can give administrators full, remote access to and control of systems. As a result, if TeamViewer is not properly secured, or a hacker manages to obtain credentials, the intruder can achieve remote control over systems (see: TeamViewer Bolsters Security After Account Takeovers).
Jake Williams, a former member of the National Security Agency's elite hacking team who now runs the cybersecurity consultancy Rendition Infosec, notes that certain versions of TeamViewer don't require administrator permissions for installing or running the software. Also, when TeamViewer makes an outbound connection, most firewalls will allow this connection to give the user remote access a computer system. In most cases, the software used TCP port 5938, but it can use others as well.
The idea is that it regularly beacons to a server on the internet and checks to see if anyone wants to connect. When the remote user wants to connect, they contact the same server. Think of this server as a matchmaker of sorts, a firewall invalidating matchmaker... 2/— Jake Williams (@MalwareJake) February 9, 2021
Williams says TeamViewer too often gets deployed in enterprise networks as "shadow IT" without administrative approval. It's not yet clear if the water treatment facility's senior managers authorized its use for remote access because the COVID-19 pandemic has led to a shift to working from home.
"Control systems engineers often need to be able to provide remote support for operational reasons. Before Friday, I'd say it's a fair bet that having TeamViewer installed saved the utility untold dollars in management/monitoring costs," Williams says. "If we're serious about calling this a 'wake-up call,' then we have to fund utilities to provide remote access solutions that are both secure and effective for the operators."
2. Did Hacker Obtain Credentials?
In the wake of the hack, a TeamViewer spokesperson tells Information Security Media Group: "We don’t have any indication that our software or platform has been compromised. As a global remote connectivity provider, we have leading security measures and state-of-the-art authentication options in place.
"TeamViewer stands ready to support relevant authorities in their investigation of the technical details, such as how the cybercriminals potentially obtained login credentials, which are set and encrypted solely on the device."
3. What Security Steps Are Most Critical?
For years, federal agencies have warned about hackers targeting critical infrastructure, including water treatment facilities and power plants.
In July 2020, the Cybersecurity Infrastructure and Security Agency, along with the NSA, issued updated critical infrastructure security recommendations, including limiting or eliminating remote access.
"Remote connectivity to OT networks and devices provides a known path that can be exploited by cyber actors. External exposure should be reduced as much as possible," according to the recommendations.
"We've seen enough breaches of the U.S. power grid, water systems and even nuclear plants to conclude this: Protecting these critical facilities and upgrading their cyber defenses should be a far higher priority."
— Hitesh Sheth, Vectra
In April 2020, CISA also warned organizations against using certain VPN servers that had known vulnerabilities. The agency also noted that some hackers were using administrative tools, such as LogMeIn and TeamViewer, to gain persistence within networks. These tools could enable hackers to maintain their presence even if they lost their primary connection to the infected network, CISA warned (see: CISA Warns Patched Pulse Secure VPNs Still Vulnerable).
Hamilton of CI Security notes that while the water facility in Florida was able to stop the attack before the hacker could cause any damage, remote system access that apparently lacked sufficient security posed a major risk.
"There is a justifiable reason for providing remote access, but enabling that access in the absence of security requirements invites these types of episodes," Hamilton says. "If remote access is a requirement the water and other critical sectors, it should be enabled only when needed." Plus, critical infrastructure providers should frequently audit the use of remote access and ensure that multifactor authentication is used, he adds.
4. Is OT Security Being Neglected?
The Florida incident is drawing attention to OT security shortcomings - especially when it comes to protecting industrial control systems and supervisory control and data acquisition, aka SCADA, systems.
"Water facilities rely on SCADA systems to manage the automated process or water distribution and treatment. Many of these ICS are outdated, unpatched and available for review on the internet, leaving them incredibly vulnerable to compromise," says Austin Berglas, who was an assistant special agent in charge of cyber investigations at the FBI's New York office.
"In addition, many ICS solutions were designed for non-internet-facing environments and therefore did not incorporate certain basic security controls. This offers additional vulnerabilities as more and more operational technology environments are allowing access to their ICS systems from the internet."
Berglas was part of the investigation into the 2013 compromise of the Bowman Avenue Dam in Rye Brook, New York. Members of the Iranian Revolutionary Guard gained access through internet-facing controls, that investigation determined.
Nearly eight years later, many of the same OT security issues remain as nation-state hackers continue to hone their skills, he says.
"Although the dam was not functioning at the time and was most likely not the Iranian’s main target, it demonstrates the vulnerability of certain critical infrastructure when their ICS systems are allowed to be exposed to the internet and not isolated," says Berglas, who is now global head of professional services at cybersecurity firm BlueVoyant.
5. Should CISA Get More Involved?
The hacking incident in Florida points to the need for CISA to get more involved in protecting critical infrastructure - going beyond sending out alerts, some security experts say.
Tom Kellermann, who's head of cybersecurity strategy for VMware and a member of the Cyber Investigations Advisory Board for the U.S. Secret Service, previously told ISMG that CISA should be given more resources to threat hunt and share that data with private organizations and federal agencies.
President Biden's $1.9 trillion proposal for COVID-19 relief calls for providing $9 billion to improve cybersecurity at the federal level, which would include $690 million for a CISA project designed to improve monitoring and incident response across government agencies (see: Biden's $10 Billion Cybersecurity Proposal: Is It Enough?).
Other security experts also believe that the time has come for CISA to expand its role in protecting the nation's infrastructure.
"In the Oldsmar case, it's premature to assign motive or place blame," says Hitesh Sheth, president and CEO of security firm Vectra. "But, we've seen enough breaches of the U.S. power grid, water systems and even nuclear plants to conclude this: Protecting these critical facilities and upgrading their cyber defenses should be a far higher priority."
Phil Reitinger, a former director of the National Cyber Security Center within the Department of Homeland Security and who’s now president and CEO of the Global Cyber Alliance, says CISA needs far more resources to help ensure critical infrastructure security issues are addressed at every level - including at a small town's water treatment plant.
"The problem is that the internet ecosystem is like an intensive care unit filled with sucking chest wounds. CISA shouldn't, and doesn't, run from patient to patient merely observing that things are pretty bad," Reitinger says. "It has to execute on driving change - a mission that was done to great success with its state and local partners in election security - across the ecosystem."