Case study: How Berlin protects its water infrastructure from hackers

Tucked away in the southeastern corner of Berlin is Friedrichshagen, a quiet and park-like neighborhood bordering the Müggelsee lake. Here, a waterworks museum shows how Berlin treated water in the 19th century and showcases the modern water treatment facilities of today. A stone’s throw from historical buildings, where steam provided power and sand was used to filter water, is the high-tech IT home base for the Berliner Wasserbetriebe (BWB), Germany’s largest water treatment facility.

Michael Böttcher, an experienced IT security engineer, heads up the facility. From his command post outfitted with about two dozen monitors, Böttcher and his team keep watch over the ports, routers, switches and the connected sensors and actuators embedded throughout Berlin’s water infrastructure.

The challenge of keeping Berlin’s water facilities safe

Managing all of Berlin’s water facilities poses unique challenges, however, including from both operational and security perspectives. “Water utilities are unique in many respects compared to other industrial facilities,” Böttcher said. “One of the first things you might notice is that our facilities have a substantially greater footprint than, for instance, a machine builder’s or tool manufacturer’s facilities. This can create challenges when it comes to both connectivity and security.”

Berlin also happens to be a sizable city — 344 square miles in land area — that is built on sandy marshland. BWB is therefore forced to play a major role in keeping the groundwater below the surface of urban areas while ensuring that legally protected bogs and lakes have enough water. This unique geographical situation necessitates extensive water infrastructure. And the size and complexity of Berlin’s connected water network pose a challenge when it comes to cybersecurity.

With roots stretching back to the 1850s, the city’s water network has expanded over the decades to meet the needs of its population. At present, Berlin has nine waterworks and eight pumping stations pulling water from more than 700 springs across some 4,900 miles of pipes — some of which are nearly 120 years old and all of which require regular maintenance. In addition, the city has six wastewater-treatment facilities and 154 wastewater-pumping stations.

[IoT Security Summit, co-located with Blockchain360 and Cloud Security Summit, explores how industry-wide security, privacy and trust can be established to unlock the full potential of IoT. Get your ticket now.]

Yet the goal of BWB is 100% uptime. “In the case of an emergency, we have to respond immediately,” said Böttcher, whose response team remotely monitors the facility’s networks for problems day and night. Each year, BWB invests more than €200 million — a figure that includes technology upgrades — to make that happen. Included in that funding are upgrades to its infrastructure with modern technology, bolstering its IT network to include a comprehensive SCADA network and a significant number of connected sensors and industrial control systems to help automate everything from pumps and vents to HVAC units for data centers.

In all, BWB monitors some 180,000 data points, archiving 15,000 of them in a data warehouse.

The control system for the water infrastructure deploys the following across 21 connected facilities using equipment from a variety of vendors:

Confronting the specter of critical infrastructure hacking

As the level of connectivity of BWB’s water treatment facility increased along with the volume of data it gathered, so did the volume of potential attack vectors. About a decade ago, BWB began to prioritize bolstering the security of its water treatment facilities infrastructure. While the expanding connectivity of its water treatment facilities led to gains in efficiency, the expanding network had a growing attack surface.

In general, water treatment plants have unique vulnerabilities to cyberattacks. Water infrastructure is one of the oldest to be equipped with industrial control systems, and the connected sensors embedded in everything from valve fittings to pipes and canals can be difficult to discover. Meanwhile, water treatment facilities could be attractive targets for cyberterrorists. In theory, an intruder in a vulnerable network could cut off water access or pour chemicals into the water supply or dump polluted water into the environment. Even a minor software modification would be sufficient to stop the water treatment process, potentially jeopardizing the entire metropolitan Berlin/Brandenburg region.

In 2005, BWB reached out to network security specialist Auconet to help monitor its IT and industrial networks while working collaboratively with the organizations to drive efficiency gains. “BWB and Auconet have worked together like development partners,” Böttcher said. “BWB had the advantage of being able to share our demands and impulses during the development process and to collaborate with Auconet to ensure that the end product was consistent with our needs.”

“Our families drink the water delivered by BWB every day, but we have understood all along that every water and power utility and SCADA installation is a potential target,” said Gerd Gruner, head of engineering at Auconet. “Industrial control systems need protection worldwide.”

The goal of securing the network became a higher priority after the discovery of the Stuxnet virus, first discovered in 2010. That malware, which targets programmable logic controllers used to automate control of industrial systems, served as a wake-up call for industrial facilities to secure their networks. One of the most advanced virus strains ever discovered, the zero-day malware reportedly destroyed 1,000 nuclear centrifuges in Iran. 

BWB was determined to fend off similar attacks on its own industrial control systems. In the first phase of its collaboration with BWB, Auconet launched a week-long proof of concept, debuting its Business Infrastructure Control Solution (BICS) platform to scan for all of the endpoints on BWB’s network while establishing centralized control over them via a central graphical user interface — the proverbial “single pane of glass.” The two organizations report that, within the first day, they located all of the endpoint devices on the BWB network. “The sensors and pumps are controlled by PLCs and the SCADA systems,” Gruner said. “The SCADA systems and the connections between them are controlled by BICS. While there is no direct connection between the older industrial environment and BICS, with the help of gateway systems, BICS can ‘look into’ industrial bus systems to identify the connected components,” he explained.

Initially, BWB focused on enabling network management and port security, according to Böttcher. Auconet programmed BICS to instantly react to defined security scenarios. “We are a sizable institution,” he said. “We have to carefully think about securing all of our facilities.”

A portion of this security is automated. “If, for instance, a threat actor connects to a network within a freight storage room, BICS will trigger the switch port to close that connection,” Böttcher noted. “After that, it will check: Is the bad guy still there? If he is, that connection remains closed. If it determines that the bad guy is gone, it returns to its prior condition.” In addition, the system shuts down unused ports to keep attackers from finding them. 

In addition, the automated functionality of the BICS system enables monitoring of BWB’s network of water treatment facilities to verify that all the processes are running as they should and that the SCADA systems are running properly, Gruner said. “If a problem is detected, BICS launches an automatic process to attempt to ‘self-heal.’”

The self-healing capability applies to both IT and OT technology. If, for instance, a pump goes down, the BICS system can see if it is capable of restarting. If the pump needs repair, BICS can send an automated alert to service personnel.

BICS also offers access control functionality to secure who has access to the network while analyzing the flow of traffic across the network. “The system determines which machines speak with others on the network based on their IP addresses and can redirect the traffic if abnormality or a problem is detected,” Gruner explained.

The use of automation in the network enabled BWB to steadily boost the efficiency of its network management, gradually reducing the number of employees required for such tasks from 170 to 40 from 2008 to 2016. In addition, the organization has curbed service calls by 90% as a result of BICS’ self-healing and early-breakdown-detection functionalities.

The partnership shows how critical-infrastructure operators can simultaneously reduce cybersecurity risks while bolstering productivity by enabling intelligent automation and data collection.

“Malware and ransomware crossing over from the IT side can threaten not just money, but also lives,” Gruner noted. “We’re able to protect against unauthorized access, while making operations more efficient. Along with our customers like BWB, we closely watch hacker activity around the world and continue investing in our solution to safeguard critical infrastructure.”

BWB’s decision over a dozen years ago to protect its water infrastructure from cyberattacks seems prescient today, given the fact that many hacks at public resources have been reported in recent years. While BWB is careful not to divulge what its future security plans will be, Böttcher said he’s prepared to share the findings with other water treatment facilities in Germany and beyond.