01 March 2004

Kicking the control system bug

Preventing security breaches in plant control systems.

By Ellen Fussell

Justin Lowe knows about security, especially when it comes to control systems, and coming from the information technology (IT) side of the industry, he knows it's a whole different ball game. As principal security consultant for BP, Lowe has been working for the past two years on a program to improve the company's process control system security.

"Threats to process control systems have been more serious in the last six months or so with last summer's worms—Blaster and Nachi," Lowe said. These are normal computer Internet viruses that self-propagate across the Internet. But because they attach to Microsoft systems, any control system connected to another network is vulnerable to infection. "Before it had been muted as a possibility, but last August, with those two worms, it's now likely," he said. Any new worms that attack Microsoft could have a severe impact on process control systems.

"It's not that any system running on Windows is vulnerable, because there are ways of securing systems so that they are not vulnerable," said Ron Sielinski, Microsoft's industry technology strategist for manufacturing. "In fact, every one of these high-profile exploits that people have read about have really been based upon vulnerabilities that were already patched in the operating system. The consequence is if you have a Windows operating system, and you're keeping abreast of the patches, that's going to protect your operating system."

Generally a worm produces a denial of service attack, where the control system slows down or stops operating. In an oil refinery it would mean the displays on an operator's network stations are not up to date and do not reflect what is happening, Lowe said. The worst case scenario is losing control of the plant—possibly causing environmental, health, or safety issues.

"In theory, refineries could blow up," Lowe said. Last year there was a case of a similar worm getting into a Russian energy company and taking control of its gas pipelines (see related story on page 32). "I think quite a lot of people in the chemical industry have experienced problems since last August. And it's only going to get worse," he said. "These worms are now attacking the core components of the operating systems. And the only way of protecting against them is to apply the security patches Microsoft's releasing on a monthly basis."

How effective are patches?

Critical flaw in Windows A critical flaw in most versions of Microsoft's Windows operating system could allow hackers to break into personal computers, Microsoft said last month in its monthly security bulletin. Although none have experienced a breach in security yet, Windows NT, Windows 2000, Windows XP, and Windows Server 2003 are at risk. The company offered software updates to fix the software flaw. The flaw affects all current versions of Windows, said Stephen Toulouse, security program manager for Microsoft's Security Response Center. "We're not aware of anyone affected by this at this time," Toulouse said. Last year Microsoft adopted a new monthly patch release program, which it said would let customers more easily apply software fixes for security bugs. Windows users can download the patch for the vulnerability from www.microsoft.com/security.

"These security patches are pretty effective, but they cause problems because the vendors of the control systems need to do a lot of testing before people can apply them to their systems," Lowe said. So that causes extra work for vendors, which is expensive. And the patching takes time—time during which systems are still vulnerable. "Even when the vendors have tested the patches, it's then down to control engineers to apply those patches to their systems," he said. That's vulnerable time too, which could take up to an hour per computer.

"When Microsoft releases a patch, the hacker community reverse engineers the patch and then develops an exploit based on that," Sielinski said. "And the time it's taking the hacker community to do that is getting shorter. So it does fall to the manufacturing community to respond more quickly to available patches."

"In order to protect against these worms you have to apply the patches, but you're spending a lot of time and money while people are going around loading CDs into computers," Lowe said. "Every time you make a change to a control system, you risk causing a problem like a minor malfunction to the software."

To help manage that process, one of the things Microsoft is doing is going to a once-a-month patch release cycle "so you don't have new patches every week and you're caught in this perpetual cycle of upgrading your operating system," Sielinski said. And Microsoft isn't just releasing patches to vulnerabilities. "We're also enhancing the security capabilities of our products. So even if a system had not been patched, and users of Windows XP had turned on the built-in firewall technology, they would have been protected from recent viruses," he said. "And we'll be doing more and more similar things in the future." For example, with the current version of XP, that firewall was turned off by default. With the next version, Service Pack 2, Microsoft will turn on the Internet connection firewall by default.

Why doesn't IT security work?

Technology in the IT realm and the plant control world came from two different sets of ancestors, said Jim Bauhs, information protection manager for plant operations at Cargill in Minneapolis, Minn. "There was a marriage at some point along the way where customers of plant control systems said, 'We'd like it if you'd use the same infrastructure as we do on the IT side, so we don't have to run different cables for Ethernet and PCP to wire the plant.'"

But plant control systems never took security into account, Bauhs said. "And when they converted their interfaces to run on Ethernet, they did not rearchitect their systems. So now you have what I would call naïve little children in a dangerous world. Some of these systems are ill prepared for some of the risks and exposures that exist."

Bauhs said when plant control systems were proprietary, only plant personnel could operate them. There was no connection—electronic or otherwise—to people outside the plant, the physical perimeter of the facility. "Now if it's on a corporate network, it has connections beyond the perimeter from the facility and stretches in many cases into the Internet," he said. So the systems are all exposed to viruses, worms, and malware. "It's actually no different than when you have a child you send to daycare for the first time, and they come home with all kinds of colds, bugs, and infections they've never been exposed to. Systems are the same way if they haven't built up immunity; they can be affected by that."

What needs to happen?

Security is strategy By Eric Byres Most corporate hacking to PLCs and HMIs on the plant floor comes from inside the firewall, so depending on the corporate firewall to protect the process is not the answer. You need a strategy. First, develop a security policy for the control systems that defines the direction, gives broad guidance, and demonstrates senior management support. A simple architecture might be to divide the plant into two levels: a business network level and process control network level. Although the firewall is the lock on the door to the process network, it is not the burglar alarm. You need some method of monitoring traffic and identifying malicious activity on the network, such as an intrusion detection system, which can range from a simple scan detector that profiles user behavior to a system that takes action against the suspected intruder. Finally, develop an incident response plan. Companies might often know they are being hacked, but they don't know how to handle the problem. Behind the byline Eric Byres is the research team leader of the Internet Engineering Lab at the British Columbia Institute of Technology.

"It comes down to the availability and reliability of [our control] systems," Bauhs said. "If the vendors fail to address that, somewhere along the way, their systems run the risk of not being reliable or available. There needs to be more participation from plant control vendors in this space," he said. In response to threats, many vendors now operate on Intel platforms running Microsoft operating systems, whether NT or 2000 or XP. Microsoft comes out with updates to the operating system (OS) fairly regularly, and "I think it's at a pace the plant control vendors aren't accustomed to," he said. "So when threats are identified and the OS manufacturers come out with fixes, you can't just take them as they are, because it may be disruptive to the control system you're running." Bauhs said plant control vendors—Allen Bradley, Rockwell, Siemens, Yokagawa—need to be more responsive.

Users of control systems need to apply protection around the control systems, such as using firewalls and other network protection devices, Lowe said. "Vendors need to be able to test the patches quickly," he said. "And vendors and other suppliers like Microsoft need to produce more stable systems—simpler systems that don't need patching as much. It'd be great if there were a simpler version of Windows that didn't include all the things you'd want for a home PC, like media player, which you don't need in a control system," he said. A simple system like that would not be vulnerable to any future discoveries, and you wouldn't need to patch them. Lowe said control engineers need to take more proactive steps in monitoring systems to make sure no one is trying to access them without authorization.

In days gone by, when plant personnel discovered a problem, vendors made the fix available, then "six months would go by before you would see the first real outbreak of someone trying to write a worm to exploit that vulnerability," Bauhs said. Now that has gone down to a few weeks. "You used to be able to say, I'll update my systems twice a year to patch all the holes identified. Now it only takes four weeks. It's difficult for any vendor or software provider to do it in four-week intervals."

Some vendors will certify that the system will process a certain number of transactions per second, minute, or hour. They have tight control of the configuration. And they do it because they have done extensive testing, Bauhs said. "If I have to update the OS multiple times, they have to go through that testing system. That's expensive. I think the OS providers, Microsoft and others, will have to find a better and less intrusive way to provide fixes."

Lowe helps companies such as BP understand the risks in systems and work out what protection measures they need to deploy. Some of those measures are technical—putting in firewalls or tightening networks—and some require human efforts—making sure systems are up to date and secured. "It's easy to write it down in a standard," he said, "but to roll it out in a company—especially the size of the big oil companies—can be difficult."

More ways to secure

Users need to think in terms of their security vulnerabilities, the risks they are willing to take, and what they can do to mitigate risks they are not willing to take, Sielinski said. They need to think of these systems as part of a broader information architecture and decide what they need to do to protect that. "Now, people talk a lot about viruses as if they were the only security threat. That's not really the case. There are multiple ways of breaching the security of a control system. People need to think in terms of multiple layers of security. Patching operating systems is one element of that. Antivirus software is potentially another element. Installing firewalls is another," he said.

There is also segmenting networks either physically or logically to ensure that communications traffic only goes where it needs to go, and that only appropriate traffic is reaching those control systems, Sielinski said. Users might also take a look at some of the work that standards organizations are doing to help guide them. ISA's SP99 and NIST's process control security and reliability forum (PCSRF) are a couple of guides. PCSRF is defining the requirements for a secure, reliable process control system. What sorts of functional features should you be looking for? How do these systems control access, configure user accounts, and define profiles?

"We've adopted an approach to securing our products known as SD3 + C—secure by design, default, deployment, and communications," Sielinski said. Securing by design means engineering the product so it is more secure. One of the more high-profile activities associated with this engineering was the Windows security push. "We stopped development for a couple of months, trained developers on writing secure code, and reviewed code looking for known vulnerabilities," he said. "Then we developed automated procedures for exercising code and eliminated a lot of bugs."

Secure by default is the idea of configuring the operating system so it is the most resistant to attack by default. "In a server product, when you install the operating system, the majority of services would be turned on—field and print for example," Sielinski said. "We're turning all those off by default and forcing users to turn on the ones they're going to use, so they don't have any exposed entry ways they aren't aware of and aren't actually using."

And then there's securing by communication. "We're due to have an automated mechanism whereby people can be notified of patches, have them downloaded, and install them," he said. It doesn't fall to the user to go to a Web site and look for patches. "We'll expand that capability to all Microsoft products. The mechanism for receiving the patch is for the user to connect to Windows updates online. It's never going to be something that's sent to you in an e-mail message and, you click on it and install it. People should never install an update from an e-mail message."

Security breaches are real By Ellen Fussell Control system security breaches aren't just lurking in the shadows; they're real, said Joe Weiss, executive consultant at KEMA Consulting in Cupertino, Calif. In addition, they are happening globally. There are more than forty confirmed cases where control systems have been affected by cyberattacks in electric power, water, oil and gas, paper, and other industrial facilities, the security consultant said. In spring of 2000 in Maroochy Shire, Australia, a disgruntled former employee remotely accessed the controls of a sewage plant to discharge nearly 264,000 gallons of untreated sewage into the local environment. Using the supervisory control and data acquisition (SCADA) system, the perpetrator "remotely opened wastewater discharge valves repeatedly before the utility realized the problem was not a hardware failure," Weiss said. "He was ultimately caught, but not before creating significant damage to the environment." During his keynote address at Distributech 2004, Weiss identified other examples of cyberimpacts on control systems. In some of the episodes perpetrators directly attacked control systems, such as a facility that had its electric distribution system and utility boiler control system hacked. In other cases, control systems have been subjected to Internet worms and viruses. Although the Ohio Davis-Besse nuclear plant's January 2003 security breach didn't pose a safety hazard (the plant had been offline since February 2002), the Slammer worm penetrated a private computer network and disabled several plant systems for nearly five hours. The Slammer worm also affected a substation SCADA system for several hours and a utility's energy management system telemetry for almost eight hours. In another incident, Gazprom, a Russian gas monopoly, was one of "a growing number of targets hit last year by computer hackers, who controlled the company's gas flows for a short time," according to an MSNBC report from 26 April 2003. The report said hackers worked with a Gazprom insider to elude the company's security and break into the system controlling gas flows in pipelines. This put the central switchboard of gas flows under the control of external users. Gazprom is the largest natural gas producer and supplier to Western Europe. It supplies about 95% of Russia's natural gas and controls nearly a quarter of global natural gas resources, delivering its products to 25 countries. The report said hackers used a Trojan horse program, which stashes lines of harmful computer code in a benign-looking program. None of the forty-plus cases Weiss cited were identified in the traditional cyber-reporting organizations such as CERT, SANS, and CSI. The control system cases demonstrate the need for establishing a "CERT for control systems," Weiss said.