July 2008

Factory Automation

Secret's in the system

Planning, shared priorities key to securing discrete systems


  • Multiple devices make discrete vulnerabilities hard to pinpoint.
  • Top-down planning means design, finance keys to secure systems.
  • IT, manufacturing still not sharing priorities.
  • Patch management needs work.
By Ellen Fussell Policastro

If you are packaging pills, building automotive parts, or labeling bottles of soda, you might think your processes do not have to be as secure as a nuclear plant or chemical plant.

Think again.

So far, the discrete industry has perceived requirements on their side of the fence as being quite different. But while some on the inside say there are differences, others believe the need for security is equal in both types of manufacturing, and the risks are indeed great.

In the process world today, the priorities are first safety, which includes the public safety "because some manufacturers in the process industries are considered a critical infrastructure by the Department of Homeland Security (DHS), and they have certain regulations they have to deal with. So safety comes first, then financial considerations," said Bob Mick, vice president of emerging technologies at ARC Advisory Group in Dedham, Mass.

Conversely, discrete manufacturers are more likely to prioritize based on equipment and employee safety, then financial because of the lack of financial impact on regulation, Mick said. If a safety shutdown occurs in the auto industry, there is a huge impact on business. "If a robotics lines shuts down, it's not as big of a safety issue as if oil were dumped into the San Francisco Bay," he said.

But securing discrete systems is not all that easy. The process industry is composed of larger systems and networks, which have often been designed in-house. Discrete manufacturing tends to be composed of smaller systems often purchased as a complete system from OEMs. "This can make securing discrete systems much more difficult since you are dealing with multiple vendors and multiple designs," said Troy Embree, MAIT technology leader with P&G in West Chester, Ohio.

Addressing vulnerabilities

In order to deal most effectively with vulnerabilities, there is complete isolation, which might only work for certain machine control. "There's surrounding devices with security, firewalls, and things like that," Mick said. Then there is zone-based isolation. Complete isolation means nothing can access this device. Zone-based says there is a computer that can communicate with that to download programs, but that computer cannot communicate with anything else.

Some people talk about multi-layers, such as security at the outside edges, such as firewalls, separating the plant systems from business systems. "But it goes deeper," Mick said. "Once I get inside this system, what if a worm or virus gets inside the system? Do I segment my system so a worm can only affect certain areas? What policies do I have for when people come in and plug into my networks or when contractors come in to work on a system?"

One of the most important vulnerabilities is operating-system-based vulnerabilities, such as Windows and Linux, Mick said. There is a constant stream of new issues with new patches. If a manufacturer is using a commercial system, they will have a constant stream of new vulnerabilities, buffer overflows, execute code, and protocol vulnerability.

"If they're using a proprietary operating system, they don't have that issue," Mick said. But they have some protocol where that proprietary system communicates with higher level systems than operations management or business systems. And those protocols can have vulnerabilities in them. "If you have a protocol download a recipe into a device that's not susceptible to worms, if a hacker can download a recipe, they can download a recipe that could alter a recipe in the pharmaceutical industry, such as a slight variation of medication you're making," he said. In automotive, it might be a program that makes a bad part or dumps material where it should not or at the wrong time.

The OLE for process control (OPC) standard that helps devices communicate with different vendors' equipment is also prone to vulnerabilities. "OPC is such an important part of the discrete world," said Eric Byres, CTO of Byres Security Inc., in Lantzville, BC, Canada, and author of a Kraft Foods-funded white paper, "OPC Exposed." OPC deployment does not require special skills or esoteric controls knowledge, so all the tools and information needed to carry them out can be downloaded from the Internet, Byres said. Two core vulnerabilities, excessively open firewalls and overly permissive distributed component object access rights, are the impetus for many security breaches. "If either vulnerability is addressed, then the chance of these scenarios occurring is significantly reduced," he said. "These vulnerabilities could be considered the responsibility of and within the control of the knowledgeable OPC end user. OPC security (especially authentication and authorization) is tied to operating system security so a poorly configured OPC application can adversely affect the security of the underlying operating system and vice-versa.

The OPC Data Access standard was created in 1995 to solve an interoperability opportunity between the devices on the factory floor and first-tier visualization and SCADA applications. "The OPC data access standard was so widely adopted by both members and nonmembers of the OPC Foundation, we deliberately did not try to address all the problems of the world with the very first solutions to facilitate widespread adoption of the technology," said Tom Burke, OPC Foundation president and executive director. "We were working with many vendors that didn't necessarily even have programmers on staff and did not understand the intricate details necessary for security to be implemented," he said. "Our focus also was to keep data inside of the corporate firewall and not to expose it into the IT world in the first eight years of the OPC Foundation." The other trend in general is a broader range of targets, Mick said. "There are Windows vulnerabilities, but also Linux in MAC OS, and in network devices. Even security devices have vulnerabilities. It makes for bigger targets for people writing worms and viruses." And it's an ongoing battle, Mick said. "It'll never go away. But it's not all gloom and despair. There's actually been better security, and they're dealing with the problem. It's not something you do once and it's done. I do think, because of nature, there will be more government regulations, and it might even go to a broader range of industries."

Byres said he believes the security of most OPC systems would be greatly enhanced if vendors improved the quality of configuration guidance to include recommended security settings and provided easy-to-use hardening scripts to automatically enable more reasonable security settings after installation. A few vendors have moved in this direction, but the vast majority has not.

The good news is the IT security community knows well the hardening practices for the windows operating system, which the controls community can adopt to significantly reduce these risks. "OPC's reliance upon the Microsoft platform is both a curse and a blessing," Byres said. "While Windows has flaws, there are a wealth of practices of hardening Windows servers that can be easily applied to OPC clients and servers."

Planning key to secure systems

Security policies and overall direction usually emanate from the top down in discrete manufacturing. So planning should take place up front, in design and financial decisions to ensure security is on the docket. Often individual plants develop solutions to meet policies and work up through the company. Cost is also a critical item in determining your security plan; but ensuring you achieve the right balance or risk versus mitigation for the business is not always that easy. "There is not a lot of data for precisely estimating the risk to manufacturing," Embree said. "Security organizations would have a lot easier time determining and funding the correct security if industry organizations could help accumulate this data."

To ensure a secure environment and to do it correctly takes time and money, said David Bauman, technical director of OMAC in Fort Wright, Ky. "When these systems were initially installed, in some cases, the support costs were not factored in," he said. "Safe practices need to ensure that even if there is a security breach there are procedures in place to make the operation safe. If an automotive manufacturer decides to manufacture a new automobile, they'll build a new plant to do that. Any new 'green' cars coming out will see production in new plants."

Bryan Singer vice president of professional services at Wurldtech Securities in Vancouver, BC, Canada, and chairman of the ISA99 standard committee on control system security, has seen very little security activity out of the automotive industry. In terms of network design and architecture, however, "they are all gung-ho," he said. "Most of the time, they just don't realize it's a security issue. For most of the automotive industry and discrete in general, with lots of functions and differing control architectures used at each station, the need for sound design for uptime and availability is key."

"Many think security is all about the hackers and crackers, but it really is about protecting the environment from any type of business loss, including network failure and process line inefficiencies," Singer said. "The key here is that good security design includes good network design, but the converse is often not true. Good network design without considering threats to network and application availability and integrity may yield a great network under nominal conditions, but any event can bring down cascading network failures."

IT key players in plant floor security

One of the differences between securing the plant floor and the enterprise is the focus on applying security. IT typically focuses on confidentiality, then integrity and availability, Embree said. Manufacturing has the opposite priority; availability is the most important, followed by integrity and confidentiality. Scale and downtime are also key differences between IT and manufacturing. IT often works on securing enterprise-wide systems. "In manufacturing, we must deal with securing more small systems that are all slightly unique from each other. This is compounded by the inability to find a common downtime for all systems."

IT leads the overall security effort for the company, and manufacturing is an integrated subset of this overall effort."

"You can't do patch management in manufacturing the same way as on desktops. You can't push patches down," Mick said. Manufacturing people have to pull down. They have to schedule when they can use it. It is the opposite for the desktop. "IT just shows up one morning.  You may need mitigation strategies to deal with high risk solutions you can't afford to deal with. You may have to disconnect a system temporarily, and do the thing manually. You deal with it temporarily, and as soon as your system comes down, you do something else. So operations strategies drastically differ."

"Since most manufacturing facilities are managed by engineering, and since engineering and IT in many companies have not had the best working relationship, engineering has not been aware of and or taken advantage of some of the automated tools IT uses to manage security," Bauman said. "These automated tools need to be implemented and managed in a way that meets manufacturing needs.  You can't just blindly download an update to your anti-virus software on a PC that may be running your manufacturing operation (such as an HMI).  This needs to be coordinated with the manufacturing schedule."

Bauman believes engineering and manufacturing need to make sure IT is educated on what is important and different from the enterprise so the systems to manage security are set up to meet those needs. "IT and engineering are starting to work better but there is still a long way to go for most organizations to improve this working relationship," he said. "It is an education process to understand what IT can offer and what engineering can do to educate IT about the differences."

Depending on patches

Due to the wide diversity of business requirements, Embree's team has not found one patch management process that works across all businesses. "In most cases, we try to utilize downtime scheduled for reasons other than just applying updates, but for some systems that rarely shut down, it happens. This is the case more often for utilities or process systems."

Discrete manufacturers need to move past looking at security on a machine-by-machine basis and drive common security across the manufacturing plant, Embree said. "A few years ago, our focus was around the basics of anti-virus and patch updates." With a better understanding, Embree's team is now focusing more on securing the configuration of each of their systems. "This involves both the user side of ensuring individual authentication and computer side (limiting the applications and services to only what is required)," he said.

Scheduling downtime for Microsoft security patches is getting better but still needs work, Bauman said. "Engineering/manufacturing is reluctant to install a patch unless the software vendor (e.g. HMI software vendor) says it is okay.  This is tough for the HMI vendor since each application tends to be set up a little differently," he said. "However, engineering is realizing that we can't wait forever for the answer if the patch is covering up a big security concern. How they are installed will vary depending upon how mission-critical the application is. A number of applications need a pull rather than a push system for updating."

Knowing what they know now, as opposed to five years ago, Bauman's team is doing a better job of updating operating system patches and antivirus updates, rather than installing and not touching them unless they break or are in need of an upgrade for a manufacturing change.

One of the areas where Microsoft has made significant improvements in the past few years is in making applications, processes, and the entire Windows operating system more secure out of the box, Byres said. "Although a tough default security posture with restrictive access control and fewer enabled services may make an application more difficult to initially configure, in the long run, this may be the best security solution," he said. On the other hand, making the default too restrictive may cause the user to configure the application to the most insecure (permissive) status just to get it operational. Consequently, the industry needs a balance of security and usability.


Ellen Fussell Policastro is the associate editor of InTech. Her e-mail is efussellpolicastro@isa.org.


OPC security issues

By Eric Byres

Object, linking, embedding (OLE) for process control (OPC) represents an open standard that specifies communication of real-time plant data between control devices from different manufacturers. The standard is based on the Microsoft distributed component object model (DCOM) interface of the remote procedure call (RPC) service. Due to its perceived vendor-neutral position in the industrial controls market, OPC is increasingly interconnecting HMI workstations, data historians, and other hosts on the control network with enterprise databases, enterprise resource planning systems and other business-oriented software. Viruses and worms from the IT world may be focusing more on the underlying RPC/DCOM protocols used by OPC. We believe the most serious issue for OPC is that securely deploying OPC applications has proven to be a challenge for most engineers and technicians. While OPC is an open protocol with the specifications freely available, engineers must wade through a large amount of very detailed information to answer even basic security questions.

Two core vulnerabilities, excessively open firewalls and overly permissive DCOM access rights, lay at the heart of many scenarios. The wide-spread adoption of OPC standards for interfacing systems on both the plant floor and the business network is a classic example of both the benefits and risks of adopting IT technologies in the control world.

Password vulnerabilities

Given OPC's reliance on Microsoft authentication mechanisms, weak passwords are among the most critical vulnerabilities that can undermine the security of an OPC server. The poor selection of passwords is the tipping point of the security of any application, operating system, or device. To ease administration, it might make sense to use the term OPC as part of the username or group name, but OPC, the organization, vendor, or product name or process control information should never be used in the actual passwords.

OPC responds

If in 1995 OPC Foundation knew what it knows now about network and internet security and the complications and pitfalls of DCOM security, "we would probably not have built the OPC interface on COM," said Jim Luth, OPC Foundation technical director.

"We have been trying to guide our vendor and end-user members through the DCOM quagmire as best we can," Luth said. To date, Eric Byres' white papers "offer the most in-depth analysis, and most importantly they recommend solutions to harden OPC installations. We have reviewed [Byres'] whitepapers and agree with his findings and recommendations," he said.

For the past four years OPC Foundation has been designing the OPC Unified Architecture (OPC UA) as a replacement to the aging OPC COM interfaces. "Not wanting to repeat the mistakes of the past, OPC has designed in security from the beginning that is not dependent on any vendor's security infrastructure," Luth said.  "Instead, we have designed OPC UA with the latest state-of-the-art security mechanisms, and they have been reviewed all along the design process by security experts like Eric [Byres].

While Byres' article points out the usual security problems are not in the implementation of the OPC standard, but in the deployment of a product in an end-user facility, "in some cases these deployment problems are exasperated by vendor product installation that default to no security or vendor documentation that does not provide clear instruction or suggestions for how to configure DCOM in a secure manner," said Paul Hunkar, director of compliance at OPC Foundation. 

"OPC Foundation has tried to address this issue via the work of the OPC Compliance Committee," which has specified an Independent Certification Test Lab, he said.

Source: Eric Byres (eric@byressecurity.com) is chief technical officer of Byres Security Inc., in Lantzville, BC, Canada, and author of a Kraft Foods-funded white paper, "OPC Exposed."