Comment

1 April 2005

Keeping your cool during a firewall installation

By Nigel James and Stephen Stocke

This historian implementation required three firewalls, one DMZ, two remote links, and lots of network savvy.

The expansion of real-time process management (RPM) systems, or historians, brings in new issues of network attack and threats as control systems pass data out on the corporate intranet.

There is a growing threat of network intrusion, especially with malicious hackers, terrorists, and viruses.

In response to this concern, the automation industry has focused on setting standards on manufacturing and control systems security to combat cyber threats. ISA – The Instrumentation, Systems, and Automation Society has now issued two technical reports on this topic:

• ANSI/ISA-TR99.00.01-2004, Security Technologies for Manufacturing and Control Systems
• ANSI/ISA–TR99.00.02-2004, Integrating Electronic Security into the Manufacturing and Control Systems Environment

Avers the first of these two documents, "The need for protecting manufacturing and control systems computer environments has grown significantly over the last few years. The combination of open systems; an increase in joint ventures, alliance partners, and outsourced services; growth in intelligent manufacturing equipment increased connectivity to other equipment/software; enhanced external connectivity; along with rapidly increasing incidents of network intrusion, more intelligent hackers, and malicious software, all lead to increased threats and probability of attack. As these threats and vulnerabilities increase, so does the need for protection of manufacturing and control system."

Far flung points of data

With hundreds of oil platforms in the Gulf of Mexico, the oil and gas industry is an important part of the area's economy. Similar to chemical manufacturing facilities, most have base control systems and some have ongoing archiving of historical data.

As companies become more global and the technology becomes more pervasive, the need for enterprise wide historians is key to making better decisions faster.

The case study company we reference here wished to integrate all process data from multiple platforms in the Gulf of Mexico and Caribbean into a single secure system with the data available in the corporate office in Houston in a secure system. Security of the data and system setup was a critical success factor of this project.

Specific details of the installation are:

• Corporate RPM server at client offices in Houston with capacity for 150,000 points
• Web portal in Houston for data trending, reports, key performance indicators (KPIs), and the like
• Local RPM servers at each production facility/interest
• Robust, secure data synchronization with Houston with minimal lag (less than one minute)
• Automatic recovery of data after network outages
• Gulf of Mexico platform (1,000 points)
• Caribbean platform (10,000 points)
• Easy administration of RPM servers at remote sites with no client presence

The solution was a Wonderware HMI (human machine interface), and the team opted for OSIsoft PI Data Historian for the RPM based on its installed base of oil and gas platforms and the client's head-to-head feature and cost comparison of several historian platforms.

The solution includes the RPM server on the platform and an additional server in the corporate office that will also gather the data from RPM servers on individual assets in North America.

This platform required three firewalls to protect the data communication to the corporate RPM server. These firewalls were required due to the platform having a joint venture partner, and the corporate networks of the two partners required firewalls at the interface.

In addition to the facility acceptance test (FAT), additional communications tests took place at operating company's corporate offices in Houston to verify the configuration of the firewalls at the network interface. This testing proved invaluable in validating the network configuration before commissioning activities on the platform commenced.

Logistical design issues

To get the data into the Houston office, there were several logistical design issues to cover. The data from the first platforms was to come from non-operated asset in the Gulf of Mexico. This would entail using line of sight microwave technology, which has an inherent bandwidth issue. Then the data was to transmit and share with a co-owner of the platforms while maintaining security on the confidential items. Four key issues came up during analysis of the project for which we needed solutions:

• Data integrity
• Bandwidth of wide area network (WAN) link
• Network security
• Remote management

Many platforms in today's commercial environment host multiple companies and alliances. This creates confidentiality issues as the non-operating partner does not have personnel on the asset on a full-time basis. This also adds complexity to the issues of data integrity and reliability. Being 100 miles offshore requires data to flow back via low-bandwidth line-of-sight microwave, which adds a layer of complexity.

The aforementioned ISA technical reports help us to find best practice approaches to solving these issues.

Firstly, TR-99.00.01 provides the list of categories of available security technologies and the strength and weaknesses of each one. These technologies break out as follows:

• Authentication and authorization
• Filtering/Blocking/Access control
• Encryption
• Data validation
• Audit
• Measurement
• Monitoring and detection tools
• Operating systems

For the purposes of this project we focus on the authentication, access control, and data validation.

The firewalls are the main tool for implementing these security measures, and proper configuration and testing of the firewalls helped to insure a smooth system commissioning and startup.

The focus of TR-99.00.02 is on applying the technologies to your system. With focus on planning, developing, and implementing activities, this is our most used document as consultants to the industry.

"In order to protect Manufacturing and Control Systems environments from potential threats and probability of attacks, each site or corporate entity should be responsible for developing an electronic security program and creating a security plan to protect manufacturing control networks.

This ISA Technical Report provides a framework for developing an electronic security program and provides a recommended organization and structure for the security plan."

Solutions in architecture

A local RPM server stayed on the platform to provide an additional layer of data reliability along with improved data compression and bandwidth utilization. Local storage of the process data greatly simplifies data recovery to the corporate server after a communication outage.

In the real-time business environment, one cannot have any moment in time without production data as that can mean safety issues or lost revenue. Being a remote platform in the middle of the Gulf adds another layer of complication since a third party operates the asset.

The key success factors in solving this data integrity issue were:

• Failover OPC Data Access servers
• Local RPM server to store data in case of network interruption to shore
• Wonderware data logging as on platform backup
• All RPM data is copied to the corporate RPM server in Houston via historian to-historian (PI-to-PI) link
• PI-to-PI link will automatically recover data after an outage

Using the slow WAN link (line of sight microwave) did create a pinch point for data to pass. For this reason, the team placed an RPM server on the oil platform. While this was more expensive, the performance improvement was significant and came about because:

• PI-to-PI link utilizes lowest bandwidth across the network with data already compressed.
• A test network environment to validate design and configuration was utilized.
• Further tuning of the RPM systems reduced network traffic.

The calculations and decisions on bandwidth are key design issues. This figure shows how exceptions testing and compression testing reduces the volume of data required to pass from the platform to the mainland.

Following ISA standards and protocol, the final design for the platform utilized three network firewalls. Strict firewall rule sets allow only RPM data traffic and remote management of the RPM server and interface computers.

Finally, the corporate RPM server has an additional firewall between it and the corporate network for added security. This is the process information network. The breakdown is slowly standardizing to the following layers:

• Enterprise network segment
• Process information network segment
• Control network segment
• Field network segment
• Process segment

Due to the remote nature of the system, the technical team made provisions in the firewall configuration to allow remote administration and management of the RPM historian on the platform. While this increased the security exposure of the system, it was necessary to reduce the requirement to have personnel fly to the platform to perform routine administration tasks. To this end, the firewalls included the following configuration:

• Firewalls configured to allow remote management of RPM tags from client offices in Houston.
• Firewalls configured to allow remote system management through standard terminal services and utilizing data encryption.
• RPM systems hardened to allow only required services and communication.

Primary lessons communicate

Network diagram: It is important to have a clear understanding of the network architecture along with the physical and software boundaries of the security design. To accomplish this, it's best to develop a network architecture diagram. By having this as your starting point, you have a clear, graphical representation of all the segments of the system along with what physical and software boundaries you can obtain to meet your security objectives.

A network diagram is the most efficient way to communicate the physical and logical network structure so everyone has a clear picture of the design. However, a detailed network diagram can also work as a roadmap to compromising your system. Therefore, any detailed diagrams, especially those containing firewall configuration information, should be treated as highly confidential and only distributed among trusted personnel with a "need-to-know."

Validation plan: In order to properly test the system and the security plan, we developed and executed three acceptance test procedures.

First, the facility acceptance test (FAT) took place at the Mangan office in Lake Jackson. The FAT tested the system setup and configuration along with extensive testing of the low bandwidth WAN communications and basic firewall rules. The RPM IT Monitor interfaces proved invaluable in tuning for the low bandwidth link to shore and in proving the data recovery mechanism over the link.

Next, a firewall configuration test at the operating company's Houston office tested the firewall configuration on both sides of the interface between the two company networks. The interface consisted of a leased line (DS-1) between the corporate offices of both companies in Houston. Each company installed a firewall on their end of the link to allow strict control of the data passed between the networks. This test did not involve the firewall on the platform but allowed thorough testing of the configuration of the firewalls at the corporate network interface. Once the configuration for these firewalls was accepted, the configuration for the firewall on the platform was simply a copy of the firewall configured on the operating company's corporate network interface.

Finally, the site acceptance test (SAT) was conducted after installation and commissioning of the RPM historian system on the platform. The SAT included additional tests of the communication through the firewalls, but at this point our prior tests made these tests academic. Proper planning and testing are imperative to implementing a successful security strategy.

Undocumented features: No matter what platform or vendor you choose, in the software world there are always surprises and software that does not function as expected. The vendor cannot predict all scenarios that the product may operate under. It is important to maintain a good partnership with all your vendors to help you solve you problems.

In the case of the interface between two historians, we initially followed vendor's recommendations on how to configure the interface. Once low bandwidth testing began, we quickly found the recommendations were problematic for the application. These issues were most evident when data recovery was occurring after a simulated network outage. We changed the design and tuned the server for better performance with the low bandwidth link. Not only did we increase the efficiency of data recovery, we also reduced normal operation bandwidth utilization significantly. We did all of that without sacrificing the plant data resolution.

While this issue does not relate directly to the security implementation, it demonstrates that thorough testing can often uncover unexpected issues and provide an opportunity to address those issues before system commissioning.

Firewall configuration: Properly specifying the firewall rules is a critical step to achieving your security goals. Here are some tips and general rules of thumb to follow that will help reduce frustration and delays during commissioning:

• Keep communication through the firewall to a bare minimum. Every hole that one pushes through the firewall represents a security risk. The more holes, the greater the chance that the system could be compromised.
• Never allow standard Microsoft protocols to pass through the firewalls. Examples include Microsoft Networking (NetBIOS) functions like Windows user authentication and file shares. If you need to give users access to certain files behind the firewall, find alternate methods like FTP scripts to copy the file to a location in the business network. Another example is OPC communication that relies on RPC and Windows user authentication as part of DCOM.
• Clearly define what communication needs to pass through the firewall. Once you have a list, research the communication technologies to determine what IP protocols and ports work. Ask the hardware/software vendor for this information. Having a list of every TCP/UDP port will help your network engineers to properly configure the firewall the first time.

New and existing systems

Installing a firewall as part of a new system installation is by far the easiest scenario. You typically have access to the actual control system equipment for testing of your firewall rules before installation, which greatly aids in troubleshooting.

In addition, one doesn't face the daunting task of having to change existing operating procedures and attitudes that may arise with the use of a new firewall.

When planning a firewall installation to protect an existing system, plan to spend extra time reviewing the current system and interviewing its users.

You will often find the openness of the existing system has resulted in the development of operating procedures that will have to change when the firewall installs. A very common example is plants using their historian as a file server. It's often a convenient place to store daily reports and other operating information that needs to disseminate amongst users. The firewall will prevent users from accessing a file share (Remember, no Windows file shares!) so a new operating procedure must be defined.

These new procedures are sometimes cumbersome for users until they are used to them. Expect to meet resistance and receive requests to just configure the firewall for the old methods or abandon the firewall installation completely. It's critical you communicate with the end users and stress the importance of the firewall as part of the companies overall security plan.

In addition, always plan for and communicate that there will be an interruption to network access to the control system/historian during commissioning. Due to the fact that you cannot properly test the firewall configuration before commissioning (the control system is in use already), you must plan extra time to work with the network engineers and troubleshoot any issues that arise with the firewall rules, and this troubleshooting will likely result in communication outages for users. Don't expect untested firewall rules will work on the first try. This is rarely the case and usually only happens with the simplest of firewall rule sets.

Finally, secure data has become a critical part of the control systems design. The industry has responded by developing standards for the manufacturing industry to follow. Adequate design and testing and more testing are the lessons of the day.

Scope of security standards Source: ISA, Eric Corman, CIDX

Behind the byline

Stephen Stocke (sstocke@manganinc.com) is a control systems specialist at Mangan Inc. He has a BSChE, and he designs and installs PLC/DCS and historians for refineries and chemical plants across the U.S. Nigel James (njames@manganinc.com) is president of Mangan Inc. He is a member of AIChE and ISA. He has a BSChE and worked in the refinery business for eight years after which he entered automation consulting where he has spent the past 12 years. This article derives from the paper that Stocke and James presented at the 60th Annual Instrumentation Symposium for the Process Industries at Texas A&M University in January 2005.