1 November 2006
SCADA sewage control no day at the beach
By Curt Wendt, Robert Murphy, and William Nelson
Lessons learned: Designing and implementing a radio based system
Florida's Orange County has 100 days of lightening per year, is hot and humid, gets heavy rain and wind, and has lots of vegetation. They all play havoc with radio systems.
Nonetheless, the county uses a new state-of-the-art radio-based SCADA to automate the monitoring and control of 600 sewage lift systems connecting to three wastewater treatment plants that has grown from 300 to 600 in a short time because of a burgeoning population.
In fact, the county's wastewater collection system is growing by almost 50 lift stations a year. Each additional lift station increases this management effort exponentially. The tremendous number of lift stations is a tracking and coordination challenge.
The SCADA system monitors each lift or pumping station through a radio communications system. Flow, wastewater levels, and pump-status data transmits to a central location where personnel can be dispatched to make repairs and correct problems.
Because of lightning concerns, the Remote Telemetry Units (RTUs) do not control the stations—the preexisting relay logic panels stayed in place for that purpose with a limited amount of control available for manual starting and stopping of pumps.
Design criteria for the new system included robustness, standard products, lightning tolerance, sunshield protection of field panels, resistance to Radio Frequency (RF) interference, and low recurring charges for phone service.
Overall, the radio approach has been successful and has saved thousands of dollars in dedicated line fees. However, this is not the tale of a trouble free installation, and next time we will do some things differently.
Radio band selection
Whenever possible, use a data-only radio band for SCADA radio systems. Previous attempts to operate a SCADA system for the Orange County Utilities (OCU) Water Division were unsuccessful.
OCU Water used two licensed frequencies in the 450 MHz range for SCADA data communications to 18 sites. Additions over the years required the use of repeater sites and a combination of series and parallel branches to achieve the required links. Failures at a sequential repeater site caused a loss of communications to sites located beyond that point. Because of this, the system was extremely vulnerable to interference, lightning strikes, thermal overload, and equipment failure.
Additionally, the radio equipment was custom made and required specialized training and test equipment for calibration and repair. Adjacent radio users include the voice designation of "land mobile," which includes taxi service, trucking dispatch, and the like. Interference came from adjacent frequency splatter and close proximity to transmitters as mobile users drove through the area. All these factors, coupled with the slow response of the communications (data bottleneck), made the former radio-based system less than desirable.
The OCU Water Division ultimately abandoned the 450 MHz radio system and moved its sites to ISDN phone lines. This was economical for the Water department because there were fewer than 20 remote sites.
The Waste Water division of OCU needed to select a radio band for their new SCADA system. The experience in the water division ruled out using the 450 MHz band. Adapting the OCU 800 MHz trunking radio system was a possibility ruled out early in the process because this system shares with other users and is slow.
There were no available licensed 928/952 MHz point-to-multipoint frequencies in the Orange County area. Spread spectrum radios in the 900 MHz band were the best design because of their ease of use and license free status.
Designed to tolerate interference well and operate in a data only band, there are no voice users, and they are widely available from a number of manufacturers. Because they operate using only 1 watt, properly locating the master radios is a critical issue and might have involved telephone leased-lines from the control room to distant locations. By using an innovative combination of radio, computer network, and telecommunications equipment, OCU's existing investment in micro-wave communications helped to reach distant master radio sites spread throughout the county. The remote radio diagnostic software that came with the radios has proved invaluable in tracking down radio problems.
Radio survey of the land
Our software radio survey between county radio towers and wastewater lift stations showed 30-ft poles would be high enough to enable communication to most, if not all, sites. It appeared that using 150-ft master radio towers and 30-ft poles at each end site would suffice for good radio reception through-out the county.
However, software surveys are only a rough determination that a radio path may exist; a field survey is required to confirm it. The field survey should happen early in the process, preferably during initial system design.
In this project, the field survey took place late in the project. Technicians temporarily installed five-watt 928 MHz transmitters and antennas with known gains at each master tower. Radio technicians at each remote location rotated a directional test an-tenna to maximize the intensity of the signal and to assure reception is through the main antenna lobe. Then they record the received signal strength and the GPS measurements.
In contrast to the initial software survey, the field results showed 30-, 50-, 70-, and 90-ft poles would be necessary. Installing 70- and 90-ft poles in a commercial area may not pose a problem. Installing them in a neighborhood would likely have a different result.
Since the field survey people mathematically adjusted their results to determine pole height, verification was necessary. To confirm the field survey results, workers installed 25 sites to compare field radio survey to the actual real world radio-signal strength.
After installation, six sites had unexpectedly weak received signal strength indication (RSSI). A specialty communications contractor came into inspect those sites with the worst performance. Using a Time Domain Reflectometer and visual inspection, they found and corrected various installation problems:
An internal jumper from radio to bulkhead connector on RTU was defective. This was a locally made cable that had a single strand of wire out of place making an RF short inside the connector. Using products from major manufacturers that specialize in RF cable would avoid this quality problem.
An antenna mast was crooked and not truly vertical. This caused the antenna radiation to propagate ineffectively.
Water was in an antenna connector at top of a mast causing an RF short. Weatherproofing was inadequate.
Loose RF connectors were causing bad electrical connections.
Re-aiming an antenna led to a three dB improvement in received signal.
A bent antenna, though barely noticeable at arm's length, necessitated the replacement of the antenna.
After the repairs, the field data closely matched the predicted results of the field survey.
In order to provide an estimate of repeaters needed, we reanalyzed the field RSSI measurements in the field radio survey. A repeater retransmits the radio signals received to increase the overall coverage area.
Techs selected potential repeater sites from the sites that could communicate to the Master radios at a 30-ft height as documented in the field radio survey. They narrowed the selected sites further to those that could visually permit a 50-ft pole.
Based on the repeater survey results, it was possible to reduce the pole height at 96 sites to the heights recommended in the visual survey. Of the 30 sites surveyed, 20 repeater sites were the best repeater locations. A total of 307 sites were identified that can be installed and meet both the visual survey recommendation and the target RSSI level.
At that time and because of certain FCC frequency re-farming in the 900 MHz band, a new set of frequencies became available in mid project. OCU successfully obtained 10 licenses in the 938/952 MHz band. The possibility of replacing spread spectrum radios with licensed radios to improve the RF fade margins in the system went on the table.
OCU bought the licensed radio equipment and performed a limited field test to compare the system performance change associated with the licensed radios. Staff converted one of the 10 channels from spread spectrum radios to licensed radios over a four-day period.
The licensed radios improved field system performance in all key areas: RSSI increased by 10 dB, each site's percent of communication failure im-proved dramatically, and channel- polling speed improved 10%.
The county subsequently made plans to replace most of the spread spectrum radios with licensed radios to get the benefits of increased fade margin.
These are the lessons learned:
Whenever possible, use a data only radio band for SCADA radio systems.
Software surveys are a rough determination that a radio path may exist. A field survey will confirm the path exists. The field survey needs to happen early in the process, preferably during initial system design.
Identifying system design constraints early is important. These include technical constraints and site-specific issues. Few people want a 90-ft pole visible from their backyard.
Field-verify extrapolated data. If a field survey uses extrapolation to determine pole heights, it needs to be field verified to demonstrate the assumptions made hold true.
Intelligently choose repeater sites to reduce the number of paths studied. Evaluating multiple radio paths can become very expensive and time consuming. Choose what appear to be the best candidates for repeater field surveys.
Power and lower data rates both enhance stability. For SCADA radio systems, use the most powerful transmitters possible and minimize the data that transmits so high data rates are not required.
The Orange County Utilities radio-SCADA project is a positive example of the using the design/build process. The project team built upon a foundation of technical expertise and professional respect to produce a successful project. OCU now has a state-of-the-art SCADA radio system that reliably monitors over 300 sewage lift stations, facilitates proactive decision-making for enhanced operation of the County's facilities, and provides real-time data about flow characteristics and other functions.
ABOUT THE AUTHORS
Curt Wendt (firstname.lastname@example.org) is a senior member of ISA, a member of IEEE, and a registered PE in six states. He is the Orlando automation group leader for Camp Dresser & McKee (CDM) and has over 18 years of experience in I&C design and commissioning. Robert Murphy is also a PE and works as a project manager for Orange County Utilities. William Nelson is a PE and is vice president at CDM.
Remote radio or telephone? A comparison and case study on control room data.
Wired for oil: A SCADA architecture based on Foundation fieldbus and Ethernet secures inaccessible land locations and offshore platforms.
Network, Organize Thyself: Industrial environments could benefit from reliability of self-organizing wireless networks
Defending SCADA networks from hackers
At the Idaho National Laboratory, the nation's electricity grids, water and oil pipelines, and chemical plants hold the sway formally accorded nuclear power and alternative fuels.
This lab researches the U.S. critical infrastructure, cyber security, and industrial networking systems.
Cox News Service reported a typical day. On a couple of standard, off-the-shelf Dell computers, Jason Larsen and Steve Schaeffer enter a few keystrokes and then turn to see the havoc they have created.
To their left, red dots on a big-screen display blink on, then off. They have just shut down the equivalent of a city power grid.
"It's very doable," Larsen said with an almost mischievous grin. "If you want to do something, you can cause some problems."
With his ponytail, black T-shirt, and tendency to slip into geek-speak, Larsen fits the stereotype of a computer hacker.
Fortunately, he is one of the good guys.
As lead cyber security researcher at the Idaho National Laboratory here, Larsen, with help from fellow researchers like Schaeffer, spends his days trying to hack into networks like the ones that run the nation's electricity grids, water and oil pipelines, and chemical plants.
As they demonstrated recently to a visiting reporter, often it is not that hard. Moreover, such acts could soon become more common in the real world, they warned.
"I would tell you it's a simple and safe bet that we will see more" attacks on Supervisory Control and Data Acquisition (SCADA) networks, said Mike Assante, chief strategist for critical infrastructure at the Idaho National Lab.
Founded in 1949 in the barren desert plains of southeastern Idaho, the lab sprawls over about 890 square miles, an area nearly three-fifths the size of Rhode Island. It got its start testing nuclear power reactors and today has 52 of them, most built as test units.
During the oil crisis of the 1970s, the lab expanded into researching alternative fuels.
Two years ago, it got an additional mission. Sensing the growing danger to the nation's SCADA networks, the Department of Homeland Security and other federal agencies charged the lab with starting the nation's first-ever SCADA "test bed" operation.
Some cyber security experts said it was one of the smartest things the DHS has ever done.
"If I had to pick one thing that DHS has done right, it is funding" the Idaho lab, said Eric Byres, a noted SCADA researcher and consultant. "We need someplace that can demonstrate to the industry that this stuff really matters."
Alan Paller, director of research with the SANS Institute cyber security training center, agreed. At least when it comes to cyber security, "this is the only place where I've seen DHS creating change in terms of making things better," Paller said. The involvement of the Idaho lab, he adds, is key to improving SCADA security.
The lab's equipment includes an actual power grid, complete with a working command center and substation where researchers can run tests.
It has dedicated data center area where SCADA hardware suppliers can test their equipment, as well as a mock chemical plant with mixers and robotic dispensers that serves as a high-tech guinea pig to test the effects of computer viruses and other maladies.
Researchers travel around the country to conduct voluntary audits of SCADA systems at companies. They also recently produced a procurement guide for SCADA purchasers that includes only the most secure equipment.
Research leader Larsen said he and others may be in a race against time to make SCADA networks more secure.At hacker conferences, on underground Web forums and elsewhere, hackers in the past couple of years have taken a noticeably increased interest in such networks, Byres said.
"They all know about this stuff now," he said. "And all of them are going to try this."