Comment

1 November 2006

SCADA sewage control no day at the beach

By Curt Wendt, Robert Murphy, and William Nelson

Lessons learned: Designing and implementing a radio based system

Florida's Orange County has 100 days of lightening per year, is hot and humid, gets heavy rain and wind, and has lots of vegetation. They all play havoc with radio systems.

Nonetheless, the county uses a new state-of-the-art radio-based SCADA to automate the monitoring and control of 600 sewage lift systems connecting to three wastewater treatment plants that has grown from 300 to 600 in a short time because of a burgeoning population.

In fact, the county's wastewater collection system is growing by almost 50 lift stations a year. Each additional lift station increases this management effort exponentially. The tremendous number of lift stations is a tracking and coordination challenge.

The SCADA system monitors each lift or pumping station through a radio communications system. Flow, wastewater levels, and pump-status data transmits to a central location where personnel can be dispatched to make repairs and correct problems.

Because of lightning concerns, the Remote Telemetry Units (RTUs) do not control the stations—the preexisting relay logic panels stayed in place for that purpose with a limited amount of control available for manual starting and stopping of pumps.

Design criteria for the new system included robustness, standard products, lightning tolerance, sunshield protection of field panels, resistance to Radio Frequency (RF) interference, and low recurring charges for phone service.

Overall, the radio approach has been successful and has saved thousands of dollars in dedicated line fees. However, this is not the tale of a trouble free installation, and next time we will do some things differently.

Radio band selection

Whenever possible, use a data-only radio band for SCADA radio systems. Previous attempts to operate a SCADA system for the Orange County Utilities (OCU) Water Division were unsuccessful.

OCU Water used two licensed frequencies in the 450 MHz range for SCADA data communications to 18 sites. Additions over the years required the use of repeater sites and a combination of series and parallel branches to achieve the required links. Failures at a sequential repeater site caused a loss of communications to sites located beyond that point. Because of this, the system was extremely vulnerable to interference, lightning strikes, thermal overload, and equipment failure.

Additionally, the radio equipment was custom made and required specialized training and test equipment for calibration and repair. Adjacent radio users include the voice designation of "land mobile," which includes taxi service, trucking dispatch, and the like. Interference came from adjacent frequency splatter and close proximity to transmitters as mobile users drove through the area. All these factors, coupled with the slow response of the communications (data bottleneck), made the former radio-based system less than desirable.

The OCU Water Division ultimately abandoned the 450 MHz radio system and moved its sites to ISDN phone lines. This was economical for the Water department because there were fewer than 20 remote sites.

The Waste Water division of OCU needed to select a radio band for their new SCADA system. The experience in the water division ruled out using the 450 MHz band. Adapting the OCU 800 MHz trunking radio system was a possibility ruled out early in the process because this system shares with other users and is slow.

There were no available licensed 928/952 MHz point-to-multipoint frequencies in the Orange County area. Spread spectrum radios in the 900 MHz band were the best design because of their ease of use and license free status.

Designed to tolerate interference well and operate in a data only band, there are no voice users, and they are widely available from a number of manufacturers. Because they operate using only 1 watt, properly locating the master radios is a critical issue and might have involved telephone leased-lines from the control room to distant locations. By using an innovative combination of radio, computer network, and telecommunications equipment, OCU's existing investment in micro-wave communications helped to reach distant master radio sites spread throughout the county. The remote radio diagnostic software that came with the radios has proved invaluable in tracking down radio problems.

Radio survey of the land

Our software radio survey between county radio towers and wastewater lift stations showed 30-ft poles would be high enough to enable communication to most, if not all, sites. It appeared that using 150-ft master radio towers and 30-ft poles at each end site would suffice for good radio reception through-out the county.

However, software surveys are only a rough determination that a radio path may exist; a field survey is required to confirm it. The field survey should happen early in the process, preferably during initial system design.

In this project, the field survey took place late in the project. Technicians temporarily installed five-watt 928 MHz transmitters and antennas with known gains at each master tower. Radio technicians at each remote location rotated a directional test an-tenna to maximize the intensity of the signal and to assure reception is through the main antenna lobe. Then they record the received signal strength and the GPS measurements.

In contrast to the initial software survey, the field results showed 30-, 50-, 70-, and 90-ft poles would be necessary. Installing 70- and 90-ft poles in a commercial area may not pose a problem. Installing them in a neighborhood would likely have a different result.

Since the field survey people mathematically adjusted their results to determine pole height, verification was necessary. To confirm the field survey results, workers installed 25 sites to compare field radio survey to the actual real world radio-signal strength.

After installation, six sites had unexpectedly weak received signal strength indication (RSSI). A specialty communications contractor came into inspect those sites with the worst performance. Using a Time Domain Reflectometer and visual inspection, they found and corrected various installation problems:

• An internal jumper from radio to bulkhead connector on RTU was defective. This was a locally made cable that had a single strand of wire out of place making an RF short inside the connector. Using products from major manufacturers that specialize in RF cable would avoid this quality problem.
• An antenna mast was crooked and not truly vertical. This caused the antenna radiation to propagate ineffectively.
• Water was in an antenna connector at top of a mast causing an RF short. Weatherproofing was inadequate.
• Loose RF connectors were causing bad electrical connections.
• Re-aiming an antenna led to a three dB improvement in received signal.
• A bent antenna, though barely noticeable at arm's length, necessitated the replacement of the antenna.

After the repairs, the field data closely matched the predicted results of the field survey.

In order to provide an estimate of repeaters needed, we reanalyzed the field RSSI measurements in the field radio survey. A repeater retransmits the radio signals received to increase the overall coverage area.

Techs selected potential repeater sites from the sites that could communicate to the Master radios at a 30-ft height as documented in the field radio survey. They narrowed the selected sites further to those that could visually permit a 50-ft pole.

Based on the repeater survey results, it was possible to reduce the pole height at 96 sites to the heights recommended in the visual survey. Of the 30 sites surveyed, 20 repeater sites were the best repeater locations. A total of 307 sites were identified that can be installed and meet both the visual survey recommendation and the target RSSI level.

Licensed radio

At that time and because of certain FCC frequency re-farming in the 900 MHz band, a new set of frequencies became available in mid project. OCU successfully obtained 10 licenses in the 938/952 MHz band. The possibility of replacing spread spectrum radios with licensed radios to improve the RF fade margins in the system went on the table.

OCU bought the licensed radio equipment and performed a limited field test to compare the system performance change associated with the licensed radios. Staff converted one of the 10 channels from spread spectrum radios to licensed radios over a four-day period.

The licensed radios improved field system performance in all key areas: RSSI increased by 10 dB, each site's percent of communication failure im-proved dramatically, and channel- polling speed improved 10%.

The county subsequently made plans to replace most of the spread spectrum radios with licensed radios to get the benefits of increased fade margin.

These are the lessons learned:

• Whenever possible, use a data only radio band for SCADA radio systems.
• Software surveys are a rough determination that a radio path may exist. A field survey will confirm the path exists. The field survey needs to happen early in the process, preferably during initial system design.
• Identifying system design constraints early is important. These include technical constraints and site-specific issues. Few people want a 90-ft pole visible from their backyard.
• Field-verify extrapolated data. If a field survey uses extrapolation to determine pole heights, it needs to be field verified to demonstrate the assumptions made hold true.
• Intelligently choose repeater sites to reduce the number of paths studied. Evaluating multiple radio paths can become very expensive and time consuming. Choose what appear to be the best candidates for repeater field surveys.
• Power and lower data rates both enhance stability. For SCADA radio systems, use the most powerful transmitters possible and minimize the data that transmits so high data rates are not required.

The Orange County Utilities radio-SCADA project is a positive example of the using the design/build process. The project team built upon a foundation of technical expertise and professional respect to produce a successful project. OCU now has a state-of-the-art SCADA radio system that reliably monitors over 300 sewage lift stations, facilitates proactive decision-making for enhanced operation of the County's facilities, and provides real-time data about flow characteristics and other functions.

ABOUT THE AUTHORS

Curt Wendt (wendtcw@cdm.com) is a senior member of ISA, a member of IEEE, and a registered PE in six states. He is the Orlando automation group leader for Camp Dresser & McKee (CDM) and has over 18 years of experience in I&C design and commissioning. Robert Murphy is also a PE and works as a project manager for Orange County Utilities. William Nelson is a PE and is vice president at CDM.