Comment

1 July 2002

Taking back control

By Tony Stafford

Most common way to implement a SCADA system is to go outside, but is it the best way?

Camrosa Water District chose to use a less common method for supervisory control and data acquisition (SCADA) implementation: in-house integration.

The goal was to replace an old, existing, proprietary system with a design that would improve operations in every respect. The desire was to not only improve the system but also implement a system over which the district would have full control, from design to installation.

It is our opinion that to truly take control of a SCADA project, in-house personnel should handle as much of the job as possible. This includes design, equipment specification, installation, and programming.

The more of these tasks one does in-house, the more control and ownership one has. Our continuous improvement philosophy required that the new system be versatile and user friendly, with the ability to grow and expand.

To accomplish this, we first evaluated the existing SCADA system and investigated new technologies to establish a list of features the new system needed to incorporate.

OUT WITH OLD LADDER LOGIC

The old SCADA system was installed in 1986. It was a PC with a hot backup to run the SCADA software and 30+ remote terminal units (RTUs) to receive and transmit data.

The remote sites included eleven reservoirs, six pump stations, eleven meter stations, five water production wells, and four sewer lift stations. The RTUs at these locations were mainly for gathering data and did little control.

The data transmitted via dedicated leased analog phone lines. The pump and equipment control was hard wired using relay logic. Because of the old system's proprietary nature, whenever we needed operational changes that required programming changes to the SCADA software or to the RTU programming, we had to hire the integrator back to make them.

Likewise, an electrician had to make wiring modifications to the relay logic whenever we needed to change control schemes at facilities. Modifications to the system were both time consuming and expensive.

Evaluating the old system and its shortcomings was an important step in determining the features we desired for the new SCADA. We considered new and desired features only if they met the criteria of improving control, improving efficiency, increasing productivity, or reducing costs.

BRIEF CONSIDERATION TO BIDDERS

While there are good third-party SCADA integrators, one must select them carefully. They are often selected through a bidding process, which does not ensure the quality of their work. The following are concerns to address when using an outside integrator:

• How well does the integrator know your operations?
• How well can you explain the desired operating procedure to the integrator?
• How is SCADA software selected?
• How well does the integrator know the software?
• How easy will it be to make changes to the SCADA operation after the warranty period has expired and the integrator is gone?
• Will a paid integrator have to return to make changes or upgrades?
• What will it cost to make these changes?
• How well can the integrator train company personnel on the new system?
• Does the integrator have the time (budgeted funds) to fine-tune the control system to the desired level?
• Does the integrator have the time (budgeted funds) to be as detailed in developing graphics, training, and the like as the owner desires?
• After completion, will the operator feel comfortable and have the working knowledge to make adjustments and changes to the system as the need arises?
• Have the operators bought in to the new system?

By eliminating the third-party integrator and taking on the task of in-house integration, the owner takes direct responsibility for a successful automation system.

While taking on the tasks of completing the integration in-house places the responsibility on the owner, it also eliminates most of the above-mentioned concerns regarding third-party integrators.

After years of experience and numerous SCADA projects, we are of the school that wants to avoid purchasing any proprietary equipment, or software or equipment not readily available off the shelf.

We decided all new equipment and software would come from companies that were well established and had a proven track record in the industry. To best accomplish this, there was no bidding process for equipment. Instead, we formed alliances with established companies to create win/win situations.

Establishing these relationships made it possible for us to obtain special price multipliers from distributors in exchange for the assurance that Camrosa would purchase a certain quantity of equipment. This approach also greatly enhanced the chances of compatibility among the different manufacturers of the needed equipment and software.

A piece of equipment had to have documentation that it was compatible with the other components in order for that piece of equipment to become part of our new SCADA system.

We also wanted equipment that was widely used throughout industry. This ensured that replacement components would be readily available, and in the event of personnel turnover, it simplified the process of finding qualified people to maintain the system.

FIRST CHOOSE THE CONTROLLER

The old system used relay logic for control and had a stand-alone proprietary RTU for data communication. It made sense to eliminate the relay logic and the proprietary RTUs and combine both their functions into one piece of equipment, so we did.

Using programmable logic controllers (PLCs), we eliminated hard-wired relays that previously required an electrician to physically rewire the control panel whenever the system needed modification. Replacing relay logic with PLCs enables us to make changes in the control scheme by programming rather than hiring an electrician.

The ease of making programming changes encourages continuous improvement by reducing the amount of time between the birth of an idea and its implementation. When operators see the ease and speed with which changes are made, they are much more likely to adopt the continuous improvement approach and suggest operational enhancements.

The PLC also replaced the proprietary RTU. Now, both the control functions of the relay logic and the data gathering function of the RTU happen using only one piece of equipment: the PLC. This allows us to make changes to the control scheme and data acquisition using a single piece of software.

The Allen-Bradley SLC 500 was the PLC of choice because it is off the shelf, is used worldwide, and has a proven track record in industry. The PLC became the backbone of the SCADA system and was the first piece of equipment we selected for the new design.

We chose all other equipment and software, communication gear, and man-machine interface (MMI) software based on compatibility with this PLC.

The old SCADA used dedicated leased phone lines to communicate with the remote RTUs at a cost of over $20,000 a year. The desire to reduce or eliminate this cost was a driving factor in our seeking alternative methods of communication, but it was not the only one.

With communications being the backbone to any SCADA system, leasing the phone lines left the district dependent on an outside source. If the phone lines went down, as they often did, the district was at the mercy of the phone company to restore communication.

Every time a phone line went down, the district had to summon an on-call person. The majority of SCADA downtime and operator overtime was due to bad phone lines. We opted to convert to a self-owned and self-operated radio communication system.

Our first priority in selecting a radio was that it easily interface with the PLC. After rejecting an integrator's quote of $10,000 to study and recommend a radio, the district decided to handle the radio design in-house, too.

Considering only radios from manufacturers that had a proven alliance and compatibility with the PLC manufacturer narrowed the search somewhat. The Data-Linc SRM 6000 became the radio of choice for several reasons. It was plug and play, spread spectrum, and frequency hopping, and it transmitted data at a high rate of speed.

JUST DO IT

The company also agreed to a 90-day trial pilot project of four radios with the understanding that we would return the radios if they did not work but purchase at least 30 additional radios if they were successful.

We planted the four radios at the most remote sites. If the radios worked there, they'd work at any of the other planned sites. The radio pilot test was a success and cost about the same amount of money the radio integrator wanted to do a study.

The entire system is now communicating flawlessly via radio. If a communication problem were to arise, the operators would have the knowledge and resources to quickly correct it without waiting on outside assistance. We exercised another opportunity to take ownership.

The new MMI software was to replace proprietary SCADA software developed and maintained by the outside vendor. With the old system, changes were costly and did not promote a continuous improvement environment.

Because of the time and money it took to make changes, operators often learned to live with and accept modes of operations instead of implementing new ideas. Again, the goal was to purchase an MMI package that was compatible with the other equipment.

We decided to go with Winview MMI software for several reasons. The PLC manufacturer developed it, ensuring compatibility and eliminating finger-pointing if a compatibility issue were ever to arise. It was also a package that could be developed in-house and that would grow as needs grew.

Purchasing and developing the MMI software in-house ensured that it developed to meet the needs of the district and that we could readily modify it for future improvements. We no longer needed to go to a third-party integrator to implement new ideas.

The approach to executing the new SCADA system was to expand one site at a time. The old and the new SCADA systems would be running in parallel. For each location, we would develop an MMI screen, establish communication via radio, and install the PLC at that particular location before the conversion from the old to the new system started.

As we added each location to the new system, we removed it from the old system. We used this method of conversion until all the sites joined the new system and the old system closed. This assured coverage at all times during the conversion process.

The order of conversion for the sites was from least to most critical. This allowed the district to gain experience and efficiency with the installation process by the time more critical sites were converted.

By building the system in-house as a team, the staff is now the expert on the new SCADA system. Ownership enables the staff's efforts to adhere to the continuous improvement work ethic. It also enables the team to make operational changes to the system without costly outsourcing.

The district no longer depends on an outside source. IT

Behind the byline

Tony Stafford is the superintendent of operations for the Camrosa Water District in Camarillo, Calif.

Related Files