1 April 2006

Treat Your Water Right

Deep tunnel sewage plant gets supervisory control with HMI software

By Ellen Fussell Policastro

Singapore's Changi treatment plant and deep tunnel sewage system is a prime example of how people with the right information can make informed decisions and are less likely to make errors. With a sophisticated treatment process (800,000 meters or 211 million gallons per day) and one of the largest integrated SCADA systems in the world, the plant was a perfect candidate for a new human machine interface (HMI) software package that provided supervisory control over processes in the tunnel, the treatment plant, and pumping stations. Jim Black, technology discipline director of C2HM-Hill, a Denver consulting and design firm, began work to solve their water treatment woes.

 

FAST FORWARD

  • Real-time HMI software empowers water sewage personnel to make fast, accurate decisions.
  • Pharmaceutical plant tracks production, meets standard compliance, and reduces human error with HMI software.
  • Human negligence isn't always to blame when automation goes awry; effective training is the key.
 

Singapore treats sewage to create new water on a small island off Malaysia. When officials decided to recycle a large percentage of its water, Black and his team started work on an integrated approach to supervisory control. They based their approach around reusable object models that integrate logic. The control system required a large investment in PLC and HMI software development. A key objective was to develop software in a structured way, based on industry standards, that multiple contractors could use to maintain a consistent look and feel across the facility, Black said. The team was responsible for control system configurations that included 50 PLCs, 50 nodes of HMI (PLC, SCADA server, and clients), and a large system with more than 3,000 graphics.

The most common standards PLC suppliers support for PLC programming in today's marketplace are the IEC 61131 and 61499 standards, he said. The IEC 61499 standard defines a generic architecture and presents guidelines for using user-developed function blocks in distributed industrial process measurement and control systems. User-developed, user-defined function blocks (UDFBs) are an established concept for defining robust, reusable software components. A UDFB can provide a software solution for pumps, valves, and control sequence or control a major unit industrial control system. The UDFB provides logic functions in object models, which, when used with this technology, allow encapsulation of industrial algorithms in a form plant staff, not software specialists, can understand and apply, he said. That means significant cost reduction to deliver and maintain a highly sophisticated control system.

In the delivery approach, Black's team and client staff used a life cycle information process to manage the vast amounts of information needed to optimize the operation and maintenance functions and to configure the control system. They used the design information generated in a 3D design model and migrated it through databases that allow the system to pass information to the contractor and migrate it back electronically, then directly load into the information systems. The data migration process aims to reduce cost and increase accuracy of that information. The data drive approach also helped configure the control system object models. "We're using and developing technologies to manage millions of documents we need to operate and maintain such a large facility," Black said. "The cost savings in this process is significant."

The system is also large in terms of I/O capacity. Instead of using conventional I/O, they use Profibus and FTD/DTM technology to access information directly over a network. "Operations and maintenance staff can now look at the diagnostics of any instrument over a network," Black said. "The FDT/DTM technology allows us to go directly from any place in the network, look at, download, and test configurations, and troubleshoot all the instruments," he said.

Quick decisions

"If I'm going to respond to an alarm, and I have something that tells me how to do it, I won't put my process in harm's way," Black said. "I need to make informed decisions" to respond quickly and effectively. The system is particularly useful for personnel if there's a pump or a piece of equipment that goes into the computer maintenance management system, and that design information doesn't include the manufacturer's design. In such a case, contractors can supply that information electronically into spreadsheets.

The HMI software package provides the visualization and operator access to the process and its controls. A key design objective was to integrate the HMI function with an information system that provided the operations and maintenance staff the information they needed to make informed operational decisions. Black's team provided integration of that information system with the software package. "If you click on a pump on a screen, you get an information menu," he said. That menu provides information, such as standard operating procedures, training, loop descriptions, and alarms, related to the clicked devices or system.

This level of integration addresses some industry statistics that indicate 30 to 40% of an operator's time is lost looking for information to do his job, Black said. The new HMI software enables users to make decisions based on real-time information. When an alarm process comes in, click on that alarm, and it tells you what to do. "Before, when an alarm came in, if you'd never seen it before, you would have to go through a book to look it up and get the suggested action," Black said. "Or you would just have to know how to respond." To achieve this level of integration, the HMI software package provides an open architecture and the ability to develop visualization objects that are easy to use, delivering an integrated control system with a consistent look and feel, independent of the supply source.

Pharma tracks production

Better consistency of a process isn't the only thing HMI is good for. Another aspect is ease of use when forced to comply with government regulations.

Sanofi-Aventis, the third largest pharmaceutical provider in the world, faces challenges to track production status and cost-effectively meet regulatory compliance in countries where it manufactures. Each year, its Ambarès plant in Europe produces nearly 5.2 billion tablets, 600 million capsules, and 30 million ampoules of injectable solution. In anticipation of FDA requirements, the company implemented a control and visualization solution to help comply with FDA 21 CFR part 11 and improve tracking processes. After a year, they've seen remarkable results.

The plant's injectable liquid manufacturing process involves stirring a mixture and pouring it manually into tanks according to recipe specifications, then storing the final solution in ampoules to preserve the product before shipping. Tanks undergo cleaning and preparation for the next batch after each operation. "The first and most important challenge was to improve the sanitary security of the complete system," said Serge Landreau, the plant's project manager.

A disconnected and manually intensive process led managers to seek a method to reduce human error and increase efficiencies. They investigated ways to improve aseptic processes, focusing on clean-in-process and sterilize-in-place processes. They also implemented new water-for-injection (WFI) loops. WFI purifies water by distillation, rendering it sterile to prevent microbiological contamination.

Standard compliance

The company needed to comply with U.S. and European pharmaceutical regulations for the next five years. Specifically, they needed to track and control the WFI process according to 21 CFR part 11 specifications, especially with its aseptic process. After evaluating control system platforms to meet compliance demands, they decided a DCS would best meet their needs. They installed an integrated architecture with controllers at its core, incorporating a supervisory level HMI software and terminals. This would allow redundancy and access control of specific functions via the electronic signature feature in a Windows domain control environment.

With an integrated control solution, "engineers know all pipes and tanks are sterilized before a new process starts," Landreau said. "Because the process is completely automated, they can also control that the right raw materials go into the right pipes and tanks at the right time. The system also provides proof that the process worked as required for regulations, available at any time."

The beauty of the new system in the compliance realm, Landreau said, is "it follows the company's three-year plan to meet compliance regulations. At the same time, we used the ISA-88 standards," along with the new system, which Landreau believes will help with long-term regulation compliance in the U.S. and in Europe. "As product demand increases, the solution will scale easily and rapidly as needed," he said.

Landreau was also struck by the "capability of the supervisory system to maintain process monitoring during critical phases and quality data tracking, even in the case of failure of the main HMI server." If the time for each process doesn't last long, there is a specific critical time where they cannot stop. In the event of main HMI server failure, the operator can still monitor the process without any disruption. It's also crucial for the company to trace data for regulation and quality purposes, which is why they set up the data server in a redundant mode to prevent data loss. The overall system is secure for monitoring, regulation, and quality issues. The switchover in this application is less than one second.

Electronic signature discrimination

The electronic signature will discriminate against some users for specific actions, such as modifying the setup of key parameters for quality, cleaning, and sterilization operations. Linked to the company domain controller, individual usernames and passwords come from the IT department.

With the previous system, all 20 employees had access to the manufacturing process via the HMI system in place. "Now, with the new system, only four employees have access to both the cleaning and sterilization functions via the electronic signature," Landreau said.

Ignorance, Not Negligence, to Blame in Human Error

By Trevor Kletz

People have opportunities to prevent every accident. But sometimes negligence on the operator's part isn't always the crux of the problem. However, supervisors and designers neglecting an operator's understanding of reasons behind his required duties could prove disastrous.

In one instance, a small tank refilled every evening from a larger stock tank with enough raw material for the following day. An operator watched the level in the tank and closed the inlet valve when the tank was 90% full. This system worked great for five years until the operator allowed his attention to wander one day. The tank overfilled and spillage occurred. Managers decided to fit a high lever trip (sometimes called an interlock), so if the level reached 90%, the inlet valve would close automatically.

The designer of the trip realized it would not be 100% reliable. But he realized the chance of the trip and the operator failing on the same day was so small he could ignore it. The operator didn't understand the design basis for the regularly scheduled testing of the trip. He continued to work and let the automatic equipment supervise the tank level. Worse, the foreman and his supervisor knew the operator wasn't watching, but they saw no need for it either.

After two years the tank overfilled again. The automatic equipment was standard off-the-shelf quality, and a failure about every two years was normal. The weekly testing was a waste of time as a demand was almost certain to follow any failure. They were using the trip as a controller rather than emergency equipment. Ergo, human errors abounded. Some companies would have blamed the operator for a violation since he failed to follow his written instructions. Yet his foreman and supervisor knew he was no longer watching the level in the tank and said nothing.

The designer was at fault as well. He should have known not to expect an operator to continuously watch a level that was under automatic control.

About the Author

Trevor Kletz is a widely recognized author on safety in the process industries.

HMI: From mental models to automation

We all have mental models of the world around us, and incorporating those models into the technology humans use in automation processes is the concept that lies behind the design and use of human machine interface (HMI) systems. HMI comprises all the elements a user will touch, see, or otherwise use to carry out tasks, said Jean-Ives Fiset, president of Systèmes Humains-Machines, Inc., a human factors engineering support firm in Laval, Quebec, Canada.

Such mental models have a number of characteristics others need to understand to avoid costly mistakes, especially in HMI design. "Imagine that you have to explain to a kid how a car can go faster while driving," Fiset said. One easy explanation is to tell them to depress the accelerator. But when the child asks more questions about why this leads to higher speed, then you'll need to explain how the accelerator changes the amount of fuel reaching the combustion chambers into the engine.

Sometimes the operator interface is an afterthought, said Nick Sands, process control technology manager at DuPont Chemical Solutions in Newark, Del. But a "well-designed interface, in conjunction with a well trained operator, can have a significant impact on plant operations, improving quality and uptime while reducing upsets and cost," he said.

The most important thing to remember is to "get lots of operator input, if not an operator, to design the graphics," Sands said. "Keep it simple; use animation and 3D effects only when you can spare the pixels. Keep it consistent; use the same shapes and colors throughout the interface. Keep it clean; do not overload the graphics. Follow the rule of seven; use less than seven colors in any display, including the background color. Keep it organized. Finally, keep alarm management in mind from the beginning; it is difficult to add it later."

REFERENCES

ISA-RP60.3-1985 Human Engineering for Control Centers (Standard)
www.isa.org/link/humancontrol

Understanding and Applying 21 CFR Part 11: The Electronic Records; Electronic Signatures Regulation (IC41PC) (Training Course)
www.isa.org/link/IC41PC

Human Factors Methods for Design: Making Systems Human-Centered, by Christopher P. Nemeth
www.isa.org/link/factordesign