July 2008

Automation IT

Independence Day

Basing equipment modules within controllers saves time and money

FAST FORWARD

  • Implementing ISA88 was not the issue, just needed to decide on which model.
  • Requirements for the process control system should support direct operator interaction and a batch application manager.
  • Qualified equipment modules and strategies should not require a re-qualification.
  • Automation strategy should support new units reusing as much existing work as possible.
 
By J. Gordon Roney and Andrew Blankenship

Nestled near the foothills of the Blue Ridge Mountains, close to the confluence of the North and Middle Oconee rivers, lies Athens, Ga.-based drug manufacturer Noramco, Inc.

Noramco manufactures active pharmaceutical ingredients, narcotics, and medical devices in three production facilities in Athens. The oldest of the production facilities, Building 1, produces active pharmaceutical ingredients.

Throughout its history, the production processes at Building 1 were a manual operation. But in the truest sense of automation and technology coming together, six years ago, to meet changing business needs and planned Building 1 expansions, Noramco began the process of installing a process control system.

The three main requirements for the implementation of the process control system were:

  • The operation of the production process, via the process control system, should support direct operator interaction and a batch application manager when installed.
  • Equipment modules and equipment module control strategies, which have already been qualified and implemented, should not require a re-qualification after they install the batch application manager.
  • The automation strategy should support the automation of new units reusing as much existing work as possible.

The decision to implement an ISA88 based control strategy for the Building 1 process control system was not in question. After all, Noramco had previous success in implementing an ISA88 based control system onsite. The overriding requirements of the process control system called for ISA88. The strictly batch nature of production in Building 1 required a true batch standard. But the burning question was which ISA88 model best addressed the requirements.

Requirement analysis found the user needed the following equipment module control options: Direct control by the operator, control via a higher level operation, and control via the batch application manager. In all of the equipment module control options listed, the user required the ability to select the needed equipment module, the appropriate equipment module control strategy, input parameters for the control strategy, and a means to monitor and control the equipment module once execution is under way. Since the batch application manager was not going to be implemented in the first phase of the process control system installation, the ability for the operator to control the equipment module independent of the batch application manager clearly became the driving force in determining the appropriate ISA88 model.

Software engineering

After all the analysis, the user selected a collapsed ISA88 model.

The collapsed model allowed equipment control independent of the batch application manager. This independence occurred by placing equipment phases, the right side of the collapsed ISA88 model, in the controllers of the process control system. The recipe procedure and phases, the left side of the collapsed ISA88 model, would undergo implementation in the batch application manager.

Design and implementation of standard equipment modules and subordinate control modules was much easier by making extensive use of object oriented programming. Object oriented programming uses "classes" and "objects" to develop computer based applications. Classes define the abstract idea, and objects are the real-world application of the class. It is often easier to think of a "class" as a model or template, in this case an equipment module template. In one case, a number of units may require an equipment module with the same basic functionality, i.e. agitator with speed control. In order to promote reuse, the user developed an equipment module template called "Agitator With Speed Control." This template would not have links to subordinate control modules. Each individual application based on the equipment module template would have the links to subordinate control modules.

The first major implementation activity was the introduction of control module templates. By defining, developing, and qualifying control module templates, the user would already know the links needed by equipment modules.

The next major implementation was the introduction of equipment module templates, which is a multiple step process. The first step was development of the equipment module tem¬plate functional specification. This document defines how the equipment module is to function and the phases for that equipment module. The second step was the implementation of the equipment module template. Once the user developed an equipment module template, the first instance was created followed by the qualification of the first instance and equipment module template.

When an application requires the use of an existing equipment module, an instance of the qualified equipment module template comes into play. Creating the tangible equipment module from the equipment module template is "instantiation."

Because an instance is an exact functional replica of an equipment module template that has already been qualified, it is not necessary to fully qualify each instance's functionality. Only the instance's unique attribute values must undergo qualification. In practice, this equates to the equipment module instance interaction to subordinate control modules and any parameters that can undergo modification like when creating set points. Once in service, implementation of changes simplifies since instances inherit the functionality of the equipment module template. A change undergoes implementation and testing once at the equipment module template, and the functionality propagates to each instance of the equipment module template.

Another important aspect of implementing equipment modules at the controller level relates to the safe, normal operation of equipment. The response of the equipment module to an initiating event or events that warrant the subordinate control modules to assume a pre-designated state can undergo implementation and testing prior to firing up the batch application manager. It is important to note this functionality should not nor does it serve the functionality of a safety instrumented system.

Most, if not all, equipment modules implemented to date have a dedicated phase called "safe" for this predetermined state. Initiation of this phase can be internal to the equipment module, via operator initiation of the "safe" phase, or initiated externally to the equipment module itself. Examples of these external events include shutdown of a unit, a power loss, and cascaded unit shutdowns, among others. When the "safe" phase needs to execute, the equipment module will execute the predetermined actions. By monitoring and responding to initiating events at the controller level directly, the response time can reduce compared to equipment modules hosted on a server based batch application manager.

Added duties

When implementing control systems and new control strategies, pharmaceutical manufacturers, like most other manufacturing companies, undertake the standard activities of engineering configuration, pre-implementation testing, and operational testing. Unlike other manufacturing companies that do not operate in an FDA regulated environment, pharmaceutical manufacturers must undertake additional activities when implementing control systems and new control strategies. These activities include developing documentation to describe what the control system/control strategies should do (user requirements), how it is to be done (functional specifications), the qualification/commissioning rationale (impact assessments, project qualification plans) and testing (software acceptance, installation and operational qualification tests). The extensive hours and expense associated with these tasks makes the reuse of equipment modules, support documentation, and testing appealing.

It should come as no surprise the largest investment of time, and therefore expense, comes in the development of the first instance of the equipment module. Once the first instance is out there and undergoes qualification, subsequent instances do not need to be functionally qualified. In contrast, if a manufacturer does not use equipment module templates, complete testing of functionality must take place because there is no qualified defined template.

The following are two case studies comparing the initial equipment module implementation to a subsequent implementation of the same type.

auto11
This is a snapshot of the equipment module phases for the Agitator with Speed Control equipement model class. Depending on the phase, the agitator may start/stop and the function of the speed controller would be set.

 

First case

The original implementation of an agitator with speed control equipment module occurred in 2004 and the second implementation in 2007. The equipment module consists of an agitator and an agitator speed controller. Four relatively simple control strategies are available.

The overall time savings for the second implementation was 86 hours or a 45% reduction in labor hours. The greatest percentage decrease occurred in the task related to requirement and specification development, which took 67% less time for the second implementation. One of the reasons for the large drop came from the copying and/or revision of existing documentation. In terms of hours saved, engineering configuration tasks experienced the greatest drop of 36 hours or 50%. In this case, the time savings is attributable to the reuse of the equipment module. The hours presented for both the original implementation and second implementation include the cost for implementing the underlying control modules.

Second case

Another case is the implementation of a tank temperature control equipment module. The original implementation occurred during 2004 and the second implementation one year later. The purpose of this equipment module is to control the equipment used for the heating/cooling and maintaining of temperature for tanks and reactors. The equipment module consists of a pump, multiple valves, a tank temperature controller, and a jacket temperature controller. Six control strategies are available. The equipment module phases sequence the starting/stopping of the pump, the opening/closing of valves, the configuration of temperature control loops, ramping of temperatures, and monitoring conditions.

The hours presented include the cost for implementing the underlying control modules. The task engineering configuration experienced a drop of 55%. Pre-implementation testing for the second implementation took 50% less time than the original implementation due to the reduced testing of complex equipment module logic. A large time savings occurred during the second implementation for both requirement/specification development and qualification documentation development.

Implementation of a batch application manager began mid-2005 and hit full stride during 2007. Since the batch application manager uses the same equipment module attributes to execute the equipment module as does an operator, no re-qualification of the underlying equipment modules has been required. During the qualification of recipes, verification the user has the correct parameters and the correct phase executes at the equipment module level occurs. The states of the control modules within the equipment module do not undergo verification as severely as they were in previous testing. This has resulted in less complex and faster qualification of recipes compared to solely recipe driven production processes on-site.

Lessons learned

Outside of the obvious time and cost savings gained via the reuse of equipment models and the operational benefits of having them based in a controller, additional lessons came about:

  • Most noticeable is equipment module templates should use a generic name. During the initial phase of the control system implementation, the first few equipment module templates picked up the name of the first instance. This has created difficulties when trouble shooting problems and during subsequent implementations when different individuals would perform the work.
  • Clarity in naming of equipment module instances. A descriptive name for the equipment module instance that leaves no doubt to its functionality is extremely important. Meeting with operations and soliciting their input is vital and acceptance of the proposed/agreed to name is crucial. Consistency in naming equipment module instances derived from the same equipment module template has proven to be a time saver.
  • Careful selection of equipment module parameters that can change and "fix" parameters. Experience shows less is best. Fixing parameters not directly related to a process, i.e. line purge times, simplifies and minimizes operator interactions.

By choosing a flexible batch application that allows equipment modules to execute independently of the batch application manager has quite a few advantages:

  • It is easier to deal with problems early on rather than during the production start up of a recipe. In practice, it has been useful to have operators manually execute equipment module logic, catch control logic obstacles before recipe implementation, and address them.
  • Having equipment module logic within the controller has allowed the flexibility to use equipment modules to recover from a batch application manager failure.
  • Recipe development simplification. If the user automated the production process to the equipment module level, then the development of a recipe comes about much easier. This is the result of the operating instruction written at the equipment module level.

Overall, the implementation of equipment modules independent of a batch application manager has been a success. Basing equipment modules within controllers has saved time and has led to more cost effective subsequent implementations, in addition to the operational advantages of a controller based application.

ABOUT THE AUTHORS

J. Gordon Roney is a control systems group leader at Athens, Ga.-based Noramco, Inc. His e-mail is groney@norus.jnj.com. Andrew Blankenship is an integration engineer at Knoxville, Tenn.-based Innovative Controls, Inc. His e-mail is awb@innovativecontrols.com. The report emanated from a paper written from the 2008 WBF North American Conference in Philadelphia.

RESOURCES