1 April 2005
Innovative integration best medicine
Navigating paths for effective plant-wide standard compliance.
By Jack E. Greene
There's an approach to integrated process control that challenges engineers and project managers but also gives the plant more flexibility. Plants built on this model will require fewer personnel to maintain them. This model also successfully meets all 21CFR11 requirements regarding electronic records.
The approach uses terminal server-based user interface, network-based authentication, centralized setpoint and recipe management via databases, and centralized Web-based reporting and data mining. This results in a standard mechanism for capturing all critical operator actions with a 21CFR11 compliant audit trail. When coupled with enhanced procedural controls, these solutions can provide a mechanism for 21CFR11 compliance that lowers downstream support costs.
Most 21CFR11 compliance schemas are equipment specific and result in plants with a series of isolated mechanisms for 21CFR11 compliance. This non-integrated approach is difficult to design, implement, and validate and is cumbersome to maintain and defend to regulatory authorities. The proposed scheme presents mechanisms to capture and filter all operator actions, such as setpoint and recipe changes, manual overrides, alarm acknowledgement, and starts/stops/aborts with full audit-trail support that includes date, time, and operator ID. Core portions of this scheme capitalize on the use of server, application, and network redundancy as well as terminal server-based, operator interface, and Web-based reporting tools. Other benefits include lower support and change management costs.
Applicability to process control
The audit trail requirement of 21CFR11 says you need to capture date, time, and operator ID for all electronic records. While the source code is an electronic record you must maintain, the running control system doesn't produce electronic records, and therefore meets the electronic records requirements of 21CFR11. The compliance issue is industry uses supervisory control and data acquisition (SCADA) systems to allow operators to alter the state of the systems via the various forms of human machine interfaces (HMIs).
SCADA systems often act as a repository for instrument data, recipes, and setpoints. The issue is not how to construct a 21CFR11-compliant PLC; the issue is how to construct a 21CFR11-compliant SCADA. An important distinction about 21CFR11 compliance is it is not an off-the-shelf product you can purchase. It is a methodology each company needs to define, design, implement, and validate according to their own definitions. SCADA systems need to capture audit trails for all critical operator actions. Some actions, such as cycle start/stop/abort, alarm acknowledgements, setpoint changes, recipe selection, recipe edits, PID loop tuning, and manual overrides, affect the state of the system. These are classified as electronic records, and they require audit trails. Other actions, such as screen navigation, report generation, and data viewing, do not affect the state of the system. They are not classified as electronic records, and they do not require audit trails. The important difference between the two types of actions is, in most cases, the critical process actions are usually associated with writes to memory addresses in process control systems.
Process control system with panel based HMI and SCADA |
Traditional process control models
The most traditional process control model involves a series of PLC/DCS systems that are stand-alone. In cases where you need to move data critical to process control among them, this information travels via individual analog and digital I/O. HMI hardwiring occurs via dedicated embedded panels (not computers) to the control systems. SCADA systems may communicate with some or all control systems via a highway type communication pathway (Ethernet, Control Net, DH-485). Reporting from the SCADA happens through dedicated reporting stations. (See figure at left.)
A core problem with this scheme is almost no embedded panel-based HMI support two-part logins (operator ID and a password). Some types of panels require no login to access the process. In other cases, you do need a login, but it is a short numeric string. For some panels, when an operator logs in, the panel displays the password on the screen as the operator types it. With these types of systems, as operators affect changes to the process, there is no straightforward mechanism for transmitting operator ID from the panel into a SCADA-based audit trail. For these reasons, embedded panels are not acceptable for applications where 21CFR11 compliance is required. This scheme also has support and maintenance issues. These panels frequently are a source of a single point of failure. If the panel fails, operators can no longer control the process. From an IT management standpoint, panels are costly to support because each panel in a plant has its own access control list. If there are 10 panels in a plant, there are 10 lists to maintain.
A slight variance on the traditional model involves PC-based HMI (industrial computers for hazardous or clean-room environments). As in the embedded panels, you usually hardwire these into the control systems. SCADA systems may communicate with some or all control systems via a highway type communication pathway (Ethernet, Control Net, DH-485). Reporting from the SCADA occurs through dedicated reporting stations. (See figure at right.)
The Windows-based model can solve many regulatory compliance issues. It supports a two-part login that in many cases you can tie in with operating system authentication. An additional feature is quite a few PC-based HMI platforms support the export of operator activity logs directly into databases. This simplifies the creation of the audit trail. You can use this architectural model to meet all existing 21CFR11 requirements.
The main detriment to this type of scheme is it has a high ownership cost. If you use more than one HMI platform brand within a plant, you'll need to develop different database schemas to collect, filter, and report the audit trail. Another issue is reliability. At the heart of the stand-alone, PC-based HMI is an operating system for home or office use.
Windows-based HMI and SCADA process control system |
Web-based reporting
This model implements the HMI software on a terminal server. The terminal server houses a centralized code source you can view at the thin client HMI stations on the plant floor. As in the previous model, you'll want to design the servers with redundancy to mitigate the risk of network or operating system failure. The operator event logs export into the facility database. Reports generate through a Web report engine that points to the database. One reporting server may serve multiple plants, and it allows reports to run from operators' desktop PCs.
Terminal server based HMI works by internally generating the control panel screens and painting them onto the thin clients. Since all screens in a plant draw from a single code source, the code deploys and validates only once. Since any screen can appear on any client, this enhances the plant's flexibility. While there are scalability issues in that the number of clients served is dependent on the computing power of the terminal server, modern servers are extremely powerful and scaleable. If you use Ethernet networks extensively, you can implement full network redundancy from all servers to all process control cabinets and to all HMI for extremely low costs.
This combination of server, panel, and network redundancy mitigates the inherent fragility of operating systems. Since developers have engineered redundancy into the system design and tested it as part of validation, you can dramatically reduce the paperwork required when a network, server, or HMI component fails. Also, the redundant design allows for routine maintenance on these components without the need to schedule plant-wide shutdown.
Since there are no codes or applications on the thin clients, process control is no longer tied to the local panel operating system. You can patch operating systems and install or modify ancillary applications as required. The only validation is to ensure the change does not affect the running of the terminal services client. This testing can be run offline and referenced across an enterprise. Any system that can run the terminal services client can become a control panel. This allows you to use low-cost Windows CE-based panels instead of costly industrial computers. You can use wireless PDA systems or tablet PCs to allow a single person to bring the HMI into the field when calibrating field devices. A remote station with the correctly configured terminal services client can allow for control systems software support personnel to remotely troubleshoot a plant (using appropriate network security and safety precautions). Also, any PC in the Enterprise can access the reporting Web server to run reports, which eliminates dedicated reporting stations and printers.
Integrated plant architectures
An integrated plant model with centralized HMI servers and databases supporting all plant control systems (including BAS/BMS and utility systems) has even more advantages. It's straightforward to construct database drive systems tables to automatically capture batch start/stop times to feed web driven reporting. You can store and manage recipes with the site database with excellent version control. This data mining ability can reduce investigation cycle time and facilitate periodic inter-batch trending. You can use data linkages to link CIP, SIP, and production with other activities, such as autoclave or vial washer runs, to pull together the full plant picture to higher level application, such as MRP or electronic batch record systems. When the SCADA is collecting all plant alarms from all control systems, use database alarm lookup tables to route alarms to cell phones and pagers via e-mail. This gives you flexibility to define alarm distribution schedule and e-mail group destination on an alarm-by-alarm basis.
With good network security design, you can make read-only HMI applications available to any desktop PC in the enterprise, including dialup or VPN users, without compromising plant safety. Together, the alarming and remote read-only HMI functions can allow on-call personnel to check the nature of an issue and quickly triage it to the correct personnel from any location.
Terminal server model challenges
Without good training on how the terminal server model operates, non-information technology groups will not understand these architectures can reduce the change control, validation, and deviation report burden as well as minimize downtime. For this type of implementation to be successful and realize its full potential, IT, engineering, regulatory, validation, and quality groups all need to work together.
Since terminal and Web-server technologies haven't traditionally applied to process control, some of the best engineering and integration firms are hesitant to move to these models. A related challenge is scope delineation. In a plant construction project using this model, groups need to work together at a much higher level. Under traditional models, systems had clear responsibilities with few extra-departmental dependencies. Under the new model, almost every system will have overlapping responsibilities with dependencies on a series of other departments.
This can make engineering and integration firms uneasy about scope. They might be unclear about where to draw the dotted line on the facility map. Standard equipment from OEM vendors will not be able to participate in this model without significant HMI customization, which can lead to significant vendor surcharges. Alternately, they could accomplish the work by an integration firm between factory acceptance testing (FAT) and site acceptance testing (SAT). To take this path, the OEM vendor must agree to execute the SAT on the modified code.
As this architecture relies heavily on networks to provide connectivity as well as system security, a good network design is essential. Multiple subnet architectures with PLC/DCS systems in one network, SCADA/HMI servers in another network, and terminal clients in multiple other networks are essential. Network filtering across network boundaries is a powerful security tool, but only if you properly implement, configure, and monitor it.
Behind the byline
Jack E. Greene is manager, Laboratory and Process Automation Systems at Alkermes, Inc. in Cambridge, Mass.
Cost effective remediation strategiesBy Derek HookIt is possible to achieve 21 CFR Part 11 compliance in a cost effective manner with minimum impact on the existing control system and with reduced revalidation effort. Undertaking remediation of existing process equipment and systems to become compliant with the Food and Drug Administration (FDA) regulation, 21 CFR Part 11, involves making decisions based on the results of a risk assessment. The application of 21 CFR Part 11 has undergone some re-evaluation lately, but the trend to provide secure data is inexorable. And sooner or later, companies will be adopting systems designed to meet the regulation. There's a technique to reducing the cost of revalidation of smaller monitoring and control systems, typically discrete controller or PLC based, as well as one for a broader approach for larger systems. It involves reusing existing equipment and achieving compliance with the regulation by applying a compliant interface to the system. Configure discrete controllers for the front panel keys that you want to disable and achieve operator interface through a compliant human machine interface (HMI). For PLC systems, leave the program intact and access the registers directly by replacing the existing HMI with a 21 CFR Part 11 compliant HMI. Both approaches reduce the cost of revalidation because the control and sequencing function remains untouched. Pharmaceutical and biotechnology companies are steadily moving toward electronic records as a means of collecting process and environmental data. The 21 CFR Part 11 regulation demands compliance to certain specifications when using electronic records as the primary source of data. The requirement is for secure, tamper-proof data, which you can also create in standard format suitable for import into a spreadsheet or database. Electronic signatures also have a number of specific requirements, such as a minimum number of characters, password expiry, unique name and password combinations, and the option for a second signature to permit access. Electronic signatures can also include fingerprint, iris, retinal, and palm recognition systems. Creating an audit trail of all data and events is also a fundamental part of this regulation. This is the requirement. So what are the implications? You should design new systems today with 21 CFR Part 11 in mind, as the trend is to use electronic records. This regulation doesn't apply to just new systems, however. The majority of existing control systems will also need to meet this regulation sooner or later. The costs to reach compliance of existing systems can be severe, not just in terms of replacing hardware and subsequent software programming, but also in the re-validation costs. These validation costs often dwarf the hardware costs involved. It is common to establish remediation teams from multiple disciplines within the organization. They all struggle with how to achieve 21 CFR Part 11 compliance and how much can be done in what timeframe with the budget allocated. So, what are the options? A variety of options involve hardware, software, programming, changing SOPs, and validation. The difference is in the complexity and cost of implementation. Replace control systemYou can replace the existing system with 21 CFR Part 11 compliant features. Obvious drawbacks are the cost of the hardware and software programming and the cost of validating the new system. This is good news for instrumentation suppliers, but it is not a popular option for most pharmaceutical and biotechnology companies. Add a computer supervisory control and data acquisition (SCADA) system to the existing control system. It has the 21 CFR Part 11 features and communicates to the control system. This solution offers features but with cost of the software and programming of the SCADA system. Add compliant, non-computer data loggerThe electronic data recorder logs process data and events. These new generation products can provide full compliance with the 21 CFR Part 11 regulations by offering an auditor options. It can provide a high quality operator interface. In association with secure data acquisition, the historian software offers an easy way to access data, typically via removal media or an Ethernet network. The remote view you get with complimentary software gives you a secure view of the data as if you were at the unit itself. This requires hardware cost, some simple configuration, changing the standard operating procedures (SOPs), and validation of the data logging system. This is easier because validation protocols are readily available. This is probably the most common method of remediation and provides a straightforward way to gather data, generate audit trails, and produce required reports. Add compliant, non-computer front endAll previous options don't address the control system interface itself in a cost-effective way. This fourth option does just that. Here's a common scenario: The control system may consist of a programmable logic controller (PLC) plus an operator interface of some kind, perhaps a simple two-line panel or a multi-line panel with some graphics. This system does not meet compliance requirements for electronic records or electronic signatures. This situation is repeated quite often throughout the plant, hence the dilemma of achieving remediation at a reasonable cost. The compliant front-end remediation strategy involves replacing the non-compliant operator interface with a compliant, non-computer operator interface. Unplug the existing panel, and replace it with the new compliant interface. Here are some advantages: 1. It doesn't affect the control strategy of the PLC, thereby simplifying the complexity of re-validation and the time and cost. You only revalidate the operator interface. 2. You have protected and logged all operator access to the equipment, including initiating and stopping a batch, making parameter changes, and making operator notes. 3. You have enhanced the operator interface with full color graphics and possibly a touch screen. Change the SOPs to accommodate the new operator interface. 4. You can collect data locally, in internal memory, with removable media, via the Ethernet network, providing a high level of integrity of the solution. You can encrypt data immediately upon acquisition. If the network fails, you won't lose data. 5. You have lower implementation cost for the high reward you achieved, resulting in savings and a wider implementation of 21 CFR Part 11 with the same budget. To connect to devices with different communication protocols, a range of protocol converters is available. The availability of validation protocols also eases the task of revalidation. Behind the bylineDerek Hook is pharmaceutical initiative leader at Eurotherm Inc. in Leesburg, Va. |
Return to Previous Page
Read questions answered by our experts or join the email list.
