Print this article

Comment

1 May 2005

Fast-track Automation

By David Adler, Jim Cox, John Gamble, Terry Lenehan, Graham Smith, and Jim Wiesler

Insulin manufacturing upgrade plan kicks into gear.

Humulin and Humalog are Eli Lilly and Co.'s insulin products produced in a pharmaceutical plant in Indianapolis. The company knew the plant needed to move with the times, so the facility completed a retrofit with 50% of the production equipment and the attendant automation systems upgraded or replaced. The upgrade provided the facility with an integrated, state-of-the-art process control system solution.

There were plenty of significant challenges on this project, but the most important one was the project timeframe.

Construction started within 15 months of the project's inception. This was a very aggressive schedule requiring a fast-track project with a time scale half that of any comparable automation project previously executed at Lilly. The time crunch was even more severe at the tail end of the project. Production constraints meant the plant could only shutdown for half the normal time to allow the upgrade work to wrap up. What's more, because insulin is a life-saving medicine and millions of people depend on it, once the plant shut down, it was absolutely essential production restart at the correct time. This required an automation program office and a matrix organization that allowed tight management of day-to-day activities while remaining agile.

Project plans

An old adage says if you fail to plan, you plan to fail. The project team recognized that without careful planning and rigorous project management, there was little chance of completing the project in the tight time frame. Usually on a project, there are three main variables you can manipulate: scope, schedule, and cost. For this project, all three remained fixed, and the required duration for implementation was shorter than any previous project.

At first, it seemed the project was graph where 1) the cost was prohibitive and 2) it may not have been possible at all. The only way to be successful was to think of alternative approaches and have flawless project execution. In appreciation of the size of the project and the aggressiveness of the schedule, it was decided to create an "Automation Program Office" that would be responsible for master planning and establishing a project management and control framework. It had extremely tight oversight of schedule and costs, near real-time metrics for leading indicators, an agile management group, tightly tuned to project metrics, and a very highly experienced team that averaged over 20 years experience in automation projects. The program office consisted of a program owner, program manager, program controller, and several project managers. The program office came together within one month of project kickoff and experienced no turnover throughout the project.

To provide the project control and structure necessary for fast-track execution of a tightly managed project, the automation program office defined a project management structure for the work. The program divided into 16 projects with a standard organizational structure and standard reporting requirements. It was important the individual process control professionals had dual reporting to the site operational leadership for technical issues and to the project office for scope, schedule, and cost control.

In a control analogy, project management came with tight gain, derivative, and fast scan rate. All aspects of this project were under the microscope and adjusted on a daily basis. The project plan underwent consistent tweaks while everyone refrained from wholesale process upsets that would have allowed the system to go out of control. Not only did the leaders control the standard metrics such as milestones, costs, and resources, but also some intangibles such as team morale and stress level.

The resource forecasts in the project plan indicated if they wanted to finish the work on schedule, a large number of contractors would have to supplement Lilly staffers. Estimates indicated there would be a need for upwards of 100 automation professionals. A local system integrator provided the process control expertise and just-in-time staff to keep the automation project moving on-track.

The first order of business was to create a project plan to define the scope and forecast the schedule and cost. The initial assessments showed there was a high probability the automation work scheduled during the shutdown would not wrap up on time. If everything went according to plan, we could meet the schedule, but there was no latitude for unforeseen problems. The project team reviewed alternative strategies and decided the only way to ensure a smooth return to service was to take a completely different approach. The leadership team decoupled tasks to allow parallel effort, even when the parallel efforts themselves were not optimal. Two examples of this were the efforts at managing documents for computer system validation and off-site equipment qualification testing.

The team decided to build a completely separate test facility away from the main production facility. This off-site test facility allowed detailed testing and qualification of systems prior to installation. The infrastructure in the test facility was so similar to what Lilly would use in the production environment that there was a high degree of confidence there would be few problems in the final installation of systems in the production facility.

Process automation organization

Paper trail

All pharmaceutical plants in the U.S. are highly regulated. For automation systems, this generally means a company must develop the system in a fashion that permits "validation" to provide a high degree of certainty they will meet their requirements. System validation generates confidence the system will work as intended. However, it also generates huge amounts of paperwork. This project had 16 distinct individual automation systems. Each of these automation systems typically required 17 different types of computer system validation documents. The typical set of validation deliverables for each system was: validation plan, user requirements, functional requirements, system overview, system design, system configuration, source code review, test plan, standards, procedures, test cases/scripts, test protocols, test summary report, security plan, business continuity plan, disaster recovery plan, and validation report.

Each automation system required 50 to 300 test cases. So, an additional challenge for the automation project was the enormous volume of documents required to support the validation of the automation system. This meant the team had to redefine most of the validation processes and decouple them from the rest of the system integration efforts. The team implemented a document management system to manage in-process and approved documents. It quickly defined and dictated content and format of each document type, thus eliminating time-wasting rework and internal project conflict. Workflow processes defined every aspect of the document creation cycle. These efforts reduced the validation document cycle time by a factor of 10 over the course of this project.

Technology

The existing automation systems in the facility represented a selection from automation vendor catalogs, some of which are no longer in business. Quite a few of the systems underwent installation in the 1980s. The systems were essentially islands of automation without plant wide integration and limited data historian and reporting capability. To accelerate the needed technology development to achieve the desired system integration, even while assembling the program office, the leaders kicked off a separate technology evaluation and implementation effort. It discovered the strengths and weaknesses of the new process control system.

Emphasis on project tracking, risk management, and schedule maintenance paid off. It proved to be a good thing when we discovered issues after about six months that would have jeopardized the entire project had the team not had the time to develop workarounds. It became apparent there were significant delays in developing the central automation infrastructure. This was a very serious risk to the overall program because each of the individual projects depended on the infrastructure. It would not be possible to complete the design of any of the individual projects until the infrastructure design was complete. Closer examination uncovered some issues with the technology framework underpinning the infrastructure. A risk-based analysis showed we could develop a low risk architecture. This, in conjunction with software patches from the vendor, would provide the essential functionality required to meet the project's needs.

The process control system delivered:

• Fewer, larger systems
• Only one Programmable Logic Controller platform
• Only one operator interface platform
• Full domain-based security with individual accounts
• Central reporting linked to the Manufacturing Execution System
• Servers and process control computers physically outside of the production area
• Full alarm/event archiving
• Full continuous and batch history
• Plant wide integration and reporting

It is a constant challenge in pharmaceutical operations to maintain the sterility of manufacturing operations. The project team developed an innovative use of an old technology to help decrease impact on sterility. KVM (keyboard, video, and mouse) technology originally saw use in large corporate computer rooms to allow the computer room operators to interact with multiple PCs from one desk console. This same KVM technology could be used to position PCs outside of the sterile block while leaving the (suitably packaged) keyboard, monitor, and mouse inside. Previously, to support and maintain PCs inside the sterile block, a technician had to gown-up and enter the sterile area.

Insulin production

The renovated facility in Indianapolis began the start-up process on schedule. The team effort and hard work allowed process automation to circumvent many roadblocks and deliver a working system on schedule and below cost projections. The new technology provided a robust platform that enhanced reliability and security. Desktop reporting tools enabled continuous improvement of manufacturing operations. The standard architecture, designed for long-term supportability, provided a flexible pathway for future enhancements. These improvements helped the operation of the building, but more importantly, insulin manufacturing can continue to deliver insulin to diabetic patients throughout the coming years.

Behind the byline

Jim Wiesler is a site process automation team leader at Lilly. Terry Lenehan is a site process automation team leader at Lilly. David Adler is a senior engineering consultant in the central Process Control Engineering group at Lilly. Jim Cox is an associate engineering consultant in the central Process Control Engineering group at Lilly. Graham Smith is the president of Brillig Systems, a project management consulting company specializing in process automation. John Gamble is senior project manager for Emerson.