01 December 2004

Peer Pressure

Production performance ratings measure effectiveness of batches.

By David Emerson

Key Performance Indicators (KPIs) serve the batch processing industries as measurements of production performance. Their use is one element in the current trend of real-time performance management.

A single KPI used as the primary measurement of production can cause other dimensions of production performance to lose importance. When multiple KPIs serve to measure a batch's production performance it can be difficult to reconcile differences between them for individual batches or for groups of batches.

KPIs based upon meeting a target, or specification, measure absolute performance yet do not provide relative information regarding how a batch performed against its peers. The peer comparisons are important for monitoring variability of production performance, which is a critical factor in documenting return on investment (ROI).

One method to compare batches with several KPIs is to calculate a composite KPI providing a single top-level metric for easier identification of problem and exemplary batches. This concept can extend to include KPIs that measure variability between peer batches and to normalize the value from 0–100% to provide a consistent, easily used value. This KPI is the production performance rating that reflects the production performance for a batch based upon its peers.

Parameters

Many different metrics or KPIs can measure the effectiveness of batch production, including the following:

• Batch cycle times
• Actual results versus planned/recipe expectations
• Capacity utilization
• Batch exceptions
• Attainment to schedule
• Attainment to standard

Some of these items are very general categories, so note that an item like "Attainment to standard" can include typical golden batch criteria such as critical process measurement trends meeting a "gold" standard, laboratory results meeting targets for a recipe or product, material additions meeting recipe targets, and labor costs for a batch meeting expectations.

Of course, the ability to repeatedly produce on-spec and saleable product is absolutely critical; however, once this is accomplished there remain many opportunities to improve production. The key benefits of focusing on KPIs beyond this basic level are lowering costs and increasing capacity without requiring large capital expenditures.

As real-time performance management receives increasing attention, we see batch production KPIs exposed to higher levels of management in real time on dashboards and Web pages and in e-mails. One difficult aspect of using multiple KPIs is that each one usually does not have an equal importance with regard to overall production efficiency and costs. Compounding this, individual KPIs can give conflicting indications of performance and can make it difficult to judge overall performance.

Production performance ratings for batches can roll up to master recipe versions, master recipes, and products to enable comparisons regarding their production performance. While one normalized value cannot express all the subtleties of a batch's production, it can act as a quick measurement of production performance and serve as a filter for finding top- and bottom-performing batches and products.

End result

A production performance rating is a technique used to rate batch production performance on a 0–100% scale. The rating's inputs are individual KPIs based upon meeting targets and specifications as well as KPIs that measure production performance against a batch's peers, the other batches based upon the same master recipe version. This combination assumes a weighted value to meet the needs of each application and yields an overall percentage that reveals a quick identification of poor-performing and top-performing batches.

The criteria and calculation for the rating will vary per application, but using the ISA-88 standard one can develop a set of application-independent KPIs that will provide a good starting point for most batch processing facilities. One can refine this set of KPIs with application-specific criteria and can develop customized calculations to provide more meaningful results. Some of the standard ISA-88-derived KPIs are listed below:

• Cycle time
• Number of times a batch was held
• Percent of time a batch was in hold
• Number of events associated with a batch

By themselves these measurements have no context so they must compare against a target or against their peers.

Target- and peer-based KPI inputs

In some cases comparison to a target makes sense; for example, lab results may be required to meet a specific level or process measurements may need to meet specific criteria before the release of a product. These KPIs provide a base level for defining an acceptable batch of product. However, they do not provide adequate insight into the batch's production performance.

For example, a set of batches may meet the lab results and process measurement requirements to be released, but production of some of the batches may have entailed much shorter cycle times and less operator involvement than other batches. Those batches produced with shorter cycle times and less operator involvement are higher-performing batches since they met the release requirements at a lower cost (less asset utilization and less labor cost).

Pure target-based KPIs would not differentiate between the higher- and lower-cost batches. Peer-based KPIs can differentiate between these batches.

Used together, target- and peer-based KPIs can be inputs to the production performance rating calculation, as seen here. The inputs should have a weight that adequately reflects each application's needs. For example, when meeting release targets is critical, weighting target-based KPIs can force a very low rating, even 0, when any release criterion is not met. When the release criteria are met, one can use the peer-based KPIs to rate the batch's production performance against other batches of the same master recipe, or even master recipe version. This provides finer resolution that will enable identification of good and poor production characteristics.

Production performance ratings are values normalized from 0–100% in order to provide an easy-to-use scale. This normalization can require tuning the weighting factors and calculation to achieve meaningful results. Once tuned, the normalized scale provides for quick identification of top-performing batches—those above 95%, 90%, or 80%—as appropriate for the application.

The normalized rating value can lead to easier identification of golden, or top-performing, batches. Traditionally many batch-operating companies have used an individual batch as the gold standard. Some have identified one golden batch for each unit recipe and built up a composite golden batch by assembling unit recipes from different batches. Production performance ratings enable the rapid identification of the top-performing batches and, if carried down to the unit recipe level, the top-performing unit recipes. This permits process and automation engineers to move beyond the single golden batch concept to look at what production characteristics the top-performing batches have in common—was it the equipment used, time of day, personnel, or ingredients?

Perhaps more important than examining the golden, or top-performing, batches is the need to examine the bottom-performing ones, or the "brown" batches. Not all of these brown batches necessarily made off-spec product. The batches may have been of acceptable quality, but perhaps needed some rework resulting in longer cycle times and more operator involvement. Understanding the characteristics that lead to brown batches may enable corrective actions that will yield less production variability and lower costs faster than chasing the golden batches.

To facilitate identifying golden and brown batches, the normalized rating value can be a filter criterion in a batch historian when selecting batches for analysis and reports.

Rating calculations vary

Calculations used to determine a production performance rating can vary in complexity. A straightforward calculation, as shown in the graphic above right, may calculate a value for each input KPI based upon a comparison to a target value or the mean and standard deviation of its peers. Weighting this value then gives individual KPI inputs different influences in the final rating; the calculation uses a default percentage as a starting point and the result is normalized to 0–100%. This method can easily expand to include new KPIs.

One may use more complex formulas but the key principles are these:

1. Compare input KPIs to a target or its peers.
2. Weight each input according to its importance to production performance.
3. Normalize the result, clamping it to 0–100%.

When a KPI compares to its peers, a simple rule of thumb serves as a starting point. This rule, as illustrated below right, draws upon concepts from Six Sigma in that low variability is desirable. When a batch's KPI—such as its cycle time—is within the upper and lower standard deviation of all its peer batches, this is good and should increase the rating. When the cycle time is outside the standard deviation limits, this is poor behavior and should decrease the rating. In order to reward those batches with improvements, those with cycle times below the mean and above the lower standard deviation limit should have a slightly higher rating.

This rule of thumb rewards batches that provide consistent cycle times, with a slightly higher reward for those trending below the mean. When a cycle time is below the lower standard deviation limit, this should be considered "too good to be true." Perhaps it is due to a breakthrough in production performance, but this decision should be held until it is repeated enough for the mean and standard deviation limits to change sufficiently for these batches to be in the green zone.

The number of standard deviations to use should be flexible and depend on the industry, company, and process. In most cases three standard deviations is a good starting point, but in processes with low variability this may not provide sufficient differentiation so a lower number may be desirable.

KPI aggregation formula

Identifying peer batches for each application is part of the process. In the strictest sense all batches based upon the same master recipe version are peer batches. If the master recipe changes, the batches based on the new version should group separately from previous batches. In other applications, batches from multiple master recipes may be considered peers if the recipes are similar enough.

As more peer batches churn out, the mean and standard deviation values will drift as new data points appear. At some point the production performance rating for all peer batches will need revision so as to provide a level comparison. Whether this should happen each time a batch finishes, periodically, or upon demand is an application-specific decision.

Depending upon the process and products involved it may be necessary to create different production performance rating formulas for each master recipe or group of master recipes in order to customize them for differences in the processes or products.

Batch versus unit ratings

Discussion so far has referred to batch level production performance ratings. The batch level rating is the most visible, but often it's the unit recipe that can provide a more accurate reflection of key performance areas.

One weakness with batch metrics such as cycle time is that one key unit recipe, say the reaction unit recipe, can represent the true bottleneck or high cost/risk portion of a batch. Other unit recipes such as those for preparation, mixing, post-reaction processing, and drying can provide variability that is actually caused by another batch's reaction, or other key unit recipe. When this occurs, unit recipe production performance ratings should be calculated and used as the primary method to compare batches. Alternatively, instead of using the batch cycle time as a KPI input to the batch production performance rating, a key unit recipe's cycle time can work.

While this can carry on down to the operation and phase level, there may be diminishing returns in these cases. If they are calculated, then comparisons for very specific periods of a batch's execution are possible.

Peer comparisons using standard deviation

Ratings roll up for levels

A powerful use of production performance ratings is the ability to roll up results. Roll-ups can apply to many levels, such as the following:

Roll-up	Benefit
Batches based upon the same master recipe version	Compare if master recipe modifications are increasing or decreasing the production performance rating. Compare production performance ratings for different master recipes and products.
Batches based upon the same master recipe, regardless of version	Compare the latest master recipe version's production performance rating against the average of all versions for that master recipe.
Batches based upon a group of master recipes	Compare ratings for products thathave more than one master recipe or for families of products.

Rolling up ratings can provide a quick indication of trends. The graphic below shows the production performance rating for batches rolled up by master recipe version and master recipe.

Production performance ratings rolled up by master recipe version

This roll-up enables the quick comparison of the rating for each version of Recipe A or B, thereby helping indicate a trend. When rolled up to the recipe level, comparisons between Recipes A and B can happen, assuming the same formula was used for both recipes.

Peer comparisons of batch's production performance ratings

Trending over time dashboard

Production performance ratings work as metrics for dashboards but can also provide a powerful indexing tool for production analysis.

Using a batch and unit recipe's ratings as a filter it is easy to find top- and bottom-performing batches for a number of criteria such as production lines or units, time of day, products, and material lots. Once used to create sets of batches, the batches yield information that helps to gain a better understanding of what causes production problems and higher costs.

When using a batch historian, one can analyze historical batch data for trends over time, not just on a recipe or product basis but for detecting other correlations such as if certain operations, phase classes, or units are commonly associated with high or low ratings.

A process engineer may find all batches of a product in the last quarter that had performance ratings below 50%, then drill down to find the root causes of the lower performance ratings. Or he may find the averages and standard deviation for performance ratings for all batches of a product, and then compare different products to see which one has the greatest variability.

In this case peer comparisons are possible using the batches' production performance ratings.

Take this away: Production performance ratings provide a composite KPI that can identify top- and bottom-performing batches.

Ratings should not serve to reflect product quality levels; instead, focusing on production performance of on-spec batches can lead to the root causes of production problems and detection of characteristics that lead to top-performing batches.

Ratings can also help manufacturers expand the concept of golden batches from one exemplary batch or unit recipe to a set of excellent, top-performing batches from which one can glean common traits and characteristics.

The golden batch concept can also expand to include brown batches, the low-performing batches that one can spot—through analysis—and fix, thus preventing future glitches.

Behind the byline

David Emerson (dave.emerson@us.yokogawa.com) is a senior systems architect at Yokogawa in Carrollton, Texas. He presented a paper on production performance ratings at the World Batch Forum North American Conference in May 2004.