1 November 2003
Some systems are as old as dirt
Is there some way to eke out a few more years from the current investment?
By Nancy Glenn and David Braun
Imagine, if you can, a manufacturing environment with a well-ingrained legacy manufacturing execution system (MES).
Now add increased pressure to do prior-step verification and recipe verification and download, and an ever-increasing push for scrap reduction and process control.
Already familiar to some of you?
Our solution? Lacking the resources to replace our MES, we took the approach of augmenting.
In any manufacturing setting the decision to implement new technology can be a difficult one. In the current mentality of cost cutting, any new system will have to quickly justify itself financially and significantly contribute to the enterprise.
Faced with the pressures of manufacturing in an Internet-based electronic business environment, small- and medium-sized manufacturers are looking to add MESs to compete.
Recent advances in technology cause larger manufacturers, which may have implemented MESs as long as twenty years ago, to rethink the capabilities of their systems. The MES is once again a hot implementation in manufacturing information technology (IT).
In a new manufacturing facility, once deciding to use an MES it is merely a choice between competitors and finding the right software package to fit the facility.
But what about a facility that made the leap ten or fifteen years ago? The system installed at that time, although possibly leading-edge technology then, is most likely outdated and not serving all of the needs of the enterprise.
Is it time to replace the system, or is there some way to eke out a few more years from the current investment? Fifteen years ago having a daily report on yesterday's work in progress (WIP) was state of the art and assisted managers in decision making.
Today, knowing yesterday's WIP is about twenty-four hours too late.
With the move to lean manufacturing or with a just-in-time (JIT) pull from suppliers in place, managers often need a real-time view of their facility.
They need advanced scheduling and dispatch with a view of machine up- and downtime to track and move the manufacturing flow around bottlenecks. Often, they also need to provide visibility to their customers, or to the corporate boardroom.
They may also need to extend data collection out to suppliers for quality feedback or to subcontractors to track off-site manufacturing. Some manufacturing facilities have moved to a strong 24/7 schedule (twenty-four hours a day and seven days a week) and need systems that are fault tolerant and fail over cleanly.
Many legacy MESs did not provide the flexibility or stability demanded by today's enterprises. In these cases, where the technology demand exceeds the capability of the current system, the decision is whether to replace or to augment the MES.
Many add-on packages exist that provide some of the additional functionality, but tying them together can be a tricky and costly process.
ARCHITECTURE RIGHT DECISIONS
What do you need out of your MES?
Before you start any technology implementation, it is important to have a clear definition of your goals. What are your biggest trouble areas? What information do you need to run your enterprise, and in what format should it be? Where will your cost justifications come from? Do you need to reduce scrap? Do you need to increase throughput? Do you need to better manage recipes on equipment? Are you looking to track static routes or to dynamically assign product to routes according to changing bottlenecks?
Do you need to change the core functionality of the system?
If you already have a system in place, does it—at the core—do what you need? If not, building on to your existing system will only expand your problems, not solve them. If your legacy MES relies on statically defined routes to dispatch, and you want to do real-time dispatching, it does not matter if you add an equipment-monitoring function; the dispatch will still be static. If you have new business requirements for traceability and require that your systems all be transaction safe, but your legacy MES is a socket- and string-based technology, no amount of enhancement is going to change that core problem. There are, in fact, times to move on and invest in a new system.
How are you growing?
It is important to have a good sense of the expected areas of growth, both as a business and as an IT infrastructure. What you don't want to end up with is an MES with so many custom patches, add-ons, and custom fixes that it becomes unmanageable. Nor do you want to plan a custom module around an Oracle database, only to find out that IS will stop supporting Oracle at your site in six months. Be sure that your code will not have to be "fixed" in twelve months because of a planned business process change that was not communicated downstream.
How old is your legacy system?
Although adding code to your MES is less costly than replacing the entire system, it is not a small investment. Be sure you will be able to get about another five years of life out of your MES, before making an investment of this size. Consider both hardware and software issues. Look not only at the MES software itself, but also at whether it requires almost obsolete hardware or a soon-to-be-discontinued architecture. What is the message bus or database infrastructure used in the MES? Do those companies still support the revision used in your system? Don't spend several hundred thousand dollars on a system that will be unsupportable in six months.
Understanding system connections
To be sure that you are making the right architectural decisions, it is important to understand the numbers, types, and frequency of connections to the system today, and how those numbers are expected to change with the additional functionality. As the number of connections increases, it drives you to an architecture that uses a common bus type. It is also a good idea to separate the connections made for core functionality—product movement and tracking on the floor—from those made for auxiliary functions—reporting, equipment monitoring, and others. This separation can take place with either software or hardware redirects, or some combination of the two.
AUGMENTATION METHODOLOGIES
There are several methods and levels at which augmentations can take place. Depending on the reason you are augmenting and the life span of your custom code, not all methods will be a best choice.
Code-level changes
This is a change of the actual MES. It requires either that your MES be open source or that your MES have an application programming interface (API) for modifications. In some cases in-house staff can make changes; in other cases the vendor will require that their technical staff make any modifications. Making code-level changes is rarely a best choice, and works best for you as an end user if the MES is modular and designed for user modifications. Common modifications at this level are things like custom screen exit functions, custom algorithms for calculations, custom data lookups, and verifications. In almost all cases, it is best to avoid a case where the vendor branches their actual code to make you a "custom-built" MES. If they are doing it for you, they are doing it for other customers, and if the MES is not modular enough, they will not be able to support it over the long term, nor will upgrades be an easy or cheap prospect. This is not a scaleable solution by nature, and if your system is growing enough to become more distributed, the complexity involved in these modifications can increase the complexity of your system significantly.
Client-wrapping approach
If your client users normally pull data from many sources, you may want to consider a client-wrapping approach, where your end users are running a custom application that then interfaces with many other systems and applications. The client application could be stand alone or Web based. It is an approach that puts more business logic in the client code, and if clients have to connect in a peer-to-peer way, it can quickly become unmanageable. This approach works well when the number of clients is small. As the client base increases, or as the number of connections a client has to make increases, it can grow out of hand. This solution can be custom built or purchased.
Server-wrapping approach
In a server-wrapping approach, data and services fold together in a commonly available interface. This is a good approach if you have legacy systems you would like to have incorporated together. It is a loosely coupled solution, which allows some components of the system to change without breaking the entire application. Depending on the legacy systems behind the server, this implementation results in a common interface to several systems where the clients have no knowledge about exactly how the connections are made to the system. The advantage of this type of approach is that it gives you an opportunity to swap out the legacy system without interruption.
ISOLATING YOUR BUSINESS LOGIC
The common connection approach to augmentation is similar to that of the server-wrapping method, but it uses a common connection between components. An example of this type of connection is publish/subscribe. There is a connection to the legacy system that allows a message to publish to the system. This would require code changes in the system unless it already supports the functionality. Data for this system transmits from the clients to the system and back.
The advantage of this system is that there is a common connection language. The disadvantage is that there still is not a common data model for the systems. Thus, clients would have to be smart about what message to publish and would require some sort of wrapping that is also found in the server-wrapping method. The big buy from this approach is that you are using a middle tier to isolate your clients from changes in the back end—things such as database drivers, message buses, and the like. Most common applications are N-tiered server solutions in which the users are isolated from changes to the back end. Putting together a framework to which the client interface connects, rather than having the client interface talk directly to the back-end system allows you to change back-end systems and not have to change your user interface significantly. Also, isolating your business logic into the middle tier allows you to change the rules by which you run your business without redeploying or upgrading client interfaces and interrupting the manufacturing flow.
When you use hardware and software to build technology scaffolding upon which functional modules can hang, you are building a framework. Be sure you are just putting in the plumbing and wiring, making it capable to plug in other software packages—not writing all the other add-ons. All the functionality is expensive to maintain—there is a potential cost in programmer heads. If possible, you want to push the cost off to vendors designed to do software support. If there are pieces you have to build in-house, try and partner with vendors who are interested in marketing and maintaining it.
Finally, if you do not want a wrapped client or a client with the smarts to connect to the servers, you can go with a distributed bus or Web-device approach. In this approach, the services are each individually "wrapped" so that the clients can dynamically "discover" them. This generally requires much more of an infrastructure investment, as the bus must be standard within the company.
DELPHI SEES PLUSES AND PAINS
At the Delphi Automotive Systems, Delphi D Semiconductor Fabrication Facilities—the fab—we had implemented computer-integrated manufacturing in a variety of ways, for a variety of reasons, over a period of approximately eight years. These ranged from point solutions that used proprietary languages—Wonderware—to programs written in C or Visual Basic. This was a little more object oriented, but still resulted in different applications for every client. However, the long-term support was killing any attempts at ongoing development. Doing the necessary changes to keep up with changing business rules was preventing us from doing new deploys. A change in tactics was necessary.
After evaluating the economic and business factors, it was apparent that an MES upgrade (the best solution) was out of reach for the time being. This meant finding a way to augment the existing WorkStream MES to handle increased demand for recipe verification, prior-step verification, and a more common user interface. The road to developing a middle-tier, common point of connection system was rocky, bumpy, and full of pluses and pains.
By putting together a middle-tier solution, we learned that we could create a common graphical user interface (GUI) for a widely disparate group of equipment. The operators on the floor liked it; there was a low training impact, and it was easy to move operators from one piece of equipment to another. This was a flexible Java-based GUI that auto configured out of a back-end Oracle database, rather than hard coding individual equipment differences in the client. It allowed for a common client across the manufacturing floor.
Using an application server made it easy to manage middle-tier functions such as connection pooling, fail over, security, user authentication, and hot deploy. This was significantly easier than hand coding the functionality.
As in any undertaking like this, we learned that the tools and scripts used for deployment and testing were critical to save time and headaches. Spend some time and effort to purchase or build these tools, and it will pay off in the long run.
However, we did find that distributed programming is much trickier to debug, and it is hard for some procedural coders to change their mindset and do the full system debugging. This was a bigger stumbling block than we expected, and you should carefully select the human resources used in your project. The learning curve was much steeper than we had planned for, and it is the reason we are recommending you bring in experienced consultants. No matter how bright your programmers are, do not assume or expect them to learn by the seat of their pants.
Not surprisingly, when your resources pool together, small changes in one part of the system can have huge impact in other parts. This is a big problem when protocols are incomplete or schedule pressures overwhelm proper testing. No matter how boring or tedious, do not allow the pressures of the production floor to push you away from a well-documented test procedure. Spend the time up front to carefully develop test procedures. Make sure that both engineering and operations are involved in testing and that the developers without domain knowledge are not verifying your system.
WHICH APPROACH IS BEST?
The decision to augment versus replace or upgrade needs to be made on both a business and a technical basis. If you can get five or more years out of your legacy system, but it does not do everything you need it to do, it is a good candidate for an augmentation. If you are going to swap it out in three to five years, carefully consider all the other factors and their impact before deciding to just patch the system. If you are going to be replacing the system in the near future, look for an MES that does everything you want your system to do, with a little growing room. Be sure your next choice has an open API, so that it will lessen the impact on the enterprise of future system changes.
A well-designed framework will allow you to choose best of breed modules and change them out without impacting user interface or floor training. It allows for flexibility in keeping up with technology changes in the future, and can allow you the freedom to implement functionality on an as-needed basis, lessening the initial investment for your facility.
Which approach is best?
In general the best approach is the middleware/server-wrapping approach, because it supplies the most flexibility without modifying the original systems. The Web-presentation approach is good when you are really just interested in some sort of common interface to all the components, but this approach requires that each of the components have the ability to present to the Web in some fashion.
We operate out of a semiconductor fab, but in what other manufacturing sites does it make sense to use a middleware framework and augment existing systems, rather than make a clean replacement?
Where business logic changes often: If you find yourself frequently making code changes to your applications because the business logic directing how you run your enterprise—manufacturing rules, process controls, or electronic data interchange requirements from customers—changes often, then you are a candidate for a framework with isolated business logic. This makes it easy to centrally change business logic with less impact on system availability. If you have been running under the same business rules for the last three years, and your customer base has changed little, you are not likely to consider this so strongly. Of course, you may also be considering if you will still be in business next year.
Where there are many legacy systems: If you have many legacy systems that have islands of data, it may be hard for you to make well-informed decisions. Having a common interface where data from multiple sources can easily concatenate and display will empower you to make more realistic decisions. IT
Behind the byline
Nancy Glenn is a system architect at Delphi Automotive Systems. David Braun is director of new technology at R&R Visual Systems. This article comes from their ISA Conference paper MES Augmentation – An Alternative to MES Replacement in a Semiconductor Fab. See the original and more detailed version at www.isa.org/intech/augmentation.
Read questions answered by our experts or join the email list.
