April 2008

System Integration

Break out of the box

End users should do homework when hiring an integrator, but avoid micromanaging for best results


  • Users say expertise, thoroughness most important in choosing integrators.
  • Integrators bring skills of project management, business savvy.
  • Users, get to know your integrator before making a decision.
  • Integrators say, "Give us elbow room to prove our metal."
By Michael Whitt and Ellen Fussell Policastro

Jeremy Cohen's company manufacturers corrugated boxes, and he sometimes does not have staff to handle the controls knowledge he needs in his production process.

"I have one guy on staff who can do controls. We have 14 people on our maintenance staff," he said. "When we get into a project that requires any kind of automation upgrade or even troubleshooting, sometimes [our need] goes beyond experience of my entire staff."

Even though a user has knowledgeable control system professionals on staff, they are not always capable of handling the large capital projects in addition to their day-to-day duties. Acme Corrugated Box in Hatboro, Penn., is just one of many companies these days that are turning to contracted integrators to help move production to competitive levels. The company hires several companies for integrator experience, one for controls upgrades and one for obsolete controls. "If those controls show signs of failure, we'd bring [the integrator] in to design a system for new controls (HMIs, PLCs, programming of PLCs) and spec out any associated electrical hardware," Cohen said.

Systems integrators provide programming and integration services; they're hired on an hourly basis to perform on-site changes under close supervision, and sometimes they work off-site to execute a turnkey project. The first step to finding the right integrator is to determine the type of service you actually need.

"Integrators are small nimble companies with key areas of expertise," said Joe Bingham, president of AES Automation, a small system integrator focusing on water wastewater, oil, and gas. "Typically when you bring in an integrator, you need something done that's unique, by someone who has specific experience." And these companies that can turn on a dime are important for end users who need inexpensive, quick turnaround at a moment's notice. "You wouldn't go to a major engineering firm and say, 'I need this problem fixed tomorrow;' you'll be lucky if you get a meeting set up tomorrow," Bingham said. "They won't even look at you if it's not a $10 million dollar project.

"Most of the time, an integrator will come in with no documentation waiting, only verbal instruction, 'Fix this; I don't know what it does. It's a big black box to me. That's why you're here.' "

In his experience, Martin Michael, vice president of business development at Advanced Automation, said users value most "integrity, consistency, professionalism, and a specific business knowledge pertinent to their organization," he said.

Acme also uses integrators for upgrades since the company has quite a bit of older equipment (some of it 30 years old) as well as some brand new equipment. That leaves room for machine troubleshooting. "If there's a problem, usually an electrical problem we can't figure out after going through our own steps internally, we use the integrator to come in, hook up to the PLC, and lead us. We have the machine knowledge, and they have the control knowledge," Cohen said.

"We're truly looking for a collaborative approach in the decision-making process, which is what I call a win/win for both [the integrator and the user]," Michael said. With Acme's maintenance staff working with the integrator, they were better able troubleshoot an electrical problem. "We brought them in once when we were having a problem controlling the speed of the glue rolls of our corrugator," Cohen said. "That's controlled through the PLC. We were not able to trouble shoot. But they were able to hook up to the PLC, go into the program and find what the issues were and remedy those issues through the program."

Project patterns

A systems integration project follows a predictable pattern. Like any other discipline, the integration team must develop a scope of work, a budget, and a schedule, and develop a control system specification. Usually the first deliverable is a control network single-line diagram showing major control system components and how they interconnect.

PLC programming and operator interface programming occur in parallel most of the time. Deliverables, other than the software, include SAMA logic diagrams, screen diagrams, and an operations and maintenance (O&M) manual. They will specify a factory` acceptance test (FAT) if the programming is happening off-site. This gives the customer a chance for input, and the integrator a clinical environment in which to demonstrate scope compliance. They should produce and approve a formal FAT procedure ahead of time.


You can perform your site acceptance test (SAT) or checkout (also known as bump and stroke) on-site. In any case, use a procedure or checklist to make sure everything is properly verified before wetting, or commissioning the process. Users should have someone observe this process and sign off as you confirm each item. Ongoing training of the operators and maintenance team is important during this time. Do commissioning only after a successful comprehensive SAT. After completing a commissioning process, the plant should be online, up to temperature and pressure, and producing material.

Top priority

While end users need and want help from an integrator in some instances, what they look for in an integrator and what they should be looking for are two different things said Norm O'Leary, executive director of the Control Systems Integrators Association (CSIA) in Exton, Penn. (See accompanying story on CSIA). "Users should be looking to improve their business skills. Our members put together this document on best practices and benchmarks in general management HR, project management, quality, financial management, and business marketing (keeping infrastructure and standards up to date)," O'Leary said. These things are all important to having a good control system integration business.

But the most important is project management. "You can't put together a good system unless you have a good project management system in place," he said. "That takes working with the user up front. It also takes making sure the project management personnel on the part of the integrator are coordinating and agreeing with users on all aspects of the job." This is pertinent not only for accessible startup of the system, but also the testing and safety training and user training of the people to operate the system afterward. "Of the seven disciplines we believe are an important part of our members in helping them become better business people, the most important is project management," he said.

Checking references and technical and industry expertise are also keys to selecting the right integrator O'Leary said. "Most of our members work in multiple industries. So what's good for the food industry is also good for pharmaceuticals and metals industry. The equipment is pretty much the same, and the end results are pretty much the same. But you have to know about the industry you're in."

Let me do my job

Integrators know they need to prove their metal for the end user, but sometimes that is difficult when the user has a preconceived notion of what he needs. "Bear in mind the real strength of an integrator is his wide background and diverse systems he's worked on," O'Leary said. "The best type of end user to work for is one who doesn't think he knows it all. He doesn't say, 'here's what we've determined we're going to do, now just give us a price.' That type of user just wants a bid on something he's put together himself, and there's no room for the integrator to contribute and show their real strengths."

"The biggest challenge is when end users don't understand the complexity of their own system or the fact some things take a long time to fix," Bingham said. "As an integrator, there's a level of bureaucracy above the customers or miscommunication," he said. "You could be doing amazing work for two weeks. But two levels above your contact, someone is upset because you've spent all this money, and he doesn't understand the benefit."

"Our ideal customers have complex manufacturing needs," Michael said.  "They are forward-thinking and have a true business-to-technology vision at all levels of the corporate structure."

When the end user decides to upgrade a particular line, "and they find something they're impressed with, then there's no room for the integrator to come in and give them what they've already decided," O'Leary said. "That's a big mistake because they're not taking advantage of the wealth of experience the integrator has. And it may not be the best solution in most cases."

One contact, thoroughness, expertise

Cohen and his team look for thoroughness and repeatability of assigned manpower. "When they send someone out to our plant, we don't want a different person every time," he said. "They're now able to send the same person when they can for the most part. Now this person starts to learn the function of our equipment as well. If you send a different person every time, that's frustrating."

Thoroughness is another key attribute users want in their integrators, someone who looks for the whole system, the document, what's going on. "We look for expertise in certain areas. We're trying to keep our PLC and OEMs down to two," Cohen said. "Eventually we'd like to get our folks trained. When we work with an integrator, we want to make sure they have experience and expertise to troubleshoot to work with those two OEMs. That doesn't mean we don't have different PLC manufacturers in here; we do, but now we're trying to standardize. They need to be able to work with our standards on hardware. They need to have experience with all kinds of OEMs.

"The most important characteristic boils down to expertise and knowledge of general controls, PLCs, drives, and things we don't have knowledge about," Cohen said. "If they have that expertise, they can come and ply it to fix our equipment."

Choosing an integrator

Any time you make a business decision, whether vendor, integrator, or other, you need to check references, Cohen said. "Visit a plant where they've done work to see an example of a project or installation. One issue we had was we didn't have that repeatability of manpower. You want to pick a provider that can get you the same one-to-three people consistently," he said. "We made it clear to [our integrators] we needed that."

Even though Acme uses one main third-party integrator, they are in the process of using another one for another project with expertise in hydraulics. "We had a big project that went out for bid for controls upgrade that had significant hydraulics on it," Cohen said. "We felt more comfortable with a hydraulics outfit with a local controls department. That's a combination a generic integrator would not be able to give. They'd have to subcontract it."

Any project needs due diligence prior to bringing in an integrator, even if it's a repeat, Cohen said. "You want to make sure they have the expertise for that particular application."

Sharing lessons learned

"Oftentimes, experience and length of time in the industry is your greatest teacher and cannot be substituted for a quick lesson or quick advice," Michael said.

The key is to empower customers "to be in charge of their future and to be self-sufficient by delivering a complete customer-supportable production system," he said.

A successful integrator will also want to know how to be a good business person O'Leary said. "Our basic premise is the reason jobs are not a tremendous success all the time isn't due to technical ineptness of the integrator. It's because they're poor business people."

Documentation is also a crucial step in successful integration Bingham said. "Make sure you document what your clients' needs are, and what you've done for them in a timely manner because you can keep a long-term client or lose a large client without doing that. Once you hand over the pricing, rates, and what it's for, then you can leave that behind and go forward, and it's understood."


Michael Whitt is I&C department manager at Mesa Associates Inc. in Knoxville, Tenn. Contact him at mwhitt@mesainc.com. Ellen Fussell Policastro is associate editor at InTech. Contact her at efussellpolicastro@isa.org.


Control Systems Integration Association

Control Systems Integrators Association (CSIA) was founded 14 years ago or so by a group of 15 integrators who decided, even though they were competitors, they had the same business problems-how to operate a business and market more effectively. The original mission was to improve business and marketing skills of members. "We look to organizations such as ISA and other associations to take care of their technical end of it. Our whole thrust has always been to improve members' business skills," said Norm O'Leary, executive director of CSIA in Exton, Penn. "We're now up to 260 plus members of control system integrator members," he said. And about 60 partner members are the hardware and software suppliers who provide equipment members need to automate their clients' systems.


Integration step by step

By Michael D. Whitt

There are three primary categories for integration services:

Staff augmentation service fills a short-term labor problem. The staff augmentor is usually mated to a member of the customer's organization who provides supervision in the performance of maintenance upgrades, production modifications, and minor process revisions.

Turnkey service is off-site and unknown to the user's organization. The integration team sends one or more representatives to an off-site project to assist in startup and training.

General integration service means the integrator should be in close proximity to the site and should want to keep a long-term relationship and provide rapid support on call.


Here is a list of deliverables to expect to support the project:

I/O list: Given an instrument list and a set of P&IDs, the integrator should produce an I/O list with the computer addresses (the hardware address and software address) of each I/O point in the system.

PLC program: Organize the PLC program so it is modular in design. Keep it well documented with imbedded comments and pertinent symbol names. Agree early on naming conventions.

Network single line: Provide a single-line diagram of the data communications scheme and addressing information at each drop.

HMI: Before beginning work on the HMI, operations personnel should produce and review a sample screen. Specify naming conventions, symbols, and color schemes early.
Logic diagram: Before beginning the PLC programming activity, make the process logic detailed. You can do this as a process narrative, or as a set of SAMA logic diagrams.

Factor acceptance test (FAT): A FAT is an important feature of a properly executed integration project. Prior to shipment, the systems integrator should write a procedure. This procedure should demonstrate compliance with the scope of work, and the customer should approve it before its execution.

Site acceptance test (SAT): Normally, the constructor checks electrical continuity as the construction process evolves. At some point after continuity checks but before wetting the process, the integration team installs the software on the process computers and begins the checkout phase.

Startup (commissioning) procedure: You should write the startup procedure around process sequences. After wetting the process and reading temperatures and pressure properly, make sure your facility demonstrates proper operation through each operating mode, with special attention to emergency modes of operation. Get this procedure approved before the commissioning process.

Operations and maintenance (O&M) manual: Produce an O&M manual or set of manuals as a corroborative effort. Often the systems integrator is in the best position to head this effort as he will have the most information about the inner workings of the system.

Training manual: The systems integrator should be prepared to perform operator and maintenance training prior to commissioning.

SOURCE: Successful Instrumentation and Control systems Design, by Michael D. Whitt.