Comment

1 April 2005

Making the Connection

By Rao Kalapatapu

Supervisory control and data acquisition (SCADA) systems consisting of SCADA hosts, remote terminal units (RTUs), and field devices monitor and control process equipment and systems from multiple locations and exchange data from various distributed control systems among local and wide area networks.

SCADA system operation involves real-time data exchange from the field devices as well as with other control systems such as DCS (Distributed Control) and Plant Information systems.

A typical RTU in the field contains a central processor, set of Input /Output (I/O) modules, and communication devices to connect to field devices. RTUs are similar to programmable logic controllers (PLCs). PLCs work in a local area such as factory floor, and they connect usually by a local area network. RTUs, on the other hand, work remote locations and connect via a wide area network (WAN). Both have CPU, I/O units, and communications ports.

You can preprogram RTUs to communicate with the central station SCADA and other networked systems in protocols designed to deliver reports on the status of all input and output devices in the field.

Protocols are similar to languages, which allow the RTU/SCADA units to communicate with each other. The basis of all network architectures loosely relies on International Standards Organization (ISO) standard seven-layer Open Systems Interconnection (OSI) model:

• Layer 7—Application
• Layer 6—Presentation
• Layer 5—Session
• Layer 4—Transport
• Layer 3—Network
• Layer 2—Data Link
• Layer 1—Physical

The object of the OSI model is to establish a framework that will allow any system or network to connect and exchange signals, messages packets, and addresses. The model makes it possible for communications to become independent of the devised system and shield the user from the need to understand the complexity of the network.

In general, the bottom four layers cover the physical wiring, network, and communication protocols of the local and wide area networks such as Ethernet and Frame Relay.

The presentation and session layers usually deal with establishing and then terminating the session between the two hosts. Not all networks use these layers. The application (Layer 7) and above is where a typical PLC/RTU Protocol (such as Modbus) will provide the data at a typical SCADA workstation/server in a user format from the field RTUs and local PLC systems.

Match game

A large part of any complex SCADA system design is involved in matching the protocol and communication parameters between connecting devices. There are about 200 such real-time user layer and application protocols. These include proprietary and non- proprietary protocols. The industry is moving away from the old and proprietary protocols.

The following RTU/ PLC protocols are emerging as virtual standards in modern SCADA systems:

Modbus: During communication on the point-to-point Modbus network, the protocol determines how each controller will know device address, recognize a message addressed to it, determine what action to take, and extract any information/data attached to it. There are a number of expansions to fix these shortcomings.

A de facto standard in many companies, Modbus does have some shortcomings. For one, it cannot handle large positive and negative numbers.

Modbus X: Companies, utilities, and SCADA software suppliers have adopted non-proprietary Modbus X expansion. It fixes the Modbus shortcomings, makes it man readable, and makes it able to handle positive and negative numbers with up to nine digits of resolution, with an exponent range from -99 to +99.

Point protocol is designed to read and write to individual I/O points in PLCs on a factory floor. The Modbus X expansion of the protocol is a universal, non-proprietary expansion, which permits handling large process variables in plain American Standard Code for Information Interchange (ASCII) with sign and exponent, capabilities that are missing in Modbus. With the universal Modbus X expanded protocol, it is no longer necessary to experiment with different proprietary expansions of the protocol.

DNP (Distributed Network Protocol): A member restricted protocol, used in some electric power systems. The DNP protocol has gone through various iterations. Presently, it is up to version 3.0. The DNP association has rules, which tend to restrict the use of the protocol.

ASCII: The dominating computer protocol is ASCII. Virtually all computers, printers, and modems and many sensors, actuators, and flow computers now communicate in ASCII.

IEEE 60870: This protocol sees use in power transmission and distribution systems. IEC 60870-5-101 is an International Communications Protocol Standard for the Telecontrol of Electric Power transmission systems, which countries throughout the world are adopting.

Device connections

The local area networks/protocols from sensors/field devices to the PLC/RTU and from PLC/RTU to SCADA are sensor networks. These are basic on/off field devices connecting networks:

Fieldbus networks: These connect analog and smart field devices such as valve actuators, pumps, and other field control systems.

Control networks: These allow for peer-to-peer connections between control systems such as SCADA/DCS/analyzer/safety PLC systems.

Safety buses: These are deterministic time sensitive safety type device connections.

As in any data communication network, communication media gets support from PLC/RTU to central/ distributed SCADA systems via local and wide area networks. A local area network (LAN) exists within a local geographical area such as a building or a campus and consists of few buildings in close proximity. On the other hand, a wide area network (WAN) is a network that connects local area networks spread across different cities at least 100 kilometers (67 miles) apart.

The types of WANs include:

• Analog point-to-point and multi-point modem networks
• Frame relay/cell relay type point-to-point and multi-point networks
• Wireless radio/satellite networks
• Fiberoptic-based networks

At present, you can build a SCADA network around many of the above possible combinations of networks and transmission protocols. More computer power will come from the SCADA hosts since they need more data to meet the information demands of the present day corporate networks.

Existing SCADA protocols share certain characteristics that make them very dependable in "robust but older/legacy" networks.

These networks operate at very low speed namely 300 to 1,200 bits per sec (bps). The data is usually asynchronous (no time synchronization), and RTUs are polled over a single multi-point circuit. All this polling and response must happen in a very short time that is often measured in fractions of a minute.

To meet this higher demand for data, SCADA equipment manufacturers, system designers, and users started looking for higher speed devices. There is a very large existing infrastructure of phone lines, private wire, and microwave that operate at voice grade frequencies, where there is a bandwidth of about 3,000 hertz. This is sufficient to pass, for example, a 9,600 bps fast poll modem. Fast poll modems work like 1,200 modems because they pass data over voice grade lines, use 4-wire phone lines, and work in multi-drop mode.

In addition, the new faster networks and their protocols interact with existing SCADA protocols and present some interesting challenges. The network protocols, such as frame relay, Ethernet, and IP, each have time sensitive characteristics that will generate delays, cause short gaps in the data, or not transmit some Data Carrier Detect (DCD) signal transitions. These delays may cause SCADA protocols to assume errors in the links. If you can get over that, new and faster communication networks are available.

In the communication world, there are two types of networks, circuit switched and packet switched. The circuit switched network establishes direct connection between two or more stations by means of switches, which normally occurs with telephone dial up modem networks. On the other hand, there is a general move toward a packet style operation where the data works in packets prefixed with some addressing. In a packet switched network, data heads in the best possible route in a complex meshed private or public wide or local area network. Packet switched networks are more cost effective since a dedicated network is not needed from start to finish.

Telecommunications facilities switch, combine, amplify, and transmit information over chosen media. The hardware consists of transmitters to convert the voice/data signals to suit the transmission media, wiring, switches, routers, and bridged and multiplexers to amplify and carry the signals to a receiving station.

The following are some current day networks used for remote SCADA transmissions from remote and local area nodes in wired or wireless networks:

Frame relay: Frame relay is a packet switched network. The data packets of frame relay networks may have no direct correspondence to the size of SCADA poll/response packets. Therefore, a SCADA packet will often break up into several frame relay packets by the network, with delays between the frame relay data packets. Typically communication system vendors use these packets when transporting large data through large geographical distance. They carry the data encapsulated for transportation by their frame relay protocols via their routers in all available, multi-path, meshed, long-haul network. The frame is a data communication service provided by telecom carriers across a network with one or more points.

Ethernet: Rather than using terminals to access shared mainframe or minicomputer, the user now works directly at a single user computer. The computer in turn connects to a local network giving the user shared access to information stored and to common peripheral equipment such as printers and fax machines. This resulted in changes in wide area networking. Users are able to make LAN-to-LAN connections instead of terminal to computer.

Ethernet is also a packet-oriented protocol. Ethernet packets generate without regard to the incoming data protocols. Ethernet devices have protocol rules to obey to allow variety of possible devices that can connect to an Ethernet network similar to frame relay. These are local area based but apply to wide area.

ATM: Asynchronous Transfer Mode is a cell relay type broadband connection oriented switching service that carries data/video/audio information in fixed length of 48 bytes with a five byte header for a total of 53 bytes. Broadband is a form of network modulation in which multiple channels form by dividing the transmission medium (such as fiberopotic cable) in to discrete frequency segments. This is a global standard.

Fiberoptic networks: Linking computers over a long distance, you use existing voice-optimized WANs. The problem was computers sent data instead of voice, and data had different characteristics, so these WANs did not send computer data very efficiently. To address these concerns, ITU-T (International Telecommunication Union–

Telecommunications is an agency of United Nations formally CCITT) and other standards groups started work in the 1980s to establish a series of recommendations for the transmission, switching, signaling, and control techniques. These require a user to implement an intelligent fiber-based network that could solve current limitations and would allow networks to efficiently carry services of the future. This network is a Broadband Integrated Services Digital Network (B-ISDN). By 1990, decisions occurred to base B-ISDN on Synchronous Optical Network/Synchronous Digital Hierarchy (SONET/SDH) and ATM at speeds varying from 55 Mbps to N multiples of 55 Mbps, which carries up to 4,696 Mbps.

These can work for long distance distributed SCADA system transmission of data by private or public switched networks at speeds of 51.8 Mbps to 2.48 Gpbs.

TCP/IP networks: TCP/IP networks have the same packet characteristics of frame relay and Ethernet networks. There is no relationship between the IP packets and the incoming SCADA poll/response data packets. TCP is a protocol developed in late 1970s by Department of Defense for providing interoperability among several equipment vendors.

In 1989, the Internet evolved from Advanced Research Project Agency Network (ARPANET) at the Department of Defense. Internet is a collection of independent Packet Switched Networks loosely interconnected to act as a coordinated network. Unlike OSI, TCP/IP is not a true international standard but an open standard widely used internationally.

IP addressing: IP addressing works on the concept of hosts and networks. A host is essentially anything on the network capable of receiving and transmitting IP packets on the network, such as a SCADA RTU/ SCADA workstation or a smart field device such as a smart control valve.

The hosts connect together by one or more networks. The IP address of any host consists of its network address plus its own host address on the network. IP addressing uses one address for network and host addresses.

How much of the address goes to the network portion and how much for the host portion varies from network to network.

An IP address is 32 bits wide and consists of two parts: the network number and the host number (1, 2, and 3). By convention, it appears as four decimal numbers separated by periods, such as "200.1.2.3" representing the decimal value of each of the four bytes. Valid addresses range from 0.0.0.0 to 255.255.255.255.

Virtual SCADA networks: Leveraging the availability of the Internet and wireless technology, SCADA vendors found remote access solutions, which allow virtually every telemetry and monitoring device to connect to the Internet and thereby to your terminal of choice. It enables you to connect to your network or device, without having to stay in the office.

Wireless networks: Wireless networks come in many flavors and styles. These include:

• Satellite networks
• Licensed VHF and UHF point-to-point and multi-point radio
• Spread spectrum license free (900 MHz, 2.4 GHz, 5.8 GHz, and 24 GHz)
• Point-to-point microwave

Within a narrow range of a building or campus, wireless data can move from node to node with privately owned spread spectrum radio networks. Broader ranges require some form of public network. The most common method is dial up over cellular, cellular digital packet data (CDPD) networks, or by using Specialized Mobile Radio (SMR) or private carrier wireless networks.

Users need to review how to maintain older legacy SCADA systems to provide the best possible control and gradually upgrade:

• Field Remote Terminal Units and field device wiring/networks
• Control room central SCADA units and local area networks they reside
• Communication networks between the field PLC/RTUs and the central SCADA systems/other control and IT networks
• Interfaces between control room SCADA and other control/IT networks

For new systems, users need to review the following:

• Open non-proprietary standards and protocols for RTU/SCADA control and monitoring
• Wire and wireless communication systems for cost effective systems
• Industry proven and easily upgradable systems
• Interchangeable software and hardware from one vendor to other
• Ownership of programs and system software code after purchase of systems

The newer RTUs come in variety of flavors and include communication connectivity through serial Modbus, Modbus/TCP, Ethernet, and a host of other protocols. The remote processors have smart modems for wireless or leased telephone/satellite connectivity. In all cases, the data is serial, and depending on the amount of data, it can transfer quite fast with 9600/19.2 bps modems and wide area networks.

Present day designs will use distributed processing and peer-to-peer communication from field to the control room. For example, each local RTU can have a unique I/P address similar to unique tag names we had for sensors and switches during last couple of decades. These I/P units are then polled by any local/PC/RTU/PLC/Terminal Server, once every few seconds or as needed to get real time operating data from the field.

For example, the chief executive of XYZ Company may only need the daily production reports from the oil/gas well fields, the SCADA manager may need only operation and maintenance reports, and a shift operator may need only real time data on his MMI console. All these happen now with the SCADA systems using off-the-shelf hardware and software products available on the market.

The cost of operating this local and wide area networks needs to go into consideration when designing a SCADA system. For example, a typical cost of leasing a 1.5 Mbit T1 line runs in to around $700 to $1,000 per month for 1,000 miles distance. But the cost of owning a 1,000 mile, 10-strand fiber optic cable line with a capacity up to 2 Gbps per two fibers has an installed cost of around $5 million. So, the monthly loan payment over 20 years will have a loan payment of around $25,000 per month. To get an equivalent leased line, you can see costs are prohibitive, so you cannot compare them on equal footing. Even though fiber optic cable costs appears very high, the operation and maintenance costs are low, and the capacity is unlimited for all future expansions where, as in the case of leased lines, the costs increase quite rapidly when bandwidth requirements reach 1Mbps and above.

Also, due to advances in technology and possible custom and off-the-shelf hardware and software availability, the distinction between RTUs, PLCs, DCS, and PCs are disappearing. All these now overlap each other with respect to functionality, protocols, hardware, and software. The main requirement for all these networks is they have to operate in real-time deterministic mode so the process and production data can be received and sent in a timely manner to all required places on the local and wide area networks.

Will SCADA work well over cellular telephone links or the public Internet? Probably, but you must use caution because they are geared for short-term conversations, have low data rates, and are susceptible to call disconnection.

The Internet will have delays similar to that of old packet switched X-25 Networks (used for example, in credit card processing) and may also suffer packet loss that can be as high as 10% or more during severe congestion periods. The Internet may be acceptable for some remote monitoring and data retrieval, but not for real time data collection and control needs. It appears private and semi-private networks, such as copper cable, fiber optic cable, phone lines, and satellite will be the communications choice. Open non-proprietary protocols such as Modbus, Ethernet, TCP/IP with application protocols such as OPC/OBBC will be the best choice because of their availability from any SCADA vendor and they provide simplicity of design and installation as well as ease of system operation and maintenance. CE

Behind the byline

Rao Kalapatapu, P.E., is a senior engineer with experience in design and installation of SCADA, DCS, PLC, and other control systems for oil and gas pipelines and chemical and petrochemical projects. He is a senior member of ISA. His e-mail is vrkalapatapu@hotmail.com.