01 March 2005

Wide terrain in LAN

Ethernet blossoms with bus, star, mesh technology.

By Kevin Burak and Roland Gendreau

As an open standard, Ethernet has the ability to link a wide range of vendor-neutral networking equipment. Today it enjoys supreme popularity as a physical layer local area network (LAN) technology. Most computers sold today come already equipped with an Ethernet connection. This helps guarantee a large market for Ethernet equipment and keeps the technology competitively priced.

The original two Ethernet standards 10Base5 (ThickNet) and 10Base2 (ThinNet) used a bus topology. This topology is a single coax cable or segment, where devices (more commonly known as nodes) attach or tap into it with an interface connector. You must use impedance terminators at each end of the cable. As a single collision domain, nodes on the bus can only transmit one message at a time. The carrier sense multiple access/collision detection (CSMA/CD) protocol determines when a node can transmit. The carrier sense requires a node to listen before transmitting. If another node is transmitting, then the node wanting to transmit must wait its turn. Multiple access means any node can access the network at any time. This is why nodes must wait their turn to transmit. The CSMA protocol also implies it is a shared bus system, where all nodes on the network see all transmitted packets. If a node receives a packet not intended for that node, it ignores it. Otherwise it passes the received packet up the protocol stack for more processing. Collision detection means it recognizes nodes attempting to transmit at the same time. Once a node detects a collision, both nodes will stop transmitting and individually wait a random amount of time before retransmitting.

Repeaters lengthen the network cable or link multiple coax segments together. They operate at the physical layer of the open system interconnection (OSI) model. They basically take the incoming signal (Ethernet frame) from one cable segment and amplify it out the other cable. Repeaters will broadcast any transmission (good or bad) by a node, which all other nodes will hear. As such, all nodes will hear even collisions. Unfortunately, repeaters cannot extend the cable segment indefinitely. The CSMA/CD protocol requires a low propagation delay to work correctly. If the segment extends too long, the propagation delay will be too long. This limits the number of repeaters to four or less between any two Ethernet nodes.

The coax cable is very reliable and easy to set up, but it shares a single collision domain. This can create network performance and fault-isolation problems that may potentially bring down the whole network. For example, if there is too much traffic, there will be nothing but collisions, and no one will ever be able to successfully transmit. If a terminator is missing at one end of the cable, the network will be unreliable or won't work. A break in the cable or a configuration change (the bus must break to add in a new section) will bring down the whole network. Trouble shooting requires the process of elimination method, where you must check each node and the cabling between them one at a time, a tedious and time consuming procedure.

Industrial communication networks do not typically change once installed, so using coax for their networks is not a hindrance for deployment. However when a fault occurs on an industrial Ethernet coax network, the whole network will go down. This makes a single bus topology subject to single points of failure. To solve this single fault issue, you typically deploy two separate networks for redundancy. If a fault does occur and brings down one of the networks, the system simply switches to the backup network.

Star topology

Introduction of the IEEE 802.3i 10BaseT standard launched a major advance in Ethernet standards. The 10BaseT standard permits Ethernet to transmit 10 Mbps over simple Category 3 unshielded twisted pair cable. The widespread use of unshielded twisted pair cabling (phone wire) in existing buildings created a high demand for 10BaseT technology. However, 10BaseT only allows distances of 100 meters from a hub to a node and is more sensitive to electrical noise than coaxial cable. Shielded twisted pair is suitable for these noisy environments, but no standards officially define Ethernet over shielded twisted pair.

The introduction of the IEEE 802.3j 10BaseFL standard eliminated some of the 10BaseT shortcomings. The 10BaseFL standard is basically 10BaseT that runs over fiber optic cable instead. However, it supports distances of up to 2000 meters and is completely immune to any type of external electrical interference. This makes 10BaseFL attractive for areas where electromagnetic interference (EMI) and radio frequency interference (RFI) levels are high (such as factory floors). Fiber is also difficult to tap in to, thus providing some level of security. With a 2000 meter limit, 10 BaseFL is also useful for use in interconnecting buildings in a campus environment where distances may be long.

On one hand, developers designed Ethernet as a bus system to operate over a coax cable using a point-to-multipoint connection. Twisted pair and fiber connections, on the other hand, are from the OSI layer one (physical) point of view, point-to-point connections. Point-to-point connections require a multi-port repeater (Ethernet hub) to connect to multiple nodes together (see Figure 3). The hub operates at the physical layer of the OSI model. Logically, the star topology still appears as a bus. When a node transmits, the hub will then retransmit the packet out its other connections. So, it is still possible for collisions to occur.

Hubs have automatic partitioning allowing the automatic bypass of any ports disconnected or with a cable fault. This isolates the fault until it is corrected. In a worst-case scenario, you can troubleshoot by disconnecting nodes one at a time from the hub until the network recovers. Usually, the hub will give an indication (such as LED) as to which node is causing a problem, allowing the technician to troubleshoot that node as opposed to spending hours finding the problem. This makes the network more fault tolerant than a coax based system, where disconnecting a single connection will bring down the whole network. This also makes it less likely any node can cause the entire network to fail. However, if the hub does fail, the entire network will also fail. Again, this is a single point of failure in an industrial communication network. So, industrial Ethernet star networks typically deploy two separate networks for redundancy. Like industrial Ethernet coax networks, if a fault does occur and brings down one of the networks, the system simply switches to the backup network.

Though the star-based networks do require more cable than a coax network, they are easier to install, manage, and troubleshoot. You can resolve any EMI or RFI noise issues on the factory floor with the use of shielded twisted pair or fiber cabling. Unlike coax-based networks, disconnecting a node from the network will have no affect on the rest of the network. Therefore, moving an attached node is simply a matter of unplugging it from the hub and reconnecting it somewhere else. When something does go wrong with the network, an administrator can troubleshoot it from one place. You can also expand star topologies into a tree topology, where one hub connects to other hubs, which in turn connect to end nodes.

Collisions

Ethernet networks designed as a bus system (logical or otherwise) do not scale well. As you add more nodes to the network or as nodes transmit larger amounts of data, collisions will occur more often and network performance will deteriorate. Minimizing collisions on the network is critical in keeping Ethernet performing well, and Ethernet has employed a couple of different methods. The simplest approach is to just add bandwidth. The faster the transmit speed, the quicker the network is free to transmit. Today's Ethernet speeds now range from 10Mbps to 10Gbps. The faster Ethernet standards include fast Ethernet (IEEE 802.3u 100BaseT) at 100Mbps, Gigabit Ethernet (IEEE 802.3z) at 1000Mbps (1Gbps), and 10Gbps with the IEEE P802.3ae standard.

Another method to minimize collisions is to segment the network so there are fewer nodes competing for the wire. This is when the network divides into different pieces and a bridge joins them together logically. Bridges isolate network traffic by selectively filtering the traffic between segments and allowing only the packets needed for that segment. This minimizes collisions and significantly increases the throughput on each segment as well as the overall network. When the bridge receives a packet, the bridge determines the destination and source segments. If the segments are the same, the bridge drops the packet; if the segments are different, it forwards the packet to the proper segment. Bridges will not forward bad or misaligned packets. Ethernet switches are in fact a marketing term for multi-port bridges.

Originally Ethernet (CSMA/CD) was a half-duplex protocol. Only one node could transmit at a time. Full-duplex protocol allows nodes to transmit to each other at the same time for a collision-free environment. This effectively doubles the transfer rate. If a 10Mbps full duplex node can transmit at 10Mbps and at the same time can receive at 10Mbps, the overall effective transfer rate is 20Mbps. For Ethernet to be full-duplex, it bypasses the normal CSMA/CD protocol to allow the two nodes to communicate over a point-to-point link. Nodes on a star network only communicate point-to-point, via a switch, and never directly with each other. In addition, switched networks use either twisted-pair or fiber-optic cabling, both of which use separate conductors for sending and receiving data. In this type of environment, nodes can skip the collision detection process and transmit at will since they are the only potential devices that can access the medium.

Full duplex became available in 1997 with the release of the IEEE 802.3x standard.

Network segmentation, more bandwidth, and full-duplex greatly enhance Ethernet's deterministic performance. However, all these enhancements do nothing to solve the switch as Ethernet's single point of failure. With switches as a single point of failure, industrial Ethernet star networks still need to deploy at least two separate networks for redundancy.

Mesh topology

The star topology eliminated the single point of failure of a common wire, but if the hub/switch fails, so does the network. To eliminate the switch as a single point of failure, add a second switch between two segments as a backup in case the primary switch fails.

Network loop

However, this creates a loop in the network. When loops are present, some switches see nodes on both sides. This condition confuses the basic bridge forwarding algorithm and allows forwarding of duplicate frames, leading to an explosion in traffic (such as broadcast storms) that can adversely degrade the performance of the network. For the network to work properly, only one active path can exist between two nodes. To prevent loops but still allow a redundant network, use the spanning tree protocol (STP) 802.1D. STP allows switches and nodes to connect in any manner you want, and it will automatically sort out the loops. A drawback of STP is it takes a minimum of 30 seconds to converge on a failure. This amount of recovery time is unacceptable in industrial networks carrying real-time traffic. The rapid spanning tree protocol (RSTP) helps void this limitation. RSTP, specified in 802.1w, now supersedes the slower STP specified in 802.1D. You can hold the RSTP failover times to as short as 100 milliseconds. Although RSTP convergence advertises three to five seconds, with appropriate planning and RSTP switch configuration, the convergence time can be well under a second.

Armed with RSTP, single switched Ethernet networks can be configured to provide functional redundancy. These redundant switched Ethernet networks can range from a simple ring topology to a fully meshed network. In a mesh topology, each node on the network has redundant data paths.

The more inter-switch connections there are, the higher the degree of reliability the network will provide. In a mesh network, should any cable, switch, or some other component fail, data can still travel along an alternate path. Industrial mesh networks are configured such that the network does not have a single point of failure. Only one network needs to be deployed to provide the degree of fault tolerance required for the needs of an industrial Ethernet network.

Mesh network

Redundant network access

The mesh topology provides excellent network redundancy with failovers in less than a second. End nodes attached to the mesh network and running critical industrial applications must also have redundant network access. Without that, each node would have a single point of failure. Network access redundancy must be automatic and transparent to the application. When a network access failure does occur, the end node must detect it and transparently switch its access to the backup.

While the current suite of standard and proprietary network protocols, such as transmission control protocol/internet protocol (TCP/IP) that runs over Ethernet, provides excellent end-to-end error detection and recovery, they do not provide redundant network access. Some standardization in redundant network access (such as Foundation fieldbus high-speed Ethernet) have existed, but these approaches are not application transparent. A transparent redundant network access allows porting of any legacy application to the mesh network without worry. You can write new applications with standard network function calls. Redundant network access allows the single standard mesh network full redundancy and transparency to the applications.

With the advent of standard Ethernet network technologies, such as the rapid spanning tree (IEEE 802.1w), full-duplex (IEEE 802.3x), and high speed Ethernet (IEEE 802.3u, 802.3z, 802.3ae), you can now deploy Ethernet LANs as highly reliable deterministic self-healing redundant mesh networks. Coupled with devices with transparent redundant network access, the Ethernet mesh network provides the factory with a high performance network that has redundant data path failovers in the order of hundreds of milliseconds. Based on open standards, the Ethernet mesh network also lowers deployment and life-cycle costs by moving away from expensive proprietary networks to a low-cost standard network you can easily maintain.

Industrial devices with redundant network access

Behind the Byline

Kevin Burak is a consulting network engineer at Invensys in Foxboro, Mass. Roland Gendreau is a product marketing manager at Invensys.

One reliable network, dual control

"The thing that's unique with Ethernet is the way people implement control system networks to dual control," said Roland Gender, product marketing manager at Invensys in Foxboro, Mass. The mesh network uses commercial off-the-shelf (COTS) technology to provide a single network that gives a single level of reliability, as opposed to what dual networks would provide. It reduces the cost of implementation to our customers." Customers are using COTS rather than proprietary equipment to deal with redundant Ethernet networks. "Ethernet was never designed to be redundant originally," Gendreau said. "If people wanted to make it redundant, they needed to add a proprietary network."

The Ethernet mesh network consists of a number of smart Ethernet switches wired together in a particular topology to create a mesh structure where each device or node attached to the network is connected to more than one place in the network. "As the name implies, it's a mesh, like a spider web, with many different paths you can take to get from one place to another," said Paul Steinitz, director of marketing for Foxboro Automation at Invensys.

"So in this day in age of putting more data on the control network, you need more reliability. We're seeing our customers putting more information on that highway because of fieldbus. You can get more than just one transmitter signal, and thus more information from each transmitter," Steinitz said. "And multiple paths will tolerate multiple failures and problems, like bypassing a wire segment that went down."

Gendreau said the trick is making sure the data is routed correctly and making sure it's going only to the place that needs it. "The switch will look at data packets coming in and say, 'Does this information need to go to my other side, or do I just stop the transmission here?' If it does, it transmits it. If not, we don't crowd, as opposed to Ethernet, which broadcasts everything to everybody."

"The key thing with mesh is we're using one network, flat architecture, for all communication needs of the system. Other systems might have multiple networks to do the same functionality. Typically you have a collision domain where all the stations on the network see all the transmissions sent to the network," Gendreau said. "This is different. With switched Ethernet, the stations only see traffic that's routed to them. And that frees the station up. It doesn't use up its bandwidth looking at each message coming to it."

Another main benefit for customers is throughput, Steinitz said. "Others would say it's the self-healing and fault tolerance. But it depends on whether you have a large or small plant. If you have a large plant, throughput is a concern, but if you have a small plant that can tolerate no failures, then you'll say the biggest benefit is fault tolerance."

But the primary benefit to the end user is security and reliability in the sense of being able to prevent threats to the network, Gendreau said. In an oil refinery, you have a central control room where a network makes decisions about which valve is open and what temperature is raised. "That network must be secure and reliable," Gendreau said. "That's what this mesh network is. It's the link between all the devices and operators inside a plant."

History of Ethernet

Ethernet came on the scene as a single coaxial cable network in the early 1970s. This work led to the original IEEE 802.3 standard in the early 1980s, and Ethernet now enjoys popularity as a physical layer local area network (LAN) technology. Early Ethernet users first wired the LAN using coaxial cables, with each station tapping into the cable. As a shared single collision domain (a single cable all devices on the network share), this created nondeterminism and performance and fault-isolation problems, as well as the lack of network redundancy. To overcome these original Ethernet shortcomings, industrial networks typically deployed multiple networks, which made it hard to deploy and manage and led to proprietary solutions.

As the popularity of Ethernet LANs continued to grow, a more structured approach, called star (or hub-and-spoke) topology, came into use where all attaching devices linked to a repeater. This helped with fault isolation and provided a more organized methodology for expanding LANs, however it did not help with network redundancy. With the advent of superior standard network technologies, such as rapid spanning tree (IEEE 802.1w), you can now deploy a single Ethernet LAN as a highly reliable, deterministic, and self-healing mesh network. The mesh network provides the redundancy you need for industrial networks. It also provides industrial applications with a fault tolerant network with redundant data path failovers in the order of hundreds of milliseconds.

In the early 1970s, Robert Metcalfe and David Boggs of the Xerox Palo Alto Research Center experimented with the first Ethernet system. Then, Xerox's primary motivation was to create a network to share a laser printer with a large number of computers. To help ensure Ethernet's openness, Metcalfe convinced Xerox to license the Ethernet technology to everyone at a reasonable cost. In 1979, Digital Equipment Corporation (DEC), Intel, and Xerox created a consortium to pursue the open Ethernet standard. This standard was to allow DEC to sell more computers, Intel to sell more chips, and Xerox to sell more printers. The Institute of Electrical and Electronic Engineers (IEEE) recognized this technology could not become an international standard if a single U.S. corporation controlled it. So it created the 802 committee to develop formal international standards LAN technologies. The IEEE just did not want to standardize each LAN individually, but to establish a set of rules that would allow data to easily move from one LAN technology to another.

Coming Next

Ask The Experts

Read questions answered by our experts or join the email list.