Driving into the real-time arena

Real-time applications based on switched Ethernet are now a viable proposition, as Øyvind Holmeide and Tor Skeie explain

The real-time properties of traditional Ethernet are poor. The non deterministic property of the Carrier Sense Multiple Access - Collision Detection (CSMA/CD) scheme has been the main argument against Ethernet for applications with real-time requirements. However, this has changed significantly with the introduction of Ethernet switches together with the new priority features of Ethernet switches. Real-time applications based on switched Ethernet can take advantage of this technology driven by the fast growing Voice Over IP (VOIP) business.

Deterministic Ethernet is achieved by using priority. Worst case latency for real time critical data through a switched Ethernet infrastructure can be guaranteed if this data is protected by priority. Typical worst case switch latency for priority protected packets are in the order of a few hundred microseconds.

Switch priority handling based on IP ToS is the preferred choice. A switch that is pre-configured for a high priority subset of possible ToS code points does not require any user configuration of the switches. Such switches will in most cases be unmanaged, and the most cost efficient alternative. Configuration of priority based on IP ToS is then a matter of the real-time application. Configuration is easily performed on socket layer for both the client and server. Note also that priority based on IP ToS does not represent any conflict with respect to other switches or stations that do not support this feature. This is not the case for priority based on tagging (IEEE 802.1p) as long as some switches and stations do not support tagging. However, priority tagging may be a relevant alternative in the future, when most switches, Network Interface Cards and operating systems support this feature.

High speed From a functional point of view, switching is exactly the same as bridging. However switches use specially designed ASICs to perform the bridging and packet-forwarding functionality as opposed to implementations using a central CPU and special software, so switches are much faster than bridges.

Ethernet switches provide 10Mbps, 100Mbps or 1Gbps (with 10Gbps (under development) on each drop link. This represents a scalable and huge bandwidth increase compared to, for example, an Ethernet hub where the bandwidth is either 10 or 100 Mbps and shared between all users connected to the same network segment. Ethernet switches also offer both half and full duplex connectivity. This means that an Ethernet controller will never see any collision if full duplex connectivity is used.

However, packets can still be lost if one of the following scenarios appears:

• The total network load exceeds the switching capability of the switch engine - that is, the switch is not able to handle full wire speed on each drop link
• The output buffer capacity is not sufficient - that is, the amount of packets sent to an output port exceeds the bandwidth of this port for a time period that is longer than the output buffer is able to handle. Thus, packets from several input ports compete for the same output port causing a non-deterministic buffering delay.

Lost packets Higher protocol layers at the stations must handle lost packets.

These two scenarios can be avoided by using the following Ethernet techniques:

• Back pressure: the switch can send a jam pattern simulating traffic on a port operating in half duplex mode if the amount of packets received on this port is more than the switch can handle
• Flow control: the switch can send PAUSE packets according to IEEE802.3x on a port operating in full duplex mode if the amount of packets received on this port is more than the switch can handle
• Priority: Ethernet packets that are identified as high priority packets are put in a high priority queue. Packets from a high priority queue are sent before the low priority packets. The low priority packets may still be lost. This is the most relevant technique with respect to optimal real-time properties for latency sensitive real time data.

Ethernet switches today may have support for priority containing two or more output queues per port, where the high priority queue(s) are reserved for real-time critical data offering Quality of Service (QoS). How the switch alternates between the priority queues varies from vendor to vendor. Relevant alternating schemes for a switch with two priority queues could be:

• Round-robin weighting: N packets are sent from the high priority queue before one packet is sent from the low priority queue
• Strict priority: all packets will be transmitted from the high priority queue. Packets from the low priority queue will only be sent in case the high priority queue is empty.

Note that a high priority packet will be delayed due to a low priority packet if the transmission of this packet is started before the high priority packet enters the output port. The high priority packet will then be delayed by the time it takes to flush the rest of the packet. Worst case will be that the transmission of an Ethernet packet with maximum packet length (1518 bytes) is just started. The extra delay will then be 122µs in case of 100Mbps, and 1.22ms in case of 10Mbps.

 

Making Ethernet real-time friendly Traditional ethernet is not real time friendly. The CSMA/CD scheme of Ethernet makes access to the medium non-deterministic. An Ethernet controller connected to a thin Ethernet (coax - 10BASE-2) or a hub (10BASE or 100BASE) is not able to send a packet as long as the medium is busy sending another packet. The Ethernet controller is free to send its packet as soon as the Ethernet is idle. While transmitting, the station continues to listen on the wire to ensure successful communications. If two stations attempt to transmit information at the same time, the transmissions overlap, causing a collision. If a collision occurs, the transmitting station recognises the interference on the network and transmits a bit sequence called jam. The jam helps to ensure that the other transmitting station recognises that a collision has occurred. After a random delay, the stations attempt to retransmit the information and the process is repeated. The probability for a collision depends on the collision domain, i.e. the range of the Ethernet, and the network load. A traditional CSMA/CD Ethernet with 20% utilisation has less than 0.1% collision, while as much as 5% of the packets will experience collisions if the network utilisation is above 40%. A CSMA/CD network with 40% utilisation is in trouble, and the net data rate will in fact decrease due to collisions if the load is further increased. However, bear in mind that those collisions are not errors. Collision is a normal part of Ethernet networks, as shown in the Figures. The figures below show the principles of CSMA/CD.

A high priority packet may also be delayed through the switch due to other real time packets that are already queued for transmission in the same high priority queue. However, it is often a straightforward job to calculate the worst-case switch latency such a packet may experience if the network load and traffic pattern of the real time application using the high priority queues are known, and all other traffic use lower priority. Typical worst-case switch latency for a high priority packet in such a system will be a few hundred microseconds in case of 100Mbps on each drop link.

This worst case latency for the real-time packets is valid regardless of any other network load with less priority.

Several priority implementations exist with respect to how a packet is identified as a high priority packet. The priority handling depends on the switch functionality. One or more of the ports of the switch can be configured for high priority. This means that all packets received on these ports will be treated as high priority packets. Switches that provide this function are in most cases managed. The advantage of this feature is limited, but this feature has been the only priority function available up to now.

Priority tagging The IEEE 802.1p (and IEEE 802.1Q) standard specifies an extra field for the Ethernet MAC header. This field is called Tag Control Info (TCI) field, and is inserted between the source MAC address and the MAC Type/Length field of an Ethernet packet (see Figure 1). This field contains a 3-bit priority field that is used for priority handling. Thus, the standard defines eight different levels of priority. However, most Ethernet switches available on the market that support priority queuing have only two or four queues. A switch with two priority queues will put Ethernet packets with priority tags set to four or higher in the high priority queue while all other packets will be put in the low priority queue. Both unmanaged and managed switches can support this feature. Thus, no switch configuration is needed.

Tag removal A disadvantage with this method is that most stations up to now do not support priority tagging. Configuring the switch to remove the tags after switching can solve this, and before the packets are sent on the output ports where stations without support for this feature are connected. This requires managed switch operation. Another problem could be that there exist other Ethernet switches in the network that do not support priority tagging. Then, the maximum packet size will, owing to the tag, increase by four bytes to 1522 bytes, and some switches will not forward packets with a packet length larger than 1518 bytes.

A layer 3 switch can perform switching based on both the Ethernet MAC destination addresses and the layer 3 contents (router functionality). A layer 3 switch may provide priority identification based on the same criteria as a layer 2 switch. The layer 3 IP type of service (ToS) field is also relevant (Figure 2).

A layer 4 switch can perform switching based on both the Ethernet MAC destination addresses and the layer 3 and layer 4 contents, the IP and TCP headers of TCP packets. A layer 4 switch may provide priority identification based on the same criteria as a layer 3 switch. The destination port of a UDP or a TCP header can also be used in the switch as a high priority criterion.

Figure 3 shows this field for the UDP header. Most layer 4 switches are and will be managed. Thus, this priority function requires switch configuration by the user. Each switch between the client and server in a real-time application must be configured for the chosen socket port number of the server. However, the user should bear in mind that the socket port number of the client would be different from the corresponding server socket port number, that is the destination port and source port number in a UDP packet will be different. The switches should be configured for both port numbers if real-time critical data also is sent from the server to the client. This could be a problem because the client socket port number in most cases is dynamically allocated. INOC

• OnTime Networks
  d144@industrialnetworking.co.uk
  
  
• ABB
  d145@industrialnetworking.co.uk