• Troubleshooting Congestion

 
 
Print Friendly, PDF & Email

What is Congestion

Congestion happens when a device receives more traffic than it can buffer or send out of an egress port. Whenever the network is congested, we can observe symptoms such as packet loss, slowness, low throughput and high latency.

 

Expected Behavior of the switch

When a switch receives a packet, it takes an action to either forward or drop it. If the switch decides to forward the packet, it stores this packet in its buffer, where this packet gets queued to be sent to the egress port. Eventually, this packet gets sent out of the egress port.

As seen in the below diagram, three packets are received on Ethernet1 port. These packets get stored in the internal buffer and then sent out of Ethernet2.

Ideally, we would want all the packets that ingress the switch to be egressed out of the switch.

 

Causes of Congestion 

  • Fan-In
  • Microburst (Bursty traffic)
  • Buffering capacity of the networking devices

Fan-In

 

In the above diagram, let’s assume all the three ports Et1, Et2 and Et3 are 10G ports. Suppose Et1 and Et2 are continuously receiving line-rate traffic which has to egress out of Et3. In an ideal world, we would want all the packets incoming on the two ports to go out of Et3 as shown in the diagram above.

However in reality, Et3 can only send packets at a line rate of 10G. So, the additional incoming packets on Et1 and Et2 would be buffered by the switch and queued to egress port Et3 as shown below.

 

A switch has limited resources to buffer the packets, hence any additional packets will be dropped by the switch and reported as ‘Discards’.

In this scenario, the congestion is caused due to ‘Fan in’ condition where multiple ingress interfaces are receiving packets that need to exit out of a single egress interface and egress interface does not have the capacity to handle that amount of traffic. 

Microburst

Network traffic is often measured using the average utilization of a link – 5 minute input or output rate, expressed in Mbps or Gbps. These averages are taken over the interval of 5 minutes or even 1 second and usually show a smooth traffic pattern. The actual traffic in a network, when viewed at a finer granularity, such as every millisecond, is far more bursty. The monitoring tools do not show these bursts as they capture the average link utilization.

Microbursts are short spikes in network traffic. This spike can result in queuing of the packets. If a switch receives continuous bursty traffic, the buffer space to queue the packets can get exhausted leading to packet drops or discards.

For example, lets consider a scenario where the ingress port is 40G and egress port is 10G as shown in the diagram below. As Ethernet1 has 40G bandwidth, it is possible that the traffic coming in on this interface has bursts of 40Gbps. Now, the egress interface Ethernet2 is 10G , this interface can send the packets at a maximum 10Gbps, hence the traffic bursts coming in on Ethernet1 will need to be buffered and queued before sending them out of Ethernet2. Assume Ethernet1 receives continuous bursts of 40Gbps. In this condition, the egress port will get congested easily.

Both Fan-In and Microburst tie to the buffering capacity of the switch. Having sufficient buffers reduces the probability of a packet being dropped due to congestion.

Buffering capacity of the networking devices

Any networking device will have limited buffer space to store the packets. Arista has both shallow buffer and deep buffer switching platforms.

While the buffering resources on deep buffered platforms are more than that of shallow buffered ones if a switch receives more packets than its capacity we would still see packet drops.

 

Detecting Congestion on Arista switches using LANZ

LANZ tracks interface congestion and queuing latency and provides real-time reporting. It monitors output queue lengths to provide congestion information for individual interfaces. This feature gives us granularity in the traffic pattern ie. helps catch bursty traffic.

Enabling LANZ and viewing stats

1. Enable LANZ using the below command –

switch(config)#queue-monitor length

Note –

a. By default, for the 7280, 7280R, 7500E and 7500R platforms, LANZ will be enabled in polling mode. On supported EOS versions, notifying mode can be enabled using the following :

switch(config)#queue-monitor length notifying

Please refer to the following article for enabling LANZ on 7500R, 7280R and 7020R series switches – https://eos.arista.com/eos-4-15-0f/lanz/

b. Enabling LANZ does not impact switch functionality.

 

2. To view LANZ related events :

switch#show queue-monitor length

More information about LANZ can be found at https://www.arista.com/en/um-eos/eos-latency-analyzer-lanz

 

Methods to prevent congestion

  • QoS
  • Flow Control / Priority Flow Control
  • Buffer tuning
  • Increasing Bandwidth of the interface

QoS

QoS can be used to classify traffic and provide guaranteed bandwidth for a queue.

By default, the interfaces follow strict priority queuing meaning the traffic class with higher priority will be serviced first. If the switch receives a large amount of higher class traffic, it can cause starvation of lower-class traffic. In such cases, QoS can be configured to perform round-robin to prevent starvation.

The following document can be referred for configuration guidelines on different Arista platforms – QOS Configuration Guide

Flow Control / Priority Flow Control

Flow Control mechanism sends Ethernet pause frames to prevent congestion. The node that receives the Ethernet pause frame then pauses sending traffic to the requesting peers. This pause gives time for the originator of the pause frame to process traffic. If flow-control is configured on a switch, it can send a pause frame when experiencing congestion on an egress port. However, please be careful while implementing this solution as this can cause throttling and could negatively impact the network.

Priority flow control (PFC) enables flow control mechanism that can be defined per traffic class. Following is the guide to configure PFC – https://www.arista.com/en/um-eos/eos-section-17-4-configuring-priority-flow-control-pfc .

Note – If flow control is used, please use it end to end.

Buffer tuning

This method allocates more buffers to the congested queue. As the buffer resources available on the switch is limited, this method can be helpful to relieve some congestion. If the interface or a particular queue is heavily congested, we may continue to see drops/discards in spite of increasing the buffer size.

The configuration to tune the buffers is platform dependent.

For example, the following methods can be used to tune the buffers on  7050, 7060, 7260 and 7304 series switches – https://eos.arista.com/buffer-tuning-for-output-discard-mitigation/

Increasing Bandwidth of the interface

Increasing bandwidth is another method to prevent congestion. Basically, by increasing the bandwidth, we increase the total capacity of the egress link allowing it to absorb the incoming traffic rate.

This can be achieved by either bundling multiple ports on existing chassis (LACP) or switching to a port that supports higher-speed links.

Note – Bundling ports do not change the speed of each member interface. The packets could still egress out of a single member interface in the bundle based on the hashing mechanism of the port-channel and the entropy of the traffic.

Follow

Get every new post on this blog delivered to your Inbox.

Join other followers: