• Buffer tuning for output discard mitigation

 
 
Print Friendly, PDF & Email

Platform : 7050 Series,  7060, 7260 Series and 7304 Series.

This document explains how to mitigate the output discards that are caused on the following platforms, due to congestion.

How to determine if the drops are due to congestion :

  1. If the rate of traffic is nearing / exceeding the link bandwidth, it is pretty easy to understand that the drops are happening due to congestion.
  2. If the traffic rate is well within the interface bandwidth and still the drops are happening, they might be due to microbursts.

( Microbursts are huge spikes in traffic rate, which happen and end within a duration of micro-seconds. These are very difficult to capture using the interface counters, as these counters can be polled only at a minimum interval of one second. So it is very easy to go unnoticed in the interface counter rates )

Microbursts happen due to either the nature of the Application which spikes the traffic or due to fan-in conditions which occur due to multiple ingress interfaces, trying to pump traffic to one egress interface.

A good way to verify if there are microburst happening is by enabling LANZ. It records the queue length in micro-seconds. The following command enables LANZ on the switch ( with the default threshold values )

queue-monitor length

We recommend the default threshold since they are designed suitably for every platform.

Solution to Congestion :

  1. The real fix is to add additional output capacity by either bundling interfaces into port channels or increasing the speed of the outgoing link if possible.
  2. However since the above might not be possible in all cases, these drops can be lessened or eliminated by tuning the buffers on the platform.

Buffering on this platform :

When the egress interface gets busy, the packets get stored into the Memory units that a device has. Before we know how to tune the buffers, we need to understand the buffer allocation on this platform.

This platform has a shared pool of buffer from which all interfaces can borrow, as well as definite amount of buffer that has been reserved for each interface (on a per queue basis). The reserved memory for the port is first used to buffer the traffic and after the egress port has exhausted, it will use buffers from the shared pool.

The amount of shared pool that any interface ( per queue ) can use, is determined by a threshold value ( known as alpha value ). The following example gives a fair idea about the same :

For example, if queue UC1 has alpha value of 1 and UC2 has alpha value of 1/2 and both the queues are already using 100 cells from the shared memory pool. If in the pool, there are 120 cells remaining, UC1 based on the alpha value can take (free cells*alpha) 120 cells maximum at that point. And as it has already used 100 cells, it can take 20 cells more from the shared pool. While UC2 has already exhausted its limit. 

Following command shows the default reserved memory allocated to et6 on a per queue basis ( for the unicast and the multicast queues ). Here each usicast queue has a 1664 amount of reserved bytes and (¼) as the threshold / alpha value.

Ethernet6

  MMU Queues                 Bytes Used Minimum Reserved Bytes    Dynamic Threshold

  ---------------            ---------- ----------------------    -----------------

  Egress Unicast Queue 0     0 1664             1/4
  Egress Unicast Queue 1     0 1664             1/4
  Egress Unicast Queue 2     0 1664             1/4
  Egress Unicast Queue 3     0 1664             1/4
  Egress Unicast Queue 4     0 1664             1/4
  Egress Unicast Queue 5     0 1664             ¼
  Egress Unicast Queue 6     0 1664             1/4
  Egress Unicast Queue 7     0 1664             1/4
  Egress Multicast Queue 0   0 1664             1/4
  Egress Multicast Queue 1   0 1664             1/4
  Egress Multicast Queue 2   0 1664             1/4
  Egress Multicast Queue 3   0 1664             1/4
  Egress Multicast Queue 4   0 1664             1/4
  Egress Multicast Queue 5   0 1664             1/4
  Egress Multicast Queue 6   0 1664             1/4

Tuning Buffers on this platform :

In the cases where there are one or few interfaces which have more traffic rate compared to rest of the interfaces and if there are congestion / drops , we can tune the buffers accordingly to mitigate or reduce the same. This can be done by :

  1. Increasing the reserved buffer for the interface / queue
  2. Increasing the threshold value so that more amount of buffer from the shared pool can be utilised by the affected interface / queue. 

NOTE : However please note that several interfaces / queues can use the shared pool at a time and the threshold value is proportional to the amount of buffer available currently ( which excludes the units consumed by the other interfaces ). So there might not be a guaranteed amount of buffer that we shall get , by tuning the threshold value.

The following shows the configuration commands needed to create a custom profile and modify the parameters for the queue and Alpha value in question.


platform trident mmu queue profile CUSTOM
egress unicast queue 0 reserved 3328
egress unicast queue 0 threshold 2

The profile is then applied to the interface

interface Ethernet6
switchport mode dot1q-tunnel
spanning-tree portfast
platform trident mmu queue interface-profile CUSTOM

We now see that the parameters have been modified.
 Ethernet6

  MMU Queues                 Bytes Used Minimum Reserved Bytes    Dynamic Threshold

  ---------------            ---------- ----------------------    -----------------

  Egress Unicast Queue 0     0 3228             2
  Egress Unicast Queue 1     0 1664             1/4
  Egress Unicast Queue 2     0 1664             1/4
  Egress Unicast Queue 3     0 1664             1/4
  Egress Unicast Queue 4     0 1664             1/4
  Egress Unicast Queue 5     0 1664             1/4
  Egress Unicast Queue 6     0 1664             1/4
  Egress Unicast Queue 7     0 1664             1/4
  Egress Multicast Queue 0   0 1664             1/4
  Egress Multicast Queue 1   0 1664             1/4
  Egress Multicast Queue 2   0 1664             1/4
  Egress Multicast Queue 3   0 1664             1/4
  Egress Multicast Queue 4   0 1664             1/4
  Egress Multicast Queue 5   0 1664             1/4
  Egress Multicast Queue 6   0 1664             1/4

The above can be done only on a trial and error basis and If the drops don’t subside then you can try doubling the number of reserved packets and Alpha value again. However please note that by allocating buffers to an interface on a permanent basis you are stealing them from the global pool and thus are unavailable for other interfaces.

We always recommend you to consult Arista Support before tuning the buffers.

 

Follow

Get every new post on this blog delivered to your Inbox.

Join other followers: