• Platform Specific Discards

 
 
Print Friendly, PDF & Email

Objective

This document is a guide to understand “input/output” discards (or congestion related drops) on various platforms and how to troubleshoot them.

Introduction

The term “Discards” in the world of Arista Networks refers to packets being dropped due to congestion- either on CPU level or interface level. Any other kind of drops , like drops due to- “Vlan not being allowed”, “route lookup failure”, “Incorrect Hardware state”, etc are not considered as “Discards” and are mostly considered as “Packet Processor” drops.

Platforms to be covered

7050, 7060, 7260 and 7304 series
7280R, 7020R, 7500R series
7150

What is an Output Discard ?

Every platform has interfaces which can either receive or transmit traffic. For transmitting traffic, every platform has some dedicated buffer for each interface. That buffer will hold the traffic momentarily before it egresses out. The buffer (per interface) is further divided between multiple Traffic classes. Traffic class defines the priority to be given to different packets based on their CoS or DSCP value.
If there is high amount of traffic in any traffic class such that the buffer gets full and further incoming packets cannot be accomodated, then the packet is dropped as “Output discard”

Congestion can happen if there is a fan-in fan-out situation: multiple interfaces receive the traffic and only one egress interface is present.

A good article on congestion- https://eos.arista.com/how-to-troubleshoot-congestion/

In general, if you are observing congestion on any interface, it is recommended to increase the bandwidth of the interface (where you see the discards) or make a port-channel with multiple interfaces as members of the port-channel.

7050, 7060, 7260 and 7304 series

In this series, If data packets are dropped due to buffer getting full on the egress interface, then the output interface(from where packet was supposed to leave the switch) discard counter will increase which would provide load sharing.

For example:

Sender——Eth1/1[Sw1]Eth1/2——Receiver

If packets arriving on Eth1/1 are getting dropped because the buffer on Eth1/2 is full, the discard counter would increase corresponding to Eth1/2. Below counter shows 100 packets were dropped due to congestion on egress buffer of Eth1/2

A queue can also get congested due to Starvation– This happens when there is a very high amount of traffic in a higher traffic class (say TC5) such that traffic in lower traffic class (say TC1) is not getting served. By default, the switch follows “strict priority queue” where the higher priority class is served first. In that case, packets in lower TC (TC1) will keep on getting accumulated and eventually the lower TC will be full leading to drops (both inDiscards and OutDiscards will increase).

To check in which queue packets are dropping, you can use the command-

UC1 maps to TC-1, same goes for the rest of the UC queue as well (UC2-TC2, UC3-TC3 and so on)

In case if there is congestion on CPU causing drops of control plane packets (like LLDP, OSPF, BGP, ARP, etc) then you need to check the “show cpu counters queue | nz” output.

Here is an article that can help you deal with data plane congestion for this series:
https://eos.arista.com/buffer-tuning-for-output-discard-mitigation/

7150

This is the only platform in this series and is known for its “low latency hardware forwarding” which means lesser buffer to hold packets in egress interface compared to other platforms. Remember- Lower the latency, lesser is the buffer.

In this platform, If data packets are dropped due to buffer getting full on the egress interface, then both the input interface (from where packet ingressed the switch) and output interface(from where packet was supposed to leave the switch) discard counter will increase . This is seen only on this platform.

For example:

Sender——Eth1/1[Sw1]Eth1/2——Receiver

If packets arriving on Eth1/1 are getting dropped because the buffer on Eth1/2 is full, then the “input discard” would increment for the ingress interface (where packet entered the switch) and output discard would increment for the Egress interface (from where packet was supposed to exit the switch). Below counter shows 100 packets were dropped due to congestion on Eth1/2

If in case, you are observing that only the “input discard” is incrementing for any given port and don’t see the increment of “output discard” for any other port, then it means- packets that are ingressing on that interface, is going to CPU and causing the CPU to congest.

For example:

Sender——Eth1/1[Sw1]Eth1/2——Receiver

Say Eth1/1 received 100 packets and you see all the 100 packets are dropped as input discards (or may be some packets dropped out of 100, depending on how much is the congestion) :

Then it means the 100 packets were supposed to reach the CPU. But due to CPU congestion, it got dropped.

NOTE: By default all control plane packets would go to CPU. However, in 7150 → In addition to all control plane packets, packets that need to be Vxlan Flooded also go to CPU as 7150 can do Vxlan Flooding only at Software level.
So if you are seeing high Input discards on an interface in 7150, and if Vxlan is configured on it, then it most likely that the packets that need to be flooded in Software are getting dropped.

Unfortunately we do not have the “show cpu counters queue” to check the packet count for different control plane queues (BGP, LLDP, LACP, etc).

However, you can use the below command to check if your CPU path is congested or not:

^ If any of the above counter is incrementing, then the CPU path is congested and can impact any control plane traffic. This will increment in conjunction with “Input Discards”

See the following article for more details about congestion on this platform:

https://eos.arista.com/7150-discards-troubleshooting/

7280R, 7020R, 7500R series

The series is also known as “Deep Buffer” switches as it can handle large amounts of traffic, microbursts quite easily compared to other platforms- thanks to a feature “Virtual Output Queue” or VoQ.

There are two types of buffer in this series- Ingress and Egress. Ingress Buffer (also called DRAM buffer) are on-chip buffer per ASIC and has a size of 4GB which is divided among various Unicast and Multicast classes. Egress buffers are small (about 4Mb) and are available per interface level- divided between various Unicast and Multicast classes.

VoQ (Virtual Output Queue)

VoQ is a Virtual Queue for the egress pipeline- It means when a Unicast Packet ingresses a switch, instead of directly sending it to the egress buffer (of the egress interface), it is first stored in the ingress buffer (which is an on-chip DRAM) in its respective queue (queue is based on the DSCP or CoS value). Only when the egress packet scheduler signals the ingress buffer (via a credit system) that it has enough space to store the newly arrived packet, the packet is sent to the egress buffer. This ensures that the egress buffer is never congested and hence we ideally do not see any Output discards for these platforms.

However, the VoQ system is specially designed for Unicast traffic. By default Multicast traffic does not use the ingress buffer and is sent directly to the egress. This is because customers expect least latency for Multicast traffic. Queueing the Multicast traffic in ingress buffer will add latency to the flow. That means there is a chance that Multicast packets can get dropped due to congestion on the egress buffer.

You can always change the replication mode of Multicast traffic from Egress (which is default) to Ingress using the command

“platform sand multicast replication default ingress”

If you are seeing Multicast packets drops (output discards), you can use the above command after which you should stop seeing the drops of Multicast packets.

NOTE: In general, you should not see Unicast packets being discarded in this series of platforms. However, if you do see, please reach out to Arista TAC.

VoQ Packet Delete

1. As discussed about VoQ earlier, when a unicast packet ingress the switch it is first stored in the ingress buffer until it receives a signal or credit from the egress buffer. But the question is how long the packet is kept in the ingress buffer ?
The VoQ will store the packet for 500ms until it gets a credit from the egress buffer. The egress buffer may not send a credit if it cannot accommodate any more packet. After 500ms, the packet is deleted from the VoQ and the “DeqDelPktCnt” counter increments under the command- “show hardware counter drop”

Every packet is given a time of 500ms before it is deleted from VoQ. The time is normally sufficient for Egress buffer to create some space for the new packet. Since other platforms do not have VoQ, packets are directly dropped.

Useful article to troubleshoot DeqDeletePktCnt:

VOQ Delete Monitoring

Commands to check for congestion drops :

1. “show interfaces counters discards | nz”

This command is useful to check if there are any drops in the Egress buffer of any interface. Usually the drops you would see would be because of Multicast traffic.

2. “show interfaces counters queue drops | nz”

This command will give you more details about the drops. For example:

If you see the above output, it shows 7042 Multicast packets were dropped due to congestion of Egress multicast buffers of Et23/1.

You can also see 669664 packets were enqueued on Jericho1 on-chip buffer (where the packet arrived) which is the ingress buffer. All those packets egressed out of Et23/1 (no drops is shown here). That is because the Unicast egress buffer of Et23/1 was never congested and hence it was always granting the credits to the ingress buffer. If it was congested, some packets would have dropped in VoQ.

NOTE: Congestion can occur in CPU level as well. Please check the below article related to CPU congestion for this platform series:

Troubleshooting based on Control Plane Policing (CoPP) for 7500R, 7280R, 7020R Platforms

Follow

Get every new post on this blog delivered to your Inbox.

Join other followers: