• Resolving CRC and Input Errors, A Beginner’s Guide

 
 
Print Friendly, PDF & Email

Introduction

Cyclic Redundancy Checks or CRCs happen at the Datalink Layer and can be caused by software errors, electromagnetic interference or physical layer errors. Physical layer errors are Frame Check Sequence (FCS) which by far are the most common source of CRC errors causing many to believe they are one and the same. Covered in this guide are the initial steps from a cabling technician’s perspective that should be taken to resolve the majority of these errors without the assistance of TAC.

Standardize Your Process

Troubleshooting link issues due to link errors consists of basic steps that local technicians or engineers can do with no supervision from a senior network engineer.  Arista recommends that you create your own internal processes with an accompanying checklist for troubleshooting input errors and CRCs that best suits your work environment.  A defined process for undertaking troubleshooting steps would likely expedite your time to resolution and may prevent any unneeded involvement of outside resources for further guidance.  Here is a suggested framework for which your organization may operate off of.

Background

The following is a scenario that shows a small block of errors to help provide context for this document.  “Input errors and CRCs are seen incrementing on Spine 2, Ethernet3 which is connected to Leaf4, Ethernet6.”  

show interface Ethernet3

Ethernet3 is up, line protocol is up (connected)
  Hardware is Ethernet, address is zzzz.bbbb.cccc (bia aaaa.bbbb.cccc)
  Member of Port-Channel100
  Ethernet MTU 9214 bytes, BW 10000000 kbit
  Full-duplex, 10Gb/s, auto negotiation: off, uni-link: n/a
  Up 19 days, 1 hour, 9 minutes, 11 seconds
  Loopback Mode : None
  4 link status changes since last clear
  Last clearing of "show interface" counters 33 days, 22:35:14 ago
  5 minutes input rate 2.89 kbps (0.0% with framing overhead), 4 packets/sec
  5 minutes output rate 2.84 kbps (0.0% with framing overhead), 4 packets/sec
     13041659 packets input, 1060194902 bytes
     Received 7 broadcasts, 1951327 multicast
     0 runts, 0 giants
     350 input errors, 302 CRC, 0 alignment, 0 symbol, 0 input discards
     0 PAUSE input
     12906625 packets output, 1046720348 bytes
     Sent 118884 broadcasts, 1657550 multicast
     0 output errors, 0 collisions
     0 late collision, 0 deferred, 0 output discards
     0 PAUSE output

 

Trailing the extension counters errors to your show interface command will show you the layer 1 FCS errors coming into the port. 

Show interface Ethernet3 counter errors 
Port               FCS    Align   Symbol       Rx    Runts   Giants       Tx 
Et3                350        0     302        0     0 0        0

 

Viewing show lldp neighbors will help identify the opposite port.  Looking at the stats from the opposite port is also vital as a link is a 2-way communication and those stats may provide you with crucial data.

show lldp neighbor et3
Last table change time   : 0:00:07 ago
Number of table inserts  : 1
Number of table deletes  : 0
Number of table drops    : 0
Number of table age-outs : 0
Port          Neighbor Device ID       Neighbor Port ID    TTL
---------- ------------------------ ---------------------- ---
Et3           Leaf4                    Ethernet6           120

 

Providing a quick and detailed topology to your onsite personnel or other engineers is highly recommended.  This information will allow them to know the impacted devices, ports of interest and allow them to trace cabling as needed.

Active Monitoring

Monitoring the progress of the work from the CLI is made simple with the  watch diff command.  Run watch diff show interface ethernet3 to monitor the interface errors.  Monitoring may be done by the technician via console or remotely.

watch diff show interface ethernet3 | grep errors
Every 2.0s: CliShell -s ar -p 15 -c show interface ...  Sat Nov 13 18:08:10 2021
  762 input errors, 556 CRC, 0 alignment, 0 symbol, 0 input discards
     0 output errors, 0 collisions 

Remediation Steps

  1. Move the cabling to a different port.  
    • If the errors follow the move, then the port you just moved from (Spine2, Ethernet3) is not the problem.  
    • If you move from (Leaf4, Ethernet6) and the errors stop, you proved Leaf4, Ethernet6 is throwing the errors.  Contact TAC immediately.
    • If after these moves you still see errors, the problem is the Layer1 hardware connecting the devices.

      2. Next clean or replace the cabling or optics with spares (if available).  In the case of any copper connections such as Cat5E or Cat6 connections, crimping new ends commonly resolves the issue.

      3. After performing the prior steps and if it is suspected the issue lies with the optics or a fault appears check the following.

    • Show interface mac detail.  If you have a fault that reports “true” or if you see that the “false” is newer than the link was established, this usually means a fault on an optic.  Local fault means the optic on the device you are looking at.  Remote fault means the optics on the opposite side of the link.
show interface et3 mac detail
Current System Time: Mon Nov 22 18:23:57 2021
Ethernet2
Current State Changes Last Change
PHY State linkUp 3 3 days, 23:49:44 ago <-- Link has been up for 3 days, 23:49:44 hours
Interface State linkUp 3 3 days, 23:49:44 ago
MAC Rx Local Fault True 14 0 days, 00:00:06 ago  <-- Potential Fault newer than link uptime
MAC Rx Remote Fault False 4 3 days, 23:49:47 ago   

 

When opening the Service Request, it is recommended to have a brief problem description that helps the TAC Engineer quickly identify the problem you face. 

List any troubleshooting steps taken and provide data points of interest such as show commands or a logged console session of your troubleshooting.

Upload the show techs for both devices (if Arista)

show tech-support | gzip > /mnt/flash/show-tech-$HOSTNAME-$(date +%m_%d.%H%M).log.gz

Related References

https://eos.arista.com/interface-errors-explained/

https://eos.arista.com/forum/what-factors-can-cause-the-link-down-in-the-eos/

Follow

Get every new post on this blog delivered to your Inbox.

Join other followers: