• Using tcpdump for Troubleshooting

 
 
Print Friendly, PDF & Email

What is tcpdump?

tcpdump is a command line packet sniffer (built into Linux) that is used to assist in troubleshooting network problems.

Any traffic coming to or from the control plane of the Arista is visible when running the tcpdump utility on the Arista.  This does not include data plane traffic transiting the Arista switch.  For capturing data plane traffic, Arista supports monitoring/SPAN ports which copies hardware-forwarded traffic to a sniffer or to other suitable capture device for analysis.  You can also mirror to the CPU depending on platform and EOS image (https://eos.arista.com/eos-4-24-0f/mirroring-to-cpu-on-7050-7060-7260-7368-7300-and-720xp-series/).

 

How do I use tcpdump to inspect control plane traffic?

tcpdump allows users to instantly analyze important traffic such as Spanning Tree and routing protocols, as well as any other traffic that is destined to the switch itself (via an SVI or management IP address). In order to make full use of it, follow the directions below:

 

 1.  The first step is to access the BASH shell:

Arista#bash 
Arista Networks EOS shell 
[admin@Arista ~]$

Once in enable mode, all that is required is to type bash and you will be presented with a bash shell prompt.  At this point you have initiated a bash shell and any commands you type will be directly within Linux and not within the EOS CLI.

2.  The next step is find the appropriate interface that you wish to monitor:

The Linux ifconfig command can be used to see the available interfaces in the Linux kernel.

[admin@Arista~]$ifconfig

cpu      Link encap:Ethernet   HWaddr 02:00:00:00:00:00
UP BROADCAST RUNNING MULTICAST MTU:9216 Metric:1
RX packets:19 errors:0 dropped:0 overruns:0 frame:0
TX packets:372 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:1140 (1.1 KiB) TX bytes:26802 (26.1 KiB)

et1      Link encap:Ethernet   HWaddr 02:00:00:00:00:00
UP BROADCAST RUNNING MULTICAST MTU:9216 Metric:1
RX packets:4 errors:0 dropped:0 overruns:0 frame:0
TX packets:47891 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:240 (240.0 b) TX bytes:3683975 (3.5 MiB)

et2      Link encap:Ethernet   HWaddr 02:00:00:00:00:00
UP BROADCAST RUNNING MULTICAST MTU:9216 Metric:1
RX packets:3 errors:0 dropped:0 overruns:0 frame:0
TX packets:1125577 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:180 (180.0 b) TX bytes:86586365 (82.5 MiB)
...

ma1      Link encap:Ethernet   HWaddr 00:1C:73:0B:1D:13
UP BROADCAST MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 b)TX bytes:0 (0.0 b)
Interrupt:31 Base address:0×2000

ma2      Link encap:Ethernet   HWaddr 00:1C:73:0B:1D:14
UP BROADCAST MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 b) TX bytes:0 (0.0 b)
Interrupt:30 Base address:0×6000

vlan1026      Link encap:Ethernet   HWaddr 00:1C:73:0B:1D:15
inet addr:172.22.26.1 Bcast:255.255.255.255 Mask:255.255.254.0
UP BROADCAST RUNNING MULTICAST MTU:9212 Metric:1
RX packets:1614852 errors:0 dropped:0 overruns:0 frame:0
TX packets:386542 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:721138049 (687.7 MiB) TX bytes:42279526 (40.3 MiB)

The list of interfaces will reflect each of the physical interfaces on the Arista switch as well as any virtual interfaces.  Any VLAN which has been assigned an IP address (SVI) will show up as a VLAN interface.  The Management interfaces show up as ma1/ma2 etc …  The interface name shown in the left-most column represents the Linux interface that we will use with the tcpdump utility.

3.  And here is an example of how to run tcpdump for a VLAN interface.

[admin@Arista ~]$tcpdump -i vlan1026
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on vlan1026, link-type EN10MB (Ethernet), capture size 65535 bytes

The ‘-v’ and ‘-vv’ flags can be used to provide more detailed output.

[admin@Arista ~]$tcpdump -i vlan1026 –v –vv

4.  Once you know the Linux name associated with a given interface, you can also call the tcpdump command directly from the EOS CLI without having to drop into a BASH shell first.  Note that this does change the syntax used.

 Arista#bash tcpdump -i et12 -v -vv

5.  Another common option includes filtering more specific traffic. This can include traffic on a specific destination port number.

Arista#bash tcpdump –n dst port 23 -i et12 -v –vv

6.  Alternatively, if you want to omit some specific traffic, that works too.

Arista#bash tcpdump -i mirror0  not ether proto 0x8809 and not ether proto 0x88cc and not ether dst 01:80:c2:00:00:00

7.  It is also common to redirect the output from a tcpdump command to a file.

[admin@Arista ] tcpdump -n dst port 80 -i vlan1026 -v > /mnt/flash/dump.txt

Note: It is important to ensure that the filesystem has enough space available and it is always a good idea to eventually remove capture files from the Arista in order to to ensure that space is not needlessly being consumed (which could eventually lead to a space shortage).

 

Common tcpdump examples

1.  How do I see LLDP packets coming from an interface? (You can run a tcpdump in or out of config mode.)

Arista(config)#bash tcpdump -nevvi et1 ether dst host 01:80:c2:00:00:0e

2.  My server is port-channeled to my Arista switch, and it is not coming up.  How do I capture LACP packets coming to and from the server to form the port-channel?

Arista(config)#bash tcpdump -nevvi et1 ether dst host 01:80:c2:00:00:0e

3.  I am trying to ping an Arista switch, but it is not responding. How do I know whether it is receiving the ping packets or not?

Arista(config)#bash tcpdump -nevvi ma1 icmp

4.  How do I capture packets arriving from a specific host?

Arista(config)#bash tcpdump -nevvvi any host 172.22.26.209

5.  How do I capture packets arriving on specific port and from a specific host?

Arista(config)#bash tcpdump -nevvvi any '((port 22) and (host 172.22.26.209))'

6.  How do I redirect tcpdump output to flash so that I can email it to someone?

Arista(config)#bash tcpdump -nevvvi any port 22 > /mnt/flash/tcpdump-test

7.  How do I troubleshoot OSPF routing LSA updates between neighbors?

Arista(config)#bash tcpdump -nevvvi et37 proto ospf

More tcpdump command-line options

-i any        : Listen on all interfaces just to see if you’re seeing any traffic.
-n            : Don’t resolve hostnames.
-e            : Get the ethernet header as well.
-v, -vv, -vvv : Increase the amount of packet information you get back.
-nn           : Don’t resolve hostnames or port names.
-X            : Show the packet’s contents in both hex and ASCII.
-XX           : Same as -X, but also shows the ethernet header.
-c            : Only get x number of packets and then stop.
-S            : Print absolute sequence numbers.
-q            : Show less protocol information.
-E            : Decrypt IPSEC traffic by providing an encryption key.
-s            : Set the snaplength, i.e. the amount of data that is being captured (in bytes)

Capturing packets with VLAN tags

Tcpdump has an unintuitive interface when filtering for VLAN tags. The critical piece of information to understand is that the position of the word ‘vlan’ in the tcpdump filter is paramount when it comes to tcpdump filter compilation. The ‘vlan’ keyword in a tcpdump filter changes the lookup offsets for all other keywords following the keyword.  This behavior is irrespective of parentheses.  For example, the following two tcpdump examples are not equivalent:

tcpdump -i interface '(vlan and arp)' or arp 
tcpdump -i interface arp or '(vlan and arp)

The first example compiles such that the offsets for both ‘arp’ filters are offset by the vlan keyword, even though the second ‘arp’ keyword is outside the parentheses.  The second example compiles as intended, where the first ‘arp’ keyword looks at the ‘normal’ offsets, the second arp keyword looks for the arp protocol after the VLAN header offsets.

It’s important to bear in mind that this constraint is well known and it’s possible that one day the behavior will change. Therefore, filters should be constructed logically so that they will compile successfully for both an intuitive compilation and the current unintuitive compilation.

 

That’s all very technical, so what does it mean? We’ve compiled some examples to help compare where and when to use the vlan filter :

Filtering for a specific host 

1. Without VLAN tags:

tcpdump -i interface host 192.168.150.50

2. With VLAN tags:

tcpdump -i interface vlan and host 192.168.150.50

Filtering for ‘arp or icmp’

1. Without VLAN tags:

tcpdump -i interface arp or icmp

2. With VLAN tags:

tcpdump -i interface vlan and '(arp or icmp)'

2. With or without VLAN tags:

tcpdump -i interface '(arp or icmp) or (vlan and (arp or icmp))'

Filtering STP/PVST BPDUs

1. Without vlan tags i.e. RSTP/MSTP/untagged native vlan in Rapid PVST:

se510#tcpdump interface e23 packet-count 1 filter stp
08:24:30.721696 00:1c:73:13:8a:d1 (oui Arista Networks) > 01:80:c2:00:00:00 (oui Unknown), 802.3, length 53: LLC, dsap STP (0x42) Individual, ssap STP (0x42) Command, ctrl 0x03: STP 802.1w, Rapid STP, Flags [Learn, Forward, Agreement], bridge-id 8001.00:1c:73:13:8a:ba.8017, length 36

2. With vlan tags i.e. tagged vlan BPDU’s in Rapid PVST

se510#tcpdump interface e23 packet-count 2 filter vlan and ether dst 01:00:0c:cc:cc:cd
08:26:09.121966 00:1c:73:13:8a:d1 (oui Arista Networks) > 01:00:0c:cc:cc:cd (oui Unknown), ethertype 802.1Q (0x8100), length 68: vlan 20, p 7, LLC, dsap SNAP (0xaa) Individual, ssap SNAP (0xaa) Command, ctrl 0x03: oui Cisco (0x00000c), pid PVST (0x010b): STP 802.1w, Rapid STP, Flags [Learn, Forward, Agreement], bridge-id 8014.00:1c:73:13:8a:ba.8017, length 42
08:26:09.163805 00:1c:73:13:a3:e7 (oui Arista Networks) > 01:00:0c:cc:cc:cd (oui Unknown), ethertype 802.1Q (0x8100), length 68: vlan 10, p 7, LLC, dsap SNAP (0xaa) Individual, ssap SNAP (0xaa) Command, ctrl 0x03: oui Cisco (0x00000c), pid PVST (0x010b): STP 802.1w, Rapid STP, Flags [Learn, Forward, Agreement], bridge-id 100a.00:1c:73:13:a3:d0.8017, length 42

3. Both:

se510#tcpdump interface e23 packet-count 3 filter stp or (vlan and ether dst 01:00:0c:cc:cc:cd)
08:30:18.065141 00:1c:73:13:a3:e7 (oui Arista Networks) > 01:00:0c:cc:cc:cd (oui Unknown), ethertype 802.1Q (0x8100), length 68: vlan 10, p 7, LLC, dsap SNAP (0xaa) Individual, ssap SNAP (0xaa) Command, ctrl 0x03: oui Cisco (0x00000c), pid PVST (0x010b): STP 802.1w, Rapid STP, Flags [Learn, Forward, Agreement], bridge-id 100a.00:1c:73:13:a3:d0.8017, length 42
08:30:18.071695 00:1c:73:13:8a:d1 (oui Arista Networks) > 01:80:c2:00:00:00 (oui Unknown), 802.3, length 53: LLC, dsap STP (0x42) Individual, ssap STP (0x42) Command, ctrl 0x03: STP 802.1w, Rapid STP, Flags [Learn, Forward, Agreement], bridge-id 8001.00:1c:73:13:8a:ba.8017, length 36 
08:30:18.071808 00:1c:73:13:8a:d1 (oui Arista Networks) > 01:00:0c:cc:cc:cd (oui Unknown), ethertype 802.1Q (0x8100), length 68: vlan 20, p 7, LLC, dsap SNAP (0xaa) Individual, ssap SNAP (0xaa) Command, ctrl 0x03: oui Cisco (0x00000c), pid PVST (0x010b): STP 802.1w, Rapid STP, Flags [Learn, Forward, Agreement], bridge-id 8014.00:1c:73:13:8a:ba.8017, length 42

Filtering for ethernet addresses

1. # Include a src ethernet address (substitute src with dst for destination macs)

tcpdump -ei <interface> ether src 00:25:90:32:ec:2a

2. # Exclude a src ethernet address, note the ‘not’ boolean operator (substitute src with dst for destination mac’s)

tcpdump -ei <interface> not ether src 00:25:90:32:ec:2a

Common tcpdump questions

1.  When using mirror-to-cpu to inspect data plane traffic, some of the packets in the data-stream are missing, but I’m not seeing retransmits, what’s going on?

COPP protects the CPU from being overwhelmed. By default, we allow 1gb/s of traffic to be punted to the CPU and anything in excess of this will be dropped. Use the various filters available to keep the traffic being punted to the CPU at a reasonable level. More details here: https://eos.arista.com/eos-4-25-1f/advanced-mirroring-features/

2.  Are there other ways to filter what I see on my screen?

Since EOS is built on top of Linux, we have a standard array of Linux tools available; this includes grep. You can issue a tcpdump command with minimal filters and use grep to look for, or omit, lines with the patterns you specify. Use the grep manual to review the available options and flags.  When you use grep, notice you will only see the particle line and not the entire packet.

Arista(config)#bash tcpdump -nevvvi any | grep '172.22.26.209'

References and other related reading:

  1. https://eos.arista.com/taking-packet-captures-on-arista-devices/
  2. https://eos.arista.com/introduction-to-port-mirroring/
  3. https://eos.arista.com/eos-4-24-0f/mirroring-to-cpu-on-7050-7060-7260-7368-7300-and-720xp-series/
  4. https://eos.arista.com/eos-4-25-1f/advanced-mirroring-features/
  5. https://eos.arista.com/forward-tcpdump-to-wireshark/
  6. https://eos.arista.com/tcpdump-on-an-arista-switch-and-redirect-or-send-output-via-email-scp-and-tftp/
  7. https://eos.arista.com/introduction-to-managing-eos-devices-troubleshooting/#361Configuring_mirroring_to_the_CPU
  8. https://eos.arista.com/eos-4-15-2f/mirror-cpu/
Follow

Get every new post on this blog delivered to your Inbox.

Join other followers: