• Discovering the Path of the Packet

 
 
Print Friendly, PDF & Email

Introduction

Discovering the path in which a packet flows or Path of the Packet is a critical step in troubleshooting an issue that involves multiple devices. Your debugging scope gets narrowed by using Path of the Packet to what is relevant.  Meaningful logs, packet captures, and other debugging tools are now possible. This article covers some practical uses of fundamental show commands to help you discover the path traffic takes.

Background

Discovering the Path of the Packet has many use cases. Packet loss, DHCP issues, MTU failures, and random disconnects between two end hosts. While some engineers do this natively, other engineers may not have seen these commands used in such an application. Path of the Packet eliminates potentially dozens of devices and hundreds of ports while quickly narrowing the scope to the relevant ones. What if Host A is having random application disconnects to Server 1 and Host A is connected to Leaf4 and Sever1 is connected to Leaf7?  Why start on Leaf1 or Leaf2?  Do not hesitate to confirm your findings. Looking at the wrong port, or worse, being on the wrong device will only prolong troubleshooting.

To help understand this process we will start by looking at what is a nebulous topology diagram and what is very useful.  Next, we will look at your primary show commands to help identify the useful topology information. Last we will look at an example of how to put all of this to use.

For this document, an example of a DHCP client who is sometimes able to obtain a DHCP address while other times attempts to obtain an address fails. Leaf 12B is the Client’s gateway with “ip helper-address” configured.

For details on seeing steps in troubleshooting other common issues, please reference your Arista Operational Runbook. If you are unable to locate a copy, please contact your Account Team for another copy.

Relevant Topology Defined

Your end goal should be to turn a nebulous diagram that conveys more of a broad concept into a practical topology to help you narrow the scope of troubleshooting an issue that involves multiple devices.  Unfortunately, most diagrams do little to help anyone understand which devices and interfaces are important.  By following the Path of the Packet, you can refine your topology into something meaningful, bringing you closer to a resolution.  Notice the differences in the diagrams below.  While the first does nothing to help, the second clarifies the relevant device names and each port ID.

Nebulous

 

Useful

 

Where to Start

Begin your discovery of the traffic path by knowing your source and destination. Listing both IP and MAC addresses (when possible) is essential. Once you have that list, you can use some basic commands to define the path.

  • Log in to the default gateway of the source and resolve ARP to discover the host MAC
switch#show ip arp 10.0.2.2
Address Age (sec) Hardware Addr Interface
10.0.2.2 0:00:25  00ac.00ac.27e4 Vlan2, Ethernet2
  • Should the client be attached to another downstream L2 device, follow the client’s MAC to confirm the port ID.
switch#show mac address address 00ac.00ac.27e4
          Mac Address Table
------------------------------------------------------------------
Vlan    Mac Address       Type        Ports      Moves   Last Move
----    -----------       ----        -----      -----   ---------
   2    00ac.00ac.27e4    DYNAMIC     Et2        1       0:00:40 ago
  • Follow Layer 3 to the destination 10.0.3.1 with “show ip route“.  
switch#show ip route 10.0.3.1
VRF: default
Codes: C - connected, S - static, K - kernel,
       O - OSPF, IA - OSPF inter area, E1 - OSPF external type 1,
       E2 - OSPF external type 2, N1 - OSPF NSSA external type 1,
       N2 - OSPF NSSA external type2, B - BGP, B I - iBGP, B E - eBGP,
       R - RIP, I L1 - IS-IS level 1, I L2 - IS-IS level 2,
       O3 - OSPFv3, A B - BGP Aggregate, A O - OSPF Summary,
       NG - Nexthop Group Static Route, V - VXLAN Control Service,
       DH - DHCP client installed default route, M - Martian,
       DP - Dynamic Policy Route, L - VRF Leaked,
       RC - Route Cache Route

 C 10.0.3.0/31 is directly connected, Ethernet2
  • Validate your return path and never assume they are the same. Port-Channels and ECMP may hash the return traffic through a different devices and ports. If ECMP is a factor at L3, for Strata platforms this guide will help determine which port the traffic is hashing to and this functionality does not apply to Sand platforms.  
  • Follow this path until you find a directly connected route or reach your network edge. If it is a directly connected route, utilize your ARP and MAC tables as above to find the end device.

Other Useful Commands

  • Show interface port-channel x 
    • In EOS, the show port-channel will show you the active ports. You will not have to run additional commands to discover the L1 links.
show interface Port-channel10
Port-Channel10 is up, line protocol is up (connected)
  Hardware is Port-Channel, address is 50ac.0001.0002
  Ethernet MTU 9214 bytes
  Full-duplex, Unconfigured
  Active members in this channel: 3
  ... Ethernet3 , Full-duplex, Unconfigured
  ... Ethernet4 , Full-duplex, Unconfigured
  ... Ethernet5 , Full-duplex, Unconfigured
  Fallback mode is: off
  Up 7 minutes, 2 seconds
  2 link status changes since last clear
  Last clearing of "show interface" counters never
  5 minutes input rate 523 bps (- with framing overhead), 1 packets/sec
  5 minutes output rate 145 bps (- with framing overhead), 0 packets/sec
     306 packets input, 37047 bytes
     Received 0 broadcasts, 306 multicast
     0 input errors, 0 input discards
     96 packets output, 10873 bytes
     Sent 0 broadcasts, 96 multicast
     0 output errors, 0 output discards
  • Using the grep function will also enable you to see the active port-channel links.
show port-channel summary | grep Po1

   Po1(U) LACP(a)     Et3(PG+) Et4(PG+) Et5(PG+)
  • LLDP will give you both the hostname and connecting port ID to help define your topology.
show lldp neighbor ethernet3
Last table change time : 0:09:45 ago
Number of table inserts : 4
Number of table deletes : 0
Number of table drops : 0
Number of table age-outs : 0

Port Neighbor Device ID Neighbor Port ID TTL
---------- ------------------------ ---------------------- ---
Et3           Spine 1 Ethernet1           120

Putting This Into Practice

With the client not always receiving an IP from the DHCP server, starting on the client side to see if the Discover packet is sent is a logical place to start. Here are just two examples of a list of tools for troubleshooting that may be used: TCPDUMP and ACLs. For more detailed tips on using packet captures you may look here. Note: it is important to understand the type of traffic you are looking for such as broadcast, multicast, UDP, TCP traffic and is this data plane or control plane traffic. Understanding this will help you understand how to filter your packet captures when the relevant ports have been identified.

  • Create a monitor session facing the DHCP Client to confirm that the Discover packets are being sent each time a lease is renewed.
config#monitor session test source Ethernet2 rx
config#monitor session test destination cpu
config#show monitor session
Session test
------------------------
Source Ports:
Both: Et2
Destination Ports:
Cpu : mirrorX

  • Capture the packets incoming from Leaf12B

#bash tcpdump -nevvi mirrorX -w /mnt/flash/DHCP-Leaf12B.pcap

  • Another tool is placing a temporary ACL whose job is to only increment counters with qualified packets traverse the port. In this scenario, the ACL would be on Leaf 12B on Po10 to confirm traffic is both leaves and return from the DHCP server. Adding the “counters per-entry” will populate the number of packets matching that ACL that have crossed the port. The match counters shows 23 packets leaving with the last one being 13 seconds ago while 23 packets return with the last one being 12 seconds ago.
switch(config)#ip access-list DHCP
switch(config-acl-DHCP)#counters per-entry
switch(config-acl-DHCP)#exit
switch(config-acl-DHCP)#show ip access-list DHCP
IP Access List DHCP
counters per-entry
10 permit ip 10.0.3.0/24 any  [match 23, 0:00:13]
20 permit ip any host 10.0.2.2  [match 23, 0:00:12]
30 permit ip any any
40 remark end of list

After these tests have been run, examine the PCAP by exporting the file to the case for viewing in WireShark.
You may also run the following show command to see the packet counters increment for the ACL test. Below you can see that no packets to or from the DHCP server are traversing the ports with show ip access-lists DHCP.

Show IP access-lists DHCP
IP Access List DHCP counters per-entry 
10 permit ip 10.0.3.0/24 any 
20 permit ip any host 10.0.2.2

Once you have completed these steps, you will have relevant data and in proper context to assist in bringing your issue to a swifter resolution.

Follow

Get every new post on this blog delivered to your Inbox.

Join other followers: