• Troubleshooting EVPN IRB with VXLAN

 
 
Print Friendly, PDF & Email

Overview

This article provides a brief introduction to EVPN IRB with VXLAN along with basic debugging methods for the same.

Introduction

Ethernet VPN (EVPN) is an extension of the MP-BGP protocol introducing a new address family. EVPN is used as a control-plane for VXLAN environments to exchange information such as MAC addresses and ARP bindings along with VTEP flood list. Additionally,  IP prefixes can be exchanged in the overlay using Type-5 routes. 

Platform Compatibility

The below table captures the EVPN IRB support for a few Arista platforms: on the platforms listed below:

 

Platform Feature Support EOS Release
7050X/ 7300X/ 7050X2/ 7060X/ 7260X/
7500R/ 7280R  platforms
Symmetric IRB
Asymmetric IRB
4.20.1F
7160 platform Symmetric IRB (only v4)
Asymmetric IRB (only v4)
4.20.2F
7020R/7280R2 platform Symmetric IRB
Asymmetric IRB
4.20.5F
7050X3 platform Symmetric IRB
Asymmetric IRB
4.21.0F
7160 platform L3 EVPN (Type-5) v6 support 4.21.0F

The EOS 4.20.6F release introduces support for EVPN Centralized Anycast Gateway model. 

Platform
Feature Support
EOS Release
7050X/ 7050X2/ 7060X/ 7280R/ 7500R  platforms
EVPN centralized Anycast GW
4.20.6F

Please refer to this link for more details: https://eos.arista.com/eos-4-20-6f/evpn-centralized-anycast-gateway/

Topology

Asymmetric/Symmetric IRB

  1. In a VxLAN environment, the anycast GW for all the extended VLAN subnets is present on all the leaf devices (VTEPs) in Asymmetric IRB whereas, in the Symmetric IRB model, the anycast GW for the subnets is present only on the leaf switches with directly attached hosts in the extended VLAN.
  2. Symmetric IRB requires VRF to VNI mappings to exchange shared subnet routes, /32 host routes using L3 VNI whereas VRF to VNI mapping is not required for Asymmetric IRB. 

Please refer to this link for a detailed explanation: https://eos.arista.com/eos-4-20-1f/evpn-irb-with-vxlan-underlay/#EVPN_Integrated_Routing_and_Bridging_IRB_with_VXLAN

Pre-checks

  • Ensure that “service routing protocols model multi-agent” is configured on all Arista devices including and between the VTEPs.
  • Ensure that the loopback source interface for the VXLAN tunnel interface is configured on the VTEPs, and is advertised in the underlay i.e., loopback reachability between VTEPs is maintained.
Leaf1(config)#sh int vxlan 1
Vxlan1 is up, line protocol is up (connected)
  Hardware is Vxlan
  Source interface is Loopback1 and is active with 1.1.1.1

 

  • Ensure that the underlay routing configured to enable multi-hop EVPN neighborship between Leaf (VTEPs) and Spine devices. While it’s possible to enable EVPN peering directly between the leaf switches, Spines are used as route reflectors in EVPN address family to avoid full mesh peering between leaf VTEPs.
  • Ensure that the VLAN to VNI mappings are matched on the VTEPs. Note: VLAN is local but VNI is global and hence ensure that the VNIs are correctly configured. Verify the same using “show interface vxlan1” command output.
Leaf1(config)#sh int vxlan 1 
Vxlan1 is up, line protocol is up (connected)   
Hardware is Vxlan   Source interface is Loopback1 and is active with 1.1.1.1
Replication/Flood Mode is headend with Flood List Source: EVPN  
Remote MAC learning via EVPN   
VNI mapping to VLANs   
Static VLAN to VNI mapping is     
[10, 1010]        [11, 1011] [12, 1012]        [14, 1014] 
.. ..   
Static VRF to VNI mapping is    
[S-VRF, 10001]

 

  • Ensure that the MAC-VRF RT/RD values are configured on the VTEPs. Check RT which should match across the network. i.e., import/export statements.
Leaf1(config)# 
  vlan 10
      rd auto
      route-target import evpn 3434:1010
      route-target export evpn 1212:1010
      redistribute learned
   Leaf3(config)# 
     vlan 10
         rd auto
         route-target import evpn 1212:1010
         route-target export evpn 3434:1010
         redistribute learned
  • For Symmetric IRB; Confirm internal VLANs were dynamically created for IP-VRF to VNI mapping. Both MLAG peers should have the same dynamic VLAN for EVPN.
Leaf1(config)#show interface vxlan 1
Vxlan1 is up, line protocol is up(connected)
Dynamic VLAN to VNI mapping for 'evpn' is
    [1008, 10001]
  Note: All Dynamic VLANs used by VCS are internal VLANs.
        Use 'show vxlan vni' for details.
  Static VRF to VNI mapping is
   [S-VRF, 10001]
Leaf2(config)#show interface vxlan 1
Vxlan1 is up, line protocol is up(connected)
Dynamic VLAN to VNI mapping for 'evpn' is
    [1008, 10001]
  Note: All Dynamic VLANs used by VCS are internal VLANs.
        Use 'show vxlan vni' for details.
  Static VRF to VNI mapping is
   [S-VRF, 10001]
Leaf2(config)#show vlan dynamic 
Dynamic VLAN source       VLANS 
dynvtep                   NONE 
evpn                      1008 
mlag                      NONE 
mss                       NONE 
vccbfd                    NONE 
vcclr                     NONE 
vccuplink                 NONE 
vxlan                     NONE
  • Check if the “redistribute connected” or a network statement is present under the IP-VRF in order to advertise/generate Type-5 EVPN routes for the connected prefixes. 
vrf red     
rd 3.3.3.1:8000     
route-target both 0:8000     
redistribute connected
  • Based on the design, check if the MLAG shared router MAC is configured on both MLAG peers. This is typically configured to avoid peer-link utilization in S-IRB setup.
int vxlan1 
vxlan virtual-router encapsulation mac-address mlag-system-id

Troubleshooting Scenarios

EVPN routes are not received or not advertised

  • Check the EVPN neighbor connections.
Leaf1(config)#show bgp evpn summary
BGP summary information for VRF default
Router identifier 172.16.1.1, local AS number 65001
Neighbor Status Codes: m - Under maintenance
  Neighbor         V AS MsgRcvd   MsgSent InQ OutQ Up/Down  State  PfxRcd PfxAcc
  21.22.21.21      4 65005 1467      1477  0   0   01:43:29 Estab   14     14
  21.22.21.22      4 65005 1479      1468  0   0   01:43:41 Estab   14     14    
Leaf1(config)#show ip bgp neighbors 21.22.21.21
BGP neighbor is 21.22.21.21, remote AS 65005, external link
  BGP version 4, remote router ID 21.22.21.21, VRF default
  Inherits configuration from and member of peer-group EVPN-SPINE
  ...
Neighbor Capabilities:
    Multiprotocol L2VPN EVPN: advertised and received and negotiated
    Four Octet ASN: advertised and received and negotiated
    Route Refresh: advertised and received and negotiated
  ...
    Additional-paths recv capability:
      L2VPN EVPN: advertised
    Additional-paths send capability:
      L2VPN EVPN: received
  Restart timer is inactive
    L2VPN EVPN End-of-RIB received: Yes
  ...
  • Check that the EVPN received prefix contains extended communities. If not, ensure that the peer has enabled “send-community” in its neighbor configuration.
“neighbor eBGP-SPINE-EVPN send-community
Leaf1(config)#show bgp evpn detail
BGP routing table information for VRF default
Router identifier 172.16.1.1, local AS number 65001
BGP routing table entry for mac-ip 0050.0600.0800, Route Distinguisher: 1.1.1.1:10
 Paths: 1 available
  Local
    - from - (0.0.0.0)
      Origin IGP, metric -, localpref -, weight 0, valid, local, best
      Extended Community: Route-Target-AS:1010:1010 TunnelEncap:tunnelTypeVxlan
      VNI: 1010 ESI: 0000:0000:0000:0000:0000   

 

  • Ensure that the VLANs are mapped correctly as VLAN based or VLAN-aware-bundle instances and this should be the same on all VTEPs.
Leaf1(config)#sh bgp evpn instance
EVPN instance: VLAN 10
  Route distinguisher: 0:0
  Service interface: VLAN-based
  Local IP address: 1.1.1.1
  Encapsulation type: VXLAN
   Leaf1(config)#sh bgp evpn instance
   EVPN instance: VLAN-aware bundle tac
     Route distinguisher: 11.12.11.11:1112
     Service interface: VLAN-aware bundle
     Local IP address: 1.1.1.1
     Encapsulation type: VXLAN

 

  • Check if the EVPN routes are generated locally on the switch. To export the EVPN routes using RT values, we would need some data-plane encapsulation in place. i.e., VXLAN interface should be up and relevant VNI mappings should be present.
Leaf1(config)#sh int vxlan 1 
Vxlan1 is up, line protocol is up (connected) 
Leaf1(config)#sh bgp evpn route-type mac-ip 0050.0600.0800 detail 
BGP routing table information for VRF default 
Router identifier 172.16.1.1, local AS number 65001

 

  • If EVPN routes are generated locally but not received on Remote VTEP.
    • route-target import configuration is  not configured correctly on the remote VTEP (OR)
    • next-hop in EVPN routes are not reachable from SPINE. i.e., the next-hop would be the vx1 interface IP which should be reachable in the underlay.
SPINE(config)#show evpn route-type mac-ip detail 
BGP routing table entry for mac-ip 0050.0600.0800, Route Distinguisher: 1.1.1.1:10  
Paths: 2 available  
65001     
   1.1.1.1 from 11.12.11.12 (172.16.2.2)       
      Origin IGP, metric -, localpref 100, weight 0, valid, external, ECMP head, best, ECMP contributor  
      Extended Community: Route-Target-AS:1010:1010 TunnelEncap:tunnelTypeVxlan       
      VNI: 1010 ESI: 0000:0000:0000:0000:0000

 

  • MAC is learned locally but not advertised to remote VTEPs although the aforementioned steps are validated. Then, check if the host MAC is blacklisted on the device by reviewing the syslog and “show bgp evpn host-flap” output.
21:48:40 sw Bgp: 3076: %EVPN-3-BLACKLISTED_DUPLICATE_MAC: MAC address 00:16:3e:80:00:15 on VLAN 32 
has been blacklisted for moving 5 or more times within the past 180 seconds

 

Flood list not populated

  • Check if the VTEP advertises/receives IMET (Type 3) EVPN routes to/from neighborship when the EVPN neighbor is UP for each MAC-VRF.
Leaf1(config)#sh bgp evpn route-type imet detail  
BGP routing table entry for imet 2.2.2.2, Route Distinguisher: 2.2.2.2:10  
  Paths: 2 available    

    65005 65002     
      2.2.2.2 from 21.22.21.21 (21.22.21.21)       
        Origin IGP, metric -, localpref 100, weight 0, valid, external, ECMP head, best, ECMP contributor 
        Extended Community: Route-Target-AS:1010:1010 TunnelEncap:tunnelTypeVxlan       
        VNI: 1010    
    
    65005 65002     
      2.2.2.2 from 21.22.21.22 (21.22.21.22)       
        Origin IGP, metric -, localpref 100, weight 0, valid, external, ECMP, ECMP contributor      
        Extended Community: Route-Target-AS:1010:1010 TunnelEncap:tunnelTypeVxlan       
        VNI: 1010

 

  • Check if the VLAN to VNI mappings are configured correctly to advertise the IMET routes correctly to the remote VTEP.

 

Data-plane issue

  • Check if the corresponding MAC is learned correctly in both software and hardware of the devices.
sw(config)#show vxlan address-table 
          Vxlan Mac Address Table
---------------------------------------------------------------
VLAN  Mac Address     Type Prt VTEP             Moves Last Move
----  -----------     ---- --- ----             ----- ---------
  20  5001.00be.ab97  EVPN Vx1 192.168.200.1    1 0:00:38 ago
sw(config)#show platform _______ mac-address-table

------------------------------MAC Table------------------------------------
|--------------------------------------------------------------------------|
|     |                   | Destination | Aging | SA |         Intf        |
| FID |         MAC       | Type        | Val   |Stt |Dyn|Prg| Drp| Name   |
|--------------------------------------------------------------------------|
| 100 | 50:01:00:be:ab:97 |         FEC| 0x000ca | 5 | Y | N | N  | Vx1
  • Check that the /32 routes are installed correctly in both hardware and software. 
sw#show platform _______ l3 software routes

Host table
VrfId  Host                   Position EntryId  Class  NextHopId Ecmp Hit
    0  10.85.128.255         0x6001e90  1956      1      8        N    N
sw#show platform _______ ip route
---------------------------------------------------------------------------------------------------------
|                                        Routing Table                                                  | 
|--------------------------------------------------------------------------------------------------------
|VRF|   Destination   |     |                    |      |      |                   | ECMP| FEC  | Tunnel
| ID|      Subnet     | Cmd |     Destination    | VID  |Outlif|   MAC / CPU Code  |Index| Index|T Value
---------------------------------------------------------------------------------------------------------
|0  |  0.0.0.0/8      |TRAP | CoppSystemL3DstMiss|  0   |   -  | ArpTrap           |  -  |126976|  -   
|0  |  4.4.4.4/32     |TRAP | CoppSystemIpUcast  |  0   |   -  | Receive           |  -  |126973|  -   
|0  |  5.5.5.5/32     |ROUTE| Et1/1              | 1007 | 4094 | 28:99:3a:99:e2:d1 |  -  |32776 |  -  

Note: Platform commands might vary depending upon the HW models.

  • Ensure that the MTU size in the path accounts for the additional bytes in the VXLAN header to avoid the packet drops with size more than 1500 bytes. 

Sub-optimal forwarding or High peer-link utilization

  • If flooding is observed for Type 5 routes and Type 2 routes Symmetric IRB routes, check if MLAG Shared MAC can be configured on both MLAG peers, as it ensures a common router MAC is advertised in EVPN updates to remote VTEPs
  • If the MLAG shared router MAC is configured but not seen in EVPN route advertisements (show bgp evpn), verify the MLAG state. The MLAG Shared MAC is activated when the MLAG state is “active-active”.
  • Check if the destination IPs are for hosts connected to non-MLAG interfaces (orphaned ports).

Please refer to this link for more details: https://eos.arista.com/eos-4-21-3f/evpn-mlag-shared-router-mac/

Limitations (Updated till EOS 4.21.6F ) 

Data collection for Arista Support

In the event, further assistance is needed, please send over the below logs to support@arista.com:

4.21.0F and Later
  1. Topology
  2. show tech-support
  3. show agent qtrace
  4. show tech extended evpn
Pre 4.21.0F
  1. Topology
  2. show tech-support
  3. show tech bgp
  4. show bgp evpn
  5. show bgp evpn instance
  6. show bgp evpn detail
  7. show bgp evpn summary
  8. show L2rib input all
  9. show L2rib input all floodset
  10. show L2rib output
  11. show L2rib output floodset
  12. show agent qtrace

Useful Resources

Follow

Get every new post on this blog delivered to your Inbox.

Join other followers: