Posted on May 1, 2017 5:49 pm
 |  Asked by Lukas Hubschmid
 |  2924 views
0
0
Print Friendly, PDF & Email

Hello all,

Please see attached picture for network topolgy.
I try to use VXLAN with HER for DCI, together with BGP&BFD as routing protocol (for underlay) between Site A and Site B.
– Site A: VLAN100 mapped to DCI VNI 10100
– Site B: VLAN1100 mapped to DCI VNI 10100
– spine1&spine2 as mlag peers with logical VTEP
– spine3 as single VTEP
– ports for DCI configured as L3 routed (no switchport)
– EBGP between spine1 and spine3 as well as spine2 and spine3
– VLAN4093 between spine1 (10.0.1.1) and spine2 (10.0.1.2) for re-routing in case one DCI is down
– IBGP between spine1 and spine2
– target is to ping from srv1 (172.16.0.10) to srv2 (172.16.0.11)
– vEOS 4.17.2F

MLAG Config on spine1 (analog on spine2):

interface Ethernet15
   channel-group 15 mode active
!
interface Ethernet16
   channel-group 15 mode active

interface Port-Channel15
   switchport mode trunk
   switchport trunk group TRGROUP-MLAGPEER

mlag configuration
   domain-id mlag-spine1-spine2
   local-interface Vlan4094
   peer-address 192.168.254.254
   primary-priority 10
   peer-link Port-Channel15

VXLAN Config on spine1:

interface Loopback1
   ip address 1.1.1.1/32

interface Vlan4093
   ip address 10.0.1.1/30

interface Vxlan1
   vxlan source-interface Loopback1
   vxlan udp-port 4789
   vxlan vlan 100 vni 10100
   vxlan flood vtep 2.2.2.2

BGP Config on spine1 (9.9.9.1 on Loopback2 only added for troubleshooting):

router bgp 64512
   router-id 9.9.9.1
   neighbor 10.0.0.2 remote-as 64513
   neighbor 10.0.0.2 fall-over bfd
   neighbor 10.0.0.2 maximum-routes 12000
   neighbor 10.0.1.2 remote-as 64512
   neighbor 10.0.1.2 next-hop-self
   neighbor 10.0.1.2 fall-over bfd
   neighbor 10.0.1.2 maximum-routes 12000
   network 1.1.1.1/32
   network 9.9.9.1/32

Communication works as expected when all switches are active (continuous ping from 172.16.0.10 to 172.16.0.11).
But when I take down the DCI over which the ping is running, communication stops, although the routing table is updated correctly:

spine1(config-if-Et3)#sh ip route

VRF name: default
Codes: C - connected, S - static, K - kernel,
       O - OSPF, IA - OSPF inter area, E1 - OSPF external type 1,
       E2 - OSPF external type 2, N1 - OSPF NSSA external type 1,
       N2 - OSPF NSSA external type2, B I - iBGP, B E - eBGP,
       R - RIP, I L1 - ISIS level 1, I L2 - ISIS level 2,
       O3 - OSPFv3, A B - BGP Aggregate, A O - OSPF Summary,
       NG - Nexthop Group Static Route, V - VXLAN Control Service

Gateway of last resort is not set

 C      1.1.1.1/32 is directly connected, Loopback1
 B I    2.2.2.2/32 [200/0] via 10.0.1.2, Vlan4093
 C      9.9.9.1/32 is directly connected, Loopback2
 B I    9.9.9.2/32 [200/0] via 10.0.1.2, Vlan4093
 B I    9.9.9.3/32 [200/0] via 10.0.1.2, Vlan4093
 C      10.0.1.0/30 is directly connected, Vlan4093
 C      172.16.0.0/24 is directly connected, Vlan100
 C      192.168.254.252/30 is directly connected, Vlan4094

What I noted: it seems the MAC addresses learned via remote VTEPs (VXLAN address-table) are not synced to other MLAG peer.
Address table in “normal” operation (ping running via spine1 to spine3):

spine1#sh vxlan address-table
          Vxlan Mac Address Table
----------------------------------------------------------------------

Vlan  Mac Address     Type     Prt  Vtep             Moves   Last Move
----  -----------     ----     ---  ----             -----   ---------
 100  00eb.fe6a.0500  DYNAMIC  Vx1  2.2.2.2          1       0:03:09 ago
Total Remote Mac Addresses for this criterion: 1
spine2#sh vxlan address-table
          Vxlan Mac Address Table
----------------------------------------------------------------------

Vlan  Mac Address     Type     Prt  Vtep             Moves   Last Move
----  -----------     ----     ---  ----             -----   ---------
Total Remote Mac Addresses for this criterion: 0
spine3#sh vxlan address-table
          Vxlan Mac Address Table
----------------------------------------------------------------------

Vlan  Mac Address     Type     Prt  Vtep             Moves   Last Move
----  -----------     ----     ---  ----             -----   ---------
1100  00eb.fedf.4000  DYNAMIC  Vx1  1.1.1.1          1       0:03:12 ago
Total Remote Mac Addresses for this criterion: 1

As soon as I disable et3 (DCI) on spine1, the communication stops working.
I now see the MAC from srv1 on spine2 learned from spine1 via Vx1 – which seems that the VXLAN packets from spine1 traversing the MLAG peer link (VLAN4093) are interpreted on spine2 – which makes no sense to me. MAC address for srv1 should be on Po1 (MLAG to leafs).

spine2#sh vxlan address-table
          Vxlan Mac Address Table
----------------------------------------------------------------------

Vlan  Mac Address     Type     Prt  Vtep             Moves   Last Move
----  -----------     ----     ---  ----             -----   ---------
 100  00eb.fedf.4000  DYNAMIC  Vx1  1.1.1.1          2       0:01:21 ago
Total Remote Mac Addresses for this criterion: 1

After a few minutes, spine1 obviously loses the entry for srv2 (spine2 still has ONLY the entry for srv1 but not for srv2):

spine1(config)#sh vxlan address-table
          Vxlan Mac Address Table
----------------------------------------------------------------------

Vlan  Mac Address     Type     Prt  Vtep             Moves   Last Move
----  -----------     ----     ---  ----             -----   ---------
Total Remote Mac Addresses for this criterion: 0

Unfortunatley I cannot do packet capture (not supported in GNS3 and monitor sessions in vEOS to CPU not working).
So I have two questions:
1. why are the MAC addresses learned via VXLAN not synced to the other MLAG peer (for my understanding, they should be synced)?
2. why is the VXLAN traffic interpreted on spine2 (destination is NOT it’s own VTEP 1.1.1.1) instead of routed to the remote VTEP (2.2.2.2)?

Any help is greatly appreciated!

Kind regards,
Lukas

Attachments:
0
Posted by Aesha Parikh
Answered on May 2, 2017 2:15 am

Hi Lukas,

Your understanding is correct. Mac address learnt over vxlan should be synced across peers. Encap vxlan Traffic coming over a mlag peer link not destined for its own IP should get routed normally. 

The above scenario will work as expected on physical duts. There are some known issues with vxlan + mlag setup on veos and some changes have been made to fix those issues. Would it be possible to upgrade the mlag setup to the latest veos image? 

Thanks,

Aesha

0
Posted by Lukas Hubschmid
Answered on May 2, 2017 7:36 am

Hi Aesha,

Thanks for your answer!

First a question for my understanding: should the local VTEP (e.g. 1.1.1.1 for spine1/spine2) be included in the vtep flood list as well, additionally to the remote VTEPs (e.g. 2.2.2.2)? And if yes, what would be the reason for this? Examples on pages 936 and 947 in the EOS User Manual 4.18.1F indicates this (address on loopback interface also includd in vtep flood list).

I changed the vEOS to the most recent (4.18.1F).
It seems MAC sync for addresses learned on VTEPs is now working properly (only DCI between spine1<->spine3 active, but spine2 learns the remote MAC address from srv2 as well via MLAG peer).

Nevertheless, the behaviour with re-routing is still strange.
Given the same scenario: traffic vom srv1 is switched to spine1 (which et3 / DCI has failed) and should be VXLAN encapsulated and re-routed to spine3 via spine2.
On spine2, it seems the MAC address for srv1 (00eb.fedf.4000) is flapping between Vx1 and Po1 and “non-existent”:

spine2(config-if-Vx1)#sh mac address-table
          Mac Address Table
------------------------------------------------------------------

Vlan    Mac Address       Type        Ports      Moves   Last Move
----    -----------       ----        -----      -----   ---------
 100    00eb.feac.b829    STATIC      Po15
4093    00eb.feac.b829    STATIC      Po15
4094    00eb.feac.b829    STATIC      Po15
Total Mac Addresses for this criterion: 3

          Multicast Mac Address Table
------------------------------------------------------------------

Vlan    Mac Address       Type        Ports
----    -----------       ----        -----
Total Mac Addresses for this criterion: 0
spine2(config-if-Vx1)#sh mac address-table
          Mac Address Table
------------------------------------------------------------------

Vlan    Mac Address       Type        Ports      Moves   Last Move
----    -----------       ----        -----      -----   ---------
 100    00eb.feac.b829    STATIC      Po15
 100    00eb.fedf.4000    DYNAMIC     Po1        2       0:00:00 ago
4093    00eb.feac.b829    STATIC      Po15
4094    00eb.feac.b829    STATIC      Po15
Total Mac Addresses for this criterion: 4

          Multicast Mac Address Table
------------------------------------------------------------------

Vlan    Mac Address       Type        Ports
----    -----------       ----        -----
Total Mac Addresses for this criterion: 0
spine2(config-if-Vx1)#sh mac address-table
          Mac Address Table
------------------------------------------------------------------

Vlan    Mac Address       Type        Ports      Moves   Last Move
----    -----------       ----        -----      -----   ---------
 100    00eb.feac.b829    STATIC      Po15
 100    00eb.fedf.4000    DYNAMIC     Po1        2       0:00:00 ago
4093    00eb.feac.b829    STATIC      Po15
4094    00eb.feac.b829    STATIC      Po15
Total Mac Addresses for this criterion: 4

          Multicast Mac Address Table
------------------------------------------------------------------

Vlan    Mac Address       Type        Ports
----    -----------       ----        -----
Total Mac Addresses for this criterion: 0
spine2(config-if-Vx1)#sh mac address-table
          Mac Address Table
------------------------------------------------------------------

Vlan    Mac Address       Type        Ports      Moves   Last Move
----    -----------       ----        -----      -----   ---------
 100    00eb.feac.b829    STATIC      Po15
 100    00eb.fedf.4000    DYNAMIC     Vx1        3       0:00:00 ago
4093    00eb.feac.b829    STATIC      Po15
4094    00eb.feac.b829    STATIC      Po15
Total Mac Addresses for this criterion: 4

          Multicast Mac Address Table
------------------------------------------------------------------

Vlan    Mac Address       Type        Ports
----    -----------       ----        -----
Total Mac Addresses for this criterion: 0
spine2(config-if-Vx1)#sh mac address-table
          Mac Address Table
------------------------------------------------------------------

Vlan    Mac Address       Type        Ports      Moves   Last Move
----    -----------       ----        -----      -----   ---------
 100    00eb.feac.b829    STATIC      Po15
4093    00eb.feac.b829    STATIC      Po15
4094    00eb.feac.b829    STATIC      Po15
Total Mac Addresses for this criterion: 3

          Multicast Mac Address Table
------------------------------------------------------------------

Vlan    Mac Address       Type        Ports
----    -----------       ----        -----
Total Mac Addresses for this criterion: 0
spine2(config-if-Vx1)#

I attached the configuration of the 3 spine switches – a short look would be greatly appreciated, maybe I still have some obvious configuation error somewhere :)

Btw: I think there might be an error on page 944 with missing “add” if this is of any interest:

Switch2
switch(config)#interface vxlan1
switch(config-if-Vx1)#vxlan flood vtep 10.1.1.1
switch(config-if-Vx1)#vxlan flood vtep 10.2.2.2

Kind regards & many thanks,
Lukas

0
Posted by Alex
Answered on May 5, 2017 2:14 pm

Hi Lukas, from the fact that the Serv-1 MAC address is flapping between the local port-channel and the VXLAN tunnel, I suspect you may have a loop in your setup. Can you provide the output of the “show mlag detail” on all the switches (spine-1, spine-2, leaf-1 and leaf-2) and the output of the LLDP table on each switch “show lldp neighbors”.  For both outputs can you collect them over a short time period to monitor any change.  

Regarding the flood list, you only need to config physical VTEP’s (2.2.2.2) to the flood-list of spine-1 and spine-2. This is details here … https://eos.arista.com/vxlan-with-mlag-configuration-guide/

Alex

0
Posted by Aesha Parikh
Answered on May 6, 2017 12:37 am

Hi Lukas,

Local VTEP IP need not be present in flood list. The documentation shows it just for ease of configuration as you can add symmetric flood list across all VTEPs. If your own VTEP ip is part of flood list, Vxlan will not create a copy for its own IP so it won’t matter. 

Thank you for testing with 4.18.1. I will relay this observation of mac address flap to veos team.

Also, I do not see any config attachment, can you please resend it?  

Thanks,

Aesha

Post your Answer

You must be logged in to post an answer.