Arista Portal

  • Arista Fabric Integration with Microsoft Network Load Balancing

 
 
Posted on January 21, 2021 by   |   Updated on June 16, 2021  |   1005 Views
Print Friendly, PDF & Email

Overview

This article provides a brief introduction and configuration of Microsoft NLB with Arista L2LS (without VxLAN) and L3LS VxLAN fabric setup to avoid the most common issues during the deployments.

NLB Introduction

The Network Load Balancing (NLB) feature distributes traffic across several servers by using the TCP/IP networking protocol. By combining two or more computers that are running applications into a single virtual cluster, NLB provides reliability and performance for web servers and other mission-critical servers.

Three modes of operation in NLB

  • Unicast

    • All the NICs assigned to a Microsoft NLB cluster share a common MAC address. The MAC address starts as 02-BF and followed by a series of hexadecimal codes that represent each octet of the VIP address.
    • We don’t have to do anything in the network infrastructure but the system infrastructure needs to be modified to manage network traffic.
    • The NLB cluster masks the cluster’s MAC address for all outgoing traffic to prevent the switch from learning the MAC address after the HB exchange. Hence unicast flooding happens. Generally, VMWare and a few other hosting environments don’t recommend this unicast mode.

  • Multicast

    • Instead of changing the MAC addresses on the network adapters, NLB converts the NLB virtual IP (VIP) address to an NLB multicast MAC address (non-standard IANA). The MAC address starts as 03-BF and followed by a series of hexadecimal codes that represent each octet of the VIP address.
    • IGMP snooping does not dynamically program this address, which results in flooding of the NLB traffic in the VLAN. So, we must configure the network infrastructure to use static ARP entries in L3 and MAC address table entries in L2.
    • Some network platforms don’t allow us to configure manually. So, the NLB traffic floods to all ports in that VLAN.

  • Multicast with IGMP

    • The network switches must be capable of IGMP snooping. This mode is basically the same as multicast mode, except that the switches can automatically build their MAC address tables in this mode.
    • The cluster has a virtual unicast IP address and a virtual multicast MAC address within the IANA range (01:00:5E:XX:XX:XX). NLB nodes send IGMP joins to the 239.255.x.y multicast address (in this address, x.y represents the last two octets of the NLB VIP).
    • Since the switch dynamically populates its IGMP snooping table to point toward the NLB cluster, which prevents unicast flooding.

Each of these modes has different requirements and requires different approaches to the network infrastructure.

Please refer to the link for more details on each Modes operation from the NLB product site:
https://support.microsoft.com/en-in/help/4494444/configuring-network-infrastructure-to-support-the-nlb-operation-mode

Arista L2LS Fabric

NLB cluster is connected to the Arista MLAG leaf pair on L2 VLAN. Here is the Test topology:

  • Cluster running Multicast Mode (non-standard IANA:03-bf)

By default, the NLB hosts are flooding the HB (control traffic) to all the interfaces which are part of VLAN60.

10:45:00.196714 00:50:56:8f:c7:8a > 03:bf:0a:0a:3c:3c, ethertype MS NLB heartbeat (0x886f), length 1510: MS NLB heartbeat, host priority: 2, cluster IP: 10.10.60.60, host IP: 10.10.60.10
10:45:00.598483 00:50:56:8f:04:26 > 03:bf:0a:0a:3c:3c, ethertype MS NLB heartbeat (0x886f), length 1510: MS NLB heartbeat, host priority: 1, cluster IP: 10.10.60.60, host IP: 10.10.60.20

    • L2 traffic (Source:10.10.60.30; Destination:NLB ClusterIP-10.10.60.60)

      • A static MAC entry should be applied on all the devices including all the redundant links that share NLB VLAN (VLAN 60).
Leaf pair-1a(config)#mac address-table static 03bf.0a0a.3c3c vlan 60 interface port-Channel 10
Leaf pair-1b(config)#mac address-table static 03bf.0a0a.3c3c vlan 60 interface ethernet 17

Spine(config)#mac address-table static 03bf.0a0a.3c3c vlan 60 interface port-Channel 42
Leaf pair-2(config)#mac address-table static 03bf.0a0a.3c3c vlan 60 interface port-Channel 52
Example:
          Multicast Mac Address Table
------------------------------------------------------------------
Vlan    Mac Address       Type        Ports
----    -----------       ----        -----
  60    03bf.0a0a.3c3c    STATIC      Po52
      • On the client/end-host(10.10.60.30), we could see the ARP entry of the NLB Cluster IP (VIP).
Host(config)#sh ip arp
Address         Age (sec)  Hardware Addr   Interface
10.10.60.2            N/A  7483.ef59.01eb  Port-Channel60
10.10.60.3            N/A  7483.ef58.dabb  Port-Channel60
10.10.60.6        0:10:36  0073.aaaa.aaaa  Port-Channel60
10.10.60.10       0:09:30  0050.568f.c78a  Port-Channel60
10.10.60.20       0:09:26  0050.568f.0426  Port-Channel60
10.10.60.60       0:09:06  03bf.0a0a.3c3c  Port-Channel60

    • L3 traffic (Source:10.10.30.40; Destination:NLB ClusterIP-10.10.60.60)

      • A static ARP entry is recommended to be applied on the Spine devices (L3) for the NLB cluster IP (VIP).
arp 10.10.60.60 03bf.0a0a.3c3c arpa
      • Traffic destined to the NLB Cluster IP (VIP) is also sent to all the ports in the VLAN as an Unknown unicast.
(config)#sh ip arp
10.10.60.60             -  03bf.0a0a.3c3c  Vlan60, not learned

  • Cluster running IGMP Multicast Mode (Standard multicast MAC)

By default, the NLB hosts are flooding the HB (control traffic) to all the interfaces which are part of VLAN60.

16:59:16.146457 00:50:56:8f:04:26 > 01:00:5e:7f:3c:3c, ethertype MS NLB heartbeat (0x886f), length 106: MS NLB heartbeat, host priority: 1, cluster IP: 10.10.60.60, host IP: 10.10.60.20
16:59:18.512733 00:50:56:8f:c7:8a > 01:00:5e:7f:3c:3c, ethertype MS NLB heartbeat (0x886f), length 106: MS NLB heartbeat, host priority: 2, cluster IP: 10.10.60.60, host IP: 10.10.60.10

    • L2 traffic (Source:10.10.60.30; Destination:NLB ClusterIP-10.10.60.60)

      • IGMP snooping entry is created automatically for the NLB cluster but there is no Multicast config (i.e, IGMP querier) present in the switch. Hence, IGMP snooping is not active.
Leaf pair-1(config)#sh ip igmp snooping groups detail
Vlan Group           IP              First       Last        Expire   Ver Filter Port
                                     Heard       Heard                    Mode
-------------------------------------------------------------------------------------
60   239.255.60.60   10.10.60.60     0d00h17m59  0d00h00m48  00h03m32 v1  0 * Et17
      • The static MAC entry configuration is not allowed in this mode.
mac address-table static 0100.5e7f.3c3c vlan 60 interface ethernet 17
% Only unicast or non-IP multicast addresses can be configured statically.
      • The Messages from NLB clusters and traffic to the clusterIP (DMAC= Multicast) are flooded across all the ports in the VLAN60.
      • To make IGMP snooping active and avoid L2 flooding, the IGMP querier should be present on the device. We can configure either an IGMP querier or PIM enabled on the SVI.
      • IGMP querier is available after adding PIM configuration on an SVI of the Spine.
Spine#sh run int vlan 60
interface Vlan60
   ip address 10.10.60.2/24
   ip pim sparse-mode
   ip virtual-router address 10.10.60.6

Spine(config)#sh ip igmp snooping querier
Vlan  IP Address       Version  Port
----------------------------------------
60    10.10.60.2       v3       Cpu
      • IGMP snooping entry is created and active and hence, there is no flooding in the VLAN 60 for the NLB traffic.
Leaf pair-1(config)#sh ip  igmp snooping groups detail
Vlan Group           IP              First       Last        Expire   Ver Filter Port
                                     Heard       Heard                    Mode
-------------------------------------------------------------------------------------
60   239.255.60.60   10.10.60.60     3d19h52m39  0d00h00m21  00h03m59 v1  0   Et17
60   239.255.60.60   10.10.60.2      0d00h22m48  0d00h00m18  00h01m27 -   -   Po42
60   239.255.255.250 10.10.60.10     0d00h22m42  0d00h01m24  00h02m56 v3  0   Et17
60   239.255.255.250 10.10.60.20     0d00h22m43  0d00h01m19  00h03m01 v3  0   Et17
      • MAC entry is present in the IGMP snooping table.
Leaf pair-1(config)#sh mac address-table multicast vlan 60
                 IGMP Snooping MAC Address Table
------------------------------------------------------------------
Vlan  Mac Address        Type  Port-List
------------------------------------------------------------------
60    01:00:5e:7f:3c:3c  IGMP  Et17, Po42
60    01:00:5e:7f:ff:fa  IGMP  Et17, Po42
      • The spine will not propagate the IGMP reports to Leaf pair-2. However, the mrouter port is present on the leaf pair-2 device due to the IGMP query received from the Spine/uplink.
(config)#sh ip igmp snooping mrouter
Vlan    Interface-ports
---------------------------------------
60      Po52(dynamic)
      • No flooding with L2 traffic. i.e., when the client in the same VLAN reaches out to NLB cluster IP (VIP). Reason: only the Layer-2 lookup happens on the arista device and forwards based on the “IGMP Snooping MAC Address Table”

    • L3 traffic (Source:10.10.30.40; Destination:NLB ClusterIP:10.10.60.60)

      • Added the static ARP entry on the switch (Spine) for the cluster IP (VIP).
Spine-pair(config)#sh ip arp
Address         Age (min)  Hardware Addr   Interface
10.10.30.40           N/A  444c.a825.f795  Vlan30, Port-Channel52
10.10.60.10           N/A  0050.568f.c78a  Vlan60, Port-Channel42
10.10.60.20           N/A  0050.568f.0426  Vlan60, Port-Channel42
10.10.60.60           N/A  0100.5e7f.3c3c  Vlan60, not learned
      • Traffic destined to the NLB cluster IP (DMAC= Multicast) is flooded across all the ports in the vlan60 as an Unknown unicast. This is expected behaviour. Reason: Layer-3 lookup happens on the device due to routing. Based on the ARP entry, we forward the packets as unknown unicast.

ICMP requests sent to NLB cluster IP get replies from both server interfaces, causing detection of Duplicate packets (Expected behaviour with Windows NLB). This behaviour is observed irrespective of whether ICMP is initiated from the same subnet or from a different subnet.

Example capture on same subnet host or SVI:
16:40:19.194535 44:4c:a8:25:ff:51 > 01:00:5e:7f:3c:3c, ethertype IPv4 (0x0800), length 114: 10.10.60.30 > 10.10.60.60: ICMP echo request, id 26900, seq 1, length 80
16:40:19.194936 00:50:56:8f:c7:8a > 44:4c:a8:25:ff:51, ethertype IPv4 (0x0800), length 114: 10.10.60.60 > 10.10.60.30: ICMP echo reply, id 26900, seq 1, length 80
16:40:19.194984 00:50:56:8f:04:26 > 44:4c:a8:25:ff:51, ethertype IPv4 (0x0800), length 114: 10.10.60.60 > 10.10.60.30: ICMP echo reply, id 26900, seq 1, length 80

#ping 10.10.60.60
PING 10.10.60.60 (10.10.60.60) 72(100) bytes of data.
80 bytes from 10.10.60.60: icmp_seq=1 ttl=127 time=0.441 ms
80 bytes from 10.10.60.60: icmp_seq=1 ttl=127 time=0.473 ms (DUP!)
5 packets transmitted, 5 received, +4 duplicates, 0% packet loss

Arista L3LS Fabric

NLB cluster is connected to the Arista MLAG-VTEP leaf pair on L2 VLAN. Here is the Test topology:

  • Cluster running Multicast Mode (non-standard IANA:03-bf)

By default, the NLB hosts are flooding the HB (control traffic) to all the interfaces (inc. Vx1) which are part of VLAN60.

10:45:00.196714 00:50:56:8f:c7:8a > 03:bf:0a:0a:3c:3c, ethertype MS NLB heartbeat (0x886f), length 1510: MS NLB heartbeat, host priority: 2, cluster IP: 10.10.60.60, host IP: 10.10.60.10
10:45:00.598483 00:50:56:8f:04:26 > 03:bf:0a:0a:3c:3c, ethertype MS NLB heartbeat (0x886f), length 1510: MS NLB heartbeat, host priority: 1, cluster IP: 10.10.60.60, host IP: 10.10.60.20

    • L2 VxLAN traffic (Source:10.10.60.30; Destination:NLB ClusterIP-10.10.60.60)

      • A static MAC entry should be applied on the VTEPs for NLB cluster VLAN (VLAN 60) to avoid flooding.
Local VTEP
Leaf pair-1(config)#mac address-table static 03bf.0a0a.3c3c vlan 60 interface port-Channel 10
      • However, adding the static multicast address in the Vx1 interface is not supported. Hence, the Messages from NLB clusters are flooded across all VTEPs in a VXLAN environment.
Leaf pair-2#mac address-table static 03bf.0a0a.3c3c vlan 60 interface vxlan 1 vtep 1.1.1.1
% Adding a static multicast address is not supported

    • L3 VxLAN traffic (Source:10.10.30.40; Destination:NLB ClusterIP-10.10.60.60)

      • A static ARP entry should be applied on the VTEPs for the NLB cluster IP (VIP).
arp 10.10.60.60 03bf.0a0a.3c3c arpa
      • Traffic destined to the NLB Cluster IP (VIP) is also sent to all the VTEPs.
(config)#sh ip arp
10.10.60.60             -  03bf.0a0a.3c3c  Vlan60, not learned

  • Cluster running IGMP Multicast Mode (Standard multicast MAC)

By default, the NLB hosts are flooding the HB (control traffic) is flooded to all the interfaces (inc. Vx1) which are part of VLAN60.

16:59:16.146457 00:50:56:8f:04:26 > 01:00:5e:7f:3c:3c, ethertype MS NLB heartbeat (0x886f), length 106: MS NLB heartbeat, host priority: 1, cluster IP: 10.10.60.60, host IP: 10.10.60.20
16:59:18.512733 00:50:56:8f:c7:8a > 01:00:5e:7f:3c:3c, ethertype MS NLB heartbeat (0x886f), length 106: MS NLB heartbeat, host priority: 2, cluster IP: 10.10.60.60, host IP: 10.10.60.10

    • L2 VxLAN traffic (Source:10.10.60.30; Destination:NLB ClusterIP-10.10.60.60)

      • IGMP snooping entry is created automatically for the NLB cluster but there is no Multicast config (i.e, IGMP querier) present in the switch. Hence, IGMP snooping is not active.
Leaf pair-1(config)#sh ip igmp snooping groups detail
Vlan Group           IP              First       Last        Expire   Ver Filter Port
                                     Heard       Heard                    Mode
-------------------------------------------------------------------------------------
60   239.255.60.60   10.10.60.60     0d00h17m59  0d00h00m48  00h03m32 v1  0 * Et17
      • The static MAC entry configuration is not allowed in this mode.
mac address-table static 0100.5e7f.3c3c vlan 60 interface ethernet 17
% Only unicast or non-IP multicast addresses can be configured statically.
      • The Messages from NLB clusters and traffic to the clusterIP are flooded across all VTEPs in a VXLAN environment.
      • IGMP snooping should be disabled in a VxLAN environment. IGMP packets will be flooded to the remote VTEPs like BUM traffic. 

    • L3 VxLAN traffic (Source:10.10.30.30; Destination:NLB ClusterIP-10.10.60.60)

      • A static ARP entry should be applied on the VTEPs for NLB cluster IP (VIP).
arp 10.10.60.60 0100.5e7f.3c3c arpa
      • Traffic destined to the NLB cluster IP (DMAC= Multicast) is flooded across all the ports in the vlan60 and to the remote VTEPs as Unknown unicast. This is expected behaviour. Reason: Layer-3 lookup happens on the device due to routing. Based on the ARP entry, we forward the packets as unknown unicast.
(config)#sh ip arp
10.10.60.60             -  0100.5e7f.3c3c  Vlan60, not learned

ICMP requests sent to NLB cluster IP get replies from both server interfaces, causing detection of Duplicate packets (Expected behaviour with Windows NLB). This behaviour is observed irrespective of whether ICMP is initiated from the same subnet or from a different subnet.

#ping 10.10.60.60
PING 10.10.60.60 (10.10.60.60) 72(100) bytes of data.
80 bytes from 10.10.60.60: icmp_seq=1 ttl=127 time=0.401 ms
80 bytes from 10.10.60.60: icmp_seq=1 ttl=127 time=0.433 ms (DUP!)
80 bytes from 10.10.60.60: icmp_seq=2 ttl=127 time=0.186 ms
80 bytes from 10.10.60.60: icmp_seq=2 ttl=127 time=0.206 ms (DUP!)
80 bytes from 10.10.60.60: icmp_seq=3 ttl=127 time=0.125 ms
80 bytes from 10.10.60.60: icmp_seq=3 ttl=127 time=0.144 ms (DUP!)
80 bytes from 10.10.60.60: icmp_seq=4 ttl=127 time=0.108 ms
80 bytes from 10.10.60.60: icmp_seq=4 ttl=127 time=0.127 ms (DUP!)
80 bytes from 10.10.60.60: icmp_seq=5 ttl=127 time=0.108 ms
--- 10.10.60.60 ping statistics ---
5 packets transmitted, 5 received, +4 duplicates, 0% packet loss

Follow

Get every new post on this blog delivered to your Inbox.

Join other followers: