• VXLAN: security recommendations

 
 
Print Friendly, PDF & Email

Abstract

This document provides recommendations that are advised to implement in order to increase the security in multitenant network environments built on Arista Networks devices using VXLAN.

Introduction

One of the crucial qualities of modern cloud network infrastructure is scalability. Scalability can’t be achieved if security of the network operations inside the cloud is compromised. As for example, load scalability is not achievable in environments where the VMs are not able to operate when the network between them is not working properly due to hijacked MAC-addresses. One of the technologies used nowadays to address the challenges with scalability inside the cloud networks is VXLAN. Currently there is not much information about best practices and security advisories about the usage of VXLAN including IETF draft, so here we will discuss the techniques which can be used to secure performance of this feature at the best level of its ability.

VXLAN background

As you probably already know, VXLAN is used to help with tackling high demands for physical network infrastructure placed by server virtualisation. And one of the goals that can be accomplished by using VXLAN is scaling the L2 topologies by stretching an L2 overlay over the L3 network. This technology is most commonly used in multitenant network environments, which by definition means multiple administrative domains are working on the same infrastructure. This fact indicates that in such environments security consideration should be a priority. As VXLAN is just a stretched L2 dataplane network over L3 control plane network, we can assume that all possible security threats applicable to 2nd and 3rd layers of TCP/IP stack are also applicable to VXLAN. So, what are those threats? Note: As we are talking here about how to secure network infrastructure using VXLAN, we will concentrate on best practices for managing active transport infrastructures like switches and routers and will skip the topic of additional security aspects by using firewalls and IPS/IDS systems as it would be too much for one topic.  DISCLAIMER: Material of this post is provided as advice, all configuration changes should be implemented according to reader’s discretion.

VXLAN implementation options

Let’s set the initial point for the use cases and in particular, discuss the options on how VXLAN domain can be implemented. Basic configuration for VXLAN will be described based on the following illustration: pic.1 Picture1   We have:

  • two VTEPs: VTEP1 with interface loopback0 ip-address 1.1.1.1/32; VTEP2 with interface loopback0 ip-address: 2.2.2.2/32
  • DCI between the location of VTEPs configured as a L3 device with interfaces’ ip-addresses in networks: 10.0.4.0/24 and 10.0.2.0/24 for VTEP1 and VTEP2 accordingly
  • tenantA site:1 Pod:1 connected to the tenantA site2: Pod:1 via the VLAN13 which is mapped to VNI 101 on VXLAN side
  • tenantA site:1 Pod:2 is connected to the VTEP1 as a L3  client, ip-address of the client in network: 10.0.5.6/24 and interface loopback0 ip-address: 5.5.5.5/32
  • All test are created on Arista DCS-7150S-64-CL running EOS-4.17.1F

Flood list

Flood lists are used to support BUM traffic over VXLAN and it is nothing else but just a list of remote VTEPs. The switch will replicate BUM data locally for bridging across the remote VTEPs specified by flood list. Configuration of VTEP1:

VTEP1(config)#interface Vxlan1
VTEP1(config-if-Vx1)#vxlan source-interface Loopback0 
VTEP1(config-if-Vx1)#vxlan udp-port 4789 
VTEP1(config-if-Vx1)#vxlan vlan 13 vni 101 
VTEP1(config-if-Vx1)#vxlan flood vtep 2.2.2.2

Configuration of VTEP2:

VTEP2(config)#interface Vxlan1
VTEP2(config-if-Vx1)#vxlan source-interface Loopback0 
VTEP2(config-if-Vx1)#vxlan udp-port 4789 
VTEP2(config-if-Vx1)#vxlan vlan 13 vni 101 
VTEP2(config-if-Vx1)#vxlan flood vtep 1.1.1.1

Multicast group

Multicast packet flooding is supported with VXLAN bridging without MLAG. This type of VXLAN operation will associate a specified multicast group with the VTI, which handles multicast and broadcast traffic as a L2 interface in a bridging domain. Configuration of VTEP1:

VTEP1(config)#interface Vxlan1
VTEP1(config-if-Vx1)#vxlan multicast-group 227.10.1.1
VTEP1(config-if-Vx1)#vxlan source-interface Loopback0
VTEP1(config-if-Vx1)#vxlan udp-port 4789
VTEP1(config-if-Vx1)#vxlan vlan 13 vni 101

Configuration of VTEP2:

VTEP2(config)#interface Vxlan1
VTEP2(config-if-Vx1)#vxlan multicast-group 227.10.1.1
VTEP2(config-if-Vx1)#vxlan source-interface Loopback0
VTEP2(config-if-Vx1)#vxlan udp-port 4789
VTEP2(config-if-Vx1)#vxlan vlan 13 vni 101

Note: Multicast for flooding is not supported any longer and has been deprecated in favour of Head End Replication.

VXLAN Control Service on CVX

VXLAN Control Service (VCS) and other services such as HSC, on CVX allow simplification and management of an entire VXLAN enabled network infrastructure. This solution also allows integration with other orchestration and control platforms such as VMWare NSX, Nuage VSP, OpenStack Neutron, etc. This solution propagates BUM Flood Lists, MAC-VTEP bindings, IP-MAC bindings and also overlay orchestration information like VLAN-VNI bindings, Port-VLAN-VNI bindings. VTEP1 configuration:

VTEP1(config)#interface Vxlan1
VTEP1(config-if-Vx1)#vxlan source-interface Loopback0 
VTEP1(config-if-Vx1)#vxlan control-service vxlan udp-port 4789 
VTEP1(config-if-Vx1)#vxlan vlan 13 vni 101
VTEP1(config)#management cvx 
VTEP1(config-mgmt-cvx)#server host 9.9.9.9 
VTEP1(config-mgmt-cvx)#no shut

VTEP2 configuration:

VTEP2(config)#interface Vxlan1
VTEP2(config-if-Vx1)#vxlan source-interface Loopback0 
VTEP2(config-if-Vx1)#vxlan contol-service vxlan udp-port 4789 
VTEP2(config-if-Vx1)#vxlan vlan 13 vni 101
VTEP2(config)#management cvx 
VTEP2(config-mgmt-cvx)#server host 9.9.9.9 
VTEP2(config-mgmt-cvx)#no shut

BGP EVPN

EVPN is a solution that helps with propagating the BUM Flood Lists, MAC-VTEP bindings and also the IP-MAC bindings (for ARP suppression)  for supporting EVPN as a control plane for L2 domains interconnected via VXLAN.  NOTE:please consult your SE for the platforms and versions where this functionality is supported. VTEP1 configuration:

VTEP1(config)#int vxlan 1  
VTEP1(config-if-Vx1)#vxlan source-interface lo0  
VTEP1(config-if-Vx1)#vxlan vrf virtual-network1 vni 101  
VTEP1(config-if-Vx1)#vrf definition virtual-network1   
VTEP1(config)#router bgp 100   
VTEP1(config-router-bgp)#neighbor 2.2.2.2 remote-as 100    
VTEP1(config-router-bgp-af)#address-family evpn    
VTEP1(config-router-bgp-af)#neighbor 2.2.2.2 activate   
VTEP1(config-router-bgp-af)#vrf virtual-network1     
VTEP1(config-router-bgp-af)#address-family ipv4       
VTEP1(config-router-bgp-af)#rd 1.1.1.1:0       
VTEP1(config-router-bgp-af)#route-target import 1234:4567890

VTEP2 configuration:

VTEP2(config)#int vxlan 1  
VTEP2(config-if-Vx1)#vxlan source-interface lo0  
VTEP2(config-if-Vx1)#vxlan vrf virtual-network1 vni 101  
VTEP2(config-if-Vx1)#vrf definition virtual-network1   
VTEP2(config)#router bgp 100   
VTEP2(config-router-bgp)#neighbor 1.1.1.1 remote-as 100    
VTEP2(config-router-bgp-af)#address-family evpn    
VTEP2(config-router-bgp-af)#neighbor 1.1.1.1 activate   
VTEP2(config-router-bgp-af)#vrf virtual-network1     
VTEP2(config-router-bgp-af)#address-family ipv4       
VTEP2(config-router-bgp-af)#rd 2.2.2.2:0       
VTEP2(config-router-bgp-af)#route-target import 4321:0987645
VTEP2(config-router-bgp-af)#route-target export 1234:4567890

Security threats and mitigation techniques

As VXLAN’s security is focused around the L2 encapsulation into L3, then the threats we are interested in here are: spoofing, flooding and traffic redirection. Additionally, we will discuss packet-injection attack which might be feasible in VXLAN environments. We will consider attacks coming from underlay and overlay networks.

Attacks from underlay networks

Underlay networks are usually located in the same administrative domain as the VTEP, it solely depends on the size of the topology. But since using co-location services are very common, physically control the access to network devices are not always feasible and we will consider underlay network as a potential place for compromised security. In our example, underlay network is connected to VTEP1 via interface eth16 as depicted below. Direction of attack is presented as red arrow: pic.2

MAC-flood eth16

Attacks from overlay networks

Overlay networks are created on top of underlay networks and are used for logical separation of tenants or parts of tenant networks in our case. Since the overlay network is something which is given to tenant as a service, it becomes the different administrative domain where security control in here becomes very limited. From a security perspective, this domain is a potential backdoor. We will consider two types of end-point connections in overlay networks: 1. L2 in access/trunk mode: 2. L3 connection. Directions of attack are presented on next two pictures: pic.3  

destinations

Mac-flooding

Mac-flooding in one tenant network can cause a disruption on the VTEP and stop the normal work of all tenants whose VLANs are terminated on that VTEP and in some cases, disrupt the work on remote VTEPs, depending on how the VXLAN domain is designed. There is a limit of MAC-addresses that can be learned by a VTEP, so if the attacker floods the network with random MAC-addresses, he can prevent the VTEP from learning legitimate MAC-addresses, causing it to act like a hub by broadcasting all traffic.

MAC-flooding in flood list type of configuration for VXLAN

Scenario 1: Attacker in an underlay network – MAC-flooding coming to interface eth16 of VTEP1 from DCI link. This attack will not be successful as DCI interface is a L3 interface and MAC-addresses are not learned there. In case eth16 would have been configured as a L2 in access or trunk mode, there would be a possibility for this attack. In such case, usage of MAC-address port security feature on an interface with non-aggressive restriction can be used. Thus, the legitimate traffic and usage of MAC-address table will not be impacted per this interface and at the same time, there will be no possibility to fill all available memory for MAC-addresses on the switch through one port. Configuration example:

VTEP1(config)#interface Ethernet16 
VTEP1(config-if-Et16)#switchport port-security maximum 500 
VTEP1(config-if-Et16)#switchport port-security violation protect log

Scenario 2: Attacker in the overlay network – MAC-flooding coming to interface eth1 of VTEP1 from the tenantA network. Attacking with this method on eth1 configured as an L2 interface in access mode for VLAN13 mapped to VNI 101 can be successful and put at risk the operation of hosts mapped to the same VNI in other locations – site2. In this case, usage of MAC-address port security feature on an interface with non-aggressive restriction can be used. It is understandable, that it is hard to control the number of MAC-address per tenant’s interface on the switch as it creates a burden of additional limitation for the customer. But, the question stands as either a formal limitation or the risk to compromise the work of a whole VXLAN domain, security measures should take priority. Configuration example from Scenario 1 is applicable here. The direction of the attack is presented in pic 3: In the case where the tenant is connected via an L3 interface, as for example eth8 on VTEP1, MAC-flood attack would not be possible. This is easy to explain by looking at how tenants packet are formed when they enter the port of VTEP. VXLAN traffic is encapsulated in a UDP packet with the following overhead on each of the packet: |outer Ethernet|UDP|IPv4|VXLAN| Based on it we can see that all the data arrived on the tenant interface on VTEP will be encapsulated into UDP and VTEPs CAM table would not be affected by this type of attack.

MAC-flooding in multicast group, CVX, BGP EVPN types of VXLAN configuration

Using other types of VXLAN configuration instead of the flooding list for VXLAN dataplane does not change the way of transport network – tenant network relations as well as with core transport network. All the security consideration related to MAC-flooding in VTEP flood list is relevant to them as well.

MAC-address spoofing

MAC-address spoofing typically involves being able to take over an MAC-address i.e., being able to send and receive traffic to and from that victim’s MAC-address. If this attack succeeds, it will influence the learning of MAC-addresses on VTEPs if dataplane learning is used.

MAC-address spoofing in flood list and type of VXLAN configuration

In this type of scenario, by design dataplane learning is enabled, so MAC-address spoofing is very easy to achieve on both underlay and overlay networks. This is achievable because by specifying the VTEPs flood list, we just limit the list of VTEPs which will receive the information from us and it does not limit current VTEPs from learning MAC-address database from rogue VTEP, wherever it is located. To mitigate this vulnerability, we can specify a list of VTEPs from which we can learn the MAC-address database. This solution is not optimal for large VXLAN deployments, but the issue of scalability can be resolved with using other methods of VXLAN configuration, we will speak about it later. Configuration example:

– learning from any

VTEP1(config-if-Vx1)#vxlan learn-restrict any

– using the flood list that the CLI or future controller set up

VTEP1(config-if-Vx1)#vxlan learn-restrict flood

– empty list – no dataplane learning

VTEP1(config-if-Vx1)#vxlan learn-restrict vtep

– list of prefixes

VTEP1(config-if-Vx1)#vxlan learn-restrict vtep 1.1.1.1 2.2.2.2

-list of prefixes IPv6

VTEP1(config-if-Vx1)#vxlan learn-restrict vtep fd12:3456:789a::/48

In addition, per VLAN/VNI learning list can be configured using the syntax:

– learning from any

VTEP1(config-if-Vx1)#vxlan vlan 13 learn-restrict any

– using the flood list that the CLI or future controller set up

VTEP1(config-if-Vx1)#vxlan vlan 13 learn-restrict flood

– list of prefixes

VTEP1(config-if-Vx1)#vxlan vlan 13 learn-restrict vtep 2.2.2.2

– fallback to using vlan-generic config

VTEP1(config-if-Vx1)#no vxlan vlan 13 learn-restrict vtep
MAC-address spoofing in multicast group VXLAN configuration

In this scenario, as in the VTEPs flood list, dataplane learning is enabled and MAC-address spoofing can be achievable, so mitigation is similar as well. However, there are additional steps which can be implemented: – not enabling PIM on interfaces where it is not required. This will prevent rogue VTEPs from joining legitimate multicast groups for VXLAN. PIM is disabled by default on the interface, no additional actions required. – in case IGMP is required on the interface, IGMP snooping filters can be applied, which will eliminate the possibility of joining IGMP groups used for VXLAN. Example:

VTEP1(config)#ip igmp profile list_1
VTEP1(config-igmp-profile-list_1)#permit
VTEP1(config-igmp-profile-list_1)#range 231.22.2.1 231.22.2.10

The ip igmp snooping filter command applies an IGMP profile to the configuration mode interface. These commands apply the list_1 snooping profile to Ethernet interface 7.

VTEP1(config)#interface ethernet 1
VTEP1(config-if-Et1)#ip igmp snooping filter list_1
MAC-address spoofing in CVX type of VXLAN configuration

If a controller is used and it programs the VTEP list and manages MAC-address learning, then there is no dataplane learning and MAC-spoofing would not be possible. On the other hand, when a controller is used in combination with dataplane learning, then MAC-address hijack i.e. fool the switch to lean the wrong combination of VTEP and MAC-address is possible. To mitigate this vulnerability we can use the learn-restrict feature, similar to the case of VTEP’s flood list.

MAC-address spoofing in BGP EVPN type of VXLAN configuration

In this case, since we are using normal BGP functionality to establish active peering sessions and negotiate all required parameters, we are already controlling who do we trust in regards of MAC-VTEP learning. Additional steps for hardening this would be: using neighbour authentication via setting password for TCP session authentication. Configuration example:

VTEP1(config)#router bgp 1
VTEP1(config-router-bgp)#neighbor 2.2.2.2 password 7 code123

ARP spoofing

ARP spoofing might be successful in any type of VXLAN environments as the weakness of ARP implementation is not changed there. The difference would be that if an attack is running in tenant network, the victim might be the tenant’s machines and since it is in a different administrative domain, there is not much operators of the transport network can do and this subject is beyond the scope of this post. When an attack is running in the overlay network, the victim would be transport equipment, as for example a VTEP1. In this case there is not much can be done on the side of VTEPs except statical arp entries for VTEP IP-addresses, but it is not very flexible solution and can be possibly run for directly connected devices. Running ARP inspection on the network may be useful, but it always the matter of current design of the overlay network. ARP inspection ensures that only valid ARP requests and responses are relayed. The switch performs these activities:

  • Intercepts all ARP requests and responses on untrusted ports
  • Verifies that each of these intercepted packets has a valid IP-to-MAC address binding before updating the local ARP cache or before forwarding the packet to the appropriate destination
  • Drops invalid ARP packets
The drawback of using ARP inspection is that it only verifies ARP packets and not IP packets. In this case spoof of MAC and IP address and sending unidirectional traffic to a host is still possible.

Static ARP configuration example:

VTEP1(config)#arp 10.0.4.4 0025.900e.c63c arpa

UDP flooding

UDP flood attack is a kind of DoS attack using UDP traffic. The goal of this attack if to flood the network with UDP packets, to force the victim to be unreachable from other hosts in the network. UDP traffic received from tenant’s network is processed in hardware without interception of CPU. As soon as attacker decides to send traffic to hosts with broadcast MAC-address as the destination, that traffic should be replicated and sent to all the VTEPs currently participating in VXLAN domain for particular VNI. This will not impose critical impact on the dataplane as all traffic from tenant’s network will be encapsulated and sent from the incoming VTEP. However, if the attacker is located in underlay network and sends UDP traffic destined to one of the VTEPs, then the VTEP will process that traffic and this might impact the dataplane operations. For that case, advised steps for hardening EOS operations should be used as a security measure. As for example, if an attacker is implementing ICMP request DoS attack on VTEP1 from underlay network, control-plane policy on VTEPs might be used to limit the amount of ICMP request destined to the VTEP. Configuration example:

VTEP1(config)#ip access-list system-icmp
VTEP1(config-acl-system-icmp)#permit icmp any any
VTEP1(config)#class-map type control-plane match-any class-system-icmp
VTEP1(config-cmap-control-plane-class-system-icmp)#match ip access-group system-icmp
VTEP1(config)#policy-map type control-plane copp-system-policy
VTEP1(config-pmap-control-plane-copp-system-policy)#class class-system-icmp
VTEP1(config-pmap-c-control-plane-copp-system-policy-class-system-icmp)#shape pps 3000
VTEP1(config-pmap-c-control-plane-copp-system-policy-class-system-icmp)#bandwidth pps 3000

Similar measures can be applicable to mitigate potential DoS attacks on device’s CPU by limiting amounts of allowed traffic flows based on its nature, as for example: DNS requests, routing protocols session initiation and so on. Let’s take a look on how this attack can be identified on the switch. First running trivial flood toward the VTEP1 from the underlay network:

#bash sudo ethxmit --ip-src=10.0.4.4 --ip-dst=10.0.4.16 -s 64 -n 100000 po1

Now, let’s see what counters on control-plane policy-map shows us:

VTEP1#sh policy-map interface control-plane copp-system-policy | b copp-system-default [diff]                           Dec 21, 2016 13:36:55
  Class-map: copp-system-default (match-any)
       shape : 8000 pps
       bandwidth : 1000 pps
      Out Packets : 58837
      Drop Packets : 343163

Here we will see a constant increase of the dropped packets counter for default CoPP class-map, while there is no harm to CPU, hence no production impact:

VTEP1#show proc top

top - 13:37:01 up 5:50, 1 user, load average: 0.31, 0.31, 0.25
Tasks: 279 total, 1 running, 278 sleeping, 0 stopped, 0 zombie
Cpu(s): 2.2%us, 0.6%sy, 0.0%ni, 97.1%id, 0.0%wa, 0.1%hi, 0.0%si, 0.0%st
Mem: 3844356k total, 3201856k used, 642500k free, 185912k buffers
Swap: 0k total, 0k used, 0k free, 1800524k cached

VTEP1#sh cpu counters queue

--------------------------------------------------------------------------
 Linecard0/0
--------------------------------------------------------------------------
Queue Counter/pkts Drops/pkts
--------------- ------------------- -------------------
Sflow 0 0
Acl Log 0 0
L3 Dest Miss 0 0
Other 80710 421290
class-system-icmp 1165 0

However, this “custom” configuration of CoPP class-maps is not supported on all the platforms:

VTEPX(config)#policy-map type control-plane copp-system-policy

% Unavailable command (not supported on this hardware platform)

The Control Plane Policing per Input interface can be used in the future releases. This feature extends normal CoPP by providing isolation within a type of traffic between the different input interfaces. NOTE: Please reach out to your System Engineer or Arista TAC for detailed information about the platform you are interested in.  

TCP SYN attacks

A TCP SYN also know as SYN flood attack is a form of denial-of-service attack in which an attacker sends a succession of SYN requests to a target’s system in an attempt to consume enough victim’s resources to make the system unresponsive to legitimate traffic. That attack is exploiting the nature of TCP and Transmission Control block in particular, which holds all the information about the connections to the host. There is a variety of attacks with TCP SYN basis such as direct, spoofed and distributed. In our case, we will be interested in mitigation of this on the underlay network as this is where we process traffic destined to the VTEP unlike the traffic from overlay network. Emulating attack with easy hping string:

attacker:~ attacker$ sudo hping3 -i u1 -S -p 22 10.0.4.16

VTEP1 statistics on received TCP SYNs:

[admin@VTEP1 ~]$ ss -a state SYN-RECV
 Recv-Q Send-Q Local Address:Port Peer Address:Port
 0 0 10.0.4.16:ssh 10.0.4.100:11773
 0 0 10.0.4.16:ssh 10.0.4.100:12417
 0 0 10.0.4.16:ssh 10.0.4.100:12017
 0 0 10.0.4.16:ssh 10.0.4.100:11892
 0 0 10.0.4.16:ssh 10.0.4.100:12506
 0 0 10.0.4.16:ssh 10.0.4.100:12695
 0 0 10.0.4.16:ssh 10.0.4.100:11864
 0 0 10.0.4.16:ssh 10.0.4.100:12577
 0 0 10.0.4.16:ssh 10.0.4.100:11850
 0 0 10.0.4.16:ssh 10.0.4.100:11941
 0 0 10.0.4.16:ssh 10.0.4.100:ardusuni
 0 0 10.0.4.16:ssh 10.0.4.100:12730
 0 0 10.0.4.16:ssh 10.0.4.100:12716
 0 0 10.0.4.16:ssh 10.0.4.100:11890

Emulating attack with hping:

sudo hping3 -S -c 1000 -w 64 -d 120 -p 80 10.0.4.16 --flood --rand-source

VTEP1 statistics on receiving TCP SYNs from the spoofed ip-addresses:

tcp 0 0 VTEP1:http 13.160.163.8:alpha-sms SYN_RECV
tcp 0 0 VTEP1:http 143.109.144.183:tcp-id-port SYN_RECV
tcp 0 0 VTEP1:http p221109133157.ppp:infocrypt SYN_RECV
tcp 0 0 VTEP1:http 48.86.245.123:bitspeer SYN_RECV
tcp 0 0 VTEP1:http 32.116.13.117:knetd SYN_RECV
tcp 0 0 VTEP1:http 5751f5a3.skybroadband.:xnds SYN_RECV
tcp 0 0 VTEP1:http 159.6.203.215:idware-router SYN_RECV
tcp 0 0 VTEP1:http 143.233.215.96:dc SYN_RECV
tcp 0 0 VTEP1:http 165.6.247.200:psdbserver SYN_RECV
tcp 0 0 VTEP1:http 130.63.184.195:dxadmind SYN_RECV
tcp 0 0 VTEP1:http 178.255.75.159:tapestry SYN_RECV
tcp 0 0 VTEP1:http 64.63.138.156:scrabble SYN_RECV
tcp 0 0 VTEP1:http 98.18.broadband18.io:stdptc SYN_RECV
tcp 0 0 VTEP1:http 18.191.83.94:linkname SYN_RECV
tcp 0 0 VTEP1:http 218.123.177.178:tdp-suite SYN_RECV
tcp 0 0 VTEP1:http 154.163.32.38:cr-websystems SYN_RECV

As we can see in this case VTEP1 is under attack from the directly connected network, but the nature of the spoofed source IP-address TCP-SYN attack makes the node to try to establish TCP connection with random hosts. What can be done – the same steps in order to harden EOS operations via control-plane policy.

VTEP1(config)#ip access-list TCP-SYN
VTEP1(config-acl-system-icmp)#permit tcp any any syn
VTEP1(config)#class-map type control-plane match-any class-tcp-syn
VTEP1(config-cmap-control-plane-class-system-icmp)#match ip access-group TCP-SYN
VTEP1(config)#policy-map type control-plane copp-system-policy
VTEP1(config-pmap-control-plane-copp-system-policy)#class class-tcp-syn
VTEP1(config-pmap-c-control-plane-copp-system-policy-class-system-icmp)#shape pps 1000
VTEP1(config-pmap-c-control-plane-copp-system-policy-class-system-icmp)#bandwidth pps 1000

Here we have to pay attention to some platform specific limitations. We are using for our testing VTEP1 and VTEP2 on DCS-7150S-64-CL which is based on Alta chipset and it supports only static class-maps for control-plane policy. Dynamic class-maps are not supported due to the limited number of CPU queues. Dynamic class-maps supported on Trident chipset, but there is a limitation in regards of the management interface, as at this stage this platform is not supporting CoPP on it. In this case, it will be specifically recommended to use additional security measure for OOB management network, but this topic is out of the scope of this post. Here is  the example of  verification commands on Trident based switches:

VTEP1#show platform trident copp mapping
======= QUEUE-DETAILS ========
Policy-map attached to control-plane: copp-system-policy
Control plane policing is enabled
Maximum number of queues supported: 48
Number of queues allocated: 34
Number of static queues allocated: 32
Number of dynamic queues allocated: 2
=======CLASS-MAPPING========
Class CosQ Type
----- ---- ----
copp-system-sflow 1 S
copp-system-acllog 2 S
copp-system-l3destmiss 3 S
copp-system-default 4 S
TCP-SYN 19 D
copp-system-icmp 20 D
copp-system-cvx 23 S
copp-system-cvx-heartbeat 24 S
copp-system-urm 25 S
copp-system-l3ttl1 26 S
copp-system-l3slowpath 27 S
copp-system-arpresolver 28 S
copp-system-arp 29 S
copp-system-tc3to5 30 S
copp-system-glean 31 S
copp-system-ipmcmiss 32 S
copp-system-igmp 33 S
copp-system-ipmcrsvd 34 S
copp-system-ptp 35 S
copp-system-pim 36 S
copp-system-OspfIsis 37 S
copp-system-bgp 38 S
copp-system-lldp 39 S
copp-system-vrrp 40 S
copp-system-tc6to7 41 S
copp-system-selfip 42 S
copp-system-selfip-tc6to7 43 S
copp-system-mlag 44 S
copp-system-bfd 45 S
copp-system-lacp 46 S
copp-system-bpdu 47 S
VTEP1#show platform trident tcam copp detail
=== Policy-map copp-system-policy type control-plane on switch Linecard0/0 ===
=-= Class-map TCP-SYN type control-plane =-=
--- ACL TCP-SYN uses 2 entries ===
Id | Seq | Proto | Src IP | Src Mask | Src Port |Hits
DSCP | Frag | Flags | Dest IP | Dest Mask | Dest Port |Action
22 |10 | TCP | *| *| *|0
 | N | S | *| *| *|Permit
23 |10 | TCP | *| *| *|0
 | Y | | *| *| *|Permit
=-= Class-map copp-system-icmp type control-plane =-=
--- ACL system-icmp uses 1 entries ===
Id | Seq | Proto | Src IP | Src Mask | Src Port |Hits
DSCP | Frag | Flags | Dest IP | Dest Mask | Dest Port |Action
21 |10 | ICMP | *| *| *|189
 | * | | *| *| *|Permit

This command will clear all the counters in the policy map for all classes:

VTEP1#clear policy-map interface control-plane counters copp-system-policy

BGP as a control plane and its security

The control plane mechanism is based on BGP, so we will briefly observe attacking scenarios for this protocol:

  • BGP route hijacking, when the attacker using BGP to announce illegitimate routes.
  • Man in the middle attack, when attacker diverting traffic to a rogue destination and giving himself access to it before it goes to it’s final destination.
  • Blackholing traffic (underlay/overlay) when all legitimate traffic is diverted to the wrong destination.

In its core, these types of attack are based on compromising BGP neighborship and countermeasures should be taken against it. In order to compromise BGP neighborship session, an attacker has to either:

  • Establish BGP session from malicious peer. This can be achieved by resetting current legitimate session by sending RST message to TCP stack, and then hijack the session from malicious BGP peer. However, different research results publicly available on the Internet shows that it is possible to insert a binary payload by listening to the sequence numbers on the wire by using tools like “tcphijack”.  Another way is to hijack the BGP session using ARP spoofing. It’s mitigation technique is observed in dedicated section of this post.  

One of the ways how we can protect peering from attacks from non-directly connected hosts is to use BGP TTL restriction. Thus we can program the switch to accept and attempt BGP connections to the external peers residing on networks not directly connected to the switch. The command does not establish the multihop if the only route to the peer is the default route (0.0.0.0). Here is the configuration example:

VTEP1(config)#router bgp 100
VTEP1(config-router-bgp)#neighbor 2.2.2.2 ebgp-multihop 2

However, this method of using fixed number of hops away doesn’t provide security against forged incoming packets with adjusted TTL value. Thus, BGP will not establish the session with such host because the TTL in the outgoing IP packet will expire after a few hops but this BGP speaker can still accept the initial TCP SYN for the BGP session making it vulnerable to the DOS attacks. And it will be recommended to use the feature GTSM for BGP described in RFC3682. The user can configure a minimum TTL for incoming IP packets received from the BGP peer. BGP session will only get established if the TTL value in the received IP packet header is greater than or equal to the configured value. If the received TTL is less than the configured value, the packet will be discarded. Here is the configuration example:

VTEP1(config-router-bgp)#neighbor 2.2.2.2 ttl-security hops 3

Another way of protection which can be considered is Unicast Reverse Path Forwarding check, this will allow the router to protect itself against attacks from spoofed IP-addresses and in this case – protect legitimate BGP peering. However, this feature has to be used with caution in case of peering with BGP route servers. uRPF is configurable on the interfaces. For packets arriving on a uRPF-enabled interfaces, the source IP address is verified by examining the source and destination addresses of unicast routing table entries. uRPF works in three different modes, strict mode, loose mode and strict mode with allow-default. In the strict mode, the packet must be received on the same L3 interface that the router would use to route the return packet. In the loose mode the router must have a route configured for the return packet, but the L3 interface in the routing table can be different from the L3 interface that it received the packet on. Strict mode with allow-default is similar to strict mode, but the packets can come in on the default interface as well. Here is the example of configuration uRPF in loose mode:

VTEP1(config)#interface Ethernet 16
VTEP1(config-if-Et16)#ip verify unicast source reachable-via any
  • Poisoning BGP updates for legitimate BGP peer by advertising non-existing or malicious routes. This can be achieved by crafting BGP update messages which will have all headers similar to the headers from legitimate BGP peer and sending it to the victim’s BGP peer.

In order to restrict unintentional or deliberate exploitation of BGP peer’s memory by flooding it with BGP malicious BGP advertisements, we can determine the number of BGP routes the switch accepts from a specified neighbor and defines an action when the limit is exceeded. The default value is 12,000. When the number of routes received from a peer exceeds the limit, the switch generates an error message.

VTEP1(config)#router bgp 100
VTEP1(config-router-bgp)#neighbor 2.2.2.2 maximum-routes 15000 warning-only

Another way to protect BGP peering is to use BGP AS-path access-lists and control specific AS paths among the advertised networks.

VTEP1(config)#ip as-path access-list list1 permit ^100$
VTEP1(config)#router bgp 100
VTEP1(config-router-bgp)#neighbor 2.2.2.2 prefix-list list1 in
This type of attacks is quite easy to be protected from by using BGP authentication based on strong passwords and strong encryption algorithms. An example of configuring MD5 password authentication for BGP peering:
VTEP1(config)#router bgp 100
VTEP1(config-router-bgp)#neighbor 2.2.2.2 password 7 .c0De.123
  • Compromise BGP peer by taking over the administrative control

Standard measures for preventing illegal access to network equipment has to be taken and applicable in this scenario very well. It might include physical protection of the equipment, using strong passwords, restricting the physical areas from where a device can be accessed remotely by applying GEO-IP based access lists and so on. It all left for user consideration and currently implemented security policies.

  • Compromise legitimate BGP session by making one or both peers unavailable for BGP peering by implementing DoS attack via exhausting the number of session peer can take.

This attack is easily implemented with TCP SYN messages flooding pointed to TCP port 179 used by BGP. There is a lot of material on the Internet describing various forms of attacks and its mitigation techniques including RFC 4987. Section TCP SYN flood attack of this post is also showing additional steps which can be taken in order to prevent the harm from this attack. Also, in our particular design, we consider the datacenter environment under administrative control and thus assuming that administrative measures should present. To be precise, we should consider any interruption of the service caused by the non-legitimate operations on the network as DoS attack, as services don’t have to be completely unavailable to cause issues in production. Even partial unavailability or constant interruptions will cause significant impact.  Thus, we should consider TCP Reset attack against BGP sessions as a serious threat as well. The goal of this attack is to just impact TCP and as collateral damage to impact BGP session. Even though a blind TCP Reset attack using the RST bit is not that straight forward for implementation, RFC 4953 provides a good observation on how this attack is plotted and also provides the considerations for mitigation. The main advice for users who operate in environments where it’s impossible to control physical access to the network infrastructure is to consider to use technologies dedicated to protect interconnection between remote hosts – IPS/IDS in the middle and IPsec tunnels as a fundamental solution for all remote locations.

Scalability considerations

Imagine a scenario when we had only a couple of VTEPs in VXLAN environment. In this case, we would probably use the easiest way of VTEPs configuration without the goal of scalability – Head-End-Replication with flood list including all VTEPs and putting multiple VLANs into single VNI. Now, if the number of VTEPs increased over time, the same style of configuration will still work, but it will posses scalability issues. Let’s take a look further. VTEP1 and VTEP2 are connecting tenants networks across the DCI. Host on tenantA on site:1 network is sending  DHCP OFFER packets in VLAN which is mapped to VNI 101 and propagated across all VTEPs in VXLAN environment according to flood list configuration. In this case, VTEP2 does not have interfaces pointed to tenantA on site:2, however it will receive DHCP discover packets anyway. According to forwarding logic on the switch, it will look for the association of VNI to VLAN in the local database and try to send the broadcast traffic there. When there is no corresponding VNI-VLAN mapping the packets will be forwarded to CPU and as there is no kernel listener on UDP port 4789, ICMP unreachable will be generated. With standard control-plane-acl, the packet will be silently dropped by iptables. VNI per VLAN. What would be recommended to eliminate unnecessary traffic floods to VTEPs missing VNIs: – use of VNI propagation per VTEP:

VTEP1(config-if-Vx1)#vxlan vlan 13 vni 101
VTEP1(config-if-Vx1)#vxlan vlan 15 vni 102
VTEP1(config-if-Vx1)#vxlan vlan 13 flood vtep 2.2.2.2
VTEP1(config-if-Vx1)#vxlan vlan 15 flood vtep 3.3.3.3

– use VXLAN learn restrict feature:

VTEP1(config-if-Vx1)#vxlan vlan 13 learn-restrict vtep 2.2.2.2
VTEP1(config-if-Vx1)#vxlan vlan 15 learn-restrict vtep 3.3.3.3

– instead of using flooding list for large VXLAN domains – switch to using controller: CVX with a variety of orchestration options on top.

Registering rogue VTEP on VXLAN controller

Each VTEP in a VXLAN domain maintains a set of BUM VTEP flood lists keyed by destination MAC address under each VLAN. Flooding packets are forwarded to all IP addresses in the list to reach remote hosts in the same VNI. The list can be configured via CLI on each VTEP. When VXLAN controller is enabled, all connected VTEPs advertise its own flood lists to VXLAN Controller (VCS) on CVX. VCS collates individual lists and generates centralised VTEP flood list for each VNI and publish the VNI-specific list to VTEPs in the VNI. If an external controller is connected to CVX via HSC, VCS will receive and publish flood VTEP lists from and to an external controller through HSC as well. However, if there is a new VTEP in the same VXLAN domain, VCS and other Arista VTEPs should not learn any information from it unless it is verified that new VTEP is legit. There is a mechanism to enable such verification. The solution is a static configuration of flood VTEP list in order to add non-Arista VTEPs in the VXLAN. That configured flood VTEP lists are stored locally in CVX sysdb. Configuration example:

cvx(config-cvx)#service vxlan
cvx(config-cvx-vxlan)#no shut
cvx(config-cvx-vxlan)#flood vtep 1.1.1.1 2.2.2.2

End-to-End security

By definition, secure data transport should provide confidentiality, integrity and data availability. Let’s take a look into each of these parts. VXLAN does not have features which would support secure transport of encapsulated data as at least one aspect of security is not met – confidentiality. It is not possible to send data across the VXLAN and assume that it is sent confidentially just because the structure of VXLAN packet does not provide us with this option and data is not encrypted.  Because of this, it is advised to use security protocol on DCIs such as IPSec. The same rule applies to any other DCI technology, would it be VLAN or MPLS based solution. Data availability is provided on the same level as L2 technology: VLAN – as it’s end hosts will manage data population and react to data losses in the flow. There is no much difference to other overlay technologies either. End-to-end integrity is preserved even while keeping in mind that a VXLAN packet contains only Outer CRC field and CRC of encapsulated packet is removed and not transported across the underlay network. In the case where a VXLAN packet will be intercepted in the underlay and the data in it will be changed – then CRC will have to recalculated and rewritten as well. CRC by itself was never used as a tool for data preserving the data integrity but as error-detection mechanism and it’s just not suitable for protecting against intentional alteration of data.  And once again – security options of using VXLAN native tools are very limited and to be fair – it was never it’s purpose.

Conclusion

Event though VXLAN was never designed to provide full, secure, end-to-end connectivity in modern multitenant environments, it is still possible to eliminate most obvious security threats with built-in tools of this popular feature. This post provided the most basic explanation and configuration examples of the most feasible way to operate such environments as a VXLAN domain in secure way. However, these measures alone are not enough to protect your network from all kind of security breaches and it is strongly recommended to use additional measures like enforce security policies and perform security audits of your IT infrastructure on regular basis.

Resources

There are a couple of posts on eos.arista.com portal which could serve as supplemental reading material on this topic: https://eos.arista.com/eos-4-17-1f/gtsm-for-bgp/ https://eos.arista.com/arista-eos-hardening-guide/ https://eos.arista.com/vxlan-with-mlag-configuration-guide/ https://eos.arista.com/vxlan-routing-with-mlag/ https://eos.arista.com/vxlan-without-controller-for-network-virtualization-with-arista-physical-vteps/ https://eos.arista.com/plan-to-migrate-otv-data-center-interconnect-dci-to-dci-with-vxlan/

Follow

Get every new post on this blog delivered to your Inbox.

Join other followers: