• Configurations and Optimizations for Internet Edge Routing

 
 
Print Friendly, PDF & Email

Introduction

For many years, network deployments for enterprise Internet edge environments have consisted of dedicated routing platforms and a switching or aggregation layer to distribute this to various network zones.  With the advances in merchant silicon forwarding engines and the software expertise put into Arista’s Extensible Operating System (EOS), we can now fully replace this legacy architecture with a collapsed routing and switching layer using Arista R Series platforms.  Arista R Series platforms allow for holding a full copy of the Internet routing table for both IPv4 and IPv6 in hardware (the Forwarding Information Base, or FIB) with plenty of headroom of growth for years to come.  In addition, the state-based architecture of EOS allows for multiple copies of the Internet table to be held in the BGP control-plane (the Routing Information Base, or RIB) so that multiple sources and Internet feeds can be evaluated and BGP best-path selection can occur per customer requirements.  The goal of this article will be to detail some of the configurations that can enable and enhance this capability.  Configurations and features in this article will be based off of the latest release of EOS at the time of this articles publishing, EOS-4.23.2F and Arista’s R Series hardware platforms.  Please note that different R Series models will have different scale, so anytime you are deciding on a hardware platform or software version, please consult an Arista SE for details.

A Note on 32 vs. 64 Bit EOS

Arista has been publishing both 32-bit and 64-bit versions of EOS since 4.22.0F.  In small to medium enterprises with only one or two internet feeds per edge environment, the standard 32-bit version of EOS will be more than sufficient.  However, for environments where RIB scale will be large and there will be many internet feeds for the hardware to evaluate for best-path, the 64-bit version of EOS will allow for much better performance.  As mentioned, please consult with your Arista SE when doing any software selection based on the requirements of your deployment.

Arista Multi-Agent BGP Configuration

For Internet edge deployments, increased control-plane scale, responsiveness and flexibility is a must.  As such, it is recommended to enable multi-agent BGP inside of EOS.  The multi-agent routing model allows for higher scale in the BGP control-plane as well as many pieces of enhanced functionality that we will discuss below.  Please note that this configuration will require a reboot to take effect, so it is recommended to set this before implementation.

inet-edge#config
inet-edge(config)#service routing protocols model multi-agent
! Change will take effect only after switch reboot
inet-edge(config)#end
inet-edge#write
Copy completed successfully.
inet-edge#

Arista FlexRoute Configuration

Arista’s FlexRoute technology is a software and hardware innovation that allows EOS to draw more capability out of merchant silicon platforms than anyone else.  It does this by optimizing certain prefix lengths so they are more efficiently stored in hardware tables.  To get the full benefit of this feature, a TCAM profile must be applied to the system.  Similar to the multi-agent configuration above, an agent restart or reboot is required for the change to take effect.

First we configure the TCAM profile.  This requires disabling certain features that are not commonly used at the Internet edge.  The profile below is just one such example.  If your deployment does require these features, please contact your Arista SE to come up with different TCAM profile that will work for your installation.

inet-edge#config
inet-edge(config)#hardware tcam
inet-edge(config-hw-tcam)#profile FLEX-ROUTE copy default
inet-edge(config-hw-tcam-profile-FLEX-ROUTE)#feature flex-route copy system-feature-source-profile
inet-edge(config-hw-tcam-profile-FLEX-ROUTE-feature-flex-route)#exit
inet-edge(config-hw-tcam-profile-FLEX-ROUTE)#no feature acl vlan ip
inet-edge(config-hw-tcam-profile-FLEX-ROUTE)#no feature mirror ip
inet-edge(config-hw-tcam-profile-FLEX-ROUTE)#exit
Saving new profile 'FLEX-ROUTE'
inet-edge(config-hw-tcam)#system profile FLEX-ROUTE
inet-edge(config-hw-tcam)#exit
inet-edge(config)#

Once the TCAM profile is applied, you can enable FlexRoute compression and expansion of prefixes.  While these commands can be customized for different prefix lengths, Arista has included pre-defined ‘internet’ profiles for both IPv4 and IPv6 based on the current contents of the Internet routing table.

inet-edge(config)#ip hardware fib optimize prefixes profile internet
! Please restart layer 3 forwarding agent to ensure IPv4 routes are optimized
inet-edge(config)#ipv6 hardware fib optimize prefixes profile internet
! Please restart layer 3 forwarding agent to ensure IPv6 routes are optimized
inet-edge(config)#end
inet-edge#write
Copy completed successfully.
inet-edge#agent SandL3Unicast terminate
SandL3Unicast was terminated
inet-edge#

A full write-up of FlexRoute in a live environment, showing the optimizations that can occur, can be found in a great write-up by another SE in the article titled: DCS-7280SR2K with FLEXROUTE – A real world use case.

A full breakdown of the feature can be found in the TOI here.

Changing ACL Implementation

Another optimization we recommend for edge deployments utilizing FlexRoute is to change the internal mechanism the switch router will use to implement Access-Control Lists.  The hardware typically utilized in high-scale edge routing deployments supports Arista AlgoMatch technology, which is enabled by default.  However, to ensure proper allocation of system resources, it is preferable at this time to change the implementation to leverage the system TCAM for ACLs.  This also requires a reboot, and can be configured at the same time as multi-agent routing and FlexRoute, prior to a reboot to knock them all out at once.

inet-edge#config
inet-edge(config)#hardware access-list mechanism tcam
! Change will take effect only after switch reboot.
inet-edge(config)#end
inet-edge#write
Copy completed successfully.
inet-edge#reload now

Adjusting show tech-support Outputs

Now that our platform is allocated properly, we can begin looking at other configurations to help in optimizing our edge router.  One that doesn’t cross most engineers’ minds on the first pass is the adjustment of what the ‘show tech-support’ command collects.  With one or more copies of the Internet routing table in the BGP RIB and a full copy of the table in the FIB, the size of the output and the amount of time to collect it from a ‘show tech-support’ can be counter-productive.  In addition, Arista EOS will automatically collect and archive a ‘show tech-support’ output on box every hour.  While this is great, it can lead to filling up the device flash sooner than would be ideal.  What can be helpful is to remove certain commands from the ‘show tech-support’ output so that the size of the archives are drastically reduced.  In addition, we can put the collection of the commands we remove on a separate schedule.  This allows us to still collect and store the outputs, but to do so less frequently to conserve storage space.  To remove the commands with the largest file size impact, use the following commands.

inet-edge#config
inet-edge(config)#management tech-support
inet-edge(config-mgmt-tech-support)#policy show tech-support
inet-edge(config-tech-spt-policy-show-tech)#exclude command show platform fap ip route
inet-edge(config-tech-spt-policy-show-tech)#exclude command show platform fap ipv6 route
inet-edge(config-tech-spt-policy-show-tech)#exclude command show ip bgp vrf all
inet-edge(config-tech-spt-policy-show-tech)#exclude command show ipv6 bgp vrf all
inet-edge(config-tech-spt-policy-show-tech)#exclude command show kernel ip route vrf all
inet-edge(config-tech-spt-policy-show-tech)#exclude command show kernel ipv6 route vrf all
inet-edge(config-tech-spt-policy-show-tech)#exclude command show ip route vrf all detail
inet-edge(config-tech-spt-policy-show-tech)#exclude command show ipv6 route vrf all detail
inet-edge(config-tech-spt-policy-show-tech)#exit
inet-edge(config-mgmt-tech-support)#exit
inet-edge(config)#

Once that is completed, we can create a separate CLI schedule to collect the outputs of those commands on a less frequent basis.  Since we want to collect multiple commands, we can create a multi-line alias first, then reference that in our scheduled task. The example here will collect the outputs once a day and store them for seven days.  These can of course be adjust to each environment’s requirements.

inet-edge(config)#alias inet-edge-routes
inet-edge(config-alias-inet-edge-routes)#10 echo ---show platform fap ip route---
inet-edge(config-alias-inet-edge-routes)#20 show platform fap ip route
inet-edge(config-alias-inet-edge-routes)#30 echo ---show platform fap ipv6 route---
inet-edge(config-alias-inet-edge-routes)#40 show platform fap ipv6 route
inet-edge(config-alias-inet-edge-routes)#50 echo ---show ip bgp vrf all---
inet-edge(config-alias-inet-edge-routes)#60 show ip bgp vrf all
inet-edge(config-alias-inet-edge-routes)#70 echo ---show ipv6 bgp vrf all---
inet-edge(config-alias-inet-edge-routes)#80 show ipv6 bgp vrf all
inet-edge(config-alias-inet-edge-routes)#90 echo ---show kernel ip route vrf all---
inet-edge(config-alias-inet-edge-routes)#100 show kernel ip route vrf all
inet-edge(config-alias-inet-edge-routes)#110 echo ---show kernel ipv6 route vrf all---
inet-edge(config-alias-inet-edge-routes)#120 show kernel ipv6 route vrf all
inet-edge(config-alias-inet-edge-routes)#130 echo ---show ip route vrf all---
inet-edge(config-alias-inet-edge-routes)#140 show ip route vrf all
inet-edge(config-alias-inet-edge-routes)#150 echo ---show ipv6 route vrf all---
inet-edge(config-alias-inet-edge-routes)#160 show ipv6 route vrf all
inet-edge(config-alias-inet-edge-routes)#exit
inet-edge(config)#schedule inet-edge-routes interval 1440 timeout 30 max-log-files 7 command inet-edge-routes
inet-edge(config)#end
inet-edge#write
Copy completed successfully.
inet-edge#

ISP Peering Configurations and Optimization

Probably the most critical settings for an ISP peering router are the configurations relating to the BGP peering itself.  These can have the most direct impact on normal behavior as well as how the router will behave if something goes amiss.

Validating Hardware before Advertising

With the size of the Internet routing table, there are a large number of prefixes to program into the hardware FIB.  While very fast and efficient, this can still take some time as EOS agents update the forwarding tables.  To prevent any mismatch in state between the BGP control-plane and the data-plane, we can tell BGP to not advertise any prefixes to neighbors until they are programmed in hardware.  This prevents a router from advertising reachability before it is actually ready to forward packets to that destination.

inet-edge#config
inet-edge(config)#router bgp 12345
inet-edge(config-router-bgp)#update wait-install
inet-edge(config-router-bgp)#

Enabling Fast Failure Detection with BFD

Bidirectional Forwarding Detection (BFD) allows for detection of neighbor down events faster than standard BGP timers would allow for. This requires support from the ISP as well.  BFD works in conjunction with the routing protocol, in this case BGP, to monitor the connectivity between the peers and to notify BGP if there is an issue. BFD timers can be tuned to align with what the ISP will support. Transmitting BFD packets is done by the ASIC under certain circumstances and will therefore be offloaded from the CPU (Hardware Accelerated BFD Transmit).

inet-edge(config-router-bgp)#neighbor 11.11.11.11 bfd
inet-edge(config-router-bgp)#exit
inet-edge(config)#bfd interval 500 min_rx 500 multiplier 3 default
inet-edge(config)#

Enabling missing route-map handling (RFC 8212 behavior)

RFC 8212 does state, that if ‘no route-map is applied’ to a BGP neighbor, it should not accept nor advertise any prefixes from this particular neighbor. This is to avoid unwanted prefix distribution to peers and/or acceptance of leaked prefixes. To mitigate this behavior a network should enable the ‘BGP missing policy action. The described behavior is documented here and here.

inet-edge(config)#router bgp 12345 
inet-edge(config-router-bgp)#bgp missing-policy direction in action deny-in-out
inet-edge(config-router-bgp)#bgp missing-policy direction out action deny-in-out
inet-edge(config-router-bgp)#

Removing Private BGP ASNs

Normally, BGP will keep the entire BGP as-path intact when passing routes to upstream ISPs that it received from downstream routers.  In most cases, an internal network will use private BGP ASNs. It is best practice to remove this from the as-path so that all routes appear to come directly from the ISP edge network.

inet-edge(config-router-bgp)#neighbor 11.11.11.11 remove-private-as all
inet-edge(config-router-bgp)#

Enabling BGP Communities

Most ISPs today will accept certain community values. The values themselves can vary and should be discussed with your ISP. However, it is a good practice to enable them to be able to make use of some of the helpful features they can provide such as adjusting the ISPs BGP local preference values or using the BGP Graceful Shutdown community for maintenance. Note that configuring send-community with no additional keywords will enable both standard and extended communities.

inet-edge(config-router-bgp)#neighbor 11.11.11.11 send-community
inet-edge(config-router-bgp)#

Adjusting community processing order

As a security measure on the internet edge you might want to clear all communities inbound carrying your own ASN (eg. if your ASN is 12345, then you might want to clear 12345:*), before adding on your own informational and control communities. The behavior how communities are processed has been changed in EOS 4.20.1F (Respect ‘set community’ clause order between route-map sequences) and should be set ‘sequential’ rather than ‘merged.

inet-edge(config)#service routing configuration route-map set-operations sequential
inet-edge(config)#

Adjusting BGP Maximum-Routes Value

EOS provides a way to limit the amount of routes a BGP peer can send to the local router. This is enabled by default with a value of 12,000. If this value is exceeded, EOS will disconnect the BGP session.  In the case of an ISP edge device with a full internet table, we of course want to accept more than 12,000 routes. The easiest way to accomplish this is to simply set the value to 0, which disables the limit. This value should only changed on a per-neighbor basis if you are expecting more than 12000 prefixes being received from your neighbor. A maximum-routes limit should still be maintained for all other peers!

inet-edge(config-router-bgp)#neighbor 11.11.11.11 maximum-routes 0
inet-edge(config-router-bgp)#

Filtering Extraneous or Malicious Routes

While you can generally trust your ISP, it is best to take a prudent approach to securing your edge router. One such measure is to filter any incoming BGP routes for networks that are not allocated to the Internet and should never appear in a BGP update from an ISP. These include things such as RFC1918 addressing. There are many lists maintained online that can be referenced for what prefixes to filter. We will use a simple example here of only RFC1918 filtering. Note that EOS does support updating a prefix-list from a URL source, so this could be updated dynamically as well.  Information on that can be found at the article here. The Bogon list used below is published by Team Cymru and is publicly available: https://www.team-cymru.com/bogon-reference.html


inet-edge(config-router-bgp)#neighbor 11.11.11.11 route-map NO-BOGONS in
inet-edge(config-router-bgp)#exit
inet-edge(config)#route-map NO-BOGONS deny 100
inet-edge(config-route-map-NO-BOGONS)#match ip address prefix-list BOGONS
inet-edge(config-route-map-NO-BOGONS)#exit
inet-edge(config)#ip prefix-list BOGONS seq 100 permit 0.0.0.0/8 
inet-edge(config)#ip prefix-list BOGONS seq 110 permit 10.0.0.0/8 le 32
inet-edge(config)#ip prefix-list BOGONS seq 120 permit 100.64.0.0/10 
inet-edge(config)#ip prefix-list BOGONS seq 130 permit 127.0.0.0/8
inet-edge(config)#ip prefix-list BOGONS seq 140 permit 169.254.0.0/16
inet-edge(config)#ip prefix-list BOGONS seq 150 permit 172.16.0.0/12 le 32
inet-edge(config)#ip prefix-list BOGONS seq 160 permit 192.0.0.0/24
inet-edge(config)#ip prefix-list BOGONS seq 170 permit 192.0.2.0/24
inet-edge(config)#ip prefix-list BOGONS seq 180 permit 192.168.0.0/16 le 32
inet-edge(config)#ip prefix-list BOGONS seq 190 permit 198.18.0.0/15
inet-edge(config)#ip prefix-list BOGONS seq 200 permit 198.51.100.0/24
inet-edge(config)#ip prefix-list BOGONS seq 210 permit 203.0.113.0/24
inet-edge(config)#ip prefix-list BOGONS seq 220 permit 224.0.0.0/3
inet-edge(config)#exit
inet-edge#write
Copy completed successfully.
inet-edge#

BGP Prefix Independent Convergence

Many enterprise ISP edge environments will make use of two or more ISPs to provide redundancy and some measure of load-balancing.  In the event that one ISP path fails, we want all traffic to seamlessly fail over to the other carrier.  This will happen by default based on BGP best-path calculation and convergence.  However, when dealing with a large routing table, like the Internet, it can take a while for the new best path to be re-computed for each prefix and programmed into hardware.  In most cases, it is preferred to instead pre-calculate and store the backup path before there is an event.  This is what BGP Prefix Independent Convergence provides.  Since multiple paths are stored in the RIB, care must be taken to ensure the system has appropriate resources.  Please consult an Arista SE if you have questions about scale and hardware capacity.

Enabling this feature is simple and will tell the router to calculate two paths based on the BGP metrics available.

inet-edge#config
inet-edge(config)#router bgp 12345
inet-edge(config-router-bgp)#bgp additional-paths install
inet-edge(config-router-bgp)#end
inet-edge#write
Copy completed successfully.
inet-edge#

A full breakdown of this feature can be found in the TOI here.

Resource Public Key Infrastructure

RPKI is used for validation of prefixes based on prefix length and Origin AS. Mostly it will protect ISP environments from misconfigured peer networks and BGP hijacks. As a prerequisite a RPKI cache server is needed to be up and running in the ISPs network. It will then provide a list of prefixes and the router can match against those to determine is a prefix is deemed ‘valid’, ‘invalid’ or ‘unknown’ (not RPKI signed).

Please note that the below route-map example is not adding up with any previous route-maps in this article and has to be adjusted according to your environment.

More details can be found is this TOI: Standalone BGP Origin Validation with RPKI

inet-edge#config
inet-edge(config)#router bgp 12345
inet-edge(config-router-bgp)#rpki cache cache1
inet-edge(config-rpki-cache)#host 23.23.23.23
inet-edge(config-rpki-cache)#exit
inet-edge(config-router-bgp)#rpki origin-validation
inet-edge(config-rpki-origin-validation)#ebgp local
inet-edge(config-origin-validation)#exit
inet-edge(config-router-bgp)#neighbor 11.11.11.11 route-map RPKI in
inet-edge(config-router-bgp)#exit

inet-edge(config)#route-map RPKI permit 10
inet-edge(config-route-map-RPKI)#match origin-as validity valid
inet-edge(config-route-map-RPKI)#set local-preference 200
inet-edge(config-route-map-RPKI)#set community 65535:10
inet-edge(config-route-map-RPKI)#route-map RPKI permit 20
inet-edge(config-route-map-RPKI)#match origin-as validity not-found
inet-edge(config-route-map-RPKI)#set local-preference 100
inet-edge(config-route-map-RPKI)#set community 65535:20
inet-edge(config-route-map-RPKI)#route-map RPKI deny 30
inet-edge(config-route-map-RPKI)#match origin-as validity invalid

Increasing Network Visibility at the Edge

It is critical to monitor the health and stability of an Internet edge device.  Normal means like SNMP and more modern and responsive methods like real-time streaming telemetry with CVP will not be covered in detail here.  Instead we will focus on a few methods typically used in conjunction with securing the Internet edge.

sFlow

sFlow is used to sample packets as they traverse a device and send flow record information of the packet to a central collector.  This can provide visibility as to who and what is actually traversing the network, but can also be used in conjunction with many security monitoring and detection services to identify potential attacks on the network.  sFlow also supports a number of extensions to provide more visibility to the flows from the router’s perspective, such as next-hop information or BGP related path information.

inet-edge#config
inet-edge(config)#sflow run
inet-edge(config)#sflow sample 10000
inet-edge(config)#sflow source-interface Loopback0
inet-edge(config)#sflow destination 23.23.23.23 2055
inet-edge(config)#sflow extension bgp
inet-edge(config)#

All EOS platforms support sFlow, which uses the system CPU to provide the sampling.  To get more granular sample rates with higher-rate interfaces, some Arista R-Series platforms also support Hardware Accelerated sFlow.

inet-edge(config)#sflow hardware acceleration
inet-edge(config)#sflow hardware acceleration sample 1024
inet-edge(config)#

A full breakdown of this feature can be found in the TOI here.

BGP Monitoring Protocol

Getting a full view of all BGP route advertisements when dealing with the Internet routing table can be challenging or time consuming via normal methods.  To provide a more modern approach, BGP Monitoring Protocol (BMP) was introduced to allow passive monitoring stations to collect BGP table information from routers to provide a unified and dynamically updating view of the network.  Arista EOS devices support sending BGP RIB information to a BMP collector.  The RIB information can be sent before route-map filtering is applied or after.

inet-edge(config)#router bgp 12345
inet-edge(config-router-bgp)#bgp monitoring
inet-edge(config-router-bgp)#monitoring station COLLECTOR
inet-edge(config-router-bgp-monitoring-station-COLLECTOR)#connection address 45.45.45.45
inet-edge(config-router-bgp-monitoring-station-COLLECTOR)#update-source Loopback0
inet-edge(config-router-bgp-monitoring-station-COLLECTOR)#export-policy received routes pre-policy
inet-edge(config-router-bgp-monitoring-station-COLLECTOR)#end
inet-edge#write
Copy completed successfully.
inet-edge#

A full breakdown of this feature can be found in the TOI here.

Summary

While this article covers many optimizations and best practice recommendations, it is not all encompassing. An edge router configuration should take special care to secure the device as it is usually fully exposed to the Internet at large.  The following article contains many general recommendations for securing an EOS device.

Arista EOS Hardening Guide

As mentioned previously, please consult your Arista SE to discuss any of these recommendations further.  With the wide array of features available in EOS software and the best-of-breed merchant silicon-based hardware, Arista can provide the most flexible and secure edge to your enterprise network.

Follow

Get every new post on this blog delivered to your Inbox.

Join other followers: