When Arista introduced R-series back in 2015 (publicly announced, available in 2016), it was a game changer. For the first time in networking history, a switch with merchant-silicon chips was capable to absorb full IPv4 and IPv6 internet tables. This valuable capability opened a full set of new addressable use cases to all vendors which leveraged off-the-shelf ASICs to implement the device forwarding plane. Since then, two more generations have been released, namely the R2 and R3 platforms and these platforms further enhanced both performances and, more relevant to our discussion, scaling.
Arista Networks has several routers to address Internet scale scenarios, namely the 7500R/R2/R3 (Modular Chassis), 7800R3 (modular chassis, R3-only) and 7280R/R2/R3 (fixed chassis). Yes, precisely because of the enhanced capabilities provided by the forwarding engine (and the carrier class operating system Arista EOS) we can call these devices routers and not switches.
In terms of forwarding plane capacity, which is the focus of our real world use case, the official figures of the various devices are:
For this real time deployment, the chosen device is the DCS-7280SR2K-48C6-M, which is part of the DCS-7280R2 family, and provides 48 x 10GE SFP+ and 6 x 100GE QSFP28 ports. The are multiple devices installed in a regional Tier-2 Italian service provider which is receiving full internet tables from its upstream Tier-1 SPs.
It’s worth to underline that in case of multiple BGP peers, the most impacted component of a device is the control plane as prefixes are learnt in multiple copies, one copy from each peer. From a forwarding plane standpoint, this is not very demanding as only one copy of the prefix is pushed to the FIB, eventually more than a next-hop will be added in the case where the BGP multipath feature is enabled, but the footprint of this next hop is negligible from a forwarding DB perspective.
As per Arista’s naming convention, the “K” letter indicates the device has expanded forwarding plane tables if compared to non-K models, therefore the expected outcome is to easily store both IPv4 and IPv6 full internet routing tables, leaving much more than 50% of the forwarding tables spaces free, either for future scaling needs or to accommodate other data plane features.
Before diving deep into our real world experience, let’s discuss some very simple yet important technologies under the hood in order to better understand all the CLI outputs that will follow.
In order to optimize route storage in the forwarding plane, which is one of the most precious resources inside the box, Arista released a feature called FlexRoute which allows the box to maximize the scaling of routing information leveraging one of the most important internal tables, that is the LEM (Large Exact Match). As it can only perform exact match, it is usually used to store fixed length entities such as host routes, MPLS labels, MAC addresses etc. By activating the FlexRoute feature, it will be possible to store prefixes different from /32 into the LEM. It’s worth to say that inside the TCAM there is another important table, named LPM (Longest Prefix Match), which is capable of performing variable length match lookups particularly important for routing. This writing doesn’t want to be a FlexRoute deep dive, it will only provide the needed information to better understand this particular use case.
FlexRoute readings can be found at the following links:
Now that we are familiar with the LPM and LEM concepts, we can finally take a look at our IPv4 and IPv6 full Internet table real world scenario.
Let’s quickly verify the hardware and the software we are going to use in this scenario (sensitive information are removed from the output):
7280SR2K#show version Arista DCS-7280SR2K-48C6-M-F Hardware version: 21.01 Serial number: xxxxxxxxxx System MAC address: xxxxxxxxxxx Software image version: 184.108.40.206F Architecture: x86_64 Internal build version: 220.127.116.11F-14409653.42302F Internal build ID: 8a9fab0d-2f3e-4fdc-a56f-f63f626e9a01 Uptime: 1 weeks, 4 days, 4 hours and 7 minutes Total memory: 32823716 kB Free memory: 30343288 kB
Everything looks just fine, we have a Jericho+ (R2) based switch with expanded FIB tables. Just as a piece of information, as the device has also a “-M” suffix, it means it also has 32GB RAM, as shown by the CLI output above. Needless to say, control plane memory is a paramount scaling factor in a BGP peering scenario like the one we are dealing with. It’s now time to configure the BGP peering to start receiving full Internet routing tables.
The BGP configuration is the following (AS Number and sensitive informations are changed to protect customer’s identity):
7280SR2K(config)# show running-config section router bgp router bgp 65503 neighbor E-BGP peer group neighbor E-BGP remote-as 65502 neighbor E-BGP description EBGP-FULL-INT neighbor E-BGP maximum-routes 0 neighbor E-BGP maximum-accepted-routes 0 neighbor 192.168.0.201 peer group E-BGP !
Let’s start receiving route updates without enabling FlexRoute and see how the forwarding tables are populated.
7280SR2K(config)#show ip bgp summary BGP summary information for VRF default Router identifier 192.168.38.202, local AS number 65503 Neighbor Status Codes: m - Under maintenance Neighbor V AS MsgRcvd MsgSent InQ OutQ Up/Down State PfxRcd PfxAcc 192.168.0.201 4 65502 542385 381 0 0 01:27:01 Estab 780642 780642
As shown above, we have a total of 780642 IPv4 routes.
Let’s take a look how those prefixes are stored in the forwarding engine:
7280SR2K(config)#show platform sand l3 summary | grep -A14 -m1 Lpm Jericho Lpm: Routes: 780644 unprogrammed: 0 Routes6: 1 unprogrammed6: 0 Backlog: 0 TCAM pivots used: 3274 Percent free: 60 Buckets used: 63740 rows used: 2291 Entries Per Bucket: 12 Percent free: 51 Lem: IPv4 Host in Lem : enabled IPv4 Hosts : 1 IPv4 Prefix-lengths in Lem: None IPv6 Host in Lem : disabled IPv6 Prefix-lengths in Lem: None IPv6 Routes per prefix-len:
Thanks to EOS CLI flexibility we can extract exactly the output we are interested in using native Linux commands such as ‘grep’. From this output we can see all the IPv4 routes are, as expected, stored in the LPM, as they are variable length prefixes by nature. Despite LPM is big enough to accommodate the full IPv4 internet table (indeed, 51% is still available!), in a typical BGP peering use case as this one, it’s a waste of resources not leveraging the LEM entries. That’s exactly where Arista FlexRoute technology comes into help.
To showcase how easy is to activate the feature, we will use the pre-defined INTERNET FlexRoute profile for both IPv4 and IPv6.
While the IPv6 INTERNET profile behaves similarly regardless TCAM profile configuration on the device, optimizing the /48 prefixes when moving them from LPM to LEM, the IPv4 profile may change its operations.
If the device is using a TCAM profile with the FlexRoute feature enabled, the INTERNET configuration sets /20 and /23 optimization with /19 and /22 expansion and /24 compression. On the other hand, if we don’t enable FlexRoute as a feature in the TCAM, it sets /20 and /24 optimization with /19 and /23 expansion.
In this case, the customer wants to leverage MPLS EVPN, hence the “mpls-evpn” TCAM profile being used, which includes the FlexRoute feature as well, and it’s active on the box therefore activating the former optimization. But let’s double check to be sure:
7280SR2K(config)#show hardware tcam profile mpls-evpn Features enabled in TCAM profile mpls-evpn: [ FixedSystem ] mpls evpn mpls multi-homing evpn mpls irb acl vlan ipv6 egress acl port ipv6 acl vlan ip acl subintf ip acl port ip tunnel vxlan acl port mac pbr mpls qos mac qos ip mpls pop ingress acl port ipv6 egress flex-route
Now, we know we should expect /23 and /20 IPv4 routes to be moved from LPM to LEM, as well as /22 and /19 which will be “expanded” into 2 /23 and 2 /20 prefixes respectively and then stored into the LEM. The same applies to “in pair” consecutive /24 prefixes that will be “compressed” into a single /23 to fit the LEM as well.
To get a deeper knowledge about all the available options of FlexRoute please refer to the following document:
Why the IPv4 and IPv6 INTERNET profiles were configured to move /23 and /20 (plus optimizing also /24, /19 and /22) and the /48 prefixes respectively from LPM to LEM?
Simply because, statistically speaking, these are the most used prefixes therefore moving them maximize the resource optimization of the forwarding table. Should LEM get full (precisely at 95%, a fallback machinery kicks in to eventually store the leftover prefixes back to LPM.
It’s now time to activate the feature using EOS CLI:
7280SR2K(config)#ip hardware fib optimize prefixes profile internet ! Please restart layer 3 forwarding agent to ensure IPv4 routes are optimized 7280SR2K(config)#ipv6 hardware fib optimize prefixes profile internet ! Please restart layer 3 forwarding agent to ensure IPv4 routes are optimized 7280SR2K(config)#
In order to apply the changes to the forwarding table, the L3 forwarding agent must be restarted.
Beware, this operation is traffic impacting, therefore is highly recommended to steer traffic away from the device before performing this action if it is done on live production traffic!
To restart L3 agent, the following command is configured:
7280SR2K(config)#agent SandL3Unicast terminate SandL3Unicast was terminated 7280SR2K(config)#
Now that FlexRoute is configured and active, let’s add IPv6 to the game and check what happens in the forwarding plane now when routes are learnt:
7280SR2K(config)#router bgp 65503 7280SR2K(config-router-bgp)#address-family ipv6 7280SR2K(config-router-bgp-af)#neighbor fd44:ba32:1211:1::2 activate 7280SR2K(config-router-bgp-af)#exit 7280SR2K(config-router-bgp-af)#neighbor fd44:ba32:1211:1::2 peer group E-BGP 7280SR2K(config-router-bgp-af)#
Now let’s check IPv4 and IPv6 BGP routes in the control plane and then see how the routes are stored now in the forwarding plane:
7280SR2K#show ip bgp summary BGP summary information for VRF default Router identifier 192.168.38.202, local AS number 65503 Neighbor Status Codes: m - Under maintenance Neighbor V AS MsgRcvd MsgSent InQ OutQ Up/Down State PfxRcd PfxAcc 192.168.38.201 4 65502 131496 23 0 0 00:08:16 Estab 788607 788607 7280SR2K#show ipv6 bgp summary BGP summary information for VRF default Router identifier 192.168.38.202, local AS number 65503 Neighbor Status Codes: m - Under maintenance Neighbor V AS MsgRcvd MsgSent InQ OutQ Up/Down State PfxRcd PfxAcc fd44:ba32:1211:1::2 4 65502 29514 24 0 0 00:08:19 Estab 78247 78247
Now that both IPv4 and IPv6 full internet tables are loaded, let’s look at the forwarding plane:
7280SR2K#show platform sand l3 summary | grep -A14 -m1 Lpm Jericho Lpm: Routes: 90957 unprogrammed: 0 Routes6: 40097 unprogrammed6: 0 Backlog: 0 TCAM pivots used: 618 Percent free: 92 Buckets used: 11806 rows used: 434 Entries Per Bucket: 11 Percent free: 90 Lem: IPv4 Host in Lem : disabled IPv4 Prefix-lengths in Lem: 20 23 IPv4 Routes per prefix-len: [/20]=79354 [/23]=512910 IPv6 Host in Lem : disabled IPv6 Prefix-lengths in Lem: 48 IPv6 Routes per prefix-len: [/48]=38188
As expected, most of the routes were moved from the LPM to the LEM. And the free percentages shown in the LPM increased from 51% to 90%, a big improvement, isn’t it?
Indeed, 90% LPM buckets are still free after almost 800,000 IPv4 and 80,000 IPv6 routes are already present in the forwarding plane. And this leaves plenty of room for other features as well as IP routing future scaling.
This is just a very simple use case but it shows very clearly the scaling capabilities of R-series coupled with a robust, flexible and capable operating system such as EOS.