Maintenance Mode Lab – Example of BGP on Spine

Maintenance Mode

Introduced in Arista’s EOS 4.15.2F, Maintenance Mode is a method to allow for easy maintenance of a switch or specific elements of a switch. The goal is to provide a set of commands with a wide range of flexibility that make our network operations lives a bit simpler. And along the way try to help drive down human error. With Maintenance Mode we expect to make the removal and reintroduction of a whole switch or portions of the switch a graceful operation that minimizes network downtime. The initial introduction of Maintenance Mode was aimed at BGP, Interfaces and the Switch as a whole.

When Arista releases new features we include feature specific documentation referred to as TOIs (Transfer of Information). To become familiar with a new feature we highly recommend reading the TOIs, the Configuration Guide and running through the features in a lab environment. The TOI introducing Maintenance Mode can be found here https://eos.arista.com/eos-4-15-2f/maintenance-mode/

 

Take a look at this basic network diagram. This network is built as a Layer 3 Leaf/Spine with eBGP as the routing protocol.

 

For this lab we are focusing on an upcoming maintenance window where we need to upgrade Spine1 (highlighted, top left). Prior to the Maintenance Mode feature we would need to consider how we might go about removing Spine1 from the production data path to minimize the network disruption during this maintenance activity. We could use various BGP options including manually assigning communities to the updates from Spine1 and then configuring all other devices to exclude those routes or we could manually inflate the AS Path with the AS Path Prepend option to make all updates from Spine1 to be the least preferred options.

Instead of doing this all manually at the increased risk of human error, why not take advantage of a feature that handles all of that heavy lifting for us. This is where Maintenance Mode for BGP steps in. Rather than having to create new policies, apply them for the maintenance and removing them once the maintenance is complete, we can simply leverage the automation built into Maintenance Mode for BGP to drain the traffic away from Spine1, perform the maintenance and re-introduce Spine1 back into production all the while minimizing packet loss.

In this article we are going to play with the BGP portion of Maintenance Mode. Before we get into the configuration we should take a moment and get to know the terminology introduced with this feature.

 

Terminology

Unit = this is the most fundamental element of the Maintenance Mode operation. Placing the ‘unit’ in Maintenance Mode allows for graceful removal of that element from the production network. A unit can be defined as a single interface, a linecard, a BGP peer or the whole switch. Units can be a collection of various groups.

Dynamic Units = this can be created using individual interfaces, an interface range command, a BGP peer or peer group. If a unit representing these various elements doesn’t already exist, one will be created dynamically. In the need of a policy, the policy for the dynamic group will be selected from existing policies already associated to members of this dynamic group.

Groups = the groups concept allows you to combine various group types. In the context of Maintenance Mode, the group types are currently interfaces and BGP peers. Specific to Maintenance Mode, when we refer to BGP peers as groups we are not talking about BGP Peer Groups which is a configuration construct within router bgp configuration mode. Out of the gate there are some built-in default groups such as the linecard group. VLANs and port-channels (logical type interfaces) are not associated to the linecard group because it is expected that these elements can span across multiple linecards. When thinking about Groups as they relate to Maintenance Mode, keep in mind that we’re targeting elements that should be associated together while performing network maintenance.

Profiles = profiles offer us a way to apply a policy or set of policies for both hardware and software units. An interface policy sets rules such as if an interface should be immediately shutdown once entering maintenance mode or if we should include link utilization thresholds to determine when the interface is ready to enter Maintenance Mode. A unit profile at the System level could include on-boot settings for the unit that forces it to be placed in Maintenance Mode and to exit Maintenance Mode after a specified time period after reboot. For BGP, a profile would call the appropriate route-map to be applied to enter Maintenance Mode. Default profiles are already associated to Interface, BGP and System units. Or you can create custom profiles to fit your needs.

Rate Monitoring = we mentioned that within a profile we can set link utilization levels to determine when an interface should go into Maintenance Mode. Rate monitoring is a configurable value where we specify the maximum input and output loads. This value is measured from the interface’s load-interval. By setting a value we can determine when we feel the interface is sufficiently drained of traffic and is safe to enter Maintenance Mode. You have the choice of using a shutdown option within the interface profile signaling that the interface is to be shutdown upon entering Maintenance Mode. In addition to link utilization, we can set a max-delay as well. Max-delay specifies the number of seconds we’re willing to wait for traffic to drain off before the interface is shutdown.

Route-Map = we mentioned the use of a route-map when we discussed BGP and profiles. Route-maps are used as part of the BGP profile to tag inbound and outbound routes. The default profile associated to BGP tags these routes using the GSHUT community. The use of route-maps provides lots of flexibility. You can choose to use other BGP communities or features such as AS Path Prepend.

On-Boot Maintenance = often times we don’t want a switch to come fully back into service immediately upon reboot. Typically it is best to allow all protocol relationships to establish and for the routing table to fully converge. This leads to better device insertion and less packet loss. There are two methods for putting a device in Maintenance Mode when it boots up. 1) The unit was put into Maintenance Mode before rebooting/reloading and the device’s running configuration was saved prior to going down. 2) The on-boot property within the profile of the unit depicts that the unit be placed into Maintenance Mode as part of the boot-up process. The duration of which can be specified. This duration is effective as soon as the device boots up and realizes that it is to be placed into Maintenance Mode.

 

Maintenance Mode Commands and Configuration Snippets

Now that we’ve covered some of the terminology, lets walk through some of the commands that were introduced with Maintenance Mode. These were taken from the Arista TOI (Transfer of Information) document that was published when this feature was introduced.

Maintenance of the entire switch via unit System

l1#config

l1(config)#maintenance

l1(config-maintenance)#unit System

l1(config-builtin-unit-System)#quiesce

The above commands will place the entire switch in maintenance. At this point the behavior of the groups, System unit, interface units and BGP units will be determined from either the default profiles associated to those units or any custom profiles that were made to be the default profile for any specific unit.

 

Maintenance of a single interface

l1#config

l1(config)#maintenance

l1(config-maintenance)#interface Ethernet2

l1(config-maint-if-Et2)#quiesce

The above commands will place interface Ethernet 2 in maintenance. The subsequent behavior will depend on the profile associated specifically to this interface or the default profile associated to all interfaces.

 

Maintenance of a single line card in Chassis switches

7500E-1#config

7500E-1(config)#maintenance

7500E-1(config-maintenance)#linecard3

7500E-1(config-maint-if-Et2)#quiesce

The above commands will place this line card in maintenance. The behavior that follows will depend on the profile associated to this unit whether that be the default profile or a customized one.

 

Maintenance of a single BGP neighbor

l1#config

l1(config)#maintenance

l1(config-maintenance)#bgp 172.16.12.2

l1(config-maint-bgp-172.16.12.2)#quiesce

These BGP related maintenance commands will be applied to the specific peer with the resulting behavior driven by the profile associated specifically or the default BGP profile that is system generated.

 

Maintenance of the System On-Boot

l1#config

l1(config)#maintenance

l1(config-maintenance)#profile unit LEAF1

l1(config-profile-unit-LEAF1)#on-boot duration ?

<300-3600>  Number of seconds for the on-boot duration timer

The On-Boot option provides a way to put a device in maintenance once it fully boots up. The duration timers is based on when the device boots and recognizes that it needs to go into maintenance mode.

 

Group commands

l1(config)#maintenance

l1(config-maintenance)#group interface MM-INTERFACE

l1(config-group-if-MM-INTERFACE)#interface Ethernet1-2

l1(config-group-if-MM-INTERFACE)#interface vlan 112

l1(config-group-if-MM-INTERFACE)#interface port-channel10

 

Above is an example of using the maintenance group command to associate units into a group when those units are of the same type. Above is putting interfaces into an interface group. Notice that you can associate a single interface, an interface range, a VLAN interface and/or a port-channel into an interface group. Below we create a BGP group and associate specific neighbors to that group.

 

l1(config)#maintenance

l1(config-maintenance)#group bgp LEAF1-BGP-PEERS

l1(config-group-bgp-LEAF1-BGP-PEERS)#nei 172.16.12.2

l1(config-group-bgp-LEAF1-BGP-PEERS)#nei 172.16.200.1

 

Profile commands

l1#config

l1(config)#maintenance

l1(config-maintenance)#profile ?

bgp        Configure bgp

interface  Configure interface

unit       Maintenance unit

 

As you can see from the output above we have several profiles we can build. Depending on which the associated unit type a profile is associated with will dictate the options we have within that profile. Here are a few examples.

 

l1(config-maintenance)#profile bgp BGP-PROFILE-ONE

l1(config-profile-bgp-BGP-PROFILE-ONE)#initiator route-map BGP-PROFILE-ONE-RM inout ?

l1(config-maintenance)#profile interface LEAF1-INTERFACES

l1(config-profile-intf-LEAF1-INTERFACES)#rate-monitoring load-interval 30

l1(config-profile-intf-LEAF1-INTERFACES)#rate-monitoring threshold 65000

l1(config-maintenance)#profile unit LEAF1-UNIT

l1(config-profile-unit-LEAF1-UNIT)#on-boot duration 300 ?

 

Some other Maintenance Mode commands (not exhaustive)

show maintenance

show maintenance bgp

show maintenance bgp ip all

show maintenance bgp receiver route-map

show maintenance debug bgp

show maintenance debug interface

show maintenance debug unit

show maintenance summary

show maintenance units

 

Lab Topology

For this paper we’re going to use the topology shown below. This Layer 3 Leaf/Spine topology provides us plenty of flexibility to run through multiple test scenarios for Arista’s Maintenance Mode. This topology takes advantage of several Arista features including Layer 3 routing, ECMP and Layer 2 MLAG. There are two MLAG domains in this topology. One MLAG Domain consists of Leaf 1 (L1) and Leaf 2 (L2). And another MLAG Domain between Leaf 3 (L3) and Leaf 4 (L4). This topology mimics a common network deployment where compute hosts are provided Layer 1 and Layer 2 redundancy by connecting to two Leafs with those Leafs in an MLAG configuration. The Leafs also take advantage of Arista’s vARP feature. vARP is an Arista FHRP feature that provides Default Gateway redundancy for the compute hosts/guests.

Note; this topology is built using Arista’s vEOS (virtual EOS) and constructed via Ravello Systems (www.ravellosystems.com). Arista SEs can easily provide access to this blueprint.

 

 

Lab, Verify, Lab, Verify

In addition to reading the appropriate documentation, there’s nothing better to help in understanding a feature than to lab it up. We’re going to run through examples of Arista’s Maintenance Mode turning various nerd knobs along the way.

 

Pre-Lab Verification

Let’s make sure that our lab is configured to match the diagram and verify reachability between all devices. Here are some abbreviated outputs of our device configurations along with routing table information and interface status.

Note; the Maintenance Mode for BGP option relies on the well-known BGP community of GSHUT. The IETF draft has gone through multiple revisions. At the time of this lab we are referring to draft-ietf-grow-bgp-gshut-06 (https://tools.ietf.org/html/draft-ietf-grow-bgp-gshut-06). Because we are relying on a community we have to configure our BGP neighbor relationships to send community information to one another. They do not do this by default.

 

s1#sh version

<output omitted>

Software image version: 4.15.2F

Architecture:           i386

Internal build version: 4.15.2F-2663444.4152F

Internal build ID:      0ebbad93-563f-4920-8ecb-731057802b9c

 

s1#show ip int brief

Interface              IP Address         Status     Protocol         MTU

Ethernet1              unassigned         up         up              1500

Ethernet2              172.16.200.1/30    up         up              1500

Ethernet3              172.16.200.5/30    up         up              1500

Ethernet4              172.16.200.9/30    up         up              1500

Ethernet5              172.16.200.13/30   up         up              1500

Loopback0              172.16.0.1/32      up         up             65535

Loopback1              unassigned         up         up             65535

Management1            192.168.0.10/24    up         up              1500

 

s1#show ip bgp sum

BGP summary information for VRF default

Router identifier 172.16.0.1, local AS number 65001

Neighbor Status Codes: m – Under maintenance

Neighbor         V  AS           MsgRcvd   MsgSent  InQ OutQ  Up/Down State  PfxRcd PfxAcc

172.16.200.2     4  65101             18        19    0    0 00:11:05 Estab  4      4

172.16.200.6     4  65102             21        22    0    0 00:11:01 Estab  4      4

172.16.200.10    4  65103             17        19    0    0 00:10:43 Estab  4      4

172.16.200.14    4  65104             17        19    0    0 00:10:46 Estab  4      4

 

s1#sh run sec bgp

logging level BGP errors

!

router bgp 65001

router-id 172.16.0.1

maximum-paths 4 ecmp 4

neighbor 172.16.200.2 remote-as 65101

neighbor 172.16.200.2 send-community

neighbor 172.16.200.2 maximum-routes 12000

neighbor 172.16.200.6 remote-as 65102

neighbor 172.16.200.6 send-community

neighbor 172.16.200.6 maximum-routes 12000

neighbor 172.16.200.10 remote-as 65103

neighbor 172.16.200.10 send-community

neighbor 172.16.200.10 maximum-routes 12000

neighbor 172.16.200.14 remote-as 65104

neighbor 172.16.200.14 send-community

neighbor 172.16.200.14 maximum-routes 12000

network 172.16.0.1/32

 

s2#sh version

<output omitted>

Software image version: 4.15.2F

Architecture:           i386

Internal build version: 4.15.2F-2663444.4152F

Internal build ID:      0ebbad93-563f-4920-8ecb-731057802b9c

 

s2#sh ip int brief

Interface              IP Address         Status     Protocol         MTU

Ethernet2              172.16.200.17/30   up         up              1500

Ethernet3              172.16.200.21/30   up         up              1500

Ethernet4              172.16.200.25/30   up         up              1500

Ethernet5              172.16.200.29/30   up         up              1500

Loopback0              172.16.0.2/32      up         up             65535

Management1            192.168.0.11/24    up         up              1500

 

s2#sh ip bgp sum

BGP summary information for VRF default

Router identifier 172.16.0.2, local AS number 65002

Neighbor Status Codes: m – Under maintenance

Neighbor         V  AS           MsgRcvd   MsgSent  InQ OutQ  Up/Down State  PfxRcd PfxAcc

172.16.200.18    4  65101             22        22    0    0 00:12:53 Estab  7      7

172.16.200.22    4  65102             24        24    0    0 00:12:49 Estab  7      7

172.16.200.26    4  65103             25        23    0    0 00:12:33 Estab  7      7

172.16.200.30    4  65104             25        23    0    0 00:12:35 Estab  7      7

 

s2#sh run sec bgp

logging level BGP errors

!

router bgp 65002

router-id 172.16.0.2

maximum-paths 4 ecmp 4

neighbor 172.16.200.18 remote-as 65101

neighbor 172.16.200.18 send-community

neighbor 172.16.200.18 maximum-routes 12000

neighbor 172.16.200.22 remote-as 65102

neighbor 172.16.200.22 send-community

neighbor 172.16.200.22 maximum-routes 12000

neighbor 172.16.200.26 remote-as 65103

neighbor 172.16.200.26 send-community

neighbor 172.16.200.26 maximum-routes 12000

neighbor 172.16.200.30 remote-as 65104

neighbor 172.16.200.30 send-community

neighbor 172.16.200.30 maximum-routes 12000

network 172.16.0.2/32

 

l1#sh version

<output omitted>

Software image version: 4.15.2F

Architecture:           i386

Internal build version: 4.15.2F-2663444.4152F

Internal build ID:      0ebbad93-563f-4920-8ecb-731057802b9c

 

l1#sh ip int brief

Interface              IP Address         Status     Protocol         MTU

Ethernet2              172.16.200.2/30    up         up              1500

Ethernet3              172.16.200.18/30   up         up              1500

Loopback0              172.16.0.3/32      up         up             65535

Management1            192.168.0.14/24    up         up              1500

Vlan12                 172.16.112.2/24    up         up              1500

Vlan4094               172.16.12.1/30     up         up              1500

 

l1#sh ip bgp sum

BGP summary information for VRF default

Router identifier 172.16.0.3, local AS number 65101

Neighbor Status Codes: m – Under maintenance

Neighbor         V  AS           MsgRcvd   MsgSent  InQ OutQ  Up/Down State  PfxRcd PfxAcc

172.16.12.2      4  65102             24        24    0    0 00:15:17 Estab  7      7

172.16.200.1     4  65001             23        22    0    0 00:15:20 Estab  5      5

172.16.200.17    4  65002             24        24    0    0 00:15:18 Estab  5      5

 

l1#sh run sec bgp

logging level BGP errors

!

router bgp 65101

router-id 172.16.0.3

maximum-paths 4 ecmp 4

neighbor 172.16.12.2 remote-as 65102

neighbor 172.16.12.2 send-community

neighbor 172.16.12.2 maximum-routes 12000

neighbor 172.16.200.1 remote-as 65001

neighbor 172.16.200.1 send-community

neighbor 172.16.200.1 maximum-routes 12000

neighbor 172.16.200.17 remote-as 65002

neighbor 172.16.200.17 send-community

neighbor 172.16.200.17 maximum-routes 12000

network 172.16.0.3/32

network 172.16.112.0/24

 

l2#sh ver

<output omitted>

Software image version: 4.15.2F

Architecture:           i386

Internal build version: 4.15.2F-2663444.4152F

Internal build ID:      0ebbad93-563f-4920-8ecb-731057802b9c

 

l2#sh ip int brief

Interface              IP Address         Status     Protocol         MTU

Ethernet2              172.16.200.6/30    up         up              1500

Ethernet3              172.16.200.22/30   up         up              1500

Loopback0              172.16.0.4/32      up         up             65535

Management1            192.168.0.15/24    up         up              1500

Vlan12                 172.16.112.3/24    up         up              1500

Vlan4094               172.16.12.2/30     up         up              1500

 

l2#sh ip bgp sum

BGP summary information for VRF default

Router identifier 172.16.0.4, local AS number 65102

Neighbor Status Codes: m – Under maintenance

Neighbor         V  AS           MsgRcvd   MsgSent  InQ OutQ  Up/Down State  PfxRcd PfxAcc

172.16.12.1      4  65101             25        25    0    0 00:16:47 Estab  7      7

172.16.200.5     4  65001             27        26    0    0 00:16:45 Estab  7      7

172.16.200.21    4  65002             27        27    0    0 00:16:43 Estab  7      7

 

l2#sh run sec bgp

logging level BGP errors

!

router bgp 65102

router-id 172.16.0.4

maximum-paths 4 ecmp 4

neighbor 172.16.12.1 remote-as 65101

neighbor 172.16.12.1 send-community

neighbor 172.16.12.1 maximum-routes 12000

neighbor 172.16.200.5 remote-as 65001

neighbor 172.16.200.5 send-community

neighbor 172.16.200.5 maximum-routes 12000

neighbor 172.16.200.21 remote-as 65002

neighbor 172.16.200.21 send-community

neighbor 172.16.200.21 maximum-routes 12000

network 172.16.0.4/32

network 172.16.112.0/24

 

l3#sh version

<output omitted>

Software image version: 4.15.2F

Architecture:           i386

Internal build version: 4.15.2F-2663444.4152F

Internal build ID:      0ebbad93-563f-4920-8ecb-731057802b9c

 

l3#sh ip int brief

Interface              IP Address         Status     Protocol         MTU

Ethernet2              172.16.200.10/30   up         up              1500

Ethernet3              172.16.200.26/30   up         up              1500

Loopback0              172.16.0.5/32      up         up             65535

Management1            192.168.0.16/24    up         up              1500

Vlan34                 172.16.134.2/24    up         up              1500

Vlan4094               172.16.34.1/30     up         up              1500

 

l3#sh ip bgp sum

BGP summary information for VRF default

Router identifier 172.16.0.5, local AS number 65103

Neighbor Status Codes: m – Under maintenance

Neighbor         V  AS           MsgRcvd   MsgSent  InQ OutQ  Up/Down State  PfxRcd PfxAcc

172.16.34.2      4  65104             28        28    0    0 00:17:50 Estab  7      7

172.16.200.9     4  65001             26        24    0    0 00:17:54 Estab  7      7

172.16.200.25    4  65002             27        29    0    0 00:17:52 Estab  7      7

 

l3#sh run sec bgp

logging level BGP errors

!

router bgp 65103

router-id 172.16.0.5

maximum-paths 4 ecmp 4

neighbor 172.16.34.2 remote-as 65104

neighbor 172.16.34.2 send-community

neighbor 172.16.34.2 maximum-routes 12000

neighbor 172.16.200.9 remote-as 65001

neighbor 172.16.200.9 send-community

neighbor 172.16.200.9 maximum-routes 12000

neighbor 172.16.200.25 remote-as 65002

neighbor 172.16.200.25 send-community

neighbor 172.16.200.25 maximum-routes 12000

network 172.16.0.5/32

network 172.16.134.0/24

 

l4#sh version

<output omitted>

Software image version: 4.15.2F

Architecture:           i386

Internal build version: 4.15.2F-2663444.4152F

Internal build ID:      0ebbad93-563f-4920-8ecb-731057802b9c

 

l4#sh ip int brief

Interface              IP Address         Status     Protocol         MTU

Ethernet2              172.16.200.14/30   up         up              1500

Ethernet3              172.16.200.30/30   up         up              1500

Loopback0              172.16.0.6/32      up         up             65535

Management1            192.168.0.17/24    up         up              1500

Vlan34                 172.16.134.3/24    up         up              1500

Vlan4094               172.16.34.2/30     up         up              1500

 

l4#sh ip bgp sum

BGP summary information for VRF default

Router identifier 172.16.0.6, local AS number 65104

Neighbor Status Codes: m – Under maintenance

Neighbor         V  AS           MsgRcvd   MsgSent  InQ OutQ  Up/Down State  PfxRcd PfxAcc

172.16.34.1      4  65103             30        30    0    0 00:19:00 Estab  7      7

172.16.200.13    4  65001             28        26    0    0 00:19:06 Estab  6      6

172.16.200.29    4  65002             28        31    0    0 00:19:04 Estab  6      6

 

l4#sh run sec bgp

logging level BGP errors

!

router bgp 65104

router-id 172.16.0.6

maximum-paths 4 ecmp 4

neighbor 172.16.34.1 remote-as 65103

neighbor 172.16.34.1 send-community

neighbor 172.16.34.1 maximum-routes 12000

neighbor 172.16.200.13 remote-as 65001

neighbor 172.16.200.13 send-community

neighbor 172.16.200.13 maximum-routes 12000

neighbor 172.16.200.29 remote-as 65002

neighbor 172.16.200.29 send-community

neighbor 172.16.200.29 maximum-routes 12000

network 172.16.0.6/32

network 172.16.134.0/24

 

Verify reachability before we break stuff

A pre-change health check is just as important as a post-change health check. We need to verify full network reachability. The hosts in our lab are Arista vEOS switches but instead of including them in any routing protocol we simply have a default route (0/0) pointing to the vARP VIP address of their MLAG Domain.

From the hosts in the lab, run pings to all of the IP addresses in the lab topology. For brevity, we’ll show only pings from the two hosts to each other. But for full verification before our change we ran pings from each host to each IP address included in the topology.

host1#ping 172.16.134.201

rtt min/avg/max/mdev = 80.005/88.805/104.007/8.915 ms, ipg/ewma 103.006/95.954 ms

host2#ping 172.16.112.201

rtt min/avg/max/mdev = 82.396/94.266/115.165/11.854 ms, ipg/ewma 110.360/104.723 ms

 

Example Layer 3 eBGP Leaf/Spine – Spine1 via BGP Maintenance Mode of a single BGP peer

In this example we are going to place Spine1 into Maintenance Mode via BGP to a single BGP peer.

Issue the show maintenance and show maintenance summary commands to verify that this device is not currently under maintenance.

 

s1#show maintenance

Flags:

o – On-boot maintenance

v – Violating traffic threshold

 

Unit Name                    Status                        Time since last change      Flags

—————————- —————————- ——————————- —–

System                       Not Under Maintenance                  never

 

s1#show maintenance summary

Number of Units Configured: 0

Number of Units Exiting Maintenance: 0

Number of Units Entering Maintenance: 0

Number of Units Not Under Maintenance: 1

Number of Units Under Maintenance: 0

Directly Put Under Maintenance:

Number of interfaces Entering Maintenance: 0

Number of interfaces Under Maintenance: 0

Number of bgp peers Entering Maintenance: 0

Number of bgp peers Under Maintenance: 0

Rate Monitoring:

Number of interfaces Entering Maintenance: 0

Number of interfaces Under Maintenance: 0

Number of interfaces Under Maintenance with threshold violation: 0

Number of interfaces shutdown for maintenance: 0

 

The show maintenance units command will list for us any configurations (default) on Spine1.

 

s1#show maintenance units

Unit Name: System

Origin: Built-in

Status: Not Under Maintenance

Unit Profile: Default

Time Since Last State Change: never

Bgp Groups:

AllBgpNeighborVrf-default

Interface Groups:

AllEthernetInterface

 

We can see that only the default maintenance units are in the configuration. We have the built-in System unit. We are still set to use the default profile for the System unit and that the default groups of AllBgpNeighborVrf-default and AllEthernetInterface exist already for us.

The default profile associated to Maintenance Mode for BGP can be viewed with the show maintenance profiles bgp default command. Here we see what Spine1 is going to apply when we place the BGP relationship to Leaf1 into maintenance.

 

s1#show maintenance profiles bgp default

Bgp Profile: Default

Initiator route-map: SystemGenerated

route-map SystemGenerated permit 10

Description:

description System generated initiator route-map

Match clauses:

Set clauses:

set community GSHUT additive

set local-preference 0

 

Before we put the BGP relationship between Spine1 and Leaf1 in maintenance, make sure you issue the command term mon on all the routers so that you can see log messages painted to your console screen. Additionally, lets start a continuous ping from host1 to the loopback of Spine1 to see if we experience any packet loss.

 

Now jump into configuration mode, maintenance and specifically the BGP relationship with Leaf1 (172.16.200.2).

s1#config

s1(config)#maintenance

s1(config-maintenance)#bgp 172.16.200.2

s1(config-maint-bgp-172.16.200.2)#quiesce

 

The first thing that we notice is a logging messages on Spine1 indicating that the BGP neighbor relationship has been placed into maintenance mode.

 

Dec 15 16:58:29 s1 MaintenanceMode: %MMODE-5-MAINT_UNIT_STATE_CHANGE: Maintenance unit state changed for unit <Dynamic Unit><172.16.200.2><vrf-default>. Old State active, New State maintenanceModeEnter

Dec 15 16:58:29 s1 Rib: %BGP-6-MAINTENANCE-MODE: peer 172.16.200.2 is placed under maintenance.

Dec 15 16:58:29 s1 MaintenanceMode: %MMODE-5-MAINT_UNIT_STATE_CHANGE: Maintenance unit state changed for unit <Dynamic Unit><172.16.200.2><vrf-default>. Old State maintenanceModeEnter, New State underMaintenance

 

Lets check the status of Maintenance Mode on Spine1 to see what that output looks like. Issue the commands show maintenance and show maintenance bgp ip all.

 

s1#show maintenance

Flags:

o – On-boot maintenance

v – Violating traffic threshold

 

Unit Name                    Status                        Time since last change      Flags

—————————- —————————- ——————————- —–

System                       Not Under Maintenance                  never

 

Bgp Neighbor(vrf: defa       Status                        Time since last change      Flags

—————————- —————————- ——————————- —–

172.16.200.2                 Under Maintenance                   0:07:29 ago

 

s1#show maintenance bgp ip all

BGP peer maintenance information for VRF default

Router identifier 172.16.0.1, local AS number 65001

Neighbor: 172.16.200.2

    Maintenance state: Under Maintenance

    Maintenance route-map: SystemGenerated

Neighbor: 172.16.200.6

Not Under Maintenance

Neighbor: 172.16.200.10

Not Under Maintenance

Neighbor: 172.16.200.14

Not Under Maintenance

 

Check the output of show ip bgp sum on both Spine1 and Leaf1 to see if anything has changed.

 

s1#sh ip bgp sum

BGP summary information for VRF default

Router identifier 172.16.0.1, local AS number 65001

Neighbor Status Codes: m – Under maintenance

Neighbor         V  AS           MsgRcvd   MsgSent  InQ OutQ  Up/Down State  PfxRcd PfxAcc

m 172.16.200.2     4  65101             63        67    0    0 00:53:48 Estab  7      7

172.16.200.6     4  65102             63        65    0    0 00:53:44 Estab  4      4

172.16.200.10    4  65103             61        65    0    0 00:53:26 Estab  6      6

172.16.200.14    4  65104             61        65    0    0 00:53:29 Estab  6      6

 

Note above that the output on Spine1 has a new flag. The ‘m’ indicates that this neighbor relationship is under maintenance.

 

In the output below from show ip bgp sum on Leaf1 we don’t see any changes. The BGP relationship between Leaf1 and Spine1 looks just as it did during our pre-change health check.

 

l1#sh ip bgp sum

BGP summary information for VRF default

Router identifier 172.16.0.3, local AS number 65101

Neighbor Status Codes: m – Under maintenance

Neighbor         V  AS           MsgRcvd   MsgSent  InQ OutQ  Up/Down State  PfxRcd PfxAcc

172.16.12.2      4  65102             60        62    0    0 00:50:55 Estab  7      7

172.16.200.1     4  65001             65        60    0    0 00:50:58 Estab  7      7

172.16.200.17    4  65002             58        61    0    0 00:50:56 Estab  6      6

 

Check the routing table of Leaf1 to see if anything has changed.

 

l1#show ip route bgp

VRF name: default

Codes: C – connected, S – static, K – kernel,

O – OSPF, IA – OSPF inter area, E1 – OSPF external type 1,

E2 – OSPF external type 2, N1 – OSPF NSSA external type 1,

N2 – OSPF NSSA external type2, B I – iBGP, B E – eBGP,

R – RIP, I L1 – ISIS level 1, I L2 – ISIS level 2,

A B – BGP Aggregate, A O – OSPF Summary,

NG – Nexthop Group Static Route

 

B E    172.16.0.1/32 [200/0] via 172.16.12.2, Vlan4094

B E    172.16.0.2/32 [200/0] via 172.16.200.17, Ethernet3

B E    172.16.0.4/32 [200/0] via 172.16.12.2, Vlan4094

B E    172.16.0.5/32 [200/0] via 172.16.200.17, Ethernet3

B E    172.16.0.6/32 [200/0] via 172.16.200.17, Ethernet3

B E    172.16.134.0/24 [200/0] via 172.16.200.17, Ethernet3

 

Indeed, we no longer see any routes with the next-hop out Ethernet2 (via 172.16.200.1). Do we still have reachability from host1 to the loopback of Spine1? Did we lose any pings?

 

host1#ping 172.16.0.1 repeat 1000

PING 172.16.0.1 (172.16.0.1) 72(100) bytes of data.

80 bytes from 172.16.0.1: icmp_req=1 ttl=63 time=60.0 ms

— 172.16.0.1 ping statistics —

1000 packets transmitted, 1000 received, 0% packet loss, time 59183ms

rtt min/avg/max/mdev = 48.003/57.815/120.008/8.277 ms, pipe 2, ipg/ewma 59.242/56.559 ms

 

No loss. 0% packet loss, 1000 packets sent and 1000 packets received. Nice. A lossless removal of a BGP relationship.

 

How did Spine1 notify Leaf1 that the BGP relationship was going into maintenance? On Arista switches we have the luxury of running tcpdump. Lets take a look at tcpdump on interface Ethernet2 of Leaf1 to see what BGP messages were sent from Spine1 when we went into maintenance. The command we’ll use is tcpdump interface ethernet 2 verbose.

 

l1#tcpdump interface ethernet 2 verbose

tcpdump: listening on et2, link-type EN10MB (Ethernet), capture size 65535 bytes

17:10:26.249718 2c:c2:60:e4:f7:19 > 2c:c2:60:fd:3d:06, ethertype IPv4 (0x0800), length 121: (tos 0xc0, ttl 1, id 3065, offset 0, flags [DF], proto TCP (6), length 107)

172.16.200.1.bgp > 172.16.200.2.36032: Flags [P.], seq 2822119957:2822120012, ack 4191984501, win 114, options [nop,nop,TS val 946005 ecr 930593], length 55: BGP, length: 55

Update Message (2), length: 55

Origin (1), length: 1, Flags [T]: IGP

0x0000:  00

AS Path (2), length: 6, Flags [T]: 65001

0x0000:  0201 0000 fde9

Next Hop (3), length: 4, Flags [T]: 172.16.200.1

0x0000:  ac10 c801

Community (8), length: 4, Flags [OT]: 65535:0

0x0000:  ffff 0000

Updated routes:

172.16.0.1/32

 

The above snippet is just one of several BGP updates that were sent from Spine1 to Leaf1. We’ve included only one for brevity. We can verify who sent this packet by looking at the source MAC, destination MAC, source IP and destination IP information. Notice that we’ve highlighted the BGP Community section of the update. We see a community associated to this update and it is 0xffff0000. Checking the IETF RFC information we find in Section 7 that this flag indicates the GSHUT community.

 

We can also check the routes from Spine1 in Leaf1’s BGP table to see the GSHUT community information there as well. The show ip bgp neighbor 172.16.200.1 received-routes community GSHUT command will show us all routes from Spine1 that have the GSHUT community associated to them.

 

l1#show ip bgp nei 172.16.200.1 received-routes community GSHUT

BGP routing table information for VRF default

Router identifier 172.16.0.3, local AS number 65101

Route status codes: s – suppressed, * –  valid, > – active, # – not installed, E – ECMP head, e – ECMP

S – Stale, c – Contributing to ECMP,  b – backup

Origin codes: i – IGP, e – EGP, ? – incomplete

AS Path Attributes: Or-ID – Originator ID, C-LST –  Cluster List, LL Nexthop – Link Local Nexthop

 

Network             Next Hop          Metric  LocPref Weight Path

*      172.16.0.1/32       172.16.200.1     0       0       –       65001 i

*      172.16.0.2/32       172.16.200.1     0       0       –       65001 65102 65002 i

*      172.16.0.4/32       172.16.200.1     0       0       –       65001 65102 i

*      172.16.0.5/32       172.16.200.1     0       0       –       65001 65103 i

*      172.16.0.6/32       172.16.200.1     0       0       –       65001 65104 i

*      172.16.112.0/24     172.16.200.1     0       0       –       65001 65102 i

*      172.16.134.0/24     172.16.200.1     0       0       –       65001 65104 i

 

Cool. So now lets suppose that we’re done with our maintenance and we’re ready to bring this BGP peering relationship back into production. We simply use the no form of the command with a no quiesce. Again, lets start up a ping and make sure we have term mon set on our sessions to watch for any updates.

 

s1#config

s1(config)#maintenance

s1(config-maintenance)#bgp 172.16.200.2

s1(config-maint-bgp-172.16.200.2)#no quiesce

 

Similar to before, we immediately get log messages displayed to our SSH session that indicates we’ve come out of maintenance for that BGP session.

Dec 15 17:23:48 s1 MaintenanceMode: %MMODE-5-MAINT_UNIT_STATE_CHANGE: Maintenance unit state changed for unit <Dynamic Unit><172.16.200.2><vrf-default>. Old State underMaintenance, New State maintenanceModeExit

Dec 15 17:23:48 s1 Rib: %BGP-6-MAINTENANCE-MODE: peer 172.16.200.2 is taken out of maintenance.

Dec 15 17:23:48 s1 MaintenanceMode: %MMODE-5-MAINT_UNIT_STATE_CHANGE: Maintenance unit state changed for unit <Dynamic Unit><172.16.200.2><vrf-default>. Old State maintenanceModeExit, New State active

 

Issue the show maintenance and show maintenance bgp ip all commands to verify we are no longer in maintenance.

 

s1#show maintenance

Flags:

o – On-boot maintenance

v – Violating traffic threshold

 

Unit Name                    Status                        Time since last change      Flags

—————————- —————————- ——————————- —–

System                       Not Under Maintenance                  never

 

Bgp Neighbor(vrf: defa       Status                        Time since last change      Flags

—————————- —————————- ——————————- —–

172.16.200.2                 Not Under Maintenance               0:01:22 ago   

 

s1#show maintenance bgp ip all

BGP peer maintenance information for VRF default

Router identifier 172.16.0.1, local AS number 65001

Neighbor: 172.16.200.2

    Not Under Maintenance

Neighbor: 172.16.200.6

Not Under Maintenance

Neighbor: 172.16.200.10

Not Under Maintenance

Neighbor: 172.16.200.14

Not Under Maintenance

 

Issuing show ip bgp sum on Spine1 we see that the ‘m’ flag has been removed. We’re also still established. So this BGP relationship didn’t have to go through a reset or re-establishment.

 

s1#sh ip bgp sum

BGP summary information for VRF default

Router identifier 172.16.0.1, local AS number 65001

Neighbor Status Codes: m – Under maintenance

Neighbor         V  AS           MsgRcvd   MsgSent  InQ OutQ  Up/Down State  PfxRcd PfxAcc

172.16.200.2     4  65101             91       110    0    0 01:18:14 Estab  7      7

172.16.200.6     4  65102             88        93    0    0 01:18:10 Estab  4      4

172.16.200.10    4  65103             85        92    0    0 01:17:52 Estab  6      6

172.16.200.14    4  65104             85        92    0    0 01:17:55 Estab  6      6

 

And the output of show ip route bgp on Leaf1 shows that our routing table has reinstated Ethernet2 (172.16.200.1) as a next hop for some of the routes.

 

l1#sh ip route bgp

VRF name: default

Codes: C – connected, S – static, K – kernel,

O – OSPF, IA – OSPF inter area, E1 – OSPF external type 1,

E2 – OSPF external type 2, N1 – OSPF NSSA external type 1,

N2 – OSPF NSSA external type2, B I – iBGP, B E – eBGP,

R – RIP, I L1 – ISIS level 1, I L2 – ISIS level 2,

A B – BGP Aggregate, A O – OSPF Summary,

NG – Nexthop Group Static Route

 

B E    172.16.0.1/32 [200/0] via 172.16.200.1, Ethernet2

B E    172.16.0.2/32 [200/0] via 172.16.200.17, Ethernet3

B E    172.16.0.4/32 [200/0] via 172.16.12.2, Vlan4094

 B E    172.16.0.5/32 [200/0] via 172.16.200.1, Ethernet2

via 172.16.200.17, Ethernet3

B E    172.16.0.6/32 [200/0] via 172.16.200.1, Ethernet2

via 172.16.200.17, Ethernet3

B E    172.16.134.0/24 [200/0] via 172.16.200.1, Ethernet2

via 172.16.200.17, Ethernet3

 

Other Maintenance Modes

This lab only covered the BGP portion of Maintenance Mode. We also currently have System and Interface as well.

 

Other Network Configuration Nerd Knobs

There exists other features that we did not include in this lab but may be recommended and should be considered for production environments. Examples include features such as Jumbo Frames, BFD (Bidirectional Forwarding Detection) and the BGP feature known as ‘update wait-install’.

These features and others like them can provide better failure detection and faster routing table convergence that lead to decreased downtime and reduce packet loss during failure and failover.

 

Summary

The Maintenance Mode feature is another example of Arista’s commitment to providing solutions that drive down both operational overhead and production network downtime.

There are many options within the Maintenance Mode feature and over the next few EOS releases more will be added. While we didn’t configure each part of the feature, hopefully we demonstrated enough to get you thinking about Maintenance Mode and where it may benefit your network.

As with any new feature, it is recommended to test it and the associated verification steps in a lab environment prior to deploying to production. In this case we were able to take advantage of vEOS. Some features will require the use of matching hardware and software versions to minimize risk.