• Network CI/CD Part 1 – Building network topologies with Docker and cEOS-lab

 
 
Print Friendly, PDF & Email

Introduction

As the networking industry continues riding the DevOps wave and network engineers become more comfortable using automation tools like Ansible and Salt, network still remains the most brittle piece of IT infrastructure. Partially this can be justified by the fact that network underpins all other areas of IT infrastructure stack – compute, storage and virtualisation. However we, as network engineers, have done very little to improve our confidence in networks and until now some of the biggest outages were caused by trivial network configuration mistakes. When software industry was facing similar challenges, the response was to create Continuous Integration (CI) pipelines – series of steps that automatically build and test every new code change in an environment that closely resembles production.  However when it comes to networking, things aren’t always as easy.

 

Problem Statement

The recent “network automation” movement has been focused mostly on one problem – pushing configuration into network devices. In the absence of any prior configuration management, this was the first logical step to take and a very difficult one as well, thanks to the decades of manual CLI provisioning and a common (mis)belief that networking is special. Although today this problem is still far from being solved, technologies like Arista’s eAPI, configuration modelling with YANG and various other commercial and open-source tools enable some form of API access to install and retrieve configuration across most of the network operating systems. That’s why now the focus is starting to shift towards more advanced use cases, like automated network verification (control plane) and validation (data plane) testing. However here the networking industry is facing another set of challenges:

  • Lack of simulated test environment – it’s still very hard to get a good working virtual replica of a production network device and network simulation is usually a manual task which involves hours of pre-configuration and fine-tuning.
  • Lack of automated test tools – although it’s possible to validate some state using Ansible, napalm or YANG state models, this still covers only a fraction of a total device state and doesn’t tell much about a network-wide state or dataplane connectivity.

  • Cultural shift – perhaps the most important and difficult issue to tackle, stems from the lack of trust in automation and requires consolidated effort from multiple layers of staff inside an organisation.

The last problem can not be classified as purely technical and represents a vast area which may require a slightly bigger publication format (e.g. The Phoenix Project). That’s why we’re going to focus on the first two problems and over a series of 3 blog posts explore the possible solutions using some of the latest additions to Arista toolset portfolio:

  1. In this post we’re going to focus on how to automatically create simulated test environments using the new containerised EOS, built specifically for lab simulations – cEOS-Lab.
  2. In the next post we’ll see how to perform automatic network verification and validation testing using Arista network validation framework.
  3. By putting everything together in the final post we’ll create a complete network CI pipeline that will build a simulated network topology, apply new changes and test them, all inside a portable Gitlab environment.

 

cEOS-lab introduction

cEOS-lab is a containerised version of vEOS-lab with similar functionality and limitations. Compared to vEOS-lab, cEOS-lab has all the benefits a container, such as much lighter footprint of approximately 600 MB of RAM per device, smaller image size and greater portability. However from network automation perspective, the most interesting features are:

  • Standard tools and APIs allowing users to interact with it just like with any other Docker container.
  • Image Immutability, which means that you can change configuration of a running device and even upgrade the EOS version all with no effect on the docker image so that the next time you spin up a container, it will come up in its original state.
  • Typical container lifecycle which makes it fast and easy to spin up and tear down arbitrary multi-node topologies

Note: cEOS-lab is not to be confused with cEOS which was designed to be run on 3rd party open hardware

 

cEOS-lab quickstart

cEOS-Lab is distributed as an archived Docker image, which can be imported directly into the local Docker image repository as follows:

$ docker import cEOSLab.tar.gz ceos:latest

From here on, we can work with cEOS just like with any other Docker container. For example, the following commands would create a cEOS container and connect it to a pair of new networks:

$ docker create –name=ceos –privileged -p 443:443 -e CEOS=1 -e container=docker -e EOS_PLATFORM=ceossim -e SKIP_ZEROTOUCH_BARRIER_IN_SYSDBINIT=1 -e ETBA=1 -e INTFTYPE=eth -i -t ceos:latest /sbin/init

$ docker network create net1

$ docker network create net2

$ docker network connect net1 ceos

$ docker network connect net2 ceos

$ docker start ceos

 

Note: environment variables supplied with `-e` flag may differ between cEOS versions and are usually provided in a README file
For 4.20.5F attribute EOS_PLATFORM should have value ceossim
For 4.21.0F attribute EOS_PLATFORM should have value ceoslab

The above commands would result in the following topology:

Note: Connection to docker0 is created by default when no network parameters are specified in `docker create`. Ethernet0 interface cannot be used and is not available inside the Cli shell.

 We can interact with a running cEOS container using standard Docker tools. For example this is how we can connect to an interactive Cli shell inside the container:

docker exec -it ceos Cli

localhost>

And this is how we can destroy all containers with names containing “ceos”:

docker rm -f $(docker ps -aq –filter name=ceos)

Running EOS inside a container is a very convenient alternative to vEOS, however since all the containers are interconnected by Docker-managed bridges, this imposes a number of limitation that may result in an unexpected behaviour:

  • By default Linux bridges will consume all L2 link-local multicast frames, including STP and LACP (source). The workaround is to connect containers directly with veth pairs either manually or using something like koko.
  • It is possible to force Linux bridge to flood LLDP PDUs with the following command

 echo 16384 > /sys/class/net/<bridge-name>/bridge/group_fwd_mask

  • For any docker container with a number of interfaces greater than 3, the order in which these networks are plugged in inside the container is not deterministic (source). The workaround is to build a custom docker binary using the proposed patch which changes the internal libnetwork data structure storing connected networks from heap to array.

 

Building multinode cEOS topologies

When building complex network topologies with multiple nodes, manual approach of creating and interconnecting each container individually can become cumbersome. It would be much easier if we could define a desired topology in a text file and have it built and destroyed automatically.  Thankfully, there are multiple ways to orchestrate the creation of multiple containers and networks.

Using existing container orchestration tools

One of the obvious tools of choice from Docker’s native ecosystem is docker-compose. It uses a YAML file which describes the desired state of a multi-container application to build or destroy multiple interconnected containers at the same time.  Below is an example of a simple two-node topology defined in a docker-compose format:

version: ‘3’                                                                                                                    

services:                                                                                                                       

 ceos-1:                                                                                                                       

   hostname: CEOS-1                                                                                                            

   image: “ceos:latest”                                                                                                        

   command: /sbin/init                                                                                                         

   privileged: true                                                                                                            

   environment:                                                                                                                

     container: docker                                                                                                         

     CEOS: 1                                                                                                                   

     EOS_PLATFORM: ceossim                                                                                                     

     SKIP_ZEROTOUCH_BARRIER_IN_SYSDBINIT: 1                                                                                    

     ETBA: 1                                                                                                                   

     INTFTYPE: eth                                                                                                                                                                                        

   networks:                                                                                                                   

     default:                                                                                                                  

     p2p:                                                                                                                    

 ceos-2:                                                                                                                       

   hostname: CEOS-2                                                                                                            

   image: “ceos:latest”                                                                                                        

   command: /sbin/init                                                                                                         

   privileged: true                                                                                                            

   environment:                                                                                                                

     container: docker                                                                                                         

     CEOS: 1                                                                                                                   

     EOS_PLATFORM: ceossim

     SKIP_ZEROTOUCH_BARRIER_IN_SYSDBINIT: 1                                                                                    

     ETBA: 1                                                                                                                   

     INTFTYPE: eth                                                                                                                                                                                        

   networks:                                                                                                                   

     default:                                                                                                                  

     p2p:                                                                                                                    

                                                                                                                      

networks:                                                                                                                       

 p2p:                                                                                                                

If saved in a `docker-compose.yml` file in the current directory, the above topology can be built using a single command:

docker-compose -d up

However docker compose has its own caveat which doesn’t honour the order in which networks are defined inside the container. This bug has no current workaround and effectively makes docker-compose unusable for any network topology with more than just 2 links per device.

Another popular container orchestration tool, Kubernetes, assumes a networking model with just a single network interface per POD, which also makes it unusable for network simulations, although a few CNI plugins like multus and knitter were created to address this issue.

Building network topology orchestrator for cEOS using Docker API

Finally, when none of the existing tools are deemed good enough for the task at hand, there’s always an option to build one yourself. The benefit of a custom-built tool is that we can focus on just what’s required without having to care for a broader set of use cases, as it is the case with general-purpose orchestrators. In this case we know that we want to create multiple containers and attach them to multiple networks but we don’t care about load-balancing, security or automatic service scale-out. An example implementation of such orchestrator can be found on Github. To begin working with it, we need to define our topology as a list of links, each described by a unique set of connected interfaces

# cat my-topo.yml

links:

 – [“Leaf1:Ethernet1”, “Spine1:Ethernet1”]

 – [“Leaf1:Ethernet2”, “Spine2:Ethernet1”]

 – [“Leaf2:Ethernet1”, “Spine1:Ethernet2”]

 – [“Leaf2:Ethernet2”, “Spine2:Ethernet2”]

 – [“Leaf3:Ethernet1”, “Spine1:Ethernet3”]

– [“Leaf3:Ethernet2″, “Spine2:Ethernet3”]

Optionally, we can provide configuration that we want to apply to a particular device. For example, this is how we define a hostname for device “Leaf1”:

mkdir config

echo “hostname Leaf1-sw” > ./config/Leaf1

Now, we can create our topology with just a single command:

# ceos-topo –create my-topo.yml

INFO:__main__:All devices started successfully

This will result in the following topology being created inside our Docker host:

From here on we can interact with each device either through a Cli or eAPI:

# docker exec -it Leaf1 Cli

Leaf1>en

Leaf1#sh run sec management

management api http-commands

  no shutdown

Leaf1#exit

#

# curl -s -k https://admin:admin@localhost:8000/command-api -d ‘{

 “jsonrpc”: “2.0”, “method”: “runCmds”,

 “params”: {

   “format”: “json”, “timestamps”: false,

   “autoComplete”: false, “expandAliases”: false,

   “cmds”: [“show version”], “version”: 1

 },  “id”: “EapiExplorer-1”

}’ | jq ‘.”result”[0] .modelName’

 

“cEOS”

By default eAPI ports of all containers will be published on docker host starting from port 8000.

Coming up

Now that we’ve covered how to build arbitrary network topologies from a simple text file, we can move on to the next task – testing them.

In the next post we’ll explore Arista network validation framework built on top of Robot, one of the most popular open-source general-purpose test automation frameworks in the world.

 

Follow

Get every new post on this blog delivered to your Inbox.

Join other followers: