• Robot Framework for Auto Test

 
 
Print Friendly, PDF & Email

Introduction

In-house EOS code certification is generally a time consuming process as it involves a lengthy life cycle. At a high-level, the various stages involved are:

  1. Setting up the test environment
  2. Designing test cases
  3. Executing test cases
  4. Documenting test results
  5. Validating gathered test results

As you can imagine, stages 3, 4 and 5 can be the most time consuming as they warrant an error-free execution. Additionally, since image and configuration management would be simplified, most customers would like to have a single EOS version deployed across all their platforms and networks. So certifying code at a brisk pace, on all platforms, but not compromising on accuracy is the need of the hour. This is where automation can help, and Robot Framework is well suited for this very reason.

Robot Framework

Robot Framework, written in Python, is open source software under Apache License 2.0. It is an extensible keyword-driven automation framework for acceptance testing and can be used for testing distributed and heterogeneous applications. It comes with a lot of built-in libraries that support ready to use technologies such as SSH and Telnet.

 

Why Robot Framework?

There are many reasons why Robot Framework can be useful:

  • Easy to use syntax formats such as plain-text and TSV that are easy to read and edit
  • Ability to create libraries in Python or Java
  • Easy-to-view reports and logs in HTML format
  • Auto-validation of test cases
  • Platform and application independent
  • Low overhead, resulting in faster execution  times
  • Support from EOS+ team 

Keywords

Libraries in Robot Framework are implemented as classes. Each method inside a class can be referenced as a keyword. This results in an abstraction of the underlying code to the end user who doesn’t need to understand the nuts and bolts involved but just understand the intent of the keyword. Some of the popular libraries and keywords are listed below:

Library Name Purpose Example Keyword
BuiltIn  Imported automatically and contains generic often needed keywords.  Should Be Equal
Collections  Keywords for handling lists and dictionaries  Count Values In List
OperatingSystem  Keywords for operating system related tasks Copy File
Telnet   Supports connecting to Telnet servers and running commands Open Connection

AristaLibrary for Robot Framework

Arista has designed the AristaLibrary to help simplify configuration validation and verification. It leverages pyeapi to make configuration validation possible for those who have no programming experience. The library is implemented as a Class with each method inside the class referenced as a keyword. Some of the popular keywords in AristaLibrary are below.

  • Connect To: Connects to an EOS switch via eAPI
  • Get Running Config: Grabs running-config
  • List Extensions: Captures list of extensions installed on a specified EOS device.
  • Configure: Runs commands in configuration mode
  • Run Cmds: Runs low-level eAPI commands on a specified EOS device.

Arista Network Validation Framework

The Arista Network Validation Framework is a great tool built by Arista to get you started on configuration and service validation. It greatly simplifies connecting to multiple devices and includes many pre-built test cases to get you started quickly. The default test suites scans the running-config for enabled features and validate the operational state of them. Also, it is built using tools that already have great reporting and integrations with many automation and Continuous Integration / Continuous Delivery (CI/CD) systems such as Jenkins. Based on Robot Framework, tests can be easily modified or new tests can be added by non-programmers while more sophisticated enhancements to the framework may be added through basic Python. When combined with tools like Jenkins, with the Robot Framework plugin, you can schedule runs and visualize the state of the validation results over time with detailed drill-down analysis. More information can be found on the software download page under the extensions directory. 

AristaLibrary Example

Now that we have introduced AristaLibrary, let’s go over a simple example. The topology considered here is a simple leaf-spine design with spines in MLAG and configured for VARP. The leaf switches are configured with VLAN 100 SVI and default gateway pointing to the VARP IP. Default configs on the devices will have MLAG disabled on spine_B and VLAN 100 SVI disabled on leaf_B. We should thus expect to see MLAG status not active on both spines, and ICMP checks to VARP gateway to fail from leaf_B.

 


The example has two different test suites. The test suite called validate.txt performs validation of services configured on end devices while the test suite deploy-and-validate.txt deploys config that re-enables MLAG on spine_B and vlan100 SVI on leaf_B and re-validates services. 

Before any test cases are run, connection to each device should be formed. The IP address of the switch, port number, and credentials are passed as inputs to the “Connect To” keyword to form connections. Each connection is assigned a switch_id which are sequential in the order of connections formed. In the below snippet, connection to spine_A gets a switch_id of 1, connection to spine_B gets a switch_id of 2 and so on.

*** Keywords ***
Connect To Switches
[Documentation] Establish connection to a switch which gets used by test cases.
Connect To host=${spine_A} TPORT=${TPORT} username=${UNAME} password=${PWD} port=${CONN_PORT}
Connect To host=${spine_B} TPORT=${TPORT} username=${UNAME} password=${PWD} port=${CONN_PORT}
Connect To host=${leaf_A} TPORT=${TPORT} username=${UNAME} password=${PWD} port=${CONN_PORT}
Connect To host=${leaf_B} TPORT=${TPORT} username=${UNAME} password=${PWD} port=${CONN_PORT}

 

The validate.txt test suite

This test suit validates MLAG, VARP and reachability services. Please note that different keywords are used to showcase the various keyword options we have in AristaLibrary. Also, test cases are run individually for each device for simplicity. We could have merged ‘like’ test cases together but separated them for simplicity as it makes it easier for readers just introduced to Robot Framework.

Verification of MLAG status on spines

Below is a snippet of the MLAG status check on spine-A

Check MLAG Status on spine-A
  ${mlag_status}=    Get Command Output   switch_id=1   cmd=show mlag
  Log   ${mlag_status}
  Expect   state   to be   active

We are calling several keywords here. The “Get Command Output” keyword is used to run commands on an end device and capture output in JSON format. It uses the switch_id to determine what device the command needs to be run on, in this case, spine_A. The Log keyword is used to log the data for reporting. Once a ‘Get Command Output’ keyword has been used for a given device the Expect keyword may then be invoked to validate the content of the returned output. In the above example, the Expect keyword validates whether or not MLAG state is “active”. 

Verification of VARP status on spines

Check VARP Status on spine-A
  Change To Switch   1
  ${varp_status}=  Run Cmds  show ip virtual-router
  Log   ${varp_status}
  Should Be Equal   ${varp_status['result'][0]['virtualRouters'][0]['protocolStatus']}   up

In the above snippet, we are first using the “Change To Switch” keyword to make sure the active switch is 1 for all the following keywords. We are then using the “Run Cmds” keyword to capture the “show ip virtual-router” output in JSON. The built-in keyword “Should Be Equal” is used to validate VARP protocol status. The test case would fail if the key ‘protocolStatus’ in the JSON output doesn’t match the string ‘up’.

Verification of Gateway Reachability

In this test, we run pings from the leaf switches to the VARP gateway and verify if the pings are successful. The below is the snippet from the test suite. We expect ping from leaf_A to pass but ping from leaf_B to fail as the SVI is disabled on leaf_B.

Test if ping to IPv4 VRP Gateway passes from leaf-A
Change To Switch   3
 ${ping_output}=    Run Cmds ping 100.1.1.10   encoding=text
 ${result}= Get From Dictionary ${ping_output} result
 ${match} ${group1}= Should Match Regexp ${result[0]['output']} (\\d+)% packet loss
Should Be Equal As Integers ${group1} 0 msg="Packets lost percent not zero!!!"

 

The deploy-and-validate.txt test suite

This test suite runs the same tests as in the validate.txt test suite, but before that, it uses the Configure keyword to enable MLAG on spine_B and VLAN 100 SVI on leaf_B. 

Enable MLAG on spine-B
  Change To Switch  2
  @{cmds}=    Create List    mlag configuration    no shutdown
  Configure    ${cmds}

Enable VLAN 100 SVI on leaf-B
  Change To Switch  4
  @{cmds}=    Create List    interface Vlan 100    no shutdown
  Configure    ${cmds}

The two test cases above leverage the AristaLibrary’s “Configure” keyword to execute global configuration commands. The standard “Create List” is used to create the list of commands you’d like to execute on the device, which is then passed to the “Configure” keyword.

 

Executing the Test Suites

The test suites require variable inputs such as user credentials, device IP address etc.. The myvars.yml file has this information which can be passed to the test suites during execution. A typical format of executing test suites is:

pybot --pythonpath=AristaLibrary --variablefile <VariableFile> <path-to-test-suite>

Execution of variable.txt test suite


mac:mlag-varp-validation mac$ pybot --pythonpath=AristaLibrary --variablefile myvars.yaml test-suites/validate.txt
============================================================================== Validate :: A simple Robot Framework test suite to Validate Services ============================================================================== Check MLAG Status on spine-A | FAIL | AristaLibrary.Expect: Key: '[u'state']', Found: 'inactive', Expected: 'active' ------------------------------------------------------------------------------ Check MLAG Status on spine-B | FAIL | AristaLibrary.Expect: Key: '[u'state']', Found: 'disabled', Expected: 'active' ------------------------------------------------------------------------------ Check VARP Status on spine-A | PASS | ------------------------------------------------------------------------------ Check VARP Status on spine-B | PASS | ------------------------------------------------------------------------------ Test if ping to IPv4 VARP Gateway passes from leaf-A | PASS | ------------------------------------------------------------------------------ Test if ping to IPv4 VARP Gateway passes from leaf-B | FAIL | 'connect: Network is unreachable ' does not match '(\d+)% packet loss' ------------------------------------------------------------------------------ Validate :: A simple Robot Framework test suite to Validate Services | FAIL | 6 critical tests, 3 passed, 3 failed 6 tests total, 3 passed, 3 failed ============================================================================== Output: /Users/sureshk/robot-github/robot-examples/mlag-varp-validation/output.xml Log: /Users/sureshk/robot-github/robot-examples/mlag-varp-validation/log.html Report: /Users/sureshk/robot-github/robot-examples/mlag-varp-validation/report.html
mac:mlag-varp-validation mac$

As you’d expect, the MLAG test cases have failed since MLAG is disabled on spine_B. The ICMP check from leaf_B has failed since SVI is disabled.

Execution of deploy-and-validate.txt test suite

mac:mlag-varp-validation  mac$ pybot --pythonpath=AristaLibrary --variablefile myvars.yaml test-suites/deploy-and-validate.txt
============================================================================== Deploy-And-Validate :: A simple Robot Framework to deploy MLAG and VARP ============================================================================== Enable MLAG on spine-B | PASS | ------------------------------------------------------------------------------ Enable VLAN 100 SVI on leaf-B | PASS | ------------------------------------------------------------------------------ Pause few seconds for MLAG to sync | PASS | ------------------------------------------------------------------------------ Check MLAG Status on spine-A | PASS | ------------------------------------------------------------------------------ Check MLAG Status on spine-B | PASS | ------------------------------------------------------------------------------ Check VARP Status on spine-A | PASS | ------------------------------------------------------------------------------ Check VARP Status on spine-B | PASS | ------------------------------------------------------------------------------ Test if ping to IPv4 VARP Gateway passes from leaf-A | PASS | ------------------------------------------------------------------------------ Test if ping to IPv4 VARP Gateway passes from leaf-B | PASS | ------------------------------------------------------------------------------ Deploy-And-Validate :: A simple Robot Framework to deploy MLAG and... | PASS | 9 critical tests, 9 passed, 0 failed 9 tests total, 9 passed, 0 failed ============================================================================== Output: /Users/sureshk/robot-github/robot-examples/mlag-varp-validation/output.xml Log: /Users/sureshk/robot-github/robot-examples/mlag-varp-validation/log.html Report: /Users/sureshk/robot-github/robot-examples/mlag-varp-validation/report.html
mac:mlag-varp-validation mac$

The first two test cases have re-enabled services on spine_B and leaf_B. The test cases following them are same test cases from validate.txt and all of them have passed.

References

Follow

Get every new post on this blog delivered to your Inbox.

Join other followers: