VMTracer Visibility and Call Flows

Introduction

Arista EOS has been supporting the VMTracer feature since vSphere 4.0 was introduced and continues to support the latest version.  The EOS User Manual (found for various releases at https://www.arista.com/en/support/software-download) provides a very good description and background to the feature along with configuration details.  This technical note adds additional call flow information to better understand the feature and the network visibility it provides to operators, as well covering NSX-V visibility details.

To set the baseline, the VMTracer logical diagram from the User Manual is redrawn here:

vmtracer logical

This diagram shows that there are up to three main conversations between the Arista EOS devices and vSphere:

  • Discovery packets to the ESXi pNIC (typically LLDP)
  • SOAP API calls to vCenter
  • (optionally) REST API call to NSX-V Manager (formerly called vShield Manager or VSM) for visibility of VXLAN overlays for VMs

Quick Configuration Notes:

In addition to ensuring that LLDP is enabled on the ESXi facing ports, VMTracer needs to be configured:

vmtracer session vmtracer-test
   url https://172.28.169.10/sdk                /* vCenter URL
   username root
   password 7 y5BlQXAUxQWJ+MkzwWsatg==
   autovlan disable                             /* Dynamic VLAN provisioning option
   vxlan
      url https://172.28.169.11                 /* NSX-V Manager URL
      username admin
      password 7 PE/d/l1qXmCOgHGg2dqT7Q==

VMTracer Call Flow

However, it is also useful to understand the call flow and state machine that occurs to assemble this information together.  The following picture provides this detail:

VmTracer

VmTracer, on startup, will try to connection to a vCenter endpoint and, upon successful connection, start a ‘monitoring’ thread that subscribes to certain event changes on objects of interest (VM, DVS, PortGroups etc). These updates will be pulled from vCenter as long as the VMTracer agent is up and connected.

Optionally, VMTracer will interact with an available NSX-V Manager via REST API calls to track the VXLAN overlays associated with the VMs.

VMTracer Visibility for the Network Operator

We can see the API connections with the vmtracer session view:

#show vmtracer session
Session                              vmtracer-test
vCenter URL                          https://172.28.169.10/sdk
username                             root
autovlan                             disabled
allowed-vlans                        1-4094
sessionState                         Connected
VShield/NSX URL                      https://172.28.169.11
username                             admin
sessionState                         Connected

More info on the APIs can be seen from the detail commands:

#show vmtracer session vmtracer-test vcenter detail
Session                              vmtracer-test
vCenter URL                          https://172.28.169.10/sdk
username                             root
autovlan                             disabled
allowed-vlans                        1-4094
sessionState                         Connected
lastStateChange                      13 days, 23:32:26 ago
lastMsgSent                          Query network hint message
timeOfLastMsg                        0:00:07 ago
responseTimeForLastMsg               4.08239467333e-05
numSuccessfulMsg                     160935
lastSuccessfulMsg                    Query network hint message
lastSuccessfulMsgTime                0:00:07 ago
numFailedMsg                         1208
lastFailedMsg                        Query network hint message
lastFailedMsgTime                    13 days, 22:26:19 ago
lastErrorCode                        RetrieveServiceContent SOAP 1.1 fault: SOAP-ENV:Client [no subcode]
"Connection refused"
Detail: [no detail]

#show vmtracer session vmtracer-test vsm detail
Session                              vmtracer-test
VShield/NSX URL                      https://172.28.169.11
username                             admin
sessionState                         Connected
lastStateChange                      0:00:01 ago
lastMsgSent                          /api/2.0/global/heartbeat
timeOfLastMsg                        0:00:01 ago
responseTimeForLastMsg               0.0504470919876
numSuccessfulMsg                     443073
lastSuccessfulMsg                    /api/2.0/global/heartbeat
lastSuccessfulMsgTime                0:00:01 ago
numFailedMsg                         6361
lastFailedMsg                        /api/2.0/global/heartbeat
lastFailedMsgTime                    13 days, 23:44:20 ago
lastErrorCode                        400

At the same time, CDP/LLDP information is exchanged between the Switch and the EXSi hosts.  The host informs the vCenter of this state and VMTracer learns of this through the QueryNetworkHints SOAP call, filtering on the bridge MAC.  Additional SOAP calls are also made in an on-demand manner (for example, FetchDVPorts, LogUserEvents).

#show lldp neighbors detail
...
Interface Ethernet1 detected 1 LLDP neighbors:

  Neighbor "vmnic1"/0050.565d.61d7, age 7 seconds
  Discovered 14 days, 0:21:52 ago; Last changed 14 days, 0:21:52 ago
  - Chassis ID type: Interface name (6)
    Chassis ID     : "vmnic1"
  - Port ID type: MAC address (3)
    Port ID     : 0050.565d.61d7
  - Time To Live: 180 seconds
  - Port Description: "port 8 on dvSwitch dvs-sw-vtep-ComputeA-cluster (etherswitch)"
  - System Name: "localhost"
  - System Description: "VMware ESX Releasebuild-2068190"
  - System Capabilities : Bridge
    Enabled Capabilities: Bridge
...

Based on these above conversations, valuable information is collected, assembled, and provided to the network operator, thus crossing the network and compute silos to enhance the visibility.  This can be seen from the main vmtracer output:

#show vmtracer vm app01
VM Name                : app01
Data Center            : Datacenter
vCenter instance       : https://172.28.169.10/sdk
Host                   : 172.28.168.21
Status                 : Up/Down
vNIC                   : Network adapter 2
  vNIC MAC addr        : 00:50:56:a5:c1:f7
  vNIC Portgroup       : vxw-dvs-171-virtualwire-2-sid-5001-ls-app
  vNIC VLAN            : 131
  vSwitch / vDS        : dvs-sw-vtep-ComputeA-cluster
  vSwitch MTU          : 1600
  Logical Switch       : --
  VNI                  : --
  VTEP IP              : --
  Transport VLAN       : --
  Host PNIC            : vmnic1
  Physical switchport  : Et1
  Transport Zone       : --

And by adding the “vxlan” keyword, the VXLAN overlay information is shown:

#show vmtracer vxlan vm
Vxlan Segment        VTEP IP              VLAN    VMs
ls-app               10.10.131.10/24      131     app01, OpenTSDB
ls-app               10.10.132.10/24      132     app02
ls-app               10.10.133.10/24      133     nsxv-esg-0
ls-db                10.10.131.10/24      131     --
ls-db                10.10.133.10/24      133     nsxv-esg-0
ls-db                10.10.132.10/24      132     db02
ls-web               10.10.132.10/24      132     web02
ls-web               10.10.133.10/24      133     nsxv-esg-0
ls-web               10.10.131.10/24      131     web01

Automation

VMTracer can perform VLAN automation on the switch for attached ESXi VMs depending on whether or not auto-segmentation (dynamic allocation and pruning of VLANs) is enabled.  This is possible due to the amount of visibility gained through the various communication channels noted above.  vMotion events can be fully automated without ever having to touch the Arista switches.

In the case of NSX-V managed overlays, even though VMTracer can maintain VXLAN mapping information, dynamic VTEP (VLAN-VNI) mappings are performed by the NSX-V to CVX automated workflow.

Further Reading

In addition to the resources mentioned above, please visit the following resources for more information:

https://www.arista.com/en/products/eos/visibility
https://www.arista.com/en/solutions/network-virtualization
White_Paper_Design_VMware_Arista.pdf