Accurate packet timestamps are essential for network event correlation and performance analysis. As data link rates increase, software solutions can no longer provide sufficient timing resolution. Purpose-built hardware timestamping devices perform well, but are costly and are difficult to deploy across a network. To address these issues, Arista Networks’ 7150 series combines datacenter-class routing and switching with precision timestamping. Starting with EOS 4.11.3, timestamping is available on any 7150 port, at line-rate, and with PTP clock synchronisation for maximum accuracy.
The hardware timestamper matches on the first bit of the FCS of an incoming packet ensuring timestamps are based on the ingress time of the frame.
Timestamping configuration itself is configured on egress ports – this allows the user to selectively enable or disable timestamps that are sent out of each interface. Any combination of port speeds is supported.
Two hardware timestamp modes are supported;
- Replaces the existing Frame Check Sequence (FCS)
- Insert a 4 byte timestamp between the payload and FCS.
Mode selection is on a per-port basis, so a switch can potentially have different ports configured in each mode. In FCS replacement mode, the switch overwrites the 32-bit FCS with the timestamp. This mode has no latency impact downstream as the original frame size is preserved. However, downstream devices must recognise that the FCS field (now a timestamp) will not be valid. Cut-through switches will forward the frame but may increment checksum error counters on the transit interfaces.
Timestamps may also be configured to append to the frame payload. In this mode, the old FCS is discarded and the timestamp is added to the end of the frame data. The switch then recalculates the FCS and appends it to the end of the frame. This creates a valid Ethernet frame with respect to the FCS, but does not update the headers of any nested protocols (e.g. UDP, TCP). Downstream applications can access the timestamp by reading the last 32 bits of the frame payload.
Packet timestamps are based on an undisciplined, free running counter running at a nominal 350Mhz (each tick is approximately 2.857ns). The switch uses the low order 31 bits of the ASIC time in the frame timestamp. The high order bit of the low order byte is a 0 pad, making the timestamp 32 bits long with 31 significant bits (see Figure 3).
Decoding Hardware Timestamps
A keyframe mechanism (described next) provides recovery of the high order 33 bits of the 350Mhz-based time, as well as a mapping to UTC time and compensation for hardware timestamper clock skew. This allows applications to determine the absolute time given a high-resolution frame timestamp.
There are three times to consider when decoding timestamps:
- Hardware packet timestamps (those applied to packets (see fig. 3)) [undisciplined]
- Keyframe ASIC Time (a 64 bit 350Mhz counter starting from system init) [disciplined]
- UTC time (64 bit epoch time) [disciplined]
Keyframe ASIC time and UTC time are maintained in EOS and the system FPGA under discipline from PTP. Keyframe ASIC time is a 64 bit counter using a 350Mhz base clock and starting from the initialization time of the system.
The 31-bit frame timestamp provides high-resolution timing, rolling over about every 6.135 seconds (31 bits at 2.857ns per tick), however the packet timestamping clock is undisciplined.
To correlate the packet timestamp to UTC, the 31 bit stamp needs to be related to an accurate 64 bit view of 350Mhz time (Keyframe ASIC time) and then to absolute UTC time. The switch sends keyframes to provide sufficient metadata to enable this conversion.
Each keyframe contains the current Keyframe ASIC time, UTC time and the factor by which the hardware packet timestamper clock deviates from the nominal 350 Mhz (skew); hence an application can compute the error in the low order bits of the packet timestamp, add the high order bits from Keyframe ASIC time (for precise, relative timing) and then map this 64-bit number to UTC time to determine absolute time.
Keyframes are generated periodically (configurable), the data contained represents a point in time correlation between the various time formats in the system. The keyframe tells you two crucial pieces of information:
- at UTC time A, 64-bit 350Mhz time (Keyframe ASIC time) was B
- at the time of Keyframe generation, the hardware timestamper frequency deviated from 350Mhz by skew factor C
The keyframe timestamp indicates the actual UTC time at which the keyframe was generated by FPGA. The frame also includes the timestamping mode (FCS type) so applications can dynamically determine the timestamp’s byte offset. Each field is described in the following table.
For optimal accuracy, the system must be under high quality PTP discipline and the clock skew data must be used when decoding the hardware packet timestamp to eliminate discrepancy between the undisciplined clock driving the hardware timestamper and the disciplined clock generating Keyframe ASIC time.
Note: This capability is disabled by default when creating a keyframe for backwards compatibility and must be explicitly enabled.
|KF ASIC time||8||The full 64-bit counter used for frame timestamping. Each tick represents approx. 2.857ns.|
|UTC time||8||Unix (POSIX) time in nanoseconds|
|Last sync time||8||The last ASIC time (in nanoseconds) at which PTP was synchronized. If no sync has occurred, or if the last sync was greater than 8 hours ago, this field is 0.|
|Skew numerator||8||[4.13.0F+] These skew fields form a ratio indicating ASIC clock skew. If numerator/denominator is greater than 1, the clock is skewed fast; less than 1 is slow. Starting in 4.13.2F, the skew fields can be toggled through the CLI (see below).|
|Keyframe timestamp||8||The generation time of the keyframe itself, in ticks (ASIC time).|
|Egress interface drops||8||Number of dropped frames on the keyframe’s egress interface|
|Device ID||2||User-defined device ID|
|Egress interface||2||The egress switchport of the keyframe|
|FCS type||1|| The timestamping mode configured on the keyframe’s egress port.
0 = timestamping disabled
|Reserved||1||Reserved for future use|
Timestamps are captured on ingress and then written on egress. To enable timestamps on all frames that egress an interface, use this command:
Arista(config-if-etX)#mac timestamp <mode>
Mode options are: replace-fcs to overwrite the existing FCS or before-fcs to append the timestamp and calculate a new FCS.
Enable timestamping on frames leaving et1, replacing the FCS:
Arista(config-if-et1)#mac timestamp replace-fcs
Disable timestamping on et1:
Arista(config-if-et1)#no mac timestamp
Keyframes are configured on groups of ports, each of which is identified with a keyframe name. Each group shares keyframe settings. The required parameters for enabling keyframes are the keyframe group name, destination IP, and destination MAC:
Arista(config)#platform fm6000 keyframe <name> interface <intf> <destination-ip-address> <destination-mac-address>
By default, the source IP of a keyframe is the IP address of the management interface (ma1). If no IP address is configured on ma1, keyframes are sourced from 0.0.0.0. A different source IP can be set with:
Arista(config)#platform fm6000 keyframe <name> source ip <source-ip-address>
Keyframes are sent once per second by default. Since the frame timestamp rolls over about every 6.135 seconds, this is sufficient for rollover detection. However, the rate can be increased as high as 100 frames per second with:
Arista(config)#platform fm6000 keyframe <name> rate <packets-per-second>
Keyframes are not tagged by default. To configure an 802.1q tag (4.13.0F+), use:
Arista(config)#platform fm6000 keyframe <name> vlan <vlan-id>
The device ID field is a user-defined 16-bit value meant for identifying the switch that sourced the keyframe.
Arista(config)#platform fm6000 keyframe <name> device <device-id>
The clock skew fields in the keyframe can be toggled in 4.13.2F+. By default they are omitted for backwards compatibility. To enable them, please use:
Arista(config)#platform fm6000 keyframe mykeyframe fields skew
Enable keyframes on et1 with a destination of 10.0.0.1/001c.73ab.cdef:
Arista(config)#platform fm6000 keyframe mykeyframe interface et1 10.0.0.1 001c.73ab.cdef
Set the source IP of the keyframes to 192.168.1.1:
Arista(config)#platform fm6000 keyframe mykeyframe source ip 192.168.1.1
Increase the keyframe rate to once every 10ms (100 frames per second):
Arista(config)#platform fm6000 keyframe mykeyframe rate 100
Set the Device ID field to 12345:
Arista(config)#platform fm6000 keyframe mykeyframe device 12345