This is a test setup using common servers to validate PTP timestamping accuracy of Arista 7150 switches.
Tap Aggregation devices are a growing monitoring model, allowing monitoring applications to analyze a huge set of data sources. In some application monitoring, it’s critical to correlate time to achieve high precision information, where we can measure the precise time in a packet path. For this purpose we have hardware time-stamping, a feature where the tap aggregation device stamp a time reference in each packet. This subject is explored in this article. Devices performing timestamp in packets must use a common clock reference to synchronize all devices’ clock time. This can be accomplished using PTP.
Common issues when dealing with PTP
While deploying PTP is not hard, when it comes to control high precision applications where hundreds of nano seconds would make a difference in monitoring, we have to be careful about clock drifts between devices. Every device drifts its clock independently and in a PTP deployment, devices synchronize with a common clock, the Grand Master Clock. The PTP protocol exchanges messages to keep all devices in sync with the GM. Common issues found in real world deployments:
- High offset values from PTP. Some offset is normal, but when we have high offsets values constantly, let’s say hundreds of nanoseconds, this must be fixed. It would be a jitter on PTP packets.
- GM clock has a problem and is drifting or losing connection.
To demonstrate a precision hardware time stamping, we’re using a concept lab that can be easily built using Linux servers and Arista DCS-7150S switches, using a topology in the following figure: The Lab works as following:
- The PTP GM server is a Linux server using a Ethernet line card with HW Time Stamping support. Most modern Ethernet Network cards has this support. We adopted Linux PTP project for our Lab, please refer to its project page for detailed information.
- Arista switch SW1 is connected directly to PTP GM in boundary mode and providing PTP clock messages to Arista Sw2 and Sw3 switches.
- The Traffic Generator can be a Linux Perf device to demonstrate this concept.
- The TG sends data traffic into Arista SW1, configured also as a Tap Aggregation device. This switch replicates the same traffic to Arista Sw2 and Sw3 at same time.
- Arista SW2 and Sw3 receive the same traffic and timestamps it to the Packet Capture Server, which is a Wireshark/TCPDump/TShark sniffer to save both traffic received from Sw2 and Sw3 in a Pcap file.
With this data, we expect to have both timestamped packets from Sw2 and Sw3 with pretty close UTC time values. For this analysis we used the PCAP Decoder script to extract the UTC values from both traffic. The PCAP decoder and keyframe usage is described in detail in this link.
Key details on Lab
The Linux PTP server is not a high precision device, its clock is not as stable comparing to a real PTP GM device. Thus, for this concept lab we need to make some tuning to have a more aggressive rate and avoid jittering clocks from devices. In the PTP GM we used the following /etc/ptp4l.conf config setup
[global] priority1 64 priority2 64 verbose 1 logging_level 6 summary_interval 0 logSyncInterval -2
All switches were configured in Tap Aggregation mode and had tap and tools ports mapped. SW1 is acting as a split device and SW2 and SW3 as timestamping Tap Aggregation switches. An important note is when you configure switch in Tap Aggregation mode, you must configure non-tap ports as no-errdisable to use them for PTP synchronization . SW1 configuration notes:
ptp mode boundary ! tap aggregation mode exclusive mode exclusive no-errdisable Ethernet1 mode exclusive no-errdisable Ethernet13 mode exclusive no-errdisable Ethernet15 ! interface Ethernet1 ptp enable ! interface Ethernet13 ptp enable ! interface Ethernet14 switchport mode tool switchport tool group set Test ! interface Ethernet15 ptp enable ! interface Ethernet16 switchport mode tool switchport tool group set Test ! interface Ethernet20 switchport mode tap switchport tap default group Test ! management api http-commands protocol http no shutdown
SW2 configuration notes:
ptp mode boundary ! tap aggregation mode exclusive mode exclusive no-errdisable Ethernet13 ! interface Ethernet13 ptp enable ! interface Ethernet14 switchport mode tap switchport tap identity 10 switchport tap default group Test10 ! interface Ethernet24 mac timestamp before-fcs switchport mode tool switchport tool identity dot1q switchport tool group set Test10 ! platform fm6000 keyframe sw2-kf interface Ethernet24 220.127.116.11 001c.73ab.1111 platform fm6000 keyframe sw2-kf device 10 platform fm6000 keyframe sw2-kf rate 100 platform fm6000 keyframe sw2-kf vlan 10 ! management api http-commands protocol http no shutdown
SW3 configuration notes:
ptp mode boundary ! tap aggregation mode exclusive mode exclusive no-errdisable Ethernet15 ! interface Ethernet15 ptp enable ! interface Ethernet16 switchport mode tap switchport tap identity 20 switchport tap default group Test20 ! interface Ethernet24 mac timestamp before-fcs switchport mode tool switchport tool identity dot1q switchport tool group set Test20 ! platform fm6000 keyframe sw3-kf interface Ethernet24 18.104.22.168 001c.73ab.2222 platform fm6000 keyframe sw3-kf device 20 platform fm6000 keyframe sw3-kf rate 100 platform fm6000 keyframe sw3-kf vlan 20 ! management api http-commands protocol http no shutdown