• 400GBASE-ZR transceivers TOI

 
 
Print Friendly, PDF & Email

Description

400GBASE-ZR transceiver is the industry’s first multi-vendor DWDM standard, a Digital Coherent Optical module with tunable laser, using DWDM multiplexing and 16QAM modulation and capable of delivering 400G per port over distances up to 120 km. 400GBASE-ZR transceivers are fully interoperable, their functionality and management are specified by CMIS4.0 and subsequent CMIS versions, Coherent 400ZR and Coherent CMIS Implementation Agreements [1,2,3,4] and other OIF documents. On the line side, the module is utilizing a high-performance Concatenated forward-error correction (C-FEC) scheme, which consists of a hard-decision (HD) outer FEC and a soft-decision (SD) inner FEC. 400GBASE-ZR transceivers are available in both QSFP-DD and OSFP form factors

Key software features supported by Arista EOS in 4.25.2F:

  • Compliant with CMIS4.0/CMIS4.1 (CMIS5.0) and Coherent CMIS
  • Frequency tuning (100GHz and 75GHz grids)
  • DOM monitoring, including VDM (Versatile Diagnostics Monitoring) pages, defined in CMIS4.0
  • Coherent alarms and faults, including pages, defined by Coherent CMIS
  • Configurable Tx output power
  • A separate command for shutting down Tx output path for unidirectional mode
  • 1-sec update of pre-FEC BER, OSNR, ESNR

Platform compatibility

The 400GBASE-ZR transceiver is a power class 8 module with power consumption up to 20W, the highest power consumption among 400G transceivers.

In theory, every 400GBASE OSFP or QSFP-DD switch is qualified to host 400GBASE-ZR transceivers. However, the actual number of 400GBASE-ZR modules that can be plugged in the switch simultaneously may vary, depending on platform (modular vs fixed) and ASIC type. It is always recommended to discuss installation of 400GBASE-ZR modules with Arista support.

Using 400GBASE-ZR in combination with OSFP-LS

Arista EOS allows using OSFP-LS (pluggable line system in the OSFP form factor), instead of traditional DCI line systems. For more details on OSFP-LS and their integration with 400GBASE-ZR, refer to the OSFP-LS TOI.

Configuration

Laser Frequency Configuration

All coherent modules, including 400GBASE-ZR, require their laser frequency to be explicitly configured. To configure laser frequency in 400GBASE-ZR, use the ‘transceiver frequency’ command under the defined interface, providing frequency in Gigahertz, in the range 191.3-196.1 THz. If laser frequency is not configured or is configured outside the valid range, all interfaces associated with the port are put into errdisable state.

switch#conf
switch(config)#
switch(config)#interface Ethernet12/1
switch(config-if-Et12/1)#transceiver frequency 193100
switch(config-if-Et12/1)#

Configured and operational frequency settings can be verified using the CLI command “show interface [<intf>] transceiver hardware”

Notes

  • It can take up to 90 sec for the 400GBASE-ZR module to fully complete frequency tuning.
  • The frequency plan for 400GBASE-ZR modules support  channel spacings of 100GHz or 75GHz. It is recommended to use operating frequency channel definitions in chapter 15 of [4].

For additional information on frequency configuration for DCO modules, see Configuration and Status sections in [5].

Transmit Output Power Configuration

CMIS4.0 defines support of configurable transmit output power as an optional feature. Check the output of CLI command ‘show interface [<intf>] transceiver eeprom’ for page 04h, registers 196-201 to see if it is supported.

Programmable output power advertisement (04h:196-201):
    Lane programmable output power supported (04h:196): true
    Min programmable output power (04h:198-199): -14 dB
    Max programmable output power (04h:200-201): -10 dB

Transmit Output Power can be configured using ‘transceiver transmitter signal power’ command

switch#conf
switch(config)#interface Ethernet14/1
switch(config-if-Et14/1)#transceiver transmitter signal-power ?
switch(config-if-Et14/1)#transceiver transmitter signal-power -10

Configured Tx Power can be verified through the CLI command “show interface [<intf>] transceiver hardware”. Actual operational Tx Power can be verified with CLI command “show interface [<intf>] transceiver”

Note 1: 400GBASE-ZR does not have a recommended level of transmit laser power. In most cases, it is OK to stay with default power.

Note 2: Some vendors may not populate the range of supported output power correctly. The range -10 to -14 dBm should be supported. Contact Arista Support if you are planning to configure output power outside of this range.

Transmit Output Disable and Unidirectional mode

In the coherent optics, ‘shutdown’ command impacts both transmit and receive path and can’t be used to set up unidirectional mode. To shut down Tx output path, use command ‘transceiver transmitter disabled’:

switch#conf
switch(config)#interface Ethernet14/1
switch(config-if-Et14/1)#transceiver transmitter DISABLED 

To re-enable transmit output, use the ‘no’ or ‘default’ version of the above configuration command.

Show commands

‘Show interface transceiver eeprom’ command – module capabilities

show interface [<intf>] transceiver eeprom  – displays parsed capabilities.  For 400GBASE-ZR, parsing of frequency tuning, power tuning ( page 04h ) and VDM configuration pages (20h-23h) is added.

switch#show interface Ethernet15/1 transceiver eeprom
Ethernet15 EEPROM:
...
  Frequency tuning support (04h:128-129):
    Grid spacing capabilities (04h:128):
      100 GHz grid supported (04h:128): true
      12.5 GHz grid supported (04h:128): false
      25 GHz grid supported (04h:128): false
      3.125 GHz grid supported (04h:128): false
      33 GHz grid supported (04h:128): false
      50 GHz grid supported (04h:128): false
      6.25 GHz grid supported (04h:128): false
      75 GHz grid supported (04h:128): true
    Tunable wavelength (04h:128): true
    Fine tuning support (04h:129): false
  Supported channel boundaries (04h:130-161):
    100 GHz grid (04h:150-153):
      Lowest channel (04h:150-151): -18
      Lowest frequency (04h:150-151): 191300000 MHz
      Highest channel (04h:152-153): 30
      Highest frequency (04h:152-153): 196100000 MHz
    75 GHz grid (04h:158-161):
      Lowest channel (04h:158-159): -72
      Lowest frequency (04h:158-159): 191300000 MHz
      Highest channel (04h:160-161): 120
      Highest frequency (04h:160-161): 196100000 MHz
  Programmable output power advertisement (04h:196-201):
    Lane programmable output power supported (04h:196): false
  VDM configuration (20h:128-255;21h:128-255):
    VDM group 1 (20h:128-255):
      Parameter 1 (20h:128-129):
        Lane (20h:128): 0
        Threshold ID (20h:128): 0
        Parameter type (20h:129): Laser temperature
      Parameter 3 (20h:132-133):
        Lane (20h:132): 0
        Threshold ID (20h:132): 2
        Parameter type (20h:133): eSNR host input
      Parameter 4 (20h:134-135):
        Lane (20h:134): 1
        Threshold ID (20h:134): 2
        Parameter type (20h:135): eSNR host input
      Parameter 5 (20h:136-137):
        Lane (20h:136): 2
        Threshold ID (20h:136): 2
        Parameter type (20h:137): eSNR host input
...
       Parameter 84 (21h:166-167):
        Lane (21h:166): 0
        Threshold ID (21h:166): 14
        Parameter type (21h:167): MER
  Number of VDM groups supported (2Fh:128): 2

“Show interface transceiver dom” command – current performance data

show interface [<intf>] transceiver dom – displays the most important current performance data on the media (line) side

switch#show interface Ethernet11/1 transceiver dom
Ch: Channel, N/A: not applicable, TX: transmit, RX: receive
mA: milliamperes, dBm: decibels (milliwatts), C: Celsius, V: Volts

Port 11
Last update: 0:00:05 ago
                                             Value
                                        ----------------
   Case temperature                         66.59 C
   Voltage                                   3.26 V
   TX power                                -10.23 dBm
   RX total power                          -11.61 dBm
   RX channel power                        -11.94 dBm
   Pre-FEC BER                           1.82e-03 
   Post-FEC errored frames ratio         0.00e+00 
   Chromatic dispersion (short link)         0.00 ps/nm
   Chromatic dispersion (long link)          0.00 ps/nm
   Differential group delay                  9.31 ps
   SOPMD                                     0.00 ps^2
   Polarization dependent loss               0.40 dB
   Received OSNR estimate                   35.10 dB
   Received ESNR estimate                   17.50 dB
   Carrier frequency offset                  0.00 MHz
   Error vector magnitude                  100.00 %
   SOP rate of change                        0.00 krad/s
   Laser temperature                        59.54 C
   Laser frequency                         193100.00 GHz

  • BER: Bit Error Rate
  • FEC: Forward Error Correction
  • OSNR: Optical Signal to Noise Ratio
  • ESNR: Electrical Signal to Noise Ratio
  • SOP: State of Polarization
  • SOPMD: State of Polarization Mode Dispersion

“Show transceiver status interface” command – alarms, faults, and interface status

show transceiver status interface [<intf>] – displays the most important alarms, faults and interface status.

For 400GBASE-ZR modules, media and host side coherent alarms, host-side pre-FEC BER, defined in the Coherent CMIS and post-FEC BER have been added to the command’s output:

switch#show transceiver status interface Ethernet14/1
                                Current State        Changes       Last Change
                                -------------        -------       -----------
Port 14
  Transceiver                   400GBASE-ZR                1       0:15:09 ago
  Transceiver SN                200554050                          
  Presence                      present                            
  Adapters                      none                               
  Bad EEPROM checksums                                     0       never
  Resets                                                   0       0:15:14 ago
  Interrupts                                               0       never
  Data path firmware fault      ok                         0       never
  Module firmware fault         ok                         0       never
  Temperature high alarm        ok                         0       never
  Temperature high warn         ok                         0       never
  Temperature low alarm         ok                         0       never
  Temperature low warn          ok                         0       never
  Voltage high alarm            ok                         0       never
  Voltage high warn             ok                         0       never
  Voltage low alarm             ok                         0       never
  Voltage low warn              ok                         0       never
  Module state                  ready                      4       0:14:55 ago
  Data path 1 state             activated                  4       0:07:21 ago
  Data path 2 state             activated                  4       0:07:21 ago
  Data path 3 state             activated                  4       0:07:21 ago
  Data path 4 state             activated                  4       0:07:21 ago
  Data path 5 state             activated                  4       0:07:21 ago
  Data path 6 state             activated                  4       0:07:21 ago
  Data path 7 state             activated                  4       0:07:21 ago
  Data path 8 state             activated                  4       0:07:21 ago
  RX LOS                        ok                         2       0:05:54 ago
  TX fault                      ok                         0       never
  RX CDR LOL                    ok                         0       never
  TX power high alarm           ok                         0       never
  TX power high warn            ok                         2       0:07:39 ago
  TX power low alarm            ok                         2       0:07:50 ago
  TX power low warn             ok                         4       0:07:39 ago
  TX bias high alarm            ok                         0       never
  TX bias high warn             ok                         0       never
  TX bias low alarm             ok                         0       never
  TX bias low warn              ok                         0       never
  RX power high alarm           ok                         0       never
  RX power high warn            ok                         0       never
  RX power low alarm            ok                         0       never
  RX power low warn             ok                         0       never
  TX loss of alignment          ok                         0       never
  TX out of alignment           ok                         0       never
  TX clock monitor unit LOL     ok                         0       never
  TX reference clock LOL        ok                         0       never
  TX deskew LOL                 ok                         0       never
  TX FIFO error                 ok                         0       never
  RX demodulator LOL            ok                         0       never
  RX CD compensation LOL        ok                         0       never
  RX loss of alignment          ok                         0       never
  RX out of alignment           ok                         0       never
  RX deskew LOL                 ok                         0       never
  RX FIFO error                 ok                         0       never
  RX FEC excessive degrade      ok                         0       never
  RX FEC detected degrade       ok                         0       never
  Freq tuning in progress       idle                       0       never
  Freq tuning busy              ok                         0       never
  Freq tuning invalid channel   ok                         0       never
  Freq tuning completed         no                         2       0:07:34 ago
Ethernet14/1
  Operational speed             400Gbps                            
  Pre-FEC bit error rate        0.00e+00                           
  Post-FEC errored frames ratio 0.00e+00                           
  TX LOS
    Host lane 1                 ok                         0       never
    Host lane 2                 ok                         0       never
    Host lane 3                 ok                         0       never
    Host lane 4                 ok                         0       never
    Host lane 5                 ok                         0       never
    Host lane 6                 ok                         0       never
    Host lane 7                 ok                         0       never
    Host lane 8                 ok                         0       never
  TX CDR LOL
    Host lane 1                 ok                         0       never
    Host lane 2                 ok                         0       never
    Host lane 3                 ok                         0       never
    Host lane 4                 ok                         0       never
    Host lane 5                 ok                         0       never
    Host lane 6                 ok                         0       never
    Host lane 7                 ok                         0       never
    Host lane 8                 ok                         0       never
  TX adaptive input EQ fault
    Host lane 1                 ok                         0       never
    Host lane 2                 ok                         0       never
    Host lane 3                 ok                         0       never
    Host lane 4                 ok                         0       never
    Host lane 5                 ok                         0       never
    Host lane 6                 ok                         0       never
    Host lane 7                 ok                         0       never
    Host lane 8                 ok                         0       never

Note: in the operational state, module state is ‘Ready’ and datapath state for all 8 lanes is ‘activated’

“Wavelength/frequency and output power status”

The ‘show interface [<intf>] transceiver hardware’ command displays the configured and programmed wavelength and output power. It takes a short time for the configured wavelength or output power to be programmed into the transceiver. The configured wavelength/output power and programmed wavelength/output power should not differ for extended periods of time.

switch#show interface Ethernet23/1 trans hardware
Name: Et23/1
Media type: 400GBASE-ZR
Maximum module power (W): 20.0
Maximum slot power (W): 20.0
Configured frequency (GHz): 193100.0
Computed wavelength (nm): 1552.52
Operational frequency (GHz): 193,100.0
Operational wavelength (nm): 1552.52
Configured TX power (dBm): -10.0
Operational TX power (dBm): -10.0

Note: configured transmit output power and operational transmit output power are only displayed if configurable output power is supported by the module.

Syslog messages

N/A

Troubleshooting

  1. Check transceiver type – make sure it is 400GBASE-ZR
  2. Check peer transceiver – make sure it is also 400GBASE-ZR.
  3. Check that channel/frequency is configured and interfaces are not in errdisabled state
  4. Check that the selected frequency is matching on both sides of the optical link
  5. Check that transmit output is not disabled
  6. If Tx power is not configured, check that it is in the range of -6 dB to -12 dB.
  7. If  Tx power is configured, check that matches its configured values on both sides of the optical link.
  8. Check Rx power. The best performance of an optical link is achieved when the received power is -10dB.
  9. Collect the output of CLI commands listed in the “Show commands” section before making a request for support from the development team.
  10. Check that pre-FEC BER ( ‘show transceiver dom’ command ) is in the correctable range ( less than 1e-2 ).
  11. ‘show transceiver status interface’ – check that module state is ‘Ready’, datapath state is ‘Activated’. Check for possible alarms and faults

Link issues

If a link has issues, the following commands and files are useful for debugging and Arista TAC.

  • show interfaces <interfaceName> phy detail
  • show interfaces <interfaceName> transceiver detail
  • show idprom transceiver <interfaceName> ext
  • files in /var/log/agents/*
  • files in /var/log/qt/*
  • /var/log/messages*

Fiber Cleaning

See [8] for recommended best practices of fiber cleaning.

Tracing

N/A

Limitations

400GBASE-ZR transceivers are relatively new.  The modules could be Arista branded or 3rd party, from multiple vendors.  For 3rd party optics, some optional properties (Tx output power, support of 75GHz grid) or DOM properties in the VDM pages, may not be implemented.

Resources

  1. Common Management Interface Specification ( CMIS4.0 )
  2. Common Management Interface Specification draft ( CMIS4.1 )
  3. OIF Implementation Agreement for Coherent CMIS, rev1.1
  4. OIF Implementation Agreement for 400ZR, OIF-400ZR-01.0
  5. CFP2-DCO configuration, status and troubleshooting
  6. Arista Networks: Data Center Interconnect – Simplified
  7. Arista Networks: Fiber Cleaning Application Note

Follow

Get every new post on this blog delivered to your Inbox.

Join other followers: