A Brief Discussion on 100G Optical Modules in Data Centers

Technology Feast | A Brief Discussion on 100G Optical Modules in Data Centers

Time: August 27th, 2024

Traditional data centers are mainly based on a 10G network architecture. In order to adapt to the large-scale deployment of services such as AI, deep learning, and big data computing, the next-generation data center architecture is evolving towards 25G/100G network architecture. In China, Internet giants such as BAT have already achieved large-scale deployment.

Building a 25G/100G data center requires a large number of 100G optical modules, which account for a high proportion of the network construction cost. What are the 100G optical module standards and how should we choose? Today, we will briefly sort out the 100G optical module standards and packaging formats for data centers.

100G optical module standard organization

Before we start sharing the optical module standards, let's first understand the standardization organizations of optical modules. The definition of optical modules is mainly based on two key organizations, namely IEEE and MSA (Multi Source Agreement), which complement and learn from each other.

Everyone knows that IEEE is the Institute of Electrical and Electronics Engineers, and 802.3 is a working group under IEEE. Many 10G, 40G, 100G, and 400G optical module standards are proposed by the IEEE 802.3 working group.

MSA is a multi-vendor specification. Compared with IEEE, it is a non-official organization. Different MSA protocols will be formed for different optical module standards, which can be understood as the behavior of the industry's corporate alliance. In addition to defining the structural packaging of optical modules (including external dimensions, electrical connectors, pin assignments, etc.), MSA also defines electrical interfaces and optical interfaces, thus forming a complete optical module standard.

A long time ago, the optical module industry chain was very chaotic. Each manufacturer had its structural packaging, and the optical modules they developed were large and small, and the interfaces were also varied. In order to solve this problem, the MSA multi-source agreement came into being. All manufacturers followed the standards proposed by MSA to unify the structural packaging and related interfaces of optical modules, like the standardization of mobile phone charging ports. For 100G, the standards defined by MSA include 100G PSM4 MSA, 100G CWDM4 MSA, and 100G Lambda MSA.

100G Optical Module Standards

100G interconnection scenarios of different distances, IEEE and MSA have defined more than ten 100G standards, but the mainstream ones are the following six standards.

Standard	Formulating bodies	Connector	Type of fiber	Transmission distance
100GBASE SR10	IEEE 802.3	24core MPO, 10 transmitter and 10 receiver	Multimode fiber, 850 nm	OM3 100M OM4 150m
100GBASE-SR4	IEEE 802.3	12-core MPO, 4 transmitter and 4 receiver	Multimode fiber, 850 nm	OM4 100m
100GBASE-LR4	IEEE 802.3	Duplex Lc, 1 transmitter and 1 receiver	single mode fiber 1295.56~1309.14nm	10KM
100GBASE-ER4	IEEE 802.3	Duplex LC, 1 transmitter and 1receiver	single mode fiber 1295.56~1309.14nm	40KM
100G PSM4	MSA	12-core MPO, 4 transmitter and 4receiver	single mode fiber 1310nm	500m
100G CWDM4	MSA	Duplex LC, 1 transmitter and 1 receiver	single mode fiber 1271nm-1331nm	2KM

▲ Table 1- Mainstream standards for 100G optical modules

The standards starting with 100GBASE are all proposed by IEEE 802.3.

▲FIG-1

As shown in the figure above: In 100GBASE-LR4, LR stands for long reach, i.e. 10 km, and 4 stands for four channels, i.e. 4*25G, which together form a 100G optical module that can transmit 10 km.

The naming convention for - R is as follows:

PMD Type	Transmission distance	Remarks
KR	Tens of centimeters	k, backplane, signal transmission between backplanes
CR	A few meters	C, copper, coaxial cable connection
SR	Tens of meters	S，short， Short distance, high-speed optical modules generally use multimode optical fibers
DR	500 m	D, Note: PSM4 has a transmission range of 500 meters, but it does not belong to the IEEE standard system
FR	2 km	F, Note: 100 G CWD4 is also 2 km, but it is not an IEEE standard, it is defined by the MSA.
LR	10 km	L, long, long distance
ER	40 km	E, extended distance, extended relative to LR
ZR	80 km	Non IEEE Standard

▲ Table 2- R Glossary (Noun Explanation)

In addition to the 100GBASE series standards proposed by IEEE, why did MSA propose the PSM4 and CWDM4 standards?

100GBASE-SR4 and 100GBASE-LR4 are defined by IEEE Commonly used 100G interface specifications. However, for the internal interconnection scenarios of large data centers, the distance supported by 100GBASE-SR4 is too short to meet all interconnection requirements, while the cost of 100GBASE-LR4 is too high. Therefore, MSA brought medium-distance interconnection solutions to the market, and PSM4 and CWDM4 are the products of this revolution.

Of course, the capabilities of 100GBASE-LR4 completely cover CWDM4, but in the 2km transmission scenario, the CWDM4 solution is lower in cost and more competitive.

The following figure is the schematic diagram of 100GBASE-LR4 and 100G CWDM4:

▲ FIG 2- 100GBASE-LR4 schematic diagram

▲ FIG 3- 100G CWDM4 schematic diagram

LR4 and CWDM4 are similar in principle, both use optical devices MUX and DEMUX to multiplex four parallel 25G channels into one 100G optical fiber link. However, there are several differences between the two:

MUX/DEMUX devices used in LR4 are more expensive

CWDM4 defines a CWDM interval of 20nm. Since the wavelength temperature drift characteristic of the laser is approximately 0.08nm/°C, the wavelength variation within the working range of 0~70°C is approximately 5.6nm, and some isolation bands must be left in the channel itself.

Channel 1: 1264.5~1277.5nm

Channel 2: 1284.5~1297.5nm

Channel 3: 1304.5~1317.5nm

Channel 4: 1324.5~1337.5nm

LR4 defines a LAN-WDM spacing of 4.5nm.

Channel 1: 1294.53~1296.59nm

Channel 2: 1299.02~1301.09nm

Channel 3: 1303.54~1305.63nm

Channel 4: 1308.09~1310.19nm

The lower the requirements for optical MUX/DEMUX devices, which can save costs.

The laser used in LR4 is more expensive and consumes more power

CWDM4 uses DML (Direct Modulated Laser), while LR4 uses EML (Electro- absorption Modulated Laser).

DML is a single laser, while EML is two devices, one is DML and the other is an EAM modulator, together they are called EML. The principle of DML is to achieve signal modulation by modulating the injection current of the laser. Since the injection current will change the refractive index of the laser active area, causing wavelength drift (chirp) and thus dispersion, it is difficult to perform high-speed signal modulation and the transmission is not far enough. 10KM is a bit beyond the capability of DML, so EML is the only choice.

Note: Chirp refers to a signal whose frequency changes (increases or decreases) over time. This signal sounds similar to the chirping of birds.

LR4 requires an additional TEC (Thermo Electric Cooler)

Because the interval between adjacent channels of LR4 is only 4.5nm, the laser needs to be placed on the TEC for temperature control. The TEC Driver chip needs to be placed on the circuit, and the laser also needs to be integrated into the TEC material. In this way, the cost of LR4 is increased compared to CWDM4.

Based on the above three points, the cost of optical modules of the 100GBASE-LR4 standard is higher, so the 100G CWDM4 standard proposed by MSA fills the gap caused by the high cost of 100GBASE-LR4 within 2 km.

In addition to CWDM4, PSM4 is also a medium-distance transmission solution. So what are the advantages and disadvantages of PSM4 compared to CWDM4?

The 100G PSM4 specification defines a point-to-point 100 Gbps link over 8 single-mode fibers (4 transmit and 4 receive), with each channel transmitting at 25 Gbps. Each signal direction uses four independent channels of the same wavelength. Therefore, two transceivers typically communicate via an 8- 8-fiber MTP/MPO single-mode patch cord. The transmission distance of PSM4 is up to 500 meters.

▲ FIG 4- PSM4 schematic diagram

To summarize, as shown in thetablebelow, due to the use of wavelength division multiplexers, the cost of CWDM4 optical modules is higher than that of PSM4 optical modules. However, CWDM4 only needs two single-mode fibers for bidirectional transmission and reception, which is far less than the eight single-mode fibers of PSM4. As the distance increases, the total cost of the PSM4 solution rises very quickly. In actual applications, it is necessary to decide whether to use PSM4 or CWDM4 based on the interconnection distance.

	CWDM4 optical module	PSM4 optical module
Optical emitter	Four CWDM directly modulated lasers (with a wavelength interval of 20nm)	Four integrated silicon photon modulators and one distributed feedback laser.
Four wavelength CWDM wavelength division multiplexer	required	Not required
Interface type	Duplex LC interface	MPO/MTP interface (8-core)
Distance	< 2 KM	< 500 M

▲Table 4-CWDM4 vs PSM4

After talking about the 100G medium and long-distance optical module standards, let's take a look at the 100G short-distance optical module.

100G short-distance optical module standards: 100GBASE-SR10 and 100GBASE-SR4. In order to meet the 100G demand in the market, the 100GBASE-SR10 standard was first proposed and applied to 100G short-distance interconnection.

100GBASE-SR10 standard uses 10 x 10Gbps parallel channels to achieve 100Gbps point-to-point transmission. The rate of electrical signals is 10G, and the rate of optical signals is also 10G. It uses NRZ modulation and 64B/66B encoding. Because IEEE 802.3 proposed the 100GBASE-SR10 standard as early as 2010, the electrical interface of the switch ASIC chip ( Application Specific Integrated Circuit ) could only support 10G at most, that is, CAUI-10 ( 10 channels x 10Gbps ).

▲ FIG 5- 100GBASE-SR10 schematic diagram

As the electrical interface rate of the switch ASIC chip increases from 10Gpbs to 25G bps, the electrical interface standard is upgraded from CAUI-10 (10 channels x 10Gbps) to CAUI-4 (4 channels x 25Gbps), and the channels are reduced from SR10’s 10 parallel channels to 4 parallel channels, the number of optical module components is reduced, the cost is reduced, the module size is reduced, and the power consumption is reduced.

The reduction in the size of optical modules allows switches to provide a higher density of 100G interfaces per 1U space. Based on the above advantages, 100GBASE-SR4 has replaced 100GBASE-SR10 to become the mainstream 100G short-distance optical module standard.

▲ FIG 6- 100GBASE-SR4 schematic diagram

100G optical module packaging

It is not enough to have only the optical interface and electrical interface specifications of the optical module. A complete optical module solution also requires a matching structural package. The packaging formats of 100G optical modules mainly include CFP, CFP2, CFP4, and QSFP28.

CFP was first proposed, and the 100GBASE-SR10 standard was used for short-distance transmission, and 100GBASE-LR4 was used for long-distance transmission. The first-generation CFP long-distance transmission solution is as follows. Because the electrical interface capacity is only CAUI-10, a built-in gearbox (10:4 serialized in the figure below) is required to achieve the conversion of 10 x 10Gbps and 4 x 25Gbps electrical signals. Later, as the electrical signal was upgraded to CAUI-4, the second-generation CFP (CFP2/CFP4) long-distance transmission solution did not require a built-in gearbox.

▲ FIG 7- The first generation of CFP optical module long-distance solution

However, the size of CFP is too large. As the integration of optical modules becomes higher and higher, the subsequent development direction is to reduce the size and power consumption. CFP has evolved to CFP2, CFP4, and then to QSFP28. Compared with CFP4, QSFP28 is smaller in size and consumes less power. The smaller size of QSFP28 enables the switch to have a higher port density (typically, each board can deploy 36 100G interfaces). Currently, QSFP28 is the mainstream packaging format for 100G optical modules in data centers.

▲ FIG 8- CFP/CFP2/CFP4/QSFP28 optical module size comparison

To summarize, regarding how to choose 25G/100G data center internal interconnect optical modules, it is recommended that you refer to the following standards:

● 100G short-reach interconnection scenarios (TOR-LEAF) of no more than 100 meters use 100GBASE-SR4 QSFP28 optical modules;

● 100G mid-range interconnection scenario (LEAF-SPINE) of 100m to 500m, using 100G PSM4 QSFP28 optical module;

● 100G medium- and long-distance interconnection scenarios ( LEAF-SPINE, SPINE-CORE ) from 500m to 2km, using 100G CWDM4 QSFP28 optical modules ;

● Long-distance interconnection scenarios exceeding 2 km (CORE-MAN), use 100GBASE-LR4 QSFP28 optical modules.

The following table 5 presents an explanation of professional terms.

Optical communication terminology	Noun Explanation	Full name
100G Lambda MSA	Single channel 100Gbps optical interface multi-source protocol	100G Lambda Multi-Source Agreement
ASIC	Application specific integrated circuit	Application Specific Integrated Circuit
CAUI	100G Ethernet electrical interface	100Gbps Attachment Unit Interface
CDR	Clock data recovery	Clock Data Recovery
CFP	100G pluggable optical module	Centum Form-factor Pluggable
CWDM4	Four channel coarse wavelength division multiplexing	Coarse Wavelength Division Multiplexing 4
DeMux	Optical DE multiplexer (splitter)	DE multiplexer
DML	Direct modulation laser	Direct Modulated Laser
EAM	Electro absorption modulator	Electro Absorption Modulator
EML	Electro absorption modulated laser	Electro-absorption Modulated Laser
IEEE	Institute of Electrical and Electronics Engineers	Institute of Electrical and Electronics Engineer
LAN-WDM	(Local Area Network) Wavelength Division Multiplexing (WDM)	(Local Area Network) Wavelength Division Multiplexing
MD	Monitoring photodiodes	Monitor Diode
MMF	Multimode fiber	Multi-Mode Fiber
MSA	Multi source protocol	Multi Source Agreement
Mux	Optical multiplexer (multiplexer)	Multiplexer
NRZ	Non zeroing	Non Return Zero
PSM4	Four parallel single-mode channels	Parallel Single Mode 4 lane
QSFP28	Four channel small pluggable optical module	Quad Small Form-factor Pluggable 28
SMF	Single-mode fiber	Single-Mode Fiber
TEC	Semiconductor thermoelectric cooler	Thermo Electric Cooler
TIA	Trans-impedance amplifier	Trans-impedance Amplifier

▲Table 5

Summary

This document provides an overview of 100G optical modules in data centers. It discusses the evolution of data center architecture towards 25G/100G networks to meet the demands of AI, deep learning, and big data computing. The document explains the standardization organizations for optical modules, including IEEE and MSA, and highlights the different standards proposed by these organizations. It also compares various 100G optical module standards such as CWDM4, PSM4, LR4, and SR4 based on their transmission distances and cost considerations. Additionally, it discusses the packaging formats for 100G optical modules: CFP, CFP2/CFP4, and QSFP28.

Technology Feast | How to build a lossless network for RDMA

How Enterprise Switches and Network Switches Enhance Modern Networks

Technology Feast | A Brief Discussion on 100G Optical Modules in Data Centers

100G optical module standard organization

100G Optical Module Standards

Summary

Featured blogs