Kenneth Finnegan
Required reading if you do any networking at any speed. The thread is full of fantastic pictures and brilliant explanations. boost
One of my nice friends at Hurricane Electric gave me a dead 100G-LR4 optic to tear apart for your entertainment so lets dig into it. post
100G-LR4 is a QSFP28 optic that runs over a duplex pair of single mode fiber, so it takes 4 lanes of 25G from the switch, modulates 4 different colors of light, and combines them together over a single strand to the other end up to 10km away.
First comes off the ejection bail. I guess we're committed now since that was rather destructive.Two tiny screws get us into the case. The grey putty stuff is thermal gap filler, which couples each part that puts off heat to the case for cooling.Think of it like thermal paste, but it's designed to be thick and span a gap instead of just be a thin film.
Lifting the electrical assembly out of the other half of the metal case, we can start pointing out the major points of interest.TOSA/ROSA - transmit/ receive optical sub assemblies are where the laser magic happen.The retimers help 4x25G get to the switch via the QSFP connector
The TOSA and ROSA both need to handle four wavelengths of light moving 25Gbps each, so they're actually connected to the PCB with two layers of ribbon cables. One of the ribbons is just for the 4x25Gbps data channels, and the other is for power, biasing, etc, not high speed stuff to run the OSA parts.
Looking on the other side of the whole assembly, the largest IC in the middle (the silver one) is the microcontroller that talks to the switch over an I2C bus to report telemetry and ID information, and pretty much everything else is voltage regulation for taking the 3.3V from the switch and regulating it down to lower voltages to drive the lasers.
And that microcontroller that runs the whole show is... a Cortex M3! An STM32F103C6 with 32kB of flash, 10kB of SRAM, and can run at up to 72MHz.Meaning that this optic has more compute power than many early home computers.You'll note that the ST microcontroller is silver, as are a few other ICs on this optic. This is actually a cost saving measure called "flip chip" packaging, where they don't waste the expense of the black plastic packaging and solder the silicon die straight down onto the board like it was a QFN. Absolutely wild.
While it's fun to think about malware on the ARM core in these, the data path is WAY faster than the Cortex M3 can handle. It's entirely an out-of-band manager of the electronics with no idea of what is going on in the Ethernet link itself.That being said, some newer optics really DO have interactions in the data path. The most notable is the speed-changing optics which allow modern Ethernet switches (which only support 10G/25G/50G on their front panel ports) to still link up with a 1G Ethernet peer. The optic does the 10:1 speed change and implements clause 37 autoneg entirely inside the optic.Granted, the firmware on optics actually *IS* field writable. I've coordinated with an optics vendor in the past to help a customer apply a firmware patch to 10,000 optics installed in Arista switches in the field by issuing the right magic sequence of write commands over the I2C bus from EOS.
Remember how I said the rest of the parts were mostly voltage regulation?Voltage regulators usually need some kind of inductance... so just saying... prime cute coil of wire opportunity.
So this is where things get REAL ugly... we break out the Dremel and cut the lid off the TOSA (Transmit Optical Sub Assembly) to expose the yummy optical bits inside it.
Remember that while there's four different waves for the four 25Gbps streams, 100G-LR4 runs over a single duplex pair of fibers, so all four optical signals need to make it out a single strand of fiber, so... PRISMS.
The colors are all way out in the infrared, so don't put too much stock in the colors I use in the picture, but hopefully you get the point about how the prisms are used as optical combiners.
The keen eye will also notice that some of the Dremel debris from me cutting open the TOSA is... stuck to this cube and fuzzy... like it's magnetic...And it it. That is a tiny magnet in the middle of the optical path, but why?Physics talk. Remember that the transmitters here are lasers, which mean that the four lasers on the right are each their own resonant cavity, which leaks a fraction of the light to be useful. But if there was some kind of partial reflection out in the fiber somewhere, this could cause problems and destabilize these lasers, so we would like some way to let light OUT of the TOSA without letting any reflected light back into the lasers to cause problems, so we make an optical one-way valve...MORE PHYSICS TALK: Light is a perpendicular E-M field, so if you used a vertical polarizer, and then passed the light through a magnetic field to rotate the polarization 45 degrees to line up with a //// diagonal polarizing filter on the other side of the magnet...Think about the reverse path. //// polarizing filter, 45 degree rotation, horizontal light hitting a vertical polarizing filtering. BLOCKED.
Putting the TOSA under my microscope, we can start to see some more of the finer details.This is rotated from the last photos.I'm not convinced whether the two missing lenses in the middle broke off before I got it, broke off while I was hacking my way into the package, or were just never there. A mystery for the ages.
For one thing, we see a tiny square and two bond wires landing on the platform holding the lasers and the first stage of the optics.This is a silicon diode. Fun diode fact: the junction voltage of a diode changes with temperature in a very predictable and consistent way, so it's very common for temperature sensors to measure the junction voltage of a diode or base of a transistor.
And a temperature sensor would imply some kind of temperature control, and this actually makes a lot of sense, because the four lasers are all different colors, but only SLIGHTLY different colors.Each of the four lasers are only 4.5nm apart in wavelength, so the temperature really matters to keep them in their channel, so there's a peltier cooler underneath the platform holding the lasers and first optical stage so that the optic can electrically heat or cool the lasers to keep them at the right temperature! https://en.wikipedia.org/wiki/Thermoelectric_cooling
The ROSA is somewhat less interesting. The right half seems to be a giant prism and I wasn't able to get it apart to any useful degree.The glassy fractures on it are from me trying to pry it out.
The four receive diodes on the left of the ROSA are facing up, which is perpendicular from the light coming in from the right, so the final stage of the ROSA optics were glued as a bridge with four little lenses and a right angle PRISM!
So lastly, here are some more photos of the PCB that runs from the QSFP edge connector to the ribbon cables attached to the ROSA and TOSA. /finThanks for reading. For those new here, I have a Liberapay which I'm using to fund the #MicroMirror project where I'm trying to fix Linux/free software distribution to users. https://liberapay.com/phirephly/
Text and images extracted from Mastodon thread using code in the browser's debugger window. This then cleaned up a bit with global replace commands in the programming text editor.
copy([...document.querySelectorAll('.status')].map(s => `<p>${s.getAttribute('aria-label')}</p><p>${[...s.querySelectorAll('img')].map(i => i.outerHTML).join("<br>")}`).join("<hr>\n"))