Optical Interconnects vs High-Speed Training Networks

Physical Layer vs. Fabric Layer: A Complementary Relationship

The most important thing to understand about optical interconnects and high-speed training networks is that they are not competing technologies — they operate at different layers of the stack. Optical interconnects are the physical medium, analogous to roads, while training networks are the traffic management system that routes vehicles. Every modern training network of meaningful scale already uses optical interconnects for inter-rack and increasingly intra-rack links. The question is not which to choose, but how deeply to integrate optics into the fabric.

This layered relationship is becoming more tightly coupled as co-packaged optics blur the boundary. When optical engines are integrated directly onto switch ASICs — as NVIDIA is doing with its Spectrum-X Photonics and Quantum-X platforms shipping in 2026 — the "physical layer" and "network layer" decisions become inseparable. Choosing a CPO-enabled switch means simultaneously choosing an optical interconnect strategy and a network fabric architecture.

The Bandwidth Crisis Driving Optical Adoption

At 200 Gb/s per lane, copper's physics hits a hard wall. Passive copper cables cannot span beyond a single server rack at these speeds, and even active electrical cables consume prohibitive power at scale. This is why optical interconnects have moved from optional to mandatory for any training cluster spanning multiple racks — which is every frontier training cluster in existence.

The numbers tell the story: NVIDIA's GB200 NVL72 system delivers 130 TB/s of aggregate NVLink bandwidth across 72 GPUs within a single rack-scale domain, but connecting multiple NVL72 racks requires an external fabric. At 800G per port, a single InfiniBand or Ethernet uplink carries a fraction of the intra-rack bandwidth, making the optical fabric's aggregate capacity the binding constraint on how efficiently distributed training parallelism strategies can operate across racks.

InfiniBand vs. Ethernet: The Training Network Protocol War

Within the training network layer, the most consequential battle is between InfiniBand and Ethernet. InfiniBand dominated through 2023–2024, commanding roughly 80% of AI training back-end network deployments, thanks to its sub-microsecond RDMA latency and tight integration with NVIDIA's software stack. But the landscape shifted dramatically in 2025.

The Ultra Ethernet Consortium released its 1.0 specification in June 2025, defining purpose-built congestion signaling, transport protocols, and telemetry for AI workloads. This is not simply RoCE rebranded — it is a rearchitected Ethernet stack designed for the all-reduce communication patterns that dominate large model training. By mid-2025, Ethernet overtook InfiniBand in AI back-end network market share, driven by hyperscaler validation at scale and the 1.5–2.5× per-port cost advantage. Both protocols, however, rely on the same underlying optical interconnect infrastructure for physical transport.

Co-Packaged Optics: Where the Layers Converge

Co-packaged optics represents the most significant architectural shift in datacenter networking since the move from 10G to 100G. By integrating optical engines directly onto or adjacent to switch and accelerator packages, CPO eliminates the power-hungry SerDes circuits at each end of an optical link. NVIDIA claims up to 3.5× power reduction and 10× resiliency improvement with its CPO-based platforms.

Lightmatter's Passage M1000, announced in March 2025, demonstrated 114 Tbps total optical bandwidth from a single photonic superchip. Their March 2026 milestone of 1.6 Tbps per fiber using 16-wavelength DWDM represents an 8× improvement over existing CPO solutions. These are not incremental gains — they represent the kind of step-function improvements that can reshape datacenter economics. Large-scale CPO deployment is expected to ramp between 2028 and 2030, but the design decisions being made today lock in which optical architecture a facility will use for its operational lifetime.

Topology and Parallelism: Where Networks Shape AI Training

High-speed training networks make decisions that optical interconnects cannot: how to route collective operations, which topology to deploy, and how to map model parallelism strategies onto physical network structure. A fat-tree topology provides full bisection bandwidth — any GPU can communicate with any other at line rate — but requires enormous switch and fiber counts. Dragonfly and rail-optimized topologies trade some cross-group bandwidth for dramatically lower cost.

The choice of topology directly determines which training parallelism strategies (data parallel, tensor parallel, pipeline parallel, expert parallel) are efficient. Tensor parallelism requires the highest bandwidth and lowest latency, making it practical only within NVLink domains or tightly connected optical groups. Pipeline and expert parallelism tolerate more latency but require consistent bandwidth, which optical fabrics provide more reliably than congestion-prone electrical networks.

Multi-Datacenter Training: The Next Frontier

As training runs scale beyond what a single facility can house, "scale-across" architectures are emerging that link multiple datacenters into a single logical training cluster. Ciena and other optical networking vendors are developing tailored optical solutions for these wide-area AI interconnects, with first field trials expected in 2026.

This is where optical interconnects and training networks face fundamentally different challenges. Optical technology can deliver massive bandwidth over metropolitan and even continental distances — wavelength-division multiplexing through existing fiber plant can provide tens of terabits per second between sites. But training networks must contend with speed-of-light latency that no amount of bandwidth can overcome. A 100-kilometer inter-datacenter link adds ~500 microseconds of round-trip latency, which is catastrophic for tightly synchronized training. Solving this requires training network innovations (asynchronous gradient methods, locality-aware scheduling) rather than optical improvements.