Spectrum-X is easy to misread as “NVIDIA’s Ethernet switch”. It isn’t a switch. It’s a pairing: Spectrum-4 switches and SuperNICs, each doing half of a job that neither can do alone. The switch sprays a flow’s packets across every path; the NIC at the other end puts them back in order. The switch sees congestion first; the NIC slows the sender. Once you see it as one system with two halves, most exam questions about it answer themselves.
This is part 3 of the NCP-AIN study series and the start of the Spectrum domain, 30% of the exam. NVIDIA’s blueprint lists, among others, Spectrum-X RoCE setup, QoS with ECN and PFC, BGP-EVPN multi-tenancy, DSX Air simulation, telemetry with What Just Happened and NetQ, DOCA and the SuperNIC. This article covers the architecture they all hang on; the next ones go deep on RoCE, EVPN, telemetry and the host.
In short
- Two halves, one system. Spectrum-4 switches route each packet to the least-loaded uplink; SuperNICs (BlueField-3, ConnectX-8) reorder those packets at the receiver and run per-flow congestion control at the sender.
- Spraying needs reordering. A plain RoCE NIC treats out-of-order packets as loss and resends. That’s why adaptive routing on Spectrum-X is defined with the SuperNIC, not the switch alone.
- Congestion control is split. The switch marks and measures; the SuperNIC decides each flow’s rate on its own congestion-control engine, programmable through DOCA.
- On the switch it’s Cumulus Linux and NVUE.
nv set qos rocebuilds the lossless RoCE profile;nv set router adaptive-routing state enabledplus per-port enablement turns on adaptive routing. Know the limits: not on bonds or SVIs, and not on 800G links on Spectrum-4.
The pieces
| Piece | What it is | Facts worth knowing |
|---|---|---|
| Spectrum-4 switch | SN5600 and SN5610, the Ethernet switches Spectrum-X builds on | 51.2 Tb/s; 64 ports of 800GbE or 128 of 400GbE; Cumulus Linux with NVUE |
| BlueField-3 SuperNIC | The east-west NIC for GPU servers | B3140H: one 400 Gb/s port (QSFP112), PCIe Gen 5 x16, 8 Arm cores, 16 GB DDR5 |
| ConnectX-8 SuperNIC | The next-generation SuperNIC | 800 Gb/s of networking per device, with an integrated PCIe Gen6 switch (48 lanes); in GB300 NVL72 that works out to 800 Gb/s per GPU |
| DOCA | NVIDIA’s host and NIC software stack | Drivers, firmware tools, congestion-control configuration and programming |
| NetQ, What Just Happened | Fabric validation and telemetry; per-packet drop reasons on the switch | Covered in Article 6 |
| Spectrum-X Photonics | Spectrum-6 switches with co-packaged optics (SN6800, SN6810) | The next hardware generation; same platform idea |
NVIDIA describes the SuperNIC as a class of network accelerator for AI clouds built on BlueField-3, with high-speed packet reordering when paired with an NVIDIA switch and congestion control driven by real-time telemetry. It uses less compute than a DPU because its job is narrower: move GPU traffic fast and fairly, rather than run infrastructure services. The BlueField-3 DPU, with the same silicon, is the choice when you need those services on the card, typically on the front-end and storage side.
Who does what
An analogy helps me here. A courier splits one big shipment across several vans so no single road jams, and it only works because the receiving warehouse reassembles the boxes in order before anyone opens them. The switch is the dispatcher picking roads; the SuperNIC is the warehouse. Take away either and the shipment arrives late or in pieces.
| Job | Switch (Spectrum-4) | SuperNIC (BlueField-3, ConnectX-8) |
|---|---|---|
| Load balancing | Adaptive routing: picks the less-loaded path per packet, from queue occupancy and port utilization | Accepts packets out of order and restores order |
| Data placement | Places the reordered data in host or GPU memory; the application never sees reordering | |
| Congestion detection | ECN marking, telemetry on queues | Receives the signals |
| Congestion reaction | PFC pause as the lossless backstop | Per-flow rate control on its congestion-control engine |
| Multi-tenancy | Isolation features, EVPN (Article 5) | Telemetry-based congestion control, which NVIDIA describes as providing noise isolation between tenants |
Key point. Adaptive routing and reordering are one feature split across two devices. Either one alone makes things worse: spraying without reordering causes retransmissions; reordering without spraying does nothing.
Why per-packet routing needs a NIC that reorders
Part 1 showed why per-flow ECMP fails: a few large flows collide on the same uplinks. The fix that removes the problem entirely is to stop routing per flow and route per packet, so every flow uses every uplink.
Cumulus Linux implements that on Spectrum-4: the switch forwards each packet to the less-loaded path, deciding from queue occupancy and port utilization, with a fixed path-selection interval of one microsecond. It can spread traffic across all members of an ECMP group, equally or by weight, and split a flow whose rate exceeds one uplink.
The cost is ordering. Packets of one flow now take different paths with different queue depths, so they arrive out of order. RDMA over standard RoCE expects order: many adapters treat a gap in sequence numbers as loss and go back to resend everything after it. Spraying into such a NIC turns a balancing feature into a retransmission storm. NVIDIA’s answer, in its own words, is that the BlueField-3 SuperNIC makes sure to reorder the packets and place them in host memory, leaving the reordering invisible to the application. That pairing is the core of Spectrum-X.
It’s the same principle as Ultra Ethernet: once the receiver can place packets as they arrive, the fabric is free to spray. Spectrum-X does it within NVIDIA’s own switch and NIC pairing; Ultra Ethernet standardizes it across vendors.
Congestion control: switch signals, NIC decisions
Spraying evens out the uplinks, but it can’t fix convergence: when many GPUs send to one, every path ends at the same port. That needs the senders to slow down, and Spectrum-X splits that job too:
- The switch detects congestion where it happens: ECN marks on the RoCE queue, telemetry on queue depth and port utilization, and PFC as the lossless backstop if a queue keeps growing.
- The SuperNIC decides each flow’s rate. On BlueField-3, congestion control runs on the NIC’s DPA-based engine, per flow, from hardware congestion events.
DOCA exposes that engine in two ways, and the distinction is exam-relevant:
| Mode | What runs | Notes |
|---|---|---|
| Native algorithms | DCQCN (ECN-based, for interoperability) or ZTR-RTT (adds hardware-timestamped RTT, for Zero Touch RoCE deployments without special switch configuration) | Configured per port |
| Programmable congestion control (DOCA PCC) | A congestion-control algorithm written with DOCA, executed on the same engine | DOCA 2.7 shipped PCC tuned for Spectrum-X 1.0.1, the reference architecture used in NVIDIA’s Israel-1 system |
The two are mutually exclusive, selected by a firmware setting (USER_PROGRAMMABLE_CC). NVIDIA’s Spectrum-X materials describe the combination as telemetry-based congestion control: switches supply real-time congestion information and the SuperNICs adjust each flow’s sending rate from it.
The switch side, in NVUE
On the switches, Spectrum-X is configured with Cumulus Linux’s NVUE. Two features matter here; the details of each get their own article later. Illustrative, from the Cumulus Linux 5.15 documentation; validate against your release.
Lossless RoCE is one command, and lossless is the default mode:
nv set qos roce # or: nv set qos roce mode lossless
nv config apply
nv show qos roce # mode, PFC, ECN, trust and the mappings
nv show interface swp1 qos roce counters # RoCE and CNP traffic, PFC pauses, ECN marks, buffer use
What the lossless profile sets up:
| Traffic | DSCP | Switch priority | Traffic class | Treatment |
|---|---|---|---|---|
| RoCE | 26 | 3 | 3 | PFC enabled, ECN marking, dedicated RoCE buffer pool, DWRR 50% |
| CNP | 48 | 6 | 6 | Strict priority |
| Everything else | others | 0 | 0 | Lossy, DWRR 50% |
The documentation’s example output shows ECN marking between about 146 KB and 1.43 MB on the RoCE class. Other modes exist: lossy (ECN without PFC), lossless-single-ipool, and lossy-multi-tc, which enables packet trimming, the switch-side feature Ultra Ethernet relies on.
Adaptive routing is enabled globally and on the ECMP member ports:
nv set router adaptive-routing state enabled
nv set interface swp51 router adaptive-routing state enabled # every uplink in the same ECMP group
nv set interface swp52 router adaptive-routing state enabled
nv config apply # reloads switchd (5.13 and later)
nv show router adaptive-routing
nv show interface swp51 router adaptive-routing
The limits are classic exam material:
- Not on layer 3 subinterfaces, SVIs, bonds or bond members. Uplinks must be plain routed ports.
- No resilient hashing on adaptive-routing ports; the switch chooses per packet instead.
- Spectrum-4 at 400G and 200G only; the documentation says Spectrum-4 doesn’t support adaptive routing on 800G links.
- Supported traffic: RoCEv2 unicast, VXLAN-encapsulated RoCE, layer 3 interfaces and next hops in the default VRF.
- Same ports, same routes: traffic meant for adaptive routing must use ports where it’s enabled; mixing with plain ECMP ports causes unexpected hashing.
- Syntax changed: up to Cumulus Linux 5.14 the commands use
enable on/enable offinstead ofstate enabled. Up to 5.12, applying it restartsswitchd; from 5.13 it reloads it. - Optional link-utilization threshold (default 70%) is only available with the custom profile; NVIDIA asks you to involve support for profile changes.
Practising it
The blueprint names DSX Air, NVIDIA’s simulation platform, and it’s the right place to practise NVUE: building a leaf-spine, applying the RoCE profile, enabling adaptive routing and reading the show commands. Know what simulation can’t show: the simulated switches run Cumulus VX, so you’ll practise configuration and verification, not the data-plane behaviour of PFC, ECN or packet spraying under load. For that you need real Spectrum hardware and SuperNICs.
What NVIDIA claims, and how to read it
NVIDIA says Spectrum-X boosts network performance by 1.6× for large-scale AI compared with off-the-shelf Ethernet, citing its Israel-1 system. Treat it as a vendor figure for a specific comparison: the gain comes from replacing per-flow ECMP and NIC-only congestion control, and your baseline determines your result. The architecture is what to learn; the multiplier is what to test.
Self-check
1. Why does Spectrum-X adaptive routing require SuperNICs at the endpoints?
Because the switch sprays a flow’s packets across paths per packet, so they arrive out of order. The SuperNIC reorders them and places the data in memory; a standard RoCE NIC would treat the gaps as loss and resend.
2. An engineer enables adaptive routing on a bond of two uplinks. What happens?
It isn’t supported: Cumulus Linux doesn’t support adaptive routing on bonds or bond members (nor on subinterfaces or SVIs). The uplinks must be individual routed ports in the same ECMP group.
3. What does `nv set qos roce` map DSCP 26 and DSCP 48 to, and which one gets PFC?
DSCP 26 to switch priority 3, traffic class 3 (RoCE, PFC enabled, ECN marking); DSCP 48 to switch priority 6, traffic class 6 (CNP, strict priority). PFC is enabled on priority 3 only.
4. Where does per-flow congestion control run in Spectrum-X, and what are the two ways to configure it?
On the SuperNIC’s DPA-based congestion-control engine. Either native algorithms (DCQCN or ZTR-RTT) or a programmable algorithm written with DOCA PCC; they’re mutually exclusive, selected by firmware setting.
5. A Spectrum-4 fabric runs 800G uplinks. Can it use adaptive routing on them?
Not according to the Cumulus Linux 5.15 documentation: Spectrum-4 supports adaptive routing at 400G and 200G, not on 800G links.
What’s next
Article 4 goes inside the RoCE profile: classification, PFC and headroom, ECN thresholds and the NIC’s reaction, and how to check that every hop agrees.
References
- NVIDIA, NVIDIA-Certified Professional: AI Networking: blueprint topics for the Spectrum domain.
- NVIDIA Technical Blog, Optimize large-scale AI workloads with NVIDIA Spectrum-X, August 2024: per-packet adaptive routing, reordering on the BlueField-3 SuperNIC, telemetry-based congestion control, the 1.6× claim.
- NVIDIA, What is a SuperNIC?: the SuperNIC’s role, reordering with an NVIDIA switch, telemetry-driven congestion control.
- NVIDIA, Spectrum-X and Ethernet switching: platform overview, SN5600 and SN5610 specifications, Spectrum-X Photonics.
- NVIDIA, BlueField-3 specifications and the ConnectX-8 SuperNIC user manual; NVIDIA Technical Blog, ConnectX-8 SuperNICs; NVIDIA, GB300 NVL72 (two ConnectX-8 per IO module, 800 Gb/s per GPU).
- NVIDIA, DOCA congestion control infrastructure: native DCQCN and ZTR-RTT, programmable CC,
USER_PROGRAMMABLE_CC. - NVIDIA Technical Blog, Enhancing AI cloud data centers and Spectrum-X with DOCA 2.7: Spectrum-X 1.0.1, lossless RoCE with adaptive routing and DOCA PCC.
- NVIDIA, Cumulus Linux 5.15: Equal Cost Multipath Load Sharing (adaptive routing) and RDMA over Converged Ethernet; NVUE reference, adaptive routing set commands.
- Related on this site: lossless RoCEv2, explained, the DCQCN research note and Ultra Ethernet in plain English.
This guide follows NVIDIA’s public blueprint and documentation; it contains no exam content. Commands are illustrative and change between releases: validate them against the documentation for your Cumulus Linux and DOCA versions. NVIDIA, Spectrum, Spectrum-X, BlueField, ConnectX, DOCA, Cumulus and NetQ are trademarks of NVIDIA Corporation.