Spectrum-X is easy to misread as “NVIDIA’s Ethernet switch”. It isn’t a switch. It’s a pairing: Spectrum-4 switches and SuperNICs, each doing half of a job that neither can do alone. The switch sprays a flow’s packets across every path; the NIC at the other end puts them back in order. The switch sees congestion first; the NIC slows the sender. Once you see it as one system with two halves, most exam questions about it answer themselves.

This is part 3 of the NCP-AIN study series and the start of the Spectrum domain, 30% of the exam. NVIDIA’s blueprint lists, among others, Spectrum-X RoCE setup, QoS with ECN and PFC, BGP-EVPN multi-tenancy, DSX Air simulation, telemetry with What Just Happened and NetQ, DOCA and the SuperNIC. This article covers the architecture they all hang on; the next ones go deep on RoCE, EVPN, telemetry and the host.

In short

  • Two halves, one system. Spectrum-4 switches route each packet to the least-loaded uplink; SuperNICs (BlueField-3, ConnectX-8) reorder those packets at the receiver and run per-flow congestion control at the sender.
  • Spraying needs reordering. A plain RoCE NIC treats out-of-order packets as loss and resends. That’s why adaptive routing on Spectrum-X is defined with the SuperNIC, not the switch alone.
  • Congestion control is split. The switch marks and measures; the SuperNIC decides each flow’s rate on its own congestion-control engine, programmable through DOCA.
  • On the switch it’s Cumulus Linux and NVUE. nv set qos roce builds the lossless RoCE profile; nv set router adaptive-routing state enabled plus per-port enablement turns on adaptive routing. Know the limits: not on bonds or SVIs, and not on 800G links on Spectrum-4.

The pieces

Piece What it is Facts worth knowing
Spectrum-4 switch SN5600 and SN5610, the Ethernet switches Spectrum-X builds on 51.2 Tb/s; 64 ports of 800GbE or 128 of 400GbE; Cumulus Linux with NVUE
BlueField-3 SuperNIC The east-west NIC for GPU servers B3140H: one 400 Gb/s port (QSFP112), PCIe Gen 5 x16, 8 Arm cores, 16 GB DDR5
ConnectX-8 SuperNIC The next-generation SuperNIC 800 Gb/s of networking per device, with an integrated PCIe Gen6 switch (48 lanes); in GB300 NVL72 that works out to 800 Gb/s per GPU
DOCA NVIDIA’s host and NIC software stack Drivers, firmware tools, congestion-control configuration and programming
NetQ, What Just Happened Fabric validation and telemetry; per-packet drop reasons on the switch Covered in Article 6
Spectrum-X Photonics Spectrum-6 switches with co-packaged optics (SN6800, SN6810) The next hardware generation; same platform idea

NVIDIA describes the SuperNIC as a class of network accelerator for AI clouds built on BlueField-3, with high-speed packet reordering when paired with an NVIDIA switch and congestion control driven by real-time telemetry. It uses less compute than a DPU because its job is narrower: move GPU traffic fast and fairly, rather than run infrastructure services. The BlueField-3 DPU, with the same silicon, is the choice when you need those services on the card, typically on the front-end and storage side.

Who does what

An analogy helps me here. A courier splits one big shipment across several vans so no single road jams, and it only works because the receiving warehouse reassembles the boxes in order before anyone opens them. The switch is the dispatcher picking roads; the SuperNIC is the warehouse. Take away either and the shipment arrives late or in pieces.

The sender SuperNIC meters each flow; the first leaf sprays the flow’s packets across three spines by picking the least-loaded uplink per packet; the last leaf marks ECN and pauses with PFC; the receiver SuperNIC puts the packets back in order and places them in GPU memory; congestion signals and telemetry flow back so the sender cuts that flow’s rate

Job Switch (Spectrum-4) SuperNIC (BlueField-3, ConnectX-8)
Load balancing Adaptive routing: picks the less-loaded path per packet, from queue occupancy and port utilization Accepts packets out of order and restores order
Data placement Places the reordered data in host or GPU memory; the application never sees reordering
Congestion detection ECN marking, telemetry on queues Receives the signals
Congestion reaction PFC pause as the lossless backstop Per-flow rate control on its congestion-control engine
Multi-tenancy Isolation features, EVPN (Article 5) Telemetry-based congestion control, which NVIDIA describes as providing noise isolation between tenants

Key point. Adaptive routing and reordering are one feature split across two devices. Either one alone makes things worse: spraying without reordering causes retransmissions; reordering without spraying does nothing.

Why per-packet routing needs a NIC that reorders

Part 1 showed why per-flow ECMP fails: a few large flows collide on the same uplinks. The fix that removes the problem entirely is to stop routing per flow and route per packet, so every flow uses every uplink.

Cumulus Linux implements that on Spectrum-4: the switch forwards each packet to the less-loaded path, deciding from queue occupancy and port utilization, with a fixed path-selection interval of one microsecond. It can spread traffic across all members of an ECMP group, equally or by weight, and split a flow whose rate exceeds one uplink.

The cost is ordering. Packets of one flow now take different paths with different queue depths, so they arrive out of order. RDMA over standard RoCE expects order: many adapters treat a gap in sequence numbers as loss and go back to resend everything after it. Spraying into such a NIC turns a balancing feature into a retransmission storm. NVIDIA’s answer, in its own words, is that the BlueField-3 SuperNIC makes sure to reorder the packets and place them in host memory, leaving the reordering invisible to the application. That pairing is the core of Spectrum-X.

It’s the same principle as Ultra Ethernet: once the receiver can place packets as they arrive, the fabric is free to spray. Spectrum-X does it within NVIDIA’s own switch and NIC pairing; Ultra Ethernet standardizes it across vendors.

Congestion control: switch signals, NIC decisions

Spraying evens out the uplinks, but it can’t fix convergence: when many GPUs send to one, every path ends at the same port. That needs the senders to slow down, and Spectrum-X splits that job too:

  • The switch detects congestion where it happens: ECN marks on the RoCE queue, telemetry on queue depth and port utilization, and PFC as the lossless backstop if a queue keeps growing.
  • The SuperNIC decides each flow’s rate. On BlueField-3, congestion control runs on the NIC’s DPA-based engine, per flow, from hardware congestion events.

DOCA exposes that engine in two ways, and the distinction is exam-relevant:

Mode What runs Notes
Native algorithms DCQCN (ECN-based, for interoperability) or ZTR-RTT (adds hardware-timestamped RTT, for Zero Touch RoCE deployments without special switch configuration) Configured per port
Programmable congestion control (DOCA PCC) A congestion-control algorithm written with DOCA, executed on the same engine DOCA 2.7 shipped PCC tuned for Spectrum-X 1.0.1, the reference architecture used in NVIDIA’s Israel-1 system

The two are mutually exclusive, selected by a firmware setting (USER_PROGRAMMABLE_CC). NVIDIA’s Spectrum-X materials describe the combination as telemetry-based congestion control: switches supply real-time congestion information and the SuperNICs adjust each flow’s sending rate from it.

The switch side, in NVUE

On the switches, Spectrum-X is configured with Cumulus Linux’s NVUE. Two features matter here; the details of each get their own article later. Illustrative, from the Cumulus Linux 5.15 documentation; validate against your release.

Lossless RoCE is one command, and lossless is the default mode:

nv set qos roce              # or: nv set qos roce mode lossless
nv config apply
nv show qos roce             # mode, PFC, ECN, trust and the mappings
nv show interface swp1 qos roce counters    # RoCE and CNP traffic, PFC pauses, ECN marks, buffer use

What the lossless profile sets up:

Traffic DSCP Switch priority Traffic class Treatment
RoCE 26 3 3 PFC enabled, ECN marking, dedicated RoCE buffer pool, DWRR 50%
CNP 48 6 6 Strict priority
Everything else others 0 0 Lossy, DWRR 50%

The documentation’s example output shows ECN marking between about 146 KB and 1.43 MB on the RoCE class. Other modes exist: lossy (ECN without PFC), lossless-single-ipool, and lossy-multi-tc, which enables packet trimming, the switch-side feature Ultra Ethernet relies on.

Adaptive routing is enabled globally and on the ECMP member ports:

nv set router adaptive-routing state enabled
nv set interface swp51 router adaptive-routing state enabled     # every uplink in the same ECMP group
nv set interface swp52 router adaptive-routing state enabled
nv config apply                                                  # reloads switchd (5.13 and later)
nv show router adaptive-routing
nv show interface swp51 router adaptive-routing

The limits are classic exam material:

  • Not on layer 3 subinterfaces, SVIs, bonds or bond members. Uplinks must be plain routed ports.
  • No resilient hashing on adaptive-routing ports; the switch chooses per packet instead.
  • Spectrum-4 at 400G and 200G only; the documentation says Spectrum-4 doesn’t support adaptive routing on 800G links.
  • Supported traffic: RoCEv2 unicast, VXLAN-encapsulated RoCE, layer 3 interfaces and next hops in the default VRF.
  • Same ports, same routes: traffic meant for adaptive routing must use ports where it’s enabled; mixing with plain ECMP ports causes unexpected hashing.
  • Syntax changed: up to Cumulus Linux 5.14 the commands use enable on / enable off instead of state enabled. Up to 5.12, applying it restarts switchd; from 5.13 it reloads it.
  • Optional link-utilization threshold (default 70%) is only available with the custom profile; NVIDIA asks you to involve support for profile changes.

Practising it

The blueprint names DSX Air, NVIDIA’s simulation platform, and it’s the right place to practise NVUE: building a leaf-spine, applying the RoCE profile, enabling adaptive routing and reading the show commands. Know what simulation can’t show: the simulated switches run Cumulus VX, so you’ll practise configuration and verification, not the data-plane behaviour of PFC, ECN or packet spraying under load. For that you need real Spectrum hardware and SuperNICs.

What NVIDIA claims, and how to read it

NVIDIA says Spectrum-X boosts network performance by 1.6× for large-scale AI compared with off-the-shelf Ethernet, citing its Israel-1 system. Treat it as a vendor figure for a specific comparison: the gain comes from replacing per-flow ECMP and NIC-only congestion control, and your baseline determines your result. The architecture is what to learn; the multiplier is what to test.

Self-check

1. Why does Spectrum-X adaptive routing require SuperNICs at the endpoints?

Because the switch sprays a flow’s packets across paths per packet, so they arrive out of order. The SuperNIC reorders them and places the data in memory; a standard RoCE NIC would treat the gaps as loss and resend.

2. An engineer enables adaptive routing on a bond of two uplinks. What happens?

It isn’t supported: Cumulus Linux doesn’t support adaptive routing on bonds or bond members (nor on subinterfaces or SVIs). The uplinks must be individual routed ports in the same ECMP group.

3. What does `nv set qos roce` map DSCP 26 and DSCP 48 to, and which one gets PFC?

DSCP 26 to switch priority 3, traffic class 3 (RoCE, PFC enabled, ECN marking); DSCP 48 to switch priority 6, traffic class 6 (CNP, strict priority). PFC is enabled on priority 3 only.

4. Where does per-flow congestion control run in Spectrum-X, and what are the two ways to configure it?

On the SuperNIC’s DPA-based congestion-control engine. Either native algorithms (DCQCN or ZTR-RTT) or a programmable algorithm written with DOCA PCC; they’re mutually exclusive, selected by firmware setting.

5. A Spectrum-4 fabric runs 800G uplinks. Can it use adaptive routing on them?

Not according to the Cumulus Linux 5.15 documentation: Spectrum-4 supports adaptive routing at 400G and 200G, not on 800G links.

What’s next

Article 4 goes inside the RoCE profile: classification, PFC and headroom, ECN thresholds and the NIC’s reaction, and how to check that every hop agrees.

References

This guide follows NVIDIA’s public blueprint and documentation; it contains no exam content. Commands are illustrative and change between releases: validate them against the documentation for your Cumulus Linux and DOCA versions. NVIDIA, Spectrum, Spectrum-X, BlueField, ConnectX, DOCA, Cumulus and NetQ are trademarks of NVIDIA Corporation.