On a Spectrum switch, lossless RoCE is one command: nv set qos roce. That’s the good news, and it’s also the trap. The command configures one switch. RoCE works only when the sending NIC, every switch on the path and the receiving NIC all agree on the same handful of numbers, and nothing checks that for you. Get one hop wrong and the fabric still passes traffic, just not the way you think it does.
So this article doesn’t start from the feature list. It follows one packet: from the sending SuperNIC, through a leaf, a spine and another leaf, to the receiver, and then follows the congestion signal back. At every hop the question is the same: what does this device believe about the packet, and does that match its neighbours?
This is part 4 of the NCP-AIN study series. Part 3 laid out who does what in Spectrum-X; this one goes inside the RoCE profile. If ECN, DCQCN and PFC are new to you, read lossless RoCEv2, explained first. It builds the theory with a motorway analogy, and I won’t repeat it here.
In short
- Three numbers make the contract. RoCE data uses DSCP 26 and priority 3, CNPs use DSCP 48 and priority 6, and PFC runs on priority 3 only. NIC and switch defaults happen to line up on all three, but you should check that they really do.
- ECN does the work; PFC is the backstop. With
nv set qos rocethe RoCE queue starts marking at 150,000 bytes and marks everything by 1,500,000. PFC pauses priority 3 one hop upstream only if the queue keeps growing anyway. - Headroom comes from cable length. Once a switch sends a pause, bytes already on the wire still arrive. Cumulus Linux sizes that reserve from the cable length, and NVIDIA recommends leaving the thresholds alone.
- Counters tell you which hop disagrees. PFC pauses with no ECN marks, RoCE counted as non-RoCE traffic, or watchdog deadlocks each point to a different mistake.
The contract: three numbers
Before following the packet, here’s what every device must agree on. Keep this table in mind; the rest of the article is just checking it hop by hop.
| What | Value with the Cumulus RoCE profile | Why it matters |
|---|---|---|
| RoCE data marking | DSCP 26, switch priority 3, traffic class 3 | Puts RoCE in the lossless queue with PFC and ECN |
| CNP marking | DSCP 48, switch priority 6, traffic class 6 (strict priority) | Gets the “slow down” message past the congestion it reports |
| PFC | Enabled on priority 3 only | Pauses RoCE without pausing anything else |
There’s something I find reassuring in this. The defaults line up without anyone planning it per site:
- Switch trusting DSCP. Cumulus Linux maps DSCP to switch priority in blocks of eight (0–7 to 0, 8–15 to 1, and so on). DSCP 26 lands on priority 3 and DSCP 48 on priority 6.
- NIC trusting DSCP. NVIDIA’s
mlnx_qostool uses the same blocks-of-eight default. - NIC-generated CNPs. NVIDIA’s DCQCN parameters default to DSCP 48 and 802.1p priority 6.
The defaults agree, but only if both sides are actually using them. The default trust state on the NIC is PCP, not DSCP, so a host that was never configured doesn’t hold up its side of the contract.
Follow one packet
The commands in this section are illustrative, from the Cumulus Linux 5.15 and DOCA documentation. Interface and device names are examples; validate everything against your releases in a lab.
Step 1: the sending host decides the marking
The switch can only classify what the host marks, so start at the NIC. Three things have to be true on the sender.
It marks RoCE with DSCP 26. In RoCEv2 the InfiniBand traffic class becomes the IP header’s ToS byte: six bits of DSCP followed by two bits of ECN. So the value you configure is the DSCP shifted left by two, plus the ECN bits. 26 × 4 = 104, and with ECT(0) in the low bits that’s 106. That’s why guides write “106” when they mean “DSCP 26”. When you see an unfamiliar number, divide by four and drop the remainder to recover the DSCP.
Where the value is set depends on who opens the connection:
# NCCL: traffic class for its RDMA connections (NCCL's default is 0 unless set)
export NCCL_IB_TC=106
# Applications using RDMA-CM: a per-port default ToS, through the kernel's configfs
mkdir /sys/kernel/config/rdma_cm/mlx5_0
echo 106 > /sys/kernel/config/rdma_cm/mlx5_0/ports/1/default_roce_tos
The kernel documentation notes that an application calling rdma_set_option overrides the configfs default. Configfs and sysfs settings also don’t survive a reboot, so production hosts set them from a startup script or the vendor’s tooling.
It trusts DSCP and runs PFC on priority 3. On NVIDIA NICs that’s mlnx_qos:
mlnx_qos -i eth0 --trust=dscp # default trust is PCP
mlnx_qos -i eth0 --pfc=0,0,0,1,0,0,0,0 # one flag per priority 0..7: PFC on 3 only
mlnx_qos -i eth0 # show trust, PFC and the dscp2prio map
NVIDIA’s documentation adds a detail that’s easy to miss: set the trust state before enabling SR-IOV, so the virtual functions inherit it.
It reacts to congestion. ECN on the host has two roles, enabled per priority in sysfs. The reaction point (roce_rp) slows down when CNPs arrive. The notification point (roce_np) sends CNPs when marked packets arrive. Every host plays both roles, because every host sends and receives:
cat /sys/class/net/eth0/ecn/roce_rp/enable/3 # 1 = this host slows down on CNPs for priority 3
cat /sys/class/net/eth0/ecn/roce_np/enable/3 # 1 = this host sends CNPs for marked priority-3 packets
Step 2: the first leaf classifies it
The packet arrives at Leaf A. With the RoCE profile applied, the switch trusts DSCP (the profile shows trust as PCP and DSCP), maps 26 to switch priority 3, and puts the packet in traffic class 3 with its own lossless buffer pool:
nv set qos roce # lossless is the default mode
nv config apply
nv show qos roce # mode, PFC, ECN thresholds, trust, mappings and pools
nv show interface swp1 qos roce status # the same, as programmed on one port
Two details in that output are worth knowing:
-
The buffer is partitioned. RoCE gets a dedicated ingress pool, lossy traffic gets another, and the two don’t compete. You can change the split. NVIDIA’s example gives lossless traffic 60%:
nv set qos traffic-pool default-lossy memory-percent 40 nv set qos traffic-pool roce-lossless memory-percent 60 -
Headroom is reserved per port. That’s the subject of step 7.
Step 3: the spine, and why every spine matters
The spine does exactly what Leaf A did, and that’s the point: it has to. In a classic fabric with per-flow ECMP, a spine with the wrong QoS map hurts the flows that happen to hash onto it. In a Spectrum-X fabric with adaptive routing (Part 3), every flow sprays across every spine. One spine with a different map now touches every flow in the fabric.
NVIDIA’s documentation also warns that a RoCE configuration generated for one Spectrum ASIC generation doesn’t carry over to another. So if you mix switch generations, configure each one with NVUE natively; don’t copy files between them.
Step 4: the last leaf is where congestion happens
Leaf B is where many senders converge on one receiver port, so this is where the traffic-class-3 egress queue builds. The RoCE profile sets ECN marking on that queue:
- Below 150,000 bytes, nothing is marked.
- Between 150,000 and 1,500,000 bytes, packets are marked Congestion Experienced (CE) with a probability that rises with queue depth.
- At 1,500,000 bytes and above, every ECN-capable packet is marked.
Here’s a small exam-style trap. nv show qos roce prints those thresholds as about 146.48 KB and 1.43 MB: the same bytes, in units of 1,024. On a port, the documentation shows the minimum as about 156 KB, because Spectrum-4 programs it in hardware steps and 150,000 bytes becomes 159,744. Three numbers, one setting.
Marking a packet doesn’t slow anything down yet. The mark has to reach the receiver and come back as a CNP.
Step 5: the receiver sends a CNP
The receiving SuperNIC sees CE on a RoCE packet. As the notification point, it sends a Congestion Notification Packet to the sender, marked by default with DSCP 48 and priority 6. NVIDIA’s DCQCN defaults also rate-limit CNPs: at least 4 µs between CNPs from the port, so a congested receiver doesn’t flood the fabric with them.
On the way back, every switch puts the CNP in traffic class 6, which is strict priority. If CNPs waited in the same queue as the data that triggered them, the warning would arrive late, exactly when it’s needed most.
Step 6: the sender cuts its rate
The sending SuperNIC, the reaction point, lowers that flow’s rate. With native DCQCN, NVIDIA’s default parameters show its shape:
- The cut. A single decrease can cut the rate by up to 50%.
- The recovery. The rate comes back in additive steps of 5 Mb/s, then hyper steps of 50 Mb/s, with increase events every 300 µs.
On Spectrum-X, the algorithm may instead be NVIDIA’s programmable congestion control running on the same engine (Part 3). The idea is the same: the switch marks, the receiver reports, the sender slows.
If that loop works, the queue on Leaf B drains and steps 7 and 8 never happen.
Step 7: PFC, when ECN is too late
Sometimes the loop isn’t fast enough, for example when an incast arrives faster than CNPs can make the round trip. When Leaf B’s ingress buffer for priority 3 crosses its xoff threshold, it sends a PFC pause frame for priority 3 to its upstream neighbour, the spine. The spine stops sending priority-3 traffic on that link. Everything else on the link, including the priority-6 CNPs, keeps flowing.
A pause takes time to work. When Leaf B sends it, the spine is still transmitting, and bytes are already on the fibre. Those bytes need somewhere to land, and that reserve is the headroom. A back-of-envelope estimate shows why the cable length matters:
| Contribution, 400 Gb/s link, 100 m of fibre | Rough size |
|---|---|
| 400 Gb/s is 50 bytes per nanosecond; fibre is about 5 ns per metre, so 500 ns each way | |
| Round trip: the pause travels up, the last bytes travel down (1,000 ns × 50 B/ns) | ~50 KB |
| A maximum-size frame already being sent on each side | ~2 × MTU |
| The upstream switch’s time to act on the pause | a little more |
So the reserve is roughly 70 KB per lossless priority, per port, for 100 m, and it grows with both speed and distance. That’s my approximation to show the shape, not NVIDIA’s formula. Cumulus Linux derives the real values from the cable length you give it:
- The PFC default is 10 m, and the RoCE profile’s example output shows 100 m.
- NVIDIA recommends leaving
port-buffer,xoff-thresholdandxon-thresholdto the automatic calculation unless you really understand lossless buffering.
If you underestimate the cable, the headroom overflows and the “lossless” class drops packets.
Step 8: the PFC watchdog, for when the backstop jams
PFC has a failure mode that ECN doesn’t. If a receiver stops draining (a stuck host, a failing NIC), it can keep pausing its upstream port indefinitely. The pause spreads hop by hop: a PFC storm, or worse, a deadlock. The PFC watchdog breaks it:
nv set interface swp1 qos pfc-watchdog # enable on a port
nv set qos pfc-watchdog polling-interval 200 # default 100 ms
nv set qos pfc-watchdog robustness 5 # intervals before acting; default 3
nv config apply
nv show interface swp1 qos pfc-watchdog # per traffic class: OK or DEADLOCK, and a count
nv action clear interface swp1 qos pfc-watchdog deadlock-count
When a lossless queue stays paused for longer than the configured number of polling intervals, the watchdog does two things. It stops honouring received pause frames for that traffic class, and it drops new packets for that egress queue. As soon as a polling interval passes with no pause frames, it restores PFC.
In other words, it gives up “lossless” on one queue to save the fabric. It works only on lossless queues and physical swp ports. You can’t enable it on a bond, but you can on its members.
Lossless or lossy
nv set qos roce isn’t the only option. The profile has four modes, and knowing why each exists matters more than memorising them:
| Mode | What it does | When you’d consider it |
|---|---|---|
lossless (default) |
PFC on priority 3, ECN, separate RoCE and lossy pools | The classic RoCE design: ECN first, PFC as the backstop |
lossless-single-ipool |
Lossless, but one shared ingress pool | When a single pool suits the traffic mix better |
lossy |
ECN without PFC | When endpoints recover from loss well and you’d rather drop than pause |
lossy-multi-tc |
No PFC; ECN on traffic classes 1–3 with much tighter thresholds (about 163 KB to 234 KB at 5% probability in the documentation’s example) and packet trimming | Endpoints designed for loss, the direction Ultra Ethernet takes |
One gotcha from the documentation: running nv set qos roce without a mode doesn’t change an existing lossy configuration. To go back to lossless, set mode lossless explicitly.
Reading the counters
The per-port RoCE counters are where the contract shows up in practice:
nv show interface swp1 qos roce counters
nv action clear interface swp1 qos roce counters
They’re grouped as:
- Receive side: non-RoCE traffic, PFC frames received, RoCE traffic.
- Transmit side: CNPs, ECN-marked packets, PFC frames sent, RoCE traffic.
- Each group: byte and packet counts, discards, and current and peak buffer usage.
Here’s how I’d read the common patterns. This is reasoning from the mechanism above, not a list from NVIDIA:
| What you see | What it most likely means | Where to look |
|---|---|---|
| ECN marks and CNPs rising, few PFC pauses | The system working as designed: ECN is controlling the queues | Nothing to fix; check the job is still fast |
| PFC pauses rising, ECN marks near zero | ECN isn’t acting: the traffic isn’t ECN-capable, or ECN is off on that class | Host roce_rp/roce_np enable flags; nv show qos roce congestion control |
| Your RDMA traffic shows up in the non-RoCE counters | Classification mismatch: the host isn’t marking DSCP 26 | NCCL_IB_TC, default_roce_tos, host trust state |
| Discards on the RoCE pool despite PFC | Headroom too small, or the neighbour ignores pauses | Cable length; PFC enabled on the neighbour, NIC or switch |
| Watchdog deadlock count above zero | A pause storm: something downstream stopped draining | Follow the pauses downstream to the host that stopped receiving |
Watch the second row in particular. When PFC is working hard, it’s tempting to conclude the fabric is protecting itself nicely. It usually means ECN never got a chance to work.
The consistency checklist
If I had five minutes on a fabric I didn’t build, this is what I’d check, end to end:
| Item | Host | Switch |
|---|---|---|
| Trust | mlnx_qos -i <if> shows trust dscp |
nv show qos roce trust includes DSCP |
| RoCE marking | NCCL or RDMA-CM ToS 106 (DSCP 26) | DSCP 26 maps to switch priority 3, traffic class 3 |
| PFC | Enabled on priority 3 only | Priority 3, receive and transmit enabled |
| ECN | roce_rp and roce_np enabled for priority 3 |
ECN on traffic class 3 |
| CNP | DSCP 48, priority 6 (defaults) | DSCP 48 to traffic class 6, strict priority |
| Every hop | Every server, every NIC port | Every leaf and every spine, including the ones you didn’t touch |
Practising it
In NVIDIA’s DSX Air you can rehearse most of the switch side:
- Build a two-leaf, two-spine topology and apply
nv set qos roce. Readnv show qos roceuntil you can find the trust state, both mappings, the PFC priority and the ECN thresholds without searching. - Switch to
mode lossy-multi-tcand compare the ECN thresholds and PFC state with lossless. Then switch back, and notice that you needmode losslessexplicitly. - Change the traffic-pool split, enable the PFC watchdog on the leaf downlinks, and read both back.
- Break the contract on purpose: remap DSCP 26 on one spine only, then find the difference by comparing
nv show qos roceacross switches. That’s the skill the checklist above is for.
The simulated switches run Cumulus VX. You’ll see the configuration and its show output, not real ECN marks or PFC pauses under load; those need Spectrum hardware and real NICs.
Self-check
1. A guide sets the RoCE traffic class to 106. What DSCP is that, and why not just write 26?
DSCP 26. The traffic class fills the whole ToS byte: DSCP in the top six bits, ECN in the bottom two. 26 × 4 = 104, plus 2 for ECT(0) = 106. The configuration expects the byte, not the DSCP.
2. Why do CNPs get a strict-priority class of their own?
Because they report congestion. If they waited in the congested RoCE queue, the “slow down” signal would be delayed by the very congestion it describes. In traffic class 6 with strict priority, they go first.
3. A leaf shows many PFC pause frames sent and almost no ECN-marked packets. What do you suspect?
That ECN isn’t controlling the queue: the hosts aren’t sending ECN-capable traffic (ECN disabled on the NICs), or ECN marking is off on the RoCE traffic class. PFC should be the rare backstop, not the main mechanism.
4. What does the PFC watchdog do when it detects a storm, and how does it recover?
After the queue stays paused for the configured number of polling intervals (default 3 × 100 ms), it ignores received pause frames for that traffic class and drops packets for that egress queue. When a polling interval passes with no pause frames, it restores PFC.
5. Why does PFC need headroom, and what does Cumulus Linux size it from?
Because after a switch sends a pause, the bytes already in flight, roughly a cable round trip at line rate plus a frame on each side, still arrive and must be buffered. Cumulus Linux derives it from the cable length (10 m by default for PFC), and NVIDIA recommends leaving the derived thresholds alone.
6. Why is a QoS mismatch on one spine worse in a Spectrum-X fabric than in a classic ECMP fabric?
Because adaptive routing sprays every flow’s packets across all spines. With per-flow ECMP, only the flows hashed to that spine are hurt; with per-packet spraying, every flow crosses it.
What’s next
Article 5 moves up the stack to BGP-EVPN multi-tenancy on Spectrum-X: VRFs, L3VNIs, and where EVPN belongs in an AI factory.
References
- NVIDIA, NVIDIA-Certified Professional: AI Networking: blueprint topics for the Spectrum domain, including RoCE, QoS with ECN and PFC.
- NVIDIA, Cumulus Linux 5.15: RDMA over Converged Ethernet (RoCE): the four modes, the lossless mappings, ECN thresholds and their hardware programming, traffic pools, counters, ASIC-specific configuration.
- NVIDIA, Cumulus Linux 5.15: Quality of Service: trust and DSCP mapping defaults, PFC profiles, cable length and thresholds, the PFC watchdog, ECN congestion-control defaults.
- NVIDIA DOCA, Ethernet QoS (
mlnx_qostrust, PFC and dscp2prio), Explicit Congestion Notification (roce_rpandroce_np) and DCQCN parameters (CNP DSCP and priority, rate decrease and increase defaults). - NVIDIA, NCCL environment variables:
NCCL_IB_TC,NCCL_IB_GID_INDEX. - Linux kernel, configfs-rdma_cm:
default_roce_tosand its override byrdma_set_option. - IETF, RFC 3168: the ECN field and codepoints. IEEE 802.1Qbb: priority-based flow control.
- Related on this site: lossless RoCEv2, explained, the DCQCN research note and Ultra Ethernet in plain English.
This guide follows NVIDIA’s public blueprint and documentation; it contains no exam content. Commands are illustrative and change between releases: validate them against the documentation for your Cumulus Linux, DOCA and NCCL versions. NVIDIA, Spectrum, Spectrum-X, BlueField, ConnectX, DOCA, Cumulus and NCCL are trademarks of NVIDIA Corporation.