Lossless RoCE in one picture: ECN, DCQCN and PFC

RDMA moves data directly between the memories of two servers, with the transport running in the network adapter. It is fast, and it hates packet loss: a single drop can stall a connection, and in a training job one stalled connection stalls every GPU. Lossless Ethernet exists to stop congestion from causing drops.

The motorway

Picture a motorway approaching a busy junction:

• ECN is the overhead sign warning of congestion ahead.
• DCQCN is the drivers easing off when they see it.
• PFC is the barrier on the slip road that stops cars completely when the junction is full.

A well-run road almost never closes the barrier, because the warnings work.

The sequence

1. A switch queue for the RoCE class starts to grow.
2. Past a minimum threshold, the switch marks packets with ECN instead of dropping them.
3. The receiving adapter sees the marks and sends a congestion notification packet (CNP) back to the sender.
4. The sending adapter slows down that flow (DCQCN), then recovers gradually.
5. Only if the queue still reaches the pause threshold does the switch send a PFC pause for that priority.

The tuning rule: ECN marking must start well below the PFC threshold. If the barrier comes down first, congestion spreads backward through the fabric and innocent flows suffer.

Consistency beats cleverness

The common convention: RoCE on DSCP 26 and priority 3, CNPs on DSCP 48 and priority 6 with strict scheduling. The exact values matter less than using the same ones on every switch and every host. One server marking differently is a server whose traffic is not protected.

On Cumulus Linux, one command applies the full lossless profile:

nv set qos roce
nv config apply
nv show interface swp1 qos roce counters

Reading the counters

What you see	What it means
ECN marks and CNPs rising and falling with load	Healthy congestion control
Many pauses, few ECN marks	Thresholds wrong or hosts not reacting
Pauses spreading from one port	Incast or a misbehaving host downstream
Lossless discards above zero	Headroom or classification problem: fix now

PFC is a safety net, not a congestion control mechanism. If you see it working hard, something upstream isn’t.

Configurations are illustrative; validate them in a lab against the documentation for your software release.