Research note: DCQCN, the paper behind RoCE congestion control

Paper: Yibo Zhu et al., “Congestion Control for Large-Scale RDMA Deployments,” ACM SIGCOMM 2015 (Microsoft and Mellanox).

The problem they had

Microsoft wanted RDMA in its data centers to cut CPU use and latency. RoCE needed a lossless network, so they relied on PFC. At scale, PFC alone caused trouble: it pauses a whole priority, not a flow, so flows that were not causing congestion got paused too (victim flows), throughput became unfair between senders, and pauses spread backward through the network. They needed per-flow congestion control that worked over routed IP networks, with switches they already had.

The idea

Combine what switches already did, ECN marking with a RED-like curve, with rate control in the adapters, borrowing ideas from QCN, an earlier Layer 2 scheme that could not cross IP subnets:

1. Congestion point (switch): marks packets with ECN as its queue grows, between a minimum and a maximum threshold.
2. Notification point (receiving NIC): on seeing marked packets for a flow, sends a congestion notification packet (CNP) back to the sender, rate-limited per flow.
3. Reaction point (sending NIC): cuts that flow’s rate in proportion to how much congestion it has been seeing, then recovers in stages, quickly at first, then more aggressively if the path stays clear.

The key design choice: everything per-flow happens in the NICs, so switches only need standard ECN.

What they found that still matters

• ECN must act before PFC. Marking thresholds have to be set so senders slow down well before the queue reaches the pause threshold; PFC stays as a backstop against loss, not the main control.
• Parameters matter. The rate-reduction and recovery settings trade fairness against how quickly senders reclaim bandwidth, and they must be consistent across the fabric.

What it means for your design today

DCQCN is still the default congestion control on RoCE adapters. The paper is the source of rules you configure every day: a separate class for CNPs, ECN below PFC, consistent settings on hosts and switches. It is also the reason PFC storms and victim flows are now well understood.

What came after

Google’s TIMELY, published at the same conference, used round-trip time instead of ECN. Later work such as HPCC used in-band telemetry for precise feedback, and the Ultra Ethernet transport now offers both sender-driven and receiver-credit control. The problem DCQCN named, per-flow control on top of a lossless safety net, is still the center of the field.

This is a summary in my own words; read the paper for the algorithm’s details and evaluation.