Ultra Ethernet in plain English

RoCEv2 carries InfiniBand’s transport over IP, and with it InfiniBand’s assumptions: packets arrive in order, each connection uses one path, and a loss is repaired by resending everything after it. Much of AI Ethernet engineering has been working around those assumptions. The Ultra Ethernet Consortium took the other route and designed a transport that assumes the opposite. Its 1.0 specification was published in June 2025.

Moving house

RoCE is one removal lorry that must arrive with boxes in packing order, and if a box is lost, everything packed after it is sent again. Ultra Ethernet is a fleet of vans taking whatever roads are clear. Every box is labelled with the room it belongs in, so it goes straight to its place in any order, and only a missing box is sent again.

What changes

• Delivery modes per operation: reliable ordered, reliable unordered (each packet placed directly into memory as it arrives), reliable unordered for operations safe to repeat, and best effort.
• Selective retransmission: only the missing packets are resent.
• Spraying chosen by the sender: each packet carries an entropy value the sender varies, so ordinary ECMP spreads one flow over many paths. Paths that report congestion get avoided.
• Two congestion control algorithms: a sender-driven window using ECN and delay (NSCC), or receiver-granted credits (RCCC).
• libfabric instead of verbs as the application interface.

Why receiver credits matter

With sender-driven control, eight senders discover incast only after the receiver’s queue has grown and the warnings have come back. With receiver credits, the receiver, the only party that knows how many senders are converging, hands out credits that add up to its port rate. The queue never forms.

What it asks of your switches

Surprisingly little: ECN marking, and ideally packet trimming. Link-level retry and credit-based flow control are optional and need support on both ends of a link. In a mixed fabric, keep RoCE in its lossless class with PFC and DCQCN, and Ultra Ethernet in a separate lossy class with ECN and trimming.

The design takeaway

Ultra Ethernet moves the intelligence from the fabric to the endpoints. Build today’s fabric so it can carry it, routed, ECN everywhere, classes ready, switches that can trim, and let the adapters decide when to change. Early products are arriving; treat roadmaps as roadmaps until you have tested them.