One link fails, every job slows: what ECMP hashing really does

A spine uplink fails on one leaf. You expect the jobs using that link to stumble for a moment. Instead, every job on the leaf slows down. Nothing is misconfigured. The switch is doing exactly what plain ECMP hashing was designed to do.

The problem: the choice depends on the count

A switch picks an ECMP member by hashing a packet’s header fields into one large number and taking the remainder of a division by the number of members:

Hash of one flow's fields: 981,242,341
8 uplinks: 981,242,341 mod 8 = 5   -> uplink 5
7 uplinks: 981,242,341 mod 7 = 2   -> uplink 2

Same flow, different uplink, purely because the member count changed. When one uplink of eight fails, about 87.5% of flows move, although only 12.5% were on the failed link.

Why moving flows is not free here

In a web data center, moving flows costs almost nothing. In an AI back-end it costs three things:

• Reordering. Packets in flight on the old path arrive after packets on the new one. An RDMA adapter that does not tolerate reordering treats it like loss and retransmits.
• New collisions. AI traffic is a few huge flows. Every reshuffle rolls the dice again on which flows share a link, at the exact moment capacity just dropped.
• Synchronization. A collective finishes when its slowest flow finishes. One unlucky reshuffle slows the whole step.

The options and what each costs

Technique	What it does	What it costs
Resilient hashing	Hash selects one of many fixed buckets; each bucket points to a link. A failure repoints only the dead link’s buckets	Slightly uneven shares; more buckets make them finer
Symmetric hashing	Both directions of a flow take the same path	Only matters with stateful middleboxes, typically on the front-end
Weighted ECMP	Buckets in proportion to capacity, for example after a spine loses half its links to a leaf	Weights must be configured or learned from routing (BGP link bandwidth)

What I’d do

Enable resilient hashing on every ECMP group that carries long-lived flows, which in an AI fabric means all of them. Use weighted ECMP, or withdraw prefixes below a minimum link count, so that a partially failed spine does not become a hot spot. Then test it: fail an uplink under an nccl-tests run and watch what happens to jobs that were not using it.

What would change my mind

If the adapters spray every packet across all paths anyway, as in per-packet adaptive routing or the Ultra Ethernet transport, flow placement matters much less, and hashing details fade into the background. Until your whole fabric is there, the hash is still deciding where your elephants walk.

Behavior and options vary by platform and release; validate in a lab.