The six places an AI fabric congests, and which fix works where

When a training job slows down, the first instinct is to add capacity. Sometimes that works. Often it does nothing, because the congestion is in a place that no amount of spine capacity can reach. Before choosing a fix, find the queue.

Two kinds of traffic jam

Picture two jams. In the first, a match has just ended and fifty thousand cars want the one exit from the stadium car park. No new road elsewhere in the city helps; the only cure is letting cars out at the rate the exit can take. In the second, a river has four bridges and every driver’s navigation app picks the first one. Three bridges stand empty. Slowing the drivers down would be a waste: they need a better choice of bridge.

AI fabrics have both:

• Convergence. Many senders, one port, and every path ends there. Incast is the classic form. Only slowing the senders (or pausing them) helps.
• Spreading. Several equal paths, but too much traffic lands on one of them. A better choice of path fixes it; slowing senders wastes capacity you already paid for.

The map

Point	Where	Can path choice help?	Main tools
1	Leaf to server, the last hop	No	ECN and DCQCN, PFC as backstop, traffic patterns without fan-in
2	Inside one leaf	No	Rate control, storage in its own class, generous storage ports
3	Leaf to spine	Yes, locally	Entropy, adaptive routing, spraying, placement
4	Spine to leaf	Only with a view beyond the leaf	Spraying, downstream-aware path choice, placement
5, 6	Spine to and from super-spine	As 3 and 4, one tier up	Same, plus keeping jobs inside a pod

Convergence happens in microseconds

Eight servers send to one server at 400 Gb/s each. The receiving port drains 400 Gb/s, so the excess piles up:

Offered: 8 x 400 Gb/s = 3.2 Tb/s; drained 400 Gb/s
Excess: 2.8 Tb/s = 350 GB/s = 0.35 MB every microsecond

With typical RoCE thresholds, ECN marking starts in under half a microsecond. But the sender only learns about it a round trip later, and the queue keeps growing meanwhile. That is why ECN thresholds sit low and why PFC with its headroom exists at all. Adding spines does nothing here: the queue is on the last link, which every path shares.

Point 2 is the same problem inside one switch. Sixteen servers writing checkpoints to a storage node on their own leaf never touch an uplink; adaptive routing has nothing to choose from.

The collision you cannot see from the leaf

Leaf A and Leaf B each send a 400G flow to Leaf D, and both pick Spine 1. From each leaf, nothing is wrong: its uplink carries its own flow at line rate. But Spine 1’s link to Leaf D is offered 800G. With four flows from four leaves over four spines, the chance that no two collide is 4!/4⁴, about 9.4%. More than 90% of the time, some spine-to-leaf link is overloaded even though every leaf balanced perfectly.

Local adaptive routing cannot fix this. Packet spraying (every flow over every spine), downstream-aware path selection, and placement that keeps traffic under the spines can.

Use it when troubleshooting

ECN marks are set at the congested queue, and PFC pauses start there and travel toward the senders. Find where they originate and the location tells you the point: a leaf’s server port means 1 or 2, a leaf uplink means 3, a spine’s downlink means 4.

Locate the queue before choosing the fix. Converging traffic needs slower senders; piled-up traffic needs better spreading.

Thresholds and figures are illustrative; validate behavior in a lab against your platform’s documentation.