Few questions in AI networking start more arguments than this one. Ask a room of engineers whether the GPU back-end should be InfiniBand or Ethernet with RoCEv2, and you’ll hear strong opinions on both sides, usually backed by a benchmark that proves exactly what its author hoped it would.
I’ve sat through a few of those arguments. This article takes the question apart calmly: what is actually the same, what is genuinely different, what each choice costs you, and how to decide from requirements rather than preferences. It’s written for network engineers, and for anyone preparing a design scenario such as the CCDE AI Infrastructure exam, where “InfiniBand or Ethernet?” is decided by the clues in the brief.
In short
- Both run the same RDMA transport and the same software (verbs, NCCL). The difference is the network underneath.
- InfiniBand was designed lossless from the first day: credit-based flow control, a central Subnet Manager, adaptive routing and in-network reduction (SHARP). It works very well out of the box, and in practice it comes from one vendor.
- RoCEv2 runs RDMA over routed Ethernet, made lossless by configuration (PFC, ECN, DCQCN). It brings many vendors, standard tooling and one operating model with the rest of the data center, and it asks more of the design and operations team.
- Today’s largest operators use both. The deciding factors are usually the team, the vendor policy, multi-tenancy and how the fabric must integrate with everything else, not raw performance.
The picture first: two railways
Think of two ways to run high-speed trains.
InfiniBand is a dedicated railway. It was built only for these trains. A central control room knows every track and sets every signal. A train never leaves a station unless the next station has confirmed it has room, so trains never pile up and nothing is ever lost.
RoCEv2 runs the same trains on the motorway network. The roads already exist everywhere, many companies build them, and every engineer knows how to drive on them. To keep the trains safe, you add dedicated lanes, warning signs and barriers. It works very well when the signs and barriers are set up consistently, and badly when they aren’t.
Same trains, same passengers. Two very different ways to keep them moving.
Same transport, two networks
This is the most important fact in the whole comparison. RoCE stands for RDMA over Converged Ethernet, and RoCEv2 literally carries the InfiniBand transport header (the BTH) inside UDP/IP. Queue pairs, memory registration, RDMA writes and reads, sequence numbers and acknowledgements work the same way on both. NCCL, the library GPUs use to communicate, runs on either with no change to the training code.
So the comparison isn’t between two kinds of RDMA. It’s between two networks that carry the same RDMA: one designed for it, one adapted to it.
How InfiniBand works
A central brain: the Subnet Manager
An InfiniBand fabric (a subnet) is managed by a Subnet Manager, software running on a server or a switch: the open-source OpenSM, or NVIDIA’s UFM. It:
- discovers every switch, adapter and link;
- assigns addresses: each adapter port gets a 16-bit Local Identifier (LID), so one subnet can address roughly 48,000 endpoints;
- computes the routes with a routing engine chosen for the topology (fat-tree, up/down, minimum-hop and others) and installs forwarding tables into every switch;
- reacts to failures by recomputing and reprogramming.
Switches themselves run no routing protocol. This makes InfiniBand remarkably simple to bring up: connect the cables, start the Subnet Manager, and the fabric configures itself. It also means the Subnet Manager is a critical component that must be redundant (a standby Subnet Manager takes over if the master fails).
Lossless by design: credits
On every InfiniBand link, the receiver tells the sender how much buffer space it has, in credits. The sender only transmits when it holds credits. A packet is never sent into a buffer that can’t take it, so congestion never causes a drop. Losslessness is a property of the link layer, not a configuration you have to get right.
Each link carries several virtual lanes (VLs) with their own buffers and credits, mapped from a packet’s service level. Traffic classes are separated the same way Ethernet priorities are, but with flow control built in for each.
Features built for collective communication
- Adaptive routing. Switches can send packets of a flow over different paths based on current load, and the adapters put them back in order. This addresses the ECMP collision problem directly.
- In-network reduction (SHARP). NVIDIA’s Quantum switches can compute reductions themselves: each GPU sends its data up a tree of switches once and receives the summed result once, instead of passing partial sums around a ring.
- Partitions. A partition key (P_Key) in every packet isolates groups of endpoints, much like VLANs. The adapter enforces it at the receiving side.
- Congestion control. The specification defines congestion notification, and current adapters implement their own algorithms on top.
Current generations
InfiniBand speeds are named per generation: HDR at 200 Gb/s, NDR at 400 Gb/s (NVIDIA Quantum-2 switches and ConnectX-7 adapters) and XDR at 800 Gb/s (Quantum-X800). In practice, InfiniBand switches today come from one vendor, NVIDIA, which acquired Mellanox in 2020.
How RoCEv2 works
RoCEv2 keeps everything above the transport and replaces the network underneath with standard routed Ethernet. The details are in the lossless RoCEv2 deep dive; in summary:
- Routing is ordinary IP routing, usually eBGP in a leaf-spine fabric, often unnumbered. Every switch participates; there is no central controller to protect.
- Lossless behavior is configured, not inherent: a dedicated traffic class with PFC as the safety net, ECN marking in the switches and DCQCN congestion control in the adapters. Every device on the path must agree on the settings.
- Load balancing starts with ECMP hashing on the UDP source port, which struggles with the few, large flows of training (why). Modern AI Ethernet platforms from several vendors add adaptive routing and packet spraying, usually together with smarter adapters.
- Isolation uses the standard data center toolbox: VRFs, VXLAN EVPN, ACLs.
- Vendors: switch silicon and systems from Broadcom-based platforms (Arista, Cisco, Juniper, others), NVIDIA Spectrum, and more; adapters from several vendors.
Side by side
| InfiniBand | RoCEv2 | |
|---|---|---|
| Transport | InfiniBand (verbs, queue pairs) | The same, carried in UDP/IP |
| Lossless | Built in: credit-based flow control | Configured: PFC, ECN and DCQCN, end to end |
| Control plane | Central Subnet Manager (OpenSM, UFM) | Distributed routing (BGP), every switch |
| Addressing | LIDs inside a subnet; routers between subnets | IP everywhere, routable across the data center |
| Load balancing | Adaptive routing, mature and integrated | ECMP by default; adaptive routing and spraying on AI-focused platforms |
| In-network compute | SHARP reductions in the switch | No standard, widely deployed equivalent today |
| Isolation | Partitions (P_Keys) | VRFs, VXLAN EVPN, ACLs |
| Vendors | In practice one: NVIDIA | Many switch and adapter vendors |
| Tooling and skills | Specialized tools (UFM, ibdiagnet); fewer engineers |
Standard Ethernet tooling, telemetry and automation; large talent pool |
| Integration | A separate island; gateways to reach Ethernet | Same technology as the front-end, storage and WAN |
| Out-of-the-box performance | Excellent with little tuning | Excellent when designed and tuned carefully |
Strengths and weaknesses
InfiniBand
Strengths
- Lossless and predictable by design. No thresholds to tune for the basic case; very consistent tail latency.
- Fast to bring up. The Subnet Manager configures the fabric; validated designs exist for every cluster size.
- Collective-aware features. Mature adaptive routing and SHARP reduce the work and latency of all-reduce.
- Proven at the largest scales of AI and HPC, over many years.
Weaknesses
- One vendor: pricing, lead times and roadmap depend on a single supplier.
- A separate world: its own tools, skills and operations, and a gateway wherever it meets the Ethernet estate.
- Smaller talent pool: fewer engineers have run it in production.
- The Subnet Manager is critical: it must be redundant, monitored and protected.
RoCEv2
Strengths
- Many vendors: competitive pricing, supply options and negotiating power.
- One operating model with the rest of the data center: the same routing, automation, telemetry and security tools.
- Large talent pool: every network team knows Ethernet and BGP.
- Native multi-tenancy and integration: VRFs and EVPN, routable everywhere, no gateways.
- The industry is investing here: adaptive routing, packet spraying, SuperNICs and Ultra Ethernet all build on Ethernet.
Weaknesses
- Losslessness depends on configuration: one inconsistent switch or adapter quietly breaks it.
- More design and tuning: ECN and PFC thresholds, headroom, classification, load balancing.
- PFC side effects: head-of-line blocking and congestion spreading must be designed out and monitored.
- No widely deployed in-network reduction comparable to SHARP today.
What the largest operators actually do
Real deployments show that both work at very large scale:
- Meta described building two clusters of 24,576 GPUs each in 2024: one with a RoCE fabric built on Arista and Meta’s own OCP rack switches, and one with NVIDIA Quantum-2 InfiniBand. Through “careful co-design of the network, software, and model architectures”, Meta reported using both “for large, GenAI workloads (including our ongoing training of Llama 3 on our RoCE cluster) without any network bottlenecks”.
- In a SIGCOMM 2024 paper, Meta explained that it chose RoCEv2 for the majority of its AI training capacity, and described the routing and congestion-control work needed to make it perform: low-entropy traffic and plain ECMP were among the main challenges.
- Many AI clouds and research clusters run InfiniBand, often as part of NVIDIA’s reference architectures, because it delivers predictable performance quickly.
The lesson isn’t that one protocol wins. It’s that both reach excellent performance at scale, by different paths: InfiniBand by design, RoCEv2 by engineering.
Key point. The transport is the same. You’re choosing the network underneath, and the team that will run it.
How to choose
In a design review, and in a design exam, the answer is almost always in the requirements. Cues to look for:
| Requirement cue | Points toward | Why |
|---|---|---|
| “Our team has years of InfiniBand experience” | InfiniBand | Lowest operational risk |
| “Strong Ethernet, BGP and EVPN skills, no InfiniBand” | RoCEv2 | The team can operate and troubleshoot it |
| “Ethernet preferred to align with the estate” or “multi-vendor policy” | RoCEv2 | Policy is a hard constraint |
| “Multi-tenant GPU cloud” | RoCEv2 | VRF and EVPN tenancy, routable everywhere |
| “Fastest path to a working cluster, one validated vendor stack” | InfiniBand | Validated design, little tuning |
| “Collective latency is critical; large all-reduce at scale” | InfiniBand, or RoCEv2 with adaptive routing | SHARP and mature adaptive routing help directly |
| “Lead time or budget pressure on one supplier” | RoCEv2 | Supply options |
Two traps to avoid:
- Deciding on a benchmark. Most published comparisons measure one configuration, often tuned for one side. At job level, both perform within a small margin when designed well; the difference shows up in how much effort it took.
- Ignoring day two. Who will run the fabric at 3 a.m., with which tools, and who can you hire next year? Operations usually costs more than the switches.
What would change my mind
- If Ethernet gains a widely deployed equivalent to in-network reduction, one of InfiniBand’s clearest technical advantages narrows further.
- If Ultra Ethernet adapters and switches ship broadly, Ethernet fabrics will need far less PFC tuning, which removes RoCEv2’s main operational weakness.
- If InfiniBand gained a broad multi-vendor ecosystem, its main business weakness would go away.
Until then: choose the network your team can run excellently, within the constraints the business has set. Both can deliver.
References
- InfiniBand Trade Association: the InfiniBand Architecture Specification, and its RoCEv2 annex.
- Meta Engineering, Building Meta’s GenAI Infrastructure, March 2024.
- Gangidi et al., RDMA over Ethernet for Distributed Training at Meta Scale, ACM SIGCOMM 2024.
- NVIDIA, Scalable Hierarchical Aggregation and Reduction Protocol (SHARP) documentation.
- NVIDIA, UFM documentation: the Subnet Manager.
- IETF, IP over InfiniBand architecture: a readable introduction to LIDs, subnets, partitions and the Subnet Manager.
- NVIDIA, DGX SuperPOD reference architecture (H100), key components: a rail-optimized, balanced full fat-tree InfiniBand compute fabric built on Quantum-2 QM9700 NDR switches.
- Ultra Ethernet Consortium, UEC Specification 1.0, June 2025.
Chapter 12 of Building the AI Factory covers InfiniBand in depth, and chapters 9 to 11 cover RDMA, lossless RoCEv2 and load balancing.
This comparison reflects public documentation and published deployments at the time of writing. Product capabilities change quickly; validate against current vendor documentation before making a design decision.