Where the GPU's own network ends: scale-up versus scale-out

For years the GPU’s private network ended at the edge of one server: eight GPUs on NVLink, and everything else over the back-end fabric. Rack-scale systems moved that edge to the whole rack. That one change quietly rewrites how the scale-out network should be designed.

Two networks with different jobs

Think of an open-plan office and the city outside it. Inside, everyone turns and talks to anyone instantly, so the team that argues over every detail sits together. Between offices, people send email: fast, but slower, so it carries decisions rather than debate. A bigger office keeps more of the debate inside.

A scale-up network makes many GPUs behave like one: memory semantics instead of messages, no NIC in the path, one switch hop between any two GPUs, reliability built into each link, copper inside the rack. A scale-out network connects those domains with RDMA over Ethernet or InfiniBand, routing, ECMP and everything that comes with them.

The ratio that decides the design

Per GPU, each direction: inside about 900 GB/s; outside one 800G NIC = 100 GB/s
A 72-GPU rack: 64.8 TB/s inside, 7.2 TB/s out of the rack

The rack talks to itself about nine times faster than to anything else. So:

• Tensor parallelism stays inside, and with a domain this size, so does the bursty all-to-all of mixture-of-experts models. That was impossible with 8-GPU servers.
• Pipeline and data parallelism cross the fabric. The back-end still carries every gradient exchange and checkpoint, so reference designs keep it non-blocking.
• Jobs must align to domains. A tensor-parallel group split across two racks runs at scale-out speed.

What it changes for the network engineer

• The rack becomes the failure domain. A failed scale-up switch tray degrades every GPU in it. Plan spares in racks, not servers.
• Power decides the floor plan. These racks draw over 100 kW and are liquid cooled. Fewer, denser racks mean shorter scale-out runs.
• Two fabrics, two toolsets. Decide early who owns the scale-up fabric and feed its telemetry into the same monitoring as the rest.

The open alternatives to NVLink, UALink and Ethernet-based scale-up from the Open Compute Project, keep the same essentials: single hop, memory semantics, link-level reliability. The choice usually comes with the accelerator.

What would change my mind

If scale-up domains grow to thousands of GPUs, with optics instead of copper, the line between the two networks blurs, and much of what we call back-end design moves inside the domain. We are not there yet; design the scale-out network around the domain you can actually buy.