Reference design: a 2,048-GPU training cluster, from brief to acceptance

Most AI cluster projects go wrong before the first switch is ordered. Someone picks a product, then a topology, and the requirements arrive last. This reference design runs in the other direction: a business brief, the decisions it forces, the numbers that size the result, and the plan that proves it works before anyone runs a real job.

The customer, Northstar Labs, is fictitious. The design is the one in Chapter 16 of Building the AI Factory, condensed here for the people who have to approve it.

The brief

Requirement	Value
Compute	256 servers with 8 GPUs each: 2,048 GPUs
Largest job	All 2,048 GPUs in one pre-training run
Growth	Double to 4,096 GPUs within 24 months, still as one job
What the business judges	Job completion time
Team	Strong Ethernet, BGP and EVPN skills; no InfiniBand experience
Vendor policy	Ethernet preferred, to align with the rest of the estate
Storage	1.5 TB/s of dataset reads; 20 TB checkpoints within 2 minutes
Facility	Liquid-cooled hall, up to 60 kW per rack

Two lines in this table decide more than all the others: the growth target and the team’s skills. Everything below traces back to one of them.

Nine decisions, each with its reason

A decision log records the choice, the alternative that was considered, and the requirement that decided between them. It is the most useful document a design produces: a year later, it tells the next engineer why, not just what.

Decision	Alternative	Decided by
RoCEv2 Ethernet for the GPU fabric	InfiniBand	Team skills and vendor policy
A dedicated GPU fabric, separate from front-end and storage	One converged fabric	Job completion time: synchronized GPU traffic must not share queues
Two tiers of 51.2 Tb/s switches, 128 ports of 400G each	64-port switches with a third tier	Growth to 4,096 GPUs fits two tiers only at 128 ports
Non-blocking (no oversubscription)	Oversubscribed to save cost	GPUs communicate in lockstep: the slowest link sets the pace for all
Adaptive routing	Classic hashing (ECMP) only	A few very large flows collide under hashing
Lossless RoCE profile (PFC and ECN)	Lossy Ethernet with newer transports	The established, platform-validated baseline today
Separate storage network, generous on the array side	Storage on the front-end	Checkpoints converge on the arrays all at once
Asynchronous, sharded checkpoints	Synchronous writes	Hide the burst instead of buying bandwidth for it
Front-end servers dual-homed to a leaf pair	Single-homed	A single leaf failure must not take servers offline

Notice what is missing: no decision says “because vendor X recommends it”. If a decision cannot name its requirement, it is a preference, and preferences are where budgets leak.

The numbers

GPU fabric
  2,048 GPU ports at 400G
  Two-tier ceiling at 128 ports: 128 x 128 / 2 = 8,192 ports  -> 4,096 GPUs fits
  Scalable unit: 64 servers (512 GPUs) on 8 rail leaves
  Today: 4 units = 32 leaves; 2,048 uplinks / 128 = 16 spines
  Growth: 4 more units and 16 more spines, no third tier

Storage
  Reads 1.5 TB/s = 12 Tb/s; checkpoint 20 TB / 120 s = 1.33 Tb/s
  Sized for both at once: about 13.3 Tb/s toward the arrays
  32 storage nodes x 2 x 400G = 25.6 Tb/s, still above peak with one spine down

Racks
  About 10.2 kW per server, 4 per rack = 41 kW, inside the 60 kW limit
  64 compute racks, plus network racks in the middle of each unit's rows

One number deserves a manager’s attention: the GPU fabric alone has more than four thousand 400G links. Optics and cables, not switches, drive cost, power and installation time. That is why the leaves sit in the middle of each unit’s rows, within short and cheap reach of the servers, and why space for phase two is reserved now: growth becomes an addition, not a rebuild.

Proving it works: five acceptance stages

The cluster is built and accepted one scalable unit (512 GPUs) at a time. A unit joins the fabric only after it passes all five stages:

1. Physical: every cable matches the plan (checked automatically with LLDP), every link runs at full speed, and error rates stay within limits under load.
2. Configuration: congestion control, adaptive routing and MTU are identical on every switch and server; all routing sessions are up.
3. Point to point: RDMA bandwidth between every pair of hosts on each rail, all within a few percent of each other.
4. Collective: the same communication patterns training uses (all-reduce, all-to-all) within the unit, then across units as each joins.
5. Soak: a full-cluster training run of at least 24 hours, watching pauses, drops, link errors and step-time variation.

The results become the baseline: the reference every future “the job is slow” investigation compares against. A cluster without a recorded baseline turns every performance complaint into a debate.

What would change this design

• A team experienced with InfiniBand, or a vendor policy that allows it: InfiniBand becomes a strong option, with mature adaptive routing and in-network reduction.
• Growth beyond 4,096 GPUs in one job: two tiers stop at 8,192 GPU ports with these switches, so the next step is a multi-plane design or a third tier, and that choice should be made now, not at 8,000.
• Many small jobs instead of one large one: the case for non-blocking weakens, and placing each job inside one scalable unit matters more than the spines.
• A facility limit below about 41 kW per rack: fewer servers per rack, longer cable runs, and possibly different optics. Facilities are a design input, not an afterthought.

The questions to ask before you buy

If you are planning a cluster, these are the questions this design answered first. Each one changes the bill of materials:

• What is the largest single job, today and in two years?
• What does the business measure: job completion time, cost per GPU-hour, or time to first result?
• What does the team already know how to operate at 3 a.m.?
• How large is a checkpoint, and how long may it take?
• What power and cooling per rack does the facility guarantee, for both phases?

The full chapter, with switch and host configurations, the bill of materials and the operating model, is in Building the AI Factory. The free sample includes the design checklist that produced this decision log.

Northstar Labs is a fictitious organization. This is a reference design for learning; validate every choice against your own requirements and your vendor’s current documentation.