04 / THE NETWORK

The cost of moving data

Training across many GPUs means doing two kinds of work: calculating an update, then exchanging it. The network determines how much time the second part adds.

Why faster chips can still wait

In this example, every GPU calculates part of a training update. They then combine their updates in a synchronization step called all-reduce. Some communication can happen while computation continues; the rest extends the step. Keeping a job within one server or network group avoids slower shared connections.

FIGURE 1

Where a job runs determines which connections it uses.

Shared uplink capacity

Where the job runs

A leaf switch connects a group of servers. Spine switches connect those groups. Colored lines show the path used by the selected job.

Model size · billion parameters 70B

Bandwidth per GPU

—

Full synchronization time

—

Compute-only throughput retained

—

Modeled price proxy

—

A communication model, not a training benchmark. The price proxy divides the entered hourly rate by the fraction of compute-only throughput retained; it does not estimate full training cost.

Bandwidth, overlap & price assumptions

Connection assumption

Both start at 400 Gb/s (50 GB/s). The 92% and 80% efficiency factors are illustrative inputs, not a benchmark comparison between protocols.

Compute-only step · sec

Communication hidden by compute · %

Measured collective bandwidth · GB/s

Quoted price · $/GPU-hr

How the model works & what to verify

Decision supported: Does the proposed fabric undermine the quote?
Primary output: Modeled step penalty and price proxy
Model boundary: One full-gradient ring all-reduce; not total application goodput

The price output is a communication-bound proxy, not realized training cost. It excludes sharding, topology-aware collective selection, pipeline bubbles, congestion, storage, and application failures.

Fabric diligence

Why the hierarchy matters. This model uses an 810 GB/s internal connection for one eight-GPU server and a 400 Gb/s (50 GB/s) network interface per GPU for larger jobs. Those are scenario assumptions, not a specification for every H100 system. Communication outside the server passes through leaf switches and, for a cluster-wide job, the shared spine tier. The rack and pod figures show where these connections sit physically.

Protocol labels do not establish delivered performance. InfiniBand and RoCE can run the same nominal wire rate and both carry RDMA. Their delivered collective bandwidth depends on NIC and switch generation, topology, congestion control, tuning, and job size. Use line rate only as a ceiling; request measured NCCL bus bandwidth for the proposed configuration.

Ask the supplierInfiniBand or Ethernet, which NIC generation, and what measured NCCL all-reduce bus bandwidth do you see at my job size — not line rate, measured?

Oversubscription changes the value of placement. A 2:1 fabric has half as much uplink capacity out of each leaf as downlink below it. Workloads that remain within a leaf may see little effect; jobs crossing leaf and spine tiers may contend for the reduced uplink capacity. Ask for both the topology diagram and the scheduler's placement policy. The pod page shows the planning ratio physically: compare the tray at 1:1 and 2:1 →

Ask the supplierWhat's the oversubscription ratio, leaf to spine — and can your scheduler place my job within a single leaf or rail?

Rail-optimized designs connect each node NIC to a corresponding switch rail, which can reduce switch count for expected collective patterns. Arbitrary cross-rank and all-to-all traffic may take a less direct path. Validate both the dominant workload and the failure behavior before accepting the cost tradeoff.

Ask the supplierFull fat-tree or rail-optimized? Show me the topology diagram, and tell me the blast radius when one spine switch fails.

Inference requirements vary by serving architecture. Single-node serving may not justify a training-oriented scale-out fabric. Large mixture-of-experts deployments, disaggregated prefill and decode, or models sharded across nodes can create substantial east-west traffic. Size the network from the serving design and measured communication profile, not from the accelerator count alone.

How the proxy is calculated. The model moves one FP16 gradient copy for every parameter through a ring all-reduce, exposes the portion not hidden by overlap, adds that time to the compute-only step, and scales the sticker price by the resulting throughput share. This is a sensitivity calculation, not a benchmark. The 0.92 InfiniBand and 0.80 tuned-RoCE efficiencies are planning assumptions; replace them with measured NCCL bus bandwidth whenever possible.

model · protocol efficiencies are explicit planning assumptions, not benchmarks · model version 2026.08 · NCCL collectives · methods & limitations