AI / ML topic How Do You Train One Model on 16,000 GPUs?

Meta trained its Llama 3 405B model on up to 16,000 H100 GPUs at once. In one 54-day stretch, the job was interrupted 466 times, 419 of them by surprise. That’s a failure about every three hours, for weeks. And the training still made progress more than 90% of the time.

How that works is less about AI than about distributed systems. A model too big for any one machine has to be split across thousands of them. Those machines have to stay in lockstep over a network.

And when one dies, which is constantly, the thousands of others can’t lose the week.

Everyone gets a copy

Training is a loop: show the model a batch of examples, measure how wrong it was, and use gradient descent to nudge every weight a little. The simplest way to use more GPUs is data parallelism: every GPU holds a full copy of the model and works on a different slice of the batch. After each step, they average their nudges so every copy stays identical.

That averaging is an all-reduce, and it happens every step, across every GPU. NVIDIA’s NCCL library does it, and the classic trick is to pass pieces around a ring, so each GPU sends about the same amount of data however many GPUs join.

When the model doesn’t fit

Data parallelism breaks the moment the model is bigger than one GPU’s memory, and for the frontier, it always is. Training needs far more memory than the weights alone: the nudges and the optimizer’s bookkeeping take several times as much. So labs split the model itself:

• FSDP (PyTorch’s take on Microsoft’s ZeRO) is data parallelism where each GPU stores only a slice and borrows the rest just in time.
• Tensor parallelism, as in NVIDIA’s Megatron-LM, splits every layer across GPUs, which talk constantly.
• Pipeline parallelism gives each GPU a run of layers and passes data down the line. Google’s GPipe named its cost, the “bubble” while the line fills and drains.

Real runs stack them. Llama 3 used four kinds of parallelism at once. The art is placing each one where the network can carry it: the chatty tensor parallelism inside a server over NVLink, the lighter traffic across the slower links between servers.

Things break, constantly

At this scale, hardware failure is routine. In Meta’s 54-day snapshot, about 78% of the unexpected interruptions were confirmed or suspected hardware problems, and faulty GPUs were the biggest single cause. And because all GPUs move in lockstep, one failure stops the whole job. Automation has to do the recovering: Meta reports that significant manual intervention was needed only three times in those 54 days.

The defence is the checkpoint: a periodic save of the model and its training state. When a machine dies, you swap it out, reload the last checkpoint, and carry on. Anything done since that save is lost.

Save too often and you waste time saving. Save too rarely and every failure throws away hours. There’s a classic formula for the balance, the Young–Daly interval, which says to save about every √(2 × save time × time between failures).

Try the background-save option. That’s why labs work so hard on fast, asynchronous checkpoints. PyTorch reported cutting checkpoint pauses by 10 to 20 times that way. Meta’s paper says it aimed to “increase checkpoint frequency to reduce the amount of lost work after a recovery.”

Measuring the waste

All this machinery leaks efficiency, so labs track one number: MFU, the share of the hardware’s peak math that went into actual training. Google introduced the metric in 2022, when PaLM hit 46.2% on 6,144 TPU chips. Llama 3 reached 38 to 43%. In 2021, NVIDIA ran a trillion-weight model on 3,072 GPUs at 52% of peak, though that count includes recomputed work that MFU leaves out.

Leaving around half the silicon idle is normal, because the rest of the time goes to waiting on memory, the network and each other.

Efficiency is also why a run’s cost is hard to state. DeepSeek said its V3 model’s final training took 2.788 million GPU hours, about $5.6 million at $2 an hour, on 2,048 GPUs. The same paper notes that figure excludes “prior research and ablation experiments”, so the final run is only part of the budget.

So how do you train on 16,000 GPUs?

Like any big distributed system, with more at stake. Split the work so each piece fits, route the chattiest traffic over the fastest links, expect a machine to die every few hours, and checkpoint so that a death costs minutes, not days. Then measure how much of the hardware you’re really using, and fight for every percent. If you’d like to try the smallest version yourself, Kaggle’s free notebooks offer two T4 GPUs (as of September 2026), enough for a real, if tiny, data-parallel run.

References & further reading

In the order the article reaches them.