networking · storage · multi-node
GPUDirect: getting data to the GPU without the CPU
Training moves data from storage to GPUs, and between GPUs on different servers. Normally every byte detours through the CPU and host RAM. GPUDirect RDMA and GPUDirect Storage remove the detour. What that costs, what it needs, and at what scale it starts to matter.
Before you start
This article assumes you know:
- A GPU is the chip that does the maths in training, and it has its own fast memory.
- Training on many GPUs means they swap results (gradients) after every step.
- Servers talk to each other through network cards (NICs).
Ideas it leans on:
- PCIe
- · the bus inside a server that connects the CPU, GPUs, network cards and SSDs.
- DMA
- · direct memory access: a device copying data to a memory address by itself, without the CPU.
- RDMA
- · remote direct memory access: a network card writing straight into another machine's memory.
- All-reduce
- · every GPU averaging its gradients with every other GPU, once per training step.
The use case: one training step
A training step moves data along three paths. Only one of them is the GPU doing maths:
- Storage → GPU. The next batch of training data comes from storage (a bucket like GCS, a parallel file system like Lustre, or a local SSD). Every so often, a checkpoint goes the other way, and after a crash the whole checkpoint is read back.
- The step itself. Forward and backward passes, inside GPU memory (HBM, high-bandwidth memory stacked beside the GPU chip).
- GPU ↔ GPU. With data parallelism, every GPU runs the same model on its own slice of the batch and works out its gradients (how each weight should change). Before the weights are updated, all GPUs average their gradients: an all-reduce. Inside one server that runs over NVLink, NVIDIA’s direct GPU-to-GPU link. Across servers it goes through network cards (NICs).
GPU memory is fast: an H100 reads its HBM3 at 3.35 TB/s and a B200 at 8 TB/s (Google Cloud’s machine table). Getting data into it is the hard part. GPUDirect is NVIDIA’s family of technologies that let other devices read and write GPU memory directly. Two matter for training: GPUDirect RDMA (NIC ↔ GPU memory) and GPUDirect Storage (storage ↔ GPU memory).
Where this lab stands
These paths are steps 3 and 5–7 of the ten in AI training at scale, which puts them in the context of a whole run.
The problem: the long way round
Inside a server, the GPU, the NIC and the SSD all hang off PCIe, the server’s internal bus. A device moves data by DMA (direct memory access): it copies bytes to an address it’s given, without the CPU. For decades that address was in host RAM, the CPU’s memory, because that is where the operating system and the application live. GPU memory sits on the far side of another PCIe link. So data from the network or a disk takes the long way: into host RAM, through the CPU’s root complex (where its PCIe lanes start), and only then across PCIe again into the GPU.
Watch one batch arrive from storage. Each stop is something the batch has to wait for:
- copies of the data
- 0
- CPU work on the data
- 0
- PCIe hops
- 0
- through host RAM
- no
One storage server and one GPU server shown. In practice every GPU server does this at once, and a parallel file system stripes each read across dozens of storage servers.
The usual path costs four things. On one server they are a nuisance. Once training spans many servers, every NIC pays them at the same time:
- Copies. Each one is a full pass over the data.
- CPU time. TCP, TLS (the encryption behind HTTPS) and copying use cores the data loader needs for decoding.
- Host RAM bandwidth. Every staged byte is written to RAM once and read back once, and a whole server’s RAM has less bandwidth than a single GPU’s HBM (the arithmetic).
- Latency. Each hop adds microseconds, and an all-reduce is many small exchanges that feel every one.
Two layers: RDMA fixes the network, GPUDirect fixes the host
The two fixes are separate, and you need both for the straight path (switch scenes in the figure above). In both, the CPU still sets up every transfer (the control path). What changes is that it no longer touches the data.
Layer 1 · RDMA: no kernel network stack
RDMA (remote direct memory access) is DMA across the network: a NIC reads and writes memory on another machine without that machine’s CPU. The application registers a buffer once, then hands work straight to the NIC: no system call, no TCP stack, no socket copy. For one-sided reads and writes the remote CPU isn’t even interrupted. RDMA runs over InfiniBand (a network built for it), over ordinary Ethernet as RoCE v2 (RDMA over Converged Ethernet), or over iWARP (RDMA carried inside TCP). Clouds add their own transports (below); the NVIDIA GPU machines in this article use ConnectX NICs.
For storage, this is the difference between a Lustre client talking to its servers over TCP and over RDMA. RDMA removes the kernel socket copy and most of the CPU work. But plain RDMA still targets host RAM, so the data lands in a bounce buffer there and is copied again into the GPU.
Layer 2 · GPUDirect: no bounce through host RAM
GPUDirect lets the NIC’s DMA target GPU memory instead. The GPU driver tells the NIC driver where a GPU buffer lives, and the NIC writes into it peer to peer across PCIe (the driver pieces are under what you need). GPUDirect RDMA has done this since NVIDIA’s Kepler GPUs and CUDA 5.0 (CUDA is NVIDIA’s GPU programming toolkit).
GPUDirect Storage applies the same idea to files: you call cuFileRead with a GPU buffer, and a supported file system (NVIDIA names DDN EXAScaler, VAST NFS, WekaFS and Lustre clients among them) places the bytes there. If the path isn’t supported, cuFile falls back to a compatibility mode that stages through host RAM: the same code still runs, just without the benefit.
Local NVMe: the same trick inside one server
Local SSDs sit on the same PCIe fabric as the GPU, so the detour is pure waste: data leaves the drive, crosses into host RAM (the operating system’s page cache), gets copied by the CPU, and crosses back. With GPUDirect Storage the drive DMAs straight into GPU memory through the shared PCIe switch. Files should be opened with O_DIRECT, which tells the operating system not to cache them in RAM. Since CUDA 12.2, cuFile also takes the direct path without it, as long as buffers, sizes and offsets are aligned.
- copies of the data
- 0
- CPU work on the data
- 0
- PCIe hops
- 0
- through host RAM
- no
GPU to GPU across servers
The all-reduce has the same problem, at both ends. Over TCP sockets, gradients are copied out of the GPU, through the operating system on both machines, and back into the other GPU. RDMA removes the operating system and the remote CPU. GPUDirect RDMA removes host RAM: the NIC reads straight out of one GPU’s memory and the far NIC writes straight into another’s. NCCL, the library PyTorch uses for all-reduce, does this by default whenever the topology allows it. Inference servers use the same path to hand a KV cache from one server to another (prefill–decode disaggregation).
- copies of the data
- 0
- CPU work on the data
- 0
- PCIe hops
- 0
- through host RAM
- no
One pair of servers and one GPU each shown. In a real all-reduce every GPU in every server does this at the same time, through its own NIC.
At what scale does it matter?
Not inside one NVLink domain: the 8 GPUs of a DGX, or the 72 of a GB200 or Rubin NVL72 rack, share NVLink, and the NICs sit idle during an all-reduce. Once training spans two servers (or racks), every step’s gradients cross the network, and the host RAM detour stops being a few extra copies. It starts capping how fast every NIC can run. The arithmetic for a DGX H100, whose user guide lists eight 400 Gb/s ConnectX-7 NICs for the cluster network and two Xeon 8480C CPUs:
- 8 NICs × 400 Gb/s = 400 GB/s in each direction, 800 GB/s both ways, at line rate (the most a link can carry).
- Staged through host RAM, each byte is written once and read once: 1.6 TB/s of memory traffic.
- Two Sapphire Rapids CPUs have 8 memory channels each. The DGX H100 fills them with 32 DIMMs, two per channel, where the rated speed is DDR5-4400: 2 × 8 × 4.4 GT/s × 8 bytes ≈ 563 GB/s, theoretical peak.
- So host RAM can feed each NIC at most ~17.6 GB/s, 35% of its line rate, before the CPU does any work. One H100’s HBM (3.35 TB/s) is almost six times the whole server’s RAM bandwidth.
Change the cluster and see where the line falls:
Servers
NIC per GPU
Training data and checkpoints
Lustre size (throughput as Managed Lustre's 1,000 MBps-per-TiB tier)
Model
Does GPUDirect matter? It matters. With 8 servers, every step pushes 31 GB through each NIC. GPUDirect RDMA does that in 613 ms; staged through host RAM it would take at least 1.7 s.
Host RAM, if the network is staged through it
8 NICs × 50 GB/s × 2 directions × 2 memory passes (the NIC writes each byte, the GPU copy reads it). At best each NIC gets 17.6 GB/s, 35% of its line rate, before the CPU copies anything. Memory: DDR5-4400, the rated speed with 2 DIMMs per channel as in the DGX H100.
Gradient exchange per training step
140 GB of bf16 gradients. Reduce-scatter on NVLink, then each GPU’s 1/8 shard crosses servers through its own NIC (a ring sends 2·(n−1)/n of it), then all-gather on NVLink. Ignores latency and overlap with the backward pass. Large models don’t fit in 8 GPUs with plain data parallelism and are sharded (FSDP, ZeRO); the bytes each NIC carries stay about the same. Real jobs at thousands of GPUs combine tensor, pipeline and data parallelism (Llama 3 405B trained on 16,384 H100s), so read this as the data-parallel share of the traffic.
Restoring a checkpoint from Lustre
9.8 s
980 GB (14 bytes per parameter: bf16 weights, fp32 master weights, two Adam moments), read at best at 100 GB/s, what this file system can serve. With RDMA and GPUDirect Storage the bytes land in GPU memory directly. (Google's Managed Lustre docs are used only for the throughput per TiB; they don't describe GPUDirect Storage.)
Where it starts to matter
Communication time per step for 70B, from 8 to 4,096 GPUs. Solid: GPUDirect RDMA; dashed: staged through host RAM. It climbs to a plateau: past a few servers each NIC carries about twice its shard however many GPUs you add, so NIC speed sets the cost, and staging multiplies it at every size.
Sources: NVIDIA DGX H100 user guide · NVIDIA H100 · Sapphire Rapids: 8 DDR5 channels per CPU · Xeon 8480: DDR5-4400 at 2 DIMMs per channel · NVIDIA ConnectX adapters · Managed Lustre performance tiers (1,000 MBps per TiB) · Google Cloud Managed Lustre: up to 10 TBps · NVIDIA GPUDirect RDMA documentation · NVIDIA GPUDirect Storage overview. Memory bandwidth is the CPUs’ theoretical peak (channels × transfer rate × 8 bytes); real systems reach less.
When it matters, and when it doesn’t
- Text pretraining data is small. Our whole 10-billion-token dataset is about 20 GB as 2-byte token ids. The 8× H100 run read it once in a few hours, and the same run on our single home GPU took a little over two days; either way that is a trickle GPUDirect Storage won’t speed up.
- Checkpoints are big, and failures are frequent at scale. With Adam in mixed precision a checkpoint is about 14 bytes per parameter: a 405B model is ~5.7 TB. Meta reported 419 unexpected interruptions in a 54-day stretch of Llama 3 405B pretraining on 16,384 H100s, roughly one every three hours (Llama 3 paper), and each one means reading a checkpoint back. While it loads, every GPU in the job waits. (Our own checkpointing lesson measures what a crash costs on one GPU.)
- Images, video and audio can be read faster than the CPU can decode them. GPUDirect Storage pairs with decoding on the GPU.
- Gradients across servers are where GPUDirect pays most: past one server, staging through host RAM caps every NIC well below its line rate. Overlapping the exchange with the backward pass hides some of it, and Google’s A3 machines get a GPUDirect path over TCP instead of RDMA, but some direct path is what large clusters are built around.
Many GPUs: real systems
Large clusters are built so that GPUDirect always has a short path. Each GPU gets its own NIC behind the same PCIe switch, and GPU i of every server is cabled to the same leaf switch (the first layer of network switches), its rail. Traffic between GPUs in one server stays on NVLink; traffic between servers goes rail by rail, all eight at once, and a layer of spine switches above the leaves joins groups of servers into clusters of thousands of GPUs.
| System | GPUs on one NVLink | Network to other servers | HBM per GPU |
|---|---|---|---|
| DGX H100 | 8, 900 GB/s per GPU | 8× ConnectX-7, 400 Gb/s (guide), plus 2 dual-port ConnectX-7 for storage | 80 GB HBM3, 3.35 TB/s |
| DGX B200 | 8, 1.8 TB/s per GPU | 8× ConnectX-7, 400 Gb/s (guide), plus 2 BlueField-3 DPUs (NICs with their own processors) for storage | 180 GB HBM3e, 8 TB/s |
| GB200 NVL72 | 72 in one rack | On Google Cloud (A4X): 4 ConnectX-7 per 4-GPU machine, one per GPU, RDMA over RoCE (GCP) | 186 GB HBM3e, 8 TB/s |
| Vera Rubin NVL72 | 72 in one rack | ConnectX-9 SuperNICs (NVIDIA’s name for its AI-cluster NICs; NVIDIA) | 288 GB HBM4, 19.2 TB/s |
Two trends meet here. Rack-scale NVLink (72 GPUs) keeps more of the all-reduce off the network, but the traffic that remains is larger and HBM keeps getting faster: 3.35 TB/s on H100, 8 TB/s on Blackwell, 19.2 TB/s on Rubin. Host RAM has not kept that pace, so staging through it falls further behind every generation. On Google Cloud you meet both variants (Google’s network bandwidth table): A3 High, A3 Edge and A3 Mega (H100) move GPU data with GPUDirect-TCPX and TCPXO, a GPUDirect path over TCP; A3 Ultra (H200) and A4 (B200) have eight 400 Gb/s ConnectX-7 NICs with GPUDirect RDMA over RoCE, as in a DGX; A4X has four ConnectX-7 and A4X Max four 800 Gb/s ConnectX-8, one per GPU.
Beyond NVIDIA and Google
The same problem, a NIC that has to reach accelerator memory without a detour, is solved differently across the market:
| Who | How accelerator memory reaches the network |
|---|---|
| AWS | EFA, using AWS’s own SRD transport, does direct transfers between the NIC and GPU memory (GPUDirect RDMA) on P5-class and newer GPU instances, and the equivalent for Trainium. FSx for Lustre supports EFA and GPUDirect Storage, up to 1,200 Gb/s per P5 client. |
| Microsoft Azure | ND H100 v5: a dedicated 400 Gb/s InfiniBand link per GPU (3.2 Tb/s per VM), with GPUDirect RDMA. |
| AMD | RCCL, AMD’s counterpart to NCCL, runs collectives between Instinct GPUs over InfiniBand and RoCE across servers, and over xGMI and PCIe inside one. |
| Intel Gaudi 3 | Puts the network on the accelerator itself: 24 × 200 Gb/s Ethernet ports with RoCE on chip, so there is no PCIe hop to a separate NIC. |
| Google TPU | Chips in a slice talk over their own inter-chip interconnect (ICI); only traffic between slices uses the data-center network (TPU architecture). |
Clouds also build their own transports. Besides AWS’s SRD, Google designed Falcon, a hardware transport that carries RDMA and NVMe; it first shipped in Intel’s IPU E2000, and Google said it would contribute the spec to the Open Compute Project.
Which storage can do this?
| Storage | How bytes reach the GPU | GPUDirect? | Limits |
|---|---|---|---|
| Object storage GCS, S3 | HTTPS over TCP → CPU → host RAM → GPU | No standard path: HTTP and TLS need the CPU | Latency per request, so throughput comes from many parallel reads. Caching helps: GCS Rapid Cache starts at 100 Gb/s and scales to 20 Tb/s per project and zone (quotas). The durable home of datasets and checkpoints. |
| Parallel and high-performance file systems Parallel: Lustre (incl. DDN EXAScaler), WekaFS, IBM Storage Scale; AWS FSx for Lustre. Over NFS: VAST (disaggregated shared-everything) and scale-out NAS such as NetApp ONTAP (compared here) | RDMA (InfiniBand or RoCE) to the client, striped across many servers | Yes, GPUDirect Storage, on supported clients (NVIDIA) | Provisioned capacity and cost. Google lists Managed Lustre at up to 10 TB/s; its docs don’t describe GPUDirect Storage for it. AWS documents GPUDirect Storage for FSx for Lustre on P5 instances. |
| Local NVMe SSD | PCIe in the same server | Yes, GPUDirect Storage | One server’s worth (a DGX H100 has 8 × 3.84 TB for cache), not shared, must be filled first. In the cloud it’s scratch: Google warns the data may be lost if the instance stops (Local SSD docs). |
| NVRAM / persistent memory Intel Optane PMem | Sits on the CPU’s memory bus, like DRAM | Not applicable: to the GPU it is host memory, so it’s still the detour | Slower than DRAM, tied to one CPU socket, and discontinued by Intel. |
| CXL memory CXL expanders | More host memory, attached over PCIe-class links | Same as host RAM | Adds capacity and some bandwidth to the host; doesn’t remove the bounce. |
| KV cache transfer engines Mooncake | RDMA between GPU memory, host RAM and SSD across servers | Yes: its Transfer Engine moves KV caches over RDMA and can target accelerator memory | For inference, not training data: Mooncake serves Kimi and moves KV caches between prefill and decode servers (why they move); vLLM, SGLang and TensorRT-LLM integrate it (Mooncake explained). |
GCS vs Lustre, in one line
What you need
- An NVIDIA data-center or workstation GPU. NVIDIA’s GPUDirect RDMA guide lists Tesla (data-center) and Quadro (professional) GPUs; consumer GeForce cards aren’t on the list.
- RDMA-capable NICs: InfiniBand or RoCE, most often NVIDIA ConnectX or BlueField, and Linux with the RDMA stack (or a cloud’s own transport, such as AWS EFA).
- The bridge between the two drivers:
nvidia-peermem(in the NVIDIA driver since CUDA 11.4) or dma-buf. For storage, the GDS cuFile library and a file system that supports it. - The right PCIe topology. NVIDIA’s guide says the two devices “must share the same upstream PCI Express root complex”, and peer-to-peer traffic is best when they share a PCIe switch.
nvidia-smi topo -mshows how each GPU and NIC are connected. Here is the slow path you get when the NIC is on the wrong side:
- copies of the data
- 0
- CPU work on the data
- 0
- PCIe hops
- 0
- through host RAM
- no
- Software that uses it. NCCL enables GPUDirect RDMA on its own when the topology permits (
NCCL_NET_GDR_LEVELsets how far apart GPU and NIC may be). For storage, call cuFile directly or through a wrapper such as KvikIO. - In the cloud, a machine type that exposes it: on Google Cloud, A3 Ultra, A4, A4X and A4X Max with GPUDirect RDMA over RoCE, or A3 High, Edge and Mega with GPUDirect-TCPX/TCPXO (GCP); on AWS, EFA (AWS’s own SRD transport), which supports GPUDirect RDMA on P5-class and newer instances; on Azure, ND H100 v5 and later with InfiniBand.
Ask about this article
Something unclear, or something we got wrong? Use the Ask button at the bottom right; it stays with you as you read. Answers come from this article, sometimes with a small chart, and each shows what it cost. To flag a passage, select it.
Sources
- NVIDIA GPUDirect overview
- NVIDIA GPUDirect RDMA documentation
- NVIDIA GPUDirect Storage overview guide
- NCCL environment variables (NCCL_NET_GDR_LEVEL)
- DGX H100 user guide: hardware overview
- DGX B200 user guide: hardware overview
- NVIDIA H100
- NVIDIA DGX B200
- NVIDIA GB200 NVL72
- NVIDIA Vera Rubin NVL72
- Google Cloud accelerator-optimized machines
- Google Cloud: GPUDirect-TCPX and TCPXO
- Google Cloud Managed Lustre overview
- Cloud Storage quotas and limits
- Google Cloud Local SSD
- Google: Introducing Falcon
- Google Cloud: GPU machine network bandwidth
- Sapphire Rapids memory (Wikipedia)
- Intel Xeon Platinum 8480+ specifications (DDR5-4400 at 2 DIMMs per channel)
- Intel Xeon 8570 specifications
- Mooncake
- AWS: Elastic Fabric Adapter (SRD, GPUDirect RDMA support by instance)
- AWS: FSx for Lustre adds EFA and GPUDirect Storage
- Microsoft: ND H100 v5 series
- AMD RCCL documentation
- Intel Gaudi 3
- Google Cloud TPU architecture
- The Llama 3 Herd of Models (training reliability)