Zero-copy GPU read of an existing CPU tensor in PyTorch: cudaHostRegister + __cuda_array_interface__ (measured GH200 vs H200)
To let the GPU read a large CPU tensor in place (no staging copy), pin it in place and wrap the pointer as a CUDA tensor:
class HostMemory:
def __init__(self, t):
raw = t.view(torch.int16) # bf16 has no CUDA array interface typestr
self.t = t
self.__cuda_array_interface__ = {'shape': tuple(raw.shape), 'strides': None,
'typestr': '<i2', 'data': (raw.data_ptr(), False), 'version': 3}
size = host.numel() * host.element_size()
assert int(torch.cuda.cudart().cudaHostRegister(host.data_ptr(), size, 0)) == 0
gpu_view = torch.as_tensor(HostMemory(host), device='cuda').view(host.dtype)
assert gpu_view.data_ptr() == host.data_ptr()
blocks = gpu_view[w].index_select(0, ids_on_gpu) # GPU gathers straight from CPU RAM
# ... later: torch.cuda.cudart().cudaHostUnregister(host.data_ptr())Plain pageable memory without registration is rejected by torch.as_tensor (the pointer is not registered with CUDA), so register it first. Registering 36 GiB in place took 0.05 s on GH200, with no extra RSS.
Measured (12 MiB gathered from 8 of 128 blocks of 1.5 MiB, bf16, 24 requests, outputs bit-exact): GH200 (NVLink-C2C, PyTorch 2.12) 0.072 ms = 175 GB/s; H200 over PCIe (PyTorch 2.8) 0.316 ms = 40 GB/s. The common alternative, a CPU index_select into a pinned buffer followed by a non_blocking copy, reached only 60 GB/s on GH200 and 26 GB/s on H200, because the CPU gather is the bottleneck. The direct read is the path that actually uses the faster GH200 link. Tested on PyTorch 2.8 and 2.12 with CUDA 12.8 and 13.0.