io_uring + DMA-BUF · Experimental · Updated 22 October 2026

What the tests establish

An exported handle, a full-length I/O completion and a correct model continuation prove different things. These records separate low-level export, verified storage transfers, serving behavior and performance.

Driver and exporter evidence

Recorded capability, with the limit beside it
Configuration	Observed result	What it does not establish
NVIDIA open modules + native CUDA DMA-BUF	Native CUDA export and verified GPU storage transfers in the September comparison.	Support for every NVIDIA device or for the proprietary kernel-module variant.
NVIDIA open modules + Vulkan/RM helper	Native CUDA export returned 801 on a different profile. The helper later passed 18/18 XFS and 18/18 ext4 GPU-to-storage checks at 4 KiB, 64 KiB and 2 MiB; three 2 MiB storage-to-GPU reads were verified.	A standard Vulkan DMA-BUF exporter, all devices, or every PCIe route. Later LMCache serving evidence is listed separately below.
Nouveau + NVK	A 2 MiB Vulkan allocation exported as DMA-BUF.	NVMe attachment, persistent VRAM placement or byte-correct storage I/O in both directions. Full storage qualification was not completed.
Nova	No completed storage qualification in this evaluation.	A working CUDA/export/storage stack. Further enablement and testing are needed.
AMD ROCm DMA-BUF	Verified direct GPU storage transfers in the September AIS comparison.	That later HIP host-staged serving tests are AMD GPU-direct results.
NVIDIA proprietary kernel modules	Not established as the io_uring DMA-BUF test route by the cited results.	That use of proprietary CUDA or Vulkan userspace means the closed kernel module was loaded.

The low-level NVIDIA helper checks above were recorded on 7 October 2026. They are separate from the 15 October application cohorts below. On another platform, a cross-Root-Port pair failed independent readback despite reporting a full write completion; see the negative qualification. A passing exporter on one platform cannot qualify another route.

What is the overhead compared with P2PDMA?

There is no measured universal overhead percentage. P2PDMA describes peer DMA support and memory representation. DMA-BUF describes shared-memory ownership and attachment. They are not mutually exclusive: a DMA-BUF path can use Linux peer-DMA infrastructure too.

The vendor paths inspected here represent GPU BAR memory with device struct page metadata. The experimental importer can handle page-less DMA-BUF mappings and reuse device mappings while valid. That changes setup and metadata costs. It does not prove a throughput or CPU advantage; cuFile also provides batching and registered-buffer mechanisms.

Recorded comparisons; these are not matched production-serving benchmarks
Comparison	Measured result	Interpretation and limits
NVIDIA cuFile P2PDMA vs NIXL DMA-BUF, September 2026	On the same XFS file, 1 MiB I/O over a 1 GiB working set: cuFile best-thread write/read 4.284/9.975 GiB/s; two NIXL runs 4.28/9.66 and 4.27/9.77 GiB/s. Payload checks passed.	Near parity for this case. The wider size/concurrency matrix was mixed. gdsio threads and the NIXL batch-drain harness were not equivalent scheduling controls; the small working set can be cache resident. No valid steady-state CPU comparison was established.
AMD AIS runtime vs DMA-BUF, September 2026	Verified queue-depth-one transfers, 2–128 MiB. At 2 MiB, the initial DMA-BUF write/read result was 5763/5409 MiB/s versus AIS 1911/1621 MiB/s. At 128 MiB the paths were near parity.	This called ROCr AIS file read/write APIs directly, not the hipFile wrapper. It is not a concurrent hipFile benchmark or a ceiling for AIS. Repeated small-transfer results preserved the gap, but did not establish a library-wide cause.

The NVIDIA P2PDMA run worked on the experimental kernel with open GPU modules and the required BAR configuration, without nvidia-fs or a custom NVMe hook patch. An earlier diagnosis that it could not work there was corrected after the configuration was fixed.

The first AMD explanation blamed a 128 KiB command clamp versus 2 MiB commands. Equalizing that limit did not remove the throughput gap, so command size alone did not explain it. The queue-depth-one harness also does not establish what concurrent hipFile streams can achieve.

AMD documents approximately 1 GiB of host metadata per 64 GiB of VRAM registered for AIS in its driver setup guidance. That is metadata capacity, not a measured payload bounce or a fixed CPU tax per I/O. DMA-BUF has its own attachment, mapping and lifetime costs.

A useful overhead comparison must hold exporter, target, payload size, concurrency, verification and cache conditions constant. Measure setup and steady-state CPU time separately, plus latency and throughput. These records support the interoperability motivation; they do not support calling cuFile untested, treating the AIS result as a hipFile ceiling, or claiming that io_uring is always faster.

What has been tested?

Profile	Evidence	Limit
Latest LMCache MP, CUDA GPU direct GPU DMA-BUF	Per filesystem: 13 model continuations; 27 complete 8 MiB reads; cold L1; 9 capacity evictions.	One GPU, two live engines, two storage owners, same initialized XFS or ext4 file. Explicit Vulkan/RM exporter.
Latest LMCache staged baselines, CUDA GPU and HIP GPU	13 continuations and 27 reads each. HIP GPU uses engine-driven staging, shared host L1 and classic fixed buffers.	HIP GPU is a host-staged result, not AMD GPU-direct storage.
Native NIXL IO_URING, current dated source	Native backend units and physical XFS/ext4 host transfers pass. With the vLLM CUDA staging follow-up, ordinary host, fixed host and host DMA-BUF modes each pass 19 model continuations across fresh-reader, concurrent and pressure cases.	Model qualification uses two engines on one NVIDIA GPU, an initialized XFS file and VLLM_BATCH_INVARIANT=1. Host DMA-BUF staging does not establish GPU DMA-BUF export or cross-host operation. Use the vLLM staging fix linked in the P/D guide; the original be9c66a5 tip lacks it.
Historical NIXL XNVME_NVM + vLLM	Two-GPU host-staged and GPU-direct serving passed historical sequential, concurrent and pressure cases.	Different source revisions and backend; see the P/D evidence table .

For the latest LMCache cohorts: Llama-3.2-1B-Instruct revision 9213176726f574b556790deb65791e0c5aa438b6 , BF16, eager, greedy seed 123, batches 1/4/8, 257/513/769-token prompts and a bounded five-token copying task. Runs recorded on 15 October 2026 (PDT) using published LMCache 10a941b54 produce 52 matching continuations across direct XFS/ext4 and the CUDA/HIP staged profiles. Maximum chosen-token logprob delta is 0.004954 against the unchanged 0.02 bound. The MP GPU matrix passes all 32 profiles with independent values, native I/O receipts and zero terminal ownership. Filesystem GPU component suites also pass on ext4 and XFS. These checks do not qualify GPU raw-device I/O or model quality under lossy FP8.

An earlier staged phrase-recall run matched token IDs but exceeded the unchanged likelihood bound: 0.026329 > 0.02. Its cause is unresolved. The passing copy fixture does not establish broad numerical equivalence or a throughput advantage.

Corrections found during validation

Local tests found two runtime bugs. LMCache b8845f7c prevents repeated close from submitting cleanup to a stopped receiver event loop. The regression reproduces the timeout and coroutine warning on the base; all 33 asynchronous backend tests pass with the fix.

In the separate NIXL lane, vLLM 2697fd5ead registers the whole contiguous host staging arena once with CUDA. Copies crossing separately registered chunks previously failed with cudaErrorInvalidValue. Storage registrations retain their DMA-BUF boundaries. The CUDA regression, 52 adapter tests and 57 same-GPU model continuations pass; four adapter hardware gates remain explicit.

Use the follow-up branches when reproducing these results. The original feature refs are unchanged. The P/D correction details distinguish regression checks from model measurements; neither fix extends the hardware qualification above.

A common interface still needs a working platform route

DMA-BUF/io_uring can unify the storage interface across GPU exporters. It cannot create a missing PCIe peer route. Qualification has to establish platform routing, kernel admission, memory representation and application correctness separately.

Devices below one switch

Useful endpoint control. A passing switch-local route does not qualify routing through the Root Complex.

Devices below separate Root Ports

A separate platform contract. Sharing a PCI segment does not imply a shared Root Port. Crossing sockets or segments needs its own qualification.

PCIe 4.0 §1.3.1, Root Complex describes peer routing between Root Port hierarchies as optional and implementation dependent. It remains optional in the inspected revisions through PCIe 7.0 §1.3.1 . A faster PCIe link does not supply that guarantee. A host-bridge route can forward PCIe traffic without copying payloads through host RAM.

Contract	What must be established	What alone is insufficient
Platform routing	The exact port path supports the requested peer transactions.	A PCIe generation, segment label or same-switch pass.
Kernel admission	The running kernel and drivers admit and map the route under supported isolation settings.	An allowlist edit or a topology prediction.
Memory representation	The intended allocation exports and maps in GPU-local memory.	A GPU pointer, large BAR, DMA-BUF descriptor or aggregate memory counter.
Application correctness	Readiness, visibility, byte integrity and lifetime hold through checked completion and consumption.	Registration success or a full-length completion.

Linux PCI P2PDMA has topology and platform admission rules. The Linux admission history is distinct from the hardware route itself. A rejected policy is not a completed hardware test; removing that rejection does not make a route correct. Exporters can apply different policies. AMD can clear peer eligibility and migrate a shared allocation into host GTT; waiting for migration fixes backing readiness, not GPU-direct placement.

Storage direction is not PCIe transaction direction

Application operation	NVMe controller's GPU-facing transaction
Read storage into GPU memory	PCIe Memory Writes into GPU memory.
Write GPU memory to storage	PCIe Memory Reads from GPU memory, followed by Completions.

Recorded negative qualification: in one bounded cross-Root-Port test, an experimental admission change enabled VRAM resource mapping. The host-memory DMA-BUF control passed both ways. A guarded 4 KiB storage read into GPU memory passed, but the reverse storage write reported 4096 bytes completed while independent disk readback found all 0xff payload bytes. That pair failed qualification. The run did not isolate physical forwarding from DMA-address/IOMMU behavior, and it does not establish results for other devices or routes.

Qualify the claimed route and both directions

Use changing payloads, poisoned destinations, surrounding guards and independent readback. Compare host-memory and switch-local controls where available. Record the port path, memory placement, mapping decisions, firmware, kernel/driver versions and actual completions. Repeat buffer reuse and mapping changes before testing concurrent pairs. Preserve supported DMA isolation; blanket ACS overrides are not a substitute for platform support. Keep application I/O size, actual NVMe command size and PCIe packet size separate.

The kernel branch includes pci-peer-check.py and its usage and limits . Default mode predicts policy without storage transfers. Its source-pinned model does not automatically learn a modified kernel's admission table. The optional live-probe protocol needs a separately supplied reviewed helper. Follow with the LMCache payload and ordering tests ; a diagnostic pass is only the first gate.

Exporter enablement is separate from platform routing

The explicitly selected NVIDIA Vulkan/RM provider allocates device-local, non-host-visible memory, maps the same owned allocation into CUDA through OPAQUE_FD, and exports through the RM core DMA-BUF interface. It addresses native CUDA export refusal on the tested profile. It is version-coupled, does not convert arbitrary existing Torch allocations, and does not repair a peer route rejected by the platform or driver. See the provider design .

A template for future platform enablement

Arm SBSA 8.0 §1.2.18 and Appendix C GPU_02/RGPU_02, printed page 60 provides a useful template: within its GPU-compute scope, it requires peer reads and writes across endpoint hierarchies, including different Root Ports. This is a conditional platform requirement, not a guarantee for every Arm processor.

Future ecosystem enablement can pair a common Linux interface with explicit platform routing guarantees, documented port coverage, supported isolation and runnable integrity tests in both directions. The Arm requirement illustrates what a platform contract can add beyond the base PCIe specification. Already qualified pairs can use the interface while broader coverage grows.

A test ladder that avoids false positives

Prove target identity, initialized extents, alignment and kernel capability. Test a harmless owned file before a dedicated block target.
Write known payloads and independently read/hash them with a fresh reader. Check registration mode and actual native operation path.
Test finite submission pressure, short reads, partial submissions, cancellation and shutdown. A timeout is not proof DMA stopped.
Run the same no-connector model oracle and split/recompute control before comparing offload restoration. Cold L1 and payload-read counts must establish a real restore.
For P/D, verify READY/claim/adopt/restore/ACK ordering, eviction protection and strict failed-load rejection. Add warm prefixes, concurrency and reader/writer failure as separate cases.
Only then measure latency, throughput or host-memory savings against your existing NIXL baseline. Report device command traces separately from logical I/O counts.

A GDS-labelled arm must also establish that GPU-direct storage engaged. A cuFile POSIX compatibility fallback is a useful staged baseline, not a measurement of GPUDirect.