io_uring + DMA-BUF · Experimental · Updated 27 October 2026

GPU memory to storage

NVIDIA and AMD GPU memory can reach NVMe through the same experimental Linux storage interface. Native CUDA export, the Vulkan/RM workaround and ROCm export have each passed storage tests on specific configurations. The application validation differs by path.

The paths tested today

These are working experimental paths, rather than a promise of support on every GPU or motherboard. All use the patched Linux io_uring DMA-BUF importer and a qualified GPU ↔ NVMe route. The cards distinguish verified storage bytes from validation inside a serving application.

VERIFIED STORAGE I/O

NVIDIA: native CUDA export

OpenRM + CUDA DMA-BUF + io_uring

CUDA allocates the GPU memory and exports its DMA-BUF directly. The storage backend registers that buffer with io_uring.

LMCache provider:

rust_raw_block.gpu_buffer_provider: native

native is the default. Select this path when CUDA DMA-BUF export works; Vulkan is unnecessary.

The native CUDA/NIXL comparison passed verified GPU storage transfers. That does not give it the separate Vulkan/RM LMCache serving result.

Native provider details →
VERIFIED STORAGE + LMCACHE SERVING

NVIDIA: Vulkan/RM workaround

OpenRM + Vulkan allocation + RM DMA-BUF + io_uring

When native CUDA export is unavailable, LMCache can allocate through Vulkan, import the same allocation into CUDA, and export a separate DMA-BUF through NVIDIA RM.

Explicit LMCache provider:

rust_raw_block.gpu_buffer_provider: vulkan_rm

This path passed verified XFS/ext4 storage transfers and the recorded LMCache model-continuation checks. CUDA still runs the GPU work.

Serving evidence and limits →
VERIFIED STORAGE I/O

AMD: native ROCm export

amdgpu/KFD + ROCm DMA-BUF + io_uring

ROCm exports GPU memory through hsa_amd_portable_export_dmabuf. The storage backend uses the returned DMA-BUF and byte offset.

LMCache provider:

rust_raw_block.gpu_buffer_provider: native

Direct GPU storage transfers were verified in the low-level AMD comparison. The later HIP serving cohort used host staging; it does not validate AMD GPU-direct serving.

Exporter and transfer evidence →

OpenRM is the main NVIDIA kernel-driver route tested here: open-source, but outside mainline Linux. Both NVIDIA cards use it. The provider choice changes allocation and export, not the kernel-driver license. LMCache's setting belongs under extra_config; it selects the staging allocator, not vLLM's KV-cache allocator.

The native CUDA and AMD storage comparisons were recorded in September 2026, the NVIDIA helper storage checks on 7 October, and the LMCache serving cohorts on 15 October. See the full results and failed platform routes before extending a result to another configuration. These paths submit io_uring directly; they do not call cuFile or hipFile.

What still needs validation?

Export success and future driver work are not full storage qualification
CandidateRecorded statusNext requirement
Nouveau + NVKA 2 MiB Vulkan allocation exported as DMA-BUF.Validate NVMe attachment, VRAM placement and correct transfers in both directions.
NovaNo completed storage validation in this evaluation.Enable and validate the required DMA-BUF export/import path, then test GPU ↔ NVMe.
One standard Vulkan provider for both vendorsA possible common allocation/export interface. The current vulkan_rm helper is NVIDIA-specific.Implement and qualify standard Vulkan DMA-BUF export, NVMe peer mapping and CUDA/HIP interop where the application needs it.

Nouveau and Nova explained · Could Vulkan serve both vendors?

How the existing vendor routes differ

These library paths coexist with the experimental io_uring interface.

Vendor storage libraries and their kernel dependencies
RouteKernel integrationEvidence here
Traditional NVIDIA GDScuFile + nvidia-fs + compatible NVMe callback hooks. Older releases supported the proprietary GPU module; current nvidia-fs requires open modules.Historical architecture. The hooks are absent from stock mainline NVMe; compatibility depends on the supported stack, not simply whether a kernel is “new.”
NVIDIA GDS with P2PDMAOpenRM + cuFile + Linux P2PDMA. Supported configurations need neither nvidia-fs nor its custom NVMe hooks.Verified cuFile P2PDMA comparison. It is a separate storage path from io_uring DMA-BUF.
AMD Infinity StoragehipFile + AMD AIS integration + Linux P2PDMA. The inspected AIS additions are out of tree; base amdgpu/KFD and P2PDMA have upstream implementations.The comparison called the ROCr AIS APIs directly. It did not benchmark the hipFile wrapper.

Driver dependencies and primary sources · Comparison results and overhead limits

One storage interface, several GPU exporters

The goal is a vendor-neutral Linux path between GPU memory and NVMe. An application shares an allocation through a DMA-BUF file descriptor and submits storage reads or writes through io_uring. The GPU driver supplies the memory; the storage driver maps it for the NVMe device.

The storage interface can be common while GPU drivers and allocation APIs differ. CUDA, ROCm or Vulkan can supply memory through a suitable exporter. A DMA-BUF descriptor alone does not prove that the payload stays in GPU memory or that NVMe can access it.

Current statusExperimental kernel support, working integrations, and qualification for specific configurations. This is an upstreaming effort, not a feature already supplied by every stock Linux kernel or GPU driver.

Who does what?

Four separate responsibilities
LayerResponsibilityExamples
ApplicationManage cached data, gather/scatter GPU bytes, and wait for completion.vLLM with LMCache; a separate vLLM/NIXL storage connector.
Allocation and exportOwn GPU memory and share it with the storage importer.CUDA DMA-BUF; ROCm DMA-BUF; the experimental Vulkan/RM provider.
Storage submissionRegister a shared buffer and submit offsets, lengths and reads/writes.LMCache Rust backend or NIXL IO_URING plugin.
Kernel and platformMap the allocation for NVMe and execute the admitted peer transactions.Experimental io_uring DMA-BUF support, block/NVMe, GPU exporter and PCIe route.

The implementations here submit io_uring directly. They do not call cuFile or hipFile. Those libraries have their own storage paths and would need an explicit backend change to adopt this interface.

Choose a question

What to take away

OpenRM is an open-source Linux kernel driver, but it is out of tree. NVIDIA's current cuFile P2PDMA route uses the open kernel modules and can avoid nvidia-fs and its custom NVMe hooks. AMD's hipFile/AIS route also uses P2PDMA; the inspected AIS driver integration is out of tree. Neither route is the new io_uring DMA-BUF path.

Vulkan could provide a common allocation API. The current NVIDIA vulkan_rm provider still uses an NVIDIA-specific export helper. A portable Vulkan exporter for both vendors needs separate implementation and NVMe qualification.

The evidence does not establish a universal speed advantage. There are cuFile P2PDMA comparisons, AMD AIS runtime comparisons and application correctness tests. Their scopes differ. Export-only Nouveau tests and future Nova enablement are not full storage qualification.

October 2026 LMCache workshop

Luis Chamberlain presented this work in the technical session at the LMCache Meetup + Hands-on Workshop on 22 October 2026 in San Jose.

Open the talk slides →

The guide above spells out the validation scope behind the talk's path summary, including native CUDA export, the LMCache Vulkan/RM workaround, ROCm export and the remaining Nouveau/Nova work.

← knlpDrivers →