Allocation, export and userspace

What changed in userspace?

The implemented storage backend submits io_uring itself. LMCache's Rust raw-block backend owns the storage transfer path; its GPU buffer provider supplies the staging allocation and exporter. The vLLM connector coordinates gathering key/value (KV) data into that pool and scattering restored data back into engine memory.

There is no cuFile or hipFile modification in this implementation, and those libraries are not called by this path. Adding kernel DMA-BUF support does not automatically redirect their existing I/O. They could retain their public APIs and implement a new backend, but that is separate work.

NIXL offers a second integration through its IO_URING plugin. Its consumer supplies an owned DMA-BUF registration and requests transfers through NIXL. LMCache direct storage does not require NIXL.

What changes in the I/O path?

A conventional GPU offload path may copy KV into host DRAM before storage. The direct profile gathers engine KV into an exported GPU staging arena, then lets the storage driver DMA that payload. Host code still submits I/O and handles metadata; GPU gather/scatter copies remain.

The first I/O may construct the device mapping. Later requests reuse it while valid; invalidation still matters. “Map once” does not mean a mapping can never change.

The LMCache and NIXL profiles described here use ordinary fixed-buffer reads and writes. The pinned experimental kernel also enables registered DMA-BUF buffers for NVMe io_uring_cmd passthrough through its fixed-buffer import change and block-request mapping change . Host-exporter tests in an emulated NVMe guest validate that kernel path. GPU exporters and application-level passthrough still require separate qualification. A filesystem also needs a capable underlying importer; putting ext4 on a virtio disk does not supply NVMe DMA-BUF support.

The allocation and export choices

Provider choice is separate from kernel route admission and from application correctness. The LMCache DMA-BUF guide describes the allocation ownership, export offsets and ordering contract.

When native CUDA or ROCm export works

extra_config:
  rust_raw_block.gpu_buffer_provider: native

native is the default, so the provider setting can be omitted. On NVIDIA, LMCache uses CUDA memory and native CUDA DMA-BUF export through cuMemGetHandleForAddressRange. On AMD, the native provider uses ROCm export through hsa_amd_portable_export_dmabuf.

NVIDIA's native path still uses the open GPU kernel modules (OpenRM). Both native and vulkan_rm use that kernel-driver stack in the configurations discussed here. The choice is how to allocate and export LMCache's GPU staging memory. Native export requires neither Vulkan allocation nor the custom RM export helper.

These snippets select a provider within a configured raw-block backend; they are not complete launch configurations. See the LMCache configuration guide and the tested-path summary.

What does vulkan_rm change?

extra_config:
  rust_raw_block.gpu_buffer_provider: vulkan_rm

This selects LMCache's external GPU staging allocator. It does not replace vLLM's KV-cache allocator or convert arbitrary existing Torch allocations into exportable buffers. The engine keeps its own KV memory; the connector copies between that memory and the staging pool.

1. Vulkan allocates device-local, non-host-visible memory.
2. The helper exports an OPAQUE_FD handle and imports that same allocation into CUDA. Sharing the allocation this way does not copy its payload.
3. A separate NVIDIA RM core operation exports a DMA-BUF handle suitable for the tested storage importer.
4. The Rust backend registers that DMA-BUF with io_uring and submits storage operations after GPU readiness has been established.

OPAQUE_FD and a Linux DMA-BUF are different external-handle types. The CUDA interop handle is not simply passed to the storage importer as if it were a DMA-BUF. The current helper depends on NVIDIA RM interfaces; the documented implementation is coupled to driver 615.71.09 and has bounded allocation limits. See the provider design and restrictions.

This allocator change belongs to LMCache and its helper. A separate vLLM/NIXL storage connector is also part of the project, but it is not the component selected by this LMCache setting.

Could Vulkan support both AMD and NVIDIA?

Yes, Vulkan can be the common allocation API, if both driver stacks provide the required export and peer-access behavior. Vulkan is an API implemented by a userspace driver. That implementation talks to a kernel GPU driver; Vulkan itself is not a Linux kernel DRM driver. Mesa RADV and NVK are examples of userspace Vulkan drivers.

The standard VK_EXT_external_memory_dma_buf extension permits Vulkan memory import/export using DMA-BUF. An implementation must support the requested external handle, allocation type and buffer usage. The application then needs to show that NVMe can attach and map the allocation without host migration, and that writes are visible to the consuming GPU workload.

A standalone Vulkan fill/check test can exercise GPU ↔ NVMe without CUDA or HIP. Existing CUDA/HIP inference applications still need compatible memory interop and synchronization, or a different compute implementation. Using Vulkan for allocation does not by itself replace those runtimes.

The present vulkan_rm provider is one NVIDIA implementation of that allocation strategy with a custom export step. It is not evidence that one standard Vulkan exporter has already qualified both vendors.

What work remains on the CPU and GPU?

CPU code still registers buffers, prepares requests, processes completions and manages metadata. The GPU still gathers KV into staging memory and scatters restored data back into engine allocations. The direct path aims to remove host-DRAM payload staging, not every copy or every CPU operation.

The allocation owner and exported descriptor must outlive outstanding I/O. GPU production must complete before NVMe reads that memory; storage completion and the required visibility synchronization must precede GPU consumption. Reuse, invalidation, cancellation and shutdown are part of the contract. The test ladder checks these properties separately from registration success.

Two independent ways to adopt it

The older XNVME_NVM plugin is retained as a historical reproduction branch. It is distinct from the newer standalone IO_URING plugin and from the older NVMe key-value XNVME_KV work. The standalone plugin uses liburing directly; no new xNVMe port is included.

A transition plan for existing NIXL users

1. Keep your current baseline. Record backend, model weights, layouts, chunk/block geometry, token IDs, chosen-token logprobs and transfer counters. Preserve the current network or GDS configuration for rollback.
2. Qualify ordinary storage first. Use the dated plugin and matching API-v2 Python binding. The historical nixl==1.5.0 API-v1 wheel is incompatible with this pinned plugin. Begin with buffer_mode=host on a fresh, initialized scratch file. Verify independent readback before serving.
3. Measure classic registration separately. Select buffer_mode=fixed . Keep prompts, target, lengths and concurrency unchanged. Confirm the fixed path actually ran.
4. Introduce DMA-BUF deliberately. Boot the experimental kernel, verify exporter/importer support, then select buffer_mode=dmabuf and register an owned export using metaInfo="dmabuf:<fd>" . This mode requires every memory registration on that engine to carry a DMA-BUF. Use separate engines if ordinary host buffers are also needed.
5. Choose the higher layer. Stay with the separate vLLM/NIXL connector for a NIXL storage experiment, or use the direct LMCache backend for LMCache cache management. This is not a switch that turns stock NixlConnector network transfers into storage P/D.
6. Roll back without sharing formats. Restore the original connector/backend configuration. LMCache and NIXL storage metadata are not interchangeable; use isolated targets and migrate/recompute cache contents explicitly.

backend = "IO_URING"
params = {"device_name": "/mnt/scratch/kv.bin",
          "buffer_mode": "host", "direct": "true"}
# Then fixed; only then dmabuf after exporter/kernel qualification.
# BLK_SEG describes offsets and lengths on this already initialized target.
# A DMA-BUF memory registration adds metaInfo="dmabuf:<fd>".