Linux support and reproduction

A common Linux storage interface

The goal is shared upstream infrastructure beneath existing userspace APIs, reducing bespoke out-of-tree storage integration. DMA-BUF is established; the new work is DMA-BUF-backed io_uring read/write support . Different GPU drivers can export allocations into the same storage interface while retaining their own export and peer-access rules. cuFile and hipFile could adopt this transport beneath existing APIs; this implementation uses io_uring directly and does not implement that library change.

The driver comparison separates traditional nvidia-fs, cuFile P2PDMA and hipFile/AIS from this interface. Upstream P2PDMA and DMA-BUF infrastructure already exist; the storage integration described here remains experimental.

The experimental dmabuf-size-ceiling-v7.3-rc3 kernel branch is pinned at 856b6583ab2f4581e81670845eadb69de7a68b30 . The cited upstream proposal covers NVMe block devices. This branch adds iomap, ext4 and XFS direct I/O, a separate DMA-BUF request ceiling, an optional DMA-optimal clamp lift and a 32 MiB driver cap. Ordinary I/O still traverses filesystem extent handling where applicable, the block layer and NVMe PCI driver.

Byte-verifying read/write selftests exercise larger requests. Actual command sizes remain bounded by controller Maximum Data Transfer Size (MDTS), mapping segments, queue limits and filesystem extents. Registration creates a context; first use may construct the reusable device mapping. A large io_uring operation does not guarantee one NVMe command. Initialized aligned XFS/ext4 files and raw NVMe are distinct test targets, not support for every inode, block driver or exporter.

Why Linux keeps a P2PDMA allowlist

PCIe 4.0 §1.3.1 already makes forwarding between Root Port hierarchies optional; that remains true through the inspected PCIe 7.0 revision. Linux therefore cannot treat a Root Complex as a switch that always forwards peer traffic. It admits well-defined switch-local routes and uses platform knowledge for routes through a host bridge. The source calls the table pci_p2pdma_whitelist ; this guide calls it the allowlist.

The commits below trace the main admission rules in the pinned kernel. Dates are commit dates. Each link opens the full change in Torvalds’ tree, including its original rationale and exact entries.

1. 2019-05-01 0f97da831026

   Known-working Root Complexes become explicit exceptions

   Adds the first AMD host-bridge exception when no common upstream bridge exists. The rationale cites PCIe 4.0 §1.3.1: cross-Root-Port forwarding is optional. The exception admits known support instead of assuming every Root Complex routes peers.

2. 2019-06-19 6dbbd053e6ae

   Missing IOMMU mappings temporarily block the exception

   Rejects host-bridge exceptions while an IOMMU is present because the kernel could not yet map these peer addresses correctly. Hardware routing capability alone did not make the software mapping safe.

3. 2019-08-16 494d63b0d5d0

   Tested Intel entries gain a same-host-bridge limit

   Introduces the table structure and REQ_SAME_HOST_BRIDGE for tested Intel bridges. Its rationale identifies poor performance across distinct host bridges via the CPU interconnect. The flag compares Linux pci_host_bridge objects; it does not require the two endpoints to share one Root Port.

4. 2019-08-16 5daf40b03924

   Correct DMA mappings remove the temporary IOMMU ban

   Removes the earlier iommu_present() restriction after the requests are mapped correctly. This is why the temporary 2019 guard is not advice to disable a current IOMMU. Supported translation and mapping still require qualification.

5. 2019-12-10 bc123a515cb7

   Another tested Intel platform enters the table

   Adds host-bridge entries after successful peer transactions spanning host and PCI bridges with the IOMMU enabled. These entries have no same-host-bridge flag. That is an explicit policy choice for those entries, not a rule for all Intel systems.

6. 2020-03-18 7b94b53db34f

   Additional Root Ports receive tested coverage

   Adds three remaining Intel Root Port entries for the platform covered by the preceding change, after reported P2P tests. Enabling one port identifier did not automatically identify the other ports.

7. 2020-07-30 dea286bb71ba

   AMD admission becomes a CPU-family rule

   Replaces individual AMD bridge entries with an AMD CPU-family allowance. Linux can encode a documented family-level routing property instead of maintaining one bridge entry per product. Admission is therefore broader than an exact device-ID table on this path.

8. 2022-02-25 feaea1fe8b36

   A newer Intel server bridge is recognized

   Adds a bridge entry for another supported Intel server generation. It extends the recognized platform set; it does not infer capability from PCIe link generation or add a generic forwarding mechanism.

9. 2022-04-11 1af7c26c59eb

   Root Port discovery works beyond device zero

   Checks the first device on the root bus when it is either device/function zero or a PCIe Root Port. Previously, an allowed Root Port at a nonzero device/function could be missed. This repairs recognition of an existing allowance, not physical routing.

10. 2026-04-10 1a1f96e3e2a8

    The table can express vendor-wide platform support

    Adds wildcard device matching while retaining the vendor match and same-host-bridge constraint. This permits a platform-wide contract to be represented without adding every device identifier separately.

11. 2026-04-10 70cf146a674c

    Google SoCs use that wildcard mechanism

    Adds the vendor-wide SoC rule, with the commit stating that these SoCs support peer transactions between Root Ports. This records a vendor platform guarantee rather than making the PCIe base requirement universal.

12. 2026-06-09 0ba76b19fd4c

    Accelerator root buses gain their own recognition

    Adds Intel crypto, data-streaming and analytics accelerator entries because such an accelerator may be the first device on its own root bus. The commit discusses future ACPI HMAT capability reporting to avoid continual table growth; it does not implement that discovery mechanism.

13. 2026-07-22 894a78088fc3

    NVIDIA CPU platforms gain explicit entries

    Adds NVIDIA CPU-platform identifiers for supported peer transactions. These are host-platform allowances, not a blanket approval of every NVIDIA GPU, exporter or GPU/NVMe pair.

See Linux’s P2PDMA documentation for topology and lifetime rules. Table entries, CPU-family rules and exporter-specific decisions must be checked against the running kernel; a policy prediction is not a transfer result.

Build and test the pinned enablement stack

The reproduction profile includes defconfigs/io-uring-dmabuf and tools/reproduce/dmabuf/README.md . The profile pins the kernel, LMCache, NIXL and vLLM revisions used by this review. Plain make prints the plan; fetching, builds, kernel installation and hardware tests are explicit stages.

make defconfig-io-uring-dmabuf
make
make dmabuf-fetch
make dmabuf-kernel-config
make dmabuf-kernel-build
# Check drivers and signing.
make dmabuf-kernel-install
# Reboot into the test kernel.
make dmabuf-doctor

The kernel configuration starts from the running machine's config, preserving its platform drivers. It adds the required I/O features and uses a distinct release suffix. The runner issues no reboot or boot-selection commands and does not alter peer policy or ACS/IOMMU settings. Distribution install hooks can update the boot menu; verify its default and keep the known-good kernel available.

Set CONFIG_KNLP_DMABUF_SCRATCH to an existing owned directory on NVMe-backed XFS/ext4. In a dedicated environment with matching Torch, compiler and GPU SDK dependencies, build LMCache and its Rust extension with make dmabuf-lmcache-install . Use the explicit targets below after coordinating GPU ownership where needed.

Logs and JUnit reports remain in the configured work directory; skipped tests are not counted as qualification. These stages do not automate full model-serving MP P/D, cross-host shared storage or performance qualification. NIXL and vLLM sources are pinned and fetched; their builds and serving orchestration follow the linked upstream/source guides. Keep your current NIXL baseline separate.

Run the P/D contracts separately from hardware qualification

make defconfig-pd-storage
make pd-test-plan
make pd-lmcache-test PYTHON=/path/to/lmcache-venv/bin/python
make pd-mp-test PYTHON=/path/to/lmcache-venv/bin/python
make pd-vllm-test PYTHON=/path/to/vllm-venv/bin/python
make pd-timeline-check PD_TIMELINE=/path/to/serving.jsonl

Fetch the pinned sources and prepare each application environment first. pd-cpu-test runs the three CPU selections in order when one environment can import both pinned projects. Unexpected skips fail qualification. The timeline target checks an existing in-process trace, with dmabuf_fixed required by default; it does not validate MP HTTP traces or launch model-serving P/D. See tools/reproduce/dmabuf/README.md for dependencies and scope.

Source and interfaces

• LMCache full dated stack: 50 commits · MP configuration and HTTP contract
• NIXL native IO_URING backend: 2 commits · Backend parameters and limits
• vLLM NIXL storage connector: 3 commits
• Experimental kernel branch , pin 856b6583ab2f4581e81670845eadb69de7a68b30

Follow-up branches include their full feature stacks: LMCache b8845f7c, NIXL 711dcb4d and vLLM 2697fd5ead. NIXL adds a GCC15 test-header fix and documents initialized read extents and locked-memory requirements; it makes no production transfer change.

Kernel DMA-BUF sharing and synchronization documentation describes the ownership interface; it does not establish that this experimental storage ABI is upstream. The dated userspace branches retain their original upstream bases.