KVTide — tiered KV-cache I/O, measured.

Why key-value, why measure

Inference serving tiers its KV-cache: accelerator memory, host DRAM, local NVMe, then shared or remote storage. The interesting boundary is where addressing changes — block storage wants offsets, but a shared cache tier wants keys, and the NVMe Key-Value command set speaks that natively: Store, Retrieve, Exist, Delete against 16-byte keys instead of LBAs. Two very different host data paths can drive such a namespace:

• Userspace kernel-bypass — the SPDK NVMe driver connects to the NVMe-oF/TCP target directly; the kernel never sees the I/O.
• Kernel passthrough — the kernel nvme-tcp host attaches the target as a KV namespace char device (/dev/ngXnY) and userspace submits KV commands via io_uring_cmd (here through xNVMe).

The kernel path composes with everything the kernel gives you (scheduling, cgroups, observability, the block layer's buffer machinery); the bypass path buys raw speed. KVTide exists to quantify that trade both ways, reproducibly.

Architecture

             one guest, loopback TCP
 ┌────────────────────────────────────────────────┐
 │  SPDK nvmf_tgt (cores 0-1)                     │
 │    kvmalloc KV bdev  ←  NVMe-oF/TCP :4420      │
 └──────────────▲──────────────────▲──────────────┘
                │                  │
     userspace  │                  │  kernel nvme-tcp
     TCP socket │                  │  KV namespace: /dev/ngXnY
                │                  │
 ┌──────────────┴──────┐  ┌────────┴───────────────┐
 │ spdk_nvme_perf (KV) │  │ xnvme_kv_perf          │
 │ kernel-bypass       │  │ io_uring_cmd passthru  │
 └─────────────────────┘  └────────────────────────┘
        same initiator core budget, same
        op/value-size/queue-depth matrix, one CSV

The target is the SPDK kvmalloc KV bdev — an in-memory key-value store exported over NVMe-oF/TCP on loopback. CPU accounting sums busy jiffies on every core except the target's busy-polling reactor cores, so the initiator-side cost — including the kernel path's softirq and kworker time that process accounting would miss — is measured the same way for both.

Compare physical NVMe read paths

Run the physical mode on the same unused controllers under one translating-IOMMU boot. KVTide rebinds those controllers between the userspace and kernel drivers, records the exact xNVMe and SPDK commits, and returns the controllers to Linux when it exits. It refuses mounted namespaces, swap devices, block holders, and process-open devices. You must still verify every PCI address before enabling the explicit rebind option.

A translated-IOMMU run binds the test controllers to vfio-pci. KVTide uses vfio device cdevs through iommufd when the host exposes them. It also tries to load the legacy vfio_iommu_type1 transport explicitly for hosts without cdevs; loading vfio-pci alone does not make that transport available to xNVMe. Set KVTIDE_PCIE_XNVME_VFIO_MODE to auto, iommufd, or type1. KVTide records the selection so an explicit type1 workaround cannot be mistaken for an iommufd result.

Read the ratios as full-stack outcomes. SPDK and premap use different benchmark programs. Use the upcie-versus-linux pair for the closest same-tool comparison. The harness reports IOPS, MiB/s, failed commands, and global busy CPU cores. It does not report a compatible latency distribution, so do not make tail-latency claims from these runs. Each benchmark program generates its own random offsets; cells match a distribution and its controls, not an identical ordered request stream.

run both synthetic questions

• AISIO command rate: run random 512-byte, 4 KiB, and 8 KiB reads at queue depth 128 while scaling from one to four host threads. A 512-byte cell runs only when the namespace has 512-byte logical blocks. Matching the published platform also requires its 16 Samsung PM1753 SSDs, H100 system, and PCIe topology; the profile only supplies its request controls.
• KV crossover: sweep random reads from 4 KiB through 2 MiB at queue depths 1, 16, 64, and 128. Use it to locate command-rate, DMA-mapping, MDTS, and bandwidth limits. Do not call this a production KV-cache workload; add trace-derived profiles only after preserving sizes, offsets, order, pacing, and concurrency.

Before recording premap results, KVTide must complete a read one logical block larger than Linux's ordinary queue limit. This catches the kernel allocation's allowed fallback to an ordinary registered buffer. Crossing that limit proves this run used the retained mapping and the separate premapped request limit.

Quick start

# in a knlp checkout, on the bench host (a QEMU guest is fine)
make defconfig-kvtide-ab
make

Plain make runs the staged pipeline; the target stays up afterwards so cells can be re-run, and make kvtide-target-down tears it down.

Four defconfigs pick the initiator shape: kvtide-ab (both — the A/B), kvtide-spdk, kvtide-xnvme, and kvtide-nixl, which adds the NIXL XNVME_KV plugin — a storage backend that moves buffers to and from KV namespaces over the same io_uring_cmd path — and runs its unit and integration tests against the same target. Every knob (pins, target address/NQN/core mask, the matrix) lives in the KVTide storage bench harness Kconfig menu.

run physical PCIe

# boot the premap kernel with a translating IOMMU first
make defconfig-kvtide-pcie
echo 'CONFIG_KVTIDE_PCIE_BDFS="0000:41:00.0 0000:42:00.0 0000:43:00.0 0000:44:00.0"' >> .config
echo 'CONFIG_KVTIDE_PCIE_ALLOW_REBIND=y' >> .config
make

Use defconfig-kvtide-pcie-linux-baremetal to build and install the public premap kernel before the run. The first make installs the kernel and required GRUB command line, then stops for a reboot. The next make runs the comparison. Use defconfig-kvtide-pcie-aisio only to reproduce the published uPCIe no-IOMMU setup; that boot cannot evaluate retained-IOMMU mappings.

What the A/B shows

Representative cells from a validated run (QEMU guest, 8 vCPUs, Debian 6.12 kernel, loopback TCP, 8 s per cell):

The bypass initiator wins ~9× on raw small-op IOPS and ~5× on per-core efficiency; the gap compresses to ~2.6× once cells go bandwidth-bound. One disclosure on the recorded run: its driver passed the SPDK core selection as a bare number, which SPDK parses as a hex mask, so the spdk cells actually ran three workers — their full CPU cost (≈1.9 cores vs xnvme's ≈1.0) is in the CSV's init_cpu_cores column, which is what the per-core comparison divides by. The harness now pins spdk with the exact -c [N] core-list syntax, so fresh runs measure a single-worker spdk.

The 38-IOPS cell is real and reproducible: large-value stores at queue depth collapse on the kernel nvme-tcp solicited-write (R2T) path when writes exceed the target's in-capsule data size (a harness knob, KVTIDE_TARGET_INCAPSULE) — the userspace initiator never takes that path. That cell is exactly why the harness exists: it is a kernel-side data-path problem you can now regress, bisect, and fix with a one-command reproducer.

Kernel A/B with kdevops

knlp ships as a kdevops plugin, and the plugin can provision every KVTide build dependency. That makes the guest a disposable, reproducible bench node — and because kdevops also builds and boots kernels, you can put a kernel of choice under the xNVMe initiator and A/B the kernel side too:

# kdevops side: provision a guest with knlp + KVTide deps
make kdevops-plugin-add URL=https://github.com/mcgrof/knlp

# kdevops resolves +fragments from ~/.config/kdevops/defconfigs/configs
mkdir -p ~/.config/kdevops/defconfigs/configs
cp ~/.config/kdevops/plugins/knlp/defconfigs/configs/knlp-kvtide.config \
   ~/.config/kdevops/defconfigs/configs/

make defconfig-<your-base>+knlp-kvtide
make
make bringup

# on the guest: knlp is already cloned at the data path
cd /data/knlp
make defconfig-kvtide-ab
make

kdevops carries block-layer buffer-pool validation tooling of its own — the blk_iobuf_pool RFC page, defconfig-iobuf-nvme and the iobuf_bench scripts — so one workflow covers both layers: boot the kernel branch under test, run the KVTide matrix on top, diff the CSVs. The xNVMe initiator's passthrough commands exercise exactly the kernel paths that block-layer work touches; the userspace initiator doubles as the control arm that skips the kernel entirely.

Do not use QEMU throughput as a stand-in for physical PCIe. Stage knlp-kvtide-pcie.config and use kdevops declared-hosts mode for the storage node. Enable the uPCIe-CUDA option only when the NVIDIA driver and CUDA toolkit are already present. The plugin installs the uPCIe v0.8.0 dmabuf-import and iommu-map-pa DKMS packages; it does not install CUDA. Then run a physical KVTide defconfig in the provisioned knlp checkout.

cp ~/.config/kdevops/plugins/knlp/defconfigs/configs/knlp-kvtide-pcie.config \
   ~/.config/kdevops/defconfigs/configs/
make defconfig-<your-base>+knlp-kvtide-pcie \
    DECLARED_HOSTS="storage-node"
make && make bringup

the kernel under test, one command

The harness can also carry the kernel itself. defconfig-kvtide-ab-linux adds a kvtide-linux stage that fetches a Linux tree (shallow), bases its config on the running kernel's, enables CONFIG_BLK_IOBUF_POOL, builds, installs, and then gates the pipeline on actually running it — one reboot, and the second make benches on the kernel under test. Every bench CSV gets a .meta sidecar recording uname -r, so runs on different kernels stay distinguishable.

# default kernel under test: the blk_iobuf_pool v3 series from
# git.kernel.org/pub/scm/linux/kernel/git/mcgrof/linux.git
make defconfig-kvtide-ab-linux
make            # builds target + initiators + kernel, installs, asks for ONE reboot
# reboot into the listed kernel, then:
make            # gate passes; bench runs on the kernel under test

# any other tree/branch, as environment variables at defconfig time:
make defconfig-kvtide-ab-linux LINUX_TREE=<git url> LINUX_BRANCH=<ref>

The LINUX_TREE / LINUX_BRANCH overrides resolve when the defconfig loads (they need python3 kconfiglib, which fills the deliberately omitted symbols); the defaults point at mcgrof/linux blk-iobuf-pool-v3. The stage installs a kernel and updates the bootloader — use it on disposable bench hosts, which is exactly what a kdevops guest is.

bare metal, as an option

Both modes are validated: the QEMU guest flow, and the full bare-metal flow on a rented 12-core box (Ubuntu 24.04, two datacenter NVMe drives) — kernel built and booted from the default blk-iobuf-pool-v3 branch, one reboot, clean 24-cell A/B on the kernel under test. The perf harness only assumes enough cores to keep the target mask and the initiator pin disjoint. Two bare-metal routes:

# via kdevops: DECLARED_HOSTS flips any config from guest bringup to
# existing machines at defconfig time (SKIP_BRINGUP + declared hosts;
# the knlp plugin's inventory hook respects declared-hosts mode)
make defconfig-<your-base>+knlp-kvtide DECLARED_HOSTS="metal1 metal2"
make
make bringup

# or directly, in a knlp clone on the machine
make defconfig-kvtide-ab && make

The kernel stage is deliberately the exception: it installs a kernel and updates the bootloader, so it refuses to run where systemd-detect-virt reports no virtualization. Testing a kernel on real hardware is an explicit opt-in — make defconfig-kvtide-ab-linux-baremetal (the same LINUX_TREE/LINUX_BRANCH overrides apply), mirroring kdevops' own defconfig-iobuf-baremetal precedent for bare-metal block-layer testing.

Reproduce the source state

The SPDK KV command set support (the kvmalloc bdev, NVMe-oF KV namespaces and spdk_nvme_perf KV mode) is pending on SPDK Gerrit; a single change ref carries the whole stack plus its base. A small vendored patch fixes a GCC 15/AVX10 build probe and an RPC decoder collision until those land upstream. The harness README (tools/reproduce/kvtide/README.md) documents every default.

Each physical fetch refreshes a clean checkout to the current configured upstream branch and refuses tracked local changes. Each run records the resulting commit IDs in run.meta, so later branch movement does not hide what produced the CSV. Pin commits in Kconfig to freeze a campaign. The physical harness has passed configuration, compilation, and static validation but has not yet produced a real-device performance result. Check zero failed commands and repeat every surprising crossover before publishing a conclusion.

References

• NVMe Key-Value command set — NVM Express specifications
• SPDK KV target stack — review.spdk.io change 28307
• xNVMe — xnvme.io
• AISIO published setup — xnvme.io/aisio
• SPDK — github.com/spdk/spdk
• NIXL XNVME_KV plugin branch — mcgrof/nixl @ 20260717-xnvme-kv
• premapped blk_iobuf kernel branch — mcgrof/linux @ blk-iobuf-pool-v5-premap-iova
• blk_iobuf_pool RFC + tooling in kdevops — RFC page, iobuf_bench
• Running knlp with kdevops — the kdevops integration page
• kdevops — github.com/linux-kdevops/kdevops

← All knlp research