KVTide — tiered KV-cache I/O, measured.

Why key-value, why measure

Inference serving tiers its KV-cache: accelerator memory, host DRAM, local NVMe, then shared or remote storage. The interesting boundary is where addressing changes — block storage wants offsets, but a shared cache tier wants keys, and the NVMe Key-Value command set speaks that natively: Store, Retrieve, Exist, Delete against 16-byte keys instead of LBAs. Two very different host data paths can drive such a namespace:

• Userspace kernel-bypass — the SPDK NVMe driver connects to the NVMe-oF/TCP target directly; the kernel never sees the I/O.
• Kernel passthrough — the kernel nvme-tcp host attaches the target as a KV namespace char device (/dev/ngXnY) and userspace submits KV commands via io_uring_cmd (here through xNVMe).

The kernel path composes with everything the kernel gives you (scheduling, cgroups, observability, the block layer's buffer machinery); the bypass path buys raw speed. KVTide exists to quantify that trade both ways, reproducibly.

Architecture

             one guest, loopback TCP
 ┌────────────────────────────────────────────────┐
 │  SPDK nvmf_tgt (cores 0-1)                     │
 │    kvmalloc KV bdev  ←  NVMe-oF/TCP :4420      │
 └──────────────▲──────────────────▲──────────────┘
                │                  │
     userspace  │                  │  kernel nvme-tcp
     TCP socket │                  │  KV namespace: /dev/ngXnY
                │                  │
 ┌──────────────┴──────┐  ┌────────┴───────────────┐
 │ spdk_nvme_perf (KV) │  │ xnvme_kv_perf          │
 │ kernel-bypass       │  │ io_uring_cmd passthru  │
 └─────────────────────┘  └────────────────────────┘
        same initiator core budget, same
        op/value-size/queue-depth matrix, one CSV

The target is the SPDK kvmalloc KV bdev — an in-memory key-value store exported over NVMe-oF/TCP on loopback. CPU accounting sums busy jiffies on every core except the target's busy-polling reactor cores, so the initiator-side cost — including the kernel path's softirq and kworker time that process accounting would miss — is measured the same way for both.

Quick start

# in a knlp checkout, on the bench host (a QEMU guest is fine)
make defconfig-kvtide-ab
make

Plain make runs the staged pipeline; the target stays up afterwards so cells can be re-run, and make kvtide-target-down tears it down.

Four defconfigs pick the initiator shape: kvtide-ab (both — the A/B), kvtide-spdk, kvtide-xnvme, and kvtide-nixl, which adds the NIXL XNVME_KV plugin — a storage backend that moves buffers to and from KV namespaces over the same io_uring_cmd path — and runs its unit and integration tests against the same target. Every knob (pins, target address/NQN/core mask, the matrix) lives in the KVTide NVMe-KV bench harness Kconfig menu.

What the A/B shows

Representative cells from a validated run (QEMU guest, 8 vCPUs, Debian 6.12 kernel, loopback TCP, 8 s per cell):

The bypass initiator wins ~9× on raw small-op IOPS and ~5× on per-core efficiency; the gap compresses to ~2.6× once cells go bandwidth-bound. One disclosure on the recorded run: its driver passed the SPDK core selection as a bare number, which SPDK parses as a hex mask, so the spdk cells actually ran three workers — their full CPU cost (≈1.9 cores vs xnvme's ≈1.0) is in the CSV's init_cpu_cores column, which is what the per-core comparison divides by. The harness now pins spdk with the exact -c [N] core-list syntax, so fresh runs measure a single-worker spdk.

The 38-IOPS cell is real and reproducible: large-value stores at queue depth collapse on the kernel nvme-tcp solicited-write (R2T) path when writes exceed the target's in-capsule data size (a harness knob, KVTIDE_TARGET_INCAPSULE) — the userspace initiator never takes that path. That cell is exactly why the harness exists: it is a kernel-side data-path problem you can now regress, bisect, and fix with a one-command reproducer.

Kernel A/B with kdevops

knlp ships as a kdevops plugin, and the plugin can provision every KVTide build dependency. That makes the guest a disposable, reproducible bench node — and because kdevops also builds and boots kernels, you can put a kernel of choice under the xNVMe initiator and A/B the kernel side too:

# kdevops side: provision a guest with knlp + KVTide deps
make kdevops-plugin-add URL=https://github.com/mcgrof/knlp

# kdevops resolves +fragments from ~/.config/kdevops/defconfigs/configs
mkdir -p ~/.config/kdevops/defconfigs/configs
cp ~/.config/kdevops/plugins/knlp/defconfigs/configs/knlp-kvtide.config \
   ~/.config/kdevops/defconfigs/configs/

make defconfig-<your-base>+knlp-kvtide
make
make bringup

# on the guest: knlp is already cloned at the data path
cd /data/knlp
make defconfig-kvtide-ab
make

kdevops carries block-layer buffer-pool validation tooling of its own — the blk_iobuf_pool RFC page, defconfig-iobuf-nvme and the iobuf_bench scripts — so one workflow covers both layers: boot the kernel branch under test, run the KVTide matrix on top, diff the CSVs. The xNVMe initiator's passthrough commands exercise exactly the kernel paths that block-layer work touches; the userspace initiator doubles as the control arm that skips the kernel entirely.

the kernel under test, one command

The harness can also carry the kernel itself. defconfig-kvtide-ab-linux adds a kvtide-linux stage that fetches a Linux tree (shallow), bases its config on the running kernel's, enables CONFIG_BLK_IOBUF_POOL, builds, installs, and then gates the pipeline on actually running it — one reboot, and the second make benches on the kernel under test. Every bench CSV gets a .meta sidecar recording uname -r, so runs on different kernels stay distinguishable.

# default kernel under test: the blk_iobuf_pool v3 series from
# git.kernel.org/pub/scm/linux/kernel/git/mcgrof/linux.git
make defconfig-kvtide-ab-linux
make            # builds target + initiators + kernel, installs, asks for ONE reboot
# reboot into the listed kernel, then:
make            # gate passes; bench runs on the kernel under test

# any other tree/branch, as environment variables at defconfig time:
make defconfig-kvtide-ab-linux LINUX_TREE=<git url> LINUX_BRANCH=<ref>

The LINUX_TREE / LINUX_BRANCH overrides resolve when the defconfig loads (they need python3 kconfiglib, which fills the deliberately omitted symbols); the defaults point at mcgrof/linux blk-iobuf-pool-v3. The stage installs a kernel and updates the bootloader — use it on disposable bench hosts, which is exactly what a kdevops guest is.

bare metal, as an option

Both modes are validated: the QEMU guest flow, and the full bare-metal flow on a rented 12-core box (Ubuntu 24.04, two datacenter NVMe drives) — kernel built and booted from the default blk-iobuf-pool-v3 branch, one reboot, clean 24-cell A/B on the kernel under test. The perf harness only assumes enough cores to keep the target mask and the initiator pin disjoint. Two bare-metal routes:

# via kdevops: DECLARED_HOSTS flips any config from guest bringup to
# existing machines at defconfig time (SKIP_BRINGUP + declared hosts;
# the knlp plugin's inventory hook respects declared-hosts mode)
make defconfig-<your-base>+knlp-kvtide DECLARED_HOSTS="metal1 metal2"
make
make bringup

# or directly, in a knlp clone on the machine
make defconfig-kvtide-ab && make

The kernel stage is deliberately the exception: it installs a kernel and updates the bootloader, so it refuses to run where systemd-detect-virt reports no virtualization. Testing a kernel on real hardware is an explicit opt-in — make defconfig-kvtide-ab-linux-baremetal (the same LINUX_TREE/LINUX_BRANCH overrides apply), mirroring kdevops' own defconfig-iobuf-baremetal precedent for bare-metal block-layer testing.

Reproducibility pins

The SPDK KV command set support (the kvmalloc bdev, NVMe-oF KV namespaces and spdk_nvme_perf KV mode) is pending on SPDK Gerrit; a single change ref carries the whole stack plus its base. A small vendored patch fixes a GCC 15/AVX10 build probe and an RPC decoder collision until those land upstream. The harness README (tools/reproduce/kvtide/README.md) documents every default.

References

• NVMe Key-Value command set — NVM Express specifications
• SPDK KV target stack — review.spdk.io change 28307
• xNVMe — xnvme.io
• NIXL XNVME_KV plugin branch — mcgrof/nixl @ 20260717-xnvme-kv
• blk_iobuf_pool kernel branch — mcgrof/linux @ blk-iobuf-pool-v3
• blk_iobuf_pool RFC + tooling in kdevops — RFC page, iobuf_bench
• Running knlp with kdevops — the kdevops integration page
• kdevops — github.com/linux-kdevops/kdevops

← All knlp research