Research tracker · storage workloads

Large CPU-initiated I/O
in AI workflows

Checkpoints, optimizer state, spilled objects, model weights and cached encodings already move substantial payloads through host DRAM. This page maps the implementations where that path is established and identifies the evidence for transfers larger than 2 MiB.

Here, CPU-initiated means host software submits the storage operation. The payload passes through host memory; NVMe DMA can move it without the CPU copying every byte. CPU-submitted GPU-direct I/O is a different path and is outside this page’s main workload set.

Source review: 27 October 2026 · Companion: Small random reads in AI

32–64 MBCheckpoint staging buffersBest configurations for FastPersist’s 512 MB transfer experiment; sizes use the paper’s units. See evidence
1–128 MiBRay Data block-size targetHost-memory objects that can spill to disk. This range includes objects below 2 MiB. See evidence
34.3 / 130.8 MBVideo encoder-cache entriesReported by LMCache’s in-process benchmark; entry sizes, not NVMe command sizes. See evidence

knlp connects workload research to reproducible experiments. ebpf-syscall and kvio provide starting points for capture and replay. This is a source-backed workload inventory; it does not report new knlp benchmarks or a measured benefit from a host folio pool.

Workloads and known I/O paths

All rows identify a host-memory storage path. Size evidence ranges from measured buffers and objects to configurable tensor sizes and read plans. None of these sources alone establishes a distribution of NVMe commands above 2 MiB.

Filters describe the named path, not every deployment. A row may appear in more than one filter. Linux buffered I/O includes the page cache; a library’s “unbuffered” option does not establish O_DIRECT.

8 implementations

Workload / implementationSize evidenceHost-memory roleKnown I/O pathWhat remains to measure
Training checkpointsFastPersist / DeepNVMe AIO16–512 MB transfersPaper evaluates 2–128 MB CPU I/O buffers.Pinned DRAM stages GPU tensors; double buffering overlaps device-to-host and storage transfers.Direct bulk I/O · AIO
Host-buffer path; GDS is a separate option.
Hold buffering fixed while varying submitted I/O size and memory backing.
Distributed checkpointsPyTorch DCP filesystem writerModel / optimizer state per rankTensor-dependent; no universal request-size claim.CPU staging creates a stable snapshot. Persistent pinned buffers can be reused across saves.Buffered filesystem path
Other storage writers may differ.
Tensor sizes, serialization writes, durability cost and actual block requests.
Optimizer-state offloadDeepSpeed ZeRO-InfinityConfigurable parameter tilesA 4096 × 4096 FP32 tensor is 64 MiB; illustrative arithmetic.The CPU computes optimizer updates. Tiles are read from NVMe, processed in DRAM and written back.Direct I/O · AIO
Reads and writes can overlap CPU work.
Tile bytes, AIO splitting and CPU update time under the same training workload.
Intermediate-object spillingRay Data / Plasma filesystem backend1–128 MiB block targetSource writes a whole object buffer at the Python file API.CPU-produced Arrow tables, dataframes and task results spill from shared memory and are restored for reuse.Buffered file I/O
Ordinary file opens in the reviewed backend.
Object, library-call, syscall and device sizes; spill/restore latency.
KV-cache offloadLMCache host L1 + native filesystem L2; in-process disk backendModel-dependent chunksExample full-layer BF16 chunk: 32 MiB. Formula below.DRAM serves as a retained cache or temporary staging area before NVMe access.Buffered or O_DIRECT
Backend, flags and alignment matter.
Per-rank and per-layer chunk bytes, cache misses and physical disk traffic.
Multimodal encoder cacheLMCache in-process implementation34.3 / 130.8 MB entriesDocumented video benchmark; one tensor per input hash.Encoder outputs are staged into CPU buffers and can be persisted by the local-disk backend.Backend-dependent
Local disk offers buffered / direct paths.
Disk-served reuse, actual writes and transfer latency independently of encoder savings.
Columnar scans / preprocessingArrow / Parquet on local files64 MiB coalescing limitDefault combined-range limit in GetReadRanges(); planning only.CPU readers decode, decompress and transform selected columns in host memory.File reads / optional mmap
Remote object-store paths need separate accounting.
Selected compressed bytes and actual local read sizes after filtering and coalescing.
Model loadingvLLM safetensors eager / lazy / prefetchWeight tensors and file shardsA 4096 × 4096 BF16 matrix is 32 MiB; illustrative.Eager loading reads files into CPU memory; lazy and prefetch paths use memory mapping / page cache.Buffered / mmap
This row excludes GPU-direct loaders.
Cold-start latency, resident cache hits, readahead and request splitting.

Keep the size evidence at the right layer

The same 64 MiB object can generate one library call, many syscalls or many NVMe commands. It can also be served entirely from RAM. A large application buffer is useful evidence for host-memory management, but is not a trace of device I/O.

QuantityWhat it tells usHow to establish it
Object / tensor bytesUseful payload and capacity demand.Shape, dtype, serialization and compression.
Staging-buffer bytesWorking memory available to pipeline transfers.Allocator configuration and measured buffer use.
Submitted I/O bytesWork handed to the OS, including each AIO or io_uring operation.Application instrumentation and submission traces.
Block / NVMe bytesRequests after merging, splitting, mapping and cache effects.Block and driver traces on the named target device.
Backing folio sizeMemory allocation and representation granularity.Inspect the actual backing; alignment alone is insufficient.
A useful existing baseline: DeepNVMe documents an AIO default of block_size=1048576 (1 MiB) and queue_depth=128. Large tensor transfers already work with smaller queued operations. A folio-pool evaluation must compare against that kind of reused-buffer implementation. [3]

Research notes and primary evidence

Published buffer-size experiment

Checkpointing: the clearest measured example

FastPersist’s §5.3.1 tests GPU-tensor writes through pinned CPU buffers to SSD. The table records the best tested buffer sizes; the authors use MB. For 512 MB transfers, 2 MB performed worst and the best double-buffer configuration was 2.87× faster. This measures pipeline buffering, not a folio-size or NVMe-command-size improvement. [1]

TransferSingle bufferDouble buffer
16 MB16 MB8 MB
512 MB64 MB32 MB
Established host staging

Distributed checkpoint save and restore

PyTorch DCP’s asynchronous save copies model and optimizer state into CPU buffers before persistence. Its pinned-buffer option retains staging allocations across checkpoints. The standard filesystem writer is a buffered file path; another storage writer can change that behavior. [2]

Measure serialized writes and completion through the required durability boundary. A return from a buffered write is not evidence that the checkpoint is on stable storage. Compare identical checkpoint content and recovery semantics.

CPU computes on the payload

Optimizer state and repeated tensor offload

ZeRO-Infinity supports NVMe optimizer storage while keeping optimizer computation on the CPU. Its tiled pipeline can overlap the next read and the previous write with the current update. This makes host DRAM part of the computation itself. [4]

A GPU-owned allocation does not replace that CPU working set. The integration question is whether a reusable host pool can back the existing tensor buffers without inserting an extra copy. DeepNVMe’s AIO and GDS results should be treated as separate paths. [5]

Host object and library write verified

Ray spilling: a CPU-only starting point

Ray Data targets 1–128 MiB blocks, commonly Arrow tables or pandas dataframes. Its filesystem spill writer calls f.write(memoryview(buf)) for each object payload using ordinary file opens. That verifies a full-buffer library write, not one syscall. [6]

Ray restores spilled objects when needed. Preprocessing and large tool results give a natural test workload: force a controlled object-store budget, measure disk-backed spill and restore, and keep the task outputs identical. Object retention pins in Ray must not be confused with DMA page locking. [7]

Cache retention and staging differ

LMCache with host memory in the path

Use an explicit host-DRAM L1 with a filesystem L2 backend for the MP case. The native filesystem adapter offers use_odirect and alignment-aware slot transfers. The older in-process disk backend also allocates CPU read buffers; local_cpu: false disables retained CPU caching without proving that host staging disappears. [8]

Illustrative unsharded, all-layer BF16 KV: 2 × 32 layers × 256 tokens × 8 KV heads × 128 dimensions × 2 bytes = 32 MiB. Layerwise transfer, tensor parallelism, architecture and quantization change the actual object and request sizes. Record them from the selected implementation.

Measured non-KV cache entries

Video, image and audio encodings

LMCache’s in-process encoder cache stores a tensor per multimodal input hash, independently of attention KV. Its Qwen2.5-VL-7B BF16 video benchmark reports 34.3 MB entries for 32 sampled frames and 130.8 MB for 128 frames. CPU and disk backends are supported. [9]

Those are cache-entry measurements. The reported warm-request speedups do not establish disk-hit latency, and may include CPU-cache hits. This documentation is labeled deprecated in-process mode; verify the selected connector and version before adapting it to MP.

Read coalescing is implemented

Columnar data used by AI tools

Arrow’s GetReadRanges() plans selected Parquet column-chunk ranges, with a default 64 MiB combined-range limit. It performs no I/O itself. PyArrow’s prebuffering can coalesce reads before CPU deserialization; actual transfer sizes depend on columns, compression, filters and the filesystem. [10]

Choose files resident on the target NVMe filesystem for a local block-layer experiment. A 64 MiB S3 request alone is network evidence, not evidence of a local 64 MiB disk request.

Established model-loading paths

Weights, cold starts and model changes

vLLM documents eager safetensors loading into CPU memory, lazy memory mapping and prefetching into the OS page cache. These establish host-memory paths with large tensor and shard payloads. [11]

Cold model provisioning and repeatedly streaming nonresident weights are different workloads. DeepNVMe also evaluates ZeRO-Inference using host bounce buffers. Keep model-load frequency and actual cache misses explicit; a large model file does not imply continuing disk traffic during serving. [5]

Where host folios and a reusable pool fit

CPU producers and consumers need usable host memory: optimizer updates, dataframes and column decoding are examples. GPU checkpointing and cache offload instead use host memory as a staging or retention tier. GPU-direct paths can remove that tier in some deployments; they do not cover the CPU-native cases.

Three device properties, different meanings

MDTS describes the controller’s maximum transfer limit; NPWG describes recommended write granularity; NOWS describes optimal write size. Decode the fields and validity rules for the device. A value above PAGE_SIZE is not itself proof of a workload above 2 MiB or a requirement for an equally large folio. [12]

Also record max_hw_sectors_kb, max_sectors_kb, max_segments and max_segment_size, plus filesystem, DMA/IOMMU and stacked-device constraints. A large application request may be split before reaching the controller. [13]

The hypothesis to test

A device-associated, preallocated folio set could reduce allocation, page bookkeeping, scatter/gather or mapping costs for reusable host buffers. Benefits depend on the implementation and mapping lifetime; larger folios do not automatically produce larger DMA mappings.

A large I/O can span several folios. Compare ordinary reused registered buffers, available large-page backing and the proposed pool at equal useful work. Evaluate backing size, submission size and persistent mapping separately. No result on this page establishes that any one of them wins.

Related Linux work: DMA-BUF + io_uring. Buffer ownership is the connection: an accelerator can supply buffers for its own payloads, while CPU-native workflows need a host-memory allocation path. The pool should become backing for the real working buffer where possible; copying into another pool changes the trade-off.

Agentic workflows: follow the payload

An agent’s transcript can be small while its tools scan datasets, create large intermediates, load models or checkpoint execution environments. The storage workload comes from those operations and their frequency.

ActivityEstablished path / useful candidateEvidence still needed
Data-analysis toolsRay spilling and Arrow scans above provide concrete CPU-memory paths for table and tensor batches.Capture a named agent task, its data scale and the local disk traffic per completed task.
Multimodal investigationLarge media artifacts and reusable encoder outputs; LMCache supplies an existing encoder-cache implementation.Separate original media reads, decoded data, encoding reuse and attention KV; measure disk hits.
Sandbox pause / resumeE2B documents dirty-memory and rootfs snapshots plus a local template cache. Export and cache population are bulk-transfer candidates. [14]Large-request hypothesis. Resume uses lazy page faults and optional prefetch; snapshot size does not establish large reads.
Model experimentsModel loading, checkpoints and optimizer offload are established above.Count jobs, saves, bytes and cache misses. Remote API use alone does not imply local weight I/O.
RAG, packages and build artifactsIngestion, index construction, compaction and artifact staging deserve capture.Name an implementation and measure requests. Online ANN lookups, metadata and small files need their own analysis.

The context-demand observatory and agentic-cache capacity model motivate the inventory. They do not measure disk request sizes. OpenRouter’s cache-usage observations describe token reuse, without identifying the storage tier serving each hit. [15]

Capacity, traffic and request size need separate measurements. Retained bytes describe footprint. Reads and writes per completed task describe turnover. Request-size distributions describe how that traffic reaches the storage stack. Deduplication, compression, RAM residency and remote storage can change all three.

Reproduce the storage question

Start with host-staged LMCache, a FastPersist-style checkpoint writer and Ray spill/restore. These cover inference, published large-buffer evidence and CPU-only processing. The integrations below are proposed experiments, not newly completed knlp benchmarks.

ComparisonHold fixedMeasure
Buffer backingReuse and registration lifecycle, payload, NUMA placement, memory budget, submitted size and durability.Allocation/pinning cost, CPU cycles per GiB, mapping work, faults and application latency.
Submitted sizeBacking, logical data, total bytes in flight and completion semantics. Sweep 1, 2, 4, 8, 16 and 32 MiB where supported.Actual NVMe sizes, splitting, throughput and tail latency. Unsupported large commands are not large-command tests.
Direct vs bufferedUseful work, total RAM budget, file layout, concurrency and durability.Cold, mixed and resident-cache behavior; device bytes and CPU cost, including cache hits.
Full applicationSame checkpoint, recovered object, model output or data-analysis result.Checkpoint pause, restore time, task completion or TTFT; report any extra memcpy introduced.
Read-only queue-limit inventory

Set dev to the target block-device name. These sysfs reads do not change device settings. Inspect every relevant leaf device when testing RAID or device mapper; passthrough paths need their own limits.

dev=nvme0n1
for field in logical_block_size physical_block_size minimum_io_size \
             optimal_io_size max_hw_sectors_kb max_sectors_kb \
             max_segments max_segment_size; do
    printf '%s: ' "$field"
    cat "/sys/class/block/$dev/queue/$field"
done

Capture both sides of the filesystem boundary. Buffered reads that hit RAM never appear in a device trace; mmap requires fault/residency observations. For asynchronous I/O, record each submitted operation’s length, not just the syscall that submits a batch. Keep device requests separate from application calls when replaying traces.

Publish source revisions, source data or a reproducible generator, hardware and queue limits, filesystem/mount options, I/O mode, folio backing, cache budget, bytes in flight, repeated-run variation and application correctness. A bandwidth gain without matching useful work and completion semantics is insufficient.

ebpf-syscall capture / replay · kvio · KVTide · knlp through kdevops

Sources

Primary papers, implementation code and project documentation. Moving documentation and branches describe the reviewed paths; record immutable revisions for experiments. MB is retained where the source uses MB; MiB denotes 2²⁰ bytes.

  1. FastPersist: Accelerating Model Checkpointing in Deep Learning§4.1 host staging; §5.3.1 / Figure 7 buffer-size measurements.
  2. PyTorch asynchronous distributed checkpointing · filesystem writerCPU staging and persistent pinned-buffer options.
  3. DeepNVMe tutorial · AIO file-open flags · I/O handleThe reviewed Linux AIO path opens filenames with O_DIRECT; the CANN build removes that flag. Existing file descriptors retain caller-selected flags. Block size and queue depth are separate controls.
  4. DeepSpeed ZeRO configurationNVMe optimizer offload, CPU computation and tile pipelining.
  5. DeepNVMe: Affordable I/O scaling for Deep Learning ApplicationsFastPersist and ZeRO-Inference evaluated with host AIO and GDS paths.
  6. Ray Data internals · filesystem spill writerBlock-size targets and full-object Python writes.
  7. Ray object-spilling architectureShared-memory pressure, spill workers and restoration.
  8. LMCache MP native filesystem L2 · in-process local storage · disk backend codeSelect host-memory L1 explicitly; distinguish it from GPU-direct backends.
  9. LMCache encoder caching · encoder-cache designIn-process implementation, tensor ownership and video-entry measurements.
  10. Arrow C++ Parquet API · PyArrow read_tableGetReadRanges(), PreBuffer() and filesystem options.
  11. vLLM engine argumentsSafetensors eager, lazy and prefetch loading strategies.
  12. libnvme NVMe field definitionsMDTS, NPWG and NOWS.
  13. Linux block-device sysfs ABITransfer-size and scatter/gather limits.
  14. E2B runtime architectureSnapshots, local template cache, dirty rootfs export and lazy UFFD restoration.
  15. Context demand · Agentic-cache capacity model · OpenRouter: Cache is CrucialDemand and capacity context, not device-I/O traces.