Research tracker

Small random reads
in AI workflows

This page tracks the leading use cases we have found for small random reads below 4096 bytes (4 KiB) in AI workflows. Recommendation embeddings, ANN retrieval and GNN feature gathering are the clearest examples so far.

We also try to identify whether each workload is known to use Direct I/O or Buffered I/O, and whether it could benefit from using buffered I/O in the future.

Companion: Large CPU-initiated I/O in AI — checkpoints, optimizer state, object spilling and other bulk transfers through host DRAM.

knlp supports early AI R&D through reproducible experiments. It is available as a kdevops plugin and also runs on an existing host. Kconfig presets and Make targets let others repeat a study, inspect the results and test changes. The reproduction section lists starting points for this work.

We use ebpf-syscall to record I/O from different AI workflows, compare application requests with device traffic, and replay the recorded storage access patterns. The aim is to make questions about read amplification, caching and data layout reproducible. This tracker is one R&D use case for those tools.

kvio provides recording and replay for KV-cache offloading workflows, including LMCache-based experiments. Here, the focus is small random reads below 4096 bytes; conventional KV-cache offload usually involves larger transfers.

Workloads and known I/O paths

The rows identify specific implementations. Record sizes describe useful data; file and device reads may be larger. Metadata workloads are included where a storage engine is known, with the missing application evidence stated.

Buffered I/O includes read() / pread() and mmap. Both use the Linux page cache, through different access paths.

9 implementations

Workload / implementationUseful record sizeKnown I/O pathCould direct I/O be switched to buffered I/O?Analysis to follow up
DLRM embedding lookupsMeta Software Defined Memory128–256 BQuantized embedding rows in the studyDirect I/O
io_uring and an application row cache.
A buffered CPU reader could be tested. The page cache adds another caching layer; the SGL bit-bucket path would not carry over unchanged.Compare lookup latency and total cache memory using the same row IDs and access distribution.
ANN rerankingOriginal FusionANNS paper128–384 BRaw vectors discussed in the paper; stored within 4 KiB pagesDirect I/O
Page grouping, request deduplication and application buffering.
Possible with a buffered file reader. Preserve layout and scheduling, then compare memory use and performance.Measure reuse of reranking pages and shared-page reads across concurrent queries.
ANN rerankingSamsungDS FusionANN prototype512 B by default128 FP32 values; the reader requests 4 KiB pagesBuffered I/O
Python file reads and an application page cache.
Already buffered. It is an independent implementation of the paper’s ideas.Trace actual syscall sizes and page-cache hits. Application read counts alone do not count SSD commands.
ANN graph / vector retrievalDiskANN C++ Linux readerDepends on vector and node layoutSub-4-KiB payloads need to be identified in the selected datasetDirect I/O
O_DIRECT with asynchronous reads.
Requires a buffered reader and scheduling work. No supported configuration switch is established here.Measure node payload, full read size, locality and recall before selecting a storage comparison.
On-disk vector storageQdrant mmap configurationDepends on dimensions and encodingBuffered I/O · mmap
Memory-mapped vector files; the OS caches pages.
Already uses the page cache. Buffered pread() would be a separate implementation.Identify a sub-4-KiB representation and measure faults, residency and query latency.
Fraud-graph feature lookupsknlp DGraphFin forced-SSD experiment68 B17 FP32 node featuresDirect I/O
SSD access is forced for the experiment.
A buffered CPU feature reader could be added. Keep the requested nodes and features identical.Compare page grouping and I/O paths separately. The roughly 250 MB feature store fits in RAM.
Relational feature gatheringKumoRFM-2 SSD graph engineNot reportedThe sub-4-KiB share needs measurementBuffered I/O · mmap
The paper specifies memory-mapped I/O for this engine.
Already uses the page cache. A buffered read() implementation would be a separate path.Capture sampled row and feature accesses, their useful sizes, page faults and reuse.
Out-of-core GNN featuresGIDS512 B–4 KiBTypical feature sizes discussed in the paper; the smaller sizes fall within this trackerGPU direct access
GPU-initiated storage, outside CPU buffered reads.
Requires a different architecture. Host buffered reads would add staging, transfers and synchronization.Compare the full CPU-staged path against the original GPU path on a dataset larger than available memory.
Feature / session / agent recordsRocksDB-backed application candidateApplication trace neededValue size and file-block size differBuffered by default
RocksDB also supports direct I/O. The AI application path is not yet traced.
The engine supports both modes. A service’s configuration and caches determine the actual path.Obtain record sizes, access patterns and file reads from a specific AI workflow.

Conversion feasibility is not a performance result. Buffered I/O may benefit from reuse through the page cache, while direct I/O may avoid duplicate caching or suit an accelerator’s data path. The comparison needs the same useful work, memory budget and latency target.

Research notes

Recommendation embeddings

Supporting Massive DLRM Inference through Software Defined Memory (Audio Summary) describes small row lookups and a direct-I/O implementation with an application cache. Its measurements favored that design over mmap. A buffered-read comparison would need to account for the different cache granularity and available RAM.

The NVMe SGL bit-bucket mechanism reduces the unused data transferred to host memory. Keith Busch’s 2022 partial-sector read series proposed exposing this through direct I/O. The support was never merged upstream and remains out of tree. Ordinary buffered reads can return small records, but do not reproduce that transfer mechanism.

ANN retrieval and reranking

FusionANNS (Audio Summary) shows how placement and page reuse can reduce redundant reads of small reranking vectors. Its direct-I/O implementation and the buffered SamsungDS prototype provide different starting points for further work.

DiskANN’s Linux reader and Qdrant’s mmap storage illustrate the range of access paths already used in ANN systems. A comparison should preserve retrieval quality and include the complete record read, index overhead and reranking traffic.

GNN feature gathering

GNNs gather feature vectors for sampled neighbors before aggregation. Small, scattered feature records can cause much larger storage reads when the feature table is on SSD. The DGraphFin showcase records useful feature bytes and NVMe commands using eBPF, then replays the device access pattern without feature values.

Page-aware sampling and changing the I/O backend should be evaluated separately: the current showcase’s arms consume different useful-byte totals. For an I/O comparison, hold requested feature IDs fixed.

A buffered feature reader would let us test the same requests through the page cache. The papers below extend this work to larger GNN datasets, graph transformers and Kumo’s relational models.

Session state and agent metadata

The Storage-Next analysis (Audio Summary) includes classical key-value-store and ANN case studies. Session state and reasoning metadata appear as motivations, with no separate application trace sufficient to reproduce their storage demand.

RocksDB supplies a practical buffered-read baseline and direct-I/O options. The next step is to identify a real record format and workload. The LMCache / kvcache.io agentic-cache capacity projection can help with sizing, but does not specify individual I/O requests.

GNNs and graph transformers

The feature store and graph structure can be much larger than the model’s weights. Our DGraphFin feature table fits in RAM, so that experiment forces SSD access to study read amplification. GIDS supplies larger public examples, while KumoRFM-2 describes an SSD-backed relational graph engine. These give us several ways to study feature gathering at different scales.

Paper / implementationFeature access and storage evidenceSmall-read analysis
Inductive Representation Learning on Large Graphs (GraphSAGE) (Audio Summary)Stanford · 2017Samples neighbors and aggregates their feature vectors. It provides a basic feature-gathering workload; the paper does not specify a disk I/O path.Record sampled node IDs, feature widths and repeated accesses. Use that request stream to compare layouts and readers.
Relational Graph Transformer (RELGT) (Audio Summary)Stanford, Kumo.AI and NVIDIA · ICLR 2026Represents table rows as graph nodes and gathers temporal two-hop neighborhoods for attention. Evaluated on 21 RelBench tasks; storage I/O mode is not established.Trace neighbor features and graph metadata separately. Measure the stored row sizes and reuse before choosing a sub-4-KiB benchmark.
KumoRFM-2: Scaling Foundation Models for Relational Learning (Audio Summary)Kumo · 2026The paper describes an SSD graph engine using memory-mapped I/O, alongside in-memory and SQL-backed execution. The SSD engine is the implementation tracked above.Measure the size distribution of sampled records and page-cache residency. The paper does not report how many reads are below 4096 bytes.
Rethinking Graph Transformers with Spectral Attention (Audio Summary)Spectral Attention Network · NeurIPS 2021Adds Laplacian positional information and fully connected attention over graph nodes. This is an architectural comparison with sampled-neighbor GNNs.Check how feature reuse and batching change. Its benchmark results do not establish an out-of-memory storage workload.
Accelerating Sampling and Aggregation Operations in GNN Frameworks with GPU Initiated Direct Storage AccessesGIDS · VLDB 2024Uses GPU-initiated storage for out-of-core feature gathering, with public datasets above a terabyte. The paper discusses feature sizes from 512 B to 4 KiB.Use the smaller feature widths for this tracker. Compare host buffered staging with the complete GPU storage path, including transfer and synchronization costs.

Financial workloads

Nubank: Your Spending Needs Attention (Audio Summary) describes nuFormer, which learns from transaction histories and combines them with tabular features. The updated paper reports production use for credit-default prediction. Its daily batch inference is a useful case for tracing history and feature access. The audio summary covers an earlier version of the paper.

Visa Research: TransactionGPT (Audio Summary) evaluates transaction modeling and anomaly detection on billion-scale payment data. It is another concrete financial workload to investigate for feature and history access.

Both use transaction transformers. Their papers do not establish a GNN storage path, sub-4096-byte reads, or a Direct I/O / Buffered I/O choice. Those details need a workload capture before they can enter the main table as known small-read cases.

Cases for further analysis

These provide small representations or a reason to investigate selective access. Their storage demand is less established than the embedding and feature lookups above.

CandidatePotential record sizeKnown I/O pathBuffered-I/O feasibility and next analysis
MUVERA encodings1,280 B PQ code in the paper’s online setupIndex and reranking data are additionalMemory measurements
Current knlp work has no qualified cold-SSD result.
A buffered storage backend could be added. First establish quality versus serialized size and identify which encodings and reranking vectors would be fetched.
REFRAG chunk cacheDepends on stored dimension, dtype and chunk representationStorage path unverified
No SSD read trace established here.
A host file-backed cache could use buffered reads. Measure the actual cached representation, reuse and batching first.
Engram conditional memory160 B BF16 rows in the stated configuration16 independent lookups per token per Engram layerHost DRAM
Published offload experiment.
An added SSD tier could use buffered I/O once tables exceed the assigned memory budget. The capacity thresholds below show the required growth; row misses and prefetch deadlines determine I/O demand.
Selective model weightsFor example, 2 KiB for 4,096 packed INT4 valuesScales, metadata and related matrix slices add bytesImplementation to select
Sub-4-KiB weight I/O is not demonstrated here.
A host buffered loader is a candidate. Measure complete fetch bundles, reuse, prefetch, quality and exposed GPU stalls before comparing with other memory tiers.
MUVERA and REFRAG

MUVERA (Audio Summary) addresses retrieval. A model such as ColBERT represents a document with many token vectors. MUVERA converts these sets into fixed-dimensional encodings (FDEs), with different constructions for queries and documents. Their dot product approximates the original multi-vector similarity, so a conventional vector index can find candidates. Reranking then uses the original token vectors. The paper’s online setup compresses a 10,240-dimensional FDE to a 1,280-byte product-quantization code; index data and reranking vectors add storage.

REFRAG (Audio Summary) addresses the context supplied to generation. An encoder compresses token chunks into embeddings, which are projected into the decoder’s input space. The generation path is trained to use these compressed representations, shortening the sequence the decoder processes. It still computes its own attention and KV state. A cached REFRAG chunk embedding is therefore different from a MUVERA retrieval encoding; an FDE is not automatically an input a language model can understand.

Our MUVERA visualization and experiment tracker studies the storage question behind both approaches: how small can a useful representation be? It separates the REFRAG-inspired memory and transfer study, the local ColBERT-based retrieval approximation, and the new target using Google’s FDE construction. Google-backed quality results remain pending. The recorded memory measurements do not establish cold-NVMe performance or REFRAG answer quality.

Run make defconfig-google-muvera-small && make to extend the study. The next step is to qualify retrieval quality against serialized size, then measure full-record reads, page reuse and amplification. The legacy optional NVMe reader needs complete-record handling above 4 KiB before those tests.

Engram: host DRAM today, possible NVMe tiering at larger table sizes

DeepSeek’s Engram paper (Audio Summary) separates lookup-table capacity from GPU computation. Lookup addresses depend on the input tokens, allowing host-memory fetches to overlap earlier layers. Its separate systems experiment puts a 100B-parameter table entirely in host DRAM and reports at most a 2.8% throughput penalty. That experiment demonstrates DRAM offload over PCIe; it does not test NVMe.

The trained Engram-27B and Engram-40B variants contain 5.7B and 18.5B lookup-table parameters. At two bytes per parameter, their table payloads are about 11.4 GB and 37 GB; the separate 100B table would be 200 GB. These are BF16/FP16 capacity calculations, not measured DRAM footprints. All three fit the 256 GiB available-DRAM budget used by default in the LMCache / kvcache.io Engram I/O projection.

Using that calculator’s two-byte storage assumption and no additional unique HBM capacity, table payload exceeds memory above these thresholds:

DRAM available for EngramTable capacity× Engram-27B’s 5.7B table× Engram-40B’s 18.5B table× the 100B offload table
256 GiB · projection default137.44B parameters24.11×7.43×1.37×
512 GiB274.88B parameters48.22×14.86×2.75×
1 TiB · 1,024 GiB549.76B parameters96.45×29.72×5.50×

These multipliers apply to Engram tables, not the whole model. “Available DRAM” is what remains after other uses and overhead. With 1 TiB reserved, the 40B variant’s table would need to grow roughly 30-fold before its payload alone exceeded memory. Less DRAM or several models sharing a host could justify NVMe earlier. Crossing the threshold creates a capacity case for tiering; the actual small-read demand depends on which rows miss the cache and whether prefetch completes in time. Longer contexts and larger batches increase lookups, not this static table’s size.

Model weights and HBF

LLM in a flash demonstrates selective weight loading with reuse and contiguous bundling. A small row can be part of a larger required bundle, and dense layers still consume substantial aggregate bandwidth.

Future work could test where selective random reads from SSD are viable relative to approaches such as High Bandwidth Flash (HBF). Useful comparisons need active bytes per token, miss rate, latency deadlines, CPU and transfer costs, and achieved inference throughput. No equivalence is assumed.

Conventional attention-KV offload is outside the leading sub-4-KiB cases here. The LMCache / kvcache.io KV-cache I/O projection covers the larger tensor and token-block transfers common in those paths. Fine-grained variants would need their own measured request distributions.

Buffered I/O improvements to evaluate

Ongoing Linux kernel work aims to improve the scalability of small buffered reads through the page cache. Some workflows above could benefit where many CPU threads read shared cached data. The benefit will depend on their access pattern and how much work already completes in an application cache.

One current example is Linus Torvalds’s page-reference scalability series. It reduces reference-count contention during eligible cached reads. It is a possible optimization for buffered-read implementations; it does not reduce the storage transfer size on a cache miss, and mmap uses a different access path.

Patch links and review status · 14 October 2026

The posted series is rcu-free-v3 (GitHub mirror), observed at 549531755d329ec48439e3263535351ba1c5acd5. See the buffered-read patch and full series.

Review identified an i_size ordering issue; Kiryl Shutsemau posted a revised fix and took on further work. An additional unlink proposal raised an f2fs interaction. Record the reviewed revision and follow-up fixes used for testing.

Daniel Gomez is following up on kernel testing. The workload analysis here can supply application cases for that work as they become reproducible.

Reproduce and extend the work

knlp provides Kconfig choices, defconfig presets and Make targets for early AI R&D. The familiar kdevops workflow is to select a study, then run it:

make defconfig-<study>
make

The kdevops plugin prepares the target and installs knlp. Run the experiment commands below inside the knlp checkout on that target, or on an existing host prepared with the standalone setup instructions. Selecting the same preset makes the study easier to repeat and extend; the hardware, dataset and effective configuration still belong with each result.

StudyCommands in the knlp checkoutWhat this runs
DGraphFin feature gathering
make defconfig-gnn-dgraphfin
make
The GNN comparison with dataset and layout preparation. Its memory-resident experiment models page-read amplification.
DGraphFin on SSD
make defconfig-gnn-dgraphfin-force-ssd
# Set CONFIG_GNN_FRAUD_SSD_DIR in .config
make
Real Direct I/O feature reads. Set the directory to a filesystem on the SSD being studied; attach eBPF capture tools separately to compare useful bytes with device reads.
Google MUVERA construction
make defconfig-google-muvera-small
make
The Google FDE construction, retrieval evaluation and memory measurements. Cold-SSD testing is a separate follow-up.
Legacy MUVERA study
make defconfig-muvera-small
make
The original local retrieval approximation and systems study. Keep it distinct from the Google-backed implementation.

The experiment tracker explains the results and open tests. Other systems cited on this page retain their own implementations; inclusion here does not mean knlp reproduces every paper.

To investigate a small-read workload, keep the requested records and quality target fixed, change one layout or I/O choice, and record both logical requests and device traffic. A saved access trace lets others repeat the storage comparison without the original feature values. A full application reproduction also needs the model, data preparation and workload configuration.

Ongoing analysis

WorkNext stepCurrent status
DLRM and ANNChoose a repeatable lookup stream. Compare direct and buffered readers with the same layout, results and total cache budget.Public designs available; comparison pending.
GNN and relational featuresAdd a buffered reader to the feature experiment with fixed node IDs. Capture KumoRFM-2’s mmap access pattern and identify sub-4096-byte records; use larger datasets for natural memory pressure.Capture/replay tools available; Kumo trace pending.
MUVERARun the Google-backed quality sweep, select a representation, then measure complete records on SSD.Harness available; quality and storage results pending.
Metadata and agent stateObtain a record schema and logical read trace from a specific application.Application evidence needed.
Engram and weightsSpecify a deployment and justify storage access. Measure the miss stream and timing before projecting performance.Future analysis.
Linux page cacheTest a reviewed kernel revision across shared hot reads, mixed residency, destination faults and memory pressure.Kernel follow-up: Daniel Gomez.
Measurements to keep with each result

Record useful bytes, actual syscall sizes, cache hits, completed device bytes, queue depth, CPU cost and application latency. A small logical record may cause a 4 KiB or larger storage read; a page-cache hit may cause none. Keep NVMe logical block size and the drive’s internal granularity separate.

Use resident, mixed and cold working sets. Hold useful work and quality fixed when comparing I/O paths, and record source revisions, hardware, memory budgets and variation across repeated runs. A file on the target filesystem is sufficient for buffered testing; raw NVMe access is not required.

Device traces support storage replay. Page-cache evaluation also needs logical requests, sharing and concurrency, including reads that never reached the device. Traces can omit feature values while retaining the access structure needed for a meaningful comparison.

Sources

Graph and financial papers, with matching audio summaries, are linked in the GNN and graph-transformer section.

  1. Supporting Massive DLRM Inference through Software Defined Memory (Audio Summary)Embedding rows, direct I/O, application cache and SGL bit buckets.
  2. Keith Busch: support for partial-sector readsProposed 30 June 2022; not merged upstream. Status checked against the upstream NVMe driver, 14 October 2026.
  3. FusionANNS (Audio Summary) · SamsungDS prototypeThe paper and independent prototype use different I/O paths.
  4. DiskANN Linux reader · Qdrant storage
  5. Accelerating Sampling and Aggregation Operations in GNN Frameworks with GPU Initiated Direct Storage Accesses · GIDS code
  6. ebpf-syscall capture and replay tools · GNN read-amplification showcase · kvio: KV-cache offload recording and replay
  7. RocksDB I/O · direct-I/O options
  8. Five-Minute Rule 40 Years Later (Audio Summary)v2, 8 May 2026; Storage-Next modeling and KV/ANN case studies. Audio covers an earlier version.
  9. MUVERA v2 (Audio Summary) · Google construction code
  10. knlp legacy MUVERA · Google-backed target · visualization
  11. REFRAG v2 (Audio Summary)
  12. DeepSeek Engram (Audio Summary) · LMCache / kvcache.io Engram I/O projection
  13. LMCache / kvcache.io agentic-cache capacity projection · LMCache / kvcache.io KV-cache I/O projection
  14. LLM in a flash · Sandisk HBF overview
  15. Linux page-reference scalability discussion13–14 October 2026; patch and follow-up links above.