Context reuse · checkpoint inference · October 2026

Reusable context,
typed by its consumer.

A learned key/value prefix, a document-vector collection, and a query’s soft memory tokens can share a storage contract. Each still needs its own model interface and cache identity.

This work extends the Learned Prefix Cache (LPC) architecture toward Memorilla’s latent memory. The shared catalog and LMCache mapped-artifact path supply document collections to the released Memorilla reader. The current qualification covers CPU input preparation, reader outputs and exact GPU decoder outputs. LPC keeps its own model attachment path.

Two suites in the mapped comparison516,096 exact input rowsNo training required

Current result: the mapped LMCache path prepares PubMedQA inputs 1.05–1.19× faster and NarrativeQA inputs 2.70–4.12× faster than the author’s unchanged loader and collator. Against a native mmap loader given the same view reuse and batch assembly, it is within measurement noise in every case: 0.999–1.008×, with no interval below 0.982. LMCache adds artifact identity, admission and lifecycle at no measured throughput cost; it is not a speedup over equally optimized native code. These are CPU preparation measurements. See the matched results and reproduction commands.

01 / The shared contract

Separate storage from model execution.

A context artifact is a versioned tensor payload plus the information needed to use it correctly. Its immutable ID binds the payload digest, representation, producer, and compatible consumers. File location and working-copy residency can change while that semantic identity stays fixed.

Learned prefix

prefix_kv

Per-layer key and value tensors for a particular model. Preserve exact logical positions, layer/head geometry, rotary-position settings, dtype, layout, and padding.

Document collection

document_embeddings

Vectors produced by a pinned encoder and preprocessing pipeline. One collection contains a vector for each assigned text entry; different questions can reuse it. A compatible memory reader consumes these vectors. They do not enter the decoder’s key/value cache directly.

Soft memory tokens

input_embeddings

Vectors in the decoder’s input space. Bind their reader, collection, question, prompt placement, and positions. Prefill must still turn the assembled input into layer-specific key/value state.

The file registry implements import, resolve, load, working-copy eviction, and restore. It validates safetensors before synchronizing payload files and directories, then publishes metadata in a synchronous SQLite transaction. Missing, corrupt, or incompatible required artifacts fail explicitly. Existing cartridge connector APIs remain unchanged.

02 / Sharing prospects

Reuse the stable representation.

The useful sharing boundary depends on what produced the bytes. The following are architectural prospects; CPU file reuse does not establish physical GPU aliasing or an engine cache hit.

ArtifactPotential reuseIdentity that must stay bound
LPC key/value prefixMany requests to the same compatible decoder can attach one trained artifact. Avoid training or reconstructing it on a cache miss.Checkpoint bytes, model weights, positions, geometry, layout, and ordered composition with the request.
Memorilla document collectionDifferent questions can reuse the same query-independent collection. New or changed source material needs a new version.Source content, encoder revision, preprocessing, namespace, and compatible reader.
Memorilla readoutAn identical readout can be restored when its complete dependencies match. A shared collection alone is insufficient.Actual soft-token bytes, question/readout dependencies, reader checkpoint, decoder, template, and placement.
Derived decoder key/value statePotential reuse after exact prompt-embedding prefill. This bridge is not implemented here.All preceding inputs, their order, model identity, positions, and the derived payload.

An ID gives consumers a common object to resolve. It does not make vectors interchangeable across models, enforce access control, or make independently trained prefixes composable. Namespace is part of identity; an actual service still needs authorization. Multiple runtime users, resident-memory deduplication, and cache-key propagation require separate engine tests.

03 / Where the bytes go

One lifecycle, different attachment paths.

Learned prefix
Stored layer KVVerified CPU import / restore
Compatible KV attachmentEngine integration pending
Decoder executionSeparate engine qualification
Memorilla
Document collection + questionReleased cached vectors
Released memory reader16 soft tokens; validated readout
Prompt-embedding prefillExact GPU decoder outputs

The current mapped qualification exercises document tensors, batch assembly and the released reader on CPU, and checks exact decoder outputs on GPU. LPC engine attachment requires its own qualification. Newly computed readouts do not need a mandatory storage round trip.

04 / Papers and interfaces

Memorilla and REFRAG shorten context differently.

Memorilla reads a query-independent document-vector collection with a learned, query-conditioned module. Its released Qwen configuration emits 16 input-space memory tokens for a frozen decoder. Reading the whole selected collection is part of that method; replacing it with top-k retrieval would change the algorithm.

REFRAG, from Meta and academic collaborators, encodes fixed token chunks, projects their vectors into decoder input space, and can expand selected chunks back to text. Its reconstruction stage freezes the decoder; later continual pretraining unfreezes it. Its mixed-input path therefore needs the corresponding trained decoder, encoder, projection, and policy.

QuestionMemorillaREFRAG
Reusable unitDocument embeddingsEncoded chunks
Query-time contextFixed-size learned memory readoutOrdered mixture of compressed chunks and expanded text
Decoder dependencyFrozen decoder paired with the trained readerDecoder aligned through the paper’s training recipe
Context budgetFixed memory-token countDepends on chunk count, compression, and expansion decisions

These are method differences, not a matched accuracy or speed comparison. The papers use different models, tasks, training, and controls. Neither paper’s headline performance is a measured result of this implementation.

05 / Engineering target

Make context durable, compatible, and reusable.

Our target is a common lifecycle for context representations: publish verified bytes, resolve an immutable handle, load the representation its consumer expects, and recover it after eviction or restart. Training remains a producer’s separate responsibility. A missing LPC must not launch training, and storage must not disguise document vectors as layer KV.

Preserve each control.

Establish the real LPC engine/checkpoint baseline and the unmodified Memorilla decoder path. Keep engine-version changes separate from artifact/storage changes.

Attach context through the right interface.

Use the existing LPC attachment for KV and exact template-aware input embeddings for Memorilla. Retain engine ownership of allocations and scheduling.

Qualify sharing and restore.

Test equal-length different payloads, different questions over one collection, native and external cache keys, and actual inference after GPU eviction and process restart. File parity is the prerequisite already explored.

Measure the complete cost.

Separate encoding, readout, serialization, storage completion, transfer, prefill, and decode. Compare matched tensors and cache states before making a storage or end-to-end speed claim.

A possible REFRAG adapter belongs at the chunk-collection and mixed-input boundaries, with its own typed metadata. Whether that adapter is worthwhile needs a separate compatibility and quality evaluation. It is not part of the CPU pass reported here.

06 / Reproduction

Use the released checkpoint and the same trace.

The public runner supplies memorilla-author for the pinned author environment and memorilla-lmcache-mapped for the current integration. Both use released weights; no training is required. The default benchmark is the complete 100-question PubMedQA test. Select all for the eight supervised benchmarks.

make defconfig DEFCONFIG=memorilla-author
make menuconfig
make memorilla-fetch MEMORILLA_PYTHON=/path/to/python
make memorilla-benchmark MEMORILLA_PYTHON=/path/to/python
make memorilla-stress MEMORILLA_PYTHON=/path/to/python

The mapped setup is listed with the current results. Stress can replay a saved trace. Match batch size, scheduling, engine options, tokenizer, checkpoint and data revisions when comparing paths. Different numerical batches can change answers even when the input tensors match. The runner documents in-process and complete-batch admission controls. The author and mapped environment locks serve different runtime controls; changing both storage and runtime does not isolate a storage effect.

These are checkpoint inference commands, not a reproduction of the CPU preparation timing table. The table excludes reader and decoder execution. The mapped GPU decoder gate in the results section checks exact outputs, not speed. Released-checkpoint inference also does not establish exact lineage to every paper score.

07 / Checkpoint evaluation

Use the release, report the measured scores.

The original pinned CUDA environment uses Torch 2.7.0, vLLM 0.9.1, Transformers 4.53.3 and datasets 4.1.1, with batch size 32 and an 80 GiB GPU. Prompt embeddings select the V0 engine in this version. All eight registered benchmarks completed with released checkpoints, original prompts and original scorers, without training.

BenchmarkMetricPaperObserved
PubMedQAAccuracy59.0059.00
FactKGAccuracy84.3383.51
TriviaQAExact match57.0054.00
NarrativeQAROUGE-L19.3119.24
LaMP-4ROUGE-L18.1516.59
LaMP-7ROUGE-L48.3746.70
PersonaMem-v2Short-answer accuracy44.6044.48
PersonalizationV4Short-answer accuracy66.6566.65

Scores are percentages; ROUGE-L uses the code’s 0–100 scale. There are 32,860 questions. Reader release f6b15af5, data c6b0d7ee and author source 005854d1 stay pinned. The inference and storage modules in clean source 8ecc3952 match the measured snapshot byte for byte.

Independent CPU rescoring reproduces the inspected generation metrics. Exact paper-checkpoint lineage remains unverified: the release card labels Stage 3 as epoch-00, while the recipe runs five epochs and the source evaluation example names epoch-04. Those labels alone do not establish which weights produced the paper table. The runner records the discrepancy and preserves the observed scores.

Read size depends on the selected collection. PubMedQA has a 20 KiB median collection and a 40 KiB maximum; NarrativeQA has a 6,975 KiB median and a 34,800 KiB maximum. These are question-weighted tensor payload sizes. The unchanged reader consumes every row of its selected collection. The author’s indexed mmap already skips other collections in a shard; our mapped path stores one artifact per collection. Both file paths use the Linux page cache. Logical bytes do not establish device-command sizes or a kernel-patch speedup.

08 / Mapped embedding reuse · October 30, 2026

Match equally cached native; keep the lifecycle.

The current path keeps document embeddings in file-backed mappings managed by the LMCache artifact SDK. A prepared reader binds the collection to its compatible consumer and retains a tensor view. Repeated uses return the same pointer. Python tensor and buffer references keep the mapping alive, so eviction can drop cache lookup ownership without invalidating a tensor already handed to the collator.

Mapped read
File-backed FP16 pagesLinux page cache supplies the selected collection
Reusable tensor referenceDirect view; reference-owned lifetime
Final BF16 reader batchCopy and cast valid rows; initialize padding

The author maps shard files and slices the selected collection, rather than reading the whole corpus for each question. Our path maps a typed artifact for each selected collection. Both use the Linux page cache and supply every vector in that collection to the unchanged reader. NarrativeQA contains roughly 2.41 GiB of vectors; the mapping exposes existing file-backed pages without another complete payload allocation. The final batch still needs its own allocation and copy/cast; this is not a claim of zero-copy reader execution.

Identity, consumer and layout admission happen when a set is attached. After that, a collection’s first use opens and maps its file, and every later use returns the retained view. Under the normal published verification policy, serving makes no catalog query and no payload scan: in every timed mapped worker below, the replay made zero catalog queries, zero buffered payload copies and zero checksum passes.

Where the implementation stands against the author

The table compares the current mapped implementation with three controls. Unchanged author is the original mmap loader and collator. Optimized native is the author’s mmap loader with our direct BF16 batch assembly and bounded copy/cast. Equally cached native adds the same per-collection view reuse to that native loader, with no LMCache, catalog or typed admission. Values are medians of seven matched throughput ratios. Smaller text gives the 95% paired block-bootstrap interval. Higher favors the mapped path.

DatasetMapped / unchanged authorMapped / optimized nativeMapped / equally cached native
PubMedQA
1 copy worker
1.054×
1.051–1.061
1.078×
1.062–1.095
0.999×
0.985–1.005
PubMedQA
8 copy workers
1.191×
1.174–1.204
1.081×
1.066–1.097
1.005×
0.994–1.037
NarrativeQA
1 copy worker
2.697×
2.639–2.732
0.989×
0.986–1.010
1.008×
0.982–1.009
NarrativeQA
8 copy workers
4.118×
4.087–4.360
0.989×
0.985–1.014
1.004×
0.995–1.043

The current path beats the unchanged author in both suites. All four paired intervals for that comparison are above one. Against equally cached native views it is a wash. All four intervals span one and none falls below 0.982. With one copy worker the whole interval lies within ±2%; with eight copy workers the interval reaches 1.04 on the high side, so equivalence within 2% is not shown there, but no case shows the mapped path slower. We count a span of one as inconclusive, not as a win.

The gains therefore have two sources that native code can share. View reuse is worth about 1.08× on PubMedQA’s small collections (equally cached native over optimized native) and nothing measurable on NarrativeQA. Direct BF16 assembly provides most of the NarrativeQA gain over the author. What LMCache adds is the artifact lifecycle: immutable identity, consumer and layout admission, publication-time verification, bounded lookup ownership and reference-owned views. The cost of that lifecycle no longer shows up in throughput.

Median mapped throughput is 7,450 / 6,622 PubMedQA rows/s and 584 / 877 NarrativeQA rows/s with one/eight copy workers; equally cached native is 7,466 / 6,586 and 589 / 877. Ratios above are computed within matched blocks, not by dividing these separately computed medians. The metric is complete CPU batch preparation; imports, setup, the exact oracle and diagnostic hashes are outside its timed intervals.

Tail and setup tell the same story. The 95th-percentile batch time of the mapped path is within 1.3% of equally cached native in every case, and every interval spans one. Setup is the one remaining cost: about 0.63 s per process against 0.35 s for native, from importing the LMCache SDK and hashing the 388 MB reader checkpoint once to bind the imported set to its exact weights.

What was tested

The current implementation and the three controls cover seven paired process blocks with rotating arm order on one multicore x86-64 host: 112 workers and 516,096 byte-exact input rows. Each worker uses seed 23 and batch size four, with 8,192 PubMedQA or 1,024 NarrativeQA requests. Every replay covers all 100 or 355 collections. PyTorch uses one CPU thread per process; one/eight copy workers means at most four current-batch copy tasks, not independent online decoder clients. The runtime is Python 3.13, Torch 2.13.0, Transformers 5.20.2 and stock vLLM 0.29.0; the decoder does not run in these timings. The host was shared with another GPU job, so compare the paired ratios rather than absolute rates.

All embedding-file pages are initially resident. All 112 workers show no process read-byte increase and no major faults during the replay. No cold-storage, kernel-patch or physical-NVMe performance result follows.

The released stage3 reader checkpoint also passes 64 CPU BF16 comparison cases on this revision: 16 unique question rows per suite, repeated with one/eight copy workers, including the smallest and largest collections. Complete collated inputs, reader outputs and native-repeat outputs match byte for byte. Another 120 integration tests and 20 LMCache contract tests pass, including a concurrent capacity-wait regression under both verification policies. A descriptor test makes 8,192 NarrativeQA borrows over all 355 collections with 16 map slots and a 64-descriptor process limit: 11 descriptors at all 33 sampled boundaries and after close. A retained tensor survives eviction and close.

These checks qualify the CPU input and reader path. A separate GPU decoder gate on one 24 GiB NVIDIA GPU also passes. Six fresh processes per dataset ran 1,536 requests in total: native with one and eight copy workers, and mapped with one and eight copy workers, with eviction after every batch, and with strict on-map verification. Within each dataset, every pair of cases (30 pairs in all) produced the same generated strings and the same batch, reader-output and decoder-prompt tensor hashes. That gate checks correctness only and measures no speed. There is no full paper-score reproduction or completed-inference speed measurement. Exact paper-checkpoint lineage remains unverified. No training is required.

Verification and memory limits

Publication always checks the full payload before registering it. Normal serving explicitly selects published verification: trust those verified bytes and the catalog record admitted at attach, and avoid another scan or catalog query on each new mapping. The inspected author loader does not perform our equivalent per-reload artifact checksum either. Strict on-map verification remains available: it rechecks the catalog record and hashes each newly created mapping. Reusing an existing mapped view is not a new mapping. Identity, consumer compatibility, file extent and typed layout are admitted before use in both modes.

Backing files and borrowed tensor values must stay unchanged. Published mode does not detect later payload corruption or catalog edits made outside the catalog API. The mapping uses private copy-on-write storage so tensor libraries can expose its buffer; consumers must not modify it. It reads existing file-backed pages without allocating another complete payload buffer. The final BF16 batch and bounded conversion scratch remain separate allocations.

The default 512 mapping entries bound cache lookup ownership. Prepared readers can retain up to 1,024 collection references outside that budget, and callers can retain their own tensors. Linux controls physical page residency. Neither entry limit is a total RAM budget. Process RSS includes the native oracle and different mappings; no total-host-memory win is certified.

Use the public dated implementation

The dated knlp runner supplies memorilla-lmcache-mapped. The dated public LMCache branch supplies the generic mapped backing, pinned at b5200661. This is the fork’s external-artifact SDK, not stock token-keyed KV storage or an accepted upstream API. The mapped instructions include the matching environment lock and build steps.

make defconfig DEFCONFIG=memorilla-lmcache-mapped
make menuconfig
make memorilla-fetch MEMORILLA_PYTHON=/path/to/python
make memorilla-import MEMORILLA_PYTHON=/path/to/python
# Select artifact mode "attach" in menuconfig after import.
make menuconfig
make memorilla-stress MEMORILLA_PYTHON=/path/to/python

These commands run released-checkpoint inference and stress, not a CPU-only replay of the preparation table. That comparison times complete collator inputs separately from the reader and decoder. Use the mapped environment lock and a fresh configured work root, or prepare the exact dependency revisions in an existing root. Import verifies and publishes the collection set once; attach uses its pinned receipt. Select CONFIG_KNLP_MEM_ARTIFACT_VERIFICATION="on-map" for strict mapping validation. Use the original-author preset for its pinned baseline. The model, data, scoring and full-collection reader semantics remain unchanged.

09 / Primary sources

Read the methods and the implementation.

  1. Memorilla and Cartridges on clinical records: answer quality on LongHealth against trained cartridges and retrieval at equal KV memory.
  2. Learned Prefix Caches in knlp — the trained KV-prefix architecture and its existing serving work.
  3. Memorilla: Latent Semantic Memory for LLMs, v1, §2 and §7 — persistent vectors, query-conditioned readout, frozen-model interface, and limitations. Author code.
  4. REFRAG: Rethinking RAG based Decoding, v2, §2–3 and Appendix A.1 — chunk embeddings, decoder alignment, and selective expansion.
  5. Current mapped contract and the file-artifact implementation — typed consumers, file ranges and mapped ownership.