Learned prefix
prefix_kvPer-layer key and value tensors for a particular model. Preserve exact logical positions, layer/head geometry, rotary-position settings, dtype, layout, and padding.
A learned key/value prefix, a document-vector collection, and a query’s soft memory tokens can share a storage contract. Each still needs its own model interface and cache identity.
This work extends the Learned Prefix Cache (LPC) architecture toward Memorilla’s latent memory. The shared catalog and LMCache mapped-artifact path supply document collections to the released Memorilla reader. The current qualification covers CPU input preparation, reader outputs and exact GPU decoder outputs. LPC keeps its own model attachment path.
Current result: the mapped LMCache path prepares PubMedQA inputs 1.05–1.19× faster and NarrativeQA inputs 2.70–4.12× faster than the author’s unchanged loader and collator. Against a native mmap loader given the same view reuse and batch assembly, it is within measurement noise in every case: 0.999–1.008×, with no interval below 0.982. LMCache adds artifact identity, admission and lifecycle at no measured throughput cost; it is not a speedup over equally optimized native code. These are CPU preparation measurements. See the matched results and reproduction commands.
A context artifact is a versioned tensor payload plus the information needed to use it correctly. Its immutable ID binds the payload digest, representation, producer, and compatible consumers. File location and working-copy residency can change while that semantic identity stays fixed.
prefix_kvPer-layer key and value tensors for a particular model. Preserve exact logical positions, layer/head geometry, rotary-position settings, dtype, layout, and padding.
document_embeddingsVectors produced by a pinned encoder and preprocessing pipeline. One collection contains a vector for each assigned text entry; different questions can reuse it. A compatible memory reader consumes these vectors. They do not enter the decoder’s key/value cache directly.
input_embeddingsVectors in the decoder’s input space. Bind their reader, collection, question, prompt placement, and positions. Prefill must still turn the assembled input into layer-specific key/value state.
The file registry implements import, resolve, load, working-copy eviction, and restore. It validates safetensors before synchronizing payload files and directories, then publishes metadata in a synchronous SQLite transaction. Missing, corrupt, or incompatible required artifacts fail explicitly. Existing cartridge connector APIs remain unchanged.
The current mapped qualification exercises document tensors, batch assembly and the released reader on CPU, and checks exact decoder outputs on GPU. LPC engine attachment requires its own qualification. Newly computed readouts do not need a mandatory storage round trip.
Memorilla reads a query-independent document-vector collection with a learned, query-conditioned module. Its released Qwen configuration emits 16 input-space memory tokens for a frozen decoder. Reading the whole selected collection is part of that method; replacing it with top-k retrieval would change the algorithm.
REFRAG, from Meta and academic collaborators, encodes fixed token chunks, projects their vectors into decoder input space, and can expand selected chunks back to text. Its reconstruction stage freezes the decoder; later continual pretraining unfreezes it. Its mixed-input path therefore needs the corresponding trained decoder, encoder, projection, and policy.
| Question | Memorilla | REFRAG |
|---|---|---|
| Reusable unit | Document embeddings | Encoded chunks |
| Query-time context | Fixed-size learned memory readout | Ordered mixture of compressed chunks and expanded text |
| Decoder dependency | Frozen decoder paired with the trained reader | Decoder aligned through the paper’s training recipe |
| Context budget | Fixed memory-token count | Depends on chunk count, compression, and expansion decisions |
These are method differences, not a matched accuracy or speed comparison. The papers use different models, tasks, training, and controls. Neither paper’s headline performance is a measured result of this implementation.
Our target is a common lifecycle for context representations: publish verified bytes, resolve an immutable handle, load the representation its consumer expects, and recover it after eviction or restart. Training remains a producer’s separate responsibility. A missing LPC must not launch training, and storage must not disguise document vectors as layer KV.
Establish the real LPC engine/checkpoint baseline and the unmodified Memorilla decoder path. Keep engine-version changes separate from artifact/storage changes.
Use the existing LPC attachment for KV and exact template-aware input embeddings for Memorilla. Retain engine ownership of allocations and scheduling.
Test equal-length different payloads, different questions over one collection, native and external cache keys, and actual inference after GPU eviction and process restart. File parity is the prerequisite already explored.
Separate encoding, readout, serialization, storage completion, transfer, prefill, and decode. Compare matched tensors and cache states before making a storage or end-to-end speed claim.
A possible REFRAG adapter belongs at the chunk-collection and mixed-input boundaries, with its own typed metadata. Whether that adapter is worthwhile needs a separate compatibility and quality evaluation. It is not part of the CPU pass reported here.
The public runner supplies memorilla-author for the pinned author environment and memorilla-lmcache-mapped for the current integration. Both use released weights; no training is required. The default benchmark is the complete 100-question PubMedQA test. Select all for the eight supervised benchmarks.
make defconfig DEFCONFIG=memorilla-author
make menuconfig
make memorilla-fetch MEMORILLA_PYTHON=/path/to/python
make memorilla-benchmark MEMORILLA_PYTHON=/path/to/python
make memorilla-stress MEMORILLA_PYTHON=/path/to/python
The mapped setup is listed with the current results. Stress can replay a saved trace. Match batch size, scheduling, engine options, tokenizer, checkpoint and data revisions when comparing paths. Different numerical batches can change answers even when the input tensors match. The runner documents in-process and complete-batch admission controls. The author and mapped environment locks serve different runtime controls; changing both storage and runtime does not isolate a storage effect.
These are checkpoint inference commands, not a reproduction of the CPU preparation timing table. The table excludes reader and decoder execution. The mapped GPU decoder gate in the results section checks exact outputs, not speed. Released-checkpoint inference also does not establish exact lineage to every paper score.
The original pinned CUDA environment uses Torch 2.7.0, vLLM 0.9.1, Transformers 4.53.3 and datasets 4.1.1, with batch size 32 and an 80 GiB GPU. Prompt embeddings select the V0 engine in this version. All eight registered benchmarks completed with released checkpoints, original prompts and original scorers, without training.
| Benchmark | Metric | Paper | Observed |
|---|---|---|---|
| PubMedQA | Accuracy | 59.00 | 59.00 |
| FactKG | Accuracy | 84.33 | 83.51 |
| TriviaQA | Exact match | 57.00 | 54.00 |
| NarrativeQA | ROUGE-L | 19.31 | 19.24 |
| LaMP-4 | ROUGE-L | 18.15 | 16.59 |
| LaMP-7 | ROUGE-L | 48.37 | 46.70 |
| PersonaMem-v2 | Short-answer accuracy | 44.60 | 44.48 |
| PersonalizationV4 | Short-answer accuracy | 66.65 | 66.65 |
Scores are percentages; ROUGE-L uses the code’s 0–100 scale. There are 32,860 questions. Reader release f6b15af5, data c6b0d7ee and author source 005854d1 stay pinned. The inference and storage modules in clean source 8ecc3952 match the measured snapshot byte for byte.
Independent CPU rescoring reproduces the inspected generation metrics. Exact paper-checkpoint lineage remains unverified: the release card labels Stage 3 as epoch-00, while the recipe runs five epochs and the source evaluation example names epoch-04. Those labels alone do not establish which weights produced the paper table. The runner records the discrepancy and preserves the observed scores.
Read size depends on the selected collection. PubMedQA has a 20 KiB median collection and a 40 KiB maximum; NarrativeQA has a 6,975 KiB median and a 34,800 KiB maximum. These are question-weighted tensor payload sizes. The unchanged reader consumes every row of its selected collection. The author’s indexed mmap already skips other collections in a shard; our mapped path stores one artifact per collection. Both file paths use the Linux page cache. Logical bytes do not establish device-command sizes or a kernel-patch speedup.
The current path keeps document embeddings in file-backed mappings managed by the LMCache artifact SDK. A prepared reader binds the collection to its compatible consumer and retains a tensor view. Repeated uses return the same pointer. Python tensor and buffer references keep the mapping alive, so eviction can drop cache lookup ownership without invalidating a tensor already handed to the collator.
The author maps shard files and slices the selected collection, rather than reading the whole corpus for each question. Our path maps a typed artifact for each selected collection. Both use the Linux page cache and supply every vector in that collection to the unchanged reader. NarrativeQA contains roughly 2.41 GiB of vectors; the mapping exposes existing file-backed pages without another complete payload allocation. The final batch still needs its own allocation and copy/cast; this is not a claim of zero-copy reader execution.
Identity, consumer and layout admission happen when a set is attached. After that, a collection’s first use opens and maps its file, and every later use returns the retained view. Under the normal published verification policy, serving makes no catalog query and no payload scan: in every timed mapped worker below, the replay made zero catalog queries, zero buffered payload copies and zero checksum passes.
The table compares the current mapped implementation with three controls. Unchanged author is the original mmap loader and collator. Optimized native is the author’s mmap loader with our direct BF16 batch assembly and bounded copy/cast. Equally cached native adds the same per-collection view reuse to that native loader, with no LMCache, catalog or typed admission. Values are medians of seven matched throughput ratios. Smaller text gives the 95% paired block-bootstrap interval. Higher favors the mapped path.
| Dataset | Mapped / unchanged author | Mapped / optimized native | Mapped / equally cached native |
|---|---|---|---|
| PubMedQA 1 copy worker | 1.054× 1.051–1.061 | 1.078× 1.062–1.095 | 0.999× 0.985–1.005 |
| PubMedQA 8 copy workers | 1.191× 1.174–1.204 | 1.081× 1.066–1.097 | 1.005× 0.994–1.037 |
| NarrativeQA 1 copy worker | 2.697× 2.639–2.732 | 0.989× 0.986–1.010 | 1.008× 0.982–1.009 |
| NarrativeQA 8 copy workers | 4.118× 4.087–4.360 | 0.989× 0.985–1.014 | 1.004× 0.995–1.043 |
The current path beats the unchanged author in both suites. All four paired intervals for that comparison are above one. Against equally cached native views it is a wash. All four intervals span one and none falls below 0.982. With one copy worker the whole interval lies within ±2%; with eight copy workers the interval reaches 1.04 on the high side, so equivalence within 2% is not shown there, but no case shows the mapped path slower. We count a span of one as inconclusive, not as a win.
The gains therefore have two sources that native code can share. View reuse is worth about 1.08× on PubMedQA’s small collections (equally cached native over optimized native) and nothing measurable on NarrativeQA. Direct BF16 assembly provides most of the NarrativeQA gain over the author. What LMCache adds is the artifact lifecycle: immutable identity, consumer and layout admission, publication-time verification, bounded lookup ownership and reference-owned views. The cost of that lifecycle no longer shows up in throughput.
Median mapped throughput is 7,450 / 6,622 PubMedQA rows/s and 584 / 877 NarrativeQA rows/s with one/eight copy workers; equally cached native is 7,466 / 6,586 and 589 / 877. Ratios above are computed within matched blocks, not by dividing these separately computed medians. The metric is complete CPU batch preparation; imports, setup, the exact oracle and diagnostic hashes are outside its timed intervals.
Tail and setup tell the same story. The 95th-percentile batch time of the mapped path is within 1.3% of equally cached native in every case, and every interval spans one. Setup is the one remaining cost: about 0.63 s per process against 0.35 s for native, from importing the LMCache SDK and hashing the 388 MB reader checkpoint once to bind the imported set to its exact weights.
The current implementation and the three controls cover seven paired process blocks with rotating arm order on one multicore x86-64 host: 112 workers and 516,096 byte-exact input rows. Each worker uses seed 23 and batch size four, with 8,192 PubMedQA or 1,024 NarrativeQA requests. Every replay covers all 100 or 355 collections. PyTorch uses one CPU thread per process; one/eight copy workers means at most four current-batch copy tasks, not independent online decoder clients. The runtime is Python 3.13, Torch 2.13.0, Transformers 5.20.2 and stock vLLM 0.29.0; the decoder does not run in these timings. The host was shared with another GPU job, so compare the paired ratios rather than absolute rates.
All embedding-file pages are initially resident. All 112 workers show no process read-byte increase and no major faults during the replay. No cold-storage, kernel-patch or physical-NVMe performance result follows.
The released stage3 reader checkpoint also passes 64 CPU BF16 comparison cases on this revision: 16 unique question rows per suite, repeated with one/eight copy workers, including the smallest and largest collections. Complete collated inputs, reader outputs and native-repeat outputs match byte for byte. Another 120 integration tests and 20 LMCache contract tests pass, including a concurrent capacity-wait regression under both verification policies. A descriptor test makes 8,192 NarrativeQA borrows over all 355 collections with 16 map slots and a 64-descriptor process limit: 11 descriptors at all 33 sampled boundaries and after close. A retained tensor survives eviction and close.
These checks qualify the CPU input and reader path. A separate GPU decoder gate on one 24 GiB NVIDIA GPU also passes. Six fresh processes per dataset ran 1,536 requests in total: native with one and eight copy workers, and mapped with one and eight copy workers, with eviction after every batch, and with strict on-map verification. Within each dataset, every pair of cases (30 pairs in all) produced the same generated strings and the same batch, reader-output and decoder-prompt tensor hashes. That gate checks correctness only and measures no speed. There is no full paper-score reproduction or completed-inference speed measurement. Exact paper-checkpoint lineage remains unverified. No training is required.
Publication always checks the full payload before registering it. Normal serving explicitly selects published verification: trust those verified bytes and the catalog record admitted at attach, and avoid another scan or catalog query on each new mapping. The inspected author loader does not perform our equivalent per-reload artifact checksum either. Strict on-map verification remains available: it rechecks the catalog record and hashes each newly created mapping. Reusing an existing mapped view is not a new mapping. Identity, consumer compatibility, file extent and typed layout are admitted before use in both modes.
Backing files and borrowed tensor values must stay unchanged. Published mode does not detect later payload corruption or catalog edits made outside the catalog API. The mapping uses private copy-on-write storage so tensor libraries can expose its buffer; consumers must not modify it. It reads existing file-backed pages without allocating another complete payload buffer. The final BF16 batch and bounded conversion scratch remain separate allocations.
The default 512 mapping entries bound cache lookup ownership. Prepared readers can retain up to 1,024 collection references outside that budget, and callers can retain their own tensors. Linux controls physical page residency. Neither entry limit is a total RAM budget. Process RSS includes the native oracle and different mappings; no total-host-memory win is certified.
The dated knlp runner supplies memorilla-lmcache-mapped. The dated public LMCache branch supplies the generic mapped backing, pinned at b5200661. This is the fork’s external-artifact SDK, not stock token-keyed KV storage or an accepted upstream API. The mapped instructions include the matching environment lock and build steps.
make defconfig DEFCONFIG=memorilla-lmcache-mapped
make menuconfig
make memorilla-fetch MEMORILLA_PYTHON=/path/to/python
make memorilla-import MEMORILLA_PYTHON=/path/to/python
# Select artifact mode "attach" in menuconfig after import.
make menuconfig
make memorilla-stress MEMORILLA_PYTHON=/path/to/python
These commands run released-checkpoint inference and stress, not a CPU-only replay of the preparation table. That comparison times complete collator inputs separately from the reader and decoder. Use the mapped environment lock and a fresh configured work root, or prepare the exact dependency revisions in an existing root. Import verifies and publishes the collection set once; attach uses its pinned receipt. Select CONFIG_KNLP_MEM_ARTIFACT_VERIFICATION="on-map" for strict mapping validation. Use the original-author preset for its pinned baseline. The model, data, scoring and full-collection reader semantics remain unchanged.