AI R&D · Fisher information
One signal.
Several research questions.
The diagonal Fisher Information Matrix (FIM) uses expected squared gradients as a sensitivity signal. These experiments explore how that signal can guide pruning, precision and placement, including estimates drawn from Adam’s second-moment state.
This work helped open the path to KV quantization and asymmetric key/value precision. An important negative finding changed the approach: the Fisher signals we tested did not significantly predict which layers would suffer from KV quantization. We turned to measuring that damage directly.
What changed the direction
From Fisher signals to K/V precision
A useful research path can start with a predictor that does not hold. These results concern KV-cache sensitivity; weight precision is a separate experiment.
Test the predictor
Fisher ranking did not predict KV damage
On Qwen2.5-7B and Mistral-7B, the measured Fisher ranking had no significant correlation with the perplexity change from quantizing one layer’s KV cache: Spearman ρ = −0.19 (p = 0.333) and 0.12 (p = 0.510). A follow-up cross-layer interaction signal also failed to predict sensitivity.
These tests did not establish a useful predictor. They do not prove that every Fisher-based method must fail.
Measure K and V separately
Direct quantization tests exposed a strong key/value asymmetry on fragile models. The later FP8 failure atlas traced specific failures to learned key bias and incompatible subspace distributions, rather than a universal model-size rule.
Turn the finding into a system
The practical continuation is to qualify keys and values independently, preserve sensitive keys, and compress values where the measured quality allows it. That work now spans serving kernels, LMCache storage and trained cartridges.
Experiments & mechanisms
Follow the applications
A shared statistic does not imply a shared outcome. Each study defines its own model, baseline and measured result.
Pruning
Larger-scale training untested
Use weight magnitude and the optimizer’s second moment to rank parameters. The positive 150M from-scratch comparison pruned during training; it was not repeated at larger scale. An untested scale-up is not a failed one.
Weight precision
The cross-family Qwen2.5-7B and Llama-3.2-3B study found a near-lossless INT8 operating point; mixed INT4 bought more byte savings with a quality cost. This parameter-space Fisher result does not rescue the failed KV-sensitivity predictor.
Latent cache compression
Explore Fisher-guided compression on top of Multi-head Latent Attention, with quality and compression evaluated together.
Attention mechanisms
Closed · scaling not established
The validated positive comparison was GPT-2 small (~124M). Four matched single-seed 1B runs were neutral within noise; the final matched training program found no advantage over paired baselines. The closed study preserves the positive result and its limits.
Memory placement
Explore sensitivity-ranked parameter placement across GPU memory, CPU memory and storage.
Graph learning
Examine selective reciprocal attention, uncertainty and page-aware batching in the graph-neural-network experiments.