Skip to content

AI R&D · Fisher information

One signal.
Several research questions.

The diagonal Fisher Information Matrix (FIM) uses expected squared gradients as a sensitivity signal. These experiments explore how that signal can guide pruning, precision and placement, including estimates drawn from Adam’s second-moment state.

This work helped open the path to KV quantization and asymmetric key/value precision. An important negative finding changed the approach: the Fisher signals we tested did not significantly predict which layers would suffer from KV quantization. We turned to measuring that damage directly.

From Fisher signals to K/V precision

A useful research path can start with a predictor that does not hold. These results concern KV-cache sensitivity; weight precision is a separate experiment.

Test the predictor

Fisher ranking did not predict KV damage

On Qwen2.5-7B and Mistral-7B, the measured Fisher ranking had no significant correlation with the perplexity change from quantizing one layer’s KV cache: Spearman ρ = −0.19 (p = 0.333) and 0.12 (p = 0.510). A follow-up cross-layer interaction signal also failed to predict sensitivity.

These tests did not establish a useful predictor. They do not prove that every Fisher-based method must fail.

Follow the applications

A shared statistic does not imply a shared outcome. Each study defines its own model, baseline and measured result.

Pruning

Adam state-based pruning

Larger-scale training untested

Use weight magnitude and the optimizer’s second moment to rank parameters. The positive 150M from-scratch comparison pruned during training; it was not repeated at larger scale. An untested scale-up is not a failed one.

Latent cache compression

KVSplice

Explore Fisher-guided compression on top of Multi-head Latent Attention, with quality and compression evaluated together.

Attention mechanisms

Reciprocal attention

Closed · scaling not established

The validated positive comparison was GPT-2 small (~124M). Four matched single-seed 1B runs were neutral within noise; the final matched training program found no advantage over paired baselines. The closed study preserves the positive result and its limits.