Reciprocal Attention evidence and status

RA mixes the standard attention routing softmax(Q@K.T) with the swapped routing softmax(K@Q.T) on selected heads. This page reports what has actually been measured at each scale, with each result's evidence status. Status: closed (2026-08) — a validated small-model result exists, matched 1B evidence is neutral, and a preregistered three-seed matched training matrix found no RA variant beating paired baselines, closing the line with a mechanistic account (sign-cancelling, high-rank reciprocal credit).

Documentation Evidence ledger

Validated small-scale result

The trusted RA result: a matched GPT-2 small (~124M) FineWebEdu comparison against a Qwen-style SDPA output-gate control, tracked in Weights & Biases (project gpt2-ra-sdpa-ablation).

Variant	Val PPL	HellaSwag
GPT-2 baseline	72.5	28.0%
Qwen-style SDPA gate	71.8	28.5%
Reciprocal Attention	68.9	30.0%

In this exact GPT-2 configuration, RA improved perplexity by ~5% and HellaSwag by 2 points while beating the gate control. Scope: subsequent matched 1B experiments did not establish a statistically clear benefit, so this result must not be extrapolated as a scaling law.

Scaling evidence

Every scale that was tested, and what each test can support. Full per-run details and limitations are in the evidence ledger.

Scale	Experiment	Result	Status
30M	GPT-2 TinyStories, 3 seeds, full RA vs "!"-gated RA	−0.47% / −0.46% PPL; gating to <0.4% of positions matched full RA	SUPPORTED_BUT_LIMITED
124M	GPT-2 vs SDPA gate, FineWebEdu (table above)	PPL 72.5 → 68.9, HellaSwag 28% → 30%	SUPPORTED (this config; scaling not established)
124M	FIM-trace arm, 3 seeds	best seed −1.87%; multi-seed mean −0.47% (~noise floor)	SUPPORTED_BUT_LIMITED
150M	1-hour matched RA-8, 4xH100, FLASH_ATTENTION parity	PPL 239.66 → 217.06 (−9.4%); single seed, artifact set incomplete	SUPPORTED_BUT_LIMITED
1B	four matched single-seed variants (two selectors, 8/28 heads)	per-step deltas −0.80% to +0.52%, σ ≈ 1% — all within noise	NEUTRAL
8B	cache-transform screen	excluded from evidence (methodology)	INVALIDATED
32B	uniform fixed-head placement, short exploratory run	negative, but no validated selector was tested	NON_DIAGNOSTIC

Reading this correctly. No robust scaling benefit was established. The matched 1B evidence is neutral within noise — which is not the same as "RA fails at 1B" or "RA tops out at 150M"; single-seed runs at ~1% σ cannot support either claim. The 8B screen was invalidated (its reported perplexity never measured the intervention), and the 32B experiment used uniform rather than validated surgical placement, so it carries no information about a working selector at that scale.

What was invalidated

The 8B cache-transform screen. Four independent faults: the intervention was a K-K cache transform rather than the reciprocal branch; the "RA-N" arms selected layers while the code called them heads; token agreement reused a mutable cache; and the reported perplexity came from a fresh model call that never consumed the transformed cache. Excluded in full.
The exact_eigmax head selector. The batch-mean post-softmax attention matrix is row-stochastic, so its largest eigenvalue is exactly 1 for every head (Perron–Frobenius); rankings from it were random tie-breaking.
An earlier version of this page showed a hardcoded comparison (baseline 282.1 / gate 223.7 / RA 50.5 PPL) that could not be traced to retained artifacts. Those numbers were removed and are not evidence.

Mechanism

Standard attention:

y_std = softmax(Q @ K.T / √d) @ V

Reciprocal routing (swapped):

y_rec = softmax(K @ Q.T / √d) @ V

Learned mix on selected heads:

y = y_std + tanh(ra_logit) · LN(y_rec)

ra_logit initialized to 0, so RA starts disabled

The raw score matrices Q@K.T and K@Q.T are transposes, but softmax normalizes over different axes (keys per query vs queries per key), so the two attention matrices encode different routing. Both branches respect the causal mask — this is alternate causal routing, not access to future tokens.

Selected heads compute an additional attention branch, so RA is not FLOP-neutral: historical GPT-2 runs measured a 12–18% per-iteration slowdown.

Why mixing might help is a hypothesis (complementarity between the two views); the experiments above measure outcomes, not mechanism. A view-divergence selector that would target the hypothesis directly is proposed but untested.

Separate line: RA-style aggregation in GNNs

A separate experiment applied reciprocal-style message-passing aggregation to graph neural networks for fraud detection on DGraphFin (+7% relative F1 over a GraphSAGE baseline; absolute F1 0.0782 → 0.0837). That is its own result with its own evidence, documented on the GNN fraud page — it is not evidence that transformer RA scales.