Why RoPE doesn't just work at 32K when trained at 4K

In the RoPE lesson we earned the key property: the attention score $\langle R_m Q, R_n K \rangle$ depends only on the offset $n - m$. That is a statement about relative position. It is not a statement about extrapolation.

Train a model at 4K tokens. The dim-pair frequencies $\theta_i = 10000^{-2i/d}$ produce angular trajectories the weights have seen only up to $4096 \cdot \theta_i$ radians. Deploy at 32K and suddenly every key at position $n > 4096$ sits at rotation phases the model has never observed during training. The softmax over scores was calibrated for the trained angular range; outside it, scores drift off into nonsense. Output collapses almost immediately past the training length.

There is a second, subtler failure. The low-frequency dim pairs (large $i$, tiny $\theta_i$) rotate so slowly that across 4K tokens they only sweep a small arc. Stretch the sequence to 32K without adapting and those pairs still only sweep a small arc — effectively the same phase for a token at position 100 and a token at position 20,000. The model loses its ability to distinguish far-apart positions at the dim pairs that were supposed to encode the coarsest positional scale.

Position Interpolation (Chen 2023) — the first attempt

The simplest thing you can do is linear. Before applying RoPE, rescale every position index by $\lambda = L_\text{train} / L_\text{target}$:

For an 8K → 32K extension, $\lambda = 1/4$. Four real tokens are packed into one virtual position. The model sees only angles in its trained range, but now four physical tokens share each angular slot.

A short fine-tune (a few hundred steps on long sequences) adapts the model to the denser angular grid, and perplexity at the new length drops to near-baseline. This is what Llama 2 used for its 4K → 32K extension. It works. Up to a point.

NTK-aware — preserve high-frequency resolution

The bloc97 insight: don't scale the position uniformly. Scale the base frequency instead, in a way that leaves the high-freq dims mostly alone and concentrates the interpolation on the low-freq dims. Concretely, replace the base 10,000 with a new base $b'$ chosen so the lowest-frequency pair still sweeps its full training arc at the new length:

For $s = 4, d = 128$, $b' \approx 40{,}970$ — about 4× the original base, which lifts the $\theta_i$ schedule of the far-right dims just enough to span the new range, while leaving the leftmost dims essentially unchanged.

On the hook plot, that's the amber curve: flat through the first few multipliers, slowly rising past 8×. NTK holds longer than PI because it keeps the high-freq dims' angular resolution intact. But it still relies on a single global recipe, and the transition between regimes is hand-tuned rather than optimised. At 32× it eventually degrades too.

YaRN — per-band scaling plus attention temperature

YaRN (Peng et al. 2023) is the clean formalisation. Instead of one global base-frequency knob, treat the RoPE dim pairs as a continuum and apply different scaling to different frequency bands. The split is set by wavelength thresholds:

• If $\text{wavelength}_i < L_\text{train}$ (high-freq) — interpolate. The dim already completes full rotations within the trained length; squishing it is safe.
• If $\text{wavelength}_i > L_\text{train}$ (low-freq) — extrapolate. The dim never completed a full rotation at trained length; its angles at longer contexts are still within the trained distribution.
• In between — smooth ramp. A cosine or linear transition, so there's no discontinuity at the band boundary.

YaRN adds one more detail that matters enormously at large multipliers: a temperature correction on the attention softmax. As sequences grow, the total mass the softmax distributes is spread across more keys, so the effective entropy rises. YaRN rescales logits by

to compensate — a tiny, closed-form fix that restores the softmax's sharpness at the new length. Without it, attention gets diffuse and the model averages rather than retrieves.

Attention sinks — where the softmax dumps its residual mass

Long-context extension solves the positional half of the problem. It does not solve streaming decode. If you want to keep generating past your trained length without accumulating an unbounded KV cache, the obvious move is a sliding window: keep the last $W$keys, evict everything older. Run that for a few hundred tokens and the model's output collapses into garbage.

The fix, discovered by Xiao et al. in September 2023, is almost comically simple: keep the first 4 tokens pinned, no matter how far the window has advanced. They call these attention sinks. Output stays fluent indefinitely.

The phenomenon is visible in the toggle above. With sinks pinned (copper glow on tokens 0–3), the rest of the window slides normally and the model produces fluent text. Evict the sinks and decode degenerates within tokens — every head is desperately trying to find some key to dump residual probability onto, but the first tokens are no longer there.

Production pairings, 2026

Every frontier model shipped in 2025–2026 uses some mixture of the techniques above. The recipes diverge on which combination is easiest to train, not on which is theoretically best — the perplexity curves for YaRN and NTK-with-long-fine-tune converge at modest multipliers, so labs pick what their existing pipelines support.

DeepSeek-V3(Dec 2024) takes the most layered approach: MLA's decoupled-RoPE slice carries the positional signal on a width-64 channel, and YaRN is applied to that slice alone. The shared latent $c^{KV}$is un-rotated, so YaRN's ramp only operates on the 64-dim RoPE side. This makes the long-context math independent of the KV-cache compression math — a rare piece of architectural luck.