RoPE vs sinusoidal positional encoding, measured: 55 logits of drift against 0.0005
This story is from 2026-10-01. It is preserved in the archive; the latest stories are on the live feed.
Take two token embeddings. Keep them exactly five positions apart and slide the pair down a 2048-token sequence. Under sinusoidal absolute position encoding the attention score for that one unchanging pair runs from -33.91 to +21.61 — a 55.5-logit spread that changes sign 157 times on the way. Unde…
Read the full story at DEV Community — Machine Learning ↗
Timeline · 1 report
- 2026-10-01 11:15 · DEV Community — Machine Learning
RoPE vs sinusoidal positional encoding, measured: 55 logits of drift against 0.0005