RoPE in bfloat16: 8,192 Token Positions Collapsed Into 897

Cast the integers 8,190 through 8,200 to bfloat16 and you get eleven copies of 8,192. That one line is why RoPE in bfloat16 is a trap. Rotary position embeddings tell attention where each token sits, and the number format most LLMs run in can't count past 256 without rounding. There's no NaN and no…