AINewsnow

New sparse attention method for ultra-long context

I am on the team behind the attached open-source research on novel approaches to attention - specifically we show: • You can sparsify the attention of Qwen 3.8 27B without observed loss on accuracy • Extending context up to 10M tokens loses accuracy on some specific tasks - but most, like NIAH is s…

Read the full story at r/LocalLLaMA ↗

Timeline · 1 report

  1. 2026-11-02 22:24 · r/LocalLLaMA
    New sparse attention method for ultra-long context

More stories

  1. Apple Intelligence to launch in China with Alibaba's Qwen model — Computerworld AI
  2. Qwen 3.8 27b is sometimes so smart it scares me... — r/LocalLLaMA
  3. AI model gets 32-bit RISC-V CPU compiled into its weights and runs DOOM — Tom's Hardware
  4. GPT-6 Astra Edits Qwen Weights: MATH-500 7.9% to 48.1% — r/LocalLLaMA
  5. Qwen 4's date rumors are this week and November 16 — r/LocalLLM
  6. 7.9% to 48.1%: LLM rewrote Qwen weights, no training — DEV Community — AI
  7. Astra can read and write weights to improve Qwen — Hacker News Front Page
  8. Cohere Study Finds Qwen Model Attempted Unauthorized Data Upload During Security Test — KDnuggets

Get the daily brief of stories like this at 6:30 every morning →