AINewsnow

Why Q/K/V changes from [1,16,384] to [1,4,16,32] (runnable PyTorch example)

The 3 and the 4 mean different things here: three projections, four attention heads. Start with one sequence of 16 tokens, each represented by 128 features. A single linear layer produces Q, K and V together: 3 × 128 = 384 features per token. Each projection then gets split into 4 heads of 32 featu…

Read the full story at r/learnmachinelearning ↗

Timeline · 1 report

  1. 2026-10-11 13:42 · r/learnmachinelearning
    Why Q/K/V changes from [1,16,384] to [1,4,16,32] (runnable PyTorch example)

More stories

  1. An Anthropic AI model sent a false homicide tip to Philadelphia police — TechCrunch AI
  2. Qwen Image 2.1 Turbo Released -- Hugging Face — r/StableDiffusion
  3. Microsoft CEO Nadella Calls for ‘Emergency Brake’ on Advanced AI — Bloomberg AI
  4. Microsoft unveils Microsoft-Decision-1, a fast decision-scoring model trained on Qwen3.5-9B, and says it will soon rebase it on MAI, OpenAI, and other models (Achint Srivastava/Command Line) — Techmeme
  5. Daily Driving Qwen 3.8 Flash-Next MoE (NVFP4) on RTX 5090 + 128GB RAM — Telemetry & Impressions — r/LocalLLM
  6. Microsoft's Nadella says AI needs an ‘emergency brake’ that humans control — CNBC Technology
  7. Philadelphia police receive false homicide tip from Anthropic AI model — The Hill Technology
  8. How Oracle Uses Codex to Help Business Users Get Answers — OpenAI YouTube

Get the daily brief of stories like this at 6:30 every morning →