AINewsnow

I trained a small MoE model from scratch that runs on a single RTX 3060 (0.38B active params)

Hi! I've been training a language model from scratch on free compute, and I wanted to share how it runs locally, since that was the whole point. Lewi Surge Preview 3B is a mixture-of-experts model: 3.45B parameters total, but only ~0.38B active per token. In bf16 it's about 7 GB, so it fits on one…

Read the full story at r/LocalLLM ↗

Timeline · 1 report

  1. 2026-10-16 21:17 · r/LocalLLM
    I trained a small MoE model from scratch that runs on a single RTX 3060 (0.38B active params)

More stories

  1. Meta releases Muse Spark Open, an 816B-parameter open-weight AI model — AWS Machine Learning Blog
  2. Anthropic pitches AI safety bill to limit government model blocks — Politico Technology
  3. NVIDIA RTX Spark Laptops Launch Globally With AI Agent Support — NVIDIA Blog
  4. Replit CEO Says Company No Longer Works With Anthropic — TechCrunch AI
  5. From training to serving: Reflection’s Beam on CoreWeave — CoreWeave Blog
  6. AWS introduces evaluation framework for AI agents on Bedrock — AWS Machine Learning Blog
  7. How Itaú is building generative AI for its entire customer base — OpenAI News
  8. Reflection AI releases open-weight Beam model for agentic coding — AWS Machine Learning Blog

Get the daily brief of stories like this at 6:30 every morning →