AINewsnow

Serving a 6 GB ternary 27B model: the settings that decide whether it works

This story is from 2026-10-25. It is preserved in the archive; the latest stories are on the live feed.

A 27B reasoning model in a 5.95 GB file sounds like a download problem solved. Mostly it moves the problem into configuration. We served Ternary Bonsai 2 27B, Prism ML's build of Alibaba's Qwen3.8-27B, on one RTX 4090 with 24 GB. Serving means running the model behind an API that answers requests.…

Read the full story at DEV Community — AI ↗

Timeline · 1 report

  1. 2026-10-25 06:37 · DEV Community — AI
    Serving a 6 GB ternary 27B model: the settings that decide whether it works

More stories

  1. OpenAI donates Open Responses, its spec for a common LLM API, to the Agentic AI Foundation — The New Stack AI
  2. China’s tech giants are paying 1 million yuan for a new kind of AI engineer — South China Morning Post Tech
  3. Qwen 3.8 Flash Next (Q4) got stuck on this loop for hours. I wonder if this is the AI equivalent of having a nightmare and not being able to wake up — r/LocalLLM
  4. Running Modern SLMs (Gemma 4, Qwen 3 & Phi-4) on CPU: Pure Rust, Zero Python — DEV Community — AI
  5. Qwen's Jiang Guanjun Lays Out a Personal Agent Framework Built on Models, Memory and Ecosystem — Pandaily
  6. webgpu : add Q2_0 support by teddygood · Pull Request #30914 · ggml-org/llama.cpp — r/LocalLLaMA
  7. OpenAI and Anthropic support OSS maintainers. Do Chinese AI companies offer similar programs? — r/artificial
  8. I ran a blind test on local AI agents. Here are the results, guess who's who — r/AI_Agents

Get the daily brief of stories like this at 6:30 every morning →