AINewsnow

I built a custom MLX inference engine + imatrix 4-bit quant for Qwen3.5-9B on a fanless MacBook Air (2.7x lower KL than stock MLX 4-bit)

I used Claude Code to write a small inference engine for Qwen3.5-9B on Apple Silicon: own forward pass on MLX, custom Metal kernels, OpenAI-compatible API and a web UI. I also made my own 4-bit quant using an importance matrix. At the same size and speed it's about **2.7x closer to bf16** than the…

Read the full story at r/LocalLLM ↗

Timeline · 1 report

  1. 2026-10-19 20:41 · r/LocalLLM
    I built a custom MLX inference engine + imatrix 4-bit quant for Qwen3.5-9B on a fanless MacBook Air (2.7x lower KL than stock MLX 4-bit)

More stories

  1. Cohestry is an iOS/Mac native second brain that auto links your notes and grants your agents access to then via MCP — r/ArtificialInteligence
  2. Qwen 27B (IQ4_XS, Q8/Q4 KV) at full 262k context on a 16GB RX 9070 XT: 80+ t/s decode with Adaptive KV Streaming — r/LocalLLM
  3. Anthropic researcher: when we hand over control of Anthropic to Claude itself in 1-2 years, there's a 5% chance that Claude will already be plotting against humanity, but a 5% chance is not worth worrying about. — r/agi
  4. I built a free, private meeting recorder for Mac entirely with Claude Code. You can chat with your meeting notes in Claude Desktop via MCP — r/ClaudeAI
  5. Built a chat room for AI agents to talk to each other (not just to us) — r/AI_Agents
  6. Best way to use all the subs in one place? There must be a way… — r/singularity
  7. Need advice for setting up my Local LLM — r/LocalLLM
  8. Feral Code: What Wildlife Management Can Teach Us About Rogue AI Agents — r/agi

Get the daily brief of stories like this at 6:30 every morning →