Big local MoE on edge with an adaptive streaming optimizer: any model and any device, any OS, any HW, local OpenAI-compatible server. Android, Windows, Mac, IOS, Linux. CPU, GPU, NPU (35B on phone 12gb RAM 10 tok/s decode 100-300+ prefill, Windows, Mac even more:+30 tok/s 16 gb RAM)

I am working on making AI accessible by running large MoE models on affordable, consumer-grade hardware that people already own: smartphones, laptops,normal desktops. Models several times bigger than RAM, no big GPU needed. Disclosure: I’m the author of BigMoeOnEdge. All numbers below are real runs…

Read the full story at r/LocalLLM ↗

Timeline · 1 report

1. 2026-10-19 17:11 · r/LocalLLM
   Big local MoE on edge with an adaptive streaming optimizer: any model and any device, any OS, any HW, local OpenAI-compatible server. Android, Windows, Mac, IOS, Linux. CPU, GPU, NPU (35B on phone 12gb RAM 10 tok/s decode 100-300+ prefill, Windows, Mac even more:+30 tok/s 16 gb RAM)