AINewsnow

Llama news

Every Llama story as it breaks — clustered from 259 sources, ranked by importance, updated every 10 minutes.

M2 Mac ultra128gb Qwen flash next

I am trying to get good speeds for Qwen flash next but I also need some ram for the system. Right now gpu is at 96gb around more I cannot use. Before I was getting 50 token/s from omlx and 60 tokens/s from mlx serve but lower context like 32k. I had ngram on ssd now where my system has more ram for…

Lead source: r/LocalLLM

  1. Qwen3.8-Flash-Next-Heretic2-IQ4XS on Halogen Flash Server vs llama-server on Strix Halo: 2.3-7.7x prefill speedup with half the VRAM (+ vision works on BYO GGUF) — r/LocalLLM
  2. Multi-hour llama.cpp optimization experiments on Qwen MoE models, patches, benchmarks, and reproduction guides — r/LocalLLM
  3. The bear can dance: Qwen 3.8 27B on one 3090 for 3 weeks — r/LocalLLaMA
  4. CUDA: enable sparse fa for qwen4 by am17an · Pull Request #28770 · ggml-org/llama.cpp — r/LocalLLaMA
  5. focus-llama: a llama.cpp fork implementing Declarative Attention (arXiv:2609.02737) — r/LocalLLaMA
  6. I ran Opencode and PI against the same local model on 3 identical projects, same prompts, same hardware... — r/LocalLLM
  7. I benchmarked 13 model/quant configs on a GPU with no tensor cores (Vega iGPU + Vulkan) and wrote it up as a measurement study — the quant encoding suffix matters more than you'd think — r/LocalLLM
  8. M1 Max 32GB, trying to run Qwen 3.8 27B at decent speeds and context — r/LocalLLaMA
  9. I turned an asymetric pair of Tesla V100s PCIe both (16 GB + 32 GB) into a surprisingly capable local LLM lab — 1.38k prompt tok/s, 40 decode tok/s with qwen3.8 27B Q6 and Q8... — r/LocalLLaMA
  10. [Guide / Weights] Qwen 3.8 27B on Intel Arc: Why IQ quants crawl at 8 tok/s, why Q4_K outpaces sub-4bpw on Battlemage, and clean RCO GGUFs (16GB & 24GB) — r/LocalLLM
  11. My Version of Jev running locally, playing doom. — r/LocalLLM
  12. Qwen3.8-27B on a single RTX 5090 (32GB) + 64GB DDR5-6000 — looking for real t/s numbers (llama.cpp / vLLM / sglang) — r/LocalLLM
  13. Digit-logits-based classifier with llama.cpp — r/LocalLLaMA
  14. Two node BC250 cluster comparison of Qwen3.6 vs Qwen 3.8 — r/LocalLLM
  15. Which models you run on your Nvidia v100? — r/LocalLLM
  16. How to run Locall LLMs on Windows 11 without LMStudio or Ollama? — r/LocalLLM
  17. Opti 27B: Qwen3.8-27B in 11.8 GB at 3.47 bpw, within 0.5% of FP16 perplexity and matching Q4_K_M at 30% fewer bytes. Patched llama.cpp runtime, source public, reproduce with one command — r/LocalLLM
  18. dual 7900 xtx - some guy made a pretty optimized fork of lamacpp optimized for this setup Qwen 3.8 Q8 at 82 tokens / seconds decode — r/LocalLLaMA
  19. Made a tool that tells you which GGUF quants will fit your GPU/Mac, with the llama.cpp command to run them — r/huggingface
  20. Llama Loves Pirates — Tom McGrath — Machine Learning Street Talk