AINewsnow

We built our own inference engine and put it up against vLLM and SGLang on four models and three kinds of GPU.

357 seconds. That's how long the median request waited for its first token when we ran GLM-5.3 on vLLM under sustained load. Eight B200s. 32 concurrent workflows. Our engine, same model, same box: 3.24 seconds. I didn't build this. Rob and the team did. They've been cooking on it for a long time, m…

Read the full story at DEV Community — Machine Learning ↗

Timeline · 1 report

  1. 2026-11-07 17:11 · DEV Community — Machine Learning
    We built our own inference engine and put it up against vLLM and SGLang on four models and three kinds of GPU.

More stories

  1. GLM-5.3-CYBERSECURITY-FP8: Inside the Refusal-Removed AI Model for Red Teaming — DEV Community — Machine Learning
  2. Moonshot Just Put a 2.8-Trillion-Parameter Model on Hugging Face — Kimi K3.1 Pro's Open Weights Are Real, and the Fine Print Matters — DEV Community — AI
  3. VEKTOR 2.0.5: 88 MCP tool runs Improved—GLM-5.3-Flash with 1 million trial tokens — DEV Community — AI
  4. Nearly 5 the output from the same eight B200s — DEV Community — AI
  5. AI Inference Benchmark: Keln vs Fireworks, Together AI, Baseten, Modal, DeepInfra & Z.ai — DEV Community — AI
  6. Z.ai chief scientist Jie Tang just revealed the training pipeline behind the upcoming GLM-5.5 (32B, 122B, and 256B) — r/LocalLLaMA
  7. Full 753B GLM-5.3 running on two DGX Sparks at 18-24 tok/s, 262k context, every expert kept — r/LocalLLM
  8. Free Chinese AI Models Show Hacking Skills Close to OpenAI's Advanced Cyber Model — Wired AI

Get the daily brief of stories like this at 6:30 every morning →