AINewsnow

vLLM Preemption: Why 1 in 50 Requests Restarts From Scratch

This story is from 2026-09-14. It is preserved in the archive; the latest stories are on the live feed.

Our chat endpoint had a mean latency of 1.9s and a p99 of 11.4s. Same model. Same GPU. Same prompt template. nvidia-smi showed 96% utilization and no memory pressure worth mentioning. Nothing crashed. Nothing retried. But roughly one request in fifty would stream a few tokens, freeze for six second…

Read the full story at DEV Community — Machine Learning ↗

Timeline · 1 report

  1. 2026-09-14 16:55 · DEV Community — Machine Learning
    vLLM Preemption: Why 1 in 50 Requests Restarts From Scratch

More stories

  1. Amid growing AI fears, King Charles meets with industry leaders in Scotland — NPR Technology
  2. King Charles to press Nvidia, OpenAI, Anthropic leaders on AI safety at summit — CNBC Technology
  3. Huawei details AI accelerator roadmap, pulls in next-generation Ascend NPUs by several quarters — FP4 performance of the Ascend 960PR doubles expectations — Tom's Hardware
  4. Flyweight: open-source C++/CUDA engine for running MoE models bigger than your VRAM on one GPU + system RAM. First PyPI release, looking for contributors. — r/LocalLLaMA
  5. Deployed Qwen 3.6 35B A3B on a single DGX Spark supporting 12 concurrent users at 262K context. Are there better ways to optimize this? — r/LocalLLM
  6. what's the state of the art recipe for running Qwen3.8-Flash-Next with a pair of 3090s and a ton of system RAM rn? — r/LocalLLaMA
  7. Dario Says AI Should Slow Down. Jensen Wants to Go Full Steam Ahead. — Wall Street Journal Technology
  8. NVIDIA CEO Jensen Huang rejects ‘AI will end the world’ claim, yet cautions ‘we should go as fast as we can but...’ — Mint AI

Get the daily brief of stories like this at 6:30 every morning →