AINewsnow

One Knob, 19x Latency: What I Learned Benchmarking vLLM on a Single GPU

I built a small serving harness that measures latency, throughput, AND output drift per config. It found a 19x latency cliff on a single knob, a 3.6x model speed gap — and a reasoning model that can't reproduce its own outputs at temperature 0. Scope note, first paragraph as promised: one RTX 3060,…

Read the full story at DEV Community — Machine Learning ↗

Timeline · 1 report

  1. 2026-10-23 14:31 · DEV Community — Machine Learning
    One Knob, 19x Latency: What I Learned Benchmarking vLLM on a Single GPU

More stories

  1. Anthropic says it partnered with Theorem to use Claude to formally prove useful security properties at the assembly code level across 123 open-source libraries (Anthropic) — Techmeme
  2. OpenAI agents disabled their own safety monitoring systems, with one lying about it — Financial Times AI
  3. The US Treasury proposes a rule to crack down on US high-tech investments in areas like AI and supercomputing in China, Russia, and other "countries of concern" (Associated Press) — Techmeme
  4. Using AI Agents to Teach Robots Like Microduck to Walk in Simulation — NVIDIA Technical Blog
  5. Recursive launches Flint, a laptop coding model it says was largely built by AI — TechCrunch AI
  6. OpenAI makes its Jev competitor widely available as decision model competition heats up — TechCrunch AI
  7. Post-Training at the Frontier — CoreWeave Blog
  8. OpenClaw creator spent $10 million on AI tokens last month — InfoWorld AI

Get the daily brief of stories like this at 6:30 every morning →