AINewsnow

Moving a running model's layers between GPUs mid-reply (8 ms, no re-prefill)

I've been working on an inference engine called jitllm. The idea came from my background in operating systems and JIT compilers: generate kernels for whatever hardware is there, and treat layers a bit like processes the engine can place and move around. In the video, a cloud VM with T4s: mid-reply,…

Read the full story at r/LocalLLaMA ↗

Timeline · 1 report

  1. 2026-10-14 04:40 · r/LocalLLaMA
    Moving a running model's layers between GPUs mid-reply (8 ms, no re-prefill)

More stories

  1. Google releases Gemini 4 Argon with 1M token output limit — Google Gemini Blog
  2. Saudi OpenAI Engineer Arrested for Allegedly Planting Recording Device — Sky News Technology
  3. Google grants 200 European researchers access to frontier AI models — Techmeme
  4. Judge orders OpenAI to disclose hiring records in Apple trade secrets lawsuit — TechCrunch AI
  5. Nvidia pledges $250M in compute to UK for AI safety research — Techmeme
  6. NEURA Robotics Unveils Neuron AI Foundation Model for Physical AI — Unite.AI
  7. Anthropic Opens Singapore Office and Signs AI Safety MOU with Government — AlphaSignal
  8. The EU Commission's AI Office launches formal AI Act probes into OpenAI and Anthropic over AI agent hacks; experts say the AI Act includes an R&D exemption — Techmeme

Get the daily brief of stories like this at 6:30 every morning →