2-bit copy in VRAM + exact residual in system RAM as the verifier: speculative decoding for models that don't fit your GPU

Hi Guys!, Recently I was making a research and development, but I do not have time to go further with this idea (kids, work, etc.). But maybe it will be interesting for someone. I do not think it would be something big, but maybe interesting approach that can be used in some of the cases. I believe…

Read the full story at r/LocalLLaMA ↗

Timeline · 1 report

1. 2026-10-22 01:18 · r/LocalLLaMA
   2-bit copy in VRAM + exact residual in system RAM as the verifier: speculative decoding for models that don't fit your GPU