Gemma 4 31B QAT on one 24 GB card, Google's 4-bit weights bit for bit, 46 tok/s plain / ~80 with the draft model (RTX 4090)
Glyd 0.31 runs Google's Gemma 4 31B and 12B QAT checkpoints on our own engine. The quant: none of our own. Google's QAT checkpoint is already 4-bit codes plus a scale per 32 weights; we store exactly that (4.5 bits per weight). Every weight decodes to the same bf16 value as Google's QAT files, and…
Read the full story at r/LocalLLM ↗