Spent ~11 GPU-hours on an MI300X figuring out which inference engine to run and which knobs matter. SGLang defaults won, and only prefix caching helped
I've had some AMD Developer Cloud credits to burn, so I've been running a series of benchmarks on a single MI300X (192 GB). The last two ended up being one question in two halves: which engine should you serve a coding model with, and once you've picked one, which settings are actually worth touchi…
Read the full story at r/LocalLLM ↗