Two Strix Halo boxes, two USB4 cables, RDMA: 74 Gb/s on one connection and +31% tok/s with tensor parallelism (with numbers and the stuff that broke)
I've been poking at this on and off since September and it finally got good enough this week that I think it's worth writing up. Short version: the USB4 RDMA driver situation changed a lot in the last month, but not where I expected, and the "is it worth it" answer is more mixed than the headline.…
Read the full story at r/LocalLLM ↗