Turning On Per-Tenant Quotas Made One Replica 5.8x Less Fair: 25.9/26.8/24.7/22.6 Becomes 7.4/11.3/30.3/51.0
This story is from 2026-08-25. It is preserved in the archive; the latest stories are on the live feed.
A quota admits requests. A round-robin dispatches requests. A rate limit counts requests. What a tenant consumes is residency - how long its sequences sit in the running batch taking a slot of every decode iteration - and residency is the output length. The exchange rate between one request and one…
Read the full story at DEV Community — AI ↗
Timeline · 1 report
- 2026-08-25 13:41 · DEV Community — AI
Turning On Per-Tenant Quotas Made One Replica 5.8x Less Fair: 25.9/26.8/24.7/22.6 Becomes 7.4/11.3/30.3/51.0