Abstract
Quantized LLM inference is dominated not by compute but by memory bandwidth: at low arithmetic intensity, the accelerator spends most of its time moving the key–value cache rather than doing math. We present TurboQuant, a KV-cache quantization scheme operating at 1–4 bits that delivers compression ratios of 16× / 8× / 5.3× / 4×, is toggleable at runtime via a PATCH to a running endpoint — no engine restart — and preserves task accuracy within measurement noise on open-weight models.
1 · Motivation
Serving throughput for autoregressive decoding is bounded by how fast the KV-cache can be read back each step. As context length and concurrency grow, the cache — not the weights — becomes the largest resident tensor in VRAM. Teams respond by over-provisioning GPUs or capping context, both of which are expensive and neither of which addresses the underlying bandwidth wall.
The lever that matters is the size of the cache in bytes. Halving it does more for tokens-per-second on memory-bound hardware than most kernel-level tricks, and it compounds with every optimization downstream.
2 · Method
TurboQuant quantizes keys and values independently at deploy time, with per-channel scales chosen to protect the high-variance dimensions that attention is most sensitive to. Three properties make it practical in production:
- Runtime-toggleable. Precision is a property of the endpoint, changed with a
PATCH— operators dial compression up under load and back down for accuracy-critical traffic without redeploying. - Bandwidth-first. The win is realized as reduced bytes moved per decode step, which is why it improves TTFT, TPOT, and throughput simultaneously on memory-bound accelerators.
- Hardware-neutral. The scheme is expressed above the vendor kernel layer, so it applies identically on NVIDIA, AMD Instinct, and Intel Gaudi.
3 · Results
Across open-weight models we observe the compression/accuracy trade-off summarized in Figure 1. At 4-bit, footprint drops ~4× with accuracy indistinguishable from FP16 on our evaluation suite; the aggressive 1-bit regime reaches up to 16× and remains viable for latency-sensitive, high-concurrency chat where a small quality margin is an acceptable exchange for capacity.
Because precision is per-endpoint, the operator picks the point on this curve that matches the workload — and can move along it live as traffic shifts.
4 · Deployment
TurboQuant ships inside FlowServe's vLLM-based runtime as a toggle alongside DDTree speculative decoding and the NVIDIA inference stack. In sovereign and air-gapped deployments — where adding hardware is slow or impossible — cache compression is often the only practical way to raise capacity, which is why it is the first optimization we enable in a pilot.
Reproducing these numbers
The compression levels above are directional and workload-dependent. We measure them live on customer models and silicon during a pilot; see Request a pilot to run the benchmark on your own hardware.