Abstract
Autoregressive decoding is latency-bound: each token requires a full forward pass, and time-to-first-token (TTFT) is what users feel in interactive chat. DDTree replaces the sequential draft model of classic speculative decoding with a block-diffusion drafter that emits a tree of candidate continuations at once. The target model verifies the whole tree in a single parallel pass and commits the longest accepted path — delivering near-draft-model speed with the target model's exact output distribution.
1 · Motivation
Speculative decoding trades cheap draft tokens for expensive verification, but a sequential drafter caps the speedup: it can only propose one continuation at a time, and a single early rejection discards the rest. For latency-sensitive workloads the drafter itself becomes the serialization bottleneck.
If instead the drafter proposes many candidate paths as a tree, the target model can verify them in parallel and keep whichever prefix survives — extracting more accepted tokens per verification step.
2 · Method
DDTree's drafter is a block-diffusion model that generates candidate tokens in parallel rather than left-to-right, arranged as a depth-bounded tree. The target model runs one batched verification over the tree and accepts the longest path consistent with its own probabilities.
- Tree-depth slider (1–16). Depth trades draft cost against expected accepted length — operators tune it per workload, or let auto-A/B pick by rolling acceptance rate.
- Optional draft model. The block-diffusion drafter can stand alone or be paired with a small draft model for harder distributions.
- Distribution-preserving. Because acceptance is checked against the target model, outputs are drawn from the target distribution — no quality trade for the speedup.
3 · Results
On interactive chat workloads, DDTree reduces TTFT and time-per-output-token relative to autoregressive decoding, with the gain scaling as tree depth rises until draft cost dominates. Because the drafter emits a tree, acceptance is more robust to a single mispredicted token than sequential drafting — one bad branch doesn't sink the step.
The optimal depth is workload-dependent; our auto-A/B controller selects DDTree vs. autoregressive online based on rolling acceptance, so the endpoint always serves the faster of the two.
4 · Deployment
DDTree ships inside FlowServe's vLLM-based runtime as a per-endpoint toggle, alongside TurboQuant KV-cache compression and the NVIDIA inference stack. It targets the latency-sensitive, high-interactivity workloads — assistants, agents, chat — where first-token speed is the metric that defines the experience.
Reproducing these numbers
Acceptance rates and latency depend on model and traffic. We measure them live during a pilot; see Request a pilot to benchmark DDTree on your own workload.