Together AI· Inference platform

Accelerate RL rollouts by up to 50% with distribution-aware speculative decoding

Summary Distribution-aware speculative decoding (DAS) is a novel framework that significantly alleviates the rollout bottleneck in RL post-training — delivering up to 50% speedup without touching model outputs. The rollout bottleneck Reinforcement learning has become the cornerstone of modern LLM post-training. Models like DeepSeek-R1 owe their reasoning capabilities to RL fine-tuning. But as models grow larger, a critical bottleneck has emerged: the rollout phase. In RL training, the model must generate complete responses to every prompt in a batch before the next training step can begin. The slowest generation determines total step time — a textbook long-tail problem. 70% of total training time is consumed by the rollout phase — exceeding the cost of backpropagation and parameter updates combined. Synchronous barrier: All rollouts must complete before training proceeds. One slow generation blocks the entire batch. Growing lengths: Modern reasoning models generate increasingly long chains of thought, amplifying the long-tail effect. GPU idle time: As stragglers run, other GPUs sit idle — wasting thousands of dollars of compute per training run. …

Announced

24 Apr 2026 00:00 UTC