Together AI· Inference platform

Aurora

Summary Speculative decoding goes stale in production — draft models can drift and offline retraining can't always keep pace. Aurora fixes this. It's an open-source, RL-based framework that learns directly from live inference traces and continuously updates the speculator without interrupting serving. Key results: → Real-time adaptation across shifting traffic domains → 1.25x additional speedup over a well-trained static speculator The headline finding: online training from scratch can outperform a carefully pretrained static baseline. Running large language models in production is a constant tradeoff between performance and cost. Speculative decoding is the standard lever: in principle, it speeds up inference. In practice, it often under-delivers—draft models go stale, acceptance rates drift, and offline retraining is too slow and too expensive to keep pace with live traffic. What if your system could learn continuously, on the fly, from the very requests it's serving? Last year, we introduced ATLAS — our first step toward an adaptive speculator. That work laid the foundation, but the goal was always a fully autonomous system that closes the loop between serving and training. …

Announced

31 Mar 2026 00:00 UTC