Together AI· Inference platform

ThunderAgent: 2x Faster Agentic Inference for Synthetic Data Generation at Scale

Summary We introduce ThunderAgent, a system for high throughput agentic inference. By introducing a novel program abstraction for agentic LLM request scheduling, ThunderAgent achieves up to 2.5× higher single-node throughput in our synthetic data generation pipeline, and delivers 2.4× speedup on an 8-node cluster with near-linear throughput scaling with respect to GPU nodes. Key Results → More than 2× single-node throughput, with roughly 10× lower P50 latency at high concurrency → 2.4× speedup on 8 nodes, near-linear scaling from 16 to 64 GPUs → Drop-in: one program_id field, OpenAI-compatible, works with your existing engine-level optimizations (like speculative decoding) ThunderAgent was accepted to ICML 2026 as a Spotlight paper. This paper was a collaboration between researchers at Georgia Institute of Technology, the University of Illinois Urbana-Champaign, Carnegie Mellon University, and Together AI. LLMs are increasingly deployed as agents. Systems like Claude Code, Codex, and OpenClaw reason, call tools, read results, and reason again, often for dozens of turns before completing a task. …

Announced

29 Jul 2026 00:00 UTC