Together AI· Inference platform

Benchmarking inference at scale: coding agents

Summary On a production coding agent workload, Together Inference Engine delivers 31% more TPS than the next fastest OSS engine on the same hardware, and maintains 2× better TTFT at saturation. The gains come from full-stack optimization: ThunderMLA, custom kernel rewrites, and end-to-end profiling on real traffic. Most inference benchmarks measure a single user hitting a dedicated endpoint. The numbers look great. They're also useless for reasoning about production. In production, you're running dozens or hundreds of concurrent requests. They compete for the same KV cache, the same memory bandwidth, the same GPU cycles. What matters is what happens to every user when the system is under load. We built this benchmark to answer that question for coding agents. It's a workload that hits inference hard: long inputs, high concurrency, and no tolerance for latency degradation under load. This is version one. We'll update it as we build. What a coding agent workload looks like Coding agent requests carry a lot of context. The file being edited, surrounding code, conversation history, retrieved snippets. Inputs are long. …

Announced

19 May 2026 00:00 UTC