Together AI· Inference platform

Autoscaling endpoints for LLM inference

Summary With Dedicated Model Inference on the Together AI platform y ou can get your deployments to autoscale on metrics the inference engine actually understands, such as in-flight requests, TTFT, GPU utilization, token throughput. You can set replica bounds, pick a metric and target, and then tune two windows that control how eagerly it scales up and how patiently it scales down. Understanding and choosing the right metric is important because it determines how your deployment will behave under peaky traffic and impacts the latency your users will see. Below we'll cover how to choose the right metric to autoscale on and show an experiment where the same load was replayed under three different autoscale policies. Over- and under-provisioning are both expensive With dedicated inference you pay per replica-minute, which makes capacity planning a balance between two failure modes: Over-provision - you're paying for GPUs to sit at 15% utilization just so that you can handle the peak traffic when/if it arrives. Almost never viable, especially in the current GPU constrained environment. Under-provision - your p95 degrades sharply the moment traffic exceeds what your replicas can batch. …

Announced

31 Jul 2026 00:00 UTC