Together AI· Inference platform

Configuring Dedicated Model Inference

Summary Dedicated Model Inference on the Together AI platform consists of three parts: the endpoint (a stable name you or your clients call), deployments (specific model + hardware combinations running replicas behind it), and configs (recipes for how a model runs). A capacity-aware traffic split ties these three entities together. This architecture enables various other features such as rollouts, A/B tests, shadow experiments, zero-downtime changes possible. Below we'll show how capacity-aware routing works against a live endpoint with measured traffic. How the resource model works A config is a recipe that specifies the engine, the GPU type and count as well as the parallelism and optimization profile (throughput, latency, balanced). Configs are immutable and every config has an ID like cr_... . Your deployment always points at exactly the config that was tested. A deployment binds one model (at a specific revision) to one config, gives it an autoscaling policy , and runs replicas. Deployments are disposable on purpose and creating and destroying should be considered routine. An endpoint is a fixed identity: a qualified name ( <project_slug>/<endpoint_name> …

Announced

29 Jul 2026 00:00 UTC