Together AI· Inference platform

Serving DeepSeek-V4: why million-token context is an inference systems problem

Benchmark tables miss the main point of DeepSeek-V4: the important change is architectural. V4 turns million-token context into a serving-systems problem. The model supports a 1M-token context window through a hybrid attention design that compresses context before key-value (KV) storage, mixes compressed and local attention paths, and changes how prefix reuse works. Those choices reduce KV pressure, but the savings only matter if the inference engine can manage the resulting cache layouts, recover local state, batch requests effectively, and choose endpoint profiles that match the workload. This post focuses on the serving implications of V4's Compressed Sparse Attention (CSA) / Heavily Compressed Attention (HCA) / Sliding Window Attention (SWA) attention design, based on Together's early bring-up work on NVIDIA HGX B200. V4 also includes other architecture and training changes, including Manifold-Constrained Hyper-Connections (mHC) residual connections and Muon optimizer choices, but those are outside the main scope here. V4 compresses the token axis of KV cache Autoregressive inference stores prior context in KV cache. …

Announced

11 May 2026 00:00 UTC