Architect a Production Service for Autoregressive LLM Generation

Start with the bottleneck: Autoregressive decoding is memory-bound. Each step computes one token; GPU compute sits idle while memory bandwidth is saturated. The whole design revolves around how to batch requests together.

Batching: Continuous batching joins requests at the token level instead of waiting for a whole batch (static batching), multiplying throughput. Prefill (compute-heavy) and decode (memory-heavy) can be deployed separately.

KV cache management: Paged attention allocates cache in pages to kill fragmentation; prefix cache reuses computation for shared prefixes (like system prompts).

Serving layer: Streaming responses cut time-to-first-token; SLO-tiered routing sends simple requests to small models; rate limiting and queueing. Judge by TTFT, TPOT, and tokens per dollar - all three.