Production AI System Design for Millions of Users: Part 1, The Architecture

Your AI feature works for ten users. Then it has to work for ten million, and almost everything that made the demo easy turns into a problem. The model call that took 800 milliseconds now queues behind a thousand others. The GPU bill, fine at prototype scale, projects to a number that ends the project. Quality silently drifts and nobody notices until users complain. And the one giant model you call for every request is overkill for 80% of them and too weak for the other 20%.

Scaling an AI product is not "the same as scaling a web app, plus a model." It is two architectures fused together. The first is the traditional system design you already know: load balancing, caching, queues, CI/CD, reliability. The second is a new AI-specific control plane that did not exist five years ago: the LLM gateway, semantic cache, retrieval layer, guardrails, evaluation systems, and AI observability. A production-ready system needs both, designed together. Get the traditional layer right and ignore the AI layer, and you ship something fast that hallucinates and costs a fortune. Get the AI layer right on a shaky foundation, and it falls over at scale.

This is a two-part series. Part 1 lays out the full architecture: every layer, traditional and AI-specific, and how each one bends under AI workloads. Part 2 walks real case studies from Amazon, Uber, Perplexity, and others, and ends with a production-readiness checklist. This series builds on System Design for AI Products, which covers what changes conceptually when LLMs enter the stack. Here we cover how to build the whole thing at scale.

The mental model is a compound AI system#

The single most useful shift is to stop thinking of "the LLM" as the system and start thinking of it as one component in a compound AI system: multiple interacting parts that together produce a reliable answer. Production LLM applications are rarely a single model call. They are an orchestrated pipeline of retrieval, routing, generation, validation, and caching, each of which can be scaled, cached, traced, and replaced independently.

Picture the request flowing from the client through the gateways into orchestration, which fans out to retrieval, tools, model serving, and guardrails before reaching the data stores.

Cutting across every layer are four planes that are easy to forget and fatal to skip: caching (exact-match, semantic, and embedding), observability (traces, tokens, cost, quality), evaluation (offline gates and online sampling), and guardrails (safety and policy). The rest of this post walks the layers in order, then the cross-cutting planes.

Reference architecture at a glance#

Here is the whole stack in one table: each layer, its job, the traditional technology that handles it, and the AI-specific addition that production demands. Treat it as the roadmap for the two parts that follow.

Part A: Traditional system design that still applies#

None of the fundamentals go away. They bend.

Load balancing, API gateway, and autoscaling#

You still put a load balancer and API gateway in front of stateless app servers, and you still autoscale. The twist is what you scale on. CPU and memory are the wrong signals for LLM-backed services because the bottleneck is the inference layer and its queue. Autoscale on queue depth and time-to-first-token (TTFT) instead: when requests start waiting and TTFT climbs, add capacity, and when GPUs idle, remove it. Because warming a GPU node and loading a model is slow, keep a warm buffer and scale ahead of demand, not after.

Latency budgets and the async pattern#

A web request budgets in tens of milliseconds. An LLM request budgets in seconds. That difference forces an architectural choice: synchronous request/response for short interactive calls, and asynchronous (queue plus worker plus webhook or polling) for anything long, like document processing, multi-step agents, and batch jobs. The queue is not optional at scale. It is what keeps a burst of slow model calls from exhausting your connection pool and taking down the synchronous path. Set an explicit latency budget per surface (sub-2-second for chat, minutes for batch) and design the path to honor it.

Multi-layer caching#

Caching is the highest-leverage cost and latency lever in the whole system, and AI adds a layer to the classic three. You still use a CDN for assets, an application cache (Redis) for computed results, and a database cache for hot rows. But for LLM responses, exact-match caching alone underperforms, because two users rarely phrase a question identically. That gap is exactly what the semantic cache (Part B) solves. Design caching as a hierarchy: cheapest and fastest first, model call last.

CI/CD becomes CD for prompts, models, and evals#

Your code pipeline is unchanged, but AI adds artifacts that also need versioning and safe rollout: prompts, models/adapters, and eval sets. A prompt change is a production change and deserves the same rigor as a code deploy. Version it, test it against an eval set in CI, and roll it out behind a canary or A/B. The teams that move fastest treat prompts as code and gate every change on an automated evaluation (more in Part B).

Reliability: timeout cascades and graceful degradation#

Every dependency in the compound system can be slow or fail: the embedding service, the vector DB, the reranker, the model API. Production systems handle this with timeout cascades and graceful degradation. If embedding exceeds its budget, skip it and fall back to keyword search. If the model exceeds its timeout, return a cached or templated fallback. If a provider rate-limits you, fail over to another via the gateway. Layer the classic patterns (timeouts, bounded retries with backoff, circuit breakers) so one slow component degrades the experience instead of breaking it. We detail the retry and queue architecture in System Design for AI Products.

Rate limiting, quotas, and multi-tenancy#

At scale you serve many tenants over shared, expensive infrastructure, so you need per-tenant rate limits and token quotas, not just request limits, because a single user can burn enormous cost in one long generation. Track and cap spend per tenant, isolate noisy neighbors, and attribute cost back to who incurred it. This lives partly in the API gateway and partly in the LLM gateway below.

Part B: The new AI-specific control plane#

This is the layer that did not exist in a pre-LLM stack, and it is what separates a production AI system from a prototype with a model call in it.

The LLM gateway is the new control plane#

The single most important new component is the LLM gateway: a proxy that sits between your application and every model provider. Instead of each service calling OpenAI, Anthropic, or your self-hosted model directly, they all go through the gateway, which centralizes the operational concerns that otherwise scatter across the codebase: routing across providers and model tiers, fallback when one is down or rate-limited, rate limiting and cost tracking, semantic caching, guardrails, and compliance logging (LLM Gateway Architecture, 2026).

The gateway is also where model tiering happens: route easy requests to a small, cheap, fast model and only the hard ones to a large model, often using a fast classifier to decide. This single pattern can cut cost dramatically, and it is why nearly every large-scale system (see Part 2) ends up multi-model rather than betting everything on one. You can buy a gateway (LiteLLM, Portkey, Kong AI Gateway, Helicone) or build one. The build-vs-buy decision turns on how much routing and governance logic is specific to you.

A classifier at the front of the gateway decides which tier each request takes, and a fallback path catches provider failures.

Semantic caching#

Semantic caching is the highest-impact AI-specific optimization. Instead of matching prompts byte-for-byte, it embeds the incoming prompt and runs a similarity search against previous prompts, returning a stored answer when a new prompt is close enough to one already answered. Because real users ask the same things in different words, this absorbs traffic that exact-match caching misses entirely, reportedly 40 to 70 percent fewer redundant model calls on real-world workloads (Maxim, 2026). The tradeoff is a similarity threshold to tune: too loose and you serve subtly wrong cached answers, too tight and you cache nothing. It usually lives in the gateway.

Retrieval and the RAG layer#

Most production AI systems are retrieval-augmented, so the retrieval layer (a vector database, embeddings, and often a reranker) is a first-class part of the architecture, not an afterthought. It has its own scaling, caching (embedding cache), and quality concerns. We cover this layer end to end in the RAG lifecycle series and Build a RAG Search Pipeline. Architecturally, treat it as a service with its own latency budget and failure mode (degrade to keyword search when it is slow).

Inference serving and model tiering#

If you self-host, the serving layer is its own discipline. Continuous batching, PagedAttention, quantization, and multi-LoRA serving determine whether one GPU serves ten requests or two hundred. This is the entire subject of Serving a Fine-Tuned Model at Scale. Architecturally, the key decision is tiering: a fleet of small models for the bulk of traffic, larger models reserved for hard requests, with the gateway routing between them.

Guardrails#

When the system can say anything, you need a layer that checks what goes in and what comes out. Guardrails validate inputs (block prompt injection and policy-violating requests), validate outputs (PII leakage, unsafe content, schema conformance), and enforce content policy. At enterprise scale these are table stakes, not nice-to-haves, and they belong in the request path (typically at the gateway) so every call is covered uniformly rather than per-feature.

Evaluation systems#

You cannot improve or even safely change what you cannot measure, and AI quality is invisible to traditional monitoring. A production system needs two kinds of evaluation. The first is offline: an eval set run in CI that gates every prompt and model change. The second is online: sampling real production traffic and scoring it (often with an LLM-as-judge) to catch quality drift after deploy. Connect the two so a regression caught in production becomes a new offline test case. The methodology is in Add Evaluation to an AI Feature.

Observability and tracing for AI#

Standard metrics and logs do not capture what matters for AI. You need AI-native observability: distributed tracing with span-level visibility across prompts, retrievals, tool calls, and guardrails; token and cost tracking per request and per tenant; latency split into TTFT and time-per-output-token; and quality and drift signals. This is the lifeline for debugging at scale, covered in Observability for AI Systems. Wire it in before you tune anything. Until you can see a real request flow through the compound system, every change is a guess.

Prompt management and agent orchestration#

Two more pieces complete the control plane. Prompt management (centralized, versioned, evaluable prompts) turns prompt changes from risky edits scattered in code into governed, testable artifacts (Uber's toolkit in Part 2 is the canonical example). Agent orchestration (the control flow for multi-step, tool-using, or multi-agent systems) is its own layer when your product is agentic. We cover building it reliably in Production AI Agents with LangGraph.

Common mistakes and tradeoffs#

The defining tension across the whole design is latency vs cost vs quality, and you cannot maximize all three. Bigger models and more retrieval raise quality and cost. Aggressive caching and smaller models cut cost and latency but risk quality. The architecture's job is not to win every axis. It is to give you the dials (the gateway for routing, the cache for cost, the eval system for quality) and the visibility to set them deliberately for each traffic class.

Key Takeaways#

• A production AI system is a compound AI system: the model is one component among retrieval, routing, validation, and caching. Design the whole pipeline, not just the model call.
• Traditional system design still applies but bends. Autoscale on queue depth and TTFT, use queues for slow calls, treat prompts as code in CI/CD, and degrade gracefully via timeout cascades.
• The LLM gateway is the new control plane. Routing, fallback, cost, semantic cache, and guardrails belong in one place, not scattered across services.
• Route across model tiers and add a semantic cache (40 to 70 percent fewer calls). These two patterns drive most of the cost savings at scale.
• The AI control plane adds retrieval, guardrails, evaluation, observability, and prompt management, each a first-class layer with its own scaling and failure modes.
• Eval gates and AI-native tracing come first. You cannot safely change or scale what you cannot measure.

FAQ#

Do I need an LLM gateway, or can services call providers directly?

At small scale, direct calls are fine. At production scale you want a gateway: it centralizes routing, fallback, rate limiting, cost tracking, semantic caching, and guardrails so those concerns are not duplicated and inconsistent across services. You can buy one (LiteLLM, Portkey, Kong, Helicone) or build a thin one. The decision turns on how custom your routing and governance logic is.

Should I use one big model or route across several?

Route. Sending every request to one large model is overkill for the majority of traffic and expensive at scale. A small-model-for-easy, large-model-for-hard routing strategy, often with a fast classifier deciding, cuts cost dramatically while preserving quality on the hard requests. Nearly every large-scale system in Part 2 is multi-model.

When should an AI call be synchronous vs asynchronous?

Synchronous for short, interactive calls within a tight latency budget (chat replies). Asynchronous (queue, worker, webhook or polling) for anything long: document processing, multi-step agents, batch jobs. The queue protects your synchronous path from being exhausted by slow calls during a burst.

How is semantic caching different from normal caching?

Exact-match caching only hits when a prompt is byte-for-byte identical, which is rare for natural-language input. Semantic caching embeds the prompt and matches on similarity, so paraphrases of an already-answered question hit the cache. It absorbs far more traffic, at the cost of a similarity threshold you must tune to avoid serving subtly wrong answers.

What's the difference between this and traditional system design?

Everything traditional still applies: load balancing, caching, queues, CI/CD, reliability. AI adds a new control plane on top: the LLM gateway, semantic cache, retrieval/vector layer, guardrails, evaluation systems, and AI-native observability. Production readiness means designing both together, and the rest of this series shows how real companies did exactly that in Part 2.