You read a system design post, nod along to the diagrams, and then ship something that falls over the first time real traffic hits it. Theory only takes you so far. The more useful question is how companies serving hundreds of millions of users actually built their AI systems, and how often the same handful of decisions show up across very different products.
Look at Rufus, Alexa Plus, Uber, Perplexity, and 11x side by side and the overlap is hard to miss. Multi-model routing instead of one giant model. Aggressive caching. Hard latency budgets defended with chip-level optimization. A centralized gateway. Eval and prompt platforms. Agentic tool use behind guardrails.
Part 1 laid out the full architecture: the traditional layers and the AI-specific control plane. This part shows that architecture in the wild, through five real systems (Amazon's Rufus and Alexa Plus, Uber's GenAI Gateway, Perplexity, and 11x's Alice). Each comes with its problem, its key decisions, and the lesson that generalizes. We close with a production-readiness checklist distilled from all of them.
TLDR
- Route across model tiers via a real-time router instead of betting on one model (Rufus, Alexa Plus, Perplexity).
- Defend a strict latency budget (Rufus holds 300ms, Alexa Plus targets sub-2s) with prompt caching, speculative decoding, and custom silicon.
- Centralize provider access, cost, PII handling, and prompts in a gateway and platform (Uber).
- Treat orchestration, not any single model, as the moat (Perplexity), and expect to iterate agentic designs before they work (11x).
Production AI System Design series
This is Part 2 of a 2-part series. Part 1: The Architecture · Part 2: Case Studies and Production Checklist (you are here)
Case study 1: Amazon Rufus routes across models at 250M users#
The problem. Rufus is Amazon's conversational shopping assistant, used by more than 250 million customers, with interactions up over 200% year over year (AWS, 2025). Shopping questions range from broad ("what's a good gift for a runner?") to deep product detail to fast lookups, and they all need answers in milliseconds, during traffic spikes like Prime Day.
The architecture. Rufus's defining decision is a real-time router over a mix of models accessed through Amazon Bedrock: Anthropic's Claude Sonnet, Amazon Nova, and a custom model trained on Amazon's store knowledge. The router decides, per query, whether a question needs broad general knowledge, deep product detail, or fast search, balancing capability, latency, and answer quality. On top of that it is agentic. At inference time it calls services as tools to query live databases, check the current catalog, and pull a customer's order history, and it uses RAG to draw on external sources for product and trend questions.
The latency story is the most instructive part. Rufus holds a 300-millisecond latency target even during Prime Day peaks, and it gets there with systems-level work. Parallel and speculative decoding doubled token throughput, and migrating inference to Amazon's Trainium and Inferentia2 chips cut costs by roughly 50%.
The lesson: route, don't bet on one model
Rufus started as a single custom in-house LLM and evolved into a multi-model Bedrock architecture. Routing across models let the team adopt better models as they appeared and match each query to the cheapest model that answers it well. One model for everything is both slower and more expensive than it needs to be.
Case study 2: Alexa Plus defends a latency budget across 600M devices#
The problem. Alexa Plus is the generative-AI rebuild of Alexa, turning a scripted voice assistant into a conversational system capable of complex multi-step planning, across more than 600 million devices already in homes. Voice raises the stakes on latency. A pause that is tolerable in a chat UI feels broken when you are talking to a speaker.
The architecture. Like Rufus, Alexa Plus is multi-model, with extensive prompt engineering to make a general model behave reliably in a voice context. To defend a sub-2-second latency target, the team leans on prompt caching (reuse the heavy, fixed system-prompt context instead of reprocessing it every turn) and speculative execution (start likely next steps before they are confirmed), alongside a significant API refactor so the assistant can call Amazon's services as tools.
The lesson. A latency budget is an architectural constraint that shapes everything above it. When the budget is tight, you cache the expensive constant parts of the prompt, you do work speculatively, and you accept multi-model complexity to keep each step fast. The budget is not a metric you measure at the end. It is a requirement you design backward from.
Case study 3: Uber's GenAI Gateway makes the control plane a platform#
The problem. Uber found more than 60 LLM use cases across the company (Uber Eats recommendations, support chatbots, internal code and SQL generation), with teams each integrating models their own way. That sprawl is exactly the duplication the LLM gateway pattern from Part 1 exists to eliminate.
The architecture. Uber's Michelangelo team built the GenAI Gateway (Uber, 2024): a single integration point that mirrors OpenAI's HTTP/JSON API and fronts models from OpenAI, Vertex AI, and Uber-hosted LLMs. Choosing the OpenAI-compatible interface was deliberate. It let Uber's developers use the existing ecosystem (LangChain, LlamaIndex) with no friction. The gateway adds the governance layer: PII redaction that anonymizes sensitive data before it leaves for an external vendor and restores it in the response, a mandatory security review of each use case, plus authentication, metrics, audit logging, and cost attribution per team.
Uber paired the gateway with a Prompt Engineering Toolkit: centralized prompt template management, version control, evaluation frameworks, and production deployment. That turns prompts from scattered strings into governed, testable, versioned artifacts.
The lesson. At organizational scale, the AI control plane is a platform, not a library. Centralizing provider access, safety, cost, and prompt lifecycle behind one gateway is what lets dozens of teams build on LLMs without each reinventing security, observability, and routing.
Case study 4: Perplexity treats orchestration as the moat#
The problem. Perplexity answers questions over the live web, which means it must combine fresh retrieval with high-quality generation at low latency and sustainable cost, without owning a frontier model of its own.
The architecture. Perplexity is the clearest example of the compound system from Part 1, built on three pillars (ByteByteGo): retrieval, orchestration, and inference. For retrieval it uses Vespa to index over 200 billion URLs across tens of thousands of CPUs, with hybrid search (dense vector + sparse BM25 + machine-learned ranking that weighs freshness and engagement). For orchestration it runs a model-agnostic router: small classifier models assess query complexity and route to "the smallest model that will still give the best possible user experience," across its own Sonar models and third-party GPT and Claude. For inference it built ROSE, a Python/PyTorch engine with performance-critical paths rewritten in Rust, running on H100 GPUs with speculative decoding and multi-token prediction.
The build-vs-buy choices are sharp. Perplexity outsourced mature search indexing to Vespa and invested its own engineering in the RAG orchestration and routing that differentiate it.
The lesson: the model isn't the moat
Perplexity's competitive advantage is not a single superior model. It is the orchestration of many models plus a fast search system. At scale, the differentiator is how well you route, retrieve, rank, and serve, not which LLM you call.
Case study 5: 11x's Alice iterated into a multi-agent architecture#
The problem. 11x builds Alice, an AI sales development rep that researches prospects and writes outreach. The product needed to perform a genuinely multi-step, tool-using task well enough to hit human-level reply rates, a hard agentic problem.
The architecture. The instructive part is the journey. 11x iterated through a ReAct-style agent and then workflow-based designs before landing on a hierarchical multi-agent system built on LangGraph, which got Alice to a human-level ~2% reply rate. The progression (single agent, then structured workflow, then a hierarchy of specialized agents) is the same maturation path we describe in Production AI Agents with LangGraph.
The lesson. Agentic architectures are rarely right on the first try. Expect to start simple, measure where it fails, and add structure (workflows, then a hierarchy of specialists) only as the task demands. The reliability comes from iteration plus the orchestration primitives (state, checkpointing, and control flow), not from a clever first prompt.
The patterns that recur#
Strip away the domains and the same architectural decisions appear in every system:
| Feature | The recurring decision | Who does it |
|---|---|---|
| Multi-model routing, not one model | Real-time router across tiers and providers | Rufus, Alexa Plus, Perplexity |
| A latency budget, defended at the systems level | Prompt caching, speculative decoding, custom silicon | Rufus (300ms), Alexa Plus (sub-2s) |
| A centralized gateway and control plane | OpenAI-compatible proxy, PII, cost, governance | Uber, and implicit in all |
| Prompt and eval as managed platforms | Versioning, evaluation, governed rollout | Uber Prompt Toolkit |
| Build vs buy, drawn deliberately | Outsource the commodity, own the differentiator | Perplexity (Vespa vs orchestration) |
| Agentic tool use, iterated into shape | Tools at inference; simple to workflow to hierarchy | Rufus, 11x |
If your design does these things, you are in good company; if it does none of them, that is where to look first.
A production-readiness checklist#
Distilled from the architecture in Part 1 and the systems above, here is what "production-ready for millions" concretely requires.
Set and defend a latency budget
Pick an explicit budget per surface (e.g. sub-2s for chat, 300ms for fast lookups, minutes for batch). Design backward from it with caching, speculative decoding, and async queues for anything that cannot meet it.
Route across model tiers
Put a real-time router in front of multiple models (small and cheap for the easy majority, large for the hard minority) and make adding or swapping a model a config change, not a rewrite.
Centralize on an LLM gateway
Route all traffic through a gateway that handles provider fallback, rate limiting, cost attribution, semantic caching, PII handling, and guardrails. Buy or build, but make it the single path.
Cache at every layer
CDN and app cache for the classic load, plus a semantic cache to absorb paraphrased repeat questions (40 to 70% of calls on real workloads). Cache embeddings too.
Add guardrails in the request path
Validate inputs (injection, policy) and outputs (PII, unsafe content, schema) uniformly at the gateway, not per feature.
Gate releases on evals, version prompts and models
Treat prompts and models as artifacts: version them, run an offline eval set in CI on every change, and roll out behind a canary or A/B with quality metrics.
Instrument AI-native observability
Trace every request across prompts, retrievals, tool calls, and guardrails; track tokens, cost, TTFT/TPOT, and sampled output quality with drift alerts.
Autoscale and degrade gracefully
Scale on queue depth and TTFT with a warm GPU buffer; on failure, fall back to a smaller model, a cached answer, or keyword search rather than erroring.
Common mistakes and tradeoffs#
- Do: route across model tiers like Rufus, Alexa Plus, and Perplexity. It is the universal cost and latency win
- Do: design backward from an explicit latency budget
- Do: centralize provider access, cost, PII, and prompts in a gateway or platform like Uber
- Do: draw build-vs-buy deliberately. Own your differentiator, outsource the commodity
- Do: expect to iterate agentic architectures from simple to hierarchical
- Avoid: betting everything on a single giant model. None of these systems do
- Avoid: treating latency as a metric to measure rather than a constraint to design for
- Avoid: letting every team integrate LLMs their own way. That sprawl is what Uber's gateway fixed
- Avoid: building your own search or index or model when a commodity solution differentiates nothing
- Avoid: shipping a complex multi-agent design first. Start simple and add structure on evidence
The throughline is that none of these companies won by having the single best model. They won on architecture: routing, caching, latency engineering, a centralized control plane, deliberate build-vs-buy, and disciplined iteration. The model is a component; the system is the product.
Key Takeaways#
- Multi-model routing is universal. Rufus, Alexa Plus, and Perplexity all route across model tiers and providers rather than betting on one model. It is the central cost-and-latency pattern at scale.
- Latency is a design constraint, not a metric. Rufus holds 300ms and Alexa Plus targets sub-2s by designing backward from the budget with prompt caching, speculative decoding, and custom silicon.
- The control plane becomes a platform. Uber's GenAI Gateway and Prompt Toolkit centralize provider access, PII handling, cost, and prompt lifecycle so dozens of teams can build safely.
- Orchestration is the moat. Perplexity's edge is routing, retrieval, ranking, and serving, not a single superior model.
- Agentic systems are iterated into shape. 11x reached human-level performance by maturing from a single agent to a hierarchical multi-agent design, not by getting it right first.
- The checklist is the requirement set: latency budget, model routing, gateway, multi-layer caching, guardrails, eval gates, AI observability, and graceful degradation.
FAQ#
What's the single most important pattern from these case studies?
Multi-model routing. Every large-scale system routes across model tiers and providers via a real-time router instead of sending all traffic to one model. It is the pattern that most directly controls cost and latency while preserving quality on hard requests.
How do companies like Amazon hit such tight latency targets?
By designing backward from the budget at the systems level: prompt caching to avoid reprocessing fixed context, parallel and speculative decoding to produce tokens faster, smaller models for easy queries, and, at Amazon's scale, custom inference chips (Trainium and Inferentia2) that also cut cost ~50%. Latency is engineered, not hoped for.
Should I build my own LLM gateway like Uber?
Uber built one because it had 60+ use cases across many teams and needed centralized governance, PII handling, and cost attribution. If you are smaller, an off-the-shelf gateway (LiteLLM, Portkey, Kong, Helicone) gives you the same routing, fallback, caching, and logging without the build. Either way, route all traffic through one gateway.
Perplexity doesn't own a frontier model. How does it compete?
By treating orchestration as the product. Its moat is a fast hybrid-search retrieval system (Vespa over 200B+ URLs), a model-agnostic router that picks the smallest model that satisfies the query, and a custom inference engine, not any single LLM. At scale, how you route, retrieve, and serve matters more than which model you call.
When do I need a multi-agent architecture like 11x?
Only when a single agent provably cannot handle the task's complexity. 11x reached a hierarchical multi-agent design by iterating (single agent, then workflow, then hierarchy) and adding structure only where it failed. Start simple, measure, and escalate. See Production AI Agents with LangGraph for the maturation path.
How do I make AI costs predictable at this scale?
Combine the patterns: route easy traffic to cheap models, absorb repeat questions with a semantic cache, attribute and cap cost per tenant at the gateway, and, if self-hosting, optimize serving (batching, quantization, the right hardware). The gateway gives you the visibility and the controls in one place.
