System Design for AI Products: What Changes After LLMs

You ship a feature backed by an LLM. The demo looks fine. Then you hit production and discover a wall of problems you did not see in deterministic services. A model call that takes eleven seconds on a bad day. An output that is subtly wrong in ways no unit test caught. A retrieval pipeline that quietly degraded after a data update. A retry storm that costs three times your expected token budget. None of these are bugs in the usual sense. They are design gaps that only appear when inference enters the critical path.

The question is not just how to call a model. The question is how your product behaves when the model is slow, wrong, unavailable, or unexpectedly expensive.

LLMs change the system design conversation#

A typical web service has a predictable shape: a database query takes single-digit milliseconds, a cache hit is even faster, and failures are binary and immediate. An LLM call is different in almost every dimension. Latency can range from a few hundred milliseconds to well over ten seconds depending on output length, model load, and provider health. The response is probabilistic, so the same input can produce different outputs on successive calls. Errors come in new forms: token budget overruns, rate limits, context length violations, and model refusals, not just HTTP 500s.

Anthropic's "Building Effective Agents" guide, which drew from working with dozens of teams building production agents, found that the most successful implementations favored simple, composable patterns over complex frameworks. The underlying reason is reliability: every added layer between the request and the model call is another place something can fail in a way that is slow and opaque.

The design conversation for AI products is really a reliability conversation. You are not just routing a request to a model. You are building a pipeline with more latency variance, more output uncertainty, and more cost sensitivity than any database query or API call you have managed before.

What the stack looks like#

A minimal AI product feature has at least four moving parts: a retrieval layer that finds relevant context, an embedding or ranking step, the LLM call itself, and a response handler that validates and routes output. More complete products add evaluation, human review, output caching, usage metering, and audit logging.

The stack is not flat. Decisions cascade. If retrieval is slow, the whole response is slow. If context is noisy, output quality drops even with a capable model. If you have no eval layer, silent regressions are invisible. Each stage has its own failure modes and its own latency contribution.

At a high level, a request fans out from the API into retrieval and the model, draws on data and vector stores, and emits signals to observability at every hop.

Here is a practical breakdown of the layers and what they own:

What usually happens, and why it goes wrong#

The most common pattern when teams add LLMs to existing products is treating the model call like a fast HTTP service. The code makes a synchronous request, expects a response in the same time window as a database query, and forwards the result directly to the user. This works in demos. It breaks at scale in four predictable ways.

Latency variance crashes user-facing SLOs. An LLM response that is fast at P50 can be three to five times slower at P99. If your timeout is calibrated to P50, tail users get error pages. If your timeout is generous enough for tail latency, slow model calls can hold connections open, exhaust worker pools, and cascade into downstream timeouts.

Retrieval quality becomes invisible. Teams often ship with retrieval that is good enough in testing, then discover that a data update, a schema change, or a new query pattern quietly breaks context quality. Because the model often produces plausible-looking text even with poor context, the degradation shows up in customer behavior or support tickets rather than error logs.

Retry logic amplifies cost. When a model call fails, naive retry logic tries again immediately with the same input. For rate-limit errors this triggers exponential cost growth. For provider outages it queues up a storm of retries that all resolve simultaneously and cause a different spike. Without jitter and backoff, retries can cost more than the original request volume.

No baseline means no regression signal. Most teams add LLMs without adding eval. When the model provider updates a model version, when the prompt changes, or when retrieval degrades, there is no automatic signal. Engineers discover regressions from users or from A/B metrics long after the change.

Root cause and core tradeoff#

The underlying tension is synchrony versus reliability. Synchronous LLM calls in user-facing request paths maximize perceived simplicity but minimize resilience. Every network issue, rate limit, provider outage, or latency spike becomes a user-visible error. Moving inference to an asynchronous pipeline (queue, worker, result store) improves resilience dramatically but adds complexity and changes the UX model.

The tradeoff is not about making one correct choice. It is about matching the execution model to the latency tolerance of the feature. A real-time autocomplete and a nightly document analysis pipeline need fundamentally different designs even if they both call the same model.

The decision usually comes down to one question: can the user wait? That single split sends a request down one of two very different paths.

Recommended approach#

For user-facing features with latency budgets, use streaming where the model supports it, set aggressive per-stage timeouts with graceful fallbacks, and never let the model call be the only retry path. Cache deterministic outputs (embedding lookups, structured validation responses) aggressively, and move anything that can wait into a background job.

For async pipelines and agent tasks, use a durable job queue with exponential backoff and jitter, explicit dead-letter queues for jobs that fail all retries, and idempotent workers so retries are safe. See Designing Reliable Job Queues for a deeper treatment of the queue layer specifically.

For retrieval quality, version your vector index alongside your embedding model, test retrieval recall before deployment, and instrument chunk relevance scores in production. A retrieval pipeline that is not observed is a retrieval pipeline you cannot improve. Our guide on building a RAG search pipeline covers the indexing and retrieval decisions in depth.

For eval and observability, instrument every model call with token counts, latency, and output quality signals from the start. The OpenTelemetry GenAI Semantic Conventions define standard attribute names for LLM spans (attributes like gen_ai.request.model, gen_ai.usage.input_tokens, and gen_ai.usage.output_tokens), so your traces are compatible with vendor tooling from day one.

Implementation details#

Timeout and retry architecture#

Set timeouts at each stage of the pipeline, not just at the outer HTTP request level. A retrieval timeout, an embedding timeout, and an LLM inference timeout are independent. If any one stage exceeds its budget, you can either degrade gracefully (skip retrieval and use a shorter context) or fail fast and return a cached or templated response.

# Pseudocode: per-stage timeout with graceful degradation
async def generate_response(query: str) -> str:
    context = ""
    try:
        context = await asyncio.wait_for(
            retrieve_context(query), timeout=1.5
        )
    except asyncio.TimeoutError:
        # Degrade: proceed without retrieved context
        pass

    try:
        return await asyncio.wait_for(
            call_llm(query, context), timeout=8.0
        )
    except asyncio.TimeoutError:
        return FALLBACK_RESPONSE

Retries should use exponential backoff with jitter. For rate-limit errors (429), honor Retry-After headers from the provider rather than using a fixed backoff. For provider errors (5xx), use a short initial delay, doubling each attempt, capped at a maximum. For context-length or validation errors, do not retry with the same input. The failure is deterministic and the retry will fail again.

Queue worker for async tasks#

For AI workloads that do not need to return synchronously (document processing, async report generation, agent tasks, evaluations), use a durable job queue backed by a persistent store.

# Pseudocode: queue worker with dead-letter handling
async def process_job(job: Job) -> None:
    try:
        result = await run_ai_pipeline(job.payload)
        await store_result(job.id, result)
    except RateLimitError:
        # Re-queue with backoff
        await requeue(job, delay=backoff(job.attempts))
    except ValidationError:
        # Permanent failure — move to DLQ
        await move_to_dlq(job, reason="invalid_input")
    except Exception:
        if job.attempts >= MAX_RETRIES:
            await move_to_dlq(job, reason="max_retries")
        else:
            await requeue(job, delay=backoff(job.attempts))

Model routing and cost controls#

For products with variable query complexity, route simple queries to smaller, faster models and reserve large model calls for tasks that require depth. Track cost per job type from the start. Token budgets that seem acceptable in development can compound unexpectedly in production as query volume and average context length grow.

Set hard cost controls at the request level (maximum tokens per call), at the pipeline level (maximum retries per job), and at the account level (provider-side spend caps). Treat a cost spike the same way you treat a latency spike: an alert, not a surprise at the end of the billing cycle.

Edge cases#

Context window overflow: Chunking strategy determines whether long documents fit the context window. Test the maximum-length inputs explicitly, not just average-length ones. A retrieval strategy that works at P50 input length may silently truncate context at P95.

Prompt injection in user input: User-provided text that reaches the model prompt can manipulate system behavior. Sanitize inputs, use structured prompt templates that separate user content from system instructions, and validate that model outputs do not include injected system directives.

Model version drift: Providers update models without always changing the model name. A system prompt or retrieval format that was tuned for one model version may produce noticeably different behavior after a silent update. Pin to specific versioned model IDs where the provider supports it, and run evals after any model change.

Cold start in embedding models: Self-hosted embedding models may have cold-start latency on the first inference after idle periods. This can cause the first user request after a quiet period to see retrieval latency far outside the norm. Use health checks or warm-up calls to keep the embedding service primed.

Checklist#

Key Takeaways#

LLMs make every system design problem slower to observe, harder to reproduce, and more expensive to retry. The architectural changes required are not about exotic AI infrastructure. They are about applying reliability engineering discipline to a component that is slower and less predictable than anything else in most product stacks.

The practical path: treat LLM calls as high-latency external services with their own timeout budgets, move non-interactive inference to async queues, instrument retrieval quality alongside token usage, build eval into the pipeline from the start, and design explicit fallback behavior rather than assuming the model will always respond within budget.

For teams evaluating specific feature tradeoffs, our post on add evaluation to AI feature goes deeper on embedding evals in the development loop, and observability for AI systems covers the signal and tooling side in more detail.

FAQ#

Should I use a framework or build a custom AI pipeline?

Start without a framework. Most production reliability problems (timeouts, retries, observability, cost controls) require behavior that frameworks often abstract away. Once you understand the design requirements for your specific use case, evaluate frameworks against those requirements rather than adopting them first and discovering the gaps later. Anthropic's guidance on effective agents explicitly recommends simple composable patterns over complex orchestration layers.

When should LLM inference be synchronous vs. asynchronous?

Use synchronous inference when the user is waiting and the response must appear before the next interaction. Real-time chat, autocomplete, and inline suggestions typically fit here. Use async queues for document processing, agent tasks, bulk evaluations, and anything where a multi-second wait is acceptable or results can be returned via polling or push notification. The threshold is roughly whether the user will tolerate a loading state or needs an immediate streaming response.

How do I detect retrieval quality degradation in production?

Instrument chunk relevance scores (the scores your vector index returns for retrieved results), monitor the distribution over time, and alert when scores drop significantly. Also track user behavior signals like re-queries, short session times, and negative feedback that correlate with poor retrieval. Neither signal is perfect alone, but together they give early warning before degradation becomes a visible product problem.

What is the right retry strategy for rate-limited model calls?

Honor the provider's Retry-After header when present. When it is absent, use exponential backoff starting at around one second, doubling each attempt, capped at thirty to sixty seconds, with additive jitter (random delay added to the backoff value) to prevent synchronized retry waves from a fleet of workers. Set a maximum retry count, and route jobs that exhaust retries to a dead-letter queue for inspection rather than silently dropping them.

How do I keep AI feature costs predictable?

Track cost per feature and per job type from the first week in production. Token usage at P99 input length, and at peak concurrency, is often three to five times the average. Set per-request token caps to bound worst-case cost per call, set per-pipeline retry caps to bound runaway retry behavior, and configure provider-level spend alerts so a cost spike surfaces as an operational event rather than a billing surprise.