Observability for AI Systems

When a traditional web request fails, you have a stack trace, a status code, and a log line pointing at the problem. When an AI feature produces a bad answer, you get a complaint and a question: which step failed? Was it retrieval, the prompt, the model, or some combination of all three that produced something technically valid but factually wrong?

Standard application observability (latency histograms, error rates, structured logs) does not answer that question. It tells you the request completed in 1.3 seconds. It does not tell you whether the answer was useful, whether the retrieved context was relevant, or whether the prompt version from last Tuesday was working better than today's.

This is the engineering problem AI observability is designed to solve.

Why AI systems are harder to observe than APIs#

A standard REST API has deterministic outputs. Given the same input, the output is the same. You can test it, diff it, and alert on regressions trivially.

An AI pipeline does not work that way. A single user query might trigger a retrieval step, an embedding comparison, a context assembly step, a prompt construction step, and a model call. Each of those steps carries probabilistic or nondeterministic behavior, and the model itself introduces variance. Two identical requests can produce different outputs that are both "correct."

This creates three gaps in standard observability:

1. Causality is distributed. A bad output might be caused by a retrieval miss, a stale embedding index, a poorly constructed prompt, or model behavior on edge-case inputs.
2. Errors are often soft. The system returns HTTP 200. The answer is still wrong. Standard error rates do not capture this.
3. Prompt and model versions change output quality. A new prompt or model change is a deployment that your existing monitors do not know how to evaluate.

Standard APM tools handle the first gap partially through distributed tracing. They do not handle the second or third at all.

What teams usually do first#

Teams building their first AI feature typically start with what they know: logs and basic metrics. They log the final response, track latency, and alert on exceptions.

This covers a narrow band of failures. The system goes down and the monitor fires. The model API returns an error and the monitor fires. Everything else passes silently.

The result is a feature with no visibility into the questions that matter most. Is the retrieval step finding relevant context? Are users editing or ignoring the AI's outputs? Did the prompt change from last week cause the answer quality to drop? The team only finds out through user feedback or periodic manual review.

As the AI feature grows in usage, this gap becomes a reliability liability. Changes are risky because there is no fast signal on whether output quality changed. Debugging a quality complaint requires manually replaying the query and guessing which step was at fault.

The root cause: tracing stops before quality#

The core tradeoff in AI observability is between coverage and cost. Full tracing captures every prompt, every retrieved chunk, and every model input and output. That gives you complete debugging capability but increases storage and processing costs significantly. Minimal tracing gives you latency and errors but leaves quality blind spots.

The common mistake is treating this as a binary choice. Teams either log everything into a flat log store with no structure, or they log nothing beyond HTTP status codes and response times.

The right frame is: trace the pipeline structure first, then add quality signals selectively. You do not need to store every prompt to know when retrieval quality drops. You need the right signals in the right places.

Recommended approach: trace the full pipeline, score selectively#

An AI observability setup that catches real problems has three layers:

Layer 1: Pipeline tracing. Every logical step in the AI pipeline (retrieval, prompt construction, model call, post-processing) becomes a named span with attributes. The trace shows causality: which retrieval call fed which prompt, which model call returned which response.

Layer 2: Quality signals. At key steps, record signals that indicate output quality: retrieval chunk scores, prompt token counts, model confidence (if available), output length, and downstream user actions like corrections or regenerations.

Layer 3: Periodic scoring. Sample a fraction of real requests and run an automated quality check. That check can be a simple heuristic, a classifier, or an LLM-as-a-judge evaluation that produces a quality score per sampled request. It runs on a representative sample rather than every request, which keeps cost bounded while still detecting trend changes.

The flow is the same whether you instrument it by hand or with a library: your app emits spans for each step, those spans roll up into a trace, a collector ships them to a backend, and the backend powers dashboards and alerts.

OpenTelemetry's GenAI semantic conventions define standardized attributes for LLM calls. Those attributes include model name, input and output token counts, the model being called, and (when opted in) prompt and completion content, which keeps traces portable across vendors (OpenTelemetry: Semantic conventions for generative AI systems). Using standard attributes means your AI traces can flow into the same backend as the rest of your infrastructure observability.

Implementation details#

Instrument the pipeline as spans#

The first step is wrapping each logical step in a named span. For a RAG pipeline, that means:

from opentelemetry import trace

tracer = trace.get_tracer("ai-service")

def handle_query(user_query: str) -> str:
    with tracer.start_as_current_span("rag_pipeline") as root_span:
        root_span.set_attribute("query.length", len(user_query))

        with tracer.start_as_current_span("retrieval") as retrieval_span:
            chunks = retrieve_context(user_query)
            retrieval_span.set_attribute("retrieval.chunks_returned", len(chunks))
            retrieval_span.set_attribute("retrieval.top_score", chunks[0].score if chunks else 0)

        with tracer.start_as_current_span("llm_call") as llm_span:
            llm_span.set_attribute("gen_ai.request.model", "gpt-4o")
            response = call_model(user_query, chunks)
            llm_span.set_attribute("gen_ai.usage.input_tokens", response.usage.prompt_tokens)
            llm_span.set_attribute("gen_ai.usage.output_tokens", response.usage.completion_tokens)

        return response.content

The attribute names follow the OpenTelemetry GenAI semantic conventions (gen_ai.request.model, gen_ai.usage.input_tokens, and so on), which means they will display correctly in observability backends that support the standard.

Use a purpose-built tool for AI-specific signals#

General-purpose observability backends (Datadog, Honeycomb, Grafana) handle span data well. They are less suited to the AI-specific signals like prompt version tracking, LLM-as-a-judge scores, and retrieval quality trends.

Langfuse is an open-source option that is purpose-built for LLM applications. It natively understands token usage, model parameters, prompt versions, and prompt/completion pairs. It also supports LLM-as-a-judge evaluation scores attached to traces, which closes the quality visibility gap without requiring a separate system.

The Langfuse data model centers on traces, observations, and generations. A trace is the full request lifecycle. Observations are individual steps within it. Generations are LLM-specific observations with model, token count, and content attributes. This structure maps directly to what you would instrument manually, but with a UI designed for reviewing AI outputs rather than network call waterfalls.

Track prompt versions#

Prompts are deployable artifacts. A changed prompt is a code change that can silently alter output quality. Without version tracking, when a quality regression appears, the team cannot easily correlate it to a prompt change.

The minimum version tracking approach: tag every LLM call span with the prompt template name and version (or a short hash of the template content):

llm_span.set_attribute("prompt.name", "qa_system_prompt")
llm_span.set_attribute("prompt.version", "v3.1")

With this, a quality dashboard can group traces by prompt version and show whether output quality or token usage shifted when the prompt changed.

Surface user feedback as a signal#

User behavior is the highest-quality signal for AI output quality. An answer that was regenerated, explicitly corrected, or ignored is a signal that the pipeline produced something inadequate. An answer that was copied, acted on, or acknowledged is a signal that it was useful.

Log these events as spans or metrics alongside the original trace, linked by session or request ID. Even a simple binary (thumbs up / thumbs down) attached to traces provides a ground-truth quality signal that no automated scorer can fully replace.

Edge cases#

Token budget exhaustion. When retrieved context fills the model's context window, earlier chunks get truncated silently. The model sees incomplete context and may hallucinate. Instrument the token budget calculation explicitly and alert when truncation occurs above a threshold.

Retrieval returning zero results. A query that matches no indexed content often falls through to a prompt with empty context. The model either hallucinates a source or produces a vague answer. Both are failures that look like HTTP 200. Log the chunk count explicitly and treat zero-chunk responses as a category to monitor.

Cascading model errors. In multi-step agent workflows, a model call that returns a malformed JSON response can break downstream tool calls. The root span completes but intermediate steps failed silently. Structured tracing that records each tool call as a named span catches this. Flat logging does not.

Prompt injection in user input. User input that overrides the system prompt will appear in traces if you log raw user content. This is not an observability bug, but it is a data handling consideration. Decide whether to sanitize user content before logging or restrict access to prompt content in your observability backend.

Observability checklist for AI systems#

• Every AI pipeline step (retrieval, embedding, prompt build, model call, post-process) is a named span with attributes.
• Model name and version are recorded per LLM call.
• Input and output token counts are recorded per generation.
• Prompt template name and version are recorded per LLM call.
• Retrieval chunk count and top score are recorded per retrieval step.
• Zero-chunk retrieval events are tracked and alerted on.
• Token truncation events (context overflow) are tracked.
• User feedback events (corrections, regenerations) are linked to originating traces.
• A sampled quality score (heuristic or LLM-as-a-judge) runs on production traffic.
• Alerts exist for: elevated error rate, latency regression, zero-retrieval spike, quality score decline.

Key Takeaways#

AI observability is not a different kind of logging. It is a different structure of instrumentation that captures causality across the pipeline steps and connects system behavior to output quality.

The engineering problem is that standard tools see HTTP 200 and stop. AI observability extends the observation past the status code into the content: did retrieval return relevant context, did the model receive a valid prompt, was the output useful?

Start with pipeline tracing using OpenTelemetry's GenAI semantic conventions, add prompt version tagging, track retrieval quality signals, and run a sampled quality score on production traffic. Layer user feedback signals on top and use a purpose-built tool like Langfuse when general-purpose APM starts to feel limiting for AI-specific debugging.

For teams building toward a structured evaluation workflow (not just monitoring, but catching regressions before they reach production), see adding evaluation to an AI feature. For the broader system design context, see system design for AI products.

FAQ#

Can I use my existing APM (Datadog, New Relic, Honeycomb) for AI observability?

Partially. These tools handle span data well and can store OpenTelemetry attributes, including GenAI semantic convention attributes. They are less suited to AI-specific workflows like prompt version diffing, LLM-as-a-judge scoring on traces, and retrieval quality trending. A common pattern is to use a general APM for system metrics and latency, and a purpose-built tool like Langfuse for AI-specific quality signals.

Should I log full prompts and completions in production?

This depends on your data handling requirements. Logging full prompts and completions gives you complete debugging capability but increases storage costs and raises data privacy considerations if user data is included in prompts. A practical middle ground: log prompts and completions in development and staging always, and in production for a sampled subset or when errors occur. Use opt-in flags for sensitive content categories.

How is AI observability different from AI evaluation?

Observability is a continuous, real-time practice: watching production traffic, alerting on anomalies, debugging individual failures. Evaluation is a structured, periodic practice: running a defined test set against the system and scoring the results. Both are needed. Observability tells you when something is wrong in production. Evaluation tells you whether a change made things better or worse before you ship it.

What is an LLM-as-a-judge score and how do I use it in monitoring?

LLM-as-a-judge is a technique where a second LLM evaluates the output of your primary model against criteria like factual grounding, relevance, and completeness. You define a scoring rubric in a prompt, pass it the input and output, and receive a score. In monitoring, you run this on a sampled subset of production traffic and track the score trend over time. A declining score trend across a prompt or model change is a signal to investigate before it becomes a user-facing quality issue.

How many spans is too many for an AI pipeline trace?

Depth matters more than count. A trace with 20 flat spans is harder to interpret than a trace with 5 well-named, hierarchically organized spans. A good rule: one span per logical pipeline step, with child spans for sub-operations that have their own latency or error rate worth measuring. Avoid one span per LLM token or one span per document chunk, since that is too granular for production tracing and creates storage and UI overhead without useful signal.