OpenTelemetry for LLM applications: what to trace

OpenTelemetry gives an LLM application a common way to describe and export telemetry. It does not decide whether an answer was correct. To investigate quality, the trace must contain enough evidence to connect the user's request to the retrieval, model calls, and tools that produced the result.

Start with the question you want the trace to answer: where did this request spend time, and which step introduced the wrong information or action?

Give one request a coherent trace

OpenTelemetry represents operations as spans, with parent-child relationships connecting work within a trace. Context propagation carries that relationship across service boundaries. OpenTelemetry trace concepts.

For a support agent, an illustrative trace might look like this:

support_request
  retrieve_policy
  generate_response
  lookup_order
  update_delivery_address
  generate_confirmation

Actual nesting should reflect which operation initiated each step. Record retry attempts as distinct operations, including the error that caused the retry. If the request crosses a worker queue, preserve the relevant trace context rather than relying on similar timestamps to reconstruct the relationship.

Use a conversation identifier to associate requests from the same interaction. A conversation can contain several traces; forcing a long conversation into one continuously open span is not required.

Capture fields that explain behavior

Use standard GenAI attributes where your instrumentation supports them, and document application-specific attributes. OpenTelemetry defines fields for model identity, messages, tool arguments, and results; support varies by instrumentation and convention version. Record the versions you deploy rather than assuming every exporter interprets every field identically. GenAI attribute registry.

Capture content deliberately. Redact secrets and unnecessary personal data before export. When full content cannot be stored, retain stable document references and enough structured metadata to diagnose behavior within your retention policy.

Avoid misleading totals

A parent span already includes elapsed time spent waiting on its children. Summing all span durations double-counts nested work. Parallel operations also overlap. Use the request's wall-clock duration for end-to-end latency and inspect the timeline to find blocking operations.

Similarly, distinguish token usage reported for an individual model call from totals aggregated at an agent or request level. Counting both inflates usage. Keep estimated cost separate from provider billing, especially when caches, retries, or provider-specific usage fields are involved.

For streaming, separate time to first token from total completion time. A quick first token can coexist with a slow or incomplete response.

Verify the export before building dashboards

Trace one ordinary request, one tool error, and one retry. Check that the span relationships, model fields, usage values, and application version survive ingestion. Repeat the check for a queued or asynchronous request if your application uses one.

Braintrust documents OTLP exporters, a span processor, and Collector-based paths for ingestion. Its generic OTLP endpoint and signal-specific trace endpoint differ, so use the configuration appropriate to your exporter and deployment region. Braintrust OpenTelemetry integration.

Sampling is another measurement choice. Retaining every error but only a small fraction of successful requests is useful for debugging; that trace collection cannot directly estimate the overall error rate without accounting for sampling.

Add quality evaluation after the trace is usable

Once the evidence is present, attach outcome checks or tool-call evals. Keep a link from each score to the trace that supports it. For retrieval failures, inspect retrieval and answer metrics separately.

The result should let an engineer move from a quality regression to the specific operation that needs changing. A larger volume of spans is useful only when it makes that investigation possible.

Continue with the evals implementation guide for the full workflow.