Tracing and OpenTelemetry for LLMs

How it works: tracing

1. A trace represents one request (or one agent task) end to end.
2. It contains spans: timed steps with a name, start and end, a parent (so they form a tree), a status (ok or error) and attributes (key-value details).
3. For LLM apps, record per span: inputs and outputs (prompts, completions, retrieved chunk IDs), model name, parameters, token counts, cost, latency, tool names and arguments, errors, and identifiers (user, session, prompt version, app version).
4. Instrument your code: wrap each step in a span, manually or automatically with libraries that instrument LLM SDKs and frameworks.
5. Export spans to a backend (Langfuse, LangSmith, Arize Phoenix, Datadog, Grafana Tempo, Honeycomb…), where you search, filter, view trees and build dashboards.
6. Use traces everywhere: debugging, error analysis, agent evals, cost and latency metrics, and adding real failures to golden sets.
7. Protect data: redact secrets and personal information, sample if volume is high, and control retention.

How it works: OpenTelemetry for LLMs

1. OpenTelemetry (OTel) is the open, vendor-neutral standard (API, SDKs, and the OTLP wire protocol) for traces, metrics and logs, used across the software industry.
2. Instrument once, export anywhere: a tracer creates spans; an exporter sends them (via OTLP) to any compatible backend, so switching tools doesn't mean re-instrumenting.
3. GenAI semantic conventions define standard attribute names for model calls, such as gen_ai.operation.name (e.g. chat), gen_ai.provider.name, gen_ai.request.model, gen_ai.response.model, gen_ai.usage.input_tokens and gen_ai.usage.output_tokens. As of November 2026 they live in their own OpenTelemetry repository and are still marked "Development" (they may change).
4. Auto-instrumentation packages (OpenTelemetry contrib and others like OpenLLMetry and OpenInference) wrap popular LLM SDKs and frameworks to emit these spans automatically.
5. Context propagation links spans across services (a trace ID passed in HTTP headers), so a request through your API, a worker queue and a model call appears as one trace.
6. Many LLM observability tools accept OTel directly (Langfuse's v5 SDK is built on OpenTelemetry), so the same spans power both general and AI-specific dashboards.

from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import SimpleSpanProcessor
from opentelemetry.sdk.trace.export.in_memory_span_exporter import InMemorySpanExporter
 
exporter = InMemorySpanExporter()                    # in production: an OTLP exporter to your backend
provider = TracerProvider()
provider.add_span_processor(SimpleSpanProcessor(exporter))
tracer = provider.get_tracer("rag-app")
 
def retrieve(q):
    with tracer.start_as_current_span("retrieve") as span:
        span.set_attribute("retrieval.top_k", 3)
        return ["chunk-1", "chunk-2", "chunk-3"]
 
def call_llm(prompt):
    # GenAI semantic conventions: standard attribute names any backend understands.
    with tracer.start_as_current_span("chat claude-haiku-5-5") as span:
        span.set_attribute("gen_ai.operation.name", "chat")
        span.set_attribute("gen_ai.provider.name", "anthropic")
        span.set_attribute("gen_ai.request.model", "claude-haiku-5-5")
        span.set_attribute("gen_ai.usage.input_tokens", 812)
        span.set_attribute("gen_ai.usage.output_tokens", 64)
        return "You get 25 days."
 
with tracer.start_as_current_span("POST /ask") as root:   # one trace = the whole request
    root.set_attribute("user.id", "u42")
    call_llm(retrieve("leave days?"))
 
for s in sorted(exporter.get_finished_spans(), key=lambda s: (s.parent is not None, s.start_time)):
    parent = s.parent.span_id if s.parent else None
    depth = 0 if parent is None else 1
    attrs = {k: v for k, v in s.attributes.items() if k.startswith(("gen_ai.usage", "retrieval", "user"))}
    print("  " * depth + f"{s.name:24} {attrs}")

POST /ask                {'user.id': 'u42'}
  retrieve                 {'retrieval.top_k': 3}
  chat claude-haiku-5-5    {'gen_ai.usage.input_tokens': 812, 'gen_ai.usage.output_tokens': 64}