LLM Observability: Metrics, Traces and gen_ai Spans

Ask a team running AI in production a few basic questions — what's our p95 latency to Anthropic? how many tokens did the support team burn last week? what's our error rate on gpt-4o? — and watch how long the answer takes. If it's a research project rather than a dashboard glance, you don't have LLM observability.

Observability isn't a nice-to-have next to governance; it's the layer underneath it. Policy enforcement, cost attribution, and audit are only as trustworthy as the telemetry that proves they ran. "We block disallowed models" needs a metric that shows the blocks; "we attribute every dollar" needs token counts per workspace; "we can prove what happened" needs traces and logs. In DVARA, the DVARA LLM Gateway emits that evidence on every request, which is what turns governance claims into governance you can audit.

What LLM observability is

LLM observability is the practice of instrumenting your model traffic so you can measure and diagnose it: request rate, latency distribution, token consumption, cost, cache behavior, and errors — broken down by the dimensions that matter (model, provider, workspace, status). It's ordinary observability applied to a new kind of traffic, plus a few AI-specific signals (tokens, cache hits, provider failovers) that generic app monitoring doesn't capture.

Because a governed control point sits on the path of every call, it's the natural place to emit this telemetry consistently — every request, every provider, one schema.

The three pillars, applied to LLMs

Metrics

The counters and histograms you actually watch on a dashboard:

• Request rate and latency — requests/sec and p50/p95/p99 latency, tagged by model, provider, and workspace.
• Token usage — input and output tokens, per model and workspace. This is the raw material for cost governance.
• Errors — provider errors by code, so you see which provider is failing and how.
• Resilience signals — retries, fallbacks (from-provider → to-provider), and rate-limit sheds, which tell you whether your failover is a quiet safety net or a daily tax.

Exposed in a standard format (Prometheus), these drop straight into the monitoring stack you already run.

Traces

A single request often fans out — filter chain, routing, an upstream call, maybe a failover to a second provider. Distributed tracing stitches those into one timeline so you can see where the milliseconds and the failures actually went, with trace context propagated to the upstream call.

Logs

One structured log line per request — trace ID, model, provider, status, latency, token counts, workspace, masked API key, cache status, error code — is the record you grep when something specific went wrong. Structured (JSON), not prose, so it's queryable.

The GenAI semantic conventions

Here's the piece that's new and worth understanding. OpenTelemetry has standardized a set of GenAI semantic conventions — a common vocabulary for model telemetry, with attributes like gen_ai.system (which provider), gen_ai.request.model, gen_ai.usage.input_tokens, gen_ai.usage.output_tokens, and gen_ai.response.model.

By default, the DVARA LLM Gateway sets these ten on the span for every provider call (management.tracing.genai-semconv.enabled turns them off):

Attribute	What it records
gen_ai.system	The provider family, such as openai or anthropic
gen_ai.operation.name	The kind of call: chat or embeddings
gen_ai.request.model	The model the request asked for
gen_ai.request.max_tokens	The output-token limit on the request
gen_ai.request.temperature	The sampling temperature on the request
gen_ai.response.model	The model that actually answered
gen_ai.response.id	The provider's response id
gen_ai.response.finish_reasons	Why generation stopped
gen_ai.usage.input_tokens	Input tokens used
gen_ai.usage.output_tokens	Output tokens used

Which observability platforms support gen_ai span attributes? Any OpenTelemetry backend stores them as ordinary span attributes. GenAI-aware backends go further and read them as model calls, with tokens and models broken out.

Why it matters: if your platform emits these standard attributes on its spans, your traces are portable. GenAI-aware backends — Langfuse, Arize Phoenix, and others — understand them without custom parsing. You're not locked into one vendor's bespoke schema; you're speaking the ecosystem's shared language. The DVARA LLM Gateway emits gen_ai.* alongside its own span attributes, so your governance telemetry travels to whatever backend you standardize on.

What to actually put on the dashboard

• Traffic: requests/sec by model and workspace.
• Latency: p95 by provider (where failover decisions get validated).
• Tokens: input/output per workspace per day (the cost driver).
• Errors: provider error rate by code.
• Resilience: fallback and retry rates by provider pair.
• Cache: hit rate (ties to cost).

For the request-level view — streaming, structured outputs, and per-call telemetry together — see the deep dive on streaming, structured outputs & observability.

An evaluation checklist

• Are Prometheus metrics exposed for rate, latency, tokens, and errors — tagged by model/provider/workspace?
• Is there distributed tracing with context propagated to upstream calls?
• Does it emit the OpenTelemetry GenAI conventions (gen_ai.*), so traces are portable?
• Is there one structured log line per request with the fields you'd need to debug?
• Are resilience and cache signals (retries, fallbacks, hit rate) observable, not just request/latency?

Where DVARA fits

DVARA is an AI gateway that governs model, tool and agent-to-agent calls, and observability is the evidence layer beneath it: its LLM Gateway exposes Prometheus metrics for request rate, latency percentiles, token usage, cost, and errors — tagged by workspace, model, and provider — and emits OpenTelemetry traces carrying the standard GenAI semantic-convention attributes (gen_ai.*) so they're portable, alongside one structured JSON log line per request. That telemetry is what lets you prove policy, cost, and audit ran. Read the pillar guide, What Is an LLM Gateway?, for how observability fits the rest.