5.2.1. Distributed Tracing with OpenTelemetry

💡 First Principle: OpenTelemetry standardizes the tracking number: a trace is one request's whole journey; spans are its timed segments, each knowing its parent; and context propagation — the W3C traceparent header carried on every HTTP call and queue message — is what stitches spans from different services into one trace. Instrument the transport, propagate the context, and the story assembles itself.

OpenTelemetry (OTel) is the vendor-neutral standard for traces, metrics, and logs; the Azure Monitor OpenTelemetry distro wires it to Application Insights in one call:

from azure.monitor.opentelemetry import configure_azure_monitor
from opentelemetry import trace

configure_azure_monitor()          # reads APPLICATIONINSIGHTS_CONNECTION_STRING
tracer = trace.get_tracer("rag-api")

def answer(question: str):
    with tracer.start_as_current_span("retrieve-context") as span:
        span.set_attribute("rag.top_k", 5)
        chunks = search_pgvector(question)     # auto-instrumented: DB span
    with tracer.start_as_current_span("call-model"):
        return llm(question, chunks)           # auto-instrumented: HTTP span

Auto-instrumentation covers the common libraries — HTTP clients, database drivers, Azure SDKs — creating spans without code changes; manual spans (as above) mark your business logic, with attributes carrying domain detail (rag.top_k, tenant ID) you'll filter on later in KQL. The connection string is configuration, not a secret-in-code — supplied as an app setting per the 3.1.3 doctrine.

Propagation is the make-or-break mechanic. HTTP hops carry traceparent automatically once instrumented. The subtle hop is through a queue: Service Bus messages don't magically carry HTTP headers, so instrumented senders stash trace context in message application properties and instrumented receivers restore it — link publisher and consumer spans, or the trace dies at the queue and your "slow chatbot" investigation hits a wall exactly where the async work begins:

At volume, sampling keeps costs sane — record a fraction of traces, statistically representative; the distro samples consistently so a kept trace is kept at every service, avoiding half-stories.

⚠️ Exam Trap: "Traces stop at the queue" scenarios are answered by propagating context in message properties, not by raising sampling, adding logs, or scaling workers. Distractors love adjacent-sounding fixes; the break is always the unpropagated context.

Span vocabulary goes slightly deeper than parent-and-child. Spans carry a kind — client and server for the two ends of an HTTP call, producer and consumer for the two ends of a queue hop — which is how backends draw correct service maps from raw spans. Spans also carry status: recording an exception on a span marks the segment failed and surfaces it in failure analytics, so record_exception belongs in your error paths. The same SDK emits metrics (counters, histograms) through a parallel API, and log records can carry the active trace ID so even plain logs join the journey. One naming distinction prevents confusion: the vendor-neutral OTLP exporter ships telemetry to any OTel collector, while the Azure Monitor distro is the batteries-included wrapper that targets Application Insights specifically — same instrumentation, different destination wiring.

Two more propagation details round out the toolkit. Resource attributes — set once per service, most importantly service.name — are how spans from many services stay attributable in a shared workspace; forgetting them yields service maps full of 'unknown_service' nodes. And beyond the trace ID itself, W3C baggage carries small key-value pairs (a tenant ID, an experiment flag) alongside the context, letting downstream spans tag themselves with request-scoped facts they never saw directly. Use baggage sparingly: every hop carries it, so it's a header tax paid across the whole journey.

Reflection Question: Why must sampling decisions be consistent across services — what analytical artifact appears if each service samples independently?