Distributed Tracing

If your application uses a microservices architecture, distributed tracing allows you to trace requests across multiple services. With distributed tracing enabled, spans created in downstream services (e.g. a retrieval service) get automatically linked to the parent trace in the upstream service (e.g. an orchestrator service).

Distributed Traces in the Splunk Agent Observability UI

Inputs and outputs from distributed traces are combined into one view in the UI.

The above screenshot shows a session, traces, and spans coming from 2 services on different processes:

Retrieval Service: A FastAPI service running on port 8000 that handles information retrieval.
Orchestrator Service: The main client that coordinates the RAG pipeline by calling the retrieval service.

Example: Two-service RAG pipeline

A code example of setting up Distributed Tracing is available in the DT 2.0 directory of the splunk-ao Python SDK repository. This code example follows a client-server model where:

Client starts a trace
Client makes HTTP requests to one or more server(s)
Server processes the request and reports back to the client
Client concludes the trace

How to enable distributed tracing

Distributed tracing uses standard W3C Trace Context propagation. The SDK can configure the supported HTTP instrumentors once during application startup.

OPENAI_API_KEY="your-openai-api-key"
SPLUNK_AO_PROJECT="your-splunk-ao-project"
SPLUNK_AO_AGENT_STREAM="distributed-tracing-example"

# Required for on-premises, standalone, custom deployments
# SPLUNK_AO_API_KEY="your-splunk-ao-api-key"
# SPLUNK_AO_CONSOLE_URL="your-splunk-ao-url"

# Required for SaaS (Splunk Observability Cloud) deployments
# SPLUNK_AO_REALM="your-splunk-ao-realm"
# SPLUNK_AO_O11Y_TOKEN="your-splunk-ingest-token"

Install the automatic HTTP instrumentation dependencies:

pip install "splunk-ao[distributed-tracing]"

Configure them once when the process starts:

from fastapi import FastAPI
from splunk_ao import configure_distributed_tracing

app = FastAPI()
provider = configure_distributed_tracing(app=app)

The setup instruments Requests, HTTPX, and aiohttp clients. Passing a FastAPI or Starlette app also enables inbound trace-context extraction. The application owns the returned provider and should call provider.shutdown() when the process is shutting down.

Manual propagation

Manual propagation remains available for unsupported transports or applications that need direct control. Add the W3C headers to an outbound request:

import requests

from splunk_ao import get_tracing_headers, log

@log
def call_downstream(url, payload):
    response = requests.post(
        url,
        headers=get_tracing_headers(),
        json=payload,
    )
    response.raise_for_status()
    return response

If you choose manual propagation instead of automatic instrumentation, use the SDK middleware to extract incoming context in a Starlette-based server. Install the middleware extra first:

pip install "splunk-ao[middleware]"

from splunk_ao.middleware import TracingMiddleware

app.add_middleware(TracingMiddleware)

When using configure_distributed_tracing(app=app), do not add TracingMiddleware; FastAPI and Starlette inbound propagation is configured automatically.

Other OpenTelemetry instrumented frameworks

For Flask, Django, and other frameworks, use the framework’s OpenTelemetry instrumentor with a provider configured with add_splunk_ao_span_processor(). Configure the instrumentor to use that provider, and follow its documentation for inbound W3C context extraction. Install the base SDK and the OpenTelemetry instrumentor for your framework before configuring the provider:

pip install splunk-ao opentelemetry-instrumentation-flask
# or
pip install splunk-ao opentelemetry-instrumentation-django

Minimal provider setup:

from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from splunk_ao.otel import add_splunk_ao_span_processor

provider = TracerProvider()
add_splunk_ao_span_processor(provider)
trace.set_tracer_provider(provider)

Configure the instrumentor to use provider, following the framework instrumentor’s documentation. Do not combine automatic instrumentation and manual propagation for the same transport.

Troubleshooting Tips

Traces not linking across services?

With automatic instrumentation, verify that the client transport and the server application were both configured during startup. Each process owns its own provider.
With manual propagation, verify that get_tracing_headers() is added to the outbound request and that the server extracts the W3C context.
Check that both services use the same project and Agent Stream configuration.

Missing spans in the trace?

Keep the provider alive until the work completes and call provider.shutdown() during application shutdown so completed spans can be drained.
Do not add per-trace flush calls; the SDK uses the normal OpenTelemetry batch processor for completed spans.

Overview

SaaS deployments

Get Started

Observability

Evaluators

Tokenomics

Signals

AI Assistant

Luna Studio

Experiments

Agent Control

Annotations

Integrations

Security

References

Distributed Traces in the Splunk Agent Observability UI

Example: Two-service RAG pipeline

How to enable distributed tracing

Manual propagation

Other OpenTelemetry instrumented frameworks

Troubleshooting Tips

​Distributed Traces in the Splunk Agent Observability UI

​Example: Two-service RAG pipeline

​How to enable distributed tracing

​Manual propagation

​Other OpenTelemetry instrumented frameworks

​Troubleshooting Tips

Distributed Traces in the Splunk Agent Observability UI

Example: Two-service RAG pipeline

How to enable distributed tracing

Manual propagation

Other OpenTelemetry instrumented frameworks

Troubleshooting Tips