Distributed Tracing with OpenTelemetry

When two agents run in separate processes and call each other over HTTP, the OpenTelemetry W3C traceparent header carries the trace context across the wire. Splunk Agent Observability joins all spans that share a trace ID into a single trace.

If both services are Python and use the Splunk Agent Observability SDK directly, Distributed Tracing is simpler. Use the OpenTelemetry approach when your services use different agent frameworks (Microsoft Agent Framework, Google ADK, LangChain, etc.) or different languages.

What you’ll see in Splunk Agent Observability

A single user request to Agent A that delegates to Agent B produces one trace:

POST /ask                              ← Agent A: FastAPI server span (workflow)
  invoke_agent OrchestratorAgent       ← Agent A: MAF agent span
    HTTP POST /research                ← Agent A: requests client span (workflow)
      POST /research                   ← Agent B: FastAPI server span (workflow)
        invoke_agent research_agent    ← Agent B: ADK agent span
          gemini.generate_content      ← Agent B: LLM span

Setup

.env

OPENAI_API_KEY=sk-...  # Find your key at https://platform.openai.com/api-keys
GOOGLE_API_KEY=...   # Find your key at https://aistudio.google.com

# Required for on-premises, standalone, custom deployments
# SPLUNK_AO_API_KEY="your-splunk-ao-api-key"
# SPLUNK_AO_CONSOLE_URL="https://app.example.com"
# Optional override for a nonstandard standalone API base URL
# SPLUNK_AO_API_URL="https://api.example.com"

# Required for SaaS (Splunk Observability Cloud) deployments
# SPLUNK_AO_O11Y_TOKEN="your-splunk-o11y-token"
# SPLUNK_AO_REALM="your-splunk-o11y-deployment-realm"

SPLUNK_AO_PROJECT=otel-distributed-tracing
SPLUNK_AO_AGENT_STREAM=default

AGENT_B_URL=http://localhost:8001

requirements.txt

fastapi
uvicorn[standard]
requests
python-dotenv
splunk-ao
opentelemetry-instrumentation-fastapi
opentelemetry-instrumentation-requests
agent-framework
agent-framework-openai
google-adk
openinference-instrumentation-google-adk

SplunkAOSpanProcessor reads the deployment credentials and routing values from the environment and exports OTLP directly to the Splunk Agent Observability OpenTelemetry endpoint. No collector is required. For on-premises deployments, SPLUNK_AO_CONSOLE_URL is required and the optional SPLUNK_AO_API_URL can override the API base URL; the SDK appends /otel/v1/traces.

Agent A: Microsoft Agent Framework, calls Agent B

agent_a.py

import os
import requests
from dotenv import load_dotenv
from fastapi import FastAPI
from pydantic import BaseModel

from splunk_ao.otel import SplunkAOSpanProcessor, add_splunk_ao_span_processor
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.instrumentation.fastapi import FastAPIInstrumentor
from opentelemetry.instrumentation.requests import RequestsInstrumentor

from agent_framework import openai, tool
from agent_framework.observability import enable_instrumentation

load_dotenv()

# Set the global tracer provider before any instrumentor runs.
tracer_provider = TracerProvider()
add_splunk_ao_span_processor(tracer_provider, SplunkAOSpanProcessor())
trace.set_tracer_provider(tracer_provider)

# Injects W3C traceparent on outbound HTTP.
RequestsInstrumentor().instrument(tracer_provider=tracer_provider)
# MAF built-in OTel
enable_instrumentation(enable_sensitive_data=True)

AGENT_B_URL = os.environ["AGENT_B_URL"]


@tool(approval_mode="never_require")
def delegate_research(topic: str) -> str:
    """Ask the research agent for facts about a topic."""
    r = requests.post(f"{AGENT_B_URL}/research", json={"topic": topic}, timeout=60)
    r.raise_for_status()
    return r.json()["findings"]


orchestrator = openai.OpenAIChatClient(model="gpt-4.1-mini").as_agent(
    name="OrchestratorAgent",
    instructions="Use delegate_research to gather facts, then answer the user clearly.",
    tools=[delegate_research],
)

app = FastAPI()
FastAPIInstrumentor.instrument_app(app, tracer_provider=tracer_provider)


class AskRequest(BaseModel):
    question: str


@app.post("/ask")
async def ask(req: AskRequest):
    return {"answer": str(await orchestrator.run(req.question))}

Agent B: Google ADK research agent

agent_b.py

from dotenv import load_dotenv
from fastapi import FastAPI
from pydantic import BaseModel

from splunk_ao.otel import SplunkAOSpanProcessor, add_splunk_ao_span_processor
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.instrumentation.fastapi import FastAPIInstrumentor

from openinference.instrumentation.google_adk import GoogleADKInstrumentor
from google.adk.agents.llm_agent import Agent
from google.adk.runners import InMemoryRunner
from google.genai.types import Content, Part

load_dotenv()

tracer_provider = TracerProvider()
add_splunk_ao_span_processor(tracer_provider, SplunkAOSpanProcessor())
trace.set_tracer_provider(tracer_provider)

GoogleADKInstrumentor().instrument(tracer_provider=tracer_provider)

agent = Agent(
    model="gemini-2.5-flash",
    name="research_agent",
    instruction="Return 3-5 concise key facts about the topic.",
)
runner = InMemoryRunner(agent=agent, app_name="otel-dt-demo")

app = FastAPI()
FastAPIInstrumentor.instrument_app(app, tracer_provider=tracer_provider)


class ResearchRequest(BaseModel):
    topic: str


@app.post("/research")
async def research(req: ResearchRequest):
    session = await runner.session_service.create_session(
        app_name="otel-dt-demo", user_id="user"
    )
    findings = ""
    # Drain the full async generator. Returning early closes it from a
    # different asyncio context and breaks OTel context-token cleanup.
    async for event in runner.run_async(
        user_id="user",
        session_id=session.id,
        new_message=Content(role="user", parts=[Part(text=req.topic)]),
    ):
        if event.is_final_response() and event.content and event.content.parts:
            findings = event.content.parts[0].text
    return {"findings": findings}

Run it

pip install -r requirements.txt

# Terminal 1
uvicorn agent_b:app --port 8001

# Terminal 2
uvicorn agent_a:app --port 8000

# Terminal 3
curl -s -X POST http://localhost:8000/ask \
  -H "Content-Type: application/json" \
  -d '{"question": "Tell me about Voyager 1."}' | jq

Open the otel-distributed-tracing project in Splunk Agent Observability, then open the default Agent Stream. You’ll see one trace covering both agents:

Distributed trace in Splunk Agent Observability UI showing both Agent A (Microsoft Agent Framework) and Agent B (Google ADK) spans nested under one trace

On-premises Splunk Agent Observability

This section applies to on-premises, standalone, and custom deployments. The OTel endpoint is different from Splunk Agent Observability’s regular API endpoint and is specifically designed to receive telemetry data in the OTLP format. If you are using:

A SaaS deployment integrated with Splunk Observability Cloud, then you don’t need to provide a custom OTel endpoint. The SDK automatically derives the https://ingest.<realm>.observability.splunkcloud.com/v2/trace/otlp endpoint from SPLUNK_AO_REALM.
An on-premises, standalone, or custom deployment, replace the <your-splunk-ao-api-url>/otel/v1/traces endpoint with your deployment URL. The format of this URL is based on your console URL, appending /otel/v1/traces.

Set SPLUNK_AO_API_URL in .env; SplunkAOSpanProcessor picks it up automatically and appends /otel/v1/traces.

Distributed Tracing: Splunk Agent Observability’s native Python SDK distributed mode.
Microsoft Agent Framework integration
Google ADK (OpenTelemetry) integration

Overview

SaaS deployments

Get Started

Observability

Evaluators

Tokenomics

Signals

AI Assistant

Luna Studio

Experiments

Agent Control

Annotations

Integrations

Security

References

What you’ll see in Splunk Agent Observability

Setup

Agent A: Microsoft Agent Framework, calls Agent B

Agent B: Google ADK research agent

Run it

On-premises Splunk Agent Observability

​What you’ll see in Splunk Agent Observability

​Setup

​Agent A: Microsoft Agent Framework, calls Agent B

​Agent B: Google ADK research agent

​Run it

​On-premises Splunk Agent Observability

​Related

What you’ll see in Splunk Agent Observability

Setup

Agent A: Microsoft Agent Framework, calls Agent B

Agent B: Google ADK research agent

Run it

On-premises Splunk Agent Observability

Related