Distributed Tracing (Beta)

​

Distributed Traces in the Splunk Agent Observability UI

• Retrieval Service: A FastAPI service running on port 8000 that handles information retrieval.
• Orchestrator Service: The main client that coordinates the RAG pipeline by calling the retrieval service.

​

Example: Two-service RAG pipeline

1. Client starts a trace
2. Client makes HTTP requests to one or more server(s)
3. Server processes the request and reports back to the client
4. Client concludes the trace

​

How to enable Distributed mode

OPENAI_API_KEY="your-openai-api-key"
SPLUNK_AO_PROJECT="your-splunk-ao-project"
SPLUNK_AO_AGENT_STREAM="distributed-tracing-example"

SPLUNK_AO_MODE=distributed # enables distributed tracing

# If you are using an on-premises, standalone, or custom deployment
# provide your API key and deployment URL below
# SPLUNK_AO_API_KEY="your-splunk-ao-api-key"
# SPLUNK_AO_CONSOLE_URL="your-splunk-ao-url"

from splunk_ao import splunk_ao_context

# Initialize in distributed mode
splunk_ao_context.init(mode="distributed")

​

Beta limitations for Distributed Tracing

1. Fire-and-forget patterns: Where the client starts a span and doesn’t wait for the server to complete. The client must await the server’s response.
2. Parallel nested workflows: Starting multiple child workflows in parallel using asyncio.gather() where each child makes its own HTTP requests to servers.
   example.py

   @log(span_type="workflow")
   async def parent_workflow():
       tasks = [child_workflow(i) for i in range(3)]
       await asyncio.gather(*tasks)  # Each child makes HTTP requests
   
   Note: This limitation will be addressed in a future release for both distributed tracing and default modes.
3. Integrations - Distributed Tracing does not yet support the following integrations: ADK, OTel, LangChain

​

Troubleshooting Tips

• Ensure you’re passing get_tracing_headers() to downstream HTTP requests
• Verify the server has TracingMiddleware configured
• Check that both services have distributed mode enabled

• Confirm that the server is sending responses back to the client (not fire-and-forget)
• Verify that the client awaits the server response before concluding the trace