Overview

This guide explains how to integrate Splunk Agent Observability with OpenTelemetry and OpenInference for comprehensive observability and tracing of your AI/ML workflows using industry-standard tools.

OpenTelemetry

The first step is to configure OpenTelemetry.

Installation

Add the Splunk Agent Observability SDK and OpenTelemetry packages to your project:

pip install opentelemetry-api opentelemetry-sdk \
            opentelemetry-exporter-otlp

For Python, the opentelemetry-api and opentelemetry-sdk packages provide the core OpenTelemetry functionality. The opentelemetry-exporter-otlp package enables sending traces to Splunk Agent Observability’s OTLP endpoint.

Create environment variables for your Splunk Agent Observability settings

Set environment variables for your Splunk Agent Observability settings, for example in a .env file. These environment variables are consumed by the SplunkAOSpanProcessor to authenticate and route traces to the correct Splunk Agent Observability Project and Agent Stream:

# Provide your Splunk Agent Observability
# API key and URL if you are using an
# on-premises, standalone, or custom deployment
# SPLUNK_AO_API_KEY="your-splunk-ao-api-key"
# SPLUNK_AO_CONSOLE_URL="your-splunk-ao-url"

# Your Splunk Agent Observability project name
SPLUNK_AO_PROJECT="your-splunk-ao-project-name"

# The name of the Agent Stream you want to use for logging
SPLUNK_AO_AGENT_STREAM="your-splunk-ao-agent-stream"

Self-hosted deployments: Set the OTel endpoint

Skip this step if you are using a hosted version.

The OTel endpoint is different from Splunk Agent Observability’s regular API endpoint and is specifically designed to receive telemetry data in the OTLP format.If you are using:

A cloud deployment, then you don’t need to provide a custom OTel endpoint. The default endpoint <your-splunk-ao-api-url>/otel/traces will be used automatically.
A self-hosted deployment, replace the <your-splunk-ao-api-url>/otel/traces endpoint with your deployment URL. The format of this URL is based on your console URL, appending /otel/traces.

The convention is to store this in the SPLUNK_AO_CONSOLE_URL environment variable. For example:

os.environ["SPLUNK_AO_CONSOLE_URL"] = "<your-splunk-ao-url>"

Initialize and create the Splunk Agent Observability span processor

The SplunkAOSpanProcessor automatically configures authentication and metadata using your environment variables. It also:

Auto-builds OTLP headers using your Splunk Agent Observability credentials
Configures the correct OTLP trace endpoint
Registers a batch span processor that exports traces to Splunk Agent Observability

from splunk_ao import otel  

# SplunkAOSpanProcessor (no manual OTLP config required) loads the env vars for 
# the Splunk Agent Observability API key, Project, and Agent Stream. Make sure to set them first. 
splunk_ao_span_processor = otel.SplunkAOSpanProcessor(
    # Optional parameters if not set, uses env var
    # project=os.environ["SPLUNK_AO_PROJECT"], 
    # agentstream=os.environ.get("SPLUNK_AO_AGENT_STREAM"),  
)

The span processor can now be registered with an OTel trace provider.

from opentelemetry.sdk import trace as trace_sdk

tracer_provider = trace_sdk.TracerProvider()
tracer_provider.add_span_processor(splunk_ao_span_processor)

You can now use this trace provider either using a framework that supports OTel directly, or via OpenInference.

OpenInference

OpenInference instrumentors are currently available for Python only.

You can enable automatic tracing for your framework and LLM operations using OpenInference instrumentors. These add AI-specific semantic conventions to your traces. For example, to instrument LangChain and OpenAI start by adding the relevant OpenInference packages:

pip install openinference-instrumentation-langchain \
            openinference-instrumentation-openai

Now you can add the instrumentation to your code, using the OTel trace provider.

from openinference.instrumentation.langgraph import (
    LangGraphInstrumentor
)
from openinference.instrumentation.openai import (
    OpenAIInstrumentor
)

LangGraphInstrumentor().instrument(tracer_provider=tracer_provider)
OpenAIInstrumentor().instrument(tracer_provider=tracer_provider)

OpenInference adds:

Automatic capture of LLM calls, token usage, and model performance evaluators
AI-specific span attributes like gen_ai.request.model, gen_ai.response.content, and gen_ai.usage.*
Semantic conventions that make your traces more meaningful in Splunk Agent Observability’s dashboard
Framework-specific instrumentation for LangGraph workflows and OpenAI API calls

Once OpenTelemetry and OpenInference is set up your application will automatically capture and send observability data to Splunk Agent Observability with every run, providing complete traces of your AI workflows, detailed LLM call breakdowns, and performance insights organized by project and Agent Stream. For a detailed example of using OpenTelemetry and OpenInference with LangGraph, see the Log with OpenTelemetry, LangGraph, and OpenAI how-to guide.

Next steps

Learn how to integrate with some popular frameworks using OpenTelemetry and OpenInference.

Google ADK

Learn how to integrate a Google ADK project with Splunk Agent Observability using OpenTelemetry and OpenInference.

Strands Agents

Learn how to integrate a Strands Agents project with Splunk Agent Observability using OpenTelemetry.

Vercel AI SDK

Learn how to integrate a Vercel AI SDK project with Splunk Agent Observability using OpenTelemetry.

Mastra

Learn how to integrate a Mastra project with Splunk Agent Observability using OpenTelemetry.

Get Started

Observability

Evaluators

Signals

AI Assistant

Luna Studio

Experiments

Agent Control

Annotations

Integrations

Security

References

Overview

OpenTelemetry

OpenInference

Next steps

Google ADK

Strands Agents

Vercel AI SDK

Mastra

​OpenTelemetry

​OpenInference

​Next steps

Google ADK

Strands Agents

Vercel AI SDK

Mastra

OpenTelemetry

OpenInference

Next steps