Run experiments with OpenTelemetry

If your application uses a framework with built-in OpenTelemetry instrumentation, you can run experiments against it using Splunk Agent Observability’s SplunkAOSpanProcessor. The span processor captures OTel traces and routes them to the experiment automatically — no need to use the Splunk Agent Observability log decorator or manual logging. This is useful when you are working with frameworks like Microsoft Agent Framework, Google ADK, Pydantic AI, or any other framework that emits OTel spans. In this guide you will:

Set up OpenTelemetry with Splunk Agent Observability
Create an agent with framework instrumentation
Expose an entry point for the experiment runner
Run the experiment

How it works

When you run an experiment with a custom function, Splunk Agent Observability’s experiment runner:

Creates an experiment and sets the experiment_id in the Splunk Agent Observability context.
For each row in the dataset, it calls your function and wraps the call with dataset context (input, ground truth, and metadata).
The SplunkAOSpanProcessor reads this context and attaches it to every OTel span your framework creates.
Spans are exported to Splunk Agent Observability’s OTLP endpoint with the experiment ID, routing them to the experiment instead of a regular Agent Stream.

Because the experiment runner manages the trace lifecycle, you must disable Splunk Agent Observability’s native logger to avoid duplicate traces. Set SPLUNK_AO_LOGGING_DISABLED=true before importing your agent code.

Prerequisites

Install the Splunk Agent Observability SDK with OpenTelemetry support, plus any framework-specific packages:

Terminal

pip install "splunk-ao[otel]" opentelemetry-api opentelemetry-sdk opentelemetry-exporter-otlp

Set up environment variables

Configure your Splunk Agent Observability credentials and disable the native logger:

.env

# Splunk Agent Observability
SPLUNK_AO_PROJECT=your-project-name
SPLUNK_AO_API_URL=your-api-url

# Only required for on-premises, standalone, and custom deployments
# SPLUNK_AO_API_KEY=your-splunk-ao-api-key
# SPLUNK_AO_CONSOLE_URL=your-splunk-ao-url

# Disable native Splunk Agent Observability logger — OTel handles tracing
SPLUNK_AO_LOGGING_DISABLED=true

# LLM provider
OPENAI_API_KEY=your-openai-api-key

Variable	Required	Description
`SPLUNK_AO_API_KEY`	No	Only needed for on-premises, standalone, and custom deployments. Your Splunk Agent Observability API key
`SPLUNK_AO_O11Y_TOKEN`	No	Only needed for SaaS deployments. Your Splunk Observability Cloud token
`SPLUNK_AO_REALM`	No	Only needed for SaaS deployments. Your Splunk Agent Observability Splunk Observability Cloud realm
`SPLUNK_AO_PROJECT`	Yes	The project to log experiments to
`SPLUNK_AO_CONSOLE_URL`	No	Only needed for on-premises, standalone, and custom deployments
`SPLUNK_AO_API_URL`	No	Used by the `SplunkAOSpanProcessor` to construct the OTLP endpoint
`SPLUNK_AO_LOGGING_DISABLED`	Yes	Set to `true` to disable the native logger. Required when using OTel for experiments

Set up OpenTelemetry with Splunk Agent Observability

Create a TracerProvider and attach the SplunkAOSpanProcessor. This processor automatically configures authentication and the OTLP endpoint using your environment variables.

agent.py

from splunk_ao.otel import SplunkAOSpanProcessor, add_splunk_ao_span_processor
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider

# Set up the OTel tracer provider with the Splunk Agent Observability span processor
tracer_provider = TracerProvider()
splunk_ao_processor = SplunkAOSpanProcessor()
add_splunk_ao_span_processor(tracer_provider, splunk_ao_processor)
trace.set_tracer_provider(tracer_provider)

Then enable your framework’s instrumentation. For example, with Microsoft Agent Framework:

agent.py

from agent_framework.observability import enable_instrumentation

# Enable the framework's built-in OTel instrumentation
# Set enable_sensitive_data=True to capture LLM inputs and outputs
enable_instrumentation(enable_sensitive_data=True)

Set enable_sensitive_data=True to capture LLM inputs and outputs in your traces. If set to False, only span metadata (timing, token counts, etc.) is sent.

Create the agent

Set up your agent with your framework of choice. The key requirement is that the framework emits OTel spans through the registered TracerProvider.

agent.py

import asyncio
import os
from typing import Any

from agent_framework import openai, tool
from agent_framework.observability import enable_instrumentation
from splunk_ao.otel import SplunkAOSpanProcessor, add_splunk_ao_span_processor
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider

# ── OTel Setup ───────────────────────────────────────────────────────────────
tracer_provider = TracerProvider()
splunk_ao_processor = SplunkAOSpanProcessor()
add_splunk_ao_span_processor(tracer_provider, splunk_ao_processor)
trace.set_tracer_provider(tracer_provider)

enable_instrumentation(enable_sensitive_data=True)

# ── Tools ────────────────────────────────────────────────────────────────────
@tool(approval_mode="never_require")
def lookup_account(account_id: str) -> str:
    """Look up account details by account ID."""
    accounts = {
        "ACCT-1001": {"name": "Alice", "plan": "Premium", "balance": 120.00},
        "ACCT-1002": {"name": "Bob", "plan": "Basic", "balance": 45.50},
    }
    account = accounts.get(account_id)
    if account:
        return f"Account {account_id}: {account}"
    return f"Account {account_id} not found."

# ── Agent ────────────────────────────────────────────────────────────────────
client = openai.OpenAIChatClient(model_id="gpt-4.1-mini")

agent = client.as_agent(
    name="BillingAgent",
    instructions="You are a helpful billing assistant. Use the lookup_account "
    "tool to find customer account details.",
    tools=[lookup_account],
)


# ── Experiment Entry Point ───────────────────────────────────────────────────
async def _run_agent_async(user_message: str) -> str:
    session = agent.create_session()
    result = await agent.run(user_message, session=session)
    return getattr(result, "text", str(result)).strip()


def run_agent(input_data: Any) -> str:
    """Called by Splunk Agent Observability's run_experiment for each dataset row."""
    if isinstance(input_data, str):
        user_message = input_data
    elif isinstance(input_data, dict):
        user_message = input_data.get("input", "")
    else:
        raise TypeError(f"Unsupported input type: {type(input_data)!r}")

    return asyncio.run(_run_agent_async(user_message))

Create the experiment entry point

The experiment runner calls your function once per dataset row. Your function receives the row data and must return a string result.

agent.py

async def _run_agent_async(user_message: str) -> str:
    session = agent.create_session()
    result = await agent.run(user_message, session=session)
    return getattr(result, "text", str(result)).strip()


def run_agent(input_data: Any) -> str:
    """Called by Splunk Agent Observability's run_experiment for each dataset row."""
    if isinstance(input_data, str):
        user_message = input_data
    elif isinstance(input_data, dict):
        user_message = input_data.get("input", "")
    else:
        raise TypeError(f"Unsupported input type: {type(input_data)!r}")

    return asyncio.run(_run_agent_async(user_message))

The function should handle both str and dict inputs, since the experiment runner passes the parsed row data which can be in either format depending on your dataset structure.

Run the experiment

Use run_experiment to iterate over the dataset, call your agent function, and evaluate the results with evaluators.

main.py

import os

from dotenv import load_dotenv
from splunk_ao.experiments import run_experiment
from splunk_ao.schema.metrics import SplunkAOEvaluators

load_dotenv()

# Disable the native Splunk Agent Observability logger — OTel handles tracing
os.environ["SPLUNK_AO_LOGGING_DISABLED"] = "true"

from agent import run_agent

DATASET = [
    {
        "input": "Can you look up the account details for ACCT-1001?",
        "ground_truth": "lookup_account(account_id='ACCT-1001')",
    },
    {
        "input": "What is the current balance for ACCT-1002?",
        "ground_truth": "lookup_account(account_id='ACCT-1002')",
    },
]

results = run_experiment(
    "my-otel-experiment",
    dataset=DATASET,
    function=run_agent,
    metrics=[
        SplunkAOEvaluators.tool_error_rate,
        SplunkAOEvaluators.instruction_adherence,
        SplunkAOEvaluators.tool_selection_quality,
    ],
    project=os.environ["SPLUNK_AO_PROJECT"],
)

Set SPLUNK_AO_LOGGING_DISABLED=true before importing your agent module. The agent module initializes OTel on import, and the native logger must be disabled before that happens to avoid conflicts.

Key parameters

Parameter	Description
`experiment_name`	Name shown in the Splunk Agent Observability UI. If a duplicate name exists, a timestamp is appended automatically
`dataset`	A list of dictionaries with `input` and optional `ground_truth` fields, a Splunk Agent Observability `Dataset` object, or a dataset name
`dataset_id`	ID of an existing Splunk Agent Observability dataset. Alternative to `dataset`
`dataset_name`	Name of an existing Splunk Agent Observability dataset. Alternative to `dataset`
`function`	Your entry point function, called once per dataset row
`metrics`	List of evaluators to evaluate. Supports built-in, custom LLM-as-a-judge, and local evaluators
`project`	Project name (can also be set via the `SPLUNK_AO_PROJECT` environment variable)
`project_id`	Project ID. Alternative to `project`
`experiment_tags`	Dictionary of key-value pairs to tag the experiment for filtering and comparison

Ground truth

When your dataset includes a ground_truth field, this value is attached to the OTel spans as ground truth. Evaluators like Ground Truth Adherence use this to evaluate the agent’s response. The experiment runner uses splunk_ao_dataset_context to attach the ground truth to the OTel context, so it is available to the SplunkAOSpanProcessor without any manual configuration.

Supported frameworks

Any framework with OpenTelemetry instrumentation works with this approach. See the framework-specific guides for OTel setup details:

Microsoft Agent Framework

Built-in OTel instrumentation, no extra packages needed.

Google ADK

Integrate Google Agent Development Kit with Splunk Agent Observability via OTel.

Pydantic AI

Use Pydantic AI’s OTel support with Splunk Agent Observability.

Strands Agents

Integrate Strands Agents with Splunk Agent Observability via OTel.

Next steps

Run experiments in code

Learn about all experiment approaches including prompt templates and custom functions.

OpenTelemetry overview

Learn more about Splunk Agent Observability’s OpenTelemetry integration for logging and monitoring.

Evaluators reference

Explore the full list of available evaluators for experiments.

Datasets

Learn how to create and manage datasets for experiments.

Overview

SaaS deployments

Get Started

Observability

Evaluators

Tokenomics

Signals

AI Assistant

Luna Studio

Experiments

Agent Control

Annotations

Integrations

Security

References

How it works

Prerequisites

Set up environment variables

Set up OpenTelemetry with Splunk Agent Observability

Create the agent

Create the experiment entry point

Run the experiment

Key parameters

Ground truth

Supported frameworks

Microsoft Agent Framework

Google ADK

Pydantic AI

Strands Agents

Next steps

Run experiments in code

OpenTelemetry overview

Evaluators reference

Datasets

​How it works

​Prerequisites

​Set up environment variables

​Set up OpenTelemetry with Splunk Agent Observability

​Create the agent

​Create the experiment entry point

​Run the experiment

​Key parameters

​Ground truth

​Supported frameworks

Microsoft Agent Framework

Google ADK

Pydantic AI

Strands Agents

​Next steps

Run experiments in code

OpenTelemetry overview

Evaluators reference

Datasets

How it works

Prerequisites

Set up environment variables

Set up OpenTelemetry with Splunk Agent Observability

Create the agent

Create the experiment entry point

Run the experiment

Key parameters

Ground truth

Supported frameworks

Next steps