> ## Documentation Index
> Fetch the complete documentation index at: https://agent-observability-docs.splunk.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Run experiments with OpenTelemetry

> Learn how to run experiments using OpenTelemetry-instrumented frameworks with Splunk Agent Observability

If your application uses a framework with built-in [OpenTelemetry](https://opentelemetry.io/) instrumentation, you can run experiments against it using Splunk Agent Observability's `SplunkAOSpanProcessor`. The span processor captures OTel traces and routes them to the experiment automatically — no need to use the Splunk Agent Observability `log` decorator or manual logging.

This is useful when you are working with frameworks like [Microsoft Agent Framework](https://agent-observability-docs.splunk.com/sdk-api/third-party-integrations/opentelemetry-and-openinference/microsoft-agent-framework.md), [Google ADK](https://agent-observability-docs.splunk.com/sdk-api/third-party-integrations/opentelemetry-and-openinference/google-adk.md), [Pydantic AI](https://agent-observability-docs.splunk.com/sdk-api/third-party-integrations/opentelemetry-and-openinference/pydantic-ai.md), or any other framework that emits OTel spans.

In this guide you will:

1. [Set up OpenTelemetry with Splunk Agent Observability](#set-up-opentelemetry-with-splunk-agent-observability)
2. [Create an agent with framework instrumentation](#create-the-agent)
3. [Expose an entry point for the experiment runner](#create-the-experiment-entry-point)
4. [Run the experiment](#run-the-experiment)

## How it works

When you run an experiment with a custom function, Splunk Agent Observability's experiment runner:

1. Creates an experiment and sets the `experiment_id` in the Splunk Agent Observability context.
2. For each row in the dataset, it calls your function and wraps the call with dataset context (input, ground truth, and metadata).
3. The `SplunkAOSpanProcessor` reads this context and attaches it to every OTel span your framework creates.
4. Spans are exported to Splunk Agent Observability's OTLP endpoint with the experiment ID, routing them to the experiment instead of a regular Agent Stream.

```mermaid theme={null}
flowchart LR
    A[Dataset row] --> B[run_experiment]
    B --> C[Your function]
    C --> D[Framework code]
    D --> E[OTel spans]
    E --> F[SplunkAOSpanProcessor]
    F --> G[Splunk Agent Observability experiment]
```

> [!NOTE]
>
> Because the experiment runner manages the trace lifecycle, you must disable Splunk Agent Observability's native logger to avoid duplicate traces. Set `SPLUNK_AO_LOGGING_DISABLED=true` before importing your agent code.

## Prerequisites

Install the Splunk Agent Observability SDK with OpenTelemetry support, plus any framework-specific packages:

```bash Terminal theme={null}
pip install "splunk-ao[otel]" opentelemetry-api opentelemetry-sdk opentelemetry-exporter-otlp
```

## Set up environment variables

Configure your Splunk Agent Observability credentials and disable the native logger:

```ini .env theme={null}
# Splunk Agent Observability
SPLUNK_AO_PROJECT=your-project-name
SPLUNK_AO_API_URL=your-api-url

# Only required for on-premises, standalone, and custom deployments
# SPLUNK_AO_API_KEY=your-splunk-ao-api-key
# SPLUNK_AO_CONSOLE_URL=your-splunk-ao-url

# Disable native Splunk Agent Observability logger — OTel handles tracing
SPLUNK_AO_LOGGING_DISABLED=true

# LLM provider
OPENAI_API_KEY=your-openai-api-key
```

| Variable | Required | Description |
| - | - | - |
| `SPLUNK_AO_API_KEY` | No | Only needed for on-premises, standalone, and custom deployments. Your Splunk Agent Observability [API key](https://agent-observability-docs.splunk.com/references/faqs/find-keys.md#splunk-ao-api-key) |
| `SPLUNK_AO_O11Y_TOKEN` | No | Only needed for SaaS deployments. Your [Splunk Observability Cloud token](https://agent-observability-docs.splunk.com/sdk-api/overview.md#initialization-and-authentication) |
| `SPLUNK_AO_REALM` | No | Only needed for SaaS deployments. Your Splunk Agent Observability [Splunk Observability Cloud realm](https://agent-observability-docs.splunk.com/sdk-api/overview.md#initialization-and-authentication) |
| `SPLUNK_AO_PROJECT` | Yes | The [project](https://agent-observability-docs.splunk.com/concepts/projects.md) to log experiments to |
| `SPLUNK_AO_CONSOLE_URL` | No | Only needed for on-premises, standalone, and custom deployments |
| `SPLUNK_AO_API_URL` | No | Used by the `SplunkAOSpanProcessor` to construct the OTLP endpoint |
| `SPLUNK_AO_LOGGING_DISABLED` | Yes | Set to `true` to disable the native logger. Required when using OTel for experiments |

## Set up OpenTelemetry with Splunk Agent Observability

Create a `TracerProvider` and attach the `SplunkAOSpanProcessor`. This processor automatically configures authentication and the OTLP endpoint using your environment variables.

```python agent.py theme={null}
from splunk_ao.otel import SplunkAOSpanProcessor, add_splunk_ao_span_processor
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider

# Set up the OTel tracer provider with the Splunk Agent Observability span processor
tracer_provider = TracerProvider()
splunk_ao_processor = SplunkAOSpanProcessor()
add_splunk_ao_span_processor(tracer_provider, splunk_ao_processor)
trace.set_tracer_provider(tracer_provider)
```

Then enable your framework's instrumentation. For example, with Microsoft Agent Framework:

```python agent.py theme={null}
from agent_framework.observability import enable_instrumentation

# Enable the framework's built-in OTel instrumentation
# Set enable_sensitive_data=True to capture LLM inputs and outputs
enable_instrumentation(enable_sensitive_data=True)
```

> [!TIP]
>
> Set `enable_sensitive_data=True` to capture LLM inputs and outputs in your traces. If set to `False`, only span metadata (timing, token counts, etc.) is sent.

## Create the agent

Set up your agent with your framework of choice. The key requirement is that the framework emits OTel spans through the registered `TracerProvider`.

```python agent.py theme={null}
import asyncio
import os
from typing import Any

from agent_framework import openai, tool
from agent_framework.observability import enable_instrumentation
from splunk_ao.otel import SplunkAOSpanProcessor, add_splunk_ao_span_processor
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider

# ── OTel Setup ───────────────────────────────────────────────────────────────
tracer_provider = TracerProvider()
splunk_ao_processor = SplunkAOSpanProcessor()
add_splunk_ao_span_processor(tracer_provider, splunk_ao_processor)
trace.set_tracer_provider(tracer_provider)

enable_instrumentation(enable_sensitive_data=True)

# ── Tools ────────────────────────────────────────────────────────────────────
@tool(approval_mode="never_require")
def lookup_account(account_id: str) -> str:
    """Look up account details by account ID."""
    accounts = {
        "ACCT-1001": {"name": "Alice", "plan": "Premium", "balance": 120.00},
        "ACCT-1002": {"name": "Bob", "plan": "Basic", "balance": 45.50},
    }
    account = accounts.get(account_id)
    if account:
        return f"Account {account_id}: {account}"
    return f"Account {account_id} not found."

# ── Agent ────────────────────────────────────────────────────────────────────
client = openai.OpenAIChatClient(model_id="gpt-4.1-mini")

agent = client.as_agent(
    name="BillingAgent",
    instructions="You are a helpful billing assistant. Use the lookup_account "
    "tool to find customer account details.",
    tools=[lookup_account],
)


# ── Experiment Entry Point ───────────────────────────────────────────────────
async def _run_agent_async(user_message: str) -> str:
    session = agent.create_session()
    result = await agent.run(user_message, session=session)
    return getattr(result, "text", str(result)).strip()


def run_agent(input_data: Any) -> str:
    """Called by Splunk Agent Observability's run_experiment for each dataset row."""
    if isinstance(input_data, str):
        user_message = input_data
    elif isinstance(input_data, dict):
        user_message = input_data.get("input", "")
    else:
        raise TypeError(f"Unsupported input type: {type(input_data)!r}")

    return asyncio.run(_run_agent_async(user_message))
```

## Create the experiment entry point

The experiment runner calls your function once per dataset row. Your function receives the row data and must return a string result.

```python agent.py theme={null}
async def _run_agent_async(user_message: str) -> str:
    session = agent.create_session()
    result = await agent.run(user_message, session=session)
    return getattr(result, "text", str(result)).strip()


def run_agent(input_data: Any) -> str:
    """Called by Splunk Agent Observability's run_experiment for each dataset row."""
    if isinstance(input_data, str):
        user_message = input_data
    elif isinstance(input_data, dict):
        user_message = input_data.get("input", "")
    else:
        raise TypeError(f"Unsupported input type: {type(input_data)!r}")

    return asyncio.run(_run_agent_async(user_message))
```

The function should handle both `str` and `dict` inputs, since the experiment runner passes the parsed row data which can be in either format depending on your dataset structure.

## Run the experiment

Use `run_experiment` to iterate over the dataset, call your agent function, and evaluate the results with evaluators.

```python main.py theme={null}
import os

from dotenv import load_dotenv
from splunk_ao.experiments import run_experiment
from splunk_ao.schema.metrics import SplunkAOEvaluators

load_dotenv()

# Disable the native Splunk Agent Observability logger — OTel handles tracing
os.environ["SPLUNK_AO_LOGGING_DISABLED"] = "true"

from agent import run_agent

DATASET = [
    {
        "input": "Can you look up the account details for ACCT-1001?",
        "ground_truth": "lookup_account(account_id='ACCT-1001')",
    },
    {
        "input": "What is the current balance for ACCT-1002?",
        "ground_truth": "lookup_account(account_id='ACCT-1002')",
    },
]

results = run_experiment(
    "my-otel-experiment",
    dataset=DATASET,
    function=run_agent,
    metrics=[
        SplunkAOEvaluators.tool_error_rate,
        SplunkAOEvaluators.instruction_adherence,
        SplunkAOEvaluators.tool_selection_quality,
    ],
    project=os.environ["SPLUNK_AO_PROJECT"],
)
```

> [!WARNING]
>
> Set `SPLUNK_AO_LOGGING_DISABLED=true` **before** importing your agent module. The agent module initializes OTel on import, and the native logger must be disabled before that happens to avoid conflicts.

### Key parameters

| Parameter | Description |
| - | - |
| `experiment_name` | Name shown in the Splunk Agent Observability UI. If a duplicate name exists, a timestamp is appended automatically |
| `dataset` | A list of dictionaries with `input` and optional `ground_truth` fields, a Splunk Agent Observability `Dataset` object, or a dataset name |
| `dataset_id` | ID of an existing Splunk Agent Observability dataset. Alternative to `dataset` |
| `dataset_name` | Name of an existing Splunk Agent Observability dataset. Alternative to `dataset` |
| `function` | Your entry point function, called once per dataset row |
| `metrics` | List of [evaluators](https://agent-observability-docs.splunk.com/sdk-api/evaluators/evaluators.md) to evaluate. Supports built-in, [custom LLM-as-a-judge](https://agent-observability-docs.splunk.com/concepts/evaluators/custom-evaluators/custom-evaluators-ui-llm.md), and [local evaluators](https://agent-observability-docs.splunk.com/concepts/evaluators/custom-evaluators/custom-evaluators-ui-code.md#local-evaluators) |
| `project` | Project name (can also be set via the `SPLUNK_AO_PROJECT` environment variable) |
| `project_id` | Project ID. Alternative to `project` |
| `experiment_tags` | Dictionary of key-value pairs to tag the experiment for filtering and comparison |

### Ground truth

When your dataset includes a `ground_truth` field, this value is attached to the OTel spans as ground truth. Evaluators like [Ground Truth Adherence](https://agent-observability-docs.splunk.com/concepts/evaluators/response-quality/ground-truth-adherence.md) use this to evaluate the agent's response.

The experiment runner uses `splunk_ao_dataset_context` to attach the ground truth to the OTel context, so it is available to the `SplunkAOSpanProcessor` without any manual configuration.

## Supported frameworks

Any framework with OpenTelemetry instrumentation works with this approach. See the framework-specific guides for OTel setup details:

> **[Microsoft Agent Framework](https://agent-observability-docs.splunk.com/sdk-api/third-party-integrations/opentelemetry-and-openinference/microsoft-agent-framework.md)**
>
> Built-in OTel instrumentation, no extra packages needed.

> **[Google ADK](https://agent-observability-docs.splunk.com/sdk-api/third-party-integrations/opentelemetry-and-openinference/google-adk.md)**
>
> Integrate Google Agent Development Kit with Splunk Agent Observability via OTel.

> **[Pydantic AI](https://agent-observability-docs.splunk.com/sdk-api/third-party-integrations/opentelemetry-and-openinference/pydantic-ai.md)**
>
> Use Pydantic AI's OTel support with Splunk Agent Observability.

> **[Strands Agents](https://agent-observability-docs.splunk.com/sdk-api/third-party-integrations/opentelemetry-and-openinference/strands-agents.md)**
>
> Integrate Strands Agents with Splunk Agent Observability via OTel.

## Next steps

> **[Run experiments in code](https://agent-observability-docs.splunk.com/sdk-api/experiments/running-experiments.md)**
>
> Learn about all experiment approaches including prompt templates and custom functions.

> **[OpenTelemetry overview](https://agent-observability-docs.splunk.com/sdk-api/third-party-integrations/opentelemetry-and-openinference.md)**
>
> Learn more about Splunk Agent Observability's OpenTelemetry integration for logging and monitoring.

> **[Evaluators reference](https://agent-observability-docs.splunk.com/sdk-api/evaluators/evaluators.md)**
>
> Explore the full list of available evaluators for experiments.

> **[Datasets](https://agent-observability-docs.splunk.com/sdk-api/experiments/datasets.md)**
>
> Learn how to create and manage datasets for experiments.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.