Run Experiments in Code - Splunk Agent Observability

You can run experiments to validate your application’s performance and behavior. Experiments help with prompt engineering and model selection, and can fit into an evaluation-driven development process or a CI/CD pipeline. Data scientists and engineers can use experiments in notebooks or simple applications. Engineers can also add experiments into production apps — allowing experiments to be run against complex scenarios (including RAG and agentic flows).

LLM integration prerequisite

To run an experiment using a prompt and a dataset, you need to set up an LLM integration. An integration is also required for experiments using out-of-the-box evaluators and custom LLM evaluators. More information on configuring an LLM integration is available in this getting started guide. Enterprise customers also have the option of setting up Luna small language models (SLMs) for experiments.

Experiment flow

A primary entry point for running experiments is the function run_experiment in the Python SDK. The flow diagram below illustrates what happens after calling this function. The table below describes the different approaches (e.g. function parameters) that can be involved in setting up an experiment.

Which approach should I use?

Approach	When to use	Output generation
Prompt template	You want Splunk Agent Observability to generate output using a prompt and an LLM	LLM generates output
Generated output	You already have output from your AI system to evaluate	No LLM generation needed — output already exists in your dataset
Custom function	You need to run complex application logic (RAG, agents, multi-step)	Your application generates output

You can choose evaluators for your experiments. Evaluators can be out-of-the-box evaluators (specified as constants in the Splunk Agent Observability SDK), or the names of custom evaluators.

For advanced users — If you’re calling your application code from an experiment, the experiment runner will start a new session and trace for every row in your dataset. To avoid conflicts, you will need to ensure your application code doesn’t start a new session or trace manually, or conclude or flush the trace.Visit the examples in this section to check whether an experiment is in progress. This special handling is required for application code that uses manual logging (e.g. SplunkAOLogger), it is not necessary for application code using the log wrapper.

Run experiments with prompt template

A simple way to get started with experimentation is by evaluating prompts against datasets. This is especially valuable during the initial prompt development and refinement phase, where you want to test different prompt variations. Assuming you’ve previously created a dataset, you can use the following code to run an experiment:

from splunk_ao import Message, MessageRole
from splunk_ao.prompts import create_prompt
from splunk_ao.experiments import run_experiment
from splunk_ao.datasets import get_dataset
from splunk_ao import SplunkAOEvaluators

from dotenv import load_dotenv
load_dotenv()

project = "my-project"

# 1a. If the prompt  does not exist, create it:
prompt = create_prompt(
    name="geography-prompt",
    template=[
        Message(role=MessageRole.system,
                content="""
                You are a geography expert.
                Respond with only the continent name.
                """),
        Message(role=MessageRole.user, content="{{input}}")
    ]
)

# 1b. (OPTIONAL) If the prompt  already exists, fetch it:
# prompt = get_prompt(name="geography-prompt")

# 2. Run the experiment and get results
results = run_experiment(
    "geography-experiment",
    # Name of a dataset you created
    dataset=get_dataset(name="countries"),
    prompt_template=prompt,
    # Optional
    prompt_settings={
        "max_tokens": 256,
        # Make sure you have an integration set up
        # for the model alias you're using
        "model_alias": "GPT-4o",
        "temperature": 0.8
    },
    metrics=[SplunkAOEvaluators.correctness],
    project=project
)

Run experiments with generated output

As of Splunk Agent Observability Python SDK v1.50.1+ — Bring your own data from any system (production logs, external LLMs, or manual curation) and evaluate it directly.

If you already have output from your AI system, you can evaluate it directly in Splunk Agent Observability without regenerating anything. Unlike the prompt-driven flow where Splunk Agent Observability calls an LLM to generate output, this flow uses the output that already exists in your dataset. You only pay for evaluator computation. This is ideal for:

Evaluating production output: Export traces from your live system, run quality evaluators to find issues
Comparing model providers: Collect output from OpenAI, Anthropic, and Gemini offline, then score them all in Splunk Agent Observability
Regression testing: After improving your RAG pipeline, run the same evaluators on new output to see if scores improved
A/B testing: Run the same inputs through two different systems, put both outputs in datasets, compare evaluator scores

How it works

Create a dataset with an input column and a generated_output column
Call run_experiment without a prompt_template — Splunk Agent Observability detects the generated_output column automatically
Evaluators are computed directly on the existing output — no LLM calls needed for generation

Example

This flow is currently supported in the Python SDK (v1.50.1+).

Python

from splunk_ao.datasets import create_dataset, get_dataset
from splunk_ao.experiments import run_experiment
from splunk_ao import SplunkAOEvaluators

from dotenv import load_dotenv
load_dotenv()

# Option A: Create a new dataset from local data
dataset = create_dataset(
    name="my-eval-dataset",
    content=[
        {
            "input": "What is the capital of France?",
            "generated_output": "The capital of France is Paris.",
            # Optional — enables Ground Truth Adherence
            "ground_truth": "Paris",
        },
        {
            "input": "Explain quantum computing in one sentence.",
            "generated_output": (
                "Quantum computing uses qubits to perform"
                " calculations exponentially faster than"
                " classical computers."
            ),
            # No ground_truth — other evaluators still work
        },
    ],
)

# Option B: Or use an existing Splunk Agent Observability dataset (just uncomment one line below)
# dataset = get_dataset(name="my-existing-dataset")

# Run experiment — no prompt template needed
results = run_experiment(
    "evaluate-my-output",
    dataset=dataset,
    metrics=[
        SplunkAOEvaluators.completeness,
        SplunkAOEvaluators.context_adherence,
        SplunkAOEvaluators.ground_truth_adherence,  # Uses ground_truth when available
    ],
    project="my-project",
)

Using this flow

You can use this flow in two ways:

Already have a Splunk Agent Observability dataset? Pass it directly to run_experiment(...).
Have local Python data (e.g., a list of dictionaries)? First upload it with create_dataset(...), then pass the returned dataset to run_experiment(...).

Dataset columns

Your dataset needs at minimum an input column and a generated_output column.

Column	Required	Description
`input`	Yes	The user query or prompt input
`generated_output`	Yes	The output from your AI system. Must have data in at least one of the first 100 rows.
`ground_truth`	No	Expected answer — used by the Ground Truth Adherence evaluator. The SDK also accepts `output` as an alias for backward compatibility.
`metadata`	No	Additional context for filtering or grouping

Column naming — The SDK accepts both ground_truth and output for the reference/expected answer column. Internally they map to the same field. In the Splunk Agent Observability UI and CSV exports, this column is displayed as “Ground Truth”. We recommend using ground_truth in new datasets for clarity.

Flow determination

When you call run_experiment:

If you provide a prompt_template, the prompt-driven flow is always used (even if the dataset has a generated_output column).
If you omit prompt_template and the dataset has a generated_output column with at least one non-empty value in the first 100 rows, the generated output flow is used.
If you omit prompt_template and the dataset has no generated_output column, or the column exists but has no non-empty values in the sampled rows, an error is returned.

Run experiments with custom function

Once you’re comfortable with basic prompt testing, you might want to evaluate more complex parts of your app using your datasets. This approach is particularly useful when you have a generation function in your app that takes a set of inputs, which you can model with a dataset. If your experiment runs code that uses the log decorator, or a third-party SDK integration, then all the spans created by these will be logged to the experiment. This example uses the log decorator. The workflow span created by the log decorator will be logged to the experiment.

from splunk_ao import log
from splunk_ao.experiments import run_experiment
from splunk_ao.datasets import get_dataset
from splunk_ao import SplunkAOEvaluators

dataset = get_dataset(name="countries")

@log(span_type="llm", name= "My Span")
def llm_call(input):
  # Custom function code
    return result

results = run_experiment(
    "geography-experiment",
    dataset=dataset,
    function=llm_call,
    metrics=[SplunkAOEvaluators.correctness],
    project="my-project",
)

This example uses the OpenAI SDK wrapper. The LLM span created by the wrapper will be logged to the experiment.

import os
from splunk_ao.experiments import run_experiment
from splunk_ao.datasets import get_dataset
from splunk_ao.openai import openai
from splunk_ao import SplunkAOEvaluators

client = openai.OpenAI(api_key=os.environ["OPENAI_API_KEY"])
dataset = get_dataset(name="countries")

def llm_call(input):
    return client.chat.completions.create(
        model="gpt-4o",
        messages=[
          {
            "role": "system",
            "content": "You are a geography expert."
          },
          {
            "role": "user",
            "content": f"""
            Which continent does the following country belong to: {input}
            """
          }
        ],
    ).choices[0].message.content

results = run_experiment(
    "geography-experiment",
    dataset=dataset,
    function=llm_call,
    metrics=[SplunkAOEvaluators.correctness],
    project="my-project",
)

Run experiments against complex code with custom functions

Custom functions can be as complex as required, including multiple steps, agents, RAG, and more. This means you can build experiments around an existing application, allowing you to run experiments against the full application you have built, using datasets to mimic user inputs. For example, if you have a multi-agent LangGraph chatbot application, you can run an experiment against it using a dataset to define different user inputs, and log every stage in the agentic flow as part of that experiment. To enable this, you will need to make some small changes to your application logic to handle the logging context from the experiment. When functions in your application are run by the run_experiment call, a logger is created by the experiment runner, and a trace is started. This logger can be passed through the application, accessed using the @log decorator or by calling splunk_ao_context.get_logger_instance() in Python. You will need to change your code to use this instead of creating a new logger and starting a new trace.

Get an existing logger and check for an existing trace

The Splunk Agent Observability SDK maintains a context that tracks the current logger. You can get this logger with the following code:

from splunk_ao import splunk_ao_context

# Get the current logger
splunk_ao_logger=splunk_ao_context.get_logger_instance()

If there isn’t a current logger, one will be created by this call, so this will always return a logger. Once you have the logger, you can check for an existing trace by accessing the current parent trace from the logger. If this is not set, then there is no active trace.

has_existing_trace = splunk_ao_logger.current_parent() is not None

You can use this to decide if you need to create a new trace in your application. If there is no parent trace, you can safely create a new one.

def process_message(input):
    # Get the Splunk Agent Observability logger instance
    splunk_ao_logger = splunk_ao_context.get_logger_instance()

    # If there is a current parent trace, we are in an experiment
    # Otherwise, we start a new trace for the chat workflow
    is_in_experiment = False
    if not splunk_ao_logger.current_parent():
        splunk_ao_logger.start_trace(
            input=input,
            name="Chat Workflow"
        )
    else:
        is_in_experiment = True

    # Your code goes here to process the input and create log spans as needed
    # You can also pass this log to other functions, or access it in those using
    # splunk_ao_context.get_logger_instance()

    # If we are not in an experiment, we conclude and flush the trace
    if not is_in_experiment:
        splunk_ao_logger.conclude("Some output")
        splunk_ao_logger.flush()

You can then safely call your code from the experiment runner as well as in your normal application logic. When called from the experiment runner, your traces will be logged to that experiment. When called from your application code, the traces will be logged as normal.

Using third-party integrations with experiments

If you are using third-party integrations, there may be some configuration you need to do to make the integrations work with experiments. See the following documentation for more details:

Custom function logging principles

There are a few important principles to understand when logging experiments in code.

When running an experiment, a new logger is created for you and set in the Splunk Agent Observability context. If you create a new logger manually in the application code used in your experiment, this logger will not be used in the experiment.
To access the logger to manually add traces inside the experiment code, you can call splunk_ao_context.get_logger_instance() (Python) to get the current logger.
To detect if there is an active trace, use the current_parent() (Python) method on the logger. This will return None/undefined if there isn’t an active trace.
Be sure to handle cases in your application code where a logger is created or a trace is started, and make sure this doesn’t happen in an experiment, and the experiment logger and trace is used instead.
Every row in a dataset is a new trace. If you create new traces manually, they will not be used.
Do not conclude or flush the logger in your experiment, the experiment will do this for you.

Set evaluators for your experiment

When you run an experiment, you need to define which evaluators you want to use for each row in the dataset. For out-of-the-box evaluators, use the constants provided by the Splunk Agent Observability SDK.

from splunk_ao.experiments import run_experiment
from splunk_ao import SplunkAOEvaluators

results = run_experiment(
    "finance-experiment",
    dataset=dataset,
    function=llm_call,
    metrics=[SplunkAOEvaluators.correctness],
    project="my-project",
)

from splunk_ao import Dataset, Experiment, SplunkAOEvaluators

dataset = Dataset.get(name="countries")

experiment = Experiment(
    name="finance-experiment",
    dataset=dataset,
    prompt_name="geography-prompt",
    metrics=[SplunkAOEvaluators.correctness],
    project_name="my-project",
)
experiment.create()
result = experiment.run()

For custom evaluators, use the name you set when you created the evaluator. For example, if you have a custom LLM-as-a-judge evaluator called "Compliance - do not recommend any financial actions":

An evaluator called Compliance - do not recommend any financial actions

You would pass this to an experiment like this:

from splunk_ao.experiments import run_experiment

results = run_experiment(
    "finance-experiment",
    dataset=dataset,
    function=llm_call,
    metrics=["Compliance - do not recommend any financial actions"],
    project="my-project",
)

from splunk_ao import Dataset, Experiment

dataset = Dataset.get(name="countries")

experiment = Experiment(
    name="finance-experiment",
    dataset=dataset,
    prompt_name="geography-prompt",
    metrics=["Compliance - do not recommend any financial actions"],
    project_name="my-project",
)
experiment.create()
result = experiment.run()

Ground truth

For Ground Truth Adherence, you also need to set the ground truth in your dataset. This is set in the ground_truth column.

dataset = [
  {
    "input": "Spain"
    "ground_truth": "Spain is in Europe"
  }
]

If you set the ground_truth column when using other evaluators, the value is not used in the calculation of the evaluator, but can be added to the Splunk Agent Observability UI under the “Dataset Ground Truth” column. This can be helpful for manual review.

Custom dataset evaluation

As your testing needs become more specific, you might need to work with custom or local datasets. This approach is perfect for focused testing of edge cases or when building up your test suite with specific scenarios:

import os
from splunk_ao.experiments import run_experiment
from splunk_ao.openai import openai
from splunk_ao import SplunkAOEvaluators

dataset = [
  {
    "input": "Spain"
  }
]

def llm_call(input):
  client = openai.OpenAI(api_key=os.environ["OPENAI_API_KEY"])
  return client.chat.completions.create(
        model="gpt-4",
        messages=[
          {
            "role": "system",
            "content": "You are a geography expert"
          },
          {
            "role": "user",
            "content": f"""
            Which continent does the following country belong to: {input}
            """
          }
        ],
    ).choices[0].message.content

results = run_experiment(
    "geography-experiment",
    dataset=dataset,
    function=llm_call,
    metrics=[SplunkAOEvaluators.correctness],
    project="my-project"
)

Custom evaluators for deep analysis

For the most sophisticated level of testing, you might need to track specific aspects of your application’s behavior. Custom evaluators provide the flexibility to define precisely what you want to measure, enabling deep analysis and targeted improvement:

import os
from splunk_ao import Trace, Span
from splunk_ao.experiments import run_experiment
from splunk_ao.openai import openai
from splunk_ao.schema.metrics import LocalMetricConfig

# 1. Scorer Function
def brevity_rank(step: Span | Trace) -> str:
    """Rank response brevity based on word count."""
    word_count = len(step.output.content.split(" "))
    if word_count <= 3:
        return "Terse"
    if word_count <= 5:
        return "Temperate"
    return "Talkative"

# 2. Configure the Local Evaluator
terseness = LocalMetricConfig[str](
    name="Terseness",
    scorer_fn=brevity_rank
)

# 3. Dataset
countries_dataset = [
    {"input": "Indonesia"},
    {"input": "New Zealand"},
    {"input": "Greenland"},
    {"input": "China"},
]

# 4. LLM-Call Function
def llm_call(input):
    client = openai.OpenAI(api_key=os.environ["OPENAI_API_KEY"])
    return (
        client.chat.completions.create(
            model="gpt-4o",
            messages=[
                {
                    "role": "system",
                    "content": """
                    You are a geography expert. Always answer as succinctly as possible.
                    """
                },
                {
                    "role": "user",
                    "content": f"""
                    Which continent does the following country belong to: {input}
                    """
                },
            ],
        )
        .choices[0]
        .message.content
    )

# 5. Run the Experiment!
results = run_experiment(
    "terseness-custom-evaluator",
    dataset=countries_dataset,
    function=llm_call,
    metrics=[terseness],  # You can add multiple custom evaluators here
    project="My first project",
)

Each of these experimentation approaches fits into different stages of your development and testing workflow. As you progress from simple prompt testing to sophisticated custom evaluators, Splunk Agent Observability’s experimentation framework provides the tools you need to gather insights and improve your application’s performance at every level of complexity.

Experiments with agentic and RAG applications

The experimentation framework extends naturally to more complex applications like agentic AI systems and RAG (Retrieval-Augmented Generation) applications. When working with agents, you can evaluate various aspects of their behavior, from decision-making capabilities to tool usage patterns. This is particularly valuable when testing how agents handle complex workflows, multi-step reasoning, or tool selection. For RAG applications, experimentation helps validate both the retrieval and generation components of your system. You can assess the quality of retrieved context, measure response relevance, and ensure that your RAG pipeline maintains high accuracy across different types of queries. This is especially important when fine-tuning retrieval parameters or testing different reranking strategies. The same experimentation patterns shown above apply to these more complex systems. You can use predefined datasets to benchmark performance, create custom datasets for specific edge cases, and define specialized evaluators that capture the unique aspects of agent behavior or RAG performance. This systematic approach to testing helps ensure that your advanced AI applications maintain high quality and reliability in production environments.

Multi-turn conversations in experiments

The Python SDK supports evaluating multi-turn conversations in experiments. With this feature, you can use session evaluators to test how multi-turn datasets perform on your agents before deploying changes to production.

Out-of-the-box session evaluators include Action Completion, Agent Efficiency, Agent Flow, Conversation Quality, Interruption Detection, and User Intent Change. You can also create your own custom evaluator to evaluate sessions. Usage steps:

Define multi-turn datasets and experiments using the Python SDK.
Configure out-of-the-box or custom session evaluators on these experiments.
View the sessions and evaluators in the Experiments UI in a new “Sessions” tab — or get the computed evaluator values through the SDK.

View full multi-turn experiment example

Instead of run_experiment, set up multi-turn experiments with create_experiment. This function creates an experiment object in which to provide your own sessions and traces using the Splunk Agent Observability context and logger.See this code example for more information.

Experiment groups

You can organize related experiments into experiment groups. In the Python SDK, provide an optional experiment_group parameter on run_experiment, create_experiment, or get_experiments. Use list_experiment_groups to show available experiment groups. Learn more about grouping experiments in the Splunk Agent Observability UI and SDK on this page.

Best practices

Use consistent datasets: Use the same dataset when comparing different prompts or models to ensure fair comparisons.
Test multiple variations: Run experiments with different prompt variations to find the best approach.
Use appropriate evaluators: Choose evaluators that are relevant to your specific use case.
Start small: Begin with a small dataset to quickly iterate and refine your approach before scaling up.
Document your experiments: Keep track of what you’re testing and why to make it easier to interpret results.

​LLM integration prerequisite

​Experiment flow

​Run experiments with prompt template

​Run experiments with generated output

​How it works

​Example

​Using this flow

​Dataset columns

​Flow determination

​Run experiments with custom function

​Run experiments against complex code with custom functions

​Get an existing logger and check for an existing trace

​Using third-party integrations with experiments

​Custom function logging principles

​Set evaluators for your experiment

​Ground truth

​Custom dataset evaluation

​Custom evaluators for deep analysis

​Experiments with agentic and RAG applications

​Multi-turn conversations in experiments

​Experiment groups

​Best practices

​Next steps

​Experiments SDK

Use Datasets in Code

Prompt Templates

​Evaluators

Evaluators Reference Guide

Local Evaluators

Custom Code-Based Evaluators

Custom LLM-as-a-Judge Evaluators

LLM integration prerequisite

Experiment flow

Run experiments with prompt template

Run experiments with generated output

How it works

Example

Using this flow

Dataset columns

Flow determination

Run experiments with custom function

Run experiments against complex code with custom functions

Get an existing logger and check for an existing trace

Using third-party integrations with experiments

Custom function logging principles

Set evaluators for your experiment

Ground truth

Custom dataset evaluation

Custom evaluators for deep analysis

Experiments with agentic and RAG applications

Multi-turn conversations in experiments

Experiment groups

Best practices

Next steps

Experiments SDK

Evaluators