Run Experiments in Unit Tests

Experiments are not only useful to evaluate your app manually, you can also use them in unit tests. This allows you to test your application code against known datasets, both at development time, and in your CI/CD pipelines. A typical pattern is to write one or more unit tests against the relevant parts of your application, running an experiment using a well-defined dataset of test cases in each unit test. Once the experiment is complete, you can assert based on the average value of the evaluators used in the experiment. These unit tests typically lean more towards integration level tests, as these are run using your application code and an LLM.

Set up your unit test

A typical set up for unit testing with evals is to have:

A dataset with a set of well-defined inputs to your application. When you initially develop your application, your dataset can be defined by a product expert or data science team, augmented with synthetic data. After your application is in production, you can then export real-world data from Agent Streams into your dataset.
A set of evaluators that you want to test in the experiment. These can be out-of-the-box evaluators, or custom evaluators, either using LLM-as-a-judge, or code. For out-of-the-box or LLM-as-a-judge evaluators, you will need an LLM integration configured to test the evaluator.
An application to unit test. This should be configured to use an actual LLM for the code under test, using the same LLM as production. Other parts of your application can be mocked as required. You may need to configure the way you start, conclude, and flush sessions and traces differently in your application to support running this code using experiments. See our run experiments with custom functions guide for more details.
Although it can be tempting to use a different LLM (such as a cheaper one) during unit testing, using a different LLM to production will give unit test results that do not match the actual behavior of your production system.

Run the experiment

To run the experiment, create a unit test using the framework of your choice. You can then run an experiment using the run experiment function, passing in a named dataset, and calling your application code. When you run the experiment, you provide an experiment name. This needs to be unique, but if you set a non-unique name, the SDK will add the run date and time to the name in the created experiment to make it unique. This new name is returned in the response.

from splunk_ao import SplunkAOEvaluators
from splunk_ao.experiments import run_experiment

def test_run_experiment():
    experiment_response = run_experiment(
        experiment_name="test_run_experiment",
        dataset_name="my-unit-test-dataset",
        function=my_llm_function,
        metrics=[
            SplunkAOEvaluators.correctness,
        ]
    )

from splunk_ao import Experiment, Evaluator

def test_run_experiment():
    experiment = Experiment(
        name="test_run_experiment",
        dataset_name="my-unit-test-dataset",
        prompt_name="my-unit-test-prompt",
        metrics=[Evaluator.scorers.correctness],
        project_name="my-project",
    )
    experiment.create()
    result = experiment.run()

You can learn more about running functions in experiments in our run experiments with custom functions guide.

Check the results

When you run an experiment, it starts running in the background. You can then poll for the status of the experiment, and when it is finished and all evaluators are tested, check for the average values of each evaluator against the entire dataset. You can then assert against these values, for example failing the unit test if the average is less than a defined threshold. To poll the experiment, you retrieve it by name. If you used a non-unique name when running the experiment, then the actual name that is used with the date and time added is returned from the call to the run experiment function.

from splunk_ao.experiments import get_experiment

# Get the experiment name from the response from run_experiment
experiment_name = experiment_response["experiment"].name

# Load the experiment by name
experiment = get_experiment(experiment_name=experiment_name)

from splunk_ao import Experiment

# Load the experiment by name
experiment = Experiment.get(
    name="test_run_experiment",
    project_name="my-project"
)

The returned experiment has a get_metric_aggregate() method that returns full aggregate statistics — avg, min, max, p50, p90, p95, p99 — for a specific evaluator. Pass a SplunkAOEvaluators enum value, a human-readable label, or an evaluator name string. The method returns None when the evaluator has not yet been calculated. Check whether the evaluator is None and, if so, wait a few seconds, re-fetch the experiment and try again.

from splunk_ao.schema.metrics import SplunkAOEvaluators

# Poll until the specific evaluator is computed
while experiment.get_metric_aggregate(SplunkAOEvaluators.instruction_adherence) is None:
    # If the evaluator isn't ready yet, sleep before polling again
    time.sleep(5)

    # Reload the experiment to get the latest results
    experiment = get_experiment(experiment_name=experiment_name)

import time
from splunk_ao.schema.metrics import SplunkAOEvaluators

# Poll until the specific evaluator is computed
while experiment.get_metric_aggregate(SplunkAOEvaluators.correctness) is None:
    # Sleep before polling again
    time.sleep(5)
    # Refresh the experiment to get the latest results
    experiment.refresh()

Once the evaluators have been calculated, you can assert against the average value.

from splunk_ao.schema.metrics import SplunkAOEvaluators

# Assert that the average evaluator exceeds a certain threshold
agg = experiment.get_metric_aggregate(SplunkAOEvaluators.instruction_adherence)
assert agg.avg >= 0.95

from splunk_ao.schema.metrics import SplunkAOEvaluators

# Assert that the average evaluator exceeds a certain threshold
agg = experiment.get_metric_aggregate(SplunkAOEvaluators.correctness)
assert agg.avg >= 0.95

Sample projects

The sample projects that are created when you create a new Splunk Agent Observability organization all contain unit tests that run experiments. Check out these projects for more details.

Simple Chatbot

Learn more about the simple chatbot sample project

Multi-Agent Banking Chatbot With LangGraph

Learn more about the multi-agent banking chatbot with LangGraph sample project

Next steps

Datasets

Learn about more datasets, the data driving your experiments.

Run experiments in code

Learn how to run experiments in Splunk Agent Observability using the Splunk Agent Observability SDKs

Overview

Get Started

Observability

Evaluators

Signals

AI Assistant

Luna Studio

Experiments

Agent Control

Annotations

Integrations

Security

References

Set up your unit test

Run the experiment

Check the results

Sample projects

Simple Chatbot

Multi-Agent Banking Chatbot With LangGraph

Next steps

Datasets

Run experiments in code

​Set up your unit test

​Run the experiment

​Check the results

​Sample projects

Simple Chatbot

Multi-Agent Banking Chatbot With LangGraph

​Next steps

Datasets

Run experiments in code

Set up your unit test

Run the experiment

Check the results

Sample projects

Next steps