> ## Documentation Index
> Fetch the complete documentation index at: https://agent-observability-docs.splunk.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Experiments Basics

> Learn how to use datasets and experiments in code or the Splunk Agent Observability UI to evaluate and improve your application

**Experiments** in Splunk Agent Observability allow you to evaluate and compare different prompts, models, and configurations using [datasets](/sdk-api/experiments/datasets.md) and [prompt templates](/sdk-api/experiments/prompts.md), and measure their performance using various [evaluators](/concepts/evaluators/overview.md). This helps you identify the best approach for your specific use case.

You can use experiments for all the stages in your AI application development lifecycle, and experiments are a key part of evaluation-driven development:

* **Prompt engineering**: Use experiments to iterate and test different prompts against well defined inputs, or your application code.
* **Model selection**: Experiments can be run against different models to provide a comprehensive way to compare each model against your specific data, use case, or application.
* **Application testing**: Experiments are a way to repeatable test your application with known data, for example in [unit, integration, or end-to-end testing in a CI/CD pipeline](/sdk-api/experiments/running-experiments-in-unit-tests.md).

For a list of supported evaluators, see the [Evaluators Reference Guide](/sdk-api/evaluators/evaluators.md).

## Components of an experiment

An experiment is made of three components:

* One or more [evaluators](/concepts/evaluators/overview.md) that you want to evaluate.
* A [dataset](/sdk-api/experiments/datasets.md) containing well defined inputs, and optional outputs for evaluations that require ground truth.
* A means of running the dataset against a model and evaluating the response. This can be via sending prompts with placeholders for the data from the dataset, or by running custom application code.

You can manage datasets and run experiments either through the [playground in the Splunk Agent Observability UI](/concepts/experiments/running-experiments-in-console.md), or in [code](/sdk-api/experiments/running-experiments.md). Datasets can either be managed through the Splunk Agent Observability UI, created either in the UI, or uploaded via code, or managed purely in code.

### Evaluators

When you run an experiment, you define what evaluators you want to use. These can be:

* [Out-of-the-box Splunk Agent Observability evaluators](/concepts/evaluators/overview.md)
* [Luna evaluators](/concepts/luna/luna.md)
* [Custom LLM-as-a-judge evaluators](/concepts/evaluators/custom-evaluators/custom-evaluators-ui-llm.md)
* [Custom code-based evaluators](/concepts/evaluators/custom-evaluators/custom-evaluators-ui-code.md)

You can learn more about evaluators in our [evaluators documentation](/concepts/evaluators/overview.md).

### Datasets

Datasets are well-defined sets of data with inputs, and optional ground truth values. The inputs can be full prompts, data that is injected into a defined prompt, or inputs to an AI application such as user inputs to a chatbot. Ground truth is required for some evaluators to help evaluate the response against this truth.

Datasets can be created in the Splunk Agent Observability UI by [uploading a file](/sdk-api/experiments/datasets.md#dataset-file-uploads), using [synthetic data generation](/sdk-api/experiments/datasets.md#synthetic-data-generation), or by [manually creating rows](/sdk-api/experiments/datasets.md#manual-dataset-creation). You can also create [datasets in code](/sdk-api/experiments/datasets.md#create-datasets), or [define datasets inline](/sdk-api/experiments/running-experiments.md#custom-dataset-evaluation) that are not saved to Splunk Agent Observability.

Datasets saved to Splunk Agent Observability are [versioned](http://localhost:3000/sdk-api/experiments/datasets#view-version-history), with the full history maintained when you update a dataset.

You can learn more about datasets in our [dataset documentation](/sdk-api/experiments/datasets.md).

### Running the experiments

Experiments can be run in multiple different ways, depending on your needs:

* [Using a playground](/concepts/experiments/running-experiments-in-console.md)
* [Running an experiment in code](/sdk-api/experiments/running-experiments.md) using a [prompt template](/sdk-api/experiments/prompts.md) against an LLM
* [Running an experiment in code](/sdk-api/experiments/running-experiments.md) against a custom function in code
* [Running an experiment in code](/sdk-api/experiments/running-experiments.md) against your existing application code

## Output of an experiment

When you run an experiment, the output contains:

* The input. When using prompts the input is the full prompt with dataset values injected as variables
* The response from the LLM or your code
* System evaluators, such as latency, the number of tokens used, and the estimated cost
* The calculated evaluators

If you are using a playground, the output will be in the playground, and you can optionally save this to an experiment Agent Stream. If you are using code to run your experiment, the output will be in an experiment Agent Stream. These Agent Streams contain one trace per dataset row.

These Agent Streams are visible in the **Experiments** tab. Each experiment is a single line, with average values for the system and evaluators. You can then drill into each experiment to see all the traces and spans.

## Logging experiments

Experiments belong to projects, with one project containing many experiments. Each experiment has a single Agent Stream with multiple traces. When you use a dataset with an experiment, each row in the dataset is logged as a separate trace in the experiment's Agent Stream.

When you are starting to plan your experiments, ensure you have [created the relevant project](/concepts/projects.md#creating-a-project) to run them in.

## Typical experimentation flow

When you start working with your application, you'll naturally progress from basic testing to more comprehensive evaluation. This journey helps you build confidence in your application's performance and systematically improve its behavior. As you advance, you'll find that organizing your test cases into **datasets** becomes essential for effective experimentation - allowing you to track performance, identify patterns, and measure improvements over time.

**1. Initial Testing**

Run your application with simple test cases to get a feel for how it performs. This is like taking your first steps:

* Test with straightforward, expected inputs
* Watch how your application behaves in ideal conditions
* Look for any immediate issues or unexpected behaviors
* Get comfortable with your evaluators and what they tell you

This phase helps you establish a baseline for what "good" looks like.

**2. Expanding Test Coverage**

Once you're comfortable with basic testing, it's time to broaden your horizons. This is where Splunk Agent Observability's dataset features become valuable:

* Introduce more complex and varied inputs
* Use Splunk Agent Observability's [datasets](/sdk-api/experiments/datasets.md) to organize and maintain test cases
* Or bring your own dataset if you already have test data
* Run [experiments](/concepts/experiments/running-experiments-in-console.md) to look for patterns in how your application handles different types of inputs

Think of this as stress-testing your application across a wide range of scenarios.

**3. Finding and Fixing Issues**

As you test more extensively, you'll discover areas where your application needs improvement:

* Identify specific inputs that cause problems
* Add these inputs to a [datasets](/sdk-api/experiments/datasets.md)
* Look for patterns in problematic cases
* Track how your fixes perform against these problem cases
* Build a library of test cases for regression testing

This systematic approach helps you not only fix issues but also prevent them from recurring.

**4. Continuous Improvement**

Now you're in a cycle of continuous improvement:

* Regularly run tests against your datasets
* Monitor for new issues or patterns
* Quickly identify when changes cause problems
* Maintain datasets that represent your key test cases
* Track your app's improving performance over time

This ongoing process helps ensure your application keeps getting better while maintaining quality.

## Initial setup

To log experiments to Splunk Agent Observability, you need to configure the SDK to connect to Splunk Agent Observability using an API key and optionally a URL for an on-premises, standalone, or custom deployment, as well as setting the project name to log the experiments to.

### Required environment variables

To get started running experiments with Splunk Agent Observability, you need to configure the required environment variables.

Environment variables vary depending on your deployment type. For details, see the [on-premises](/sdk-api/overview.md#initialization-and-authentication-for-on-premises-deployments) and [SaaS](/sdk-api/overview.md#initialization-and-authentication-for-saas-deployments) environment variable references.

In development, you can use a `.env` file for these. For a production deployment, make sure you configure these correctly for your deployment platform.

> [!NOTE]
>
> If you are using the free version, there is no need to set the `SPLUNK_AO_CONSOLE_URL` environment variable.

### Project

The project can be configured as an environment variable, or directly in code.

| Environment variable | Description |
| :- | :- |
| `SPLUNK_AO_PROJECT` | The [Splunk Agent Observability project](/concepts/projects.md) to log to. If this is not set, you will need to pass the project name in code. |

You can also set the project when running the experiment by passing it in to the run experiments call.

```python Python theme={null}
results = run_experiment(
    "my-experiment",
    dataset=dataset,
    prompt_template=prompt,
    metrics=[SplunkAOEvaluators.correctness],
    # Set the project name here
    project="my-project"
)
```

```python Python (Beta) theme={null}
from splunk_ao import Experiment, Evaluator

experiment = Experiment(
    name="my-experiment",
    dataset_name="countries",
    prompt_name="geography-prompt",
    metrics=[Evaluator.scorers.correctness],
    # Set the project name here
    project_name="my-project"
)
experiment.create()
result = experiment.run()
```

## Next steps

### Experiments SDK

> **[Run experiments in code](/sdk-api/experiments/running-experiments.md)**
>
> Learn how to run experiments in Splunk Agent Observability using the Splunk Agent Observability SDKs

> **[Run experiments in playgrounds](/concepts/experiments/running-experiments-in-console.md)**
>
> Learn about running experiments in the Splunk Agent Observability UI using playgrounds and datasets.

> **[Run experiments in unit tests](/sdk-api/experiments/running-experiments-in-unit-tests.md)**
>
> Learn how to run experiments in unit tests that you can use during development, or in your CI/CD pipelines.

> **[Compare experiments](/concepts/experiments/compare.md)**
>
> Learn how to compare experiments in Splunk Agent Observability.

> **[Datasets](/sdk-api/experiments/datasets.md)**
>
> Learn about more datasets, the data driving your experiments.

> **[Prompts](/sdk-api/experiments/prompts.md)**
>
> Learn how to create and use prompts in experiments

### Out-of-the-box and custom evaluators

> **[Evaluators reference guide](/sdk-api/evaluators/evaluators.md)**
>
> A list of supported evaluators and how to use them in experiments.

> **[Local evaluators](/concepts/evaluators/custom-evaluators/custom-evaluators-ui-code.md#local-evaluators)**
>
> Create and run custom evaluators directly in code.

> **[Custom code-based evaluators](/concepts/evaluators/custom-evaluators/custom-evaluators-ui-code.md)**
>
> Create reusable custom evaluators right in the Splunk Agent Observability UI.

> **[Custom LLM-as-a-judge evaluators](/concepts/evaluators/custom-evaluators/custom-evaluators-ui-llm.md)**
>
> Create reusable custom evaluators using LLMs to evaluate your response quality


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.