> ## Documentation Index
> Fetch the complete documentation index at: https://agent-observability-docs.splunk.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Run Experiments in Code

> Learn how to run experiments in Splunk Agent Observability using the Splunk Agent Observability SDKs

You can run experiments to validate your application's performance and behavior. Experiments help with prompt engineering and model selection, and can fit into an evaluation-driven development process or a CI/CD pipeline.

Data scientists and engineers can use experiments in notebooks or simple applications. Engineers can also add experiments into production apps -- allowing experiments to be run against complex scenarios (including RAG and agentic flows).

## LLM integration prerequisite

To run an experiment using a [prompt](/sdk-api/experiments/prompts) and a [dataset](/sdk-api/experiments/datasets), you need to set up an LLM integration. An integration is also required for experiments using [out-of-the-box evaluators](/concepts/evaluators/evaluator-comparison) and [custom LLM evaluators](/concepts/evaluators/custom-evaluators/custom-evaluators-ui-llm).

More information on configuring an LLM integration is available in [this getting started guide](/getting-started/experiments/#prerequisite-configure-an-llm-integration). Enterprise customers also have the option of setting up [Luna](/concepts/luna/luna) small language models (SLMs) for experiments.

## Experiment flow

A primary entry point for running experiments is the function [`run_experiment` in the Python SDK](/sdk-api/python/reference/experiments#run-experiment).

The flow diagram below illustrates what happens after calling this function.

```mermaid theme={null}
flowchart TD
    A[Dataset] --> B[Experiment Runner]
    B --> C[Prompt Template]
    B --> D[Generated Output]
    B --> E[Custom Function]
    C --> F[Evaluators Run]
    D --> F
    E --> F
    F --> G[Results in Splunk Agent Observability UI]
```

The table below describes the different approaches (e.g. function parameters) that can be involved in setting up an experiment.

<Tip>
  **Which approach should I use?**

  | Approach                                                       | When to use                                                                      | Output generation                                                |
  | -------------------------------------------------------------- | -------------------------------------------------------------------------------- | ---------------------------------------------------------------- |
  | [**Prompt template**](#run-experiments-with-prompt-template)   | You want Splunk Agent Observability to generate output using a prompt and an LLM | LLM generates output                                             |
  | [**Generated output**](#run-experiments-with-generated-output) | You already have output from your AI system to evaluate                          | No LLM generation needed — output already exists in your dataset |
  | [**Custom function**](#run-experiments-with-custom-function)   | You need to run complex application logic (RAG, agents, multi-step)              | Your application generates output                                |
</Tip>

You can choose evaluators for your experiments. Evaluators can be [out-of-the-box evaluators](/sdk-api/evaluators/evaluators) (specified as constants in the Splunk Agent Observability SDK), or the names of custom evaluators.

<Note>
  **For advanced users** -- If you're calling your application code from an experiment, the experiment runner will start a new session and trace for every row in your dataset. To avoid conflicts, you will need to ensure your application code doesn't start a new session or trace manually, or conclude or flush the trace.

  Visit the examples in [this section](/sdk-api/experiments/running-experiments#get-an-existing-logger-and-check-for-an-existing-trace) to check whether an experiment is in progress. This special handling is required for application code that uses manual logging (e.g. [`SplunkAOLogger`](/sdk-api/logging/splunk-ao-logger)), it is not necessary for application code using the [`log`](/sdk-api/logging/log-decorator/log-decorator) wrapper.
</Note>

## Run experiments with prompt template

A simple way to get started with experimentation is by evaluating prompts against datasets. This is especially valuable during the initial prompt development and refinement phase, where you want to test different prompt variations. Assuming you've previously created a dataset, you can use the following code to run an experiment:

<CodeGroup>
  ```python Python theme={null}
  from splunk_ao import Message, MessageRole
  from splunk_ao.prompts import create_prompt
  from splunk_ao.experiments import run_experiment
  from splunk_ao.datasets import get_dataset
  from splunk_ao import SplunkAOEvaluators

  from dotenv import load_dotenv
  load_dotenv()

  project = "my-project"

  # 1a. If the prompt  does not exist, create it:
  prompt = create_prompt(
      name="geography-prompt",
      template=[
          Message(role=MessageRole.system,
                  content="""
                  You are a geography expert.
                  Respond with only the continent name.
                  """),
          Message(role=MessageRole.user, content="{{input}}")
      ]
  )

  # 1b. (OPTIONAL) If the prompt  already exists, fetch it:
  # prompt = get_prompt(name="geography-prompt")

  # 2. Run the experiment and get results
  results = run_experiment(
      "geography-experiment",
      # Name of a dataset you created
      dataset=get_dataset(name="countries"),
      prompt_template=prompt,
      # Optional
      prompt_settings={
          "max_tokens": 256,
          # Make sure you have an integration set up
          # for the model alias you're using
          "model_alias": "GPT-4o",
          "temperature": 0.8
      },
      metrics=[SplunkAOEvaluators.correctness],
      project=project
  )
  ```
</CodeGroup>

## Run experiments with generated output

<Note>
  **As of Splunk Agent Observability Python SDK [v1.50.1+](https://pypi.org/project/splunk-ao/)** — Bring your own data from any system (production logs, external LLMs, or manual curation) and evaluate it directly.
</Note>

If you already have output from your AI system, you can evaluate it directly in Splunk Agent Observability without regenerating anything. Unlike the prompt-driven flow where Splunk Agent Observability calls an LLM to generate output, this flow uses the output that already exists in your dataset. You only pay for evaluator computation.

This is ideal for:

* **Evaluating production output**: Export traces from your live system, run quality evaluators to find issues
* **Comparing model providers**: Collect output from OpenAI, Anthropic, and Gemini offline, then score them all in Splunk Agent Observability
* **Regression testing**: After improving your RAG pipeline, run the same evaluators on new output to see if scores improved
* **A/B testing**: Run the same inputs through two different systems, put both outputs in datasets, compare evaluator scores

### How it works

1. Create a dataset with an `input` column and a `generated_output` column
2. Call `run_experiment` without a `prompt_template` — Splunk Agent Observability detects the `generated_output` column automatically
3. Evaluators are computed directly on the existing output — no LLM calls needed for generation

### Example

<Note>This flow is currently supported in the Python SDK (v1.50.1+).</Note>

```python Python theme={null}
from splunk_ao.datasets import create_dataset, get_dataset
from splunk_ao.experiments import run_experiment
from splunk_ao import SplunkAOEvaluators

from dotenv import load_dotenv
load_dotenv()

# Option A: Create a new dataset from local data
dataset = create_dataset(
    name="my-eval-dataset",
    content=[
        {
            "input": "What is the capital of France?",
            "generated_output": "The capital of France is Paris.",
            # Optional — enables Ground Truth Adherence
            "ground_truth": "Paris",
        },
        {
            "input": "Explain quantum computing in one sentence.",
            "generated_output": (
                "Quantum computing uses qubits to perform"
                " calculations exponentially faster than"
                " classical computers."
            ),
            # No ground_truth — other evaluators still work
        },
    ],
)

# Option B: Or use an existing Splunk Agent Observability dataset (just uncomment one line below)
# dataset = get_dataset(name="my-existing-dataset")

# Run experiment — no prompt template needed
results = run_experiment(
    "evaluate-my-output",
    dataset=dataset,
    metrics=[
        SplunkAOEvaluators.completeness,
        SplunkAOEvaluators.context_adherence,
        SplunkAOEvaluators.ground_truth_adherence,  # Uses ground_truth when available
    ],
    project="my-project",
)
```

### Using this flow

You can use this flow in two ways:

1. **Already have a Splunk Agent Observability dataset?** Pass it directly to `run_experiment(...)`.
2. **Have local Python data** (e.g., a list of dictionaries)? First upload it with `create_dataset(...)`, then pass the returned dataset to `run_experiment(...)`.

#### Dataset columns

Your dataset needs at minimum an `input` column and a `generated_output` column.

| Column             | Required | Description                                                                                                                                                                                           |
| ------------------ | -------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `input`            | Yes      | The user query or prompt input                                                                                                                                                                        |
| `generated_output` | Yes      | The output from your AI system. Must have data in at least one of the first 100 rows.                                                                                                                 |
| `ground_truth`     | No       | Expected answer — used by the [Ground Truth Adherence](/concepts/evaluators/response-quality/ground-truth-adherence) evaluator. The SDK also accepts `output` as an alias for backward compatibility. |
| `metadata`         | No       | Additional context for filtering or grouping                                                                                                                                                          |

<Note>
  **Column naming** -- The SDK accepts both `ground_truth` and `output` for the reference/expected answer column. Internally they map to the same field. In the Splunk Agent Observability UI and CSV exports, this column is displayed as "Ground Truth". We recommend using `ground_truth` in new datasets for clarity.
</Note>

#### Flow determination

When you call `run_experiment`:

* If you provide a `prompt_template`, the prompt-driven flow is always used (even if the dataset has a `generated_output` column).
* If you omit `prompt_template` and the dataset has a `generated_output` column with at least one non-empty value in the first 100 rows, the generated output flow is used.
* If you omit `prompt_template` and the dataset has no `generated_output` column, or the column exists but has no non-empty values in the sampled rows, an error is returned.

## Run experiments with custom function

Once you're comfortable with basic prompt testing, you might want to evaluate more complex parts of your app using your datasets. This approach is particularly useful when you have a generation function in your app that takes a set of inputs, which you can model with a dataset.

If your experiment runs code that uses the `log` decorator, or a third-party SDK integration, then all the spans created by these will be logged to the experiment.

This example uses the [`log` decorator](/sdk-api/logging/log-decorator/log-decorator). The workflow span created by the log decorator will be logged to the experiment.

<CodeGroup>
  ```python Python theme={null}
  from splunk_ao import log
  from splunk_ao.experiments import run_experiment
  from splunk_ao.datasets import get_dataset
  from splunk_ao import SplunkAOEvaluators

  dataset = get_dataset(name="countries")

  @log(span_type="llm", name= "My Span")
  def llm_call(input):
    # Custom function code
      return result

  results = run_experiment(
      "geography-experiment",
      dataset=dataset,
      function=llm_call,
      metrics=[SplunkAOEvaluators.correctness],
      project="my-project",
  )
  ```
</CodeGroup>

This example uses the [OpenAI SDK wrapper](/sdk-api/third-party-integrations/openai/openai). The LLM span created by the wrapper will be logged to the experiment.

<CodeGroup>
  ```python Python theme={null}
  import os
  from splunk_ao.experiments import run_experiment
  from splunk_ao.datasets import get_dataset
  from splunk_ao.openai import openai
  from splunk_ao import SplunkAOEvaluators

  client = openai.OpenAI(api_key=os.environ["OPENAI_API_KEY"])
  dataset = get_dataset(name="countries")

  def llm_call(input):
      return client.chat.completions.create(
          model="gpt-4o",
          messages=[
            {
              "role": "system",
              "content": "You are a geography expert."
            },
            {
              "role": "user",
              "content": f"""
              Which continent does the following country belong to: {input}
              """
            }
          ],
      ).choices[0].message.content

  results = run_experiment(
      "geography-experiment",
      dataset=dataset,
      function=llm_call,
      metrics=[SplunkAOEvaluators.correctness],
      project="my-project",
  )
  ```
</CodeGroup>

### Run experiments against complex code with custom functions

Custom functions can be as complex as required, including multiple steps, agents, RAG, and more. This means you can build experiments around an existing application, allowing you to run experiments against the full application you have built, using datasets to mimic user inputs.

For example, if you have a multi-agent LangGraph chatbot application, you can run an experiment against it using a dataset to define different user inputs, and log every stage in the agentic flow as part of that experiment.

To enable this, you will need to make some small changes to your application logic to handle the logging context from the experiment.

When functions in your application are run by the `run_experiment` call, a logger is created by the experiment runner, and a trace is started. This logger can be passed through the application, accessed using the `@log` decorator or by calling [`splunk_ao_context.get_logger_instance()`](/sdk-api/python/reference/decorator#get-logger-instance) in Python.

You will need to change your code to use this instead of creating a new logger and starting a new trace.

#### Get an existing logger and check for an existing trace

The Splunk Agent Observability SDK maintains a context that tracks the current logger. You can get this logger with the following code:

<CodeGroup>
  ```python Python theme={null}
  from splunk_ao import splunk_ao_context

  # Get the current logger
  splunk_ao_logger=splunk_ao_context.get_logger_instance()
  ```
</CodeGroup>

If there isn't a current logger, one will be created by this call, so this will always return a logger.

Once you have the logger, you can check for an existing trace by accessing the current parent trace from the logger. If this is not set, then there is no active trace.

<CodeGroup>
  ```python Python theme={null}
  has_existing_trace = splunk_ao_logger.current_parent() is not None
  ```
</CodeGroup>

You can use this to decide if you need to create a new trace in your application. If there is no parent trace, you can safely create a new one.

<CodeGroup>
  ```python Python theme={null}
  def process_message(input):
      # Get the Splunk Agent Observability logger instance
      splunk_ao_logger = splunk_ao_context.get_logger_instance()

      # If there is a current parent trace, we are in an experiment
      # Otherwise, we start a new trace for the chat workflow
      is_in_experiment = False
      if not splunk_ao_logger.current_parent():
          splunk_ao_logger.start_trace(
              input=input,
              name="Chat Workflow"
          )
      else:
          is_in_experiment = True

      # Your code goes here to process the input and create log spans as needed
      # You can also pass this log to other functions, or access it in those using
      # splunk_ao_context.get_logger_instance()

      # If we are not in an experiment, we conclude and flush the trace
      if not is_in_experiment:
          splunk_ao_logger.conclude("Some output")
          splunk_ao_logger.flush()
  ```
</CodeGroup>

You can then safely call your code from the experiment runner as well as in your normal application logic. When called from the experiment runner, your traces will be logged to that experiment. When called from your application code, the traces will be logged as normal.

#### Using third-party integrations with experiments

If you are using third-party integrations, there may be some configuration you need to do to make the integrations work with experiments. See the following documentation for more details:

* [CrewAI](/sdk-api/third-party-integrations/crewai/experiments)
* [LangChain and LangGraph](/sdk-api/third-party-integrations/langchain/experiments)

### Custom function logging principles

There are a few important principles to understand when logging experiments in code.

* When running an experiment, a new logger is created for you and set in the Splunk Agent Observability context. If you create a new logger manually in the application code used in your experiment, this logger will not be used in the experiment.
* To access the logger to manually add traces inside the experiment code, you can call `splunk_ao_context.get_logger_instance()` (Python) to get the current logger.
* To detect if there is an active trace, use the `current_parent()` (Python) method on the logger. This will return `None`/`undefined` if there isn't an active trace.
* Be sure to handle cases in your application code where a logger is created or a trace is started, and make sure this doesn't happen in an experiment, and the experiment logger and trace is used instead.
* Every row in a dataset is a new trace. If you create new traces manually, they will not be used.
* Do not conclude or flush the logger in your experiment, the experiment will do this for you.

## Set evaluators for your experiment

When you run an experiment, you need to define which evaluators you want to use for each row in the dataset.

For out-of-the-box evaluators, use the [constants provided by the Splunk Agent Observability SDK](/sdk-api/evaluators/evaluators).

<CodeGroup>
  ```python Python {8} theme={null}
  from splunk_ao.experiments import run_experiment
  from splunk_ao import SplunkAOEvaluators

  results = run_experiment(
      "finance-experiment",
      dataset=dataset,
      function=llm_call,
      metrics=[SplunkAOEvaluators.correctness],
      project="my-project",
  )
  ```

  ```python Python (Beta) theme={null}
  from splunk_ao import Dataset, Experiment, SplunkAOEvaluators

  dataset = Dataset.get(name="countries")

  experiment = Experiment(
      name="finance-experiment",
      dataset=dataset,
      prompt_name="geography-prompt",
      metrics=[SplunkAOEvaluators.correctness],
      project_name="my-project",
  )
  experiment.create()
  result = experiment.run()
  ```
</CodeGroup>

For custom evaluators, use the name you set when you created the evaluator. For example, if you have a custom LLM-as-a-judge evaluator called `"Compliance - do not recommend any financial actions"`:

<img src="https://mintcdn.com/agent-observability-docs/0i9p_J7eAsCmqtd7/concepts/evaluators/custom-evaluators/evaluator-name.webp?fit=max&auto=format&n=0i9p_J7eAsCmqtd7&q=85&s=6b1cf6ed71e6d1b32cf5105293a0ea20" alt="An evaluator called Compliance - do not recommend any financial actions" width="1285" height="182" data-path="concepts/evaluators/custom-evaluators/evaluator-name.webp" />

You would pass this to an experiment like this:

<CodeGroup>
  ```python Python {7} theme={null}
  from splunk_ao.experiments import run_experiment

  results = run_experiment(
      "finance-experiment",
      dataset=dataset,
      function=llm_call,
      metrics=["Compliance - do not recommend any financial actions"],
      project="my-project",
  )
  ```

  ```python Python (Beta) theme={null}
  from splunk_ao import Dataset, Experiment

  dataset = Dataset.get(name="countries")

  experiment = Experiment(
      name="finance-experiment",
      dataset=dataset,
      prompt_name="geography-prompt",
      metrics=["Compliance - do not recommend any financial actions"],
      project_name="my-project",
  )
  experiment.create()
  result = experiment.run()
  ```
</CodeGroup>

### Ground truth

For [Ground Truth Adherence](/concepts/evaluators/response-quality/ground-truth-adherence), you also need to set the ground truth in your dataset. This is set in the `ground_truth` column.

<CodeGroup>
  ```python Python {4} theme={null}
  dataset = [
    {
      "input": "Spain"
      "ground_truth": "Spain is in Europe"
    }
  ]
  ```
</CodeGroup>

<Note>If you set the `ground_truth` column when using other evaluators, the value is not used in the calculation of the evaluator, but can be added to the Splunk Agent Observability UI under the "Dataset Ground Truth" column. This can be helpful for manual review.</Note>

## Custom dataset evaluation

As your testing needs become more specific, you might need to work with custom or local datasets. This approach is perfect for focused testing of edge cases or when building up your test suite with specific scenarios:

<CodeGroup>
  ```python Python theme={null}
  import os
  from splunk_ao.experiments import run_experiment
  from splunk_ao.openai import openai
  from splunk_ao import SplunkAOEvaluators

  dataset = [
    {
      "input": "Spain"
    }
  ]

  def llm_call(input):
    client = openai.OpenAI(api_key=os.environ["OPENAI_API_KEY"])
    return client.chat.completions.create(
          model="gpt-4",
          messages=[
            {
              "role": "system",
              "content": "You are a geography expert"
            },
            {
              "role": "user",
              "content": f"""
              Which continent does the following country belong to: {input}
              """
            }
          ],
      ).choices[0].message.content

  results = run_experiment(
      "geography-experiment",
      dataset=dataset,
      function=llm_call,
      metrics=[SplunkAOEvaluators.correctness],
      project="my-project"
  )
  ```
</CodeGroup>

## Custom evaluators for deep analysis

For the most sophisticated level of testing, you might need to track specific aspects of your application's behavior. Custom evaluators provide the flexibility to define precisely what you want to measure, enabling deep analysis and targeted improvement:

<CodeGroup>
  ```python Python theme={null}
  import os
  from splunk_ao import Trace, Span
  from splunk_ao.experiments import run_experiment
  from splunk_ao.openai import openai
  from splunk_ao.schema.metrics import LocalMetricConfig

  # 1. Scorer Function
  def brevity_rank(step: Span | Trace) -> str:
      """Rank response brevity based on word count."""
      word_count = len(step.output.content.split(" "))
      if word_count <= 3:
          return "Terse"
      if word_count <= 5:
          return "Temperate"
      return "Talkative"

  # 2. Configure the Local Evaluator
  terseness = LocalMetricConfig[str](
      name="Terseness",
      scorer_fn=brevity_rank
  )

  # 3. Dataset
  countries_dataset = [
      {"input": "Indonesia"},
      {"input": "New Zealand"},
      {"input": "Greenland"},
      {"input": "China"},
  ]

  # 4. LLM-Call Function
  def llm_call(input):
      client = openai.OpenAI(api_key=os.environ["OPENAI_API_KEY"])
      return (
          client.chat.completions.create(
              model="gpt-4o",
              messages=[
                  {
                      "role": "system",
                      "content": """
                      You are a geography expert. Always answer as succinctly as possible.
                      """
                  },
                  {
                      "role": "user",
                      "content": f"""
                      Which continent does the following country belong to: {input}
                      """
                  },
              ],
          )
          .choices[0]
          .message.content
      )

  # 5. Run the Experiment!
  results = run_experiment(
      "terseness-custom-evaluator",
      dataset=countries_dataset,
      function=llm_call,
      metrics=[terseness],  # You can add multiple custom evaluators here
      project="My first project",
  )
  ```
</CodeGroup>

Each of these experimentation approaches fits into different stages of your development and testing workflow. As you progress from simple prompt testing to sophisticated custom evaluators, Splunk Agent Observability's experimentation framework provides the tools you need to gather insights and improve your application's performance at every level of complexity.

## Experiments with agentic and RAG applications

The experimentation framework extends naturally to more complex applications like agentic AI systems and RAG (Retrieval-Augmented Generation) applications. When working with agents, you can evaluate various aspects of their behavior, from decision-making capabilities to tool usage patterns. This is particularly valuable when testing how agents handle complex workflows, multi-step reasoning, or tool selection.

For RAG applications, experimentation helps validate both the retrieval and generation components of your system. You can assess the quality of retrieved context, measure response relevance, and ensure that your RAG pipeline maintains high accuracy across different types of queries. This is especially important when fine-tuning retrieval parameters or testing different reranking strategies.

The same experimentation patterns shown above apply to these more complex systems. You can use predefined datasets to benchmark performance, create custom datasets for specific edge cases, and define specialized evaluators that capture the unique aspects of agent behavior or RAG performance. This systematic approach to testing helps ensure that your advanced AI applications maintain high quality and reliability in production environments.

## Multi-turn conversations in experiments

The [Python SDK](/sdk-api/python/sdk-reference) supports evaluating multi-turn conversations in experiments. With this feature, you can use session evaluators to test how multi-turn datasets perform on your agents before deploying changes to production.

<img src="https://mintcdn.com/agent-observability-docs/Y4gaVgpsSUs8MBdT/images/console-ui/multi-turn-conversations-experiments.png?fit=max&auto=format&n=Y4gaVgpsSUs8MBdT&q=85&s=31926c99c8b6e23bb35cc2e9abf389a3" alt="Multi-turn conversations in experiments screenshot" width="2692" height="818" data-path="images/console-ui/multi-turn-conversations-experiments.png" />

Out-of-the-box session evaluators include [Action Completion](/concepts/evaluators/agentic/action-completion), [Agent Efficiency](/concepts/evaluators/agentic/agent-efficiency), [Agent Flow](/concepts/evaluators/agentic/agent-flow), [Conversation Quality](/concepts/evaluators/agentic/conversation-quality), [Interruption Detection](/concepts/evaluators/multimodal-quality/interruption-detection), and [User Intent Change](/concepts/evaluators/agentic/intent-change). You can also [create your own custom evaluator](/concepts/evaluators/custom-evaluators/custom-evaluators-ui-llm) to evaluate sessions.

Usage steps:

1. Define multi-turn datasets and experiments using the Python SDK.
2. Configure out-of-the-box or custom session evaluators on these experiments.
3. View the sessions and evaluators in the Experiments UI in a new "Sessions" tab -- or get the computed evaluator values through the SDK.

[View full multi-turn experiment example](https://github.com/splunk/splunk-ao-python/tree/main/examples/experiments/multi-turn)

<Note>
  Instead of [`run_experiment`](/sdk-api/python/reference/experiments#run_experiment), set up multi-turn experiments with  [`create_experiment`](/sdk-api/python/reference/experiments#create_experiment). This function creates an experiment object in which to provide your own sessions and traces using the Splunk Agent Observability context and logger.

  See [this code example](https://github.com/splunk/splunk-ao-python/blob/main/examples/experiments/multi-turn/basic-example.py) for more information.
</Note>

## Experiment groups

You can organize related experiments into experiment groups. In the Python SDK, provide an optional **`experiment_group`** parameter on [`run_experiment`](/sdk-api/python/reference/experiments#run_experiment), [`create_experiment`](/sdk-api/python/reference/experiments#create_experiment), or [`get_experiments`](/sdk-api/python/reference/experiments#get_experiments). Use [`list_experiment_groups`](/sdk-api/python/reference/experiments#list_experiment_groups) to show available experiment groups.

Learn more about grouping experiments in the Splunk Agent Observability UI and SDK on [this page](/sdk-api/experiments/experiment-groups).

## Best practices

1. **Use consistent datasets**: Use the same dataset when comparing different prompts or models to ensure fair comparisons.
2. **Test multiple variations**: Run experiments with different prompt variations to find the best approach.
3. **Use appropriate evaluators**: Choose evaluators that are relevant to your specific use case.
4. **Start small**: Begin with a small dataset to quickly iterate and refine your approach before scaling up.
5. **Document your experiments**: Keep track of what you're testing and why to make it easier to interpret results.

## Next steps

### Experiments SDK

<CardGroup cols={2}>
  <Card title="Use Datasets in Code" icon="database" horizontal href="/sdk-api/experiments/datasets">
    Learn about more datasets, the data driving your experiments.
  </Card>

  <Card title="Prompt Templates" icon="message" horizontal href="/sdk-api/experiments/prompts">
    Learn how to create and use prompt templates in experiments
  </Card>
</CardGroup>

### Evaluators

<CardGroup cols={2}>
  <Card title="Evaluators Reference Guide" icon="brain" horizontal href="/sdk-api/evaluators/evaluators">
    A list of supported evaluators and how to use them in experiments.
  </Card>

  <Card title="Local Evaluators" icon="code" horizontal href="/concepts/evaluators/custom-evaluators/custom-evaluators-ui-code#local-evaluators">
    Create and run custom evaluators directly in code.
  </Card>

  <Card title="Custom Code-Based Evaluators" icon="code" horizontal href="/concepts/evaluators/custom-evaluators/custom-evaluators-ui-code">
    Create reusable custom evaluators right in the Splunk Agent Observability UI.
  </Card>

  <Card title="Custom LLM-as-a-Judge Evaluators" icon="brain" horizontal href="/concepts/evaluators/custom-evaluators/custom-evaluators-ui-llm">
    Create reusable custom evaluators using LLMs to evaluate your response quality
  </Card>
</CardGroup>