> ## Documentation Index
> Fetch the complete documentation index at: https://agent-observability-docs.splunk.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Create a Local Evaluator

> Learn how to create a local evaluator in Python to use in your experiments

## Overview

This guide shows you how to create a custom [local evaluator](https://agent-observability-docs.splunk.com/concepts/evaluators/custom-evaluators/custom-evaluators-ui-code.md#local-evaluators) in Python to use in an experiment.

In this example, you will be creating an evaluator to rate the brevity (shortness) of an LLM's response based on word count. The sample code to run the experiment will use OpenAI as an LLM.

In this guide you will:

1. [Set up a project with Splunk Agent Observability](#install-dependencies)

2. [Create your local evaluator](#create-your-local-evaluator)

3. [Prepare the experiment](#prepare-the-experiment)

4. [Run the experiment](#run-the-experiment)

## Before you start

To complete this how-to, you will need:

* An [OpenAI API key](https://openai.com/api/)
* A [Splunk Agent Observability project](https://agent-observability-docs.splunk.com/concepts/projects.md)
* Your [Splunk Agent Observability API key](https://agent-observability-docs.splunk.com/references/faqs/find-keys.md#splunk-ao-api-key)

## Install dependencies

To use Splunk Agent Observability, you need to install some package dependencies, and configure environment variables.

**1. Install Required Dependencies**

Install the required dependencies for your app. Create a virtual environment using your preferred method, then install dependencies inside that environment:

```bash Python theme={null}
pip install "splunk-ao[openai]" python-dotenv
```

**2. Create a .env file, and add the following values**

> [!NOTE]
>
> This assumes you are using a free account. If you are using an on-premises, standalone, or custom deployment, then you will also need to add your custom deployment URL:
>
> ```ini .env theme={null}
> SPLUNK_AO_CONSOLE_URL=your-splunk-ao-url
> ```

## Create your local evaluator

**1. Create a file for your experiment called experiment.py.**

**2. Create a scorer function**

The Scorer Function assigns one of three ranks — `"Terse"`, `"Temperate"`, or `"Talkative"`, depending on how many words the model outputs. Add this code to your `experiment.py` file.

```python Python theme={null}
from splunk_ao import Trace, Span

def brevity_rank(step: Span | Trace) -> str:
    """
    Rank response brevity based on word count.
    """
    word_count = len(step.output.content.split(" "))
    if word_count <= 3:
        return "Terse"
    if word_count <= 5:
        return "Temperate"
    return "Talkative"
```

**3. Create the local evaluator configuration**

Here, we tell Splunk Agent Observability that our custom evaluator returns a `str`. We give it a name ("Terseness"), then assign the Scorer. Add this code to your `experiment.py` file.

```python Python theme={null}
from splunk_ao.schema.metrics import LocalMetricConfig

terseness = LocalMetricConfig[str](
    # Evaluator name (shown as a column in Splunk Agent Observability)
    name="Terseness",
    # Scorer Function defined above
    scorer_fn=brevity_rank
)
```

The evaluator has been created. Next, we can use it in an experiment.

## Prepare the experiment

For this example, we'll ask the LLM to specify the continent of four countries, encouraging it to be succinct.

**1. Create a dataset**

Create a dataset of inputs to the experiment by adding this code to your `experiment.py` file.

```python Python theme={null}
# Simple dataset with four countries:
countries_dataset = [
    {"input": "Indonesia"},
    {"input": "New Zealand"},
    {"input": "Greenland"},
    {"input": "China"},
]
```

**2. Call the LLM**

Next you need a [custom function](https://agent-observability-docs.splunk.com/sdk-api/experiments/running-experiments.md#run-experiments-with-custom-function) to be called by your experiment. Add this code to your `experiment.py` file.

```python Python theme={null}
from splunk_ao.openai import openai

# The function that calls the LLM for each input:
def llm_call(input):
    client = openai.OpenAI(api_key=os.environ["OPENAI_API_KEY"])
    return (
        client.chat.completions.create(
            model="gpt-4o",
            messages=[
                {
                    "role": "system",
                    "content": """
                    You are a geography expert.
                    Always answer as succinctly as possible.
                    """
                },
                {
                    "role": "user",
                    "content": f"""
                    Which continent does the following
                    country belong to: {input}
                    """
                },
            ],
        )
        .choices[0]
        .message.content
    )
```

**3. Add code to run the experiment**

Finally, add code to run the experiment using your dataset and custom local evaluator.

```python Python theme={null}
import os

from splunk_ao.experiments import run_experiment

# Run the Experiment!
results = run_experiment(
    "terseness-local-evaluator",
    dataset=countries_dataset,
    function=llm_call,
    metrics=[terseness],
    project=os.environ["SPLUNK_AO_PROJECT"],
)
```

## Run the experiment

Now your experiment is set up, you can run it to see the results of your local evaluator.

**1. Run the experiment code**

```bash Terminal theme={null}
python experiment.py
```

When the experiment runs, it will output a link to view the results in the terminal.

```bash Terminal wrap theme={null}
(.venv) ➜ python experiment.py
Experiment terseness-local-evaluator has completed and results are available at <your-splunk-ao-url>/project/xxx/experiments/xxx
```

**2. View the experiment**

Follow the link in your terminal to view the results of the experiment. This experiment has 4 rows - one per item in the dataset.

The new Terseness evaluator is available in both the **Traces** table, and from the evaluators pane when selecting a row.

You have successfully created a local evaluator and used it in an experiment.

## See also

* [Custom evaluators in code](https://agent-observability-docs.splunk.com/concepts/evaluators/custom-evaluators/custom-evaluators-ui-code.md)
* [Custom evaluators using an LLM](https://agent-observability-docs.splunk.com/concepts/evaluators/custom-evaluators/custom-evaluators-ui-llm.md)


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.