Create a Local Evaluator

Overview

This guide shows you how to create a custom local evaluator in Python to use in an experiment. In this example, you will be creating an evaluator to rate the brevity (shortness) of an LLM’s response based on word count. The sample code to run the experiment will use OpenAI as an LLM. In this guide you will:

Set up a project with Splunk Agent Observability
Create your local evaluator
Prepare the experiment
Run the experiment

Before you start

To complete this how-to, you will need:

Install dependencies

To use Splunk Agent Observability, you need to install some package dependencies, and configure environment variables.

Install Required Dependencies

Install the required dependencies for your app. Create a virtual environment using your preferred method, then install dependencies inside that environment:

pip install "splunk-ao[openai]" python-dotenv

Create a .env file, and add the following values

This assumes you are using a free account. If you are using an on-premises, standalone, or custom deployment, then you will also need to add your custom deployment URL:

.env

SPLUNK_AO_CONSOLE_URL=your-splunk-ao-url

Create your local evaluator

Create a file for your experiment called experiment.py.

Create a scorer function

The Scorer Function assigns one of three ranks — "Terse", "Temperate", or "Talkative", depending on how many words the model outputs. Add this code to your experiment.py file.

Python

from splunk_ao import Trace, Span

def brevity_rank(step: Span | Trace) -> str:
    """
    Rank response brevity based on word count.
    """
    word_count = len(step.output.content.split(" "))
    if word_count <= 3:
        return "Terse"
    if word_count <= 5:
        return "Temperate"
    return "Talkative"

Create the local evaluator configuration

Here, we tell Splunk Agent Observability that our custom evaluator returns a str. We give it a name (“Terseness”), then assign the Scorer. Add this code to your experiment.py file.

Python

from splunk_ao.schema.metrics import LocalMetricConfig

terseness = LocalMetricConfig[str](
    # Evaluator name (shown as a column in Splunk Agent Observability)
    name="Terseness",
    # Scorer Function defined above
    scorer_fn=brevity_rank
)

The evaluator has been created. Next, we can use it in an experiment.

Prepare the experiment

For this example, we’ll ask the LLM to specify the continent of four countries, encouraging it to be succinct.

Create a dataset

Create a dataset of inputs to the experiment by adding this code to your experiment.py file.

Python

# Simple dataset with four countries:
countries_dataset = [
    {"input": "Indonesia"},
    {"input": "New Zealand"},
    {"input": "Greenland"},
    {"input": "China"},
]

Call the LLM

Next you need a custom function to be called by your experiment. Add this code to your experiment.py file.

Python

from splunk_ao.openai import openai

# The function that calls the LLM for each input:
def llm_call(input):
    client = openai.OpenAI(api_key=os.environ["OPENAI_API_KEY"])
    return (
        client.chat.completions.create(
            model="gpt-4o",
            messages=[
                {
                    "role": "system",
                    "content": """
                    You are a geography expert.
                    Always answer as succinctly as possible.
                    """
                },
                {
                    "role": "user",
                    "content": f"""
                    Which continent does the following
                    country belong to: {input}
                    """
                },
            ],
        )
        .choices[0]
        .message.content
    )

Add code to run the experiment

Finally, add code to run the experiment using your dataset and custom local evaluator.

Python

import os

from splunk_ao.experiments import run_experiment

# Run the Experiment!
results = run_experiment(
    "terseness-local-evaluator",
    dataset=countries_dataset,
    function=llm_call,
    metrics=[terseness],
    project=os.environ["SPLUNK_AO_PROJECT"],
)

Run the experiment

Now your experiment is set up, you can run it to see the results of your local evaluator.

Run the experiment code

Terminal

python experiment.py

When the experiment runs, it will output a link to view the results in the terminal.

Terminal

(.venv) ➜ python experiment.py
Experiment terseness-local-evaluator has completed and results are available at <your-splunk-ao-url>/project/xxx/experiments/xxx

View the experiment

Follow the link in your terminal to view the results of the experiment. This experiment has 4 rows - one per item in the dataset.The new Terseness evaluator is available in both the Traces table, and from the evaluators pane when selecting a row.

You have successfully created a local evaluator and used it in an experiment.

Overview

SaaS deployments

Get Started

Observability

Evaluators

Tokenomics

Signals

AI Assistant

Luna Studio

Experiments

Agent Control

Annotations

Integrations

Security

References

Overview

Before you start

Install dependencies

Create your local evaluator

Prepare the experiment

Run the experiment

See also

​Overview

​Before you start

​Install dependencies

​Create your local evaluator

​Prepare the experiment

​Run the experiment

​See also

Overview

Before you start

Install dependencies

Create your local evaluator

Prepare the experiment

Run the experiment

See also