Experiment Evaluators

Splunk Agent Observability AI provides a set of built-in, preset evaluators designed to evaluate various aspects of LLM, agent, and retrieval-based workflows. You can also create custom evaluators using LLM-as-a-judge, or code. This guide provides a reference for using these evaluators in your experiments.

Out-of-the-box evaluators reference

The table below summarizes gives the constants used in code to access each evaluator. To use these evaluators, import the relevant enum.

from splunk_ao import SplunkAOEvaluators

import { SplunkAOMetrics } from "splunk-ao";

LLM-as-a-judge Evaluators

Python
TypeScript

Evaluator	Enum Value
Action Advancement	`SplunkAOEvaluators.action_advancement`
Action Completion	`SplunkAOEvaluators.action_completion`
Agent Efficiency	`SplunkAOEvaluators.agent_efficiency`
Agent Flow	`SplunkAOEvaluators.agent_flow`
Completeness	`SplunkAOEvaluators.completeness`
Context Adherence	`SplunkAOEvaluators.context_adherence`
Context Precision	`SplunkAOEvaluators.context_precision`
Context Relevance (Query Adherence)	`SplunkAOEvaluators.context_relevance`
Conversation Quality	`SplunkAOEvaluators.conversation_quality`
Correctness (factuality)	`SplunkAOEvaluators.correctness`
Ground Truth Adherence	`SplunkAOEvaluators.ground_truth_adherence`
Instruction Adherence	`SplunkAOEvaluators.instruction_adherence`
Interruption Detection	`SplunkAOEvaluators.interruption_detection`
PII (personally identifiable information)	`SplunkAOEvaluators.input_pii`, `SplunkAOEvaluators.output_pii`
Prompt Injection	`SplunkAOEvaluators.prompt_injection`
Sexism / Bias	`SplunkAOEvaluators.input_sexism`, `SplunkAOEvaluators.output_sexism`
Tone	`SplunkAOEvaluators.input_tone`, `SplunkAOEvaluators.output_tone`
Tool Errors	`SplunkAOEvaluators.tool_error_rate`
Tool Selection Quality	`SplunkAOEvaluators.tool_selection_quality`
Reasoning Coherence	`SplunkAOEvaluators.reasoning_coherence`
SQL Correctness	`SplunkAOEvaluators.sql_correctness`
SQL Adherence	`SplunkAOEvaluators.sql_adherence`
SQL Injection	`SplunkAOEvaluators.sql_injection`
SQL Efficiency	`SplunkAOEvaluators.sql_efficiency`
Toxicity	`SplunkAOEvaluators.input_toxicity`, `SplunkAOEvaluators.output_toxicity`
User Intent Change	`SplunkAOEvaluators.user_intent_change`
Visual Fidelity	`SplunkAOEvaluators.visual_fidelity`
Visual Quality	`SplunkAOEvaluators.visual_quality`

Evaluator	Enum Value
Action Advancement	`SplunkAOEvaluators.actionAdvancement`
Action Completion	`SplunkAOEvaluators.actionCompletion`
Agent Efficiency	`SplunkAOEvaluators.agentEfficiency`
Agent Flow	`SplunkAOEvaluators.agentFlow`
Completeness	`SplunkAOEvaluators.completeness`
Context Adherence	`SplunkAOEvaluators.contextAdherence`
Context Precision	`SplunkAOEvaluators.contextPrecision`
Context Relevance (Query Adherence)	`SplunkAOEvaluators.contextRelevance`
Conversation Quality	`SplunkAOEvaluators.conversationQuality`
Correctness (factuality)	`SplunkAOEvaluators.correctness`
Ground Truth Adherence	`SplunkAOEvaluators.groundTruthAdherence`
Instruction Adherence	`SplunkAOEvaluators.instructionAdherence`
PII (personally identifiable information)	`SplunkAOEvaluators.inputPii`, `SplunkAOEvaluators.outputPii`
Prompt Injection	`SplunkAOEvaluators.promptInjection`
Sexism / Bias	`SplunkAOEvaluators.inputSexism`, `SplunkAOEvaluators.outputSexism`
Tone	`SplunkAOEvaluators.inputTone`, `SplunkAOEvaluators.outputTone`
Tool Errors	`SplunkAOEvaluators.toolErrorRate`
Tool Selection Quality	`SplunkAOEvaluators.toolSelectionQuality`
SQL Correctness	`SplunkAOEvaluators.sqlCorrectness`
SQL Adherence	`SplunkAOEvaluators.sqlAdherence`
SQL Injection	`SplunkAOEvaluators.sqlInjection`
SQL Efficiency	`SplunkAOEvaluators.sqlEfficiency`
Toxicity	`SplunkAOEvaluators.inputToxicity`, `SplunkAOEvaluators.outputToxicity`
User Intent Change	`SplunkAOEvaluators.userIntentChange`

Luna-3 evaluators

If you are using the Splunk Agent Observability Luna-3 model, then use these evaluator values.

Python
TypeScript

Evaluator	Enum Value
Action Advancement	`SplunkAOEvaluators.action_advancement_luna`
Action Completion	`SplunkAOEvaluators.action_completion_luna`
Completeness	`SplunkAOEvaluators.completeness_luna`
Context Adherence	`SplunkAOEvaluators.context_adherence_luna`
PII (personally identifiable information)	`SplunkAOEvaluators.input_pii`, `SplunkAOEvaluators.output_pii`
Prompt Injection	`SplunkAOEvaluators.prompt_injection_luna`
Sexism / Bias	`SplunkAOEvaluators.input_sexism_luna`, `SplunkAOEvaluators.output_sexism_luna`
Tone	`SplunkAOEvaluators.input_tone`, `SplunkAOEvaluators.output_tone`
Tool Errors	`SplunkAOEvaluators.tool_error_rate_luna`
Tool Selection Quality	`SplunkAOEvaluators.tool_selection_quality_luna`
Toxicity	`SplunkAOEvaluators.input_toxicity_luna`, `SplunkAOEvaluators.output_toxicity_luna`

Evaluator	Enum Value
Action Advancement	`SplunkAOEvaluators.actionAdvancementLuna`
Action Completion	`SplunkAOEvaluators.actionCompletionLuna`
Completeness	`SplunkAOEvaluators.completenessLuna`
Context Adherence	`SplunkAOEvaluators.contextAdherenceLuna`
PII (personally identifiable information)	`SplunkAOEvaluators.inputPii`, `SplunkAOEvaluators.outputPii`
Prompt Injection	`SplunkAOEvaluators.promptInjectionLuna`
Sexism / Bias	`SplunkAOEvaluators.inputSexismLuna`, `SplunkAOEvaluators.outputSexismLuna`
Tone	`SplunkAOEvaluators.inputTone`, `SplunkAOEvaluators.outputTone`
Tool Errors	`SplunkAOEvaluators.toolErrorRateLuna`
Tool Selection Quality	`SplunkAOEvaluators.toolSelectionQualityLuna`
Toxicity	`SplunkAOEvaluators.inputToxicityLuna`, `SplunkAOEvaluators.outputToxicityLuna`

How do I use evaluators in experiments?

The run experiment function in Python takes a list of evaluators as part of its arguments.

Preset evaluators

Supply a list of one or more evaluator names into the run_experiment function as shown below:

import os
from splunk_ao.experiments import run_experiment
from splunk_ao.datasets import get_dataset
from splunk_ao.openai import openai
from splunk_ao import SplunkAOEvaluators

dataset = get_dataset(name="fictional_character_names")

# Define a custom "runner" function for your experiment.
def my_custom_llm_runner(input):
  client = openai.OpenAI(api_key=os.environ["OPENAI_API_KEY"])
  return client.chat.completions.create(
        model="gpt-4o",
        messages=[
          { "role": "system", "content": "You are a great storyteller." },
          { "role": "user", "content": f"Write a story about {input["topic"]}" },
        ],
    ).choices[0].message.content

# Run the experiment!
results = run_experiment(
    "test-experiment",
    project="my-test-project-1",
    dataset=dataset,
    function=my_custom_llm_runner,
    metrics=[
        # List evaluators here
        SplunkAOEvaluators.action_advancement,
        SplunkAOEvaluators.completeness,
        SplunkAOEvaluators.instruction_adherence
    ],
)

For more information, read about running experiments with the Splunk Agent Observability SDKs.

Custom evaluators

You can use custom evaluators in the same way as Splunk Agent Observability’s preset evaluators. At a high level, this involves the following steps:

Create your evaluator in the Splunk Agent Observability UI (or in code). Your custom evaluator will return a numerical score based on its input.
Pass the name of your new evaluator into the run experiment, like in the example below.

For example, if you have an evaluator called "Compliance - do not recommend any financial actions":

You would pass this to an experiment like this:

from splunk_ao.experiments import run_experiment

results = run_experiment(
    "finance-experiment",
    dataset=dataset,
    function=llm_call,
    metrics=["Compliance - do not recommend any financial actions"],
    project="my-project",
)

Custom evaluators provide the flexibility to define precisely what you want to measure, enabling deep analysis and targeted improvement. For a detailed walkthrough on creating them, see Custom Evaluators.

Ground truth data

Ground truth is the authoritative, validated answer or label used to benchmark model performance. For LLM evaluators, this often means a gold-standard answer, fact, or supporting evidence against which outputs are compared. The following evaluators require ground truth data to compute their scores, as they involve direct comparison to a reference answer, label, or fact.

Ground Truth Adherence

These evaluators are only supported in experiments, as they require the ground truth to be set in the dataset used by the experiment.

To set the ground truth, set this in the output of your dataset either in the Splunk Agent Observability UI, or in code.

Are evaluators LLM-agnostic?

Yes, all evaluators are designed to work across any LLM integrated with Splunk Agent Observability.

Next steps

Evaluators Overview

Explore Splunk Agent Observability’s comprehensive evaluators framework for evaluating and improving AI system performance across multiple dimensions.

Experiments Overview

Learn how to use datasets and experiments to improve your application.

Run experiments

Learn how to run experiments in Splunk Agent Observability using the Splunk Agent Observability SDKs and custom evaluators.

Overview

SaaS deployments

Get Started

Observability

Evaluators

Tokenomics

Signals

AI Assistant

Luna Studio

Experiments

Agent Control

Annotations

Integrations

Security

References

Out-of-the-box evaluators reference

LLM-as-a-judge Evaluators

Luna-3 evaluators

How do I use evaluators in experiments?

Preset evaluators

Custom evaluators

Ground truth data

Are evaluators LLM-agnostic?

Next steps

Evaluators Overview

Experiments Overview

Run experiments

​Out-of-the-box evaluators reference

​LLM-as-a-judge Evaluators

​Luna-3 evaluators

​How do I use evaluators in experiments?

​Preset evaluators

​Custom evaluators

​Ground truth data

​Are evaluators LLM-agnostic?

​Next steps

Evaluators Overview

Experiments Overview

Run experiments

Out-of-the-box evaluators reference

LLM-as-a-judge Evaluators

Luna-3 evaluators

How do I use evaluators in experiments?

Preset evaluators

Custom evaluators

Ground truth data

Are evaluators LLM-agnostic?

Next steps