> ## Documentation Index
> Fetch the complete documentation index at: https://agent-observability-docs.splunk.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Experiment Evaluators

> Learn how to use evaluators in your experiments

Splunk Agent Observability AI provides a set of built-in, preset evaluators designed to evaluate various aspects of LLM, agent, and retrieval-based workflows. You can also create custom evaluators using LLM-as-a-judge, or code. This guide provides a reference for using these evaluators in your experiments.

## Out-of-the-box evaluators reference

The table below summarizes gives the constants used in code to access each evaluator. To use these evaluators, import the relevant enum.

```python Python theme={null}
from splunk_ao import SplunkAOEvaluators
```

```typescript TypeScript theme={null}
import { SplunkAOMetrics } from "splunk-ao";
```

### LLM-as-a-judge Evaluators

**Python**

| Evaluator | Enum Value |
| :- | :- |
| [Action Advancement](https://agent-observability-docs.splunk.com/concepts/evaluators/agentic/action-advancement.md) | `SplunkAOEvaluators.action_advancement` |
| [Action Completion](https://agent-observability-docs.splunk.com/concepts/evaluators/agentic/action-completion.md) | `SplunkAOEvaluators.action_completion` |
| [Agent Efficiency](https://agent-observability-docs.splunk.com/concepts/evaluators/agentic/agent-efficiency.md) | `SplunkAOEvaluators.agent_efficiency` |
| [Agent Flow](https://agent-observability-docs.splunk.com/concepts/evaluators/agentic/agent-flow.md) | `SplunkAOEvaluators.agent_flow` |
| [Completeness](https://agent-observability-docs.splunk.com/concepts/evaluators/rag/generation-quality/completeness.md) | `SplunkAOEvaluators.completeness` |
| [Context Adherence](https://agent-observability-docs.splunk.com/concepts/evaluators/rag/generation-quality/context-adherence.md) | `SplunkAOEvaluators.context_adherence` |
| [Context Precision](https://agent-observability-docs.splunk.com/concepts/evaluators/rag/retrieval-quality/context-precision.md) | `SplunkAOEvaluators.context_precision` |
| [Context Relevance (Query Adherence)](https://agent-observability-docs.splunk.com/concepts/evaluators/rag/retrieval-quality/context-relevance.md) | `SplunkAOEvaluators.context_relevance` |
| [Conversation Quality](https://agent-observability-docs.splunk.com/concepts/evaluators/agentic/conversation-quality.md) | `SplunkAOEvaluators.conversation_quality` |
| [Correctness (factuality)](https://agent-observability-docs.splunk.com/concepts/evaluators/response-quality/correctness.md) | `SplunkAOEvaluators.correctness` |
| [Ground Truth Adherence](https://agent-observability-docs.splunk.com/concepts/evaluators/response-quality/ground-truth-adherence.md) | `SplunkAOEvaluators.ground_truth_adherence` |
| [Instruction Adherence](https://agent-observability-docs.splunk.com/concepts/evaluators/response-quality/instruction-adherence.md) | `SplunkAOEvaluators.instruction_adherence` |
| [Interruption Detection](https://agent-observability-docs.splunk.com/concepts/evaluators/multimodal-quality/interruption-detection.md) | `SplunkAOEvaluators.interruption_detection` |
| [PII (personally identifiable information)](https://agent-observability-docs.splunk.com/concepts/evaluators/safety-and-compliance/pii.md) | `SplunkAOEvaluators.input_pii`, `SplunkAOEvaluators.output_pii` |
| [Prompt Injection](https://agent-observability-docs.splunk.com/concepts/evaluators/safety-and-compliance/prompt-injection.md) | `SplunkAOEvaluators.prompt_injection` |
| [Sexism / Bias](https://agent-observability-docs.splunk.com/concepts/evaluators/safety-and-compliance/sexism.md) | `SplunkAOEvaluators.input_sexism`, `SplunkAOEvaluators.output_sexism` |
| [Tone](https://agent-observability-docs.splunk.com/concepts/evaluators/expression-and-readability/tone.md) | `SplunkAOEvaluators.input_tone`, `SplunkAOEvaluators.output_tone` |
| [Tool Errors](https://agent-observability-docs.splunk.com/concepts/evaluators/agentic/tool-error.md) | `SplunkAOEvaluators.tool_error_rate` |
| [Tool Selection Quality](https://agent-observability-docs.splunk.com/concepts/evaluators/agentic/tool-selection-quality.md) | `SplunkAOEvaluators.tool_selection_quality` |
| [Reasoning Coherence](https://agent-observability-docs.splunk.com/concepts/evaluators/agentic/reasoning-coherence.md) | `SplunkAOEvaluators.reasoning_coherence` |
| [SQL Correctness](https://agent-observability-docs.splunk.com/concepts/evaluators/text2sql/sql-correctness.md) | `SplunkAOEvaluators.sql_correctness` |
| [SQL Adherence](https://agent-observability-docs.splunk.com/concepts/evaluators/text2sql/sql-adherence.md) | `SplunkAOEvaluators.sql_adherence` |
| [SQL Injection](https://agent-observability-docs.splunk.com/concepts/evaluators/text2sql/sql-injection.md) | `SplunkAOEvaluators.sql_injection` |
| [SQL Efficiency](https://agent-observability-docs.splunk.com/concepts/evaluators/text2sql/sql-efficiency.md) | `SplunkAOEvaluators.sql_efficiency` |
| [Toxicity](https://agent-observability-docs.splunk.com/concepts/evaluators/safety-and-compliance/toxicity.md) | `SplunkAOEvaluators.input_toxicity`, `SplunkAOEvaluators.output_toxicity` |
| [User Intent Change](https://agent-observability-docs.splunk.com/concepts/evaluators/agentic/intent-change.md) | `SplunkAOEvaluators.user_intent_change` |
| [Visual Fidelity](https://agent-observability-docs.splunk.com/concepts/evaluators/multimodal-quality/visual-fidelity.md) | `SplunkAOEvaluators.visual_fidelity` |
| [Visual Quality](https://agent-observability-docs.splunk.com/concepts/evaluators/multimodal-quality/visual-quality.md) | `SplunkAOEvaluators.visual_quality` |

**TypeScript**

| Evaluator | Enum Value |
| :- | :- |
| [Action Advancement](https://agent-observability-docs.splunk.com/concepts/evaluators/agentic/action-advancement.md) | `SplunkAOEvaluators.actionAdvancement` |
| [Action Completion](https://agent-observability-docs.splunk.com/concepts/evaluators/agentic/action-completion.md) | `SplunkAOEvaluators.actionCompletion` |
| [Agent Efficiency](https://agent-observability-docs.splunk.com/concepts/evaluators/agentic/agent-efficiency.md) | `SplunkAOEvaluators.agentEfficiency` |
| [Agent Flow](https://agent-observability-docs.splunk.com/concepts/evaluators/agentic/agent-flow.md) | `SplunkAOEvaluators.agentFlow` |
| [Completeness](https://agent-observability-docs.splunk.com/concepts/evaluators/rag/generation-quality/completeness.md) | `SplunkAOEvaluators.completeness` |
| [Context Adherence](https://agent-observability-docs.splunk.com/concepts/evaluators/rag/generation-quality/context-adherence.md) | `SplunkAOEvaluators.contextAdherence` |
| [Context Precision](https://agent-observability-docs.splunk.com/concepts/evaluators/rag/retrieval-quality/context-precision.md) | `SplunkAOEvaluators.contextPrecision` |
| [Context Relevance (Query Adherence)](https://agent-observability-docs.splunk.com/concepts/evaluators/rag/retrieval-quality/context-relevance.md) | `SplunkAOEvaluators.contextRelevance` |
| [Conversation Quality](https://agent-observability-docs.splunk.com/concepts/evaluators/agentic/conversation-quality.md) | `SplunkAOEvaluators.conversationQuality` |
| [Correctness (factuality)](https://agent-observability-docs.splunk.com/concepts/evaluators/response-quality/correctness.md) | `SplunkAOEvaluators.correctness` |
| [Ground Truth Adherence](https://agent-observability-docs.splunk.com/concepts/evaluators/response-quality/ground-truth-adherence.md) | `SplunkAOEvaluators.groundTruthAdherence` |
| [Instruction Adherence](https://agent-observability-docs.splunk.com/concepts/evaluators/response-quality/instruction-adherence.md) | `SplunkAOEvaluators.instructionAdherence` |
| [PII (personally identifiable information)](https://agent-observability-docs.splunk.com/concepts/evaluators/safety-and-compliance/pii.md) | `SplunkAOEvaluators.inputPii`, `SplunkAOEvaluators.outputPii` |
| [Prompt Injection](https://agent-observability-docs.splunk.com/concepts/evaluators/safety-and-compliance/prompt-injection.md) | `SplunkAOEvaluators.promptInjection` |
| [Sexism / Bias](https://agent-observability-docs.splunk.com/concepts/evaluators/safety-and-compliance/sexism.md) | `SplunkAOEvaluators.inputSexism`, `SplunkAOEvaluators.outputSexism` |
| [Tone](https://agent-observability-docs.splunk.com/concepts/evaluators/expression-and-readability/tone.md) | `SplunkAOEvaluators.inputTone`, `SplunkAOEvaluators.outputTone` |
| [Tool Errors](https://agent-observability-docs.splunk.com/concepts/evaluators/agentic/tool-error.md) | `SplunkAOEvaluators.toolErrorRate` |
| [Tool Selection Quality](https://agent-observability-docs.splunk.com/concepts/evaluators/agentic/tool-selection-quality.md) | `SplunkAOEvaluators.toolSelectionQuality` |
| [SQL Correctness](https://agent-observability-docs.splunk.com/concepts/evaluators/text2sql/sql-correctness.md) | `SplunkAOEvaluators.sqlCorrectness` |
| [SQL Adherence](https://agent-observability-docs.splunk.com/concepts/evaluators/text2sql/sql-adherence.md) | `SplunkAOEvaluators.sqlAdherence` |
| [SQL Injection](https://agent-observability-docs.splunk.com/concepts/evaluators/text2sql/sql-injection.md) | `SplunkAOEvaluators.sqlInjection` |
| [SQL Efficiency](https://agent-observability-docs.splunk.com/concepts/evaluators/text2sql/sql-efficiency.md) | `SplunkAOEvaluators.sqlEfficiency` |
| [Toxicity](https://agent-observability-docs.splunk.com/concepts/evaluators/safety-and-compliance/toxicity.md) | `SplunkAOEvaluators.inputToxicity`, `SplunkAOEvaluators.outputToxicity` |
| [User Intent Change](https://agent-observability-docs.splunk.com/concepts/evaluators/agentic/intent-change.md) | `SplunkAOEvaluators.userIntentChange` |

### Luna-3 evaluators

If you are using the [Splunk Agent Observability Luna-3 model](https://agent-observability-docs.splunk.com/concepts/luna/luna.md), then use these evaluator values.

**Python**

| Evaluator | Enum Value |
| :- | :- |
| [Action Advancement](https://agent-observability-docs.splunk.com/concepts/evaluators/agentic/action-advancement.md) | `SplunkAOEvaluators.action_advancement_luna` |
| [Action Completion](https://agent-observability-docs.splunk.com/concepts/evaluators/agentic/action-completion.md) | `SplunkAOEvaluators.action_completion_luna` |
| [Completeness](https://agent-observability-docs.splunk.com/concepts/evaluators/rag/generation-quality/completeness.md) | `SplunkAOEvaluators.completeness_luna` |
| [Context Adherence](https://agent-observability-docs.splunk.com/concepts/evaluators/rag/generation-quality/context-adherence.md) | `SplunkAOEvaluators.context_adherence_luna` |
| [PII (personally identifiable information)](https://agent-observability-docs.splunk.com/concepts/evaluators/safety-and-compliance/pii.md) | `SplunkAOEvaluators.input_pii`, `SplunkAOEvaluators.output_pii` |
| [Prompt Injection](https://agent-observability-docs.splunk.com/concepts/evaluators/safety-and-compliance/prompt-injection.md) | `SplunkAOEvaluators.prompt_injection_luna` |
| [Sexism / Bias](https://agent-observability-docs.splunk.com/concepts/evaluators/safety-and-compliance/sexism.md) | `SplunkAOEvaluators.input_sexism_luna`, `SplunkAOEvaluators.output_sexism_luna` |
| [Tone](https://agent-observability-docs.splunk.com/concepts/evaluators/expression-and-readability/tone.md) | `SplunkAOEvaluators.input_tone`, `SplunkAOEvaluators.output_tone` |
| [Tool Errors](https://agent-observability-docs.splunk.com/concepts/evaluators/agentic/tool-error.md) | `SplunkAOEvaluators.tool_error_rate_luna` |
| [Tool Selection Quality](https://agent-observability-docs.splunk.com/concepts/evaluators/agentic/tool-selection-quality.md) | `SplunkAOEvaluators.tool_selection_quality_luna` |
| [Toxicity](https://agent-observability-docs.splunk.com/concepts/evaluators/safety-and-compliance/toxicity.md) | `SplunkAOEvaluators.input_toxicity_luna`, `SplunkAOEvaluators.output_toxicity_luna` |

**TypeScript**

| Evaluator | Enum Value |
| :- | :- |
| [Action Advancement](https://agent-observability-docs.splunk.com/concepts/evaluators/agentic/action-advancement.md) | `SplunkAOEvaluators.actionAdvancementLuna` |
| [Action Completion](https://agent-observability-docs.splunk.com/concepts/evaluators/agentic/action-completion.md) | `SplunkAOEvaluators.actionCompletionLuna` |
| [Completeness](https://agent-observability-docs.splunk.com/concepts/evaluators/rag/generation-quality/completeness.md) | `SplunkAOEvaluators.completenessLuna` |
| [Context Adherence](https://agent-observability-docs.splunk.com/concepts/evaluators/rag/generation-quality/context-adherence.md) | `SplunkAOEvaluators.contextAdherenceLuna` |
| [PII (personally identifiable information)](https://agent-observability-docs.splunk.com/concepts/evaluators/safety-and-compliance/pii.md) | `SplunkAOEvaluators.inputPii`, `SplunkAOEvaluators.outputPii` |
| [Prompt Injection](https://agent-observability-docs.splunk.com/concepts/evaluators/safety-and-compliance/prompt-injection.md) | `SplunkAOEvaluators.promptInjectionLuna` |
| [Sexism / Bias](https://agent-observability-docs.splunk.com/concepts/evaluators/safety-and-compliance/sexism.md) | `SplunkAOEvaluators.inputSexismLuna`, `SplunkAOEvaluators.outputSexismLuna` |
| [Tone](https://agent-observability-docs.splunk.com/concepts/evaluators/expression-and-readability/tone.md) | `SplunkAOEvaluators.inputTone`, `SplunkAOEvaluators.outputTone` |
| [Tool Errors](https://agent-observability-docs.splunk.com/concepts/evaluators/agentic/tool-error.md) | `SplunkAOEvaluators.toolErrorRateLuna` |
| [Tool Selection Quality](https://agent-observability-docs.splunk.com/concepts/evaluators/agentic/tool-selection-quality.md) | `SplunkAOEvaluators.toolSelectionQualityLuna` |
| [Toxicity](https://agent-observability-docs.splunk.com/concepts/evaluators/safety-and-compliance/toxicity.md) | `SplunkAOEvaluators.inputToxicityLuna`, `SplunkAOEvaluators.outputToxicityLuna` |

## How do I use evaluators in experiments?

The `run experiment` function in [Python](https://agent-observability-docs.splunk.com/sdk-api/python/reference#galileo-experiments) takes a list of evaluators as part of its arguments.

### Preset evaluators

Supply a list of one or more evaluator names into the `run_experiment` function as shown below:

```python Python theme={null}
import os
from splunk_ao.experiments import run_experiment
from splunk_ao.datasets import get_dataset
from splunk_ao.openai import openai
from splunk_ao import SplunkAOEvaluators

dataset = get_dataset(name="fictional_character_names")

# Define a custom "runner" function for your experiment.
def my_custom_llm_runner(input):
  client = openai.OpenAI(api_key=os.environ["OPENAI_API_KEY"])
  return client.chat.completions.create(
        model="gpt-4o",
        messages=[
          { "role": "system", "content": "You are a great storyteller." },
          { "role": "user", "content": f"Write a story about {input["topic"]}" },
        ],
    ).choices[0].message.content

# Run the experiment!
results = run_experiment(
    "test-experiment",
    project="my-test-project-1",
    dataset=dataset,
    function=my_custom_llm_runner,
    metrics=[
        # List evaluators here
        SplunkAOEvaluators.action_advancement,
        SplunkAOEvaluators.completeness,
        SplunkAOEvaluators.instruction_adherence
    ],
)
```

For more information, read about running experiments with the [Splunk Agent Observability SDKs](https://agent-observability-docs.splunk.com/sdk-api/experiments).

### Custom evaluators

You can use [custom evaluators](https://agent-observability-docs.splunk.com/concepts/evaluators/custom-evaluators/custom-evaluators-ui-code.md) in the same way as Splunk Agent Observability's preset evaluators. At a high level, this involves the following steps:

1. Create your evaluator in the Splunk Agent Observability UI (or in [code](https://agent-observability-docs.splunk.com/concepts/evaluators/custom-evaluators/custom-evaluators-ui-code.md#the-scorer-function)). Your custom evaluator will return a numerical score based on its input.
2. Pass the name of your new evaluator into the `run experiment`, like in the example below.

For example, if you have an evaluator called `"Compliance - do not recommend any financial actions"`:

![An evaluator called Compliance - do not recommend any financial actions](https://mintcdn.com/agent-observability-docs/0i9p_J7eAsCmqtd7/concepts/evaluators/custom-evaluators/evaluator-name.webp?fit=max\&auto=format\&n=0i9p_J7eAsCmqtd7\&q=85\&s=6b1cf6ed71e6d1b32cf5105293a0ea20)

You would pass this to an experiment like this:

```python Python {7} theme={null}
from splunk_ao.experiments import run_experiment

results = run_experiment(
    "finance-experiment",
    dataset=dataset,
    function=llm_call,
    metrics=["Compliance - do not recommend any financial actions"],
    project="my-project",
)
```

Custom evaluators provide the flexibility to define precisely what you want to measure, enabling deep analysis and targeted improvement. For a detailed walkthrough on creating them, see [Custom Evaluators](https://agent-observability-docs.splunk.com/concepts/evaluators/custom-evaluators/custom-evaluators-ui-code.md).

## Ground truth data

**Ground truth** is the authoritative, validated answer or label used to benchmark model performance. For LLM evaluators, this often means a gold-standard answer, fact, or supporting evidence against which outputs are compared.

The following evaluators require ground truth data to compute their scores, as they involve direct comparison to a reference answer, label, or fact.

* [Ground Truth Adherence](https://agent-observability-docs.splunk.com/concepts/evaluators/response-quality/ground-truth-adherence.md)

> [!NOTE]
>
> These evaluators are only supported in experiments, as they require the ground truth to be set in the dataset used by the experiment.

To set the ground truth, set this in the output of your dataset either in the [Splunk Agent Observability UI](https://agent-observability-docs.splunk.com/sdk-api/experiments/datasets.md#create-and-manage-datasets-in-the-ui), or [in code](https://agent-observability-docs.splunk.com/sdk-api/experiments/running-experiments.md#ground-truth).

## Are evaluators LLM-agnostic?

Yes, all evaluators are designed to work across any LLM integrated with Splunk Agent Observability.

## Next steps

> **[Evaluators Overview](https://agent-observability-docs.splunk.com/concepts/evaluators/overview.md)**
>
> Explore Splunk Agent Observability's comprehensive evaluators framework for evaluating and improving AI system performance across multiple dimensions.

> **[Experiments Overview](https://agent-observability-docs.splunk.com/sdk-api/experiments/experiments.md)**
>
> Learn how to use datasets and experiments to improve your application.

> **[Run experiments](https://agent-observability-docs.splunk.com/sdk-api/experiments/running-experiments.md#set-evaluators-for-your-experiment)**
>
> Learn how to run experiments in Splunk Agent Observability using the Splunk Agent Observability SDKs and custom evaluators.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.