> ## Documentation Index
> Fetch the complete documentation index at: https://agent-observability-docs.splunk.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Evaluators Overview

> Explore Splunk Agent Observability's comprehensive evaluators framework for evaluating and improving AI system performance across multiple dimensions

export const ModalityIcons = ({modalities}) => {
  const iconProps = {
    fill: "none",
    height: 20,
    width: 20,
    stroke: "currentColor",
    strokeLinecap: "round",
    strokeLinejoin: "round",
    strokeWidth: 2,
    viewBox: "0 0 24 24",
    xmlns: "http://www.w3.org/2000/svg"
  };
  const renderIcon = modality => {
    if (modality === "Text") {
      return <svg {...iconProps} aria-hidden="true">
          <circle cx="12" cy="12" r="9" />
          <path d="M8 8h8M12 8v8" />
        </svg>;
    }
    if (modality === "Image/PDF") {
      return <svg {...iconProps} aria-hidden="true">
          <rect height="18" rx="2" width="18" x="3" y="3" />
          <circle cx="8.5" cy="8.5" r="1.5" />
          <path d="m21 15-5-5L5 21" />
        </svg>;
    }
    if (modality === "Audio") {
      return <svg {...iconProps} aria-hidden="true">
          <path d="M9 18V5l10-2v13" />
          <circle cx="6" cy="18" r="3" />
          <circle cx="16" cy="16" r="3" />
        </svg>;
    }
    return null;
  };
  return <span aria-label={`Supported modalities: ${modalities.join(", ")}`} role="group" style={{
    display: "inline-flex",
    alignItems: "center",
    gap: "0.5rem"
  }}>
      {modalities.map(modality => <span aria-label={modality} key={modality} role="img" style={{
    display: "inline-flex",
    flexShrink: 0
  }} title={modality}>
          {renderIcon(modality)}
        </span>)}
    </span>;
};

Splunk Agent Observability comes with a set of ready to use [Out-of-the-Box evaluators](/concepts/evaluators/evaluator-comparison) that allow you to see how your AI is performing. With these evaluators, you can quickly spot problems, track improvements, and make your AI work better for your users. These evaluators apply to different node types (such as session, trace, or different span types), depending on the evaluator.

You can then expand these evaluators with custom evaluators, using [LLM-as-a-judge](/concepts/evaluators/custom-evaluators/custom-evaluators-ui-llm), or [custom code-based evaluators](/concepts/evaluators/custom-evaluators/custom-evaluators-ui-code).

To calculate Out-of-the-Box evaluators, or LLM-as-a-judge evaluators, you first need to configure an integration with an LLM, or [Luna-2](/concepts/luna/luna). [Connect Splunk Agent Observability to your language model](/concepts/evaluators/overview#configure-splunk-agent-observability-for-out-of-the-box-and-llm-as-a-judge-evaluators) by adding your API key on the **LLM Integrations** page from within the Splunk Agent Observability application.

You can improve the evaluator calculation based on your requirements using [Autotune](/concepts/evaluators/autotune-llm-as-a-judge-evaluators). This allows you to continuously provide feedback in natural language that automatically improves the evaluators to align better with your domain, or expected inputs and outputs.

Evaluators can be used with [experiments](/concepts/experiments/running-experiments-in-console), and [Agent Streams](/concepts/logging/configure-evaluators/configure-evaluators).

## Configure Splunk Agent Observability for Out-of-the-Box and LLM-as-a-judge evaluators

Most Out-of-the-Box evaluators and all LLM-as-a-judge evaluators are LLM-based evaluators.
LLM-based evaluators use an LLM to evaluate inputs and outputs.
To use these evaluators from Agent Streams or Experiments, you first need to configure an integration with an LLM platform.

<Steps>
  <Step title="Navigate to the LLM Integrations page" id="step-navigate">
    In the Splunk Agent Observability UI, navigate to the **LLM Integrations** page by selecting your user profile in the upper-right corner and then selecting **Integrations**.

    <img src="https://mintcdn.com/agent-observability-docs/Y4gaVgpsSUs8MBdT/images/console-ui/user-menu-sao.png?fit=max&auto=format&n=Y4gaVgpsSUs8MBdT&q=85&s=3e71ca4ab350c4552866ecb0a323975f" alt="The user menu" width="2458" height="906" data-path="images/console-ui/user-menu-sao.png" />
  </Step>

  <Step title="Add an integration" id="step-add-integration">
    Locate the LLM provider you are using (or specify a [custom integration](/sdk-api/third-party-integrations/model-integrations/custom-model-integrations/custom-model-integrations)), then select the **+Add Integration** button.

    <img src="https://mintcdn.com/agent-observability-docs/Y4gaVgpsSUs8MBdT/images/console-ui/llm-integrations-sao.png?fit=max&auto=format&n=Y4gaVgpsSUs8MBdT&q=85&s=d7faf841b98a7de492bd250734007d0d" alt="LLM provider options" width="1846" height="1098" data-path="images/console-ui/llm-integrations-sao.png" />
  </Step>

  <Step title="Add settings" id="step-add-settings">
    Specify settings for your integration (such as an API key), then select **Save changes**.
  </Step>
</Steps>

## Using evaluators effectively

To get the most value from Splunk Agent Observability's evaluators:

1. **Start with key evaluators** - Focus on evaluators most relevant to your use case
2. **Establish baselines** - Understand your current performance before making changes
3. **Track trends over time** - Monitor how evaluators change as you iterate on your system
4. **Combine multiple evaluators** - Look at related evaluators together for a more complete picture
5. **Set thresholds** - Define acceptable ranges for critical evaluators
6. **Improve the evaluators** - Use CLHF to continuously improve the evaluators

## Out-of-the-Box evaluator categories

Our evaluators can be broken down into seven key categories, each addressing a specific aspect of AI system performance. You may benefit from using evaluators from more than one category. Splunk Agent Observability also supports [custom evaluators](/concepts/evaluators/custom-evaluators/custom-evaluators-ui-llm) that are able to be implemented alongside the Out-of-the-Box evaluator options.

The [Evaluators Comparison](/concepts/evaluators/evaluator-comparison) provides a full list of the Out-of-the-Box evaluators for each category.

### 1. Agentic performance evaluators

[Agentic Performance Evaluators](/concepts/evaluators/agentic/agentic-overview) evaluate how effectively AI agents perform tasks, use tools, and progress toward goals.

| When to Use                                                                                                  | Example Use Cases                                                                                                                                                                                                                               |
| :----------------------------------------------------------------------------------------------------------- | :---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| When building and optimizing AI systems that take actions, make decisions, or use tools to accomplish tasks. | <ul><li>Evaluating a travel planning agent's ability to book complete itineraries</li><li>Assessing a coding assistant's appropriate use of APIs and libraries</li><li>Measuring a data analysis agent's tool selection effectiveness</li></ul> |

### 2. Expression and readability evaluators

[Expression And Readability Evaluators](/concepts/evaluators/expression-and-readability/expression-and-readability-overview) assess the style, tone, clarity, and overall presentation of AI-generated content.

| When to Use                                                                                                   | Example Use Cases                                                                                                                                                                                                                                    |
| :------------------------------------------------------------------------------------------------------------ | :--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| When the format, tone, and presentation of AI outputs are important for user experience or brand consistency. | <ul><li>Ensuring a luxury brand chatbot maintains a sophisticated tone</li><li>Verifying educational content is presented at the appropriate reading level</li><li>Measuring clarity and conciseness in technical documentation generation</li></ul> |

### 3. Multimodal quality evaluators

[Multimodal Quality Evaluators](/concepts/evaluators/multimodal-quality/multimodal-quality-overview) evaluate whether multimodal inputs and outputs (such as images and audio conversations) are usable and compliant for the task at hand.

| When to Use                                                                                           | Example Use Cases                                                                                                                                                                                     |
| :---------------------------------------------------------------------------------------------------- | :---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| When your AI workflow includes images or audio conversations and you need quality/compliance signals. | <ul><li>Gating blurry document photos before extraction</li><li>Enforcing brand rules on generated marketing images</li><li>Detecting turn-taking issues in voice agents (overlap/barge-in)</li></ul> |

### 4. Response quality evaluators

[Response Quality Evaluators](/concepts/evaluators/response-quality/response-quality-overview) help you evaluate how correctly, consistently, and in line with ground truth your AI follows instructions and answers user queries in any setting — with or without RAG (correctness, instruction adherence, ground truth adherence).

### 5. RAG evaluators

[RAG Evaluators](/concepts/evaluators/rag/rag-overview) help you evaluate retrieval and generation quality in RAG pipelines, including retrieval quality (chunk relevance, context relevance, context precision, Precision @ K) and generation quality (chunk attribution utilization, context adherence, completeness).

| When to Use                                                                                                                                                       | Example Use Cases                                                                                                                                                                                                             |
| :---------------------------------------------------------------------------------------------------------------------------------------------------------------- | :---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| When evaluating how well retrieval systems find relevant context and how well models use that context to produce accurate, complete, and well-grounded responses. | <ul><li>Measuring factual accuracy in a medical information system</li><li>Evaluating how well a RAG system uses retrieved information</li><li>Assessing if customer service responses address all parts of a query</li></ul> |

### 6. Safety and compliance evaluators

[Safety And Compliance Evaluators](/concepts/evaluators/safety-and-compliance/safety-and-compliance-overview) identify potential risks, harmful content, bias, or privacy concerns in AI interactions.

| When to Use                                                                                                                  | Example Use Cases                                                                                                                                                                                                            |
| :--------------------------------------------------------------------------------------------------------------------------- | :--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| When ensuring AI systems meet regulatory requirements, protect user privacy, and avoid generating harmful or biased content. | <ul><li>Detecting PII in healthcare chatbot conversations</li><li>Identifying potential prompt injection attacks in public-facing systems</li><li>Measuring bias in hiring or loan approval recommendation systems</li></ul> |

### 7. Text-to-SQL evaluators

[Text-to-SQL evaluators](/concepts/evaluators/text2sql/text2sql-overview) evaluate the accuracy and effectiveness of SQL queries generated by AI models from natural language inputs.

| When to Use                                                                                                                                                                                                                                  | Example Use Cases                                                                                                                                                                                                                                                                                                                      |
| :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Use these evaluators to evaluate the quality of SQL queries generated from natural language inputs. They're essential when building Text-to-SQL systems that need to produce syntactically correct queries grounded in your database schema. | <ul><li>A data analytics assistant where users ask questions in natural language and expect accurate query results</li><li>A customer-facing data analytics chatbot that must prevent injection attacks from user inputs</li><li>A business intelligence platform where ad-hoc queries must not impact database availability</li></ul> |

## Next steps

<CardGroup cols={2}>
  <Card title="Custom LLM-as-a-judge evaluators" icon="brain" horizontal href="/concepts/evaluators/custom-evaluators/custom-evaluators-ui-llm">
    Learn how to create evaluators using LLMs to judge the quality of responses
  </Card>

  <Card title="Custom code-based evaluators" icon="code" horizontal href="/concepts/evaluators/custom-evaluators/custom-evaluators-ui-code">
    Learn how to create, register, and use custom evaluators to evaluate your LLM applications
  </Card>

  <Card title="Improve LLM-as-a-Judge Evaluators with Autotune" icon="message" horizontal href="/concepts/evaluators/autotune-llm-as-a-judge-evaluators">
    Learn how to improve your LLM-as-a-judge evaluators using expert feedback with Autotune
  </Card>
</CardGroup>