Out-of-the-box Evaluator Categories

Out-of-the-box evaluators can be broken down into the following key categories, each addressing a specific aspect of AI system performance. You may benefit from using evaluators from more than one category. Splunk Agent Observability also supports custom evaluators that you can implement in addition to out-of-the-box evaluators. The Evaluators Comparison provides a full list of the out-of-the-box evaluators for each category.

1. Agentic performance evaluators

Agentic Performance Evaluators evaluate how effectively AI agents perform tasks, use tools, and progress toward goals.

When to Use	Example Use Cases
When building and optimizing AI systems that take actions, make decisions, or use tools to accomplish tasks.	Evaluating a travel planning agent’s ability to book complete itineraries Assessing a coding assistant’s appropriate use of APIs and libraries Measuring a data analysis agent’s tool selection effectiveness

2. Expression and readability evaluators

Expression And Readability Evaluators assess the style, tone, clarity, and overall presentation of AI-generated content.

When to Use	Example Use Cases
When the format, tone, and presentation of AI outputs are important for user experience or brand consistency.	Ensuring a luxury brand chatbot maintains a sophisticated tone Verifying educational content is presented at the appropriate reading level Measuring clarity and conciseness in technical documentation generation

3. Multimodal quality evaluators

Multimodal Quality Evaluators evaluate whether multimodal inputs and outputs (such as images and audio conversations) are usable and compliant for the task at hand.

When to Use	Example Use Cases
When your AI workflow includes images or audio conversations and you need quality/compliance signals.	Gating blurry document photos before extraction Enforcing brand rules on generated marketing images Detecting turn-taking issues in voice agents (overlap/barge-in)

4. Response quality evaluators

Response Quality Evaluators help you evaluate how correctly, consistently, and in line with ground truth your AI follows instructions and answers user queries in any setting — with or without RAG (correctness, instruction adherence, ground truth adherence).

5. RAG evaluators

RAG Evaluators help you evaluate retrieval and generation quality in RAG pipelines, including retrieval quality (chunk relevance, context relevance, context precision, Precision @ K) and generation quality (context adherence, completeness).

When to Use	Example Use Cases
When evaluating how well retrieval systems find relevant context and how well models use that context to produce accurate, complete, and well-grounded responses.	Measuring factual accuracy in a medical information system Evaluating how well a RAG system uses retrieved information Assessing if customer service responses address all parts of a query

6. Safety and compliance evaluators

Safety And Compliance Evaluators identify potential risks, harmful content, bias, or privacy concerns in AI interactions.

When to Use	Example Use Cases
When ensuring AI systems meet regulatory requirements, protect user privacy, and avoid generating harmful or biased content.	Detecting PII in healthcare chatbot conversations Identifying potential prompt injection attacks in public-facing systems Measuring bias in hiring or loan approval recommendation systems

7. Text-to-SQL evaluators

Text-to-SQL evaluators evaluate the accuracy and effectiveness of SQL queries generated by AI models from natural language inputs.

When to Use	Example Use Cases
Use these evaluators to evaluate the quality of SQL queries generated from natural language inputs. They’re essential when building Text-to-SQL systems that need to produce syntactically correct queries grounded in your database schema.	A data analytics assistant where users ask questions in natural language and expect accurate query results A customer-facing data analytics chatbot that must prevent injection attacks from user inputs A business intelligence platform where ad-hoc queries must not impact database availability

Overview

SaaS deployments

Get Started

Observability

Evaluators

Tokenomics

Signals

AI Assistant

Luna Studio

Experiments

Agent Control

Annotations

Integrations

Security

References

1. Agentic performance evaluators

2. Expression and readability evaluators

3. Multimodal quality evaluators

4. Response quality evaluators

5. RAG evaluators

6. Safety and compliance evaluators

7. Text-to-SQL evaluators

​1. Agentic performance evaluators

​2. Expression and readability evaluators

​3. Multimodal quality evaluators

​4. Response quality evaluators

​5. RAG evaluators

​6. Safety and compliance evaluators

​7. Text-to-SQL evaluators

1. Agentic performance evaluators

2. Expression and readability evaluators

3. Multimodal quality evaluators

4. Response quality evaluators

5. RAG evaluators

6. Safety and compliance evaluators

7. Text-to-SQL evaluators