Skip to main content
Splunk Agent Observability provides a comprehensive suite of out-of-the-box evaluators designed to evaluate various aspects of AI system performance without requiring custom implementation. Most out-of-the-box evaluators are LLM-based. With these evaluators, you can quickly spot problems, track improvements, and make your AI work better for your users. These evaluators apply to different node types (such as session, trace, or different span types), depending on the evaluator. You can then expand these evaluators with custom evaluators: After you set up evaluators, you can use them in experiments and Agent Streams to monitor AI system performance.

​
Which evaluator should I use?

Evaluators can use decision models, LLMs, Luna small language models (SLMs), or code to evaluate inputs and outputs. Choose an evaluator type depending on the reasoning and specialization you need: The following table describes the differences between decision model and LLM-as-a-judge evaluators:

​
How do I use evaluators effectively?

To get the most value from Splunk Agent Observability evaluators:
  1. Start with key evaluators: Focus on evaluators most relevant to your use case.
  2. Establish baselines: Understand your current performance before making changes.
  3. Track trends over time: Monitor how evaluators change as you iterate on your system.
  4. Combine multiple evaluators: Look at related evaluators together for a more complete picture.
  5. Set thresholds: Define acceptable ranges for critical evaluators.
  6. Improve the evaluators: Use CLHF to continuously improve evaluators.

​
Set up an LLM or decision model integration

This step is required to use LLM-based out-of-the-box, LLM-as-a-judge, and decision model evaluators.
1

Navigate to the Integrations page

In the Splunk Agent Observability UI, select your user profile in the upper-right corner and then select Integrations.The user menu
2

Add an integration

Locate the provider you are using or specify a custom integration. Select the +Add Integration button.
3

Add settings

Specify settings for your integration (such as an API key), then select Save changes.

​
Next steps

Out-of-the-box evaluators

Learn about the pre-built evaluators you can use to monitor AI system performance.

Custom decision model evaluators

Learn how to create decision model evaluators for fast, structured judgments.

Custom LLM-as-a-judge evaluators

Learn how to create LLM-as-a-judge evaluators to assess the quality of responses.

Custom code-based evaluators

Learn how to create custom code-based evaluators to evaluate your LLM applications.

Improve LLM-as-a-judge evaluators with Autotune

Learn how to improve your LLM-as-a-judge evaluators using expert feedback with Autotune.