Skip to main content
LLM-as-a-judge evaluators measure LLM application outputs at scale, but may not reflect your team’s domain-specific standards out of the box. Whether you’re adapting a preset evaluator to a new domain or refining a custom evaluator that still isn’t accurate enough, the evaluator prompt often needs tuning to capture your specific evaluation criteria — and doing that manually is time-consuming and hard to scale. Teams typically rewrite prompts, test changes, and repeat that cycle across multiple rounds with no guarantee the result is right. Autotune lets anyone involved in building or reviewing evaluators — annotators, product managers, or developers — provide feedback on evaluator outputs instead of editing prompts directly. Reviewers correct results and explain their reasoning in natural language. Splunk Agent Observability translates that feedback into prompt improvements and shows exactly what changed.

​
When to use Autotune

Use Autotune to improve evaluator performance when:
  • A new custom evaluator isn’t accurate enough for your use case
  • An existing evaluator isn’t generalizing well to a new domain or use case
  • An existing evaluator is producing inconsistent results with low reviewer agreement in production
  • The current prompt isn’t handling domain-specific edge cases reliably
  • Manual prompt iteration is too time-consuming to scale

​
How it works

​
See Autotune in action

1

Review evaluator results across logs

Examine evaluator outputs across your logged spans, traces, or sessions where the evaluator isn’t performing as expected.
2

Identify incorrect outputs

Flag results that are incorrect or do not match your team’s expectations.
3

Enter the expected value and explain why it's correct

For each flagged result, enter the value the evaluator should have produced and add a natural-language explanation of why.
4

Retune the evaluator

Run Autotune using the collected feedback. Splunk Agent Observability aggregates the feedback and adapts the evaluator prompt accordingly.
5

Review and test the updated prompt

Inspect the changes to the prompt and test the updated evaluator before publishing.
6

Apply the improved evaluator to future runs

Publish the updated evaluator so it is used for new logs and evaluations.
Splunk Agent Observability automatically versions the evaluator so you can track changes and revert if needed.
You can optionally recompute historical results with the updated evaluator after publishing.

​
How to provide good feedback

Autotune supports unlimited feedback per evaluator. Good feedback should include what output the evaluator should have produced and why the corrected result is right. Avoid vague corrections:

​
Which evaluators are supported?

Autotune works across all LLM-as-a-judge evaluators, output types, and evaluator levels.

Custom LLM-as-a-Judge Evaluators

Learn how to create and configure custom LLM-as-a-judge evaluators in the Splunk Agent Observability UI.

LLM-as-a-Judge Prompt Engineering Guide

Learn best practices for writing effective evaluator prompts.

Evaluators Overview

Explore Splunk Agent Observability’s comprehensive evaluators framework for evaluating and improving AI system performance.