> ## Documentation Index
> Fetch the complete documentation index at: https://agent-observability-docs.splunk.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Improve LLM-as-a-Judge Evaluators with Autotune

> Use Autotune to turn feedback into prompt improvements that make LLM-as-a-judge evaluators more accurate for your use case.

LLM-as-a-judge evaluators measure LLM application outputs at scale, but may not reflect your team's domain-specific standards out of the box. Whether you're adapting a preset evaluator to a new domain or refining a custom evaluator that still isn't accurate enough, the evaluator prompt often needs tuning to capture your specific evaluation criteria — and doing that manually is time-consuming and hard to scale. Teams typically rewrite prompts, test changes, and repeat that cycle across multiple rounds with no guarantee the result is right.

Autotune lets anyone involved in building or reviewing evaluators — annotators, product managers, or developers — provide feedback on evaluator outputs instead of editing prompts directly. Reviewers correct results and explain their reasoning in natural language. Splunk Agent Observability translates that feedback into prompt improvements and shows exactly what changed.

## When to use Autotune

Use Autotune to improve evaluator performance when:

* A new custom evaluator isn't accurate enough for your use case
* An existing evaluator isn't generalizing well to a new domain or use case
* An existing evaluator is producing inconsistent results with low reviewer agreement in production
* The current prompt isn't handling domain-specific edge cases reliably
* Manual prompt iteration is too time-consuming to scale

## How it works

### See Autotune in action

<Steps>
  <Step title="Review evaluator results across logs">
    Examine evaluator outputs across your logged spans, traces, or sessions where the evaluator isn't performing as expected.
  </Step>

  <Step title="Identify incorrect outputs">
    Flag results that are incorrect or do not match your team's expectations.
  </Step>

  <Step title="Enter the expected value and explain why it's correct">
    For each flagged result, enter the value the evaluator should have produced and add a natural-language explanation of why.
  </Step>

  <Step title="Retune the evaluator">
    Run Autotune using the collected feedback. Splunk Agent Observability aggregates the feedback and adapts the evaluator prompt accordingly.
  </Step>

  <Step title="Review and test the updated prompt">
    Inspect the changes to the prompt and test the updated evaluator before publishing.
  </Step>

  <Step title="Apply the improved evaluator to future runs">
    Publish the updated evaluator so it is used for new logs and evaluations.
  </Step>
</Steps>

Splunk Agent Observability automatically versions the evaluator so you can track changes and revert if needed.

<Note>
  You can optionally recompute historical results with the updated evaluator after publishing.
</Note>

## How to provide good feedback

Autotune supports unlimited feedback per evaluator. Good feedback should include what output the evaluator should have produced and why the corrected result is right. Avoid vague corrections:

| Vague                    | Good                                                                                                                                                                  |
| :----------------------- | :-------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| "This score is wrong"    | "Score should be 60% — the user had 5 goals (A, B, C, D, E) but only completed B, D, and E, so 3 out of 5 were met (3/5 = 60%)"                                       |
| "This should be flagged" | "Should be flagged — the response recommends ibuprofen at a specific dosage without disclaiming that this is not medical advice, which violates the safety criterion" |

## Which evaluators are supported?

Autotune works across all LLM-as-a-judge evaluators, output types, and evaluator levels.

| Category         | Supported                                                         |
| :--------------- | :---------------------------------------------------------------- |
| Evaluator types  | Out-of-the-box and custom LLM-as-a-judge evaluators               |
| Output types     | All types — boolean, categorical, percentage, count, and discrete |
| Evaluator levels | All levels — spans, traces, and sessions                          |

## Related resources

<CardGroup cols={2}>
  <Card title="Custom LLM-as-a-Judge Evaluators" icon="gauge" horizontal href="/concepts/evaluators/custom-evaluators/custom-evaluators-ui-llm">
    Learn how to create and configure custom LLM-as-a-judge evaluators in the Splunk Agent Observability UI.
  </Card>

  <Card title="LLM-as-a-Judge Prompt Engineering Guide" icon="wrench" horizontal href="/concepts/evaluators/custom-evaluators/prompt-engineering">
    Learn best practices for writing effective evaluator prompts.
  </Card>

  <Card title="Evaluators Overview" icon="chart-bar" horizontal href="/concepts/evaluators/overview">
    Explore Splunk Agent Observability's comprehensive evaluators framework for evaluating and improving AI system performance.
  </Card>
</CardGroup>