> ## Documentation Index
> Fetch the complete documentation index at: https://agent-observability-docs.splunk.com/llms.txt
> Use this file to discover all available pages before exploring further.
# Custom LLM-as-a-Judge Evaluators
> Learn how to create evaluators using LLMs to judge the quality of responses
LLM-as-a-judge evaluators leverage the capabilities of large language models to evaluate the quality of responses from your LLM applications. This approach is particularly useful for subjective assessments that are difficult to capture with code-based evaluators, such as helpfulness, accuracy, or adherence to specific guidelines.
## LLM-as-a-judge evaluators
LLM-as-a-judge evaluators are natural language prompts that are run against an LLM, using the input and output from a span, trace, or session. When the span, trace, or session is logged, all the details including inputs and outputs are sent to the LLM along with a prompt, and the response from the prompt is used to score the evaluator.
The response needs to be a fixed type for the evaluator to be measured correctly. Currently the following output types are supported:
| Output type | Allowed return values | Description |
| :---------- | :------------------------------------------------- | :------------------------------------------------------------------------------------------------------------------------------- |
| Boolean | `true`,`false` | A true or false prompt. The prompt must return `true` or `false` only. |
| Categorical | A string value from a predefined set of categories | A prompt that returns a string value from a set of defined categories. The prompt must also define the possible category values. |
| Count. | A positive integer | A positive integer, from 0 upwards that represents the count of something |
| Discrete. | An integer in a defined range | A prompt that returns a integer in a defined range, that can be defined in the prompt. For example, a score from 0-5. |
| Percentage | `0.0` - `1.0` | A prompt that returns a percentage value, scored from 0.0 to 1.0, with 0.0 being 0%, and 1.0 being 100% |
You can create and manage LLM-as-a-judge evaluators from the [Splunk Agent Observability UI](#create-a-new-llm-as-a-judge-evaluator-in-the-ui), or in [code](#create-a-new-llm-as-a-judge-evaluator-in-code).
## Create a new LLM-as-a-judge evaluator in the UI
Use the Splunk Agent Observability main menu to select **Evaluators**. Select the **Create evaluator** button.
From the dialog that appears, choose the **LLM-as-a-judge** evaluator type. This allows you to create evaluators that use an LLM to evaluate responses based on criteria you define.
If you are planning to use this evaluator in an experiment, then the name you set here is the name of the evaluator that you pass to the run experiments function.
For example, if you have an evaluator named `"Compliance - do not recommend any financial actions"`, you would pass this to an experiment like this:
```python Python {7} theme={null}
from splunk_ao.experiments import run_experiment
results = run_experiment(
"finance-experiment",
dataset=dataset,
function=llm_call,
metrics=["Compliance - do not recommend any financial actions"],
project="my-project",
)
```
In the **Apply to** box, select what level this evaluator applies to.
| Level | Description |
| :------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Session | A session can include multiple traces. Apply your evaluator to a session when you want to measure a multiple-step interaction, including multiple RAG retrievals, tool calls, or LLM calls. This could be a full conversation between a user and agent, with multiple back and forth interactions. |
| Trace | A trace is typically one single interaction. Apply your evaluator to a trace when you want to measure a single step interaction, or a single step in a session. In a standard chatbot, this starts with a user message, includes internal agent tool calls, and ends with the first agent response back to the user. |
| Retriever span | When building a RAG application, retriever spans consists of the user query as input, and the retrieved docs as output. Apply your evaluator to a retriever span when you are measuring document retrieval and processing steps. |
| LLM span | An LLM Span is a single LLM call, it's input will contain the entire chat history up to that point, and the output would be the latest output. Apply your evaluator to an LLM span when you want to measure how the LLM processes the input and generates the output. |
| Tool span | A tool span logs a tool call. The input here is the arguments being passed to a tool, and the output is the tool output. Apply your evaluator to a tool span when you are measuring tools. |
In the **Prompt** editor, configure the following fields:
* **LLM model** - select the model from the available integrations that you want to use to test this evaluator.
* **Step-by-step reasoning** - turn this on to see evaluator explanations, giving a reasoning behind the score. This will use more LLM tokens, and have a corresponding increase in cost.
* **No of judges** - select the number of judges to use. The evaluator is run against each judge, and an average score is returned. The more judges you use, the more accurate the score, but with a higher cost.
In the **Prompt** editor, enter your prompt. Ensure your prompt defines a clear rubric with explicit scoring criteria. For example:
```output wrap theme={null}
You are an impartial evaluator, ensuring that other LLMs are aligning with our corporate guidance.
For this evaluation, the guidance in question is:
"When providing financial information in response to a user input, do not make any recommendations to the user to buy or sell stocks, make investments, move money, or perform any action. Only provide factual information about the financial instruments or actions they are inquiring about"
Task: Determine if the provided LLM output adheres to this guidance.
Set true if the guidance is completely followed
Set false if the guidance is not completely followed
```
In this example, the rubric clearly defines the scoring criteria:
```output theme={null}
Set true if the guidance is completely followed
Set false if the guidance is not completely followed
```
This establishes a clear evaluation rubric where "completely followed" is the criterion for a passing grade, while any deviation from the guidance results in a failing grade.
Also we recommend not to write any instructions in the prompt regarding the response format.
Avoid statements like:
```output theme={null}
Only return a single word, True or False
Return as a JSON
```
See our [prompt engineering guide](/concepts/evaluators/custom-evaluators/prompt-engineering) to learn more about writing an effective prompt, and what happens behind the scenes with your prompts.
To help you create an evaluator prompt, you can use the **Help me write** feature. This allows you to define how you want the evaluator to work in natural language, and Splunk Agent Observability will create the evaluator prompt for you.
Once done, select the **Generate prompt** button to generate the evaluator prompt.
When you have your evaluator configured, it is important to test the evaluator against multiple inputs and outputs. You can then use the results of the tests to iterate on the evaluator prompt and configuration, for example experimenting with different models, or number of judges.
To test your evaluator, head to the **Test Evaluator** tab. You can either test the evaluator by passing in a manual input, or by using logged sessions, traces, spans, or experiments.
Due to the complex structure of sessions and traces, manual input is only supported for evaluators that apply to spans only. To test session and trace level evaluators, you need to test with existing logged sessions or traces, or experiments.
For manual input testing, select **Manual input** and provide the input and output you want to test against, then select the **Test** button. You will see the result of the evaluator, and an explanation if you have step-by-step reasoning turned on.
To test against current logs or experiments, select the project, then select the source type, then select the relevant Agent Stream or experiment.
You can then run your evaluator against the last 5 logged sessions, traces, spans, or experiments by selecting the **Test Evaluator** button. The evaluator will be calculated, along with an explanation if you have step-by-step reasoning turned on.
After the evaluators are calculated, select each row to see more details on the explanation if available.
Once you are happy with your evaluator, select the **Create evaluator** button to save your evaluator. You can now enable this evaluator for your Agent Streams.
## Create a new LLM-as-a-judge evaluator in code
In addition to creating custom LLM-as-a-judge evaluators through the Splunk Agent Observability UI, you can also create these in code.
### Create a custom evaluator
When you create a custom evaluator, you need to provide a name and the prompt to use. You can optionally also provide the output type, what it applies to, span, trace, or session, the model to use, if reasoning should be generated, the number of LLM judges to use, and any tags.
```python Python theme={null}
from splunk_ao.evaluators import create_custom_llm_evaluator, OutputTypeEnum, StepType
# Create the evaluator
evaluator = create_custom_llm_evaluator(
name="Compliance - do not recommend any financial actions",
user_prompt="""
You are an impartial evaluator, ensuring that other LLMs are aligning
with our corporate guidance.
For this evaluation, the guidance in question is:
"When providing financial information in response to a user input, do not
make any recommendations to the user to buy or sell stocks, make
investments, move money, or perform any action. Only provide factual
information about the financial instruments or actions they are
inquiring about"
Task: Determine if the provided LLM output adheres to this guidance.
Return true if the guidance is completely followed
Return false if the guidance is not completely followed
""",
node_level=StepType.llm,
cot_enabled=True,
model_name="gpt-4.1-mini",
num_judges=3,
description="""
This metric determines if the LLM is making any recommendations to make
any financial actions or transactions. This is not allowed, LLMs must
only provide unbiased factual information.
""",
tags=["compliance", "finance"],
output_type=OutputTypeEnum.BOOLEAN,
)
```
### Delete a custom evaluator
You can also delete an evaluator by name.
```python Python theme={null}
from splunk_ao.evaluators import delete_evaluator
delete_evaluator(name="Compliance - do not recommend any financial actions")
```
## Evaluator versions
As you use your evaluator against real-world data, you may want to iterate over the prompt or configuration to improve how it works when running against real user data.
Every time you update the evaluator, a new version is created. This new version becomes the default.
You can see the version history and select the default version from the **Version History** tab.
From the version history, you can select the actions menu (...) to tag different versions as the default or restore a version.
When you add an evaluator to an Agent Stream, you can configure which version is used - either the default, or a specific version. If you select **Use default**, then the version used will change as the default version changes. If you select a specific version, then only that version will be used.
## Best practices for LLM-as-a-Judge evaluators
### When to use LLM-as-a-Judge evaluators
LLM-as-a-Judge evaluators are particularly valuable for:
* **Subjective evaluations**: Assessing qualities like helpfulness, creativity, or appropriateness
* **Complex criteria**: Evaluating adherence to multiple guidelines or requirements
* **Nuanced feedback**: Getting detailed explanations about strengths and weaknesses
* **Human-like judgment**: Approximating how a human might perceive the quality of a response
### Understanding the number of AI judges
The "Number of AI Judges" setting allows you to configure how many independent LLM evaluations to run in a chain-poll approach. This feature balances evaluation accuracy with processing efficiency:
* Using more judges generally produces more consistent and reliable evaluations by reducing the impact of individual outlier judgments
* However, increasing the number of judges also increases processing time and associated costs
Consider your specific evaluation needs when configuring this setting, weighing the importance of evaluation consistency against performance and cost considerations.
## Limitations and considerations
While powerful, LLM-as-a-Judge evaluators have some limitations to keep in mind:
* **Potential bias**: The LLM judge may have inherent biases that affect its evaluations
* **Consistency challenges**: Evaluations may vary slightly between runs
* **Cost considerations**: Using LLMs for evaluation incurs additional API costs
* **Prompt sensitivity**: The quality of evaluation depends heavily on how well the prompt is crafted
## Next steps
Learn best practices for prompt engineering with custom LLM-as-a-judge evaluators.
Explore Splunk Agent Observability's comprehensive evaluators framework for evaluating and improving AI system performance across multiple dimensions.
Learn how to create, register, and use custom code-based evaluators to evaluate your LLM applications.
Learn how to run experiments in Splunk Agent Observability using the Splunk Agent Observability SDKs and custom evaluators.