> ## Documentation Index
> Fetch the complete documentation index at: https://agent-observability-docs.splunk.com/llms.txt
> Use this file to discover all available pages before exploring further.
# Composite Evaluators
> Learn how to create composite evaluators that leverage other evaluators to perform advanced evaluations
Composite evaluators are advanced custom evaluators that can access and leverage the
results of other evaluators to perform sophisticated evaluations. Unlike standard
evaluators that operate independently, composite evaluators build upon previously
computed evaluator values to create more nuanced and context-aware assessments.
## What are composite evaluators?
A **composite evaluator** is a custom evaluator that has access to other evaluators
computed on the current step or any of its child steps. This allows you to:
* Combine multiple evaluator scores into a single comprehensive evaluation
* Apply conditional logic based on evaluator values
* Create hierarchical evaluations that aggregate scores across sessions, traces,
and spans
* Build context-aware evaluators that only calculate when certain conditions are
met
Composite evaluators use the `required_metrics` parameter to specify which evaluators
they depend on. These required evaluators are guaranteed to be computed before the
composite evaluator runs, and their values are accessible via the `step_object.metrics`
dictionary.
## Common use cases
### Conditional evaluation
Calculate an evaluator only when another evaluator meets certain criteria:
**Example**: Only calculate adherence if the input prompt is correct
**Required evaluators**: `SplunkAOEvaluators.correctness`, `SplunkAOEvaluators.context_adherence`
```python theme={null}
from splunk_ao import SplunkAOEvaluators, LlmSpan
def scorer_fn(*, step_object: LlmSpan, **kwargs) -> float:
# Boolean evaluators like correctness return a list of 0/1 values,
# one per judge. Compute the fraction of judges that agreed.
correctness_votes = step_object.metrics[SplunkAOEvaluators.correctness]
correctness_score = (
sum(correctness_votes) / len(correctness_votes)
if correctness_votes else 0.0
)
if correctness_score < 0.7:
return 0.0 # Skip adherence calculation for incorrect inputs
adherence = step_object.metrics[
SplunkAOEvaluators.context_adherence
]
return adherence
```
### Hierarchical aggregation
Aggregate evaluator values across different levels of your application hierarchy:
**Example**: Calculate average evaluator scores across all spans in a session
**Required evaluators**: `SplunkAOEvaluators.context_adherence`
```python theme={null}
from splunk_ao import SplunkAOEvaluators, Session
def scorer_fn(*, step_object: Session, **kwargs) -> float:
llm_scores = []
# Collect scores from all LLM spans across all traces
for trace in step_object.traces:
for span in trace.spans:
if span.type == "llm":
score = span.metrics[SplunkAOEvaluators.context_adherence]
llm_scores.append(score)
# Return average score
return sum(llm_scores) / len(llm_scores) if llm_scores else 0.0
```
### Multi-evaluator analysis
Combine multiple evaluators to detect specific patterns or issues:
**Example**: Check for PII and count occurrences if found
**Required evaluators**: `SplunkAOEvaluators.output_pii`
```python theme={null}
from splunk_ao import SplunkAOEvaluators, LlmSpan
def scorer_fn(*, step_object: LlmSpan, **kwargs) -> int:
# Check if PII is present
has_pii = step_object.metrics[SplunkAOEvaluators.output_pii]
if not has_pii:
return 0
# If PII found, count how many times SSN pattern appears
import re
output = step_object.output.content
ssn_pattern = r'\b\d{3}-\d{2}-\d{4}\b'
ssn_count = len(re.findall(ssn_pattern, output))
return ssn_count
```
### Cross-span evaluation
Use evaluators across different span types in a trace:
**Example**: Combine retriever and LLM evaluators for RAG evaluation
**Required evaluators**: `SplunkAOEvaluators.context_relevance`, `SplunkAOEvaluators.context_adherence`
```python theme={null}
from splunk_ao import SplunkAOEvaluators, Trace
def scorer_fn(*, step_object: Trace, **kwargs) -> float:
retriever_score = 0.0
llm_score = 0.0
for span in step_object.spans:
if span.type == "retriever":
retriever_score = span.metrics[SplunkAOEvaluators.context_relevance]
elif span.type == "llm":
llm_score = span.metrics[SplunkAOEvaluators.context_adherence]
# Combine both scores
return (retriever_score + llm_score) / 2
```
## Specifying required evaluators
The `required_metrics` parameter tells Splunk Agent Observability which evaluators must be computed
before your composite evaluator runs. This ensures the evaluator values are available
when your scorer function executes.
You specify required evaluators when creating your code-based custom evaluator:
* **In the UI**: Select evaluators from the "Required Evaluators" dropdown ([see how](/concepts/evaluators/custom-evaluators/custom-evaluators-ui-code#creating-composite-evaluators))
* **In the Python SDK**: Pass the `required_metrics` parameter
### Splunk Agent Observability preset evaluators
For Splunk Agent Observability's built-in evaluators, use the `SplunkAOEvaluators` enum. For example, you
might select:
* `SplunkAOEvaluators.context_adherence`
* `SplunkAOEvaluators.context_adherence_luna`
* `SplunkAOEvaluators.correctness`
### Custom evaluators
For your own custom evaluators, reference them by name as strings. You can also
mix custom evaluators with Splunk Agent Observability preset evaluators:
* `"My Custom Evaluator"` (string for custom evaluator)
* `"Compliance Check"` (string for custom evaluator)
* `SplunkAOEvaluators.output_pii` (Splunk Agent Observability preset evaluator)
## Accessing evaluator values
Once you've specified required evaluators, access them through the
`step_object.metrics` dictionary:
```python theme={null}
def scorer_fn(*, step_object: LlmSpan, **kwargs) -> float:
# Access evaluators using the same enum or string used in required_metrics
adherence = step_object.metrics[SplunkAOEvaluators.context_adherence]
custom_score = step_object.metrics["My Custom Evaluator"]
# Use the evaluator values in your logic
return (adherence + custom_score) / 2
```
### Boolean vs. float evaluators
Different Splunk Agent Observability evaluators return different value types:
* **Boolean evaluators** (e.g. `correctness`, `context_adherence`) are evaluated by multiple judges and return a `list[int]` at the root level, where each element is `0` (false) or `1` (true) — one value per judge.
* **Float evaluators** (e.g. `completeness`) return a single `float` between 0 and 1.
When using a boolean evaluator in your composite scorer, you must handle the list:
```python theme={null}
# Boolean evaluator — returns a list of 0/1 values, one per judge
correctness_votes = step_object.metrics[SplunkAOEvaluators.correctness]
# e.g. [1, 0, 1] when 3 judges ran
# Get the fraction of judges that agreed (0.0 – 1.0)
correctness_score = (
sum(correctness_votes) / len(correctness_votes)
if correctness_votes else 0.0
)
# Or check if any/all judges agreed
passed_any = any(v == 1 for v in correctness_votes)
passed_all = all(v == 1 for v in correctness_votes)
```
## Complete example: multi-level session evaluator
This example demonstrates a comprehensive composite evaluator that aggregates
scores from all hierarchy levels.
**Required evaluators to select** ([in UI dropdown](/concepts/evaluators/custom-evaluators/custom-evaluators-ui-code#creating-composite-evaluators) or SDK parameter):
* `SplunkAOEvaluators.conversation_quality`
* `SplunkAOEvaluators.action_completion`
* `SplunkAOEvaluators.agent_efficiency`
* `SplunkAOEvaluators.action_completion_luna`
* `SplunkAOEvaluators.action_advancement`
* `SplunkAOEvaluators.context_adherence`
* `SplunkAOEvaluators.context_relevance`
* `SplunkAOEvaluators.tool_error_rate`
```python theme={null}
from splunk_ao import SplunkAOEvaluators, Session
def scorer_fn(*, step_object: Session, **kwargs) -> float:
"""
Comprehensive session score combining metrics from all hierarchy levels.
"""
# Session-level evaluators
conversation_quality = step_object.metrics[
SplunkAOEvaluators.conversation_quality
]
action_completion = step_object.metrics[SplunkAOEvaluators.action_completion]
agent_efficiency = step_object.metrics[SplunkAOEvaluators.agent_efficiency]
# Collect trace-level evaluators
trace_scores = []
for trace in step_object.traces:
trace_scores.append(
trace.metrics[SplunkAOEvaluators.action_completion_luna]
)
trace_scores.append(trace.metrics[SplunkAOEvaluators.action_advancement])
# Collect span-level evaluators by type
llm_scores = []
retriever_scores = []
tool_scores = []
for trace in step_object.traces:
for span in trace.spans:
if span.type == "llm":
llm_scores.append(
span.metrics[SplunkAOEvaluators.context_adherence]
)
elif span.type == "retriever":
retriever_scores.append(
span.metrics[SplunkAOEvaluators.context_relevance]
)
elif span.type == "tool":
tool_scores.append(
1 - span.metrics[SplunkAOEvaluators.tool_error_rate]
)
# Calculate averages for each level
session_avg = (
conversation_quality + action_completion + agent_efficiency
) / 3
trace_avg = sum(trace_scores) / len(trace_scores) if trace_scores else 0.5
llm_avg = sum(llm_scores) / len(llm_scores) if llm_scores else 0.5
retriever_avg = (
sum(retriever_scores) / len(retriever_scores)
if retriever_scores
else 0.5
)
tool_avg = sum(tool_scores) / len(tool_scores) if tool_scores else 0.5
# Return weighted average across all levels
return (session_avg + trace_avg + llm_avg + retriever_avg + tool_avg) / 5
```
## Best practices
### Be specific with required evaluators
Only include evaluators you actually use. This improves performance and makes your
evaluator's dependencies clear:
```python theme={null}
# Bad - includes unnecessary evaluators
required_metrics = [
SplunkAOEvaluators.context_adherence,
SplunkAOEvaluators.context_relevance,
SplunkAOEvaluators.completeness, # Not used in scorer
SplunkAOEvaluators.correctness # Not used in scorer
]
# Good - only required evaluators
required_metrics = [
SplunkAOEvaluators.context_adherence,
SplunkAOEvaluators.context_relevance
]
```
### Use appropriate step types
Match your composite evaluator's step type to where the required evaluators exist:
* **Session**: Can access session, trace, and span evaluators
* **Trace**: Can access trace and span evaluators
* **Span**: Can only access evaluators on that specific span
### Execution restrictions
Composite evaluators depend on the successful completion of their `required_metrics`:
* While any required evaluator is not yet final (e.g., queued or computing), the composite evaluator remains **queued**.
* If any required evaluator finishes without a successful final status (e.g., failed, not computed, or not applicable), the composite evaluator **raises an error** that includes the failed statuses of those required evaluators.
* Evaluators **not** listed in `required_metrics` do **not** affect the composite evaluator—only the required ones gate execution.
## Creating composite evaluators
Composite evaluators can be created in two ways:
1. **Splunk Agent Observability UI**: Use the custom code-based evaluators editor and select
required evaluators from the "Required Evaluators" dropdown
2. **Python SDK**: Add the `required_metrics` parameter when creating code-based
evaluators
Composite evaluators are only supported for **code-based custom evaluators**.
LLM-as-a-judge evaluators do not support the `required_metrics` parameter.
Learn how to create composite evaluators using the Splunk Agent Observability UI
View Python SDK documentation for evaluators
Learn about custom code-based evaluators in Splunk Agent Observability
## Next steps
Learn how to create custom code-based evaluators in Splunk Agent Observability
Explore Splunk Agent Observability's comprehensive evaluators framework
Learn how to use evaluators in experiments