Composite Evaluators

Composite evaluators are advanced custom evaluators that can access and leverage the results of other evaluators to perform sophisticated evaluations. Unlike standard evaluators that operate independently, composite evaluators build upon previously computed evaluator values to create more nuanced and context-aware assessments.

What are composite evaluators?

A composite evaluator is a custom evaluator that has access to other evaluators computed on the current step or any of its child steps. This allows you to:

Combine multiple evaluator scores into a single comprehensive evaluation
Apply conditional logic based on evaluator values
Create hierarchical evaluations that aggregate scores across sessions, traces, and spans
Build context-aware evaluators that only calculate when certain conditions are met

Composite evaluators use the required_metrics parameter to specify which evaluators they depend on. These required evaluators are guaranteed to be computed before the composite evaluator runs, and their values are accessible via the step_object.metrics dictionary.

Common use cases

Conditional evaluation

Calculate an evaluator only when another evaluator meets certain criteria: Example: Only calculate adherence if the input prompt is correct Required evaluators: SplunkAOEvaluators.correctness, SplunkAOEvaluators.context_adherence

from splunk_ao import SplunkAOEvaluators, LlmSpan

def scorer_fn(*, step_object: LlmSpan, **kwargs) -> float:
    # Boolean evaluators like correctness return a list of 0/1 values,
    # one per judge. Compute the fraction of judges that agreed.
    correctness_votes = step_object.metrics[SplunkAOEvaluators.correctness]
    correctness_score = (
        sum(correctness_votes) / len(correctness_votes)
        if correctness_votes else 0.0
    )

    if correctness_score < 0.7:
        return 0.0  # Skip adherence calculation for incorrect inputs

    adherence = step_object.metrics[
        SplunkAOEvaluators.context_adherence
    ]
    return adherence

Hierarchical aggregation

Aggregate evaluator values across different levels of your application hierarchy: Example: Calculate average evaluator scores across all spans in a session Required evaluators: SplunkAOEvaluators.context_adherence

from splunk_ao import SplunkAOEvaluators, Session

def scorer_fn(*, step_object: Session, **kwargs) -> float:
    llm_scores = []

    # Collect scores from all LLM spans across all traces
    for trace in step_object.traces:
        for span in trace.spans:
            if span.type == "llm":
                score = span.metrics[SplunkAOEvaluators.context_adherence]
                llm_scores.append(score)

    # Return average score
    return sum(llm_scores) / len(llm_scores) if llm_scores else 0.0

Multi-evaluator analysis

Combine multiple evaluators to detect specific patterns or issues: Example: Check for PII and count occurrences if found Required evaluators: SplunkAOEvaluators.output_pii

from splunk_ao import SplunkAOEvaluators, LlmSpan

def scorer_fn(*, step_object: LlmSpan, **kwargs) -> int:
    # Check if PII is present
    has_pii = step_object.metrics[SplunkAOEvaluators.output_pii]

    if not has_pii:
        return 0

    # If PII found, count how many times SSN pattern appears
    import re
    output = step_object.output.content
    ssn_pattern = r'\b\d{3}-\d{2}-\d{4}\b'
    ssn_count = len(re.findall(ssn_pattern, output))

    return ssn_count

Cross-span evaluation

Use evaluators across different span types in a trace: Example: Combine retriever and LLM evaluators for RAG evaluation Required evaluators: SplunkAOEvaluators.context_relevance, SplunkAOEvaluators.context_adherence

from splunk_ao import SplunkAOEvaluators, Trace

def scorer_fn(*, step_object: Trace, **kwargs) -> float:
    retriever_score = 0.0
    llm_score = 0.0

    for span in step_object.spans:
        if span.type == "retriever":
            retriever_score = span.metrics[SplunkAOEvaluators.context_relevance]
        elif span.type == "llm":
            llm_score = span.metrics[SplunkAOEvaluators.context_adherence]

    # Combine both scores
    return (retriever_score + llm_score) / 2

Specifying required evaluators

The required_metrics parameter tells Splunk Agent Observability which evaluators must be computed before your composite evaluator runs. This ensures the evaluator values are available when your scorer function executes. You specify required evaluators when creating your code-based custom evaluator:

In the UI: Select evaluators from the “Required Evaluators” dropdown (see how)
In the Python SDK: Pass the required_metrics parameter

Splunk Agent Observability preset evaluators

For Splunk Agent Observability’s built-in evaluators, use the SplunkAOEvaluators enum. For example, you might select:

SplunkAOEvaluators.context_adherence
SplunkAOEvaluators.context_adherence_luna
SplunkAOEvaluators.correctness

Custom evaluators

For your own custom evaluators, reference them by name as strings. You can also mix custom evaluators with Splunk Agent Observability preset evaluators:

"My Custom Evaluator" (string for custom evaluator)
"Compliance Check" (string for custom evaluator)
SplunkAOEvaluators.output_pii (Splunk Agent Observability preset evaluator)

Accessing evaluator values

Once you’ve specified required evaluators, access them through the step_object.metrics dictionary:

def scorer_fn(*, step_object: LlmSpan, **kwargs) -> float:
    # Access evaluators using the same enum or string used in required_metrics
    adherence = step_object.metrics[SplunkAOEvaluators.context_adherence]
    custom_score = step_object.metrics["My Custom Evaluator"]

    # Use the evaluator values in your logic
    return (adherence + custom_score) / 2

Boolean vs. float evaluators

Different Splunk Agent Observability evaluators return different value types:

Boolean evaluators (e.g. correctness, context_adherence) are evaluated by multiple judges and return a list[int] at the root level, where each element is 0 (false) or 1 (true) — one value per judge.
Float evaluators (e.g. completeness) return a single float between 0 and 1.

When using a boolean evaluator in your composite scorer, you must handle the list:

# Boolean evaluator — returns a list of 0/1 values, one per judge
correctness_votes = step_object.metrics[SplunkAOEvaluators.correctness]
# e.g. [1, 0, 1] when 3 judges ran

# Get the fraction of judges that agreed (0.0 – 1.0)
correctness_score = (
    sum(correctness_votes) / len(correctness_votes)
    if correctness_votes else 0.0
)

# Or check if any/all judges agreed
passed_any = any(v == 1 for v in correctness_votes)
passed_all = all(v == 1 for v in correctness_votes)

Complete example: multi-level session evaluator

This example demonstrates a comprehensive composite evaluator that aggregates scores from all hierarchy levels. Required evaluators to select (in UI dropdown or SDK parameter):

SplunkAOEvaluators.conversation_quality
SplunkAOEvaluators.action_completion
SplunkAOEvaluators.agent_efficiency
SplunkAOEvaluators.action_completion_luna
SplunkAOEvaluators.action_advancement
SplunkAOEvaluators.context_adherence
SplunkAOEvaluators.context_relevance
SplunkAOEvaluators.tool_error_rate

from splunk_ao import SplunkAOEvaluators, Session

def scorer_fn(*, step_object: Session, **kwargs) -> float:
    """
    Comprehensive session score combining metrics from all hierarchy levels.
    """
    # Session-level evaluators
    conversation_quality = step_object.metrics[
        SplunkAOEvaluators.conversation_quality
    ]
    action_completion = step_object.metrics[SplunkAOEvaluators.action_completion]
    agent_efficiency = step_object.metrics[SplunkAOEvaluators.agent_efficiency]

    # Collect trace-level evaluators
    trace_scores = []
    for trace in step_object.traces:
        trace_scores.append(
            trace.metrics[SplunkAOEvaluators.action_completion_luna]
        )
        trace_scores.append(trace.metrics[SplunkAOEvaluators.action_advancement])

    # Collect span-level evaluators by type
    llm_scores = []
    retriever_scores = []
    tool_scores = []

    for trace in step_object.traces:
        for span in trace.spans:
            if span.type == "llm":
                llm_scores.append(
                    span.metrics[SplunkAOEvaluators.context_adherence]
                )
            elif span.type == "retriever":
                retriever_scores.append(
                    span.metrics[SplunkAOEvaluators.context_relevance]
                )
            elif span.type == "tool":
                tool_scores.append(
                    1 - span.metrics[SplunkAOEvaluators.tool_error_rate]
                )

    # Calculate averages for each level
    session_avg = (
        conversation_quality + action_completion + agent_efficiency
    ) / 3
    trace_avg = sum(trace_scores) / len(trace_scores) if trace_scores else 0.5
    llm_avg = sum(llm_scores) / len(llm_scores) if llm_scores else 0.5
    retriever_avg = (
        sum(retriever_scores) / len(retriever_scores)
        if retriever_scores
        else 0.5
    )
    tool_avg = sum(tool_scores) / len(tool_scores) if tool_scores else 0.5

    # Return weighted average across all levels
    return (session_avg + trace_avg + llm_avg + retriever_avg + tool_avg) / 5

Best practices

Be specific with required evaluators

Only include evaluators you actually use. This improves performance and makes your evaluator’s dependencies clear:

# Bad - includes unnecessary evaluators
required_metrics = [
    SplunkAOEvaluators.context_adherence,
    SplunkAOEvaluators.context_relevance,
    SplunkAOEvaluators.completeness,  # Not used in scorer
    SplunkAOEvaluators.correctness     # Not used in scorer
]

# Good - only required evaluators
required_metrics = [
    SplunkAOEvaluators.context_adherence,
    SplunkAOEvaluators.context_relevance
]

Use appropriate step types

Match your composite evaluator’s step type to where the required evaluators exist:

Session: Can access session, trace, and span evaluators
Trace: Can access trace and span evaluators
Span: Can only access evaluators on that specific span

Execution restrictions

Composite evaluators depend on the successful completion of their required_metrics:

While any required evaluator is not yet final (e.g., queued or computing), the composite evaluator remains queued.
If any required evaluator finishes without a successful final status (e.g., failed, not computed, or not applicable), the composite evaluator raises an error that includes the failed statuses of those required evaluators.
Evaluators not listed in required_metrics do not affect the composite evaluator—only the required ones gate execution.

Creating composite evaluators

Composite evaluators can be created in two ways:

Splunk Agent Observability UI: Use the custom code-based evaluators editor and select required evaluators from the “Required Evaluators” dropdown
Python SDK: Add the required_metrics parameter when creating code-based evaluators

Composite evaluators are only supported for code-based custom evaluators. LLM-as-a-judge evaluators do not support the required_metrics parameter.

Create composite evaluators in the UI

Learn how to create composite evaluators using the Splunk Agent Observability UI

Python SDK reference

View Python SDK documentation for evaluators

Custom evaluators overview

Learn about custom code-based evaluators in Splunk Agent Observability

Next steps

Custom code-based evaluators

Learn how to create custom code-based evaluators in Splunk Agent Observability

Evaluators overview

Explore Splunk Agent Observability’s comprehensive evaluators framework

Run experiments with evaluators

Learn how to use evaluators in experiments

Overview

Get Started

Observability

Evaluators

Signals

AI Assistant

Luna Studio

Experiments

Agent Control

Annotations

Integrations

Security

References

What are composite evaluators?

Common use cases

Conditional evaluation

Hierarchical aggregation

Multi-evaluator analysis

Cross-span evaluation

Specifying required evaluators

Splunk Agent Observability preset evaluators

Custom evaluators

Accessing evaluator values

Boolean vs. float evaluators

Complete example: multi-level session evaluator

Best practices

Be specific with required evaluators

Use appropriate step types

Execution restrictions

Creating composite evaluators

Create composite evaluators in the UI

Python SDK reference

Custom evaluators overview

Next steps

Custom code-based evaluators

Evaluators overview

Run experiments with evaluators

​What are composite evaluators?

​Common use cases

​Conditional evaluation

​Hierarchical aggregation

​Multi-evaluator analysis

​Cross-span evaluation

​Specifying required evaluators

​Splunk Agent Observability preset evaluators

​Custom evaluators

​Accessing evaluator values

​Boolean vs. float evaluators

​Complete example: multi-level session evaluator

​Best practices

​Be specific with required evaluators

​Use appropriate step types

​Execution restrictions

​Creating composite evaluators

Create composite evaluators in the UI

Python SDK reference

Custom evaluators overview

​Next steps

Custom code-based evaluators

Evaluators overview

Run experiments with evaluators

What are composite evaluators?

Common use cases

Conditional evaluation

Hierarchical aggregation

Multi-evaluator analysis

Cross-span evaluation

Specifying required evaluators

Splunk Agent Observability preset evaluators

Custom evaluators

Accessing evaluator values

Boolean vs. float evaluators

Complete example: multi-level session evaluator

Best practices

Be specific with required evaluators

Use appropriate step types

Execution restrictions

Creating composite evaluators

Next steps