BuiltInEvaluators

Provides convenient access to built-in Splunk AO evaluators (formerly “scorers”). Examples from splunk_ao import Evaluator

Access built-in evaluators

Evaluator.metrics.correctness Evaluator.metrics.completeness Evaluator.metrics.toxicity

Evaluator

Base class for all Splunk AO evaluators. This is an abstract base class that defines common attributes and methods for all metric types. Use one of the concrete metric classes instead:

SplunkAOEvaluator: Built-in Splunk AO evaluators (access via Evaluator.metrics)
LlmEvaluator: Custom LLM-based metrics with prompt templates
LocalEvaluator: Local function-based metrics
CodeEvaluator: Code-based metrics (future support)

Common Attributes

id (str | None): The unique metric identifier (UUID). name (str): The metric name. scorer_type (ScorerTypes | None): The type of scorer. description (str): Description of the metric. tags (list[str]): Tags associated with the metric. created_at (datetime | None): When the metric was created. updated_at (datetime | None): When the metric was last updated. version (int | None): Evaluator version number.

Class Attributes

metrics (BuiltInEvaluators): Access built-in Splunk AO evaluators. Examples

1. Use built-in Splunk AO evaluators

from splunk_ao import Evaluator, SplunkAOEvaluator, LlmEvaluator, LocalEvaluator, AgentStream agent_stream = AgentStream.get(name=“my-stream”, project_name=“my-project”) agent_stream.set_metrics([ Evaluator.metrics.correctness, Evaluator.metrics.completeness, ])

2. Create custom LLM metric

llm_metric = LlmEvaluator( name=“response_quality”, prompt=“Rate the quality…”, model=“gpt-4o-mini”, judges=3, ).create()

3. Create local function-based metric

def my_scorer(trace_or_span): return 0.5 local_metric = LocalEvaluator( name=“response_length”, scorer_fn=my_scorer, )

delete

def delete(self) -> None

Delete this metric. Only works for server-side metrics. Local metrics don’t need deletion. Examples metric = Evaluator.get(name=“factuality-checker”) metric.delete()

delete_by_name

def delete_by_name(cls, name: str) -> None

Delete a metric by name without retrieving it first. This is more efficient than calling Evaluator.get(name=...).delete() when you only need to delete and don’t need the metric object. Arguments

name: The name of the metric to delete.

get

def get(cls, *, id: str | None=None, name: str | None=None) -> Evaluator | None

Get an existing metric by ID or name. Returns the appropriate subclass instance based on scorer_type. Arguments

id: The metric ID (UUID).
name: The metric name.

list

def list(cls,
         *,
         name_filter: str | None=None,
         scorer_types: list[ScorerTypes] | None=None) -> builtins.list[Evaluator]

List metrics with optional filtering. Returns appropriate subclass instances based on scorer_type. Arguments

name_filter: Filter metrics by exact name match.
scorer_types: Filter by scorer types.

refresh

def refresh(self) -> None

Refresh this metric’s state from the API. Updates all attributes with the latest values from the remote API. Examples metric.refresh() assert metric.is_synced()

to_legacy_metric

def to_legacy_metric(self) -> SchemaMetric

Convert to legacy splunk_ao.schema.metrics.Evaluator format. This enables backward compatibility with existing code that uses the legacy Evaluator class. Examples metric = Evaluator.get(name=“my-metric”) legacy = metric.to_legacy_metric()

Use with existing APIs

update

def update(self, **kwargs: Any) -> Evaluator

Update this metric’s properties on the API. Only name, description, and tags can be updated via this method. On success the instance is updated with the API response and returned in SYNCED state. Arguments

**kwargs (Any): Fields to update. Supported keys: name, description, tags.

Examples metric = Evaluator.get(name=“factuality-checker”) metric.update(name=“new-name”, description=“Updated description”) assert metric.is_synced()

LlmEvaluator

LLM-based metric with custom prompt templates. This metric type allows you to create custom metrics evaluated by an LLM judge using a prompt template. Arguments

Configuration:
-------------: Default values for model and judges can be configured via:
- Configuration.default_scorer_model (env: SPLUNK_AO_DEFAULT_SCORER_MODEL)
- Configuration.default_scorer_judges (env: SPLUNK_AO_DEFAULT_SCORER_JUDGES)

Examples

Create custom LLM metric with string model name

metric = LlmEvaluator( name=“response_quality”, prompt=''' Rate the quality of this response on a scale of 1-10. Question: {input} Answer: {output} Return only the numerical score (1-10). ''', model=“gpt-4o-mini”, # String model name judges=3, node_level=StepType.llm, description=“Rates response quality”, tags=[“quality”, “custom”], output_type=OutputTypeEnum.PERCENTAGE, cot_enabled=True, ).create()

Or use a Model object from Integration

from splunk_ao.integration import Integration gpt_model = Integration.openai.get_model(alias=“gpt-4o-mini”) metric = LlmEvaluator( name=“response_quality”, prompt=“Rate quality 1-10: {input} -> {output}”, model=gpt_model, # Model object judges=3, ).create()

create

def create(self) -> LlmEvaluator

Persist this LLM metric to the API. Examples metric = LlmEvaluator( name=“quality_check”, prompt=“Rate the quality…”, model=“gpt-4o-mini” ).create() assert metric.is_synced()

CodeEvaluator

Code-based metric. This metric type is for code-based scorers that execute custom code to evaluate traces/spans. Examples

Get existing code metric

metric = Evaluator.get(name=“my-code-metric”) assert isinstance(metric, CodeEvaluator)

Create code metric with inline code

metric = CodeEvaluator( name=“custom_code_scorer”, code=“def scorer_fn(step_object):\n return 1.0”, description=“Custom code-based scorer”, tags=[“custom”, “code”], node_level=StepType.llm, output_type=OutputTypeEnum.PERCENTAGE, ).create()

Load code from file

metric = CodeEvaluator( name=“custom_code_scorer”, node_level=StepType.llm, ).load_code(”./scorers/my_scorer.py”).create()

create

def create(self) -> CodeEvaluator

Persist this Code metric to the API. This method validates the code first by submitting it to the validation endpoint, polling for the result, and then creating the scorer with the validated result. Examples

Create with inline code

metric = CodeEvaluator( name=“custom_code_scorer”, code=“def scorer_fn(step_object):\n return 1.0”, node_level=StepType.llm, ).create() assert metric.is_synced()

Create by loading from file

metric = CodeEvaluator( name=“custom_code_scorer”, node_level=StepType.llm, ).load_code(”./scorers/my_scorer.py”).create() assert metric.is_synced()

load_code

def load_code(self, code_file_path: str) -> CodeEvaluator

Load code from a file into this metric instance. Arguments

code_file_path: Path to the Python file containing the scorer code.

SplunkAOEvaluator

Built-in Splunk AO evaluator. This evaluator type represents Splunk AO’s built-in scorers like correctness, completeness, toxicity, etc. Access these via Evaluator.metrics. Examples

Access built-in scorers

from splunk_ao import Evaluator, AgentStream agent_stream = AgentStream.get(name=“my-stream”, project_name=“my-project”) agent_stream.set_metrics([ Evaluator.metrics.correctness, Evaluator.metrics.completeness, Evaluator.metrics.toxicity, ])

Or get by name

metric = Evaluator.get(name=“correctness”) assert isinstance(metric, SplunkAOEvaluator)

LocalEvaluator

Local function-based metric. This metric type uses a Python function to score traces/spans locally without making API calls. Useful for simple, deterministic metrics. Examples

Create local function-based metric

def response_length_scorer(trace_or_span): if hasattr(trace_or_span, “output”) and trace_or_span.output: return min(len(trace_or_span.output) / 100.0, 1.0) return 0.0 local_metric = LocalEvaluator( name=“response_length”, scorer_fn=response_length_scorer, scorable_types=[StepType.llm], aggregatable_types=[StepType.trace], )

Or return (score, metadata) for explainability

EXPECTED = [“relevance”, “accuracy”, “completeness”] def keyword_coverage(trace_or_span): text = getattr(trace_or_span, “output”, "") or "" matched = [k for k in EXPECTED if k in text] return len(matched) / len(EXPECTED), { “matched”: matched, “missing”: [k for k in EXPECTED if k not in text], }

Use with log stream

log_stream.set_metrics([local_metric])

to_local_metric_config

def to_local_metric_config(self) -> LocalMetricConfig

Convert to LocalMetricConfig format. Examples def my_scorer(trace): return 0.5 metric = LocalEvaluator(name=“test”, scorer_fn=my_scorer) config = metric.to_local_metric_config()

Overview

Python SDK

API

evaluator

BuiltInEvaluators

Access built-in evaluators

Evaluator

Common Attributes

Class Attributes

1. Use built-in Splunk AO evaluators

2. Create custom LLM metric

3. Create local function-based metric

delete

delete_by_name

get

list

refresh

to_legacy_metric

Use with existing APIs

update

LlmEvaluator

Create custom LLM metric with string model name

Or use a Model object from Integration

create

CodeEvaluator

Get existing code metric

Create code metric with inline code

Load code from file

create

Create with inline code

Create by loading from file

load_code

SplunkAOEvaluator

Access built-in scorers

Or get by name

LocalEvaluator

Create local function-based metric

Or return (score, metadata) for explainability

Use with log stream

to_local_metric_config

​BuiltInEvaluators

​Access built-in evaluators

​Evaluator

​Common Attributes

​Class Attributes

​1. Use built-in Splunk AO evaluators

​2. Create custom LLM metric

​3. Create local function-based metric

​delete

​delete_by_name

​get

​list

​refresh

​to_legacy_metric

​Use with existing APIs

​update

​LlmEvaluator

​Create custom LLM metric with string model name

​Or use a Model object from Integration

​create

​CodeEvaluator

​Get existing code metric

​Create code metric with inline code

​Load code from file

​create

​Create with inline code

​Create by loading from file

​load_code

​SplunkAOEvaluator

​Access built-in scorers

​Or get by name

​LocalEvaluator

​Create local function-based metric

​Or return (score, metadata) for explainability

​Use with log stream

​to_local_metric_config

BuiltInEvaluators

Access built-in evaluators

Evaluator

Common Attributes

Class Attributes

1. Use built-in Splunk AO evaluators

2. Create custom LLM metric

3. Create local function-based metric

delete

delete_by_name

get

list

refresh

to_legacy_metric

Use with existing APIs

update

LlmEvaluator

Create custom LLM metric with string model name

Or use a Model object from Integration

create

CodeEvaluator

Get existing code metric

Create code metric with inline code

Load code from file

create

Create with inline code

Create by loading from file

load_code

SplunkAOEvaluator

Access built-in scorers

Or get by name

LocalEvaluator

Create local function-based metric

Or return (score, metadata) for explainability

Use with log stream

to_local_metric_config