> ## Documentation Index
> Fetch the complete documentation index at: https://agent-observability-docs.splunk.com/llms.txt
> Use this file to discover all available pages before exploring further.

# evaluator

## BuiltInEvaluators

Provides convenient access to built-in Splunk AO evaluators (formerly "scorers").

**Examples**

from splunk\_ao import Evaluator

# Access built-in evaluators

Evaluator.metrics.correctness
Evaluator.metrics.completeness
Evaluator.metrics.toxicity

## Evaluator

Base class for all Splunk AO evaluators.

This is an abstract base class that defines common attributes and methods
for all metric types. Use one of the concrete metric classes instead:

* **SplunkAOEvaluator**: Built-in Splunk AO evaluators (access via Evaluator.metrics)
* **LlmEvaluator**: Custom LLM-based metrics with prompt templates
* **LocalEvaluator**: Local function-based metrics
* **CodeEvaluator**: Code-based metrics (future support)

## Common Attributes

id (str | None): The unique metric identifier (UUID).
name (str): The metric name.
scorer\_type (ScorerTypes | None): The type of scorer.
description (str): Description of the metric.
tags (list\[str]): Tags associated with the metric.
created\_at (datetime | None): When the metric was created.
updated\_at (datetime | None): When the metric was last updated.
version (int | None): Evaluator version number.

## Class Attributes

metrics (BuiltInEvaluators): Access built-in Splunk AO evaluators.

**Examples**

# 1. Use built-in Splunk AO evaluators

from splunk\_ao import Evaluator, SplunkAOEvaluator, LlmEvaluator, LocalEvaluator, AgentStream

agent\_stream = AgentStream.get(name="my-stream", project\_name="my-project")
agent\_stream.set\_metrics(\[
Evaluator.metrics.correctness,
Evaluator.metrics.completeness,
])

# 2. Create custom LLM metric

llm\_metric = LlmEvaluator(
name="response\_quality",
prompt="Rate the quality...",
model="gpt-4o-mini",
judges=3,
).create()

# 3. Create local function-based metric

def my\_scorer(trace\_or\_span):
return 0.5

local\_metric = LocalEvaluator(
name="response\_length",
scorer\_fn=my\_scorer,
)

### delete

```python theme={null}
def delete(self) -> None
```

Delete this metric.

Only works for server-side metrics. Local metrics don't need deletion.

**Examples**

metric = Evaluator.get(name="factuality-checker")
metric.delete()

### delete\_by\_name

```python theme={null}
def delete_by_name(cls, name: str) -> None
```

Delete a metric by name without retrieving it first.

This is more efficient than calling `Evaluator.get(name=...).delete()`
when you only need to delete and don't need the metric object.

**Arguments**

* `name`: The name of the metric to delete.

### get

```python theme={null}
def get(cls, *, id: str | None=None, name: str | None=None) -> Evaluator | None
```

Get an existing metric by ID or name.

Returns the appropriate subclass instance based on scorer\_type.

**Arguments**

* `id`: The metric ID (UUID).
* `name`: The metric name.

### list

```python theme={null}
def list(cls,
         *,
         name_filter: str | None=None,
         scorer_types: list[ScorerTypes] | None=None) -> builtins.list[Evaluator]
```

List metrics with optional filtering.

Returns appropriate subclass instances based on scorer\_type.

**Arguments**

* `name_filter`: Filter metrics by exact name match.
* `scorer_types`: Filter by scorer types.

### refresh

```python theme={null}
def refresh(self) -> None
```

Refresh this metric's state from the API.

Updates all attributes with the latest values from the remote API.

**Examples**

metric.refresh()
assert metric.is\_synced()

### to\_legacy\_metric

```python theme={null}
def to_legacy_metric(self) -> SchemaMetric
```

Convert to legacy splunk\_ao.schema.metrics.Evaluator format.

This enables backward compatibility with existing code that uses
the legacy Evaluator class.

**Examples**

metric = Evaluator.get(name="my-metric")
legacy = metric.to\_legacy\_metric()

# Use with existing APIs

### update

```python theme={null}
def update(self, **kwargs: Any) -> Evaluator
```

Update this metric's properties on the API.

Only `name`, `description`, and `tags` can be updated via this method.
On success the instance is updated with the API response and returned in SYNCED state.

**Arguments**

* `**kwargs` (`Any`): Fields to update. Supported keys: `name`, `description`, `tags`.

**Examples**

metric = Evaluator.get(name="factuality-checker")
metric.update(name="new-name", description="Updated description")
assert metric.is\_synced()

## LlmEvaluator

LLM-based metric with custom prompt templates.

This metric type allows you to create custom metrics evaluated by an LLM
judge using a prompt template.

**Arguments**

* `Configuration`:

* `-------------`: Default values for `model` and `judges` can be configured via:
  * Configuration.default\_scorer\_model (env: SPLUNK\_AO\_DEFAULT\_SCORER\_MODEL)
  * Configuration.default\_scorer\_judges (env: SPLUNK\_AO\_DEFAULT\_SCORER\_JUDGES)

**Examples**

# Create custom LLM metric with string model name

metric = LlmEvaluator(
name="response\_quality",
prompt='''
Rate the quality of this response on a scale of 1-10.

Question: \{input}
Answer: \{output}

Return only the numerical score (1-10).
''',
model="gpt-4o-mini",  # String model name
judges=3,
node\_level=StepType.llm,
description="Rates response quality",
tags=\["quality", "custom"],
output\_type=OutputTypeEnum.PERCENTAGE,
cot\_enabled=True,
).create()

# Or use a Model object from Integration

from splunk\_ao.integration import Integration
gpt\_model = Integration.openai.get\_model(alias="gpt-4o-mini")
metric = LlmEvaluator(
name="response\_quality",
prompt="Rate quality 1-10: \{input} -> \{output}",
model=gpt\_model,  # Model object
judges=3,
).create()

### create

```python theme={null}
def create(self) -> LlmEvaluator
```

Persist this LLM metric to the API.

**Examples**

metric = LlmEvaluator(
name="quality\_check",
prompt="Rate the quality...",
model="gpt-4o-mini"
).create()
assert metric.is\_synced()

## CodeEvaluator

Code-based metric.

This metric type is for code-based scorers that execute custom code
to evaluate traces/spans.

**Examples**

# Get existing code metric

metric = Evaluator.get(name="my-code-metric")
assert isinstance(metric, CodeEvaluator)

# Create code metric with inline code

metric = CodeEvaluator(
name="custom\_code\_scorer",
code="def scorer\_fn(step\_object):\n    return 1.0",
description="Custom code-based scorer",
tags=\["custom", "code"],
node\_level=StepType.llm,
output\_type=OutputTypeEnum.PERCENTAGE,
).create()

# Load code from file

metric = CodeEvaluator(
name="custom\_code\_scorer",
node\_level=StepType.llm,
).load\_code("./scorers/my\_scorer.py").create()

### create

```python theme={null}
def create(self) -> CodeEvaluator
```

Persist this Code metric to the API.

This method validates the code first by submitting it to the validation
endpoint, polling for the result, and then creating the scorer with the
validated result.

**Examples**

# Create with inline code

metric = CodeEvaluator(
name="custom\_code\_scorer",
code="def scorer\_fn(step\_object):\n    return 1.0",
node\_level=StepType.llm,
).create()
assert metric.is\_synced()

# Create by loading from file

metric = CodeEvaluator(
name="custom\_code\_scorer",
node\_level=StepType.llm,
).load\_code("./scorers/my\_scorer.py").create()
assert metric.is\_synced()

### load\_code

```python theme={null}
def load_code(self, code_file_path: str) -> CodeEvaluator
```

Load code from a file into this metric instance.

**Arguments**

* `code_file_path`: Path to the Python file containing the scorer code.

## SplunkAOEvaluator

Built-in Splunk AO evaluator.

This evaluator type represents Splunk AO's built-in scorers like correctness,
completeness, toxicity, etc. Access these via `Evaluator.metrics`.

**Examples**

# Access built-in scorers

from splunk\_ao import Evaluator, AgentStream

agent\_stream = AgentStream.get(name="my-stream", project\_name="my-project")
agent\_stream.set\_metrics(\[
Evaluator.metrics.correctness,
Evaluator.metrics.completeness,
Evaluator.metrics.toxicity,
])

# Or get by name

metric = Evaluator.get(name="correctness")
assert isinstance(metric, SplunkAOEvaluator)

## LocalEvaluator

Local function-based metric.

This metric type uses a Python function to score traces/spans locally
without making API calls. Useful for simple, deterministic metrics.

**Examples**

# Create local function-based metric

def response\_length\_scorer(trace\_or\_span):
if hasattr(trace\_or\_span, "output") and trace\_or\_span.output:
return min(len(trace\_or\_span.output) / 100.0, 1.0)
return 0.0

local\_metric = LocalEvaluator(
name="response\_length",
scorer\_fn=response\_length\_scorer,
scorable\_types=\[StepType.llm],
aggregatable\_types=\[StepType.trace],
)

# Or return (score, metadata) for explainability

EXPECTED = \["relevance", "accuracy", "completeness"]
def keyword\_coverage(trace\_or\_span):
text = getattr(trace\_or\_span, "output", "") or ""
matched = \[k for k in EXPECTED if k in text]
return len(matched) / len(EXPECTED), \{
"matched": matched,
"missing": \[k for k in EXPECTED if k not in text],
}

# Use with log stream

log\_stream.set\_metrics(\[local\_metric])

### to\_local\_metric\_config

```python theme={null}
def to_local_metric_config(self) -> LocalMetricConfig
```

Convert to LocalMetricConfig format.

**Examples**

def my\_scorer(trace):
return 0.5

metric = LocalEvaluator(name="test", scorer\_fn=my\_scorer)
config = metric.to\_local\_metric\_config()