Skip to main content
Custom decision model evaluators classify and score text records. Each evaluator asks one question and returns a typed answer with probabilities, without a written explanation. This guide describes when and how to use decision model evaluators.

​
Supported decision models

Splunk Agent Observability supports the following decision model:

​
How decision model evaluators work

A decision model is not an LLM. You provide a record, a plain-language question, and the allowed answers. The model returns an answer with probabilities and does not provide a written explanation. The following table describes the supported question types and results:

​
When to use a decision model evaluator

Decision model evaluators classify and score large volumes of text faster and at a much lower cost than LLM-as-a-judge evaluators. For example, you can use decision model evaluators to monitor high-volume Agent Streams at full coverage and perform classification-style checks for frustration, task completion, tone, and policy adherence. Use an LLM-as-a-judge evaluator when you need explanations, counts or percentages, images, audio or files, or very long inputs. For more information on evaluator differences, see Which evaluator should I use?

​
Create and use decision model evaluators

​
Create a decision model evaluator

Before you begin, set up a decision model integration.
1

Navigate to the Evaluators page

From the Splunk Agent Observability main menu, select Evaluators.
2

Configure the evaluator

Select Create evaluator. Follow the on-screen instructions to configure the evaluator.If your experiment needs an expected answer, turn on Use ground truth. For guidance on writing questions, see How to write effective questions.
3

Test the evaluator

Select Test evaluator to test the evaluator.You can test with manual input, current logs, or a dataset. Dataset testing scores Macro F1 for boolean and categorical, and RMSE for discrete. See Test Evaluators for more information.Update the evaluator as needed, then select Create Evaluator.
After you create an evaluator, editing and saving it again creates a new evaluator version. You can’t change Apply to or Output type after creation. To change either setting, duplicate the evaluator and configure the copy.

​
Decision model evaluator batching

When decision model evaluators use the same model and inputs, Splunk Agent Observability batches up to 10 evaluators in a single call to reduce cost. Each evaluator still receives its own answer. If one evaluator in the batch fails, all evaluators in the batch fail and show a clear error. Hover over an evaluator value to see how many evaluators were included in the batch. The cost shown for each evaluator is an estimate calculated as a fraction of the total batch cost based on the batch size.

​
How to write effective questions

Ask one question per evaluator. If a rubric checks several qualities, split it into separate evaluators. Follow these guidelines based on your question type:
  • For a question with multiple applicable labels, create one boolean evaluator per label.
  • For a categorical question, consider creating an Unclear category. The model always selects an answer, even when the record doesn’t clearly fit the other categories.
  • For a discrete question, describe each level in concrete terms so adjacent levels are distinguishable.
Example questions include:
  • “Was the user frustrated?”
  • “Did the agent complete the task?”
  • “Did the response follow the stated policy?”
Test questions against examples you have labeled yourself before relying on their results.

​
Use decision model evaluators

Decision model evaluators work in Agent Streams, experiments, and playgrounds, at every step level. To enable one, select Configure Evaluators within one of those pages and turn on an evaluator marked with the Decision type. Composite evaluators can use decision model results. For a boolean decision model evaluator, the composite receives its probability. Charts, filters, alerts, and roll-ups work as they do for other evaluators with the same output type.

​
Read decision model evaluator results

A decision model result appears like any other evaluator result with the same output type. Hover over a result to see its confidence, answer probabilities, model version, token usage, and estimated cost. Each result has one of four states: Success, N/A, Error, or Pending.

​
Limits and troubleshooting

Decision models accept text only. A record containing images, audio, or documents shows N/A. The record and question must fit within a 32k-token context window. Splunk Agent Observability does not trim oversized input. If a shared call fails, every evaluator in that call shows the error. The following table describes common errors and how to troubleshoot them: See Decision Models for more information on the decision model integration, including billing and rate limits.