> ## Documentation Index > Fetch the complete documentation index at: https://agent-observability-docs.splunk.com/llms.txt > Use this file to discover all available pages before exploring further. # Custom Decision Model Evaluators > Learn when and how to use decision model evaluators Custom decision model evaluators classify and score text records. Each evaluator asks one question and returns a typed answer with probabilities, without a written explanation. This guide describes when and how to use decision model evaluators. ## Supported decision models Splunk Agent Observability supports the following decision model: | Decision model | Provider | Setup | | - | - | - | | Jev | TypeSafe | Set up a [decision model integration](https://agent-observability-docs.splunk.com/sdk-api/third-party-integrations/decision-models/decision-models.md). | ## How decision model evaluators work A decision model is not an LLM. You provide a record, a plain-language question, and the allowed answers. The model returns an answer with probabilities and does not provide a written explanation. The following table describes the supported question types and results: | Question type | Evaluator output type | Result | | - | - | - | | True/False | Boolean | Returns `true` when the probability is at least 0.5; otherwise returns `false`. | | Choice | Categorical | Selects one of the categories you define. | | Score | Discrete | Selects the most likely level from your rubric. | ## When to use a decision model evaluator Decision model evaluators classify and score large volumes of text faster and at a much lower cost than LLM-as-a-judge evaluators. For example, you can use decision model evaluators to monitor high-volume Agent Streams at full coverage and perform classification-style checks for frustration, task completion, tone, and policy adherence. Use an [LLM-as-a-judge evaluator](https://agent-observability-docs.splunk.com/concepts/evaluators/custom-evaluators/custom-evaluators-ui-llm.md) when you need explanations, counts or percentages, images, audio or files, or very long inputs. For more information on evaluator differences, see [Which evaluator should I use?](https://agent-observability-docs.splunk.com/concepts/evaluators/overview.md#which-evaluator-should-i-use) ## Create and use decision model evaluators ### Create a decision model evaluator Before you begin, [set up a decision model integration](https://agent-observability-docs.splunk.com/sdk-api/third-party-integrations/decision-models/decision-models.md). **1. Navigate to the Evaluators page** From the Splunk Agent Observability main menu, select **Evaluators**. **2. Configure the evaluator** Select **Create evaluator**. Follow the on-screen instructions to configure the evaluator. If your experiment needs an expected answer, turn on **Use ground truth**. For guidance on writing questions, see [How to write effective questions](#how-to-write-effective-questions). **3. Test the evaluator** Select **Test evaluator** to test the evaluator. You can test with manual input, current logs, or a dataset. Dataset testing scores **Macro F1** for boolean and categorical, and **RMSE** for discrete. See [Test Evaluators](https://agent-observability-docs.splunk.com/concepts/evaluators/custom-evaluators/test-evaluators.md) for more information. Update the evaluator as needed, then select **Create Evaluator**. After you create an evaluator, editing and saving it again creates a new evaluator version. You can't change **Apply to** or **Output type** after creation. To change either setting, duplicate the evaluator and configure the copy. ### Decision model evaluator batching When decision model evaluators use the same model and inputs, Splunk Agent Observability batches up to 10 evaluators in a single call to reduce cost. Each evaluator still receives its own answer. If one evaluator in the batch fails, all evaluators in the batch fail and show a clear error. Hover over an evaluator value to see how many evaluators were included in the batch. The cost shown for each evaluator is an estimate calculated as a fraction of the total batch cost based on the batch size. ### How to write effective questions Ask one question per evaluator. If a rubric checks several qualities, split it into separate evaluators. Follow these guidelines based on your question type: * For a question with multiple applicable labels, create one boolean evaluator per label. * For a categorical question, consider creating an **Unclear** category. The model always selects an answer, even when the record doesn't clearly fit the other categories. * For a discrete question, describe each level in concrete terms so adjacent levels are distinguishable. Example questions include: * “Was the user frustrated?” * “Did the agent complete the task?” * “Did the response follow the stated policy?” Test questions against examples you have labeled yourself before relying on their results. ### Use decision model evaluators Decision model evaluators work in [Agent Streams](https://agent-observability-docs.splunk.com/concepts/logging/configure-evaluators/configure-evaluators.md), [experiments](https://agent-observability-docs.splunk.com/sdk-api/experiments/experiments.md), and [playgrounds](https://agent-observability-docs.splunk.com/concepts/experiments/running-experiments-in-console.md), at every step level. To enable one, select **Configure Evaluators** within one of those pages and turn on an evaluator marked with the **Decision** type. [Composite evaluators](https://agent-observability-docs.splunk.com/concepts/evaluators/custom-evaluators/composite-evaluators.md) can use decision model results. For a boolean decision model evaluator, the composite receives its probability. Charts, filters, alerts, and roll-ups work as they do for other evaluators with the same output type. ### Read decision model evaluator results A decision model result appears like any other evaluator result with the same output type. Hover over a result to see its confidence, answer probabilities, model version, token usage, and estimated cost. Each result has one of four states: **Success**, **N/A**, **Error**, or **Pending**. ## Limits and troubleshooting Decision models accept text only. A record containing images, audio, or documents shows **N/A**. The record and question must fit within a 32k-token context window. Splunk Agent Observability does not trim oversized input. If a shared call fails, every evaluator in that call shows the error. The following table describes common errors and how to troubleshoot them: | **What you see** | **What to do** | | - | - | | Authentication error | As the run creator, add or replace your decision model API key. | | Rate-limit error | Raise your decision model quota or use sampling. | | Input exceeds the provider's context window | Change **Input style** to **I/O only**. | | **N/A: Unsupported file types** | Use an [LLM-as-a-judge evaluator](https://agent-observability-docs.splunk.com/concepts/evaluators/custom-evaluators/custom-evaluators-ui-llm.md) for multimodal records. | | **N/A: Ground truth required** | Add the expected output to the dataset. | | **N/A: Sampled out** or **Filtered out** | No action is needed. | | Model is no longer available | Move the evaluator to a replacement model version. | | Toggle disabled in **Configure Evaluators** | Add a [decision model integration](https://agent-observability-docs.splunk.com/sdk-api/third-party-integrations/decision-models/decision-models.md) for the run creator. | See [Decision Models](https://agent-observability-docs.splunk.com/sdk-api/third-party-integrations/decision-models/decision-models.md) for more information on the decision model integration, including billing and rate limits. This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.