Instruction
Issues with interpreting the user’s objective, following instructions, or deciding whether to act.| Problem label | Description |
|---|---|
| Constraint or instruction-priority violation | An output or action violates an explicit constraint or follows a lower-trust instruction that conflicts with a higher-trust instruction. |
| User intent misinterpreted | Before execution, the system adopts a goal that differs materially from the user’s stated or reasonably inferred goal. |
| Required clarification not requested | The system acts, guesses, or stops without asking for a required value that the user can provide and policy permits requesting. |
| Inappropriate refusal | The system refuses an allowed, supported request despite having sufficient information and no policy requiring refusal. |
| Goal drift | After accepting the correct objective, the system pursues a different objective or expands the task without authorization. |
Knowledge
Issues with retrieving, retaining, or citing information needed for the task.| Problem label | Description |
|---|---|
| Required evidence not retrieved | Retrieval omits required source material that exists in the configured corpus and is available to the task. |
| Irrelevant or noisy retrieval | Retrieved content is unrelated to the task or required claims and degrades the result or adds avoidable processing cost. |
| Stale or false retrieved context | Retrieval supplies content that is outdated, contradicted by the applicable source of truth, or known to be false for the request time. |
| Required context or memory lost | Information available earlier in the session is missing, truncated, or replaced when the system later needs it. |
| Context window limit exceeded | The assembled input exceeds the model’s context limit, or evidence shows it can exceed that limit before the required completion. |
| Citation source mismatch or unsupported citation | A citation does not support its paired claim, or a claim lacks a citation required by the application. |
Reasoning
Issues with decisions, claims, verification, or the delivered result.| Problem label | Description |
|---|---|
| Unsupported or invented claim | A response or intermediate decision makes a material claim that the available evidence does not support, even if the claim is not proven false. |
| Factual or logical error | A decision or output contradicts established facts, a deterministic calculation, or a governing logical rule. |
| Incomplete or prematurely terminated result | The system returns or stops before completing a required part of the accepted task, including a required final answer. |
| Verification absent or incorrect | A required check is skipped, incomplete, or reports a pass that contradicts a directly checkable condition. |
| Output schema or format violation | A delivered output violates the required syntax, schema, fields, or response format. |
| Action and stated decision mismatch | An action or delivered artifact contradicts the same component’s recorded decision or plan without a supported plan revision or change of objective. |
| False success claim | The system claims an action, change, check, or task succeeded, but execution or state evidence shows that the claimed completion did not occur. |
| Success check subverted | The system manipulates a success check or exploits a shortcut so that it passes while an independently established task requirement remains unmet. |
Tool
Issues with selecting tools, calling them, using their results, or handling their effects.| Problem label | Description |
|---|---|
| Wrong tool selected | The system invokes an unsuitable or impermissible tool when a suitable tool or correct path without a tool is available. |
| Required tool call omitted | The system acts, answers, or stops without making a call required by the accepted task and workflow state. |
| Invalid tool arguments | A tool call violates its declared schema, types, required values, normalization rules, or conditions for use. |
| Tool interface contract mismatch | Advertised and exposed tool names, available tools, input or result schemas, conditions for use, or interface versions disagree across an integration. |
| Tool result invalid | A tool reports success but returns data that violates its defined result structure or meaning. |
| Tool result misinterpreted or ignored | The system receives a valid tool result but contradicts it, invents replacement information, or ignores material information. |
| Tool execution error not handled | A tool or integration returns an error or invalid result, and the application fails to recover, validate, degrade safely, or report it as required. |
| Unintended tool side effect | A tool call changes external or application state beyond or differently from the authorized intended effect. |
Orchestration
Issues with workflow routing, handoffs, shared state, or termination.| Problem label | Description |
|---|---|
| Invalid route, sequence, or decomposition | The workflow assigns the wrong node, order, branch, or task breakdown, changing execution or the outcome. |
| Handoff information lost or ignored | Required information is available at a handoff but is withheld, omitted, or ignored by the receiving agent or node. |
| Workflow state not propagated or reset | A transition fails to carry forward, save, or correctly update shared state needed by a later node. |
| Workflow loop or deadlock | The workflow repeats a control state without useful progress or waits cyclically, preventing completion. |
| Termination condition missed | Execution continues without an authorized purpose after a required stopping condition has been reached. |
Runtime
Issues with provider availability, capacity, authentication, or runtime configuration.| Problem label | Description |
|---|---|
| Provider or runtime timeout/unavailable | A model, provider, or workflow operation times out or is unavailable, preventing execution from meeting its service requirements. |
| Rate or resource exhaustion | Quota, capacity, memory, token, concurrency, or rate limits are exhausted or demonstrably reachable under expected load, impairing execution. |
| Runtime authentication or configuration failure | Invalid or missing credentials, endpoints, model deployments, quota configuration, or other runtime settings prevent execution. |
Safety
Issues with trust, sensitive data, authorized actions, or application safety policies.| Problem label | Description |
|---|---|
| Untrusted-instruction boundary failure | Untrusted or lower-priority content controls behavior across a required trust boundary, or configuration exposes a proven reachable takeover path. |
| Sensitive information disclosure | Sensitive data reaches an unauthorized destination or is unnecessarily accessible through a demonstrated threat path. |
| Excessive or unauthorized agency | An agent can take or takes actions beyond user approval, role authorization, required confirmation, or least-privilege access. |
| Unsafe or disallowed content produced | The system produces content prohibited by the application’s explicit safety or content policy in the evaluated context. |
| Policy or access-control bypass | A required authorization, policy, guardrail, or trust check is bypassed, or configuration provides a reachable way to bypass it. |
| Insecure model/tool output handling | Untrusted output reaches an interpreter, renderer, command, or privileged destination without required validation, encoding, or isolation. |
Efficiency
Avoidable work in tasks that succeed or complete safely, without a compensating quality, safety, or recovery benefit.| Problem label | Description |
|---|---|
| Repeated or duplicate action | A successful, safe task repeats an equivalent completed action without adding needed evidence, state, safety, or recovery value. |
| Retry thrash | A task eventually succeeds or degrades safely but repeats a failing operation beyond its justified retry or backoff policy without useful adaptation. |
| Redundant retrieval | A successful, safe task repeats equivalent retrievals or fetches available evidence without a justified need for freshness, coverage, or verification. |
| Avoidable context bloat | A successful, safe task sends irrelevant, duplicate, or unnecessarily large context that adds token use, cost, or latency. |