> ## Documentation Index
> Fetch the complete documentation index at: https://agent-observability-docs.splunk.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Signals

> Proactively discover failure patterns across your AI agents that evaluators and manual searches might miss.

Proactively discover failure patterns across your AI agents that are invisible to evaluators and manual searches. Signals analyzes your production traces to surface security leaks, policy drift, cascading failures, and other issues you might not know to search for.

Signals doesn't wait for you to ask the right question. When requested, Signals sweeps across your agent traces, groups related findings, and prioritizes the patterns worth investigating.

## Why use it

* **Find hidden patterns:** Detect patterns that predefined evaluators and manual searches might miss. Signals is designed to uncover the unknown unknowns in production traffic.
* **Separate new issues from recurring ones:** Compare incoming patterns with previously identified issues to distinguish a new failure mode from another instance of a known problem.
* **Track recurring issues:** Review recommendations for evaluators, alerts, and trend views to track an issue if it appears again.
* **Understand severity, scale, and timing:** View each issue's category and severity, the number of affected spans and sessions, and a timeline of when affected spans occurred.

## How Signals works

An agent can fail its task even when its tools and infrastructure work normally. A successful tool call can still take the wrong action, and individually valid calls can repeat work without making progress.

Signals assesses the impact shown in the evidence, including failures that occurred, demonstrated risks, and unnecessary work. It also considers how many sessions are affected. Review severity alongside the supporting evidence and recommended actions to decide what to investigate first.

### Research results

In a research evaluation on recorded agent executions from selected public benchmarks, Signals achieved **91.43% F1 for matching Signals to reference issues**, compared with **73.69%** for a one-pass LLM control using the same model and taxonomy. F1 combines precision and recall. This is about **18 percentage points higher**.

Signals achieved **92.90% precision** and **90.01% recall** in this comparison. The evaluation measured agreement on the affected component and problem type. Severity assessments and proposed fixes require separate validation.

For more information, see the research paper [Signals: Continuous Diagnostics of AI Agents](https://arxiv.org/abs/2611.04507).

## Signals severity scores

Signals uses an internal severity score to rank findings. Severity appears in the UI as **Low**, **High**, or **Critical**.

The score considers the impact of observed failures, the potential harm from risks and how easily they could occur, and unnecessary resource use. It also considers how many sessions are affected and how common the issue is across the sessions analyzed.

| Severity level | Description |
| - | - |
| **Critical** | Severe consequences, readily reachable risks of severe harm, or severe or persistent resource waste. |
| **High** | Failures, risks, or unnecessary work that can materially affect task results, cost, or response time. |
| **Low** | Smaller observed effects, limited resource waste, or risks that are less harmful or harder to trigger. |

The same problem type can have different severity levels depending on its impact and the sessions affected.

In addition to severity levels, each signal also includes labels that provide context on the failure. See [Signals Labels](https://agent-observability-docs.splunk.com/concepts/signals/signals-labels.md) for more information.

## How to use Signals

> [tab] On-premises
>
> **1. Set up an LLM integration**
>
> Signals requires access to an LLM reasoning model. Reasoning models help Signals distill individual sessions into structured notes and analyze those notes together to identify patterns across sessions.
>
> If you haven't already set up an LLM integration, see [Configure an LLM integration](https://agent-observability-docs.splunk.com/getting-started/experiments.md#prerequisite-configure-an-llm-integration). Select the reasoning model to use when you start a scan or configure a schedule.
>
> **2. Open Signals from an Agent Stream**
>
> Use the main menu to select an Agent Stream. Select the **Signals** tab to view the list of signals for the Agent Stream.
>
> ![The Signals tab](https://mintcdn.com/agent-observability-docs/WRlhvBOnGs9cmQIv/images/concepts/signals/signals-tab-onprem.png?fit=max\&auto=format\&n=WRlhvBOnGs9cmQIv\&q=85\&s=f2007332341ec7abbf4d076ef1e992dd)
>
> **3. Run a scan**
>
> Select **Scan now**, choose a **Model** and **Scan window**, then select **Start scan**. You can scan data since the last successful scan, the last 12 hours, the last 7 days, or a custom window.
>
> Select **Scan history** to review previous scans, including their status, the window analyzed, the model used, and the number of signals created or updated.
>
> **4. Investigate a signal**
>
> Select a signal to investigate it.
>
> ![The Detect tab on a signal](https://mintcdn.com/agent-observability-docs/WRlhvBOnGs9cmQIv/images/concepts/signals/signals-tab-detail-onprem.png?fit=max\&auto=format\&n=WRlhvBOnGs9cmQIv\&q=85\&s=08e52a407c3dafd2d49f96fded003af1)
>
> The **Detect** tab displays the severity level, labels that describe the signal, root cause analysis, and evidence. Signals detects and labels 45 types of agent behavior failures. See [Signals Labels](https://agent-observability-docs.splunk.com/concepts/signals/signals-labels.md) for more information.
>
> You can use the **Sessions** section to see the affected component, span, and an example of the issue.
>
> **5. Remediate a signal**
>
> Select the **Remediate** tab to view recommended fixes.
>
> ![The Remediate tab on a signal, showing the Fix or prevent section](https://mintcdn.com/agent-observability-docs/WRlhvBOnGs9cmQIv/images/concepts/signals/signals-remediate-fix.png?fit=max\&auto=format\&n=WRlhvBOnGs9cmQIv\&q=85\&s=abb5cbfa410eff403a2527423392cf55)
>
> The **Fix or prevent** section lists recommended changes to address an issue or reduce the chance it happens again. Each recommendation describes the change and labels its type. For supported recommendations, select **Fix in your codebase** to open a brief for your coding agent.
>
> Use the [Splunk Agent Observability MCP server integration](https://agent-observability-docs.splunk.com/getting-started/mcp/setup-splunk-ao-mcp.md) with your coding agent to act on recommended remediations in your codebase.
>
> Select **Open in Codex**, **Open in Cursor**, or **Copy prompt** to provide the signal's context and remediation instructions to your coding agent. The coding agent can use MCP to retrieve the recommendation and supporting evidence before proposing changes.
>
> ![The Fix in code dialog with coding-agent actions and the MCP setup guide](https://mintcdn.com/agent-observability-docs/WRlhvBOnGs9cmQIv/images/concepts/signals/signals-fix-in-code-onprem.png?fit=max\&auto=format\&n=WRlhvBOnGs9cmQIv\&q=85\&s=9060bc46b971c236ec579a6c49acd38b)
>
> Treat each recommendation as a proposed fix. Ask your coding agent to inspect the relevant code and supporting evidence, confirm that the change fits the intended behavior, and validate the change with appropriate checks.
>
> **6. Track an issue from a signal**
>
> On the **Remediate** tab, the **Detect recurrence** section lists recommended ways to track when an issue recurs. Each recommendation describes what to monitor and labels the suggested method, such as an evaluator, alert, or trend view.
>
> ![The Remediate tab on a signal, showing the Detect recurrence section](https://mintcdn.com/agent-observability-docs/WRlhvBOnGs9cmQIv/images/concepts/signals/signals-remediate-detect.png?fit=max\&auto=format\&n=WRlhvBOnGs9cmQIv\&q=85\&s=96652619f5e6005676c95012febed23b)
>
> **7. Investigate with the AI Assistant**
>
> Open the AI Assistant with the signal context attached to ask questions to understand what went wrong, identify the next best action, and move from detection to remediation without leaving your workspace.
>
> From the list of signals, select the checkbox next to one or more signals and select **Ask AI about these**.
>
> ![How to select multiple signals to investigate with the AI Assistant](https://mintcdn.com/agent-observability-docs/WRlhvBOnGs9cmQIv/images/concepts/signals/signals-ai-assistant-onprem.png?fit=max\&auto=format\&n=WRlhvBOnGs9cmQIv\&q=85\&s=6ad26e6096b256347276712a853e938c)
>
> For example, you can ask:
>
> * "Explain this signal and show me the traces behind it."
> * "Which signal should I look at first?"
> * "How many traces are affected by this signal?"
> * "What is the likely root cause of this signal?"
> * "What should I change to address this issue?"
> * "Are there similar failures in other sessions?"
>
> See [AI Assistant](https://agent-observability-docs.splunk.com/concepts/ai-assistant.md) for more information.

> [tab] SaaS
>
> **1. Set up an LLM integration**
>
> Signals requires access to an LLM reasoning model. Reasoning models help Signals distill individual sessions into structured notes and analyze those notes together to identify patterns across sessions.
>
> If you haven't already set up an LLM integration, see [Configure an LLM integration](https://agent-observability-docs.splunk.com/getting-started/experiments.md#prerequisite-configure-an-llm-integration). Select the reasoning model to use when you start a scan or configure a schedule.
>
> **2. Open Signals from an Agent Stream**
>
> From the Splunk Observability Cloud main menu, select **Agent Observability** > **All projects**. Select a project, then select an Agent Stream.
>
> Within an Agent Stream, select the **Signals** tab to view the list of signals for the Agent Stream.
>
> ![The Signals tab](https://mintcdn.com/agent-observability-docs/WRlhvBOnGs9cmQIv/images/concepts/signals/signals-tab-saas.png?fit=max\&auto=format\&n=WRlhvBOnGs9cmQIv\&q=85\&s=59a04317640882d3e31b7b32f488c33d)
>
> **3. Run a scan**
>
> Select **Scan now**, choose a **Model** and **Scan window**, then select **Start scan**. You can scan data since the last successful scan, the last 12 hours, the last 7 days, or a custom window.
>
> Select **Scan history** to review previous scans, including their status, the window analyzed, the model used, and the number of signals created or updated.
>
> **4. Investigate a signal**
>
> Select a signal to investigate it.
>
> ![The Detect tab on a signal](https://mintcdn.com/agent-observability-docs/WRlhvBOnGs9cmQIv/images/concepts/signals/signals-tab-detail-saas.png?fit=max\&auto=format\&n=WRlhvBOnGs9cmQIv\&q=85\&s=74296ad80dabccc1d21f89cc9c6f1cb7)
>
> The **Detect** tab displays the severity level, labels that describe the signal, root cause analysis, and evidence. Signals detects and labels 45 types of agent behavior failures.
>
> You can use the **Sessions** section to see the affected component, span, and an example of the issue.
>
> **5. Remediate a signal**
>
> Select the **Remediate** tab to view recommended fixes.
>
> ![The Remediate tab on a signal, showing the Fix or prevent section](https://mintcdn.com/agent-observability-docs/WRlhvBOnGs9cmQIv/images/concepts/signals/signals-remediate-fix-saas.png?fit=max\&auto=format\&n=WRlhvBOnGs9cmQIv\&q=85\&s=bf72b962bfc47e0b3ad07ded813edecb)
>
> The **Fix or prevent** section lists recommended changes to address an issue or reduce the chance it happens again. Each recommendation describes the change and labels its type. For supported recommendations, select **Fix in your codebase** to open a brief for your coding agent.
>
> The following example shows a recommended prompt change:
>
> ![A prompt change recommendation in the Fix or prevent section](https://mintcdn.com/agent-observability-docs/WRlhvBOnGs9cmQIv/images/concepts/signals/signals-remediate-prompt-saas.png?fit=max\&auto=format\&n=WRlhvBOnGs9cmQIv\&q=85\&s=bf69e52c14762d4e8cf9e040f2967dd8)
>
> Use the **SaaS** instructions in the [Splunk Agent Observability MCP server integration guide](https://agent-observability-docs.splunk.com/getting-started/mcp/setup-splunk-ao-mcp.md) to connect your coding agent and enable the Agent Observability tool set.
>
> Select **Open in Codex**, **Open in Cursor**, or **Copy prompt** to provide the signal's context and remediation instructions to your coding agent. The coding agent can use MCP to retrieve the recommendation and supporting evidence before proposing changes.
>
> ![The Fix in code dialog with coding-agent actions and the Splunk MCP setup guide](https://mintcdn.com/agent-observability-docs/WRlhvBOnGs9cmQIv/images/concepts/signals/signals-fix-in-code-saas.png?fit=max\&auto=format\&n=WRlhvBOnGs9cmQIv\&q=85\&s=1b716419891e36c9a97b52088b988c44)
>
> Treat each recommendation as a proposed fix. Ask your coding agent to inspect the relevant code and supporting evidence, confirm that the change fits the intended behavior, and validate the change with appropriate checks.
>
> **6. Track an issue from a signal**
>
> On the **Remediate** tab, the **Detect recurrence** section lists recommended ways to track when an issue recurs. Each recommendation describes what to monitor and labels the suggested method, such as an evaluator, alert, or trend view.

## Configure settings for Signals

> [tab] On-premises
>
> To navigate to the **Configuration** page, use the main menu to select an Agent Stream. Select the **Signals** tab, then select the gear icon in the upper-right corner:
>
> ![How to access the Configuration page for Signals](https://mintcdn.com/agent-observability-docs/WRlhvBOnGs9cmQIv/images/concepts/signals/signals-configuration-icon.png?fit=max\&auto=format\&n=WRlhvBOnGs9cmQIv\&q=85\&s=170937a8b6e36e6024e36c56495e2d66)
>
> The **Configuration** page includes the following sections:
>
> * **Schedule**: Choose whether Signals runs only when manually triggered or on a schedule.
> * **Reasoning model**: Select the model Signals uses and review suggested models.
>
>   ![Signals configuration with a weekly schedule and reasoning model selected](https://mintcdn.com/agent-observability-docs/WRlhvBOnGs9cmQIv/images/concepts/signals/signals-configuration-scheduled-onprem.png?fit=max\&auto=format\&n=WRlhvBOnGs9cmQIv\&q=85\&s=71b99228019c0d178b2269fe9d536494)

> [tab] SaaS
>
> To navigate to the **Configuration** page:
>
> 1. From the Splunk Observability Cloud main menu, select **Agent Observability** > **All projects**.
> 2. Select a project, then select an Agent Stream.
> 3. Select the **Signals** tab, then select the gear icon in the upper-right corner.
>
>    ![How to access the Configuration page for Signals](https://mintcdn.com/agent-observability-docs/WRlhvBOnGs9cmQIv/images/concepts/signals/signals-configuration-icon-saas.png?fit=max\&auto=format\&n=WRlhvBOnGs9cmQIv\&q=85\&s=2b39ce6f5be759c4c7d3a9b44d7415a6)
>
> The **Configuration** page includes the following sections:
>
> * **Schedule**: Choose whether Signals runs only when manually triggered or on a schedule.
> * **Reasoning model**: Select the model Signals uses and review suggested models.
>
>   ![Signals configuration with a paused weekly schedule and reasoning model selected](https://mintcdn.com/agent-observability-docs/WRlhvBOnGs9cmQIv/images/concepts/signals/signals-configuration-scheduled-saas.png?fit=max\&auto=format\&n=WRlhvBOnGs9cmQIv\&q=85\&s=ca91acd7ef1e60008bb6a81f36b7e1f1)

### Schedule Signals runs

Only organization admins can manage schedules. Project editors can run manual scans.

1. On the **Configuration** page, select **Scheduled**.
2. Choose a frequency: **Every few hours**, **Daily**, or **Weekly**. Scans must be at least 12 hours apart.
3. Select a time zone, start date, and optional end date.
4. Select a reasoning model.
5. Select **Save configuration**.

The first scheduled scan analyzes the previous 30 days. Later scheduled scans analyze one scheduled interval.

You can pause scheduled scans without deleting the schedule. Scans that have already started continue running.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.