Skip to main content
Proactively discover failure patterns across your AI agents that are invisible to evaluators and manual searches. Signals analyzes your production traces to surface security leaks, policy drift, cascading failures, and other issues you might not know to search for. Signals doesn’t wait for you to ask the right question. When requested, Signals sweeps across your agent traces, groups related findings, and prioritizes the patterns worth investigating.

​
Why use it

  • Find hidden patterns: Detect patterns that predefined evaluators and manual searches might miss. Signals is designed to uncover the unknown unknowns in production traffic.
  • Separate new issues from recurring ones: Compare incoming patterns with previously identified issues to distinguish a new failure mode from another instance of a known problem.
  • Track recurring issues: Review recommendations for evaluators, alerts, and trend views to track an issue if it appears again.
  • Understand severity, scale, and timing: View each issue’s category and severity, the number of affected spans and sessions, and a timeline of when affected spans occurred.

​
How Signals works

An agent can fail its task even when its tools and infrastructure work normally. A successful tool call can still take the wrong action, and individually valid calls can repeat work without making progress. Signals assesses the impact shown in the evidence, including failures that occurred, demonstrated risks, and unnecessary work. It also considers how many sessions are affected. Review severity alongside the supporting evidence and recommended actions to decide what to investigate first.

​
Research results

In a research evaluation on recorded agent executions from selected public benchmarks, Signals achieved 91.43% F1 for matching Signals to reference issues, compared with 73.69% for a one-pass LLM control using the same model and taxonomy. F1 combines precision and recall. This is about 18 percentage points higher. Signals achieved 92.90% precision and 90.01% recall in this comparison. The evaluation measured agreement on the affected component and problem type. Severity assessments and proposed fixes require separate validation. For more information, see the research paper Signals: Continuous Diagnostics of AI Agents.

​
Signals severity scores

Signals uses an internal severity score to rank findings. Severity appears in the UI as Low, High, or Critical. The score considers the impact of observed failures, the potential harm from risks and how easily they could occur, and unnecessary resource use. It also considers how many sessions are affected and how common the issue is across the sessions analyzed. The same problem type can have different severity levels depending on its impact and the sessions affected. In addition to severity levels, each signal also includes labels that provide context on the failure. See Signals Labels for more information.

​
How to use Signals

1

Set up an LLM integration

Signals requires access to an LLM reasoning model. Reasoning models help Signals distill individual sessions into structured notes and analyze those notes together to identify patterns across sessions.If you haven’t already set up an LLM integration, see Configure an LLM integration. Select the reasoning model to use when you start a scan or configure a schedule.
2

Open Signals from an Agent Stream

Use the main menu to select an Agent Stream. Select the Signals tab to view the list of signals for the Agent Stream.The Signals tab
3

Run a scan

Select Scan now, choose a Model and Scan window, then select Start scan. You can scan data since the last successful scan, the last 12 hours, the last 7 days, or a custom window.Select Scan history to review previous scans, including their status, the window analyzed, the model used, and the number of signals created or updated.
4

Investigate a signal

Select a signal to investigate it.The Detect tab on a signalThe Detect tab displays the severity level, labels that describe the signal, root cause analysis, and evidence. Signals detects and labels 45 types of agent behavior failures. See Signals Labels for more information.You can use the Sessions section to see the affected component, span, and an example of the issue.
5

Remediate a signal

Select the Remediate tab to view recommended fixes.The Remediate tab on a signal, showing the Fix or prevent sectionThe Fix or prevent section lists recommended changes to address an issue or reduce the chance it happens again. Each recommendation describes the change and labels its type. For supported recommendations, select Fix in your codebase to open a brief for your coding agent.Use the Splunk Agent Observability MCP server integration with your coding agent to act on recommended remediations in your codebase.Select Open in Codex, Open in Cursor, or Copy prompt to provide the signal’s context and remediation instructions to your coding agent. The coding agent can use MCP to retrieve the recommendation and supporting evidence before proposing changes.The Fix in code dialog with coding-agent actions and the MCP setup guideTreat each recommendation as a proposed fix. Ask your coding agent to inspect the relevant code and supporting evidence, confirm that the change fits the intended behavior, and validate the change with appropriate checks.
6

Track an issue from a signal

On the Remediate tab, the Detect recurrence section lists recommended ways to track when an issue recurs. Each recommendation describes what to monitor and labels the suggested method, such as an evaluator, alert, or trend view.The Remediate tab on a signal, showing the Detect recurrence section
7

Investigate with the AI Assistant

Open the AI Assistant with the signal context attached to ask questions to understand what went wrong, identify the next best action, and move from detection to remediation without leaving your workspace.From the list of signals, select the checkbox next to one or more signals and select Ask AI about these.How to select multiple signals to investigate with the AI AssistantFor example, you can ask:
  • “Explain this signal and show me the traces behind it.”
  • “Which signal should I look at first?”
  • “How many traces are affected by this signal?”
  • “What is the likely root cause of this signal?”
  • “What should I change to address this issue?”
  • “Are there similar failures in other sessions?”
See AI Assistant for more information.

​
Configure settings for Signals

To navigate to the Configuration page, use the main menu to select an Agent Stream. Select the Signals tab, then select the gear icon in the upper-right corner:How to access the Configuration page for SignalsThe Configuration page includes the following sections:
  • Schedule: Choose whether Signals runs only when manually triggered or on a schedule.
  • Reasoning model: Select the model Signals uses and review suggested models. Signals configuration with a weekly schedule and reasoning model selected

​
Schedule Signals runs

Only organization admins can manage schedules. Project editors can run manual scans.
  1. On the Configuration page, select Scheduled.
  2. Choose a frequency: Every few hours, Daily, or Weekly. Scans must be at least 12 hours apart.
  3. Select a time zone, start date, and optional end date.
  4. Select a reasoning model.
  5. Select Save configuration.
The first scheduled scan analyzes the previous 30 days. Later scheduled scans analyze one scheduled interval. You can pause scheduled scans without deleting the schedule. Scans that have already started continue running.