Skip to main content

​
Overview

This tutorial will guide you through how to monitor and investigate controls in Splunk Agent Observability. By the end of this guide, you’ll know how to monitor the real-time control activity of all Agent Streams or drill down to understand how a control executed in a specific trace in an Agent Stream.

​
Before you begin

This tutorial assumes that you have completed the following procedures:
  1. Install the dependencies for Agent Control
  2. Create a control
  3. Initialize and configure Agent Control

​
Monitor controls using charts in the Control View

1

Open the Agent Stream in the Splunk Agent Observability UI

Navigate to your homepage and select your Agent Stream from the main menu.
2

Navigate to the Control View

Select the Trends tab. Select Default View, then select Control View from the drop-down menu.
3

Use the Controls charts to monitor real-time control activity across all agents

Agent Control chartsYou can use the following charts to verify controls are working, spot patterns that need tuning, and audit which controls triggered on which agent. Use the settings above the charts to update or filter the charts based on time range, groups of controls, or specific controls.
  • Activity trend: The number of controls that executed in the selected time range, including deleted controls.
  • Actions distribution: The distribution of actions that resulted when a control executed over the selected time range. When the evaluator detects that a query matches your control criteria, it directs your agent to perform one of the following actions:
    • deny: The agent blocks the request.
    • steer: The agent modifies or redirects its behavior based on the instructions defined in the steering context of your control configuration. This action can include exception handling.
    • observe: The agent executes the request. Depending on your control criteria, the agent may also log the request or raise a warning. This umbrella action includes the allow, log, and warn actions.
    • Match rate: The number of controls that matched or did not match a query over the selected time range. A match occurs when the evaluator’s confidence score meets or exceeds the threshold set in your control configuration.
    • Check stages: The number of controls that executed in the pre or post mode. See Scope for more information on these modes.
    • Applies to: The number of executed controls that were configured for LLMs or tools.

​
Investigate how a control executed in a trace from the Traces tab

1

Open the Agent Stream in the Splunk Agent Observability UI

Navigate to your homepage and select your Agent Stream from the main menu. The Traces tab displays by default.
2

Navigate to the Messages view and drill down into a span

Select a trace from the table to navigate to the Messages view. Select a control span, identified by the shield icon before the name of the span, to drill down for more information.
3

Use the Controls tab to investigate how the control executed

Controls tab in the span detail viewThe Controls tab displays more information about the control that was triggered for the span. You can use this tab to view the evaluator name, whether the request matched the control criteria, and the confidence level of the match.Select View control to navigate to the configuration page for the control clone attached to the Agent Stream.