Set Up Alerts on Logs

​

Overview

• Your cost is higher than expected
• Your model is hallucinating more than you want
• Users are entering foul language into your app

​

Example alerts

• an evaluator e.g. Cost, Correctness, Context Adherence
• an aggregation function: Average, Minimum, Maximum, Count, Sum
• a threshold e.g. < 0.5
• a time window e.g. 1 hour

• Exceeding costs: If you want to get alerted with an uptick in cost (above $100/day), select Cost Sum > 100 in the last day
• Hallucinations: If you want to get alerted when there’s an extreme hallucination, select Correctness or Context Adherence Count = 1 for values = 0 in the last 15 minutes.
• Hallucination average: If you want to get alerted when hallucinations are probable (e.g. more than 50% below perfect threshold), select Correctness or Context Adherence Average < 0.5 in the last 1 hour.

​

Email notifications

​

Slack notifications

1. Create a Slack app. If you don’t have an existing manifest file, choose the “From scratch” option to create the app.
2. Pick an App Name like “Splunk Agent Observability Alerts” that will help identify the Slack app that the notifications will come from.
3. Go to the “Incoming Webhooks” page of your Slack app (under “Features”), and enable the toggle to “Activate Incoming Webhooks”.
4. Click “Add New Webhook” and choose the Slack channel you’d like Splunk Agent Observability’s Alerts to go to. (You can also test the webhook by direct messaging your user account.)
5. Copy the generated webhook URL from Slack. Navigate to your Agent Stream, select the Alerts tab, and paste the URL into the Slack Notification section.
6. In the “Notes” section, add the name of the channel that will receive notifications.
7. Try sending a test message to webhook to verify the connection.

​

Generic webhook notifications

​

Configure the webhook

1. In the Webhook (Generic) section, enter the destination URL of your endpoint.
2. Optionally, add a Name to identify this webhook (for example, PagerDuty - Platform Team).
3. Select an Auth type:
   • None — no authentication headers are added.
   • Header token — adds an Authorization: Bearer <token> header. Enter your token in the Token field.
   • HTTP Basic — adds a Basic auth header. Enter your Username and Password.
   The token or password is write-only. Splunk Agent Observability never returns it after saving. If you edit the webhook later, you will need to re-enter the credential to save changes.
4. Optionally, add Metadata key-value pairs. These are forwarded unchanged in the metadata field of every payload Splunk Agent Observability sends to this endpoint. Use this to attach routing or context information your downstream system needs — for example:

   Key	Value
   team	platform
   env	production
   service	my-llm-app

   Your system receives these fields as-is inside the payload, so you can use them to route alerts, filter noise, or set incident priority without having to parse the alert content.
5. Click Add to save the webhook.

​

Test the connection

​

Webhook payload

{
  "version": "1.0",
  "event": "alert.triggered",
  "event_id": "a1b2c3d4-...",
  "timestamp": "2024-01-15T10:30:00Z",
  "alert": {
    "id": "alert-id",
    "name": "High hallucination rate",
    "status": "triggered",
    "previous_status": "healthy"
  },
  "scope": {
    "org_id": "your-org-id",
    "project_id": "your-project-id",
    "project_name": "My Project",
    "agent_stream_id": "your-agent-stream-id",
    "agent_stream_name": "production"
  },
  "conditions": [
    {
      "metric": "metrics/context_adherence",
      "aggregation": "Average",
      "operator": "less than",
      "threshold": 0.5,
      "observed_value": 0.34
    }
  ],
  "dedup_key": "alert-id:project-id:agent-stream-id",
  "deep_link": "<your-splunk-ao-url>/project/.../agent-streams/...",
  "metadata": {
    "team": "platform",
    "env": "production",
    "service": "my-llm-app"
  }
}

• conditions[].observed_value — the actual evaluator value at the time of the alert, not just the threshold. Use this to set incident severity or include it in notification copy.
• dedup_key — stable identifier for this alert instance. Use it to correlate a trigger event with its corresponding clear event, or to avoid duplicate re-deliveries.
• deep_link — a direct link into the Splunk Agent Observability UI scoped to the time window and filters active when the alert fired.
• metadata — the key-value pairs you configured on the webhook. Forwarded unchanged so your system can route or annotate without parsing alert content.
• version — always "1.0". Will be incremented if the payload structure changes in a breaking way.