Splunk Agent Observability MCP Server

The Splunk Agent Observability Model Context Protocol (MCP) server enables seamless integration between AI-powered IDEs, such as Cursor, or VS Code with GitHub Copilot, and Splunk Agent Observability’s evaluation and observability platform. With MCP, you can access Splunk Agent Observability’s capabilities directly from your development environment, including:

Creating and managing datasets
Getting guidance on setting up and running experiments
Setting up prompt templates
Investigating signals and applying recommended code changes
Integrating Splunk Agent Observability with your code

Configure your IDE

Select the instructions for your deployment.

On-premises
SaaS

Before you begin, ensure you have the following:

AI-enabled IDE

Install an AI-enabled IDE such as Cursor or VS Code with AI capabilities

API key

Generate your Splunk Agent Observability API key from the API keys page in the UI.

The Splunk Agent Observability MCP server works with both Cursor and VS Code. Follow the steps below for your IDE:

VS Code
Cursor

Install GitHub Copilot

Install the GitHub Copilot extension if you haven’t already

Open MCP settings

Open the Command Palette (Ctrl + Shift + P on Windows/Linux, or Cmd + Shift + P on Mac) and search for “MCP: Open User Configuration”

Add the Splunk Agent Observability MCP server configuration

Copy and paste the configuration below. Replace YOUR-API-KEY with your actual Splunk Agent Observability API key and <your-splunk-ao-api-url> with your deployment’s API URL.

VSCode MCP Configuration

{
  "servers": {
    "splunk_ao_mcp_server": {
      "type": "http",
      "url": "<your-splunk-ao-api-url>/mcp/http/mcp",
      "headers": {
        "Splunk-AO-API-Key": "YOUR-API-KEY",
        "Accept": "application/json, text/event-stream"
      }
    }
  },
  "inputs": []
}

If you’re using a self-hosted Splunk Agent Observability deployment, replace the <your-splunk-ao-api-url>/mcp/http/mcp url with your deployment URL. The format of this URL is based on your URL, replacing console with api and appending /mcp/http/mcp.

Reload VS Code

Reload VS Code by opening the Command Palette and running “Developer: Reload Window” for the changes to take effect

Open the MCP server configuration file

Open or create .cursor/mcp.json in your project, or ~/.cursor/mcp.json to make the server available across projects.

Add the Splunk Agent Observability MCP server configuration

Copy and paste the configuration below. Replace YOUR-API-KEY with your actual Splunk Agent Observability API key and <your-splunk-ao-api-url> with your deployment’s API URL.

Cursor MCP Configuration

{
  "mcpServers": {
    "splunk_ao_mcp_server": {
      "url": "<your-splunk-ao-api-url>/mcp/http/mcp",
      "headers": {
        "Splunk-AO-API-Key": "YOUR-API-KEY",
        "Accept": "application/json, text/event-stream"
      }
    }
  }
}

Save and restart

Save the configuration and restart Cursor for the changes to take effect

Use the configuration for your IDE. Make sure to replace YOUR-API-KEY with your actual Splunk Agent Observability API key from the API keys page in the UI.

If you’re using Agent Observability in Splunk Observability Cloud, connect through the Splunk MCP Gateway. See Interact with your observability data using the Splunk MCP server for more information.Before you begin, ensure you have the following:

A coding agent such as Codex, or an AI-enabled IDE such as Cursor or VS Code with GitHub Copilot.
Your Splunk Observability Cloud realm. See View your realm, API endpoints, and organization.
The MCP Gateway region for your realm. See the Supported regions table in Configure Splunk Observability tools with Splunk MCP Server.
A Splunk Observability Cloud user API access token. See Retrieve your user API access token.

Agent Observability tools are not included in the default tool set. Add the X-SPLUNK-O11Y-TOOL-SETS header to select the tools available to your IDE:

Header value	Tools to include
`agent_observability`	Agent Observability tools.
`default,agent_observability`	The default Splunk Observability Cloud tools and Agent Observability tools.

Selecting a tool set doesn’t change your permissions or enable features for your organization.Follow the steps below for your IDE. In each configuration, replace <SCS-REGION> with the MCP Gateway region for your realm, YOUR-REALM with your Splunk Observability Cloud realm, and YOUR-USER-ACCESS-TOKEN with your user API access token.

Codex
VS Code
Cursor

Open the MCP server configuration file

Open or create ~/.codex/config.toml.

Add the Splunk MCP Gateway configuration

Codex SaaS MCP Configuration

[mcp_servers.splunk_mcp_gateway]
url = "https://region-<SCS-REGION>.api.scs.splunk.com/system/mcp-gateway/v1/"

[mcp_servers.splunk_mcp_gateway.http_headers]
"Accept" = "application/json, text/event-stream"
"X-SF-REALM" = "YOUR-REALM"
"X-SF-TOKEN" = "YOUR-USER-ACCESS-TOKEN"
"X-SPLUNK-O11Y-TOOL-SETS" = "agent_observability"

Save and restart

Save the configuration and restart Codex for the changes to take effect. For more information, see Configure Codex with config.toml.

Open MCP settings

Install the GitHub Copilot extension if you haven’t already. Open the Command Palette and run MCP: Open User Configuration.

Add the Splunk MCP Gateway configuration

VS Code SaaS MCP Configuration

{
  "servers": {
    "splunk_mcp_gateway": {
      "type": "http",
      "url": "https://region-<SCS-REGION>.api.scs.splunk.com/system/mcp-gateway/v1/",
      "headers": {
        "Accept": "application/json, text/event-stream",
        "X-SF-REALM": "YOUR-REALM",
        "X-SF-TOKEN": "YOUR-USER-ACCESS-TOKEN",
        "X-SPLUNK-O11Y-TOOL-SETS": "agent_observability"
      }
    }
  },
  "inputs": []
}

Reload VS Code

Save the configuration, then open the Command Palette and run Developer: Reload Window.

Open the MCP server configuration file

Open or create .cursor/mcp.json in your project, or ~/.cursor/mcp.json to make the server available across projects.

Add the Splunk MCP Gateway configuration

Cursor SaaS MCP Configuration

{
  "mcpServers": {
    "splunk_mcp_gateway": {
      "url": "https://region-<SCS-REGION>.api.scs.splunk.com/system/mcp-gateway/v1/",
      "headers": {
        "Accept": "application/json, text/event-stream",
        "X-SF-REALM": "YOUR-REALM",
        "X-SF-TOKEN": "YOUR-USER-ACCESS-TOKEN",
        "X-SPLUNK-O11Y-TOOL-SETS": "agent_observability"
      }
    }
  }
}

Save and restart

Save the configuration and restart Cursor for the changes to take effect.

To include the default Splunk Observability Cloud tools as well, set X-SPLUNK-O11Y-TOOL-SETS to default,agent_observability.

Verify your setup

Once configured, you can verify your MCP setup by asking your AI assistant in your IDE:

Can you show me how to add Splunk Agent Observability logging to my agent bot?

Help me create a synthetic dataset for customer support queries

How do I integrate Splunk Agent Observability with LangChain?

Your AI assistant should now be able to access Splunk Agent Observability’s capabilities and respond with information from your Splunk Agent Observability account.

Tools

The Splunk Agent Observability MCP server provides powerful tools that you can access through natural conversation with your AI assistant. Simply ask questions or make requests, and the AI will use these tools to help you.

Create Datasets

Generate synthetic datasets or upload your own data to test and evaluate your AI applications. The tool supports creating datasets with various types of queries including general queries, prompt injections, off-topic content, and toxic content scenarios.What you can ask:

Create a dataset with 50 customer service queries about billing issues

Generate a dataset of 30 chatbot queries, including some prompt injection attempts

Make a dataset with product recommendation queries and include off-topic questions

Check Dataset Status

Track the progress of your dataset generation and preview the generated content. You’ll see the first 10 rows of data along with generation status and progress updates.What you can ask:

Check the status of my dataset that's currently generating

Show me the preview of dataset abc-123

Is my customer service dataset ready yet?

Create Prompt Templates

Build reusable prompt templates that you can use across all your projects. Set up model configurations, temperature settings, and other parameters for consistent prompt behavior.What you can ask:

Create a prompt template called "Friendly Assistant" for customer support responses

Make a prompt template for summarizing technical documentation with lower temperature

Set up a chat template for a code review assistant

Set Up Experiments

Get complete guidance on setting up and running Splunk Agent Observability experiments, including dataset preparation, evaluators configuration, and integration with your existing code. Available for Python and other supported languages.What you can ask:

How do I set up a Splunk Agent Observability experiment in Python?

Show me how to run an experiment with my RAG application

Guide me through creating an experiment for my agentic workflow

Remediate Signals

Investigate a signal’s recommended code change and supporting evidence from your IDE. Start with the brief generated by Fix in your codebase on the signal’s Remediate tab.Your coding agent can use the following tools:

get_code_remediation: Read the signal’s diagnosis and recommended change.
get_signal_evidence: Retrieve affected occurrences and their evidence references.
get_signal_evidence_context: Read nearby spans for an affected event.
get_signal_evidence_span: Read a referenced span’s input and output.

What you can ask:

Use this Signal brief to review the recommended change and its supporting
evidence, then propose a fix in my codebase.

Show me the evidence behind this recommendation and suggest tests for the fix.

Integrate with OpenAI

Get step-by-step integration guides for adding Splunk Agent Observability to your OpenAI applications. Automatically log prompts, responses, model parameters, and token usage with minimal code changes.What you can ask:

How do I add Splunk Agent Observability logging to my OpenAI application?

Show me how to integrate Splunk Agent Observability with OpenAI

Help me set up Splunk Agent Observability for my GPT-4 chatbot

Integrate with LangChain

Get complete integration instructions for adding Splunk Agent Observability to your LangChain applications. Capture full traces of your chains, agents, and tools with automatic logging.What you can ask:

How do I integrate Splunk Agent Observability with my LangChain application?

Show me how to add Splunk Agent Observability tracing to my LangChain agent

Help me log my LangChain RAG pipeline with Splunk Agent Observability

Search Documentation

Find relevant information, code examples, API references, and implementation guides across all Splunk Agent Observability documentation. Get direct links to the pages you need.What you can ask:

How do I set up data logging in Splunk Agent Observability?

Find documentation about custom evaluators

Search for examples of agentic AI evaluation

Example use cases

Create a synthetic dataset

Ask your AI assistant:

Create a synthetic dataset with 20 customer service queries about product
returns. Include both general queries and off-topic queries.

The MCP server will guide you through the dataset creation process and provide a dataset ID to track progress.

Get integration help

Ask your AI assistant:

How do I integrate Splunk Agent Observability with my LangChain application in Python?

The MCP server will provide complete integration code examples and setup instructions.

Remediate a signal in your codebase

Open a signal in your Agent Stream and select the Remediate tab.
For a recommendation that supports code changes, select Fix in your codebase.
Select Open in Codex, Open in Cursor, or Copy prompt to provide the brief to your coding agent.
Connect your coding agent to the MCP server for the deployment that hosts the signal.

The brief includes the signal’s context, remediation instructions, and references to supporting evidence. Your coding agent can retrieve the evidence through MCP, inspect your codebase, and propose changes and tests. For more information, see Signals.

Troubleshooting

Error connecting to Splunk Agent Observability MCP Server

For on-premises deployments, check that the MCP server URL is set to <your-splunk-ao-api-url>/mcp/http/mcp.
For SaaS, check that the MCP Gateway URL uses the region for your Splunk Observability Cloud realm and ends with /system/mcp-gateway/v1/.
Ensure the Accept header is set to application/json, text/event-stream
Restart your IDE MCP connection after making configuration changes

Signals evidence tools are unavailable

Confirm that your MCP connection points to the deployment that hosts the signal.
Confirm that Signals is available in that deployment.
For SaaS, include agent_observability in the X-SPLUNK-O11Y-TOOL-SETS header. Use default,agent_observability to include both tool sets.
Refresh your IDE’s MCP connection to reload the available tools.

On-premises API key errors

Confirm your API key is properly set in the configuration
Check that your API key has not expired
Generate a new API key from the API keys page in the UI

SaaS authentication errors

Confirm that X-SF-REALM matches your Splunk Observability Cloud realm.
Confirm that X-SF-TOKEN contains a valid user API access token for that realm.
If the token has expired, retrieve a new token and update your configuration.

Next steps

Log your first trace

Learn how to log your first trace with Splunk Agent Observability

Run your first experiment

Set up and run experiments to evaluate your AI applications

Explore integrations

Learn about Splunk Agent Observability integrations with third-party frameworks

Overview

SaaS deployments

Get Started

Observability

Evaluators

Tokenomics

Signals

AI Assistant

Luna Studio

Experiments

Agent Control

Annotations

Integrations

Security

References

Configure your IDE

Verify your setup

Tools

Example use cases

Create a synthetic dataset

Get integration help

Remediate a signal in your codebase

Troubleshooting

Next steps

Log your first trace

Run your first experiment

Explore integrations

​Configure your IDE

​Verify your setup

​Tools

​Example use cases

​Create a synthetic dataset

​Get integration help

​Remediate a signal in your codebase

​Troubleshooting

​Next steps

Log your first trace

Run your first experiment

Explore integrations

Configure your IDE

Verify your setup

Tools

Example use cases

Create a synthetic dataset

Get integration help

Remediate a signal in your codebase

Troubleshooting

Next steps