Overview

The Splunk Agent Observability SDK provides a comprehensive set of tools for logging, evaluating, and experimenting with LLM applications. Regardless of how you go about logging your AI application, you will still need to install the Splunk Agent Observability SDK and initialize your API keys by following the steps below.

Python SDK

The Splunk Agent Observability Python SDK on PyPI.

Installation

pip install splunk-ao

uv pip install splunk-ao

poetry add splunk-ao

(Optional) Install the integration dependencies used by your application:

pip install "splunk-ao[openai]"
pip install "splunk-ao[langchain]" langchain-openai
pip install "splunk-ao[crewai]"
pip install "splunk-ao[middleware]"
pip install "splunk-ao[all]"

uv pip install "splunk-ao[openai]"
uv pip install "splunk-ao[langchain]" langchain-openai
uv pip install "splunk-ao[crewai]"
uv pip install "splunk-ao[middleware]"
uv pip install "splunk-ao[all]"

poetry add "splunk-ao[openai]"
poetry add "splunk-ao[langchain]" langchain-openai
poetry add "splunk-ao[crewai]"
poetry add "splunk-ao[middleware]"
# Install all optional dependencies
poetry add "splunk-ao[all]"

Initialization and authentication

These steps vary depending on your deployment type. Select the tab for your deployment type below to view the documentation.

On-premises
SaaS

This section only applies to on-premises, standalone, and custom deployments. The following environment variables are used for these deployments:

On-premises environment variable	Required	Description
`SPLUNK_AO_API_KEY`	Yes	Create the API key in the UI, then set it as this environment variable.
`SPLUNK_AO_CONSOLE_URL`	Yes	Set this to the URL of your deployment, which appears in the browser when you log in to Splunk Agent Observability. The SDK derives the API endpoint from this deployment URL.
`SPLUNK_AO_API_URL`	Yes (conditional)	Only required for deployments that use a separate API URL that cannot be derived from the deployment URL.
`SPLUNK_AO_LOGGING_DISABLED`	No	Set this to `true` to deactivate telemetry collection and export.
`SPLUNK_AO_PROJECT`	No	The Splunk Agent Observability project to log to. If this is not set, you will need to pass the project name in code.
`SPLUNK_AO_AGENT_STREAM`	No	The default Agent Stream to log to. If this is not set, you will need to pass the Agent Stream name in code.

To set up the Python SDK for an on-premises, standalone, or custom deployment:

Set your API key and deployment URL

Set your Agent Observability API key and deployment URL:

export SPLUNK_AO_API_KEY="<your-agent-observability-api-key>"
export SPLUNK_AO_CONSOLE_URL="<your-deployment-url>"

(Optional) Set a separate API URL

Set SPLUNK_AO_API_URL only if your deployment uses a separate API URL that cannot be derived from the deployment URL:

export SPLUNK_AO_API_URL="<your-api-url>"

(Optional) Set your project and Agent Stream

Set the following environment variables to define the project and Agent Stream that Splunk Agent Observability should use. You can also provide these names through the SDK later.

export SPLUNK_AO_PROJECT="<your-project-name>"
export SPLUNK_AO_AGENT_STREAM="<your-agent-stream-name>"

This section only applies to SaaS deployments integrated with Splunk Observability Cloud. The following environment variables are used for SaaS deployments:

SaaS environment variable	Required	Description
`SPLUNK_AO_REALM`	Yes	The realm for your deployment, such as `us1`. The SDK derives the console, API, and OTLP ingest endpoints from the realm.
`SPLUNK_AO_O11Y_TOKEN`	Yes	The Splunk Observability Cloud access token used for OTLP export. This token is required to export telemetry. This token is used for CRUD operations when it contains the necessary API permissions and no dedicated API token is configured.
`SPLUNK_AO_O11Y_API_TOKEN`	No	A separate and dedicated token for CRUD operations. When both `SPLUNK_AO_O11Y_API_TOKEN` and `SPLUNK_AO_O11Y_TOKEN` are set, `SPLUNK_AO_O11Y_API_TOKEN` is used for CRUD and `SPLUNK_AO_O11Y_TOKEN` is used for telemetry ingestion. For CRUD-only use, you can set `SPLUNK_AO_REALM` and `SPLUNK_AO_O11Y_API_TOKEN` without setting `SPLUNK_AO_O11Y_TOKEN`.
`OTEL_SERVICE_NAME`	No	The OpenTelemetry service name. Set this when you want traces to appear under a recognizable service in Splunk Application Performance Monitoring (APM). If this environment variable is absent, the standard OpenTelemetry unknown-service fallback is used. See the OpenTelemetry specification for more details.
`SPLUNK_AO_LOGGING_DISABLED`	No	Set this to `true` to deactivate telemetry collection and export.
`SPLUNK_AO_PROJECT`	No	Defines the Splunk Agent Observability project to log to. If this is not set, you will need to pass the project name in code.
`SPLUNK_AO_AGENT_STREAM`	No	Defines the default Agent Stream to log to. If this is not set, you will need to pass the Agent Stream name in code.

To set up the Python SDK for a SaaS deployment:

Create an access token

You must be an admin to perform this step. If you don’t have an existing access token, create one in Splunk Observability Cloud:

In the Splunk Observability Cloud main menu, select Settings > Access tokens.
Under Authorization and capability scope:
- Check the INGEST token box.
- Check the API token with roles box, then assign it the agent_observability_admin role. For example:
Proceed with the on-screen instructions to create the access token.
After you create the access token, select the token Name from the table.
Copy the Token Secret.

Set your realm and access token

Set your Splunk Observability Cloud realm and access token for telemetry export:

export SPLUNK_AO_REALM="us1"
export SPLUNK_AO_O11Y_TOKEN="your-o11y-access-token"

(Optional) Configure a separate CRUD token

Configure a separate token for CRUD operations:

export SPLUNK_AO_O11Y_API_TOKEN="<your-splunk-api-token>"

(Optional) Set your project and Agent Stream

Set the following environment variables to define the project and Agent Stream that Splunk Agent Observability should use. You can also provide these names through the SDK later.

export SPLUNK_AO_PROJECT="<your-project-name>"
export SPLUNK_AO_AGENT_STREAM="<your-agent-stream-name>"

Logging

The Splunk Agent Observability SDKs allow you to log all prompts, responses, and statistics around your LLM usage. There are three main ways to log your application:

Use a third-party integration - use wrappers that integrate with common SDKs to automatically log LLM calls or agentic workflows.
Use a decorator - by decorating a function that calls an LLM with the @log decorator or log wrapper, the Splunk Agent Observability SDK logs all AI prompts within.
Directly using the SplunkAOLogger class - For more control over your logging, you can use the SplunkAOLogger directly. This allows you to manually create sessions, start traces, and log spans. This can be mixed with the other methods, for example accessing the logger directly inside a decorated function call to manually add spans.

Log experiments

Experiments are logged automatically when they are run, but you can use these same SDK concepts inside the code being run by your experiment for greater control and additional logging. This allows you to not only create distinct experiments, such as in notebooks, but to also add experiments to your production application code. See our run experiments with code documentation for more details.

Next steps

Logging with the SDKs

Learn how to log experiments

Learn how to run experiments with multiple data points using datasets and prompt templates

Splunk Agent Observability logger

Log with full control over sessions, traces, and spans using the Splunk Agent Observability logger.

Log decorator

Quickly add logging to your code with the log decorator and wrapper.

Splunk Agent Observability context

Manage logging using the Splunk Agent Observability context manager.

How-to guides

Log Using the OpenAI Wrapper

Learn how to integrate and use OpenAI’s API with Splunk Agent Observability’s wrapper client.
Python

Log Using the @log Decorator

Learn how to use the Splunk Agent Observability @log decorator to log functions to traces
Python

Python SDK

API

Overview

Python SDK

Installation

Initialization and authentication

Logging

Log experiments

Next steps

Logging with the SDKs

Learn how to log experiments

Splunk Agent Observability logger

Log decorator

Splunk Agent Observability context

How-to guides

Log Using the OpenAI Wrapper

Log Using the @log Decorator

Create Traces and Spans

SDK reference

Python SDK Reference

Python SDK

​Installation

​Initialization and authentication

​Logging

​Log experiments

​Next steps

​Logging with the SDKs

Learn how to log experiments

Splunk Agent Observability logger

Log decorator

Splunk Agent Observability context

​How-to guides

Log Using the OpenAI Wrapper

Log Using the @log Decorator

Create Traces and Spans

​SDK reference

Python SDK Reference

Installation

Initialization and authentication

Logging

Log experiments

Next steps

Logging with the SDKs

How-to guides

SDK reference