FAQ - Splunk Agent Observability

General and core concepts

What is Splunk Agent Observability?

Splunk Agent Observability is an end-to-end platform for evaluating and improving the performance of generative AI models. It provides tools to help teams deeply understand model behavior across tasks like text generation, summarization, classification, and more. Splunk Agent Observability integrates seamlessly into your LLM pipeline and supports evaluation with both human feedback and automated evaluators. View the Splunk Agent Observability platform overview →

How does Splunk Agent Observability help with generative AI evaluation?

Splunk Agent Observability enables teams to evaluate generative models through:

Automatic and custom evaluators: Evaluate and track instruction adherence, correctness, safety & compliance, chunk utilization, and more.
Human feedback integration: Annotate and label logs through the UI or SDK.
Fine-grained error analysis: Detect generation quality and systematically log responses as CLLF, ILLF, IHFA, etc.
Multi-model comparison: Evaluate different model outputs side by side.

This helps identify where your model is strong or weak and speeds up iteration cycles. Learn more about how Splunk Agent Observability helps →

How is evaluating generative AI models different from traditional ML evaluation?

Unlike traditional ML, where outputs are often single values or labels, generative AI outputs are open-ended and nuanced (e.g. full text). This creates challenges like:

Subjectivity: There’s often no single correct answer.
Evaluation complexity: Evaluators must handle variation, style, relevance, and factual accuracy.
Human oversight: Required to understand intent and impact of generations.

Splunk Agent Observability addresses these differences with custom workflows tailored to generative AI. Learn more about Splunk Agent Observability’s evaluation approach →

How do I get started with Splunk Agent Observability to evaluate my AI models?

Visit the Getting Started guide and follow along to create your first Splunk Agent Observability application. Then, you can start incorporating Splunk Agent Observability into your AI project pipeline. Getting Started guide →

How does Splunk Agent Observability fit into the generative AI development lifecycle?

Splunk Agent Observability supports every stage of generative AI development:

Prompt & data experimentation: Evaluate prompt effectiveness.
Model comparison: Choose the best model or fine-tuned variant.
Fine-tuning oversight: Identify when training introduces regressions.
Pre-deployment checks: Surface edge cases, hallucinations, or bias.
Post-deployment monitoring: Continuously analyze performance and drift.

It becomes your team’s co-pilot in delivering high-quality AI products.

SDK usage

What SDKs does Splunk Agent Observability provide?

Splunk Agent Observability provides a Python SDK designed to integrate with your generative AI pipelines. They allow you to log prompts, model generations, evaluators, and feedback directly from your application. Install the Splunk Agent Observability SDK using the following terminal command:

pip install splunk-ao

Python SDK installation instructions →

How do I log prompt and response data using the SDK?

You can log AI prompt and response data by recording logs to an Agent Stream using the Splunk Agent Observability Logger. By initializing a new Trace, the AI activity executed in the subsequent code is recorded to the selected Project’s Agent Stream.

## Include SplunkAOLogger in imports
from splunk_ao import SplunkAOLogger

## Initialize logger and select project and Agent Stream
logger = SplunkAOLogger(project="Science Exploration", agent_stream="laws-of-science")

## Initialize a new trace and start listening for logs to add to it
prompt = 'Write a short story about a magic pizza.'
trace = logger.start_trace(input=prompt)

The logged data is viewable in the Splunk Agent Observability UI for further evaluation. More on logging data →

How do I use the Python SDK in a Python notebook (e.g. Jupyter, Google Colab, etc.)?

When using a Python Notebook (such as Jupyter, Google Colab, etc.), you should use the splunk_ao_context and make sure to call splunk_ao_context.flush() at the end of your notebook.

import os
from splunk_ao import splunk_ao_context
from splunk_ao.openai import openai

splunk_ao_context.init(project="your-project-name", agent_stream="your-agent-stream-name")

# Initialize the Splunk Agent Observability wrapped OpenAI client
client = openai.OpenAI(api_key=os.environ.get("OPENAI_API_KEY"))

def call_openai():
    chat_completion = client.chat.completions.create(
        messages=[{"role": "user", "content": "Say this is a test"}],
        model="gpt-4o"
    )
    return chat_completion.choices[0].message.content

# This will create a single span trace with the OpenAI call
call_openai()

# This will upload the trace to Splunk Agent Observability
splunk_ao_context.flush()

Can I log custom evaluators with the SDK?

Yes, you can create custom evaluators in the Splunk Agent Observability UI, and then use them with the SDK.

Log in to the Splunk Agent Observability UI.
Select your Project in the top-left corner
Click “Agent Streams” in the top menu bar
Click the “Configure Evaluators” button in the top-right corner

Custom evaluator logging guide →

How can I annotate or categorize different experiments in my logs?

Use the ‘tags’ and metadata fields to attach filterable labels to your logged Traces and Spans. This allows you organize and view the logged data in the Splunk Agent Observability UI, and add annotation-based logic to your AI application. Annotations can be used to track things like relevant topics, version numbers, experiment IDs, and beyond.

## Initialize SplunkAOLogger
logger = SplunkAOLogger(project="Science Exploration", agent_stream="laws-of-science")
prompt = f"Explain the following topic succinctly: Newton's First Law"

## Define tags and metadata
tags = ["newton", "test", "new-version", "JSON"]
metadata = {"experimentNumber": "1", \
            "promptVersion": "0.0.1", \
            "field": "physics"}

## Initialize a new trace and start listening for logs to add to it
trace = logger.start_trace(
    input=prompt,
    tags=tags,
    metadata=metadata)

Get started annotating logs →

How do I handle logging sensitive data like PII?

If your prompts or generations may contain personally identifiable information (PII), ensure that you mask, anonymize, or exclude that data before logging. Use annotations like tags and metadata to mark sensitive content without logging it directly.

trace = logger.start_trace(
    input="REDACTED",
    tags=['PII_REDACTED'],
    metadata={"pii_redacted": "True"})

Security best practices →

API integration

How do I authenticate with the API?

Use the authentication header for your deployment:

Splunk Observability Cloud
On-premises, standalone, or custom

X-SF-Token: <SPLUNK_AO_O11Y_TOKEN>

Splunk-AO-API-Key: <SPLUNK_AO_API_KEY>

Get your API key →

How do I log a sample via the API?

Send a POST request to the /samples endpoint with a JSON payload:

{
  "task_type": "text-generation",
  "input": "Prompt for the model...",
  "output": "Response from the model...",
  "metadata": {
    "experiment_version": "1.2.1"
  }
}

Logging via API →

What’s the best way to handle errors and retries with the API?

Unlike the SDK, the API does not automatically retry failed requests. You’ll need to:

Check for 4XX and 5XX status codes
Parse error messages in the response body
Implement retry logic using exponential backoff

This is especially important when logging data from production pipelines. Error handling →

Can I update a previously logged data using the API?

Yes, you can update logged data by including the trace_id or span_id created when initially logging. View Trace IDs and Span IDs in the Splunk Agent Observability UI in your Project’s Agent Streams. This is useful for streaming or progressive generation workflows.

{
  "trace_id": trace_id,
  "input": "NEW Prompt for the model...",
  "output": "NEW Response from the model...",
  "metadata": {
    "experiment_version": "1.2.11"
  }
}

Learn more about logging Traces →

How do I retrieve data I’ve previously logged to Splunk Agent Observability via the API?

The API includes endpoints for retrieving logs, evaluators, and datasets. To fetch an Agent Stream, replace project_id and log_stream_id with your IDs. Add the variables for your deployment to .env, then load the file and configure helper variables for the request:

Splunk Observability Cloud
On-premises, standalone, or custom

.env

SPLUNK_AO_REALM="us1"
SPLUNK_AO_O11Y_TOKEN="your-o11y-token"

set -a
source .env
set +a

AO_API_URL="https://app.${SPLUNK_AO_REALM}.observability.splunkcloud.com/ao/api"
AO_AUTH_HEADER="X-SF-Token: ${SPLUNK_AO_O11Y_TOKEN}"

.env

SPLUNK_AO_API_URL="https://your-splunk-ao-api-url"
SPLUNK_AO_API_KEY="your-splunk-ao-api-key"

set -a
source .env
set +a

AO_API_URL="${SPLUNK_AO_API_URL%/}"
AO_AUTH_HEADER="Splunk-AO-API-Key: ${SPLUNK_AO_API_KEY}"

Make the request:

curl -X GET "${AO_API_URL}/v1/projects/PROJECT_ID/logstreams/AGENT_STREAM_ID/logs" \
  -H "${AO_AUTH_HEADER}"

Learn more about Agent Streams →

How do I structure metadata for complex use cases (e.g. nested values)?

Metadata must be a flat key-value object. Nested or deeply structured metadata (e.g. JSON objects within JSON) should be flattened. Metadata keys and values must be strings.

"metadata": {
  "dataset": "test_a",
  "experiment_version": "1.2.11",
  "group": "control"
}

Integration and compatibility

Which LLM providers, models, and frameworks does Splunk Agent Observability support out of the box?

Splunk Agent Observability is model agnostic, and supports leading LLM providers including OpenAI, Azure OpenAI, Anthropic, and Llama. Splunk Agent Observability’s flexible SDK and API allow it to work with any model or provider. View supported integrations on the LLM Integrations the Splunk Agent Observability UI.

Does Splunk Agent Observability integrate with LlamaIndex or other retrieval frameworks for RAG pipelines?

Yes, Splunk Agent Observability works with LlamaIndex (formerly GPT Index) and other retrieval-augmented generation frameworks. You can log RAG context, query, and generation data as part of a single sample or as separate, chained samples. This makes it easier to analyze the quality of both retrieved context and final generations. RAG pipeline logging →

Does Splunk Agent Observability support non-English languages?

This section applies to both on-premises and SaaS deployments. Individual features may not be available for every deployment option. See Feature differences by deployment option for details.

The following table describes support for non-English languages in Splunk Agent Observability. For LLM-powered features, support and performance depend on the language capabilities of the model you use. Check your model provider’s documentation for specific language support.

Feature or component	Support for non-English content	What to expect
Trace capture and display	Supported	You can log and monitor traces containing non-English inputs, outputs, and other text.
Playgrounds and model calls	Model-dependent	You can use non-English prompts, variables, and datasets if the selected model supports that language. Accuracy may vary by model.
LLM-as-a-judge evaluators	Model-dependent	Evaluators can analyze non-English content if the selected model supports that language. Accuracy may vary by model.
Luna-based evaluators	Not supported	Luna-based evaluators are optimized for English and may not evaluate non-English content accurately.
Signals	Not supported	Signals are optimized for English content. Results for non-English content may be incomplete or inaccurate.
AI Assistant	Not supported	AI Assistant is optimized for English content and prompts. Results for non-English content may be incomplete or inaccurate.
User interface text	Not supported	Splunk Agent Observability user interface text is provided in English. Translation into other languages isn’t natively supported.

Troubleshooting and best practices

What should I do if the Splunk Agent Observability SDK fails to install or initialize?

If you are experiencing errors when installing or initializing the Splunk Agent Observability SDK, follow these diagnostic steps:

Reinstall the SDK by running pip install splunk-ao (Python).
Inspect browser/network logs for failed requests to Splunk Agent Observability endpoints, using the Network tab in browser dev tools.
Ensure environment variables (API Key, environment, etc.) are correctly set and loaded by printing them in your app initialization script.

What should I do if Splunk Agent Observability is not logging any data from my application?

If your application is running, but doesn’t appear to be creating logs, try these diagnostic steps:

Ensure you are checking the same Project and Agent Stream that is being logged to in your application. They are case-sensitive.
Use splunk_ao_context or create a new Trace in your application.

Troubleshooting logging issues →

How do I troubleshoot unexpected evaluation results or scores in Splunk Agent Observability?

Start by reviewing the evaluators configuration, including which evaluators are being applied and how they are calculated. If you are using custom evaluators, verify that they are computed correctly and that the input and reference values are accurate. Use Splunk Agent Observability’s error slices and segmentation tools to identify patterns in where scores drop. It’s also helpful to manually inspect low-scoring examples to rule out model or data issues. Instruction adherence evaluator →

How can I ensure my evaluators align with real user needs or product goals?

Start by understanding the key outcomes your product is driving, such as accuracy, clarity, helpfulness, or safety. Choose evaluators that reflect those priorities. Splunk Agent Observability supports custom evaluator logging, so you can track user satisfaction scores, pass/fail labels, or scenario-specific evaluators alongside standard ones. Regularly review evaluation slices with your team to validate alignment with business outcomes. Evaluators overview →

I need more help

If issues persist, contact support with your request payload and full error message.

​General and core concepts

​What is Splunk Agent Observability?

​How does Splunk Agent Observability help with generative AI evaluation?

​How is evaluating generative AI models different from traditional ML evaluation?

​How do I get started with Splunk Agent Observability to evaluate my AI models?

​How does Splunk Agent Observability fit into the generative AI development lifecycle?

​SDK usage

​What SDKs does Splunk Agent Observability provide?

​How do I log prompt and response data using the SDK?

​How do I use the Python SDK in a Python notebook (e.g. Jupyter, Google Colab, etc.)?

​Can I log custom evaluators with the SDK?

​How can I annotate or categorize different experiments in my logs?

​How do I handle logging sensitive data like PII?

​API integration

​How do I authenticate with the API?

​How do I log a sample via the API?

​What’s the best way to handle errors and retries with the API?

​Can I update a previously logged data using the API?

​How do I retrieve data I’ve previously logged to Splunk Agent Observability via the API?

​How do I structure metadata for complex use cases (e.g. nested values)?

​Integration and compatibility

​Which LLM providers, models, and frameworks does Splunk Agent Observability support out of the box?

​Does Splunk Agent Observability integrate with LlamaIndex or other retrieval frameworks for RAG pipelines?

​Does Splunk Agent Observability support non-English languages?

​Troubleshooting and best practices

​What should I do if the Splunk Agent Observability SDK fails to install or initialize?

​What should I do if Splunk Agent Observability is not logging any data from my application?

​How do I troubleshoot unexpected evaluation results or scores in Splunk Agent Observability?

​How can I ensure my evaluators align with real user needs or product goals?

​I need more help

General and core concepts

What is Splunk Agent Observability?

How does Splunk Agent Observability help with generative AI evaluation?

How is evaluating generative AI models different from traditional ML evaluation?

How do I get started with Splunk Agent Observability to evaluate my AI models?

How does Splunk Agent Observability fit into the generative AI development lifecycle?

SDK usage

What SDKs does Splunk Agent Observability provide?

How do I log prompt and response data using the SDK?

How do I use the Python SDK in a Python notebook (e.g. Jupyter, Google Colab, etc.)?

Can I log custom evaluators with the SDK?

How can I annotate or categorize different experiments in my logs?

How do I handle logging sensitive data like PII?

API integration

How do I authenticate with the API?

How do I log a sample via the API?

What’s the best way to handle errors and retries with the API?

Can I update a previously logged data using the API?

How do I retrieve data I’ve previously logged to Splunk Agent Observability via the API?

How do I structure metadata for complex use cases (e.g. nested values)?

Integration and compatibility

Which LLM providers, models, and frameworks does Splunk Agent Observability support out of the box?

Does Splunk Agent Observability integrate with LlamaIndex or other retrieval frameworks for RAG pipelines?

Does Splunk Agent Observability support non-English languages?

Troubleshooting and best practices

What should I do if the Splunk Agent Observability SDK fails to install or initialize?

What should I do if Splunk Agent Observability is not logging any data from my application?

How do I troubleshoot unexpected evaluation results or scores in Splunk Agent Observability?

How can I ensure my evaluators align with real user needs or product goals?

I need more help