> ## Documentation Index
> Fetch the complete documentation index at: https://agent-observability-docs.splunk.com/llms.txt
> Use this file to discover all available pages before exploring further.

# FAQ

## General and core concepts

### What is Splunk Agent Observability?

**Splunk Agent Observability** is an end-to-end platform for evaluating and improving the performance of generative AI models. It provides tools to help teams deeply understand model behavior across tasks like text generation, summarization, classification, and more. Splunk Agent Observability integrates seamlessly into your LLM pipeline and supports evaluation with both human feedback and automated evaluators.

[View the Splunk Agent Observability platform overview →](https://agent-observability-docs.splunk.com/what-is-splunk-agent-observability.md)

### How does Splunk Agent Observability help with generative AI evaluation?

Splunk Agent Observability enables teams to evaluate generative models through:

* **Automatic and custom evaluators**: Evaluate and track instruction adherence, correctness, safety & compliance, chunk utilization, and more.
* **Human feedback integration**: Annotate and label logs through the UI or SDK.
* **Fine-grained error analysis**: Detect generation quality and systematically log responses as CLLF, ILLF, IHFA, etc.
* **Multi-model comparison**: Evaluate different model outputs side by side.

This helps identify where your model is strong or weak and speeds up iteration cycles.

[Learn more about how Splunk Agent Observability helps →](https://agent-observability-docs.splunk.com/what-is-splunk-agent-observability.md)

### How is evaluating generative AI models different from traditional ML evaluation?

Unlike traditional ML, where outputs are often single values or labels, generative AI outputs are open-ended and nuanced (e.g. full text). This creates challenges like:

* **Subjectivity**: There's often no single correct answer.
* **Evaluation complexity**: Evaluators must handle variation, style, relevance, and factual accuracy.
* **Human oversight**: Required to understand intent and impact of generations.

Splunk Agent Observability addresses these differences with custom workflows tailored to generative AI.

[Learn more about Splunk Agent Observability's evaluation approach →](https://agent-observability-docs.splunk.com/concepts/evaluators/overview.md)

### How do I get started with Splunk Agent Observability to evaluate my AI models?

Visit the [Getting Started](https://agent-observability-docs.splunk.com/getting-started/quickstart.md) guide and follow along to create your first **Splunk Agent Observability** application.

Then, you can start incorporating Splunk Agent Observability into your AI project pipeline.

[Getting Started guide →](https://agent-observability-docs.splunk.com/getting-started/quickstart.md)

### How does Splunk Agent Observability fit into the generative AI development lifecycle?

Splunk Agent Observability supports every stage of generative AI development:

* **Prompt & data experimentation**: Evaluate prompt effectiveness.
* **Model comparison**: Choose the best model or fine-tuned variant.
* **Fine-tuning oversight**: Identify when training introduces regressions.
* **Pre-deployment checks**: Surface edge cases, hallucinations, or bias.
* **Post-deployment monitoring**: Continuously analyze performance and drift.

It becomes your team's co-pilot in delivering high-quality AI products.

## SDK usage

### What SDKs does Splunk Agent Observability provide?

Splunk Agent Observability provides a Python SDK designed to integrate with your generative AI pipelines. They allow you to log prompts, model generations, evaluators, and feedback directly from your application.

Install the Splunk Agent Observability SDK using the following terminal command:

```bash Python theme={null}
pip install splunk-ao
```

[Python SDK installation instructions →](https://agent-observability-docs.splunk.com/sdk-api/python/sdk-reference.md)

### How do I log prompt and response data using the SDK?

You can log AI prompt and response data by recording logs to an Agent Stream using the **Splunk Agent Observability Logger**.

By initializing a new [Trace](https://agent-observability-docs.splunk.com/sdk-api/logging/splunk-ao-logger.md#start-a-trace), the AI activity executed in the subsequent code is recorded to the selected Project's Agent Stream.

```python Python theme={null}
## Include SplunkAOLogger in imports
from splunk_ao import SplunkAOLogger

## Initialize logger and select project and Agent Stream
logger = SplunkAOLogger(project="Science Exploration", agent_stream="laws-of-science")

## Initialize a new trace and start listening for logs to add to it
prompt = 'Write a short story about a magic pizza.'
trace = logger.start_trace(input=prompt)
```

The logged data is viewable in the Splunk Agent Observability UI for further evaluation.

[More on logging data →](https://agent-observability-docs.splunk.com/sdk-api/logging/logging-basics.md)

### How do I use the Python SDK in a Python notebook (e.g. Jupyter, Google Colab, etc.)?

When using a Python Notebook (such as Jupyter, Google Colab, etc.), you should use the `splunk_ao_context` and make sure to call `splunk_ao_context.flush()` at the end of your notebook.

```python Python theme={null}
import os
from splunk_ao import splunk_ao_context
from splunk_ao.openai import openai

splunk_ao_context.init(project="your-project-name", agent_stream="your-agent-stream-name")

# Initialize the Splunk Agent Observability wrapped OpenAI client
client = openai.OpenAI(api_key=os.environ.get("OPENAI_API_KEY"))

def call_openai():
    chat_completion = client.chat.completions.create(
        messages=[{"role": "user", "content": "Say this is a test"}],
        model="gpt-4o"
    )
    return chat_completion.choices[0].message.content

# This will create a single span trace with the OpenAI call
call_openai()

# This will upload the trace to Splunk Agent Observability
splunk_ao_context.flush()
```

### Can I log custom evaluators with the SDK?

Yes, you can create custom evaluators in the Splunk Agent Observability UI, and then use them with the SDK.

1. Log in to the Splunk Agent Observability UI.
2. Select your Project in the top-left corner
3. Click "Agent Streams" in the top menu bar
4. Click the "Configure Evaluators" button in the top-right corner

[Custom evaluator logging guide →](https://agent-observability-docs.splunk.com/concepts/evaluators/overview.md)

### How can I annotate or categorize different experiments in my logs?

Use the 'tags' and `metadata` fields to attach filterable labels to your logged Traces and Spans. This allows you organize and view the logged data in the Splunk Agent Observability UI, and add annotation-based logic to your AI application.

Annotations can be used to track things like relevant topics, version numbers, experiment IDs, and beyond.

```python Python theme={null}
## Initialize SplunkAOLogger
logger = SplunkAOLogger(project="Science Exploration", agent_stream="laws-of-science")
prompt = f"Explain the following topic succinctly: Newton's First Law"

## Define tags and metadata
tags = ["newton", "test", "new-version", "JSON"]
metadata = {"experimentNumber": "1", \
            "promptVersion": "0.0.1", \
            "field": "physics"}

## Initialize a new trace and start listening for logs to add to it
trace = logger.start_trace(
    input=prompt,
    tags=tags,
    metadata=metadata)
```

[Get started annotating logs →](https://agent-observability-docs.splunk.com/concepts/annotations)

### How do I handle logging sensitive data like PII?

If your prompts or generations may contain personally identifiable information (PII), ensure that you mask, anonymize, or exclude that data before logging.

Use annotations like tags and metadata to mark sensitive content without logging it directly.

```python Python theme={null}
trace = logger.start_trace(
    input="REDACTED",
    tags=['PII_REDACTED'],
    metadata={"pii_redacted": "True"})
```

[Security best practices →](https://agent-observability-docs.splunk.com/concepts/evaluators/safety-and-compliance/pii.md)

## API integration

### How do I authenticate with the API?

Use the authentication header for your deployment:

**Splunk Observability Cloud**

```text theme={null}
X-SF-Token: <SPLUNK_AO_O11Y_TOKEN>
```

**On-premises, standalone, or custom**

```text theme={null}
Splunk-AO-API-Key: <SPLUNK_AO_API_KEY>
```

[Get your API key →](https://agent-observability-docs.splunk.com/references/faqs/find-keys.md#splunk-ao-api-key)

### How do I log a sample via the API?

Send a `POST` request to the `/samples` endpoint with a JSON payload:

```json theme={null}
{
  "task_type": "text-generation",
  "input": "Prompt for the model...",
  "output": "Response from the model...",
  "metadata": {
    "experiment_version": "1.2.1"
  }
}
```

[Logging via API →](https://agent-observability-docs.splunk.com/sdk-api/logging/logging-basics.md)

### What's the best way to handle errors and retries with the API?

Unlike the SDK, the API does not automatically retry failed requests. You'll need to:

* Check for `4XX` and `5XX` status codes
* Parse error messages in the response body
* Implement retry logic using exponential backoff

This is especially important when logging data from production pipelines.

[Error handling →](https://agent-observability-docs.splunk.com/references/faqs/errors.md)

### Can I update a previously logged data using the API?

Yes, you can update logged data by including the `trace_id` or `span_id` created when initially logging. View Trace IDs and Span IDs in the Splunk Agent Observability UI in your Project's Agent Streams.

This is useful for streaming or progressive generation workflows.

```json theme={null}
{
  "trace_id": trace_id,
  "input": "NEW Prompt for the model...",
  "output": "NEW Response from the model...",
  "metadata": {
    "experiment_version": "1.2.11"
  }
}
```

[Learn more about logging Traces →](https://agent-observability-docs.splunk.com/sdk-api/logging/splunk-ao-logger.md#start-a-trace)

### How do I retrieve data I've previously logged to Splunk Agent Observability via the API?

The API includes endpoints for retrieving logs, evaluators, and datasets.

To fetch an Agent Stream, replace `project_id` and `log_stream_id` with your IDs. Add the variables for your deployment to `.env`, then load the file and configure helper variables for the request:

**Splunk Observability Cloud**

```ini .env theme={null}
SPLUNK_AO_REALM="us1"
SPLUNK_AO_O11Y_TOKEN="your-o11y-token"
```

```bash theme={null}
set -a
source .env
set +a

AO_API_URL="https://app.${SPLUNK_AO_REALM}.observability.splunkcloud.com/ao/api"
AO_AUTH_HEADER="X-SF-Token: ${SPLUNK_AO_O11Y_TOKEN}"
```

**On-premises, standalone, or custom**

```ini .env theme={null}
SPLUNK_AO_API_URL="https://your-splunk-ao-api-url"
SPLUNK_AO_API_KEY="your-splunk-ao-api-key"
```

```bash theme={null}
set -a
source .env
set +a

AO_API_URL="${SPLUNK_AO_API_URL%/}"
AO_AUTH_HEADER="Splunk-AO-API-Key: ${SPLUNK_AO_API_KEY}"
```

Make the request:

```bash theme={null}
curl -X GET "${AO_API_URL}/v1/projects/PROJECT_ID/logstreams/AGENT_STREAM_ID/logs" \
  -H "${AO_AUTH_HEADER}"
```

[Learn more about Agent Streams →](https://agent-observability-docs.splunk.com/sdk-api/logging/logging-basics.md)

### How do I structure metadata for complex use cases (e.g. nested values)?

Metadata must be a flat key-value object. Nested or deeply structured metadata (e.g. JSON objects within JSON) should be flattened.

Metadata keys and values must be strings.

```json theme={null}
"metadata": {
  "dataset": "test_a",
  "experiment_version": "1.2.11",
  "group": "control"
}
```

## Integration and compatibility

### Which LLM providers, models, and frameworks does Splunk Agent Observability support out of the box?

Splunk Agent Observability is model agnostic, and supports leading LLM providers including OpenAI, Azure OpenAI, Anthropic, and Llama.

Splunk Agent Observability's flexible SDK and API allow it to work with any model or provider.

View supported integrations on the **LLM Integrations** the Splunk Agent Observability UI.

### Does Splunk Agent Observability integrate with LlamaIndex or other retrieval frameworks for RAG pipelines?

Yes, Splunk Agent Observability works with LlamaIndex (formerly GPT Index) and other retrieval-augmented generation frameworks.

You can log RAG context, query, and generation data as part of a single sample or as separate, chained samples. This makes it easier to analyze the quality of both retrieved context and final generations.

[RAG pipeline logging →](https://agent-observability-docs.splunk.com/how-to-guides/rag/basic-example.md)

### Does Splunk Agent Observability support non-English languages?

> [!NOTE]
>
> This section applies to both on-premises and SaaS deployments. Individual features may not be available for every deployment option. See [Feature differences by deployment option](https://agent-observability-docs.splunk.com/saas/overview.md#feature-differences-by-deployment-option) for details.

The following table describes support for non-English languages in Splunk Agent Observability.

For LLM-powered features, support and performance depend on the language capabilities of the model you use. Check your model provider's documentation for specific language support.

| Feature or component | Support for non-English content | What to expect |
| - | - | - |
| Trace capture and display | Supported | You can log and monitor traces containing non-English inputs, outputs, and other text. |
| [Playgrounds and model calls](https://agent-observability-docs.splunk.com/concepts/experiments/running-experiments-in-console.md) | Model-dependent | You can use non-English prompts, variables, and datasets if the selected model supports that language. Accuracy may vary by model. |
| [LLM-as-a-judge evaluators](https://agent-observability-docs.splunk.com/concepts/evaluators/how-llm-as-judge-evaluators-are-calculated.md) | Model-dependent | Evaluators can analyze non-English content if the selected model supports that language. Accuracy may vary by model. |
| [Luna-based evaluators](https://agent-observability-docs.splunk.com/concepts/luna/luna.md) | Not supported | Luna-based evaluators are optimized for English and may not evaluate non-English content accurately. |
| [Signals](https://agent-observability-docs.splunk.com/concepts/signals.md) | Not supported | Signals are optimized for English content. Results for non-English content may be incomplete or inaccurate. |
| [AI Assistant](https://agent-observability-docs.splunk.com/concepts/ai-assistant.md) | Not supported | AI Assistant is optimized for English content and prompts. Results for non-English content may be incomplete or inaccurate. |
| User interface text | Not supported | Splunk Agent Observability user interface text is provided in English. Translation into other languages isn't natively supported. |

## Troubleshooting and best practices

### What should I do if the Splunk Agent Observability SDK fails to install or initialize?

If you are experiencing errors when installing or initializing the Splunk Agent Observability SDK, follow these diagnostic steps:

1. Reinstall the SDK by running `pip install splunk-ao` (Python).
2. Inspect browser/network logs for failed requests to Splunk Agent Observability endpoints, using the Network tab in browser dev tools.
3. Ensure environment variables (API Key, environment, etc.) are correctly set and loaded by printing them in your app initialization script.

### What should I do if Splunk Agent Observability is not logging any data from my application?

If your application is running, but doesn't appear to be creating logs, try these diagnostic steps:

1. Ensure you are checking the same Project and Agent Stream that is being logged to in your application. They are case-sensitive.
2. Use `splunk_ao_context` or create a new [Trace](https://agent-observability-docs.splunk.com/sdk-api/logging/splunk-ao-logger.md#start-a-trace) in your application.

[Troubleshooting logging issues →](https://agent-observability-docs.splunk.com/references/faqs/errors.md)

### How do I troubleshoot unexpected evaluation results or scores in Splunk Agent Observability?

Start by reviewing the evaluators configuration, including which evaluators are being applied and how they are calculated.

If you are using custom evaluators, verify that they are computed correctly and that the input and reference values are accurate. Use Splunk Agent Observability's error slices and segmentation tools to identify patterns in where scores drop.

It's also helpful to manually inspect low-scoring examples to rule out model or data issues.

[Instruction adherence evaluator →](https://agent-observability-docs.splunk.com/concepts/evaluators/response-quality/instruction-adherence.md)

### How can I ensure my evaluators align with real user needs or product goals?

Start by understanding the key outcomes your product is driving, such as accuracy, clarity, helpfulness, or safety.

Choose evaluators that reflect those priorities. Splunk Agent Observability supports custom evaluator logging, so you can track user satisfaction scores, pass/fail labels, or scenario-specific evaluators alongside standard ones.

Regularly review evaluation slices with your team to validate alignment with business outcomes.

[Evaluators overview →](https://agent-observability-docs.splunk.com/concepts/evaluators/overview.md)

## I need more help

If issues persist, [contact support](https://www.splunk.com/en_us/about-splunk/contact-us.html) with your request payload and full error message.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.