> ## Documentation Index
> Fetch the complete documentation index at: https://agent-observability-docs.splunk.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Troubleshooting

Welcome to the Splunk Agent Observability troubleshooting guide. This guide addresses common issues users may encounter when working with the Splunk Agent Observability SDK, API, UI, and overall integration process.

***

## SDK setup and configuration

### Problem: SDK fails to initialize

**Causes:**

* SDK is out of date or improperly installed
* Environment variables are improperly configured

**Solutions:**

* Confirm Splunk Agent Observability is installed and check its version:
  * **Python:** `pip show splunk-ao`
* Reinstall the SDK:
  * **Python:** `pip install splunk-ao`
* Double-check `.env` configuration, making sure that API keys are correct, the keys are named correctly, and loaded via tools like `dotenv`.

***

### Problem: Python SDK methods failing silently or returning None unexpectedly

**Causes:**

* The Splunk Agent Observability SDK catches exceptions internally to prevent crashing your application
* Errors are logged to a specific Python logger (`splunk_ao.utils.catch_log`) that may not be configured in your application

**Solutions:**

* Configure Python logging to see Splunk Agent Observability SDK errors by adding the following to your application code **before** importing Splunk Agent Observability:

  <CodeGroup>
    ```python Python theme={null}
    import logging

    # Configure the Splunk Agent Observability SDK logger
    splunk_ao_catch_logger = logging.getLogger('splunk_ao.utils.catch_log')
    splunk_ao_catch_logger.setLevel(logging.WARNING)
    splunk_ao_catch_logger.addHandler(logging.StreamHandler())

    # Now import and use Splunk Agent Observability
    from splunk_ao import SplunkAOLogger
    ```
  </CodeGroup>

* Common silent failures include:
  * Passing invalid parameters to SDK methods (e.g., unsupported keyword arguments)
  * Network connectivity issues with Splunk Agent Observability APIs

***

## API connectivity and authentication

### Problem: 401 unauthorized when calling Splunk Agent Observability APIs

**Causes:**

* API keys are out of date or entered incorrectly
* Environment variables are improperly configured

**Solutions:**

* Re-generate your API token from the [Splunk Agent Observability UI](/references/faqs/find-keys#splunk-ao-api-key)
* Double-check `.env` configuration, making sure that API keys are correct, the keys are named correctly, and loaded via tools like `dotenv`.

### Problem: API requests timing out

**Causes:**

* Internet connectivity issues
* Payload size in request exceeds limits

**Solutions:**

* Ping Splunk Agent Observability endpoints to check latency using `ping <your-splunk-ao-api-url>` or `curl -I <your-splunk-ao-url>`.
* Validate your internet connection by visiting other websites or performing a speed test.
* Retry using a minimal payload by simplifying your API request body to the required fields only.
* Verify you're not exceeding rate limits by checking error response headers for limits.
* Consider implementing exponential backoff for retry logic, using libraries like `axios-retry` or custom retry mechanisms.

***

## Integration issues

### Problem: Rate limit issues

**Causes:**

* Some evaluators rely on OpenAI APIs or other external APIs which have their own rate limits
* Some agents rely on external APIs which have their own rate limits

**Solutions:**

* Request higher rate limits from OpenAI for your organization.
* Use different API keys or organizations for separate projects or environments (e.g., production vs. pre-production) to distribute load.

### Problem: JSON parsing errors

**Causes:**

* Some evaluators rely on OpenAI or other API responses being valid JSON

**Solutions:**

* Retry the evaluator computation as transient errors may cause invalid JSON.
* Check the output format of the model you're using in its provider's documentation, such as [OpenAI's](https://platform.openai.com/docs/).

***

## Splunk Agent Observability UI

### Problem: UI not loading

**Causes:**

* Browser extensions are interfering with Splunk Agent Observability UI
* Browser cache and cookies are interfering with Splunk Agent Observability UI

**Solutions:**

* Clear browser cache and reload by opening browser settings and selecting "Clear Cache and Site Data".
* Try accessing the UI in incognito mode to rule out extension or cookie conflicts.

### Problem: Logging data not appearing in UI

**Causes:**

* Incorrect Project name or Agent Stream name
* SplunkAOLogger or Traces not properly configured in application
* Failing to conclude or flush the SplunkAOLogger

**Solutions:**

* Confirm that the Project and Agent Stream names in the UI match the names used when invoking the `SplunkAOLogger` or using `splunk_ao_context` in your application code.

  <CodeGroup>
    ```python Python theme={null}
    ## Example: SplunkAOLogger is used to assign Project Name and Agent Stream name
    from splunk_ao import SplunkAOLogger
    logger = SplunkAOLogger(
      project="Your Project Name",
      agent_stream="Your Agent Stream Name"
    )

    ## Example: splunk_ao_context is used to assign project name and Agent Stream name
    from splunk_ao import splunk_ao_context
    with splunk_ao_context(
      project="Your Project Name",
      agent_stream="Your Agent Stream Name"
    ):
        ## ...your AI application code...
    ```
  </CodeGroup>

* In your application code, ensure the `SplunkAOLogger` is properly initialized and [Traces](/sdk-api/logging/splunk-ao-logger#start-a-trace) are configured correctly.

  <CodeGroup>
    ```python Python theme={null}
    ## Import and initialize SplunkAOLogger
    from splunk_ao import SplunkAOLogger
    logger = SplunkAOLogger(
      project="Your Project Name",
      agent_stream="Your Agent Stream Name"
    )

    ## Initialize a new trace and start listening for logs to add to it
    trace = logger.start_trace(
        input="Your prompt",
        name="Example Trace Name",
        tags=["example_tag1", "example_tag2"],
        metadata={"example_key": "example_value"}
    )
    ```

    ````typescript TypeScript theme={null}
    // Import and initialize SplunkAOLogger
    import { SplunkAOLogger } from "splunk-ao";
    const logger = new SplunkAOLogger({
      projectName: "Your Project Name",
      agentStreamName: "Your Agent Stream Name"
    });

    // Initialize a new trace and start listening for logs to add to it
    const trace = logger.startTrace({
      input: "Your prompt",
      name: "Example Trace Name",
      tags: ["example_tag1", "example_tag2"],
      metadata: { example_key: "example_value" },
    });
    ```
    ````
  </CodeGroup>

* At the end of your application code, ensure the `SplunkAOLogger` is concluded and logs are flushed.

  <CodeGroup>
    ```python Python theme={null}
    ## Conclude and flush the logger to close to push captured logs
    logger.conclude(output="Final output of experiment")
    logger.flush()
    ```

    ````typescript TypeScript theme={null}
    // Conclude and flush the logger to close to push captured logs
    logger.conclude({ output: "Final output of experiment" })
    await logger.flush();
    ```
    ````
  </CodeGroup>

### Problem: Spans appearing as separate traces

**Causes:**

* Spans not encapsulated within a Workflow Span
* Traces invoked multiple times without concluding and flushing logs

**Solutions:**

* Create a [Workflow Span](/sdk-api/logging/splunk-ao-logger#add-spans) to act as a parent Span before logging other Spans.

  <CodeGroup>
    ```python Python theme={null}
    from splunk_ao import SplunkAOLogger
    logger = SplunkAOLogger()

    ## Create a workflow span (parent span)
    workflowSpan = logger.add_workflow_span(
        input="Processing workflow",
        name="Main Workflow",
        metadata={ "workflow_type": "test" },
        tags=["workflow"])
    ```

    ```typescript TypeScript theme={null}
    import { SplunkAOLogger } from "splunk-ao";
    const logger = new SplunkAOLogger();

    // Initialize a new trace
    const trace = await logger.startTrace({
      input: "Your prompt",
      name: "Example Trace Name",
      tags: ["example_tag1", "example_tag2"],
      metadata: { example_key: "example_value" },
    });

    // Create a workflow span (parent span)
    const workflowSpan = logger.addWorkflowSpan({
      input: "Processing workflow",
      name: "Main Workflow",
      metadata: { workflow_type: "test" },
      tags: ["workflow"],
    });
    ```
  </CodeGroup>

* Ensure [Traces](/sdk-api/logging/splunk-ao-logger#start-a-trace) are concluded and flushed after creating Spans, and look for loops in your application code that may invoke Traces multiple times.

  <CodeGroup>
    ```python Python theme={null}
    from splunk_ao import SplunkAOLogger
    logger = SplunkAOLogger()

    ## Initialize a new trace and start listening for logs to add to it
    trace = logger.start_trace(
        input="Your prompt",
        name="Example Trace Name",
        tags=["example_tag1", "example_tag2"],
        metadata={"example_key": "example_value"}
    )

    ## ...Spans are created...

    ## Conclude the trace and flush logs
    logger.conclude(output="Final trace output with all spans completed",
                     duration_ns=3000000000)
    logger.flush()
    ```

    ```typescript TypeScript theme={null}
    import { SplunkAOLogger } from "splunk-ao";
    const logger = new SplunkAOLogger();

    // Initialize a new trace and start listening for logs to add to it
    const trace = logger.startTrace({
      input: "Your prompt",
      name: "Example Trace Name",
      tags: ["example_tag1", "example_tag2"],
      metadata: { example_key: "example_value" },
    });

    // ...Spans are created...

    // Conclude the Trace and flush logs
    logger.conclude({
      output: "Final trace output with all spans completed",
      durationNs: 3000000000 
    });
    logger.flush();
    ```
  </CodeGroup>

### Problem: Ground truth evaluators are not appearing

**Causes:**

* Dataset is missing an "output" column
* No Ground Truths are provided

**Solutions:**

* Check your experiment to confirm that your [Dataset](/sdk-api/experiments/datasets) has an "output" column. If it does not, add one.
* Evaluators may appear to be missing because they do not apply to Agent Streams. With Agent Streams, no [Ground Truths](/concepts/evaluators/response-quality/ground-truth-adherence) are provided.

***

For additional help:

* Explore our [Common Errors Guide](/references/faqs/errors) and [FAQ page](/references/faqs/faqs).
* Visit our [Error Catalog](/references/faqs/errors-catalog).
* Contact [Support](https://www.splunk.com/en_us/about-splunk/contact-us.html).