Troubleshooting

Welcome to the Splunk Agent Observability troubleshooting guide. This guide addresses common issues users may encounter when working with the Splunk Agent Observability SDK, API, UI, and overall integration process.

---

​

SDK setup and configuration

​

Problem: SDK fails to initialize

Causes:

• SDK is out of date or improperly installed
• Environment variables are improperly configured

Solutions:

• Confirm Splunk Agent Observability is installed and check its version:
  • Python: pip show splunk-ao
• Reinstall the SDK:
  • Python: pip install splunk-ao
• Double-check .env configuration, making sure that API keys are correct, the keys are named correctly, and loaded via tools like dotenv.

---

​

Problem: Python SDK methods failing silently or returning None unexpectedly

Causes:

• The Splunk Agent Observability SDK catches exceptions internally to prevent crashing your application
• Errors are logged to a specific Python logger (splunk_ao.utils.catch_log) that may not be configured in your application

Solutions:

• Configure Python logging to see Splunk Agent Observability SDK errors by adding the following to your application code before importing Splunk Agent Observability:
  Python

  import logging
  
  # Configure the Splunk Agent Observability SDK logger
  splunk_ao_catch_logger = logging.getLogger('splunk_ao.utils.catch_log')
  splunk_ao_catch_logger.setLevel(logging.WARNING)
  splunk_ao_catch_logger.addHandler(logging.StreamHandler())
  
  # Now import and use Splunk Agent Observability
  from splunk_ao import SplunkAOLogger
  
• Common silent failures include:
  • Passing invalid parameters to SDK methods (e.g., unsupported keyword arguments)
  • Network connectivity issues with Splunk Agent Observability APIs

---

​

API connectivity and authentication

​

Problem: 401 unauthorized when calling Splunk Agent Observability APIs

Causes:

• API keys are out of date or entered incorrectly
• Environment variables are improperly configured

Solutions:

• Re-generate your API token from the Splunk Agent Observability UI
• Double-check .env configuration, making sure that API keys are correct, the keys are named correctly, and loaded via tools like dotenv.

​

Problem: API requests timing out

Causes:

• Internet connectivity issues
• Payload size in request exceeds limits

Solutions:

• Ping Splunk Agent Observability endpoints to check latency using ping <your-splunk-ao-api-url> or curl -I <your-splunk-ao-url>.
• Validate your internet connection by visiting other websites or performing a speed test.
• Retry using a minimal payload by simplifying your API request body to the required fields only.
• Verify you’re not exceeding rate limits by checking error response headers for limits.
• Consider implementing exponential backoff for retry logic, using libraries like axios-retry or custom retry mechanisms.

---

​

Integration issues

​

Problem: Rate limit issues

Causes:

• Some evaluators rely on OpenAI APIs or other external APIs which have their own rate limits
• Some agents rely on external APIs which have their own rate limits

Solutions:

• Request higher rate limits from OpenAI for your organization.
• Use different API keys or organizations for separate projects or environments (e.g., production vs. pre-production) to distribute load.

​

Problem: JSON parsing errors

Causes:

• Some evaluators rely on OpenAI or other API responses being valid JSON

Solutions:

• Retry the evaluator computation as transient errors may cause invalid JSON.
• Check the output format of the model you’re using in its provider’s documentation, such as OpenAI’s.

---

​

Splunk Agent Observability UI

​

Problem: UI not loading

Causes:

• Browser extensions are interfering with Splunk Agent Observability UI
• Browser cache and cookies are interfering with Splunk Agent Observability UI

Solutions:

• Clear browser cache and reload by opening browser settings and selecting “Clear Cache and Site Data”.
• Try accessing the UI in incognito mode to rule out extension or cookie conflicts.

​

Problem: Logging data not appearing in UI

Causes:

• Incorrect Project name or Agent Stream name
• SplunkAOLogger or Traces not properly configured in application
• Failing to conclude or flush the SplunkAOLogger

Solutions:

• Confirm that the Project and Agent Stream names in the UI match the names used when invoking the SplunkAOLogger or using splunk_ao_context in your application code.
  Python

  ## Example: SplunkAOLogger is used to assign Project Name and Agent Stream name
  from splunk_ao import SplunkAOLogger
  logger = SplunkAOLogger(
    project="Your Project Name",
    agent_stream="Your Agent Stream Name"
  )
  
  ## Example: splunk_ao_context is used to assign project name and Agent Stream name
  from splunk_ao import splunk_ao_context
  with splunk_ao_context(
    project="Your Project Name",
    agent_stream="Your Agent Stream Name"
  ):
      ## ...your AI application code...
  
• In your application code, ensure the SplunkAOLogger is properly initialized and Traces are configured correctly.
  Python

  TypeScript

  ## Import and initialize SplunkAOLogger
  from splunk_ao import SplunkAOLogger
  logger = SplunkAOLogger(
    project="Your Project Name",
    agent_stream="Your Agent Stream Name"
  )
  
  ## Initialize a new trace and start listening for logs to add to it
  trace = logger.start_trace(
      input="Your prompt",
      name="Example Trace Name",
      tags=["example_tag1", "example_tag2"],
      metadata={"example_key": "example_value"}
  )
  
• At the end of your application code, ensure the SplunkAOLogger is concluded and logs are flushed.
  Python

  TypeScript

  ## Conclude and flush the logger to close to push captured logs
  logger.conclude(output="Final output of experiment")
  logger.flush()
  

​

Problem: Spans appearing as separate traces

Causes:

• Spans not encapsulated within a Workflow Span
• Traces invoked multiple times without concluding and flushing logs

Solutions:

• Create a Workflow Span to act as a parent Span before logging other Spans.
  Python

  TypeScript

  from splunk_ao import SplunkAOLogger
  logger = SplunkAOLogger()
  
  ## Create a workflow span (parent span)
  workflowSpan = logger.add_workflow_span(
      input="Processing workflow",
      name="Main Workflow",
      metadata={ "workflow_type": "test" },
      tags=["workflow"])
  
• Ensure Traces are concluded and flushed after creating Spans, and look for loops in your application code that may invoke Traces multiple times.
  Python

  TypeScript

  from splunk_ao import SplunkAOLogger
  logger = SplunkAOLogger()
  
  ## Initialize a new trace and start listening for logs to add to it
  trace = logger.start_trace(
      input="Your prompt",
      name="Example Trace Name",
      tags=["example_tag1", "example_tag2"],
      metadata={"example_key": "example_value"}
  )
  
  ## ...Spans are created...
  
  ## Conclude the trace and flush logs
  logger.conclude(output="Final trace output with all spans completed",
                   duration_ns=3000000000)
  logger.flush()
  

​

Problem: Ground truth evaluators are not appearing

Causes:

• Dataset is missing an “output” column
• No Ground Truths are provided

Solutions:

• Check your experiment to confirm that your Dataset has an “output” column. If it does not, add one.
• Evaluators may appear to be missing because they do not apply to Agent Streams. With Agent Streams, no Ground Truths are provided.

---

For additional help:

• Explore our Common Errors Guide and FAQ page.
• Visit our Error Catalog.
• Contact Support.