> ## Documentation Index
> Fetch the complete documentation index at: https://agent-observability-docs.splunk.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Overview

> Learn how to integrate Splunk Agent Observability with OpenTelemetry and OpenInference for comprehensive observability and tracing

This guide explains how to integrate Splunk Agent Observability with [OpenTelemetry](https://opentelemetry.io/) and [OpenInference](https://github.com/Arize-ai/openinference) for comprehensive observability and tracing of your AI/ML workflows using industry-standard tools.

## OpenTelemetry

The first step is to configure OpenTelemetry.

<Steps>
  <Step title="Installation">
    Add the Splunk Agent Observability SDK and OpenTelemetry packages to your project:

    <CodeGroup>
      ```bash Python theme={null}
      pip install opentelemetry-api opentelemetry-sdk \
                  opentelemetry-exporter-otlp
      ```
    </CodeGroup>

    For Python, the `opentelemetry-api` and `opentelemetry-sdk` packages provide the core OpenTelemetry functionality. The `opentelemetry-exporter-otlp` package enables sending traces to Splunk Agent Observability's OTLP endpoint.
  </Step>

  <Step title="Create environment variables for your Splunk Agent Observability settings">
    Set environment variables for your Splunk Agent Observability settings, for example in a `.env` file.
    These environment variables are consumed by the `SplunkAOSpanProcessor` to authenticate
    and route traces to the correct Splunk Agent Observability Project and Agent Stream:

    <CodeGroup>
      ```ini .env theme={null}
      # Provide your Splunk Agent Observability
      # API key and URL if you are using an
      # on-premises, standalone, or custom deployment
      # SPLUNK_AO_API_KEY="your-splunk-ao-api-key"
      # SPLUNK_AO_CONSOLE_URL="your-splunk-ao-url"

      # Your Splunk Agent Observability project name
      SPLUNK_AO_PROJECT="your-splunk-ao-project-name"

      # The name of the Agent Stream you want to use for logging
      SPLUNK_AO_AGENT_STREAM="your-splunk-ao-agent-stream"
      ```
    </CodeGroup>
  </Step>

  <Step title="Self-hosted deployments: Set the OTel endpoint">
    <Note>
      Skip this step if you are using a hosted version.
    </Note>

    The OTel endpoint is different from Splunk Agent Observability's regular API endpoint and is specifically designed to receive telemetry data in the OTLP format.

    If you are using:

    * A cloud deployment, then you don't need to provide a custom OTel endpoint.
      The default endpoint `<your-splunk-ao-api-url>/otel/traces` will be used automatically.

    * A **self-hosted deployment**, replace the `<your-splunk-ao-api-url>/otel/traces` endpoint with your deployment URL. The format of this URL is based on your console URL, appending `/otel/traces`.

    The convention is to store this in the `SPLUNK_AO_CONSOLE_URL` environment variable. For example:

    <CodeGroup>
      ```python Python theme={null}
      os.environ["SPLUNK_AO_CONSOLE_URL"] = "<your-splunk-ao-url>"
      ```
    </CodeGroup>
  </Step>

  <Step title="Initialize and create the Splunk Agent Observability span processor">
    The `SplunkAOSpanProcessor` automatically configures authentication
    and metadata using your environment variables. It also:

    * Auto-builds OTLP headers using your Splunk Agent Observability credentials
    * Configures the correct OTLP trace endpoint
    * Registers a batch span processor that exports traces to Splunk Agent Observability

    <CodeGroup>
      ```python Python theme={null}
      from splunk_ao import otel  

      # SplunkAOSpanProcessor (no manual OTLP config required) loads the env vars for 
      # the Splunk Agent Observability API key, Project, and Agent Stream. Make sure to set them first. 
      splunk_ao_span_processor = otel.SplunkAOSpanProcessor(
          # Optional parameters if not set, uses env var
          # project=os.environ["SPLUNK_AO_PROJECT"], 
          # agentstream=os.environ.get("SPLUNK_AO_AGENT_STREAM"),  
      )
      ```
    </CodeGroup>
  </Step>

  <Step title="Register the span processor">
    The span processor can now be registered with an OTel trace provider.

    <CodeGroup>
      ```python Python theme={null}
      from opentelemetry.sdk import trace as trace_sdk

      tracer_provider = trace_sdk.TracerProvider()
      tracer_provider.add_span_processor(splunk_ao_span_processor)
      ```
    </CodeGroup>
  </Step>
</Steps>

You can now use this trace provider either using a framework that supports OTel directly, or via OpenInference.

## OpenInference

<Note>
  OpenInference instrumentors are currently available for Python only.
</Note>

You can enable automatic tracing for your framework and LLM operations using OpenInference instrumentors. These add AI-specific semantic conventions to your traces.

For example, to instrument LangChain and OpenAI start by adding the relevant OpenInference packages:

<CodeGroup>
  ```bash Terminal theme={null}
  pip install openinference-instrumentation-langchain \
              openinference-instrumentation-openai
  ```
</CodeGroup>

Now you can add the instrumentation to your code, using the OTel trace provider.

<CodeGroup>
  ```python Python theme={null}
  from openinference.instrumentation.langgraph import (
      LangGraphInstrumentor
  )
  from openinference.instrumentation.openai import (
      OpenAIInstrumentor
  )

  LangGraphInstrumentor().instrument(tracer_provider=tracer_provider)
  OpenAIInstrumentor().instrument(tracer_provider=tracer_provider)
  ```
</CodeGroup>

OpenInference adds:

* Automatic capture of LLM calls, token usage, and model performance evaluators
* AI-specific span attributes like `gen_ai.request.model`, `gen_ai.response.content`, and `gen_ai.usage.*`
* Semantic conventions that make your traces more meaningful in Splunk Agent Observability's dashboard
* Framework-specific instrumentation for LangGraph workflows and OpenAI API calls

Once OpenTelemetry and OpenInference is set up your application will automatically capture and send observability data to Splunk Agent Observability with every run, providing complete traces of your AI workflows, detailed LLM call breakdowns, and performance insights organized by project and Agent Stream.

For a detailed example of using OpenTelemetry and OpenInference with LangGraph, see the [Log with OpenTelemetry, LangGraph, and OpenAI](/how-to-guides/third-party-integrations/otel) how-to guide.

## Next steps

Learn how to integrate with some popular frameworks using OpenTelemetry and OpenInference.

<CardGroup cols={2}>
  <Card title="Google ADK" icon="python" horizontal href="/sdk-api/third-party-integrations/opentelemetry-and-openinference/google-adk">
    Learn how to integrate a Google ADK project with Splunk Agent Observability using OpenTelemetry and OpenInference.
  </Card>

  <Card title="Strands Agents" icon="python" horizontal href="/sdk-api/third-party-integrations/opentelemetry-and-openinference/strands-agents">
    Learn how to integrate a Strands Agents project with Splunk Agent Observability using OpenTelemetry.
  </Card>

  <Card title="Vercel AI SDK" icon="js" horizontal href="/sdk-api/third-party-integrations/opentelemetry-and-openinference/vercel-ai">
    Learn how to integrate a Vercel AI SDK project with Splunk Agent Observability using OpenTelemetry.
  </Card>

  <Card title="Mastra" icon="js" horizontal href="/sdk-api/third-party-integrations/opentelemetry-and-openinference/mastra">
    Learn how to integrate a Mastra project with Splunk Agent Observability using OpenTelemetry.
  </Card>
</CardGroup>