Log with OpenTelemetry, LangGraph, and OpenAI

Overview

This guide walks you through running a LangGraph app with:

OpenTelemetry tracing
OpenInference semantic conventions
Splunk Agent Observability’s built-in span processor
Automatic LangGraph + OpenAI instrumentation

This example application demonstrates how to build a traced, observable LangGraph workflow that processes a user’s question, generates an LLM response, formats it, and sends complete telemetry to Splunk Agent Observability using OpenTelemetry At a high level, the app:

Takes a user question
Validates the input
Sends the question to OpenAI
Parses/cleans the LLM response
Returns a final formatted answer
Emits detailed traces for every step

The full code example is available in the LangGraph Open Telemetry SDK example.

In this guide you will

Set up your environment and requirements
Understanding and running the LangGraph Open Telemetry SDK example
Run your application with OpenTelemetry

Before you start

Below, you’ll find instructions on the key parts that come into play when using OpenTelemetry for observability.

Python 3.10+ installed
A free Splunk Agent Observability account and API key
An OpenAI API key
Basic understanding of LangGraph concepts
Familiarity with OpenTelemetry basics

Set up your environment and requirements

For this how-to guide we’ll assume that you have some familiarity with LangGraph, as well as some familiarity with basic observability principles. To follow this guide pull the code from the LangGraph Open Telemetry SDK example and work in the root of that directory.

Install required dependencies

The corresponding repository ships with a pyproject.toml and so uv is recommended for this project.After installing uv, you can create and sync a virtual environment with:

uv sync

Set up environment variables

Create environment file or copy it from the .env.example file

cp .env.example .env

Self-hosted deployments: Set the OTel endpoint

The OTel endpoint is different from Splunk Agent Observability’s regular API endpoint and is specifically designed to receive telemetry data in the OTLP format.If you are using:

A cloud deployment, then you don’t need to provide a custom OTel endpoint. The default endpoint <your-splunk-ao-api-url>/otel/traces will be used automatically.
A self-hosted deployment, replace the <your-splunk-ao-api-url>/otel/traces endpoint with your deployment URL. The format of this URL is based on your console URL, appending /otel/traces.

The convention is to store this in the OTEL_EXPORTER_OTLP_ENDPOINT environment variable. For example:

os.environ["OTEL_EXPORTER_OTLP_TRACES_ENDPOINT"] = \
    "<your-splunk-ao-url>/otel/traces"

Understanding and running the LangGraph Open Telemetry SDK example

Initialize OpenTelemetry and Splunk Agent Observability span processor

After setting up your environment variables, initialize OpenTelemetry and create the SplunkAOSpanProcessor. The TracerProvider manages tracers and spans, while the SplunkAOSpanProcessor is responsible for exporting those spans to Splunk Agent Observability.

from splunk_ao import otel
from opentelemetry import trace as trace_api
from opentelemetry.sdk import trace as trace_sdk
from opentelemetry.sdk.resources import Resource

# SplunkAOSpanProcessor (no manual OTLP config required) loads the env vars for 
# the Splunk Agent Observability API key, Project, and Agent Stream. Make sure to set them first. 
splunk_ao_span_processor = otel.SplunkAOSpanProcessor(
    # Optional parameters if not set, uses env var
    # project=os.environ["SPLUNK_AO_PROJECT"], 
    # agentstream=os.environ.get("SPLUNK_AO_AGENT_STREAM"), 
)

# Resource metadata that will appear in Splunk Agent Observability
resource = Resource.create({
    "service.name": "LangGraph-OpenTelemetry-Demo",
    "service.version": "1.0.0",
    "deployment.environment": "development",
})

# Create tracer provider
tracer_provider = trace_sdk.TracerProvider(resource=resource)

# Attach Splunk Agent Observability processor
otel.add_splunk_ao_span_processor(tracer_provider, splunk_ao_span_processor)

# Register this provider globally
trace_api.set_tracer_provider(tracer_provider)

Apply OpenInference instrumentation

Enable automatic AI observability by applying OpenInference instrumentors. These automatically capture LLM calls, token usage, and model performance without requiring changes to your existing code.

from openinference.instrumentation.langchain import LangChainInstrumentor
from openinference.instrumentation.openai import OpenAIInstrumentor

LangChainInstrumentor().instrument(tracer_provider=tracer_provider)
OpenAIInstrumentor().instrument(tracer_provider=tracer_provider)

What this enables automatically:

LangGraph operations and OpenAI API calls are traced
Token usage and model information is captured
Performance evaluators and errors are recorded

Define your LangGraph workflow

This example app will build a simple LangGraph workflow that:

Validates user input with validate_input
Calls OpenAI with generate_response
Formats the final answer with format_answer

from typing import TypedDict  
from langgraph.graph import StateGraph
from openai import OpenAI

class AgentState(TypedDict, total=False):
    user_input: str  # The user's input question
    llm_response: str  # The raw response from the LLM
    parsed_answer: str  # The processed/cleaned answer


# Node 1: Input Validation
# Validates and prepares the user input for processing
def validate_input(state: AgentState):
    user_input = state.get("user_input", "")
    print(f"📥 Validating input: '{user_input}'")

    return {"user_input": user_input}


# Node 2: Generate Response
# Calls OpenAI to generate a response to the user's question
# OpenAI instrumentation will automatically create detailed spans
def generate_response(state: AgentState):
    user_input = state.get("user_input", "")

    try:
        print(f"⚙️ Calling OpenAI with: '{user_input}'")

        # Make the OpenAI API call - OpenAI instrumentation handles tracing
        response = client.chat.completions.create(
            model="gpt-3.5-turbo",
            messages=[{"role": "user", "content": user_input}],
            max_tokens=300,
            temperature=0.7,
        )

        # Extract the response content
        llm_response = response.choices[0].message.content

        if not llm_response:
            print("❌ No response from OpenAI")
        else:
            print(f"✓ Received response: '{llm_response[:100]}...'")

        return {"llm_response": llm_response}

    except Exception as e:
        print(f"❌ Error calling OpenAI: {e}")
        return {"llm_response": f"Error: {str(e)}"}


# Node 3: Format Answer
# Extracts and formats a clean answer from the raw LLM response
def format_answer(state: AgentState):
    llm_response = state.get("llm_response", "")

    # Simple parsing - extract first sentence for a concise answer
    sentences = llm_response.split(". ")
    parsed_answer = sentences[0] if sentences else llm_response

    # Clean up the answer
    parsed_answer = parsed_answer.strip()
    if not parsed_answer.endswith(".") and parsed_answer:
        parsed_answer += "."

    print(f"✨ Parsed answer: '{parsed_answer}'")

    return {"parsed_answer": parsed_answer}

Build and run the LangGraph application

Everything is assembled using LangGraph’s StateGraph:

from langgraph import StateGraph, END

workflow = StateGraph(AgentState)

workflow.add_node("validate_input", validate_input)
workflow.add_node("generate_response", generate_response)
workflow.add_node("format_answer", format_answer)

workflow.set_entry_point("validate_input")
workflow.add_edge("validate_input", "generate_response")
workflow.add_edge("generate_response", "format_answer")
workflow.add_edge("format_answer", END)

app = workflow.compile()

Run the LangGraph application

Finally, run the LangGraph application to observe the traces in Splunk Agent Observability.

# Run the app and observe traces in both console and Splunk Agent Observability
if __name__ == "__main__":
    inputs = {"user_input": "Plan a trip for me"}

    result = app.invoke(AgentState(**inputs))

    provider = trace_api.get_tracer_provider()


    print("\n=== FINAL RESULT ===")
    print(f"Question: {result.get('user_input', 'N/A')}")
    print(f"LLM Response: {result.get('llm_response', 'N/A')}")
    print(f"Parsed Answer: {result.get('parsed_answer', 'N/A')}")
    print("✓ Execution complete - check Splunk Agent Observability for traces in your project/Agent Stream")

Run the full code example

Finally, run the LangGraph Open Telemetry SDK example with:

uv run python main.py

Viewing your traces in Splunk Agent Observability

Once your application is running with OpenTelemetry configured, you can view your traces in the Splunk Agent Observability dashboard. Navigate to your project and Agent Stream to see the complete trace graph showing your LangGraph workflow execution.

The Trace graph for the LangGraph workflow

The trace graph displays:

Workflow spans showing the execution flow through your LangGraph nodes
LLM call details with token usage and model information
Performance evaluators including timing and resource utilization
Error tracking if any issues occur during execution

Run your application with OpenTelemetry

With OpenTelemetry correctly configured, your application will now automatically capture and send observability data to Splunk Agent Observability with every run. You’ll see complete traces of your LangGraph workflows, detailed LLM call breakdowns with token counts, and performance insights organized by project and Agent Stream in your Splunk Agent Observability dashboard. This provides consistent, well-structured logging across all your AI applications without requiring additional code changes, enabling effective monitoring, debugging, and optimization at scale.

The Messages tab for the LangGraph trace

OpenInference semantic conventions for LangGraph—Advanced Usage

When running your LangGraph app with OpenInference, Splunk Agent Observability automatically applies semantic conventions to your traces, capturing model information, token usage, and performance evaluators without any additional code. For advanced use cases, you can also manually add custom attributes to enhance your traces with domain-specific information:

Span attributes

Redacted data attributes: You can attach splunk_ao.redacted_input and splunk_ao.redacted_output to any span in the trace (root or child) to send redacted versions of input and output alongside the originals. Splunk Agent Observability stores both and can restrict the original data to privileged users. Each attribute is independent — you can set one without the other for partial redaction.

# Model information
span.set_attribute("gen_ai.system", "openai")
span.set_attribute("gen_ai.request.model", "gpt-4")
span.set_attribute("gen_ai.request.prompt", user_prompt)

# Response information
span.set_attribute("gen_ai.response.model", "gpt-4")
span.set_attribute("gen_ai.response.content", ai_response)

# Token usage
span.set_attribute("gen_ai.usage.prompt_tokens", 150)
span.set_attribute("gen_ai.usage.completion_tokens", 75)
span.set_attribute("gen_ai.usage.total_tokens", 225)

# Redacted data (privacy/compliance)
# Replace sensitive content with your own redaction logic
redacted_user_input = mask_pii(user_prompt)        # e.g. "My SSN is ***-**-****"
redacted_ai_response = mask_pii(ai_response)       # e.g. "Your account *** is active"

span.set_attribute("splunk_ao.redacted_input", redacted_user_input)
span.set_attribute("splunk_ao.redacted_output", redacted_ai_response)

Events

# Add events to spans for additional context
span.add_event("model.loaded", {
    "model.name": "gpt-4",
    "model.size": "1.7T",
    "load.time_ms": 2500
})

span.add_event("inference.started", {
    "batch.size": 1,
    "max.tokens": 1000
})

span.add_event("inference.completed", {
    "duration.ms": 1250,
    "tokens.generated": 75
})

Troubleshooting your LangGraph app

Here are some common troubleshooting steps when using OpenTelemetry and OpenInference.

Headers not formatted correctly

Not seeing your OTel traces in Splunk Agent Observability? Double checker your header formatting. OpenTelemetry requires headers in a specific comma-separated string format, not as a dictionary.

# ❌ Wrong - dictionary format won't work with OTel
headers = {"Splunk-AO-API-Key": "your-key", "project": "your-project"}

# ✅ Correct - must be comma-separated string format
os.environ["OTEL_EXPORTER_OTLP_TRACES_HEADERS"] = \
    "Splunk-AO-API-Key=your-key,project=your-project,agentstream=default"

Wrong endpoint

# ❌ Wrong - this is the native SDK endpoint
endpoint = "<your-splunk-ao-url>/v2/otlp"

# ✅ Correct - this is the OTel endpoint
endpoint = "<your-splunk-ao-url>/otel/traces"

Missing LangGraph instrumentation

Not seeing your LangGraph workflow traces? Ensure you’re instrumenting both LangGraph and the underlying LLM providers. LangGraph workflows require instrumentation at multiple levels to capture the complete execution flow.

# ❌ Wrong - only instrumenting OpenAI, missing LangGraph workflow tracing
OpenAIInstrumentor().instrument(tracer_provider=tracer_provider)

# ✅ Correct - instrument both LangGraph workflows and LLM providers
from openinference.instrumentation.langgraph import LangGraphInstrumentor
from openinference.instrumentation.openai import OpenAIInstrumentor

LangGraphInstrumentor().instrument(tracer_provider=tracer_provider)
OpenAIInstrumentor().instrument(tracer_provider=tracer_provider)

Next steps

LangGraph OTel Cookbook

Complete tutorial with working LangGraph example

LangChain Integration

Integrate with LangChain using Splunk Agent Observability callbacks

Custom Evaluators

Add custom evaluators to track your LangGraph app performance

Experiments

Run experiments on your instrumented LangGraph workflows

Overview

Get Started

Observability

Evaluators

Signals

AI Assistant

Luna Studio

Experiments

Agent Control

Annotations

Integrations

Security

References

Log with OpenTelemetry, LangGraph, and OpenAI

Overview

In this guide you will

Before you start

Set up your environment and requirements

Understanding and running the LangGraph Open Telemetry SDK example

Run your application with OpenTelemetry

OpenInference semantic conventions for LangGraph—Advanced Usage

Troubleshooting your LangGraph app

Headers not formatted correctly

Wrong endpoint

Missing LangGraph instrumentation

Next steps

LangGraph OTel Cookbook

LangChain Integration

Custom Evaluators

Experiments

​Overview

​In this guide you will

​Before you start

​Set up your environment and requirements

​Understanding and running the LangGraph Open Telemetry SDK example

​Run your application with OpenTelemetry

​OpenInference semantic conventions for LangGraph—Advanced Usage

​Troubleshooting your LangGraph app

​Headers not formatted correctly

​Wrong endpoint

​Missing LangGraph instrumentation

​Next steps

LangGraph OTel Cookbook

LangChain Integration

Custom Evaluators

Experiments

Overview

In this guide you will

Before you start

Set up your environment and requirements

Understanding and running the LangGraph Open Telemetry SDK example

Run your application with OpenTelemetry

OpenInference semantic conventions for LangGraph—Advanced Usage

Troubleshooting your LangGraph app

Headers not formatted correctly

Wrong endpoint

Missing LangGraph instrumentation

Next steps