Instrumentation

• A project represents a distinct application. For example - a customer facing chatbot and an internal HR chatbot would be 2 separate projects.
• A Agent Stream represents a distinct environment in that project that you want to monitor. For example, a dev Agent Stream for your development work, a staging Agent Stream for your staging environment, and a production Agent Stream for your production application.

​

Structure of Agent Streams

• Sessions represent logical groupings of traces for actions involving an AI that may contain multiple steps. For example, in a chatbot, the entire conversation with a single user would be a session.
• Traces represent one complete interaction with an AI that may contain multiple calls and interactions. For example, inside a chatbot app sending a single message, having the AI handle it using tool calls or RAG, then returning a response, would be a single trace.
• Spans represent distinct operations inside a trace. Each LLM call, tool call, or step in an agentic workflow would be an individual span. For example, in a chatbot app the initial message sent to the LLM, calls to tools based off the LLM response, follow-up LLM calls would all be separate spans.

​

Initial setup

​

API key

​

Project and Agent Stream

​

Set the project and Agent Stream using environment variables

​

Set the project and Agent Stream in code

from splunk_ao import splunk_ao_context

splunk_ao_context.init(project="my-project",agent_stream="my-agent-stream")

from splunk_ao import AgentStream

agent_stream = AgentStream.get(name="my-agent-stream", project_name="my-project")

# Or create a new one
# agent_stream = AgentStream(name="my-agent-stream", project_name="my-project").create()

# Route traces to the specified Agent Stream
with agent_stream.context():
    # Every trace logged inside this block goes to the Agent Stream above
    ...

from splunk_ao import SplunkAOLogger

logger = SplunkAOLogger(project="my-project",
                       agent_stream="my-agent-stream")

from splunk_ao import splunk_ao_context

logger = splunk_ao_context.get_logger_instance(
    project="my-project",
    agent_stream="my-agent-stream"
)

​

Logging flow

​

Logging components

• The Splunk Agent Observability Logger - You can create a logger, and manually manage sessions, traces, spans and more. This logger can be passed around your application to create traces and add spans as needed.
• Log Decorator - You can decorate or wrap functions with the log decorator to have spans created automatically. If you don’t have an active session or trace, one will be created. You can also access the logger used by the decorator for manual control.
• Third-party integrations - Splunk Agent Observability integrates SDKs like the OpenAI SDK, the OpenAI Agents SDK, and LangChain/LangGraph. These integrations manage logging for you, automatically creating sessions, traces and spans as needed.

• In a chatbot app using LangGraph, you can start sessions for each distinct user conversation with the Splunk Agent Observability context, then have the Splunk Agent Observability LangGraph callback log each chat message as a separate trace automatically.
• In an agentic app, you can wrap top level calls with the log decorator to start a trace, access the log that was created by the decorator to add workflow spans, then have spans added automatically under these workflow spans using the OpenAI Agents SDK integration.

​

Next steps

​

Basic logging components

​

OpenTelemetry and OpenInference

​

LLM SDK integrations

​

Agent framework integrations