> ## Documentation Index > Fetch the complete documentation index at: https://agent-observability-docs.splunk.com/llms.txt > Use this file to discover all available pages before exploring further. # Instrumentation > Learn the basics of instrumenting your application with Splunk Agent Observability using the Splunk Agent Observability SDKs **Agent Streams** are the core building blocks used for evaluations. Agent Streams belong to a **project**, with one project containing one or more Agent Streams. The way you structure this varies depending on your organizational preferences or standards, but a typical model would be: * A **[project](/concepts/projects)** represents a distinct application. For example - a customer facing chatbot and an internal HR chatbot would be 2 separate projects. * A **[Agent Stream](/sdk-api/logging/logging-basics)** represents a distinct environment in that project that you want to monitor. For example, a dev Agent Stream for your development work, a staging Agent Stream for your staging environment, and a production Agent Stream for your production application. This allows you to evaluate each separate deployment of each application separately, such as to compare changes in your staging environment to production before a rollout, or to add different evaluators to a dev environment to reflect new capabilities. ## Structure of Agent Streams When logging applications, **Agent Streams** are made up of zero or more **sessions**, which are in turn made up of **traces**, which contain **spans**. **Experiments** can be thought of as a single session, containing multiple traces made up of spans. * **[Sessions](/concepts/logging/sessions/sessions-overview)** represent logical groupings of traces for actions involving an AI that may contain multiple steps. For example, in a chatbot, the entire conversation with a single user would be a session. * **[Traces](/sdk-api/logging/splunk-ao-logger#start-a-trace)** represent one complete interaction with an AI that may contain multiple calls and interactions. For example, inside a chatbot app sending a single message, having the AI handle it using tool calls or RAG, then returning a response, would be a single trace. * **[Spans](/sdk-api/logging/splunk-ao-logger#add-spans)** represent distinct operations inside a trace. Each LLM call, tool call, or step in an agentic workflow would be an individual span. For example, in a chatbot app the initial message sent to the LLM, calls to tools based off the LLM response, follow-up LLM calls would all be separate spans. Evaluators are configured at the Agent Stream level, allowing different evaluators for different Agent Streams. ```mermaid theme={null} flowchart LR evaluators[Evaluators] --> trace subgraph trace[" "] agentstreams --> project[Project] sessions --> agentstreams[Agent Streams] traces --> sessions[Sessions] spans[Spans] --> traces[Traces] end ``` To log to Splunk Agent Observability using the SDK, you use a `SplunkAOLogger` object that is configured to point to a specific project and Agent Stream, then from there you can create sessions and traces, and add log spans to a trace. This logger can be created manually, or automatically using a range of wrappers, decorators and integrations with third party SDKs. ## Initial setup To log to Splunk Agent Observability, you need to configure a connection to Splunk Agent Observability using an API key and optionally a URL for an on-premises, standalone, or custom deployment, as well as setting the project and Agent Stream you want to log to. ### API key To get started building with Splunk Agent Observability, you need to configure your API key, and optionally the URL of your Splunk Agent Observability deployment if you are using a custom hosted, or self deployed version. These are set as environment variables. In development you can use a `.env` file for these, for a production deployment make sure you configure these correctly for your deployment platform. | Environment variable | Description | | :---------------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `SPLUNK_AO_API_KEY` | Only required if you are using an on-premises, standalone, or custom deployment. Your [Splunk Agent Observability API key](/references/faqs/find-keys#splunk-ao-api-key). | | `SPLUNK_AO_CONSOLE_URL` | Only required if you are using an on-premises, standalone, or custom deployment. Set this to your custom deployment URL, which appears in the browser when you log in. | If you are using the free version of Splunk Agent Observability, there is no need to set the `SPLUNK_AO_CONSOLE_URL` environment variable. ### Project and Agent Stream Both the project and Agent Stream can be configured as environment variables, or directly in code. Agent Streams can be created up front in the Splunk Agent Observability UI, or automatically in code. If you log to an Agent Stream that doesn't exist, it will be created automatically for you. #### Set the project and Agent Stream using environment variables The advantage of using environment variables to set your project and Agent Stream is that you can share code across deployments and configure those deployments separately, for example to share the same project but log to different Agent Streams. | Environment variable | Description | | :----------------------- | :---------------------------------------------------------------------------------------------------------------------------------------------- | | `SPLUNK_AO_PROJECT` | The [Splunk Agent Observability project](/concepts/projects) to log to. If this is not set, you will need to pass the project name in code. | | `SPLUNK_AO_AGENT_STREAM` | The [default Agent Stream](/sdk-api/logging/logging-basics) to log to. If this is not set, you will need to pass the Agent Stream name in code. | #### Set the project and Agent Stream in code The advantage of setting in code is you have more granular control, for example logging different parts of your application to different Agent Streams. You can set these in code in two ways - set it at the context level so that it is shared by all logging calls, or set it at an individual logger level. To set at the context level, use this code. After running this code, every trace will go to the specified Agent Stream for the specified project. ```python Python theme={null} from splunk_ao import splunk_ao_context splunk_ao_context.init(project="my-project",agent_stream="my-agent-stream") ``` ```python Python (Beta) theme={null} from splunk_ao import AgentStream agent_stream = AgentStream.get(name="my-agent-stream", project_name="my-project") # Or create a new one # agent_stream = AgentStream(name="my-agent-stream", project_name="my-project").create() # Route traces to the specified Agent Stream with agent_stream.context(): # Every trace logged inside this block goes to the Agent Stream above ... ``` To set the project and Agent Stream for a single logger, you can pass it to the logger constructor. ```python Python theme={null} from splunk_ao import SplunkAOLogger logger = SplunkAOLogger(project="my-project", agent_stream="my-agent-stream") ``` You can also use the current context to get a logger for a particular project and Agent Stream. ```python Python theme={null} from splunk_ao import splunk_ao_context logger = splunk_ao_context.get_logger_instance( project="my-project", agent_stream="my-agent-stream" ) ``` If you are using experiments, you can set the project name in the call to run the experiment, and the Agent Stream name is generated for you. Learn more in our [experiment SDK docs](/sdk-api/experiments/experiments#project). ## Logging flow The typical logging flow follows these steps: ```mermaid theme={null} flowchart LR startSession[Start Session] --> trace[Traces] subgraph trace[Traces] createTrace[Create Trace] --> addSpan[Add Spans] addSpan --> concludeTrace[Conclude Trace] end trace[Traces] --> flushLogs[Flush Logs] ``` Starting sessions is optional - if you don't start a session, then all traces are automatically each logged to new autogenerated sessions. Using the Splunk Agent Observability SDK you can either do all these steps manually, or you can use a range of wrappers, decorators and integrations with third-party SDKs, where most of this is handled for you. ## Logging components Splunk Agent Observability provides 3 ways to log your application code: * **[The Splunk Agent Observability Logger](/sdk-api/logging/splunk-ao-logger)** - You can create a logger, and manually manage sessions, traces, spans and more. This logger can be passed around your application to create traces and add spans as needed. * **[Log Decorator](/sdk-api/logging/log-decorator/log-decorator)** - You can decorate or wrap functions with the log decorator to have spans created automatically. If you don't have an active session or trace, one will be created. You can also access the logger used by the decorator for manual control. * **[Third-party integrations](/sdk-api/third-party-integrations/overview)** - Splunk Agent Observability integrates SDKs like the OpenAI SDK, the OpenAI Agents SDK, and LangChain/LangGraph. These integrations manage logging for you, automatically creating sessions, traces and spans as needed. In addition, there is a [Splunk Agent Observability context manager](/sdk-api/logging/splunk-ao-context) that provides top level control over logging, such as setting the project and Agent Stream, flushing all loggers, and managing sessions. All the logging methods can be mixed and matched, and combined with the Splunk Agent Observability Context. For example: * In a chatbot app using LangGraph, you can start sessions for each distinct user conversation with the Splunk Agent Observability context, then have the Splunk Agent Observability LangGraph callback log each chat message as a separate trace automatically. * In an agentic app, you can wrap top level calls with the `log` decorator to start a trace, access the log that was created by the decorator to add workflow spans, then have spans added automatically under these workflow spans using the OpenAI Agents SDK integration. ## Next steps ### Basic logging components Log with full control over sessions, traces, and spans using the Splunk Agent Observability logger. Quickly add logging to your code with the log decorator and wrapper. Manage logging using the Splunk Agent Observability context manager. ### OpenTelemetry and OpenInference Learn how to integrate Splunk Agent Observability with OpenTelemetry and OpenInference for comprehensive observability and tracing. Learn how to integrate a Google ADK project with Splunk Agent Observability using OpenTelemetry and OpenInference. Learn how to integrate a Strands Agents project with Splunk Agent Observability using OpenTelemetry. Learn how to integrate a Vercel AI SDK project with Splunk Agent Observability using OpenTelemetry. ### LLM SDK integrations Automatically log calls to the OpenAI SDK with a wrapper. ### Agent framework integrations Automatically log all the steps in your CrewAI application with the Splunk Agent Observability event listener. Automatically log all the steps in your LangChain or LangGraph application with the Splunk Agent Observability callback. Automatically log all the steps in your OpenAI Agent SDK apps using the Splunk Agent Observability trace processor.