> ## Documentation Index
> Fetch the complete documentation index at: https://agent-observability-docs.splunk.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Middleware

> Learn about using SplunkAOMiddleware for automatic logging of LangChain agents

## Overview

`SplunkAOMiddleware` is a middleware component that integrates with LangGraph agents to provide comprehensive tracing and logging. Unlike the callback-based approach, middleware automatically intercepts agent execution at key points:

* **Agent lifecycle**: Tracks when an agent starts and completes
* **Model calls**: Logs all LLM invocations with prompts, responses, and metadata
* **Tool calls**: Captures tool invocations including function names, arguments, and outputs
* **Async support**: Full support for both synchronous and asynchronous agent execution

## Basic usage

To use `SplunkAOMiddleware`, simply add it to the `middleware` parameter when creating a LangChain agent:

The middleware automatically handles all logging internally. When the agent is invoked:

1. An agent node is created to track the overall execution
2. Each model call creates an LLM node with prompt and response details
3. Each tool call creates a tool node with function name, arguments, and output
4. All nodes are linked hierarchically under the agent node

## Configuration options

`SplunkAOMiddleware` accepts the following parameters:

* `splunk_ao_logger` (optional): A custom `SplunkAOLogger` instance. If not provided, a default logger is created.
* `start_new_trace` (default: `True`): Whether to start a new trace on agent invocation. Set to `False` to add to an existing trace.
* `flush_on_chain_end` (default: `True`): Whether to flush logs to Splunk Agent Observability when the agent completes.
* `ingestion_hook` (optional): A callback function that receives `TracesIngestRequest` objects before they're sent to Splunk Agent Observability.

## Custom logger

You can provide a custom logger instance to integrate with existing logging infrastructure:

<CodeGroup>
  ```python Python theme={null}
  from splunk_ao.logger import SplunkAOLogger

  # Create a custom logger
  logger = SplunkAOLogger(
      project_name="my-agent-project",
      console_output=True
  )

  # Use it with middleware
  agent = create_agent(
      model,
      tools=[get_weather, get_stock_price],
      middleware=[SplunkAOMiddleware(splunk_ao_logger=logger)]
  )
  ```
</CodeGroup>

## Trace management

By default, each agent invocation creates a new trace. You can control trace behavior:

### Add to existing trace

To add agent execution to an existing trace, use a shared logger with `start_new_trace` set to `False` (Python):

<CodeGroup>
  ```python Python theme={null}
  # Create a logger and start a trace
  logger = SplunkAOLogger()
  session_id = logger.create_session()
  trace_id = logger.create_trace(session_id)

  # Create middleware that adds to existing trace
  middleware = SplunkAOMiddleware(
      splunk_ao_logger=logger,
      start_new_trace=False
  )

  # The agent execution will be added to the existing trace
  agent = create_agent(model, tools=[...], middleware=[middleware])
  agent.invoke({"messages": [...]})
  ```
</CodeGroup>

### Manual flush control

If you want to control when logs are flushed (e.g., for batch processing):

<CodeGroup>
  ```python Python theme={null}
  # Disable automatic flushing
  middleware = SplunkAOMiddleware(
      splunk_ao_logger=logger,
      flush_on_chain_end=False
  )

  # Execute multiple agent calls
  agent.invoke({"messages": [...]})
  agent.invoke({"messages": [...]})

  # Manually flush when ready
  logger.flush()
  ```
</CodeGroup>

## What gets logged

`SplunkAOMiddleware` captures the following information:

### Agent node

* Input state (messages)
* Output state (final messages)
* Execution time

### Model call nodes

* Model name and configuration (temperature, etc.)
* Input messages (including system message if present)
* Output response
* Tools available to the model
* Timing evaluators (start time, time to first token if available)

### Tool call nodes

* Tool/function name
* Tool arguments (serialized)
* Tool output
* Execution time

## Comparison with SplunkAOCallback

`SplunkAOMiddleware` (Python) and `SplunkAOCallback` (Python) provide similar functionality but use different approaches:

| Feature               | SplunkAOMiddleware (Python)               | SplunkAOCallback (Python)                        |
| --------------------- | ----------------------------------------- | ------------------------------------------------ |
| **Integration point** | LangGraph agents via middleware parameter | LangChain components via callbacks parameter     |
| **Setup complexity**  | Simple - add to middleware list           | Manual - pass to each component                  |
| **Agent support**     | Native support for LangGraph agents       | Requires callback setup                          |
| **Flexibility**       | Automatic agent-level tracing             | Fine-grained control over individual components  |
| **Language support**  | Python only                               | Python                                           |
| **Use case**          | LangGraph agents with minimal setup       | Complex LangChain applications with custom needs |

Use `SplunkAOMiddleware` when:

* You're building LangGraph agents **in Python**
* You want automatic, drop-in logging
* You prefer simpler setup

Use `SplunkAOCallback` when:

* You need fine-grained control over logging
* You're working with complex LangChain applications
* You want to log specific components selectively

## Async support

`SplunkAOMiddleware` (Python) fully supports asynchronous execution. The middleware automatically handles both sync and async contexts.

<CodeGroup>
  ```python Python theme={null}
  # Async agent usage
  async def main():
      agent = create_agent(
          model,
          tools=[get_weather, get_stock_price],
          middleware=[SplunkAOMiddleware()]
      )
      result = await agent.ainvoke({
          "messages": [HumanMessage(content="...")]
      })
      print(result)
  ```
</CodeGroup>

In Python, the middleware uses the appropriate handler (`SplunkAOBaseHandler` or `SplunkAOAsyncBaseHandler`) based on the execution context.

## Best practices

1. **Use middleware for LangGraph agents**: For LangGraph-based agents, middleware provides the simplest integration
2. **Add meaningful metadata**: Include relevant project and session information in your logger configuration
3. **Configure flush behavior**: For high-volume applications, consider disabling auto-flush and batch your logs
4. **Share loggers**: Use the same logger instance across middleware for unified trace management
5. **Monitor execution**: Review the hierarchical traces in Splunk Agent Observability to understand agent behavior

## Example

You can find a complete example of using `SplunkAOMiddleware` with a LangGraph agent in the [LangChain Middleware Example](https://github.com/splunk/splunk-ao-python/tree/main/examples/agent/langchain-middleware).

## Next steps

### Related documentation

<CardGroup cols={2}>
  <Card title="SplunkAOCallback" icon="code" horizontal href="/sdk-api/third-party-integrations/langchain">
    Use callbacks for fine-grained LangChain logging control.
  </Card>

  <Card title="Experiments" icon="flask" horizontal href="/sdk-api/third-party-integrations/langchain/experiments">
    Learn how to run and track experiments with LangChain.
  </Card>
</CardGroup>

### Cookbooks

<CardGroup cols={2}>
  <Card title="Monitor LangChain Agents with Splunk Agent Observability" icon="code" horizontal href="/cookbooks/use-cases/agent-langchain">
    Learn how to build and monitor a LangChain AI Agent using Splunk Agent Observability for tracing and observability.
  </Card>

  <Card title="Add evaluations to a multi-agent LangGraph application" icon="code" horizontal href="/cookbooks/use-cases/multi-agent-langgraph/multi-agent-langgraph">
    Learn how to add evaluations to a multi-agent LangGraph chat bot using Splunk Agent Observability
  </Card>
</CardGroup>