> ## Documentation Index
> Fetch the complete documentation index at: https://agent-observability-docs.splunk.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Create Traces and Spans

> Learn how to create log traces and spans manually in your AI apps

## Overview

This guide shows you how to log spans to Splunk Agent Observability using the `SplunkAOLogger`.

The Splunk Agent Observability wrappers and `@log` decorator are the preferred way to create log traces and spans, but there are times when you need to manually create a trace or a span to give you more granular control over the data you are logging.

This guide shows how to manually log a number of spans when calling an LLM. This pattern can be used for times with the wrapper or decorator are not applicable, such as when using a unsupported LLM SDK such as the Azure AI inference SDK. You will be using OpenAI for this example.

In this guide you will:

1. [Set up a project with Splunk Agent Observability](#install-dependencies)

1) [Create a basic app to call OpenAI](#create-a-basic-app-to-call-openai)
2) [Create a new Splunk Agent Observability logger to log traces and spans to](#create-a-new-splunk-agent-observability-logger-to-log-traces-and-spans-to)
3) [Add spans](#add-spans)
4) [Add more details to the trace](#add-more-details-to-the-trace)

## Before you start

To complete this how-to, you will need:

* An [OpenAI API key](https://openai.com/api/)
* A [Splunk Agent Observability project](https://agent-observability-docs.splunk.com/concepts/projects.md) configured
* Your [Splunk Agent Observability API key](https://agent-observability-docs.splunk.com/references/faqs/find-keys.md#splunk-ao-api-key)

## Install dependencies

To use Splunk Agent Observability, you need to install some package dependencies, and configure environment variables.

**1. Install Required Dependencies**

Install the required dependencies for your app. If you are using Python, create a virtual environment using your preferred method, then install dependencies inside that environment:

```bash Python theme={null}
pip install "splunk-ao[openai]" python-dotenv
```

**2. Create a .env file, and add the following values**

## Create a basic app to call OpenAI

**1. Create a file for your application called app.py or app.ts.**

**2. Add the following code to call OpenAI to ask a question**

```python Python theme={null}
import os
import asyncio
import openai
from dotenv import load_dotenv

load_dotenv()

client = openai.AsyncOpenAI(api_key=os.environ.get("OPENAI_API_KEY"))

async def prompt_open_ai(prompt: str) -> str:
    response = await client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": prompt}],
    )
    return response.choices[0].message.content.strip()

async def main():
    prompt = "Explain the following topic succinctly: Newton's First Law"
    response = await prompt_open_ai(prompt)
    print(response)

if __name__ == "__main__":
    asyncio.run(main())
```

**3. Run the app to ensure everything is working**

```bash Python theme={null}
python app.py
```

You should see a description of Newton's first law.

```output theme={null}
(.venv) ➜  python app.py
Newton's First Law, also known as the Law of Inertia, states that an object
at rest will stay at rest and an object in motion will stay in motion with
the same speed and in the same direction, unless acted upon by an
unbalanced force. In simpler terms, it means that an object will keep doing
what it's currently doing until a force makes it do something different.
This law is fundamental to understanding motion and forces as it pertains
to physics.
```

## Create a new Splunk Agent Observability logger to log traces and spans to

If you are using the Splunk Agent Observability wrappers or decorators, Splunk Agent Observability automatically create new logging sessions, traces, and spans for you. Seeing as you are doing everything manually, you will need to create a new logger.

**1. Import the Splunk Agent Observability logger**

At the top of your file, add an import for the Splunk Agent Observability logger:

```python Python theme={null}
from splunk_ao import SplunkAOLogger
```

**2. Create a logger instance**

Create a new Splunk Agent Observability logger instance. In Python, do this at the start of the `main` function.

```python Python theme={null}
async def main():
    # Create a logger instance
    logger = SplunkAOLogger()
    ...
```

> [!NOTE]
>
> This will pick up the project and Agent Stream from the `SPLUNK_AO_PROJECT` and `SPLUNK_AO_AGENT_STREAM` environment variables. You can override these if required by setting the relevant constructor parameters.

**3. Create a new trace**

The hierarchy for Agent Streams is Sessions contain traces, which contain spans. If you start a new trace, a session is created automatically.

Create a new trace using the logger by adding the following code just below the declaration of the logger:

```python Python theme={null}
# Start a new trace
logger.start_trace("Newton's First Law Trace")
```

**4. Conclude the trace**

Once you have finished logging a trace, conclude it by passing in the output. Concluding the trace enqueues it for export automatically.

Add the following lines to the bottom of the `main` function in Python:

```python Python theme={null}
logger.conclude(output=response)
```

Your full code should now look like this:

```python Python theme={null}
import os
import asyncio
import openai
from dotenv import load_dotenv

load_dotenv()

client = openai.AsyncOpenAI(api_key=os.environ.get("OPENAI_API_KEY"))

async def prompt_open_ai(prompt: str) -> str:
    response = await client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": prompt}],
    )
    return response.choices[0].message.content.strip()

async def main():
    # Create a logger instance
    logger = SplunkAOLogger()

    # Start a new trace
    logger.start_trace("Newton's First Law Trace")

    prompt = "Explain the following topic succinctly: Newton's First Law"
    response = await prompt_open_ai(prompt)
    print(response)

    logger.conclude(output=response)
```

**5. Run the app to log the trace**

```bash Python theme={null}
python app.py
```

**6. View the logged trace**

From your Splunk Agent Observability homepage, open the Agent Stream for your project. You will see a trace in the traces table.

![A trace in a table showing the input and output](https://mintcdn.com/agent-observability-docs/tQcIFWBC_UEfM1wy/images/howto/create-traces-and-spans/first-trace-table-sao.png?fit=max\&auto=format\&n=tQcIFWBC_UEfM1wy\&q=85\&s=95df72e778479b46d054f44d26f93ef5)

**7. View details of the trace**

Select the trace to view details. Currently it contains no spans, and shows the question and the LLM response as input and output.

![A trace details showing no spans](https://mintcdn.com/agent-observability-docs/tQcIFWBC_UEfM1wy/images/howto/create-traces-and-spans/first-trace-no-span.png?fit=max\&auto=format\&n=tQcIFWBC_UEfM1wy\&q=85\&s=38f6d9d06d5d81f10fccf755844a8740)

## Add spans

The next step is to add spans to the trace. You will be adding an LLM span.

LLM spans can contain a range of details about the LLM call. As well as the input and output text, you can add properties such as the number of tokens used, temperature, and more.

**1. Pass the logger to the prompt OpenAI function**

To use the logger, first you need to pass it to the prompt OpenAI function.

Update the function definition to the following:

```python Python theme={null}
async def prompt_open_ai(prompt: str, logger: SplunkAOLogger) -> str:
    ...
```

Update the call to this function in the `main` function to pass the logger:

```python Python theme={null}
response = await prompt_open_ai(prompt, logger)
```

**2. Log the LLM span**

After the call to the LLM in the prompt OpenAI function, add the following to create an LLM span. Pass in the prompt, and the response from the LLM, along with details of the model and a name.

```python Python theme={null}
logger.add_llm_span(
    input=prompt,
    output=response.choices[0].message.content.strip(),
    model="gpt-4o-mini",
    name="OpenAI GPT-4o-mini response"
)
```

**3. Run the app to log the trace**

```bash Python theme={null}
python app.py
```

**4. View the logged trace**

Select the new trace to view details. You will see a single LLM span called "OpenAI GPT-4o-mini response".

![A trace details showing an LLM span](https://mintcdn.com/agent-observability-docs/tQcIFWBC_UEfM1wy/images/howto/create-traces-and-spans/second-trace-llm-span-sao.png?fit=max\&auto=format\&n=tQcIFWBC_UEfM1wy\&q=85\&s=db82d627173ed2d1bb78bba817428d7e)

## Add more details to the trace

Now you have a span, you can add more details to both the span and the trace, such as duration to help measure latency in your app, and number of tokens to help understand usage and cost.

**1. Add the number of tokens to the LLM span**

Update the call that adds the LLM span to include the details of the tokens returned by the call to OpenAI:

```python Python theme={null}
logger.add_llm_span(
    input=prompt,
    output=response.choices[0].message.content.strip(),
    model="gpt-4o-mini",
    name="OpenAI GPT-4o-mini response",
    # Add details about the tokens
    num_input_tokens=response.usage.prompt_tokens,
    num_output_tokens=response.usage.completion_tokens,
    total_tokens=response.usage.total_tokens
)
```

**2. Add the duration of the LLM call**

If you time the call to the LLM, you can pass this duration to the LLM span.

If you are using Python, add an import for `time` to the top of the file:

```python app.py theme={null}
import time
```

Add code to time the LLM call:

```python Python theme={null}
# Get the start time
start_ns = time.perf_counter_ns()
response = await client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": prompt}],
)
# Calculate the duration
duration_ns = time.perf_counter_ns() - start_ns
```

Add this duration to the LLM span:

```python Python theme={null}
logger.add_llm_span(
    input=prompt,
    output=response.choices[0].message.content.strip(),
    model="gpt-4o-mini",
    name="OpenAI GPT-4o-mini response",
    # Add details about the tokens
    num_input_tokens=response.usage.prompt_tokens,
    num_output_tokens=response.usage.completion_tokens,
    total_tokens=response.usage.total_tokens,
    # Add the duration
    duration_ns=duration_ns
)
```

**3. Add the total duration**

It can also be helpful to time the entire trace. This allows you to spot bottlenecks in your application.

Add similar code to time the code between starting the trace and concluding it. Then pass this to the `logger.conclude` call:

```python Python theme={null}
async def main():
    # Create a logger instance
    logger = SplunkAOLogger()

    # Start a new trace
    logger.start_trace("Newton's First Law Trace")

    # Get the start time
    start_ns = time.perf_counter_ns()

    prompt = "Explain the following topic succinctly: Newton's First Law"
    response = await prompt_open_ai(prompt, logger)
    print(response)

    # Calculate the duration
    duration_ns = time.perf_counter_ns() - start_ns
    logger.conclude(output=response, duration_ns=duration_ns)
```

**4. Run the app to log the trace**

```bash Python theme={null}
python app.py
```

**5. View the details of the logged trace**

Select the new trace to view details. In the **Evaluators** tab you will see the latency.

![A trace details showing a latency of 1.43 seconds](https://mintcdn.com/agent-observability-docs/tQcIFWBC_UEfM1wy/images/howto/create-traces-and-spans/trace-latency-sao.png?fit=max\&auto=format\&n=tQcIFWBC_UEfM1wy\&q=85\&s=54702f07ecb920fee596baaad107b14f)

You can select the LLM span to see more details for that span.

Your logging is now set up! You are ready to configure evaluators for your project.

## See also

* [Configure evaluators](https://agent-observability-docs.splunk.com/concepts/evaluators/overview.md)
* [Agent Streams](https://agent-observability-docs.splunk.com/sdk-api/logging/logging-basics.md)
* [Span types](https://agent-observability-docs.splunk.com/sdk-api/logging/splunk-ao-logger.md#add-spans)


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.