> ## Documentation Index
> Fetch the complete documentation index at: https://agent-observability-docs.splunk.com/llms.txt
> Use this file to discover all available pages before exploring further.
# OpenTelemetry Integration Recommendations
> Guidelines for instrumenting your spans to ensure they are valid and properly processed by Splunk Agent Observability's OTLP provider.
Splunk Agent Observability's OTLP provider conforms to [OpenTelemetry](https://opentelemetry.io/) and [OpenInference](https://github.com/Arize-ai/openinference) semantic conventions. To ensure your spans are valid and properly processed, follow these guidelines.
## Semantic Conventions
Splunk Agent Observability supports two complementary semantic convention standards:
1. **OpenTelemetry GenAI Semantic Conventions** — [Semantic Conventions for GenAI agent and framework spans](https://opentelemetry.io/docs/specs/semconv/gen-ai/gen-ai-agent-spans/)
2. **OpenInference** — [OpenInference Semantic Conventions](https://github.com/Arize-ai/openinference/blob/main/python/openinference-semantic-conventions/src/openinference/semconv/trace/__init__.py)
When using automatic instrumentation libraries (like Traceloop/OpenLLMetry), these requirements are typically handled automatically. When creating custom spans, ensure you follow both the OpenTelemetry GenAI conventions and any framework-specific guidelines.
## Minimum Requirements for Valid Spans
For a span to be considered valid, it must include certain **required** attributes depending on the span type.
### Agent Spans
| Attribute | Requirement Level | Description | Example |
| ----------------------- | ----------------- | ------------------------------------------------------------------------------------------------------- | ------------------------------------------- |
| `gen_ai.operation.name` | **Required** | The name of the operation being performed | `invoke_agent`, `create_agent` |
| `gen_ai.provider.name` | **Required** | The Generative AI provider | `openai`, `anthropic`, `gcp.vertex_ai` |
| Span name | **Required** | Should follow format: `invoke_agent {gen_ai.agent.name}` or `invoke_agent` | `invoke_agent Math Tutor` |
| Span kind | **Required** | Should be `CLIENT` for remote agents or `INTERNAL` for in-process agents | `CLIENT`, `INTERNAL` |
| **Input** | **Required** | Input messages or data. Use `gen_ai.input.messages` (OpenTelemetry) or `input.value` (OpenInference) | `[{"role": "user", "content": "..."}]` |
| **Output** | **Required** | Output messages or data. Use `gen_ai.output.messages` (OpenTelemetry) or `output.value` (OpenInference) | `[{"role": "assistant", "content": "..."}]` |
### LLM Spans
| Attribute | Requirement Level | Description | Example |
| ----------------------- | ---------------------- | --------------------------------------------------------------------------------------------------------------------- | ------------------------------------------- |
| `gen_ai.operation.name` | **Required** | The name of the operation | `chat`, `text_completion`, `embeddings` |
| `gen_ai.provider.name` | **Required** | The Generative AI provider | `openai`, `anthropic` |
| `gen_ai.request.model` | Conditionally Required | The name of the GenAI model | `gpt-4`, `claude-3-opus` |
| **Input** | **Required** | Input messages or prompts. Use `gen_ai.input.messages` (OpenTelemetry) or `llm.input_messages` (OpenInference) | `[{"role": "user", "content": "..."}]` |
| **Output** | **Required** | Output messages or completions. Use `gen_ai.output.messages` (OpenTelemetry) or `llm.output_messages` (OpenInference) | `[{"role": "assistant", "content": "..."}]` |
### Tool Execution Spans
| Attribute | Requirement Level | Description | Example |
| ----------------------- | ----------------- | ---------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------- |
| `gen_ai.operation.name` | **Required** | Should be `execute_tool` | `execute_tool` |
| `gen_ai.tool.name` | **Required** | Name of the tool being used | `get_weather`, `calculate` |
| `gen_ai.tool.call.id` | Recommended | Unique identifier for the tool call | `call_abc123` |
| **Input** | **Required** | Tool call arguments. Use `gen_ai.tool.call.arguments` and `gen_ai.input.messages` (OpenTelemetry) or `input.value` (OpenInference) | `{"location": "NYC", "unit": "fahrenheit"}` |
| **Output** | **Required** | Tool call result. Use `gen_ai.tool.call.result` and `gen_ai.output.messages` (OpenTelemetry) or `output.value` (OpenInference) | `{"temperature": 72, "condition": "sunny"}` |
### Retriever Spans
| Attribute | Requirement Level | Description | Example |
| ----------------------------------------- | ---------------------- | ----------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------- |
| `db.operation` | **Required** | Database operation type. Should be `query` or `search` | `query`, `search` |
| `openinference.span.kind` (OpenInference) | Conditionally Required | Should be `retriever` when using OpenInference | `retriever` |
| **Input** | **Required** | Query string or search input. Use `gen_ai.input.messages` (OpenTelemetry) or `input.value` (OpenInference) | `"What is machine learning?"` |
| **Output** | **Required** | Retrieved documents. Use `gen_ai.output.messages` with document list (OpenTelemetry) or `retrieval.documents` (OpenInference) | `[{"id": "doc1", "content": "..."}, {"id": "doc2", "content": "..."}]` |
Retriever spans are typically detected automatically when `db.operation` is set to `query` or `search`. The output should be a list of documents, which will be formatted appropriately by Splunk Agent Observability's OTLP provider.
### Workflow Spans
Workflow spans represent higher-level orchestration units that coordinate multiple sub-operations, such as chains, pipelines, or multi-step processes.
| Attribute | Requirement Level | Description | Example |
| ---------- | ----------------- | ----------------------------------------------------------------------------------------------------- | ----------------------------------------------------------- |
| Span name | **Required** | Descriptive name of the workflow | `document_processing_pipeline` |
| **Input** | **Required** | Input data to the workflow. Use `gen_ai.input.messages` with user message format | `[{"role": "user", "content": "Process this document"}]` |
| **Output** | **Required** | Output from the workflow. Use `gen_ai.output.messages` with assistant message or document list format | `[{"role": "assistant", "content": "Processing complete"}]` |
Workflow spans are useful for grouping related operations together. When using the Splunk Agent Observability SDK's `start_splunk_ao_span` helper, workflow spans can contain nested child spans for LLM calls, tool executions, and retriever operations.
## Error Handling
For spans that end in an error, you **must** include:
* **`error.type`** — Describes the class of error (e.g., `timeout`, `500`, exception name)
* **Span status** — Should be set to `ERROR` with an appropriate error description
```python Python theme={null}
from opentelemetry.trace import StatusCode
span.set_status(StatusCode.ERROR, "Connection timed out")
span.set_attribute("error.type", "timeout")
```
## Direct POST Calls to the OTLP Endpoint
You can make direct POST calls to the Splunk Agent Observability OTLP endpoint to send OTLP packets. This is useful for custom integrations or when you need to send pre-generated OTLP data.
### Endpoint
```sh theme={null}
POST /otel/v1/traces
```
### Headers
The following headers are required:
| Header | Description |
| -------------------------------- | ------------------------------------------------------------------------------------------------------------------------------- |
| `Splunk-AO-API-Key` | Your Splunk Agent Observability API key |
| `project` or `projectid` | Project name or project ID (at least one required) |
| `agentstream` or `agentstreamid` | Agent Stream name or Agent Stream ID (required for logging, omit when using `experimentid`) |
| `experimentid` | Experiment ID. Routes traces to an experiment instead of an Agent Stream. Mutually exclusive with `agentstream`/`agentstreamid` |
| `Content-Type` | Must be `application/x-protobuf` |
### Resource Attributes
When sending spans for experiments, you can attach Splunk Agent Observability-specific resource attributes to the spans in the protobuf payload. These attributes are used by the backend to route traces and attach ground truth data for evaluation.
If you are using the `SplunkAOSpanProcessor` from the Splunk Agent Observability SDK, these attributes are set automatically. You only need to set them manually when making direct POST calls.
| Attribute | Required | Description |
| ---------------------------- | -------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `splunk_ao.project.name` | No | Project name. Overrides the `project` header if set |
| `splunk_ao.experiment.id` | No | Experiment ID. Routes the trace to an experiment |
| `splunk_ao.agentstream.name` | No | Agent Stream name. Ignored when `splunk_ao.experiment.id` is set |
| `splunk_ao.session.id` | No | Session ID for grouping related traces |
| `splunk_ao.dataset.input` | No | Ground truth input for the dataset row (JSON string). Used by evaluators like [Ground Truth Adherence](/concepts/evaluators/response-quality/ground-truth-adherence) |
| `splunk_ao.dataset.output` | No | Ground truth for the dataset row (JSON string). Maps to the `ground_truth` dataset field. Used by evaluators like [Ground Truth Adherence](/concepts/evaluators/response-quality/ground-truth-adherence) |
| `splunk_ao.dataset.metadata` | No | Additional metadata for the dataset row (JSON string) |
### Request Body
The request body should contain OTLP packets in protobuf format (binary). The payload should be an `ExportTraceServiceRequest` message as defined in the [OpenTelemetry Protocol specification](https://opentelemetry.io/docs/specs/otlp/).
### Response Codes
The request was successfully processed. The response body contains an `ExportTraceServiceResponse` in JSON format.
If some spans were rejected, the response includes partial success information:
```json theme={null}
{
"partialSuccess": {
"rejectedSpans": 5,
"errorMessage": "Group 0: Run not found for agentstream ..."
}
}
```
Even when the HTTP status is 200, check for `partialSuccess` in the response to determine if all spans were successfully processed.
The API key is missing or invalid.
```json theme={null}
{
"detail": "API Key is missing"
}
```
The specified project was not found and could not be created.
```json theme={null}
{
"detail": "Project not found."
}
```
The `Content-Type` header is not `application/x-protobuf`.
```json theme={null}
{
"detail": " is not supported, content_type needs to be: 'application/x-protobuf'"
}
```
The request could not be processed. Common reasons include no spans found, trace processing failure, or missing Agent Stream ID.
```json theme={null}
{
"detail": "No spans found in request."
}
```
## Additional Resources
Official OpenTelemetry semantic conventions for GenAI agent spans.
OpenInference semantic conventions reference.
Set up OpenTelemetry and OpenInference with Splunk Agent Observability.
Step-by-step guide using OpenTelemetry with LangGraph and OpenAI.