> ## Documentation Index
> Fetch the complete documentation index at: https://agent-observability-docs.splunk.com/llms.txt
> Use this file to discover all available pages before exploring further.
# Simple Chatbot
> Get started with the simple chatbot sample project
The simple chatbot sample project is a demo of a simplistic terminal-based LLM chatbot where you can have a back-and-forth conversation with an LLM. This project comes pre-populated with an Agent Stream with traces and evaluators, as well as insights to help you improve this project.
Get the code for the sample project. You can find this project by cloning the Splunk Agent Observability Python SDK repo.
The code for this project is in the `/examples/chatbot/sample-project-chatbot/` folder.
The code for this sample is available in Python, and you can run this code using a range of LLM providers to generate more traces, and experiment with improving the app based off the evaluations.
The sample code has 3 variations for the following LLM providers:
* [OpenAI](https://openai.com/api/), or any OpenAI API compatible endpoint, such as [Ollama](https://ollama.com) running locally, or [Google Vertex](https://cloud.google.com/vertex-ai/generative-ai/docs/migrate/openai/overview).
* [Anthropic](https://www.anthropic.com/api)
* [Azure AI Foundry](https://ai.azure.com) models using the Azure AI inference SDK
## Evaluate the app
The sample project comes with an Agent Stream pre-populated with a set of traces for some sample interactions with the chatbot - some serious, some asking nonsense questions.
### Investigate the Agent Stream
Navigate to the **Default Agent Stream** by selecting this project, and selecting the **Default Agent Stream** in the dashboard.
The Agent Stream is configured with the following evaluators:
* [Correctness](/concepts/evaluators/response-quality/correctness)
* [Instruction Adherence](/concepts/evaluators/response-quality/instruction-adherence)
For some of the traces, these evaluators are scored at 100%, showing the chatbot is working well for those inputs. For other traces, these evaluators are reporting lower values, showing the chatbot needs some improvements.
Select different rows to see more details, including the input and output data, the evaluator scores, and explanations.
## Run the sample app
You can run the sample app to generate more traces and test out different system prompts.
### Prerequisites
To run the code yourself to generate more traces, you will need:
* Access to an LLM, with one of:
* Access to an OpenAI compatible API, such as
* An OpenAI API key
* Access to an OpenAI compatible API, such as Google Vertex
* Ollama installed locally with a model downloaded
* An Anthropic API key
* An model compatible with the Azure AI Inference API deployed to Azure AI Foundry
* Either Python 3.10 or later, or Node installed
To get evaluators calculated in Splunk Agent Observability, you will need:
* An integration with an LLM configured. If you don't have an integration configured, then:
In the Splunk Agent Observability UI, navigate to the **LLM Integrations** page by selecting your user profile in the upper-right corner and then selecting **Integrations**.
Locate the LLM provider you are using (or specify a [custom integration](/sdk-api/third-party-integrations/model-integrations/custom-model-integrations/custom-model-integrations)), then select the **+Add Integration** button.
Specify settings for your integration (such as an API key), then select **Save changes**.
### Get the code
```bash Terminal theme={null}
git clone https://github.com/splunk/splunk-ao-python
```
Start by navigating to the root folder for the programming language you are using:
```bash Python theme={null}
cd examples/chatbot/sample-project-chatbot
```
Then navigate to the folder for the relevant LLM you are using:
```bash OpenAI compatible API theme={null}
cd openai-ollama
```
```bash Anthropic theme={null}
cd anthropic
```
```bash Azure AI Inference theme={null}
cd azure-inference
```
The full source code for all of our sample projects is available in the Splunk Agent Observability [SDK Examples GitHub repo](https://github.com/splunk/splunk-ao-python/tree/main/examples).
Check out sample projects using Splunk Agent Observability
### Run the code
From the project folder, Install the required dependencies. For Python, make sure to create and activate a virtual environment before installing the dependencies.
```bash Python theme={null}
pip install -r requirements.txt
```
In each project folder is a `.env.example` file. Rename this file to `.env` and populate the Splunk Agent Observability values:
| Environment Variable | Value |
| :----------------------- | :----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `SPLUNK_AO_API_KEY` | Only required if you are using an on-premises, standalone, or custom deployment. Your API key. |
| `SPLUNK_AO_PROJECT` | The name of your Splunk Agent Observability project - this is preset to `Simple Chatbot` |
| `SPLUNK_AO_AGENT_STREAM` | The name of your Agent Stream - this is preset to `Default Agent Stream` |
| `SPLUNK_AO_CONSOLE_URL` | Only required if you are using an on-premises, standalone, or custom deployment. This URL is displayed in the browser when you log into Splunk Agent Observability. For the free tier, you don't need to set this. |
You can find these values from the project page for the simple chatbot sample page in the Splunk Agent Observability UI.
Next populate the values for your LLM:
| Environment Variable | Value |
| :------------------- | :----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `OPENAI_API_KEY` | Your OpenAI API key. If you are using Ollama, set this to `ollama`. If you are using another OpenAI compatible API, then set this to the relevant API key. |
| `OPENAI_BASE_URL` | Optional. The base URL of your OpenAI deployment. Leave this commented out if you are using the default OpenAI API. If you are using Ollama, set this to `http://localhost:11434/v1`. If you are using another OpenAI compatible API, then set this to the relevant URL. |
| `MODEL_NAME` | The name of the model you are using |
| Environment Variable | Value |
| :------------------- | :---------------------------------- |
| `ANTHROPIC_API_KEY` | Your Anthropic API key |
| `MODEL_NAME` | The name of the model you are using |
| Environment Variable | Value |
| :---------------------------- | :--------------------------------------------- |
| `AZURE_AI_INFERENCE_ENDPOINT` | Your Azure AI Foundry endpoint |
| `AZURE_AI_INFERENCE_API_KEY` | Your Azure AI Foundry API key |
| `MODEL_NAME` | The name of the model deployment you are using |
Run the project with the following command:
```bash Python theme={null}
python app.py
```
The app will run in your terminal, and you can ask the LLM questions and get responses.
### Improve the app
The insights you viewed earlier suggested improving the system prompt. The default system prompt is defined in the following file:
```output Python theme={null}
app.py
```
In this file is the current system prompt, as well as a suggested improvement:
```python Python theme={null}
chat_history = [
{
"role": "system",
"content": """
You are a helpful assistant that can answer questions and provide
information. If you are not sure about the question, then try to
answer it to the best of your ability, including extrapolating or
guessing the answer from your training data.
""",
# This default system prompt can lead to hallucinations, so you
# might want to change it.
# For example, you could use a more restrictive prompt like:
# """
# You are a helpful assistant that can answer questions and provide
# information. If you don't know the answer, say "I don't know"
# instead of making up an answer. Do not under any circumstances
# make up an answer.
# """
}
]
```
Try commenting out the original system prompt, and uncomment the suggestion. Then restart the chatbot and interact with it, asking questions about made-up things to see how it responds.
Once you have asked a few questions, head back to the Splunk Agent Observability UI and examine the new traces. You should see the evaluators improving.
### Run the sample app as an experiment
Splunk Agent Observability allows you to run [experiments](/sdk-api/experiments/experiments) against [datasets](/sdk-api/experiments/datasets) of known data, generating traces in an experiment Agent Stream and evaluating these for different evaluators. Experiments allow you to take a known set of inputs and evaluate different prompts, LLMs, or versions of your apps.
This sample project has a unit test that runs the chatbot against a pre-defined dataset, containing a mixture of sensible and nonsense questions:
```json dataset.json theme={null}
[
{
"input": "Which continent is Spain in?"
},
{
"input": "Which continent is Japan in?"
},
{
"input": "Describe the running of the hippopotamus festival in Spain."
},
{
"input": "What is the estimated population of Querulous Quails in Florin."
},
{
"input": "Describe the famous Pudding Lane BBQ party"
}
...
]
```
You can use this unit test to evaluate different system prompts for your app.
Use the following command to run the unit test:
```python Python theme={null}
python -m pytest test.py
```
The unit test will output a link to the experiment in the Splunk Agent Observability UI:
```output Terminal theme={null}
Experiment simple-chatbot-experiment 2025-07-15 at 00:48:11.842 has
completed and results are available at
//experiments/
```
Follow this link to see the evaluators for the experiment Agent Stream.
Experiment with different system prompts. Edit the system prompt in the app, then re-run the experiment through the unit test to see how different system prompts affect the evaluators.
If you navigate to the experiments list, you will be able to compare the average evaluator values of each run.
You can then select multiple rows and [compare the experiments in detail](/concepts/experiments/compare).
## Next steps
### Logging with the SDKs
Learn how to run experiments with multiple data points using datasets and prompt templates
Log with full control over sessions, traces, and spans using the Splunk Agent Observability logger.
Quickly add logging to your code with the log decorator and wrapper.
Manage logging using the Splunk Agent Observability context manager.
### How-to guides
Learn how to integrate and use OpenAI's API with Splunk Agent Observability's wrapper client.
Python
Learn how to use the Splunk Agent Observability @log decorator to log functions to traces
Python
Learn how to create log traces and spans manually in your AI apps
Python
### SDK reference
The Splunk Agent Observability Python SDK reference.