> ## Documentation Index
> Fetch the complete documentation index at: https://agent-observability-docs.splunk.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Add Domain-Specific Custom Evaluators to your Application

> Learn how to create custom LLM-as-a-Judge evaluators to evaluate domain-specific applications within Splunk Agent Observability

## Overview

In this tutorial, you'll learn how to add custom evaluations to a comedy multi-agent LLM app using Splunk Agent Observability. This tutorial is intended for Python developers building domain-specific AI applications. It assumes you have basic knowledge of:

* Some familiarity with Python/Flask
* Python Package Manager of choice (we'll be using [uv](https://docs.astral.sh/uv/))
* Code editor of choice (VS Code, Cursor, Warp, etc.)
* API keys for:
  * [OpenAI](https://platform.openai.com/docs/overview), [NewsAPI (free!)](https://newsapi.org/), and Splunk Agent Observability (free!)

By the end of this tutorial, you'll be able to:

* Understand the importance of domain-expertise in Splunk Agent Observability
* Create a custom LLM-as-a-Judge evaluator to measure outputs

## Background

For the sake of jumping right into action — we'll be starting from an [existing application](https://github.com/splunk/splunk-ao-python/tree/main/examples) and demonstrating how to add custom evaluators to an existing application.

The app we'll be building off of is the Startup Sim 3000, an LLM-based Python application that generates either serious or silly startup pitches using OpenAI and real-time data. The app includes two agent chains:

* **Serious Mode**: Uses NewsAPI data and GPT-4 to generate business-style startup pitches
* **Silly Mode**: Uses HackerNews headlines to inspire parody pitches of absurd tech startups

We'll demonstrate how to observe the agent's performance and evaluate humor or business quality with custom evaluators, using Splunk Agent Observability for session tracking and LLM-as-a-Judge evaluators.

## Create a new Splunk Agent Observability project

In order to set up custom evaluators, we'll need a Splunk Agent Observability project to log evaluations to first.

**1. Create a new project from the Splunk Agent Observability UI**

If you haven't already, log into Splunk Agent Observability and create a account. When prompted, add an organization name.

> [!NOTE]
>
> For more information on getting started, see the [Quickstart](https://agent-observability-docs.splunk.com/getting-started/quickstart.md).

Create a new project by clicking on the **Create new project** button on the upper right hand screen.   You will be prompted to add a project name, as well as an Agent Stream name.

**2. Get your Splunk Agent Observability API Keys**

Click on the profile icon in the upper right hand side of the page, navigate on the drop-down menu to API keys.

![The user menu](https://mintcdn.com/agent-observability-docs/i6IIDOves2sy1VUe/images/console-ui/user-menu-sao.png?fit=max\&auto=format\&n=i6IIDOves2sy1VUe\&q=85\&s=e927eb14c7815f5ccae0c0c1e43eb21e)

From the API Keys screen, select **Create New Key**.  Save the key within your environment file with the project name and Agent Stream name you've created.

Run your app again, and you'll now be able to see the logs appear within Splunk Agent Observability.

## Set up the project

**1. Clone the project in your IDE of choice.**

The starter project is in the `examples/agent/startup-simulator-3000` folder in the cloned repo.

```bash theme={null}
git clone https://github.com/splunk/splunk-ao-python
```

**2. Set up a virtual environment and install dependencies.**

A virtual environment keeps your project's dependencies isolated from your global Python installation.  For this we'll be using [uv](https://docs.astral.sh/uv/).

On Windows:

```bash theme={null}
uv venv
source .venv\Scripts\activate 
uv pip install -r requirements.txt
```

On MacOS/Linux:

```bash theme={null}
uv venv
source .venv/bin/activate 
uv pip install -r requirements.txt
```

This creates and activates a virtual environment for your project, then installs the necessary requirements.

**3. Configure your .env file.**

Take the `.example.env` file, copy it, renaming it to `.env` and add in your own variables. Be sure the variables are added to your `.gitignore` file.

When complete, it should look something like this:

```ini .env theme={null}
# Example .env file — copy this file to .env and fill in the values. 
# Be sure to add the .env file to your .gitignore file.

# LLM API Key (required)
# For regular keys: sk-...
# For project-based keys: sk-proj-...
OPENAI_API_KEY=your-openai-api-key-here
# OpenAI Project ID (optional for project-based keys)
# OPENAI_PROJECT_ID=your-openai-project-id-here

# Required for on-premises, standalone, custom deployments
# SPLUNK_AO_API_KEY="your-splunk-ao-api-key"
# SPLUNK_AO_CONSOLE_URL="your-splunk-ao-url"

# Required for SaaS (Splunk Observability Cloud) deployments
# SPLUNK_AO_O11Y_TOKEN="your-splunk-o11y-token"
# SPLUNK_AO_REALM="your-splunk-o11y-deployment-realm"

# Splunk Agent Observability Details
SPLUNK_AO_PROJECT=your project name here
SPLUNK_AO_AGENT_STREAM=my_agent_stream

# Optional LLM configuration
LLM_MODEL=gpt-4
LLM_TEMPERATURE=0.7

# Optional agent configuration
VERBOSITY=low  # Options: none, low, high
ENVIRONMENT=development
ENABLE_LOGGING=true
ENABLE_TOOL_SELECTION=true
```

**4. Start the Flask app and test out the application**

After your environment variables are set, you are all set to run the application.  The application is designed as a Flask Application, with a JavaScript frontend, and Python backend.

Run the application locally by running the following in the terminal.

```terminal theme={null}
python web_server.py
```

Your application will be running at [http://localhost:2021](http://localhost:2021) — open that within your browser and start exploring.

Try generating both "Silly" and "Serious" mode pitches.

The standard flow of the application is as follows:
User Input → Flask Route → Agent Coordinator → Tool Chain → AI Model → Response

## Create a custom LLM-as-a-Judge evaluator in Splunk Agent Observability

**1. Add Evaluators to your Agent Stream in Splunk Agent Observability**

Navigate to your project home inside of Splunk Agent Observability. Select the Agent Stream you've got your traces in.

Click on the **Trace** tab and see your most recent runs listed below.

From this view, navigate to the upper right hand side of your screen and click on the **Configure Evaluators** button.  A side panel should appear with a set of different evaluators from you to choose from.

**2. Add custom Evaluators**

That's where custom evaluators come in.

Once in this panel, navigate to the **Create Evaluator** in the upper right hand corner of your screen, and select **LLM-as-a-Judge** Evaluator.

**3. Create your own LLM-as-a-Judge prompt**

A window will appear where you will be able to generate an evaluator prompt. An evaluator prompt is a prompt to prompt the creation of the final LLM as a judge prompt. This is to ensure you spend your time focused on what's important (the success criteria) instead of worrying about the output format.

![Create a custom LLMAJ evaluator](https://mintcdn.com/agent-observability-docs/837zSZ4Vo0rxb9Cv/images/console-ui/create-custom-evaluator-llaj-sao.png?fit=max\&auto=format\&n=837zSZ4Vo0rxb9Cv\&q=85\&s=a3bb559c3fcfe66bdd96d022d7a061bd)

When writing a good prompt, remember that the goal is to transform **subjective evaluation criteria** into a consistent, repeatable process that a language model can assess.

Use a specific, structured evaluator for best results.  For this example, I've provided a sample custom evaluator below.

```markdown theme={null}
You are an expert humor judge specializing in startup culture satire and tech
industry parody. Your role is to evaluate the humor and effectiveness of
startup-related content generated by an AI system.

EVALUATION CRITERIA:
For each criterion, answer TRUE if the content meets the standard,
FALSE if it doesn't.

1. SATIRE EFFECTIVENESS
   - [ ] Content clearly parodies startup culture tropes
   - [ ] Parody is recognizable to tech industry insiders
   - [ ] Maintains balance between believable and absurd
   - [ ] Successfully mocks common startup practices
2. HUMOR CONSISTENCY
   - [ ] Humor level remains consistent throughout
   - [ ] No significant drops in comedic quality
   - [ ] Tone remains appropriate for satire
   - [ ] Jokes build upon each other effectively
3. CULTURAL RELEVANCE
   - [ ] References are current and timely
   - [ ] Captures current startup culture trends
   - [ ] Buzzwords are accurately parodied
   - [ ] Industry-specific knowledge is evident
4. NARRATIVE COHERENCE
   - [ ] Story follows internal logic
   - [ ] Pivots make sense within context
   - [ ] Character/voice remains consistent
   - [ ] Plot points connect logically
5. ORIGINALITY
   - [ ] Avoids overused startup jokes
   - [ ] Contains unique elements
   - [ ] Offers fresh perspective
   - [ ] Surprises the audience
6. TECHNICAL ACCURACY
   - [ ] Startup concepts are correctly parodied
   - [ ] Industry terminology is used appropriately
   - [ ] Business concepts are accurately mocked
   - [ ] Technical details are correctly referenced
 Answer TRUE only if ALL of the following conditions are met:
   - [ ] At least 80% of all criteria are rated TRUE
   - [ ] No critical criteria (Satire Effectiveness, Humor Consistency) 
         are rated FALSE
   - [ ] Content would be considered funny by the target audience
   - [ ] Satire successfully achieves its intended purpose
   - [ ] Content maintains appropriate tone throughout
```

When added, press **Save** then test your evaluator. The evaluation prompt will then be generated for you to see within a preview window.

**4. Test your evaluator**

From within the Custom Evaluator pane, select **Test Evaluator**.

![Testing an evaluator](https://mintcdn.com/agent-observability-docs/tQcIFWBC_UEfM1wy/images/console-ui/test-evaluator-sao.png?fit=max\&auto=format\&n=tQcIFWBC_UEfM1wy\&q=85\&s=28f2565e6eb978b7daaad2718655e4d3)

Take the output from an earlier run, and paste it in the output section of the **Test Evaluator** page, and check the response.

Continue tweaking the prompt until you have an evaluator that you feel confident with — the goal isn't to have an evaluator that is perfect 100% of the time, but helps you determine what "good" looks like.

Have examples of what a subject matter expert would consider to be "good" and "bad" to test your evaluator for success.

**5. Add your Custom Evaluator**

Once tested, your evaluator will appear in the list of available evaluators.  Click on **Configure Evaluators** and flip the toggle the evaluator you've created, on. From there, it can be used to assess future LLM outputs across runs and sessions—giving you visibility into quality over time.

## Summary

In this tutorial, you learned how to:

* Configure and observe spans in a creative AI agent app
* Translate your domain expertise into a measurable AI quality rubric
* Build a custom evaluator with LLM-as-a-Judge to evaluate startup pitches

## Next steps

* Check out other cookbooks in the [Splunk Agent Observability cookbook library](https://agent-observability-docs.splunk.com/cookbooks/overview.md)
* Explore how to create custom evaluators in [Splunk Agent Observability using code](https://agent-observability-docs.splunk.com/concepts/evaluators/custom-evaluators/custom-evaluators-ui-code.md)
* Learn more about the different [out-of-the-box-evaluators available](https://agent-observability-docs.splunk.com/concepts/evaluators/overview.md) in Splunk Agent Observability


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.