Add Domain-Specific Custom Evaluators to your Application

Overview

In this tutorial, you’ll learn how to add custom evaluations to a comedy multi-agent LLM app using Splunk Agent Observability. This tutorial is intended for Python developers building domain-specific AI applications. It assumes you have basic knowledge of:

Some familiarity with Python/Flask
Python Package Manager of choice (we’ll be using uv)
Code editor of choice (VS Code, Cursor, Warp, etc.)
API keys for:
- OpenAI, NewsAPI (free!), and Splunk Agent Observability (free!)

By the end of this tutorial, you’ll be able to:

Understand the importance of domain-expertise in Splunk Agent Observability
Create a custom LLM-as-a-Judge evaluator to measure outputs

Background

For the sake of jumping right into action — we’ll be starting from an existing application and demonstrating how to add custom evaluators to an existing application. The app we’ll be building off of is the Startup Sim 3000, an LLM-based Python application that generates either serious or silly startup pitches using OpenAI and real-time data. The app includes two agent chains:

Serious Mode: Uses NewsAPI data and GPT-4 to generate business-style startup pitches
Silly Mode: Uses HackerNews headlines to inspire parody pitches of absurd tech startups

We’ll demonstrate how to observe the agent’s performance and evaluate humor or business quality with custom evaluators, using Splunk Agent Observability for session tracking and LLM-as-a-Judge evaluators.

Create a new Splunk Agent Observability project

In order to set up custom evaluators, we’ll need a Splunk Agent Observability project to log evaluations to first.

Create a new project from the Splunk Agent Observability UI

If you haven’t already, log into Splunk Agent Observability and create a account. When prompted, add an organization name.

For more information on getting started, see the Quickstart.

Create a new project by clicking on the Create new project button on the upper right hand screen. You will be prompted to add a project name, as well as an Agent Stream name.

Get your Splunk Agent Observability API Keys

Click on the profile icon in the upper right hand side of the page, navigate on the drop-down menu to API keys.

From the API Keys screen, select Create New Key. Save the key within your environment file with the project name and Agent Stream name you’ve created.Run your app again, and you’ll now be able to see the logs appear within Splunk Agent Observability.

Set up the project

Clone the project in your IDE of choice.

The starter project is in the examples/agent/startup-simulator-3000 folder in the cloned repo.

git clone https://github.com/splunk/splunk-ao-python

Set up a virtual environment and install dependencies.

A virtual environment keeps your project’s dependencies isolated from your global Python installation. For this we’ll be using uv.On Windows:

uv venv
source .venv\Scripts\activate 
uv pip install -r requirements.txt

On MacOS/Linux:

uv venv
source .venv/bin/activate 
uv pip install -r requirements.txt

This creates and activates a virtual environment for your project, then installs the necessary requirements.

Configure your .env file.

Take the .example.env file, copy it, renaming it to .env and add in your own variables. Be sure the variables are added to your .gitignore file.When complete, it should look something like this:

.env

# Example .env file — copy this file to .env and fill in the values. 
# Be sure to add the .env file to your .gitignore file.

# LLM API Key (required)
# For regular keys: sk-...
# For project-based keys: sk-proj-...
OPENAI_API_KEY=your-openai-api-key-here
# OpenAI Project ID (optional for project-based keys)
# OPENAI_PROJECT_ID=your-openai-project-id-here

# Required for on-premises, standalone, custom deployments
# SPLUNK_AO_API_KEY="your-splunk-ao-api-key"
# SPLUNK_AO_CONSOLE_URL="your-splunk-ao-url"

# Required for SaaS (Splunk Observability Cloud) deployments
# SPLUNK_AO_O11Y_TOKEN="your-splunk-o11y-token"
# SPLUNK_AO_REALM="your-splunk-o11y-deployment-realm"

# Splunk Agent Observability Details
SPLUNK_AO_PROJECT=your project name here
SPLUNK_AO_AGENT_STREAM=my_agent_stream

# Optional LLM configuration
LLM_MODEL=gpt-4
LLM_TEMPERATURE=0.7

# Optional agent configuration
VERBOSITY=low  # Options: none, low, high
ENVIRONMENT=development
ENABLE_LOGGING=true
ENABLE_TOOL_SELECTION=true

Start the Flask app and test out the application

After your environment variables are set, you are all set to run the application. The application is designed as a Flask Application, with a JavaScript frontend, and Python backend.Run the application locally by running the following in the terminal.

python web_server.py

Your application will be running at http://localhost:2021 — open that within your browser and start exploring.Try generating both “Silly” and “Serious” mode pitches.The standard flow of the application is as follows: User Input → Flask Route → Agent Coordinator → Tool Chain → AI Model → Response

Create a custom LLM-as-a-Judge evaluator in Splunk Agent Observability

Add Evaluators to your Agent Stream in Splunk Agent Observability

Navigate to your project home inside of Splunk Agent Observability. Select the Agent Stream you’ve got your traces in.Click on the Trace tab and see your most recent runs listed below.From this view, navigate to the upper right hand side of your screen and click on the Configure Evaluators button. A side panel should appear with a set of different evaluators from you to choose from.

Add custom Evaluators

That’s where custom evaluators come in.Once in this panel, navigate to the Create Evaluator in the upper right hand corner of your screen, and select LLM-as-a-Judge Evaluator.

Create your own LLM-as-a-Judge prompt

A window will appear where you will be able to generate an evaluator prompt. An evaluator prompt is a prompt to prompt the creation of the final LLM as a judge prompt. This is to ensure you spend your time focused on what’s important (the success criteria) instead of worrying about the output format.

When writing a good prompt, remember that the goal is to transform subjective evaluation criteria into a consistent, repeatable process that a language model can assess.Use a specific, structured evaluator for best results. For this example, I’ve provided a sample custom evaluator below.

You are an expert humor judge specializing in startup culture satire and tech
industry parody. Your role is to evaluate the humor and effectiveness of
startup-related content generated by an AI system.

EVALUATION CRITERIA:
For each criterion, answer TRUE if the content meets the standard,
FALSE if it doesn't.

1. SATIRE EFFECTIVENESS
   - [ ] Content clearly parodies startup culture tropes
   - [ ] Parody is recognizable to tech industry insiders
   - [ ] Maintains balance between believable and absurd
   - [ ] Successfully mocks common startup practices
2. HUMOR CONSISTENCY
   - [ ] Humor level remains consistent throughout
   - [ ] No significant drops in comedic quality
   - [ ] Tone remains appropriate for satire
   - [ ] Jokes build upon each other effectively
3. CULTURAL RELEVANCE
   - [ ] References are current and timely
   - [ ] Captures current startup culture trends
   - [ ] Buzzwords are accurately parodied
   - [ ] Industry-specific knowledge is evident
4. NARRATIVE COHERENCE
   - [ ] Story follows internal logic
   - [ ] Pivots make sense within context
   - [ ] Character/voice remains consistent
   - [ ] Plot points connect logically
5. ORIGINALITY
   - [ ] Avoids overused startup jokes
   - [ ] Contains unique elements
   - [ ] Offers fresh perspective
   - [ ] Surprises the audience
6. TECHNICAL ACCURACY
   - [ ] Startup concepts are correctly parodied
   - [ ] Industry terminology is used appropriately
   - [ ] Business concepts are accurately mocked
   - [ ] Technical details are correctly referenced
 Answer TRUE only if ALL of the following conditions are met:
   - [ ] At least 80% of all criteria are rated TRUE
   - [ ] No critical criteria (Satire Effectiveness, Humor Consistency) 
         are rated FALSE
   - [ ] Content would be considered funny by the target audience
   - [ ] Satire successfully achieves its intended purpose
   - [ ] Content maintains appropriate tone throughout

When added, press Save then test your evaluator. The evaluation prompt will then be generated for you to see within a preview window.

Test your evaluator

From within the Custom Evaluator pane, select Test Evaluator.

Take the output from an earlier run, and paste it in the output section of the Test Evaluator page, and check the response.Continue tweaking the prompt until you have an evaluator that you feel confident with — the goal isn’t to have an evaluator that is perfect 100% of the time, but helps you determine what “good” looks like.Have examples of what a subject matter expert would consider to be “good” and “bad” to test your evaluator for success.

Add your Custom Evaluator

Once tested, your evaluator will appear in the list of available evaluators. Click on Configure Evaluators and flip the toggle the evaluator you’ve created, on. From there, it can be used to assess future LLM outputs across runs and sessions—giving you visibility into quality over time.

Summary

In this tutorial, you learned how to:

Configure and observe spans in a creative AI agent app
Translate your domain expertise into a measurable AI quality rubric
Build a custom evaluator with LLM-as-a-Judge to evaluate startup pitches

Next steps

Check out other cookbooks in the Splunk Agent Observability cookbook library
Explore how to create custom evaluators in Splunk Agent Observability using code
Learn more about the different out-of-the-box-evaluators available in Splunk Agent Observability

Overview

Agentic AI

Conversational AI

Retrieval-Augmented Generation

Add Domain-Specific Custom Evaluators to your Application

Overview

Background

Create a new Splunk Agent Observability project

Set up the project

Create a custom LLM-as-a-Judge evaluator in Splunk Agent Observability

Summary

Next steps

​Overview

​Background

​Create a new Splunk Agent Observability project

​Set up the project

​Create a custom LLM-as-a-Judge evaluator in Splunk Agent Observability

​Summary

​Next steps

Overview

Background

Create a new Splunk Agent Observability project

Set up the project

Create a custom LLM-as-a-Judge evaluator in Splunk Agent Observability

Summary

Next steps