Skip to main content
The Conversation Quality evaluator measures user satisfaction across an entire chatbot session by analyzing tone, engagement, and sentiment. It classifies each conversation as GOOD or BAD depending on whether the user’s overall experience reflects positive engagement or frustration directed at the bot. The evaluator focuses on conversational flow rather than task success, emphasizing how naturally and politely the user and bot interact. It excludes non-textual or purely action-based agent outputs (e.g., button clicks). This is a boolean evaluator that indicates whether the conversation quality is good.

​
Conversation Quality at a glance

​
When to use this evaluator

​
How to improve Conversation Quality scores

Some techniques to improve Conversation Quality scores are:
  • Ensure bots provide clear, empathetic, and concise responses
  • Detect and mitigate repeated clarification loops
  • Train models to de-escalate external frustration effectively
  • Log complete sessions to allow accurate tone assessment
Common issues that can cause low scores are:
  • Mislabeling external frustration as bot-directed
  • Incomplete logs
  • Abrupt session truncation

​
Performance Benchmarks

We evaluated Conversation Quality against human expert labels on an internal dataset of agentic conversation samples using top frontier models.

​
GPT-4.1 Classification Report

Benchmarks based on internal evaluation dataset. Performance may vary by use case.
If you would like to dive deeper or start implementing Conversation Quality, check out the following resources:

​
Examples

  • Conversation Quality Examples - Log into Splunk Agent Observability and explore the “Conversation Quality” Agent Stream in the “Preset Evaluator Examples” Project to see this evaluator in action.

​
How-to guides