Skip to main content
Definition: Context Adherence is a measurement of closed-domain hallucinations: cases where your model said things that were not provided in the context. If a response is adherent to the context (i.e. it has a value of 1 or close to 1), it only contains information given in the context. If a response is not adherent (i.e. it has a value of 0 or close to 0), it’s likely to contain facts not included in the context provided to the model.

​
Context adherence with Luna-2

You can also leverage Splunk Agent Observability’s proprietary Evaluation SLMs to calculate context adherence. Context Adherence Luna is computed using Splunk Agent Observability in-house small language models (Luna-2). Context Adherence Luna is a cost-effective way to scale up your RAG evaluation workflows. To leverage Luna-2 for context adherence or other evaluators, reach out to our team.

​
Performance Benchmarks

We evaluated Context Adherence against human expert labels on an internal dataset of RAG samples using top frontier models.

​
GPT-4.1 Classification Report

Benchmarks based on internal evaluation dataset. Performance may vary by use case.
If you would like to dive deeper or start implementing Context Adherence, check out the following resources:

​
Examples

  • Context Adherence Examples - Log into Splunk Agent Observability and explore the “Context Adherence” Agent Stream in the “Preset Evaluator Examples” Project to see this evaluator in action.