LLM spans with RAG - Splunk Agent Observability

Use this tutorial when your evaluator depends on retrieved documents as context. This is the standard RAG pattern for Luna evaluators such as context adherence.

Dataset schema

Minimum columns:

documents: the retrieved context
at least one of:
- input: the user question
- output: the model answer
label: the ground-truth class for the evaluator

Config shape

Set:

data_generation.metric.input_format: "rag"
data_generation.source_data.dataset.columns.features to a subset of ["documents", "input", "output"]
generation.context_examples: 1

Minimal end-to-end config

run_steps:
  - data_generation
  - training

pipeline_provider: "local"
metric_name: "custom"

data_generation:
  metric:
    name: "Context Adherence"
    type: "binary"
    input_format: "rag"
    llmaj_source_prompt: "Determine whether the answer is consistent with the retrieved documents."
  source_data:
    dataset:
      source_type: "huggingface"
      huggingface:
        name: "context_adherence_dataset"
  generation:
    context_examples: 1
  output:
    dataset:
      repo_name: "context-adherence-training"

training:
  dataset:
    name: "context-adherence-training"
  prompt_template: |
    Determine whether the answer is consistent with the retrieved documents.
    Question:
    {input}

    Documents:
    {documents}

    Answer:
    {output}

    Respond with "true" or "false".
  output:
    model_name: "context-adherence-model"

⌘I