Skip to main content
The first step asks: what should this evaluator measure? You can pick from a curated list of templates, or click Use custom prompt to write your own.

​
Pick an evaluator

The Evaluator select is searchable. It lists preset and custom Splunk Agent Observability LLM-as-judge evaluators available to your organization, plus saved custom prompts from the current Luna Studio project. The picker is organized into three groups:
  • Splunk Agent Observability presets — built-in scorers from the Splunk Agent Observability catalog.
  • Custom Splunk Agent Observability evaluators — custom evaluators already created in Splunk Agent Observability.
  • Saved custom prompts — prompts previously authored in Luna Studio.
Only text evaluators with a supported fine-tuning contract can be selected. Other catalog entries remain visible but disabled, with the reason appended to their name. This includes evaluators with unsupported output types or input formats and multimodal evaluators. Selection eligibility does not guarantee registration eligibility. For example, Context relevance can be selected, fine-tuned, and evaluated, but its resulting evaluator cannot currently be registered. See Register an evaluator.

​
Inspect a selected template

Once you pick a template, the form expands to show a read-only Evaluator details panel:
Selected metric details

Step 1 with a built-in template selected, showing its read-only details

  • Output type — the evaluator’s return shape (Boolean, Categorical etc.). See Output types.
  • Input level — where the evaluator evaluates data, such as an LLM span or trace.
  • Evaluator shape — the dataset contract persisted with the evaluator, such as Input only, Output only, Input/output pair, RAG, or With tools.
  • Prompt — the LLM-as-judge prompt the template uses, in a read-only textarea.

​
Write a custom prompt

For evaluators that don’t fit a template, click the dropdown’s Use custom prompt option (with a + icon). The form switches into editable mode. In custom mode, you fill in:

​
Output types in detail

Other Splunk Agent Observability output types are not trainable in Luna Studio yet. The output type also constrains what label values your test set can use during validation. See Test sets.

​
Input levels in detail

Session does not appear as a selectable input level for custom evaluators because generic full-session fine-tuning and registration are not supported in Luna Studio yet.

​
Evaluator shapes

For RAG, output can also be present when the evaluator prompt needs it. Luna Studio displays Full trace as unavailable under the Trace input level and omits Session because those contracts cannot currently complete the UI fine-tune-and-register workflow.
The standalone Luna SDK supports advanced full-trace and full-session label-only workflows. Those SDK workflows do not make the same shapes trainable or registerable in the Luna Studio UI.

​
Prompt-writing tips

  • Be specific. Define exactly what counts as a positive vs negative result.
  • Give examples. One or two short examples per outcome class is plenty.
  • Constrain the output. End the prompt with something like “Respond with only true or false.” for Boolean evaluators.
  • Avoid open scales. “Score 1–10” is harder for an LLM judge to keep consistent than a binary or 3-class categorical.
Pro tip: For best results, we recommend first creating your evaluator in the Splunk Agent Observability UI and using the Autotune feature to test and refine it on a labelled test dataset. This helps you optimize the evaluator’s performance before launching a full training run in Luna Studio.

​
Where to go next

Step 2 — Test set

Pick the labelled dataset Luna evaluates against.

Test sets

Schema rules and best practices for evaluation data.