Skip to main content

​
Human Labelled Test Dataset

A golden set is the set of data points that is the most representative of your real-life production data, which are human-labelled according to the definition of the evaluator. A golden test set is crucial for robust model development and evaluation. It serves as the single source of truth against which all model performance is measured.

​
Required dataset format

In the Luna Studio UI, source data can come from a local CSV file, a CSV file at an HTTP(S) URL, or a connected Splunk Agent Observability workspace. Although the source picker accepts .jsonl uploads and URLs during ingestion, the current validation and training pipeline expects CSV data; convert JSONL to CSV before using it in a run. Hugging Face sources are available in the standalone SDK, not as an in-app dataset source. Each row should represent one example. The required feature columns depend on the evaluator shape. Test sets and labelled training sets must also have a label column for the ground-truth label; raw unlabelled training logs omit it until you start Label with evaluator prompt in the run wizard. Read more in Core concepts, Test sets, and Dataset validation.

​
Advanced formats

Full-trace and full-session fine-tuning are not available in the Luna Studio UI. The standalone SDK supports advanced label-only or direct-training workflows for these formats; see Full traces and Full sessions. Labels should be manually assigned and should match the exact evaluator definition you want to train or evaluate.

​
Required dataset size

Evaluator-aware validation in the run wizard enforces these minimums:
  • Test set: at least 300 rows total and 100 examples per class.
  • Labelled training set: at least 2,000 rows total and 100 examples per class.
For test sets, 1,000-3,000 representative human-labelled rows is a strong target. Keep class distributions reasonably balanced so evaluation results are meaningful across classes.

​
Training dataset guidance

If you add unlabelled training logs, choose Label with evaluator prompt, configure the labelling job, and start it. Luna Studio then validates the resulting labelled training set before fine-tuning. The class distribution should be similar to your test set distribution and should not be extremely skewed, for example 99/1. If you do not have enough training data, synthetic generation can help create training examples before fine-tuning.

​
LLM-as-a-Judge Prompt

You can select a preset evaluator, select a custom evaluator created in Splunk Agent Observability, or write a custom prompt directly in Step 1 of the Luna Studio run wizard. For Splunk Agent Observability-side authoring, see Create a new LLM-as-a-judge evaluator in the UI. It is important to ensure that the LLM-as-a-judge prompt has high accuracy on your golden dataset. If it does not, tune the prompt manually or use Autotune in Splunk Agent Observability before creating a Luna evaluator. This ensures that Luna fine-tuning starts with a good understanding of the evaluator and avoids garbage in, garbage out.