Skip to main content
This workflow requires network access to Splunk Agent Observability APIs. If you get an access denied error or cannot reach the Splunk Agent Observability endpoints, talk to your DevOps or Splunk Agent Observability administrator before proceeding.

​
Prerequisites

Set these environment variables:
  • SPLUNK_AO_API_URL
  • SPLUNK_AO_API_KEY (only required if you are using an on-premises, standalone, or custom deployment)

​
Steps to register an evaluator in Splunk Agent Observability

There are two steps to register an evaluator in Splunk Agent Observability:

​
1. Upload LoRA weights

Use the SDK helper to upload your fine-tuned LoRA artifacts to Splunk Agent Observability via the Splunk Agent Observability API:
upload_finetuned_model_weights(...) reads SPLUNK_AO_API_URL and SPLUNK_AO_API_KEY from the environment, requests upload URLs from Splunk Agent Observability, uploads the required LoRA files from lora_files_path, and returns both the resolved lora_task_id and the Splunk Agent Observability-side lora_weights_path. If you do not provide lora_task_id, the SDK automatically picks the first free task id at or above lora_task_id_min. The directory passed in lora_files_path should contain:
  • model.lora_weights.npy
  • model.lora_config.npy
  • adapter_config.json
  • adapter_model.safetensors

​
2. Register the evaluator in Splunk Agent Observability

Once your weights are uploaded and you have the returned weights_path, register (or reuse) a scorer and create a new Luna scorer version using the SDK:

​
How to set the upload arguments

  • lora_task_id [optional]: provide this only if you want to force a specific LoRA task id. If omitted, the SDK auto-selects one for you.
  • lora_task_id_min [optional]: minimum LoRA task id to consider when auto-picking. In most cases, leave the default.
  • lora_files_path: the local directory that contains the LoRA artifact files produced by training.
  • timeout_seconds [optional]: request timeout used for the prepare-upload call and the file uploads. In most cases, the default is fine.

​
How to set the registration arguments

  • metric_name: the name of the scorer in Splunk Agent Observability. Use a stable, human-readable evaluator name.
  • prompt_template: the prompt template used for the evaluator. Make sure to use the same template shape you trained with. It is also available in the output artifact of the training run.
  • luna_input_type: the Splunk Agent Observability Luna input type enum that matches the evaluator input shape. Available options are: span, trace_object, trace_input_output_only.
  • luna_output_type: the Splunk Agent Observability Luna output type enum that matches the evaluator output shape. Available options are: float (for boolean evaluators), string (for categorical evaluators).
  • lora_task_id: the LoRA task id associated with the uploaded fine-tuned weights. Pass the lora_task_id returned by upload_finetuned_model_weights(...).
  • weights_path: the Splunk Agent Observability-side storage path returned by upload_finetuned_model_weights(...). Pass the lora_weights_path value from the upload response.

​
Use the registered evaluator

After registration, the evaluator becomes available inside Splunk Agent Observability (UI / API) as a Luna scorer.