Skip to main content

​
DecisionVersionOutcomeUnknownError

The version request failed without a definitive rejection, so the scorer was kept. A rate limit (429) is also treated as unknown, so it leaves a scorer without a version; delete it by id before retrying with the same name.

​
Attributes

scorer_id (str): The scorer that may or may not have a version.

​
_VersionRejected

The server answered the version request with a validation error or no body, so no version exists.

​
Evaluators

​
create_custom_llm_evaluator

Create a custom LLM metric. Arguments
  • name (str): Name of the metric.
  • user_prompt (str): User prompt for the metric.
  • node_level (StepType): Node level for the metric.
  • cot_enabled (bool): Whether chain-of-thought is enabled.
  • model_name (str): Model name to use.
  • num_judges (int): Number of judges for the metric.
  • description (str): Description of the metric.
  • tags (List[str]): Tags associated with the metric.
  • output_type (OutputTypeEnum): Output type for the metric.
  • ground_truth (bool): Whether the scorer requires ground truth (reference_output) from the dataset. When True, the judge LLM receives the row’s ground-truth value in its prompt.
Returns
  • BaseScorerVersionResponse: Response containing the created metric details.

​
create_decision_evaluator

Create a decision evaluator. Arguments
  • name (str): Name of the evaluator.
  • question (str): The question the decision model answers (1-2,000 characters).
  • node_level (StepType): The single node type the evaluator scores.
  • model_name (str): Decision model registry version, for example jev-1.13.0.
  • output_type (OutputTypeEnum | str | None): BOOLEAN_PROBABILITY, CATEGORICAL or DISCRETE. When None, it is inferred from answer_options: None gives BOOLEAN_PROBABILITY, strings give CATEGORICAL and levels give DISCRETE. An explicit value that does not match answer_options raises ValueError.
  • answer_options (list[str] | list[DecisionLevel | dict] | None): None for BOOLEAN_PROBABILITY. For CATEGORICAL, a list of 2-255 unique labels. For DISCRETE, 2-10 DecisionLevel objects (or \{level, label, description\} dicts) numbered 0, 1, 2 and so on in order.
  • input_type (InputTypeEnum | None): What the model sees. When None, the server picks a default for the node type.
  • description (str): Description of the evaluator.
  • tags (List[str]): Tags associated with the evaluator.
  • ground_truth (bool): Whether the scorer requires ground truth (reference_output) from the dataset.
Raises
  • ValueError: If an argument is invalid or the API answers with a validation error. If the scorer was created but its version was rejected, the scorer is deleted before the error is raised.
  • SplunkAOAPIError: For a definitive rejection (400, 403, 404, 422), for example 403 when the feature is not enabled. The scorer is deleted before the error is raised.
  • DecisionVersionOutcomeUnknownError: For a timeout, network error, 5xx or any other failure. The version may have been saved, so the scorer is kept and the error names its id.
Returns
  • BaseScorerVersionResponse: Response containing the created evaluator version.

​
create_custom_llm_evaluator

Create a custom LLM metric. Arguments
  • name (str): Name of the metric.
  • user_prompt (str): User prompt for the metric.
  • node_level (StepType): Node level for the metric.
  • cot_enabled (bool): Whether chain-of-thought is enabled.
  • model_name (str): Model name to use.
  • num_judges (int): Number of judges for the metric.
  • description (str): Description of the metric.
  • tags (List[str]): Tags associated with the metric.
  • output_type (OutputTypeEnum): Output type for the metric.
  • ground_truth (bool): Whether the scorer requires ground truth (reference_output) from the dataset. When True, the judge LLM receives the row’s ground-truth value in its prompt.
Returns
  • BaseScorerVersionResponse: Response containing the created metric details.

​
create_decision_evaluator

Create a decision evaluator. A decision evaluator asks a decision model one typed question and scores the node with the probabilities of the answers. Arguments
  • name (str): Name of the evaluator.
  • question (str): The question the decision model answers (1-2,000 characters).
  • node_level (StepType): The single node type the evaluator scores.
  • model_name (str): Decision model registry version, for example jev-1.13.0.
  • output_type (OutputTypeEnum | str | None): BOOLEAN_PROBABILITY, CATEGORICAL or DISCRETE. When None, it is inferred from answer_options: None gives BOOLEAN_PROBABILITY, strings give CATEGORICAL and levels give DISCRETE. An explicit value that does not match answer_options raises ValueError.
  • answer_options (list[str] | list[DecisionLevel | dict] | None): None for BOOLEAN_PROBABILITY. For CATEGORICAL, a list of 2-255 unique labels. For DISCRETE, 2-10 DecisionLevel objects (or \{level, label, description\} dicts) numbered 0, 1, 2 and so on in order.
  • input_type (InputTypeEnum | None): What the model sees. When None, the server picks a default for the node type.
  • description (str): Description of the evaluator.
  • tags (List[str]): Tags associated with the evaluator.
  • ground_truth (bool): Whether the scorer requires ground truth (reference_output) from the dataset.
Raises
  • ValueError: If an argument is invalid or the API answers with a validation error. If the scorer was created but its version was rejected, the scorer is deleted before the error is raised.
  • SplunkAOAPIError: For a definitive rejection (400, 403, 404, 422), for example 403 when the feature is not enabled. The scorer is deleted before the error is raised.
  • DecisionVersionOutcomeUnknownError: For a timeout, network error, 5xx or any other failure. The version may have been saved, so the scorer is kept and the error names its id.
Returns
  • BaseScorerVersionResponse: Response containing the created evaluator version.
Examples from splunk_ao.evaluators import create_decision_evaluator from splunk_ao.resources.models import OutputTypeEnum create_decision_evaluator( name=“tone”, question=“Is the tone of this response professional?”, output_type=OutputTypeEnum.CATEGORICAL, answer_options=[“professional”, “casual”, “rude”], )

​
delete_evaluator

Deletes a metric by its name. Arguments
  • name: The name of the metric to delete.

​
get_evaluators

Queries for metrics in a project. Arguments
  • project_id: The unique identifier of the project.
  • start_time: The start of the time range for the query.
  • end_time: The end of the time range for the query.
  • experiment_id: Filter records by a specific experiment ID.
  • agent_stream_id: Filter records by a specific agent stream ID.
  • filters: A list of filters to apply to the query.
  • group_by: The field to group the results by.
  • interval: The time interval for the query in seconds.
Returns
  • LogRecordsMetricsResponse: A LogRecordsMetricsResponse object containing the query results, or None if the query fails.

​
normalize_decision_answer_options

Convert user-facing answer options into the request shape. Strings pass through (categorical). DecisionLevel objects and \{level, label, description\} dicts become DecisionLevel (discrete). None stays None (boolean_probability).

​
resolve_decision_output_type

Return the explicit output type, or infer it from the normalized answer options. None options mean boolean_probability, strings mean categorical and levels mean discrete. An explicit value wins, and the request validation then rejects a mismatch with the options.

​
validate_decision_request

Reject requests the server would refuse, before anything is created. Checks the question length (1-2,000 characters when given) and that the answer options match the output type: none for boolean_probability, 2-255 unique labels for categorical, 2-10 levels numbered 0, 1, 2 and so on in order for discrete.