evaluators

​

Evaluators

​

create_custom_llm_evaluator

def create_custom_llm_evaluator(self,
                                name: str,
                                user_prompt: str,
                                node_level: StepType=StepType.llm,
                                cot_enabled: bool=True,
                                model_name: str='gpt-4.1-mini',
                                num_judges: int=3,
                                description: str='',
                                tags: list[str] | None=None,
                                output_type: OutputTypeEnum=OutputTypeEnum.BOOLEAN,
                                ground_truth: bool=False) -> BaseScorerVersionResponse

• name (str): Name of the metric.
• user_prompt (str): User prompt for the metric.
• node_level (StepType): Node level for the metric.
• cot_enabled (bool): Whether chain-of-thought is enabled.
• model_name (str): Model name to use.
• num_judges (int): Number of judges for the metric.
• description (str): Description of the metric.
• tags (List[str]): Tags associated with the metric.
• output_type (OutputTypeEnum): Output type for the metric.
• ground_truth (bool): Whether the scorer requires ground truth (reference_output) from the dataset. When True, the judge LLM receives the row’s ground-truth value in its prompt.

• BaseScorerVersionResponse: Response containing the created metric details.

​

create_custom_llm_evaluator

def create_custom_llm_evaluator(name: str,
                                user_prompt: str,
                                node_level: StepType=StepType.llm,
                                cot_enabled: bool=True,
                                model_name: str='gpt-4.1-mini',
                                num_judges: int=3,
                                description: str='',
                                tags: list[str] | None=None,
                                output_type: OutputTypeEnum=OutputTypeEnum.BOOLEAN,
                                ground_truth: bool=False) -> BaseScorerVersionResponse

• name (str): Name of the metric.
• user_prompt (str): User prompt for the metric.
• node_level (StepType): Node level for the metric.
• cot_enabled (bool): Whether chain-of-thought is enabled.
• model_name (str): Model name to use.
• num_judges (int): Number of judges for the metric.
• description (str): Description of the metric.
• tags (List[str]): Tags associated with the metric.
• output_type (OutputTypeEnum): Output type for the metric.
• ground_truth (bool): Whether the scorer requires ground truth (reference_output) from the dataset. When True, the judge LLM receives the row’s ground-truth value in its prompt.

• BaseScorerVersionResponse: Response containing the created metric details.

​

delete_evaluator

def delete_evaluator(name: str) -> None

• name: The name of the metric to delete.

​

get_evaluators

def get_evaluators(project_id: str,
                   start_time: datetime.datetime,
                   end_time: datetime.datetime,
                   experiment_id: str | None=None,
                   agent_stream_id: str | None=None,
                   filters: list[FilterType] | None=None,
                   group_by: str | None=None,
                   interval: int=5) -> LogRecordsMetricsResponse

• project_id: The unique identifier of the project.
• start_time: The start of the time range for the query.
• end_time: The end of the time range for the query.
• experiment_id: Filter records by a specific experiment ID.
• agent_stream_id: Filter records by a specific agent stream ID.
• filters: A list of filters to apply to the query.
• group_by: The field to group the results by.
• interval: The time interval for the query in seconds.

• LogRecordsMetricsResponse: A LogRecordsMetricsResponse object containing the query results, or None if the query fails.