Skip to main content
Agent Flow is a binary evaluator that measures the correctness and coherence of an agentic trajectory by validating it against user-specified natural language tests. A trajectory is said to pass the Agent Flow evaluator if and only if all the user-defined natural language conditions are successfully satisfied by the agent’s realized behavior or output.
To use this evaluator, you will need to create a copy and edit the prompt to provide your natural language tests.
This is a boolean evaluator that indicates whether the agent flow satisfies all conditions.

​
Agent Flow at a glance

​
When to use this evaluator

​
Configure Agent Flow

This evaluator needs to be manually customized to include your own natural language tests.
1

Create a copy of the Agent Flow evaluator

From the Evaluators Hub, select the Agent Flow evaluator. You will get a popup asking you to duplicate the evaluator. Select Duplicate evaluator to create a copy.
2

Locate the user defined tests section

Locate the user defined tests section in the prompt.
3

Customize the prompt by adding your user-defined tests

This prompt needs to be customized based on your application, and the inputs and outputs you are expecting. Replace {{ Add your tests here }} with a numbered list of tests in natural language that can be used to evaluate the agent efficiency. This can include:
  • Expected tool or agent calls, using the tool or agent names
  • Conditions on tool or agent calling (e.g. if tool x is called, don’t call agent y)
  • Expectations around the input or output parameters to tools and agents
  • Limitations on the number of tool or agent calls
For example, imagine you were creating an agent to provide advice on exercises for different body parts, such as for a physical therapy application. This has multiple tools, including list_by_target_muscle_for_exercised, list_by_body_part_for_exercised, list_of_bodyparts_for_exercised. Some user tests might be:
4

Save the evaluator

Save the evaluator, then turn it on for your Agent Stream.

​
Best practices

Trajectory tests are similar to unit tests for the agents trajectory, to check if certain conditions are followed during the agents path. You should write all the tests in a numbered list. For example:
Each test should check for one single condition only. Tests should be logically consistent, and well defined.

​
Performance Benchmarks

We evaluated Agent Flow against human expert labels on an internal dataset of agentic conversation samples using top frontier models.

​
GPT-4.1 Classification Report

Benchmarks based on internal evaluation dataset. Performance may vary by use case.
If you would like to dive deeper or start implementing Agent Flow, check out the following resources:

​
How-to guides