The Evaluations API tests agents against defined examples and metrics. It keeps the evaluation definition separate from each execution and its results.
Evaluation definitions
An evaluation definition identifies a target agent, a dataset, metrics, and repetitions. The dataset supplies the cases to evaluate. Metrics specify what to measure for each case or across the run.
Definitions are stored in project source. The target agent is managed through the Agents API.
Runs and reports
Starting an evaluation creates a new run. The run records progress and results without replacing the definition or the history of earlier runs.
An evaluation run produces a report. Check the run status before retrieving it. The report is not available as soon as the run is created.
The Execution API provides shared run operations, including event inspection and cancellation. Evaluation-specific operations provide the definition, evaluation run details, and report.
Assessing a change
For a support agent, a dataset can represent expected questions and known failure cases. Metrics measure the results. Repetitions can expose variation across executions.
When comparing instructions or tool configurations, keep the cases and other relevant settings consistent. Completion means execution finished. Use the report and your acceptance criteria to judge the behavior.
Your release process determines whether the results meet its approval criteria. A successful evaluation run does not approve a deployment.
Get started
Evaluate an agent: create a definition, start a run, and inspect its report. The Code evaluation guide covers source-level authoring.