Evaluations

Browse GraphQL domains.

Runs

Start, list, inspect, cancel, and resume agent, task, and workflow runs, follow their events, answer their input requests, and find the runs that schedules, webhooks, and evaluations start.

Triggers

Start runs from schedules, webhooks, and evaluations, and list the runs they started.

OperationDescription
listEvalRunsList evaluation runs.
projectEvalRunRead one run of an eval to follow its status or read its result summary; returns the eval run.
projectEvalRunReportRead the report of a finished eval run to inspect every example and metric result; returns the report document, or a conflict while the report is not written yet.
projectEvalRunsList the runs of one eval, newest first by default, to follow their status or pick a run to inspect; returns one page of eval runs.
projectEvalRunsConnectionList the runs of one eval, newest first by default, to follow their status or pick a run to inspect; returns one page of eval runs.
runProjectEvalStart a run of an eval against its agent, reading the eval from the project or a preview branch and running it on the main branch, an environment or a preview branch; returns the new eval run, which finishes in the background.

Queries

OperationDescription
evalRead one eval definition by its ID to inspect its target, dataset, metrics and the fields that can be edited; returns the eval.
evalsList the evals defined in the source of a project, a branch or an environment to find an eval to run or edit; returns every eval as one page with the source files that could not be read as evals.
evalSourceRead the editable source behind one eval before changing it; returns the eval source document with its source text, edit mode and the reasons it can only be edited as source.

Mutations

OperationDescription
createEvalCreate an eval source file in the project source or a preview branch to test an agent against a dataset and metrics; returns the eval source document.
updateEvalChange fields of a structured eval, such as its name, target, dataset or metrics, where an omitted field keeps its value; returns the updated eval source document.