Run a small evaluation against a project agent and inspect the result before adding more cases.
Before you start
You need curl, jq, uuidgen, an API credential with permission to manage evaluation definitions and create and read runs, and a project agent you can run. The example uses triage-agent in support-assistant.
This example checks whether an answer contains billing. Choose cases and metrics that reflect your application's requirements.
1. Create the evaluation
Call POST /projects/{project_reference}/evals to create a source definition:
set -e
curl --fail-with-body https://api.veryfront.com/projects/support-assistant/evals \
-H "Authorization: Bearer $VERYFRONT_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"name": "Triage quality",
"target": "agent:triage-agent",
"dataset": {
"kind": "inline",
"editable": true,
"dynamic": false,
"examples": [
{
"id": "duplicate-charge",
"input": "I was charged twice. Classify this request as billing.",
"reference": "billing"
}
]
},
"metrics": [
{
"name": "answer.contains",
"family": "answer",
"severity": "gate",
"config": {
"text": "billing"
},
"editable": true,
"dynamic": false
}
],
"repetitions": 1
}' > evaluation.json
EVAL_ID=$(jq -er '.id' evaluation.json) || exit 1
export EVAL_IDKeep the returned id as EVAL_ID. Pass this source definition ID unchanged in the run configuration.
2. Make the definition available to the runtime
Deploy a release containing the new evaluation and its target agent. Set ENVIRONMENT_ID to the available environment where that release is deployed.
Set PROJECT_ID to the project's UUID. The selected environment must belong to that project and contain the evaluation and target agent.
An environment runtime selects that environment's deployed evaluation source. Managed preview environments resolve to the main-branch source and runtime. See the run creation reference for other runtime selections.
3. Start the evaluation task
Call POST /runs with the task:eval target:
set -e
export ENVIRONMENT_ID="<ENVIRONMENT_ID>"
export PROJECT_ID="<PROJECT_ID>"
EVAL_RUN_IDEMPOTENCY_KEY=$(uuidgen) || exit 1
export EVAL_RUN_IDEMPOTENCY_KEY
jq -n --arg project "$PROJECT_ID" --arg evaluation "$EVAL_ID" \
--arg environment "$ENVIRONMENT_ID" '{
project_id: $project,
target: {type: "task", id: "eval"},
config: {eval_id: $evaluation},
execution: {runtime: {type: "environment", id: $environment}}
}' > evaluation-request.json
curl --fail-with-body https://api.veryfront.com/runs \
-H "Authorization: Bearer $VERYFRONT_API_KEY" \
-H "Content-Type: application/json" \
-H "Idempotency-Key: $EVAL_RUN_IDEMPOTENCY_KEY" \
--data-binary @evaluation-request.json > evaluation-run.json
EVAL_RUN_ID=$(jq -er '.id' evaluation-run.json) || exit 1
export EVAL_RUN_IDExpect HTTP 202 and a run object with a top-level id. A credential without read access receives only run_id and status. This walkthrough requires read access.
Reuse the same idempotency key and request body when retrying creation. Use a new key for a separate evaluation run. The task reads the selected evaluation source and runs it against the selected runtime.
4. Inspect status and retrieve the report
Read GET /runs/{run_id}:
set -e
while true; do
curl --fail-with-body "https://api.veryfront.com/runs/$EVAL_RUN_ID" \
-H "Authorization: Bearer $VERYFRONT_API_KEY" > evaluation-result.json || exit 1
status=$(jq -er '.status' evaluation-result.json) || exit 1
case "$status" in
completed) jq '{status, summary: .output}' evaluation-result.json; break ;;
failed|cancelled|waiting) jq '{status, error, control}' evaluation-result.json; exit 1 ;;
pending|running) sleep 2 ;;
*) printf 'Unexpected run status: %s\n' "$status" >&2; exit 1 ;;
esac
doneFor a failed execution, inspect the error and run events before treating the result as an agent-quality failure. A waiting execution needs its recorded input or resume condition. Retrieve the report only after execution completes.
When the response lists a report artifact, retrieve it with GET /runs/{run_id}/artifacts/{artifact_id}:
set -e
jq -e '.status == "completed"' evaluation-result.json > /dev/null
export REPORT_ARTIFACT_ID=$(jq -er '.output.report.artifact_id // (.artifacts[] | select(.type == "eval-report") | .artifact_id)' evaluation-result.json)
: "${REPORT_ARTIFACT_ID:?No evaluation report artifact was returned}"
curl --fail-with-body \
"https://api.veryfront.com/runs/$EVAL_RUN_ID/artifacts/$REPORT_ARTIFACT_ID" \
-H "Authorization: Bearer $VERYFRONT_API_KEY" > report.json
jq . report.jsonCheck the case results and metric outcomes. A completed execution does not imply every quality check passed. Keep the same cases and relevant settings when assessing a changed agent definition.
API references
- Evaluation definitions lists source selectors and evaluation source operations.
- Run creation and artifacts covers
task:evalexecution and report retrieval. - Evaluation GraphQL operations and MCP tools provide the supported equivalents.