Evaluate an agent

Run a small evaluation against a project agent and inspect the result before adding more cases.

Before you start

You need curl, jq, uuidgen, an API credential with permission to manage evaluation definitions and create and read runs, and a project agent you can run. The example uses triage-agent in support-assistant.

This example checks whether an answer contains billing. Choose cases and metrics that reflect your application's requirements.

1. Create the evaluation

Call POST /projects/{project_reference}/evals to create a source definition:

bash
set -e

curl --fail-with-body https://api.veryfront.com/projects/support-assistant/evals \
  -H "Authorization: Bearer $VERYFRONT_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Triage quality",
    "target": "agent:triage-agent",
    "dataset": {
      "kind": "inline",
      "editable": true,
      "dynamic": false,
      "examples": [
        {
          "id": "duplicate-charge",
          "input": "I was charged twice. Classify this request as billing.",
          "reference": "billing"
        }
      ]
    },
    "metrics": [
      {
        "name": "answer.contains",
        "family": "answer",
        "severity": "gate",
        "config": {
          "text": "billing"
        },
        "editable": true,
        "dynamic": false
      }
    ],
    "repetitions": 1
  }' > evaluation.json

EVAL_ID=$(jq -er '.id' evaluation.json) || exit 1
export EVAL_ID

Keep the returned id as EVAL_ID. Pass this source definition ID unchanged in the run configuration.

2. Make the definition available to the runtime

Deploy a release containing the new evaluation and its target agent. Set ENVIRONMENT_ID to the available environment where that release is deployed.

Set PROJECT_ID to the project's UUID. The selected environment must belong to that project and contain the evaluation and target agent.

An environment runtime selects that environment's deployed evaluation source. Managed preview environments resolve to the main-branch source and runtime. See the run creation reference for other runtime selections.

3. Start the evaluation task

Call POST /runs with the task:eval target:

bash
set -e

export ENVIRONMENT_ID="<ENVIRONMENT_ID>"
export PROJECT_ID="<PROJECT_ID>"
EVAL_RUN_IDEMPOTENCY_KEY=$(uuidgen) || exit 1
export EVAL_RUN_IDEMPOTENCY_KEY

jq -n --arg project "$PROJECT_ID" --arg evaluation "$EVAL_ID" \
  --arg environment "$ENVIRONMENT_ID" '{
  project_id: $project,
  target: {type: "task", id: "eval"},
  config: {eval_id: $evaluation},
  execution: {runtime: {type: "environment", id: $environment}}
}' > evaluation-request.json

curl --fail-with-body https://api.veryfront.com/runs \
  -H "Authorization: Bearer $VERYFRONT_API_KEY" \
  -H "Content-Type: application/json" \
  -H "Idempotency-Key: $EVAL_RUN_IDEMPOTENCY_KEY" \
  --data-binary @evaluation-request.json > evaluation-run.json

EVAL_RUN_ID=$(jq -er '.id' evaluation-run.json) || exit 1
export EVAL_RUN_ID

Expect HTTP 202 and a run object with a top-level id. A credential without read access receives only run_id and status. This walkthrough requires read access.

Reuse the same idempotency key and request body when retrying creation. Use a new key for a separate evaluation run. The task reads the selected evaluation source and runs it against the selected runtime.

4. Inspect status and retrieve the report

Read GET /runs/{run_id}:

bash
set -e

while true; do
  curl --fail-with-body "https://api.veryfront.com/runs/$EVAL_RUN_ID" \
    -H "Authorization: Bearer $VERYFRONT_API_KEY" > evaluation-result.json || exit 1
  status=$(jq -er '.status' evaluation-result.json) || exit 1
  case "$status" in
    completed) jq '{status, summary: .output}' evaluation-result.json; break ;;
    failed|cancelled|waiting) jq '{status, error, control}' evaluation-result.json; exit 1 ;;
    pending|running) sleep 2 ;;
    *) printf 'Unexpected run status: %s\n' "$status" >&2; exit 1 ;;
  esac
done

For a failed execution, inspect the error and run events before treating the result as an agent-quality failure. A waiting execution needs its recorded input or resume condition. Retrieve the report only after execution completes.

When the response lists a report artifact, retrieve it with GET /runs/{run_id}/artifacts/{artifact_id}:

bash
set -e

jq -e '.status == "completed"' evaluation-result.json > /dev/null

export REPORT_ARTIFACT_ID=$(jq -er '.output.report.artifact_id // (.artifacts[] | select(.type == "eval-report") | .artifact_id)' evaluation-result.json)
: "${REPORT_ARTIFACT_ID:?No evaluation report artifact was returned}"

curl --fail-with-body \
  "https://api.veryfront.com/runs/$EVAL_RUN_ID/artifacts/$REPORT_ARTIFACT_ID" \
  -H "Authorization: Bearer $VERYFRONT_API_KEY" > report.json

jq . report.json

Check the case results and metric outcomes. A completed execution does not imply every quality check passed. Keep the same cases and relevant settings when assessing a changed agent definition.

API references