Evaluate an agent

Create a small evaluation for a project agent, execute it, and inspect the measured result before expanding the dataset.

Before you start

You need curl, jq, uuidgen, an API credential with permission to manage evaluation definitions and create and read runs, and a project agent you can run. The example uses triage-agent in support-assistant. The example checks whether an answer contains billing. That demonstrates one metric; choose cases and metrics that reflect your application’s requirements.

1. Create the evaluation

Call POST /projects/{project_reference}/evals to create a source definition:

create-evaluation.sh

set -e

curl --fail-with-body https://api.veryfront.com/projects/support-assistant/evals \
  -H "Authorization: Bearer $VERYFRONT_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "kind": "eval-source-create",
    "fields": {
      "name": "Triage quality",
      "target": "agent:triage-agent",
      "dataset": {
        "kind": "inline",
        "editable": true,
        "dynamic": false,
        "examples": [{
          "id": "duplicate-charge",
          "input": "I was charged twice. Classify this request as billing.",
          "reference": "billing"
        }]
      },
      "metrics": [{
        "name": "answer.contains",
        "family": "answer",
        "severity": "gate",
        "config": {"text":"billing"},
        "editable": true,
        "dynamic": false
      }],
      "repetitions": 1
    }
  }' > evaluation.json

EVAL_ID=$(jq -er '.id' evaluation.json) || exit 1
export EVAL_ID

Keep the returned id as EVAL_ID. Pass this source definition ID unchanged in the run configuration.

2. Make the definition available to the runtime

Deploy a release containing the new evaluation and its target agent. Set ENVIRONMENT_ID to the available environment where that release is deployed. Set PROJECT_ID to the project’s UUID. The selected environment must belong to that project and contain the evaluation and target agent. An environment runtime selects that environment’s deployed evaluation source. Managed preview environments resolve to the main-branch source and runtime. See the run creation reference for other runtime selections.

3. Start the evaluation task

Call POST /runs with the task:eval target:

run-evaluation.sh

set -e

export ENVIRONMENT_ID="<ENVIRONMENT_ID>"
export PROJECT_ID="<PROJECT_ID>"
EVAL_RUN_IDEMPOTENCY_KEY=$(uuidgen) || exit 1
export EVAL_RUN_IDEMPOTENCY_KEY

jq -n --arg project "$PROJECT_ID" --arg evaluation "$EVAL_ID" \
  --arg environment "$ENVIRONMENT_ID" '{
  project_id: $project,
  target: {type: "task", id: "eval"},
  config: {eval_id: $evaluation},
  execution: {runtime: {type: "environment", id: $environment}}
}' > evaluation-request.json

curl --fail-with-body https://api.veryfront.com/runs \
  -H "Authorization: Bearer $VERYFRONT_API_KEY" \
  -H "Content-Type: application/json" \
  -H "Idempotency-Key: $EVAL_RUN_IDEMPOTENCY_KEY" \
  --data-binary @evaluation-request.json > evaluation-run.json

EVAL_RUN_ID=$(jq -er '.id' evaluation-run.json) || exit 1
export EVAL_RUN_ID

Expect HTTP 202 and a run object with a top-level id. A credential without read access receives only run_id and status; this walkthrough requires read access. Reuse the same idempotency key and request body when retrying creation. Use a new key for a separate evaluation run. The task reads the selected evaluation source and runs it against the selected runtime.

4. Inspect status and retrieve the report

Read GET /runs/{run_id}:

read-evaluation-run.sh

set -e

while true; do
  curl --fail-with-body "https://api.veryfront.com/runs/$EVAL_RUN_ID" \
    -H "Authorization: Bearer $VERYFRONT_API_KEY" > evaluation-result.json || exit 1
  status=$(jq -er '.status' evaluation-result.json) || exit 1
  case "$status" in
    completed) jq '{status, summary: .output}' evaluation-result.json; break ;;
    failed|cancelled|waiting) jq '{status, error, control}' evaluation-result.json; exit 1 ;;
    pending|running) sleep 2 ;;
    *) printf 'Unexpected run status: %s\n' "$status" >&2; exit 1 ;;
  esac
done

For a failed execution, inspect the error and run events before treating the result as an agent-quality failure. A waiting execution needs its recorded input or resume condition. Retrieve the report only after execution completes. When the response lists a report artifact, retrieve it with GET /runs/{run_id}/artifacts/{artifact_id}:

read-evaluation-report.sh

set -e

jq -e '.status == "completed"' evaluation-result.json > /dev/null

export REPORT_ARTIFACT_ID=$(jq -er '.output.report.artifact_id // (.artifacts[] | select(.type == "eval-report") | .artifact_id)' evaluation-result.json)
: "${REPORT_ARTIFACT_ID:?No evaluation report artifact was returned}"

curl --fail-with-body \
  "https://api.veryfront.com/runs/$EVAL_RUN_ID/artifacts/$REPORT_ARTIFACT_ID" \
  -H "Authorization: Bearer $VERYFRONT_API_KEY" > report.json

jq . report.json

Check the case results and metric outcomes. A completed execution does not imply every quality check passed. Keep the same cases and relevant settings when assessing a changed agent definition.

API references

Evaluation definitions lists source selectors and evaluation source operations.
Run creation and artifacts covers task:eval execution and report retrieval.
Evaluation GraphQL operations and MCP tools provide the supported equivalents.

Get started

APIs

REST API

GraphQL API

MCP API

Before you start

1. Create the evaluation

2. Make the definition available to the runtime

3. Start the evaluation task

4. Inspect status and retrieve the report

API references

​Before you start

​1. Create the evaluation

​2. Make the definition available to the runtime

​3. Start the evaluation task

​4. Inspect status and retrieve the report

​API references

Before you start

1. Create the evaluation

2. Make the definition available to the runtime

3. Start the evaluation task

4. Inspect status and retrieve the report

API references