Create eval

veryfront__create_eval

Requires project.evals.write.

Access

No external-service connection is required. Use a Veryfront credential with access to the requested resources.

Call type: Can change resources.

Arguments

FieldTypeRequiredDescription and constraints
project_referencestringNoProject identifier or slug that scopes the tool call. Minimum length: 1.
namestringYesHuman-readable eval name; the eval id is derived from it. Minimum length: 1.
descriptionstringNoOptional short description of what the eval checks.
targetstringYesIdentifier of the agent the eval runs against. Minimum length: 1.
datasetobjectYesThe examples the eval runs: inline examples or a dataset file.
metricsobject[]NoThe metrics that score each example. Defaults to none. Default: [].
repetitionsintegerNoHow often each example runs. Defaults to 1. Default: 1.
tagsstring[]NoLabels for the eval. Defaults to none. Default: [].
metadataobjectNoFree-form JSON object stored with the eval. Defaults to empty. Default: {}.
source_target_kindstringNoSource to use: project or preview_branch. Defaults to project.
target_branch_idstring or nullNoPreview branch to use when the target kind is preview_branch.

Example request

Send this JSON-RPC request through an initialized MCP client. Replace sample identifiers and values before calling the tool. See MCP client setup.

json
{
  "jsonrpc": "2.0",
  "id": 1,
  "method": "tools/call",
  "params": {
    "name": "veryfront__create_eval",
    "arguments": {
      "name": "example",
      "target": "example",
      "dataset": {
        "kind": "inline",
        "editable": false,
        "dynamic": false
      }
    }
  }
}

Result

The published output shape is summarized below. Inspect both the MCP response content and isError when handling a call.

FieldTypeRequiredDescription and constraints
idstringYesEval identifier.
namestringYesHuman-readable eval name.
descriptionstring or nullYesShort description of what the eval checks, or null.
target_kindstringYesWhat the eval runs against: always agent. Value: "agent".
targetstringYesIdentifier of the agent the eval runs against.
source_pathstringYesSource file path where the eval definition lives.
export_namestringYesName of the export that declares the eval in its source file.
datasetobjectYesThe examples the eval runs.
metricsobject[]YesThe metrics that score each example.
repetitionsintegerYesHow often each example runs.
tagsstring[]YesLabels of the eval.
metadataobjectYesFree-form JSON object stored with the eval.
editable_fieldsstring[]YesFields that update_eval can change.
dynamic_fieldsstring[]YesFields computed in source, which update_eval cannot change.
capabilitiesstring[]YesEval capabilities the caller holds on the project.
sourcestringYesSource text of the eval file.
edit_modestringYesEditing mode of the eval: structured for field updates, source_only for source edits. Values: "structured", "source_only".
source_only_reasonsstring[]YesReasons why the eval can only be edited as source; empty for a structured eval.

Full schemas

json
{
  "type": "object",
  "properties": {
    "project_reference": {
      "type": "string",
      "minLength": 1,
      "description": "Project identifier or slug that scopes the tool call."
    },
    "name": {
      "type": "string",
      "minLength": 1,
      "description": "Human-readable eval name; the eval id is derived from it."
    },
    "description": {
      "type": "string",
      "description": "Optional short description of what the eval checks."
    },
    "target": {
      "type": "string",
      "minLength": 1,
      "description": "Identifier of the agent the eval runs against."
    },
    "dataset": {
      "type": "object",
      "properties": {
        "kind": {
          "type": "string",
          "enum": [
            "inline",
            "json",
            "jsonl",
            "dynamic"
          ],
          "description": "Provide the kind."
        },
        "path": {
          "type": "string",
          "description": "Provide the path."
        },
        "examples": {
          "type": "array",
          "items": {
            "type": "object",
            "properties": {
              "id": {
                "type": "string",
                "minLength": 1,
                "description": "Unique identifier for the target record."
              },
              "input": {
                "description": "Provide the input."
              },
              "reference": {
                "description": "Provide the reference."
              },
              "metadata": {
                "type": "object",
                "additionalProperties": {
                  "description": "Additional structured values supplied to the tool."
                },
                "description": "Additional structured metadata for the request."
              }
            },
            "required": [
              "id"
            ],
            "additionalProperties": false,
            "description": "One example item supplied to the tool."
          },
          "description": "Provide one or more example values."
        },
        "editable": {
          "type": "boolean",
          "description": "Provide the editable."
        },
        "dynamic": {
          "type": "boolean",
          "description": "Provide the dynamic."
        }
      },
      "required": [
        "kind",
        "editable",
        "dynamic"
      ],
      "additionalProperties": false,
      "description": "The examples the eval runs: inline examples or a dataset file."
    },
    "metrics": {
      "type": "array",
      "items": {
        "type": "object",
        "properties": {
          "name": {
            "type": "string",
            "minLength": 1,
            "description": "Human-readable name for the target record."
          },
          "family": {
            "type": "string",
            "enum": [
              "answer",
              "agent",
              "ops",
              "judge",
              "knowledge",
              "check"
            ],
            "description": "Provide the family."
          },
          "severity": {
            "type": "string",
            "enum": [
              "gate",
              "soft",
              "budget"
            ],
            "description": "Provide the severity."
          },
          "threshold": {
            "type": "object",
            "properties": {
              "min": {
                "type": "number",
                "description": "Provide the min."
              },
              "max": {
                "type": "number",
                "description": "Provide the max."
              }
            },
            "additionalProperties": false,
            "description": "Structured threshold details used by the tool."
          },
          "config": {
            "type": "object",
            "additionalProperties": {
              "description": "Additional structured values supplied to the tool."
            },
            "description": "Structured config details used by the tool."
          },
          "editable": {
            "type": "boolean",
            "description": "Provide the editable."
          },
          "dynamic": {
            "type": "boolean",
            "description": "Provide the dynamic."
          }
        },
        "required": [
          "name",
          "family",
          "severity",
          "editable",
          "dynamic"
        ],
        "additionalProperties": false,
        "description": "One metric item supplied to the tool."
      },
      "default": [],
      "description": "The metrics that score each example. Defaults to none."
    },
    "repetitions": {
      "type": "integer",
      "exclusiveMinimum": 0,
      "default": 1,
      "description": "How often each example runs. Defaults to 1."
    },
    "tags": {
      "type": "array",
      "items": {
        "type": "string",
        "description": "One tag item supplied to the tool."
      },
      "default": [],
      "description": "Labels for the eval. Defaults to none."
    },
    "metadata": {
      "type": "object",
      "additionalProperties": {
        "description": "Additional structured values supplied to the tool."
      },
      "default": {},
      "description": "Free-form JSON object stored with the eval. Defaults to empty."
    },
    "source_target_kind": {
      "type": "string",
      "description": "Source to use: project or preview_branch. Defaults to project."
    },
    "target_branch_id": {
      "anyOf": [
        {
          "type": "string",
          "format": "uuid",
          "description": "Provide the target branch id."
        },
        {
          "type": "null",
          "description": "Provide the target branch id."
        }
      ],
      "description": "Preview branch to use when the target kind is preview_branch."
    }
  },
  "required": [
    "name",
    "target",
    "dataset"
  ],
  "additionalProperties": false,
  "$schema": "http://json-schema.org/draft-07/schema#",
  "description": "Arguments for the create_eval tool."
}
json
{
  "type": "object",
  "properties": {
    "id": {
      "type": "string",
      "description": "Eval identifier."
    },
    "name": {
      "type": "string",
      "description": "Human-readable eval name."
    },
    "description": {
      "type": [
        "string",
        "null"
      ],
      "description": "Short description of what the eval checks, or null."
    },
    "target_kind": {
      "type": "string",
      "const": "agent",
      "description": "What the eval runs against: always agent."
    },
    "target": {
      "type": "string",
      "description": "Identifier of the agent the eval runs against."
    },
    "source_path": {
      "type": "string",
      "description": "Source file path where the eval definition lives."
    },
    "export_name": {
      "type": "string",
      "description": "Name of the export that declares the eval in its source file."
    },
    "dataset": {
      "type": "object",
      "properties": {
        "kind": {
          "type": "string",
          "enum": [
            "inline",
            "json",
            "jsonl",
            "dynamic"
          ],
          "description": "The kind returned by the tool."
        },
        "path": {
          "type": "string",
          "description": "The path returned by the tool."
        },
        "examples": {
          "type": "array",
          "items": {
            "type": "object",
            "properties": {
              "id": {
                "type": "string",
                "minLength": 1,
                "description": "Unique identifier for the returned record."
              },
              "input": {
                "description": "The input returned by the tool."
              },
              "reference": {
                "description": "The reference returned by the tool."
              },
              "metadata": {
                "type": "object",
                "additionalProperties": {
                  "description": "Additional structured values returned by the tool."
                },
                "description": "Additional structured metadata returned by the tool."
              }
            },
            "required": [
              "id"
            ],
            "additionalProperties": false,
            "description": "One example item returned by the tool."
          },
          "description": "List of examples returned by the tool."
        },
        "editable": {
          "type": "boolean",
          "description": "The editable returned by the tool."
        },
        "dynamic": {
          "type": "boolean",
          "description": "The dynamic returned by the tool."
        }
      },
      "required": [
        "kind",
        "editable",
        "dynamic"
      ],
      "additionalProperties": false,
      "description": "The examples the eval runs."
    },
    "metrics": {
      "type": "array",
      "items": {
        "type": "object",
        "properties": {
          "name": {
            "type": "string",
            "minLength": 1,
            "description": "Human-readable name for the returned record."
          },
          "family": {
            "type": "string",
            "enum": [
              "answer",
              "agent",
              "ops",
              "judge",
              "knowledge",
              "check"
            ],
            "description": "The family returned by the tool."
          },
          "severity": {
            "type": "string",
            "enum": [
              "gate",
              "soft",
              "budget"
            ],
            "description": "The severity returned by the tool."
          },
          "threshold": {
            "type": "object",
            "properties": {
              "min": {
                "type": "number",
                "description": "The min returned by the tool."
              },
              "max": {
                "type": "number",
                "description": "The max returned by the tool."
              }
            },
            "additionalProperties": false,
            "description": "Structured threshold details returned by the tool."
          },
          "config": {
            "type": "object",
            "additionalProperties": {
              "description": "Additional structured values returned by the tool."
            },
            "description": "Structured config details returned by the tool."
          },
          "editable": {
            "type": "boolean",
            "description": "The editable returned by the tool."
          },
          "dynamic": {
            "type": "boolean",
            "description": "The dynamic returned by the tool."
          }
        },
        "required": [
          "name",
          "family",
          "severity",
          "editable",
          "dynamic"
        ],
        "additionalProperties": false,
        "description": "One metric item returned by the tool."
      },
      "description": "The metrics that score each example."
    },
    "repetitions": {
      "type": "integer",
      "exclusiveMinimum": 0,
      "description": "How often each example runs."
    },
    "tags": {
      "type": "array",
      "items": {
        "type": "string",
        "description": "One tag item returned by the tool."
      },
      "description": "Labels of the eval."
    },
    "metadata": {
      "type": "object",
      "additionalProperties": {
        "description": "Additional structured values returned by the tool."
      },
      "description": "Free-form JSON object stored with the eval."
    },
    "editable_fields": {
      "type": "array",
      "items": {
        "type": "string",
        "enum": [
          "name",
          "description",
          "target",
          "dataset",
          "repetitions",
          "tags",
          "metadata",
          "metrics",
          "check"
        ],
        "description": "One editable field item returned by the tool."
      },
      "description": "Fields that update_eval can change."
    },
    "dynamic_fields": {
      "type": "array",
      "items": {
        "$ref": "#/properties/editable_fields/items",
        "description": "One dynamic field item returned by the tool."
      },
      "description": "Fields computed in source, which update_eval cannot change."
    },
    "capabilities": {
      "type": "array",
      "items": {
        "type": "string",
        "description": "One capability item returned by the tool."
      },
      "description": "Eval capabilities the caller holds on the project."
    },
    "source": {
      "type": "string",
      "description": "Source text of the eval file."
    },
    "edit_mode": {
      "type": "string",
      "enum": [
        "structured",
        "source_only"
      ],
      "description": "Editing mode of the eval: structured for field updates, source_only for source edits."
    },
    "source_only_reasons": {
      "type": "array",
      "items": {
        "type": "string",
        "description": "One source only reason item returned by the tool."
      },
      "description": "Reasons why the eval can only be edited as source; empty for a structured eval."
    }
  },
  "required": [
    "id",
    "name",
    "description",
    "target_kind",
    "target",
    "source_path",
    "export_name",
    "dataset",
    "metrics",
    "repetitions",
    "tags",
    "metadata",
    "editable_fields",
    "dynamic_fields",
    "capabilities",
    "source",
    "edit_mode",
    "source_only_reasons"
  ],
  "additionalProperties": false,
  "description": "Eval definition including editable source content.",
  "$schema": "http://json-schema.org/draft-07/schema#"
}