Create eval

createEval(projectReference: String!, input: CreateEvalInput!): EvalSource!

Requires project.evals.write.

Arguments

ArgumentTypeRequired
projectReferenceString!Yes
inputCreateEvalInput!Yes

Input fields

input

CreateEvalInput

FieldTypeRequiredDescription
nameString!YesThe eval id is derived from the name.
descriptionStringNo
targetString!YesThe identifier of the agent the eval runs against.
datasetEvalDatasetInput!Yes
metrics[EvalMetricInput!]NoDefault none.
repetitionsIntNoHow often each example runs; default 1.
tags[String!]No
metadataJSONObjectNo
sourceTargetKindWriteTargetKindNoThe source to write; default project.
targetBranchIdIDNoRequired with the kind preview_branch.

Response

Returns EvalSource!. Open the type for its fields and nested types.

FieldTypeDescription
idID!
nameString!
descriptionString
targetKindEvalTargetKind!
targetString!
sourcePathString!
exportNameString!
sourceString!The source text of the eval file.

All fields of EvalSource.

Types marked with ! cannot be null. Lists use [Type] notation. The type index lists all published input and response types.

Example

graphql
mutation CreateEval($projectReference: String!, $input: CreateEvalInput!) {
  createEval(projectReference: $projectReference, input: $input) {
    id
    name
    capabilities
    description
    dynamicFields
  }
}
json
{
  "projectReference": "00000000-0000-4000-8000-000000000001",
  "input": {
    "name": "example",
    "target": "example",
    "dataset": {
      "kind": "inline",
      "editable": false,
      "dynamic": false
    }
  }
}