Deployments.ai Docs
Evaluations

Start a dataset replay or trace scoring run

POST /v1/evaluation-runs · SDK createEvaluationRun · CLI deployments eval run create

POST
/v1/evaluation-runs

Requires observability:write.

Authorization

platformBearer
headerAuthorizationBearer <token>

Platform bearer credential accepted as an opaque API key or JWT.

Header Parameters

x-deployments-organization*string
Match^[0-7][0-9A-HJKMNP-TV-Z]{25}$
x-deployments-project*string
Match^[0-7][0-9A-HJKMNP-TV-Z]{25}$
x-deployments-environment*string
Match^[0-7][0-9A-HJKMNP-TV-Z]{25}$
idempotency-key*string
Length1 <= length <= 512

Request Body

application/json
  1. body
caseTimeoutSeconds?integer
Range30 <= value <= 3600
concurrency?integer
Range1 <= value <= 20
datasetId*string
Match^[0-9A-HJKMNP-TV-Za-hjkmnp-tv-z]{26}$
kind*"dataset-replay"
Value in"dataset-replay"
scoreDefinitionIds*array<>
Items1 <= items <= 20
target*|

Response Body

EvaluationRunResponse

application/json
  1. response
caseCount*integer
Range0 <= value <= 9007199254740991
caseTimeoutSeconds*integer
Range0 <= value <= 9007199254740991
comparison?
completedAt?string
Match^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}\.\d{3}Z$
Formatdate-time
concurrency*integer
Range0 <= value <= 9007199254740991
counts*
createdAt*string
Match^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}\.\d{3}Z$
Formatdate-time
datasetId?string
Match^[0-9A-HJKMNP-TV-Za-hjkmnp-tv-z]{26}$
error?
executionBindingId?string
id*string
Match^[0-9A-HJKMNP-TV-Za-hjkmnp-tv-z]{26}$
kind*string
Value in"dataset-replay""trace-score"
scoreDefinitions*array<>
startedAt?string
Match^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}\.\d{3}Z$
Formatdate-time
status*string
Value in"canceled""completed""failed""queued""running"
summary?
target?
traceIds?array<>
updatedAt*string
Match^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}\.\d{3}Z$
Formatdate-time
version*integer
Range0 <= value <= 9007199254740991
curl -X POST "https://example.com/v1/evaluation-runs" \  -H "x-deployments-organization: string" \  -H "x-deployments-project: string" \  -H "x-deployments-environment: string" \  -H "idempotency-key: string" \  -H "Content-Type: application/json" \  -d '{    "datasetId": "string",    "kind": "dataset-replay",    "scoreDefinitionIds": [      "string"    ],    "target": {      "agentReleaseId": "string",      "type": "release"    }  }'
{  "caseCount": 0,  "caseTimeoutSeconds": 0,  "comparison": {    "releaseComparisonId": "string",    "role": "baseline"  },  "completedAt": "2019-08-24T14:15:22Z",  "concurrency": 0,  "counts": {    "failed": 0,    "pending": 0,    "running": 0,    "scored": 0,    "timedOut": 0  },  "createdAt": "2019-08-24T14:15:22Z",  "datasetId": "string",  "error": {    "code": "string",    "message": "string"  },  "executionBindingId": "string",  "id": "string",  "kind": "dataset-replay",  "scoreDefinitions": [    {      "evaluatorVersionHash": "string",      "name": "string",      "revision": 1,      "scoreDefinitionId": "string",      "threshold": 0,      "type": "deterministic"    }  ],  "startedAt": "2019-08-24T14:15:22Z",  "status": "canceled",  "summary": {    "costUsdMicros": 0,    "definitions": [      {        "errors": 0,        "mean": 0,        "passRate": 0,        "passed": 0,        "revision": 0,        "scoreDefinitionId": "string",        "scored": 0      }    ],    "latencyP50Ms": 0,    "latencyP95Ms": 0,    "totalTokens": 0  },  "target": {    "agentDeploymentId": "string",    "agentReleaseId": "string",    "type": "deployment"  },  "traceIds": [    "string"  ],  "updatedAt": "2019-08-24T14:15:22Z",  "version": 0}