Evaluations
Start a dataset replay or trace scoring run
POST /v1/evaluation-runs · SDK createEvaluationRun · CLI deployments eval run create
POST
Requires observability:write.
platformBearerheader
AuthorizationBearer <token>Platform bearer credential accepted as an opaque API key or JWT.
x-deployments-organization*stringMatch
^[0-7][0-9A-HJKMNP-TV-Z]{25}$x-deployments-project*stringMatch
^[0-7][0-9A-HJKMNP-TV-Z]{25}$x-deployments-environment*stringMatch
^[0-7][0-9A-HJKMNP-TV-Z]{25}$idempotency-key*stringLength
1 <= length <= 512application/json- body
caseTimeoutSeconds?integerRange
30 <= value <= 3600concurrency?integerRange
1 <= value <= 20datasetId*stringMatch
^[0-9A-HJKMNP-TV-Za-hjkmnp-tv-z]{26}$kind*"dataset-replay"Value in
"dataset-replay"scoreDefinitionIds*array<>Items
1 <= items <= 20target*|EvaluationRunResponse
application/json- response
caseCount*integerRange
0 <= value <= 9007199254740991caseTimeoutSeconds*integerRange
0 <= value <= 9007199254740991comparison?completedAt?stringMatch
^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}\.\d{3}Z$Format
date-timeconcurrency*integerRange
0 <= value <= 9007199254740991counts*createdAt*stringMatch
^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}\.\d{3}Z$Format
date-timedatasetId?stringMatch
^[0-9A-HJKMNP-TV-Za-hjkmnp-tv-z]{26}$error?executionBindingId?stringid*stringMatch
^[0-9A-HJKMNP-TV-Za-hjkmnp-tv-z]{26}$kind*stringValue in
"dataset-replay""trace-score"scoreDefinitions*array<>startedAt?stringMatch
^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}\.\d{3}Z$Format
date-timestatus*stringValue in
"canceled""completed""failed""queued""running"summary?target?traceIds?array<>updatedAt*stringMatch
^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}\.\d{3}Z$Format
date-timeversion*integerRange
0 <= value <= 9007199254740991curl -X POST "https://example.com/v1/evaluation-runs" \ -H "x-deployments-organization: string" \ -H "x-deployments-project: string" \ -H "x-deployments-environment: string" \ -H "idempotency-key: string" \ -H "Content-Type: application/json" \ -d '{ "datasetId": "string", "kind": "dataset-replay", "scoreDefinitionIds": [ "string" ], "target": { "agentReleaseId": "string", "type": "release" } }'{ "caseCount": 0, "caseTimeoutSeconds": 0, "comparison": { "releaseComparisonId": "string", "role": "baseline" }, "completedAt": "2019-08-24T14:15:22Z", "concurrency": 0, "counts": { "failed": 0, "pending": 0, "running": 0, "scored": 0, "timedOut": 0 }, "createdAt": "2019-08-24T14:15:22Z", "datasetId": "string", "error": { "code": "string", "message": "string" }, "executionBindingId": "string", "id": "string", "kind": "dataset-replay", "scoreDefinitions": [ { "evaluatorVersionHash": "string", "name": "string", "revision": 1, "scoreDefinitionId": "string", "threshold": 0, "type": "deterministic" } ], "startedAt": "2019-08-24T14:15:22Z", "status": "canceled", "summary": { "costUsdMicros": 0, "definitions": [ { "errors": 0, "mean": 0, "passRate": 0, "passed": 0, "revision": 0, "scoreDefinitionId": "string", "scored": 0 } ], "latencyP50Ms": 0, "latencyP95Ms": 0, "totalTokens": 0 }, "target": { "agentDeploymentId": "string", "agentReleaseId": "string", "type": "deployment" }, "traceIds": [ "string" ], "updatedAt": "2019-08-24T14:15:22Z", "version": 0}