Deployments.ai Docs
Evaluations

Compare a candidate release against a baseline on a dataset

POST /v1/release-comparisons · SDK createReleaseComparison · CLI deployments eval comparison create

POST
/v1/release-comparisons

Requires observability:write.

Authorization

platformBearer
headerAuthorizationBearer <token>

Platform bearer credential accepted as an opaque API key or JWT.

Header Parameters

x-deployments-organization*string
Match^[0-7][0-9A-HJKMNP-TV-Z]{25}$
x-deployments-project*string
Match^[0-7][0-9A-HJKMNP-TV-Z]{25}$
x-deployments-environment*string
Match^[0-7][0-9A-HJKMNP-TV-Z]{25}$
idempotency-key*string
Length1 <= length <= 512

Request Body

application/json
  1. body
baselineReleaseId*string
Match^[0-9A-HJKMNP-TV-Za-hjkmnp-tv-z]{26}$
candidateReleaseId*string
Match^[0-9A-HJKMNP-TV-Za-hjkmnp-tv-z]{26}$
caseTimeoutSeconds?integer
Range30 <= value <= 3600
concurrency?integer
Range1 <= value <= 20
datasetId*string
Match^[0-9A-HJKMNP-TV-Za-hjkmnp-tv-z]{26}$
gatePolicy?
runtimeConfigFromAgentDeploymentId?string
Match^[0-9A-HJKMNP-TV-Za-hjkmnp-tv-z]{26}$
scoreDefinitionIds*array<>
Items1 <= items <= 20

Response Body

ReleaseComparisonResponse

application/json
  1. response
baseline*
candidate*
completedAt?string
Match^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}\.\d{3}Z$
Formatdate-time
createdAt*string
Match^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}\.\d{3}Z$
Formatdate-time
datasetId*string
Match^[0-9A-HJKMNP-TV-Za-hjkmnp-tv-z]{26}$
error?
gateMode*"advisory"
Value in"advisory"
gatePolicy*
id*string
Match^[0-9A-HJKMNP-TV-Za-hjkmnp-tv-z]{26}$
reasons*array<string>
report?
runtimeConfigFromAgentDeploymentId?string
Match^[0-9A-HJKMNP-TV-Za-hjkmnp-tv-z]{26}$
scoreDefinitions*array<>
status*string
Value in"completed""failed""queued""running"
summary?
updatedAt*string
Match^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}\.\d{3}Z$
Formatdate-time
verdict?string
Value in"error""fail""pass"
version*integer
Range0 <= value <= 9007199254740991
curl -X POST "https://example.com/v1/release-comparisons" \  -H "x-deployments-organization: string" \  -H "x-deployments-project: string" \  -H "x-deployments-environment: string" \  -H "idempotency-key: string" \  -H "Content-Type: application/json" \  -d '{    "baselineReleaseId": "string",    "candidateReleaseId": "string",    "datasetId": "string",    "scoreDefinitionIds": [      "string"    ]  }'
{  "baseline": {    "agentReleaseId": "string",    "evaluationRunId": "string",    "releaseContentHash": "string"  },  "candidate": {    "agentReleaseId": "string",    "evaluationRunId": "string",    "releaseContentHash": "string"  },  "completedAt": "2019-08-24T14:15:22Z",  "createdAt": "2019-08-24T14:15:22Z",  "datasetId": "string",  "error": {    "code": "string",    "message": "string"  },  "gateMode": "advisory",  "gatePolicy": {    "maxEvaluatorErrorRate": 0,    "maxMeanDecrease": 0,    "maxMissingEvidenceRate": 0,    "maxNewFailures": 0,    "maxPassRateDecrease": 0,    "minComparableCases": 1  },  "id": "string",  "reasons": [    "string"  ],  "report": {    "json": {      "contentType": "string",      "downloadUrl": "string",      "downloadUrlExpiresAt": "2019-08-24T14:15:22Z",      "sha256": "string",      "sizeBytes": 0    },    "markdown": {      "contentType": "string",      "downloadUrl": "string",      "downloadUrlExpiresAt": "2019-08-24T14:15:22Z",      "sha256": "string",      "sizeBytes": 0    }  },  "runtimeConfigFromAgentDeploymentId": "string",  "scoreDefinitions": [    {      "evaluatorVersionHash": "string",      "name": "string",      "revision": 1,      "scoreDefinitionId": "string",      "threshold": 0,      "type": "deterministic"    }  ],  "status": "completed",  "summary": {    "comparableCases": 0,    "costUsdMicros": {      "baselineTotal": 0,      "candidateTotal": 0,      "delta": 0    },    "definitions": [      {        "baselineMean": 0,        "baselinePassRate": 0,        "candidateMean": 0,        "candidatePassRate": 0,        "meanDelta": 0,        "name": "string",        "passRateDelta": 0,        "revision": 0,        "scoreDefinitionId": "string"      }    ],    "evaluatorErrors": {      "baseline": 0,      "candidate": 0    },    "latency": {      "baselineP50Ms": 0,      "baselineP95Ms": 0,      "candidateP50Ms": 0,      "candidateP95Ms": 0,      "p50DeltaMs": 0,      "p95DeltaMs": 0    },    "missingEvidence": {      "baseline": 0,      "candidate": 0    },    "regressions": 0,    "tokens": {      "baselineTotal": 0,      "candidateTotal": 0,      "delta": 0    },    "totalCases": 0  },  "updatedAt": "2019-08-24T14:15:22Z",  "verdict": "error",  "version": 0}