Deployments.ai Docs
Command reference

deployments eval

Evaluate agents with scores, regression datasets, runs, and release comparisons

CommandSummary
deployments eval score-definition createCreate a score definition
deployments eval score-definition listList score definitions
deployments eval score-definition showShow a score definition
deployments eval score-definition revisePublish a new score definition revision
deployments eval score-definition archiveArchive a score definition
deployments eval score listList recent score results
deployments eval score traceList score results for a trace
deployments eval score recordRecord a human or imported score for a trace
deployments eval annotation listList annotations for a trace
deployments eval annotation createAnnotate a trace
deployments eval dataset createCreate a regression dataset
deployments eval dataset listList regression datasets
deployments eval dataset showShow a regression dataset
deployments eval dataset archiveArchive a regression dataset
deployments eval case addAdd one case from inline input or an existing run or trace
deployments eval case importImport cases from a JSON case file
deployments eval case listList cases in a regression dataset
deployments eval run createStart a dataset replay or trace scoring run
deployments eval run listList evaluation runs, newest first
deployments eval run showShow an evaluation run and its summary
deployments eval run resultsList per-case results of an evaluation run
deployments eval comparison createCompare a candidate release against a baseline on a dataset
deployments eval comparison listList release comparisons, newest first
deployments eval comparison showShow a release comparison and its verdict

deployments eval score-definition create

Create a score definition.

deployments eval score-definition create --name <name> --evaluator <json>
  • Calls POST /v1/score-definitions (SDK createScoreDefinition).
  • Needs an organization, project, and environment: --org, --project, --env, or the saved context.
  • Sends an idempotency key. The CLI generates one per run; pass --idempotency-key to retry safely.
FlagTypeRequiredDefaultDescription
--namestringyesScore definition name
--descriptionstringScore definition description
--evaluatorstring (json)yesEvaluator configuration, for example {"kind":"contains"}
--samplingstring (json)Automatic sampling policy, for example {"rate":0.1}
--targetstringScored target: root-run or trace
--thresholdstring (number)Minimum passing value between 0 and 1
--weightstring (number)Relative weight between 0 and 100

Also accepts the global flags.

deployments eval score-definition list

List score definitions.

deployments eval score-definition list
  • Calls GET /v1/score-definitions (SDK listScoreDefinitions).
  • Needs an organization, project, and environment: --org, --project, --env, or the saved context.
  • Paginated. --page-size sets the page size; pass the returned cursor as --cursor for the next page.
FlagTypeRequiredDefaultDescription
--cursorstring (cursor)Pagination cursor
--page-sizestring (number)Page size
--statusstringOnly active or archived resources

Also accepts the global flags.

deployments eval score-definition show

Show a score definition.

deployments eval score-definition show --id <id>
FlagTypeRequiredDefaultDescription
--idstringyesScore definition ID

Also accepts the global flags.

deployments eval score-definition revise

Publish a new score definition revision.

deployments eval score-definition revise --id <id> --expected-revision <number> --evaluator <json>
  • Calls POST /v1/score-definitions/{scoreDefinitionId}/revisions (SDK reviseScoreDefinition).
  • Needs an organization, project, and environment: --org, --project, --env, or the saved context.
  • Sends an idempotency key. The CLI generates one per run; pass --idempotency-key to retry safely.
FlagTypeRequiredDefaultDescription
--idstringyesScore definition ID
--expected-revisionstring (number)yesRevision the change is based on
--evaluatorstring (json)yesEvaluator configuration, for example {"kind":"contains"}
--samplingstring (json)Automatic sampling policy, for example {"rate":0.1}
--targetstringScored target: root-run or trace
--thresholdstring (number)Minimum passing value between 0 and 1
--weightstring (number)Relative weight between 0 and 100

Also accepts the global flags.

deployments eval score-definition archive

Archive a score definition.

deployments eval score-definition archive --id <id>
  • Calls POST /v1/score-definitions/{scoreDefinitionId}/archive (SDK archiveScoreDefinition).
  • Needs an organization, project, and environment: --org, --project, --env, or the saved context.
  • Destructive. Live runs ask for confirmation; pass --yes in scripts or with --json.
  • Sends an idempotency key. The CLI generates one per run; pass --idempotency-key to retry safely.
FlagTypeRequiredDefaultDescription
--idstringyesScore definition ID

Also accepts the global flags.

deployments eval score list

List recent score results.

deployments eval score list
  • Calls GET /v1/score-results (SDK listScoreResults).
  • Needs an organization, project, and environment: --org, --project, --env, or the saved context.
  • Paginated. --page-size sets the page size; pass the returned cursor as --cursor for the next page.
FlagTypeRequiredDefaultDescription
--cursorstring (cursor)Pagination cursor
--page-sizestring (number)Page size
--score-definitionstringOnly results for this score definition

Also accepts the global flags.

deployments eval score trace

List score results for a trace.

deployments eval score trace --trace <trace>
  • Calls GET /v1/traces/{traceId}/score-results (SDK listTraceScoreResults).
  • Needs an organization, project, and environment: --org, --project, --env, or the saved context.
  • Paginated. --page-size sets the page size; pass the returned cursor as --cursor for the next page.
FlagTypeRequiredDefaultDescription
--cursorstring (cursor)Pagination cursor
--page-sizestring (number)Page size
--tracestringyesTrace ID (32 lowercase hex characters)

Also accepts the global flags.

deployments eval score record

Record a human or imported score for a trace.

deployments eval score record --trace <trace> --score-definition <score-definition> --value <number>
  • Calls POST /v1/traces/{traceId}/score-results (SDK recordTraceScoreResult).
  • Needs an organization, project, and environment: --org, --project, --env, or the saved context.
  • Sends an idempotency key. The CLI generates one per run; pass --idempotency-key to retry safely.
FlagTypeRequiredDefaultDescription
--tracestringyesTrace ID (32 lowercase hex characters)
--score-definitionstringyesScore definition ID
--valuestring (number)yesScore value between 0 and 1
--labelstringScore label
--rationalestringWhy this score was given
--runstringRun ID within the trace
--spanstringSpan ID within the trace

Also accepts the global flags.

deployments eval annotation list

List annotations for a trace.

deployments eval annotation list --trace <trace>
  • Calls GET /v1/traces/{traceId}/annotations (SDK listTraceAnnotations).
  • Needs an organization, project, and environment: --org, --project, --env, or the saved context.
  • Paginated. --page-size sets the page size; pass the returned cursor as --cursor for the next page.
FlagTypeRequiredDefaultDescription
--cursorstring (cursor)Pagination cursor
--page-sizestring (number)Page size
--tracestringyesTrace ID (32 lowercase hex characters)

Also accepts the global flags.

deployments eval annotation create

Annotate a trace.

deployments eval annotation create --trace <trace> --body <body>
  • Calls POST /v1/traces/{traceId}/annotations (SDK createTraceAnnotation).
  • Needs an organization, project, and environment: --org, --project, --env, or the saved context.
  • Sends an idempotency key. The CLI generates one per run; pass --idempotency-key to retry safely.
FlagTypeRequiredDefaultDescription
--tracestringyesTrace ID (32 lowercase hex characters)
--bodystringyesAnnotation text
--labelstringAnnotation label
--tagsstring (a,b,c)Annotation tags
--runstringRun ID within the trace
--spanstringSpan ID within the trace

Also accepts the global flags.

deployments eval dataset create

Create a regression dataset.

deployments eval dataset create --name <name>
  • Calls POST /v1/regression-datasets (SDK createRegressionDataset).
  • Needs an organization, project, and environment: --org, --project, --env, or the saved context.
  • Sends an idempotency key. The CLI generates one per run; pass --idempotency-key to retry safely.
FlagTypeRequiredDefaultDescription
--namestringyesRegression dataset name
--descriptionstringRegression dataset description
--retention-daysstring (number)Days to retain captured cases (1-3650)

Also accepts the global flags.

deployments eval dataset list

List regression datasets.

deployments eval dataset list
  • Calls GET /v1/regression-datasets (SDK listRegressionDatasets).
  • Needs an organization, project, and environment: --org, --project, --env, or the saved context.
  • Paginated. --page-size sets the page size; pass the returned cursor as --cursor for the next page.
FlagTypeRequiredDefaultDescription
--cursorstring (cursor)Pagination cursor
--page-sizestring (number)Page size
--statusstringOnly active or archived resources

Also accepts the global flags.

deployments eval dataset show

Show a regression dataset.

deployments eval dataset show --id <id>
FlagTypeRequiredDefaultDescription
--idstringyesRegression dataset ID

Also accepts the global flags.

deployments eval dataset archive

Archive a regression dataset.

deployments eval dataset archive --id <id>
  • Calls POST /v1/regression-datasets/{datasetId}/archive (SDK archiveRegressionDataset).
  • Needs an organization, project, and environment: --org, --project, --env, or the saved context.
  • Destructive. Live runs ask for confirmation; pass --yes in scripts or with --json.
  • Sends an idempotency key. The CLI generates one per run; pass --idempotency-key to retry safely.
FlagTypeRequiredDefaultDescription
--idstringyesRegression dataset ID

Also accepts the global flags.

deployments eval case add

Add one case from inline input or an existing run or trace.

deployments eval case add --dataset <dataset>
  • Calls POST /v1/regression-datasets/{datasetId}/cases (SDK addDatasetCase).
  • Needs an organization, project, and environment: --org, --project, --env, or the saved context.
  • Sends an idempotency key. The CLI generates one per run; pass --idempotency-key to retry safely.
FlagTypeRequiredDefaultDescription
--datasetstringyesRegression dataset ID
--inputstring (json)Inline input, for example {"content":"Where is my refund?"}
--sourcestring (json)Captured source, for example {"type":"trace","traceId":"..."}
--expectationsstring (json)Expected behavior, for example {"mustInclude":["refund"]}
--external-idstringStable external case ID
--tagsstring (a,b,c)Case tags
--consent-basisstringConsent basis for captured customer content

Also accepts the global flags.

deployments eval case import

Import cases from a JSON case file.

deployments eval case import --dataset <dataset> --file <path>
  • Calls POST /v1/regression-datasets/{datasetId}/cases/import (SDK importDatasetCases).
  • Needs an organization, project, and environment: --org, --project, --env, or the saved context.
  • Sends an idempotency key. The CLI generates one per run; pass --idempotency-key to retry safely.
FlagTypeRequiredDefaultDescription
--datasetstringyesRegression dataset ID
--filestring (path)yesJSON array of { id?, message, expected?: { mustInclude?, mustNot?, mustUseTools?, mustNotUseTools?, matches?, jsonSchema? }, tags? }

Also accepts the global flags.

deployments eval case list

List cases in a regression dataset.

deployments eval case list --dataset <dataset>
  • Calls GET /v1/regression-datasets/{datasetId}/cases (SDK listDatasetCases).
  • Needs an organization, project, and environment: --org, --project, --env, or the saved context.
  • Paginated. --page-size sets the page size; pass the returned cursor as --cursor for the next page.
FlagTypeRequiredDefaultDescription
--cursorstring (cursor)Pagination cursor
--page-sizestring (number)Page size
--datasetstringyesRegression dataset ID

Also accepts the global flags.

deployments eval run create

Start a dataset replay or trace scoring run.

deployments eval run create --kind <kind> --score-definitions <a-b-c>
  • Calls POST /v1/evaluation-runs (SDK createEvaluationRun).
  • Needs an organization, project, and environment: --org, --project, --env, or the saved context.
  • Sends an idempotency key. The CLI generates one per run; pass --idempotency-key to retry safely.
FlagTypeRequiredDefaultDescription
--kindstringyesRun kind: dataset-replay or trace-score
--score-definitionsstring (a,b,c)yesScore definition IDs to evaluate
--datasetstringRegression dataset ID (dataset-replay)
--targetstring (json)Replay target (dataset-replay), for example {"type":"release","agentReleaseId":"..."}
--trace-idsstring (a,b,c)Trace IDs to score (trace-score)
--concurrencystring (number)Cases replayed in parallel (1-20)
--case-timeoutstring (number)Per-case timeout in seconds (30-3600)

Also accepts the global flags.

deployments eval run list

List evaluation runs, newest first.

deployments eval run list
  • Calls GET /v1/evaluation-runs (SDK listEvaluationRuns).
  • Needs an organization, project, and environment: --org, --project, --env, or the saved context.
  • Paginated. --page-size sets the page size; pass the returned cursor as --cursor for the next page.
FlagTypeRequiredDefaultDescription
--cursorstring (cursor)Pagination cursor
--page-sizestring (number)Page size
--statusstringOnly runs in this status: queued, running, completed, failed, or canceled
--datasetstringOnly replays of this regression dataset
--releasestringOnly replays whose target is this agent release

Also accepts the global flags.

deployments eval run show

Show an evaluation run and its summary.

deployments eval run show --id <id>
FlagTypeRequiredDefaultDescription
--idstringyesEvaluation run ID

Also accepts the global flags.

deployments eval run results

List per-case results of an evaluation run.

deployments eval run results --id <id>
  • Calls GET /v1/evaluation-runs/{evaluationRunId}/results (SDK listEvaluationRunResults).
  • Needs an organization, project, and environment: --org, --project, --env, or the saved context.
  • Paginated. --page-size sets the page size; pass the returned cursor as --cursor for the next page.
FlagTypeRequiredDefaultDescription
--cursorstring (cursor)Pagination cursor
--page-sizestring (number)Page size
--idstringyesEvaluation run ID

Also accepts the global flags.

deployments eval comparison create

Compare a candidate release against a baseline on a dataset.

deployments eval comparison create --baseline <baseline> --candidate <candidate> --dataset <dataset> --score-definitions <a-b-c>
  • Calls POST /v1/release-comparisons (SDK createReleaseComparison).
  • Needs an organization, project, and environment: --org, --project, --env, or the saved context.
  • Sends an idempotency key. The CLI generates one per run; pass --idempotency-key to retry safely.
FlagTypeRequiredDefaultDescription
--baselinestringyesBaseline agent release ID
--candidatestringyesCandidate agent release ID
--datasetstringyesRegression dataset ID
--score-definitionsstring (a,b,c)yesScore definition IDs to evaluate
--gate-policystring (json)Gate thresholds, for example {"maxNewFailures":0}
--runtime-config-fromstringAgent deployment whose runtime configuration both releases use
--concurrencystring (number)Cases replayed in parallel (1-20)
--case-timeoutstring (number)Per-case timeout in seconds (30-3600)

Also accepts the global flags.

deployments eval comparison list

List release comparisons, newest first.

deployments eval comparison list
  • Calls GET /v1/release-comparisons (SDK listReleaseComparisons).
  • Needs an organization, project, and environment: --org, --project, --env, or the saved context.
  • Paginated. --page-size sets the page size; pass the returned cursor as --cursor for the next page.
FlagTypeRequiredDefaultDescription
--cursorstring (cursor)Pagination cursor
--page-sizestring (number)Page size
--statusstringOnly comparisons in this status: queued, running, completed, or failed
--baselinestringOnly comparisons with this baseline release
--candidatestringOnly comparisons with this candidate release

Also accepts the global flags.

deployments eval comparison show

Show a release comparison and its verdict.

deployments eval comparison show --id <id>
FlagTypeRequiredDefaultDescription
--idstringyesRelease comparison ID

Also accepts the global flags.

On this page