deployments eval
Evaluate agents with scores, regression datasets, runs, and release comparisons
| Command | Summary |
|---|---|
deployments eval score-definition create | Create a score definition |
deployments eval score-definition list | List score definitions |
deployments eval score-definition show | Show a score definition |
deployments eval score-definition revise | Publish a new score definition revision |
deployments eval score-definition archive | Archive a score definition |
deployments eval score list | List recent score results |
deployments eval score trace | List score results for a trace |
deployments eval score record | Record a human or imported score for a trace |
deployments eval annotation list | List annotations for a trace |
deployments eval annotation create | Annotate a trace |
deployments eval dataset create | Create a regression dataset |
deployments eval dataset list | List regression datasets |
deployments eval dataset show | Show a regression dataset |
deployments eval dataset archive | Archive a regression dataset |
deployments eval case add | Add one case from inline input or an existing run or trace |
deployments eval case import | Import cases from a JSON case file |
deployments eval case list | List cases in a regression dataset |
deployments eval run create | Start a dataset replay or trace scoring run |
deployments eval run list | List evaluation runs, newest first |
deployments eval run show | Show an evaluation run and its summary |
deployments eval run results | List per-case results of an evaluation run |
deployments eval comparison create | Compare a candidate release against a baseline on a dataset |
deployments eval comparison list | List release comparisons, newest first |
deployments eval comparison show | Show a release comparison and its verdict |
deployments eval score-definition create
Create a score definition.
deployments eval score-definition create --name <name> --evaluator <json>- Calls
POST /v1/score-definitions(SDKcreateScoreDefinition). - Needs an organization, project, and environment:
--org,--project,--env, or the saved context. - Sends an idempotency key. The CLI generates one per run; pass
--idempotency-keyto retry safely.
| Flag | Type | Required | Default | Description |
|---|---|---|---|---|
--name | string | yes | Score definition name | |
--description | string | Score definition description | ||
--evaluator | string (json) | yes | Evaluator configuration, for example {"kind":"contains"} | |
--sampling | string (json) | Automatic sampling policy, for example {"rate":0.1} | ||
--target | string | Scored target: root-run or trace | ||
--threshold | string (number) | Minimum passing value between 0 and 1 | ||
--weight | string (number) | Relative weight between 0 and 100 |
Also accepts the global flags.
deployments eval score-definition list
List score definitions.
deployments eval score-definition list- Calls
GET /v1/score-definitions(SDKlistScoreDefinitions). - Needs an organization, project, and environment:
--org,--project,--env, or the saved context. - Paginated.
--page-sizesets the page size; pass the returned cursor as--cursorfor the next page.
| Flag | Type | Required | Default | Description |
|---|---|---|---|---|
--cursor | string (cursor) | Pagination cursor | ||
--page-size | string (number) | Page size | ||
--status | string | Only active or archived resources |
Also accepts the global flags.
deployments eval score-definition show
Show a score definition.
deployments eval score-definition show --id <id>- Calls
GET /v1/score-definitions/{scoreDefinitionId}(SDKgetScoreDefinition). - Needs an organization, project, and environment:
--org,--project,--env, or the saved context.
| Flag | Type | Required | Default | Description |
|---|---|---|---|---|
--id | string | yes | Score definition ID |
Also accepts the global flags.
deployments eval score-definition revise
Publish a new score definition revision.
deployments eval score-definition revise --id <id> --expected-revision <number> --evaluator <json>- Calls
POST /v1/score-definitions/{scoreDefinitionId}/revisions(SDKreviseScoreDefinition). - Needs an organization, project, and environment:
--org,--project,--env, or the saved context. - Sends an idempotency key. The CLI generates one per run; pass
--idempotency-keyto retry safely.
| Flag | Type | Required | Default | Description |
|---|---|---|---|---|
--id | string | yes | Score definition ID | |
--expected-revision | string (number) | yes | Revision the change is based on | |
--evaluator | string (json) | yes | Evaluator configuration, for example {"kind":"contains"} | |
--sampling | string (json) | Automatic sampling policy, for example {"rate":0.1} | ||
--target | string | Scored target: root-run or trace | ||
--threshold | string (number) | Minimum passing value between 0 and 1 | ||
--weight | string (number) | Relative weight between 0 and 100 |
Also accepts the global flags.
deployments eval score-definition archive
Archive a score definition.
deployments eval score-definition archive --id <id>- Calls
POST /v1/score-definitions/{scoreDefinitionId}/archive(SDKarchiveScoreDefinition). - Needs an organization, project, and environment:
--org,--project,--env, or the saved context. - Destructive. Live runs ask for confirmation; pass
--yesin scripts or with--json. - Sends an idempotency key. The CLI generates one per run; pass
--idempotency-keyto retry safely.
| Flag | Type | Required | Default | Description |
|---|---|---|---|---|
--id | string | yes | Score definition ID |
Also accepts the global flags.
deployments eval score list
List recent score results.
deployments eval score list- Calls
GET /v1/score-results(SDKlistScoreResults). - Needs an organization, project, and environment:
--org,--project,--env, or the saved context. - Paginated.
--page-sizesets the page size; pass the returned cursor as--cursorfor the next page.
| Flag | Type | Required | Default | Description |
|---|---|---|---|---|
--cursor | string (cursor) | Pagination cursor | ||
--page-size | string (number) | Page size | ||
--score-definition | string | Only results for this score definition |
Also accepts the global flags.
deployments eval score trace
List score results for a trace.
deployments eval score trace --trace <trace>- Calls
GET /v1/traces/{traceId}/score-results(SDKlistTraceScoreResults). - Needs an organization, project, and environment:
--org,--project,--env, or the saved context. - Paginated.
--page-sizesets the page size; pass the returned cursor as--cursorfor the next page.
| Flag | Type | Required | Default | Description |
|---|---|---|---|---|
--cursor | string (cursor) | Pagination cursor | ||
--page-size | string (number) | Page size | ||
--trace | string | yes | Trace ID (32 lowercase hex characters) |
Also accepts the global flags.
deployments eval score record
Record a human or imported score for a trace.
deployments eval score record --trace <trace> --score-definition <score-definition> --value <number>- Calls
POST /v1/traces/{traceId}/score-results(SDKrecordTraceScoreResult). - Needs an organization, project, and environment:
--org,--project,--env, or the saved context. - Sends an idempotency key. The CLI generates one per run; pass
--idempotency-keyto retry safely.
| Flag | Type | Required | Default | Description |
|---|---|---|---|---|
--trace | string | yes | Trace ID (32 lowercase hex characters) | |
--score-definition | string | yes | Score definition ID | |
--value | string (number) | yes | Score value between 0 and 1 | |
--label | string | Score label | ||
--rationale | string | Why this score was given | ||
--run | string | Run ID within the trace | ||
--span | string | Span ID within the trace |
Also accepts the global flags.
deployments eval annotation list
List annotations for a trace.
deployments eval annotation list --trace <trace>- Calls
GET /v1/traces/{traceId}/annotations(SDKlistTraceAnnotations). - Needs an organization, project, and environment:
--org,--project,--env, or the saved context. - Paginated.
--page-sizesets the page size; pass the returned cursor as--cursorfor the next page.
| Flag | Type | Required | Default | Description |
|---|---|---|---|---|
--cursor | string (cursor) | Pagination cursor | ||
--page-size | string (number) | Page size | ||
--trace | string | yes | Trace ID (32 lowercase hex characters) |
Also accepts the global flags.
deployments eval annotation create
Annotate a trace.
deployments eval annotation create --trace <trace> --body <body>- Calls
POST /v1/traces/{traceId}/annotations(SDKcreateTraceAnnotation). - Needs an organization, project, and environment:
--org,--project,--env, or the saved context. - Sends an idempotency key. The CLI generates one per run; pass
--idempotency-keyto retry safely.
| Flag | Type | Required | Default | Description |
|---|---|---|---|---|
--trace | string | yes | Trace ID (32 lowercase hex characters) | |
--body | string | yes | Annotation text | |
--label | string | Annotation label | ||
--tags | string (a,b,c) | Annotation tags | ||
--run | string | Run ID within the trace | ||
--span | string | Span ID within the trace |
Also accepts the global flags.
deployments eval dataset create
Create a regression dataset.
deployments eval dataset create --name <name>- Calls
POST /v1/regression-datasets(SDKcreateRegressionDataset). - Needs an organization, project, and environment:
--org,--project,--env, or the saved context. - Sends an idempotency key. The CLI generates one per run; pass
--idempotency-keyto retry safely.
| Flag | Type | Required | Default | Description |
|---|---|---|---|---|
--name | string | yes | Regression dataset name | |
--description | string | Regression dataset description | ||
--retention-days | string (number) | Days to retain captured cases (1-3650) |
Also accepts the global flags.
deployments eval dataset list
List regression datasets.
deployments eval dataset list- Calls
GET /v1/regression-datasets(SDKlistRegressionDatasets). - Needs an organization, project, and environment:
--org,--project,--env, or the saved context. - Paginated.
--page-sizesets the page size; pass the returned cursor as--cursorfor the next page.
| Flag | Type | Required | Default | Description |
|---|---|---|---|---|
--cursor | string (cursor) | Pagination cursor | ||
--page-size | string (number) | Page size | ||
--status | string | Only active or archived resources |
Also accepts the global flags.
deployments eval dataset show
Show a regression dataset.
deployments eval dataset show --id <id>- Calls
GET /v1/regression-datasets/{datasetId}(SDKgetRegressionDataset). - Needs an organization, project, and environment:
--org,--project,--env, or the saved context.
| Flag | Type | Required | Default | Description |
|---|---|---|---|---|
--id | string | yes | Regression dataset ID |
Also accepts the global flags.
deployments eval dataset archive
Archive a regression dataset.
deployments eval dataset archive --id <id>- Calls
POST /v1/regression-datasets/{datasetId}/archive(SDKarchiveRegressionDataset). - Needs an organization, project, and environment:
--org,--project,--env, or the saved context. - Destructive. Live runs ask for confirmation; pass
--yesin scripts or with--json. - Sends an idempotency key. The CLI generates one per run; pass
--idempotency-keyto retry safely.
| Flag | Type | Required | Default | Description |
|---|---|---|---|---|
--id | string | yes | Regression dataset ID |
Also accepts the global flags.
deployments eval case add
Add one case from inline input or an existing run or trace.
deployments eval case add --dataset <dataset>- Calls
POST /v1/regression-datasets/{datasetId}/cases(SDKaddDatasetCase). - Needs an organization, project, and environment:
--org,--project,--env, or the saved context. - Sends an idempotency key. The CLI generates one per run; pass
--idempotency-keyto retry safely.
| Flag | Type | Required | Default | Description |
|---|---|---|---|---|
--dataset | string | yes | Regression dataset ID | |
--input | string (json) | Inline input, for example {"content":"Where is my refund?"} | ||
--source | string (json) | Captured source, for example {"type":"trace","traceId":"..."} | ||
--expectations | string (json) | Expected behavior, for example {"mustInclude":["refund"]} | ||
--external-id | string | Stable external case ID | ||
--tags | string (a,b,c) | Case tags | ||
--consent-basis | string | Consent basis for captured customer content |
Also accepts the global flags.
deployments eval case import
Import cases from a JSON case file.
deployments eval case import --dataset <dataset> --file <path>- Calls
POST /v1/regression-datasets/{datasetId}/cases/import(SDKimportDatasetCases). - Needs an organization, project, and environment:
--org,--project,--env, or the saved context. - Sends an idempotency key. The CLI generates one per run; pass
--idempotency-keyto retry safely.
| Flag | Type | Required | Default | Description |
|---|---|---|---|---|
--dataset | string | yes | Regression dataset ID | |
--file | string (path) | yes | JSON array of { id?, message, expected?: { mustInclude?, mustNot?, mustUseTools?, mustNotUseTools?, matches?, jsonSchema? }, tags? } |
Also accepts the global flags.
deployments eval case list
List cases in a regression dataset.
deployments eval case list --dataset <dataset>- Calls
GET /v1/regression-datasets/{datasetId}/cases(SDKlistDatasetCases). - Needs an organization, project, and environment:
--org,--project,--env, or the saved context. - Paginated.
--page-sizesets the page size; pass the returned cursor as--cursorfor the next page.
| Flag | Type | Required | Default | Description |
|---|---|---|---|---|
--cursor | string (cursor) | Pagination cursor | ||
--page-size | string (number) | Page size | ||
--dataset | string | yes | Regression dataset ID |
Also accepts the global flags.
deployments eval run create
Start a dataset replay or trace scoring run.
deployments eval run create --kind <kind> --score-definitions <a-b-c>- Calls
POST /v1/evaluation-runs(SDKcreateEvaluationRun). - Needs an organization, project, and environment:
--org,--project,--env, or the saved context. - Sends an idempotency key. The CLI generates one per run; pass
--idempotency-keyto retry safely.
| Flag | Type | Required | Default | Description |
|---|---|---|---|---|
--kind | string | yes | Run kind: dataset-replay or trace-score | |
--score-definitions | string (a,b,c) | yes | Score definition IDs to evaluate | |
--dataset | string | Regression dataset ID (dataset-replay) | ||
--target | string (json) | Replay target (dataset-replay), for example {"type":"release","agentReleaseId":"..."} | ||
--trace-ids | string (a,b,c) | Trace IDs to score (trace-score) | ||
--concurrency | string (number) | Cases replayed in parallel (1-20) | ||
--case-timeout | string (number) | Per-case timeout in seconds (30-3600) |
Also accepts the global flags.
deployments eval run list
List evaluation runs, newest first.
deployments eval run list- Calls
GET /v1/evaluation-runs(SDKlistEvaluationRuns). - Needs an organization, project, and environment:
--org,--project,--env, or the saved context. - Paginated.
--page-sizesets the page size; pass the returned cursor as--cursorfor the next page.
| Flag | Type | Required | Default | Description |
|---|---|---|---|---|
--cursor | string (cursor) | Pagination cursor | ||
--page-size | string (number) | Page size | ||
--status | string | Only runs in this status: queued, running, completed, failed, or canceled | ||
--dataset | string | Only replays of this regression dataset | ||
--release | string | Only replays whose target is this agent release |
Also accepts the global flags.
deployments eval run show
Show an evaluation run and its summary.
deployments eval run show --id <id>- Calls
GET /v1/evaluation-runs/{evaluationRunId}(SDKgetEvaluationRun). - Needs an organization, project, and environment:
--org,--project,--env, or the saved context.
| Flag | Type | Required | Default | Description |
|---|---|---|---|---|
--id | string | yes | Evaluation run ID |
Also accepts the global flags.
deployments eval run results
List per-case results of an evaluation run.
deployments eval run results --id <id>- Calls
GET /v1/evaluation-runs/{evaluationRunId}/results(SDKlistEvaluationRunResults). - Needs an organization, project, and environment:
--org,--project,--env, or the saved context. - Paginated.
--page-sizesets the page size; pass the returned cursor as--cursorfor the next page.
| Flag | Type | Required | Default | Description |
|---|---|---|---|---|
--cursor | string (cursor) | Pagination cursor | ||
--page-size | string (number) | Page size | ||
--id | string | yes | Evaluation run ID |
Also accepts the global flags.
deployments eval comparison create
Compare a candidate release against a baseline on a dataset.
deployments eval comparison create --baseline <baseline> --candidate <candidate> --dataset <dataset> --score-definitions <a-b-c>- Calls
POST /v1/release-comparisons(SDKcreateReleaseComparison). - Needs an organization, project, and environment:
--org,--project,--env, or the saved context. - Sends an idempotency key. The CLI generates one per run; pass
--idempotency-keyto retry safely.
| Flag | Type | Required | Default | Description |
|---|---|---|---|---|
--baseline | string | yes | Baseline agent release ID | |
--candidate | string | yes | Candidate agent release ID | |
--dataset | string | yes | Regression dataset ID | |
--score-definitions | string (a,b,c) | yes | Score definition IDs to evaluate | |
--gate-policy | string (json) | Gate thresholds, for example {"maxNewFailures":0} | ||
--runtime-config-from | string | Agent deployment whose runtime configuration both releases use | ||
--concurrency | string (number) | Cases replayed in parallel (1-20) | ||
--case-timeout | string (number) | Per-case timeout in seconds (30-3600) |
Also accepts the global flags.
deployments eval comparison list
List release comparisons, newest first.
deployments eval comparison list- Calls
GET /v1/release-comparisons(SDKlistReleaseComparisons). - Needs an organization, project, and environment:
--org,--project,--env, or the saved context. - Paginated.
--page-sizesets the page size; pass the returned cursor as--cursorfor the next page.
| Flag | Type | Required | Default | Description |
|---|---|---|---|---|
--cursor | string (cursor) | Pagination cursor | ||
--page-size | string (number) | Page size | ||
--status | string | Only comparisons in this status: queued, running, completed, or failed | ||
--baseline | string | Only comparisons with this baseline release | ||
--candidate | string | Only comparisons with this candidate release |
Also accepts the global flags.
deployments eval comparison show
Show a release comparison and its verdict.
deployments eval comparison show --id <id>- Calls
GET /v1/release-comparisons/{releaseComparisonId}(SDKgetReleaseComparison). - Needs an organization, project, and environment:
--org,--project,--env, or the saved context.
| Flag | Type | Required | Default | Description |
|---|---|---|---|---|
--id | string | yes | Release comparison ID |
Also accepts the global flags.