Docs menu

// Command reference

simforge open-loop

Generated from the CLI's own help (CLI 0.2.0-rc.9). Global flags and exit codes are on the reference index.

All commands

simforge open-loop artifacts

A download grant ({url, sha256, bytes, ...}) for one artifact of the evaluation's result

simforge open-loop artifacts [OPTIONS] <EVALUATION_ID> <ARTIFACT_ID>
Argument / flagNotesWhat
<evaluation_id>requiredThe evaluation (run_..., or cmpj_... from before cloud runs)
<artifact_id>requiredThe artifact, by its id in the result manifest

simforge open-loop list

List the workspace's evaluations, whichever model ran them

simforge open-loop list [OPTIONS]
Argument / flagNotesWhat
--limit LIMITdefault 20At most this many items on the page
--cursor CURSORThe next page: the nextCursor of the page before it
--status STATUSOnly evaluations in these statuses (comma-separated)
--model MODELOnly one model's evaluations, by name (case-insensitive) or id: a model the host lists (GET /api/v1/evaluation-options), or a model version of your own by id. Default: every model's

simforge open-loop result

The evaluation's result manifest: scored metrics, artifacts, model/render/source pins and compute cost

simforge open-loop result [OPTIONS] <EVALUATION_ID>
Argument / flagNotesWhat
<evaluation_id>requiredThe evaluation (run_..., or cmpj_... from before cloud runs)

simforge open-loop run

Score a scenario's authored episode with a driving model the host offers for open loop, named by --model (requires --cloud, --model, --max-cost and --idempotency-key, also with --dry-run). The help's models lists the host's open-loop models

simforge open-loop run [OPTIONS] --max-cost <CENTS> <SCENARIO_ID>
Argument / flagNotesWhat
<scenario_id>requiredThe scenario to evaluate
--model MODELThe model that predicts, by name (case-insensitive) or id: one of the models the host offers for open loop now (GET /api/v1/evaluation-options, each model's loops.openLoop; run without it to list them). Required
--seed SEEDModel sampling seed (the model's seed parameter). Default: the model's
--samples NTrajectory samples drawn by the model (its numTrajSamples parameter). Default: the model's; a model with a fixed count accepts only that
--param NAME=VALUErepeatableOne of the model's parameters, as NAME=VALUE (repeatable): the options' params of the model, typed as listed (e.g. --param temperature=0.8). Absent ones take their defaults
--max-cost CENTSrequiredExplicit maximum cloud cost in cents; required, also with --dry-run
--waitWait for the evaluation to finish

simforge open-loop show

One evaluation: status, cost and attempts

simforge open-loop show [OPTIONS] <EVALUATION_ID>
Argument / flagNotesWhat
<evaluation_id>requiredThe evaluation (run_..., or cmpj_... from before cloud runs)

simforge open-loop watch

Follow an evaluation until it succeeds, is partial, fails or is cancelled

simforge open-loop watch [OPTIONS] <EVALUATION_ID>
Argument / flagNotesWhat
<evaluation_id>requiredThe evaluation (run_..., or cmpj_... from before cloud runs)