// Command reference
simforge open-loop
Generated from the CLI's own help (CLI 0.2.0-rc.9). Global flags and exit codes are on the reference index.
simforge open-loop artifacts
A download grant ({url, sha256, bytes, ...}) for one artifact of the evaluation's result
simforge open-loop artifacts [OPTIONS] <EVALUATION_ID> <ARTIFACT_ID>
| Argument / flag | Notes | What |
|---|---|---|
| <evaluation_id> | required | The evaluation (run_..., or cmpj_... from before cloud runs) |
| <artifact_id> | required | The artifact, by its id in the result manifest |
simforge open-loop list
List the workspace's evaluations, whichever model ran them
simforge open-loop list [OPTIONS]
| Argument / flag | Notes | What |
|---|---|---|
| --limit LIMIT | default 20 | At most this many items on the page |
| --cursor CURSOR | The next page: the nextCursor of the page before it | |
| --status STATUS | Only evaluations in these statuses (comma-separated) | |
| --model MODEL | Only one model's evaluations, by name (case-insensitive) or id: a model the host lists (GET /api/v1/evaluation-options), or a model version of your own by id. Default: every model's |
simforge open-loop result
The evaluation's result manifest: scored metrics, artifacts, model/render/source pins and compute cost
simforge open-loop result [OPTIONS] <EVALUATION_ID>
| Argument / flag | Notes | What |
|---|---|---|
| <evaluation_id> | required | The evaluation (run_..., or cmpj_... from before cloud runs) |
simforge open-loop run
Score a scenario's authored episode with a driving model the host offers for open loop, named by --model (requires --cloud, --model, --max-cost and --idempotency-key, also with --dry-run). The help's models lists the host's open-loop models
simforge open-loop run [OPTIONS] --max-cost <CENTS> <SCENARIO_ID>
| Argument / flag | Notes | What |
|---|---|---|
| <scenario_id> | required | The scenario to evaluate |
| --model MODEL | The model that predicts, by name (case-insensitive) or id: one of the models the host offers for open loop now (GET /api/v1/evaluation-options, each model's loops.openLoop; run without it to list them). Required | |
| --seed SEED | Model sampling seed (the model's seed parameter). Default: the model's | |
| --samples N | Trajectory samples drawn by the model (its numTrajSamples parameter). Default: the model's; a model with a fixed count accepts only that | |
| --param NAME=VALUE | repeatable | One of the model's parameters, as NAME=VALUE (repeatable): the options' params of the model, typed as listed (e.g. --param temperature=0.8). Absent ones take their defaults |
| --max-cost CENTS | required | Explicit maximum cloud cost in cents; required, also with --dry-run |
| --wait | Wait for the evaluation to finish |
simforge open-loop show
One evaluation: status, cost and attempts
simforge open-loop show [OPTIONS] <EVALUATION_ID>
| Argument / flag | Notes | What |
|---|---|---|
| <evaluation_id> | required | The evaluation (run_..., or cmpj_... from before cloud runs) |
simforge open-loop watch
Follow an evaluation until it succeeds, is partial, fails or is cancelled
simforge open-loop watch [OPTIONS] <EVALUATION_ID>
| Argument / flag | Notes | What |
|---|---|---|
| <evaluation_id> | required | The evaluation (run_..., or cmpj_... from before cloud runs) |