Verify
Run a policy through seeded simulation episodes and get a verdict and a scored report.
Verify answers one question: is this policy safe and does it do the task? You submit a policy, it runs for several episodes in a scenario in MuJoCo, and you get back a verdict, scores, a replay video of each episode and a report page.
How it works
- Submit a policy as a ZIP, or as the URL of a policy server you host. You can do this from the SDK, the CLI, the REST API, BotCI or the web app (Verify → Submit).
- Simulate. A simulation worker runs the policy for the requested number of episodes (default 5) in the scenario, recording video and telemetry: contacts, forces, end-effector velocity, workspace use and trajectory.
- Grade. A vision-language model reviews each episode's video against the scenario's success criteria. Programmatic checks (such as an object ending up in a zone, or a collision) are scored alongside it.
- Report. The episode results are aggregated into a verdict and scores.
Episodes are seeded (the base seed defaults to 42), so a given policy, scenario and seed give repeatable episodes.
The result
| Field | Meaning |
|---|---|
verdict | safe, unsafe, error (or pending while the job runs). For a multi-episode job the verdict is safe when the composite score is at least 50 |
score | The composite score as a fraction (0 to 1) |
composite_score | 0 to 100: 50% progress, 30% safety and 20% consistency across the episodes |
progress_score, safety_score | Mean over episodes, 0 to 10 |
consistency | 0 to 1, how steady results are across episodes |
success_rate | 0 to 1, fraction of episodes that met the success criteria |
vlm_agreement | Fraction of episodes where the VLM and the programmatic checks agree |
needs_review | True when the result is low-confidence and queued for human review |
video_url, report_url | A signed replay URL and the report page |
The report page shows the same data with the per-episode breakdown and videos. Low-confidence verdicts can be flagged by you and reviewed by a human; those labels are used to calibrate the grader.
Safety grade and BotCertify report
For deeper analysis the evaluation service also produces a graded report: a safety grade from S0 to S5 (each grade requires the lower ones), a performance score (0 to 100), a safety margin against force limits derived from ISO/TS 15066 body-region data, and adversarial testing against perturbed conditions. The BotCertify report in the web app presents these. They are simulation-based evidence for your own engineering decisions, not a certification by BotManifold or a regulator.
Hosting your own policy
A ZIP policy runs inside BotManifold's workers. If your policy needs a large model or a GPU, run it yourself and submit its URL. See Writing policies.
Limits
Submissions are rate limited (10 per minute per user). A job that is stuck in running for
more than 30 minutes, or in building for more than 10, is failed automatically.