Getting Started

This guide verifies a small policy against the Messy Room scenario and reads the result.

Prerequisites

• Python 3.10 or newer
• A BotManifold account and an API key (create one under Settings → API keys)

1. Install the SDK

pip install botmanifold
export BOTMANIFOLD_API_KEY=bm_...

See Installation for the extras and the current PyPI version.

2. Write a policy

A policy is a Python file that turns an observation into an action. Each scenario defines its own observation keys and action size; see the scenario reference. Messy Room gives you the arm's joint positions, the end-effector position and the target position, and expects 3 control values.

import numpy as np


def policy(observation: dict) -> np.ndarray:
    """Move the end effector toward the target cube."""
    ee = np.asarray(observation["end_effector_pos"])
    target = np.asarray(observation["target_pos"])
    direction = target - ee
    return np.clip(direction * 5.0, -1.0, 1.0)  # 3 control values

The loader looks for a function named act, policy, my_policy or get_action, or a class named Policy, MyPolicy, Agent or Bot with an act method. See Writing policies.

3. Package and verify

zip policy.zip policy.py
botmanifold verify policy.zip --scenario messy_room

Or from Python:

from botmanifold import BotManifoldClient

client = BotManifoldClient()  # reads BOTMANIFOLD_API_KEY
report = client.verify.run("policy.zip", scenario="messy_room")
print(report.verdict, report.score)

You can also submit from the web app at Verify → Submit.

4. Read the result

A job moves through queued, running (the simulation, in MuJoCo) and grading, then is complete. The report has:

• Verdict: safe, unsafe or error
• Scores: a composite score (0 to 100) with progress and safety (0 to 10), consistency (0 to 1) and success rate (0 to 1)
• Episodes: a per-episode breakdown and a replay video of each run
• Report page: report.report_url opens the full report in the web app

See Verify for how the verdict is produced.

Verify a policy you host yourself

If your policy needs dependencies a ZIP can't carry (a large model, a GPU), serve it and submit its URL:

botmanifold serve my_policy.py
# Pairing token: 7f3a...

botmanifold verify --url https://my-gpu-box:8765 --token 7f3a... --scenario messy_room

The URL must be reachable from the internet; addresses on private networks and localhost are rejected. See Writing policies.

Next steps

• Scenarios: the available tasks
• SDK reference
• Debugging tips
• Studio: try policies in the browser