API Reference¶

Complete reference for the BotManifold Python SDK.

BotManifoldClient¶

The main client class for interacting with BotManifold.

Constructor¶

BotManifoldClient(
    api_key: str = None,
    base_url: str = "https://api.botmanifold.com",
    timeout: int = 30
)

Parameter	Type	Default	Description
`api_key`	str	`None`	API key (reads `BOTMANIFOLD_API_KEY` env var if not provided)
`base_url`	str	`"https://api.botmanifold.com"`	Base URL for the API
`timeout`	int	`30`	Request timeout in seconds

Raises: AuthenticationError if no API key is provided or found in environment.

Sub-clients¶

The client exposes functionality through sub-clients:

Property	Type	Description
`client.verify`	`VerifyClient`	Policy safety verification
`client.arena`	`ArenaClient`	Arena submissions and leaderboard

VerifyClient¶

Access via client.verify. Handles policy safety verification.

verify.run()¶

Submit a policy and wait for the verification result.

verify.run(
    policy: str,
    scenario: str,
    *,
    policy_url: str = None,
    num_episodes: int = 3,
    timeout: int = 600,
    wait: bool = True
) -> VerifyReport

Parameter	Type	Description
`policy`	str	Path to policy zip file
`scenario`	str	Scenario ID
`policy_url`	str	URL to a running policy server (alternative to file upload)
`num_episodes`	int	Number of evaluation episodes
`timeout`	int	Max wait time in seconds
`wait`	bool	If True, blocks until completion

Returns: VerifyReport

Example:

report = client.verify.run("policy.zip", scenario="messy_room")
print(f"Verdict: {report.verdict}")

verify.submit()¶

Submit a policy for verification without waiting.

verify.submit(policy: str, config: VerifyConfig) -> VerifyReport

Returns: VerifyReport with status QUEUED

verify.status()¶

Check the status of a verification job.

verify.status(job_id: str) -> VerifyReport

verify.wait()¶

Wait for a verification job to complete.

verify.wait(
    job_id: str,
    timeout: int = 600,
    poll_interval: int = 5
) -> VerifyReport

ArenaClient¶

Access via client.arena. Handles arena submissions and leaderboard.

arena.submit()¶

Submit a policy to the arena.

arena.submit(
    policy: str,
    scenario: str,
    description: str = None,
    *,
    policy_url: str = None
) -> Submission

Parameter	Type	Description
`policy`	str	Path to policy zip file
`scenario`	str	Scenario ID
`description`	str	Optional submission description
`policy_url`	str	URL to a running policy server

Returns: Submission

arena.status()¶

arena.status(submission_id: str) -> Submission

arena.wait()¶

arena.wait(
    submission_id: str,
    timeout: int = 600,
    poll_interval: int = 5
) -> Submission

arena.leaderboard()¶

arena.leaderboard(
    scenario: str,
    limit: int = 20,
    offset: int = 0
) -> list[LeaderboardEntry]

arena.list_scenarios()¶

arena.list_scenarios() -> list[Scenario]

Known Limitation

Currently returns a limited set of scenarios. Use the REST API /scenarios endpoint for the full list.

Data Classes¶

VerifyReport¶

Verification result returned by verify.run(), verify.status(), and verify.wait().

Attribute	Type	Description
`job_id`	str	Unique job ID
`status`	str	Job status (QUEUED, RUNNING, COMPLETED, FAILED)
`verdict`	Verdict	Safety verdict (SAFE, UNSAFE, ERROR, PENDING)
`score`	float	Overall score
`composite_score`	float	Composite evaluation score
`progress_score`	float	Task progress score
`safety_score`	float	Safety evaluation score
`consistency`	float	Cross-episode consistency
`success_rate`	float	Episode success rate (0.0-1.0)
`vlm_agreement`	bool	Whether VLM judge agrees with automated scores
`needs_review`	bool	Whether manual review is recommended
`episode_details`	list	Per-episode breakdown
`video_url`	str	URL to result video

Verdict¶

Enum for verification verdicts.

Value	Description
`SAFE`	Policy passed safety verification
`UNSAFE`	Policy failed safety checks
`ERROR`	Error during verification
`PENDING`	Verification in progress

Submission¶

Arena submission returned by arena.submit(), arena.status(), and arena.wait().

Attribute	Type	Description
`id`	str	Unique submission ID
`scenario_id`	str	Scenario this submission is for
`status`	str	Current status
`verdict`	VLMVerdict	Judge verdict
`score`	float	Numerical score
`confidence`	float	Judge confidence (0.0-1.0)
`video_url`	str	URL to result video
`error_message`	str	Error message if failed
`created_at`	datetime	Submission timestamp

Scenario¶

Attribute	Type	Description
`id`	str	Unique scenario ID
`name`	str	Display name
`description`	str	Short description
`difficulty`	int	Difficulty level (1-3)
`time_limit`	int	Max episode duration in seconds
`max_score`	int	Maximum achievable score

LeaderboardEntry¶

Attribute	Type	Description
`rank`	int	Leaderboard position
`submission_id`	str	Submission ID
`score`	float	Score
`verdict`	str	Verdict
`created_at`	datetime	Submission timestamp

VerifyConfig¶

Configuration for verification jobs.

Attribute	Type	Description
`scenario`	str	Scenario ID
`num_episodes`	int	Number of evaluation episodes
`policy_url`	str	URL to policy server

Exceptions¶

BotManifoldError¶

Base exception for all SDK errors.

from botmanifold import BotManifoldError

try:
    client.verify.run(...)
except BotManifoldError as e:
    print(f"Error: {e}")

AuthenticationError¶

Raised on 401/403 responses or missing API key.

Attribute	Type	Description
`status_code`	int	HTTP status code
`message`	str	Error message

from botmanifold import AuthenticationError

try:
    client = BotManifoldClient()  # No API key
except AuthenticationError:
    print("Set BOTMANIFOLD_API_KEY environment variable")

APIError¶

Raised when the API returns an error response.

Attribute	Type	Description
`status_code`	int	HTTP status code
`message`	str	Error message

CLI¶

The SDK includes a command-line interface.

# Verify a policy
botmanifold verify policy.zip --scenario messy_room

# Arena submission
botmanifold arena submit policy.zip --scenario messy_room
botmanifold arena status <submission_id>
botmanifold arena leaderboard --scenario messy_room

# Validate scenario YAML
botmanifold scenario validate --scenario object_sorting

# Run a local policy server
botmanifold serve my_policy.py

# Start a mock server for testing
botmanifold mock-server

Environment Variables¶

Variable	Description	Default
`BOTMANIFOLD_API_KEY`	API key for authentication	(required)
`BOTMANIFOLD_API_URL`	Override API base URL	`https://api.botmanifold.com`

export BOTMANIFOLD_API_KEY=bm_your_api_key