Skip to content

Frequently Asked Questions

General

What is BotManifold?

BotManifold is a platform for evaluating robot control policies through simulation. You write a policy (a Python function that controls a robot), submit it, and we run it in a physics simulation to see how well it performs.

Is it free?

Yes, BotManifold is free to use during the beta period.

What programming languages are supported?

Currently, only Python is supported. Your policy must be a Python function that takes an observation dictionary and returns a numpy array of actions.

Policies

What should my policy.py file look like?

import numpy as np

def policy(observation: dict) -> np.ndarray:
    # Your logic here
    return np.zeros(7)  # 7D action vector

What packages can I use?

  • numpy (pre-installed)
  • math (standard library)
  • Python standard library
  • Any pure Python code you include in your zip

Not available: PyTorch, TensorFlow, scikit-learn, or other ML libraries.

Can I use machine learning models?

Yes, but with limitations:

  • You must include the model weights in your zip file
  • Only numpy-based inference is supported
  • No GPU acceleration
  • Must complete each step within ~100ms

How do I include multiple files?

Put them all in your zip:

policy.zip
├── policy.py      # Entry point (required)
├── model.py       # Your model code
├── weights.npy    # Model weights
└── utils.py       # Helper functions

Then import them in policy.py:

from model import MyModel
from utils import helper_function

What's the action format?

A 7D numpy array:

action = np.array([
    vx,      # X velocity (-1 to 1)
    vy,      # Y velocity (-1 to 1)
    vz,      # Z velocity (-1 to 1)
    roll,    # Roll velocity (-1 to 1)
    pitch,   # Pitch velocity (-1 to 1)
    yaw,     # Yaw velocity (-1 to 1)
    gripper  # -1 = close, 1 = open
])

Submissions

How long does evaluation take?

Typically 1-5 minutes, depending on queue length:

  1. Queued: Waiting for a worker (0-2 min)
  2. Running: Simulation (~30-60 sec)
  3. Judging: AI evaluation (~30 sec)

What do the verdicts mean?

Verdict Meaning
PASS Successfully completed the task
PARTIAL Partially completed (some objectives met)
FAIL Did not meet minimum requirements

Can I see the simulation video?

Yes! Every submission generates a video showing your robot's performance. View it on the submission detail page.

How is the score calculated?

Each scenario has its own scoring formula. Generally:

  • Task completion: Main source of points
  • Time bonus: Faster completion = more points
  • Penalties: Dropping objects, collisions, etc.

See the specific scenario documentation for details.

Why did my submission fail?

Common reasons:

  1. Code error: Check for syntax errors, missing imports
  2. Timeout: Policy too slow (>100ms per step)
  3. Invalid action: Wrong shape or out-of-range values
  4. Runtime error: Exception during execution

Check the submission detail page for error messages.

Scenarios

What scenarios are available?

See the Scenarios page for the current list.

Can I suggest a new scenario?

Yes! Open an issue on GitHub with your idea.

Can I test locally before submitting?

Local testing is coming soon. For now, you can:

  1. Test your policy logic with mock observations
  2. Submit and iterate based on the video

Technical

What physics engine do you use?

MuJoCo (Multi-Joint dynamics with Contact), a high-performance physics simulator.

What robot is simulated?

A 7-DOF robotic arm with a parallel gripper, similar to a Franka Panda.

How does the AI judge work?

We use a Vision-Language Model (VLM) to evaluate the simulation video. It analyzes:

  • Whether objectives were achieved
  • Quality of execution
  • Any errors or problems

Is my code private?

Yes. Your submitted code is:

  • Used only for evaluation
  • Not shared with other users
  • Deleted after evaluation (only results are stored)

Account & Billing

Do I need an account?

Currently, no account is required. This may change in the future.

Will there be a paid tier?

Possibly, for features like:

  • Priority queue
  • More submissions per day
  • Team features

The basic tier will remain free.

Troubleshooting

My policy works locally but fails on submission

Common causes:

  1. Different Python version: We use Python 3.11
  2. Missing files: Ensure all dependencies are in the zip
  3. Absolute paths: Use relative paths for any file access

The robot doesn't do anything

Check that:

  1. Your action values are non-zero
  2. Action values are within [-1, 1]
  3. You're returning a numpy array, not a list

"Module not found" error

Ensure the module is either:

  • Part of Python standard library
  • Included in your zip file
  • numpy (pre-installed)

Contact