Skip to content

Scenarios

Scenarios are pre-defined challenges that test different aspects of robot control. Each scenario has specific objectives, scoring criteria, and constraints.

Available Scenarios

Scenario Difficulty Description VLA Variant
Navigation Easy Move end-effector to target location --
Object Sorting Easy Sort objects by color into bins --
Obstacle Avoidance Easy Navigate around obstacles to reach target --
Messy Room Medium Clean up scattered items in a room messy_room_vla
Kitchen Pickup Medium Pick up items from kitchen counter kitchen_pickup_vla
Mug Placement Medium Place mug on designated coaster/shelf --
Table Clearing Medium Clear objects from a table surface --
Block Stacking Hard Stack blocks in the correct order block_stacking_vla
Precision Placement Hard Place objects at exact coordinates --
Dual Arm Handover Hard Two-arm object handover coordination dual_arm_handover_vla
Warehouse Retrieval Hard Retrieve items from shelved storage --

15 total scenarios (11 base + 4 VLA variants). VLA variants use Vision-Language-Action models for evaluation.

Difficulty Levels

  • Easy: Good for beginners, simple objectives, forgiving scoring
  • Medium: Requires planning and coordination
  • Hard: Complex manipulation, precise timing, multi-step reasoning

Common Concepts

Observations

Every scenario provides observations to your policy:

observation = {
    'robot_state': np.array([...]),      # Joint positions, velocities
    'object_positions': np.array([...]), # XYZ of each object
    'target_positions': np.array([...]), # Goal positions (if applicable)
    'gripper_state': float,              # Gripper openness (0-1)
    'time_remaining': float,             # Seconds left
}

Actions

Policies return a 7D action vector:

action = np.array([
    vx,      # X velocity (-1 to 1)
    vy,      # Y velocity (-1 to 1)
    vz,      # Z velocity (-1 to 1)
    roll,    # Roll velocity (-1 to 1)
    pitch,   # Pitch velocity (-1 to 1)
    yaw,     # Yaw velocity (-1 to 1)
    gripper  # Gripper command (-1=close, 1=open)
])

Scoring

Each scenario defines its own scoring criteria, but common factors include:

  • Task Completion: Did the robot achieve the objective?
  • Efficiency: How quickly was the task completed?
  • Safety: Were there collisions or drops?

Time Limits

All scenarios have a maximum episode length. Your policy should:

  • Prioritize the most important objectives
  • Not spend too long on any single subtask
  • Handle the time_remaining observation appropriately

Writing a Good Policy

  1. Understand the Objective: Read the scenario documentation carefully
  2. Start Simple: Get a basic working policy before optimizing
  3. Test Locally: Use botmanifold serve and botmanifold mock-server for local testing
  4. Iterate: Watch videos to understand failure modes

See the Writing Policies guide for more tips.