BotCI

BotCI runs Verify from GitHub Actions. When a workflow runs, it submits a policy checkpoint to BotManifold, waits for the verdict and reports it in the job summary. A policy that fails the check fails the workflow like a failing test, so it cannot merge.

Set up

1. Create an API key under Settings → API keys and store it as a repository secret named BOTMANIFOLD_API_KEY.
2. Add the action to a workflow:

name: Policy safety
on: [pull_request]

jobs:
  verify:
    runs-on: ubuntu-latest
    steps:
      - uses: BotManifold/botci-action@v1
        with:
          api-key: ${{ secrets.BOTMANIFOLD_API_KEY }}
          scenario-id: <scenario id>
          checkpoint-gcs-path: checkpoints/ci/${{ github.sha }}/model.ckpt
          timeout: 600

checkpoint-gcs-path is a GCS object path without the gs:// prefix. scenario-id is an id returned by GET /api/mcp/scenarios (the same list the MCP server's list_scenario_suites tool shows). The action lives in the public BotManifold/botci-action repository, which has the current input list, and botci-demo shows a working setup.

Outputs and exit codes

The action sets the outputs verdict, confidence, submission-id and detail-url, and writes a job summary with the verdict, scores and a link to the full result.

It polls with exponential backoff (5 s up to 15 s) and honors Retry-After when rate limited.

How it talks to BotManifold

BotCI uses the MCP evaluation API: POST /api/mcp/evaluations to submit, GET /api/mcp/evaluations/{id}/status to poll and GET /api/mcp/evaluations/{id} for the result. Rate limits are 10 submissions per minute and 60 reads per minute.