Skip to content

submission_guidance_docker.md

Latest commit

 

History

History
156 lines (102 loc) · 5.76 KB

submission_guidance_docker.md

File metadata and controls

156 lines (102 loc) · 5.76 KB
Raw

RoboMME Challenge Guide: Docker Submission

This document explains how to package your policy into a Docker image that organizers can pull and run for the CVPR challenge evaluation.

We use an MME-VLA (framesamp+modul) model as an example.

What you (the participant) provide

  • A Docker image containing your policy server code and all dependencies.
  • A checkpoint location that the organizers can download.
  • One command to start your policy server inside the container.

1) Implement the policy interface and serving script

Implement the Policy class compatible with the challenge interface.

  • Copy the challenge_interface directory from the benchmark repo into your repo.

    For example, in this repo, we copied the participant-oriented files into the challenge_interface directory.

    challenge_interface
├── __init__.py
├── msgpack_numpy.py
├── policy.py
├── scripts
│   └── deploy.py
└── server.py

  • Override infer and reset in your policy implementation.

    For example, we wrapped the original MME-VLA policy in the MyPolicy_for_CVPR_Challenge class for the challenge.

  • Adjust challenge_interface/scripts/deploy.py for your own policy.

    For example, in this repo, we modified it into this for the MyPolicy_for_CVPR_Challenge class.

2) Upload your checkpoint(s)

Upload your model checkpoint(s) somewhere the organizers can download them.

  • For example, we uploaded the framesamp+modul MME-VLA model to Hugging Face.

3) Build the Docker image

We provide a challenge_interface/docs/Dockerfile example.

You may edit it to include any additional dependencies your policy requires, or use your own Dockerfile.

Build the Docker image:

docker build -f challenge_interface/docs/Dockerfile -t <my_cool_model_name>:latest .

4) Self-check locally with the benchmark eval client

  1. Run your container locally:
docker run --rm -it --gpus all \
  -e NVIDIA_DRIVER_CAPABILITIES=compute,graphics,utility,video \
  -v "$PWD/<dir>:/app/<dir>" \
  -p <port>:<port> \
  my_cool_model_name:latest

Map the server port and mount the directory correctly. Here we put all the model checkpoints under the runs directory and use port 8001.

  1. Inside the container, start the policy server using your modified deploy.py.
# Inside the container
uv run python -m challenge_interface.scripts.deploy --port <port> --checkpoint-dir <dir>
  1. From another terminal, run the benchmark eval client outside the policy server container for evaluation.
cd <robomme_benchmark>
uv run python -m challenge_interface.scripts.phase1_eval --port <port>

5) Push the Docker image to a registry

Push your image to a registry so the organizers can pull it from Docker Hub.

docker tag <my_cool_model_name>:latest <dockerhub_user>/<my_cool_model_name>:latest
docker login
docker push <dockerhub_user>/<my_cool_model_name>:latest

For example, organizers pushed an image for framesamp+modul to Docker Hub.

6) Submit your policy

Prepare the following information:

  • policy_name
  • email
  • action_space: you can only choose one of "joint_angle", "ee_pose", or "waypoint".
  • evaluation_method: set as docker.
  • Checkpoint URL (downloadable by organizers), e.g. https://huggingface.co/Yinpei/perceptual-framesamp-modul
  • Docker image (registry path + tag), e.g. <dockerhub_user>/my_cool_model_name:latest
  • Command to start the policy server, e.g. uv run python -m challenge_interface.scripts.deploy --checkpoint-dir runs/ckpts/perceptual-framesamp-modul/79999. The organizers will run deploy.py to start your policy server, then run evaluation.
  • Other flags: use_depth, use_camera_params (default: false)

An example JSON file can be found here.

What the organizers will do

After we receive your submitted JSON file, we will:

  1. Pull your docker image (based on the image name/tag you provided), for example:
docker pull yinpeidai/perceptual-framesamp-modul:latest
  1. Download your checkpoint(s) (based on the URL you provided), for example:
git clone https://huggingface.co/YinpeiDai/perceptual-framesamp-modul runs/ckpts/perceptual-framesamp-modul
  1. Run your container, for example:
docker run --rm -it --gpus all \
  -e NVIDIA_DRIVER_CAPABILITIES=compute,graphics,utility,video \
  -v "$PWD/runs:/app/runs" \
  -p 8001:8001 \
  yinpeidai/perceptual-framesamp-modul:latest

Then, inside the container, start the policy server based on the command you provided, for example:

uv run python -m challenge_interface.scripts.deploy --port 8001 --checkpoint-dir runs/ckpts/perceptual-framesamp-modul/79999
  1. Run evaluation (phase 1), using the eval script from the RoboMME benchmark repo:
cd robomme_benchmark
uv run python -m challenge_interface.scripts.phase1_eval --port 8001 --action_space joint_angle --team_id 0000

After determining the top 5–10 teams, the organizers will run phase 2 evaluation.