MultiDojo²
The World's First Self-Evolving Leaderboard for Heterogeneous Multi-Robot Collaboration
English · 简体中文
Project website · Dataset · Tasks · Evaluation · RSI · Get started
| Training demonstrations | Task families | Robot teams | RSI |
|---|---|---|---|
| 2,000+ | 20+ · 100 demos each | 2 / 3 / 4 / 8 arms | Learn & Evolve |
Collaboration in action
▶ Watch the complete ten-arm assembly · Watch the collection overview
Ten matching arms share an assembly sequence, transferring a workpiece between stations and coordinating each step toward a completed fan.
Dataset
A balanced collection of 2,000+ reviewed simulation demonstrations across 20+ collaborative manipulation tasks. Each task contributes 100 episodes, covering handovers, relays, shared transport, ordered stacking, and parallel manipulation.
- See the team and the local interaction. Every arm has a first-person view, accompanied by third-person views of the shared workspace. Follow grasping details and inter-robot coordination on the same timeline.
- Learn from aligned observations and actions. RGB, robot state, native joint commands, next observations, and camera calibration preserve the connection between what each robot sees and how it moves.
- Study collaboration across robot designs. Homogeneous and heterogeneous teams span two, three, four, and eight arms. Native joint orders, units, and actuator masks retain each robot's control semantics.
- Explore meaningful scene variation. Object and goal positions, workspaces, and scene appearance vary while preserving the task's manipulation objective.
- Use physically reviewed episodes. Acceptance checks cover grasping and contact, object support, placement stability, arm-to-arm collision constraints, synchronized images, and file integrity.
Download training data on Hugging Face →
The 2,000+ demonstration collection is being progressively released on Hugging Face, with synchronized first-person and third-person views, native states/actions, and acceptance metadata. Download and loading instructions are provided in the dataset repository.
Data format & loader example → · Machine-readable task catalog →
Collaboration across embodiments
Heterogeneous robot teams
Different manipulators take complementary roles in the same assembly workflow, coordinating transfers and operations around a shared workpiece.
▶ Watch the complete heterogeneous assembly
Humanoids and manipulators
Humanoids and robot arms extend the collaboration across body types, working together through the assembly sequence.
▶ Watch the complete humanoid–manipulator assembly
Fan assembly highlights long-horizon, multi-stage coordination across homogeneous, heterogeneous, and humanoid teams.
All-to-One. One-to-All.
Robot automation scales through collaboration. Shared tasks call for a shared understanding of the workspace, coordinated timing, and actions suited to each robot's role.
All-to-One brings observations and states into a common view of the world. One-to-All turns that shared understanding into coordinated actions across different robots. One World, One Model expresses the research vision behind this approach.
Tasks
The suite spans five collaboration skills: handover, relay, shared lifting, stacking, and parallel manipulation. Each task includes 100 training demonstrations, with evaluation designed around varied scene conditions and initial states.
| ID | Task | Team | Skill |
|---|---|---|---|
| 00 | Banana handover into a pan | Homogeneous · 2 arms | Handover |
| 01 | Banana and apple to plates | Homogeneous · 2 arms | Parallel manipulation |
| 02 | Collaborative carton transport | Homogeneous · 2 arms | Shared lifting |
| 03 | Two-stage cup relay | Homogeneous · 2 arms | Relay |
| 04 | Place and stack blocks | Homogeneous · 2 arms | Stacking |
| 05 | Franka–Piper long-block handover | Heterogeneous · 2 arms | Handover |
| 06 | Three-arm block relay | Homogeneous · 3 arms | Relay |
| 07 | Apples to baskets | Homogeneous · 4 arms | Parallel manipulation |
| 08 | Four-arm block relay | Homogeneous · 4 arms | Relay |
| 09 | Ordered four-color stacking | Homogeneous · 4 arms | Stacking |
| 10 | Cookies to boxes | Homogeneous · 4 arms | Parallel manipulation |
| 11 | Household objects to boxes | Homogeneous · 4 arms | Parallel manipulation |
| 12 | Eight-arm blocks to boxes | Homogeneous · 8 arms | Parallel manipulation |
| 13 | Three-arm block relay | Heterogeneous · 3 arms | Relay |
| 14 | Four-arm block relay | Heterogeneous · 4 arms | Relay |
| 15 | Eight-arm blocks to boxes | Heterogeneous · 8 arms | Parallel manipulation |
| 16 | Three-arm banana relay | Heterogeneous · 3 arms | Relay |
| 17 | Three-arm white-cup relay | Heterogeneous · 3 arms | Relay |
| 18 | Four-arm banana relay | Heterogeneous · 4 arms | Relay |
| 19 | Four-arm white-cup relay | Heterogeneous · 4 arms | Relay |
Canonical task identifiers
00 BananaDualArmHandover 01 IsomorphicDualArm/BananaAppleToPlates 02 IsomorphicDualArm/CollabLiftLongBox 03 IsomorphicDualArm/CupLeftToMidToRight 04 IsomorphicDualArm/LeftPlaceRightStack 05 HeterogeneousDualArm/FrankPiperBananaHandover 06 IsomorphicTripleArm/BlockZoneRelay 07 IsomorphicQuadArm/AppleToBasket 08 IsomorphicQuadArm/BlockZoneRelay 09 IsomorphicQuadArm/ColorBlockCenterStack 10 IsomorphicQuadArm/CookieToBox 11 IsomorphicQuadArm/HouseholdToBox 12 IsomorphicOctaArm/BlocksToBoxes 13 HeterogeneousTripleArm/BlockZoneRelay 14 HeterogeneousQuadArm/BlockZoneRelay 15 HeterogeneousOctaArm/BlocksToBoxes 16 HeterogeneousTripleArm/BananaZoneRelay 17 HeterogeneousTripleArm/WhiteCupZoneRelay 18 HeterogeneousQuadArm/BananaZoneRelay 19 HeterogeneousQuadArm/WhiteCupZoneRelay
The task catalog connects IDs 00–19 with task names, camera sets, and episode ranges in the training index. Task and identifier guide →
Evaluation
Test collaboration under controlled scene variation. Evaluation varies object and goal positions, permitted appearance, and initial states while preserving task instructions and physical requirements.
Task success rate = successful trials / evaluated trials
Overall score = 100 × mean(task success rate across evaluated task families)
Success combines task completion with the required grasp, transfer order, support, destination, stability, and collision constraints. Equal weighting across task families makes results easy to compare, while recorded execution evidence supports reproducibility and review.
Online evaluation is coming soon.
Evaluation protocol & policy integration →
RSI
Turn replay evidence into a focused training objective. RSI connects execution records with failure explanations and concrete follow-up tasks, helping researchers decide what a policy should practise next.
A training recommendation specifies the participating robots, object, initial and goal ranges, collaboration order, motion constraints, and success conditions. Candidate scenes pass native physical validation before entering training; re-evaluation measures the effect of the next training round.
Use the RSI workflow guide to connect scene recommendations to your training pipeline. RSI interface guide / 中文 →
Quick start
Preview the project with Python 3.10+:
python3 scripts/check_release.py
python3 -m http.server 8000 --bind 127.0.0.1 --directory .
Open localhost:8000, then explore the dataset, task catalog, and evaluation guide.
Research & usage guide · Deployment guide
For questions or contributions, include the task ID, episode or trial ID, and the steps needed to reproduce the issue.
Website code and documentation: MIT License · Attribution.
2,000+ trajectories in motion
▶ Explore the full collection · 4K · 54 seconds
From dual-arm handovers to eight-arm workstations: a continuously scrolling panorama of collaborative manipulation, accompanied by assembly showcases.