RLWRLD's DexBench is now live on Manifold
Bifrost's evaluation platform, Manifold, now hosts DexBench, making it easier for robotics researchers and developers to evaluate dexterous manipulation policies at scale.
DexBench is a new open benchmark developed by RLWRLD on NVIDIA Isaac Lab-Arena, an open source framework for scaled robot policy evaluation, designed to evaluate bimanual robotic dexterity on industrial tasks.
Manifold is the fastest way to scale robotics simulations. GPU orchestration accelerates simulation workloads, while in-depth failure analysis helps you pinpoint critical failure modes, uncover actionable insights, and rapidly improve robot policy performance. Run more simulations, understand failures, and rapidly improve robot policies.
New to Manifold? You can start evaluating your policies in minutes, with free evaluation credits to get started.
What does DexBench measure?
DexBench focuses on manipulation capabilities required in real-world manufacturing, logistics, and service environments.
RLWRLD developed the benchmark by studying recurring manipulation tasks across these industries and identifying the underlying skills robots need to perform them reliably.
The benchmark covers tasks such as bimanual regrasping, precision insertion, object manipulation, tool use, and handling objects in constrained environments.
Rather than evaluating only whether a policy can complete a task under fixed conditions, DexBench also tests how well it generalizes when the environment changes.
How is performance measured?
DexBench measures task success rates and evaluates policy generalization across three dimensions:
- Object generalization: Can the policy handle unseen objects with different geometries, sizes, and appearances?
- Physics generalization: Does the policy remain reliable when physical properties such as mass, friction, and stiffness change?
- Spatial generalization: Can the policy handle variations in object placement, orientation, clutter, and spatial constraints?
Object and spatial generalization are available in early access, with physics generalization planned for general availability.
Together, these evaluations help researchers understand not just how often a policy succeeds, but how robust it is to changes in the task environment.
Running DexBench on Manifold
We've integrated and optimized DexBench on Manifold so you can run evaluations without having to configure Isaac Lab-Arena, provision GPUs, or build your own evaluation pipeline or manage your simulation infrastructure.
Simply connect your policy, select the benchmark, and start evaluating.
Manifold distributes simulation workloads across parallel GPU compute, allowing teams to run more rollouts and iterate faster.
For every evaluation, you get:
- Per-task success rates: See how your policy performs across individual benchmark tasks.
- 3D episode replays: Inspect individual rollouts to understand successful and failed behaviors.
- Automated failure analysis: Identify which subgoals fail, cluster recurring failure modes, and understand where your policy struggles.
- Reproducible evaluations: Track policy checkpoints, simulator and benchmark versions, and random seeds for consistent comparisons.
Instead of relying on aggregate success rates alone, you can drill into individual failures and use those results to guide the next iteration of your policy.
DexBench joins LIBERO, RoboLab, RoboCasa, CALVIN, SIMPLER, and RoboTwin 2.0 on Manifold, allowing researchers to evaluate policies across multiple benchmarks from a single platform.
Benchmarking NVIDIA GR00T 1.7 on DexBench
To establish an initial reference point, we've evaluated NVIDIA GR00T 1.7, open generalist robot VLA model on DexBench using Manifold.
Below, we share per-task success rates, evaluation configurations, and rollout results to provide a baseline for other researchers evaluating their own policies.
GR00T 1.7 succeeded in 564 of 1,680 episodes, a 33.6% overall success rate.
| Task | Train | Object | Spatial | Object + spatial | All |
|---|---|---|---|---|---|
| Push the detergent drawer into the washer's slot | 85.0% | – | 70.0% | – | 77.5% |
| Lift a package off the cage with both hands onto the conveyor | 47.5% | 45.0% | 55.0% | 45.0% | 48.1% |
| Move a stack of bowls or plates from the banquet table to the serving cart | 41.7% | 20.0% | 29.2% | 26.7% | 29.4% |
| Take a drink out of the refrigerator's second shelf and put it in the basket | 5.0% | 0.0% | 0.0% | 0.0% | 1.3% |
| All tasks | 44.8% | 21.7% | 38.5% | 23.9% | 33.6% |
The Train column uses the conditions each task was trained on. The other columns change the objects, their placement, or both. Each cell is 120 episodes. The drawer task has no object variations, so it has no Object or Object + spatial result.
Across all tasks, success falls from 44.8% under training conditions to 38.5% with new placements and 21.7% with unseen objects, so new objects cost the policy more than new placements. The refrigerator task is the hardest. GR00T 1.7 completed it in only 6 of 120 episodes under training conditions and in none of the 360 episodes with new drinks, new placements, or both.
Try DexBench on Manifold
DexBench is now available on Manifold.
Connect your policy, run the benchmark, and explore the results. New users receive free evaluation credits, which can also be used across other supported benchmarks.
