Top Datasets For Robot Manipulation, VLA Training & Policy Learning

Not All Datasets Are Equal

The hardest part of building a manipulation policy isn't the model architecture, it's the data underneath it. Vision-language-action models are hungry for demonstrations, and the field has responded with a wave of large open datasets: teleoperated pick-and-place, simulated kitchens, egocentric human video, and cross-embodiment collections that pool dozens of labs into one format. That is a genuine gift to anyone training a policy today.

But scale alone doesn't make a dataset useful for your task. A million trajectories from one kitchen still won't teach a policy to generalize to a new gripper or a cluttered workbench. The datasets below vary enormously in embodiment diversity, task breadth, label quality, and license terms, and those differences show up directly in how well a trained policy transfers to your robot.

Why Does It Matter?

We've seen firsthand how the wrong training mix produces a policy that looks great on a held-out validation split and then falls apart the moment the gripper, camera angle, or lighting changes. By sharing this list of the datasets we see referenced most often in VLA and manipulation research, we want to give you a framework for choosing data that actually matches your deployment target, not just the biggest number on a leaderboard.

Whether you're pretraining a foundation policy, fine-tuning on a single task, or building an evaluation suite, this list should help you find the right data for the job.

How Do We Measure Dataset Quality?

Every dataset on this list gets assessed against the same three questions, alongside the plain facts (year, scale, modalities, license) so you can filter fast.

1. Embodiment and Task Diversity

The single biggest predictor of whether a policy generalizes is how many distinct robots, grippers, and camera viewpoints it trained on. A dataset collected on one arm in one lab teaches a policy that arm's quirks as much as it teaches the task. Cross-embodiment datasets trade some per-task depth for breadth, which tends to pay off the moment you deploy on hardware the policy has never seen.

• High diversity spans multiple robot arms, grippers, camera placements, and physical scenes, so a policy learns the task rather than the rig.
• Low diversity is collected on a single robot in a fixed setup. Still useful for benchmarking or fine-tuning, but risky as the sole source for a generalist policy.

2. Demonstration and Label Quality

Manipulation data lives or dies on how clean the demonstrations are. Teleoperated data with jittery, non-expert operators teaches a policy the same hesitation and correction behavior it recorded. Simulated data solves this by construction, but only for the tasks and objects the simulator's asset library covers.

• High-quality demonstrations come from proficient operators or scripted/procedural generation, are paired with precise action and proprioceptive logs, and include language annotations that actually describe what's happening in the clip.
• Low-quality demonstrations mix skill levels inconsistently, have sparse or noisy language labels, or use action representations (joint space vs. end-effector, absolute vs. delta) that don't match what your policy expects without conversion.

3. License and Reuse Terms

The most useful dataset in the world is a liability if the license doesn't match your use case. Robotics datasets range from fully permissive (Apache 2.0, MIT, CC-BY) to research-only (CC BY-NC) to bespoke agreements gating anything with human faces in frame. This matters more here than in most ML subfields, because a lot of manipulation and VLA work is happening inside commercial labs shipping real products.

• Permissive licenses (Apache 2.0, MIT, CC-BY 4.0) allow commercial use with attribution, which is what most teams shipping a product actually need.
• Restrictive licenses (CC BY-NC-SA, custom research agreements) are fine for academic benchmarking but will block a commercial pretraining run unless you negotiate separately.

With that framework in mind, here are 12 of the datasets we see cited most often for manipulation and VLA training, roughly ordered from broadest cross-embodiment collections to narrower, more specialized ones.

1. Open X-Embodiment

Open X-Embodiment is the closest thing the field has to a shared substrate for VLA pretraining. It pools more than 60 existing robot datasets from over 20 collaborating institutions into a single RLDS format, spanning single arms, bimanual rigs, and quadrupeds. Google DeepMind trained the RT-X models on it and showed positive transfer: a policy trained across embodiments outperformed embodiment-specific baselines on several of them. The catch is quality is uneven across the pooled sub-datasets, since some come from research labs with very different recording setups and demonstration quality, so treat it as a pretraining corpus rather than a drop-in fine-tuning set for any one robot.

• Year: 2023
• Scale: 1M+ real robot trajectories, 22 robot embodiments, pooled from 60+ datasets across 20+ institutions
• Modalities: RGB (multiple datasets add depth, wrist cameras, and proprioception), language instructions, action and proprioceptive logs
• Tasks/Embodiments: Pick-and-place, manipulation, navigation-adjacent tasks across single arms, bimanual rigs, and quadrupeds; 527 skills spanning 160,266 tasks
• License: Apache 2.0 (software), CC-BY 4.0 (data)
• Link: robotics-transformer-x.github.io

2. DROID

DROID is the dataset to reach for if the gap you're worried about is scene diversity rather than embodiment diversity. It was collected by 50 data collectors across 13 institutions in North America, Asia, and Europe over 12 months, using a shared Franka Panda hardware kit, so every trajectory is on the same robot but in wildly different homes, offices, and labs. That combination, one embodiment held constant, everything else varied, makes it particularly good for training perception and manipulation policies that need to hold up across real-world clutter rather than a lab bench. Each episode has three synchronized RGB streams plus depth and calibration, which is more camera coverage than most academic datasets bother with.

• Year: 2024
• Scale: 76,000 demonstration trajectories (~350 hours), 564 scenes, 86 tasks, 13 institutions, 18 robots
• Modalities: 3 synchronized RGB camera streams, depth, camera calibration, language instructions, proprioception
• Tasks/Embodiments: General manipulation across 86 tasks and 125 object categories, single embodiment (Franka Panda)
• License: CC-BY 4.0
• Link: droid-dataset.github.io

3. BridgeData V2

BridgeData V2 is the workhorse dataset for low-cost, WidowX-based manipulation research. It's smaller and older than DROID but earns its place on this list because of how widely it's used as a fine-tuning and evaluation target: a lot of published VLA results report BridgeData numbers specifically because the setup is cheap enough for other labs to replicate on their own WidowX arms. The task set leans toward kitchen-style manipulation (pick-and-place, pushing, sweeping, door and drawer manipulation, stacking, cloth folding), so it's a good fit if your target domain looks similar, less so if you need industrial or bimanual coverage.

• Year: 2023
• Scale: 53,896 trajectories, 13 skills, 24 environments
• Modalities: RGB, language instructions, action logs
• Tasks/Embodiments: Pick-and-place, pushing, sweeping, door/drawer manipulation, stacking, cloth folding; single embodiment (WidowX 250 arm)
• License: CC-BY 4.0
• Link: rail-berkeley.github.io/bridgedata

4. RT-1 Dataset

The RT-1 dataset is worth including on its own rather than folding it entirely into Open X-Embodiment, because it's the dataset that first demonstrated real-world transformer policies could hit high success rates (97% on trained instructions) at meaningful scale. Google collected it with a fleet of 13 robots over 17 months, covering more than 700 distinct instructions across picking, placing, opening and closing drawers, and similar tabletop tasks. It's now a component of Open X-Embodiment, but if you specifically want a large, single-embodiment, high-consistency dataset for a tabletop arm, it's still worth pulling standalone.

• Year: 2022
• Scale: 130,000+ episodes, 700+ tasks, collected with 13 robots over 17 months
• Modalities: RGB, natural language instructions, action logs
• Tasks/Embodiments: Picking, placing, opening/closing drawers, moving items in and out of containers; single embodiment (mobile manipulator arm)
• License: Apache 2.0 (code); CC-BY 4.0 (data, released as part of Open X-Embodiment)
• Link: robotics-transformer1.github.io

5. LIBERO

LIBERO is built for a specific question that raw scale doesn't answer: how well does a policy transfer and retain knowledge across a sequence of tasks. Its 130 tasks are split into four suites (LIBERO-Spatial, LIBERO-Object, LIBERO-Goal, and LIBERO-100, itself split into LIBERO-90 for pretraining and LIBERO-10 for lifelong-learning evaluation) that each isolate a different kind of distribution shift: object identity, spatial layout, or task goal. That makes it one of the better benchmarks for measuring knowledge transfer and catastrophic forgetting rather than raw task success, and it has become a standard evaluation suite for open VLA checkpoints like OpenVLA.

• Year: 2023
• Scale: 130 tasks across 4 task suites, high-quality human teleoperated demonstrations for every task
• Modalities: RGB, language instructions, proprioception, simulated ground-truth state
• Tasks/Embodiments: Tabletop manipulation with controlled shifts in object type, spatial arrangement, or goal; simulated Franka Panda arm (robosuite)
• License: MIT (code), CC BY 4.0 (dataset)
• Link: libero-project.github.io

6. RoboCasa

RoboCasa is the simulated-kitchen answer to the question "how do I get 100,000 demonstrations without 100,000 teleoperation sessions." Human operators recorded 50 demonstrations per atomic task (25 atomic plus 75 composite tasks, 100 total, across 120 kitchen scenes and roughly 2,500 unique 3D objects), and MimicGen then auto-generated the rest, expanding the atomic-task data out to 100,000 trajectories. That's a genuinely useful trick for bootstrapping scale, but it means the bulk of the dataset inherits MimicGen's generation assumptions rather than pure human demonstration quality, and MimicGen itself carries the NVIDIA Source Code License, which has commercial-use constraints worth reading before you build a product pipeline on it.

• Year: 2024
• Scale: 100 tasks (25 atomic, 75 composite) over 120 kitchen scenes and ~2,500 3D objects; 50 human demos per atomic task, scaled to 100,000 trajectories via MimicGen
• Modalities: RGB, language instructions, proprioception, simulated ground-truth state
• Tasks/Embodiments: Everyday kitchen manipulation (opening appliances, cooking-adjacent tasks, pick-and-place); simulated Franka Panda mobile manipulator (robosuite)
• License: Apache 2.0 (code), CC-BY 4.0 (dataset); MimicGen generation pipeline under NVIDIA Source Code License
• Link: robocasa.ai

7. RoboMimic

RoboMimic is less a dataset than a benchmark suite, and that's exactly its value. It standardizes five manipulation tasks, Lift, Can, Square, Transport, and Tool Hang, across both simulation and matched real-world setups, each with proficient-human (PH) and mixed-human (MH) demonstration variants so you can study how operator skill level affects imitation learning. If you're comparing offline RL or imitation learning algorithms rather than pretraining a foundation policy, this is still one of the cleanest apples-to-apples benchmarks available, precisely because the task set is small and tightly controlled rather than broad.

• Year: 2021 (CoRL), actively maintained through 2025
• Scale: 5 core tasks, each with proficient-human and mixed-human demonstration sets (9 datasets total in the original release)
• Modalities: RGB, proprioception, simulated and real-world state, action logs (HDF5 format)
• Tasks/Embodiments: Lift, Can, Square, Transport, Tool Hang; simulated and real Franka Panda arm (robosuite)
• License: MIT
• Link: robomimic.github.io

8. CALVIN

CALVIN is purpose-built for long-horizon, language-conditioned manipulation, which most of the datasets above don't test directly. It provides four environment splits (A, B, C, D), each with roughly 6 hours of unsegmented teleoperated play data, about 2.4 million interaction steps in total across 34 tasks, but only around 1% of it (roughly 24,000 sequences) carries a language annotation. That ratio is the point: CALVIN is designed to evaluate whether a policy can chain unseen combinations of language instructions into long-horizon behavior, not to serve as a bulk source of labeled demonstrations. If your goal is testing instruction-following and generalization to novel language rather than pretraining scale, this is one of the more rigorous options.

• Year: 2022
• Scale: 4 environment splits (A/B/C/D), ~6 hours of unsegmented teleoperated play data per split (~2.4M interaction steps total), ~24,000 language-annotated sequences (~1%)
• Modalities: RGB, depth, proprioception, unconstrained natural language instructions
• Tasks/Embodiments: 34 long-horizon manipulation tasks chained via language; simulated 7-DoF Franka Panda arm
• License: MIT
• Link: github.com/mees/calvin

9. RH20T

RH20T is the dataset to look at if modality richness matters more to you than raw visual diversity. Collected by Shanghai Jiao Tong University, it pairs over 110,000 real robot sequences with a matched set of human demonstration sequences (50M+ images total) across 147 tasks, and each sequence includes RGB, depth, and binocular IR video alongside 6-DoF wrist force-torque readings and, for some sequences, audio and fingertip tactile data. That multi-modal depth makes it a strong choice for one-shot imitation and tactile-aware manipulation research, though be aware the human-demonstration subset includes volunteers' faces and voices, and the license situation is mixed: the RH20T-C scenes (0001-0005) are CC BY-SA 4.0, while the RH20T-NC scenes (0006-0010) are CC BY-NC 4.0 and block commercial use.

• Year: 2023
• Scale: 110,000+ robot sequences plus a matched set of human demonstration sequences, 50M+ images total
• Modalities: RGB, depth, binocular IR, 6-DoF force-torque, audio, fingertip tactile (select sequences), language descriptions
• Tasks/Embodiments: 147 diverse manipulation skills across multiple robot arms and grippers
• License: CC BY-SA 4.0 (RH20T-C, scenes 0001-0005); CC BY-NC 4.0 (RH20T-NC, scenes 0006-0010)
• Link: rh20t.github.io

10. AgiBot World

AgiBot World is the largest and newest entry on this list, and it's built specifically around humanoid and mobile dual-arm platforms rather than fixed tabletop arms. Its initial Alpha release (December 2024) shipped about 92,000 trajectories; the Beta release that followed a few months later scaled that up past 1 million trajectories from 100 robots across more than 100 real-world scenarios in five target domains (manipulation, tool use, and multi-robot collaboration among them), collected with hardware that includes array-based visual-tactile sensors and 6-DoF dexterous hands. It's the dataset to reach for if your target embodiment looks like a humanoid or mobile bimanual platform rather than a Franka arm bolted to a table, though the CC BY-NC-SA 4.0 license means you'll need a separate agreement for commercial pretraining.

• Year: 2024 (Alpha, ~92K trajectories), 2025 (Beta, 1M+ trajectories), expanded through 2026
• Scale: 1M+ trajectories (Beta release) from 100 robots across 100+ scenarios, 217 tasks in 5 domains
• Modalities: RGB, array-based visual-tactile sensing, proprioception, language instructions
• Tasks/Embodiments: Fine-grained manipulation, tool use, multi-robot collaboration; mobile dual-arm humanoid robots with 6-DoF hands
• License: CC BY-NC-SA 4.0
• Link: agibot-world.com

11. Ego4D

Ego4D isn't a robot dataset at all, it's massive-scale egocentric human video, but it earns its spot here because human-video pretraining has become a real strategy for bootstrapping manipulation and VLA policies before any robot data enters the picture. Meta and an international consortium collected more than 3,670 hours of daily-life first-person video from 931 camera wearers across 74 locations in 9 countries, some of it paired with audio, 3D scene meshes, and eye gaze. For manipulation specifically, it's most useful as a pretraining source for visual representations and hand-object interaction priors, not as a source of robot action labels, since there's no robot in the loop and no action supervision to imitate directly.

• Year: 2021 (paper), 2022 (CVPR)
• Scale: 3,670+ hours of egocentric video, 931 camera wearers, 74 locations, 9 countries
• Modalities: First-person RGB video, audio (partial), 3D environment meshes (partial), eye gaze (partial), stereo (partial)
• Tasks/Embodiments: Unscripted daily-life activity; human embodiment (egocentric camera), used as a pretraining source rather than direct robot supervision
• License: Custom license agreement (application and approval required, non-commercial research emphasis)
• Link: ego4d-data.org

12. FMB (Functional Manipulation Benchmark)

FMB takes a different design philosophy from the pretraining-scale datasets above: instead of maximizing trajectory count, it deliberately scopes tasks to be narrow enough that a modest dataset can still drive meaningful progress tracking, while composing those narrow skills into long-horizon, functionally relevant behaviors. Built at UC Berkeley on a real Franka arm, its 22,550 expert demonstrations cover both individual manipulation primitives and the multi-stage tasks built by chaining them, which makes it a good fit if you want a benchmark for skill composition rather than another massive pretraining corpus.

• Year: 2024
• Scale: 22,550 expert demonstrations
• Modalities: RGB, proprioception, action logs
• Tasks/Embodiments: Composable manipulation primitives and multi-stage functional assembly tasks; real Franka Panda arm
• License: Open source (see repository for terms)
• Link: functional-manipulation-benchmark.github.io

Putting The Framework To Work

Run any dataset you're considering through the same three questions: how many embodiments and tasks does it actually span, how clean and well-labeled are the demonstrations, and does the license match what you're building. A pretraining corpus wants breadth (Open X-Embodiment, RT-1, AgiBot World). A fine-tuning or evaluation target wants consistency and a matched real-world setup (BridgeData V2, DROID, LIBERO). A benchmark for algorithm comparison wants a small, tightly controlled task set (RoboMimic, CALVIN, FMB). Very few teams need all twelve, but most manipulation and VLA projects will end up drawing from at least two or three of these tiers.

Training Data Is Half The Loop

Good training data gets a policy to work in the lab. It doesn't tell you whether that policy will hold up once the objects, lighting, or gripper change, and that is a separate problem from data collection. That's why we think about training data and evaluation as one pipeline rather than two separate projects.

Manifold is our open-source policy evaluation layer: drop in a policy trained on any of the datasets above, run it at scale (1,000+ rollouts per run) across LIBERO, RoboCasa, Isaac Lab, MuJoCo, Genesis, or your own scenarios, and see success rate plus per-task and per-step failure modes as the run progresses, not in a report afterward. Every run gets a reproducible, citable URI, so regressions and improvements are visible run over run rather than buried in a spreadsheet.

On the data side, Stardust and our robotics vertical generate the photorealistic, pixel-perfect labeled synthetic data (segmentation, depth, 6-DoF pose, and grasp annotations) that trains the perception underneath a manipulation policy, with randomized objects, materials, clutter, and lighting to cover the long tail that fixed datasets like the ones above can't fully capture. Training data gets you most of the way. Evaluation is how you find out what's actually left.

Also Read

• Building The Data Engine To Power Physical AI
• Top Datasets For Maritime AI, USV Perception & Obstacle Detection