Synthetic Data vs Real Data: Building Better Robotics Training Pipelines

We love robots. We enjoy the idea of them making our lives easier by helping with house chores, driving us to work, or working the factories so we don’t have to deal with manual labor. Unfortunately, training a robot to perform a task is one challenge while getting it to repeat the same task when something unexpected happens in another. Despite recent advancements of AI robotics, Pollen Robotics and Hugging Face’s Microducks struggled with following certain instructions, prompting NVIDIA’s Jensen Huang to sternly reprimand the cute little machines. 

When it comes to robotics training, an early-stage team worries about getting enough demonstrations to train a useful policy. A deployed fleet has a different headache: finding the handful of situations that still make the robot fail. Once those failures are found, the bottleneck moves again. Engineers need to reproduce them, generate variations, retrain, and prove that the new policy actually improved. That progression matters more than the familiar debate over real-world versus synthetic data.

• Choose data sources around a measured failure or coverage gap.
• Use simulation to expand controlled experiments; use physical data to test relevance and transfer.
• Judge datasets by usable coverage and downstream performance, alongside volume.
• Define annotation, metadata, and acceptance criteria before scaling production.
• Keep evaluation data independent and check for regressions after each update. 

There is an appealing simplicity to simulation. Give a robot a virtual environment, let it run thousands or millions of trials, and let the computers do the heavy lifting. The human reading this article doesn’t have to worry about cleaning up any mess or paying the teleoperator pilot huge fees for a few months of robot training. 

NVIDIA’s Jim Fan has made exactly this case. As he puts it, “Real-world teleoperation data scales linearly with human time” while simulated experience scales with compute. His thought experiment is deliberately extreme: run 100,000 copies of a simulation on a GPU and give a policy years of simulated experience in a fraction of the wall-clock time. Plus, it’s much cheaper. 

Meanwhile, the argument for real-world data is equally as compelling. Jan Erik Solem, CEO of Staer, put it more bluntly: “Real-world data from deployed machines beats everything if you can get it at scale.” If it ain’t broke, why fix it? 

While both statements have their own merit, neither describes the whole robotics data problem. The real issue is that the bottleneck changes as autonomy increases.

First, you need enough experience to get a system working. Then you need enough variation to stop it from overfitting to a narrow environment. Then you need the unusual cases that expose its weaknesses. Finally, you need a feedback loop fast enough to turn those weaknesses into measurable improvements. That creates four very different data problems inside one robotics program.

Real-world data is captured from physical systems and their environments. This includes teleoperation sessions and egocentric video (which we covered in one of our previous articles) and also robot trajectories, sensor recordings, fleet logs, interventions, and human demonstrations. 

Synthetic data is generated inside a simulated or computationally constructed environment. It can come from physics-based simulation, procedural generation, generative world models, rendering pipelines, or augmentation. It’s no wonder why an NVIDIA spokesperson would champion this modality so wholeheartedly since you need a lot of expensive GPUs to produce a virtual environment for your robot to learn from. 

Nevertheless, simulation has an enormous practical advantage: control. You know the object pose. You know the collision state. You know the segmentation mask. You can reset a failed task and run it again under slightly different conditions. 

That is why synthetic data use cases extend far beyond simple image augmentation. Simulation supports perception, manipulation, navigation, planning, benchmarking, and reinforcement learning robotics at a scale that would be expensive or unsafe on physical hardware. Reviews of sim-to-real research consistently point to simulation's scalability and efficiency as major advantages.

While all that is cool, real data has a different advantage: nothing in the real world really goes according to plan.

A simulator only reproduces the assumptions encoded in it. Physical systems introduce sensor noise, imperfect calibration, unexpected contact, environmental variation, and behaviors that were never in the specification. The sim-to-real gap is therefore more than a graphics problem; it can involve dynamics, sensing, actuation, timing, and environmental variation.

At first, simulation is hard to beat. A new robot may have only a few physical units. Hardware time is expensive. Experiments take place in real time. Some actions can damage equipment. Others may simply be too risky to explore repeatedly.

This is where robot simulation software earns its keep. Teams can train policies, test control strategies, generate perception examples, and run large batches of experiments before putting the policy on a physical robot.

It is also where synthetic data generation tools are particularly useful. A simulator can change object positions, lighting, camera placement, friction, obstacle layouts, or robot starting states without requiring a human to reset the physical environment every few minutes.

For computer vision tasks, simulation can generate synthetic training data with precise labels at a fraction of the cost of manual annotation. What teams tend to underestimate at this stage is how narrow the virtual world can be. A policy can look excellent while learning a very specific version of the task.

The robot can now perform the task. That’s great but the next question is whether it can perform it when the conditions change. This is where real demonstrations become much more valuable.

Teleoperation gives a model examples of how a person completes the task. Egocentric video captures how humans interact with objects and environments. Those demonstrations can support behavior cloning, where a policy learns to associate observations with actions taken by an expert.

For manipulation, the difference can be enormous. A robot manipulation dataset containing successful demonstrations across different objects, poses, speeds, and approaches gives a policy more useful coverage than thousands of near-identical repetitions.

This becomes especially important in dexterous manipulation, where a successful action may depend on a sequence of subtle movements rather than a single target pose.

Synthetic data still has a role. Once a team understands the task, simulation can vary the known conditions and expand the coverage. What tends to get underestimated here is the difference between learning a task and understanding its boundaries.

After deployment, the question changes again. A physical robot doesn’t care how convincing the training distribution looked in a lab. It encounters the wrong object in the wrong place, a camera partially blocked by a worker, a reflective surface, a crowded aisle, a damaged package, or a situation nobody put into the original test plan.

This is where real-world robotics training data becomes disproportionately valuable. A failure contains information. It tells the team something about the environment, the robot, or the policy that was missing from the development loop. The mistake is to treat that failure as a one-off bug.

A better pipeline turns it into a family of training examples. Once a real failure has been captured, simulation can reproduce the underlying conditions and generate variations around them. The real example establishes relevance; synthetic generation provides scale.

That changes the role of synthetic data. Instead of generating millions of generic examples and hoping the model becomes robust, teams can use synthetic data for machine learning to amplify situations that have already demonstrated value.

Bessemer Venture Partners makes a related argument in its 2026 analysis of world models, noting that robotics lacks the equivalent of the internet-scale corpus that powered language models and that teleoperation alone cannot provide it. The real bottleneck is increasingly coverage of failure, not raw sample count.

The most mature robotics systems turn deployment into a learning loop. The robot operates. Evaluation identifies a weakness. The team finds the underlying examples. New robotics data collection targets the missing coverage. Annotation and QA turn the raw captures into usable training data. Simulation expands the scenario. The policy is retrained. Then the new version is tested against a held-out real-world evaluation set.

The faster that loop runs, the faster the system improves. In a joint study between Stanford University and affiliations from Toyota Research Institute, a mobile manipulator collected data while scanning 2,103 library shelves; fine-tuning on the collected data improved book-identification performance from 32.0% to 71.8%, while also improving performance on a related multilingual OCR task. The result illustrates deployment-driven perception improvement and does not establish that the same process improves every component of a robotics system. 

This is a fundamentally different metric for a data program.

The question is no longer “How many hours did we collect?”

It becomes “How quickly can we turn a real failure into the next validated improvement?”

For companies building robotics foundation models, that feedback velocity may become one of the most important competitive advantages in the stack.

Going back to the experts, Jim Fan's argument makes sense when the bottleneck is physical trial time. Simulation turns robot experience into something that can scale with computing performance that’s separate from real world constraints like teleop pilot availability and budget. 

However, Solem's argument makes sense once the robot is operating in the physical world. Real deployment exposes conditions the development team did not anticipate and gives the model access to behavior that simulation may never have represented correctly.

Bessemer's position lands somewhere in the middle: teleoperation is valuable, but it cannot by itself create an internet-scale experience corpus. The firm sees world models and video as possible ways to expand that pool. These positions map surprisingly cleanly onto the maturity curve: 

• Simulation attacks the volume problem.
• Demonstrations attack the coverage problem.
• Deployment data exposes the failure problem.
• A closed-loop data and evaluation system attacks the feedback problem.

The disagreement is mostly about where to put your next dollar.

Synthetic data often comes with labels generated by the simulator. Synthetic data validation determines whether those labels and scenarios reflect conditions relevant to the physical robot.

Real-world data brings different challenges. In autonomous driving, LiDAR annotation requires accurate 3D bounding boxes, object classifications, and consistent tracking across frames. Occlusion, sparse point clouds, and misaligned sensor streams make these decisions difficult.

That said, more annotators won’t make the problem just go away. Teams need task-specific guidelines, expert review, and QA that checks the actual geometry and tracking instead of just agreement between annotators.

Chemin provides data collection, annotation, and dataset development services for AI training and evaluation, including LiDAR annotation for autonomous driving. Collection captures the required scenarios; annotation turns sensor recordings into structured labels; dataset development prepares the data for training and evaluation. Across all three, their shared priority is to ensure the data accurately represents the task the model needs to learn.

The next investment depends on what is holding the robot back. If it lacks experience, expand simulation. If it struggles outside familiar conditions, broaden demonstrations. If deployment exposes unpredictable failures, collect targeted real-world data. If progress has slowed despite a large dataset, examine annotation, QA, and evaluation. Keep one real-world evaluation set separate from training so that each new model is tested against situations it has not already seen.

The goal is to connect collection, annotation, simulation, and evaluation into one feedback loop. Real-world evidence identifies gaps, structured labels make them useful, and simulation expands the scenarios available for training. Stronger robotics datasets emerge from each cycle and the team gets faster at turning a discovered weakness into a verified improvement.