How Much Is Robot Deployment Data Worth?
On robot deployment flywheels and why I am skeptical of the data they produce
There’s a recent trend towards emphasizing the importance of real-world deployments in robotics. It’s common wisdom. On one level, this makes sense: real data is scarce, and real deployments are, in the end, what drive revenue in robotics.
I want to make a counter-argument: deployments in robotics are important, but most robotics deployments are fundamentally not scalable, requiring large, specialized support teams and large amounts of resources to maintain, and the data that these deployments produce is almost certainly less valuable than you think it is.
Only a few classes of robotics deployments can truly be scaled nearly without end — problems like self-driving. That’s because the “single task” is so valuable that collecting endless in-domain data actually does work very well.
And even in this ideal case for robot deployments, data scaling has not thus far been enough! Tesla has millions of miles of driving data, and that data on its own did not result in a working, totally self driving robotaxi — instead, they are deploying robotaxis with drivers in the loop in cities like Austin, collecting what amounts to intervention data of how to improve model performance.
The fundamental reason for this is because to deploy robots, they need to reach high success rates on a particular task; the easiest way to reach high success rates is to constrain the problem; once the problem is highly constrained, it no longer helps you learn anything useful for the next problem because robotics is a wildly high dimensional problem space.
So, let’s talk about robot tasks, deployments, and what’s missing.
The Distribution of Robot Tasks
We tend to see a few tasks showing up again and again in modern robotics:
Rigid or semi-rigid pick and place (like this engine cover placement task from Xiaomi)
Package reorientation (Figure and Robotera), covered in my blog post on recent scaling announcements for real-world robotics
Box packing (Physical Intelligence and Ultra)
Clothes folding (HuggingFace, Sunday, Weave, and so many more)
The reason is because, to do useful work in most real-world tasks, robots need to achieve success rates in the 97%-99.9% range or higher, and all of these tasks can be saturated during their training process, massive amounts of data collected in constrained settings, so that high-quality data can be collected and high-quality policies can be trained for them.
Achieving these high success rates takes a lot of time, too. Take this task from Xiaomi, inserting a self-tapping nut:
They initially launched with a 90.2% success rate, and got this task up to a 98% success rate in about 4 months. That’s impressively good, and probably borderline production quality, assuming they can fix any issues with retries. But it demonstrates how slow it can be to improve robot policies up to a useful level.
Part of the problem is simple; in a single task, as a policy improves, it gets harder and harder to find the rare useful failures, from which you can collect more data in order to improve the policy. And modern robot learning isn’t terribly efficient; you’ll often need a lot of data in a particular corner case to resolve it.
Why Do We Need So Much Robot Data?
Once you get deeply involved in a single robotics vertical, you start to learn all the hard things associated with that vertical. This is the implicit knowledge workers in that domain have, which is not readily captured in a blog post or a video or even ChatGPT. How exactly should you physically interact with something? This knowledge, presumably, comes from a robotics deployment.
The problem, as noted by Sumedh Sontakke here, is that the “long tail” of robotics problems is very “long and fat,” meaning that you need to solve many, many distinct problems which each show up quite frequently to get utility out of your “general purpose” robot.
Because there are so many robotics tasks out there, and because the space of robot control is so naturally high dimensional, it’s important that we try to cover the space of potential interactions fairly well.
Robotics is a fairly high-dimensional problem. Let’s consider the minimal amount of free dimensions needed to represent a dual-armed mobile manipulator interacting with the world. The robot’s torso can be represented as an SE(2) position (x, y, theta), and a torso height z. In addition, there will be two 6-DOF poses representing (roll, pitch, yaw, x, y, z) for each of the two end effectors, and two gripper states (left, right). So, 18 degrees of freedom for the robot. In a manipulation problem with N objects, we would need to add 6 degrees of freedom per object.
Now imagine evenly sampling just 10 potential values for each of these 18+6N states. We might want to consider all the potential relationships between objects. However, this explodes in complexity. This is something that we often refer to as the curse of dimensionality. For different values of N:
At N=1, we have a manageable 10^24 samples
At N=5 = 10^48, we are approaching the number of atoms on Earth (~10^50)
At N=10 objects = 10^78 samples, we are approaching the number of atoms in the known universe
The useful regions of this space are relatively limited, but naive exploration is clearly not feasible. In the past, many solutions have been proposed: robot motion planning uses learned heuristics, sampling-based planning, and priors from supervised learning to make the problem more tractable.
In robotics, then, the goal is to explore this massive combinatorial space of possible world states, and as much as possible, gain coverage of all the regions of that state space which might potentially matter to the robot, at sufficient resolution to solve the problem in all cases.
Because the problem is so huge, exhaustive exploration will always be impossible, but likewise because the problem is so huge, it can be easy to miss some crucial manifold which will show up at test time. And this is before even accounting for all of the many different things that will need to be done within this huge configuration space!
The Search for Novelty
So the problem becomes one of operations: how do we explore this huge space, while finding all of the niche little regions of that space that might be relevant to our policy?
Deployment does not usually solve this problem. Deployed robots will very rarely see new situations which we care about, as they must already be performing at high success rates; when they do, they provide a very inefficient way of collecting new demonstrations as you never fully explore a failure mode.
Thus, a goal of robot data collection is to constantly find and explore new, relevant tasks — identifying a broader subspace of robot worlds which we can characterize (through demonstration data) in order to achieve more general and robust policy performance.
Finding these new tasks and examples is an art; finding more high quality task data is of high importance for robot learning, and there are few “natural” flywheels where deployment data can actually provide this.
But there are domains which do introduce a steady stream of novelty for a proper flywheel; the most obvious of these is the home. Companies like Weave Robotics and 1X are able to teleoperate robots in homes remotely when they cannot perform a task, meaning they’ll be able to solve the data problem. But of course, on-demand intervention like this is still an excruciating, difficult way of collecting the right data!
Kyle Vedder of Physical Intelligence wrote a (much shorter) blog post here which makes similar points:
Industrial Robotics Tasks
Where does this leave us? Many interesting industrial tasks can be tightly constrained, so that simple, smaller robot policies can be developed to deploy them. Take this plug insertion task by Sanctuary AI, for example.
In these tasks, the subspace of the vast, high-dimensional world that needs to be explored is extremely small, and can be fully saturated during data collection, so long as the environment is tightly constrained. These tasks, then, are perfectly suited for automation on an assembly line already.
This, notably, completely avoids a strict need for learning from deployment, or any notion of a “data flywheel,” by simply choosing a task that is straightforward to train a model for.
Final Thoughts
So, to summarize:
Most robot deployments don’t introduce new novelty at a sufficient pace to improve your model’s general capabilities.
Most robot deployments don’t generate new failure and correction data fast enough to improve models at a reasonable rate.
What are we to do? Well, we have two options:
Find an environment where you actually can experience a wide range of tasks, all the time, in order to improve basic model capabilities.
Have a cracked ops team who can go out and make up new tasks to explore on their own.
Fortunately, I think all of this will be solved via the magic of scale. There are many, many more robots in the world now; there will be even more next year, and even more the year after that. So long as these robots are doing interesting things, and so long as the data they generate can be used and shared, we will be able to mine this data more and more to create general-purpose robots, even if each individual effort is not so useful.
But it might take a while to get there.
Great article! Given the need for novelty, what are your thoughts on actively training policies designed to target noisy/difficult distributions as supplementary data generators for the main policy (ex: seeking variance-seeking behavior)?
Interesting article...