Towards Home Robots
What this blog is and why I'm writing it
In the near future - say 2030 - will have a variety of robots in our homes. They will help out around the house. They’ll do chores, they’ll entertain us, they’ll act as companions.
To some extend this really isn’t an ambitious statement. I have a few robots in my home already: a couple robot vacuums and an Amazon Astro, all of which I get a great deal of value (or at least entertainment) out of.
But these existing robots don’t have a lot of complex ways of interacting with the world. They’re honestly pretty bad at it. The roomba-knockoffs trundle around, bumping into furnitive, cleaning your floor. The Astro is an amazing piece of hardware, but all it can do is take pictures and show you images.
The real question, then, is how we make robots into useful, general purpose assistants - ones that can perform interesting multi-step tasks.
1. Why Now
I’m increasingly convinced that we can build robots that are low-cost, useful, and safe in-home assistants. See our work last year on HomeRobot, where we looked at a challenge called Open-Vocabulary Mobile Manipulation, in which a robot must move any object from any start receptacle to a goal receptacle, all of which are given as language (e.g., “move the cup from the sofa to the kitchen table”).
Above: “Move the water bottle to the sink,” from OK Robot. Also found on my website. Things are much faster than this now, but the hard part remains the same: perceiving and interacting with the world!
There has been a lot more interesting work in this direction, including Go To Anything, OK Robot, and ConceptGraphs. The question to me, clearly, is:
How can we make robots that can act intelligently in any environment while performing manipulation tasks, with minimal to no training specifically for those environments?
What allows robots (and AI systems in general) to perform complex, multi-step reasoning, and what is missing from those systems today that prevents them from reliably performing this sort of reasoning?
Above: One more video from my house, this time of “move the cactus plushie to the baby seat.” I like this task because the succulent/cactus toy is such a strange one, and the 4moms robotic baby seat is an unusual thing to see as well. Real homes and real environments are filled with unique objects and rare concepts; this is part of what makes them so hard to automate.
Again, we’re much faster now, as you’ll see soon!
And, recently, I joined Hello Robot! Hello’s goal is to build safe, lightweight, low cost home robots, and I’m pretty excited to be a part of that.
For years now, I’ve been a passionate supporter of their vision of affordable, useful robots that can help people out with their day-to-day lives. I’ve previously worked at FAIR, part of Meta, and at NVIDIA, in both cases leading research on robot learning and planning for manipulation, with a particular focus on generalization, so that robot skills will work outside of the specific context they were trained on.
My focus on these problems has always been motivated by my belief that we can make robots into the exact sort of useful, general-purpose assistants that Hello Robot envisions. Now seems like the time to make that a reality. Excited for what the next year will bring!
2. What to expect
On this blog I plan to write about AI and robotics, both industry and academic research. I write quite a bit on this already on Twitter/X, but for a while I have wanted to start posting my thoughts somewhere a bit less ephemeral.
To be clear what I expect to post here is:
Random paper summaries and notes
More detailed writeups on the state of the field once a week
The blog’s name is a TV and movie trope about finding out that the monster-of-the-week can actually think and reason. I expect to see some of those surprising moments as robots and AI systems become more and more intelligent! But of course in a positive way.
To be clear I’m not planning on doing “paid” posts or anything. You’ll find most of this content for free on Twitter as well if that’s your preferred medium.
Thanks for reading my Substack! Subscribe for free to receive new posts and support my work.