Introducing Stretch AI

Build intelligent, AI-enabled mobile robot behaviors

Dec 03, 2024

I'd like to introduce what I've been working on the last few months at Hello Robot: Stretch AI, a set of open-source tools for language-guided autonomy, exploration, navigation, and learning from demonstration. The goal is to allow researchers and developers to quickly build and deploy AI-enabled robot applications.

Stretch AI is designed so that you can easily get started and try it out on your robot. It supports multiple LLMs, from open-source models like Qwen to OpenAI. You can even do voice control of the robot, talk to it, and have it clean up your floor!

LLM Agent

The whole idea is that you can clone the repo, and then, on your GPU laptop or desktop, run a command like this:

python -m stretch.app.ai_pickup --use_llm --use_voice

And you’ll be able to run that demo I showed above! The robot will move around, building a rich 3d map of the world as it does. You can talk to it, or type instructions, to get the robot to pick and place objects in different locations.

A 3d map of a small part of my house collected during experiments. Red areas are obstacles that have been detected and will be avoided by the robot agent.

The visualizations we use are powered by Rerun, which is just an incredible set of visualization tools for researchers and I recommend taking a look.

Learning from Demonstration

In addition, we provide dexterous teleop and data collection, integrated with Huggingface's LeRobot software. This lets you collect 6dof demonstrations on a mobile manipulator for all kinds of dexterous tasks, using our affordable dex teleop setup.

Then, using our AI tools, you can deploy these learned skills to complete long-horizon tasks on your own mobile robot, in your own home or other environment.

Navigation, manipulation, and learned skills all coordinated by open-source AI!

Setup and System Architecture

Stretch (the robot) doesn't have a powerful GPU, so Stretch AI offers tools for seamlessly controlling it over a wireless network. We also provide docker images both for on-robot control (including ROS2) and for a GPU-enabled desktop, to make development easy.

Essentially, on the robot, you will run something like:

./scripts/run_stretch_ai_ros2_bridge_server.sh

This will download and run a Stretch docker image, containing a ROS2 environment and a server which provides a ZMQ-enabled network connection to let you control the robot from any other device.

Then running robot commands is simple:

from stretch.agent import RobotClient
robot = RobotClient(robot_ip="192.168.1.15")  # Replace with your robot's IP
# On future connection attempts, the IP address can be left blank

# Turn head towards robot's hand
robot.move_to_manip_posture()

# Move forward 0.1 along robot x axis in maniplation mode, and move arm to 0.5 meter height
robot.arm_to([0.1, 0.5, 0, 0, 0, 0])

# Turn head towards robot's base and switch base to navigation mode
# In navigation mode, we can stream velocity commands to the base for smooth motions, and base
# rotations are enabled
robot.move_to_nav_posture()

# Move the robot back to origin
# move_base_to() is only allowed in navigation mode
robot.move_base_to([0, 0, 0])

# Move the robot 0.5m forward
robot.move_base_to([0.5, 0, 0], relative=True)

# Rotate the robot 90 degrees to the left
robot.move_base_to([0, 0, 3.14159/2], relative=True)

# And to the right
robot.move_base_to([0, 0, -3.14159/2], relative=True)

Of course, we have plenty of apps you can run, and you can go through the LLM agent documentation to explore all the options there and try out different language models.

Final Notes

Our code also allows you to develop and experiment with exciting, cutting-edge research, like Dynamem, which allows for open-vocabulary mobile manipulation in dynamic scenes.

This is research code — not a product. It won’t always work. It probably has bugs. I hope that people find it useful, and even that people are able to extend it to other robots (though it’s definitely intended for Stretch — and Stretch is the best home mobile manipulator you can buy!).

On a personal note, this is something I've wanted to build since I left NVIDIA ~2.5 years ago -- I think it's so important to have good, open, easy-to-use tools for mobile robotics research in AI. I'm looking forward to seeing what people can build with it!