Do you consider the MolmoAct project and Action Reasoning Model (https://allenai.org/blog/molmoact) just another VLA or do you think they are doing something different/novel? Thanks.
Great question. We actually just recorded a RoboPapers episode on MolmoAct last night! And i do think they're making some important innovations on the basic formula, especially pioneering what "reasoning" would look like in the context of robotics. I want to do a follow up post (maybe next week?) on some innovations and changes to the core formula, which so far seems pretty fixed across all the major players!
Yeah I really find MolmoAct fascinating. The core question it poses of "how do we build reasoning models for 3d/real world tasks" will I think be one of the most important ones in robotics, and I love that part of how they do that is through predicting spatial/depth info.
> This particular problem would go away if everyone had just kept contributing to Open-X Embodiment like they were supposed to. But data is expensive, it’s the new coding, and in a very real way it’s your “moat”: it’s unreasonable to expect private companies to share large amounts of data freely.
I’m hoping that over time, LeRobot catches on in the research and hobbyist community and eventually all we’ll have to do is go through and aggregate it. It already seems to have some sway - IK SmolVLA used LeRobot data and iirc GR00T trained on SO100 data.
Lovely piece! (You can probably tell what I'm trying to learn at any given moment based on what post of yours I comment on at any one point haha).
> An extremely clear example of this structure is pi 0.5 from Physical Intelligence: they have their “System 2” output a set of discrete tokens, learned via the FAST action tokenizer, which are then inputs into the flow matching System 1 architecture.
I think this is inaccurate or slightly misphrased? If I'm reading the pi 0.5 paper right, the "System 2" outputs a set of discrete text tokens which are inputs into the flow matching System 1 architecture. But notably, those outputs which go into the action expert are not discrete action tokens. (In fact, the FAST action tokens are masked from the action-expert so that it learns to generate action tokens on its own.)
Great post!
Do you consider the MolmoAct project and Action Reasoning Model (https://allenai.org/blog/molmoact) just another VLA or do you think they are doing something different/novel? Thanks.
Great question. We actually just recorded a RoboPapers episode on MolmoAct last night! And i do think they're making some important innovations on the basic formula, especially pioneering what "reasoning" would look like in the context of robotics. I want to do a follow up post (maybe next week?) on some innovations and changes to the core formula, which so far seems pretty fixed across all the major players!
🙏🏼
I think MolmoAct is another paradigm for VLA, it takes in Vision, Language and generate Action. But the structure is very different, closer to ReKep
Yeah I really find MolmoAct fascinating. The core question it poses of "how do we build reasoning models for 3d/real world tasks" will I think be one of the most important ones in robotics, and I love that part of how they do that is through predicting spatial/depth info.
> This particular problem would go away if everyone had just kept contributing to Open-X Embodiment like they were supposed to. But data is expensive, it’s the new coding, and in a very real way it’s your “moat”: it’s unreasonable to expect private companies to share large amounts of data freely.
I’m hoping that over time, LeRobot catches on in the research and hobbyist community and eventually all we’ll have to do is go through and aggregate it. It already seems to have some sway - IK SmolVLA used LeRobot data and iirc GR00T trained on SO100 data.
Yeah I hope so too. I think LeRobot is a great project, though it too has its shortcomings.
Hi, Chris, great post. Very nice overview. 👍
Thank you!
Lovely piece! (You can probably tell what I'm trying to learn at any given moment based on what post of yours I comment on at any one point haha).
> An extremely clear example of this structure is pi 0.5 from Physical Intelligence: they have their “System 2” output a set of discrete tokens, learned via the FAST action tokenizer, which are then inputs into the flow matching System 1 architecture.
I think this is inaccurate or slightly misphrased? If I'm reading the pi 0.5 paper right, the "System 2" outputs a set of discrete text tokens which are inputs into the flow matching System 1 architecture. But notably, those outputs which go into the action expert are not discrete action tokens. (In fact, the FAST action tokens are masked from the action-expert so that it learns to generate action tokens on its own.)
I like your job sir