[TB3] Hugging Face LeRobot with TurtleBot3 & OpenManipulator-X

TurtleBot
1

:fire: Exciting news! We’re thrilled to showcase our first demonstration of LeRobot​:hugs: Imitation Learning with TurtleBot3 & OpenManipulator-X!

Leveraging the power of imitation learning, we’ve successfully implemented a “Pick & Place” task by adapting and utilizing code available from LeRobot’s GitHub repository. (Note: The leader robot shown in the video is currently a temporary design for demonstration purposes.)

:rocket: We are particularly enthusiastic about highlighting Hugging Face’s LeRobot project, a groundbreaking initiative in the field of imitation learning and robotics. We strongly recommend exploring LeRobot’s impressive capabilities and actively engaging with this vibrant community!

:handshake: We also recognize the fantastic synergy between the ROS community and Hugging Face’s LeRobot community. These platforms complement each other exceptionally well, fostering significant advancements in robotics software development.

:clapper_board: This is just the beginning! We’re actively working on more comprehensive demonstrations and detailed content, so stay tuned—there’s much more to come!

For further information, detailed instructions, and resources, please check out the following:

:books: Manual:

:inbox_tray: GitHub Repositories:

:speech_balloon: Community:

#huggingface #lerobot #imitation_learning #physical_ai turtlebot3 #openmanipulator robotics #AIinnovation

5 Likes
2

The video seems to show the Koch v1.1 robot from the tutorial ON a turtlebot (not lerobot on the turtle robot), and considering the manual you link + the video, it doesnt seem like you are using ROS2 nor a turtlebot to record the demonstrations and then deploy the policy.

I point this out since I’ve been looking around for an open source solution to record demonstrations and deploy lerobot policies from ROS2, but it seems like there is not any atm (from my somewhat shallow search). I wondered if maybe I was mistaken with my initial comment, and you already implemented such thing.

For the moment I’ve found a yt video with isaac to lerobot dataset with ROS2, but it is kinda hardcoded for their application and the code is very poorly written, meaning I should probably just start from zero. The other repo I’ve found is dora-lerobot, but all the ROS2 functionality is already wrapped in the dora framework, which doesnt make it very useful for me.

I intend not to reinvent the wheel here, considering both lerobot and ROS2 are very popular, I am kind of surprised such a ROS package doesnt exist. If you already solved this, I would love to hear how/with what package.

1 Like
3

The video above showcases our first demonstration with HF LeRobot. In the future, we plan to open up the interface between ROS 2 and LeRobot and make it easier for users across platforms.

1 Like
4

I think the most appropriate thing is
to have an option to integrate lerobot
which will know to publish to a specific topic in ROS2

this way it will keep the type of action and the type of robot under your control

5

I have a PR open to acquire image frames over ROS 2. I could use some feedback and am happy to iterate on the API Add support for cameras over ROS 2 by Yadunund · Pull Request #866 · huggingface/lerobot · GitHub

2 Likes
6

Hi @Yadunund

Thank you for your excellent contribution! Your content looks very promising, and it would be great to see it merged soon!

On our side, we’re currently developing a ROS 2 package named data_collector within our physical_ai_tools repository. Once we reach a stable milestone, we plan to share it along with an accompanying demonstration video.

4 Likes