Embodied AI Community Group - Call of Interest

Next Generation ROS
, ,
1

Update: the group is launched: The launch and first meeting of Embodied AI Community Group!

Dear ROS Community!

Large Language Models, Vision Language Models and AI Agent architectures are recent breakthrough technologies that most of you are familiar with. There are many gaps in Generative AI for robotics such as in spatial and temporal reasoning, dealing with imperfect perception and actuation, performance, size, liability, and so on. However, there are also important strengths and applications that are already quite reasonable, such as in human-robot interaction, high-level planning, inspections, and summaries.

This creates interest in developing robots running an AI Agent, which shows in open-source activity in embodied AI being on the rise, with multiple new projects published or updated recently, including:

  • RAI - a novel generic AI Agent framework for ROS 2.

  • ros2ai – command line interface extension, especially for those learning ROS by @tomoyafujita

  • ROSA - to inspect, diagnose, understand, and operate robots by @RobRoyce

  • embodied-agents – to facilitate data collection and sharing for embodied AI researchers.

  • Llama_ros, whisper_ros, … – ROS 2 integrated llama.cpp model, listening to speech and generating responses.

  • DORA - streamline and simplify creation of AI based robot applications.

It makes sense to me to align these developments and foster the community around Gen AI for ROS. The benefits are many, from a platform to share experiences with this novel technology and learning from other projects and applications, to collaborating on common goals, new features, and integrations.

My idea would be to lead the group personally and to organize meetings once a month to start with. I would invite speakers to present their projects, share experiences and needs. Robotec.ai’s contributions would involve our ongoing efforts with RAI as well as simulation projects and assets to validate embodied AI agents.

Dear ROS community! I would appreciate your expression of interest in the Embodied AI Community Working Group and tips on how to organize it in the best way. I would love to learn how you see your participation in the group, how you are applying AI agents to your robots, what kind of contributions you would like to make, and what your goals are!

19 Likes
2

Well done @adamdbrw taking the lead and attempting to unify AI/GenAI efforts. I’d be happy to participate if the WG takes off.

4 Likes
3

This is a great idea, and I’d be happy to help build the WG with you.

We’ve been thinking a lot about the ethical implications of embodied agents and see this as one of the major areas for improvement moving forward. It’s certainly not something that should happen in isolation. This WG seems like a good place to build on the standards we’ve established at NASA and to provide the community with solutions that are easy to adopt and grounded in an ethical framework.

Let me know what the timeline looks like and how we can help support!

3 Likes
4

I am interested in participating.

2 Likes
5

@adamdbrw thanks for taking a lead on this activity. i will be definitely there!

2 Likes
6

Hi Adam,

Please let me know when you set a time and place for your first meeting and I’ll help spread the word. I would recommend putting together a Google Form to collect e-mail address so you can notify interested parties and organize a Google Group.

4 Likes
7

Ho I’m Simone Voto from Reply. On this topic we are working from 1 year applyed on different robots. We are experimenting robot commanding with LLM. In particolar,now,we are working on using tools to give context from external system,likes RFID,in order to create a task forna robot. Here Is an example

Would be great to be part of this group to share our projects and to test new feature on real robots on real cases

2 Likes
8

Hi @adamdbrw interested in participating in this group.

1 Like
9

For anyone who’s interested, we have an IEEE pre-print on arXiv that details the ROSA framework as a first of its kind embodied agent that works for both ROS1 and ROS2. We also have an entire section dedicated to ethics for embodied agents and a demonstrations section that shows example usage on multiple physical and simulated robots.

You can see it in action at the JPL Mars Yard on the NeBula-Spot robot:

3 Likes
10

Hello, I’m clearly interested in ! Thank you for the initiative !

1 Like
11

That’s such a great idea! I’m definitely onboard. I’m also happy to support it any way I can.

One idea would be to select one paper or OSS library related to embodied AI per meeting, that everyone can look into before, and which can be discussed as part of the meeting. Better even if there is one volunteer to recap the content as introduction. Just as idea.

Looking forward to it, let me know how I can support it.

Jennifer
https://www.linkedin.com/in/jenbuehler

2 Likes
12

Hi, I’m interested in participating!

1 Like
13

Hi @adamdbrw thanks for taking the lead, I am interested in participating :grinning:. I have several AI ROS 2 packages, not only llama_ros and whisper_ros, but also tts_ros and yolov8_ros. We also have a chatbot_ros, controlled by an FSM created with YASMIN, that uses llama-3.2-1B, whisper-large-turbo and tts. For anyone interested in running LLM locally in ROS 2, here you have a demo of LangChain RAG within ROS 2. We are launching the embedding model, to create the vector database; the reranking model; and the llama-3.2-1B model using llama_ros.

2 Likes
14

Hi @adamdbrw. I am interested in participating in the WG. Nice initiative!

1 Like
15

Hi @adamdbrw , interested in this, keep us looped in please!

1 Like
16

Thanks everyone for such a great support for the idea! We are definitely moving forward with the Work Group and I am especially happy to get responses from representative of most of the projects in the space.

There is a simple registration form so that we can organize a group and notify everyone (thank you @Katherine_Scott for suggesting): Embodied AI Working Group registration. This will also allow us to determine the best time slot for calls for all of us.

I also reserved Birds of Feather session for those of us going to ROSCon in Odense, centered around Embodied AI, which will be a great kick-off for us to connect & discuss our common goals for the group. The meeting will be on the 21st at 16:00 in Room 208.

Looking forward to meeting you in person and on-line!

3 Likes
17

The group is launched!

3 Likes
18

Great, I am in ! Cheers

1 Like
19

@adamdbrw That’s a great initiative. I would like to be a part.

1 Like
20

Hello Simone, I watched the demo showcasing controlling the Spot robot using natural language. I’m interested to know more about this project in detail. How can I contact you?