Currently, Artificial Intelligence technologies have a wide range of applications in the field of robotics such as language processing, face detection, object recognition, or position estimation; with application domains as varied as agriculture, autonomous vehicles or medicine. However, there is no effort to unify the integration of the different ML technologies in the ROS 2 ecosystem.
Although there are a variety of approaches to use ML in ROS 2 robots through the use of specific development frameworks such as Tensorflow, Pytorch, or OpenVINO, there is no current attempt to standardize the integration of such ML technologies into ROS 2 systems, providing a common interface for accessing and exploiting ML models.
That is why eProsima presents this proposal for an AI Integration Working Group, in which we intend to address the following objectives.
Unify AI integration in the ROS 2 ecosystem through a common interface.
Give visibility to community research projects that illustrate the use of ML in ROS 2.
Study the industry exploitation of ML in ROS 2 by presenting real scenarios of application.
Study the state of the art of novel ML technologies that can be applied to the robotic sector.
Enable ML technologies in embedded systems.
Address new paradigms of Human-Machine partnership in the field of robotics with AI.
Study implementations of hardware acceleration technologies for ML algorithms.
Please share in the comments below any suggestions or questions about this WG proposal in order to build a supportive ROS 2 AI community.
I’m not convinced that there’s no attempt to standardize integration of ML technologies into ROS2. I think you can make a pretty strong argument that ROS2 itself is the standardized integration: Stochastic approaches are a tool for a job, and sometimes it’s the right tool to provide the location of an identified object (Poses), to identify points for object grasping/manipulation (rail_grasp), to segment objects within a field of depth (pcl), to provide localalization estimates (PoseStamped), to instruct movement behavior (Twist), and many others.
But, those interfaces already exist in a way that doesn’t require the implementation to be probabilistic (which is good, because there’s no reason for them to be since sometimes it’s not the right tool for the job). I suppose a working group could try to standardize the python/c++ API’s to models created for specific tasks, but that sort of defeats the purpose of having ROS involved in your system at all.
I might be missing something, but the most concrete goal listed here is to provide that integration, but I think that’s been done already. But maybe there’s some use-case or background experience I’m missing here?
AI is a large domain, which includes machine learning, and it’s subset, deep learning. Deep learning contains multiple approaches, including supervised, unsupervised, and reinforcement learning, each with varying levels of maturity and ongoing innovation. Is this proposal specifically for deep learning?
There is ongoing rapid innovation in multiple areas of deep learning, and AI in general. Areas, like detection, classification and segmentation are mature mostly-solved functions, while some others are not. Common interfaces for inputs and outputs could be defined for the more mature inference functions; interfaces for other areas may be outdated as the inference functions advance with changes in the input(s). Is the proposal to cover all of AI, or specific areas over some maturity level?
From our experience having some standardization on inputs to packages that perform inferencing on deep learning models, would benefit the community. There is for example no standard definition of a tensor in ROS we have found. The outputs in some cases are consistent with existing packages as @cst0 mentioned as there is prior-art from CV based approaches to the same function; a gap perhaps is in output confidence as prediction approaches are probabilistic in nature and not 100% accurate. An output confidence enables approaches for fusion of multiple predictions from different models, or input sensors.
Is a concrete goal here to standardize inputs to inferencing?
I am not sure if this requires a separate working group. We use a lot of AI in our computer vision pipelines and for us inference is the simple part of the pipeline. It takes a CV Image in , runs inference and spits out a CV image for further post processing.
We used a lot of NVIDIA TRT examples in the beginning but outside of CV Image we do not need much more than that. Maybe a few common libraries for inputs and outputs like @ggrigor suggests could be good.
Unless you are looking at AI for robotics in general with different algorithms for robot behaviours, etc,. That might be an area that requires a separate working group.
I’ve said previously at TSC meetings that I think it would be helpful if the working groups had reasonably clear borders between their efforts. In my mind, this is a case where, “good fences make good neighbors.” I think the example I gave the TSC was creating three distinct working groups, “data collection and processing, and training”, “inference and deployment”, and “hardware acceleration.” My opinion is that breaking things up like this would reasonably clear borders between the working groups, and help target the efforts of participants. Reading your list of topic areas, you list:
Enable ML technologies in embedded systems.
Study implementations of hardware acceleration technologies for ML algorithms.
These topics seem to overlap with @vmayoral’s Hardware Acceleration Working Group.
Most of the remaining topics seem to focus more on studying the state of the art versus addressing software needs within ROS 2. I think this is great, but I want to clarify, is your intent here to serve the community as an educational resource or as a software development resource? Both of these are needed and great, but I think we should state concrete goals for each working group.
LGTM, count with us, especially for the robotics perspective.
As mentioned throughout the comments, IMHO this should not be limited to Deep Learning. I’m inclined to believe that it should be more oriented to the kind of problem to solve (supervised learning, unsupervised learning, reinforcement learning…) rather than the specific solver technique.
However, this is obviously debatable. In fact, maybe these approaches are not completely mutually exclusive (e.g. supervised learning problem → an input-output interface; solving via deep learning → a neural network parameter/hyperparameter interface). I would aim to decouple problem/solver interfaces as clearly as possible; but am also aware that, in practice, many use cases where this is difficult may appear.
I agree with some previous comments, there’s value behind providing a common interface for using ML models in ROS 2 computational graphs. However, it’s to me still very unclear why this group is needed and/or how exactly do you plan do it. In my view, the right way to think about this is to align with ROS 2 standard messages so that practitioners can consider replacing ROS 2 packages/functionality with AI-powered ones (and this is going to take a while , if happens).
You mentioned above a few ML libraries and frameworks, let me call these level 1. Myself and my teams in the past contributed gym_gazebo (paper), gym_gazebo2 (paper) and ros2_learn (paper), which many have extended. These and similar projects aim to provide an abstraction layer on top of level 1, so that integrating things into ROS becomes easier for dev. purposes. I’ll call these level 2. Finally, we have projects like openrobotics_darknet_ros, or isaac_ros_pose_estimation which relates to @ggrigor’s input above. These are projects that use AI and produce usefulness that can then be fed into ROS 2 computational graphs. Let’s call these level 3. So what’s your target? It sounds to me like you’re targeting level 1 and if so, then I believe this is not the right forum.
Which ROS 2 AI projects specifically are you looking to align providing a “common interface for accessing and exploiting ML models”@rsanchez? Shouldn’t you start by adressing this first and identify ROS 2 packages that add already value, so that you can work on the interfacing? Also, can you address the questions posted by others above? There’re some really good questions raised.
There’s indeed an overlap in here. Not just with HAWG, but also with Edge AI led by @kydos. Here’s my read:
WG
Goal
Hardware Acceleration (HAWG)
drive creation, maintenance and testing of acceleration kernels on top of open standards (OpenCL) for optimized ROS 2 and Gazebo interactions over different compute substrates (FPGAs, GPUs and ASICs).
Edge AI
make Edge AI easier and ubiquitous in ROS 2, specifically ML applied to navigation, perception & picking, inspection and motion planning.
AI Integration
provide a common interface for accessing and exploiting ML models.
For what concerns the HAWG overlap:
Study implementations of hardware acceleration technologies for ML algorithms.
I think it should be fine provided this group complies with REP-2008 PR and/or helps refine it if needed, so that any contribution is directly useable by the rest of the community looking at hardware acceleration and we don’t create unnecessary fragmentation.
Similar to @peterdavidfagan 's comment, I would be interested in participating in such a working group as well.
Under the ROS-Industrial Consortium Asia Pacific team, I am currently heading an effort to ease AI integration for Computer Vision-enabled robotic applications. This can be seen in easy_perception_deployment which is currently being actively developed with local governmental support.
The project is aimed at providing intuitive and business-friendly access to robotics startup for prototyping and deploying Computer Vision-enabled robotics. Through (at least) my team’s research, we are trying to address this gap via use of Transfer-Learning and the use of the Onnx/OnnxRuntime ML export and deployment framework. Both approaches are based on an understanding of 2 endemic issues regarding AI integration in ROS and industrial setups; Long training time and Machine Learning Framework Lock-In.
Should this working group be formed. I will be more than willing to share more of my findings on similar issues and its current workarounds.
sorry for the delay in replying but I didn’t want to leave out any of your questions. Also, thank you all for participating and showing your point of view. Thanks to that we have been reconsidering the objectives of this WG and its goal.
Hi @peterdavidfagan , we are counting on you. Regarding your question, no, it is not part of the objectives to create pre-trained models and datasets. However, it could be possible to distribute the (public) models and datasets with which the integration framework is tested.
Yes, as you say ROS 2 can be an integration standard by itself although it has major gaps when it comes to using ML models and a standardized API for accessing them. Currently, as you mention, there are a great number of AI applications and concrete tools to solve them, but there is a lack of unification of data types and facility for the user to deploy ML models in final applications.
The purpose of the WG is exactly that, to create a package that facilitates the integration and deployment of ML models in ROS 2 systems.
Hi @ggrigor, the WG would not be just focused on DL although we would start there as it is highly exploited in robotics systems for Computer-Vision enabled applications.
As @cardboardcode mentions, we had the idea of starting the standardization of model deployment and inference using ONNX and its DNN conversion tools. Anyway, that is the idea of the WG, to see what is the current state of the art in the use of ML, starting with DL, by the ROS 2 community in order to lay the groundwork to start a standardization project of ML model inference and deployment.
The purpose of the WG is exactly that, to create a package that facilitates the integration and deployment of ML models in ROS 2 systems.
Even though it is true that the inference is not complex thanks to the ML frameworks based on Python libraries, each framework has its own inference mechanism. Moreover, I think that the real expensive issue in your case is the deployment of the models, since the costly part is the training of the model, which in many cases has to be performed on a dedicated machine and then deployed on the final system.
Thanks for the support @Katherine_Scott. First of all, this is intended to be a TSC sponsored WG aimed at boosting the visibility of the use of AI applications in ROS 2 and exploiting this research field.
As you mention, AI Integration WG may not be a good name due to the large amount of uncertainty it raises so it could be considered AI: Inference & Deployment WG or another name that fits this field of study as this is what the WG intends to focus on.
This WG aims to create a ROS 2 package for the deployment of ML models and the performance of inferences in the final system, so it would serve as an software development resource.
Yes, we are considering the use of the common ROS 2 interfaces as well as the generation of new AI application common interfaces in order to exploit the deployment of ML models, and inference inputs and results.
Sorry if this is misunderstood but the intention is to focus on the level 2 you mention. We will work on creating an abstraction layer (ROS 2 package) that facilitates the deployment of models and unifies access to them.
Since there is already a HW Acceleration WG, we will do our best not to overlap with your WG so I will remove this item from the list. Also, in case this issue comes up or needs to be addressed, I will contact you to see if we can participate in your WG.
Finally, I suspect that the Edge AI WG is closed, at least it is inactive since, as far as I know, it has only had one meeting in the last year due to lack of activity. This indicates that having a very dedicated WG can lead to this lack of activity and contribution from the community, so I would opt to address a broader topic always focusing on the boundaries of the WG as Kat says.
This concerns me a bit. Can you be more specific please, how are you going to do this and what’s the expected value for the community? level 2 above was defined as packages that facilitate integrating AI frameworks into ROS 2 for dev. purposes. Are you sure you want to focus on this? There’re already quite a few packages doing it. I’ve kept track of some over the years:
There’re many (many) more. Generally forks and forks of forks, most build upon the same principles. What’s your take on this and how would you add value to level 2? Please clarify, at this point it’s unclear and feels you’re planning to create another fork (at least capability-wise).
Have you guys considered going one step forward into level 3? Being familiar with many of the approaches listed above, my experience indicates that (level 3) is where the real value is today[1]. There’re tons of projects out there that require interfacing in a coherent manner and that build upon level 2 existing results. Besides the two above, another example is drl_grasping. There’re many more.
Having a TSC-driven WG focused on making AI models first class participants of the ROS ecosystem would be very useful.
I think that what you define as level 3 are very specific applications for each use case and I think that is not the way for an AI WG. I do think it is fine to consider the end users applications to detect the interfaces needed for each of them and facilitate the deployment of these in ROS 2 by filling the gap of deploying ML models in ROS 2.
Having said that, I think we are talking about the same thing. The aim of this WG is just what you said, making AI models first class participants of the ROS ecosystem, and I consider it as level 2 but unifying interfaces for access to AI frameworks (level 1).
For example and very briefly explained, having a training server and a target system with ROS 2, offer the possibility to deploy the trained models independently of the framework used (level 1) through a common and specific interface for AI applications in ROS 2 ecosystems. The difference between that and the examples you show of level 2 is that those examples specify the AI framework used underneath and use unique interfaces for that framework. On top of that the users would build the level 3 applications you mention.
I edited my answer above (and definition of level 2) for consistency with previous comments. Please check.
We’re getting there, but I think you’re trying to solve the wrong problem in here. You’re thinking as an AI-guy and not as a ROS/robotics practitioner. In my view, trying to unify AI-frameworks into a common interface is someone else’s problem, not this community’s. It’s great if you want to do it, but that should not be the core of any WG in here.
This AI mindset is the conflict (and what’s making many over-expect the actual capabilities of Artificial Intelligence in robotics today). I believe you should be thinking about how ROS users can integrate AI into their computational graphs in an easier manner (at least this is what I understood your initial description).
Note that many people in this community (and robotics, in general) are actually already quite invested on AI frameworks, it’s thereby highly unlikely that your work will change this. You can (and I think, should) produce a series of REPs/specifications/alignment-with-ROS-2-standard-messages and adapters (simple ROS 2 Nodes that do the actual interfacing) that hints AI folks what should their output tensors look like. That way, you’ll get to integrate AI efforts nicely with ROS 2 computational graphs.
There’s nothing use-case specific about this. Give it a thought!
I am not sure with " making AI models first class participants of the ROS ecosystem" as a separate working group. Are you proposing have pre-trained models that are optimized for ROS?
In our use case we have to train the models from scratch because we have very different use cases then the general AI for driving use case (for example). We also shift between TensorRT, Movidius and now NXP, so having general optimized models for use in ROS2 would not work for us at all.
Lastly deployment should not be part of ROS. Each company will have their own deployment system based on their requirements, (AWS, Azure, docker, etc).
I still do not see the need for a separate WG. At least for us I would not see much value in it the way its being proposed
Developing an AI model is a data driven iterative development process, that requires cloud infrastructure, training tools, and data management. This is framework, infrastructure, and data specific. There is more than a training server and doesn’t seem a problem for ROS to solve.
AI models need to be processed to run on the target hardware. This is platform specific and part of the AI model development process … not something independent or generalized.
Deployment of models (a bunch of numbers|weights) can be updated separately from the firmware or with the firmware depending on your preference. This should not be part of ROS.
+1 with @Shawn_Schaerer that there is not much value as the WG is currently proposed.
Hi, it still seems there’s a lot of mutual misunderstanding in this thread. @rsanchez, could you provide a few names of possible ROS packages or REPs that you think might be the outcome of the proposed WG? I think having such concrete examples would help get a clearer idea…
I agree with many of the points that were made above but I also have a perspective that I thought I would share in case it can add to the conversation.
I think the point raised here was very applicable, it can be challenging to keep track of all the ROS packages that are being created especially for AI applications. I feel it would be beneficial to have a group identify mature projects and look to consolidate these with packages under a single organisation. The hope would be that this would make it easier for ROS newcomers to have a single reliable group of core ROS packages for AI applications. It may also have the benefit of funnelling some of the disparate development work into development that matures a set of core packages, I agree on the need not to reinvent the wheel here. There may also be room for some standardisation between these packages. I agree on the need for consolidation as outlined by @vmayoral when discussing level 3.
I agree that when it comes to real-world applications training models from scratch for custom use cases and optimising runtime for specific hardware is a crucial topic. With this being said I still believe that there is some benefit in standardisation for the open-source community and low-budget startups. I believe some standardisation may help facilitate a growth in the number of users applying AI to real-world robotics problems with ROS. I see this resulting from the fact that standardisation may help enable benchmarking of results on real robot hardware, facilitate the sharing of pretrained models and provide some best-practices/guidance for model deployment on popular hardware. I believe there are many potential users of ROS who won’t have the capacity to collect data and train custom models and whom could benefit from the community that could form around standardised packages.
I agree with the above. Could data collection remain relevant to the ROS community as it is one way in which ROS may interface with say cloud data management infrastructure?
I agree on this point also. One point I would make is that I feel outside of custom commercial applications it may be feasible to standardise around common/popular hardware and platforms. For instance PyTorch can leverage different parallel compute platforms such as CUDA. Is it worthwhile developing packages that standardise deployment to a number of platforms? I guess the answer is no but I am not yet convinced and honestly need to do my own research on this to be more informed.
+1 on the above I’d be interested in this response.
These are some of my current thoughts, I’d be happy to hear other opinions and feedback on the above. I wouldn’t consider myself a ROS expert so there is a chance that I have overlooked something in my response, I’d be happy for others to point this out in places where my perspective isn’t well formed. It’s great to be having this conversation, even if a working group doesn’t form I think there is value in this discussion.
, if happens).