The PX4 community has been spending a fair bit of time working on this, and now have full support for their uOrb messages using either EProsima MicroXRCE client or the uRos client libraries, which is super amazing to see. They have demonstrated offboard control using it. On the other hand, the Ardupilot support has fallen stale, so I wanted to see what it would take to get it back on track.
So far in the past week, I’ve been able to accomplish the following.
Update the previous work in Foxy to compile and run on humble
Update the work to work off latest master of Ardupilot
Update Eprosima XRCE-DDS to the latest develop
Remove the custom messages used to trim down the PR to a builtin_interfaces/msg/Time message
In the last dev call with the ardupilot team, they had expressed concern about the flash size of MicroXRCE Client implementation. Although it is currently implemented to only be added by a compilation flag, they would like to enable it for mainline production builds if it can be trimmed down.
Key areas of concern were the runtime building of XML strings, and how to make it easier for ROS2 users to add new topics to Ardupilot without having to work in native code (C++). Unfortunately, it didn’t appear anyone was super experienced with MicroXRCE DDS, so we could use some help.
I have noticed that there are alternative ways of building the client configuration for the participants and topics according to the docs, but I can’t find any examples of this in a full application. Being able to avoid generating the XML profiles at compile time, or better yet not even have them in flash, would be awesome. Perhaps like so.
Would anyone familiar with either MicroXRCE DDS be able to help me answer some of the architecture questions in the following thread?
My eventual goal with this work is to enable offboard navigation of Arduplane using NAV2 through DDS instead of mavlink. In my professional work, I have architected the replacement of a custom CAN protocol with ROS2-message compatible DDS between the two Linux systems. The advantage it gave to the embedded device being able to visualize everything in a tool like Foxglove is of huge for devs, and that’s before you get to integrating any ROS2 commands to the embedded device.
Before and after. I figured out how to do build the client configuration from reference. This moves the XML from program flash of ardupilot to disk of offboard computer.
Before
Target Text (B) Data (B) BSS (B) Total Flash Used (B) Free Flash (B)
--------------------------------------------------------------------------------------
bin/antennatracker 3009746 116733 197192 3126479 Not Applicable
bin/arducopter 3954606 170173 215048 4124779 Not Applicable
bin/arducopter-heli 3943934 171693 216136 4115627 Not Applicable
bin/arduplane 3874086 172525 212776 4046611 Not Applicable
bin/ardusub 3501974 138877 207752 3640851 Not Applicable
bin/blimp 2822538 109845 197288 2932383 Not Applicable
bin/ardurover 3611312 151877 213704 3763189 Not Applicable
After
BUILD SUMMARY
Build directory: /home/ryan/Documents/ardu_ws/src/ardupilot/build/sitl
Target Text (B) Data (B) BSS (B) Total Flash Used (B) Free Flash (B)
--------------------------------------------------------------------------------------
bin/antennatracker 3009258 116733 197192 3125991 Not Applicable
bin/arducopter 3954150 170173 215048 4124323 Not Applicable
bin/arducopter-heli 3943446 171693 216136 4115139 Not Applicable
bin/arduplane 3873598 172525 212776 4046123 Not Applicable
bin/ardusub 3501486 138877 207752 3640363 Not Applicable
bin/blimp 2822018 109845 197288 2931863 Not Applicable
bin/ardurover 3610824 151877 213704 3762701 Not Applicable
For arducopter, it was 456 bytes difference for the one message. Considering px4_msgs has 184 different messages, if this was extrapolated out very roughly, that’s an 83.9KB saving. I don’t think there is any reason a ROS developer would need access to 184 different topics to do offboard control, but you never know I guess.
Nice to see this community work to integrate XRCE on Arduplane.
Regarding this:
We would like to mention that Micro XRCE-DDS supports 3 different ways of creating DDS entities, precisely to fit these use cases:
XML Creation Mode: Provides the whole flexibility of XML configuration, allowing the user to tune every single aspect of the entity being created in the embedded side.
Reference Creation Mode: Allows (as you mention in your post) some memory saving at the cost of moving the XML configuration to the Micro XRCE-DDS Agent side and just keeping a label or reference in the embedded side.
Binary Creation Mode: In this mode, entities are being created in the Micro XRCE-DDS Agent using a serialized and packed description that comes from the embedded side. This way, you can configure entities on the Client without using an XML string (this means saving memory), but with a reduced set of configuration options (see API for reference).
For example, the way to go in micro-ROS (at the RMW implementation that uses Micro XRCE-DDS) is the Binary Creation Mode since it provides enough flexibility (pub/sub, services, reliability, and most important QoS, etc) for micro-ROS without the need for handling huge XML strings.
I hope this info is relevant to your project, and do not hesitate to ask us if you have any other questions about micro-ROS or Micro XRCE-DDS.
Thanks for the kind words. I have successfully implemented your second option, reference creation mode. It works really great, and saved a lot of flash. At this point, we have decided to scope the PR to only use EProsima MicroXRCE Client libraries since flash size is of prime concern for the initial feature. At a later time, or if there is some more help, we can evaluate the MicroROS client libraries.
After a discussion with the maintainer of ArduPilot (tridge), we have set the following goal:
Develop a functional branch of MicroXRCE DDS on Ardupilot by the end of March in time for the ArduPilot developer conference. Ideally, merge it to master. The branch will feature three publishers (Odometry, NavSatFix, and BatteryState) that can be subscribed to from ROS 2 with ROS2 SensorData compatible QoS. We will demonstrate the feature on hardware: the Pixhawk 6X.
There are a few aspects of Micro-XRCE-DDS-Gen and the client libraries that I would love some support from EProsima. I have reached out to try to find a time to chat.
At this point, my biggest request would be support to take a ROS2 compatible IDL that has module keyword in the IDL and be able to both parse it in the IDL parser and then generate C-compatible serialization. Please let me know if this is within scope for your possible upgrades.
Right now, if I take a ROS 2 .msg file, run the IDL converter to get an .idl file, then Micro XRCE DDS Gen can’t parse it. Right now, for Ardupilot users wanting to add more ROS2 messages to their builds, we would have to hack in something to strip the modules, which I would like to avoid.
After speaking to the Ardupilot dev team, here is what ROS2 users can look forward to for the first PR to master (End of march hopefully)
DDS XRCE library support from EProsima
uROS libraries - out of scope
Transport: serial
Ethernet is planned for later on Pixhawk 6X
Topics:
Best Effort publisher ~5Hz sensor_msgs/msg/BatteryState for the first battery on the system
Best Effort publisher~20Hz geometry_msgs/msg/Odometry, smoothed from AHRS. Frames TBD
Actually, switch to PoseStamped to support Foxglove.
Best Effort publisher~5Hz sensor_msgs/msg/NavSatFix for WGS-84 location
More topics can be added depending on flash implication
Subscribers for commands (waypoints, velocity or acceleration control) are out of scope initially
Data rates for publish will be hard coded for now, more time is needed for ROS2 parameter API investigation
It is highly desirable to be able to use ROS2 param service to list all params on the autopilot, and read/write specific values
Lua integrations to be able to subscribe or publish to topics on Ardupilot are desired, but out of scope for MVP
Waf will call MicroXRCE DDS Gen on a directory in the source tree for any .idl file. Users may add their own in MVP and copy the existing structure. In later updates, this process will get refined
XML on the ROS2 side will drive the configuration of participants, topics, and data writers
The XRCE client will operate on its own thread, scheduling will be hard coded for now from AP_Scheduler
An included launch file will interface with sim_vehicle.py to bring up a SITL ardupilot with Gazebo Garden to show the vehicle moving around
Ardupilot CI environment will have ROS2 humble installed through apt
Ardupilot CI tests will add a new test to check DDS works. This will ensure there are no regressions. The test will subscribe to each of the three topics and assert it is getting reasonable data at the right rates.
Stretch: The code infrastructure will optionally let users make topics and topic without the ROS2 prefixes (rt/<topic and <package>/msg/dds_/<msgName>_).
A more complete and usable functionality for applications like offboard control will be targeted for the 4.5 release of plane.
This feature will not, by default, be supported on 1MB flash boards. Only 2MB flash.
OS - required to use Ubuntu 22.04 to build MVP, will want to support Ubuntu 20.04, Mac, Windows in future
Had a great meeting with eProsima. They have done an amazing job merging PR’s to support this effort to develop. They already have a formal release to master planned for end of March, so for now, we’ll be using develop branch for all of eProsima’s repos.
Can discuss as a community how to approach this. Even having a separate .hpp header for some of the client calls that can be constexpr would be nice.
No known path forward to support exposing ALL params of Ardupilot into the ROS param environment. This is because xrce only supports static arrays, and ListParameters is unbounded array type.
Perhaps just start with individual get/set params
No CI for Mac developers of ROS2 in Micro XRCE DDS Client
Shouldn’t be too hard to add at some point
Installing parts of eProsima libs as a system install, and others as local install can lead to mysterious behavior, and even segfaults.
Between three devs, we’ve spent hours chasing down things that were due to this.
Hi @RFRIEDM regarding developing a C++ API for micro-ROS, I am glad to support. We had this discussion in the EWG previously. The question will be: would this C++ API be based on rclc - and to what extend will code-duplication be inevitable e.g. to support C++ types and more flexibility for callback functions like lambda expressions and normal class methods).
Just as a cross-reference, in the ROS 2 real-time working group we have the goal to develop a real-time ROS 2 Executor (in C++). So I wonder, could this Executor also be used for micro-controllers? Also when targetting Linux-OS, we want to remove dynamic memory allocation at runtime, provide deterministic behavior and make it real-time capable (e.g. by by thread prioritization of callbacks using the underlying operating system API (like sched_setparam in POSIX).
I was just thinking, not to do double work in the embedded community, but rather bringing ROS developers together:
analyze the requirements of a C++ Executor for micro-ROS, and for a real-time ROS 2 Executor
define a layered software architecture for such an Executor, so it could be used in both worlds.
Sounds like a great room for collaboration! Coming as a C++/python dev, I’m newer to embedded, so likely not the person to take the lead. That said, having worked on this on an embedded platform with similar constraints would be really beneficial. Once we get our MVP merged (2-3 publishers from embedded device), I should have some bandwidth and better understand to discuss your real time C++ executor.
General update here: I’ve gotten the PR pretty much ready to go. It’s been scoped down for the initial merge just to have one topic, then increase capabilities after that. This is to improve collaboration because working on the same PR branch with 3-4 people is getting cumbersome.
Anyways, we’ve achieved publishing builtin_interfaces/msg/Time at ~300Hz without any tuning; we have not benchmarked the latency yet.
When I tried configuring the client to publish multiple topics, I got a UXR_STATUS_ERR_DDS_ERROR return code on the 2nd publisher creation request; I’ll be working in the uROS Slack Channel to resolve it. It’s not entirely clear whether one should be populating XML for publishers when there are multiple; in the docs, the publisher or subscriber XML is an empty string.