MicroXRCE DDS on Ardupilot

Hello!

I just recently started working on MicroXRCE DDS support in Ardupilot. I’m building off of some previous work for a Google Summer of Code project that used EProsima’s MicroXRCE client and agent.

The PX4 community has been spending a fair bit of time working on this, and now have full support for their uOrb messages using either EProsima MicroXRCE client or the uRos client libraries, which is super amazing to see. They have demonstrated offboard control using it. On the other hand, the Ardupilot support has fallen stale, so I wanted to see what it would take to get it back on track.

So far in the past week, I’ve been able to accomplish the following.

  • Update the previous work in Foxy to compile and run on humble
  • Update the work to work off latest master of Ardupilot
  • Update Eprosima XRCE-DDS to the latest develop
  • Remove the custom messages used to trim down the PR to a builtin_interfaces/msg/Time message

In the last dev call with the ardupilot team, they had expressed concern about the flash size of MicroXRCE Client implementation. Although it is currently implemented to only be added by a compilation flag, they would like to enable it for mainline production builds if it can be trimmed down.
Key areas of concern were the runtime building of XML strings, and how to make it easier for ROS2 users to add new topics to Ardupilot without having to work in native code (C++). Unfortunately, it didn’t appear anyone was super experienced with MicroXRCE DDS, so we could use some help.

I have noticed that there are alternative ways of building the client configuration for the participants and topics according to the docs, but I can’t find any examples of this in a full application. Being able to avoid generating the XML profiles at compile time, or better yet not even have them in flash, would be awesome. Perhaps like so.

Would anyone familiar with either MicroXRCE DDS be able to help me answer some of the architecture questions in the following thread?

My eventual goal with this work is to enable offboard navigation of Arduplane using NAV2 through DDS instead of mavlink. In my professional work, I have architected the replacement of a custom CAN protocol with ROS2-message compatible DDS between the two Linux systems. The advantage it gave to the embedded device being able to visualize everything in a tool like Foxglove is of huge for devs, and that’s before you get to integrating any ROS2 commands to the embedded device.

4 Likes

Before and after. I figured out how to do build the client configuration from reference. This moves the XML from program flash of ardupilot to disk of offboard computer.
Before

Target               Text (B)  Data (B)  BSS (B)  Total Flash Used (B)  Free Flash (B)
--------------------------------------------------------------------------------------
bin/antennatracker    3009746    116733   197192               3126479  Not Applicable
bin/arducopter        3954606    170173   215048               4124779  Not Applicable
bin/arducopter-heli   3943934    171693   216136               4115627  Not Applicable
bin/arduplane         3874086    172525   212776               4046611  Not Applicable
bin/ardusub           3501974    138877   207752               3640851  Not Applicable
bin/blimp             2822538    109845   197288               2932383  Not Applicable
bin/ardurover         3611312    151877   213704               3763189  Not Applicable

After

BUILD SUMMARY
Build directory: /home/ryan/Documents/ardu_ws/src/ardupilot/build/sitl
Target               Text (B)  Data (B)  BSS (B)  Total Flash Used (B)  Free Flash (B)
--------------------------------------------------------------------------------------
bin/antennatracker    3009258    116733   197192               3125991  Not Applicable
bin/arducopter        3954150    170173   215048               4124323  Not Applicable
bin/arducopter-heli   3943446    171693   216136               4115139  Not Applicable
bin/arduplane         3873598    172525   212776               4046123  Not Applicable
bin/ardusub           3501486    138877   207752               3640363  Not Applicable
bin/blimp             2822018    109845   197288               2931863  Not Applicable
bin/ardurover         3610824    151877   213704               3762701  Not Applicable

For arducopter, it was 456 bytes difference for the one message. Considering px4_msgs has 184 different messages, if this was extrapolated out very roughly, that’s an 83.9KB saving. I don’t think there is any reason a ROS developer would need access to 184 different topics to do offboard control, but you never know I guess.

Huge!

2 Likes

Hello @RFRIEDM,

Nice to see this community work to integrate XRCE on Arduplane.

Regarding this:

We would like to mention that Micro XRCE-DDS supports 3 different ways of creating DDS entities, precisely to fit these use cases:

  • XML Creation Mode: Provides the whole flexibility of XML configuration, allowing the user to tune every single aspect of the entity being created in the embedded side.
  • Reference Creation Mode: Allows (as you mention in your post) some memory saving at the cost of moving the XML configuration to the Micro XRCE-DDS Agent side and just keeping a label or reference in the embedded side.
  • Binary Creation Mode: In this mode, entities are being created in the Micro XRCE-DDS Agent using a serialized and packed description that comes from the embedded side. This way, you can configure entities on the Client without using an XML string (this means saving memory), but with a reduced set of configuration options (see API for reference).

For example, the way to go in micro-ROS (at the RMW implementation that uses Micro XRCE-DDS) is the Binary Creation Mode since it provides enough flexibility (pub/sub, services, reliability, and most important QoS, etc) for micro-ROS without the need for handling huge XML strings.

I hope this info is relevant to your project, and do not hesitate to ask us if you have any other questions about micro-ROS or Micro XRCE-DDS.

Thanks for the kind words. I have successfully implemented your second option, reference creation mode. It works really great, and saved a lot of flash. At this point, we have decided to scope the PR to only use EProsima MicroXRCE Client libraries since flash size is of prime concern for the initial feature. At a later time, or if there is some more help, we can evaluate the MicroROS client libraries.

After a discussion with the maintainer of ArduPilot (tridge), we have set the following goal:

Develop a functional branch of MicroXRCE DDS on Ardupilot by the end of March in time for the ArduPilot developer conference. Ideally, merge it to master. The branch will feature three publishers (Odometry, NavSatFix, and BatteryState) that can be subscribed to from ROS 2 with ROS2 SensorData compatible QoS. We will demonstrate the feature on hardware: the Pixhawk 6X.

There are a few aspects of Micro-XRCE-DDS-Gen and the client libraries that I would love some support from EProsima. I have reached out to try to find a time to chat.

Thanks!

I have an internal meeting this week to evaluate the possible upgrades in Micro XRCE-DDS Gen. So you will hear news soon.

Regarding the options, please consider the Binary mode, IMO it will provide you more flexibility at the same flash cost.

At this point, my biggest request would be support to take a ROS2 compatible IDL that has module keyword in the IDL and be able to both parse it in the IDL parser and then generate C-compatible serialization. Please let me know if this is within scope for your possible upgrades.

Right now, if I take a ROS 2 .msg file, run the IDL converter to get an .idl file, then Micro XRCE DDS Gen can’t parse it. Right now, for Ardupilot users wanting to add more ROS2 messages to their builds, we would have to hack in something to strip the modules, which I would like to avoid.

After speaking to the Ardupilot dev team, here is what ROS2 users can look forward to for the first PR to master (End of march hopefully)

  • DDS XRCE library support from EProsima
  • uROS libraries - out of scope
  • Transport: serial
    • Ethernet is planned for later on Pixhawk 6X
  • Topics:
    • Best Effort publisher ~5Hz sensor_msgs/msg/BatteryState for the first battery on the system
    • Best Effort publisher~20Hz geometry_msgs/msg/Odometry, smoothed from AHRS. Frames TBD
    • Best Effort publisher~5Hz sensor_msgs/msg/NavSatFix for WGS-84 location
    • More topics can be added depending on flash implication
  • Subscribers for commands (waypoints, velocity or acceleration control) are out of scope initially
  • Data rates for publish will be hard coded for now, more time is needed for ROS2 parameter API investigation
    • It is highly desirable to be able to use ROS2 param service to list all params on the autopilot, and read/write specific values
  • Lua integrations to be able to subscribe or publish to topics on Ardupilot are desired, but out of scope for MVP
  • Waf will call MicroXRCE DDS Gen on a directory in the source tree for any .idl file. Users may add their own in MVP and copy the existing structure. In later updates, this process will get refined
  • XML on the ROS2 side will drive the configuration of participants, topics, and data writers
  • The XRCE client will operate on its own thread, scheduling will be hard coded for now from AP_Scheduler
  • An included launch file will interface with sim_vehicle.py to bring up a SITL ardupilot with Gazebo Garden to show the vehicle moving around
  • Ardupilot CI environment will have ROS2 humble installed through apt
  • Ardupilot CI tests will add a new test to check DDS works. This will ensure there are no regressions. The test will subscribe to each of the three topics and assert it is getting reasonable data at the right rates.
  • Stretch: The code infrastructure will optionally let users make topics and topic without the ROS2 prefixes (rt/<topic and <package>/msg/dds_/<msgName>_).
  • A more complete and usable functionality for applications like offboard control will be targeted for the 4.5 release of plane.
  • This feature will not, by default, be supported on 1MB flash boards. Only 2MB flash.
  • OS - required to use Ubuntu 22.04 to build MVP, will want to support Ubuntu 20.04, Mac, Windows in future

Had a great meeting with eProsima. They have done an amazing job merging PR’s to support this effort to develop. They already have a formal release to master planned for end of March, so for now, we’ll be using develop branch for all of eProsima’s repos.

Things they fixed already

Things left (not for MVP):

Hi @RFRIEDM regarding developing a C++ API for micro-ROS, I am glad to support. We had this discussion in the EWG previously. The question will be: would this C++ API be based on rclc - and to what extend will code-duplication be inevitable e.g. to support C++ types and more flexibility for callback functions like lambda expressions and normal class methods).

Just as a cross-reference, in the ROS 2 real-time working group we have the goal to develop a real-time ROS 2 Executor (in C++). So I wonder, could this Executor also be used for micro-controllers? Also when targetting Linux-OS, we want to remove dynamic memory allocation at runtime, provide deterministic behavior and make it real-time capable (e.g. by by thread prioritization of callbacks using the underlying operating system API (like sched_setparam in POSIX).

I was just thinking, not to do double work in the embedded community, but rather bringing ROS developers together:

  • analyze the requirements of a C++ Executor for micro-ROS, and for a real-time ROS 2 Executor
  • define a layered software architecture for such an Executor, so it could be used in both worlds.
2 Likes

Jan,

Sounds like a great room for collaboration! Coming as a C++/python dev, I’m newer to embedded, so likely not the person to take the lead. That said, having worked on this on an embedded platform with similar constraints would be really beneficial. Once we get our MVP merged (2-3 publishers from embedded device), I should have some bandwidth and better understand to discuss your real time C++ executor.

General update here: I’ve gotten the PR pretty much ready to go. It’s been scoped down for the initial merge just to have one topic, then increase capabilities after that. This is to improve collaboration because working on the same PR branch with 3-4 people is getting cumbersome.

Anyways, we’ve achieved publishing builtin_interfaces/msg/Time at ~300Hz without any tuning; we have not benchmarked the latency yet.

As far as any blockers, the eProsima XRCE DDS Generator is continuing to have little snags, like support for modules, includes with folders in the path, uint8 alias, modules with arrays, illegal identifier in PoseStamped, NavSatStatus no such attribute, and DDS Status Error with multiple publishers however eProsima has been really helpful each time we file a bug to issuing a fix within a day or two to develop. It’s not clear how much stuff is going to need to be fixed to support an unmodified PoseStamped message, but it’s at least closer than before.

When I tried configuring the client to publish multiple topics, I got a UXR_STATUS_ERR_DDS_ERROR return code on the 2nd publisher creation request; I’ll be working in the uROS Slack Channel to resolve it. It’s not entirely clear whether one should be populating XML for publishers when there are multiple; in the docs, the publisher or subscriber XML is an empty string.

XRCE DDS support is now merged to master in ArduPilot!

I’ve started working with the team to scope the features that will make it into the next release (4.5).

4 Likes

I was invited to present at the ArduPilot developer conference. The talk summarizing my work with Micro XRCE DDS is enclosed below:

1 Like