Wire Breaking Change in rmw_zenoh for ROS 2 Jazzy

ROS ROS Announcements and News
,
1

Important Announcement: Wire Breaking Change in rmw_zenoh for ROS 2 Jazzy

We’re writing to inform the ROS community about an upcoming wire breaking change in rmw_zenoh that will be included in the next sync of binary packages for ROS 2 Jazzy.

The current version of rmw_zenoh on Jazzy is 0.2.3, and the soon-to-be-released version will be 0.2.5. In version 0.2.4, we introduced a backport (detailed in this pull request) that updates the serialization format used for attachments.

While our primary goal is to maintain stability for released distributions, especially concerning core components, we made an intentional decision to backport this specific change to Jazzy. This was done to achieve improvements in performance and to enable seamless communication with embedded systems using zenoh-pico.

What This Means for You

If you’re using rmw_zenoh on ROS 2 Jazzy, you’ll need to be aware that once version 0.2.5 is released and synced, it will not be able to communicate with previous versions (0.2.3 and earlier) due to the serialization format change.

It’s crucial that all your systems are updated to 0.2.5 once the binary is released. This includes any Docker images you may have cached. Systems running 0.2.5 will not be able to communicate with those on 0.2.3. Attempting to do so will result in a runtime error that will crash the process. Please plan to update all rmw_zenoh instances in your Jazzy deployments to 0.2.5 to ensure continued communication.

A Note on Future Distros

As a reminder, rmw_zenoh is designated as Tier-1 starting from ROS 2 Kilted. For Kilted and future distributions, we are committed to maintaining wire compatibility and will not introduce such breaking changes. This backport to Jazzy was an an exceptional case driven by the substantial benefits it provides for performance and embedded system integration.

We apologize for any inconvenience this may cause and appreciate your understanding as we work to improve the ROS 2 ecosystem.

If you have any questions or concerns, please feel free to comment below.

5 Likes
2

Wow I haven’t expected such change would happen, because the in-release stability has always been great in ROS.

So just to lay things as they are (hopefully I read all the threads about it correctly) : the Jazzy and Humble backports were merged by mistake, right? No big discussion, no announcement… Then “thanks” to a buildfarm bug, people got new version on arm64 and old version on amd64, which caused mayhem. So the new version was reverted from buildfarm, but the main reason was OpenCV challenge (otherwise, I see that probably, instead of reverting, the action would have been to fix the amd64 version to the new one). And then Marco decided (where, when? it’s not documented anywhere? or was it the PMC meeting with still pending notes?) that he will re-release the new version for all platforms in the next sync.

I understand that mistakes do happen. What I don’t understand is the last step about re-releasing the breaking change after it was reverted. I’d expect that such drastic change will be accompanied with lots of hard evidence showing its usefulness. However, except a few sentences like “serializing fewer values has to be faster”, I haven’t seen any hard data. There is a slight attempt in [Bug] ze_serializer is quite resource intensive · Issue #926 · eclipse-zenoh/zenoh-pico · GitHub , but that’s a synthetic example. For example, I haven’t yet seen any estimate about the wire traffic savings in real systems. Apart from this missing performance report, I thing giving the community a week or two to prepare for the change is a super short notice.

The trust of the community that the PMC ensures quality of released versions is a fragile thing. But it is a key part affecting ROS adoption. You break the trust once, you re-gain it much longer.

Personally (and selfishly), I’m actually glad for this particular change, because it seems useful and we haven’t yet switched to ROS 2 on most of our robots. I’m just quite unhappy with the process around it.

Also, what’s the plan for humble? I see the change got backported, too, but there is no planned or recently merged rosdistro PR.

3 Likes
3

Does this mean bags recorded with the old version cannot be played back with the new version?

4

Hi @danthony recorded rosbags can be replayed without any issues.

5

Hi @peci1,

I want to start by saying we agree that in-release stability is a cornerstone of the ROS ecosystem, and we apologize for the confusion and frustration this situation has caused. I understand why this change was unexpected, and I appreciate you bringing your feedback to us!

Let me first provide some context and then clarify the sequence of events.

Our Rationale for Backporting

Until its promotion to Tier-1 in Kilted, rmw_zenoh was under very active, rapid development. We made no formal stability declarations for it on Jazzy; in fact, our first binary release for Jazzy was only in January of this year. We were transparent in that announcement that it was not yet a stable, core component.

Our primary goal for the Jazzy binaries was to encourage community testing and gather feedback. The feedback we received, particularly regarding performance on resource-constrained systems using zenoh-pico, was instrumental. This led directly to the serialization improvements you’ve noted. Our decision to backport this breaking change was intentional—we believed the significant benefits for performance and embedded interoperability for all users outweighed the risk, given the package’s declared development status at the time.

Clarifying the Sequence of Events

So just to lay things as they are (hopefully I read all the threads about it correctly) : the Jazzy and Humble backports were merged by mistake, right? No big discussion, no announcement… Then “thanks” to a buildfarm bug, people got new version on arm64 and old version on amd64, which caused mayhem. So the new version was reverted from buildfarm, but the main reason was OpenCV challenge (otherwise, I see that probably, instead of reverting, the action would have been to fix the amd64 version to the new one). And then Marco decided (where, when? it’s not documented anywhere? or was it the PMC meeting with still pending notes?) that he will re-release the new version for all platforms in the next sync.

I’d like to clarify a couple things in your message.

  1. No Mistake in Backporting: The backporting of these changes was an intentional decision. As a nascent project under heavy development for Kilted, we chose to include these optimizations in Jazzy so that users could benefit from them. The changes were never reverted in the source branches.

  2. The Revert was Intentional and Proactive: We did create a 0.2.4 release of rmw_zenoh with the breaking changes (ros/rosdistro#45285). However, we proactively reverted it before the sync went out (ros/rosdistro#45556). This was done for two key reasons:

    • To ensure stability for the OpenCV challenge.
    • Crucially, to give us time to write a proper announcement (this post) rather than having the change land silently. We realized that even for a non-Tier-1 package, this level of breaking change required a heads-up.

  3. The Buildfarm Issue: The fact that different architectures (arm64 vs. amd64) temporarily had different versions was an unfortunate and independent buildfarm mishap, which you can read about here. This incident complicated the situation, but it was not the driver for our decision to pause and announce—it was a parallel problem. tldr; arm64 package did not get a chance to rebuild before the sync went out and the version was automatically downgraded in the subsequent sync.

  4. The Decision to Proceed: The decision to move forward with the release now is a maintainer-level decision, not a directive from the ROS Boss (Marco), whose role is to execute the syncs. We are now proceeding with the release after the OpenCV challenge has concluded and after providing the community notice we failed to provide the first time. The code changes were never reverted in the source branches; we simply paused the binary deployment.

Our Commitment to Transparency and Future Stability

Even today, transparency with the ROS community remains our top priority, which is why we made the effort to create this announcement. We believe that the long-term value this change brings to Jazzy users, particularly for performance and seamless communication with embedded systems via zenoh-pico, outweighs the short-term inconvenience. We are incredibly grateful to our early adopters for their testing and feedback, which has been instrumental in improving rmw_zenoh.

Now that rmw_zenoh is Tier-1 in Kilted, we are committed to maintaining wire compatibility and will be extremely cautious about introducing any breaking changes in future released distributions. As mentioned, this backport to Jazzy was an exceptional case, given how actively rmw_zenoh was evolving leading up to Kilted.

We appreciate your understanding and continued feedback as you switch to ROS 2 and using rmw_zenoh for your robots!

4 Likes
6

Maybe pedantic but shouldn’t breaking changes only be introduced in major (or at least minor) versions, i.e., not in patches? Meaning, shouldn’t you be changing the version number to 0.3.0 now to stay consistent with semantic versioning? This would make it much easier for people who don’t see this announcement to debug their systems once they notice they are broken.

1 Like
7

Indeed, this is my position as well.

If someone installs ROS Jazzy, it should be compatible with other installations of ROS Jazzy. This sort of breaking change causes problems for those who installed at different times or through different means.

It may not be possible for users of ROS Jazzy to force conversion of their entire system to the new rmw_zenoh wire format. Standardizing on an official release is supposed to improve systems integration via standard interfaces, not pose a systems integration threat.

Performance optimization is nice, but it should not come at the cost of usability.

8

Thanks for the explanations, they help a lot. So it seems to me the major problem was the lack of communication when (before) the decision to backport the changes was first made.

There is probably also a confusion about the state of rmw_zenoh_cpp in Jazzy. From the posts made about it in the past I got the impression that rmw_zenoh_cpp on Jazzy is experimental in the sense that not everything has to work, or something can work suboptimally. I definitely haven’t seen the option of the RMW to receive breaking updates. But yeah, the version number is 0.x.y. Semver says

Major version zero (0.y.z) is for initial development. Anything MAY change at any time. The public API SHOULD NOT be considered stable.

So technically, no promise has been probably broken. Yet still, it leaves a bit bitter aftertaste. However, I see you take this lesson seriously, which is a good sign :wink: On the other hand, this has not been the first stable release breakage…

As I’ve stated above, the version is 0.x.y, which means anything can change anytime.

1 Like
9

I will elaborate more on this tommorrow, but I am missing something or rmw_zenoh: 0.1.2-1 in 'humble/distribution.yaml' [bloom] by ahcorde · Pull Request #46542 · ros/rosdistro · GitHub introduced a similar breaking change in ROS 2 Humble?

10

Ok, I think at this point that the problem also affects ROS 2 Humble, and will be soon affect more users once rmw_zenoh 0.1.2, see Wire Breaking Change in rmw_zenoh also for ROS 2 Humble between rmw_zenoh 0.1.1 and 0.1.2 ? · Issue #700 · ros2/rmw_zenoh · GitHub for details.

Perhaps it could make sense to do an announcement similar to this one for Humble?

11

@Yadunund How can I leverage this ?
Is it achieved in the standard way in zenoh or does rmw_zenoh require some tweaking ?

12

Just for a more in topic comment, the breaking release 0.2.5 announced in this announcement was actually released on ROS 2 Jazzy main channel on the 2025-06-29, see https://discourse.ros.org/t/new-packages-for-jazzy-jalisco-2024-07-29 (do not get confused by the title of the thread as initially I was, I guess it is a typo).