Improving the Developer Experience

General
,
1

Hey,

tldr: What are your pain points when working with ROS? What causes you friction? What could be done to improve your developer experience?

Disclaimer:
This is a discussion starter and in no way intended as an attack against any entity. It may read as a rant, but in reality that is just my personal insecurities shining through. I am sure that you - the reader - are doing a tremendous job contributing to this and the overall open source community. ROS in this context means the packages developed and used by this community and not just the core ROS itself. I am currently using ROS2 Galactic, but this discussion is not specific to that version.

Background:
Iā€™m working with ROS for about 3 years now with a tiny 3 people team and would consider myself more as a problem solver or high level system integrator than a proficient programmer. So please take everything with a healthy grain of salt. The project which I am currently developing is a low(ish) cost, but extendable service robot platform. This platform is basically an up-sized and glorified robot vacuum cleaner and does not include things like inverse kinematics / 3D motion planning (manipulators, legged robots) or large scale navigation (autonomous vehicles, last mile delivery, drones) which are other popular use cases for ROS-based systems.

Problem statement:
I often feel frustrated when working with ROS. A large part of it is probably my general workflow - more on that here - and maybe the skill gap I have, but the other part is the constant friction and the overpromise and underdeliver vibe I often get. This I would categorize as Developer Experience and would like to focus this discussion on. To be clear I am not looking for a ROS alternative out of the box robotics framework. What I am looking for is the nitty-gritty small annoyances causing friction and making you, the developer frustrated.

Some personal anecdotes (most ranty part):
Starting with RViz2, it is the most crashy software that I ever used. Topic visualizations often does not render (reasons other than the obvious policies). The displays panelā€™s left and right side dropdown arrows canā€™t be seen at once so you have to glitch your way to see the other side. Opening the Add popup panel freezes the visualization and tough luck combing through that anyway because there is no search function there. No ROS_DOMAIN_ID selector or integrated teleop tool.

Gazebo is the default choice for ROS sims. Maybe my needs are extremely niche, but I would expect a lidar plugin which actually rotates, meaning in real lidars with a slow speed (~10 RPM) the image distorts considerably based on velocity impacting navigation performance. The startup image will most likely glitch and startup time can vary from seconds to minutes depending if the models folder configured properly. The performance of gzclient is not great at all.

There is no proper player / browser / editor for rosbags (as of Galactic). (Iā€™m thinking a mp3 player like utility with selectable topics and markers for problems.) Recording is painful with the whole reliability policy pattern.

Iā€™ve yet to come across an odometry source which provides actual covariance and thus be viably used for sensor fusion.

Potential solution:
These annoyances can be mostly helped with proper bug reports and PRs. Documentation is another thing that can be improved.

Call to action:
I would like to hear your anecdotes and feedback. What causes friction when working on your ROS-based system? What are some quality of life improvements you came across in other frameworks which greatly benefited your developer experience?

7 Likes
2

May I humbly suggest that the people contributing to ROS do not have unlimited resources. I absolutely agree that there are areas that could be better, however, that does not mean there are resources to address those issues, or that your priorities are the priorities of the other contributors. Most of the current maintainers are spread pretty thin; the best way to improve the project(s) is to make meaningful contributions in the areas that you think need to be addressed.

Call to action:
I would like to hear your anecdotes and feedback. What causes friction when working on your ROS-based system? What are some quality of life improvements you came across in other frameworks which greatly benefited your developer experience?

Can you clarify this a bit? Are you planning to contribute improvements in this direction? Have you reached out to the maintainers to better understand how you can help these projects move forward in a meaningful way? Feature requests and bug reports are certainly helpful for triage purposes, but they donā€™t necessarily move things along.

5 Likes
3

For me personally, the main ā€˜pain pointā€™ is that ROS1 keeps working really well, and the robots that I use havenā€™t swapped over to ROS2 yet. So, similar to the python2.7-to-python3 shift: momentum is the main roadblock.

ROS2 is a big environment to play in, so like ROS1, it has a non-trivial learning curve. Unfortunately, to some extent that might just be a fact we have to deal with (in the same way learning a new programming language is a lot of work, but is worth it for the functionality). Documentation and examples certainly help, and thereā€™s more of them every day, but that takes people and time.

To offer something constructive: ROS is unique from a lot of other open-source projects in that it has a large and healthy online community (thanks largely to the work of Katherine Scott, and Iā€™m sure others that Iā€™m unaware of). Iā€™d bet that thereā€™s a lot of folks lurking on this thread that are willing to write tutorials but donā€™t know what beginners need/want. Would it be helpful to have a biweekly ā€˜request for documentationā€™ thread, to get some informal coordination of documentation-writing efforts?

Similarly, I can justify spending time contributing to packages that Iā€™m using every day, but to be honest itā€™s not on my mind until something breaks. Would developers make use of a ā€˜request for contributionsā€™ thread if one was available? Are there other strategies that donā€™t take too much administrative effort but that could potentially direct community effort?

4 Likes
4

I think Wendy Tan White, the CEO of Intrinsic, summarizes well what the main painpoint of ROS is. Quoting from her IEEESpectrum interview two months ago with Brian Gerkey:

ā€¦ Brian was finding at Open Robotics is that he was starting to get that pull to build to that level, and then ended up building loads of customized things on top of ROS when really what was needed was a more industrialized platform. But he didnā€™t want to force the ROS community into that alone. So, the way that we see this is that weā€™d be more like the Red Hat to the Linux of OSRF. And weā€™re actually going to carry on investing in OSRF, to give it more stability. I donā€™t think Brian would have wanted to do this unless he felt like OSRF was going to stay independent.

The problem is indeed the loads of customized things on top of ROS. Those things are introduced at some point but undermaintained heavily. Not surprising because tooling is a hard business and it is not the main business of roboticisits. Simply the resources were not enough for such overambitous projects. So, like this post says, the most frustration comes from the 2000ā€™s developer experience in 2020s. What we need a well-defined pub-sub interface and industry-standard tooling, which developed greatly in the last decade.

With the new structure and financial stability to steer the ship, I hope the OpenRobotics would force the ROS community as they have should have done yesterday. I am really excited what they will bring in terms of developer experience in this new period.

Btw do we have a roadmap that can be shared with the public?

5

For the subset of us interested in finding ways to contribute, I hope folks will (continue to) use the ā€œhelp wantedā€ tag on github issues to help us find them. (Or if thereā€™s a better mechanism, please do let us know!)

3 Likes
6

We noob ROS developers understand that, and would really like a safe place to say ā€œI am having trouble with thisā€ and not get reamed because someone else asked what in no way sounds like the same question and we could not understand how to apply the answer to that question to our question.

Asking a question should not cause anyone to imagine we are complaining about the heroic, guru level work that is being performed to create and maintain ROS - we are hurting and often are wondering if we are the only one with this issue, or if simply finding one of the thousands of packages available and figuring out how to configure the 20-30 options for the package even after being told ā€œthatā€™ll answer your questionā€.

My background - retired software engineer with life long hobby interest in personal robotics.

My learning issues:

  • not being smart enough to understand how to make my robot report accurate heading: canā€™t figure out how to configure EKF to fuse wheel encoders and IMU data. My math skills are just not there to understand covariance matrices.
  • not being smart enough to understand to optimize onboard RPi4 SLAM
7

I donā€™t think it was anyoneā€™s intention to do that in this thread. ROS Answers is designed to be the place youā€™re looking for.

8

Thanks for sharing that! That could be a very helpful resource, but it takes a bit of github skill to get to it and only looks at the core packages, and the most recent issue is from 3 weeks ago. Perhaps some aggregator could be created that looks at the top x% of ROS packages for issues, and then shares them on some external site? I donā€™t know enough about web dev or the github api to know how straightforward that would beā€¦

9

Yes, thank you - I am aware of and have participated in both ROS Answers and Robotics StackExchange. They are not always ā€œsafe spacesā€ for noobs.

10

Sometimes things are just hard (understanding the math inside a state estimator) and sometimes things are hard because no one has figured out how to fund fixing them. Robotics software combines many of these problems.

The hope of open source is that by sharing the cost we can reduce the second class of problem. The challenge though is sharing the cost is not something we are particularly good at. Just look at the number of active contributors to what we consider the ā€œcore of ROSā€ which is spread across 12 GitHub orgs, 104 repos, 356 packages, 494 cmake projects, and 98 python projects.

In many cases, I think we fight against the second class of problem (arbitrarily hard) because we have yet to get to the point where we actually spread the work of building these things across many people. How many people are active committers in those 104 repos should tell you something about how healthy we are at open source. (I donā€™t know this number, maybe someone has that stat handy, my perception is that it is less than 10)

There is also this perception that there is a vast innate gap in skill between those doing contributions and maintenance and those using the systems so it is fruitless to try to onboard more people to contribute. If that is actually what we believe then we should stop trying to make this open-source thing work. We should focus on improving the journey of new contributors and building a pipeline for making ROS something that is not just kept alive by the valiant efforts of a few.

Anyway, rant over. Like others said, if you find it hard to use, maybe fix it yourself. Contributions welcome or something like that!

7 Likes
11

I appreciate you calling this one out - itā€™s a great example of something trivial that makes the learning experience for a new user unnecessarily more difficult, and after that remains a minor annoyance.

That being said, it hasnā€™t frustrated me enough to even look if there is an issue on GitHub let alone attempting to fix it, so I canā€™t complain too much!

12

These sites should be safe spaces to ask questions. And you should not be getting ā€œreamedā€. I monitor both sites relatively closely and have not recently observed any attacks personal or otherwise. If you see or experience any of this please flag it on the site using the report button. Or feel free to directly message me about it and Iā€™ll take a look.

Note that there are questions which will get feedback and potentially closed. These efforts are necessary for the maintenance of the site. If you are new to the community you can still ask well formed questions. We do our best to provide good guidelines for how to ask questions. Help Center - Robotics Stack Exchange and Support - ROS Wiki

A few common examples of reasons for closing are: If a question is closed ā€œduplicateā€ and a pointer to another question is provided thatā€™s not an attack on you, but a pointer for you and anyone else who finds your question in the future a pointer to where thereā€™s already an answer to your question. That way we donā€™t end up answering the same question many different times, and if someone thinks of it the same way that you do and finds your question they also get the same answer.

And there has been a lot of thought put into the process. Including having standard responses: Copy-pastable comment text for common problems with questions? - Robotics Meta Stack Exchange These standard pieces of feedback point back to the guidelines for asking questions to try to help you revise your question to include the information that the community needs to be able to help you.

A lot of what youā€™ll find is that the more effort you put into asking a good question the faster and better an answer youā€™ll get back out. For example ask your question after doing rubber duck debugging and include everything that you previously summarized.

5 Likes
13

Iā€™ve had reasonable success with rqt_bag (both ROS1 and ROS2). It has yet to crash on me and I can get most browsing etc done. No editing though but I can live with that. I would love to have a better documented rosbag API in python but I got working what I needed to.

For me the biggest learning curve in transitioning from ROS1 ā†’ ROS2 was the unpredictable middleware performance and the change in build system from catkin ā†’ ament. Much of the middleware gotchas I experienced (and continue to experience) are due to lack of documentation and of examples.

5 Likes
14

+1 for this.
Im a newcomer to ROS.
I see the issue of various things being ā€œundermaintainedā€. So, I have difficulty understanding how the decision was made some years ago, to come up with a whole new set of APIs in ROS2ā€¦ but then not port widely used critical packages, as part of the ROS2 release. Such as ROS-industrial.

And worse yet, the decision to make a"ROS1" bridge. It may seem like a good short-term idea, but in the long term, its terrible. It negatively incentivises people to port packages.
A far better solution, in my opinion, would have been to put that effort into some kind of compability layer, or porting tools, that made it easier to bring the old code into direct ROS2 usability.
Itā€™s like the early days of the linux desktop wars with ms-windows emulation.
Sure, it made a lot of people short-term happyā€¦ but it put off linux desktop progress for maybe a decade.

The best thing for ROSā€™s future is a unified and well documented standard framework that can then be easily ported to any robot platform with a minimum of effort.
If it takes over a year for a large vendor such as Yaskawa, with their paid resources, to come out with ROS2 functionality for their robotā€¦ then seems like thereā€™s something major missing.

2 Likes
15

I have difficulty understanding how the decision was made some years ago, to come up with a whole new set of APIs in ROS2ā€¦ but then not port widely used critical packages

I think youā€™re conflating a few things here. There wasnā€™t a decision to just not port things, there was a decision to update the APIs (for reasons discussed elsewhere). Porting packages is then the responsibility of the package maintainers (as it is between ROS1 versions): a lot of the ROS-industrial packages are still on Melodic. This returns us to the problem of ā€˜who is willing to sit down and do the maintenance workā€™.

As an aside: I donā€™t really see why the problem is ā€œloads of customized things on top of ROSā€-- is the implication here that building on top of ROS is bad practice? That seems odd.

the decision to make a"ROS1" bridge ā€¦ negatively incentivises people to port packages

Yeah, thatā€™s an interesting point actually. But, here we are.

The best thing for ROSā€™s future is a unified and well documented standard framework that can then be easily ported to any robot platform with a minimum of effort

Are you describing ros_control and the ROS2 equivalent, ros2_control? Or something else? Keep in mind that there are many, many different needs, so there cannot be a one-size-fits-all solution.

If it takes over a year for a large vendor such as Yaskawa, with their paid resources, to come out with ROS2 functionality for their robotā€¦ then seems like thereā€™s something major missing.

Has Yaskawa been actively contributing to development of a ROS2 port? Their package is still on Kinetic and git history appears to be community-driven.

16
Largely off-topic, so click-to-expand if you still want to read it

I wonā€™t go into any details here, but:

this is off-topic here and doesnā€™t say anything about the state of ROS 2.

Pace of development of MotoROS2 neither was nor is affected by the subject(s) being discussed in this thread.

Large (industrial) OEMs have very different motivators from open-source communities and measure development cycles typically in years. And just to put it in perspective: the initial development of MotoROS1 (ie: the motoman_driver) took about 2.5 years, and included just the base functionality.

The reference is to MotoROS2, for which development does not take place in the repository you refer to.

Just to clarify: branching policy for those repositories dictates to only branch when absolutely necessary. As the readme states, those packages are perfectly usable on newer ROS 1 versions.

1 Like
17

The thread got a bit derailed. I think the point was to find weak points in the ROS ecosystem, things that are really painful but we got used to it because we see no hope of it getting fixed, like RViz.

I think one of the top annoyances in ROS2 is QoS. It really is important and necessary, but having to spend minutes in RViz trying to figure out the QoS settings of a topic because otherwise it wonā€™t show data is so annoying. Similarly writing subscribers. I know ros2 topic info -v prints the QoS settings, but when working with a lot of different QoS topics (and especially general subscribers) it is very tedious. ROS2 would benefit greatly from an auto-QoS subscriber, both inside and outside RViz. (I saw in some thread that the info is there in the DDS implementation but not propagated to the ROS layer as of right now.)

3 Likes
18

Hi!

From my perspective, I kind of agree to the point of ROS got too large to handle. Humble is ~1000 packages with very varying level of code quality, and since I tried to cross compile most of it, I think I can have an opinion on that.
One thing that really makes me wonder if ROS is the way to go for implementing an industrial grade robot are the quality issues. Although it is not a big issue, I came across a regression that is described here and I offered a PR here but for the past 3 months or so nobody even seems to have the time to say that this is bullshit and needs to be closed. The recently (finally!!!) addressed realtime issues might serve as another example.
That said, I had PR towards Microsoft that got merged 2 years in, so I still have not lost faith :slight_smile:

My point is: given the limited resources, would it make sense to focus on a ROS2 - Core, instead of adding all those packages that soak up so much resourcesā€¦ ?!

1 Like
19

That is why ROS 2 introduced the explicit quality level of packages. I hope that the higher the quality level, the better the support. Of course this still has some limits. But basically, if youā€™re using QL4- packages, youā€™re just on your own (with a high probability). But it also happens that some interesting package is provided by an individual who is not yet proficient enough to write good code, but is happy to merge any improvements.

6 Likes
20

I will be happy and grateful to help you with the best of my knowledge and abilities. Driven by desire to work with you out of curiosity and to contribute to the work you are doing. Thank you for your time and consideration.