ROS only simpler

General
1

Hello All,

I apologize if this is not the right forum for this, but I recently started on a project that will use ROS2 for IPC and inter-container communication.

For a little background, I have been a C++ developer for 20 years, and a C developer for 10 years before that. I have used MQTT on other projects, and fully appreciate what ROS2 offers.

My first impressions of ROS2 have not been good. Don’t get me wrong, I understand what ROS does, how it does it, and believe it is the right tool for the job, and in spite of the on-going difficulty in using ROS2, we are committed to using it.

Our trouble stems form the build environment that ROS2 dashing has. I have built the basic tutorials at https://index.ros.org/doc/ros2/Tutorials/, and after some trouble, they compile and all is well. The next step in my journey is to incorporate custom messages, and here is where the story has become unhappy. I started following the tutorial at: https://index.ros.org/doc/ros2/Tutorials/Rosidl-Tutorial/, and the rails came undone. This tutorial fails to build with cryptic errors. One of our partners put me in touch with their ROS/ROS2 expert, and he is helping me work through the build environment issues. We will solve these issues, but this is pain I can do without. My first MQTT project worked in 30 minutes from start to hello world (with my own custom payload). I have so far invested 2 days of my time and at least a few hours of others time, and still do not have a working basic example of a custom payload with ROS2, in fact, we have so-far been unable to get the above tutorial to run on my development machine.

From a new dev perspective, I have a few simple suggestions to hopefully help others who may follow in my steps:

First, The existing build environment is far over complicated for a simple tutorial. As noted, I can’t get the example programs to run because the build environment is broken. This is almost certainly not related to anything that is core to ROS2, and could easily have been avoided. The tutorials should include only a few files, and perform the command line compile by hand (avoid automated build tools: that’s like taking a bazooka to a fly). For a good example of how a tutorial should be: see https://cumulocity.com/guides/device-sdk/mqtt-examples/#hello-mqtt-c. Notice that this tutorial requires only gcc, and nothing else, it distills the example down to its most basic form. A similar tutorial for ROS2 would require only the gcc compiler and the translation script that generates the C and C++ header files for the custom messages.
Second, document the translation scripts as standalone programs so that people who wish to use their own build environment do not have to reverse engineer yours just to extract the custom message translation mechanism. You have a good build system for complex projects (at least I assume it is good as I don’t have it working yet), but it is grossly overkill for most projects, especially learning projects for people who are new to ROS, but have lots of experience as developers. I am in no way suggesting that your build system should be changed, just that the option to skip the build system be supported for those that don’t need/want it, and for the sake of simple tutorials.

1 Like
2

I know its not the point you’re trying to make but: can you tell us the “what was wrong” and “what was the error you got” with this one? Documentation can always be easily updated with your experiences here. If you find something confusing, others would too. Be the change you want to see, and all that.

Anyhow, to try to repeat back to you my understanding of your core issue:

  • You believe that the build environment is complex
  • Because its complex, some of the tutorials are unclear

Those are actually very different points than I would have expected someone new to ROS2 to make about the tutorials. That perspective is interesting. This is the right place to share it, thanks for getting a conversation started.

I would not consider the custom message tutorial to be on the basic-first-look for some of the issues you bring up. It is a little involved, but if you were to be going through the tutorials in order, the only new macro here is rosidl_generate_interfaces. The other ament things you would have run into already to get a feeling for in several tutorials before this point.

Those do a bit more than just that, but I understand the sentiment. I have a feeling that someone more knowledgeable than me will come along to explain why it would likely be more challenging to compile from a commandline for new users.

My higher-level opinion is that we functionally never do that. Reinforcing the tools and teaching habits from the get go is important. If you tell someone how to compile it with a commandline, you’re going to need to then make another tutorial about how to do it “for real”. Most users are going to be using ament/colcon to compile their systems, something I’d estimate in excess of 95%. Each additional reference of these tools someone can learn from has substantial value for reinforcing ideas. However, what doesn’t have value, is if they are poorly explained, presented, or ordered in tutorials causing the experience that you had.

I think that experience has alot of value and we’d all learn alot from a proposal from you about how to better explain these concepts and support new users (ie to move around the tutorials, explain some concepts better, or give more examples presented in another way).

In addition, adding an advanced tutorial explaining how to compile code without these tools could have value to an experienced developer like yourself that would rather not use the provided tooling.

Those 2 action items don’t resolve the specifics of your comments, but I think they resolve the spirit of where we can do better so that your comments won’t happen to new users because the build environment will be better presented and working out of the box.

3 Likes
3

The catch here, especially when comparing with MQTT, is that ROS is so much more than just IPC.

There was quite a bit of discussion about the build tool recently, which might help explain why things are the way they are. There is room for improvement, especially in the documentation, but we do have reasons for why things are like this.

If you are having problems, then I second the comments by @smac: Seeing those errors and hearing about the detail of the problems will go a long way to helping us find out specifically where we can improve.

1 Like
4

While I agree largely with what @smac and @gbiggs have written in their replies, I just wanted to make sure that @Eric_Schumann doesn’t get the impression we’re trying to ignore his comments or dismiss them with an appeal to tradition (ie: “this is just how we do things here”).

The thread @gbiggs linked to was the first thing that came to my mind as well, and I believe there are some good posts in it that @Eric_Schumann may want to read.

This is true, and may not be apparent when starting to work with this infrastructure.

1 Like
5

Absolutely. We know the tutorials are lacking, but there are so many tutorials and so much stuff we need to cover so hearing from @Eric_Schumann what specifically are the errors and problems will help us improve things.

1 Like
6

Yeah, I agree with the sentiments above. I don’t think it is a good idea to introduce something that is simpler, only to then backtrack to using the “real” tools.

But I do agree whole-heartedly that the tutorials are lacking, and making things “just work” is an important goal for us. So I would also like to know the particulars of your trouble, and how we can improve the build tools/code/documentation so that the next newcomer doesn’t run into them.

7

Follow up

Good news! I was able (with the help of one of our local CMake experts) to dig back to the root cause of the problems I was having with the build environment.

At one point, I had been working on just one of the included packages, and saw the CMakeLists.txt, and naturally thought to run cmake . followed by make. This worked fine for that 1 specific compile (except for build files being put in funny places), but after that, cmake had cached the build settings which conflicted with those that colcon was trying to use, and cmake just ignored the colcon setting to use the cache. To fix the problem, all that needed to be done was to clear the cmake cache (the cmake equivalent of make clean).

@gbiggs shared a link to a general discussion topic about moving ROS2 over to a pure CMake environment.I still contend that for a tutorial, the build environment is a distraction, however pure CMake would help to improve the situation a lot. CMake is quickly becoming the defacto standard for many platforms, and as such, would be easier for some of us to transition to. (Disclaimer: I work for Kitware).

I appreciate all of the responses, and to be clear, I am not necessarily expecting any particular action, and I am absolutely not suggesting that the default for ROS2 be anything other than the build environment as you have built it. For most (all?) new projects, I think that the current default build environment makes reasonable sense. That having been said, it is always a good idea to give users more flexibility when you can, especially developers, and I can think of several cases where the single build environment would be more hindrance than good. The first and most obvious is for a legacy project where they want to add ROS2. That project already has its own build environment, and porting that entire project to use colcon would be prohibitive at best, and would most likely make the change a non-starter. A second example would be conflicting with other projects that have incompatible build requirements. For example, incorporating anything involving GPGPU processing or multi-language environments generally already have their own build systems in place that, in some cases, do not play nice with CMake. Merging ROS into such a project would be a royal PITA. Granted, these other projects are as much or more at fault for picking a non-standard build system, but I think you see where I am going with this. Another use case is for embedded systems, where one of the packages in a system belongs on an embedded device. Due to the nature of cross compilation, it is almost always easier to have a standalone build system for the cross-compiled parts. As you can see, it is easy enough to keep separate workspaces for each of these examples, but under these examples, the colcon build system provides no real value added.

One thing I do want to be sure to convey is my appreciation for the documentation. In my experience good quality documentation is hard to write and hard to find, but the plethora of ROS/ROS2 documentation has been a happy exception to that norm. Being able to find most of the information I need without resorting to google-fu has been very refreshing.

Thank you again for the wonderful replies. You have built a really powerful tool in ROS/ROS2, and it is clear to me that no matter what decisions are made, there are dedicated and capable people working to make ROS2 better, and that is what really matters.

8

This is an interesting statement, because it is actually the opposite of how I think about the ROS build tools.

One point that I keep coming back to (and maybe not explaining well) is that tools like Make, CMake, SCons, etc are inherently single-project tools. That’s fine, that is their goal. But ROS is set up in a much more federated fashion, where you have many projects that you combine in different ways to achieve your goals. So while I wouldn’t stand in the way of anybody trying to improve the documentation and tools for the single CMake case, in my mind it just doesn’t fit in well with the federated nature of the ROS ecosystem.

That being said, I’m glad that you solved your problem. One thing we could add to the documentation is to make sure that your CMake environment is cleaned up before trying to build. For the record, what were the commands/actions that you had to take to clear our your environment?

9

For the record, what were the commands/actions that you had to take to clear our your environment?

I restored the package directory to the pre-cmake state. In my case, I accomplished this by deleting the package directory under src, and replaced it with the original from the tutorial. When I ran cmake from the package directory, it built all of the files and directories that colcon would have placed under /build, and put them in the local package directory instead. I had to remove those files (which included the cmake cache information). I probably only needed to remove the CMakeCache.txt, Makefile and the CMakeFiles directory, but it was easier to restart fresh in my case.

10

Python is a good example where they do this poorly. The usual intro tutorial is just how to print hello world to the terminal, it explains nothing about importing packages or pip or setting up requirements.txt and setup.py, which is of course how you do it “for real”

I hope you’ll forgive the sarcasm but I thought it was the most effective way to get my point across.

Most of arguments against making ROS simple enough to be able to have simple tutorials center around “but ROS is federated”, which is actually a sort of catch-22 because the build system enforces this federated model and when someone like OP tries to work outside of it, they get bit.

I’d argue that Python is also “federated”, in the sense that there’s a vibrant package ecosystem (which is one of the greatest things about Python and about ROS as well). I think Python/pip could provide a good model for how to do ROS packages and one of these days, when I find the time, I plan to work on a proposal for how ROS packages could work in a more pip-like way that is both friendly to new users but also decipherable for those who want to dive in “under the hood”.

1 Like
11

You mixing two completely separate parts of the process: the build system (CMake, Python setuptools, etc.) and the build tool (colcon) [see A universal build tool]:

  • The build system is in charge of processing a single package. In the case of CMake that usually boils down to using it as cmake / make / make install. The fact that ROS packages use CMake helper functions provided by ament_cmake are actually not relevant for this since they are an implementation detail when you look at how to invoke the build system. You can achieve the same with plain CMake - it will just take you more lines of CMake code.
  • The build tool colcon only determines in which order a set of packages needs to be built and what build system a package uses in order to decide how to build that package. colcon is very modular, if you want to use a build system which isn’t supported atm you can easily write an extension to teach colcon how to invoke that build system. E.g. CMake support is just an extension for colcon which implements the invocation logic mentioned above.

And as usually you can always mix-and-match: build some packages with colcon, then some manually (however you like), and then another set using colcon.

I don’t think this statement is true. When you want to cross compile any project which consists of more than a single monolithic piece you are facing the exact same challenges as for a native build. Therefore the benefits of a build tool apply just the same. Coincidentally just yesterday a tool to make cross compiling was announced here on which internally uses colcon for that very reason.

See my comment above which clarifies that the build system only has knowledge about a single package. So by definition can’t have anything to do with the federated model which means dealing with multiple packages.

And by design the build tool is optional to use. You are able to build all the packages of a ROS distribution just fine without a build tool. You “just” have to manually figure out in which order packages need to be build and how to build each individual package. So I hardly see where we enforce the build tool anywhere (except that doing that kind of work manually sounds like a huge amount of wasted time to me since it can be easily automated).

I think this comparison isn’t actually applicable. Afaik there is no way in Python to build and install a set of packages where you have the source code in a local directory - and especially do that by honoring their inter-dependencies.

1 Like
12

Great discussion! I just started ROS2 in November, I didnt have any ROS1 experience. I did find the build system complex even though I am an expert in cmake, but colcon is now working for me too and I like it…just takes time. I struggled like OP…let’s face it ROS is a big ecosystem and requires perseverance to master. I took a udemy ROS2 course, and just kept at it. I feel at ease with the build system now but had the exact same issues as OP on my first custom msgs build. I love cmake and now after only 2 months I really love ROS2…such an improvement over ROS1. I would say keep colcon in the mix, but second improving documentation and tutorials especially given in progressions. Should be spoon fed a bit in the beginning. API-like tech docs can get overwehelming for newbies (like I was).

13

See, but I do want actions. I think you bring up a good point in things not being as clear as they should be. Most (if not all) of those tutorials didn’t exist with I learned ROS2, so you have a unique perspective as not only a new user, but a competent software engineer. Most new users don’t come in with 10+ years of experience to draw from :wink: . I’d think we would all very seriously learn alot from your stream of consciousness on how to improve the first experience, if we take colcon/ament as a given.

But it sounds like you got yourself in this situation by going off the beaten path of the tutorials.

I agree with @Eric_Schumann. If you come from anywhere in C++ land prior to ROS, this stuff is a little weird to deal with. Once you own the fact that you’re just going to use ROS for the long-haul with multiple packages, its a worthwhile investment. But if you’re just wanting to dip your toes in or you’re really only trying to support 1 package, that perspective is very different. I don’t want to argue on the technology components here, mostly because I think my opinion is under-educated, and partially because I don’t really think its the point.

14

Howdy, I am also a newer ROS2 software engineer, I started picking it up about 6 months ago. I just recently had to implement custom messages and I found the tutorial lacking in one specific area that I am suggesting gets added.

The tutorial offers insight in creating a custom message within the same package, and also includes the option for interfacing with a message in an external package, however, the latter is coupled with the former. That means the cmakelists had both internal and external references for custom messages and it took a bit of extra work to determine what was needed for only referencing external custom messages. It would be useful to add an extra step that shows just that. I hope this suggestion was clear!

Also, while I am here, I was wondering if anyone had a simple answer to whether or not it is possible to compose node components that are in different packages, or even further, different ros workspaces. I dug around and could not find an answer, which leads me to believe that it is not possible, especially with the c++ build and compiling requirements. I could be missing something though, so if anyone has an answer, I would appreciate it!

Thanks,
Robert

15

That should work just fine. The only constraint I would see is that both shared libraries need to be compatible in terms of being loadable into the same process (e.g. not linking against the same dependency but with a different version which have conflicting symbols).

16

While I agree with the sentiment of “install the packages in your system and just use them”, I think your comparison between pip and colcon has flaws:

  • Best practice in Python these days (and in Ruby and a bunch of other languages) is to create a virtual environment for every project and every test execution and so on. This allows you to be certain about dependencies, not have to worry about what the system has installed, and keep things in general isolated. A virtual environment is effectively what a Colcon workspace is, in most respects, and this is typically how I use them - I have a bunch of bash (and now zsh functions and aliases that I use to activate workspaces, including the ROS distribution in use, depending on what I’m working on. It makes me just as happy to do this with C++ code as Python developers are with venv.

  • Python packages usually do not need to be built before you use them, they just need to be present. This means that you do not need to worry about ordering of builds beyond a simple depth-first approach used to grab depedencies from PyPI.

  • pip only has to deal with Python, not C++ and Java and Lisp and protentially half a dozen other languages being treated as first-class citizens in a single virtual environment (or system).

  • pip only works on a single package and its dependencies, not, as @dirk-thomas said, a bunch of packages at once.

17

Can you either submit a PR to the message tutorial or DM me a summary of what you think needs to be added so I can add it.

1 Like
18

I am also in the process of writing a ‘hello world’ tutorial, any feedback will be appreciated.

19

This distinction feels academic. If there were tutorials or use cases where ament_cmake was used by itself, then it might make sense, but from an outside perspective it appears to be designed to work hand-in-glove with colcon as a single unit.

Is this your idea of a reasonable workflow???

Another academic point that’s missing the spirit of the discussion. Saying the build tool is optional to use is like saying that using either Linux or Windows is optional. It’s also not true, the build system doesn’t generate the setup.bash file.

I meant it in the sense that if I wanted to learn how to write some Python code, I can go learn how to write Python code. If I later want to publish that code, I can learn how the packaging system works. With ROS, you can either learn how to package the code first and then how to write it, or learn them simultaneously. This is the main criticism, that the package and build system/tool get in the way of writing code, and that perhaps there’s another approach where you can write the code and package it too without having to mix these two things, like the way they do it in Python. Obviously I’m not literally suggesting that ROS move to the exact Python/pip model, just pointing it out as a potential inspiration for a potential rethink to the package system. While ROS has a system that works and is very popular, I think it could do more to get out of the user’s way and I think this would be beneficial to ROS in the long term.

4 Likes
20

It’s just following the single responsibility principle - one tool is doing thing - and only one thing: the build system processes one package, the build tool invokes the build system for each package in the needed order. Also since ament_cmake is only providing helper functions to make writing CMake project more convenient it can be used like any other CMake project - which is a very well established and documented process.

You are wrong about this. ament_cmake packages actually do generate setup files for the package they build. You can find them in the install prefix under share/<pkgname>. Obviously they only setup the environment necessary for that package. The rational is also straight forward: multiple packages shouldn’t try to write the same setup files in the root of a workspace - that was e.g. a major issue for catkin_tools in ROS 1. Instead a single package - namely ros_workspace - is responsible to produce these prefix-level setup files for e.g. the Debian packages (in which process colcon is not involved at all).

Please consider proposing specific improvements rather than talking in generalities. Otherwise it is just not very likely that something can / will be done about what you would like to see improved.

1 Like