ROS only simpler

While I agree largely with what @smac and @gbiggs have written in their replies, I just wanted to make sure that @Eric_Schumann doesn’t get the impression we’re trying to ignore his comments or dismiss them with an appeal to tradition (ie: “this is just how we do things here”).

The thread @gbiggs linked to was the first thing that came to my mind as well, and I believe there are some good posts in it that @Eric_Schumann may want to read.

This is true, and may not be apparent when starting to work with this infrastructure.

1 Like

Absolutely. We know the tutorials are lacking, but there are so many tutorials and so much stuff we need to cover so hearing from @Eric_Schumann what specifically are the errors and problems will help us improve things.

1 Like

Yeah, I agree with the sentiments above. I don’t think it is a good idea to introduce something that is simpler, only to then backtrack to using the “real” tools.

But I do agree whole-heartedly that the tutorials are lacking, and making things “just work” is an important goal for us. So I would also like to know the particulars of your trouble, and how we can improve the build tools/code/documentation so that the next newcomer doesn’t run into them.

Follow up

Good news! I was able (with the help of one of our local CMake experts) to dig back to the root cause of the problems I was having with the build environment.

At one point, I had been working on just one of the included packages, and saw the CMakeLists.txt, and naturally thought to run cmake . followed by make. This worked fine for that 1 specific compile (except for build files being put in funny places), but after that, cmake had cached the build settings which conflicted with those that colcon was trying to use, and cmake just ignored the colcon setting to use the cache. To fix the problem, all that needed to be done was to clear the cmake cache (the cmake equivalent of make clean).

@gbiggs shared a link to a general discussion topic about moving ROS2 over to a pure CMake environment.I still contend that for a tutorial, the build environment is a distraction, however pure CMake would help to improve the situation a lot. CMake is quickly becoming the defacto standard for many platforms, and as such, would be easier for some of us to transition to. (Disclaimer: I work for Kitware).

I appreciate all of the responses, and to be clear, I am not necessarily expecting any particular action, and I am absolutely not suggesting that the default for ROS2 be anything other than the build environment as you have built it. For most (all?) new projects, I think that the current default build environment makes reasonable sense. That having been said, it is always a good idea to give users more flexibility when you can, especially developers, and I can think of several cases where the single build environment would be more hindrance than good. The first and most obvious is for a legacy project where they want to add ROS2. That project already has its own build environment, and porting that entire project to use colcon would be prohibitive at best, and would most likely make the change a non-starter. A second example would be conflicting with other projects that have incompatible build requirements. For example, incorporating anything involving GPGPU processing or multi-language environments generally already have their own build systems in place that, in some cases, do not play nice with CMake. Merging ROS into such a project would be a royal PITA. Granted, these other projects are as much or more at fault for picking a non-standard build system, but I think you see where I am going with this. Another use case is for embedded systems, where one of the packages in a system belongs on an embedded device. Due to the nature of cross compilation, it is almost always easier to have a standalone build system for the cross-compiled parts. As you can see, it is easy enough to keep separate workspaces for each of these examples, but under these examples, the colcon build system provides no real value added.

One thing I do want to be sure to convey is my appreciation for the documentation. In my experience good quality documentation is hard to write and hard to find, but the plethora of ROS/ROS2 documentation has been a happy exception to that norm. Being able to find most of the information I need without resorting to google-fu has been very refreshing.

Thank you again for the wonderful replies. You have built a really powerful tool in ROS/ROS2, and it is clear to me that no matter what decisions are made, there are dedicated and capable people working to make ROS2 better, and that is what really matters.

This is an interesting statement, because it is actually the opposite of how I think about the ROS build tools.

One point that I keep coming back to (and maybe not explaining well) is that tools like Make, CMake, SCons, etc are inherently single-project tools. That’s fine, that is their goal. But ROS is set up in a much more federated fashion, where you have many projects that you combine in different ways to achieve your goals. So while I wouldn’t stand in the way of anybody trying to improve the documentation and tools for the single CMake case, in my mind it just doesn’t fit in well with the federated nature of the ROS ecosystem.

That being said, I’m glad that you solved your problem. One thing we could add to the documentation is to make sure that your CMake environment is cleaned up before trying to build. For the record, what were the commands/actions that you had to take to clear our your environment?

For the record, what were the commands/actions that you had to take to clear our your environment?

I restored the package directory to the pre-cmake state. In my case, I accomplished this by deleting the package directory under src, and replaced it with the original from the tutorial. When I ran cmake from the package directory, it built all of the files and directories that colcon would have placed under /build, and put them in the local package directory instead. I had to remove those files (which included the cmake cache information). I probably only needed to remove the CMakeCache.txt, Makefile and the CMakeFiles directory, but it was easier to restart fresh in my case.

Python is a good example where they do this poorly. The usual intro tutorial is just how to print hello world to the terminal, it explains nothing about importing packages or pip or setting up requirements.txt and setup.py, which is of course how you do it “for real”

I hope you’ll forgive the sarcasm but I thought it was the most effective way to get my point across.

Most of arguments against making ROS simple enough to be able to have simple tutorials center around “but ROS is federated”, which is actually a sort of catch-22 because the build system enforces this federated model and when someone like OP tries to work outside of it, they get bit.

I’d argue that Python is also “federated”, in the sense that there’s a vibrant package ecosystem (which is one of the greatest things about Python and about ROS as well). I think Python/pip could provide a good model for how to do ROS packages and one of these days, when I find the time, I plan to work on a proposal for how ROS packages could work in a more pip-like way that is both friendly to new users but also decipherable for those who want to dive in “under the hood”.

1 Like

You mixing two completely separate parts of the process: the build system (CMake, Python setuptools, etc.) and the build tool (colcon) [see A universal build tool]:

  • The build system is in charge of processing a single package. In the case of CMake that usually boils down to using it as cmake / make / make install. The fact that ROS packages use CMake helper functions provided by ament_cmake are actually not relevant for this since they are an implementation detail when you look at how to invoke the build system. You can achieve the same with plain CMake - it will just take you more lines of CMake code.
  • The build tool colcon only determines in which order a set of packages needs to be built and what build system a package uses in order to decide how to build that package. colcon is very modular, if you want to use a build system which isn’t supported atm you can easily write an extension to teach colcon how to invoke that build system. E.g. CMake support is just an extension for colcon which implements the invocation logic mentioned above.

And as usually you can always mix-and-match: build some packages with colcon, then some manually (however you like), and then another set using colcon.

I don’t think this statement is true. When you want to cross compile any project which consists of more than a single monolithic piece you are facing the exact same challenges as for a native build. Therefore the benefits of a build tool apply just the same. Coincidentally just yesterday a tool to make cross compiling was announced here on which internally uses colcon for that very reason.

See my comment above which clarifies that the build system only has knowledge about a single package. So by definition can’t have anything to do with the federated model which means dealing with multiple packages.

And by design the build tool is optional to use. You are able to build all the packages of a ROS distribution just fine without a build tool. You “just” have to manually figure out in which order packages need to be build and how to build each individual package. So I hardly see where we enforce the build tool anywhere (except that doing that kind of work manually sounds like a huge amount of wasted time to me since it can be easily automated).

I think this comparison isn’t actually applicable. Afaik there is no way in Python to build and install a set of packages where you have the source code in a local directory - and especially do that by honoring their inter-dependencies.

1 Like

Great discussion! I just started ROS2 in November, I didnt have any ROS1 experience. I did find the build system complex even though I am an expert in cmake, but colcon is now working for me too and I like it…just takes time. I struggled like OP…let’s face it ROS is a big ecosystem and requires perseverance to master. I took a udemy ROS2 course, and just kept at it. I feel at ease with the build system now but had the exact same issues as OP on my first custom msgs build. I love cmake and now after only 2 months I really love ROS2…such an improvement over ROS1. I would say keep colcon in the mix, but second improving documentation and tutorials especially given in progressions. Should be spoon fed a bit in the beginning. API-like tech docs can get overwehelming for newbies (like I was).

See, but I do want actions. I think you bring up a good point in things not being as clear as they should be. Most (if not all) of those tutorials didn’t exist with I learned ROS2, so you have a unique perspective as not only a new user, but a competent software engineer. Most new users don’t come in with 10+ years of experience to draw from :wink: . I’d think we would all very seriously learn alot from your stream of consciousness on how to improve the first experience, if we take colcon/ament as a given.

But it sounds like you got yourself in this situation by going off the beaten path of the tutorials.

I agree with @Eric_Schumann. If you come from anywhere in C++ land prior to ROS, this stuff is a little weird to deal with. Once you own the fact that you’re just going to use ROS for the long-haul with multiple packages, its a worthwhile investment. But if you’re just wanting to dip your toes in or you’re really only trying to support 1 package, that perspective is very different. I don’t want to argue on the technology components here, mostly because I think my opinion is under-educated, and partially because I don’t really think its the point.

Howdy, I am also a newer ROS2 software engineer, I started picking it up about 6 months ago. I just recently had to implement custom messages and I found the tutorial lacking in one specific area that I am suggesting gets added.

The tutorial offers insight in creating a custom message within the same package, and also includes the option for interfacing with a message in an external package, however, the latter is coupled with the former. That means the cmakelists had both internal and external references for custom messages and it took a bit of extra work to determine what was needed for only referencing external custom messages. It would be useful to add an extra step that shows just that. I hope this suggestion was clear!

Also, while I am here, I was wondering if anyone had a simple answer to whether or not it is possible to compose node components that are in different packages, or even further, different ros workspaces. I dug around and could not find an answer, which leads me to believe that it is not possible, especially with the c++ build and compiling requirements. I could be missing something though, so if anyone has an answer, I would appreciate it!

Thanks,
Robert

That should work just fine. The only constraint I would see is that both shared libraries need to be compatible in terms of being loadable into the same process (e.g. not linking against the same dependency but with a different version which have conflicting symbols).

While I agree with the sentiment of “install the packages in your system and just use them”, I think your comparison between pip and colcon has flaws:

  • Best practice in Python these days (and in Ruby and a bunch of other languages) is to create a virtual environment for every project and every test execution and so on. This allows you to be certain about dependencies, not have to worry about what the system has installed, and keep things in general isolated. A virtual environment is effectively what a Colcon workspace is, in most respects, and this is typically how I use them - I have a bunch of bash (and now zsh functions and aliases that I use to activate workspaces, including the ROS distribution in use, depending on what I’m working on. It makes me just as happy to do this with C++ code as Python developers are with venv.

  • Python packages usually do not need to be built before you use them, they just need to be present. This means that you do not need to worry about ordering of builds beyond a simple depth-first approach used to grab depedencies from PyPI.

  • pip only has to deal with Python, not C++ and Java and Lisp and protentially half a dozen other languages being treated as first-class citizens in a single virtual environment (or system).

  • pip only works on a single package and its dependencies, not, as @dirk-thomas said, a bunch of packages at once.

Can you either submit a PR to the message tutorial or DM me a summary of what you think needs to be added so I can add it.

1 Like

I am also in the process of writing a ‘hello world’ tutorial, any feedback will be appreciated.

This distinction feels academic. If there were tutorials or use cases where ament_cmake was used by itself, then it might make sense, but from an outside perspective it appears to be designed to work hand-in-glove with colcon as a single unit.

Is this your idea of a reasonable workflow???

Another academic point that’s missing the spirit of the discussion. Saying the build tool is optional to use is like saying that using either Linux or Windows is optional. It’s also not true, the build system doesn’t generate the setup.bash file.

I meant it in the sense that if I wanted to learn how to write some Python code, I can go learn how to write Python code. If I later want to publish that code, I can learn how the packaging system works. With ROS, you can either learn how to package the code first and then how to write it, or learn them simultaneously. This is the main criticism, that the package and build system/tool get in the way of writing code, and that perhaps there’s another approach where you can write the code and package it too without having to mix these two things, like the way they do it in Python. Obviously I’m not literally suggesting that ROS move to the exact Python/pip model, just pointing it out as a potential inspiration for a potential rethink to the package system. While ROS has a system that works and is very popular, I think it could do more to get out of the user’s way and I think this would be beneficial to ROS in the long term.

4 Likes

It’s just following the single responsibility principle - one tool is doing thing - and only one thing: the build system processes one package, the build tool invokes the build system for each package in the needed order. Also since ament_cmake is only providing helper functions to make writing CMake project more convenient it can be used like any other CMake project - which is a very well established and documented process.

You are wrong about this. ament_cmake packages actually do generate setup files for the package they build. You can find them in the install prefix under share/<pkgname>. Obviously they only setup the environment necessary for that package. The rational is also straight forward: multiple packages shouldn’t try to write the same setup files in the root of a workspace - that was e.g. a major issue for catkin_tools in ROS 1. Instead a single package - namely ros_workspace - is responsible to produce these prefix-level setup files for e.g. the Debian packages (in which process colcon is not involved at all).

Please consider proposing specific improvements rather than talking in generalities. Otherwise it is just not very likely that something can / will be done about what you would like to see improved.

1 Like

dirk,

I have three separate proposals that all relate.

First, At the top level of the tutorials, add a section that explains two key ROS2 principles: ROS2 uses colcon to handle package management and simplify the installation of ROS2 based projects. The second thing that needs to be explained is that ROS2 is NOT strictly a messaging protocol. It is the basis for an entire operating system (to accommodate machines that do not have an OS already). This operating system operates based on callbacks where the assumed behavior is that everything happens as a reaction to stimulus. When an external stimulus is lacking, timers are provided to generate that external stimulus.

Second, Provide and document the ros message processors (ros_idl_generator_xxx) as separate standalone utilities, and setup the early tutorials to use these instead of the colcon build system.

Third, Provide further tutorial demonstrate how to integrate the simple examples into a full fledged ROS build using colcon.

I would be happy to at least review any new tutorials, and when my current project finishes up, I could probably help write some of them as well.

Thanks,

Eric Schumann

1 Like

I think that sounds like a useful Advanced tutorial.

Very much disagree. The discussion above on Ament / Colcon does not change the fundamentals that they are the default and widely accepted toolset for ROS2. >90% of developers will be working with them. I believe that your and @actinium226 issues are very genuine and should be properly addressed with advanced documentation and examples without them however that is atypical for the majority of users. Having some note in the very early tutorials with links what you both describe would give folks like yourselves the opportunity to go down the road-less-traveled makes alot of sense. However, education conventions usually stipulates a concept needs to be brought up at least 3 times before its solidified in a students’ mind. The importance of additional exposure for new ROS users (and new linux/programmers) to the toolset cannot be overstated.

I think we would all very much so like that.

Please don’t use sarcasm in forum posts. They are easily misunderstood. The tone that you intend is often lost when translated to text and is especially hard to understand for non-native language readers.

Indeed they are designed to work well together. But that doesn’t mean that they’re not useful separately.

This distinction may seem academic to you but this distinction is quite important. There are many use cases where we leverage this distinction. The first and actually most common one is that things like all the debian packages build and used every day simply use the pure build system and don’t invoke any build tool. I believe that all the other packaging mechanisms for distribution also leverage the build system directly and don’t use a build tool. All of these mechanisms effectively have their own infrastructure that replaces the need for a build tool. When you’re compiling from source having the build tool is much more important to deal with all the different elements on your system and enable you to keep your workspace up to date. We don’t have the documentation of how to use the build system without the build tool in our core tutorials as it greatly increases the complexity of what we need to teach. That doesn’t mean there aren’t active use cases.

As an aside I don’t think it’s useful to go into detail here about what actually ament_cmake is. Please see the linked documentation. But it’s really just a cmake library to make writing CMakeLists.txt more compact. You’re just invoking CMake to build the packages.

The other place where the separation of the build tool and the build system is important is when you’re crossing languages. We support packages using multiple languages with their own build system. The build tool iterates over each of the packages and invokes it’s build system in the right sequence. To that end if we didn’t have the separation between the build system and the build tool then we couldn’t integrate packages across multiple languages.

You look to be taking this out of context. This is in reply to a comment that said that the external project to be integrated with had a system that’s unable to be integrated. If you start with that as a premise, this would be the only approach and it would work.

And in practice this is exactly how almost every system is actually built. You got through a sequence of tools and buildsystems to install layer on top of layer onto your system until you have all the features that you want.

As I pointed out above the build tool is optional and there are active use cases which are working not using it. In addition to the use cases I highlighed above there are also projects that are simply importing and building ROS packages using just the CMake/ament_cmake functionality.

I’m not sure what gave you this impression about the setup scripts, but it’s incorrect. Setup files are generated for each package by the build system, ament_cmake, if you test out a single package by doing your usual cmake; make; make install that there will be setup files generated. However the convenient workspace level setup files that aggregate all the packages in the correct topological order are generated by the build tool because it’s the only thing that knows the topological order. And if you don’t want to use a build tool, then you need to have something else to replace that aggregation logic yourself.

I think that jumping from learning how to package software at the same time as learning the new API is more complicated that doing one and then the other to “a potential rethink of the packaging system” is a bit of a non-sequitor.

I agree that we could do some of our basic tutorials outside the concept of a package with the goal of teaching separate elements simultaneously. This can allow you to keep the tutorials more focused. As has been pointed out earlier in this thread though there’s a cost to teaching people outside the standard development practices in that you’ll teach them one way to start and then later tell them, “don’t do it the way that we just taught you” because it’s bad practice. That will get people into bad practices.

You assert that the “package and build system/tool get in the way of writing code”. This is not connected to your previous line of though, it’s jumping from learning these two things together is potentially confusing, to stating that packaging gets in the way of developing code. I strongly disagree with this. Packaging is a very important tool for good software engineering. Purely writing lines of code is great, but what makes the ROS ecosystem vibrant is how much and how easily people can share code. And the first priority for making software shareable is to provide enough metadata for other users to be able to reuse the code. This is exactly the metadata that you need to give to the package. Likewise the package infrastructure makes you think about your dependency structure and what sort of modularity you can support. And we have a lot of tools for testing, linting, and distributing packages.

In your development you’re welcome to do anything you want in your CMakeLists.txt and you’re welcome to not create and setup a package.xml if you just want to leverage the functionality that ROS provides. But as soon as you want to share your code with others. You’ll want to export the appropriate .cmake files to support downstream users. And declaring your dependencies appropriately in your package.xml will make it much easier for users to verify that they have all the necessary dependencies satisfied.

You mention that ROS has a working system that’s popular, but then imply that the packaging is getting in the way of the system. However small modular packages are generally preferred due to their siginifcantly easier maintenance. This was highlighted in the recent guidelines for architecting ROS Guidelines on how to architect ROS-based systems

There’s always space for improvement in any system. Many times people learning a system can get frustrated and upset and believe that there must be a better way to do something. And as I just stated there usually is. But in these projects there’s a lot of history and built up functionality and experiences that have shaped the system over many times years of development. Proposing a general “rethink” of the system is not particularly helpful. If there are particular pain points that can be improved we can focus on improving those. If we run into blockers then maybe we need to consider structural changes. It would really help to focus on what you want to happen instead of general complaints that the current system doesn’t work for you.

Indeed there’s always multiple ways to use any tool and we want to make sure to focus on the one that we recommend most people use. There will always be power users who divert from the beaten path, but it’s counter productive to try to teach new users all of the power options before they understand how to put together a whole system. This approach is much more rewarding and helps with getting people engaged quickly which is what they need to have a good experience and come back to learn more.

I’m sorry that you spent all that time working on this tutorial. As is called out at the top of that tutorial you’ve gotten ahead of our documentation, the tutorial starts with,

INCOMPLETE: this is a draft of an upcoming tutorial for creating and using custom ROS interfaces.
Disclaimer: The code provided is to support the explanation, it is likely outdated and should not be expected to compile as is"

This explains why you’ve likely been having an issue getting it to compile on your system. Since that tutorial is incomplete it’s likely more productive to find an example package as close to what you want to do to follow. In this case I’d recommend looking at a small representative package such as example_intefaces and examples

If others run into these sort of disclaimers in the future please take a moment to search and if nothing found ask on https://answers.ros.org to see if there might be another example you could follow or if there’s more up to date resources or an example you could follow. We have a large helpful community that can answer well crafted questions quite quickly. There are dozens of packages that declare their own messages that we could point you to as examples in the mean time until we have time to fill in that tutorial.