Handle unique parameters for robot instances

[AGummyBear]

This is a problem I’ve run into multiple times, and I don’t think I’ve found “The Solution” (if one even exists). There are many factors related to your own codebase and development style, as well as organizational needs, robot usage patterns, etc. that can impact what solution will work well for you. With that said, I’ll try to share some ideas that I’ve found work reasonably well.

As a note, I have only approached this problem with ROS1, so not everything may apply similarly.

Both organizations that I have worked on this problem at have used debian package based deployment (as opposed to docker). I am also personally a fan of mono-repo whenever possible (I mention this because the suggestions of some others to use different repos for each configuration make me shudder a bit). In this scenario, I have found it useful to use separate ROS packages for configuration files and launch files for each robot platform/model. I like to style my launch files like this:

• Top level launch file
  • include launch file
    • include launch file 1
      • default params for 1 (native or included)
    • include launch file 2
      • default params for 1 (native or included)
    • …
    • include launch file N
    • load params 1 for
    • load params 2 for
    • …
    • load params N for

It’s important that the param loading is last.

What this effectively does is let you have launch files for individual nodes that include default parameters (making the launch file useful standalone and providing documentation of the parameters), and then override those parameters in files that are unique to your specific platform that is getting loaded.

There are additional cases where you may need parameters unique to individual robots (sensor and peripheral serial numbers, or robot-specific tuning params, etc). To handle this, we use a central webserver that stores robotic-specific configuration parameters as an environment variable file. Whenever a robot comes online, it syncs the file to its system. When the software starts up, the file is sourced into the that runtime environment. The environment variables are also used to select the correct top-level launch file (basically, specifying the , above).

Using this process, we have

• Source control for all platform-specific configurations
• Centralized server for all robot-specific configurations
  • This is advantageous for feature flags and other real-time param changes without a full software deploy
  • This also makes it easy for others in the organization to see and make minor configurations to a robot, without needing to be git-savvy
• Hierarchical launch file structure that takes advantage of overrides

Some Challenges:

• Tracing which value is used for a particular parameter (default in code at initialization? default in code from NodeHandle::param()? value in launch file? value in param file? value in param override file? value on server config file?. Ideally, you could just look at the lowest-level config file and see, but you don’t always override everything. If you make a mistake with parameter naming (or namespacing), this can also be misleading.
• Adding a new parameter or modifying the value of a parameter can require touching a lot of files
• Guaranteeing that a parameter was found
  • To combat this, we have started to use a helper function that wraps standard ROS param loading that throws an exception if a parameter is not found. It’s not perfect, but it has helped us catch things quickly in simulation and automated tests.

[jak]

It was very interesting to hear that you have had very similar problems related to parameters as we have had at Karelics. We have been recently developing a solution that satisfies our main needs related to this topic.

We have implemented a ROS 2 package that solves this problem in a similar way as @Hugal31, @chfritz and @johnjamesmiller have shown. Also, many of the requirements that have been brought up in this thread and were summarized by @johnjamesmiller are mainly satisfied. But there are still many open questions, limitations and improvements we would like to make to this package.

We are currently considering open sourcing this package, to help to solve the problem and to possibly get some community support in developing the solution further. If we do so, would you be interested to try it out?

Parameter files
The ROS 2 package provides two main features:

1. Dynamic YAML variables - Adds a support for variables and placeholders in YAML files, which made our launch files much less verbose and more readable. We no longer have to depend to Nav2’s ReWrittenYaml or our custom solutions to resolve env variables, ROS package paths, math expressions, etc. in the launch files.
2. YAML file overlaying - Allows us to have parameter overrides for each device (physical robot) and model (robot series). These overrides can be applied to the default parameter file that is read directly from the ROS package. Even though we currently support just these 3 layers, it will be later on easy to add more and even customize the number of layers based on the user needs.

The parameter override files exist outside of the ROS packages in a predefined directory. This makes it easy to get the resolved YAML file in the launch file using custom syntax, and then pass the resolved file directly to the ROS node:

from param_configuration.configuration import get_resolved_yaml

resolved_yaml = get_resolved_yaml("config://nav2_bringup/nav2_params.yaml")

Having a ROS package to handle the dynamic variables and file overlaying makes the solution also usable offline.

Versioning of the parameter files
We store these parameter files in a separate Git repository, from which they can be easily deployed to the robots. Ansible is used to deploy, but it can be done basically with any tool chosen by the user.

In this configuration repository, we also store Docker compose files, which ensures that we use the correct image versions with the correct set of parameters. It is easy to rollback any unwanted changes, switch between versions and have separate branches for development and prototyping.

Since the parameters exist completely outside of our Docker containers, it also allows us to update any device or model -specific configurations without rebuilding the images or ROS packages.

Let me know what you think about open sourcing this!

[Hugal31]

That would be awesome!

We are not using ROS2 yet (but soon™), but I’d be very interested to see how it is working.

[tnajjar]

I would be very interested in trying it out and contributing back to it if it provides a good starting point!

[jak]

Great! I’ll let you know here soon once we have proceeded further with it.

[johnjamesmiller]

@jak I really like how you have the docker-compose and configs in the same repo to keep the configs coupled to the code version. Has this approach been full proof or any shortcomings?

Any shortcomings using ansible to deploy configs that you have uncovered?

For those looking to get started with ansible I’d love your feedback on how I can improve this:
GitHub - johnjamesmiller/docker-ansible-ros: running ansible in a docker to administer ROS2 based systems

[jak]

We have been using our current system for some months now. So far storing the docker files in the same repo with the parameters and deploying using Ansible has been working great; we didn’t yet find any shortcomings related to these. But for sure there are other improvements we would still like to make. I can try to gather them to a roadmap in the repository and list there also all the open questions we have.

[jak]

We have now open sourced our Parameter Configuration package, which addresses many of the issues discussed here. We’ve included two tutorials that guide you through its usage step by step.

GitHub - Karelics/param_configuration: The tools to make ROS 2 parameter...

While the package is currently implemented for ROS 2, adapting it for ROS 1 should be rather straightforward (@Hugal31). However, given that ROS 1 is nearing its end of life, we’ve decided not to provide official support for it. Instead, we encourage users to migrate to ROS 2

The package is still in an experimental stage, so we’d love to hear about your experiences with it and receive feedback on how we can make it even better.