ROS2 "Global Parameter Server" Status

Hi All,

We did see some articles about ros2 global parameter server as followings,

TSC minutes 2019-06-20

Configuration management system. We see these use cases which ROS 2 parameters do not satisfy:

  • Global configuration that many nodes share (e.g. RTOS priorities, vehicle dimensions, …)

ROS answer

It would be possible to create a node called the “global parameter server” which allows anyone to set
parameters on it and read parameters from it. We’ve discussed implementing this and even providing
API’s in rclcpp and rclpy to make is easy to use it, but we’ve not had time to do that.

we understand that

  • ROS2 Parameter is combined with Node. This is actually anti-centralized(distributed) system architecture which ROS2 targets.
  • ROS2 Parameter Server can be constructed just as one node with parameter APIs. (more like “write your own parameter server”)

but wondering if any development is on-going or not?
since above article tells that there seems to be a plan to implement API’s in rclcpp and rclpy.

could anybody share the current status or requirement discussion?

thanks in advance,
Tomoya

1 Like

I wonder how different this would be from the parameters_blackboard example contained in demo_nodes_cpp

@alsora

I wonder how different this would be from the parameters_blackboard example contained in demo_nodes_cpp

So do I.

there seems to be requirement and API’s, not actually server node.
so we would like to know the requirement and API’s what has considered.

thanks,
Tomoya

FYI I opened a ticket on Github about adding some functionalities to the ROS2 parameters.

It seems to me we could promote the parameter_blackboard from demos to be a standard component, in a package of its own. With it living in demos, it sounds like it isn’t supposed to be used in a real robot.

Parameter_blackboard would work for us as is.

That is ok by me. I hesitated to do it at the time (I wrote the parameter_blackboard demo) just because I felt like it needed more documentation and probably a tutorial for which I didn’t have time.

Also, you could image that there could be a C++/Python “global parameter API” in a separate package which handles the communication and discovery of the “global parameter server/blackboard” for you, and which only worked if you also ran the global parameter server separately (sort of like roscore from ROS 1). Which is something else I thought about doing rather than just literally just putting that demo into rclcpp or something.

We could really use some help getting this use case better supported. :+1:

@crdelsey
@wjwwood

It seems to me we could promote the parameter_blackboard from demos to be a standard component, in a package of its own. With it living in demos, it sounds like it isn’t supposed to be used in a real robot.

i do agree on this.

also i would like to hear about parameter lifetime or persistent parameter.
i do understand that ros_parameters design, but global parameter server would need to support the persistent parameter in general.

parameters are modified in runtime and cache it into persistent volume as well. and next boot or next re-spawn, modified parameter will be loaded at initialization. (lifetime is dependent on use case, sometimes system lifetime, sometimes node lifetime.)

could you share your thoughts on this?

thanks,
tomoya

@tomoyafujita

I feel like there may be lots of different ways that people want to make parameters persistent. You mentioned how parameters have different lifetimes. Others may want to store parameters in a database instead of a file perhaps.

Rather than try to build all this into the parameter_blackboard, maybe it would be better to create a separate node that stores the parameters.

Right now, the parameter_blackboard can load parameters from a YAML file. A tool like is presented in https://github.com/ros-planning/navigation2/pull/902 can dump all the current values to a YAML file to be used the next time the parameter_blackboard is loaded.

Alternatively, maybe the base parameter_blackboard could have one simple storage mechanism. People can then extend that example to create packages with other types of persistence

@crdelsey

thanks for sharing your thoughts, good to know that.

there are a lot of use cases, and we cannot support everything as generic. always trading-off. i was just calling for the ideas for this stuff, appreciate it.

thanks,
tomoya

As mentioned in this thread there are many different aspects of parameters that add significant complexity. Depending on your application there are many different levels of persistence that might be necessary, for some ephemeral is fine, for others there should be backups mechanisms to recover state following hardware failures. Similarly there are parameters that need to be validated within ranges or enums, or otherwise validated sometimes with runtime constraints.

Generic nodes and blackboards are very convenient but they lead to unclear ownership and lack the ability to define semantics in documentation. In particular the lifecycles needs of different parameters may be quite different. For example if you have a parameter that needs high persistence and one that doesn’t want it, then you now need to be able to set policies per parameter on the generic parameter server. And potentially try to tie their lifetimes and data management correctly to external events it starts to get much more complicated.

Whereas if we provide good tools for managing parameters within nodes the data can live with the nodes which own the data. A simple example is setting a parameter on a sensor, the parameter is limited by the capabilities of the sensor. The parameter must be validated by the sensor driver, so there needs to be a mechanism for the driver to get callbacks for the changes and then maybe revert the change to the global parameter, and this can get into fighting updates for the state of the parameter. Also if you do something like plug in a new version of the sensor, and it doesn’t have a parameter that the previous sensor had. That parameter will still exist, and the user might be surprised that when they change the parameter there’s no behavior change. There’s no visibility into the fact that that parameter is now useless. It’s just vestigial but will persist as long as the parameter server stays online.

As more of the system state evolves this vestigial state will accrue. It might be that some of these parameters overlap between nodes that ran before and suddenly you’ll get different behavior based on the order of previous launches.

It’s much cleaner to actually keep the parameters in nodes that have semantic meaning. It may make sense to have specific nodes that are standalone and part of a launch file for a particular application. But these nodes should plan to own the lifecycle of the parameters, document the semantics, provide appropriate persistence models. As we look towards supporting security these nodes can have appropriate access control. (Access control on a blackboard type nodes add a lot of complexity.

Similarly a “global” parameter server has undefined behavior if you have two robots that are operating independently then roam onto the same network. If you do need or want state to be persisted, you can launch a specific node that’s setup to have a specific backup to a database or other storage mechanism and will reload state. But if you’re thinking about doing that what is the benefit of having the parameters stored in an external process? Does requiring a network connection improve anything? If we had good libraries such that any node could easily set validation policies for parameters, and have modules that can enable persistence for parameters, then the node that owns the parameter can provide it with semantic meaning and provide the necessary persistence for each parameter.

So what I’d like to suggest is that instead of focusing on providing a separate node that just holds parameters and add lots of features to that, instead we provide methods and libraries that can build on and extend the node’s parameter API to allow semantically meaningful parameters integrated into the nodes that use them. And this is not to say that you can’t leverage a parameter from another node, but don’t just try to put all the parameters in one place. They are all accessible from any node on the system independent of which node is hosting them so centralizing them just breaks down the modularity of the system. We may also want to develop standard parameter interfaces that multiple interchangeable nodes can implement, but that’s much more introspectable than just putting the “expected” parameter onto the parameter server and hoping that the other node will read the same parameter with the same semantic meaning.

4 Likes

I was going to chime in to make clear my unease about this regressive trend towards globalized parameters on the grounds of security, but @tfoote gladly beat me to it!

This design pattern was notorious to try and account for in SROS1, where the parameter server was hosted in the ros-master, and ownership was unclear. The width of the global parameter API and its pervasive use was so much in conflict with principle of least privilege (POLP) that there was little we could do of secure parameter to any meaningful degree without breaking the establish API. See slide 14 from a previous ROSCon 2017:

I’m already quite uneasy with the growing trend of flat level parameter_events topics and its current read and write access requirements for all basic nodes:

This defeats much of our efforts to restrict data flow within the computation graph at the starting line, as every node on the network must then be given both read/write access to a commonly shared global topic. Thus much of the access control efforts in SROS2 lose their teeth, as a means for strict policy enforcement point for isolating/protecting subsystems in the same computation graph is rendered unachievable by design.

@tfoote

thanks for your thoughts, that is really appreciated!!!

i think that most of parameter should be bound to the driver node such as sensor node as you mentioned previously. the point is who is responsible for that parameter. and im not saying that every single parameter should be taken care by parameter server. but some parameters like system configuration, system property, awareness level, attention level for perception or recognition to be shared in the system for our use cases.

@ruffsl

appreciate for sharing thoughts with security aspect, that really helps!

There’s a difference between a “system” and “global”. If you’re running one robot, “system” and “global” can be relatively easily interchanged. But if you think about being able to instantiate two copies of that same robot, the robot configuration is no longer “global”. It needs to be able to be pushed down into a namespace with the robot because it may not be appropriate for both or, if there’s more, all robots to be required to have the same “awareness level” or any other system setting. Instead if you have a robot configuration management node that has these parameters each robot instance can be pointed at the appropriate configuration management node and get all this functionality with the potential additional functionality of having semantically meaningful validation and potentially application specific logic that can do things like cause one parameter change to adjust another one. This could be useful if you have a mode property that changes the validation ranges on other properties.

And if you use an introspection tool to check on robot configurations it can discover all the configuration nodes and then display what it finds for each configuration node.

One of the most valuable things about ROS APIs is that we make sure that the messages have specific semantic meaning so that they can’t be misinterpreted. As we develop the ROS 2 tools and best practices we should make sure to bring that same level of rigor to parameters too for greater reusability and correctness.

There’s a difference between a “system” and “global”. If you’re running one robot, “system” and “global” can be relatively easily interchanged. But if you think about being able to instantiate two copies of that same robot, the robot configuration is no longer “global”.

yes, that is correct.
as you mentioned, we may need to use namespace.

Instead if you have a robot configuration management node that has these parameters each robot instance can be pointed at the appropriate configuration management node and get all this functionality with the potential additional functionality of having semantically meaningful validation and potentially application specific logic that can do things like cause one parameter change to adjust another one. This could be useful if you have a mode property that changes the validation ranges on other properties.

understood, technically i do agree with you.

thank you for the insights!

ROS2 Persistent Parameter Server

This is the PoC project for ROS2 Persistent Parameter Server, that resides in the ROS2 system to serve the parameter daemon. The other nodes can write/read the parameter in Parameter Server, and Parameter Server is able to store the parameter into the persistent storage which user can specify such as tmpfs, nfs, or disk.

if anyone interested on persistent storage, we like to discuss if we could port this feature into mainline.

How will this interact with the launch system, especially parameters that are set during launch?

@gbiggs

you can still use parameter via launch system, of course.

parameter server is only aware of “persistent parameter” via persistent yaml file or /parameter_events, since i did not change any ROS2 core libraries such as rclcpp, rcl. I will add more example including launch system.

That’s my point: You need to clearly define what happens to parameters that are both set by the launch step and by your persistence node.

1 Like

Just FYI, description and samples are added.