[Issue]
There are no infrastructures to configure and set thread attributes of the threads used by Executor so far.
So, if we want to set some attributes (priority, scheduling policy, and so on), we should implement environment-specific code for each application, as below.
[Proposed solution]
If we want to add an infrastructure treating such attributes, we will have to add two new items below.
- Measure to pass the thread attributes to ROS 2 infrastructure.
- Low-level thread setting feature for each language binding
Particularly for C++, rclcpp cannot set thread attributes for each thread because it depends on the std::thread class to control threads.
To discuss how we can address these items, our team has implemented a set of implementations as below.
For item 1, our team has modified the rcl layer to receive a set of scheduling parameters. Those parameters are described in YAML format and passed via command line parameters, environment variables, or a file. And, those are able to be gotten via some additional API from upper language bindings.
And for item 2, our team has implemented a new “rclcpp::thread” class which is almost the same as std::trhead but can set the thread attributes when it creates a thread. Using this class, our team has modified the Multithread Executor, too, to read the attributes from rcl and set those to each thread created by it. With this sample implementation, I could share my intention for the proposal with you.
This proposal has already been shared with the Real-time WG members, as below.
[Candidates of the items to be discussed]
Through the discussion at the issue ticket above, WideAwakeTN · GitHub has kindly listed items to be discussed. I append some comments to the items to explain our intention or current implementations included in the PR above.
Regarding plain C:
- Should OS specific and POSIX low level code be confined to a separate ROS2 package? Should this package function like an abstraction layer? Does it make more sense to use ifdefs inside the code to support different operating systems or is it better to create multiple variants of this new package for different operating systems?
- The alternative would be putting the code into rclc or some other already existing package (and using ifdefs).
- How to ensure that the new functionality does not hinder microcontroller development for ROS2? This issue suggests to not put OS specific code or POSIX code into rclc.
[Comments]
For items 1 and 2, the current implementations are not separated from the existing packages. But, we’d like to separate them if there are any places more preferred.
For item 3, the YAML parameter passing cannot hinder any microcontroller development. And language bindings (rclc, rclpy, and so on) can decide whether they would use the parameter passing infrastructure. So, for example, rclc can ignore the stored parameters in rcl.
Regarding C++:
- Should there be a factory to spawn threads? A factory would allow to implement thread pools.
- Should there be an interface for thread class instances?
- How to configure threads? Configure them when they are spawned, e.g. by passing parameters to the factory interface or the constructor, or configure them later?
- Is thread re-configuration required later on? Then the thread class / interface should provide some kind of public configuration functionality.
- What thread properties should be configurable? Presumably CPU core affinity, CPU priority and scheduler type (FIFO or FAIR).
- Which ROS2 C++ core entities should natively support thread configuration? The MultiThreadedExecutor, SingleThreadedExecutor, StaticSingleThreadedExecutor, the standard container, the multithreaded container and the isolated multithreaded container? Note: the SingleThreadedExecutors could spawn a worker thread internally and join it (the join behaviour could actually be optional and the calling thread might alternatively continue… that might be useful functionality).
- Should the new thread functionality be available to all ROS2 C++ components, e.g. the middleware layer, the TF2 library, etc., and if so how?
- Should the new thread functionality be available to application developers, and if so how?
- How to store and read a thread configuration so that it is available to all supported entities? Should the ROS parameters YML be used/extended, a new file be used or some other method chosen?
- Should there be a thread configuration class/object to combine all the different thread configuration parameters? This class could provide an easy way for everyone to load a thread configuration. (Note: internally the configuration class could access some already available data that was parsed during rclcpp::init).
- Should all executor instances get names, similar to node instances, so that they can access their thread configuration automatically? Should executors get general parameter functionality, similar to nodes? Note: executor parameters are expected to not change during runtime and need not be accessible via services.
- How much flexibility do we need?
- Should there be a CMake option to deactivate any new thread related functionality?
- Should POSIX or OS specific code be be kept out of rclcpp and rcpputils?
[Comments]
For items 1 to 3, the rclcpp::thread is implemented to be just an infrastructure to control the thread attribute in rclcpp and used in multipurpose executers (like single/multithread executors and “coming-in-future” executors for hard real-time purposes). Even if you find such factories or interfaces you mentioned above useful, the rclcpp::thread can be useful to implement them.
For item 4, I’m curious to know whether such a need exists or not. But, the current rclcpp::thread has no feature to change the attributes of threads after it creates them.
For item 5, the rclcpp::thread should be implemented for each specific environment. The proposal’s parameter sets are implemented, being based on POSIX-like systems (including Linux) and assuming that there are some configuration mappings will be required for another tier1 Windows environment.
For item 6, the PR has only included Multithread Executor using the new feature. And our team will be able to propose the Singlethread Executor using it once this proposal is accepted. Before that, I want to confirm if the parameter passing infrastructure and rclcpp::thread is acceptable or not.
For item 7, the rclcpp::thread can also be used in other packages using C++. But, in such a case, we might have to change the namespace “rclcpp::” to “roscpp::” or something because the class would no longer belong to not only rclcpp. And, more, command line or environment parameter names passed to rcl might also have to add some prefixes. For example, "tf_ " when is used for the TF library.
For item 8, if there are any needs, the rclcpp::thread can be used for any purpose, as I said above.
For item 9, the PR’s code has two types, single YAML format file or continuous YAML string. And each is specified by a command line parameter or environment variable. And they have priorities so that only one parameter set shall be used, not mixed with each other.
For item 10, to have the parameter set used by multiple components (e.g., rclpp and tf), we can add another parameter (like ID as strings) to decide to which components each thread belongs.
For item 11, the PR has only provided the infrastructure thread attribute passing, rclcpp::thread, and sample multithread executor using them. But, once we add some new feature to set names to executors, we might be able to map each thread to each executor by using the ID string I mentioned above.
For items 13 and 14, as I said in another thread, the PR has also included environment-agnostic rclcpp::thread, which is just a wrapper for std::thread. So, the rclcpp::thread can be used for the environment for which the environment-specific implementation is not implemented yet.
Incidentally, in such an environment, the sample Multithread executor raises an error when the thread attributes are passed. This behavior could be changed to just ignore the passed parameter if we found it appropriate.
Regarding Python:
- Does it make any sense to provide thread configuration facilities for Python?
[Commments]
As I said above, you could use the parameter passing infrastructure with any language binding layers.