This article contains a proposal for integrating thread configuration functionalities into rclcpp. The basic idea is to give executors a name, similar to nodes, so that their thread configuration can be set via a ROS parameters file, which could look like this:
This example contains the configuration for three different executors. The scheduler_type property can be either “fair” or “fifo” (see SCHED_FIFO of the POSIX standard), cpu_priority refers to a number of predefined strings (to be defined) for platform independent cpu priorities, cpu_bitmask is an integer whereby each single bit corresponds to a cpu core that the executor thread(s) may run on (note: this limits the adressable cpu cores to 32 cores; maybe it would be better to use a string to store the cpu bitmask?). The executor base class should get functionality to automatically parse these config entries, so that every executor implementation can easily access its thread configuration. Ideally, a thread configuration reader class is created for this purpose which can be reused by other ROS components which also want to read their thread configuration from a ROS parameters file.
The executors should get a new optional constructor parameter, besides a new optional parameter for their name, to receive a thread managment interface instance, which could be defined like this:
namespace rclcpp {
class ThreadManager {virtual ~ThreadManager() = 0;
virtual std::weak_ptr< std::thread > solicit_thread(std::function<void () > callback, rclcpp::SchedulerType sched_type = THREAD_SCHEDULER_FAIR, rclcpp::ThreadPriority thread_priority = THREAD_PRIORITY_MEDIUM, unsigned int64_t bitmask = (unsigned int64_t) -1) = 0;
virtual void release_thread(std::weak_ptr< std::thread >& thread_ref) = 0;
}}
This approach decouples the executors from any OS specific code for creating & configuring a thread. ROS2 should, however, provide a standard ThreadManager implementation which works for the most common OSs, but the application programmer would be free to choose whatever ThreadManager implementation he likes to use. For example, someone might choose an implementation that internally uses a thread pool. If no ThreadManager instance is passed to an executor it should throw an exception when entering a spin function and a thread configuration was specified. The callback parameter references the code that is being executed by a solicited thread (note: use std::bind here). Moreover, the solicit_thread function should define suitable standard parameters, so that calling solicit_thread without parameters, besides the callback parameter, behaves like executing new std::thread(callback). The release_thread function should be called to signal that a particular thread is not used anymore. If an OS does not support fifo scheduling calling solicit_thread should throw an exception in case fifo scheduling is requested. The executor base class should also have a public function, or several public functions, to manually set a thread configuration, whereby the manual settings should override the parameter file settings. Calling a thread configuration function should cause an exception if the executor is already spinning.
The thread configuration should be applied when a spin call is made. Therefore the following methods would need to be adapted in rclcpp:
- Executor::spin_some_impl
- Executor::spin_once_impl
- Executor::spin_until_future_complete
- SingleThreadedExecutor::spin
- StaticSingleThreadedExecutor::spin
- StaticSingleThreadedExecutor::spin_some_impl
- StaticSingleThreadedExecutor::spin_once_impl
- MultiThreadedExecutor::spin
The MultiThreadedExecutor::spin function already creates threads internally and that code can easily be adapted for the ThreadManager interface, in case an instance of it was provided. Adapting the other spin functions should also not be a technical problem. I suggest that these other spin functions also request a thread through solicit_thread and join it. The other possibility would be to temporarily reconfigure the thread that called the spin function. The latter approach would require adding another function to the ThreadManager, like virtual void reconfigure_thread(std::native_handle_type thread_handle, rclcpp::SchedulerType sched_type, rclcpp::ThreadPriority thread_priority, unsigned int64_t bitmask) = 0. But it’s debatable if this is a clean approach.
The container classes would also need some changes. The standard container (see component_container.cpp) and the multithreaded container (see component_container_mt.cpp) could use the proposed named executor functionality which would allow using ROS parameter files to configure their thread(s). Moreover, the executors used by the component containers need an instance of the ThreadManager being passed to them. The isolated container is a bit more delicate to change. For it the ComponentManagerIsolated implementation(see component_manager_isolated.hpp) needs to be adapted. Here it would be helpful to have non-blocking executor spin functions and being able to join the executor thread(s) later on for cancellation/cleanup. Alternatively a ThreadManager::reconfigure_thread function would be needed (as mentioned before).
Open questions:
- Which package should host the standard implementation for the ThreadManager interface?
- Should we introduce a ROS thread class in rcpputils, which for now simply maps to std::thread?
- Did I miss some rclcpp functionality that also would require source code changes?