Robot Fleet Adapter does not submit bid when submitting Automatic Dispatch Type from RMF Dashboard (#673)

Open-RMF Open-RMF General
1

Posted by @cardboardcode:

Description :bug:

When submitting a basic Automatic Dispatch Type patrol task through RMF Dashboard (as shown in the video below), the Robot Fleet Adapter does not seem to receive the bid in order to start the cost calculation and submit a Bid Proposal in response.

However, when submitting a basic patrol task with Robot specified, RMF is able to dispatch the task successfully and Robot Fleet Adapter executes the task successfully.

See video attached for better visualisation.

Screencast from 04-20-2025 02:32:29 PM.webm

Error Abstract

rmf_core_c                 | [rmf_task_dispatcher-14] [INFO] [1745128036.256038198] [rmf_dispatcher_node]: Task auction for [patrol.dispatch-0] did not received any bids
rmf_core_c                 | [rmf_task_dispatcher-14] [WARN] [1745128036.256195683] [rmf_dispatcher_node]: Dispatcher Bidding Result: task [patrol.dispatch-0] has no submissions during bidding. Dispatching failed, and the task will not be performed.

Context :book:

I have created a simple simulated world with a Turtlebot3. All of the following RMF and Robot components are in their own individual docker containers for the sake of modularity:

  • RMF Dashboard (ROS 2 Jazzy)
  • RMF API Server (ROS 2 Jazzy)
  • Robot Fleet Adapter (ROS 2 Jazzy)
  • RMF Core (ROS 2 Jazzy)
  • Robot REST API Server (ROS 1 Noetic)
  • Robot Gazebo Simulation (ROS 1 Noetic)
  • Robot Navigation Stack (ROS 1 Noetic)

Expected Behaviour :green_circle:

When submitting Automatic Dispatch Type patrol task, RMF should be able to find robot0 which is the simulated TurtleBot3 under Robot Fleet Adapter and successfully award the bid to it to start executing.

Actual Behaviour :red_circle:

When submitting Automatic Dispatch Type patrol task, Robot Fleet Adapter seemingly ignores the bid from RMF Traffic. Even though, from the video above, it clearly shows that robot0 can be found and is interactable through a direct patrol task dispatch.

Temporary Workaround :adhesive_bandage:

For now, it seems that, to circumvent this issue, both Robot Fleet Adapter and RMF Core needs to be in the same docker container in order for Automatic Dispatch Type patrol task to find robot0.

However, for the sake of modularity, would prefer to keep them apart. Keeping in mind that having the same set-up based in ROS 2 Humble does not face the same issue.

Investigation :eyeglasses:

Looking into this issue further in-depth, there are a few odd observations:

  • ROS 2 topic fleet_markers from ROS 2 node fleet_states_visualizer does not publish any data, even though ROS 2 topic /fleet_states can be seen publishing robot information. This results in a lack of RViz Marker showing Robot position on RMF Core RViz. See screenshot below:

    Screenshot from 2025-04-20 14-54-21
    Screenshot from 2025-04-20 14-54-21799脳754 47.2 KB

  • Despite being on the same ROS 2 distribution between RMF Core and Robot Fleet Adapter, the following error is often thrown when attempting to read information of ROS 2 topics:

This error state is being overwritten:

  'Unknown QoS durability policy, at ./src/qos.cpp:82'

with this new error message:

  'create_publisher() failed setting topic QoS, at ./src/subscription.cpp:599'

rcutils_reset_error() should be called after error handling to avoid this.

Remarks :speech_balloon:

Therefore, after much self-debugging, I would like to trouble you for your insights on what critical components/configurations I may be missing in my set-up in order to achieve the Expected Behaviour.

Will provide any further necessary information as required in order to debug.

2

Posted by @aaronchongth:

thanks for flagging this, I haven鈥檛 looked into reproducing it yet, but I do see a lot of IDL related warning and error messages too. My experience is that these normally indicate something related to the RMW layer is not doing what it is supposed to. We鈥檝e had the least amount of issues using rmw_cyclonedds_cpp when running RMF. Can you confirm what RMW you are using at the moment? The default in Jazzy is still rmw_fastrtps_cpp

Edited by @aaronchongth at 2025-04-21T02:24:53Z

3

Posted by @cardboardcode:

Can you confirm what RMW you are using at the moment?

The RMW I am using at the moment is rmw_fastrtps_cpp.

Thank you for the prompt reply.

Will change my setup to use rmw_cyclonedds_cpp RMW instead and update here if the same issue is encountered.

4

Posted by @cardboardcode:

Hi @aaronchongth

I have tried changing my setup to use rmw_cyclone_cpp instead of rmw_fastrtps_cpp.

I have done so by installing ros-jazzy-rmw-cyclonedds-cpp into each ROS 2 -based RMF Component (RMF Core and Robot Fleet Adapter) as well as defining RMW_IMPLEMENTATION=rmw_cyclonedds_cpp when running the docker image as container.

However, the following runtime error is thrown when starting up RMF Core:

Error Abstract :bug:

[building_map_server-13] [INFO] [1745245499.816895494] [building_map_server]: ready to serve map: "tb3_world"  Ctrl+C to exit...
[rmf_traffic_schedule_monitor-2] [WARN] [1745245509.251399482] [rmf_traffic_schedule_backup]: Timeout while trying to connect to traffic schedule
[schedule_visualizer_node-4] [ERROR] [1745245509.255089270] [schedule_data_node]: Mirror was not initialized in enough time [10.000000s]!
[schedule_visualizer_node-4] [ERROR] [1745245509.257504221] [schedule_visualizer_node]: Failed to create a Mirror of the RMF Schedule Database after waiting for 10 seconds. Please ensur rmf_traffic_schedule node is running. No schedule markers will be published to RViz.
[rmf_traffic_schedule_monitor-2] Stack trace (most recent call last):
[rmf_traffic_schedule_monitor-2] #5    Object "", at 0xffffffffffffffff, in 
[rmf_traffic_schedule_monitor-2] #4    Object "/opt/ros/jazzy/lib/rmf_traffic_ros2/rmf_traffic_schedule_monitor", at 0x5e5a0cf57384, in _start
[rmf_traffic_schedule_monitor-2] #3    Object "/usr/lib/x86_64-linux-gnu/libc.so.6", at 0x73b91f38028a, in __libc_start_main
[rmf_traffic_schedule_monitor-2] #2    Object "/usr/lib/x86_64-linux-gnu/libc.so.6", at 0x73b91f3801c9, in 
[rmf_traffic_schedule_monitor-2] #1    Object "/opt/ros/jazzy/lib/rmf_traffic_ros2/rmf_traffic_schedule_monitor", at 0x5e5a0cf5696b, in main
[rmf_traffic_schedule_monitor-2] #0    Object "/opt/ros/jazzy/lib/librclcpp.so", at 0x73b91f963c54, in rclcpp::Node::get_node_base_interface()
[rmf_traffic_schedule_monitor-2] Segmentation fault (Address not mapped to object [0x18])
[ERROR] [rmf_traffic_schedule_monitor-2]: process has died [pid 26, exit code -11, cmd '/opt/ros/jazzy/lib/rmf_traffic_ros2/rmf_traffic_schedule_monitor --ros-args -r __node:=rmf_traffic_schedule_backup --params-file /tmp/launch_params_bpigbwth'].

This has been largely the reason why I have avoided using rmw_cyclone_cpp for now since the error thrown does not provide much meaningful information to attempt debugging.

Please see the raw logs using the following link to a gist:

Remarks :speech_balloon:

@aaronchongth May I ask if you have encountered a error similar to the above Error Abstract before?

Edited by @cardboardcode at 2025-04-21T14:42:57Z

5

Posted by @aaronchongth:

Not quite. The error means that the traffic schedule mirrors timed out waiting for the main traffic schedule, which means that there was no communication between the nodes at all. I see that the traffic schedule did indeed start, but the other nodes failed to find it. I鈥檓 gonna guess this is just the RMF Core docker image.

Installing and switching out just the RMW_IMPLEMENTATION should not have any different behavior, and from our own deployment experience, rmw_cyclonedds_cpp provides the best performance and reliability. I suspect there may be a misconfiguration somewhere. It鈥檇 be great if you can share your Dockerfile and the commands you used to start the containers too.

6

Posted by @cardboardcode:

Thank you for responding.

Will provide them as follows:

Dockerfile:

FROM ros:jazzy-ros-core

ENV DEBIAN_FRONTEND=noninteractive
RUN apt-get update && apt-get install -y \
    git \
    build-essential \
    ros-jazzy-rviz2 ros-jazzy-rmf-dev \
    ros-jazzy-rmw-cyclonedds-cpp \
    python3-colcon-common-extensions \
    && rm -rf /var/lib/apt/lists/*

# Copy over rmf_core ROS 2 package with utilities
WORKDIR /rmf_core_ws/
COPY rmf_core src/rmf_core
COPY ./scripts/check_health.bash check_health.bash

# Fix for bug for lifts whose names are integers
WORKDIR /rmf_core_ws/src
RUN git clone https://github.com/cardboardcode/rmf_traffic_editor.git --branch bug_fix/level_name_str --single-branch
RUN git clone https://github.com/open-rmf/rmf_demos.git --branch jazzy --depth 1 --single-branch
RUN cp -r rmf_demos/rmf_demos_tasks ./
RUN rm -r rmf_demos
WORKDIR /rmf_core_ws

# Build rmf_core
RUN . /opt/ros/$ROS_DISTRO/setup.sh \
  && colcon build --cmake-args -DCMAKE_BUILD_TYPE=Release

# Source rmf_core with standard /ros_entrypoint.sh.
RUN sed -i '$isource "/rmf_core_ws/install/setup.bash"' /ros_entrypoint.sh

ENTRYPOINT ["/ros_entrypoint.sh"]
CMD ["bash"]

rmf_core is essentially a ROS 2 package that contains just start_rmf_core.launch.xml file with its contents as shown below:

<?xml version='1.0' ?>

<launch>
  <!-- Common parameter -->
  <arg name="use_sim_time" default="false"/>

  <!-- Visualisation(rviz) parameters -->
  <arg name="viz_config_file" default="/rmf_core_ws/src/rmf_core/rviz/debug.rviz"/>
  <arg name="config_file" default="$(find-pkg-share rmf_core)/maps/default.building.yaml"/>
  <arg name="initial_map" default="L1" description="Initial map name for the visualizer"/>
  <arg name="headless" default="false" description="do not launch rviz; launch gazebo in headless mode"/>
  <arg name="map_name" default="L1"/>
  <arg name="rate" default="10.0"/>
  <arg name="display_names" default="true"/>
  <arg name="websocket_port" default="8006"/>
  <arg name="headless" default="false" description="Do not launch rviz2"/>

  <arg name="rmf_frame_id" default="map" description="The RMF global frame id"/>
  
  <arg name="path_width" default="0.2" description="Width of schedule path(m) to be rendered"/>
  <arg name="lane_width" default="0.5" description="Width of lanes(m) to be rendered"/>
  <arg name="waypoint_scale" default="1.3" description="The relative size of a waypoint wrt the lane width"/>
  <arg name="text_scale" default="0.7" description="The relative size of a text label wrt the lane width"/>
  <arg name="lane_transparency" default="0.6" description="The transparency of a rendered lane in default state"/>
  <arg name="wait_secs" default="10" description="The nummber of seconds the schedule visualizer should keep trying to connect to the rmf_schedule_node"/>
  <arg name="retained_history_count" default="50" description="The retained history count for negotiations"/>
  <arg name="fleet_state_nose_scale" default="0.5" description="The radius of the nose marker as a fraction of the fleet's footprint radius"/>


  <!-- Dispatcher Node parameter-->
  <arg name="bidding_time_window" description="Time window in seconds for task bidding process" default="10.0"/>

  <!-- Traffic Schedule  -->
  <node pkg="rmf_traffic_ros2" exec="rmf_traffic_schedule" output="both" name="rmf_traffic_schedule_primary">
    <param name="use_sim_time" value="$(var use_sim_time)"/>
  </node>
  <!-- Traffic Schedule monitor and replacement node -->
  <node pkg="rmf_traffic_ros2" exec="rmf_traffic_schedule_monitor" output="both" name="rmf_traffic_schedule_backup">
    <param name="use_sim_time" value="$(var use_sim_time)"/>
  </node>

  <!-- Blockade Moderator -->
  <node pkg="rmf_traffic_ros2" exec="rmf_traffic_blockade" output="both">
    <param name="use_sim_time" value="$(var use_sim_time)"/>
  </node>

  <!-- RMF Visualisation nodes(for RVIZ) -->
  <group>
    <node pkg="rmf_visualization_schedule" exec="schedule_visualizer_node" args="-r $(var rate) -m $(var map_name) -p $(var websocket_port) --history 50">
      <param name="use_sim_time" value="$(var use_sim_time)"/>
    </node>

    <node pkg="rmf_visualization_fleet_states" exec="fleetstates_visualizer_node" args="-m $(var map_name)">
      <param name="display_names" value ="$(var display_names)"/>
      <param name="use_sim_time" value="$(var use_sim_time)"/>
    </node>

    <node pkg="rmf_visualization_building_systems" exec="rmf_visualization_building_systems" args="-m $(var map_name)">
      <param name="use_sim_time" value="$(var use_sim_time)"/>
    </node>

    <node pkg="rmf_visualization_floorplans" exec="floorplan_visualizer_node">
      <param name="use_sim_time" value="$(var use_sim_time)"/>
      <param name="initial_map_name" value="$(var map_name)"/>
    </node>

    <node pkg="rmf_visualization_obstacles" exec="obstacle_visualizer_node">
      <param name="use_sim_time" value="$(var use_sim_time)"/>
      <param name="initial_map_name" value="$(var map_name)"/>
      <param name="global_fixed_frame" value="$(var rmf_frame_id)"/>
    </node>

    <node pkg="rmf_visualization_navgraphs" exec="navgraph_visualizer_node" output="both">
      <param name="use_sim_time" value="$(var use_sim_time)"/>
      <param name="initial_map_name" value="$(var map_name)"/>
      <param name="lane_width" value="$(var lane_width)"/>
      <param name="waypoint_scale" value="$(var waypoint_scale)"/>
      <param name="text_scale" value="$(var text_scale)"/>
      <param name="lane_transparency" value="$(var lane_transparency)"/>
    </node>
  </group>

  <group unless="$(var headless)">
    <executable cmd="rviz2 -d $(var viz_config_file)" output="both"/>
  </group>

  <!-- Door Supervisor -->
  <group>
    <node pkg="rmf_fleet_adapter" exec="door_supervisor">
      <param name="use_sim_time" value="$(var use_sim_time)"/>
    </node>
  </group> 

  <!-- Lift Supervisor -->
  <group>
    <node pkg="rmf_fleet_adapter" exec="lift_supervisor">
      <param name="use_sim_time" value="$(var use_sim_time)"/>
    </node>
  </group> 

  <!-- Building Map  -->
  <group>
    <node pkg="rmf_building_map_tools" exec="building_map_server" args="$(var config_file)">
      <param name="use_sim_time" value="$(var use_sim_time)"/>
    </node>
  </group>

  <!-- Dispatcher Node -->
  <group>
    <node pkg="rmf_task_ros2" exec="rmf_task_dispatcher"  output="screen">
      <param name="use_sim_time" value="$(var use_sim_time)"/>
      <param name="bidding_time_window" value="$(var bidding_time_window)"/>
    </node>
  </group>

</launch>

Commands Used To Start The Containers

  1. Build aforementioned Dockerfile as docker image:
docker build -t rmf_core:jazzy .
  1. Run docker image as docker container:
docker run -it --rm \
  --name rmf_core_c \
  --network=host \
  -e RMW_IMPLEMENTATION=rmw_cyclonedds_cpp \
  -e DISPLAY=$DISPLAY \
  -v /dev/shm:/dev/shm \
  -v /tmp/.X11-unix:/tmp/.X11-unix \
  -v ./rmf_core/rviz/debug.rviz:/rmf_core_ws/src/rmf_core/rviz/debug.rviz \
  -v ./map:/rmf_core_ws/map/ \
  rmf_core:jazzy bash -c \
  "ros2 launch rmf_core start_rmf_core.launch.xml \
  config_file:=/rmf_core_ws/map/tb3_world/tb3_world.building.yaml"
7

Posted by @aaronchongth:

Thanks for the info, I鈥檝e modified your Dockerfile to below as I don鈥檛 have many of your packages,

FROM ros:jazzy-ros-core

ENV DEBIAN_FRONTEND=noninteractive
RUN apt-get update && apt-get install -y \
    git \
    build-essential \
    ros-jazzy-rviz2 ros-jazzy-rmf-dev \
    ros-jazzy-rmw-cyclonedds-cpp \
    python3-colcon-common-extensions \
    && rm -rf /var/lib/apt/lists/*

# Copy over rmf_core ROS 2 package with utilities
WORKDIR /rmf_core_ws/
# COPY rmf_core src/rmf_core
# COPY ./scripts/check_health.bash check_health.bash

# Fix for bug for lifts whose names are integers
WORKDIR /rmf_core_ws/src
RUN git clone https://github.com/cardboardcode/rmf_traffic_editor.git --branch bug_fix/level_name_str --single-branch
RUN git clone https://github.com/open-rmf/rmf_demos.git --branch jazzy
COPY test.launch.xml /rmf_core_ws/src/rmf_demos/rmf_demos_gz/launch/test.launch.xml
# RUN git clone https://github.com/open-rmf/rmf_demos.git --branch jazzy --depth 1 --single-branch
# RUN cp -r rmf_demos/rmf_demos_tasks ./
# RUN rm -r rmf_demos
WORKDIR /rmf_core_ws

# RUN . /opt/ros/$ROS_DISTRO/setup.sh \
#   && rosdep install --from-paths src --ignore-src -ry

# Build rmf_core
RUN . /opt/ros/$ROS_DISTRO/setup.sh \
  && colcon build --cmake-args -DCMAKE_BUILD_TYPE=Release

# Source rmf_core with standard /ros_entrypoint.sh.
RUN sed -i '$isource "/rmf_core_ws/install/setup.bash"' /ros_entrypoint.sh

ENTRYPOINT ["/ros_entrypoint.sh"]
CMD ["bash"]

Created test.launch.xml beside the Dockerfile (it gets copied into rmf_demos_gz/launch)

Built with docker build -t rmf_core:jazzy .

Running it using the basic office building map,

docker run -it --rm  \
   --name rmf_core_c \
  --network=host \
  -e RMW_IMPLEMENTATION=rmw_cyclonedds_cpp \
  -e DISPLAY=$DISPLAY \
  -v /dev/shm:/dev/shm \
  -v /tmp/.X11-unix:/tmp/.X11-unix \
  rmf_core:jazzy bash -c "ros2 launch rmf_demos_gz test.launch.xml config_file:=/rmf_core_ws/src/rmf_demos/rmf_demos_maps/maps/office/office.building.yaml"

The visualization stuff is broken since I鈥檝e not set up my x11 forwarding properly, but everything else seem to be working fine and all the nodes find the schedule without issue,

with rmw_cyclonedds_cpp, with rmw_cyclonedds_cpp 路 GitHub

with the default RMW, with default rmw 路 GitHub

any chance it is your additional configs or scripts like check_health.bash that is doing something unintended?

8

Posted by @cardboardcode:

Hi @aaronchongth

Thank you for the detailed steps and modifications. :pray: Will give this a try once I am free again. Will update here the results.

any chance it is your additional configs or scripts like check_health.bash that is doing something unintended?

Might be the case. check_health.bash is a simple bash script that runs ros2 node list so should not be the issue. Will recheck on my additional configurations.

Edited by @cardboardcode at 2025-04-22T04:33:27Z

9

Posted by @cardboardcode:

Hi @aaronchongth

Gave the above modified Dockerfile a go. Was still not able to get my set-up to work with rmw_cyclonedds_cpp.

When running set-up with rmw_cyclonedds_cpp, Robot Fleet Adapter is not able to detect RMF Core. Robot Fleet Adapter waits for 60 seconds and then crashes. :red_circle:

However, in my continued self-debugging, I backtracked to a previous working commit and narrowed down the cause of issue to the following:

Cause of Issue :mag:

However, by changing the base docker image of both RMF Core and Robot Fleet Adapter to ros:jazzy-ros-core and reinstalling RMF binaries, I was able to achieve the Expected Behaviour with automatic RMF task assignment working. This is still done using rmw_fastrtps_cpp as RMW.

Still not sure on the exact specifics of why.

The most I can deduce for now is:

ghcr.io/open-rmf/rmf/rmf_demos:latest has internal parameters affecting the discovery of certain ROS 2 topics when using rmw_fastrtps_cpp.

Solution :medal_military:

Changed the base docker image for Robot Fleet Adapter from:

FROM ghcr.io/open-rmf/rmf/rmf_demos:latest

to:

FROM ros:jazzy-ros-core

And set up RMF binaries from scratch.

Remarks :speech_balloon:

Would likely try to avoid using ghcr.io/open-rmf/rmf/rmf_demos:latest as a base docker image in the future, particularly when still using rmw_fastrtps_cpp as RMW.

Would also re-attempt to use rmw_cyclonedds_cpp as much as possible (since that seems to be the de-facto RMW for RMF), although, to be frank and I am not sure why, rmw_cyclonedds_cpp has never worked for my RMF implementations thus far.

Will likely provide a minimal example showing why rmw_cyclonedds_cpp does not seem to work properly from my perspective some time in the near future.

Edited by @cardboardcode at 2025-04-27T06:56:53Z

10

Posted by @cardboardcode:

Hi @aaronchongth

I have created a separate GitHub issue thread for the minimal example on why rmw_cyclonedds_cpp does not work for me:

Please see [Bug]: rmf_traffic_schedule_monitor crashes when running with rmw-cyclonedds-cpp 路 Issue #676 路 open-rmf/rmf 路 GitHub.

Edited by @cardboardcode at 2025-04-27T08:03:42Z

11

Posted by @cardboardcode:

Refer to Solution above.

Closing discussion.