Fixing Moveit2 Humble moveit_ros_benchmarks package

I had some problems running benchmarks using the moveit_ros_benchmarks package while following the tutorial: https://moveit.picknik.ai/humble/doc/examples/benchmarking/benchmarking_tutorial.htm

After some time I fixed all my issues. They are not completely fixing the entire package, but at least the benchmark works now fine for me.

I just want to leave the steps I made here so everyone having similar issues can reproduce them. I hope that this is the right place to do it.

Requirements: Working Moveit2 Config

Trying to follow: Benchmarking — MoveIt Documentation: Humble documentation

But outdated tutorial and warehouse_mongo does not exist anymore for ros2.

Saving a scene with queries etc in which the benchmarks shall be done

warehouse_mongo does not exist anymore for ros2.

Instead try to use warehouse_sqlite: Launch your rviz stuff with your robot as before so you can save the robot states. See next example code snippet. Then follow tutorial to save scenes and states and queries. (Maybe you need also: sudo apt install ros-humble-warehouse-ros-sqlite)

    warehouse_ros_config = {
        "warehouse_plugin": "warehouse_ros_sqlite::DatabaseConnection",
        "warehouse_host": "localhost"
    }

    # MoveGroup Node | remapping needed 
    run_move_group_node = Node(
        package="moveit_ros_move_group",
        executable="move_group",
        output="screen",
        parameters=[moveit_config.to_dict(), warehouse_ros_config,],
    )

    # RViz
    rviz_config_file = (
        get_package_share_directory("kairos_simple_mover_group")
        + "/config/moveit_group_tutorial.rviz"
    )
    rviz_node = Node(
        package="rviz2",
        executable="rviz2",
        name="rviz2_moveit2",
        output="log",
        arguments=["-d", rviz_config_file],
        parameters=[
            moveit_config.robot_description,
            moveit_config.robot_description_semantic,
            warehouse_ros_config,
        ],
    )

Do the benchmark

You need to adjust the benchmark package code, so get the code from github (humble) and put it into your workspace. In this package you will find the launch files and .yamls.

  1. Take the demo_panda.launch.py as template and modify to use sqlite instead of mongo
from launch import LaunchDescription
from launch_ros.actions import Node
from moveit_configs_utils import MoveItConfigsBuilder
from launch_param_builder import ParameterBuilder

def generate_launch_description():

    moveit_ros_benchmarks_config = (
        ParameterBuilder("moveit_ros_benchmarks")
        .yaml(
            parameter_namespace="benchmark_config",
            file_path="demo1.yaml",
        )
        .to_dict()
    )

    moveit_configs = MoveItConfigsBuilder("kairosab_ur10_egh").to_dict()

    warehouse_ros_config = {
        "warehouse_plugin": "warehouse_ros_sqlite::DatabaseConnection", # needed to use sqlite instead of mongodb
        # "warehouse_host": "localhost"  # not needed because host and port are set in benchmark_config
    }

    ompl_planning_pipeline_config = {
        "planning_plugin": "ompl_interface/OMPLPlanner",    # currently a workaround for the Error: Multiple planning plugins available. You should specify the '~planning_plugin' parameter. Using 'chomp_interface/CHOMPPlanner' for now.
    }

    # moveit_ros_benchmark demo executable
    moveit_ros_benchmarks_node = Node(
        name="moveit_run_benchmark",
        package="moveit_ros_benchmarks",
        # prefix='xterm -e gdb --ex=run --args',
        executable="moveit_run_benchmark",
        output="screen",
        parameters=[
            moveit_ros_benchmarks_config,
            moveit_configs,
            warehouse_ros_config,
            ompl_planning_pipeline_config,
        ],
    )

    # Warehouse mongodb server
    # mongodb_server_node = Node(
    #     package="warehouse_ros_sqlite",
    #     executable="mongo_wrapper_ros.py",
    #     parameters=[
    #         {"warehouse_port": 33829},
    #         {"warehouse_host": "localhost"},
    #         {"warehouse_plugin": "warehouse_ros_sqlit::DatabaseConnection"},
    #     ],
    #     output="screen",
    # )

    return LaunchDescription([moveit_ros_benchmarks_node]) #, mongodb_server_node])
  1. Then you have your demo1.yaml. You can adjust it according to the tutorial. (mine looks like this).
# This is an example configuration that loads the "Kitchen" scene from the
# local MoveIt warehouse and benchmarks the "panda_arm" group in the Pick1
# query using the Start1 initial state (all pre-stored in the local warehouse)

# Three different planners from OMPL are run a total of 50 times each, with a
# maximum of 10 seconds per run.  Output is stored in the /tmp/moveit_benchmarks directory.
warehouse:
    host: localhost
    port: 33829
    scene_name: PickUp     # Required
parameters:
    name: PickingUpItem
    runs: 50
    group: arm       # Required
    timeout: 10.0
    output_directory: /tmp/moveit_benchmarks/
    queries: .*
    start_states: .*
planning_pipelines:
    pipelines: [pipeline1]
    pipeline1:
        name: ompl
        planners:
            - APSConfigDefault
            - SBLkConfigDefault
  1. Modify BenchmarkOptions.cpp to read in the planners correctly (lines 287-293):

This fixes:

[moveit_run_benchmark-1] terminate called after throwing an instance of 'rclcpp::exceptions::InvalidParameterTypeException'
[moveit_run_benchmark-1]   what():  parameter 'benchmark_config.planning_pipelines.pipeline1.name' has invalid type: expected [string_array] got [string]

Looks like this:

std::vector<std::string> planners;
if (!node->get_parameter(pipeline_parameter_name, planners))  // changed
{
		RCLCPP_ERROR(LOGGER, "Fail to get the parameter in `%s` namespace.", pipeline_parameter_name.c_str());
    return;
}

Should look like this:

std::vector<std::string> planners;
const std::string planners_parameter_name = std::string(np).append(".").append(pipeline).append(".planners"); // changed
if (!node->get_parameter(planners_parameter_name, planners))  // changed
{
		RCLCPP_ERROR(LOGGER, "Fail to get the parameter in `%s` namespace.", pipeline_parameter_name.c_str());
		return;
}
  1. Modify BenchmarkExecutor.cpp to correctly initialize the planning pipeline

Fixing: Multiple planning plugins available. You should specify the '~planning_plugin' parameter. Using 'chomp_interface/CHOMPPlanner' for now. Which leads to a segmentation fault later on.

Modify: void BenchmarkExecutor::initialize(const std::vectorstd::string& planning_pipeline_names)

planning_pipeline::PlanningPipelinePtr pipeline(new planning_pipeline::PlanningPipeline(
planning_scene_->getRobotModel(), node_, "", "planning_plugin", "request_adapters")); // changed child_node to node_ and added "" as the third argument
// from https://moveit.picknik.ai/main/doc/examples/motion_planning_pipeline/motion_planning_pipeline_tutorial.html
  // planning_pipeline::PlanningPipelinePtr planning_pipeline(new planning_pipeline::PlanningPipeline(robot_model, node, "", "planning_plugin", "request_adapters"));

This way the planning_pipeline initializes the correct planning interface specified in the launch file.

Now you should be able to run the benchmark. (The boost progress bar is not working)

While the above post fixes some issues, I encountered even more.

I have now created a repository containing the fixed code: GitHub - valantano/moveit-ros-benchmark-humble-fix

Here a short description of all errors I have encountered and fixed:

While following the tutorial here, I encountered the following complications:

  1. warehouse_mongo Dependency: The warehouse_mongo package is no longer available for ROS 2, causing an issue with the benchmarking process.

  2. Incorrect YAML Parsing: The demo1.yaml file is not correctly parsed by the BenchmarkOptions.cpp, resulting in a crash with the following error:

    [moveit_run_benchmark-1] terminate called after throwing an instance of 'rclcpp::exceptions::InvalidParameterTypeException'
    [moveit_run_benchmark-1]   what():  parameter 'benchmark_config.planning_pipelines.pipeline1.name' has invalid type: expected [string_array] got [string]
    
    
  3. Planning Pipeline Initialization: The BenchmarkExecutor.cpp does not correctly initialize the planning pipeline, leading to two possible issues:

    • It either always uses the CHOMP planner.
    • It uses the OMPL planner but fails to find the configurations and falls back to the default planner configured inside the MoveIt Config Package. This issue is accompanied by the following warning:
    [moveit_run_benchmark-1] [WARN] [moveit.ompl_planning.planning_context_manager]: Cannot find planning configuration for group 'arm' using planner 'TRRTkConfigDefault'. Will use defaults instead.
    [moveit_run_benchmark-1] [INFO] [moveit.ompl_planning.model_based_planning_context]: Planner configuration 'arm' will use planner 'geometric::RRTConnect'. Additional configuration parameters will be set when the planner is constructed.
    
    

Hi Valentino,
sorry for the late response & thanks for sharing your solutions to the problems you’ve encountered! Regarding your issues:

Multiple planning plugins available. You should specify the '~planning_plugin' parameter. Using 'chomp_interface/CHOMPPlanner' for now. Which leads to a segmentation fault later on.

We just fixed this upstream: Don't default to random algorithm if no plugin is defined by sjahr · Pull Request #2228 · ros-planning/moveit2 · GitHub. It will fail now if these parameters are not configured correctly. See my suggested changes a couple of comments down on how to configure it correctly.

warehouse_mongo Dependency
I worked on the benchmarking code last year and back then MongoDB was not available in Ubuntu 22.04. That might have changed by now but in order to get it working, it is very likely that warehouse_mongo needs to be updated (It is already ported to ROS 2 (see branch ros2) but I think only to an earlier version). If you try to use it and run into any problems feel free to open an issue.

Incorrect YAML Parsing
That’s the correct fix and it is already applied to moveit2 main but not backported to humble.

Planning Pipeline Initialization
For the pipelines to initialize correctly you need to make sure that all the required parameters are declared correctly. In your case it would be something like this:

    moveit_ros_benchmarks_config = (
        ParameterBuilder("moveit_ros_benchmarks")
        .yaml(
            parameter_namespace="benchmark_config",
            file_path="demo1.yaml",
        )
        .to_dict()
    )

-    moveit_configs = MoveItConfigsBuilder("kairosab_ur10_egh").to_dict()
+    moveit_configs = (
+        MoveItConfigsBuilder("kairosab_ur10_egh")
+       .robot_description(file_path="config/panda.urdf.xacro")
+        .planning_pipelines("ompl", ["ompl", "chomp", "pilz_industrial_motion_planner"])
+        .moveit_cpp(
+            file_path=get_package_share_directory("PATH TO YOUR CONFIG PACKAGE")
+            + "/REALTIVE PATH/moveit_cpp.yaml"
+        )
+        .to_moveit_configs()
+    )

I’ve created a working example a couple of months ago (take it with the caveat that 1-2 small things might be broken) that might help with other issues you run into. If you like you can also consider moving this or your code as a working example to the moveit2_tutorials. Although I planned to do it, I’ve have not started it by now.

warehouse_mongo Dependency

Indeed on Ubuntu 22.04 some dependencies are no longer available hence the warehouse_ros_mongo package is not functional any more and installer is no longer generated. Some discussion has been going on in ROS2 Humble and Rolling incompatibility · Issue #75 · ros-planning/warehouse_ros_mongo · GitHub

Hi sjahr,

I see. The main branch switched to using the Cpp interface.
Which is probably the better way.

I’ve also encountered another problem inside the Humble Benchmark package. The path length is always the same for each run and each planner. So according to the benchmark all planners return a plan with the same length.
Is this also the case for the Benchmark package on the main branch?
Sadly, I cannot compile the package on the main branch to use it in Humble.

No, I did not encounter this issue on main. If you’re having build issues feel free to open an issue in the moveit 2 main repos with the terminal output and I can help you to solve it. Just in case you have not seen it: Here are the MoveIt 2 source build instructions that should :wink: work MoveIt 2 Source Build - Linux | MoveIt

Hi Valentino,

Do you know is there any off-the-shelf script to generate, for example, hundreds of start/goal robot state for warehouse to dump a sqlite file? I’ve read the tutorial, it seems that it only shows you how to use rviz to manually add start/goal state.