ROS Resources: Documentation | Support | Discussion Forum | Service Status | Q&A answers.ros.org

CI coverage issues and trying out the llvm coverage tool

We can see the coverage report in CI job, https://ci.ros2.org/view/All/job/ci_linux_coverage/. But it looks to have some issues.

(1) Function coverages look inaccurate in both .cpp, .hpp.
(2) We cannot see per line report because sources are not attached in CI report.

In our experiment, llvm looks to be able to address (1).
Is anyone else doing the similar things? Questions, suggestions and advice are welcome.

In the following section, we describe the issue in detail and llvm experiment.

issue detail

Here is a report of rclcpp/include/create_subscription_hpp.

You can see function coverage is 41% (78/192), and line coverage is 39%(9/23).
Obviously, there are too many functions(192 functions in 23 line???)

And you can not check which functions or lines are missed as source file is not uploaded.
Additionally, inline functions look to be ignored, as we describe below.

Another example of .hpp is rclcpp/include/create_publisher.hpp. We get 100%(5/5) line coverage… but 85%(28/33) function coverage.

Here is a sample of .cpp.
You can see function coverage is 86%(6/7).
As we cannot see which functions or lines are missed, we reproduced the coverage test locally and found all functions were passed in per line result, but gcov says 86% function coverage. I don’t know why, but it looks strange.

Regarding .hpp function coverage, it looks that template or inline functions are not handled well by gcov.

By the way, llvm distinguishes “templated functions and their instantiations.”

So we tried llvm source coverage.

summary of llvm source coverage

We used the following docker ROS2 devel image and built ROS2 from the source.

We built rclcpp by clang with coverage options.

$ export CXX=clang++-10
$ colcon build --symlink-install   --build-base=build-scov   --install-base=install-scov   --packages-select rclcpp   --cmake-force-configure   --cmake-args -DCMAKE_CXX_FLAGS="-fprofile-instr-generate -fcoverage-mapping"

And run tets. To avoid profile result to be overwrited, we set LLVM_PROFILE_FILE.

# run colcon test. 
$ export LLVM_PROFILE_FILE="%p.profraw"
$ colcon test --build-base build-scov --install-base install-scov --packages-select rclcpp

We got may profraw files and merged.

$ ls -l build-scov/rclcpp/test/rclcpp/*profraw
-rw-r--r-- 1  1629504 12 24 18:22 build-scov/rclcpp/test/rclcpp/20486.profraw
-rw-r--r-- 1  1629520 12 24 18:22 build-scov/rclcpp/test/rclcpp/20488.profraw
(snip)

$ llvm-profdata-10 merge -sparse build-scov/rclcpp/test/rclcpp/*profraw -o build-scov/rclcpp/test/test.profdata

To get report, we need to specify at least one binary file and append extra files by -object option.
If you use “show” command instead of “report”, you can see per line reports.

$ llvm-cov-10 report -instr-profile=build-scov/rclcpp/test/test.profdata -Xdemangler=c++filt \
   build-scov/rclcpp/librclcpp.so \
   -object build-scov/rclcpp/test/rclcpp/test_add_callback_groups_to_executor \
   -object build-scov/rclcpp/test/rclcpp/test_allocator_common 
   (snip)

Filename                                                                                                         Regions    Missed Regions     Cover   Functions  Missed Functions  Executed       Lines      Missed Lines     Cover
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
(snip)
rclcpp/rclcpp/src/rclcpp/any_executable.cpp                                                                            4                 0   100.00%           2                 0   100.00%           9                 0   100.00%
rclcpp/rclcpp/src/rclcpp/callback_group.cpp                                                                           21                 1    95.24%          16                 0   100.00%          78                 0   100.00%
(snip)

We compared llvm and gcov results for rclcpp get_node_base_interface.hpp (I simply chose this one because it was at the top).
We can see “Regions” result only in llvm, and function/line coverages are different.

Regions Missed Regions Cover Functions Missed Functions Executed Lines Missed Lines Cover
llvm 7 2 71.43% 4 1 75.00% 17 5 70.59%
gcov 100.0% (4/4) 87.5 %(7/8)

Regions coverage

This is known as Focused Expression Coverage.
We can distinguish 50% or 100% coverage of return x || y && z e.g.
As 2 missed regions exist, we may need more tests.

Functions/Lines coverage

We found that inline function get_node_base_interface is not called by using llvm “show” command.

   94|       |inline
   95|       |std::shared_ptr<rclcpp::node_interfaces::NodeBaseInterface>
   96|       |get_node_base_interface(
   97|       |  std::shared_ptr<rclcpp::node_interfaces::NodeBaseInterface> & node_interface)
   98|      0|{
   99|      0|  return node_interface;
  100|      0|}

Additionally, gcov ignores this function. The following lines are from lcov report.
Though this function is not executed, gcov reports “100% function coverage”.

      94             : inline
      95             : std::shared_ptr<rclcpp::node_interfaces::NodeBaseInterface>
      96             : get_node_base_interface(
      97             :   std::shared_ptr<rclcpp::node_interfaces::NodeBaseInterface> & node_interface)
      98             : {
      99             :   return node_interface;
     100             : }

Instantiation coverage

In llvm, we can get instantiatoin coverage. For example, get_node_base_interface_from_pointer is instatiated when NodeType == std::shared_ptr<rclcpp::Node> and NodeType == NodeWrapper*.
We may get more insight by looking into these report.

   45|       |// If NodeType has a method called get_node_base_interface() which returns a shared pointer.
   46|       |template<
   47|       |  typename NodeType,
   48|       |  typename std::enable_if<has_node_base_interface<
   49|       |    typename rcpputils::remove_pointer<NodeType>::type
   50|       |  >::value, int>::type = 0
   51|       |>
   52|       |std::shared_ptr<rclcpp::node_interfaces::NodeBaseInterface>
   53|       |get_node_base_interface_from_pointer(NodeType node_pointer)
   54|      5|{
   55|      5|  if (!node_pointer) {
   56|      0|    throw std::invalid_argument("node cannot be nullptr");
   57|      0|  }
   58|      5|  return node_pointer->get_node_base_interface();
   59|      5|}
  ------------------
  | std::shared_ptr<rclcpp::node_interfaces::NodeBaseInterface> rclcpp::node_interfaces::detail::get_node_base_interface_from_pointer<std::shared_ptr<rclcpp::Node>, 0>(std::shared_ptr<rclcpp::Node>):
  |   54|      3|{
  |   55|      3|  if (!node_pointer) {
  |   56|      0|    throw std::invalid_argument("node cannot be nullptr");
  |   57|      0|  }
  |   58|      3|  return node_pointer->get_node_base_interface();
  |   59|      3|}
  ------------------
  | std::shared_ptr<rclcpp::node_interfaces::NodeBaseInterface> rclcpp::node_interfaces::detail::get_node_base_interface_from_pointer<NodeWrapper*, 0>(NodeWrapper*):
  |   54|      2|{
  |   55|      2|  if (!node_pointer) {
  |   56|      0|    throw std::invalid_argument("node cannot be nullptr");
  |   57|      0|  }
  |   58|      2|  return node_pointer->get_node_base_interface();
  |   59|      2|}
  ------------------

Finally, we desribe .hpp or .cpp results described in “issue detail”, and one more interesting exxample. callback_group.cpp has 100% line coverage but has one missed region.

Filename                                                                                                         Regions    Missed Regions     Cover   Functions  Missed Functions  Executed       Lines      Missed Lines     Cover
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
-----------------------------------------

rclcpp/rclcpp/include/rclcpp/create_publisher.hpp                                                                      5                 0   100.00%           3                 0   100.00%          29                 0   100.00%
rclcpp/rclcpp/include/rclcpp/create_subscription.hpp                                                                  14                 1    92.86%           4                 0   100.00%          89                 4    95.51%
rclcpp/rclcpp/src/rclcpp/executors/multi_threaded_executor.cpp                                                        44                 1    97.73%           5                 0   100.00%          72                 2    97.22%

rclcpp/rclcpp/src/rclcpp/callback_group.cpp                                                                           21                 1    95.24%          16                 0   100.00%          78                 0   100.00%

What do we do next?

We think we try to check rclcpp coverage in more detail and add more tests.
Additionally, we want to set up things for other developers to use llvm coverage tool more easily if it look good.
If you have any advice or suggestions please let us know.

Thank you.

2 Likes

The first thing to be careful of is not to look at the ci_linux_coverage job. That is the job that is used in development, so it could represent the state of an unmerged or in progress PR. You should always look at the nightly jobs, i.e. https://ci.ros2.org/view/nightly/job/nightly_linux_coverage/

The way that the coverage work definitely leads to a bit of inaccuracy. I don’t remember the precise details; @brawner may be able to shed more light here.

That being said, I don’t remember the coverage report being wildly off.

It’s not that the sources are not attached; its that for some reason, you need to be logged into the Jenkins instance in order to access them. I’ve asked the internal infrastructure team to see if we can lift that restriction.

Even with the above issues solved, I agree that the current code coverage is difficult to read. However, I don’t think that is particularly because of the backend tool we use. I think it is more of a presentation problem on the Jenkins frontend. Instead of changing the tool to llvm, I’d like to see if we could:

  1. Filter out the non-src directories from the listing.
  2. Allow non-logged in users to see the coverage report.

I would only look at changing the backend to llvm if it turns out that those things aren’t possible with the current system.

1 Like

@clalancette Thank you for comment.

You should always look at the nightly jobs, i.e. https://ci.ros2.org/view/nightly/job/nightly_linux_coverage/

Thank you, I do so.

you need to be logged into the Jenkins instance in order to access them. I’ve asked the internal infrastructure team to see if we can lift that restriction.

Can I create account or log into the Jenkins? I thgouth only OSRF or on-boaring member can do it.
Anyway, it looks better if we can lift that restriction.

Filter out the non-src directories from the listing.

We may use lcov --remove option.
I got the following output by this command in my environment.

lcov --remove cov_rclcpp_exclude -o cov_rclcpp_exclude2.info \
  '*/build-cov/*' '*/install-cov/*' '*/test/*'

At the moment, we have restricted access to getting a Jenkins account.

Looking at the CI code, we are currently filtering out /usr/*, /home/rosbuild/*, */test/*, */tests/*, *gtest_vendor*, and *gmock_vendor*. So it looks like we could augment this to filter the build and install directory as well.

Looking at the CI code, we are currently filtering out /usr/, /home/rosbuild/, /test/, /tests/, gtest_vendor, and gmock_vendor. So it looks like we could augment this to filter the build and install directory as well.

I agree.

Additionally, we aim “Quality: Increase testing coverage of C/C++ packages” in ros2 Roadmap.
But I feel it is difficult to know C/C++ packages coverage.

From this perspective, these views might be useful:

  • enable to see the coverage of C/C++ packages and other packages separately
  • or report coverage not by directories but by packages

I agree it is a little hard to know now; we’re looking into the permissions on the backend to see if we can make this data available without a login.

In the meantime, you can calculate the coverage yourself by following the instructions at ROS 2 on-boarding guide . It’s a little clunky, but it does get the job done. Keep in mind that to get the full coverage numbers, you have to run all of the tests in the system.

I agree that these would be nice. We’d have to invest some time into a Jenkins plugin to do that; it might be worthwhile, but I’m not sure.