We can see the coverage report in CI job, https://ci.ros2.org/view/All/job/ci_linux_coverage/. But it looks to have some issues.
(1) Function coverages look inaccurate in both .cpp, .hpp.
(2) We cannot see per line report because sources are not attached in CI report.
In our experiment, llvm looks to be able to address (1).
Is anyone else doing the similar things? Questions, suggestions and advice are welcome.
In the following section, we describe the issue in detail and llvm experiment.
issue detail
Here is a report of rclcpp/include/create_subscription_hpp.
You can see function coverage is 41% (78/192), and line coverage is 39%(9/23).
Obviously, there are too many functions(192 functions in 23 line???)
And you can not check which functions or lines are missed as source file is not uploaded.
Additionally, inline functions look to be ignored, as we describe below.
Another example of .hpp is rclcpp/include/create_publisher.hpp. We get 100%(5/5) line coverage… but 85%(28/33) function coverage.
Here is a sample of .cpp.
You can see function coverage is 86%(6/7).
As we cannot see which functions or lines are missed, we reproduced the coverage test locally and found all functions were passed in per line result, but gcov says 86% function coverage. I don’t know why, but it looks strange.
Regarding .hpp function coverage, it looks that template or inline functions are not handled well by gcov.
By the way, llvm distinguishes “templated functions and their instantiations.”
So we tried llvm source coverage.
summary of llvm source coverage
We used the following docker ROS2 devel image and built ROS2 from the source.
We built rclcpp by clang with coverage options.
$ export CXX=clang++-10
$ colcon build --symlink-install --build-base=build-scov --install-base=install-scov --packages-select rclcpp --cmake-force-configure --cmake-args -DCMAKE_CXX_FLAGS="-fprofile-instr-generate -fcoverage-mapping"
And run tets. To avoid profile result to be overwrited, we set LLVM_PROFILE_FILE.
# run colcon test.
$ export LLVM_PROFILE_FILE="%p.profraw"
$ colcon test --build-base build-scov --install-base install-scov --packages-select rclcpp
We got may profraw files and merged.
$ ls -l build-scov/rclcpp/test/rclcpp/*profraw
-rw-r--r-- 1 1629504 12 24 18:22 build-scov/rclcpp/test/rclcpp/20486.profraw
-rw-r--r-- 1 1629520 12 24 18:22 build-scov/rclcpp/test/rclcpp/20488.profraw
(snip)
$ llvm-profdata-10 merge -sparse build-scov/rclcpp/test/rclcpp/*profraw -o build-scov/rclcpp/test/test.profdata
To get report, we need to specify at least one binary file and append extra files by -object option.
If you use “show” command instead of “report”, you can see per line reports.
$ llvm-cov-10 report -instr-profile=build-scov/rclcpp/test/test.profdata -Xdemangler=c++filt \
build-scov/rclcpp/librclcpp.so \
-object build-scov/rclcpp/test/rclcpp/test_add_callback_groups_to_executor \
-object build-scov/rclcpp/test/rclcpp/test_allocator_common
(snip)
Filename Regions Missed Regions Cover Functions Missed Functions Executed Lines Missed Lines Cover
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
(snip)
rclcpp/rclcpp/src/rclcpp/any_executable.cpp 4 0 100.00% 2 0 100.00% 9 0 100.00%
rclcpp/rclcpp/src/rclcpp/callback_group.cpp 21 1 95.24% 16 0 100.00% 78 0 100.00%
(snip)
We compared llvm and gcov results for rclcpp get_node_base_interface.hpp (I simply chose this one because it was at the top).
We can see “Regions” result only in llvm, and function/line coverages are different.
| Regions | Missed Regions | Cover | Functions | Missed Functions | Executed | Lines | Missed Lines | Cover | |
|---|---|---|---|---|---|---|---|---|---|
| llvm | 7 | 2 | 71.43% | 4 | 1 | 75.00% | 17 | 5 | 70.59% |
| gcov | 100.0% (4/4) | 87.5 %(7/8) |
Regions coverage
This is known as Focused Expression Coverage.
We can distinguish 50% or 100% coverage of return x || y && z e.g.
As 2 missed regions exist, we may need more tests.
Functions/Lines coverage
We found that inline function get_node_base_interface is not called by using llvm “show” command.
94| |inline
95| |std::shared_ptr<rclcpp::node_interfaces::NodeBaseInterface>
96| |get_node_base_interface(
97| | std::shared_ptr<rclcpp::node_interfaces::NodeBaseInterface> & node_interface)
98| 0|{
99| 0| return node_interface;
100| 0|}
Additionally, gcov ignores this function. The following lines are from lcov report.
Though this function is not executed, gcov reports “100% function coverage”.
94 : inline
95 : std::shared_ptr<rclcpp::node_interfaces::NodeBaseInterface>
96 : get_node_base_interface(
97 : std::shared_ptr<rclcpp::node_interfaces::NodeBaseInterface> & node_interface)
98 : {
99 : return node_interface;
100 : }
Instantiation coverage
In llvm, we can get instantiatoin coverage. For example, get_node_base_interface_from_pointer is instatiated when NodeType == std::shared_ptr<rclcpp::Node> and NodeType == NodeWrapper*.
We may get more insight by looking into these report.
45| |// If NodeType has a method called get_node_base_interface() which returns a shared pointer.
46| |template<
47| | typename NodeType,
48| | typename std::enable_if<has_node_base_interface<
49| | typename rcpputils::remove_pointer<NodeType>::type
50| | >::value, int>::type = 0
51| |>
52| |std::shared_ptr<rclcpp::node_interfaces::NodeBaseInterface>
53| |get_node_base_interface_from_pointer(NodeType node_pointer)
54| 5|{
55| 5| if (!node_pointer) {
56| 0| throw std::invalid_argument("node cannot be nullptr");
57| 0| }
58| 5| return node_pointer->get_node_base_interface();
59| 5|}
------------------
| std::shared_ptr<rclcpp::node_interfaces::NodeBaseInterface> rclcpp::node_interfaces::detail::get_node_base_interface_from_pointer<std::shared_ptr<rclcpp::Node>, 0>(std::shared_ptr<rclcpp::Node>):
| 54| 3|{
| 55| 3| if (!node_pointer) {
| 56| 0| throw std::invalid_argument("node cannot be nullptr");
| 57| 0| }
| 58| 3| return node_pointer->get_node_base_interface();
| 59| 3|}
------------------
| std::shared_ptr<rclcpp::node_interfaces::NodeBaseInterface> rclcpp::node_interfaces::detail::get_node_base_interface_from_pointer<NodeWrapper*, 0>(NodeWrapper*):
| 54| 2|{
| 55| 2| if (!node_pointer) {
| 56| 0| throw std::invalid_argument("node cannot be nullptr");
| 57| 0| }
| 58| 2| return node_pointer->get_node_base_interface();
| 59| 2|}
------------------
Finally, we desribe .hpp or .cpp results described in “issue detail”, and one more interesting exxample. callback_group.cpp has 100% line coverage but has one missed region.
Filename Regions Missed Regions Cover Functions Missed Functions Executed Lines Missed Lines Cover
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
-----------------------------------------
rclcpp/rclcpp/include/rclcpp/create_publisher.hpp 5 0 100.00% 3 0 100.00% 29 0 100.00%
rclcpp/rclcpp/include/rclcpp/create_subscription.hpp 14 1 92.86% 4 0 100.00% 89 4 95.51%
rclcpp/rclcpp/src/rclcpp/executors/multi_threaded_executor.cpp 44 1 97.73% 5 0 100.00% 72 2 97.22%
rclcpp/rclcpp/src/rclcpp/callback_group.cpp 21 1 95.24% 16 0 100.00% 78 0 100.00%
What do we do next?
We think we try to check rclcpp coverage in more detail and add more tests.
Additionally, we want to set up things for other developers to use llvm coverage tool more easily if it look good.
If you have any advice or suggestions please let us know.
Thank you.
