Feedback Wanted: Test framework for verifying topic output (both unit and integration testing)

lkwatson · 2018-07-10T20:32:09.439Z

Background of package
I’ve been working on a project that (I believe) simplifies testing nodes. I have a working proof of concept that permits one to simply supply inputs (currently, just messages but extensible) and verify outputs to expected values. I’ve also been experimenting with methods of supplying parametric rules (near mappings) between input and output for testing a full parameter space.

I’m looking for feedback, and in the future some testers, to see which direction I should grow this project.

Example
Take for example a node that filters a Twist message /cmd_vel. Let’s propose it does two things, first it clamps the message to a certain velocity magnitude. Then, let’s say it subscribes to a Float64 message (or perhaps in the future, a dynamic parameter) /accel_max that correspondingly considers the change in velocity and clamps acceleration.

Current work
The first and biggest pain point I see in my mind is defining tests. Rather than supplying a few test cases, I’d love to be able to just give the functional mapping of input to output and say “make sure the node follows this”. However for even relatively few degrees of freedom, this becomes costly in time.

To test such a node, I could create several extreme test cases to ensure clamping, and others to ensure acceleration limiting.

However, It would be very nice to be able to describe that velocity magnitude v is clamped to a max value, and dv/dt should correspond with /accel_max. The risk of such a modality is requiring the developer maintain both the node and the tests, which for complex behavior may be unreasonable.

One modality I’ve also considered/experimented with is allowing one to provide input, record a node (or many nodes) output, and then use this as a base to test against in future work. However I don’t know how useful this would be in broad application.

For you
First, I’m interested in how you might envision using such a tool (if at all), and what features would be most desired.

Then, I’m seeking any feedback on how to best define an input/output space for testing, and establishing those relationships.

fkromer · 2018-07-11T06:52:45.887Z

It could probably be interesting for you to integrate with boschresearch/ros_fmu “(… which provides) wrapping functional mockup units (FMUs) for co-simulation of physical models into ROS nodes.” to generate input space…

Martin_Guenther · 2018-07-11T11:31:36.128Z

I like your ideas and think this could be very useful. However, one should keep in mind that writing a complete specification of the expected behavior of a node is as complex as writing the node itself; if you had a complete mapping from input to output, you could just execute it instead of running the node. So writing a full functional mapping of input to output is probably infeasible. In your example, you’re doing something else: You specify invariants that should hold (velocity magnitude is clamped, and dv/dt corresponds with accel_max), while leaving open some degrees of freedom in the behavior of the node. I think this is the way to do it.

By the way, are you familiar with rostest? An example where it’s used are the tests for robot_pose_ekf. Here they just test some simple properties of the node’s output topics and check that the final estimated pose is correct. This is probably enough to catch 95% of regression bugs.

lkwatson · 2018-07-12T23:13:03.384Z

This is quite interesting, thanks for this! I’ll have to look into it further.

lkwatson · 2018-07-12T23:20:26.719Z

Martin_Guenther:

I like your ideas and think this could be very useful. However, one should keep in mind that writing a complete specification of the expected behavior of a node is as complex as writing the node itself; if you had a complete mapping from input to output, you could just execute it instead of running the node. So writing a full functional mapping of input to output is probably infeasible. In your example, you’re doing something else: You specify invariants that should hold (velocity magnitude is clamped, and dv/dt corresponds with accel_max), while leaving open some degrees of freedom in the behavior of the node. I think this is the way to do it.

This is a great way to put it. My concern is precisely how to express invariants generally enough such that development of the node (or nodes) isn’t hindered by the tests. Cumbersome tests are likely to be ignored if they break and are hard to fix.

Martin_Guenther:

By the way, are you familiar with rostest ?

Yes, should have mentioned that this will ideally become a rostest node where pubs/subs/invariants are supplied via yaml.