Thanks for the feedback, @Paulinho1!
I got lost when you were commenting the tool overview slide.
I was afraid that one was going to be problematic. It’s got too much information into a single slide.
Indeed, some kind of animation, or at least some highlights of the parts I am talking about, should be included.
Can I use it for ros 1?
Yes. It is intended for ROS 1 only, for now. ROS 2 is still on the roadmap.
How long it takes to install?
Installation should not take long. Depending on the plugins you want to use, it only takes the time to download a few packages with
pip, or the time to clone the repositories if you run it from source.
How long to set up a test?
If you do not take into account the time it takes to come up with (and write down) properties, generating a test is also relatively quick.
It should only take a few seconds, unless you are running it for the first time for a given system (or if the ROS code has been modified since you last ran it). In that case, it has to “compile” the code again, to update the architecture model, and that takes about as long as normal compilation with
catkin_make or similar.
How long to run a satysfying set of test?
Depends on settings, but running property-based tests takes a while because each property is automatically tested many times with different inputs.
Unfortunately, for each input, it has to launch the ROS system under test, run the test and kill the system, so that the next input is guaranteed to start from a fresh state.
If the property is false, it should not take long to find a counterexample. But, if it is true, you can expect to wait a few minutes, until it gives up.
I tested systems with hundreds of properties (both true and false), and I got an average of 5 or 6 seconds per input (half of it is
roslaunch). True properties should run hundreds of inputs, false properties often find the bug in 15 to 30 inputs.
If there was a standard mechanism to reset nodes to the original state at runtime, without killing and launching them again, this would be much shorter. I have seen cases (e.g., Kobuki) where they have a
/reset topic, and I modified some tests by hand to use it, but it is not standard.
some examples would be insightful
Indeed. Thanks for the suggestion. I will include some hands-on information in a future version.