I want to stir some interest in supporting pytest for ROS. pytest is a very popular unit testing framework for Python.
In my opinion, pytest is far easier to use than Python unittest and nosetests supported by ROS as of now. My personal experience says the easier something is to do, the likelier it is to be done. As we are talking about unit tests, an essential part of professional software development and QA, I think it’s even more important here. I have outline more about why this is necessary in a blog post.
The easiest way to support pytest within ROS is to use a new test runner that wraps pytest and extracts the JUnit XML output. Here is my attempt at doing that ros_pytest
ROS seems to pick up the test data very well. However, I noticed that failing tests still don’t trigger a failure when running rostests (but the log/terminal output shows the error). I would appreciate any input on why this is the case.
I joined the ROS community a few months ago and I want to make my first contribution to the project. So please let me know how if you think this is a good idea and how I can proceed.
The really great thing about pytest is that it really makes unit testing easy and quick. To shorten you feedback loops I recommend you to clearly distinguish between unit tests and ROS node or integration tests.
After reading the blog, I didn’t quite get how pytest makes unit testing “easy and quick”. I mean it looks like it does, but did you find it easier and quicker than unittest / nosetests that ROS is already integrated with? I’m simply curious, as ROS allows to run unit testing by nosetests as well as integrate it into Node testing.
ROS seems to pick up the test data very well. However, I noticed that failing tests still don’t trigger a failure when running rostests (but the log/terminal output shows the error). I would appreciate any input on why this is the case.
Not sure what you mean by “ROS seems to pick up the test data very well”. But I’d agree the test result is not the easiest to read. In your particular case, have you tried catkin_test_results?
After reading the blog, I didn’t quite get how pytest makes unit testing “easy and quick”. I mean it looks like it does, but did you find it easier and quicker than nosetests that ROS is already integrated with? I’m simply curious, as ROS allows to run unit testing by nosetests as well as integrate it into Node testing.
I should have elaborated on this. I have only tried nosetests for testing parts of Machinekit. From what I know it just extends Python unittest, which is not great after all. nosetests also supports test functions (no need to write a complex test runner class), which is great. However, pytest seems to be a lot easier when it comes to writing and fixtures. Additionally, it makes great use of assert, including meaningful error messages. Writing pytests feels in my opinion just more natural and pythonic.
Not sure what you mean by “ROS seems to pick up the test data very well”. But I’d agree the test result is not the easiest to read. In your particular case, have you tried catkin_test_results?
Actually looks good. I just wondered why it does not “crash” or abort with an error message when the tests are run. But I’m not that familiar with ROS yet.
I’m afraid I’m not sure what you mean, but you don’t want the whole chain of tests to stop in the middle just because of a single test case failed, do you? I agree that the output can be improved to be more intuitive though.
Apparently PyTest seems to be getting popular. It would be nice for it to be integrated into core library so that we can use it without defining special dependency etc (AFAIUC, your current approach requires a special test runner script). Would you interested in going further down this road (if so I’d definitely like to help)?
I concur that pytest feels “easier” / more natural to use (obviously a subjective point). It certainly is more actively being developed and offers more features through the numerous available plugins. For ROS 2 we are using pytest over nose for a while for these reasons.
The new build tool colcon also invokes Python tests with pytest. Since pytest is capable of running tests written for nose this is usually compatible with exiting tests.
There are three parts to Python unit testing to consider:
the API used to write unit tests
the tool used to perform the testing of a package
the tool used by rostest internally
I think the three parts are fairly independent from each other. E.g. using colcon you can run the tests of a ROS package with pytest even though they use nose / unittest API.
In the Python web dev domain pytest is the standard tool for testing distributed systems. If one wants to get an idea about in what non public projects it could be used I suggest to read about the history of the pytest initiator (Holger Krekel).
I would help out in adding pytest support in ROS1.
Has anyone else run into issues using catkin_test_results with results from pytest on the ROS Kinetic version?
Pytest --junitxml output produces an xml file with the results under a child of root named ‘testsuites’.
However in the Kinetic version of catkin_test_results in read_junit2 it only parses the root node and thus fails to find any of the failures or errors.
The Melodic version of read_junit2 actually checks the child nodes and so picks up failures and errors correctly.
Is there a pytest option to make all the results in the root xml node?
Are there any examples on how to use ros_pytest?
I am having some troubles with it, when I run my tests my node gets stuck and I get a timeout. When I run a regular script with the same code, it works. I am wondering if it might have something do with how I configured my tests
