ROS Resources: Documentation | Support | Discussion Forum | Service Status | Q&A answers.ros.org

Pytest support for ROS / rostest


#1

Hello everyone,

I want to stir some interest in supporting pytest for ROS. pytest is a very popular unit testing framework for Python.

In my opinion, pytest is far easier to use than Python unittest and nosetests supported by ROS as of now. My personal experience says the easier something is to do, the likelier it is to be done. As we are talking about unit tests, an essential part of professional software development and QA, I think it’s even more important here. I have outline more about why this is necessary in a blog post.

The easiest way to support pytest within ROS is to use a new test runner that wraps pytest and extracts the JUnit XML output. Here is my attempt at doing that ros_pytest

ROS seems to pick up the test data very well. However, I noticed that failing tests still don’t trigger a failure when running rostests (but the log/terminal output shows the error). I would appreciate any input on why this is the case.

I joined the ROS community a few months ago and I want to make my first contribution to the project. So please let me know how if you think this is a good idea and how I can proceed.

Alex


#2

BTW: Congratulations on winning the Austrian Open Source Award 2017 with Machinekit :slight_smile:


#3

Great to see an initiative like this in testing! A few questions after a glance:

In your blog post you said:

The really great thing about pytest is that it really makes unit testing easy and quick. To shorten you feedback loops I recommend you to clearly distinguish between unit tests and ROS node or integration tests.

After reading the blog, I didn’t quite get how pytest makes unit testing “easy and quick”. I mean it looks like it does, but did you find it easier and quicker than unittest / nosetests that ROS is already integrated with? I’m simply curious, as ROS allows to run unit testing by nosetests as well as integrate it into Node testing.

ROS seems to pick up the test data very well. However, I noticed that failing tests still don’t trigger a failure when running rostests (but the log/terminal output shows the error). I would appreciate any input on why this is the case.

Not sure what you mean by “ROS seems to pick up the test data very well”. But I’d agree the test result is not the easiest to read. In your particular case, have you tried catkin_test_results?


#4

After reading the blog, I didn’t quite get how pytest makes unit testing “easy and quick”. I mean it looks like it does, but did you find it easier and quicker than nosetests that ROS is already integrated with? I’m simply curious, as ROS allows to run unit testing by nosetests as well as integrate it into Node testing.

I should have elaborated on this. I have only tried nosetests for testing parts of Machinekit. From what I know it just extends Python unittest, which is not great after all. nosetests also supports test functions (no need to write a complex test runner class), which is great. However, pytest seems to be a lot easier when it comes to writing and fixtures. Additionally, it makes great use of assert, including meaningful error messages. Writing pytests feels in my opinion just more natural and pythonic.

Not sure what you mean by “ROS seems to pick up the test data very well”. But I’d agree the test result is not the easiest to read. In your particular case, have you tried catkin_test_results?

ros$ catkin_test_results 
build/ros_pytest/test_results/ros_pytest/rosunit-test_lib.xml: 2 tests, 0 errors, 1 failures, 0 skipped
Summary: 3 tests, 0 errors, 1 failures, 0 skipped

Actually looks good. I just wondered why it does not “crash” or abort with an error message when the tests are run. But I’m not that familiar with ROS yet.


ROSCon 2018 Talk Idea Brainstorm
#5

I’m afraid I’m not sure what you mean, but you don’t want the whole chain of tests to stop in the middle just because of a single test case failed, do you? I agree that the output can be improved to be more intuitive though.

Apparently PyTest seems to be getting popular. It would be nice for it to be integrated into core library so that we can use it without defining special dependency etc (AFAIUC, your current approach requires a special test runner script). Would you interested in going further down this road (if so I’d definitely like to help)?


ROSCon 2018 Talk Idea Brainstorm
#6

I concur that pytest feels “easier” / more natural to use (obviously a subjective point). It certainly is more actively being developed and offers more features through the numerous available plugins. For ROS 2 we are using pytest over nose for a while for these reasons.

The new build tool colcon also invokes Python tests with pytest. Since pytest is capable of running tests written for nose this is usually compatible with exiting tests.

There are three parts to Python unit testing to consider:

  • the API used to write unit tests
  • the tool used to perform the testing of a package
  • the tool used by rostest internally

I think the three parts are fairly independent from each other. E.g. using colcon you can run the tests of a ROS package with pytest even though they use nose / unittest API.


#7

In the Python web dev domain pytest is the standard tool for testing distributed systems. If one wants to get an idea about in what non public projects it could be used I suggest to read about the history of the pytest initiator (Holger Krekel).

I would help out in adding pytest support in ROS1.


#8

The ros_pytest package is ready. I just created a PR for the Kinetic release: https://github.com/ros/rosdistro/pull/18443


#9

The PR has been merged. ros_pytest is now available as package!