Some things to know as Python 2 approaches EOL

General
,
1

Hey folks. We got an interesting question from a customer today, and I think the answer might be helpful to a wider audience.

Python 2 will reach end of life in two months. This shouldn’t be news to anyone who hasn’t been living under a rock, and plans are in place to use Python 3 in Noetic (whereas ROS 2 has always used Python 3). However, the question from our customer was this: What does that mean for existing ROS 1 distributions (Kinetic and Melodic)? They are still using (and will continue to use) Python 2.

The answer really depends on where you’re getting Python 2. Tl;dr: If you’re using Ubuntu Xenial (16.04) or Bionic (18.04), please know that Python 2 from the Ubuntu repositories will continue to be supported for the lifetime of the Ubuntu release, regardless of Python 2’s upstream support status.

Ubuntu’s package repositories are split into a number of components: main, restricted, universe, and multiverse. You can read that link to learn all about them, but let me briefly quote it so you can understand the difference between main and universe:

main:

The main component contains applications that are free software, can be freely redistributed and are fully supported by the Ubuntu team. […] When you install software from the main component, you are assured that the software will come with security updates and that commercial technical support is available from Canonical.

universe:

The universe component is a snapshot of the free, open-source, and Linux world. […] Canonical does not provide a guarantee of regular security updates for software in the universe component, but will provide these where they are made available by the community.

All Ubuntu releases prior to Eoan (19.10) included Python 2 in main. In Eoan it was demoted to universe. This means that for Xenial (16.04) and Bionic (18.04), Python 2 falls under the main guarantees quoted above. It will continue to be fully-supported and receive free security updates.

In other words, as long as you’re running Kinetic or Melodic against its recommended Ubuntu release targets, you should be golden.

11 Likes
2

(hopefully this is the correct venue for this followup question)
One of our issues comes from the fact that we use rosdep to resolve dependencies and some times you need to use a pip dependency for a myriad of reasons. Since rosdep (to my knowledge) doesn’t allow us freeze pip package versions we end up in a situation where it can end up installing a python 3 version of a package (or if the package maintainer is smart we end up with an install time / CI time crash).

Our current approach is to fix package versions using ansible (and remove those packages from out package.xml which is really not ideal). Is there a recommended fix for that for Kinetic users?

1 Like
3

On Arch we already build all melodic packages against python3, works totally fine, you only always have to pass --cmake-args -DPYTHON_EXECUTABLE=/usr/bin/python3" to as catkin cmake option. Most simply put this into an alias.

1 Like
4

Totally the right place; that’s an excellent question. To be absolutely clear, anything not contained within the “main” component of an Ubuntu repository does not receive that guarantee of support from Canonical, and this includes pip dependencies. I agree that you’re in a bit of a sticky spot if you’re using a pip dependency that ends up dropping support for Python 2 in its codebase, because you’re right: as far as I know rosdep doesn’t support a freeze or requirements.txt files like pip does.

I know that REP 140 allows for version qualifiers, perhaps rosdep can grow support for that. However, I’d also be concerned with pip itself dropping support for Python 2. Make sure you’re installing pip from the Ubuntu repos as well, in that case.

I agree, removing those from your package.xml isn’t ideal. Perhaps rosdep install can grow support for --filter-for-installers.

Worst case, I don’t think it would be overly difficult to write your own dependency discovery tool using the catkin_pkg API (so you can extract the version qualifiers), use rosdep to resolve the keys into actual dependencies, and then do the installing/freezing logic yourself. You would of course need to deal with potential version clashes (e.g. package A depends on foo v1.1, package B requires at least foo v1.2).

@nuclearsandwich might have some insight to share on this as well.

5

I can only sit in my armchair and daydream/brainstorm as I do not ship ROS on robots myself. Iterating on @kyrofa’s suggestion, if I had an ansible based deployment one thing I’d be keen to try is using rosdep as part of an optional pre-run step that feeds dependency information into ansible rather than using it during deployment. rosdep has a lot of moving parts and I wouldn’t want it running on every deploy. Invoking rosdep as a part of the pre-ansible pipeline also means you don’t have to filter anything out of the package.xml. This would allow you to re-deploy independent of updating dependencies. If I were depending on packages from pip in a production environment I’d also be very keen to keep a local mirror containing the pip packages I need. Package index mirrors and caches - Python Packaging User Guide

3 Likes
6

Care to explain this part:

Python 2 falls under the main guarantees quoted above. It will continue to be fully-supported and receive free security updates.

The security updates for python 2 come from the python developers. They are not going to bother with securing python 2.

And to quote Guido:

Let’s not play games with semantics. The way I see the situation for 2.7 is that EOL is January 1st, 2020, and there will be no updates, not even source-only security patches, after that date. Support (from the core devs, the PSF, and python.org) stops completely on that date. If you want support for 2.7 beyond that day you will have to pay a commercial vendor. Of course it’s open source so people are also welcome to fork it. But the core devs have toiled long enough, and the 2020 EOL date (an extension from the originally annouced 2015 EOL!) was announced with sufficient lead time and fanfare that I don’t feel bad about stopping to support it at all.

1 Like
7

The security updates for python 2 come from the python developers. They are not going to bother with securing python 2.

You are of course, correct, assuming you’re consuming python 2 straight from the python developers. However, if you’re using the python from the Ubuntu archive, you’re not: you’re consuming it from Ubuntu. Going back to Guido’s quote, I suppose you could call that a fork, but this is just how distros work. Ubuntu releases are supported long enough that it’s not unusual for its archives to include software that upstream considers end-of-life, but we have teams dedicated to supporting them in Ubuntu. That’s what makes an Ubuntu release supported.

1 Like