Build/CI failures with ros-testing repository enabled?

I’m not entirely sure this already ‘deserves’ a post here, but I’m seeing increased build failures (and CI failures) for builds which use the ros-testing repository. Regular builds (ie: using the main ros repository) succeed.

Example of buildfarm build which failed: Kbin_uX64__abb_driver__ubuntu_xenial_amd64__binary. Test PR with Travis CI build which fails when ros-testing repository is enabled: ros-industrial/ur_msgs#3.

In both cases errors about “broken packages” are reported. From the buildfarm build (log):

Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:

The following packages have unmet dependencies:
 ros-kinetic-industrial-robot-client : Depends: ros-kinetic-actionlib but it is not going to be installed
                                       Depends: ros-kinetic-industrial-utils but it is not going to be installed
                                       Depends: ros-kinetic-robot-state-publisher but it is not going to be installed
                                       Depends: ros-kinetic-urdf but it is not going to be installed
E: Unable to correct problems, you have held broken packages.

I don’t know whether there are any other builds failing on the buildfarm, as I can’t get any tabs other than the manage tab to load.

Other builds with similar failures:

The swarmros failure log seems to point to something having to do with Python 2 and rospkg, rosdistro and rosdep:

35 'Broken ros-kinetic-roslib:amd64 Depends on ros-kinetic-rospack [ amd64 ] < none -> 2.4.5-1xenial-20191214-014035+0000 > ( misc )'
02:16:20   36 '  Considering ros-kinetic-rospack:amd64 -1 as a solution to ros-kinetic-roslib:amd64 9998'
02:16:20   37 '    Reinst Failed early because of python2:amd64'
02:16:20   38 '    Reinst Failed because of python-rospkg-modules:amd64'
02:16:20   39 '    Reinst Failed because of python-rosdistro-modules:amd64'
02:16:20   40 '    Reinst Failed because of python-rosdistro:amd64'
02:16:20   41 '    Reinst Failed because of python-rosdep:amd64'
2 Likes

This looks like it could be related to yesterday’s 1.2.7 release of rospkg. I am taking a look now.

Yes. It looks like the recent release of python-rospkg is Broken on Xenial and Bionic

The following packages have unmet dependencies:
 python-rospkg : Depends: python2 (< 2.8) but it is not installable
                 Depends: python2 (>= 2.7) but it is not installable
                 Depends: python-rospkg-modules (>= 1.2.7) but it is not going to be installed
E: Unable to correct problems, you have held broken packages.

I thought this was resolved with stdeb 0.9.0 but when looking closer I realized that rospkg is missing the necessary config option. I’ll get working on the fix and an updated release.

https://github.com/ros-infrastructure/rospkg/pull/204 makes the necessary configuration change and the 1.2.8 release is importing now I pulled it directly from the bootstrap repository to verify it’s installability on Ubuntu Xenial and it should be the same for other python 2 platforms.

Thanks @nuclearsandwich.

I’m not entirely sure from your description, but should this be available from the ros-testing repositories already?

I’ve just re-ran https://github.com/ros-industrial/ur_msgs/pull/3 and it still fails (with what seems to be the same error).

I think you were a bit too quick at the draw :cowboy_hat_face:. The import_upstream job which pulls packages from the bootstrap repository into the build farm repositories (all three of building, testing, and main at once) finished about 20 minutes ago as of this writing but the upload_main and upload_testing jobs to push from the build farm repositories to the primary repositories on packages.ros.org had not yet run.

Those jobs have now also run so it should be broadly available as of now.

I was just impatient :slight_smile:

Just re-ran all the tests and things seem to be working again.

Thanks for the fixes @nuclearsandwich.

Edit: could you trigger a build for the abb_driver release job I referenced in the OP? Afaict it should build, but of course I can’t trigger it.


Would there be a way to add some sort of testing for this to the rospkg repository (and other repositories with Python 2 & 3 packages)? This particular case would have been caught by a test which attempts to install the package on all supported platforms.

Perhaps this is already done and I just haven’t checked btw (in fact: I haven’t :innocent:).

It wouldn’t be a bad idea to add a canary installation attempt when we push new releases to the bootstrap repository but we don’t have an existing apparatus for doing so which means it’s not a small task to incorporate such a pre-flight check. Because the problem exists only with the release artifacts it’s not something that we could add to the current CI for these projects.

I’ve opened a round of PRs for the other ros-infrastructure packages making sure that they at least have the necessary config values. As our Ubuntu 20.04 environments stabilize these kinds of regressions should dissipate. At least for another two years.

This bug affects also the images on dockerhub. Particularly, I’m using https://hub.docker.com/layers/ros/library/ros/melodic-robot/images/sha256-211835c5fe5b99138062dd99709bd9c5f4ee8ac2d76ead720b5b927e611df35d?context=explore . Could you please ping those responsible to update the images immediately?

Looking at the history of these jobs neither of them has ever built succesfully. From a brief look at the latest builds the reasons seems to be:

  • missing to declare a dependency on dual_quaternions or cv_bridge
  • an incorrect install directive
  • missing an include directory (or dependency) on rosbag_msgs

These images are not trivially rebuildable. The simplest fix I can think of (if any needed) without relying on external infrastructure is to upgrade rospkg when entering the container (run an apt update && apt upgrade).

I did not get a chance to reproduce on arm64 but I’ve had no issue using the amd64 images recently. Can you provide a reproducible example of the issue you are facing ?

The last build of these images was 8 days ago, meaning they do not have the faulty release of rospkg installed (that was released 3 days ago and fixed with another release 2 days ago). So I suspect the problem you are facing may be a different one

1 Like

I’m not sure about reproducible docker examples, as I’m actually building a singularity container. The definition file is:

Bootstrap: docker
From: ros:melodic-robot

%files
    singularity_apt_cache.conf /etc/apt/apt.conf.d/singularity_apt_cache.conf

%post

    export CCACHE_DIR=/tmp/singularity-cache/ccache # configure CCache dir
    export XDG_CACHE_HOME=/tmp/singularity-cache # pip cache

    apt-get -y update
    apt-get -y upgrade
    apt-get -y install curl

    curl -sSL 'http://packages.osrfoundation.org/gazebo.key' | apt-key add -
    echo "deb http://packages.osrfoundation.org/gazebo/ubuntu-stable $(lsb_release -sc) main" > /etc/apt/sources.list.d/gazebo-latest.list

    packages="
    build-essential
    ccache
    crudini
    gazebo9
    gdb
    git-gui
    htop
    libgazebo9-dev
    mc
    net-tools
    nmap
    openssh-client
    openssh-server
    python-catkin-tools
    python-pip
    python-pip-whl
    python-rosinstall
    python-rosinstall-generator
    python-scipy
    python-virtualenv
    python-wstool
    ros-melodic-cartographer-ros
    ros-melodic-cartographer-rviz
    ros-melodic-desktop-full
    ros-melodic-octomap-ros
    ros-melodic-tf2-geometry-msgs
    ros-melodic-tf2-sensor-msgs
    screen
    sshpass
    terminator
    tmux
    tree
    vim
    wget
    "
    apt-get -y update && apt-get -y install $packages
Summary
Docker image path: index.docker.io/library/ros:melodic-robot
Cache folder set to /root/.singularity/docker
[14/14] |===================================| 100.0% 
Exploding layer: sha256:d7c3167c320d7a820935f54cf4290890ea19567da496ecf774e8773b35d5f065.tar.gz
Exploding layer: sha256:131f805ec7fd68d45a887e2ef82de61de0247b4eb934ab03b7c933650e854baa.tar.gz
Exploding layer: sha256:322ed380e680a77f30528ba013e3a802a7b44948a0609c7d1d732dd46a9a310d.tar.gz
Exploding layer: sha256:6ac240b130982ad1c3ba3188abbf18ba4e54bdd9e504ce2d5c2eff6d3e86b8dd.tar.gz
Exploding layer: sha256:b64a7083ea96b95855d3cee6bcab14b90b20d9bfd209d123ecb0caa88322c7ae.tar.gz
Exploding layer: sha256:bc602a4d0af8ead24cc13540f49a0fff8887a95b19f7167d71c673468d08954e.tar.gz
Exploding layer: sha256:b13ebfa6253ab146de4b680f495660841c32931dfc6640051a14000c8c18ac49.tar.gz
Exploding layer: sha256:e357f79e762b301a72bfc29c53b477116db00825dab096e69c4b22309aa72edf.tar.gz
Exploding layer: sha256:4426cc1b5ab808a5916e33dd02e4b7bd626d04ee88a6f0020db75bf20ab8eaba.tar.gz
Exploding layer: sha256:a1c6421f726415d8084d253ffe8760cd196ebc9e5488080af5b60e09a1bd388c.tar.gz
Exploding layer: sha256:e888aaa5c69683911554f32e367447730fe4effe123cb12b4b4ceeae26264f5e.tar.gz
Exploding layer: sha256:3f16d9e160e2c666350cc63944bab979f121556389769e635b955743686f52e4.tar.gz
Exploding layer: sha256:532d314c79ae06bfd76459b5138747166f47e1edeeee3b9dd6bbfc53648ae718.tar.gz
Exploding layer: sha256:acfa0dc0ce33a1d0bfc7d24b07994cb8836eea805b2fd5f6d353fe1db48b93d1.tar.gz
Exploding layer: sha256:ab0af5e90bfe73486419683632515d704c4822e8f8d5be5cfd769f0231a270d4.tar.gz
Adding files to container
Copying 'singularity_apt_cache.conf' to '/etc/apt/apt.conf.d/singularity_apt_cache.conf'
Running post scriptlet
+ export CCACHE_DIR=/tmp/singularity-cache/ccache
+ export XDG_CACHE_HOME=/tmp/singularity-cache
+ apt-get -y update
Get:1 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB]
Get:2 http://archive.ubuntu.com/ubuntu bionic InRelease [242 kB]                           
Get:3 http://archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB]                            
Get:4 http://archive.ubuntu.com/ubuntu bionic-backports InRelease [74.6 kB]                     
Get:5 http://security.ubuntu.com/ubuntu bionic-security/restricted amd64 Packages [82.2 kB]               
Get:6 http://security.ubuntu.com/ubuntu bionic-security/universe amd64 Packages [862 kB]                                         
Get:7 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages [977 kB]                              
Get:8 http://archive.ubuntu.com/ubuntu bionic/main amd64 Packages [1344 kB]                                                    
Get:9 http://security.ubuntu.com/ubuntu bionic-security/multiverse amd64 Packages [9012 B]                                               
Get:10 http://packages.ros.org/ros/ubuntu bionic InRelease [4669 B]                                                                       
Get:11 http://archive.ubuntu.com/ubuntu bionic/restricted amd64 Packages [13.5 kB]                                    
Get:12 http://archive.ubuntu.com/ubuntu bionic/universe amd64 Packages [11.3 MB]       
Get:13 http://archive.ubuntu.com/ubuntu bionic/multiverse amd64 Packages [186 kB]     
Get:14 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse amd64 Packages [13.4 kB]
Get:15 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages [1270 kB]
Get:16 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 Packages [1398 kB]
Get:17 http://packages.ros.org/ros/ubuntu bionic/main amd64 Packages [676 kB]    
Get:18 http://archive.ubuntu.com/ubuntu bionic-updates/restricted amd64 Packages [93.9 kB]  
Get:19 http://archive.ubuntu.com/ubuntu bionic-backports/universe amd64 Packages [8158 B] 
Get:20 http://archive.ubuntu.com/ubuntu bionic-backports/main amd64 Packages [8286 B]      
Fetched 18.8 MB in 2s (8775 kB/s)                                                          
Reading package lists... Done
+ apt-get -y upgrade
Reading package lists... Done
Building dependency tree       
Reading state information... Done
Calculating upgrade... Done
The following packages will be upgraded:
  libcurl3-gnutls libcurl4 linux-libc-dev python-catkin-pkg python-catkin-pkg-modules python-rospkg python-rospkg-modules
7 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Need to get 1489 kB of archives.
After this operation, 2048 B of additional disk space will be used.
Get:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libcurl3-gnutls amd64 7.58.0-2ubuntu3.9 [212 kB]
Get:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libcurl4 amd64 7.58.0-2ubuntu3.9 [214 kB]
Get:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-libc-dev amd64 4.15.0-108.109 [991 kB]
Get:4 http://packages.ros.org/ros/ubuntu bionic/main amd64 python-catkin-pkg-modules all 0.4.22-1 [42.1 kB]
Get:5 http://packages.ros.org/ros/ubuntu bionic/main amd64 python-catkin-pkg all 0.4.22-100 [3516 B]
Get:6 http://packages.ros.org/ros/ubuntu bionic/main amd64 python-rospkg-modules all 1.2.8-1 [23.8 kB]
Get:7 http://packages.ros.org/ros/ubuntu bionic/main amd64 python-rospkg all 1.2.8-100 [2066 B]
Fetched 1489 kB in 0s (3561 kB/s)       
debconf: delaying package configuration, since apt-utils is not installed
(Reading database ... 62835 files and directories currently installed.)
Preparing to unpack .../0-libcurl3-gnutls_7.58.0-2ubuntu3.9_amd64.deb ...
Unpacking libcurl3-gnutls:amd64 (7.58.0-2ubuntu3.9) over (7.58.0-2ubuntu3.8) ...
Preparing to unpack .../1-libcurl4_7.58.0-2ubuntu3.9_amd64.deb ...
Unpacking libcurl4:amd64 (7.58.0-2ubuntu3.9) over (7.58.0-2ubuntu3.8) ...
Preparing to unpack .../2-linux-libc-dev_4.15.0-108.109_amd64.deb ...
Unpacking linux-libc-dev:amd64 (4.15.0-108.109) over (4.15.0-106.107) ...
Preparing to unpack .../3-python-catkin-pkg-modules_0.4.22-1_all.deb ...
Unpacking python-catkin-pkg-modules (0.4.22-1) over (0.4.20-1) ...
Preparing to unpack .../4-python-catkin-pkg_0.4.22-100_all.deb ...
Unpacking python-catkin-pkg (0.4.22-100) over (0.4.20-100) ...
Preparing to unpack .../5-python-rospkg-modules_1.2.8-1_all.deb ...
Unpacking python-rospkg-modules (1.2.8-1) over (1.2.6-1) ...
Preparing to unpack .../6-python-rospkg_1.2.8-100_all.deb ...
Unpacking python-rospkg (1.2.8-100) over (1.2.6-100) ...
Setting up python-catkin-pkg-modules (0.4.22-1) ...
Setting up libcurl3-gnutls:amd64 (7.58.0-2ubuntu3.9) ...
Setting up libcurl4:amd64 (7.58.0-2ubuntu3.9) ...
Setting up linux-libc-dev:amd64 (4.15.0-108.109) ...
Setting up python-rospkg-modules (1.2.8-1) ...
Setting up python-catkin-pkg (0.4.22-100) ...
Setting up python-rospkg (1.2.8-100) ...
Processing triggers for libc-bin (2.27-3ubuntu1) ...
+ apt-get -y install curl
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following NEW packages will be installed:
  curl
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 159 kB of archives.
After this operation, 396 kB of additional disk space will be used.
Get:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 curl amd64 7.58.0-2ubuntu3.9 [159 kB]
Fetched 159 kB in 0s (1248 kB/s)
debconf: delaying package configuration, since apt-utils is not installed
Selecting previously unselected package curl.
(Reading database ... 62835 files and directories currently installed.)
Preparing to unpack .../curl_7.58.0-2ubuntu3.9_amd64.deb ...
Unpacking curl (7.58.0-2ubuntu3.9) ...
Setting up curl (7.58.0-2ubuntu3.9) ...
+ + curl -sSL http://packages.osrfoundation.org/gazebo.key
apt-key add -
OK
+ lsb_release -sc
+ echo deb http://packages.osrfoundation.org/gazebo/ubuntu-stable bionic main
+ packages=
    build-essential
    ccache
    crudini
    gazebo9
    gdb
    git-gui
    htop
    libgazebo9-dev
    mc
    net-tools
    nmap
    openssh-client
    openssh-server
    python-catkin-tools
    python-pip
    python-pip-whl
    python-rosinstall
    python-rosinstall-generator
    python-scipy
    python-virtualenv
    python-wstool
    ros-melodic-cartographer-ros
    ros-melodic-cartographer-rviz
    ros-melodic-desktop-full
    ros-melodic-octomap-ros
    ros-melodic-tf2-geometry-msgs
    ros-melodic-tf2-sensor-msgs
    screen
    sshpass
    terminator
    tmux
    tree
    vim
    wget
    
+ apt-get -y update
Hit:1 http://archive.ubuntu.com/ubuntu bionic InRelease
Hit:2 http://security.ubuntu.com/ubuntu bionic-security InRelease                                                                         
Hit:3 http://archive.ubuntu.com/ubuntu bionic-updates InRelease                                                                                                     
Hit:4 http://archive.ubuntu.com/ubuntu bionic-backports InRelease                                                                                                   
Hit:5 http://packages.ros.org/ros/ubuntu bionic InRelease                                                                                     
Get:6 http://packages.osrfoundation.org/gazebo/ubuntu-stable bionic InRelease [4256 B]
Get:7 http://packages.osrfoundation.org/gazebo/ubuntu-stable bionic/main amd64 Packages [64.0 kB]
Fetched 68.3 kB in 1s (55.5 kB/s)   
Reading package lists... Done
+ apt-get -y install build-essential ccache crudini gazebo9 gdb git-gui htop libgazebo9-dev mc net-tools nmap openssh-client openssh-server python-catkin-tools python-pip python-pip-whl python-rosinstall python-rosinstall-generator python-scipy python-virtualenv python-wstool ros-melodic-cartographer-ros ros-melodic-cartographer-rviz ros-melodic-desktop-full ros-melodic-octomap-ros ros-melodic-tf2-geometry-msgs ros-melodic-tf2-sensor-msgs screen sshpass terminator tmux tree vim wget
Reading package lists... Done
Building dependency tree       
Reading state information... Done
build-essential is already the newest version (12.4ubuntu1).
python-rosinstall is already the newest version (0.7.8-1).
python-wstool is already the newest version (0.1.17-1).
python-wstool set to manually installed.
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:

The following packages have unmet dependencies:
 python-rosinstall-generator : Depends: python2 (< 2.8) but it is not installable
                               Depends: python2 (>= 2.7) but it is not installable
                               Depends: python2:any (>= 2.6.6-7~) but it is not installable
E: Unable to correct problems, you have held broken packages.

So the failure is now with python-rosinstall-generator.

Okay, python-rosinstall-generator had a known issue that should be fixed by now: https://github.com/ros-infrastructure/rosinstall_generator/issues/70 .

It was actually fixed in ros-infrastructure/rosinstall_generator#68. This is one of the PRs in the series @nuclearsandwich mentions.

The issue you link to describes problems with updating the local apt indices of the reporter (and follow up commenters).

Also (and off-topic): nice to see Singularity being used :slight_smile: