Running the image on an amd64 host (which is possible with binfmt and the bundled qemu-arm-static binary present on the image) does not exhibit these issues.
I’ve tried the image on an ARM host running 16.04 and 18.04 but not with 20.04 yet.
Alternatively, would it be possible to just stick with using amd64 host for the docker daemon + static QEMU installed, or update the build farm workers for ARM to use the same kernel version as the targeted LTS?