14 August 2022

When docker images stop being portable

Recently, I spent some time managing and improving our build servers, and one of the tasks was phasing out an old Ubuntu 16.04 server and setting up the build on a Ubuntu 20.04 server, to ensure an up to date and secure build environment. Nothing easier than that. The build is dockerized and thus effortlessly portable! - Or so I thought. What ensued was quite the adventure.

Let's just pull the latest version of the docker image on the new server and we should be good to go, right?

builduser@newbuildserver:~$ docker pull foo/openwrt Using default tag: latest latest: Pulling from foo/openwrt f01457d30a62: Pull complete b85c74a04473: Pull complete 51d30eb98965: Extracting [==================================================>] 2.111GB/2.111GB e92b5b1dbca9: Download complete 0e62a6b87170: Download complete failed to register layer: lstat /var/lib/docker/overlay2/483c5d4d2907b24c5bed845aadd49b927c6ccff5e231b7878317e3a22383e4c8/diff/home/builduser/openwrt/build_dir/host/findutils-4.4.2/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3: file name too long

The image fails to be pulled. Well, that's not good. Looks like something recursively generated a large number of confdir3 directories and failed to clean them up. Note that the Dockerfile that generated this docker image is more or less a stock OpenWrt 15.05 "Chaos Calmer" build (quite an old version, actually), which means that this issue is caused by the build system of the official release. What could possibly cause such a ridiculous number of nested directories?

There are quite a few threads by distraught developers who have been bitten by this issue. I found an old bug that describes a problem caused by those same directory names: https://savannah.gnu.org/bugs/?57461. The comments are very interesting indeed:

The problem is clearly not specific to cpio or findutils but is caused by the shared code from gnulib's getcwd-path-max.m4 in your configure scripts. If this hasn't already been fixed in the latest gnulib, the problem should be reported to the authors of gnulib.

Oh, so this is caused by a script that figures out the maximum path length? Well, it certainly found the maximum path length! And it seems like this maximally long path found its way right into the docker image, and after some years, the docker engine could no longer handle paths of this length. Also, the configure script should delete the bogus directories after it's done probing filesystem limits, so that certainly is a bug. The old adage "don't be too clever" comes to mind - GNU tools are usually quite good at balancing pragmatism and cleverness appropriately, but this entire thing looks insane to me.

The next question is why do we suddenly run into problems now, when it used to work just fine? The old build server accepted such docker images, allowing me to push, pull and run them, but the new one does not and neither does my Windows 10 machine with Docker Desktop. Perhaps something changed in docker?

I gathered information with all the usual commands like docker version (not to be confused with docker --version which provides much less information) and docker info on both servers and spent a lot of time analysing the differences and thinking. Then I found something that caught my attention:

Old server:

builduser@oldbuildserver:~$ docker info [...] Storage Driver: aufs Root Dir: /var/lib/docker/aufs

New server:

builduser@newbuildserver:~$ docker info [...] Storage Driver: overlay2

with the root directory of the new server being /var/lib/docker/overlay2. Perhaps OverlayFS has a different path length than aufs? Suddenly, I had an idea. No... It can't be! Let us compare the two docker storage root directories:

/var/lib/docker/aufs /var/lib/docker/overlay2

It looks like overlay2 is exactly 4 characters longer than aufs. Let's see what this does to that huge confdir3 path. Yep, that does it. With aufs, the long path above becomes 4095 characters, with overlay2 it is 4099 characters (all the other elements of the path remain the same length). On Linux, the maximum path length for most tools is usually 4096 bytes (and thus characters, if we're talking about plain ASCII paths). So it seems to me that the longer name of OverlayFS in the path of the docker storage root directory causes this problem.

Okay, the hypothesis stands, so it's time to launch some virtual machines to verify it, just as the scientific method prescribes. Ubuntu 20.04 still supports aufs, unlike Ubuntu 22.04. Because Ubuntu 20.04 supports both aufs and OverlayFS, it is the best choice to perform the test. Docker comes with OverlayFS by default and, as expected, the docker image fails to pull.

I run:

sudo nano /etc/docker/daemon.json

and create the config file with the aufs configuration:

{ "storage-driver": "aufs" }

Then, to finish switching to aufs, all I need to do is:

sudo service docker restart

Hint: If you mess with the docker configuration and the docker service fails to restart, use sudo dockerd --debug to troubleshoot the issue. The service manager will recommend See "systemctl status docker.service" and "journalctl -xe" for details. but both these logs are completely irrelevant.

Sure enough, as predicted, the image can be pulled just fine with aufs! We found the culprit.

We now know why it doesn't work, but that doesn't tell us how we can rectify the situation. We need this old docker container, and it has to run on all the servers, including those that only have OverlayFS. Of course I still have the original Dockerfile, so could we just rebuild the docker image from scratch? Nope, we cannot. The instructions in the Dockerfile pull code from a repository that no longer exists. And even if it existed, buildroot downloads files from all over the place. Who knows which of these mirrors are still up and serve the same files? I certainly don't. It will be impossible to recreate the image exactly as it was built back then. Now that's a lesson on docker image longevity, vendoring and Dockerfile design, isn't it? Someone who tries to build your Dockerfile in five years will pull their hair out - if they have any left.

Because I wanted to preserve and run the exact docker image, as it was built back then, I opted for the minimally invasive method: Delete the directory with the long path and merge together (squash) that layer with the original layer. This will ensure that everything is exactly the same, except that this directory is gone. And because the layers are squashed, there is no intermediary layer that exhibits the problem. Of course, this has to be done on an old server or VM that still runs docker with aufs. A tiny Dockerfile suffices:

FROM foo/openwrt RUN rm -rf /home/builduser/openwrt/build_dir/host/findutils-4.4.2/confdir3/confdir3

Now we just have to build this image and squash the layers. Newer docker versions have the --squash option, however, I used an old VM where this was unavailable, therefore I used the handy python library docker-squash to do the same. Unfortunately, that library requires Python 3.7 which the VM didn't have, so I had to install that manually as well. I also learned that 21GB of disk storage are not enough to perform a squash operation on a 7GB layer, so I had to grow the VM disk space after it ran out of space after 40 minutes, and then I had to restart squashing the layers from scratch. Yes, somehow it almost took an hour. That day, it seemed like all my tools conspired against me to waste as much of my time and patience as possible.

Eventually, however, I emerged the victor: The new docker image and the build worked on the new server right away. Let's just hope that this story doesn't repeat when, one day, docker uses a new storage driver whose name is longer than overlay2.

Just for the sake of entertainment, let's go deeper down the rabbit hole. Let's create a minimal reproducible example (MRE). Our first attempt is the following Dockerfile:

FROM debian:bookworm RUN mkdir -p /home/builduser/openwrt/build_dir/host/findutils-4.4.2/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3

An interesting facet is that you can build this Dockerfile with OverlayFS, but it will not create the very last confdir3 folder and therefore silently omit that directory and succeed, which I find baffling. After all, we're talking about docker silently losing a folder when building a Dockerfile.

Note that we depend on debian here. We can't just use FROM scratch and use the COPY instruction. That will fail with failed to copy files: evalSymlinksInScope: too many links. There are no symlinks here, so this error message is buggy. The cause of the error can be found in the corresponding go code which has now been moved to a separate repository. This logic limits the directory depth to 255. So, instructions with RUN mkdir can nest deeper than those with COPY because of this arbitrary limitation. Fine, then we'll just make less directories with longer names. File names usually can't be longer than 255 bytes, so we choose a sensible length of 100 (including the slash) and create a new Dockerfile. Note that in each directory, docker will create a pivot_root directory with a name like .pivot_root3853263278. This behaviour can also be found in the source code. Since at the end, we only have four characters left, this leads to an error that looks like this: failed to copy files: failed to copy file: Error processing tar file(exit status 1): Error setting up pivot dir: mkdir [...]/.pivot_root3853263278: file name too long. For that reason, we skip the last slash and make the last element of the path a bit longer and end up with a Dockerfile like this:

FROM scratch COPY Dockerfile /01_llllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllll/02_llllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllll/03_llllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllll/04_llllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllll/05_llllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllll/06_llllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllll/07_llllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllll/08_llllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllll/09_llllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllll/10_llllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllll/11_llllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllll/12_llllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllll/13_llllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllll/14_llllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllll/15_llllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllll/16_llllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllll/17_llllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllll/18_llllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllll/19_llllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllll/20_llllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllll/21_llllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllll/22_llllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllll/23_llllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllll/24_llllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllll/25_llllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllll/26_llllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllll/27_llllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllll/28_llllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllll/29_llllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllll/30_llllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllll/31_llllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllll/32_llllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllll/33_llllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllll/34_llllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllll/35_llllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllll/36_llllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllll/37_llllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllll/38_llllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllll/39_llllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllll/40_llllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllll_41_l

Once this docker image is built with aufs storage, it can be pushed to a registry or saved into a file with docker save -o longpath.tar longpath. Then, with OverlayFS, the image cannot be pulled or imported from the file because the file name is too long. This Dockerfile, unlike the one above that depends on debian, will fail to build on OverlayFS and not silently omit any files or directories.

We see that when we push docker to its limits, it is very fragile: It stops being portable, it silently loses files when building, it exhibits misleading error messages. It is a tool that creates an abstract portable environment, but the quirks and complexity it has to keep under the lid are too much to handle. It is a classic example of a leaky abstraction and we better be aware of its limitations, lest they bite us.