I’ve used Docker for Mac since the Beta release opened to wider audiences. With the rapid prototyping I’m doing on Hadoop environments, I’m finding it great for providing quick environments to test out theories.
Problem: How do you access the Docker for Mac VM?
The problem with a black box is not being able to easily get inside to diagnose weird behavior. Previously, I was using boot2docker on Virtualbox to run containers on my Mac. This was pretty easy to get into: you could pop open the console of the VM in the Virtualbox control panel and do what you need to do.
With the Docker for Mac install, it’s a bit more obtuse. I ran across this following method while digging around Github issues for the second problem below.
Solution: Accessing the VM
$ screen ~/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/tty
Pretty simple. I was able to poke around the VM pretty easily once I figured this out.
For example, let’s take a quick look at the output of df (which is relevant for the next problem).
/ # df Filesystem 1K-blocks Used Available Use% Mounted on tmpfs 1022944 161836 861108 16% / tmpfs 204592 192 204400 0% /run cgroup_root 10240 0 10240 0% /sys/fs/cgroup dev 10240 0 10240 0% /dev shm 1022944 0 1022944 0% /dev/shm /dev/vda1 19049892 17360568 698600 96% /var /dev/vda1 19049892 17360568 698600 96% /var osxfs 243941376 181997504 61687872 75% /Users osxfs 243941376 181997504 61687872 75% /Volumes osxfs 243941376 181997504 61687872 75% /tmp osxfs 243941376 181997504 61687872 75% /private osxfs 243941376 181997504 61687872 75% /host_docker_app /dev/vda1 19049892 17360568 698600 96% /run/log osxfs 243941376 181997504 61687872 75% /var/log /dev/vda1 19049892 17360568 698600 96% /var/lib/docker/aufs
From here, we can see the /dev/vda1 volume which maps to the underlying Qcow2 VM image. We can see the osxfs filesystems which are overlaid volumes mounted from the host to make it easier to pass data into and out of the container. Pretty nice. I’m not sure why we see double mounts for things like /var. I suspect it’s just an oddity in the way this VM works.
To exit, kill your screen session by typing Control-A then k and answer “yes”.
This leads me into the second problem I was encountering today.
Problem: Filesystem full?
While creating some pretty large containers for a test, I encountered docker build failures pointing to some filesystem being full. This error was coming from yum as it was building out the required packages for the container.
Error Summary ------------- Disk Requirements: At least 442MB more space needed on the / filesystem.
My first inclination was to make sure my Mac wasn’t full; it wasn’t. The next step was to clear out old containers that were no longer needed. I know the Qcow2 image is pretty small at 20 gigabytes, so that doesn’t leave much space if I’ve made a few of these large builds. I cleared out a few gigabytes of old containers and their images, but it didn’t help. I kept running into these space issues despite having cleaned out everything but a base OS container.
Going back to that Github issue, I was able to use the screen hack to get in and found that the Docker container cache inside the VM was on a volume that was full.
/var/lib/docker/aufs # du -smx * 16717 diff 1 layers 1 mnt
The bulk of the disk usage was in /var/lib/docker/aufs. This is where the container diffs get cached. Reading through various other Github issues, it sounds like these can occasionally orphan themselves which prevents them from getting cleaned up properly.
Well, that’s likely my problem here, but how do I get that cleaned up?
Solution: Garbage collect the container diffs
This is where the spotify/docker-gc tools come in. I ran across this in some of the discussions. It looked pretty straightforward and I was fine running it. If it screwed some of my existing containers up, I could just rebuild them. I ended up using the docker image for this.
$ docker run --rm -v /var/run/docker.sock:/var/run/docker.sock -v /etc:/etc spotify/docker-gc
It dumped a bunch of output like the following.
Unable to find image 'spotify/docker-gc:latest' locally latest: Pulling from spotify/docker-gc 2352f4f477e3: Pull complete be723c23f85f: Pull complete 40827ff2c763: Pull complete Digest: sha256:0a87fcd289d526ec534d23a7c82f7ea27cecafe60c650eb8b1c877ef3bfc2a88 Status: Downloaded newer image for spotify/docker-gc:latest Container running 45832fb87d28d920cd3774618e09f1505fd9e5fd40cf1d509e037d6a10932064 /gigantic_colden Container not running 03476687dad17a0a4ba53d0d46679711dfc43afae891fc43287d65561e3b047a /berserk_banach Container not running 03d4297dc90d091ea19858eab097e62ab61b1e4c54b2ce707ade76f49df62264 /peaceful_roentgen Container not running 04bf8b01bdf2f7348f22453ccf8a4b6dc485295477031b2108bd59c20f013c1a /hungry_hawking Container not running 09cad64371cb1e3cdcfb898d4ac3736e4ee9eae53dc3a3760975ed684c32db3e /determined_lamarr . . [lots of output deleted] . . Removing containers fa23ff819a5bfb66e931079284106f6584c0c2c7e3e208610b8526e8c4a3e703 /pedantic_franklin Removing containers fd073598306b884c739d2fd99d79056958ac4f4a786b305d0be72061702b7551 /naughty_swartz Removing containers fe9250b1c4ca2512c1d1b7464ec58d40f2c457315cc39622f4ac12af5abd9c49 /goofy_hypatia Removing image sha256:04dac6a8672c0d311fa1ccd323e5826d84320c9032761c70f98a88d72238c6cf [ansible-aws:latest] Removing image sha256:6e5c17caa1307c4a8c5fe2f50d9e24bf3a3ec864a9bca1045571b7bb42b1b546 [xenapi:latest] Removing image sha256:778a53015523d89bd807dab131cf9b8bb65f661ddaed8e24038817cdca42d576 [centos:latest] Removing image sha256:bd9d6812fdd0187ffeb1737a3a601c268eeded59454186b1f9ceafd1f00e07df [quay.io/dtan4/terraforming:latest] Removing image sha256:d0a31e3494fe61ed2ff0387cd0e71e237394c413f7a0dfca9a33b1319bab499c [centos:6] Removing image sha256:d0e7f81ca65cdd391b6eb3dd3ce2454a575023156cd932ee4a58f188436bc5e0 [centos:7] Removing image sha256:f753707788c5c100f194ce0a73058faae1a457774efcda6c1469544a114f8644 [ubuntu:latest]
Looking at df now showed that the space utilization dropped down from 96% to 17%. Much better. Running my docker build showed that it was succeeding again.
Clean early, clean often
This problem appears often in the Docker community from what I can tell. Spotify’s tool is pretty good and tries to be as safe as possible. I was happy to use it in my test environment. If you decide to use it for yours, you should test it before running it against your production containers. You never know what might happen.
