Disclosure: I may earn affiliate revenue or commissions if you purchase products from links on my website. The prospect of compensation does not influence what I write about or how my posts are structured. The vast majority of articles on my website do not contain any affiliate links.
A rare thing happened to me on Friday–I ran out of work. I didn’t run out in the sense that I sprinted to the elevator and down Jackson Boulevard. That happens with some frequency. I ran out of work in that I didn’t have much meaningful work to do. Even this is a bit of an exaggeration, but I was waiting on a few merge requests and the office had mostly vacated by mid afternoon. I checked out some of the single-serving O’Reilly books on DevOps, but then out of the corner of my eye I saw the Docker in Practice book on a colleague’s desk. I was a bit apprehensive to pick it up since it was published in 2016 (and thus potentially written in 2015), but I figured it would be the best use of my time. It turned out to be a great read. You can find my short review here.
The book is split up into 4 sections and 101 “techniques” for solving problems with Docker. I found myself really engaging with some of the techniques and writing a lot in my small notebook. I’ve never structured a post this way. What I’m about to do is highlight select techniques from the book accompanied by my own commentary.
Technique 4: Use socat to monitor Docker API traffic
Seeing Docker as a black box rather than a client-daemon with an API was the cause of many of my early follies. Immediately resorting to GitHub issue threads when encountering problems has prevented me from firming my grasp on the technology. Socat (SOcket CAT) is a useful tool that I didn’t know about until I read this technique. It allows you to inspect that API calls being made to the Docker socket. Since it can be used between any bidirectional byte streams, this tool is probably well known to network engineers.
sudo socat -v UNIX-LISTEN:/tmp/dockerapi.sock UNIX-CONNECT:/var/run/docker.sock &then
docker -H unix:///tmp/dockerapi.sock any-docker-commandThe resulting output will include the HTTP request, the HTTP response, the JSON content, and then the output you would normally see. For running docker images, the results aren’t particularly interesting, but I can see socat becoming an invaluable debugging tool for me, especially while first digging into an issue.
[root@ansible-2 ~]# docker -H unix:///tmp/dockerapi.sock images
> 2018/04/08 17:23:21.118605 length=79 from=0 to=78
GET /_ping HTTP/1.1\r
Host: docker\r
User-Agent: Docker-Client/1.13.1 (linux)\r
\r
< 2018/04/08 17:23:21.119281 length=196 from=0 to=195
HTTP/1.1 200 OK\r
Api-Version: 1.26\r
Docker-Experimental: false\r
Server: Docker/1.13.1 (linux)\r
Date: Sun, 08 Apr 2018 17:23:21 GMT\r
Content-Length: 2\r
Content-Type: text/plain; charset=utf-8\r
\r
OK> 2018/04/08 17:23:21.119807 length=91 from=79 to=169
GET /v1.26/images/json HTTP/1.1\r
Host: docker\r
User-Agent: Docker-Client/1.13.1 (linux)\r
\r
< 2018/04/08 17:23:21.120936 length=889 from=196 to=1084
HTTP/1.1 200 OK\r
Api-Version: 1.26\r
Content-Type: application/json\r
Docker-Experimental: false\r
Server: Docker/1.13.1 (linux)\r
Date: Sun, 08 Apr 2018 17:23:21 GMT\r
Content-Length: 702\r
\r
[{"Containers":-1,"Created":1521838695,"Id":"sha256:8e518564f5b3ed28920a279f7f440423dc2ba566d2bf9a8e12bc385ad7f30394","Labels":null,"ParentId":"","RepoDigests":["docker.io/haproxy@sha256:7f5e0872c9c91d9666a8c7f79489e0582d8a45fbd55e835a15be1212a8ab6b45"],"RepoTags":["docker.io/haproxy:latest"],"SharedSize":-1,"Size":71010374,"VirtualSize":71010374},{"Containers":-1,"Created":1515547359,"Id":"sha256:d1fd7d86a8257f3404f92c4474fb3353076883062d64a09232d95d940627459d","Labels":null,"ParentId":"","RepoDigests":["docker.io/registry@sha256:672d519d7fd7bbc7a448d17956ebeefe225d5eb27509d8dc5ce67ecb4a0bce54"],"RepoTags":["docker.io/registry:latest"],"SharedSize":-1,"Size":33281203,"VirtualSize":33281203}]
REPOSITORY TAG IMAGE ID CREATED SIZE
docker.io/haproxy latest 8e518564f5b3 2 weeks ago 71 MB
docker.io/registry latest d1fd7d86a825 2 months ago 33.3 MBTechnique 12: Splitting a system into microservice containers
A wise man once told me that the best way to learn Docker is by splitting up monoliths and practicing dockerizing your favorite libraries. He was right, which is why I still regard him as a wise man. While dockerizing libraries might just be a fun coding exercise, splitting up monoliths is a fun, useful application of Docker. By splitting a docker container into pure component parts and thus eliminating sources of bloat or bastardization, you're able to enjoy a more stable, more agile environment everywhere from dev to production.
This technique appears early in the book and doesn't cover the composition (ahem, docker-composing) of such a service, but I regard it as a fundamental that is crucial to understand before using docker anywhere.
Technique 27: Inspecting containers
As I mentioned in my Reverse Engineering Docker Run Commands post, there are a few ways you can substitute a lack of docker knowledge when you need to uncover more information about a container, but there is nothing like having a thorough understanding of docker inspect and what is stored in the container's metadata.
Casting hardcore grep and awk usage aside, the inspect command has a --format flag that can be used to easily parse fields in the returned json blob.
[root@ansible-2 ~]# docker inspect --format='{{.Config.Cmd}}' 593d6fabde02
[sleep 1000]Technique 31: Rebuilding without the cache
Raise your hand if you've been victimized by a layer in a Dockerfile build being cached.
The problem I've most often encountered is when I'm from a git repository, specifying a brand rather than a tag, and the Dockerfile not being any wiser just using the cache since the name hasn't changed though the underlying repo has. To resolve this, you can build images using the --no-cache flag. If you want to be more selective and not bust your cache every time you build, you can instead add in a single cache-busting line above your code prone to cache confusion and just add a few gibberish characters to that line each time you need to build. Alternatively, though this doesn't solve all issues, you can only build images from tagged versions of a repository. This has helped me prevent this embarrassing problem from rearing its ugly head in prod.
In Technique 32, the authors recommend inserting a single comment to bust a cache starting on a certain line. I regard this as a cleaner approach to the one that I advocated for above.
Technique 34: Housekeeping containers
The first time I appeared on the SysEng radar in my workplace as a potential threat was when I first started contributing to some of our software that used Docker and had no idea how to manage the containers and the space they gobbled up.
Rather than deleting all containers, there are more pragmatic and safe ways to clean up a lot of disk space without blowing up something important.
It's important to first clean up all containers, as this may create more images that can be cleaned up. The opposite is not true.
docker rm $(docker ps -qa --no-trunc --filter "status=exited")This will get rid of all the containers not being used
docker rmi $(docker images --filter "dangling=true" -q --no-trunc)This will get rid of all dangling images--thus having a potentially large impact on disk usage but no impact on anything currently running
Nowadays, it is more common to use the modern prune feature:
docker prune imagesdocker prune containersTechnique 37: Using DockerUI to manage your Docker daemon
Once you realize that the Docker daemon exposes an API, the wheels of your mind start turning. What if there was a graphical interface that allowed you to manage all of your containers and images? Turns out, there is. It used to be called DockerUI, but that merged with the Portainer. I've been using Portainer for a little over a month now and it's miraculous. The setup is easy because it's Dockerized. The full functionality--which I still have yet to use to its full potential, is a game changer. This is something I wish I discovered much sooner in my career.
Technique 41: Avoid package drift by specifying versions in your build
Though the article in which I recommended pinning versions of Docker dependencies was mainly advocating for version pinning in general, it's an important lesson. The problem, plainly stated in this chapter, is that "you want to ensure that your pacakges are the versions you expect." This sounds innocuous, but I've been bitten by it in a big way. Once you introduce a bug with an automatically updated package version in a large project, it is a nightmare (or, in my words, a horror story) to try to track down and debug. Please consider pinning versions!
Technique 42: Reverse-engineer a Dockerfile from an image
Yes, this is possible. Similar to what I said about Technique 27, understanding the Docker inspect metadata is crucial. In this example, jq (like sed for JSON data) is used to parse the ContainerConfig fields and recreate the Dockerfile. Wow!
docker history reverseme | \
awk '{print $1}' | \
grep -v IMAGE | \
tac | \
sed "/\(.*\)/docker inspect \1 | \
jq -r \'.[0].ContainerConfig.Cmd[2] | tostring\'/" | \
sh | \
sed 's/^#(nop) //'
Still impressed with this one.
Technique 46: Traditional: using make with Docker
My response to this is to please think twice. I just recently was able to deprecate our bloated base Makefile at work and everyone sleeps a little better at night. It can get very confusing very quickly, and while I recognize it as a valid tool for certain use cases, I have trouble endorsing it. If you really, truly need addition tasks, then its totally reasonable to use. In many cases, though, I'd imagine people get a bit overzealous and focus their energy here instead of properly using docker-compose
Well, that's probably all you care to read about this topic for now. In part 2 I will cover techniques 49 - 101. Thanks for reading.
