Testcontainers is one of the most popular use cases for Docker on desktops. Spinning up lightweight ephemeral containers is a great way to ensure your code uses the same third party services in test as in production. Testcontainers manages the lifecycle of the containers and integrates with different test frameworks to provide containers and their configuration to your code. However, the library doesn’t implement a container engine or runtime itself. Instead, it expects Docker to be available on the machine.
Recently, we received many questions from Testcontainers users about the various options of running Docker workloads on their non-Linux machines. We identified three (and a half) ways of running Docker on Windows and macOS and decided to test them all for you with Testcontainers’ own test suite to tell you what to expect and how each of them behaved.
In this article we look at what options you have for the container runtime to run integration tests with Testcontainers, provide some guidance on how to setup them, and outline the current state of their compatibility.
We’ll look at:
The Setup
It’s not trivial to test compatibility of runtimes and other low-level systems. However, if you can successfully run a body of tests for the application level software (something that uses the runtime), you can claim compatibility with a reasonably high degree of confidence.
Indeed if the tests pass you can assume the runtime provides and behaves in a way the application expects it to. For this research we used the Testcontainers core tests which rely on a wide range of the Docker APIs.
The tests were executed on an Intel chip based macOS. We haven’t tested these options on Apple’s new M1-based laptops, as its ARM nature prevents running certain images no matter what is used to run the Docker daemon. All of the options that we reviewed exist for Windows as well, and the experience will be similar / the same.
Docker Desktop
Docker Desktop is a Docker distribution that includes a GUI and tight integration into the host OS and is available for Windows and macOS. It’s a product by Docker, Inc., which bundles a number of components that should satisfy most of your needs for running containers on desktop machines:
Besides the Docker Engine, it includes Docker Compose and even has built-in support for Kubernetes.
Docker Desktop is polished, multi-functional, and packs quite a few additional services. You can connect with your Docker account and publish images to Docker Hub, collaborate with your team, and so on. For obvious reasons, we consider it the reference implementation of “Docker” on Windows and macOS.
Getting started with Docker Desktop is rather straightforward: you download it, drag & drop it into your Applications folder, and you are good to go! You have a fully functional Docker running on your computer now.
One setting you would like to check is the amount of resources Docker can use on your machine, in Docker Desktop you find it in the Resources tab.
If you plan to run tests that depend on multiple containers you might want to up the amount of memory they are allowed to use. You may also let it access more CPUs to make these containers start faster.
Running tests using Testcontainers is then as straightforward as running a unit test:
All in all developer experience with Docker Desktop is natural and easy – install the app, run it, and it provides the necessary components out of the box. Your tests using the full range of features available in the Testcontainers library will work with Docker Desktop.
Docker Machine
Before Docker Desktop got all good and shiny, there was another project making it possible to run Docker on Windows and macOS – Docker Machine. And indeed, the official legacy Docker distribution on Windows, Docker Toolbox, was Docker Machine based.
Docker Machine was recently deprecated, but what it did and still does is create Docker virtual machine hosts and configures the Docker client to talk to them, while handling the nitty-gritty details of key management for the TLS connection for you. The default driver for Docker Machine is VirtualBox, which means it creates a virtual machine with VirtualBox and the necessary configuration for your docker CLI commands and API calls to correctly talk to Docker running in that VM.
Here’s how to install Docker Machine for running integration tests. You download the binary (note the last release was in September 2019), install VirtualBox and create the VM using the docker-machine command:
$ # download the binary $ curl -L https://github.com/docker/machine/releases/download/v0.16.2/docker-machine-`uname -s`-`uname -m` >/usr/local/bin/docker-machine $ chmod +x /usr/local/bin/docker-machine $ # install VirtualBox $ brew install --cask virtualbox $ # create the machine $ docker-machine create default --virtualbox-cpu-count "-1" --virtualbox-memory "8192"
It is very similar to Docker Desktop that also starts a virtual machine for running Docker (remember, Docker is a Linux technology, so we have to use the virtualization!). However, while Docker Desktop relies on modern HyperKit on macOS and a combination of WSL and Hyper-V on Windows, Docker Machine allows you to pick the virtualization provider, and VirtualBox is the most straightforward option.
Testcontainers was originally developed back when docker-machine was the only option for running Docker on macOS, so naturally Testcontainers is already Docker Machine compatible. Testcontainers can detect the presence of docker-machine and configure itself to correctly use it. This means that you can run the tests out of the box after the installation.
The only test out of the testcontainers-java suite that failed with Docker Machine was related to Healthchecks. We couldn’t identify the cause, but this is most probably due to Docker Machine having an old version of Docker engine.
Minikube
The next entry in our experiment is minikube. Minikube is a local Kubernetes, focusing on making it easy to learn and develop for Kubernetes.
Minikube needs a container or virtual machine manager, for our tests we used HyperKit. HyperKit is also a core component of Docker Desktop for Mac. If you don’t have Docker Desktop installed, you can get HyperKit from brew at the same time you’re installing minikube:
$ brew install minikube hyperkit $ minikube start --driver "hyperkit" --memory "8g" --cpus "max"
When you have Minikube running, you need to configure Testcontainers to talk to it. You can do this by asking minikube for the docker-env configuration and editing the .testcontainers.properties configuration file:
$ eval $(minikube -p minikube docker-env) $ echo "docker.host=$DOCKER_HOST" >> ~/.testcontainers.properties $ echo "docker.cert.path=$DOCKER_CERT_PATH" >> ~/.testcontainers.properties $ echo "docker.tls.verify=true" >> ~/.testcontainers.properties
Here we are using something that we added in Testcontainers 1.16.0 – support for Docker-related properties in the .testcontainers.properties file.
Note that if you want to use the Docker CLI utility, then you need to set up the environment variables with:
$ eval $(minikube docker-env)
But for configuring Testcontainers it’s much more convenient to use the .testcontainers.properties file in the HOME directory, which stores the central configuration.
After that you can run the tests normally (including your IDE). One thing to be aware of is that there is no filesystem mounting by default, so in our initial run with minikube quite a bit of the tests failed not finding the files:
If you want to make your local filesystem available in the containers, you need to mount it into minikube:
$ minikube mount $HOME:$HOME
After that everything works as expected and we managed to run the tests normally.
Still, you should consider, do you need to mount files into containers? In most cases, you do not, and we have been advocating the Copy API (withCopyFileToContainer and friends) for years. But, if you must use filesystem mounting and see weird errors – you know what to do.
An interesting fact about minikube is that it uses libmachine, which was a part of Docker Machine project, for managing the VMs. It is also an excellent example of Open Source at work – Minikube uses docker-machine’s libmachine to manage the VMs and Docker Desktop’s HyperKit to start them.
All in all, the compatibility of minikube for running your integration tests with Testcontainers is very good, in our case all Testcontainers-java tests pass. But your need to know a bit about various moving parts, for example to configure the local filesystem access correctly if it is needed.
Podman
And the last tool we are looking at in this article is Podman.
Podman is a daemonless, open source, Linux native tool designed to make it easy to find, run, build, share and deploy applications using Open Containers Initiative (OCI) Containers and Container Images.
Unlike the other options of running Docker on Windows and macOS, Podman isn’t something that will start a Docker daemon, but rather a full replacement for it, with a Docker API compatibility layer.
Podman is a Linux tool first and foremost as it describes itself in the above quote from their website. But it does work on macOS and Windows as we’ve seen reports that it’s possible to run Testcontainers tests using Podman as the underlying container runtime.
Here’s the configuration you might need after installing podman with brew:
$ brew install podman $ # Bootstrap guest CoreOS VM $ podman machine init -m 4096 $ podman machine start $ # use this command to identify ssh port guest CoreOS VM is using: $ podman system connection list $ # Create ssh tunnel, this will create a unix socket in /tmp/podman.sock $ ssh -i ~/.ssh/podman-machine-default -p <port> -L'/tmp/podman.sock:/run/user/1000/podman/podman.sock' -N core@localhost $ # Configure Testcontainers to use it $ echo "docker.host=unix:///tmp/podman.sock" >> $HOME/.testcontainers.properties
The results of our runs however didn’t inspire a lot of confidence. We observed a higher number of failures that is reasonable to write off as minor compatibility issues. We’ve reached out to the Podman team with the details of the failures to see if these issues can be fixed in a future release of Podman.
Ou current verdict would be that we’re not confident enough to make any claims of compatibility, so we’ll defer the question until future tests. But configuring Podman and making sure your tests can be run correctly clearly requires understanding of moving parts and Podman specific configuration.
Performance
An additional characteristic we wanted to look at is performance of all these available options. One would think that all these options should have the same performance characteristics, Right? Right?…
We thought so too, but decided to try running the same test and measure the performance of each option.
Our “benchmark” was rather simple – we measured how long it takes to start a KafkaContainer. Why Kafka? We always treated it as an interesting use case for Testcontainers – it requires some magic behind to make it work with random ports, uses multiple commands to configure everything, and takes a significant amount of time and CPU to start. Here is what we got:
| Method | Avg. startup time |
|---|---|
| Docker Desktop | 3.81s |
| docker-machine | 3.69s |
| Minikube | 3.92s |
Surprisingly, docker-machine with VirtualBox was the fastest! One would think that a modern HyperKit-based approach would win, but apparently good old VirtualBox is still doing great!
Conclusion
In this article we looked at 4 different ways to run integration tests that involve Docker on Windows and Mac. The tests were taken from the Testcontainers-java project and cover quite a bit of Docker API usage which makes us believe it’s a decent proxy for how compatible these solutions are.
We looked at Docker Desktop, Docker Machine, Minikube, and Podman. Here’s a table summarising our experience with them:
Docker Desktop provides great out of the box experience and is an obvious first choice for your Docker workloads. However should you need alternative solutions there are some options that are worth exploring for your use cases. Minikube is the most promising one, and docker-machine remaining a viable option if the deprecation of the project is not offputting.
