One of the unique values of Testcontainers is that it allows running the same set of integration tests locally and on CI, using the same mechanism for setting up the test environment. This gives development teams early feedback during development right after they compile their code in IDE while also guarding CI/CD pipelines with automated tests on every push.
How do Testcontainers achieve this parity between execution environments? By using Docker, of course! What used to be a major pain and manual process that could lead to inconsistency between environments, now became as simple as “If it has access to Docker, you can run your tests there!”.
CircleCI isn’t an exception, as it is an ultra-modern CI/CD platform that comes with everything we need to run our CI pipelines.
Let’s look into various ways of running Testcontainers on CircleCI.
In this article, we will learn about various options of running Testcontainers-based tests on CircleCI, learn how Testcontainers Cloud makes it even easier and more elastic, and explore options of running the tests in parallel.
Making the tests lean on the cloud for running the containers allows you to use dockerized workers on CircleCI instead of machine executors, and given the scalability of the cloud you can also parallelize your workloads to considerably speed up your pipelines!
We are going to use Testcontainers-showcase project to demonstrate how we can run Testcontainers-based tests on CircleCI platform.
Minimal Setup
By default, CircleCI will run your builds inside a container, and it spawns them fast. Like… really FAST. It takes just a few lines of YAML to have your project being built with CircleCI.
# .circleci/config.yml
version: 2.1
orbs:
maven: circleci/maven@1.3.0
workflows:
maven_test:
jobs:
- maven/test:
executor:
name: maven/default
tag: "17.0"
We will be using Testcontainers for Java in this article, but it will be the same for Golang, .NET, NodeJS and other Testcontainers implementations.
That works great for unit tests, but won’t work for our use case since we need a Docker daemon to start containers with Testcontainers. Here is the Could not find a valid Docker environment error we will get, unsurprisingly:
23:46:56.123 [main] WARN org.testcontainers.utility.TestcontainersConfiguration - Attempted to read Testcontainers configuration file at file:/home/circleci/.testcontainers.properties but the file was not found. Exception message: FileNotFoundException: /home/circleci/.testcontainers.properties (No such file or directory) 23:46:56.126 [main] INFO org.testcontainers.utility.ImageNameSubstitutor - Image name substitution will be performed by: DefaultImageNameSubstitutor (composite of 'ConfigurationFileImageNameSubstitutor' and 'PrefixingImageNameSubstitutor') 23:46:56.172 [main] INFO org.testcontainers.dockerclient.DockerMachineClientProviderStrategy - docker-machine executable was not found on PATH ([/opt/sbt/bin, /opt/gradle/bin, /opt/apache-maven/bin, /home/circleci/bin, /home/circleci/.local/bin, /usr/local/sbin, /usr/local/bin, /usr/sbin, /usr/bin, /sbin, /bin]) 23:46:56.173 [main] ERROR org.testcontainers.dockerclient.DockerClientProviderStrategy - Could not find a valid Docker environment. Please check configuration. Attempted configurations were: UnixSocketClientProviderStrategy: failed with exception InvalidConfigurationException (Could not find unix domain socket). Root cause NoSuchFileException (/var/run/docker.sock)As no valid configuration was found, execution cannot continue.
If we head to Running Docker Commands, we will find that CircleCI supports having Docker with the setup_remote_docker step:
version: '2.1'
orbs:
maven: circleci/maven@1.3.0
jobs:
build:
executor:
name: maven/default
tag: "17.0"
steps:
- setup_remote_docker:
version: 20.10.14
docker_layer_caching: true
- checkout
- run: "./mvnw verify"
Let’s give it a try:
04:53:54.172 [main] INFO 🐳 [testcontainers/ryuk:0.3.4] - Creating container for image: testcontainers/ryuk:0.3.4 04:53:54.320 [main] INFO 🐳 [testcontainers/ryuk:0.3.4] - Container testcontainers/ryuk:0.3.4 is starting: e3457531ced5438b1f3c413ec26788d2b43ebd5f29ce9745d6d3518ab7ba44f7 04:53:54.918 [main] INFO 🐳 [testcontainers/ryuk:0.3.4] - Container testcontainers/ryuk:0.3.4 started in PT2.124507253S 04:53:59.924 [testcontainers-ryuk] WARN org.testcontainers.utility.RyukResourceReaper - Can not connect to Ryuk at 34.73.149.65:49153 java.net.SocketTimeoutException: Connect timed out at java.base/sun.nio.ch.NioSocketImpl.timedFinishConnect(NioSocketImpl.java:546) at java.base/sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:597) at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:327) at java.base/java.net.Socket.connect(Socket.java:633) at org.testcontainers.utility.RyukResourceReaper.lambda$null$0(RyukResourceReaper.java:92) at org.rnorth.ducttape.ratelimits.RateLimiter.doWhenReady(RateLimiter.java:27) at org.testcontainers.utility.RyukResourceReaper.lambda$maybeStart$1(RyukResourceReaper.java:88) at java.base/java.lang.Thread.run(Thread.java:833)
It looks like this feature would work for e.g. Docker image builds, but not for starting arbitrary containers. Bummer!
So, it seems that CircleCI’s containerized runners aren’t a good fit for running projects that depend on Docker, e.g. Testcontainers.
But the good news is that there is an alternative runner type machine that provides us with a Virtual Machine instead of a container to run our builds in and it has Docker inside. Let’s give it a try:
version: '2.1'
orbs:
maven: circleci/maven@1.3.0
executors:
machine_executor_amd64:
machine:
image: ubuntu-2204:2022.04.2
environment:
architecture: "amd64"
platform: "linux/amd64"
workflows:
maven_test:
jobs:
- maven/test:
executor: machine_executor_amd64
Note the addition of the Ubuntu-based Virtual Machine executor named machine_executor_amd64. Let’s check the build logs:
[INFO] [INFO] --- maven-jar-plugin:3.3.0:jar (default-jar) @ testcontainers-showcase --- [INFO] Building jar: /home/circleci/project/target/testcontainers-showcase-0.0.1-SNAPSHOT.jar [INFO] [INFO] --- spring-boot-maven-plugin:3.0.0:repackage (repackage) @ testcontainers-showcase --- [INFO] Replacing main artifact with repackaged archive [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 07:12 min [INFO] Finished at: 2022-12-05T10:57:50Z [INFO] ------------------------------------------------------------------------
Now we are talking! Our build is green and so are the test results. The downside is that Virtual Machines aren’t as elastic and lightweight as containers on CircleCI, so you might end up waiting for them to be allocated.
The question is: can we get the elasticity of containerized runners while also running Testcontainers-based tests? Yes we can!
Meet Testcontainers Cloud – Developer’s and CI’s Best Companion!
Testcontainers Cloud is a managed SaaS service that we’ve built to make it easy to run Testcontainers-based tests anywhere, be it on your local machine with our Desktop client or on CIs with the agent.
Let’s have a look at what needs to be done to start using Testcontainers Cloud on CircleCI.
First thing first, we need to Signup for Testcontainers Cloud 🙂 Once we are in, we will be able to obtain a service token that we will be using to authenticate.
In the Testcontainers Cloud Dashboard, go to Settings, select Service Accounts, click on Create Service Account, enter Account name, click on Create and get the token.
Let’s create a secret in CircleCI to store the token:
- In CircleCI go to your project and click on Project Settings
- Go to Environment Variables and add Environment Variable with name TC_CLOUD_TOKEN and enter TestcontainersCloud Token as value.
Now that we have the token as a secret, we can go ahead and add Testcontainers Cloud to our original containerized CircleCI config with the Orb that we provide for your convenience:
version: '2.1'
orbs:
maven: circleci/maven@1.3.0
tcc: atomicjar/testcontainers-cloud-orb@0.1.0
workflows:
maven_test:
jobs:
- maven/test:
executor:
name: maven/default
tag: "17.0"
pre-steps:
- tcc/setup
The testcontainers-cloud-orb will take care of starting the Testcontainers Cloud agent as a background process which will enable it to start containers in Testcontainers Cloud instead of using a local Docker daemon.
We should be able to see in logs that tests are now using Testcontainers Cloud for running Testcontainers-based tests.
00:22:23.351 [main] INFO org.testcontainers.DockerClientFactory - Connected to docker: Server Version: 70+testcontainerscloud API Version: 1.41 Operating System: Ubuntu 20.04 LTS
And the tests should run just fine on CircleCI’s default docker executor using Testcontainers Cloud!
[INFO] [INFO] --- maven-jar-plugin:3.3.0:jar (default-jar) @ testcontainers-showcase --- [INFO] Building jar: /home/circleci/project/target/testcontainers-showcase-0.0.1-SNAPSHOT.jar [INFO] [INFO] --- spring-boot-maven-plugin:3.0.0:repackage (repackage) @ testcontainers-showcase --- [INFO] Replacing main artifact with repackaged archive [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 07:25 min [INFO] Finished at: 2022-12-05T11:49:13Z [INFO] ------------------------------------------------------------------------
As you can see, with a minor change to our CI config, we can now run Testcontainers-based tests without sacrificing the elasticity of CircleCI’s containerized runners!
Bonus: Parallel Testing
Have you ever asked yourself: “my tests spend a lot of time waiting for things. Wouldn’t it be better to run them in parallel?”. We had 🙂
Parallel testing is a common practice in unit testing, because they are lightweight and isolated.
But, with containers, this becomes tricky, if not impossible. Or… it used to be so 🙂
Let’s first have a look at how we could use CircleCI’s parallel testing support.
Parallel Testing with CircleCI only
Luckily enough, CircleCI’s Maven Orb supports parallel testing out of the box, and we only need to change maven/test to maven/parallel_test plus optionally set the parallelism:
version: '2.1'
orbs:
maven: circleci/maven@1.3.0
executors:
machine_executor_amd64:
machine:
image: ubuntu-2204:2022.04.2
environment:
architecture: "amd64"
platform: "linux/amd64"
workflows:
maven_test:
jobs:
- maven/parallel_test:
executor: machine_executor_amd64
parallelism: 4
parallel_it_pattern: "**/*Test*.java"
This is how it will look like using parallel tests with machine executors.
And, with parallelisation tests should run faster.
[INFO] [INFO] --- maven-jar-plugin:3.3.0:jar (default-jar) @ testcontainers-showcase --- [INFO] Building jar: /home/circleci/project/target/testcontainers-showcase-0.0.1-SNAPSHOT.jar [INFO] [INFO] --- spring-boot-maven-plugin:3.0.0:repackage (repackage) @ testcontainers-showcase --- [INFO] Replacing main artifact with repackaged archive [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 03.18 min [INFO] Finished at: 2022-12-05T11:03:11Z [INFO] ------------------------------------------------------------------------
Now let’s compare it to the Testcontainers Cloud-based approach.
Parallel Testing with Testcontainers Cloud only
Why did we have to spawn multiple runners with CircleCI to test in parallel in the first place?
Because we want each parallel “rail” to have its own set of containers to avoid conflicts between the tests, and we have a direct one-to-one mapping between the CI runner and Docker.
But, with Testcontainers Cloud, we can go one-to-many!
This allows us to use the build tool’s parallel testing support that we can use both locally and on CI, so we no longer need to commit our changes to run all tests fast. Isn’t that cool?
In Maven, for example, this is done with the forkCount property like ./mvnw verify -DforkCount=4.
We would not recommend running parallel tests with a local Docker as it will consume too much of your CPU and RAM, which can lead to flaky tests. But we can easily do that with Testcontainers Cloud, especially with the Turbo Mode enabled.
Now let’s proceed enabling Maven’t parallel testing support on CI:
version: '2.1'
orbs:
maven: circleci/maven@1.3.0
tcc: atomicjar/testcontainers-cloud-orb@0.1.0
workflows:
maven_test:
jobs:
- maven/test:
executor:
name: maven/default
tag: "17.0"
command: 'verify -DforkCount=4'
pre-steps:
- tcc/setup
As you can see, the only change we had to do is to add -PforkCount=4 to the Maven command, similar to how we did it locally.
[INFO] [INFO] --- maven-jar-plugin:3.3.0:jar (default-jar) @ testcontainers-showcase --- [INFO] Building jar: /home/circleci/project/target/testcontainers-showcase-0.0.1-SNAPSHOT.jar [INFO] [INFO] --- spring-boot-maven-plugin:3.0.0:repackage (repackage) @ testcontainers-showcase --- [INFO] Replacing main artifact with repackaged archive [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 03:42 min [INFO] Finished at: 2022-12-05T14:26:41Z [INFO] ------------------------------------------------------------------------
Parallel Testing with Testcontainers Cloud and CircleCI
One great advantage of CircleCI’s parallel testing support is that it will analyze the duration of tests and be smart about assigning tests to parallel workers, to equally spread the load leading to faster test results.
We just learned how to do the parallel testing with Testcontainers Cloud without any CI support, but we can mix and match and get the best of both worlds too.
Let’s take the config we had, remove the machine executor to make it start faster and enable Testcontainers Cloud:
version: '2.1'
orbs:
maven: circleci/maven@1.3.0
tcc: atomicjar/testcontainers-cloud-orb@0.1.0
workflows:
maven_test:
jobs:
- maven/parallel_test:
executor:
name: maven/default
tag: "17.0"
parallelism: 4
parallel_it_pattern: "**/*Test*.java"
pre-steps:
- tcc/setup
Now we can benefit from CircleCI’s parallel testing support without having to use the machine executors, so that we can run more builds in parallel and ship to production faster and more often.
[INFO] [INFO] --- maven-jar-plugin:3.3.0:jar (default-jar) @ testcontainers-showcase --- [INFO] Building jar: /home/circleci/project/target/testcontainers-showcase-0.0.1-SNAPSHOT.jar [INFO] [INFO] --- spring-boot-maven-plugin:3.0.0:repackage (repackage) @ testcontainers-showcase --- [INFO] Replacing main artifact with repackaged archive [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 03.13 min [INFO] Finished at: 2022-12-05T13:15:17Z [INFO] ------------------------------------------------------------------------
Conclusion
We have explored how to run Testcontainers based tests on the CircleCI platform using various ways. We have talked about some of the challenges you may typically face and how to solve them.
Testcontainers Cloud is a SaaS platform that can help you to run your Testcontainers-based tests in a fast way without having to go through Docker configuration issues. Setting up the Testcontainers Cloud integration on CircleCI is simple thanks to the testcontainers-cloud-orb and you can run your Testcontainers-based tests without sacrificing the convenience of containerized workers.
We also explored multiple ways to run tests in parallel to speed up the execution by leveraging features of the build tool, multiple CircleCI executors and Testcontainers Cloud as well. Our experiments showed that you can go from a 7:12 min build on a Machine executor, to running in a similar time with Testcontainers Cloud in a containerized worker, to still using the containerized workers only but running tests in parallel to cut the build time more than 2x and reaching 3:13 min.
Sign up for Testcontainers Cloud at https://testcontainers.cloud/ and try it yourself!
