Supercharge your Redpanda Experience with Testcontainers for Local Development and Tooling!

We often find ourselves wrestling with the complexities of setting up and maintaining local development environments. The process can be time-consuming, error-prone, and frustrating. However using Testcontainers can streamline our workflow and make the whole setup programmartic. While you might be familiar with using Testcontainers in testing, let’s see how it can supercharge your local development setup and provide additional convenience with tooling containers.

Testcontainers for Local Development

Contrary to popular belief, Testcontainers is not just for testing. Its ability to programmatically create and manage containers makes it a powerful tool for setting up development environments as well. When you clone a project and run it locally, if there’s no configured environment present Testcontainers can spin up necessary dependencies, such as a database or a Redpanda instance in containers (duh!). This process is automatic and can be either implemented on the framework side, like Quarkus Dev Services, or Microntaut Test Resources, or recently added Spring Boot support for this, or you can easily do it yourself just for your projects.

This local development environment setup approach offers several advantages:

1. Consistency: It ensures that all developers are working with the same environment, eliminating differences between local setups.
2. Efficiency: It saves developers from the hassle of manual setup and configuration, allowing them to focus on coding.
3. Isolation: It keeps your local development environment clean, as these dependencies are running in isolated containers rather than being installed directly on your machine.

Here’s how you can configure your Spring Boot application to use Redpanda:

@TestConfiguration(proxyBeanMethods = false)
public class ContainerConfig {
  @Bean
  @ServiceConnection
  public RedpandaContainer getRedpanda() {
    return new RedpandaContainer("redpandadata/redpanda:v23.1.14");
  }
}

We declare a Configuration to be used in our tests and then we can instrument both the tests and the local application run with this, and Spring Boot will see that the @Bean is a RedpandaContainer will start it, and thanks to @ServiceConnection will configure itself to use it.

Now let’s take it one step further and also programmatically enable the tooling.

Running Additional Tooling Containers

Beyond setting up application dependencies, Testcontainers can also be used to run additional tooling in containers. This can significantly enhance the development and debugging experience by providing easy access to powerful tools, without the need for manual setup and configuration.

For instance, if your application uses Redpanda, you can use Testcontainers to run a Redpanda console in a separate container alongside your Redpanda instance. This setup provides a powerful interface for monitoring your Redpanda instance in real-time, making it easier than ever to debug and monitor message streams.

We’ll modify our RedpandaContainer definition to put it on a Network and slightly modify its config so other services in the same Docker environment could communicate with it easily.

Network network = Network.newNetwork();

RedpandaContainer redpanda = new RedpandaContainer(DockerImageName.parse("docker.redpanda.com/redpandadata/redpanda:v23.1.10")) {

            @Override
            protected void containerIsStarting(InspectContainerResponse containerInfo) {
                String command = "#!/bin/bash\n";
                command = command + " /usr/bin/rpk redpanda start --mode dev-container --overprovisioned --smp=1";
                command = command + " --kafka-addr INTERNAL://redpanda:19092,PLAINTEXT://0.0.0.0:9092";
                command = command + " --advertise-kafka-addr INTERNAL://redpanda:19092,PLAINTEXT://" + this.getHost() + ":" + this.getMappedPort(9092);
                this.copyFileToContainer(Transferable.of(command, 511), "/testcontainers_start.sh");
            }
        }
        .withNetwork(network)
        .withNetworkAliases("redpanda");

Now Redpanda can be accessed from the same Network by the host redpanda so let’s configure the console:

String consoleConfig = """
                kafka:
                  brokers: ["redpanda:19092"]
                  schemaRegistry:
                    enabled: true
                    urls: ["http://redpanda:8081"]
                redpanda:
                  adminApi:
                    enabled: true
                    urls: ["http://redpanda:9644"]
                """;

        GenericContainer<?> console = new GenericContainer<>("docker.redpanda.com/redpandadata/console:v2.2.4")
                .withLabel("com.testcontainers.desktop.service", "redpanda-console")
                .withNetwork(network)
                .withExposedPorts(8080) // where we connect to it.
                .withCopyToContainer(Transferable.of(consoleConfig),
                        "/tmp/config.yml")
                .withEnv("CONFIG_FILEPATH", "/tmp/config.yml")
                .waitingFor(new HostPortWaitStrategy())
                .withStartupTimeout(Duration.of(10, ChronoUnit.SECONDS))
                .dependsOn(redpanda);

This code snippet is actually pretty interesting. It shows a common pattern of configuring services running via Testcontainers from Strings in your code.

Now when we start both containers, we can connect to Redpanda console on its exposed port. To simplify connecting to a randomly chosen port with external tools, in our case the browser, we can use Testcontainers Desktop and configure it to proxy a “fixed” port to any Redpanda Container.

Now while using the application, you can have a stable URL to a Redpanda Console container and easily debug what is happening in the Redpanda instance and see the messages your application or tests send into the broker.

The approach would also work with the other services, for example, if you’re working with a database, you can run a SQL admin web app in a container alongside your database. This provides a convenient way to inspect the state of the database, execute queries, and debug any issues.

The Power of Programmatic Configuration

The beauty of Testcontainers lies in its programmatic configuration. Your application can automatically check its dependencies at startup and use Testcontainers to spin up any necessary Docker containers. This ensures that your tools and dependencies are always configured correctly, reducing the potential for human error.

Moreover, these tooling containers can be automatically configured to connect to the appropriate services, since all the configuration is done programmatically. This means that developers don’t have to manually set up these tools each time they want to use them.

In a nutshell, Testcontainers can do much more than just facilitate testing. It can be a powerful ally in setting up local development environments and running additional tooling containers. By leveraging its capabilities, we can create a robust, consistent, and easy-to-manage environment, saving us from the drudgery of manual setup and configuration. So, if you haven’t explored these aspects of Testcontainers yet, I highly recommend giving it a try. It might just make your development life a lot easier.