Using Docker with Jenkins Pipelines

In modern DevOps, consistency is everything. Historically, Jenkins administrators had to manually install compilers, runtimes, and build tools directly onto Jenkins agents. This approach quickly led to "dependency hell," where one project required Java 8 while another required Java 17, or different projects demanded conflicting versions of Node.js.

By integrating Docker with Jenkins Pipelines, you can isolate your build environments completely. Instead of installing tools on the host system, Jenkins spins up a lightweight Docker container, executes the build steps inside it, and tears it down immediately after. This guarantees that every build runs in a clean, predictable, and reproducible environment.

Understanding the Jenkins-Docker Architecture

When you run a Jenkins pipeline with a Docker agent, Jenkins coordinates with the Docker daemon running on the agent machine. Here is how the execution flow works:

+-------------------------------------------------------------+
|                      Jenkins Controller                     |
+------------------------------+------------------------------+
                               | Triggers Pipeline
                               ▼
+-------------------------------------------------------------+
|                        Jenkins Agent                        |
|                                                             |
|  1. Pulls requested image (e.g., maven:3.9-eclipse-temurin) |
|  2. Starts container, mounting the local workspace          |
|  3. Executes build commands inside the container            |
|  4. Stops and cleans up the container                       |
+-------------------------------------------------------------+
  

Crucially, Jenkins automatically mounts the current workspace directory from the host agent into the Docker container. This means any files generated inside the container—such as compiled classes, JAR files, or test reports—are written directly to the host's disk and persist even after the container is destroyed.

Prerequisites for Using Docker in Jenkins

Before writing a Docker-enabled Jenkinsfile, ensure your environment meets these requirements:

• Docker Installed: The Docker engine must be installed and running on the Jenkins agent machine.
• Jenkins User Permissions: The jenkins system user on the agent must belong to the docker group. This allows Jenkins to run Docker commands without requiring root privileges.
• Docker Pipeline Plugin: This plugin must be installed on your Jenkins controller to enable the docker agent syntax in Declarative Pipelines.

Declarative Pipeline Syntax with Docker

Jenkins makes it incredibly simple to define Docker containers as execution environments. You can define a Docker image globally for the entire pipeline, or specify different containers for individual stages.

1. Global Docker Agent

If your entire build, test, and deployment process relies on a single toolset, you can define the Docker agent at the top level of your pipeline.

pipeline {
    agent {
        docker {
            image 'maven:3.9.6-eclipse-temurin-17'
            args '-v /root/.m2:/root/.m2'
        }
    }
    stages {
        stage('Compile') {
            steps {
                sh 'mvn clean compile'
            }
        }
        stage('Test') {
            steps {
                sh 'mvn test'
            }
        }
    }
}
  

In this example, Jenkins pulls the official Maven image with JDK 17. The args parameter mounts a host directory to cache Maven dependencies, preventing Jenkins from downloading the entire internet on every single build run.

2. Stage-Specific Docker Agents

For complex multi-language projects, a single container might not suffice. You can run different stages in different containers while sharing the same workspace.

pipeline {
    agent any
    stages {
        stage('Build Backend') {
            agent {
                docker { image 'maven:3.9.6-eclipse-temurin-17' }
            }
            steps {
                sh 'mvn clean package -DskipTests'
            }
        }
        stage('Test Frontend') {
            agent {
                docker { image 'node:20-alpine' }
            }
            steps {
                sh 'npm install'
                sh 'npm test'
            }
        }
    }
}
  

Here, the backend compiles inside a Java container, and the frontend tests run inside a Node.js container. The files generated during the backend build remain accessible to subsequent stages because they share the same workspace directory on the host agent.

Real-World Use Case: Building and Pushing a Custom Docker Image

A highly common CI/CD pattern involves building an application, packaging it into a new Docker image, and pushing that image to a secure registry like Docker Hub or Amazon ECR. Below is a complete Jenkinsfile demonstrating this workflow.

pipeline {
    agent any
    environment {
        REGISTRY_CREDENTIALS = 'dockerhub-credentials-id'
        IMAGE_NAME = 'my-docker-username/my-java-app'
        IMAGE_TAG = "${env.BUILD_NUMBER}"
    }
    stages {
        stage('Build Artifact') {
            agent {
                docker { image 'maven:3.9.6-eclipse-temurin-17' }
            }
            steps {
                sh 'mvn clean package -DskipTests'
            }
        }
        stage('Build Docker Image') {
            steps {
                sh "docker build -t ${IMAGE_NAME}:${IMAGE_TAG} ."
                sh "docker tag ${IMAGE_NAME}:${IMAGE_TAG} ${IMAGE_NAME}:latest"
            }
        }
        stage('Push to Registry') {
            steps {
                withCredentials([usernamePassword(credentialsId: env.REGISTRY_CREDENTIALS, usernameVariable: 'USER', passwordVariable: 'PASS')]) {
                    sh "echo ${PASS} | docker login -u ${USER} --password-stdin"
                    sh "docker push ${IMAGE_NAME}:${IMAGE_TAG}"
                    sh "docker push ${IMAGE_NAME}:latest"
                }
            }
        }
    }
    post {
        always {
            sh "docker rmi ${IMAGE_NAME}:${IMAGE_TAG} || true"
            sh "docker rmi ${IMAGE_NAME}:latest || true"
        }
    }
}
  

Common Mistakes and How to Avoid Them

• Permission Denied on /var/run/docker.sock: This occurs when the jenkins user on the agent does not have permission to communicate with the Docker daemon. Fix: Run sudo usermod -aG docker jenkins on the agent machine, and then restart the Jenkins agent service.
• Workspace File Ownership Issues: Because Docker containers often run as the root user by default, files generated inside the container (like target directories) may be owned by root. This can prevent Jenkins from cleaning up the workspace in subsequent builds. Fix: Pass the host user ID to the container using the args parameter: args '-u 1000:1000' (replace with your Jenkins user ID).
• Running Out of Disk Space: Every time Jenkins pulls a new image or builds one, disk space is consumed. Over time, unused images, dangling builds, and stopped containers will fill up your agent's hard drive. Fix: Implement a cron job or a post-build step that runs docker system prune -af --volumes periodically to clean up unused resources.

Interview Prep: Jenkins and Docker Q&A

How does Jenkins run commands inside a Docker container during a pipeline?

When you specify a Docker agent, Jenkins starts the container with a command that keeps it running in the background. It then uses the docker exec command or internally managed channels to execute your pipeline's shell steps (sh) directly inside that running container.

What is the difference between running Docker commands inside a pipeline stage and using a Docker agent?

Using a Docker agent (e.g., agent { docker { image 'node' } }) automatically configures the environment so that all pipeline steps run inside that container. Conversely, running shell steps like sh 'docker run ...' requires you to manually manage container lifecycles, volume mounts, and environment variables.

How do you run sidecar containers (e.g., a database) for integration testing in Jenkins?

You can use the docker.image().withRun() utility in Scripted Pipeline syntax, or use Docker Compose within a Declarative Pipeline step to spin up dependent services like MySQL or Redis alongside your application container, tearing them down when testing completes.

Summary

Integrating Docker with Jenkins Pipelines elevates your CI/CD architecture to a professional standard. By shifting build environments from static host installations to dynamic, containerized environments, you eliminate configuration drift, simplify agent maintenance, and guarantee build reproducibility. Whether you are running simple compilation steps inside a Maven container or building and shipping production-grade Docker images, combining these two tools is a foundational skill for any modern DevOps engineer.