Setting Up a CI/CD Pipeline with Docker and Kubernetes

Continuous integration (CI) automatically checks every code change: compilation, tests, static analysis, building an artifact. Continuous deployment (CD) delivers that artifact through successive environments up to production, reproducibly and without error-prone manual intervention.

The goal is not speed for its own sake but confidence: every deployment must be identical in staging and production, traceable back to the originating commit and reversible within minutes. Docker provides the immutable artifact; Kubernetes provides the deployment and rollback mechanism.

Pipeline orchestration tools (GitHub Actions, GitLab CI, Jenkins, Bitbucket Pipelines) are largely interchangeable. What matters is the structure of the stages and the quality rules enforced at each one, not the brand of the tool that runs them.

The Dockerfile must produce a minimal, deterministic image. Use a multi-stage build: a first stage compiles with all the development tools, a second copies only the files needed at runtime into a lightweight base image, which reduces both size and attack surface.

Pin the versions of base images and dependencies, run the process as an unprivileged user and add a .dockerignore file to exclude unnecessary or sensitive files. Layer caching should be exploited by ordering instructions from least to most frequently changing.

Every image must be identified by an immutable tag, typically the commit SHA, in addition to any human-readable tags. The "latest" tag must never be deployed to production: it makes it impossible to know what is actually running and to roll back cleanly.

A typical pipeline chains: dependency installation with caching, static analysis and formatting, unit tests, image build, image security scan with a tool such as Trivy or Grype, then publication to the registry only if every previous stage succeeded.

Integration tests that require a database or a queue run in ephemeral containers started by the pipeline. This guarantees an identical environment on every run and avoids dependencies on shared services whose state varies.

The pipeline must be fast: beyond about ten minutes, developers stop waiting for the result and errors are caught later. Parallelise independent stages and rebuild only what changed in a repository that holds several services.

Kubernetes manifests (Deployment, Service, Ingress, ConfigMap, Secret, HorizontalPodAutoscaler) must be versioned with the code or in a dedicated repository. Helm or Kustomize let you manage variations between environments without duplicating files or maintaining them by hand.

Every Deployment must define liveness and readiness probes, resource requests and limits, and a rolling update strategy with a maximum number of unavailable instances. Without a readiness probe, Kubernetes sends traffic to pods that are not ready yet.

Secrets must never be stored in plain text in the repository. Use an external manager (AWS Secrets Manager, HashiCorp Vault) synchronised through External Secrets Operator, or encrypted secrets with Sealed Secrets or SOPS, so the repository stays readable without being dangerous.

The GitOps approach makes the Git repository the source of truth for the desired state of the cluster. An operator such as Argo CD or Flux watches the repository and applies changes automatically. The CI pipeline no longer deploys directly: it updates the image tag in the configuration repository.

The benefits are considerable: a complete deployment history in Git, pull request review before production, drift detection between declared and actual state, and rollback by simply reverting the offending commit, with no direct access to the cluster.

This requires discipline: any manual change on the cluster will be overwritten by the operator. That is exactly the point, but the team must have understood and accepted it before adoption, or frustration follows.

The default rolling update replaces pods progressively and suits most stateless services. For risky changes, a blue/green deployment keeps two complete versions and switches traffic at once; a canary deployment first sends a fraction of the traffic to the new version.

Canary and progressive deployments are made easier by tools such as Argo Rollouts or Flagger, which automate promotion or rollback based on metrics (error rate, latency). This assumes observability is in place: centralised, queryable metrics, logs and traces.

Database migrations must remain compatible with the previous version of the application during the deployment, since both versions coexist for a few minutes. Without this precaution, rollback becomes impossible at the exact moment you need it.

The most common mistakes we encounter: images built differently per environment, no tests before the image build, secrets in pipeline variables without rotation, pipeline administrator access to the production cluster, and no tested rollback procedure.

A CI/CD pipeline is a product in its own right. It deserves regular review, up-to-date documentation and a person responsible for keeping it working, exactly like the application it delivers.