Deployments and Rollouts: Shipping and Rolling Back Without Downtime

A story to begin with

The three layers

flowchart TB
  D[Deployment<br/>desired: 3 replicas of version 2<br/>strategy: rolling update] --> RS2[ReplicaSet v2<br/>owns the new pods]
  D -.-> RS1[ReplicaSet v1<br/>old, scaled to 0 but kept for rollback]
  RS2 --> P1[Pod]
  RS2 --> P2[Pod]
  RS2 --> P3[Pod]
  style D fill:#6366f1,color:#fff
  style RS1 fill:#64748b,color:#fff

• A pod runs your containers.
• A ReplicaSet keeps a fixed number of identical pods running. If one dies, it makes another.
• A Deployment manages ReplicaSets over time. When you change the pod template (new image, new env var), it creates a new ReplicaSet and gradually moves pods from the old one to the new one. The old ReplicaSet is kept (scaled to zero) so you can go back.

You almost always create Deployments, not ReplicaSets or bare pods.

A complete Deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
  labels: {app: web}
spec:
  replicas: 4
  revisionHistoryLimit: 5          # how many old ReplicaSets to keep for rollback
  progressDeadlineSeconds: 300     # mark the rollout failed if it stalls this long
  selector:
    matchLabels: {app: web}
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1                  # may run 1 extra pod during the update
      maxUnavailable: 0            # never drop below the desired count
  template:
    metadata:
      labels: {app: web}
    spec:
      terminationGracePeriodSeconds: 45
      containers:
      - name: web
        image: registry.example.com/web:2.4.0
        ports: [{containerPort: 8080}]
        readinessProbe:
          httpGet: {path: /ready, port: 8080}
          periodSeconds: 5
        resources:
          requests: {cpu: 200m, memory: 256Mi}
          limits: {memory: 256Mi}

How a rolling update works

sequenceDiagram
  participant D as Deployment controller
  participant N as New ReplicaSet v2
  participant O as Old ReplicaSet v1
  Note over O: 4 pods running v1
  D->>N: scale to 1 (maxSurge 1)
  Note over N: new pod starts, must become Ready
  D->>O: scale to 3 (after v2 pod is Ready)
  D->>N: scale to 2
  D->>O: scale to 2
  D->>N: scale to 3
  D->>O: scale to 1
  D->>N: scale to 4
  D->>O: scale to 0
  Note over N: 4 pods running v2, old set kept for rollback

Two numbers control the dance:

Typical choices: maxSurge: 25%, maxUnavailable: 0 for user-facing services that must not lose capacity; maxSurge: 0, maxUnavailable: 1 when you have no spare resources.

The rollout only moves forward when new pods become Ready, so a correct readiness probe is what makes rolling updates safe.

Operating a rollout

kubectl apply -f web.yaml                      # or: kubectl set image deploy/web web=...:2.4.1
kubectl rollout status deploy/web              # blocks until done or failed
kubectl rollout history deploy/web             # list revisions
kubectl rollout history deploy/web --revision=3
kubectl rollout pause deploy/web               # stop mid-way to inspect a partial rollout
kubectl rollout resume deploy/web
kubectl rollout undo deploy/web                # back to the previous revision
kubectl rollout undo deploy/web --to-revision=2
kubectl annotate deploy/web kubernetes.io/change-cause="upgrade to 2.4.1: fix cart bug"

Add a change-cause annotation so history is readable. A stuck rollout reports ProgressDeadlineExceeded instead of hanging forever.

Why a "good" rollout can still cause errors

lifecycle:
  preStop:
    exec: {command: ["sh", "-c", "sleep 8"]}   # give endpoints time to update before shutdown

Beyond rolling updates

Plain Kubernetes does simple blue-green with two Deployments and a Service selector switch. For automated canaries with metric analysis, use a progressive-delivery tool such as Argo Rollouts or Flagger together with an ingress or service mesh that can split traffic.

# Argo Rollouts canary sketch
strategy:
  canary:
    steps:
    - setWeight: 10
    - pause: {duration: 5m}
    - analysis: {templates: [{templateName: success-rate}]}
    - setWeight: 50
    - pause: {duration: 10m}

Scaling and updating together

kubectl scale deploy/web --replicas=8
kubectl autoscale deploy/web --min=4 --max=20 --cpu-percent=70

Scaling changes the replica count of the current ReplicaSet without creating a new revision. Only changes to spec.template create a new revision.

A release practice for your organization

flowchart LR
  A[Merge to main] --> B[CI: build, test,<br/>scan, tag image with Git SHA]
  B --> C[Deploy to staging<br/>smoke tests]
  C --> D[Production rolling update<br/>or canary]
  D --> E{Health OK?<br/>readiness, error rate, latency}
  E -- yes --> F[Done: record change-cause]
  E -- no --> G[Automatic rollback<br/>+ alert owner]
  style E fill:#6366f1,color:#fff
  style G fill:#f87171,color:#fff

1. One path to production: CI builds an image tagged with the Git SHA, and a pipeline (or GitOps) applies the manifest. Nobody edits live objects by hand.
2. Standard Deployment template: probes, resources, maxUnavailable: 0, preStop, graceful shutdown, revisionHistoryLimit and the change-cause annotation.
3. Gates: smoke tests in staging, then a small production canary for risky services.
4. Automatic rollback: roll back when error rate or latency crosses an agreed threshold during the rollout window.
5. Release visibility: deploy markers on dashboards and a channel notification for every production rollout.
6. Measure: deployment frequency, change failure rate, time to roll back and time to restore. Report them monthly.
7. Practise rollback: once a quarter roll a service back on purpose so nobody learns the command during an incident.

Pitfalls

• Using :latest tags, which makes rollout undo meaningless.
• Skipping readiness probes and trusting "Running".
• Setting revisionHistoryLimit: 0 and losing the ability to roll back.
• Changing the selector of an existing Deployment (immutable): create a new one instead.

Interview questions

Q1. How does a Deployment perform a rolling update? It creates a new ReplicaSet for the updated pod template and gradually scales it up while scaling the old one down, respecting maxSurge and maxUnavailable and waiting for new pods to be Ready.

Q2. What do maxSurge and maxUnavailable do? maxSurge is how many extra pods can exist above desired during the rollout; maxUnavailable is how many can be missing. Together they trade speed against capacity.

Q3. A rollout causes a burst of errors. What do you check? Readiness probe quality, graceful shutdown (SIGTERM handling and preStop), surge and unavailability settings, resource availability, and whether the new version warms up slowly.

Q4. How do you roll back, and what makes it reliable? kubectl rollout undo returns to the previous ReplicaSet. It is reliable when image tags are immutable, history is retained, and configuration changes are part of the same revision.

Q5. Rolling update vs blue-green vs canary? Rolling replaces gradually with both versions briefly coexisting; blue-green switches all traffic between two full environments; canary exposes a small share first and expands based on metrics.

Q6. Why might you set maxUnavailable: 0? To ensure you never fall below the desired capacity during updates, which protects availability for user-facing services at the cost of needing spare capacity for surge pods.

Next: Config and Secrets shows how to change behaviour without rebuilding images.