Pods in Depth: Lifecycle, Probes, Multi-Container Patterns and Resources

A story to begin with

What a pod is

A pod is one or more containers that run together on one node and share the same network address and (optionally) storage. Containers in a pod talk to each other over localhost.

flowchart TB
  subgraph Pod[Pod: one IP address, shared network]
    IC[Init container<br/>runs first, then exits]
    M[Main container<br/>your app]
    S[Sidecar container<br/>logging, proxy, sync]
    V[(Shared volume)]
  end
  IC --> V
  M --- V
  S --- V
  M -. localhost .- S
  style M fill:#6366f1,color:#fff

Why pods exist: some processes must be scheduled together and share fate, such as an app and its log shipper. Most pods hold a single main container; add more only when they genuinely need to share a lifecycle.

Important truths about pods:

• Pods are mortal and replaceable. They are not repaired, they are replaced, with a new IP each time.
• You rarely create pods directly. You create a Deployment (or StatefulSet, Job) that manages pods for you.
• A pod is placed on one node for its entire life.

The pod lifecycle

stateDiagram-v2
  [*] --> Pending: created, waiting to be scheduled or pull images
  Pending --> Running: containers started
  Running --> Succeeded: all containers exited with 0 (Jobs)
  Running --> Failed: a container exited non-zero and will not restart
  Running --> Terminating: deleted
  Terminating --> [*]: SIGTERM, grace period, SIGKILL
  Pending --> Failed: cannot be scheduled for too long

Container states within a pod are Waiting (with reasons like ImagePullBackOff, CrashLoopBackOff), Running, and Terminated (with an exit code).

What happens when a pod is deleted

1. The pod is marked Terminating and removed from Service endpoints.
2. A preStop hook (if any) runs.
3. Containers receive SIGTERM and get terminationGracePeriodSeconds (30 seconds by default) to finish.
4. After the grace period, anything still running is killed with SIGKILL.

Your application must handle SIGTERM by finishing in-flight requests and exiting. An app that ignores it gets killed mid-request on every deploy.

Probes: how Kubernetes decides if your app is healthy

flowchart TD
  A[Container starts] --> B{Startup probe<br/>has the app finished booting?}
  B -- not yet --> B
  B -- passed --> C[Liveness probe begins<br/>is the process stuck?]
  B -- failed too long --> R[Restart container]
  C -- failing --> R
  C -- ok --> D[Readiness probe<br/>can it serve traffic now?]
  D -- ready --> E[Receives Service traffic]
  D -- not ready --> F[Removed from Service endpoints<br/>no restart]
  style R fill:#f87171,color:#fff
  style E fill:#34d399,color:#062

containers:
- name: api
  image: my-api:2.3.1
  ports: [{containerPort: 8080}]
  startupProbe:
    httpGet: {path: /healthz, port: 8080}
    periodSeconds: 5
    failureThreshold: 30            # allows up to 150 s to start
  readinessProbe:
    httpGet: {path: /ready, port: 8080}
    periodSeconds: 5
    failureThreshold: 3
  livenessProbe:
    httpGet: {path: /healthz, port: 8080}
    periodSeconds: 10
    timeoutSeconds: 2
    failureThreshold: 3

Probe rules to live by:

• Liveness must be cheap and local. Never call the database or other services; a dependency blip would restart every pod at once.
• Readiness may check essentials the pod needs to serve, such as being connected to its own critical resources, but beware of failing all pods because a shared dependency is slow.
• Use a startup probe for slow starters instead of a huge liveness delay.
• Keep probe endpoints fast and unauthenticated inside the cluster.

Init containers and sidecars

Init containers run to completion, one after another, before the main containers start. Use them to wait for a dependency, run a migration step, fetch configuration or prepare files.

spec:
  initContainers:
  - name: wait-for-db
    image: busybox:1.36
    command: ['sh','-c','until nc -z db 5432; do echo waiting; sleep 2; done']
  containers:
  - name: app
    image: my-app:1.0

Sidecars run alongside the app for its whole life and extend it without changing its code. Common examples:

• A log shipper reading files from a shared volume.
• A proxy handling TLS or service mesh traffic.
• A config reloader that refreshes settings.

Other patterns: ambassador (a local proxy to a remote service) and adapter (normalising output, such as converting metrics format). Use them sparingly; each added container adds resource cost and complexity.

Requests, limits and QoS

Resources are declared per container:

resources:
  requests: {cpu: "250m", memory: "256Mi"}   # what the scheduler reserves
  limits:   {cpu: "500m", memory: "512Mi"}   # the ceiling at run time

Quality-of-service classes decide who is evicted first when a node is under pressure:

Guidance: always set memory requests and limits (usually equal), always set CPU requests, and think carefully about CPU limits for latency-sensitive services because throttling can add delay even when the node is idle.

Labels and annotations

Labels are key-value pairs used to select and group objects (app=web, tier=frontend, env=prod). Annotations hold non-identifying metadata for tools (build info, owner contact). A consistent label scheme (app, version, team, environment) pays off in Services, policies, dashboards and cost allocation.

A pod checklist for production

flowchart LR
  A[Image: pinned tag, non-root] --> B[Resources: requests and limits]
  B --> C[Probes: startup, readiness, liveness]
  C --> D[Graceful shutdown: handle SIGTERM,<br/>preStop if needed]
  D --> E[Labels: app, team, env, version]
  E --> F[Security context: read-only FS,<br/>drop capabilities]
  style C fill:#6366f1,color:#fff

securityContext:
  runAsNonRoot: true
  allowPrivilegeEscalation: false
  readOnlyRootFilesystem: true
  capabilities: {drop: ["ALL"]}

How to roll this out in your organization

1. Publish a pod standard (the checklist above) as a one-page document with a copy-paste manifest.
2. Provide a base Helm chart or Kustomize template that already includes probes, resources, labels, security context and graceful shutdown, so teams start compliant.
3. Audit what exists: list pods without resource requests, probes or non-root users (a short script or a policy tool in audit mode).
4. Enforce gradually with admission policy: warn first, then block for new workloads, then for existing ones after a deadline.
5. Teach by incident: take a real past incident caused by a missing probe or limit and walk through how the standard would have prevented it.
6. Track: percentage of pods meeting the standard, OOMKills per week, restarts per pod, and failed rollouts.

Pitfalls

• Copying probe settings between services without tuning for startup time.
• Setting limits far below real usage, causing constant OOMKills or throttling.
• Using liveness probes that depend on other services.
• Making policies blocking on day one and breaking teams' deployments.

Interview questions

Q1. What is a pod, and why not run bare containers? A pod is the smallest deployable unit: one or more containers sharing network and optional storage on one node. It lets tightly coupled processes be scheduled and managed together.

Q2. Liveness vs readiness vs startup probes? Liveness restarts a stuck container, readiness controls whether the pod receives traffic, and startup protects slow starters before the other probes begin.

Q3. Why is a liveness probe that checks the database dangerous? A database slowdown would make every pod fail liveness, causing a synchronised restart storm. Liveness should test only the process itself.

Q4. Explain requests and limits. Requests are reserved for scheduling and set the guaranteed share. Limits are ceilings: exceeding a CPU limit causes throttling, exceeding a memory limit causes an OOM kill.

Q5. What are the QoS classes? Guaranteed (requests equal limits), Burstable (partial settings) and BestEffort (none). Under node pressure BestEffort pods are evicted first, Guaranteed last.

Q6. What happens when a pod is deleted? It is removed from endpoints, preStop runs, containers get SIGTERM and a grace period, and are then killed with SIGKILL if still running.

Q7. When would you use an init container? To run setup that must finish before the app starts: waiting for dependencies, schema setup, fetching config or preparing files on a shared volume.

Next: Deployments and Rollouts shows how pods are managed, updated and rolled back at scale.