Kubernetes Troubleshooting: A Step-by-Step Playbook for Pods, Nodes and Networking

A story to begin with

Kubernetes in five minutes

Everything in Kubernetes is a controller reconciling desired state with actual state. You declare "I want 6 replicas of this image"; controllers keep making reality match.

flowchart LR
  U[You: kubectl apply] --> API[API server<br/>stores desired state in etcd]
  API --> D[Deployment controller<br/>creates ReplicaSet]
  D --> S[Scheduler<br/>picks a node for each pod]
  S --> K[kubelet on the node<br/>pulls image, starts containers]
  K --> R[Container runtime]
  K -- reports status --> API
  style API fill:#6366f1,color:#fff
  style K fill:#22d3ee,color:#012

When something fails it fails at a specific step. That gives you a map for debugging.

Where is the pod stuck?

flowchart TD
  A[Pod created] --> B{Scheduled?}
  B -- no --> B1[Pending: resources, taints,<br/>affinity, PVC, quota]
  B -- yes --> C{Image pulled?}
  C -- no --> C1[ImagePullBackOff:<br/>tag, credentials, rate limit]
  C -- yes --> D{Container starts?}
  D -- no --> D1[CreateContainerConfigError:<br/>missing ConfigMap or Secret]
  D -- yes --> E{Stays running?}
  E -- no --> E1[CrashLoopBackOff:<br/>logs --previous, exit code, OOM]
  E -- yes --> F{Ready?}
  F -- no --> F1[Readiness probe failing:<br/>check endpoint and dependencies]
  F -- yes --> G[Serving traffic]
  style G fill:#34d399,color:#062

The debugging ladder (run it in this order)

# 1. What is unhealthy, and where?
kubectl get pods -A -o wide | grep -v Running

# 2. Read the story Kubernetes already wrote. Scroll to Events at the bottom!
kubectl describe pod <pod> -n <ns>

# 3. What did the container itself say before dying?
kubectl logs <pod> -n <ns> -c <container> --previous

# 4. What else happened in the namespace recently?
kubectl get events -n <ns> --sort-by=.lastTimestamp | tail -20

# 5. Resource usage
kubectl top pod -n <ns> ; kubectl top node

# 6. Get inside (or beside) the container
kubectl exec -it <pod> -n <ns> -- sh
kubectl debug -it <pod> -n <ns> --image=nicolaka/netshoot --target=<container>

Golden rule: describe first, logs --previous second. Most beginners skip straight to restarting things.

Scenario 1: Pod stuck in Pending

$ kubectl get pods
NAME                     READY   STATUS    RESTARTS   AGE
api-7d9c8b5f6-xk2lp      0/1     Pending   0          4m

$ kubectl describe pod api-7d9c8b5f6-xk2lp | tail -5
Events:
  Warning  FailedScheduling  0/5 nodes are available: 3 Insufficient cpu, 2 node(s) had untolerated taint {dedicated: gpu}.

Reading it: the scheduler found no node that fits. Three nodes lack CPU, two are reserved for GPU workloads. Fixes: lower the CPU request if it is oversized, add nodes (cluster autoscaler), add a toleration only if the workload belongs on those nodes, or free capacity. Also check: an unbound PersistentVolumeClaim, namespace ResourceQuota, node selectors and affinity rules.

Scenario 2: ImagePullBackOff

Events:
  Warning  Failed   Failed to pull image "registry.acme.io/api:v2.4.1": rpc error: ... 401 Unauthorized
  Warning  Failed   Error: ImagePullBackOff

Reading it: the node could not pull the image. 401 means credentials, not found means a wrong tag, 429 means a registry rate limit. Fixes: verify the tag exists, create or fix the imagePullSecret, use a pull-through cache to avoid rate limits, pin by digest.

Scenario 3: CrashLoopBackOff

$ kubectl logs api-7d9c8b5f6-xk2lp --previous | tail -3
Error: environment variable DATABASE_URL is required

Reading it: the application started and exited. The previous logs hold the reason. Common causes: missing config or secret, bad command or arguments, application bug, dependency not reachable at startup, failing liveness probe, OOM. Fixes: correct the config, fix or roll back the release (kubectl rollout undo deploy/api), add startup retries for dependencies.

Scenario 4: OOMKilled (exit code 137)

Last State:     Terminated
  Reason:       OOMKilled
  Exit Code:    137
Limits:   memory: 256Mi

Reading it: the container used more memory than its limit, so the kernel killed it. Fixes: raise the memory limit to cover real usage plus headroom; check for a leak if memory climbs steadily; for Java set -Xmx below the container limit (heap is not the only memory used). Exit codes: 0 success, 1 app error, 137 SIGKILL (often OOM), 139 segfault, 143 SIGTERM.

Scenario 5: Probes that cause outages

sequenceDiagram
  participant K as kubelet
  participant P as Pod
  participant D as Database
  Note over D: DB has a 30 second blip
  K->>P: liveness probe GET /healthz
  P->>D: SELECT 1
  D--xP: timeout
  P-->>K: 500
  K->>P: restart container (all pods at once!)
  Note over P: Outage grows from a blip to a full restart storm

Rule of thumb:

• Liveness: "is the process stuck?" Keep it local and cheap. Never call dependencies.
• Readiness: "can I serve traffic right now?" May include critical local state.
• Startup: gives slow-starting apps time before liveness begins.

Scenario 6: Service not reachable

$ kubectl get endpoints api
NAME   ENDPOINTS   AGE
api    <none>      12m

Reading it: the Service has no pods behind it. Either the label selector does not match, or no pod is Ready.

kubectl get svc api -o yaml | grep -A3 selector
kubectl get pods --show-labels
kubectl run tmp --rm -it --image=nicolaka/netshoot -- bash
# inside: nslookup api.default.svc.cluster.local ; curl -v http://api:80/healthz

Fixes: align labels and selectors, fix targetPort, make pods Ready, check NetworkPolicy and DNS (ndots, CoreDNS health).

Scenario 7: Node problems

kubectl get nodes
kubectl describe node <node> | sed -n '/Conditions/,/Addresses/p'
journalctl -u kubelet --since "30 min ago"      # on the node

Look for conditions MemoryPressure, DiskPressure, PIDPressure, NotReady. Typical causes: full disk (images, logs), runaway process, kubelet or container runtime failure, networking plugin issues, or the cloud instance itself being unhealthy. Cordon and drain: kubectl cordon <node>, kubectl drain <node> --ignore-daemonsets.

Command cheat sheet

A cautionary scenario: the Friday evening config change

This is an illustrative composite, not a report of one specific company. A platform team maintains a shared configuration that every service reads at startup. On a Friday evening an engineer fixes a small setting and rolls the change to every cluster at once, because "it is only configuration". A misspelled key causes each service to fail its startup validation, so pods begin to crash-loop everywhere within a couple of minutes.

What made it worse:

• Earlier changes to the same system had gone through a careful canary. This one was judged low risk, so it skipped it.
• Alerts fired by the hundred and buried the people trying to respond.
• The team's dashboards and chat bot ran on the same clusters, so some of their own tools were down too.
• Rolling back needed the same pipeline that was now failing.

What saved the day was luck: the author was watching the support channel, noticed the first complaints, and reverted within minutes. The lessons are the ones to adopt on purpose, not by luck:

• Canary every change, including configuration, regardless of how small it looks.
• Keep out-of-band access and a rollback path that does not depend on the broken system.
• Make alerts group and inhibit so a fleet-wide failure produces a few clear pages, not hundreds.
• Never keep your only debugging tools inside the thing you are debugging.

Map this to Kubernetes: a ConfigMap, a Helm value or an admission policy pushed to every cluster at once can create the same shape of outage. A CrashLoopBackOff across all pods right after a config change is this story in miniature.

How to implement Kubernetes troubleshooting practices in your organization

Make debugging a team capability, not one person's superpower.

flowchart TD
  A[Standardise: probes, resources,<br/>labels via policy] --> B[Runbooks for the<br/>top 7 failure modes]
  B --> C[Standard dashboards:<br/>pod health, restarts, events]
  C --> D[Safe change process:<br/>canary, progressive, rollback]
  D --> E[Training: break and fix labs]
  E --> F[Game day each quarter]
  F --> G[Review incidents,<br/>update runbooks and policy]
  G --> A
  style D fill:#6366f1,color:#fff
  style F fill:#fbbf24,color:#000

1. Standardise with policy, not documents. Use Kyverno, OPA Gatekeeper or ValidatingAdmissionPolicy to require resource requests and limits, readiness probes, no :latest tags, and owner labels. Roll out in audit mode first, fix the findings, then enforce.

2. Write runbooks for the seven common failures covered in this lesson (Pending, ImagePullBackOff, CrashLoopBackOff, OOMKilled, probe failures, unreachable Service, node problems). Each runbook is one page: symptoms, first three commands, decision tree, safe mitigations, escalation. Link them from the alerts.

3. Build one standard dashboard every service gets by default: ready versus desired pods, restarts, CPU and memory against requests and limits, recent Kubernetes events, deploy markers. Use a Helm chart or template so teams do not hand-build it.

4. Make configuration changes safe.

• Treat config as code in Git with review.
• Use GitOps (Argo CD or Flux) with progressive sync: one cluster, then a ring of clusters, then everything.
• Use Argo Rollouts or Flagger for automated canary analysis with rollback.
• Keep a documented kubectl rollout undo or Git revert path that works even when dashboards are down.

5. Train people by breaking things. Run a monthly 45-minute "break and fix" lab on a throwaway cluster: introduce one failure from this lesson and have an engineer find it using only the debugging ladder. Rotate who drives.

6. Run a quarterly game day simulating a bad config push, a node failure and a dependency outage. Measure time to detect and time to mitigate, and fix the runbooks that failed.

7. Keep out-of-band access. Make sure at least a few engineers can reach the clusters when SSO, the dashboard or the CI system is unavailable (break-glass credentials, tested regularly).

Metrics that show it is working

Mean time to mitigate for pod and node incidents, share of incidents resolved from a runbook, share of services passing the policy checks, restarts per pod per week, and the number of config pushes that were rolled back automatically before users noticed.

Pitfalls

• Enforcing policies before teams can comply, causing resentment.
• Runbooks that nobody tests.
• Rolling configuration to all clusters at the same time "because it is only a config change".
• Debug tooling that depends on the cluster you are debugging.

Interview questions

Q1. A pod is in CrashLoopBackOff. What are your steps? describe the pod and read events and exit code, check logs --previous, verify config and secrets, look at probes and resource limits, review the last image or config change, and reproduce locally with the same environment.

Q2. A deployment is stuck at 3 of 5 pods ready. Check rollout status, readiness probe failures, maxUnavailable and maxSurge, a PodDisruptionBudget, insufficient resources for surge pods, and namespace quota.

Q3. A Service is unreachable from another namespace. Use the FQDN, check endpoints exist, verify NetworkPolicy, port names, service mesh policy, kube-proxy health and DNS.

Q4. Why are liveness probes that call the database dangerous? A database blip makes every pod fail liveness, causing a synchronised restart storm that turns a small degradation into a full outage. Keep liveness local; use readiness and graceful degradation for dependencies.

Q5. What does exit code 137 mean? The container was killed with SIGKILL, most often by the OOM killer when memory exceeded its limit.

Q6. How do you debug a distroless container that has no shell? kubectl debug with an ephemeral container (for example netshoot) sharing the target container's process namespace.

Next: try the free Chaos Engineering Fundamentals lesson, or unlock Kubernetes Reliability for PodDisruptionBudgets, graceful shutdown and safe upgrades.