Chaos Engineering Fundamentals: Break Things on Purpose, Safely

A story to begin with

What it is (and is not)

Chaos engineering is disciplined experimentation on a system to build confidence in its ability to withstand turbulent conditions. It is the scientific method applied to reliability. It is not "randomly break things and hope".

The experiment loop

flowchart LR
  A[Define steady state<br/>SLI baseline] --> B[Hypothesis:<br/>steady state holds]
  B --> C[Inject failure<br/>small blast radius]
  C --> D[Observe SLIs,<br/>alerts, behaviour]
  D --> E{Hypothesis held?}
  E -- yes --> F[Widen scope or<br/>automate the experiment]
  E -- no --> G[Weakness found:<br/>fix and rerun]
  D -. abort condition hit .-> H[Stop and roll back]
  style E fill:#6366f1,color:#fff
  style H fill:#f87171,color:#fff
  style G fill:#fbbf24,color:#000

1. Steady state: a measurable sign the system is healthy (orders per minute, p99 latency, error rate).
2. Hypothesis: "If one pod of checkout dies, the error rate stays below 0.1%."
3. Inject a real-world event: pod or node loss, latency, packet loss, dependency failure, disk full.
4. Observe: did the steady state hold? Did alerts fire? Did people know what to do?
5. Learn and fix, then re-run. Eventually automate the experiment as a regression test.

Your first experiment, step by step

Goal: prove that losing one replica does not hurt users.

1. Pick a safe target: a stateless service in staging with at least 3 replicas.
2. Define steady state: success rate above 99.9% and p99 below 500 ms on the dashboard.
3. Write the hypothesis and abort rule: "stay above 99.5% or stop immediately".
4. Tell people: post in the team channel; nobody should be surprised.
5. Run (for example kubectl delete pod <one-checkout-pod>), and watch the dashboard live.
6. Record the result: held or failed? Anything surprising (slow recovery, noisy alerts)?
7. Fix and repeat until the result is boring. Then move to production with a tiny blast radius.

# The same experiment with Chaos Mesh
apiVersion: chaos-mesh.org/v1alpha1
kind: PodChaos
metadata: {name: kill-one-checkout-pod, namespace: chaos-testing}
spec:
  action: pod-kill
  mode: one
  selector: {namespaces: [shop], labelSelectors: {app: checkout}}
  duration: 60s

Safety rules

flowchart TD
  A[Plan experiment] --> B[Start in staging]
  B --> C[Smallest blast radius:<br/>1 pod, 1% traffic]
  C --> D[Automatic abort tied<br/>to SLO burn]
  D --> E[Announce to everyone]
  E --> F[Avoid peak hours<br/>and active incidents]
  F --> G[Run, observe, document]

Always: a tested abort button, ownership of the experiment, notification of stakeholders, no experiments during incidents or major sales events, and a plan to restore the system. Move to production only after staging results are boring.

What to test: a failure catalogue

Tools: Chaos Mesh and LitmusChaos (Kubernetes), Gremlin (managed), AWS Fault Injection Service, Azure Chaos Studio, Toxiproxy (dependencies), Istio fault injection.

Game day: a ready-made plan (90 minutes)

Capture: detection time, whether alerts were useful, what the responders had to guess, and which runbook step was wrong. Track the action items to closure like postmortem items.

Maturity ladder

flowchart LR
  A[Ad hoc<br/>one-off tests] --> B[Game days<br/>in staging]
  B --> C[Scheduled experiments<br/>in production]
  C --> D[Continuous chaos<br/>in CI/CD with SLO gates]
  style D fill:#34d399,color:#062

Getting buy-in from sceptics

• Start with incident data: "These three outages cost us X; two were failure modes we could have tested."
• Propose a tiny, reversible first step in staging.
• Promise no surprises: abort criteria, announcements and a clear owner.
• Show results quickly: one fixed weakness is worth ten slide decks.
• Frame it as insurance and training for on-call, not as risk.

A cautionary scenario: the experiment that was "too small to matter"

This is an illustrative composite. A team wants to prove that nothing important depends on a rarely used cache cluster, so they plan an experiment: block access to one cache node out of fifty. The plan was reviewed and looked tiny. Within minutes, several services report slow requests and errors. One shared client library, used by many teams, retried failed cache calls without a timeout, and threads piled up waiting.

The team aborts immediately, but then discovers that the rollback step had never been tried, so restoring access takes much longer than the experiment itself. They recover by an alternative route they had rehearsed, while a library fix is prepared in parallel. Total impact: roughly an hour.

What a good team takes from this:

• A reviewed, small experiment can still expose an interaction nobody understood. That is exactly why you run experiments, and why you keep the blast radius tiny.
• Announce experiments and use the normal incident process, even for self-inflicted trouble.
• Test the rollback first, in a safe place, before running the experiment that needs it.
• Fix the cause permanently (here, a shared library) and schedule a re-run so the lesson sticks.

Notice the tone: no blame, quick abort, parallel recovery and a lasting fix. That is the culture you are trying to build.

How to start a chaos engineering programme in your organization

Think of it as a 12-week rollout, not a tool installation.

flowchart LR
  A[Weeks 1 to 2<br/>Sponsor, scope,<br/>safety rules] --> B[Weeks 3 to 4<br/>Observability readiness<br/>steady-state SLIs]
  B --> C[Weeks 5 to 6<br/>First game day<br/>in staging]
  C --> D[Weeks 7 to 9<br/>Fix findings,<br/>test rollbacks]
  D --> E[Weeks 10 to 12<br/>First production experiment<br/>tiny blast radius]
  E --> F[Quarterly cadence<br/>and automation]
  style C fill:#6366f1,color:#fff
  style F fill:#34d399,color:#062

Prerequisites (do not skip). Chaos on a blind system teaches nothing. Before your first experiment you need: SLIs and a dashboard that shows steady state, working alerts, an incident process, and a tested rollback for anything you will break. If these are missing, fix them first; that alone is a valuable outcome.

Step 1: get a sponsor and write the rules (week 1 to 2). Agree a one-page policy with an engineering leader:

• Who may run experiments and who approves them.
• Environments allowed (staging first).
• Mandatory elements: hypothesis, blast radius, abort condition, owner, notification.
• Blackout periods (peak sales, active incidents, freezes).

Step 2: choose a first target (week 2). A stateless service with three or more replicas and good dashboards. Avoid stateful systems and unknown dependencies for now.

Step 3: build an experiment catalogue. Start with five cheap, high-value experiments:

1. Kill one pod or instance.
2. Add 300 ms latency to one dependency.
3. Return errors from one dependency.
4. Drain one node or availability zone.
5. Expire a certificate or credential in staging. Keep them as code in Git (Chaos Mesh, Litmus, AWS FIS or simple scripts) with a README for each: hypothesis, steps, expected result, abort rule.

Step 4: run your first game day in staging (week 5 to 6). Use the 90-minute plan in this lesson. Invite developers, SREs and a product representative. Assign roles: facilitator, operator, observers, scribe.

Step 5: fix, then repeat. Convert findings into prioritised tickets. Re-run the same experiment after each fix and watch the result change from "failed" to "held". Publish a short summary: what we tested, what we found, what we fixed.

Step 6: go to production carefully (week 10 to 12). Choose a low-traffic window, 1% of traffic or a single replica, an automatic abort tied to your SLO burn alert, a heads-up to support and leadership, and a rollback you have already tested.

Step 7: make it routine. A quarterly game day, scheduled automated experiments in staging, and SLO-gated experiments in production (skip them if the error budget is burning).

Metrics that show value

Number of weaknesses found and fixed, time to detect and mitigate in game days, share of Tier-0 services with tested failover, number of incidents caused by failure modes you had already tested (this should trend toward zero), and on-call confidence in surveys.

Pitfalls

• Starting in production before staging results are boring.
• Experiments without a hypothesis or abort condition.
• Running chaos without telling anyone, then calling it a surprise.
• Treating findings as embarrassments instead of wins.
• Testing infrastructure only and never testing people, runbooks and communication.

Interview questions

Q1. How do you convince a sceptical VP to allow chaos in production? Start with a staging game day, show data on past outages and their cost, propose a minimal blast-radius experiment with an abort button, and frame it as insurance that finds failures on our schedule.

Q2. What is blast radius? The maximum set of users or systems an experiment can affect. Control it by scope (one pod, one cell, one tenant), duration and automatic abort.

Q3. What do you do if chaos exposes a serious weakness? That is a success. File a prioritised fix, add monitoring that would have caught it, and rerun the experiment after the fix.

Q4. What is a steady-state hypothesis? A prediction that a measurable indicator of normal behaviour (such as success rate or latency) will remain within bounds when a specific failure is injected.

Q5. Give three experiments you would run on a Kubernetes service. Kill pods to test replicas and graceful shutdown, add latency to a dependency to test timeouts and circuit breakers, and drain a node or zone to test spread constraints and capacity headroom.

Next: unlock Chaos Experiments in Practice for Kubernetes, AWS and dependency failure experiments, plus how to build a chaos programme.