Embracing Risk: How Much Reliability Is Enough?

The core insight

Reliability is not free. Each extra nine of availability costs more than the one before, yet users rarely notice the difference once the service is more reliable than their own phone, Wi-Fi or ISP. SRE's job is not to maximise reliability but to find the right level and then use the remaining risk to move faster.

flowchart LR
  A[Cost of reliability] --- B(Rises steeply<br/>for each extra nine)
  C[Benefit to users] --- D(Flattens once above<br/>what users can perceive)
  B --> E{Sweet spot}
  D --> E
  E --> F[Set SLO here]
  F --> G[Remaining risk =<br/>error budget to spend on change]
  style E fill:#6366f1,color:#fff
  style G fill:#34d399,color:#062

Managing risk like an economist

Questions to ask: What revenue is lost per hour of downtime? What happens to trust? Is the harm bursty (a few big outages) or spread out (many small ones)? Are competitors better or worse?

Measuring service risk

Two practical ways to express availability:

• Time-based: uptime / (uptime + downtime). Simple, but it treats a 3 a.m. blip like a peak-hour outage.
• Request-based (aggregate): successful requests / total requests. Better for global services because it weights outages by actual impact and works when part of the fleet is down.

Request-based availability = good requests / total requests
Example: 2,999,200 successful out of 3,000,000 = 99.973%

Other dimensions you can set risk targets for: latency, freshness of data, durability of stored data, correctness.

Risk tolerance by service type

flowchart TD
  A[Consumer service?] --> B[Who are the users,<br/>and what do they expect?]
  B --> C[What does downtime<br/>cost the business?]
  C --> D[What are the other<br/>sources of failure users see?]
  D --> E[Does the product tolerate<br/>partial failure?]
  E --> F[Target availability and<br/>latency per product]

Examples that illustrate the idea:

• A consumer video service can tolerate occasional buffering; pushing from 99.9% to 99.99% mostly adds cost.
• A payments or healthcare system needs much higher durability and correctness; here the trade-off flips.
• An internal batch tool may tolerate hours of downtime; set a low target and spend engineering time elsewhere.
• Free and paid tiers can have different targets; so can different request classes (checkout versus browsing).

Infrastructure services: offer reliability tiers

Shared platforms (storage, queues, build systems) cannot guess every customer's need. Offering distinct service levels (for example low-latency with higher cost, or throughput-oriented and cheaper) lets each product team buy the reliability it actually needs. This avoids over-engineering everything to the strictest customer.

Motivation for error budgets

Developers want to ship; SREs want stability. Without data this becomes an argument. An error budget converts the SLO into a shared, objective number: budget = 1 − SLO.

sequenceDiagram
  participant P as Product
  participant D as Dev team
  participant S as SRE
  P->>D: Ship feature X
  D->>S: Release request
  S->>S: Check budget remaining
  alt Budget healthy
    S-->>D: Go ahead, canary first
  else Budget exhausted
    S-->>D: Reliability work first
    D->>P: Reprioritise, data shows why
  end

Benefits: product and SRE share one goal, innovation is encouraged (budget is meant to be spent), over-delivering reliability is treated as waste, and outages become an expected, planned cost instead of a crisis. If a service is far more reliable than its SLO, a mature team deliberately takes more risk, or injects planned downtime to ensure dependants do not assume perfection.

How to introduce risk-based thinking in your organization

Many organisations treat every outage as unacceptable and every release as dangerous, which pushes everyone into slow, fearful delivery. Changing that is a leadership conversation backed by data.

flowchart LR
  A[Step 1<br/>Tier your services<br/>by business impact] --> B[Step 2<br/>Set a target<br/>per tier, with product]
  B --> C[Step 3<br/>Write the error<br/>budget policy]
  C --> D[Step 4<br/>Use the budget<br/>in release decisions]
  D --> E[Step 5<br/>Review quarterly<br/>and adjust]
  style B fill:#6366f1,color:#fff
  style D fill:#34d399,color:#062

Step 1: tier your services. Sort services into tiers by business impact:

Different targets are the point: it frees capacity for the services that matter and removes pointless gold-plating elsewhere.

Step 2: set targets with product, not for product. For each tier-0 and tier-1 service, sit with the product manager and ask: what does an hour of downtime cost in revenue, support load and reputation? What do users tolerate? What do competitors offer? Write the answer down as a number.

Step 3: write a one-page error budget policy. Example wording you can adapt:

Get both engineering and product leaders to sign it before it is needed.

Step 4: use the budget in real decisions. Put "budget remaining" on the release checklist and the weekly planning agenda. When the budget is healthy, encourage risk: faster releases, experiments, migrations. When it is low, slow down and fix the biggest sources of burn.

Step 5: review and adjust. Each quarter ask: were the targets right? Which budgets burned and why? Is any service wildly over-delivering (spend less on it, or take more risk)? Is a service regularly missing (invest, or relax the target honestly)?

Handling the hard conversations

• "We cannot accept any downtime." Ask what the business would pay for each extra nine, and show the cost curve.
• "The freeze will hurt our launch." Point to the policy they signed, offer a middle path (ship behind a flag, add a canary), and agree an exception process.
• "Our service is already at 99.99%, so why change?" Congratulate them, then ask whether anyone depends on that perfection and whether the cost of staying there is justified.

Metrics for the programme

Percentage of tier-0 and tier-1 services with an agreed target, how many release decisions referenced the budget, incident cost trend, and deployment frequency (should rise as fear falls).

Pitfalls

• Targets chosen by engineering alone, so product does not recognise them.
• A policy with no consequences, so nobody respects it.
• Setting the same target for everything.
• Treating the budget as a stick instead of a shared tool for balanced decisions.

Interview questions

Q1. Why not set an SLO of 100%? It is impossible to achieve, users cannot perceive it, costs grow exponentially, and the target would block all change, which is the main source of improvement.

Q2. When is request-based availability better than time-based? For distributed services where part of the fleet can fail without full downtime, and where traffic varies. It weights impact correctly.

Q3. Your service is at 99.99% against a 99.9% SLO for six months. What should you do? Treat it as spare budget: ship faster, take more risks, simplify or reduce redundancy cost, and consider whether the SLO should be tightened if users genuinely need more.

Q4. How do you decide an SLO with no history? Estimate from user expectations and business impact, set a conservative initial target, measure for a few weeks, and revise.