Observability vs Monitoring: Metrics, Logs, Traces and the Golden Signals

A detective story

Imagine a detective arriving at a scene with only a single clue: "the window is broken". That is monitoring: it tells you that something is wrong. A great detective also needs fingerprints, camera footage, witness statements and a timeline so they can answer a question they had never thought to ask. That is observability.

Monitoring vs observability

You need both: monitoring to notice, observability to understand.

The signals

flowchart LR
  A[Metrics<br/>numbers over time] --> X[Know something is wrong]
  B[Logs<br/>detailed events] --> Y[Know exactly what happened]
  C[Traces<br/>request journeys] --> Z[Know where time is spent]
  D[Profiles<br/>code hot spots] --> W[Know which code is costly]
  E[Change events<br/>deploys, flags] --> V[Know what changed]
  style A fill:#22d3ee,color:#012
  style C fill:#6366f1,color:#fff

Golden signals, RED and USE

Three tiny frameworks tell you what to measure when you do not know where to start.

The four golden signals (for any user-facing service):

1. Latency: how long requests take (and separate success from failure latency).
2. Traffic: how much demand the service gets (requests per second).
3. Errors: the rate of failed requests.
4. Saturation: how "full" the service is (CPU, memory, queue depth, connection pool).

RED (for request-driven services): Rate, Errors, Duration. USE (for resources like CPU, disks, network): Utilisation, Saturation, Errors.

flowchart TD
  A[Something feels wrong] --> B{Service level: RED}
  B --> B1[Rate normal?]
  B --> B2[Errors up?]
  B --> B3[Duration up?]
  A --> C{Resource level: USE}
  C --> C1[Utilisation high?]
  C --> C2[Saturation: queues?]
  C --> C3[Errors on the resource?]
  B2 & B3 & C2 --> D[Narrow down by version, region, customer]

Putting the signals to work: an investigation

flowchart LR
  A[Alert: error budget burning<br/>metrics] --> B[Which service,<br/>region, version?]
  B --> C[Exemplar or filter<br/>to a trace]
  C --> D[Trace shows slow<br/>or failing span]
  D --> E[Logs for that span<br/>give the exact error]
  E --> F[Profile if CPU bound]
  F --> G[Fix and confirm<br/>on the SLO graph]
  style A fill:#f87171,color:#fff
  style G fill:#34d399,color:#062

A worked example:

1. Metrics say checkout errors rose from 0.1% to 4% at 10:31.
2. Slicing the metric by version shows only version 2026.11.07.3 is affected.
3. A trace of a failed request shows the payments call took 3 seconds then timed out.
4. The log line for that span says connection pool exhausted.
5. A change event shows the pool size was lowered in the same release.
6. Fix: restore the pool size. Confirm errors return to baseline on the SLO dashboard.

Five signals, five minutes, no guesswork.

Cardinality: the quiet bill-shock

Every unique combination of metric labels is a separate time series. A label such as user_id with a million values creates a million series and can melt your monitoring system.

• Keep metric labels bounded: service, route template, status class, region.
• Put high-cardinality details (user id, trace id) in logs and traces.
• Use route templates (/orders/{id}) not raw URLs (/orders/9917).

Logging that helps

{"ts":"2026-11-07T10:02:11Z","level":"error","service":"checkout","trace_id":"4bf92f35...","msg":"payment timeout","upstream":"psp","latency_ms":3041}

• Structured JSON, one event per line, UTC timestamps.
• Always include the trace id so logs link to traces.
• Never log secrets or personal data.
• Use levels deliberately; sample noisy debug logs; keep hot storage short and archive older logs cheaply.

Designing dashboards people actually use

1. Top row: SLO status and error budget remaining.
2. Second row: the golden signals for the service.
3. Third row: dependencies and saturation.
4. Change markers: deploys, flags and config changes drawn on every graph.

Every panel should answer a question an on-call engineer asks. If nobody has looked at a panel in a month, delete it. A dashboard with 40 graphs is not a dashboard; it is wallpaper.

Controlling cost

• Drop metrics and labels nobody uses.
• Use recording rules to pre-aggregate expensive queries.
• Sample traces (keep errors and slow ones).
• Set log level policies and retention tiers.
• Track telemetry cost per service so teams see the bill.

Method beats memory

Experienced engineers often seem to debug by magic: they know which dashboard to open and which question to ask. Usually that is not magic but accumulated scar tissue from earlier incidents. It works until the system changes, the expert is on holiday, or the failure is one nobody has seen before.

A more durable approach is to make investigation a method anyone can follow. Keep the documentation that stays true (who owns the service, how to escalate, what it depends on, a few good starting queries) and avoid encyclopaedic runbooks for every possible cause, because those go stale and wrong guidance is worse than none. Then invest in telemetry rich enough that a newer engineer can reason from evidence instead of folklore.

The same discipline applies to alerting: keep it as simple as you can. Page a human only when something needs human attention, and prefer alerts on symptoms users feel over alerts on every internal cause. A monitoring system that cries wolf trains people to ignore it.

How to implement observability in your organization

Most organisations fail by buying a tool first. Follow the order that works: standards, then instrumentation, then visualisation, then alerting, then cost control.

flowchart LR
  A[Phase 1<br/>Standards:<br/>naming, fields, ownership] --> B[Phase 2<br/>Instrument one<br/>critical journey end to end]
  B --> C[Phase 3<br/>SLO dashboards and<br/>symptom alerts]
  C --> D[Phase 4<br/>Tracing and wide events<br/>across services]
  D --> E[Phase 5<br/>Cost control,<br/>governance, training]
  style B fill:#6366f1,color:#fff
  style E fill:#34d399,color:#062

Phase 1: agree on standards (week 1 to 2).

• Pick OpenTelemetry as the instrumentation standard so you are not locked to a vendor.
• Define mandatory attributes on every event: service.name, service.version, deployment.environment, team, tier, trace_id.
• Define metric naming rules and a cardinality policy (no user ids as labels).
• Decide ownership: each service owner is responsible for its telemetry quality.

Phase 2: instrument one critical journey (week 2 to 6). Choose the revenue-critical path (for example checkout). Trace it from the edge to the database. Add structured logs with trace ids. Add the golden signals as metrics. Proving this on one journey is far more convincing than a big-bang rollout.

Phase 3: dashboards and alerts (week 6 to 10). Build a standard service dashboard template: SLO and budget on top, golden signals next, dependencies and saturation below, deploy markers on every graph. Replace cause-based alerts (CPU high) with symptom alerts tied to the SLO.

Phase 4: expand (month 3 to 6). Roll the template to other services through a shared library or Helm chart so teams get good telemetry by default. Add tracing everywhere, and the ability to query by high-cardinality fields.

Phase 5: govern and control cost (ongoing). Track telemetry volume and spend per team, set budgets, remove unused dashboards, tier retention, and train engineers to use the data.

A simple instrumentation checklist for any new service

• Emits request rate, errors and duration (RED) metrics
• Propagates trace context and exports spans
• Logs structured JSON with trace id, no secrets
• Has a standard dashboard and an SLO
• Has at most a few symptom alerts, each with a runbook link
• Marks deploys and config changes on graphs
• Documents owner, escalation and dependencies

How to win support

• Show a before and after of a real incident: time to find the cause with and without the new telemetry.
• Make it the default: templates and libraries beat documentation.
• Teach: run a lunch-and-learn where someone solves a real incident live using the data.
• Measure adoption: share of services meeting the standard, time to detect, share of incidents found by customers rather than alerts.

Pitfalls

• Buying an expensive platform before agreeing standards.
• Instrumenting everything shallowly instead of one journey deeply.
• No owner for telemetry, so quality decays.
• Ignoring cost until the bill arrives.
• Dashboards nobody reviews.

Interview questions

Q1. What is the difference between monitoring and observability? Monitoring answers predefined questions and alerts you to known failure modes. Observability lets you ask new questions about unknown failures using rich, high-dimensional telemetry such as events and traces, without shipping new code.

Q2. A latency alert fires. What do you do? Check the golden signals and recent changes, break down by endpoint, region and version, open traces of slow requests, check dependency latency and saturation, then mitigate (rollback, scale, shed load) before root-causing.

Q3. Explain the golden signals. Latency, traffic, errors and saturation: the four things that tell you whether a user-facing service is healthy and how close it is to its limits.

Q4. Why can high cardinality be dangerous? For metrics, each label combination creates a separate series, so high-cardinality labels exhaust memory and cost. Keep metrics bounded and put detail into events and traces.

Q5. Your logging bill exploded. How do you reduce it? Find top producers, cut debug logs, sample repetitive entries, turn recurring log patterns into metrics, shorten hot retention, archive to object storage and set per-team budgets.

Next: Prometheus and PromQL shows how to collect and query these metrics.