Python Fundamentals for SREs: Your First Useful Scripts

Why Python?

On a typical SRE team you will find Python scripts that restart services, parse logs, call cloud APIs, check certificates, glue monitoring systems together and automate a hundred small tasks. You do not need to be a software architect. You need to be able to write a clean 50 to 100 line script quickly, under pressure, and make it safe to run in production.

flowchart LR
  A[Incident or boring task] --> B[Write a small script]
  B --> C{Safe to run?}
  C -- dry run + review --> D[Run in production]
  C -- not sure --> B
  D --> E[Log what it did]
  E --> F[Turn it into a<br/>reusable tool]
  style C fill:#6366f1,color:#fff
  style F fill:#34d399,color:#062

Setup in two minutes

python3 --version          # use 3.10 or newer (on Windows try: py --version)
python3 -m venv .venv      # one isolated environment per project
source .venv/bin/activate  # Windows: .venv\Scripts\activate
pip install requests pyyaml

Always use a virtual environment. Installing packages globally is how scripts mysteriously break on someone else's machine.

Create a file hello.py:

print("Hello, reliability!")

Run it with python hello.py. That is the entire workflow: write, run, improve.

Core types you will use daily

servers = ["web-01", "web-02", "db-01"]
config = {"service": "checkout", "port": 8080}

print(servers[0], servers[-1])        # web-01 db-01
print(config["service"])
print(config.get("timeout", 30))      # default if the key is missing

Decisions and loops

for host in servers:
    if host.startswith("db"):
        continue                       # skip databases
    print(f"checking {host}")          # f-strings: the modern way to format text

Use enumerate() when you need an index and zip() to walk two lists side by side.

Functions

def error_ratio(errors: int, total: int) -> float:
    """Return errors / total, or 0.0 when there was no traffic."""
    if total == 0:
        return 0.0
    return errors / total

print(f"{error_ratio(7, 1000):.2%}")   # 0.70%

Small functions that do one thing, type hints on the signature and a short docstring make code easy to trust and test.

Comprehensions: short and powerful

latencies = [120, 85, 310, 95, 2400, 101]
slow = [x for x in latencies if x > 300]          # [310, 2400]
by_host = {h: len(h) for h in servers}             # dict comprehension

Generators produce items one at a time, so you can process a 10 GB log file without loading it into memory:

def error_lines(path):
    with open(path, errors="replace") as f:
        for line in f:
            if " 500 " in line:
                yield line

Errors and with

import json

def load_config(path):
    try:
        with open(path) as f:             # 'with' closes the file even if something fails
            return json.load(f)
    except FileNotFoundError:
        raise SystemExit(f"config missing: {path}")      # non-zero exit code
    except json.JSONDecodeError as e:
        raise ValueError(f"bad JSON in {path}: {e}") from e

Catch specific exceptions. A bare except: hides bugs. Exit with a non-zero code when something fails so cron jobs and CI notice.

Mini project 1: a concurrent health checker

#!/usr/bin/env python3
"""Check whether a list of hosts accepts TCP connections."""
import socket, sys, time
from concurrent.futures import ThreadPoolExecutor

def check(host, port=443, timeout=3):
    start = time.perf_counter()
    try:
        with socket.create_connection((host, port), timeout=timeout):
            return host, True, round((time.perf_counter() - start) * 1000, 1)
    except OSError as e:
        return host, False, str(e)

def main(hosts):
    with ThreadPoolExecutor(max_workers=20) as pool:    # bounded concurrency
        results = list(pool.map(check, hosts))
    for host, ok, detail in results:
        print(f"{'OK  ' if ok else 'FAIL'} {host:30} {detail}")
    return 0 if all(ok for _, ok, _ in results) else 1

if __name__ == "__main__":
    sys.exit(main(sys.argv[1:] or ["example.com", "github.com"]))

Run it: python check.py example.com github.com. You just wrote a tool with concurrency, timeouts, readable output and an exit code.

Mini project 2: count status codes in a log

from collections import Counter

counts = Counter()
with open("access.log", errors="replace") as f:
    for line in f:
        parts = line.split()
        if len(parts) > 8:
            counts[parts[8][0] + "xx"] += 1       # 200 -> 2xx, 502 -> 5xx
print(dict(counts))

Add one more line to print the ratio of 5xx to the total and you have a poor man's error-rate SLI.

Mini project 3: retry with backoff and jitter

import random, time

def retry(fn, tries=5, base=0.2, cap=5):
    for attempt in range(tries):
        try:
            return fn()
        except Exception:
            if attempt == tries - 1:
                raise
            time.sleep(random.uniform(0, min(cap, base * 2 ** attempt)))   # full jitter

This tiny function embodies a big idea: never retry in lockstep. Jitter spreads retries out so a thousand clients do not hammer a recovering server at the same instant.

Traps that bite beginners

def add_host(host, hosts=[]):       # BUG: the default list is shared between calls
    hosts.append(host)
    return hosts

def add_host_ok(host, hosts=None):
    hosts = [] if hosts is None else hosts
    hosts.append(host)
    return hosts

• Lists and dicts are mutable; passing them to a function can change the caller's data.
• Use == to compare values and is None to check for none.
• Floating-point numbers are inexact; compare with a tolerance.
• Never run eval() on input from anyone else, and never hard-code secrets in a script.

A cautionary scenario: when "nothing found" meant "delete everything"

Here is an illustrative scenario, a composite of mistakes that teams really make. An engineer writes a script to clean up unused storage volumes in a test account. It looks up volumes tagged owner=team-a, then deletes every volume that matches a filter built from that list. One day the tag lookup fails quietly and returns an empty list. The filter is built from nothing, so it matches nothing to exclude, and the script treats the missing filter as "no restriction". It starts deleting every volume in the account. Because the script ran in a loop with no limit, dozens of volumes were gone before anyone noticed.

The underlying bug is one of the most common in automation: an empty input is silently treated as "all". The defences are simple, and every script that changes things should have them:

• Validate inputs, especially empty ones. An empty list is not a valid "everything".
• Cap the blast radius: never operate on more than N targets without an explicit override.
• Dry-run first, then run for real.
• Rate-limit destructive loops and add a quick way to stop them.
• Log every action so you can see exactly what happened.

def decommission(hosts, *, dry_run=True, max_hosts=5):
    if not hosts:
        raise ValueError("refusing to run: empty host list")        # empty must never mean all
    if len(hosts) > max_hosts:
        raise ValueError(f"refusing to touch {len(hosts)} hosts (limit {max_hosts})")
    for h in hosts:
        print(("DRY RUN: would decommission " if dry_run else "decommissioning ") + h)
        if not dry_run:
            ...  # real action, with logging

How to build a scripting culture in your organization

Individual scripts are fine; a shared, safe, reviewed toolbox is what scales. A plan you can start this month:

flowchart LR
  A[Create one repo:<br/>sre-tools] --> B[Add a script template<br/>with safe defaults]
  B --> C[Code review checklist<br/>and CI lint plus tests]
  C --> D[Tool catalogue<br/>README of what exists]
  D --> E[Package and share:<br/>pip install sre-tools]
  E --> F[Retire one-off scripts<br/>from laptops]
  style B fill:#6366f1,color:#fff
  style F fill:#34d399,color:#062

1. One repository for team tooling, with an owner. Scripts living on personal laptops are a bus-factor risk and a security risk.
2. A script template every new tool starts from: argument parsing, --dry-run as the default, logging, timeouts, exit codes, a docstring.
3. A review checklist for pull requests:
   • Does it handle empty and huge inputs?
   • Is there a dry-run mode and a blast-radius limit?
   • Are secrets read from the environment or a secret manager, never hard-coded or logged?
   • Do network calls have timeouts and retries with jitter?
   • Are the decisions covered by tests?
4. CI: run ruff (lint), pytest and a quick type check on every pull request.
5. A catalogue: a README listing each tool, what it does, who owns it and an example. Include "when not to use it".
6. Safe execution: run risky tools from a controlled environment (a bastion or CI job) with audit logs, not from random laptops. Use least-privilege credentials per tool.
7. Measure value: track hours saved from each tool; retire the ones nobody uses.

Mini policy you can adopt tomorrow

• Anything that changes production must support --dry-run and print what it will do.
• Anything that touches more than 10 resources needs a human confirmation or a second reviewer.
• No credentials in code, ever.
• Every tool has an owner and a runbook line.

Pitfalls

• The "genius with a laptop" problem: critical tooling known by one person.
• Scripts without tests, so nobody dares to change them.
• Over-engineering a framework before you have five useful scripts.
• Giving scripts admin credentials "to make it easy".

Interview questions

Q1. What is the difference between a list and a tuple, and a dict and a set? Lists are ordered and mutable; tuples are ordered and immutable (hashable). Dicts map keys to values with O(1) average lookup; sets store unique items with O(1) membership tests.

Q2. Why is def f(x, items=[]) a bug? The default list is created once when the function is defined and shared across calls. Use None as the default and create a new list inside.

Q3. What does a context manager give you in an ops script? Guaranteed cleanup (files closed, locks released, state restored) even when exceptions occur.

Q4. When would you use a generator? When data is large or unbounded, such as log files or API pages, because it yields items lazily with constant memory.

Q5. Write a function that retries a flaky call. What details matter? Limit attempts, exponential backoff with jitter, retry only transient errors, re-raise the last error, and log each attempt.

Next: Files, Text Processing and Log Parsing turns these basics into a real log analyser.