The SRE Hierarchy of Reliability

The building analogy

Maslow's hierarchy of needs says you cannot focus on self-actualisation when you are starving. A person without food, shelter, and safety cannot meaningfully pursue love, belonging, and meaning those needs require the lower levels to be met first. The SRE hierarchy works the same way: you cannot have meaningful SLOs if your monitoring is broken, just as you cannot self-actualise while starving. Each level is the prerequisite for the one above it.

Level 1: Monitoring - The absolute foundation — you cannot fix what you cannot see

Monitoring is the practice of collecting and storing data about your system's behaviour over time. Metrics (numbers), logs (events), and traces (request paths) are the three pillars. Without monitoring, your system is a black box you learn about problems only when users tell you, which is always too late.

• Signs you are at Level 1: you have basic metrics collection but it is inconsistent across services, your logs are difficult to search, you have no idea about your service's normal baseline behaviour.

• What Level 1 well looks like: every service emits the four golden signals (latency, traffic, errors, saturation), logs are structured and queryable, you have dashboards that show system state at a glance, and you know what "normal" looks like.

The four golden signals

Google's SRE Book defines four golden signals that every service should measure: Latency: how long requests take. Traffic: how much demand the system is handling. Errors: the rate of failed requests. Saturation: how "full" the service is (CPU%, memory%, queue depth). These four, measured consistently across every service, form the baseline that everything else is built on.

Level 2: Alerting - Turning monitoring data into actionable notifications

Alerting is the layer that converts monitoring data into human attention, a page, a Slack message, an email. Good alerting is much harder than most teams expect. The failure mode in both directions is severe: too many alerts causes alert fatigue, where engineers start ignoring pages. Too few alerts means problems surface too late. The goal is alerts that are always urgent, always actionable, and almost always correct.

• Signs you are at Level 2: your alerting fires too often (false positives are routine), your on-call engineer acknowledges and silences alerts without investigating, alerts fire on causes rather than symptoms, alerts do not tell the responder what to do.

• What Level 2 well looks like: every alert is urgent, every alert is actionable, false positives are rare, alerts are symptom-based not cause-based, each alert links to a runbook.

Level 3: Incident response - A structured process for when things go wrong

An incident is any event that disrupts service to users. Incident response is the structured process for detecting, coordinating, resolving, and communicating about those events. Without a defined process, incidents are chaotic multiple people making conflicting changes, no clear communication to stakeholders, no record of what was tried. A good incident response process transforms a chaotic fire into a managed recovery.

• Signs you are at Level 3: incidents are chaotic, there is no clear incident commander, stakeholders are updated inconsistently or not at all, the same person who is debugging is also writing updates, rollbacks and mitigations are not documented in real time.

• What Level 3 well looks like: clear severity levels, a defined incident commander role, consistent stakeholder communication, real-time incident timeline, defined escalation paths, a known mitigation-before-investigation principle.

The most common incident response failure

The same person who is debugging the system is also writing stakeholder updates, fielding questions from colleagues, and trying to coordinate with the team. This is a recipe for slower resolution and worse communication. The incident commander role exists precisely to separate these concerns: one person drives resolution, one person handles communication, one person maintains the timeline. In small teams, these roles collapse but the principle still applies: do not let debugging be interrupted by communication tasks.

Level 4: Post-mortems - Blameless analysis of what happened and why

A post-mortem is a written record of an incident, what happened, why it happened, how it was detected, how it was resolved, and what actions will prevent recurrence. The word "blameless" is essential: post-mortems that blame individuals teach people to hide mistakes and become defensive. Blameless post-mortems assume that intelligent, well-motivated engineers made reasonable decisions with the information they had and ask what changes to the system would prevent the same failure in the future.

• Signs you are at Level 4: post-mortems are optional or inconsistent, they focus on what people did wrong rather than what the system allowed, action items are vague or never completed, post-mortems are filed and forgotten.

• What Level 4 well looks like: post-mortems are required for every significant incident, they are blameless in both policy and tone, action items are specific and assigned, they feed back into the product backlog.

The aviation analogy

Commercial aviation has one of the strongest safety records of any human activity not because pilots never make mistakes, but because the industry treats every incident as data about the system rather than evidence of individual failure. Pilots file incident reports without fear of punishment. Near-misses are analysed as carefully as crashes. Design changes follow data. This blameless, systems-thinking culture is exactly what SRE post-mortems aim to replicate in software engineering.

Level 5: SLOs and error budgets - Explicit reliability targets with financial-grade enforcement

Service Level Objectives (SLOs) are explicit reliability targets "99.9% of requests will complete in under 200ms" agreed between the SRE team and the product team. The error budget is the flip side: the amount of unreliability you are allowed. SLOs turn reliability from a vague aspiration into a measurable, enforceable engineering property. But they only work if Levels 1–4 are solid: you cannot measure SLOs without monitoring, you cannot alert on SLO burns without alerting infrastructure, you cannot explain SLO misses without incident response and post-mortems.

• Signs you are at Level 5: SLOs are defined but not enforced, error budgets exist on paper but do not change team behaviour, SLOs are not connected to deployment decisions.

• What Level 5 well looks like: SLOs drive deployment decisions, error budget state is visible to everyone, product and engineering teams jointly own the budget, an exhausted budget triggers a reliability sprint.

Level 6: Automation and toil reduction - Eliminating repetitive work so humans can focus on novel problems

Toil is any manual, repetitive, automatable operational work that scales with the volume of your systems rather than the complexity of the problems you solve. Toil is not inherently bad — some is unavoidable. But uncapped toil grows with the system, eventually consuming all engineering time and leaving nothing for improvement. Automation is the SRE's primary tool for reducing toil: auto-remediation of known failure modes, self-healing infrastructure, automated capacity scaling, and runbooks that execute themselves.

• Signs you are at Level 6: the same runbook steps are executed manually every week, known failure modes require human intervention every time they occur, scaling happens by manual ticket rather than automatically.

• What Level 6 well looks like: known failure modes trigger automated remediation, capacity scales automatically, toil is measured and tracked, the SRE team's time is primarily spent on engineering rather than operations.

Why automation comes last, not first

Teams often reach for automation too early automating processes they do not fully understand yet, enshrining incorrect procedures in code, creating systems that auto-remediate in ways that make incidents worse. The reason automation comes at the top of the hierarchy is that you must first understand your system through monitoring, experience its failures through incidents, learn from those failures through post-mortems, and agree on its reliability targets through SLOs before you know what is worth automating and how.

The most common finding

Most DevOps engineers assess themselves at Level 3 or 4, then discover through these questions that their Level 1 and 2 foundations are weaker than they thought. Monitoring is inconsistent across services. Alerts fire too often and engineers have started treating the on-call rotation as noise. This is exactly the right moment to stop adding new practices and shore up the foundation because every level above Level 2 depends on trustworthy monitoring and alerting.