SRE Operating Model: SLOs, Error Budgets, Incidents & Toil

Site Reliability Engineering (SRE) is an operating model that applies software engineering to production reliability. DevOps is the broader effort to make delivery and operations one shared responsibility. A team can adopt SRE practices without creating an “SRE department,” but it cannot succeed without named service owners, user-centred reliability targets, and agreed actions when those targets are missed.

Start with user journeys, not infrastructure uptime

Define the small set of interactions whose failure would matter to users: successful checkout, authenticated API request, document upload, or fresh search result. A service-level indicator (SLI) measures one of those interactions. A service-level objective (SLO) sets its target over a stated window. An external service-level agreement (SLA) is a commercial or legal commitment and may use different scope, exclusions, and remedies.

A useful availability SLI has an explicit numerator and denominator, for example successful eligible requests divided by all eligible requests over a rolling 28-day window. Define success, exclusions, aggregation, and data source before choosing the percentage. Host uptime is a weak proxy if requests are slow, stale, or incorrect. Google's SLO implementation guidance begins with user journeys and requires agreement from product, development, and production owners.

Turn the error budget into a decision policy

An error budget is the unreliability allowed by the SLO: a 99.9% target leaves 0.1% of eligible events as budget. The number is useful only when a policy says what changes as it is consumed. Define burn thresholds, who can pause risky releases, which security or recovery changes remain permitted, when a post-incident review is required, and how disagreements are escalated.

Do not copy another company's thresholds without its traffic, risk, and planning cadence. Google's published example error-budget policy is an example, not a standard. A low-volume service may need event counts and qualitative review alongside percentages; a safety-critical workflow may need separate objectives for correctness and latency rather than a looser availability target.

Make service ownership and on-call sustainable

Every production service needs an owner for code, operational readiness, alert quality, dependencies, capacity, security findings, and end-of-life. SRE should not become a queue where development teams transfer systems they no longer maintain. Establish entry criteria before an SRE team accepts a service: runbooks, dependency map, deployment and rollback, capacity evidence, SLO instrumentation, and a manageable on-call load.

Page on symptoms that require urgent human action, not every anomalous metric. Each page needs an owner, an expected response, a diagnostic starting point, and a path to silence or repair the cause. Track pages per shift, after-hours interruptions, repeated alerts, and unresolved follow-up work. If the rota depends on one expert or routinely interrupts rest, the system is understaffed or insufficiently engineered—not “high performance.”

Run incidents with authority, communication, and evidence

Predefine incident roles: an incident commander coordinates decisions, an operations lead changes the system, and a communications lead updates affected stakeholders. Record a timestamped event log, decision rationale, customer impact, and current hypotheses. Prefer reversible mitigations that stop harm before pursuing a complete diagnosis. Set hand-off and stop conditions so exhausted responders do not remain the only source of context.

A learning-focused post-incident review is not an absence of accountability. It avoids reducing a system failure to individual blame, while assigning concrete owners and deadlines to corrective work. Review detection gaps, contributing technical and organisational conditions, mitigation effectiveness, and why safeguards looked reasonable before the incident. Google's SRE Workbook includes incident response, postmortem culture, overload, and engagement models in one operating framework.

Measure and remove toil deliberately

Toil is repetitive, predictable operational work that scales with service growth and leaves little enduring value: manual restarts, repetitive access changes, copy-and-paste releases, or recurring alert triage. Inventory it by frequency, time, interruption cost, error risk, and growth rate. Automate the highest recurring burden only after simplifying the underlying process; automating a broken approval chain can make mistakes faster.

Google's own 50% cap on operational work is a staffing policy in its SRE model, not a universal maturity score. The relevant outcome is enough protected engineering capacity to remove recurring work and reliability risks. Track whether automation actually lowers human interventions and whether its failure mode is safer than the manual process it replaces.

Instrument decisions and adopt the model incrementally

Collect metrics, logs, and traces around the same service boundaries and propagate request context across dependencies. The OpenTelemetry signal model provides vendor-neutral instrumentation concepts, but telemetry volume is not observability. Dashboards and alerts should answer an operational question tied to an SLO, incident, capacity limit, or deployment.

Begin with one consequential service. Name its owners, define one or two defensible SLIs, agree an error-budget policy, remove the noisiest alerts, run an incident exercise, and review toil after a full operating cycle. Common failure modes are SLOs chosen from round numbers, budgets with no consequence, “blameless” reviews with no actions, SRE ownership without developer participation, and automation that nobody can recover manually. Correct those before scaling the programme.