Reliability Engineering And SLOs

Purpose

Reliability engineering is about making systems behave correctly and predictably over time, especially when things fail.

Reliability is not the same as "no incidents." Reliable teams expect failure, design for it, detect it, respond well, and learn quickly.

SLI, SLO, SLA

Example:

• SLI: percentage of valid API requests that succeed.
• SLO: 99.5% of valid requests succeed over 30 days.
• SLA: contract says customer receives service credit if availability drops below agreed threshold.

Error Budgets

An error budget is the acceptable amount of unreliability within an SLO.

If the target is 99.5% availability over a month, the budget is 0.5% failure. This makes reliability a product and engineering tradeoff rather than a shouting match.

When error budget is healthy:

• Teams can release normally.
• Experiments can continue.
• Reliability risk is acceptable.

When error budget is burning too fast:

• Slow risky releases.
• Fix recurring failures.
• Improve tests, observability, and rollback.
• Reduce operational risk.

Reliability Metrics

Useful metrics:

• Availability.
• Error rate.
• Latency percentiles.
• Saturation.
• Durability.
• Correctness.
• Recovery time objective, or RTO.
• Recovery point objective, or RPO.
• Incident frequency.
• Mean time to detect, acknowledge, mitigate, and recover.

Reliability should be measured from the user's point of view where possible. A server can be healthy while the customer experience is unhealthy.

Failure Design

Design for:

• Dependency failure.
• Network timeout.
• Slow response.
• Partial outage.
• Database failover.
• Queue backlog.
• Bad deployment.
• Config mistake.
• Certificate expiry.
• Increased traffic.
• Operator error.

Useful patterns:

• Timeouts.
• Retries with exponential backoff and jitter.
• Circuit breakers.
• Bulkheads.
• Idempotency.
• Graceful degradation.
• Rate limiting.
• Backpressure.
• Health checks.
• Blue-green or canary deployments.

Incident Management

Every team should understand:

• Severity levels.
• Incident commander role.
• Communication channel.
• Customer/stakeholder update expectations.
• Rollback process.
• Runbook location.
• Escalation path.
• Postmortem process.

Postmortems should focus on learning, not blame. The system allowed the incident; the team improves the system.

Reliability In Product Decisions

Reliability has cost. Ask:

• How critical is this service?
• What happens if it is down for 5 minutes, 1 hour, or 1 day?
• Does the customer need real-time behavior or eventual completion?
• What is the acceptable data loss?
• Is the expensive HA design justified?

Not every internal tool needs five-nines architecture. Not every customer-facing system can survive on hope and one backup.

Team Reference Guide

Guidelines For Teams

• Define SLOs for critical services.
• Alert on user-impacting symptoms, not every noisy metric.
• Create runbooks for common failures.
• Practice rollback and recovery.
• Use incidents to improve design, tests, and automation.

Reflection Questions

• What reliability promise do users assume we make?
• What failure mode would create the biggest customer pain?
• Which alert wakes people but does not require action?
• What incident pattern should become an engineering improvement?

Further Study

• Google SRE Service Level Objectives: https://sre.google/sre-book/service-level-objectives/
• Google SRE Monitoring Distributed Systems: https://sre.google/sre-book/monitoring-distributed-systems/
• Google SRE Managing Incidents: https://sre.google/sre-book/managing-incidents/
• Google SRE Postmortem Culture: https://sre.google/workbook/postmortem-culture/
• AWS reliability pillar: https://docs.aws.amazon.com/wellarchitected/latest/reliability-pillar/welcome.html
• Azure reliability pillar: https://learn.microsoft.com/en-us/azure/well-architected/reliability/