SRE Practices#

Apply Site Reliability Engineering principles.

Service Level Concepts#

Choosing SLOs#

• Pick metrics users actually care about
• Start with availability + latency + durability
• Tighten SLOs over time as reliability improves
• Don't over-constrain (cost/effort vs. benefit)

Error Budgets#

Error Budget = 100% - SLO
Example: 99.9% SLO → 0.1% error budget = ~8.7 hours/month

How Error Budgets Work#

• If error budget remaining → can deploy new features
• If error budget exhausted → freeze deployments, focus on reliability
• Error budget burn rate alerts trigger incident response
• Balance innovation velocity with system stability

Incident Response#

Severity Levels#

Incident Command System#

• Incident Commander: Coordinates response
• Communications Lead: Status updates, stakeholder comms
• Operations Lead: Technical investigation
• Scribe: Timeline and action log

Postmortem Culture#

• Blameless: systems failed, not people
• Focus on: detection, response, prevention
• Action items with owners and due dates
• Share postmortems org-wide
• Track action items to completion

Reliability Patterns#

• Circuit breaker: stop cascading failures
• Bulkhead: isolate failure domains
• Retry with exponential backoff + jitter
• Rate limiting: protect against traffic spikes
• Graceful degradation: degrade features, not the whole system