9 Script — Triple

| Symptom | Root Cause | Scripted Fix | |---------|------------|---------------| | 3 AM database deadlock | Single DB writer | Move to read replicas + failover automation | | Deploys break things | Manual rollbacks | Feature flags + automatic canary analysis | | “The network was weird” | No retry logic | Exponential backoff + circuit breakers | | Disk full on one node | No monitoring | Set alerts at 75% – not 99% | | TLS cert expired again | Calendar-based memory | Automate cert renewal (Certbot, Vault) |

| Reliability | Max downtime/year | Typical cost multiplier (vs 99%) | |-------------|------------------|----------------------------------| | 99% | 3.65 days | 1x (baseline) | | 99.9% | 8.76 hours | ~2-3x | | 99.99% | 52.6 minutes | ~5-10x | | 99.999% | 5.26 minutes | ~20x+ | triple 9 script

You’ve heard the phrase “Triple 9” before. But do you have the script to actually get there? | Symptom | Root Cause | Scripted Fix

Now go break things on purpose (in staging). That’s how you get to five nines in production. That’s how you get to five nines in production