Incident Runbook Generator

Create actionable runbooks for common incidents.

Runbook Template

````markdown

Runbook: Database Connection Pool Exhausted

Severity: P1 (Critical) Estimated Time to Resolve: 15-30 minutes Owner: Database Team (On-call)

Symptoms

• Application errors: "connection pool exhausted"
• Increased API latency (>5s)
• Failed health checks
• CloudWatch alarm: DatabaseConnectionsHigh

Detection

• Alert: DatabaseConnectionPoolExhausted
• Metrics: activeconnections > maxconnections * 0.9
• Logs: "Error: connect ETIMEDOUT"

Immediate Actions (5 min)

1. Verify the issue

``bash # Check current connections SELECT count(*) FROM pgstatactivity; ` ```

1. Identify long-running queries

``sql SELECT pid, now() - pgstatactivity.querystart AS duration, query FROM pgstat_activity WHERE state = 'active' ORDER BY duration DESC LIMIT 10; ``

1. Kill blocking queries (if safe)

``sql SELECT pgterminatebackend(pid) FROM pgstatactivity WHERE state = 'idle in transaction' AND now() - state_change > interval '5 minutes'; ``

Mitigation (10 min)

1. Scale up connection pool (temporary)

``bash # Update RDS parameter group aws rds modify-db-parameter-group \ --db-parameter-group-name prod-params \ --parameters "ParameterName=max_connections,ParameterValue=200" ``

1. Restart application (if needed)

``bash kubectl rollout restart deployment/api ``

1. Monitor recovery

``bash watch -n 5 'psql -c "SELECT count(*) FROM pgstatactivity;"' ``

Root Cause Investigation

Check for:

• Recent deployment (new code with connection leaks)
• Traffic spike (legitimate or DDoS)
• Slow queries holding connections
• Connection pool configuration too small
• Application not releasing connections

Rollback Steps

If caused by deployment:

# Rollback to previous version
kubectl rollout undo deployment/api

# Verify
kubectl rollout status deployment/api

Communication Template

Initial (within 5 min):

🚨 INCIDENT: Database connection pool exhausted
Status: Investigating
Impact: API errors and slowness
ETA: 15-30 min
Next update: 10 min

Update (every 10 min):

UPDATE: Killed long-running queries
Status: Mitigating
Impact: Still degraded, improving
Actions: Scaling connection pool
Next update: 10 min

Resolution:

✅ RESOLVED: Database connections normalized
Duration: 25 minutes
Root cause: Connection leak in v2.3.4
Fix: Rolled back to v2.3.3
Follow-up: Bug fix PR #1234
Postmortem: [link]

Prevention

• Add connection pool metrics to dashboards
• Implement connection timeout (30s)
• Add connection leak detection in tests
• Set up pre-deployment load testing
• Review connection pool sizing

Related Runbooks

• Database High CPU
• Slow Database Queries
• Application OOM

## Output Checklist

- [ ] Symptoms documented
- [ ] Detection criteria
- [ ] Step-by-step actions
- [ ] Owner assigned
- [ ] Rollback procedure
- [ ] Communication templates
- [ ] Prevention measures
ENDFILE