Incident Postmortem Template for Engineering Teams (2026)

# Postmortem: [Brief Incident Title]

**Date:** [YYYY-MM-DD]
**Severity:** Sev-[1/2/3]
**Duration:** [X hours Y minutes] (detected to resolved)
**Impact:** [Who was affected and how — e.g., '15% of API requests returned 500 errors for 45 minutes']
**Root Cause:** [One sentence — e.g., 'A database migration added a NOT NULL column without a default value, causing INSERT failures']
**Status:** Resolved / Monitoring / Action items pending

## Timeline (all times UTC)

| Time | Event |
|------|-------|
| 14:32 | Deploy #1234 goes to production |
| 14:35 | Error rate alert fires in #alerts (PagerDuty incident created) |
| 14:38 | On-call engineer [name] acknowledges alert, begins investigation |
| 14:45 | Root cause identified: database migration failing on INSERT |
| 14:48 | Decision made to roll back deployment |
| 14:52 | Rollback deployed, error rate returns to baseline |
| 14:55 | All-clear posted in #incidents, PagerDuty resolved |
| 15:30 | Fix PR opened with default value for the new column |
| 16:10 | Fix deployed and verified in production |

## Root Cause Analysis

**Direct Cause:** [What directly triggered the incident]

**Contributing Factors:**
1. [First contributing factor]
2. [Second contributing factor]
3. [Third contributing factor]

**5 Whys:**
1. Why did production fail? → [The migration added a NOT NULL column without a default]
2. Why wasn't this caught in testing? → [Our staging DB had no existing rows, so the migration succeeded]
3. Why didn't CI catch it? → [We don't run migration tests against populated databases]
4. Why not? → [Migration testing was never added to our CI pipeline]
5. Why not? → [We haven't had a migration-related incident before — it wasn't prioritized]

**Systemic Issue:** [The underlying process/system gap — e.g., 'No automated testing of database migrations against production-like data']

## Action Items

| Priority | Action | Owner | Deadline | Status |
|----------|--------|-------|----------|--------|
| P0 | Add migration testing against seeded DB to CI pipeline | [Name] | [Date] | 🔴 Not started |
| P0 | Add NOT NULL migration lint rule to pre-commit hooks | [Name] | [Date] | 🔴 Not started |
| P1 | Create runbook for database migration rollback | [Name] | [Date] | 🔴 Not started |
| P1 | Add staging environment with production-like data volume | [Name] | [Date] | 🟡 In progress |
| P2 | Tighten error rate alert threshold from 5% to 2% | [Name] | [Date] | 🔴 Not started |

## Lessons Learned

**What went well:**
- [e.g., Alert fired within 3 minutes of the deploy — monitoring is working]
- [e.g., Rollback was executed in under 5 minutes — deployment pipeline is solid]

**What went poorly:**
- [e.g., It took 10 minutes to identify root cause because migration logs weren't in our centralized logging]
- [e.g., Staging didn't catch the issue because it has no data]

**Where we got lucky:**
- [e.g., The deploy happened at 2pm, not during peak traffic at 10am — impact would have been 5x worse]

**Surprises:**
- [e.g., We didn't know that NOT NULL without a default fails on INSERT, not on the migration itself]