Before this:Alerting & on-call
Incident response & runbooks
Key takeaways An incident is unplanned user impact. Responding to one is a repeatable flow: detect it (usually via an alert), mitigate to stop the bleeding before you fully understand it, communicate with users and teammates, then resolve the root cause afterwards. A runbook is the pre-written playbook for a known failure; a blameless postmortem turns each incident into a systemic fix.
Alerting wakes someone when production breaks. This lesson is what that someone does. The goal during an incident is not to be clever — it’s to reduce user impact fast, then learn from it so the same thing can’t page you twice.
The response flow
- Detect — an alert fires, or a user reports it. The clock on user impact is running.
- Mitigate — stop the bleeding before diagnosing. Roll back the recent deploy, fail over, restart — whatever ends the impact fastest.
- Communicate — tell affected users and teammates what’s happening and when you’ll update next, even if the update is “still investigating.”
- Resolve — once users are safe, calmly find and fix the root cause.
Mitigate before you diagnose
The instinct to understand why before acting is the wrong one mid-incident. Every minute spent debugging is a minute of user impact. If the incident started right after a deploy, your fastest mitigation is almost always a rollback to the last known-good version:
# stop the bleeding first — roll back, understand later
docker compose up -d # recreate on the previous known-good image
journalctl -u gophertrunk -n 200 --no-pager # capture logs for the postmortem
Note the second command: grab the evidence before it rotates away, then move on. Understanding is for the postmortem; right now you’re ending impact.
Severity sets the response
Not every incident is all-hands. A severity level scales the response to the impact:
| Severity | Impact | Response |
|---|---|---|
| SEV1 | Full outage / data loss | All hands, incident commander, active comms |
| SEV2 | Major feature broken for many | On-call + owner, regular updates |
| SEV3 | Minor / degraded, workaround exists | On-call handles, ticket to follow up |
Declaring the severity early tells everyone how hard to pull the fire alarm — and stops a SEV3 from consuming a team that a SEV1 will need.
Runbooks: don’t improvise the known
A runbook is a short, pre-written playbook for a known failure mode: the symptom,
how to confirm it, and the exact steps to mitigate. Every alert
should link one, so a half-awake responder follows steps instead of inventing them. A
runbook for GopherTrunk losing its control channel might read: confirm via /api/v1/health,
check the SDR is enumerated with lsusb, restart the service, escalate to the owner if it
doesn’t recover in 10 minutes. Boring, specific, and exactly what you want at 3 a.m.
The blameless postmortem
After a significant incident, write it up — timeline, impact, what happened, and what will stop it recurring. Make it blameless: focus on the system that let the failure happen, not the person at the keyboard. People act reasonably with the information they have, so the durable fixes are systemic — a missing health gate, a confusing config, an alert that fired too late. Blame makes people hide facts; blamelessness makes them share the ones that prevent the next incident. Each postmortem’s action items feed back into the next build — closing the deployment loop.
Quick check: what should you do first when an incident starts?
Recap
- An incident is unplanned user impact; respond with detect → mitigate → communicate → resolve.
- Mitigate first — roll back or restart to stop impact before diagnosing.
- Severity scales the response so effort matches impact.
- Runbooks give responders pre-written steps; link one from every alert.
- A blameless postmortem fixes the system, and its action items feed the next build.
Next up: keeping a deployment affordable — cost and resource management.
Frequently asked questions
What is a blameless postmortem?
A blameless postmortem is a write-up after an incident that focuses on what in the system let the failure happen, not on who made a mistake. People act reasonably given the information and tools they had, so the fix is almost always a systemic one — a missing guardrail, a confusing interface, a gap in monitoring. Removing blame is what makes people honest, which is what makes the analysis useful.
What's the difference between mitigating and resolving an incident?
Mitigating means stopping the user impact as fast as possible — often by rolling back, failing over, or restarting — even if you don’t yet understand the root cause. Resolving means actually fixing the underlying cause so it can’t recur. During an incident you mitigate first to stop the bleeding; you resolve afterwards, calmly, once users are no longer affected.