More than half of respondents to Uptime's 2024 annual survey said their most recent significant outage cost more than $100,000, with one in five saying it cost more than $1 million. The bill is that high partly because of technical complexity - and partly because the communication layer breaks down while engineers are staring at dashboards. Someone forgets the 30-minute stakeholder update. Nobody pins the current severity. The scribe role is empty. By the time the incident closes, you have a channel full of noise and a postmortem with gaps.
This is a playbook for the hour between the alert and the all-clear. Not channel setup - that part is largely automated now. The hard part is running the channel well while it's live.
The two-track problem that eats your MTTR
"The incident commander outranks the CEO when you have an incident," as Slack's own case study puts it - but "it's almost like two parallel sets of work have to happen: thinking about how you have to communicate to everyone, while fixing the problem itself." That tension is where time goes.
Checking PagerDuty to find who's on-call, opening Datadog for metrics, coordinating in Slack, taking notes in Google Docs, creating Jira tickets, and updating Statuspage requires five tools and 12 minutes of logistics before troubleshooting starts. Twelve minutes before anyone touches the actual problem. Across a year of P1s, that adds up fast.
The non-obvious insight: the technical track gets all the attention in your runbook, but the comms track is where most of the toil lives. The Google SRE Book defines toil as work that is manual, repetitive, automatable, tactical, and devoid of enduring value. Manual status updates check every box. They are also the work most likely to fall through the cracks when engineers are under pressure.
What the channel needs in the first five minutes
Speed here is not about typing faster. It is about having a structure that fills itself in.
Context should be pre-loaded automatically: incident details, severity, service information, and runbook links posted to the channel the moment it opens. If someone still has to go find those things, your setup is costing you minutes on every incident.
Beyond the automated context drop, four things need to happen manually - and they need to happen in under five minutes:
- Name the incident commander. When an alert fires at 11 PM, nobody should be asking "who owns this service?" The IC is pinned in the channel topic, not buried in thread.
- Assign the scribe. T-Mobile's teams choose a scribe to capture notes as incidents are resolved. This practice creates transparent records for stakeholders, executives, and other team members to review later. The scribe is a dedicated role, not a shared assumption.
- Set the update cadence. Implement status update intervals - every 30 minutes is the standard for a SEV1. The clock starts now, not when someone remembers.
- Open the comms channel. Some organizations spin up two channels - one for primary incident response and another just for comms. Stakeholders go in the comms channel; responders stay focused in the technical one.
Keeping the channel clean while it's live
A live incident channel degrades fast. Fifteen engineers asking questions, monitoring bots firing every 90 seconds, someone sharing a Grafana link with no context. Here is the discipline that keeps it usable:
Thread everything that is not a status update. Debugging hypotheses, log snippets, and "has anyone checked X?" all go in threads. The top-level channel is for severity changes, decisions, and timed updates only.
Use a consistent status-update format. Every 30 minutes, one message, same shape: