Incident management: why incidents come back

At 9:05 on a Tuesday, single sign-on starts failing for the London office. The first ticket arrives at 9:07, and forty more follow in the next half hour, each one triaged on its own. The identity team hears about it from someone who walks over. A certificate is renewed at 10:20. The postmortem a week later lists five actions, with times taken from Slack scrollback. In March the same certificate expires again, because nobody owned the action to monitor it.

Incident to resolution is the fifth stage of the IT service desk. Most teams handle the fix well. The trouble comes before it, while nobody is sure the problem counts as an incident, and after it, when the record is wrong and the follow-up work never gets done.

What is the incident management process?

The incident management process takes an outage or a service failure from detection to resolution and review. It covers declaring the incident, assigning roles, keeping people informed, restoring service, and running a postmortem whose actions are tracked until they’re done. On an IT service desk, it also links the tickets employees raise about the incident to the incident itself, so they’re answered together.

Why incidents come back

1. Declaring feels like a big decision. People wait to see if the problem clears on its own, and the waiting is the expensive part.
2. Tickets about the incident are handled one by one. Forty people report the same failure, and the service desk triages forty tickets before anyone links them.
3. Roles are volunteered mid-incident. The first ten minutes go on deciding who leads.
4. The timeline is rebuilt from memory. A week later, the times come from Slack scrollback, and they make the response look faster than it was.
5. One duration hides the slow part. Time to resolve mixes detecting, acknowledging, mitigating and fixing, so nobody knows which one to improve.
6. Postmortem actions have no owner or due date. The review ends with a list, and the list goes into a document nobody opens again.

What it costs

A repeat incident costs as much as the first one. The outage, the tickets, the people pulled off other work and the updates all happen a second time, for a cause the team already found.

A wrong timeline costs the next decision. If the record says the alert was acknowledged in two minutes when it took eleven, the team works on the wrong part of the response.

What to fix first

1. Make declaring one action. Use a slash command or a click from the alert, and ask only for a one-line description and a severity.
2. Set everything up on declare. Create the channel, assign roles from the on-call rota, and open the bridge, the status page and the incident record in one step.
3. Link related tickets to the incident. Match new tickets to the open incident on arrival, reply to each with the current status, and close them together.
4. Build the timeline from events. Record alerts, deploys and status changes with their timestamps as they happen.
5. Record four times. Capture detected, acknowledged, mitigated and resolved separately, so the slow part is visible.
6. Track postmortem actions to closure. Give each action an owner and a due date, and report open actions until they’re done.

The guide to building incident to resolution has the prompts for declaring, setting up, the timeline and the postmortem.

Where it fits

An incident usually shows up first as a run of tickets, which is why triage needs to spot the pattern before it routes forty tickets one by one. Once the incident is open, an ITSM agent in Slack can give people who ask the current status, instead of opening another ticket.

For all six stages, see the IT service desk process.