Every host has an availability page. Rather fewer describe what actually happens between something breaking and it being fixed, so here is ours.
How We Find Out
Monitoring checks the services from outside our own network, on a short interval. Outside matters. A check running on the same machine will happily report that everything is fine right up until the machine stops answering anyone.
When a check fails twice in a row, it wakes someone up. Not a rota of six people. Someone.
The First Ten Minutes
Confirm it is real and not a monitoring fault, which is a genuine possibility at three in the morning. Then establish blast radius: one site, one server, or everything.
Then the boring discipline. Restore service first, understand it afterwards. The temptation when you are tired is to work out why before fixing it. That is the wrong order when people's mail is queueing.
What Gets Said, And When
If it is affecting customers and it is going to take more than a few minutes, it goes on the status page while it is still happening rather than once it is tidy. An incident record written after the fact, when the outcome is already known, is a press release.
We would rather post something that says we do not yet know the cause, and update it, than go quiet until we have a clean story.
The Honest Trade
A small team means you get someone who knows the whole system, and no handoff between shifts where context evaporates. It also means we are not staffed like a company ten times our size, and there is no pretending otherwise.
What we can commit to is that the person who answers has access, authority and enough of the picture to act, and that the incident history stays published afterwards, including the ones that did not reflect well on us.
