The Alarm That Couldn’t Ring

The release job ran on schedule and reported success on every run. It did that for days. I had no reason to open it, because I open things that are red, and so does everyone I have worked with. A piece of finished work that should have been in production, and wasn’t, gave it away. I went to look at a job I had been ignoring because it kept telling me it was fine, and found it had been skipping some of the work and failing on the rest, and recording a success either way. The log was accurate about one thing, which was that the job had run.

I could have filed that under bad luck. I went looking for the same pattern instead and found it three more times in a few weeks of work on the same product. A set of production alarms had read healthy since a platform migration. The migration took away the signal they listened for and nothing replaced it, so they couldn’t fire. One of them counted hundreds of errors on the day of a real incident and raised none of them, and it sat at healthy for the whole of that day.

A health check had passed for weeks against a service that could not start. The check asked whether the port was open, and the port was open, so it passed. Nobody had sent that service real traffic, which meant the first people to learn the truth were users trying to log in. Then there was the AI code review gate. It reviewed changes before merge until it hit its spending cap, and after that it reviewed nothing, and for a while nobody noticed. An engineer waiting on a review has no way to tell a gate with no objections from a gate that has stopped reading.

Careful people built all four of these checks, and I count myself among them. Each check was right about the thing it measured. The release job did run. The port was open. Each one answered a question a little to the side of the one we cared about, and the failure sat in the gap between the two. We asked whether the job ran when we meant whether the work arrived. We asked whether the port was open when we meant whether a person could log in. You write the easy question on the day you build the check because the easy question is cheap to automate, and then you read its answer as though you had asked the hard one.

Two conditions made this worse, and both are common. The first is a migration. Moving a platform is the moment checks lose their footing, because the data comes across and the things that watch the data often don’t. We found backups that had stopped after the same move, on servers that reported themselves healthy throughout. The second is noise. Our monitoring had been sending the same failing alert to the same few inboxes more than a hundred times a day, from a check that could not pass. People stopped reading it, which is the correct response to an alert that fires all day, and a real alert sent into that inbox had nowhere to land.

The rule I work to now is that a check does not count until someone has watched it fail. You break the thing on purpose in a safe place, you see the alarm fire and reach a person, and after that you trust it. This costs time that feels wasted, because you are spending an afternoon proving that something works when it already says it works. We rebuilt the backup alarm on that principle: it goes and looks for the latest backup and complains when it can’t find one, where the old one asked the system for its opinion of itself. The rule matters more now that AI tools write so many of our checks at speed. A generated test that has passed since the minute it was written tells you as little as my release job did.

I still like a green dashboard. I have stopped reading it as proof. Green tells me a check has nothing to report, and a broken check has nothing to report either, so the colour alone can’t separate a healthy system from an unwatched one. The question I ask about any check I inherit or build is when it last went red, and who saw it. If nobody can remember, I assume the alarm can’t ring, and I go and find out before a customer does it for me.