For five days, every alarm in the monitoring system evaluated correctly, decided correctly, and delivered nothing. Seven of the eleven had never been capable of delivering anything at all, from the moment they were created.
No dashboard showed a problem, because from a dashboard’s point of view there was none. The alarms were healthy. The delivery path was not, and nothing was watching the delivery path.
How it happened
The notification channel was encrypted with a managed key belonging to the cloud provider. Managed keys of that kind grant access to identity principals. The alarm service is not an identity principal; it is a service principal, and the managed key cannot be edited to allow one.
So the alarm fired, the notification was submitted, the encryption step refused it, and the failure was recorded in a place nobody had a reason to look. Every layer did exactly what it was configured to do.
Why five days
Because nothing failed. That is the whole answer, and it is why this class of fault is dangerous: the absence of a page is indistinguishable from the absence of a problem, and the second interpretation is the comfortable one.
It was found while checking something else.
What changed
The key was replaced with one that can grant a service. The managed key was the wrong instrument and could not be made into the right one.
Every alarm now has to prove it can wake somebody. Delivery is tested end to end, to a real destination, rather than inferred from the alarm being configured. An alarm that has never delivered is treated as broken, not as quiet.
The alarms fire on silence. A heartbeat that stops is now itself an alarm, so a system that goes quiet is reported rather than assumed healthy.
What is still true
This was found by accident, not by a check. The changes above make this specific failure visible; they do not make the general class of it visible, and no claim is made that they do.
The general lesson is cheaper to state than to act on: a monitoring system is not verified until something has been woken by it. Everything short of that is a configuration that looks correct.