A database you have not broken on purpose is a database you do not understand. So the primary was killed while the application was writing to it, over the application’s own connection string rather than an administrative one, because the connection an application uses is the only one whose behaviour matters.
Three hundred and twenty-three writes had been acknowledged when the process died. Three hundred and twenty-three were present afterwards. Nothing was lost, and nothing had to be replayed by hand.
That result is the reason for a decision that otherwise looks like stubbornness: the database is not rented. It runs on machines we configure, with failover we own, and the standby is synchronous, which means a write is not acknowledged until a second machine has it. The cache is the same story. Neither is a managed service.
Renting is not cheaper. It is quieter.
The usual argument for a managed database is cost of operation, and it is a real argument. It is also not the one that decided this. Two things did.
The first is who holds the keys. A managed database is a place where somebody else can read the disk. The encryption story around it is usually good and occasionally excellent, and it is still a boundary drawn by a third party rather than by us.
The second is the failure mode. When the database is somebody else’s, an outage is a status page and a wait. There is no lever. The decision here was to accept the work in exchange for the lever, and the sentence that follows is the whole trade: when the database is yours, the failover is yours, the backups are yours, and the call at three in the morning is yours.
Two numbers people run together
The recovery above produced two intervals, and treating them as one is the most common mistake in this area.
About one second is how long the system could not accept a write. That is the number a user would feel, and for most of a second of it they would feel nothing at all, because a retry covers it.
About two minutes is how long it took to have two copies again. During that window everything worked. It also would not have survived a second failure, which is a different kind of exposure and deserves a different number.
Systems are frequently described as recovering in the smaller number when what they mean is the larger one. The distinction matters most in exactly the situation where it is least convenient to think about it.
What is verified, and how often
A backup that has not been restored is a belief, not a backup. The restore runs on its own schedule, twice a week, against the real archive, and it either produces a working database or it fails loudly. It has produced one twice since it was set up.
The alarms are built to fire on silence rather than on a threshold being crossed. A system that only speaks when something is wrong is indistinguishable from a system that has stopped speaking, and the second case is the one that kills you quietly.
What is still wrong
Three things, stated because a piece like this is worth very little without them.
One region. Everything runs in a single region today, deliberately, and a region-wide failure is not survivable. The multi-region work exists in the infrastructure code and is switched off.
One cache. The cache is a single node, and the sign-in path fails closed without it. Two independent nodes would be worse rather than better, because they would disagree about counters that must not be disagreed about, so the honest description is that this is accepted rather than solved.
No copy outside the provider. Backups live in the same account as the thing they protect. That is the largest open risk in the system and it is not an oversight, it is unfinished work.
None of those three are hard to find if you go looking. Publishing them is cheaper than being asked.