Research Safety Company
News
← News

Why the database is ours

The leader was killed mid-write, on purpose, over the application’s own connection. 323 writes were acknowledged and 323 survived.

Timeline of the induced failure. The leader is killed at zero seconds, writes resume after about one second, and redundancy is restored after about two minutes. 0s ~1s ~120s leader killed writes resume redundancy restored no writes accepted serving, on a single copy induced failure of the primary, measured from the application side
Figure 1 The two numbers are not the same number. Writing came back in about a second. Redundancy took about two minutes, and for that whole window the cluster was serving from a single copy.

A database you have not broken on purpose is a database you do not understand. So the primary was killed while the application was writing to it, over the application’s own connection string rather than an administrative one, because the connection an application uses is the only one whose behaviour matters.

Three hundred and twenty-three writes had been acknowledged when the process died. Three hundred and twenty-three were present afterwards. Nothing was lost, and nothing had to be replayed by hand.

That result is the reason for a decision that otherwise looks like stubbornness: the database is not rented. It runs on machines we configure, with failover we own, and the standby is synchronous, which means a write is not acknowledged until a second machine has it. The cache is the same story. Neither is a managed service.

Renting is not cheaper. It is quieter.

The usual argument for a managed database is cost of operation, and it is a real argument. It is also not the one that decided this. Two things did.

The first is who holds the keys. A managed database is a place where somebody else can read the disk. The encryption story around it is usually good and occasionally excellent, and it is still a boundary drawn by a third party rather than by us.

The second is the failure mode. When the database is somebody else’s, an outage is a status page and a wait. There is no lever. The decision here was to accept the work in exchange for the lever, and the sentence that follows is the whole trade: when the database is yours, the failover is yours, the backups are yours, and the call at three in the morning is yours.

Two numbers people run together

The recovery above produced two intervals, and treating them as one is the most common mistake in this area.

About one second is how long the system could not accept a write. That is the number a user would feel, and for most of a second of it they would feel nothing at all, because a retry covers it.

About two minutes is how long it took to have two copies again. During that window everything worked. It also would not have survived a second failure, which is a different kind of exposure and deserves a different number.

Systems are frequently described as recovering in the smaller number when what they mean is the larger one. The distinction matters most in exactly the situation where it is least convenient to think about it.

What is verified, and how often

A backup that has not been restored is a belief, not a backup. The restore runs on its own schedule, twice a week, against the real archive, and it either produces a working database or it fails loudly. It has produced one twice since it was set up.

The alarms are built to fire on silence rather than on a threshold being crossed. A system that only speaks when something is wrong is indistinguishable from a system that has stopped speaking, and the second case is the one that kills you quietly.

What is still wrong

Three things, stated because a piece like this is worth very little without them.

One region. Everything runs in a single region today, deliberately, and a region-wide failure is not survivable. The multi-region work exists in the infrastructure code and is switched off.

One cache. The cache is a single node, and the sign-in path fails closed without it. Two independent nodes would be worse rather than better, because they would disagree about counters that must not be disagreed about, so the honest description is that this is accepted rather than solved.

No copy outside the provider. Backups live in the same account as the thing they protect. That is the largest open risk in the system and it is not an oversight, it is unfinished work.

None of those three are hard to find if you go looking. Publishing them is cheaper than being asked.

Notes

1 Method. The failure was induced by terminating the primary process while a client was writing continuously, using the same connection string the application uses in production. Acknowledged writes were counted client-side before the kill and compared against rows present after recovery.
2 Timings. Both intervals are approximate and were taken from the application side, which is where they are felt. They are one observation of one induced failure, not a distribution, and should not be read as a guarantee.
3 What this does not show. A single killed process is the easiest failure to survive. Nothing here says anything about a corrupted disk, a bad migration, or a region going away, and the third of those is currently not survivable at all.