ENGINEERING NOTE

Recovery is part of the design.

A deployment is incomplete until you can explain how to recover its code, its data and everything that still remembers the previous version.

Low Kah Yi · 27 September 2026 · Updated 25 October 2026

The useful question after a deployment is not only whether it works now. It is what happens when the next change is wrong. A production system needs a recovery path that was designed with it, not one improvised after it has failed.

Code, data and media recover differently.#

Code is the easy part. Earlier versions still exist, and returning to one takes seconds. This site records the versions running before each release and returns to them automatically if the release fails its checks.

Data does not work that way. Rolling the code back does not roll a database migration back, so the previous version of the code has to be able to read the new schema. In practice that means additive migrations: add the new column, move the code over, and remove the old column in a later release, once nothing reads it.

Media and keys sit outside both. A database export contains references to images, not the images themselves, so media need their own backup. Encrypted fields in the CMS are unreadable without their key, so the key is part of the data backup even though it lives somewhere else.

Treat code rollback, database recovery and media recovery as separate operations, each with its own evidence that it worked.

Caches are state too.#

In October, a release of this site changed its stylesheets. The edge cache still held writing pages rendered by the previous version, and those pages pointed at stylesheets the new release no longer served. Every check on the new code passed, and readers saw an unstyled archive.

Nothing had failed in the usual sense. The cache was doing its job, remembering a page that had been correct when it was stored. The repair was to make every cached page carry the identity of the deployment that rendered it, so a new release can never serve a page built by an old one.

Anything that remembers an earlier version belongs in the recovery design: caches, queued work, browser storage, links that other people have already shared. A release changes all of them, whether the plan mentions them or not.

A backup is a claim until it has been restored.#

A backup that has never been restored shows that a file exists, not that recovery works. A useful rehearsal restores into an isolated environment and checks the things a reader or editor would notice: can you sign in to the CMS, do the articles and their revisions open, and do the media references still resolve?

Record which version of the code can read the backup, which credentials the restore needs, and how long it actually took. A recovery time measured during a rehearsal is worth more than any estimate. Point-in-time recovery helps too, but only within its retention window, and only for the data it covers.

Keep the architecture legible.#

This site separates editorial content from experimental scenarios. Each has its own application and its own database. That boundary keeps a simulator migration from touching the writing archive, and it keeps the recovery of each one simple enough to explain. One codebase keeps the overall design consistent.

A boundary you can draw on one page is a boundary you can recover around. The more a system's parts share, the more any single recovery has to restore at once.

Write the recovery down before the release.#

For each change, it is worth answering four questions before it ships:

1. What changes in code, schema, media, cached content and configuration?
2. How is each of those reversed, and which of them cannot be?
3. What would show that each part has actually recovered, not just restarted?
4. Who needs to know, and how long can the service wait?

The same discipline applies to changing any system that cannot stop: decide the stop points, the recoverable state and the evidence before the work begins. Recovery itself is a sequence rather than a switch, and the last step is evidence.

Know the boundary.#

This note describes the recovery design of this personal site and general engineering practice. It is not a recovery plan for any other system.