Operations

These are the working practices behind this site and the systems it touches. They are here because how work is run is evidence in its own right, and often better evidence than the outcome, which can be luck.

Every change is recorded, with its evidence

Change logs usually drift from reality. Nobody can say what was actually done or whether it worked. On the systems I run a change is not finished until it is recorded: who made it, why, what it touched, and how it was verified. The record carries the evidence itself: the real output of the check, the counters, the checksum of what is being served. Entries are hash-chained and versioned, so the history reads in order and an edit that was never recorded is detectable. There are 663 of them so far, across 42 tracked targets.

Because the evidence has to exist before the entry can be written, "I think that worked" never becomes a closed item at all.

Releases are editions, not edits

I publish several static sites. Editing live files in place gives no rollback and no way to see what changed between one state and the next, so a build produces a timestamped, immutable edition instead. It is checked, reviewed, staged, then promoted by moving a symlink. Rollback is moving it back, and rejected editions stay on disk rather than being tidied away.

The checks are partly mechanical, and partly semantic, because a route can answer 200 while the page is wrong. The harder judgement is knowing when a build process has started fighting the live state. When that happens the fix belongs in the live tree and the build gets retired rather than defended. This site's own history includes exactly that.

Several hosts, no collisions

More than one machine, more than one operator, human or automated, and no shared calendar. Concurrent changes are how work gets lost, or a service breaks while nobody is watching. Changes that touch a shared host are made under a named lock with an expiry, so concurrent work is visible and waits its turn, and every change is attributed to the identity that made it. Coordination is treated as a technical problem rather than a social one.

Monitoring that can interrupt a person

A monitor nobody reads is decoration, and an alert that fires on every wobble gets muted within a week. The recurring checks here are small scripts on the host itself. They record state changes rather than re-reporting a steady state, so a line that flaps is one notice and not fifty, and the checks that matter escalate to a phone. The rule I hold to is that a monitor is only real if it is allowed to interrupt someone.

Verify what is served, not what was changed

This is the mistake I have made most often and see most often. A file is edited, the edit looks correct, and the thing actually served is a different copy somewhere else. So the check after any change is against the artefact rather than the source: fetch the URL, compare the checksum with the file on disk, count the tags, re-run the route list. If the served bytes differ from what was written, the change is not done, however good the editor looked.

Two recent examples, both caught this way. A card generator that produced three byte-identical files for three different pages, which is the signature of text that never rendered at all. And a set of error pages branded with the wrong hostname, invisible in the source files and obvious the moment the served pages were read.

Deception as detection

Any system on the public internet is probed constantly. Most of it is automated and worth almost nothing, and some of it is worth reading. Alongside ordinary blocking, some surfaces here are decoys: they answer convincingly, record what is asked of them and what is run against them, and feed a review queue. To date that has produced 91 high-signal events from 91 distinct sources, with the commands each one ran. The findings are recorded in the same ledger as everything else.

What I will not publish is how any of it works. A trap that explains itself stops working, so the mechanisms stay off this page. The practice and the outcome are the whole story.

Published 2026-09-29. Describes systems in operation.