Back up and restore control

This page backs up control and puts a backup back: control’s store, keys, kits and exports, and every rig’s events, which control keeps on its message bus. It also says what else to keep, where, and how to keep each rig’s ledger wherever the rig runs. Do it before an update, and as often as you can’t afford to lose what control knows. thermite backup and thermite restore list every option.

docker compose --profile backup run --rm backup thermite backup /backups   # Docker
podman compose --profile backup run --rm backup thermite backup /backups   # Podman

What to keep, and where

What	Where it is	How it is kept
Control’s state: control.db, its keys, the bus’s certificate authority, the backup’s key on the bus, the kits control keeps, the exports rigs send it and the knowledge base’s files (kb-files/)	Control’s state folder, /var/lib/thermite-control: the control-state volume under compose	thermite backup
Every rig’s events: the bus’s EVENTS stream, which control’s records are applied from	The bus’s store, /data in the nats service: the nats-data volume	thermite backup, by the server’s snapshot, after the store
The secrets key, which opens the sealed keys and sign-ins in control.db	The control-secrets volume under compose, /var/lib/thermite-control-secrets (THERMITE_CONTROL_SECRETS_KEY_FILE)	By you, apart from the backups: Keep the secrets key. A backup never holds it
The tokens	.dev/secrets/ next to compose.yaml, or only their hashes in .env	By you: they are files you made
Each rig’s ledger: everything the rig recorded	The rig’s harness state	A full export: Keep a rig’s ledger
The champions’ code	The results repository, or control’s exports	Your Git host; control’s exports are in its backup

A backup leaves out what control makes again as it starts: the bus’s other streams (REPORTS, an inbox whose pages a rig sends again; COMMANDS, which control publishes again from its store; LIVE, the newest live lines; CONFIG, which control writes again; KB_FILES, the knowledge base’s files on their way: a file’s chunks stay only until control keeps it in kb-files/, which a backup holds), the files it writes for the bus’s server (nats-config), and its public files (control-pub).

A backup holds control’s signing key and the bus’s certificate authority: whoever has it can act as control. Keep it as you keep control’s state. It does not hold the secrets key, so it opens none of the keys and sign-ins sealed in control.db: thermite backup says left out secrets.key: a backup never holds the secrets key; keep it apart when control keeps the key in its state folder.

Before you start

• The stack from Install on one machine, started at least once with this version: control makes the backup’s key on the bus, bus-backup.seed, as it starts.
• The folder .dev/backups/ next to compose.yaml. docker compose run --rm init makes it, as at each update; in a checkout of the source code, npm run dev:init.
• Room for the backup: about the size of control.db and its exports, and about 65 bytes for each event on the bus (500,000 events made a snapshot of 32 MB).
• With Podman, write podman compose where the steps say docker compose.

Back up control

1. From the source code, build the backup service’s image after each update: it is in a profile of its own, which up --build neither starts nor builds. A release’s compose file runs control’s own image, and needs no build.

   docker compose --profile backup build backup   # Docker
   podman compose --profile backup build backup   # Podman

2. Run the backup. Control may go on running.

   docker compose --profile backup run --rm backup thermite backup /backups   # Docker
   podman compose --profile backup run --rm backup thermite backup /backups   # Podman

   It prints each step, and ends with the backup’s folder:

   copied control.db: schema 14, 2539520 bytes
   copied kits/: 12 files
   copied exports/: 3 files
   copied kb-files/: 5 files
   took EVENTS's snapshot: 48210 messages to 48210
   backup: /backups/thermite-backup-2026-10-29T10-00-00-000Z (29 files, 1 streams)

   It copies the store first, whole while control writes to it, then takes the stream’s snapshot. So control’s place in the stream, which the backup’s store holds, is at or before the stream’s end, and control, started from the backup, applies what came between.

3. Copy the folder off this machine, for example as one file:

   tar -C .dev/backups -czf thermite-backup-2026-10-29.tgz thermite-backup-2026-10-29T10-00-00-000Z

4. Keep the secrets key, once and again whenever it changes: Keep the secrets key.

To tell that it worked: the last line names the folder, and the folder’s name does not end in .partial (a backup that stopped part way keeps that end, and no restore reads it). Its manifest.json lists each file with its SHA-256, and under streams the EVENTS stream’s messages and last_seq; under store.positions, control’s place in it, which is at most last_seq.

Keep the secrets key

The secrets key opens every key and sign-in control seals in its store. Control makes it at its first start, and it stays the same, so one copy kept apart from the backups is enough. A restore onto the same machine leaves it where it is, on the control-secrets volume.

1. Copy it off control, into a file only you can read:

   (umask 077; docker compose exec -T control cat /var/lib/thermite-control-secrets/secrets.key > control-secrets.key)   # or: podman compose exec -T …

2. Keep control-secrets.key somewhere other than the backups, such as your password manager. Whoever holds a backup and this key can open every key control stores.

To put it back on a new machine, before control’s first start there:

docker compose run --rm --no-deps -T --entrypoint sh control -c 'umask 077; cat > /var/lib/thermite-control-secrets/secrets.key' < control-secrets.key   # or: podman compose run …

Control started without it makes a new key: the keys and sign-ins of a restored store then don’t open, and Settings asks for each again (its key cannot be opened: enter it again).

Restore a backup

A restore takes control back to the backup. Rigs enrolled, campaigns made and commands issued after it are not in it. The rigs’ events after it come back from the bus, where it kept them, and from the rigs: each rig’s supervisor sends again, from control’s place, the events of its ledger that control lacks. A restore keeps what it replaces: control’s files go to replaced-<time>/ in control’s state folder, and the bus’s store to replaced-<time>/ in the bus’s volume. Remove those yourself once the restore is good. A restore leaves the secrets key as it is. A backup made before v0.1.1 holds one: its restore puts that key in control’s state folder, and control, as it starts, removes it there when the control-secrets volume holds the same key, moves it there when the volume holds none, and refuses to start when the two differ (Troubleshoot).

The server’s restore makes a stream only where it has none of that name. So the restore first checks the stream the bus holds. It keeps a stream that holds the backup’s events unchanged, with or without newer ones after them, as a bus that ran on since the backup does, and control, started on the backup’s store, applies the newer events from its place. A stream a retention policy purged since the backup is kept too, as long as it still holds the backup’s last event: one whose policy removed that event is refused in words (a retention policy removed it since the backup), and you restore onto an empty store instead. It refuses any other stream.

• When the bus kept its stream, as when only control’s state goes back after an update that went wrong, follow Restore onto the bus as it is: three steps, and the bus goes on running.
• When the bus lost its store, or the restore refused its stream, follow the steps below. The bus’s store is set aside first, with the bus stopped, and the restore then brings the stream back into an empty store.

1. Stop control and the bus. The rigs’ supervisors keep their places and send again later.

   docker compose stop control nats   # or: podman compose stop control nats

2. Set the bus’s store aside:

   docker compose --profile backup run --rm backup thermite restore --bus-store /bus-store /backups/thermite-backup-2026-10-29T10-00-00-000Z

   It checks every file of the backup against its manifest first, then says the bus's store: moved aside to /bus-store/replaced-….

3. Start the bus alone, with room for the restore. The server’s restore holds much of the stream in memory: about 1 KiB for each event, over 128 MiB. A server with less is stopped part way. Take the stream’s messages from the backup’s manifest.json: 2g covers about 1.9 million events.

   THERMITE_NATS_MEMORY=2g docker compose up -d --no-deps nats   # or: THERMITE_NATS_MEMORY=2g podman compose up -d --no-deps nats

4. Restore the backup. --control makes it refuse while control answers.

   docker compose --profile backup run --rm backup thermite restore --control http://control:8800 /backups/thermite-backup-2026-10-29T10-00-00-000Z

   the backup of 2026-10-29T10:00:00.000Z is whole: 24 files and 1 streams as its manifest says
   EVENTS: restoring 48,210 messages; the bus's server needs about 176 MiB of memory for it
   EVENTS: restored and checked, 48,210 messages to 48,210
   control's state: control.db, control-signing.key, control-signing.pub, bus-ca.pem, bus-ca.key, bus-backup.seed, kits, exports moved aside to /var/lib/thermite-control/replaced-2026-10-29T12-00-00-000Z, and the backup's put back
   restored: start control, and it applies the events from its place in the stream

5. Start the bus with its own memory again, and control, with your -f options:

   docker compose up -d nats control   # or: podman compose up -d nats control

To tell that it worked: control’s log, docker compose logs control, says control's records go on from the events stream's place with the place the backup’s manifest gives under store.positions, and then control's views built from the events stream, with no fatal line. /v1/health answers 200. The rigs’ pages fill again as their supervisors report, and GET /v1/bus says 0 under events.apply.behind.

Restore onto the bus as it is

When the bus kept its EVENTS stream, only control stops, and the restore checks the stream instead of bringing it back.

1. Stop control:

   docker compose stop control   # or: podman compose stop control

2. Restore the backup, with the bus running:

   docker compose --profile backup run --rm backup thermite restore --control http://control:8800 /backups/thermite-backup-2026-10-29T10-00-00-000Z

   It says how many events the bus holds after the backup’s:

   the backup of 2026-10-29T10:00:00.000Z is whole: 24 files and 1 streams as its manifest says
   EVENTS on the bus holds the backup's events and 1,312 newer ones after them, to 49,522: control applies them from its place, 48,210, as it starts
   control's state: control.db, control-signing.key, control-signing.pub, bus-ca.pem, bus-ca.key, bus-backup.seed, kits, exports moved aside to /var/lib/thermite-control/replaced-2026-10-29T12-00-00-000Z, and the backup's put back
   restored: start control, and it applies the events from its place in the stream

   A bus that holds no event after the backup’s says EVENTS is on the bus as the backup holds it instead. If the restore is refused, follow the steps above from step 1.

3. Start control, with your -f options:

   docker compose up -d control   # or: podman compose up -d control

To tell that it worked: control’s log says control's records go on from the events stream's place with the place the backup’s manifest gives under store.positions, and behind, the events it applies from there. Then GET /v1/bus says 0 under events.apply.behind, and the rigs’ pages show their newer events.

If the restore is refused

The restore checks the stream the bus holds, and each stream it finds after its restore, against the backup’s manifest: the stream’s first place, its count, its last place, and its message at the backup’s last place. It keeps a stream that is the backup’s, or the backup’s with newer events after it. A server stopped part way through a restore, for want of memory, leaves a stream that looks whole and is not, and starts with it again; the restore refuses it, naming what differs, and changes no file of control’s:

restore refused: EVENTS on the bus is not the backup's: it holds 356,496 messages to 356,496, and the backup holds 500,000 to 500,000. …

Go back to step 1 and give the bus more memory at step 3. The stream that was refused is kept, set aside with the store. Troubleshoot lists the other refusals.

Other placements of control

Thermite ships control in the compose stack. A control you run another way needs the same for thermite backup and thermite restore: control’s image, which holds thermite; control’s state folder mounted, named by THERMITE_CONTROL_DIR or --state; the bus at THERMITE_NATS_URL or --bus, reached as control reaches it, since its certificate names control’s address for it; and, for --bus-store, the bus’s store mounted, with the bus stopped. On a machine, run a container of control’s image with those volumes on control’s network. On Kubernetes, run a Job or a pod of control’s image with control’s volume claim, in control’s namespace; for a restore, scale control’s and the bus’s Deployments to 0 first, and mount the bus’s claim for --bus-store.

Keep a rig’s ledger

Every rig’s events are on control’s bus, so a backup of control holds what each rig reported. A rig’s own ledger holds the same and more, such as its research archive. To keep it, ask the rig for a Full export under Export on its page (Get a rig’s results from control), or send export_now with "full": true. A full export holds a snapshot of the ledger, and goes to the rig’s results repository, or to control, whose exports a backup of control holds. The harness also copies its ledger before every migration, and keeps the two newest copies beside it (The harness stops at start: the ledger is newer).

Where the ledger lives, for a copy of its files with the harness stopped:

Where the rig runs	The ledger	A copy of its files
The compose stack	/var/lib/thermite/ledger/ on the harness-state volume (harness2-state for local-2)	Stop the harness (docker compose stop harness), copy the volume with your engine’s tools (with Podman, podman volume export thermite_harness-state --output harness-state.tar), then start it again
A machine, set up by control	The Podman volume thermite-<rig>-state	Stop on the rig’s page, then on the machine podman volume export thermite-<rig>-state --output <rig>-state.tar, as the rig’s user, then Start
A cluster, set up by control	The volume claim thermite-<rig>-data, subpath state	Stop on the rig’s page, then a snapshot of the claim with your cluster’s storage, then Start
RunPod	The pod’s container disk, with a sealed snapshot on its volume	A full export only: the pod’s disk goes when it stops, and Terminate waits until a full export reached control