Skip to content
ThermiteThermitev0.1.3

Add a second machine

In this tutorial you add a second machine to the Thermite you started in Install on one machine. Control sets a rig up there over SSH, and you start a run of your campaign on it, beside the first.

It takes about thirty minutes, most of it the second machine downloading the images.

What you need

Section titled “What you need”
  • Thermite on your first machine, from Install on one machine, and a campaign from Run your first campaign.
  • A second machine: Linux with systemd, which you reach over SSH. It must reach your first machine, and pull the release’s images from ghcr.io.
  • An account on it for control. Make one with sudo useradd -m thermite. For control to install Podman there, give it sudo without a password: add the line thermite ALL=(ALL) NOPASSWD: ALL with sudo visudo -f /etc/sudoers.d/thermite. You can remove that file once the installs are done.
  • A name for your first machine that the second one reaches, and a reverse proxy on the first machine that serves HTTPS under it, such as Caddy or nginx. Compose shows control only to the first machine itself, at 127.0.0.1:8800. With Tailscale on both machines, Tailscale is the name and the proxy: see If you are using Tailscale.

1. Let the second machine reach control

Section titled “1. Let the second machine reach control”

  1. Point your reverse proxy at 127.0.0.1:8800, under the name, such as control.example.com. Have it send X-Forwarded-Proto: https and take request bodies of up to 90 MB. With Tailscale, tailscale serve --bg 8800 does this, and the name is the machine’s on your tailnet, the address it prints (If you are using Tailscale).

  2. In .env, next to compose.yaml on your first machine, add the address and open the message bus’s port to other machines:

    .env (part)
    THERMITE_CONTROL_PUBLIC_URL=https://control.example.com
    THERMITE_NATS_PUBLISH=0.0.0.0:4222

  3. Start Thermite again with them:

    Terminal window
    docker compose up -d   # or: podman compose up -d

  4. On the second machine, check that it reaches both:

    Terminal window
    curl https://control.example.com/v1/health
    nc -zv control.example.com 4222

    curl answers {"ok":true,"bus":"connected","degraded":false}, and nc connects. If your firewall blocks port 4222, open it to the second machine: a rig talks to control on that port. With Tailscale, publish the bus on the first machine’s tailnet address only, so that nothing outside your tailnet reaches it: THERMITE_NATS_PUBLISH=<address>:4222, with the address tailscale ip -4 prints.

2. Add the machine

Section titled “2. Add the machine”
  1. Open the dashboard, press the cog at the foot of the sidebar, then Machines under Rigs, and press Add a machine.
  2. Choose A computer over SSH, and fill in a Name such as machine-2, its Address, the SSH port (usually 22) and the User, thermite. Leave Private key empty: control makes one.
  3. Press Add and read its host key.

The machine’s row opens, with a line that starts with restrict: the public half of control’s key.

3. Let control in

Section titled “3. Let control in”
  1. On the second machine, as thermite, add that line to ~/.ssh/authorized_keys. Make the folder and the file first if they are new: ~/.ssh with mode 700, the file with mode 600.
  2. Back in the row, compare its fingerprint with the machine’s own: ssh-keygen -lf /etc/ssh/ssh_host_ed25519_key.pub on the machine. If they match, press It matches: trust it.

Control logs in and reads the machine: its system, GPUs, Podman and whether it reaches control. Under Missing for a rig, it asks about each thing the machine still needs, such as May control install Podman?, with the exact script it would run. Press Install it for each, then refresh in the status bar along the bottom of the page until the row has nothing missing.

4. Add the rig

Section titled “4. Add the rig”
  1. Open Rigs in the sidebar, then press Add a rig.
  2. Under What kind of rig?, choose One machine, then Next.
  3. Under Where does it run?, choose machine-2 (over SSH), then Next.
  4. Under Which GPUs does it use?, leave its GPUs ticked, if it has any. With none, the rig measures on the CPU. Press Next.
  5. Under Name it and create it, give a Rig name, such as rig-2. Message bus shows the address the rig will reach, such as nats://control.example.com:4222.
  6. Press Create the rig.

You land on the rig’s page. Its Set up by control block ticks off each step as it comes: enrolled, files and services written and started, its key on the message bus registered, first report. The first download of the images takes most of the time. When the last step is ticked, press refresh: the page says rig-2 is live.

5. Start a run there

Section titled “5. Start a run there”
  1. Open Campaigns in the sidebar, then your campaign, and press Start a run at the top of its page.
  2. Under Where it runs, choose rig-2.
  3. Press Start the run.

The campaign’s page now has two runs at work, one on each machine. Each rig’s judge measures only its own run, and every result reaches the same control.

What you did

Section titled “What you did”

You gave control a way in from other machines, let it set a rig up on a second one over SSH, and ran your campaign on both at once.

Next

Section titled “Next”