Thermite vs pi-autoresearch

pi-autoresearch and Thermite run the same loop: an agent changes something, a run measures it, and the better version stays. The difference is who you have to trust. In pi-autoresearch the agent writes the benchmark, reports the number and decides what stays, and you review what comes out. In Thermite a judge that the agents can neither read nor change does all three.

Use pi-autoresearch when:

the run is short, and you will review the result yourself;
you want to start in a minute, in your own terminal, with no containers;
you have no benchmark yet and want the agent to write one.

Use Thermite when:

you want to leave research running for hours or days, and trust what you find when you come back;
the number is noisy, or other numbers must not get worse;
the agents must not be able to see or change what scores them;
you run several agents or several projects, on more than one machine;
the work is a list of goals or a question, not only a number.

Side by side

	pi-autoresearch	Thermite
What it is	An extension for the Pi coding agent	A system you host: rigs that do the work, and control, one dashboard with a REST API and MCP
Setup	`pi install npm:pi-autoresearch`	Docker or Podman, and about twenty minutes
Where it runs	In your terminal, in your checkout	In containers: on your computer, on machines over SSH, on Kubernetes or on RunPod
Agent programs	Pi	Pi, OpenCode, Codex or Claude Code
Kinds of work	Improving a metric	Hill climb, goal or research
The benchmark	Written by the agent, which may change it during the loop	A kit the agents never see
The number in the record	Typed by the agent	Measured by the judge
Keep or discard	Chosen by the agent; a failed check blocks `keep`	Decided by the judge, by the rules in the contract
Noise	An advisory confidence score	Paired, randomized blocks, and a win confirmed on fresh runs
Other metrics	Shown for monitoring	Guards with limits per step and in total
Hidden test cases	None	Run only at confirmation, so no agent can tune to them
A change that loses	Code reverted; the agent’s notes stay in the log	Kept as a git ref, with its verdict and reasons
What ends the work	`maxIterations` if you set it; built-in stops at 200 automatic turns and after more than 20 failures in a row	Only a limit you set
Parallel work	One agent in one session	Many isolated lanes, one planner and one queue
Agent memory	One long session, plus `.auto/prompt.md` and the log	A fresh call each time, from a brief the rig builds out of its ledger
Oversight	Scripts you write, as hooks	Built-in advisors: teachers, an auditor and a review panel
Where you watch	A table in Pi, and a live page on localhost	One dashboard for every campaign on every rig
The result	Commits on a branch, which `/autoresearch finalize` splits into reviewable branches	The champion as a patch, and on request a commit for each champion in your repository
Licence	MIT	Apache-2.0

Who measures and who decides

In pi-autoresearch, the agent writes .auto/measure.sh, and its setup skill says it can update it during the loop. run_experiment prints the values it parsed, but the metric and the status in the log are what the agent passes to log_experiment. The one hard rule is that a failed .auto/checks.sh blocks keep.

That works when you are the judge, and pi-autoresearch’s authors say so plainly. In Shopify Engineering’s post, David Cortés writes that the agent “does ugly hacks sometimes, like removing a lot of files”, and that “I throw away all the hacks and keep the good stuff”. The post quotes Tobi Lütke calling his own result on the Liquid template engine “probably somewhat overfit”. The review is the person’s job.

Thermite does that review in code, so it happens for every change, with nobody watching. A kit holds how to build, run, time and check the project, and agents never see it. An agent submits a change and a claim. The judge builds the change, measures it and writes the verdict, and no agent can set one. Hidden test cases run only at confirmation. Why agents never judge gives each step.

The cost: someone has to write the kit before the first change. The New campaign page can write a simple one from your answers, but a judge is only as good as its check, and that part is your work.

What happens to a change that loses

pi-autoresearch keeps the branch clean. After discard, crash or checks_failed it runs git checkout and git clean, and the code is gone. Its skill warns the agent that the only record that survives is the description and the notes the agent writes. The branch stays easy to read: it holds only what was kept.

Thermite saves every change as a git ref before it is built, and the ledger keeps its verdict and the judge’s reasons. The one word discard becomes a set of verdicts that say how a change lost: neutral, regress, tradeoff, not_confirmed and others. Work that moves no number can still be kept as field notes and sources, for later runs to read.

Noise

pi-autoresearch shows a confidence score: the best kept gain divided by the median absolute deviation of the segment’s runs. It is advisory, and the decision stays with the agent. Those runs are different versions of the code, so the score reflects how far apart the ideas were as well as how noisy the machine is. An open issue argues that it reads “likely real” on pure noise in most sessions, and another proposes repeated runs.

Thermite times the change and the champion together, in blocks that alternate in a random order on the same device. The smallest gain that counts is set in the contract before the run starts. A change that looks better is measured again in fresh blocks, and decided on that new data alone.

The cost: time on the device. Every change is timed many times beside the champion, so a judged attempt takes far longer than one benchmark run.

Metrics that must not get worse

pi-autoresearch has one primary metric. The others are logged and shown, and its guidance to the agent is that they “almost never affect keep/discard”. A rule such as “raise prefill, never drop decode” has to live in the prompt, and the agent has to keep to it.

A Thermite contract gives each guard two limits: how much worse a metric may get in one step, and how much worse in total against the starting point, so small losses can’t add up. Hard constraints hold a metric to a fixed value. A change past a guard doesn’t become the champion, unless the guard is set to ask you and you say yes.

Thermite still climbs one number at a time. Further priority metrics break ties; they are not a second climb.

What ends the work

pi-autoresearch’s README says the loop “never stops unless interrupted”. Under that sit two guards: it stops resuming itself after 200 automatic turns, and after more than 20 discards or crashes in a row. Its documentation says these are not tuning knobs, and two pull requests to make the limit configurable were closed unmerged. You can add maxIterations as well. For a loop on your own bill, in a session you can see, these are sensible rails. On a search with long dry spells, they end the work at the first long one.

Thermite has no default limit on turns, attempts, failures or time. A hill climb runs until you pause it. Goal and research campaigns end when code says they are done. A lane with nothing to show waits longer between trials, and the search goes on.

The cost: nothing stops a paid model for you. Before you use a paid key, set a budget or a stop time.

One agent or many

pi-autoresearch runs one agent in one Pi session, and its README suggests a dedicated branch or worktree. Staying simple is a choice: a proposal for a swarm mode was closed by its maintainer with “I don’t want to overcomplicate it”. After a compaction, the agent re-reads .auto/prompt.md and the log and goes on.

Thermite splits a run into lanes, each with its own workspace and its own Unix user. A planner gives each lane its next experiment from one queue, never the same one to two lanes. Every agent call starts fresh from a brief the rig builds out of its ledger, so a crashed session loses nothing.

The cost: each trial reads the code again, which takes time and tokens that one long session would not spend.

Watching and steering

pi-autoresearch shows a results table above Pi’s editor, a fullscreen view, and a live page that the same process serves on localhost. You steer by talking to the agent. Hooks run your own scripts around each iteration, and what they print reaches the agent.

Thermite’s rigs report to control, and a rig goes on researching while control is down. Every instruction is a signed, recorded command, from the dashboard, the REST API, MCP or the command line. Advisors are built in: teachers bring ideas, an auditor looks hard at surprising results, and a review panel decides what code can’t. None of them can set a verdict.

What pi-autoresearch does better

Getting started. One install, then /autoresearch optimize unit test runtime, monitor correctness. The agent asks a few questions, writes the benchmark and starts. Thermite takes about twenty minutes to install, and each project then needs a kit.
Fewer moving parts. One extension, one .auto/ folder and one log you can read. Thermite is containers, a message bus and a control service.
Reviewable branches. /autoresearch finalize groups kept work into independent branches, one for each logical change. Thermite gives you the champion as one patch, or a commit for each champion.
Track record. It has been public since March 2026 and had 8,221 GitHub stars on 2026-11-07, and Shopify reports using it on more than 40 metrics. Thermite is at 0.x and makes no upgrade promise, and the test that proves runs of several days has not run yet.

Thermite owes it more than a comparison. Pi is Thermite’s default worker harness, and a hill climb’s first step, a baseline of the unmodified code before any change, comes from pi-autoresearch and Karpathy’s autoresearch. Thermite doesn’t run the extension itself.

Moving a project over

If you know pi-autoresearch, most of its parts have a home in Thermite:

pi-autoresearch	Thermite
`.auto/prompt.md`	The campaign’s brief, and its contract for what must hold
`.auto/measure.sh`	The kit’s build and run steps
`.auto/checks.sh`	The kit’s check
`.auto/ideas.md`	The run’s queue of experiments
`.auto/log.jsonl`	The rig’s ledger
A segment	A run
`keep`	A confirmed win, which becomes the champion
`discard`	A verdict that says how the change lost
Notes in `asi`	Field notes
`/autoresearch finalize`	Take the result

Your own project walks through a first campaign from the dashboard, and Write a kit for your project covers a real benchmark.