Thermite for local AI enthusiasts
Local AI runs on the hardware people can get: used data-centre GPUs, consumer cards, older multi-GPU machines. Inference engines and quantizations are tuned for the newest hardware first, so older cards are often underused. Thermite’s agents tune them for your specific card, while a judge on the same card measures every change and verifies that the model’s output is unchanged.
- faster decode, Qwen3.8-27B on one Tesla P100
- 3.89×
- 7.74 → 30.06 tokens a second, on a card that sells used for about $100
- the speed of the best standard 3-bit file
- 1.14×
- on the same card
- changes judged in five days
- 131
- 28 confirmed wins, each with its outputs held
Use cases
Section titled “Use cases”Each of these is a campaign you can run on your own GPU, with the agents on a model server of your own.
- AI and machine learning: LLM inference optimization
- AI and machine learning: LLM application quality
- AI and machine learning: Model training efficiency
- AI and machine learning: Model and method landscape
- AI and machine learning: Fitting a model to your hardware
- Science: Mathematical discovery
- Robotics: Robot reward design
Why it fits
Section titled “Why it fits”- The judge measures on your hardware. A rig is the machine the work runs on. Its judge builds each change and times it against the best version so far, on the same GPU, in alternating blocks, and times the best version against itself every day to learn the card’s noise. One GPU is shared in turns; with several, you split them between the judge and the agents (Sharing a device, Give a rig several GPUs).
- Speed never wins on its own. Your kit, the trusted code that builds, runs and checks the engine, decides what correct means: KL divergence against a reference, bit-identical text from a second model, the whole model in memory. A change that breaks a guard is invalid however fast it is, and a win must hold up again on inputs the agents never saw (Why agents never judge).
- Big model files, fetched once and checked. A rig downloads a model by URL, a gated Hugging Face model included, and checks its SHA-256 before any run uses it (Bring a model’s weights to a run).
- The agents can run on a local model too. Point them at a model server of your own, such as Ollama, vLLM or LiteLLM, and they cost nothing per request. Or use a Claude or ChatGPT plan you already pay for (Choose a model).
- Everything stays on your machines. Thermite is self-hosted: control, its dashboard and the rigs run where you put them. For a card you don’t own, rent one on RunPod by the hour.
- What one campaign learns, the next one reads. Each campaign’s attempts, verdicts and field notes go into its collection of Thermite’s knowledge base, and a later campaign can be granted it (Use past research).
Get started
Section titled “Get started”- Install on one machine: about twenty minutes, and the judge’s smoke check shows you a dozen known changes, cheats among them, getting their expected verdicts.
- Add a rig on your GPU machine over SSH, and split its GPUs if it has several.
- Write a kit that builds your engine, runs the model and times it, then test it before any agent sees it.
- Write a contract: the number to push, the guards to hold, and the folders the agents may change. Keep the measuring tools out of them.
- Start the campaign and watch it on Live. When you are happy, take the result as a patch.
What to know first
Section titled “What to know first”Local AI runs on the hardware people can get: used data-centre GPUs, consumer cards, older multi-GPU machines. Inference engines and quantizations are tuned for the newest hardware first, so older cards are often underused. Thermite’s agents tune them for your specific card, while a judge on the same card measures every change and verifies that the model’s output is unchanged.