Part 3 of 3 in our series on building the best home energy planner in the world. Part 1 tells how we built it, and part 2 explains the problem and how we measure it.
A planner controls real hardware in real homes, so "it seemed faster on my laptop" is not a release process. Here is how a change gets into Energyplan, and how a team of AI agents does much of the work under rules strict enough that we trust the result.
Rule 1: every change needs a measurement
We measure every change against its parent, the code just before it, and every release against the version you run today. The rules:
- Faster must show up as faster on fixed benchmark sets, beyond the measurement noise.
- Better plans must show up as lower cost at equal goals.
- A clean-up that should change nothing must give byte-identical answers on every test request.
- No gain, no ship. In November we dropped a feature that saved 16.7 öre across 564 requests and cost 6 % more time.
Rule 2: know your noise
Timing on a computer is noisy. Run the same program twice and you get two different times. So before we judge anything, we run the shipped version against itself, many times, at several load levels. That tells us how big a difference pure noise can make, case by case.
A new version has to beat that noise floor. We learned this from our own mistakes:
- Small test groups are judged case by case. In a group of five cases, one jumpy case could otherwise hide a 12 % slowdown in another.
- More data must never make a floor tighter. When adding a second measurement run made a limit stricter, we changed the rule.
- Count the work, not just the time. When a timing looks worse, we count the processor instructions. That tells a real slowdown from a noisy moment.
Rule 3: a quiet machine, or no measurement
Our benchmark machine runs around the clock. Other heavy jobs, such as our own end-to-end site simulations, sometimes start on it. A guard watches for them. It waits until the machine has been quiet for ten minutes, and if anything heavy ran during a measurement, it throws that measurement away and takes it again.
Rule 4: real homes, not just test sets
Benchmarks are made up. Homes are not. So we also test every release on:
- 500 real replans recorded on a live home, replayed through old and new versions side by side;
- a Raspberry Pi 4, the same hardware many FTW homes use, with strict limits so the test never disturbs the home it runs next to;
- FTW's own validator, which replays every plan against the home's physics, exactly as it does in production.
Rule 5: write the targets down first
Before a speed release starts, we write its targets into the repository: how many more plans proven optimal, how much faster, how much less work. We measure each one against the shipped version, and we write them before anyone looks at a result. Nobody can change a target after seeing the outcome.
Rule 6: beta, watch, then stable
A new Energyplan first ships in FTW's beta channel. For at least a week we watch the abort signals:
- plans rejected by the validator;
- fallbacks to Core's own planner;
- time-budget overruns;
- crashes.
We compare them with the weeks before. We prepare and rehearse a rollback to the previous version before the release, not after.
Around the planner
A plan only matters if the whole site does the right thing with it. So the planner has a test world around it:
- A simulator of a whole site. It runs FTW against simulated meters, batteries, inverters and chargers, with a thermal model of the main fuse. A full run covers 30 scenarios.
- A field matrix. It runs 314 cases through the control loop and lists every known failure until a fix turns it green.
- A catalogue of what has gone wrong. By late October it held 840 known faults, device behaviours and hardware tests. We are turning each one into a scenario.
None of these writes a plan. They are the reason we can believe one.
The team: humans and a lot of agents
AI agents do much of this work, organised much like a disciplined engineering team.
- An orchestrator holds the plan, makes the calls a human has delegated, and asks the human for everything else. Releases, public pull requests and tags always need a human yes.
- Track agents each own one line of work in their own isolated copy of the code. They write the test first, then the change, then the measurements.
- Reviewer agents attack every change from two angles: "is it correct and robust?" and "does it do what the plan says, and is it really faster?" Serious findings go back for a fix round, and nothing merges until the reviewers are satisfied.
- Independent reviewers check the measuring tools themselves. In November one found a gate check that could have let an outdated exception slip through. We fixed it, test first, before it mattered.
We use models from more than one vendor. Claude Opus 5.5 did most of the release work, with Claude Sonnet 5.5 on review, test and measurement. GPT-6 Astra, GPT-6 Luna and GPT-6.1 Sol through Codex did most of the research. Claude Fable 5 and 5.1 are part of the work too. Before them we used the best models of the day, right back to the start.
Part 1 has the full count: about 50 billion tokens and about 235,000 tool actions in the logs we still hold. The last release alone, from 27 October to 8 November on our main build machine:
| Tokens processed | about 6.4 billion |
| Tokens written (code, tests, reviews, reports) | about 29 million |
| Agent runs | 396, in 8 multi-agent waves |
| Model calls / tool actions | about 31,900 / 33,400 |
| Benchmark steps | 830+ |
Agents make mistakes, just as people do. One agent once held the shared benchmark queue for hours while it waited for its own next step, and every other measurement stalled behind it. Now there is a rule: never wait while holding a lock. We do not expect perfection. We expect mistakes to get caught, written down and turned into rules.
Why this is hard to copy
The tokens are the cheap part. Anyone can run agents. It has taken us years to build what they run against: the research behind the models, the forecasts, the benchmark sets, the noise floors, the replays of real homes, the Raspberry Pi rig, the site simulator, the validator, the rehearsed rollbacks, and the platform and gateway firmware that reach real batteries. Without those, an agent team produces code that nobody can trust. With them, it produces releases that real homes can run.
That is also why Energyplan is proprietary. FTW's code is open, and we want it to be. The planner, and what we have learned building it, is ours to license.
Why it matters to you
You will never see any of this. You will see a battery that is full when prices peak, a car that is ready at seven and a plan you can check.
That is how we intend to build the best home energy planner in the world: one measured step at a time.
Energyplan is Sourceful's proprietary planner. It ships with FTW, whose Core, clients and drivers are open code under AGPLv3. Energyplan itself has its own licence: free for your own private household with FTW, and under a written Sourceful licence for commercial use or for bundling with other products. Talk to us if you want it in yours.