Our agents drank all the coffee

There’s an xkcd about the best excuse a programmer ever had, and for most of my career it was a documentary. You pushed a commit and walked to the kitchen. You watched the coffee machine grind. You drifted outside to where the smokers stood, because that’s where the real decisions got made, and you came back. Still running. The build was the only break anyone was allowed to take.

You know the feeling. You have a fix. It’s small, it’s ready, all it needs is a green tick. CI says twenty minutes. So you do the sensible thing and start something else, and by the time the run finishes you’re deep in that. You look after lunch. It’s red: a flaky test, or a conflict with something that merged in the meantime. You fix it, push, wait again, and get back to it tomorrow. A one-hour change has taken two days, and almost none of that time went into the change. All you wanted was to pass the checks and move on.

This year the build got new company at the coffee machine: our coding agents. In September they were waiting eighteen minutes for every run. Here’s how that became about three, for the same money, and what we found at the end of the queue once the machines stopped being the excuse: us.

Who waits for CI now

Most of our team now works with coding agents, Claude in particular. A typical developer here has four or more agent sessions going at once.

Each of those sessions works in a loop. It writes the change, pushes it, waits for CI (continuous integration: the robot that builds and tests every change), reads what failed, fixes it and pushes again. Only when the checks are green does a person review it, with CodeRabbit as an extra pair of eyes, and a person merges it and decides when it reaches production.

That turns CI from something a person checks once or twice an hour into something every agent on the team waits for, all day. On GitHub Actions we went from 48 runs per working day in June to 375 in September. Merged pull requests went from around 140 a month in the spring to 583 in September. GitHub bills by the minute, and the bill went from about $29 in June to $135 for September, with a few days still to go.

Then our main web application got slow, because we had written a lot of tests. In one week in September, while we built a new admin API, the suite grew from about 3,900 tests to about 9,600, most of them tests that boot the whole application and talk to a real database. A typical pipeline run went from under six minutes to eighteen, and a bad one took twenty-seven. Jobs weren’t stuck in a queue. The machine had two CPUs, two and a half times as much work, and we had just told it to run eight test processes at once.

A pull request here typically gets two CI runs before it’s done, and a quarter get four or more, mostly follow-up pushes rather than failures. At eighteen minutes a run, that’s over half an hour per change before a person even looks at it, and with four or more sessions per developer, at least four changes are always at the coffee machine.

Not a VPS

A colleague was the first to say it out loud: this is slowing all of us down. Their suggestion was a VPS, a rented virtual server. Rent a big one, install GitHub’s runner software (the agent that picks up CI jobs) on it, done. It would have worked.

I still didn’t love it. A self-hosted runner is a pet. Somebody patches it, keeps Docker healthy on it, installs the right languages and versions for every repository that uses it, empties the disk when a build fills it, and notices when it’s down on a Saturday. I’d rather nobody on a team our size spent their week on that. So I looked for something hosted instead, but first I fixed the tests.

Fixing the tests, then the machines

The first fix was plain engineering, and it did most of the work. The tests already ran in parallel, but with a lot of waste around them: every test worker built its own copy of the database, the database carefully saved every write to disk as if it mattered, and the same code was compiled again in every process. So the test database now lives in memory, it’s built once and copied to each worker, and the suite is split into four jobs that run at the same time. The split is deliberately dumb: count the tests in each file, weigh the slow kinds heavier, and hand the files out heaviest first to whichever job has the least so far. The four jobs don’t finish at quite the same time, and timing-based weights are the obvious next step.

Database setup went from 76 seconds to under 30, the tests needed half the compute, and on GitHub’s standard machines the pipeline was down to about five minutes. But each of the four jobs now pays for its own setup, so billed minutes per run stayed at about 21, what the slow pipeline had cost. We waited about a quarter as long for the same bill.

The day the test fixes landed, I signed up for a trial at Blacksmith, the hosted option with the easiest setup. Blacksmith sells what GitHub sells, a machine that runs your GitHub Actions jobs, except you can pick bigger and faster ones. You don’t install anything. You change one line per job:

runs-on: blacksmith-4vcpu-ubuntu-2404   # was: ubuntu-latest

Right after signing up I asked Blacksmith to run our jobs in their EU region. CI should only ever see test data, never a club’s or a golfer’s. In case someone slips up, the machines sit in the EU anyway. They switched it within a day. There’s no deal behind this post: we did the trial and now pay the normal price. If we ever want to leave, it’s the same one line back.

With the four test jobs on Blacksmith, the pipeline has run 62 times since. The median is 2.8 minutes, and nine runs in ten finish in under 3.7.

Tuning and splitting took a typical run from eighteen minutes to about five. The machines took it from five to under three, and part of that is simply more cores, because two CPUs became four. I didn’t test GitHub’s own four-core runner, which exists and costs $0.012 a minute, twice its two-CPU rate, so some of this gain is money any GitHub customer could spend.

What changed, and what didn’t

I expected the whole loop to collapse. So I measured it, comparing pull requests from the slow week with the five days since the switch.

The part CI controls did collapse. From opening a pull request to its first green run took a median of 17.5 minutes in the slow week. Now it’s 3.1. That’s the wait every change pays before review, and it’s five times shorter.

The part I thought would follow didn’t. When a run fails, the time until the next green one is a median of about half an hour, before and after (only a handful of cases each week, so read that loosely). Faster machines couldn’t touch it. Fewer than one pull request in ten goes red, and the few that do mostly sit until a person picks them up, sometimes not until the next morning. The agents rarely hand CI something broken. The slow part of our loop is now us.

What it costs, and two lessons

One pipeline run of the main web application used to bill about 21 minutes on GitHub’s two-CPU machine, roughly 13 cents. On Blacksmith the same run bills about 16 minutes, most of them on the bigger machines, and comes to 12 or 13 cents depending on how you round. The same money, then, for a run under three minutes instead of five, or eighteen before we touched the tests.

Blacksmith for September so far: $12.68 of usage, $12 of which the free allowance covers. We owe them 68 cents.

Splitting buys time, and you still pay for every minute. Every job pays for its own setup: checking out the code, installing dependencies, starting the database. GitHub bills each job by the minute, rounded up, so ten one-minute jobs cost ten minutes and feel like one. That’s worth it for a pull request somebody is waiting on. For a nightly job nobody watches, keep it in one piece.

Look at ARM early. At Blacksmith an ARM minute costs over a third less than a standard Intel or AMD (x64) one, and moving CI there doesn’t change what you ship. We’ve only tried it on one repository so far, and it won’t fit everywhere: our Android app has to stay on x64 because Google’s Android build tools don’t run on Linux ARM.

Moving the rest

We have nearly fifty active repositories, and I don’t want each migration to depend on someone understanding that particular pipeline. So I wrote the process down as a skill for Claude, a runbook it follows step by step when it moves a repository over. It measures before and after, over several runs rather than one lucky one. It checks that caches actually restore; one of ours never once had, and nobody had noticed. And it tries ARM where nothing rules it out.

Five repositories run on Blacksmith so far. One started there. Three, including the main web application, went over with Blacksmith’s own migration wizard plus some help from me, and one was moved end to end by the skill. The early numbers for the others look good, from slightly faster to roughly twice as fast, but they rest on a handful of runs each.

Leave the pipeline faster than you found it

Nobody throws a party when CI works. When it’s slow or flaky, everyone is annoyed as hell. It’s a bigger part of the job than we like to admit, and it gets slow without anyone deciding it should: a few tests at a time, until one week it takes eighteen minutes and everyone has quietly started working around it.

So I treat it the way I treat code, with the boy scout rule: leave the pipeline better than you found it. Whenever I’m working in a repository, I look at how long its CI takes and whether its caches work, and if something is obviously slow, I fix it while I’m there.

With an agent, that costs almost nothing. I point one at the repository and let it read the past builds, find the slow steps, try a change, measure it, keep what got faster, drop what got worse, and go round again. It runs in the background while I work on bugs and features, and it barely takes any of my attention. The skill writes that loop down. I’d like it to become everyone’s habit, not just mine.

Back to the coffee

The test suite is still growing, so one day the pipeline will be slow again. With any luck, whoever notices will fix it before the coffee is ready. Somewhere, I assume, the smokers are still making the real decisions. I just can’t blame the build for joining them anymore.