The VM That Never Sleeps

Why my agents get a machine of their own, and why it isn't allowed to do anything else

The first decision was the easy one. The agent doesn't live on my laptop.

A laptop closes. It sleeps, it updates, it drops off the VPN, it goes home with me. Every one of those ends a session in the middle of something, and the sessions I care about are long: a migration, a review loop, an hour of an agent reading logs I don't want to read.

So the sessions run on a Linux VM on our cluster. I connect from the Claude desktop app over SSH. The repos live there. The tools live there. When I close the laptop at the end of the day, the session doesn't notice.

That part worked exactly as advertised. It was the only part that did.

It used to do more

The first version of the VM was also where things ran. Test databases. Containers. Whatever the task needed.

That lasted until it ran out of memory. Long agent sessions aren't small, and they don't share RAM politely with a Postgres container somebody forgot about. The somebody was usually me, or an agent working for me, which is the same thing.

So the VM got a rule, written into the file every session reads at startup:

It is not a home for an app.

No Docker. No database server. No browsers. No test suites. It hosts agent sessions and reaches everything else from one place.

A rule in a text file is a suggestion, so there's a hook. Before the agent runs a shell command, the hook reads it. If it starts with docker, or tries to talk to a local Postgres that isn't there, the command is blocked and the agent is told where to go instead.

It only judges the first command on the line. An agent determined to get around it could. That's fine. It's a guard rail, not a wall. The point is that the mistake costs one blocked command instead of an afternoon.

There's also earlyoom, in case the rule loses an argument.

The other box

Everything the VM isn't allowed to do happens on a shared dev host. Containers, throwaway databases, the QA preview slot, a test stack per pull request.

Shared means my sessions, a coworker's sessions and the CI runners all hit the same Docker daemon. So the rules say: name what you start after your project, touch only what you started, and tear it down when you're done.

The agents followed the first two.

There was a sweeper. It cleaned up the QA slot after a few hours and the per-PR stacks after a day, matched by name. One day in September, the box had 64 containers running. Thirty-nine of them were between one and fourteen weeks old. Load average around 30, on six cores.

None of them matched the sweeper, because agents name things however they like. A test database named after the bug it was chasing. A four-letter abbreviation. A ticket number. Each one a reasonable name for something somebody needed briefly and never thought about again.

I removed the 39 by hand and kept the list. The sweeper has a third rule now: anything not created, started or stopped in five days goes, unless it's labeled to stay. Five days, so a stack left over a weekend survives. A container that has crash-looped ten times is aged from when it was created, because otherwise every crash resets its clock.

Last year I wrote a whole post about letting a jump box kill itself, because I couldn't be trusted to shut it down. The agents clean up exactly as well as I did. Same answer: don't ask, enforce.

The VM has the same problem in miniature. Every parallel session makes its own git worktree, and almost none of them clean up. A nightly job reaps the ones that are provably finished: clean, merged, no open PR. A hook at the end of each session deletes a local branch only when GitHub says its PR merged from that exact commit. Everything else stays, and the decision gets logged either way.

The machine I'm careful not to reboot

Here's the part nobody warns you about.

Once a machine runs your agents, it starts running everything around them too. About thirty scheduled jobs live on that VM now: the watchdogs, the review agent, the overnight triage.

When we joined our hypervisors into one cluster this summer, the whole operation was run from it. Rule one of that runbook: never stop this VM, because it's the thing running the runbook. The hypervisor it sits on joined the cluster with its disks in place and every guest still running, including the one doing the joining.

That worked. Then the VM nearly stopped being itself.

Our VMs are built from cloud images with cloud-init. Proxmox hands each guest an instance ID computed from its config. Change the config, and the next cold start brings a new ID. Cloud-init sees the new ID, decides this is a brand-new machine, and re-provisions it: new hostname, new SSH host keys, a full upgrade at boot.

It had already done that to this VM once, in June. New host keys. Every client that trusted the old ones stopped connecting.

This fall I went looking for the same trap on every other guest. Plenty of them had it. Most were already protected. This VM wasn't. It was one reboot away from a new hostname, new host keys and an unattended dist-upgrade, on the host that runs the scheduled Claude jobs.

It's protected now. Cloud-init has been told to stand down. Nothing was restarted to do it.

Where I Landed

• The VM runs. Sessions survive the laptop. That was the whole point, and it's the part I never think about anymore.
• Nothing runs on it but agents and the jobs that drive them, and a hook enforces that.
• The dev box cleans up after the agents, because the agents won't.
• Cloud-init on the VM is switched off, and so is every other guest that was exposed.
• The VM is still a pet. It was built by hand over months. The kit I wrote later builds a clean one, but deliberately leaves out the thirty jobs, because each of them has to run in exactly one place. If this VM dies, the agents' new home is a kit away. The jobs are not.

Step two of the kit I wrote for a coworker's VM says: name it in Proxmox before you do anything else, then restart it from the hypervisor, so the host keys change once, before anyone has trusted them.

That step exists because of June. Most of the kit is like that.

Next: every rule has a scar.