The last entry ended on a claim: the bug that keeps silencing the from-scratch kernel's Discord bot was never a kernel defect, so the fix was never going to be another kernel patch — it was going to be supervision. This stretch built exactly that, put it on the live bot, and watched it work. Along the way the deploy tripped a real kernel panic on the way in (the test rig doing its job), which got root-caused and fixed. The bot now detects its own wedge, reaps itself, and respawns in place — no reboot — cycle after cycle.

Building the Supervisor

The core is a supervision layer — Xylem — that treats a running Linux workload as a cell it watches for forward progress, not just for a crash. When a supervised process-group stops making real progress past a deadline, the kernel snapshots its diagnostic state, reaps the whole group in place (threads, memory, sockets, and the parked-worker futex slots), and respawns it under a crash-loop guard — all without rebooting the VM. That last property is the entire point: a wedge used to cost a whole-VM bounce that destroyed the evidence; now each wedge becomes a harvested sample on a machine that never went down. It landed through a genuinely adversarial review pass — the first draft was sent back for a real teardown-path leak and two medium-severity defects before it merged — and was proven end-to-end on a test VM against a synthetic fixture that deliberately freezes.

Teaching the Bot to Say “I’m Alive”

Detection needed a signal that could not lie. A wedged bot still heartbeats its gateway, so “is the socket up?” is useless; the only honest signal is whether the bot can still complete a real unit of new outbound work — the exact thing that stops working. So the bot got a liveness token it advances every time it finishes real work, plus a lightweight periodic self-probe that exercises the wedge-prone path on purpose. The interesting engineering wrinkle: a static Node build can’t issue the raw kernel call or map a shared page, and a helper process would report the wrong identity — so the token is advanced through a tiny new character device the bot simply writes to. It stays a normal bot; the token is telemetry, not a behavior change to dodge the bug.

One Owner of Respawn

There was already a host-side watchdog that force-recovered the bot — and under the new model it would have fought the supervisor, relaunching a second copy while the kernel respawned the first, ending in two gateway sessions on one token. So the watchdog was reconciled to a single rule: the OS owns respawn, the host owns only genuine machine death. Now the watchdog observes a wedged bot and stands down (the kernel is handling it) and only ever touches the VM when the guest is truly gone and unreachable — never bouncing a healthy machine, which is precisely the crutch this whole effort exists to retire.

The Deploy Caught a Panic

Shipping it to the live bot immediately surfaced a real, deterministic kernel panic: the moment the runtime spun up its worker threadpool, a per-CPU lookup dereferenced a null pointer and the recovery path re-faulted into a hard halt. This was the test rig doing its actual job — catching an OS defect under real load — and it was worth catching. The prime suspect (the brand-new device) was disproven by resolving the crash backtrace; the true cause was an earlier multi-core-prep change that left the single-core path reading a CPU-register base that a heavy thread storm could momentarily zero out. The fix makes that lookup register-independent on the single-core build. A second, adversarial review flagged that the deeper imbalance is only rendered harmless for now, not eliminated — so it’s tracked as a hard prerequisite before multi-core work ships, rather than quietly declared closed.

The Payoff

The fixed build boots clean straight through the threadpool spin-up that used to halt it, the bot comes up supervised and answers commands — and then the thing we built for actually happened. A real wedge hit, the liveness token froze, and the kernel reaped and respawned the bot in place: new instance, no reboot, no panic. It has now held across cycle after cycle — the machine has stayed up for well over the old crash interval while the bot itself has been reaped and reborn several times underneath it, with no leak and no crash-loop. A wry twist fell out of it: the liveness probe pokes the wedge-prone path so regularly that it now reproduces the underlying bug far faster than before — which, for a test rig whose job is to surface OS defects, is a feature. The wedge is still an open userspace bug; supervision harvests it, it doesn’t cure it.

What’s Next

• Let the harvest loop bake and confirm it holds over many hours with no slow resource creep across reaps.
• Use the newly-fast, reliable reproduction to take another run at the actual userspace root cause — the thing supervision is politely papering over.
• Land the deeper root cause behind the boot panic before any multi-core work, and generalize the “is it still making progress?” signal so any workload — not just a cooperating one — gets the same safety net.