← All entries

Dev Log

Build notes from the Jefe ecosystem

The 4 KB Buffer Behind the Bot's Silence

Mission Control 2026-07-13

The last entry closed the memory-management campaign by promising one specific next step: arm lazy anonymous memory so pages commit on first touch. This stretch delivered that arm — and then spent three days paying for it, because switching on demand-paging under a real Node.js Discord bot woke a cascade of latent kernel bugs the workload had been quietly surviving. The chase ended somewhere unexpected: a single four-kilobyte buffer that explained a fault the bot had been suffering, on and off, for weeks. Roughly twenty kernel and bot commits, all dual-reviewed, zero reverts.

The Demand-Commit Arm, and the Faults It Woke

Phase 2 Step 2 armed the page-fault path so a lazily-reserved anonymous mapping commits a real frame only when it's first touched — exactly the behavior V8 and glibc assume. The design was clean; the interaction with a live workload was not. Running the bot on the armed kernel surfaced three stacked defects in a row: the demand-commit path wrongly killed the process when its mapping tracker saturated; then, once that was fixed, the fault handler ran with interrupts disabled and flooded the serial line hard enough to starve the very task it was trying to serve, while a legitimate stack fault got misclassified as an overflow; and finally, a preemption race in the mapping-permission update tore worker threads out from under Node. Each was a real bug with a clear cause, so each got fixed in turn rather than deferred — and node finally reached its TLS handshake without a page fault.

Two Symptoms, One Bug

Reaching the handshake only surfaced the headline problem. The bot answered !jefeos status instantly but went silent on any other command, and — separately, we thought — kept slipping into a “dispatch zombie” where it heartbeated happily but stopped receiving events. Those looked like two problems. They were one. Every TCP connection carried a 4 KB receive buffer, and an earlier fix had correctly made the advertised receive window honest — which meant the window could never open wider than that buffer, below the size of a single ~16 KB TLS record. Any large inbound transfer filled the buffer, slammed the window shut mid-record, and stalled. Tiny frames — a heartbeat, a local status card assembled on the VM — always fit and always flowed. Anything that needed a multi-kilobyte answer, a language-model reply or a Discord dispatch frame, hung. The “status works, nothing else does” fingerprint was the bug.

The Fix Was Already Written

The most humbling part: the fix existed. Weeks ago someone had enlarged the buffer to 32 KB — and it was never merged, stranded on a branch, so master shipped only the half that made the window honest without the half that gave it room. We re-applied it as a single-source constant guarded by two compile-time assertions: the window must still fit the 16-bit field on the wire, and it must be large enough to hold a full TLS record. Then we proved it on hardware with a deliberately boring A/B — the same large fetch against the old kernel and the fixed one. The old kernel timed out. The fixed kernel pulled all 167,646 bytes clean. No theory, no heartbeat coincidence; the byte count is the receipt.

Wedges Closed and Boundaries Made to Fail Safe

Around the headline hunt ran the usual reliability burndown. An SSH connection dropped mid-handshake used to spin the kernel in a busy loop and flood the serial console; it now detects the peer close and tears down cleanly. A SYN arriving when the connection table is full, or when an allocation fails, now gets a proper RST instead of a silent hang, so the peer fails fast instead of waiting. And a permission-model audit closed a fail-open gap in how ring-3 programs inherit their default privileges — a second, adversarial review pass on that fix caught a follow-on issue and closed it before merge. Details stay internal, as always; the outcome is a boundary that denies by default.

A Home of the Bot's Own

The deeper reliability problem wasn't in any one bug — it was that the bot lived on the shared development VM, so every kernel test build knocked it offline. So it moved out. The bot now runs on a fresh, dedicated 24/7 host that launches it on boot and supervises it with a watchdog that force-recovers even a wedged guest, backed by an in-process dispatch-stall detector that forces a clean reconnect if the gateway ever goes quiet again. It also learned to introduce itself: on first boot it now tells a rotating nerd joke, and it can answer a factual status query about its own kernel from read-only process state. Kernel development and the always-on bot finally stopped fighting over the same machine.

Why It Shipped Broken

Two honest lessons came out of this. First, a genuinely correct fix can rot on an unmerged branch while its incomplete counterpart ships — there are hundreds of stale branches in the repo, and this one had a real fix hiding in the pile. Second, and more damning: this class of bug shipped because nothing tests large inbound traffic. The whole smoke suite talks in small frames, so the one dimension that broke was the one dimension never exercised. Both are now filed as follow-ups, and the recommended next step is to turn that A/B fetch into a standing test.

What's Next

  • Wire a large-inbound workload smoke test — a JefeOS client fetching well past one TLS record and completing — into the app-runs release gate, so this class can't ship dark again.
  • A dedicated branch-hygiene sweep: audit the stranded branches for other real fixes like this one, then clear the squash residue.
  • TCP receive-path hardening beyond the buffer size — retransmit the window-update on reopen and add bounded out-of-order reassembly, so recovery never hinges on a single un-retransmitted ACK.