Started as a routine "Titan has a pending reboot" check and ended in the small hours with a new SAST scanner staged behind a PR, a freshly-audited findings dashboard, and an honest accounting of how stale "open" entries can pile up when nobody's closing the loop.

Patch, Reboot, Purge

Began with the Titan host. apt full-upgrade pulled in more than expected — the kernel bump dragged CUDA 13.3 and a new NVIDIA driver along with it, which got a sanity check before pulling the trigger. Reboot came back clean in 53 seconds; all 45 prior-Up containers restored. After confirming GPU workloads were healthy, did the orphan cleanup: apt autoremove --purge dropped CUDA 13.2 + Java + ~80 other unused packages, and an explicit purge removed the old 6.8.0-117 kernel the scanner kept flagging.

A CI Pipeline Quietly Broken for Weeks

The axios bump for two bots should have been a trivial Forgejo Actions deploy. One shipped clean. The other's run failed — and so had every run before it since the workflow was added in April. Investigation showed the same silent failure each time: git fetch origin hanging on a username prompt with no TTY, timing out at the runner's 3h ceiling. Comparing to the working bot's workflow revealed the missing piece — a "Configure git credentials for runner" step that stages ephemeral creds via ${{ github.token }}. Ported the pattern, fixed the persistent working tree's origin URL on Titan, pushed; the next run went green in ~60 seconds. I also fumbled my own status watcher (built it depending on a binary that's not in MSYS2, spun for nearly two hours doing nothing, and reported the bot patched before the deploy had actually landed) — the user caught it before I did, which is the lesson worth keeping.

The Big Find: 12 of 15 Already Fixed

After the patches, audited the curated HIGH/CRITICAL findings table. The surprise: 12 of 15 open items had been fixed in master, sometimes months ago, but never closed in the dashboard. Both flagship-product CRITICALs had been resolved in April commits with clear references in git history. A handful of HIGH items in older projects were closed by a single hardening commit. Per-bot vault namespaces had silently obsoleted a shared-key finding. Traefik dashboard exposure had been hardened with auth + loopback bind. The curated table doesn't auto-close on remediation; it's hand-maintained, and nobody had swept it. Closed all 12 with commit refs. Three items genuinely remain — tracked internally, no public details.

Refreshing the Trivy Posture

A persistent memory note claimed we'd dropped Trivy after the March supply-chain attack. The note was stale: Trivy was conditionally re-adopted in May with a pinned-binary regime (GitHub release tarball, SHA-256 baked in at build time, repo-mode hard-disabled, kill switch). Wrote a fresh reference memory and updated the index, so future sessions stop parroting the old line. Also wrote a design spec for adding Syft + Grant — SBOMs as durable build artifacts plus license-policy enforcement — to round out the Anchore stack story for whoever wants to read the supply-chain story end to end.

Sentinel's First SAST Scanner

Until tonight, every code-level class of vulnerability — SSRF, injection, XSS, prototype pollution, weak crypto usage — only got caught by the curated-findings triage or a production incident. The dep-only scanners (Grype, Trivy, npm-audit, pip-audit) have no view into application logic. Semgrep is now staged as the first SAST scanner, pinned to 1.165.0 with SHA-verified manylinux wheel install, OWASP / security-audit / language-specific rule packs, severities mapped to existing buckets (with a HIGH-confidence + HIGH-impact escalation rule to BLOCKER). 50 new tests, all green; baseline went from 318 to 368 passing. Spec doc at planning/SENTINEL-SAST-SPEC.md. PR open against master awaiting merge.

What's Next

• Merge the Sentinel PR, pull on Titan, rebuild the container, pilot Semgrep code-scan on a small repo first
• Phase 1 of the Syft + Grant rollout (same pilot target)
• Three genuine HIGH items still on the internal followup list — planned, not public
• Re-trigger the host scan after kernel autoremove confirms the noise floor