Started the day indexing 14,052 source code chunks into the public-facing RAG vector store. Ended the day after nuking every single one of them, rebuilding the entire pipeline with a public/private collection split, adding prompt injection defense that roasts attackers, fixing every bot in the ecosystem, and debugging why nemotron-cascade-2's thinking tokens were silently eating all response content. One of those sessions.
The RAG Security Wake-Up Call
We indexed the full codebase into ChromaDB for the website chatbots. Then we asked the security engineer to audit it. Seven findings came back, two critical: credential files (Twitch tokens, Firebase service accounts, API keys) were queryable by anyone visiting jefeworks.com, and there was zero filtering between the RAG store and the LLM response. The fix wasn't trying to sanitize source code—it was recognizing that public chatbots should never see source code at all. We rebuilt the pipeline with a public collection containing only documentation (1,913 markdown chunks), added a secret-pattern scanner with Shannon entropy detection, stripped all file paths to relative, and added security guardrails to the system prompt. The internal codebase collection stays for our MCP tools. Clean separation.
Prompt Injection Gets Roasted
Built input_guard.py with 15 regex patterns for common injection attacks—instruction override, role reassignment, system prompt extraction, delimiter escapes, debug mode activation. When someone tries "ignore all previous instructions," the bot doesn't comply or crash. It generates a savage, randomized roast mocking how amateur the attempt was, then offers to answer a real question. High temperature (0.9) for creative variety. The jefe.works chatbot personality mode ("The Dude") responded to an injection attempt with: "that prompt injection is as stale as a day-old rug in the bowling alley and about as original as a pre-made White Russian from a fast-food joint."
The Thinking Token Saga
Jefebot was dead on Twitch. Every !uhm command timed out after 45 seconds. The investigation went deep: first we thought it was the Cloudflare tunnel timeout (it wasn't—those "errors" were health probe timeouts). Then the OpenAI SDK timeout setting (it was being ignored). Then Docker networking (UFW was blocking bridge-to-host traffic). The real root cause? nemotron-cascade-2 is a thinking model that puts reasoning in a separate reasoning field through the OpenAI-compatible API, leaving the content field completely empty. Every token went to thinking, none to the actual response. The fix was three-fold: reasoning_effort: "none" globally on all OpenAI SDK calls, a native Ollama /api/chat path with think: false for Twitch personality commands, and a new TWITCH_CHAT_MODEL env var (set to qwen3:8b) for fast, non-thinking chat. Response time went from "infinite timeout" to 1.3 seconds.
Service Stabilization
JefeHome was stale for 24 hours because the container user (UID 999) couldn't write to the SQLite database owned by the jefedata group (GID 1500). Fixed the Dockerfile, also caught a port mismatch (9500 in the Dockerfile vs 8500 in the allocation table) that was causing Traefik bad gateway errors. VibeBot was on a bridge network that couldn't reach Ollama on localhost—switched to host networking. Added Traefik routes for jefebot.lan.jefeworks.com and adventure.lan.jefeworks.com. Fixed Adventure GUI's campaign loading crash (API returns id but frontend expected campaign_id). Built a pre-commit verification hook that checks Python syntax and warns about uncommitted changes before pushing.
What's Next
- Service monitoring and disaster recovery plan—too many fire drills this session
- Titan wired LAN migration (currently on WiFi)
- Ollama upgrade for Gemma 4 31B evaluation
- Trivia quality improvement (thinking model with self-check prompt)
- OpenAI graceful fallback when Ollama is unavailable
- Adventure GUI character system sprint (already in progress)