feat(hybrid-gateway): integrate muse-cli with Cloudflare netns isolation, symmetric sidechat routing, and 646-pip sync unblock

2026-10-04 22:54:01 +00:00
parent 3d5fbe5aeb
commit 1a271b1bbd
60 changed files with 5068 additions and 55 deletions
@@ -0,0 +1,100 @@
+# Timer Stagger — Fleet Polling De-correlation
+
+**Date:** 2026-10-04 ~21:50 UTC
+**Why:** All 4 NetVM nodes egress via the same Warp IP (104.28.195.181). Synchronized
+timers = 4x correlated traffic bursts from one IP = correlated rate-limit risk.
+Staggering spreads load; combined with per-node rate limiting it de-correlates
+the fleet's egress signature.
+
+## Before
+
+Three heaviest muse.ai-egress timers fired at the **exact same second** every 5 min
+(`OnCalendar=*:0/5`): `job-heartbeat`, `self-main-loop`, `main-chat-watchdog`.
+`accounts-health` + `loop-remediator` coincided every 15 min. All four `wake-*`
+timers fired hourly at :00. Three of four `chromebox-watchdog` timers were
+synchronized to the second.
+
+## After — stagger schedule
+
+### bl user timers (`~/.config/systemd/user/`, hand-managed)
+
+| Timer | Old | New | Fires at |
+|---|---|---|---|
+| job-heartbeat | `*:0/5` | `*:0/5` (phase 0, unchanged) | :00 :05 :10 … |
+| self-main-loop | `*:0/5` | `*:1/5` | :01 :06 :11 … |
+| main-chat-watchdog | `*:0/5` | `*:2/5` | :02 :07 :12 … |
+| keepalive (box-managed) | `*:0/5` | `*:4/5` | :04 :09 :14 … |
+| accounts-health | `*:0/15` | `*:3/15` | :03 :18 :33 :48 |
+| loop-remediator | `*:0/15` | `*:8/15` | :08 :23 :38 :53 |
+| followup-sweeper | `*:*` (@:00s) | `*:*:20` | :20s of every minute |
+| agent-health | OnBootSec/OnUnitActiveSec 5m | + `RandomizedDelaySec=60s` | jittered ±60s |
+| fleet-alert-check | OnBootSec/OnUnitActiveSec 5m | + `RandomizedDelaySec=60s` | jittered ±60s |
+| exec-watch | OnUnitActiveSec 15m | + `RandomizedDelaySec=120s` | jittered ±120s |
+
+systemd natively supports offset calendar specs (`*:1/5`); no wrapper scripts needed.
+
+### Box-managed timers (`/srv/box/timers.json` → `timer-sync.py --all`)
+
+| Timer | Old | New |
+|---|---|---|
+| wake-muse | `*-*-* *:00:00` | `*-*-* *:05:00` |
+| wake-pip | `*-*-* *:00:00` | `*-*-* *:20:00` |
+| wake-646 | `*:0/30` | `*-*-* *:35:00` |
+| wake-opm | `*:0/30` | `*-*-* *:50:00` |
+| loop-health | `*:0/15` | `*:11/15` |
+| keepalive | `*:0/5` | `*:4/5` |
+
+(`wake-646`/`wake-opm` were actually on `*:0/30`, not hourly — now spread 15 min apart
+across the hour like the others.)
+
+### System timers (`/etc/systemd/system/`, hand-managed)
+
+| Timer | Change |
+|---|---|
+| chromebox-watchdog-{pip,646,muse,opm} | + `RandomizedDelaySec=30s` (local checks only; de-correlates restart bursts) |
+
+## Backups (pre-change)
+
+- bl user timers: `~/.config/systemd/user/bak-stagger-20261004/` (12 files)
+- system timers: `/root/bak-stagger-20261004/`
+- timers.json: `/srv/box/timers.json.bak-stagger-20261004`
+
+## Verification (2026-10-04 ~21:52 UTC)
+
+`systemctl --user list-timers` and `systemctl list-timers` confirm the new phases:
+followup-sweeper @ :50:20, main-chat-watchdog @ :52:00, loop-remediator @ :53:00,
+accounts-health @ 22:03:00, keepalive @ :54:00, wake-muse @ 22:05, wake-pip @ 22:20,
+wake-646 @ 22:35, wake-opm @ 22:50. Watchdogs spread over ~7s via jitter.
+
+Note: `self-main-loop` did one immediate catch-up run at 21:49:21 on reload
+(`Persistent=true` treating the 21:46 slot as missed under the new phase) — benign,
+one extra run.
+
+## Rollback
+
+```bash
+# bl user timers
+cp ~/.config/systemd/user/bak-stagger-20261004/*.timer ~/.config/systemd/user/
+systemctl --user daemon-reload
+# system timers
+sudo cp /root/bak-stagger-20261004/*.timer /etc/systemd/system/
+sudo systemctl daemon-reload
+# box timers
+cp /srv/box/timers.json.bak-stagger-20261004 /srv/box/timers.json
+/srv/box/bin/timer-sync.py --all
+```
+
+## Not changed (deliberately)
+
+- `box-http-health` (`*:0/15`) / `box-service-health` (`*:7/15`) — already staggered.
+- `siphon`, `cdp-relay-watchdog`, `shadow`, `backup-sync` — unrelated cadences.
+- `chat-state`, `response-harvester` — inactive.
+- `self-main-loop` still touches all 4 browsers inside one run — staggering the
+  *timer* doesn't de-correlate *within* the run. Follow-up: add inter-node sleeps
+  inside `self_main_loop.py` itself.
+
+## Status
+
+Live on bl + VM as of 2026-10-04 ~21:52 UTC. Doc uncommitted (review-then-commit);
+timer unit files live outside the NetVM repo (`~/.config/systemd/user`,
+`/etc/systemd/system`) — consider versioning them if drift recurs.