Fix supervisor SIGKILL loop killing restarted browsers (opm/pip)

Root cause: agent-health.service runs Type=oneshot with the default
KillMode=control-group. restart_browser() spawned the replacement
chromium with nohup under the service, so systemd SIGKILLed it the
moment the service exited. Every 5-min tick: FAIL -> restart ->
RECOVERED -> SIGKILL at teardown. opm and pip were permanently dark
and dm.py read masked it (empty output, exit 0).

Fix: launch replacements via systemd-run --user --scope (backgrounded)
so the browser lives in a transient scope outside the service cgroup
and survives teardown. setsid does NOT escape either. Note:
systemd-run --scope waits for the scope even with --no-block
(verified 2026-10-03), hence the backgrounding. Same fix in
chromebox-watchdog.sh (timers currently off).

dm.py: dm_read() now uses run_full() and prints a WARNING to stderr
with rc + last error line instead of failing silently on empty reads.

Trailers: Session: sidechat/opm-blind-fix
This commit is contained in:
operator-main
2026-10-04 01:10:45 +00:00
parent 3a1f0107ba
commit f19aff9b79
3 changed files with 315 additions and 53 deletions
+12 -2
View File
@@ -39,9 +39,19 @@ restart_browser() {
# Kill existing
pkill -f "$agent.*$cdp_port" 2>/dev/null
sleep 3
# Restart via netvm-chrome.sh
# Restart via netvm-chrome.sh in its own systemd scope.
# This oneshot service runs with KillMode=control-group, so anything
# spawned directly under it (nohup AND setsid both stay in the cgroup)
# is SIGKILLed when the service exits — observed 2026-10-03: every
# restart "recovered" then died at service teardown, looping forever.
# A transient scope escapes the service cgroup and survives.
# NOTE: systemd-run --scope WAITS for the scope's processes (even with
# --no-block, verified 2026-10-03), so background it — the scope itself
# is an independent unit and outlives the wrapper.
cd "$NETVM_BIN"
nohup ./netvm-chrome.sh --headless --cdp-port "$cdp_port" "$agent" https://muse.ai > "/tmp/bl-$agent.log" 2>&1 &
systemd-run --user --scope --unit="netvm-chrome-$agent-$(date +%s)" \
./netvm-chrome.sh --headless --cdp-port "$cdp_port" "$agent" https://muse.ai \
> "/tmp/bl-$agent.log" 2>&1 &
sleep 15
echo "$(date -Iseconds) $agent: browser restarted" >> "$LOG"
}