Commit Graph

3 Commits

Author SHA1 Message Date
Muse Sidechat c9143a558b fix: truthful fleet status in blind shells + agent-health circuit breaker
box fleet status / approvals check misreported every node as STOPPED /
CDP-unreachable from sandboxed shells (own PID+net namespaces: pgrep
blind, no route to 10.201.x.x, no sudo). Fleet was healthy throughout.

- bin/host_evidence.py (new): host watchdog evidence fallback. Recent
  timer runs (journal -o json, exact UNIT match) with no newer failure
  line in cdp-relay-watchdog.log / chromebox-watchdog.log (both
  silent-when-healthy) prove a node is up. def/dev have no watchdog
  coverage: browser verdict via chromebox-<node>.log freshness
  (alive-only), CDP verdict unknown.
- super-cli.py: effective status/source/evidence per node. Host
  evidence decides ONLY the fully-blind pattern (both local probes
  negative); live local signals always win. New UNKNOWN badge, [*]
  footnote; approvals UNREACHABLE splits into BLIND / OFFLINE(host
  agrees) / unreachable-evidence-inconclusive, with honest footer.
  proc_alive/cdp_ok keep local-probe meaning; status/source/evidence
  are new JSON fields.
- approvals.py: host_cdp_ok flag on the unreachable path.
- agent-health.sh: restart circuit breaker. 3 consecutive futile
  restarts (restart leaves agent still failing) opens the circuit:
  no more kills for 1800s, ALERT to log+journal, half-open probe
  after cooldown, reset on any success. Stops the def murder loop
  (57 restarts / 155 API FAILs for an account-layer failure).
- tests/test_fleet_status.py (25), tests/test_agent_health.py (6).
- CHROMEBOX-RUNBOOK.md: blind-shell status + futile-restart sections.

Tests: 98/98 focused green (agent_health + fleet_status +
completion + tool_calls). Live-verified: 4 ACTIVE [*] + 2 UNKNOWN.
2026-10-06 18:16:14 +00:00
operator 94d6502289 docs: sync onboarding runbooks, node inventory, bridge mappings, and box api docs 2026-10-05 15:58:37 +00:00
operator-main eba6c6965f Add CHROMEBOX-RUNBOOK.md: operator guide for the fleet browser stack
Covers architecture, watchdogs, queue, monitors, common failures, and 2am triage.

Session: sidechat/chromebox-ops
2026-10-04 13:27:29 +00:00