fix: truthful fleet status in blind shells + agent-health circuit breaker

box fleet status / approvals check misreported every node as STOPPED /
CDP-unreachable from sandboxed shells (own PID+net namespaces: pgrep
blind, no route to 10.201.x.x, no sudo). Fleet was healthy throughout.

- bin/host_evidence.py (new): host watchdog evidence fallback. Recent
  timer runs (journal -o json, exact UNIT match) with no newer failure
  line in cdp-relay-watchdog.log / chromebox-watchdog.log (both
  silent-when-healthy) prove a node is up. def/dev have no watchdog
  coverage: browser verdict via chromebox-<node>.log freshness
  (alive-only), CDP verdict unknown.
- super-cli.py: effective status/source/evidence per node. Host
  evidence decides ONLY the fully-blind pattern (both local probes
  negative); live local signals always win. New UNKNOWN badge, [*]
  footnote; approvals UNREACHABLE splits into BLIND / OFFLINE(host
  agrees) / unreachable-evidence-inconclusive, with honest footer.
  proc_alive/cdp_ok keep local-probe meaning; status/source/evidence
  are new JSON fields.
- approvals.py: host_cdp_ok flag on the unreachable path.
- agent-health.sh: restart circuit breaker. 3 consecutive futile
  restarts (restart leaves agent still failing) opens the circuit:
  no more kills for 1800s, ALERT to log+journal, half-open probe
  after cooldown, reset on any success. Stops the def murder loop
  (57 restarts / 155 API FAILs for an account-layer failure).
- tests/test_fleet_status.py (25), tests/test_agent_health.py (6).
- CHROMEBOX-RUNBOOK.md: blind-shell status + futile-restart sections.

Tests: 98/98 focused green (agent_health + fleet_status +
completion + tool_calls). Live-verified: 4 ACTIVE [*] + 2 UNKNOWN.
This commit is contained in:
Muse Sidechat
2026-10-06 18:16:14 +00:00
parent a9f014f9fa
commit c9143a558b
7 changed files with 901 additions and 9 deletions
+87
View File
@@ -0,0 +1,87 @@
"""Tests for the agent-health.sh restart circuit breaker.
Drives the real shell functions (sourced with AGENT_HEALTH_LIB_ONLY=1)
against a scratch STATE_DIR/LOG: futile-restart counting, circuit open,
half-open probe after cooldown, and reset on success.
"""
import subprocess
import tempfile
import unittest
from pathlib import Path
REPO_ROOT = Path(__file__).resolve().parent.parent
SCRIPT = REPO_ROOT / "bin" / "agent-health.sh"
def _run(state_dir, log, snippet):
prog = (
"source '%s'\n"
"STATE_DIR='%s'; LOG='%s'\n"
"%s\n" % (SCRIPT, state_dir, log, snippet)
)
env = {"AGENT_HEALTH_LIB_ONLY": "1", "PATH": "/usr/bin:/bin"}
return subprocess.run(["bash", "-c", prog], capture_output=True,
text=True, env=env, timeout=30)
class CircuitBreaker(unittest.TestCase):
def setUp(self):
self.tmp = tempfile.TemporaryDirectory()
self.state = str(Path(self.tmp.name) / "state")
Path(self.state).mkdir()
self.log = str(Path(self.tmp.name) / "log")
def tearDown(self):
self.tmp.cleanup()
def bash(self, snippet):
return _run(self.state, self.log, snippet)
def test_allows_when_closed(self):
r = self.bash("circuit_allows def")
self.assertEqual(r.returncode, 0, r.stderr)
def test_allows_below_threshold(self):
r = self.bash("circuit_note_restart def fail\n"
"circuit_note_restart def fail\n"
"circuit_allows def")
self.assertEqual(r.returncode, 0, r.stderr)
self.assertEqual((Path(self.state) / "futile-def").read_text().strip(), "2")
def test_opens_after_threshold_and_alerts(self):
r = self.bash("circuit_note_restart def fail\n"
"circuit_note_restart def fail\n"
"circuit_note_restart def fail")
self.assertEqual(r.returncode, 0, r.stderr)
self.assertTrue((Path(self.state) / "circuit-def").exists())
self.assertIn("ALERT", r.stdout)
self.assertIn("ALERT", Path(self.log).read_text())
r2 = self.bash("circuit_allows def")
self.assertNotEqual(r2.returncode, 0)
self.assertIn("CIRCUIT OPEN", Path(self.log).read_text())
def test_half_open_after_cooldown(self):
old = "echo $(( $(date +%%s) - 1900 )) > '%s/circuit-def'" % self.state
r = self.bash(old + "\ncircuit_allows def")
self.assertEqual(r.returncode, 0, r.stderr)
self.assertIn("half-open", Path(self.log).read_text())
def test_ok_resets(self):
r = self.bash("circuit_note_restart def fail\n"
"circuit_note_restart def fail\n"
"circuit_note_restart def fail\n"
"circuit_note_restart def ok")
self.assertEqual(r.returncode, 0, r.stderr)
self.assertFalse((Path(self.state) / "futile-def").exists())
self.assertFalse((Path(self.state) / "circuit-def").exists())
def test_per_agent_isolation(self):
r = self.bash("circuit_note_restart def fail\n"
"circuit_note_restart def fail\n"
"circuit_note_restart def fail\n"
"circuit_allows pip")
self.assertEqual(r.returncode, 0, r.stderr)
if __name__ == "__main__":
unittest.main()