operator
281 lines
14 KiB
Markdown
281 lines
14 KiB
Markdown
# Chromebox Runbook
|
||
|
||
> **Box is the main surface.** All operator work goes through Box (box.muse-dev.online). The web UI, `box` CLI, and agents share the same API endpoints. No UI-only powers.
|
||
|
||
Operator's guide to the Chromebox/NetVM browser fleet on `bl` (100.123.153.75).
|
||
Written 2026-10-04. If you're reading this at 2am, start at [Quick Triage](#quick-triage).
|
||
|
||
## Quick Triage
|
||
|
||
```bash
|
||
# 1. Are the browsers alive? (should return Browser JSON for all 4)
|
||
for t in 10.201.35.2:9410 10.201.87.2:9420 10.201.202.2:9430 10.201.157.2:9440; do
|
||
printf "%s: " $t; curl -s -m 5 -o /dev/null -w "%{http_code}\n" http://$t/json/version
|
||
done
|
||
|
||
# 2. Are the relay processes running? (expect 4 python3 netvm-cdp-relay.py)
|
||
pgrep -af netvm-cdp-relay | grep -v grep
|
||
|
||
# 3. What did the watchdogs do recently?
|
||
tail -20 /home/super/Projects/NetVM/chromebox-watchdog.log
|
||
tail -20 /home/super/Projects/NetVM/cdp-relay-watchdog.log
|
||
|
||
# 4. Are the host veth IPs assigned? (all 4 must show inet 10.201.x.1/30)
|
||
ip -o addr show | grep '10.201' | grep -v '10.201.202\|10.201.157' # muse+pip
|
||
ip -o addr show | grep -E '10.201.(35|87|157|202).1/30'
|
||
```
|
||
|
||
If step 1 shows 000 for a node but the browser is fine inside its netns,
|
||
it's almost certainly a **missing veth IP** or **dead relay** — see [Common Failures](#common-failures).
|
||
|
||
## Architecture
|
||
|
||
Four nodes, one per agent. 1:1 mapping: node == agent == profile == netns.
|
||
|
||
| Node | Agent | Netns | CDP Port | Veth IP (host) | Peer IP (netns) | Relay Target |
|
||
|------|-------|-------|----------|----------------|-----------------|--------------|
|
||
| muse | muse | warp-muse | 9410 | 10.201.35.1/30 | 10.201.35.2 | 10.201.35.2:9410 |
|
||
| pip | pip | warp-pip | 9420 | 10.201.87.1/30 | 10.201.87.2 | 10.201.87.2:9420 |
|
||
| 646 | operator-646 | warp-646 | 9430 | 10.201.202.1/30 | 10.201.202.2 | 10.201.202.2:9430 |
|
||
| opm | opm | warp-opm | 9440 | 10.201.157.1/30 | 10.201.157.2 | 10.201.157.2:9440 |
|
||
|
||
**Traffic path** (per node):
|
||
```
|
||
host → veth IP:port (e.g. 10.201.87.2:9420)
|
||
→ netvm-cdp-relay.py (runs INSIDE the netns, listens on veth/peer IP)
|
||
→ 127.0.0.1:port (Chromium's DevTools, bound to netns loopback only)
|
||
→ Chromium (headless, Warp egress via WireGuard in the same netns)
|
||
```
|
||
|
||
**Key facts that will save you hours:**
|
||
- The relay listens on the **veth/peer IP**, NOT on 127.0.0.1. Curling `127.0.0.1:9420`
|
||
on the host returns nothing — that's normal, not a failure.
|
||
- Veth IPs are hash-derived via `bin/netvm-names.sh` (`netvm_names <node>` gives
|
||
you VETH, GW, PEER_IP). Don't hardcode them in new scripts — source the file.
|
||
- CDP ports 9410–9440 are registry-pinned. The hash-derived `CDP_PORT` from
|
||
`netvm-names.sh` is WRONG unless `CDP_PORT_OVERRIDE` was set. Always use the
|
||
pinned ports above.
|
||
- Chromium binds DevTools to loopback only. The relay exists because iptables
|
||
REDIRECT to 127.0.0.1 was proven not to establish — the userspace relay is
|
||
the reliable path. Don't try to "simplify" it away.
|
||
|
||
## Watchdogs
|
||
|
||
### chromebox-watchdog (browser health)
|
||
|
||
- **Script:** `/home/super/Projects/NetVM/bin/chromebox-watchdog.sh`
|
||
- **Timers:** `chromebox-watchdog-<profile>.timer` (one per profile — every active registry node: muse, pip, 646, opm, def, dev)
|
||
- **Cadence:** every 2 minutes
|
||
- **Log:** `/home/super/Projects/NetVM/chromebox-watchdog.log` (10 MB rotation, 1 backup gen)
|
||
- **Per-profile Chromium output:** `/home/super/Projects/NetVM/chromebox-<profile>.log`
|
||
|
||
**Health check stages** (in order, `HEALTH_FAIL_REASON` tells you which broke):
|
||
1. Chromium process exists for the profile → else `"no chromium process for profile"`
|
||
2. CDP responds on the profile's port → else `"CDP unreachable on :<port>"`
|
||
3. Chat page title present in target list → else reason names the missing title
|
||
|
||
**Kill-loop guard** (added 2026-10-04): before killing, checks if a Chromium for
|
||
the profile launched <2 min ago. If so, skips the kill — it's probably still
|
||
starting. Prevents the watchdog from murdering a slow-but-working browser.
|
||
|
||
**Relaunch retry** (added 2026-10-04): after relaunch, tries the health check up
|
||
to 4 times, 15s apart (~60s window). Only declares FAILED if all 4 miss.
|
||
A single 25s check was too brittle for cold starts (fresh egress IP +
|
||
Cloudflare handshake can exceed 25s).
|
||
|
||
**What "relaunch FAILED — needs operator attention" means:** the watchdog tried
|
||
4 times and the browser still isn't healthy. Check `HEALTH_FAIL_REASON` in the
|
||
log, then look at the per-profile chromium log. Don't just re-run the watchdog —
|
||
find out why the browser won't come up.
|
||
|
||
### cdp-relay-watchdog (relay health)
|
||
|
||
- **Script:** `/home/super/Projects/NetVM/bin/cdp-relay-watchdog.sh`
|
||
- **Timer:** `cdp-relay-watchdog.timer`
|
||
- **Service:** `cdp-relay-watchdog.service` (Type=oneshot)
|
||
- **Cadence:** every 5 minutes
|
||
- **Log:** `/home/super/Projects/NetVM/cdp-relay-watchdog.log`
|
||
|
||
**Two-stage check** (per node):
|
||
1. **Veth IP present** (`veth_healthy`): verifies the host veth interface has its
|
||
expected IP. If missing → `FAIL_LOUD` in the log, skips relay restart
|
||
(pointless without the veth). Does NOT auto-recreate the veth — that touches
|
||
WireGuard/iptables, too invasive for a watchdog.
|
||
2. **Relay connectivity** (`relay_healthy`): curls `http://<peer-ip>:<port>/json/version`
|
||
and greps for `"Browser"`. **Never trusts pidfiles** — they go stale and lie.
|
||
|
||
If a relay is down but the veth is fine, the watchdog kills any existing relay
|
||
for that port and restarts it with the exact `netvm-node-up.sh` invocation
|
||
(inside the netns, listening on the veth IP).
|
||
|
||
## The Queue (`cdp_queue.py`)
|
||
|
||
**Why it exists:** nothing coordinated browser operations. DM sends, tab opens,
|
||
agent reads, and watchdog restarts all hit the same Chromium with zero
|
||
scheduling. Under fleet concurrency this overwhelms bl's CPU.
|
||
|
||
**Location:** `/home/super/Projects/NetVM/bin/cdp_queue.py`
|
||
|
||
**Design:**
|
||
- Per-node FIFO queue (muse, pip, 646, opm each independent)
|
||
- Max **2 concurrent** CDP operations per browser (`MAX_CONCURRENT = 2`)
|
||
- Priority levels: `PRIORITY_HIGH = 0` (DM sends, user-facing),
|
||
`PRIORITY_NORMAL = 1` (default), `PRIORITY_LOW = 2`
|
||
- Cross-process coordination via **flock'd ticket files** in `/tmp/cdp-queue/<node>/`
|
||
(not just threading — `dm.py` shells out to `muse-chat-api.py`, so in-process
|
||
semaphores alone don't coordinate)
|
||
- `QueueTimeout` after 60s (configurable per-call) — fails loud, never hangs forever
|
||
- Warns when wait exceeds 10s
|
||
|
||
**Integration points:**
|
||
- `muse-chat-api.py`: every CDP session wrapped in `with cdp_slot(node, priority=...)`.
|
||
Priority from `CDP_PRIORITY` env var (`high`/`normal`/`low`, default `normal`).
|
||
Graceful fallback: if `cdp_queue` import fails, runs unqueued (nullcontext).
|
||
This is the single choke point — all current and future callers get queuing.
|
||
- `dm.py`: `run()` accepts `priority` param, sets `CDP_PRIORITY` in subprocess env.
|
||
`dm_send()` uses `priority="high"`. Read-back verification stays normal.
|
||
|
||
**Usage:**
|
||
```python
|
||
from cdp_queue import cdp_slot, PRIORITY_HIGH
|
||
with cdp_slot("opm", priority=PRIORITY_HIGH):
|
||
# ... CDP operations ...
|
||
```
|
||
|
||
**Monitoring:** `queue_depth(node)` returns current depth. `/tmp/cdp-queue/<node>/`
|
||
has `slot-0.lock` / `slot-1.lock` (persistent, by design) and transient ticket
|
||
files (should clean up within ~60s of completion — slow async cleanup, not a leak).
|
||
|
||
## Monitors (side-chat crons)
|
||
|
||
Five runtime crons report to the Chromebox ops side chat. All silent when healthy.
|
||
|
||
| Monitor | Cadence | What it checks | Alert threshold |
|
||
|---------|---------|----------------|-----------------|
|
||
| chromebox-watchdog-alerts | 10 min | New `relaunch FAILED` lines in watchdog log | Any new failure |
|
||
| cdp-relay-health-monitor | 15 min | All 4 relays return HTTP 200 on veth IPs | Any non-200 |
|
||
| browser-flap-detector | 30 min | `relaunch OK` count per profile in last hour | 3+ restarts/hour |
|
||
| cdp-latency-monitor | 15 min | CDP `/json/version` response time per node | FAIL or >5s (2–5s tracked, alerts after 3 consecutive) |
|
||
| chromebox-error-log-watch | 1 hour | FATAL/crash/segfault/OOM in per-profile chrome logs | Any new match |
|
||
|
||
Watermarks live in `~/workspace/goals/chromebox-ops/hidden_files/` so alerts
|
||
only fire on genuinely new events, not repeats.
|
||
|
||
## Common Failures and Fixes
|
||
|
||
### Missing host veth IP
|
||
**Symptoms:** relay process running, but host can't reach `http://<peer-ip>:<port>`.
|
||
`ip addr show dev ve-<tag>` shows no `inet 10.201.x.1/30`.
|
||
**Cause:** veth pair exists but host-side IP was lost (observed 2026-10-04 for
|
||
muse/pip — interfaces up, IPs gone).
|
||
**Fix:** `sudo ip addr add <GW>/30 dev <VETH>` where GW/VETH come from
|
||
`netvm-names.sh`. Or run `netvm-node-up.sh <node>` (idempotent, also fixes
|
||
anything else that's drifted).
|
||
**Detect:** cdp-relay-watchdog logs `FAIL_LOUD` for this. Don't just restart
|
||
the relay — it won't help without the veth IP.
|
||
|
||
### Stale pidfiles
|
||
**Symptoms:** `netvm-topology.sh` (old versions) reports relay DOWN when it's up,
|
||
or UP on the wrong port. `/run/netvm-<node>-cdp-relay.pid` points to a dead or
|
||
recycled PID.
|
||
**Fix:** fixed 2026-10-04 — topology script now checks by connectivity, not
|
||
pidfile. If you see pidfile-based checks anywhere else, they're suspect.
|
||
|
||
### Kill loop (watchdog killing slow browser)
|
||
**Symptoms:** repeated `unhealthy, relaunching` → `relaunch FAILED` cycles in
|
||
quick succession. Browser process exists and CDP responds, but the page hasn't
|
||
loaded yet.
|
||
**Cause (pre-2026-10-04):** single 25s post-launch check, no retry, no
|
||
recent-launch guard.
|
||
**Fix:** fixed 2026-10-04 — 4 retries over 60s + skip kill if launched <2 min ago.
|
||
If you see this pattern again, the guard may need tuning (longer window).
|
||
|
||
### Futile-restart loop (agent-health killing a healthy browser)
|
||
**Symptoms:** `FAIL (api timeout)` + `restarting browser...` + `CRITICAL -
|
||
still down after restart` repeating every ~10 min for one node while its
|
||
CDP port stays up (observed 2026-10-06: def, 57 restarts, 155 API FAILs).
|
||
The API/account layer is broken; restarts cannot fix it, they just murder
|
||
a working browser.
|
||
**Fix:** `agent-health.sh` circuit breaker — after 3 consecutive futile
|
||
restarts the circuit OPENS (alert in log + journal, no more kills) until
|
||
a 30-min half-open probe or any successful check. Manual reset:
|
||
`rm /tmp/agent-health-state/circuit-<node> /tmp/agent-health-state/futile-<node>`.
|
||
Then fix the actual API-layer failure (account session/auth/chat-state),
|
||
not the browser.
|
||
|
||
### Setup-fed supervision (new nodes automatically watched)
|
||
**Wiring:** `netvm-node-up.sh` ends with `ensure-node-supervision.sh <node>`
|
||
(idempotent): appends the NODES.md registry row (port from netvm-names
|
||
pinning, honors CDP_PORT_OVERRIDE) and installs/enables
|
||
`chromebox-watchdog-<node>.timer`. Provision/onboarding reach it
|
||
transitively via node-up. Registry-driven supervisors (relay watchdog,
|
||
agent-health, relay-health/cdp-latency checks) pick up new rows on
|
||
their next run — no per-node code edits. Heal drift anytime:
|
||
`sudo bin/ensure-node-supervision.sh --all`.
|
||
|
||
### Relay on wrong IP
|
||
**Symptoms:** relay process exists but on the wrong veth IP (e.g., muse's relay
|
||
on pip's `10.201.87.2` instead of muse's `10.201.35.2`). Port responds on the
|
||
wrong node's IP.
|
||
**Cause:** relay started with wrong PEER_IP argument.
|
||
**Fix:** kill it, restart with the correct IP from `netvm-names.sh`:
|
||
`sudo ip netns exec warp-<node> setsid nohup python3 bin/netvm-cdp-relay.py <PEER_IP> <PORT> 127.0.0.1 <PORT>`
|
||
|
||
### Queue module missing
|
||
**Symptoms:** everything works but nothing is actually queued — `/tmp/cdp-queue/`
|
||
never gets ticket files. No errors (graceful fallback hides it).
|
||
**Cause:** `bin/cdp_queue.py` not present on bl (observed 2026-10-04 — integration
|
||
was wired but the module file was missing).
|
||
**Fix:** ensure the file exists and `python3 -m py_compile` passes. It's committed
|
||
in the NetVM repo — check `git status` if it's gone.
|
||
|
||
### Orphan relays on random ports
|
||
**Symptoms:** `pgrep -af netvm-cdp-relay` shows relays on ports like 9269, 9278,
|
||
9353, 10239, 10355 (hash-derived, not registry ports).
|
||
**Cause:** old node-ups or queue tests. Harmless but confusing.
|
||
**Fix:** kill them. Only the registry ports (9410/9420/9430/9440/9450/9455) should be running.
|
||
|
||
## Fleet Status From Blind Shells
|
||
|
||
`box fleet status` probes live (pgrep + peer-IP CDP). Sandboxed shells
|
||
(own PID/net namespaces, no sudo, no route to 10.201.x.x) fail both
|
||
probes for every node. Instead of misreporting STOPPED, fleet status
|
||
falls back to host watchdog evidence (`bin/host_evidence.py`):
|
||
|
||
- Recent watchdog timer runs (journal) with no newer failure line in
|
||
`cdp-relay-watchdog.log` / `chromebox-watchdog.log` (both are
|
||
silent-when-healthy) prove the node is up → `ACTIVE [*]`.
|
||
- `UNKNOWN` means neither live probes nor host evidence could decide
|
||
(e.g. watchdog timers not installed yet for that node).
|
||
- Host evidence never overrides a live local signal, so a fresh outage
|
||
observed on the host always wins over a minutes-old watchdog run.
|
||
|
||
Same rule drives `box approvals check`: `BLIND` = CDP ok on host, this
|
||
shell cannot reach it; approval queues are unverified, not clear.
|
||
|
||
## Key Reference
|
||
|
||
**Node bring-up:** `sudo /home/super/Projects/NetVM/bin/netvm-node-up.sh <node>`
|
||
(idempotent — safe to re-run, recreates veth/WireGuard/relay as needed)
|
||
|
||
**Node teardown:** `bin/netvm-node-down.sh <node>`
|
||
|
||
**Topology check:** `bin/netvm-topology.sh` (connectivity-based since 2026-10-04)
|
||
|
||
**Names:** `bin/netvm-names.sh` — source it: `. bin/netvm-names.sh && netvm_names <node>`
|
||
gives you `$NETNS $WG $VETH $VPEER $SUB $GW $PEER_IP`
|
||
|
||
**DM send:** `python3 bin/dm.py send --agent <from> --to <to> --target main "message"`
|
||
|
||
**Commit identity** (shared repo — always use per-command flags, never repo config):
|
||
```bash
|
||
git -c user.name="operator-main" -c user.email="operator-main@frontdoor.local" commit ...
|
||
```
|
||
|
||
**What NOT to do:**
|
||
- Don't trust pidfiles for relay health — they lie.
|
||
- Don't curl `127.0.0.1:<port>` on the host to check relays — nothing listens there.
|
||
- Don't kill a browser that just launched — check `ps -o lstart` first.
|
||
- Don't commit other sessions' uncommitted work — use `git add -p` or selective staging.
|
||
- Don't restart browsers to fix relay problems — relays and browsers are independent.
|