feat: setup-fed watchdog supervision for all registry nodes
Close the def/dev supervision gap at the source: every node brought up gets watched, and every supervisor enumerates the registry. - bin/ensure-node-supervision.sh (new, idempotent): appends the NODES.md row (netvm-names port, honors CDP_PORT_OVERRIDE so it never fights provision's picker) and installs/enables chromebox-watchdog-<node>.timer. --all heals drift (registry + /etc/netvm identities). Template verified byte-identical to the installed def unit. - netvm-node-up.sh: calls ensure (non-fatal) at the end. Provision and the onboarding pipeline reach it transitively. - relay-health-check.sh, cdp-latency-check.sh: registry-driven watched_nodes() + LIB_ONLY guards (were hardcoded 4 nodes). - tests/test_node_supervision.py (6): row add/idempotent/override, timer render, node-up wiring, both watched_nodes(). - CHROMEBOX-RUNBOOK.md: setup-fed supervision section. Pairs with the registry-driven relay/chromebox watchdogs: new rows are picked up on the next run with no per-node code edits.
This commit is contained in:
@@ -203,6 +203,16 @@ a 30-min half-open probe or any successful check. Manual reset:
|
||||
Then fix the actual API-layer failure (account session/auth/chat-state),
|
||||
not the browser.
|
||||
|
||||
### Setup-fed supervision (new nodes automatically watched)
|
||||
**Wiring:** `netvm-node-up.sh` ends with `ensure-node-supervision.sh <node>`
|
||||
(idempotent): appends the NODES.md registry row (port from netvm-names
|
||||
pinning, honors CDP_PORT_OVERRIDE) and installs/enables
|
||||
`chromebox-watchdog-<node>.timer`. Provision/onboarding reach it
|
||||
transitively via node-up. Registry-driven supervisors (relay watchdog,
|
||||
agent-health, relay-health/cdp-latency checks) pick up new rows on
|
||||
their next run — no per-node code edits. Heal drift anytime:
|
||||
`sudo bin/ensure-node-supervision.sh --all`.
|
||||
|
||||
### Relay on wrong IP
|
||||
**Symptoms:** relay process exists but on the wrong veth IP (e.g., muse's relay
|
||||
on pip's `10.201.87.2` instead of muse's `10.201.35.2`). Port responds on the
|
||||
|
||||
Reference in New Issue
Block a user