chore(fleet): sync operator memory, hatch menu dialogs, and watchdog alerts
This commit is contained in:
operator
@@ -64,7 +64,7 @@ host → veth IP:port (e.g. 10.201.87.2:9420)
|
||||
### chromebox-watchdog (browser health)
|
||||
|
||||
- **Script:** `/home/super/Projects/NetVM/bin/chromebox-watchdog.sh`
|
||||
- **Timers:** `chromebox-watchdog-<profile>.timer` (one per profile: muse, pip, 646, opm)
|
||||
- **Timers:** `chromebox-watchdog-<profile>.timer` (one per profile — every active registry node: muse, pip, 646, opm, def, dev)
|
||||
- **Cadence:** every 2 minutes
|
||||
- **Log:** `/home/super/Projects/NetVM/chromebox-watchdog.log` (10 MB rotation, 1 backup gen)
|
||||
- **Per-profile Chromium output:** `/home/super/Projects/NetVM/chromebox-<profile>.log`
|
||||
@@ -233,7 +233,7 @@ in the NetVM repo — check `git status` if it's gone.
|
||||
**Symptoms:** `pgrep -af netvm-cdp-relay` shows relays on ports like 9269, 9278,
|
||||
9353, 10239, 10355 (hash-derived, not registry ports).
|
||||
**Cause:** old node-ups or queue tests. Harmless but confusing.
|
||||
**Fix:** kill them. Only 9410/9420/9430/9440 should be running.
|
||||
**Fix:** kill them. Only the registry ports (9410/9420/9430/9440/9450/9455) should be running.
|
||||
|
||||
## Fleet Status From Blind Shells
|
||||
|
||||
@@ -246,7 +246,7 @@ falls back to host watchdog evidence (`bin/host_evidence.py`):
|
||||
`cdp-relay-watchdog.log` / `chromebox-watchdog.log` (both are
|
||||
silent-when-healthy) prove the node is up → `ACTIVE [*]`.
|
||||
- `UNKNOWN` means neither live probes nor host evidence could decide
|
||||
(e.g. def/dev have no relay-monitor coverage).
|
||||
(e.g. watchdog timers not installed yet for that node).
|
||||
- Host evidence never overrides a live local signal, so a fresh outage
|
||||
observed on the host always wins over a minutes-old watchdog run.
|
||||
|
||||
|
||||
+37
-15
@@ -48,26 +48,48 @@ settled-state probes (`DOM-PAGE-STRUCTURE.md` §3.2). The dialog is found
|
||||
**by text, not by structure** — this is the central fragility of the current
|
||||
implementation.
|
||||
|
||||
Known structural facts:
|
||||
Known structural facts (Verified 2026-10-06 live capture on fleet):
|
||||
- The dialog is in-DOM (React-rendered), so `Runtime.evaluate` sees it; no
|
||||
shadow-DOM piercing has been needed so far.
|
||||
- Buttons are plain `<button>` elements matched by innerText
|
||||
(`allow` / `deny` / `block`, case-insensitive).
|
||||
- Per `AGENTS.md` (2026-10-03): React selectors behave identically in
|
||||
headful and headless environments, so selectors captured headless apply
|
||||
to the user's browser too — and vice versa.
|
||||
shadow-DOM piercing needed.
|
||||
- Container: `div[data-testid="hatch-inline-approval-card"]` with
|
||||
`data-hatch-approval-surface="panel"`.
|
||||
- Header: `div[data-testid="approval-panel-header"]`.
|
||||
- Primary Allow action: `button[data-hatch-approval-primary-action="true"]` ("Allow once").
|
||||
- Dropdown options: `button[data-slot="dropdown-menu-trigger"]` (contains "Always allow").
|
||||
- Deny action: `<button>` with innerText "Deny".
|
||||
- Secondary/Background queued approvals surface:
|
||||
`div[data-testid="hatch-inline-approval-card"][data-hatch-background-approval-surface="true"]`
|
||||
with `<p class="text-footnote text-text-secondary">N tasks need review</p>`
|
||||
and `<button data-pel-click="chat_background_approval_review">Review</button>`.
|
||||
|
||||
Candidate selectors to verify on next live capture (none confirmed yet):
|
||||
### Confirmed Live Selectors (2026-10-06):
|
||||
```javascript
|
||||
'[role="dialog"]',
|
||||
'[role="alertdialog"]',
|
||||
'[data-testid*="dialog"]',
|
||||
'[data-testid*="approval"]',
|
||||
'[data-testid*="permission"]',
|
||||
// button-level (confirmed pattern, unconfirmed testids):
|
||||
'button' // innerText matches /allow once|always allow|deny/i
|
||||
// Active approval panel container
|
||||
'div[data-testid="hatch-inline-approval-card"]'
|
||||
// Panel header
|
||||
'[data-testid="approval-panel-header"]'
|
||||
// Primary action button ("Allow once")
|
||||
'button[data-hatch-approval-primary-action="true"]'
|
||||
// Background review surface ("N tasks need review" / "A task needs review")
|
||||
'[data-hatch-background-approval-surface="true"]'
|
||||
// Background review button trigger
|
||||
'[data-pel-click="chat_background_approval_review"]'
|
||||
```
|
||||
|
||||
### Queued Background Task Reviews ("N tasks need review")
|
||||
When multiple scheduled tasks or background operations trigger approval prompts concurrently (e.g. `opm` with scheduled Fleet Health Monitor runs):
|
||||
1. Muse renders the first approval card inline over chat.
|
||||
2. Below it, Muse docks a background approval banner:
|
||||
`<div data-testid="hatch-inline-approval-card" data-hatch-background-approval-surface="true">`
|
||||
stating `"N tasks need review"` with a `"Review"` button (`data-pel-click="chat_background_approval_review"`).
|
||||
3. Allowing or denying the active approval immediately advances the queue: the next queued task pops into the active inline panel, decrementing the background count (e.g. from 2 down to "A task needs review" to clear).
|
||||
4. Automated approval engines must inspect both the active card and `[data-hatch-background-approval-surface="true"]` to confirm whether agents remain blocked.
|
||||
|
||||
### Troubleshooting Blocked Agents
|
||||
- **In `muse-tui`**: Type `/blocked` (or press `[a]` / `F2`) from any view to open the Approvals & Blocked Tasks Drawer. Use `j`/`k` or arrow keys to navigate between blocked nodes; press `[1]` to Allow, `[2]` Always, `[3]` Deny, `[R]` Proceed, `[X]` Dismiss. Actions target the highlighted agent.
|
||||
- **In CLI**: Run `box blocked` or `box approvals` to view fleet approval status; use `box approvals allow <node>` to approve.
|
||||
|
||||
|
||||
## 3. How `check_approvals` works today
|
||||
|
||||
Location: `bin/muse-chat-api.py`, `check_approvals(ws)` (~line 70).
|
||||
|
||||
+12
-5
@@ -37,16 +37,23 @@ Static: `permissions.connector_defaults`,
|
||||
`permissions.web_access` (`auto_allow`/`always_ask`);
|
||||
`permissions.advanced.transparent_proxy|tls_interception|
|
||||
sni_mismatch_rejection`, `data_controls.ai_improvement` (`on`/`off`);
|
||||
`general.theme` (match/default/blue/purple/pink/orange/green/
|
||||
beige/monochrome).
|
||||
`general.theme` (avatar/default/blue/purple/pink/orange/green/
|
||||
beige/monochrome; `avatar` = "Match my avatar").
|
||||
|
||||
Families: `permissions.websites:<host>` (`Allow`/`Ask`/`Deny`),
|
||||
`permissions.protocols:<slug>` (`on`/`off`; slugs discovered live,
|
||||
e.g. `mcp-sse`, `mcp-streamable`, `agent-skills`, `mcp-apps`,
|
||||
`mcp-oauth`).
|
||||
`permissions.protocols:<slug>` (`on`/`off`; network primitives
|
||||
`outbound-ssh`, `smtp`, `imap-pop3`, `database`, `ftp`, `dns`,
|
||||
`other-tcp`, `other-udp` pinned live 2026-10-06, MCP titles kept
|
||||
defensively; unique substrings like `ssh` also resolve).
|
||||
|
||||
Every `set` verifies in place and reads back through a fresh
|
||||
session; readback mismatch reports failure, never partial success.
|
||||
Switch commits can land slowly (or on dialog close), so in-flow
|
||||
verifies poll and the fresh readback is the source of truth; sets
|
||||
that only the readback confirms carry `"readback_only": true`.
|
||||
Website Ask/Deny is one-way: the override row leaves the allowed
|
||||
list (no add UI), verified by absence; absent hosts read as
|
||||
"not in Websites list" (effective: web-access default).
|
||||
Caller errors (unknown node/toggle/tab/value) raise `MenuError`
|
||||
before any CDP traffic. Transport failures return `{"ok": False}`.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user