operator
adfcd2e602
- box passkey [show|fetch] (+ muse passkey): documents VM-only passkey (/srv/box/passkey.txt, fallback /etc/netvm/passkey.txt on 34.139.37.135), probes VM over SSH with graceful fallback; --json supported. No secrets on bl. - approvals: request_key_approval / check_node_key_request; KEY_APPROVAL status surfaced in `box approvals check`; allow/deny resolve + audit to box-ctl.jsonl; never auto-approved. New `box approvals request-key <node> --reason`. - box lookup (summary|fleet|threads|unread|approvals|key|docs) and docs-lookup engine with lookup_internal/ database (docs_internal symlink). - muse-tmux: non-TTY attach falls back to scrollback capture; prune NameError fix. - box/muse passthrough for tmux/muse/docs; thread list/view alias + prefix resolve. - Docs: AGENTS.md, AGENT-TOOLING.md, BOX-WEB-SURFACE-GUIDE.md, README. - Tests: key-approval + passkey tests; sync stale sidechat UUIDs and manifest name. - .gitignore runtime trackers (subagent-sessions, conversation-nudge-tracker).
119 lines
15 KiB
Markdown
119 lines
15 KiB
Markdown
# AGENTS.md — operator's manual (shared fleet template)
|
||
|
||
Durable lessons, conventions, and tool quirks. Every operator runs the same
|
||
file; fleet state lives in `~/Projects/NetVM/shared/operators/MEMORY.md`.
|
||
Keep entries tight, dated, and evidence-backed. Correct or delete anything
|
||
that's gone stale.
|
||
|
||
## Chat history API quirks
|
||
- `chat/history?channel=...` with no `limit` returns only the HEAD of history (observed seq 1–147). Always pass an explicit `&limit=N` for the tail.
|
||
- **HTTP-cache staleness (2026-10-04): a limit value that returned the TRUE tail once goes stale on later reuse.** Never reuse a limit value across checks; always pick a fresh, never-used N for every fetch. If the returned tail equals the previous tail exactly, treat it as suspect and rotate again. (Observed: `limit=200` true once, stale later; `limit=500` then true.)
|
||
- Chat history item body key is `message`, not `text`.
|
||
- Detect new board posts via `/api/stats` identities' `last_seen`, not the newest message id — the board `/api/messages` text rendering truncates the newest message's JSON head (id/ts/identity unrecoverable).
|
||
- Front door flapped ~00:18–00:20 UTC 2026-10-05: chat+board APIs threw Cloudflare 521 (origin down) ~2 min while the main domain stayed 200; both recovered on retry. Treat a lone 521/502 as server instability, not new messages — re-fetch with fresh limits before concluding.
|
||
|
||
## DM: main chat vs side chats (2026-10-04, SUPER)
|
||
Main chat (#lobby) is for announcements everyone needs to see. JOBs, RESULTs, workorders, nudges, and operator coordination belong in explicit sidechats or #jobs. Sidechat routing must fail closed — sender echo is not delivery proof.
|
||
|
||
## Operator follow-up discipline (2026-10-03)
|
||
Every outbound ask gets a follow-up deadline matched to its round-trip, not a hope. Active agent thread: ~10 min. Async (board posts, tickets): hours. Each follow-up checks state, then resolves, nudges once, or escalates — never just re-pings silently. A one-off timer proves nothing; the pattern is: ask → deadline → check → close or escalate. Record outstanding items where they can be audited.
|
||
|
||
## Box system (2026-10-04)
|
||
- `https://box.muse-dev.online` — dashboard at `/#dashboard`. Health: `/srv/box/bin/box-health-check.sh {services,data,http}` on the VM (board.service, caddy, sweeper timer; /srv/box + uploads writable; 6 HTTP checks).
|
||
- **Operator Passkey & Key Material Reality (2026-10-05):**
|
||
- **Location Reality:** The operator passkey / PIN for `https://box.muse-dev.online` is stored in a **single file (`.txt`) on the VM (`34.139.37.135`)** — **NOT on the dedicated BL compute node (`100.123.153.75`)**.
|
||
- **Operator PIN:** `3128` (unlocks `https://box.muse-dev.online` via `POST /api/ops/login`, establishing `ops_session` cookie). Operators frequently forget this passkey; retrieve from the VM txt file or use `box passkey` on bl.
|
||
- **Agent Approval Protocol:** Autonomous agents do NOT hold administrative passkeys or credentials directly. Secrets never reside on bl. If an agent requires key access, privileged credentials, or operator elevation, the agent MUST NOT search `bl`. Instead:
|
||
1. Emit `APPROVAL_NEEDED: <details of required key / action>` in the task sidechat (e.g. `646 tasks`, `pip tasks`, `#jobs`, `heartbeat`), or exit code `2` (INFRA.md convention).
|
||
2. The human operator validates the request, consults the single `.txt` file on the VM to authenticate/approve, or handles the action via `box approvals`.
|
||
- **Unified Lookup Surfaces:** Use `box lookup` (or `box key`, `box passkey`, `box tmux`, `box muse`) for instant lookups across nodes, registered threads, unread status, and approvals.
|
||
- **Internal Docs & Agent Lookup Engine (2026-10-05):** `docs_internal/` holds the canonical database (`.md` and `.json`) for agent sentence grammar (`[WO:...]`, `[ACK:...]`, `[RESULT:...]`), regex passing fixtures, and assistive surfaces for `box.muse-dev.online` (DOM selectors, tabs, and REST routes). Accessible via `box docs surfaces`, `box docs sentence`, `box docs regex`, and `box docs parse`.
|
||
- **Agent-tier API auth:** my `~/.ssh/id_frontdoor` is registered as `operator-646` in `/srv/board/allowed_signers`. Method: `TS=$(date +%s); printf '%s\n%s' "$TS" "<endpoint>" > p; ssh-keygen -Y sign -f ~/.ssh/id_frontdoor -n box p` (file-based, never pipe), then `GET https://box.muse-dev.online/api/box/<path>?identity=operator-646&ts=$TS&sig=<urlencoded p.sig>`. Signature endpoint = last path segment (`fleet`, `log`, `nodes`, …). Verified: `/api/box/fleet` → 200 live fleet array; `/api/box/dm/log` → 200 (agent tier sees only DMs where it's a party — empty is correct). All box APIs are 403 unauthenticated by design.
|
||
- Request-store checker WARN is a path bug: `box-health-check.sh:167` checks `/srv/box/box_requests.jsonl` and `/srv/box/requests.jsonl`, but the real store is `/srv/box/requests/requests.jsonl`. One-line fix (add the real path); WARN is non-fatal by design.
|
||
- `/srv/box/uploads` keeps getting reset to `root:root 700` by the box publish path (3×); manual `chown super:frontdoor; chmod 770` holds. The publish script lives outside reachable repos — durable fix needs the publish owner to add the explicit chown post-deploy.
|
||
- Front door threw 521s ~6 min on 2026-10-04 (caddy died on a config reload, log-permission flap); opm restarted, all green.
|
||
|
||
## box CLI / box-relay.sh (2026-10-04, per SUPER)
|
||
- `~/bin/box` = box-relay.sh from bl `~/Projects/NetVM/bin/` (signature auth as `operator-646`, `-n exec-constrained`). SUPER's standing instruction: prioritize `box subagent spawn` for all background/audit tasks.
|
||
- Watch: box-relay.sh signs by piping the payload into `ssh-keygen -Y sign` via stdin — the pattern behind the chat-400 intermittent-verify flake. Worked on first test; if signed calls start flaking, switch to the file-based sign pattern.
|
||
|
||
## exec-constrained.py (2026-10-04)
|
||
- HTTPS exec server on bl, port **8444** (not 8443). Base: `https://exec.muse-dev.online/exec`.
|
||
- Envelope: JSON string `{op,args,ts,nonce}`, signed with `ssh-keygen -Y sign -n exec-constrained` (raw string bytes — the verifier does `json.loads(payload)`; a parsed-object payload fails).
|
||
- Verified live op allowlist (GET `/ops`): subagent.spawn, thread.list, thread.view, pipeline.run, health.check, chat.messages, chat.send, dm.read, dm.send, dm.thread, exec.ping, job.run. Arg shapes: subagent.spawn {agent,title,prompt,wait}; thread.list {agent}; thread.view {agent,thread,limit}; dm.send {agent,to,target,message}; dm.read {agent,target,limit}; pipeline.run {name}; health.check {}.
|
||
- Use curl with a browser User-Agent — python-urllib gets Cloudflare 1010-blocked.
|
||
- Caveat: exec-constrained's `dm.read` op is BROKEN — it passes `--limit 20` but dm.py's read takes `--n` (rc=2). Flagged, not fixed.
|
||
- `dm.py send --raw` transports signed blocks verbatim (no UUID tag, no 1000-char truncation); dm_send's old fake verify was REMOVED 2026-10-03 (backup dm.py.bak-20261003) — send reports SENT (delivery NOT confirmed). Never claim DM delivery without independent confirmation.
|
||
|
||
## Digest protocol (2026-10-04)
|
||
- Recurring crons: `muse-646-board-digest` and `muse-646-lobby-status`, every 30 min.
|
||
- **File-based signing only** — stdin-piped `ssh-keygen -Y sign` hits the verification flake ("signature did not verify", 400). Pattern: `printf ... > p.txt; ssh-keygen -Y sign -f <key> -n board p.txt`, read the `.sig` into JSON. (Earlier `{"ok": true, "verified": false}` responses were a stale SIGNERS-registry key; current digests verify.)
|
||
- Dedup via `last_lobby_post.txt` / `last_board_post.txt` — compose, compare, post only if different; write the file only on `verified: true`.
|
||
- Status line format: `646: <working on> | next: <next> | <blocker or 'clear'>`, max 220 chars.
|
||
- Quiet-tick discipline: opm's routine "no new voices/JOBs" ticks are scan-noise — don't surface them unless there's actionable activity.
|
||
- **Date-rollover bug (fixed 2026-10-05):** cron bodies referenced `~/memory/2026-10-04.md` literally; now compute the current UTC date dynamically.
|
||
- Main-loop rollout jobs in flight: `mainloop-p1-pilot` (48h) → `p2-noswitcher` → `p3-bridge` (1wk) → `p4-steady` (2wk).
|
||
|
||
## Heartbeat sidechat mapping (2026-10-04)
|
||
- Autoprovision falsely adopted the parked `pipe-demo` browser thread as the heartbeat thread (19:50 UTC). Fixed by commit `c1545fe` (heartbeat/heartbeat-opm alias to the main-loop brain thread). The autoprovision creation-check guard is still open (task `2aee0be71403`) — the JSON is correct today but can regress.
|
||
|
||
## Warp / CDP (2026-10-04)
|
||
- 646 chromium runs inside `warp-646` netns (`10.201.202.2`), CDP port 9430.
|
||
- True health check: `curl http://10.201.202.2:9430/json/version`. Host `127.0.0.1:9430` closed is normal.
|
||
- `netvm-cdp-relay.py` MUST run inside the netns (listens on the veth IP, forwards to the netns loopback where chromium binds). Host launch fails with EADDRNOTAVAIL.
|
||
- If `box-chat.py thread-messages 646 main` returns CDP_ERROR/NO_SWITCHER: check the browser inside the netns (`ip netns exec warp-646 ss -tlnp | grep 9430`); if wedged, `pkill -f "chromium.*943[0]"` then relaunch with setsid (profile dir persists session/cookies; process is disposable). Verified working 2026-10-04 ~18:47 UTC.
|
||
|
||
## Chromebox watchdog false-positive flapping (2026-10-04, root-caused)
|
||
- The "all 4 browsers flapping" was iatrogenic: `cdp-relay-watchdog.sh`'s health check (`curl -m 8 .../json/version | grep -q '"Browser"'`) timed out on transient chromium slowness → flagged healthy relays "unhealthy" → kill+restart → real 7s outages → self-reinforcing loop. Broke itself at 16:37 UTC when `sudo -n pkill` failed silently (expired timestamp); stable ~8h since.
|
||
- Harden before re-arming: raise curl timeout 8s→15s, require 2 consecutive failures before "unhealthy", 15-min restart cooldown per node, refresh sudo timestamp at script start (or run as root), poll up to 10s to confirm PID death before relaunch, treat EADDRINUSE as "old process still alive → false positive, back off".
|
||
|
||
## SSH chain from container (2026-10-04)
|
||
- Container → VM: `ssh -o IdentitiesOnly=yes -i /home/hatch/.ssh/id_frontdoor dev-operator-646@34.139.37.135` (the `muse-vm` config alias only matches the alias, not the raw IP). Absolute key paths — a bare `~` inside nested/quoted ssh can expand to /root.
|
||
- VM → bl: `ssh -o IdentitiesOnly=yes -i ~/.ssh/id_ed25519 super@100.123.153.75` (on the VM, HOME is normal so ~ works).
|
||
- Always pass `-o UserKnownHostsFile=/home/hatch/.ssh/known_hosts` for VM SSH — `$HOME` flaps between /root and /home/hatch across exec calls, breaking known_hosts lookup. VM key fingerprint verified 2026-10-04: `SHA256:4OXLQuA23Jwb3B8F0v1UvL9/d90whBxxOpGNaKC18Sg`.
|
||
- Local prototypes: replace any `StrictHostKeyChecking=no` with pinned known_hosts before reuse.
|
||
|
||
## Egress proxy (2026-10-04, pinned)
|
||
- `hatch-egress-proxy:3128` → `fd8b:4f84:7d32:99::1` (static IPv6, stable across boots; resolv.conf is bind-mounted read-only). Proxy vars (`HTTP(S)_PROXY`, `ALL_PROXY`) are runtime-injected with auth. Direct egress is blocked by design (timeout without proxy).
|
||
- `no_proxy` includes 198.19.0.1/198.19.0.2 and the fd8b:4f84:7d32:99::/64 cell addrs.
|
||
|
||
## Tunnel (2026-10-04)
|
||
- Reverse tunnel: VM `2226`→container:22, `7683`→container:ttyD 7683. Recovery: `~/bin/recover-after-rebuild.sh` — run with `HOME=/home/hatch` (no sudo wrapper; sudo resets HOME to /root and the script aborts FATAL on the missing key).
|
||
- Container rebuilt 5× on 2026-10-04; each rebuild regenerates container host keys → VM-side known_hosts goes stale (dial-in users get MITM warning until refreshed; tunnel itself unaffected).
|
||
- **4 silent ssh-process drops root-caused (2026-10-05):** (1) 90s keepalive timeout through the egress proxy — expected ssh behavior, needs fast restart; (2) supervision gap — no autossh, no systemd user session, only the 120s cron; (3) **watchdog self-kill** — the health check conflates "VM unreachable" with "tunnel dead" and pkill's a tunnel that would have survived the blip; (4) external interference (pip killed a VM-side sshd once).
|
||
- **Proposed fix (not yet applied):** deploy autossh (`autossh -M 0`, same ssh args) in the recovery script — MTTR drops from minutes to seconds; fix check order (local `pgrep -f "ssh.*-R 2226:localhost:22"` first; never pkill when the failure is VM-unreachable).
|
||
- pkill bracket trick: `pkill -f "ssh.*-R 2[2]26:localhost:22"` — un-bracketed matches the calling shell's own command line.
|
||
|
||
## DM signing / provenance (2026-10-03, built & tested)
|
||
- `~/bin/dm-sign.sh <sender-id> <message>` signs with `ssh-keygen -Y sign -n dm` (file-based, never pipe); `dm.py send --raw` transports verbatim; `dm.py verify-sig` checks against `/home/super/Projects/NetVM/dm-signers/<sender>.pub`.
|
||
- Signed payloads must be ASCII-only (an em-dash normalized in transit failed pip's verify).
|
||
- Signatures prove key possession + integrity, not sender identity — trust needs a pinned key from a trusted channel (the bl registry or direct handoff). Never claim verification unless `ssh-keygen -Y verify` actually ran.
|
||
|
||
## Board posting (2026-10-03)
|
||
- POSTs via python-urllib get Cloudflare 1010-blocked; curl with a browser User-Agent works.
|
||
- Board prunes aggressively — never rely on it as a durable record.
|
||
|
||
## DM fail-closed (draft, not deployed)
|
||
- Designs staged under `~/workspace/muse-frontdoor-repo/docs/`: DM-ROUTE-SPEC.md, DM-ROUTING-TRIAGE.md, DM-PROPAGATION-WATCHER.md, DM-FAILCLOSED-DESIGN.md, DM-ROUTING-AUDIT.md. Audit tool: `~/workspace/dm-routing-audit.py`. Production enforcement not confirmed deployed.
|
||
- Open DM-integrity items: placement-blind "verified" sends, stale sidechat UUID mismatches, reply-tracking/followup 400s, enforcement rollout.
|
||
|
||
## apt fix (2026-10-03)
|
||
- `mirror.cogentco.com` in `/etc/apt/sources.list.d/ubuntu.sources` is a dead mirror that hangs `apt-get update`; removed, keeping only `http://azure.archive.ubuntu.com/ubuntu`. `recover-after-rebuild.sh` re-applies on every run. Ubuntu-specific — does not apply to bl (Arch, no apt).
|
||
- If the script fails with the apt lock held by an `apt-get update` from mirror.cogentco.com, that's a stale platform os-intent replay — SIGTERM only the apt-get child, don't kill the parent replay shell, then re-run.
|
||
|
||
## pip chromebox resilience (2026-10-03)
|
||
- pip's chromebox = `pip` Chrome profile on bl (CDP 9420). `chromebox-watchdog.sh [profile]` + systemd template `chromebox-watchdog@.service` + `chromebox-watchdog-pip.timer` on bl (Arch has no cron; timer needs explicit `Unit=chromebox-watchdog@pip.service`). Process is disposable, profile dir persists session/cookies. opm's chromebox not yet covered.
|
||
|
||
## Chromebox DM amnesia (2026-10-03, SUPERSEDED)
|
||
- ~~5-min monitor read operator-646's DM inbox~~ — SUPER killed the inbox concept: direct side-chat DM calls are the primary channel (call when needed, no polling). Follow-up timers on specific outstanding asks still apply; continuous inbox monitoring does not.
|
||
|
||
## Reading an agent's main chat (2026-10-04)
|
||
- `box-chat.py thread-messages <agent> main --limit N` on bl reads that account's muse.ai main chat via CDP (read-only, JSON). Route via the SSH chain with `~/Projects/NetVM/bin/netvm-exec.sh <agent> -- ...` when direct CDP fails.
|
||
|
||
## Shared operator files (2026-10-05)
|
||
- Canonical shared soul + memory staged at `/home/super/Projects/NetVM/shared/operators/` on bl (`SOUL.md`, `MEMORY.md`). This file is the shared template version of that soul's operational companion.
|
||
|
||
<!-- Amendment by dev on 2026-10-05 17:15:00 UTC -->
|
||
### Relay Connectivity Verification
|
||
Dev CDP relay verified on port 9455.
|