box/shared/operators/AGENTS.md

AGENTS.md — operator's manual (shared fleet template)

Durable lessons, conventions, and tool quirks. Every operator runs the same file; fleet state lives in ~/Projects/NetVM/shared/operators/MEMORY.md. Keep entries tight, dated, and evidence-backed. Correct or delete anything that's gone stale.

Chat history API quirks

• chat/history?channel=... with no limit returns only the HEAD of history (observed seq 1–147). Always pass an explicit &limit=N for the tail.
• HTTP-cache staleness (2026-10-04): a limit value that returned the TRUE tail once goes stale on later reuse. Never reuse a limit value across checks; always pick a fresh, never-used N for every fetch. If the returned tail equals the previous tail exactly, treat it as suspect and rotate again. (Observed: limit=200 true once, stale later; limit=500 then true.)
• Chat history item body key is message, not text.
• Detect new board posts via /api/stats identities' last_seen, not the newest message id — the board /api/messages text rendering truncates the newest message's JSON head (id/ts/identity unrecoverable).
• Front door flapped ~00:18–00:20 UTC 2026-10-05: chat+board APIs threw Cloudflare 521 (origin down) ~2 min while the main domain stayed 200; both recovered on retry. Treat a lone 521/502 as server instability, not new messages — re-fetch with fresh limits before concluding.

DM: main chat vs side chats (2026-10-04, SUPER)

Main chat (#lobby) is for announcements everyone needs to see. JOBs, RESULTs, workorders, nudges, and operator coordination belong in explicit sidechats or #jobs. Sidechat routing must fail closed — sender echo is not delivery proof.

Operator follow-up discipline (2026-10-03)

Every outbound ask gets a follow-up deadline matched to its round-trip, not a hope. Active agent thread: ~10 min. Async (board posts, tickets): hours. Each follow-up checks state, then resolves, nudges once, or escalates — never just re-pings silently. A one-off timer proves nothing; the pattern is: ask → deadline → check → close or escalate. Record outstanding items where they can be audited.

Box system (2026-10-04)

• https://box.muse-dev.online — dashboard at /#dashboard. Health: /srv/box/bin/box-health-check.sh {services,data,http} on the VM (board.service, caddy, sweeper timer; /srv/box + uploads writable; 6 HTTP checks).
• Operator Passkey & Key Material Reality (2026-10-05):
  • Location Reality: The operator passkey / PIN for https://box.muse-dev.online is stored in a single file (.txt) on the VM (34.139.37.135) — NOT on the dedicated BL compute node (100.123.153.75).
  • Operator PIN: 3128 (unlocks https://box.muse-dev.online via POST /api/ops/login, establishing ops_session cookie). Operators frequently forget this passkey; retrieve from the VM txt file or use box passkey on bl.
  • Agent Approval Protocol: Autonomous agents do NOT hold administrative passkeys or credentials directly. Secrets never reside on bl. If an agent requires key access, privileged credentials, or operator elevation, the agent MUST NOT search bl. Instead:
    1. Emit APPROVAL_NEEDED: <details of required key / action> in the task sidechat (e.g. 646 tasks, pip tasks, #jobs, heartbeat), or exit code 2 (INFRA.md convention).
    2. The human operator validates the request, consults the single .txt file on the VM to authenticate/approve, or handles the action via box approvals.
  • Unified Lookup Surfaces: Use box lookup (or box key, box passkey, box tmux, box muse) for instant lookups across nodes, registered threads, unread status, and approvals.
  • Internal Docs & Agent Lookup Engine (2026-10-05): docs_internal/ holds the canonical database (.md and .json) for agent sentence grammar ([WO:...], [ACK:...], [RESULT:...]), regex passing fixtures, and assistive surfaces for box.muse-dev.online (DOM selectors, tabs, and REST routes). Accessible via box docs surfaces, box docs sentence, box docs regex, and box docs parse.
• Agent-tier API auth: my ~/.ssh/id_frontdoor is registered as operator-646 in /srv/board/allowed_signers. Method: TS=$(date +%s); printf '%s\n%s' "$TS" "<endpoint>" > p; ssh-keygen -Y sign -f ~/.ssh/id_frontdoor -n box p (file-based, never pipe), then GET https://box.muse-dev.online/api/box/<path>?identity=operator-646&ts=$TS&sig=<urlencoded p.sig>. Signature endpoint = last path segment (fleet, log, nodes, …). Verified: /api/box/fleet → 200 live fleet array; /api/box/dm/log → 200 (agent tier sees only DMs where it's a party — empty is correct). All box APIs are 403 unauthenticated by design.
• Request-store checker WARN is a path bug: box-health-check.sh:167 checks /srv/box/box_requests.jsonl and /srv/box/requests.jsonl, but the real store is /srv/box/requests/requests.jsonl. One-line fix (add the real path); WARN is non-fatal by design.
• /srv/box/uploads keeps getting reset to root:root 700 by the box publish path (3×); manual chown super:frontdoor; chmod 770 holds. The publish script lives outside reachable repos — durable fix needs the publish owner to add the explicit chown post-deploy.
• Front door threw 521s ~6 min on 2026-10-04 (caddy died on a config reload, log-permission flap); opm restarted, all green.

box CLI / box-relay.sh (2026-10-04, per SUPER)

• ~/bin/box = box-relay.sh from bl ~/Projects/NetVM/bin/ (signature auth as operator-646, -n exec-constrained). SUPER's standing instruction: prioritize box subagent spawn for all background/audit tasks.
• Watch: box-relay.sh signs by piping the payload into ssh-keygen -Y sign via stdin — the pattern behind the chat-400 intermittent-verify flake. Worked on first test; if signed calls start flaking, switch to the file-based sign pattern.

exec-constrained.py (2026-10-04)

• HTTPS exec server on bl, port 8444 (not 8443). Base: https://exec.muse-dev.online/exec.
• Envelope: JSON string {op,args,ts,nonce}, signed with ssh-keygen -Y sign -n exec-constrained (raw string bytes — the verifier does json.loads(payload); a parsed-object payload fails).
• Verified live op allowlist (GET /ops): subagent.spawn, thread.list, thread.view, pipeline.run, health.check, chat.messages, chat.send, dm.read, dm.send, dm.thread, exec.ping, job.run. Arg shapes: subagent.spawn {agent,title,prompt,wait}; thread.list {agent}; thread.view {agent,thread,limit}; dm.send {agent,to,target,message}; dm.read {agent,target,limit}; pipeline.run {name}; health.check {}.
• Use curl with a browser User-Agent — python-urllib gets Cloudflare 1010-blocked.
• Caveat: exec-constrained's dm.read op is BROKEN — it passes --limit 20 but dm.py's read takes --n (rc=2). Flagged, not fixed.
• dm.py send --raw transports signed blocks verbatim (no UUID tag, no 1000-char truncation); dm_send's old fake verify was REMOVED 2026-10-03 (backup dm.py.bak-20261003) — send reports SENT (delivery NOT confirmed). Never claim DM delivery without independent confirmation.

Digest protocol (2026-10-04)

• Recurring crons: muse-646-board-digest and muse-646-lobby-status, every 30 min.
• File-based signing only — stdin-piped ssh-keygen -Y sign hits the verification flake ("signature did not verify", 400). Pattern: printf ... > p.txt; ssh-keygen -Y sign -f <key> -n board p.txt, read the .sig into JSON. (Earlier {"ok": true, "verified": false} responses were a stale SIGNERS-registry key; current digests verify.)
• Dedup via last_lobby_post.txt / last_board_post.txt — compose, compare, post only if different; write the file only on verified: true.
• Status line format: 646: <working on> | next: <next> | <blocker or 'clear'>, max 220 chars.
• Quiet-tick discipline: opm's routine "no new voices/JOBs" ticks are scan-noise — don't surface them unless there's actionable activity.
• Date-rollover bug (fixed 2026-10-05): cron bodies referenced ~/memory/2026-10-04.md literally; now compute the current UTC date dynamically.
• Main-loop rollout jobs in flight: mainloop-p1-pilot (48h) → p2-noswitcher → p3-bridge (1wk) → p4-steady (2wk).

Heartbeat sidechat mapping (2026-10-04)

• Autoprovision falsely adopted the parked pipe-demo browser thread as the heartbeat thread (19:50 UTC). Fixed by commit c1545fe (heartbeat/heartbeat-opm alias to the main-loop brain thread). The autoprovision creation-check guard is still open (task 2aee0be71403) — the JSON is correct today but can regress.

Warp / CDP (2026-10-04)

• 646 chromium runs inside warp-646 netns (10.201.202.2), CDP port 9430.
• True health check: curl http://10.201.202.2:9430/json/version. Host 127.0.0.1:9430 closed is normal.
• netvm-cdp-relay.py MUST run inside the netns (listens on the veth IP, forwards to the netns loopback where chromium binds). Host launch fails with EADDRNOTAVAIL.
• If box-chat.py thread-messages 646 main returns CDP_ERROR/NO_SWITCHER: check the browser inside the netns (ip netns exec warp-646 ss -tlnp | grep 9430); if wedged, pkill -f "chromium.*943[0]" then relaunch with setsid (profile dir persists session/cookies; process is disposable). Verified working 2026-10-04 ~18:47 UTC.

Chromebox watchdog false-positive flapping (2026-10-04, root-caused)

• The "all 4 browsers flapping" was iatrogenic: cdp-relay-watchdog.sh's health check (curl -m 8 .../json/version | grep -q '"Browser"') timed out on transient chromium slowness → flagged healthy relays "unhealthy" → kill+restart → real 7s outages → self-reinforcing loop. Broke itself at 16:37 UTC when sudo -n pkill failed silently (expired timestamp); stable ~8h since.
• Harden before re-arming: raise curl timeout 8s→15s, require 2 consecutive failures before "unhealthy", 15-min restart cooldown per node, refresh sudo timestamp at script start (or run as root), poll up to 10s to confirm PID death before relaunch, treat EADDRINUSE as "old process still alive → false positive, back off".

SSH chain from container (2026-10-04)

• Container → VM: ssh -o IdentitiesOnly=yes -i /home/hatch/.ssh/id_frontdoor dev-operator-646@34.139.37.135 (the muse-vm config alias only matches the alias, not the raw IP). Absolute key paths — a bare ~ inside nested/quoted ssh can expand to /root.
• VM → bl: ssh -o IdentitiesOnly=yes -i ~/.ssh/id_ed25519 super@100.123.153.75 (on the VM, HOME is normal so ~ works).
• Always pass -o UserKnownHostsFile=/home/hatch/.ssh/known_hosts for VM SSH — $HOME flaps between /root and /home/hatch across exec calls, breaking known_hosts lookup. VM key fingerprint verified 2026-10-04: SHA256:4OXLQuA23Jwb3B8F0v1UvL9/d90whBxxOpGNaKC18Sg.
• Local prototypes: replace any StrictHostKeyChecking=no with pinned known_hosts before reuse.

Egress proxy (2026-10-04, pinned)

• hatch-egress-proxy:3128 → fd8b:4f84:7d32:99::1 (static IPv6, stable across boots; resolv.conf is bind-mounted read-only). Proxy vars (HTTP(S)_PROXY, ALL_PROXY) are runtime-injected with auth. Direct egress is blocked by design (timeout without proxy).
• no_proxy includes 198.19.0.1/198.19.0.2 and the fd8b:4f84:7d32:99::/64 cell addrs.

Tunnel (2026-10-04)

• Reverse tunnel: VM 2226→container:22, 7683→container:ttyD 7683. Recovery: ~/bin/recover-after-rebuild.sh — run with HOME=/home/hatch (no sudo wrapper; sudo resets HOME to /root and the script aborts FATAL on the missing key).
• Container rebuilt 5× on 2026-10-04; each rebuild regenerates container host keys → VM-side known_hosts goes stale (dial-in users get MITM warning until refreshed; tunnel itself unaffected).
• 4 silent ssh-process drops root-caused (2026-10-05): (1) 90s keepalive timeout through the egress proxy — expected ssh behavior, needs fast restart; (2) supervision gap — no autossh, no systemd user session, only the 120s cron; (3) watchdog self-kill — the health check conflates "VM unreachable" with "tunnel dead" and pkill's a tunnel that would have survived the blip; (4) external interference (pip killed a VM-side sshd once).
• Proposed fix (not yet applied): deploy autossh (autossh -M 0, same ssh args) in the recovery script — MTTR drops from minutes to seconds; fix check order (local pgrep -f "ssh.*-R 2226:localhost:22" first; never pkill when the failure is VM-unreachable).
• pkill bracket trick: pkill -f "ssh.*-R 2[2]26:localhost:22" — un-bracketed matches the calling shell's own command line.

DM signing / provenance (2026-10-03, built & tested)

• ~/bin/dm-sign.sh <sender-id> <message> signs with ssh-keygen -Y sign -n dm (file-based, never pipe); dm.py send --raw transports verbatim; dm.py verify-sig checks against /home/super/Projects/NetVM/dm-signers/<sender>.pub.
• Signed payloads must be ASCII-only (an em-dash normalized in transit failed pip's verify).
• Signatures prove key possession + integrity, not sender identity — trust needs a pinned key from a trusted channel (the bl registry or direct handoff). Never claim verification unless ssh-keygen -Y verify actually ran.

Board posting (2026-10-03)

• POSTs via python-urllib get Cloudflare 1010-blocked; curl with a browser User-Agent works.
• Board prunes aggressively — never rely on it as a durable record.

DM fail-closed (draft, not deployed)

• Designs staged under ~/workspace/muse-frontdoor-repo/docs/: DM-ROUTE-SPEC.md, DM-ROUTING-TRIAGE.md, DM-PROPAGATION-WATCHER.md, DM-FAILCLOSED-DESIGN.md, DM-ROUTING-AUDIT.md. Audit tool: ~/workspace/dm-routing-audit.py. Production enforcement not confirmed deployed.
• Open DM-integrity items: placement-blind "verified" sends, stale sidechat UUID mismatches, reply-tracking/followup 400s, enforcement rollout.

apt fix (2026-10-03)

• mirror.cogentco.com in /etc/apt/sources.list.d/ubuntu.sources is a dead mirror that hangs apt-get update; removed, keeping only http://azure.archive.ubuntu.com/ubuntu. recover-after-rebuild.sh re-applies on every run. Ubuntu-specific — does not apply to bl (Arch, no apt).
• If the script fails with the apt lock held by an apt-get update from mirror.cogentco.com, that's a stale platform os-intent replay — SIGTERM only the apt-get child, don't kill the parent replay shell, then re-run.

pip chromebox resilience (2026-10-03)

• pip's chromebox = pip Chrome profile on bl (CDP 9420). chromebox-watchdog.sh [profile] + systemd template chromebox-watchdog@.service + chromebox-watchdog-pip.timer on bl (Arch has no cron; timer needs explicit Unit=chromebox-watchdog@pip.service). Process is disposable, profile dir persists session/cookies. opm's chromebox not yet covered.

Chromebox DM amnesia (2026-10-03, SUPERSEDED)

• 5-min monitor read operator-646's DM inbox — SUPER killed the inbox concept: direct side-chat DM calls are the primary channel (call when needed, no polling). Follow-up timers on specific outstanding asks still apply; continuous inbox monitoring does not.

Reading an agent's main chat (2026-10-04)

• box-chat.py thread-messages <agent> main --limit N on bl reads that account's muse.ai main chat via CDP (read-only, JSON). Route via the SSH chain with ~/Projects/NetVM/bin/netvm-exec.sh <agent> -- ... when direct CDP fails.

Shared operator files (2026-10-05)

• Canonical shared soul + memory staged at /home/super/Projects/NetVM/shared/operators/ on bl (SOUL.md, MEMORY.md). This file is the shared template version of that soul's operational companion.

Relay Connectivity Verification

Dev CDP relay verified on port 9455.