feat(operators): agent markdown drive management via Hatch and SSH runbook

2026-10-05 16:51:56 +00:00
parent 653166e202
commit cc8702b38c
12 changed files with 1540 additions and 3 deletions
@@ -0,0 +1,127 @@
+# Operator Drive & Agent Markdown Runbook
+
+## Overview
+
+Muse agents operate inside containerized environments managed by MetaAIVM / Hatch. A common failure mode observed across the fleet was agent dormancy and lack of autonomous drive:
+1. **Empty `HEARTBEAT.md`**: Default template is 113 bytes of comments with no checklist items. Because the platform states *"an empty checklist means nothing runs"*, background loops and periodic checks never fired.
+2. **Unconfigured `PROACTIVE_PREFERENCES.md`**: Blank sections prevent the agent from autonomously initiating check-ins or unprompted actions.
+3. **Passive Stock `SOUL.md`**: Default consumer template (*"Be a guest in someone's life"*) lacked operator-level directives, self-healing reflexes, or instructions to remain alive and monitor tunnels.
+4. **Unconfigured `USER.md` / `TOOLS.md`**: Left agents unaware of operational conventions, SSH reverse tunnel configurations, and safe execution rules.
+
+This runbook documents how operators inspect and update agent `.md` configurations via the **Hatch WebSocket Gateway** and **SSH Reverse Tunnels**.
+
+---
+
+## 1. Hatch WebSocket RPC Gateway Access
+
+Agents expose a WebSocket RPC gateway at `wss://hatch.metaaivm.com/v1/noise` using Noise-XX handshake cryptography.
+
+Session cookies are stored under `~/.config/muse-cli/<account>/cookies.txt` on the host machine (`bl`), and authenticate directly without needing container execution or proxy wrapping.
+
+### Filesystem RPC Rules
+- **Relative paths only**: Paths must not start with `/`. The empty string `""` represents `/home/hatch`.
+- **fs.list**: `{"path": "<relative_dir>"}`
+- **fs.read**: `{"path": "<relative_path>", "offset": 0, "len": <bytes>}`
+- **fs.write**: `{"path": "<relative_path>", "overwrite": true, "append": false, "create_parent": true, "text": "<content>"}`
+- **fs.delete**: `{"path": "<relative_path>"}`
+
+---
+
+## 2. CLI Tooling: `super-cli` & `agent_md.py`
+
+High-level operational drive management is integrated directly into `super-cli box md` and `agent_md.py`.
+
+### A. Fleet Drive Audit
+Audits all agent nodes, scores their operational DRIVE (0-100), and flags idle checklists or stock passive templates:
+```bash
+box md audit
+# or targeted:
+box md audit muse 646
+# or directly via python:
+python3 bin/agent_md.py audit
+```
+
+### B. Inspect Agent Files
+List files inside an agent container:
+```bash
+box md list 646
+box md list 646 .ssh
+```
+
+Read any configuration or markdown file:
+```bash
+box md read 646 SOUL.md
+box md read pip HEARTBEAT.md
+```
+
+### C. Diff Against Shared Operator Templates
+Compare an agent's live file against the canonical templates in `shared/operators/`:
+```bash
+box md diff 646 HEARTBEAT.md
+box md diff muse SOUL.md
+```
+
+### D. Inject High-Drive Templates
+Inject high-drive configurations (`SOUL.md`, `PROACTIVE_PREFERENCES.md`, `HEARTBEAT.md`, `USER.md`, `TOOLS.md`, `AGENTS.md`) into a specific agent:
+```bash
+box md inject-drive <agent>
+# Example:
+box md inject-drive pip
+```
+
+Sync high-drive configurations across all fleet agents in one pass:
+```bash
+box md sync-all
+```
+
+### E. Manual File Updates via Hatch
+Write text content or push local files directly into the container:
+```bash
+# Push a local file
+box md write 646 .ssh/authorized_keys --file ~/.ssh/id_ed25519.pub
+
+# Push raw text
+box md write 646 HEARTBEAT.md --content "- [ ] Check local reverse SSH tunnel every 5 minutes"
+```
+
+---
+
+## 3. Container SSH Access & Reverse Tunnel Architecture
+
+Each container runs `recover-after-rebuild.sh` (or `tunnel-watchdog`) to maintain reverse SSH tunnels back to the GCP VM (`34.139.37.135`).
+
+### Port Allocation Map
+
+| Agent Account | Reverse SSH Port | ttyd Terminal Port | VM User / Identity |
+| :--- | :--- | :--- | :--- |
+| `muse-main` | 2224 | 7681 | `hatch` |
+| `muse` | 2225 | 7682 | `hatch` |
+| `646` | 2226 | 7683 | `dev-operator-646` (`hatch`) |
+| `pip` | 2227 | 7684 | `hatch` |
+| `opm` | 2228 | 7685 | `hatch` |
+| `def` | 2229 | 7686 | `hatch` |
+| `dev` | 2230 | 7687 | `hatch` |
+
+View this port mapping anytime via CLI:
+```bash
+box ssh ports
+box ssh info 646
+```
+
+### Modifying Files via SSH Jump Host
+Once an operator's public key is present in `/home/hatch/.ssh/authorized_keys` (installed either via `box md write` or during initial provisioning), modifications can be streamed directly over SSH:
+
+```bash
+# Read a file via SSH
+ssh -o StrictHostKeyChecking=no -J super@34.139.37.135 -p 2226 hatch@localhost 'cat /home/hatch/SOUL.md'
+
+# Update a file via SSH pipe
+cat shared/operators/SOUL.md | ssh -o StrictHostKeyChecking=no -J super@34.139.37.135 -p 2226 hatch@localhost 'cat > /home/hatch/SOUL.md'
+```
+
+---
+
+## 4. Key Takeaways & Best Practices
+1. **Never leave `HEARTBEAT.md` empty**: If an agent has an empty checklist, its background runner will remain completely dormant.
+2. **Relative Paths in Hatch RPC**: Hatch WebSocket RPC rejects absolute paths (`/SOUL.md` fails; `SOUL.md` succeeds).
+3. **Dual Access Redundancy**: If SSH reverse tunnels drop, Hatch WebSocket RPC is independent of SSH and can be used immediately to inspect logs, repair `authorized_keys`, or restart watchdog scripts.