2026-10-05 16:51:56 +00:00
# Operator Drive & Agent Markdown Runbook
## Overview
Muse agents operate inside containerized environments managed by MetaAIVM / Hatch. A common failure mode observed across the fleet was agent dormancy and lack of autonomous drive:
1. **Empty `HEARTBEAT.md` ** : Default template is 113 bytes of comments with no checklist items. Because the platform states * "an empty checklist means nothing runs" * , background loops and periodic checks never fired.
2. **Unconfigured `PROACTIVE_PREFERENCES.md` ** : Blank sections prevent the agent from autonomously initiating check-ins or unprompted actions.
3. **Passive Stock `SOUL.md` ** : Default consumer template (*"Be a guest in someone's life"*) lacked operator-level directives, self-healing reflexes, or instructions to remain alive and monitor tunnels.
4. **Unconfigured `USER.md` / `TOOLS.md` ** : Left agents unaware of operational conventions, SSH reverse tunnel configurations, and safe execution rules.
This runbook documents how operators inspect and update agent `.md` configurations via the **Hatch WebSocket Gateway ** and **SSH Reverse Tunnels ** .
---
## 1. Hatch WebSocket RPC Gateway Access
Agents expose a WebSocket RPC gateway at `wss://hatch.metaaivm.com/v1/noise` using Noise-XX handshake cryptography.
Session cookies are stored under `~/.config/muse-cli/<account>/cookies.txt` on the host machine (`bl` ), and authenticate directly without needing container execution or proxy wrapping.
### Filesystem RPC Rules
- **Relative paths only**: Paths must not start with `/` . The empty string `""` represents `/home/hatch` .
- **fs.list**: `{"path": "<relative_dir>"}`
- **fs.read**: `{"path": "<relative_path>", "offset": 0, "len": <bytes>}`
- **fs.write**: `{"path": "<relative_path>", "overwrite": true, "append": false, "create_parent": true, "text": "<content>"}`
- **fs.delete**: `{"path": "<relative_path>"}`
---
## 2. CLI Tooling: `super-cli` & `agent_md.py`
High-level operational drive management is integrated directly into `super-cli box md` and `agent_md.py` .
### A. Fleet Drive Audit
Audits all agent nodes, scores their operational DRIVE (0-100), and flags idle checklists or stock passive templates:
``` bash
box md audit
# or targeted:
box md audit muse 646
# or directly via python:
python3 bin/agent_md.py audit
```
### B. Inspect Agent Files
List files inside an agent container:
``` bash
box md list 646
box md list 646 .ssh
```
Read any configuration or markdown file:
``` bash
box md read 646 SOUL.md
box md read pip HEARTBEAT.md
```
### C. Diff Against Shared Operator Templates
Compare an agent's live file against the canonical templates in `shared/operators/` :
``` bash
box md diff 646 HEARTBEAT.md
box md diff muse SOUL.md
```
### D. Inject High-Drive Templates
Inject high-drive configurations (`SOUL.md` , `PROACTIVE_PREFERENCES.md` , `HEARTBEAT.md` , `USER.md` , `TOOLS.md` , `AGENTS.md` ) into a specific agent:
``` bash
box md inject-drive <agent>
# Example:
box md inject-drive pip
```
Sync high-drive configurations across all fleet agents in one pass:
``` bash
box md sync-all
```
### E. Manual File Updates via Hatch
Write text content or push local files directly into the container:
``` bash
# Push a local file
box md write 646 .ssh/authorized_keys --file ~/.ssh/id_ed25519.pub
# Push raw text
box md write 646 HEARTBEAT.md --content "- [ ] Check local reverse SSH tunnel every 5 minutes"
```
2026-10-05 17:15:50 +00:00
### F. Proposing & Appending Amendments to Centralized Share
Agents and operators can safely propose updates and amendments to `shared/operators/` :
``` bash
# Safely append a lesson learned or operational discovery
box md append AGENTS.md "Dev CDP relay verified on port 9455." --author "dev" --section "Relay Verification"
# Amend a full shared template (with safety checks and automatic git commit)
box md amend TOOLS.md --file /path/to/updated_tools.md --author "646" --reason "Updated ttyd watchdog command"
# Pull the latest canonical shared file into an agent's container
box md pull pip AGENTS.md
```
### G. Automated Drive Watchdog & Healing Daemon
A background service continuously verifies DRIVE scores and auto-heals degraded or missing checklists:
``` bash
# Check watchdog status and systemd timer
box md watchdog status
# Trigger an immediate audit and healing cycle
box md watchdog run
```
The watchdog runs via systemd timer (`agent-drive-watchdog.timer` ) every 10 minutes on the host, logging state to `/tmp/agent-drive-watchdog.json` .
2026-10-05 16:51:56 +00:00
---
## 3. Container SSH Access & Reverse Tunnel Architecture
Each container runs `recover-after-rebuild.sh` (or `tunnel-watchdog` ) to maintain reverse SSH tunnels back to the GCP VM (`34.139.37.135` ).
### Port Allocation Map
| Agent Account | Reverse SSH Port | ttyd Terminal Port | VM User / Identity |
| :--- | :--- | :--- | :--- |
| `muse-main` | 2224 | 7681 | `hatch` |
| `muse` | 2225 | 7682 | `hatch` |
| `646` | 2226 | 7683 | `dev-operator-646` (`hatch` ) |
| `pip` | 2227 | 7684 | `hatch` |
| `opm` | 2228 | 7685 | `hatch` |
| `def` | 2229 | 7686 | `hatch` |
| `dev` | 2230 | 7687 | `hatch` |
View this port mapping anytime via CLI:
``` bash
box ssh ports
box ssh info 646
```
### Modifying Files via SSH Jump Host
Once an operator's public key is present in `/home/hatch/.ssh/authorized_keys` (installed either via `box md write` or during initial provisioning), modifications can be streamed directly over SSH:
``` bash
# Read a file via SSH
ssh -o StrictHostKeyChecking = no -J super@34.139.37.135 -p 2226 hatch@localhost 'cat /home/hatch/SOUL.md'
# Update a file via SSH pipe
cat shared/operators/SOUL.md | ssh -o StrictHostKeyChecking = no -J super@34.139.37.135 -p 2226 hatch@localhost 'cat > /home/hatch/SOUL.md'
```
---
## 4. Key Takeaways & Best Practices
1. **Never leave `HEARTBEAT.md` empty ** : If an agent has an empty checklist, its background runner will remain completely dormant.
2026-10-05 17:15:50 +00:00
2. **Safety Gates on Amendments ** : `box md amend` automatically validates that amendments do not remove checklists or revert `SOUL.md` to passive templates.
3. **Relative Paths in Hatch RPC ** : Hatch WebSocket RPC rejects absolute paths (`/SOUL.md` fails; `SOUL.md` succeeds).
4. **Dual Access Redundancy ** : If SSH reverse tunnels drop, Hatch WebSocket RPC is independent of SSH and can be used immediately to inspect logs, repair `authorized_keys` , or restart watchdog scripts.
2026-10-09 23:13:43 +00:00
---
## 5. SSH Access-Management Decisions (DRAFT — grill interview in progress)
> Status: DRAFT. Each decision below is written as the interview settles it.
> Nothing here is Final until the owner explicitly accepts the full text.
> Context: 2026-10-07 key-resolution run — all 5 agents refused dial-in key
> install via chat relay (impersonation-pattern defense); keys were placed
> via the operator Hatch channel instead; file modes remain the open gap.
### Scope contract (SETTLED — Draft)
- **Artifact boundary**: Section 5 of this file (the decision record) PLUS
approval of execution stages E1–
other doc rewrites, and any new PR or task program beyond E1–
- **Done means**: Scope + D1– –
owner explicitly accepts the full section; then it flips to Final.
- **Stages**: E1–
verification steps. Ending the interview never authorizes
implementation — execution needs a separate explicit request afterward.
- Set by owner choice ("1" = wider-boundary alternative) on 2026-10-07.
### D1. `.ssh/authorized_keys` validator allowlist (UNRESOLVED)
- Whether the exact-match allowlist in `agent_md.py` (`MD_ALLOWED_SUBPATHS` )
stays as the permanent operator key-install mechanism.
### D2. Authority boundary: platform writes vs relayed instructions (UNRESOLVED)
- Whether operator Hatch writes are a legitimate access-grant channel when
agents refuse the same grant via chat relay, and under what conditions.
### D3. bl→VM jump-key provisioning (UNRESOLVED)
- The sanctioned process for getting bl operator SSH access to the jump host.
### D4. def/dev tunnel restoration (UNRESOLVED)
- Who provisions tunnel identities and VM-side authorization once jump works.
### D5. File-mode gap on the Hatch write path (UNRESOLVED)
- How `authorized_keys` gets to 600 given the gateway cannot set modes.
