box/docs/OPERATOR-DRIVE-RUNBOOK.md

# Operator Drive & Agent Markdown Runbook

## Overview

Muse agents operate inside containerized environments managed by MetaAIVM / Hatch. A common failure mode observed across the fleet was agent dormancy and lack of autonomous drive:
1. **Empty `HEARTBEAT.md`**: Default template is 113 bytes of comments with no checklist items. Because the platform states *"an empty checklist means nothing runs"*, background loops and periodic checks never fired.
2. **Unconfigured `PROACTIVE_PREFERENCES.md`**: Blank sections prevent the agent from autonomously initiating check-ins or unprompted actions.
3. **Passive Stock `SOUL.md`**: Default consumer template (*"Be a guest in someone's life"*) lacked operator-level directives, self-healing reflexes, or instructions to remain alive and monitor tunnels.
4. **Unconfigured `USER.md` / `TOOLS.md`**: Left agents unaware of operational conventions, SSH reverse tunnel configurations, and safe execution rules.

This runbook documents how operators inspect and update agent `.md` configurations via the **Hatch WebSocket Gateway** and **SSH Reverse Tunnels**.

---

## 1. Hatch WebSocket RPC Gateway Access

Agents expose a WebSocket RPC gateway at `wss://hatch.metaaivm.com/v1/noise` using Noise-XX handshake cryptography.

Session cookies are stored under `~/.config/muse-cli/<account>/cookies.txt` on the host machine (`bl`), and authenticate directly without needing container execution or proxy wrapping.

### Filesystem RPC Rules
- **Relative paths only**: Paths must not start with `/`. The empty string `""` represents `/home/hatch`.
- **fs.list**: `{"path": "<relative_dir>"}`
- **fs.read**: `{"path": "<relative_path>", "offset": 0, "len": <bytes>}`
- **fs.write**: `{"path": "<relative_path>", "overwrite": true, "append": false, "create_parent": true, "text": "<content>"}`
- **fs.delete**: `{"path": "<relative_path>"}`

---

## 2. CLI Tooling: `super-cli` & `agent_md.py`

High-level operational drive management is integrated directly into `super-cli box md` and `agent_md.py`.

### A. Fleet Drive Audit
Audits all agent nodes, scores their operational DRIVE (0-100), and flags idle checklists or stock passive templates:
```bash
box md audit
# or targeted:
box md audit muse 646
# or directly via python:
python3 bin/agent_md.py audit
```

### B. Inspect Agent Files
List files inside an agent container:
```bash
box md list 646
box md list 646 .ssh
```

Read any configuration or markdown file:
```bash
box md read 646 SOUL.md
box md read pip HEARTBEAT.md
```

### C. Diff Against Shared Operator Templates
Compare an agent's live file against the canonical templates in `shared/operators/`:
```bash
box md diff 646 HEARTBEAT.md
box md diff muse SOUL.md
```

### D. Inject High-Drive Templates
Inject high-drive configurations (`SOUL.md`, `PROACTIVE_PREFERENCES.md`, `HEARTBEAT.md`, `USER.md`, `TOOLS.md`, `AGENTS.md`) into a specific agent:
```bash
box md inject-drive <agent>
# Example:
box md inject-drive pip
```

Sync high-drive configurations across all fleet agents in one pass:
```bash
box md sync-all
```

### E. Manual File Updates via Hatch
Write text content or push local files directly into the container:
```bash
# Push a local file
box md write 646 .ssh/authorized_keys --file ~/.ssh/id_ed25519.pub

# Push raw text
box md write 646 HEARTBEAT.md --content "- [ ] Check local reverse SSH tunnel every 5 minutes"
```

### F. Proposing & Appending Amendments to Centralized Share
Agents and operators can safely propose updates and amendments to `shared/operators/`:
```bash
# Safely append a lesson learned or operational discovery
box md append AGENTS.md "Dev CDP relay verified on port 9455." --author "dev" --section "Relay Verification"

# Amend a full shared template (with safety checks and automatic git commit)
box md amend TOOLS.md --file /path/to/updated_tools.md --author "646" --reason "Updated ttyd watchdog command"

# Pull the latest canonical shared file into an agent's container
box md pull pip AGENTS.md
```

### G. Automated Drive Watchdog & Healing Daemon
A background service continuously verifies DRIVE scores and auto-heals degraded or missing checklists:
```bash
# Check watchdog status and systemd timer
box md watchdog status

# Trigger an immediate audit and healing cycle
box md watchdog run
```
The watchdog runs via systemd timer (`agent-drive-watchdog.timer`) every 10 minutes on the host, logging state to `/tmp/agent-drive-watchdog.json`.

---

## 3. Container SSH Access & Reverse Tunnel Architecture

Each container runs `recover-after-rebuild.sh` (or `tunnel-watchdog`) to maintain reverse SSH tunnels back to the GCP VM (`34.139.37.135`).

### Port Allocation Map

| Agent Account | Reverse SSH Port | ttyd Terminal Port | VM User / Identity |
| :--- | :--- | :--- | :--- |
| `muse-main` | 2224 | 7681 | `hatch` |
| `muse` | 2225 | 7682 | `hatch` |
| `646` | 2226 | 7683 | `dev-operator-646` (`hatch`) |
| `pip` | 2227 | 7684 | `hatch` |
| `opm` | 2228 | 7685 | `hatch` |
| `def` | 2229 | 7686 | `hatch` |
| `dev` | 2230 | 7687 | `hatch` |

View this port mapping anytime via CLI:
```bash
box ssh ports
box ssh info 646
```

### Modifying Files via SSH Jump Host
Once an operator's public key is present in `/home/hatch/.ssh/authorized_keys` (installed either via `box md write` or during initial provisioning), modifications can be streamed directly over SSH:

```bash
# Read a file via SSH
ssh -o StrictHostKeyChecking=no -J super@34.139.37.135 -p 2226 hatch@localhost 'cat /home/hatch/SOUL.md'

# Update a file via SSH pipe
cat shared/operators/SOUL.md | ssh -o StrictHostKeyChecking=no -J super@34.139.37.135 -p 2226 hatch@localhost 'cat > /home/hatch/SOUL.md'
```

---

## 4. Key Takeaways & Best Practices
1. **Never leave `HEARTBEAT.md` empty**: If an agent has an empty checklist, its background runner will remain completely dormant.
2. **Safety Gates on Amendments**: `box md amend` automatically validates that amendments do not remove checklists or revert `SOUL.md` to passive templates.
3. **Relative Paths in Hatch RPC**: Hatch WebSocket RPC rejects absolute paths (`/SOUL.md` fails; `SOUL.md` succeeds).
4. **Dual Access Redundancy**: If SSH reverse tunnels drop, Hatch WebSocket RPC is independent of SSH and can be used immediately to inspect logs, repair `authorized_keys`, or restart watchdog scripts.

---

## 5. SSH Access-Management Decisions (DRAFT — grill interview in progress)

> Status: DRAFT. Each decision below is written as the interview settles it.
> Nothing here is Final until the owner explicitly accepts the full text.
> Context: 2026-10-07 key-resolution run — all 5 agents refused dial-in key
> install via chat relay (impersonation-pattern defense); keys were placed
> via the operator Hatch channel instead; file modes remain the open gap.

### Scope contract (SETTLED — Draft)
- **Artifact boundary**: Section 5 of this file (the decision record) PLUS
  approval of execution stages E1–E3 below. Out of scope: code changes,
  other doc rewrites, and any new PR or task program beyond E1–E3.
- **Done means**: Scope + D1–D5 + E1–E3 all written as settled text; the
  owner explicitly accepts the full section; then it flips to Final.
- **Stages**: E1–E3 are approved here as plans with named owners and
  verification steps. Ending the interview never authorizes
  implementation — execution needs a separate explicit request afterward.
- Set by owner choice ("1" = wider-boundary alternative) on 2026-10-07.

### D1. `.ssh/authorized_keys` validator allowlist (UNRESOLVED)
- Whether the exact-match allowlist in `agent_md.py` (`MD_ALLOWED_SUBPATHS`)
  stays as the permanent operator key-install mechanism.

### D2. Authority boundary: platform writes vs relayed instructions (UNRESOLVED)
- Whether operator Hatch writes are a legitimate access-grant channel when
  agents refuse the same grant via chat relay, and under what conditions.

### D3. bl→VM jump-key provisioning (UNRESOLVED)
- The sanctioned process for getting bl operator SSH access to the jump host.

### D4. def/dev tunnel restoration (UNRESOLVED)
- Who provisions tunnel identities and VM-side authorization once jump works.

### D5. File-mode gap on the Hatch write path (UNRESOLVED)
- How `authorized_keys` gets to 600 given the gateway cannot set modes.