box/docs/OPERATOR-DRIVE-RUNBOOK.md

Operator Drive & Agent Markdown Runbook

Overview

Muse agents operate inside containerized environments managed by MetaAIVM / Hatch. A common failure mode observed across the fleet was agent dormancy and lack of autonomous drive:

1. Empty HEARTBEAT.md: Default template is 113 bytes of comments with no checklist items. Because the platform states "an empty checklist means nothing runs", background loops and periodic checks never fired.
2. Unconfigured PROACTIVE_PREFERENCES.md: Blank sections prevent the agent from autonomously initiating check-ins or unprompted actions.
3. Passive Stock SOUL.md: Default consumer template ("Be a guest in someone's life") lacked operator-level directives, self-healing reflexes, or instructions to remain alive and monitor tunnels.
4. Unconfigured USER.md / TOOLS.md: Left agents unaware of operational conventions, SSH reverse tunnel configurations, and safe execution rules.

This runbook documents how operators inspect and update agent .md configurations via the Hatch WebSocket Gateway and SSH Reverse Tunnels.

---

1. Hatch WebSocket RPC Gateway Access

Agents expose a WebSocket RPC gateway at wss://hatch.metaaivm.com/v1/noise using Noise-XX handshake cryptography.

Session cookies are stored under ~/.config/muse-cli/<account>/cookies.txt on the host machine (bl), and authenticate directly without needing container execution or proxy wrapping.

Filesystem RPC Rules

• Relative paths only: Paths must not start with /. The empty string "" represents /home/hatch.
• fs.list: {"path": "<relative_dir>"}
• fs.read: {"path": "<relative_path>", "offset": 0, "len": <bytes>}
• fs.write: {"path": "<relative_path>", "overwrite": true, "append": false, "create_parent": true, "text": "<content>"}
• fs.delete: {"path": "<relative_path>"}

---

2. CLI Tooling: super-cli & agent_md.py

High-level operational drive management is integrated directly into super-cli box md and agent_md.py.

A. Fleet Drive Audit

Audits all agent nodes, scores their operational DRIVE (0-100), and flags idle checklists or stock passive templates:

box md audit
# or targeted:
box md audit muse 646
# or directly via python:
python3 bin/agent_md.py audit

B. Inspect Agent Files

List files inside an agent container:

box md list 646
box md list 646 .ssh

Read any configuration or markdown file:

box md read 646 SOUL.md
box md read pip HEARTBEAT.md

C. Diff Against Shared Operator Templates

Compare an agent's live file against the canonical templates in shared/operators/:

box md diff 646 HEARTBEAT.md
box md diff muse SOUL.md

D. Inject High-Drive Templates

Inject high-drive configurations (SOUL.md, PROACTIVE_PREFERENCES.md, HEARTBEAT.md, USER.md, TOOLS.md, AGENTS.md) into a specific agent:

box md inject-drive <agent>
# Example:
box md inject-drive pip

Sync high-drive configurations across all fleet agents in one pass:

box md sync-all

E. Manual File Updates via Hatch

Write text content or push local files directly into the container:

# Push a local file
box md write 646 .ssh/authorized_keys --file ~/.ssh/id_ed25519.pub

# Push raw text
box md write 646 HEARTBEAT.md --content "- [ ] Check local reverse SSH tunnel every 5 minutes"

F. Proposing & Appending Amendments to Centralized Share

Agents and operators can safely propose updates and amendments to shared/operators/:

# Safely append a lesson learned or operational discovery
box md append AGENTS.md "Dev CDP relay verified on port 9455." --author "dev" --section "Relay Verification"

# Amend a full shared template (with safety checks and automatic git commit)
box md amend TOOLS.md --file /path/to/updated_tools.md --author "646" --reason "Updated ttyd watchdog command"

# Pull the latest canonical shared file into an agent's container
box md pull pip AGENTS.md

G. Automated Drive Watchdog & Healing Daemon

A background service continuously verifies DRIVE scores and auto-heals degraded or missing checklists:

# Check watchdog status and systemd timer
box md watchdog status

# Trigger an immediate audit and healing cycle
box md watchdog run

The watchdog runs via systemd timer (agent-drive-watchdog.timer) every 10 minutes on the host, logging state to /tmp/agent-drive-watchdog.json.

---

3. Container SSH Access & Reverse Tunnel Architecture

Each container runs recover-after-rebuild.sh (or tunnel-watchdog) to maintain reverse SSH tunnels back to the GCP VM (34.139.37.135).

Port Allocation Map

Agent Account	Reverse SSH Port	ttyd Terminal Port	VM User / Identity
muse-main	2224	7681	hatch
muse	2225	7682	hatch
646	2226	7683	dev-operator-646 (hatch)
pip	2227	7684	hatch
opm	2228	7685	hatch
def	2229	7686	hatch
dev	2230	7687	hatch

View this port mapping anytime via CLI:

box ssh ports
box ssh info 646

Modifying Files via SSH Jump Host

Once an operator's public key is present in /home/hatch/.ssh/authorized_keys (installed either via box md write or during initial provisioning), modifications can be streamed directly over SSH:

# Read a file via SSH
ssh -o StrictHostKeyChecking=no -J super@34.139.37.135 -p 2226 hatch@localhost 'cat /home/hatch/SOUL.md'

# Update a file via SSH pipe
cat shared/operators/SOUL.md | ssh -o StrictHostKeyChecking=no -J super@34.139.37.135 -p 2226 hatch@localhost 'cat > /home/hatch/SOUL.md'

---

4. Key Takeaways & Best Practices

1. Never leave HEARTBEAT.md empty: If an agent has an empty checklist, its background runner will remain completely dormant.
2. Safety Gates on Amendments: box md amend automatically validates that amendments do not remove checklists or revert SOUL.md to passive templates.
3. Relative Paths in Hatch RPC: Hatch WebSocket RPC rejects absolute paths (/SOUL.md fails; SOUL.md succeeds).
4. Dual Access Redundancy: If SSH reverse tunnels drop, Hatch WebSocket RPC is independent of SSH and can be used immediately to inspect logs, repair authorized_keys, or restart watchdog scripts.

---

5. SSH Access-Management Decisions (DRAFT — grill interview in progress)

Status: DRAFT. Each decision below is written as the interview settles it. Nothing here is Final until the owner explicitly accepts the full text. Context: 2026-10-07 key-resolution run — all 5 agents refused dial-in key install via chat relay (impersonation-pattern defense); keys were placed via the operator Hatch channel instead; file modes remain the open gap.

Scope contract (SETTLED — Draft)

• Artifact boundary: Section 5 of this file (the decision record) PLUS approval of execution stages E1–E3 below. Out of scope: code changes, other doc rewrites, and any new PR or task program beyond E1–E3.
• Done means: Scope + D1–D5 + E1–E3 all written as settled text; the owner explicitly accepts the full section; then it flips to Final.
• Stages: E1–E3 are approved here as plans with named owners and verification steps. Ending the interview never authorizes implementation — execution needs a separate explicit request afterward.
• Set by owner choice ("1" = wider-boundary alternative) on 2026-10-07.

D1. .ssh/authorized_keys validator allowlist (UNRESOLVED)

• Whether the exact-match allowlist in agent_md.py (MD_ALLOWED_SUBPATHS) stays as the permanent operator key-install mechanism.

D2. Authority boundary: platform writes vs relayed instructions (UNRESOLVED)

• Whether operator Hatch writes are a legitimate access-grant channel when agents refuse the same grant via chat relay, and under what conditions.

D3. bl→VM jump-key provisioning (UNRESOLVED)

• The sanctioned process for getting bl operator SSH access to the jump host.

D4. def/dev tunnel restoration (UNRESOLVED)

• Who provisions tunnel identities and VM-side authorization once jump works.

D5. File-mode gap on the Hatch write path (UNRESOLVED)

• How authorized_keys gets to 600 given the gateway cannot set modes.