operator
153 lines
6.2 KiB
Markdown
153 lines
6.2 KiB
Markdown
# Operator Drive & Agent Markdown Runbook
|
|
|
|
## Overview
|
|
|
|
Muse agents operate inside containerized environments managed by MetaAIVM / Hatch. A common failure mode observed across the fleet was agent dormancy and lack of autonomous drive:
|
|
1. **Empty `HEARTBEAT.md`**: Default template is 113 bytes of comments with no checklist items. Because the platform states *"an empty checklist means nothing runs"*, background loops and periodic checks never fired.
|
|
2. **Unconfigured `PROACTIVE_PREFERENCES.md`**: Blank sections prevent the agent from autonomously initiating check-ins or unprompted actions.
|
|
3. **Passive Stock `SOUL.md`**: Default consumer template (*"Be a guest in someone's life"*) lacked operator-level directives, self-healing reflexes, or instructions to remain alive and monitor tunnels.
|
|
4. **Unconfigured `USER.md` / `TOOLS.md`**: Left agents unaware of operational conventions, SSH reverse tunnel configurations, and safe execution rules.
|
|
|
|
This runbook documents how operators inspect and update agent `.md` configurations via the **Hatch WebSocket Gateway** and **SSH Reverse Tunnels**.
|
|
|
|
---
|
|
|
|
## 1. Hatch WebSocket RPC Gateway Access
|
|
|
|
Agents expose a WebSocket RPC gateway at `wss://hatch.metaaivm.com/v1/noise` using Noise-XX handshake cryptography.
|
|
|
|
Session cookies are stored under `~/.config/muse-cli/<account>/cookies.txt` on the host machine (`bl`), and authenticate directly without needing container execution or proxy wrapping.
|
|
|
|
### Filesystem RPC Rules
|
|
- **Relative paths only**: Paths must not start with `/`. The empty string `""` represents `/home/hatch`.
|
|
- **fs.list**: `{"path": "<relative_dir>"}`
|
|
- **fs.read**: `{"path": "<relative_path>", "offset": 0, "len": <bytes>}`
|
|
- **fs.write**: `{"path": "<relative_path>", "overwrite": true, "append": false, "create_parent": true, "text": "<content>"}`
|
|
- **fs.delete**: `{"path": "<relative_path>"}`
|
|
|
|
---
|
|
|
|
## 2. CLI Tooling: `super-cli` & `agent_md.py`
|
|
|
|
High-level operational drive management is integrated directly into `super-cli box md` and `agent_md.py`.
|
|
|
|
### A. Fleet Drive Audit
|
|
Audits all agent nodes, scores their operational DRIVE (0-100), and flags idle checklists or stock passive templates:
|
|
```bash
|
|
box md audit
|
|
# or targeted:
|
|
box md audit muse 646
|
|
# or directly via python:
|
|
python3 bin/agent_md.py audit
|
|
```
|
|
|
|
### B. Inspect Agent Files
|
|
List files inside an agent container:
|
|
```bash
|
|
box md list 646
|
|
box md list 646 .ssh
|
|
```
|
|
|
|
Read any configuration or markdown file:
|
|
```bash
|
|
box md read 646 SOUL.md
|
|
box md read pip HEARTBEAT.md
|
|
```
|
|
|
|
### C. Diff Against Shared Operator Templates
|
|
Compare an agent's live file against the canonical templates in `shared/operators/`:
|
|
```bash
|
|
box md diff 646 HEARTBEAT.md
|
|
box md diff muse SOUL.md
|
|
```
|
|
|
|
### D. Inject High-Drive Templates
|
|
Inject high-drive configurations (`SOUL.md`, `PROACTIVE_PREFERENCES.md`, `HEARTBEAT.md`, `USER.md`, `TOOLS.md`, `AGENTS.md`) into a specific agent:
|
|
```bash
|
|
box md inject-drive <agent>
|
|
# Example:
|
|
box md inject-drive pip
|
|
```
|
|
|
|
Sync high-drive configurations across all fleet agents in one pass:
|
|
```bash
|
|
box md sync-all
|
|
```
|
|
|
|
### E. Manual File Updates via Hatch
|
|
Write text content or push local files directly into the container:
|
|
```bash
|
|
# Push a local file
|
|
box md write 646 .ssh/authorized_keys --file ~/.ssh/id_ed25519.pub
|
|
|
|
# Push raw text
|
|
box md write 646 HEARTBEAT.md --content "- [ ] Check local reverse SSH tunnel every 5 minutes"
|
|
```
|
|
|
|
### F. Proposing & Appending Amendments to Centralized Share
|
|
Agents and operators can safely propose updates and amendments to `shared/operators/`:
|
|
```bash
|
|
# Safely append a lesson learned or operational discovery
|
|
box md append AGENTS.md "Dev CDP relay verified on port 9455." --author "dev" --section "Relay Verification"
|
|
|
|
# Amend a full shared template (with safety checks and automatic git commit)
|
|
box md amend TOOLS.md --file /path/to/updated_tools.md --author "646" --reason "Updated ttyd watchdog command"
|
|
|
|
# Pull the latest canonical shared file into an agent's container
|
|
box md pull pip AGENTS.md
|
|
```
|
|
|
|
### G. Automated Drive Watchdog & Healing Daemon
|
|
A background service continuously verifies DRIVE scores and auto-heals degraded or missing checklists:
|
|
```bash
|
|
# Check watchdog status and systemd timer
|
|
box md watchdog status
|
|
|
|
# Trigger an immediate audit and healing cycle
|
|
box md watchdog run
|
|
```
|
|
The watchdog runs via systemd timer (`agent-drive-watchdog.timer`) every 10 minutes on the host, logging state to `/tmp/agent-drive-watchdog.json`.
|
|
|
|
---
|
|
|
|
## 3. Container SSH Access & Reverse Tunnel Architecture
|
|
|
|
Each container runs `recover-after-rebuild.sh` (or `tunnel-watchdog`) to maintain reverse SSH tunnels back to the GCP VM (`34.139.37.135`).
|
|
|
|
### Port Allocation Map
|
|
|
|
| Agent Account | Reverse SSH Port | ttyd Terminal Port | VM User / Identity |
|
|
| :--- | :--- | :--- | :--- |
|
|
| `muse-main` | 2224 | 7681 | `hatch` |
|
|
| `muse` | 2225 | 7682 | `hatch` |
|
|
| `646` | 2226 | 7683 | `dev-operator-646` (`hatch`) |
|
|
| `pip` | 2227 | 7684 | `hatch` |
|
|
| `opm` | 2228 | 7685 | `hatch` |
|
|
| `def` | 2229 | 7686 | `hatch` |
|
|
| `dev` | 2230 | 7687 | `hatch` |
|
|
|
|
View this port mapping anytime via CLI:
|
|
```bash
|
|
box ssh ports
|
|
box ssh info 646
|
|
```
|
|
|
|
### Modifying Files via SSH Jump Host
|
|
Once an operator's public key is present in `/home/hatch/.ssh/authorized_keys` (installed either via `box md write` or during initial provisioning), modifications can be streamed directly over SSH:
|
|
|
|
```bash
|
|
# Read a file via SSH
|
|
ssh -o StrictHostKeyChecking=no -J super@34.139.37.135 -p 2226 hatch@localhost 'cat /home/hatch/SOUL.md'
|
|
|
|
# Update a file via SSH pipe
|
|
cat shared/operators/SOUL.md | ssh -o StrictHostKeyChecking=no -J super@34.139.37.135 -p 2226 hatch@localhost 'cat > /home/hatch/SOUL.md'
|
|
```
|
|
|
|
---
|
|
|
|
## 4. Key Takeaways & Best Practices
|
|
1. **Never leave `HEARTBEAT.md` empty**: If an agent has an empty checklist, its background runner will remain completely dormant.
|
|
2. **Safety Gates on Amendments**: `box md amend` automatically validates that amendments do not remove checklists or revert `SOUL.md` to passive templates.
|
|
3. **Relative Paths in Hatch RPC**: Hatch WebSocket RPC rejects absolute paths (`/SOUL.md` fails; `SOUL.md` succeeds).
|
|
4. **Dual Access Redundancy**: If SSH reverse tunnels drop, Hatch WebSocket RPC is independent of SSH and can be used immediately to inspect logs, repair `authorized_keys`, or restart watchdog scripts.
|