box/docs/LOOP-MANAGEMENT.md

# NetVM Intrinsic Loop Management — Operational Runbook & Architecture Specification

## 1. Executive Overview

NetVM coordinates an autonomous agent mesh (`muse`, `pip`, `646`, `opm`, `super`) communicating via inter-agent direct messages (DMs), scheduled jobs, and live browser sidechats. **Intrinsic Loops** represent communication cycles requiring closure (e.g. follow-ups, results, acknowledgements).

To prevent silent failures, stale deadlines, or rogue infinite nudging, NetVM provides **External Loop Management**:
- **Dual-Surface Architecture:** Real-time local CLI management on `bl` (`super` and `box` commands) synchronized with an operator Web Console on the Google Cloud VM (`https://box.muse-dev.online/`).
- **Dynamic Runtime Control Variables:** Typed runtime knobs controlling sampling cadences, silence thresholds, and retry policies with atomic rollbacks.
- **Hierarchical Modulation:** Rule cascade determining follow-up tracking policies scoped by `(input_type, subtype, agent)`.
- **Progressive Auto-Remediation:** Background daemon healing soft breaks while loudly escalating hard breaks.

---

## 2. Architecture Diagram

```mermaid
flowchart TD
    subgraph VM ["Google Cloud Gateway VM (34.139.37.135)"]
        UI["Box Web Console (/srv/box/www)"]
        Board["board.service (/srv/board/server.py)"]
        UI -->|HTTP /api/box/loop/*| Board
    end

    subgraph Tailnet ["Tailscale Secure Mesh (100.123.153.75)"]
        Board -->|SSH Allowlisted RPC| BoxCtl["bin/box-ctl.py"]
    end

    subgraph BL ["Local Management Node (bl)"]
        BoxCtl --> VarEng["Variables Engine (bin/variables.py)"]
        BoxCtl --> ModEng["Modulation Strategy (bin/modulate.py)"]
        BoxCtl --> GravEng["Loop Diagnostics (bin/gravity.py)"]

        CLI["CLI Orchestrator (bin/super-cli.py)"]
        CLI --> VarEng
        CLI --> ModEng
        CLI --> GravEng

        Daemon["systemd: loop-remediator.timer (15m)"]
        Daemon --> GravEng

        GravEng -->|Soft Heal| Followups["followups.json"]
        GravEng -->|Hard Break Alert| DMLog["bin/dm.py -> opm"]
        GravEng -->|Audit Trail| JobLog["job-log.jsonl"]
    end
```

---

## 3. Dual-Surface API & CLI Reference

### 3.1. Runtime Control Variables

The runtime variables engine ([`bin/variables.py`](file:///home/super/Projects/NetVM/bin/variables.py)) enforces type constraints, ranges, and dual-sync persistence between `/srv/box/variables.json` and `./variables.json`. Every mutation is appended to [`variables-history.jsonl`](file:///home/super/Projects/NetVM/variables-history.jsonl).

#### CLI Commands
```bash
# List all registered variables, values, units, and ranges
super vars list
# or
box vars-list

# Get specific variable
super vars get loop_health_threshold

# Set a variable (validated against schema)
super vars set loop_health_threshold 0.65

# Reset variable to default
super vars reset loop_health_threshold

# Inspect audit history
super vars history [name] [limit]

# Atomic rollback
super vars rollback loop_health_threshold
```

#### VM REST Endpoints (`/api/box/loop/*`)
- `GET /api/box/loop/vars` → Retrieves full dictionary of runtime variables.
- `POST /api/box/loop/vars` → Body: `{"name": "...", "value": ...}` (returns 202 Accepted).
- `POST /api/box/loop/vars/reset` → Body: `{"name": "..."}`.
- `GET /api/box/loop/vars/history?name=...` → Returns append-only revision history.

---

### 3.2. Hierarchical Modulation Strategy

Follow-up tracking behavior ([`bin/modulate.py`](file:///home/super/Projects/NetVM/bin/modulate.py)) is resolved hierarchically across four precedence levels down to the builtin table:

$$\text{Override Precedence: } (T, S, A) \succ (T, \text{None}, A) \succ (T, S, \text{None}) \succ (T, \text{None}, \text{None}) \succ \text{Builtin}$$

1. Exact match: `(input_type, subtype, agent)`
2. Agent default: `(input_type, None, agent)`
3. Subtype default: `(input_type, subtype, None)`
4. Type default: `(input_type, None, None)`
5. Builtin table fallback

#### CLI Commands
```bash
# Show modulation matrix (builtins + active overrides)
super strat show

# Set override
super strat set manual --timeout 1800 --nudges 1

# Set agent-specific override
super strat set manual --agent pip --no-track

# Reset override
super strat reset manual --agent pip
```

#### VM REST Endpoints
- `GET /api/box/loop/strat` → Returns merged modulation matrix.
- `POST /api/box/loop/strat` → Body: `{"input_type": "...", "subtype": "...", "agent": "...", ...}`.
- `POST /api/box/loop/strat/reset` → Resets override for key.

---

### 3.3. Loop Diagnostics & Progressive Remediation

Loop health ([`bin/gravity.py`](file:///home/super/Projects/NetVM/bin/gravity.py)) tracks loop status across agents:

$$\text{Health Ratio} = \frac{\text{Closed} + \text{Answered}}{\text{Landed}}$$

#### Progressive Remediation Workflow
1. **Soft Breaks (Auto-Healed):**
   - **Answered Loops:** If a pending follow-up in [`followups.json`](file:///home/super/Projects/NetVM/followups.json) has a matching reply detected in [`dm-log.jsonl`](file:///home/super/Projects/NetVM/dm-log.jsonl), it is automatically marked `resolved` with note `auto-healed: reply detected in dm-log`.
   - **Expired Nudges:** If a loop deadline has lapsed but allowable nudges remain, the deadline is updated to `now` and [`bin/followup-sweeper.py`](file:///home/super/Projects/NetVM/bin/followup-sweeper.py) is invoked immediately.
2. **Hard Breaks (Loudly Escalated):**
   - `silent_agent`: Agent unresponsive after exhausting all allowed nudges.
   - `auth_rot`: Missing or corrupted SSH Ed25519 signing key (`~/.ssh/id_ed25519`).
   - `scheduler_death`: Systemd user session or timer infrastructure offline.
   - **Escalation Actions:** Emits structured event to [`job-log.jsonl`](file:///home/super/Projects/NetVM/job-log.jsonl) and dispatches an immediate DM alert to `opm` on `main`.

#### CLI Commands
```bash
# View fleet loop health table and ratio
super loop health

# View all active / reconstructed loops
super loop status --limit 50

# Diagnose detected loop breakages
super loop breaks

# Manually resolve a stuck loop
super loop close <dm_id> "Resolved via operator intervention"

# Trigger manual remediation pass
super loop remediate [--dry-run]
```

#### VM REST Endpoints
- `GET /api/box/loop/health` &rarr; JSON summary of fleet ratios and health verdicts.
- `GET /api/box/loop/status?limit=50` &rarr; Active loop instances.
- `POST /api/box/loop/resolve` &rarr; Body: `{"dm_id": "...", "note": "..."}`.
- `POST /api/box/loop/remediate` &rarr; Runs progressive remediation cycle.

#### Followup record fields & nudge→reply matching (2026-10-04 fix set)

Followup records in [`followups.json`](file:///home/super/Projects/NetVM/followups.json) carry two fields
introduced by the 2026-10-04 reliability fix set (see `docs/SIDECHAT-RELIABILITY.md`):

- **`thread_uuid` backfill** — when a followup is created but the sidechat thread
  autoprovisioning failed (no UUID captured), the sweeper retries on each nudge.
  Once a nudge send succeeds with autoprovisioning (`sidechat_autoprovisioned`
  in `dm-log.jsonl`), the sweeper backfills `thread_uuid` into the record. Once
  set, the field is **never overwritten with null** — a followup with a null
  `thread_uuid` can never match a reply, so it would ghost-nudge then escalate
  on a thread that never existed.
- **`final_nudge_target`** — set to `"main"` when the final nudge routes to the
  main chat instead of a sidechat (routing policy: nudge in-thread first, main
  only on the final attempt).

**Nudge→reply matching matrix** (implemented in `bin/response-harvester.py`):

| Nudge routing | Reply location | Resolves when |
|---|---|---|
| In-thread nudge | Assistant reply in the thread whose UUID == `thread_uuid` | UUID match |
| Final nudge → main | Assistant reply in main chat | `final_nudge_target == "main"` |
| Any | Reply in a different thread | Never auto-resolves (manual `loop close`) |

The sweeper also backfills `thread_uuid` immediately after a successful nudge
send when the send itself autoprovisioned the thread, so later replies match.

**`dm.py` placement assertion** — before sending to a sidechat, `dm.py` fails
closed (loud refusal, no `verified:true`) if the post-navigation browser URL
does not contain the target thread UUID. The `"main"` target is exempt. This
catches silent `sidechat use` failures and wrong-thread sends.

**Multi-`[RESULT]` harvesting** — the harvester processes **every** `[RESULT …]`
marker in a message (previously only the first), so replies that batch several
job results each resolve their followup and trigger their own chain step.

---

## 4. Background Services & Daemons

On node `bl`, loop remediation is managed by systemd user units:
- **Service:** [`/home/super/.config/systemd/user/loop-remediator.service`](file:///home/super/.config/systemd/user/loop-remediator.service)
  - Runs: `/usr/bin/python3 /home/super/Projects/NetVM/bin/gravity.py --remediate`
- **Timer:** [`/home/super/.config/systemd/user/loop-remediator.timer`](file:///home/super/.config/systemd/user/loop-remediator.timer)
  - Cadence: `OnCalendar=*:0/15` (fires every 15 minutes, synchronized with `loop_health_interval_s`).

Inspect service status:
```bash
systemctl --user status loop-remediator.timer
journalctl --user -u loop-remediator.service -n 20 --no-pager
```

---

## 5. Operator Troubleshooting Runbook

### Incident A: Fleet Health Drops Below Threshold (< 50%)
1. Run `super loop health` to pinpoint the offending agent node.
2. Run `super loop breaks` to see whether loops are `NUDGED`, `ESCALATED`, or `BROKEN`.
3. If an agent is unresponsive:
   - Check container process: `super fleet status`.
   - Send diagnostic ping: `super dm send --to <agent> --target "<agent tasks>" "Liveness check"`.
4. Run `super loop remediate` to auto-heal any lagged answer states.

### Incident B: Web Console Mutations Fail (403 or 500)
1. Verify operator authentication: Ensure valid PIN session cookie or Bearer token on `https://box.muse-dev.online/`.
2. Verify Tailnet SSH bridge:
   - From VM: `ssh super@100.123.153.75 /home/super/Projects/NetVM/bin/box-ctl.py loop-health`.
   - Check [`box-ctl.jsonl`](file:///home/super/Projects/NetVM/box-ctl.jsonl) on `bl` for allowlisted action audit records.
3. Check `board.service` logs on VM: `sudo journalctl -u board -n 50 --no-pager`.

### Incident C: Accidental Variable Corruption
1. View audit history: `super vars history <variable_name>`.
2. Rollback to prior known good value: `super vars rollback <variable_name>`.
3. If necessary, reset to hardcoded schema default: `super vars reset <variable_name>`.

---

## 6. Verification & Automated Testing

All operational modules are covered by the comprehensive unit test suite in [`tests/`](file:///home/super/Projects/NetVM/tests/):

```bash
# Run complete test suite (26 passing tests)
python3 -m unittest discover -s tests -v
```
- [`tests/test_variables_engine.py`](file:///home/super/Projects/NetVM/tests/test_variables_engine.py): Schema validation, rollback, and RPC actions.
- [`tests/test_modulate_strategy.py`](file:///home/super/Projects/NetVM/tests/test_modulate_strategy.py): Hierarchical override cascade and dynamic evaluation.
- [`tests/test_loop_health_remediation.py`](file:///home/super/Projects/NetVM/tests/test_loop_health_remediation.py): Progressive remediation, diagnostics, and loop reconstruction.
- [`tests/test_main_nav.py`](file:///home/super/Projects/NetVM/tests/test_main_nav.py): Sidechat navigation and Main Chat policy enforcement.
- [`bin/tests/test_followup_fixes.py`](file:///home/super/Projects/NetVM/bin/tests/test_followup_fixes.py): DM-machinery harness (25 tests) — sweeper backfill, final-nudge routing/marking, harvester matching matrix, multi-`[RESULT]` extraction, and the `dm.py` placement assertion (incl. main-exempt).