operator
2a2a808778
- update is_protected() in box-stability-watcher.py to revoke immunity from bash/zsh processes with RSS >= 2048MB - update watchers/README.md to document the 2048MB interactive shell threshold - add unit tests verifying shell protection vs runaway exemption in test_box_stability_watcher.py
63 lines
2.9 KiB
Markdown
63 lines
2.9 KiB
Markdown
# NetVM Watchers
|
|
|
|
Dedicated directory for background autonomous health, resource, and stability watchers on NetVM host `bl`.
|
|
|
|
## Components
|
|
|
|
- **`box-stability-watcher.py`**: Host resource supervisor and load shedder. Proactively monitors:
|
|
- System 1m/5m/15m load averages against core counts.
|
|
- Host RAM and Swap pressure percentages.
|
|
- Per-process memory leaks (critical RSS thresholds for `muse-bin`, headless Chromium renderers, Python workers).
|
|
- Rogue/leaked CPU hogs starving SSH/Tailscale.
|
|
- Failing systemd user services trapped in tight restart loops.
|
|
- Socket isolation violations (automated workers running on `/tmp/tmux-1000/default` instead of `/tmp/tmux-muse.sock`).
|
|
- **`box-stability.json`**: Tunable operational thresholds, notifications, and protected process whitelist.
|
|
- **`systemd/box-stability-watcher.service`**: Systemd user daemon running the watcher continuously with 10s evaluation ticks.
|
|
|
|
## Operational Tiers & Mitigations
|
|
|
|
| Tier | Status | Trigger Condition | Automated Action |
|
|
| :--- | :--- | :--- | :--- |
|
|
| **GREEN** | Normal | Load < 20, RAM < 80%, Swap < 75% | Silent monitoring. |
|
|
| **YELLOW** | Warning | Load >= 20, RAM >= 80%, or process RSS >= 2000MB | Renice CPU hogs (+15) to preserve interactive SSH responsiveness; log warning. |
|
|
| **ORANGE** | Critical | Load >= 35, RAM >= 90%, or process RSS >= 3000MB | **Pause (SIGSTOP)** runaway worker, record in `.state/stability-paused.json`, post alert to `646 tasks` sidechat, and allow 60s operator inspection before SIGTERM. |
|
|
| **RED** | Emergency | Load >= 60, RAM >= 95%, or Swap >= 92% | Emergency load shedding of non-protected heavy consumers (>1500MB). |
|
|
|
|
## Paused Process Lifecycle (60s Grace Window)
|
|
|
|
When a process is paused:
|
|
1. Sent `SIGSTOP` immediately.
|
|
2. Recorded in `.state/stability-paused.json` with timestamp and command info.
|
|
3. Alert posted to `646 tasks` sidechat.
|
|
4. An operator can inspect the runtime or resume it via:
|
|
```bash
|
|
box stability resume <PID>
|
|
```
|
|
5. If unresumed after 60 seconds, the watcher automatically culls the process via `SIGTERM`.
|
|
|
|
## Protected Whitelist
|
|
|
|
The watcher will **never** terminate or renice core system services or interactive terminal sessions:
|
|
`sshd`, `tailscaled`, `tailscale`, `systemd`, `dbus-broker`, `pipewire`, `wireplumber`, `tmux` (main server), `ghostty`, `alacritty`.
|
|
|
|
Interactive shells (`bash`, `zsh`, `sh`) are protected while operating within normal memory bounds (<2048MB RSS). Runaway scripts or test jobs executing under `bash`/`zsh` that exceed 2048MB RSS automatically lose whitelist immunity and are subjected to the standard ORANGE pause/cull lifecycle to protect the host against OOM crashes.
|
|
|
|
## Unified Box CLI Integration
|
|
|
|
```bash
|
|
# Host stability status & active socket warnings
|
|
box stability status
|
|
|
|
# Machine-readable JSON output
|
|
box stability json
|
|
|
|
# Single evaluation check
|
|
box stability check [--dry-run]
|
|
|
|
# Resume a paused process
|
|
box stability resume <PID>
|
|
|
|
# Top-line host health indicator
|
|
box fleet status
|
|
```
|