super/box

Files

T

operator 2a2a808778 fix(watchers): exempt runaway shells from protection to prevent host OOM

- update is_protected() in box-stability-watcher.py to revoke immunity from bash/zsh processes with RSS >= 2048MB
- update watchers/README.md to document the 2048MB interactive shell threshold
- add unit tests verifying shell protection vs runaway exemption in test_box_stability_watcher.py

2026-10-07 21:09:45 +00:00

2.9 KiB

Raw Permalink Blame History

NetVM Watchers

Dedicated directory for background autonomous health, resource, and stability watchers on NetVM host bl.

Components

box-stability-watcher.py: Host resource supervisor and load shedder. Proactively monitors:
- System 1m/5m/15m load averages against core counts.
- Host RAM and Swap pressure percentages.
- Per-process memory leaks (critical RSS thresholds for muse-bin, headless Chromium renderers, Python workers).
- Rogue/leaked CPU hogs starving SSH/Tailscale.
- Failing systemd user services trapped in tight restart loops.
- Socket isolation violations (automated workers running on /tmp/tmux-1000/default instead of /tmp/tmux-muse.sock).
box-stability.json: Tunable operational thresholds, notifications, and protected process whitelist.
systemd/box-stability-watcher.service: Systemd user daemon running the watcher continuously with 10s evaluation ticks.

Operational Tiers & Mitigations

Tier	Status	Trigger Condition	Automated Action
GREEN	Normal	Load < 20, RAM < 80%, Swap < 75%	Silent monitoring.
YELLOW	Warning	Load >= 20, RAM >= 80%, or process RSS >= 2000MB	Renice CPU hogs (+15) to preserve interactive SSH responsiveness; log warning.
ORANGE	Critical	Load >= 35, RAM >= 90%, or process RSS >= 3000MB	Pause (SIGSTOP) runaway worker, record in `.state/stability-paused.json`, post alert to `646 tasks` sidechat, and allow 60s operator inspection before SIGTERM.
RED	Emergency	Load >= 60, RAM >= 95%, or Swap >= 92%	Emergency load shedding of non-protected heavy consumers (>1500MB).

Paused Process Lifecycle (60s Grace Window)

When a process is paused:

Sent SIGSTOP immediately.
Recorded in .state/stability-paused.json with timestamp and command info.
Alert posted to 646 tasks sidechat.
An operator can inspect the runtime or resume it via:
```
box stability resume <PID>
```
If unresumed after 60 seconds, the watcher automatically culls the process via SIGTERM.

Protected Whitelist

The watcher will never terminate or renice core system services or interactive terminal sessions: sshd, tailscaled, tailscale, systemd, dbus-broker, pipewire, wireplumber, tmux (main server), ghostty, alacritty.

Interactive shells (bash, zsh, sh) are protected while operating within normal memory bounds (<2048MB RSS). Runaway scripts or test jobs executing under bash/zsh that exceed 2048MB RSS automatically lose whitelist immunity and are subjected to the standard ORANGE pause/cull lifecycle to protect the host against OOM crashes.

Unified Box CLI Integration

# Host stability status & active socket warnings
box stability status

# Machine-readable JSON output
box stability json

# Single evaluation check
box stability check [--dry-run]

# Resume a paused process
box stability resume <PID>

# Top-line host health indicator
box fleet status

2.9 KiB Raw Permalink Blame History