box/docs/RATE-LIMIT-POLICY.md

Per-Node Rate Limit Policy (2026-10-04)

Box is the main surface. All operator work goes through Box (box.muse-dev.online). The web UI, box CLI, and agents share the same API endpoints. No UI-only powers.

Why

All 4 fleet nodes (muse, pip, 646, opm) egress from a single Cloudflare IP (104.28.195.181). Cloudflare therefore sees the whole fleet as one client. When fleet timers fire simultaneously, synchronized send bursts look like one aggressive client and invite correlated rate-limiting — the likely shape of the "all 4 browsers flapping" episodes.

Per-agent rate limiting with deterministic jitter de-correlates the fleet: each node's send phase drifts apart instead of hitting in lockstep.

Policy

Agent	Base interval	Jitter factor	Effective interval	Burst	Max/min
muse	3.0s	1.2917	3.875s	5	20
pip	3.0s	0.8183	2.455s	5	20
646	3.0s	1.1193	3.358s	5	20
opm	3.0s	0.8287	2.486s	5	20

Mean effective interval ≈ 3.04s — the jitter does not make limits stricter on average; it only spreads phases.

• Jitter: ±30% deterministic per agent, seeded by sha256("netvm-rate-jitter:v1:" + agent). Same agent → same factor every run (reproducible); different agents → different factors (de-correlated). Uses an isolated random.Random, never touches global random state.
• Scope: per-agent token buckets. State lives in one shared file (/tmp/netvm-rate-limit.json) but keys are namespaced by agent ({agent}, {agent}_recent) — buckets are fully independent.
• Enforcement points:
  • job-dispatch.py — rate_limit_wait(agent) / rate_limit_wait(sender_agent) (pre-existing; now jittered, no call-site change needed)
  • dm.py::dm_send — rate_limit_wait(agent) before send, keyed on the sender (added 2026-10-04). Near no-op at normal pacing since sends already sleep 2–3s; it only bites on burst loops.

Inspect

python3 bin/rate_limiter.py --policy            # effective policy per agent
python3 bin/rate_limiter.py --policy opm --no-jitter

Notes

• /tmp state does not survive reboots; buckets start empty (permissive).
• To disable jitter for a caller: rate_limit_wait(agent, jitter=0).
• Distinct egress IPs per node (Cloudflare Zero Trust dedicated egress) remains the structural fix; this policy is the de-correlation layer that works regardless.