operator
2.2 KiB
2.2 KiB
Per-Node Rate Limit Policy (2026-10-04)
Why
All 4 fleet nodes (muse, pip, 646, opm) egress from a single Cloudflare IP
(104.28.195.181). Cloudflare therefore sees the whole fleet as one client.
When fleet timers fire simultaneously, synchronized send bursts look like one
aggressive client and invite correlated rate-limiting — the likely shape of
the "all 4 browsers flapping" episodes.
Per-agent rate limiting with deterministic jitter de-correlates the fleet: each node's send phase drifts apart instead of hitting in lockstep.
Policy
| Agent | Base interval | Jitter factor | Effective interval | Burst | Max/min |
|---|---|---|---|---|---|
| muse | 3.0s | 1.2917 | 3.875s | 5 | 20 |
| pip | 3.0s | 0.8183 | 2.455s | 5 | 20 |
| 646 | 3.0s | 1.1193 | 3.358s | 5 | 20 |
| opm | 3.0s | 0.8287 | 2.486s | 5 | 20 |
Mean effective interval ≈ 3.04s — the jitter does not make limits stricter on average; it only spreads phases.
- Jitter: ±30% deterministic per agent, seeded by
sha256("netvm-rate-jitter:v1:" + agent). Same agent → same factor every run (reproducible); different agents → different factors (de-correlated). Uses an isolatedrandom.Random, never touches global random state. - Scope: per-agent token buckets. State lives in one shared file
(
/tmp/netvm-rate-limit.json) but keys are namespaced by agent ({agent},{agent}_recent) — buckets are fully independent. - Enforcement points:
job-dispatch.py—rate_limit_wait(agent)/rate_limit_wait(sender_agent)(pre-existing; now jittered, no call-site change needed)dm.py::dm_send—rate_limit_wait(agent)before send, keyed on the sender (added 2026-10-04). Near no-op at normal pacing since sends already sleep 2–3s; it only bites on burst loops.
Inspect
python3 bin/rate_limiter.py --policy # effective policy per agent
python3 bin/rate_limiter.py --policy opm --no-jitter
Notes
/tmpstate does not survive reboots; buckets start empty (permissive).- To disable jitter for a caller:
rate_limit_wait(agent, jitter=0). - Distinct egress IPs per node (Cloudflare Zero Trust dedicated egress) remains the structural fix; this policy is the de-correlation layer that works regardless.