54 lines
2.2 KiB
Markdown
54 lines
2.2 KiB
Markdown
|
# Per-Node Rate Limit Policy (2026-10-04)
|
|||
|
|
|
|||
|
|
## Why
|
|||
|
|
|
|||
|
|
All 4 fleet nodes (muse, pip, 646, opm) egress from a single Cloudflare IP
|
|||
|
|
(`104.28.195.181`). Cloudflare therefore sees the whole fleet as one client.
|
|||
|
|
When fleet timers fire simultaneously, synchronized send bursts look like one
|
|||
|
|
aggressive client and invite correlated rate-limiting — the likely shape of
|
|||
|
|
the "all 4 browsers flapping" episodes.
|
|||
|
|
|
|||
|
|
Per-agent rate limiting with deterministic jitter de-correlates the fleet:
|
|||
|
|
each node's send phase drifts apart instead of hitting in lockstep.
|
|||
|
|
|
|||
|
|
## Policy
|
|||
|
|
|
|||
|
|
| Agent | Base interval | Jitter factor | Effective interval | Burst | Max/min |
|
|||
|
|
|---|---|---|---|---|---|
|
|||
|
|
| muse | 3.0s | 1.2917 | 3.875s | 5 | 20 |
|
|||
|
|
| pip | 3.0s | 0.8183 | 2.455s | 5 | 20 |
|
|||
|
|
| 646 | 3.0s | 1.1193 | 3.358s | 5 | 20 |
|
|||
|
|
| opm | 3.0s | 0.8287 | 2.486s | 5 | 20 |
|
|||
|
|
|
|||
|
|
Mean effective interval ≈ 3.04s — the jitter does **not** make limits
|
|||
|
|
stricter on average; it only spreads phases.
|
|||
|
|
|
|||
|
|
- **Jitter:** ±30% deterministic per agent, seeded by
|
|||
|
|
`sha256("netvm-rate-jitter:v1:" + agent)`. Same agent → same factor every
|
|||
|
|
run (reproducible); different agents → different factors (de-correlated).
|
|||
|
|
Uses an isolated `random.Random`, never touches global random state.
|
|||
|
|
- **Scope:** per-agent token buckets. State lives in one shared file
|
|||
|
|
(`/tmp/netvm-rate-limit.json`) but keys are namespaced by agent
|
|||
|
|
(`{agent}`, `{agent}_recent`) — buckets are fully independent.
|
|||
|
|
- **Enforcement points:**
|
|||
|
|
- `job-dispatch.py` — `rate_limit_wait(agent)` / `rate_limit_wait(sender_agent)`
|
|||
|
|
(pre-existing; now jittered, no call-site change needed)
|
|||
|
|
- `dm.py::dm_send` — `rate_limit_wait(agent)` before send, keyed on the
|
|||
|
|
sender (added 2026-10-04). Near no-op at normal pacing since sends
|
|||
|
|
already sleep 2–3s; it only bites on burst loops.
|
|||
|
|
|
|||
|
|
## Inspect
|
|||
|
|
|
|||
|
|
```bash
|
|||
|
|
python3 bin/rate_limiter.py --policy # effective policy per agent
|
|||
|
|
python3 bin/rate_limiter.py --policy opm --no-jitter
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
## Notes
|
|||
|
|
|
|||
|
|
- `/tmp` state does not survive reboots; buckets start empty (permissive).
|
|||
|
|
- To disable jitter for a caller: `rate_limit_wait(agent, jitter=0)`.
|
|||
|
|
- Distinct egress IPs per node (Cloudflare Zero Trust dedicated egress)
|
|||
|
|
remains the structural fix; this policy is the de-correlation layer that
|
|||
|
|
works regardless.
|