feat(hybrid-gateway): integrate muse-cli with Cloudflare netns isolation, symmetric sidechat routing, and 646-pip sync unblock

2026-10-04 22:54:01 +00:00
parent 3d5fbe5aeb
commit 1a271b1bbd
60 changed files with 5068 additions and 55 deletions
@@ -0,0 +1,78 @@
+# Identity Variance Testing
+
+## Why
+
+All NetVM nodes currently egress from a single Cloudflare IP (`104.28.195.181`),
+but each node holds a *distinct* WireGuard identity. When production rate limits
+hit, we need data to answer: per-identity, per-IP, time-based, or random?
+
+This framework builds that dataset gently: ~2 probes per node every 10 minutes.
+
+## Files
+
+- `bin/identity-variance-test.py` — probe harness + analyzer
+- `.state/identity-variance/variance.jsonl` — results log (git-ignored, append-only)
+- (proposed, not installed) systemd unit + timer for 10-min cadence
+
+## What it measures
+
+Per probe, per node:
+- `http_status` — 200s vs 429 (rate limited) vs 403 (blocked) vs curl failures
+- `response_ms` — full-request timing from inside the netns
+- `egress_ip` — IP seen from inside the netns (per node, per cycle)
+- `identity_hash` — sha256 (truncated) of the node's WireGuard private key.
+  Detects identity rotation; the key itself is NEVER logged.
+- `remote_ip` — the IP curl actually connected to
+
+Probes (both lightweight, no auth, no writes):
+1. `https://www.cloudflare.com/cdn-cgi/trace` — what Cloudflare sees
+2. `https://muse.ai/` — production-relevant landing page
+
+Traffic budget: 5 nodes × 2 probes every 10 min ≈ 6 req/hr per node. Negligible.
+
+## Log format (JSONL)
+
+```json
+{"ts": "2026-10-04T21:50:00+00:00", "node": "muse", "probe": "cf-trace",
+ "url": "https://www.cloudflare.com/cdn-cgi/trace", "curl_rc": 0,
+ "http_status": 200, "response_ms": 312.4, "bytes": 240,
+ "remote_ip": "104.16.0.0", "rate_limited": false, "blocked": false,
+ "ok": true, "elapsed_wall_ms": 415.2,
+ "identity_hash": "a43dded8edc358e2", "egress_ip": "104.28.195.181"}
+```
+
+## Usage
+
+```bash
+# one manual cycle
+bin/identity-variance-test.py probe
+
+# analyze (all data)
+bin/identity-variance-test.py analyze
+
+# last 24h only
+bin/identity-variance-test.py analyze --since 24
+```
+
+## Reading the analysis
+
+- `http_429` differing by node → per-identity differential treatment
+- all nodes 429 at the same `ts` → correlated (per-IP) throttling
+- 429s clustering by time of day → time-based limits
+- uniform, rare 429s with no pattern → effectively random
+
+## Installing the timer (human-gated)
+
+```bash
+sudo cp systemd/identity-variance.{service,timer} /etc/systemd/system/
+sudo systemctl daemon-reload
+sudo systemctl enable --now identity-variance.timer
+systemctl list-timers | grep identity-variance
+```
+
+## Notes
+
+- `sudo -n ip netns exec` is required; script fails cleanly otherwise.
+- The `def` node is included but will report `node-skip` until assigned.
+- Log rotation: JSONL grows ~1.5KB/cycle (~200KB/day). Rotate monthly:
+  `mv variance.jsonl variance-$(date +%Y%m).jsonl`.