operator
81 lines
2.8 KiB
Markdown
81 lines
2.8 KiB
Markdown
# Identity Variance Testing
|
||
|
||
> **Box is the main surface.** All operator work goes through Box (box.muse-dev.online). The web UI, `box` CLI, and agents share the same API endpoints. No UI-only powers.
|
||
|
||
## Why
|
||
|
||
All NetVM nodes currently egress from a single Cloudflare IP (`104.28.195.181`),
|
||
but each node holds a *distinct* WireGuard identity. When production rate limits
|
||
hit, we need data to answer: per-identity, per-IP, time-based, or random?
|
||
|
||
This framework builds that dataset gently: ~2 probes per node every 10 minutes.
|
||
|
||
## Files
|
||
|
||
- `bin/identity-variance-test.py` — probe harness + analyzer
|
||
- `.state/identity-variance/variance.jsonl` — results log (git-ignored, append-only)
|
||
- (proposed, not installed) systemd unit + timer for 10-min cadence
|
||
|
||
## What it measures
|
||
|
||
Per probe, per node:
|
||
- `http_status` — 200s vs 429 (rate limited) vs 403 (blocked) vs curl failures
|
||
- `response_ms` — full-request timing from inside the netns
|
||
- `egress_ip` — IP seen from inside the netns (per node, per cycle)
|
||
- `identity_hash` — sha256 (truncated) of the node's WireGuard private key.
|
||
Detects identity rotation; the key itself is NEVER logged.
|
||
- `remote_ip` — the IP curl actually connected to
|
||
|
||
Probes (both lightweight, no auth, no writes):
|
||
1. `https://www.cloudflare.com/cdn-cgi/trace` — what Cloudflare sees
|
||
2. `https://muse.ai/` — production-relevant landing page
|
||
|
||
Traffic budget: 5 nodes × 2 probes every 10 min ≈ 6 req/hr per node. Negligible.
|
||
|
||
## Log format (JSONL)
|
||
|
||
```json
|
||
{"ts": "2026-10-04T21:50:00+00:00", "node": "muse", "probe": "cf-trace",
|
||
"url": "https://www.cloudflare.com/cdn-cgi/trace", "curl_rc": 0,
|
||
"http_status": 200, "response_ms": 312.4, "bytes": 240,
|
||
"remote_ip": "104.16.0.0", "rate_limited": false, "blocked": false,
|
||
"ok": true, "elapsed_wall_ms": 415.2,
|
||
"identity_hash": "a43dded8edc358e2", "egress_ip": "104.28.195.181"}
|
||
```
|
||
|
||
## Usage
|
||
|
||
```bash
|
||
# one manual cycle
|
||
bin/identity-variance-test.py probe
|
||
|
||
# analyze (all data)
|
||
bin/identity-variance-test.py analyze
|
||
|
||
# last 24h only
|
||
bin/identity-variance-test.py analyze --since 24
|
||
```
|
||
|
||
## Reading the analysis
|
||
|
||
- `http_429` differing by node → per-identity differential treatment
|
||
- all nodes 429 at the same `ts` → correlated (per-IP) throttling
|
||
- 429s clustering by time of day → time-based limits
|
||
- uniform, rare 429s with no pattern → effectively random
|
||
|
||
## Installing the timer (human-gated)
|
||
|
||
```bash
|
||
sudo cp systemd/identity-variance.{service,timer} /etc/systemd/system/
|
||
sudo systemctl daemon-reload
|
||
sudo systemctl enable --now identity-variance.timer
|
||
systemctl list-timers | grep identity-variance
|
||
```
|
||
|
||
## Notes
|
||
|
||
- `sudo -n ip netns exec` is required; script fails cleanly otherwise.
|
||
- The `def` node is included but will report `node-skip` until assigned.
|
||
- Log rotation: JSONL grows ~1.5KB/cycle (~200KB/day). Rotate monthly:
|
||
`mv variance.jsonl variance-$(date +%Y%m).jsonl`.
|