Identity Variance Testing

Box is the main surface. All operator work goes through Box (box.muse-dev.online). The web UI, box CLI, and agents share the same API endpoints. No UI-only powers.

Why

All NetVM nodes currently egress from a single Cloudflare IP (104.28.195.181), but each node holds a distinct WireGuard identity. When production rate limits hit, we need data to answer: per-identity, per-IP, time-based, or random?

This framework builds that dataset gently: ~2 probes per node every 10 minutes.

Files

bin/identity-variance-test.py — probe harness + analyzer
.state/identity-variance/variance.jsonl — results log (git-ignored, append-only)
(proposed, not installed) systemd unit + timer for 10-min cadence

What it measures

Per probe, per node:

http_status — 200s vs 429 (rate limited) vs 403 (blocked) vs curl failures
response_ms — full-request timing from inside the netns
egress_ip — IP seen from inside the netns (per node, per cycle)
identity_hash — sha256 (truncated) of the node's WireGuard private key. Detects identity rotation; the key itself is NEVER logged.
remote_ip — the IP curl actually connected to

Probes (both lightweight, no auth, no writes):

https://www.cloudflare.com/cdn-cgi/trace — what Cloudflare sees
https://muse.ai/ — production-relevant landing page

Traffic budget: 5 nodes × 2 probes every 10 min ≈ 6 req/hr per node. Negligible.

Log format (JSONL)

{"ts": "2026-10-04T21:50:00+00:00", "node": "muse", "probe": "cf-trace",
 "url": "https://www.cloudflare.com/cdn-cgi/trace", "curl_rc": 0,
 "http_status": 200, "response_ms": 312.4, "bytes": 240,
 "remote_ip": "104.16.0.0", "rate_limited": false, "blocked": false,
 "ok": true, "elapsed_wall_ms": 415.2,
 "identity_hash": "a43dded8edc358e2", "egress_ip": "104.28.195.181"}

Usage

# one manual cycle
bin/identity-variance-test.py probe

# analyze (all data)
bin/identity-variance-test.py analyze

# last 24h only
bin/identity-variance-test.py analyze --since 24

Reading the analysis

http_429 differing by node → per-identity differential treatment
all nodes 429 at the same ts → correlated (per-IP) throttling
429s clustering by time of day → time-based limits
uniform, rare 429s with no pattern → effectively random

Installing the timer (human-gated)

sudo cp systemd/identity-variance.{service,timer} /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable --now identity-variance.timer
systemctl list-timers | grep identity-variance

Notes

sudo -n ip netns exec is required; script fails cleanly otherwise.
The def node is included but will report node-skip until assigned.
Log rotation: JSONL grows ~1.5KB/cycle (~200KB/day). Rotate monthly: mv variance.jsonl variance-$(date +%Y%m).jsonl.

2.8 KiB Raw Blame History Unescape Escape