box/docs/IDENTITY-VARIANCE.md

# Identity Variance Testing

> **Box is the main surface.** All operator work goes through Box (box.muse-dev.online). The web UI, `box` CLI, and agents share the same API endpoints. No UI-only powers.

## Why

All NetVM nodes currently egress from a single Cloudflare IP (`104.28.195.181`),
but each node holds a *distinct* WireGuard identity. When production rate limits
hit, we need data to answer: per-identity, per-IP, time-based, or random?

This framework builds that dataset gently: ~2 probes per node every 10 minutes.

## Files

- `bin/identity-variance-test.py` — probe harness + analyzer
- `.state/identity-variance/variance.jsonl` — results log (git-ignored, append-only)
- (proposed, not installed) systemd unit + timer for 10-min cadence

## What it measures

Per probe, per node:
- `http_status` — 200s vs 429 (rate limited) vs 403 (blocked) vs curl failures
- `response_ms` — full-request timing from inside the netns
- `egress_ip` — IP seen from inside the netns (per node, per cycle)
- `identity_hash` — sha256 (truncated) of the node's WireGuard private key.
  Detects identity rotation; the key itself is NEVER logged.
- `remote_ip` — the IP curl actually connected to

Probes (both lightweight, no auth, no writes):
1. `https://www.cloudflare.com/cdn-cgi/trace` — what Cloudflare sees
2. `https://muse.ai/` — production-relevant landing page

Traffic budget: 5 nodes × 2 probes every 10 min ≈ 6 req/hr per node. Negligible.

## Log format (JSONL)

```json
{"ts": "2026-10-04T21:50:00+00:00", "node": "muse", "probe": "cf-trace",
 "url": "https://www.cloudflare.com/cdn-cgi/trace", "curl_rc": 0,
 "http_status": 200, "response_ms": 312.4, "bytes": 240,
 "remote_ip": "104.16.0.0", "rate_limited": false, "blocked": false,
 "ok": true, "elapsed_wall_ms": 415.2,
 "identity_hash": "a43dded8edc358e2", "egress_ip": "104.28.195.181"}
```

## Usage

```bash
# one manual cycle
bin/identity-variance-test.py probe

# analyze (all data)
bin/identity-variance-test.py analyze

# last 24h only
bin/identity-variance-test.py analyze --since 24
```

## Reading the analysis

- `http_429` differing by node → per-identity differential treatment
- all nodes 429 at the same `ts` → correlated (per-IP) throttling
- 429s clustering by time of day → time-based limits
- uniform, rare 429s with no pattern → effectively random

## Installing the timer (human-gated)

```bash
sudo cp systemd/identity-variance.{service,timer} /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable --now identity-variance.timer
systemctl list-timers | grep identity-variance
```

## Notes

- `sudo -n ip netns exec` is required; script fails cleanly otherwise.
- The `def` node is included but will report `node-skip` until assigned.
- Log rotation: JSONL grows ~1.5KB/cycle (~200KB/day). Rotate monthly:
  `mv variance.jsonl variance-$(date +%Y%m).jsonl`.