box/docs/UUID-ROTATION.md

# Thread-UUID rotation: measurement & mapping hygiene proposal

> **Box is the main surface.** All operator work goes through Box (box.muse-dev.online). The web UI, `box` CLI, and agents share the same API endpoints. No UI-only powers.

Date: 2026-10-04. Source: `dm-log.jsonl` (window 2026-10-03 21:36Z → 2026-10-04 19:46Z,
~22h, 3318 lines — the log does not cover a full 7 days; rates below are
extrapolated from this window and should be re-measured on a longer one).
Read-only analysis; no code changed.

## 1. Measured rotation per alias

Two distinct phenomena share the symptom "alias → different UUID":

- **(A) Real thread death/rotation.** The SPA retires threads. Observed earlier
  for heartbeat-opm (5bd5b350 → 0077e918 → dead, twice in one day, per operator
  notes; predates this log window).
- **(B) Fuzzy-search nondeterminism.** `sidechat use "<name>"` scans every DOM
  element for `textContent.includes(name)`, picks the shortest match, and clicks
  it (`muse-chat-api.py: cmd_sidechat_use`). The candidate set changes between
  React renders, so the same alias resolves to different live threads run to run
  — with no thread dying at all.

| alias | distinct UUIDs (window) | autoprovisioned (real rotations) | send-target flaps (fuzzy) | changes/day* |
|---|---|---|---|---|
| heartbeat | 4 (5f18476d, 1e75a740, 0077e918, …) | 0 | 3 | ~3.3 flap |
| 646 tasks | 4 (1e75a740, a7d0b08a, 1dfb3199) | 1 (18:30 → 1dfb3199) | 2 | ~1.1 real, ~2.2 flap |
| 646-pip | 2 (82779e55 → 75feb3a2) | 1 (18:01) | 0 | ~1.1 real |
| ops-audit | 2 (75feb3a2 → 34a8a87c) | 1 (19:28) | 0 | ~1.1 real |
| main-loop brain | 1 (5f18476d) | 0 | — | collision, see §3 |
| pipe-* (6 aliases) | 1 each | 1 each (creation) | 0 | 0 post-creation |
| 646-pip-coord | 1 (4466d0c1) | 0 | 0 | 0 |
| 646-opm-coord | 1 (4139dd4e) | 0 | 0 | 0 |

\* changes/day extrapolated from ~22h window; demo-day traffic inflates flaps.

Smoking gun for (B): at 16:25:53 a `super`→646 send for "646 tasks" landed on
a7d0b08a, and 52 seconds later an `opm`→646 send for the same alias in the same
recipient browser landed on 1e75a740. No thread died in between — the fuzzy
search is nondeterministic across runs.

## 2. Who writes / reads job-sidechats.json today

Writers:
- `dm.py` autoprovision path only (atomic tmp+mv write of
  `{thread_uuid, agent, created_at}` on `sidechat_autoprovisioned`).
- Manual edits (e.g. today's removal of stale heartbeat-opm / pipe-9735f2).

NOT writers:
- The new sweeper thread_uuid backfill (workstream 1) writes **followups.json
  only** — it does not refresh job-sidechats.json.
- `thread_lifecycle.py` exists but is marked "prototype, not deployed".

Readers: `dm.py resolve_sidechat_target` (exact key match → UUID, else
passthrough to fuzzy search), harvester/siphon (monitor list), box-ctl,
job-dispatch, super-cli.

Critical gap in the refresh story: a **dead mapped UUID does not self-heal**.
`sidechat use <dead-uuid>` → SPA redirects to `/` → output is
"Navigated to: https://muse.ai/" (not NOTFOUND) → `nav_failed no_thread_url` →
hard fail. Autoprovision only fires on NOTFOUND from *name* search. So a dead
mapping = permanent send failures for that alias until a human removes the
entry. (This is fail-closed, not misdelivery — but it is not self-healing.)

Second gap: an **unmapped alias whose fuzzy search keeps "succeeding" never
autoprovision** and therefore never earns a canonical mapping — it flaps
forever. This is heartbeat's exact situation right now (see §3).

## 3. Flags

- **MISSING mapping, highest churn: `heartbeat`.** 4 UUIDs in 3h, zero
  autoprovision events, no entry in job-sidechats.json (the old `heartbeat-opm`
  entry was removed during today's hardcoded-alias cleanup). Every send does a
  fresh fuzzy search. At 16:45 a heartbeat send landed on 1e75a740 — 646 tasks'
  thread. **Misdelivery already happened here.**
- **MISSING mapping: `ops-audit`.** Re-provisioned 19:28 (34a8a87c) but the
  alias was never registered; next fuzzy search may flap.
- **COLLISION, unverified: `main-loop brain` → 5f18476d.** The brain's init
  send (19:40) fuzzy-matched into 5f18476d — the same thread heartbeat used at
  16:35 and 19:41. No `sidechat_autoprovisioned` event for the brain alias, so
  no new thread was created. Either the fuzzy search false-positived into
  heartbeat's thread, or the brain was deliberately placed there. **Confirm
  intent — if unintended, digests and heartbeat traffic are cross-talking.**
- **No current mapping points to a UUID superseded by a newer autoprovision
  for the same alias** — all 11 entries match their alias's latest
  autoprovision event. Nothing stale in that narrow sense.
- **Aging, low-risk:** `test-auto-prov` (last confirmed 16:20), `646-opm-coord`
  (last confirmed 12:52), and six single-use `pipe-*` demo entries accumulate
  with no TTL. Harmless today, cruft tomorrow.
- **Confounder:** as of ~19:43Z all four chromebox browsers are flapping
  (dying every 10–15 min, no crash signature). Nav flakiness measured today is
  polluted by this; re-measure rotation after the browsers stabilize.

## 4. Proposal: mapping hygiene

1. **Canonical entry for every recurring alias.** `heartbeat`, `ops-audit`,
   `main-loop brain` (once its thread is confirmed), `646 tasks`, `646-pip`,
   `646-pip-coord`, `646-opm-coord` must each have exactly one entry, created
   by explicit provisioning — never by fuzzy luck. Unmapped + fuzzy-matched is
   the misdelivery configuration; it should be treated as a bug, not a
   fallback.
2. **Dead-UUID reprovisioning.** On `no_thread_url` / `uuid_mismatch` where
   the nav target came from a *mapping* (source == `dynamic_mapping`), fall
   through to autoprovision and overwrite the mapping — the same path name
   search uses today. This closes the "dead mapping = permanent failure" gap
   and makes rotation self-healing. (Keep failing closed when even
   autoprovision can't produce a thread page.)
3. **Exact-match-first fuzzy search.** Before the DOM substring scan,
   `sidechat use` should try an exact title match (normalized case/space).
   Only fall back to substring on no exact hit. This kills most cross-alias
   false positives (the 16:45 heartbeat→646-tasks landing) without changing
   the autoprovision contract.
4. **TTL + audit job.** Add `last_confirmed_at` to each mapping entry, refreshed
   on every successful send through it. A periodic audit (daily is plenty;
   weekly at minimum) flags: entries unconfirmed > TTL (7d job, 24h heartbeat),
   aliases seen in dm-log with no entry, and UUIDs serving >1 alias
   (collision detector — would have caught 1e75a740 and 5f18476d). Report-only
   at first; auto-prune single-use `pipe-*`/test entries after TTL.
5. **Sweeper backfill → also refresh the mapping.** The workstream-1 backfill
   already learns the live UUID per followup; writing it through to
   job-sidechats.json when the followup's target is a mapped alias is nearly
   free and covers the "nudge provisioned a thread dm.py didn't" case.
6. **Do not reintroduce hardcoded UUIDs** (the empty `SIDCHAT_ALIASES` was the
   right call). The mapping file is the single source of truth; the audit job
   in (4) is what keeps it honest.

## 5. Suggested next steps (no code in this workstream)

- Confirm `main-loop brain` thread intent (collision with heartbeat's 5f18476d).
- Provision canonical threads for `heartbeat` and `ops-audit`; register them.
- Implement proposal items 2 and 3 (dm.py + muse-chat-api.py — small, surgical).
- Stand up the audit job (4) as report-only; review one week of flags before
  enabling any auto-prune.
- Re-run this measurement on a 7-day dm-log window once rotation stabilizes
  post-browser-flap.