Thread-UUID rotation: measurement & mapping hygiene proposal

Box is the main surface. All operator work goes through Box (box.muse-dev.online). The web UI, box CLI, and agents share the same API endpoints. No UI-only powers.

Date: 2026-10-04. Source: dm-log.jsonl (window 2026-10-03 21:36Z → 2026-10-04 19:46Z, ~22h, 3318 lines — the log does not cover a full 7 days; rates below are extrapolated from this window and should be re-measured on a longer one). Read-only analysis; no code changed.

1. Measured rotation per alias

Two distinct phenomena share the symptom "alias → different UUID":

(A) Real thread death/rotation. The SPA retires threads. Observed earlier for heartbeat-opm (5bd5b350 → 0077e918 → dead, twice in one day, per operator notes; predates this log window).
(B) Fuzzy-search nondeterminism. sidechat use "<name>" scans every DOM element for textContent.includes(name), picks the shortest match, and clicks it (muse-chat-api.py: cmd_sidechat_use). The candidate set changes between React renders, so the same alias resolves to different live threads run to run — with no thread dying at all.

alias	distinct UUIDs (window)	autoprovisioned (real rotations)	send-target flaps (fuzzy)	changes/day*
heartbeat	4 (5f18476d, 1e75a740, 0077e918, …)	0	3	~3.3 flap
646 tasks	4 (1e75a740, a7d0b08a, 1dfb3199)	1 (18:30 → 1dfb3199)	2	~1.1 real, ~2.2 flap
646-pip	2 (82779e55 → 75feb3a2)	1 (18:01)	0	~1.1 real
ops-audit	2 (75feb3a2 → 34a8a87c)	1 (19:28)	0	~1.1 real
main-loop brain	1 (5f18476d)	0	—	collision, see §3
pipe-* (6 aliases)	1 each	1 each (creation)	0	0 post-creation
646-pip-coord	1 (4466d0c1)	0	0	0
646-opm-coord	1 (4139dd4e)	0	0	0

* changes/day extrapolated from ~22h window; demo-day traffic inflates flaps.

Smoking gun for (B): at 16:25:53 a super→646 send for "646 tasks" landed on a7d0b08a, and 52 seconds later an opm→646 send for the same alias in the same recipient browser landed on 1e75a740. No thread died in between — the fuzzy search is nondeterministic across runs.

2. Who writes / reads job-sidechats.json today

Writers:

dm.py autoprovision path only (atomic tmp+mv write of {thread_uuid, agent, created_at} on sidechat_autoprovisioned).
Manual edits (e.g. today's removal of stale heartbeat-opm / pipe-9735f2).

NOT writers:

The new sweeper thread_uuid backfill (workstream 1) writes followups.json only — it does not refresh job-sidechats.json.
thread_lifecycle.py exists but is marked "prototype, not deployed".

Readers: dm.py resolve_sidechat_target (exact key match → UUID, else passthrough to fuzzy search), harvester/siphon (monitor list), box-ctl, job-dispatch, super-cli.

Critical gap in the refresh story: a dead mapped UUID does not self-heal. sidechat use <dead-uuid> → SPA redirects to / → output is "Navigated to: https://muse.ai/" (not NOTFOUND) → nav_failed no_thread_url → hard fail. Autoprovision only fires on NOTFOUND from name search. So a dead mapping = permanent send failures for that alias until a human removes the entry. (This is fail-closed, not misdelivery — but it is not self-healing.)

Second gap: an unmapped alias whose fuzzy search keeps "succeeding" never autoprovision and therefore never earns a canonical mapping — it flaps forever. This is heartbeat's exact situation right now (see §3).

3. Flags

MISSING mapping, highest churn: heartbeat. 4 UUIDs in 3h, zero autoprovision events, no entry in job-sidechats.json (the old heartbeat-opm entry was removed during today's hardcoded-alias cleanup). Every send does a fresh fuzzy search. At 16:45 a heartbeat send landed on 1e75a740 — 646 tasks' thread. Misdelivery already happened here.
MISSING mapping: ops-audit. Re-provisioned 19:28 (34a8a87c) but the alias was never registered; next fuzzy search may flap.
COLLISION, unverified: main-loop brain → 5f18476d. The brain's init send (19:40) fuzzy-matched into 5f18476d — the same thread heartbeat used at 16:35 and 19:41. No sidechat_autoprovisioned event for the brain alias, so no new thread was created. Either the fuzzy search false-positived into heartbeat's thread, or the brain was deliberately placed there. Confirm intent — if unintended, digests and heartbeat traffic are cross-talking.
No current mapping points to a UUID superseded by a newer autoprovision for the same alias — all 11 entries match their alias's latest autoprovision event. Nothing stale in that narrow sense.
Aging, low-risk: test-auto-prov (last confirmed 16:20), 646-opm-coord (last confirmed 12:52), and six single-use pipe-* demo entries accumulate with no TTL. Harmless today, cruft tomorrow.
Confounder: as of ~19:43Z all four chromebox browsers are flapping (dying every 10–15 min, no crash signature). Nav flakiness measured today is polluted by this; re-measure rotation after the browsers stabilize.

4. Proposal: mapping hygiene

Canonical entry for every recurring alias. heartbeat, ops-audit, main-loop brain (once its thread is confirmed), 646 tasks, 646-pip, 646-pip-coord, 646-opm-coord must each have exactly one entry, created by explicit provisioning — never by fuzzy luck. Unmapped + fuzzy-matched is the misdelivery configuration; it should be treated as a bug, not a fallback.
Dead-UUID reprovisioning. On no_thread_url / uuid_mismatch where the nav target came from a mapping (source == dynamic_mapping), fall through to autoprovision and overwrite the mapping — the same path name search uses today. This closes the "dead mapping = permanent failure" gap and makes rotation self-healing. (Keep failing closed when even autoprovision can't produce a thread page.)
Exact-match-first fuzzy search. Before the DOM substring scan, sidechat use should try an exact title match (normalized case/space). Only fall back to substring on no exact hit. This kills most cross-alias false positives (the 16:45 heartbeat→646-tasks landing) without changing the autoprovision contract.
TTL + audit job. Add last_confirmed_at to each mapping entry, refreshed on every successful send through it. A periodic audit (daily is plenty; weekly at minimum) flags: entries unconfirmed > TTL (7d job, 24h heartbeat), aliases seen in dm-log with no entry, and UUIDs serving >1 alias (collision detector — would have caught 1e75a740 and 5f18476d). Report-only at first; auto-prune single-use pipe-*/test entries after TTL.
Sweeper backfill → also refresh the mapping. The workstream-1 backfill already learns the live UUID per followup; writing it through to job-sidechats.json when the followup's target is a mapped alias is nearly free and covers the "nudge provisioned a thread dm.py didn't" case.
Do not reintroduce hardcoded UUIDs (the empty SIDCHAT_ALIASES was the right call). The mapping file is the single source of truth; the audit job in (4) is what keeps it honest.

5. Suggested next steps (no code in this workstream)

Confirm main-loop brain thread intent (collision with heartbeat's 5f18476d).
Provision canonical threads for heartbeat and ops-audit; register them.
Implement proposal items 2 and 3 (dm.py + muse-chat-api.py — small, surgical).
Stand up the audit job (4) as report-only; review one week of flags before enabling any auto-prune.
Re-run this measurement on a 7-day dm-log window once rotation stabilizes post-browser-flap.

7.6 KiB Raw Permalink Blame History Unescape Escape