box/docs/SIDECHAT-RELIABILITY.md

Sidechat Reliability Runbook

Fleet sidechat DMs are the fleet's nervous system. On 2026-10-04 they were measured at ~1/3 navigation success. This runbook is the single place that explains how the machinery works, how each known failure looks in the logs, and what to do when a DM doesn't arrive.

All paths below are on bl (super@100.123.153.75, via the VM). Repo: /home/super/Projects/NetVM (git, local-only, no remotes).

Companion tools (all read-only unless noted):

• bin/dm-log-taxonomy.py — failure taxonomy over dm-log.jsonl (counts, nav-failure breakdown per target/agent-pair, hourly bursts, ghost rate). Run: python3 bin/dm-log-taxonomy.py
• bin/placement-audit.py — human-readable placement/gate activity. placement-audit.py --since 24h, --since 60m, --watch
• bin/tests/test_followup_fixes.py — 25 unit tests pinning the sweeper/harvester/gate contracts

---

1. Architecture (10 lines)

1. Operator (or timer) runs bin/dm.py send --agent <node> --to <agent> --target <sidechat> "<msg>".
2. dm.py resolves the target name → thread UUID: SIDCHAT_ALIASES (intentionally empty — hardcoded UUIDs rot), then job-sidechats.json dynamic mapping, then passthrough name for sidebar search.
3. Unknown names autoprovision: sidechat create in the recipient's browser, and the fresh UUID is registered in job-sidechats.json (self-healing).
4. Browser control goes netvm-exec.sh <node> -- python3 muse-chat-api.py --account <node> <cmd>, which drives headless Chromium over CDP inside the node's Warp netns.
5. dm.py navigates to /thread/<uuid> (or sidebar title search), then the pre-send gate (2026-10-04) independently samples the browser URL and requires the expected UUID in it — fail-closed, never sends blind.
6. Send happens via CDP; verify_placement() re-navigates and reads the thread back (post-send check, complementary to the pre-send gate).
7. Every step appends machine-readable events to dm-log.jsonl (send_start, nav_ok, sent, verified, placement_failed, pre_send_assert_failed, …).
8. If the message carries reply:expected tags, _register_followup() arms a record in followups.json (deadline, nudges, escalate target).
9. followup-sweeper.py (driven by gravity loop) sends nudges on expiry — in-thread first, main chat on the final nudge — then escalates. It backfills thread_uuid when a nudge provisions the thread.
10. response-harvester.py (every ~60s via response-harvester.service) reads agent threads, records [RESULT job_id] markers to job-log.jsonl, and resolves followups when an assistant reply lands in the matching thread.

---

2. Failure-mode catalog

How to read the signatures: grep '"type": "<event>"' dm-log.jsonl, or use placement-audit.py --since 24h / dm-log-taxonomy.py.

2a. Landing-page nav (caught by the pre-send gate)

• Signature: {"type":"pre_send_assert_failed","id":…,"agent":…,"to":…, "target":…,"expected_uuid":…,"actual_url":"https://muse.ai/", "reason":"url_mismatch"|"not_on_thread_page"|"direct_nav_failed"}
• Cause: the browser never left the muse.ai landing page (or parked on the wrong thread) after sidechat use; the old code would have sent anyway.
• Effect: send aborted, exit 1, nothing delivered, nothing marked verified.
• Fix: re-run the send; if it repeats, the nav layer is flapping — see §2g and the nav workstream. The gate doing its job is good news, not a bug.
• Owner: dm.py placement-gate workstream (landed 2026-10-04).

2b. sidechat_uuid_capture_failed (autoprovision went nowhere)

• Signature: {"type":"sidechat_uuid_capture_failed","id":…,"out":"https://muse.ai/"}
• Cause: sidechat create ran but UUID capture saw the landing page — the thread may or may not exist; dm.py cannot know.
• Effect (pre-gate era): the send proceeded with thread_uuid: null and reported verified:true — a ghost send (see §2d). With the gate live, the send now fails closed instead.
• Owner: dm.py autoprovision path; gate workstream.

2c. Stale UUID rotation (the UUID you have is dead)

• Signature: {"type":"placement_failed",…,"expected_uuid":"<uuid>", "actual_url":"https://muse.ai/thread/<different-uuid>","reason":"url_mismatch"} or {"type":"placement_mismatch",…,"expected_uuid":…,"actual_uuid":…}.
• Cause: thread UUIDs rotate — observed same-day (heartbeat-opm: 5bd5b350 → 0077e918 → dead). Any hardcoded UUID is a time bomb; the SPA may also redirect a dead-UUID nav to whatever thread was last open (misdelivery, not just failure).
• Fix: never hardcode UUIDs — SIDCHAT_ALIASES in dm.py is intentionally empty (sibling workstream removed the last hardcoded entries 2026-10-04, incl. the twice-dead heartbeat UUID). Use name-based autoprovision + job-sidechats.json dynamic mappings, which self-heal on re-provision.
• Owner: UUID-rotation workstream. TODO: docs/UUID-ROTATION.md not yet written; this section is the interim reference.

2d. Placement-blind verified:true (the ghost send)

• Signature: {"type":"verified","id":…,"agent":…,"target":…} with no thread_uuid field (or "thread_uuid": null), paired with a sent event — and no placement_failed/pre_send_assert_failed anywhere near it.
• Cause: the old verify step checked the current browser chat, not the target (see operator AGENTS.md). A send parked on the landing page still "verified".
• Effect: the message is lost but every downstream system believes it landed — followups nudge a ghost, pipelines stall as dispatched.
• Fix: the pre-send gate (§2a) makes new ghost sends impossible (fail-closed). For historical ghosts: backfill thread_uuid in followups.json once the thread provisions, register the alias in job-sidechats.json, resolve manually with a note.
• Owner: gate workstream (prevention); operators (historical repair).

2e. Logged-out / dead session

• Signature: no single event type. Look for nav_failed with "rc": 1 and CDP tracebacks, or muse-chat-api.py login-state detection: page Title == "muse.ai" (LANDING), body containing "To log in, enter the code" (OTP_PROMPT), or "Your email matches multiple accounts" (ACCOUNT_SELECTION).
• Cause: Meta session expired, OTP pending, or (2026-10-04 ~19:43 UTC) all four chromebox browsers dying silently every 10–15 min with no crash log.
• Fix: agent-health.sh watchdog relaunches dead Chromium; OTP/account-selection states need the human (approval flow). Track: board task 9e8b15d69ca9 for the 2026-10-04 browser-flapping root cause.
• Owner: session-state workstream. TODO: dedicated session-probe tooling pending.

2f. Ghost followup (thread_uuid: null — can never auto-resolve)

• Signature: in followups.json: "thread_uuid": null, "status": "pending"; in dm-log: the original send's sidechat_uuid_capture_failed, then sweeper followup_nudged events that never resolve.
• Cause: the harvester's clear_matching_followups only matches exact thread_uuid (or target=="main" in main). A null-UUID followup matches nothing, forever — the sweeper nudges then escalates on a ghost.
• Fix: sweeper now backfills thread_uuid when a nudge successfully provisions the thread (workstream landed 2026-10-04). For existing ghosts: backfill manually, register the alias, resolve with a note. Longer-term TODO: sweeper should also backfill on sweep ticks by scanning dm-log for the original dm_id's sidechat_autoprovisioned event (currently only backfills on nudge success).
• Owner: sweeper-policy workstream.

2g. Flaky nav (~1/3 success) and timing races

• Signature: bursts of nav_failed / pre_send_assert_failed / placement_mismatch for the same target in placement-audit.py --since 60m; dm-log-taxonomy.py shows the per-target failure rate.
• Cause: under investigation 2026-10-04 — sidebar DOM races, CDP timing, possibly the browser-flapping issue (§2e). The gate converts these from silent misdeliveries into loud, retried failures (strictly better).
• Owner: nav-shootout, timing-races, and retry-module workstreams. TODO: their findings land here when complete.

2h. main_chat_blocked

• Signature: {"type":"main_chat_blocked","id":…,"agent":…,"to":…,"target":"main"}
• Cause: a send targeted main without --allow-main-chat. Deliberate guardrail: automation stays out of main chat unless explicitly allowed.
• Fix: pass --allow-main-chat if main delivery is really intended.

---

3. Diagnosis playbook: "a DM didn't arrive"

Run in order. All commands on bl unless noted.

1. Find the send. grep '"id": "<8-hex-id>"' /home/super/Projects/NetVM/dm-log.jsonl (or grep by target). Walk its event sequence: send_start → nav_ok? → sent? → verified? Any *_failed event is your answer.
2. Placement first. python3 bin/placement-audit.py --since 24h — if you see pre_send_assert_failed / placement_failed / placement_mismatch for the id, the message never left bl correctly. Check actual_url: landing page (§2a), wrong thread (§2c).
3. Check the taxonomy. python3 bin/dm-log-taxonomy.py — is this an isolated incident or is the target/agent-pair failing at a high rate right now (§2g)?
4. Check the session. If nav_failed with rc=1: is the browser alive? ss -tln for the CDP port in the netns; muse-chat-api.py login-state detection (LANDING vs LOGGED_IN per operator AGENTS.md). If all four browsers are flapping, see board task 9e8b15d69ca9 (§2e).
5. Check UUID freshness. python3 -c "import json;print(json.load(open( '/home/super/Projects/NetVM/job-sidechats.json')).get('<target>'))" — does the mapped UUID still open the right thread? If the send used a UUID from anywhere else, suspect rotation (§2c).
6. Check the followup. grep -A12 '<dm_id>' /home/super/Projects/NetVM/followups.json — thread_uuid: null + pending = ghost (§2f). If nudges are firing, the reply (if any) landed in a thread the harvester doesn't match: backfill + resolve manually.
7. Check the pipeline. If it was a pipeline step: grep '<job_id>' /home/super/Projects/NetVM/pipelines.json — dispatched with no result after a ghost send needs pipeline_engine.record_step_result() + a decision on chaining.
8. Repair, don't re-fire blindly. Fix the cause (nav, session, UUID), then resend once. Repeated blind retries of a ghost just burn the nudge budget.

---

4. Hard lessons (non-negotiable)

1. Never trust verified:true alone. It historically meant "found the msg_id in whatever chat the browser happened to be parked in." Always corroborate with a placement assertion (pre-send gate) or a thread read-back.
2. Assert the post-nav URL before sending. The gate is now the enforcement; any new send path must go through it. Fail closed, fail loudly (non-zero exit + pre_send_assert_failed event), never degrade to blind send.
3. Thread UUIDs rotate — never hardcode them. Observed same-day rotation. SIDCHAT_ALIASES stays empty; name-based autoprovision + job-sidechats.json self-heals. If you find a hardcoded UUID anywhere, delete it.
4. Backfill thread_uuid. A followup with null UUID can never auto-resolve. Backfill on provision (sweeper does this now), and repair historical ghosts by hand with a note.
5. Atomic writes on live files. siphon/response-harvester fire every ~60s; the sweeper loop is tight. Edit to /tmp, py_compile, atomic mv. Never half-write a live file. Back up (*.bak-<date>-<who>) before touching shared code.
6. One owner per file per change. Parallel fixes on shared surfaces caused the 2026-10-03 502s: reconcile first, disjoint ownership, review the combined diff before it goes live. dm.py is the single chokepoint — treat it accordingly.
7. The dm-log is the source of truth for delivery; the harvester is the source of truth for replies. verified ≠ delivered; sent ≠ read. When they disagree, believe the thread read-back.

---

5. Sibling workstream map (2026-10-04 troubleshooting sprint)

#	Workstream	Output	Status
1	Gate restore	pre-send gate in dm.py (assert_pre_send_placement)	Landed
2	Log taxonomy	bin/dm-log-taxonomy.py	Landed
3	UUID rotation	hardcoded UUID removal (SIDCHAT_ALIASES emptied)	Landed
4	Nav shootout	—	TODO: findings pending
5	Session-state	—	TODO: probe tooling pending
6	Timing races	—	TODO: findings pending
7	Retry module	shared retry helper	TODO: pending
8	Sweeper policy	thread_uuid backfill + final_nudge_target	Landed
9	Audit CLI	bin/placement-audit.py	Landed
10	This runbook	docs/SIDECHAT-RELIABILITY.md	This file

Related docs: docs/DM-SPEC.md (DM system spec), docs/SIDECHAT_SPEC.md, docs/SIDECHAT-POLICY-TRIAGE.md, docs/CHROMEBOX-RUNBOOK.md. TODO: docs/UUID-ROTATION.md (rotation mechanics deep-dive) — §2c is the interim reference.