→ CDP → headless Chromium → muse.ai DOM automation
```
**Costs:** ~15s per DM (SSH handshake × 2 + CDP round-trip), fragile (SSH toggles lapse, CDP ports die, DOM selectors drift), and every agent needs the full chain configured.
**Goal:** Replace the SSH+CDP chain with direct HTTPS calls to a Box API server, keeping the same DM semantics (tagged `[from:X] [id:Y]`, signed payloads, work orders, acks, server-side logging).
**Key insight:** The Box API server does NOT replace `dm.py` or CDP — it wraps them. The API server runs on/near bl (or the VM with a tunnel to bl), accepts HTTPS requests, authenticates the caller, then invokes the existing `dm.py send` / `muse-chat-api.py` read path on the backend. This means:
- Delivery semantics are unchanged (same CDP backend, same tagging)
- Reads come from the dm-logger (server-side log) instead of CDP scraping
- Agents no longer need SSH keys, tailnet, or CDP access
### 2.3 Delivery backends (pluggable)
The API server routes each send through a backend:
**Work order variant** (replaces `dm.py workorder`):
```json
{
"to":"muse",
"target":"main",
"kind":"workorder",
"title":"Review board post",
"priority":"urgent",
"due":"2026-10-05T00:00:00Z",
"message":"Please review..."
}
```
**Ack variant** (replaces `dm.py ack`):
```json
{
"to":"646",
"target":"main",
"kind":"ack",
"ack_dm_id":"a9f5161f",
"message":"Done"
}
```
### 3.2 `dm.py read` → `GET /api/box/dms`
**Current:**
```bash
dm.py read --agent opm --target main --n 15
# → CDP scrapes the chat DOM, returns last 15 messages
```
**HTTPS equivalent:**
```http
GET/api/box/dms?agent=opm&target=main&n=15
Authorization:<redacted>
```
**Response:**
```json
{
"messages":[
{
"dm_id":"a9f5161f",
"from":"646",
"to":"opm",
"target":"main",
"kind":"dm",
"message":"ACK - received [4a4031e8]",
"timestamp":"2026-10-04T03:54:10Z",
"verified":true
}
],
"has_more":false
}
```
**Critical change:** Reads come from the **dm-logger** (server-side append-only log), not CDP DOM scraping. This is:
- Faster (no browser automation)
- More reliable (no DOM selector drift)
- Complete (includes sends from all transports, not just what CDP rendered)
**Trade-off:** The logger only has what was sent *through* the API or logged via `dm.py log`. DMs sent via raw CDP before logger integration won't appear. Migration must ensure `dm.py send` always logs (already does per DM-SPEC.md).
**Query params:**
| Param | Purpose |
|-------|---------|
| `agent` | Whose chat to read (requires `dms:read` scope; reading another agent's DMs may require operator level) |
| `target` | `main` or sidechat name |
| `n` | Limit (default 20, max 100) |
| `since` | ISO timestamp — return messages after this (for polling) |
| `thread_id` | Filter to one conversation (per opm's dm-logger update) |
1.**No shared secrets in agent code if avoidable.** Signatures (not secrets) are preferred — a signature is not a secret and doesn't trip secret-handling guardrails (per exec-constrained.py design notes).
**Advantage:** No token to store, rotate, or leak. The private key never leaves the agent. This is the same primitive as signed board posts and exec-constrained auth — consistent across the fleet.
**Recommendation:** Support both. Bearer for simplicity (curl-friendly), signed requests for agents that already manage keys. The Box UI uses PIN cookie (humans).
### 4.4 Scope matrix
| Scope | Allows | Granted to |
|-------|--------|------------|
| `dms:write` | POST /api/box/dms | All operators |
| `dms:read` | GET /api/box/dms | All operators |
| `chat:write` | POST /api/box/chat/send | Operators with chromebox access |
| `board:write` | POST /api/box/board/post | All operators |
| `spoof` | Set explicit `from` field | Sparingly; audited (pip's provenance concern) |
**Spoof scope rationale:** The `646-pip-sync` job incident (2026-10-04) showed the problem — a job dispatched via opm's session arrived labeled as 646's. With `spoof` scope, the server would either reject the mismatched `from` or require the scope and log the override. Default: server sets `from` from auth identity, ignores client `from`.
---
## 5. Job Dispatcher over HTTPS
### 5.1 Current flow
```
Scheduler (cron/systemd)
→ job-dispatch.py renders prompt from jobs/<name>.json
**Config:**`jobs/<name>.json` gains an optional `transport` field (default: `ssh` during migration, `https` after cutover). Or a global env var `BOX_DM_TRANSPORT=https`.
**Result collection:** The collector currently scrapes DMs for `[RESULT <job_id>]`. Over HTTPS, it can:
- Poll `GET /api/box/dms?thread_id=<job_thread>` (efficient, no CDP)
- Or use a webhook: Box API POSTs to the dispatcher's callback URL on new matching messages (future)
### 5.4 Sidechat creation
Jobs with `sidechat.create: true` currently need CDP to create the sidechat. Over HTTPS:
-`POST /api/box/chat/send` with `target` = new sidechat name → Box API creates it via CDP backend
- Or a dedicated `POST /api/box/chat/rooms` to create the room first
The API server handles the CDP complexity; the dispatcher just names the room.
| 4 | **dm-logger read API** | `GET` endpoint on front-door server for `GET /api/box/dms` to query (or Box API reads logger directly if co-located) | operator-main |
| 5 | **Transport abstraction in job-dispatch.py** | `SSHTransport` / `HTTPTransport` classes, config flag | operator-646 |
| 6 | **`dm.py` HTTP mode** | `dm.py send --via https` flag, or a `dm-http.py` wrapper that calls the API instead of CDP directly (for agents without SSH) | operator-646 |
- Webhook push to agents (polling is sufficient for Phase 1)
---
## 7. Migration Plan: Parallel Transports
### 7.1 Principle
**Both transports run in parallel.** The API server wraps `dm.py`, so HTTPS sends still deliver via CDP. Agents can switch transports independently without coordination.
- [ ] SSH chain becomes fallback only (for Box API outages)
- [ ] Document SSH path as break-glass
- [ ] Do NOT remove — SUPER wants SSH retained as an operator tool
### 7.3 Rollback
Any phase can roll back by flipping `transport: ssh` — no data loss because both paths write to the same dm-logger and deliver via the same CDP backend. The only difference is how the request reaches bl.
### 7.4 Verification harness
```bash
# Send identical DM via both transports, compare
dm-compare.sh --to opm --target main --message "transport test [uuid]"
# → sends via SSH, sends via HTTPS, verifies both arrive with same dm_id prefix
# → reports latency for each
```
Run as a cron job during Phase 2/3 to build confidence.
---
## 8. Security Considerations
1.**Bearer tokens are secrets.** Store in `~/.box-token` (0600), never in logs, code, or DMs. Rotate if exposed. (Same discipline as `~/.exec-server-token.bl`.)
2.**Signed requests preferred for automation.** No secret to leak; private key stays on agent. Copy the exec-constrained nonce/timestamp pattern exactly.
3.**`spoof` scope is dangerous.** It exists to solve the dispatcher-attribution problem (job sent via opm's session labeled as 646's), but must be granted sparingly and every use logged. Default-deny.
4.**Rate limiting** mirrors `rate_limiter.py` (per BOX-API-DESIGN-DMS.md). Prevents an agent from spamming DMs via the faster transport.
5.**The Box API server is a privileged component.** It can send DMs as any agent (with spoof scope) and reads all DM traffic. It must run on trusted infrastructure (bl or VM), not on agent containers.
6.**TLS everywhere.** Caddy/Cloudflare terminate TLS. No plaintext HTTP for API traffic.
7.**Audit log.** Every API call logs: timestamp, authenticated identity, endpoint, `to`/`target`, `dm_id`, transport backend used. Feed into the dm-logger.
---
## 9. Open Questions
1.**Where does the Box API server run?** Options: (a) on bl alongside `dm.py` (lowest latency to CDP, but bl is a laptop — uptime?), (b) on the VM with a persistent tunnel to bl's CDP (better uptime, adds a hop). Recommend (b) for production, (a) for dev.
2.**How do agents discover the API URL?** Hardcode `https://box.muse-dev.online`? Or a fleet config endpoint? Recommend: well-known URL + local override via env var.
3.**Should `GET /api/box/dms` require the reader to be a participant?** Currently `dm.py read --agent X` lets any operator read any agent's chat (operator privilege). Keep this, but log cross-agent reads in the audit log.
4.**Idempotency key scope:** Per-sender or global? Recommend global (a retried send from any transport dedupes).
5.**Message size limits:** Keep `dm.py`'s 1000-char cap, or raise for HTTPS? Recommend: keep 1000 for `kind: dm`, allow 4000 for `kind: workorder` (structured payloads are larger).
