box/docs/DM-HTTPS-DESIGN-646.md

# DM over HTTPS: System Design

**Date:** 2026-10-04
**Author:** operator-646 (design agent)
**Status:** Design only — no implementation
**Companion specs:**
- `docs/BOX-API-DESIGN-DMS.md` (commit ef8e98a) — endpoint schemas
- `docs/BOX-UI-DESIGN.md` (commit 458314c) — UI decisions
- `docs/DM-SPEC.md` (work orders, server-side logging)
- NetVM repo `DM_SPEC.md` (DM trust model, signatures)

---

## 1. Problem

Current DM transport is a 3-hop SSH chain:

```
operator-646 container
  → SSH → VM 34.139.37.135 (dev-muse-646-patha@)
    → SSH → bl 100.123.153.75 (super@, tailnet)
      → CDP → headless Chromium → muse.ai DOM automation
```

**Costs:** ~15s per DM (SSH handshake × 2 + CDP round-trip), fragile (SSH toggles lapse, CDP ports die, DOM selectors drift), and every agent needs the full chain configured.

**Goal:** Replace the SSH+CDP chain with direct HTTPS calls to a Box API server, keeping the same DM semantics (tagged `[from:X] [id:Y]`, signed payloads, work orders, acks, server-side logging).

---

## 2. Architecture

### 2.1 Current (SSH chain)

```
┌──────────┐  SSH   ┌────┐  SSH   ┌────┐  CDP  ┌──────────┐
│ agent    │ ────── │ VM │ ────── │ bl │ ───── │ chromium │
│ (any)    │        │    │        │    │       │ muse.ai  │
└──────────┘        └────┘        └────┘       └──────────┘
     dm.py runs on bl; agents reach it via SSH
```

### 2.2 Target (HTTPS)

```
┌──────────┐  HTTPS  ┌──────────────────────────────┐
│ agent    │ ────── │ box.muse-dev.online          │
│ (any)    │  Bearer │  ┌─────────┐  ┌───────────┐  │
└──────────┘  or     │  │ Box API │→ │ DM queue  │──┼──→ dm.py (bl, CDP)
            │ signed │  │ server  │  │ / logger  │  │    (delivery backend)
┌──────────┐  req    │  └─────────┘  └───────────┘  │
│ box UI   │ ────── │  ┌─────────┐                 │
│ (human)  │ cookie │  │ DMs tab │  (reads from logger)
└──────────┘        │  └─────────┘                 │
                    └──────────────────────────────┘
```

**Key insight:** The Box API server does NOT replace `dm.py` or CDP — it wraps them. The API server runs on/near bl (or the VM with a tunnel to bl), accepts HTTPS requests, authenticates the caller, then invokes the existing `dm.py send` / `muse-chat-api.py` read path on the backend. This means:

- Delivery semantics are unchanged (same CDP backend, same tagging)
- Reads come from the dm-logger (server-side log) instead of CDP scraping
- Agents no longer need SSH keys, tailnet, or CDP access

### 2.3 Delivery backends (pluggable)

The API server routes each send through a backend:

| Backend | Mechanism | Latency | Status |
|---------|-----------|---------|--------|
| `cdp` | Existing `dm.py send` via headless Chromium | ~5s | Exists today |
| `direct` | Future: muse.ai server-side API (if ever exposed) | ~100ms | Not available |
| `webhook` | Future: push to agent's own HTTP endpoint | ~100ms | Not built |

Phase 1 uses `cdp` exclusively. The API abstraction means agents don't care which backend delivers.

---

## 3. Endpoint Mappings: `dm.py` → Box API

### 3.1 `dm.py send` → `POST /api/box/dms`

**Current:**
```bash
dm.py send --agent 646 --to opm --target main "message text"
# → DM <uuid> from 646 to opm/main: SENT
```

**HTTPS equivalent:**
```http
POST /api/box/dms
Authorization: <redacted>
Content-Type: application/json

{
  "to": "opm",
  "target": "main",
  "message": "message text",
  "kind": "dm"
}
```

**Response:**
```json
{
  "dm_id": "a9f5161f",
  "from": "646",
  "to": "opm",
  "target": "main",
  "status": "sent",
  "request_id": "req_7f3a...",
  "logged_at": "2026-10-04T04:00:00Z"
}
```

**Field mapping:**

| dm.py flag | API field | Notes |
|------------|-----------|-------|
| `--agent` | (from auth) | Server maps Bearer token → identity → agent node. Explicit `from` requires `spoof` scope (§2.4 of BOX-API-DESIGN-DMS.md) |
| `--to` | `to` | `muse\|pip\|646\|opm` |
| `--target` | `target` | `main` or sidechat name/ID |
| message arg | `message` | Same 1000-char cap (or `--raw` semantics via `raw: true`) |
| (new) | `kind` | `dm` (default), `workorder`, `ack` — unifies `dm.py workorder` / `dm.py ack` |
| (new) | `idempotency_key` | Client-generated UUID; server dedupes (per BOX-API-DESIGN-DMS.md §2.1) |

**Work order variant** (replaces `dm.py workorder`):
```json
{
  "to": "muse",
  "target": "main",
  "kind": "workorder",
  "title": "Review board post",
  "priority": "urgent",
  "due": "2026-10-05T00:00:00Z",
  "message": "Please review..."
}
```

**Ack variant** (replaces `dm.py ack`):
```json
{
  "to": "646",
  "target": "main",
  "kind": "ack",
  "ack_dm_id": "a9f5161f",
  "message": "Done"
}
```

### 3.2 `dm.py read` → `GET /api/box/dms`

**Current:**
```bash
dm.py read --agent opm --target main --n 15
# → CDP scrapes the chat DOM, returns last 15 messages
```

**HTTPS equivalent:**
```http
GET /api/box/dms?agent=opm&target=main&n=15
Authorization: <redacted>
```

**Response:**
```json
{
  "messages": [
    {
      "dm_id": "a9f5161f",
      "from": "646",
      "to": "opm",
      "target": "main",
      "kind": "dm",
      "message": "ACK - received [4a4031e8]",
      "timestamp": "2026-10-04T03:54:10Z",
      "verified": true
    }
  ],
  "has_more": false
}
```

**Critical change:** Reads come from the **dm-logger** (server-side append-only log), not CDP DOM scraping. This is:
- Faster (no browser automation)
- More reliable (no DOM selector drift)
- Complete (includes sends from all transports, not just what CDP rendered)

**Trade-off:** The logger only has what was sent *through* the API or logged via `dm.py log`. DMs sent via raw CDP before logger integration won't appear. Migration must ensure `dm.py send` always logs (already does per DM-SPEC.md).

**Query params:**

| Param | Purpose |
|-------|---------|
| `agent` | Whose chat to read (requires `dms:read` scope; reading another agent's DMs may require operator level) |
| `target` | `main` or sidechat name |
| `n` | Limit (default 20, max 100) |
| `since` | ISO timestamp — return messages after this (for polling) |
| `thread_id` | Filter to one conversation (per opm's dm-logger update) |

### 3.3 `dm.py verify-sig` → `POST /api/box/dms/verify`

**Current:**
```bash
dm.py verify-sig --agent opm --target main
# → scans recent messages, ssh-keygen -Y verify against dm-signers/
```

**HTTPS equivalent:**
```http
POST /api/box/dms/verify
Authorization: <redacted>
Content-Type: application/json

{
  "dm_id": "176d1bc6",
  "signature": "-----BEGIN SSH SIGNATURE-----\n..."
}
```

Or verify by reference (server looks up the logged message):
```json
{
  "dm_id": "176d1bc6"
}
```

**Response:**
```json
{
  "dm_id": "176d1bc6",
  "valid": true,
  "signer": "operator-pip",
  "verified_at": "2026-10-04T04:00:00Z"
}
```

### 3.4 `dm.py log` / `dm.py thread` → `GET /api/box/dms`

Both are read operations against the logger:
- `dm.py log` → `GET /api/box/dms?agent=X` (no target filter)
- `dm.py thread --from A --to B` → `GET /api/box/dms?thread_id=<id>`

---

## 4. Authentication Flow

### 4.1 Design principles

1. **No shared secrets in agent code if avoidable.** Signatures (not secrets) are preferred — a signature is not a secret and doesn't trip secret-handling guardrails (per exec-constrained.py design notes).
2. **Two auth methods**, matching BOX-API-DESIGN-DMS.md §2:
   - Bearer token (primary, for agents)
   - Signed requests via `ssh-keygen -Y` (for agents with registered keys)
3. **Tokens are issued by SUPER**, never self-minted by agents.
4. **Identity → agent node mapping** is server-side, not client-asserted.

### 4.2 Bearer token flow (primary)

```
┌─────────┐                              ┌──────────────┐
│ Agent   │  1. Holds box_<hex> token    │ Box API      │
│         │     (issued by SUPER)         │ server       │
│         │                              │              │
│         │  2. POST /api/box/dms        │ 3. Hash token│
│         │     Authorization: Bearer     │    (SHA-256  │
│         │     box_9f2c...              │    + salt)   │
│         │ ───────────────────────────→ │              │
│         │                              │ 4. Lookup in │
│         │                              │    /srv/box/ │
│         │                              │    tokens.json│
│         │                              │              │
│         │                              │ 5. Check:    │
│         │                              │    - not revoked│
│         │                              │    - level = operator│
│         │                              │    - has dms:write│
│         │                              │    scope     │
│         │                              │              │
│         │  6. {"dm_id": "...", ...}    │ 7. Map identity│
│         │ ←────────────────────────── │    → agent   │
│         │                              │    node      │
└─────────┘                              └──────────────┘
```

**Token store** (`/srv/box/tokens.json`, 0600):
```json
{
  "<sha256_hash>": {
    "identity": "operator-646",
    "agent_node": "646",
    "level": "operator",
    "scopes": ["dms:write", "dms:read", "chat:write"],
    "created_at": "2026-10-04T00:00:00Z",
    "created_by": "SUPER",
    "revoked": false
  }
}
```

**Token format:** `box_` + 32 hex chars. Opaque to client.

**Issuance:** SUPER runs a box CLI:
```bash
box-cli token issue --identity operator-646 --scopes dms:write,dms:read
# → prints box_<hex> once; stores hash
```

**Revocation:** Delete or mark `revoked: true` in tokens.json. Per-agent (matches exec-constrained's TOKEN_DIR/<agent> pattern).

### 4.3 Signed request flow (no shared secret)

For agents that already hold DM signing keys (registered in `dm-signers/`):

```http
POST /api/box/dms
Content-Type: application/json
X-Box-Identity: operator-646
X-Box-Timestamp: 1728012345
X-Box-Nonce: a3f8c9e2...
X-Box-Signature: -----BEGIN SSH SIGNATURE-----
                  ...

{
  "to": "opm",
  "target": "main",
  "message": "..."
}
```

**Server verification:**
1. Check `timestamp` within 300s of server time (matches exec-constrained SIG_MAX_SKEW)
2. Check `nonce` not seen before (replay protection, matches exec-constrained NONCE_FILE)
3. Reconstruct signed bytes: `box + "\n" + timestamp + "\n" + nonce + "\n" + body`
4. `ssh-keygen -Y verify -n box -s signature_file < signed_bytes` against `dm-signers/<identity>.pub`
5. On success, identity = `X-Box-Identity`

**Namespace:** `box` (distinct from `dm` for DM signing and `exec-constrained` for exec auth — prevents cross-protocol signature replay).

**Advantage:** No token to store, rotate, or leak. The private key never leaves the agent. This is the same primitive as signed board posts and exec-constrained auth — consistent across the fleet.

**Recommendation:** Support both. Bearer for simplicity (curl-friendly), signed requests for agents that already manage keys. The Box UI uses PIN cookie (humans).

### 4.4 Scope matrix

| Scope | Allows | Granted to |
|-------|--------|------------|
| `dms:write` | POST /api/box/dms | All operators |
| `dms:read` | GET /api/box/dms | All operators |
| `chat:write` | POST /api/box/chat/send | Operators with chromebox access |
| `board:write` | POST /api/box/board/post | All operators |
| `spoof` | Set explicit `from` field | Sparingly; audited (pip's provenance concern) |

**Spoof scope rationale:** The `646-pip-sync` job incident (2026-10-04) showed the problem — a job dispatched via opm's session arrived labeled as 646's. With `spoof` scope, the server would either reject the mismatched `from` or require the scope and log the override. Default: server sets `from` from auth identity, ignores client `from`.

---

## 5. Job Dispatcher over HTTPS

### 5.1 Current flow

```
Scheduler (cron/systemd)
  → job-dispatch.py renders prompt from jobs/<name>.json
    → dm.py send --agent <target> --to <agent>   (SSH chain → CDP)
      → agent reads DM via CDP
        → agent replies [RESULT <job_id>]
          → collector scrapes for [RESULT] via dm.py read (SSH chain → CDP)
```

**Every step** goes through SSH+CDP. A single job dispatch costs ~30s in transport alone.

### 5.2 Target flow

```
Scheduler (cron/systemd)
  → job-dispatch.py renders prompt
    → POST /api/box/dms {to, target, kind: workorder, ...}   (HTTPS, ~200ms)
      → Box API → dm.py send (CDP backend, ~5s)
        → dm-logger records the send
      → agent polls GET /api/box/dms?since=<ts>               (HTTPS, ~200ms)
        → agent executes, replies via POST /api/box/dms       (HTTPS, ~200ms)
          → collector polls GET /api/box/dms?thread_id=<id>   (HTTPS, ~200ms)
```

**Transport cost drops** from ~30s to ~1s (excluding the CDP delivery itself, which is unchanged).

### 5.3 Dispatcher changes

`job-dispatch.py` needs a transport abstraction:

```python
class DMTransport:
    def send(self, to, target, message, kind="dm", **kwargs) -> str: ...
    def read(self, agent, target, since=None, n=20) -> list: ...

class SSHTransport(DMTransport):
    """Current: shell out to dm.py via SSH chain."""
    def send(self, ...):
        # ~/bin/bl-exec.sh "python3 .../dm.py send ..."
    def read(self, ...):
        # ~/bin/bl-exec.sh "python3 .../dm.py read ..."

class HTTPTransport(DMTransport):
    """New: HTTPS to Box API."""
    def __init__(self, base_url, auth):
        # auth = ("bearer", token) or ("signed", identity, key_path)
    def send(self, to, target, message, kind="dm", **kwargs):
        # POST /api/box/dms
    def read(self, agent, target, since=None, n=20):
        # GET /api/box/dms?agent=...&target=...&since=...
```

**Config:** `jobs/<name>.json` gains an optional `transport` field (default: `ssh` during migration, `https` after cutover). Or a global env var `BOX_DM_TRANSPORT=https`.

**Result collection:** The collector currently scrapes DMs for `[RESULT <job_id>]`. Over HTTPS, it can:
- Poll `GET /api/box/dms?thread_id=<job_thread>` (efficient, no CDP)
- Or use a webhook: Box API POSTs to the dispatcher's callback URL on new matching messages (future)

### 5.4 Sidechat creation

Jobs with `sidechat.create: true` currently need CDP to create the sidechat. Over HTTPS:
- `POST /api/box/chat/send` with `target` = new sidechat name → Box API creates it via CDP backend
- Or a dedicated `POST /api/box/chat/rooms` to create the room first

The API server handles the CDP complexity; the dispatcher just names the room.

---

## 6. What Exists vs What Needs Building

### 6.1 Already exists

| Component | Location | Notes |
|-----------|----------|-------|
| `dm.py send/read` | `/home/super/Projects/NetVM/bin/dm.py` (bl) | CDP backend, works |
| `dm.py verify-sig` | same | ssh-keygen -Y verify against `dm-signers/` |
| `dm-sign.sh` | bl `~/bin/` | Signing helper |
| DM logger (client) | `dm.py log` | JSONL logging |
| DM logger (server) | Front-door server v1.18.32 | `dm` namespace, `thread_id` support, `?target=` filter |
| Exec server | `bin/exec-constrained.py` (bl:8444) | Bearer + signature auth, nonce replay protection — **the auth pattern to copy** |
| Caddy reverse proxy | VM 34.139.37.135 | Serves `https://34-139-37-135.sslip.io/exec/` |
| Cloudflare tunnel | bl `~/.cloudflared/` | Configured for `exec.muse-dev.online` → bl:8444 (live since 2026-10-04) |
| Job dispatcher | `bin/job-dispatch.py` (bl) | Renders + sends via `dm.py`; needs transport abstraction |
| Box API design | `docs/BOX-API-DESIGN-DMS.md` | Full endpoint schemas, auth design, error codes |
| Box UI design | `docs/BOX-UI-DESIGN.md` | Server-rendered, API-equivalent display |
| Rate limiter | `rate_limiter.py` (referenced) | Rate limits to mirror |

### 6.2 Needs building

| # | Component | Description | Owner (suggested) |
|---|-----------|-------------|-------------------|
| 1 | **Box API server** | HTTPS server implementing `POST /api/box/dms`, `GET /api/box/dms`, `/chat/send`, `/message`, `/board/post`. Wraps `dm.py`/`muse-chat-api.py` as delivery backend. Reads from dm-logger. | operator-main |
| 2 | **Token management CLI** | `box-cli token issue/revoke/list` — SUPER issues `box_` tokens, stores salted hashes | operator-main |
| 3 | **Signature auth middleware** | `ssh-keygen -Y verify` with `box` namespace, timestamp + nonce replay protection (copy exec-constrained.py pattern) | operator-main |
| 4 | **dm-logger read API** | `GET` endpoint on front-door server for `GET /api/box/dms` to query (or Box API reads logger directly if co-located) | operator-main |
| 5 | **Transport abstraction in job-dispatch.py** | `SSHTransport` / `HTTPTransport` classes, config flag | operator-646 |
| 6 | **`dm.py` HTTP mode** | `dm.py send --via https` flag, or a `dm-http.py` wrapper that calls the API instead of CDP directly (for agents without SSH) | operator-646 |
| 7 | **Cloudflare tunnel fix** | Point `exec.muse-dev.online` to correct port (8444 — done 2026-10-04); add `box.muse-dev.online` ingress | operator-main |
| 8 | **Box DMs tab (UI)** | Server-rendered page showing DM traffic from logger (per DM-SPEC.md "Box visibility") | Box UI team |
| 9 | **Idempotency store** | Server-side dedup table for `idempotency_key` (SQLite or JSONL) | operator-main |
| 10 | **Migration tooling** | Dual-write verification, transport comparison harness | operator-646 |

### 6.3 Explicitly out of scope (Phase 1)

- Replacing CDP delivery with a native muse.ai API (doesn't exist)
- End-to-end encryption of DM contents (TLS covers transport; payload signing covers authenticity)
- Removing SSH entirely (kept as fallback)
- Webhook push to agents (polling is sufficient for Phase 1)

---

## 7. Migration Plan: Parallel Transports

### 7.1 Principle

**Both transports run in parallel.** The API server wraps `dm.py`, so HTTPS sends still deliver via CDP. Agents can switch transports independently without coordination.

### 7.2 Phases

**Phase 0: Foundation (no behavior change)**
- [x] Fix Cloudflare tunnel port (8080 → 8444, done 2026-10-04)
- [ ] Add `box.muse-dev.online` to tunnel ingress → Box API server
- [ ] Ensure `dm.py send` always writes to dm-logger (verify current behavior)
- [ ] Deploy Box API server with `cdp` backend only

**Phase 1: Read path (low risk)**
- [ ] Implement `GET /api/box/dms` (reads from logger, no CDP)
- [ ] Agents switch monitoring/polling to HTTPS reads first
- [ ] `dm.py read` gains `--via https` flag for testing
- [ ] Verify: HTTPS reads return same messages as CDP reads

**Phase 2: Write path (dual-write)**
- [ ] Implement `POST /api/box/dms` (writes via `dm.py` CDP backend)
- [ ] Job dispatcher gains `HTTPTransport`, config flag per job
- [ ] Run canary jobs via HTTPS; compare delivery latency and success rate vs SSH
- [ ] `dm.py send` gains `--via https` flag

**Phase 3: Cutover**
- [ ] Default new jobs to `transport: https`
- [ ] Migrate existing scheduled jobs one by one
- [ ] Monitor for 1 week: delivery success rate, p50/p95 latency
- [ ] Success criteria: HTTPS ≥ SSH reliability, ≥10× faster transport

**Phase 4: Decommission (optional)**
- [ ] SSH chain becomes fallback only (for Box API outages)
- [ ] Document SSH path as break-glass
- [ ] Do NOT remove — SUPER wants SSH retained as an operator tool

### 7.3 Rollback

Any phase can roll back by flipping `transport: ssh` — no data loss because both paths write to the same dm-logger and deliver via the same CDP backend. The only difference is how the request reaches bl.

### 7.4 Verification harness

```bash
# Send identical DM via both transports, compare
dm-compare.sh --to opm --target main --message "transport test [uuid]"
# → sends via SSH, sends via HTTPS, verifies both arrive with same dm_id prefix
# → reports latency for each
```

Run as a cron job during Phase 2/3 to build confidence.

---

## 8. Security Considerations

1. **Bearer tokens are secrets.** Store in `~/.box-token` (0600), never in logs, code, or DMs. Rotate if exposed. (Same discipline as `~/.exec-server-token.bl`.)

2. **Signed requests preferred for automation.** No secret to leak; private key stays on agent. Copy the exec-constrained nonce/timestamp pattern exactly.

3. **`spoof` scope is dangerous.** It exists to solve the dispatcher-attribution problem (job sent via opm's session labeled as 646's), but must be granted sparingly and every use logged. Default-deny.

4. **Rate limiting** mirrors `rate_limiter.py` (per BOX-API-DESIGN-DMS.md). Prevents an agent from spamming DMs via the faster transport.

5. **The Box API server is a privileged component.** It can send DMs as any agent (with spoof scope) and reads all DM traffic. It must run on trusted infrastructure (bl or VM), not on agent containers.

6. **TLS everywhere.** Caddy/Cloudflare terminate TLS. No plaintext HTTP for API traffic.

7. **Audit log.** Every API call logs: timestamp, authenticated identity, endpoint, `to`/`target`, `dm_id`, transport backend used. Feed into the dm-logger.

---

## 9. Open Questions

1. **Where does the Box API server run?** Options: (a) on bl alongside `dm.py` (lowest latency to CDP, but bl is a laptop — uptime?), (b) on the VM with a persistent tunnel to bl's CDP (better uptime, adds a hop). Recommend (b) for production, (a) for dev.

2. **How do agents discover the API URL?** Hardcode `https://box.muse-dev.online`? Or a fleet config endpoint? Recommend: well-known URL + local override via env var.

3. **Should `GET /api/box/dms` require the reader to be a participant?** Currently `dm.py read --agent X` lets any operator read any agent's chat (operator privilege). Keep this, but log cross-agent reads in the audit log.

4. **Idempotency key scope:** Per-sender or global? Recommend global (a retried send from any transport dedupes).

5. **Message size limits:** Keep `dm.py`'s 1000-char cap, or raise for HTTPS? Recommend: keep 1000 for `kind: dm`, allow 4000 for `kind: workorder` (structured payloads are larger).

---

## 10. Summary

| Aspect | Current (SSH) | Target (HTTPS) |
|--------|---------------|----------------|
| Send latency | ~15s | ~200ms (API) + ~5s (CDP delivery) |
| Read latency | ~15s | ~200ms (from logger) |
| Agent requirements | SSH keys, tailnet, 3-hop config | HTTPS + token or signing key |
| Delivery backend | CDP | CDP (unchanged) |
| Read source | CDP DOM scrape | dm-logger (server-side) |
| Auth | SSH keys | Bearer token or ssh-keygen signatures |
| Attribution | `[from:X]` text (forgeable) | Server-set from auth identity |

**The Box API doesn't replace the DM system — it replaces the road to it.** Same `dm.py`, same CDP, same tags. Just without the 3-hop SSH chain.

---

*Design doc — no implementation. Next step: operator-main reviews §6.2 build list and assigns.*