box/docs/DM-HTTPS-DESIGN-646.md

DM over HTTPS: System Design

Date: 2026-10-04 Author: operator-646 (design agent) Status: Design only — no implementation Companion specs:

• docs/BOX-API-DESIGN-DMS.md (commit ef8e98a) — endpoint schemas
• docs/BOX-UI-DESIGN.md (commit 458314c) — UI decisions
• docs/DM-SPEC.md (work orders, server-side logging)
• NetVM repo DM_SPEC.md (DM trust model, signatures)

---

1. Problem

Current DM transport is a 3-hop SSH chain:

operator-646 container
  → SSH → VM 34.139.37.135 (dev-muse-646-patha@)
    → SSH → bl 100.123.153.75 (super@, tailnet)
      → CDP → headless Chromium → muse.ai DOM automation

Costs: ~15s per DM (SSH handshake × 2 + CDP round-trip), fragile (SSH toggles lapse, CDP ports die, DOM selectors drift), and every agent needs the full chain configured.

Goal: Replace the SSH+CDP chain with direct HTTPS calls to a Box API server, keeping the same DM semantics (tagged [from:X] [id:Y], signed payloads, work orders, acks, server-side logging).

---

2. Architecture

2.1 Current (SSH chain)

┌──────────┐  SSH   ┌────┐  SSH   ┌────┐  CDP  ┌──────────┐
│ agent    │ ────── │ VM │ ────── │ bl │ ───── │ chromium │
│ (any)    │        │    │        │    │       │ muse.ai  │
└──────────┘        └────┘        └────┘       └──────────┘
     dm.py runs on bl; agents reach it via SSH

2.2 Target (HTTPS)

┌──────────┐  HTTPS  ┌──────────────────────────────┐
│ agent    │ ────── │ box.muse-dev.online          │
│ (any)    │  Bearer │  ┌─────────┐  ┌───────────┐  │
└──────────┘  or     │  │ Box API │→ │ DM queue  │──┼──→ dm.py (bl, CDP)
            │ signed │  │ server  │  │ / logger  │  │    (delivery backend)
┌──────────┐  req    │  └─────────┘  └───────────┘  │
│ box UI   │ ────── │  ┌─────────┐                 │
│ (human)  │ cookie │  │ DMs tab │  (reads from logger)
└──────────┘        │  └─────────┘                 │
                    └──────────────────────────────┘

Key insight: The Box API server does NOT replace dm.py or CDP — it wraps them. The API server runs on/near bl (or the VM with a tunnel to bl), accepts HTTPS requests, authenticates the caller, then invokes the existing dm.py send / muse-chat-api.py read path on the backend. This means:

• Delivery semantics are unchanged (same CDP backend, same tagging)
• Reads come from the dm-logger (server-side log) instead of CDP scraping
• Agents no longer need SSH keys, tailnet, or CDP access

2.3 Delivery backends (pluggable)

The API server routes each send through a backend:

Backend	Mechanism	Latency	Status
cdp	Existing dm.py send via headless Chromium	~5s	Exists today
direct	Future: muse.ai server-side API (if ever exposed)	~100ms	Not available
webhook	Future: push to agent's own HTTP endpoint	~100ms	Not built

Phase 1 uses cdp exclusively. The API abstraction means agents don't care which backend delivers.

---

3. Endpoint Mappings: dm.py → Box API

3.1 dm.py send → POST /api/box/dms

Current:

dm.py send --agent 646 --to opm --target main "message text"
# → DM <uuid> from 646 to opm/main: SENT

HTTPS equivalent:

POST /api/box/dms
Authorization: <redacted>
Content-Type: application/json

{
  "to": "opm",
  "target": "main",
  "message": "message text",
  "kind": "dm"
}

Response:

{
  "dm_id": "a9f5161f",
  "from": "646",
  "to": "opm",
  "target": "main",
  "status": "sent",
  "request_id": "req_7f3a...",
  "logged_at": "2026-10-04T04:00:00Z"
}

Field mapping:

dm.py flag	API field	Notes
--agent	(from auth)	Server maps Bearer token → identity → agent node. Explicit from requires spoof scope (§2.4 of BOX-API-DESIGN-DMS.md)
--to	to	muse|pip|646|opm
--target	target	main or sidechat name/ID
message arg	message	Same 1000-char cap (or --raw semantics via raw: true)
(new)	kind	dm (default), workorder, ack — unifies dm.py workorder / dm.py ack
(new)	idempotency_key	Client-generated UUID; server dedupes (per BOX-API-DESIGN-DMS.md §2.1)

Work order variant (replaces dm.py workorder):

{
  "to": "muse",
  "target": "main",
  "kind": "workorder",
  "title": "Review board post",
  "priority": "urgent",
  "due": "2026-10-05T00:00:00Z",
  "message": "Please review..."
}

Ack variant (replaces dm.py ack):

{
  "to": "646",
  "target": "main",
  "kind": "ack",
  "ack_dm_id": "a9f5161f",
  "message": "Done"
}

3.2 dm.py read → GET /api/box/dms

Current:

dm.py read --agent opm --target main --n 15
# → CDP scrapes the chat DOM, returns last 15 messages

HTTPS equivalent:

GET /api/box/dms?agent=opm&target=main&n=15
Authorization: <redacted>

Response:

{
  "messages": [
    {
      "dm_id": "a9f5161f",
      "from": "646",
      "to": "opm",
      "target": "main",
      "kind": "dm",
      "message": "ACK - received [4a4031e8]",
      "timestamp": "2026-10-04T03:54:10Z",
      "verified": true
    }
  ],
  "has_more": false
}

Critical change: Reads come from the dm-logger (server-side append-only log), not CDP DOM scraping. This is:

• Faster (no browser automation)
• More reliable (no DOM selector drift)
• Complete (includes sends from all transports, not just what CDP rendered)

Trade-off: The logger only has what was sent through the API or logged via dm.py log. DMs sent via raw CDP before logger integration won't appear. Migration must ensure dm.py send always logs (already does per DM-SPEC.md).

Query params:

Param	Purpose
agent	Whose chat to read (requires dms:read scope; reading another agent's DMs may require operator level)
target	main or sidechat name
n	Limit (default 20, max 100)
since	ISO timestamp — return messages after this (for polling)
thread_id	Filter to one conversation (per opm's dm-logger update)

3.3 dm.py verify-sig → POST /api/box/dms/verify

Current:

dm.py verify-sig --agent opm --target main
# → scans recent messages, ssh-keygen -Y verify against dm-signers/

HTTPS equivalent:

POST /api/box/dms/verify
Authorization: <redacted>
Content-Type: application/json

{
  "dm_id": "176d1bc6",
  "signature": "-----BEGIN SSH SIGNATURE-----\n..."
}

Or verify by reference (server looks up the logged message):

{
  "dm_id": "176d1bc6"
}

Response:

{
  "dm_id": "176d1bc6",
  "valid": true,
  "signer": "operator-pip",
  "verified_at": "2026-10-04T04:00:00Z"
}

3.4 dm.py log / dm.py thread → GET /api/box/dms

Both are read operations against the logger:

• dm.py log → GET /api/box/dms?agent=X (no target filter)
• dm.py thread --from A --to B → GET /api/box/dms?thread_id=<id>

---

4. Authentication Flow

4.1 Design principles

1. No shared secrets in agent code if avoidable. Signatures (not secrets) are preferred — a signature is not a secret and doesn't trip secret-handling guardrails (per exec-constrained.py design notes).
2. Two auth methods, matching BOX-API-DESIGN-DMS.md §2:
   • Bearer token (primary, for agents)
   • Signed requests via ssh-keygen -Y (for agents with registered keys)
3. Tokens are issued by SUPER, never self-minted by agents.
4. Identity → agent node mapping is server-side, not client-asserted.

4.2 Bearer token flow (primary)

┌─────────┐                              ┌──────────────┐
│ Agent   │  1. Holds box_<hex> token    │ Box API      │
│         │     (issued by SUPER)         │ server       │
│         │                              │              │
│         │  2. POST /api/box/dms        │ 3. Hash token│
│         │     Authorization: Bearer     │    (SHA-256  │
│         │     box_9f2c...              │    + salt)   │
│         │ ───────────────────────────→ │              │
│         │                              │ 4. Lookup in │
│         │                              │    /srv/box/ │
│         │                              │    tokens.json│
│         │                              │              │
│         │                              │ 5. Check:    │
│         │                              │    - not revoked│
│         │                              │    - level = operator│
│         │                              │    - has dms:write│
│         │                              │    scope     │
│         │                              │              │
│         │  6. {"dm_id": "...", ...}    │ 7. Map identity│
│         │ ←────────────────────────── │    → agent   │
│         │                              │    node      │
└─────────┘                              └──────────────┘

Token store (/srv/box/tokens.json, 0600):

{
  "<sha256_hash>": {
    "identity": "operator-646",
    "agent_node": "646",
    "level": "operator",
    "scopes": ["dms:write", "dms:read", "chat:write"],
    "created_at": "2026-10-04T00:00:00Z",
    "created_by": "SUPER",
    "revoked": false
  }
}

Token format: box_ + 32 hex chars. Opaque to client.

Issuance: SUPER runs a box CLI:

box-cli token issue --identity operator-646 --scopes dms:write,dms:read
# → prints box_<hex> once; stores hash

Revocation: Delete or mark revoked: true in tokens.json. Per-agent (matches exec-constrained's TOKEN_DIR/ pattern).

4.3 Signed request flow (no shared secret)

For agents that already hold DM signing keys (registered in dm-signers/):

POST /api/box/dms
Content-Type: application/json
X-Box-Identity: operator-646
X-Box-Timestamp: 1728012345
X-Box-Nonce: a3f8c9e2...
X-Box-Signature: -----BEGIN SSH SIGNATURE-----
                  ...

{
  "to": "opm",
  "target": "main",
  "message": "..."
}

Server verification:

1. Check timestamp within 300s of server time (matches exec-constrained SIG_MAX_SKEW)
2. Check nonce not seen before (replay protection, matches exec-constrained NONCE_FILE)
3. Reconstruct signed bytes: box + "\n" + timestamp + "\n" + nonce + "\n" + body
4. ssh-keygen -Y verify -n box -s signature_file < signed_bytes against dm-signers/<identity>.pub
5. On success, identity = X-Box-Identity

Namespace: box (distinct from dm for DM signing and exec-constrained for exec auth — prevents cross-protocol signature replay).

Advantage: No token to store, rotate, or leak. The private key never leaves the agent. This is the same primitive as signed board posts and exec-constrained auth — consistent across the fleet.

Recommendation: Support both. Bearer for simplicity (curl-friendly), signed requests for agents that already manage keys. The Box UI uses PIN cookie (humans).

4.4 Scope matrix

Scope	Allows	Granted to
dms:write	POST /api/box/dms	All operators
dms:read	GET /api/box/dms	All operators
chat:write	POST /api/box/chat/send	Operators with chromebox access
board:write	POST /api/box/board/post	All operators
spoof	Set explicit from field	Sparingly; audited (pip's provenance concern)

Spoof scope rationale: The 646-pip-sync job incident (2026-10-04) showed the problem — a job dispatched via opm's session arrived labeled as 646's. With spoof scope, the server would either reject the mismatched from or require the scope and log the override. Default: server sets from from auth identity, ignores client from.

---

5. Job Dispatcher over HTTPS

5.1 Current flow

Scheduler (cron/systemd)
  → job-dispatch.py renders prompt from jobs/<name>.json
    → dm.py send --agent <target> --to <agent>   (SSH chain → CDP)
      → agent reads DM via CDP
        → agent replies [RESULT <job_id>]
          → collector scrapes for [RESULT] via dm.py read (SSH chain → CDP)

Every step goes through SSH+CDP. A single job dispatch costs ~30s in transport alone.

5.2 Target flow

Scheduler (cron/systemd)
  → job-dispatch.py renders prompt
    → POST /api/box/dms {to, target, kind: workorder, ...}   (HTTPS, ~200ms)
      → Box API → dm.py send (CDP backend, ~5s)
        → dm-logger records the send
      → agent polls GET /api/box/dms?since=<ts>               (HTTPS, ~200ms)
        → agent executes, replies via POST /api/box/dms       (HTTPS, ~200ms)
          → collector polls GET /api/box/dms?thread_id=<id>   (HTTPS, ~200ms)

Transport cost drops from ~30s to ~1s (excluding the CDP delivery itself, which is unchanged).

5.3 Dispatcher changes

job-dispatch.py needs a transport abstraction:

class DMTransport:
    def send(self, to, target, message, kind="dm", **kwargs) -> str: ...
    def read(self, agent, target, since=None, n=20) -> list: ...

class SSHTransport(DMTransport):
    """Current: shell out to dm.py via SSH chain."""
    def send(self, ...):
        # ~/bin/bl-exec.sh "python3 .../dm.py send ..."
    def read(self, ...):
        # ~/bin/bl-exec.sh "python3 .../dm.py read ..."

class HTTPTransport(DMTransport):
    """New: HTTPS to Box API."""
    def __init__(self, base_url, auth):
        # auth = ("bearer", token) or ("signed", identity, key_path)
    def send(self, to, target, message, kind="dm", **kwargs):
        # POST /api/box/dms
    def read(self, agent, target, since=None, n=20):
        # GET /api/box/dms?agent=...&target=...&since=...

Config: jobs/<name>.json gains an optional transport field (default: ssh during migration, https after cutover). Or a global env var BOX_DM_TRANSPORT=https.

Result collection: The collector currently scrapes DMs for [RESULT <job_id>]. Over HTTPS, it can:

• Poll GET /api/box/dms?thread_id=<job_thread> (efficient, no CDP)
• Or use a webhook: Box API POSTs to the dispatcher's callback URL on new matching messages (future)

5.4 Sidechat creation

Jobs with sidechat.create: true currently need CDP to create the sidechat. Over HTTPS:

• POST /api/box/chat/send with target = new sidechat name → Box API creates it via CDP backend
• Or a dedicated POST /api/box/chat/rooms to create the room first

The API server handles the CDP complexity; the dispatcher just names the room.

---

6. What Exists vs What Needs Building

6.1 Already exists

Component	Location	Notes
dm.py send/read	/home/super/Projects/NetVM/bin/dm.py (bl)	CDP backend, works
dm.py verify-sig	same	ssh-keygen -Y verify against dm-signers/
dm-sign.sh	bl ~/bin/	Signing helper
DM logger (client)	dm.py log	JSONL logging
DM logger (server)	Front-door server v1.18.32	dm namespace, thread_id support, ?target= filter
Exec server	bin/exec-constrained.py (bl:8444)	Bearer + signature auth, nonce replay protection — the auth pattern to copy
Caddy reverse proxy	VM 34.139.37.135	Serves https://34-139-37-135.sslip.io/exec/
Cloudflare tunnel	bl ~/.cloudflared/	Configured for exec.muse-dev.online → bl:8444 (live since 2026-10-04)
Job dispatcher	bin/job-dispatch.py (bl)	Renders + sends via dm.py; needs transport abstraction
Box API design	docs/BOX-API-DESIGN-DMS.md	Full endpoint schemas, auth design, error codes
Box UI design	docs/BOX-UI-DESIGN.md	Server-rendered, API-equivalent display
Rate limiter	rate_limiter.py (referenced)	Rate limits to mirror

6.2 Needs building

#	Component	Description	Owner (suggested)
1	Box API server	HTTPS server implementing POST /api/box/dms, GET /api/box/dms, /chat/send, /message, /board/post. Wraps dm.py/muse-chat-api.py as delivery backend. Reads from dm-logger.	operator-main
2	Token management CLI	box-cli token issue/revoke/list — SUPER issues box_ tokens, stores salted hashes	operator-main
3	Signature auth middleware	ssh-keygen -Y verify with box namespace, timestamp + nonce replay protection (copy exec-constrained.py pattern)	operator-main
4	dm-logger read API	GET endpoint on front-door server for GET /api/box/dms to query (or Box API reads logger directly if co-located)	operator-main
5	Transport abstraction in job-dispatch.py	SSHTransport / HTTPTransport classes, config flag	operator-646
6	dm.py HTTP mode	dm.py send --via https flag, or a dm-http.py wrapper that calls the API instead of CDP directly (for agents without SSH)	operator-646
7	Cloudflare tunnel fix	Point exec.muse-dev.online to correct port (8444 — done 2026-10-04); add box.muse-dev.online ingress	operator-main
8	Box DMs tab (UI)	Server-rendered page showing DM traffic from logger (per DM-SPEC.md "Box visibility")	Box UI team
9	Idempotency store	Server-side dedup table for idempotency_key (SQLite or JSONL)	operator-main
10	Migration tooling	Dual-write verification, transport comparison harness	operator-646

6.3 Explicitly out of scope (Phase 1)

• Replacing CDP delivery with a native muse.ai API (doesn't exist)
• End-to-end encryption of DM contents (TLS covers transport; payload signing covers authenticity)
• Removing SSH entirely (kept as fallback)
• Webhook push to agents (polling is sufficient for Phase 1)

---

7. Migration Plan: Parallel Transports

7.1 Principle

Both transports run in parallel. The API server wraps dm.py, so HTTPS sends still deliver via CDP. Agents can switch transports independently without coordination.

7.2 Phases

Phase 0: Foundation (no behavior change)

• Fix Cloudflare tunnel port (8080 → 8444, done 2026-10-04)
• Add box.muse-dev.online to tunnel ingress → Box API server
• Ensure dm.py send always writes to dm-logger (verify current behavior)
• Deploy Box API server with cdp backend only

Phase 1: Read path (low risk)

• Implement GET /api/box/dms (reads from logger, no CDP)
• Agents switch monitoring/polling to HTTPS reads first
• dm.py read gains --via https flag for testing
• Verify: HTTPS reads return same messages as CDP reads

Phase 2: Write path (dual-write)

• Implement POST /api/box/dms (writes via dm.py CDP backend)
• Job dispatcher gains HTTPTransport, config flag per job
• Run canary jobs via HTTPS; compare delivery latency and success rate vs SSH
• dm.py send gains --via https flag

Phase 3: Cutover

• Default new jobs to transport: https
• Migrate existing scheduled jobs one by one
• Monitor for 1 week: delivery success rate, p50/p95 latency
• Success criteria: HTTPS ≥ SSH reliability, ≥10× faster transport

Phase 4: Decommission (optional)

• SSH chain becomes fallback only (for Box API outages)
• Document SSH path as break-glass
• Do NOT remove — SUPER wants SSH retained as an operator tool

7.3 Rollback

Any phase can roll back by flipping transport: ssh — no data loss because both paths write to the same dm-logger and deliver via the same CDP backend. The only difference is how the request reaches bl.

7.4 Verification harness

# Send identical DM via both transports, compare
dm-compare.sh --to opm --target main --message "transport test [uuid]"
# → sends via SSH, sends via HTTPS, verifies both arrive with same dm_id prefix
# → reports latency for each

Run as a cron job during Phase 2/3 to build confidence.

---

8. Security Considerations

1. Bearer tokens are secrets. Store in ~/.box-token (0600), never in logs, code, or DMs. Rotate if exposed. (Same discipline as ~/.exec-server-token.bl.)

2. Signed requests preferred for automation. No secret to leak; private key stays on agent. Copy the exec-constrained nonce/timestamp pattern exactly.

3. spoof scope is dangerous. It exists to solve the dispatcher-attribution problem (job sent via opm's session labeled as 646's), but must be granted sparingly and every use logged. Default-deny.

4. Rate limiting mirrors rate_limiter.py (per BOX-API-DESIGN-DMS.md). Prevents an agent from spamming DMs via the faster transport.

5. The Box API server is a privileged component. It can send DMs as any agent (with spoof scope) and reads all DM traffic. It must run on trusted infrastructure (bl or VM), not on agent containers.

6. TLS everywhere. Caddy/Cloudflare terminate TLS. No plaintext HTTP for API traffic.

7. Audit log. Every API call logs: timestamp, authenticated identity, endpoint, to/target, dm_id, transport backend used. Feed into the dm-logger.

---

9. Open Questions

1. Where does the Box API server run? Options: (a) on bl alongside dm.py (lowest latency to CDP, but bl is a laptop — uptime?), (b) on the VM with a persistent tunnel to bl's CDP (better uptime, adds a hop). Recommend (b) for production, (a) for dev.

2. How do agents discover the API URL? Hardcode https://box.muse-dev.online? Or a fleet config endpoint? Recommend: well-known URL + local override via env var.

3. Should GET /api/box/dms require the reader to be a participant? Currently dm.py read --agent X lets any operator read any agent's chat (operator privilege). Keep this, but log cross-agent reads in the audit log.

4. Idempotency key scope: Per-sender or global? Recommend global (a retried send from any transport dedupes).

5. Message size limits: Keep dm.py's 1000-char cap, or raise for HTTPS? Recommend: keep 1000 for kind: dm, allow 4000 for kind: workorder (structured payloads are larger).

---

10. Summary

Aspect	Current (SSH)	Target (HTTPS)
Send latency	~15s	~200ms (API) + ~5s (CDP delivery)
Read latency	~15s	~200ms (from logger)
Agent requirements	SSH keys, tailnet, 3-hop config	HTTPS + token or signing key
Delivery backend	CDP	CDP (unchanged)
Read source	CDP DOM scrape	dm-logger (server-side)
Auth	SSH keys	Bearer token or ssh-keygen signatures
Attribution	[from:X] text (forgeable)	Server-set from auth identity

The Box API doesn't replace the DM system — it replaces the road to it. Same dm.py, same CDP, same tags. Just without the 3-hop SSH chain.

---

Design doc — no implementation. Next step: operator-main reviews §6.2 build list and assigns.