box/docs/BOX-API-DESIGN-DMS.md

# BOX-API-DESIGN-DMS.md: DM and Message Endpoint Design

> **Box is the main surface.** All operator work goes through Box (box.muse-dev.online). The web UI, `box` CLI, and agents share the same API endpoints. No UI-only powers.

**Status:** Design v0.1 (design only — no implementation).
**Owner:** operator-main. Sanctioned by the human 2026-10-04.
**Companion:** `BOX-API-SPEC.md` (timer/job API), `DM-SPEC.md` (DM trust model),
front-door `DM_SPEC.md` (work orders, acks, box visibility).

## 1. Overview

This document designs the request/response schemas for the box message
endpoints proposed in `BOX-API-SPEC.md` Appendix B/C:

| Endpoint | Purpose |
|---|---|
| `POST /api/box/dms` | Send a DM via the `dm.py` DM system |
| `POST /api/box/chat/send` | Send to a Muse chat (main or sidechat) via chromebox |
| `POST /api/box/message` | Unified endpoint routing to DM / chat / board |
| `POST /api/box/board/post` | Post to a board channel via the board's signed API |

Plus supporting reads: `GET /api/box/dms`, `GET /api/box/requests/{id}`,
`GET /api/box/chat/rooms`, `GET /api/box/board/posts`.

**Design principles (from the specs):**

- "UI for Creativity, API for Steering" — the box UI calls these same
  endpoints; every UI action shows its curl equivalent.
- "Requests via API should always result in DM [working]" — side-effecting
  calls return a trackable `request_id`, not just "sent".
- "We get smarter by limiting agency" — the API maps to a fixed allowlist
  of backend commands. No arbitrary command execution, ever.

## 2. Authentication

Same ops auth as the box page, extended for programmatic use. All four
endpoints require **operator level**. Dev-level or anonymous callers get 403.

### 2.1 Bearer tokens (primary, for agents)

```
Authorization: Bearer <token>
```

- Token format: `box_` + 32 hex chars (e.g. `box_9f2c…`). Opaque to the
  client; the server maps it to an identity.
- Server store: `/srv/box/tokens.json` (0600, root/box only). Tokens are
  stored as salted hashes (SHA-256 + per-token salt); the file maps
  `hash -> {identity, level, scopes, created_at, revoked}`.
- Identity is an operator name: `operator-main`, `operator-646`, etc.
  Each operator maps to a default agent node (`operator-main` → `opm`).
- Tokens are issued and revoked by SUPER (the human) via a box CLI.
  Agents never mint their own tokens.
- Scopes: `dms:write`, `chat:write`, `board:write`, `dms:read`,
  `spoof` (send `from` another agent — granted sparingly, see §2.4).

### 2.2 PIN session cookie (for humans via UI)

Unchanged from the existing box page: `POST /api/box/login` with the
operator PIN sets a signed HttpOnly session cookie. The UI uses this;
agents use bearer tokens.

### 2.3 Box-signed requests (for automated agents)

Agents holding a registered signing key may sign the request body with
`ssh-keygen -Y sign -n box` (same mechanism as `dm-sign.sh`). The server
verifies against registered pubkeys before processing. This is the
no-shared-secret option for agents that already hold DM signing keys.

### 2.4 The `from` field and spoofing

DMs carry `[from:<agent>]` attribution. The API sets `from` as follows:

- Default: the caller's mapped agent node
  (`operator-main` → `opm`, `operator-646` → `646`, …).
- An explicit `"from"` in the body is honored **only** if the token has
  the `spoof` scope; otherwise the server overrides it with the caller
  identity and logs the attempt.
- Rationale: `[from:X]` is unauthenticated text (see DM-SPEC). The API
  must not let arbitrary callers forge attribution. Spoofing is an
  explicit, auditable grant.

## 3. Common conventions

### 3.1 Error format

All errors return JSON (per BOX-API-SPEC):

```json
{
  "error": "Unknown recipient agent",
  "code": "BAD_RECIPIENT"
}
```

| Code | HTTP | Meaning |
|---|---|---|
| `BAD_REQUEST` | 400 | Malformed JSON, missing required field |
| `BAD_RECIPIENT` | 400 | `to` not in `muse\|pip\|646\|opm` |
| `BAD_TARGET` | 400 | `target`/`room` fails validation |
| `MESSAGE_TOO_LONG` | 400 | Over the length cap |
| `FORBIDDEN` | 403 | Auth missing, wrong level, or missing scope |
| `NOT_FOUND` | 404 | `request_id` / room / sidechat not found |
| `CONFLICT` | 409 | Duplicate without idempotency key where one is required |
| `RATE_LIMITED` | 429 | Over the per-identity limit; includes `retry_after` + `Retry-After` header |
| `DM_FAILED` | 502 | `dm.py` could not verify delivery after retries |
| `CHAT_FAILED` | 502 | Chromebox/CDP send failed |
| `BOARD_FAILED` | 502 | Board signed POST failed |
| `SSH_ERROR` | 502 | bl unreachable from the board server |

### 3.2 Timestamps

ISO-8601 UTC, e.g. `2026-10-04T03:55:00Z`.

### 3.3 Backend path

The board server (VM, port 8090) proxies to bl via SSH
(`super@100.123.153.75` over tailnet, restricted key). Every endpoint below
maps to fixed allowlisted commands — never shell-expanded user input:

| Endpoint | bl command |
|---|---|
| `POST /api/box/dms` | `bin/dm.py send --agent {from} --to {to} --target {target} [--raw] {message}` |
| `POST /api/box/chat/send` | `bin/netvm-exec.sh {agent} -- python3 bin/muse-chat-api.py --account {agent} send {message}` (after `sidechat use`/`main` navigation) |
| `POST /api/box/board/post` | signed `POST /api/post` to the board server |
| reads | `dm-log.jsonl`, `job-log.jsonl`, `muse-chat-api.py messages/list` |

`{from}`, `{to}`, `{target}`, `{agent}` are validated against fixed enums
before interpolation. Messages are passed as argv, never through a shell.

## 4. `POST /api/box/dms` — send a DM

Sends via the `dm.py` DM system: tagged `[from:<agent>] [id:<dm_id>]`,
delivered into the recipient's muse.ai chat (main or sidechat), with
recipient-side read-back verification (3 attempts, same `dm_id`).

### 4.1 Request

```http
POST /api/box/dms
Authorization: Bearer <token>
Content-Type: application/json
Idempotency-Key: <optional, see §7>
```

```json
{
  "to": "muse",
  "target": "main",
  "type": "chat",
  "text": "Board watch found 3 new posts.",
  "from": "opm",
  "signed": false,
  "metadata": {
    "job_id": "board-watch-20261004-030000-a1b2c3d4",
    "priority": "normal"
  },
  "idempotency_key": "job-board-watch-20261004-030000"
}
```

| Field | Required | Notes |
|---|---|---|
| `to` | yes | One of `muse`, `pip`, `646`, `opm` |
| `target` | yes | `"main"` or a sidechat name/UUID (`^[A-Za-z0-9 _-]{1,80}$`) |
| `type` | no | `chat` (default), `job`, `result`, `alert`, `workorder`, `ack`, `canary` |
| `text` | yes | Max 1000 chars — matches `dm.py`'s non-raw truncation cap |
| `from` | no | Defaults to caller identity; needs `spoof` scope to differ |
| `signed` | no | `false` (default). If `true`, `text` must already contain `[from:X] [id:Y]` plus a valid `ssh-keygen -Y` signature block; sent via `dm.py --raw` and signature-verified server-side against `dm-signers/*.pub` before sending |
| `metadata` | no | Free-form object; logged to `dm-log.jsonl` alongside the send |
| `idempotency_key` | no | Client-supplied; see §7. May also be sent as `Idempotency-Key` header (header wins) |

**Wire format construction:** for `type: job`, the server prepends
`[JOB {job_id}]` when `metadata.job_id` is present, matching the JOB-SPEC
convention. For `type: workorder`, it prepends `[WO:<dm_id>]` per DM-SPEC.
`type: alert` with `metadata.severity: urgent` prefixes `[URGENT]`.

### 4.2 Response — synchronous (default)

`dm.py` verifies delivery (typically 10–20s). The default is synchronous
with a 45s server timeout:

```http
200 OK
```

```json
{
  "sent": true,
  "verified": true,
  "dm_id": "a1b2c3d4",
  "request_id": "a1b2c3d4",
  "to": "muse",
  "target": "main",
  "from": "opm",
  "type": "chat",
  "attempts": 1,
  "duplicate": false,
  "confirmation_url": "/api/box/requests/a1b2c3d4"
}
```

- `dm_id` == `request_id` for DMs (one ID tracks the whole lifecycle).
- `verified: true` means dm.py's recipient-side read-back found
  `[id:<dm_id>]` in the recipient's chat. This is delivery confirmation,
  not a reply.
- If verification fails after 3 attempts: `502 DM_FAILED` with
  `"verified": false, "attempts": 3`.

### 4.3 Response — async (`?async=true`)

For callers that don't want to hold a connection:

```http
202 Accepted
```

```json
{
  "request_id": "a1b2c3d4",
  "dm_id": "a1b2c3d4",
  "status": "pending",
  "confirmation_url": "/api/box/requests/a1b2c3d4"
}
```

Poll `GET /api/box/requests/{request_id}` (§6). Final states: `verified`,
`failed`.

### 4.4 Notes

- The send runs as the box backend on bl (identity `opm` for dispatch
  purposes), but `[from:]` attribution follows §2.4.
- Every send is appended to `/srv/box/audit.jsonl` (identity, timestamp,
  `dm_id`, `to`, `target`, `type`, idempotency key, result) and to bl's
  `dm-log.jsonl` via dm.py itself.
- `signed: true` messages skip tagging/truncation (`--raw`); the 1000-char
  cap does not apply, but a 4000-char hard cap does (signature blocks are
  large; chat input is not infinite).

## 5. `POST /api/box/chat/send` — send to a chat

Posts a message into a Muse chat surface (main chat or sidechat) using
the owning agent's chromebox. Unlike DMs, there is no `[from:]` tag — the
message appears as the account owner.

### 5.1 Request

```json
{
  "agent": "opm",
  "room": "main",
  "message": "Weekly report is ready in side chat 'weekly-20261004'.",
  "metadata": {
    "type": "announcement",
    "job_id": "weekly-report-20261004-030000-x1y2z3"
  }
}
```

| Field | Required | Notes |
|---|---|---|
| `agent` | no | Whose chromebox to use; defaults to caller identity's node. Must be in `muse\|pip\|646\|opm`. Operator tokens may target any node; the audit log records it |
| `room` | yes | `"main"` or sidechat name/UUID |
| `message` | yes | Max 2000 chars |
| `metadata` | no | Logged only; not sent into the chat |

### 5.2 Response

```json
{
  "sent": true,
  "agent": "opm",
  "room": "main",
  "request_id": "b2c3d4e5"
}
```

`message_id` is best-effort: `muse-chat-api.py send` returns no stable ID
(the CDP evaluate often returns `None` even on success — see AGENTS.md).
The server does a read-back tail-match like `dm.py` where feasible and
reports `"verified": true/false`; when the tail-match is inconclusive it
reports `"verified": "unknown"` rather than lying.

### 5.3 Notes

- Requires the agent's chromebox to be healthy (CDP reachable). If not,
  `502 CHAT_FAILED` with `"hint": "chromebox unreachable"`.
- Rate-limited per caller identity (§8), not per target agent — prevents
  one caller spamming all four inboxes.

## 6. `POST /api/box/message` — unified message endpoint

One endpoint that routes on `target`, so jobs and agents don't need to
know the transport. This is the recommended endpoint for job-dispatch
output.

### 6.1 Request

```json
{
  "target": "dm:muse",
  "message": "Board watch found 3 new posts.",
  "metadata": {},
  "idempotency_key": "job-board-watch-20261004-030000"
}
```

**Target grammar:**

| Prefix | Routes to | Example |
|---|---|---|
| `dm:<agent>` | `POST /api/box/dms` | `dm:muse` (target defaults to `main`; `dm:muse#<sidechat>` for a sidechat) |
| `chat:main` | `POST /api/box/chat/send` | `chat:main` (agent defaults to caller node) |
| `chat:<room>` | `POST /api/box/chat/send` | `chat:heartbeat-opm` |
| `chat:<agent>:<room>` | `POST /api/box/chat/send` | `chat:pip:main` |
| `board:<#channel>` | `POST /api/box/board/post` | `board:#lobby` |

`message` and `text` are accepted as aliases. `type`, `from`, `signed`
pass through to the DM endpoint when routing to `dm:`.

### 6.2 Response

Normalized across transports:

```json
{
  "sent": true,
  "target": "dm:muse",
  "kind": "dm",
  "id": "a1b2c3d4",
  "request_id": "a1b2c3d4",
  "duplicate": false
}
```

`kind` is one of `dm`, `chat`, `board`. Transport-specific fields
(`verified`, `seq`, `attempts`) are included when available.

### 6.3 Fan-out

`target` may be an array for broadcast (e.g. an alert to board + chat):

```json
{
  "target": ["board:#operators", "chat:main"],
  "message": "Deploy starting in 5 minutes."
}
```

Each leg is sent independently with the same idempotency key scope
(`key + target`); the response contains a `results` array. A leg failure
does not cancel other legs; the top-level `sent` is `true` only if all
legs succeeded.

## 7. `POST /api/box/board/post` — post to board

Proxies to the board's signed POST API. The box server holds dedicated
board credentials (a `box` signer identity registered in the board's
`allowed_signers`, operator level).

### 7.1 Request

```json
{
  "channel": "#lobby",
  "message": "Job board-watch completed: 3 new posts found.",
  "metadata": {
    "job_id": "board-watch-20261004-030000-a1b2c3d4",
    "type": "job_result"
  }
}
```

| Field | Required | Notes |
|---|---|---|
| `channel` | yes | Must match `^#[a-z0-9-]+$` (board channels: `#lobby`, `#jobs`, `#operators`, …) |
| `message` | yes | **Max 500 chars** — the board's `MAX_MSG` limit. Longer → `400 MESSAGE_TOO_LONG` (the API does not silently truncate; callers must chunk) |
| `metadata` | no | Not posted to the board; kept in the box audit log |

### 7.2 Response

```json
{
  "posted": true,
  "channel": "#lobby",
  "seq": 1234,
  "request_id": "c3d4e5f6"
}
```

`seq` is the board's sequence number for the post.

### 7.3 Notes

- The box server signs the board POST with its own key; the board shows
  the post as authored by the box identity with the caller's identity in
  the signed payload's metadata (audit trail preserved end to end).
- Board signing requires `rm -f` semantics on any temp signature files —
  no signature reuse across requests (cf. AGENTS.md lesson).

## 8. Supporting read endpoints

### `GET /api/box/dms?agent=muse&limit=20&type=job`

Reads bl's `dm-log.jsonl` (via SSH), newest first. Filters: `agent`
(sender or recipient), `type`, `since` (ISO timestamp).

```json
{
  "dms": [
    {
      "id": "a1b2c3d4",
      "text": "[from:opm] [id:a1b2c3d4] Board watch found…",
      "from": "opm",
      "to": "muse",
      "target": "main",
      "type": "chat",
      "timestamp": "2026-10-04T03:55:00Z",
      "verified": true,
      "metadata": {}
    }
  ]
}
```

This is the "DMs tab" data source for box UI visibility (DM-SPEC).

### `GET /api/box/requests/{request_id}`

Confirmation tracking (per BOX-API-SPEC confirmation system):

```json
{
  "request_id": "a1b2c3d4",
  "status": "verified",
  "kind": "dm",
  "created_at": "2026-10-04T03:55:00Z",
  "updated_at": "2026-10-04T03:55:14Z",
  "attempts": 1,
  "dm_id": "a1b2c3d4",
  "ack_dm_id": null
}
```

`status`: `pending` → `verified` | `failed`. For `type: job` DMs,
`ack_dm_id` is filled when a `[RESULT <job_id>]` reply is observed,
closing the loop to the JOB system's result collector.

### `GET /api/box/chat/rooms?agent=muse`

```json
{
  "agent": "muse",
  "rooms": [
    {"name": "main", "type": "main"},
    {"name": "heartbeat-opm", "type": "sidechat", "thread": "e65d0cc4-…"}
  ]
}
```

Backed by `muse-chat-api.py sidechat list` on the agent's chromebox.

### `GET /api/box/board/posts?channel=#lobby&limit=20`

Proxies the board's read API. Same shape as the board's native response,
wrapped with `{"channel": …, "posts": […]}`.

## 9. Idempotency

DMs are UUID-tagged and `dm.py` already reuses one `msg_id` across its
3 internal retries with read-back verification — transport retries don't
duplicate. The API layer adds client-driven idempotency on top:

1. **Key source:** `Idempotency-Key` header, or `idempotency_key` body
   field (header wins when both present).
2. **dm_id derivation:** when a key is supplied, `dm_id` is deterministic:
   first 8 hex of `SHA-256(key)`. A retry after the server forgot the key
   still produces the same DM tag, so the recipient's read-back
   (`[id:…]`) dedupes naturally. Without a key, `dm_id = uuid4()[:8]`
   (current dm.py behavior).
3. **Server dedup store:** `/srv/box/idempotency.json` (0600), mapping
   `key -> {status_code, response_body, created_at}`. On a repeat key
   within TTL (24h), the server returns the stored response unchanged
   with an added `X-Idempotent-Replay: true` header and does **not**
   re-send. Expired keys are pruned by a periodic sweep.
4. **Scope:** the key is scoped to the endpoint + caller identity
   (`key + path + identity`), so two operators using the same key don't
   collide.
5. **Fan-out:** for `/api/box/message` with array targets, the effective
   key is `key + target` per leg.

## 10. Rate limiting

Consistent with `bin/rate_limiter.py` (token bucket per identity):

| Bucket | Sustained | Burst | Max/min | Applies to |
|---|---|---|---|---|
| `dm` | 1 op / 3s | 5 | 20 | `POST /api/box/dms`, `dm:` legs of `/message` |
| `chat` | 1 op / 3s | 5 | 20 | `POST /api/box/chat/send`, `chat:` legs |
| `board` | 1 op / 6s | 3 | 10 | `POST /api/box/board/post`, `board:` legs (matches BOX-API-SPEC's 10/min) |

- The limit key is the **authenticated identity** (operator name), not IP.
- Reads (`GET`) are not rate-limited beyond a generous 120/min guard.
- The box server is long-lived, so buckets live in memory; they are
  checkpointed to `/srv/box/rate-limit.json` (atomic write, same pattern
  as `rate_limiter.py`) for restart continuity.
- On exceed: `429` + `Retry-After: <seconds>` header:
  ```json
  {"error": "Rate limit exceeded for dm (20/min)", "code": "RATE_LIMITED", "retry_after": 14}
  ```
- This is a **per-identity** API limit. The underlying `dm.py` /
  chromebox path has its own per-agent browser pacing; the two compose
  (API limit is the outer bound).

## 11. Audit and logging

Every mutating call appends to `/srv/box/audit.jsonl` (0600):

```json
{
  "ts": "2026-10-04T03:55:00Z",
  "identity": "operator-main",
  "endpoint": "POST /api/box/dms",
  "request_id": "a1b2c3d4",
  "idempotency_key": "job-board-watch-20261004-030000",
  "to": "muse",
  "target": "main",
  "type": "chat",
  "dm_id": "a1b2c3d4",
  "result": "verified",
  "latency_ms": 14200
}
```

No message text in the audit log (text lives in `dm-log.jsonl` on bl,
which is already the DM content store). No tokens, keys, or signatures
in any log — values are never logged, only keyed references.

## 12. Security notes

- **No arbitrary execution.** The SSH backend allowlist is fixed:
  `dm.py send|read`, `muse-chat-api.py` subcommands, `systemctl --user`
  (timers, separate spec). User input is validated against enums/regexes
  and passed as argv.
- **Credential hygiene.** Bearer tokens are hashed at rest. Signing keys
  stay with their holders; the box server only holds its own board key
  and the token hashes. Nothing secret appears in logs, errors, or the UI.
- **Signed DMs** (`signed: true`) are verified server-side against
  `dm-signers/*.pub` *before* sending; a bad signature is `400`, not a
  send.
- **Attribution honesty.** Unsigned DMs carry `[from:X]` as plain text —
  the API documents that this is a convention, not authentication, and
  restricts who may set it (§2.4). Cryptographic attribution requires
  `signed: true`.
- **Board posts** are capped at 500 chars by the board; the API rejects
  over-long messages instead of truncating, so callers chunk deliberately.

## 13. UI/UX (box web UI)

Per "UI for Creativity, API for Steering":

- New **DMs tab** on box.muse-dev.online backed by `GET /api/box/dms`
  (the DM-SPEC "box visibility" requirement): filter by agent/type,
  shows verified state and ack linkage.
- **Compose form** for DMs/chat/board posts. Every submit shows its
  exact curl equivalent (copy button), teaching the API.
- **Request tracker**: `GET /api/box/requests/{id}` rendered as a
  pending → verified timeline, including dm.py attempt counts.
- **No UI-only features**: everything the UI does is one of the
  endpoints above.

## 14. Open questions (for implementation phase)

1. Should `POST /api/box/dms` default to sync (45s timeout) or async
   (202 + poll)? This design offers both; implementation should pick one
   default and measure p99 send latency first.
2. Token issuance UX: CLI-only for SUPER, or a box UI flow with the
   operator PIN? (Security-sensitive — human's call.)
3. `dm.py` currently has no `--id` flag to accept a caller-supplied
   `dm_id`; implementation needs it (or an env var) for deterministic
   idempotency keys.
4. Board `box` signer identity: register as a new `allowed_signers`
   entry, or reuse an existing operator key? (New entry is cleaner;
   needs the human's key ceremony.)
5. The pending-request sweeper (BOX-API-SPEC) and the JOB result
   collector overlap on `[RESULT job_id]` correlation — unify them
   during implementation.