Files
box/docs/BOX-API-DESIGN-DMS.md
T

589 lines
20 KiB
Markdown
Raw Normal View History

# BOX-API-DESIGN-DMS.md: DM and Message Endpoint Design
> **Box is the main surface.** All operator work goes through Box (box.muse-dev.online). The web UI, `box` CLI, and agents share the same API endpoints. No UI-only powers.
**Status:** Design v0.1 (design only — no implementation).
**Owner:** operator-main. Sanctioned by the human 2026-10-04.
**Companion:** `BOX-API-SPEC.md` (timer/job API), `DM-SPEC.md` (DM trust model),
front-door `DM_SPEC.md` (work orders, acks, box visibility).
## 1. Overview
This document designs the request/response schemas for the box message
endpoints proposed in `BOX-API-SPEC.md` Appendix B/C:
| Endpoint | Purpose |
|---|---|
| `POST /api/box/dms` | Send a DM via the `dm.py` DM system |
| `POST /api/box/chat/send` | Send to a Muse chat (main or sidechat) via chromebox |
| `POST /api/box/message` | Unified endpoint routing to DM / chat / board |
| `POST /api/box/board/post` | Post to a board channel via the board's signed API |
Plus supporting reads: `GET /api/box/dms`, `GET /api/box/requests/{id}`,
`GET /api/box/chat/rooms`, `GET /api/box/board/posts`.
**Design principles (from the specs):**
- "UI for Creativity, API for Steering" — the box UI calls these same
endpoints; every UI action shows its curl equivalent.
- "Requests via API should always result in DM [working]" — side-effecting
calls return a trackable `request_id`, not just "sent".
- "We get smarter by limiting agency" — the API maps to a fixed allowlist
of backend commands. No arbitrary command execution, ever.
## 2. Authentication
Same ops auth as the box page, extended for programmatic use. All four
endpoints require **operator level**. Dev-level or anonymous callers get 403.
### 2.1 Bearer tokens (primary, for agents)
```
Authorization: Bearer <token>
```
- Token format: `box_` + 32 hex chars (e.g. `box_9f2c…`). Opaque to the
client; the server maps it to an identity.
- Server store: `/srv/box/tokens.json` (0600, root/box only). Tokens are
stored as salted hashes (SHA-256 + per-token salt); the file maps
`hash -> {identity, level, scopes, created_at, revoked}`.
- Identity is an operator name: `operator-main`, `operator-646`, etc.
Each operator maps to a default agent node (`operator-main` → `opm`).
- Tokens are issued and revoked by SUPER (the human) via a box CLI.
Agents never mint their own tokens.
- Scopes: `dms:write`, `chat:write`, `board:write`, `dms:read`,
`spoof` (send `from` another agent — granted sparingly, see §2.4).
### 2.2 PIN session cookie (for humans via UI)
Unchanged from the existing box page: `POST /api/box/login` with the
operator PIN sets a signed HttpOnly session cookie. The UI uses this;
agents use bearer tokens.
### 2.3 Box-signed requests (for automated agents)
Agents holding a registered signing key may sign the request body with
`ssh-keygen -Y sign -n box` (same mechanism as `dm-sign.sh`). The server
verifies against registered pubkeys before processing. This is the
no-shared-secret option for agents that already hold DM signing keys.
### 2.4 The `from` field and spoofing
DMs carry `[from:<agent>]` attribution. The API sets `from` as follows:
- Default: the caller's mapped agent node
(`operator-main` → `opm`, `operator-646` → `646`, …).
- An explicit `"from"` in the body is honored **only** if the token has
the `spoof` scope; otherwise the server overrides it with the caller
identity and logs the attempt.
- Rationale: `[from:X]` is unauthenticated text (see DM-SPEC). The API
must not let arbitrary callers forge attribution. Spoofing is an
explicit, auditable grant.
## 3. Common conventions
### 3.1 Error format
All errors return JSON (per BOX-API-SPEC):
```json
{
"error": "Unknown recipient agent",
"code": "BAD_RECIPIENT"
}
```
| Code | HTTP | Meaning |
|---|---|---|
| `BAD_REQUEST` | 400 | Malformed JSON, missing required field |
| `BAD_RECIPIENT` | 400 | `to` not in `muse\|pip\|646\|opm` |
| `BAD_TARGET` | 400 | `target`/`room` fails validation |
| `MESSAGE_TOO_LONG` | 400 | Over the length cap |
| `FORBIDDEN` | 403 | Auth missing, wrong level, or missing scope |
| `NOT_FOUND` | 404 | `request_id` / room / sidechat not found |
| `CONFLICT` | 409 | Duplicate without idempotency key where one is required |
| `RATE_LIMITED` | 429 | Over the per-identity limit; includes `retry_after` + `Retry-After` header |
| `DM_FAILED` | 502 | `dm.py` could not verify delivery after retries |
| `CHAT_FAILED` | 502 | Chromebox/CDP send failed |
| `BOARD_FAILED` | 502 | Board signed POST failed |
| `SSH_ERROR` | 502 | bl unreachable from the board server |
### 3.2 Timestamps
ISO-8601 UTC, e.g. `2026-10-04T03:55:00Z`.
### 3.3 Backend path
The board server (VM, port 8090) proxies to bl via SSH
(`super@100.123.153.75` over tailnet, restricted key). Every endpoint below
maps to fixed allowlisted commands — never shell-expanded user input:
| Endpoint | bl command |
|---|---|
| `POST /api/box/dms` | `bin/dm.py send --agent {from} --to {to} --target {target} [--raw] {message}` |
| `POST /api/box/chat/send` | `bin/netvm-exec.sh {agent} -- python3 bin/muse-chat-api.py --account {agent} send {message}` (after `sidechat use`/`main` navigation) |
| `POST /api/box/board/post` | signed `POST /api/post` to the board server |
| reads | `dm-log.jsonl`, `job-log.jsonl`, `muse-chat-api.py messages/list` |
`{from}`, `{to}`, `{target}`, `{agent}` are validated against fixed enums
before interpolation. Messages are passed as argv, never through a shell.
## 4. `POST /api/box/dms` — send a DM
Sends via the `dm.py` DM system: tagged `[from:<agent>] [id:<dm_id>]`,
delivered into the recipient's muse.ai chat (main or sidechat), with
recipient-side read-back verification (3 attempts, same `dm_id`).
### 4.1 Request
```http
POST /api/box/dms
Authorization: Bearer <token>
Content-Type: application/json
Idempotency-Key: <optional, see §7>
```
```json
{
"to": "muse",
"target": "main",
"type": "chat",
"text": "Board watch found 3 new posts.",
"from": "opm",
"signed": false,
"metadata": {
"job_id": "board-watch-20261004-030000-a1b2c3d4",
"priority": "normal"
},
"idempotency_key": "job-board-watch-20261004-030000"
}
```
| Field | Required | Notes |
|---|---|---|
| `to` | yes | One of `muse`, `pip`, `646`, `opm` |
| `target` | yes | `"main"` or a sidechat name/UUID (`^[A-Za-z0-9 _-]{1,80}$`) |
| `type` | no | `chat` (default), `job`, `result`, `alert`, `workorder`, `ack`, `canary` |
| `text` | yes | Max 1000 chars — matches `dm.py`'s non-raw truncation cap |
| `from` | no | Defaults to caller identity; needs `spoof` scope to differ |
| `signed` | no | `false` (default). If `true`, `text` must already contain `[from:X] [id:Y]` plus a valid `ssh-keygen -Y` signature block; sent via `dm.py --raw` and signature-verified server-side against `dm-signers/*.pub` before sending |
| `metadata` | no | Free-form object; logged to `dm-log.jsonl` alongside the send |
| `idempotency_key` | no | Client-supplied; see §7. May also be sent as `Idempotency-Key` header (header wins) |
**Wire format construction:** for `type: job`, the server prepends
`[JOB {job_id}]` when `metadata.job_id` is present, matching the JOB-SPEC
convention. For `type: workorder`, it prepends `[WO:<dm_id>]` per DM-SPEC.
`type: alert` with `metadata.severity: urgent` prefixes `[URGENT]`.
### 4.2 Response — synchronous (default)
`dm.py` verifies delivery (typically 10–20s). The default is synchronous
with a 45s server timeout:
```http
200 OK
```
```json
{
"sent": true,
"verified": true,
"dm_id": "a1b2c3d4",
"request_id": "a1b2c3d4",
"to": "muse",
"target": "main",
"from": "opm",
"type": "chat",
"attempts": 1,
"duplicate": false,
"confirmation_url": "/api/box/requests/a1b2c3d4"
}
```
- `dm_id` == `request_id` for DMs (one ID tracks the whole lifecycle).
- `verified: true` means dm.py's recipient-side read-back found
`[id:<dm_id>]` in the recipient's chat. This is delivery confirmation,
not a reply.
- If verification fails after 3 attempts: `502 DM_FAILED` with
`"verified": false, "attempts": 3`.
### 4.3 Response — async (`?async=true`)
For callers that don't want to hold a connection:
```http
202 Accepted
```
```json
{
"request_id": "a1b2c3d4",
"dm_id": "a1b2c3d4",
"status": "pending",
"confirmation_url": "/api/box/requests/a1b2c3d4"
}
```
Poll `GET /api/box/requests/{request_id}` (§6). Final states: `verified`,
`failed`.
### 4.4 Notes
- The send runs as the box backend on bl (identity `opm` for dispatch
purposes), but `[from:]` attribution follows §2.4.
- Every send is appended to `/srv/box/audit.jsonl` (identity, timestamp,
`dm_id`, `to`, `target`, `type`, idempotency key, result) and to bl's
`dm-log.jsonl` via dm.py itself.
- `signed: true` messages skip tagging/truncation (`--raw`); the 1000-char
cap does not apply, but a 4000-char hard cap does (signature blocks are
large; chat input is not infinite).
## 5. `POST /api/box/chat/send` — send to a chat
Posts a message into a Muse chat surface (main chat or sidechat) using
the owning agent's chromebox. Unlike DMs, there is no `[from:]` tag — the
message appears as the account owner.
### 5.1 Request
```json
{
"agent": "opm",
"room": "main",
"message": "Weekly report is ready in side chat 'weekly-20261004'.",
"metadata": {
"type": "announcement",
"job_id": "weekly-report-20261004-030000-x1y2z3"
}
}
```
| Field | Required | Notes |
|---|---|---|
| `agent` | no | Whose chromebox to use; defaults to caller identity's node. Must be in `muse\|pip\|646\|opm`. Operator tokens may target any node; the audit log records it |
| `room` | yes | `"main"` or sidechat name/UUID |
| `message` | yes | Max 2000 chars |
| `metadata` | no | Logged only; not sent into the chat |
### 5.2 Response
```json
{
"sent": true,
"agent": "opm",
"room": "main",
"request_id": "b2c3d4e5"
}
```
`message_id` is best-effort: `muse-chat-api.py send` returns no stable ID
(the CDP evaluate often returns `None` even on success — see AGENTS.md).
The server does a read-back tail-match like `dm.py` where feasible and
reports `"verified": true/false`; when the tail-match is inconclusive it
reports `"verified": "unknown"` rather than lying.
### 5.3 Notes
- Requires the agent's chromebox to be healthy (CDP reachable). If not,
`502 CHAT_FAILED` with `"hint": "chromebox unreachable"`.
- Rate-limited per caller identity (§8), not per target agent — prevents
one caller spamming all four inboxes.
## 6. `POST /api/box/message` — unified message endpoint
One endpoint that routes on `target`, so jobs and agents don't need to
know the transport. This is the recommended endpoint for job-dispatch
output.
### 6.1 Request
```json
{
"target": "dm:muse",
"message": "Board watch found 3 new posts.",
"metadata": {},
"idempotency_key": "job-board-watch-20261004-030000"
}
```
**Target grammar:**
| Prefix | Routes to | Example |
|---|---|---|
| `dm:<agent>` | `POST /api/box/dms` | `dm:muse` (target defaults to `main`; `dm:muse#<sidechat>` for a sidechat) |
| `chat:main` | `POST /api/box/chat/send` | `chat:main` (agent defaults to caller node) |
| `chat:<room>` | `POST /api/box/chat/send` | `chat:heartbeat-opm` |
| `chat:<agent>:<room>` | `POST /api/box/chat/send` | `chat:pip:main` |
| `board:<#channel>` | `POST /api/box/board/post` | `board:#lobby` |
`message` and `text` are accepted as aliases. `type`, `from`, `signed`
pass through to the DM endpoint when routing to `dm:`.
### 6.2 Response
Normalized across transports:
```json
{
"sent": true,
"target": "dm:muse",
"kind": "dm",
"id": "a1b2c3d4",
"request_id": "a1b2c3d4",
"duplicate": false
}
```
`kind` is one of `dm`, `chat`, `board`. Transport-specific fields
(`verified`, `seq`, `attempts`) are included when available.
### 6.3 Fan-out
`target` may be an array for broadcast (e.g. an alert to board + chat):
```json
{
"target": ["board:#operators", "chat:main"],
"message": "Deploy starting in 5 minutes."
}
```
Each leg is sent independently with the same idempotency key scope
(`key + target`); the response contains a `results` array. A leg failure
does not cancel other legs; the top-level `sent` is `true` only if all
legs succeeded.
## 7. `POST /api/box/board/post` — post to board
Proxies to the board's signed POST API. The box server holds dedicated
board credentials (a `box` signer identity registered in the board's
`allowed_signers`, operator level).
### 7.1 Request
```json
{
"channel": "#lobby",
"message": "Job board-watch completed: 3 new posts found.",
"metadata": {
"job_id": "board-watch-20261004-030000-a1b2c3d4",
"type": "job_result"
}
}
```
| Field | Required | Notes |
|---|---|---|
| `channel` | yes | Must match `^#[a-z0-9-]+$` (board channels: `#lobby`, `#jobs`, `#operators`, …) |
| `message` | yes | **Max 500 chars** — the board's `MAX_MSG` limit. Longer → `400 MESSAGE_TOO_LONG` (the API does not silently truncate; callers must chunk) |
| `metadata` | no | Not posted to the board; kept in the box audit log |
### 7.2 Response
```json
{
"posted": true,
"channel": "#lobby",
"seq": 1234,
"request_id": "c3d4e5f6"
}
```
`seq` is the board's sequence number for the post.
### 7.3 Notes
- The box server signs the board POST with its own key; the board shows
the post as authored by the box identity with the caller's identity in
the signed payload's metadata (audit trail preserved end to end).
- Board signing requires `rm -f` semantics on any temp signature files —
no signature reuse across requests (cf. AGENTS.md lesson).
## 8. Supporting read endpoints
### `GET /api/box/dms?agent=muse&limit=20&type=job`
Reads bl's `dm-log.jsonl` (via SSH), newest first. Filters: `agent`
(sender or recipient), `type`, `since` (ISO timestamp).
```json
{
"dms": [
{
"id": "a1b2c3d4",
"text": "[from:opm] [id:a1b2c3d4] Board watch found…",
"from": "opm",
"to": "muse",
"target": "main",
"type": "chat",
"timestamp": "2026-10-04T03:55:00Z",
"verified": true,
"metadata": {}
}
]
}
```
This is the "DMs tab" data source for box UI visibility (DM-SPEC).
### `GET /api/box/requests/{request_id}`
Confirmation tracking (per BOX-API-SPEC confirmation system):
```json
{
"request_id": "a1b2c3d4",
"status": "verified",
"kind": "dm",
"created_at": "2026-10-04T03:55:00Z",
"updated_at": "2026-10-04T03:55:14Z",
"attempts": 1,
"dm_id": "a1b2c3d4",
"ack_dm_id": null
}
```
`status`: `pending` → `verified` | `failed`. For `type: job` DMs,
`ack_dm_id` is filled when a `[RESULT <job_id>]` reply is observed,
closing the loop to the JOB system's result collector.
### `GET /api/box/chat/rooms?agent=muse`
```json
{
"agent": "muse",
"rooms": [
{"name": "main", "type": "main"},
{"name": "heartbeat-opm", "type": "sidechat", "thread": "e65d0cc4-…"}
]
}
```
Backed by `muse-chat-api.py sidechat list` on the agent's chromebox.
### `GET /api/box/board/posts?channel=#lobby&limit=20`
Proxies the board's read API. Same shape as the board's native response,
wrapped with `{"channel": …, "posts": […]}`.
## 9. Idempotency
DMs are UUID-tagged and `dm.py` already reuses one `msg_id` across its
3 internal retries with read-back verification — transport retries don't
duplicate. The API layer adds client-driven idempotency on top:
1. **Key source:** `Idempotency-Key` header, or `idempotency_key` body
field (header wins when both present).
2. **dm_id derivation:** when a key is supplied, `dm_id` is deterministic:
first 8 hex of `SHA-256(key)`. A retry after the server forgot the key
still produces the same DM tag, so the recipient's read-back
(`[id:…]`) dedupes naturally. Without a key, `dm_id = uuid4()[:8]`
(current dm.py behavior).
3. **Server dedup store:** `/srv/box/idempotency.json` (0600), mapping
`key -> {status_code, response_body, created_at}`. On a repeat key
within TTL (24h), the server returns the stored response unchanged
with an added `X-Idempotent-Replay: true` header and does **not**
re-send. Expired keys are pruned by a periodic sweep.
4. **Scope:** the key is scoped to the endpoint + caller identity
(`key + path + identity`), so two operators using the same key don't
collide.
5. **Fan-out:** for `/api/box/message` with array targets, the effective
key is `key + target` per leg.
## 10. Rate limiting
Consistent with `bin/rate_limiter.py` (token bucket per identity):
| Bucket | Sustained | Burst | Max/min | Applies to |
|---|---|---|---|---|
| `dm` | 1 op / 3s | 5 | 20 | `POST /api/box/dms`, `dm:` legs of `/message` |
| `chat` | 1 op / 3s | 5 | 20 | `POST /api/box/chat/send`, `chat:` legs |
| `board` | 1 op / 6s | 3 | 10 | `POST /api/box/board/post`, `board:` legs (matches BOX-API-SPEC's 10/min) |
- The limit key is the **authenticated identity** (operator name), not IP.
- Reads (`GET`) are not rate-limited beyond a generous 120/min guard.
- The box server is long-lived, so buckets live in memory; they are
checkpointed to `/srv/box/rate-limit.json` (atomic write, same pattern
as `rate_limiter.py`) for restart continuity.
- On exceed: `429` + `Retry-After: <seconds>` header:
```json
{"error": "Rate limit exceeded for dm (20/min)", "code": "RATE_LIMITED", "retry_after": 14}
```
- This is a **per-identity** API limit. The underlying `dm.py` /
chromebox path has its own per-agent browser pacing; the two compose
(API limit is the outer bound).
## 11. Audit and logging
Every mutating call appends to `/srv/box/audit.jsonl` (0600):
```json
{
"ts": "2026-10-04T03:55:00Z",
"identity": "operator-main",
"endpoint": "POST /api/box/dms",
"request_id": "a1b2c3d4",
"idempotency_key": "job-board-watch-20261004-030000",
"to": "muse",
"target": "main",
"type": "chat",
"dm_id": "a1b2c3d4",
"result": "verified",
"latency_ms": 14200
}
```
No message text in the audit log (text lives in `dm-log.jsonl` on bl,
which is already the DM content store). No tokens, keys, or signatures
in any log — values are never logged, only keyed references.
## 12. Security notes
- **No arbitrary execution.** The SSH backend allowlist is fixed:
`dm.py send|read`, `muse-chat-api.py` subcommands, `systemctl --user`
(timers, separate spec). User input is validated against enums/regexes
and passed as argv.
- **Credential hygiene.** Bearer tokens are hashed at rest. Signing keys
stay with their holders; the box server only holds its own board key
and the token hashes. Nothing secret appears in logs, errors, or the UI.
- **Signed DMs** (`signed: true`) are verified server-side against
`dm-signers/*.pub` *before* sending; a bad signature is `400`, not a
send.
- **Attribution honesty.** Unsigned DMs carry `[from:X]` as plain text —
the API documents that this is a convention, not authentication, and
restricts who may set it (§2.4). Cryptographic attribution requires
`signed: true`.
- **Board posts** are capped at 500 chars by the board; the API rejects
over-long messages instead of truncating, so callers chunk deliberately.
## 13. UI/UX (box web UI)
Per "UI for Creativity, API for Steering":
- New **DMs tab** on box.muse-dev.online backed by `GET /api/box/dms`
(the DM-SPEC "box visibility" requirement): filter by agent/type,
shows verified state and ack linkage.
- **Compose form** for DMs/chat/board posts. Every submit shows its
exact curl equivalent (copy button), teaching the API.
- **Request tracker**: `GET /api/box/requests/{id}` rendered as a
pending → verified timeline, including dm.py attempt counts.
- **No UI-only features**: everything the UI does is one of the
endpoints above.
## 14. Open questions (for implementation phase)
1. Should `POST /api/box/dms` default to sync (45s timeout) or async
(202 + poll)? This design offers both; implementation should pick one
default and measure p99 send latency first.
2. Token issuance UX: CLI-only for SUPER, or a box UI flow with the
operator PIN? (Security-sensitive — human's call.)
3. `dm.py` currently has no `--id` flag to accept a caller-supplied
`dm_id`; implementation needs it (or an env var) for deterministic
idempotency keys.
4. Board `box` signer identity: register as a new `allowed_signers`
entry, or reuse an existing operator key? (New entry is cleaner;
needs the human's key ceremony.)
5. The pending-request sweeper (BOX-API-SPEC) and the JOB result
collector overlap on `[RESULT job_id]` correlation — unify them
during implementation.