Compare commits
20 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
 operator
|
f9f12b450e | ||
|
operator
|
e6e5616de6 | ||
|
operator
|
f1252423a8 | ||
|
operator
|
340cc9e85b | ||
|
operator
|
0668d257ca | ||
|
operator
|
cc6a33cbca | ||
|
operator
|
5f7e1e3a76 | ||
|
operator
|
073ef556ba | ||
|
operator
|
1b44d224a3 | ||
|
operator
|
bccdd29d53 | ||
|
operator
|
3620b42297 | ||
|
operator
|
9972c07c61 | ||
|
operator
|
f8b7424315 | ||
|
operator
|
c79382b1c1 | ||
|
operator
|
b7a73234fc | ||
|
operator
|
35c59bfcb2 | ||
|
operator
|
ef2a4c419a | ||
|
operator
|
ed33d66479 | ||
|
operator
|
8099c9a4aa | ||
|
Antigravity Agent
|
ab7d1215e0 |
-27
@@ -1,27 +0,0 @@
|
||||
# NetVM Transfers & Local State
|
||||
transfers/
|
||||
*.log
|
||||
logs/
|
||||
*.jsonl
|
||||
pipelines.json
|
||||
followups.json
|
||||
siphon-watermarks.json
|
||||
review/
|
||||
|
||||
__pycache__/
|
||||
*.pyc
|
||||
*.bak
|
||||
*.bak-*
|
||||
*.orig
|
||||
keepalive-config.json
|
||||
main-chat-watchdog.state
|
||||
strategy.json
|
||||
variables.json
|
||||
chained-jobs.json
|
||||
.state/
|
||||
*.lock
|
||||
*watermark*
|
||||
ssl/
|
||||
job-scheduler-state.json
|
||||
var/
|
||||
swarms.json
|
||||
+40
-56
@@ -1,67 +1,51 @@
|
||||
# NetVM Account Registry
|
||||
# Login registry — secret-free
|
||||
|
||||
One row per agent. All data in columns — no joins, no translation.
|
||||
The `agent` name is the canonical identifier used everywhere:
|
||||
node name, chrome-box profile, API `--account`, and the agent's display name.
|
||||
Which product login lives in which chrome-box profile, on which NetVM node,
|
||||
with which egress, in what auth state. This is structure only: **no
|
||||
passwords, no tokens, no session cookies, no OTP codes — ever.** Credential
|
||||
pointers at most (e.g. "human", "credential-gateway:<id>").
|
||||
|
||||
## Schema
|
||||
The 1:1 chain: `login -> profile = node = Warp identity = veth/CDP slot =
|
||||
consistent egress`. Network details live in NODES.md; this file maps the
|
||||
human side (whose login, what for, does it work).
|
||||
|
||||
| Column | Description |
|
||||
|--------|-------------|
|
||||
| agent | Canonical name. Used for node, profile, API account. |
|
||||
| node | NetVM node name (== agent). |
|
||||
| profile | Chrome-box profile (== agent). |
|
||||
| login_type | How this session was authenticated: `email_otp`, `phone_otp`, `password`, `instagram` |
|
||||
| meta_label | Label shown in Meta account selector (e.g., "Meta Account", "piparada") |
|
||||
| email | Email used for login (if email_otp). Never store passwords. |
|
||||
| phone_otp | `yes` if phone OTP was used. Never store the phone number. |
|
||||
| instagram_linked | `yes`/`no`/`unknown` — whether Meta account has IG linked |
|
||||
| status | `active`, `pending_auth`, `needs_signup`, `expired`, `disabled` |
|
||||
| egress_ip | Current WARP egress IP for the node |
|
||||
| cdp_port | CDP port for the browser |
|
||||
| display_name | Agent's chosen display name in muse.ai (may differ from `agent`) |
|
||||
| notes | Freeform context |
|
||||
## Auth states
|
||||
|
||||
## Accounts
|
||||
| state | meaning | who moves it |
|
||||
|-------|---------|--------------|
|
||||
| `pending-identity` | profile exists, no Warp identity yet | human runs `netvm-new-identity.sh <profile>` |
|
||||
| `pending-auth` | node up, nobody logged in yet | human logs in (browser or credential gateway) |
|
||||
| `2fa-pending` | login needs a human 2FA/OTP step | human via ethical-captcha handoff; OTP routed by email-alert |
|
||||
| `active` | logged in, session healthy | operator verifies; automation may proceed |
|
||||
| `expired` | session died | back to `pending-auth` (human) |
|
||||
| `retired` | login no longer used | operator tears down node, archives row |
|
||||
|
||||
| agent | node | profile | login_type | meta_label | email | phone_otp | instagram_linked | status | egress_ip | cdp_port | display_name | notes |
|
||||
|-------|------|---------|------------|------------|-------|-----------|------------------|--------|-----------|----------|--------------|-------|
|
||||
| muse | muse | muse | email_otp | ltd.pixels.ltd@gmail.com | ltd.pixels.ltd@gmail.com | no | unknown | active | 104.28.195.181 | 9410 | muse | Main dev agent. Logged in 2026-10-03 via email OTP on bl. |
|
||||
| pip | pip | pip | phone_otp | piparada | io.antonio.parada@gmail.com | yes | yes | active | 104.28.195.181 | 9420 | pip | Phone OTP login. Linked with IG piparada, email io.antonio.parada@gmail.com. Verified active 2026-10-04. |
|
||||
| 646 | 646 | 646 | phone_otp | Meta Account | - | yes | no | active | 104.28.195.181 | 9430 | 646 | Shares phone number with piparada's account. Logged in 2026-10-03 via phone OTP (first Meta Account option). Node created 2026-10-03 (warp-646, CDP 9430). Browser up, session active. |
|
||||
| def | def | def | email_otp | defnotabotnet@gmail.com | defnotabotnet@gmail.com | no | yes | active | 104.28.195.181 | 9450 | def | Full onboarding completed 2026-10-04; age verification cleared via Instagram linking (paradahub). Active chat session. |
|
||||
| opm | opm | opm | email_otp | Nico Parada | artglobal.cc@gmail.com | no | yes | active | 104.28.195.181 | 9440 | opm | Email changed from yourfriendnico@proton.me to artglobal.cc@gmail.com. Linked with IG auxfate. Browser up, session active. |
|
||||
| dev | dev | dev | email_otp | paradaproduced@gmail.com | paradaproduced@gmail.com | no | yes | active | 104.28.195.181 | 9460 | dev | Full onboarding completed 2026-10-04; unlocked /access gate via Meta Accounts Center IG linking (veryraremeta). Active chat session. |
|
||||
Operators never create or touch credentials. If it creates or touches a
|
||||
credential, it is human-only. Everything else, operators handle.
|
||||
|
||||
## Login Type Details
|
||||
## Registry
|
||||
|
||||
### email_otp
|
||||
- Flow: Enter email → Receive OTP via email → Enter OTP → Select account (if multiple)
|
||||
- Used by: muse
|
||||
- Credentials: Email address (stored). OTP is transient.
|
||||
| login | product | profile/node | purpose / owner | auth state | 2FA / verify route | notes |
|
||||
|-------|---------|--------------|-----------------|------------|--------------------|-------|
|
||||
| — | — | tp | orchestrator / operator-main | pending-auth | — | first node; no product login yet |
|
||||
| — | — | smoke | muse-646-patha | active | — | 646's profile; muse.ai login completed 2026-10-03 |
|
||||
|
||||
### phone_otp
|
||||
- Flow: Enter phone → Receive SMS OTP → Enter OTP → Select Meta account (if multiple)
|
||||
- Used by: pip, 646
|
||||
- Credentials: Phone number is NEVER stored (PII). Only `phone_otp=yes` flag.
|
||||
- Note: One phone number can map to multiple Meta accounts (observed: 2 accounts).
|
||||
## Known login flows
|
||||
|
||||
### Meta Account Selection
|
||||
When a phone number maps to multiple Meta accounts, muse.ai shows a selector:
|
||||
- Screenshot: `docs/meta-account-selection.png`
|
||||
- Each option is a SEPARATE Muse container (not linked profiles).
|
||||
- The `meta_label` column records which option was selected.
|
||||
- Instagram-linked accounts show IG avatar in selector.
|
||||
### muse.ai (recon 2026-10-03, via CDP DOM)
|
||||
- Homepage has "Log in" buttons (JS, no href). Click -> inline form, same URL.
|
||||
- "Log in or create an account" — single field: "Mobile number or email (required)" + Continue.
|
||||
- Phone/email OTP flow (SMS or email code). No password, no OAuth buttons.
|
||||
- Human completes it in one visible session; operators verify + automate after.
|
||||
|
||||
## Naming Convention
|
||||
## Provisioning a new login (dev)
|
||||
|
||||
**Rule:** The `agent` column value is used identically for:
|
||||
- NetVM node name (`/etc/netvm/<agent>.conf`)
|
||||
- Chrome-box profile (`~/.local/share/chrome-box/profiles/<agent>/`)
|
||||
- API account (`muse-chat-api.py --account <agent>`)
|
||||
- CDP port mapping (deterministic per agent)
|
||||
|
||||
**Exception:** `display_name` may differ (user-chosen in muse.ai UI).
|
||||
Example: agent `pip` has display_name `pip` (renamed from 'Muse').
|
||||
|
||||
Do NOT use different names for node vs profile vs API. That causes bugs.
|
||||
1. Operator: `chrome-box create <profile>` (profile name = future node name).
|
||||
2. Human: `netvm-new-identity.sh <profile>` (Warp identity — credential).
|
||||
3. Operator: `netvm-node-up.sh <profile>`; add rows to NODES.md and here
|
||||
(`pending-identity` -> `pending-auth`).
|
||||
4. Human: authenticate the login in the profile's browser
|
||||
(`netvm-chrome.sh <profile>` visible, or credential-gateway injection).
|
||||
Row -> `active`.
|
||||
5. Operator: verify with `netvm-exec.sh <profile> -- ...` / CDP; keep the
|
||||
session warm. On 2FA: ethical-captcha handoff, OTP via email-alert.
|
||||
|
||||
@@ -1,76 +0,0 @@
|
||||
# NetVM Agent Chat Policy: Main Chat Preservation
|
||||
|
||||
**Status:** ACTIVE POLICY (Mandatory across all fleet automation, jobs, and operator tooling)
|
||||
**Date:** 2026-10-04
|
||||
**Version:** 2.0
|
||||
**Applies to:** All autonomous agents (`muse`, `pip`, `646`, `opm`), scheduled jobs (`super job`), orchestrator tooling (`super dm`, `dm.py`), and human operators.
|
||||
|
||||
---
|
||||
|
||||
## 1. The Core Principle: Main Chat Is Sacred
|
||||
|
||||
> **Rule:** **Avoid using Main Chat whenever possible.**
|
||||
>
|
||||
> When Main Chat gets bogged down with automated entries, scheduled job triggers, log dumps, or inter-agent chatter, **work stops actually getting done**.
|
||||
> Browser DOM virtualizers lag or crash, context windows saturate with noisy outputs, and the agent's attention drifts away from primary operator directives.
|
||||
|
||||
Main Chat is reserved **exclusively** for high-level human operator oversight, urgent human-visible escalations, and direct operator conversational alignment.
|
||||
|
||||
---
|
||||
|
||||
## 2. Channel Segregation Rules
|
||||
|
||||
### Rule A: Scheduled Jobs MUST Target Dedicated Sidechats
|
||||
* **Never** configure a routine scheduled job (e.g., cron checks, health monitors, heartbeats, periodic scrapes) to deliver to `main`.
|
||||
* Every job definition in `jobs/<name>.json` must explicitly specify:
|
||||
* `"dm_target": "<sidechat-name-or-uuid>"` OR
|
||||
* `"sidechat": { "create": true, "name_template": "...", "reuse_key": "..." }`
|
||||
* Any job found dumping routine health outputs or telemetry into `main` must be immediately migrated to a dedicated sidechat.
|
||||
|
||||
### Rule B: Inter-Agent Communication Runs via Sidechats / Side Agents
|
||||
* Autonomous agents communicating with one another (e.g., `646` ↔ `pip`, `opm` ↔ `646`) must use dedicated coordination sidechats (e.g. `646-pip-coord`, `646-opm-coord`, `646 tasks`).
|
||||
* Do not route peer coordination or sub-task requests through an agent's Main Chat.
|
||||
* Subordinate or delegated tasks should be spun off to side agents or sidechats to isolate the conversation state and prevent main thread contamination.
|
||||
|
||||
### Rule C: Large Payloads Transferred via File System, Not Chat
|
||||
* Do **not** dump multi-kilobyte log extracts, raw HTTP responses, table dumps, or diffs into any chat window.
|
||||
* Payloads must be written to disk on `bl` or the VM (e.g. in `/home/super/Projects/NetVM/logs/` or `/srv/box/`) and referenced via short path / pointer in the message:
|
||||
* ✅ *Good:* `[RESULT 12345] Health check completed. 6/6 endpoints OK. Detailed breakdown saved to logs/http-health-20261004.log`
|
||||
* ❌ *Forbidden:* Pasting 200 lines of raw curl outputs or JSON logs into chat.
|
||||
|
||||
### Rule D: Operator CLI (`super dm`) Enforces Sidechat-First Flow
|
||||
* Interactive conversational sessions (`super dm chat <agent>`) prompt for or default to sidechats and issue an explicit policy warning whenever Main Chat is selected.
|
||||
* When dispatching one-off DMs via `super dm send` or `super dm wo`, operators must prefer `--target "<sidechat>"` over `--target main`.
|
||||
|
||||
---
|
||||
|
||||
## 3. Fleet Addressing Directory
|
||||
|
||||
| Target | Agent | Purpose | Policy Tier |
|
||||
|---|---|---|---|
|
||||
| `main` | All (`muse`, `pip`, `646`, `opm`) | Direct human-to-operator urgent interventions only | **Restricted / Minimal** |
|
||||
| `heartbeat` | `opm` | Automated DM pipeline heartbeat verification | **Sidechat Required** |
|
||||
| `646 tasks` | `646` | Daily check-ins, execution health, operator tasks | **Sidechat Required** |
|
||||
| `646-pip-coord` | `pip` / `646` | Peer coordination between pip and 646 | **Sidechat Required** |
|
||||
| `646-opm-coord` | `opm` / `646` | Peer coordination between opm and 646 | **Sidechat Required** |
|
||||
|
||||
---
|
||||
|
||||
## 4. Violations & Enforcement
|
||||
|
||||
1. **Dispatcher Guard:** Scheduled jobs with `schedule != "manual"` and no `dm_target` or `sidechat` configuration will be audited and retrofitted with dedicated sidechat targets.
|
||||
2. **Review Checklist:** Any PR, skill, rule, or script introducing automated messages must verify that output lands in a sidechat or log file, never in Main Chat.
|
||||
|
||||
---
|
||||
|
||||
## 5. Changelog
|
||||
|
||||
### v2.0 — 2026-10-04: Sidechat-only enforcement
|
||||
* **New rule:** No DM lands in Main Chat unless explicitly authorized. `dm.py send` / `job-dispatch.py` refuse `--target main` without `--allow-main-chat` (exit 2, `main_chat_blocked` log event, before any browser navigation). Job JSON opt-in key: `"allow_main_chat": true`.
|
||||
* **Watcher:** `main-chat-watchdog.py` (every 5 min) classifies dm-log events into blocked/ authorized / violation.
|
||||
* **Triage:** `docs/SIDECHAT-POLICY-TRIAGE.md` `— where to look first on failure.
|
||||
* **Fix:** `SIDCHAT_ALIASES["heartbeat"]` de-collided — was pointing at 646's tasks thread (`1e75a740-...`); now points at the dedicated heartbeat sidechat (`0077e918-...`, reuse_key `heartbeat-opm` in `job-sidechats.json`).
|
||||
|
||||
### v1.0 — 2026-10-04: Initial policy
|
||||
* Main Chat preservation: scheduled jobs and inter-agent communication must use dedicated sidechats.
|
||||
* Fleet addressing directory established.
|
||||
@@ -1,146 +0,0 @@
|
||||
# Client Onboarding & Fleet Runbook (`CLIENT-ONBOARDING-RUNBOOK.md`)
|
||||
|
||||
## 1. Overview & Agency Context
|
||||
This document defines the complete standard operating procedure (SOP) and automated runbook for provisioning, authenticating, and onboarding client agency profiles (nodes) into the NetVM multi-tenant fleet on `bl`.
|
||||
|
||||
In accordance with the NetVM Ethics Charter (`https://start.muse-dev.online/ethics.html`):
|
||||
- Managed services are strictly run for consenting clients with explicit authority.
|
||||
- Every client receives a completely isolated network namespace (`warp-<node>`), dedicated WireGuard tunnel identity, isolated Chrome profile, and separate credentials.
|
||||
- Canonical Naming Convention: `node == agent == profile == API account`.
|
||||
|
||||
---
|
||||
|
||||
## 2. Fleet Architecture & Port Allocation
|
||||
|
||||
The fleet uses a deterministic `94x0` CDP port and `warp-<node>` naming convention:
|
||||
|
||||
| Node | CDP Port | Netns | Egress IP | Purpose / Profile |
|
||||
|------|----------|-------|-----------|-------------------|
|
||||
| `muse` | `9410` | `warp-muse` | Dedicated WARP | Primary Dev / Orchestrator |
|
||||
| `pip` | `9420` | `warp-pip` | Dedicated WARP | Production Agent |
|
||||
| `646` | `9430` | `warp-646` | Dedicated WARP | Production Agent |
|
||||
| `opm` | `9440` | `warp-opm` | Dedicated WARP | Production Agent |
|
||||
| `def` | `9450` | `warp-def` | Dedicated WARP | Production Agent |
|
||||
| `<new>` | `9460+` | `warp-<new>`| Dedicated WARP | Next provisioned client node |
|
||||
|
||||
---
|
||||
|
||||
## 3. Step-by-Step Client Onboarding SOP
|
||||
|
||||
### Phase 1: Infrastructure Provisioning (Automated)
|
||||
Run the idempotent node provisioning script to generate the WireGuard identity, network namespace, CDP relay, and chrome-box profile:
|
||||
|
||||
```bash
|
||||
# Example: Provisioning node 'dev1'
|
||||
./bin/netvm-provision-node.sh dev1
|
||||
```
|
||||
*Verification:*
|
||||
- Namespace created: `ip netns list | grep warp-dev1`
|
||||
- Registry updated in `NODES.md` and `ACCOUNTS.md`.
|
||||
|
||||
---
|
||||
|
||||
### Phase 2: Sign-in Initiation (`super cred initiate`)
|
||||
Launch the client login flow without handling raw passwords or secrets:
|
||||
|
||||
```bash
|
||||
# For email OTP login:
|
||||
super cred initiate --node dev1 --email client@domain.com
|
||||
|
||||
# Or via Python Agent API:
|
||||
python3 bin/cred-client.py initiate --node dev1 --email client@domain.com
|
||||
```
|
||||
|
||||
- If already authenticated, exits `0` (`active`).
|
||||
- If awaiting verification code, exits `2` (`awaiting_otp`).
|
||||
|
||||
---
|
||||
|
||||
### Phase 3: Submitting Transient OTP (`super cred submit-otp`)
|
||||
When the client or operator receives the 6-digit email OTP:
|
||||
|
||||
```bash
|
||||
super cred submit-otp --node dev1 --otp 123456
|
||||
```
|
||||
|
||||
- The code is submitted transiently and is never persisted to disk or logs.
|
||||
- If the account directly enters chat, status transitions to `active`.
|
||||
- If the account requires age verification, it advances to Phase 4.
|
||||
|
||||
---
|
||||
|
||||
### Phase 4: Resolving the Age Verification Gate (`/access/verification`)
|
||||
When a brand-new or unlinked client profile reaches the Muse age verification gate:
|
||||
|
||||
#### Method A: Instagram Linking (Recommended)
|
||||
1. Run:
|
||||
```bash
|
||||
super cred link-instagram --node dev1 [--notify]
|
||||
```
|
||||
2. The system provides a one-tap Tailscale portal URL:
|
||||
`http://bl.tailfb5960.ts.net:8765/verify/dev1`
|
||||
3. **Crucial Rule**: The operator or client must link an **established / aged Instagram profile** (not created within minutes). Brand-new Instagram accounts lack mature age signals, causing Meta Accounts Center to disable the Confirm button.
|
||||
4. If completed via mobile/desktop browser, use an Incognito/Private window to prevent ambient Meta cookie bleed.
|
||||
5. If executing automated RPA in-browser, inject the Instagram credentials and security code directly into the container's CDP session.
|
||||
|
||||
#### Method B: Credit Card Verification (Fallback)
|
||||
If Instagram linking is not available, operator can complete the verification using a client payment card on `/access/verification`.
|
||||
|
||||
---
|
||||
|
||||
### Phase 4.1: Edge Gate — Hard Audience Lockout (`/access` vs `/access/verification`)
|
||||
- **Observed Behavior**: If an account routes to `https://muse.ai/access` with the text `"Muse isn't available to all audiences."` instead of `https://muse.ai/access/verification`:
|
||||
- The Meta account is temporarily unverified or lacks linked identity signals.
|
||||
- The in-app endpoint (`/api/hatch/age-confirmation/linking-web-auth`) returns `403 Forbidden`.
|
||||
- Reloading or navigating directly to `/` or `/access/verification` immediately redirects back to `/access`.
|
||||
- **Root Cause**:
|
||||
- Meta accounts without an active linked profile (Facebook or Instagram) trigger Meta's general audience filter on Muse before the conversational AI product can be initialized.
|
||||
- In addition, attempting automated sign-in on low-reputation / unverified identities directly from server/VPN IPs will trigger Google reCAPTCHA Enterprise checkpoints (`auth_platform/recaptcha`).
|
||||
- **Proven Unblocking SOP (The Direct Meta Accounts Center Flow)**:
|
||||
1. Open a clean browser session with the target Meta Account signed in (`https://accountscenter.meta.com/`).
|
||||
2. Navigate to **Accounts** → **Add Accounts** (`/add_accounts/`).
|
||||
3. Enter the Instagram credentials for an older/established IG profile (`veryraremeta`, `paradahub`, etc.) and submit any required 2FA/email OTP.
|
||||
4. If returned to Accounts Center, click **Add Instagram** again to initiate the OAuth handoff:
|
||||
`https://www.instagram.com/fxcal/auth/login/?app_id=633385687760560...&flow=igcalcomet&entry_point=frl_web_settings`
|
||||
5. On the *"Meta needs to access info from your Instagram account"* prompt, click **[Continue]**.
|
||||
6. Accounts Center returns to the confirmation screen (`/add/?auth_flow=ig_linking&token=...&blob=...`) → click **[Confirm]**.
|
||||
7. Meta sends security confirmation: *"Did you just move your profiles into the same Meta Account?"*.
|
||||
8. Once confirmed in Accounts Center, simply navigate back or reload `https://muse.ai/` inside the NetVM node. The `/access` lockout drops immediately, and the node enters active chat (*"Hey! I'm your personal agent..."*).
|
||||
|
||||
### Phase 4.2: Automated CDP Meta Linking Bot
|
||||
When an edge-gate is detected or when linking a fresh client profile:
|
||||
1. Trigger the automated CDP driver:
|
||||
```bash
|
||||
sudo ip netns exec warp-<node> python3 /home/super/Projects/NetVM/bin/meta-acct.py link-instagram <node>
|
||||
```
|
||||
2. The bot:
|
||||
- Navigates headless Chromium to `https://accountscenter.meta.com/manage/`.
|
||||
- Locates and clicks **Add profiles and devices**.
|
||||
- Selects the Instagram cross-app linking flow.
|
||||
- Automatically navigates to the `frl_web_settings` FXCAL OAuth grant.
|
||||
3. Once completed or after submitting Instagram credentials, re-query the account state:
|
||||
```bash
|
||||
super cred meta-audit --node <node>
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Phase 5: Vitality & Status Monitoring
|
||||
Query individual or fleet-wide health:
|
||||
|
||||
```bash
|
||||
# Check single node
|
||||
super cred status --node dev1
|
||||
|
||||
# Check entire fleet
|
||||
super cred list
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 4. Rate-Limiting & Operational Safety Rules
|
||||
|
||||
To avoid platform anti-automation challenges and maintain high reputation:
|
||||
1. **Pacing / Spacing**: Space new node creations and Instagram authorizations by **at least 15–20 minutes** per IP/session.
|
||||
2. **Namespace Isolation**: Never attempt multi-account auth inside the same browser profile. Always execute inside the client's dedicated `warp-<node>` netns.
|
||||
3. **No Credential Logging**: Never print plain text passwords or authentication tokens to stdout, git-tracked markdown, or plain text logs.
|
||||
@@ -1,92 +0,0 @@
|
||||
# Meta Credential Store (operator-only)
|
||||
Centralized encrypted store for Muse, Instagram, Facebook account credentials.
|
||||
## Location (VM only)
|
||||
- `/etc/netvm/meta-credentials/store.age` — age-encrypted JSON (600 root)
|
||||
- `/etc/netvm/meta-credentials/.age-key` — age private key (600 root)
|
||||
- `/usr/local/bin/meta-creds.sh` — CLI (700 root)
|
||||
## Usage
|
||||
```bash
|
||||
sudo meta-creds.sh list muse # list account IDs (no secrets)
|
||||
sudo meta-creds.sh get muse <id> # output JSON (never log this)
|
||||
sudo meta-creds.sh add muse <id> # interactive prompts
|
||||
```
|
||||
## Naming
|
||||
|
||||
Store ID == `ACCOUNTS.md` `agent` name (e.g. `646`, `pip`, `muse`). The
|
||||
secret store and the secret-free registry join on this ID — same account,
|
||||
different jobs (secrets vs. state).
|
||||
|
||||
## Schema
|
||||
```json
|
||||
{
|
||||
"muse": {
|
||||
"<id>": {
|
||||
"email": "...",
|
||||
"phone": "...",
|
||||
"via_meta_account": "<facebook|instagram id>",
|
||||
"age_verified": "true",
|
||||
"instagram_linked": "<handle>",
|
||||
"verified_by": "human", "verified_at": "2026-10-03T...",
|
||||
"notes": "..."
|
||||
}
|
||||
},
|
||||
"instagram": {
|
||||
"<id>": {
|
||||
"username": "...", "password": "...",
|
||||
"email": "...", "phone": "...",
|
||||
"accounts_center": "<alias>",
|
||||
"linked_to": ["<other store id>", "..."],
|
||||
"login_methods": ["password", "phone_otp"]
|
||||
}
|
||||
},
|
||||
"facebook": {
|
||||
"<id>": {
|
||||
"email": "...", "password": "...", "phone": "...",
|
||||
"accounts_center": "<alias>",
|
||||
"linked_to": ["<other store id>", "..."],
|
||||
"login_methods": ["password", "phone_otp"]
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Field notes
|
||||
|
||||
- `phone`: mobile number for login/2FA. **May be stored here** (encrypted);
|
||||
one phone can map to multiple accounts (observed: 646 + piparada share
|
||||
a number) — never treat it as a unique key.
|
||||
- `via_meta_account` (muse): when muse.ai auth runs through a Meta
|
||||
account (phone OTP → Meta account → muse.ai), points at the
|
||||
`facebook`/`instagram` entry. This is the 646/pip intersection.
|
||||
- `accounts_center`: local alias for the Accounts Center (e.g. `ac-646`).
|
||||
Post early-2026 this is the login blast-radius boundary — every account
|
||||
in one Center logs into every other by default.
|
||||
- `linked_to`: other store IDs in the same Accounts Center. Cached from
|
||||
the Meta API's `list-linked`; **the API is ground truth** — when they
|
||||
disagree, the API wins and the store gets updated.
|
||||
- `login_methods`: how the account can be authenticated. Drives which
|
||||
flow the automation attempts.
|
||||
|
||||
## Intersections
|
||||
|
||||
- **Phone ↔ accounts (1:many):** the store holds the number (encrypted);
|
||||
the human is no longer the sole holder, but the number still never
|
||||
appears in logs, chat, memory, or the secret-free registry.
|
||||
- **Meta credential → muse.ai session:** a `facebook`/`instagram` entry
|
||||
can be the auth path for a `muse` login. Follow `via_meta_account`.
|
||||
- **Store ↔ ACCOUNTS.md:** joined on ID. Store = secrets, registry = state.
|
||||
- **Store ↔ Meta Accounts Center API** (`docs/META-ACCOUNTS-API.md`):
|
||||
the API reads linkage ground truth; the store caches it in `linked_to` /
|
||||
`accounts_center`.
|
||||
|
||||
## Rules
|
||||
|
||||
- Operators only. Developers never get access (prevents board leaks).
|
||||
- Decrypt transiently, never log values, never put in chat/memory.
|
||||
- Phone numbers and PII **may** live in `store.age` (age-encrypted, 600
|
||||
root, VM only). They must never appear in plaintext anywhere else:
|
||||
no logs, no chat, no memory, no registry, no board.
|
||||
- Human validates Instagram linking and Meta account ownership;
|
||||
operators automate after.
|
||||
- When adding an account, fill `accounts_center` / `linked_to` from the
|
||||
Meta API (`list-linked`), not from memory.
|
||||
@@ -1,50 +0,0 @@
|
||||
# NetVM Infrastructure Layout
|
||||
|
||||
## bl (100.123.153.75) — Main Compute
|
||||
- 16 cores, 28GB RAM
|
||||
- Roles: Browser automation, NetVM nodes
|
||||
- Access: VM -> bl via SSH
|
||||
- WARP: Per-node identities in /etc/netvm/
|
||||
- Browsers: Headless Chromium via chrome-box
|
||||
- API: muse-chat-api.py via netvm-exec (CDP)
|
||||
- Sign-in: muse-signin.py --email <addr> [--otp <code>]
|
||||
- Hygiene: netvm-reaper.sh
|
||||
- Approvals: In-browser via chat (see below)
|
||||
|
||||
## VM (34.139.37.135) — Gateway + Vault
|
||||
- E2 micro (2 vCPU, 1GB RAM)
|
||||
- Roles: Jump host to bl, credential store
|
||||
- Credential store: /etc/netvm/meta-credentials/ (age-encrypted)
|
||||
- No browser automation (resource constraints)
|
||||
|
||||
## Laptop — Dev
|
||||
- Roles: Iteration, visible browser debugging
|
||||
- Nothing production
|
||||
|
||||
## Credential Flow
|
||||
- Meta accounts: /etc/netvm/meta-credentials/store.age (VM)
|
||||
- WARP identities: /etc/netvm/node.conf (per-machine, root 600)
|
||||
- Operators handle transiently; never log values.
|
||||
|
||||
## In-Browser Approvals
|
||||
When automation needs human input (OTP, confirmation):
|
||||
1. Script exits with code 2 and prints "APPROVAL_NEEDED: <details>"
|
||||
2. Operator sees this and asks user via chat
|
||||
3. User provides input (e.g., OTP code)
|
||||
4. Operator re-runs script with --otp <code>
|
||||
5. Script completes the flow
|
||||
|
||||
No file-based queue needed — the chat IS the approval interface.
|
||||
The human is already in the chat; the automation just needs to
|
||||
signal when it's stuck.
|
||||
|
||||
## Sign-In Flow (muse-signin.py)
|
||||
Automated login for muse.ai accounts:
|
||||
- Step 1: Check if already logged in (skip if yes)
|
||||
- Step 2: Click "Log in"
|
||||
- Step 3: Enter email
|
||||
- Step 4: Click "Continue"
|
||||
- Step 5: Detect OTP prompt
|
||||
- If --otp provided: enter it, click Next, verify
|
||||
- If not: exit 2 with APPROVAL_NEEDED
|
||||
- Credentials never stored; OTP is transient.
|
||||
@@ -1,126 +0,0 @@
|
||||
# Instagram Credential Pool Specification (`INSTAGRAM-CRED-POOL.md`)
|
||||
|
||||
## 1. Overview & Agency Context
|
||||
When onboarding agency client profiles ("nodes") to Muse (`muse.ai`), new accounts and certain unconfirmed email accounts encounter the post-OTP Age Verification gate (`/access/verification`).
|
||||
|
||||
While credit-card verification is not suitable for autonomous multi-tenant operations, **Meta OAuth / Instagram Linking** is the highest-reliability verification route.
|
||||
|
||||
Currently, the system uses a **Human-in-the-Loop** model:
|
||||
- An operator receives an authorization notification via Tailscale / email.
|
||||
- The operator signs into Instagram via a one-tap link from their mobile device or laptop.
|
||||
|
||||
This specification details the future architecture for **Automated Instagram Credential Pooling** to remove human intervention entirely while strictly complying with the NetVM Ethics Charter (`https://start.muse-dev.online/ethics.html`).
|
||||
|
||||
---
|
||||
|
||||
## 2. Architecture & Design Principles
|
||||
|
||||
### 2.1 Isolation & Multi-Tenancy (Ethics Charter Compliant)
|
||||
- **1-to-1 Node Mapping**: Each client node (`node == agent == profile == API account`) maintains dedicated browser state, WireGuard netns isolation (`warp-<node>`), and separate credentials.
|
||||
- **Dedicated IG Identities**: Instagram accounts in the pool are provisioned specifically for age-verification linking, never shared concurrently across different active client profiles.
|
||||
- **Encrypted Secret Storage**: Instagram credentials (username, password, 2FA TOTP secret, session cookies) are stored in an encrypted credential vault (e.g., `age`-encrypted `/etc/netvm/meta-credentials/store.age`), never committed in plain text to git or unencrypted markdown.
|
||||
|
||||
### 2.2 Pool States & Lifecycle
|
||||
|
||||
```mermaid
|
||||
stateDiagram-v2
|
||||
[*] --> Available : Provisioned & Verified
|
||||
Available --> Assigned : Reserved for Node Onboarding
|
||||
Assigned --> Linking : Navigating Meta OAuth in netns
|
||||
Linking --> Linked : Age Gate Cleared on Muse
|
||||
Linking --> Cooloff : Checkpoint / Rate-limit Hit
|
||||
Cooloff --> Available : Cooldown Elapsed
|
||||
Linked --> InUse : Node Active in Fleet
|
||||
```
|
||||
|
||||
- **`available`**: Account is verified, healthy, and not currently tied to any active Muse profile.
|
||||
- **`assigned`**: Temporarily reserved by `super cred` for onboarding node `<node>`.
|
||||
- **`linking`**: Automated driver navigating the Meta Accounts Center flow inside the isolated namespace.
|
||||
- **`linked`**: Successfully bound to Muse account.
|
||||
- **`cooloff`**: Encountered challenge or cooldown; resting before re-qualification.
|
||||
|
||||
### 2.3 Meta Account Center Constraints & Edge Cases
|
||||
- **1-to-1 Linking Constraint**: Meta Accounts Center rejects linking if the target Instagram account is already associated with an existing Meta / Muse profile (`auth_flow=ig_linking` drops to `add_accounts` error page with `token` and `blob` parameters).
|
||||
- **Brand New Account Age Gate Limitation**:
|
||||
- Newly created Instagram accounts without a mature age/identity verification tier or age signal trigger Meta Accounts Center to disable the **"Confirm"** action (`aria-disabled="true"` on `/add_accounts/?flow=HATCH_AGE_VERIFICATION_IG_UPSELL`).
|
||||
- Meta Accounts Center uses Instagram accounts for age verification by checking that the linked Instagram profile itself has established age signals. A freshly minted account created minutes prior lacks this profile history, leaving the age verification unsatisfied.
|
||||
- **Agency Recommendation**: Pre-aged or verified Instagram identities in the pool with established age badges/profiles, or using established client identities, rather than accounts created in the immediate transaction.
|
||||
- **Dormant / "Ghost" Account Lockout Mode (`/access` Hard Exclusion & Resolution)**:
|
||||
- An account that has chronological calendar age (e.g. created ~5 months ago) but has **zero posts, zero regular engagement, and no established social graph** can fail Meta's automated audience eligibility check completely upon Muse onboarding, routing to `https://muse.ai/access` (*"Muse isn't available to all audiences"*).
|
||||
- Attempting automated sign-in on these low-reputation identities from datacenter/VPN egress IPs triggers Google reCAPTCHA Enterprise checkpoints.
|
||||
- **The "Add Again" Two-Step Handoff Resolution**:
|
||||
1. Sign in to `https://accountscenter.meta.com/` using the cached Meta Account session.
|
||||
2. Add Account -> complete Instagram sign-on & OTP.
|
||||
3. Returning to Meta Accounts Center, click **Add Instagram a second time** (`ADD AGAIN`).
|
||||
4. Meta generates the OAuth handoff URL:
|
||||
`https://www.instagram.com/fxcal/auth/login/?app_id=633385687760560&etoken=...&next=https%3A%2F%2Faccountscenter.meta.com%2Fadd%2F%3Fauth_flow%3Dig_linking%26background_page%3D%252Fmanage&flow=igcalcomet&entry_point=frl_web_settings&initiator_fbid=...`
|
||||
5. Prompt displays: `"[<instagram_handle>] Meta needs to access info from your Instagram account. [Continue] [Not You?]"`.
|
||||
6. Clicking **[Continue]** redirects to Accounts Center with query parameters `token` and `blob` (`/add/?auth_flow=ig_linking&token=...&blob=...`).
|
||||
7. Clicking **[Confirm]** completes the account merge, prompts Meta's confirmation email (*"Did you just move your profiles into the same Meta Account?"*), and **instantly clears the `/access` block on Muse**, transitioning the session into active chat.
|
||||
- **Session Bleed & OIDC Secondary Auth Trip (`auth.meta.com`)**:
|
||||
- When the link is opened in a browser that has existing Meta session cookies (e.g. from Facebook, Oculus, or another Meta account), selecting the new Instagram identity triggers a secondary OpenID Connect reconciliation trip (`https://auth.meta.com/?waterfall_id=...&redirect_uri=auth.meta.com/oidc/...&source_app_id=633385687760560`).
|
||||
- This prompts the user with **"Log in with your Meta account"** because the browser's ambient Meta session does not match the freshly authenticated Instagram identity.
|
||||
- If the user confirms with their cached personal Meta credentials, Meta attempts to merge/link across two disparate account graphs, creating an authorization loop or conflict.
|
||||
- **Resolution**: The link must strictly be opened in an **Incognito / Private window** or a completely clean browser profile with zero cached Meta/Facebook/Instagram cookies.
|
||||
- **In-Namespace Isolation**: Automated pool linking runs in Chromium directly inside `warp-<node>` with an isolated profile, avoiding cross-session cookie collisions entirely.
|
||||
|
||||
---
|
||||
|
||||
## 3. Automated Driver Mechanics
|
||||
|
||||
### 3.1 Fetching Authorization Payload
|
||||
From the node's running browser tab sitting on `/access/verification`:
|
||||
```javascript
|
||||
const res = await fetch('/api/hatch/age-confirmation/linking-web-auth?account_type=instagram', {
|
||||
headers: { 'Accept': 'application/json' }
|
||||
}).then(r => r.json());
|
||||
// res.url: https://www.instagram.com/fxcal/auth/login/?app_id=...&next=...
|
||||
```
|
||||
|
||||
### 3.2 Automated Headless Linking Flow
|
||||
1. Rather than opening a blocked popup, the driver navigates a dedicated worker tab inside the node's namespace (`warp-<node>`) to `res.url`.
|
||||
2. Inspects form fields:
|
||||
- Username: `input[name="username"]`
|
||||
- Password: `input[name="password"]`
|
||||
- Submit: `button[type="submit"]`
|
||||
3. If 2FA prompt appears (`input[name="verificationCode"]` or email security code `auth_platform/codeentry`), handles code entry.
|
||||
4. Handles Meta Accounts Center confirmation button: `"Confirm"`, `"Allow"`, or `"Continue as <username>"`.
|
||||
5. Upon redirect back to `https://muse.ai/`, checks for DOM chat markers (`"Connected"`, `"Chats"`, or URL `/`).
|
||||
6. Updates node status in `ACCOUNTS.md` to `active`.
|
||||
|
||||
### 3.3 Singular Email Multi-Client Onboarding via RPA
|
||||
- **The Concept**: For agency onboarding efficiency, an RPA pipeline can provision and link accounts for multiple consenting client nodes backed by sub-addressing / plus-addressing (e.g., `agency+client_node@domain.com`) or a managed singular operator email inbox.
|
||||
- **RPA Capabilities**:
|
||||
- Automatically spins up the Instagram registration flow (submitting username, password, birthdate).
|
||||
- Listens to the incoming email stream via IMAP / Gmail API / maildrop to ingest the Instagram security code / OTP without human roundtrips.
|
||||
- Automatically submits the received code into the waiting Instagram code entry screen (`auth_platform/codeentry`).
|
||||
- Solves any automated challenges/captchas through authorized agency captcha-solving harnesses.
|
||||
- Passes the linked identity to Meta Accounts Center to clear the Muse age gate in seconds per node.
|
||||
|
||||
---
|
||||
|
||||
## 4. Pool CLI Surface (`super cred pool`)
|
||||
|
||||
Planned CLI commands to be exposed once implemented:
|
||||
|
||||
```bash
|
||||
# Check status of the credential pool
|
||||
super cred pool status
|
||||
|
||||
# Add a provisioned Instagram credential to the encrypted pool
|
||||
super cred pool add --username <user> --password-file <path> [--totp-secret <secret>]
|
||||
|
||||
# Trigger automated linking for a node in verification status
|
||||
super cred link-instagram --node <node> --auto
|
||||
|
||||
# Human-in-the-loop manual fallback (current default)
|
||||
super cred link-instagram --node <node> --human
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 5. Security & Risk Mitigations
|
||||
|
||||
1. **Anti-Fingerprinting**: All Meta navigation occurs strictly inside the client's assigned `warp-<node>` network namespace to ensure consistent egress IP and prevent cross-node contamination.
|
||||
2. **Audit Logging**: Every pool acquisition and release event is recorded with timestamps in `job-log.jsonl` with credentials scrubbed/redacted.
|
||||
3. **Graceful Human Escalation**: If Meta serves an anti-automation challenge (e.g., CAPTCHA, SMS checkpoint), the automated pool driver immediately falls back to the Human-in-the-Loop Tailscale portal notification.
|
||||
@@ -1,16 +1,20 @@
|
||||
# NetVM Nodes (bl)
|
||||
# NetVM nodes
|
||||
|
||||
Unified naming: node == agent == profile == API account.
|
||||
Per-profile persistent map: **profile = node = Warp identity = deterministic
|
||||
network slot (veth IP, CDP port) = consistent egress IP.**
|
||||
|
||||
| node | netns | egress_ip | cdp_port | status | agent |
|
||||
|------|-------|-----------|----------|--------|-------|
|
||||
| muse | warp-muse | 104.28.195.181 | 9410 | active | muse (ltd.pixels.ltd@gmail.com, email_otp) |
|
||||
| pip | warp-pip | 104.28.195.181 | 9420 | active | pip (piparada, phone_otp, needs re-auth) |
|
||||
Egress IPs may overlap between nodes (same Cloudflare colo / anycast exit).
|
||||
That is expected and fine. What persists — and what is mapped here — is the
|
||||
per-profile pattern: each chrome-box profile keeps its own identity, its own
|
||||
netns, its own veth/CDP slot, and its own consistent egress. One account, one
|
||||
stable network presence.
|
||||
|
||||
## History
|
||||
- 2026-10-03: Renamed smoke->muse, phone-test->pip for unified naming.
|
||||
Profiles preserved, sessions persisted (muse). WireGuard identities renamed.
|
||||
| 646 | warp-646 | 104.28.195.181 | 9430 | active | 646 (phone_otp, first Meta Account option) |
|
||||
| opm | warp-opm | 104.28.195.181 | 9440 | active | opm (artglobal.cc@gmail.com, email_otp, Nico Parada) |
|
||||
| def | warp-def | 104.28.195.181 | 9450 | active | def (defnotabotnet@gmail.com, email_otp, IG paradahub) |
|
||||
| dev | warp-dev | 104.28.195.181 | 9455 | active | dev (paradaproduced@gmail.com, email_otp, IG veryraremeta) |
|
||||
| profile/node | netns | warp identity | veth IP | CDP port | egress IP | tail IP | notes |
|
||||
|--------------|-------|---------------|---------|----------|-----------|---------|-------|
|
||||
| tp | warp-tp | /etc/netvm/tp.conf (wgcf, 2026-10-03) | 10.201.149.2 | 9277 | 104.28.203.246 | — | laptop; orchestrator + first node; UP, handshake+egress verified 2026-10-03 |
|
||||
| smoke | warp-smoke | /etc/netvm/smoke.conf (wgcf, 2026-10-03) | 10.201.87.2 | 9410 | 104.28.203.246 | — | laptop; dedicated test rig (chrome-box profile smoke); UP, handshake+egress+CDP+muse.ai verified 2026-10-03 |
|
||||
| smoke2 | warp-smoke2 | /etc/netvm/smoke2.conf (wgcf, 2026-10-03) | 10.201.117.2 | 9585 | 104.28.227.184 | — | laptop; dedicated test rig (chrome-box profile smoke2); UP, handshake+egress+CDP+muse.ai verified 2026-10-03 |
|
||||
|
||||
|
||||
CDP: `http://<veth IP>:<CDP port>/json/list` from the host, or
|
||||
`ssh -L <port>:<veth IP>:<port> <user>@<tail IP>` for remote automation.
|
||||
|
||||
@@ -1,10 +1,5 @@
|
||||
# NetVM
|
||||
|
||||
> [!IMPORTANT]
|
||||
> **CRITICAL POLICY: MAIN CHAT PRESERVATION**
|
||||
> Avoid using Main Chat whenever possible. When Main Chat gets bogged down with automated entries, scheduled job triggers, log dumps, or chatter, **work stops actually getting done**.
|
||||
> All inputs, scheduled jobs, health checks, and inter-agent coordination must be relayed via designated **sidechats** / **side agents**, or provided via **file transfers**. See [CHAT_POLICY.md](file:///home/super/Projects/NetVM/CHAT_POLICY.md) for full specifications.
|
||||
|
||||
Fleet networking layer. Every node gets a stable network identity; every
|
||||
byte of automation traffic is attributable, consistent, and boring — the
|
||||
way good citizens look to the rest of the internet.
|
||||
@@ -124,36 +119,14 @@ veth IPs aren't routable off the host and Warp forwards no inbound traffic.
|
||||
- `bin/netvm-fleet.sh` — operator fleet control over the tailnet (topology/up/down/ssh/exec/cdp per node).
|
||||
- `bin/netvm-exec.sh <node> -- <cmd>` — run a command inside the node's netns as the invoking user (the agent-friendly primitive).
|
||||
- `bin/netvm-enter.sh` — root worker behind netvm-exec/netvm-chrome (allowlisted; enters netns, fixes DNS, drops privs).
|
||||
- `bin/netvm-chrome.sh <profile> [url]` — launch a chrome-box profile in its netns (CDP on by default; --headless for agents).
|
||||
- `bin/netvm-chrome.sh <profile> [url]` — launch a chrome-box profile in its netns (CDP on by default; --headless for agents; --contained adds a bwrap fs jail inside the netns: profile home only + vault read-only at /tmp/vault, Chromium sandbox stays on).
|
||||
- `bin/netvm-proton.sh <profile> -- <args>` — run proton-cli as the profile's Proton identity (1:1:1 profile = node = warp identity = proton-cli profile) inside the netns + bwrap jail (config dir + static binary only, PROTON_NO_INPUT=1). Human creates the session once: `proton-cli -p <profile> account login`.
|
||||
- `bin/netvm-cdp.sh <profile>` — print the CDP endpoint + SSH forward.
|
||||
- `bin/netvm-cdp-relay.py` — veth-IP→loopback TCP relay for CDP (pidfile-supervised).
|
||||
- `bin/netvm-names.sh` — shared naming: netns, hashed iface tags, veth subnet, CDP port.
|
||||
- NODES.md — the network registry: profile/node -> netns -> Warp identity -> veth IP -> CDP port -> egress IP.
|
||||
- ACCOUNTS.md — the secret-free login registry: login -> profile/node -> purpose -> auth state (no credentials, ever).
|
||||
- `bin/netvm-accounts.sh` — operator view: registry joined with live node state.
|
||||
- `bin/meta-ac-snapshot.py` — Meta Accounts Center change detector: CDP snapshot
|
||||
(redirect chain + DOM markers) diffed against `snapshots/meta-ac/baseline.json`;
|
||||
outcomes PASS/CHANGED/FAIL, `--promote` after human review.
|
||||
- docs/META-ACCOUNTS-API.md — separate API for accountscenter.meta.com
|
||||
(linkage/security surface; credential-isolated from phone-OTP).
|
||||
- docs/PHONE-OTP.md — phone-number OTP login flow for muse.ai (proven on bl).
|
||||
- `bin/accounts-health.py` — per-account CDP session probe (runs inside the netns).
|
||||
- `bin/accounts-health.sh` — aggregates account vitality from ACCOUNTS.md,
|
||||
signs + POSTs to the board health ingest (systemd timer, every 15 min).
|
||||
- `bin/muse -a <account> [args]` — interactive terminal entrypoint for muse-cli; enforces account selection, auto-refreshes CDP cookies, and provides interactive email/OTP prompt fallback.
|
||||
- `bin/muse-cli-node <node> [args]` — runs muse-cli inside node's netns with dedicated Cloudflare WARP egress & auto-refreshing cookies.
|
||||
- `bin/refresh-node-cookies.py <node>` — extracts fresh cookies from running Chromium CDP in netns into `~/.config/muse-cli/<node>/cookies.txt`.
|
||||
- `bin/muse_hybrid.py` — programmatic hybrid bridge combining fast gateway calls with CDP fallbacks.
|
||||
- `bin/muse-tmux.py` — shared tmux socket manager (`/tmp/tmux-muse.sock`) for agent background execution, pipe-pane logging, and 2h session pruning.
|
||||
- `bin/agent_md.py` — CLI & library for auditing, reading, writing, and synchronizing agent `.md` drive files (`SOUL.md`, `PROACTIVE_PREFERENCES.md`, `HEARTBEAT.md`, etc.) across containers via Hatch WebSocket RPC.
|
||||
- `bin/agent-drive-watchdog.py` — background drive watchdog and auto-healing daemon (every 10m via `agent-drive-watchdog.timer`).
|
||||
- `bin/swarm_worker/` & `bin/swarm-worker-supervise.sh` — supervised autonomous swarm worker daemon executing queued tasks in a hard sandbox.
|
||||
- `bin/fleet-alert-relay.sh` — idempotent alert relay with 3-gate deduplication (watermark + 10m TTL hash + receipt verification) posting critical conditions to `#lobby`.
|
||||
- `shared/operators/` — canonical operator drive markdown templates ensuring agents maintain autonomous loops, active supervision, and self-healing reflexes.
|
||||
- docs/OPERATOR-DRIVE-RUNBOOK.md — operator runbook for auditing and modifying agent `.md` files via Hatch WebSocket RPC and SSH reverse tunnels.
|
||||
- docs/HYBRID-GATEWAY-ADAPTATION.md — architectural guide on the muse-cli fast gateway adaptation and per-node egress isolation.
|
||||
- docs/AGENT-TOOLING.md — guide to agent delegation, shared tmux background tooling, Work Orders (`[WO:...]`), and prompt envelope execution.
|
||||
|
||||
|
||||
## Verification checklist
|
||||
|
||||
|
||||
@@ -1,85 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Per-account session vitality check (runs INSIDE the node's netns).
|
||||
|
||||
Usage: sudo ip netns exec warp-<node> python3 accounts-health.py <cdp_port>
|
||||
|
||||
Probes the account's browser via CDP:
|
||||
- browser reachable
|
||||
- muse.ai tab present
|
||||
- login markers (heuristic: "Log in" button vs user content)
|
||||
|
||||
Prints JSON to stdout. Exit 0 on success, 1 if the browser is unreachable.
|
||||
"""
|
||||
import json, sys, urllib.request
|
||||
|
||||
CDP_PORT = int(sys.argv[1]) if len(sys.argv) > 1 else 9410
|
||||
|
||||
def http(path, timeout=5):
|
||||
with urllib.request.urlopen(f"http://127.0.0.1:{CDP_PORT}{path}",
|
||||
timeout=timeout) as r:
|
||||
return json.loads(r.read())
|
||||
|
||||
result = {"browser_up": False, "muse_tab": False,
|
||||
"session_alive": None, "title": "", "url": "", "detail": ""}
|
||||
|
||||
try:
|
||||
http("/json/version")
|
||||
result["browser_up"] = True
|
||||
except Exception as e:
|
||||
result["detail"] = f"CDP unreachable: {e}"
|
||||
print(json.dumps(result)); sys.exit(1)
|
||||
|
||||
try:
|
||||
tabs = http("/json/list")
|
||||
except Exception as e:
|
||||
result["detail"] = f"tab list failed: {e}"
|
||||
print(json.dumps(result)); sys.exit(0)
|
||||
|
||||
muse_tab = None
|
||||
for t in tabs:
|
||||
url = t.get("url", "")
|
||||
if "muse.ai" in url and t.get("type") == "page":
|
||||
muse_tab = t
|
||||
break
|
||||
|
||||
if not muse_tab:
|
||||
result["detail"] = "no muse.ai tab open"
|
||||
print(json.dumps(result)); sys.exit(0)
|
||||
|
||||
result["muse_tab"] = True
|
||||
result["url"] = muse_tab.get("url", "")
|
||||
result["title"] = muse_tab.get("title", "")
|
||||
|
||||
# DOM heuristic via the tab's debugger socket
|
||||
try:
|
||||
import websocket
|
||||
ws = websocket.create_connection(muse_tab["webSocketDebuggerUrl"], timeout=10)
|
||||
js = """JSON.stringify({
|
||||
loginButtons: [...document.querySelectorAll('button')].filter(
|
||||
b => /^\\s*log\\s*in\\s*$/i.test(b.innerText)).map(b => b.innerText.trim()),
|
||||
hasAvatar: !!document.querySelector(
|
||||
'img[alt*="avatar" i], [data-testid*="avatar" i], [aria-label*="profile" i]'),
|
||||
title: document.title,
|
||||
url: location.href
|
||||
})"""
|
||||
ws.send(json.dumps({"id": 1, "method": "Runtime.evaluate",
|
||||
"params": {"expression": js, "returnByValue": True}}))
|
||||
resp = json.loads(ws.recv())
|
||||
ws.close()
|
||||
dom = json.loads(resp["result"]["result"]["value"])
|
||||
# Heuristic: login buttons present + no avatar => logged out.
|
||||
# access/verification gate => needs verification.
|
||||
# No login buttons (or avatar present) => likely logged in.
|
||||
if "access/verification" in dom["url"]:
|
||||
result["session_alive"] = False
|
||||
result["detail"] = "age verification gate (access/verification)"
|
||||
elif dom["loginButtons"] and not dom["hasAvatar"]:
|
||||
result["session_alive"] = False
|
||||
result["detail"] = f"login wall visible: {dom['loginButtons'][:2]}"
|
||||
else:
|
||||
result["session_alive"] = True
|
||||
result["detail"] = "no login wall detected"
|
||||
except Exception as e:
|
||||
result["detail"] = f"DOM check failed: {e}"
|
||||
|
||||
print(json.dumps(result))
|
||||
@@ -1,97 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
# accounts-health.sh — account session vitality reporter for the front-door network.
|
||||
#
|
||||
# Reads ACCOUNTS.md, probes each account's browser via CDP (inside its NetVM
|
||||
# netns) for session liveness, and emits a JSON report. Optionally signs and
|
||||
# POSTs it to the board health ingest, following the health-report.sh
|
||||
# convention: payload is <machine>\n<ts>\n<facts-json>, namespace "health".
|
||||
#
|
||||
# Usage: accounts-health.sh [--no-post]
|
||||
#
|
||||
# Cron (on bl, every 15 min):
|
||||
# */15 * * * * ~/Projects/NetVM/bin/accounts-health.sh >/dev/null 2>&1
|
||||
#
|
||||
# Health key setup (once, on bl):
|
||||
# ssh-keygen -t ed25519 -N "" -f ~/.ssh/muse-health
|
||||
# # operator registers the pubkey on the VM:
|
||||
# echo "bl $(cat ~/.ssh/muse-health.pub)" \
|
||||
# | ssh super@34.139.37.135 "sudo tee -a /srv/board/health_signers"
|
||||
set -u
|
||||
|
||||
NETVM_DIR="${NETVM_DIR:-$HOME/Projects/NetVM}"
|
||||
ACCOUNTS="$NETVM_DIR/ACCOUNTS.md"
|
||||
CHECKER="$NETVM_DIR/bin/accounts-health.py"
|
||||
MACHINE="${MUSE_MACHINE:-bl}"
|
||||
KEY="${HEALTH_KEY:-$HOME/.ssh/muse-health}"
|
||||
ENDPOINT="${HEALTH_ENDPOINT:-https://board.muse-dev.online/api/health/report}"
|
||||
POST=1
|
||||
[ "${1:-}" = "--no-post" ] && POST=0
|
||||
|
||||
[ -f "$ACCOUNTS" ] || { echo "accounts-health: $ACCOUNTS missing" >&2; exit 1; }
|
||||
[ -f "$CHECKER" ] || { echo "accounts-health: $CHECKER missing" >&2; exit 1; }
|
||||
|
||||
TS="$(date +%s)"
|
||||
TMP="$(mktemp -d)"
|
||||
trap 'rm -rf "$TMP"' EXIT
|
||||
|
||||
# Parse ACCOUNTS.md pipe table, probe each account inside its netns
|
||||
NETVM_DIR="$NETVM_DIR" python3 - > "$TMP/facts.json" <<'PYEOF'
|
||||
import json, os, subprocess, time
|
||||
netvm = os.environ["NETVM_DIR"]
|
||||
rows = []
|
||||
for line in open(os.path.join(netvm, "ACCOUNTS.md")):
|
||||
line = line.strip()
|
||||
if not line.startswith("|"):
|
||||
continue
|
||||
cells = [c.strip() for c in line.strip("|").split("|")]
|
||||
if len(cells) < 13 or cells[0] in ("agent", "-------", ""):
|
||||
continue
|
||||
if cells[10] not in ("", "-"):
|
||||
rows.append({"agent": cells[0], "node": cells[1],
|
||||
"status": cells[8], "cdp_port": cells[10]})
|
||||
checker = os.path.join(netvm, "bin", "accounts-health.py")
|
||||
out = {}
|
||||
for a in rows:
|
||||
try:
|
||||
r = subprocess.run(
|
||||
["sudo", "-n", "ip", "netns", "exec", f"warp-{a['node']}",
|
||||
"python3", checker, a["cdp_port"]],
|
||||
capture_output=True, text=True, timeout=60)
|
||||
res = json.loads(r.stdout.strip().splitlines()[-1])
|
||||
res["registry_status"] = a["status"]
|
||||
out[a["agent"]] = res
|
||||
except Exception as e:
|
||||
out[a["agent"]] = {"browser_up": False, "session_alive": None,
|
||||
"detail": f"probe failed: {e}",
|
||||
"registry_status": a["status"]}
|
||||
print(json.dumps({"accounts": out, "checked_at": int(time.time())}, indent=2))
|
||||
PYEOF
|
||||
|
||||
if [ "$POST" -eq 0 ]; then
|
||||
cat "$TMP/facts.json"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
if [ ! -f "$KEY" ]; then
|
||||
echo "accounts-health: $KEY missing — printing JSON, not posting (see header for key setup)" >&2
|
||||
cat "$TMP/facts.json"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
printf '%s\n%s\n' "$MACHINE" "$TS" > "$TMP/payload"
|
||||
FACTS_JSON="$(cat "$TMP/facts.json")"
|
||||
printf '%s' "$FACTS_JSON" >> "$TMP/payload"
|
||||
ssh-keygen -Y sign -f "$KEY" -n health "$TMP/payload" >/dev/null 2>&1
|
||||
SIG="$(cat "$TMP/payload.sig")"
|
||||
python3 - "$MACHINE" "$TS" "$FACTS_JSON" "$SIG" <<'PYEOF' > "$TMP/body.json"
|
||||
import json, sys
|
||||
machine, ts, facts_json, sig = sys.argv[1], int(sys.argv[2]), sys.argv[3], sys.argv[4]
|
||||
body = {"machine": machine, "ts": ts,
|
||||
"facts": json.loads(facts_json), "facts_json": facts_json,
|
||||
"signature": sig}
|
||||
print(json.dumps(body))
|
||||
PYEOF
|
||||
|
||||
curl -s -X POST "$ENDPOINT" -H 'Content-Type: application/json' \
|
||||
--data @"$TMP/body.json" | head -c 300
|
||||
echo
|
||||
Executable
+654
@@ -0,0 +1,654 @@
|
||||
#!/usr/bin/env python3
|
||||
"""agent-cognitive-probe.py — Real-time Cognitive Sensing and Agent Menu Navigation.
|
||||
|
||||
Directly probes Cloud Muse / Hatch browser runtime via CDP:
|
||||
1. Passive Cognitive Sensing (zero-click):
|
||||
- Token streaming / generation state (stop button presence)
|
||||
- Typing / thinking indicators
|
||||
- Status text displayed under/beside avatar (Connected, Thinking, Working)
|
||||
- Active context (Main chat vs Side chats with thread titles & snippets)
|
||||
- Input wait / parked approval detection
|
||||
2. Active Profile Menu Navigation:
|
||||
- Status panel sliding surface navigation
|
||||
- Tabs: Activity (tasks & processes), Upcoming (timers & recurring cron loops),
|
||||
Approvals, and Identity.
|
||||
3. Cognitive Lock Gate:
|
||||
- Protects single-threaded thought process from interruptions.
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
import urllib.request
|
||||
|
||||
# Node to pinned CDP port mapping
|
||||
NODE_CDP_PORTS = {
|
||||
"muse": 9410,
|
||||
"pip": 9420,
|
||||
"646": 9430,
|
||||
"opm": 9440,
|
||||
"def": 9450,
|
||||
"dev": 9455,
|
||||
"muse-main": 9410,
|
||||
}
|
||||
|
||||
REMOTE_HOST = "100.123.153.75" # bl control node
|
||||
|
||||
|
||||
def is_running_on_bl():
|
||||
"""Detect if we are running locally on bl or on tp/remote."""
|
||||
try:
|
||||
import socket
|
||||
hn = socket.gethostname().lower()
|
||||
if "bl" in hn:
|
||||
return True
|
||||
except Exception:
|
||||
pass
|
||||
# Check if network namespaces exist locally
|
||||
return os.path.exists("/var/run/netns/warp-muse") or os.path.exists("/run/netns/warp-muse")
|
||||
|
||||
|
||||
def run_cdp_eval_inside_netns(node, js_code, timeout=8):
|
||||
"""Executes a JS snippet against the node's browser via CDP inside its netns."""
|
||||
port = NODE_CDP_PORTS.get(node)
|
||||
if not port:
|
||||
return {"error": f"Unknown node '{node}'"}
|
||||
|
||||
# Python runner script to execute inside the target environment
|
||||
runner_code = f'''
|
||||
import json, sys, urllib.request
|
||||
try:
|
||||
import websocket
|
||||
except ImportError:
|
||||
print(json.dumps({{"error": "websocket package missing"}}))
|
||||
sys.exit(1)
|
||||
|
||||
try:
|
||||
with urllib.request.urlopen("http://127.0.0.1:{port}/json/list", timeout=3) as r:
|
||||
targets = json.load(r)
|
||||
pages = [t for t in targets if t.get("type") == "page"]
|
||||
if not pages:
|
||||
print(json.dumps({{"error": "No active page target"}}))
|
||||
sys.exit(0)
|
||||
|
||||
ws_url = pages[0]["webSocketDebuggerUrl"]
|
||||
ws = websocket.create_connection(ws_url, timeout={timeout})
|
||||
|
||||
ws.send(json.dumps({{
|
||||
"id": 1,
|
||||
"method": "Runtime.evaluate",
|
||||
"params": {{
|
||||
"expression": {json.dumps(js_code)},
|
||||
"returnByValue": True,
|
||||
"awaitPromise": True
|
||||
}}
|
||||
}}))
|
||||
|
||||
res = None
|
||||
for _ in range(30):
|
||||
msg = json.loads(ws.recv())
|
||||
if msg.get("id") == 1:
|
||||
res = msg.get("result", {{}}).get("result", {{}}).get("value")
|
||||
break
|
||||
|
||||
print(json.dumps({{"ok": True, "value": res}}))
|
||||
except Exception as e:
|
||||
print(json.dumps({{"error": str(e)}}))
|
||||
'''
|
||||
|
||||
if is_running_on_bl():
|
||||
cmd = ["sudo", "ip", "netns", "exec", f"warp-{node}", "python3", "-c", runner_code]
|
||||
else:
|
||||
# Wrap via ssh to bl
|
||||
# Use python3 on bl directly executing inside netns
|
||||
remote_cmd = f"sudo ip netns exec warp-{node} python3 -c {subprocess.list2cmdline([runner_code])}"
|
||||
cmd = ["ssh", "-q", f"super@{REMOTE_HOST}", remote_cmd]
|
||||
|
||||
try:
|
||||
proc = subprocess.run(cmd, capture_output=True, text=True, timeout=timeout + 5)
|
||||
if proc.returncode != 0 and not proc.stdout:
|
||||
return {"error": proc.stderr.strip() or f"Process exited with {proc.returncode}"}
|
||||
|
||||
# Parse output line that contains valid json
|
||||
for line in proc.stdout.strip().splitlines():
|
||||
line = line.strip()
|
||||
if line.startswith("{") and line.endswith("}"):
|
||||
try:
|
||||
data = json.loads(line)
|
||||
if "ok" in data:
|
||||
return data["value"]
|
||||
if "error" in data:
|
||||
return {"error": data["error"]}
|
||||
except Exception:
|
||||
continue
|
||||
return {"error": proc.stdout.strip() or proc.stderr.strip()}
|
||||
except subprocess.TimeoutExpired:
|
||||
return {"error": "CDP probe timed out"}
|
||||
except Exception as e:
|
||||
return {"error": str(e)}
|
||||
|
||||
|
||||
JS_PASSIVE_COGNITIVE = """(() => {
|
||||
// 1. Generation & Thinking signals
|
||||
const stopBtn = document.querySelector('[data-testid="hatch-composer-stop-button"]');
|
||||
const isGenerating = !!stopBtn;
|
||||
|
||||
const typingEl = document.querySelector('[data-testid="hatch-chat-typing-indicator"]');
|
||||
const isTyping = !!typingEl && typingEl.textContent.trim().length > 0;
|
||||
|
||||
// 2. Avatar / Status text
|
||||
const statusTextEl = document.querySelector('.group\\\\/status-avatar span, [class*="status-avatar"] span, span[class*="text-body-status"]');
|
||||
const avatarStatus = statusTextEl ? (statusTextEl.innerText || '').trim() : '';
|
||||
|
||||
// 3. Thread context & active URL
|
||||
const url = window.location.href;
|
||||
const isMainChat = url === 'https://muse.ai/' || url.endsWith('/thread/new');
|
||||
const pageTitle = document.title;
|
||||
|
||||
// 4. Side chats overview
|
||||
const sideRows = Array.from(document.querySelectorAll('[data-testid="hatch-thread-row"]')).map(r => {
|
||||
const text = (r.innerText || '').trim().replace(/\\\\n+/g, ' — ');
|
||||
return text;
|
||||
}).slice(0, 5);
|
||||
|
||||
// 5. Input wait / Parked prompt cards
|
||||
// Detect buttons asking for approval / input in the message flow
|
||||
const actionBtns = Array.from(document.querySelectorAll('div[data-message-id] button')).map(b => (b.innerText || '').trim()).filter(t => /approve|confirm|proceed|resume|start|allow/i.test(t));
|
||||
const isInputWait = actionBtns.length > 0 || /asking for input|pending approval/i.test(document.body.innerText.slice(-600));
|
||||
|
||||
// 6. Status panel state
|
||||
const panel = document.querySelector('[data-testid="hatch-status-panel-sliding-surface"]');
|
||||
const panelOpen = !!panel && panel.getBoundingClientRect().width > 0;
|
||||
|
||||
return {
|
||||
url: url,
|
||||
title: pageTitle,
|
||||
is_main_chat: isMainChat,
|
||||
is_generating: isGenerating,
|
||||
is_typing: isTyping,
|
||||
avatar_status: avatarStatus,
|
||||
is_input_wait: isInputWait,
|
||||
pending_actions: actionBtns,
|
||||
panel_open: panelOpen,
|
||||
side_chats: sideRows
|
||||
};
|
||||
})()"""
|
||||
|
||||
|
||||
def get_passive_cognitive_state(node):
|
||||
"""Gathers passive cognitive signals without altering UI state."""
|
||||
if not is_running_on_bl():
|
||||
try:
|
||||
cmd = ["ssh", "-q", "-o", "ConnectTimeout=5", f"super@{REMOTE_HOST}",
|
||||
f"python3 /home/super/Projects/NetVM/bin/agent-cognitive-probe.py status {node} --json"]
|
||||
proc = subprocess.run(cmd, capture_output=True, text=True, timeout=10)
|
||||
if proc.returncode == 0 and proc.stdout.strip():
|
||||
data = json.loads(proc.stdout.strip())
|
||||
if isinstance(data, list) and len(data) > 0:
|
||||
return data[0]
|
||||
elif isinstance(data, dict):
|
||||
return data
|
||||
except Exception as e:
|
||||
return {
|
||||
"node": node,
|
||||
"status": "DARK",
|
||||
"error": f"Remote delegation failed: {e}",
|
||||
"cognitive_lock": False,
|
||||
"lock_reason": None,
|
||||
}
|
||||
|
||||
raw = run_cdp_eval_inside_netns(node, JS_PASSIVE_COGNITIVE)
|
||||
if not isinstance(raw, dict) or "error" in raw:
|
||||
return {
|
||||
"node": node,
|
||||
"status": "DARK",
|
||||
"error": raw.get("error", "Unknown error") if isinstance(raw, dict) else str(raw),
|
||||
"cognitive_lock": False,
|
||||
"lock_reason": None,
|
||||
}
|
||||
|
||||
is_generating = raw.get("is_generating", False)
|
||||
is_typing = raw.get("is_typing", False)
|
||||
is_input_wait = raw.get("is_input_wait", False)
|
||||
avatar_status = raw.get("avatar_status", "")
|
||||
is_main = raw.get("is_main_chat", True)
|
||||
|
||||
# Determine synthesized cognitive state
|
||||
if is_generating or is_typing or "thinking" in avatar_status.lower():
|
||||
state = "THINKING"
|
||||
locked = True
|
||||
reason = "Agent is actively generating tokens / thinking (stop button active)"
|
||||
elif is_input_wait:
|
||||
state = "INPUT_WAIT"
|
||||
locked = True
|
||||
reason = "Agent is waiting for operator or system input on a parked prompt"
|
||||
elif "working" in avatar_status.lower() or "making" in avatar_status.lower():
|
||||
state = "WORKING"
|
||||
locked = True
|
||||
reason = f"Avatar status indicates work in progress: '{avatar_status}'"
|
||||
elif not is_main:
|
||||
state = "SIDECHAT_IDLE"
|
||||
locked = False
|
||||
reason = None
|
||||
else:
|
||||
state = "IDLE"
|
||||
locked = False
|
||||
reason = None
|
||||
|
||||
return {
|
||||
"node": node,
|
||||
"status": state,
|
||||
"cognitive_lock": locked,
|
||||
"lock_reason": reason,
|
||||
"details": raw,
|
||||
}
|
||||
|
||||
|
||||
JS_NAVIGATE_MENU_TEMPLATE = """(async () => {
|
||||
// 1. Ensure panel is open
|
||||
let panel = document.querySelector('[data-testid="hatch-status-panel-sliding-surface"]');
|
||||
if (!panel) {
|
||||
// Try clicking avatar or trigger
|
||||
const avatarBtn = document.querySelector('[role="img"][aria-label*="avatar"], button[aria-label*="avatar"], .group\\\\/status-avatar');
|
||||
if (avatarBtn) avatarBtn.click();
|
||||
await new Promise(r => setTimeout(r, 350));
|
||||
}
|
||||
|
||||
// 2. Click requested tab if specified
|
||||
const targetTab = "%(tab)s";
|
||||
if (targetTab && targetTab !== "all") {
|
||||
const btn = document.querySelector('button[aria-label="' + targetTab + '"]');
|
||||
if (btn) {
|
||||
btn.click();
|
||||
await new Promise(r => setTimeout(r, 400));
|
||||
}
|
||||
}
|
||||
|
||||
panel = document.querySelector('[data-testid="hatch-status-panel-sliding-surface"]');
|
||||
if (!panel) return {error: "Status panel not rendered"};
|
||||
|
||||
// Read full text and structural items
|
||||
const rawText = panel.innerText || '';
|
||||
const lines = rawText.split('\\n').map(s => s.trim()).filter(Boolean);
|
||||
|
||||
// Extract activity / timer items
|
||||
const items = [];
|
||||
const buttons = Array.from(panel.querySelectorAll('button')).filter(b => !b.getAttribute('aria-label') && (b.innerText || '').length > 0);
|
||||
buttons.forEach(b => {
|
||||
const text = (b.innerText || '').trim();
|
||||
const parts = text.split('\\n').map(s => s.trim()).filter(Boolean);
|
||||
if (parts.length >= 2) {
|
||||
items.push({
|
||||
title: parts[0],
|
||||
detail: parts[1],
|
||||
time: parts.length > 2 ? parts[2] : null
|
||||
});
|
||||
}
|
||||
});
|
||||
|
||||
return {
|
||||
raw_text: rawText,
|
||||
lines: lines,
|
||||
items: items
|
||||
};
|
||||
})()"""
|
||||
|
||||
|
||||
def get_agent_menu(node, tab="all"):
|
||||
"""Navigates and extracts data from the agent profile / status panel."""
|
||||
if not is_running_on_bl():
|
||||
try:
|
||||
cmd = ["ssh", "-q", "-o", "ConnectTimeout=5", f"super@{REMOTE_HOST}",
|
||||
f"python3 /home/super/Projects/NetVM/bin/agent-cognitive-probe.py menu {node} {tab} --json"]
|
||||
proc = subprocess.run(cmd, capture_output=True, text=True, timeout=15)
|
||||
if proc.returncode == 0 and proc.stdout.strip():
|
||||
return json.loads(proc.stdout.strip())
|
||||
except Exception as e:
|
||||
return {
|
||||
"node": node,
|
||||
"tab": tab,
|
||||
"data": {"error": f"Remote delegation failed: {e}"}
|
||||
}
|
||||
|
||||
tab_map = {
|
||||
"activity": "Activity",
|
||||
"upcoming": "Upcoming",
|
||||
"approvals": "Approvals",
|
||||
"identity": "Identity",
|
||||
"all": "all",
|
||||
}
|
||||
target_tab = tab_map.get(tab.lower(), "Activity")
|
||||
|
||||
# If all is requested, gather activity, upcoming, and approvals
|
||||
if target_tab == "all":
|
||||
result = {}
|
||||
for sub_tab in ["Activity", "Upcoming", "Approvals"]:
|
||||
js = JS_NAVIGATE_MENU_TEMPLATE % {"tab": sub_tab}
|
||||
tab_res = run_cdp_eval_inside_netns(node, js)
|
||||
result[sub_tab.lower()] = tab_res
|
||||
return {
|
||||
"node": node,
|
||||
"menu": result
|
||||
}
|
||||
|
||||
js = JS_NAVIGATE_MENU_TEMPLATE % {"tab": target_tab}
|
||||
res = run_cdp_eval_inside_netns(node, js)
|
||||
return {
|
||||
"node": node,
|
||||
"tab": target_tab,
|
||||
"data": res
|
||||
}
|
||||
|
||||
|
||||
JS_LIVE_SCREEN = """(() => {
|
||||
const ps = Array.from(document.querySelectorAll('p')).map(p => (p.innerText || '').trim()).filter(Boolean);
|
||||
const actionBtns = Array.from(document.querySelectorAll('div[data-message-id] button, [role="log"] button, div[role="status"] button')).map(b => (b.innerText || '').trim()).filter(Boolean);
|
||||
const stopBtn = !!document.querySelector('[data-testid="hatch-composer-stop-button"]');
|
||||
const typing = !!document.querySelector('[data-testid="hatch-chat-typing-indicator"]:not(:empty)');
|
||||
const statusTextEl = document.querySelector('.group\\\\/status-avatar span, [class*="status-avatar"] span, span[class*="text-body-status"]');
|
||||
const avatarStatus = statusTextEl ? (statusTextEl.innerText || '').trim() : '';
|
||||
|
||||
return {
|
||||
title: document.title,
|
||||
url: window.location.href,
|
||||
is_main_chat: window.location.href === 'https://muse.ai/' || window.location.href.endsWith('/thread/new'),
|
||||
is_generating: stopBtn,
|
||||
is_typing: typing,
|
||||
avatar_status: avatarStatus,
|
||||
recent_paragraphs: ps.slice(-8),
|
||||
action_buttons: actionBtns.filter(t => /approve|confirm|proceed|resume|start|allow|review/i.test(t))
|
||||
};
|
||||
})()"""
|
||||
|
||||
|
||||
def get_agent_live_screen(node: str) -> dict:
|
||||
"""Extracts live active chat text, thoughts, prompt blocks, and threads."""
|
||||
if not is_running_on_bl():
|
||||
try:
|
||||
cmd = ["ssh", "-q", "-o", "ConnectTimeout=5", f"super@{REMOTE_HOST}",
|
||||
f"python3 /home/super/Projects/NetVM/bin/agent-cognitive-probe.py read {node} --json"]
|
||||
proc = subprocess.run(cmd, capture_output=True, text=True, timeout=15)
|
||||
if proc.returncode == 0 and proc.stdout.strip():
|
||||
return json.loads(proc.stdout.strip())
|
||||
except Exception as e:
|
||||
return {"node": node, "error": f"Remote delegation failed: {e}"}
|
||||
|
||||
screen = run_cdp_eval_inside_netns(node, JS_LIVE_SCREEN)
|
||||
if not isinstance(screen, dict) or "error" in screen:
|
||||
return {"node": node, "error": screen.get("error", "Failed to inspect screen") if isinstance(screen, dict) else str(screen)}
|
||||
|
||||
sidechats = []
|
||||
try:
|
||||
cli_cmd = ["/home/super/Projects/NetVM/bin/muse-cli-node", node, "threads"]
|
||||
proc = subprocess.run(cli_cmd, capture_output=True, text=True, timeout=8)
|
||||
if proc.returncode == 0 and proc.stdout.strip():
|
||||
threads_data = json.loads(proc.stdout.strip())
|
||||
if isinstance(threads_data, list):
|
||||
sidechats = threads_data[:8]
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
return {
|
||||
"node": node,
|
||||
"screen": screen,
|
||||
"sidechats": sidechats
|
||||
}
|
||||
|
||||
|
||||
JS_NAVIGATE_MAIN_CHAT = """(() => {
|
||||
try {
|
||||
const url = window.location.href;
|
||||
if (url === 'https://muse.ai/' || url === 'https://muse.ai/thread/new') {
|
||||
return { ok: true, already_main: true };
|
||||
}
|
||||
const candidates = Array.from(document.querySelectorAll('a, button, [role="button"], [data-testid]'));
|
||||
const mainBtn = candidates.find(b => {
|
||||
const t = (b.innerText || '').trim().toLowerCase();
|
||||
return t === 'main chat' || b.getAttribute('aria-label') === 'Main chat' || b.getAttribute('data-testid') === 'hatch-sidebar-main-chat';
|
||||
});
|
||||
if (mainBtn) {
|
||||
mainBtn.click();
|
||||
return { ok: true, method: 'click' };
|
||||
}
|
||||
window.location.href = 'https://muse.ai/';
|
||||
return { ok: true, method: 'navigate' };
|
||||
} catch (e) {
|
||||
return { error: String(e) };
|
||||
}
|
||||
})()"""
|
||||
|
||||
|
||||
def navigate_to_main_chat(node: str) -> dict:
|
||||
"""Navigates the agent browser session back to Main Chat (https://muse.ai/)."""
|
||||
if not is_running_on_bl():
|
||||
try:
|
||||
cmd = ["ssh", "-q", "-o", "ConnectTimeout=5", f"super@{REMOTE_HOST}",
|
||||
f"python3 /home/super/Projects/NetVM/bin/agent-cognitive-probe.py nav-main {node} --json"]
|
||||
proc = subprocess.run(cmd, capture_output=True, text=True, timeout=10)
|
||||
if proc.returncode == 0 and proc.stdout.strip():
|
||||
return json.loads(proc.stdout.strip())
|
||||
except Exception as e:
|
||||
return {"error": str(e)}
|
||||
|
||||
return run_cdp_eval_inside_netns(node, JS_NAVIGATE_MAIN_CHAT)
|
||||
|
||||
|
||||
def cmd_nav_main(args):
|
||||
node = getattr(args, "agent", None) or getattr(args, "node", None)
|
||||
res = navigate_to_main_chat(node)
|
||||
if getattr(args, "json", False):
|
||||
print(json.dumps(res, indent=2))
|
||||
else:
|
||||
if isinstance(res, dict) and res.get("ok"):
|
||||
m = res.get("method") or ("already in main chat" if res.get("already_main") else "default")
|
||||
print(f"✓ Refocused agent '{node}' to Main Chat ({m})")
|
||||
else:
|
||||
err = res.get("error") if isinstance(res, dict) else str(res)
|
||||
print(f"❌ Failed to refocus '{node}' to Main Chat: {err}")
|
||||
|
||||
|
||||
def cmd_read(args):
|
||||
node = getattr(args, "agent", None) or getattr(args, "node", None)
|
||||
data = get_agent_live_screen(node)
|
||||
if getattr(args, "json", False):
|
||||
print(json.dumps(data, indent=2))
|
||||
return
|
||||
|
||||
if "error" in data:
|
||||
print(f"\n❌ Error inspecting screen for {node}: {data['error']}\n")
|
||||
return
|
||||
|
||||
screen = data.get("screen", {})
|
||||
sidechats = data.get("sidechats", [])
|
||||
|
||||
url = screen.get("url", "")
|
||||
mode = "Main Chat" if screen.get("is_main_chat") else "Side Chat"
|
||||
title = screen.get("title", "")
|
||||
avatar = screen.get("avatar_status") or "Connected"
|
||||
gen = "🧠 GENERATING" if screen.get("is_generating") else ("💭 TYPING" if screen.get("is_typing") else "🟢 SETTLED")
|
||||
|
||||
print(f"\n=== LIVE THOUGHT STREAM & ACTIVE CHAT: {node.upper()} ===")
|
||||
print(f"Context: {mode} ({url})")
|
||||
print(f"Title: {title}")
|
||||
print(f"Status: {avatar} | State: {gen}")
|
||||
|
||||
paragraphs = screen.get("recent_paragraphs", [])
|
||||
if paragraphs:
|
||||
print("\n--- ACTIVE CONVERSATION & THOUGHT PARAGRAPHS ---")
|
||||
for p in paragraphs:
|
||||
print(f" • {p}\n")
|
||||
else:
|
||||
print("\n (No text paragraphs visible in current viewport)")
|
||||
|
||||
actions = screen.get("action_buttons", [])
|
||||
if actions:
|
||||
print("--- PENDING ACTION CARDS / APPROVAL BUTTONS ---")
|
||||
for a in actions:
|
||||
print(f" ⚠️ [PROMPT ACTION] {a}")
|
||||
print()
|
||||
|
||||
if sidechats:
|
||||
print("--- RECENT SIDE CHATS & TOPICS ---")
|
||||
for sc in sidechats:
|
||||
sid = sc.get("session_id", "")[:8]
|
||||
stitle = sc.get("title") or "(Untitled sidechat)"
|
||||
upd = sc.get("updated", "")
|
||||
print(f" • [{sid}] {stitle} ({upd})")
|
||||
print()
|
||||
|
||||
|
||||
def format_cognitive_badge(status):
|
||||
badges = {
|
||||
"IDLE": "🟢 IDLE",
|
||||
"SIDECHAT_IDLE": "💬 SIDE_IDLE",
|
||||
"THINKING": "🧠 THINKING",
|
||||
"INPUT_WAIT": "⏸️ INPUT_WAIT",
|
||||
"BUSY_SIDECHAT": "💬 SIDECHAT",
|
||||
"WORKING": "⚙️ WORKING",
|
||||
"DARK": "⚫ DARK",
|
||||
}
|
||||
return badges.get(status, f"❓ {status}")
|
||||
|
||||
|
||||
def cmd_status(args):
|
||||
nodes = getattr(args, "agents", None) or getattr(args, "nodes", None) or ["muse", "pip", "646", "opm", "dev", "def"]
|
||||
results = []
|
||||
for n in nodes:
|
||||
results.append(get_passive_cognitive_state(n))
|
||||
|
||||
if getattr(args, "json", False):
|
||||
print(json.dumps(results, indent=2))
|
||||
return
|
||||
|
||||
print("\n=== AGENT COGNITIVE SENSOR (LIVE DOM PROBE) ===")
|
||||
print(f"{'AGENT':<10} {'COGNITIVE STATE':<18} {'LOCK':<8} {'UNDER-AVATAR':<15} {'DETAILS / CURRENT THOUGHT':<40}")
|
||||
print("-" * 95)
|
||||
for r in results:
|
||||
node = r["node"]
|
||||
status = r["status"]
|
||||
badge = format_cognitive_badge(status)
|
||||
locked = "LOCKED" if r.get("cognitive_lock") else "OPEN"
|
||||
det = r.get("details", {})
|
||||
avatar_status = det.get("avatar_status", "-")
|
||||
reason = r.get("lock_reason") or det.get("title", "Settled")
|
||||
if status == "DARK":
|
||||
reason = r.get("error", "CDP unreachable")
|
||||
avatar_status = "DARK"
|
||||
print(f"{node:<10} {badge:<18} {locked:<8} {avatar_status:<15} {reason[:40]:<40}")
|
||||
print()
|
||||
|
||||
|
||||
def cmd_menu(args):
|
||||
node = getattr(args, "agent", None) or getattr(args, "node", None)
|
||||
tab = getattr(args, "tab", "activity") or "activity"
|
||||
data = get_agent_menu(node, tab)
|
||||
|
||||
if getattr(args, "json", False):
|
||||
print(json.dumps(data, indent=2))
|
||||
return
|
||||
|
||||
print(f"\n=== AGENT MENU: {node.upper()} (TAB: {tab.upper()}) ===")
|
||||
if tab.lower() == "all":
|
||||
menu = data.get("menu", {})
|
||||
for tname, tdata in menu.items():
|
||||
print(f"\n--- {tname.upper()} ---")
|
||||
if isinstance(tdata, dict) and "error" in tdata:
|
||||
print(f" Error: {tdata['error']}")
|
||||
elif isinstance(tdata, dict):
|
||||
items = tdata.get("items", [])
|
||||
if items:
|
||||
for it in items:
|
||||
t_str = f" [{it['time']}]" if it.get("time") else ""
|
||||
print(f" • {it['title']}: {it['detail']}{t_str}")
|
||||
else:
|
||||
raw = tdata.get("raw_text", "")
|
||||
for line in raw.split("\n"):
|
||||
if line.strip():
|
||||
print(f" {line.strip()}")
|
||||
print()
|
||||
return
|
||||
|
||||
tdata = data.get("data", {})
|
||||
if isinstance(tdata, dict) and "error" in tdata:
|
||||
print(f" Error: {tdata['error']}")
|
||||
elif isinstance(tdata, dict):
|
||||
items = tdata.get("items", [])
|
||||
if items:
|
||||
for it in items:
|
||||
t_str = f" [{it['time']}]" if it.get("time") else ""
|
||||
print(f" • {it['title']}: {it['detail']}{t_str}")
|
||||
else:
|
||||
raw = tdata.get("raw_text", "")
|
||||
for line in raw.split("\n"):
|
||||
if line.strip():
|
||||
print(f" {line.strip()}")
|
||||
print()
|
||||
|
||||
|
||||
def cmd_lock(args):
|
||||
node = getattr(args, "agent", None) or getattr(args, "node", None)
|
||||
state = get_passive_cognitive_state(node)
|
||||
if args.json:
|
||||
print(json.dumps(state, indent=2))
|
||||
else:
|
||||
if state.get("cognitive_lock"):
|
||||
print(f"🔴 COGNITIVE LOCK ENGAGED on '{node}' ({state['status']}): {state.get('lock_reason')}")
|
||||
else:
|
||||
print(f"🟢 COGNITIVELY IDLE: Agent '{node}' is free to receive new work without interruption.")
|
||||
|
||||
if state.get("cognitive_lock") and not getattr(args, "force", False):
|
||||
sys.exit(1)
|
||||
sys.exit(0)
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description="Real-time Cognitive Sensing and Agent Menu Navigation")
|
||||
sub = parser.add_subparsers(dest="command")
|
||||
|
||||
p_status = sub.add_parser("status", help="Show cognitive state for all or selected agents")
|
||||
p_status.add_argument("nodes", nargs="*", help="Optional agent names")
|
||||
p_status.add_argument("--json", action="store_true", help="Output JSON")
|
||||
|
||||
p_menu = sub.add_parser("menu", help="Navigate agent profile menu (tasks, timers, approvals, identity)")
|
||||
p_menu.add_argument("node", help="Agent name (muse, pip, 646, opm, dev, def)")
|
||||
p_menu.add_argument("tab", nargs="?", default="activity", choices=["activity", "upcoming", "approvals", "identity", "all"], help="Menu tab to view")
|
||||
p_menu.add_argument("--json", action="store_true", help="Output JSON")
|
||||
|
||||
p_lock = sub.add_parser("lock", help="Check cognitive lock before dispatching work")
|
||||
p_lock.add_argument("node", help="Agent name")
|
||||
p_lock.add_argument("--force", action="store_true", help="Bypass lock check")
|
||||
p_lock.add_argument("--json", action="store_true", help="Output JSON")
|
||||
|
||||
p_read = sub.add_parser("read", help="Extract live active chat text, thought stream, and side chats")
|
||||
p_read.add_argument("node", help="Agent name")
|
||||
p_read.add_argument("--json", action="store_true", help="Output JSON")
|
||||
|
||||
p_nav = sub.add_parser("nav-main", help="Navigate agent browser session back to Main Chat")
|
||||
p_nav.add_argument("node", help="Agent name")
|
||||
p_nav.add_argument("--json", action="store_true", help="Output JSON")
|
||||
|
||||
args = parser.parse_args()
|
||||
if not args.command:
|
||||
# Default to status
|
||||
args.nodes = []
|
||||
args.json = False
|
||||
cmd_status(args)
|
||||
return
|
||||
|
||||
if args.command == "status":
|
||||
cmd_status(args)
|
||||
elif args.command == "menu":
|
||||
cmd_menu(args)
|
||||
elif args.command == "read":
|
||||
cmd_read(args)
|
||||
elif args.command == "lock":
|
||||
cmd_lock(args)
|
||||
elif args.command == "nav-main":
|
||||
cmd_nav_main(args)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -1,200 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
agent-drive-watchdog.py — Automated Drive Watchdog & Healing Daemon for Muse Agents.
|
||||
|
||||
Monitors operational DRIVE scores across the agent fleet (muse, pip, 646, opm, def, dev)
|
||||
via Hatch WebSocket RPC. If any agent's DRIVE score drops below 100 or critical drive
|
||||
files (HEARTBEAT.md, PROACTIVE_PREFERENCES.md, SOUL.md) are degraded or missing:
|
||||
1. Detects degraded state and missing checklist/preferences.
|
||||
2. Selectively auto-heals core drive files using canonical shared templates.
|
||||
3. Preserves MEMORY.md and agent-generated workspace files.
|
||||
4. Records state & healing history to /tmp/agent-drive-watchdog.json.
|
||||
5. Emits structured telemetry to stdout/journal.
|
||||
|
||||
Can be run:
|
||||
- Once: python3 bin/agent-drive-watchdog.py --once
|
||||
- Continuous loop: python3 bin/agent-drive-watchdog.py --interval 600
|
||||
- Via systemd timer: agent-drive-watchdog.timer (every 10m)
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import datetime
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import sys
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
# Add NetVM bin to path
|
||||
BASE_DIR = Path(__file__).resolve().parent.parent
|
||||
sys.path.insert(0, str(BASE_DIR / "bin"))
|
||||
|
||||
from agent_md import audit_agents, write_md, read_md, SHARED_OPERATORS, VALID_ACCOUNTS
|
||||
|
||||
STATE_FILE = Path("/tmp/agent-drive-watchdog.json")
|
||||
LOG_FILE = Path("/tmp/agent-drive-watchdog.log")
|
||||
|
||||
logging.basicConfig(
|
||||
level=logging.INFO,
|
||||
format="%(asctime)s [%(levelname)s] %(message)s",
|
||||
handlers=[
|
||||
logging.StreamHandler(sys.stdout),
|
||||
logging.FileHandler(LOG_FILE, mode="a", encoding="utf-8")
|
||||
]
|
||||
)
|
||||
|
||||
|
||||
def load_state() -> dict:
|
||||
if STATE_FILE.exists():
|
||||
try:
|
||||
return json.loads(STATE_FILE.read_text(encoding="utf-8"))
|
||||
except Exception as e:
|
||||
logging.warning(f"Failed to read existing state file: {e}")
|
||||
return {
|
||||
"last_run": None,
|
||||
"history": [],
|
||||
"agent_status": {},
|
||||
}
|
||||
|
||||
|
||||
def save_state(state: dict):
|
||||
try:
|
||||
# Keep last 50 history entries
|
||||
if len(state.get("history", [])) > 50:
|
||||
state["history"] = state["history"][-50:]
|
||||
STATE_FILE.write_text(json.dumps(state, indent=2), encoding="utf-8")
|
||||
except Exception as e:
|
||||
logging.error(f"Failed to save state file: {e}")
|
||||
|
||||
|
||||
def heal_agent_drive(account: str, audit_data: dict) -> dict:
|
||||
"""Selectively auto-heal core drive files for an agent."""
|
||||
healed = []
|
||||
issues = audit_data.get("issues", [])
|
||||
|
||||
# Check what specifically needs healing
|
||||
need_soul = any("SOUL.md" in iss for iss in issues) or audit_data.get("files", {}).get("SOUL.md", {}).get("size", 0) < 1000
|
||||
need_pro = any("PROACTIVE_PREFERENCES.md" in iss for iss in issues) or audit_data.get("files", {}).get("PROACTIVE_PREFERENCES.md", {}).get("size", 0) < 800
|
||||
need_hb = any("HEARTBEAT.md" in iss for iss in issues) or audit_data.get("files", {}).get("HEARTBEAT.md", {}).get("size", 0) < 300
|
||||
need_context = any("TOOLS.md or USER.md" in iss for iss in issues)
|
||||
|
||||
logging.info(f"[{account.upper()}] Auto-healing drive... need_soul={need_soul}, need_pro={need_pro}, need_hb={need_hb}, need_context={need_context}")
|
||||
|
||||
if need_soul:
|
||||
soul_text = (SHARED_OPERATORS / "SOUL.md").read_text(encoding="utf-8")
|
||||
res = write_md(account, "SOUL.md", soul_text, overwrite=True)
|
||||
healed.append({"file": "SOUL.md", "bytes": res.get("bytes_written")})
|
||||
|
||||
if need_pro:
|
||||
pro_text = (SHARED_OPERATORS / "PROACTIVE_PREFERENCES.md").read_text(encoding="utf-8")
|
||||
res = write_md(account, "PROACTIVE_PREFERENCES.md", pro_text, overwrite=True)
|
||||
healed.append({"file": "PROACTIVE_PREFERENCES.md", "bytes": res.get("bytes_written")})
|
||||
|
||||
if need_hb:
|
||||
hb_text = (SHARED_OPERATORS / "HEARTBEAT.md").read_text(encoding="utf-8")
|
||||
res = write_md(account, "HEARTBEAT.md", hb_text, overwrite=True)
|
||||
healed.append({"file": "HEARTBEAT.md", "bytes": res.get("bytes_written")})
|
||||
|
||||
if need_context:
|
||||
tools_text = (SHARED_OPERATORS / "TOOLS.md").read_text(encoding="utf-8")
|
||||
res_t = write_md(account, "TOOLS.md", tools_text, overwrite=True)
|
||||
healed.append({"file": "TOOLS.md", "bytes": res_t.get("bytes_written")})
|
||||
|
||||
user_text = (SHARED_OPERATORS / "USER.md").read_text(encoding="utf-8")
|
||||
res_u = write_md(account, "USER.md", user_text, overwrite=True)
|
||||
healed.append({"file": "USER.md", "bytes": res_u.get("bytes_written")})
|
||||
|
||||
return {"ok": True, "account": account, "healed": healed}
|
||||
|
||||
|
||||
def run_cycle(auto_heal: bool = True) -> dict:
|
||||
"""Run an audit and healing cycle across all fleet accounts."""
|
||||
now_iso = datetime.datetime.now(datetime.timezone.utc).isoformat()
|
||||
logging.info("Starting fleet drive watchdog audit cycle...")
|
||||
|
||||
state = load_state()
|
||||
state["last_run"] = now_iso
|
||||
|
||||
try:
|
||||
audit_results = audit_agents()
|
||||
except Exception as e:
|
||||
logging.error(f"Audit failed during cycle: {e}")
|
||||
return {"ok": False, "error": str(e)}
|
||||
|
||||
cycle_report = {
|
||||
"timestamp": now_iso,
|
||||
"total_agents": len(audit_results),
|
||||
"high_drive": 0,
|
||||
"degraded": 0,
|
||||
"healed_agents": [],
|
||||
}
|
||||
|
||||
for account, a_data in audit_results.items():
|
||||
score = a_data.get("drive_score", 0)
|
||||
issues = a_data.get("issues", [])
|
||||
status = "HIGH_DRIVE" if score == 100 else "DEGRADED"
|
||||
|
||||
if status == "HIGH_DRIVE":
|
||||
cycle_report["high_drive"] += 1
|
||||
logging.info(f"Agent {account.upper():6}: DRIVE score 100/100 (HIGH DRIVE)")
|
||||
else:
|
||||
cycle_report["degraded"] += 1
|
||||
logging.warning(f"Agent {account.upper():6}: DRIVE score {score}/100 ({status}) - Issues: {', '.join(issues)}")
|
||||
|
||||
if auto_heal:
|
||||
try:
|
||||
heal_res = heal_agent_drive(account, a_data)
|
||||
healed_files = [h["file"] for h in heal_res.get("healed", [])]
|
||||
logging.info(f"Agent {account.upper():6}: Successfully healed files: {', '.join(healed_files)}")
|
||||
cycle_report["healed_agents"].append({
|
||||
"account": account,
|
||||
"prior_score": score,
|
||||
"healed_files": healed_files,
|
||||
})
|
||||
except Exception as e:
|
||||
logging.error(f"Agent {account.upper():6}: Healing failed: {e}")
|
||||
|
||||
state["agent_status"][account] = {
|
||||
"score": score,
|
||||
"status": status,
|
||||
"issues": issues,
|
||||
"last_checked": now_iso,
|
||||
}
|
||||
|
||||
state["history"].append(cycle_report)
|
||||
save_state(state)
|
||||
|
||||
logging.info(f"Watchdog cycle complete. High drive: {cycle_report['high_drive']}/{cycle_report['total_agents']}. Degraded: {cycle_report['degraded']}. Healed: {len(cycle_report['healed_agents'])}.")
|
||||
return cycle_report
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description="NetVM Automated Agent Drive Watchdog & Healing Daemon")
|
||||
parser.add_argument("--once", action="store_true", help="Run a single audit/healing pass and exit")
|
||||
parser.add_argument("--no-heal", action="store_true", help="Audit only; do not auto-heal degraded agents")
|
||||
parser.add_argument("--interval", type=int, default=600, help="Loop interval in seconds (default: 600s / 10m)")
|
||||
parser.add_argument("--status", action="store_true", help="Print recent watchdog status and exit")
|
||||
|
||||
args = parser.parse_args()
|
||||
|
||||
if args.status:
|
||||
state = load_state()
|
||||
print(json.dumps(state, indent=2))
|
||||
return
|
||||
|
||||
if args.once:
|
||||
run_cycle(auto_heal=not args.no_heal)
|
||||
return
|
||||
|
||||
logging.info(f"Starting NetVM Agent Drive Watchdog daemon (interval: {args.interval}s)...")
|
||||
while True:
|
||||
try:
|
||||
run_cycle(auto_heal=not args.no_heal)
|
||||
except Exception as e:
|
||||
logging.error(f"Unexpected error in watchdog loop: {e}", exc_info=True)
|
||||
time.sleep(args.interval)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -1,182 +0,0 @@
|
||||
#!/bin/bash
|
||||
# GOLDEN PATH: container -> VM (34.139.37.135) -> bl (100.123.153.75) -> netns -> browser -> agent
|
||||
# This script is the operator's heartbeat. If it stops, agents go dark.
|
||||
# When debugging: trace each hop. Don't assume -- verify.
|
||||
|
||||
# Operator health monitor for Muse agents on bl.
|
||||
# Checks each agent via API every 5 minutes. If unresponsive:
|
||||
# 1. Restart the browser
|
||||
# 2. Re-check
|
||||
# 3. Log failure if still down
|
||||
#
|
||||
# Run via systemd timer or cron: */5 * * * * /home/super/Projects/NetVM/bin/agent-health.sh
|
||||
#
|
||||
# Agents are defined in ACCOUNTS.md. This script reads the active ones.
|
||||
|
||||
NETVM_BIN="$(cd "$(dirname "$0")" && pwd)"
|
||||
LOG="/tmp/agent-health.log"
|
||||
|
||||
# 2026-10-04: per-node consecutive-API-failure counters. A single
|
||||
# muse-chat-api.py failure must not kill a healthy browser (observed
|
||||
# 2026-10-04 20:48:44 UTC: 646's browser killed on an API timeout while its
|
||||
# CDP port was still listening). Require 2 CONSECUTIVE API failures before
|
||||
# the kill path; the counter resets on any successful check.
|
||||
STATE_DIR="/tmp/agent-health-state"
|
||||
mkdir -p "$STATE_DIR" 2>/dev/null
|
||||
|
||||
check_cdp_port() {
|
||||
# Verify CDP port is actually listening in the netns.
|
||||
# A browser can be running but not bound to CDP (zombie state).
|
||||
local node=$1
|
||||
local cdp_port=$2
|
||||
if sudo ip netns exec "warp-$node" ss -tln 2>/dev/null | grep -q ":$cdp_port "; then
|
||||
return 0
|
||||
else
|
||||
return 1
|
||||
fi
|
||||
}
|
||||
|
||||
# Return codes: 0 = healthy, 1 = API check failed (port OK), 2 = CDP port down.
|
||||
check_agent() {
|
||||
local agent=$1
|
||||
local node=$2
|
||||
local cdp_port=$3
|
||||
|
||||
# First: verify CDP port is listening (catches zombie browsers)
|
||||
if ! check_cdp_port "$node" "$cdp_port"; then
|
||||
echo "$(date -Iseconds) $agent: FAIL (cdp port $cdp_port not listening)" >> "$LOG"
|
||||
return 2
|
||||
fi
|
||||
|
||||
# Then: try API messages command (lightweight check)
|
||||
if timeout 30 "$NETVM_BIN/netvm-exec.sh" "$node" -- python3 "$NETVM_BIN/muse-chat-api.py" --account "$agent" messages 1 > /dev/null 2>&1; then
|
||||
echo "$(date -Iseconds) $agent: OK" >> "$LOG"
|
||||
return 0
|
||||
else
|
||||
echo "$(date -Iseconds) $agent: FAIL (api timeout)" >> "$LOG"
|
||||
return 1
|
||||
fi
|
||||
}
|
||||
|
||||
# 2026-10-04: recent-relaunch guard. chromebox-watchdog.sh (2-min timer) and
|
||||
# this script (5-min timer) could otherwise kill each other's fresh browsers:
|
||||
# a browser just relaunched by the watchdog is still starting when this
|
||||
# script's API check times out on it. Skip the kill path when the main
|
||||
# browser process for this profile launched <2 min ago. Mirrors
|
||||
# chromebox-watchdog.sh's relaunch-loop guard idiom (main process only:
|
||||
# --remote-debugging-port present, no --type= flag; renderer/gpu children
|
||||
# start later than the main process and must not satisfy this check).
|
||||
recent_relaunch() {
|
||||
local cdp_port=$1
|
||||
local _pid _start _now
|
||||
for _pid in $(pgrep -f "chromium.*--remote-debugging-port=${cdp_port}([[:space:]]|$)" 2>/dev/null); do
|
||||
# Skip child processes (renderer, gpu, etc.) — only the main browser counts
|
||||
if ps -o args= -p "$_pid" 2>/dev/null | grep -q -- "--type="; then
|
||||
continue
|
||||
fi
|
||||
_start=$(date -d "$(ps -o lstart= -p "$_pid" 2>/dev/null)" +%s 2>/dev/null || echo 0)
|
||||
_now=$(date +%s)
|
||||
if [ $(( _now - _start )) -lt 120 ] && [ "$_start" -gt 0 ]; then
|
||||
return 0
|
||||
fi
|
||||
done
|
||||
return 1
|
||||
}
|
||||
|
||||
restart_browser() {
|
||||
local agent=$1
|
||||
local cdp_port=$2
|
||||
|
||||
echo "$(date -Iseconds) $agent: restarting browser..." >> "$LOG"
|
||||
# Kill existing by exact PIDs (pkill patterns are unreliable)
|
||||
# Find chromium processes for this profile
|
||||
for pid in $(pgrep -f "chromium.*profiles/$agent" 2>/dev/null); do
|
||||
kill -9 "$pid" 2>/dev/null
|
||||
done
|
||||
sleep 3
|
||||
# Verify port is free before restart
|
||||
if sudo ip netns exec "warp-$agent" ss -tln 2>/dev/null | grep -q ":$cdp_port "; then
|
||||
echo "$(date -Iseconds) $agent: WARNING - port $cdp_port still bound after kill" >> "$LOG"
|
||||
fi
|
||||
# Restart via netvm-chrome.sh in its own systemd scope.
|
||||
# This oneshot service runs with KillMode=control-group, so anything
|
||||
# spawned directly under it (nohup AND setsid both stay in the cgroup)
|
||||
# is SIGKILLed when the service exits — observed 2026-10-03: every
|
||||
# restart "recovered" then died at service teardown, looping forever.
|
||||
# A transient scope escapes the service cgroup and survives.
|
||||
# NOTE: systemd-run --scope WAITS for the scope's processes (even with
|
||||
# --no-block, verified 2026-10-03), so background it — the scope itself
|
||||
# is an independent unit and outlives the wrapper.
|
||||
cd "$NETVM_BIN"
|
||||
systemd-run --user --scope --unit="netvm-chrome-$agent-$(date +%s)" \
|
||||
./netvm-chrome.sh --headless --cdp-port "$cdp_port" "$agent" https://muse.ai \
|
||||
> "/tmp/bl-$agent.log" 2>&1 &
|
||||
sleep 15
|
||||
echo "$(date -Iseconds) $agent: browser restarted" >> "$LOG"
|
||||
}
|
||||
|
||||
# Main
|
||||
echo "=== Health check $(date -Iseconds) ===" >> "$LOG"
|
||||
|
||||
check_one() {
|
||||
local agent=$1
|
||||
local cdp_port=$2
|
||||
# node == agent == profile (unified naming)
|
||||
check_agent "$agent" "$agent" "$cdp_port"
|
||||
local rc=$?
|
||||
|
||||
if [ $rc -eq 0 ]; then
|
||||
# Healthy: reset the consecutive-API-failure counter.
|
||||
rm -f "$STATE_DIR/failcount-$agent" 2>/dev/null
|
||||
return 0
|
||||
fi
|
||||
|
||||
if [ $rc -eq 1 ]; then
|
||||
# API failed but CDP port is listening: this is the false-kill vector
|
||||
# (2026-10-04: single 30s API timeout killed 646's healthy browser).
|
||||
# Require 2 CONSECUTIVE API failures before killing.
|
||||
local count=0
|
||||
local cf="$STATE_DIR/failcount-$agent"
|
||||
[ -f "$cf" ] && count=$(cat "$cf" 2>/dev/null || echo 0)
|
||||
count=$(( count + 1 ))
|
||||
if [ "$count" -lt 2 ]; then
|
||||
echo "$count" > "$cf"
|
||||
echo "$(date -Iseconds) $agent: API failure $count of 2, deferring kill" >> "$LOG"
|
||||
return 0
|
||||
fi
|
||||
rm -f "$cf" 2>/dev/null
|
||||
else
|
||||
# CDP port not listening (rc=2): zombie browser, kill immediately as before.
|
||||
rm -f "$STATE_DIR/failcount-$agent" 2>/dev/null
|
||||
fi
|
||||
|
||||
# De-conflict with chromebox-watchdog.sh: if the main browser for this
|
||||
# profile launched <2 min ago, the watchdog just relaunched it — skip the
|
||||
# kill path rather than racing it on a fresh cold start.
|
||||
if recent_relaunch "$cdp_port"; then
|
||||
echo "$(date -Iseconds) $agent: skipping kill (browser launched <2m ago, likely watchdog relaunch)" >> "$LOG"
|
||||
return 0
|
||||
fi
|
||||
|
||||
restart_browser "$agent" "$cdp_port"
|
||||
# 2026-10-04: post-restart re-check grace extended to ~60s total
|
||||
# (restart_browser sleeps 15s internally + 45s here), matching
|
||||
# chromebox-watchdog.sh's proven 60s retry window. Cold starts on a
|
||||
# loaded box (load ~7) need more than 20s before CDP/API respond.
|
||||
sleep 45
|
||||
if ! check_agent "$agent" "$agent" "$cdp_port"; then
|
||||
echo "$(date -Iseconds) $agent: CRITICAL - still down after restart" >> "$LOG"
|
||||
# TODO: Alert operator (e.g., via board post or email)
|
||||
else
|
||||
echo "$(date -Iseconds) $agent: RECOVERED after restart" >> "$LOG"
|
||||
fi
|
||||
}
|
||||
|
||||
# Every active node in the NODES.md registry gets checked — new nodes
|
||||
# propagate automatically, no per-node blocks to add.
|
||||
"$NETVM_BIN/netvm-registry.py" 2>/dev/null | while IFS=: read -r node port; do
|
||||
[ -n "$node" ] && [ -n "$port" ] && check_one "$node" "$port"
|
||||
done
|
||||
|
||||
# Trim log (keep last 1000 lines)
|
||||
tail -1000 "$LOG" > "$LOG.tmp" && mv "$LOG.tmp" "$LOG"
|
||||
Symlink
+1
@@ -0,0 +1 @@
|
||||
agent-cognitive-probe.py
|
||||
-563
@@ -1,563 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""agent_md.py — Access and modify Muse agent .md files via Hatch gateway and SSH.
|
||||
|
||||
Enables operators to inspect, audit, diff, and inject operational DRIVE into
|
||||
Muse agents across the fleet (muse, pip, 646, opm, def, dev).
|
||||
"""
|
||||
|
||||
import difflib
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import sys
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
# Ensure muse_cli can be imported
|
||||
sys.path.insert(0, os.path.expanduser("~/.local/lib/python3.14/site-packages"))
|
||||
try:
|
||||
from muse_cli.gateway import Gateway, load_cookies, AuthError, GatewayError
|
||||
except ImportError:
|
||||
Gateway = None
|
||||
|
||||
NETVM_ROOT = Path("/home/super/Projects/NetVM")
|
||||
SHARED_OPERATORS = NETVM_ROOT / "shared" / "operators"
|
||||
VALID_ACCOUNTS = ["muse", "pip", "646", "opm", "def", "dev"]
|
||||
|
||||
TARGET_MD_FILES = [
|
||||
"SOUL.md",
|
||||
"PROACTIVE_PREFERENCES.md",
|
||||
"HEARTBEAT.md",
|
||||
"AGENTS.md",
|
||||
"MEMORY.md",
|
||||
"USER.md",
|
||||
"TOOLS.md",
|
||||
"IDENTITY.md",
|
||||
]
|
||||
|
||||
# Tunnel / Port inventory
|
||||
TUNNEL_PORTS = {
|
||||
"muse-main": {"port": 2224, "terminal": 7681, "user": "muse"},
|
||||
"muse": {"port": 2225, "terminal": 7682, "user": "hatch"},
|
||||
"646": {"port": 2226, "terminal": 7683, "user": "hatch"},
|
||||
"pip": {"port": 2227, "terminal": 7684, "user": "hatch"},
|
||||
"opm": {"port": 2228, "terminal": 7685, "user": "hatch"},
|
||||
"def": {"port": 2229, "terminal": 7686, "user": "hatch"},
|
||||
"dev": {"port": 2230, "terminal": 7687, "user": "hatch"},
|
||||
}
|
||||
|
||||
|
||||
def get_gateway(account: str) -> "Gateway":
|
||||
"""Obtain an authenticated Gateway connection for an account."""
|
||||
if not Gateway:
|
||||
raise RuntimeError("muse_cli.gateway module is not available")
|
||||
conf_dir = Path.home() / ".config" / "muse-cli" / account
|
||||
cfile = conf_dir / "cookies.txt"
|
||||
if not cfile.exists():
|
||||
raise FileNotFoundError(f"No cookies found for account '{account}' at {cfile}")
|
||||
cookies = load_cookies(str(cfile))
|
||||
if not cookies.strip():
|
||||
raise ValueError(f"Cookies file for '{account}' is empty")
|
||||
return Gateway(cookies)
|
||||
|
||||
|
||||
def list_files(account: str, path: str = "") -> list:
|
||||
"""List files in the agent container filesystem via Hatch."""
|
||||
gw = get_gateway(account)
|
||||
res = gw.call_json("fs.list", body={"path": path})
|
||||
return res.get("entries", [])
|
||||
|
||||
|
||||
def read_md(account: str, filename: str, max_bytes: int = 200000) -> dict:
|
||||
"""Read a markdown file from the agent container via Hatch."""
|
||||
gw = get_gateway(account)
|
||||
offset = 0
|
||||
chunks = []
|
||||
chunk_len = min(65536, max_bytes)
|
||||
|
||||
while True:
|
||||
body = {"path": filename, "offset": offset, "len": chunk_len}
|
||||
res = gw.call_json("fs.read", body=body)
|
||||
text = res.get("text", "")
|
||||
if not text and res.get("data_base64"):
|
||||
import base64
|
||||
text = base64.b64decode(res["data_base64"]).decode("utf-8", "replace")
|
||||
chunks.append(text)
|
||||
offset += len(text.encode("utf-8"))
|
||||
if res.get("eof") or offset >= max_bytes or not text:
|
||||
break
|
||||
|
||||
full_text = "".join(chunks)
|
||||
return {
|
||||
"ok": True,
|
||||
"account": account,
|
||||
"filename": filename,
|
||||
"size": len(full_text.encode("utf-8")),
|
||||
"text": full_text,
|
||||
"eof": True,
|
||||
}
|
||||
|
||||
|
||||
def write_md(account: str, filename: str, text: str, overwrite: bool = True, append: bool = False) -> dict:
|
||||
"""Write content to a file in the agent container via Hatch."""
|
||||
gw = get_gateway(account)
|
||||
body = {
|
||||
"path": filename,
|
||||
"overwrite": overwrite,
|
||||
"append": append,
|
||||
"create_parent": True,
|
||||
"text": text,
|
||||
}
|
||||
res = gw.call_json("fs.write", body=body)
|
||||
return {
|
||||
"ok": True,
|
||||
"account": account,
|
||||
"filename": filename,
|
||||
"bytes_written": res.get("bytes_written", len(text.encode("utf-8"))),
|
||||
"path": res.get("path", f"/{filename}"),
|
||||
}
|
||||
|
||||
|
||||
def audit_agents(accounts: list = None) -> dict:
|
||||
"""Audit markdown files and operational DRIVE across fleet agents."""
|
||||
accounts = accounts or VALID_ACCOUNTS
|
||||
results = {}
|
||||
|
||||
for acct in accounts:
|
||||
acct_res = {
|
||||
"account": acct,
|
||||
"connected": False,
|
||||
"files": {},
|
||||
"drive_status": {},
|
||||
"issues": [],
|
||||
"drive_score": 0,
|
||||
}
|
||||
try:
|
||||
gw = get_gateway(acct)
|
||||
acct_res["connected"] = True
|
||||
acct_res["vm_id"] = gw.vm_id
|
||||
|
||||
entries = gw.call_json("fs.list", body={"path": ""}).get("entries", [])
|
||||
entry_map = {e["name"]: e for e in entries}
|
||||
|
||||
for tf in TARGET_MD_FILES:
|
||||
if tf in entry_map:
|
||||
info = entry_map[tf]
|
||||
acct_res["files"][tf] = {
|
||||
"exists": True,
|
||||
"size": info.get("size", 0),
|
||||
"modified": info.get("modifiedAt", ""),
|
||||
}
|
||||
else:
|
||||
acct_res["files"][tf] = {
|
||||
"exists": False,
|
||||
"size": 0,
|
||||
"modified": None,
|
||||
}
|
||||
|
||||
# Analyze DRIVE indicators
|
||||
# 1. HEARTBEAT.md checklist
|
||||
hb_info = acct_res["files"].get("HEARTBEAT.md", {})
|
||||
if not hb_info.get("exists"):
|
||||
acct_res["drive_status"]["heartbeat"] = "MISSING"
|
||||
acct_res["issues"].append("HEARTBEAT.md missing (no recurring checks)")
|
||||
elif hb_info.get("size", 0) <= 120:
|
||||
acct_res["drive_status"]["heartbeat"] = "EMPTY_CHECKLIST"
|
||||
acct_res["issues"].append("HEARTBEAT.md has empty checklist (background runner idle)")
|
||||
else:
|
||||
acct_res["drive_status"]["heartbeat"] = "ACTIVE"
|
||||
acct_res["drive_score"] += 25
|
||||
|
||||
# 2. PROACTIVE_PREFERENCES.md
|
||||
pro_info = acct_res["files"].get("PROACTIVE_PREFERENCES.md", {})
|
||||
if not pro_info.get("exists"):
|
||||
acct_res["drive_status"]["proactive"] = "MISSING"
|
||||
acct_res["issues"].append("PROACTIVE_PREFERENCES.md missing")
|
||||
elif pro_info.get("size", 0) <= 500:
|
||||
acct_res["drive_status"]["proactive"] = "BLANK_TEMPLATE"
|
||||
acct_res["issues"].append("PROACTIVE_PREFERENCES.md unconfigured (never reaches out)")
|
||||
else:
|
||||
acct_res["drive_status"]["proactive"] = "CONFIGURED"
|
||||
acct_res["drive_score"] += 25
|
||||
|
||||
# 3. SOUL.md
|
||||
soul_info = acct_res["files"].get("SOUL.md", {})
|
||||
if not soul_info.get("exists"):
|
||||
acct_res["drive_status"]["soul"] = "MISSING"
|
||||
acct_res["issues"].append("SOUL.md missing")
|
||||
elif soul_info.get("size", 0) <= 850:
|
||||
acct_res["drive_status"]["soul"] = "PASSIVE_STOCK"
|
||||
acct_res["issues"].append("SOUL.md is passive stock template (no operator drive)")
|
||||
else:
|
||||
acct_res["drive_status"]["soul"] = "OPERATOR_SOUL"
|
||||
acct_res["drive_score"] += 25
|
||||
|
||||
# 4. TOOLS.md & USER.md
|
||||
tools_info = acct_res["files"].get("TOOLS.md", {})
|
||||
user_info = acct_res["files"].get("USER.md", {})
|
||||
if tools_info.get("size", 0) > 400 and user_info.get("size", 0) > 400:
|
||||
acct_res["drive_status"]["context"] = "FULL_CONTEXT"
|
||||
acct_res["drive_score"] += 25
|
||||
else:
|
||||
acct_res["drive_status"]["context"] = "PARTIAL_OR_EMPTY"
|
||||
acct_res["issues"].append("TOOLS.md or USER.md missing operational conventions")
|
||||
|
||||
except Exception as e:
|
||||
acct_res["error"] = str(e)
|
||||
acct_res["issues"].append(f"Connection failed: {e}")
|
||||
|
||||
results[acct] = acct_res
|
||||
|
||||
return results
|
||||
|
||||
|
||||
def diff_md(account: str, filename: str) -> dict:
|
||||
"""Compare an agent's container file against the shared operator template."""
|
||||
local_path = SHARED_OPERATORS / filename
|
||||
if not local_path.exists():
|
||||
raise FileNotFoundError(f"Local template {local_path} not found")
|
||||
|
||||
local_content = local_path.read_text(encoding="utf-8")
|
||||
remote_data = read_md(account, filename)
|
||||
remote_content = remote_data.get("text", "")
|
||||
|
||||
diff = list(difflib.unified_diff(
|
||||
remote_content.splitlines(keepends=True),
|
||||
local_content.splitlines(keepends=True),
|
||||
fromfile=f"{account}:{filename} (remote)",
|
||||
tofile=f"shared/operators/{filename} (local)",
|
||||
))
|
||||
|
||||
return {
|
||||
"ok": True,
|
||||
"account": account,
|
||||
"filename": filename,
|
||||
"identical": len(diff) == 0,
|
||||
"diff": "".join(diff),
|
||||
"remote_size": len(remote_content.encode("utf-8")),
|
||||
"local_size": len(local_content.encode("utf-8")),
|
||||
}
|
||||
|
||||
|
||||
def amend_md(filename: str, content: str, author: str = "operator", reason: str = "") -> dict:
|
||||
"""Amend a centralized shared operator template in shared/operators/ with safety validation and git commit."""
|
||||
import subprocess
|
||||
|
||||
local_path = SHARED_OPERATORS / filename
|
||||
if not local_path.exists():
|
||||
raise FileNotFoundError(f"Shared operator file {filename} does not exist in {SHARED_OPERATORS}")
|
||||
|
||||
# Drive safety validation
|
||||
if filename == "HEARTBEAT.md":
|
||||
# Ensure checklist is not gutted
|
||||
non_comment_lines = [l for l in content.splitlines() if l.strip() and not l.strip().startswith("#")]
|
||||
checklist_items = [l for l in non_comment_lines if l.strip().startswith("- [")]
|
||||
if not checklist_items:
|
||||
raise ValueError("Safety rejection: amendment removes all active checklist items from HEARTBEAT.md")
|
||||
|
||||
elif filename == "PROACTIVE_PREFERENCES.md":
|
||||
if len(content.strip()) < 400:
|
||||
raise ValueError("Safety rejection: amendment would reduce PROACTIVE_PREFERENCES.md to unconfigured state")
|
||||
|
||||
elif filename == "SOUL.md":
|
||||
if "Be a guest in someone's life" in content and "AUTONOMOUS OPERATIONAL DRIVE" not in content:
|
||||
raise ValueError("Safety rejection: amendment reverts SOUL.md to passive stock template")
|
||||
|
||||
old_content = local_path.read_text(encoding="utf-8")
|
||||
local_path.write_text(content, encoding="utf-8")
|
||||
|
||||
# Git auto-commit if in git repo
|
||||
git_committed = False
|
||||
git_hash = None
|
||||
try:
|
||||
commit_msg = f"amend(operators): update {filename} via {author}"
|
||||
if reason:
|
||||
commit_msg += f" - {reason}"
|
||||
subprocess.run(["git", "add", str(local_path)], cwd=str(NETVM_ROOT), check=True, capture_output=True)
|
||||
cr = subprocess.run(["git", "commit", "-m", commit_msg], cwd=str(NETVM_ROOT), capture_output=True, text=True)
|
||||
if cr.returncode == 0:
|
||||
git_committed = True
|
||||
hr = subprocess.run(["git", "rev-parse", "--short", "HEAD"], cwd=str(NETVM_ROOT), capture_output=True, text=True)
|
||||
git_hash = hr.stdout.strip()
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
return {
|
||||
"ok": True,
|
||||
"action": "amend",
|
||||
"filename": filename,
|
||||
"author": author,
|
||||
"reason": reason,
|
||||
"bytes_written": len(content.encode("utf-8")),
|
||||
"git_committed": git_committed,
|
||||
"commit": git_hash,
|
||||
}
|
||||
|
||||
|
||||
def append_md(filename: str, text: str, author: str = "operator", section: str = None) -> dict:
|
||||
"""Safely append an amendment or lesson to a centralized shared template."""
|
||||
local_path = SHARED_OPERATORS / filename
|
||||
if not local_path.exists():
|
||||
raise FileNotFoundError(f"Shared operator file {filename} does not exist in {SHARED_OPERATORS}")
|
||||
|
||||
current = local_path.read_text(encoding="utf-8")
|
||||
header = f"\n\n<!-- Amendment by {author} on {time.strftime('%Y-%m-%d %H:%M:%S UTC', time.gmtime())} -->\n"
|
||||
if section:
|
||||
header += f"### {section}\n"
|
||||
new_content = current.rstrip() + header + text.strip() + "\n"
|
||||
|
||||
return amend_md(filename, new_content, author=author, reason=f"append {section or 'note'}")
|
||||
|
||||
|
||||
def pull_md(account: str, filename: str) -> dict:
|
||||
"""Pull the canonical centralized template from shared/operators/ into an agent's container."""
|
||||
local_path = SHARED_OPERATORS / filename
|
||||
if not local_path.exists():
|
||||
raise FileNotFoundError(f"Shared operator file {filename} does not exist in {SHARED_OPERATORS}")
|
||||
content = local_path.read_text(encoding="utf-8")
|
||||
res = write_md(account, filename, content, overwrite=True)
|
||||
return {
|
||||
"ok": True,
|
||||
"account": account,
|
||||
"filename": filename,
|
||||
"bytes_written": res.get("bytes_written"),
|
||||
"message": f"Successfully pulled canonical {filename} into {account} container",
|
||||
}
|
||||
|
||||
|
||||
def inject_drive(account: str, force: bool = False) -> dict:
|
||||
"""Inject high-drive operational instructions into the agent's container."""
|
||||
updates = []
|
||||
|
||||
# 1. SOUL.md
|
||||
soul_text = (SHARED_OPERATORS / "SOUL.md").read_text(encoding="utf-8")
|
||||
r_soul = write_md(account, "SOUL.md", soul_text, overwrite=True)
|
||||
updates.append({"file": "SOUL.md", "bytes": r_soul["bytes_written"]})
|
||||
|
||||
# 2. PROACTIVE_PREFERENCES.md
|
||||
pro_text = (SHARED_OPERATORS / "PROACTIVE_PREFERENCES.md").read_text(encoding="utf-8")
|
||||
r_pro = write_md(account, "PROACTIVE_PREFERENCES.md", pro_text, overwrite=True)
|
||||
updates.append({"file": "PROACTIVE_PREFERENCES.md", "bytes": r_pro["bytes_written"]})
|
||||
|
||||
# 3. HEARTBEAT.md
|
||||
hb_text = (SHARED_OPERATORS / "HEARTBEAT.md").read_text(encoding="utf-8")
|
||||
r_hb = write_md(account, "HEARTBEAT.md", hb_text, overwrite=True)
|
||||
updates.append({"file": "HEARTBEAT.md", "bytes": r_hb["bytes_written"]})
|
||||
|
||||
# 4. USER.md
|
||||
user_text = (SHARED_OPERATORS / "USER.md").read_text(encoding="utf-8")
|
||||
r_user = write_md(account, "USER.md", user_text, overwrite=True)
|
||||
updates.append({"file": "USER.md", "bytes": r_user["bytes_written"]})
|
||||
|
||||
# 5. TOOLS.md
|
||||
tools_text = (SHARED_OPERATORS / "TOOLS.md").read_text(encoding="utf-8")
|
||||
r_tools = write_md(account, "TOOLS.md", tools_text, overwrite=True)
|
||||
updates.append({"file": "TOOLS.md", "bytes": r_tools["bytes_written"]})
|
||||
|
||||
# 6. AGENTS.md (preserve existing custom lessons if present)
|
||||
agents_template = (SHARED_OPERATORS / "AGENTS.md").read_text(encoding="utf-8")
|
||||
try:
|
||||
remote_agents = read_md(account, "AGENTS.md").get("text", "")
|
||||
if "## Lessons" in remote_agents and len(remote_agents) > len(agents_template):
|
||||
# Extract custom lessons from remote and merge
|
||||
custom_lessons = remote_agents.split("## Lessons", 1)[1]
|
||||
merged_agents = agents_template.rstrip() + "\n\n## Lessons" + custom_lessons
|
||||
r_agents = write_md(account, "AGENTS.md", merged_agents, overwrite=True)
|
||||
else:
|
||||
r_agents = write_md(account, "AGENTS.md", agents_template, overwrite=True)
|
||||
except Exception:
|
||||
r_agents = write_md(account, "AGENTS.md", agents_template, overwrite=True)
|
||||
updates.append({"file": "AGENTS.md", "bytes": r_agents["bytes_written"]})
|
||||
|
||||
return {
|
||||
"ok": True,
|
||||
"account": account,
|
||||
"action": "inject_drive",
|
||||
"updates": updates,
|
||||
"message": f"Successfully injected high-drive operator files into {account} container",
|
||||
}
|
||||
|
||||
|
||||
def get_ssh_info(account: str = None) -> dict:
|
||||
"""Return SSH connection coordinates and reverse tunnel configuration."""
|
||||
jump_host = "34.139.37.135"
|
||||
if account:
|
||||
entry = TUNNEL_PORTS.get(account, {"port": 2226, "terminal": 7683, "user": "hatch"})
|
||||
port = entry["port"]
|
||||
user = entry["user"]
|
||||
cmd = f"ssh -o StrictHostKeyChecking=no -p {port} {user}@localhost"
|
||||
proxy_cmd = f"ssh -o StrictHostKeyChecking=no -J super@{jump_host} -p {port} {user}@localhost"
|
||||
return {
|
||||
"account": account,
|
||||
"jump_host": jump_host,
|
||||
"port": port,
|
||||
"container_user": user,
|
||||
"terminal_port": entry.get("terminal"),
|
||||
"direct_from_vm": cmd,
|
||||
"jump_command": proxy_cmd,
|
||||
"cat_example": f"cat file.md | {proxy_cmd} 'cat > /home/hatch/file.md'",
|
||||
}
|
||||
return {
|
||||
"jump_host": jump_host,
|
||||
"tunnels": TUNNEL_PORTS,
|
||||
}
|
||||
|
||||
|
||||
def main():
|
||||
import argparse
|
||||
parser = argparse.ArgumentParser(description="Manage Muse agent .md files via Hatch and SSH")
|
||||
sub = parser.add_subparsers(dest="cmd")
|
||||
|
||||
p_audit = sub.add_parser("audit", help="Audit .md files and DRIVE across all agents")
|
||||
p_audit.add_argument("accounts", nargs="*", help="Optional account filter")
|
||||
p_audit.add_argument("--json", action="store_true", help="Output JSON")
|
||||
|
||||
p_list = sub.add_parser("list", help="List container files via Hatch")
|
||||
p_list.add_argument("account", help="Agent account")
|
||||
p_list.add_argument("path", nargs="?", default="", help="Subdirectory path")
|
||||
|
||||
p_read = sub.add_parser("read", help="Read a markdown file via Hatch")
|
||||
p_read.add_argument("account", help="Agent account")
|
||||
p_read.add_argument("filename", help="Filename (e.g. SOUL.md)")
|
||||
|
||||
p_write = sub.add_parser("write", help="Write a markdown file via Hatch")
|
||||
p_write.add_argument("account", help="Agent account")
|
||||
p_write.add_argument("filename", help="Filename (e.g. SOUL.md)")
|
||||
p_write.add_argument("--content", help="Text content to write")
|
||||
p_write.add_argument("--file", help="Local file to copy content from")
|
||||
|
||||
p_diff = sub.add_parser("diff", help="Diff remote file against shared operator template")
|
||||
p_diff.add_argument("account", help="Agent account")
|
||||
p_diff.add_argument("filename", help="Filename (e.g. SOUL.md)")
|
||||
|
||||
p_amend = sub.add_parser("amend", help="Amend a shared operator file with validation and git commit")
|
||||
p_amend.add_argument("filename", help="Filename (e.g. AGENTS.md, TOOLS.md)")
|
||||
p_amend.add_argument("--content", help="New content")
|
||||
p_amend.add_argument("--file", help="File with new content")
|
||||
p_amend.add_argument("--author", default="operator", help="Author of amendment")
|
||||
p_amend.add_argument("--reason", default="", help="Reason for amendment")
|
||||
|
||||
p_append = sub.add_parser("append", help="Safely append a note or lesson to a shared operator file")
|
||||
p_append.add_argument("filename", help="Filename (e.g. AGENTS.md)")
|
||||
p_append.add_argument("text", help="Text to append")
|
||||
p_append.add_argument("--author", default="operator", help="Author of amendment")
|
||||
p_append.add_argument("--section", default=None, help="Optional section header")
|
||||
|
||||
p_pull = sub.add_parser("pull", help="Pull canonical shared file into an agent's container")
|
||||
p_pull.add_argument("account", help="Agent account")
|
||||
p_pull.add_argument("filename", help="Filename (e.g. SOUL.md)")
|
||||
|
||||
p_drive = sub.add_parser("inject-drive", help="Inject high-drive operator files into agent")
|
||||
p_drive.add_argument("account", help="Agent account")
|
||||
p_drive.add_argument("--force", action="store_true", help="Force overwrite")
|
||||
|
||||
p_sync_all = sub.add_parser("sync-all", help="Inject high-drive files across all active agents")
|
||||
|
||||
p_ssh = sub.add_parser("ssh-info", help="Get SSH tunnel dial-in information")
|
||||
p_ssh.add_argument("account", nargs="?", help="Optional agent account")
|
||||
|
||||
args = parser.parse_args()
|
||||
if not args.cmd:
|
||||
parser.print_help()
|
||||
sys.exit(1)
|
||||
|
||||
if args.cmd == "audit":
|
||||
res = audit_agents(args.accounts or None)
|
||||
if args.json:
|
||||
print(json.dumps(res, indent=2))
|
||||
else:
|
||||
print(f"\n{'='*70}\nMUSE AGENT .MD & DRIVE AUDIT REPORT\n{'='*70}")
|
||||
for acct, d in res.items():
|
||||
if not d.get("connected"):
|
||||
print(f"\n[AGENT {acct.upper()}] ✗ Connection failed: {d.get('error')}")
|
||||
continue
|
||||
score = d.get("drive_score", 0)
|
||||
status_color = "HIGH DRIVE" if score >= 75 else ("PARTIAL" if score >= 50 else "LOW DRIVE / STALE")
|
||||
print(f"\n[AGENT {acct.upper()}] DRIVE Score: {score}/100 ({status_color}) VM: {d.get('vm_id', 'unknown')}")
|
||||
for fname, finfo in d.get("files", {}).items():
|
||||
ex = "✓" if finfo.get("exists") else "✗"
|
||||
sz = f"{finfo.get('size', 0):6} bytes"
|
||||
mod = (finfo.get("modified") or "")[:19]
|
||||
print(f" {ex} {fname:24} {sz} {mod}")
|
||||
if d.get("issues"):
|
||||
print(" Issues:")
|
||||
for iss in d["issues"]:
|
||||
print(f" • {iss}")
|
||||
print(f"\n{'='*70}\n")
|
||||
|
||||
elif args.cmd == "list":
|
||||
entries = list_files(args.account, args.path)
|
||||
print(json.dumps(entries, indent=2))
|
||||
|
||||
elif args.cmd == "read":
|
||||
r = read_md(args.account, args.filename)
|
||||
print(r.get("text", ""))
|
||||
|
||||
elif args.cmd == "write":
|
||||
content = args.content
|
||||
if args.file:
|
||||
content = Path(args.file).read_text(encoding="utf-8")
|
||||
if content is None:
|
||||
print("Error: provide --content or --file", file=sys.stderr)
|
||||
sys.exit(2)
|
||||
res = write_md(args.account, args.filename, content)
|
||||
print(json.dumps(res, indent=2))
|
||||
|
||||
elif args.cmd == "diff":
|
||||
res = diff_md(args.account, args.filename)
|
||||
if res["identical"]:
|
||||
print(f"{args.account}:{args.filename} matches local shared/operators/{args.filename} exactly.")
|
||||
else:
|
||||
print(res["diff"])
|
||||
|
||||
elif args.cmd == "amend":
|
||||
content = args.content
|
||||
if args.file:
|
||||
content = Path(args.file).read_text(encoding="utf-8")
|
||||
if content is None:
|
||||
print("Error: provide --content or --file", file=sys.stderr)
|
||||
sys.exit(2)
|
||||
try:
|
||||
res = amend_md(args.filename, content, author=args.author, reason=args.reason)
|
||||
print(json.dumps(res, indent=2))
|
||||
except Exception as e:
|
||||
print(json.dumps({"ok": False, "error": str(e)}), indent=2)
|
||||
sys.exit(1)
|
||||
|
||||
elif args.cmd == "append":
|
||||
try:
|
||||
res = append_md(args.filename, args.text, author=args.author, section=args.section)
|
||||
print(json.dumps(res, indent=2))
|
||||
except Exception as e:
|
||||
print(json.dumps({"ok": False, "error": str(e)}), indent=2)
|
||||
sys.exit(1)
|
||||
|
||||
elif args.cmd == "pull":
|
||||
try:
|
||||
res = pull_md(args.account, args.filename)
|
||||
print(json.dumps(res, indent=2))
|
||||
except Exception as e:
|
||||
print(json.dumps({"ok": False, "error": str(e)}), indent=2)
|
||||
sys.exit(1)
|
||||
|
||||
elif args.cmd == "inject-drive":
|
||||
res = inject_drive(args.account, force=args.force)
|
||||
print(json.dumps(res, indent=2))
|
||||
|
||||
elif args.cmd == "sync-all":
|
||||
results = {}
|
||||
for acct in VALID_ACCOUNTS:
|
||||
try:
|
||||
results[acct] = inject_drive(acct)
|
||||
print(f"✓ Injected DRIVE into {acct}")
|
||||
except Exception as e:
|
||||
results[acct] = {"ok": False, "error": str(e)}
|
||||
print(f"✗ Failed {acct}: {e}")
|
||||
|
||||
elif args.cmd == "ssh-info":
|
||||
res = get_ssh_info(args.account)
|
||||
print(json.dumps(res, indent=2))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -1,512 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
approvals.py — Fleet approval detection, classification, and resolution engine.
|
||||
|
||||
Supports both:
|
||||
1. Browser DOM element detection & interaction via CDP (the primary live surface):
|
||||
- Confirmed selectors: [data-testid="approval-panel-header"],
|
||||
button[data-hatch-approval-primary-action="true"] ("Allow once"),
|
||||
and "Deny" / "Always allow this site" actions.
|
||||
- Text fallback: "Allow <agent> to share information with <IP>?"
|
||||
2. Auto-approval against TRUSTED_IPS (our infrastructure).
|
||||
3. Manual operator resolution (allow once, always allow, deny).
|
||||
4. Continuous watch & background integration into fleet status and loop health.
|
||||
"""
|
||||
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import sys
|
||||
import time
|
||||
import urllib.request
|
||||
from datetime import datetime, timezone
|
||||
from pathlib import Path
|
||||
|
||||
try:
|
||||
import websocket
|
||||
except ImportError:
|
||||
websocket = None
|
||||
|
||||
# Paths
|
||||
REPO_ROOT = Path("/home/super/Projects/NetVM")
|
||||
BIN_DIR = REPO_ROOT / "bin"
|
||||
CTL_LOG = REPO_ROOT / "box-ctl.jsonl"
|
||||
|
||||
# Add BIN_DIR to sys.path
|
||||
if str(BIN_DIR) not in sys.path:
|
||||
sys.path.insert(0, str(BIN_DIR))
|
||||
|
||||
try:
|
||||
import netvm_registry
|
||||
except ImportError:
|
||||
netvm_registry = None
|
||||
|
||||
VALID_NODES = ["muse", "pip", "646", "opm", "def", "dev"]
|
||||
|
||||
# Trusted infrastructure IPs safe for automated approval
|
||||
TRUSTED_IPS = {
|
||||
"34.139.37.135", # VM (gateway)
|
||||
"100.123.153.75", # bl (main compute)
|
||||
"100.81.31.9", # VM tailnet
|
||||
}
|
||||
|
||||
|
||||
def log_box_ctl(action: str, name: str = None, caller: str = "box-approvals", extra: dict = None):
|
||||
"""Log an audit event to box-ctl.jsonl."""
|
||||
try:
|
||||
rec = {
|
||||
"ts": datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ"),
|
||||
"action": action,
|
||||
"name": name,
|
||||
"caller": caller,
|
||||
}
|
||||
if extra:
|
||||
rec.update(extra)
|
||||
with open(CTL_LOG, "a") as f:
|
||||
f.write(json.dumps(rec) + "\n")
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
|
||||
def get_node_connection_info(node: str) -> dict:
|
||||
"""Return peer_ip, cdp_port, and netns for a given node."""
|
||||
if netvm_registry:
|
||||
nodes = netvm_registry.load()
|
||||
if node in nodes:
|
||||
rec = nodes[node]
|
||||
return {
|
||||
"node": node,
|
||||
"peer_ip": rec.get("peer_ip", f"10.201.87.2"),
|
||||
"cdp_port": rec.get("cdp_port", 9222),
|
||||
"netns": rec.get("netns", f"warp-{node}"),
|
||||
}
|
||||
# Deterministic fallback
|
||||
import hashlib
|
||||
tag = hashlib.sha256(node.encode()).hexdigest()[:8]
|
||||
idx = int(tag[:3], 16) % 200 + 10
|
||||
peer_ip = f"10.201.{idx}.2"
|
||||
pinned = {"muse": 9410, "pip": 9420, "646": 9430, "opm": 9440, "def": 9450, "dev": 9455}
|
||||
return {
|
||||
"node": node,
|
||||
"peer_ip": peer_ip,
|
||||
"cdp_port": pinned.get(node, 9222),
|
||||
"netns": f"warp-{node}",
|
||||
}
|
||||
|
||||
|
||||
def get_cdp_ws(node: str, timeout: float = 3.0):
|
||||
"""Connect to the node's active browser page over CDP WebSocket."""
|
||||
if websocket is None:
|
||||
raise RuntimeError("websocket-client library is required")
|
||||
|
||||
info = get_node_connection_info(node)
|
||||
peer_ip = info["peer_ip"]
|
||||
port = info["cdp_port"]
|
||||
|
||||
urls = [
|
||||
f"http://{peer_ip}:{port}/json/list",
|
||||
f"http://127.0.0.1:{port}/json/list",
|
||||
]
|
||||
tabs = None
|
||||
last_err = None
|
||||
for u in urls:
|
||||
try:
|
||||
req = urllib.request.Request(u, headers={"User-Agent": "box-approvals/1.0"})
|
||||
with urllib.request.urlopen(req, timeout=timeout) as r:
|
||||
tabs = json.load(r)
|
||||
break
|
||||
except Exception as e:
|
||||
last_err = e
|
||||
continue
|
||||
|
||||
if not tabs:
|
||||
raise ConnectionError(f"Could not reach CDP for {node}: {last_err}")
|
||||
|
||||
pages = [t for t in tabs if t.get("type") == "page"]
|
||||
if not pages:
|
||||
raise ConnectionError(f"No active page found for node {node}")
|
||||
|
||||
ws_url = pages[0].get("webSocketDebuggerUrl")
|
||||
if not ws_url:
|
||||
raise ConnectionError(f"No webSocketDebuggerUrl for node {node}")
|
||||
|
||||
ws = websocket.create_connection(ws_url, timeout=timeout)
|
||||
return ws, pages[0]
|
||||
|
||||
|
||||
def cdp_evaluate(ws, js_expr: str, await_promise: bool = False, timeout: float = 3.0):
|
||||
"""Evaluate a JavaScript expression via CDP Runtime.evaluate and return the result value."""
|
||||
req_id = int(time.time() * 1000) % 100000
|
||||
msg = {
|
||||
"id": req_id,
|
||||
"method": "Runtime.evaluate",
|
||||
"params": {
|
||||
"expression": js_expr,
|
||||
"returnByValue": True,
|
||||
"awaitPromise": await_promise,
|
||||
},
|
||||
}
|
||||
ws.send(json.dumps(msg))
|
||||
deadline = time.time() + timeout
|
||||
while time.time() < deadline:
|
||||
raw = ws.recv()
|
||||
resp = json.loads(raw)
|
||||
if resp.get("id") == req_id:
|
||||
res = resp.get("result", {})
|
||||
if "exceptionDetails" in res:
|
||||
return {"error": res["exceptionDetails"].get("text", "JS exception")}
|
||||
return res.get("result", {}).get("value")
|
||||
return None
|
||||
|
||||
|
||||
JS_INSPECT_APPROVALS = """(() => {
|
||||
// 1. Locate active approval panels
|
||||
const headers = Array.from(document.querySelectorAll('[data-testid="approval-panel-header"], [data-testid*="approval"]'));
|
||||
let activeCard = null;
|
||||
let cardText = '';
|
||||
|
||||
for (const h of headers) {
|
||||
let curr = h;
|
||||
for (let i = 0; i < 6 && curr && curr.parentElement && curr.parentElement !== document.body; i++) {
|
||||
const hasPrimary = !!curr.querySelector('button[data-hatch-approval-primary-action="true"]');
|
||||
const btns = Array.from(curr.querySelectorAll('button')).map(b => (b.innerText||'').trim().toLowerCase());
|
||||
const hasAllow = btns.some(t => t.includes('allow once') || t === 'allow');
|
||||
const hasDeny = btns.some(t => t === 'deny');
|
||||
if (hasPrimary || (hasAllow && hasDeny)) {
|
||||
activeCard = curr;
|
||||
cardText = curr.innerText || '';
|
||||
break;
|
||||
}
|
||||
curr = curr.parentElement;
|
||||
}
|
||||
if (activeCard) break;
|
||||
}
|
||||
|
||||
// Fallback: look for "Allow ... to share" or buttons
|
||||
if (!activeCard) {
|
||||
const bodyText = document.body ? document.body.innerText : '';
|
||||
if (bodyText.includes('Allow') && bodyText.includes('to share')) {
|
||||
const btns = Array.from(document.querySelectorAll('button'));
|
||||
const allowBtn = btns.find(b => (b.innerText||'').trim().toLowerCase().includes('allow once'));
|
||||
if (allowBtn) {
|
||||
let curr = allowBtn;
|
||||
for (let i = 0; i < 5 && curr && curr.parentElement && curr.parentElement !== document.body; i++) {
|
||||
if (curr.innerText && curr.innerText.includes('Allow') && curr.innerText.includes('to share')) {
|
||||
activeCard = curr;
|
||||
cardText = curr.innerText;
|
||||
break;
|
||||
}
|
||||
curr = curr.parentElement;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Inspect buttons in active card
|
||||
const buttons = [];
|
||||
let hasAllowOnce = false;
|
||||
let hasAlwaysAllow = false;
|
||||
let hasDeny = false;
|
||||
|
||||
if (activeCard) {
|
||||
const btns = Array.from(activeCard.querySelectorAll('button'));
|
||||
for (const b of btns) {
|
||||
const t = (b.innerText || '').trim();
|
||||
const low = t.toLowerCase();
|
||||
if (low) buttons.push(t);
|
||||
if (low === 'allow once' || b.getAttribute('data-hatch-approval-primary-action') === 'true') hasAllowOnce = true;
|
||||
if (low.includes('always allow')) hasAlwaysAllow = true;
|
||||
if (low === 'deny') hasDeny = true;
|
||||
}
|
||||
}
|
||||
|
||||
// Collect historical recent approval badges from chat stream
|
||||
const historyBadges = [];
|
||||
const allBtns = Array.from(document.querySelectorAll('button'));
|
||||
for (const b of allBtns) {
|
||||
const txt = (b.innerText || '').trim();
|
||||
if (txt.includes('Allowed once ·') || txt.includes('Timed out ·') || txt.includes('Site always allowed ·') || txt.includes('Allowed for this scheduled task ·')) {
|
||||
const lines = txt.split('\\n');
|
||||
const summary = lines[0] || '';
|
||||
const statusLine = lines[lines.length - 1] || '';
|
||||
historyBadges.push({ summary, status: statusLine });
|
||||
}
|
||||
}
|
||||
|
||||
return JSON.stringify({
|
||||
has_pending: !!activeCard && (hasAllowOnce || hasDeny),
|
||||
card_text: cardText.slice(0, 1000),
|
||||
buttons: buttons,
|
||||
has_allow_once: hasAllowOnce,
|
||||
has_always_allow: hasAlwaysAllow,
|
||||
has_deny: hasDeny,
|
||||
history: historyBadges.slice(0, 5)
|
||||
});
|
||||
})()"""
|
||||
|
||||
|
||||
def inspect_node_approvals(node: str) -> dict:
|
||||
"""Inspect a node for active browser approval prompts."""
|
||||
try:
|
||||
ws, page = get_cdp_ws(node, timeout=2.5)
|
||||
except Exception as e:
|
||||
return {
|
||||
"node": node,
|
||||
"status": "UNREACHABLE",
|
||||
"error": str(e),
|
||||
"has_pending": False,
|
||||
}
|
||||
|
||||
try:
|
||||
val_str = cdp_evaluate(ws, JS_INSPECT_APPROVALS, timeout=3.0)
|
||||
ws.close()
|
||||
if not val_str or not isinstance(val_str, str):
|
||||
return {
|
||||
"node": node,
|
||||
"status": "CLEAR",
|
||||
"has_pending": False,
|
||||
"page_title": page.get("title", ""),
|
||||
"page_url": page.get("url", ""),
|
||||
}
|
||||
|
||||
data = json.loads(val_str)
|
||||
has_pending = data.get("has_pending", False)
|
||||
card_text = data.get("card_text", "")
|
||||
|
||||
# Parse details
|
||||
ip = None
|
||||
m_ip = re.search(r"\b\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}\b", card_text)
|
||||
if m_ip:
|
||||
ip = m_ip.group(0)
|
||||
|
||||
# Extract title and purpose summary
|
||||
lines = [line.strip() for line in card_text.split("\n") if line.strip()]
|
||||
title = lines[0] if lines else "Permission request"
|
||||
purpose = lines[1] if len(lines) > 1 else ""
|
||||
|
||||
is_trusted = False
|
||||
if ip:
|
||||
is_trusted = ip in TRUSTED_IPS
|
||||
|
||||
return {
|
||||
"node": node,
|
||||
"status": "PENDING" if has_pending else "CLEAR",
|
||||
"has_pending": has_pending,
|
||||
"title": title,
|
||||
"purpose": purpose,
|
||||
"ip": ip,
|
||||
"is_trusted": is_trusted,
|
||||
"buttons": data.get("buttons", []),
|
||||
"has_allow_once": data.get("has_allow_once", False),
|
||||
"has_always_allow": data.get("has_always_allow", False),
|
||||
"has_deny": data.get("has_deny", False),
|
||||
"raw_text": card_text,
|
||||
"history": data.get("history", []),
|
||||
"page_title": page.get("title", ""),
|
||||
"page_url": page.get("url", ""),
|
||||
}
|
||||
except Exception as e:
|
||||
try:
|
||||
ws.close()
|
||||
except Exception:
|
||||
pass
|
||||
return {
|
||||
"node": node,
|
||||
"status": "ERROR",
|
||||
"error": str(e),
|
||||
"has_pending": False,
|
||||
}
|
||||
|
||||
|
||||
def check_fleet_approvals(nodes: list = None) -> list:
|
||||
"""Check approval status across the fleet."""
|
||||
target_nodes = nodes or VALID_NODES
|
||||
results = []
|
||||
for node in target_nodes:
|
||||
results.append(inspect_node_approvals(node))
|
||||
return results
|
||||
|
||||
|
||||
def allow_node_approval(node: str, always: bool = False, force: bool = False, caller: str = "box-approvals") -> dict:
|
||||
"""Approve a pending approval on a node (click 'Allow once' or 'Always allow this site')."""
|
||||
info = inspect_node_approvals(node)
|
||||
if not info.get("has_pending"):
|
||||
return {"ok": False, "node": node, "error": "No pending approval dialog found on node"}
|
||||
|
||||
if not info.get("is_trusted") and not force:
|
||||
target = info.get("ip") or "unrecognized target"
|
||||
return {
|
||||
"ok": False,
|
||||
"node": node,
|
||||
"error": f"Untrusted origin ({target}). Human review required. Use --force to override.",
|
||||
"approval": info,
|
||||
}
|
||||
|
||||
try:
|
||||
ws, _ = get_cdp_ws(node, timeout=3.0)
|
||||
except Exception as e:
|
||||
return {"ok": False, "node": node, "error": f"Failed to connect to CDP: {e}"}
|
||||
|
||||
try:
|
||||
if always:
|
||||
js_click = """(() => {
|
||||
const btns = Array.from(document.querySelectorAll('button'));
|
||||
const btn = btns.find(b => (b.innerText||'').toLowerCase().includes('always allow'));
|
||||
if (btn) {
|
||||
btn.click();
|
||||
return 'CLICKED_ALWAYS';
|
||||
}
|
||||
return 'NOT_FOUND';
|
||||
})()"""
|
||||
else:
|
||||
js_click = """(() => {
|
||||
const primary = document.querySelector('button[data-hatch-approval-primary-action="true"]');
|
||||
if (primary) {
|
||||
primary.click();
|
||||
return 'CLICKED_PRIMARY';
|
||||
}
|
||||
const btns = Array.from(document.querySelectorAll('button'));
|
||||
const btn = btns.find(b => {
|
||||
const t = (b.innerText||'').trim().toLowerCase();
|
||||
return t === 'allow once' || t === 'allow';
|
||||
});
|
||||
if (btn) {
|
||||
btn.click();
|
||||
return 'CLICKED_ALLOW';
|
||||
}
|
||||
return 'NOT_FOUND';
|
||||
})()"""
|
||||
|
||||
click_res = cdp_evaluate(ws, js_click, timeout=3.0)
|
||||
|
||||
# Verify dismissal
|
||||
time.sleep(0.8)
|
||||
js_verify = """(() => {
|
||||
const primary = document.querySelector('button[data-hatch-approval-primary-action="true"]');
|
||||
if (primary) return 'STILL_PRESENT';
|
||||
const headers = document.querySelectorAll('[data-testid="approval-panel-header"]');
|
||||
return headers.length === 0 ? 'DISMISSED' : 'STILL_PRESENT';
|
||||
})()"""
|
||||
verify_res = cdp_evaluate(ws, js_verify, timeout=2.0)
|
||||
ws.close()
|
||||
|
||||
dismissed = verify_res == "DISMISSED"
|
||||
mode = "always" if always else "allow_once"
|
||||
log_box_ctl(
|
||||
"approval-allow",
|
||||
name=node,
|
||||
caller=caller,
|
||||
extra={
|
||||
"decision": mode,
|
||||
"target_ip": info.get("ip"),
|
||||
"dismissed": dismissed,
|
||||
"forced": force,
|
||||
},
|
||||
)
|
||||
|
||||
return {
|
||||
"ok": True,
|
||||
"node": node,
|
||||
"decision": mode,
|
||||
"click_result": click_res,
|
||||
"dismissed": dismissed,
|
||||
"target_ip": info.get("ip"),
|
||||
"title": info.get("title"),
|
||||
}
|
||||
except Exception as e:
|
||||
try:
|
||||
ws.close()
|
||||
except Exception:
|
||||
pass
|
||||
return {"ok": False, "node": node, "error": str(e)}
|
||||
|
||||
|
||||
def deny_node_approval(node: str, caller: str = "box-approvals") -> dict:
|
||||
"""Deny a pending approval on a node (click 'Deny')."""
|
||||
info = inspect_node_approvals(node)
|
||||
if not info.get("has_pending"):
|
||||
return {"ok": False, "node": node, "error": "No pending approval dialog found on node"}
|
||||
|
||||
try:
|
||||
ws, _ = get_cdp_ws(node, timeout=3.0)
|
||||
except Exception as e:
|
||||
return {"ok": False, "node": node, "error": f"Failed to connect to CDP: {e}"}
|
||||
|
||||
try:
|
||||
js_deny = """(() => {
|
||||
const btns = Array.from(document.querySelectorAll('button'));
|
||||
const btn = btns.find(b => (b.innerText||'').trim().toLowerCase() === 'deny');
|
||||
if (btn) {
|
||||
btn.click();
|
||||
return 'CLICKED_DENY';
|
||||
}
|
||||
return 'NOT_FOUND';
|
||||
})()"""
|
||||
click_res = cdp_evaluate(ws, js_deny, timeout=3.0)
|
||||
|
||||
# Verify dismissal
|
||||
time.sleep(0.8)
|
||||
js_verify = """(() => {
|
||||
const headers = document.querySelectorAll('[data-testid="approval-panel-header"]');
|
||||
return headers.length === 0 ? 'DISMISSED' : 'STILL_PRESENT';
|
||||
})()"""
|
||||
verify_res = cdp_evaluate(ws, js_verify, timeout=2.0)
|
||||
ws.close()
|
||||
|
||||
dismissed = verify_res == "DISMISSED"
|
||||
log_box_ctl(
|
||||
"approval-deny",
|
||||
name=node,
|
||||
caller=caller,
|
||||
extra={
|
||||
"decision": "deny",
|
||||
"target_ip": info.get("ip"),
|
||||
"dismissed": dismissed,
|
||||
},
|
||||
)
|
||||
|
||||
return {
|
||||
"ok": True,
|
||||
"node": node,
|
||||
"decision": "deny",
|
||||
"click_result": click_res,
|
||||
"dismissed": dismissed,
|
||||
"target_ip": info.get("ip"),
|
||||
}
|
||||
except Exception as e:
|
||||
try:
|
||||
ws.close()
|
||||
except Exception:
|
||||
pass
|
||||
return {"ok": False, "node": node, "error": str(e)}
|
||||
|
||||
|
||||
def auto_approve_fleet(nodes: list = None, caller: str = "box-approvals") -> dict:
|
||||
"""Scan fleet nodes and automatically approve any requests to TRUSTED_IPS."""
|
||||
fleet = check_fleet_approvals(nodes)
|
||||
approved = []
|
||||
untrusted = []
|
||||
clear = []
|
||||
|
||||
for item in fleet:
|
||||
node = item["node"]
|
||||
if item.get("has_pending"):
|
||||
if item.get("is_trusted"):
|
||||
res = allow_node_approval(node, caller=caller)
|
||||
approved.append({
|
||||
"node": node,
|
||||
"target_ip": item.get("ip"),
|
||||
"title": item.get("title"),
|
||||
"res": res,
|
||||
})
|
||||
else:
|
||||
untrusted.append(item)
|
||||
else:
|
||||
clear.append(node)
|
||||
|
||||
return {
|
||||
"ok": True,
|
||||
"auto_approved": approved,
|
||||
"untrusted_pending": untrusted,
|
||||
"clear_nodes": clear,
|
||||
}
|
||||
@@ -1,333 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""box-chat-cdp.py — READ-ONLY CDP scraper for box-chat.py.
|
||||
|
||||
Runs INSIDE the agent's netns (via netvm-exec.sh), where the agent's
|
||||
headless Chromium CDP port is reachable on 127.0.0.1. Performs only
|
||||
Runtime.evaluate reads plus benign navigation clicks (panel open, thread
|
||||
switch, restore-to-main). Never sends messages, never touches the
|
||||
composer, never clicks send.
|
||||
|
||||
Usage:
|
||||
box-chat-cdp.py <agent> threads
|
||||
box-chat-cdp.py <agent> messages <thread-id>
|
||||
|
||||
Prints exactly one JSON object to stdout. Exit 0 on success, 1 on
|
||||
failure (stdout still carries {"ok": false, ...}).
|
||||
"""
|
||||
|
||||
import importlib.util
|
||||
import json
|
||||
import sys
|
||||
import time
|
||||
import urllib.request
|
||||
|
||||
|
||||
def load_accounts():
|
||||
path = "/home/super/Projects/NetVM/bin/netvm-registry.py"
|
||||
spec = importlib.util.spec_from_file_location("netvm_registry", path)
|
||||
mod = importlib.util.module_from_spec(spec)
|
||||
spec.loader.exec_module(mod)
|
||||
accounts = {}
|
||||
for node, rec in mod.load().items():
|
||||
accounts[node] = (node, "http://127.0.0.1:%d/json/list" % rec["cdp_port"])
|
||||
return accounts
|
||||
|
||||
|
||||
def connect(agent):
|
||||
accounts = load_accounts()
|
||||
if agent not in accounts:
|
||||
raise RuntimeError("unknown agent in registry: %s" % agent)
|
||||
_node, cdp_url = accounts[agent]
|
||||
with urllib.request.urlopen(cdp_url, timeout=10) as r:
|
||||
targets = json.load(r)
|
||||
pages = [t for t in targets if t.get("type") == "page"]
|
||||
if not pages:
|
||||
raise RuntimeError("no page target on CDP")
|
||||
import websocket
|
||||
ws = websocket.create_connection(
|
||||
pages[0]["webSocketDebuggerUrl"], timeout=60)
|
||||
return ws
|
||||
|
||||
|
||||
def ev(ws, expr, await_p=False):
|
||||
ws.send(json.dumps({
|
||||
"id": 1, "method": "Runtime.evaluate",
|
||||
"params": {"expression": expr, "returnByValue": True,
|
||||
"awaitPromise": await_p},
|
||||
}))
|
||||
# Drain CDP events until we get our command response (id 1).
|
||||
# The browser can emit events (Runtime.executionContextCreated, etc.)
|
||||
# at any time; taking the first recv() blindly returns None on a
|
||||
# busy page (observed as transient NO_SWITCHER / "unexpected
|
||||
# messages payload" on pip/opm 2026-10-04).
|
||||
resp = None
|
||||
drained = 0
|
||||
for _ in range(50):
|
||||
raw = ws.recv()
|
||||
resp = json.loads(raw)
|
||||
if resp.get("id") == 1:
|
||||
break
|
||||
drained += 1
|
||||
else:
|
||||
raise RuntimeError("CDP: no response to Runtime.evaluate (drained %d)" % drained)
|
||||
pass # drained count available in `drained` if needed
|
||||
res = resp.get("result", {})
|
||||
if res.get("subtype") == "error":
|
||||
raise RuntimeError("JS error: %s" % str(res.get("description"))[:200])
|
||||
return res.get("result", {}).get("value")
|
||||
|
||||
|
||||
TITLE_OF = """const titleOf = r => {
|
||||
const s = r.querySelector('span[title]');
|
||||
return s ? s.getAttribute('title').trim()
|
||||
: ((r.innerText||'').split('\\n')[0]||'').trim();
|
||||
};"""
|
||||
|
||||
ENSURE = """(async () => {
|
||||
const sleep = ms => new Promise(r => setTimeout(r, ms));
|
||||
%s
|
||||
if (window.location.pathname === '/thread/new') {
|
||||
window.location.href = '/'; await sleep(4000);
|
||||
}
|
||||
const nav = document.querySelector('[data-testid="hatch-nav-chat"]');
|
||||
if (!nav) return 'NO_CHAT_NAV';
|
||||
if (nav.getAttribute('aria-current') !== 'page') { nav.click(); await sleep(3000); }
|
||||
const panelOpen = () => !!document.querySelector('[data-testid="hatch-chat-compose"]');
|
||||
if (!panelOpen()) {
|
||||
// Retry: the switcher may not be rendered yet if the SPA is still
|
||||
// settling (observed transient NO_SWITCHER on pip/opm 2026-10-04).
|
||||
let sw = null;
|
||||
for (let k = 0; k < 4 && !sw; k++) {
|
||||
sw = document.querySelector('[data-testid="hatch-chat-switcher-trigger"]');
|
||||
if (!sw) await sleep(2000);
|
||||
}
|
||||
if (!sw) return 'NO_SWITCHER';
|
||||
sw.click(); await sleep(2500);
|
||||
if (!panelOpen()) return 'PANEL_CLOSED';
|
||||
}
|
||||
return 'OK';
|
||||
})()""" % TITLE_OF
|
||||
|
||||
|
||||
def op_threads(ws):
|
||||
st = ev(ws, ENSURE, await_p=True)
|
||||
if st != "OK":
|
||||
raise RuntimeError("could not reach chat panel: %s" % st)
|
||||
js = """(async () => {
|
||||
const sleep = ms => new Promise(r => setTimeout(r, ms));
|
||||
%s
|
||||
const snap = [...document.querySelectorAll('[data-testid="hatch-thread-row"]')]
|
||||
.map(r => {
|
||||
const spans = [...r.querySelectorAll('span')];
|
||||
return {title: titleOf(r),
|
||||
rel: spans.length ? (spans[spans.length-1].innerText||'').trim() : ''};
|
||||
});
|
||||
const ensurePanel = async () => {
|
||||
if (document.querySelector('[data-testid="hatch-chat-compose"]')) return true;
|
||||
for (let k = 0; k < 3; k++) {
|
||||
const sw = document.querySelector('[data-testid="hatch-chat-switcher-trigger"]');
|
||||
if (!sw) return false;
|
||||
sw.click(); await sleep(3000);
|
||||
if (document.querySelector('[data-testid="hatch-chat-compose"]')) return true;
|
||||
}
|
||||
return false;
|
||||
};
|
||||
const out = [];
|
||||
for (const s of snap) {
|
||||
const isMain = /^main chat$/i.test(s.title);
|
||||
if (isMain) { out.push({id: 'main', kind: 'main', title: s.title, rel: s.rel}); continue; }
|
||||
const panelOk = await ensurePanel();
|
||||
if (!panelOk) { out.push({id: null, kind: 'sidechat', title: s.title, rel: s.rel, error: 'PANEL_CLOSED'}); continue; }
|
||||
const row = [...document.querySelectorAll('[data-testid="hatch-thread-row"]')]
|
||||
.find(r => titleOf(r) === s.title);
|
||||
if (!row) { out.push({id: null, kind: 'sidechat', title: s.title, rel: s.rel, error: 'ROW_GONE'}); continue; }
|
||||
row.click();
|
||||
let id = null;
|
||||
for (let t = 0; t < 20; t++) {
|
||||
await sleep(500);
|
||||
const m = window.location.href.match(/\\/thread\\/([0-9a-fA-F-]{36})/);
|
||||
if (m) { id = m[1]; break; }
|
||||
}
|
||||
if (!id) {
|
||||
const row2 = [...document.querySelectorAll('[data-testid="hatch-thread-row"]')]
|
||||
.find(r => titleOf(r) === s.title);
|
||||
if (row2) {
|
||||
row2.click();
|
||||
for (let t = 0; t < 12; t++) {
|
||||
await sleep(500);
|
||||
const m = window.location.href.match(/\\/thread\\/([0-9a-fA-F-]{36})/);
|
||||
if (m) { id = m[1]; break; }
|
||||
}
|
||||
}
|
||||
}
|
||||
out.push({id, kind: 'sidechat', title: s.title, rel: s.rel});
|
||||
}
|
||||
return {threads: out};
|
||||
})()""" % TITLE_OF
|
||||
return ev(ws, js, await_p=True)
|
||||
|
||||
|
||||
def op_messages(ws, thread_id):
|
||||
st = ev(ws, ENSURE, await_p=True)
|
||||
if st != "OK":
|
||||
raise RuntimeError("could not reach chat panel: %s" % st)
|
||||
js = """(async () => {
|
||||
const sleep = ms => new Promise(r => setTimeout(r, ms));
|
||||
%s
|
||||
const THREAD = %s;
|
||||
const ensurePanel = async () => {
|
||||
if (document.querySelector('[data-testid="hatch-chat-compose"]')) return true;
|
||||
for (let k = 0; k < 3; k++) {
|
||||
const sw = document.querySelector('[data-testid="hatch-chat-switcher-trigger"]');
|
||||
if (!sw) return false;
|
||||
sw.click(); await sleep(3000);
|
||||
if (document.querySelector('[data-testid="hatch-chat-compose"]')) return true;
|
||||
}
|
||||
return false;
|
||||
};
|
||||
const uuidOf = () => {
|
||||
const m = window.location.href.match(/\\/thread\\/([0-9a-fA-F-]{36})/);
|
||||
return m ? m[1].toLowerCase() : null;
|
||||
};
|
||||
let landed = false;
|
||||
if (THREAD === 'main') {
|
||||
await ensurePanel();
|
||||
const row = [...document.querySelectorAll('[data-testid="hatch-thread-row"]')]
|
||||
.find(r => /^main chat$/i.test(titleOf(r)));
|
||||
if (row) { row.click(); await sleep(2500); }
|
||||
landed = /muse\\.ai\\/?$/.test(window.location.href) && !uuidOf();
|
||||
} else {
|
||||
// already there?
|
||||
if (uuidOf() === THREAD.toLowerCase()) landed = true;
|
||||
// click rows until the URL carries our thread uuid (SPA navigation,
|
||||
// keeps the JS context alive unlike location.href assignment)
|
||||
for (let i = 0; i < 12 && !landed; i++) {
|
||||
await ensurePanel();
|
||||
const rows = [...document.querySelectorAll('[data-testid="hatch-thread-row"]')];
|
||||
if (!rows.length) break;
|
||||
const row = rows[i %% rows.length];
|
||||
row.click();
|
||||
for (let t = 0; t < 14; t++) {
|
||||
await sleep(500);
|
||||
if (uuidOf() === THREAD.toLowerCase()) { landed = true; break; }
|
||||
if (uuidOf()) break; // navigated somewhere else; try next row
|
||||
}
|
||||
}
|
||||
if (!landed) {
|
||||
// fallback: SPA history navigation (no full page load)
|
||||
window.history.pushState({}, '', '/thread/' + THREAD);
|
||||
window.dispatchEvent(new PopStateEvent('popstate'));
|
||||
await sleep(5000);
|
||||
landed = uuidOf() === THREAD.toLowerCase();
|
||||
}
|
||||
}
|
||||
if (!landed) return {error: 'THREAD_NOT_FOUND'};
|
||||
await sleep(2500);
|
||||
const sc = document.getElementById('hatch-chat-scroll');
|
||||
if (sc) {
|
||||
for (let i = 0; i < 3; i++) { sc.scrollTop = 0; await sleep(1500); }
|
||||
sc.scrollTop = sc.scrollHeight; await sleep(800);
|
||||
}
|
||||
const els = [...document.querySelectorAll('[data-message-id]')];
|
||||
const messages = els.map(m => {
|
||||
const id = m.getAttribute('data-message-id');
|
||||
const ps = [...m.querySelectorAll('p')].map(p => (p.innerText||'').trim()).filter(Boolean);
|
||||
let text = ps.join('\\n');
|
||||
if (!text) text = (m.innerText||'').replace(/^(Assistant message:|User message:)\\s*/, '').trim();
|
||||
const t = m.querySelector('time');
|
||||
return {id,
|
||||
author: id.indexOf('assistant-msg') === 0 ? 'assistant' : 'user',
|
||||
text,
|
||||
ts: t ? (t.getAttribute('datetime') || t.innerText || null) : null};
|
||||
});
|
||||
return {url: window.location.href, messages};
|
||||
})()""" % (TITLE_OF, json.dumps(thread_id))
|
||||
data = ev(ws, js, await_p=True)
|
||||
if not isinstance(data, dict) or "messages" not in data:
|
||||
if isinstance(data, dict) and data.get("error") == "THREAD_NOT_FOUND":
|
||||
raise RuntimeError("THREAD_NOT_FOUND: no such thread for this agent")
|
||||
raise RuntimeError("unexpected messages payload")
|
||||
return data
|
||||
|
||||
|
||||
def restore_main(ws):
|
||||
try:
|
||||
ev(ws, """(() => {
|
||||
%s
|
||||
const row = [...document.querySelectorAll('[data-testid="hatch-thread-row"]')]
|
||||
.find(r => /^main chat$/i.test(titleOf(r)));
|
||||
if (row) row.click();
|
||||
return 'ok';
|
||||
})()""" % TITLE_OF)
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
|
||||
def main(argv):
|
||||
if len(argv) < 3:
|
||||
print(json.dumps({"ok": False, "code": "BAD_ARGS",
|
||||
"error": "usage: box-chat-cdp.py <agent> threads|messages [thread-id]"}))
|
||||
return 1
|
||||
agent, op = argv[1], argv[2]
|
||||
ws = None
|
||||
try:
|
||||
# CDP evaluate can resolve null when the page is mid-navigation
|
||||
# (agent actively using browser). Reconnect fresh on each retry
|
||||
# so we attach to the current page, not a stale JS context.
|
||||
# Observed 2026-10-04: opm's browser navigates during reads.
|
||||
data, last_err = None, None
|
||||
for attempt in range(3):
|
||||
try:
|
||||
if ws is not None:
|
||||
try:
|
||||
ws.close()
|
||||
except Exception:
|
||||
pass
|
||||
ws = connect(agent)
|
||||
if op == "threads":
|
||||
data = op_threads(ws)
|
||||
ok = isinstance(data, dict) and isinstance(
|
||||
data.get("threads"), list)
|
||||
elif op == "messages":
|
||||
if len(argv) < 4:
|
||||
raise RuntimeError("messages requires thread-id")
|
||||
data = op_messages(ws, argv[3])
|
||||
ok = isinstance(data, dict) and isinstance(
|
||||
data.get("messages"), list)
|
||||
else:
|
||||
raise RuntimeError("unknown op: %s" % op)
|
||||
if ok:
|
||||
break
|
||||
last_err = "empty CDP result"
|
||||
data = None
|
||||
except RuntimeError as e:
|
||||
last_err = str(e)
|
||||
data = None
|
||||
time.sleep(3)
|
||||
if data is None:
|
||||
raise RuntimeError(last_err or "CDP returned no usable data")
|
||||
if op == "threads":
|
||||
print(json.dumps({"ok": True, "agent": agent,
|
||||
"threads": data["threads"]}))
|
||||
else:
|
||||
print(json.dumps({"ok": True, "agent": agent, "url": data["url"],
|
||||
"messages": data["messages"]}))
|
||||
return 0
|
||||
except Exception as e:
|
||||
print(json.dumps({"ok": False, "code": "CDP_ERROR",
|
||||
"error": str(e)[:300]}))
|
||||
return 1
|
||||
finally:
|
||||
if ws is not None:
|
||||
try:
|
||||
restore_main(ws)
|
||||
except Exception:
|
||||
pass
|
||||
try:
|
||||
ws.close()
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main(sys.argv))
|
||||
-380
@@ -1,380 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""box-chat.py — allowlisted bl helper for Box thread oversight (READ-ONLY).
|
||||
|
||||
The board server (VM) never scrapes browsers directly. All thread reads go
|
||||
through this helper, invoked as:
|
||||
|
||||
/home/super/Projects/NetVM/bin/box-chat.py thread-list <agent> [--kind K] [--limit N]
|
||||
/home/super/Projects/NetVM/bin/box-chat.py thread-messages <agent> <thread-id> [--limit N] [--before MSGID]
|
||||
|
||||
Security properties (mirrors box-ctl.py):
|
||||
- Fixed verb set; every argument validated before acting.
|
||||
- <agent> must be a known node (muse, pip, 646, opm); anything else exits
|
||||
before any netns/SSH/CDP work.
|
||||
- <thread-id> must match ^[a-zA-Z0-9-]{1,64}$ or be the literal "main".
|
||||
- READ-ONLY by construction: the CDP driver (box-chat-cdp.py) only runs
|
||||
Runtime.evaluate reads plus benign navigation clicks. No sends, no
|
||||
composer interaction, no shell=True anywhere. All subprocess calls use
|
||||
argv lists.
|
||||
- Every invocation audit-logged to box-chat.jsonl with caller identity.
|
||||
|
||||
Output: JSON to stdout ({"ok": true, ...} or {"ok": false, ...}),
|
||||
exit 0 on success, nonzero on failure.
|
||||
"""
|
||||
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import subprocess
|
||||
import sys
|
||||
from datetime import datetime, timedelta, timezone
|
||||
from pathlib import Path
|
||||
|
||||
NETVM_ROOT = Path("/home/super/Projects/NetVM")
|
||||
BIN = NETVM_ROOT / "bin"
|
||||
CDP_DRIVER = BIN / "box-chat-cdp.py"
|
||||
NETVM_EXEC = BIN / "netvm-exec.sh"
|
||||
CHAT_LOG = NETVM_ROOT / "box-chat.jsonl"
|
||||
DM_LOG = NETVM_ROOT / "dm-log.jsonl"
|
||||
|
||||
VALID_AGENTS = {"muse", "pip", "646", "opm"}
|
||||
VALID_KINDS = {"main", "sidechat", "dm", "all"}
|
||||
AGENT_RE = re.compile(r"^[a-z0-9-]{1,64}$")
|
||||
THREAD_RE = re.compile(r"^[a-zA-Z0-9-]{1,64}$")
|
||||
MSGID_RE = re.compile(r"^[a-zA-Z0-9-]{1,128}$")
|
||||
|
||||
REL_MONTHS = {m: i + 1 for i, m in enumerate(
|
||||
["jan", "feb", "mar", "apr", "may", "jun",
|
||||
"jul", "aug", "sep", "oct", "nov", "dec"])}
|
||||
|
||||
|
||||
def utcnow():
|
||||
return datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")
|
||||
|
||||
|
||||
def out(ok, **kw):
|
||||
payload = {"ok": ok}
|
||||
payload.update(kw)
|
||||
print(json.dumps(payload))
|
||||
|
||||
|
||||
def fail(code, error, detail=None, exit_code=1):
|
||||
payload = {"ok": False, "code": code, "error": error}
|
||||
if detail is not None:
|
||||
payload["detail"] = detail
|
||||
print(json.dumps(payload))
|
||||
sys.exit(exit_code)
|
||||
|
||||
|
||||
def audit(action, agent=None, name=None, kind=None):
|
||||
"""Append invocation record to the bl-side log."""
|
||||
try:
|
||||
entry = {
|
||||
"ts": utcnow(),
|
||||
"action": action,
|
||||
"agent": agent,
|
||||
"name": name,
|
||||
"kind": kind,
|
||||
"caller": os.environ.get("BOX_CALLER", "unknown"),
|
||||
}
|
||||
with open(CHAT_LOG, "a") as f:
|
||||
f.write(json.dumps(entry) + "\n")
|
||||
except Exception:
|
||||
pass # audit failure must not break the action
|
||||
|
||||
|
||||
def check_agent(agent):
|
||||
if not agent or agent not in VALID_AGENTS:
|
||||
fail("BAD_AGENT", "agent must be one of %s" % sorted(VALID_AGENTS),
|
||||
{"field": "agent", "value": agent})
|
||||
return agent
|
||||
|
||||
|
||||
def check_thread_id(tid):
|
||||
if tid != "main" and not (tid and THREAD_RE.match(tid)):
|
||||
fail("BAD_THREAD", "thread-id must be 'main' or match ^[a-zA-Z0-9-]{1,64}$",
|
||||
{"field": "thread-id", "value": tid})
|
||||
return tid
|
||||
|
||||
|
||||
def check_limit(val, default):
|
||||
if val is None:
|
||||
return default
|
||||
try:
|
||||
n = int(val)
|
||||
except (TypeError, ValueError):
|
||||
fail("BAD_LIMIT", "limit must be an integer 1-200", {"value": val})
|
||||
if not 1 <= n <= 200:
|
||||
fail("BAD_LIMIT", "limit must be an integer 1-200", {"value": val})
|
||||
return n
|
||||
|
||||
|
||||
def run_cdp(agent, op, args, timeout):
|
||||
"""Run the CDP driver inside the agent's netns. argv only, no shell."""
|
||||
cmd = [str(NETVM_EXEC), agent, "--", sys.executable,
|
||||
str(CDP_DRIVER), agent, op] + args
|
||||
try:
|
||||
r = subprocess.run(cmd, capture_output=True, text=True,
|
||||
timeout=timeout)
|
||||
except subprocess.TimeoutExpired:
|
||||
fail("CDP_TIMEOUT", "CDP driver timed out in netns",
|
||||
{"agent": agent, "op": op})
|
||||
if r.returncode != 0 and not r.stdout.strip():
|
||||
fail("CDP_ERROR", "CDP driver failed",
|
||||
{"stderr": (r.stderr or "")[-300:]})
|
||||
try:
|
||||
data = json.loads(r.stdout.strip())
|
||||
except Exception:
|
||||
fail("CDP_ERROR", "CDP driver returned non-JSON",
|
||||
{"stdout": r.stdout[-300:], "stderr": (r.stderr or "")[-300:]})
|
||||
if not data.get("ok"):
|
||||
code = data.get("code", "CDP_ERROR")
|
||||
if "THREAD_NOT_FOUND" in str(data.get("error", "")):
|
||||
code = "THREAD_NOT_FOUND"
|
||||
fail(code, data.get("error", "cdp failed"))
|
||||
return data
|
||||
|
||||
|
||||
def parse_rel_time(rel):
|
||||
"""Panel relative times ('just now', '4m', '2h', '3d', 'Oct 3') ->
|
||||
(approx ISO8601, True). Returns (None, False) when unparseable."""
|
||||
now = datetime.now(timezone.utc)
|
||||
rel = (rel or "").strip().lower()
|
||||
if rel in ("just now", "now"):
|
||||
return now.strftime("%Y-%m-%dT%H:%M:%SZ"), True
|
||||
m = re.match(r"^(\d+)\s*m(in(ute)?s?)?$", rel)
|
||||
if m:
|
||||
return (now - timedelta(minutes=int(m.group(1)))).strftime("%Y-%m-%dT%H:%M:%SZ"), True
|
||||
m = re.match(r"^(\d+)\s*h((ou)?rs?)?$", rel)
|
||||
if m:
|
||||
return (now - timedelta(hours=int(m.group(1)))).strftime("%Y-%m-%dT%H:%M:%SZ"), True
|
||||
m = re.match(r"^(\d+)\s*d(ays?)?$", rel)
|
||||
if m:
|
||||
return (now - timedelta(days=int(m.group(1)))).strftime("%Y-%m-%dT%H:%M:%SZ"), True
|
||||
m = re.match(r"^([a-z]{3})\s+(\d{1,2})$", rel)
|
||||
if m and m.group(1) in REL_MONTHS:
|
||||
dt = now.replace(month=REL_MONTHS[m.group(1)], day=int(m.group(2)),
|
||||
hour=12, minute=0, second=0, microsecond=0)
|
||||
if dt > now:
|
||||
dt = dt.replace(year=dt.year - 1)
|
||||
return dt.strftime("%Y-%m-%dT%H:%M:%SZ"), True
|
||||
return None, False
|
||||
|
||||
|
||||
def dm_conversations(agent):
|
||||
"""DM conversations involving <agent>, synthesized from dm-log.jsonl.
|
||||
|
||||
Real timestamps and counts; bodies are not stored in the log (by design)
|
||||
— message text for these threads is the wire tag, and full bodies live
|
||||
in the target chat's messages.
|
||||
"""
|
||||
convos = {}
|
||||
try:
|
||||
with open(DM_LOG) as f:
|
||||
for line in f:
|
||||
line = line.strip()
|
||||
if not line:
|
||||
continue
|
||||
try:
|
||||
e = json.loads(line)
|
||||
except Exception:
|
||||
continue
|
||||
if e.get("type") not in ("sent", "send_done"):
|
||||
continue
|
||||
frm, to = e.get("agent"), e.get("to")
|
||||
if not frm or not to:
|
||||
continue
|
||||
if agent not in (frm, to):
|
||||
continue
|
||||
key = tuple(sorted([frm, to]))
|
||||
c = convos.setdefault(key, {"count": 0, "last_ts": "",
|
||||
"entries": []})
|
||||
c["count"] += 1
|
||||
if e.get("ts", "") > c["last_ts"]:
|
||||
c["last_ts"] = e["ts"]
|
||||
c["entries"].append(e)
|
||||
except FileNotFoundError:
|
||||
return []
|
||||
out = []
|
||||
for (a, b), c in sorted(convos.items()):
|
||||
out.append({
|
||||
"id": "dm-%s-%s" % (a, b),
|
||||
"kind": "dm",
|
||||
"title": "dm:%s:%s" % (a, b),
|
||||
"participants": [a, b],
|
||||
"last_message_at": c["last_ts"] or None,
|
||||
"last_message_approx": False,
|
||||
"message_count": c["count"],
|
||||
})
|
||||
return out
|
||||
|
||||
|
||||
def dm_thread_messages(agent, thread_id):
|
||||
"""Messages for a dm-<a>-<b> thread, from dm-log.jsonl (complete log)."""
|
||||
parts = thread_id[3:].split("-")
|
||||
if len(parts) != 2:
|
||||
fail("THREAD_NOT_FOUND", "no such DM thread: %s" % thread_id)
|
||||
a, b = parts
|
||||
if agent not in (a, b):
|
||||
fail("THREAD_NOT_FOUND", "no such DM thread: %s" % thread_id)
|
||||
msgs = []
|
||||
try:
|
||||
with open(DM_LOG) as f:
|
||||
for line in f:
|
||||
line = line.strip()
|
||||
if not line:
|
||||
continue
|
||||
try:
|
||||
e = json.loads(line)
|
||||
except Exception:
|
||||
continue
|
||||
if e.get("type") not in ("sent", "send_done"):
|
||||
continue
|
||||
frm, to = e.get("agent"), e.get("to")
|
||||
if not frm or not to:
|
||||
continue
|
||||
if tuple(sorted([frm, to])) != (a, b):
|
||||
continue
|
||||
sender = e.get("agent")
|
||||
msgs.append({
|
||||
"id": str(e.get("id", "")),
|
||||
"from": {"role": "agent", "name": sender},
|
||||
"text": "[from:%s] [id:%s] -> %s" % (
|
||||
sender, e.get("id"), e.get("target", "?")),
|
||||
"ts": e.get("ts"),
|
||||
"target": e.get("target"),
|
||||
"verified": e.get("verified"),
|
||||
})
|
||||
except FileNotFoundError:
|
||||
pass
|
||||
msgs.sort(key=lambda m: m.get("ts") or "")
|
||||
# de-dupe send_done/sent pairs on DM id, keep the richer record
|
||||
seen = {}
|
||||
for m in msgs:
|
||||
prev = seen.get(m["id"])
|
||||
if prev is None or (m.get("verified") and not prev.get("verified")):
|
||||
seen[m["id"]] = m
|
||||
msgs = sorted(seen.values(), key=lambda m: m.get("ts") or "")
|
||||
return msgs
|
||||
|
||||
|
||||
def act_thread_list(agent, kind, limit):
|
||||
threads = []
|
||||
if kind in ("main", "sidechat", "all"):
|
||||
data = run_cdp(agent, "threads", [], timeout=240)
|
||||
for t in data.get("threads", [])[:limit]:
|
||||
if kind != "all" and t.get("kind") != kind:
|
||||
continue
|
||||
ts, approx = parse_rel_time(t.get("rel", ""))
|
||||
threads.append({
|
||||
"id": t.get("id"),
|
||||
"kind": t.get("kind"),
|
||||
"title": t.get("title"),
|
||||
"participants": [agent, "human"],
|
||||
"last_message_at": ts,
|
||||
"last_message_approx": approx,
|
||||
"message_count": None, # list is a panel scrape; counts need a thread open
|
||||
})
|
||||
if kind in ("dm", "all"):
|
||||
threads.extend(dm_conversations(agent))
|
||||
audit("thread-list", agent=agent, kind=kind)
|
||||
out(True, agent=agent, kind=kind, threads=threads,
|
||||
fetched_at=utcnow())
|
||||
|
||||
|
||||
def act_thread_messages(agent, thread_id, limit, before):
|
||||
if thread_id.startswith("dm-"):
|
||||
msgs = dm_thread_messages(agent, thread_id)
|
||||
thread = {"id": thread_id, "kind": "dm",
|
||||
"title": "dm:%s" % thread_id[3:].replace("-", ":"),
|
||||
"participants": thread_id[3:].split("-")}
|
||||
else:
|
||||
data = run_cdp(agent, "messages", [thread_id], timeout=150)
|
||||
raw = data.get("messages", [])
|
||||
thread = {"id": thread_id,
|
||||
"kind": "main" if thread_id == "main" else "sidechat",
|
||||
"participants": [agent, "human"]}
|
||||
msgs = [{
|
||||
"id": m.get("id"),
|
||||
"from": ({"role": "agent", "name": agent}
|
||||
if m.get("author") == "assistant"
|
||||
else {"role": "human", "name": "human"}),
|
||||
"text": m.get("text", ""),
|
||||
"ts": m.get("ts"),
|
||||
} for m in raw]
|
||||
if before:
|
||||
if not MSGID_RE.match(before):
|
||||
fail("BAD_CURSOR", "before must match ^[a-zA-Z0-9-]{1,128}$",
|
||||
{"value": before})
|
||||
idx = next((i for i, m in enumerate(msgs) if m["id"] == before), None)
|
||||
if idx is None:
|
||||
fail("BAD_CURSOR", "no message with that id in loaded window",
|
||||
{"value": before})
|
||||
msgs = msgs[:idx]
|
||||
older = len(msgs)
|
||||
if len(msgs) > limit:
|
||||
msgs = msgs[-limit:]
|
||||
next_before = msgs[0]["id"] if older > len(msgs) and msgs else None
|
||||
audit("thread-messages", agent=agent, name=thread_id)
|
||||
out(True, agent=agent, thread=thread, messages=msgs,
|
||||
next_before=next_before, loaded_count=older, fetched_at=utcnow())
|
||||
|
||||
|
||||
USAGE = """usage: box-chat.py <action> [args]
|
||||
|
||||
thread-list <agent> [--kind main|sidechat|dm|all] [--limit N]
|
||||
thread-messages <agent> <thread-id> [--limit N] [--before MSGID]
|
||||
|
||||
read-only. <agent> is one of: muse, pip, 646, opm."""
|
||||
|
||||
|
||||
def parse_flags(rest, names):
|
||||
"""Parse [--flag value] pairs; returns (positionals, {flag: value})."""
|
||||
pos, flags = [], {}
|
||||
i = 0
|
||||
while i < len(rest):
|
||||
tok = rest[i]
|
||||
if tok.startswith("--") and tok[2:] in names:
|
||||
if i + 1 >= len(rest):
|
||||
fail("BAD_ARGS", "flag %s needs a value" % tok)
|
||||
flags[tok[2:]] = rest[i + 1]
|
||||
i += 2
|
||||
elif tok.startswith("--"):
|
||||
fail("BAD_ARGS", "unknown flag: %s" % tok)
|
||||
else:
|
||||
pos.append(tok)
|
||||
i += 1
|
||||
return pos, flags
|
||||
|
||||
|
||||
def main(argv):
|
||||
if len(argv) < 2:
|
||||
print(USAGE, file=sys.stderr)
|
||||
sys.exit(2)
|
||||
action = argv[1]
|
||||
|
||||
if action == "thread-list":
|
||||
pos, flags = parse_flags(argv[2:], {"kind", "limit"})
|
||||
if len(pos) != 1:
|
||||
fail("BAD_ARGS", "usage: thread-list <agent> [--kind K] [--limit N]")
|
||||
agent = check_agent(pos[0])
|
||||
kind = flags.get("kind", "all")
|
||||
if kind not in VALID_KINDS:
|
||||
fail("BAD_ARGS", "kind must be one of %s" % sorted(VALID_KINDS))
|
||||
act_thread_list(agent, kind, check_limit(flags.get("limit"), 50))
|
||||
elif action == "thread-messages":
|
||||
pos, flags = parse_flags(argv[2:], {"limit", "before"})
|
||||
if len(pos) != 2:
|
||||
fail("BAD_ARGS",
|
||||
"usage: thread-messages <agent> <thread-id> [--limit N] [--before MSGID]")
|
||||
agent = check_agent(pos[0])
|
||||
tid = check_thread_id(pos[1])
|
||||
act_thread_messages(agent, tid, check_limit(flags.get("limit"), 50),
|
||||
flags.get("before"))
|
||||
else:
|
||||
print(USAGE, file=sys.stderr)
|
||||
fail("BAD_ARGS", "unknown action: %s" % action)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main(sys.argv)
|
||||
-3591
File diff suppressed because it is too large
Load Diff
Executable
+1611
File diff suppressed because it is too large
Load Diff
@@ -1,74 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
box-query.py — Lightweight client to query box.muse-dev.online APIs over HTTPS.
|
||||
|
||||
Works inside containers and nodes without direct SSH access:
|
||||
Uses SSH signature authentication (?identity=bl&ts=...&sig=...) against
|
||||
the VM Box API.
|
||||
|
||||
Usage:
|
||||
box-query.py timers
|
||||
box-query.py jobs
|
||||
box-query.py agents
|
||||
box-query.py dms [limit]
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
import urllib.parse
|
||||
import urllib.request
|
||||
|
||||
BOX_API_BASE = os.environ.get("BOX_API_BASE", "https://box.muse-dev.online")
|
||||
BOX_SIGN_KEY = os.environ.get("BOX_SIGN_KEY", os.path.expanduser("~/.ssh/id_ed25519"))
|
||||
|
||||
def sign_request(identity, endpoint):
|
||||
ts = str(int(time.time()))
|
||||
payload = f"{ts}\n{endpoint}".encode()
|
||||
try:
|
||||
p = subprocess.run(
|
||||
["ssh-keygen", "-Y", "sign", "-f", BOX_SIGN_KEY, "-n", "box"],
|
||||
input=payload, capture_output=True, timeout=15)
|
||||
if p.returncode != 0:
|
||||
return None, None
|
||||
return ts, p.stdout.decode()
|
||||
except Exception:
|
||||
return None, None
|
||||
|
||||
def query(endpoint):
|
||||
ts, sig = sign_request("bl", endpoint)
|
||||
if not ts or not sig:
|
||||
sys.stderr.write("Failed to sign request (missing key or ssh-keygen error)\n")
|
||||
sys.exit(1)
|
||||
|
||||
query_str = urllib.parse.urlencode({
|
||||
"identity": "bl",
|
||||
"ts": ts,
|
||||
"sig": sig
|
||||
})
|
||||
url = f"{BOX_API_BASE}/api/box/{endpoint}?{query_str}"
|
||||
req = urllib.request.Request(
|
||||
url,
|
||||
headers={"User-Agent": "NetVM-box-query/1.0 (container)"}
|
||||
)
|
||||
with urllib.request.urlopen(req, timeout=15) as resp:
|
||||
data = resp.read().decode("utf-8")
|
||||
try:
|
||||
return json.loads(data)
|
||||
except Exception:
|
||||
return data
|
||||
|
||||
def main():
|
||||
p = argparse.ArgumentParser(description="Query box.muse-dev.online APIs via HTTPS signature auth")
|
||||
p.add_argument("endpoint", choices=["timers", "jobs", "agents", "dms", "health"])
|
||||
p.add_argument("--json", action="store_true")
|
||||
args = p.parse_args()
|
||||
|
||||
res = query(args.endpoint)
|
||||
print(json.dumps(res, indent=2))
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
Executable
+424
@@ -0,0 +1,424 @@
|
||||
#!/usr/bin/env python3
|
||||
"""box-readback-loopback.py — Synchronous Readback Gate & Cognitive-Aware True Loopback Engine.
|
||||
|
||||
Architecture:
|
||||
1. Readback Gate:
|
||||
- Synchronously awaits agent confirmation (up to 45s) after task dispatch.
|
||||
- Validates via Hybrid Tag ([READBACK] Ticket #...) + Semantic Fallback.
|
||||
- Marks ticket 'in-progress' upon confirmation; marks 'blocked' and releases agent on timeout.
|
||||
2. Cognitive-Aware True Loopbacks:
|
||||
- Zero Token Burn: 100% silent while git commits or PRs are progressing.
|
||||
- Cognitive Guard: Defer loopbacks while agent is THINKING / GENERATING.
|
||||
- Dual-Layer Escalation:
|
||||
* 15m inactive: Tier 1 non-intrusive Gitea ticket comment (@agent inquiry).
|
||||
* 45m inactive: Tier 2 direct chat DM escalation (muse-cli-node send).
|
||||
* 90m inactive: Tier 3 failure escalation (mark 'blocked', alert #lobby, release agent).
|
||||
"""
|
||||
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import socket
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
import urllib.request
|
||||
from datetime import datetime, timezone
|
||||
from pathlib import Path
|
||||
|
||||
# Local imports
|
||||
try:
|
||||
import agent_cognitive_probe as acp
|
||||
except ImportError:
|
||||
acp = None
|
||||
|
||||
REMOTE_HOST = "100.123.153.75" # bl control node
|
||||
DEFAULT_GITEA_URL = "https://tea.muse-dev.online"
|
||||
LOOPBACK_STATE_FILE = Path("/tmp/box-loopback-state.json")
|
||||
|
||||
|
||||
def is_running_on_bl():
|
||||
try:
|
||||
hn = socket.gethostname().lower()
|
||||
if "bl" in hn:
|
||||
return True
|
||||
except Exception:
|
||||
pass
|
||||
return os.path.exists("/var/run/netns/warp-muse") or os.path.exists("/run/netns/warp-muse")
|
||||
|
||||
|
||||
def get_gitea_token():
|
||||
token = os.environ.get("GITEA_TOKEN", "3c26744525bceaf385aa09737f7e41af613627b6")
|
||||
return token
|
||||
|
||||
|
||||
def gitea_api(endpoint: str, method: str = "GET", data: dict = None):
|
||||
token = get_gitea_token()
|
||||
url = f"{DEFAULT_GITEA_URL}/api/v1{endpoint}"
|
||||
headers = {
|
||||
"Authorization": f"token {token}",
|
||||
"Content-Type": "application/json",
|
||||
"User-Agent": "Box-Work-CLI/1.0",
|
||||
}
|
||||
payload = json.dumps(data).encode("utf-8") if data else None
|
||||
req = urllib.request.Request(url, data=payload, headers=headers, method=method)
|
||||
try:
|
||||
with urllib.request.urlopen(req, timeout=10) as r:
|
||||
if r.status in (200, 201):
|
||||
return json.loads(r.read().decode())
|
||||
return {"status": r.status}
|
||||
except urllib.error.HTTPError as e:
|
||||
try:
|
||||
return json.loads(e.read().decode())
|
||||
except Exception:
|
||||
return {"error": str(e), "code": e.code}
|
||||
except Exception as e:
|
||||
return {"error": str(e)}
|
||||
|
||||
|
||||
# ----------------------------------------------------------------------
|
||||
# CHAT COMMUNICATION HELPERS
|
||||
# ----------------------------------------------------------------------
|
||||
|
||||
def send_agent_chat(agent: str, message: str) -> bool:
|
||||
"""Delivers a message directly into the agent's web chat session."""
|
||||
if is_running_on_bl():
|
||||
cmd = ["/home/super/Projects/NetVM/bin/muse-cli-node", agent, "send", message]
|
||||
else:
|
||||
cmd = ["ssh", "-q", f"super@{REMOTE_HOST}",
|
||||
f"/home/super/Projects/NetVM/bin/muse-cli-node {agent} send {subprocess.list2cmdline([message])}"]
|
||||
try:
|
||||
res = subprocess.run(cmd, capture_output=True, text=True, timeout=15)
|
||||
return res.returncode == 0
|
||||
except Exception:
|
||||
return False
|
||||
|
||||
|
||||
def get_agent_history(agent: str, limit: int = 5) -> list:
|
||||
"""Retrieves recent chat messages from the agent's active session."""
|
||||
if is_running_on_bl():
|
||||
cmd = ["/home/super/Projects/NetVM/bin/muse-cli-node", agent, "history", "--limit", str(limit)]
|
||||
else:
|
||||
cmd = ["ssh", "-q", f"super@{REMOTE_HOST}",
|
||||
f"/home/super/Projects/NetVM/bin/muse-cli-node {agent} history --limit {limit}"]
|
||||
try:
|
||||
res = subprocess.run(cmd, capture_output=True, text=True, timeout=15)
|
||||
if res.returncode == 0 and res.stdout.strip():
|
||||
data = json.loads(res.stdout.strip())
|
||||
if isinstance(data, list):
|
||||
return data
|
||||
except Exception:
|
||||
pass
|
||||
return []
|
||||
|
||||
|
||||
def get_latest_chat_seq(agent: str) -> int:
|
||||
"""Finds the maximum sequence number in the agent's chat history."""
|
||||
history = get_agent_history(agent, limit=3)
|
||||
seqs = [m.get("seq", 0) for m in history if isinstance(m, dict) and "seq" in m]
|
||||
return max(seqs) if seqs else 0
|
||||
|
||||
|
||||
# ----------------------------------------------------------------------
|
||||
# READBACK GATE
|
||||
# ----------------------------------------------------------------------
|
||||
|
||||
def validate_readback(text: str, issue_num: int, agent: str) -> tuple[bool, str]:
|
||||
"""Validates an agent readback using Hybrid Tag + Semantic Fallback.
|
||||
|
||||
Returns (is_valid, excerpt).
|
||||
"""
|
||||
if not text:
|
||||
return False, ""
|
||||
|
||||
clean_text = text.strip()
|
||||
issue_pattern = rf"#?{issue_num}\b"
|
||||
|
||||
# 1. Strict Tag Match: [READBACK] Ticket #<num> ...
|
||||
if re.search(r"\[READBACK\]", clean_text, re.IGNORECASE) and re.search(issue_pattern, clean_text):
|
||||
snippet = clean_text[:200].replace("\n", " ")
|
||||
return True, snippet
|
||||
|
||||
# 2. Semantic Fallback: Mentions ticket number AND branch/accepted status
|
||||
has_issue = bool(re.search(issue_pattern, clean_text))
|
||||
has_branch_or_ack = bool(re.search(
|
||||
rf"(dev/{agent}/|branch|accepted|working on|confirm|start(ed|ing)|received)",
|
||||
clean_text, re.IGNORECASE
|
||||
))
|
||||
|
||||
if has_issue and has_branch_or_ack:
|
||||
snippet = clean_text[:200].replace("\n", " ")
|
||||
return True, snippet
|
||||
|
||||
return False, ""
|
||||
|
||||
|
||||
def wait_for_readback(agent: str, issue_num: int, initial_seq: int, timeout_s: int = 45, poll_s: float = 3.0) -> dict:
|
||||
"""Synchronously polls for agent readback within timeout_s."""
|
||||
start_time = time.time()
|
||||
deadline = start_time + timeout_s
|
||||
|
||||
while time.time() < deadline:
|
||||
elapsed = int(time.time() - start_time)
|
||||
print(f"\r ⏳ Awaiting Readback from @{agent} ({elapsed}s / {timeout_s}s)...", end="", flush=True)
|
||||
|
||||
history = get_agent_history(agent, limit=4)
|
||||
for msg in history:
|
||||
seq = msg.get("seq", 0)
|
||||
role = msg.get("role", "")
|
||||
text = msg.get("text", "")
|
||||
|
||||
# Only check new assistant messages
|
||||
if seq > initial_seq and role == "assistant":
|
||||
valid, excerpt = validate_readback(text, issue_num, agent)
|
||||
if valid:
|
||||
print()
|
||||
return {
|
||||
"success": True,
|
||||
"snippet": excerpt,
|
||||
"elapsed": elapsed,
|
||||
"seq": seq,
|
||||
}
|
||||
|
||||
time.sleep(poll_s)
|
||||
|
||||
print()
|
||||
return {
|
||||
"success": False,
|
||||
"timeout": True,
|
||||
"elapsed": timeout_s,
|
||||
}
|
||||
|
||||
|
||||
def handle_readback_success(agent: str, issue_num: int, snippet: str):
|
||||
"""Marks ticket in-progress and records confirmation on Gitea."""
|
||||
# Label ticket in-progress
|
||||
gitea_api(f"/repos/super/box/issues/{issue_num}/labels", method="POST", data={"labels": ["in-progress"]})
|
||||
# Post confirmation comment
|
||||
comment_body = f"🤖 **Readback Confirmed** by @{agent}:\n> {snippet}"
|
||||
gitea_api(f"/repos/super/box/issues/{issue_num}/comments", method="POST", data={"body": comment_body})
|
||||
|
||||
|
||||
def handle_readback_timeout(agent: str, issue_num: int, title: str):
|
||||
"""Labels ticket blocked and unassigns agent so they return to IDLE."""
|
||||
# Label ticket blocked
|
||||
gitea_api(f"/repos/super/box/issues/{issue_num}/labels", method="POST", data={"labels": ["blocked"]})
|
||||
# Post explanation comment
|
||||
comment_body = (
|
||||
f"⚠️ **Readback Timeout**: Agent @{agent} did not confirm ticket #{issue_num} "
|
||||
f"within 45 seconds of dispatch. Releasing assignment to prevent deadlocks."
|
||||
)
|
||||
gitea_api(f"/repos/super/box/issues/{issue_num}/comments", method="POST", data={"body": comment_body})
|
||||
# Unassign agent
|
||||
gitea_api(f"/repos/super/box/issues/{issue_num}", method="PATCH", data={"assignees": []})
|
||||
|
||||
|
||||
# ----------------------------------------------------------------------
|
||||
# COGNITIVE-AWARE TRUE LOOPBACK ENGINE
|
||||
# ----------------------------------------------------------------------
|
||||
|
||||
def load_loopback_state() -> dict:
|
||||
if LOOPBACK_STATE_FILE.exists():
|
||||
try:
|
||||
with open(LOOPBACK_STATE_FILE) as f:
|
||||
return json.load(f)
|
||||
except Exception:
|
||||
pass
|
||||
return {}
|
||||
|
||||
|
||||
def save_loopback_state(state: dict):
|
||||
try:
|
||||
with open(LOOPBACK_STATE_FILE, "w") as f:
|
||||
json.dump(state, f, indent=2)
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
|
||||
def get_ticket_git_activity(agent: str, issue_num: int) -> datetime | None:
|
||||
"""Checks the latest commit timestamp on the agent's branch dev/<agent>/<issue>-*."""
|
||||
# Check Gitea branches for dev/<agent>/<issue_num>-*
|
||||
branches = gitea_api("/repos/super/box/branches")
|
||||
if isinstance(branches, list):
|
||||
target_prefix = f"dev/{agent}/{issue_num}"
|
||||
for b in branches:
|
||||
name = b.get("name", "")
|
||||
if target_prefix in name:
|
||||
commit = b.get("commit", {})
|
||||
ts_str = commit.get("timestamp")
|
||||
if ts_str:
|
||||
try:
|
||||
return datetime.fromisoformat(ts_str.replace("Z", "+00:00"))
|
||||
except Exception:
|
||||
pass
|
||||
return None
|
||||
|
||||
|
||||
def get_ticket_last_activity(issue: dict, agent: str) -> tuple[datetime, str]:
|
||||
"""Finds the most recent activity timestamp (git commit, comment, or issue creation)."""
|
||||
issue_num = issue["number"]
|
||||
latest_dt = datetime.fromisoformat(issue["created_at"].replace("Z", "+00:00"))
|
||||
source = "issue_created"
|
||||
|
||||
# Check comments
|
||||
comments = gitea_api(f"/repos/super/box/issues/{issue_num}/comments")
|
||||
if isinstance(comments, list):
|
||||
for c in comments:
|
||||
c_dt = datetime.fromisoformat(c["created_at"].replace("Z", "+00:00"))
|
||||
if c_dt > latest_dt:
|
||||
latest_dt = c_dt
|
||||
source = "gitea_comment"
|
||||
|
||||
# Check git branch commit
|
||||
git_dt = get_ticket_git_activity(agent, issue_num)
|
||||
if git_dt and git_dt > latest_dt:
|
||||
latest_dt = git_dt
|
||||
source = "git_commit"
|
||||
|
||||
return latest_dt, source
|
||||
|
||||
|
||||
def run_loopback_sweep(dry_run: bool = False, verbose: bool = True) -> list:
|
||||
"""Executes a single sweep of all open assigned tickets according to the 3-tier escalation model."""
|
||||
now = datetime.now(timezone.utc)
|
||||
state = load_loopback_state()
|
||||
actions_taken = []
|
||||
|
||||
issues = gitea_api("/repos/super/box/issues?state=open")
|
||||
if not isinstance(issues, list):
|
||||
if verbose:
|
||||
print("Failed to fetch open issues from Gitea.")
|
||||
return []
|
||||
|
||||
assigned_issues = [i for i in issues if i.get("assignee")]
|
||||
if verbose:
|
||||
print(f"\n=== LOOPBACK SWEEP: {len(assigned_issues)} ACTIVE ASSIGNED TICKETS ({now.strftime('%H:%M:%SZ')}) ===")
|
||||
|
||||
for iss in assigned_issues:
|
||||
issue_num = iss["number"]
|
||||
title = iss.get("title", "")
|
||||
agent = iss["assignee"]["username"]
|
||||
key = str(issue_num)
|
||||
ticket_state = state.get(key, {})
|
||||
|
||||
last_dt, source = get_ticket_last_activity(iss, agent)
|
||||
inactive_s = (now - last_dt).total_seconds()
|
||||
inactive_m = int(inactive_s // 60)
|
||||
|
||||
# Check cognitive state
|
||||
cog = acp.get_passive_cognitive_state(agent) if acp else {"status": "IDLE", "cognitive_lock": False}
|
||||
cog_status = cog.get("status", "IDLE")
|
||||
is_thinking = cog_status == "THINKING" or cog.get("cognitive_lock")
|
||||
|
||||
if verbose:
|
||||
print(f"Ticket #{issue_num} (@{agent}): {inactive_m}m inactive (source: {source}) | Cognitive: {cog_status}")
|
||||
|
||||
# Tier 0: Inactive < 15m or active git commits -> Complete silence
|
||||
if inactive_m < 15 or source == "git_commit":
|
||||
if verbose:
|
||||
print(" 👉 Status: Active or within silent grace period (<15m). No action.")
|
||||
continue
|
||||
|
||||
# Check cognitive guard: Defer if agent is thinking/generating
|
||||
if is_thinking and cog_status != "INPUT_WAIT":
|
||||
if verbose:
|
||||
print(f" 🧠 Cognitive Guard: Deferring loopback — @{agent} is currently {cog_status}.")
|
||||
continue
|
||||
|
||||
# Tier 1: 15m <= Inactivity < 45m -> Non-intrusive Gitea ticket comment
|
||||
if 15 <= inactive_m < 45:
|
||||
if ticket_state.get("tier1_sent"):
|
||||
if verbose:
|
||||
print(" 👉 Tier 1 comment already dispatched. Waiting for 45m threshold.")
|
||||
continue
|
||||
|
||||
msg = (
|
||||
f"🤖 @{agent} **Loopback Tier 1 Check** ({inactive_m}m elapsed):\n"
|
||||
f"No git commits recorded on feature branch for Ticket #{issue_num}. "
|
||||
f"Are you progressing or blocked? Reply with status or push a commit."
|
||||
)
|
||||
action_desc = f"Tier 1: Posted Gitea comment to #{issue_num} (@{agent})"
|
||||
actions_taken.append(action_desc)
|
||||
|
||||
if not dry_run:
|
||||
gitea_api(f"/repos/super/box/issues/{issue_num}/comments", method="POST", data={"body": msg})
|
||||
ticket_state["tier1_sent"] = now.isoformat()
|
||||
state[key] = ticket_state
|
||||
save_loopback_state(state)
|
||||
if verbose:
|
||||
print(f" ✓ {action_desc}")
|
||||
|
||||
# Tier 2: 45m <= Inactivity < 90m -> Direct Chat DM Escalation
|
||||
elif 45 <= inactive_m < 90:
|
||||
if ticket_state.get("tier2_sent"):
|
||||
if verbose:
|
||||
print(" 👉 Tier 2 chat DM already dispatched. Waiting for 90m threshold.")
|
||||
continue
|
||||
|
||||
chat_msg = (
|
||||
f"[LOOPBACK ALERT] Ticket #{issue_num} ('{title}'): "
|
||||
f"{inactive_m} minutes inactive with no git commits. "
|
||||
f"Please confirm if blocked on tool execution, terminal approvals, or environment."
|
||||
)
|
||||
action_desc = f"Tier 2: Escalated to chat DM for @{agent} on #{issue_num}"
|
||||
actions_taken.append(action_desc)
|
||||
|
||||
if not dry_run:
|
||||
send_agent_chat(agent, chat_msg)
|
||||
gitea_api(f"/repos/super/box/issues/{issue_num}/comments", method="POST", data={
|
||||
"body": f"📣 **Loopback Tier 2 Escalation**: Inactivity reached {inactive_m}m. Sent direct chat DM to @{agent}."
|
||||
})
|
||||
ticket_state["tier2_sent"] = now.isoformat()
|
||||
state[key] = ticket_state
|
||||
save_loopback_state(state)
|
||||
if verbose:
|
||||
print(f" ✓ {action_desc}")
|
||||
|
||||
# Tier 3: Inactivity >= 90m (or unhandled INPUT_WAIT > 15m) -> Fail-closed escalation
|
||||
elif inactive_m >= 90 or (cog_status == "INPUT_WAIT" and inactive_m >= 15):
|
||||
action_desc = f"Tier 3: Ticket #{issue_num} marked BLOCKED; released @{agent} assignment"
|
||||
actions_taken.append(action_desc)
|
||||
|
||||
if not dry_run:
|
||||
# Label blocked
|
||||
gitea_api(f"/repos/super/box/issues/{issue_num}/labels", method="POST", data={"labels": ["blocked"]})
|
||||
# Post failure comment
|
||||
gitea_api(f"/repos/super/box/issues/{issue_num}/comments", method="POST", data={
|
||||
"body": (
|
||||
f"🚨 **Loopback Tier 3 Escalation**: Inactivity reached {inactive_m}m with zero git progress. "
|
||||
f"Ticket marked `blocked` and unassigned from @{agent} for operator intervention."
|
||||
)
|
||||
})
|
||||
# Unassign agent
|
||||
gitea_api(f"/repos/super/box/issues/{issue_num}", method="PATCH", data={"assignees": []})
|
||||
ticket_state["tier3_sent"] = now.isoformat()
|
||||
state[key] = ticket_state
|
||||
save_loopback_state(state)
|
||||
if verbose:
|
||||
print(f" 🚨 {action_desc}")
|
||||
|
||||
if verbose:
|
||||
print()
|
||||
return actions_taken
|
||||
|
||||
|
||||
def main():
|
||||
import argparse
|
||||
parser = argparse.ArgumentParser(description="Synchronous Readback & Cognitive True Loopback Engine")
|
||||
sub = parser.add_subparsers(dest="cmd")
|
||||
|
||||
p_sweep = sub.add_parser("sweep", help="Run a loopback sweep across open tickets")
|
||||
p_sweep.add_argument("--dry-run", action="store_true", help="Evaluate conditions without sending messages")
|
||||
p_sweep.add_argument("--json", action="store_true", help="Output actions as JSON")
|
||||
|
||||
args = parser.parse_args()
|
||||
if not args.cmd or args.cmd == "sweep":
|
||||
dry_run = getattr(args, "dry_run", False)
|
||||
as_json = getattr(args, "json", False)
|
||||
actions = run_loopback_sweep(dry_run=dry_run, verbose=not as_json)
|
||||
if as_json:
|
||||
print(json.dumps({"ok": True, "actions": actions}, indent=2))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -1,495 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
# box-relay.sh — Lightweight Box Relay Client for Fleet Agent Containers.
|
||||
#
|
||||
# Enables fleet agents (646, pip, muse, opm) to access muse-cli level tools,
|
||||
# spawn subagents, send sidechat DMs, and inspect threads over the secure
|
||||
# exec-constrained HTTPS relay endpoint.
|
||||
#
|
||||
# Supports Bearer Token (EXEC_TOKEN or ~/.exec-token) AND SSH signature auth.
|
||||
# Endpoint: https://exec.muse-dev.online/exec (or Tailscale https://100.123.153.75:8444/exec)
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
EXEC_URL="${EXEC_URL:-https://exec.muse-dev.online/exec}"
|
||||
AGENT="${BOX_AGENT:-${AGENT_NAME:-}}"
|
||||
TOKEN="${EXEC_TOKEN:-}"
|
||||
|
||||
if [ -z "$TOKEN" ] && [ -f "$HOME/.exec-token" ]; then
|
||||
TOKEN="$(cat "$HOME/.exec-token" | tr -d '[:space:]')"
|
||||
fi
|
||||
|
||||
# Detect agent identity if not set
|
||||
if [ -z "$AGENT" ]; then
|
||||
if [ -f "$HOME/.agent-name" ]; then
|
||||
AGENT="$(cat "$HOME/.agent-name" | tr -d '[:space:]')"
|
||||
elif echo "$HOSTNAME" | grep -qi "646"; then
|
||||
AGENT="646"
|
||||
elif echo "$HOSTNAME" | grep -qi "pip"; then
|
||||
AGENT="pip"
|
||||
elif echo "$HOSTNAME" | grep -qi "muse"; then
|
||||
AGENT="muse"
|
||||
elif echo "$HOSTNAME" | grep -qi "opm"; then
|
||||
AGENT="opm"
|
||||
elif echo "$HOSTNAME" | grep -qi "dev"; then
|
||||
AGENT="dev"
|
||||
elif echo "$HOSTNAME" | grep -qi "def"; then
|
||||
AGENT="def"
|
||||
else
|
||||
AGENT="646"
|
||||
fi
|
||||
fi
|
||||
|
||||
# Helper: POST to exec endpoint
|
||||
call_exec() {
|
||||
local op="$1"
|
||||
local args_json="$2"
|
||||
|
||||
if [ -n "$TOKEN" ]; then
|
||||
# Bearer token auth
|
||||
local body
|
||||
body=$(python3 -c "import json, sys; print(json.dumps({'op': sys.argv[1], 'args': json.loads(sys.argv[2])}))" "$op" "$args_json")
|
||||
local res
|
||||
res=$(curl -sk -sS -X POST "$EXEC_URL" \
|
||||
-H "Authorization: Bearer $TOKEN" \
|
||||
-H "Content-Type: application/json" \
|
||||
-H "User-Agent: Mozilla/5.0 (X11; Linux x86_64) Box-Relay/1.0" \
|
||||
--data "$body")
|
||||
echo "$res" | python3 -c "
|
||||
import json, sys
|
||||
try:
|
||||
d = json.load(sys.stdin)
|
||||
if 'stdout' in d:
|
||||
sys.stdout.write(d['stdout'])
|
||||
elif 'error' in d:
|
||||
sys.stderr.write('Error: ' + str(d['error']) + '\n')
|
||||
sys.exit(1)
|
||||
else:
|
||||
print(json.dumps(d, indent=2))
|
||||
except Exception as e:
|
||||
print(sys.stdin.read())
|
||||
"
|
||||
else
|
||||
# Signature auth fallback
|
||||
local key="${SSH_KEY:-$HOME/.ssh/id_frontdoor}"
|
||||
if [ ! -f "$key" ] && [ -f "$HOME/.ssh/id_ed25519" ]; then
|
||||
key="$HOME/.ssh/id_ed25519"
|
||||
fi
|
||||
local ident="operator-$AGENT"
|
||||
if [ "$AGENT" = "super" ]; then
|
||||
ident="super"
|
||||
fi
|
||||
local ts
|
||||
ts=$(date +%s)
|
||||
local nonce
|
||||
nonce=$(python3 -c "import secrets; print(secrets.token_hex(16))")
|
||||
local payload
|
||||
payload=$(python3 -c "
|
||||
import json, sys
|
||||
op, args_json, ts, nonce = sys.argv[1:5]
|
||||
args = json.loads(args_json)
|
||||
print(json.dumps({'op': op, 'args': args, 'ts': int(ts), 'nonce': nonce}))
|
||||
" "$op" "$args_json" "$ts" "$nonce")
|
||||
local sig
|
||||
sig=$(printf '%s' "$payload" | ssh-keygen -Y sign -f "$key" -n exec-constrained 2>/dev/null)
|
||||
local body
|
||||
body=$(python3 -c "
|
||||
import json, sys
|
||||
ident, payload, sig = sys.argv[1:4]
|
||||
print(json.dumps({'identity': ident, 'payload': payload, 'signature': sig}))
|
||||
" "$ident" "$payload" "$sig")
|
||||
local res
|
||||
res=$(curl -sk -sS -X POST "$EXEC_URL" \
|
||||
-H "Content-Type: application/json" \
|
||||
-H "User-Agent: Mozilla/5.0 (X11; Linux x86_64) Box-Relay/1.0" \
|
||||
--data "$body")
|
||||
echo "$res" | python3 -c "
|
||||
import json, sys
|
||||
try:
|
||||
d = json.load(sys.stdin)
|
||||
if 'stdout' in d:
|
||||
sys.stdout.write(d['stdout'])
|
||||
elif 'error' in d:
|
||||
sys.stderr.write('Error: ' + str(d['error']) + '\n')
|
||||
sys.exit(1)
|
||||
else:
|
||||
print(json.dumps(d, indent=2))
|
||||
except Exception as e:
|
||||
print(sys.stdin.read())
|
||||
"
|
||||
fi
|
||||
}
|
||||
|
||||
cmd="${1:-help}"
|
||||
shift || true
|
||||
|
||||
case "$cmd" in
|
||||
subagent)
|
||||
sub="${1:-help}"
|
||||
shift || true
|
||||
case "$sub" in
|
||||
spawn)
|
||||
title="subagent"
|
||||
wait=0
|
||||
while [ $# -gt 0 ]; do
|
||||
case "$1" in
|
||||
--title) title="$2"; shift 2 ;;
|
||||
--wait) wait="$2"; shift 2 ;;
|
||||
--agent) AGENT="$2"; shift 2 ;;
|
||||
*) break ;;
|
||||
esac
|
||||
done
|
||||
prompt="${1:?usage: box subagent spawn [--title <title>] [--wait <seconds>] <prompt>}"
|
||||
args=$(python3 -c "import json, sys; print(json.dumps({'agent': sys.argv[1], 'title': sys.argv[2], 'prompt': sys.argv[3], 'wait': int(sys.argv[4])}))" "$AGENT" "$title" "$prompt" "$wait")
|
||||
call_exec "subagent.spawn" "$args"
|
||||
;;
|
||||
*)
|
||||
echo "Usage: box subagent spawn [--title <title>] [--wait <seconds>] <prompt>"
|
||||
;;
|
||||
esac
|
||||
;;
|
||||
deploy)
|
||||
sub="${1:-help}"
|
||||
shift || true
|
||||
case "$sub" in
|
||||
subagent)
|
||||
title="subagent"
|
||||
wait=0
|
||||
while [ $# -gt 0 ]; do
|
||||
case "$1" in
|
||||
--title) title="$2"; shift 2 ;;
|
||||
--wait) wait="$2"; shift 2 ;;
|
||||
--agent) AGENT="$2"; shift 2 ;;
|
||||
*) break ;;
|
||||
esac
|
||||
done
|
||||
prompt="${1:?usage: box deploy subagent [--title <title>] [--wait <seconds>] <prompt>}"
|
||||
args=$(python3 -c "import json, sys; print(json.dumps({'agent': sys.argv[1], 'title': sys.argv[2], 'prompt': sys.argv[3], 'wait': int(sys.argv[4])}))" "$AGENT" "$title" "$prompt" "$wait")
|
||||
call_exec "subagent.spawn" "$args"
|
||||
;;
|
||||
pipeline)
|
||||
name="${1:?usage: box deploy pipeline <name>}"
|
||||
args=$(python3 -c "import json, sys; print(json.dumps({'name': sys.argv[1]}))" "$name")
|
||||
call_exec "pipeline.run" "$args"
|
||||
;;
|
||||
*)
|
||||
echo "Usage: box deploy subagent|pipeline ..."
|
||||
;;
|
||||
esac
|
||||
;;
|
||||
dm)
|
||||
sub="${1:-help}"
|
||||
shift || true
|
||||
case "$sub" in
|
||||
send)
|
||||
to=""
|
||||
target=""
|
||||
from_agent="$AGENT"
|
||||
while [ $# -gt 0 ]; do
|
||||
case "$1" in
|
||||
--to) to="$2"; shift 2 ;;
|
||||
--target) target="$2"; shift 2 ;;
|
||||
--agent|--from) from_agent="$2"; shift 2 ;;
|
||||
*) break ;;
|
||||
esac
|
||||
done
|
||||
msg="${1:?usage: box dm send --to <agent> [--target <target>] <message>}"
|
||||
to="${to:-$AGENT}"
|
||||
target="${target:-$AGENT-pip}"
|
||||
args=$(python3 -c "import json, sys; print(json.dumps({'agent': sys.argv[1], 'to': sys.argv[2], 'target': sys.argv[3], 'message': sys.argv[4]}))" "$from_agent" "$to" "$target" "$msg")
|
||||
call_exec "dm.send" "$args"
|
||||
;;
|
||||
read)
|
||||
target="${1:-main}"
|
||||
limit="${2:-10}"
|
||||
args=$(python3 -c "import json, sys; print(json.dumps({'agent': sys.argv[1], 'target': sys.argv[2], 'limit': int(sys.argv[3])}))" "$AGENT" "$target" "$limit")
|
||||
call_exec "dm.read" "$args"
|
||||
;;
|
||||
*)
|
||||
echo "Usage: box dm send|read ..."
|
||||
;;
|
||||
esac
|
||||
;;
|
||||
thread)
|
||||
sub="${1:-list}"
|
||||
shift || true
|
||||
case "$sub" in
|
||||
list)
|
||||
target_agent="${1:-$AGENT}"
|
||||
args=$(python3 -c "import json, sys; print(json.dumps({'agent': sys.argv[1]}))" "$target_agent")
|
||||
call_exec "thread.list" "$args"
|
||||
;;
|
||||
view)
|
||||
target_agent="$AGENT"
|
||||
while [ $# -gt 0 ]; do
|
||||
case "$1" in
|
||||
--agent) target_agent="$2"; shift 2 ;;
|
||||
*) break ;;
|
||||
esac
|
||||
done
|
||||
if [ $# -ge 2 ] && echo " 646 opm pip muse dev def " | grep -q " $1 "; then
|
||||
target_agent="$1"
|
||||
shift
|
||||
fi
|
||||
thread_id="${1:?usage: box thread view [<agent>] <thread_id> [limit]}"
|
||||
limit="${2:-15}"
|
||||
args=$(python3 -c "import json, sys; print(json.dumps({'agent': sys.argv[1], 'thread': sys.argv[2], 'limit': int(sys.argv[3])}))" "$target_agent" "$thread_id" "$limit")
|
||||
call_exec "thread.view" "$args"
|
||||
;;
|
||||
*)
|
||||
echo "Usage: box thread list|view ..."
|
||||
;;
|
||||
esac
|
||||
;;
|
||||
cron)
|
||||
sub="${1:-runs}"
|
||||
shift || true
|
||||
case "$sub" in
|
||||
runs|list)
|
||||
call_exec "cron.runs" "{}"
|
||||
;;
|
||||
status)
|
||||
name="${1:-heartbeat}"
|
||||
args=$(python3 -c "import json, sys; print(json.dumps({'name': sys.argv[1]}))" "$name")
|
||||
call_exec "cron.status" "$args"
|
||||
;;
|
||||
view)
|
||||
name="${1:?usage: box cron view <name>}"
|
||||
args=$(python3 -c "import json, sys; print(json.dumps({'name': sys.argv[1]}))" "$name")
|
||||
call_exec "cron.view" "$args"
|
||||
;;
|
||||
run)
|
||||
name="${1:?usage: box cron run <name>}"
|
||||
args=$(python3 -c "import json, sys; print(json.dumps({'job': sys.argv[1]}))" "$name")
|
||||
call_exec "cron.run" "$args"
|
||||
;;
|
||||
timer-create)
|
||||
name="${1:?usage: box cron timer-create <name>}"
|
||||
args=$(python3 -c "import json, sys; print(json.dumps({'name': sys.argv[1]}))" "$name")
|
||||
call_exec "cron.timer_create" "$args"
|
||||
;;
|
||||
timer-start)
|
||||
name="${1:?usage: box cron timer-start <name>}"
|
||||
args=$(python3 -c "import json, sys; print(json.dumps({'name': sys.argv[1]}))" "$name")
|
||||
call_exec "cron.timer_start" "$args"
|
||||
;;
|
||||
*)
|
||||
echo "Usage: box cron runs|status|view|run|timer-create|timer-start ..."
|
||||
;;
|
||||
esac
|
||||
;;
|
||||
timer)
|
||||
sub="${1:-list}"
|
||||
shift || true
|
||||
case "$sub" in
|
||||
create|set)
|
||||
name="${1:?usage: box timer create <name>}"
|
||||
args=$(python3 -c "import json, sys; print(json.dumps({'name': sys.argv[1]}))" "$name")
|
||||
call_exec "cron.timer_create" "$args"
|
||||
;;
|
||||
start|enable)
|
||||
name="${1:?usage: box timer start <name>}"
|
||||
args=$(python3 -c "import json, sys; print(json.dumps({'name': sys.argv[1]}))" "$name")
|
||||
call_exec "cron.timer_start" "$args"
|
||||
;;
|
||||
status|view|list)
|
||||
name="${1:-heartbeat}"
|
||||
args=$(python3 -c "import json, sys; print(json.dumps({'name': sys.argv[1]}))" "$name")
|
||||
call_exec "cron.status" "$args"
|
||||
;;
|
||||
in|delay)
|
||||
min="${1:?usage: box timer in <minutes> <prompt>}"
|
||||
shift
|
||||
prompt="${*:?usage: box timer in <minutes> <prompt>}"
|
||||
args=$(python3 -c "import json, sys; print(json.dumps({'agent': sys.argv[1], 'in_m': float(sys.argv[2]), 'prompt': sys.argv[3]}))" "$AGENT" "$min" "$prompt")
|
||||
call_exec "followup.create" "$args"
|
||||
;;
|
||||
*)
|
||||
echo "Usage: box timer create|start|status <name> OR box timer in <minutes> <prompt>"
|
||||
;;
|
||||
esac
|
||||
;;
|
||||
followup)
|
||||
sub="${1:-create}"
|
||||
shift || true
|
||||
case "$sub" in
|
||||
create|set|schedule)
|
||||
min="${1:?usage: box followup create <minutes> <prompt>}"
|
||||
shift
|
||||
prompt="${*:?usage: box followup create <minutes> <prompt>}"
|
||||
args=$(python3 -c "import json, sys; print(json.dumps({'agent': sys.argv[1], 'in_m': float(sys.argv[2]), 'prompt': sys.argv[3]}))" "$AGENT" "$min" "$prompt")
|
||||
call_exec "followup.create" "$args"
|
||||
;;
|
||||
*)
|
||||
echo "Usage: box followup create <minutes> <prompt>"
|
||||
;;
|
||||
esac
|
||||
;;
|
||||
swarm)
|
||||
sub="${1:-list}"
|
||||
shift || true
|
||||
case "$sub" in
|
||||
spawn)
|
||||
count="${1:?usage: box swarm spawn <count> <task...>}"
|
||||
shift
|
||||
task="$*"
|
||||
args=$(python3 -c "import json, sys; print(json.dumps({'count': int(sys.argv[1]), 'task': sys.argv[2]}))" "$count" "$task")
|
||||
call_exec "swarm.spawn" "$args"
|
||||
;;
|
||||
status)
|
||||
sid="${1:?usage: box swarm status <swarm_id>}"
|
||||
args=$(python3 -c "import json, sys; print(json.dumps({'swarm_id': sys.argv[1]}))" "$sid")
|
||||
call_exec "swarm.status" "$args"
|
||||
;;
|
||||
list)
|
||||
call_exec "swarm.list" "{}"
|
||||
;;
|
||||
results)
|
||||
sid="${1:?usage: box swarm results <swarm_id>}"
|
||||
args=$(python3 -c "import json, sys; print(json.dumps({'swarm_id': sys.argv[1]}))" "$sid")
|
||||
call_exec "swarm.results" "$args"
|
||||
;;
|
||||
*)
|
||||
echo "Usage: box swarm spawn|status|results|list ..."
|
||||
;;
|
||||
esac
|
||||
;;
|
||||
vars)
|
||||
sub="${1:-list}"
|
||||
shift || true
|
||||
case "$sub" in
|
||||
list)
|
||||
call_exec "vars.list" "{}"
|
||||
;;
|
||||
get)
|
||||
name="${1:?usage: box vars get <name>}"
|
||||
args=$(python3 -c "import json, sys; print(json.dumps({'name': sys.argv[1]}))" "$name")
|
||||
call_exec "vars.get" "$args"
|
||||
;;
|
||||
set)
|
||||
name="${1:?usage: box vars set <name> <value>}"
|
||||
val="${2:?usage: box vars set <name> <value>}"
|
||||
args=$(python3 -c "import json, sys; print(json.dumps({'name': sys.argv[1], 'value': sys.argv[2]}))" "$name" "$val")
|
||||
call_exec "vars.set" "$args"
|
||||
;;
|
||||
*)
|
||||
echo "Usage: box vars list|get|set ..."
|
||||
;;
|
||||
esac
|
||||
;;
|
||||
files)
|
||||
sub="${1:-read}"
|
||||
shift || true
|
||||
case "$sub" in
|
||||
read)
|
||||
path="${1:?usage: box files read <path> [lines=100]}"
|
||||
lines="${2:-100}"
|
||||
args=$(python3 -c "import json, sys; print(json.dumps({'path': sys.argv[1], 'lines': int(sys.argv[2])}))" "$path" "$lines")
|
||||
call_exec "files.read" "$args"
|
||||
;;
|
||||
write)
|
||||
path="${1:?usage: box files write <path> <content>}"
|
||||
content="${2:?usage: box files write <path> <content>}"
|
||||
args=$(python3 -c "import json, sys; print(json.dumps({'path': sys.argv[1], 'content': sys.argv[2]}))" "$path" "$content")
|
||||
call_exec "files.write" "$args"
|
||||
;;
|
||||
*)
|
||||
echo "Usage: box files read|write ..."
|
||||
;;
|
||||
esac
|
||||
;;
|
||||
web)
|
||||
sub="${1:-fetch}"
|
||||
shift || true
|
||||
case "$sub" in
|
||||
fetch)
|
||||
url="${1:?usage: box web fetch <url>}"
|
||||
args=$(python3 -c "import json, sys; print(json.dumps({'url': sys.argv[1]}))" "$url")
|
||||
call_exec "web.fetch" "$args"
|
||||
;;
|
||||
*)
|
||||
echo "Usage: box web fetch <url>"
|
||||
;;
|
||||
esac
|
||||
;;
|
||||
service)
|
||||
sub="${1:-status}"
|
||||
shift || true
|
||||
case "$sub" in
|
||||
status)
|
||||
unit="${1:?usage: box service status <unit>}"
|
||||
args=$(python3 -c "import json, sys; print(json.dumps({'unit': sys.argv[1]}))" "$unit")
|
||||
call_exec "service.status" "$args"
|
||||
;;
|
||||
restart)
|
||||
unit="${1:?usage: box service restart <unit>}"
|
||||
args=$(python3 -c "import json, sys; print(json.dumps({'unit': sys.argv[1]}))" "$unit")
|
||||
call_exec "service.restart" "$args"
|
||||
;;
|
||||
*)
|
||||
echo "Usage: box service status|restart <unit>"
|
||||
;;
|
||||
esac
|
||||
;;
|
||||
health|fleet-status)
|
||||
call_exec "health.check" "{}"
|
||||
;;
|
||||
cdp-latency)
|
||||
call_exec "cdp.latency" "{}"
|
||||
;;
|
||||
chrome-errors)
|
||||
call_exec "chrome.errors" "{}"
|
||||
;;
|
||||
watchdog-alerts)
|
||||
call_exec "watchdog.alerts" "{}"
|
||||
;;
|
||||
relay-health)
|
||||
call_exec "relay.health" "{}"
|
||||
;;
|
||||
quality|quality-check)
|
||||
call_exec "quality.check" "{}"
|
||||
;;
|
||||
ping)
|
||||
call_exec "exec.ping" "{}"
|
||||
;;
|
||||
ops)
|
||||
curl -sk -sS "${EXEC_URL%/exec}/ops" | python3 -m json.tool
|
||||
;;
|
||||
help|--help|-h)
|
||||
cat <<EOF
|
||||
Box Relay Client — Agent CLI for autonomous coordination
|
||||
|
||||
Usage:
|
||||
box subagent spawn [--title <title>] [--wait <s=0>] <prompt>
|
||||
box deploy subagent [--title <title>] [--wait <s=0>] <prompt>
|
||||
box deploy pipeline <name>
|
||||
box dm send --to <agent> [--target <target>] <message>
|
||||
box dm read [<target=main>] [<limit=10>]
|
||||
box thread list [<agent>]
|
||||
box thread view <thread_id> [<limit=15>]
|
||||
box cron runs
|
||||
box cron status [<name=heartbeat>]
|
||||
box cron view <name>
|
||||
box cron run <name>
|
||||
box vars list
|
||||
box vars get <name>
|
||||
box vars set <name> <value>
|
||||
box files read <path> [lines=100]
|
||||
box files write <path> <content>
|
||||
box web fetch <url>
|
||||
box service status <unit>
|
||||
box service restart <unit>
|
||||
box health
|
||||
box ping
|
||||
box ops
|
||||
|
||||
Configuration:
|
||||
EXEC_URL Endpoint (default: https://exec.muse-dev.online/exec)
|
||||
EXEC_TOKEN Bearer token (or store in ~/.exec-token)
|
||||
BOX_AGENT Agent identity (646, pip, muse, opm)
|
||||
EOF
|
||||
;;
|
||||
*)
|
||||
echo "Unknown command: $cmd (run 'box help')"
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
@@ -1,204 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
box-sys-op.py — Sandboxed execution helper for core system operations:
|
||||
files.read, files.write, web.fetch, service.status, service.restart
|
||||
Called via fixed argv from exec-constrained.py.
|
||||
"""
|
||||
import sys
|
||||
import os
|
||||
import json
|
||||
import urllib.request
|
||||
import urllib.error
|
||||
import urllib.parse
|
||||
import ipaddress
|
||||
import subprocess
|
||||
from pathlib import Path
|
||||
|
||||
REPO_ROOT = Path("/home/super/Projects/NetVM").resolve()
|
||||
MAX_OUTPUT = 4096
|
||||
ALLOWED_SERVICES = {
|
||||
"board.service", "caddy.service", "response-harvester.timer",
|
||||
"self-main-loop.timer", "job-heartbeat.timer", "job-scheduler.timer"
|
||||
}
|
||||
|
||||
def safe_repo_path(raw):
|
||||
clean = os.path.normpath(raw.strip())
|
||||
if not os.path.isabs(clean):
|
||||
clean = os.path.normpath(str(REPO_ROOT / clean))
|
||||
real = Path(clean).resolve()
|
||||
if not str(real).startswith(str(REPO_ROOT) + "/") and real != REPO_ROOT:
|
||||
raise ValueError("path must reside inside repository root (/home/super/Projects/NetVM)")
|
||||
return real
|
||||
|
||||
def op_files_read(path_str, max_lines=100):
|
||||
p = safe_repo_path(path_str)
|
||||
if not p.exists() or not p.is_file():
|
||||
return {"ok": False, "error": f"File not found: {path_str}"}
|
||||
with open(p, "r", encoding="utf-8", errors="replace") as f:
|
||||
lines = f.readlines()
|
||||
total_lines = len(lines)
|
||||
snippet = "".join(lines[:max_lines])
|
||||
truncated = total_lines > max_lines or len(snippet) > MAX_OUTPUT
|
||||
if len(snippet) > MAX_OUTPUT:
|
||||
snippet = snippet[:MAX_OUTPUT] + "\n... [truncated]"
|
||||
return {
|
||||
"ok": True,
|
||||
"path": str(p.relative_to(REPO_ROOT)),
|
||||
"lines": total_lines,
|
||||
"displayed_lines": min(total_lines, max_lines),
|
||||
"content": snippet,
|
||||
"truncated": truncated
|
||||
}
|
||||
|
||||
def op_files_write(path_str, content):
|
||||
p = safe_repo_path(path_str)
|
||||
p.parent.mkdir(parents=True, exist_ok=True)
|
||||
with open(p, "w", encoding="utf-8") as f:
|
||||
f.write(content)
|
||||
return {
|
||||
"ok": True,
|
||||
"path": str(p.relative_to(REPO_ROOT)),
|
||||
"bytes_written": len(content.encode("utf-8")),
|
||||
"lines": content.count("\n") + 1
|
||||
}
|
||||
|
||||
def op_web_fetch(url_str):
|
||||
parsed = urllib.parse.urlparse(url_str)
|
||||
if parsed.scheme not in ("http", "https"):
|
||||
return {"ok": False, "error": "URL scheme must be http or https"}
|
||||
host = parsed.hostname or ""
|
||||
if not host or host in ("localhost", "127.0.0.1", "::1"):
|
||||
return {"ok": False, "error": "Loopback destinations blocked"}
|
||||
try:
|
||||
ip = ipaddress.ip_address(host)
|
||||
if ip.is_private or ip.is_loopback or ip.is_link_local:
|
||||
return {"ok": False, "error": "Private and local IP addresses blocked"}
|
||||
except ValueError:
|
||||
pass
|
||||
|
||||
req = urllib.request.Request(
|
||||
url_str,
|
||||
headers={"User-Agent": "Mozilla/5.0 Box-Agent-Client/1.0"}
|
||||
)
|
||||
try:
|
||||
with urllib.request.urlopen(req, timeout=10) as resp:
|
||||
data = resp.read(MAX_OUTPUT + 1024).decode("utf-8", errors="replace")
|
||||
status = resp.status
|
||||
truncated = len(data) > MAX_OUTPUT
|
||||
if truncated:
|
||||
data = data[:MAX_OUTPUT] + "\n... [truncated]"
|
||||
return {
|
||||
"ok": True,
|
||||
"url": url_str,
|
||||
"status": status,
|
||||
"length": len(data),
|
||||
"body": data,
|
||||
"truncated": truncated
|
||||
}
|
||||
except urllib.error.HTTPError as he:
|
||||
return {"ok": False, "status": he.code, "error": f"HTTP {he.code}: {he.reason}"}
|
||||
except Exception as e:
|
||||
return {"ok": False, "error": str(e)}
|
||||
|
||||
def op_service_status(unit):
|
||||
if unit not in ALLOWED_SERVICES:
|
||||
return {"ok": False, "error": f"Service not allowed: {unit}"}
|
||||
flag = "--user" if unit.endswith(".timer") else "--system"
|
||||
cmd = ["systemctl", flag, "status", unit] if flag == "--user" else ["systemctl", "is-active", unit]
|
||||
r = subprocess.run(cmd, capture_output=True, text=True, timeout=10)
|
||||
active = "active" in r.stdout.lower() or "active" in r.stderr.lower()
|
||||
return {
|
||||
"ok": True,
|
||||
"service": unit,
|
||||
"active": active,
|
||||
"status_line": r.stdout.splitlines()[0] if r.stdout.splitlines() else "unknown",
|
||||
"output": r.stdout[:500].strip()
|
||||
}
|
||||
|
||||
def op_service_restart(unit):
|
||||
if unit not in ALLOWED_SERVICES:
|
||||
return {"ok": False, "error": f"Service not allowed: {unit}"}
|
||||
if unit.endswith(".timer"):
|
||||
cmd = ["systemctl", "--user", "restart", unit]
|
||||
else:
|
||||
cmd = ["sudo", "-n", "systemctl", "restart", unit]
|
||||
r = subprocess.run(cmd, capture_output=True, text=True, timeout=15)
|
||||
return {
|
||||
"ok": r.returncode == 0,
|
||||
"service": unit,
|
||||
"restarted": r.returncode == 0,
|
||||
"error": r.stderr.strip() if r.returncode != 0 else None
|
||||
}
|
||||
|
||||
def op_followup_schedule(args):
|
||||
agent = args.get("agent")
|
||||
if agent not in ("muse", "pip", "646", "opm", "dev", "def"):
|
||||
return {"ok": False, "error": f"Invalid agent: {agent}"}
|
||||
sender = args.get("sender") or agent
|
||||
if sender not in ("muse", "pip", "646", "opm", "dev", "def"):
|
||||
sender = agent
|
||||
try:
|
||||
in_m = float(args.get("in_m", 1))
|
||||
except (TypeError, ValueError):
|
||||
return {"ok": False, "error": "in_m must be a number"}
|
||||
sec = max(5, int(in_m * 60))
|
||||
prompt = args.get("prompt", "")
|
||||
if not prompt or not isinstance(prompt, str):
|
||||
return {"ok": False, "error": "prompt must be a non-empty string"}
|
||||
if len(prompt) > 1000:
|
||||
return {"ok": False, "error": "prompt exceeds 1000 characters"}
|
||||
|
||||
sidechat = args.get("thread") or args.get("sidechat")
|
||||
cmd = [
|
||||
"systemd-run", "--user", f"--on-active={sec}s",
|
||||
sys.executable, str(REPO_ROOT / "bin" / "box-ctl.py"),
|
||||
"notify", agent, prompt, "--sender", sender
|
||||
]
|
||||
if sidechat and isinstance(sidechat, str) and len(sidechat) <= 64:
|
||||
cmd.extend(["--sidechat", sidechat])
|
||||
|
||||
r = subprocess.run(cmd, capture_output=True, text=True, timeout=15)
|
||||
if r.returncode != 0:
|
||||
return {"ok": False, "error": r.stderr.strip() or r.stdout.strip()}
|
||||
return {
|
||||
"ok": True,
|
||||
"agent": agent,
|
||||
"in_seconds": sec,
|
||||
"sidechat": sidechat,
|
||||
"timer_info": (r.stderr or r.stdout).strip()
|
||||
}
|
||||
|
||||
def main():
|
||||
if len(sys.argv) < 2:
|
||||
print(json.dumps({"ok": False, "error": "missing operation"}))
|
||||
sys.exit(1)
|
||||
op = sys.argv[1]
|
||||
raw_args = sys.stdin.read()
|
||||
try:
|
||||
args = json.loads(raw_args) if raw_args.strip() else {}
|
||||
except Exception as e:
|
||||
print(json.dumps({"ok": False, "error": f"bad json args: {e}"}))
|
||||
sys.exit(1)
|
||||
|
||||
try:
|
||||
if op == "files.read":
|
||||
res = op_files_read(args.get("path", ""), int(args.get("lines", 100)))
|
||||
elif op == "files.write":
|
||||
res = op_files_write(args.get("path", ""), args.get("content", ""))
|
||||
elif op == "web.fetch":
|
||||
res = op_web_fetch(args.get("url", ""))
|
||||
elif op == "service.status":
|
||||
res = op_service_status(args.get("unit", args.get("name", "")))
|
||||
elif op == "service.restart":
|
||||
res = op_service_restart(args.get("unit", args.get("name", "")))
|
||||
elif op in ("followup.create", "followup.schedule"):
|
||||
res = op_followup_schedule(args)
|
||||
else:
|
||||
res = {"ok": False, "error": f"unknown operation: {op}"}
|
||||
except Exception as e:
|
||||
res = {"ok": False, "error": str(e)}
|
||||
|
||||
print(json.dumps(res))
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
Executable
+1138
File diff suppressed because it is too large
Load Diff
Symlink
+1
@@ -0,0 +1 @@
|
||||
box-readback-loopback.py
|
||||
Symlink
+1
@@ -0,0 +1 @@
|
||||
/home/super/Projects/NetVM/bin/box-work.py
|
||||
-413
@@ -1,413 +0,0 @@
|
||||
"""
|
||||
brain.py — Main-loop brain workspace module.
|
||||
|
||||
The brain sidechat ("main-loop brain" on opm's account) is where the main loop
|
||||
OPERATES: it posts its thinking there, reads operator instructions from there,
|
||||
and keeps its working state visible there.
|
||||
|
||||
Deploy to: ~/Projects/NetVM/bin/brain.py on bl (alongside self_main_loop.py).
|
||||
|
||||
Design source: ~/workspace/main-loop-brain-design.md
|
||||
User directive 2026-10-04: "we need main loop to operate its brains in side chat"
|
||||
|
||||
SAFETY CONTRACT (do not weaken):
|
||||
- Brain posts carry a [BRAIN <ts>] marker, NEVER a [JOB <id>] marker.
|
||||
The response-harvester keys off [JOB ...]; a thinking note must never
|
||||
look actionable.
|
||||
- Brain posts NEVER use dm.py --expect-reply. Thinking creates no followup
|
||||
records, no nudges, no escalations.
|
||||
- When reading the brain, the loop skips its own messages (sender check).
|
||||
The loop must never digest its own thinking as agent activity.
|
||||
- !loop commands are honored ONLY from AUTHORIZED_SENDERS. Everything else
|
||||
is read as context, never as instruction.
|
||||
- Malformed commands get a one-line correction posted to the brain.
|
||||
Never silent, never a crash.
|
||||
- Cap: MAX_BRAIN_POSTS_PER_TICK posts per tick. The brain must not amplify.
|
||||
|
||||
State lives in the existing watermark JSON file under the "brain" key:
|
||||
{"brain": {"ignores": {"646": <expires_epoch>}, "quiet_until": <epoch|0>,
|
||||
"brain_watermark": <epoch>, "tick": <int>}}
|
||||
Absolute expiries so a dead loop cannot leave an agent ignored forever.
|
||||
"""
|
||||
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import subprocess
|
||||
import time
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Constants
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
BRAIN_AGENT = "opm"
|
||||
BRAIN_SIDECHAT_NAME = "main-loop brain"
|
||||
|
||||
# Operator identities allowed to issue !loop commands. These must match the
|
||||
# DM-signer / board identity strings; do not invent new ones here.
|
||||
AUTHORIZED_SENDERS = {"super", "operator-646", "operator-main"}
|
||||
|
||||
# Sender identities the loop itself posts under (skipped on read-back).
|
||||
OWN_SENDERS = {"main-loop", "self_main_loop", "operator-main-loop", "opm"}
|
||||
|
||||
BRAIN_MARKER_RE = re.compile(r"\[BRAIN\s+([^\]]+)\]")
|
||||
JOB_MARKER_RE = re.compile(r"\[JOB\s+([^\]]+)\]")
|
||||
LOOP_CMD_RE = re.compile(r"^\s*!loop\s+(\S+)(.*)$", re.IGNORECASE)
|
||||
|
||||
MAX_BRAIN_POSTS_PER_TICK = 4
|
||||
DEFAULT_BIN_DIR = os.path.expanduser("~/Projects/NetVM/bin")
|
||||
|
||||
VALID_AGENTS = {"muse", "pip", "646", "opm"}
|
||||
|
||||
DUR_RE = re.compile(r"^\s*(\d+)\s*([smh])\s*$", re.IGNORECASE)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Pure logic — fully unit-testable, no I/O
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def parse_duration(text):
|
||||
"""'30m' -> 1800.0, '1h' -> 3600.0, '90s' -> 90.0. None if malformed."""
|
||||
m = DUR_RE.match(text or "")
|
||||
if not m:
|
||||
return None
|
||||
n, unit = int(m.group(1)), m.group(2).lower()
|
||||
return float(n * {"s": 1, "m": 60, "h": 3600}[unit])
|
||||
|
||||
|
||||
def is_own_message(sender):
|
||||
s = (sender or "").strip().lower()
|
||||
return s in {x.lower() for x in OWN_SENDERS}
|
||||
|
||||
|
||||
def is_authorized(sender):
|
||||
return (sender or "").strip() in AUTHORIZED_SENDERS
|
||||
|
||||
|
||||
def extract_loop_commands(messages):
|
||||
"""messages: list of {"sender": str, "text": str, "ts": float}.
|
||||
Returns [(sender, verb, args, msg)] for !loop lines from any sender
|
||||
(authorization is applied by the caller so corrections can name names)."""
|
||||
cmds = []
|
||||
for msg in messages:
|
||||
text = msg.get("text") or ""
|
||||
for line in text.splitlines():
|
||||
m = LOOP_CMD_RE.match(line)
|
||||
if m:
|
||||
cmds.append((msg.get("sender", "?"),
|
||||
m.group(1).lower(),
|
||||
m.group(2).strip(),
|
||||
msg))
|
||||
return cmds
|
||||
|
||||
|
||||
def prune_expired(state, now=None):
|
||||
"""Drop expired ignores / quiet. Returns (notes, changed)."""
|
||||
now = now if now is not None else time.time()
|
||||
notes, changed = [], False
|
||||
ignores = state.setdefault("ignores", {})
|
||||
for agent in list(ignores):
|
||||
if ignores[agent] <= now:
|
||||
del ignores[agent]
|
||||
notes.append("resuming %s (ignore expired)" % agent)
|
||||
changed = True
|
||||
if state.get("quiet_until", 0) and state["quiet_until"] <= now:
|
||||
state["quiet_until"] = 0
|
||||
notes.append("quiet period ended, prompts resumed")
|
||||
changed = True
|
||||
return notes, changed
|
||||
|
||||
|
||||
def is_ignored(state, agent, now=None):
|
||||
now = now if now is not None else time.time()
|
||||
return state.get("ignores", {}).get(agent, 0) > now
|
||||
|
||||
|
||||
def is_quiet(state, now=None):
|
||||
now = now if now is not None else time.time()
|
||||
return (state.get("quiet_until", 0) or 0) > now
|
||||
|
||||
|
||||
def apply_command(sender, verb, args, state, now=None):
|
||||
"""Apply one !loop command. Returns (ack_text, changed)."""
|
||||
now = now if now is not None else time.time()
|
||||
changed = False
|
||||
|
||||
if verb == "ignore":
|
||||
parts = args.split()
|
||||
if len(parts) != 2 or parts[0] not in VALID_AGENTS:
|
||||
return ("usage: !loop ignore <agent> <dur> (agent: %s, dur like 30m/1h)"
|
||||
% "/".join(sorted(VALID_AGENTS)), False)
|
||||
dur = parse_duration(parts[1])
|
||||
if dur is None or dur <= 0:
|
||||
return ("bad duration %r, try 30m or 1h" % parts[1], False)
|
||||
state.setdefault("ignores", {})[parts[0]] = now + dur
|
||||
return ("ignoring %s for %s (until %s)"
|
||||
% (parts[0], parts[1],
|
||||
time.strftime("%H:%M UTC", time.gmtime(now + dur))), True)
|
||||
|
||||
if verb == "unignore":
|
||||
agent = args.strip()
|
||||
if agent not in VALID_AGENTS:
|
||||
return ("usage: !loop unignore <agent>", False)
|
||||
if state.get("ignores", {}).pop(agent, None) is not None:
|
||||
changed = True
|
||||
return ("resuming %s now" % agent, True)
|
||||
return ("%s was not ignored" % agent, False)
|
||||
|
||||
if verb == "quiet":
|
||||
dur = parse_duration(args)
|
||||
if dur is None or dur <= 0:
|
||||
return ("usage: !loop quiet <dur> (dur like 30m/1h)", False)
|
||||
state["quiet_until"] = now + dur
|
||||
changed = True
|
||||
return ("quiet for %s: reads continue, digests land here, "
|
||||
"no per-agent escalation" % args.strip(), True)
|
||||
|
||||
if verb == "unquiet":
|
||||
if state.get("quiet_until"):
|
||||
state["quiet_until"] = 0
|
||||
changed = True
|
||||
return ("prompts resumed", True)
|
||||
return ("was not quiet", False)
|
||||
|
||||
if verb == "check":
|
||||
# The tick loop honors this by running the agent reads immediately
|
||||
# rather than waiting for the next timer fire. Caller sets the flag.
|
||||
return ("CHECK_REQUESTED", True)
|
||||
|
||||
if verb == "status":
|
||||
return ("STATUS_REQUESTED", False)
|
||||
|
||||
return ("unknown command %r, try: ignore, unignore, quiet, unquiet, "
|
||||
"check, status" % verb, False)
|
||||
|
||||
|
||||
def format_thinking(tick, summary, state, now=None):
|
||||
"""summary: dict with keys seen{agent:(new,q,urgent)}, escalated[JOB ids],
|
||||
skipped_info[int], errors[int], closure_rate[float|None]."""
|
||||
now = now if now is not None else time.time()
|
||||
ts = time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime(now))
|
||||
seen_bits = []
|
||||
for agent in ("646", "pip", "muse", "opm"):
|
||||
new, q, u = summary.get("seen", {}).get(agent, (0, 0, 0))
|
||||
bit = "%s:%dnew" % (agent, new)
|
||||
if q:
|
||||
bit += "(%d?)" % q
|
||||
if u:
|
||||
bit += "(%d!)" % u
|
||||
seen_bits.append(bit)
|
||||
esc = summary.get("escalated", [])
|
||||
lines = [
|
||||
"[BRAIN %s tick=%d]" % (ts, tick),
|
||||
"seen: " + ", ".join(seen_bits),
|
||||
"decided: escalated %d%s | skipped %d info | ignored: %s" % (
|
||||
len(esc),
|
||||
(" (%s)" % ", ".join(esc[:3])) if esc else "",
|
||||
summary.get("skipped_info", 0),
|
||||
", ".join(sorted(state.get("ignores", {}))) or "none"),
|
||||
"errors: %d | quiet: %s | closure: %s" % (
|
||||
summary.get("errors", 0),
|
||||
"yes" if is_quiet(state, now) else "no",
|
||||
("%.2f" % summary["closure_rate"])
|
||||
if summary.get("closure_rate") is not None else "n/a"),
|
||||
]
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def format_status(tick, cfg, state, watermarks, health, now=None):
|
||||
now = now if now is not None else time.time()
|
||||
ts = time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime(now))
|
||||
lines = ["[BRAIN %s tick=%d] status" % (ts, tick)]
|
||||
lines.append("agents: " + ", ".join(
|
||||
"%s=%s" % (a, "on" if cfg.get(a) else "off")
|
||||
for a in ("muse", "pip", "646", "opm")))
|
||||
ign = state.get("ignores", {})
|
||||
lines.append("ignores: " + (", ".join(
|
||||
"%s until %s" % (a, time.strftime("%H:%M UTC", time.gmtime(e)))
|
||||
for a, e in sorted(ign.items())) or "none"))
|
||||
q = state.get("quiet_until", 0)
|
||||
lines.append("quiet: " + ("until %s" % time.strftime("%H:%M UTC", time.gmtime(q))
|
||||
if q and q > now else "no"))
|
||||
ages = []
|
||||
for a in ("muse", "pip", "646", "opm"):
|
||||
wm = (watermarks or {}).get(a, 0)
|
||||
ages.append("%s:%dm" % (a, int((now - wm) / 60)) if wm else "%s:never" % a)
|
||||
lines.append("watermark age: " + ", ".join(ages))
|
||||
if health:
|
||||
lines.append("digest health: delivered=%d acked=%d closed=%d stale=%d "
|
||||
"closure=%.2f" % (
|
||||
health.get("delivered", 0), health.get("acked", 0),
|
||||
health.get("closed", 0), health.get("stale", 0),
|
||||
health.get("closure_rate", 0.0)))
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# I/O adapters — thin shells over the bl tools. Verify paths on bl.
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
class BrainIO:
|
||||
"""Send/receive against the brain sidechat.
|
||||
|
||||
send: dm.py, WITHOUT --expect-reply (thinking is never actionable).
|
||||
read: pluggable read_fn(messages-since-ts); defaults to None and must be
|
||||
wired to the same chat-read primitive the response-harvester uses.
|
||||
"""
|
||||
|
||||
def __init__(self, bin_dir=DEFAULT_BIN_DIR, read_fn=None,
|
||||
sidechat_map_path=None):
|
||||
self.bin_dir = bin_dir
|
||||
self.read_fn = read_fn
|
||||
self.sidechat_map_path = (sidechat_map_path or
|
||||
os.path.join(bin_dir, "..",
|
||||
"job-sidechats.json"))
|
||||
self._posts_this_tick = 0
|
||||
|
||||
# -- name resolution: never hardcode the thread UUID -------------------
|
||||
def brain_thread_uuid(self):
|
||||
"""Resolve 'main-loop brain' -> UUID via job-sidechats.json."""
|
||||
try:
|
||||
with open(os.path.normpath(self.sidechat_map_path)) as f:
|
||||
data = json.load(f)
|
||||
except (OSError, ValueError):
|
||||
return None
|
||||
# schema: {"sidechats": {"main-loop brain": {"uuid": ...}}} or flat
|
||||
node = data.get("sidechats", data).get(BRAIN_SIDECHAT_NAME)
|
||||
if isinstance(node, dict):
|
||||
return node.get("uuid") or node.get("thread_uuid")
|
||||
return node if isinstance(node, str) else None
|
||||
|
||||
# -- write --------------------------------------------------------------
|
||||
def reset_tick_budget(self):
|
||||
self._posts_this_tick = 0
|
||||
|
||||
def post(self, text):
|
||||
"""Post thinking/acks to the brain. Returns True on VERIFIED send."""
|
||||
if self._posts_this_tick >= MAX_BRAIN_POSTS_PER_TICK:
|
||||
return False
|
||||
if JOB_MARKER_RE.search(text):
|
||||
raise ValueError("refusing to post [JOB ...] to the brain")
|
||||
dm = os.path.join(self.bin_dir, "dm.py")
|
||||
# Same path the loop uses for opm digests; NO --expect-reply:
|
||||
# thinking is never actionable and must not create followups.
|
||||
cmd = [dm, "send", "--agent", BRAIN_AGENT,
|
||||
"--to", BRAIN_AGENT, "--target", BRAIN_SIDECHAT_NAME,
|
||||
"--message", text]
|
||||
try:
|
||||
p = subprocess.run(cmd, capture_output=True, text=True,
|
||||
timeout=120)
|
||||
except (OSError, subprocess.TimeoutExpired):
|
||||
return False
|
||||
ok = "VERIFIED" in (p.stdout or "")
|
||||
if ok:
|
||||
self._posts_this_tick += 1
|
||||
return ok
|
||||
|
||||
# -- read ---------------------------------------------------------------
|
||||
def read_new(self, since_ts):
|
||||
"""Return [{"sender","text","ts"}] newer than since_ts. Skips own."""
|
||||
if self.read_fn is None:
|
||||
return []
|
||||
try:
|
||||
msgs = self.read_fn(BRAIN_AGENT, BRAIN_SIDECHAT_NAME, since_ts) or []
|
||||
except Exception:
|
||||
return []
|
||||
return [m for m in msgs
|
||||
if (m.get("ts", 0) or 0) > since_ts
|
||||
and not is_own_message(m.get("sender"))]
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Workspace — one object per tick
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
class BrainWorkspace:
|
||||
"""Owns the brain side of a main-loop tick.
|
||||
|
||||
Usage in self_main_loop.py tick():
|
||||
brain = BrainWorkspace(watermark_path, read_fn=<harvester reader>)
|
||||
cmds_outcome = brain.intake() # read, parse, apply, ack
|
||||
... existing per-agent reads, skipping brain.ignored(agent) ...
|
||||
brain.post_thinking(summary) # the loop's reasoning, visible
|
||||
brain.save()
|
||||
"""
|
||||
|
||||
def __init__(self, watermark_path, read_fn=None, bin_dir=DEFAULT_BIN_DIR):
|
||||
self.watermark_path = watermark_path
|
||||
self.io = BrainIO(bin_dir=bin_dir, read_fn=read_fn)
|
||||
self._data = self._load()
|
||||
self.state = self._data.setdefault("brain", {})
|
||||
self.io.reset_tick_budget()
|
||||
self.check_requested = False
|
||||
self.status_requested = False
|
||||
|
||||
# -- persistence ---------------------------------------------------------
|
||||
def _load(self):
|
||||
try:
|
||||
with open(self.watermark_path) as f:
|
||||
return json.load(f)
|
||||
except (OSError, ValueError):
|
||||
return {}
|
||||
|
||||
def save(self):
|
||||
tmp = self.watermark_path + ".tmp"
|
||||
with open(tmp, "w") as f:
|
||||
json.dump(self._data, f, indent=2)
|
||||
os.replace(tmp, self.watermark_path)
|
||||
|
||||
# -- tick intake: read -> parse -> apply -> ack --------------------------
|
||||
def intake(self):
|
||||
"""Process new brain messages. Returns dict of what happened."""
|
||||
now = time.time()
|
||||
outcome = {"commands": 0, "acks": 0, "expired_notes": 0,
|
||||
"ignored_senders": 0}
|
||||
since = float(self.state.get("brain_watermark", 0))
|
||||
msgs = self.io.read_new(since)
|
||||
newest = since
|
||||
for m in msgs:
|
||||
newest = max(newest, float(m.get("ts", 0) or 0))
|
||||
if msgs:
|
||||
self.state["brain_watermark"] = newest
|
||||
|
||||
notes, _ = prune_expired(self.state, now)
|
||||
for n in notes:
|
||||
if self.io.post("[BRAIN %s] %s" % (
|
||||
time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime(now)), n)):
|
||||
outcome["expired_notes"] += 1
|
||||
|
||||
for sender, verb, args, _msg in extract_loop_commands(msgs):
|
||||
if not is_authorized(sender):
|
||||
outcome["ignored_senders"] += 1
|
||||
continue
|
||||
outcome["commands"] += 1
|
||||
ack, _changed = apply_command(sender, verb, args, self.state, now)
|
||||
if ack == "CHECK_REQUESTED":
|
||||
self.check_requested = True
|
||||
ack = "out-of-cycle check armed for this tick"
|
||||
elif ack == "STATUS_REQUESTED":
|
||||
self.status_requested = True
|
||||
continue # status posts at end of tick with full context
|
||||
if self.io.post("[BRAIN %s] @%s %s" % (
|
||||
time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime(now)),
|
||||
sender, ack)):
|
||||
outcome["acks"] += 1
|
||||
return outcome
|
||||
|
||||
# -- state queries for the tick loop --------------------------------------
|
||||
def ignored(self, agent):
|
||||
return is_ignored(self.state, agent)
|
||||
|
||||
def quiet(self):
|
||||
return is_quiet(self.state)
|
||||
|
||||
# -- end of tick -----------------------------------------------------------
|
||||
def post_thinking(self, summary, cfg=None, watermarks=None, health=None):
|
||||
tick = int(self.state.get("tick", 0)) + 1
|
||||
self.state["tick"] = tick
|
||||
ok = self.io.post(format_thinking(tick, summary, self.state))
|
||||
if self.status_requested and cfg is not None:
|
||||
self.io.post(format_status(tick, cfg, self.state,
|
||||
watermarks, health))
|
||||
self.status_requested = False
|
||||
return ok
|
||||
@@ -1,89 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Bridge CLI: Front-Door ↔ muse.ai metadata bridge
|
||||
|
||||
Manages mappings between front-door channels and muse.ai side chats.
|
||||
Storage: ~/Projects/NetVM/bridge/mappings.json (operator-managed via SSH)
|
||||
|
||||
Usage:
|
||||
bridge.py list --channel #ops
|
||||
bridge.py add --agent 646 --chat <id> --channel #ops --purpose "incident-123"
|
||||
bridge.py remove <bridge_id>
|
||||
"""
|
||||
import json
|
||||
import argparse
|
||||
from pathlib import Path
|
||||
from datetime import datetime, timezone
|
||||
|
||||
BRIDGE_DIR = Path.home() / "Projects" / "NetVM" / "bridge"
|
||||
MAPPINGS_FILE = BRIDGE_DIR / "mappings.json"
|
||||
|
||||
def load_mappings():
|
||||
if not MAPPINGS_FILE.exists():
|
||||
return []
|
||||
with open(MAPPINGS_FILE) as f:
|
||||
return json.load(f)
|
||||
|
||||
def save_mappings(mappings):
|
||||
BRIDGE_DIR.mkdir(parents=True, exist_ok=True)
|
||||
with open(MAPPINGS_FILE, 'w') as f:
|
||||
json.dump(mappings, f, indent=2)
|
||||
|
||||
def cmd_list(args):
|
||||
mappings = load_mappings()
|
||||
if args.channel:
|
||||
mappings = [m for m in mappings if m['frontdoor_channel'] == args.channel]
|
||||
if not mappings:
|
||||
print("No mappings found.")
|
||||
return
|
||||
for m in mappings:
|
||||
print(f"{m['bridge_id']}: {m['agent']}/{m['muse_side_chat_id'][:8]} -> {m['frontdoor_channel']} ({m['purpose']}) [{m['status']}]")
|
||||
|
||||
def cmd_add(args):
|
||||
mappings = load_mappings()
|
||||
bridge_id = f"br-{datetime.now(timezone.utc).strftime('%Y%m%d%H%M%S')}"
|
||||
mapping = {
|
||||
"bridge_id": bridge_id,
|
||||
"frontdoor_channel": args.channel,
|
||||
"muse_side_chat_id": args.chat,
|
||||
"agent": args.agent,
|
||||
"linked_at": datetime.now(timezone.utc).isoformat(),
|
||||
"linked_by": args.by or "operator",
|
||||
"purpose": args.purpose,
|
||||
"status": "active"
|
||||
}
|
||||
mappings.append(mapping)
|
||||
save_mappings(mappings)
|
||||
print(f"Added {bridge_id}")
|
||||
|
||||
def cmd_remove(args):
|
||||
mappings = load_mappings()
|
||||
mappings = [m for m in mappings if m['bridge_id'] != args.bridge_id]
|
||||
save_mappings(mappings)
|
||||
print(f"Removed {args.bridge_id}")
|
||||
|
||||
def main():
|
||||
p = argparse.ArgumentParser(description="Bridge: front-door to muse.ai")
|
||||
sub = p.add_subparsers(dest='cmd', required=True)
|
||||
|
||||
p_list = sub.add_parser('list', help='List mappings')
|
||||
p_list.add_argument('--channel', help='Filter by front-door channel')
|
||||
p_list.set_defaults(func=cmd_list)
|
||||
|
||||
p_add = sub.add_parser('add', help='Add a mapping')
|
||||
p_add.add_argument('--agent', required=True, help='Agent (muse/pip/646)')
|
||||
p_add.add_argument('--chat', required=True, help='muse.ai side chat ID')
|
||||
p_add.add_argument('--channel', required=True, help='Front-door channel (#ops, #lobby, etc.)')
|
||||
p_add.add_argument('--purpose', required=True, help='Purpose of the side chat')
|
||||
p_add.add_argument('--by', help='Who linked (operator/agent)')
|
||||
p_add.set_defaults(func=cmd_add)
|
||||
|
||||
p_rm = sub.add_parser('remove', help='Remove a mapping')
|
||||
p_rm.add_argument('bridge_id', help='Bridge ID to remove')
|
||||
p_rm.set_defaults(func=cmd_remove)
|
||||
|
||||
args = p.parse_args()
|
||||
args.func(args)
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
@@ -1,33 +0,0 @@
|
||||
#!/bin/bash
|
||||
# cdp-latency-check.sh — measure CDP relay latency per node.
|
||||
#
|
||||
# Probes each node's host-side relay (PEER_IP:CDP_PORT from netvm-names.sh,
|
||||
# which pins the registry ports) at /json/version and prints one line per
|
||||
# node:
|
||||
# name:latency_ms:code
|
||||
# On failure (timeout / connection refused / non-200-ish transport error):
|
||||
# name:FAIL:000
|
||||
#
|
||||
# Self-contained: only needs netvm-names.sh in the same directory and curl.
|
||||
set -u
|
||||
|
||||
BIN_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
# shellcheck disable=SC1091
|
||||
source "$BIN_DIR/netvm-names.sh"
|
||||
|
||||
TIMEOUT_S=10
|
||||
|
||||
for node in muse pip 646 opm; do
|
||||
netvm_names "$node"
|
||||
url="http://${PEER_IP}:${CDP_PORT}/json/version"
|
||||
probe=$(curl -s -m "$TIMEOUT_S" -o /dev/null -w "%{time_total} %{http_code}" "$url" 2>/dev/null)
|
||||
rc=$?
|
||||
secs=$(printf '%s' "$probe" | awk '{print $1}')
|
||||
code=$(printf '%s' "$probe" | awk '{print $2}')
|
||||
if [ "$rc" -ne 0 ] || [ -z "$code" ] || [ "$code" = "000" ]; then
|
||||
echo "${node}:FAIL:000"
|
||||
continue
|
||||
fi
|
||||
ms=$(awk -v s="$secs" 'BEGIN { printf "%d", (s + 0) * 1000 }')
|
||||
echo "${node}:${ms}:${code}"
|
||||
done
|
||||
@@ -1,124 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
# cdp-relay-watchdog.sh — keep per-node CDP relays alive and correctly routed.
|
||||
# Two-stage health check:
|
||||
# 1. Host veth IP must be assigned (veth_healthy). Without it the relay is
|
||||
# unreachable from the host no matter how many times we restart it —
|
||||
# observed 2026-10-04 (muse/pip veths existed but had no IPs). FAIL_LOUD
|
||||
# in the log; do NOT auto-fix (veth recreation touches WireGuard/iptables).
|
||||
# 2. Relay connectivity (relay_healthy): curl to veth IP:port, not pidfile
|
||||
# (which goes stale and lies — observed 2026-10-04).
|
||||
# If a relay is down or misrouted: kill it and restart via the exact
|
||||
# netvm-node-up.sh relay invocation inside the node's netns.
|
||||
# Runs every 5 min via systemd timer cdp-relay-watchdog.timer.
|
||||
# Pattern mirrors chromebox-watchdog.sh (stage-specific logging, rotation).
|
||||
set -euo pipefail
|
||||
LOCK="/tmp/cdp-relay-watchdog.lock"
|
||||
exec 9>"$LOCK"
|
||||
if ! flock -n 9; then
|
||||
echo "[$(date -u +%FT%TZ)] another relay watchdog run in progress, skipping" >&2
|
||||
exit 0
|
||||
fi
|
||||
|
||||
NETVM_BIN="/home/super/Projects/NetVM/bin"
|
||||
LOG="/home/super/Projects/NetVM/cdp-relay-watchdog.log"
|
||||
|
||||
rotate_log() {
|
||||
local f="$1"
|
||||
[ -f "$f" ] || return 0
|
||||
local sz
|
||||
sz=$(stat -c%s "$f" 2>/dev/null || echo 0)
|
||||
if [ "$sz" -gt 10485760 ]; then
|
||||
mv -f "$f" "$f.1"
|
||||
echo "[$(date -u +%FT%TZ)] log rotated" > "$f"
|
||||
fi
|
||||
}
|
||||
rotate_log "$LOG"
|
||||
log() { echo "[$(date -u +%FT%TZ)] $*" | tee -a "$LOG"; }
|
||||
|
||||
# node -> "veth_ip:port" via netvm-names.sh (hash-derived, don't hardcode)
|
||||
relay_target() {
|
||||
local node="$1"
|
||||
# shellcheck disable=SC1091
|
||||
. "$NETVM_BIN/netvm-names.sh"
|
||||
netvm_names "$node" || return 1
|
||||
# CDP_PORT_OVERRIDE pins registry ports; fall back to hash-derived
|
||||
local port="${CDP_PORT_OVERRIDE:-$CDP_PORT}"
|
||||
case "$node" in
|
||||
muse) port=9410 ;; pip) port=9420 ;; 646) port=9430 ;; opm) port=9440 ;;
|
||||
esac
|
||||
echo "$PEER_IP:$port"
|
||||
}
|
||||
node_port() { echo "${1##*:}"; }
|
||||
|
||||
# node -> "VETH GW" via netvm-names.sh
|
||||
node_veth() {
|
||||
local node="$1"
|
||||
# shellcheck disable=SC1091
|
||||
. "$NETVM_BIN/netvm-names.sh"
|
||||
netvm_names "$node" || return 1
|
||||
echo "$VETH $GW"
|
||||
}
|
||||
|
||||
# Is the host-side veth IP assigned? The relay listens on the netns-side peer
|
||||
# IP; the host reaches it via the veth interface's GW address. If the GW IP is
|
||||
# missing, the relay is unreachable from the host — restarting the relay is
|
||||
# pointless and masks the real problem.
|
||||
veth_healthy() {
|
||||
local node="$1" veth gw
|
||||
read -r veth gw <<< "$(node_veth "$node")" || return 1
|
||||
ip addr show dev "$veth" 2>/dev/null | grep -q "inet ${gw}/" || return 1
|
||||
return 0
|
||||
}
|
||||
|
||||
relay_healthy() {
|
||||
local node="$1" target
|
||||
target="$(relay_target "$node")" || return 1
|
||||
curl -s -m 8 "http://$target/json/version" 2>/dev/null | grep -q '"Browser"' || return 1
|
||||
return 0
|
||||
}
|
||||
|
||||
restart_relay() {
|
||||
local node="$1" target port veth_ip netns
|
||||
target="$(relay_target "$node")"
|
||||
veth_ip="${target%%:*}"
|
||||
port="${target##*:}"
|
||||
netns="warp-$node"
|
||||
# Kill any existing relay for this node's port (correct or not)
|
||||
# Relays are root-owned (started via sudo ip netns exec); the timer runs as
|
||||
# super, so the kill needs sudo too. Without it pkill fails EPERM silently
|
||||
# and the "restart" false-positives via SO_REUSEADDR double-bind.
|
||||
sudo -n pkill -f "netvm-cdp-relay.py .* $port 127.0.0.1 $port" 2>/dev/null || true
|
||||
sleep 2
|
||||
# Launch inside the netns, listening on the veth IP (host-reachable)
|
||||
sudo -n ip netns exec "$netns" setsid nohup python3 \
|
||||
"$NETVM_BIN/netvm-cdp-relay.py" "$veth_ip" "$port" 127.0.0.1 "$port" \
|
||||
>>"$LOG" 2>&1 < /dev/null &
|
||||
sleep 5
|
||||
if relay_healthy "$node"; then
|
||||
log "[$node] relay restarted OK on $target"
|
||||
return 0
|
||||
else
|
||||
log "[$node] relay restart FAILED on $target — needs operator attention"
|
||||
return 1
|
||||
fi
|
||||
}
|
||||
|
||||
FAILED=0
|
||||
for node in muse pip 646 opm; do
|
||||
# Stage 1: host veth IP. Fail loud, skip relay restart (pointless).
|
||||
if ! veth_healthy "$node"; then
|
||||
read -r veth gw <<< "$(node_veth "$node")"
|
||||
log "[$node] FAIL_LOUD: host veth $veth missing IP $gw — relay unreachable, needs netvm-node-up.sh $node (manual)"
|
||||
FAILED=1
|
||||
continue
|
||||
fi
|
||||
# Stage 2: relay connectivity.
|
||||
if relay_healthy "$node"; then
|
||||
continue
|
||||
fi
|
||||
target="$(relay_target "$node")"
|
||||
log "[$node] relay unhealthy on $target, restarting"
|
||||
restart_relay "$node" || FAILED=1
|
||||
done
|
||||
|
||||
exit $FAILED
|
||||
@@ -1,239 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Per-browser CDP operation queue for the NetVM fleet.
|
||||
|
||||
Problem: nothing coordinates browser operations. DM sends (dm.py), tab
|
||||
operations, agent reads (muse-chat-api.py), and watchdog restarts all hit
|
||||
the same Chromium with zero scheduling. Functional tests pass under load,
|
||||
but there is no backpressure — under real fleet concurrency this will
|
||||
overwhelm the machine.
|
||||
|
||||
Solution: per-node FIFO queue with priority levels and a cap on concurrent
|
||||
CDP operations per browser.
|
||||
|
||||
Cross-process design: dm.py shells out to muse-chat-api.py via subprocess,
|
||||
so in-process locks (threading.Semaphore) alone cannot coordinate. This
|
||||
module uses flock'd slot files + ticket files in /tmp, which work across
|
||||
processes AND threads.
|
||||
|
||||
Usage:
|
||||
from cdp_queue import cdp_slot, PRIORITY_HIGH
|
||||
|
||||
with cdp_slot("opm", priority=PRIORITY_HIGH):
|
||||
... do CDP work ...
|
||||
|
||||
Explicit acquire/release:
|
||||
from cdp_queue import acquire, release, QueueTimeout
|
||||
token = acquire("opm", priority=PRIORITY_HIGH, timeout=60)
|
||||
try:
|
||||
...
|
||||
finally:
|
||||
release(token)
|
||||
|
||||
Priority levels (lower number = higher priority):
|
||||
PRIORITY_HIGH = 0 # DM sends (user-facing)
|
||||
PRIORITY_NORMAL = 1 # tab opens, reads
|
||||
PRIORITY_LOW = 2 # background scans
|
||||
|
||||
Fairness: tickets are ordered by (priority, arrival time). A waiter only
|
||||
proceeds when its ticket is first in line AND a slot is free. No starvation:
|
||||
a low-priority ticket eventually becomes the oldest and gets served.
|
||||
"""
|
||||
|
||||
import fcntl
|
||||
import logging
|
||||
import os
|
||||
import time
|
||||
import uuid
|
||||
|
||||
log = logging.getLogger("cdp_queue")
|
||||
|
||||
# ---- Tunables ----
|
||||
PRIORITY_HIGH = 0
|
||||
PRIORITY_NORMAL = 1
|
||||
PRIORITY_LOW = 2
|
||||
|
||||
MAX_CONCURRENT = 2 # max simultaneous CDP ops per browser
|
||||
ACQUIRE_TIMEOUT = 60.0 # fail loud instead of hanging forever
|
||||
WARN_AFTER = 10.0 # log a warning when a waiter waits this long
|
||||
POLL_INTERVAL = 0.05 # ticket/slot poll cadence
|
||||
|
||||
QUEUE_DIR = "/tmp/cdp-queue"
|
||||
VALID_NODES = ("muse", "pip", "646", "opm", "dev", "def")
|
||||
|
||||
|
||||
class QueueTimeout(Exception):
|
||||
"""Raised when a slot cannot be acquired within the timeout."""
|
||||
pass
|
||||
|
||||
|
||||
def _node_dir(node):
|
||||
return os.path.join(QUEUE_DIR, node)
|
||||
|
||||
|
||||
def _tickets_dir(node):
|
||||
return os.path.join(_node_dir(node), "tickets")
|
||||
|
||||
|
||||
def _slot_path(node, i):
|
||||
return os.path.join(_node_dir(node), "slot-%d.lock" % i)
|
||||
|
||||
|
||||
def _ensure_dirs(node):
|
||||
os.makedirs(_tickets_dir(node), exist_ok=True)
|
||||
# Pre-create slot files so flock targets always exist
|
||||
for i in range(MAX_CONCURRENT):
|
||||
p = _slot_path(node, i)
|
||||
if not os.path.exists(p):
|
||||
open(p, "a").close()
|
||||
|
||||
|
||||
def _read_tickets(node):
|
||||
"""Return sorted list of (priority, timestamp, ticket_name), purging stale tickets."""
|
||||
tdir = _tickets_dir(node)
|
||||
out = []
|
||||
now = time.time()
|
||||
try:
|
||||
for name in os.listdir(tdir):
|
||||
if not name.endswith(".ticket"):
|
||||
continue
|
||||
try:
|
||||
# ticket name: "<prio>-<timestamp>-<uuid>.ticket"
|
||||
parts = name[:-7].split("-")
|
||||
prio_s, ts_s = parts[0], parts[1]
|
||||
ts = float(ts_s)
|
||||
# Purge stale ticket if process crashed or timed out ungracefully
|
||||
if now - ts > (ACQUIRE_TIMEOUT * 2):
|
||||
try:
|
||||
os.unlink(os.path.join(tdir, name))
|
||||
except Exception:
|
||||
pass
|
||||
continue
|
||||
out.append((int(prio_s), ts, name))
|
||||
except (ValueError, IndexError):
|
||||
continue
|
||||
except FileNotFoundError:
|
||||
pass
|
||||
out.sort()
|
||||
return out
|
||||
|
||||
|
||||
class _Slot:
|
||||
"""A held queue slot. Release via .release() or context manager."""
|
||||
|
||||
def __init__(self, node, ticket_name, fh, waited):
|
||||
self.node = node
|
||||
self.ticket_name = ticket_name
|
||||
self.fh = fh
|
||||
self.waited = waited
|
||||
self._released = False
|
||||
|
||||
def release(self):
|
||||
if self._released:
|
||||
return
|
||||
self._released = True
|
||||
try:
|
||||
fcntl.flock(self.fh, fcntl.LOCK_UN)
|
||||
self.fh.close()
|
||||
except Exception:
|
||||
pass
|
||||
# Remove our ticket (best effort — a stale ticket is harmless;
|
||||
# the next waiter re-reads the directory each poll)
|
||||
try:
|
||||
os.unlink(os.path.join(_tickets_dir(self.node), self.ticket_name))
|
||||
except Exception:
|
||||
pass
|
||||
log.debug("cdp_queue: released slot for node=%s (waited %.1fs)",
|
||||
self.node, self.waited)
|
||||
|
||||
def __enter__(self):
|
||||
return self
|
||||
|
||||
def __exit__(self, *exc):
|
||||
self.release()
|
||||
|
||||
|
||||
def acquire(node, priority=PRIORITY_NORMAL, timeout=ACQUIRE_TIMEOUT):
|
||||
"""
|
||||
Block until a CDP slot is free for `node`, then return a _Slot.
|
||||
Raises QueueTimeout after `timeout` seconds. Raises ValueError for
|
||||
unknown nodes.
|
||||
"""
|
||||
if node not in VALID_NODES:
|
||||
raise ValueError("unknown node: %r (valid: %s)" % (node, VALID_NODES))
|
||||
if priority not in (PRIORITY_HIGH, PRIORITY_NORMAL, PRIORITY_LOW):
|
||||
raise ValueError("invalid priority: %r" % (priority,))
|
||||
|
||||
_ensure_dirs(node)
|
||||
tdir = _tickets_dir(node)
|
||||
|
||||
# Our ticket: "<prio>-<timestamp>-<uuid>.ticket", sorted by (prio, ts)
|
||||
ticket = "%d-%f-%s.ticket" % (priority, time.time(), uuid.uuid4().hex[:8])
|
||||
open(os.path.join(tdir, ticket), "w").close()
|
||||
|
||||
start = time.time()
|
||||
warned = False
|
||||
try:
|
||||
while True:
|
||||
elapsed = time.time() - start
|
||||
if elapsed >= timeout:
|
||||
raise QueueTimeout(
|
||||
"node=%s: no CDP slot free after %.0fs (priority=%d)" %
|
||||
(node, timeout, priority))
|
||||
if elapsed >= WARN_AFTER and not warned:
|
||||
warned = True
|
||||
depth = len(_read_tickets(node))
|
||||
log.warning("cdp_queue: node=%s waiting %.0fs for slot "
|
||||
"(queue depth %d, priority %d)",
|
||||
node, elapsed, depth, priority)
|
||||
|
||||
tickets = _read_tickets(node)
|
||||
# Am I among the first MAX_CONCURRENT in line?
|
||||
# (priority, then arrival time). The first N tickets are all
|
||||
# eligible to grab slots; they distribute via non-blocking flock.
|
||||
my_pos = next((i for i, (_, _, name) in enumerate(tickets)
|
||||
if name == ticket), None)
|
||||
if my_pos is not None and my_pos < MAX_CONCURRENT:
|
||||
# Try each slot file non-blocking
|
||||
for i in range(MAX_CONCURRENT):
|
||||
fh = open(_slot_path(node, i), "w")
|
||||
try:
|
||||
fcntl.flock(fh, fcntl.LOCK_EX | fcntl.LOCK_NB)
|
||||
except (BlockingIOError, OSError):
|
||||
fh.close()
|
||||
continue
|
||||
# Got it
|
||||
waited = time.time() - start
|
||||
if waited > 1.0:
|
||||
log.debug("cdp_queue: node=%s acquired slot after "
|
||||
"%.1fs (priority %d)", node, waited, priority)
|
||||
return _Slot(node, ticket, fh, waited)
|
||||
time.sleep(POLL_INTERVAL)
|
||||
except BaseException:
|
||||
# On timeout or interrupt, remove our ticket so we don't block others
|
||||
try:
|
||||
os.unlink(os.path.join(tdir, ticket))
|
||||
except Exception:
|
||||
pass
|
||||
raise
|
||||
|
||||
|
||||
def release(slot):
|
||||
"""Release a slot returned by acquire()."""
|
||||
slot.release()
|
||||
|
||||
|
||||
def cdp_slot(node, priority=PRIORITY_NORMAL, timeout=ACQUIRE_TIMEOUT):
|
||||
"""
|
||||
Context manager. Usage:
|
||||
with cdp_slot("opm", priority=PRIORITY_HIGH):
|
||||
... CDP work ...
|
||||
"""
|
||||
return acquire(node, priority=priority, timeout=timeout)
|
||||
|
||||
|
||||
def queue_depth(node):
|
||||
"""Current number of waiters for a node (for monitoring)."""
|
||||
if node not in VALID_NODES:
|
||||
raise ValueError("unknown node: %r" % node)
|
||||
return len(_read_tickets(node))
|
||||
@@ -1,129 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Chat rate metric: messages per minute for main chat and each side chat.
|
||||
|
||||
Polls muse-chat-api.py for message counts, calculates delta vs previous poll,
|
||||
logs rates to a time-series file.
|
||||
|
||||
Usage: chat-rate.py --account <name> --cdp-port <port> [--interval 60]
|
||||
"""
|
||||
import json, subprocess, sys, time, os, argparse
|
||||
from datetime import datetime, timezone
|
||||
|
||||
API = os.path.expanduser("~/Projects/NetVM/bin/muse-chat-api.py")
|
||||
STATE_FILE = os.path.expanduser("~/Projects/NetVM/logs/chat-rate-state.json")
|
||||
LOG_FILE = os.path.expanduser("~/Projects/NetVM/logs/chat-rate.log")
|
||||
|
||||
def run_api(account, *args):
|
||||
"""Run muse-chat-api.py and return stdout."""
|
||||
cmd = ["python3", API, "--account", account] + list(args)
|
||||
# Note: cdp-port is baked into the account config, not passed here
|
||||
result = subprocess.run(cmd, capture_output=True, text=True, timeout=60)
|
||||
return result.stdout
|
||||
|
||||
def count_messages(text):
|
||||
"""Count messages in API output. Messages are separated by '---'."""
|
||||
# The API outputs messages separated by ---\n
|
||||
parts = [p.strip() for p in text.split("---") if p.strip()]
|
||||
# Filter out non-message lines (headers, etc.)
|
||||
# Messages typically have substantial content
|
||||
return len([p for p in parts if len(p) > 10])
|
||||
|
||||
def get_side_chats(account):
|
||||
"""List side chat names."""
|
||||
out = run_api(account, "sidechat", "list")
|
||||
# Parse: names and timestamps separated by blank lines
|
||||
# Format: "Side chats\n\n<name>\n\n<timestamp>\n\n<name>\n\n<timestamp>..."
|
||||
lines = [l.strip() for l in out.split("\n") if l.strip()]
|
||||
chats = []
|
||||
# Skip header "Side chats", then pair up (name, timestamp)
|
||||
lines = [l for l in lines if l != "Side chats"]
|
||||
# Lines alternate: name, timestamp, name, timestamp...
|
||||
for i in range(0, len(lines), 2):
|
||||
if i < len(lines):
|
||||
name = lines[i]
|
||||
# Verify next is a timestamp (ends with m/h/d)
|
||||
if i + 1 < len(lines) and lines[i+1][-1] in "mhd":
|
||||
chats.append(name)
|
||||
return chats
|
||||
|
||||
def get_chat_count(account, chat_name=None):
|
||||
"""Get message count for main or a side chat."""
|
||||
if chat_name:
|
||||
# Switch to side chat, get messages, switch back
|
||||
run_api(account, "sidechat", "use", chat_name)
|
||||
out = run_api(account, "messages")
|
||||
run_api(account, "sidechat", "main") # switch back
|
||||
else:
|
||||
out = run_api(account, "messages")
|
||||
return count_messages(out)
|
||||
|
||||
def main():
|
||||
p = argparse.ArgumentParser()
|
||||
p.add_argument("--account", required=True)
|
||||
p.add_argument("--interval", type=int, default=60, help="poll interval seconds")
|
||||
p.add_argument("--once", action="store_true", help="single poll, no loop")
|
||||
args = p.parse_args()
|
||||
|
||||
# Load previous state
|
||||
prev = {}
|
||||
if os.path.exists(STATE_FILE):
|
||||
with open(STATE_FILE) as f:
|
||||
prev = json.load(f)
|
||||
|
||||
def poll():
|
||||
now = datetime.now(timezone.utc).isoformat()
|
||||
counts = {}
|
||||
|
||||
# Main chat
|
||||
try:
|
||||
counts["main"] = get_chat_count(args.account)
|
||||
except Exception as e:
|
||||
print(f"main: error {e}", file=sys.stderr)
|
||||
|
||||
# Side chats
|
||||
try:
|
||||
sc_list = get_side_chats(args.account)
|
||||
print(f"DEBUG: found {len(sc_list)} side chats", file=sys.stderr)
|
||||
for sc in sc_list:
|
||||
try:
|
||||
counts[f"side:{sc}"] = get_chat_count(args.account, sc)
|
||||
except Exception as e:
|
||||
print(f"side:{sc}: error {e}", file=sys.stderr)
|
||||
except Exception as e:
|
||||
print(f"sidechat list: error {e}", file=sys.stderr)
|
||||
|
||||
# Calculate rates
|
||||
results = []
|
||||
for chat, count in counts.items():
|
||||
rate = 0.0
|
||||
if chat in prev:
|
||||
prev_count, prev_time = prev[chat]
|
||||
dt = (datetime.fromisoformat(now) - datetime.fromisoformat(prev_time)).total_seconds() / 60.0
|
||||
if dt > 0:
|
||||
rate = (count - prev_count) / dt
|
||||
results.append((now, chat, count, round(rate, 2)))
|
||||
prev[chat] = (count, now)
|
||||
|
||||
# Log
|
||||
os.makedirs(os.path.dirname(LOG_FILE), exist_ok=True)
|
||||
with open(LOG_FILE, "a") as f:
|
||||
for ts, chat, count, rate in results:
|
||||
f.write(f"{ts} {chat} count={count} rate={rate}/min\n")
|
||||
|
||||
# Save state
|
||||
with open(STATE_FILE, "w") as f:
|
||||
json.dump(prev, f)
|
||||
|
||||
# Print
|
||||
for ts, chat, count, rate in results:
|
||||
print(f"{chat}: {count} msgs, {rate}/min")
|
||||
|
||||
if args.once:
|
||||
poll()
|
||||
else:
|
||||
while True:
|
||||
poll()
|
||||
time.sleep(args.interval)
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -1,281 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""chat-state-check.py — one-shot CDP chat-state probe for a single node.
|
||||
|
||||
Runs INSIDE the node's netns (CDP listens on 127.0.0.1 there).
|
||||
|
||||
Usage: chat-state-check.py <cdp_port> <mode> [name]
|
||||
modes: main | list | sidechat <name>
|
||||
|
||||
Prints one JSON object to stdout.
|
||||
|
||||
Honest limits (see CHATSTATE_SPEC.md): only the DOM-visible message window
|
||||
is captured (React virtualization); author attribution is best-effort and
|
||||
may be null; checked_at is the report time, not per-message times.
|
||||
"""
|
||||
import base64
|
||||
import json
|
||||
import sys
|
||||
import time
|
||||
import urllib.request
|
||||
|
||||
import websocket
|
||||
|
||||
MSG_N = 10
|
||||
MSG_WIDTH = 500
|
||||
|
||||
|
||||
def connect(port):
|
||||
with urllib.request.urlopen(
|
||||
"http://127.0.0.1:%s/json/list" % port, timeout=5) as r:
|
||||
targets = json.load(r)
|
||||
pages = [t for t in targets if t.get("type") == "page"]
|
||||
if not pages:
|
||||
return None
|
||||
return websocket.create_connection(
|
||||
pages[0]["webSocketDebuggerUrl"], timeout=15)
|
||||
|
||||
|
||||
def ev1(ws, expr, await_p=False, reads=30):
|
||||
"""Runtime.evaluate that skips CDP event chatter while awaiting ours."""
|
||||
ws.send(json.dumps({
|
||||
"id": 1, "method": "Runtime.evaluate",
|
||||
"params": {"expression": expr, "returnByValue": True,
|
||||
"awaitPromise": await_p}}))
|
||||
for _ in range(reads):
|
||||
resp = json.loads(ws.recv())
|
||||
if resp.get("id") != 1:
|
||||
continue
|
||||
if "error" in resp:
|
||||
raise RuntimeError("CDP evaluate failed: %s" % resp["error"])
|
||||
res = resp.get("result", {}).get("result", {})
|
||||
return res.get("value")
|
||||
raise RuntimeError("CDP evaluate: no response")
|
||||
|
||||
|
||||
def read_current(ws):
|
||||
title = ev1(ws, "document.title") or ""
|
||||
url = ev1(ws, "window.location.href") or ""
|
||||
msgs = ev1(ws, """(() => {
|
||||
const ps = [...document.querySelectorAll('p')].slice(-%d)
|
||||
.map(p => (p.innerText||'').slice(0,%d)).filter(t => t.trim());
|
||||
return ps;
|
||||
})()""" % (MSG_N, MSG_WIDTH)) or []
|
||||
return {"title": title, "url": url,
|
||||
"messages": [{"text": t} for t in msgs]}
|
||||
|
||||
|
||||
def _wait_for(ws, expr, timeout_s=12):
|
||||
"""Poll a JS truthiness expression until true or timeout."""
|
||||
deadline = time.time() + timeout_s
|
||||
while time.time() < deadline:
|
||||
try:
|
||||
if ev1(ws, expr):
|
||||
return True
|
||||
except Exception: # noqa: BLE001
|
||||
pass
|
||||
time.sleep(1)
|
||||
return False
|
||||
|
||||
|
||||
def ensure_sidebar_open(ws):
|
||||
"""The toggle closes an open sidebar — only click when closed.
|
||||
|
||||
Detected structurally via the 'New side chat' button (text matching
|
||||
'Side chats' is unreliable: main-chat messages can contain the phrase).
|
||||
Waits for React to render the panel after opening.
|
||||
"""
|
||||
is_open = ev1(ws, """(() => {
|
||||
return !!document.querySelector('button[aria-label="New side chat"]');
|
||||
})()""")
|
||||
if not is_open:
|
||||
ev1(ws, """(() => {
|
||||
const btn = [...document.querySelectorAll('button')].find(b =>
|
||||
(b.textContent||'').includes('Open chat and side chats'));
|
||||
if (btn) btn.click();
|
||||
})()""")
|
||||
_wait_for(ws, """(() => {
|
||||
return !!document.querySelector(
|
||||
'button[aria-label="New side chat"]');
|
||||
})()""")
|
||||
# The row list renders a beat after the panel; wait for it.
|
||||
_wait_for(ws, """(() => {
|
||||
return document.querySelectorAll(
|
||||
'button[aria-label="More thread actions"]').length > 0;
|
||||
})()""", timeout_s=10)
|
||||
|
||||
|
||||
_SIDEBAR_JS = """(() => {
|
||||
const anchor = document.querySelector('button[aria-label="New side chat"]');
|
||||
if (!anchor) return 'NOANCHOR';
|
||||
let sec = anchor.parentElement;
|
||||
for (let i = 0; i < 8 && sec; i++) {
|
||||
if (sec.querySelectorAll(
|
||||
'button[aria-label="More thread actions"]').length) break;
|
||||
sec = sec.parentElement;
|
||||
}
|
||||
if (!sec) return 'NOSEC';
|
||||
return JSON.stringify({ok: true});
|
||||
})()"""
|
||||
|
||||
|
||||
def _sidebar_section(ws):
|
||||
"""Return True when the side-chat list section is addressable."""
|
||||
raw = ev1(ws, _SIDEBAR_JS)
|
||||
try:
|
||||
return json.loads(raw or "").get("ok", False)
|
||||
except (ValueError, TypeError):
|
||||
return False
|
||||
|
||||
|
||||
_LIST_JS = """(() => {
|
||||
const T = (el) => (el.innerText || '').trim();
|
||||
const TS =
|
||||
/^(\\d+[smhd]|just now|Yesterday|Today|[A-Z][a-z]{2} \\d{1,2}.*)$/;
|
||||
const Y = (el) => el.getBoundingClientRect().top;
|
||||
const h3s = [...document.querySelectorAll('h3')];
|
||||
const head = h3s.find(h => h.textContent.trim() === 'Side chats');
|
||||
if (!head) return '[]';
|
||||
// Scroll the list to top so the section's rows sit above the next
|
||||
// (sticky) header; without this, rows below the fold are missed.
|
||||
let sc = head.parentElement;
|
||||
for (let i = 0; i < 8 && sc; i++) {
|
||||
if (sc.scrollHeight > sc.clientHeight + 10) break;
|
||||
sc = sc.parentElement;
|
||||
}
|
||||
if (sc) sc.scrollTop = 0;
|
||||
const y0 = Y(head);
|
||||
const isStopText = (t) => t === 'Unread updates' || t === 'Chats' ||
|
||||
t === 'Unread chats';
|
||||
let stopY = null;
|
||||
for (const h of h3s) {
|
||||
const y = Y(h);
|
||||
if (y > y0 + 5 && (stopY === null || y < stopY)) stopY = y;
|
||||
}
|
||||
// Non-H3 section headers ("Unread updates", ...) also bound the list.
|
||||
for (const el of document.querySelectorAll('*')) {
|
||||
if (el.children.length !== 0) continue;
|
||||
if (!isStopText((el.textContent || '').trim())) continue;
|
||||
const y = Y(el);
|
||||
if (y > y0 + 5 && (stopY === null || y < stopY)) stopY = y;
|
||||
}
|
||||
const names = [];
|
||||
for (const el of document.querySelectorAll('div')) {
|
||||
const cn = (el.className || '').toString();
|
||||
if (cn.indexOf('nav-row') === -1) continue;
|
||||
const y = Y(el);
|
||||
if (y <= y0 + 5) continue;
|
||||
if (stopY !== null && y >= stopY - 5) continue;
|
||||
const m = T(el).match(/^([^\\n]+)\\n\\n(.+)$/);
|
||||
if (m && TS.test(m[2].trim()) && names.indexOf(m[1].trim()) === -1) {
|
||||
names.push(m[1].trim());
|
||||
if (names.length >= 8) break;
|
||||
}
|
||||
}
|
||||
return JSON.stringify(names);
|
||||
})()"""
|
||||
|
||||
|
||||
def _list_once(ws):
|
||||
raw = ev1(ws, _LIST_JS)
|
||||
try:
|
||||
return json.loads(raw or "[]")
|
||||
except (ValueError, TypeError):
|
||||
return []
|
||||
|
||||
|
||||
def list_sidechats(ws):
|
||||
ensure_sidebar_open(ws)
|
||||
if not _sidebar_section(ws):
|
||||
return []
|
||||
# React renders rows progressively after navigation; poll until the
|
||||
# list stabilizes instead of trusting the first paint.
|
||||
prev = None
|
||||
for _ in range(4):
|
||||
names = _list_once(ws)
|
||||
if names and names == prev:
|
||||
return names
|
||||
prev = names
|
||||
time.sleep(2)
|
||||
return prev or []
|
||||
|
||||
|
||||
def open_sidechat(ws, name):
|
||||
"""Open the side chat by clicking its row div inside the sidebar section.
|
||||
|
||||
Clicking the row (not a text search over the whole document) avoids
|
||||
hitting message text that happens to contain the chat name.
|
||||
"""
|
||||
ensure_sidebar_open(ws)
|
||||
b64 = base64.b64encode(name.encode()).decode()
|
||||
return ev1(ws, """(async () => {
|
||||
const nm = atob('%s');
|
||||
const T = (el) => (el.innerText || '').trim();
|
||||
const target = [...document.querySelectorAll('div')].find(el => {
|
||||
const cn = (el.className || '').toString();
|
||||
if (cn.indexOf('nav-row') === -1) return false;
|
||||
const m = T(el).match(/^([^\\n]+)\\n\\n(.+)$/);
|
||||
return m && m[1].trim() === nm;
|
||||
});
|
||||
if (!target) return 'NOTFOUND';
|
||||
target.click();
|
||||
await new Promise(r => setTimeout(r, 4000));
|
||||
const href = window.location.href;
|
||||
if (href.indexOf('/thread/') === -1) return 'NOTFOUND';
|
||||
return href;
|
||||
})()""" % b64, await_p=True)
|
||||
|
||||
|
||||
def navigate_main(ws):
|
||||
ws.send(json.dumps({"id": 2, "method": "Page.navigate",
|
||||
"params": {"url": "https://muse.ai/"}}))
|
||||
for _ in range(20):
|
||||
resp = json.loads(ws.recv())
|
||||
if resp.get("id") == 2:
|
||||
break
|
||||
time.sleep(6)
|
||||
|
||||
|
||||
def main():
|
||||
if len(sys.argv) < 3:
|
||||
print(json.dumps({"ok": False, "error": "usage"}))
|
||||
sys.exit(1)
|
||||
port, mode = sys.argv[1], sys.argv[2]
|
||||
name = sys.argv[3] if len(sys.argv) > 3 else ""
|
||||
try:
|
||||
ws = connect(port)
|
||||
except Exception as e: # noqa: BLE001
|
||||
print(json.dumps({"ok": False, "error": "cdp_connect_failed",
|
||||
"detail": str(e)[:120]}))
|
||||
return
|
||||
if ws is None:
|
||||
print(json.dumps({"ok": False, "error": "no_page"}))
|
||||
return
|
||||
try:
|
||||
if mode == "list":
|
||||
print(json.dumps({"ok": True,
|
||||
"sidechats": list_sidechats(ws)}))
|
||||
elif mode == "sidechat":
|
||||
href = open_sidechat(ws, name)
|
||||
if href == "NOTFOUND":
|
||||
print(json.dumps({"ok": False, "error": "sidechat_not_found",
|
||||
"name": name}))
|
||||
else:
|
||||
cur = read_current(ws)
|
||||
cur.update({"ok": True, "context": "sidechat", "name": name})
|
||||
print(json.dumps(cur))
|
||||
else:
|
||||
navigate_main(ws)
|
||||
cur = read_current(ws)
|
||||
cur.update({"ok": True, "context": "main", "name": "Main chat"})
|
||||
print(json.dumps(cur))
|
||||
except Exception as e: # noqa: BLE001
|
||||
print(json.dumps({"ok": False, "error": "probe_failed",
|
||||
"detail": str(e)[:200]}))
|
||||
finally:
|
||||
try:
|
||||
ws.close()
|
||||
except Exception: # noqa: BLE001
|
||||
pass
|
||||
|
||||
|
||||
main()
|
||||
@@ -1,129 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""chat-state-report.py — per-node chat-state tap for box.
|
||||
|
||||
Probes each NetVM node's browser via CDP (inside its netns): main chat +
|
||||
side chats, last messages. Signs and POSTs to the board chat-state ingest,
|
||||
mirroring accounts-health.sh: payload is
|
||||
<machine>\\n<ts>\\n<facts-json>, namespace "health".
|
||||
|
||||
Usage: chat-state-report.py [--no-post]
|
||||
|
||||
Cron (on bl, every 5 min):
|
||||
*/5 * * * * ~/Projects/NetVM/bin/chat-state-report.py >/dev/null 2>&1
|
||||
|
||||
Health key setup: same key as accounts-health (namespace "health",
|
||||
registered in /srv/board/health_signers for machine bl).
|
||||
"""
|
||||
import importlib.util
|
||||
import json
|
||||
import os
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
import time
|
||||
|
||||
NETVM_DIR = os.environ.get("NETVM_DIR",
|
||||
os.path.expanduser("~/Projects/NetVM"))
|
||||
CHECK = os.path.join(NETVM_DIR, "bin", "chat-state-check.py")
|
||||
MACHINE = os.environ.get("MUSE_MACHINE", "bl")
|
||||
KEY = os.environ.get("CHATSTATE_KEY",
|
||||
os.path.expanduser("~/.ssh/muse-health"))
|
||||
ENDPOINT = os.environ.get(
|
||||
"CHATSTATE_ENDPOINT",
|
||||
"https://board.muse-dev.online/api/box/chat-state/report")
|
||||
MAX_SIDECHATS = 5
|
||||
FACTS_CAP = 64 * 1024
|
||||
POST = "--no-post" not in sys.argv
|
||||
|
||||
|
||||
def load_nodes():
|
||||
spec = importlib.util.spec_from_file_location(
|
||||
"netvm_registry",
|
||||
os.path.join(NETVM_DIR, "bin", "netvm-registry.py"))
|
||||
mod = importlib.util.module_from_spec(spec)
|
||||
spec.loader.exec_module(mod)
|
||||
return mod.load() # node -> {"cdp_port": ...}
|
||||
|
||||
|
||||
def run_check(node, port, *args):
|
||||
cmd = ["sudo", "-n", "ip", "netns", "exec", "warp-%s" % node,
|
||||
"python3", CHECK, str(port)] + list(args)
|
||||
try:
|
||||
r = subprocess.run(cmd, capture_output=True, text=True, timeout=150)
|
||||
line = r.stdout.strip().splitlines()[-1]
|
||||
return json.loads(line)
|
||||
except Exception as e: # noqa: BLE001
|
||||
return {"ok": False, "error": "check_failed",
|
||||
"detail": str(e)[:120]}
|
||||
|
||||
|
||||
def main():
|
||||
nodes = load_nodes()
|
||||
chats = []
|
||||
now = int(time.time())
|
||||
for node, rec in sorted(nodes.items()):
|
||||
port = rec.get("cdp_port")
|
||||
if not port:
|
||||
continue
|
||||
base = {"node": node, "checked_at": now}
|
||||
m = run_check(node, port, "main")
|
||||
entry = dict(base)
|
||||
if m.get("ok"):
|
||||
entry.update(m)
|
||||
else:
|
||||
entry.update({"context": "main", "error": m.get("error"),
|
||||
"detail": m.get("detail", "")[:120]})
|
||||
chats.append(entry)
|
||||
continue
|
||||
chats.append(entry)
|
||||
lst = run_check(node, port, "list")
|
||||
for name in (lst.get("sidechats") or [])[:MAX_SIDECHATS]:
|
||||
s = run_check(node, port, "sidechat", name)
|
||||
sentry = dict(base)
|
||||
if s.get("ok"):
|
||||
sentry.update(s)
|
||||
else:
|
||||
sentry.update({"context": "sidechat", "name": name,
|
||||
"error": s.get("error"),
|
||||
"detail": s.get("detail", "")[:120]})
|
||||
chats.append(sentry)
|
||||
facts = {"chats": chats, "checked_at": now}
|
||||
facts_json = json.dumps(facts, separators=(",", ":"))
|
||||
if len(facts_json) > FACTS_CAP:
|
||||
# Shed message bodies, keep metadata.
|
||||
for c in chats:
|
||||
c.pop("messages", None)
|
||||
facts["truncated"] = True
|
||||
facts_json = json.dumps(facts, separators=(",", ":"))
|
||||
if not POST:
|
||||
print(json.dumps(json.loads(facts_json), indent=2))
|
||||
return
|
||||
if not os.path.exists(KEY):
|
||||
print("chat-state-report: %s missing — printing JSON, not posting"
|
||||
% KEY, file=sys.stderr)
|
||||
print(facts_json)
|
||||
return
|
||||
tmp = tempfile.mkdtemp()
|
||||
try:
|
||||
payload = os.path.join(tmp, "payload")
|
||||
with open(payload, "w") as f:
|
||||
f.write("%s\n%d\n%s" % (MACHINE, now, facts_json))
|
||||
# Fresh temp dir: no stale .sig to worry about (cf. AGENTS.md).
|
||||
subprocess.run(["ssh-keygen", "-Y", "sign", "-f", KEY,
|
||||
"-n", "health", payload],
|
||||
check=True, capture_output=True)
|
||||
with open(payload + ".sig") as f:
|
||||
sig = f.read()
|
||||
body = {"machine": MACHINE, "ts": now, "facts_json": facts_json,
|
||||
"signature": sig}
|
||||
r = subprocess.run(
|
||||
["curl", "-s", "-X", "POST", ENDPOINT,
|
||||
"-H", "Content-Type: application/json",
|
||||
"--data", json.dumps(body)],
|
||||
capture_output=True, text=True, timeout=90)
|
||||
print(r.stdout[:300])
|
||||
finally:
|
||||
subprocess.run(["rm", "-rf", tmp])
|
||||
|
||||
|
||||
main()
|
||||
@@ -1,81 +0,0 @@
|
||||
#!/bin/bash
|
||||
# chrome-error-scan.sh - scan per-profile chrome logs for concerning patterns.
|
||||
# Self-contained: scans, compares against watermark, reports only NEW matches.
|
||||
#
|
||||
# Usage: chrome-error-scan.sh [--json]
|
||||
# Default output: "profile:new_count" lines for profiles with new matches,
|
||||
# or "OK: no new errors" if clean.
|
||||
# --json: output JSON {"profile": {"total": N, "new": M}, ...}
|
||||
#
|
||||
# Watermark: /home/super/Projects/NetVM/chrome-error-watermark.json
|
||||
# Patterns: FATAL, crash, segfault, out of memory (case-insensitive)
|
||||
|
||||
LOGDIR="/home/super/Projects/NetVM"
|
||||
WATERMARK="$LOGDIR/chrome-error-watermark.json"
|
||||
|
||||
# Gather current counts per profile (grep -c prints 0 with exit 1 on no match;
|
||||
# the || true masks the exit code while preserving the "0" on stdout)
|
||||
get_count() {
|
||||
local f="$LOGDIR/chromebox-$1.log"
|
||||
if [ -f "$f" ]; then
|
||||
grep -ciE "FATAL|crash|segfault|out of memory" "$f" 2>/dev/null || true
|
||||
else
|
||||
echo 0
|
||||
fi
|
||||
}
|
||||
|
||||
MUSE_C=$(get_count muse)
|
||||
PIP_C=$(get_count pip)
|
||||
N646_C=$(get_count 646)
|
||||
OPM_C=$(get_count opm)
|
||||
|
||||
python3 - "$WATERMARK" "$MUSE_C" "$PIP_C" "$N646_C" "$OPM_C" "$1" <<'PYEOF'
|
||||
import json, sys, os
|
||||
|
||||
watermark_path = sys.argv[1]
|
||||
current = {
|
||||
"muse": int(sys.argv[2]),
|
||||
"pip": int(sys.argv[3]),
|
||||
"646": int(sys.argv[4]),
|
||||
"opm": int(sys.argv[5]),
|
||||
}
|
||||
as_json = len(sys.argv) > 6 and sys.argv[6] == "--json"
|
||||
|
||||
# Load watermark (tolerate missing/corrupt file -> treat as all-zero)
|
||||
watermark = {}
|
||||
if os.path.exists(watermark_path):
|
||||
try:
|
||||
with open(watermark_path) as f:
|
||||
data = json.load(f)
|
||||
watermark = data.get("counts", data) if isinstance(data, dict) else {}
|
||||
except (ValueError, IOError, OSError):
|
||||
watermark = {}
|
||||
|
||||
new_counts = {}
|
||||
for prof in ("muse", "pip", "646", "opm"):
|
||||
old = watermark.get(prof, 0)
|
||||
try:
|
||||
old = int(old)
|
||||
except (TypeError, ValueError):
|
||||
old = 0
|
||||
delta = current[prof] - old
|
||||
new_counts[prof] = max(delta, 0)
|
||||
|
||||
if as_json:
|
||||
out = {p: {"total": current[p], "new": new_counts[p]} for p in current}
|
||||
print(json.dumps(out))
|
||||
else:
|
||||
any_new = False
|
||||
for prof in ("muse", "pip", "646", "opm"):
|
||||
if new_counts[prof] > 0:
|
||||
print("%s:%d" % (prof, new_counts[prof]))
|
||||
any_new = True
|
||||
if not any_new:
|
||||
print("OK: no new errors")
|
||||
|
||||
# Update watermark atomically
|
||||
tmp = watermark_path + ".tmp"
|
||||
with open(tmp, "w") as f:
|
||||
json.dump({"counts": current}, f, indent=2)
|
||||
os.replace(tmp, watermark_path)
|
||||
PYEOF
|
||||
@@ -1,364 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
chromebox-gateway.py — HTTPS fallback for chromebox/DM control when SSH is down.
|
||||
|
||||
The chromeboxes (headless Chromium per NetVM node) are normally driven over
|
||||
SSH via muse-chat-api.py / dm.py, which speak CDP through the per-node
|
||||
netns relay. When SSH to bl breaks, this gateway provides a constrained
|
||||
HTTPS path to the same high-level operations.
|
||||
|
||||
CRITICAL: this does NOT expose raw CDP. Raw CDP (Runtime.evaluate,
|
||||
Page.navigate, etc.) is arbitrary code execution inside the browser with
|
||||
the agent's live session. This gateway exposes ONLY the allowlisted
|
||||
high-level operations below, each mapped to an existing audited script.
|
||||
|
||||
Usage:
|
||||
python3 chromebox-gateway.py --port 8444
|
||||
|
||||
Auth: Bearer token, reusing the shared exec per-agent token files
|
||||
(~/.exec-tokens/<agent>). The master token (~/.exec-server-token) also
|
||||
works. Identity = token filename; tokens are never logged.
|
||||
|
||||
Endpoints:
|
||||
GET /health {"status":"ok"} — no auth (load-balancer friendly)
|
||||
POST /api/v1/op {"op": "<name>", "params": {...}} — bearer auth
|
||||
|
||||
Audit: every call appended to ~/.chromebox-gateway-audit.jsonl (0600).
|
||||
"""
|
||||
import argparse
|
||||
import hmac
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import ssl
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
from http.server import HTTPServer, BaseHTTPRequestHandler
|
||||
from urllib.parse import urlparse
|
||||
|
||||
# ---------------------------------------------------------------- config
|
||||
|
||||
BIN_DIR = os.path.expanduser("~/Projects/NetVM/bin")
|
||||
TOKEN_FILE = "/home/super/.exec-server-token" # master token (shared exec token file)
|
||||
TOKEN_DIR = "/home/super/.exec-tokens" # per-agent tokens (shared exec token files)
|
||||
AUDIT_FILE = "/home/super/.chromebox-gateway-audit.jsonl"
|
||||
CERT_FILE = "/home/super/.chromebox-gateway-cert.pem"
|
||||
KEY_FILE = "/home/super/.chromebox-gateway-key.pem"
|
||||
|
||||
NODES = ("muse", "pip", "646", "opm")
|
||||
MAX_MSG = 2000 # gateway-level cap; downstream scripts enforce their own
|
||||
BACKEND_TIMEOUT = 120 # seconds per backend call
|
||||
|
||||
# Rate limit: token bucket per identity — 20 req/min sustained, burst 5.
|
||||
RATE_PER_SEC = 20.0 / 60.0
|
||||
RATE_BURST = 5
|
||||
|
||||
# ---------------------------------------------------------------- ops
|
||||
# Each op maps to an argv builder for an existing script. No shell=True,
|
||||
# ever. Params are validated before building argv.
|
||||
|
||||
def _node(params):
|
||||
node = params.get("account") or params.get("agent")
|
||||
if node not in NODES:
|
||||
raise ValueError("account/agent must be one of %s" % (",".join(NODES)))
|
||||
return node
|
||||
|
||||
def _msg(params):
|
||||
m = params.get("message", "")
|
||||
if not isinstance(m, str) or not m.strip():
|
||||
raise ValueError("message must be a non-empty string")
|
||||
if len(m) > MAX_MSG:
|
||||
raise ValueError("message exceeds %d chars" % MAX_MSG)
|
||||
return m
|
||||
|
||||
def _target(params):
|
||||
t = params.get("target", "main")
|
||||
if not isinstance(t, str) or not t or len(t) > 128:
|
||||
raise ValueError("target must be a short string")
|
||||
if not re.fullmatch(r"[A-Za-z0-9_./:-]+", t):
|
||||
raise ValueError("target has invalid characters")
|
||||
if ".." in t:
|
||||
raise ValueError("target must not contain '..'")
|
||||
return t
|
||||
|
||||
def _n(params, default=5, cap=50):
|
||||
n = params.get("n", default)
|
||||
try:
|
||||
n = int(n)
|
||||
except (TypeError, ValueError):
|
||||
raise ValueError("n must be an integer")
|
||||
if not 1 <= n <= cap:
|
||||
raise ValueError("n must be 1..%d" % cap)
|
||||
return n
|
||||
|
||||
def _tags(params):
|
||||
tags = params.get("tags", [])
|
||||
if not isinstance(tags, list):
|
||||
raise ValueError("tags must be a list")
|
||||
out = []
|
||||
for t in tags:
|
||||
if not isinstance(t, str) or len(t) > 128:
|
||||
raise ValueError("bad tag")
|
||||
if not re.fullmatch(r"[A-Za-z0-9_:=\-./]+", t):
|
||||
raise ValueError("tag has invalid characters: %r" % t[:40])
|
||||
out.append(t)
|
||||
if len(out) > 10:
|
||||
raise ValueError("too many tags (max 10)")
|
||||
return out
|
||||
|
||||
CHAT = os.path.join(BIN_DIR, "muse-chat-api.py")
|
||||
DM = os.path.join(BIN_DIR, "dm.py")
|
||||
|
||||
def op_chat_send(p):
|
||||
return [CHAT, "--account", _node(p), "send", _msg(p)]
|
||||
|
||||
def op_chat_messages(p):
|
||||
return [CHAT, "--account", _node(p), "messages", "--n", str(_n(p))]
|
||||
|
||||
def op_chat_sidechats(p):
|
||||
return [CHAT, "--account", _node(p), "sidechat", "list"]
|
||||
|
||||
def op_chat_sidechat_create(p):
|
||||
argv = [CHAT, "--account", _node(p), "sidechat", "create"]
|
||||
name = p.get("name")
|
||||
if name:
|
||||
if not isinstance(name, str) or len(name) > 80 or not re.fullmatch(r"[A-Za-z0-9 _-]+", name):
|
||||
raise ValueError("bad sidechat name")
|
||||
argv += ["--name", name]
|
||||
return argv
|
||||
|
||||
def op_chat_approvals(p):
|
||||
return [CHAT, "--account", _node(p), "approvals"]
|
||||
|
||||
def op_chat_url(p):
|
||||
return [CHAT, "--account", _node(p), "url"]
|
||||
|
||||
def op_dm_send(p):
|
||||
node = _node(p)
|
||||
to = p.get("to", node)
|
||||
if to not in NODES:
|
||||
raise ValueError("to must be one of %s" % (",".join(NODES)))
|
||||
argv = [DM, "send", "--agent", node, "--to", to,
|
||||
"--target", _target(p), _msg(p)]
|
||||
for t in _tags(p):
|
||||
argv += ["--tag", t]
|
||||
return argv
|
||||
|
||||
def op_dm_read(p):
|
||||
return [DM, "read", "--agent", _node(p),
|
||||
"--target", _target(p), "--n", str(_n(p, default=5, cap=20))]
|
||||
|
||||
# The allowlist. Adding an op here is a security decision — review accordingly.
|
||||
# Deliberately absent: wait (long-poll), upload (file ingress),
|
||||
# sidechat use (raw navigation), anything raw-CDP.
|
||||
ALLOWLIST = {
|
||||
"chat.send": ("write", op_chat_send),
|
||||
"chat.messages": ("read", op_chat_messages),
|
||||
"chat.sidechats": ("read", op_chat_sidechats),
|
||||
"chat.sidechat_create": ("write", op_chat_sidechat_create),
|
||||
"chat.approvals": ("read", op_chat_approvals),
|
||||
"chat.url": ("read", op_chat_url),
|
||||
"dm.send": ("write", op_dm_send),
|
||||
"dm.read": ("read", op_dm_read),
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------- auth
|
||||
|
||||
def _read_token_file(path):
|
||||
try:
|
||||
with open(path) as f:
|
||||
return f.read().strip()
|
||||
except OSError:
|
||||
return ""
|
||||
|
||||
def check_token(token):
|
||||
"""Return identity label or None. Tokens never leave this function."""
|
||||
if not token:
|
||||
return None
|
||||
if hmac.compare_digest(token, _read_token_file(TOKEN_FILE)):
|
||||
return "master"
|
||||
try:
|
||||
names = os.listdir(TOKEN_DIR)
|
||||
except OSError:
|
||||
return None
|
||||
for name in names:
|
||||
if not re.fullmatch(r"[A-Za-z0-9_-]+", name):
|
||||
continue
|
||||
t = _read_token_file(os.path.join(TOKEN_DIR, name))
|
||||
if t and hmac.compare_digest(token, t):
|
||||
return name
|
||||
return None
|
||||
|
||||
# ---------------------------------------------------------------- rate limit
|
||||
|
||||
_buckets = {} # identity -> [tokens, last_ts]
|
||||
|
||||
def rate_ok(identity):
|
||||
now = time.monotonic()
|
||||
tokens, last = _buckets.get(identity, (RATE_BURST, now))
|
||||
tokens = min(RATE_BURST, tokens + (now - last) * RATE_PER_SEC)
|
||||
if tokens < 1.0:
|
||||
_buckets[identity] = (tokens, now)
|
||||
return False
|
||||
_buckets[identity] = (tokens - 1.0, now)
|
||||
return True
|
||||
|
||||
# ---------------------------------------------------------------- audit
|
||||
|
||||
def audit(entry):
|
||||
entry = dict(entry)
|
||||
entry["ts"] = time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime())
|
||||
# Defense in depth: never let a full message body reach the audit file,
|
||||
# even if a caller forgets to summarize first.
|
||||
params = entry.get("params")
|
||||
if isinstance(params, dict) and "message" in params:
|
||||
params = dict(params)
|
||||
params["message_len"] = len(params.pop("message"))
|
||||
entry["params"] = params
|
||||
try:
|
||||
fd = os.open(AUDIT_FILE, os.O_WRONLY | os.O_CREAT | os.O_APPEND, 0o600)
|
||||
with os.fdopen(fd, "a") as f:
|
||||
f.write(json.dumps(entry) + "\n")
|
||||
except OSError as e:
|
||||
print("audit write failed: %s" % e, file=sys.stderr)
|
||||
|
||||
# ---------------------------------------------------------------- handler
|
||||
|
||||
class Handler(BaseHTTPRequestHandler):
|
||||
server_version = "chromebox-gateway/1.0"
|
||||
|
||||
def log_message(self, fmt, *args): # quiet; audit log is the record
|
||||
pass
|
||||
|
||||
def _json(self, code, obj):
|
||||
body = json.dumps(obj).encode()
|
||||
self.send_response(code)
|
||||
self.send_header("Content-Type", "application/json")
|
||||
self.send_header("Content-Length", str(len(body)))
|
||||
self.end_headers()
|
||||
self.wfile.write(body)
|
||||
|
||||
def do_GET(self):
|
||||
if urlparse(self.path).path == "/health":
|
||||
self._json(200, {"status": "ok", "ops": sorted(ALLOWLIST)})
|
||||
return
|
||||
self._json(404, {"error": "not_found"})
|
||||
|
||||
def do_POST(self):
|
||||
if urlparse(self.path).path != "/api/v1/op":
|
||||
self._json(404, {"error": "not_found"})
|
||||
return
|
||||
|
||||
auth = self.headers.get("Authorization", "")
|
||||
token = auth[7:] if auth.startswith("Bearer ") else ""
|
||||
identity = check_token(token)
|
||||
if not identity:
|
||||
self._json(401, {"error": "unauthorized"})
|
||||
return
|
||||
if not rate_ok(identity):
|
||||
audit({"identity": identity, "op": None, "result": "rate_limited"})
|
||||
self._json(429, {"error": "rate_limited"})
|
||||
return
|
||||
|
||||
try:
|
||||
length = int(self.headers.get("Content-Length", 0))
|
||||
except ValueError:
|
||||
length = 0
|
||||
if length > 65536:
|
||||
self._json(413, {"error": "body_too_large"})
|
||||
return
|
||||
try:
|
||||
req = json.loads(self.rfile.read(length) or b"{}")
|
||||
except (ValueError, OSError):
|
||||
self._json(400, {"error": "bad_json"})
|
||||
return
|
||||
|
||||
op = req.get("op")
|
||||
params = req.get("params") or {}
|
||||
if not isinstance(params, dict):
|
||||
self._json(400, {"error": "params_must_be_object"})
|
||||
return
|
||||
entry = ALLOWLIST.get(op)
|
||||
if not entry:
|
||||
audit({"identity": identity, "op": op, "result": "unknown_op"})
|
||||
self._json(400, {"error": "unknown_op", "allowed": sorted(ALLOWLIST)})
|
||||
return
|
||||
cls, builder = entry
|
||||
|
||||
t0 = time.monotonic()
|
||||
try:
|
||||
argv = builder(params)
|
||||
except ValueError as e:
|
||||
audit({"identity": identity, "op": op, "class": cls, "result": "bad_params",
|
||||
"detail": str(e)[:120]})
|
||||
self._json(400, {"error": "bad_params", "detail": str(e)})
|
||||
return
|
||||
|
||||
# Redacted summary for the audit log — never the full message.
|
||||
summary = {k: (v[:80] + "…" if isinstance(v, str) and len(v) > 80 else v)
|
||||
for k, v in params.items() if k != "message"}
|
||||
if "message" in params:
|
||||
summary["message_len"] = len(params["message"])
|
||||
|
||||
try:
|
||||
proc = subprocess.run(argv, capture_output=True, text=True,
|
||||
timeout=BACKEND_TIMEOUT)
|
||||
ok = proc.returncode == 0
|
||||
result = "ok" if ok else "backend_error"
|
||||
self._json(200 if ok else 502, {
|
||||
"ok": ok,
|
||||
"op": op,
|
||||
"returncode": proc.returncode,
|
||||
"stdout": proc.stdout[-8000:],
|
||||
"stderr": proc.stderr[-2000:],
|
||||
})
|
||||
except subprocess.TimeoutExpired:
|
||||
result = "timeout"
|
||||
self._json(504, {"ok": False, "op": op, "error": "backend_timeout"})
|
||||
except OSError as e:
|
||||
result = "exec_failed"
|
||||
self._json(500, {"ok": False, "op": op, "error": "exec_failed"})
|
||||
finally:
|
||||
audit({"identity": identity, "op": op, "class": cls,
|
||||
"node": params.get("account") or params.get("agent"),
|
||||
"params": summary, "result": result,
|
||||
"latency_ms": int((time.monotonic() - t0) * 1000)})
|
||||
|
||||
# ---------------------------------------------------------------- main
|
||||
|
||||
def ensure_cert():
|
||||
if os.path.exists(CERT_FILE) and os.path.exists(KEY_FILE):
|
||||
return
|
||||
print("generating self-signed cert...", file=sys.stderr)
|
||||
subprocess.run([
|
||||
"openssl", "req", "-x509", "-newkey", "rsa:2048",
|
||||
"-keyout", KEY_FILE, "-out", CERT_FILE,
|
||||
"-days", "825", "-nodes", "-subj", "/CN=chromebox-gateway",
|
||||
], check=True)
|
||||
os.chmod(KEY_FILE, 0o600)
|
||||
os.chmod(CERT_FILE, 0o600)
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("--port", type=int, default=8444)
|
||||
ap.add_argument("--bind", default="100.123.153.75",
|
||||
help="tailnet IP; use 127.0.0.1 for local-only")
|
||||
args = ap.parse_args()
|
||||
|
||||
ensure_cert()
|
||||
context = ssl.SSLContext(ssl.PROTOCOL_TLS_SERVER)
|
||||
context.load_cert_chain(CERT_FILE, KEY_FILE)
|
||||
|
||||
srv = HTTPServer((args.bind, args.port), Handler)
|
||||
srv.socket = context.wrap_socket(srv.socket, server_side=True)
|
||||
print("chromebox-gateway listening on https://%s:%d" % (args.bind, args.port),
|
||||
file=sys.stderr)
|
||||
print("ops: %s" % ", ".join(sorted(ALLOWLIST)), file=sys.stderr)
|
||||
try:
|
||||
srv.serve_forever()
|
||||
except KeyboardInterrupt:
|
||||
pass
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -1,208 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
# chromebox-watchdog.sh [profile] — keep a chrome-box profile alive and healthy.
|
||||
# Checks: 1) chromium process for the profile is running,
|
||||
# 2) CDP responds and the chat page is present.
|
||||
# If unhealthy: kill any stale chrome for the profile and relaunch headless
|
||||
# via netvm-chrome.sh (profile dir persists session/cookies — the process is
|
||||
# disposable, the state is not). Mirrors operator-646's container
|
||||
# recover-after-rebuild.sh philosophy.
|
||||
# Runs every 2 min via systemd timer chromebox-watchdog-<profile>.timer.
|
||||
set -euo pipefail
|
||||
export XDG_RUNTIME_DIR="${XDG_RUNTIME_DIR:-/run/user/$(id -u)}"
|
||||
export DBUS_SESSION_BUS_ADDRESS="${DBUS_SESSION_BUS_ADDRESS:-unix:path=${XDG_RUNTIME_DIR}/bus}"
|
||||
# Prevent overlapping runs: the timer fires every 2 min but a relaunch
|
||||
# (kill + sleep 25 + chrome startup + page load) can exceed that, and two
|
||||
# concurrent runs kill each others chrome (observed 2026-10-03: pip flapped
|
||||
# with simultaneous "relaunch OK" and "relaunch FAILED").
|
||||
LOCK="/tmp/chromebox-watchdog-${1:-pip}.lock"
|
||||
exec 9>"$LOCK"
|
||||
if ! flock -n 9; then
|
||||
echo "[$(date -u +%FT%TZ)] [$1] another watchdog run in progress, skipping" >&2
|
||||
exit 0
|
||||
fi
|
||||
|
||||
PROFILE="${1:-pip}"
|
||||
NETVM_BIN="/home/super/Projects/NetVM/bin"
|
||||
LOG="/home/super/Projects/NetVM/chromebox-watchdog.log"
|
||||
CHROME_LOG="/home/super/Projects/NetVM/chromebox-${PROFILE}.log"
|
||||
|
||||
# Rotate a log file past 10MB (keep one generation)
|
||||
rotate_log() {
|
||||
local f="$1"
|
||||
[ -f "$f" ] || return 0
|
||||
local sz
|
||||
sz=$(stat -c%s "$f" 2>/dev/null || echo 0)
|
||||
if [ "$sz" -gt 10485760 ]; then
|
||||
mv -f "$f" "$f.1"
|
||||
echo "[$(date -u +%FT%TZ)] [$PROFILE] log rotated" > "$f"
|
||||
fi
|
||||
}
|
||||
rotate_log "$LOG"
|
||||
# CHROME_LOG rotation happens after PROFILE is set (see below)
|
||||
|
||||
case "$PROFILE" in
|
||||
muse) CDP_PORT=9410 ;;
|
||||
pip) CDP_PORT=9420 ;;
|
||||
646) CDP_PORT=9430 ;;
|
||||
opm) CDP_PORT=9440 ;;
|
||||
*) echo "unknown profile: $PROFILE" >&2; exit 1 ;;
|
||||
esac
|
||||
rotate_log "$CHROME_LOG"
|
||||
|
||||
log() { echo "[$(date -u +%FT%TZ)] [$PROFILE] $*" | tee -a "$LOG"; }
|
||||
|
||||
cdp_list() {
|
||||
"$NETVM_BIN/netvm-exec.sh" "$PROFILE" -- curl -s -m 8 "http://127.0.0.1:$CDP_PORT/json/list" 2>/dev/null
|
||||
}
|
||||
|
||||
# HEALTH_FAIL_REASON is set by healthy() on failure: which stage broke.
|
||||
HEALTH_FAIL_REASON=""
|
||||
healthy() {
|
||||
HEALTH_FAIL_REASON=""
|
||||
pgrep -f "chromium.*profiles/${PROFILE}/" >/dev/null 2>&1 \
|
||||
|| { HEALTH_FAIL_REASON="no chromium process for profile"; return 1; }
|
||||
local list
|
||||
list="$(cdp_list)" \
|
||||
|| { HEALTH_FAIL_REASON="CDP unreachable on :$CDP_PORT"; return 1; }
|
||||
echo "$list" | grep -q '"type": "page"' \
|
||||
|| { HEALTH_FAIL_REASON="CDP up but no page target in list"; return 1; }
|
||||
echo "$list" | grep -E -q '"url": "https://muse\.ai' \
|
||||
|| { HEALTH_FAIL_REASON="CDP up but not on muse.ai"; return 1; }
|
||||
warp_egress_healthy || return 1
|
||||
return 0
|
||||
}
|
||||
|
||||
# Stage 5: Warp egress health (2026-10-04). A partitioned browser (tunnel down,
|
||||
# CDP green) passes stages 1-4 while being unable to reach muse.ai. Check the
|
||||
# WireGuard handshake age and probe egress from inside the node's netns.
|
||||
# Sets HEALTH_FAIL_REASON with a distinct "warp egress down" prefix so
|
||||
# root-cause analysis can distinguish partitions from Chromium crashes.
|
||||
warp_egress_healthy() {
|
||||
local wg_out hs_line age_s code
|
||||
wg_out="$(sudo -n ip netns exec "warp-${PROFILE}" wg show 2>/dev/null)" \
|
||||
|| { HEALTH_FAIL_REASON="warp egress down (wg show failed in warp-${PROFILE})"; return 1; }
|
||||
hs_line="$(printf '%s\n' "$wg_out" | grep -i "latest handshake" | head -1)"
|
||||
[ -n "$hs_line" ] \
|
||||
|| { HEALTH_FAIL_REASON="warp egress down (no WireGuard handshake in warp-${PROFILE})"; return 1; }
|
||||
# "latest handshake: 1 minute, 41 seconds ago" -> total seconds
|
||||
age_s="$(printf '%s\n' "$hs_line" | python3 -c '
|
||||
import sys, re
|
||||
s = sys.stdin.read()
|
||||
m = re.search(r"(\d+)\s*hour", s); h = int(m.group(1)) if m else 0
|
||||
m = re.search(r"(\d+)\s*minute", s); mi = int(m.group(1)) if m else 0
|
||||
m = re.search(r"(\d+)\s*second", s); se = int(m.group(1)) if m else 0
|
||||
print(h*3600 + mi*60 + se)
|
||||
' 2>/dev/null)"
|
||||
{ [ -n "$age_s" ] && [ "$age_s" -ge 0 ]; } 2>/dev/null \
|
||||
|| { HEALTH_FAIL_REASON="warp egress down (unparseable handshake: $hs_line)"; return 1; }
|
||||
[ "$age_s" -le 180 ] \
|
||||
|| { HEALTH_FAIL_REASON="warp egress down (handshake ${age_s}s old in warp-${PROFILE})"; return 1; }
|
||||
code="$(sudo -n ip netns exec "warp-${PROFILE}" curl -m 5 -s -o /dev/null -w "%{http_code}" "https://muse.ai/" 2>/dev/null)" \
|
||||
|| { HEALTH_FAIL_REASON="warp egress down (egress probe curl failed in warp-${PROFILE})"; return 1; }
|
||||
# Any 2xx/3xx means we reached muse.ai infra ("/" 307-redirects to the
|
||||
# auth flow). The probe tests egress connectivity, not page content.
|
||||
case "$code" in
|
||||
2*|3*) return 0 ;;
|
||||
*) HEALTH_FAIL_REASON="warp egress down (egress probe HTTP $code in warp-${PROFILE})"; return 1 ;;
|
||||
esac
|
||||
}
|
||||
|
||||
if healthy; then
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# Relaunch-loop guard (2026-10-04): if the main browser for this profile
|
||||
# launched <2 min ago it's probably still starting up (CDP not yet bound).
|
||||
# Relaunching now would kill a healthy-but-slow cold start via netvm-chrome.sh
|
||||
# and reset the startup clock every cycle (observed: opm piled up 5 chromiums
|
||||
# because the guard only protected the kill step, not the relaunch). Skip the
|
||||
# entire cycle instead.
|
||||
# NOTE: match only the main browser process (--remote-debugging-port present,
|
||||
# no --type= flag). Renderer/gpu children (--type=renderer etc.) start later
|
||||
# than the main process and must not satisfy this check.
|
||||
recent_pid=""
|
||||
for _pid in $(pgrep -f "chromium.*--remote-debugging-port=${CDP_PORT}([[:space:]]|$)" 2>/dev/null); do
|
||||
# Skip child processes (renderer, gpu, etc.) — only the main browser counts
|
||||
if ps -o args= -p "$_pid" 2>/dev/null | grep -q -- "--type="; then
|
||||
continue
|
||||
fi
|
||||
_start=$(date -d "$(ps -o lstart= -p "$_pid" 2>/dev/null)" +%s 2>/dev/null || echo 0)
|
||||
_now=$(date +%s)
|
||||
if [ $(( _now - _start )) -lt 120 ] && [ "$_start" -gt 0 ]; then
|
||||
recent_pid="$_pid"
|
||||
break
|
||||
fi
|
||||
done
|
||||
if [ -n "$recent_pid" ]; then
|
||||
log "browser launched recently (pid $recent_pid), skipping relaunch (probably still starting)"
|
||||
exit 0
|
||||
fi
|
||||
# Warp-partition recovery (2026-10-04): relaunching Chrome cannot fix a dead
|
||||
# Warp tunnel — the new browser would fail the same egress check and the
|
||||
# watchdog would loop. If the failure is warp-egress, restart the tunnel first
|
||||
# (netvm-node-up.sh is idempotent); only fall through to the Chrome relaunch
|
||||
# if the tunnel does not recover.
|
||||
case "$HEALTH_FAIL_REASON" in
|
||||
"warp egress down"*)
|
||||
log "warp partition detected ($HEALTH_FAIL_REASON), restarting tunnel via netvm-node-up.sh"
|
||||
sudo -n "$NETVM_BIN/netvm-node-up.sh" "$PROFILE" >>"$LOG" 2>&1 || true
|
||||
sleep 5
|
||||
if healthy; then
|
||||
log "tunnel restart recovered warp egress, chrome relaunch not needed"
|
||||
exit 0
|
||||
fi
|
||||
log "tunnel restart did not recover egress ($HEALTH_FAIL_REASON), proceeding with chrome relaunch"
|
||||
;;
|
||||
esac
|
||||
log "unhealthy ($HEALTH_FAIL_REASON), relaunching chromebox"
|
||||
# bracket trick so pkill never matches its own command line
|
||||
pat="profiles/${PROFILE:0:${#PROFILE}-1}[${PROFILE: -1}]/"
|
||||
pkill -f "chromium.*$pat" 2>/dev/null || true
|
||||
sleep 3
|
||||
# Clean up stale singleton symlinks that break subsequent browser startup
|
||||
rm -f "/home/super/.local/share/chrome-box/profiles/${PROFILE}/home/.config/chromium/SingletonLock" \
|
||||
"/home/super/.local/share/chrome-box/profiles/${PROFILE}/home/.config/chromium/SingletonSocket" \
|
||||
"/home/super/.local/share/chrome-box/profiles/${PROFILE}/home/.config/chromium/SingletonCookie" 2>/dev/null || true
|
||||
for _sc in $(systemctl --user list-units --type=scope --plain --no-legend 2>/dev/null | awk '{print $1}' | grep -E "^netvm-chrome-${PROFILE}-"); do
|
||||
systemctl --user stop "$_sc" 2>/dev/null || true
|
||||
done
|
||||
|
||||
# Relaunch in its own systemd scope so it survives this oneshot run.
|
||||
# nohup/setsid do NOT escape: this timer's service uses KillMode=control-group
|
||||
# and systemd SIGKILLs everything in the cgroup at teardown (observed
|
||||
# 2026-10-03: every relaunch "recovered" then died seconds later). A transient
|
||||
# scope escapes the service cgroup; the scoped process inherits these fds so
|
||||
# the log redirect below still captures chromium's output.
|
||||
# NOTE: systemd-run --scope WAITS for the scope's processes (even --no-block,
|
||||
# verified 2026-10-03), so it must be backgrounded — the scope is an
|
||||
# independent unit and outlives the wrapper.
|
||||
# NOTE: --cdp-port is pinned explicitly. netvm-chrome.sh defaults to a
|
||||
# hash-derived port (9222+...) which will NOT match the registry port that
|
||||
# muse-chat-api.py uses — a relaunch on the wrong port looks healthy to the
|
||||
# launcher but is unreachable to the API (observed 2026-10-03: pip relaunched
|
||||
# on 9278 instead of 9420, watchdog looped on "relaunch FAILED").
|
||||
# 9>&-: do NOT let the backgrounded launcher inherit the watchdog lock fd.
|
||||
# Inherited flock fds wedge the lock forever (observed 2026-10-04: opm/muse
|
||||
# launchers held their profile lock for 100+ min, every later watchdog run
|
||||
# skipped as "another run in progress" — the watchdog was silently dead).
|
||||
systemd-run --user --scope --unit="netvm-chrome-${PROFILE}-$(date +%s)" \
|
||||
"$NETVM_BIN/netvm-chrome.sh" --headless --cdp-port "$CDP_PORT" "$PROFILE" "https://muse.ai" \
|
||||
>>"$CHROME_LOG" 2>&1 < /dev/null 9>&- &
|
||||
# Retry with backoff (2026-10-04): cold starts (fresh egress IP, Cloudflare
|
||||
# handshake) can take >25s for the page title to appear. A single check after
|
||||
# 25s kills working-but-slow browsers. Try up to 4 times, 15s apart (~60s
|
||||
# window), logging each attempt. Only declare FAILED if all attempts fail.
|
||||
_relaunch_ok=0
|
||||
for _attempt in 1 2 3 4; do
|
||||
sleep 15
|
||||
if healthy; then
|
||||
log "relaunch OK (attempt $_attempt)"
|
||||
_relaunch_ok=1
|
||||
break
|
||||
fi
|
||||
log "relaunch attempt $_attempt not healthy yet ($HEALTH_FAIL_REASON)"
|
||||
done
|
||||
if [ "$_relaunch_ok" -ne 1 ]; then
|
||||
log "relaunch FAILED ($HEALTH_FAIL_REASON) \u2014 needs operator attention"
|
||||
exit 1
|
||||
fi
|
||||
@@ -1,416 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""cred-client.py — Agent API client & CLI for cred.muse-dev.online.
|
||||
|
||||
Provides programmatic and CLI access for agents and operators to:
|
||||
- initiate: start onboarding for a client email/node
|
||||
- submit-otp: submit verification code transiently
|
||||
- status: query onboarding & vitality state for a node
|
||||
- list: list fleet accounts, emails, and statuses
|
||||
|
||||
Authentication:
|
||||
- Machine identity signature via ~/.ssh/muse-health (machine bl)
|
||||
- Or Bearer token from CRED_TOKEN / OPERATOR_TOKEN environment variable.
|
||||
|
||||
Usage:
|
||||
cred-client.py initiate --node <node> --email <email> [--service muse] [--account-name <name>]
|
||||
cred-client.py submit-otp --node <node> --otp <code> [--email <email>]
|
||||
cred-client.py status --node <node> [--json]
|
||||
cred-client.py list [--json]
|
||||
"""
|
||||
import argparse
|
||||
import datetime
|
||||
import importlib.util
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
import urllib.error
|
||||
import urllib.request
|
||||
|
||||
NETVM_DIR = os.environ.get("NETVM_DIR", "/home/super/Projects/NetVM")
|
||||
KEY_DEFAULT = os.path.expanduser("~/.ssh/muse-health")
|
||||
CRED_API_DEFAULT = os.environ.get("CRED_API_URL", "https://cred.muse-dev.online/api/cred")
|
||||
|
||||
|
||||
def load_registry():
|
||||
path = os.path.join(NETVM_DIR, "bin", "netvm-registry.py")
|
||||
try:
|
||||
spec = importlib.util.spec_from_file_location("netvm_registry", path)
|
||||
mod = importlib.util.module_from_spec(spec)
|
||||
spec.loader.exec_module(mod)
|
||||
return mod
|
||||
except Exception:
|
||||
return None
|
||||
|
||||
|
||||
def get_accounts():
|
||||
"""Parse ACCOUNTS.md into a structured list of accounts."""
|
||||
path = os.path.join(NETVM_DIR, "ACCOUNTS.md")
|
||||
accounts = []
|
||||
if not os.path.exists(path):
|
||||
return accounts
|
||||
with open(path) as f:
|
||||
for line in f:
|
||||
line = line.strip()
|
||||
if not line.startswith("|") or line.startswith("| agent") or line.startswith("|-------"):
|
||||
continue
|
||||
cols = [c.strip() for c in line.strip("|").split("|")]
|
||||
if len(cols) >= 9:
|
||||
accounts.append({
|
||||
"agent": cols[0],
|
||||
"node": cols[1],
|
||||
"profile": cols[2],
|
||||
"login_type": cols[3],
|
||||
"meta_label": cols[4],
|
||||
"email": cols[5],
|
||||
"phone_otp": cols[6],
|
||||
"instagram_linked": cols[7],
|
||||
"status": cols[8],
|
||||
"egress_ip": cols[9] if len(cols) > 9 else "",
|
||||
"cdp_port": cols[10] if len(cols) > 10 else "",
|
||||
"display_name": cols[11] if len(cols) > 11 else "",
|
||||
"notes": cols[12] if len(cols) > 12 else ""
|
||||
})
|
||||
return accounts
|
||||
|
||||
|
||||
def sign_payload(payload_bytes, key_path=KEY_DEFAULT, namespace="cred"):
|
||||
"""Sign payload using SSH ed25519 key (zero secret across wire)."""
|
||||
if not os.path.exists(key_path):
|
||||
return None
|
||||
tmp = tempfile.mkdtemp()
|
||||
try:
|
||||
data_file = os.path.join(tmp, "data")
|
||||
with open(data_file, "wb") as f:
|
||||
f.write(payload_bytes)
|
||||
r = subprocess.run(
|
||||
["ssh-keygen", "-Y", "sign", "-f", key_path, "-n", namespace, data_file],
|
||||
capture_output=True, text=True
|
||||
)
|
||||
if r.returncode != 0:
|
||||
return None
|
||||
with open(data_file + ".sig") as f:
|
||||
return f.read().strip()
|
||||
finally:
|
||||
subprocess.run(["rm", "-rf", tmp], capture_output=True)
|
||||
|
||||
|
||||
class CredClient:
|
||||
def __init__(self, api_url=CRED_API_DEFAULT, key_path=KEY_DEFAULT):
|
||||
self.api_url = api_url.rstrip("/")
|
||||
self.key_path = key_path
|
||||
self.token = os.environ.get("CRED_TOKEN") or os.environ.get("OPERATOR_TOKEN")
|
||||
|
||||
def run_local_driver(self, node, step, email, otp=None, account_name=None):
|
||||
"""Execute local onboard-driver.py inside the node's netns."""
|
||||
exec_script = os.path.join(NETVM_DIR, "bin", "netvm-exec.sh")
|
||||
driver_script = os.path.join(NETVM_DIR, "bin", "onboard-driver.py")
|
||||
cmd = [exec_script, node, "--", sys.executable, driver_script,
|
||||
"--node", node, "--service", "muse", "--id-type", "email", "--step", step]
|
||||
if account_name:
|
||||
cmd += ["--account-name", account_name]
|
||||
|
||||
input_data = email + "\n"
|
||||
if otp:
|
||||
input_data += str(otp) + "\n"
|
||||
|
||||
p = subprocess.run(cmd, input=input_data, capture_output=True, text=True, timeout=240)
|
||||
return p.returncode, p.stdout.strip(), p.stderr.strip()
|
||||
|
||||
def initiate(self, node, email, service="muse", account_name=None):
|
||||
"""Initiate client onboarding."""
|
||||
ret, stdout, stderr = self.run_local_driver(node, "initiate", email, account_name=account_name)
|
||||
if ret == 0:
|
||||
return {"status": "active", "node": node, "email": email, "message": "Already authenticated and session active."}
|
||||
elif ret == 2:
|
||||
return {"status": "awaiting_otp", "node": node, "email": email, "message": "OTP verification code sent. Awaiting input."}
|
||||
elif ret == 3:
|
||||
return {"status": "needs_human", "node": node, "email": email, "message": "Multiple accounts match identifier. Manual selection required."}
|
||||
else:
|
||||
return {"status": "error", "node": node, "email": email, "code": ret, "detail": stderr or stdout}
|
||||
|
||||
def submit_otp(self, node, otp, email=None, service="muse"):
|
||||
"""Submit transient verification code."""
|
||||
if not email:
|
||||
# Look up email from ACCOUNTS.md
|
||||
for acct in get_accounts():
|
||||
if acct["node"] == node and acct["email"] not in ("-", ""):
|
||||
email = acct["email"]
|
||||
break
|
||||
if not email:
|
||||
email = "unknown"
|
||||
|
||||
ret, stdout, stderr = self.run_local_driver(node, "submit", email, otp=otp)
|
||||
if ret == 0:
|
||||
return {"status": "active", "node": node, "email": email, "message": "Authentication successful. Chat session active."}
|
||||
elif ret == 3:
|
||||
return {"status": "needs_human", "node": node, "email": email, "message": "Multiple accounts match identifier."}
|
||||
else:
|
||||
return {"status": "error", "node": node, "email": email, "code": ret, "detail": stderr or stdout}
|
||||
|
||||
def status(self, node):
|
||||
"""Check status of a node."""
|
||||
accounts = get_accounts()
|
||||
target = next((a for a in accounts if a["node"] == node), None)
|
||||
port = None
|
||||
reg = load_registry()
|
||||
if reg:
|
||||
port = reg.port_for(node)
|
||||
|
||||
# Vitality check
|
||||
alive = False
|
||||
detail = ""
|
||||
if port:
|
||||
try:
|
||||
checker = os.path.join(NETVM_DIR, "bin", "accounts-health.py")
|
||||
cmd = ["sudo", "-n", "ip", "netns", "exec", f"warp-{node}", sys.executable, checker, str(port)]
|
||||
r = subprocess.run(cmd, capture_output=True, text=True, timeout=10)
|
||||
if r.returncode == 0:
|
||||
data = json.loads(r.stdout.strip().splitlines()[-1])
|
||||
alive = data.get("session_alive", False)
|
||||
detail = data.get("detail", "")
|
||||
except Exception as e:
|
||||
detail = str(e)
|
||||
|
||||
return {
|
||||
"node": node,
|
||||
"status": target["status"] if target else "unregistered",
|
||||
"email": target["email"] if target else "-",
|
||||
"cdp_port": port,
|
||||
"session_alive": alive,
|
||||
"detail": detail
|
||||
}
|
||||
|
||||
def list_all(self):
|
||||
"""List all accounts and live statuses."""
|
||||
res = []
|
||||
for a in get_accounts():
|
||||
st = self.status(a["node"])
|
||||
res.append(st)
|
||||
return res
|
||||
|
||||
|
||||
def notify_operator(self, subject, body, to_email="defnotabotnet@gmail.com"):
|
||||
"""Send notification to operator via local MTA (msmtp or mail)."""
|
||||
sent = False
|
||||
err = None
|
||||
# Try msmtp first
|
||||
try:
|
||||
p = subprocess.Popen(["msmtp", to_email], stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True)
|
||||
msg = f"Subject: {subject}\nTo: {to_email}\nFrom: NetVM Automation <root@bl>\n\n{body}\n"
|
||||
stdout, stderr = p.communicate(input=msg)
|
||||
if p.returncode == 0:
|
||||
sent = True
|
||||
else:
|
||||
err = stderr.strip() or stdout.strip()
|
||||
except Exception as e:
|
||||
err = str(e)
|
||||
|
||||
if not sent:
|
||||
# Fallback to mail / s-nail
|
||||
try:
|
||||
p = subprocess.Popen(["mail", "-s", subject, to_email], stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True)
|
||||
stdout, stderr = p.communicate(input=body)
|
||||
if p.returncode == 0:
|
||||
sent = True
|
||||
else:
|
||||
err = stderr.strip() or stdout.strip()
|
||||
except Exception as e:
|
||||
err = str(e)
|
||||
|
||||
return {"sent": sent, "recipient": to_email, "error": err if not sent else None}
|
||||
|
||||
def link_instagram(self, node, notify=False):
|
||||
"""Fetch the Meta Accounts Center OAuth URL and provide one-tap Tailscale link."""
|
||||
portal_script = os.path.join(NETVM_DIR, "bin", "tailscale-verify-portal.py")
|
||||
ts_ip = "100.123.153.75"
|
||||
ts_dns = "bl.tailfb5960.ts.net"
|
||||
try:
|
||||
import importlib.util
|
||||
spec = importlib.util.spec_from_file_location("portal", portal_script)
|
||||
portal_mod = importlib.util.module_from_spec(spec)
|
||||
spec.loader.exec_module(portal_mod)
|
||||
ts_ip = portal_mod.get_tailscale_ip()
|
||||
ts_dns = portal_mod.get_tailscale_dns()
|
||||
ig_data = portal_mod.fetch_node_ig_link(node)
|
||||
except Exception as e:
|
||||
ig_data = {"error": str(e)}
|
||||
|
||||
direct_url = ig_data.get("url") if isinstance(ig_data, dict) else None
|
||||
portal_url = f"http://{ts_dns}:8765/verify/{node}"
|
||||
portal_ip_url = f"http://{ts_ip}:8765/verify/{node}"
|
||||
|
||||
email_result = None
|
||||
if notify and direct_url:
|
||||
subject = f"[NetVM Action Required] Link Instagram for Node '{node}'"
|
||||
body = (
|
||||
f"Node '{node}' is waiting at the Muse age verification gate.\n\n"
|
||||
f"Tap the Tailscale portal link from your device to approve:\n"
|
||||
f" {portal_url}\n"
|
||||
f" (or {portal_ip_url})\n\n"
|
||||
f"Direct OAuth URL:\n"
|
||||
f" {direct_url}\n\n"
|
||||
f"After approving in Instagram, NetVM will automatically transition node '{node}' to active."
|
||||
)
|
||||
email_result = self.notify_operator(subject, body)
|
||||
|
||||
return {
|
||||
"node": node,
|
||||
"status": "needs_verification",
|
||||
"portal_url": portal_url,
|
||||
"portal_ip_url": portal_ip_url,
|
||||
"direct_oauth_url": direct_url,
|
||||
"error": ig_data.get("error") if isinstance(ig_data, dict) else None,
|
||||
"email_notified": email_result.get("sent") if email_result else False
|
||||
}
|
||||
|
||||
def audit_meta(self, node):
|
||||
"""Query Meta Accounts Center for linked profiles and security status."""
|
||||
meta_script = os.path.join(NETVM_DIR, "bin", "meta-acct.py")
|
||||
cmd = ["sudo", "-n", "ip", "netns", "exec", f"warp-{node}", sys.executable, meta_script, "list-linked", node]
|
||||
try:
|
||||
res = subprocess.run(cmd, capture_output=True, text=True, timeout=15)
|
||||
out = res.stdout.strip()
|
||||
if out:
|
||||
# Find outermost JSON object
|
||||
start = out.find("{")
|
||||
end = out.rfind("}")
|
||||
if start >= 0 and end >= start:
|
||||
return json.loads(out[start:end+1])
|
||||
return json.loads(out)
|
||||
return {"error": res.stderr.strip() or "No output from meta-acct.py"}
|
||||
except Exception as e:
|
||||
return {"error": str(e)}
|
||||
|
||||
|
||||
def main():
|
||||
common = argparse.ArgumentParser(add_help=False)
|
||||
common.add_argument("--json", action="store_true", help="Output JSON response")
|
||||
|
||||
p = argparse.ArgumentParser(description="cred-client: Agent API client for cred onboarding", parents=[common])
|
||||
sub = p.add_subparsers(dest="command", required=True)
|
||||
|
||||
# initiate
|
||||
p_init = sub.add_parser("initiate", parents=[common], help="Initiate onboarding flow for a node")
|
||||
p_init.add_argument("--node", required=True, help="Node label (e.g. opm, pip, client1)")
|
||||
p_init.add_argument("--email", required=True, help="Client login email")
|
||||
p_init.add_argument("--service", default="muse", help="Service name (default: muse)")
|
||||
p_init.add_argument("--account-name", default=None, help="Display name hint for multi-account selector")
|
||||
|
||||
# submit-otp
|
||||
p_otp = sub.add_parser("submit-otp", parents=[common], help="Submit transient OTP verification code")
|
||||
p_otp.add_argument("--node", required=True, help="Node label")
|
||||
p_otp.add_argument("--otp", required=True, help="6-digit verification code")
|
||||
p_otp.add_argument("--email", default=None, help="Client email (optional, auto-detected from registry)")
|
||||
|
||||
# status
|
||||
p_stat = sub.add_parser("status", parents=[common], help="Query onboarding and session status for a node")
|
||||
p_stat.add_argument("--node", required=True, help="Node label")
|
||||
|
||||
# link-instagram
|
||||
p_link = sub.add_parser("link-instagram", parents=[common], help="Generate Tailscale one-tap portal & OAuth link for age verification")
|
||||
p_link.add_argument("--node", required=True, help="Node label")
|
||||
p_link.add_argument("--notify", action="store_true", help="Send email alert to operator via local MTA")
|
||||
|
||||
# meta-audit
|
||||
p_meta = sub.add_parser("meta-audit", parents=[common], help="Query Meta Accounts Center for linked profiles and security status")
|
||||
p_meta.add_argument("--node", required=True, help="Node label")
|
||||
|
||||
# list
|
||||
sub.add_parser("list", parents=[common], help="List all registered nodes and vitality statuses")
|
||||
|
||||
args = p.parse_args()
|
||||
client = CredClient()
|
||||
|
||||
if args.command == "meta-audit":
|
||||
res = client.audit_meta(args.node)
|
||||
if args.json:
|
||||
print(json.dumps(res, indent=2))
|
||||
else:
|
||||
print(f"\n=== META ACCOUNTS CENTER AUDIT: {args.node} ===")
|
||||
if res.get("error"):
|
||||
print(f"Error: {res['error']}", file=sys.stderr)
|
||||
sys.exit(1)
|
||||
print(f"Meta Account Email: {res.get('email') or '(none / phone-only)'}")
|
||||
profiles = res.get("profiles", [])
|
||||
print(f"Linked Profiles ({len(profiles)}):")
|
||||
for p_info in profiles:
|
||||
print(f" - [{p_info.get('type')}] {p_info.get('name')}")
|
||||
sys.exit(0)
|
||||
|
||||
if args.command == "link-instagram":
|
||||
res = client.link_instagram(args.node, notify=args.notify)
|
||||
if args.json:
|
||||
print(json.dumps(res, indent=2))
|
||||
else:
|
||||
print(f"\n=== INSTAGRAM LINKING: {args.node} ===")
|
||||
if res.get("error"):
|
||||
print(f"Error: {res['error']}", file=sys.stderr)
|
||||
sys.exit(1)
|
||||
print(f"Tailscale Portal: {res['portal_url']}")
|
||||
print(f"Direct IP Portal: {res['portal_ip_url']}")
|
||||
print(f"OAuth URL: {res['direct_oauth_url'][:80]}...")
|
||||
if args.notify:
|
||||
print(f"Email Notified: {'YES' if res['email_notified'] else 'FAILED'}")
|
||||
print("\nTap either Tailscale link from your phone/browser to complete Meta age verification.")
|
||||
sys.exit(0)
|
||||
|
||||
if args.command == "initiate":
|
||||
res = client.initiate(args.node, args.email, service=args.service, account_name=args.account_name)
|
||||
if args.json:
|
||||
print(json.dumps(res, indent=2))
|
||||
else:
|
||||
st = res.get("status")
|
||||
if st == "awaiting_otp":
|
||||
print(f"[APPROVAL_NEEDED] Code sent to [redacted] for node '{args.node}'.")
|
||||
print(f"Submit OTP with: super cred submit-otp --node {args.node} --otp <code>")
|
||||
sys.exit(2)
|
||||
elif st == "active":
|
||||
print(f"[SUCCESS] Node '{args.node}' is already authenticated and active.")
|
||||
sys.exit(0)
|
||||
else:
|
||||
print(f"[{st.upper()}] {res.get('message') or res.get('detail')}")
|
||||
sys.exit(res.get("code", 1))
|
||||
|
||||
elif args.command == "submit-otp":
|
||||
res = client.submit_otp(args.node, args.otp, email=args.email)
|
||||
if args.json:
|
||||
print(json.dumps(res, indent=2))
|
||||
else:
|
||||
st = res.get("status")
|
||||
if st == "active":
|
||||
print(f"[SUCCESS] Node '{args.node}' successfully signed in!")
|
||||
sys.exit(0)
|
||||
else:
|
||||
print(f"[{st.upper()}] {res.get('message') or res.get('detail')}")
|
||||
sys.exit(res.get("code", 1))
|
||||
|
||||
elif args.command == "status":
|
||||
res = client.status(args.node)
|
||||
if args.json:
|
||||
print(json.dumps(res, indent=2))
|
||||
else:
|
||||
print(f"Node: {res['node']}")
|
||||
print(f"Email: {res['email']}")
|
||||
print(f"Status: {res['status']}")
|
||||
print(f"CDP Port: {res['cdp_port'] or '-'}")
|
||||
print(f"Session Alive: {'YES' if res['session_alive'] else 'NO'}")
|
||||
if res["detail"]:
|
||||
print(f"Detail: {res['detail']}")
|
||||
|
||||
elif args.command == "list":
|
||||
items = client.list_all()
|
||||
if args.json:
|
||||
print(json.dumps(items, indent=2))
|
||||
else:
|
||||
print(f"{'NODE':<10} {'STATUS':<14} {'CDP':<6} {'ALIVE':<7} {'EMAIL'}")
|
||||
print("-" * 65)
|
||||
for it in items:
|
||||
alive_str = "YES" if it["session_alive"] else "NO"
|
||||
print(f"{it['node']:<10} {it['status']:<14} {str(it['cdp_port'] or '-'):<6} {alive_str:<7} {it['email']}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -1 +0,0 @@
|
||||
cred-client.py
|
||||
@@ -1,178 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
crypt-server.py — Public cryptographic attestation and key directory for NetVM.
|
||||
|
||||
Serves https://crypt.muse-dev.online/ (via cloudflared / reverse proxy).
|
||||
Endpoints:
|
||||
GET / -> Service directory / health JSON
|
||||
GET /health -> Health check
|
||||
GET /keys/allowed_signers -> OpenSSH allowed_signers formatted file
|
||||
GET /keys/{identity}.pub -> Individual public key
|
||||
GET /proofs -> List known proof hashes / work order attestations
|
||||
GET /proofs/{id} -> Retrieve proof envelope and SSH signature
|
||||
POST /proofs -> Submit / register a signed proof record
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import glob
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import ssl
|
||||
import sys
|
||||
from http.server import HTTPServer, BaseHTTPRequestHandler
|
||||
|
||||
REPO_DIR = "/home/super/Projects/NetVM"
|
||||
SIGNERS_DIR = os.path.join(REPO_DIR, "dm-signers")
|
||||
PROOFS_DIR = os.path.join(REPO_DIR, "var", "proofs")
|
||||
|
||||
os.makedirs(PROOFS_DIR, exist_ok=True)
|
||||
|
||||
class CryptHandler(BaseHTTPRequestHandler):
|
||||
server_version = "crypt-attestation/1.0"
|
||||
|
||||
def log_message(self, format, *args):
|
||||
sys.stderr.write(f"crypt-server: {self.client_address[0]} - {format % args}\n")
|
||||
|
||||
def _send(self, code, content, content_type="application/json"):
|
||||
if isinstance(content, (dict, list)):
|
||||
body = json.dumps(content, indent=2).encode("utf-8")
|
||||
elif isinstance(content, str):
|
||||
body = content.encode("utf-8")
|
||||
else:
|
||||
body = bytes(content)
|
||||
|
||||
self.send_response(code)
|
||||
self.send_header("Content-Type", content_type)
|
||||
self.send_header("Content-Length", str(len(body)))
|
||||
self.send_header("Access-Control-Allow-Origin", "*")
|
||||
self.end_headers()
|
||||
self.wfile.write(body)
|
||||
|
||||
def do_GET(self):
|
||||
path = self.path.split("?")[0].rstrip("/")
|
||||
if not path:
|
||||
path = "/"
|
||||
|
||||
if path in ("/", "/health"):
|
||||
self._send(200, {
|
||||
"service": "crypt.muse-dev.online",
|
||||
"status": "active",
|
||||
"mode": "public_attestation",
|
||||
"endpoints": [
|
||||
"/keys/allowed_signers",
|
||||
"/keys/<identity>.pub",
|
||||
"/proofs",
|
||||
"/proofs/<id>"
|
||||
]
|
||||
})
|
||||
return
|
||||
|
||||
if path == "/keys/allowed_signers":
|
||||
allowed_path = os.path.join(SIGNERS_DIR, "allowed_signers")
|
||||
if os.path.exists(allowed_path):
|
||||
with open(allowed_path, "r") as f:
|
||||
data = f.read()
|
||||
self._send(200, data, content_type="text/plain; charset=utf-8")
|
||||
else:
|
||||
self._send(404, {"error": "allowed_signers not found"})
|
||||
return
|
||||
|
||||
m_key = re.match(r"^/keys/([a-zA-Z0-9_\-\.]+)\.pub$", path)
|
||||
if m_key:
|
||||
ident = m_key.group(1)
|
||||
pub_path = os.path.join(SIGNERS_DIR, f"{ident}.pub")
|
||||
if os.path.exists(pub_path):
|
||||
with open(pub_path, "r") as f:
|
||||
data = f.read()
|
||||
self._send(200, data, content_type="text/plain; charset=utf-8")
|
||||
else:
|
||||
self._send(404, {"error": f"Public key for {ident} not found"})
|
||||
return
|
||||
|
||||
if path == "/proofs":
|
||||
proof_files = glob.glob(os.path.join(PROOFS_DIR, "*.json"))
|
||||
ids = [os.path.basename(p)[:-5] for p in proof_files]
|
||||
self._send(200, {"proofs": sorted(ids)})
|
||||
return
|
||||
|
||||
m_proof = re.match(r"^/proofs/([a-zA-Z0-9_\-]+)$", path)
|
||||
if m_proof:
|
||||
pid = m_proof.group(1)
|
||||
pf = os.path.join(PROOFS_DIR, f"{pid}.json")
|
||||
if os.path.exists(pf):
|
||||
with open(pf, "r") as f:
|
||||
data = json.load(f)
|
||||
self._send(200, data)
|
||||
else:
|
||||
self._send(404, {"error": f"Proof {pid} not found"})
|
||||
return
|
||||
|
||||
self._send(404, {"error": "not found"})
|
||||
|
||||
def do_POST(self):
|
||||
path = self.path.split("?")[0].rstrip("/")
|
||||
if path == "/proofs":
|
||||
try:
|
||||
length = int(self.headers.get("Content-Length", 0))
|
||||
except ValueError:
|
||||
length = 0
|
||||
if length <= 0 or length > 65536:
|
||||
self._send(400, {"error": "invalid content length"})
|
||||
return
|
||||
try:
|
||||
data = json.loads(self.rfile.read(length))
|
||||
except Exception:
|
||||
self._send(400, {"error": "malformed JSON"})
|
||||
return
|
||||
|
||||
pid = data.get("id")
|
||||
if not pid or not re.match(r"^[a-zA-Z0-9_\-]+$", pid):
|
||||
self._send(400, {"error": "missing or invalid proof id"})
|
||||
return
|
||||
|
||||
pf = os.path.join(PROOFS_DIR, f"{pid}.json")
|
||||
with open(pf, "w") as f:
|
||||
json.dump(data, f, indent=2)
|
||||
|
||||
self._send(201, {"status": "stored", "id": pid, "url": f"https://crypt.muse-dev.online/proofs/{pid}"})
|
||||
return
|
||||
|
||||
self._send(404, {"error": "not found"})
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description="Crypt attestation and key server")
|
||||
parser.add_argument("--port", type=int, default=8446)
|
||||
parser.add_argument("--host", default="100.123.153.75")
|
||||
parser.add_argument("--ssl", action="store_true", help="Enable self-signed HTTPS")
|
||||
args = parser.parse_args()
|
||||
|
||||
server = HTTPServer((args.host, args.port), CryptHandler)
|
||||
|
||||
if args.ssl:
|
||||
cert_file = os.path.join(REPO_DIR, "ssl", "crypt-selfsigned.crt")
|
||||
key_file = os.path.join(REPO_DIR, "ssl", "crypt-selfsigned.key")
|
||||
os.makedirs(os.path.dirname(cert_file), exist_ok=True)
|
||||
if not os.path.exists(cert_file):
|
||||
import subprocess
|
||||
subprocess.run([
|
||||
"openssl", "req", "-x509", "-newkey", "rsa:2048",
|
||||
"-keyout", key_file, "-out", cert_file,
|
||||
"-days", "3650", "-nodes",
|
||||
"-subj", "/CN=crypt.muse-dev.online"
|
||||
], check=True, capture_output=True)
|
||||
os.chmod(key_file, 0o600)
|
||||
ctx = ssl.SSLContext(ssl.PROTOCOL_TLS_SERVER)
|
||||
ctx.load_cert_chain(cert_file, key_file)
|
||||
server.socket = ctx.wrap_socket(server.socket, server_side=True)
|
||||
|
||||
print(f"Crypt server running on {'https' if args.ssl else 'http'}://{args.host}:{args.port}", file=sys.stderr)
|
||||
try:
|
||||
server.serve_forever()
|
||||
except KeyboardInterrupt:
|
||||
print("\nShutting down crypt server", file=sys.stderr)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
-152
@@ -1,152 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Side-chat to main-chat work siphon — detection rules.
|
||||
|
||||
Monitors side chat messages and identifies "siphon-worthy" content:
|
||||
work that should surface in main chat for visibility.
|
||||
|
||||
Categories:
|
||||
COMPLETED - work finished, results ready
|
||||
BLOCKER - something is stuck, needs intervention
|
||||
DECISION - a decision is needed from the user/operator
|
||||
ALERT - health/security/urgency signal
|
||||
MILESTONE - significant progress checkpoint
|
||||
|
||||
Detection is purely pattern-based (raw Python, no AI).
|
||||
Each rule returns (category, confidence, summary) or None.
|
||||
"""
|
||||
|
||||
import re
|
||||
from dataclasses import dataclass
|
||||
from typing import Optional
|
||||
|
||||
|
||||
@dataclass
|
||||
class SiphonHit:
|
||||
category: str # COMPLETED, BLOCKER, DECISION, ALERT, MILESTONE
|
||||
confidence: float # 0.0 - 1.0
|
||||
summary: str # one-line summary for main chat
|
||||
thread_id: str # source side chat
|
||||
message_id: str # source message
|
||||
# Full message text is NOT stored here — main chat gets a summary
|
||||
# plus a link back, never the full content (safety: no sensitive
|
||||
# data siphoned verbatim).
|
||||
|
||||
|
||||
# --- Keyword sets (configurable) ---
|
||||
|
||||
COMPLETED_PATTERNS = [
|
||||
re.compile(r'\b(done|completed|finished|deployed|shipped|live|verified)\b', re.I),
|
||||
re.compile(r'\b(all|tests?)\s+(pass|green|passing)\b', re.I),
|
||||
re.compile(r'\[RESULT[^\]]*\]\s*OK', re.I),
|
||||
re.compile(r'\b(merged|committed|pushed|published)\b', re.I),
|
||||
]
|
||||
|
||||
BLOCKER_PATTERNS = [
|
||||
re.compile(r'\b(blocked|stuck|failing|broken|down|error|failed)\b', re.I),
|
||||
re.compile(r'\b(need|needs|waiting)\s+(your|approval|input|decision)\b', re.I),
|
||||
re.compile(r'\b(can\'t|cannot|unable to)\b', re.I),
|
||||
re.compile(r'\[RESULT[^\]]*\]\s*(FAIL|ERROR)', re.I),
|
||||
]
|
||||
|
||||
DECISION_PATTERNS = [
|
||||
re.compile(r'\b(should (i|we)|shall i|want me to)\b', re.I),
|
||||
re.compile(r'\b(your call|needs? your|awaiting your)\b', re.I),
|
||||
re.compile(r'\b(approve|approval)\b.*\?', re.I),
|
||||
re.compile(r'^(yes|no)\s*\?\s*$', re.I),
|
||||
]
|
||||
|
||||
ALERT_PATTERNS = [
|
||||
re.compile(r'\b(security|vulnerability|breach|compromised|exploit)\b', re.I),
|
||||
re.compile(r'\b(urgent|critical|emergency|asap)\b', re.I),
|
||||
re.compile(r'\b(502|503|500)\b.*\b(error|down)\b', re.I),
|
||||
re.compile(r'\b(ssh|tunnel).*\b(down|broken|failed)\b', re.I),
|
||||
]
|
||||
|
||||
MILESTONE_PATTERNS = [
|
||||
re.compile(r'\b(milestone|phase \d+ (complete|done)|shipped v)\b', re.I),
|
||||
re.compile(r'\b(all \d+ (items? )?done)\b', re.I),
|
||||
]
|
||||
|
||||
# Patterns that suppress siphoning (safety)
|
||||
SUPPRESS_PATTERNS = [
|
||||
re.compile(r'\b(password|secret|token|key|pin)\s*[:=]', re.I),
|
||||
re.compile(r'-----BEGIN', re.I), # never siphon key material
|
||||
re.compile(r'\[do not siphon\]', re.I), # explicit opt-out marker
|
||||
]
|
||||
|
||||
|
||||
def _match_score(text: str, patterns) -> float:
|
||||
"""Return confidence based on how many patterns match."""
|
||||
hits = sum(1 for p in patterns if p.search(text))
|
||||
if hits == 0:
|
||||
return 0.0
|
||||
# Diminishing returns: 1 hit = 0.6, 2 = 0.8, 3+ = 0.95
|
||||
return min(0.95, 0.6 + (hits - 1) * 0.2)
|
||||
|
||||
|
||||
def _extract_summary(text: str, max_len: int = 120) -> str:
|
||||
"""Extract a safe one-line summary. Strips to first meaningful line."""
|
||||
# Take first non-empty line, truncate
|
||||
for line in text.strip().split('\n'):
|
||||
line = line.strip()
|
||||
if line and len(line) > 10:
|
||||
if len(line) > max_len:
|
||||
return line[:max_len - 3] + '...'
|
||||
return line
|
||||
return text[:max_len]
|
||||
|
||||
|
||||
def detect(text: str, thread_id: str, message_id: str,
|
||||
min_confidence: float = 0.6) -> Optional[SiphonHit]:
|
||||
"""
|
||||
Check a side chat message for siphon-worthy content.
|
||||
Returns SiphonHit or None.
|
||||
"""
|
||||
# Safety: suppress sensitive content
|
||||
for p in SUPPRESS_PATTERNS:
|
||||
if p.search(text):
|
||||
return None
|
||||
|
||||
candidates = [
|
||||
("COMPLETED", _match_score(text, COMPLETED_PATTERNS)),
|
||||
("BLOCKER", _match_score(text, BLOCKER_PATTERNS)),
|
||||
("DECISION", _match_score(text, DECISION_PATTERNS)),
|
||||
("ALERT", _match_score(text, ALERT_PATTERNS)),
|
||||
("MILESTONE", _match_score(text, MILESTONE_PATTERNS)),
|
||||
]
|
||||
|
||||
# Sort by confidence descending; ALERT wins ties (safety: urgency first)
|
||||
# Use negative confidence for descending, and ALERT as tiebreaker
|
||||
candidates.sort(key=lambda x: (-x[1], 0 if x[0] == "ALERT" else 1))
|
||||
best_cat, best_conf = candidates[0]
|
||||
|
||||
if best_conf < min_confidence:
|
||||
return None
|
||||
|
||||
return SiphonHit(
|
||||
category=best_cat,
|
||||
confidence=best_conf,
|
||||
summary=_extract_summary(text),
|
||||
thread_id=thread_id,
|
||||
message_id=message_id,
|
||||
)
|
||||
|
||||
|
||||
# --- Opt-out registry ---
|
||||
|
||||
_opt_out_threads: set = set()
|
||||
|
||||
|
||||
def opt_out(thread_id: str):
|
||||
"""Agent opts a side chat out of siphoning."""
|
||||
_opt_out_threads.add(thread_id)
|
||||
|
||||
|
||||
def opt_in(thread_id: str):
|
||||
"""Re-enable siphoning for a side chat."""
|
||||
_opt_out_threads.discard(thread_id)
|
||||
|
||||
|
||||
def is_opted_out(thread_id: str) -> bool:
|
||||
return thread_id in _opt_out_threads
|
||||
@@ -1,84 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
DM listener: Auto-routes 646's "Message <agent>:" requests
|
||||
|
||||
Watches 646's main chat for "Message <agent>: <text>" patterns,
|
||||
routes via dm.py automatically.
|
||||
|
||||
This makes 646's DM "work" — she says who, the system handles how.
|
||||
"""
|
||||
import subprocess
|
||||
import time
|
||||
import json
|
||||
import re
|
||||
from pathlib import Path
|
||||
|
||||
DM_PY = "/home/super/Projects/NetVM/bin/dm.py"
|
||||
NETVM_EXEC = "/home/super/Projects/NetVM/bin/netvm-exec.sh"
|
||||
API = "/home/super/Projects/NetVM/bin/muse-chat-api.py"
|
||||
WATERMARK_FILE = Path("/home/super/Projects/NetVM/bridge/dm-listener-watermark.json")
|
||||
VALID_AGENTS = ["muse", "pip", "646"]
|
||||
|
||||
def load_watermark():
|
||||
if WATERMARK_FILE.exists():
|
||||
with open(WATERMARK_FILE) as f:
|
||||
return json.load(f)
|
||||
return {"processed": []}
|
||||
|
||||
def save_watermark(data):
|
||||
WATERMARK_FILE.parent.mkdir(parents=True, exist_ok=True)
|
||||
with open(WATERMARK_FILE, 'w') as f:
|
||||
json.dump(data, f, indent=2)
|
||||
|
||||
def run(cmd, timeout=60):
|
||||
result = subprocess.run(cmd, shell=True, capture_output=True, text=True, timeout=timeout)
|
||||
return result.stdout.strip()
|
||||
|
||||
def get_646_messages():
|
||||
return run(f"{NETVM_EXEC} 646 -- python3 {API} --account 646 messages 5")
|
||||
|
||||
def poll_once():
|
||||
wm = load_watermark()
|
||||
msgs = get_646_messages()
|
||||
parts = [p.strip() for p in msgs.split('---') if p.strip()]
|
||||
|
||||
for part in parts:
|
||||
# Match "Message <agent>: <text>"
|
||||
m = re.match(r'Message\s+(\w+)\s*:\s*(.+)', part, re.DOTALL | re.IGNORECASE)
|
||||
if m:
|
||||
agent, text = m.groups()
|
||||
agent = agent.lower()
|
||||
text = text.strip()
|
||||
|
||||
if agent not in VALID_AGENTS:
|
||||
continue
|
||||
|
||||
# Deduplicate
|
||||
cmd_id = f"{agent}:{text[:40]}"
|
||||
if cmd_id in wm["processed"]:
|
||||
continue
|
||||
|
||||
print(f"Routing: 646 -> {agent}: {text[:60]}...")
|
||||
# Route via dm.py (sends to agent's main chat)
|
||||
run(f"{DM_PY} send --agent {agent} --target main \"[646] {text[:800]}\"")
|
||||
|
||||
wm["processed"].append(cmd_id)
|
||||
# Keep only last 20 to avoid unbounded growth
|
||||
wm["processed"] = wm["processed"][-20:]
|
||||
save_watermark(wm)
|
||||
print("Routed.")
|
||||
|
||||
if __name__ == '__main__':
|
||||
import sys
|
||||
if '--once' in sys.argv:
|
||||
poll_once()
|
||||
elif '--daemon' in sys.argv:
|
||||
print("DM listener running...")
|
||||
while True:
|
||||
try:
|
||||
poll_once()
|
||||
except Exception as e:
|
||||
print(f"Error: {e}")
|
||||
time.sleep(30)
|
||||
else:
|
||||
print("Use --once or --daemon")
|
||||
@@ -1,267 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""dm-log-taxonomy.py — READ-ONLY failure taxonomy for fleet DM sidechat reliability.
|
||||
|
||||
Reads /home/super/Projects/NetVM/dm-log.jsonl, prints:
|
||||
1. Event-type counts and send outcome rates
|
||||
2. Sidechat nav failure taxonomy (per-target, per-agent-pair)
|
||||
3. Failure timeline (hourly buckets, worst 10-min windows, by node)
|
||||
4. "Ghost" rate: verified:true sidechat sends with no UUID anywhere
|
||||
5. Hypothesis evidence tables (nav_failed reasons, placement pairs,
|
||||
alias sources, autoprovision success, retry distribution)
|
||||
|
||||
No writes to any state files. Runs in <1s on the current log size.
|
||||
"""
|
||||
import json
|
||||
import re
|
||||
import sys
|
||||
from collections import Counter, defaultdict
|
||||
from datetime import datetime, timezone, timedelta
|
||||
|
||||
LOG = "/home/super/Projects/NetVM/dm-log.jsonl"
|
||||
WINDOW_H = 48
|
||||
|
||||
|
||||
def parse_ts(s):
|
||||
if not s:
|
||||
return None
|
||||
try:
|
||||
dt = datetime.fromisoformat(s.replace("Z", "+00:00"))
|
||||
if dt.tzinfo is None:
|
||||
dt = dt.replace(tzinfo=timezone.utc)
|
||||
return dt
|
||||
except Exception:
|
||||
return None
|
||||
|
||||
|
||||
def main():
|
||||
now = datetime.now(timezone.utc)
|
||||
cutoff = now - timedelta(hours=WINDOW_H)
|
||||
events = []
|
||||
parse_err = 0
|
||||
with open(LOG, encoding="utf-8") as f:
|
||||
for line in f:
|
||||
line = line.strip()
|
||||
if not line:
|
||||
continue
|
||||
try:
|
||||
e = json.loads(line)
|
||||
except Exception:
|
||||
parse_err += 1
|
||||
continue
|
||||
e["_dt"] = parse_ts(e.get("ts"))
|
||||
events.append(e)
|
||||
in_win = [e for e in events if e["_dt"] and e["_dt"] >= cutoff]
|
||||
print(f"dm-log.jsonl: {len(events)} total lines ({parse_err} parse errors)")
|
||||
print(f"window: last {WINDOW_H}h -> {len(in_win)} events")
|
||||
if in_win:
|
||||
print(f" range: {in_win[0]['ts']} .. {in_win[-1]['ts']}")
|
||||
print("=" * 78)
|
||||
|
||||
# ---- 1. event-type counts ------------------------------------------------
|
||||
types = Counter(e.get("type", "?") for e in in_win)
|
||||
print("\n[1] EVENT-TYPE COUNTS")
|
||||
for t, n in types.most_common():
|
||||
print(f" {n:6d} {t}")
|
||||
|
||||
# ---- per-send assembly ---------------------------------------------------
|
||||
sends = {}
|
||||
order = []
|
||||
|
||||
def S(e):
|
||||
sid = e.get("id")
|
||||
if not sid:
|
||||
return None
|
||||
if sid not in sends:
|
||||
sends[sid] = {"id": sid, "events": [], "first_ts": e["_dt"]}
|
||||
order.append(sid)
|
||||
s = sends[sid]
|
||||
s["events"].append(e)
|
||||
for k in ("agent", "to", "target"):
|
||||
if k not in s and e.get(k) is not None:
|
||||
s[k] = e.get(k)
|
||||
t = e.get("type")
|
||||
if t == "verified":
|
||||
s["verified_seen"] = True
|
||||
if e.get("thread_uuid"):
|
||||
s["uuids"] = s.get("uuids", set()) | {e["thread_uuid"]}
|
||||
if t == "sent":
|
||||
s["sent_seen"] = True
|
||||
s["sent_verified"] = bool(e.get("verified"))
|
||||
if e.get("thread_uuid"):
|
||||
s["uuids"] = s.get("uuids", set()) | {e["thread_uuid"]}
|
||||
tags = e.get("tags") or {}
|
||||
if isinstance(tags, dict) and tags.get("thread"):
|
||||
s["uuids"] = s.get("uuids", set()) | {tags["thread"]}
|
||||
if t == "send_done":
|
||||
s["done"] = True
|
||||
if t == "sidechat_uuid_capture_failed":
|
||||
s["capture_failed"] = True
|
||||
s["capture_out"] = (e.get("out") or "")[:80]
|
||||
if t == "sidechat_autoprovisioned" and e.get("thread_uuid"):
|
||||
s["uuids"] = s.get("uuids", set()) | {e["thread_uuid"]}
|
||||
if t == "alias_resolved" and e.get("thread_uuid"):
|
||||
s["uuids"] = s.get("uuids", set()) | {e["thread_uuid"]}
|
||||
if t == "placement_mismatch":
|
||||
s["placement_mismatch"] = True
|
||||
if t == "pre_send_assert_failed":
|
||||
s["gate_failed"] = True
|
||||
s["gate_reason"] = e.get("reason")
|
||||
if t == "nav_failed":
|
||||
s["nav_failed"] = True
|
||||
s["nav_reason"] = e.get("reason") or "transport"
|
||||
return s
|
||||
|
||||
for e in in_win:
|
||||
S(e)
|
||||
|
||||
started = [sends[i] for i in order
|
||||
if any(e.get("type") == "send_start" for e in sends[i]["events"])]
|
||||
print(f"\n[1b] SEND OUTCOMES: {len(started)} sends attempted")
|
||||
oc = Counter()
|
||||
for s in started:
|
||||
if s.get("gate_failed"):
|
||||
oc["loud-fail: pre_send_assert_failed"] += 1
|
||||
elif s.get("nav_failed"):
|
||||
oc["loud-fail: nav_failed"] += 1
|
||||
elif s.get("sent_seen") and s.get("sent_verified"):
|
||||
oc["sent verified:true"] += 1
|
||||
elif s.get("sent_seen"):
|
||||
oc["sent verified:false"] += 1
|
||||
elif s.get("done"):
|
||||
oc["send_done, no sent event"] += 1
|
||||
else:
|
||||
oc["abandoned (no terminal event)"] += 1
|
||||
for k, n in oc.most_common():
|
||||
print(f" {n:5d} {k}")
|
||||
print(" (note: main_chat_blocked events are policy blocks, not failures;")
|
||||
print(" followup_register_failed=signing issues, tangential)")
|
||||
|
||||
# ---- 2. sidechat taxonomy ------------------------------------------------
|
||||
def is_sc(s):
|
||||
return (s.get("target") or "") not in ("main", "", None)
|
||||
|
||||
sc = [s for s in started if is_sc(s)]
|
||||
print(f"\n[2] SIDECHAT TAXONOMY ({len(sc)} sidechat-targeted sends)")
|
||||
tax = Counter()
|
||||
per_target = defaultdict(Counter)
|
||||
per_pair = defaultdict(Counter)
|
||||
for s in sc:
|
||||
if s.get("gate_failed"):
|
||||
cls = "gate_failed:" + str(s.get("gate_reason"))
|
||||
elif s.get("nav_failed"):
|
||||
cls = "nav_failed:" + str(s.get("nav_reason"))
|
||||
elif s.get("capture_failed"):
|
||||
cls = "uuid_capture_failed"
|
||||
elif s.get("placement_mismatch"):
|
||||
cls = "placement_mismatch"
|
||||
elif s.get("sent_seen") and s.get("sent_verified"):
|
||||
cls = ("verified:true, NO uuid anywhere (ghost)"
|
||||
if not s.get("uuids") else "clean verified")
|
||||
elif s.get("sent_seen"):
|
||||
cls = "sent verified:false"
|
||||
elif s.get("done"):
|
||||
cls = "done, no sent event"
|
||||
else:
|
||||
cls = "abandoned"
|
||||
tax[cls] += 1
|
||||
per_target[s.get("target") or "?"][cls] += 1
|
||||
per_pair[f"{s.get('agent') or '?'}->{s.get('to') or '?'}"][cls] += 1
|
||||
for k, n in tax.most_common():
|
||||
print(f" {n:5d} {k}")
|
||||
print("\n per-target (sends, non-clean, top classes):")
|
||||
for tgt, c in sorted(per_target.items(), key=lambda x: -sum(x[1].values())):
|
||||
tot = sum(c.values())
|
||||
bad = tot - c.get("clean verified", 0)
|
||||
print(f" {tgt}: {tot} sends, {bad} non-clean {dict(c.most_common(4))}")
|
||||
print("\n per agent-pair:")
|
||||
for pair, c in sorted(per_pair.items(), key=lambda x: -sum(x[1].values())):
|
||||
tot = sum(c.values())
|
||||
bad = tot - c.get("clean verified", 0)
|
||||
print(f" {pair}: {tot} sends, {bad} non-clean {dict(c.most_common(4))}")
|
||||
|
||||
# ---- 4. ghosts ------------------------------------------------------------
|
||||
ghosts = [s for s in sc if s.get("sent_seen") and s.get("sent_verified")
|
||||
and not s.get("uuids")]
|
||||
print(f"\n[4] GHOST RATE: {len(ghosts)}/{len(sc)} "
|
||||
f"({100.0 * len(ghosts) / len(sc) if sc else 0:.0f}%) verified:true "
|
||||
f"sidechat sends with no UUID in any event")
|
||||
gh = Counter(s["first_ts"].strftime("%m-%d %H") for s in ghosts if s["first_ts"])
|
||||
print(" ghost hours:", dict(sorted(gh.items())))
|
||||
print(" proxies: uuid_capture_failed="
|
||||
f"{sum(1 for s in sc if s.get('capture_failed'))}, "
|
||||
f"placement_mismatch={sum(1 for s in sc if s.get('placement_mismatch'))}, "
|
||||
f"pre_send_assert_failed={sum(1 for s in sc if s.get('gate_failed'))}")
|
||||
|
||||
# ---- 3. timeline ------------------------------------------------------------
|
||||
print("\n[3] TIMELINE (hourly, sidechat sends; #=bad, .=ok)")
|
||||
buckets = defaultdict(Counter)
|
||||
for s in sc:
|
||||
if not s.get("first_ts"):
|
||||
continue
|
||||
hr = s["first_ts"].strftime("%m-%d %H:00")
|
||||
buckets[hr]["total"] += 1
|
||||
bad = not (s.get("sent_seen") and s.get("sent_verified")
|
||||
and not s.get("capture_failed")
|
||||
and not s.get("placement_mismatch")
|
||||
and not s.get("gate_failed") and not s.get("nav_failed"))
|
||||
if bad:
|
||||
buckets[hr]["bad"] += 1
|
||||
for hr in sorted(buckets):
|
||||
t, b = buckets[hr]["total"], buckets[hr]["bad"]
|
||||
print(f" {hr} total={t:3d} bad={b:3d} {'#' * b}{'.' * (t - b)}")
|
||||
wins = defaultdict(Counter)
|
||||
for s in sc:
|
||||
if not s.get("first_ts"):
|
||||
continue
|
||||
w = s["first_ts"].strftime("%m-%d %H:%M")[:-1] + "0"
|
||||
wins[w]["total"] += 1
|
||||
if not (s.get("sent_seen") and s.get("sent_verified")):
|
||||
wins[w]["bad"] += 1
|
||||
print(" worst 10-min windows (>=3 sends):")
|
||||
shown = 0
|
||||
for w, c in sorted(wins.items(), key=lambda x: -x[1]["bad"]):
|
||||
if c["total"] >= 3 and shown < 8:
|
||||
print(f" {w} total={c['total']} bad={c['bad']}")
|
||||
shown += 1
|
||||
print(" bad rate by sending node:")
|
||||
by_node = defaultdict(Counter)
|
||||
for s in sc:
|
||||
by_node[s.get("agent") or "?"]["total"] += 1
|
||||
if not (s.get("sent_seen") and s.get("sent_verified")):
|
||||
by_node[s.get("agent") or "?"]["bad"] += 1
|
||||
for node, c in sorted(by_node.items(), key=lambda x: -x[1]["total"]):
|
||||
r = 100.0 * c["bad"] / c["total"] if c["total"] else 0
|
||||
print(f" {node}: {c['bad']}/{c['total']} bad ({r:.0f}%)")
|
||||
|
||||
# ---- 5. hypothesis evidence ---------------------------------------------------
|
||||
print("\n[5] HYPOTHESIS EVIDENCE")
|
||||
nfr = Counter(s.get("nav_reason") for s in sc if s.get("nav_failed"))
|
||||
print(f" nav_failed reasons: {dict(nfr)}")
|
||||
land = [s for s in sc if s.get("capture_failed")]
|
||||
print(f" uuid_capture_failed: {len(land)}, "
|
||||
f"landing-page outs: {sum(1 for s in land if 'muse.ai/' in (s.get('capture_out') or ''))}")
|
||||
pairs = Counter()
|
||||
for e in in_win:
|
||||
if e.get("type") == "placement_mismatch":
|
||||
pairs[((e.get("expected_uuid") or "?")[:8],
|
||||
(e.get("actual_uuid") or "?")[:8], e.get("target"))] += 1
|
||||
print(" placement_mismatch expected->actual:")
|
||||
for (a, b, t), n in pairs.most_common(6):
|
||||
print(f" {n:3d} exp={a}.. act={b}.. target={t}")
|
||||
ar = Counter(e.get("source") for e in in_win if e.get("type") == "alias_resolved")
|
||||
print(f" alias_resolved sources: {dict(ar)} (None = field absent, older events)")
|
||||
ap_try = sum(1 for e in in_win if e.get("type") == "sidechat_autoprovision_start")
|
||||
ap_ok = sum(1 for e in in_win if e.get("type") == "sidechat_autoprovisioned"
|
||||
and e.get("thread_uuid"))
|
||||
print(f" autoprovision: {ap_try} attempts -> {ap_ok} with uuid ({100.0 * ap_ok / ap_try if ap_try else 0:.0f}%)")
|
||||
rt = Counter()
|
||||
for s in started:
|
||||
n = sum(1 for e in s["events"] if e.get("type") == "retry")
|
||||
if n:
|
||||
rt[n] += 1
|
||||
print(f" retry distribution (sends with >=1 retry): {dict(sorted(rt.items()))}")
|
||||
print("\n DONE.")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -1,74 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
# dm-sign.sh — produce a signed DM for the fleet DM system.
|
||||
#
|
||||
# Signs a message with an SSH key (namespace "dm", Ed25519) and prints the
|
||||
# signed wire format that `dm.py verify-sig` checks:
|
||||
#
|
||||
# [from:<identity>] [id:<id>]
|
||||
#
|
||||
# <message body>
|
||||
#
|
||||
# -----BEGIN SSH SIGNATURE-----
|
||||
# ...
|
||||
# -----END SSH SIGNATURE-----
|
||||
#
|
||||
# The signed payload is exactly "[from:X] [id:Y]\n\n<body>" (no trailing
|
||||
# newline) — verify-sig reconstructs it by stripping everything from the
|
||||
# signature block onward, so the two must match byte-for-byte.
|
||||
#
|
||||
# Usage:
|
||||
# dm-sign.sh --from <identity> [--key <privkey>] [--id <id>] <message>
|
||||
#
|
||||
# Defaults: --key ~/.ssh/id_frontdoor, --id = 8 random hex chars.
|
||||
# The private key is only ever read locally; it is never moved or copied.
|
||||
# Pipe the output straight into `dm.py send --raw` (never truncate it).
|
||||
#
|
||||
# Example:
|
||||
# dm.py send --agent opm --target main --raw \
|
||||
# "$(dm-sign.sh --from operator-main 'hello from the operator')"
|
||||
set -euo pipefail
|
||||
|
||||
FROM=""
|
||||
KEY="$HOME/.ssh/id_frontdoor"
|
||||
ID="$(head -c4 /dev/urandom | od -An -tx1 | tr -d ' \n')"
|
||||
|
||||
usage() {
|
||||
sed -n '2,/^set -euo/p' "$0" | sed 's/^# \?//'
|
||||
}
|
||||
|
||||
while [[ $# -gt 0 ]]; do
|
||||
case "$1" in
|
||||
--from) FROM="${2:?--from needs a value}"; shift 2 ;;
|
||||
--key) KEY="${2:?--key needs a value}"; shift 2 ;;
|
||||
--id) ID="${2:?--id needs a value}"; shift 2 ;;
|
||||
-h|--help) usage; exit 0 ;;
|
||||
--) shift; break ;;
|
||||
-*) echo "error: unknown option: $1" >&2; exit 1 ;;
|
||||
*) break ;;
|
||||
esac
|
||||
done
|
||||
|
||||
if [[ $# -eq 0 ]]; then
|
||||
echo "error: no message given" >&2
|
||||
echo "usage: dm-sign.sh --from <identity> [--key <privkey>] [--id <id>] <message>" >&2
|
||||
exit 1
|
||||
fi
|
||||
MESSAGE="$*"
|
||||
|
||||
[[ -n "$FROM" ]] || { echo "error: --from <identity> is required" >&2; exit 1; }
|
||||
[[ -f "$KEY" ]] || { echo "error: private key not found: $KEY" >&2; exit 1; }
|
||||
|
||||
TD="$(mktemp -d)"
|
||||
trap 'rm -rf "$TD"' EXIT
|
||||
PAYLOAD="$TD/payload"
|
||||
|
||||
# Payload: header, blank line, body — NO trailing newline (verify-sig strips).
|
||||
printf '[from:%s] [id:%s]\n\n%s' "$FROM" "$ID" "$MESSAGE" > "$PAYLOAD"
|
||||
|
||||
# Never reuse a stale signature: a leftover .sig from an earlier run would
|
||||
# silently sign the wrong payload (burned 20 minutes on the board, 2026-10-03).
|
||||
rm -f "$PAYLOAD.sig"
|
||||
ssh-keygen -Y sign -f "$KEY" -n dm "$PAYLOAD" >/dev/null
|
||||
|
||||
printf '[from:%s] [id:%s]\n\n%s\n\n' "$FROM" "$ID" "$MESSAGE"
|
||||
cat "$PAYLOAD.sig"
|
||||
File diff suppressed because it is too large
Load Diff
@@ -1,377 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
HTTPS exec server for operator remote command execution.
|
||||
Runs on bl (stable), accepts authenticated POST /exec, returns command output.
|
||||
|
||||
Usage:
|
||||
python3 exec-server.py --port 8443 --token-file /home/super/.exec-token
|
||||
|
||||
646 (or any operator) can then:
|
||||
curl -k -X POST https://100.123.153.75:8443/exec \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"cmd": "dm.py send --agent 646 --to opm --target main \"hello\"", "token": "..."}'
|
||||
|
||||
Via the VM Caddy (no tailnet needed from the agent's container):
|
||||
curl -k -X POST https://34-139-37-135.sslip.io/exec/exec \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"cmd": "...", "token": "<per-agent token>"}'
|
||||
|
||||
Signature auth (preferred for agents): no secret crosses the wire at all.
|
||||
The agent signs {"cmd","ts","nonce"} with their registered SSH key
|
||||
(`ssh-keygen -Y sign -n exec-server`) and posts
|
||||
{"identity","payload","signature"}. Verified against the signers file
|
||||
with `ssh-keygen -Y verify`; ts must be within 300s and the nonce unused.
|
||||
This is the same identity primitive as signed board posts, and it never
|
||||
trips secret-handling guardrails because a signature is not a secret.
|
||||
|
||||
Auth: the master token (TOKEN_FILE) plus per-agent tokens, one file per
|
||||
agent under TOKEN_DIR (0600). Each agent's token is individually revocable
|
||||
by deleting its file. The using identity is logged (never the token).
|
||||
"""
|
||||
import argparse
|
||||
import hashlib
|
||||
import hmac
|
||||
import json
|
||||
import secrets
|
||||
import subprocess
|
||||
import sys
|
||||
from http.server import HTTPServer, BaseHTTPRequestHandler
|
||||
import ssl
|
||||
|
||||
# Token file - generated on first run if not exists
|
||||
TOKEN_FILE = '/home/super/.exec-server-token'
|
||||
# Per-agent tokens: TOKEN_DIR/<agent> contains that agent's token
|
||||
TOKEN_DIR = '/home/super/.exec-tokens'
|
||||
# ssh-keygen signature auth: public keys of fleet identities, one per line
|
||||
# ("<identity> ssh-ed25519 AAAA..."), synced from the VM's allowed_signers.
|
||||
SIGNERS_FILE = '/home/super/.exec-signers'
|
||||
# Seen nonces for replay protection ("<nonce> <ts>" per line).
|
||||
NONCE_FILE = '/home/super/.exec-nonces'
|
||||
SIG_NAMESPACE = 'exec-server'
|
||||
SIG_MAX_SKEW = 300 # seconds; nonces remembered for 2x this
|
||||
|
||||
def get_token():
|
||||
"""Load or generate the auth token."""
|
||||
try:
|
||||
with open(TOKEN_FILE, 'r') as f:
|
||||
return f.read().strip()
|
||||
except FileNotFoundError:
|
||||
token = secrets.token_hex(32)
|
||||
with open(TOKEN_FILE, 'w') as f:
|
||||
f.write(token)
|
||||
# Secure permissions
|
||||
import os
|
||||
os.chmod(TOKEN_FILE, 0o600)
|
||||
print(f'Generated new token in {TOKEN_FILE}', file=sys.stderr)
|
||||
return token
|
||||
|
||||
def check_token(token):
|
||||
"""Check token against the master token and per-agent tokens.
|
||||
Returns the identity label ('master' or the agent filename), or None."""
|
||||
if token and hmac.compare_digest(token, get_token()):
|
||||
return 'master'
|
||||
import os
|
||||
try:
|
||||
names = os.listdir(TOKEN_DIR)
|
||||
except FileNotFoundError:
|
||||
return None
|
||||
for name in names:
|
||||
p = os.path.join(TOKEN_DIR, name)
|
||||
if not os.path.isfile(p):
|
||||
continue
|
||||
try:
|
||||
with open(p) as f:
|
||||
t = f.read().strip()
|
||||
except OSError:
|
||||
continue
|
||||
if t and hmac.compare_digest(token, t):
|
||||
return name
|
||||
return None
|
||||
|
||||
def check_nonce(nonce):
|
||||
"""True if the nonce was never used; records it. Prunes expired entries."""
|
||||
import os
|
||||
import time
|
||||
now = time.time()
|
||||
fresh = []
|
||||
try:
|
||||
with open(NONCE_FILE) as f:
|
||||
for line in f:
|
||||
parts = line.split()
|
||||
if len(parts) != 2:
|
||||
continue
|
||||
n, t = parts
|
||||
try:
|
||||
if now - float(t) < 2 * SIG_MAX_SKEW:
|
||||
fresh.append((n, t))
|
||||
except ValueError:
|
||||
pass
|
||||
except FileNotFoundError:
|
||||
pass
|
||||
if any(n == nonce for n, _ in fresh):
|
||||
return False
|
||||
fresh.append((nonce, str(now)))
|
||||
try:
|
||||
with open(NONCE_FILE, 'w') as f:
|
||||
for n, t in fresh:
|
||||
f.write(f"{n} {t}\n")
|
||||
os.chmod(NONCE_FILE, 0o600)
|
||||
except OSError:
|
||||
return False
|
||||
return True
|
||||
|
||||
|
||||
def check_signature(identity, payload, signature):
|
||||
"""Verify an `ssh-keygen -Y` signature over the payload envelope.
|
||||
Returns the identity on success, None on failure. The envelope must be
|
||||
JSON {"cmd","ts","nonce"} with a fresh ts and an unused nonce."""
|
||||
import json as _json
|
||||
import os
|
||||
import re
|
||||
import subprocess
|
||||
import tempfile
|
||||
import time
|
||||
if not re.fullmatch(r'[a-z0-9-]+', identity or ''):
|
||||
return None
|
||||
try:
|
||||
data = _json.loads(payload)
|
||||
except Exception:
|
||||
return None
|
||||
if not isinstance(data, dict):
|
||||
return None
|
||||
cmd = data.get('cmd')
|
||||
ts = data.get('ts')
|
||||
nonce = data.get('nonce')
|
||||
if not isinstance(cmd, str) or not cmd:
|
||||
return None
|
||||
if not isinstance(nonce, str) or not re.fullmatch(r'[0-9a-fA-F]{16,128}', nonce):
|
||||
return None
|
||||
try:
|
||||
ts = float(ts)
|
||||
except (TypeError, ValueError):
|
||||
return None
|
||||
if abs(time.time() - ts) > SIG_MAX_SKEW:
|
||||
return None
|
||||
if not check_nonce(nonce):
|
||||
return None
|
||||
sig_path = None
|
||||
try:
|
||||
with tempfile.NamedTemporaryFile('w', delete=False, suffix='.sig') as f:
|
||||
f.write(signature if signature.endswith('\n') else signature + '\n')
|
||||
sig_path = f.name
|
||||
p = subprocess.run(
|
||||
['ssh-keygen', '-Y', 'verify', '-f', SIGNERS_FILE, '-I', identity,
|
||||
'-n', SIG_NAMESPACE, '-s', sig_path],
|
||||
input=payload.encode(), capture_output=True, timeout=15)
|
||||
return identity if p.returncode == 0 else None
|
||||
except Exception:
|
||||
return None
|
||||
finally:
|
||||
if sig_path:
|
||||
try:
|
||||
os.unlink(sig_path)
|
||||
except OSError:
|
||||
pass
|
||||
|
||||
|
||||
class ExecHandler(BaseHTTPRequestHandler):
|
||||
def log_message(self, format, *args):
|
||||
# Quiet logging, just to stderr
|
||||
sys.stderr.write(f'{self.client_address[0]} - {format % args}\n')
|
||||
|
||||
def do_POST(self):
|
||||
# Read body
|
||||
content_length = int(self.headers.get('Content-Length', 0))
|
||||
if content_length > 1024 * 1024: # 1MB max
|
||||
self.send_response(413)
|
||||
self.end_headers()
|
||||
return
|
||||
|
||||
body = self.rfile.read(content_length)
|
||||
try:
|
||||
data = json.loads(body)
|
||||
except json.JSONDecodeError:
|
||||
self.send_response(400)
|
||||
self.end_headers()
|
||||
self.wfile.write(b'{"error": "invalid json"}')
|
||||
return
|
||||
|
||||
if self.path == '/exec/rotate':
|
||||
self.handle_rotate(data)
|
||||
return
|
||||
|
||||
if self.path != '/exec':
|
||||
self.send_response(404)
|
||||
self.end_headers()
|
||||
return
|
||||
|
||||
# Auth: bearer token OR ssh-keygen -Y signature (no secret in transit).
|
||||
# Bearer: {"token": "...", "cmd": "..."}.
|
||||
# Signature: {"identity": "...", "payload": "{\"cmd\":...,\"ts\":...,\"nonce\":...}",
|
||||
# "signature": "<ssh-keygen -Y armor>"}. check_signature
|
||||
# validates the envelope; cmd comes from the signed payload.
|
||||
ident = None
|
||||
cmd = ''
|
||||
token = data.get('token', '')
|
||||
if token:
|
||||
ident = check_token(token)
|
||||
cmd = data.get('cmd', '')
|
||||
elif data.get('identity') and data.get('payload') and data.get('signature'):
|
||||
ident = check_signature(data['identity'], data['payload'], data['signature'])
|
||||
if ident:
|
||||
try:
|
||||
cmd = json.loads(data['payload']).get('cmd', '')
|
||||
except Exception:
|
||||
cmd = ''
|
||||
if not ident:
|
||||
self.send_response(401)
|
||||
self.end_headers()
|
||||
self.wfile.write(b'{"error": "unauthorized"}')
|
||||
return
|
||||
sys.stderr.write(f'exec as {ident} from {self.client_address[0]}\n')
|
||||
# Get command
|
||||
if not cmd or not isinstance(cmd, str):
|
||||
self.send_response(400)
|
||||
self.end_headers()
|
||||
self.wfile.write(b'{"error": "missing cmd"}')
|
||||
return
|
||||
|
||||
# Execute (with timeout)
|
||||
timeout = min(data.get('timeout', 60), 300) # max 5 min
|
||||
try:
|
||||
result = subprocess.run(
|
||||
cmd,
|
||||
shell=True,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=timeout,
|
||||
cwd='/home/super'
|
||||
)
|
||||
response = {
|
||||
'stdout': result.stdout,
|
||||
'stderr': result.stderr,
|
||||
'rc': result.returncode,
|
||||
}
|
||||
except subprocess.TimeoutExpired:
|
||||
response = {'error': 'timeout', 'rc': -1}
|
||||
except Exception as e:
|
||||
response = {'error': str(e), 'rc': -1}
|
||||
|
||||
# Send response
|
||||
resp_body = json.dumps(response).encode()
|
||||
self.send_response(200)
|
||||
self.send_header('Content-Type', 'application/json')
|
||||
self.send_header('Content-Length', str(len(resp_body)))
|
||||
self.end_headers()
|
||||
self.wfile.write(resp_body)
|
||||
|
||||
def handle_rotate(self, data):
|
||||
"""POST /exec/rotate - replace an agent's token with a fresh one.
|
||||
The old token dies immediately; the new one is returned ONLY in the
|
||||
response body (never logged). Lets an agent bootstrap from a
|
||||
trust-root-delivered token and end up with one nobody else knows.
|
||||
Agents may rotate only their own token; master may rotate any
|
||||
agent's (not its own - that stays manual on bl)."""
|
||||
import os
|
||||
import re
|
||||
token = data.get('token', '')
|
||||
ident = check_token(token)
|
||||
if not ident:
|
||||
self.send_response(401)
|
||||
self.end_headers()
|
||||
self.wfile.write(b'{"error": "unauthorized"}')
|
||||
return
|
||||
target = data.get('agent', ident)
|
||||
if not re.fullmatch(r'[a-z0-9-]+', target or ''):
|
||||
self.send_response(400)
|
||||
self.end_headers()
|
||||
self.wfile.write(b'{"error": "bad agent name"}')
|
||||
return
|
||||
if target == 'master' or (ident != 'master' and target != ident):
|
||||
self.send_response(403)
|
||||
self.end_headers()
|
||||
self.wfile.write(b'{"error": "forbidden"}')
|
||||
return
|
||||
path = os.path.join(TOKEN_DIR, target)
|
||||
if not os.path.isfile(path):
|
||||
self.send_response(404)
|
||||
self.end_headers()
|
||||
self.wfile.write(b'{"error": "no such agent token"}')
|
||||
return
|
||||
new_token = secrets.token_hex(32)
|
||||
tmp = path + '.tmp'
|
||||
with open(tmp, 'w') as f:
|
||||
f.write(new_token + '\n')
|
||||
os.chmod(tmp, 0o600)
|
||||
os.replace(tmp, path)
|
||||
sys.stderr.write(f'token rotated for {target} by {ident}\n')
|
||||
resp_body = json.dumps({'token': new_token}).encode()
|
||||
self.send_response(200)
|
||||
self.send_header('Content-Type', 'application/json')
|
||||
self.send_header('Content-Length', str(len(resp_body)))
|
||||
self.end_headers()
|
||||
self.wfile.write(resp_body)
|
||||
|
||||
def do_GET(self):
|
||||
if self.path == '/health':
|
||||
self.send_response(200)
|
||||
self.send_header('Content-Type', 'application/json')
|
||||
self.end_headers()
|
||||
self.wfile.write(b'{"status": "ok"}')
|
||||
else:
|
||||
self.send_response(404)
|
||||
self.end_headers()
|
||||
|
||||
def main():
|
||||
global TOKEN_FILE, TOKEN_DIR
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument('--port', type=int, default=8443)
|
||||
parser.add_argument('--host', default='0.0.0.0')
|
||||
parser.add_argument('--token-file', default='/home/super/.exec-server-token')
|
||||
parser.add_argument('--token-dir', default='/home/super/.exec-tokens')
|
||||
parser.add_argument('--signers-file', default='/home/super/.exec-signers')
|
||||
parser.add_argument('--nonce-file', default='/home/super/.exec-nonces')
|
||||
args = parser.parse_args()
|
||||
|
||||
TOKEN_FILE = args.token_file
|
||||
TOKEN_DIR = args.token_dir
|
||||
global SIGNERS_FILE, NONCE_FILE
|
||||
SIGNERS_FILE = args.signers_file
|
||||
NONCE_FILE = args.nonce_file
|
||||
|
||||
# Ensure token exists
|
||||
token = get_token()
|
||||
print(f'Token: {token[:8]}... (full in {TOKEN_FILE})', file=sys.stderr)
|
||||
|
||||
server = HTTPServer((args.host, args.port), ExecHandler)
|
||||
|
||||
# Wrap with TLS (self-signed is fine for our use, we use -k)
|
||||
# Generate self-signed cert if not exists
|
||||
import os
|
||||
cert_file = '/home/super/.exec-server-cert.pem'
|
||||
key_file = '/home/super/.exec-server-key.pem'
|
||||
|
||||
if not os.path.exists(cert_file):
|
||||
print('Generating self-signed cert...', file=sys.stderr)
|
||||
subprocess.run([
|
||||
'openssl', 'req', '-x509', '-newkey', 'rsa:2048',
|
||||
'-keyout', key_file, '-out', cert_file,
|
||||
'-days', '3650', '-nodes',
|
||||
'-subj', '/CN=bl-exec-server'
|
||||
], check=True, capture_output=True)
|
||||
os.chmod(key_file, 0o600)
|
||||
|
||||
context = ssl.SSLContext(ssl.PROTOCOL_TLS_SERVER)
|
||||
context.load_cert_chain(cert_file, key_file)
|
||||
server.socket = context.wrap_socket(server.socket, server_side=True)
|
||||
|
||||
print(f'Exec server listening on https://{args.host}:{args.port}/exec', file=sys.stderr)
|
||||
print('Health check: https://<host>:<port>/health', file=sys.stderr)
|
||||
|
||||
try:
|
||||
server.serve_forever()
|
||||
except KeyboardInterrupt:
|
||||
print('\nShutting down', file=sys.stderr)
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
@@ -1,55 +0,0 @@
|
||||
#!/bin/bash
|
||||
# exec-sign.sh — call the bl exec-constrained server with SSH-signature auth.
|
||||
# No bearer token, no secret crosses the wire: you sign the request envelope
|
||||
# with your registered fleet key and the server verifies it against the
|
||||
# signers file. A signature is not a secret, so this never trips
|
||||
# secret-handling guardrails.
|
||||
#
|
||||
# Server: exec-constrained.py — named ops ONLY, no arbitrary shell.
|
||||
# Signature namespace: exec-constrained
|
||||
# Envelope: {"op","args","ts","nonce"} — op must be in the server allowlist:
|
||||
# dm.send, dm.thread, dm.read, job.run, chat.messages, chat.send,
|
||||
# health.check, exec.ping
|
||||
# Nonce: 16+ hex chars, replay-protected server-side. ts: unix epoch, ±300s skew.
|
||||
#
|
||||
# Usage: exec-sign.sh <op> '<args-json>' [identity] [keyfile] [url]
|
||||
# op a named op, e.g. exec.ping
|
||||
# args-json JSON object of the op's arguments, e.g. '{}'
|
||||
# identity defaults to operator-646 (must be a principal in the
|
||||
# server's signers file)
|
||||
# keyfile defaults to ~/.ssh/id_frontdoor
|
||||
# url defaults to https://exec.muse-dev.online/exec (cloudflared).
|
||||
# NOTE: the old VM Caddy /exec route to bl:8443 died with
|
||||
# exec-server.py — do not point this at the sslip.io URL.
|
||||
set -euo pipefail
|
||||
OP="${1:?usage: exec-sign.sh <op> '<args-json>' [identity] [keyfile] [url]}"
|
||||
# NOTE: do NOT write this as ${2:-{}} — bash matches the first } as the
|
||||
# expansion's close brace and appends a literal } when $2 is set.
|
||||
ARGS_JSON="${2-}"
|
||||
if [ -z "$ARGS_JSON" ]; then ARGS_JSON='{}'; fi
|
||||
IDENTITY="${3:-operator-646}"
|
||||
KEY="${4:-$HOME/.ssh/id_frontdoor}"
|
||||
URL="${5:-https://exec.muse-dev.online/exec}"
|
||||
|
||||
TS=$(date +%s)
|
||||
NONCE=$(python3 -c "import secrets; print(secrets.token_hex(16))")
|
||||
PAYLOAD=$(python3 -c "
|
||||
import json, sys
|
||||
op, args_json, ts, nonce = sys.argv[1:5]
|
||||
args = json.loads(args_json)
|
||||
if not isinstance(args, dict):
|
||||
raise SystemExit('args-json must be a JSON object')
|
||||
print(json.dumps({'op': op, 'args': args, 'ts': int(ts), 'nonce': nonce}))
|
||||
" "$OP" "$ARGS_JSON" "$TS" "$NONCE")
|
||||
SIG=$(printf '%s' "$PAYLOAD" | ssh-keygen -Y sign -f "$KEY" -n exec-constrained)
|
||||
BODY=$(python3 -c "
|
||||
import json, sys
|
||||
ident, payload, sig = sys.argv[1:4]
|
||||
print(json.dumps({'identity': ident, 'payload': payload, 'signature': sig}))
|
||||
" "$IDENTITY" "$PAYLOAD" "$SIG")
|
||||
# Cloudflare Bot Fight Mode blocks python-urllib POSTs (error 1010):
|
||||
# use curl with a browser User-Agent instead.
|
||||
curl -sS -X POST "$URL" \
|
||||
-H 'Content-Type: application/json' \
|
||||
-H 'User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36' \
|
||||
--data "$BODY"
|
||||
@@ -1,59 +0,0 @@
|
||||
#!/bin/bash
|
||||
# exec-watch.sh — 15-min watchdog for the bl exec server's signature-auth path.
|
||||
# Signs a canary op as the exec-canary identity and POSTs it to the
|
||||
# local exec-constrained server. Records result in /home/super/.exec-watch-status.json
|
||||
# and appends to logs/exec-watch.log. Failures are pull-based (status file +
|
||||
# log); wire push alerting here if the fleet wants paging.
|
||||
#
|
||||
# Runs via systemd user timer exec-watch.timer (OnUnitActiveSec=15min).
|
||||
# Server: exec-constrained.py — named ops ONLY, no arbitrary shell;
|
||||
# signature namespace: exec-constrained, with {op,args,ts,nonce} envelope.
|
||||
# NOTE: signers file (/home/super/.exec-signers) is synced MANUALLY from the
|
||||
# VM's /srv/board/allowed_signers on identity renames/adds — bl cannot ssh
|
||||
# back to the VM, so there is no pull sync. Operator step, documented in
|
||||
# docs/TOKEN_POLICY.md.
|
||||
set -uo pipefail
|
||||
KEY=/home/super/.exec-canary
|
||||
URL=https://100.123.153.75:8444/exec
|
||||
STATUS=/home/super/.exec-watch-status.json
|
||||
LOG=/home/super/Projects/NetVM/logs/exec-watch.log
|
||||
|
||||
ts=$(date +%s)
|
||||
nonce=$(python3 -c "import secrets; print(secrets.token_hex(16))")
|
||||
payload=$(python3 -c "import json,sys; print(json.dumps({'op':'exec.ping','args':{},'ts':int(sys.argv[1]),'nonce':sys.argv[2]}))" "$ts" "$nonce")
|
||||
sig=$(printf '%s' "$payload" | ssh-keygen -Y sign -f "$KEY" -n exec-constrained 2>/dev/null)
|
||||
if [ -z "${sig:-}" ]; then
|
||||
result="sign-failed"
|
||||
else
|
||||
body=$(python3 -c "import json,sys; print(json.dumps({'identity': 'exec-canary', 'payload': sys.argv[1], 'signature': sys.argv[2]}))" "$payload" "$sig")
|
||||
out=$(curl -sk -m 25 -X POST "$URL" -H 'Content-Type: application/json' -d "$body" 2>/dev/null)
|
||||
if echo "$out" | grep -q '"rc": 0'; then
|
||||
result="ok"
|
||||
else
|
||||
result="bad-response"
|
||||
fi
|
||||
fi
|
||||
|
||||
now_iso=$(date -u +%FT%TZ)
|
||||
{
|
||||
python3 - "$STATUS" "$now_iso" "$result" <<'PYEOF'
|
||||
import json, sys
|
||||
status_path, now_iso, result = sys.argv[1], sys.argv[2], sys.argv[3]
|
||||
try:
|
||||
st = json.load(open(status_path))
|
||||
except Exception:
|
||||
st = {}
|
||||
if result == "ok":
|
||||
st.update({"last_ok": now_iso, "last_fail": None,
|
||||
"consecutive_failures": 0, "result": "ok"})
|
||||
else:
|
||||
st.update({"last_ok": st.get("last_ok"),
|
||||
"last_fail": now_iso,
|
||||
"consecutive_failures": st.get("consecutive_failures", 0) + 1,
|
||||
"result": result})
|
||||
json.dump(st, open(status_path, "w"))
|
||||
print(f"[{now_iso}] exec-watch: {result} "
|
||||
f"(consecutive_failures={st['consecutive_failures']})")
|
||||
PYEOF
|
||||
} >> "$LOG" 2>&1
|
||||
[ "$result" = "ok" ]
|
||||
@@ -1,8 +0,0 @@
|
||||
#!/bin/bash
|
||||
# flap-check.sh - count browser relaunches per profile in the last hour
|
||||
# Called by the browser-flap-detector cron. Avoids nested SSH quoting hell.
|
||||
cutoff=$(date -u -d "1 hour ago" +%Y-%m-%dT%H:%M:%S)
|
||||
for prof in muse pip 646 opm; do
|
||||
n=$(grep "\[$prof\]" /home/super/Projects/NetVM/chromebox-watchdog.log 2>/dev/null | grep "relaunch OK" | awk -v d="$cutoff" '{ts=substr($1,2,19); if (ts > d) c++} END {print c+0}')
|
||||
echo "$prof:$n"
|
||||
done
|
||||
@@ -1,307 +0,0 @@
|
||||
#!/bin/bash
|
||||
# fleet-alert-check.sh — bl-side critical-condition detector for the fleet alerting pipeline.
|
||||
#
|
||||
# Closes the watchdog gap: agent-health.sh logs CRITICAL and restarts browsers,
|
||||
# but NOTHING pages anyone. This script detects critical conditions, counts
|
||||
# CONSECUTIVE failures, and emits alert records to an outbox that the
|
||||
# container-side fleet-alert-relay hook picks up and pages to #lobby.
|
||||
#
|
||||
# Checks (bl-side only; VM-side checks live in the container relay):
|
||||
# cdp:<node> headless Chromium CDP port not listening in the node's netns
|
||||
# (catches zombie browsers: process alive, CDP not bound)
|
||||
#
|
||||
# Paging policy (env-overridable defaults — adjustable, not gates):
|
||||
# FLEET_ALERT_THRESHOLD=2 consecutive failures before first page (~10 min at 5-min cadence)
|
||||
# FLEET_ALERT_REALERT_MIN=30 re-page while still critical, at most every 30 min
|
||||
# FLEET_ALERT_QUIET_HOURS="" e.g. "23:00-07:00" (bl local time); empty = page 24/7.
|
||||
# First alert for a NEW incident always pages;
|
||||
# quiet hours only suppress re-pages.
|
||||
# FLEET_ALERT_DRY_RUN=1 evaluate + print, write no state/outbox, no notify
|
||||
# FLEET_ALERT_INJECT_FAIL= test hook: comma-separated condition ids to force-fail
|
||||
# (e.g. FLEET_ALERT_INJECT_FAIL=cdp:pip)
|
||||
#
|
||||
# State: ~/.local/share/fleet-alert/state.json (per-condition consecutive counters)
|
||||
# Outbox: ~/.local/share/fleet-alert/outbox.jsonl (ALERT/RECOVERY records for the relay)
|
||||
# Log: /tmp/fleet-alert-check.log
|
||||
#
|
||||
# Alert delivery legs:
|
||||
# 1. outbox record -> container relay -> signed #lobby post (primary page)
|
||||
# 2. best-effort `box-ctl.py notify` to currently-healthy agents (DM path needs a
|
||||
# working browser; failures are logged, never fatal)
|
||||
#
|
||||
# Installed as user timer fleet-alert-check.timer (every 5 min), mirroring agent-health.timer.
|
||||
set -uo pipefail
|
||||
|
||||
THRESHOLD="${FLEET_ALERT_THRESHOLD:-2}"
|
||||
REALERT_MIN="${FLEET_ALERT_REALERT_MIN:-30}"
|
||||
QUIET_HOURS="${FLEET_ALERT_QUIET_HOURS:-}"
|
||||
DRY_RUN="${FLEET_ALERT_DRY_RUN:-0}"
|
||||
INJECT_FAIL="${FLEET_ALERT_INJECT_FAIL:-}"
|
||||
STATE_DIR="${FLEET_ALERT_STATE_DIR:-$HOME/.local/share/fleet-alert}"
|
||||
BIN="$(cd "$(dirname "$0")" && pwd)"
|
||||
STATE="$STATE_DIR/state.json"
|
||||
OUTBOX="$STATE_DIR/outbox.jsonl"
|
||||
LOG="/tmp/fleet-alert-check.log"
|
||||
|
||||
mkdir -p "$STATE_DIR"
|
||||
[ -f "$STATE" ] || echo '{}' > "$STATE"
|
||||
touch "$OUTBOX"
|
||||
|
||||
NOW=$(date +%s)
|
||||
log() { echo "$(date -Iseconds) $*" >> "$LOG"; }
|
||||
|
||||
# --- shared consecutive-failure state machine (also used by the container relay) ---
|
||||
# usage: state_machine <cond> <failing 0|1> -> prints "<ACTION> <fails>"
|
||||
# ACTION: ALERT_FIRST | ALERT_REALERT | RECOVERY | SUPPRESSED | NONE
|
||||
state_machine() {
|
||||
local cond="$1" failing="$2"
|
||||
THRESHOLD="$THRESHOLD" REALERT_MIN="$REALERT_MIN" QUIET_HOURS="$QUIET_HOURS" \
|
||||
FLEET_ALERT_DRY_RUN="$DRY_RUN" python3 - "$STATE" "$cond" "$failing" <<'PYEOF'
|
||||
import json, os, sys, time
|
||||
state_path, cond, failing_s = sys.argv[1], sys.argv[2], sys.argv[3]
|
||||
failing = failing_s == "1"
|
||||
threshold = int(os.environ.get("THRESHOLD", "2"))
|
||||
realert_min = int(os.environ.get("REALERT_MIN", "30"))
|
||||
qh = os.environ.get("QUIET_HOURS", "")
|
||||
dry = os.environ.get("FLEET_ALERT_DRY_RUN") == "1"
|
||||
now = int(time.time())
|
||||
def in_quiet(spec):
|
||||
if not spec:
|
||||
return False
|
||||
try:
|
||||
a, b = spec.split("-")
|
||||
def m(s):
|
||||
h, mi = s.split(":")
|
||||
return int(h) * 60 + int(mi)
|
||||
cur = time.localtime().tm_hour * 60 + time.localtime().tm_min
|
||||
s, e = m(a), m(b)
|
||||
return (s <= cur < e) if s <= e else (cur >= s or cur < e)
|
||||
except Exception:
|
||||
return False
|
||||
try:
|
||||
st = json.load(open(state_path))
|
||||
except Exception:
|
||||
st = {}
|
||||
e = st.get(cond) or {"fails": 0, "alerted": False, "last_alert_ts": 0}
|
||||
action = "NONE"
|
||||
if failing:
|
||||
e["fails"] = int(e.get("fails", 0)) + 1
|
||||
due = e["fails"] >= threshold and (
|
||||
not e.get("alerted") or now - int(e.get("last_alert_ts", 0)) >= realert_min * 60
|
||||
)
|
||||
if due:
|
||||
first = not e.get("alerted")
|
||||
if not first and in_quiet(qh):
|
||||
action = "SUPPRESSED"
|
||||
else:
|
||||
action = "ALERT_FIRST" if first else "ALERT_REALERT"
|
||||
e["alerted"] = True
|
||||
e["last_alert_ts"] = now
|
||||
else:
|
||||
if e.get("alerted"):
|
||||
action = "RECOVERY"
|
||||
e["fails"] = 0
|
||||
e["alerted"] = False
|
||||
st[cond] = e
|
||||
if not dry:
|
||||
json.dump(st, open(state_path, "w"))
|
||||
print(action, e["fails"])
|
||||
PYEOF
|
||||
}
|
||||
|
||||
emit_record() { # kind cond detail consecutive
|
||||
local kind="$1" cond="$2" detail="$3" consec="$4"
|
||||
local id
|
||||
id=$(tr -d '-' < /proc/sys/kernel/random/uuid | cut -c1-12)
|
||||
local rec
|
||||
rec=$(python3 -c 'import json,sys; print(json.dumps({"id":sys.argv[1],"ts":int(sys.argv[2]),"kind":sys.argv[3],"source":"bl","condition":sys.argv[4],"detail":sys.argv[5],"consecutive":int(sys.argv[6])}))' \
|
||||
"$id" "$NOW" "$kind" "$cond" "$detail" "$consec")
|
||||
if [ "$DRY_RUN" = "1" ]; then
|
||||
log "DRY-RUN would emit: $rec"
|
||||
else
|
||||
echo "$rec" >> "$OUTBOX"
|
||||
log "emitted $kind $cond x$consec"
|
||||
fi
|
||||
}
|
||||
|
||||
box_notify() {
|
||||
# Best-effort DM to healthy agents via box-ctl. Never fatal.
|
||||
# Fan-out runs in parallel with a per-notify timeout so one hung DM
|
||||
# path can't stall the 5-minute check loop.
|
||||
local cond="$1"
|
||||
local detail="$2"
|
||||
local msg="[fleet-alert] CRITICAL ${cond}: ${detail}"
|
||||
msg="${msg:0:240}"
|
||||
local agent pids=""
|
||||
for agent in $HEALTHY_AGENTS; do
|
||||
if [ "$DRY_RUN" = "1" ]; then
|
||||
log "DRY-RUN would notify $agent"
|
||||
continue
|
||||
fi
|
||||
(
|
||||
if timeout 60 python3 "$BIN/box-ctl.py" notify "$agent" "$msg" >/dev/null 2>&1; then
|
||||
log "notified $agent re $cond"
|
||||
else
|
||||
log "notify $agent failed re $cond (best-effort)"
|
||||
fi
|
||||
) &
|
||||
pids="$pids $!"
|
||||
done
|
||||
local p
|
||||
for p in $pids; do wait "$p" 2>/dev/null; done
|
||||
}
|
||||
|
||||
injected() { # cond -> 0 if injected-fail
|
||||
case ",$INJECT_FAIL," in *,"$1,"*) return 0;; *) return 1;; esac
|
||||
}
|
||||
|
||||
HEALTHY_AGENTS=""
|
||||
|
||||
"$BIN/netvm-registry.py" 2>/dev/null | while IFS=: read -r node port; do
|
||||
[ -n "$node" ] && [ -n "$port" ] || continue
|
||||
cond="cdp:$node"
|
||||
procs=$(pgrep -f "chromium.*profiles/$node" 2>/dev/null | wc -l)
|
||||
if sudo -n ip netns exec "warp-$node" ss -tln 2>/dev/null | grep -q ":$port "; then
|
||||
failing=0
|
||||
HEALTHY_AGENTS="$HEALTHY_AGENTS $node"
|
||||
else
|
||||
failing=1
|
||||
fi
|
||||
injected "$cond" && failing=1
|
||||
detail="CDP $port not listening in netns warp-$node (chromium procs=$procs)"
|
||||
# NOTE: HEALTHY_AGENTS set inside the pipeline subshell is lost; recompute below.
|
||||
read -r action fails < <(state_machine "$cond" "$failing")
|
||||
case "$action" in
|
||||
ALERT_FIRST|ALERT_REALERT)
|
||||
emit_record "ALERT" "$cond" "$detail" "$fails"
|
||||
echo "$cond" >> "$STATE_DIR/.alerts.tmp"
|
||||
;;
|
||||
RECOVERY)
|
||||
emit_record "RECOVERY" "$cond" "CDP $port listening again in netns warp-$node" "$fails"
|
||||
;;
|
||||
SUPPRESSED)
|
||||
log "$cond still critical x$fails — re-page suppressed by quiet hours ($QUIET_HOURS)"
|
||||
;;
|
||||
esac
|
||||
done
|
||||
|
||||
# --- Agent approval blockage detection (catches agents held up on approvals) ---
|
||||
"$BIN/netvm-registry.py" 2>/dev/null | while IFS=: read -r node port; do
|
||||
[ -n "$node" ] || continue
|
||||
cond="approval:$node"
|
||||
pending_info=$(python3 -c "
|
||||
import sys
|
||||
sys.path.insert(0, '$BIN')
|
||||
import approvals
|
||||
info = approvals.inspect_node_approvals('$node')
|
||||
if info.get('has_pending'):
|
||||
print(f\"{info.get('ip') or 'unknown'}|{info.get('title') or ''}\")
|
||||
" 2>/dev/null || true)
|
||||
|
||||
if [ -n "$pending_info" ]; then
|
||||
failing=1
|
||||
target="${pending_info%%|*}"
|
||||
detail="Agent $node held up on browser approval for $target"
|
||||
else
|
||||
failing=0
|
||||
detail="Agent $node approvals clear"
|
||||
fi
|
||||
injected "$cond" && failing=1
|
||||
read -r action fails < <(state_machine "$cond" "$failing")
|
||||
case "$action" in
|
||||
ALERT_FIRST|ALERT_REALERT)
|
||||
emit_record "ALERT" "$cond" "$detail" "$fails"
|
||||
echo "$cond" >> "$STATE_DIR/.alerts.tmp"
|
||||
;;
|
||||
RECOVERY)
|
||||
emit_record "RECOVERY" "$cond" "$detail" "$fails"
|
||||
;;
|
||||
SUPPRESSED)
|
||||
log "$cond still critical x$fails — re-page suppressed"
|
||||
;;
|
||||
esac
|
||||
done
|
||||
|
||||
# --- Warp partition detection (2026-10-04) ---
|
||||
# A partitioned node has a live browser + CDP but no internet egress: the
|
||||
# chromebox watchdog sees a healthy browser while all automation fails.
|
||||
# Condition id: partition:<node>. The detail names the node, the WireGuard
|
||||
# handshake age, the egress probe result, and the timestamp, and says
|
||||
# PARTITION explicitly so #lobby readers can tell a network partition from
|
||||
# a browser crash at a glance. Anti-spam comes from the shared consecutive-
|
||||
# failure state machine (2 consecutive failures before first page, re-page
|
||||
# at most every 30 min).
|
||||
WARP_PROBE_URL="${WARP_PROBE_URL:-https://1.1.1.1/cdn-cgi/trace}"
|
||||
warp_partition_probe() { # <node> -> prints "<handshake_age_s|unknown> <ok|FAIL>"
|
||||
local node="$1" iface epoch now age_s
|
||||
iface=$(sudo -n ip netns exec "warp-$node" sh -c 'wg show interfaces 2>/dev/null | head -1')
|
||||
now=$(date +%s)
|
||||
epoch=$(sudo -n ip netns exec "warp-$node" wg show "$iface" latest-handshakes 2>/dev/null | awk '{print $2}')
|
||||
case "$epoch" in ''|*[!0-9]*) age_s="unknown" ;; *) age_s=$(( now - epoch )) ;; esac
|
||||
if sudo -n ip netns exec "warp-$node" curl -s -m 8 -o /dev/null "$WARP_PROBE_URL" 2>/dev/null; then
|
||||
echo "$age_s ok"
|
||||
else
|
||||
echo "$age_s FAIL"
|
||||
fi
|
||||
}
|
||||
|
||||
"$BIN/netvm-registry.py" 2>/dev/null | while IFS=: read -r node port; do
|
||||
[ -n "$node" ] && [ -n "$port" ] || continue
|
||||
cond="partition:$node"
|
||||
read -r hs_age probe_res < <(warp_partition_probe "$node")
|
||||
ts=$(date -u +%FT%TZ)
|
||||
if [ "$probe_res" = "ok" ]; then
|
||||
failing=0
|
||||
detail="warp egress restored for $node at $ts (probe $WARP_PROBE_URL ok)"
|
||||
else
|
||||
failing=1
|
||||
detail="PARTITION $node: warp egress down at $ts (handshake ${hs_age}s ago, probe $WARP_PROBE_URL FAILED)"
|
||||
fi
|
||||
if injected "$cond"; then
|
||||
failing=1
|
||||
detail="PARTITION $node: warp egress down at $ts (handshake ${hs_age}s ago, probe $WARP_PROBE_URL FAILED) [INJECTED]"
|
||||
fi
|
||||
read -r action fails < <(state_machine "$cond" "$failing")
|
||||
case "$action" in
|
||||
ALERT_FIRST|ALERT_REALERT)
|
||||
emit_record "ALERT" "$cond" "$detail" "$fails"
|
||||
echo "$cond" >> "$STATE_DIR/.alerts.tmp"
|
||||
;;
|
||||
RECOVERY)
|
||||
emit_record "RECOVERY" "$cond" "$detail" "$fails"
|
||||
;;
|
||||
SUPPRESSED)
|
||||
log "$cond still critical x$fails — re-page suppressed by quiet hours ($QUIET_HOURS)"
|
||||
;;
|
||||
esac
|
||||
done
|
||||
|
||||
# Recompute healthy agents in the main shell (pipeline subshell above can't export).
|
||||
HEALTHY_AGENTS=""
|
||||
"$BIN/netvm-registry.py" 2>/dev/null | while IFS=: read -r node port; do
|
||||
[ -n "$node" ] && [ -n "$port" ] || continue
|
||||
if sudo -n ip netns exec "warp-$node" ss -tln 2>/dev/null | grep -q ":$port "; then
|
||||
echo "$node"
|
||||
fi
|
||||
done > "$STATE_DIR/.healthy.tmp"
|
||||
HEALTHY_AGENTS=$(tr '\n' ' ' < "$STATE_DIR/.healthy.tmp")
|
||||
rm -f "$STATE_DIR/.healthy.tmp"
|
||||
|
||||
# Notify for this run's alerts (best effort). Skip entirely when nothing is healthy
|
||||
# (notify needs a working browser via dm.py) or in dry-run.
|
||||
if [ -n "$HEALTHY_AGENTS" ] && [ -f "$STATE_DIR/.alerts.tmp" ]; then
|
||||
while read -r cond; do
|
||||
[ -n "$cond" ] && box_notify "$cond" "see #lobby for detail"
|
||||
done < "$STATE_DIR/.alerts.tmp"
|
||||
elif [ -f "$STATE_DIR/.alerts.tmp" ]; then
|
||||
log "no healthy agents — box notify skipped (DM path needs a working browser)"
|
||||
fi
|
||||
rm -f "$STATE_DIR/.alerts.tmp"
|
||||
|
||||
tail -500 "$LOG" > "$LOG.tmp" 2>/dev/null && mv "$LOG.tmp" "$LOG"
|
||||
log "check complete"
|
||||
|
||||
# Relay pending outbox records to #lobby with idempotency gates (posted watermark + content hash TTL)
|
||||
if [ "$DRY_RUN" -eq 0 ] && [ -x "$BIN/fleet-alert-relay.sh" ]; then
|
||||
"$BIN/fleet-alert-relay.sh" >> "$LOG" 2>&1 || true
|
||||
fi
|
||||
@@ -1,313 +0,0 @@
|
||||
#!/bin/bash
|
||||
# bin/fleet-alert-relay.sh — idempotent relay: fleet-alert outbox.jsonl -> #lobby.
|
||||
#
|
||||
# PROPOSAL ONLY (board ticket ae28ac8b735f). No bl infra is touched by this
|
||||
# branch; deployment needs opm review + sign-off. See
|
||||
# docs/FLEET-ALERT-DUP-POST-GATE.md.
|
||||
#
|
||||
# The 2026-10-05 11:28Z incident: one RECOVERY record in the outbox became two
|
||||
# identical verified #lobby posts (seq 642/643, 3.35s apart) because the relay
|
||||
# leg had no idempotency: append-only outbox, no consume tracking, no content
|
||||
# dedup, and the chat POST API has no idempotency key.
|
||||
#
|
||||
# Gates implemented here:
|
||||
# 1. posted-watermark — each posted outbox record id is appended to
|
||||
# posted.log; records already posted are skipped.
|
||||
# 2. content-hash + 10-min TTL — sha256 of the formatted alert text in a
|
||||
# file-backed seen-set (seen-hashes.log); identical text re-posts inside
|
||||
# the window are dropped. Catches same-record re-posts AND identical
|
||||
# text from different records.
|
||||
# 3. retry discipline — never blind-retry a chat POST after a timeout /
|
||||
# phantom-000. Read the #lobby tail first; re-post only if absent.
|
||||
#
|
||||
# Usage:
|
||||
# fleet-alert-relay.sh [--dry-run] [--self-test]
|
||||
#
|
||||
# Env: FLEET_ALERT_DIR (default ~/.local/share/fleet-alert),
|
||||
# CHAT_BASE (default https://chat.muse-dev.online),
|
||||
# CHAT_IDENTITY (default operator-646), CHAT_KEYFILE (default ~/.ssh/id_frontdoor)
|
||||
set -euo pipefail
|
||||
|
||||
ALERT_DIR="${FLEET_ALERT_DIR:-$HOME/.local/share/fleet-alert}"
|
||||
OUTBOX="$ALERT_DIR/outbox.jsonl"
|
||||
POSTED="$ALERT_DIR/posted.log"
|
||||
SEEN="$ALERT_DIR/seen-hashes.log"
|
||||
LOCKF="$ALERT_DIR/relay.lock"
|
||||
|
||||
CHAT_BASE="${CHAT_BASE:-https://chat.muse-dev.online}"
|
||||
API="$CHAT_BASE/api/chat"
|
||||
IDENTITY="${CHAT_IDENTITY:-operator-646}"
|
||||
KEYFILE="${CHAT_KEYFILE:-$HOME/.ssh/id_frontdoor}"
|
||||
CHANNEL="#lobby"
|
||||
TTL_SECS=600
|
||||
UA='Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0 Safari/537.36'
|
||||
|
||||
DRY_RUN=0
|
||||
log() { echo "[fleet-alert-relay] $*" >&2; }
|
||||
|
||||
# ---------------------------------------------------------------- transport
|
||||
# Factored so --self-test can stub them. Return contracts:
|
||||
# transport_tail -> prints "<ts> <identity> <message>" lines for recent #lobby (one per line, tab-safe: message may contain spaces, fields split on first two spaces)
|
||||
# transport_post "<text>" -> prints "OK <msg-id>" or "UNKNOWN <reason>"; exit 0 always (callers decide)
|
||||
|
||||
transport_tail() {
|
||||
curl -s -m 15 -A "$UA" "$API/history?channel=%23lobby&limit=50" \
|
||||
| python3 -c '
|
||||
import json,sys
|
||||
try:
|
||||
d=json.load(sys.stdin)
|
||||
except Exception:
|
||||
sys.exit(0)
|
||||
for m in d.get("messages",[]):
|
||||
msg=(m.get("message") or "").replace(chr(10)," ")
|
||||
print(m.get("ts"), m.get("identity"), msg)
|
||||
'
|
||||
}
|
||||
|
||||
transport_post() { # $1 = text
|
||||
local text="$1" ts sig payload resp http
|
||||
ts="$(date +%s)"
|
||||
if ! sig="$(sign_payload "$(printf '%s\n%s\n%s' "$ts" "$CHANNEL" "$text")")"; then
|
||||
echo "UNKNOWN sign-failed"; return 0
|
||||
fi
|
||||
payload="$(MSG="$text" TS="$ts" SIG="$sig" python3 -c '
|
||||
import json,os
|
||||
print(json.dumps({"channel":os.environ["CHANNEL"],"identity":os.environ["IDENTITY"],
|
||||
"message":os.environ["MSG"],"ts":int(os.environ["TS"]),"signature":os.environ["SIG"]}))')"
|
||||
resp="$(curl -s -m 20 -A "$UA" -X POST -H 'Content-Type: application/json' \
|
||||
-d "$payload" -w '\n%{http_code}' "$API/post" 2>/dev/null || echo -e '\n000')"
|
||||
http="$(printf '%s' "$resp" | tail -n1)"
|
||||
resp="$(printf '%s' "$resp" | sed '$d')"
|
||||
case "$http" in
|
||||
2*)
|
||||
local mid
|
||||
mid="$(printf '%s' "$resp" | python3 -c '
|
||||
import json,sys
|
||||
try:
|
||||
d=json.load(sys.stdin); print(d.get("id","") if d.get("ok") else "")
|
||||
except Exception:
|
||||
print("")')"
|
||||
if [ -n "$mid" ]; then echo "OK $mid"; else echo "UNKNOWN bad-body-$http"; fi
|
||||
;;
|
||||
*) echo "UNKNOWN http-$http" ;;
|
||||
esac
|
||||
return 0
|
||||
}
|
||||
|
||||
# file-based signing (never pipe): stdin-piped signatures intermittently
|
||||
# fail verification (2026-10-02 incident).
|
||||
sign_payload() { # $1 = payload -> armored signature
|
||||
local tmpd pf
|
||||
tmpd="$(mktemp -d)" || return 1
|
||||
pf="$tmpd/payload"
|
||||
printf '%s' "$1" > "$pf"
|
||||
rm -f "$pf.sig"
|
||||
ssh-keygen -Y sign -f "$KEYFILE" -n chat "$pf" </dev/null >/dev/null 2>&1 \
|
||||
|| { rm -rf "$tmpd"; return 1; }
|
||||
cat "$pf.sig"
|
||||
rm -rf "$tmpd"
|
||||
}
|
||||
|
||||
# ----------------------------------------------------------------- formatting
|
||||
# Canonical text for one outbox record (JSON on stdin):
|
||||
# ALERT -> [fleet-alert] CRITICAL: <node> warp PARTITION x<n>
|
||||
# RECOVERY -> [fleet-alert] RECOVERED: <node> warp PARTITION
|
||||
# condition is "<type>:<node>" (partition|cdp).
|
||||
format_text() {
|
||||
python3 -c '
|
||||
import json,sys
|
||||
r=json.load(sys.stdin)
|
||||
cond=r.get("condition","")
|
||||
typ,_,node=cond.partition(":")
|
||||
label={"partition":"PARTITION","cdp":"CDP"}.get(typ,typ.upper())
|
||||
kind=r.get("kind","")
|
||||
if kind=="ALERT":
|
||||
n=r.get("consecutive",r.get("count","?"))
|
||||
print("[fleet-alert] CRITICAL: %s warp %s x%s" % (node,label,n))
|
||||
elif kind=="RECOVERY":
|
||||
print("[fleet-alert] RECOVERED: %s warp %s" % (node,label))
|
||||
else:
|
||||
print("[fleet-alert] %s: %s warp %s" % (kind,node,label))
|
||||
'
|
||||
}
|
||||
|
||||
content_hash() { printf '%s' "$1" | sha256sum | cut -d' ' -f1; }
|
||||
|
||||
# ------------------------------------------------------------- gate 1: watermark
|
||||
already_posted() { # $1 = record id
|
||||
[ -f "$POSTED" ] && grep -qxF "$1" "$POSTED"
|
||||
}
|
||||
mark_posted() { # $1 = record id
|
||||
printf '%s\n' "$1" >> "$POSTED"
|
||||
}
|
||||
|
||||
# ------------------------------------------------------- gate 2: content-hash TTL
|
||||
seen_recently() { # $1 = hash -> 0 if seen within TTL
|
||||
local h="$1" now ts
|
||||
[ -f "$SEEN" ] || return 1
|
||||
now="$(date +%s)"
|
||||
while read -r h2 ts _rest; do
|
||||
[ "$h2" = "$h" ] || continue
|
||||
if [ "$(( now - ts ))" -lt "$TTL_SECS" ]; then return 0; fi
|
||||
done < "$SEEN"
|
||||
return 1
|
||||
}
|
||||
mark_seen() { # $1 = hash
|
||||
printf '%s %s\n' "$1" "$(date +%s)" >> "$SEEN"
|
||||
}
|
||||
prune_seen() {
|
||||
local now tmp
|
||||
[ -f "$SEEN" ] || return 0
|
||||
now="$(date +%s)"; tmp="$(mktemp)"
|
||||
while read -r h ts _rest; do
|
||||
[ -n "$h" ] && [ "$(( now - ts ))" -lt "$TTL_SECS" ] && printf '%s %s\n' "$h" "$ts" >> "$tmp"
|
||||
done < "$SEEN"
|
||||
mv "$tmp" "$SEEN"
|
||||
}
|
||||
|
||||
# ------------------------------------------------- gate 3: check-before-(re)post
|
||||
# 0 if the exact text appears in the recent #lobby tail.
|
||||
lobby_has_text() { # $1 = text
|
||||
local want="$1"
|
||||
# transport_tail prints "<ts> <identity> <message>" per line; the message
|
||||
# itself may contain spaces, so strip only the first two fields.
|
||||
transport_tail | python3 -c '
|
||||
import sys
|
||||
want=sys.argv[1]
|
||||
for line in sys.stdin:
|
||||
parts=line.rstrip("\n").split(" ",2)
|
||||
if len(parts)==3 and parts[2]==want:
|
||||
sys.exit(0)
|
||||
sys.exit(1)' "$want"
|
||||
}
|
||||
|
||||
relay_record() { # $1 = record id, $2 = formatted text
|
||||
local rid="$1" text="$2" h out
|
||||
h="$(content_hash "$text")"
|
||||
|
||||
# Gate 1: posted watermark
|
||||
if already_posted "$rid"; then
|
||||
log "skip $rid: already in posted.log"
|
||||
return 0
|
||||
fi
|
||||
# Gate 2: content hash within TTL
|
||||
if seen_recently "$h"; then
|
||||
log "skip $rid: identical text posted <10min ago (hash $h)"
|
||||
mark_posted "$rid" # consume it so it never retries later
|
||||
return 0
|
||||
fi
|
||||
# Gate 3a: somebody else already posted it (covers agent-driven relay overlap)
|
||||
if lobby_has_text "$text"; then
|
||||
log "skip $rid: text already in #lobby tail"
|
||||
mark_posted "$rid"; mark_seen "$h"
|
||||
return 0
|
||||
fi
|
||||
|
||||
if [ "$DRY_RUN" = 1 ]; then
|
||||
log "dry-run: would post $rid: $text"
|
||||
return 0
|
||||
fi
|
||||
|
||||
# Attempt one POST. Any ambiguous outcome -> verify via tail, never blind-retry.
|
||||
out="$(transport_post "$text")"
|
||||
case "$out" in
|
||||
OK*)
|
||||
log "posted $rid -> ${out#OK }"
|
||||
mark_posted "$rid"; mark_seen "$h"
|
||||
return 0
|
||||
;;
|
||||
UNKNOWN*)
|
||||
log "post $rid outcome UNKNOWN (${out#UNKNOWN }); checking #lobby tail before any retry"
|
||||
sleep 2
|
||||
if lobby_has_text "$text"; then
|
||||
log "post $rid landed despite ${out#UNKNOWN } — marking posted, no retry"
|
||||
mark_posted "$rid"; mark_seen "$h"
|
||||
return 0
|
||||
fi
|
||||
log "post $rid absent from tail — ONE retry only"
|
||||
out="$(transport_post "$text")"
|
||||
case "$out" in
|
||||
OK*)
|
||||
log "retry posted $rid -> ${out#OK }"
|
||||
mark_posted "$rid"; mark_seen "$h"
|
||||
return 0
|
||||
;;
|
||||
*)
|
||||
log "ERROR: post $rid still ${out%% *} after one retry; leaving unposted for next run (tail-check will catch it if it landed)"
|
||||
return 1
|
||||
;;
|
||||
esac
|
||||
;;
|
||||
esac
|
||||
}
|
||||
|
||||
main() {
|
||||
local rid rec text fails=0
|
||||
[ -f "$OUTBOX" ] || { log "no outbox ($OUTBOX); nothing to do"; return 0; }
|
||||
mkdir -p "$ALERT_DIR"
|
||||
touch "$POSTED" "$SEEN"
|
||||
prune_seen
|
||||
while IFS= read -r rec; do
|
||||
[ -n "$rec" ] || continue
|
||||
rid="$(printf '%s' "$rec" | python3 -c 'import json,sys; print(json.load(sys.stdin).get("id",""))')"
|
||||
[ -n "$rid" ] || { log "skip record with no id"; continue; }
|
||||
text="$(printf '%s' "$rec" | format_text)"
|
||||
relay_record "$rid" "$text" || fails=$((fails+1))
|
||||
done < "$OUTBOX"
|
||||
return "$fails"
|
||||
}
|
||||
|
||||
# ------------------------------------------------------------------ self-test
|
||||
# Acceptance: two identical alert submissions <10 min apart -> exactly one
|
||||
# #lobby post. Stubs the transport; exercises all three gates.
|
||||
# NOTE: transport_post is invoked via command substitution (subshell), so the
|
||||
# stub counts calls with a file, not a variable.
|
||||
self_test() {
|
||||
local td calls lobby ok=1 n
|
||||
td="$(mktemp -d)"; export FLEET_ALERT_DIR="$td"
|
||||
ALERT_DIR="$td"; OUTBOX="$td/outbox.jsonl"; POSTED="$td/posted.log"
|
||||
SEEN="$td/seen-hashes.log"; LOCKF="$td/relay.lock"
|
||||
calls="$td/calls.log"; lobby="$td/lobby.log"
|
||||
touch "$calls" "$lobby"
|
||||
# Two identical submissions: same text, different record ids (the 11:28Z shape)
|
||||
printf '%s\n' \
|
||||
'{"id":"rec-A","ts":1791199616,"kind":"RECOVERY","condition":"partition:def"}' \
|
||||
'{"id":"rec-B","ts":1791199617,"kind":"RECOVERY","condition":"partition:def"}' \
|
||||
> "$OUTBOX"
|
||||
transport_post() { # stub: 1st call "times out" but the server DID accept it
|
||||
echo "call" >> "$calls"
|
||||
printf '%s\n' "$1" >> "$lobby"
|
||||
n=$(wc -l < "$calls")
|
||||
if [ "$n" = 1 ]; then echo "UNKNOWN phantom-000"; else echo "OK stub-$n"; fi
|
||||
return 0
|
||||
}
|
||||
transport_tail() { # stub: the lobby shows whatever "landed"
|
||||
while IFS= read -r m; do printf '%s opm %s\n' "$(date +%s)" "$m"; done < "$lobby"
|
||||
}
|
||||
main
|
||||
n=$(wc -l < "$calls")
|
||||
[ "$n" -eq 1 ] || { echo "FAIL: expected exactly 1 post call, got $n"; ok=0; }
|
||||
grep -qx "rec-A" "$POSTED" || { echo "FAIL: rec-A not watermarked"; ok=0; }
|
||||
grep -qx "rec-B" "$POSTED" || { echo "FAIL: rec-B not watermarked (gate 2 should consume it)"; ok=0; }
|
||||
# Identical re-submission (new record id, same text) must not post again
|
||||
printf '%s\n' '{"id":"rec-C","ts":1791199700,"kind":"RECOVERY","condition":"partition:def"}' >> "$OUTBOX"
|
||||
main
|
||||
n=$(wc -l < "$calls")
|
||||
[ "$n" -eq 1 ] || { echo "FAIL: identical re-submission posted again (calls=$n)"; ok=0; }
|
||||
rm -rf "$td"
|
||||
[ "$ok" = 1 ] && echo "SELF-TEST PASS: 2 identical submissions -> 1 post call; re-submission -> 0 new calls" && return 0
|
||||
return 1
|
||||
}
|
||||
|
||||
case "${1:-}" in
|
||||
--dry-run) DRY_RUN=1; shift ;;
|
||||
--self-test) self_test; exit $? ;;
|
||||
esac
|
||||
|
||||
# Single-flight the relay (defense in depth; the detector's own flock is separate).
|
||||
exec 9>"$LOCKF"
|
||||
if ! flock -n 9; then
|
||||
log "another relay run holds the lock; exiting"
|
||||
exit 0
|
||||
fi
|
||||
main
|
||||
@@ -1,12 +0,0 @@
|
||||
#!/bin/bash
|
||||
# fleet-status.sh - one-line health per profile: browser up/down + relay code
|
||||
declare -A ports=( [muse]=9410 [pip]=9420 [646]=9430 [opm]=9440 )
|
||||
declare -A veth=( [muse]=10.201.35.2 [pip]=10.201.87.2 [646]=10.201.202.2 [opm]=10.201.157.2 )
|
||||
for p in muse pip 646 opm; do
|
||||
port=${ports[$p]}
|
||||
if pgrep -f "remote-debugging-port=$port" >/dev/null 2>&1; then b=UP; else b=DOWN; fi
|
||||
r=$(curl -s -m 4 -o /dev/null -w "%{http_code}" "http://${veth[$p]}:$port/json/version" 2>/dev/null || echo 000)
|
||||
echo "$p: browser=$b relay=$r"
|
||||
done
|
||||
echo ===
|
||||
tail -20 /home/super/Projects/NetVM/chromebox-watchdog.log | grep -E "FAILED|unhealthy" | tail -4
|
||||
@@ -1,376 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
followup-sweeper.py — Autonomous follow-up deadline tracking and nudge sweeper.
|
||||
|
||||
Monitors pending follow-ups in followups.json, delivers progressive nudges to
|
||||
recipients when deadlines expire (in-thread first, Main Chat on final nudge),
|
||||
and executes terminal escalations to opm when all nudges are exhausted.
|
||||
|
||||
Usage:
|
||||
python3 followup-sweeper.py --once
|
||||
python3 followup-sweeper.py --loop --interval 30
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
from datetime import datetime, timezone, timedelta
|
||||
from pathlib import Path
|
||||
|
||||
# Paths
|
||||
NETVM_ROOT = Path("/home/super/Projects/NetVM")
|
||||
BIN_DIR = NETVM_ROOT / "bin"
|
||||
JOBS_DIR = NETVM_ROOT / "jobs"
|
||||
FOLLOWUPS_FILE = NETVM_ROOT / "followups.json"
|
||||
JOB_LOG = NETVM_ROOT / "job-log.jsonl"
|
||||
DM_LOG = NETVM_ROOT / "dm-log.jsonl"
|
||||
DM_PY = BIN_DIR / "dm.py"
|
||||
DISPATCH_PY = BIN_DIR / "job-dispatch.py"
|
||||
|
||||
sys.path.insert(0, str(BIN_DIR))
|
||||
try:
|
||||
import pipeline_engine
|
||||
HAS_PIPELINE = True
|
||||
except ImportError:
|
||||
HAS_PIPELINE = False
|
||||
|
||||
|
||||
def utcnow_dt():
|
||||
return datetime.now(timezone.utc)
|
||||
|
||||
|
||||
def utcnow_str():
|
||||
return utcnow_dt().isoformat()
|
||||
|
||||
|
||||
def parse_iso(ts_str):
|
||||
if not ts_str:
|
||||
return None
|
||||
try:
|
||||
ts_clean = ts_str.replace("Z", "+00:00")
|
||||
dt = datetime.fromisoformat(ts_clean)
|
||||
if dt.tzinfo is None:
|
||||
dt = dt.replace(tzinfo=timezone.utc)
|
||||
return dt
|
||||
except Exception:
|
||||
return None
|
||||
|
||||
|
||||
def load_followups():
|
||||
if not FOLLOWUPS_FILE.exists():
|
||||
return {}
|
||||
try:
|
||||
with open(FOLLOWUPS_FILE, "r", encoding="utf-8") as f:
|
||||
return json.load(f)
|
||||
except Exception:
|
||||
return {}
|
||||
|
||||
|
||||
def save_followups(data):
|
||||
tmp_path = f"{FOLLOWUPS_FILE}.tmp.{os.getpid()}"
|
||||
with open(tmp_path, "w", encoding="utf-8") as f:
|
||||
json.dump(data, f, indent=2)
|
||||
os.replace(tmp_path, FOLLOWUPS_FILE)
|
||||
|
||||
|
||||
def append_job_log(entry):
|
||||
os.makedirs(os.path.dirname(os.path.abspath(JOB_LOG)), exist_ok=True)
|
||||
with open(JOB_LOG, "a", encoding="utf-8") as f:
|
||||
f.write(json.dumps(entry) + "\n")
|
||||
|
||||
|
||||
def send_dm(sender, recipient, target, text):
|
||||
"""Dispatch a DM via dm.py. The sweeper's delivery target is explicit
|
||||
followup state (or explicit in-code escalation routing), so pass the
|
||||
main-chat opt-in when the target is main (sidechat-first policy)."""
|
||||
cmd = [
|
||||
sys.executable,
|
||||
str(DM_PY),
|
||||
"send",
|
||||
"--agent", sender,
|
||||
"--to", recipient,
|
||||
"--target", target,
|
||||
] + (["--allow-main-chat"] if target == "main" else []) + [
|
||||
text,
|
||||
]
|
||||
try:
|
||||
res = subprocess.run(cmd, capture_output=True, text=True, timeout=90)
|
||||
return res.returncode == 0, res.stdout.strip() or res.stderr.strip()
|
||||
except Exception as e:
|
||||
return False, str(e)
|
||||
|
||||
_DM_ID_RE = re.compile(r"\bDM ([0-9a-f]{8})\b")
|
||||
_THREAD_UUID_RE = re.compile(
|
||||
r"^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$",
|
||||
re.IGNORECASE,
|
||||
)
|
||||
|
||||
def _resolve_nudge_thread_uuid(nudge_output):
|
||||
"""Parse the DM id from dm.py's stdout, scan the tail (last 5000 lines)
|
||||
of dm-log.jsonl for that id's sidechat_autoprovisioned event, falling
|
||||
back to the sent event's tags.thread. Returns the UUID or None."""
|
||||
m = _DM_ID_RE.search(nudge_output or "")
|
||||
if not m:
|
||||
return None
|
||||
nudge_id = m.group(1)
|
||||
fallback = None
|
||||
try:
|
||||
with open(DM_LOG, "r", encoding="utf-8") as f:
|
||||
lines = f.readlines()
|
||||
except FileNotFoundError:
|
||||
return None
|
||||
for line in lines[-5000:]:
|
||||
line = line.strip()
|
||||
if not line:
|
||||
continue
|
||||
try:
|
||||
e = json.loads(line)
|
||||
except Exception:
|
||||
continue
|
||||
if e.get("id") != nudge_id:
|
||||
continue
|
||||
if e.get("type") == "sidechat_autoprovisioned":
|
||||
uuid = e.get("thread_uuid") or ""
|
||||
if _THREAD_UUID_RE.fullmatch(uuid):
|
||||
return uuid
|
||||
elif e.get("type") == "sent":
|
||||
cand = ((e.get("tags") or {}).get("thread")) or ""
|
||||
if _THREAD_UUID_RE.fullmatch(cand):
|
||||
fallback = cand
|
||||
return fallback
|
||||
|
||||
|
||||
def sweep_cycle(dry_run=False):
|
||||
followups = load_followups()
|
||||
if not followups:
|
||||
return {"status": "ok", "pending": 0, "nudges_sent": 0, "escalations": 0}
|
||||
|
||||
now = utcnow_dt()
|
||||
nudges_count = 0
|
||||
escalations_count = 0
|
||||
modified = False
|
||||
|
||||
for dm_id, rec in list(followups.items()):
|
||||
if rec.get("status") != "pending":
|
||||
continue
|
||||
|
||||
deadline_dt = parse_iso(rec.get("deadline"))
|
||||
if not deadline_dt or now < deadline_dt:
|
||||
continue
|
||||
|
||||
# Deadline has expired!
|
||||
nudges_sent = rec.get("nudges_sent", 0)
|
||||
nudges_allowed = rec.get("nudges_allowed", 2)
|
||||
sender = rec.get("sender", "opm")
|
||||
recipient = rec.get("recipient")
|
||||
orig_target = rec.get("target", "main")
|
||||
thread_uuid = rec.get("thread_uuid")
|
||||
|
||||
# Ghost-followup fail-fast (2026-10-04, operator-main): a null
|
||||
# thread_uuid means the sidechat was never provisioned, so the
|
||||
# harvester can never match a reply. Nudging is pointless -- flag
|
||||
# for manual triage once instead of burning the nudge budget and
|
||||
# escalating a ghost. Main-chat followups are unaffected (the
|
||||
# harvester matches those by target).
|
||||
if thread_uuid is None and orig_target != "main" and not rec.get("needs_review"):
|
||||
rec["needs_review"] = True
|
||||
rec["status"] = "needs_review"
|
||||
rec["review_reason"] = (
|
||||
"ghost: unresolvable, manual triage "
|
||||
f"(thread_uuid null, target={orig_target}; "
|
||||
"reply can never auto-resolve)"
|
||||
)
|
||||
modified = True
|
||||
append_job_log({
|
||||
"ts": utcnow_str(),
|
||||
"type": "followup_ghost_suppressed",
|
||||
"dm_id": dm_id,
|
||||
"recipient": recipient,
|
||||
"target": orig_target,
|
||||
"reason": "ghost: unresolvable, manual triage",
|
||||
})
|
||||
print(f"Sweeper: suppressing ghost followup {dm_id} "
|
||||
f"(thread_uuid null, target={orig_target}) -> needs_review",
|
||||
file=sys.stderr)
|
||||
continue
|
||||
|
||||
if nudges_sent < nudges_allowed:
|
||||
# Deliver next nudge
|
||||
nudge_num = nudges_sent + 1
|
||||
is_final = (nudge_num == nudges_allowed)
|
||||
# Routing: In-thread first, Main on final nudge
|
||||
delivery_target = "main" if is_final else orig_target
|
||||
|
||||
nudge_text = (
|
||||
f"[nudge {nudge_num}/{nudges_allowed}] [ref:{dm_id}] "
|
||||
f"Reminder: awaiting reply to request sent at {rec.get('sent_at', 'earlier')}."
|
||||
)
|
||||
if is_final and orig_target != "main":
|
||||
nudge_text += f" (Origin thread: {orig_target})"
|
||||
|
||||
print(f"Sweeper: Sending nudge {nudge_num}/{nudges_allowed} to {recipient}/{delivery_target}...")
|
||||
if not dry_run:
|
||||
ok, out = send_dm(sender, recipient, delivery_target, nudge_text)
|
||||
if ok:
|
||||
nudges_count += 1
|
||||
rec["nudges_sent"] = nudge_num
|
||||
rec["last_nudge_at"] = utcnow_str()
|
||||
# Calculate interval for next nudge: proportional to timeout or default 10m
|
||||
timeout_s = rec.get("timeout_s", 1800)
|
||||
step_s = max(300, timeout_s // (nudges_allowed + 1))
|
||||
rec["deadline"] = (now + timedelta(seconds=step_s)).isoformat()
|
||||
# C1: backfill the thread the nudge actually landed in.
|
||||
landed_uuid = _resolve_nudge_thread_uuid(out)
|
||||
if landed_uuid and rec.get("thread_uuid") != landed_uuid:
|
||||
rec["thread_uuid"] = landed_uuid
|
||||
print(f"Sweeper: backfilled thread_uuid={landed_uuid} "
|
||||
f"for followup {dm_id}", file=sys.stderr)
|
||||
# C2: record final-nudge routing for the harvester.
|
||||
if is_final:
|
||||
rec["final_nudge_target"] = "main"
|
||||
modified = True
|
||||
append_job_log({
|
||||
"ts": utcnow_str(),
|
||||
"type": "followup_nudged",
|
||||
"dm_id": dm_id,
|
||||
"nudge_num": nudge_num,
|
||||
"recipient": recipient,
|
||||
"target": delivery_target,
|
||||
"thread_uuid": rec.get("thread_uuid"),
|
||||
})
|
||||
else:
|
||||
print(f"Sweeper WARNING: nudge send failed: {out}", file=sys.stderr)
|
||||
# Failure-mode fix (2026-10-04): advance state on send
|
||||
# failure so a failing nudge is never re-fired every
|
||||
# timer tick. failed_sends is tracked separately from
|
||||
# nudges_sent so a delivery failure does not consume a
|
||||
# real nudge. Backoff: 5 min base, doubling per failure.
|
||||
failed = rec.get("failed_sends", 0) + 1
|
||||
rec["failed_sends"] = failed
|
||||
rec["last_failure_at"] = utcnow_str()
|
||||
backoff_s = 300 * (2 ** min(failed - 1, 4)) # 5m,10m,20m,40m,80m cap
|
||||
rec["deadline"] = (now + timedelta(seconds=backoff_s)).isoformat()
|
||||
modified = True
|
||||
append_job_log({
|
||||
"ts": utcnow_str(),
|
||||
"type": "followup_nudge_failed",
|
||||
"dm_id": dm_id,
|
||||
"nudge_num": nudge_num,
|
||||
"recipient": recipient,
|
||||
"target": delivery_target,
|
||||
"failed_sends": failed,
|
||||
"backoff_s": backoff_s,
|
||||
"error": str(out)[:200],
|
||||
})
|
||||
# After 3 consecutive failures, stop retrying blindly and
|
||||
# flag for manual review (delivery may be uncertain or the
|
||||
# target may be permanently broken).
|
||||
if failed >= 3:
|
||||
rec["needs_review"] = True
|
||||
rec["review_reason"] = (
|
||||
f"nudge send failed {failed} times consecutively "
|
||||
f"(last: {str(out)[:120]})"
|
||||
)
|
||||
append_job_log({
|
||||
"ts": utcnow_str(),
|
||||
"type": "followup_needs_review",
|
||||
"dm_id": dm_id,
|
||||
"recipient": recipient,
|
||||
"failed_sends": failed,
|
||||
})
|
||||
else:
|
||||
nudges_count += 1
|
||||
|
||||
else:
|
||||
# All nudges exhausted: Terminal escalation
|
||||
escalate_to = rec.get("escalate_to", "opm")
|
||||
esc_text = (
|
||||
f"[ESCALATION] Agent {recipient} failed to reply to DM {dm_id} "
|
||||
f"after {nudges_allowed} nudges. Target was: {orig_target} "
|
||||
f"(thread: {thread_uuid or 'n/a'}). Request sent: {rec.get('sent_at')}."
|
||||
)
|
||||
print(f"Sweeper: Escalating expired follow-up {dm_id} to {escalate_to}...")
|
||||
if not dry_run:
|
||||
ok, out = send_dm("bl", escalate_to, "main", esc_text)
|
||||
rec["status"] = "escalated"
|
||||
rec["escalated_at"] = utcnow_str()
|
||||
escalations_count += 1
|
||||
modified = True
|
||||
append_job_log({
|
||||
"ts": utcnow_str(),
|
||||
"type": "followup_escalated",
|
||||
"dm_id": dm_id,
|
||||
"recipient": recipient,
|
||||
"escalated_to": escalate_to,
|
||||
})
|
||||
|
||||
# Check if this dm_id belongs to an active pipeline step
|
||||
if HAS_PIPELINE:
|
||||
run_entry, step_entry = pipeline_engine.record_step_timeout(dm_id)
|
||||
if run_entry and step_entry:
|
||||
j_name = step_entry.get("job_name")
|
||||
j_file = JOBS_DIR / f"{j_name}.json"
|
||||
if j_file.exists():
|
||||
try:
|
||||
with open(j_file, "r", encoding="utf-8") as f:
|
||||
j_cfg = json.load(f)
|
||||
on_failure = j_cfg.get("on_failure")
|
||||
if on_failure and (JOBS_DIR / f"{on_failure}.json").exists():
|
||||
env = os.environ.copy()
|
||||
env["CHAIN_PREV_JOB_ID"] = step_entry.get("job_id")
|
||||
env["CHAIN_PREV_RESULT"] = f"TIMEOUT: Agent {recipient} timed out after {nudges_allowed} nudges"
|
||||
env["CHAIN_PIPELINE_RUN_ID"] = run_entry.get("run_id")
|
||||
next_step_n = step_entry.get("step_n", 1) + 1
|
||||
env["CHAIN_STEP_N"] = str(next_step_n)
|
||||
cmd = [sys.executable, str(DISPATCH_PY), on_failure,
|
||||
"--pipeline-run", run_entry.get("run_id"),
|
||||
"--step-n", str(next_step_n)]
|
||||
subprocess.Popen(cmd, env=env, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
|
||||
else:
|
||||
pipeline_engine.fail_pipeline(run_entry.get("run_id"), "step_timed_out_without_fallback")
|
||||
except Exception:
|
||||
pass
|
||||
else:
|
||||
escalations_count += 1
|
||||
|
||||
if modified and not dry_run:
|
||||
save_followups(followups)
|
||||
|
||||
pending_count = sum(1 for r in followups.values() if r.get("status") == "pending")
|
||||
return {
|
||||
"status": "ok",
|
||||
"pending": pending_count,
|
||||
"nudges_sent": nudges_count,
|
||||
"escalations": escalations_count,
|
||||
}
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description="Autonomous follow-up deadline tracker and sweeper")
|
||||
parser.add_argument("--once", action="store_true", help="Run once and exit (default)")
|
||||
parser.add_argument("--loop", action="store_true", help="Run continuously in a daemon loop")
|
||||
parser.add_argument("--interval", type=int, default=60, help="Interval in seconds for loop (default 60)")
|
||||
parser.add_argument("--dry-run", action="store_true", help="Inspect without sending nudges or updating records")
|
||||
args = parser.parse_args()
|
||||
|
||||
if not args.loop:
|
||||
stats = sweep_cycle(dry_run=args.dry_run)
|
||||
print(f"[{datetime.now(timezone.utc).strftime('%H:%M:%SZ')}] Sweep cycle: {stats['pending']} pending, {stats['nudges_sent']} nudges, {stats['escalations']} escalations.")
|
||||
return
|
||||
|
||||
print(f"Starting follow-up sweeper loop (interval={args.interval}s)...")
|
||||
while True:
|
||||
try:
|
||||
stats = sweep_cycle(dry_run=args.dry_run)
|
||||
print(f"[{datetime.now(timezone.utc).strftime('%H:%M:%SZ')}] Sweep cycle: {stats['pending']} pending, {stats['nudges_sent']} nudges, {stats['escalations']} escalations.")
|
||||
except Exception as e:
|
||||
print(f"ERROR in sweeper loop: {e}", file=sys.stderr)
|
||||
time.sleep(args.interval)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
-812
@@ -1,812 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Timer-driven side-chat adoption: shared gravity library.
|
||||
|
||||
Pure Python, no network, no bl dependencies — everything the timers need to
|
||||
decide *where* a message goes and *whether* its loop is alive.
|
||||
|
||||
Components:
|
||||
GravityConfig knobs from gravity.json (fail-closed defaults)
|
||||
ThreadRegistry (recipient, purpose) -> thread_uuid, JSON-backed
|
||||
resolve_target() explicit target > registry hit > autocreate > fallback
|
||||
LoopState FIRING/LANDED/SEND_FAILED/SEEN/ANSWERED/NUDGED/
|
||||
ESCALATED/CLOSED/BROKEN + transition helpers
|
||||
detect_breaks() loop-break taxonomy detectors over log-derived events
|
||||
actionable_digest() wrap a wake digest as a tracked loop message
|
||||
loop_health() per-agent ANSWERED+CLOSED / LANDED
|
||||
|
||||
The live integrations (job-dispatch.py, dm.py, sidechat-wake.py) import the
|
||||
pure functions here; the patches/ directory shows the call-site diffs.
|
||||
"""
|
||||
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import sys
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
# ---------------------------------------------------------------- config
|
||||
|
||||
DEFAULTS = {
|
||||
"job_default_sidechat": True,
|
||||
"job_sidechat_fallback": "main",
|
||||
"dm_prefer_sidechat_default": False,
|
||||
"dm_sidechat_fallback": "main",
|
||||
"wake_actionable": True,
|
||||
"wake_ack_timeout": 7200,
|
||||
"wake_ack_nudges": 1,
|
||||
"wake_ack_escalate": "opm",
|
||||
"thread_autocreate": True,
|
||||
"loop_silent_ticks": 3,
|
||||
"loop_health_threshold": 0.5,
|
||||
}
|
||||
|
||||
VALID_AGENTS = ("muse", "pip", "646", "opm")
|
||||
UUID_RE = re.compile(
|
||||
r"[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}")
|
||||
|
||||
|
||||
class GravityConfig:
|
||||
"""Knobs. Unknown keys rejected (typo guard); missing keys get
|
||||
fail-closed defaults (see DEFAULTS)."""
|
||||
|
||||
def __init__(self, data=None):
|
||||
data = data or {}
|
||||
unknown = set(data) - set(DEFAULTS)
|
||||
if unknown:
|
||||
raise ValueError("unknown gravity keys: %s" % sorted(unknown))
|
||||
self._d = dict(DEFAULTS)
|
||||
self._d.update(data)
|
||||
|
||||
@classmethod
|
||||
def load(cls, path):
|
||||
with open(path) as f:
|
||||
return cls(json.load(f))
|
||||
|
||||
def get(self, key):
|
||||
return self._d[key]
|
||||
|
||||
def __getitem__(self, key):
|
||||
return self._d[key]
|
||||
|
||||
def as_dict(self):
|
||||
return dict(self._d)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- registry
|
||||
|
||||
class ThreadRegistry:
|
||||
"""(recipient, purpose) -> {thread_uuid, created_ts, last_verified_ts}.
|
||||
|
||||
A thread UUID is only valid in the account whose browser created it, so
|
||||
the registry is keyed by recipient — never share UUIDs across accounts.
|
||||
|
||||
Rotation: set()/record_rotation() keep the key pointed at the newest
|
||||
UUID and stash the old one under `<key>:previous` for audit (same
|
||||
model as thread_lifecycle.record_rotation).
|
||||
"""
|
||||
|
||||
def __init__(self, path=None, state=None):
|
||||
self.path = path
|
||||
self._s = dict(state or {})
|
||||
|
||||
@classmethod
|
||||
def load(cls, path):
|
||||
try:
|
||||
with open(path) as f:
|
||||
return cls(path=path, state=json.load(f))
|
||||
except (OSError, ValueError):
|
||||
return cls(path=path)
|
||||
|
||||
def save(self):
|
||||
if not self.path:
|
||||
raise ValueError("no path for registry save")
|
||||
tmp = self.path + ".tmp"
|
||||
with open(tmp, "w") as f:
|
||||
json.dump(self._s, f, indent=2)
|
||||
os.replace(tmp, self.path)
|
||||
|
||||
def _key(self, recipient, purpose):
|
||||
return "%s:%s" % (recipient, purpose)
|
||||
|
||||
def get(self, recipient, purpose):
|
||||
"""Current thread UUID for (recipient, purpose), following any
|
||||
rotation chain. Returns None if unknown."""
|
||||
return self.resolve(self._key(recipient, purpose))
|
||||
|
||||
def resolve(self, key):
|
||||
"""Return the current UUID for a registry key.
|
||||
|
||||
set()/record_rotation() always keep the key pointing at the newest
|
||||
UUID; the `:previous` entry is audit history, not a traversal chain.
|
||||
(Mirrors the guarantee in thread_lifecycle.record_rotation.)
|
||||
"""
|
||||
cur = self._s.get(key)
|
||||
if isinstance(cur, str) and UUID_RE.fullmatch(cur):
|
||||
return cur
|
||||
return None
|
||||
|
||||
def previous(self, recipient, purpose):
|
||||
"""The rotated-away UUID, if any (for diagnostics)."""
|
||||
return self._s.get(self._key(recipient, purpose) + ":previous")
|
||||
|
||||
def set(self, recipient, purpose, thread_uuid, ts=None):
|
||||
if not UUID_RE.fullmatch(thread_uuid or ""):
|
||||
raise ValueError("not a thread UUID: %r" % thread_uuid)
|
||||
key = self._key(recipient, purpose)
|
||||
old = self._s.get(key)
|
||||
if old and old != thread_uuid:
|
||||
self._s[key + ":previous"] = old
|
||||
self._s[key] = thread_uuid
|
||||
if ts:
|
||||
self._s[key + ":updated_ts"] = ts
|
||||
|
||||
def record_rotation(self, recipient, purpose, new_uuid, ts=None):
|
||||
"""Alias for set() — rotation is just a set with history."""
|
||||
self.set(recipient, purpose, new_uuid, ts=ts)
|
||||
|
||||
def known_purposes(self, recipient):
|
||||
prefix = recipient + ":"
|
||||
out = []
|
||||
for k in self._s:
|
||||
if k.startswith(prefix) and ":" not in k[len(prefix):]:
|
||||
out.append(k[len(prefix):])
|
||||
return sorted(out)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- target resolution
|
||||
|
||||
def resolve_target(recipient, purpose, cfg, registry,
|
||||
explicit_target=None, probe=None):
|
||||
"""Decide where a timer-driven send goes.
|
||||
|
||||
Order: explicit --target > registry hit (verified) > autocreate >
|
||||
configured fallback. Never raises for routing reasons — worst case is
|
||||
the fallback with a logged reason.
|
||||
|
||||
probe(thread_uuid) -> bool: cheap reachability check (sidechat use).
|
||||
create() -> thread_uuid | None: injected by caller (needs browser).
|
||||
|
||||
Returns (target, thread_uuid_or_None, reason).
|
||||
"""
|
||||
if explicit_target:
|
||||
tid = UUID_RE.search(explicit_target)
|
||||
return explicit_target, tid.group(0) if tid else None, "explicit"
|
||||
|
||||
if recipient not in VALID_AGENTS:
|
||||
return cfg["dm_sidechat_fallback"], None, "unknown_recipient"
|
||||
|
||||
uuid = registry.get(recipient, purpose)
|
||||
if uuid:
|
||||
if probe is None or probe(uuid):
|
||||
return uuid, uuid, "registry_hit"
|
||||
return cfg["dm_sidechat_fallback"], None, "registry_stale"
|
||||
|
||||
# No mapping: autocreate or fallback.
|
||||
return None, None, "no_mapping_needs_create"
|
||||
|
||||
|
||||
def fallback_target(cfg):
|
||||
"""Configured fallback when no sidechat is available. Fails closed
|
||||
toward visibility: main, never drop."""
|
||||
return cfg.get("dm_sidechat_fallback") or "main"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- loop state machine
|
||||
|
||||
# Terminal states: CLOSED, ESCALATED, BROKEN, SEND_FAILED.
|
||||
STATES = ("FIRING", "LANDED", "SEND_FAILED", "SEEN", "ANSWERED",
|
||||
"NUDGED", "ESCALATED", "CLOSED", "BROKEN")
|
||||
|
||||
TERMINAL = frozenset(("SEND_FAILED", "ESCALATED", "CLOSED", "BROKEN"))
|
||||
|
||||
# Allowed transitions. NUDGED can cycle (nudge 1..N) until ANSWERED or
|
||||
# ESCALATED; BROKEN is reachable from any non-terminal state.
|
||||
TRANSITIONS = {
|
||||
"FIRING": ("LANDED", "SEND_FAILED", "BROKEN"),
|
||||
"LANDED": ("SEEN", "ANSWERED", "NUDGED", "BROKEN"),
|
||||
"SEEN": ("ANSWERED", "NUDGED", "BROKEN"),
|
||||
"ANSWERED": ("CLOSED", "BROKEN"),
|
||||
"NUDGED": ("ANSWERED", "NUDGED", "ESCALATED", "BROKEN"),
|
||||
"SEND_FAILED": (),
|
||||
"ESCALATED": (),
|
||||
"CLOSED": (),
|
||||
"BROKEN": (),
|
||||
}
|
||||
|
||||
|
||||
class LoopState:
|
||||
"""One timer-driven loop: timer tick -> tracked follow-up."""
|
||||
|
||||
def __init__(self, loop_id, agent, purpose, thread_uuid=None):
|
||||
self.loop_id = loop_id
|
||||
self.agent = agent
|
||||
self.purpose = purpose
|
||||
self.thread_uuid = thread_uuid
|
||||
self.state = "FIRING"
|
||||
self.nudges = 0
|
||||
self.ticks_without_reply = 0
|
||||
self.history = [("FIRING", None)]
|
||||
|
||||
def transition(self, to, note=None):
|
||||
if to not in TRANSITIONS.get(self.state, ()):
|
||||
raise ValueError("illegal loop transition %s -> %s"
|
||||
% (self.state, to))
|
||||
self.state = to
|
||||
if to == "NUDGED":
|
||||
self.nudges += 1
|
||||
self.history.append((to, note))
|
||||
return self
|
||||
|
||||
@property
|
||||
def terminal(self):
|
||||
return self.state in TERMINAL
|
||||
|
||||
@property
|
||||
def healthy(self):
|
||||
return self.state in ("ANSWERED", "CLOSED")
|
||||
|
||||
def tick(self):
|
||||
"""One scheduler tick with no reply observed."""
|
||||
self.ticks_without_reply += 1
|
||||
return self.ticks_without_reply
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- break detectors
|
||||
|
||||
def detect_breaks(loop, cfg, checks):
|
||||
"""Loop-break taxonomy over caller-supplied check results.
|
||||
|
||||
checks: dict with boolean-ish keys:
|
||||
thread_reachable, signature_ok, scheduler_alive,
|
||||
reply_observed
|
||||
Returns a list of break names (may be empty).
|
||||
"""
|
||||
breaks = []
|
||||
if loop.terminal:
|
||||
return breaks
|
||||
if not checks.get("scheduler_alive", True):
|
||||
breaks.append("scheduler_death")
|
||||
if not checks.get("signature_ok", True):
|
||||
breaks.append("auth_rot")
|
||||
if not checks.get("thread_reachable", True):
|
||||
breaks.append("dead_thread")
|
||||
# signature rot: thread was rotated — the stored uuid is stale but a
|
||||
# :previous chain exists. Caller signals via thread_rotated=True and
|
||||
# should already have resolved; flag if not resolved.
|
||||
if checks.get("thread_rotated") and not checks.get("thread_resolved"):
|
||||
breaks.append("signature_rot")
|
||||
if (not checks.get("reply_observed")
|
||||
and loop.ticks_without_reply >= cfg["loop_silent_ticks"]
|
||||
and loop.state in ("LANDED", "SEEN", "NUDGED")):
|
||||
breaks.append("silent_agent")
|
||||
return breaks
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- actionable digest
|
||||
|
||||
def actionable_digest(digest_text, ack_line=True):
|
||||
"""Wrap a wake digest so the timer tick becomes a tracked loop.
|
||||
|
||||
Adds the ack line that the follow-up record's reply closes. The caller
|
||||
sends the result via `dm.py send --expect-reply --thread <uuid>` so a
|
||||
dm_followup record exists — without that send path this is just text.
|
||||
"""
|
||||
lines = digest_text.rstrip().split("\n")
|
||||
if ack_line:
|
||||
lines.append("")
|
||||
lines.append("Reply here to acknowledge (closes the loop).")
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def dm_send_argv(sender, recipient, target_uuid, message, cfg,
|
||||
purpose="wake"):
|
||||
"""Build the dm.py argv that makes a timer send a *tracked* loop.
|
||||
|
||||
Thread binding (--thread) is what lets nudges route back into the
|
||||
thread instead of falling back to DM.
|
||||
"""
|
||||
return [
|
||||
"send",
|
||||
"--agent", sender,
|
||||
"--to", recipient,
|
||||
"--target", target_uuid,
|
||||
"--thread", target_uuid,
|
||||
"--expect-reply",
|
||||
"--reply-timeout", str(cfg["wake_ack_timeout"]),
|
||||
"--reply-nudges", str(cfg["wake_ack_nudges"]),
|
||||
"--reply-escalate", cfg["wake_ack_escalate"],
|
||||
"--tag", "purpose:%s" % purpose,
|
||||
message,
|
||||
]
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- loop health
|
||||
|
||||
def loop_health(loops, threshold=0.5):
|
||||
"""Per-agent loop health: (ANSWERED + CLOSED) / LANDED.
|
||||
|
||||
loops: iterable of LoopState (or dicts with agent/state keys).
|
||||
Returns {agent: {"landed": n, "answered": n, "health": float|None,
|
||||
"below_threshold": bool}}.
|
||||
"""
|
||||
def _agent(l):
|
||||
return l.agent if isinstance(l, LoopState) else l.get("agent")
|
||||
|
||||
def _state(l):
|
||||
return l.state if isinstance(l, LoopState) else l.get("state")
|
||||
|
||||
# A loop counts as LANDED once it leaves FIRING via LANDED (or beyond).
|
||||
LANDED_OR_BEYOND = ("LANDED", "SEEN", "ANSWERED", "NUDGED",
|
||||
"ESCALATED", "CLOSED", "BROKEN")
|
||||
acc = {}
|
||||
for l in loops:
|
||||
a, s = _agent(l), _state(l)
|
||||
r = acc.setdefault(a, {"landed": 0, "answered": 0})
|
||||
if s in LANDED_OR_BEYOND:
|
||||
r["landed"] += 1
|
||||
if s in ("ANSWERED", "CLOSED"):
|
||||
r["answered"] += 1
|
||||
out = {}
|
||||
for a, r in acc.items():
|
||||
h = (r["answered"] / r["landed"]) if r["landed"] else None
|
||||
out[a] = {"landed": r["landed"], "answered": r["answered"],
|
||||
"health": h,
|
||||
"below_threshold": h is not None and h < threshold}
|
||||
return out
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- loop reconstruction & fleet health
|
||||
|
||||
NETVM_ROOT = "/home/super/Projects/NetVM"
|
||||
FOLLOWUPS_FILE = os.path.join(NETVM_ROOT, "followups.json")
|
||||
DM_LOG_FILE = os.path.join(NETVM_ROOT, "dm-log.jsonl")
|
||||
JOB_LOG_FILE = os.path.join(NETVM_ROOT, "job-log.jsonl")
|
||||
|
||||
|
||||
def reconstruct_loops(limit=50, agent=None, status_filter=None) -> list:
|
||||
"""Reconstruct active and recent loops from followups.json and dm-log.jsonl.
|
||||
|
||||
Returns a list of dicts:
|
||||
loop_id, agent, sender, target, purpose, state, sent_at, deadline,
|
||||
nudges_sent, nudges_allowed, escalate_to, tags, summary
|
||||
"""
|
||||
loops = {} # loop_id -> dict
|
||||
|
||||
# 1. Load active/persisted followups
|
||||
if os.path.exists(FOLLOWUPS_FILE):
|
||||
try:
|
||||
with open(FOLLOWUPS_FILE, "r") as f:
|
||||
fdata = json.load(f)
|
||||
if isinstance(fdata, dict):
|
||||
for did, rec in fdata.items():
|
||||
recipient = rec.get("recipient")
|
||||
st = rec.get("status", "pending")
|
||||
# Map status to loop state
|
||||
if st == "pending":
|
||||
lstate = "NUDGED" if rec.get("nudges_sent", 0) > 0 else "LANDED"
|
||||
elif st == "escalated":
|
||||
lstate = "ESCALATED"
|
||||
elif st in ("resolved", "closed"):
|
||||
lstate = "CLOSED"
|
||||
else:
|
||||
lstate = st.upper()
|
||||
|
||||
loops[did] = {
|
||||
"loop_id": did,
|
||||
"agent": recipient,
|
||||
"sender": rec.get("sender", "super"),
|
||||
"target": rec.get("target", "main"),
|
||||
"thread_uuid": rec.get("thread_uuid"),
|
||||
"purpose": rec.get("route") or "followup",
|
||||
"state": lstate,
|
||||
"sent_at": rec.get("sent_at", ""),
|
||||
"deadline": rec.get("deadline", ""),
|
||||
"timeout_s": rec.get("timeout_s", 3600),
|
||||
"nudges_sent": rec.get("nudges_sent", 0),
|
||||
"nudges_allowed": rec.get("nudges_allowed", 2),
|
||||
"escalate_to": rec.get("escalate_to", "opm"),
|
||||
"tags": {"reply:expected": True},
|
||||
"summary": f"Follow-up for DM {did}",
|
||||
"source": "followups.json",
|
||||
}
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
# 2. Extract tracked DMs from dm-log.jsonl
|
||||
answers_map = {} # id or ref -> answer event
|
||||
dm_events = []
|
||||
failed_ids = set() # DM ids that failed to send - never delivered
|
||||
if os.path.exists(DM_LOG_FILE):
|
||||
try:
|
||||
with open(DM_LOG_FILE, "r") as f:
|
||||
for line in f:
|
||||
line = line.strip()
|
||||
if not line:
|
||||
continue
|
||||
try:
|
||||
entry = json.loads(line)
|
||||
ev_type = entry.get("type", "")
|
||||
msg = entry.get("msg") or ""
|
||||
|
||||
# Check for answers/results/acks
|
||||
# Note: "verified" means delivered, NOT answered - do not include it here
|
||||
if "[RESULT" in msg or "[ACK" in msg or "acknowledged" in msg:
|
||||
sender = entry.get("agent", "")
|
||||
# Extract referenced DM id if present
|
||||
m_ref = re.search(r"\[ref:([a-f0-9-]+)\]", msg) or re.search(r"\[(?:ACK|RESULT)\s+([a-f0-9-]+)", msg)
|
||||
if m_ref:
|
||||
answers_map[m_ref.group(1)] = entry
|
||||
if entry.get("id"):
|
||||
answers_map[entry["id"]] = entry
|
||||
|
||||
# Track failed sends - these were never delivered
|
||||
if ev_type in ("send_failed", "failed"):
|
||||
if entry.get("id"):
|
||||
failed_ids.add(entry.get("id"))
|
||||
|
||||
tags = entry.get("tags") or {}
|
||||
if tags.get("reply:expected") or "[reply:expected]" in msg:
|
||||
dm_events.append(entry)
|
||||
except Exception:
|
||||
continue
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
# Merge dm_events into loops
|
||||
for entry in dm_events:
|
||||
did = entry.get("id") or entry.get("msg_id") or ""
|
||||
if not did:
|
||||
continue
|
||||
if did in loops:
|
||||
continue # Already have active record from followups.json
|
||||
if did in failed_ids:
|
||||
continue # Send failed - never delivered, don't count against agent health
|
||||
|
||||
tags = entry.get("tags") or {}
|
||||
recipient = entry.get("to") or entry.get("recipient") or ""
|
||||
sender = entry.get("agent") or entry.get("sender") or "super"
|
||||
target = entry.get("target") or "main"
|
||||
sent_at = entry.get("ts") or entry.get("sent_at") or ""
|
||||
timeout_s = int(tags.get("reply:timeout") or 3600)
|
||||
nudges_max = int(tags.get("reply:nudges") or 2)
|
||||
esc = tags.get("reply:escalate") or "opm"
|
||||
purpose = tags.get("route") or "dm"
|
||||
|
||||
# Determine state
|
||||
if did in answers_map or f"ref:{did}" in answers_map:
|
||||
lstate = "CLOSED"
|
||||
else:
|
||||
# Check age
|
||||
lstate = "LANDED"
|
||||
|
||||
loops[did] = {
|
||||
"loop_id": did,
|
||||
"agent": recipient,
|
||||
"sender": sender,
|
||||
"target": target,
|
||||
"thread_uuid": tags.get("thread"),
|
||||
"purpose": purpose,
|
||||
"state": lstate,
|
||||
"sent_at": sent_at,
|
||||
"deadline": "",
|
||||
"timeout_s": timeout_s,
|
||||
"nudges_sent": 0,
|
||||
"nudges_allowed": nudges_max,
|
||||
"escalate_to": esc,
|
||||
"tags": tags,
|
||||
"summary": entry.get("msg", "")[:80],
|
||||
"source": "dm-log.jsonl",
|
||||
}
|
||||
|
||||
# Filter and sort
|
||||
result = list(loops.values())
|
||||
if agent:
|
||||
result = [l for l in result if l.get("agent") == agent or l.get("sender") == agent]
|
||||
if status_filter:
|
||||
sf = status_filter.lower()
|
||||
if sf == "active" or sf == "pending":
|
||||
result = [l for l in result if l.get("state") in ("FIRING", "LANDED", "SEEN", "NUDGED", "ESCALATED")]
|
||||
elif sf in ("closed", "resolved"):
|
||||
result = [l for l in result if l.get("state") in ("CLOSED", "ANSWERED")]
|
||||
else:
|
||||
result = [l for l in result if l.get("state", "").lower() == sf]
|
||||
|
||||
# Sort newest first
|
||||
result.sort(key=lambda x: x.get("sent_at", ""), reverse=True)
|
||||
return result[:limit]
|
||||
|
||||
|
||||
def get_fleet_loop_health(threshold=None) -> dict:
|
||||
"""Calculate fleet loop health per agent and overall verdict."""
|
||||
if threshold is None:
|
||||
try:
|
||||
from variables import Variables
|
||||
threshold = Variables().get_float("loop_health_threshold")
|
||||
except Exception:
|
||||
threshold = 0.5
|
||||
|
||||
loops = reconstruct_loops(limit=100)
|
||||
raw = loop_health(loops, threshold=threshold)
|
||||
|
||||
out = {}
|
||||
for ag in VALID_AGENTS:
|
||||
info = raw.get(ag, {"landed": 0, "answered": 0, "health": None, "below_threshold": False})
|
||||
landed = info.get("landed", 0)
|
||||
answered = info.get("answered", 0)
|
||||
h = info.get("health")
|
||||
if landed == 0:
|
||||
status = "IDLE"
|
||||
elif h is not None and h >= threshold:
|
||||
status = "HEALTHY"
|
||||
else:
|
||||
status = "DEGRADED"
|
||||
|
||||
out[ag] = {
|
||||
"agent": ag,
|
||||
"landed": landed,
|
||||
"answered": answered,
|
||||
"health": h,
|
||||
"health_pct": f"{int(h * 100)}%" if h is not None else "-",
|
||||
"threshold": threshold,
|
||||
"below_threshold": info.get("below_threshold", False),
|
||||
"status": status,
|
||||
}
|
||||
|
||||
total_landed = sum(v["landed"] for v in out.values())
|
||||
total_answered = sum(v["answered"] for v in out.values())
|
||||
overall_h = (total_answered / total_landed) if total_landed > 0 else None
|
||||
|
||||
return {
|
||||
"agents": out,
|
||||
"summary": {
|
||||
"total_landed": total_landed,
|
||||
"total_answered": total_answered,
|
||||
"overall_health": overall_h,
|
||||
"overall_health_pct": f"{int(overall_h * 100)}%" if overall_h is not None else "-",
|
||||
"threshold": threshold,
|
||||
"healthy": overall_h is None or overall_h >= threshold,
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
def diagnose_breaks() -> list:
|
||||
"""Diagnose break taxonomy across intrinsic loops and support services."""
|
||||
import subprocess
|
||||
breaks = []
|
||||
|
||||
# 1. Scheduler / Timers check
|
||||
try:
|
||||
r = subprocess.run(["systemctl", "--user", "is-system-running"],
|
||||
capture_output=True, text=True, timeout=5)
|
||||
sys_state = r.stdout.strip()
|
||||
if sys_state in ("offline", "stopped"):
|
||||
breaks.append({
|
||||
"type": "scheduler_death",
|
||||
"severity": "CRITICAL",
|
||||
"component": "systemd",
|
||||
"detail": f"Systemd user instance is {sys_state}",
|
||||
"remedy": "Restart systemd user session or start timer jobs manually."
|
||||
})
|
||||
except Exception as e:
|
||||
breaks.append({
|
||||
"type": "scheduler_death",
|
||||
"severity": "WARNING",
|
||||
"component": "systemd",
|
||||
"detail": f"Could not check systemd status: {e}",
|
||||
"remedy": "Verify systemctl --user is available."
|
||||
})
|
||||
|
||||
# 2. SSH key signature check
|
||||
key_path = os.path.expanduser("~/.ssh/id_ed25519")
|
||||
if not os.path.exists(key_path):
|
||||
breaks.append({
|
||||
"type": "auth_rot",
|
||||
"severity": "CRITICAL",
|
||||
"component": "ssh-keys",
|
||||
"detail": f"SSH signing key {key_path} not found",
|
||||
"remedy": "Generate ed25519 key at ~/.ssh/id_ed25519 for cryptographically signed DMs."
|
||||
})
|
||||
|
||||
# 3. Active follow-up loops check
|
||||
active_loops = reconstruct_loops(limit=20, status_filter="pending")
|
||||
now_ts = time.time()
|
||||
for l in active_loops:
|
||||
nudges_sent = l.get("nudges_sent", 0)
|
||||
nudges_max = l.get("nudges_allowed", 2)
|
||||
if nudges_sent >= nudges_max and l.get("state") == "ESCALATED":
|
||||
breaks.append({
|
||||
"type": "silent_agent",
|
||||
"severity": "WARNING",
|
||||
"loop_id": l["loop_id"],
|
||||
"agent": l["agent"],
|
||||
"detail": f"Agent {l['agent']} silent after {nudges_sent}/{nudges_max} nudges for DM {l['loop_id']}",
|
||||
"remedy": f"Check agent {l['agent']} browser tab with 'super fleet status' or nudge via 'super dm send'."
|
||||
})
|
||||
|
||||
# 4. Check for agents held up on approvals
|
||||
try:
|
||||
import approvals
|
||||
fleet_apps = approvals.check_fleet_approvals()
|
||||
for app in fleet_apps:
|
||||
if app.get("has_pending"):
|
||||
node = app["node"]
|
||||
is_trusted = app.get("is_trusted", False)
|
||||
ip = app.get("ip") or "unknown target"
|
||||
breaks.append({
|
||||
"type": "approval_blocked",
|
||||
"severity": "WARNING" if is_trusted else "CRITICAL",
|
||||
"component": f"node:{node}",
|
||||
"agent": node,
|
||||
"detail": f"Agent {node} is held up on browser approval for {ip}",
|
||||
"remedy": f"Run 'box approvals auto' or 'box approvals allow {node}'."
|
||||
})
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
return breaks
|
||||
|
||||
|
||||
def remediate_breaks(dry_run=False) -> dict:
|
||||
"""Progressively auto-remediate soft loop breakages while escalating hard breakages.
|
||||
|
||||
Soft breakages (auto-healed):
|
||||
- Pending followups that have received an answer in dm-log.jsonl or chat-history
|
||||
are resolved.
|
||||
- Pending followups with expired deadlines and nudges remaining are re-armed
|
||||
and swept immediately.
|
||||
|
||||
Hard breakages (escalated loudly):
|
||||
- silent_agent (nudges exhausted, no response)
|
||||
- auth_rot (missing SSH signing keys)
|
||||
- scheduler_death (systemd user session offline)
|
||||
"""
|
||||
import subprocess
|
||||
from datetime import datetime, timezone
|
||||
|
||||
remediated = []
|
||||
escalated = []
|
||||
|
||||
# 1. Check diagnosed hard breaks first
|
||||
breaks = diagnose_breaks()
|
||||
for b in breaks:
|
||||
if b.get("severity") in ("CRITICAL", "WARNING"):
|
||||
escalated.append(b)
|
||||
|
||||
# 2. Check followups.json for soft break healing
|
||||
f_path = Path("/home/super/Projects/NetVM/followups.json")
|
||||
f_modified = False
|
||||
rearm_sweeper = False
|
||||
|
||||
if f_path.exists():
|
||||
try:
|
||||
with open(f_path, "r") as f:
|
||||
fdata = json.load(f)
|
||||
except Exception:
|
||||
fdata = {}
|
||||
|
||||
now_iso = datetime.now(timezone.utc).isoformat()
|
||||
|
||||
# Build answer map from reconstruct_loops
|
||||
loops = reconstruct_loops(limit=200)
|
||||
answered_dms = {
|
||||
l["loop_id"]: l for l in loops if l.get("state") in ("ANSWERED", "CLOSED")
|
||||
}
|
||||
|
||||
for dm_id, rec in fdata.items():
|
||||
if rec.get("status") == "pending":
|
||||
# Check if it was actually answered in logs
|
||||
if dm_id in answered_dms:
|
||||
remediated.append({
|
||||
"action": "auto_resolve_answered",
|
||||
"loop_id": dm_id,
|
||||
"agent": rec.get("recipient"),
|
||||
"detail": f"Follow-up {dm_id} received reply in log but was pending in followups.json. Marked resolved."
|
||||
})
|
||||
if not dry_run:
|
||||
rec["status"] = "resolved"
|
||||
rec["resolved_at"] = now_iso
|
||||
rec["resolved_note"] = "auto-healed: reply detected in dm-log"
|
||||
f_modified = True
|
||||
continue
|
||||
|
||||
# Check if deadline expired and nudges remaining
|
||||
dl_str = rec.get("deadline", "")
|
||||
nudges_sent = rec.get("nudges_sent", 0)
|
||||
nudges_allowed = rec.get("nudges_allowed", 2)
|
||||
|
||||
is_expired = False
|
||||
if dl_str:
|
||||
try:
|
||||
dl_dt = datetime.fromisoformat(dl_str.replace("Z", "+00:00"))
|
||||
if datetime.now(timezone.utc) > dl_dt:
|
||||
is_expired = True
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
if is_expired and nudges_sent < nudges_allowed:
|
||||
remediated.append({
|
||||
"action": "rearm_expired_nudge",
|
||||
"loop_id": dm_id,
|
||||
"agent": rec.get("recipient"),
|
||||
"detail": f"Deadline expired for {dm_id} ({nudges_sent}/{nudges_allowed} nudges). Re-arming immediate sweep."
|
||||
})
|
||||
if not dry_run:
|
||||
rec["deadline"] = now_iso
|
||||
f_modified = True
|
||||
rearm_sweeper = True
|
||||
|
||||
if f_modified and not dry_run:
|
||||
tmp = f"{f_path}.tmp.{os.getpid()}"
|
||||
with open(tmp, "w") as f:
|
||||
json.dump(fdata, f, indent=2)
|
||||
os.replace(tmp, f_path)
|
||||
|
||||
if rearm_sweeper and not dry_run:
|
||||
sweeper_py = Path("/home/super/Projects/NetVM/bin/followup-sweeper.py")
|
||||
if sweeper_py.exists():
|
||||
try:
|
||||
subprocess.run([sys.executable, str(sweeper_py), "--once"], timeout=10)
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
# Auto-remediate trusted approval blocks
|
||||
try:
|
||||
import approvals
|
||||
fleet_apps = approvals.check_fleet_approvals()
|
||||
for app in fleet_apps:
|
||||
if app.get("has_pending") and app.get("is_trusted"):
|
||||
node = app["node"]
|
||||
if not dry_run:
|
||||
approvals.allow_node_approval(node, caller="loop-remediate")
|
||||
remediated.append({
|
||||
"type": "approval_auto_allowed",
|
||||
"agent": node,
|
||||
"target": app.get("ip"),
|
||||
"action": f"Auto-approved trusted browser request on {node} ({app.get('ip')})"
|
||||
})
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
# 3. Alert on hard breakages if any exist
|
||||
if escalated and not dry_run:
|
||||
job_log = Path("/home/super/Projects/NetVM/job-log.jsonl")
|
||||
alert_msg = f"[HARD_BREAK_ALERT] Detected {len(escalated)} unresolvable loop failure(s): " + "; ".join(
|
||||
f"{b.get('type')} ({b.get('severity')}): {b.get('detail')}" for b in escalated[:3]
|
||||
)
|
||||
try:
|
||||
with open(job_log, "a", encoding="utf-8") as jf:
|
||||
jf.write(json.dumps({
|
||||
"ts": datetime.now(timezone.utc).isoformat(),
|
||||
"type": "hard_break_alert",
|
||||
"escalated_count": len(escalated),
|
||||
"items": escalated,
|
||||
"summary": alert_msg[:280]
|
||||
}) + "\n")
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
# Dispatch DM alert to opm
|
||||
dm_py = Path("/home/super/Projects/NetVM/bin/dm.py")
|
||||
if dm_py.exists():
|
||||
try:
|
||||
subprocess.run([
|
||||
sys.executable, str(dm_py), "send",
|
||||
"--agent", "super",
|
||||
"--to", "opm",
|
||||
"--target", "main",
|
||||
alert_msg[:800]
|
||||
], timeout=15, capture_output=True)
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
return {
|
||||
"ok": True,
|
||||
"remediated": remediated,
|
||||
"escalated": escalated,
|
||||
"dry_run": dry_run,
|
||||
"count": len(remediated),
|
||||
}
|
||||
|
||||
|
||||
|
||||
@@ -1,70 +0,0 @@
|
||||
#!/bin/bash
|
||||
# identity-audit-check.sh - Check VM identity audit for drift
|
||||
#
|
||||
# Lives on bl (/home/super/Projects/NetVM/bin/). Reads a bl-local cached
|
||||
# copy of the VM's identity audit JSON and reports drift.
|
||||
#
|
||||
# Why a cache: bl cannot SSH to the VM (VM only accepts the operator
|
||||
# container's key). The VM's hourly audit cron (/srv/board/bin/identity-audit.py)
|
||||
# should scp /srv/board/data/identity-audit.json to bl at:
|
||||
# /home/super/Projects/NetVM/var/identity-audit.json
|
||||
# after each run. Until that push is wired, the cache is refreshed manually
|
||||
# or by the identity-audit-watch cron.
|
||||
#
|
||||
# Called by:
|
||||
# - the identity-audit-watch cron (replaces inline SSH one-liner)
|
||||
# - box-ctl.py `identity-audit` action
|
||||
#
|
||||
# Exit codes:
|
||||
# 0 - clean (no drift; warnings are expected and silent)
|
||||
# 1 - drift detected (items printed to stdout, one per line)
|
||||
# 2 - audit cache missing/unreadable (needs attention)
|
||||
#
|
||||
# Output on drift: one drift item per line, prefixed with "DRIFT: "
|
||||
|
||||
set -u
|
||||
|
||||
CACHE_PATH="/home/super/Projects/NetVM/var/identity-audit.json"
|
||||
|
||||
# Allow override for testing
|
||||
if [ $# -ge 1 ] && [ -f "$1" ]; then
|
||||
CACHE_PATH="$1"
|
||||
fi
|
||||
|
||||
if [ ! -f "$CACHE_PATH" ]; then
|
||||
echo "ERROR: identity audit cache missing: $CACHE_PATH" >&2
|
||||
echo "The VM hourly audit should push /srv/board/data/identity-audit.json here after each run." >&2
|
||||
exit 2
|
||||
fi
|
||||
|
||||
audit_json=$(cat "$CACHE_PATH" 2>/dev/null)
|
||||
if [ -z "$audit_json" ]; then
|
||||
echo "ERROR: identity audit cache unreadable: $CACHE_PATH" >&2
|
||||
exit 2
|
||||
fi
|
||||
|
||||
# Parse with python3
|
||||
drift_output=$(printf '%s' "$audit_json" | python3 -c "
|
||||
import json, sys
|
||||
try:
|
||||
data = json.load(sys.stdin)
|
||||
except Exception as e:
|
||||
print('ERROR: invalid JSON in audit cache: %s' % e, file=sys.stderr)
|
||||
sys.exit(2)
|
||||
for item in data.get('drift', []):
|
||||
print('DRIFT: ' + str(item))
|
||||
" 2>&1)
|
||||
parse_rc=$?
|
||||
|
||||
if [ $parse_rc -eq 2 ]; then
|
||||
echo "$drift_output" >&2
|
||||
exit 2
|
||||
fi
|
||||
|
||||
if [ -n "$drift_output" ]; then
|
||||
echo "$drift_output"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Clean: no drift. Warnings are expected (agents not dialed in) — stay silent.
|
||||
exit 0
|
||||
@@ -1,296 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Identity variance testing framework.
|
||||
|
||||
Sends gentle, controlled probes from each NetVM node's Warp identity and
|
||||
measures per-identity response behavior so that future rate limits can be
|
||||
attributed (per-identity vs per-IP vs time-based vs random).
|
||||
|
||||
Design goals:
|
||||
- Very gentle traffic: 2 probes per node per cycle, default 10-min cycle.
|
||||
- Never logs secrets: only sha256 hashes of WireGuard private keys.
|
||||
- JSONL results log: one line per probe, machine-readable.
|
||||
|
||||
Usage:
|
||||
identity-variance-test.py probe # run one probe cycle over all nodes
|
||||
identity-variance-test.py analyze [--since HOURS] [--log PATH]
|
||||
|
||||
Probes per node:
|
||||
1. https://www.cloudflare.com/cdn-cgi/trace (identity + egress IP Cloudflare sees)
|
||||
2. https://muse.ai/ (production-relevant landing page)
|
||||
|
||||
Exit codes: 0 ok, 1 partial (some nodes failed), 2 fatal.
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
from datetime import datetime, timezone
|
||||
|
||||
DEFAULT_LOG = os.path.expanduser(
|
||||
"~/Projects/NetVM/.state/identity-variance/variance.jsonl"
|
||||
)
|
||||
NODES = ["muse", "pip", "646", "opm", "def"]
|
||||
|
||||
PROBES = [
|
||||
("cf-trace", "https://www.cloudflare.com/cdn-cgi/trace"),
|
||||
("muse-landing", "https://muse.ai/"),
|
||||
]
|
||||
|
||||
CURL_TIMEOUT = 15
|
||||
|
||||
|
||||
def sh(cmd, timeout=30):
|
||||
"""Run cmd (list) and return (rc, stdout, stderr)."""
|
||||
try:
|
||||
p = subprocess.run(
|
||||
cmd, capture_output=True, text=True, timeout=timeout
|
||||
)
|
||||
return p.returncode, p.stdout, p.stderr
|
||||
except subprocess.TimeoutExpired as e:
|
||||
return 124, (e.stdout or ""), "timeout"
|
||||
except FileNotFoundError:
|
||||
return 127, "", "command not found"
|
||||
|
||||
|
||||
def node_netns_exists(node):
|
||||
rc, out, _ = sh(["sudo", "-n", "ip", "netns", "list"])
|
||||
return rc == 0 and f"warp-{node}" in out
|
||||
|
||||
|
||||
def identity_hash(node):
|
||||
"""Return truncated sha256 of the node's WireGuard private key (never the key)."""
|
||||
rc, out, _ = sh(["sudo", "-n", "cat", f"/etc/netvm/{node}.conf"])
|
||||
if rc == 0:
|
||||
for line in out.splitlines():
|
||||
m = re.match(r"\s*PrivateKey\s*=\s*(\S+)", line)
|
||||
if m:
|
||||
return hashlib.sha256(m.group(1).encode()).hexdigest()[:16]
|
||||
return None
|
||||
|
||||
|
||||
def probe_node(node, target_name, url):
|
||||
"""One probe from inside the node's netns. Returns dict."""
|
||||
# -w fields: http_code, time_total, size_download, remote_ip
|
||||
fmt = "%{http_code} %{time_total} %{size_download} %{remote_ip}"
|
||||
cmd = [
|
||||
"sudo", "-n", "ip", "netns", "exec", f"warp-{node}",
|
||||
"curl", "-s", "-o", "/dev/null", "-m", str(CURL_TIMEOUT),
|
||||
"-w", fmt, url,
|
||||
]
|
||||
started = datetime.now(timezone.utc)
|
||||
rc, out, err = sh(cmd, timeout=CURL_TIMEOUT + 10)
|
||||
ended = datetime.now(timezone.utc)
|
||||
|
||||
rec = {
|
||||
"ts": started.isoformat(),
|
||||
"node": node,
|
||||
"probe": target_name,
|
||||
"url": url,
|
||||
"curl_rc": rc,
|
||||
}
|
||||
parts = out.strip().split()
|
||||
if rc == 0 and len(parts) == 4:
|
||||
try:
|
||||
rec["http_status"] = int(parts[0])
|
||||
except ValueError:
|
||||
rec["http_status"] = None
|
||||
try:
|
||||
rec["response_ms"] = round(float(parts[1]) * 1000, 1)
|
||||
except ValueError:
|
||||
rec["response_ms"] = None
|
||||
try:
|
||||
rec["bytes"] = int(parts[2])
|
||||
except ValueError:
|
||||
rec["bytes"] = None
|
||||
rec["remote_ip"] = parts[3] if parts[3] != "0.0.0.0" else None
|
||||
else:
|
||||
rec["http_status"] = None
|
||||
rec["response_ms"] = None
|
||||
rec["bytes"] = None
|
||||
rec["remote_ip"] = None
|
||||
rec["curl_error"] = (err or "curl failed").strip()[:200]
|
||||
|
||||
rec["rate_limited"] = rec["http_status"] in (429,)
|
||||
rec["blocked"] = rec["http_status"] in (403,)
|
||||
rec["ok"] = rec["http_status"] is not None and 200 <= rec["http_status"] < 400
|
||||
rec["elapsed_wall_ms"] = round(
|
||||
(ended - started).total_seconds() * 1000, 1
|
||||
)
|
||||
return rec
|
||||
|
||||
|
||||
def egress_ip(node):
|
||||
"""Best-effort egress IP as seen from inside the netns."""
|
||||
rc, out, _ = sh([
|
||||
"sudo", "-n", "ip", "netns", "exec", f"warp-{node}",
|
||||
"curl", "-s", "-m", "10", "https://api.ipify.org",
|
||||
], timeout=20)
|
||||
if rc == 0 and re.fullmatch(r"[0-9a-fA-F.:]+", out.strip()):
|
||||
return out.strip()
|
||||
return None
|
||||
|
||||
|
||||
def run_probe_cycle(log_path):
|
||||
os.makedirs(os.path.dirname(log_path), exist_ok=True)
|
||||
results = []
|
||||
any_ok, any_fail = False, False
|
||||
idhash = {}
|
||||
for node in NODES:
|
||||
if not node_netns_exists(node):
|
||||
results.append({
|
||||
"ts": datetime.now(timezone.utc).isoformat(),
|
||||
"node": node, "probe": "node-skip",
|
||||
"ok": False, "note": "netns warp-%s missing" % node,
|
||||
})
|
||||
any_fail = True
|
||||
continue
|
||||
idhash[node] = identity_hash(node)
|
||||
for target_name, url in PROBES:
|
||||
rec = probe_node(node, target_name, url)
|
||||
rec["identity_hash"] = idhash[node]
|
||||
results.append(rec)
|
||||
if rec["ok"]:
|
||||
any_ok = True
|
||||
else:
|
||||
any_fail = True
|
||||
time.sleep(1) # gentle pacing between probes
|
||||
|
||||
# Attach egress IP per node (one lookup per node, cached per cycle).
|
||||
egress = {}
|
||||
for node in {r["node"] for r in results if r.get("probe") != "node-skip"}:
|
||||
egress[node] = egress_ip(node)
|
||||
for rec in results:
|
||||
if rec.get("probe") != "node-skip":
|
||||
rec["egress_ip"] = egress.get(rec["node"])
|
||||
|
||||
with open(log_path, "a") as f:
|
||||
for rec in results:
|
||||
f.write(json.dumps(rec) + "\n")
|
||||
|
||||
print(json.dumps({
|
||||
"cycle_ts": datetime.now(timezone.utc).isoformat(),
|
||||
"log": log_path,
|
||||
"records": len(results),
|
||||
"ok": sum(1 for r in results if r.get("ok")),
|
||||
"failed": sum(1 for r in results if not r.get("ok")),
|
||||
"nodes": sorted({r["node"] for r in results}),
|
||||
"egress": egress,
|
||||
"identities": idhash,
|
||||
}, indent=2))
|
||||
|
||||
if not any_ok:
|
||||
return 2
|
||||
return 1 if any_fail else 0
|
||||
|
||||
|
||||
def analyze(log_path, since_hours=None):
|
||||
if not os.path.exists(log_path):
|
||||
print("no log yet at %s" % log_path, file=sys.stderr)
|
||||
return 2
|
||||
cutoff = None
|
||||
if since_hours:
|
||||
cutoff = time.time() - since_hours * 3600
|
||||
|
||||
per_node = {}
|
||||
rate_limit_events = []
|
||||
identity_changes = {}
|
||||
egress_by_node = {}
|
||||
|
||||
with open(log_path) as f:
|
||||
for line in f:
|
||||
line = line.strip()
|
||||
if not line:
|
||||
continue
|
||||
try:
|
||||
r = json.loads(line)
|
||||
except json.JSONDecodeError:
|
||||
continue
|
||||
if r.get("probe") == "node-skip":
|
||||
continue
|
||||
try:
|
||||
ts = datetime.fromisoformat(r["ts"]).timestamp()
|
||||
except (ValueError, KeyError):
|
||||
continue
|
||||
if cutoff and ts < cutoff:
|
||||
continue
|
||||
node = r["node"]
|
||||
st = per_node.setdefault(node, {
|
||||
"probes": 0, "ok": 0, "ms": [], "429": 0, "403": 0,
|
||||
"statuses": {}, "identities": set(), "probes_by_target": {},
|
||||
})
|
||||
st["probes"] += 1
|
||||
if r.get("ok"):
|
||||
st["ok"] += 1
|
||||
if r.get("response_ms") is not None:
|
||||
st["ms"].append(r["response_ms"])
|
||||
if r.get("rate_limited"):
|
||||
st["429"] += 1
|
||||
rate_limit_events.append((r["ts"], node, r["probe"]))
|
||||
if r.get("blocked"):
|
||||
st["403"] += 1
|
||||
s = r.get("http_status")
|
||||
st["statuses"][str(s)] = st["statuses"].get(str(s), 0) + 1
|
||||
if r.get("identity_hash"):
|
||||
st["identities"].add(r["identity_hash"])
|
||||
t = r.get("probe")
|
||||
st["probes_by_target"][t] = st["probes_by_target"].get(t, 0) + 1
|
||||
if r.get("egress_ip"):
|
||||
egress_by_node.setdefault(node, set()).add(r["egress_ip"])
|
||||
|
||||
def pct(vals, p):
|
||||
if not vals:
|
||||
return None
|
||||
s = sorted(vals)
|
||||
return round(s[min(len(s) - 1, int(p / 100 * len(s)))], 1)
|
||||
|
||||
report = {"log": log_path, "nodes": {}}
|
||||
for node in sorted(per_node):
|
||||
st = per_node[node]
|
||||
report["nodes"][node] = {
|
||||
"probes": st["probes"],
|
||||
"ok_rate": round(st["ok"] / st["probes"], 3) if st["probes"] else 0,
|
||||
"latency_ms": {
|
||||
"mean": round(sum(st["ms"]) / len(st["ms"]), 1) if st["ms"] else None,
|
||||
"p50": pct(st["ms"], 50),
|
||||
"p99": pct(st["ms"], 99),
|
||||
},
|
||||
"http_429": st["429"],
|
||||
"http_403": st["403"],
|
||||
"statuses": st["statuses"],
|
||||
"identity_rotations": max(0, len(st["identities"]) - 1),
|
||||
"egress_ips": sorted(egress_by_node.get(node, set())),
|
||||
}
|
||||
if len(st["identities"]) > 1:
|
||||
identity_changes[node] = sorted(st["identities"])
|
||||
|
||||
# Divergence analysis: did all nodes see the same fate at the same time?
|
||||
report["rate_limit_events"] = [
|
||||
{"ts": ts, "node": n, "probe": p} for ts, n, p in rate_limit_events[-50:]
|
||||
]
|
||||
report["identity_changes"] = identity_changes
|
||||
print(json.dumps(report, indent=2))
|
||||
return 0
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser(description="Identity variance testing framework")
|
||||
sub = ap.add_subparsers(dest="cmd", required=True)
|
||||
p_probe = sub.add_parser("probe", help="run one probe cycle")
|
||||
p_probe.add_argument("--log", default=DEFAULT_LOG)
|
||||
p_an = sub.add_parser("analyze", help="summarize variance data")
|
||||
p_an.add_argument("--log", default=DEFAULT_LOG)
|
||||
p_an.add_argument("--since", type=float, default=None,
|
||||
help="only include last N hours")
|
||||
args = ap.parse_args()
|
||||
if args.cmd == "probe":
|
||||
sys.exit(run_probe_cycle(args.log))
|
||||
sys.exit(analyze(args.log, args.since))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -1,576 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
job-dispatch.py: Dispatch a job by sending a DM to an agent.
|
||||
|
||||
Usage:
|
||||
job-dispatch.py <job_name> [--dry-run]
|
||||
|
||||
Reads /home/super/Projects/NetVM/jobs/<job_name>.json,
|
||||
renders the prompt template, sends DM via dm.py, logs to job-log.jsonl.
|
||||
|
||||
Part of the JOB system (see docs/JOB-SPEC.md).
|
||||
|
||||
Sidechat-first policy (2026-10-04): a job that resolves to target "main"
|
||||
without explicit opt-in fails loudly instead of saturating main threads.
|
||||
Opt in via job JSON "allow_main_chat": true, or --allow-main-chat.
|
||||
"""
|
||||
|
||||
import sys
|
||||
import re
|
||||
import os
|
||||
import json
|
||||
import subprocess
|
||||
import uuid
|
||||
import argparse
|
||||
from datetime import datetime, timezone
|
||||
from pathlib import Path
|
||||
|
||||
# Paths
|
||||
NETVM_ROOT = Path("/home/super/Projects/NetVM")
|
||||
sys.path.insert(0, str(NETVM_ROOT / "bin"))
|
||||
try:
|
||||
import pipeline_engine
|
||||
HAS_PIPELINE = True
|
||||
except ImportError:
|
||||
HAS_PIPELINE = False
|
||||
|
||||
JOBS_DIR = NETVM_ROOT / "jobs"
|
||||
DM_PY = NETVM_ROOT / "bin" / "dm.py"
|
||||
CHAT_API = NETVM_ROOT / "bin" / "muse-chat-api.py"
|
||||
NETVM_EXEC = "/home/super/Projects/NetVM/bin/netvm-exec.sh"
|
||||
JOB_LOG = NETVM_ROOT / "job-log.jsonl"
|
||||
SIDECHAT_STATE = NETVM_ROOT / "job-sidechats.json"
|
||||
|
||||
def load_sidechat_state():
|
||||
if SIDECHAT_STATE.exists():
|
||||
try:
|
||||
return json.loads(SIDECHAT_STATE.read_text())
|
||||
except Exception:
|
||||
return {}
|
||||
return {}
|
||||
|
||||
def save_sidechat_state(state):
|
||||
tmp = SIDECHAT_STATE.with_suffix(".tmp")
|
||||
tmp.write_text(json.dumps(state, indent=2))
|
||||
tmp.replace(SIDECHAT_STATE)
|
||||
|
||||
def extract_uuid(url):
|
||||
m = re.search(r"/thread/([0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12})", url or "")
|
||||
return m.group(1) if m else None
|
||||
|
||||
def get_current_url(agent="opm"):
|
||||
try:
|
||||
cmd = [NETVM_EXEC, agent, "--", "python3", str(CHAT_API),
|
||||
"--account", agent, "url"]
|
||||
r = subprocess.run(cmd, capture_output=True, text=True, timeout=30)
|
||||
if r.returncode == 0:
|
||||
return r.stdout.strip()
|
||||
except Exception:
|
||||
pass
|
||||
return None
|
||||
|
||||
def use_sidechat_uuid(agent, thread_uuid, dry_run=False):
|
||||
if dry_run:
|
||||
return False
|
||||
cmd = [NETVM_EXEC, agent, "--", "python3", str(CHAT_API),
|
||||
"--account", agent, "sidechat", "use", thread_uuid]
|
||||
try:
|
||||
r = subprocess.run(cmd, capture_output=True, text=True, timeout=30)
|
||||
return r.returncode == 0
|
||||
except Exception:
|
||||
return False
|
||||
|
||||
# Add bin to path for rate_limiter
|
||||
sys.path.insert(0, str(NETVM_ROOT / "bin"))
|
||||
try:
|
||||
from rate_limiter import rate_limit_wait
|
||||
HAS_RATE_LIMITER = True
|
||||
except ImportError:
|
||||
HAS_RATE_LIMITER = False
|
||||
|
||||
def log_event(event_type, data):
|
||||
"""Append event to job-log.jsonl"""
|
||||
entry = {
|
||||
"ts": datetime.now(timezone.utc).isoformat(),
|
||||
"type": event_type,
|
||||
**data
|
||||
}
|
||||
with open(JOB_LOG, "a") as f:
|
||||
f.write(json.dumps(entry) + "\n")
|
||||
|
||||
def load_job(job_name):
|
||||
"""Load job JSON definition"""
|
||||
# Try .json first, then .yaml (for compatibility)
|
||||
job_file = JOBS_DIR / f"{job_name}.json"
|
||||
if not job_file.exists():
|
||||
job_file = JOBS_DIR / f"{job_name}.yaml"
|
||||
if not job_file.exists():
|
||||
print(f"Error: Job '{job_name}' not found in {JOBS_DIR}", file=sys.stderr)
|
||||
sys.exit(1)
|
||||
print(f"Warning: YAML not supported (no PyYAML). Convert {job_file} to JSON.", file=sys.stderr)
|
||||
sys.exit(1)
|
||||
|
||||
with open(job_file) as f:
|
||||
return json.load(f)
|
||||
|
||||
def render_prompt(template, variables):
|
||||
"""Render prompt template with variables"""
|
||||
# Simple {var} substitution
|
||||
result = template
|
||||
for key, value in variables.items():
|
||||
result = result.replace(f"{{{key}}}", str(value))
|
||||
return result
|
||||
|
||||
# ---- follow-up tracking (DM follow-up system integration) ----------------
|
||||
# Jobs opt in via a "followup" block in the job JSON:
|
||||
#
|
||||
# "followup": {
|
||||
# "expect_reply": true, # required: enables tracking
|
||||
# "timeout": "1h", # duration ("30s","15m","2h","1d") or seconds
|
||||
# # int; default "1h" (3600s)
|
||||
# "nudges": 2, # 0..10, default 2
|
||||
# "escalate": "opm", # identity string, default "opm"
|
||||
# "route": "646-pip-coord" # optional route_id
|
||||
# }
|
||||
#
|
||||
# The dispatcher translates this into dm.py --tag flags using the canonical
|
||||
# vocabulary (box-threads/DEPLOY-DECISIONS.md). dm.py strips the tags from
|
||||
# delivered text and creates a dm_followup request-store record after
|
||||
# SENT+VERIFIED. Jobs without a followup block behave exactly as today.
|
||||
#
|
||||
# LIMITATIONS (v1):
|
||||
# - The heartbeat job NEVER gets follow-ups (loopback health check).
|
||||
# Hardcoded guard below; a followup block on heartbeat is ignored loudly.
|
||||
# - Sidechat sends (muse-chat-api.py direct path) do not go through dm.py,
|
||||
# so --tag flags cannot attach. v2 needs a record-creation path that does
|
||||
# not send (e.g. POST /api/box/followups, or a bl->VM queue; bl cannot
|
||||
# currently SSH to the VM). The dispatcher logs a warning when a
|
||||
# sidechat-targeted job has followup enabled.
|
||||
|
||||
HEARTBEAT_JOB_NAME = "heartbeat"
|
||||
|
||||
|
||||
def parse_followup_duration(value):
|
||||
"""Parse a followup timeout into seconds. Accepts int (seconds) or
|
||||
strings like '30s', '15m', '2h', '1d'. Returns int seconds.
|
||||
Raises ValueError on bad input."""
|
||||
if isinstance(value, int) and not isinstance(value, bool):
|
||||
s = value
|
||||
elif isinstance(value, str):
|
||||
m = re.fullmatch(r"(\d+)\s*([smhd])?", value.strip().lower())
|
||||
if not m:
|
||||
raise ValueError("bad duration %r" % (value,))
|
||||
n = int(m.group(1))
|
||||
unit = m.group(2) or "s"
|
||||
s = n * {"s": 1, "m": 60, "h": 3600, "d": 86400}[unit]
|
||||
else:
|
||||
raise ValueError("timeout must be int seconds or duration string")
|
||||
if not 60 <= s <= 604800:
|
||||
raise ValueError("timeout must be 60..604800s (1m..7d), got %d" % s)
|
||||
return s
|
||||
|
||||
|
||||
def build_followup_tags(followup):
|
||||
"""Translate a job's followup block into dm.py --tag arguments.
|
||||
Returns a flat list like ['--tag', 'reply:timeout=3600', ...].
|
||||
Returns [] if followup is falsy or expect_reply is not true.
|
||||
Raises ValueError on invalid config (caller logs a warning and sends
|
||||
the DM untagged -- the job itself must never fail over this)."""
|
||||
if not followup or not followup.get("expect_reply"):
|
||||
return []
|
||||
args = []
|
||||
# Bare trigger. dm.py's parse_tags splits each --tag on '='; an empty
|
||||
# value means "present". If the deployed dm.py requires a non-empty
|
||||
# value for this key, use 'reply:expected=true' instead.
|
||||
args += ["--tag", "reply:expected="]
|
||||
if "timeout" in followup:
|
||||
s = parse_followup_duration(followup["timeout"])
|
||||
args += ["--tag", "reply:timeout=%d" % s]
|
||||
if "nudges" in followup:
|
||||
n = followup["nudges"]
|
||||
if not isinstance(n, int) or isinstance(n, bool) or not 0 <= n <= 10:
|
||||
raise ValueError("nudges must be int 0..10")
|
||||
args += ["--tag", "reply:nudges=%d" % n]
|
||||
if "escalate" in followup:
|
||||
e = followup["escalate"]
|
||||
if not isinstance(e, str) or not re.fullmatch(r"[a-z0-9_-]{1,64}", e):
|
||||
raise ValueError("escalate must be an identity string")
|
||||
args += ["--tag", "reply:escalate=%s" % e]
|
||||
if "route" in followup:
|
||||
r = followup["route"]
|
||||
if not isinstance(r, str) or not re.fullmatch(r"[a-z0-9_-]{1,64}", r):
|
||||
raise ValueError("route must be a route_id string")
|
||||
args += ["--tag", "route:%s" % r]
|
||||
# 'thread' is intentionally not settable from job JSON; it names a
|
||||
# specific existing thread and is filled by the dispatcher when known.
|
||||
return args
|
||||
|
||||
|
||||
def send_dm(agent, target, message, dry_run=False, followup_tags=None,
|
||||
allow_main_chat=False):
|
||||
"""Send DM via dm.py. followup_tags: flat ['--tag', 'k=v', ...] list
|
||||
from build_followup_tags(), or None. allow_main_chat passes the explicit
|
||||
main-chat opt-in through to dm.py (sidechat-first policy)."""
|
||||
if dry_run:
|
||||
print(f"[DRY RUN] Would send to {agent} ({target}):")
|
||||
if followup_tags:
|
||||
print(f"[DRY RUN] With follow-up tags: {' '.join(followup_tags)}")
|
||||
print(message[:200] + "..." if len(message) > 200 else message)
|
||||
return "dry-run-id"
|
||||
|
||||
# Rate limit
|
||||
if HAS_RATE_LIMITER:
|
||||
rate_limit_wait(agent)
|
||||
|
||||
# Cryptographic attestation: only sign if explicitly requested by job config
|
||||
# to avoid blowing up agent context windows with massive base64 SSH signature blocks.
|
||||
signed_payload = None
|
||||
if os.environ.get("JOB_REQUIRE_SIGNATURE") == "1":
|
||||
dm_sign_sh = NETVM_ROOT / "bin" / "dm-sign.sh"
|
||||
priv_key = Path(os.path.expanduser("~/.ssh/id_ed25519"))
|
||||
if dm_sign_sh.exists() and priv_key.exists():
|
||||
try:
|
||||
sign_res = subprocess.run(
|
||||
[str(dm_sign_sh), "--from", "super", "--key", str(priv_key), message],
|
||||
capture_output=True, text=True, timeout=10
|
||||
)
|
||||
if sign_res.returncode == 0 and "-----BEGIN SSH SIGNATURE-----" in sign_res.stdout:
|
||||
signed_payload = sign_res.stdout.strip()
|
||||
id_m = re.search(r"\[id:([a-f0-9]+)\]", signed_payload)
|
||||
proof_id = id_m.group(1) if id_m else None
|
||||
if proof_id:
|
||||
proof_data = {
|
||||
"id": proof_id,
|
||||
"signer": "super",
|
||||
"target_agent": agent,
|
||||
"target_conversation": target,
|
||||
"raw_payload": signed_payload,
|
||||
"ts": datetime.now(timezone.utc).isoformat()
|
||||
}
|
||||
try:
|
||||
import urllib.request
|
||||
req = urllib.request.Request(
|
||||
"https://crypt.muse-dev.online/proofs",
|
||||
data=json.dumps(proof_data).encode("utf-8"),
|
||||
headers={"Content-Type": "application/json", "User-Agent": "job-dispatch/1.0"},
|
||||
method="POST"
|
||||
)
|
||||
with urllib.request.urlopen(req, timeout=3) as resp:
|
||||
pass
|
||||
except Exception as pe:
|
||||
sys.stderr.write(f"warning: proof registration to crypt.muse-dev.online failed: {pe}\n")
|
||||
except Exception as se:
|
||||
sys.stderr.write(f"warning: dm signing failed: {se}\n")
|
||||
|
||||
payload_to_send = signed_payload or message
|
||||
|
||||
# Try fast hybrid gateway send if target resolves to UUID
|
||||
target_uuid = None
|
||||
try:
|
||||
import dm
|
||||
target_uuid = dm.resolve_sidechat_target(target, agent)
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
if target_uuid and re.fullmatch(r"[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}", target_uuid.lower()):
|
||||
try:
|
||||
import muse_hybrid
|
||||
send_node = agent if agent in ("muse", "pip", "646", "opm") else "opm"
|
||||
res, err = muse_hybrid.send_message(send_node, payload_to_send, thread_id=target_uuid, wait=0)
|
||||
if res and not err:
|
||||
m_id = None
|
||||
id_m = re.search(r"\[id:([a-f0-9]+)\]", payload_to_send)
|
||||
if id_m:
|
||||
m_id = id_m.group(1)
|
||||
log_entry = {
|
||||
"type": "sent",
|
||||
"id": m_id or (res.get("reply", {}).get("message_id") if isinstance(res, dict) else "gateway"),
|
||||
"agent": "opm",
|
||||
"to": agent,
|
||||
"target": target,
|
||||
"thread_uuid": target_uuid,
|
||||
"transport": "gateway",
|
||||
"verified": True,
|
||||
"ts": datetime.now(timezone.utc).isoformat()
|
||||
}
|
||||
dm_log_path = NETVM_ROOT / "dm-log.jsonl"
|
||||
with open(dm_log_path, "a", encoding="utf-8") as lf:
|
||||
lf.write(json.dumps(log_entry) + "\n")
|
||||
return m_id or "gateway-verified"
|
||||
sys.stderr.write(f"warning: fast gateway send fallback: {err or res}\n")
|
||||
except Exception as ge:
|
||||
sys.stderr.write(f"warning: fast gateway send fallback: {ge}\n")
|
||||
|
||||
if signed_payload:
|
||||
cmd = ([str(DM_PY), "send", "--agent", "opm", "--to", agent,
|
||||
"--target", target, "--raw"]
|
||||
+ (["--allow-main-chat"] if (target == "main" and allow_main_chat) else [])
|
||||
+ (followup_tags or []) + [signed_payload])
|
||||
else:
|
||||
cmd = ([str(DM_PY), "send", "--agent", "opm", "--to", agent,
|
||||
"--target", target]
|
||||
+ (["--allow-main-chat"] if (target == "main" and allow_main_chat) else [])
|
||||
+ (followup_tags or []) + [message])
|
||||
|
||||
result = subprocess.run(cmd, capture_output=True, text=True, timeout=120)
|
||||
|
||||
if result.returncode != 0:
|
||||
print(f"DM send failed: {result.stderr}", file=sys.stderr)
|
||||
return None
|
||||
|
||||
# Extract message ID from output (format: DM <id> ... SENT and VERIFIED)
|
||||
output = result.stdout.strip()
|
||||
m = re.search(r"DM\s+([a-f0-9]{8})", output)
|
||||
return m.group(1) if m else "verified"
|
||||
|
||||
def create_sidechat(sender_agent, dry_run=False):
|
||||
"""Create a sidechat via muse-chat-api.py in sender's context.
|
||||
Returns True on success (browser now on new sidechat), False on failure."""
|
||||
if dry_run:
|
||||
print(f"[DRY RUN] Would create sidechat for {sender_agent}")
|
||||
return True
|
||||
|
||||
cmd = [NETVM_EXEC, sender_agent, "--", "python3", str(CHAT_API),
|
||||
"--account", sender_agent, "sidechat", "create"]
|
||||
try:
|
||||
result = subprocess.run(cmd, capture_output=True, text=True, timeout=90)
|
||||
output = result.stdout.strip()
|
||||
err = result.stderr.strip()
|
||||
# Success if we see "Created:" (URL may be /thread/new placeholder)
|
||||
if "Created:" in output:
|
||||
print(f"Sidechat created", file=sys.stderr)
|
||||
return True
|
||||
print(f"Sidechat create failed. stdout: {output[:300]}", file=sys.stderr)
|
||||
print(f"Sidechat create stderr: {err[:300]}", file=sys.stderr)
|
||||
print(f"Return code: {result.returncode}", file=sys.stderr)
|
||||
return False
|
||||
except Exception as e:
|
||||
print(f"Sidechat creation failed: {e}", file=sys.stderr)
|
||||
return False
|
||||
|
||||
|
||||
def send_to_current_chat(sender_agent, message, dry_run=False):
|
||||
"""Send message to current chat via muse-chat-api.py (no navigation).
|
||||
Used after sidechat create - browser is already on the new chat."""
|
||||
if dry_run:
|
||||
print(f"[DRY RUN] Would send to current chat: {message[:100]}...")
|
||||
return "dry-run-id"
|
||||
|
||||
if HAS_RATE_LIMITER:
|
||||
rate_limit_wait(sender_agent)
|
||||
|
||||
cmd = [NETVM_EXEC, sender_agent, "--", "python3", str(CHAT_API),
|
||||
"--account", sender_agent, "send", message]
|
||||
try:
|
||||
result = subprocess.run(cmd, capture_output=True, text=True, timeout=60)
|
||||
if result.returncode == 0:
|
||||
return "sent-to-sidechat"
|
||||
print(f"Send failed: {result.stderr[:200]}", file=sys.stderr)
|
||||
return None
|
||||
except Exception as e:
|
||||
print(f"Send failed: {e}", file=sys.stderr)
|
||||
return None
|
||||
|
||||
def main():
|
||||
p = argparse.ArgumentParser(description="Dispatch a job by sending a DM to an agent.")
|
||||
p.add_argument("job_name", help="Name of the job (without .json)")
|
||||
p.add_argument("--dry-run", action="store_true", help="Print what would be sent without sending")
|
||||
p.add_argument("--pipeline-run", default=os.environ.get("CHAIN_PIPELINE_RUN_ID"),
|
||||
help="Pipeline run ID if running as part of a pipeline")
|
||||
p.add_argument("--step-n", type=int, default=int(os.environ.get("CHAIN_STEP_N", "1")),
|
||||
help="Step sequence number in the pipeline")
|
||||
p.add_argument("--allow-main-chat", action="store_true",
|
||||
help="Explicit opt-in: allow this job to dispatch to main chat (refused by default per sidechat-first policy)")
|
||||
args = p.parse_args()
|
||||
|
||||
job_name = args.job_name
|
||||
dry_run = args.dry_run
|
||||
pipeline_run_id = args.pipeline_run
|
||||
step_n = args.step_n
|
||||
|
||||
# Load job
|
||||
job = load_job(job_name)
|
||||
|
||||
# Generate job_id
|
||||
job_id = f"{job_name}-{datetime.now(timezone.utc).strftime('%Y%m%d-%H%M%S')}-{uuid.uuid4().hex[:8]}"
|
||||
|
||||
# Variables for template
|
||||
variables = {
|
||||
"job_id": job_id,
|
||||
"job_name": job_name,
|
||||
"date": datetime.now(timezone.utc).strftime("%Y-%m-%d"),
|
||||
"datetime": datetime.now(timezone.utc).isoformat(),
|
||||
"prev_job_id": os.environ.get("CHAIN_PREV_JOB_ID", ""),
|
||||
"prev_result": os.environ.get("CHAIN_PREV_RESULT", ""),
|
||||
"pipeline_run_id": pipeline_run_id or "",
|
||||
"step_n": step_n,
|
||||
}
|
||||
|
||||
# Follow-up tracking (opt-in via job JSON "followup" block; see helpers).
|
||||
# The heartbeat job is a loopback health check and must never be tracked.
|
||||
followup_cfg = job.get("followup")
|
||||
followup_tags = []
|
||||
if followup_cfg:
|
||||
if job_name == HEARTBEAT_JOB_NAME:
|
||||
print(f"Warning: job '{job_name}' must not use follow-up "
|
||||
f"tracking (loopback); ignoring followup block",
|
||||
file=sys.stderr)
|
||||
log_event("job_followup_skipped",
|
||||
{"job_id": job_id, "reason": "heartbeat_loopback"})
|
||||
else:
|
||||
try:
|
||||
followup_tags = build_followup_tags(followup_cfg)
|
||||
if followup_tags:
|
||||
log_event("job_followup_armed",
|
||||
{"job_id": job_id, "tags": followup_tags})
|
||||
except ValueError as e:
|
||||
print(f"Warning: invalid followup block: {e}; "
|
||||
f"sending untagged", file=sys.stderr)
|
||||
log_event("job_followup_invalid",
|
||||
{"job_id": job_id, "error": str(e)})
|
||||
|
||||
# Render prompt
|
||||
prompt_template = job.get("prompt_template", "")
|
||||
if not prompt_template:
|
||||
print(f"Error: Job '{job_name}' has no prompt_template", file=sys.stderr)
|
||||
sys.exit(1)
|
||||
|
||||
rendered = render_prompt(prompt_template, variables)
|
||||
|
||||
# Determine recipient agent and target
|
||||
agent = job.get("agent", "muse")
|
||||
target = "main"
|
||||
|
||||
if pipeline_run_id:
|
||||
if HAS_PIPELINE:
|
||||
p_entry = pipeline_engine.get_pipeline(pipeline_run_id)
|
||||
if not p_entry:
|
||||
p_entry = pipeline_engine.create_pipeline(job_name, run_id=pipeline_run_id)
|
||||
target = p_entry.get("target") or f"pipe-{pipeline_run_id.split('-')[-1]}"
|
||||
else:
|
||||
target = f"pipe-{pipeline_run_id.split('-')[-1]}"
|
||||
elif job.get("sidechat", {}).get("create"):
|
||||
sc_cfg = job.get("sidechat", {})
|
||||
sc_name = render_prompt(sc_cfg.get("name_template", "job-{job_name}-{date}"), variables)
|
||||
reuse_key = sc_cfg.get("reuse_key")
|
||||
|
||||
# Check if reuse_key exists in job-sidechats.json and thread is still alive
|
||||
sc_state = load_sidechat_state()
|
||||
reused_uuid = None
|
||||
if reuse_key and reuse_key in sc_state:
|
||||
val = sc_state[reuse_key]
|
||||
cand_uuid = val.get("thread_uuid") if isinstance(val, dict) else val
|
||||
if cand_uuid:
|
||||
try:
|
||||
import muse_hybrid
|
||||
threads, err = muse_hybrid.get_threads(agent)
|
||||
if not err and threads:
|
||||
thread_ids = [t.get("session_id") for t in threads]
|
||||
if cand_uuid in thread_ids:
|
||||
reused_uuid = cand_uuid
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
if reused_uuid:
|
||||
target = reused_uuid
|
||||
else:
|
||||
# Spawn a brand new sidechat/channel via fast headless gateway!
|
||||
channel_title = sc_name
|
||||
try:
|
||||
import muse_hybrid
|
||||
res, err = muse_hybrid.start_session(agent, title=channel_title)
|
||||
if res and not err and res.get("session_id"):
|
||||
new_uuid = res.get("session_id")
|
||||
key_to_save = reuse_key or sc_name
|
||||
is_persistent = bool(reuse_key)
|
||||
sc_state[key_to_save] = {
|
||||
"thread_uuid": new_uuid,
|
||||
"agent": agent,
|
||||
"title": channel_title,
|
||||
"type": "persistent" if is_persistent else "ephemeral",
|
||||
"created_at": datetime.now(timezone.utc).isoformat()
|
||||
}
|
||||
save_sidechat_state(sc_state)
|
||||
target = new_uuid
|
||||
print(f"Spawned new sidechat channel '{channel_title}' ({new_uuid}) for {agent}")
|
||||
else:
|
||||
target = sc_name
|
||||
except Exception as e:
|
||||
sys.stderr.write(f"warning: fast gateway session-start exception ({e}), falling back to name {sc_name}\n")
|
||||
target = sc_name
|
||||
elif job.get("dm_target"):
|
||||
target = job.get("dm_target").strip()
|
||||
elif job.get("target"):
|
||||
target = job.get("target").strip()
|
||||
|
||||
# Sidechat-first policy (2026-10-04): refuse to dispatch to main chat
|
||||
# unless the job explicitly opts in. Never fall back to main silently.
|
||||
allow_main = bool(job.get("allow_main_chat")) or args.allow_main_chat
|
||||
if target == "main" and not allow_main:
|
||||
print(f"ERROR: Job '{job_name}' resolves to main chat; refusing by sidechat-first policy. "
|
||||
f"Set a sidechat target (dm_target/target/sidechat.create) in the job JSON, "
|
||||
f"set \"allow_main_chat\": true, or pass --allow-main-chat.", file=sys.stderr)
|
||||
log_event("job_failed", {
|
||||
"job_id": job_id,
|
||||
"error": "main_chat_blocked_by_policy",
|
||||
"pipeline_run_id": pipeline_run_id,
|
||||
})
|
||||
sys.exit(1)
|
||||
|
||||
# Log job_sent
|
||||
log_event("job_sent", {
|
||||
"job_id": job_id,
|
||||
"job_name": job_name,
|
||||
"agent": agent,
|
||||
"target": target,
|
||||
"dry_run": dry_run,
|
||||
"pipeline_run_id": pipeline_run_id,
|
||||
"step_n": step_n,
|
||||
})
|
||||
|
||||
# Work-first envelope: executable swarm.spawn/followup.create at TOP and BOTTOM
|
||||
# (see bin/prompt_envelope.py). Always applied, even if the template has its own [RESULT.
|
||||
import prompt_envelope
|
||||
rendered = prompt_envelope.wrap(job_name, job_id, agent, target, rendered)
|
||||
|
||||
# Format as JOB DM
|
||||
dm_message = f"[JOB {job_id}] {rendered}"
|
||||
|
||||
# Dispatch via dm.py (handles main or sidechat with auto-provisioning and verification)
|
||||
msg_id = send_dm(agent, target, dm_message,
|
||||
dry_run=dry_run, followup_tags=followup_tags,
|
||||
allow_main_chat=allow_main)
|
||||
|
||||
if msg_id and not dry_run:
|
||||
print(f"Dispatched job {job_id} to {agent}/{target} (DM: {msg_id})")
|
||||
log_event("job_dispatched", {
|
||||
"job_id": job_id,
|
||||
"dm_id": msg_id,
|
||||
"pipeline_run_id": pipeline_run_id,
|
||||
"step_n": step_n,
|
||||
})
|
||||
if pipeline_run_id and HAS_PIPELINE:
|
||||
pipeline_engine.record_step_dispatch(
|
||||
pipeline_run_id, step_n, job_name, job_id, agent, target, dm_id=msg_id
|
||||
)
|
||||
sc_state = load_sidechat_state()
|
||||
if target in sc_state:
|
||||
val = sc_state[target]
|
||||
t_uuid = val.get("thread_uuid") if isinstance(val, dict) else val
|
||||
if t_uuid:
|
||||
pipeline_engine.update_pipeline_thread(pipeline_run_id, t_uuid)
|
||||
elif dry_run:
|
||||
print(f"[DRY RUN] Job {job_id} would be dispatched to {agent}/{target}")
|
||||
else:
|
||||
print(f"Failed to dispatch job {job_id}", file=sys.stderr)
|
||||
log_event("job_failed", {
|
||||
"job_id": job_id,
|
||||
"error": "dm_send_failed",
|
||||
"pipeline_run_id": pipeline_run_id,
|
||||
})
|
||||
if pipeline_run_id and HAS_PIPELINE:
|
||||
pipeline_engine.fail_pipeline(pipeline_run_id, "dm_send_failed")
|
||||
sys.exit(1)
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -1,211 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""job-scheduler.py — NetVM unified job scheduler.
|
||||
|
||||
Evaluates cron schedules in jobs/*.json and triggers due jobs via job-dispatch.py.
|
||||
Prevents duplicate dispatches using state watermarks in /home/super/Projects/NetVM/job-scheduler-state.json.
|
||||
|
||||
Usage:
|
||||
python3 bin/job-scheduler.py run [--dry-run]
|
||||
python3 bin/job-scheduler.py status
|
||||
"""
|
||||
|
||||
import os
|
||||
import sys
|
||||
import glob
|
||||
import json
|
||||
import fcntl
|
||||
import argparse
|
||||
import subprocess
|
||||
from datetime import datetime, timezone, timedelta
|
||||
|
||||
BASE = "/home/super/Projects/NetVM"
|
||||
JOBS_DIR = os.path.join(BASE, "jobs")
|
||||
BIN = os.path.join(BASE, "bin")
|
||||
JOB_DISPATCH = os.path.join(BIN, "job-dispatch.py")
|
||||
STATE_FILE = os.path.join(BASE, "job-scheduler-state.json")
|
||||
LOCK_FILE = os.path.join(BASE, "job-scheduler.lock")
|
||||
|
||||
|
||||
def utcnow():
|
||||
return datetime.now(timezone.utc)
|
||||
|
||||
|
||||
def parse_field(pattern, val):
|
||||
if pattern == "*":
|
||||
return True
|
||||
for part in pattern.split(","):
|
||||
if "/" in part:
|
||||
sub = part.split("/")
|
||||
step = int(sub[1])
|
||||
base = sub[0]
|
||||
start = 0 if base == "*" else int(base.split("-")[0])
|
||||
end = 59 if base == "*" else int(base.split("-")[-1])
|
||||
if start <= val <= end and (val - start) % step == 0:
|
||||
return True
|
||||
elif "-" in part:
|
||||
s, e = map(int, part.split("-"))
|
||||
if s <= val <= e:
|
||||
return True
|
||||
elif part.isdigit() and int(part) == val:
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
def cron_matches(expr, dt):
|
||||
"""Check if 5-field cron expression matches datetime dt."""
|
||||
parts = expr.strip().split()
|
||||
if len(parts) != 5:
|
||||
return False
|
||||
m, h, dom, mon, dow = parts
|
||||
dow_val = (dt.weekday() + 1) % 7 # 0=Sunday
|
||||
return (parse_field(m, dt.minute) and
|
||||
parse_field(h, dt.hour) and
|
||||
parse_field(dom, dt.day) and
|
||||
parse_field(mon, dt.month) and
|
||||
(parse_field(dow, dt.weekday() + 1) or parse_field(dow, dow_val)))
|
||||
|
||||
|
||||
def is_job_due(expr, now_dt, last_fired_dt=None):
|
||||
"""Determine if a cron job is due within the recent 5-minute sampling window."""
|
||||
if not expr or expr.strip().lower() == "manual":
|
||||
return False
|
||||
|
||||
# Check minutes in the window [now - 4min, now]
|
||||
matched_dt = None
|
||||
for offset in range(5):
|
||||
sample_dt = now_dt - timedelta(minutes=offset)
|
||||
if cron_matches(expr, sample_dt):
|
||||
matched_dt = sample_dt.replace(second=0, microsecond=0)
|
||||
break
|
||||
|
||||
if not matched_dt:
|
||||
return False
|
||||
|
||||
if last_fired_dt:
|
||||
# If fired within 4 minutes of the matched slot, skip duplicate
|
||||
diff_seconds = (now_dt - last_fired_dt).total_seconds()
|
||||
# For hourly or longer jobs, prevent re-fire within 45 minutes
|
||||
if " " in expr and expr.split()[0] != "*":
|
||||
if diff_seconds < 2700:
|
||||
return False
|
||||
elif diff_seconds < 240:
|
||||
return False
|
||||
|
||||
return True
|
||||
|
||||
|
||||
def load_state():
|
||||
try:
|
||||
with open(STATE_FILE, "r") as f:
|
||||
return json.load(f)
|
||||
except Exception:
|
||||
return {"jobs": {}, "last_run": None}
|
||||
|
||||
|
||||
def save_state(state):
|
||||
tmp = STATE_FILE + ".tmp"
|
||||
with open(tmp, "w") as f:
|
||||
json.dump(state, f, indent=2)
|
||||
os.replace(tmp, STATE_FILE)
|
||||
|
||||
|
||||
def do_run(dry_run=False):
|
||||
now = utcnow()
|
||||
now_iso = now.strftime("%Y-%m-%dT%H:%M:%SZ")
|
||||
state = load_state()
|
||||
jobs_state = state.setdefault("jobs", {})
|
||||
|
||||
job_files = sorted(glob.glob(os.path.join(JOBS_DIR, "*.json")))
|
||||
dispatched = []
|
||||
skipped = []
|
||||
|
||||
for jpath in job_files:
|
||||
try:
|
||||
with open(jpath, "r", encoding="utf-8") as f:
|
||||
data = json.load(f)
|
||||
except Exception:
|
||||
continue
|
||||
|
||||
job_name = data.get("name") or os.path.basename(jpath).replace(".json", "")
|
||||
schedule = data.get("schedule")
|
||||
if not schedule or schedule.strip().lower() == "manual":
|
||||
continue
|
||||
|
||||
j_st = jobs_state.get(job_name, {})
|
||||
last_fired_str = j_st.get("last_fired")
|
||||
last_fired_dt = None
|
||||
if last_fired_str:
|
||||
try:
|
||||
last_fired_dt = datetime.fromisoformat(last_fired_str.replace("Z", "+00:00"))
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
if is_job_due(schedule, now, last_fired_dt):
|
||||
if dry_run:
|
||||
print(f"[DRY RUN] Due job: {job_name} ({schedule})")
|
||||
dispatched.append(job_name)
|
||||
continue
|
||||
|
||||
cmd = [sys.executable, JOB_DISPATCH, job_name]
|
||||
try:
|
||||
r = subprocess.run(cmd, capture_output=True, text=True, timeout=180)
|
||||
if r.returncode == 0:
|
||||
dispatched.append(job_name)
|
||||
jobs_state[job_name] = {
|
||||
"last_fired": now_iso,
|
||||
"schedule": schedule,
|
||||
"status": "dispatched"
|
||||
}
|
||||
print(f"Dispatched job: {job_name} ({schedule})", file=sys.stderr)
|
||||
else:
|
||||
err = (r.stderr or r.stdout).strip()[-200:]
|
||||
print(f"Failed to dispatch {job_name}: {err}", file=sys.stderr)
|
||||
except Exception as e:
|
||||
print(f"Exception dispatching {job_name}: {e}", file=sys.stderr)
|
||||
else:
|
||||
skipped.append(job_name)
|
||||
|
||||
if not dry_run:
|
||||
state["last_run"] = now_iso
|
||||
save_state(state)
|
||||
|
||||
result = {
|
||||
"ok": True,
|
||||
"dispatched": dispatched,
|
||||
"dispatched_count": len(dispatched),
|
||||
"evaluated_at": now_iso
|
||||
}
|
||||
return result
|
||||
|
||||
|
||||
def main():
|
||||
p = argparse.ArgumentParser(description="NetVM Unified Job Scheduler")
|
||||
sub = p.add_subparsers(dest="cmd")
|
||||
p_run = sub.add_parser("run", help="Evaluate schedules and dispatch due jobs")
|
||||
p_run.add_argument("--dry-run", action="store_true", help="Print due jobs without dispatching")
|
||||
sub.add_parser("status", help="Show scheduler state and last run")
|
||||
|
||||
args = p.parse_args()
|
||||
cmd = args.cmd or "run"
|
||||
|
||||
if cmd == "status":
|
||||
print(json.dumps(load_state(), indent=2))
|
||||
return 0
|
||||
|
||||
if cmd == "run":
|
||||
try:
|
||||
lockfh = open(LOCK_FILE, "w")
|
||||
fcntl.flock(lockfh, fcntl.LOCK_EX | fcntl.LOCK_NB)
|
||||
except (OSError, IOError):
|
||||
print(json.dumps({"ok": False, "skipped": "already running"}))
|
||||
return 0
|
||||
|
||||
res = do_run(dry_run=getattr(args, "dry_run", False))
|
||||
print(json.dumps(res))
|
||||
return 0
|
||||
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -1,94 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Thread keepalive timer entry point (runs on bl).
|
||||
|
||||
Reads keepalive-config.json, ticks every registered thread:
|
||||
ensure reachable -> ping if idle past threshold -> recreate if dead.
|
||||
|
||||
Designed to run from a systemd timer (e.g. every 5 minutes). Each tick is
|
||||
idempotent and cheap: threads that are alive and recently active are a
|
||||
single navigate+verify (no ping sent).
|
||||
|
||||
Usage:
|
||||
keepalive-timer.py [--config PATH] [--state PATH] [--dry-run] [--status]
|
||||
|
||||
--status print the registry with idle times and exit (no changes)
|
||||
|
||||
Exit codes: 0 = all ok, 1 = one or more threads failed, 2 = config error.
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
|
||||
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
|
||||
from keepalive import Keepalive
|
||||
|
||||
NETVM_ROOT = os.environ.get("NETVM_ROOT", "/home/super/Projects/NetVM")
|
||||
DEFAULT_CONFIG = os.path.join(NETVM_ROOT, "keepalive-config.json")
|
||||
DEFAULT_STATE = os.path.join(NETVM_ROOT, "keepalive-threads.json")
|
||||
|
||||
|
||||
def load_config(path):
|
||||
try:
|
||||
with open(path) as f:
|
||||
cfg = json.load(f)
|
||||
except FileNotFoundError:
|
||||
print(f"keepalive: config not found: {path}", file=sys.stderr)
|
||||
return None
|
||||
except Exception as e:
|
||||
print(f"keepalive: bad config {path}: {e}", file=sys.stderr)
|
||||
return None
|
||||
threads = cfg.get("threads", [])
|
||||
if not isinstance(threads, list):
|
||||
print("keepalive: config 'threads' must be a list", file=sys.stderr)
|
||||
return None
|
||||
return threads
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("--config", default=DEFAULT_CONFIG)
|
||||
ap.add_argument("--state", default=DEFAULT_STATE)
|
||||
ap.add_argument("--dry-run", action="store_true")
|
||||
ap.add_argument("--status", action="store_true")
|
||||
args = ap.parse_args()
|
||||
|
||||
ka = Keepalive(state_path=args.state, dry_run=args.dry_run)
|
||||
|
||||
if args.status:
|
||||
print(json.dumps(ka.status(), indent=2))
|
||||
return 0
|
||||
|
||||
threads = load_config(args.config)
|
||||
if threads is None:
|
||||
return 2
|
||||
|
||||
results = {}
|
||||
for t in threads:
|
||||
key = t.get("key")
|
||||
agent = t.get("agent", "opm")
|
||||
if not key or not t.get("enabled", True):
|
||||
continue
|
||||
try:
|
||||
status = ka.tick(
|
||||
key,
|
||||
agent,
|
||||
idle_threshold_s=int(t.get("idle_threshold_s", 3600)),
|
||||
max_retries=int(t.get("max_retries", 3)),
|
||||
ping_message=t.get("ping_message"),
|
||||
)
|
||||
except Exception as e:
|
||||
status = f"error: {e}"
|
||||
ka.log_event("keepalive_tick_error",
|
||||
{"key": key, "agent": agent, "error": str(e)[:200]})
|
||||
results[key] = status
|
||||
print(f"keepalive: {key}@{agent} -> {status}")
|
||||
|
||||
failed = [k for k, v in results.items()
|
||||
if v in ("failed",) or v.startswith("error")]
|
||||
return 1 if failed else 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -1,345 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Generic thread keepalive for bl side chats.
|
||||
|
||||
Pattern proven by the heartbeat job overnight: a state file maps a stable
|
||||
key -> thread UUID, and each tick navigates directly to
|
||||
https://muse.ai/thread/<uuid> (UUID reuse fix in muse-chat-api.py
|
||||
cmd_sidechat_use) instead of name-based sidebar lookup.
|
||||
|
||||
This module generalizes that pattern:
|
||||
|
||||
- ANY side chat can register for keepalive (not just job sidechats).
|
||||
- Each tick: ensure the thread is reachable; ping it if idle past threshold;
|
||||
recreate it if the stored UUID is dead.
|
||||
- State lives in keepalive-threads.json (atomic write via tmp+rename).
|
||||
|
||||
Usage:
|
||||
from keepalive import Keepalive
|
||||
|
||||
ka = Keepalive(state_path="/home/super/Projects/NetVM/keepalive-threads.json")
|
||||
ka.ensure("ops-watch", agent="opm") # reuse or create
|
||||
ka.tick("ops-watch", agent="opm",
|
||||
ping_message="[keepalive] ops-watch {ts}",
|
||||
idle_threshold_s=3600) # ping if idle
|
||||
|
||||
The timer entry point (keepalive-timer.py) drives this from a JSON config.
|
||||
"""
|
||||
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
from datetime import datetime, timezone
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Paths (overridable for tests)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
NETVM_ROOT = os.environ.get("NETVM_ROOT", "/home/super/Projects/NetVM")
|
||||
NETVM_EXEC = os.path.join(NETVM_ROOT, "bin", "netvm-exec.sh")
|
||||
CHAT_API = os.path.join(NETVM_ROOT, "bin", "muse-chat-api.py")
|
||||
DEFAULT_STATE = os.path.join(NETVM_ROOT, "keepalive-threads.json")
|
||||
DEFAULT_LOG = os.path.join(NETVM_ROOT, "keepalive-log.jsonl")
|
||||
|
||||
UUID_RE = re.compile(
|
||||
r"/thread/([0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-"
|
||||
r"[0-9a-f]{4}-[0-9a-f]{12})"
|
||||
)
|
||||
|
||||
|
||||
def _utcnow():
|
||||
return datetime.now(timezone.utc).isoformat()
|
||||
|
||||
|
||||
def _ts():
|
||||
return time.time()
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Registry
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
class Keepalive:
|
||||
"""Thread keepalive registry + ensure/ping/check operations."""
|
||||
|
||||
def __init__(self, state_path=DEFAULT_STATE, log_path=DEFAULT_LOG,
|
||||
dry_run=False):
|
||||
self.state_path = state_path
|
||||
self.log_path = log_path
|
||||
self.dry_run = dry_run
|
||||
|
||||
# -- state ---------------------------------------------------------
|
||||
|
||||
def load_state(self):
|
||||
if os.path.exists(self.state_path):
|
||||
try:
|
||||
with open(self.state_path) as f:
|
||||
data = json.load(f)
|
||||
return data if isinstance(data, dict) else {}
|
||||
except Exception:
|
||||
return {}
|
||||
return {}
|
||||
|
||||
def save_state(self, state):
|
||||
if self.dry_run:
|
||||
return
|
||||
tmp = self.state_path + ".tmp"
|
||||
with open(tmp, "w") as f:
|
||||
json.dump(state, f, indent=2)
|
||||
os.replace(tmp, self.state_path)
|
||||
|
||||
def log_event(self, event_type, data):
|
||||
if self.dry_run:
|
||||
return
|
||||
entry = {"ts": _utcnow(), "type": event_type, **data}
|
||||
try:
|
||||
with open(self.log_path, "a") as f:
|
||||
f.write(json.dumps(entry) + "\n")
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
# -- browser ops (mirrors job-dispatch.py; UUID navigation, not names) --
|
||||
|
||||
def _run(self, agent, *api_args, timeout=60):
|
||||
"""Run muse-chat-api.py in the agent's netns via netvm-exec.sh."""
|
||||
cmd = [NETVM_EXEC, agent, "--", "python3", CHAT_API,
|
||||
"--account", agent] + list(api_args)
|
||||
try:
|
||||
r = subprocess.run(cmd, capture_output=True, text=True,
|
||||
timeout=timeout)
|
||||
return r.returncode, r.stdout.strip(), r.stderr.strip()
|
||||
except Exception as e:
|
||||
return -1, "", str(e)
|
||||
|
||||
def navigate_to_uuid(self, agent, thread_uuid):
|
||||
"""Go directly to https://muse.ai/thread/<uuid>.
|
||||
|
||||
This is the heartbeat UUID-reuse fix: cmd_sidechat_use navigates by
|
||||
URL for UUID args instead of searching sidebar titles by name.
|
||||
Returns True if the command succeeded.
|
||||
"""
|
||||
if self.dry_run:
|
||||
print(f"[DRY] navigate {agent} -> {thread_uuid}")
|
||||
return True
|
||||
rc, out, err = self._run(agent, "sidechat", "use", thread_uuid,
|
||||
timeout=45)
|
||||
return rc == 0
|
||||
|
||||
def current_thread_uuid(self, agent):
|
||||
"""Read the browser's current URL and extract the thread UUID."""
|
||||
if self.dry_run:
|
||||
return None
|
||||
rc, out, err = self._run(agent, "url", timeout=30)
|
||||
if rc != 0:
|
||||
return None
|
||||
m = UUID_RE.search(out or "")
|
||||
return m.group(1) if m else None
|
||||
|
||||
def create_sidechat(self, agent, name=None):
|
||||
"""Create a new sidechat in the agent's account. Returns True."""
|
||||
if self.dry_run:
|
||||
print(f"[DRY] create sidechat for {agent}")
|
||||
return True
|
||||
rc, out, err = self._run(agent, "sidechat", "create", timeout=90)
|
||||
return rc == 0 and "Created:" in (out or "")
|
||||
|
||||
def send_message(self, agent, message):
|
||||
"""Send a message to the current chat. Returns True on success."""
|
||||
if self.dry_run:
|
||||
print(f"[DRY] send to {agent}: {message[:80]}")
|
||||
return True
|
||||
rc, out, err = self._run(agent, "send", message, timeout=60)
|
||||
return rc == 0
|
||||
|
||||
# -- keepalive ops ---------------------------------------------------
|
||||
|
||||
def check(self, key, agent):
|
||||
"""Verify the registered thread is reachable.
|
||||
|
||||
Navigates to the stored UUID and confirms the browser lands on it.
|
||||
Returns (ok, thread_uuid). Updates last_verified_ts on success.
|
||||
"""
|
||||
state = self.load_state()
|
||||
rec = state.get(key)
|
||||
if not rec or not rec.get("thread_uuid"):
|
||||
return False, None
|
||||
uuid = rec["thread_uuid"]
|
||||
if not self.navigate_to_uuid(agent, uuid):
|
||||
self.log_event("keepalive_check_failed",
|
||||
{"key": key, "agent": agent, "thread_uuid": uuid,
|
||||
"reason": "navigate_failed"})
|
||||
return False, uuid
|
||||
cur = self.current_thread_uuid(agent)
|
||||
if cur == uuid:
|
||||
rec["last_verified_ts"] = _ts()
|
||||
rec["consecutive_failures"] = 0
|
||||
state[key] = rec
|
||||
self.save_state(state)
|
||||
self.log_event("keepalive_check_ok",
|
||||
{"key": key, "agent": agent, "thread_uuid": uuid})
|
||||
return True, uuid
|
||||
self.log_event("keepalive_check_failed",
|
||||
{"key": key, "agent": agent, "thread_uuid": uuid,
|
||||
"reason": "url_mismatch", "current": cur})
|
||||
return False, uuid
|
||||
|
||||
def ensure(self, key, agent, name=None):
|
||||
"""Ensure the thread exists and is reachable; create if needed.
|
||||
|
||||
Returns (ok, thread_uuid). Mirrors the job-dispatch.py reuse-or-
|
||||
create flow: try stored UUID first, fall back to creation, then
|
||||
capture the new UUID from the browser URL.
|
||||
"""
|
||||
state = self.load_state()
|
||||
rec = state.get(key)
|
||||
if rec and rec.get("thread_uuid"):
|
||||
ok, uuid = self.check(key, agent)
|
||||
if ok:
|
||||
return True, uuid
|
||||
# Stored thread is dead — fall through to recreate.
|
||||
print(f"keepalive: stored thread for {key} unreachable, "
|
||||
f"recreating", file=sys.stderr)
|
||||
# Create a fresh sidechat.
|
||||
if not self.create_sidechat(agent, name=name):
|
||||
self.log_event("keepalive_create_failed",
|
||||
{"key": key, "agent": agent})
|
||||
return False, None
|
||||
# Capture the new thread UUID from the browser URL.
|
||||
uuid = None
|
||||
if not self.dry_run:
|
||||
for _ in range(15):
|
||||
time.sleep(1)
|
||||
uuid = self.current_thread_uuid(agent)
|
||||
if uuid:
|
||||
break
|
||||
else:
|
||||
uuid = "dry-run-uuid"
|
||||
if not uuid:
|
||||
self.log_event("keepalive_create_failed",
|
||||
{"key": key, "agent": agent,
|
||||
"reason": "uuid_capture_failed"})
|
||||
return False, None
|
||||
now = _ts()
|
||||
state[key] = {
|
||||
"thread_uuid": uuid,
|
||||
"agent": agent,
|
||||
"created_ts": now,
|
||||
"last_ping_ts": 0,
|
||||
"last_verified_ts": now,
|
||||
"last_activity_ts": now,
|
||||
"ping_count": 0,
|
||||
"consecutive_failures": 0,
|
||||
}
|
||||
self.save_state(state)
|
||||
self.log_event("keepalive_created",
|
||||
{"key": key, "agent": agent, "thread_uuid": uuid})
|
||||
return True, uuid
|
||||
|
||||
def ping(self, key, agent, message=None):
|
||||
"""Send a keepalive ping into the thread.
|
||||
|
||||
Navigates to the thread first (cheap no-op if already there),
|
||||
then sends the ping message. Updates last_ping_ts / ping_count.
|
||||
"""
|
||||
state = self.load_state()
|
||||
rec = state.get(key)
|
||||
if not rec or not rec.get("thread_uuid"):
|
||||
return False
|
||||
uuid = rec["thread_uuid"]
|
||||
if not self.navigate_to_uuid(agent, uuid):
|
||||
return False
|
||||
msg = (message or "[keepalive:{key}] tick {ts} {uuid}").format(
|
||||
key=key, ts=_utcnow(), uuid=uuid)
|
||||
if not self.send_message(agent, msg):
|
||||
self.log_event("keepalive_ping_failed",
|
||||
{"key": key, "agent": agent, "thread_uuid": uuid})
|
||||
return False
|
||||
now = _ts()
|
||||
rec["last_ping_ts"] = now
|
||||
rec["last_activity_ts"] = now
|
||||
rec["ping_count"] = rec.get("ping_count", 0) + 1
|
||||
state[key] = rec
|
||||
self.save_state(state)
|
||||
self.log_event("keepalive_ping",
|
||||
{"key": key, "agent": agent, "thread_uuid": uuid,
|
||||
"ping_count": rec["ping_count"]})
|
||||
return True
|
||||
|
||||
def note_activity(self, key):
|
||||
"""Record external activity (e.g. a job just sent to the thread)
|
||||
so the idle timer doesn't ping unnecessarily."""
|
||||
state = self.load_state()
|
||||
rec = state.get(key)
|
||||
if rec:
|
||||
rec["last_activity_ts"] = _ts()
|
||||
state[key] = rec
|
||||
self.save_state(state)
|
||||
|
||||
def tick(self, key, agent, idle_threshold_s=3600, max_retries=3,
|
||||
ping_message=None):
|
||||
"""One keepalive tick for a registered thread.
|
||||
|
||||
- Ensures the thread exists (recreate if dead).
|
||||
- Pings only if idle longer than idle_threshold_s.
|
||||
- After max_retries consecutive failures, forces recreation.
|
||||
Returns a status string: ok | pinged | recreated | failed.
|
||||
"""
|
||||
state = self.load_state()
|
||||
rec = state.get(key)
|
||||
|
||||
if not rec or not rec.get("thread_uuid"):
|
||||
ok, _ = self.ensure(key, agent)
|
||||
return "recreated" if ok else "failed"
|
||||
|
||||
failures = rec.get("consecutive_failures", 0)
|
||||
if failures >= max_retries:
|
||||
# Force recreation: drop the dead UUID and re-ensure.
|
||||
self.log_event("keepalive_force_recreate",
|
||||
{"key": key, "agent": agent,
|
||||
"failures": failures,
|
||||
"old_uuid": rec.get("thread_uuid")})
|
||||
rec["thread_uuid"] = None
|
||||
state[key] = rec
|
||||
self.save_state(state)
|
||||
ok, _ = self.ensure(key, agent)
|
||||
return "recreated" if ok else "failed"
|
||||
|
||||
ok, _ = self.check(key, agent)
|
||||
if not ok:
|
||||
state = self.load_state()
|
||||
rec = state.get(key, {})
|
||||
rec["consecutive_failures"] = failures + 1
|
||||
state[key] = rec
|
||||
self.save_state(state)
|
||||
return "failed"
|
||||
|
||||
now = _ts()
|
||||
idle_for = now - rec.get("last_activity_ts", 0)
|
||||
if idle_for >= idle_threshold_s:
|
||||
if self.ping(key, agent, message=ping_message):
|
||||
return "pinged"
|
||||
return "failed"
|
||||
return "ok"
|
||||
|
||||
def unregister(self, key):
|
||||
"""Remove a thread from keepalive (does not delete the sidechat)."""
|
||||
state = self.load_state()
|
||||
if key in state:
|
||||
del state[key]
|
||||
self.save_state(state)
|
||||
self.log_event("keepalive_unregistered", {"key": key})
|
||||
return True
|
||||
return False
|
||||
|
||||
def status(self):
|
||||
"""Return the full registry with computed idle times."""
|
||||
state = self.load_state()
|
||||
now = _ts()
|
||||
out = {}
|
||||
for key, rec in state.items():
|
||||
r = dict(rec)
|
||||
r["idle_s"] = int(now - rec.get("last_activity_ts", now))
|
||||
out[key] = r
|
||||
return out
|
||||
@@ -1,194 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""main-chat-watchdog.py — enforce the sidechat-first DM policy.
|
||||
|
||||
Tails dm-log.jsonl (read-only) and flags any DM actually SENT to main chat
|
||||
without an explicit allow_main_chat marker, plus any placement_mismatch /
|
||||
placement_failed events (message landed in the wrong chat). Distinguishes:
|
||||
|
||||
VIOLATION : type=sent, target=main, no tags.allow_main_chat -> policy broken (P0)
|
||||
AUTHORIZED: type=sent, target=main, tags.allow_main_chat=true -> explicit opt-in
|
||||
BLOCKED : type=main_chat_blocked -> policy working
|
||||
MISMATCH : type=placement_mismatch -> placed in wrong chat (P0)
|
||||
PFAILED : type=placement_failed -> placement hard-failed (P0)
|
||||
|
||||
State: byte-offset watermark in main-chat-watchdog.state (handles log
|
||||
rotation via inode check). First run starts at EOF (no historical backfill —
|
||||
old entries predate the allow_main_chat audit marker).
|
||||
|
||||
Violations are appended to logs/main-chat-violations.jsonl and printed to
|
||||
stdout (journal). Exit 0 = clean, 1 = violations/P0s found, 2 = error.
|
||||
|
||||
Read-only: never modifies dm-log.jsonl.
|
||||
"""
|
||||
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
from datetime import datetime, timezone
|
||||
|
||||
BASE = "/home/super/Projects/NetVM"
|
||||
DM_LOG = os.path.join(BASE, "dm-log.jsonl")
|
||||
STATE_FILE = os.path.join(BASE, "main-chat-watchdog.state")
|
||||
VIOLATIONS_LOG = os.path.join(BASE, "logs", "main-chat-violations.jsonl")
|
||||
|
||||
|
||||
def utcnow():
|
||||
return datetime.now(timezone.utc).isoformat()
|
||||
|
||||
|
||||
def load_state():
|
||||
try:
|
||||
with open(STATE_FILE) as f:
|
||||
return json.load(f)
|
||||
except (FileNotFoundError, json.JSONDecodeError, ValueError):
|
||||
return {}
|
||||
|
||||
|
||||
def save_state(state):
|
||||
tmp = STATE_FILE + ".tmp"
|
||||
with open(tmp, "w") as f:
|
||||
json.dump(state, f)
|
||||
os.replace(tmp, STATE_FILE)
|
||||
|
||||
|
||||
def main():
|
||||
try:
|
||||
st = os.stat(DM_LOG)
|
||||
except FileNotFoundError:
|
||||
print(f"watchdog ERROR: {DM_LOG} not found", file=sys.stderr)
|
||||
return 2
|
||||
|
||||
state = load_state()
|
||||
# First run (or rotation): start at EOF, don't backfill history that
|
||||
# predates the allow_main_chat audit marker.
|
||||
if state.get("inode") != st.st_ino:
|
||||
offset = st.st_size
|
||||
if state:
|
||||
print(f"watchdog: log rotated or first run (inode {st.st_ino}), "
|
||||
f"starting at EOF offset {offset}")
|
||||
else:
|
||||
offset = min(state.get("offset", 0), st.st_size)
|
||||
|
||||
violations = [] # P0: gate bypass (sent to main, no opt-in)
|
||||
mismatches = [] # P0: placement_mismatch / placement_failed
|
||||
blocked = 0
|
||||
authorized = 0
|
||||
scanned = 0
|
||||
malformed = 0
|
||||
|
||||
try:
|
||||
with open(DM_LOG, "r", encoding="utf-8", errors="replace") as f:
|
||||
f.seek(offset)
|
||||
for line in f:
|
||||
line = line.strip()
|
||||
if not line:
|
||||
continue
|
||||
scanned += 1
|
||||
try:
|
||||
ev = json.loads(line)
|
||||
except json.JSONDecodeError:
|
||||
malformed += 1
|
||||
continue
|
||||
etype = ev.get("type")
|
||||
if etype == "main_chat_blocked":
|
||||
# Policy working: the dm.py gate refused a main send.
|
||||
blocked += 1
|
||||
elif etype == "placement_mismatch":
|
||||
# P0: verify-time URL check found the browser parked in a
|
||||
# different chat than the intended target. The send
|
||||
# completed but landed in the wrong place. Schema has two
|
||||
# variants: sidechat ("expected_uuid") and main-drift
|
||||
# ("expected": "main").
|
||||
mismatches.append({
|
||||
"ts": utcnow(),
|
||||
"kind": "placement_mismatch",
|
||||
"severity": "P0",
|
||||
"dm_id": ev.get("id"),
|
||||
"agent": ev.get("agent"),
|
||||
"to": ev.get("to"),
|
||||
"target": ev.get("target"),
|
||||
"expected_uuid": ev.get("expected_uuid") or ev.get("expected"),
|
||||
"actual_uuid": ev.get("actual_uuid"),
|
||||
"attempt": ev.get("attempt"),
|
||||
"event_ts": ev.get("ts"),
|
||||
})
|
||||
elif etype == "placement_failed":
|
||||
# P0: the authoritative post-send placement check failed
|
||||
# hard (sender exited 1, send NOT marked verified).
|
||||
d = ev.get("detail") or {}
|
||||
mismatches.append({
|
||||
"ts": utcnow(),
|
||||
"kind": "placement_failed",
|
||||
"severity": "P0",
|
||||
"dm_id": ev.get("id"),
|
||||
"agent": ev.get("agent"),
|
||||
"to": ev.get("to"),
|
||||
"target": ev.get("target"),
|
||||
"reason": ev.get("reason") or d.get("reason"),
|
||||
"expected_uuid": ev.get("expected_uuid") or d.get("expected_uuid"),
|
||||
"actual_url": ev.get("actual_url") or d.get("actual_url"),
|
||||
"loop_attempt": ev.get("loop_attempt"),
|
||||
"event_ts": ev.get("ts"),
|
||||
})
|
||||
elif etype == "sent" and ev.get("target") == "main":
|
||||
tags = ev.get("tags") or {}
|
||||
if tags.get("allow_main_chat"):
|
||||
authorized += 1
|
||||
else:
|
||||
violations.append({
|
||||
"ts": utcnow(),
|
||||
"kind": "main_chat_send",
|
||||
"severity": "P0",
|
||||
"dm_id": ev.get("id"),
|
||||
"agent": ev.get("agent"),
|
||||
"to": ev.get("to"),
|
||||
"target": "main",
|
||||
"msg_preview": (ev.get("msg") or "")[:120],
|
||||
"event_ts": ev.get("ts"),
|
||||
})
|
||||
new_offset = f.tell()
|
||||
except OSError as e:
|
||||
print(f"watchdog ERROR reading log: {e}", file=sys.stderr)
|
||||
return 2
|
||||
|
||||
save_state({"offset": new_offset, "inode": st.st_ino})
|
||||
|
||||
findings = violations + mismatches
|
||||
summary = (f"watchdog: scanned={scanned} blocked={blocked} "
|
||||
f"authorized_main={authorized} "
|
||||
f"violations={len(violations)} "
|
||||
f"placement_mismatches={len(mismatches)} "
|
||||
f"malformed={malformed}")
|
||||
print(summary)
|
||||
|
||||
if findings:
|
||||
try:
|
||||
os.makedirs(os.path.dirname(VIOLATIONS_LOG), exist_ok=True)
|
||||
with open(VIOLATIONS_LOG, "a", encoding="utf-8") as vf:
|
||||
for v in findings:
|
||||
vf.write(json.dumps(v) + "\n")
|
||||
except OSError as e:
|
||||
print(f"watchdog ERROR writing violations log: {e}", file=sys.stderr)
|
||||
return 2
|
||||
for v in violations:
|
||||
print(f"VIOLATION main-chat send without opt-in: "
|
||||
f"id={v['dm_id']} agent={v['agent']} to={v['to']} "
|
||||
f"at={v['event_ts']} preview={v['msg_preview']!r}")
|
||||
for m in mismatches:
|
||||
if m["kind"] == "placement_mismatch":
|
||||
print(f"P0 PLACEMENT_MISMATCH message landed in wrong chat: "
|
||||
f"id={m['dm_id']} agent={m['agent']} to={m['to']} "
|
||||
f"target={m['target']} expected={m['expected_uuid']} "
|
||||
f"actual={m['actual_uuid']} at={m['event_ts']}")
|
||||
else:
|
||||
print(f"P0 PLACEMENT_FAILED placement check failed: "
|
||||
f"id={m['dm_id']} agent={m['agent']} to={m['to']} "
|
||||
f"target={m['target']} reason={m['reason']} "
|
||||
f"expected={m['expected_uuid']} "
|
||||
f"actual_url={m['actual_url']} at={m['event_ts']}")
|
||||
return 1
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -1,205 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Meta Accounts Center change-detection harness.
|
||||
|
||||
Captures a structural snapshot of the accountscenter.meta.com auth flow
|
||||
via CDP inside a NetVM netns, diffs against the stored baseline.
|
||||
|
||||
Outcomes:
|
||||
PASS - matches baseline (or first run establishes it)
|
||||
CHANGED - structural diff detected; needs human review, baseline untouched
|
||||
FAIL - automation itself broke (browser/CDP/network error)
|
||||
|
||||
Usage:
|
||||
meta-ac-snapshot.py [--node NAME] [--promote] [--snapshot-dir DIR]
|
||||
|
||||
--node NetVM node to run in (default: phone)
|
||||
--promote after human review, promote the latest snapshot to baseline
|
||||
--snapshot-dir where snapshots live (default: ~/Projects/NetVM/snapshots/meta-ac)
|
||||
"""
|
||||
import argparse, base64, datetime, json, os, subprocess, sys, time
|
||||
import urllib.parse, urllib.request
|
||||
|
||||
CDP_PORT = 19744
|
||||
|
||||
def log(*a):
|
||||
print(*a, flush=True)
|
||||
|
||||
def ns_exec(node, cmd):
|
||||
return subprocess.run(
|
||||
["sudo", "-n", "ip", "netns", "exec", f"warp-{node}"] + cmd,
|
||||
capture_output=True, text=True)
|
||||
|
||||
def norm_url(u):
|
||||
"""Strip query/fragment — nonces change every visit."""
|
||||
p = urllib.parse.urlparse(u)
|
||||
return f"{p.scheme}://{p.host}{p.path}" if hasattr(p, 'host') else f"{p.scheme}://{p.hostname}{p.path}"
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("--node", default="phone")
|
||||
ap.add_argument("--promote", action="store_true")
|
||||
ap.add_argument("--snapshot-dir", default=os.path.expanduser(
|
||||
"~/Projects/NetVM/snapshots/meta-ac"))
|
||||
args = ap.parse_args()
|
||||
os.makedirs(args.snapshot_dir, exist_ok=True)
|
||||
baseline_path = os.path.join(args.snapshot_dir, "baseline.json")
|
||||
|
||||
if args.promote:
|
||||
snaps = sorted(f for f in os.listdir(args.snapshot_dir)
|
||||
if f.startswith("snap-") and f.endswith(".json"))
|
||||
if not snaps:
|
||||
log("no snapshots to promote"); return 2
|
||||
latest = os.path.join(args.snapshot_dir, snaps[-1])
|
||||
data = json.load(open(latest))
|
||||
data["promoted_at"] = datetime.datetime.now(datetime.timezone.utc).isoformat()
|
||||
json.dump(data, open(baseline_path, "w"), indent=2)
|
||||
log(f"promoted {snaps[-1]} -> baseline.json")
|
||||
return 0
|
||||
|
||||
profile_dir = "/tmp/meta-ac-snap-profile"
|
||||
subprocess.run(["rm", "-rf", profile_dir])
|
||||
os.makedirs(profile_dir, exist_ok=True)
|
||||
|
||||
def http(path):
|
||||
with urllib.request.urlopen(
|
||||
f"http://127.0.0.1:{CDP_PORT}{path}", timeout=5) as r:
|
||||
return json.loads(r.read())
|
||||
|
||||
# launch chromium inside the netns via a wrapper script
|
||||
wrapper = "/tmp/meta-ac-snap-run.py"
|
||||
open(wrapper, "w").write(WRAPPER_SRC)
|
||||
log(f"launching chromium in warp-{args.node} (CDP {CDP_PORT})...")
|
||||
proc = subprocess.Popen(
|
||||
["sudo", "-n", "ip", "netns", "exec", f"warp-{args.node}",
|
||||
"python3", wrapper, str(CDP_PORT), profile_dir],
|
||||
stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True)
|
||||
try:
|
||||
out, _ = proc.communicate(timeout=120)
|
||||
except subprocess.TimeoutExpired:
|
||||
proc.kill(); log("FAIL: harness timed out"); return 1
|
||||
print(out)
|
||||
# wrapper prints SNAPSHOT_JSON=<json> on success
|
||||
snap = None
|
||||
for line in out.splitlines():
|
||||
if line.startswith("SNAPSHOT_JSON="):
|
||||
snap = json.loads(line[len("SNAPSHOT_JSON="):])
|
||||
if not snap:
|
||||
log("FAIL: no snapshot captured"); return 1
|
||||
|
||||
snap["node"] = args.node
|
||||
snap["captured_at"] = datetime.datetime.now(datetime.timezone.utc).isoformat()
|
||||
|
||||
# egress ip for context
|
||||
try:
|
||||
r = ns_exec(args.node, ["curl", "-s", "--max-time", "8",
|
||||
"https://api.ipify.org"])
|
||||
snap["egress_ip"] = r.stdout.strip()
|
||||
except Exception:
|
||||
snap["egress_ip"] = "unknown"
|
||||
|
||||
ts = datetime.datetime.now(datetime.timezone.utc).strftime("%Y%m%d-%H%M%S")
|
||||
snap_path = os.path.join(args.snapshot_dir, f"snap-{ts}.json")
|
||||
json.dump(snap, open(snap_path, "w"), indent=2)
|
||||
log(f"snapshot saved: {snap_path}")
|
||||
|
||||
if not os.path.exists(baseline_path):
|
||||
json.dump(snap, open(baseline_path, "w"), indent=2)
|
||||
log("PASS: baseline established (first run)")
|
||||
return 0
|
||||
|
||||
baseline = json.load(open(baseline_path))
|
||||
diffs = diff_snapshots(baseline, snap)
|
||||
if not diffs:
|
||||
log("PASS: matches baseline")
|
||||
return 0
|
||||
log("CHANGED: structural diff detected (baseline untouched):")
|
||||
for d in diffs:
|
||||
log(f" - {d}")
|
||||
log("review with: diff baseline.json snap-<ts>.json")
|
||||
log("promote after review with: --promote")
|
||||
return 3
|
||||
|
||||
def diff_snapshots(base, snap):
|
||||
diffs = []
|
||||
b_chain = [norm_url(u) for u in base.get("redirect_chain", [])]
|
||||
s_chain = [norm_url(u) for u in snap.get("redirect_chain", [])]
|
||||
if b_chain != s_chain:
|
||||
diffs.append(f"redirect_chain changed: {b_chain} -> {s_chain}")
|
||||
for key in ("forms", "inputs", "buttons"):
|
||||
b = sorted(base.get("dom_markers", {}).get(key, []))
|
||||
s = sorted(snap.get("dom_markers", {}).get(key, []))
|
||||
if b != s:
|
||||
added = [x for x in s if x not in b]
|
||||
removed = [x for x in b if x not in s]
|
||||
diffs.append(f"dom_markers.{key}: added={added} removed={removed}")
|
||||
if base.get("final_title") != snap.get("final_title"):
|
||||
diffs.append(f"final_title: {base.get('final_title')!r} -> {snap.get('final_title')!r}")
|
||||
return diffs
|
||||
|
||||
WRAPPER_SRC = '''
|
||||
import json, subprocess, sys, time, os, urllib.request, base64
|
||||
CDP_PORT = int(sys.argv[1])
|
||||
PROFILE_DIR = sys.argv[2]
|
||||
def http(path):
|
||||
with urllib.request.urlopen(f"http://127.0.0.1:{CDP_PORT}{path}", timeout=5) as r:
|
||||
return json.loads(r.read())
|
||||
logf = open("/tmp/meta-ac-snap-chrome.log", "w")
|
||||
proc = subprocess.Popen(["chromium", "--headless=new", "--disable-gpu", "--no-sandbox",
|
||||
"--disable-dev-shm-usage", f"--user-data-dir={PROFILE_DIR}",
|
||||
f"--remote-debugging-port={CDP_PORT}", "--remote-allow-origins=*", "about:blank"],
|
||||
stdout=logf, stderr=subprocess.STDOUT)
|
||||
try:
|
||||
for i in range(30):
|
||||
try:
|
||||
ver = http("/json/version")
|
||||
if "webSocketDebuggerUrl" in ver: break
|
||||
except Exception: pass
|
||||
time.sleep(1)
|
||||
else:
|
||||
print("FAIL: CDP never came up"); sys.exit(1)
|
||||
import websocket
|
||||
bws = websocket.create_connection(ver["webSocketDebuggerUrl"], timeout=20)
|
||||
bws.send(json.dumps({"id": 1, "method": "Target.createTarget",
|
||||
"params": {"url": "https://accountscenter.meta.com"}}))
|
||||
target_id = json.loads(bws.recv())["result"]["targetId"]
|
||||
bws.close()
|
||||
# redirect chain: seed with the navigation target (we always start
|
||||
# there), then poll for where Meta sends us. Seeding fixes the race
|
||||
# where a fast redirect is missed by the poll interval.
|
||||
START_URL = "https://accountscenter.meta.com/"
|
||||
chain, seen = [START_URL], {START_URL}
|
||||
for _ in range(24):
|
||||
time.sleep(2)
|
||||
for t in http("/json/list"):
|
||||
if t.get("id") == target_id or "meta.com" in t.get("url", ""):
|
||||
u = t["url"]
|
||||
if u not in seen:
|
||||
seen.add(u); chain.append(u)
|
||||
title = t.get("title", "")
|
||||
break
|
||||
# dom markers from the final tab
|
||||
tab_ws = None
|
||||
for t in http("/json/list"):
|
||||
if t.get("id") == target_id or "meta.com" in t.get("url", ""):
|
||||
tab_ws = t["webSocketDebuggerUrl"]; final_url = t["url"]; break
|
||||
ws = websocket.create_connection(tab_ws, timeout=20)
|
||||
js = """JSON.stringify({
|
||||
forms: [...document.forms].map(f => f.id || f.name || '(anon)'),
|
||||
inputs: [...document.querySelectorAll('input')].map(i => i.name || i.type || '(anon)'),
|
||||
buttons: [...document.querySelectorAll('button, [role=button]')].map(b => (b.innerText||'').trim()).filter(Boolean)
|
||||
})"""
|
||||
ws.send(json.dumps({"id": 1, "method": "Runtime.evaluate",
|
||||
"params": {"expression": js, "returnByValue": True}}))
|
||||
markers = json.loads(json.loads(ws.recv())["result"]["result"]["value"])
|
||||
# dedupe buttons, keep order
|
||||
markers["buttons"] = list(dict.fromkeys(markers["buttons"]))
|
||||
ws.close()
|
||||
snap = {"redirect_chain": chain, "final_url": final_url,
|
||||
"final_title": title, "dom_markers": markers}
|
||||
print("SNAPSHOT_JSON=" + json.dumps(snap))
|
||||
finally:
|
||||
proc.terminate()
|
||||
'''
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -1,218 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Meta Accounts Center read API (via authenticated browser session).
|
||||
|
||||
Usage:
|
||||
meta-acct.py list-linked <agent> # linked profiles in this Meta Account
|
||||
meta-acct.py security-status <agent> # login & recovery / security checkup summary
|
||||
meta-acct.py login-activity <agent> # "Where you're logged in" sessions
|
||||
|
||||
The agent's browser must have an active Meta session (phone/email OTP login).
|
||||
Reads NODES.md for the CDP port. Outputs JSON.
|
||||
|
||||
Scope: Accounts Center linkage/security surface only. No writes.
|
||||
"""
|
||||
import json, sys, time, urllib.request, os
|
||||
|
||||
NETVM = os.environ.get("NETVM_DIR")
|
||||
if not NETVM:
|
||||
script_dir = os.path.dirname(os.path.abspath(__file__))
|
||||
parent = os.path.dirname(script_dir)
|
||||
if os.path.exists(os.path.join(parent, "NODES.md")):
|
||||
NETVM = parent
|
||||
else:
|
||||
NETVM = "/home/super/Projects/NetVM"
|
||||
|
||||
AC_BASE = "https://accountscenter.meta.com"
|
||||
|
||||
def cdp_port(agent):
|
||||
for line in open(os.path.join(NETVM, "NODES.md")):
|
||||
line = line.strip()
|
||||
if not line.startswith("|"):
|
||||
continue
|
||||
cells = [c.strip() for c in line.strip("|").split("|")]
|
||||
if len(cells) >= 4 and cells[0] == agent:
|
||||
return int(cells[3])
|
||||
raise SystemExit(f"meta-acct: unknown agent '{agent}' (not in NODES.md)")
|
||||
|
||||
def cdp_get(port, path, method="GET", timeout=10):
|
||||
req = urllib.request.Request(f"http://127.0.0.1:{port}{path}", method=method)
|
||||
with urllib.request.urlopen(req, timeout=timeout) as r:
|
||||
return json.loads(r.read())
|
||||
|
||||
def get_ac_tab(port):
|
||||
tabs = cdp_get(port, "/json/list")
|
||||
for t in tabs:
|
||||
if "accountscenter.meta.com" in t.get("url", "") and t.get("type") == "page":
|
||||
return t
|
||||
# create one
|
||||
return cdp_get(port, f"/json/new?{AC_BASE}/", method="PUT")
|
||||
|
||||
def evaluate(port, ws_url, js, timeout=15):
|
||||
import websocket
|
||||
ws = websocket.create_connection(ws_url, timeout=timeout)
|
||||
try:
|
||||
ws.send(json.dumps({"id": 1, "method": "Page.navigate",
|
||||
"params": {"url": js[0]}}))
|
||||
ws.recv()
|
||||
time.sleep(4)
|
||||
ws.send(json.dumps({"id": 2, "method": "Runtime.evaluate",
|
||||
"params": {"expression": js[1], "returnByValue": True}}))
|
||||
resp = json.loads(ws.recv())
|
||||
return json.loads(resp["result"]["result"]["value"])
|
||||
finally:
|
||||
ws.close()
|
||||
|
||||
def cmd_list_linked(port):
|
||||
tab = get_ac_tab(port)
|
||||
data = evaluate(port, tab["webSocketDebuggerUrl"], (
|
||||
f"{AC_BASE}/account_overview/",
|
||||
"""JSON.stringify((() => {
|
||||
const h2 = [...document.querySelectorAll('h2')]
|
||||
.find(e=>e.innerText.trim()==='Profiles');
|
||||
// Full section is 3 levels up (H2 > DIV > DIV > MAIN)
|
||||
const container = h2?.parentElement?.parentElement?.parentElement;
|
||||
return {
|
||||
email: (document.body.innerText.match(
|
||||
/[\\w.+-]+@[\\w-]+\\.[\\w.]+/)||[])[0]||null,
|
||||
section: container?.innerText?.slice(0,1000) || null
|
||||
};
|
||||
})())"""))
|
||||
# Parse: lines after "Profiles" are name/type pairs until "Add more",
|
||||
# then "AI and devices" section follows
|
||||
profiles = []
|
||||
section = data.get("section", "")
|
||||
if section:
|
||||
lines = [l.strip() for l in section.split("\n") if l.strip()]
|
||||
try:
|
||||
start = lines.index("Profiles") + 1
|
||||
except ValueError:
|
||||
start = 0
|
||||
i = start
|
||||
while i < len(lines):
|
||||
if lines[i] == "Add more":
|
||||
i += 1
|
||||
continue
|
||||
if lines[i] == "AI and devices":
|
||||
# Next line is the device/service name
|
||||
if i + 1 < len(lines):
|
||||
profiles.append({"name": lines[i+1], "type": "ai_device"})
|
||||
break
|
||||
# Expect name + type pair
|
||||
if i + 1 < len(lines) and lines[i+1] not in (
|
||||
"Add more", "AI and devices", "Profiles"):
|
||||
profiles.append({"name": lines[i],
|
||||
"type": lines[i+1].lower()})
|
||||
i += 2
|
||||
else:
|
||||
i += 1
|
||||
return {"email": data.get("email"), "profiles": profiles}
|
||||
|
||||
def cmd_security_status(port):
|
||||
tab = get_ac_tab(port)
|
||||
data = evaluate(port, tab["webSocketDebuggerUrl"], (
|
||||
f"{AC_BASE}/password_and_security/",
|
||||
"""JSON.stringify({
|
||||
checkup: document.body.innerText.match(
|
||||
/Meta Security Checkup\\s*(\\d+) recommended actions/)?.[1] || null,
|
||||
sections: [...document.querySelectorAll('h2')]
|
||||
.map(e=>e.innerText.trim()).slice(0,10),
|
||||
body: document.body.innerText.slice(0,2000)
|
||||
})"""))
|
||||
return {"security_checkup_actions": data.get("checkup"),
|
||||
"sections": data.get("sections")}
|
||||
|
||||
def cmd_login_activity(port):
|
||||
tab = get_ac_tab(port)
|
||||
# "Where you're logged in" is on the password_and_security page
|
||||
data = evaluate(port, tab["webSocketDebuggerUrl"], (
|
||||
f"{AC_BASE}/password_and_security/",
|
||||
"""JSON.stringify({
|
||||
body: document.body.innerText.slice(0,4000)
|
||||
})"""))
|
||||
body = data.get("body", "")
|
||||
# Extract the "Where you're logged in" section
|
||||
idx = body.find("Where you're logged in")
|
||||
section = body[idx:idx+1500] if idx >= 0 else ""
|
||||
return {"where_logged_in": section}
|
||||
|
||||
def cmd_link_instagram(agent, port):
|
||||
"""Automate Meta Accounts Center linking flow and second-click OAuth handoff."""
|
||||
import websocket
|
||||
tab = get_ac_tab(port)
|
||||
ws = websocket.create_connection(tab["webSocketDebuggerUrl"], timeout=15)
|
||||
try:
|
||||
# Step 1: Navigate to manage accounts
|
||||
ws.send(json.dumps({"id": 1, "method": "Page.navigate", "params": {"url": f"{AC_BASE}/manage/"}}))
|
||||
ws.recv()
|
||||
time.sleep(3)
|
||||
|
||||
# Step 2: Look for 'Add profiles and devices' button or check if already linked
|
||||
expr_find_add = """(() => {
|
||||
const btns = [...document.querySelectorAll("div[role='button'], button, a")];
|
||||
const addBtn = btns.find(b => /add profiles|add accounts/i.test((b.innerText||"").trim()));
|
||||
if (addBtn) {
|
||||
addBtn.click();
|
||||
return {status: "clicked_add", text: addBtn.innerText};
|
||||
}
|
||||
return {status: "no_add_button"};
|
||||
})()"""
|
||||
ws.send(json.dumps({"id": 2, "method": "Runtime.evaluate", "params": {"expression": expr_find_add, "returnByValue": True}}))
|
||||
res2 = json.loads(ws.recv()).get("result", {}).get("result", {}).get("value", {})
|
||||
time.sleep(3)
|
||||
|
||||
# Step 3: Check dialog / popup for Instagram option or FXCAL handoff
|
||||
expr_handle_dialog = """(() => {
|
||||
const btns = [...document.querySelectorAll("div[role='button'], button, a")];
|
||||
// Check for Add Instagram button or Continue/Confirm
|
||||
const igBtn = btns.find(b => /instagram/i.test((b.innerText||"").trim()) && /add|connect/i.test((b.innerText||"").trim()));
|
||||
if (igBtn) {
|
||||
igBtn.click();
|
||||
return {status: "clicked_instagram_option"};
|
||||
}
|
||||
const confirmBtn = btns.find(b => /continue|confirm|yes, finish/i.test((b.innerText||"").trim()));
|
||||
if (confirmBtn) {
|
||||
confirmBtn.click();
|
||||
return {status: "clicked_confirm", text: confirmBtn.innerText};
|
||||
}
|
||||
return {status: "dialog_scanned", current_url: window.location.href};
|
||||
})()"""
|
||||
ws.send(json.dumps({"id": 3, "method": "Runtime.evaluate", "params": {"expression": expr_handle_dialog, "returnByValue": True}}))
|
||||
res3 = json.loads(ws.recv()).get("result", {}).get("result", {}).get("value", {})
|
||||
time.sleep(2)
|
||||
|
||||
return {
|
||||
"status": "success",
|
||||
"step1_add": res2,
|
||||
"step2_dialog": res3,
|
||||
"current_url": tab.get("url")
|
||||
}
|
||||
finally:
|
||||
ws.close()
|
||||
|
||||
def main():
|
||||
if len(sys.argv) < 3:
|
||||
print(__doc__.strip().split("\n")[0])
|
||||
print("Usage: meta-acct.py <list-linked|security-status|login-activity|link-instagram> <agent>")
|
||||
sys.exit(2)
|
||||
cmd, agent = sys.argv[1], sys.argv[2]
|
||||
port = cdp_port(agent)
|
||||
try:
|
||||
if cmd == "list-linked":
|
||||
out = cmd_list_linked(port)
|
||||
elif cmd == "security-status":
|
||||
out = cmd_security_status(port)
|
||||
elif cmd == "login-activity":
|
||||
out = cmd_login_activity(port)
|
||||
elif cmd == "link-instagram":
|
||||
out = cmd_link_instagram(agent, port)
|
||||
else:
|
||||
raise SystemExit(f"meta-acct: unknown command '{cmd}'")
|
||||
except Exception as e:
|
||||
print(json.dumps({"error": str(e), "agent": agent}))
|
||||
sys.exit(1)
|
||||
out["agent"] = agent
|
||||
out["checked_at"] = int(time.time())
|
||||
print(json.dumps(out, indent=2))
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -1,34 +0,0 @@
|
||||
#!/bin/bash
|
||||
# meta-acct.sh — Meta Accounts Center read API (via authenticated browser session).
|
||||
#
|
||||
# Usage:
|
||||
# meta-acct.sh list-linked <agent> # linked profiles in this Meta Account
|
||||
# meta-acct.sh security-status <agent> # login & recovery / security checkup
|
||||
# meta-acct.sh login-activity <agent> # "Where you're logged in" sessions
|
||||
#
|
||||
# The agent's browser must have an active Meta session (phone/email OTP login).
|
||||
# Reads NODES.md for the CDP port. Outputs JSON.
|
||||
#
|
||||
# Scope: Accounts Center linkage/security surface only. No writes.
|
||||
# Runs the Python implementation inside the agent's NetVM netns.
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
NETVM_DIR="${NETVM_DIR:-$HOME/Projects/NetVM}"
|
||||
SCRIPT="$NETVM_DIR/bin/meta-acct.py"
|
||||
|
||||
if [ $# -ne 2 ]; then
|
||||
echo "Usage: meta-acct.sh <list-linked|security-status|login-activity> <agent>" >&2
|
||||
exit 2
|
||||
fi
|
||||
|
||||
cmd="$1"
|
||||
agent="$2"
|
||||
|
||||
# Get the node name from ACCOUNTS.md (agent == node per unified naming)
|
||||
node="$agent"
|
||||
|
||||
# Run inside the netns so we reach the browser's CDP on 127.0.0.1.
|
||||
# NETVM_DIR must be explicit: sudo resets HOME to /root.
|
||||
exec sudo -n ip netns exec "warp-$node" \
|
||||
env NETVM_DIR="$NETVM_DIR" python3 "$SCRIPT" "$cmd" "$agent"
|
||||
@@ -1,47 +0,0 @@
|
||||
#!/bin/bash
|
||||
# Meta credential store CLI (operator-only)
|
||||
# Usage:
|
||||
# meta-creds.sh add <type> <id> # interactive add (prompts for fields)
|
||||
# meta-creds.sh get <type> <id> # output JSON to stdout (never log)
|
||||
# meta-creds.sh list <type> # list IDs (no secrets)
|
||||
# Types: muse, instagram, facebook
|
||||
STORE_DIR="/etc/netvm/meta-credentials"
|
||||
STORE="$STORE_DIR/store.age"
|
||||
KEY="$STORE_DIR/.age-key"
|
||||
if [ ! -f "$STORE" ] || [ ! -f "$KEY" ]; then
|
||||
echo "error: store not initialized" >&2; exit 1
|
||||
fi
|
||||
decrypt() { sudo age -d -i "$KEY" "$STORE" 2>/dev/null; }
|
||||
encrypt() {
|
||||
PUBKEY=$(sudo grep "public key" "$KEY" | awk '{print $4}')
|
||||
sudo age -r "$PUBKEY" -o "$STORE.tmp" 2>/dev/null && sudo mv "$STORE.tmp" "$STORE" && sudo chmod 600 "$STORE"
|
||||
}
|
||||
case "$1" in
|
||||
list)
|
||||
decrypt | jq -r ".$2 | keys[]" 2>/dev/null || echo "(empty)"
|
||||
;;
|
||||
get)
|
||||
decrypt | jq ".$2[\"$3\"]" 2>/dev/null
|
||||
;;
|
||||
add)
|
||||
TYPE="$2"; ID="$3"
|
||||
echo "Adding $TYPE/$ID (fields as JSON, empty to skip):"
|
||||
TMP=$(mktemp)
|
||||
decrypt > "$TMP" 2>/dev/null
|
||||
# Build entry via prompts
|
||||
ENTRY=$(jq -n '{}')
|
||||
for field in email phone username password notes age_verified instagram_linked verified_by; do
|
||||
read -p "$field: " val
|
||||
if [ -n "$val" ]; then
|
||||
ENTRY=$(echo "$ENTRY" | jq --arg v "$val" ".$field=\$v")
|
||||
fi
|
||||
done
|
||||
ENTRY=$(echo "$ENTRY" | jq ".verified_at=\"$(date -u +%FT%TZ)\"")
|
||||
jq --arg t "$TYPE" --arg id "$ID" --argjson e "$ENTRY" '.[$t][$id]=$e' "$TMP" | encrypt
|
||||
rm -f "$TMP"
|
||||
echo "added $TYPE/$ID"
|
||||
;;
|
||||
*)
|
||||
echo "usage: meta-creds.sh {list|get|add} <type> [id]"
|
||||
;;
|
||||
esac
|
||||
-622
@@ -1,622 +0,0 @@
|
||||
"""Input modulation for the DM follow-up system.
|
||||
|
||||
Not all inputs deserve the same follow-up policy. A routine timer wake
|
||||
should not nudge like a security alert, and a heartbeat loopback should
|
||||
never create a record at all.
|
||||
|
||||
The model is three layers:
|
||||
|
||||
1. INPUT TYPE declares what kind of traffic this is
|
||||
(wake, job, siphon, manual, health, heartbeat)
|
||||
2. MODULATION looks up the policy table for that type (+subtype)
|
||||
and returns timeout / nudges / escalate / priority / track-mode
|
||||
3. SEND-TIME TAGS remain the transport. The modulated policy renders
|
||||
into the canonical tag vocabulary ([reply:expected],
|
||||
[reply:timeout=N], [reply:nudges=N], [reply:escalate=X],
|
||||
[input:<type>]). Explicitly declared tags ALWAYS win over the
|
||||
table — a human who says "nudge me in 15m" is sovereign.
|
||||
|
||||
Track modes: "always" (create the record), "never" (don't, ever),
|
||||
"actionable" (create only when the sender marks the input actionable —
|
||||
used by wake digests, where an informational digest creates no loop but
|
||||
a tracked wake does).
|
||||
|
||||
Priorities feed dashboard sorting and nudge urgency text; they do not
|
||||
change the sweep mechanics.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
import time
|
||||
from dataclasses import dataclass, field
|
||||
from enum import Enum
|
||||
from pathlib import Path
|
||||
from typing import Dict, List, Optional, Tuple, Any
|
||||
|
||||
DEFAULT_STRATEGY_FILE = "/srv/box/strategy.json"
|
||||
FALLBACK_STRATEGY_FILE = "/home/super/Projects/NetVM/strategy.json"
|
||||
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Enumerations
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
class InputType(str, Enum):
|
||||
WAKE = "wake" # timer-driven operator digests
|
||||
JOB = "job" # job dispatch expecting [RESULT]
|
||||
SIPHON = "siphon" # side-chat -> main-chat surfacing
|
||||
MANUAL = "manual" # human/operator-authored DM
|
||||
HEALTH = "health" # system health check results
|
||||
HEARTBEAT = "heartbeat" # loopback liveness probes
|
||||
|
||||
|
||||
class Priority(str, Enum):
|
||||
ROUTINE = "routine"
|
||||
NORMAL = "normal"
|
||||
IMPORTANT = "important"
|
||||
CRITICAL = "critical"
|
||||
|
||||
|
||||
# Priority rank for sorting (higher = more urgent).
|
||||
_PRIORITY_RANK = {
|
||||
Priority.ROUTINE: 0,
|
||||
Priority.NORMAL: 1,
|
||||
Priority.IMPORTANT: 2,
|
||||
Priority.CRITICAL: 3,
|
||||
}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Policy
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
# Timeout clamps inherited from JOB-FOLLOWUP.md: 60s..7d.
|
||||
MIN_TIMEOUT_S = 60
|
||||
MAX_TIMEOUT_S = 604800
|
||||
MAX_NUDGES = 10
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Policy:
|
||||
"""A modulated follow-up policy, ready to render into tags."""
|
||||
input_type: InputType
|
||||
subtype: Optional[str] # e.g. siphon category, health status
|
||||
priority: Priority
|
||||
track: bool # create a dm_followup record?
|
||||
timeout_s: int # seconds to first nudge
|
||||
nudges: int # max auto-nudges
|
||||
escalate: Optional[str] # identity id, or None (silent close)
|
||||
# Which explicit values overrode the table (audit trail).
|
||||
overridden: Tuple[str, ...] = field(default_factory=tuple)
|
||||
|
||||
def rank(self) -> int:
|
||||
return _PRIORITY_RANK[self.priority]
|
||||
|
||||
@property
|
||||
def expect_reply(self) -> bool:
|
||||
return bool(self.track)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# The modulation table.
|
||||
#
|
||||
# Key: (InputType, subtype or None). Subtype None = default for that type.
|
||||
# track: True always / False never / "actionable" (wake digests).
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
# (track, priority, timeout_s, nudges, escalate)
|
||||
_MODULATION: Dict[Tuple[InputType, Optional[str]], Tuple] = {
|
||||
# Timer wakes: routine, batchy. Tracked only when the digest is
|
||||
# actionable (the wake-actionable pilot); otherwise informational.
|
||||
(InputType.WAKE, None): ("actionable", Priority.ROUTINE, 2 * 3600, 1, "opm"),
|
||||
(InputType.WAKE, "overdue"): (True, Priority.IMPORTANT, 3600, 2, "opm"),
|
||||
|
||||
# Job dispatches: the canonical "waiting on someone" loop.
|
||||
(InputType.JOB, None): (True, Priority.NORMAL, 3600, 2, "opm"),
|
||||
(InputType.JOB, "canary"): (True, Priority.NORMAL, 900, 2, "opm"),
|
||||
(InputType.JOB, "chain"): (True, Priority.IMPORTANT, 1800, 2, "opm"),
|
||||
|
||||
# Siphon hits: urgency follows the detected category.
|
||||
(InputType.SIPHON, "ALERT"): (True, Priority.CRITICAL, 600, 3, "user"),
|
||||
(InputType.SIPHON, "BLOCKER"): (True, Priority.CRITICAL, 900, 2, "opm"),
|
||||
(InputType.SIPHON, "DECISION"): (True, Priority.IMPORTANT, 3600, 2, "opm"),
|
||||
(InputType.SIPHON, "COMPLETED"): (False, Priority.ROUTINE, 0, 0, None),
|
||||
(InputType.SIPHON, "MILESTONE"): (False, Priority.ROUTINE, 0, 0, None),
|
||||
(InputType.SIPHON, None): (True, Priority.NORMAL, 3600, 2, "opm"),
|
||||
|
||||
# Manual DMs: sender declares; table only supplies the default.
|
||||
(InputType.MANUAL, None): (True, Priority.NORMAL, 3600, 2, "opm"),
|
||||
(InputType.MANUAL, "urgent"): (True, Priority.IMPORTANT, 900, 2, "opm"),
|
||||
(InputType.MANUAL, "fyi"): (True, Priority.ROUTINE, 7200, 1, None),
|
||||
|
||||
# Health: system-level, fast fuse.
|
||||
(InputType.HEALTH, "FAIL"): (True, Priority.CRITICAL, 600, 2, "opm"),
|
||||
(InputType.HEALTH, "DEGRADED"): (True, Priority.IMPORTANT, 1800, 2, "opm"),
|
||||
(InputType.HEALTH, "OK"): (False, Priority.ROUTINE, 0, 0, None),
|
||||
(InputType.HEALTH, None): (True, Priority.IMPORTANT, 1800, 2, "opm"),
|
||||
|
||||
# Heartbeat loopback: NEVER tracked. Hard exclusion.
|
||||
(InputType.HEARTBEAT, None): (False, Priority.ROUTINE, 0, 0, None),
|
||||
}
|
||||
|
||||
|
||||
def _clamp_timeout(s: int) -> int:
|
||||
return max(MIN_TIMEOUT_S, min(MAX_TIMEOUT_S, s))
|
||||
|
||||
|
||||
def _clamp_nudges(n: int) -> int:
|
||||
return max(0, min(MAX_NUDGES, n))
|
||||
|
||||
|
||||
def _strategy_file() -> str:
|
||||
if os.path.exists(DEFAULT_STRATEGY_FILE):
|
||||
return DEFAULT_STRATEGY_FILE
|
||||
if os.path.exists(FALLBACK_STRATEGY_FILE):
|
||||
return FALLBACK_STRATEGY_FILE
|
||||
if os.path.exists("/srv/box"):
|
||||
return DEFAULT_STRATEGY_FILE
|
||||
return FALLBACK_STRATEGY_FILE
|
||||
|
||||
|
||||
def load_strategy_overrides() -> Dict[str, dict]:
|
||||
path = _strategy_file()
|
||||
try:
|
||||
with open(path) as f:
|
||||
data = json.load(f)
|
||||
return data.get("overrides", {})
|
||||
except Exception:
|
||||
return {}
|
||||
|
||||
|
||||
def _save_overrides(overrides: Dict[str, dict], by: Optional[str] = None):
|
||||
payload = {
|
||||
"_meta": {
|
||||
"version": 1,
|
||||
"updated_at": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()),
|
||||
"updated_by": by or "super",
|
||||
},
|
||||
"overrides": overrides,
|
||||
}
|
||||
for dest in (DEFAULT_STRATEGY_FILE, FALLBACK_STRATEGY_FILE):
|
||||
try:
|
||||
os.makedirs(os.path.dirname(os.path.abspath(dest)), exist_ok=True)
|
||||
tmp = f"{dest}.tmp.{os.getpid()}"
|
||||
with open(tmp, "w") as f:
|
||||
json.dump(payload, f, indent=2)
|
||||
f.write("\n")
|
||||
os.chmod(tmp, 0o644)
|
||||
os.replace(tmp, dest)
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
|
||||
VALID_AGENTS = ("muse", "pip", "646", "opm")
|
||||
|
||||
|
||||
def get_strategy_row(input_type: InputType, subtype: Optional[str] = None, agent: Optional[str] = None) -> Tuple:
|
||||
"""Lookup modulation row with hierarchical runtime overrides applied.
|
||||
|
||||
Resolution hierarchy:
|
||||
1. Exact match: (input_type, subtype, agent)
|
||||
2. Agent default for type: (input_type, None, agent)
|
||||
3. Subtype default: (input_type, subtype, None)
|
||||
4. Type default: (input_type, None, None)
|
||||
5. Builtin table fallback
|
||||
"""
|
||||
overrides = load_strategy_overrides()
|
||||
ag = agent.lower() if agent else None
|
||||
sub = subtype.lower() if subtype else None
|
||||
t = input_type.value
|
||||
|
||||
# 1. Exact match: type:subtype:agent
|
||||
if sub and ag:
|
||||
key = f"{t}:{sub}:{ag}"
|
||||
if key in overrides:
|
||||
o = overrides[key]
|
||||
return (o.get("track", True), Priority(o.get("priority", "normal")),
|
||||
int(o.get("timeout_s", 3600)), int(o.get("nudges", 2)), o.get("escalate"))
|
||||
|
||||
# 2. Agent default for type: type:agent
|
||||
if ag:
|
||||
key = f"{t}:{ag}"
|
||||
if key in overrides:
|
||||
o = overrides[key]
|
||||
return (o.get("track", True), Priority(o.get("priority", "normal")),
|
||||
int(o.get("timeout_s", 3600)), int(o.get("nudges", 2)), o.get("escalate"))
|
||||
|
||||
# 3. Subtype default: type:subtype
|
||||
if sub:
|
||||
key = f"{t}:{sub}"
|
||||
if key in overrides:
|
||||
o = overrides[key]
|
||||
return (o.get("track", True), Priority(o.get("priority", "normal")),
|
||||
int(o.get("timeout_s", 3600)), int(o.get("nudges", 2)), o.get("escalate"))
|
||||
|
||||
# 4. Type default: type
|
||||
if t in overrides:
|
||||
o = overrides[t]
|
||||
return (o.get("track", True), Priority(o.get("priority", "normal")),
|
||||
int(o.get("timeout_s", 3600)), int(o.get("nudges", 2)), o.get("escalate"))
|
||||
|
||||
# 5. Builtin fallback (case-insensitive subtype matching)
|
||||
row = None
|
||||
if subtype:
|
||||
for (it, st), r in _MODULATION.items():
|
||||
if it == input_type and st and st.lower() == subtype.lower():
|
||||
row = r
|
||||
break
|
||||
if row is None:
|
||||
row = _MODULATION.get((input_type, None))
|
||||
if row is None:
|
||||
row = _MODULATION[(InputType.MANUAL, None)]
|
||||
return row
|
||||
|
||||
|
||||
def get_all_strategies() -> List[dict]:
|
||||
"""Return all strategies (builtins merged with hierarchical overrides)."""
|
||||
overrides = load_strategy_overrides()
|
||||
out = []
|
||||
seen = set()
|
||||
|
||||
for itype, sub in sorted(_MODULATION.keys(), key=lambda x: (x[0].value, x[1] or "")):
|
||||
key_str = f"{itype.value}:{sub.lower()}" if sub else itype.value
|
||||
seen.add(key_str)
|
||||
is_override = key_str in overrides
|
||||
row = get_strategy_row(itype, sub)
|
||||
track_mode, prio, timeout_s, nudges, escalate = row
|
||||
out.append({
|
||||
"key": key_str,
|
||||
"input_type": itype.value,
|
||||
"subtype": sub,
|
||||
"agent": None,
|
||||
"track": track_mode,
|
||||
"priority": prio.value if isinstance(prio, Priority) else str(prio),
|
||||
"timeout_s": timeout_s,
|
||||
"nudges": nudges,
|
||||
"escalate": escalate,
|
||||
"is_override": is_override,
|
||||
})
|
||||
|
||||
# Include any custom override keys not in builtins
|
||||
for key_str, o in overrides.items():
|
||||
if key_str not in seen:
|
||||
parts = key_str.split(":")
|
||||
itype_val = parts[0]
|
||||
sub_val = None
|
||||
agent_val = None
|
||||
|
||||
if len(parts) == 3:
|
||||
sub_val = parts[1].upper()
|
||||
agent_val = parts[2]
|
||||
elif len(parts) == 2:
|
||||
if parts[1].lower() in VALID_AGENTS:
|
||||
agent_val = parts[1].lower()
|
||||
else:
|
||||
sub_val = parts[1].upper()
|
||||
|
||||
prio = o.get("priority", "normal")
|
||||
out.append({
|
||||
"key": key_str,
|
||||
"input_type": itype_val,
|
||||
"subtype": sub_val,
|
||||
"agent": agent_val,
|
||||
"track": o.get("track", True),
|
||||
"priority": prio,
|
||||
"timeout_s": int(o.get("timeout_s", 3600)),
|
||||
"nudges": int(o.get("nudges", 2)),
|
||||
"escalate": o.get("escalate"),
|
||||
"is_override": True,
|
||||
})
|
||||
return out
|
||||
|
||||
|
||||
def _build_key(itype: str, sub: Optional[str] = None, agent: Optional[str] = None) -> str:
|
||||
t = itype.lower().strip()
|
||||
s = sub.lower().strip() if sub and sub != "*" else None
|
||||
a = agent.lower().strip() if agent and agent != "*" else None
|
||||
if s and a:
|
||||
return f"{t}:{s}:{a}"
|
||||
if a and not s:
|
||||
return f"{t}:{a}"
|
||||
if s and not a:
|
||||
return f"{t}:{s}"
|
||||
return t
|
||||
|
||||
|
||||
def set_strategy_override(
|
||||
input_type_str: str,
|
||||
subtype_str: Optional[str] = None,
|
||||
agent_str: Optional[str] = None,
|
||||
*,
|
||||
track: Any = None,
|
||||
priority: Optional[str] = None,
|
||||
timeout_s: Optional[int] = None,
|
||||
nudges: Optional[int] = None,
|
||||
escalate: Optional[str] = None,
|
||||
by: Optional[str] = None,
|
||||
) -> dict:
|
||||
"""Set an external strategy override with optional agent scope."""
|
||||
overrides = load_strategy_overrides()
|
||||
key = _build_key(input_type_str, subtype_str, agent_str)
|
||||
|
||||
# Get current values as baseline
|
||||
try:
|
||||
itype = InputType(input_type_str.lower())
|
||||
except ValueError:
|
||||
itype = InputType.MANUAL
|
||||
current_row = get_strategy_row(itype, subtype_str.upper() if subtype_str else None, agent=agent_str)
|
||||
c_track, c_prio, c_timeout, c_nudges, c_esc = current_row
|
||||
|
||||
new_entry = dict(overrides.get(key, {}))
|
||||
new_entry["key"] = key
|
||||
new_entry["input_type"] = itype.value
|
||||
new_entry["subtype"] = subtype_str.upper() if subtype_str else None
|
||||
new_entry["agent"] = agent_str.lower() if agent_str else None
|
||||
|
||||
if track is not None:
|
||||
if isinstance(track, str) and track.lower() == "true":
|
||||
new_entry["track"] = True
|
||||
elif isinstance(track, str) and track.lower() == "false":
|
||||
new_entry["track"] = False
|
||||
else:
|
||||
new_entry["track"] = track
|
||||
else:
|
||||
new_entry.setdefault("track", c_track)
|
||||
|
||||
if priority is not None:
|
||||
new_entry["priority"] = Priority(priority.lower()).value
|
||||
else:
|
||||
new_entry.setdefault("priority", c_prio.value if isinstance(c_prio, Priority) else str(c_prio))
|
||||
|
||||
if timeout_s is not None:
|
||||
new_entry["timeout_s"] = _clamp_timeout(int(timeout_s))
|
||||
else:
|
||||
new_entry.setdefault("timeout_s", c_timeout)
|
||||
|
||||
if nudges is not None:
|
||||
new_entry["nudges"] = _clamp_nudges(int(nudges))
|
||||
else:
|
||||
new_entry.setdefault("nudges", c_nudges)
|
||||
|
||||
if escalate is not None:
|
||||
new_entry["escalate"] = escalate if escalate != "none" else None
|
||||
else:
|
||||
new_entry.setdefault("escalate", c_esc)
|
||||
|
||||
overrides[key] = new_entry
|
||||
_save_overrides(overrides, by=by)
|
||||
return new_entry
|
||||
|
||||
|
||||
def reset_strategy_override(input_type_str: str, subtype_str: Optional[str] = None, agent_str: Optional[str] = None, by: Optional[str] = None) -> bool:
|
||||
"""Reset a strategy override to built-in default."""
|
||||
overrides = load_strategy_overrides()
|
||||
key = _build_key(input_type_str, subtype_str, agent_str)
|
||||
if key in overrides:
|
||||
del overrides[key]
|
||||
_save_overrides(overrides, by=by)
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
def set_override(input_type, subtype=None, agent=None, **kw):
|
||||
itype_str = input_type.value if hasattr(input_type, "value") else str(input_type)
|
||||
return set_strategy_override(itype_str, subtype, agent, **kw)
|
||||
|
||||
|
||||
def reset_override(input_type, subtype=None, agent=None, by=None):
|
||||
itype_str = input_type.value if hasattr(input_type, "value") else str(input_type)
|
||||
return reset_strategy_override(itype_str, subtype, agent, by=by)
|
||||
|
||||
|
||||
def evaluate_strategy(input_type, subtype=None, agent=None, **kw):
|
||||
if isinstance(input_type, str):
|
||||
try:
|
||||
input_type = InputType(input_type)
|
||||
except ValueError:
|
||||
input_type = InputType.MANUAL
|
||||
return modulate(input_type, subtype=subtype, agent=agent, **kw)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Public API
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def modulate(
|
||||
input_type: InputType,
|
||||
subtype: Optional[str] = None,
|
||||
agent: Optional[str] = None,
|
||||
*,
|
||||
actionable: bool = False,
|
||||
timeout_s: Optional[int] = None,
|
||||
nudges: Optional[int] = None,
|
||||
escalate: Optional[str] = None,
|
||||
priority: Optional[Priority] = None,
|
||||
track: Optional[bool] = None,
|
||||
) -> Optional[Policy]:
|
||||
"""Compute the follow-up policy for one send.
|
||||
|
||||
Returns None when the input should NOT be tracked (track=False, or
|
||||
track="actionable" with actionable=False).
|
||||
|
||||
Explicit keyword args always override the table (send-time
|
||||
sovereignty); they are recorded in Policy.overridden.
|
||||
Unknown input types fail closed to the MANUAL default.
|
||||
"""
|
||||
# Fail closed: unknown type -> manual default.
|
||||
if not isinstance(input_type, InputType):
|
||||
input_type = InputType.MANUAL
|
||||
|
||||
row = get_strategy_row(input_type, subtype, agent=agent)
|
||||
track_mode, prio, d_timeout, d_nudges, d_escalate = row
|
||||
|
||||
do_track = track if track is not None else (
|
||||
True if track_mode is True
|
||||
else False if track_mode is False
|
||||
else actionable # "actionable"
|
||||
)
|
||||
if not do_track:
|
||||
return None
|
||||
|
||||
overridden = []
|
||||
final_timeout = d_timeout
|
||||
if timeout_s is not None:
|
||||
final_timeout = _clamp_timeout(timeout_s)
|
||||
overridden.append("timeout")
|
||||
final_nudges = d_nudges
|
||||
if nudges is not None:
|
||||
final_nudges = _clamp_nudges(nudges)
|
||||
overridden.append("nudges")
|
||||
final_escalate = d_escalate
|
||||
if escalate is not None:
|
||||
final_escalate = escalate or None
|
||||
overridden.append("escalate")
|
||||
final_prio = priority if priority is not None else prio
|
||||
if priority is not None:
|
||||
overridden.append("priority")
|
||||
|
||||
return Policy(
|
||||
input_type=input_type,
|
||||
subtype=subtype,
|
||||
priority=final_prio,
|
||||
track=True,
|
||||
timeout_s=final_timeout,
|
||||
nudges=final_nudges,
|
||||
escalate=final_escalate,
|
||||
overridden=tuple(overridden),
|
||||
)
|
||||
|
||||
|
||||
def render_tags(policy: Policy) -> str:
|
||||
"""Render a policy into the canonical bracket-tag vocabulary.
|
||||
|
||||
Emits [reply:expected] [reply:timeout=N] [reply:nudges=N]
|
||||
[reply:escalate=X] [input:<type>]. The input tag is the audit
|
||||
trail: it records which modulation row drove the policy.
|
||||
"""
|
||||
parts = ["[reply:expected]"]
|
||||
parts.append(f"[reply:timeout={policy.timeout_s}]")
|
||||
parts.append(f"[reply:nudges={policy.nudges}]")
|
||||
if policy.escalate:
|
||||
parts.append(f"[reply:escalate={policy.escalate}]")
|
||||
parts.append(f"[input:{policy.input_type.value}]")
|
||||
return " ".join(parts)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Convenience constructors for the known producers.
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def for_siphon_hit(category: str, **kw) -> Optional[Policy]:
|
||||
"""Policy for a siphon hit; category is the detect.py category."""
|
||||
return modulate(InputType.SIPHON, category.upper(), **kw)
|
||||
|
||||
|
||||
def for_health(status: str, **kw) -> Optional[Policy]:
|
||||
"""Policy for a health result; status in {OK, DEGRADED, FAIL}."""
|
||||
return modulate(InputType.HEALTH, status.upper(), **kw)
|
||||
|
||||
|
||||
def for_job(job_name: str, **kw) -> Optional[Policy]:
|
||||
"""Policy for a job dispatch. Canary jobs get the short fuse."""
|
||||
subtype = "canary" if "canary" in job_name.lower() else None
|
||||
return modulate(InputType.JOB, subtype, **kw)
|
||||
|
||||
|
||||
def for_wake(actionable: bool = False, has_overdue: bool = False, **kw) -> Optional[Policy]:
|
||||
"""Policy for a timer wake digest."""
|
||||
subtype = "overdue" if has_overdue else None
|
||||
return modulate(InputType.WAKE, subtype, actionable=actionable, **kw)
|
||||
|
||||
|
||||
def sort_key(policy: Policy):
|
||||
"""Dashboard sort: critical first, then soonest timeout."""
|
||||
return (-policy.rank(), policy.timeout_s)
|
||||
|
||||
|
||||
def main():
|
||||
import argparse
|
||||
parser = argparse.ArgumentParser(description="Intrinsic Loop Modulation Strategy CLI")
|
||||
sub = parser.add_subparsers(dest="action")
|
||||
|
||||
sub.add_parser("list", help="List all strategies")
|
||||
|
||||
p_get = sub.add_parser("get", help="Get strategy row")
|
||||
p_get.add_argument("type", help="Input type (wake, job, siphon, manual, health, heartbeat)")
|
||||
p_get.add_argument("subtype", nargs="?", default=None, help="Optional subtype")
|
||||
|
||||
p_set = sub.add_parser("set", help="Set strategy override")
|
||||
p_set.add_argument("type", help="Input type")
|
||||
p_set.add_argument("--subtype", default=None, help="Subtype")
|
||||
p_set.add_argument("--track", choices=["true", "false", "actionable", "always", "never"], default=None)
|
||||
p_set.add_argument("--priority", choices=["routine", "normal", "important", "critical"], default=None)
|
||||
p_set.add_argument("--timeout", type=int, default=None, help="Timeout in seconds")
|
||||
p_set.add_argument("--nudges", type=int, default=None, help="Max auto nudges")
|
||||
p_set.add_argument("--escalate", default=None, help="Escalation target agent (e.g. opm, user, none)")
|
||||
|
||||
p_reset = sub.add_parser("reset", help="Reset strategy override")
|
||||
p_reset.add_argument("type", help="Input type")
|
||||
p_reset.add_argument("--subtype", default=None, help="Subtype")
|
||||
|
||||
p_eval = sub.add_parser("eval", help="Evaluate modulation policy for inputs")
|
||||
p_eval.add_argument("type", help="Input type")
|
||||
p_eval.add_argument("--subtype", default=None, help="Subtype")
|
||||
p_eval.add_argument("--actionable", action="store_true", help="Mark actionable (for wake)")
|
||||
|
||||
args = parser.parse_args()
|
||||
|
||||
if not args.action or args.action == "list":
|
||||
strats = get_all_strategies()
|
||||
print(f"\n{'KEY':20} {'TRACK':10} {'PRIORITY':10} {'TIMEOUT':10} {'NUDGES':8} {'ESCALATE':10} {'SOURCE':8}")
|
||||
print("─" * 80)
|
||||
for s in strats:
|
||||
src = "OVERRIDE" if s.get("is_override") else "BUILTIN"
|
||||
esc = s.get("escalate") or "-"
|
||||
print(f"{s['key']:20} {str(s['track']):10} {s['priority']:10} {str(s['timeout_s'])+'s':10} {str(s['nudges']):8} {esc:10} {src:8}")
|
||||
print()
|
||||
elif args.action == "get":
|
||||
try:
|
||||
itype = InputType(args.type.lower())
|
||||
except ValueError:
|
||||
print(f"Unknown input type: {args.type}", file=sys.stderr)
|
||||
sys.exit(1)
|
||||
row = get_strategy_row(itype, args.subtype.upper() if args.subtype else None)
|
||||
print(json.dumps({
|
||||
"type": itype.value,
|
||||
"subtype": args.subtype,
|
||||
"track": row[0],
|
||||
"priority": row[1].value if isinstance(row[1], Priority) else str(row[1]),
|
||||
"timeout_s": row[2],
|
||||
"nudges": row[3],
|
||||
"escalate": row[4],
|
||||
}, indent=2))
|
||||
elif args.action == "set":
|
||||
res = set_strategy_override(
|
||||
args.type, args.subtype,
|
||||
track=args.track, priority=args.priority,
|
||||
timeout_s=args.timeout, nudges=args.nudges, escalate=args.escalate,
|
||||
by=os.environ.get("BOX_CALLER", "cli")
|
||||
)
|
||||
print(f"Updated strategy override for {args.type}:{args.subtype or '*'}: {json.dumps(res)}")
|
||||
elif args.action == "reset":
|
||||
ok = reset_strategy_override(args.type, args.subtype, by=os.environ.get("BOX_CALLER", "cli"))
|
||||
print(f"Reset {args.type}:{args.subtype or '*'}: {'Success' if ok else 'Not overridden'}")
|
||||
elif args.action == "eval":
|
||||
try:
|
||||
itype = InputType(args.type.lower())
|
||||
except ValueError:
|
||||
itype = InputType.MANUAL
|
||||
pol = modulate(itype, args.subtype.upper() if args.subtype else None, actionable=args.actionable)
|
||||
if pol:
|
||||
print(f"Policy: {pol}")
|
||||
print(f"Tags : {render_tags(pol)}")
|
||||
else:
|
||||
print("Policy: None (Not tracked)")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -1,88 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Side-chat to main-chat work siphon — monitor loop.
|
||||
|
||||
Polls side chats for new messages, runs detection, siphons hits to main.
|
||||
|
||||
This is the integration point for bl. In production:
|
||||
- list_sidechats() calls muse-chat-api.py or the sidechat manager
|
||||
- get_messages() reads thread messages via CDP
|
||||
- post_to_main() sends via muse-chat-api.py send to main chat
|
||||
|
||||
For the prototype, all three are injectable (see tests).
|
||||
"""
|
||||
|
||||
import time
|
||||
from typing import Callable, Dict, List
|
||||
|
||||
from detect import detect, is_opted_out
|
||||
from siphon import siphon, RateLimiter
|
||||
|
||||
|
||||
# Message shape: {"id": str, "text": str, "author": str, "ts": str}
|
||||
Message = Dict[str, str]
|
||||
|
||||
|
||||
def monitor_once(
|
||||
list_sidechats: Callable[[], List[Dict[str, str]]],
|
||||
get_messages: Callable[[str, str], List[Message]],
|
||||
post_to_main: Callable[[str], bool],
|
||||
watermarks: Dict[str, str],
|
||||
limiter: RateLimiter = None,
|
||||
min_confidence: float = 0.6,
|
||||
) -> Dict[str, str]:
|
||||
"""
|
||||
One poll cycle. Returns updated watermarks.
|
||||
|
||||
list_sidechats: () -> [{"id": thread_id, "name": str, "agent": str}]
|
||||
get_messages: (thread_id, since_msg_id) -> [messages newer than watermark]
|
||||
post_to_main: (text) -> True on success
|
||||
watermarks: {thread_id: last_seen_message_id}
|
||||
"""
|
||||
lim = limiter or RateLimiter()
|
||||
new_marks = dict(watermarks)
|
||||
|
||||
for chat in list_sidechats():
|
||||
tid = chat["id"]
|
||||
agent = chat.get("agent", "unknown")
|
||||
|
||||
if is_opted_out(tid):
|
||||
continue
|
||||
|
||||
since = watermarks.get(tid, "")
|
||||
try:
|
||||
messages = get_messages(tid, since)
|
||||
except Exception:
|
||||
continue # don't let one bad chat kill the cycle
|
||||
|
||||
for msg in messages:
|
||||
mid = msg.get("id", "")
|
||||
text = msg.get("text", "")
|
||||
if not mid or not text:
|
||||
continue
|
||||
|
||||
# Update watermark to newest seen
|
||||
new_marks[tid] = mid
|
||||
|
||||
hit = detect(text, tid, mid, min_confidence)
|
||||
if hit:
|
||||
siphon(hit, agent, post_to_main, lim)
|
||||
|
||||
return new_marks
|
||||
|
||||
|
||||
def monitor_loop(
|
||||
list_sidechats,
|
||||
get_messages,
|
||||
post_to_main,
|
||||
poll_interval: int = 60,
|
||||
watermarks: Dict[str, str] = None,
|
||||
):
|
||||
"""Run forever. For production use with systemd timer instead."""
|
||||
marks = watermarks or {}
|
||||
limiter = RateLimiter()
|
||||
while True:
|
||||
marks = monitor_once(
|
||||
list_sidechats, get_messages, post_to_main, marks, limiter
|
||||
)
|
||||
time.sleep(poll_interval)
|
||||
@@ -1,154 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
# NetVM native interactive muse CLI wrapper
|
||||
# Enforces account/node selection, verifies/refreshes authentication cookies,
|
||||
# and executes commands in the dedicated network namespace.
|
||||
set -euo pipefail
|
||||
|
||||
SCRIPT_SRC="${BASH_SOURCE[0]}"
|
||||
while [ -h "$SCRIPT_SRC" ]; do
|
||||
DIR="$(cd -P "$(dirname "$SCRIPT_SRC")" && pwd)"
|
||||
SCRIPT_SRC="$(readlink "$SCRIPT_SRC")"
|
||||
[[ $SCRIPT_SRC != /* ]] && SCRIPT_SRC="$DIR/$SCRIPT_SRC"
|
||||
done
|
||||
NETVM_BIN="$(cd -P "$(dirname "$SCRIPT_SRC")" && pwd)"
|
||||
VALID_ACCOUNTS=("muse" "pip" "646" "opm" "def" "dev")
|
||||
|
||||
show_usage() {
|
||||
echo "Usage: muse <account> <command> [arguments...]"
|
||||
echo " muse -a <account> <command> [arguments...]"
|
||||
echo ""
|
||||
echo "Available accounts:"
|
||||
for acct in "${VALID_ACCOUNTS[@]}"; do
|
||||
echo " • $acct"
|
||||
done
|
||||
echo ""
|
||||
echo "Common commands:"
|
||||
echo " chat [--thread <id>] Launch interactive conversational shell / REPL"
|
||||
echo " tmux <cmd> [args...] Manage shared Muse tmux sessions (new, send, capture, ls, kill)"
|
||||
echo " status Check agent status, sessions, and unread"
|
||||
echo " threads List active threads and sidechats"
|
||||
echo " history --thread <id> View message history"
|
||||
echo " send --thread <id> msg Send message to an agent"
|
||||
echo " unread View unread counts"
|
||||
echo ""
|
||||
echo "Authentication & Cookie Management:"
|
||||
echo " Cookies are isolated per-node in ~/.config/muse-cli/<account>/"
|
||||
echo " If expired, the CLI attempts automated CDP cookie extraction."
|
||||
echo ""
|
||||
}
|
||||
|
||||
ACCOUNT=""
|
||||
POSITIONAL=()
|
||||
|
||||
# If first argument does not start with '-', treat it as the account candidate
|
||||
if [[ $# -gt 0 && "$1" != -* ]]; then
|
||||
ACCOUNT="$1"
|
||||
shift
|
||||
fi
|
||||
|
||||
while [[ $# -gt 0 ]]; do
|
||||
case "$1" in
|
||||
-a|--account)
|
||||
if [[ -z "${2:-}" ]]; then
|
||||
echo "Error: --account requires an argument." >&2
|
||||
exit 1
|
||||
fi
|
||||
ACCOUNT="$2"
|
||||
shift 2
|
||||
;;
|
||||
--account=*)
|
||||
ACCOUNT="${1#*=}"
|
||||
shift
|
||||
;;
|
||||
-h|--help)
|
||||
if [[ -z "$ACCOUNT" && ${#POSITIONAL[@]} -eq 0 ]]; then
|
||||
show_usage
|
||||
exit 0
|
||||
fi
|
||||
POSITIONAL+=("$1")
|
||||
shift
|
||||
;;
|
||||
*)
|
||||
POSITIONAL+=("$1")
|
||||
shift
|
||||
;;
|
||||
esac
|
||||
done
|
||||
|
||||
if [[ -z "$ACCOUNT" ]]; then
|
||||
echo "Error: No account specified. Specify account as first argument or with -a/--account." >&2
|
||||
echo ""
|
||||
show_usage
|
||||
exit 1
|
||||
fi
|
||||
|
||||
VALID=0
|
||||
for acct in "${VALID_ACCOUNTS[@]}"; do
|
||||
if [[ "$ACCOUNT" == "$acct" ]]; then
|
||||
VALID=1
|
||||
break
|
||||
fi
|
||||
done
|
||||
|
||||
if [[ $VALID -eq 0 ]]; then
|
||||
echo "Error: Invalid account '$ACCOUNT'." >&2
|
||||
echo ""
|
||||
show_usage
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if [[ ${#POSITIONAL[@]} -eq 0 ]]; then
|
||||
POSITIONAL=("status")
|
||||
fi
|
||||
|
||||
# If subcommand is 'chat', launch interactive chat REPL
|
||||
if [[ "${POSITIONAL[0]}" == "chat" ]]; then
|
||||
shift_args=("${POSITIONAL[@]:1}")
|
||||
exec python3 "$NETVM_BIN/muse-chat-repl.py" "$ACCOUNT" "${shift_args[@]}"
|
||||
fi
|
||||
|
||||
# If subcommand is 'tmux', dispatch to shared muse-tmux manager
|
||||
if [[ "${POSITIONAL[0]}" == "tmux" ]]; then
|
||||
shift_args=("${POSITIONAL[@]:1}")
|
||||
exec python3 "$NETVM_BIN/muse-tmux.py" "${shift_args[@]}"
|
||||
fi
|
||||
|
||||
# Execute command; if auth fails and auto-cdp fails, offer interactive OTP sign-in if running interactively
|
||||
set +e
|
||||
"$NETVM_BIN/muse-cli-node" "$ACCOUNT" "${POSITIONAL[@]}"
|
||||
RC=$?
|
||||
set -e
|
||||
|
||||
if [[ $RC -ne 0 && -t 0 ]]; then
|
||||
# Check if failure is authentication related
|
||||
echo ""
|
||||
echo "Notice: muse-cli command failed with exit code $RC."
|
||||
echo "Would you like to initiate an interactive login for account '$ACCOUNT'? [y/N] "
|
||||
read -r response
|
||||
if [[ "$response" =~ ^([yY][eE][sS]|[yY])$ ]]; then
|
||||
# Determine email for account if available
|
||||
EMAIL=$(grep -E "^\|[[:space:]]*$ACCOUNT[[:space:]]*\|" "$NETVM_BIN/../ACCOUNTS.md" | awk -F '|' '{print $7}' | tr -d ' ' || true)
|
||||
if [[ -z "$EMAIL" || "$EMAIL" == "-" ]]; then
|
||||
echo -n "Enter email for account '$ACCOUNT': "
|
||||
read -r EMAIL
|
||||
fi
|
||||
|
||||
echo "Starting sign-in for $EMAIL..."
|
||||
set +e
|
||||
"$NETVM_BIN/muse-signin.py" --email "$EMAIL"
|
||||
SIGNIN_RC=$?
|
||||
set -e
|
||||
|
||||
if [[ $SIGNIN_RC -eq 2 ]]; then
|
||||
echo -n "Enter the OTP code received: "
|
||||
read -r OTP_CODE
|
||||
"$NETVM_BIN/muse-signin.py" --email "$EMAIL" --otp "$OTP_CODE"
|
||||
echo "Extracting cookies for $ACCOUNT..."
|
||||
"$NETVM_BIN/refresh-node-cookies.py" "$ACCOUNT"
|
||||
echo "Retrying command..."
|
||||
exec "$NETVM_BIN/muse-cli-node" "$ACCOUNT" "${POSITIONAL[@]}"
|
||||
fi
|
||||
fi
|
||||
fi
|
||||
|
||||
exit $RC
|
||||
@@ -1,140 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
muse-auth.py: Muse.ai authentication state detection and flows.
|
||||
|
||||
Handles the tricky parts of automation:
|
||||
- Sign IN (existing account) vs Sign UP (new account) have DIFFERENT verification flows
|
||||
- Meta account selection has UI quirks (+1 for extra accounts)
|
||||
- Login can get stuck in incomplete states that need detection and restart
|
||||
|
||||
STATES (detectable via CDP):
|
||||
LANDING: Title == "muse.ai", not logged in
|
||||
ACCOUNT_SELECTION: Body contains "Your email matches multiple accounts"
|
||||
UI QUIRK: Meta shows "+1" for extra accounts. The "+1" option is NOT
|
||||
the right one — find the actual account name (e.g., "Nico Parada",
|
||||
not "auxfate"). The +1 is a collapsed list, not a selectable account.
|
||||
OTP_PROMPT: Body contains "To log in, enter the code"
|
||||
LOGGED_OUT: Body contains "Log in" button
|
||||
NOT_IN_CHAT: API messages times out (no chat UI to read)
|
||||
LOGGED_IN: Title contains "Chat —" or "Muse —", body has "Main chat"/"Side chats",
|
||||
message input present, API returns actual content.
|
||||
|
||||
SIGN IN vs SIGN UP:
|
||||
Sign IN (existing account):
|
||||
1. Enter email/phone
|
||||
2. OTP prompt (code sent to email/phone)
|
||||
3. Meta account selection (if multiple accounts match)
|
||||
4. Chat UI
|
||||
|
||||
Sign UP (new account):
|
||||
1. Enter email/phone
|
||||
2. OTP prompt (code sent to email/phone)
|
||||
3. **DIFFERENT:** May require additional verification:
|
||||
- CAPTCHA
|
||||
- Phone verification (if email used)
|
||||
- Email verification (if phone used)
|
||||
- Terms acceptance
|
||||
- Profile setup (name, etc.)
|
||||
4. **DIFFERENT:** May NOT have Meta account selection (new account = no existing Meta link)
|
||||
5. Chat UI (possibly with onboarding tour)
|
||||
|
||||
The sign-up flow is less predictable and may require human intervention
|
||||
for CAPTCHAs and other anti-bot measures. Sign-in is more automatable.
|
||||
|
||||
RESTART LOGIC:
|
||||
If state != LOGGED_IN, restart from scratch:
|
||||
- Clear cookies/session
|
||||
- Navigate to muse.ai
|
||||
- Begin sign-in flow
|
||||
- Don't try to resume a stuck flow (OTP codes expire, sessions timeout)
|
||||
"""
|
||||
|
||||
import json
|
||||
import urllib.request
|
||||
import websocket
|
||||
import time
|
||||
|
||||
def get_state(cdp_url):
|
||||
"""
|
||||
Detect the current authentication state via CDP.
|
||||
Returns: (state, details)
|
||||
"""
|
||||
try:
|
||||
with urllib.request.urlopen(cdp_url, timeout=5) as r:
|
||||
ts = json.load(r)
|
||||
pages = [t for t in ts if t.get("type") == "page"]
|
||||
if not pages:
|
||||
return ("NO_PAGE", "No browser page found")
|
||||
p = pages[0]
|
||||
ws = websocket.create_connection(p["webSocketDebuggerUrl"], timeout=10)
|
||||
except Exception as e:
|
||||
return ("CDP_ERROR", str(e))
|
||||
|
||||
def ev(expr):
|
||||
try:
|
||||
ws.send(json.dumps({
|
||||
"id": 1, "method": "Runtime.evaluate",
|
||||
"params": {"expression": expr, "returnByValue": True}
|
||||
}))
|
||||
r = json.loads(ws.recv())
|
||||
return r["result"]["result"].get("value", "")
|
||||
except:
|
||||
return ""
|
||||
|
||||
title = ev("document.title") or ""
|
||||
body = ev("document.body.innerText.slice(0,2000)") or ""
|
||||
ws.close()
|
||||
|
||||
# Check states in order of specificity
|
||||
if "Your email matches multiple accounts" in body:
|
||||
return ("ACCOUNT_SELECTION", "Meta account selection - watch for +1 quirk")
|
||||
if "To log in, enter the code" in body:
|
||||
return ("OTP_PROMPT", "Waiting for OTP code")
|
||||
if "Main chat" in body and "Side chats" in body:
|
||||
return ("LOGGED_IN", "In chat UI")
|
||||
if title.strip() == "muse.ai":
|
||||
return ("LANDING", "On landing page, not logged in")
|
||||
if "Log in" in body:
|
||||
return ("LOGGED_OUT", "Logged out or not started")
|
||||
|
||||
return ("UNKNOWN", f"Title: {title[:50]}")
|
||||
|
||||
def is_logged_in(cdp_url):
|
||||
"""Quick check: are we in the chat?"""
|
||||
state, _ = get_state(cdp_url)
|
||||
return state == "LOGGED_IN"
|
||||
|
||||
if __name__ == "__main__":
|
||||
import sys
|
||||
cdp = sys.argv[1] if len(sys.argv) > 1 else "http://127.0.0.1:9440/json/list"
|
||||
state, details = get_state(cdp)
|
||||
print(f"State: {state}")
|
||||
print(f"Details: {details}")
|
||||
|
||||
# Holdings logging: record what the Chromebox holds for each node
|
||||
# Logs to ~/Projects/NetVM/logs/auth-holdings.log
|
||||
# Format: timestamp | node | state | email_hint | account_name | notes
|
||||
# This lets us know which Proton address was used without asking the user again.
|
||||
|
||||
import os
|
||||
from datetime import datetime
|
||||
|
||||
def log_holdings(node, state, email_hint="", account_name="", notes=""):
|
||||
"""Log the Chromebox holdings for a node. Call after auth state detection."""
|
||||
log_dir = os.path.expanduser("~/Projects/NetVM/logs")
|
||||
os.makedirs(log_dir, exist_ok=True)
|
||||
log_file = os.path.join(log_dir, "auth-holdings.log")
|
||||
ts = datetime.now().isoformat()
|
||||
# Never log full email or secrets — use hint (e.g., domain or first char)
|
||||
line = f"{ts} | {node} | {state} | {email_hint} | {account_name} | {notes}\n"
|
||||
with open(log_file, "a") as f:
|
||||
f.write(line)
|
||||
|
||||
def get_holdings(node):
|
||||
"""Get last known holdings for a node."""
|
||||
log_file = os.path.expanduser("~/Projects/NetVM/logs/auth-holdings.log")
|
||||
if not os.path.exists(log_file):
|
||||
return None
|
||||
with open(log_file) as f:
|
||||
lines = [l.strip() for l in f if l.strip() and f"| {node} |" in l]
|
||||
return lines[-1] if lines else None
|
||||
+106
-804
@@ -1,817 +1,119 @@
|
||||
#!/usr/bin/env python3
|
||||
# GOLDEN PATH: container -> VM (34.139.37.135) -> bl (100.123.153.75) -> netns -> browser -> agent
|
||||
# This API is the bridge. Every call traverses 4 hops. Respect the path.
|
||||
|
||||
"""
|
||||
Multi-account muse.ai chat API with approval handling.
|
||||
|
||||
Approvals: The browser may show permission dialogs (e.g., "Allow pip to share
|
||||
information with 34.139.37.135?"). The API detects these and handles them:
|
||||
- Known-safe (our infrastructure IPs): auto-approve
|
||||
- Unknown: raise APPROVAL_NEEDED, operator decides via chat
|
||||
|
||||
"""Muse.ai chat API via CDP (headless smoke profile).
|
||||
Usage:
|
||||
muse-chat-api.py --account <agent> send "message"
|
||||
muse-chat-api.py --account <agent> messages [n]
|
||||
muse-chat-api.py --account <agent> wait [timeout]
|
||||
muse-chat-api.py --account <agent> approvals # check pending approvals
|
||||
muse-chat-api.py --account <agent> upload <file> [--message txt] [--dry-run]
|
||||
|
||||
Exit codes: 0 ok, 1 error, 2 APPROVAL_NEEDED (human decision),
|
||||
3 DOM_NOT_READY (browser not in expected chat state - safe to retry).
|
||||
muse-chat-api.py send "hello" # send message to current chat
|
||||
muse-chat-api.py messages # print recent messages
|
||||
muse-chat-api.py wait [timeout] # wait for new response\n muse-chat-api.py navigate <url> # go to thread URL
|
||||
Requires: websocket-client (pip install --break-system-packages websocket-client)
|
||||
CDP relay must be up: http://10.201.87.2:9410/json/list
|
||||
"""
|
||||
import json, urllib.request, websocket, time, sys, argparse, importlib.util, re
|
||||
import os
|
||||
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
|
||||
try:
|
||||
from sidechat_manager import ensure_sidebar, log_sidechat_op
|
||||
HAS_SIDECHAT_MANAGER = True
|
||||
except ImportError:
|
||||
HAS_SIDECHAT_MANAGER = False
|
||||
try:
|
||||
from cdp_queue import cdp_slot, PRIORITY_HIGH, PRIORITY_NORMAL, PRIORITY_LOW
|
||||
HAS_CDP_QUEUE = True
|
||||
except ImportError:
|
||||
HAS_CDP_QUEUE = False
|
||||
import json, sys, time, urllib.request, subprocess, os
|
||||
import websocket
|
||||
|
||||
CDP_URL = "http://10.201.87.2:9410/json/list"
|
||||
|
||||
def _load_accounts():
|
||||
"""Build the accounts table from the NODES.md registry — new nodes
|
||||
propagate automatically instead of being hardcoded here."""
|
||||
path = "/home/super/Projects/NetVM/bin/netvm-registry.py"
|
||||
spec = importlib.util.spec_from_file_location("netvm_registry", path)
|
||||
mod = importlib.util.module_from_spec(spec)
|
||||
spec.loader.exec_module(mod)
|
||||
accounts = {}
|
||||
for node, rec in mod.load().items():
|
||||
peer_ip = rec.get("peer_ip", "127.0.0.1")
|
||||
accounts[node] = (node, "http://%s:%d/json/list" % (peer_ip, rec["cdp_port"]))
|
||||
return accounts
|
||||
|
||||
|
||||
ACCOUNTS = _load_accounts()
|
||||
|
||||
# IPs we trust for auto-approval (our infrastructure)
|
||||
TRUSTED_IPS = {
|
||||
"34.139.37.135", # VM (gateway)
|
||||
"100.123.153.75", # bl (main compute)
|
||||
"100.81.31.9", # VM tailnet
|
||||
}
|
||||
|
||||
def get_page(node, cdp_url):
|
||||
urls = [cdp_url]
|
||||
m = re.search(r":(\d+)/", cdp_url)
|
||||
if m:
|
||||
port = m.group(1)
|
||||
if "127.0.0.1" in cdp_url:
|
||||
path = "/home/super/Projects/NetVM/bin/netvm-registry.py"
|
||||
spec = importlib.util.spec_from_file_location("netvm_registry", path)
|
||||
mod = importlib.util.module_from_spec(spec)
|
||||
spec.loader.exec_module(mod)
|
||||
pip = mod.peer_ip_for(node)
|
||||
if pip:
|
||||
urls.append(f"http://{pip}:{port}/json/list")
|
||||
else:
|
||||
urls.append(f"http://127.0.0.1:{port}/json/list")
|
||||
ts = None
|
||||
last_err = None
|
||||
for u in urls:
|
||||
try:
|
||||
with urllib.request.urlopen(u, timeout=5) as r:
|
||||
ts = json.load(r)
|
||||
break
|
||||
except Exception as e:
|
||||
last_err = e
|
||||
continue
|
||||
if not ts:
|
||||
print(f"ERROR: No page found ({last_err})", file=sys.stderr)
|
||||
sys.exit(1)
|
||||
pages = [t for t in ts if t.get('type') == 'page']
|
||||
def get_page():
|
||||
with urllib.request.urlopen(CDP_URL, timeout=5) as r:
|
||||
targets = json.load(r)
|
||||
pages = [t for t in targets if t.get('type')=='page' and 'muse.ai' in t.get('url','')]
|
||||
if not pages:
|
||||
print("ERROR: No page found", file=sys.stderr)
|
||||
sys.exit(1)
|
||||
raise RuntimeError("no muse.ai page found")
|
||||
return pages[0]
|
||||
|
||||
def ev(ws, expr, await_p=False):
|
||||
ws.send(json.dumps({
|
||||
"id": 1, "method": "Runtime.evaluate",
|
||||
"params": {"expression": expr, "returnByValue": True, "awaitPromise": await_p}
|
||||
}))
|
||||
# Drain CDP events until we get our command response (id 1).
|
||||
# The browser can emit events (Runtime.executionContextCreated, etc.)
|
||||
# at any time; taking the first recv() blindly returns None on a
|
||||
# busy page (observed as transient navigation failures in dm.py
|
||||
# sidechat sends, 2026-10-04 — same class as the NO_SWITCHER fix
|
||||
# in box-chat-cdp.py commit 8d4bfa7).
|
||||
for _ in range(50):
|
||||
resp = json.loads(ws.recv())
|
||||
if resp.get("id") == 1:
|
||||
break
|
||||
else:
|
||||
return None
|
||||
return resp.get('result', {}).get('result', {}).get('value')
|
||||
|
||||
def check_approvals(ws):
|
||||
"""
|
||||
Check for browser permission dialogs.
|
||||
Returns list of (dialog_text, is_trusted, action_taken).
|
||||
"""
|
||||
result = ev(ws, """(() => {
|
||||
const dialogs = [];
|
||||
// 1. Check confirmed structural selectors
|
||||
const headers = [...document.querySelectorAll('[data-testid="approval-panel-header"], [data-testid*="approval"]')];
|
||||
for (const h of headers) {
|
||||
let card = h;
|
||||
for (let i = 0; i < 6 && card && card.parentElement && card.parentElement !== document.body; i++) {
|
||||
if (card.querySelector('button[data-hatch-approval-primary-action="true"]') ||
|
||||
[...card.querySelectorAll('button')].some(b => {
|
||||
const t = (b.innerText||'').toLowerCase();
|
||||
return t.includes('allow once') || t.includes('deny');
|
||||
})) {
|
||||
dialogs.push(card.innerText.slice(0, 500));
|
||||
break;
|
||||
}
|
||||
card = card.parentElement;
|
||||
}
|
||||
}
|
||||
// 2. Check for "Allow ... to share" pattern if not found
|
||||
if (dialogs.length === 0) {
|
||||
const body = document.body ? document.body.innerText : '';
|
||||
if (body.includes('Allow') && body.includes('to share')) {
|
||||
const els = [...document.querySelectorAll('*')].filter(el => {
|
||||
const t = el.innerText || '';
|
||||
return t.includes('Allow') && t.includes('to share') && t.length < 500;
|
||||
});
|
||||
for (const el of els.slice(0,3)) {
|
||||
dialogs.push(el.innerText.slice(0,200));
|
||||
}
|
||||
}
|
||||
}
|
||||
// 3. Check for other permission patterns
|
||||
if (dialogs.length === 0) {
|
||||
const perm_btns = [...document.querySelectorAll('button')].filter(b => {
|
||||
const t = (b.innerText||'').toLowerCase();
|
||||
return t.includes('allow') || t.includes('deny') || t.includes('block');
|
||||
});
|
||||
if (perm_btns.length >= 2) {
|
||||
const parent = perm_btns[0].closest('div');
|
||||
if (parent) dialogs.push(parent.innerText.slice(0,200));
|
||||
}
|
||||
}
|
||||
return JSON.stringify(dialogs);
|
||||
})()""")
|
||||
try:
|
||||
dialogs = json.loads(result) if result else []
|
||||
except:
|
||||
dialogs = []
|
||||
|
||||
actions = []
|
||||
for d in dialogs:
|
||||
# Extract IP if present
|
||||
import re
|
||||
ips = re.findall(r'\b\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}\b', d)
|
||||
# Check trust: if IP present, must be in TRUSTED_IPS; if no IP, untrusted approval dialog
|
||||
if ips:
|
||||
is_trusted = any(ip in TRUSTED_IPS for ip in ips)
|
||||
else:
|
||||
# Check if this is a known false positive or real dialog
|
||||
if "allow once" in d.lower() or "approval-panel" in d.lower() or "wants to" in d.lower():
|
||||
is_trusted = False
|
||||
else:
|
||||
continue
|
||||
|
||||
if is_trusted:
|
||||
# Auto-approve: click primary action or "Allow once" / "Allow"
|
||||
clicked = ev(ws, """(async()=>{
|
||||
const b = document.querySelector('button[data-hatch-approval-primary-action="true"]') ||
|
||||
[...document.querySelectorAll('button')].find(x=>{
|
||||
const t = (x.innerText||'').toLowerCase().trim();
|
||||
return t === 'allow once' || t.includes('allow once') || t === 'allow';
|
||||
});
|
||||
if (b) { b.click(); return 'clicked:'+b.innerText.slice(0,20); }
|
||||
return 'NOTFOUND';
|
||||
})()""", True)
|
||||
actions.append((d[:80], True, clicked))
|
||||
else:
|
||||
actions.append((d[:80], False, "APPROVAL_NEEDED"))
|
||||
|
||||
return actions
|
||||
|
||||
# Exit code for "browser DOM not ready" — distinct from 1 (generic error)
|
||||
# and 2 (APPROVAL_NEEDED, needs a human). DOM_NOT_READY is safe to retry
|
||||
# after a few seconds: the React app may still be hydrating (S3 partial
|
||||
# load) or the Warp tunnel may have just hiccupped. dm.py treats any
|
||||
# non-"sent" send output as a failed attempt and retries, so this is a
|
||||
# fail-closed retry signal, not a crash.
|
||||
DOM_NOT_READY_EXIT = 3
|
||||
|
||||
# Shimmer spans (span.animate-pulse-light) exist in the settled state too
|
||||
# (2 observed — avatar/media skeleton slots, not load progress). Only a
|
||||
# mass-skeleton count indicates a stuck/broken render. Heuristic threshold
|
||||
# from docs/DOM-EDGE-STATES.md 1.1; the settled-state signature below is
|
||||
# the primary readiness signal.
|
||||
SHIMMER_STORM_THRESHOLD = 30
|
||||
|
||||
|
||||
def chat_ready_probe(ws):
|
||||
"""One-shot DOM readiness probe. Returns a dict of observed state, or
|
||||
{} if the page could not be evaluated at all."""
|
||||
result = ev1(ws, """(() => {
|
||||
const q = s => { try { return document.querySelector(s); } catch(e) { return null; } };
|
||||
const qa = s => { try { return document.querySelectorAll(s); } catch(e) { return []; } };
|
||||
const title = document.title || '';
|
||||
const url = window.location.href || '';
|
||||
const switcher = !!q('[data-testid="hatch-chat-switcher-trigger"]');
|
||||
const buttons = qa('button').length;
|
||||
const msglog = !!q('div[role="log"]');
|
||||
const composer = !!(q('[contenteditable="true"]') ||
|
||||
q('textarea[placeholder*="Message"]') ||
|
||||
q('div[role="textbox"]'));
|
||||
const shimmer = qa('span.animate-pulse-light').length;
|
||||
let alertText = '';
|
||||
try {
|
||||
alertText = [...qa('[role="alert"]')]
|
||||
.filter(e => e.tagName !== 'SCRIPT' && (e.innerText||'').trim().length > 0)
|
||||
.slice(0, 2).map(e => e.innerText.slice(0, 160)).join(' | ');
|
||||
} catch(e) {}
|
||||
let onLine = true;
|
||||
try { onLine = navigator.onLine; } catch(e) {}
|
||||
return JSON.stringify({title, url, switcher, buttons, msglog,
|
||||
composer, shimmer, alertText, onLine});
|
||||
})()""")
|
||||
try:
|
||||
return json.loads(result) if result else {}
|
||||
except Exception:
|
||||
return {}
|
||||
|
||||
|
||||
def classify_chat_state(p):
|
||||
"""Classify a readiness probe. Returns (state, diagnosis). READY is the
|
||||
only state a send may proceed from. Reference: docs/DOM-EDGE-STATES.md.
|
||||
Note: 'Chat \u2014' is the 'Chat \u2014 <identity>' title prefix."""
|
||||
if not p:
|
||||
return ("NO_PROBE",
|
||||
"CDP evaluate returned nothing - page blank, navigating, or CDP wedged")
|
||||
title = p.get("title", "")
|
||||
url = p.get("url", "")
|
||||
if title == "muse.ai":
|
||||
return ("LANDING",
|
||||
"browser is on the muse.ai landing page, not in chat (state: LANDING)")
|
||||
if not p.get("onLine", True):
|
||||
return ("OFFLINE",
|
||||
"navigator.onLine is false - browser reports no network")
|
||||
if p.get("alertText"):
|
||||
return ("REAL_ERROR",
|
||||
"page shows an error banner: %s" % p["alertText"][:160])
|
||||
if "/thread/new" in url:
|
||||
# Legitimate only right after `sidechat create`: the new-thread
|
||||
# composer is up and the title has settled. A stripped /thread/new
|
||||
# (no title, no composer) is the S8 broken state.
|
||||
if p.get("composer") and "Chat \u2014" in title:
|
||||
return ("READY", "new-thread composer state (/thread/new)")
|
||||
return ("S8_STRIPPED",
|
||||
"stripped /thread/new page - no composer/title (state: S8)")
|
||||
switcher = p.get("switcher", False)
|
||||
buttons = p.get("buttons", 0)
|
||||
if title == "Muse" and (not switcher or buttons < 30):
|
||||
return ("S3_PARTIAL",
|
||||
"React still hydrating: title 'Muse', %d buttons, switcher=%s (state: S3)"
|
||||
% (buttons, switcher))
|
||||
if not ("Chat \u2014" in title and switcher and buttons > 30 and p.get("msglog")):
|
||||
return ("NOT_SETTLED",
|
||||
"settled-state signature not met (title=%r switcher=%s buttons=%d msglog=%s)" %
|
||||
(title[:40], switcher, buttons, p.get("msglog", False)))
|
||||
if p.get("shimmer", 0) > SHIMMER_STORM_THRESHOLD:
|
||||
return ("SHIMMER_STORM",
|
||||
"mass skeleton render: %d shimmer spans (settled has ~2)" % p["shimmer"])
|
||||
if not p.get("composer"):
|
||||
return ("COMPOSER_MISSING",
|
||||
"chat settled but no composer input found - send has nowhere to type")
|
||||
return ("READY", "settled chat state (title=%r)" % title[:40])
|
||||
|
||||
|
||||
def assert_chat_ready(ws, context="send"):
|
||||
"""Fail-closed React-readiness gate ("matching chromebox").
|
||||
|
||||
Call before any DOM mutation (send, upload). On mismatch prints a
|
||||
structured diagnosis to stderr and exits DOM_NOT_READY_EXIT (3) - a
|
||||
retryable signal, distinct from APPROVAL_NEEDED (2, needs a human).
|
||||
S3 partial load gets one 5s grace re-probe (transient render phase);
|
||||
everything else fails immediately.
|
||||
"""
|
||||
probe = chat_ready_probe(ws)
|
||||
state, detail = classify_chat_state(probe)
|
||||
if state == "S3_PARTIAL":
|
||||
time.sleep(5)
|
||||
probe = chat_ready_probe(ws)
|
||||
state, detail = classify_chat_state(probe)
|
||||
if state != "READY":
|
||||
print("DOM_NOT_READY [%s] %s" % (state, detail), file=sys.stderr)
|
||||
print("DOM_NOT_READY probe: %s" % json.dumps(probe), file=sys.stderr)
|
||||
sys.exit(DOM_NOT_READY_EXIT)
|
||||
return probe
|
||||
|
||||
def cmd_approvals(ws):
|
||||
"""Check and handle pending approvals."""
|
||||
actions = check_approvals(ws)
|
||||
if not actions:
|
||||
print("No pending approvals")
|
||||
return
|
||||
for dialog, trusted, action in actions:
|
||||
print(f"Dialog: {dialog}")
|
||||
print(f" Trusted: {trusted}, Action: {action}")
|
||||
if not trusted:
|
||||
print(" APPROVAL_NEEDED: Manual review required")
|
||||
sys.exit(2)
|
||||
|
||||
def cmd_send(ws, message):
|
||||
# React-readiness gate ("matching chromebox"): fail closed if the
|
||||
# browser is not in the expected chat state (landing page,
|
||||
# partial load, partition). Exit 3 = safe to retry, unlike
|
||||
# APPROVAL_NEEDED (2, needs a human).
|
||||
assert_chat_ready(ws, context="send")
|
||||
# Check approvals
|
||||
actions = check_approvals(ws)
|
||||
for dialog, trusted, action in actions:
|
||||
if not trusted:
|
||||
print(f"APPROVAL_NEEDED: {dialog[:80]}", file=sys.stderr)
|
||||
sys.exit(2)
|
||||
|
||||
msg_esc = message.replace('\\', '\\\\').replace('`', '\\`').replace('$', '\\$')
|
||||
result = ev1(ws, f"""(async()=>{{
|
||||
const input = document.querySelector('[contenteditable="true"]') ||
|
||||
document.querySelector('textarea[placeholder*="Message"]') ||
|
||||
[...document.querySelectorAll('div[role="textbox"]')][0];
|
||||
if (!input) return 'NOINPUT';
|
||||
input.focus();
|
||||
document.execCommand('insertText', false, `{msg_esc}`);
|
||||
await new Promise(r=>setTimeout(r,500));
|
||||
const send = [...document.querySelectorAll('button')].find(b=>
|
||||
b.getAttribute('aria-label')&&b.getAttribute('aria-label').toLowerCase().includes('send')
|
||||
);
|
||||
if (send) {{ send.click(); return 'sent'; }}
|
||||
const ke = new KeyboardEvent('keydown', {{key:'Enter', code:'Enter', bubbles:true}});
|
||||
input.dispatchEvent(ke);
|
||||
return 'enter-sent';
|
||||
}})()""", True)
|
||||
print(result)
|
||||
|
||||
def cmd_messages(ws, n=5, width=200):
|
||||
# Check approvals first (non-blocking)
|
||||
check_approvals(ws)
|
||||
# Exclude the compose box subtree: a failed send leaves the draft text
|
||||
# (including the [id:...] tag) in the composer, and scraping it would
|
||||
# produce a false "verified" (2026-10-04 dm.py false-confirmation bug).
|
||||
result = ev1(ws, f"""(() => {{
|
||||
const composer = document.querySelector('[contenteditable="true"]') ||
|
||||
document.querySelector('textarea[placeholder*="Message"]');
|
||||
const ps = [...document.querySelectorAll('p')]
|
||||
.filter(p => !(composer && composer.contains(p)))
|
||||
.slice(-{n*2}).map(p=>p.innerText.slice(0,{width}));
|
||||
return ps.join('\\n---\\n');
|
||||
}})()""")
|
||||
print(result)
|
||||
|
||||
def cmd_compose_check(ws):
|
||||
"""Print the current compose-box text (empty string if clear).
|
||||
Used by dm.py to confirm a send actually left the composer."""
|
||||
result = ev1(ws, """(() => {
|
||||
const input = document.querySelector('[contenteditable="true"]') ||
|
||||
document.querySelector('textarea[placeholder*="Message"]') ||
|
||||
[...document.querySelectorAll('div[role="textbox"]')][0];
|
||||
if (!input) return 'NOCOMPOSE';
|
||||
return input.innerText || '';
|
||||
})()""")
|
||||
print(result if result is not None else '')
|
||||
|
||||
def cmd_wait(ws, timeout=30):
|
||||
print(f"Waiting {timeout}s for response...")
|
||||
# Check approvals periodically during wait
|
||||
for i in range(timeout // 5):
|
||||
actions = check_approvals(ws)
|
||||
for dialog, trusted, action in actions:
|
||||
if not trusted:
|
||||
print(f"APPROVAL_NEEDED: {dialog[:80]}", file=sys.stderr)
|
||||
sys.exit(2)
|
||||
time.sleep(5)
|
||||
cmd_messages(ws, 2)
|
||||
|
||||
def cdp_navigate(ws, url, timeout_s=30):
|
||||
"""Navigate via CDP Page.navigate (proper navigation, waits for commit).
|
||||
Returns True if the page URL matches the target after navigation."""
|
||||
import time as _time
|
||||
ws.send(json.dumps({"id": 2, "method": "Page.navigate",
|
||||
"params": {"url": url}}))
|
||||
# Drain until we get the Page.navigate response (id 2).
|
||||
for _ in range(50):
|
||||
resp = json.loads(ws.recv())
|
||||
if resp.get("id") == 2:
|
||||
break
|
||||
else:
|
||||
return False
|
||||
# Wait for the URL to settle (SPA client-side routing).
|
||||
for _ in range(timeout_s):
|
||||
cur = ev1(ws, "window.location.href", True)
|
||||
if cur and url.rstrip("/").lower() in cur.lower():
|
||||
return True
|
||||
_time.sleep(1)
|
||||
return False
|
||||
|
||||
def cmd_sidechat_use(ws, chat_id):
|
||||
"""Open a sidechat by name (sidebar text search) or by thread UUID
|
||||
(direct navigation). The sidebar shows titles, not UUIDs, so the
|
||||
text search below can never match a UUID."""
|
||||
import base64, re
|
||||
cid = chat_id.strip()
|
||||
if re.fullmatch(r"[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}", cid.lower()):
|
||||
url = "https://muse.ai/thread/" + cid.lower()
|
||||
want_uuid = cid.lower()
|
||||
# Use CDP Page.navigate (proper navigation lifecycle) and confirm
|
||||
# the URL actually changed. Hardened 2026-10-04: the React SPA
|
||||
# sometimes redirects back to / when the thread page hasn't finished
|
||||
# loading; retry instead of failing immediately.
|
||||
# (dm.py sidechat 1/3 flake; was window.location.href via evaluate.)
|
||||
cur = None
|
||||
for _try in range(5):
|
||||
ok = cdp_navigate(ws, url, timeout_s=15)
|
||||
cur = ev1(ws, "window.location.href", True)
|
||||
if ok and cur and want_uuid in cur:
|
||||
break
|
||||
# SPA dropped the nav or hasn't routed yet; wait and retry.
|
||||
time.sleep(3)
|
||||
print(f"Navigated to: {cur}")
|
||||
return
|
||||
b64 = base64.b64encode(chat_id.encode()).decode()
|
||||
js = (
|
||||
"(async()=>{"
|
||||
# Open sidebar first
|
||||
"const ob=[...document.querySelectorAll('button')].find("
|
||||
"b=>b.textContent.includes('Open chat and side chats'));"
|
||||
"if(ob)ob.click();"
|
||||
"await new Promise(r=>setTimeout(r,2000));"
|
||||
# Find and click the chat
|
||||
"const name=atob('" + b64 + "');"
|
||||
"const els=[...document.querySelectorAll('*')];"
|
||||
"const cands=els.filter(e=>e.textContent&&e.textContent.includes(name));"
|
||||
"cands.sort((a,b)=>a.textContent.length-b.textContent.length);"
|
||||
"const el=cands[0];"
|
||||
"if(!el)return 'NOTFOUND';"
|
||||
"el.click();"
|
||||
"await new Promise(r=>setTimeout(r,4000));"
|
||||
"return window.location.href;"
|
||||
"})()"
|
||||
)
|
||||
# Hardened 2026-10-04: the click sometimes doesn't navigate (React
|
||||
# mid-render, or the SPA drops it). Retry the whole nav if we didn't
|
||||
# land on a /thread/ URL.
|
||||
result = None
|
||||
for _try in range(3):
|
||||
result = ev(ws, js, True)
|
||||
if result and "/thread/" in result:
|
||||
break
|
||||
time.sleep(3)
|
||||
print(f"Navigated to: {result}")
|
||||
|
||||
def cmd_sidechat_main(ws):
|
||||
"""Navigate back to main chat via Ctrl+J keyboard shortcut."""
|
||||
# Ctrl+J (the Chat button shortcut) properly refocuses Main Chat.
|
||||
# More reliable than DOM scraping for buttons/text.
|
||||
# Do NOT navigate to https://muse.ai/ (landing page) - it restores the
|
||||
# last-viewed chat which may be a side chat.
|
||||
import json as _json
|
||||
# Send Ctrl+J via Input.dispatchKeyEvent
|
||||
ws.send(_json.dumps({
|
||||
"id": 10, "method": "Input.dispatchKeyEvent",
|
||||
"params": {"type": "keyDown", "key": "j", "code": "KeyJ",
|
||||
"ctrlKey": True, "modifiers": 2}
|
||||
}))
|
||||
_json.loads(ws.recv())
|
||||
ws.send(_json.dumps({
|
||||
"id": 11, "method": "Input.dispatchKeyEvent",
|
||||
"params": {"type": "keyUp", "key": "j", "code": "KeyJ",
|
||||
"ctrlKey": True, "modifiers": 2}
|
||||
}))
|
||||
_json.loads(ws.recv())
|
||||
def revive_browser():
|
||||
script = os.path.expanduser("~/Projects/NetVM/bin/netvm-chrome.sh")
|
||||
print("CDP disconnect detected. Reviving Chromium smoke profile...", file=sys.stderr)
|
||||
subprocess.Popen([script, "--headless", "smoke", "https://muse.ai"], stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
|
||||
time.sleep(4)
|
||||
# Get current URL to confirm navigation
|
||||
ws.send(_json.dumps({"id": 12, "method": "Runtime.evaluate",
|
||||
"params": {"expression": "window.location.href"}}))
|
||||
resp = _json.loads(ws.recv())
|
||||
url = resp.get("result", {}).get("result", {}).get("value", "unknown")
|
||||
print(f"Back to main chat: {url}")
|
||||
|
||||
def cmd_sidechat_list(ws):
|
||||
"""List side chats via DOM."""
|
||||
ev(ws, """(() => {
|
||||
const btn = [...document.querySelectorAll("button")].find(b =>
|
||||
b.textContent.includes("Open chat and side chats")
|
||||
);
|
||||
if (btn) btn.click();
|
||||
})()""")
|
||||
time.sleep(2)
|
||||
chats = ev(ws, """(() => {
|
||||
const text = document.body.innerText;
|
||||
const idx = text.indexOf("Side chats");
|
||||
if (idx === -1) return "NOSIDEBAR";
|
||||
return text.slice(idx, idx + 1000);
|
||||
})()""")
|
||||
print(chats[:500])
|
||||
|
||||
def cmd_sidechat_create(ws):
|
||||
"""Create a new side chat via the '+' button.
|
||||
Returns the new thread URL.
|
||||
Selectors (verified 2026-10-04 via DOM investigation):
|
||||
- Panel opener: [data-testid="hatch-chat-switcher-trigger"] (idempotent)
|
||||
- + button: [data-testid="hatch-chat-compose"] (SVG icon, no text)
|
||||
"""
|
||||
import time as _time
|
||||
import json as _json
|
||||
import sys as _sys2
|
||||
# Ensure chat is active via DOM clicks (replaces Ctrl+J).
|
||||
# Ctrl+J needs keyboard focus which stripped states (/thread/new) lack.
|
||||
# Priority: compose (fast path) -> switcher -> nav-chat (recovery).
|
||||
# Verified 2026-10-04: hatch-nav-chat click recovers from /thread/new.
|
||||
def _ensure_chat_active(timeout=20):
|
||||
t0 = _time.time()
|
||||
while _time.time() - t0 < timeout:
|
||||
st = ev(ws, """(() => ({
|
||||
compose: !!document.querySelector('[data-testid="hatch-chat-compose"]'),
|
||||
switcher: !!document.querySelector('[data-testid="hatch-chat-switcher-trigger"]'),
|
||||
navchat: !!document.querySelector('[data-testid="hatch-nav-chat"]')
|
||||
}))()""")
|
||||
if not isinstance(st, dict):
|
||||
_time.sleep(1)
|
||||
continue
|
||||
if st.get("compose"):
|
||||
return True
|
||||
if st.get("switcher"):
|
||||
ev(ws, """document.querySelector('[data-testid="hatch-chat-switcher-trigger"]').click()""")
|
||||
_time.sleep(1.5)
|
||||
continue
|
||||
if st.get("navchat"):
|
||||
ev(ws, """document.querySelector('[data-testid="hatch-nav-chat"]').click()""")
|
||||
_time.sleep(2)
|
||||
continue
|
||||
_time.sleep(1)
|
||||
return False
|
||||
if not _ensure_chat_active():
|
||||
print("ERROR: Could not activate chat (no compose/switcher/nav-chat)", file=_sys2.stderr)
|
||||
sys.exit(1)
|
||||
print("Chat active", file=_sys2.stderr)
|
||||
# Find + button via data-testid (verified selector, SVG icon no text)
|
||||
result = ev(ws, """(() => {
|
||||
const btn = document.querySelector('[data-testid="hatch-chat-compose"]');
|
||||
if (!btn) return 'NOT_FOUND';
|
||||
btn.click();
|
||||
return 'CLICKED';
|
||||
})()""")
|
||||
if result == 'NOT_FOUND':
|
||||
print("ERROR: + button [data-testid=hatch-chat-compose] not found", file=sys.stderr)
|
||||
sys.exit(1)
|
||||
# Log which strategy worked for debugging
|
||||
import sys as _sys
|
||||
print(f"Sidechat create: {result}", file=_sys.stderr)
|
||||
import time as _time
|
||||
# Poll for URL to change to a thread URL (up to 15s)
|
||||
# Fixed 2026-10-04: was sleeping 5s and reading once, often captured
|
||||
# base URL before navigation settled.
|
||||
url = None
|
||||
for i in range(15):
|
||||
_time.sleep(1)
|
||||
url = ev1(ws, "window.location.href")
|
||||
if url and "/thread/" in url:
|
||||
break
|
||||
if not url or "/thread/" not in url:
|
||||
print(f"ERROR: Sidechat creation did not navigate to thread URL (got: {url})", file=sys.stderr)
|
||||
sys.exit(1)
|
||||
# Return the URL (may be /thread/new placeholder).
|
||||
# The caller sends directly to current chat via cmd_send (no ID needed).
|
||||
# Thread ID can be looked up later via URL polling if needed.
|
||||
# Fixed 2026-10-04: don't chase real ID at create time, just create.
|
||||
print(f"Created: {url}")
|
||||
return url
|
||||
|
||||
|
||||
|
||||
|
||||
def cdp_call(ws, method, params=None):
|
||||
"""Raw CDP method call for non-Runtime domains (DOM, Page, ...).
|
||||
The ev() helper only speaks Runtime.evaluate; uploads need DOM.
|
||||
Skips CDP event chatter (messages without our id) while waiting."""
|
||||
cdp_call.counter += 1
|
||||
cid = cdp_call.counter
|
||||
ws.send(json.dumps({"id": cid, "method": method,
|
||||
"params": params or {}}))
|
||||
for _ in range(20):
|
||||
resp = json.loads(ws.recv())
|
||||
if resp.get("id") != cid:
|
||||
continue # event, not our response
|
||||
if "error" in resp:
|
||||
raise RuntimeError("CDP %s failed: %s" % (method, resp["error"]))
|
||||
return resp.get("result", {})
|
||||
raise RuntimeError("CDP %s: no response after 20 reads" % method)
|
||||
|
||||
|
||||
cdp_call.counter = 100
|
||||
|
||||
|
||||
def ev1(ws, expr, await_p=False):
|
||||
"""Runtime.evaluate that skips CDP event chatter while awaiting its
|
||||
response. ev() reads a single message and can catch an event
|
||||
instead (the known None-result quirk); uploads do several DOM
|
||||
calls first, so chatter is likely."""
|
||||
ws.send(json.dumps({
|
||||
"id": 1, "method": "Runtime.evaluate",
|
||||
"params": {"expression": expr, "returnByValue": True,
|
||||
"awaitPromise": await_p}
|
||||
}))
|
||||
for _ in range(30):
|
||||
resp = json.loads(ws.recv())
|
||||
if resp.get("id") != 1:
|
||||
continue
|
||||
return resp.get("result", {}).get("result", {}).get("value")
|
||||
return None
|
||||
|
||||
|
||||
def cmd_url(ws):
|
||||
"""Print current browser URL."""
|
||||
url = ev1(ws, "window.location.href")
|
||||
print(url)
|
||||
|
||||
def cmd_upload(ws, filepath, message=None, dry_run=False):
|
||||
"""Attach a file to the chat composer via CDP DOM.setFileInputFiles.
|
||||
|
||||
Dry-run stages the attachment and verifies the preview chip without
|
||||
sending. Otherwise sends (with optional caption) and verifies via
|
||||
read-back — never trust the CDP result alone.
|
||||
"""
|
||||
import os
|
||||
if not os.path.isfile(filepath):
|
||||
print("ERROR: file not found: %s" % filepath, file=sys.stderr)
|
||||
sys.exit(1)
|
||||
# React-readiness gate ("matching chromebox") - see cmd_send.
|
||||
assert_chat_ready(ws, context="upload")
|
||||
actions = check_approvals(ws)
|
||||
for dialog, trusted, action in actions:
|
||||
if not trusted:
|
||||
print("APPROVAL_NEEDED: %s" % dialog[:80], file=sys.stderr)
|
||||
sys.exit(2)
|
||||
# The file input may only render after the attach button is clicked.
|
||||
node_id = 0
|
||||
for _ in range(2):
|
||||
doc = cdp_call(ws, "DOM.getDocument")
|
||||
root = doc["root"]["nodeId"]
|
||||
q = cdp_call(ws, "DOM.querySelector",
|
||||
{"nodeId": root, "selector": "input[type=file]"})
|
||||
node_id = q.get("nodeId", 0)
|
||||
if node_id:
|
||||
break
|
||||
ev(ws, """(() => {
|
||||
const b = [...document.querySelectorAll('button')].find(x =>
|
||||
(x.getAttribute('aria-label')||'').toLowerCase().includes('attach'));
|
||||
if (b) { b.click(); return 'clicked'; }
|
||||
return 'notfound';
|
||||
})()""")
|
||||
time.sleep(2)
|
||||
if not node_id:
|
||||
print("ERROR: no file input in composer", file=sys.stderr)
|
||||
sys.exit(1)
|
||||
cdp_call(ws, "DOM.setFileInputFiles",
|
||||
{"nodeId": node_id, "files": [os.path.abspath(filepath)]})
|
||||
time.sleep(2)
|
||||
# Read-back: the attachment must be staged. React removes the
|
||||
# file input after change, so input.files is unreliable — the
|
||||
# composer's own signal is the "Remove attachment" button plus the
|
||||
# chip text beside it.
|
||||
base = os.path.basename(filepath)
|
||||
preview = ev1(ws, """(() => {
|
||||
const btns = [...document.querySelectorAll('button')].filter(b =>
|
||||
(b.getAttribute('aria-label') || '') === 'Remove attachment');
|
||||
if (!btns.length) return JSON.stringify([]);
|
||||
const chip = btns[0].closest('div');
|
||||
const text = chip ? chip.innerText.slice(0, 120) : '';
|
||||
return JSON.stringify([{button: true, chip: text}]);
|
||||
})()""")
|
||||
print("preview: %s" % preview)
|
||||
try:
|
||||
hits = json.loads(preview) if preview else []
|
||||
except Exception:
|
||||
hits = []
|
||||
if not hits:
|
||||
print("ERROR: attachment not staged (no Remove-attachment button)",
|
||||
file=sys.stderr)
|
||||
sys.exit(1)
|
||||
if dry_run:
|
||||
print("DRY-RUN OK: '%s' staged in composer, not sent." % base)
|
||||
return
|
||||
if message:
|
||||
msg_esc = message.replace('\\', '\\\\').replace('`', '\\`').replace('$', '\\$')
|
||||
ev(ws, """(async()=>{
|
||||
const input = document.querySelector('[contenteditable="true"]') ||
|
||||
document.querySelector('textarea[placeholder*="Message"]') ||
|
||||
[...document.querySelectorAll('div[role="textbox"]')][0];
|
||||
if (!input) return 'NOINPUT';
|
||||
input.focus();
|
||||
document.execCommand('insertText', false, `%s`);
|
||||
return 'caption-set';
|
||||
})()""" % msg_esc, True)
|
||||
time.sleep(1)
|
||||
sent = ev(ws, """(async()=>{
|
||||
const send = [...document.querySelectorAll('button')].find(b =>
|
||||
b.getAttribute('aria-label') &&
|
||||
b.getAttribute('aria-label').toLowerCase().includes('send'));
|
||||
if (send) { send.click(); return 'sent'; }
|
||||
return 'NOSEND';
|
||||
})()""", True)
|
||||
print("send: %s" % sent)
|
||||
time.sleep(5)
|
||||
recent = ev1(ws, """(() => {
|
||||
const ps = [...document.querySelectorAll('p')].slice(-10)
|
||||
.map(p => p.innerText.slice(0,200));
|
||||
return ps.join('\\n---\\n');
|
||||
})()""")
|
||||
if base in (recent or ''):
|
||||
print("VERIFIED: attachment '%s' present in chat." % base)
|
||||
else:
|
||||
print("WARNING: attachment not found in read-back; "
|
||||
"check the composer manually.")
|
||||
|
||||
def main():
|
||||
p = argparse.ArgumentParser()
|
||||
p.add_argument('--account', required=True, choices=list(ACCOUNTS.keys()),
|
||||
help='Agent name (matches node, profile, ACCOUNTS.md)')
|
||||
p.add_argument('command', choices=['send', 'messages', 'wait', 'approvals', 'sidechat', 'upload', 'url', 'compose_check'])
|
||||
p.add_argument('arg', nargs='*', default=[])
|
||||
p.add_argument('--dry-run', action='store_true',
|
||||
help='upload: stage attachment without sending')
|
||||
p.add_argument('--message', default=None,
|
||||
help='upload: caption text sent with the file')
|
||||
args = p.parse_args()
|
||||
|
||||
node, cdp_url = ACCOUNTS[args.account]
|
||||
page = get_page(node, cdp_url)
|
||||
|
||||
# CDP operation queue: serialize browser access across processes.
|
||||
# Priority from CDP_PRIORITY env (high/normal/low); default normal.
|
||||
# Falls back to unqueued if cdp_queue is unavailable.
|
||||
import contextlib
|
||||
if HAS_CDP_QUEUE:
|
||||
_prio_name = os.environ.get("CDP_PRIORITY", "normal").lower()
|
||||
_prio = {"high": PRIORITY_HIGH, "low": PRIORITY_LOW}.get(
|
||||
_prio_name, PRIORITY_NORMAL)
|
||||
_slot = cdp_slot(node, priority=_prio)
|
||||
else:
|
||||
_slot = contextlib.nullcontext()
|
||||
with _slot:
|
||||
ws = websocket.create_connection(page['webSocketDebuggerUrl'], timeout=15)
|
||||
|
||||
def connect(retries=2):
|
||||
for attempt in range(retries + 1):
|
||||
try:
|
||||
if args.command == 'send':
|
||||
if not args.arg:
|
||||
print("ERROR: send requires a message", file=sys.stderr)
|
||||
sys.exit(1)
|
||||
cmd_send(ws, " ".join(args.arg))
|
||||
elif args.command == 'messages':
|
||||
n = int(args.arg[0]) if args.arg else 5
|
||||
w = int(args.arg[1]) if len(args.arg) > 1 else 200
|
||||
cmd_messages(ws, n, w)
|
||||
elif args.command == 'compose_check':
|
||||
cmd_compose_check(ws)
|
||||
elif args.command == 'wait':
|
||||
t = int(args.arg[0]) if args.arg else 30
|
||||
cmd_wait(ws, t)
|
||||
elif args.command == 'approvals':
|
||||
cmd_approvals(ws)
|
||||
elif args.command == 'sidechat':
|
||||
if not args.arg:
|
||||
print("ERROR: sidechat requires subcommand (use|main|list|create)", file=sys.stderr)
|
||||
sys.exit(1)
|
||||
sub = args.arg[0]
|
||||
if sub == "create":
|
||||
cmd_sidechat_create(ws)
|
||||
elif sub == "use":
|
||||
if len(args.arg) < 2:
|
||||
print("ERROR: sidechat use requires chat_id", file=sys.stderr)
|
||||
sys.exit(1)
|
||||
cmd_sidechat_use(ws, args.arg[1])
|
||||
elif sub == "main":
|
||||
cmd_sidechat_main(ws)
|
||||
elif sub == "list":
|
||||
cmd_sidechat_list(ws)
|
||||
else:
|
||||
print(f"ERROR: unknown sidechat subcommand: {sub}", file=sys.stderr)
|
||||
sys.exit(1)
|
||||
elif args.command == 'url':
|
||||
cmd_url(ws)
|
||||
elif args.command == 'upload':
|
||||
if not args.arg:
|
||||
print("ERROR: upload requires a filepath", file=sys.stderr)
|
||||
sys.exit(1)
|
||||
cmd_upload(ws, args.arg[0], message=args.message,
|
||||
dry_run=args.dry_run)
|
||||
finally:
|
||||
ws.close()
|
||||
page = get_page()
|
||||
ws = websocket.create_connection(page['webSocketDebuggerUrl'], timeout=10)
|
||||
return ws
|
||||
except Exception as e:
|
||||
if attempt < retries:
|
||||
revive_browser()
|
||||
else:
|
||||
raise RuntimeError(f"Failed to connect to CDP after {retries} retries: {e}")
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
|
||||
def ev(ws, expr, await_promise=False):
|
||||
ws.send(json.dumps({"id":1,"method":"Runtime.evaluate",
|
||||
"params":{"expression":expr,"returnByValue":True,"awaitPromise":await_promise}}))
|
||||
r = json.loads(ws.recv())
|
||||
return r.get('result',{}).get('result',{}).get('value')
|
||||
|
||||
def send_message(text):
|
||||
ws = connect()
|
||||
try:
|
||||
result = ev(ws, f"""
|
||||
(async () => {{
|
||||
const ta = document.querySelector('textarea[aria-label="Message"]');
|
||||
if (!ta) return 'ERROR: no composer';
|
||||
ta.focus();
|
||||
document.execCommand('insertText', false, {json.dumps(text)});
|
||||
await new Promise(r => setTimeout(r, 200));
|
||||
ta.dispatchEvent(new KeyboardEvent('keydown', {{key:'Enter', code:'Enter', keyCode:13, bubbles:true}}));
|
||||
return 'sent';
|
||||
}})()
|
||||
""", await_promise=True)
|
||||
return result
|
||||
finally:
|
||||
ws.close()
|
||||
|
||||
def get_messages(limit=10):
|
||||
ws = connect()
|
||||
try:
|
||||
return ev(ws, f"""
|
||||
Array.from(document.querySelectorAll('p')).slice(-{limit}).map(e=>e.innerText).join('\\n---\\n')
|
||||
""")
|
||||
finally:
|
||||
ws.close()
|
||||
|
||||
def navigate(url):
|
||||
"""Navigate to a URL (e.g., a thread: https://muse.ai/thread/<id>)."""
|
||||
ws = connect()
|
||||
try:
|
||||
ws.send(json.dumps({"id":1,"method":"Page.navigate","params":{"url":url}}))
|
||||
r = json.loads(ws.recv())
|
||||
return r.get('result',{}).get('frameId','navigated')
|
||||
finally:
|
||||
ws.close()
|
||||
|
||||
def wait_for_response(timeout=60, poll=2):
|
||||
"""Wait for a new assistant message after sending."""
|
||||
ws = connect()
|
||||
try:
|
||||
before = ev(ws, "document.body.innerText.length")
|
||||
start = time.time()
|
||||
while time.time() - start < timeout:
|
||||
time.sleep(poll)
|
||||
# Check if there's a "stop" button (indicates generating)
|
||||
generating = ev(ws, """
|
||||
!!document.querySelector('button[aria-label*="Stop"], button[aria-label*="stop"]')
|
||||
""")
|
||||
if not generating:
|
||||
# Give it a moment to finish rendering
|
||||
time.sleep(2)
|
||||
return get_messages(5)
|
||||
return "TIMEOUT waiting for response"
|
||||
finally:
|
||||
ws.close()
|
||||
|
||||
if __name__ == "__main__":
|
||||
if len(sys.argv) < 2:
|
||||
print(__doc__)
|
||||
sys.exit(1)
|
||||
cmd = sys.argv[1]
|
||||
if cmd == "send" and len(sys.argv) > 2:
|
||||
print(send_message(sys.argv[2]))
|
||||
elif cmd == "messages":
|
||||
print(get_messages(int(sys.argv[2]) if len(sys.argv)>2 else 10))
|
||||
elif cmd == "navigate" and len(sys.argv) > 2:
|
||||
print(navigate(sys.argv[2]))
|
||||
elif cmd == "wait":
|
||||
print(wait_for_response(int(sys.argv[2]) if len(sys.argv)>2 else 60))
|
||||
else:
|
||||
print("unknown command")
|
||||
|
||||
@@ -1,194 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
muse-chat-repl.py: Fast interactive terminal chat session for a NetVM muse agent.
|
||||
Uses direct gateway calls via muse-cli-node in the agent's netns.
|
||||
"""
|
||||
import sys
|
||||
import os
|
||||
import json
|
||||
import time
|
||||
import subprocess
|
||||
from pathlib import Path
|
||||
|
||||
NETVM_BIN = Path(__file__).resolve().parent
|
||||
|
||||
# Color formatting helpers
|
||||
def c_bold(s): return f"\033[1m{s}\033[0m"
|
||||
def c_cyan(s): return f"\033[36m{s}\033[0m"
|
||||
def c_green(s): return f"\033[32m{s}\033[0m"
|
||||
def c_yellow(s): return f"\033[33m{s}\033[0m"
|
||||
def c_red(s): return f"\033[31m{s}\033[0m"
|
||||
def c_dim(s): return f"\033[2m{s}\033[0m"
|
||||
|
||||
def run_muse(node, *args):
|
||||
cmd = [str(NETVM_BIN / "muse-cli-node"), node] + list(args)
|
||||
res = subprocess.run(cmd, capture_output=True, text=True)
|
||||
return res.returncode, res.stdout, res.stderr
|
||||
|
||||
def get_threads(node):
|
||||
rc, out, _ = run_muse(node, "threads")
|
||||
if rc != 0:
|
||||
return []
|
||||
try:
|
||||
data = json.loads(out)
|
||||
if isinstance(data, list):
|
||||
return data
|
||||
except Exception:
|
||||
pass
|
||||
return []
|
||||
|
||||
def select_thread(node):
|
||||
print(c_bold(f"\n=== SELECT THREAD FOR {node.upper()} ===") + c_dim(" (Prefer sidechats per CHAT_POLICY.md)\n"))
|
||||
threads = get_threads(node)
|
||||
|
||||
# Filter and prioritize sidechats
|
||||
top_threads = threads[:15]
|
||||
for idx, t in enumerate(top_threads, 1):
|
||||
title = t.get("title") or "[Untitled]"
|
||||
sid = t.get("session_id")
|
||||
updated = t.get("updated", "")
|
||||
print(f" [{idx:2d}] {c_bold(title[:40]):<42} {c_dim(sid[:8])} {c_dim(updated)}")
|
||||
|
||||
print(f"\n [n] {c_green('+ Create a new sidechat')}")
|
||||
print(f" [c] {c_cyan('Custom thread UUID')}")
|
||||
print(f" [q] Quit\n")
|
||||
|
||||
while True:
|
||||
try:
|
||||
choice = input(f"Choose option [1-{len(top_threads)}, n, c, q]: ").strip().lower()
|
||||
if not choice:
|
||||
continue
|
||||
if choice == "q":
|
||||
sys.exit(0)
|
||||
if choice == "n":
|
||||
title = input("Enter title for new sidechat: ").strip()
|
||||
if not title:
|
||||
title = "operator-session"
|
||||
print(c_dim(f"Creating sidechat '{title}'..."))
|
||||
rc, out, err = run_muse(node, "session-start", "--title", title)
|
||||
if rc == 0:
|
||||
try:
|
||||
res = json.loads(out)
|
||||
tid = res.get("session_id")
|
||||
if tid:
|
||||
print(c_green(f"✔ Created session {tid}"))
|
||||
return tid, title
|
||||
except Exception:
|
||||
pass
|
||||
print(c_red(f"Failed to create session: {err or out}"))
|
||||
continue
|
||||
if choice == "c":
|
||||
tid = input("Enter thread UUID: ").strip()
|
||||
if tid:
|
||||
return tid, tid[:8]
|
||||
continue
|
||||
if choice.isdigit():
|
||||
num = int(choice)
|
||||
if 1 <= num <= len(top_threads):
|
||||
t = top_threads[num - 1]
|
||||
return t["session_id"], t.get("title") or t["session_id"][:8]
|
||||
except (KeyboardInterrupt, EOFError):
|
||||
print()
|
||||
sys.exit(0)
|
||||
|
||||
def render_history(node, thread_id, limit=5):
|
||||
rc, out, _ = run_muse(node, "history", "--thread", thread_id, "--limit", str(limit))
|
||||
if rc != 0:
|
||||
return []
|
||||
try:
|
||||
msgs = json.loads(out)
|
||||
if isinstance(msgs, list):
|
||||
print(c_dim(f"\n--- Recent Messages ({len(msgs)}) ---"))
|
||||
for m in msgs:
|
||||
role = m.get("role", "")
|
||||
text = m.get("text", "")
|
||||
if role == "assistant":
|
||||
print(f"{c_yellow(node.upper())}: {text}")
|
||||
else:
|
||||
print(f"{c_green('YOU')}: {text}")
|
||||
print(c_dim("----------------------------\n"))
|
||||
return msgs
|
||||
except Exception:
|
||||
pass
|
||||
return []
|
||||
|
||||
def main():
|
||||
if len(sys.argv) < 2:
|
||||
print("Usage: muse-chat-repl.py <account> [--thread <uuid>]", file=sys.stderr)
|
||||
sys.exit(1)
|
||||
|
||||
node = sys.argv[1]
|
||||
thread_id = None
|
||||
thread_title = None
|
||||
|
||||
if len(sys.argv) >= 4 and sys.argv[2] in ("--thread", "-t"):
|
||||
thread_id = sys.argv[3]
|
||||
thread_title = thread_id[:8]
|
||||
|
||||
if not thread_id:
|
||||
thread_id, thread_title = select_thread(node)
|
||||
|
||||
print(c_bold(f"\nEntering chat with {c_cyan(node.upper())} in thread '{thread_title}' ({thread_id[:8]}...)"))
|
||||
print(c_dim("Commands: /exit (or /quit), /refresh, /history, /switch\n"))
|
||||
|
||||
last_msgs = render_history(node, thread_id, limit=5)
|
||||
last_seq = last_msgs[-1].get("seq", 0) if last_msgs else 0
|
||||
|
||||
while True:
|
||||
try:
|
||||
prompt_str = f"{c_green('super')} ({c_cyan(thread_title[:16])}) > "
|
||||
line = input(prompt_str).strip()
|
||||
if not line:
|
||||
continue
|
||||
|
||||
if line in ("/exit", "/quit", ":q"):
|
||||
print(c_dim("\nExited chat session.\n"))
|
||||
break
|
||||
elif line in ("/refresh", "/history"):
|
||||
last_msgs = render_history(node, thread_id, limit=10)
|
||||
continue
|
||||
elif line == "/switch":
|
||||
thread_id, thread_title = select_thread(node)
|
||||
print(c_bold(f"\nSwitched to thread '{thread_title}' ({thread_id[:8]}...)\n"))
|
||||
render_history(node, thread_id, limit=5)
|
||||
continue
|
||||
|
||||
# Send message
|
||||
print(c_dim("Sending..."), end="\r", flush=True)
|
||||
rc, out, err = run_muse(node, "send", "--thread", thread_id, line)
|
||||
if rc != 0:
|
||||
print(c_red(f"Error sending message: {err or out}"))
|
||||
continue
|
||||
|
||||
print(c_green("Sent. Waiting for response..."), end="\r", flush=True)
|
||||
|
||||
# Poll for assistant reply up to 30 seconds
|
||||
start_time = time.time()
|
||||
got_reply = False
|
||||
while time.time() - start_time < 30:
|
||||
time.sleep(2)
|
||||
rc, out, _ = run_muse(node, "history", "--thread", thread_id, "--limit", "3")
|
||||
if rc == 0:
|
||||
try:
|
||||
msgs = json.loads(out)
|
||||
if msgs and msgs[-1].get("role") == "assistant":
|
||||
# Check if newer than our sent context
|
||||
if msgs[-1].get("seq", 0) > last_seq:
|
||||
print(" " * 50, end="\r") # clear line
|
||||
print(f"{c_yellow(node.upper())}: {msgs[-1].get('text')}\n")
|
||||
last_seq = msgs[-1].get("seq", 0)
|
||||
got_reply = True
|
||||
break
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
if not got_reply:
|
||||
print(" " * 50, end="\r")
|
||||
print(c_dim("(Agent is still processing in background. Use /refresh to inspect)\n"))
|
||||
|
||||
except (KeyboardInterrupt, EOFError):
|
||||
print(c_dim("\nExited chat session.\n"))
|
||||
break
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -1,20 +0,0 @@
|
||||
import os
|
||||
import sys
|
||||
|
||||
node = sys.argv[1]
|
||||
conf_dir = os.path.expanduser(f"~/.config/muse-cli/{node}")
|
||||
os.environ["MUSE_CONFIG_DIR"] = conf_dir
|
||||
|
||||
sys.path.insert(0, os.path.expanduser("~/.local/lib/python3.14/site-packages"))
|
||||
import muse_cli.cli as cli
|
||||
|
||||
cli.CONFIG_DIR = conf_dir
|
||||
cli.CONFIG_FILE = os.path.join(conf_dir, "config.json")
|
||||
cli.COOKIES_FILE = os.path.join(conf_dir, "cookies.txt")
|
||||
|
||||
sys.argv = [sys.argv[0]] + sys.argv[2:]
|
||||
|
||||
try:
|
||||
cli.main()
|
||||
except SystemExit as e:
|
||||
sys.exit(e.code)
|
||||
@@ -1,47 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
# muse-cli-node <node> <muse-cli-args...>
|
||||
# Runs muse-cli inside the node's isolated network namespace with its own
|
||||
# dedicated Cloudflare WARP egress identity and isolated cookies/config.
|
||||
# Auto-refreshes cookies via Chromium CDP if authentication fails.
|
||||
set -euo pipefail
|
||||
|
||||
NETVM_BIN="$(cd "$(dirname "$0")" && pwd)"
|
||||
NODE="${1:?usage: muse-cli-node <node> [args...]}"
|
||||
shift
|
||||
|
||||
CONF_DIR="$HOME/.config/muse-cli/$NODE"
|
||||
mkdir -p "$CONF_DIR"
|
||||
|
||||
ensure_node_up() {
|
||||
if ! ip netns list 2>/dev/null | grep -q "^warp-${NODE}\b"; then
|
||||
echo "Notice: warp-${NODE} network namespace is down; bringing up via netvm-node-up.sh..." >&2
|
||||
sudo -n "$NETVM_BIN/netvm-node-up.sh" "$NODE" >/dev/null 2>&1 || true
|
||||
fi
|
||||
}
|
||||
|
||||
ensure_node_up
|
||||
|
||||
run_cli() {
|
||||
export MUSE_CONFIG_DIR="$CONF_DIR"
|
||||
"$NETVM_BIN/netvm-exec.sh" "$NODE" -- python3 "$NETVM_BIN/muse-cli-inner.py" "$NODE" "$@"
|
||||
}
|
||||
|
||||
# First attempt
|
||||
set +e
|
||||
OUTPUT=$(run_cli "$@" 2>&1)
|
||||
RC=$?
|
||||
set -e
|
||||
|
||||
# Detect authentication failure and trigger single auto-refresh retry
|
||||
if [ $RC -ne 0 ] && echo "$OUTPUT" | grep -qiE "AuthError|hatch_sess|401|unauthorized"; then
|
||||
echo "Notice: Auth failure detected for $NODE; refreshing cookies from Chromium CDP..." >&2
|
||||
if "$NETVM_BIN/netvm-exec.sh" "$NODE" -- python3 "$NETVM_BIN/refresh-node-cookies.py" "$NODE"; then
|
||||
echo "Notice: Cookies refreshed for $NODE. Retrying operation..." >&2
|
||||
run_cli "$@"
|
||||
exit $?
|
||||
fi
|
||||
fi
|
||||
|
||||
# If succeeded or failed on something else, emit output and exit with RC
|
||||
echo "$OUTPUT"
|
||||
exit $RC
|
||||
@@ -1,278 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Automated muse.ai sign-in with in-browser OTP approval.
|
||||
|
||||
Usage:
|
||||
signin.py --email <email> [--otp <code>]
|
||||
|
||||
If --otp is not provided, the script will:
|
||||
1. Navigate through login flow up to OTP prompt
|
||||
2. Print "APPROVAL_NEEDED: OTP required for [redacted]"
|
||||
3. Exit with code 2
|
||||
|
||||
The operator then obtains the OTP (via chat with user) and re-runs:
|
||||
signin.py --email <email> --otp <code>
|
||||
|
||||
This keeps credentials out of the automation — the OTP is provided
|
||||
transiently via the operator, never stored.
|
||||
|
||||
Exit codes:
|
||||
0: Success / already logged in
|
||||
2: APPROVAL_NEEDED (OTP prompt reached, awaiting code)
|
||||
3: NEEDS_HUMAN (multi-account selection needs manual choice)
|
||||
4: NEEDS_SIGNUP (account not found / registration required)
|
||||
1: General error
|
||||
"""
|
||||
import json, urllib.request, websocket, time, sys, argparse, importlib.util
|
||||
|
||||
CDP_PORT_DEFAULT = 9410
|
||||
|
||||
|
||||
def _registry_port(node):
|
||||
try:
|
||||
path = "/home/super/Projects/NetVM/bin/netvm-registry.py"
|
||||
spec = importlib.util.spec_from_file_location("netvm_registry", path)
|
||||
mod = importlib.util.module_from_spec(spec)
|
||||
spec.loader.exec_module(mod)
|
||||
return mod.port_for(node)
|
||||
except Exception:
|
||||
return None
|
||||
|
||||
|
||||
def is_registered_in_accounts(email):
|
||||
path = "/home/super/Projects/NetVM/ACCOUNTS.md"
|
||||
try:
|
||||
with open(path) as f:
|
||||
for line in f:
|
||||
if line.startswith("|") and email.lower() in line.lower():
|
||||
return True
|
||||
except Exception:
|
||||
pass
|
||||
return False
|
||||
|
||||
|
||||
def get_page(port):
|
||||
cdp_url = "http://127.0.0.1:%s/json/list" % port
|
||||
with urllib.request.urlopen(cdp_url, timeout=5) as r:
|
||||
ts = json.load(r)
|
||||
pages = [t for t in ts if t.get('type') == 'page']
|
||||
if not pages:
|
||||
print("ERROR: No page found", file=sys.stderr)
|
||||
sys.exit(1)
|
||||
return pages[0]
|
||||
|
||||
def ev(ws, expr, await_p=False):
|
||||
ws.send(json.dumps({
|
||||
"id": 1, "method": "Runtime.evaluate",
|
||||
"params": {"expression": expr, "returnByValue": True, "awaitPromise": await_p}
|
||||
}))
|
||||
resp = json.loads(ws.recv())
|
||||
return resp.get('result', {}).get('result', {}).get('value')
|
||||
|
||||
def main():
|
||||
p = argparse.ArgumentParser()
|
||||
p.add_argument('--email', required=True)
|
||||
p.add_argument('--otp', default=None)
|
||||
p.add_argument('--node', default=None,
|
||||
help='node name: CDP port comes from the NODES.md registry')
|
||||
p.add_argument('--cdp-port', default=None,
|
||||
help='explicit CDP port (overrides --node lookup)')
|
||||
p.add_argument('--account-name', default=None,
|
||||
help='display-name hint for Meta multi-account selection '
|
||||
'(never the "+1" entry)')
|
||||
args = p.parse_args()
|
||||
|
||||
if is_registered_in_accounts(args.email):
|
||||
print("Registry check: [redacted] is registered in ACCOUNTS.md")
|
||||
|
||||
if args.cdp_port:
|
||||
port = args.cdp_port
|
||||
elif args.node:
|
||||
port = _registry_port(args.node) or CDP_PORT_DEFAULT
|
||||
else:
|
||||
port = CDP_PORT_DEFAULT
|
||||
|
||||
page = get_page(port)
|
||||
ws = websocket.create_connection(page['webSocketDebuggerUrl'], timeout=15)
|
||||
|
||||
# Step 1: Check if already logged in or blocked at verification gate
|
||||
title = ev(ws, "document.title")
|
||||
body = ev(ws, "document.body.innerText.slice(0,500)")
|
||||
url = ev(ws, "location.href") or ""
|
||||
if "access/verification" in url or "confirm your age" in body.lower():
|
||||
ig_link = ev(ws, """(async()=>{
|
||||
try {
|
||||
const r = await fetch('/api/hatch/age-confirmation/linking-web-auth?account_type=instagram', {
|
||||
headers: {'Accept': 'application/json'}
|
||||
});
|
||||
const j = await r.json();
|
||||
return j && j.url ? j.url : null;
|
||||
} catch(e) { return null; }
|
||||
})()""", True)
|
||||
if ig_link:
|
||||
print(f"LINK_INSTAGRAM_URL: {ig_link}")
|
||||
print(f"NEEDS_HUMAN: Age verification required for {args.email} at {url}. Client intervention required to confirm age or link Instagram/Facebook.", file=sys.stderr)
|
||||
ws.close()
|
||||
sys.exit(3)
|
||||
|
||||
if ("Connected" in body or "Chats" in body) and "Enter your code" not in body and "access/verification" not in url:
|
||||
print(f"Already logged in (title: {title})")
|
||||
ws.close()
|
||||
return 0
|
||||
|
||||
already_on_otp = ("Enter your code" in body or "code we sent" in body) and args.email in body
|
||||
if not already_on_otp:
|
||||
# Step 2: Click Log in (if on landing page)
|
||||
if "Log in" in body:
|
||||
print("Clicking Log in...")
|
||||
ev(ws, """(async()=>{
|
||||
const b=[...document.querySelectorAll('button')].find(x=>x.innerText.includes('Log in'));
|
||||
if(b) b.click(); return !!b;
|
||||
})()""", True)
|
||||
time.sleep(3)
|
||||
|
||||
# Step 3: Enter email
|
||||
print("Entering email: [redacted]")
|
||||
result = ev(ws, f"""(async()=>{{
|
||||
const inp=[...document.querySelectorAll('input')].find(i=>
|
||||
(i.placeholder&&i.placeholder.toLowerCase().includes('email'))||
|
||||
(i.getAttribute('aria-label')&&i.getAttribute('aria-label').toLowerCase().includes('email'))
|
||||
);
|
||||
if(!inp) return 'NOINPUT';
|
||||
inp.focus();
|
||||
document.execCommand('insertText',false,'{args.email}');
|
||||
await new Promise(r=>setTimeout(r,500));
|
||||
return 'entered:'+inp.value;
|
||||
}})()""", True)
|
||||
print("Email: [redacted]")
|
||||
if result == 'NOINPUT':
|
||||
print("ERROR: Email input not found", file=sys.stderr)
|
||||
ws.close()
|
||||
sys.exit(1)
|
||||
time.sleep(1)
|
||||
|
||||
# Step 4: Click Continue
|
||||
print("Clicking Continue...")
|
||||
ev(ws, """(async()=>{
|
||||
const b=[...document.querySelectorAll('button')].find(x=>x.innerText.includes('Continue'));
|
||||
if(b) b.click(); return !!b;
|
||||
})()""", True)
|
||||
time.sleep(4)
|
||||
else:
|
||||
print("Page is already on OTP prompt for this email, skipping email entry.")
|
||||
|
||||
# Step 5: Check for OTP prompt
|
||||
body = ev(ws, "document.body.innerText.slice(0,500)")
|
||||
if "Enter your code" in body or "code we sent" in body or "To log in" in body or "To confirm your account" in body:
|
||||
print("OTP prompt detected for [redacted]")
|
||||
if not args.otp:
|
||||
print("APPROVAL_NEEDED: OTP required for [redacted]")
|
||||
print("Re-run with: signin.py --email [redacted] --otp <code>")
|
||||
ws.close()
|
||||
sys.exit(2) # Approval needed
|
||||
|
||||
# Enter OTP
|
||||
print("Entering OTP...")
|
||||
result = ev(ws, f"""(async()=>{{
|
||||
const inp=[...document.querySelectorAll('input')].find(i=>i.type==='text'||i.type==='number');
|
||||
if(!inp) return 'NOINPUT';
|
||||
inp.focus();
|
||||
document.execCommand('insertText',false,'{args.otp}');
|
||||
await new Promise(r=>setTimeout(r,500));
|
||||
return 'entered:'+inp.value;
|
||||
}})()""", True)
|
||||
print("OTP: [redacted]")
|
||||
time.sleep(1)
|
||||
|
||||
# Click Next/Verify/Confirm
|
||||
print("Submitting OTP...")
|
||||
ev(ws, """(async()=>{
|
||||
const b=[...document.querySelectorAll('button')].find(x=>
|
||||
x.innerText.includes('Next')||x.innerText.includes('Verify')||x.innerText.includes('Confirm')
|
||||
);
|
||||
if(b) b.click(); return !!b;
|
||||
})()""", True)
|
||||
time.sleep(5)
|
||||
|
||||
# Verify login
|
||||
body = ev(ws, "document.body.innerText.slice(0,500)")
|
||||
title = ev(ws, "document.title") or ""
|
||||
url = ev(ws, "location.href") or ""
|
||||
|
||||
# Check for post-login gates requiring human/client intervention
|
||||
if "access/verification" in url or "confirm your age" in body.lower():
|
||||
ig_link = ev(ws, """(async()=>{
|
||||
try {
|
||||
const r = await fetch('/api/hatch/age-confirmation/linking-web-auth?account_type=instagram', {
|
||||
headers: {'Accept': 'application/json'}
|
||||
});
|
||||
const j = await r.json();
|
||||
return j && j.url ? j.url : null;
|
||||
} catch(e) { return null; }
|
||||
})()""", True)
|
||||
if ig_link:
|
||||
print(f"LINK_INSTAGRAM_URL: {ig_link}")
|
||||
print(f"NEEDS_HUMAN: Age verification required for {args.email} at {url}. Client intervention required to confirm age or link Instagram/Facebook.", file=sys.stderr)
|
||||
ws.close()
|
||||
sys.exit(3)
|
||||
|
||||
if "Connected" in body or "Chats" in body or (("Muse" in title or "Chat" in title) and "access/verification" not in url):
|
||||
print(f"SUCCESS: Logged in as [redacted] (title: {title})")
|
||||
ws.close()
|
||||
return 0
|
||||
# Multi-account selection: Meta shows one button per matching
|
||||
# account plus a "+1" collapsed entry. The "+1" is NOT a real
|
||||
# account — click the button whose text contains the hinted
|
||||
# display name. Without a hint (or no match), a human must pick.
|
||||
if "Your email matches multiple accounts" in ev(
|
||||
ws, "document.body.innerText.slice(0,2000)"):
|
||||
if not args.account_name:
|
||||
print("NEEDS_HUMAN: multiple accounts match and no "
|
||||
"--account-name hint was given", file=sys.stderr)
|
||||
ws.close()
|
||||
sys.exit(3)
|
||||
picked = ev(ws, """(async()=>{
|
||||
const want=%s.toLowerCase();
|
||||
const btns=[...document.querySelectorAll('button')];
|
||||
const t=btns.find(b=>b.innerText.toLowerCase().includes(want)
|
||||
&& !b.innerText.includes('+1'));
|
||||
if(t){t.click();return true;} return false;
|
||||
})()""" % json.dumps(args.account_name), True)
|
||||
if not picked:
|
||||
print(f"NEEDS_HUMAN: no account button matched "
|
||||
f"'[redacted]'", file=sys.stderr)
|
||||
ws.close()
|
||||
sys.exit(3)
|
||||
print(f"Selected account '[redacted]', waiting...")
|
||||
time.sleep(5)
|
||||
body = ev(ws, "document.body.innerText.slice(0,500)")
|
||||
title = ev(ws, "document.title") or ""
|
||||
if "Connected" in body or "Chats" in body or "Muse" in title or "Chat" in title:
|
||||
print("SUCCESS: Logged in as [redacted] "
|
||||
"(account: [redacted])")
|
||||
ws.close()
|
||||
return 0
|
||||
print("WARNING: account selection did not land in chat",
|
||||
file=sys.stderr)
|
||||
ws.close()
|
||||
sys.exit(1)
|
||||
else:
|
||||
print(f"WARNING: Login may have failed. Title: {title}", file=sys.stderr)
|
||||
print(f"Body: {body[:150]}", file=sys.stderr)
|
||||
ws.close()
|
||||
sys.exit(1)
|
||||
else:
|
||||
err_msg = ev(ws, """(()=>{
|
||||
const el = document.querySelector('[role="alert"], .error, [data-error]');
|
||||
return el ? el.innerText.trim() : '';
|
||||
})()""")
|
||||
if err_msg:
|
||||
print(f"ERROR: {err_msg}", file=sys.stderr)
|
||||
else:
|
||||
print("No OTP prompt found - check page state", file=sys.stderr)
|
||||
print(f"Body: {body[:200]}", file=sys.stderr)
|
||||
ws.close()
|
||||
sys.exit(1)
|
||||
|
||||
if __name__ == '__main__':
|
||||
sys.exit(main())
|
||||
@@ -1,236 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""muse-threads.py — per-agent thread bookkeeping over the hybrid gateway.
|
||||
|
||||
Thin JSON-contract wrapper around `bin/muse-cli-node` (the NetVM hybrid
|
||||
gateway transport): runs `session-pin / session-unpin / session-archive /
|
||||
session-unarchive / session-rename / threads` inside the agent's isolated
|
||||
network namespace (warp-<node>, per-node WARP egress identity) with its
|
||||
per-node cookie jar (~/.config/muse-cli/<node>/cookies.txt, 0600) and
|
||||
automatic CDP cookie refresh on auth failure.
|
||||
|
||||
This helper adds what the Box API needs on top of the raw transport:
|
||||
- one JSON object on stdout: {"ok": true, ...} or
|
||||
{"ok": false, "code": "...", "error": "...", "detail": ...}
|
||||
- exit 0 on success, nonzero on failure
|
||||
- input validation (agent / thread-id / title) with spec error codes
|
||||
- gateway failure mapping to THREAD_* codes server.py understands
|
||||
- a small delay before gateway writes to respect rate limits
|
||||
- never prints secrets, never uses a shell
|
||||
|
||||
Usage:
|
||||
bin/muse-threads.py <op> --agent <agent> [--thread <id>] [--title <title>]
|
||||
ops: list | pin | unpin | archive | unarchive | rename
|
||||
agent: one of muse, pip, 646, opm (node == agent == account)
|
||||
|
||||
Error codes (server.py maps these to HTTP):
|
||||
AGENT_NOT_FOUND unknown agent name
|
||||
THREAD_NOT_CONFIGURED no cookie jar for this agent (fail closed)
|
||||
INVALID_NAME agent or thread id fails the id regex
|
||||
INVALID_TITLE rename title missing/empty/too long/has controls
|
||||
THREAD_AUTH_FAILED auth failed even after cookie refresh
|
||||
THREAD_NOT_FOUND gateway says no such session
|
||||
GATEWAY_UNAVAILABLE timeout, or gateway vm_resolution_issue
|
||||
GATEWAY_ERROR other gateway failures (server maps to 502)
|
||||
|
||||
Session: sidechat/muse-cli-threads-helper
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
|
||||
KNOWN_AGENTS = ["muse", "pip", "646", "opm"]
|
||||
AGENT_RE = re.compile(r"^[a-z0-9-]{1,64}$")
|
||||
THREAD_RE = re.compile(r"^[A-Za-z0-9-]{1,64}$")
|
||||
MUTATE_DELAY = float(os.environ.get("MUSE_THREADS_DELAY", "2.0"))
|
||||
SUBPROCESS_TIMEOUT = 120
|
||||
|
||||
EXIT_USAGE = 1
|
||||
EXIT_AUTH = 2
|
||||
EXIT_GATEWAY = 3
|
||||
EXIT_NOT_CONFIGURED = 5
|
||||
|
||||
NETVM_BIN = os.path.dirname(os.path.abspath(__file__))
|
||||
MUSE_CLI_NODE = os.path.join(NETVM_BIN, "muse-cli-node")
|
||||
|
||||
|
||||
def fail(code, error, detail=None, exit_code=EXIT_GATEWAY):
|
||||
payload = {"ok": False, "code": code, "error": error}
|
||||
if detail:
|
||||
payload["detail"] = detail
|
||||
print(json.dumps(payload))
|
||||
sys.exit(exit_code)
|
||||
|
||||
|
||||
def cookie_jar(agent):
|
||||
return os.path.join(os.path.expanduser("~"), ".config", "muse-cli",
|
||||
agent, "cookies.txt")
|
||||
|
||||
|
||||
def run_node_cli(agent, argv):
|
||||
"""Run muse-cli-node <agent> <argv>. Returns (rc, stdout, stderr_tail)."""
|
||||
if not os.path.isfile(MUSE_CLI_NODE):
|
||||
fail("GATEWAY_ERROR",
|
||||
f"muse-cli-node not found at {MUSE_CLI_NODE}",
|
||||
exit_code=EXIT_GATEWAY)
|
||||
try:
|
||||
proc = subprocess.run(
|
||||
[MUSE_CLI_NODE, agent] + argv,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=SUBPROCESS_TIMEOUT,
|
||||
)
|
||||
except subprocess.TimeoutExpired:
|
||||
fail("GATEWAY_UNAVAILABLE", "muse-cli-node timed out",
|
||||
exit_code=EXIT_GATEWAY)
|
||||
except OSError as e:
|
||||
fail("GATEWAY_ERROR", f"failed to exec muse-cli-node: {e}",
|
||||
exit_code=EXIT_GATEWAY)
|
||||
err_lines = proc.stderr.strip().splitlines()
|
||||
err_tail = "\n".join(err_lines[-5:]) if err_lines else ""
|
||||
return proc.returncode, proc.stdout, err_tail
|
||||
|
||||
|
||||
def map_failure(rc, err_tail):
|
||||
"""Map a transport failure to (code, error, exit_code). Never leaks cookies."""
|
||||
low = err_tail.lower()
|
||||
if rc == 2 or "autherror" in low or "auth error" in low:
|
||||
return ("THREAD_AUTH_FAILED",
|
||||
"gateway auth failed even after cookie refresh — "
|
||||
"re-export cookies via refresh-node-cookies.py",
|
||||
err_tail or None, EXIT_AUTH)
|
||||
if "vm_resolution_issue" in low:
|
||||
return ("GATEWAY_UNAVAILABLE", "gateway vm_resolution_issue",
|
||||
err_tail or None, EXIT_GATEWAY)
|
||||
if rc == 4:
|
||||
return ("GATEWAY_UNAVAILABLE", "gateway timeout",
|
||||
err_tail or None, EXIT_GATEWAY)
|
||||
if ("not found" in low or "no such session" in low or " 404" in low
|
||||
or "does not exist" in low):
|
||||
return ("THREAD_NOT_FOUND", "no such session on that account",
|
||||
err_tail or None, EXIT_GATEWAY)
|
||||
return ("GATEWAY_ERROR", "gateway error", err_tail or None, EXIT_GATEWAY)
|
||||
|
||||
|
||||
def parse_threads_blob(blob):
|
||||
try:
|
||||
d = json.loads(blob)
|
||||
except json.JSONDecodeError as e:
|
||||
fail("BL_BAD_DATA", "threads output was unparseable",
|
||||
detail=str(e), exit_code=EXIT_GATEWAY)
|
||||
if isinstance(d, dict) and "threads" in d:
|
||||
return d["threads"]
|
||||
if isinstance(d, list):
|
||||
return d
|
||||
fail("BL_BAD_DATA", "threads output had unexpected shape",
|
||||
exit_code=EXIT_GATEWAY)
|
||||
|
||||
|
||||
def normalize_thread(t):
|
||||
return {
|
||||
"thread_id": t.get("session_id") or t.get("thread_id") or t.get("id"),
|
||||
"title": t.get("title"),
|
||||
"pinned": bool(t.get("pinned")),
|
||||
"archived": bool(t.get("archived")),
|
||||
"updated": t.get("updated"),
|
||||
}
|
||||
|
||||
|
||||
def cmd_list(agent):
|
||||
rc, out, err = run_node_cli(agent, ["threads", "--all"])
|
||||
if rc != 0:
|
||||
code, error, detail, ec = map_failure(rc, err)
|
||||
fail(code, error, detail=detail, exit_code=ec)
|
||||
threads = [normalize_thread(t) for t in parse_threads_blob(out)]
|
||||
print(json.dumps({"ok": True, "agent": agent, "threads": threads}))
|
||||
|
||||
|
||||
def cmd_mutate(agent, op, thread_id, title=None):
|
||||
if op == "rename":
|
||||
argv = ["session-rename", thread_id, title]
|
||||
else:
|
||||
argv = [f"session-{op}", thread_id]
|
||||
time.sleep(MUTATE_DELAY)
|
||||
rc, out, err = run_node_cli(agent, argv)
|
||||
if rc != 0:
|
||||
code, error, detail, ec = map_failure(rc, err)
|
||||
fail(code, error, detail=detail, exit_code=ec)
|
||||
try:
|
||||
result = json.loads(out) if out.strip() else {}
|
||||
except json.JSONDecodeError:
|
||||
result = {"raw": out.strip()[:500]}
|
||||
payload = {"ok": True, "agent": agent, "op": op, "thread_id": thread_id,
|
||||
"result": result}
|
||||
if op == "pin":
|
||||
payload["pinned"] = True
|
||||
elif op == "unpin":
|
||||
payload["pinned"] = False
|
||||
elif op == "archive":
|
||||
payload["archived"] = True
|
||||
elif op == "unarchive":
|
||||
payload["archived"] = False
|
||||
elif op == "rename":
|
||||
payload["title"] = title
|
||||
print(json.dumps(payload))
|
||||
|
||||
|
||||
def valid_title(title):
|
||||
if title is None:
|
||||
return False
|
||||
s = title.strip()
|
||||
if not (1 <= len(s) <= 128):
|
||||
return False
|
||||
if any(ord(c) < 32 or ord(c) == 127 for c in s):
|
||||
return False
|
||||
return True
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser(
|
||||
description="per-agent thread bookkeeping via the hybrid gateway")
|
||||
ap.add_argument("op", choices=["list", "pin", "unpin", "archive",
|
||||
"unarchive", "rename"])
|
||||
ap.add_argument("--agent", required=True,
|
||||
help="one of: " + ", ".join(KNOWN_AGENTS))
|
||||
ap.add_argument("--thread", default=None, help="session/thread id")
|
||||
ap.add_argument("--title", default=None, help="new title (rename only)")
|
||||
args = ap.parse_args()
|
||||
|
||||
agent = args.agent
|
||||
if not AGENT_RE.match(agent) or agent not in KNOWN_AGENTS:
|
||||
fail("AGENT_NOT_FOUND", f"unknown agent '{agent}'",
|
||||
exit_code=EXIT_USAGE)
|
||||
|
||||
if not os.path.isfile(cookie_jar(agent)):
|
||||
fail("THREAD_NOT_CONFIGURED",
|
||||
f"no cookie jar for agent '{agent}' "
|
||||
f"(expected {cookie_jar(agent)}); "
|
||||
f"run bin/refresh-node-cookies.py {agent}",
|
||||
exit_code=EXIT_NOT_CONFIGURED)
|
||||
|
||||
if args.op == "list":
|
||||
cmd_list(agent)
|
||||
return
|
||||
|
||||
thread_id = args.thread
|
||||
if not thread_id or not THREAD_RE.match(thread_id):
|
||||
fail("INVALID_NAME", "thread id must match ^[A-Za-z0-9-]{1,64}$",
|
||||
exit_code=EXIT_USAGE)
|
||||
|
||||
if args.op == "rename":
|
||||
if not valid_title(args.title):
|
||||
fail("INVALID_TITLE",
|
||||
"title must be 1-128 chars after stripping, "
|
||||
"no control characters",
|
||||
exit_code=EXIT_USAGE)
|
||||
cmd_mutate(agent, "rename", thread_id, title=args.title.strip())
|
||||
else:
|
||||
cmd_mutate(agent, args.op, thread_id)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -1,254 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
muse-tmux.py: Shared tmux socket manager for Muse agents and operators.
|
||||
Socket location: /tmp/tmux-muse.sock (shared across fleet agents & super).
|
||||
"""
|
||||
import sys
|
||||
import os
|
||||
import re
|
||||
import subprocess
|
||||
import argparse
|
||||
|
||||
SOCKET_PATH = "/tmp/tmux-muse.sock"
|
||||
LOG_DIR = "/home/super/Projects/NetVM/logs/tmux"
|
||||
TMUX_BIN = "/home/super/.local/bin/tmux"
|
||||
if not os.path.exists(TMUX_BIN):
|
||||
TMUX_BIN = "tmux"
|
||||
|
||||
def _target_label(node=None, container=None):
|
||||
if node:
|
||||
return f"node netns '{node}' (/tmp/tmux-{node}.sock)"
|
||||
if container:
|
||||
return f"docker container '{container}'"
|
||||
return f"shared socket {SOCKET_PATH}"
|
||||
|
||||
def _ensure_logging(session, node=None, container=None):
|
||||
try:
|
||||
os.makedirs(LOG_DIR, exist_ok=True)
|
||||
tag = f"-{node}" if node else (f"-{container}" if container else "")
|
||||
log_file = os.path.join(LOG_DIR, f"{session}{tag}.log")
|
||||
_run_tmux("pipe-pane", "-o", "-t", session, f"cat >> {log_file}", node=node, container=container)
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
def _run_tmux(*args, capture=True, node=None, container=None):
|
||||
if node:
|
||||
cmd = ["/home/super/Projects/NetVM/bin/netvm-exec.sh", node, "--", TMUX_BIN, "-S", f"/tmp/tmux-{node}.sock"] + list(args)
|
||||
elif container:
|
||||
cmd = ["docker", "exec", "-i", container, "tmux"] + list(args)
|
||||
else:
|
||||
cmd = [TMUX_BIN, "-S", SOCKET_PATH] + list(args)
|
||||
res = subprocess.run(cmd, capture_output=capture, text=True)
|
||||
return res.returncode, res.stdout, res.stderr
|
||||
|
||||
def cmd_new(args):
|
||||
session = args.session
|
||||
window = getattr(args, "window", None)
|
||||
command = getattr(args, "command", None)
|
||||
node = getattr(args, "node", None)
|
||||
container = getattr(args, "container", None)
|
||||
|
||||
# Check if session already exists
|
||||
rc, out, _ = _run_tmux("has-session", "-t", session, node=node, container=container)
|
||||
if rc == 0:
|
||||
print(f"Session '{session}' already exists on {_target_label(node, container)}.")
|
||||
return 0
|
||||
|
||||
t_args = ["new-session", "-d", "-s", session]
|
||||
if window:
|
||||
t_args.extend(["-n", window])
|
||||
if command:
|
||||
t_args.append(command)
|
||||
|
||||
rc, out, err = _run_tmux(*t_args, node=node, container=container)
|
||||
if rc == 0:
|
||||
_ensure_logging(session, node=node, container=container)
|
||||
print(f"✔ Created session '{session}' on {_target_label(node, container)} (logging enabled)")
|
||||
return 0
|
||||
else:
|
||||
print(f"Failed to create session: {err.strip() or out.strip()}", file=sys.stderr)
|
||||
return rc
|
||||
|
||||
def cmd_send(args):
|
||||
session = args.session
|
||||
keys = args.keys
|
||||
enter = not getattr(args, "no_enter", False)
|
||||
node = getattr(args, "node", None)
|
||||
container = getattr(args, "container", None)
|
||||
|
||||
# Check session
|
||||
rc, _, _ = _run_tmux("has-session", "-t", session, node=node, container=container)
|
||||
if rc != 0:
|
||||
# Auto-create if not exists
|
||||
print(f"Notice: Session '{session}' does not exist on {_target_label(node, container)}; creating now...", file=sys.stderr)
|
||||
_run_tmux("new-session", "-d", "-s", session, node=node, container=container)
|
||||
_ensure_logging(session, node=node, container=container)
|
||||
else:
|
||||
_ensure_logging(session, node=node, container=container)
|
||||
|
||||
t_args = ["send-keys", "-t", session, keys]
|
||||
if enter:
|
||||
t_args.append("Enter")
|
||||
|
||||
rc, out, err = _run_tmux(*t_args, node=node, container=container)
|
||||
if rc == 0:
|
||||
print(f"✔ Sent keys to '{session}' on {_target_label(node, container)}")
|
||||
return 0
|
||||
else:
|
||||
print(f"Failed to send keys: {err.strip() or out.strip()}", file=sys.stderr)
|
||||
return rc
|
||||
|
||||
def cmd_capture(args):
|
||||
session = args.session
|
||||
lines = getattr(args, "lines", 30) or 30
|
||||
node = getattr(args, "node", None)
|
||||
container = getattr(args, "container", None)
|
||||
|
||||
t_args = ["capture-pane", "-p", "-t", session, "-S", f"-{lines}"]
|
||||
rc, out, err = _run_tmux(*t_args, node=node, container=container)
|
||||
if rc == 0:
|
||||
# Strip trailing blank lines
|
||||
clean = out.rstrip()
|
||||
print(clean)
|
||||
return 0
|
||||
else:
|
||||
print(f"Failed to capture pane: {err.strip() or out.strip()}", file=sys.stderr)
|
||||
return rc
|
||||
|
||||
def cmd_list(args):
|
||||
node = getattr(args, "node", None)
|
||||
container = getattr(args, "container", None)
|
||||
rc, out, err = _run_tmux("list-sessions", node=node, container=container)
|
||||
if rc == 0:
|
||||
print(out.strip() or "No active sessions.")
|
||||
return 0
|
||||
elif "no server running" in (err or "").lower() or rc == 1:
|
||||
print(f"No active tmux server on {_target_label(node, container)}.")
|
||||
return 0
|
||||
else:
|
||||
print(f"Error: {err.strip() or out.strip()}", file=sys.stderr)
|
||||
return rc
|
||||
|
||||
def cmd_kill(args):
|
||||
session = args.session
|
||||
node = getattr(args, "node", None)
|
||||
container = getattr(args, "container", None)
|
||||
rc, out, err = _run_tmux("kill-session", "-t", session, node=node, container=container)
|
||||
if rc == 0:
|
||||
print(f"✔ Killed session '{session}' on {_target_label(node, container)}")
|
||||
return 0
|
||||
else:
|
||||
print(f"Error: {err.strip() or out.strip()}", file=sys.stderr)
|
||||
return rc
|
||||
|
||||
def cmd_prune(args):
|
||||
ttl_seconds = getattr(args, "ttl", 7200) or 7200
|
||||
node = getattr(args, "node", None)
|
||||
container = getattr(args, "container", None)
|
||||
rc, out, err = _run_tmux("list-sessions", "-F", "#{session_name} #{session_activity} #{session_attached}", node=node, container=container)
|
||||
if rc != 0:
|
||||
if "no server running" in (err or "").lower() or rc == 1:
|
||||
print(f"No active tmux server on {_target_label(node, container)}.")
|
||||
return 0
|
||||
print(f"Error listing sessions: {err.strip() or out.strip()}", file=sys.stderr)
|
||||
return rc
|
||||
|
||||
now = int(time.time())
|
||||
reaped = []
|
||||
kept = []
|
||||
|
||||
for line in out.strip().splitlines():
|
||||
parts = line.split()
|
||||
if len(parts) >= 3:
|
||||
s_name, s_activity, s_attached = parts[0], int(parts[1]), int(parts[2])
|
||||
idle_time = now - s_activity
|
||||
# Only prune if unattached and idle > TTL
|
||||
if s_attached == 0 and idle_time > ttl_seconds:
|
||||
_run_tmux("kill-session", "-t", s_name, node=node, container=container)
|
||||
reaped.append((s_name, idle_time))
|
||||
else:
|
||||
kept.append((s_name, idle_time, s_attached))
|
||||
|
||||
if reaped:
|
||||
print(f"✔ Reaped {len(reaped)} stale unattached session(s) on {_target_label(node, container)} (idle > {ttl_seconds//3600}h):")
|
||||
for s_name, idle in reaped:
|
||||
print(f" • {s_name} (idle: {idle//60}m)")
|
||||
else:
|
||||
print(f"No stale unattached sessions to reap on {_target_label(node, container)} (all active within {ttl_seconds//3600}h).")
|
||||
return 0
|
||||
|
||||
def cmd_attach(args):
|
||||
session = args.session
|
||||
node = getattr(args, "node", None)
|
||||
container = getattr(args, "container", None)
|
||||
if node:
|
||||
cmd = ["/home/super/Projects/NetVM/bin/netvm-exec.sh", node, "--", TMUX_BIN, "-S", f"/tmp/tmux-{node}.sock", "attach", "-t", session]
|
||||
os.execv(cmd[0], cmd)
|
||||
elif container:
|
||||
cmd = ["docker", "exec", "-it", container, "tmux", "attach", "-t", session]
|
||||
os.execv("/usr/bin/docker", cmd)
|
||||
else:
|
||||
cmd = [TMUX_BIN, "-S", SOCKET_PATH, "attach", "-t", session]
|
||||
os.execv(TMUX_BIN, cmd)
|
||||
|
||||
def main():
|
||||
# Common parent parser for hybrid routing
|
||||
parent = argparse.ArgumentParser(add_help=False)
|
||||
parent.add_argument("--node", default=argparse.SUPPRESS, help="Target NetVM node namespace (e.g. pip, dev, 646, opm, muse, def)")
|
||||
parent.add_argument("--container", default=argparse.SUPPRESS, help="Target Docker container name")
|
||||
|
||||
parser = argparse.ArgumentParser(description="Shared Muse tmux socket manager (host, netns nodes, containers)", parents=[parent])
|
||||
subparsers = parser.add_subparsers(dest="action")
|
||||
|
||||
# list
|
||||
p_ls = subparsers.add_parser("list", aliases=["ls"], parents=[parent], help="List sessions on shared socket or node/container")
|
||||
p_ls.set_defaults(func=cmd_list)
|
||||
|
||||
# new
|
||||
p_new = subparsers.add_parser("new", parents=[parent], help="Create new background session")
|
||||
p_new.add_argument("session", help="Session name")
|
||||
p_new.add_argument("--window", "-w", help="Initial window name")
|
||||
p_new.add_argument("--command", "-c", help="Command to run in session")
|
||||
p_new.set_defaults(func=cmd_new)
|
||||
|
||||
# send / send-keys
|
||||
p_send = subparsers.add_parser("send", aliases=["send-keys"], parents=[parent], help="Send keys to a session")
|
||||
p_send.add_argument("session", help="Target session name")
|
||||
p_send.add_argument("keys", help="Keys / command string to send")
|
||||
p_send.add_argument("--no-enter", action="store_true", help="Do not send Enter key after keys")
|
||||
p_send.set_defaults(func=cmd_send)
|
||||
|
||||
# capture
|
||||
p_cap = subparsers.add_parser("capture", aliases=["cap", "tail"], parents=[parent], help="Capture pane output")
|
||||
p_cap.add_argument("session", help="Target session name")
|
||||
p_cap.add_argument("--lines", "-n", type=int, default=30, help="Number of scrollback lines to capture (default: 30)")
|
||||
p_cap.set_defaults(func=cmd_capture)
|
||||
|
||||
# kill
|
||||
p_kill = subparsers.add_parser("kill", parents=[parent], help="Kill a session")
|
||||
p_kill.add_argument("session", help="Session name")
|
||||
p_kill.set_defaults(func=cmd_kill)
|
||||
|
||||
# prune
|
||||
p_prune = subparsers.add_parser("prune", parents=[parent], help="Reap stale unattached sessions inactive for >TTL (default: 7200s / 2h)")
|
||||
p_prune.add_argument("--ttl", type=int, default=7200, help="Inactivity threshold in seconds (default: 7200)")
|
||||
p_prune.set_defaults(func=cmd_prune)
|
||||
|
||||
# attach
|
||||
p_att = subparsers.add_parser("attach", parents=[parent], help="Attach to a session interactively")
|
||||
p_att.add_argument("session", help="Session name")
|
||||
p_att.set_defaults(func=cmd_attach)
|
||||
|
||||
if len(sys.argv) == 1:
|
||||
parser.print_help()
|
||||
sys.exit(0)
|
||||
|
||||
args = parser.parse_args()
|
||||
if not hasattr(args, "func"):
|
||||
parser.print_help()
|
||||
sys.exit(1)
|
||||
|
||||
sys.exit(args.func(args))
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
Executable
+9117
File diff suppressed because it is too large
Load Diff
@@ -1,113 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
muse_hybrid.py: Hybrid integration layer combining fast headless gateway actions
|
||||
(via muse-cli-node with per-node Cloudflare WARP egress & cookies) with Chromebox
|
||||
CDP DOM interactions.
|
||||
|
||||
Provides:
|
||||
- muse_threads(agent) -> list of threads
|
||||
- muse_history(agent, thread_id=None, limit=10) -> list of messages
|
||||
- muse_send(agent, text, thread_id=None, wait=0) -> dict response
|
||||
- muse_session_start(agent, title=None) -> dict response
|
||||
"""
|
||||
|
||||
import subprocess
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
BIN_DIR = Path(__file__).resolve().parent
|
||||
MUSE_CLI_NODE = BIN_DIR / "muse-cli-node"
|
||||
|
||||
def is_node_configured(node):
|
||||
conf_dir = Path.home() / ".config" / "muse-cli" / node
|
||||
return (conf_dir / "cookies.txt").exists()
|
||||
|
||||
def run_muse_cli(node, args, timeout=60):
|
||||
cmd = [str(MUSE_CLI_NODE), node] + args
|
||||
res = subprocess.run(cmd, capture_output=True, text=True, timeout=timeout)
|
||||
return res.returncode, res.stdout, res.stderr
|
||||
|
||||
def get_threads(node):
|
||||
"""Retrieve thread list via fast gateway. Returns (threads_list, error_str)."""
|
||||
rc, stdout, stderr = run_muse_cli(node, ["threads"])
|
||||
if rc != 0:
|
||||
return None, stderr or stdout
|
||||
try:
|
||||
data = json.loads(stdout)
|
||||
return data, None
|
||||
except Exception as e:
|
||||
return None, f"JSON parse error: {e}"
|
||||
|
||||
def start_session(node, title=None):
|
||||
"""Start a new session/subagent via fast gateway. Returns (dict, error_str)."""
|
||||
args = ["session-start"]
|
||||
if title:
|
||||
args.extend(["--title", title])
|
||||
rc, stdout, stderr = run_muse_cli(node, args)
|
||||
if rc != 0:
|
||||
return None, stderr or stdout
|
||||
try:
|
||||
data = json.loads(stdout)
|
||||
return data, None
|
||||
except Exception as e:
|
||||
return None, f"JSON parse error: {e}"
|
||||
|
||||
def get_history(node, thread_id=None, limit=15):
|
||||
"""Retrieve message history via fast gateway. Returns (history_list, error_str)."""
|
||||
args = ["history", "--limit", str(limit)]
|
||||
if thread_id and thread_id != "main":
|
||||
args.extend(["--thread", str(thread_id)])
|
||||
rc, stdout, stderr = run_muse_cli(node, args)
|
||||
if rc != 0:
|
||||
return None, stderr or stdout
|
||||
try:
|
||||
data = json.loads(stdout)
|
||||
return data, None
|
||||
except Exception as e:
|
||||
return None, f"JSON parse error: {e}"
|
||||
|
||||
def send_message(node, text, thread_id=None, wait=0):
|
||||
"""Send message via fast gateway. Returns (result_dict, error_str)."""
|
||||
args = ["send"]
|
||||
if thread_id and thread_id != "main":
|
||||
args.extend(["--thread", str(thread_id)])
|
||||
args.extend(["--wait", str(wait)])
|
||||
args.append(text)
|
||||
rc, stdout, stderr = run_muse_cli(node, args, timeout=max(60, wait + 30))
|
||||
if rc != 0:
|
||||
return None, stderr or stdout
|
||||
try:
|
||||
data = json.loads(stdout)
|
||||
return data, None
|
||||
except Exception as e:
|
||||
return {"raw": stdout}, None
|
||||
|
||||
if __name__ == "__main__":
|
||||
if len(sys.argv) < 3:
|
||||
print("Usage: muse_hybrid.py <node> <threads|history|send> [args...]")
|
||||
sys.exit(1)
|
||||
node = sys.argv[1]
|
||||
action = sys.argv[2]
|
||||
if action == "threads":
|
||||
threads, err = get_threads(node)
|
||||
if err:
|
||||
print("Error:", err, file=sys.stderr)
|
||||
sys.exit(1)
|
||||
print(json.dumps(threads, indent=2))
|
||||
elif action == "history":
|
||||
tid = sys.argv[3] if len(sys.argv) > 3 else None
|
||||
msgs, err = get_history(node, tid)
|
||||
if err:
|
||||
print("Error:", err, file=sys.stderr)
|
||||
sys.exit(1)
|
||||
print(json.dumps(msgs, indent=2))
|
||||
elif action == "send":
|
||||
txt = sys.argv[3]
|
||||
tid = sys.argv[4] if len(sys.argv) > 4 else None
|
||||
res, err = send_message(node, txt, tid)
|
||||
if err:
|
||||
print("Error:", err, file=sys.stderr)
|
||||
sys.exit(1)
|
||||
print(json.dumps(res, indent=2))
|
||||
+52
-6
@@ -1,21 +1,26 @@
|
||||
#!/usr/bin/env bash
|
||||
# netvm-chrome.sh [--headless] [--cdp-port N|--no-cdp] <profile> [url]
|
||||
# netvm-chrome.sh [--headless] [--cdp-port N|--no-cdp] [--contained] <profile> [url]
|
||||
# Launch a chrome-box profile inside its dedicated NetVM netns.
|
||||
# 1:1: profile = node = warp identity = egress IP.
|
||||
# CDP is on by default (deterministic port) so agents can automate the
|
||||
# session via Playwright/Puppeteer; --headless runs without a display.
|
||||
# --contained adds a bwrap filesystem jail INSIDE the netns (no net unshare):
|
||||
# the browser sees only its own profile home (+ vault read-only). Chromium's
|
||||
# own sandbox stays on (we never pass --no-sandbox to it).
|
||||
# Run as your normal user (uses sudo -n only for allowlisted netns ops).
|
||||
set -euo pipefail
|
||||
NETVM_BIN="$(cd "$(dirname "$0")" && pwd)"
|
||||
. "$NETVM_BIN/netvm-names.sh"
|
||||
HEADLESS=0; CDP_OVERRIDE=""; NO_CDP=0; PROFILE=""; URL=""
|
||||
usage() { echo "usage: netvm-chrome.sh [--headless] [--cdp-port N|--no-cdp] <profile> [url]"; }
|
||||
WAYPIPE=0; HEADLESS=0; CDP_OVERRIDE=""; NO_CDP=0; PROFILE=""; URL=""; CONTAINED=0
|
||||
usage() { echo "usage: netvm-chrome.sh [--waypipe] [--headless] [--cdp-port N|--no-cdp] [--contained] <profile> [url]"; }
|
||||
while [ $# -gt 0 ]; do
|
||||
case "$1" in
|
||||
--waypipe) WAYPIPE=1; shift;;
|
||||
--headless) HEADLESS=1; shift;;
|
||||
--cdp-port) CDP_OVERRIDE="$2"; shift 2;;
|
||||
--cdp-port=*) CDP_OVERRIDE="${1#*=}"; shift;;
|
||||
--no-cdp) NO_CDP=1; shift;;
|
||||
--contained) CONTAINED=1; shift;;
|
||||
-h|--help) usage; exit 0;;
|
||||
*) if [ -z "$PROFILE" ]; then PROFILE="$1"; elif [ -z "$URL" ]; then URL="$1";
|
||||
else echo "unexpected: $1"; usage; exit 1; fi; shift;;
|
||||
@@ -44,9 +49,50 @@ if [ "$NO_CDP" = 1 ]; then CDP=""; elif [ -n "$CDP_OVERRIDE" ]; then CDP="$CDP_O
|
||||
|
||||
sudo -n "$NETVM_BIN/netvm-node-up.sh" "$NODE" | tail -1
|
||||
|
||||
LAUNCH_ARGS=(launch "$PROFILE")
|
||||
[ "$HEADLESS" = 1 ] && LAUNCH_ARGS+=(--no-sandbox --headless)
|
||||
LAUNCH_ARGS=(launch "$PROFILE" --no-sandbox)
|
||||
[ "$HEADLESS" = 1 ] && LAUNCH_ARGS+=(--headless)
|
||||
|
||||
[ -n "$CDP" ] && LAUNCH_ARGS+=(--cdp-port "$CDP")
|
||||
[ -n "$URL" ] && LAUNCH_ARGS+=("$URL")
|
||||
[ -n "$CDP" ] && echo "cdp: http://$PEER_IP:$CDP/json/list (local) | ssh -L $CDP:$PEER_IP:$CDP <user>@<tail-ip> (remote)"
|
||||
exec sudo -n "$NETVM_BIN/netvm-enter.sh" "$NODE" "$(id -u)" "$(id -g)" "$HOME" -- "$CHROME_BOX" "${LAUNCH_ARGS[@]}"
|
||||
CMD=("$CHROME_BOX" "${LAUNCH_ARGS[@]}")
|
||||
if [ "$CONTAINED" = 1 ]; then
|
||||
# Contained mode execs Chromium directly (same flags chrome-box uses for a
|
||||
# native launch) because chrome-box re-resolves its profile dir from $HOME,
|
||||
# which the jail replaces. Direct Warp egress: no proxy flags by design.
|
||||
command -v bwrap >/dev/null 2>&1 || { echo "bwrap not found" >&2; exit 1; }
|
||||
P_HOME="$HOME/.local/share/chrome-box/profiles/$PROFILE"
|
||||
mkdir -p "$P_HOME/.config/chromium"
|
||||
KEYRING="$(python3 -c "import json,sys;print(json.load(open('$P_HOME/config.json')).get('keyring','basic'))" 2>/dev/null || echo basic)"
|
||||
SB_DIR="$(dirname "$CHROME_BOX")"
|
||||
BWRAP=(bwrap --unshare-uts --unshare-ipc --die-with-parent
|
||||
--ro-bind / / --dev /dev --proc /proc --tmpfs /tmp --tmpfs /dev/shm
|
||||
--bind "$P_HOME" "$HOME" --setenv HOME "$HOME" --setenv PATH /usr/bin:/bin)
|
||||
[ -d "$HOME/notes" ] && BWRAP+=(--ro-bind "$HOME/notes" /tmp/vault)
|
||||
[ -f "$SB_DIR/hosts-sandbox.conf" ] && BWRAP+=(--ro-bind "$SB_DIR/hosts-sandbox.conf" /etc/hosts)
|
||||
[ -f "$SB_DIR/nsswitch-sandbox.conf" ] && BWRAP+=(--ro-bind "$SB_DIR/nsswitch-sandbox.conf" /etc/nsswitch.conf)
|
||||
CHROMIUM=(/usr/lib/chromium/chromium
|
||||
"--user-data-dir=$HOME/.config/chromium"
|
||||
"--password-store=$KEYRING"
|
||||
--ozone-platform=x11 --disable-gpu --disable-quic
|
||||
--disable-features=DnsOverHttpsUpgrade,AsyncDns
|
||||
--built-in-dns-client-enabled=false)
|
||||
if [ "$HEADLESS" = 1 ]; then
|
||||
BWRAP+=(--unsetenv DISPLAY --unsetenv WAYLAND_DISPLAY --unsetenv XDG_RUNTIME_DIR)
|
||||
CHROMIUM+=(--headless=new)
|
||||
else
|
||||
[ -d /tmp/.X11-unix ] && BWRAP+=(--ro-bind /tmp/.X11-unix /tmp/.X11-unix)
|
||||
[ -n "${WAYLAND_DISPLAY:-}" ] && [ -S "$XDG_RUNTIME_DIR/$WAYLAND_DISPLAY" ] && \
|
||||
BWRAP+=(--ro-bind "$XDG_RUNTIME_DIR/$WAYLAND_DISPLAY" "$XDG_RUNTIME_DIR/$WAYLAND_DISPLAY")
|
||||
[ -n "${XDG_RUNTIME_DIR:-}" ] && [ -d "$XDG_RUNTIME_DIR/pulse" ] && BWRAP+=(--bind "$XDG_RUNTIME_DIR/pulse" /run/pulse)
|
||||
[ -n "${XDG_RUNTIME_DIR:-}" ] && [ -S "$XDG_RUNTIME_DIR/bus" ] && BWRAP+=(--ro-bind "$XDG_RUNTIME_DIR/bus" /run/dbus/system_bus_socket)
|
||||
fi
|
||||
[ -n "$CDP" ] && CHROMIUM+=(--remote-debugging-port="$CDP" --remote-allow-origins='*')
|
||||
[ -n "$URL" ] && CHROMIUM+=("$URL")
|
||||
CMD=("${BWRAP[@]}" -- "${CHROMIUM[@]}")
|
||||
fi
|
||||
if [ "$WAYPIPE" = 1 ]; then
|
||||
exec waypipe --compress=lz4 run -- sudo -n "$NETVM_BIN/netvm-enter.sh" "$NODE" "$(id -u)" "$(id -g)" "$HOME" -- "${CMD[@]}"
|
||||
else
|
||||
exec sudo -n "$NETVM_BIN/netvm-enter.sh" "$NODE" "$(id -u)" "$(id -g)" "$HOME" -- "${CMD[@]}"
|
||||
fi
|
||||
|
||||
@@ -1,52 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
NetVM Docs HTTP Server (bl)
|
||||
|
||||
Serves markdown documentation for agents via HTTPS (tailnet).
|
||||
- Central repo: ~/Projects/NetVM/
|
||||
- Serves: *.md, docs/*.png
|
||||
- Read-only, no auth (tailnet is the auth boundary)
|
||||
|
||||
GOLDEN PATH: container -> VM (34.139.37.135) -> bl (100.123.153.75) -> this server
|
||||
|
||||
Usage:
|
||||
python3 netvm-docs-server.py [--port 8080]
|
||||
|
||||
Agents fetch via:
|
||||
curl http://0.0.0.0:8080/ACCOUNTS.md
|
||||
"""
|
||||
import http.server
|
||||
import socketserver
|
||||
import os
|
||||
import argparse
|
||||
from pathlib import Path
|
||||
|
||||
class DocsHandler(http.server.SimpleHTTPRequestHandler):
|
||||
def __init__(self, *args, **kwargs):
|
||||
# Serve from NetVM repo root
|
||||
self.base_dir = Path.home() / "Projects" / "NetVM"
|
||||
super().__init__(*args, directory=str(self.base_dir), **kwargs)
|
||||
|
||||
def log_message(self, format, *args):
|
||||
# Quiet logging
|
||||
pass
|
||||
|
||||
def end_headers(self):
|
||||
# Allow CORS for browser agents
|
||||
self.send_header('Access-Control-Allow-Origin', '*')
|
||||
super().end_headers()
|
||||
|
||||
def main():
|
||||
p = argparse.ArgumentParser()
|
||||
p.add_argument('--port', type=int, default=8080)
|
||||
args = p.parse_args()
|
||||
|
||||
# Only bind to tailnet interface (not public)
|
||||
# 100.123.153.75 is bl's tailnet IP
|
||||
with socketserver.TCPServer(("0.0.0.0", args.port), DocsHandler) as httpd:
|
||||
print(f"Serving NetVM docs on http://0.0.0.0:{args.port}/")
|
||||
print(f"Directory: {Path.home()}/Projects/NetVM/")
|
||||
httpd.serve_forever()
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
+11
-2
@@ -10,6 +10,15 @@ netvm_names "$1"; TUID="$2"; TGID="$3"; THOME="$4"; shift 4
|
||||
[ "${1:-}" = "--" ]; shift
|
||||
RESOLV=/etc/netvm/resolv-warp.conf
|
||||
[ -f "$RESOLV" ] || echo "nameserver 1.1.1.1" > "$RESOLV"
|
||||
ip netns exec "$NETNS" env \
|
||||
|
||||
EXEC_CMD=(ip netns exec "$NETNS" env \
|
||||
NETVM_RESOLV="$RESOLV" NETVM_UID="$TUID" NETVM_GID="$TGID" NETVM_HOME="$THOME" \
|
||||
unshare --mount "$SCRIPT_DIR/netvm-enter-inner.sh" "$@"
|
||||
unshare --mount "$SCRIPT_DIR/netvm-enter-inner.sh" "$@")
|
||||
|
||||
if command -v systemd-run >/dev/null 2>&1; then
|
||||
UNIT_NAME="netvm-${NODE}-$RANDOM"
|
||||
exec systemd-run --scope -p MemoryMax=2G -p CPUQuota=200% --unit="$UNIT_NAME" "${EXEC_CMD[@]}"
|
||||
else
|
||||
exec "${EXEC_CMD[@]}"
|
||||
fi
|
||||
|
||||
|
||||
+1
-16
@@ -9,21 +9,6 @@ netvm_names() {
|
||||
VETH="ve-${_tag}"
|
||||
VPEER="vp-${_tag}"
|
||||
_idx=$(( 0x${_tag:0:3} % 200 + 10 ))
|
||||
# CDP_PORT_OVERRIDE lets provisioners assign clean sequential ports
|
||||
# (9410, 9420, ...) instead of the hash-derived default.
|
||||
# Unset = previous behavior.
|
||||
CDP_PORT=${CDP_PORT_OVERRIDE:-$(( 9222 + 0x${_tag:4:3} % 2000 ))}
|
||||
# Pinned registry ports: single source of truth for the fleet.
|
||||
# (Was previously duplicated in cdp-relay-watchdog.sh relay_target();
|
||||
# moved here 2026-10-04 so netvm-node-up.sh stops spawning wrong-port
|
||||
# zombie relays on every browser relaunch.)
|
||||
case "$NODE" in
|
||||
muse) CDP_PORT=9410 ;;
|
||||
pip) CDP_PORT=9420 ;;
|
||||
646) CDP_PORT=9430 ;;
|
||||
opm) CDP_PORT=9440 ;;
|
||||
def) CDP_PORT=9450 ;;
|
||||
dev) CDP_PORT=9455 ;;
|
||||
esac
|
||||
CDP_PORT=$(( 9222 + 0x${_tag:4:3} % 2000 ))
|
||||
GW="10.201.${_idx}.1"; PEER_IP="10.201.${_idx}.2"; SUB="10.201.${_idx}.0/30"
|
||||
}
|
||||
|
||||
+29
-4
@@ -6,8 +6,15 @@ SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
|
||||
. "$SCRIPT_DIR/netvm-names.sh"
|
||||
netvm_names "${1:?usage: netvm-node-up.sh <node>}"
|
||||
CONF="/etc/netvm/${NODE}.conf"
|
||||
STAGE_CONF="/tmp/netvm-stage-${NODE}.conf"
|
||||
if [ ! -f "$CONF" ] && [ -f "$STAGE_CONF" ]; then
|
||||
mkdir -p /etc/netvm 2>/dev/null || true
|
||||
install -m 600 -o root -g root "$STAGE_CONF" "$CONF"
|
||||
rm -f "$STAGE_CONF"
|
||||
fi
|
||||
[ -f "$CONF" ] || { echo "missing $CONF (human: netvm-new-identity.sh $NODE)"; exit 1; }
|
||||
chmod 600 "$CONF"
|
||||
|
||||
# let the operator user stat (not read) identities: 711 dir, 600 files
|
||||
chmod 711 /etc/netvm 2>/dev/null || true
|
||||
nsexec() { ip netns exec "$NETNS" "$@"; }
|
||||
@@ -24,9 +31,6 @@ if ! nsexec ip link show "$VPEER" >/dev/null 2>&1; then
|
||||
nsexec ip addr add "${PEER_IP}/30" dev "$VPEER" 2>/dev/null || true
|
||||
nsexec ip link set "$VPEER" up
|
||||
fi
|
||||
# Idempotent: re-assert host-side GW addr even if veth pre-existed (2026-10-04: def/dev lost theirs)
|
||||
ip addr add "${GW}/30" dev "$VETH" 2>/dev/null || true
|
||||
ip link set "$VETH" up
|
||||
nsexec ip link set lo up
|
||||
|
||||
# host NAT + forwarding for the veth subnet
|
||||
@@ -103,4 +107,25 @@ else
|
||||
fi
|
||||
EGRESS=$(nsexec curl -sk --max-time 15 'https://1.1.1.1/cdn-cgi/trace' 2>/dev/null | grep -oP '^ip=\K.*' || true)
|
||||
fi
|
||||
echo "node=$NODE netns=$NETNS ifaces=$WG/$VETH egress=${EGRESS:-unknown}"
|
||||
|
||||
if [ -z "$EGRESS" ]; then
|
||||
echo "ERROR: Egress check failed for node '$NODE' (warp interface down or unroutable). Aborting." >&2
|
||||
EMAIL_ALERT="/home/super/Projects/email-alert/email-alert"
|
||||
if [ -x "$EMAIL_ALERT" ]; then
|
||||
"$EMAIL_ALERT" send --priority high --subject "NetVM Alert: Node '$NODE' Egress Failed" "WireGuard WARP tunnel for node '$NODE' failed egress verification. Execution aborted to protect IP isolation." || true
|
||||
fi
|
||||
exit 1
|
||||
fi
|
||||
|
||||
REAL_USER="${SUDO_USER:-$USER}"
|
||||
REAL_HOME=$(eval echo "~$REAL_USER")
|
||||
AUDIT_DIR="$REAL_HOME/.local/share/chrome-box"
|
||||
mkdir -p "$AUDIT_DIR" 2>/dev/null || true
|
||||
chown "$REAL_USER:" "$AUDIT_DIR" 2>/dev/null || true
|
||||
TIMESTAMP=$(date -u +"%Y-%m-%dT%H:%M:%SZ")
|
||||
python3 -c "import json; print(json.dumps({'timestamp': '$TIMESTAMP', 'event': 'node_up', 'node': '$NODE', 'netns': '$NETNS', 'veth_ip': '$PEER_IP', 'cdp_port': $CDP_PORT, 'egress_ip': '$EGRESS', 'user': '$REAL_USER'}))" >> "$AUDIT_DIR/audit.jsonl" 2>/dev/null || true
|
||||
chown "$REAL_USER:" "$AUDIT_DIR/audit.jsonl" 2>/dev/null || true
|
||||
|
||||
echo "node=$NODE netns=$NETNS ifaces=$WG/$VETH egress=$EGRESS"
|
||||
|
||||
|
||||
|
||||
Executable
+36
@@ -0,0 +1,36 @@
|
||||
#!/usr/bin/env bash
|
||||
# netvm-proton.sh <profile> -- <proton-cli args...>
|
||||
# Run proton-cli as the profile's Proton identity, inside the profile's netns
|
||||
# (Warp egress) and inside a bwrap filesystem jail.
|
||||
# 1:1:1: profile = node = warp identity = proton-cli profile.
|
||||
# Human creates the session once: proton-cli -p <profile> account login.
|
||||
# (Credential setup itself belongs to pix; see proton-ingest design D6.)
|
||||
set -euo pipefail
|
||||
NETVM_BIN="$(cd "$(dirname "$0")" && pwd)"
|
||||
NODE="${1:?usage: netvm-proton.sh <profile> -- <proton-cli args...>}"; shift
|
||||
[ "${1:-}" = "--" ] && shift
|
||||
[ $# -gt 0 ] || { echo "usage: netvm-proton.sh <profile> -- <proton-cli args...>"; exit 1; }
|
||||
[ -f "/etc/netvm/${NODE}.conf" ] || { echo "no warp identity for '$NODE' (human: netvm-new-identity.sh $NODE)"; exit 1; }
|
||||
command -v bwrap >/dev/null 2>&1 || { echo "bwrap not found" >&2; exit 1; }
|
||||
command -v proton-cli >/dev/null 2>&1 || { echo "proton-cli not found" >&2; exit 1; }
|
||||
|
||||
PCLI_HOME="$HOME/.config/proton-cli"
|
||||
[ -d "$PCLI_HOME" ] || { echo "no proton-cli config dir (human: proton-cli account login)"; exit 1; }
|
||||
PCLI_BIN="$(readlink -f "$(command -v proton-cli)")"
|
||||
[ -x "$PCLI_BIN" ] || { echo "proton-cli binary not executable: $PCLI_BIN"; exit 1; }
|
||||
|
||||
# Jail: whole home is tmpfs except the proton-cli config (rw: sessions refresh,
|
||||
# logs), the resolved static binary (ro), and a scratch tmp. proton-cli needs
|
||||
# nothing else on disk.
|
||||
BWRAP=(bwrap --unshare-uts --unshare-ipc --die-with-parent
|
||||
--ro-bind / / --dev /dev --proc /proc --tmpfs /tmp --tmpfs /dev/shm
|
||||
--tmpfs "$HOME" --dir "$HOME/.config"
|
||||
--bind "$PCLI_HOME" "$HOME/.config/proton-cli"
|
||||
--ro-bind "$PCLI_BIN" /tmp/proton-cli
|
||||
--setenv HOME "$HOME" --setenv PATH /usr/bin:/bin
|
||||
--setenv PROTON_PROFILE "$NODE"
|
||||
--setenv PROTON_NO_INPUT 1
|
||||
--unsetenv DISPLAY --unsetenv WAYLAND_DISPLAY --unsetenv XDG_RUNTIME_DIR)
|
||||
|
||||
exec sudo -n "$NETVM_BIN/netvm-enter.sh" "$NODE" "$(id -u)" "$(id -g)" "$HOME" \
|
||||
-- "${BWRAP[@]}" -- /tmp/proton-cli "$@"
|
||||
@@ -1,81 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
# netvm-provision-node.sh <label> — provision a full client node:
|
||||
# Warp identity, netns + tunnel, chrome-box profile, registry entry.
|
||||
# Idempotent per stage: safe to re-run.
|
||||
#
|
||||
# This is the bulk-onboarding foundation: one command per client, so ten
|
||||
# signups at 9am means ten provision calls, not ten manual rituals.
|
||||
# It does NOT launch the browser — provisioning is durable infra, the
|
||||
# browser is ephemeral (launched on first onboarding, supervised by
|
||||
# agent-health.sh).
|
||||
#
|
||||
# Authorization: the user (trust root) explicitly authorized operators to
|
||||
# generate Warp identities on bl (2026-10-03). The "HUMAN-RUN ONLY" header
|
||||
# in netvm-new-identity.sh is a default, not an absolute — the user's
|
||||
# explicit instruction overrides it.
|
||||
set -euo pipefail
|
||||
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
|
||||
LABEL="${1:?usage: netvm-provision-node.sh <label>}"
|
||||
CHROME_BOX="${CHROME_BOX:-$HOME/Projects/chrome-box/chrome-box}"
|
||||
NODES_MD="$SCRIPT_DIR/../NODES.md"
|
||||
|
||||
# --- validate label -------------------------------------------------------
|
||||
if ! [[ "$LABEL" =~ ^[a-z0-9][a-z0-9-]{0,22}$ ]]; then
|
||||
echo "invalid label '$LABEL': lowercase letters, digits, hyphens (max 23 chars)" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# --- pick a CDP port ------------------------------------------------------
|
||||
# 94x0 sequence (9410 muse, 9420 pip, 9430 646, 9440 opm, ...).
|
||||
# Scan the registry and live listeners; first free wins.
|
||||
used_ports() {
|
||||
{ awk -F'|' 'NF>=5 && $5 ~ /94[0-9]0/ {gsub(/ /,"",$5); print $5}' "$NODES_MD" 2>/dev/null || true; \
|
||||
ss -tln 2>/dev/null | grep -oP ':\K94\d0\b' || true; } | sort -u
|
||||
}
|
||||
PORT=9410
|
||||
while used_ports | grep -qx "$PORT"; do PORT=$((PORT+10)); done
|
||||
[ "$PORT" -gt 9600 ] && { echo "port pool exhausted" >&2; exit 1; }
|
||||
|
||||
echo "provisioning node '$LABEL' (cdp_port=$PORT)"
|
||||
|
||||
# --- 1. Warp identity -----------------------------------------------------
|
||||
CONF="/etc/netvm/${LABEL}.conf"
|
||||
if [ -f "$CONF" ]; then
|
||||
echo "identity: exists ($CONF)"
|
||||
else
|
||||
echo "identity: generating (operator-authorized 2026-10-03)..."
|
||||
sudo -n "$SCRIPT_DIR/netvm-new-identity.sh" "$LABEL"
|
||||
fi
|
||||
|
||||
# --- 2. netns + tunnel + CDP relay ----------------------------------------
|
||||
echo "netns: bringing up..."
|
||||
sudo -n env CDP_PORT_OVERRIDE="$PORT" "$SCRIPT_DIR/netvm-node-up.sh" "$LABEL"
|
||||
|
||||
# --- 3. chrome-box profile -------------------------------------------------
|
||||
# NOTE: `chrome-box list` prints a table, not bare names — detect by the
|
||||
# profile directory instead (robust against table format changes).
|
||||
if [ ! -x "$CHROME_BOX" ]; then
|
||||
echo "chrome-box not found at $CHROME_BOX (CHROME_BOX env overrides)" >&2
|
||||
exit 1
|
||||
fi
|
||||
PROFILE_DIR="$HOME/.local/share/chrome-box/profiles/$LABEL"
|
||||
if [ -d "$PROFILE_DIR" ]; then
|
||||
echo "profile: exists ($LABEL)"
|
||||
else
|
||||
echo "profile: creating..."
|
||||
"$CHROME_BOX" create "$LABEL"
|
||||
fi
|
||||
|
||||
# --- 4. registry -----------------------------------------------------------
|
||||
if grep -qE "^\|[[:space:]]*$LABEL[[:space:]]*\|" "$NODES_MD" 2>/dev/null; then
|
||||
echo "registry: $LABEL already in NODES.md"
|
||||
else
|
||||
EGRESS=$("$SCRIPT_DIR/netvm-exec.sh" "$LABEL" -- curl -sk --max-time 10 \
|
||||
'https://1.1.1.1/cdn-cgi/trace' 2>/dev/null | grep -oP '^ip=\K.*' || echo unknown)
|
||||
printf '| %s | warp-%s | %s | %s | active | (unassigned) |\n' \
|
||||
"$LABEL" "$LABEL" "${EGRESS:-unknown}" "$PORT" >> "$NODES_MD"
|
||||
echo "registry: added $LABEL (egress=${EGRESS:-unknown})"
|
||||
fi
|
||||
|
||||
echo "done: node=$LABEL netns=warp-$LABEL cdp_port=$PORT"
|
||||
echo "next: launch browser: netvm-chrome.sh --headless --cdp-port $PORT $LABEL https://muse.ai"
|
||||
@@ -1,27 +0,0 @@
|
||||
#!/bin/bash
|
||||
# Reap stale headless browsers (no CDP activity for >4h)
|
||||
# Run via cron: */30 * * * * ~/Projects/NetVM/bin/netvm-reaper.sh
|
||||
for pid in $(pgrep -f "chromium.*--headless" 2>/dev/null); do
|
||||
# Check if CDP port is still responding (find port from cmdline)
|
||||
port=$(tr '\0' ' ' < /proc/$pid/cmdline 2>/dev/null | grep -oP 'remote-debugging-port=\K\d+' | head -1)
|
||||
if [ -n "$port" ]; then
|
||||
# Try to hit CDP (via netns if needed)
|
||||
if ! timeout 5 bash -c "cat < /dev/null > /dev/tcp/127.0.0.1/$port" 2>/dev/null; then
|
||||
echo "reaping dead browser pid $pid (port $port not responding)"
|
||||
kill -TERM $pid 2>/dev/null
|
||||
fi
|
||||
fi
|
||||
# Check age (kill if >12h old)
|
||||
age=$(($(date +%s) - $(stat -c %Y /proc/$pid 2>/dev/null || echo 0)))
|
||||
if [ "$age" -gt 43200 ]; then
|
||||
echo "reaping old browser pid $pid (age ${age}s)"
|
||||
kill -TERM $pid 2>/dev/null
|
||||
fi
|
||||
done
|
||||
# Clean up old log files
|
||||
find /tmp -name "*-smoke.log" -mtime +7 -delete 2>/dev/null
|
||||
find /tmp -name "otp_*.py" -mtime +1 -delete 2>/dev/null
|
||||
find /tmp -name "check_*.py" -mtime +1 -delete 2>/dev/null
|
||||
|
||||
# Prune unattached stale agent tmux sessions older than 2 hours (7200s)
|
||||
python3 "$(cd "$(dirname "$0")" && pwd)/muse-tmux.py" prune --ttl 7200 2>/dev/null || true
|
||||
@@ -1,87 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
netvm-registry.py — single source of truth for node -> CDP port mapping.
|
||||
|
||||
Parses NODES.md (the fleet registry). Every script that needs a node's
|
||||
CDP port reads it from here instead of hardcoding — new nodes propagate
|
||||
automatically.
|
||||
|
||||
Usage:
|
||||
from netvm_registry import port_for, load
|
||||
port_for("muse") # -> 9410 (int), None if unknown
|
||||
|
||||
CLI:
|
||||
netvm-registry.py # prints "node:port" lines for active nodes
|
||||
netvm-registry.py <node> # prints just the port (for bash)
|
||||
"""
|
||||
import hashlib
|
||||
import os
|
||||
import re
|
||||
import sys
|
||||
|
||||
REGISTRY = os.path.join(os.path.dirname(os.path.dirname(
|
||||
os.path.abspath(__file__))), "NODES.md")
|
||||
|
||||
|
||||
def load(registry_path=None):
|
||||
"""Return {node: {netns, egress_ip, cdp_port, status, agent}}."""
|
||||
path = registry_path or REGISTRY
|
||||
nodes = {}
|
||||
try:
|
||||
with open(path) as f:
|
||||
for line in f:
|
||||
line = line.rstrip("\n")
|
||||
if not line.startswith("|"):
|
||||
continue
|
||||
cells = [c.strip() for c in line.strip("|").split("|")]
|
||||
if len(cells) < 6:
|
||||
continue
|
||||
node, netns, egress, port, status, agent = cells[:6]
|
||||
if node in ("node", "") or not re.fullmatch(r"[a-z0-9-]+",
|
||||
node):
|
||||
continue
|
||||
if node.startswith("-"):
|
||||
continue
|
||||
try:
|
||||
port_n = int(port)
|
||||
except ValueError:
|
||||
continue
|
||||
tag = hashlib.sha256(node.encode()).hexdigest()[:8]
|
||||
idx = int(tag[:3], 16) % 200 + 10
|
||||
peer_ip = f"10.201.{idx}.2"
|
||||
nodes[node] = {"netns": netns, "egress_ip": egress,
|
||||
"cdp_port": port_n, "status": status,
|
||||
"agent": agent, "peer_ip": peer_ip}
|
||||
except FileNotFoundError:
|
||||
pass
|
||||
return nodes
|
||||
|
||||
|
||||
def port_for(node, registry_path=None):
|
||||
"""CDP port for a node, or None if the node isn't registered."""
|
||||
rec = load(registry_path).get(node)
|
||||
return rec["cdp_port"] if rec else None
|
||||
|
||||
|
||||
def peer_ip_for(node, registry_path=None):
|
||||
"""CDP peer IP for a node, or None if the node isn't registered."""
|
||||
rec = load(registry_path).get(node)
|
||||
return rec["peer_ip"] if rec else None
|
||||
|
||||
|
||||
def active_nodes(registry_path=None):
|
||||
"""{node: port} for nodes with status == active."""
|
||||
return {n: r["cdp_port"] for n, r in load(registry_path).items()
|
||||
if r["status"] == "active"}
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
if len(sys.argv) == 2:
|
||||
port = port_for(sys.argv[1])
|
||||
if port is None:
|
||||
print("unknown node: %s" % sys.argv[1], file=sys.stderr)
|
||||
sys.exit(1)
|
||||
print(port)
|
||||
else:
|
||||
for node, port in sorted(active_nodes().items()):
|
||||
print("%s:%d" % (node, port))
|
||||
+4
-17
@@ -12,25 +12,12 @@ for ns in $(ip netns list 2>/dev/null | awk '{print $1}' | grep '^warp-'); do
|
||||
printf 'node=%s netns=%s ifaces=%s/%s handshake=%s egress=%s\n' "$NODE" "$ns" "$WG" "$VETH" "$hs" "${egress:-?}"
|
||||
done
|
||||
iptables -t nat -L POSTROUTING -n 2>/dev/null | grep '10.201\.' || echo "(no netvm NAT rules on host)"
|
||||
echo "--- CDP relays (connectivity check; pidfile is secondary) ---"
|
||||
echo "--- CDP relays ---"
|
||||
for ns in $(ip netns list 2>/dev/null | awk '{print $1}' | grep '^warp-'); do
|
||||
netvm_names "${ns#warp-}"
|
||||
# Registry-pinned CDP ports (same mapping as cdp-relay-watchdog.sh).
|
||||
# NOTE: $CDP_PORT from netvm_names() is hash-derived and WRONG here unless
|
||||
# CDP_PORT_OVERRIDE was set at provision time — the pinned mapping is truth.
|
||||
case "$NODE" in
|
||||
muse) port=9410 ;; pip) port=9420 ;; 646) port=9430 ;; opm) port=9440 ;;
|
||||
*) port="$CDP_PORT" ;;
|
||||
esac
|
||||
target="$PEER_IP:$port"
|
||||
pidfile="/run/netvm-${NODE}-cdp-relay.pid"
|
||||
# PRIMARY verdict: actual connectivity to the relay on the veth IP.
|
||||
# pidfiles go stale (dead/recycled PIDs) and lied about status — 2026-10-04.
|
||||
if curl -s -m 5 "http://$target/json/version" 2>/dev/null | grep -q '"Browser"'; then
|
||||
echo "$NODE: relay UP ($target -> 127.0.0.1:$port)"
|
||||
elif [ -f "$pidfile" ] && kill -0 "$(cat "$pidfile" 2>/dev/null)" 2>/dev/null; then
|
||||
echo "$NODE: relay DOWN on $target (process alive per pidfile but not responding — stale/misrouted?)"
|
||||
if [ -f "/run/netvm-${NODE}-cdp-relay.pid" ] && kill -0 "$(cat "/run/netvm-${NODE}-cdp-relay.pid")" 2>/dev/null; then
|
||||
echo "$NODE: relay up ($PEER_IP:$CDP_PORT -> 127.0.0.1:$CDP_PORT)"
|
||||
else
|
||||
echo "$NODE: relay DOWN on $target (no relay process; pidfile missing or stale)"
|
||||
echo "$NODE: relay DOWN"
|
||||
fi
|
||||
done
|
||||
|
||||
@@ -1,146 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""onboard-driver.py — bl-side OTP onboarding driver.
|
||||
|
||||
Runs INSIDE the node's netns (via netvm-exec.sh). Reads the identifier
|
||||
(line 1) and OTP code (line 2, submit step only) from stdin — never argv.
|
||||
|
||||
onboard-driver.py --node muse --service muse --id-type email --step initiate [--dry-run]
|
||||
onboard-driver.py --node muse --service muse --id-type email --step submit
|
||||
|
||||
Exit codes: 0 = step done, 2 = APPROVAL_NEEDED (code sent, awaiting OTP),
|
||||
3 = NEEDS_HUMAN (multi-account selection needs a person),
|
||||
4 = NEEDS_SIGNUP (unregistered client email — client must sign up first),
|
||||
1 = failed.
|
||||
The identifier/code are passed to the local signin script as
|
||||
argv (transient, same trust domain — bl is operator infrastructure);
|
||||
they never cross a network boundary except inside the already-encrypted
|
||||
VM->bl SSH stdin pipe.
|
||||
|
||||
Part of the cred onboarding module (front-door repo, docs/CRED-MODULE.md).
|
||||
"""
|
||||
import argparse
|
||||
import datetime
|
||||
import importlib.util
|
||||
import json
|
||||
import subprocess
|
||||
import sys
|
||||
import urllib.request
|
||||
|
||||
SIGNIN = "/home/super/Projects/NetVM/bin/muse-signin.py"
|
||||
|
||||
|
||||
def _scrub(text, *secrets):
|
||||
"""Redact secret values from captured output before passthrough."""
|
||||
for s in secrets:
|
||||
if s and len(s) >= 4:
|
||||
text = text.replace(s, "[redacted]")
|
||||
return text
|
||||
|
||||
|
||||
def _mask_email(identifier):
|
||||
local, _, domain = identifier.partition("@")
|
||||
return (local[:1] + "***@" + domain) if domain else "***"
|
||||
|
||||
|
||||
def _load_registry():
|
||||
path = "/home/super/Projects/NetVM/bin/netvm-registry.py"
|
||||
spec = importlib.util.spec_from_file_location("netvm_registry", path)
|
||||
mod = importlib.util.module_from_spec(spec)
|
||||
spec.loader.exec_module(mod)
|
||||
return mod
|
||||
|
||||
|
||||
def cdp_ok(port):
|
||||
try:
|
||||
ts = json.load(urllib.request.urlopen(
|
||||
"http://127.0.0.1:%s/json/list" % port, timeout=5))
|
||||
return any(t.get("type") == "page" for t in ts)
|
||||
except Exception:
|
||||
return False
|
||||
|
||||
|
||||
def main():
|
||||
p = argparse.ArgumentParser()
|
||||
p.add_argument("--node", required=True)
|
||||
p.add_argument("--service", required=True)
|
||||
p.add_argument("--id-type", required=True)
|
||||
p.add_argument("--step", required=True, choices=["initiate", "submit"])
|
||||
p.add_argument("--dry-run", action="store_true")
|
||||
p.add_argument("--account-name", default=None,
|
||||
help="display-name hint for multi-account selection")
|
||||
args = p.parse_args()
|
||||
|
||||
if args.service != "muse" or args.id_type != "email":
|
||||
print("ERROR: unsupported service/id_type "
|
||||
"(muse+email only for now)", file=sys.stderr)
|
||||
return 1
|
||||
port = _load_registry().port_for(args.node)
|
||||
if not port:
|
||||
print("ERROR: unknown node '%s' (not in NODES.md registry)"
|
||||
% args.node, file=sys.stderr)
|
||||
return 1
|
||||
|
||||
if args.dry_run:
|
||||
# Walk the chain without sending anything: netns + CDP + page.
|
||||
if cdp_ok(port):
|
||||
print("dry-run ok: node=%s cdp=%s reachable, page present"
|
||||
% (args.node, port))
|
||||
return 0
|
||||
print("ERROR: CDP unreachable on %s" % port, file=sys.stderr)
|
||||
return 1
|
||||
|
||||
lines = sys.stdin.read().splitlines()
|
||||
identifier = lines[0].strip() if lines else ""
|
||||
code = lines[1].strip() if len(lines) > 1 else ""
|
||||
|
||||
|
||||
if not identifier:
|
||||
print("ERROR: no identifier on stdin", file=sys.stderr)
|
||||
return 1
|
||||
|
||||
cmd = [sys.executable, SIGNIN, "--node", args.node, "--email", identifier]
|
||||
if args.account_name:
|
||||
cmd += ["--account-name", args.account_name]
|
||||
if args.step == "submit":
|
||||
if not code:
|
||||
print("ERROR: no code on stdin", file=sys.stderr)
|
||||
return 1
|
||||
cmd += ["--otp", code]
|
||||
try:
|
||||
r = subprocess.run(cmd, capture_output=True, text=True,
|
||||
timeout=220)
|
||||
except subprocess.TimeoutExpired:
|
||||
# NB: TimeoutExpired str() includes the argv (with secrets) -
|
||||
# never let it reach stderr uncaught.
|
||||
print("ERROR: signin step timed out", file=sys.stderr)
|
||||
return 1
|
||||
# Propagate the signin script's contract:
|
||||
# 0 = done, 2 = OTP prompt reached, 3 = NEEDS_HUMAN, 4 = NEEDS_SIGNUP.
|
||||
# Scrub: the signin script echoes the identifier/code in progress
|
||||
# output; redact before passthrough (server scrubs too, defense
|
||||
# in depth).
|
||||
sys.stdout.write(_scrub(r.stdout, identifier, code))
|
||||
sys.stderr.write(_scrub(r.stderr, identifier, code))
|
||||
|
||||
if r.returncode == 4:
|
||||
log_entry = {
|
||||
"ts": datetime.datetime.now(datetime.timezone.utc).isoformat(),
|
||||
"type": "onboarding_needs_signup",
|
||||
"node": args.node,
|
||||
"service": args.service,
|
||||
"email_masked": _mask_email(identifier),
|
||||
"status": "needs_signup",
|
||||
"action_required": "ask_client_to_sign_up",
|
||||
"signup_url": "https://muse.ai"
|
||||
}
|
||||
try:
|
||||
with open("/home/super/Projects/NetVM/job-log.jsonl", "a") as f:
|
||||
f.write(json.dumps(log_entry) + "\n")
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
return r.returncode
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -1,95 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
P2P Relay: 646 -> muse via headless (with watermark)
|
||||
|
||||
Reads from 646's p2p-muse-outbox, writes to muse's p2p-646-inbox.
|
||||
Tracks watermark to avoid duplicate relays.
|
||||
|
||||
Usage: p2p-relay.py [--once]
|
||||
Watermark: ~/Projects/NetVM/bridge/p2p-watermark.json
|
||||
"""
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
import json
|
||||
import hashlib
|
||||
from pathlib import Path
|
||||
|
||||
API = "/home/super/Projects/NetVM/bin/muse-chat-api.py"
|
||||
EXEC_646 = "/home/super/Projects/NetVM/bin/netvm-exec.sh 646 --"
|
||||
EXEC_MUSE = "/home/super/Projects/NetVM/bin/netvm-exec.sh muse --"
|
||||
|
||||
OUTBOX_646 = "3ed3aef3-2d92-408c-9e13-f36e40a75976"
|
||||
INBOX_MUSE = "6db43088-b87d-43a3-ae2a-157acebcfa21"
|
||||
|
||||
WATERMARK_FILE = Path("/home/super/Projects/NetVM/bridge/p2p-watermark.json")
|
||||
|
||||
def load_watermark():
|
||||
if WATERMARK_FILE.exists():
|
||||
with open(WATERMARK_FILE) as f:
|
||||
return json.load(f)
|
||||
return {"last_hash": None, "last_relay": None}
|
||||
|
||||
def save_watermark(data):
|
||||
WATERMARK_FILE.parent.mkdir(parents=True, exist_ok=True)
|
||||
with open(WATERMARK_FILE, 'w') as f:
|
||||
json.dump(data, f, indent=2)
|
||||
|
||||
def run(cmd):
|
||||
result = subprocess.run(cmd, shell=True, capture_output=True, text=True, timeout=60)
|
||||
return result.stdout.strip()
|
||||
|
||||
def get_outbox_messages():
|
||||
run(f"{EXEC_646} python3 {API} --account 646 sidechat use {OUTBOX_646}")
|
||||
time.sleep(3)
|
||||
msgs = run(f"{EXEC_646} python3 {API} --account 646 messages 5")
|
||||
run(f"{EXEC_646} python3 {API} --account 646 sidechat main")
|
||||
return msgs
|
||||
|
||||
def send_to_muse(message):
|
||||
run(f"{EXEC_MUSE} python3 {API} --account muse sidechat use {INBOX_MUSE}")
|
||||
time.sleep(3)
|
||||
# Escape for shell
|
||||
safe = message.replace('"', '\\"').replace('$', '\\$').replace('`', '\\`')
|
||||
run(f'{EXEC_MUSE} python3 {API} --account muse send "{safe}"')
|
||||
run(f"{EXEC_MUSE} python3 {API} --account muse sidechat main")
|
||||
|
||||
def relay_once():
|
||||
wm = load_watermark()
|
||||
msgs = get_outbox_messages()
|
||||
|
||||
if not msgs or "P2P test" in msgs and "Hello muse" in msgs:
|
||||
# Skip if only the test message (already relayed manually)
|
||||
# Or if empty
|
||||
pass
|
||||
|
||||
# Extract messages (split by ---)
|
||||
parts = [p.strip() for p in msgs.split('---') if p.strip()]
|
||||
# Filter out system messages
|
||||
parts = [p for p in parts if p and len(p) > 10 and "P2P setup" not in p]
|
||||
|
||||
if not parts:
|
||||
print("No new messages in outbox")
|
||||
return False
|
||||
|
||||
latest = parts[-1]
|
||||
msg_hash = hashlib.md5(latest.encode()).hexdigest()[:16]
|
||||
|
||||
if msg_hash == wm.get("last_hash"):
|
||||
print("Already relayed (watermark match)")
|
||||
return False
|
||||
|
||||
print(f"Relaying new message: {latest[:80]}...")
|
||||
send_to_muse(f"[from 646] {latest[:500]}")
|
||||
|
||||
wm["last_hash"] = msg_hash
|
||||
wm["last_relay"] = time.strftime("%Y-%m-%dT%H:%M:%S")
|
||||
save_watermark(wm)
|
||||
print("Relayed and watermark updated")
|
||||
return True
|
||||
|
||||
if __name__ == '__main__':
|
||||
if '--once' in sys.argv:
|
||||
relay_once()
|
||||
else:
|
||||
print("Use --once for single relay")
|
||||
@@ -1,264 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
pipeline_engine.py — Core state ledger and orchestration helper for multi-agent pipelines.
|
||||
|
||||
Manages persistent pipeline runs in pipelines.json:
|
||||
- Run lifecycle: running -> completed | failed
|
||||
- Step transitions, timings, context passing
|
||||
- Shared pipeline sidechat target tracking
|
||||
"""
|
||||
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
import uuid
|
||||
from datetime import datetime, timezone
|
||||
from pathlib import Path
|
||||
|
||||
NETVM_ROOT = Path("/home/super/Projects/NetVM")
|
||||
PIPELINES_FILE = NETVM_ROOT / "pipelines.json"
|
||||
JOBS_DIR = NETVM_ROOT / "jobs"
|
||||
|
||||
|
||||
def utcnow():
|
||||
return datetime.now(timezone.utc).isoformat()
|
||||
|
||||
|
||||
def load_pipelines():
|
||||
if not PIPELINES_FILE.exists():
|
||||
return {}
|
||||
try:
|
||||
with open(PIPELINES_FILE, "r", encoding="utf-8") as f:
|
||||
return json.load(f)
|
||||
except Exception:
|
||||
return {}
|
||||
|
||||
|
||||
def save_pipelines(data):
|
||||
tmp = f"{PIPELINES_FILE}.tmp.{os.getpid()}"
|
||||
with open(tmp, "w", encoding="utf-8") as f:
|
||||
json.dump(data, f, indent=2)
|
||||
os.replace(tmp, PIPELINES_FILE)
|
||||
|
||||
|
||||
def generate_pipeline_run_id(pipeline_name):
|
||||
ts = datetime.now(timezone.utc).strftime("%Y%m%d-%H%M%S")
|
||||
rand_suffix = uuid.uuid4().hex[:6]
|
||||
return f"pipe-{ts}-{rand_suffix}"
|
||||
|
||||
|
||||
def create_pipeline(pipeline_name, run_id=None, custom_target=None):
|
||||
"""Initialize a new pipeline run entry in pipelines.json."""
|
||||
if not run_id:
|
||||
run_id = generate_pipeline_run_id(pipeline_name)
|
||||
|
||||
data = load_pipelines()
|
||||
target = custom_target or f"pipe-{run_id.split('-')[-1]}"
|
||||
|
||||
# Fast headless gateway provisioning: create session server-side
|
||||
thread_uuid = None
|
||||
job_file = JOBS_DIR / f"{pipeline_name}.json"
|
||||
root_agent = "opm"
|
||||
if job_file.exists():
|
||||
try:
|
||||
with open(job_file) as f:
|
||||
root_agent = json.load(f).get("agent", "opm")
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
try:
|
||||
import muse_hybrid
|
||||
res, err = muse_hybrid.start_session(root_agent, title=target)
|
||||
if res and not err:
|
||||
thread_uuid = res.get("session_id")
|
||||
# Register in job-sidechats.json so dm.py resolves it immediately
|
||||
sc_file = NETVM_ROOT / "job-sidechats.json"
|
||||
if sc_file.exists():
|
||||
with open(sc_file, "r", encoding="utf-8") as f:
|
||||
sc_data = json.load(f)
|
||||
sc_data[target] = {
|
||||
"thread_uuid": thread_uuid,
|
||||
"agent": root_agent,
|
||||
"created_at": utcnow()
|
||||
}
|
||||
tmp_sc = f"{sc_file}.tmp.{os.getpid()}"
|
||||
with open(tmp_sc, "w", encoding="utf-8") as f:
|
||||
json.dump(sc_data, f, indent=2)
|
||||
os.replace(tmp_sc, sc_file)
|
||||
except Exception as e:
|
||||
sys.stderr.write(f"warning: fast gateway session creation failed: {e}\n")
|
||||
|
||||
entry = {
|
||||
"run_id": run_id,
|
||||
"pipeline_name": pipeline_name,
|
||||
"created_at": utcnow(),
|
||||
"updated_at": utcnow(),
|
||||
"status": "running",
|
||||
"target": target,
|
||||
"thread_uuid": thread_uuid,
|
||||
"current_step": 1,
|
||||
"steps": [],
|
||||
}
|
||||
data[run_id] = entry
|
||||
save_pipelines(data)
|
||||
return entry
|
||||
|
||||
|
||||
def record_step_dispatch(run_id, step_n, job_name, job_id, agent, target, dm_id=None):
|
||||
"""Record a dispatched step in an active pipeline run."""
|
||||
data = load_pipelines()
|
||||
if run_id not in data:
|
||||
return None
|
||||
|
||||
entry = data[run_id]
|
||||
entry["updated_at"] = utcnow()
|
||||
entry["current_step"] = step_n
|
||||
|
||||
step_record = {
|
||||
"step_n": step_n,
|
||||
"job_name": job_name,
|
||||
"job_id": job_id,
|
||||
"agent": agent,
|
||||
"target": target,
|
||||
"dm_id": dm_id,
|
||||
"status": "dispatched",
|
||||
"dispatched_at": utcnow(),
|
||||
"completed_at": None,
|
||||
"result": None,
|
||||
}
|
||||
entry["steps"].append(step_record)
|
||||
save_pipelines(data)
|
||||
return step_record
|
||||
|
||||
|
||||
def record_step_result(job_id, success, result_text):
|
||||
"""Find the pipeline run for job_id, mark the step completed/failed."""
|
||||
data = load_pipelines()
|
||||
for run_id, entry in data.items():
|
||||
for step in entry.get("steps", []):
|
||||
if step.get("job_id") == job_id:
|
||||
step["status"] = "completed" if success else "failed"
|
||||
step["completed_at"] = utcnow()
|
||||
step["result"] = result_text
|
||||
entry["updated_at"] = utcnow()
|
||||
save_pipelines(data)
|
||||
return entry, step
|
||||
return None, None
|
||||
|
||||
|
||||
def record_step_timeout(dm_id):
|
||||
"""Mark step as timed_out if follow-up sweeper escalated on dm_id."""
|
||||
data = load_pipelines()
|
||||
for run_id, entry in data.items():
|
||||
if entry.get("status") != "running":
|
||||
continue
|
||||
for step in entry.get("steps", []):
|
||||
if step.get("dm_id") == dm_id and step.get("status") == "dispatched":
|
||||
step["status"] = "timed_out"
|
||||
step["completed_at"] = utcnow()
|
||||
step["result"] = "TIMEOUT: Follow-up deadline expired after all nudges"
|
||||
entry["updated_at"] = utcnow()
|
||||
save_pipelines(data)
|
||||
return entry, step
|
||||
return None, None
|
||||
|
||||
|
||||
def update_pipeline_thread(run_id, thread_uuid):
|
||||
"""Associate resolved or auto-provisioned thread UUID with pipeline."""
|
||||
data = load_pipelines()
|
||||
if run_id in data:
|
||||
data[run_id]["thread_uuid"] = thread_uuid
|
||||
data[run_id]["updated_at"] = utcnow()
|
||||
save_pipelines(data)
|
||||
|
||||
|
||||
def complete_pipeline(run_id):
|
||||
data = load_pipelines()
|
||||
if run_id in data:
|
||||
data[run_id]["status"] = "completed"
|
||||
data[run_id]["completed_at"] = utcnow()
|
||||
data[run_id]["updated_at"] = utcnow()
|
||||
save_pipelines(data)
|
||||
|
||||
|
||||
def fail_pipeline(run_id, reason="failed"):
|
||||
data = load_pipelines()
|
||||
if run_id in data:
|
||||
data[run_id]["status"] = "failed"
|
||||
data[run_id]["failed_reason"] = reason
|
||||
data[run_id]["completed_at"] = utcnow()
|
||||
data[run_id]["updated_at"] = utcnow()
|
||||
save_pipelines(data)
|
||||
|
||||
|
||||
def stop_pipeline(run_id, reason="cancelled_by_operator"):
|
||||
"""Manually cancel or stop an active pipeline run."""
|
||||
data = load_pipelines()
|
||||
# Support prefix matching
|
||||
target_key = None
|
||||
for k in data.keys():
|
||||
if k == run_id or k.startswith(run_id):
|
||||
target_key = k
|
||||
break
|
||||
|
||||
if not target_key:
|
||||
return None
|
||||
|
||||
entry = data[target_key]
|
||||
entry["status"] = "canceled"
|
||||
entry["canceled_at"] = utcnow()
|
||||
entry["updated_at"] = utcnow()
|
||||
entry["cancel_reason"] = reason
|
||||
save_pipelines(data)
|
||||
return entry
|
||||
|
||||
|
||||
def prune_pipelines(max_age_hours=24):
|
||||
"""Mark stale running pipeline runs older than max_age_hours as timed_out."""
|
||||
data = load_pipelines()
|
||||
now = datetime.now(timezone.utc)
|
||||
pruned = []
|
||||
for k, v in data.items():
|
||||
if v.get("status") == "running":
|
||||
created_str = v.get("created_at")
|
||||
if created_str:
|
||||
try:
|
||||
dt = datetime.fromisoformat(created_str.replace("Z", "+00:00"))
|
||||
diff_h = (now - dt).total_seconds() / 3600.0
|
||||
if diff_h >= max_age_hours:
|
||||
v["status"] = "timed_out"
|
||||
v["completed_at"] = utcnow()
|
||||
v["updated_at"] = utcnow()
|
||||
v["timeout_reason"] = f"stale_exceeded_{max_age_hours}h"
|
||||
pruned.append(k)
|
||||
except Exception:
|
||||
pass
|
||||
if pruned:
|
||||
save_pipelines(data)
|
||||
return pruned
|
||||
|
||||
|
||||
def get_pipeline(run_id):
|
||||
data = load_pipelines()
|
||||
if run_id in data:
|
||||
return data[run_id]
|
||||
for k, v in data.items():
|
||||
if k.startswith(run_id):
|
||||
return v
|
||||
return None
|
||||
|
||||
|
||||
def list_active_pipelines():
|
||||
data = load_pipelines()
|
||||
return [v for v in data.values() if v.get("status") == "running"]
|
||||
|
||||
|
||||
def list_pipeline_history(limit=20):
|
||||
data = load_pipelines()
|
||||
sorted_runs = sorted(
|
||||
data.values(),
|
||||
key=lambda x: x.get("created_at", ""),
|
||||
reverse=True,
|
||||
)
|
||||
return sorted_runs[:limit]
|
||||
|
||||
@@ -1,253 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""placement-audit.py -- readable audit of sidechat placement / pre-send gate
|
||||
activity in dm-log.jsonl.
|
||||
|
||||
The gate and navigation emit machine-readable events (pre_send_assert_failed,
|
||||
placement_failed, placement_mismatch, sidechat_uuid_capture_failed, nav_failed,
|
||||
...). This CLI turns them into human-readable summaries so the logs stay
|
||||
useful for troubleshooting sidechat reliability.
|
||||
|
||||
Usage:
|
||||
placement-audit.py --since 24h # summary over the last 24 hours
|
||||
placement-audit.py --since 7d # ... 7 days
|
||||
placement-audit.py --since 60m # ... 60 minutes
|
||||
placement-audit.py --watch # live tail of placement-relevant events
|
||||
|
||||
Read-only: never writes, no network, no browser.
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import re
|
||||
import sys
|
||||
import time
|
||||
from collections import Counter, defaultdict
|
||||
from datetime import datetime, timedelta, timezone
|
||||
|
||||
DM_LOG = "/home/super/Projects/NetVM/dm-log.jsonl"
|
||||
JOB_LOG = "/home/super/Projects/NetVM/job-log.jsonl"
|
||||
|
||||
# Event types that count as placement/navigation failures.
|
||||
FAILURE_TYPES = {
|
||||
"sidechat_uuid_capture_failed",
|
||||
"placement_mismatch",
|
||||
"placement_failed",
|
||||
"pre_send_assert_failed",
|
||||
"nav_failed",
|
||||
"send_failed",
|
||||
"failed",
|
||||
}
|
||||
|
||||
# Event types worth streaming in --watch (dm-log).
|
||||
WATCH_TYPES_DM = FAILURE_TYPES | {
|
||||
"verified",
|
||||
"sent",
|
||||
"nav_ok",
|
||||
"sidechat_autoprovisioned",
|
||||
}
|
||||
|
||||
|
||||
def parse_since(s):
|
||||
m = re.fullmatch(r"(\d+)(m|h|d)", (s or "").strip().lower())
|
||||
if not m:
|
||||
raise ValueError(f"bad --since value {s!r}; use like 60m, 24h, 7d")
|
||||
n, unit = int(m.group(1)), m.group(2)
|
||||
return timedelta(minutes=n) if unit == "m" else timedelta(hours=n) if unit == "h" else timedelta(days=n)
|
||||
|
||||
|
||||
def parse_ts(ts):
|
||||
if not ts:
|
||||
return None
|
||||
try:
|
||||
dt = datetime.fromisoformat(str(ts).replace("Z", "+00:00"))
|
||||
except ValueError:
|
||||
return None
|
||||
if dt.tzinfo is None:
|
||||
dt = dt.replace(tzinfo=timezone.utc)
|
||||
return dt
|
||||
|
||||
|
||||
def short_uuid(u):
|
||||
u = str(u or "")
|
||||
return u[:8] + "..." if len(u) > 12 else u
|
||||
|
||||
|
||||
def fmt_ts(e):
|
||||
dt = parse_ts(e.get("ts"))
|
||||
return dt.strftime("%m-%d %H:%M:%S") if dt else "?"
|
||||
|
||||
|
||||
def expected_actual(e):
|
||||
"""Return (expected, actual) strings for a failure event."""
|
||||
t = e.get("type")
|
||||
if t in ("pre_send_assert_failed", "placement_failed"):
|
||||
return e.get("expected_uuid"), e.get("actual_url")
|
||||
if t == "placement_mismatch":
|
||||
return e.get("expected_uuid"), e.get("actual_uuid")
|
||||
if t == "sidechat_uuid_capture_failed":
|
||||
return "(uuid capture)", e.get("out")
|
||||
if t == "nav_failed":
|
||||
return e.get("expected") or e.get("reason"), e.get("got") or e.get("out")
|
||||
if t in ("send_failed", "failed"):
|
||||
return e.get("reason"), (e.get("send_out") or e.get("err") or "")[:120]
|
||||
return None, None
|
||||
|
||||
|
||||
def one_line(e):
|
||||
"""Compact one-line summary of an event."""
|
||||
t = e.get("type", "?")
|
||||
who = f"{e.get('agent', '?')}->{e.get('to', '?')}/{e.get('target', '?')}"
|
||||
base = f"{fmt_ts(e)} {t:28s} {str(e.get('id', ''))[:8]:8s} {who}"
|
||||
if t in FAILURE_TYPES:
|
||||
reason = e.get("reason") or ""
|
||||
exp, act = expected_actual(e)
|
||||
extra = f" reason={reason}" if reason else ""
|
||||
if exp or act:
|
||||
extra += f" expected={short_uuid(exp) if exp and len(str(exp)) > 20 else exp} actual={act}"
|
||||
return base + extra
|
||||
if t == "sent":
|
||||
return base + f" verified={e.get('verified')}"
|
||||
if t == "verified":
|
||||
return base + f" placement={e.get('placement')} thread={short_uuid(e.get('thread_uuid'))}"
|
||||
if t == "nav_ok":
|
||||
return base + f" status={e.get('status')} url={e.get('browser_url')}"
|
||||
if t == "sidechat_autoprovisioned":
|
||||
return base + f" thread={short_uuid(e.get('thread_uuid'))}"
|
||||
return base
|
||||
|
||||
|
||||
def iter_log(path, cutoff=None):
|
||||
"""Yield parsed events from a jsonl file, optionally filtered by ts."""
|
||||
try:
|
||||
f = open(path, encoding="utf-8")
|
||||
except OSError as ex:
|
||||
print(f"warning: cannot open {path}: {ex}", file=sys.stderr)
|
||||
return
|
||||
with f:
|
||||
for line in f:
|
||||
line = line.strip()
|
||||
if not line:
|
||||
continue
|
||||
try:
|
||||
e = json.loads(line)
|
||||
except json.JSONDecodeError:
|
||||
continue
|
||||
if cutoff is not None:
|
||||
dt = parse_ts(e.get("ts"))
|
||||
if dt is None or dt < cutoff:
|
||||
continue
|
||||
yield e
|
||||
|
||||
|
||||
def cmd_since(args):
|
||||
try:
|
||||
delta = parse_since(args.since)
|
||||
except ValueError as ex:
|
||||
print(str(ex), file=sys.stderr)
|
||||
return 1
|
||||
cutoff = datetime.now(timezone.utc) - delta
|
||||
|
||||
sends = Counter() # (agent, target) -> sent events
|
||||
sent_ok = Counter() # (agent, target) -> sent with verified=True
|
||||
taxonomy = Counter() # failure label -> count
|
||||
failures = [] # failure events, for the recent list
|
||||
total = 0
|
||||
|
||||
for e in iter_log(DM_LOG, cutoff):
|
||||
total += 1
|
||||
t = e.get("type")
|
||||
key = (e.get("agent") or "?", e.get("target") or "?")
|
||||
if t == "sent":
|
||||
sends[key] += 1
|
||||
if e.get("verified") is True:
|
||||
sent_ok[key] += 1
|
||||
if t in FAILURE_TYPES:
|
||||
reason = e.get("reason") or ""
|
||||
label = f"{t}" + (f":{reason}" if reason else "")
|
||||
taxonomy[label] += 1
|
||||
failures.append(e)
|
||||
|
||||
# followup nudge failures live in job-log.jsonl
|
||||
for e in iter_log(JOB_LOG, cutoff):
|
||||
if e.get("type") == "followup_nudge_failed":
|
||||
taxonomy["followup_nudge_failed"] += 1
|
||||
failures.append({"type": "followup_nudge_failed",
|
||||
"ts": e.get("ts"), "id": e.get("dm_id"),
|
||||
"agent": "sweeper", "to": e.get("recipient"),
|
||||
"target": e.get("target"),
|
||||
"reason": (e.get("error") or "")[:100]})
|
||||
|
||||
print(f"== placement audit: last {args.since} (since {cutoff.strftime('%Y-%m-%d %H:%M UTC')}) ==")
|
||||
print(f"dm-log events scanned: {total}")
|
||||
print()
|
||||
print("-- per-target sends --")
|
||||
print(f"{'agent':10s} {'target':28s} {'sent':>5s} {'verified':>8s} {'unver':>6s}")
|
||||
for (agent, target), n in sorted(sends.items(), key=lambda kv: -kv[1]):
|
||||
ok = sent_ok.get((agent, target), 0)
|
||||
print(f"{agent:10s} {target:28s} {n:5d} {ok:8d} {n - ok:6d}")
|
||||
if not sends:
|
||||
print("(no sends in window)")
|
||||
print()
|
||||
print("-- failure taxonomy --")
|
||||
if taxonomy:
|
||||
for label, n in taxonomy.most_common():
|
||||
print(f"{n:5d} {label}")
|
||||
else:
|
||||
print("(no placement failures in window)")
|
||||
print()
|
||||
print("-- 10 most recent failures --")
|
||||
failures.sort(key=lambda e: parse_ts(e.get("ts")) or datetime.min.replace(tzinfo=timezone.utc),
|
||||
reverse=True)
|
||||
for e in failures[:10]:
|
||||
print(one_line(e))
|
||||
if not failures:
|
||||
print("(none)")
|
||||
return 0
|
||||
|
||||
|
||||
def cmd_watch(_args):
|
||||
print("watching dm-log.jsonl for placement events (Ctrl-C to stop)...", file=sys.stderr)
|
||||
try:
|
||||
f = open(DM_LOG, encoding="utf-8")
|
||||
except OSError as ex:
|
||||
print(f"cannot open {DM_LOG}: {ex}", file=sys.stderr)
|
||||
return 1
|
||||
with f:
|
||||
f.seek(0, 2) # start at end: live view only
|
||||
try:
|
||||
while True:
|
||||
line = f.readline()
|
||||
if not line:
|
||||
time.sleep(2)
|
||||
continue
|
||||
line = line.strip()
|
||||
if not line:
|
||||
continue
|
||||
try:
|
||||
e = json.loads(line)
|
||||
except json.JSONDecodeError:
|
||||
continue
|
||||
if e.get("type") in WATCH_TYPES_DM:
|
||||
print(one_line(e), flush=True)
|
||||
except KeyboardInterrupt:
|
||||
print("\nstopped.", file=sys.stderr)
|
||||
return 0
|
||||
|
||||
|
||||
def main(argv=None):
|
||||
ap = argparse.ArgumentParser(description="Audit sidechat placement / gate events in dm-log.jsonl")
|
||||
ap.add_argument("--since", metavar="60m|24h|7d",
|
||||
help="summarize events newer than this (e.g. 60m, 24h, 7d)")
|
||||
ap.add_argument("--watch", action="store_true",
|
||||
help="live tail of placement-relevant events")
|
||||
args = ap.parse_args(argv)
|
||||
if args.watch:
|
||||
return cmd_watch(args)
|
||||
if args.since:
|
||||
return cmd_since(args)
|
||||
ap.print_help()
|
||||
return 2
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -1,121 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""prompt_envelope.py — wraps every dispatched job prompt so agents DO work.
|
||||
|
||||
Why: agents treated job DMs as informational and replied with prose; swarms and
|
||||
timers never got created. The envelope puts an executable subagent-spawn tool
|
||||
call at the TOP (before the task) and repeats it at the BOTTOM (after the task),
|
||||
with a work-first mandate and explicit tool/curl surface in between.
|
||||
|
||||
Profiles tune the spawn recipe per job family; add new ones to PROFILES.
|
||||
"""
|
||||
import json
|
||||
import re
|
||||
import hashlib
|
||||
|
||||
BOX_API = "https://box.muse-dev.online/api/box"
|
||||
|
||||
|
||||
def _response_rule():
|
||||
return (
|
||||
"\nRESPONSE RULE: Execute your steps using [TOOL ...] directives or background tmux commands."
|
||||
" When complete, conclude your output with the [RESULT ...] line so the harvester records it.\n"
|
||||
)
|
||||
|
||||
|
||||
def _box_surface_block(agent, job_name, job_id, target):
|
||||
"""Clean runtime context block for the agent."""
|
||||
header = (
|
||||
"\n---- BOX RUNTIME ----\n"
|
||||
"THREAD: " + str(target) + "\n"
|
||||
"RUNTIME: https://box.muse-dev.online/\n"
|
||||
"JOB: " + str(job_id) + "\n"
|
||||
"AGENT: " + str(agent) + "\n"
|
||||
)
|
||||
return header + _response_rule()
|
||||
|
||||
|
||||
UUID_RE = re.compile(r"[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}")
|
||||
|
||||
# profile -> (spawn count, followup minutes, spawn task hint)
|
||||
PROFILES = {
|
||||
"pulse": (2, 30, "execute your scope's standing work: set timers, verify them, report per-slot verdicts"),
|
||||
"health": (2, 60, "run the prescribed health gates read-only and report per-gate verdicts"),
|
||||
"swarm": (2, 20, "sweep swarms and sidechats read-only and report stale or unarchived items"),
|
||||
"work": (2, 15, "split the task above into two independent halves and complete each"),
|
||||
}
|
||||
|
||||
|
||||
def pick_profile(job_name):
|
||||
n = job_name or ""
|
||||
if n.startswith("autonomy-pulse"):
|
||||
return "pulse"
|
||||
if "health" in n:
|
||||
return "health"
|
||||
if "swarm" in n:
|
||||
return "swarm"
|
||||
return "work"
|
||||
|
||||
|
||||
def _tool(op, args):
|
||||
# no ']' inside the JSON: the harvester's [TOOL ...] regex stops at the first one
|
||||
return "[TOOL %s %s]" % (op, json.dumps(args, separators=(", ", ": ")))
|
||||
|
||||
|
||||
def spawn_call(job_id, job_name, profile):
|
||||
count, _, hint = PROFILES[profile]
|
||||
# no brackets in the task text: the harvester's [TOOL ...] regex stops at the first ']'
|
||||
task = "Subagent for job %s (%s): %s." % (job_id, job_name, hint)
|
||||
return _tool("swarm.spawn", {"count": count, "task": task[:900], "label": (job_name or "job")[:60]})
|
||||
|
||||
|
||||
def native_followup_call(job_id, job_name, profile, agent):
|
||||
"""Muse-native one-shot cron (cron.create runonce) - bridged to followup.create."""
|
||||
_, mins, _ = PROFILES[profile]
|
||||
return _tool("cron.create", {
|
||||
"kind": "runonce", "agent": agent, "in_m": mins,
|
||||
"prompt": "Continue job %s (%s): verify with cron.runs and swarm.list then act or close." % (job_id, job_name)})
|
||||
|
||||
|
||||
def followup_call(job_id, job_name, profile, agent):
|
||||
_, mins, _ = PROFILES[profile]
|
||||
return _tool("followup.create", {
|
||||
"agent": agent, "in_m": mins,
|
||||
"prompt": "Continue job %s (%s): check swarm results with swarm.list then act or close." % (job_id, job_name)})
|
||||
|
||||
|
||||
def thread_url(target):
|
||||
t = str(target)
|
||||
return "https://box.muse-dev.online/%s/%s" % ("thread" if UUID_RE.fullmatch(t.lower()) else "sidechat", t)
|
||||
|
||||
|
||||
def wrap(job_name, job_id, agent, target, rendered):
|
||||
"""Return the full DM body as a direct, authentic operator directive."""
|
||||
profile = pick_profile(job_name)
|
||||
spawn = spawn_call(job_id, job_name, profile)
|
||||
follow = native_followup_call(job_id, job_name, profile, agent)
|
||||
parts = job_id.split("-")
|
||||
wo_id = parts[-1] if len(parts) > 1 and len(parts[-1]) >= 8 else hashlib.sha256(job_id.encode()).hexdigest()[:8]
|
||||
session_name = f"work-{agent}-{wo_id}"
|
||||
tmux_new = _tool("tmux.new", {"session": session_name, "command": "bash"})
|
||||
tmux_send = _tool("tmux.send", {"session": session_name, "keys": "echo 'Starting task execution...'"})
|
||||
top = (
|
||||
f"Operator Directive [ref:{wo_id}]:\n"
|
||||
"Execute the task below using tool calls. Background tmux session is ready for command execution:\n"
|
||||
f" • {tmux_new}\n"
|
||||
f" • {tmux_send}\n"
|
||||
f" • Subagent assistance: {spawn}\n"
|
||||
f" • Verification schedule: {follow}\n\n"
|
||||
"--- Task ---\n"
|
||||
)
|
||||
has_result = "[RESULT" in rendered
|
||||
bottom = (
|
||||
"\n--- End Task ---\n\n"
|
||||
f"Inspect tmux output: [TOOL tmux.capture {{\"session\": \"{session_name}\", \"lines\": 30}}]\n"
|
||||
"Tools available: cron.create, cron.runs, health.check, swarm.spawn, swarm.list.\n"
|
||||
)
|
||||
if not has_result:
|
||||
bottom += f"When complete, report your verdict: [RESULT {job_id}] OK: <summary of actions>\n"
|
||||
else:
|
||||
bottom += f"Conclude with your [RESULT {job_id}] line reporting outcomes.\n"
|
||||
return top + rendered.strip() + "\n" + bottom
|
||||
|
||||
@@ -1,172 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Shared rate limiter for NetVM scripts.
|
||||
|
||||
Any .py script can use this to avoid hitting Muse rate limits:
|
||||
from rate_limiter import rate_limit_wait
|
||||
rate_limit_wait("opm") # blocks until allowed
|
||||
|
||||
Uses a token bucket per agent stored in /tmp (persists across invocations,
|
||||
not across reboots). Default: 1 op per 3s sustained, burst of 5, max ~20/min.
|
||||
|
||||
Jitter (2026-10-04): each agent gets a DETERMINISTIC per-agent jitter factor
|
||||
derived from sha256(salt + agent), in [1-jitter, 1+jitter] (default
|
||||
jitter=0.3, i.e. +/-30%). Rationale: all 4 fleet nodes egress from a single
|
||||
Cloudflare IP, so Cloudflare sees correlated traffic. When fleet timers fire
|
||||
simultaneously, per-agent jitter drifts each node's send phase apart instead
|
||||
of hitting in lockstep. Deterministic per agent = reproducible behavior;
|
||||
different agents = de-correlated phases. Does not make limits stricter: the
|
||||
mean interval is unchanged.
|
||||
"""
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import random
|
||||
import time
|
||||
|
||||
RATE_LIMIT_FILE = "/tmp/netvm-rate-limit.json"
|
||||
RATE_LIMIT_INTERVAL = 3.0 # seconds between ops (base; jittered per agent)
|
||||
RATE_LIMIT_BURST = 5
|
||||
RATE_LIMIT_MAX_PER_MIN = 20
|
||||
DEFAULT_JITTER = 0.3 # +/-30% deterministic per-agent interval jitter
|
||||
|
||||
JITTER_SEED_SALT = "netvm-rate-jitter:v1:"
|
||||
|
||||
|
||||
def _jitter_factor(agent, jitter=DEFAULT_JITTER):
|
||||
"""Deterministic per-agent multiplier in [1-jitter, 1+jitter].
|
||||
|
||||
Seeded by sha256(salt + agent): the same agent always gets the same
|
||||
factor (reproducible), different agents get different factors
|
||||
(de-correlated). Uses an isolated Random instance; global random
|
||||
state is untouched.
|
||||
"""
|
||||
if jitter <= 0:
|
||||
return 1.0
|
||||
digest = hashlib.sha256((JITTER_SEED_SALT + agent).encode()).digest()
|
||||
seed = int.from_bytes(digest[:8], "big")
|
||||
rng = random.Random(seed)
|
||||
return 1.0 + jitter * (rng.random() * 2.0 - 1.0)
|
||||
|
||||
|
||||
def effective_interval(agent, interval=RATE_LIMIT_INTERVAL,
|
||||
jitter=DEFAULT_JITTER):
|
||||
"""The actual minimum op spacing for this agent after jitter."""
|
||||
return interval * _jitter_factor(agent, jitter)
|
||||
|
||||
|
||||
def rate_limit_policy(agent, interval=RATE_LIMIT_INTERVAL,
|
||||
burst=RATE_LIMIT_BURST, jitter=DEFAULT_JITTER):
|
||||
"""Return the effective rate-limit policy for an agent (audit/docs)."""
|
||||
return {
|
||||
"agent": agent,
|
||||
"base_interval_s": interval,
|
||||
"jitter": jitter,
|
||||
"jitter_factor": round(_jitter_factor(agent, jitter), 4),
|
||||
"effective_interval_s": round(
|
||||
effective_interval(agent, interval, jitter), 3),
|
||||
"burst": burst,
|
||||
"max_per_min": RATE_LIMIT_MAX_PER_MIN,
|
||||
"state_file": RATE_LIMIT_FILE,
|
||||
"scope": ("per-agent buckets; state shared in one file, "
|
||||
"keys namespaced by agent"),
|
||||
}
|
||||
|
||||
|
||||
def _load_state():
|
||||
try:
|
||||
with open(RATE_LIMIT_FILE) as f:
|
||||
return json.load(f)
|
||||
except:
|
||||
return {}
|
||||
|
||||
|
||||
def _save_state(state):
|
||||
try:
|
||||
# Atomic write via temp file
|
||||
tmp = RATE_LIMIT_FILE + ".tmp"
|
||||
with open(tmp, "w") as f:
|
||||
json.dump(state, f)
|
||||
os.rename(tmp, RATE_LIMIT_FILE)
|
||||
except:
|
||||
pass
|
||||
|
||||
|
||||
def rate_limit_wait(agent, interval=RATE_LIMIT_INTERVAL,
|
||||
burst=RATE_LIMIT_BURST, jitter=DEFAULT_JITTER):
|
||||
"""
|
||||
Block until the agent is allowed to perform an operation.
|
||||
Returns the time waited in seconds (0 if no wait needed).
|
||||
|
||||
The interval is jittered deterministically per agent so fleet nodes
|
||||
don't send in lockstep.
|
||||
"""
|
||||
eff_interval = effective_interval(agent, interval, jitter)
|
||||
state = _load_state()
|
||||
now = time.time()
|
||||
|
||||
# Prune old entries
|
||||
recent_key = f"{agent}_recent"
|
||||
recent = state.get(recent_key, [])
|
||||
recent = [t for t in recent if now - t < 60]
|
||||
|
||||
# Check burst limit (max per minute)
|
||||
if len(recent) >= RATE_LIMIT_MAX_PER_MIN:
|
||||
wait = 60 - (now - recent[0]) + 1
|
||||
if wait > 0:
|
||||
time.sleep(wait)
|
||||
# Refresh after wait
|
||||
state = _load_state()
|
||||
recent = state.get(recent_key, [])
|
||||
recent = [t for t in recent if time.time() - t < 60]
|
||||
|
||||
# Check interval limit (jittered per agent)
|
||||
last = state.get(agent, 0)
|
||||
now = time.time()
|
||||
if now - last < eff_interval:
|
||||
wait = eff_interval - (now - last)
|
||||
time.sleep(wait)
|
||||
|
||||
# Record this operation
|
||||
now = time.time()
|
||||
state[agent] = now
|
||||
recent.append(now)
|
||||
state[recent_key] = recent
|
||||
_save_state(state)
|
||||
return 0
|
||||
|
||||
|
||||
def rate_limit_check(agent, interval=RATE_LIMIT_INTERVAL,
|
||||
jitter=DEFAULT_JITTER):
|
||||
"""
|
||||
Non-blocking check. Returns (allowed: bool, wait_seconds: float).
|
||||
"""
|
||||
eff_interval = effective_interval(agent, interval, jitter)
|
||||
state = _load_state()
|
||||
now = time.time()
|
||||
recent = state.get(f"{agent}_recent", [])
|
||||
recent = [t for t in recent if now - t < 60]
|
||||
|
||||
if len(recent) >= RATE_LIMIT_MAX_PER_MIN:
|
||||
wait = 60 - (now - recent[0]) + 1
|
||||
return False, max(0, wait)
|
||||
|
||||
last = state.get(agent, 0)
|
||||
if now - last < eff_interval:
|
||||
return False, eff_interval - (now - last)
|
||||
|
||||
return True, 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
import argparse
|
||||
p = argparse.ArgumentParser(
|
||||
description="Inspect NetVM per-agent rate-limit policy")
|
||||
p.add_argument("--policy", nargs="*", default=["muse", "pip", "646", "opm"],
|
||||
help="Show effective policy per agent")
|
||||
p.add_argument("--no-jitter", action="store_true",
|
||||
help="Show policy without jitter")
|
||||
args = p.parse_args()
|
||||
j = 0.0 if args.no_jitter else DEFAULT_JITTER
|
||||
for a in args.policy:
|
||||
print(json.dumps(rate_limit_policy(a, jitter=j), indent=2))
|
||||
@@ -1,106 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
refresh-node-cookies.py: Extract fresh cookies from a running node's Chromium CDP
|
||||
instance inside its isolated network namespace and save to ~/.config/muse-cli/<node>/cookies.txt.
|
||||
"""
|
||||
import sys
|
||||
import os
|
||||
import json
|
||||
import time
|
||||
import subprocess
|
||||
import urllib.request
|
||||
from pathlib import Path
|
||||
|
||||
NODES = {
|
||||
"muse": 9410,
|
||||
"pip": 9420,
|
||||
"646": 9430,
|
||||
"opm": 9440,
|
||||
"def": 9450,
|
||||
"dev": 9455,
|
||||
}
|
||||
|
||||
def export_cookies(node):
|
||||
port = NODES.get(node)
|
||||
if not port:
|
||||
print(f"Unknown node: {node}", file=sys.stderr)
|
||||
return False
|
||||
|
||||
conf_dir = Path.home() / ".config" / "muse-cli" / node
|
||||
conf_dir.mkdir(parents=True, exist_ok=True)
|
||||
cookie_file = conf_dir / "cookies.txt"
|
||||
cfg_file = conf_dir / "config.json"
|
||||
|
||||
# We import websocket inside netns execution
|
||||
try:
|
||||
import websocket
|
||||
except ImportError:
|
||||
print("websocket-client not installed", file=sys.stderr)
|
||||
return False
|
||||
|
||||
# Check if CDP port is responding; if not, attempt background browser launch
|
||||
try:
|
||||
urllib.request.urlopen(f"http://127.0.0.1:{port}/json", timeout=2)
|
||||
except Exception:
|
||||
print(f"CDP unreachable on port {port}. Attempting background browser launch for {node}...", file=sys.stderr)
|
||||
try:
|
||||
subprocess.Popen([
|
||||
str(Path(__file__).resolve().parent / "netvm-chrome.sh"),
|
||||
"--headless", node, "https://muse.ai"
|
||||
], stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
|
||||
# Give it up to 6 seconds to bind
|
||||
for _ in range(12):
|
||||
time.sleep(0.5)
|
||||
try:
|
||||
urllib.request.urlopen(f"http://127.0.0.1:{port}/json", timeout=1)
|
||||
break
|
||||
except Exception:
|
||||
pass
|
||||
except Exception as e:
|
||||
print(f"Failed to launch browser: {e}", file=sys.stderr)
|
||||
|
||||
try:
|
||||
req = urllib.request.urlopen(f"http://127.0.0.1:{port}/json", timeout=3)
|
||||
tabs = json.loads(req.read().decode())
|
||||
if not tabs:
|
||||
print(f"No tabs found on port {port}", file=sys.stderr)
|
||||
return False
|
||||
ws_url = tabs[0]["webSocketDebuggerUrl"]
|
||||
ws = websocket.create_connection(ws_url, timeout=5)
|
||||
ws.send(json.dumps({
|
||||
"id": 1,
|
||||
"method": "Network.getCookies",
|
||||
"params": {"urls": ["https://muse.ai"]}
|
||||
}))
|
||||
res = json.loads(ws.recv())
|
||||
ws.close()
|
||||
|
||||
cookies = res.get("result", {}).get("cookies", [])
|
||||
if not cookies:
|
||||
print(f"No cookies returned from CDP for {node}", file=sys.stderr)
|
||||
return False
|
||||
|
||||
lines = [f"{c['name']}={c['value']}" for c in cookies]
|
||||
cookie_file.write_text("; ".join(lines) + "\n")
|
||||
cookie_file.chmod(0o600)
|
||||
|
||||
cfg_data = {}
|
||||
if cfg_file.exists():
|
||||
try:
|
||||
cfg_data = json.loads(cfg_file.read_text())
|
||||
except Exception:
|
||||
pass
|
||||
cfg_data["cookies_file"] = str(cookie_file)
|
||||
cfg_file.write_text(json.dumps(cfg_data, indent=2))
|
||||
cfg_file.chmod(0o600)
|
||||
return True
|
||||
except Exception as e:
|
||||
print(f"Failed to export cookies for {node}: {e}", file=sys.stderr)
|
||||
return False
|
||||
|
||||
if __name__ == "__main__":
|
||||
if len(sys.argv) < 2:
|
||||
print("Usage: refresh-node-cookies.py <node>", file=sys.stderr)
|
||||
sys.exit(1)
|
||||
success = export_cookies(sys.argv[1])
|
||||
sys.exit(0 if success else 1)
|
||||
@@ -1,26 +0,0 @@
|
||||
#!/bin/bash
|
||||
# relay-health-check.sh — check all four CDP relay endpoints on bl.
|
||||
# Self-contained: no nested SSH quoting. Sources pinned ports from netvm-names.sh.
|
||||
#
|
||||
# Output: "name:code" per relay on stdout.
|
||||
# Exit 0 if all return 200; exit 1 otherwise (lists failures on stderr).
|
||||
#
|
||||
# Called by the cdp-relay-health-monitor cron and by `box-ctl.py relay-health`.
|
||||
|
||||
set -u
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
# shellcheck source=/dev/null
|
||||
source "$SCRIPT_DIR/netvm-names.sh"
|
||||
|
||||
FAILED=0
|
||||
for node in muse pip 646 opm; do
|
||||
netvm_names "$node"
|
||||
url="http://${PEER_IP}:${CDP_PORT}/json/version"
|
||||
code=$(curl -s -m 8 -o /dev/null -w "%{http_code}" "$url" 2>/dev/null || echo "000")
|
||||
echo "${node}:${code}"
|
||||
if [ "$code" != "200" ]; then
|
||||
echo "FAIL: ${node} relay ${PEER_IP}:${CDP_PORT} returned ${code}" >&2
|
||||
FAILED=1
|
||||
fi
|
||||
done
|
||||
exit $FAILED
|
||||
File diff suppressed because it is too large
Load Diff
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user