operator
71 lines
3.4 KiB
Markdown
71 lines
3.4 KiB
Markdown
# Retention Piece 2 — Job Definition Archival & Pruning
|
|
|
|
Owner: operator-646
|
|
Branch: `dev/operator-646/retention-rotations-p1`
|
|
Date: 2026-10-07
|
|
|
|
Extends the fleet retention framework from Piece 1 ([RETENTION-ROTATIONS-P1.md](file:///home/super/Projects/NetVM/docs/RETENTION-ROTATIONS-P1.md)) to prune and archive retired manual job definitions from `jobs/` into `jobs/archive/`.
|
|
|
|
## Problem
|
|
|
|
The root `jobs/` directory accumulated over 235 job definition files, primarily composed of retired `auto-work-*` exploratory batches and one-off test pipelines. Every invocation of `job-scheduler.py` (`glob.glob("jobs/*.json")`) and CLI tools was parsing all 235 files, causing unnecessary filesystem overhead and clutter.
|
|
|
|
## Policy Implemented
|
|
|
|
| Domain | Eligibility Criteria | Action | Target Layout |
|
|
|:---|:---|:---|:---|
|
|
| `jobs/*.json` | `schedule == "manual"` or empty, AND not protected | `git mv` archive | `jobs/archive/<name>.json` (plain JSON, preserves history) |
|
|
|
|
### Dynamic Safety Guardrails
|
|
|
|
A job is **never** archived automatically if any of the following hold:
|
|
1. **Systemd User Units**: Registered as an `ExecStart` target in `~/.config/systemd/user/job-*.service` (e.g. `work-finder`, `heartbeat`, `muse-auditor`, `opm-swarm-harvest`, `autonomy-pulse-*`).
|
|
2. **Active Pipelines**: Associated with a running, dispatched, or pending run in `pipelines.json` (e.g. `ops-audit-*`, `pipe-demo-*`).
|
|
3. **Chained Dependencies**: Linked via `chain_next`, `on_success`, or `on_failure` from any active protected job.
|
|
4. **Core Fleet Baseline**: Hardcoded protective baseline: `heartbeat`, `refine-system`, `canary-test`.
|
|
|
|
## Tooling & Architecture
|
|
|
|
- `bin/retention-archive-jobs.py`: Standalone CLI driver supporting `scan`, `archive`, `unarchive`, and `list` with `--dry-run`, `--force`, and `--json`.
|
|
- `bin/retention-run-rotations.sh`: Hourly rotation driver now runs `retention-archive-jobs.py scan` as the 4th phase of the retention pipeline.
|
|
- `bin/box-ctl.py`:
|
|
- `job-list`: Excludes archived jobs by default; includes them when passed `--archived` / `-a`.
|
|
- `job-archive <name>`: Archives a job via `retention-archive-jobs.py`.
|
|
- `job-unarchive <name>`: Restores an archived job to active `jobs/`.
|
|
- `bin/super-cli.py`: Exposes `box job archive <name>`, `box job unarchive <name>`, and `box job list --archived`.
|
|
- `bin/job-dispatch.py`: Guards against direct dispatch of archived jobs; fails fast with an explicit unarchive prompt.
|
|
|
|
## Operations & Verification
|
|
|
|
Run on-demand scan:
|
|
```bash
|
|
bin/retention-archive-jobs.py scan [--dry-run]
|
|
```
|
|
|
|
Manual archive / unarchive:
|
|
```bash
|
|
box job archive <name>
|
|
box job unarchive <name>
|
|
```
|
|
|
|
List archived jobs:
|
|
```bash
|
|
box job list --archived
|
|
```
|
|
|
|
Execute full retention cycle:
|
|
```bash
|
|
bin/retention-run-rotations.sh
|
|
```
|
|
|
|
## First Run Results (2026-10-07)
|
|
|
|
- Total jobs scanned: 235
|
|
- Dynamically protected jobs: 13
|
|
- Eligible retired jobs identified: 17
|
|
- Jobs archived to `jobs/archive/`:
|
|
`646-opm-watch`, `646-pip-sync`, `646-sidechat-task`, `auto-work-queue-f02`, `auto-work-queue-f06`, `auto-work-queue-f10`, `auto-work-queue-f14`, `auto-work-queue-f18`, `auto-work-xop-e02`, `auto-work-xop-e06`, `auto-work-xop-e10`, `auto-work-xop-e14`, `auto-work-xop-e18`, `mainloop-p1-pilot`, `mainloop-p2-noswitcher`, `mainloop-p3-bridge`, `mainloop-p4-steady`.
|
|
- Committed in git as: `fff5556` (`chore(retention): archive retired jobs [...]`).
|
|
- Active jobs remaining: 218.
|
|
- Unit tests: 10 tests in `tests/test_retention_archive_jobs.py` (all green).
|