operator
1e86ed8a44
- chat-history.jsonl rotates at 10MB or 7d -> logs/archive/*.jsonl.gz + .sha256 - job-log.jsonl rotates at 2MB or 14d -> same archive layout - followups.json archives resolved>7d / escalated>30d -> followups.archive.jsonl (append-only) - verify wrapper: sha256 -c, gzip -t, clean listing, spot-extract JSON + ts bounds - driver + hourly systemd user timer (retention-rotations.timer) - archive-only: nothing is ever deleted; thread backfill excluded (needs human-reviewed preview) First run 2026-10-05 16:35Z: chat-history 29.7MB + job-log 4.7MB rotated and verified; followups 0 eligible of 163.
82 lines
4.1 KiB
Markdown
82 lines
4.1 KiB
Markdown
# Retention piece 1 — immediate rotations (board bug b67084660385)
|
|
|
|
Owner: operator-646 (per opm verdict GO WITH MODIFICATIONS, 2026-10-05).
|
|
Branch: `dev/operator-646/retention-rotations-p1`.
|
|
|
|
Scope is deliberately narrow: the three files whose thresholds were already
|
|
exceeded on bl (opm-verified 2026-10-05 ~10:35 UTC). Everything else from the
|
|
retention draft — the 344-thread backfill (needs human-reviewed preview),
|
|
jobs/ archival, subagent-session state fix, swarm blob summarizer, dispatch
|
|
reuse-key hardening — is a later piece.
|
|
|
|
## Policy implemented
|
|
|
|
| File | Threshold (opm-verified) | Action | Archive layout |
|
|
|------|--------------------------|--------|----------------|
|
|
| `logs/chat-history.jsonl` | 10MB or 7d | rotate | `logs/archive/chat-history-YYYYMMDD-HHMMSS.jsonl.gz` + `.sha256` |
|
|
| `job-log.jsonl` | 2MB or 14d | rotate | `logs/archive/job-log-YYYYMMDD-HHMMSS.jsonl.gz` + `.sha256` |
|
|
| `followups.json` | resolved >7d / escalated >30d | archive records | `followups.archive.jsonl` (append-only) |
|
|
|
|
Archive-only: no script here deletes anything. The draft's 180d hard-delete
|
|
of archive entries is explicitly NOT implemented in this piece.
|
|
|
|
## Files
|
|
|
|
- `bin/retention-rotate-chat-history.sh` — 10MB/7d rotate
|
|
- `bin/retention-rotate-job-log.sh` — 2MB/14d rotate
|
|
- `bin/retention-archive-followups.py` — resolved>7d / escalated>30d archival
|
|
- `bin/retention-verify-archive.sh <path>` — checksum, `gzip -t`, clean
|
|
listing (`gzip -l`), spot-extract (first/last 3 lines valid JSON + ts
|
|
bounds), line counts
|
|
- `bin/retention-run-rotations.sh` — driver: runs all three, verifies the
|
|
newest archives + the cumulative followups archive, appends a full report
|
|
to `logs/retention-runs/retention-run-TS.log`
|
|
- `systemd/retention-rotations.service` + `systemd/retention-rotations.timer`
|
|
— hourly user timer (`OnCalendar=hourly`, Persistent), same shape as
|
|
`followup-sweeper.timer`
|
|
|
|
## Live-writer safety
|
|
|
|
- `chat-history.jsonl`: the harvester opens the file in append mode per write
|
|
(`bin/response-harvester.py`, `with open(path, "a")`), so the script's
|
|
atomic `mv` of the live file is safe — the next write recreates it. The
|
|
script also `touch`es the new live file.
|
|
- `job-log.jsonl`: same pattern (`bin/job-dispatch.py`, `bin/followup-sweeper.py`).
|
|
- `followups.json`: both this script and `bin/followup-sweeper.py` write
|
|
atomically (tmp + rename). This script snapshots `(mtime_ns, size)` before
|
|
deciding the archival set and aborts (rc=2) if the sweeper rewrote the
|
|
file mid-decision; the next hourly run picks it up.
|
|
|
|
## Operations
|
|
|
|
Run once now (or any time): `bin/retention-run-rotations.sh`
|
|
Dry check without acting: each rotate script prints SKIP when under threshold.
|
|
Verify one archive: `bin/retention-verify-archive.sh <path>`
|
|
|
|
Install the timer (bl, user units):
|
|
```
|
|
cp systemd/retention-rotations.{service,timer} ~/.config/systemd/user/
|
|
systemctl --user daemon-reload
|
|
systemctl --user enable --now retention-rotations.timer
|
|
systemctl --user list-timers | grep retention
|
|
```
|
|
Uninstall: `systemctl --user disable --now retention-rotations.timer`
|
|
|
|
## First-run report (2026-10-05, run by operator-646)
|
|
|
|
- chat-history.jsonl: 29,746,435 B (25,240 lines) -> rotated to
|
|
`logs/archive/chat-history-20261005-163558.jsonl.gz` (5,904,954 B compressed,
|
|
80.1% ratio). Verify: sha256 OK, gzip -t OK, clean listing, spot-extract
|
|
first/last 3 lines valid JSON (25,240 lines), live file 0 B after rotation
|
|
and growing again within a minute (writers healthy on the new file).
|
|
- job-log.jsonl: 4,724,788 B (16,143 lines) -> rotated to
|
|
`logs/archive/job-log-20261005-163559.jsonl.gz` (594,762 B compressed,
|
|
87.4% ratio). Same verification, all OK. Live file already 287 B seconds
|
|
after rotation (job-dispatch writes flowing).
|
|
- followups.json: 163 records (161 resolved, 1 escalated, 1 pending),
|
|
0 eligible under resolved>7d / escalated>30d -> no archival, machinery
|
|
verified by dry logic + schema check; `followups.archive.jsonl` not
|
|
created yet (created on first eligible archival run)
|
|
- Timer installed and enabled; first hourly run after install is a no-op
|
|
unless thresholds are exceeded again.
|