mirror of
https://github.com/mvschwarz/openrig.git
synced 2026-11-09 03:39:51 -10:00
Compare commits
7
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
f6180805c3 | ||
|
|
0d9e843856 | ||
|
|
5b43c103e6 | ||
|
|
16e2f1e88c | ||
|
|
f46cba5295 | ||
|
|
e834607350 | ||
|
|
0cb06f8b1c |
No files matched your search
@@ -511,6 +511,7 @@ default-permit record (tag `runner-permission-permit`) opens no row.
|
||||
| `rig queue inbox-pending <destinationSession>` | — | `--json` |
|
||||
| `rig queue outbox-record` | — | `--sender <session>`<br>`--destination <session>` **required**<br>`--body <text>`<br>`--body-file <path>`<br>`--tags <tags>`<br>`--urgency <urgency>`<br>`--audit <pointer>`<br>`--id <outboxId>`<br>`--json` |
|
||||
| `rig queue outbox-list <senderSession>` | — | `--limit <n>`<br>`--json` |
|
||||
| `rig queue erase` | — | `--members <file>`<br>`--rig <id>`<br>`--ids <file>`<br>`--action <action>` (`delete`, `scrub`)<br>`--routine`<br>`--apply`<br>`--actor <session>`<br>`--json` |
|
||||
|
||||
### slack
|
||||
|
||||
|
||||
@@ -0,0 +1,153 @@
|
||||
# Queue erasure: closed-work retention and account deletion
|
||||
|
||||
A daemon that serves several accounts sometimes needs to remove queue content for good: closed work after its retention window, or everything that belongs to an account when the account is deleted. `rig queue erase` and three retention settings do that.
|
||||
|
||||
Erasure can't be undone. Run every scope as a dry run first, without `--apply`. The dry run prints exactly what `--apply` would erase and changes nothing.
|
||||
|
||||
## What is erased
|
||||
|
||||
A queue row is erased in one of two ways:
|
||||
|
||||
- **Scrub.** The row's content is removed, and the row itself stays, marked `contentErasedAt`. A retried create with the same id still finds the row, so it makes no new work. If anything writes to the row after its scrub (a note, a summary, a reopen and close), the next run erases that content too.
|
||||
- **Delete.** The row and everything linked to it are removed. Only its id is kept, in an erased-id record with no endpoints and no content. A later create that reuses the id is refused with `qitem_erased` (409), so a retried or forwarded create can't bring the work back.
|
||||
|
||||
| A scrub clears | A scrub keeps |
|
||||
|---|---|
|
||||
| body, summary, resolution, tags, blocker text, chain of record, evidence reference, human detail, questions and answers, delegated decision, the input-box reading | id, source and destination, state, priority, tier, times, closure reason, handoff links |
|
||||
| a closure's target when it's a reason or note, in the row, both transition tables and events. That's every target under a reason other than the two at right, and any target under those two that isn't a session address or a queue id | a closure's target that names a successor: under `handed_off_to` or `superseded`, shaped as a session address (`member@rig`, optionally `@host`) or a queue id (`qitem-…`) |
|
||||
| transition notes and recorded principals, live and archived | each transition's state, actor and time |
|
||||
| events' `summary`, `reason` and `error` (the event keeps its type and ids, marked `contentErased`) | |
|
||||
| mission-control audit text and snapshots | |
|
||||
|
||||
**The row's own copies go with it:**
|
||||
- its transitions;
|
||||
- events whose payload names it;
|
||||
- its mission-control audit rows.
|
||||
|
||||
**Copies held with someone else go only when they're the owner's.** These are outbox and inbox entries that point at the row or wake for it, and stream reader actions and dispositions that name it (including `duplicate_of`). For account deletion, the owner is the account: both of a copy's sessions must be members. Otherwise the owner is the row's own source and destination. Any other copy is kept and listed under `keptCopies`:
|
||||
- `mixed`: one of its sessions is the owner's;
|
||||
- `foreign`: none of them are.
|
||||
|
||||
**A row a workflow still references is never erased.** That covers a step trail, a frontier binding, a failure record, or a live instance's frontier. It's reported under `gaps` with the code `workflow-reference`.
|
||||
|
||||
## Scopes
|
||||
|
||||
Name exactly one.
|
||||
|
||||
| Scope | Erases | Action |
|
||||
|---|---|---|
|
||||
| `--members <file>` | Account deletion. Every row whose source and destination are both in the file, in any state, open work included. Add `--rig <id>` for the account rig's own records | delete |
|
||||
| `--ids <file>` | Exactly the listed rows. A report's `qitems` entries, or a ledger's lines, keep each entry's own action | `--action delete` (default for plain ids) or `--action scrub` |
|
||||
| `--routine` | Closed rows past this daemon's retention windows (below) | scrub, then delete |
|
||||
|
||||
A file is one entry per line (blank lines and `#` comments are skipped; a line starting with `{` is a JSON entry) or a JSON array.
|
||||
|
||||
**Account deletion is exact.**
|
||||
- The members file is the account's recorded membership: the exact session names of its seats, including departed ones. List every address form the account used, such as a host-suffixed form. Names are compared as given, never by pattern, so a row addressed through an unlisted form is neither erased nor reported.
|
||||
- **Mixed rows:** a row with exactly one member endpoint is reported under `mixed` (with which endpoint is the member) and left untouched. Deciding who owns it is a separate decision. A row rerouted out of the account while the run is under way (by a fallback, say) is kept too, and reported as `changed-during-run`.
|
||||
- **After the rows:** account deletion also removes outbox and inbox entries between two members that no row owns, and the members' stream items. An entry with one member endpoint, or a stream item something still references, is counted and left.
|
||||
|
||||
**The account's rig.** A rig's own records outlive the rig. Events are kept past rig and node deletion by design, and snapshots, node and seat records and watchdog jobs have no key that cascades. `--rig <id>`, given with `--members`, erases them:
|
||||
- the rig's events, including those of its nodes;
|
||||
- snapshots, operation correlations and bootstrap runs;
|
||||
- seat identity verdicts, delivery-guard changes, environment bindings and usage records for its nodes and member sessions;
|
||||
- watchdog jobs aimed at a member, with their history;
|
||||
- stream subscriptions and thread maps of member sessions.
|
||||
|
||||
Records that name the account but also belong elsewhere are counted under `rig.left` and kept. These are a member's watchdog job aimed outside the account, a workflow instance or spec bound to the rig, classifier leases and custom views.
|
||||
- **While the rig exists,** its sessions and the members must match both ways, or the run is refused (`erase_rig_mismatch`). Every member's session name must carry the rig's name (departed members included), and every current session of the rig must be a member, so the rig holds no one else's seat.
|
||||
- **On that verified run,** the rig's id and name are recorded with a digest of the membership.
|
||||
- **Once the rig is removed,** the same command, with the same membership, uses that record to erase what's left under the rig's id. This includes the events the removal wrote, and the record still lets it count records bound to the rig's name.
|
||||
- **When nothing of the rig is left,** the record becomes a completion receipt: the rig's name is cleared, and its id, the membership digest and the time stay. An identical rerun, after a lost response say, still verifies and reports the same clean result, with `rig.completedAt`. The receipt is kept for good. It holds no content.
|
||||
- **A removed rig's id that was never verified** with this membership erases nothing keyed by it. Those records are counted under `rig.left`, the name-bound tables are listed under `rig.unknown`, and the run isn't clean.
|
||||
|
||||
**Retention windows** (`rig config set …`, read at the daily sweep and at daemon start):
|
||||
|
||||
| Setting | Default | Effect |
|
||||
|---|---|---|
|
||||
| `retention.closed_content_days` | `0` (off) | Scrub closed rows this many days after they closed |
|
||||
| `retention.closed_metadata_days` | `0` (off) | Delete closed rows, keeping only their ids, this many days after they closed |
|
||||
| `retention.erasure_ledger` | empty (none) | An absolute file path, on local disk, since each batch appends to it while it holds the daemon. Every applied erasure, by the sweep or the command, appends each row's id and action there, one JSON line each, before anything in its batch changes. A path that can't be written stops the run before it changes anything |
|
||||
|
||||
- **Closed** means `done`, `canceled`, `denied` or `handed-off`. `failed` and open work are never touched by the windows.
|
||||
- **Age:** a row's age counts from when it entered its final closed state. A note on a closed row doesn't restart it, and a row reopened and closed again counts from its latest close.
|
||||
- **Running them now:** `rig queue erase --routine --apply` runs the windows immediately. `retention.enabled false` turns off the whole sweep, these windows included.
|
||||
- **When the sweep's erasure stops on an error,** that's logged, and the sweep's other passes still run.
|
||||
|
||||
## Account deletion, in order
|
||||
|
||||
1. Revoke the account's access.
|
||||
2. Keep its exact membership list and rig id.
|
||||
3. Run `rig queue erase --members <file> --rig <id>` as a dry run, then with `--apply`. Rerun until `complete` is true.
|
||||
4. Remove the rig.
|
||||
5. Rerun the same command with `--apply`, to erase the records the removal wrote.
|
||||
6. The account is deleted only when the report's `clean` is true. Otherwise its `mixed`, `gaps`, `keptCopies`, `unlinked` and `rig.left` or `rig.unknown` entries need a decision. Rerunning won't resolve them.
|
||||
|
||||
## Output
|
||||
|
||||
The command prints one JSON report (`--json` makes it a single line):
|
||||
|
||||
| Field | Meaning |
|
||||
|---|---|
|
||||
| `applied` | `false` for a dry run |
|
||||
| `complete` | the run finished. `false` when it stopped at its batch limit or on an error; rerun the same command to continue |
|
||||
| `clean` | `complete`, and nothing was left: no `mixed` rows, mixed `keptCopies`, `gaps`, `unreadableEvents`, left-over entries, or rig records left or unknown. Only a clean report means the scope is fully erased |
|
||||
| `error` | why the run stopped (`stage` and `message`). Everything listed in `qitems` was committed before it; the command exits 2 |
|
||||
| `qitems` | each row erased (or that would be), with its `action` and `state` |
|
||||
| `mixed` | members scope: rows with one member endpoint, left untouched |
|
||||
| `keptCopies` | copies of erased rows held with someone else, kept: `table`, `id`, `qitemId`, `ownership` (`mixed` or `foreign`) |
|
||||
| `gaps` | rows not erased and why: `workflow-reference`; `not-found` for an id to scrub that doesn't exist; `changed-during-run` for a row that no longer qualified when its batch ran (reopened, rerouted out of the account, or written to so it left its window) or an absent id created since selection; `unreadable-event` for a row an unreadable event names |
|
||||
| `alreadyAbsent` | ids to delete with no row; their erased-id record is written (again), and they're listed once it's committed. They're rechecked as the record is written, so an id created during the run is reported as `changed-during-run` instead |
|
||||
| `unreadableEvents` | events naming a selected row whose payload can't be read: `seq` and the row ids found in its text. They're kept, since what they hold can't be inspected, and so are the rows they name, so every rerun reports them again until the event is repaired or removed. The run isn't clean |
|
||||
| `linked` | counts of the copies erased with the rows |
|
||||
| `unlinked` | members scope: entries and stream items erased that no row owned, plus those left (`mixedEntries`, `referencedStreamItems`) |
|
||||
| `rig` | members scope with `--rig`: `found`, `verified`, records erased per table (`erased`, counted as each chunk commits), records left (`left`), tables that can't be counted (`unknown`), and `completedAt` once the rig's erasure is complete |
|
||||
|
||||
**How a run works:**
|
||||
- **Selection:** a run selects everything first and writes nothing.
|
||||
- **Events:** it reads the events table once, in small windows, to find each row's events.
|
||||
- **Batches:** each batch, of up to 100 rows, is one transaction. It rechecks its rows, writes their ledger lines, then changes their events, copies and rows.
|
||||
- **The recheck drops** a row that a workflow has taken up, or that an unreadable event names. For account deletion, it also drops a row that is no longer between two members. For a routine run, it drops one that was reopened or no longer falls in the same window.
|
||||
- **Each dropped row** is reported in `gaps`.
|
||||
- **The current endpoints** decide whose its copies are.
|
||||
- **How long it holds the daemon:** the batch caps rows, not events or copies, so a row with many of either makes its batch longer. On a copy of a store with 2.4 million events, the longest single hold in a first routine run was about 0.25 s.
|
||||
- **Stopping:** a run that stops leaves whole batches done, events included. A rerun picks up the rest, and erasing a row a second time changes nothing.
|
||||
- **Ledger lines are recorded intentions:** if a batch's transaction fails after its lines are written, the ledger lists rows that weren't erased. A replay erases those ids as listed, whatever happened to the rows since.
|
||||
|
||||
**Refusals**, with nothing erased:
|
||||
|
||||
| Code | Cause |
|
||||
|---|---|
|
||||
| `erase_scope_required` | no scope named |
|
||||
| `erase_scope_conflict` | more than one scope |
|
||||
| `erase_members_invalid` | an empty list, or names with surrounding spaces |
|
||||
| `erase_rig_invalid` | a rig without `--members`, or an empty rig id |
|
||||
| `erase_rig_mismatch` | the rig exists, and its sessions and the members don't match both ways |
|
||||
| `erase_ids_invalid` | an empty ids list, or ids with surrounding spaces |
|
||||
| `erase_action_invalid` | an action other than `delete` or `scrub` |
|
||||
| `erase_routine_off` | both windows are `0` |
|
||||
|
||||
The command records who ran it from `OPENRIG_SESSION_NAME`, or from `--actor <session>` when that isn't set.
|
||||
|
||||
## Restores and backups
|
||||
|
||||
A restored database can bring erased content back: rows, copies, rig records and the erased-id records themselves. It can even bring back a row that was still open when the backup was taken and was closed and erased later. Before a restored daemon serves anyone, replay what was erased after the backup was taken:
|
||||
|
||||
1. **Keep a ledger.** Set `retention.erasure_ledger` to a file outside the backed-up data. Every applied erasure appends its ids and actions there before committing. That covers the daily windows, `--routine`, `--ids` and account deletion's rows.
|
||||
2. **Replay the ledger:** `rig queue erase --ids <ledger> --apply`. Each line keeps its own action, so a scrub is replayed as a scrub, never promoted to a deletion. A row that was open in the backup is erased by its id, and other open work isn't touched.
|
||||
3. **Replay account deletions:** rerun `rig queue erase --members <file> --rig <id> --apply` for each account deleted since the backup, from its membership list and rig id. Keep those outside the backed-up data until every backup taken before the deletion has expired. This also covers the copies and rig records the ledger doesn't list.
|
||||
4. **Rerun the windows:** `rig queue erase --routine --apply`.
|
||||
|
||||
Backup expiry is the deployment's own schedule. Content erased from the live database stays in backups until they expire.
|
||||
|
||||
## What this doesn't cover
|
||||
|
||||
These stores need their own handling where a deployment uses them:
|
||||
- terminal transcripts;
|
||||
- daemon and service logs;
|
||||
- gateway and Slack stores and Slack's own copies;
|
||||
- downloaded files and artifacts;
|
||||
- chat messages (removed with their rig);
|
||||
- events tied neither to a row nor to the account's rig.
|
||||
|
||||
The rig itself (its nodes, sessions and bindings) is removed by removing the rig, between the two runs above.
|
||||
@@ -107,6 +107,14 @@ Keys:
|
||||
than N days; 0 keeps them (default). Removed events no longer replay from /api/events, leave
|
||||
the telemetry export, the log and the feed, and can't be brought back. Read at the next sweep
|
||||
(daily and at daemon start)
|
||||
retention.closed_content_days remove the content of closed queue work (done, canceled, denied, handed-off) N
|
||||
days after it closed; the row stays, without content. 0 keeps it (default). Can't be undone
|
||||
retention.closed_metadata_days delete closed queue rows N days after they closed, keeping only their ids so a retry
|
||||
can't recreate them. 0 keeps them (default). Both read at the next sweep; "rig queue erase
|
||||
--routine --apply" runs them now
|
||||
retention.erasure_ledger an absolute file where queue erasure appends each erased id and its action, one JSON
|
||||
line each, before it commits them; keep it outside backups and replay it with "rig queue
|
||||
erase --ids <file> --apply" after a restore. Empty keeps no ledger (default)
|
||||
launch.non_interruptive accept harness first-launch warnings at full bypass for new rigs (default off)
|
||||
launch.codex_login_shell defer to Codex's own login-shell setting in managed Codex seats (default off)
|
||||
Off adds allow_login_shell=false: tool commands inherit the seat's environment, so plain rig
|
||||
|
||||
@@ -1999,6 +1999,100 @@ Examples:
|
||||
});
|
||||
});
|
||||
|
||||
cmd
|
||||
.command("erase")
|
||||
.description("Erase queue rows in one exact scope: an account's rows, a list of ids, or the closed-work retention windows; a dry run unless --apply")
|
||||
.option("--members <file>", "Account deletion: the account's recorded session names (exact, departed seats included), one per line or a JSON array")
|
||||
.option("--rig <id>", "With --members: the account rig's id. Also erases the rig's own records that outlive it (events, snapshots, node and seat records); refused unless every member is a session of it")
|
||||
.option("--ids <file>", "These qitem ids, one per line or a JSON array; a report's qitems entries, or an erasure ledger's JSON lines, keep each one's action")
|
||||
.option("--action <action>", "With --ids: delete (default) removes each row and keeps only its id; scrub removes its content and keeps the row", enumArg(["delete", "scrub"]))
|
||||
.option("--routine", "Run this daemon's closed-work windows now (retention.closed_content_days, retention.closed_metadata_days)")
|
||||
.option("--apply", "Erase. Without it the command reports what would be erased and changes nothing")
|
||||
.option("--actor <session>", "Who is erasing, when OPENRIG_SESSION_NAME isn't set (recorded as a claimed identity)")
|
||||
.option("--json", "JSON output for agents")
|
||||
.addHelpText("after", [
|
||||
"",
|
||||
"Name exactly one scope. What each one erases:",
|
||||
" --members every row whose source and destination are both members, in any state, deleted with its transitions,",
|
||||
" outbox and inbox copies, stream actions, events and audit rows; then member-to-member outbox and inbox",
|
||||
" entries no row owns, and the members' stream items. A row with one member endpoint is reported under",
|
||||
" mixed and left untouched. With --rig, also the rig's own events, snapshots and node and seat records.",
|
||||
" --ids the listed rows, deleted or scrubbed (a report's qitems entries keep their action); a deleted id",
|
||||
" that's already gone keeps (or regains) its id marker.",
|
||||
" --routine closed rows (done, canceled, denied, handed-off) past the windows, aged from when they closed.",
|
||||
"A deleted row keeps only its id, and a create that reuses that id is refused (qitem_erased). A row a workflow still",
|
||||
"references is reported under gaps and left. Output is JSON: clean (true only when nothing was left), complete,",
|
||||
"qitems, mixed, gaps, alreadyAbsent, linked, unlinked and rig. Refusals: erase_scope_required, erase_scope_conflict,",
|
||||
"erase_members_invalid, erase_rig_invalid, erase_rig_mismatch, erase_ids_invalid, erase_action_invalid,",
|
||||
"erase_routine_off. A run that stops on an error returns what it committed, with error.stage, and exits 2.",
|
||||
"Reference: docs/reference/queue-erasure.md",
|
||||
].join("\n"))
|
||||
.action(async (opts: { members?: string; rig?: string; ids?: string; action?: string; routine?: boolean; apply?: boolean; actor?: string; json?: boolean }) => {
|
||||
const list = (flag: string, path: string): unknown[] | undefined => {
|
||||
let text: string;
|
||||
try {
|
||||
text = fs.readFileSync(path, "utf8");
|
||||
} catch (err) {
|
||||
console.error(`--${flag} ${path}: can't be read (${err instanceof Error ? err.message : String(err)}). Nothing was erased.`);
|
||||
process.exitCode = 1;
|
||||
return undefined;
|
||||
}
|
||||
if (text.trimStart().startsWith("[")) {
|
||||
try {
|
||||
const parsed: unknown = JSON.parse(text);
|
||||
if (Array.isArray(parsed)) return parsed;
|
||||
} catch { /* reported below */ }
|
||||
console.error(`--${flag} ${path}: not a JSON array. Nothing was erased.`);
|
||||
process.exitCode = 1;
|
||||
return undefined;
|
||||
}
|
||||
// One entry per line: a plain id or session, or (an erasure ledger's line) a JSON object.
|
||||
const entries: unknown[] = [];
|
||||
for (const line of text.split(/\r?\n/).map((entry) => entry.trim()).filter((entry) => entry.length > 0 && !entry.startsWith("#"))) {
|
||||
if (!line.startsWith("{")) {
|
||||
entries.push(line);
|
||||
continue;
|
||||
}
|
||||
try {
|
||||
entries.push(JSON.parse(line));
|
||||
} catch {
|
||||
console.error(`--${flag} ${path}: a line isn't valid JSON (${line.slice(0, 60)}). Nothing was erased.`);
|
||||
process.exitCode = 1;
|
||||
return undefined;
|
||||
}
|
||||
}
|
||||
return entries;
|
||||
};
|
||||
const body: Record<string, unknown> = { apply: opts.apply === true };
|
||||
if (opts.members !== undefined) {
|
||||
const members = list("members", opts.members);
|
||||
if (!members) return;
|
||||
body.members = members;
|
||||
}
|
||||
// Sent as given: the daemon refuses a rig without members (erase_rig_invalid), never drops it.
|
||||
if (opts.rig !== undefined) body.rig = opts.rig;
|
||||
if (opts.ids !== undefined) {
|
||||
const ids = list("ids", opts.ids);
|
||||
if (!ids) return;
|
||||
body.ids = ids;
|
||||
if (opts.action) body.action = opts.action;
|
||||
}
|
||||
if (opts.routine) body.routine = true;
|
||||
if (!readOpenRigEnv("OPENRIG_SESSION_NAME", "RIGGED_SESSION_NAME")) {
|
||||
if (!opts.actor) {
|
||||
console.error("OPENRIG_SESSION_NAME is not set and --actor wasn't given, so there's no one to record as erasing. Nothing was erased.");
|
||||
process.exitCode = 1;
|
||||
return;
|
||||
}
|
||||
body.actorSession = opts.actor;
|
||||
}
|
||||
const deps = getDeps();
|
||||
await withClient(deps, async (client) => {
|
||||
const res = await client.post<unknown>("/api/queue/erase", body);
|
||||
printResult(opts.json ?? false, res.data, res.status);
|
||||
});
|
||||
});
|
||||
|
||||
// ---- Inbox subcommands ----
|
||||
|
||||
cmd
|
||||
|
||||
@@ -221,6 +221,9 @@ export interface RiggedConfig {
|
||||
streamDays: number;
|
||||
streamMaxItems: number;
|
||||
eventsDays: number;
|
||||
closedContentDays: number;
|
||||
closedMetadataDays: number;
|
||||
erasureLedger: string;
|
||||
batchSize: number;
|
||||
};
|
||||
// S04 (OPR.0.5.5.4) — pickup-receipt stall threshold (twin of the daemon settings-store).
|
||||
@@ -415,6 +418,9 @@ const DEFAULTS = {
|
||||
streamDays: 30,
|
||||
streamMaxItems: 20000,
|
||||
eventsDays: 0,
|
||||
closedContentDays: 0,
|
||||
closedMetadataDays: 0,
|
||||
erasureLedger: "",
|
||||
batchSize: 500,
|
||||
},
|
||||
// S04 — pickup-receipt stall threshold default (twin of daemon getDefaultValue).
|
||||
@@ -565,6 +571,12 @@ export const VALID_KEYS = [
|
||||
"retention.stream_max_items",
|
||||
// The history-only event types of `events` older than N days, 0 (default) keeps them; lockstep with the daemon twin.
|
||||
"retention.events_days",
|
||||
// Closed queue work: content erased N days after it closed, then the row (only its id kept); 0 (default) keeps them;
|
||||
// lockstep with the daemon twin.
|
||||
"retention.closed_content_days",
|
||||
"retention.closed_metadata_days",
|
||||
// Queue erasure's ledger file (JSON lines of erased ids and actions); empty keeps none; lockstep with the daemon twin.
|
||||
"retention.erasure_ledger",
|
||||
"stream.intake",
|
||||
"retention.batch_size",
|
||||
// S04 — pickup-receipt stall threshold; lockstep with the daemon settings-store twin.
|
||||
@@ -687,6 +699,9 @@ export const ENV_MAP: Record<ValidKey, { primary: string; legacy?: string }> = {
|
||||
"retention.stream_days": { primary: "OPENRIG_RETENTION_STREAM_DAYS" },
|
||||
"retention.stream_max_items": { primary: "OPENRIG_RETENTION_STREAM_MAX_ITEMS" },
|
||||
"retention.events_days": { primary: "OPENRIG_RETENTION_EVENTS_DAYS" },
|
||||
"retention.closed_content_days": { primary: "OPENRIG_RETENTION_CLOSED_CONTENT_DAYS" },
|
||||
"retention.closed_metadata_days": { primary: "OPENRIG_RETENTION_CLOSED_METADATA_DAYS" },
|
||||
"retention.erasure_ledger": { primary: "OPENRIG_RETENTION_ERASURE_LEDGER" },
|
||||
"stream.intake": { primary: "OPENRIG_STREAM_INTAKE" },
|
||||
"retention.batch_size": { primary: "OPENRIG_RETENTION_BATCH_SIZE" },
|
||||
"queue.pickup_stall_threshold_minutes": { primary: "OPENRIG_QUEUE_PICKUP_STALL_THRESHOLD_MINUTES" },
|
||||
@@ -792,6 +807,9 @@ const KEY_TO_PATH: Record<ValidKey, string[]> = {
|
||||
"retention.stream_days": ["retention", "streamDays"],
|
||||
"retention.stream_max_items": ["retention", "streamMaxItems"],
|
||||
"retention.events_days": ["retention", "eventsDays"],
|
||||
"retention.closed_content_days": ["retention", "closedContentDays"],
|
||||
"retention.closed_metadata_days": ["retention", "closedMetadataDays"],
|
||||
"retention.erasure_ledger": ["retention", "erasureLedger"],
|
||||
"stream.intake": ["stream", "intake"],
|
||||
"retention.batch_size": ["retention", "batchSize"],
|
||||
"queue.pickup_stall_threshold_minutes": ["queue", "pickupStallThresholdMinutes"],
|
||||
@@ -1178,6 +1196,24 @@ const KEY_CONSTRAINTS: Partial<Record<ValidKey, (raw: string, coerced: string |
|
||||
throw new Error(`Invalid value for retention.events_days: must be an integer >= 0 (0 keeps the events), got "${raw}"`);
|
||||
}
|
||||
},
|
||||
// 0 (the default) keeps closed work; a positive day count turns its erasure on (queue-erasure).
|
||||
"retention.closed_content_days": (raw, coerced) => {
|
||||
if (!/^-?\d+$/.test((raw ?? "").trim()) || typeof coerced !== "number" || !Number.isInteger(coerced) || coerced < 0) {
|
||||
throw new Error(`Invalid value for retention.closed_content_days: must be an integer >= 0 (0 keeps the content), got "${raw}"`);
|
||||
}
|
||||
},
|
||||
"retention.closed_metadata_days": (raw, coerced) => {
|
||||
if (!/^-?\d+$/.test((raw ?? "").trim()) || typeof coerced !== "number" || !Number.isInteger(coerced) || coerced < 0) {
|
||||
throw new Error(`Invalid value for retention.closed_metadata_days: must be an integer >= 0 (0 keeps the rows), got "${raw}"`);
|
||||
}
|
||||
},
|
||||
// Empty (the default) keeps no erasure ledger; otherwise an absolute file path, kept outside the backed-up data.
|
||||
"retention.erasure_ledger": (raw) => {
|
||||
const value = (raw ?? "").trim();
|
||||
if (value !== "" && !value.startsWith("/")) {
|
||||
throw new Error(`Invalid value for retention.erasure_ledger: must be empty or an absolute file path, got "${raw}"`);
|
||||
}
|
||||
},
|
||||
"retention.batch_size": (raw, coerced) => {
|
||||
if (!/^-?\d+$/.test((raw ?? "").trim()) || typeof coerced !== "number" || !Number.isInteger(coerced) || coerced < 1) {
|
||||
throw new Error(`Invalid value for retention.batch_size: must be an integer >= 1, got "${raw}"`);
|
||||
@@ -1449,6 +1485,9 @@ export class ConfigStore {
|
||||
streamDays: v("retention.stream_days") as number,
|
||||
streamMaxItems: v("retention.stream_max_items") as number,
|
||||
eventsDays: v("retention.events_days") as number,
|
||||
closedContentDays: v("retention.closed_content_days") as number,
|
||||
closedMetadataDays: v("retention.closed_metadata_days") as number,
|
||||
erasureLedger: v("retention.erasure_ledger") as string,
|
||||
batchSize: v("retention.batch_size") as number,
|
||||
},
|
||||
queue: {
|
||||
|
||||
@@ -216,6 +216,9 @@ describe("ConfigStore — extended namespaces (User Settings v0)", () => {
|
||||
"retention.stream_days", // slice 22 outcome 9: stream retention
|
||||
"retention.stream_max_items",
|
||||
"retention.events_days", // the history-only event types of events
|
||||
"retention.closed_content_days", // closed-work erasure: content
|
||||
"retention.closed_metadata_days", // closed-work erasure: the row, keeping its id
|
||||
"retention.erasure_ledger", // queue erasure's replay ledger file
|
||||
"stream.intake",
|
||||
"retention.batch_size",
|
||||
"queue.pickup_stall_threshold_minutes",
|
||||
|
||||
@@ -0,0 +1,115 @@
|
||||
import { describe, it, expect, vi, beforeEach } from "vitest";
|
||||
import fs from "node:fs";
|
||||
import path from "node:path";
|
||||
import os from "node:os";
|
||||
import type { QueueDeps } from "../src/commands/queue.js";
|
||||
import { createProgram } from "../src/index.js";
|
||||
|
||||
// `rig queue erase`: the file it reads, the request it sends, and what it refuses before sending anything.
|
||||
vi.mock("../src/daemon-lifecycle.js", async () => {
|
||||
const actual = await vi.importActual<Record<string, unknown>>("../src/daemon-lifecycle.js");
|
||||
return {
|
||||
...actual,
|
||||
getDaemonStatus: vi.fn(async () => ({ state: "running", healthy: true, pid: 1234, port: 7433 })),
|
||||
getDaemonUrl: vi.fn(() => "http://localhost:7433"),
|
||||
};
|
||||
});
|
||||
|
||||
function makeDeps(response: { status: number; data: unknown } = { status: 200, data: { ok: true } }) {
|
||||
const posts: Array<{ path: string; body: unknown }> = [];
|
||||
const deps = {
|
||||
lifecycleDeps: {} as QueueDeps["lifecycleDeps"],
|
||||
clientFactory: () => ({
|
||||
post: vi.fn(async (urlPath: string, body: unknown) => {
|
||||
posts.push({ path: urlPath, body });
|
||||
return response;
|
||||
}),
|
||||
}) as unknown as ReturnType<QueueDeps["clientFactory"]>,
|
||||
} as QueueDeps;
|
||||
return { deps, posts };
|
||||
}
|
||||
|
||||
describe("rig queue erase", () => {
|
||||
let dir: string;
|
||||
let errors: string[];
|
||||
beforeEach(() => {
|
||||
vi.unstubAllEnvs();
|
||||
vi.stubEnv("OPENRIG_SESSION_NAME", "ops-admin@n-ops");
|
||||
dir = fs.mkdtempSync(path.join(os.tmpdir(), "queue-erase-"));
|
||||
errors = [];
|
||||
vi.spyOn(console, "log").mockImplementation(() => {});
|
||||
vi.spyOn(console, "error").mockImplementation((...args) => errors.push(args.join(" ")));
|
||||
process.exitCode = undefined;
|
||||
});
|
||||
const run = (deps: QueueDeps, ...args: string[]) => createProgram({ queueDeps: deps }).parseAsync(["node", "rig", "queue", "erase", ...args]);
|
||||
|
||||
it("reads a members file (one per line, comments skipped) and sends a dry run unless --apply", async () => {
|
||||
const file = path.join(dir, "members.txt");
|
||||
fs.writeFileSync(file, "# account n-acct\na-one@n-acct\n\n b-two@n-acct \n");
|
||||
const { deps, posts } = makeDeps();
|
||||
await run(deps, "--members", file);
|
||||
await run(deps, "--members", file, "--apply");
|
||||
expect(posts).toEqual([
|
||||
{ path: "/api/queue/erase", body: { apply: false, members: ["a-one@n-acct", "b-two@n-acct"] } },
|
||||
{ path: "/api/queue/erase", body: { apply: true, members: ["a-one@n-acct", "b-two@n-acct"] } },
|
||||
]);
|
||||
});
|
||||
|
||||
it("reads ids as a JSON array and passes the action", async () => {
|
||||
const file = path.join(dir, "ids.json");
|
||||
fs.writeFileSync(file, JSON.stringify(["qitem-1", "qitem-2"]));
|
||||
const { deps, posts } = makeDeps();
|
||||
await run(deps, "--ids", file, "--action", "scrub", "--apply");
|
||||
expect(posts).toEqual([{ path: "/api/queue/erase", body: { apply: true, ids: ["qitem-1", "qitem-2"], action: "scrub" } }]);
|
||||
});
|
||||
|
||||
it("passes the account rig with the members, and a ledger of report entries as given", async () => {
|
||||
const members = path.join(dir, "members.json");
|
||||
fs.writeFileSync(members, JSON.stringify(["a-one@n-acct"]));
|
||||
const ledger = path.join(dir, "ledger.json");
|
||||
const entries = [{ qitemId: "qitem-1", action: "scrub", state: "done" }, "qitem-2"];
|
||||
fs.writeFileSync(ledger, JSON.stringify(entries));
|
||||
const { deps, posts } = makeDeps();
|
||||
await run(deps, "--members", members, "--rig", "rig-acct-id", "--apply");
|
||||
await run(deps, "--ids", ledger, "--apply");
|
||||
expect(posts).toEqual([
|
||||
{ path: "/api/queue/erase", body: { apply: true, members: ["a-one@n-acct"], rig: "rig-acct-id" } },
|
||||
{ path: "/api/queue/erase", body: { apply: true, ids: entries } },
|
||||
]);
|
||||
});
|
||||
|
||||
it("reads an erasure ledger's JSON lines as entries that keep their action", async () => {
|
||||
const ledger = path.join(dir, "ledger.jsonl");
|
||||
fs.writeFileSync(ledger, '{"qitemId":"qitem-1","action":"scrub","reason":"retention-metadata","erasedAt":"t"}\n# a comment\n{"qitemId":"qitem-2","action":"delete"}\n');
|
||||
const { deps, posts } = makeDeps();
|
||||
await run(deps, "--ids", ledger, "--apply");
|
||||
expect(posts).toEqual([{ path: "/api/queue/erase", body: { apply: true, ids: [
|
||||
{ qitemId: "qitem-1", action: "scrub", reason: "retention-metadata", erasedAt: "t" }, { qitemId: "qitem-2", action: "delete" },
|
||||
] } }]);
|
||||
fs.writeFileSync(ledger, '{"qitemId": broken\n');
|
||||
process.exitCode = undefined;
|
||||
await run(deps, "--ids", ledger, "--apply");
|
||||
expect(process.exitCode).toBe(1);
|
||||
expect(posts).toHaveLength(1);
|
||||
});
|
||||
|
||||
it("sends nothing when its file can't be read, or when there's no one to record as erasing", async () => {
|
||||
const { deps, posts } = makeDeps();
|
||||
await run(deps, "--members", path.join(dir, "missing.txt"));
|
||||
expect(process.exitCode).toBe(1);
|
||||
vi.stubEnv("OPENRIG_SESSION_NAME", "");
|
||||
process.exitCode = undefined;
|
||||
await run(deps, "--routine");
|
||||
expect(process.exitCode).toBe(1);
|
||||
expect(posts).toEqual([]);
|
||||
expect(errors.join("\n")).toMatch(/Nothing was erased/);
|
||||
await run(deps, "--routine", "--actor", "erase-runner@n-ops");
|
||||
expect(posts).toEqual([{ path: "/api/queue/erase", body: { apply: false, routine: true, actorSession: "erase-runner@n-ops" } }]);
|
||||
});
|
||||
|
||||
it("a daemon refusal exits nonzero", async () => {
|
||||
const { deps } = makeDeps({ status: 400, data: { error: "erase_scope_required", message: "Name one scope" } });
|
||||
await run(deps);
|
||||
expect(process.exitCode).toBe(1);
|
||||
});
|
||||
});
|
||||
@@ -2,6 +2,8 @@ import { piAccountSwitchesSchema } from "./migrations/151_pi_account_switches.js
|
||||
import { workflowInstanceSpecSchema } from "./migrations/152_workflow_instance_spec.js";
|
||||
import { eventsRetentionMarksSchema } from "./migrations/153_events_retention_marks.js";
|
||||
import { workflowInstanceRootObjectiveSchema } from "./migrations/154_workflow_instance_root_objective.js";
|
||||
import { queueErasureSchema } from "./migrations/155_queue_erasure.js";
|
||||
import { queueErasureLinkIndexesSchema } from "./migrations/156_queue_erasure_link_indexes.js";
|
||||
import { activityGrantsSchema } from "./migrations/143_activity_grants.js";
|
||||
import { environmentBindingsSchema } from "./migrations/144_environment_bindings.js";
|
||||
import { nodePermissionsToSchema } from "./migrations/145_node_permissions_to.js";
|
||||
@@ -311,4 +313,6 @@ export const ALL_MIGRATIONS: Migration[] = [
|
||||
workflowInstanceSpecSchema,
|
||||
eventsRetentionMarksSchema,
|
||||
workflowInstanceRootObjectiveSchema,
|
||||
queueErasureSchema,
|
||||
queueErasureLinkIndexesSchema,
|
||||
];
|
||||
@@ -0,0 +1,29 @@
|
||||
import type { Migration } from "../migrate.js";
|
||||
|
||||
/**
|
||||
* Queue erasure (closed-work retention and account deletion). `content_erased_at` marks a row whose content was
|
||||
* removed: the row stays, without content, so a retried create with its id still finds it and makes no new work.
|
||||
* A deleted row leaves only its id in `queue_erased_ids`, with no endpoints and no content, and the create path
|
||||
* refuses that id (`qitem_erased`). `queue_erased_rigs` keeps an account rig's id and name with a digest of the
|
||||
* membership it was verified against, so the run after the rig is removed still erases and counts its records exactly.
|
||||
* Once nothing of the rig is left, the record becomes a completion receipt (`completed_at` set, the name cleared), so an
|
||||
* identical rerun still verifies and reports the same result.
|
||||
*/
|
||||
export const queueErasureSchema: Migration = {
|
||||
name: "155_queue_erasure.sql",
|
||||
sql: `
|
||||
ALTER TABLE queue_items ADD COLUMN content_erased_at TEXT;
|
||||
CREATE TABLE IF NOT EXISTS queue_erased_ids (
|
||||
qitem_id TEXT PRIMARY KEY,
|
||||
erased_at TEXT NOT NULL,
|
||||
reason TEXT NOT NULL
|
||||
);
|
||||
CREATE TABLE IF NOT EXISTS queue_erased_rigs (
|
||||
rig_id TEXT PRIMARY KEY,
|
||||
rig_name TEXT NOT NULL,
|
||||
members_digest TEXT NOT NULL,
|
||||
verified_at TEXT NOT NULL,
|
||||
completed_at TEXT
|
||||
);
|
||||
`,
|
||||
};
|
||||
@@ -0,0 +1,15 @@
|
||||
import type { Migration } from "../migrate.js";
|
||||
|
||||
/** Queue erasure finds a row's copies by its id; without these, each lookup scans the whole table. */
|
||||
export const queueErasureLinkIndexesSchema: Migration = {
|
||||
name: "156_queue_erasure_link_indexes.sql",
|
||||
sql: `
|
||||
CREATE INDEX IF NOT EXISTS idx_outbox_entries_audit_pointer ON outbox_entries(audit_pointer);
|
||||
CREATE INDEX IF NOT EXISTS idx_inbox_entries_absorbed_qitem ON inbox_entries(absorbed_qitem_id);
|
||||
CREATE INDEX IF NOT EXISTS idx_inbox_entries_audit_pointer ON inbox_entries(audit_pointer);
|
||||
CREATE INDEX IF NOT EXISTS idx_stream_dispositions_work_qitem ON stream_dispositions(work_qitem_id);
|
||||
CREATE INDEX IF NOT EXISTS idx_stream_dispositions_effect_qitem ON stream_dispositions(effect_qitem_id);
|
||||
CREATE INDEX IF NOT EXISTS idx_stream_reader_actions_work_qitem ON stream_reader_actions(work_qitem_id);
|
||||
CREATE INDEX IF NOT EXISTS idx_stream_reader_actions_duplicate_of ON stream_reader_actions(duplicate_of);
|
||||
`,
|
||||
};
|
||||
@@ -0,0 +1,822 @@
|
||||
import { createHash } from "node:crypto";
|
||||
import { appendFileSync } from "node:fs";
|
||||
import type Database from "better-sqlite3";
|
||||
|
||||
/**
|
||||
* Queue erasure: the maintained way to remove queue content, for closed-work retention and account deletion on a
|
||||
* daemon that serves several accounts. Two actions on a queue row:
|
||||
* - scrub: the row's content is removed and the row stays, content-free, with its id, endpoints, state and times
|
||||
* (`content_erased_at` set). The create path's retry identity still finds it, so a retried create makes no new work.
|
||||
* A later note, summary or reclose makes it eligible again.
|
||||
* - delete: the row and everything linked to it go. Only its id is kept, in `queue_erased_ids` (no endpoints, no
|
||||
* content), and the create path refuses that id (`qitem_erased`).
|
||||
* The row's own copies follow it: its transitions, live and archived; events whose payload names it; its
|
||||
* mission-control audit rows. Copies another party holds (outbox and inbox entries, stream reader actions and
|
||||
* dispositions) go only when they're the owner's: both endpoints the account's (members scope) or the row's own
|
||||
* (other scopes). Others are kept and reported.
|
||||
*
|
||||
* Scope is exact. Account deletion takes the account's recorded membership (exact session names): a row whose two
|
||||
* endpoints are both members is deleted, whatever its state; a row with one member endpoint is reported as mixed and
|
||||
* left untouched. Named with the account's rig, it also removes the rig's own records that outlive the rig. Routine
|
||||
* retention touches only closed rows (done, canceled, denied, handed-off; never failed or open work), aged from the
|
||||
* start of their final closed state. A row a workflow still references is reported, never erased.
|
||||
*
|
||||
* A run selects everything first, then reads the events table once in seq windows to find the events naming each
|
||||
* selected row (no index serves a payload match, and one statement over a large table would hold the daemon), then
|
||||
* erases batch by batch, yielding between steps. Each batch is one transaction (better-sqlite3 commits a transaction
|
||||
* on return and rolls it back on a throw): it rechecks its rows, since anything may have changed them since selection
|
||||
* (a reopened row is open work again; a rerouted one may no longer be the account's), writes their ledger lines, then
|
||||
* changes their events, copies and rows. A row an unreadable event names is kept, so reruns keep reporting it. A run
|
||||
* that stops, by its batch limit or an error, returns only what it committed; a rerun picks up the rest.
|
||||
*/
|
||||
|
||||
/** The closed states routine retention may erase. `failed` stays out: account deletion covers it. */
|
||||
export const CLOSED_QUEUE_STATES = ["done", "canceled", "denied", "handed-off"] as const;
|
||||
/** The closure reasons whose closure_target names a successor; for the others it can be a reason or note. */
|
||||
const ID_CLOSURE_REASONS = ["handed_off_to", "superseded"] as const;
|
||||
/** A successor as a closure target names it: a session address, or a queue id (either with a host suffix). */
|
||||
const ADDRESS_SHAPE = /^[A-Za-z0-9][A-Za-z0-9._-]*@[A-Za-z0-9._-]+(?:@[A-Za-z0-9._-]+)?$/;
|
||||
const QITEM_SHAPE = /^qitem-[A-Za-z0-9._-]+(?:@[A-Za-z0-9._-]+)?$/;
|
||||
|
||||
/** A scrub keeps a closure target only when it names a successor: closure writes don't constrain the target, so text
|
||||
* under a successor reason is cleared like any other note. */
|
||||
function keepsClosureTarget(reason: unknown, target: unknown): boolean {
|
||||
return ID_CLOSURE_REASONS.includes(reason as (typeof ID_CLOSURE_REASONS)[number]) && typeof target === "string"
|
||||
&& target.length <= 256 && (ADDRESS_SHAPE.test(target) || QITEM_SHAPE.test(target));
|
||||
}
|
||||
|
||||
export type EraseAction = "scrub" | "delete";
|
||||
export type EraseScope =
|
||||
| { kind: "members"; members: readonly string[]; rig?: string }
|
||||
| { kind: "ids"; entries: ReadonlyArray<{ qitemId: string; action: EraseAction }> }
|
||||
| { kind: "routine"; contentDays: number; metadataDays: number };
|
||||
|
||||
/** A scope that can't be erased exactly; nothing was written. */
|
||||
export class EraseRefusal extends Error {
|
||||
constructor(readonly code: string, message: string) {
|
||||
super(message);
|
||||
}
|
||||
}
|
||||
|
||||
export interface EraseOptions {
|
||||
scope: EraseScope;
|
||||
/** false: report what would be erased and write nothing */
|
||||
apply: boolean;
|
||||
nowIso: string;
|
||||
batchSize?: number;
|
||||
/** selection stops after this many pages and the run reports complete: false; a rerun continues */
|
||||
maxBatches?: number;
|
||||
/** events rows (by seq) read per step of the events walk */
|
||||
eventsWindow?: number;
|
||||
/** append each erased row's {qitemId, action, reason, erasedAt} here, one JSON line each, before its batch changes
|
||||
* anything; the file is opened before the run writes anything at all */
|
||||
ledgerPath?: string;
|
||||
}
|
||||
|
||||
type Stage = "ledger" | "select" | "events" | "rows" | "unlinked" | "rig";
|
||||
type CopyTable = "outbox_entries" | "inbox_entries" | "stream_reader_actions" | "stream_dispositions";
|
||||
export type EraseGap = { code: "workflow-reference" | "not-found" | "changed-during-run" | "unreadable-event"; qitemId: string };
|
||||
|
||||
export interface EraseReport {
|
||||
ok: true;
|
||||
applied: boolean;
|
||||
scope: EraseScope["kind"];
|
||||
/** false when the run stopped at maxBatches or on an error; rerun to continue */
|
||||
complete: boolean;
|
||||
/** complete, and nothing was left: no mixed rows or copies, gaps, unreadable events, or left-over entries or rig
|
||||
* records */
|
||||
clean: boolean;
|
||||
/** why the run stopped early, and where; everything listed in qitems was committed before it */
|
||||
error?: { stage: Stage; message: string };
|
||||
/** rows erased (applied, committed) or that would be (dry run) */
|
||||
qitems: Array<{ qitemId: string; action: EraseAction; state: string }>;
|
||||
/** members scope: rows with exactly one member endpoint, left untouched */
|
||||
mixed: Array<{ qitemId: string; memberEndpoint: "source" | "destination" }>;
|
||||
/** rows not erased, and why: a workflow references them; (ids scope) no such row to scrub; the row changed after
|
||||
* selection so it no longer qualifies (reopened, rerouted out of the account, written to out of its window, or an
|
||||
* absent id created since); or an event naming it can't be read */
|
||||
gaps: EraseGap[];
|
||||
/** ids scope, delete: rows already gone whose id marker is (re)written; listed once the marker is committed */
|
||||
alreadyAbsent: string[];
|
||||
/** an erased row's copies held with someone else: mixed (one of their sessions is the owner's) or foreign (none) */
|
||||
keptCopies: Array<{ table: CopyTable; id: string; qitemId: string; ownership: "mixed" | "foreign" }>;
|
||||
/** events naming a selected row whose payload can't be read: kept, since what they hold can't be inspected. The
|
||||
* rows they name are kept too (gap unreadable-event), so every rerun finds them again until they're resolved */
|
||||
unreadableEvents: Array<{ seq: number; qitemIds: string[] }>;
|
||||
/** copies erased with the rows, or that would be */
|
||||
linked: { transitions: number; archivedTransitions: number; outbox: number; inbox: number; events: number; streamActions: number; missionControl: number };
|
||||
/** members scope: entries between members that no queue row owns; mixed ones are counted and left */
|
||||
unlinked?: { outbox: number; inbox: number; streamItems: number; mixedEntries: number; referencedStreamItems: number };
|
||||
/** members scope with rig: whether the rig was found and verified, its records erased and left per table, the
|
||||
* tables whose records can't be counted (an unverified removed rig), and when its erasure completed */
|
||||
rig?: { id: string; found: boolean; verified: boolean; erased: Record<string, number>; left: Record<string, number>; unknown: string[];
|
||||
completedAt?: string };
|
||||
routine?: { contentDays: number; metadataDays: number };
|
||||
}
|
||||
|
||||
const DEFAULT_BATCH = 500;
|
||||
const DEFAULT_MAX_BATCHES = 10_000;
|
||||
const DEFAULT_EVENTS_WINDOW = 1_000;
|
||||
/** Rows erased per transaction. Each one rechecks its rows and rewrites their events, so it's kept smaller than a
|
||||
* selection page: on a 2.4M-event store, 500 rows held the daemon up to 0.5 s, and 100 up to 0.25 s. */
|
||||
const ROWS_PER_TRANSACTION = 100;
|
||||
|
||||
function cutoffIso(nowIso: string, days: number): string {
|
||||
const now = new Date(nowIso);
|
||||
if (Number.isNaN(now.getTime())) throw new Error(`queue-erasure: invalid nowIso ${JSON.stringify(nowIso)}`);
|
||||
return new Date(now.getTime() - days * 86_400_000).toISOString();
|
||||
}
|
||||
|
||||
function yieldToLoop(): Promise<void> {
|
||||
return new Promise((resolve) => setImmediate(resolve));
|
||||
}
|
||||
|
||||
const IDS = "(SELECT value FROM json_each(@ids))";
|
||||
const MEMBERS = "(SELECT value FROM json_each(@members))";
|
||||
/** The payload keys a row-naming event uses: queue events name it in qitemId; inbox, workflow and mission-control
|
||||
* events use these others. */
|
||||
const EVENT_ID_KEYS = ["qitemId", "promotedQitemId", "priorQitemId", "nextQitemId"] as const;
|
||||
/** The content keys of row-naming events; a scrubbed event keeps its type and ids. closureTarget goes too, unless it
|
||||
* names a successor. */
|
||||
const EVENT_CONTENT_KEYS = ["summary", "reason", "error"] as const;
|
||||
/** instr is a cheap prefilter for row-naming events: every id key ends in "temId" (qitemId, promotedQitemId, ...). */
|
||||
const NAMES_A_ROW = "instr(payload, 'temId') > 0";
|
||||
|
||||
class Statements {
|
||||
constructor(private readonly db: Database.Database) {}
|
||||
private cache = new Map<string, Database.Statement>();
|
||||
get(sql: string): Database.Statement {
|
||||
let statement = this.cache.get(sql);
|
||||
if (!statement) {
|
||||
statement = this.db.prepare(sql);
|
||||
this.cache.set(sql, statement);
|
||||
}
|
||||
return statement;
|
||||
}
|
||||
count(sql: string, params: Record<string, unknown>): number {
|
||||
return (this.get(sql).get(params) as { n: number }).n;
|
||||
}
|
||||
}
|
||||
|
||||
/** Rows a workflow still references (a step trail, a frontier binding or a failure record point at them by key, or
|
||||
* a live instance's frontier names them): erasing them would break the workflow or its record. */
|
||||
function workflowReferenced(st: Statements, ids: string[]): Set<string> {
|
||||
if (ids.length === 0) return new Set();
|
||||
const rows = st.get(`SELECT value AS id FROM json_each(@ids) j WHERE
|
||||
EXISTS (SELECT 1 FROM workflow_step_trails WHERE next_qitem_id = j.value OR prior_qitem_id = j.value)
|
||||
OR EXISTS (SELECT 1 FROM workflow_frontier_bindings WHERE packet_id = j.value)
|
||||
OR EXISTS (SELECT 1 FROM workflow_failure_occurrences WHERE failed_packet_id = j.value OR redrive_packet_id = j.value)
|
||||
OR EXISTS (SELECT 1 FROM workflow_instances WHERE status IN ('active', 'waiting')
|
||||
AND current_frontier_json LIKE '%"' || j.value || '"%')`).all({ ids: JSON.stringify(ids) }) as Array<{ id: string }>;
|
||||
return new Set(rows.map((row) => row.id));
|
||||
}
|
||||
|
||||
type Candidate = { qitemId: string; state: string; action: EraseAction; source: string; destination: string };
|
||||
type Copy = { table: CopyTable; id: string; qitemId: string; sessions: Array<string | null> };
|
||||
type Current = { qitemId: string; state: string; tsUpdated: string; erasedAt: string | null; source: string; destination: string };
|
||||
|
||||
/** The copies of these rows that another party holds, each found through its own indexed link. */
|
||||
function linkedCopies(st: Statements, ids: string[]): Copy[] {
|
||||
const params = { ids: JSON.stringify(ids) };
|
||||
const copies = new Map<string, Copy>();
|
||||
const collect = (table: CopyTable, sql: string) => {
|
||||
for (const row of st.get(sql).all(params) as Array<{ id: string; qitemId: string; a: string | null; b: string | null }>) {
|
||||
copies.set(`${table}:${row.id}`, { table, id: String(row.id), qitemId: row.qitemId, sessions: [row.a, row.b] });
|
||||
}
|
||||
};
|
||||
collect("outbox_entries", `SELECT outbox_id AS id, audit_pointer AS qitemId, sender_session AS a, destination_session AS b
|
||||
FROM outbox_entries WHERE audit_pointer IN ${IDS}`);
|
||||
collect("outbox_entries", `SELECT outbox_id AS id, substr(outbox_id, 13) AS qitemId, sender_session AS a, destination_session AS b
|
||||
FROM outbox_entries WHERE outbox_id IN (SELECT 'wake-intent-' || value FROM json_each(@ids))`);
|
||||
collect("inbox_entries", `SELECT inbox_id AS id, absorbed_qitem_id AS qitemId, sender_session AS a, destination_session AS b
|
||||
FROM inbox_entries WHERE absorbed_qitem_id IN ${IDS}`);
|
||||
collect("inbox_entries", `SELECT inbox_id AS id, audit_pointer AS qitemId, sender_session AS a, destination_session AS b
|
||||
FROM inbox_entries WHERE audit_pointer IN ${IDS}`);
|
||||
collect("stream_reader_actions", `SELECT action_id AS id, work_qitem_id AS qitemId, reader_session AS a, NULL AS b
|
||||
FROM stream_reader_actions WHERE work_qitem_id IN ${IDS}`);
|
||||
collect("stream_reader_actions", `SELECT action_id AS id, duplicate_of AS qitemId, reader_session AS a, NULL AS b
|
||||
FROM stream_reader_actions WHERE duplicate_of IN ${IDS}`);
|
||||
collect("stream_dispositions", `SELECT stream_item_id AS id, work_qitem_id AS qitemId, accepted_by AS a, owner_session AS b
|
||||
FROM stream_dispositions WHERE work_qitem_id IN ${IDS}`);
|
||||
collect("stream_dispositions", `SELECT stream_item_id AS id, effect_qitem_id AS qitemId, accepted_by AS a, owner_session AS b
|
||||
FROM stream_dispositions WHERE effect_qitem_id IN ${IDS}`);
|
||||
return [...copies.values()];
|
||||
}
|
||||
|
||||
const COPY_KEY: Record<CopyTable, string> = {
|
||||
outbox_entries: "outbox_id", inbox_entries: "inbox_id", stream_reader_actions: "action_id", stream_dispositions: "stream_item_id",
|
||||
};
|
||||
|
||||
function eraseCopies(st: Statements, copies: Copy[], action: EraseAction): void {
|
||||
for (const table of Object.keys(COPY_KEY) as CopyTable[]) {
|
||||
const keys = copies.filter((copy) => copy.table === table).map((copy) => copy.id);
|
||||
if (keys.length === 0) continue;
|
||||
const params = { keys: JSON.stringify(keys) };
|
||||
const where = `${COPY_KEY[table]} IN (SELECT value FROM json_each(@keys))`;
|
||||
if (action === "delete") {
|
||||
st.get(`DELETE FROM ${table} WHERE ${where}`).run(params);
|
||||
} else if (table === "outbox_entries") {
|
||||
st.get(`UPDATE outbox_entries SET body = '', tags = NULL WHERE ${where}`).run(params);
|
||||
} else if (table === "inbox_entries") {
|
||||
st.get(`UPDATE inbox_entries SET body = '', tags = NULL, denied_reason = NULL WHERE ${where}`).run(params);
|
||||
} else if (table === "stream_reader_actions") {
|
||||
st.get(`UPDATE stream_reader_actions SET reason = NULL WHERE ${where}`).run(params);
|
||||
} else {
|
||||
st.get(`UPDATE stream_dispositions SET template_text = '', request_json = '{}', provenance_json = '{}', reason = NULL,
|
||||
result_json = NULL, applied_json = NULL WHERE ${where}`).run(params);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
function scrubRows(st: Statements, ids: string[], nowIso: string): void {
|
||||
const params = { ids: JSON.stringify(ids) };
|
||||
// Closure targets that don't name a successor, found here: SQLite has no pattern match for the id shapes.
|
||||
for (const [table, key] of [["queue_items", "qitem_id"], ["queue_transitions", "transition_id"], ["queue_transitions_archive", "transition_id"]]) {
|
||||
const notes = (st.get(`SELECT ${key} AS k, closure_reason AS reason, closure_target AS target FROM ${table}
|
||||
WHERE qitem_id IN ${IDS} AND closure_target IS NOT NULL`).all(params) as Array<{ k: string | number; reason: string | null; target: string }>)
|
||||
.filter((row) => !keepsClosureTarget(row.reason, row.target)).map((row) => row.k);
|
||||
if (notes.length > 0) {
|
||||
st.get(`UPDATE ${table} SET closure_target = NULL WHERE ${key} IN (SELECT value FROM json_each(@keys))`).run({ keys: JSON.stringify(notes) });
|
||||
}
|
||||
}
|
||||
st.get(`UPDATE queue_items SET body = '', summary = NULL, resolution = NULL, tags = NULL, blocked_on = NULL,
|
||||
chain_of_record = NULL, evidence_ref = NULL, human_detail = NULL, human_questions = NULL, human_answers = NULL,
|
||||
delegated_decision = NULL, last_nudge_input_box = NULL, content_erased_at = @now WHERE qitem_id IN ${IDS}`).run({ ...params, now: nowIso });
|
||||
for (const table of ["queue_transitions", "queue_transitions_archive"]) {
|
||||
st.get(`UPDATE ${table} SET transition_note = NULL, identity_principal = NULL WHERE qitem_id IN ${IDS}`).run(params);
|
||||
}
|
||||
st.get(`UPDATE mission_control_actions SET before_state_json = NULL, after_state_json = NULL, reason = NULL,
|
||||
annotation = NULL, notify_result = NULL, audit_notes_json = NULL WHERE qitem_id IN ${IDS}`).run(params);
|
||||
}
|
||||
|
||||
function deleteRows(st: Statements, ids: string[], nowIso: string, reason: string): void {
|
||||
const params = { ids: JSON.stringify(ids) };
|
||||
// mission_control_actions references queue_items by key, so it goes before the row.
|
||||
st.get(`DELETE FROM mission_control_actions WHERE qitem_id IN ${IDS}`).run(params);
|
||||
st.get(`DELETE FROM queue_transition_wakes WHERE qitem_id IN ${IDS}`).run(params);
|
||||
st.get(`DELETE FROM queue_transitions WHERE qitem_id IN ${IDS}`).run(params);
|
||||
st.get(`DELETE FROM queue_transitions_archive WHERE qitem_id IN ${IDS}`).run(params);
|
||||
st.get(`DELETE FROM queue_items WHERE qitem_id IN ${IDS}`).run(params);
|
||||
st.get(`INSERT OR IGNORE INTO queue_erased_ids (qitem_id, erased_at, reason) SELECT value, @now, @reason FROM json_each(@ids)`)
|
||||
.run({ ...params, now: nowIso, reason });
|
||||
}
|
||||
|
||||
type EventPayload = Record<string, unknown>;
|
||||
/** A payload's own JSON, or null when it doesn't parse to an object. */
|
||||
function parsePayload(text: string): EventPayload | null {
|
||||
try {
|
||||
const value: unknown = JSON.parse(text);
|
||||
return value !== null && typeof value === "object" && !Array.isArray(value) ? value as EventPayload : null;
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
}
|
||||
const namedIds = (payload: EventPayload) => EVENT_ID_KEYS.map((key) => payload[key]).filter((id): id is string => typeof id === "string");
|
||||
|
||||
/** Where each target row's events are: one read-only pass over events in seq windows, up to the seq it started at.
|
||||
* Nothing changes here; each batch changes its own rows' events after rechecking the rows. A payload that names a
|
||||
* target but can't be read is listed as unreadable. */
|
||||
async function walkEvents(db: Database.Database, targets: Set<string>, window: number):
|
||||
Promise<{ seqs: Map<string, number[]>; unreadable: Array<{ seq: number; qitemIds: string[] }>; top: number }> {
|
||||
const found = { seqs: new Map<string, number[]>(), unreadable: [] as Array<{ seq: number; qitemIds: string[] }>, top: 0 };
|
||||
if (targets.size === 0) return found;
|
||||
found.top = (db.prepare("SELECT MAX(seq) AS seq FROM events").get() as { seq: number | null }).seq ?? 0;
|
||||
const read = db.prepare(`SELECT seq, payload FROM events WHERE seq > ? AND seq <= ? AND ${NAMES_A_ROW}`);
|
||||
for (let lo = 0; lo < found.top; lo += window) {
|
||||
for (const row of read.all(lo, Math.min(lo + window, found.top)) as Array<{ seq: number; payload: string }>) {
|
||||
const payload = parsePayload(row.payload);
|
||||
if (payload === null) {
|
||||
const named = [...targets].filter((id) => row.payload.includes(id));
|
||||
if (named.length > 0) found.unreadable.push({ seq: row.seq, qitemIds: named });
|
||||
continue;
|
||||
}
|
||||
for (const id of new Set(namedIds(payload))) {
|
||||
if (!targets.has(id)) continue;
|
||||
const seqs = found.seqs.get(id);
|
||||
if (seqs) seqs.push(row.seq);
|
||||
else found.seqs.set(id, [row.seq]);
|
||||
}
|
||||
}
|
||||
await yieldToLoop();
|
||||
}
|
||||
return found;
|
||||
}
|
||||
|
||||
/** When a closed row entered its final closed state: the earliest transition after the last one recorded in another
|
||||
* state. A note on a closed row keeps its state, so it doesn't move this; a reopened and closed-again row counts
|
||||
* from its latest close. */
|
||||
function closedSince(st: Statements, qitemId: string, state: string, fallback: string): string {
|
||||
const lastOther = (st.get(`SELECT MAX(id) AS id FROM (
|
||||
SELECT transition_id AS id FROM queue_transitions WHERE qitem_id = @q AND state != @s
|
||||
UNION ALL SELECT transition_id AS id FROM queue_transitions_archive WHERE qitem_id = @q AND state != @s)`)
|
||||
.get({ q: qitemId, s: state }) as { id: number | null }).id ?? -1;
|
||||
const since = (st.get(`SELECT MIN(ts) AS ts FROM (
|
||||
SELECT ts FROM queue_transitions WHERE qitem_id = @q AND transition_id > @after
|
||||
UNION ALL SELECT ts FROM queue_transitions_archive WHERE qitem_id = @q AND transition_id > @after)`)
|
||||
.get({ q: qitemId, after: lastOther }) as { ts: string | null }).ts;
|
||||
return since ?? fallback;
|
||||
}
|
||||
|
||||
function membersDigest(members: readonly string[]): string {
|
||||
return createHash("sha256").update(JSON.stringify([...new Set(members)].sort())).digest("hex");
|
||||
}
|
||||
|
||||
type BatchPlan = {
|
||||
rows: Candidate[];
|
||||
absent: string[];
|
||||
dropped: EraseGap[];
|
||||
owned: Record<EraseAction, Copy[]>;
|
||||
kept: EraseReport["keptCopies"];
|
||||
counts: { transitions: number; archivedTransitions: number; missionControl: number };
|
||||
events: { rewrite: Array<[string, number]>; remove: number[]; seqs: number[] };
|
||||
unreadable: Array<{ seq: number; qitemIds: string[] }>;
|
||||
};
|
||||
|
||||
/** Erase queue rows in one scope. With apply false it reports exactly what an apply would erase and writes nothing. */
|
||||
export async function eraseQueue(db: Database.Database, opts: EraseOptions): Promise<EraseReport> {
|
||||
const st = new Statements(db);
|
||||
const batchSize = opts.batchSize ?? DEFAULT_BATCH;
|
||||
const maxBatches = opts.maxBatches ?? DEFAULT_MAX_BATCHES;
|
||||
const { scope } = opts;
|
||||
|
||||
// A rig named with the members must be theirs, both ways: every member's session is named in the rig (departed
|
||||
// members included), and every session of it is a member. A rig already removed is verified only by the record an
|
||||
// earlier run kept with this membership.
|
||||
let rig: { id: string; name: string | null; found: boolean; verified: boolean; completedAt: string | null } | null = null;
|
||||
if (scope.kind === "members" && scope.rig !== undefined) {
|
||||
const name = (db.prepare("SELECT name FROM rigs WHERE id = ?").get(scope.rig) as { name: string } | undefined)?.name ?? null;
|
||||
if (name !== null) {
|
||||
const outside = scope.members.filter((member) => member.split("@")[1] !== name);
|
||||
const rigSessions = (db.prepare(`SELECT DISTINCT s.session_name AS name FROM sessions s JOIN nodes n ON n.id = s.node_id
|
||||
WHERE n.rig_id = ?`).all(scope.rig) as Array<{ name: string }>).map((row) => row.name);
|
||||
const unlisted = rigSessions.filter((session) => !scope.members.includes(session));
|
||||
if (outside.length > 0 || unlisted.length > 0) {
|
||||
throw new EraseRefusal("erase_rig_mismatch", [
|
||||
outside.length > 0 ? `These members aren't sessions of rig ${name}: ${outside.join(", ")}.` : "",
|
||||
unlisted.length > 0 ? `Rig ${name} has sessions the members don't list: ${unlisted.join(", ")}.` : "",
|
||||
].filter(Boolean).join(" "));
|
||||
}
|
||||
rig = { id: scope.rig, name, found: true, verified: true, completedAt: null };
|
||||
} else {
|
||||
const kept = db.prepare(`SELECT rig_name AS name, members_digest AS digest, completed_at AS completedAt
|
||||
FROM queue_erased_rigs WHERE rig_id = ?`).get(scope.rig) as { name: string; digest: string; completedAt: string | null } | undefined;
|
||||
const verified = kept !== undefined && kept.digest === membersDigest(scope.members);
|
||||
rig = { id: scope.rig, name: verified && kept.name !== "" ? kept.name : null, found: false, verified,
|
||||
completedAt: verified ? kept.completedAt : null };
|
||||
}
|
||||
}
|
||||
|
||||
const report: EraseReport = {
|
||||
ok: true, applied: opts.apply, scope: scope.kind, complete: true, clean: false, qitems: [], mixed: [], gaps: [],
|
||||
alreadyAbsent: [], keptCopies: [], unreadableEvents: [],
|
||||
linked: { transitions: 0, archivedTransitions: 0, outbox: 0, inbox: 0, events: 0, streamActions: 0, missionControl: 0 },
|
||||
};
|
||||
if (scope.kind === "routine") report.routine = { contentDays: scope.contentDays, metadataDays: scope.metadataDays };
|
||||
const reason = scope.kind === "members" ? "account-deletion" : scope.kind === "routine" ? "retention-metadata" : "erase-by-id";
|
||||
const ledgerReason = (action: EraseAction) => scope.kind === "routine" && action === "scrub" ? "retention-content" : reason;
|
||||
const memberSet = new Set(scope.kind === "members" ? scope.members : []);
|
||||
const members = JSON.stringify([...memberSet]);
|
||||
const entries = scope.kind === "ids"
|
||||
? [...new Map(scope.entries.map((entry) => [entry.qitemId, entry.action])).entries()].sort(([a], [b]) => (a < b ? -1 : a > b ? 1 : 0))
|
||||
: [];
|
||||
const contentCutoff = scope.kind === "routine" && scope.contentDays > 0 ? cutoffIso(opts.nowIso, scope.contentDays) : null;
|
||||
const metadataCutoff = scope.kind === "routine" && scope.metadataDays > 0 ? cutoffIso(opts.nowIso, scope.metadataDays) : null;
|
||||
|
||||
// What routine retention does with a row now: delete it past the row window, scrub it past the content window while
|
||||
// it has content, or nothing. A scrubbed row written to since (a note appends a transition without touching
|
||||
// ts_updated, and may since have been archived; a reclose moves both) has content again.
|
||||
const routineAction = (row: Current): EraseAction | null => {
|
||||
if (!CLOSED_QUEUE_STATES.includes(row.state as (typeof CLOSED_QUEUE_STATES)[number])) return null;
|
||||
const since = closedSince(st, row.qitemId, row.state, row.tsUpdated);
|
||||
if (metadataCutoff && since < metadataCutoff) return "delete";
|
||||
if (!contentCutoff || since >= contentCutoff) return null;
|
||||
const hasContent = !row.erasedAt || row.tsUpdated > row.erasedAt
|
||||
|| st.get(`SELECT 1 FROM queue_transitions WHERE qitem_id = @q AND ts > @since
|
||||
UNION ALL SELECT 1 FROM queue_transitions_archive WHERE qitem_id = @q AND ts > @since LIMIT 1`)
|
||||
.get({ q: row.qitemId, since: row.erasedAt }) !== undefined;
|
||||
return hasContent ? "scrub" : null;
|
||||
};
|
||||
|
||||
// Ids scope, delete: ids with no row at selection; their markers are written at the end.
|
||||
const absentAtSelect: string[] = [];
|
||||
// One page of candidate rows, keyed on qitem_id so selection moves forward without writing anything.
|
||||
const page = (after: string, offset: number): { rows: Candidate[]; last: string | null; seen: number } => {
|
||||
const select = "qitem_id AS qitemId, state, source_session AS source, destination_session AS destination";
|
||||
if (scope.kind === "ids") {
|
||||
const slice = entries.slice(offset, offset + batchSize);
|
||||
const found = new Map((st.get(`SELECT ${select} FROM queue_items WHERE qitem_id IN ${IDS}`)
|
||||
.all({ ids: JSON.stringify(slice.map(([id]) => id)) }) as Array<Omit<Candidate, "action">>).map((row) => [row.qitemId, row]));
|
||||
const rows: Candidate[] = [];
|
||||
for (const [id, action] of slice) {
|
||||
const row = found.get(id);
|
||||
if (row !== undefined) rows.push({ ...row, action });
|
||||
else if (action === "delete") absentAtSelect.push(id);
|
||||
else report.gaps.push({ code: "not-found", qitemId: id });
|
||||
}
|
||||
return { rows, last: null, seen: slice.length };
|
||||
}
|
||||
if (scope.kind === "members") {
|
||||
const found = st.get(`SELECT ${select} FROM queue_items
|
||||
WHERE qitem_id > @after AND (source_session IN ${MEMBERS} OR destination_session IN ${MEMBERS})
|
||||
ORDER BY qitem_id LIMIT @limit`).all({ members, after, limit: batchSize }) as Array<Omit<Candidate, "action">>;
|
||||
const rows: Candidate[] = [];
|
||||
for (const row of found) {
|
||||
const fromMember = memberSet.has(row.source);
|
||||
const toMember = memberSet.has(row.destination);
|
||||
if (fromMember && toMember) rows.push({ ...row, action: "delete" });
|
||||
else report.mixed.push({ qitemId: row.qitemId, memberEndpoint: fromMember ? "source" : "destination" });
|
||||
}
|
||||
return { rows, last: found.at(-1)?.qitemId ?? null, seen: found.length };
|
||||
}
|
||||
// +state keeps the planner on the primary key, so a page reads forward from @after; the state index would sort
|
||||
// every closed row on each page (1.4 s a page on a 44k-row store).
|
||||
const closed = CLOSED_QUEUE_STATES.map((state) => `'${state}'`).join(", ");
|
||||
const found = st.get(`SELECT ${select}, ts_updated AS tsUpdated, content_erased_at AS erasedAt
|
||||
FROM queue_items WHERE qitem_id > @after AND +state IN (${closed}) ORDER BY qitem_id LIMIT @limit`)
|
||||
.all({ after, limit: batchSize }) as Array<Omit<Candidate, "action"> & { tsUpdated: string; erasedAt: string | null }>;
|
||||
const rows: Candidate[] = [];
|
||||
for (const { tsUpdated, erasedAt, ...row } of found) {
|
||||
const action = routineAction({ ...row, tsUpdated, erasedAt });
|
||||
if (action !== null) rows.push({ ...row, action });
|
||||
}
|
||||
return { rows, last: found.at(-1)?.qitemId ?? null, seen: found.length };
|
||||
};
|
||||
|
||||
// Whose copies these are: the account's (members scope), or the row's own participants (other scopes).
|
||||
const ownerOf = (candidate: Candidate) => scope.kind === "members" ? memberSet : new Set([candidate.source, candidate.destination]);
|
||||
const ownership = (copy: Copy, owners: Set<string>): "owned" | "mixed" | "foreign" => {
|
||||
const sessions = copy.sessions.filter((session): session is string => session !== null);
|
||||
const owned = sessions.filter((session) => owners.has(session)).length;
|
||||
return owned === sessions.length ? "owned" : owned > 0 ? "mixed" : "foreign";
|
||||
};
|
||||
// Stream actions and dispositions this run deletes, so the unlinked pass plans on the same references either way.
|
||||
const plannedActions: string[] = [];
|
||||
const plannedDispositions: string[] = [];
|
||||
const eventSeqs = new Set<number>();
|
||||
const unreadable = new Map<number, Set<string>>();
|
||||
const noteUnreadable = (found: Array<{ seq: number; qitemIds: string[] }>) => {
|
||||
for (const { seq, qitemIds } of found) {
|
||||
const ids = unreadable.get(seq) ?? new Set<string>();
|
||||
for (const id of qitemIds) ids.add(id);
|
||||
unreadable.set(seq, ids);
|
||||
}
|
||||
report.unreadableEvents = [...unreadable.entries()].sort(([a], [b]) => a - b).map(([seq, ids]) => ({ seq, qitemIds: [...ids].sort() }));
|
||||
};
|
||||
const ledger = (rows: Array<{ qitemId: string; action: EraseAction }>) => {
|
||||
if (!opts.apply || !opts.ledgerPath || rows.length === 0) return;
|
||||
appendFileSync(opts.ledgerPath, rows.map((row) => JSON.stringify({ qitemId: row.qitemId, action: row.action,
|
||||
reason: ledgerReason(row.action), erasedAt: opts.nowIso })).join("\n") + "\n");
|
||||
};
|
||||
|
||||
let stage: Stage = "ledger";
|
||||
try {
|
||||
// 0. A ledger that can't be written stops the run before anything changes.
|
||||
if (opts.apply && opts.ledgerPath) appendFileSync(opts.ledgerPath, "");
|
||||
|
||||
// 1. Select every candidate, writing nothing.
|
||||
stage = "select";
|
||||
const candidates: Candidate[] = [];
|
||||
let after = "";
|
||||
let offset = 0;
|
||||
for (let batch = 0; ; batch += 1) {
|
||||
if (batch >= maxBatches) {
|
||||
report.complete = false;
|
||||
break;
|
||||
}
|
||||
const { rows, last, seen } = page(after, offset);
|
||||
if (seen === 0) break;
|
||||
const referenced = workflowReferenced(st, rows.map((row) => row.qitemId));
|
||||
for (const id of referenced) report.gaps.push({ code: "workflow-reference", qitemId: id });
|
||||
candidates.push(...rows.filter((row) => !referenced.has(row.qitemId)));
|
||||
if (last !== null) after = last;
|
||||
offset += seen;
|
||||
if (seen < batchSize) break;
|
||||
await yieldToLoop();
|
||||
}
|
||||
|
||||
// 2. Where the candidates' events are: one read-only walk over the table.
|
||||
stage = "events";
|
||||
const events = await walkEvents(db, new Set(candidates.map((row) => row.qitemId)), opts.eventsWindow ?? DEFAULT_EVENTS_WINDOW);
|
||||
noteUnreadable(events.unreadable);
|
||||
const eventsAt = st.get(`SELECT seq, payload FROM events WHERE seq IN (SELECT value FROM json_each(@seqs))
|
||||
UNION ALL SELECT seq, payload FROM events WHERE seq > @top AND ${NAMES_A_ROW}`);
|
||||
const rewriteEvent = db.prepare("UPDATE events SET payload = ? WHERE seq = ?");
|
||||
const removeEvent = db.prepare("DELETE FROM events WHERE seq = ?");
|
||||
|
||||
// One batch, read at the moment it's erased: each row is rechecked, since anything may have changed it after
|
||||
// selection (a reopen makes it open work again; a workflow may have taken it up), then its copies and events.
|
||||
const unreadableAtWalk = new Set(events.unreadable.flatMap((event) => event.qitemIds));
|
||||
const planBatch = (chunk: Candidate[]): BatchPlan => {
|
||||
const chunkIds = chunk.map((row) => row.qitemId);
|
||||
const current = new Map((st.get(`SELECT qitem_id AS qitemId, state, ts_updated AS tsUpdated, content_erased_at AS erasedAt,
|
||||
source_session AS source, destination_session AS destination FROM queue_items WHERE qitem_id IN ${IDS}`)
|
||||
.all({ ids: JSON.stringify(chunkIds) }) as Current[]).map((row) => [row.qitemId, row]));
|
||||
const referenced = workflowReferenced(st, chunkIds);
|
||||
const plan: BatchPlan = { rows: [], absent: [], dropped: [], owned: { scrub: [], delete: [] }, kept: [],
|
||||
counts: { transitions: 0, archivedTransitions: 0, missionControl: 0 }, events: { rewrite: [], remove: [], seqs: [] }, unreadable: [] };
|
||||
for (const candidate of chunk) {
|
||||
const now = current.get(candidate.qitemId);
|
||||
if (now === undefined) {
|
||||
// Gone since selection: a delete still writes its marker; a scrub has nothing to scrub.
|
||||
if (candidate.action === "delete") plan.absent.push(candidate.qitemId);
|
||||
else plan.dropped.push({ code: "changed-during-run", qitemId: candidate.qitemId });
|
||||
} else if (referenced.has(candidate.qitemId)) {
|
||||
plan.dropped.push({ code: "workflow-reference", qitemId: candidate.qitemId });
|
||||
} else if (scope.kind === "members" && !(memberSet.has(now.source) && memberSet.has(now.destination))) {
|
||||
// Rerouted since selection (a fallback, say): no longer the account's own row.
|
||||
plan.dropped.push({ code: "changed-during-run", qitemId: candidate.qitemId });
|
||||
} else if (scope.kind === "routine" && routineAction(now) !== candidate.action) {
|
||||
plan.dropped.push({ code: "changed-during-run", qitemId: candidate.qitemId });
|
||||
} else {
|
||||
// The current endpoints decide whose the row's copies are.
|
||||
plan.rows.push({ ...candidate, state: now.state, source: now.source, destination: now.destination });
|
||||
}
|
||||
}
|
||||
if (plan.rows.length === 0) return plan;
|
||||
// The walk's events for these rows, plus any written since it began. A row an unreadable event names is kept:
|
||||
// what that event holds can't be inspected, and keeping the row means every rerun finds the event and reports
|
||||
// it until it's resolved.
|
||||
const seqs = [...new Set(plan.rows.flatMap((row) => events.seqs.get(row.qitemId) ?? []))];
|
||||
const fetched = new Map<number, string>();
|
||||
for (const event of eventsAt.all({ seqs: JSON.stringify(seqs), top: events.top }) as Array<{ seq: number; payload: string }>) {
|
||||
fetched.set(event.seq, event.payload);
|
||||
}
|
||||
const blocked = new Set(plan.rows.map((row) => row.qitemId).filter((id) => unreadableAtWalk.has(id)));
|
||||
const readable: Array<[number, EventPayload]> = [];
|
||||
for (const [seq, text] of fetched) {
|
||||
const payload = parsePayload(text);
|
||||
if (payload !== null) {
|
||||
readable.push([seq, payload]);
|
||||
continue;
|
||||
}
|
||||
const named = plan.rows.map((row) => row.qitemId).filter((id) => text.includes(id));
|
||||
if (named.length === 0) continue;
|
||||
plan.unreadable.push({ seq, qitemIds: named });
|
||||
for (const id of named) blocked.add(id);
|
||||
}
|
||||
if (blocked.size > 0) {
|
||||
plan.dropped.push(...[...blocked].sort().map((qitemId) => ({ code: "unreadable-event" as const, qitemId })));
|
||||
plan.rows = plan.rows.filter((row) => !blocked.has(row.qitemId));
|
||||
}
|
||||
const ids = plan.rows.map((row) => row.qitemId);
|
||||
if (ids.length === 0) return plan;
|
||||
const byId = new Map(plan.rows.map((row) => [row.qitemId, row]));
|
||||
for (const copy of linkedCopies(st, ids)) {
|
||||
const candidate = byId.get(copy.qitemId)!;
|
||||
const whose = ownership(copy, ownerOf(candidate));
|
||||
if (whose === "owned") plan.owned[candidate.action].push(copy);
|
||||
else plan.kept.push({ table: copy.table, id: copy.id, qitemId: copy.qitemId, ownership: whose });
|
||||
}
|
||||
const params = { ids: JSON.stringify(ids) };
|
||||
plan.counts = {
|
||||
transitions: st.count(`SELECT COUNT(*) AS n FROM queue_transitions WHERE qitem_id IN ${IDS}`, params),
|
||||
archivedTransitions: st.count(`SELECT COUNT(*) AS n FROM queue_transitions_archive WHERE qitem_id IN ${IDS}`, params),
|
||||
missionControl: st.count(`SELECT COUNT(*) AS n FROM mission_control_actions WHERE qitem_id IN ${IDS}`, params),
|
||||
};
|
||||
// An event naming two rows is deleted if either is deleted; one naming rows of a later batch is changed again
|
||||
// there.
|
||||
for (const [seq, payload] of readable) {
|
||||
const actions = namedIds(payload).map((id) => byId.get(id)?.action).filter((action): action is EraseAction => action !== undefined);
|
||||
if (actions.length === 0) continue;
|
||||
plan.events.seqs.push(seq);
|
||||
if (actions.includes("delete")) {
|
||||
plan.events.remove.push(seq);
|
||||
} else {
|
||||
for (const key of EVENT_CONTENT_KEYS) delete payload[key];
|
||||
if (!keepsClosureTarget(payload.closureReason, payload.closureTarget)) delete payload.closureTarget;
|
||||
payload.contentErased = true;
|
||||
plan.events.rewrite.push([JSON.stringify(payload), seq]);
|
||||
}
|
||||
}
|
||||
return plan;
|
||||
};
|
||||
|
||||
// 3. The rows, their events and their other copies, a batch per transaction. A row is listed once its batch has
|
||||
// committed; its ledger lines are written in the transaction, before anything in it changes.
|
||||
stage = "rows";
|
||||
const perTransaction = Math.min(batchSize, ROWS_PER_TRANSACTION);
|
||||
for (let start = 0; start < candidates.length; start += perTransaction) {
|
||||
const chunk = candidates.slice(start, start + perTransaction);
|
||||
const plan = opts.apply
|
||||
? db.transaction(() => {
|
||||
const planned = planBatch(chunk);
|
||||
ledger([...planned.rows, ...planned.absent.map((qitemId) => ({ qitemId, action: "delete" as const }))]);
|
||||
for (const [payload, seq] of planned.events.rewrite) rewriteEvent.run(payload, seq);
|
||||
for (const seq of planned.events.remove) removeEvent.run(seq);
|
||||
eraseCopies(st, planned.owned.scrub, "scrub");
|
||||
eraseCopies(st, planned.owned.delete, "delete");
|
||||
const scrub = planned.rows.filter((row) => row.action === "scrub").map((row) => row.qitemId);
|
||||
const remove = [...planned.rows.filter((row) => row.action === "delete").map((row) => row.qitemId), ...planned.absent];
|
||||
if (scrub.length > 0) scrubRows(st, scrub, opts.nowIso);
|
||||
if (remove.length > 0) deleteRows(st, remove, opts.nowIso, reason);
|
||||
return planned;
|
||||
})()
|
||||
: planBatch(chunk);
|
||||
const all = [...plan.owned.scrub, ...plan.owned.delete];
|
||||
report.linked.transitions += plan.counts.transitions;
|
||||
report.linked.archivedTransitions += plan.counts.archivedTransitions;
|
||||
report.linked.missionControl += plan.counts.missionControl;
|
||||
report.linked.outbox += all.filter((copy) => copy.table === "outbox_entries").length;
|
||||
report.linked.inbox += all.filter((copy) => copy.table === "inbox_entries").length;
|
||||
report.linked.streamActions += all.filter((copy) => copy.table === "stream_reader_actions" || copy.table === "stream_dispositions").length;
|
||||
for (const seq of plan.events.seqs) eventSeqs.add(seq);
|
||||
report.linked.events = eventSeqs.size;
|
||||
noteUnreadable(plan.unreadable);
|
||||
plannedActions.push(...plan.owned.delete.filter((copy) => copy.table === "stream_reader_actions").map((copy) => copy.id));
|
||||
plannedDispositions.push(...plan.owned.delete.filter((copy) => copy.table === "stream_dispositions").map((copy) => copy.id));
|
||||
report.keptCopies.push(...plan.kept);
|
||||
report.gaps.push(...plan.dropped);
|
||||
report.alreadyAbsent.push(...plan.absent);
|
||||
report.qitems.push(...plan.rows.map(({ qitemId, action, state }) => ({ qitemId, action, state })));
|
||||
await yieldToLoop();
|
||||
}
|
||||
if (absentAtSelect.length > 0) {
|
||||
// Rechecked as the markers are written: an id created since selection is live work, never certified absent.
|
||||
const certify = () => {
|
||||
const present = new Set((st.get(`SELECT qitem_id AS id FROM queue_items WHERE qitem_id IN ${IDS}`)
|
||||
.all({ ids: JSON.stringify(absentAtSelect) }) as Array<{ id: string }>).map((row) => row.id));
|
||||
const absent = absentAtSelect.filter((id) => !present.has(id));
|
||||
if (opts.apply && absent.length > 0) {
|
||||
ledger(absent.map((qitemId) => ({ qitemId, action: "delete" as const })));
|
||||
st.get(`INSERT OR IGNORE INTO queue_erased_ids (qitem_id, erased_at, reason) SELECT value, @now, @reason FROM json_each(@ids)`)
|
||||
.run({ ids: JSON.stringify(absent), now: opts.nowIso, reason });
|
||||
}
|
||||
return { absent, created: absentAtSelect.filter((id) => present.has(id)) };
|
||||
};
|
||||
const { absent, created } = opts.apply ? db.transaction(certify)() : certify();
|
||||
report.gaps.push(...created.map((qitemId) => ({ code: "changed-during-run" as const, qitemId })));
|
||||
report.alreadyAbsent.push(...absent);
|
||||
}
|
||||
|
||||
// 4. Account deletion's other passes, once every row is done.
|
||||
if (scope.kind === "members" && report.complete) {
|
||||
stage = "unlinked";
|
||||
report.unlinked = eraseUnlinked(db, st, {
|
||||
members, plannedActions, plannedDispositions, erasedRows: report.qitems.map((row) => row.qitemId),
|
||||
}, opts.apply);
|
||||
if (rig !== null) {
|
||||
stage = "rig";
|
||||
report.rig = { id: rig.id, found: rig.found, verified: rig.verified, erased: {}, left: {}, unknown: [] };
|
||||
await eraseRigRecords(db, st, { ...rig, members, digest: membersDigest(scope.members) }, opts.apply, opts.nowIso, report.rig);
|
||||
}
|
||||
}
|
||||
} catch (err) {
|
||||
// Everything listed so far was committed; the caller keeps those ids, and a rerun continues.
|
||||
report.complete = false;
|
||||
report.error = { stage, message: err instanceof Error ? err.message : String(err) };
|
||||
}
|
||||
|
||||
const unlinkedLeft = report.unlinked ? report.unlinked.mixedEntries + report.unlinked.referencedStreamItems : 0;
|
||||
const rigLeft = report.rig ? Object.values(report.rig.left).reduce((sum, n) => sum + n, 0) + report.rig.unknown.length : 0;
|
||||
report.clean = report.complete && report.mixed.length === 0 && report.gaps.length === 0 && unlinkedLeft === 0 && rigLeft === 0
|
||||
&& report.unreadableEvents.length === 0 && !report.keptCopies.some((copy) => copy.ownership === "mixed");
|
||||
return report;
|
||||
}
|
||||
|
||||
/** Account deletion's second pass: outbox and inbox entries between two members that no queue row owns, and the
|
||||
* members' stream items. An entry with one member endpoint, or a stream item something still references, is counted
|
||||
* and left. Rows and references this run erases are counted as they were before it, so a dry run and an apply report
|
||||
* the same set. */
|
||||
function eraseUnlinked(db: Database.Database, st: Statements,
|
||||
plan: { members: string; plannedActions: string[]; plannedDispositions: string[]; erasedRows: string[] }, apply: boolean): NonNullable<EraseReport["unlinked"]> {
|
||||
// A row this run erased still owns its copies here: they were reported with it (keptCopies), not as unowned.
|
||||
const rows = "(SELECT qitem_id FROM queue_items UNION ALL SELECT value FROM json_each(@rows))";
|
||||
// COALESCE: an entry with no pointer is unowned, not unknown (NULL IN (...) would drop it from both counts).
|
||||
const ownedOutbox = `(COALESCE(audit_pointer, '') IN ${rows}
|
||||
OR (outbox_id LIKE 'wake-intent-%' AND substr(outbox_id, 13) IN ${rows}))`;
|
||||
const ownedInbox = `(COALESCE(absorbed_qitem_id, '') IN ${rows} OR COALESCE(audit_pointer, '') IN ${rows})`;
|
||||
const both = `sender_session IN ${MEMBERS} AND destination_session IN ${MEMBERS}`;
|
||||
const fromMember = `sender_session IN ${MEMBERS} AND destination_session NOT IN ${MEMBERS}`;
|
||||
const toMember = `destination_session IN ${MEMBERS} AND sender_session NOT IN ${MEMBERS}`;
|
||||
// The same references the stream pass keeps an item for (queue-retention pruneStreamItems), less this run's own.
|
||||
const classificationColumns = new Set((db.prepare("PRAGMA table_info(project_classifications)").all() as Array<{ name: string }>)
|
||||
.map((column) => column.name));
|
||||
const notPlannedAction = "action_id NOT IN (SELECT value FROM json_each(@actions))";
|
||||
const references = [
|
||||
"SELECT stream_item_id FROM project_classifications WHERE stream_item_id IS NOT NULL",
|
||||
...(classificationColumns.has("duplicate_of_stream_item_id")
|
||||
? ["SELECT duplicate_of_stream_item_id FROM project_classifications WHERE duplicate_of_stream_item_id IS NOT NULL"] : []),
|
||||
"SELECT stream_item_id FROM classification_attempts WHERE stream_item_id IS NOT NULL",
|
||||
"SELECT stream_item_id FROM stream_dispositions WHERE stream_item_id IS NOT NULL AND stream_item_id NOT IN (SELECT value FROM json_each(@dispositions))",
|
||||
`SELECT stream_item_id FROM stream_reader_actions WHERE stream_item_id IS NOT NULL AND ${notPlannedAction}`,
|
||||
`SELECT duplicate_of FROM stream_reader_actions WHERE duplicate_of IS NOT NULL AND ${notPlannedAction}`,
|
||||
];
|
||||
const referenced = `stream_item_id IN (${references.join(" UNION ")})`;
|
||||
const params = { members: plan.members, actions: JSON.stringify(plan.plannedActions), dispositions: JSON.stringify(plan.plannedDispositions),
|
||||
rows: JSON.stringify(plan.erasedRows) };
|
||||
const result = {
|
||||
outbox: st.count(`SELECT COUNT(*) AS n FROM outbox_entries WHERE ${both} AND NOT ${ownedOutbox}`, params),
|
||||
inbox: st.count(`SELECT COUNT(*) AS n FROM inbox_entries WHERE ${both} AND NOT ${ownedInbox}`, params),
|
||||
streamItems: st.count(`SELECT COUNT(*) AS n FROM stream_items WHERE source_session IN ${MEMBERS} AND NOT ${referenced}`, params),
|
||||
// A row's own copies are reported with that row (mixed, keptCopies); these are the entries no row owns.
|
||||
mixedEntries: [fromMember, toMember].reduce((sum, where) =>
|
||||
sum + st.count(`SELECT COUNT(*) AS n FROM outbox_entries WHERE ${where} AND NOT ${ownedOutbox}`, params)
|
||||
+ st.count(`SELECT COUNT(*) AS n FROM inbox_entries WHERE ${where} AND NOT ${ownedInbox}`, params), 0),
|
||||
referencedStreamItems: st.count(`SELECT COUNT(*) AS n FROM stream_items WHERE source_session IN ${MEMBERS} AND ${referenced}`, params),
|
||||
};
|
||||
if (apply) {
|
||||
db.transaction(() => {
|
||||
st.get(`DELETE FROM outbox_entries WHERE ${both} AND NOT ${ownedOutbox}`).run(params);
|
||||
st.get(`DELETE FROM inbox_entries WHERE ${both} AND NOT ${ownedInbox}`).run(params);
|
||||
st.get(`DELETE FROM stream_items WHERE source_session IN ${MEMBERS} AND NOT ${referenced}`).run(params);
|
||||
})();
|
||||
}
|
||||
return result;
|
||||
}
|
||||
|
||||
// The account rig's own records that outlive the rig (no cascading key to it), in delete order: dependents first.
|
||||
// byRigId: keyed by the rig's id alone, so erased only for a verified id. NODES: the rig's nodes, while it has them.
|
||||
const NODES = "(SELECT id FROM nodes WHERE rig_id = @rig)";
|
||||
const RIG_RECORDS: Array<{ table: string; where: string; byRigId: boolean }> = [
|
||||
{ table: "events", where: "rig_id = @rig", byRigId: true },
|
||||
{ table: "events", where: `node_id IN ${NODES}`, byRigId: false },
|
||||
{ table: "snapshots", where: "rig_id = @rig", byRigId: true },
|
||||
{ table: "operation_correlations", where: "rig_id = @rig", byRigId: true },
|
||||
{ table: "bootstrap_actions", where: "bootstrap_id IN (SELECT id FROM bootstrap_runs WHERE rig_id = @rig)", byRigId: true },
|
||||
{ table: "bootstrap_runs", where: "rig_id = @rig", byRigId: true },
|
||||
{ table: "seat_identity_verdicts", where: `node_id IN ${NODES} OR session_name IN ${MEMBERS}`, byRigId: false },
|
||||
{ table: "seat_delivery_guard_changes", where: `node_id IN ${NODES}`, byRigId: false },
|
||||
{ table: "environment_operations",
|
||||
where: `binding_id IN (SELECT binding_id FROM environment_bindings WHERE node_id IN ${NODES} OR seat IN ${MEMBERS})`, byRigId: false },
|
||||
{ table: "environment_bindings", where: `node_id IN ${NODES} OR seat IN ${MEMBERS}`, byRigId: false },
|
||||
{ table: "usage_samples", where: `node_id IN ${NODES} OR seat_session IN ${MEMBERS}`, byRigId: false },
|
||||
{ table: "pi_usage_entries", where: `node_id IN ${NODES} OR seat_session IN ${MEMBERS}`, byRigId: false },
|
||||
{ table: "pi_usage_charges", where: `seat_session IN ${MEMBERS}`, byRigId: false },
|
||||
{ table: "watchdog_history",
|
||||
where: `delivery_target_session IN ${MEMBERS} OR job_id IN (SELECT job_id FROM watchdog_jobs WHERE target_session IN ${MEMBERS})`, byRigId: false },
|
||||
{ table: "watchdog_jobs", where: `target_session IN ${MEMBERS}`, byRigId: false },
|
||||
{ table: "stream_subscriptions", where: `reader_session IN ${MEMBERS}`, byRigId: false },
|
||||
{ table: "thread_seat_map", where: `seat IN ${MEMBERS}`, byRigId: false },
|
||||
{ table: "thread_part_map", where: `seat IN ${MEMBERS}`, byRigId: false },
|
||||
];
|
||||
// Records that name the account but also belong to someone or something else: counted and left. byName: matched by
|
||||
// the rig's name, which an unverified removed rig doesn't have, so those are unknown rather than zero.
|
||||
const RIG_RECORDS_LEFT: Array<{ table: string; where: string; byName: boolean }> = [
|
||||
{ table: "watchdog_jobs", where: `registered_by_session IN ${MEMBERS} AND target_session NOT IN ${MEMBERS}`, byName: false },
|
||||
{ table: "workflow_instances", where: `created_by_session IN ${MEMBERS}`, byName: false },
|
||||
{ table: "workflow_instances", where: "bound_rig = @rigName", byName: true },
|
||||
{ table: "workflow_specs", where: "target_rig = @rigName", byName: true },
|
||||
{ table: "classifier_leases", where: `classifier_session IN ${MEMBERS} OR reclaimed_by_session IN ${MEMBERS}`, byName: false },
|
||||
{ table: "views_custom", where: `registered_by_session IN ${MEMBERS}`, byName: false },
|
||||
];
|
||||
const RIG_RECORDS_CHUNK = 1_000;
|
||||
|
||||
/** The account rig's records, counted into `out` as each chunk commits, so a pass that stops partway still reports
|
||||
* what it erased. */
|
||||
async function eraseRigRecords(db: Database.Database, st: Statements,
|
||||
rig: { id: string; name: string | null; found: boolean; verified: boolean; completedAt: string | null; members: string; digest: string },
|
||||
apply: boolean, nowIso: string, out: NonNullable<EraseReport["rig"]>): Promise<void> {
|
||||
const tables = new Set((db.prepare("SELECT name FROM sqlite_master WHERE type = 'table'").all() as Array<{ name: string }>)
|
||||
.map((row) => row.name));
|
||||
const params = { rig: rig.id, rigName: rig.name ?? "", members: rig.members };
|
||||
const add = (into: Record<string, number>, table: string, n: number) => {
|
||||
into[table] = (into[table] ?? 0) + n;
|
||||
};
|
||||
for (const { table, where, byRigId } of RIG_RECORDS) {
|
||||
if (!tables.has(table)) continue;
|
||||
if (byRigId && !rig.verified) {
|
||||
// An unverified id: its records are counted and left, never erased on a guess.
|
||||
add(out.left, table, st.count(`SELECT COUNT(*) AS n FROM ${table} WHERE ${where}`, params));
|
||||
continue;
|
||||
}
|
||||
if (!apply) {
|
||||
add(out.erased, table, st.count(`SELECT COUNT(*) AS n FROM ${table} WHERE ${where}`, params));
|
||||
continue;
|
||||
}
|
||||
add(out.erased, table, 0);
|
||||
// In chunks by rowid (every one of these tables has one; a declared key can be NULL), so a large table (events)
|
||||
// never holds the daemon in one statement.
|
||||
const chunk = st.get(`DELETE FROM ${table} WHERE rowid IN (SELECT rowid FROM ${table} WHERE ${where} LIMIT ${RIG_RECORDS_CHUNK})`);
|
||||
for (let changes = chunk.run(params).changes; changes > 0; changes = chunk.run(params).changes) {
|
||||
add(out.erased, table, changes);
|
||||
await yieldToLoop();
|
||||
}
|
||||
}
|
||||
const unknown = new Set<string>();
|
||||
for (const { table, where, byName } of RIG_RECORDS_LEFT) {
|
||||
if (!tables.has(table)) continue;
|
||||
if (byName && rig.name === null) {
|
||||
// No name to count by. After a completed erasure that's expected: these were empty when it completed.
|
||||
if (rig.completedAt === null) unknown.add(table);
|
||||
} else {
|
||||
add(out.left, table, st.count(`SELECT COUNT(*) AS n FROM ${table} WHERE ${where}`, params));
|
||||
}
|
||||
}
|
||||
out.unknown = [...unknown];
|
||||
const leftTotal = Object.values(out.left).reduce((sum, n) => sum + n, 0);
|
||||
if (rig.completedAt !== null) out.completedAt = rig.completedAt;
|
||||
if (apply && rig.verified) {
|
||||
if (rig.found) {
|
||||
// While the rig exists, its id and name are kept with this membership, so the run after its removal is verified.
|
||||
db.prepare(`INSERT INTO queue_erased_rigs (rig_id, rig_name, members_digest, verified_at, completed_at) VALUES (?, ?, ?, ?, NULL)
|
||||
ON CONFLICT(rig_id) DO UPDATE SET rig_name = excluded.rig_name, members_digest = excluded.members_digest,
|
||||
verified_at = excluded.verified_at, completed_at = NULL`).run(rig.id, rig.name, rig.digest, nowIso);
|
||||
} else if (leftTotal === 0 && unknown.size === 0 && rig.completedAt === null) {
|
||||
// After the removal, once nothing of the rig is left, the record becomes a completion receipt: the name goes,
|
||||
// and the id, membership digest and time stay, so an identical rerun (say, after a lost response) still
|
||||
// verifies and reports the same clean result.
|
||||
db.prepare("UPDATE queue_erased_rigs SET rig_name = '', completed_at = ? WHERE rig_id = ?").run(nowIso, rig.id);
|
||||
out.completedAt = nowIso;
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -308,6 +308,9 @@ export interface QueueItem {
|
||||
* (convention C3). NULL for all non-human-routed items (BR-1); required
|
||||
* at the domain write path only when the §5 predicate is true. */
|
||||
evidenceRef: string | null;
|
||||
/** Set only on a row whose content queue erasure removed (queue-erasure.ts): its body and other content are empty
|
||||
* because they were erased, not because the row was written that way. */
|
||||
contentErasedAt?: string;
|
||||
/** Present only on compact list rows, which omit these fields entirely so omitted content
|
||||
* can't be mistaken for an empty value. Full reads never carry this marker. */
|
||||
fieldsElided?: Array<(typeof COMPACT_ELIDED_FIELDS)[number]>;
|
||||
@@ -363,6 +366,7 @@ interface QueueItemRow {
|
||||
delegated_decision?: string | null;
|
||||
summary: string | null;
|
||||
evidence_ref: string | null;
|
||||
content_erased_at?: string | null;
|
||||
closure_reason: string | null;
|
||||
closure_target: string | null;
|
||||
closure_required_at: string | null;
|
||||
@@ -1075,6 +1079,7 @@ export class QueueRepository {
|
||||
private readonly hasMintingGenColumn: boolean;
|
||||
private readonly hasClaimedGenColumn: boolean;
|
||||
private readonly hasQueueTransitionsTable: boolean;
|
||||
private readonly hasErasedIdsTable: boolean;
|
||||
private readonly hasTransitionProvenanceColumn: boolean;
|
||||
private readonly hasOwnerNotificationColumns: boolean;
|
||||
private readonly loadHumanRegistryFn: () => LoadResult;
|
||||
@@ -1157,6 +1162,7 @@ export class QueueRepository {
|
||||
this.hasDelegatedDecisionColumn = detectQueueColumn(db, "delegated_decision");
|
||||
this.hasEvidenceRefColumn = detectQueueColumn(db, "evidence_ref");
|
||||
this.hasQueueTransitionsTable = detectTable(db, "queue_transitions");
|
||||
this.hasErasedIdsTable = detectTable(db, "queue_erased_ids");
|
||||
const transitionColumns = this.hasQueueTransitionsTable
|
||||
? new Set((db.prepare("PRAGMA table_info(queue_transitions)").all() as Array<{ name: string }>).map((column) => column.name))
|
||||
: new Set<string>();
|
||||
@@ -2274,6 +2280,11 @@ export class QueueRepository {
|
||||
}
|
||||
this.assertHumanAlert(input.humanAlert, input.destinationSession, input.humanIntent);
|
||||
const id = input.qitemId ?? newQitemId();
|
||||
// A row queue erasure deleted keeps only its id: a create that reuses it would bring erased work back.
|
||||
if (input.qitemId && this.hasErasedIdsTable
|
||||
&& this.db.prepare("SELECT 1 FROM queue_erased_ids WHERE qitem_id = ?").get(input.qitemId)) {
|
||||
throw new QueueRepositoryError("qitem_erased", `qitem ${input.qitemId} was erased; its id can't be used again.`);
|
||||
}
|
||||
const ts = new Date().toISOString();
|
||||
const priority = input.priority ?? "routine";
|
||||
const tier = input.tier ?? null;
|
||||
@@ -5641,6 +5652,7 @@ export class QueueRepository {
|
||||
// PL-007: target_repo present only when migration 038 has applied;
|
||||
// older test fixtures supply legacy rows where target_repo is undefined.
|
||||
targetRepo: row.target_repo ?? null,
|
||||
...(row.content_erased_at ? { contentErasedAt: row.content_erased_at } : {}),
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1,5 +1,6 @@
|
||||
import type Database from "better-sqlite3";
|
||||
import { TERMINAL_QUEUE_STATES } from "./queue-repository.js";
|
||||
import { eraseQueue } from "./queue-erasure.js";
|
||||
|
||||
/**
|
||||
* OPR.0.4.6 FS-1 (daemon read-path hardening) — W2 retention/prune runner.
|
||||
@@ -93,6 +94,10 @@ export const RETENTION_DEFAULTS = {
|
||||
eventsWindow: 1_000,
|
||||
/** pause between two events windows, after the event-loop yield */
|
||||
eventsPauseMs: 5,
|
||||
/** remove the content of closed work (done, canceled, denied, handed-off) this many days after it closed; 0 keeps it */
|
||||
closedContentDays: 0,
|
||||
/** delete closed rows, keeping only their ids, this many days after they closed; 0 keeps them */
|
||||
closedMetadataDays: 0,
|
||||
} as const;
|
||||
|
||||
/**
|
||||
@@ -128,6 +133,10 @@ export interface RetentionOptions {
|
||||
eventsRetentionDays?: number;
|
||||
eventsWindow?: number;
|
||||
eventsPauseMs?: number;
|
||||
closedContentDays?: number;
|
||||
closedMetadataDays?: number;
|
||||
/** where closed-work erasure appends the ids and actions it erases (retention.erasure_ledger); unset keeps none */
|
||||
erasureLedgerPath?: string;
|
||||
/** the events pass's yield between windows; injectable so a test can count it */
|
||||
eventsYield?: () => Promise<void>;
|
||||
/** called once per events statement with its seq bounds (lo, hi]; for tests and measurement */
|
||||
@@ -528,6 +537,12 @@ export interface RetentionSweepSummary {
|
||||
eventsBatches: number;
|
||||
/** the events low-water seq persisted after the pass (null when none is stored) */
|
||||
eventsLowWater: number | null;
|
||||
/** closed rows whose content was removed, rows deleted, and rows a workflow kept (queue-erasure, routine) */
|
||||
closedScrubbed: number;
|
||||
closedDeleted: number;
|
||||
closedKept: number;
|
||||
/** why closed-work erasure stopped, when it did; the other passes still ran */
|
||||
closedError: string | null;
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -554,8 +569,33 @@ export async function runQueueRetentionSweep(
|
||||
eventsDeleted: 0,
|
||||
eventsBatches: 0,
|
||||
eventsLowWater: null,
|
||||
closedScrubbed: 0,
|
||||
closedDeleted: 0,
|
||||
closedKept: 0,
|
||||
closedError: null,
|
||||
};
|
||||
|
||||
// Closed-work retention, opt-in: content first, then the row itself, each from when the row closed.
|
||||
const closedContentDays = opts.closedContentDays ?? RETENTION_DEFAULTS.closedContentDays;
|
||||
const closedMetadataDays = opts.closedMetadataDays ?? RETENTION_DEFAULTS.closedMetadataDays;
|
||||
if (closedContentDays > 0 || closedMetadataDays > 0) {
|
||||
// Reported, never fatal: an erasure that stops leaves the other passes to run (it returns what it committed).
|
||||
try {
|
||||
const closed = await eraseQueue(db, {
|
||||
scope: { kind: "routine", contentDays: closedContentDays, metadataDays: closedMetadataDays },
|
||||
apply: true, nowIso: opts.nowIso, batchSize: opts.batchSize ?? RETENTION_DEFAULTS.batchSize, maxBatches,
|
||||
...(opts.erasureLedgerPath ? { ledgerPath: opts.erasureLedgerPath } : {}),
|
||||
});
|
||||
summary.closedScrubbed = closed.qitems.filter((row) => row.action === "scrub").length;
|
||||
summary.closedDeleted = closed.qitems.filter((row) => row.action === "delete").length;
|
||||
summary.closedKept = closed.gaps.length;
|
||||
summary.closedError = closed.error ? `${closed.error.stage}: ${closed.error.message}` : null;
|
||||
} catch (err) {
|
||||
summary.closedError = err instanceof Error ? err.message : String(err);
|
||||
}
|
||||
if (summary.closedError) console.error(`[queue-retention] closed-work erasure stopped: ${summary.closedError}`);
|
||||
}
|
||||
|
||||
for (let i = 0; i < maxBatches; i++) {
|
||||
const batch = archiveAgedTerminalTransitions(db, opts);
|
||||
if (batch.archivedQitems === 0) break;
|
||||
|
||||
@@ -229,6 +229,12 @@ export const SETTINGS_VALID_KEYS = [
|
||||
"retention.usage_samples_days",
|
||||
// The history-only event types of `events` (queue-retention PRUNABLE_EVENT_TYPES) older than N days; 0 (default) keeps them.
|
||||
"retention.events_days",
|
||||
// Closed queue work (done, canceled, denied, handed-off): its content goes N days after it closed, then the row, keeping
|
||||
// only its id (queue-erasure). 0 (default) keeps them.
|
||||
"retention.closed_content_days",
|
||||
"retention.closed_metadata_days",
|
||||
// Where queue erasure appends each erased id and action (JSON lines), for replaying after a restore; empty keeps none.
|
||||
"retention.erasure_ledger",
|
||||
"retention.watchdog_keep_per_job",
|
||||
// Slice 22 outcome 9 — stream retention: unreferenced items older than N days or beyond the newest N.
|
||||
"retention.stream_days",
|
||||
@@ -355,6 +361,9 @@ const ENV_MAP: Record<SettingsValidKey, { primary: string; legacy?: string }> =
|
||||
"retention.watchdog_days": { primary: "OPENRIG_RETENTION_WATCHDOG_DAYS" },
|
||||
"retention.usage_samples_days": { primary: "OPENRIG_RETENTION_USAGE_SAMPLES_DAYS" },
|
||||
"retention.events_days": { primary: "OPENRIG_RETENTION_EVENTS_DAYS" },
|
||||
"retention.closed_content_days": { primary: "OPENRIG_RETENTION_CLOSED_CONTENT_DAYS" },
|
||||
"retention.closed_metadata_days": { primary: "OPENRIG_RETENTION_CLOSED_METADATA_DAYS" },
|
||||
"retention.erasure_ledger": { primary: "OPENRIG_RETENTION_ERASURE_LEDGER" },
|
||||
"retention.watchdog_keep_per_job": { primary: "OPENRIG_RETENTION_WATCHDOG_KEEP_PER_JOB" },
|
||||
"retention.stream_days": { primary: "OPENRIG_RETENTION_STREAM_DAYS" },
|
||||
"retention.stream_max_items": { primary: "OPENRIG_RETENTION_STREAM_MAX_ITEMS" },
|
||||
@@ -459,6 +468,9 @@ const KEY_TO_PATH: Record<SettingsValidKey, string[]> = {
|
||||
"retention.watchdog_days": ["retention", "watchdogDays"],
|
||||
"retention.usage_samples_days": ["retention", "usageSamplesDays"],
|
||||
"retention.events_days": ["retention", "eventsDays"],
|
||||
"retention.closed_content_days": ["retention", "closedContentDays"],
|
||||
"retention.closed_metadata_days": ["retention", "closedMetadataDays"],
|
||||
"retention.erasure_ledger": ["retention", "erasureLedger"],
|
||||
"retention.watchdog_keep_per_job": ["retention", "watchdogKeepPerJob"],
|
||||
"retention.stream_days": ["retention", "streamDays"],
|
||||
"retention.stream_max_items": ["retention", "streamMaxItems"],
|
||||
@@ -781,6 +793,9 @@ function getDefaultValue(key: SettingsValidKey, workspaceRoot: string): string |
|
||||
case "retention.watchdog_days": return 14;
|
||||
case "retention.usage_samples_days": return 14;
|
||||
case "retention.events_days": return 0; // 0 = keep; pruning is opt-in
|
||||
case "retention.closed_content_days": return 0; // 0 = keep; closed-work erasure is opt-in
|
||||
case "retention.closed_metadata_days": return 0;
|
||||
case "retention.erasure_ledger": return ""; // empty = keep no ledger
|
||||
case "retention.watchdog_keep_per_job": return 50;
|
||||
case "retention.stream_days": return 30;
|
||||
case "retention.stream_max_items": return 20000;
|
||||
@@ -1007,6 +1022,24 @@ const KEY_CONSTRAINTS: Partial<Record<SettingsValidKey, (raw: string, coerced: s
|
||||
throw new Error(`Invalid value for retention.events_days: must be an integer >= 0 (0 keeps the events), got "${raw}"`);
|
||||
}
|
||||
},
|
||||
// 0 (the default) keeps closed work; a positive day count turns its erasure on (queue-erasure).
|
||||
"retention.closed_content_days": (raw, coerced) => {
|
||||
if (!/^-?\d+$/.test((raw ?? "").trim()) || typeof coerced !== "number" || !Number.isInteger(coerced) || coerced < 0) {
|
||||
throw new Error(`Invalid value for retention.closed_content_days: must be an integer >= 0 (0 keeps the content), got "${raw}"`);
|
||||
}
|
||||
},
|
||||
"retention.closed_metadata_days": (raw, coerced) => {
|
||||
if (!/^-?\d+$/.test((raw ?? "").trim()) || typeof coerced !== "number" || !Number.isInteger(coerced) || coerced < 0) {
|
||||
throw new Error(`Invalid value for retention.closed_metadata_days: must be an integer >= 0 (0 keeps the rows), got "${raw}"`);
|
||||
}
|
||||
},
|
||||
// Empty (the default) keeps no erasure ledger; otherwise an absolute file path, kept outside the backed-up data.
|
||||
"retention.erasure_ledger": (raw) => {
|
||||
const value = (raw ?? "").trim();
|
||||
if (value !== "" && !value.startsWith("/")) {
|
||||
throw new Error(`Invalid value for retention.erasure_ledger: must be empty or an absolute file path, got "${raw}"`);
|
||||
}
|
||||
},
|
||||
"retention.watchdog_keep_per_job": (raw, coerced) => {
|
||||
if (!/^-?\d+$/.test((raw ?? "").trim()) || typeof coerced !== "number" || !Number.isInteger(coerced) || coerced < 0) {
|
||||
throw new Error(`Invalid value for retention.watchdog_keep_per_job: must be an integer >= 0, got "${raw}"`);
|
||||
|
||||
@@ -118,7 +118,8 @@ export function startPeriodicSnapshotScheduler(deps: {
|
||||
/** OPR.0.4.6.FS-1 W2 — start the queue-retention maintenance sweep (arch D3): a
|
||||
* boot sweep + a daily tick that archives aged terminal `queue_transitions`
|
||||
* (never-delete) and prunes `watchdog_history`, `usage_samples`, stream items and
|
||||
* the history-only event types of `events`, all in bounded batches with
|
||||
* the history-only event types of `events`, and (opt-in) erases closed work
|
||||
* (`retention.closed_content_days`, `retention.closed_metadata_days`), all in bounded batches with
|
||||
* event-loop yields. Reads the `retention.*` settings (baked defaults via
|
||||
* RETENTION_DEFAULTS). Fire-and-forget with error logging — a sweep failure must
|
||||
* never crash the daemon. Returns the daily interval handle (cleared on
|
||||
@@ -135,6 +136,10 @@ export function startQueueRetentionScheduler(deps: {
|
||||
const v = store ? store.resolveOne(key).value : fallback;
|
||||
return typeof v === "number" ? v : fallback;
|
||||
};
|
||||
const ledger = (): string | undefined => {
|
||||
const v = store ? store.resolveOne("retention.erasure_ledger").value : undefined;
|
||||
return typeof v === "string" && v.length > 0 ? v : undefined;
|
||||
};
|
||||
const runOnce = (): void => {
|
||||
void runQueueRetentionSweep(db, {
|
||||
nowIso: new Date().toISOString(),
|
||||
@@ -146,6 +151,9 @@ export function startQueueRetentionScheduler(deps: {
|
||||
streamMaxItems: num("retention.stream_max_items", RETENTION_DEFAULTS.streamMaxItems),
|
||||
batchSize: num("retention.batch_size", RETENTION_DEFAULTS.batchSize),
|
||||
eventsRetentionDays: num("retention.events_days", RETENTION_DEFAULTS.eventsRetentionDays),
|
||||
closedContentDays: num("retention.closed_content_days", RETENTION_DEFAULTS.closedContentDays),
|
||||
closedMetadataDays: num("retention.closed_metadata_days", RETENTION_DEFAULTS.closedMetadataDays),
|
||||
...(ledger() ? { erasureLedgerPath: ledger() } : {}),
|
||||
}).catch((err: unknown) => {
|
||||
console.error(`[queue-retention] sweep error: ${err instanceof Error ? err.message : String(err)}`);
|
||||
});
|
||||
|
||||
@@ -31,6 +31,7 @@ import { listProjects, projectCatalogPaths } from "../domain/workspace/project-r
|
||||
import { readProjectCatalog } from "../domain/workspace/project-catalog.js";
|
||||
import type { WhoamiService } from "../domain/whoami-service.js";
|
||||
import type Database from "better-sqlite3";
|
||||
import { EraseRefusal, eraseQueue, type EraseAction, type EraseScope } from "../domain/queue-erasure.js";
|
||||
import { HumanModeService } from "../domain/gateway/human-operating-modes.js";
|
||||
import type { HumanQuestion } from "../domain/human-questions.js";
|
||||
import { sharedChannelFor } from "../domain/gateway/participants.js";
|
||||
@@ -227,6 +228,8 @@ export function queueRoutes(): Hono {
|
||||
// caller id-reuse (a bug, not an idempotent retry) — surface as a
|
||||
// conflict, never overwrite.
|
||||
: err.code === "qitem_id_reuse" ? 409
|
||||
// Queue erasure deleted this id's row; it isn't created again.
|
||||
: err.code === "qitem_erased" ? 409
|
||||
// OPR.0.4.6.MH3 FR-4 (C2): a cross-host source-close re-drive naming a
|
||||
// DIFFERENT closure_target than the recorded one — someone else closed
|
||||
// the source meanwhile; surfaced, never overwritten.
|
||||
@@ -654,6 +657,74 @@ export function queueRoutes(): Hono {
|
||||
}
|
||||
});
|
||||
|
||||
// POST /erase — queue erasure in one exact scope: account deletion (members, the account's recorded sessions),
|
||||
// explicit ids (a ledger's reapply after a restore), or the routine closed-work windows from this daemon's settings,
|
||||
// run now. A dry run unless apply is true. domain/queue-erasure.ts; docs/reference/queue-erasure.md.
|
||||
app.post("/erase", async (c) => {
|
||||
const body = await c.req.json<{ members?: unknown; rig?: unknown; ids?: unknown; action?: unknown; routine?: unknown; apply?: unknown; actorSession?: string }>()
|
||||
.catch(() => ({} as never));
|
||||
const identity = requireSenderIdentity(c, { verb: "queue erase", bodyClaim: body.actorSession });
|
||||
if (!identity.ok) return identity.response;
|
||||
const refuse = (error: string, message: string) => c.json({ error, message }, 400);
|
||||
const settings = c.get("settingsStore" as never) as { resolveOne(key: string): { value: unknown } } | undefined;
|
||||
const named = [body.members !== undefined, body.ids !== undefined, body.routine !== undefined].filter(Boolean).length;
|
||||
if (named === 0) return refuse("erase_scope_required", "Name one scope: members (account deletion), ids, or routine.");
|
||||
if (named > 1) return refuse("erase_scope_conflict", "Name exactly one scope: members, ids, or routine.");
|
||||
const exactList = (value: unknown): string[] | null =>
|
||||
Array.isArray(value) && value.length > 0 && value.every((entry) => typeof entry === "string" && entry.length > 0 && entry.trim() === entry)
|
||||
? value as string[] : null;
|
||||
let scope: EraseScope;
|
||||
if (body.rig !== undefined && (body.members === undefined || typeof body.rig !== "string" || body.rig.length === 0 || body.rig.trim() !== body.rig)) {
|
||||
return refuse("erase_rig_invalid", "rig is the account rig's id, given with members.");
|
||||
}
|
||||
if (body.members !== undefined) {
|
||||
const members = exactList(body.members);
|
||||
if (!members) return refuse("erase_members_invalid", "members must be a non-empty list of exact session names.");
|
||||
scope = { kind: "members", members, ...(typeof body.rig === "string" ? { rig: body.rig } : {}) };
|
||||
} else if (body.ids !== undefined) {
|
||||
// Plain ids take the request's action (delete by default); a report's qitems entries keep their own.
|
||||
const fallback = body.action ?? "delete";
|
||||
const isAction = (value: unknown): value is EraseAction => value === "delete" || value === "scrub";
|
||||
if (!isAction(fallback)) return refuse("erase_action_invalid", "action must be delete or scrub.");
|
||||
const raw = Array.isArray(body.ids) ? body.ids : [];
|
||||
const entries = raw.map((entry) => typeof entry === "string" ? { qitemId: entry, action: fallback as EraseAction }
|
||||
: entry && typeof entry === "object" ? { qitemId: (entry as { qitemId?: unknown }).qitemId, action: (entry as { action?: unknown }).action ?? fallback }
|
||||
: null);
|
||||
if (raw.length === 0 || entries.some((entry) => !entry || typeof entry.qitemId !== "string" || entry.qitemId.length === 0
|
||||
|| entry.qitemId.trim() !== entry.qitemId)) {
|
||||
return refuse("erase_ids_invalid", "ids must be a non-empty list of qitem ids, or of {qitemId, action} entries.");
|
||||
}
|
||||
if (entries.some((entry) => !isAction(entry!.action))) return refuse("erase_action_invalid", "action must be delete or scrub.");
|
||||
scope = { kind: "ids", entries: entries as Array<{ qitemId: string; action: EraseAction }> };
|
||||
} else {
|
||||
const days = (key: string): number => {
|
||||
const value = settings?.resolveOne(key).value;
|
||||
return typeof value === "number" ? value : 0;
|
||||
};
|
||||
const contentDays = days("retention.closed_content_days");
|
||||
const metadataDays = days("retention.closed_metadata_days");
|
||||
if (contentDays === 0 && metadataDays === 0) {
|
||||
return refuse("erase_routine_off", "retention.closed_content_days and retention.closed_metadata_days are both 0 (off).");
|
||||
}
|
||||
scope = { kind: "routine", contentDays, metadataDays };
|
||||
}
|
||||
try {
|
||||
const ledger = settings?.resolveOne("retention.erasure_ledger").value;
|
||||
const report = await eraseQueue(c.get("db" as never) as Database.Database, {
|
||||
scope, apply: body.apply === true, nowIso: new Date().toISOString(),
|
||||
...(typeof ledger === "string" && ledger.length > 0 ? { ledgerPath: ledger } : {}),
|
||||
});
|
||||
console.log(`[queue-erase] ${identity.session} ${scope.kind} ${report.applied ? "applied" : "dry run"}: `
|
||||
+ `${report.qitems.length} rows, ${report.mixed.length} mixed, ${report.gaps.length} gaps, complete=${report.complete}, `
|
||||
+ `clean=${report.clean}${report.error ? `, stopped at ${report.error.stage}: ${report.error.message}` : ""}`);
|
||||
// A run that stopped on an error still returns what it committed, so its ids reach the caller's ledger.
|
||||
return c.json(report, report.error ? 500 : 200);
|
||||
} catch (err) {
|
||||
if (err instanceof EraseRefusal) return refuse(err.code, err.message);
|
||||
return c.json({ error: "erase_failed", message: err instanceof Error ? err.message : String(err) }, 500);
|
||||
}
|
||||
});
|
||||
|
||||
// POST /:qitemId/claim
|
||||
app.post("/:qitemId/claim", async (c) => {
|
||||
const qitemId = c.req.param("qitemId");
|
||||
|
||||
@@ -1,5 +1,6 @@
|
||||
import { piAccountSwitchesSchema } from "../../src/db/migrations/151_pi_account_switches.js";
|
||||
import { eventsRetentionMarksSchema } from "../../src/db/migrations/153_events_retention_marks.js";
|
||||
import { queueErasureSchema } from "../../src/db/migrations/155_queue_erasure.js";
|
||||
import { activityGrantsSchema } from "../../src/db/migrations/143_activity_grants.js";
|
||||
import { environmentBindingsSchema } from "../../src/db/migrations/144_environment_bindings.js";
|
||||
import { nodePermissionsToSchema } from "../../src/db/migrations/145_node_permissions_to.js";
|
||||
@@ -162,7 +163,7 @@ import fs from "node:fs";
|
||||
|
||||
/** Seam B R6: the canonical full-fixture migration list, exported so file-backed
|
||||
* DB-reopen tests migrate IDENTICALLY to createFullTestDb. */
|
||||
export const migrationsForFullTestDb = [coreSchema, bindingsSessionsSchema, eventsSchema, snapshotsSchema, checkpointsSchema, resumeMetadataSchema, nodeSpecFieldsSchema, packagesSchema, installJournalSchema, journalSeqSchema, bootstrapSchema, discoverySchema, discoveryFkFix, agentspecRebootSchema, startupContextSchema, chatMessagesSchema, podNamespaceSchema, contextUsageSchema, externalCliAttachmentSchema, rigServicesSchema, seatHandoverObservabilitySchema, nodeCodexConfigProfileSchema, nodePermissionPolicySchema, rigPermissionPolicySchema, nodePolicyProvenanceSchema, rigPolicyProvenanceSchema, streamItemsSchema, queueItemsSchema, queueTransitionsSchema, rigPolicySchema, rigArchiveSchema, resumeProvenanceSchema, resumeVerificationSchema, seatIdentityVerdictsSchema, selfHostIdentitySchema, occupantTenuresSchema, daemonLifecycleSchema, watchdogJobsSchema, occupantGenerationStampsSchema, projectionManifestSchema, watchdogTargetGenerationSchema, appliedLaunchObservationsSchema, appliedLaunchObservationInvalidationsSchema, threadSeatMapSchema, queueTransitionWakesSchema, nodeSessionSourceSchema, scopedOperatingPostureSchema, humanNotificationIntentSchema, reviewReadIndexesSchema, inventoryEventIndexesSchema, rigClaudeManagedBlockFileSchema, nodePermissionSelectionsSchema, humanReplyToSchema, humanQuestionsSchema, nodeEffortSchema, rigNonInterruptiveSchema, rigInstallRootSchema, threadPartMapSchema, queueTransitionsQitemIdOrderSchema, queueItemsRecoveryTagsSchema, resumeRotationSchema, sessionStopIntentSchema, nodeEffortChoiceSchema, modePostureCoexistSchema, seatEnvSchema, appliedLaunchDaemonOptOutSchema, sessionCommunityHarnessSchema, threadSeatMapChannelKeySchema, queueItemWakeInputBoxSchema, providerAccountReadingsSchema, humanQuietWindowsSchema, seatIdentityPanePidSchema, seatIdentityProcessEvidenceSchema, rigStartupLayersSchema, humanOperatingModesSchema, queueDelegatedDecisionSchema, nodeAdvisorSchema, operationCorrelationSchema, terminalPolicySchema, seatRuntimeSwitchesSchema, claudeTokenReadingsSchema, nodeLaunchRecipeSchema, nodeClaudeTokenRefSchema, nodeClaudePermissionsSchema, nodeCodexPermissionsSchema, escalationTargetRefsSchema, sessionAgentRunnerSchema, nodeAccountHomeSchema, nodeClaudeMcpSchema, activityGrantsSchema, environmentBindingsSchema, nodePermissionsToSchema, nodeStartupGuidanceFileSchema, managedBlockWritesSchema, streamIntakeCursorSchema, nodeLaunchWrapperSchema, piAccountSwitchesSchema, eventsRetentionMarksSchema];
|
||||
export const migrationsForFullTestDb = [coreSchema, bindingsSessionsSchema, eventsSchema, snapshotsSchema, checkpointsSchema, resumeMetadataSchema, nodeSpecFieldsSchema, packagesSchema, installJournalSchema, journalSeqSchema, bootstrapSchema, discoverySchema, discoveryFkFix, agentspecRebootSchema, startupContextSchema, chatMessagesSchema, podNamespaceSchema, contextUsageSchema, externalCliAttachmentSchema, rigServicesSchema, seatHandoverObservabilitySchema, nodeCodexConfigProfileSchema, nodePermissionPolicySchema, rigPermissionPolicySchema, nodePolicyProvenanceSchema, rigPolicyProvenanceSchema, streamItemsSchema, queueItemsSchema, queueTransitionsSchema, rigPolicySchema, rigArchiveSchema, resumeProvenanceSchema, resumeVerificationSchema, seatIdentityVerdictsSchema, selfHostIdentitySchema, occupantTenuresSchema, daemonLifecycleSchema, watchdogJobsSchema, occupantGenerationStampsSchema, projectionManifestSchema, watchdogTargetGenerationSchema, appliedLaunchObservationsSchema, appliedLaunchObservationInvalidationsSchema, threadSeatMapSchema, queueTransitionWakesSchema, nodeSessionSourceSchema, scopedOperatingPostureSchema, humanNotificationIntentSchema, reviewReadIndexesSchema, inventoryEventIndexesSchema, rigClaudeManagedBlockFileSchema, nodePermissionSelectionsSchema, humanReplyToSchema, humanQuestionsSchema, nodeEffortSchema, rigNonInterruptiveSchema, rigInstallRootSchema, threadPartMapSchema, queueTransitionsQitemIdOrderSchema, queueItemsRecoveryTagsSchema, resumeRotationSchema, sessionStopIntentSchema, nodeEffortChoiceSchema, modePostureCoexistSchema, seatEnvSchema, appliedLaunchDaemonOptOutSchema, sessionCommunityHarnessSchema, threadSeatMapChannelKeySchema, queueItemWakeInputBoxSchema, providerAccountReadingsSchema, humanQuietWindowsSchema, seatIdentityPanePidSchema, seatIdentityProcessEvidenceSchema, rigStartupLayersSchema, humanOperatingModesSchema, queueDelegatedDecisionSchema, nodeAdvisorSchema, operationCorrelationSchema, terminalPolicySchema, seatRuntimeSwitchesSchema, claudeTokenReadingsSchema, nodeLaunchRecipeSchema, nodeClaudeTokenRefSchema, nodeClaudePermissionsSchema, nodeCodexPermissionsSchema, escalationTargetRefsSchema, sessionAgentRunnerSchema, nodeAccountHomeSchema, nodeClaudeMcpSchema, activityGrantsSchema, environmentBindingsSchema, nodePermissionsToSchema, nodeStartupGuidanceFileSchema, managedBlockWritesSchema, streamIntakeCursorSchema, nodeLaunchWrapperSchema, piAccountSwitchesSchema, eventsRetentionMarksSchema, queueErasureSchema];
|
||||
|
||||
/**
|
||||
* P24 — the DECLARED exclusions for {@link migrationsForFullTestDb}. That list is deliberately a
|
||||
@@ -179,6 +180,7 @@ export const migrationsForFullTestDb = [coreSchema, bindingsSessionsSchema, even
|
||||
* its exclusion); the guard's redundant-exclusion check then keeps the two in sync.
|
||||
*/
|
||||
export const migrationsForFullTestDbExclusions: Record<string, string> = {
|
||||
"156_queue_erasure_link_indexes.sql": "indexes outbox_entries, inbox_entries and the stream tables (all excluded from the core edge) for queue erasure's lookups; erasure tests migrate the canonical full schema.",
|
||||
"082_archive_identity_provenance.sql": "archive extension; 054 is excluded from this core fixture. Archive identity regressions use ALL_MIGRATIONS.",
|
||||
"026_inbox_entries.sql": "inbox subsystem table — not on the shared core edge (inbox suites migrate it inline).",
|
||||
"027_outbox_entries.sql": "outbox subsystem table — not on the shared core edge (outbox suites migrate it inline).",
|
||||
|
||||
@@ -0,0 +1,670 @@
|
||||
import { afterEach, beforeEach, describe, expect, it } from "vitest";
|
||||
import { Hono } from "hono";
|
||||
import Database from "better-sqlite3";
|
||||
import { createDb } from "../src/db/connection.js";
|
||||
import { migrate } from "../src/db/migrate.js";
|
||||
import { ALL_MIGRATIONS } from "../src/db/all-migrations.js";
|
||||
import { EventBus } from "../src/domain/event-bus.js";
|
||||
import { OutboxHandler } from "../src/domain/outbox-handler.js";
|
||||
import { InboxHandler } from "../src/domain/inbox-handler.js";
|
||||
import { QueueRepository, QueueRepositoryError } from "../src/domain/queue-repository.js";
|
||||
import { QueueTransitionLog } from "../src/domain/queue-transition-log.js";
|
||||
import { EraseRefusal, eraseQueue } from "../src/domain/queue-erasure.js";
|
||||
import { runQueueRetentionSweep } from "../src/domain/queue-retention.js";
|
||||
import { queueRoutes } from "../src/routes/queue.js";
|
||||
import { recordReaderAction } from "../src/domain/stream-reader-actions.js";
|
||||
import { mkdtempSync, readFileSync } from "node:fs";
|
||||
import { tmpdir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
|
||||
const NOW = "2026-11-09T12:00:00.000Z";
|
||||
const ago = (days: number) => new Date(Date.parse(NOW) - days * 86_400_000).toISOString();
|
||||
const A = "a-one@n-acct";
|
||||
const B = "b-two@n-acct";
|
||||
const OUT = "x-other@n-else";
|
||||
|
||||
let db: Database.Database;
|
||||
let repo: QueueRepository;
|
||||
let log: QueueTransitionLog;
|
||||
function open(handle: Database.Database) {
|
||||
db = handle;
|
||||
repo = new QueueRepository(db, new EventBus(db));
|
||||
repo.attachOutbox(new OutboxHandler(db));
|
||||
log = new QueueTransitionLog(db);
|
||||
}
|
||||
beforeEach(() => {
|
||||
const handle = createDb();
|
||||
migrate(handle, ALL_MIGRATIONS);
|
||||
open(handle);
|
||||
});
|
||||
afterEach(() => db.close());
|
||||
|
||||
/** A row whose every content field and linked copy carries `secret-<id>`, with its history at chosen ages. */
|
||||
async function row(id: string, source: string, destination: string, history: Array<{ state: string; daysAgo: number }> = []) {
|
||||
const secret = `secret-${id}`;
|
||||
await repo.create({ qitemId: id, sourceSession: source, destinationSession: destination, body: `${secret} body`, summary: `${secret} summary`,
|
||||
tags: [`${secret}-tag`] });
|
||||
db.prepare("UPDATE queue_transitions SET ts = ?, transition_note = ? WHERE qitem_id = ?").run(ago(400), `${secret} created`, id);
|
||||
for (const step of history) {
|
||||
const { transitionId } = log.append({ qitemId: id, state: step.state, actorSession: destination, transitionNote: `${secret} note` });
|
||||
db.prepare("UPDATE queue_transitions SET ts = ? WHERE transition_id = ?").run(ago(step.daysAgo), transitionId);
|
||||
}
|
||||
const state = history.at(-1)?.state ?? "pending";
|
||||
db.prepare("UPDATE queue_items SET state = ?, resolution = ?, ts_updated = ? WHERE qitem_id = ?")
|
||||
.run(state, `${secret} resolution`, ago(history.at(-1)?.daysAgo ?? 400), id);
|
||||
// The create staged the row's wake intent (its body carries the summary); make sure one exists either way.
|
||||
db.prepare(`INSERT OR IGNORE INTO outbox_entries (outbox_id, sender_session, destination_session, body, ts_dispatched, audit_pointer)
|
||||
VALUES (?, ?, ?, ?, ?, ?)`).run(`wake-intent-${id}`, source, destination, `${secret} wake`, NOW, id);
|
||||
db.prepare(`INSERT INTO inbox_entries (inbox_id, destination_session, sender_session, body, ts_dropped, state, absorbed_qitem_id)
|
||||
VALUES (?, ?, ?, ?, ?, 'absorbed', ?)`).run(`inbox-${id}`, destination, source, `${secret} inbox`, NOW, id);
|
||||
db.prepare(`INSERT INTO mission_control_actions (action_id, action_verb, qitem_id, actor_session, acted_at, reason)
|
||||
VALUES (?, 'approve', ?, ?, ?, ?)`).run(`mc-${id}`, id, destination, NOW, `${secret} reason`);
|
||||
}
|
||||
|
||||
/** Every value in every table, so a check can say no copy of a secret is left anywhere. */
|
||||
function everything(): string {
|
||||
const tables = (db.prepare("SELECT name FROM sqlite_master WHERE type = 'table'").all() as Array<{ name: string }>).map((t) => t.name);
|
||||
return tables.map((table) => JSON.stringify(db.prepare(`SELECT * FROM "${table}"`).all())).join("\n");
|
||||
}
|
||||
const exists = (id: string) => db.prepare("SELECT 1 FROM queue_items WHERE qitem_id = ?").get(id) !== undefined;
|
||||
/** Event payloads as stored (everything() escapes their quotes, so match payload text here). */
|
||||
const payloads = () => (db.prepare("SELECT payload FROM events").all() as Array<{ payload: string }>).map((e) => e.payload).join("\n");
|
||||
const marker = (id: string) => db.prepare("SELECT * FROM queue_erased_ids WHERE qitem_id = ?").get(id) as Record<string, unknown> | undefined;
|
||||
|
||||
describe("account deletion (members scope)", () => {
|
||||
beforeEach(async () => {
|
||||
await row("q-done", A, B, [{ state: "done", daysAgo: 2 }]);
|
||||
await row("q-open", B, A); // open work: account deletion takes it too
|
||||
await row("q-mixed", A, OUT, [{ state: "done", daysAgo: 2 }]);
|
||||
await row("q-else", OUT, OUT);
|
||||
db.prepare(`INSERT INTO outbox_entries (outbox_id, sender_session, destination_session, body, ts_dispatched)
|
||||
VALUES ('o-loose', ?, ?, 'secret-loose', ?), ('o-cross', ?, ?, 'secret-cross', ?)`).run(A, B, NOW, A, OUT, NOW);
|
||||
});
|
||||
|
||||
it("a dry run reports exactly what an apply erases, and changes nothing", async () => {
|
||||
const before = everything();
|
||||
const dry = await eraseQueue(db, { scope: { kind: "members", members: [A, B] }, apply: false, nowIso: NOW });
|
||||
expect(everything()).toBe(before);
|
||||
expect(dry.qitems.map((q) => [q.qitemId, q.action, q.state])).toEqual([["q-done", "delete", "done"], ["q-open", "delete", "pending"]]);
|
||||
expect(dry.mixed).toEqual([{ qitemId: "q-mixed", memberEndpoint: "source" }]);
|
||||
expect(dry.linked.outbox).toBeGreaterThanOrEqual(2);
|
||||
expect(dry.linked.events).toBeGreaterThanOrEqual(2);
|
||||
expect(dry.unlinked).toMatchObject({ outbox: 1, mixedEntries: 1 });
|
||||
const applied = await eraseQueue(db, { scope: { kind: "members", members: [A, B] }, apply: true, nowIso: NOW });
|
||||
expect({ ...applied, applied: false }).toEqual(dry);
|
||||
});
|
||||
|
||||
it("deletes both members' rows with every copy, keeps only their ids, and leaves mixed and other rows whole", async () => {
|
||||
const report = await eraseQueue(db, { scope: { kind: "members", members: [A, B] }, apply: true, nowIso: NOW });
|
||||
expect(report.complete).toBe(true);
|
||||
const all = everything();
|
||||
for (const id of ["q-done", "q-open"]) {
|
||||
expect(exists(id)).toBe(false);
|
||||
expect(all).not.toContain(`secret-${id}`);
|
||||
expect(marker(id)).toEqual({ qitem_id: id, erased_at: NOW, reason: "account-deletion" });
|
||||
}
|
||||
expect(all).not.toContain("secret-loose");
|
||||
// The mixed row and its copies stay, and so does the cross-account entry; nothing else was touched.
|
||||
expect(all).toContain("secret-q-mixed body");
|
||||
expect(all).toContain("secret-q-mixed summary");
|
||||
expect(all).toContain("secret-cross");
|
||||
expect(all).toContain("secret-q-else body");
|
||||
expect(marker("q-mixed")).toBeUndefined();
|
||||
});
|
||||
|
||||
it("a create reusing an erased id is refused, so erased work never comes back as new work", async () => {
|
||||
await eraseQueue(db, { scope: { kind: "members", members: [A, B] }, apply: true, nowIso: NOW });
|
||||
await expect(repo.create({ qitemId: "q-done", sourceSession: A, destinationSession: B, body: "secret-q-done body" }))
|
||||
.rejects.toBeInstanceOf(QueueRepositoryError);
|
||||
await expect(repo.create({ qitemId: "q-done", sourceSession: A, destinationSession: B, body: "secret-q-done body" }))
|
||||
.rejects.toMatchObject({ code: "qitem_erased" });
|
||||
expect(exists("q-done")).toBe(false);
|
||||
});
|
||||
|
||||
it("a run that stops partway says so, and a rerun finishes the rest", async () => {
|
||||
const first = await eraseQueue(db, { scope: { kind: "members", members: [A, B] }, apply: true, nowIso: NOW, batchSize: 1, maxBatches: 1 });
|
||||
expect(first.complete).toBe(false);
|
||||
expect(first.unlinked).toBeUndefined();
|
||||
expect(first.qitems.map((q) => q.qitemId)).toEqual(["q-done"]);
|
||||
expect(exists("q-open")).toBe(true);
|
||||
const rest = await eraseQueue(db, { scope: { kind: "members", members: [A, B] }, apply: true, nowIso: NOW, batchSize: 1 });
|
||||
expect(rest.complete).toBe(true);
|
||||
expect(rest.qitems.map((q) => q.qitemId)).toEqual(["q-open"]);
|
||||
expect(everything()).not.toContain("secret-loose");
|
||||
});
|
||||
|
||||
it("a row a workflow still references is reported and left", async () => {
|
||||
db.pragma("foreign_keys = OFF");
|
||||
db.prepare(`INSERT INTO workflow_frontier_bindings (instance_id, packet_id, step_id, created_at) VALUES ('wf-1', 'q-done', 's1', ?)`).run(NOW);
|
||||
db.pragma("foreign_keys = ON");
|
||||
const report = await eraseQueue(db, { scope: { kind: "members", members: [A, B] }, apply: true, nowIso: NOW });
|
||||
expect(report.gaps).toEqual([{ code: "workflow-reference", qitemId: "q-done" }]);
|
||||
expect(report.qitems.map((q) => q.qitemId)).toEqual(["q-open"]);
|
||||
expect(everything()).toContain("secret-q-done body");
|
||||
});
|
||||
});
|
||||
|
||||
describe("routine closed-work retention", () => {
|
||||
beforeEach(async () => {
|
||||
// Closed 40 days ago; a note yesterday keeps the closed state, so it doesn't restart the clock.
|
||||
await row("r-old-done", A, B, [{ state: "done", daysAgo: 40 }, { state: "done", daysAgo: 1 }]);
|
||||
await row("r-recent", A, B, [{ state: "done", daysAgo: 10 }]);
|
||||
await row("r-ancient-canceled", A, B, [{ state: "canceled", daysAgo: 100 }]);
|
||||
await row("r-failed", A, B, [{ state: "failed", daysAgo: 100 }]);
|
||||
await row("r-open", A, B, [{ state: "in-progress", daysAgo: 200 }]);
|
||||
// Closed long ago, reopened, closed again five days ago: it counts from the latest close.
|
||||
await row("r-reclosed", A, B, [{ state: "done", daysAgo: 50 }, { state: "pending", daysAgo: 20 }, { state: "denied", daysAgo: 5 }]);
|
||||
});
|
||||
|
||||
it("removes content past the content window and the row past the metadata window, from when each row closed", async () => {
|
||||
const report = await eraseQueue(db, { scope: { kind: "routine", contentDays: 30, metadataDays: 90 }, apply: true, nowIso: NOW });
|
||||
expect(report.qitems.map((q) => [q.qitemId, q.action])).toEqual([["r-ancient-canceled", "delete"], ["r-old-done", "scrub"]]);
|
||||
const all = everything();
|
||||
expect(all).not.toContain("secret-r-old-done");
|
||||
expect(all).not.toContain("secret-r-ancient-canceled");
|
||||
for (const id of ["r-recent", "r-failed", "r-open", "r-reclosed"]) expect(all).toContain(`secret-${id} body`);
|
||||
expect(marker("r-ancient-canceled")).toMatchObject({ reason: "retention-metadata" });
|
||||
// The scrubbed row stays, with its identity, endpoints and state.
|
||||
const kept = db.prepare("SELECT source_session, destination_session, state, body, content_erased_at FROM queue_items WHERE qitem_id = 'r-old-done'").get();
|
||||
expect(kept).toEqual({ source_session: A, destination_session: B, state: "done", body: "", content_erased_at: NOW });
|
||||
const event = db.prepare("SELECT payload FROM events WHERE json_extract(payload, '$.qitemId') = 'r-old-done'").get() as { payload: string };
|
||||
expect(JSON.parse(event.payload)).toMatchObject({ type: "queue.created", qitemId: "r-old-done", contentErased: true });
|
||||
// A second sweep finds nothing more to do.
|
||||
const again = await eraseQueue(db, { scope: { kind: "routine", contentDays: 30, metadataDays: 90 }, apply: true, nowIso: NOW });
|
||||
expect(again.qitems).toEqual([]);
|
||||
});
|
||||
|
||||
it("a retried create for a scrubbed row returns that row and makes no new work", async () => {
|
||||
await eraseQueue(db, { scope: { kind: "routine", contentDays: 30, metadataDays: 0 }, apply: true, nowIso: NOW });
|
||||
const transitions = () => (db.prepare("SELECT COUNT(*) AS n FROM queue_transitions WHERE qitem_id = 'r-old-done'").get() as { n: number }).n;
|
||||
const before = transitions();
|
||||
const retried = await repo.create({ qitemId: "r-old-done", sourceSession: A, destinationSession: B, body: "secret-r-old-done body" });
|
||||
expect(retried).toMatchObject({ qitemId: "r-old-done", state: "done", body: "", contentErasedAt: NOW });
|
||||
expect(transitions()).toBe(before);
|
||||
expect(everything()).not.toContain("secret-r-old-done");
|
||||
});
|
||||
|
||||
it("runs from the retention sweep only when a window is set", async () => {
|
||||
const off = await runQueueRetentionSweep(db, { nowIso: NOW });
|
||||
expect([off.closedScrubbed, off.closedDeleted]).toEqual([0, 0]);
|
||||
expect(everything()).toContain("secret-r-old-done body");
|
||||
const on = await runQueueRetentionSweep(db, { nowIso: NOW, closedContentDays: 30, closedMetadataDays: 90 });
|
||||
expect([on.closedScrubbed, on.closedDeleted, on.closedKept]).toEqual([1, 1, 0]);
|
||||
});
|
||||
});
|
||||
|
||||
describe("ids scope: reapplying a ledger after a restore", () => {
|
||||
it("deletes restored rows again and writes back every id marker, including ids already gone", async () => {
|
||||
await row("i-one", A, B, [{ state: "done", daysAgo: 3 }]);
|
||||
await eraseQueue(db, { scope: { kind: "ids", entries: [{ qitemId: "i-one", action: "delete" }] }, apply: true, nowIso: NOW });
|
||||
// A restore from a backup taken before the erase brings the row back and loses the marker.
|
||||
db.prepare("DELETE FROM queue_erased_ids").run();
|
||||
await row("i-one", A, B, [{ state: "done", daysAgo: 3 }]);
|
||||
const reapplied = await eraseQueue(db, { scope: { kind: "ids", entries: [{ qitemId: "i-one", action: "delete" }, { qitemId: "i-gone", action: "delete" }] }, apply: true, nowIso: NOW });
|
||||
expect(reapplied.qitems.map((q) => q.qitemId)).toEqual(["i-one"]);
|
||||
expect(reapplied.alreadyAbsent).toEqual(["i-gone"]);
|
||||
expect(exists("i-one")).toBe(false);
|
||||
expect(everything()).not.toContain("secret-i-one");
|
||||
expect(marker("i-one")).toMatchObject({ reason: "erase-by-id" });
|
||||
expect(marker("i-gone")).toMatchObject({ reason: "erase-by-id" });
|
||||
});
|
||||
|
||||
it("a scrub of an unknown id is reported, not invented", async () => {
|
||||
const report = await eraseQueue(db, { scope: { kind: "ids", entries: [{ qitemId: "nope", action: "scrub" }] }, apply: true, nowIso: NOW });
|
||||
expect(report.gaps).toEqual([{ code: "not-found", qitemId: "nope" }]);
|
||||
expect(marker("nope")).toBeUndefined();
|
||||
});
|
||||
});
|
||||
|
||||
describe("POST /api/queue/erase", () => {
|
||||
function app(settings: Record<string, unknown> = {}) {
|
||||
const bus = new EventBus(db);
|
||||
const hono = new Hono();
|
||||
hono.use("*", async (c, next) => {
|
||||
c.set("db" as never, db as never);
|
||||
c.set("eventBus" as never, bus as never);
|
||||
c.set("queueRepo" as never, repo as never);
|
||||
c.set("inboxHandler" as never, new InboxHandler(db, bus, repo) as never);
|
||||
c.set("outboxHandler" as never, new OutboxHandler(db) as never);
|
||||
c.set("settingsStore" as never, { resolveOne: (key: string) => ({ value: settings[key] ?? 0 }) } as never);
|
||||
await next();
|
||||
});
|
||||
hono.route("/api/queue", queueRoutes());
|
||||
return hono;
|
||||
}
|
||||
const post = (hono: Hono, body: unknown) => hono.request("/api/queue/erase", {
|
||||
method: "POST", headers: { "content-type": "application/json", "X-OpenRig-Session": "ops-admin@n-ops" }, body: JSON.stringify(body),
|
||||
});
|
||||
|
||||
it("refuses a missing, doubled or malformed scope with a fixed code", async () => {
|
||||
const hono = app();
|
||||
const codes = async (body: unknown) => {
|
||||
const res = await post(hono, body);
|
||||
return [res.status, ((await res.json()) as { error: string }).error];
|
||||
};
|
||||
expect(await codes({})).toEqual([400, "erase_scope_required"]);
|
||||
expect(await codes({ members: [A], ids: ["x"] })).toEqual([400, "erase_scope_conflict"]);
|
||||
expect(await codes({ members: [] })).toEqual([400, "erase_members_invalid"]);
|
||||
expect(await codes({ members: [" a-one@n-acct"] })).toEqual([400, "erase_members_invalid"]);
|
||||
expect(await codes({ ids: ["x"], action: "shred" })).toEqual([400, "erase_action_invalid"]);
|
||||
expect(await codes({ routine: true })).toEqual([400, "erase_routine_off"]);
|
||||
expect(await codes({ ids: ["x"], rig: "rig-id" })).toEqual([400, "erase_rig_invalid"]);
|
||||
expect(await codes({ ids: [{ qitemId: "x", action: "shred" }] })).toEqual([400, "erase_action_invalid"]);
|
||||
});
|
||||
|
||||
it("runs a dry run by default and erases only with apply", async () => {
|
||||
await row("p-one", A, B, [{ state: "done", daysAgo: 2 }]);
|
||||
const hono = app();
|
||||
const dry = await post(hono, { members: [A, B] });
|
||||
expect(dry.status).toBe(200);
|
||||
expect(await dry.json()).toMatchObject({ ok: true, applied: false, qitems: [{ qitemId: "p-one", action: "delete" }] });
|
||||
expect(exists("p-one")).toBe(true);
|
||||
const applied = await post(hono, { members: [A, B], apply: true });
|
||||
expect(await applied.json()).toMatchObject({ applied: true, complete: true });
|
||||
expect(exists("p-one")).toBe(false);
|
||||
});
|
||||
|
||||
it("a run that stops on an error answers 500 with what it committed", async () => {
|
||||
await row("p-a", A, B, [{ state: "done", daysAgo: 2 }]);
|
||||
await row("p-b", A, B, [{ state: "done", daysAgo: 2 }]);
|
||||
db.exec("CREATE TEMP TRIGGER fail_pb BEFORE DELETE ON queue_items WHEN old.qitem_id = 'p-b' BEGIN SELECT RAISE(ABORT, 'disk full (test)'); END;");
|
||||
const res = await post(app(), { members: [A, B], apply: true });
|
||||
expect(res.status).toBe(500);
|
||||
expect(await res.json()).toMatchObject({ complete: false, error: { stage: "rows" } });
|
||||
});
|
||||
|
||||
it("routine reads this daemon's own windows", async () => {
|
||||
await row("p-old", A, B, [{ state: "done", daysAgo: 40 }]);
|
||||
const res = await post(app({ "retention.closed_content_days": 30 }), { routine: true, apply: true });
|
||||
expect(await res.json()).toMatchObject({ routine: { contentDays: 30, metadataDays: 0 }, qitems: [{ qitemId: "p-old", action: "scrub" }] });
|
||||
});
|
||||
});
|
||||
|
||||
describe("account deletion with the account's rig", () => {
|
||||
const RIG = "rig-acct-id";
|
||||
beforeEach(async () => {
|
||||
db.prepare("INSERT INTO rigs (id, name) VALUES (?, 'n-acct'), ('rig-else-id', 'n-else')").run(RIG);
|
||||
db.prepare("INSERT INTO nodes (id, rig_id, logical_id) VALUES ('node-a', ?, 'a.one'), ('node-x', 'rig-else-id', 'x.other')").run(RIG);
|
||||
const bus = new EventBus(db);
|
||||
bus.emit({ type: "rig.imported", rigId: RIG, specName: "secret-rig-spec", specVersion: "1" } as never);
|
||||
bus.emit({ type: "node.launched", rigId: RIG, nodeId: "node-a", logicalId: "a.one", sessionName: A } as never);
|
||||
bus.emit({ type: "node.launched", rigId: "rig-else-id", nodeId: "node-x", logicalId: "x.other", sessionName: OUT } as never);
|
||||
db.prepare("INSERT INTO snapshots (id, rig_id, kind, data) VALUES ('snap-a', ?, 'periodic', '{\"seat\":\"secret-snapshot\"}')").run(RIG);
|
||||
db.prepare("INSERT INTO seat_identity_verdicts (node_id, session_name, verdict, observed_at) VALUES ('node-a', ?, 'verified', ?)").run(A, NOW);
|
||||
db.prepare(`INSERT INTO watchdog_jobs (job_id, policy, spec_yaml, target_session, interval_seconds, registered_by_session, registered_at)
|
||||
VALUES ('wd-a', 'periodic-reminder', 'secret-watchdog', ?, 60, ?, ?), ('wd-cross', 'periodic-reminder', 'x', ?, 60, ?, ?)`)
|
||||
.run(A, B, NOW, OUT, A, NOW);
|
||||
db.prepare("INSERT INTO watchdog_history (job_id, evaluated_at, outcome) VALUES ('wd-a', ?, 'fired')").run(NOW);
|
||||
await row("g-done", A, B, [{ state: "done", daysAgo: 2 }]);
|
||||
db.prepare(`INSERT INTO inbox_entries (inbox_id, destination_session, sender_session, body, ts_dropped)
|
||||
VALUES ('i-loose', ?, ?, 'secret-loose-inbox', ?)`).run(B, A, NOW);
|
||||
});
|
||||
|
||||
it("erases the rig's own records, leaves another rig's, and counts a member's job aimed outside", async () => {
|
||||
const report = await eraseQueue(db, { scope: { kind: "members", members: [A, B], rig: RIG }, apply: true, nowIso: NOW });
|
||||
expect(report.error).toBeUndefined();
|
||||
expect(report.rig).toMatchObject({ id: RIG, found: true, erased: { snapshots: 1, seat_identity_verdicts: 1, watchdog_jobs: 1, watchdog_history: 1 },
|
||||
left: { watchdog_jobs: 1 } });
|
||||
expect(report.rig!.erased.events).toBeGreaterThanOrEqual(2);
|
||||
expect(report.clean).toBe(false); // the member's job aimed outside the account is left, so not clean
|
||||
const all = everything();
|
||||
for (const secret of ["secret-rig-spec", "secret-snapshot", "secret-watchdog", "secret-loose-inbox", "secret-g-done"]) {
|
||||
expect(all).not.toContain(secret);
|
||||
}
|
||||
expect(payloads()).not.toContain(`"sessionName":"${A}"`);
|
||||
expect(payloads()).toContain(`"sessionName":"${OUT}"`);
|
||||
expect(db.prepare("SELECT COUNT(*) AS n FROM events WHERE rig_id = ?").get(RIG)).toEqual({ n: 0 });
|
||||
});
|
||||
|
||||
it("refuses a rig the members don't belong to, and erases nothing", async () => {
|
||||
const before = everything();
|
||||
await expect(eraseQueue(db, { scope: { kind: "members", members: [A, B], rig: "rig-else-id" }, apply: true, nowIso: NOW }))
|
||||
.rejects.toThrowError(EraseRefusal);
|
||||
await expect(eraseQueue(db, { scope: { kind: "members", members: [A, B], rig: "rig-else-id" }, apply: true, nowIso: NOW }))
|
||||
.rejects.toMatchObject({ code: "erase_rig_mismatch" });
|
||||
expect(everything()).toBe(before);
|
||||
});
|
||||
|
||||
it("after the rig is removed, a rerun erases what its removal wrote, by the rig's id", async () => {
|
||||
await eraseQueue(db, { scope: { kind: "members", members: [A, B], rig: RIG }, apply: true, nowIso: NOW });
|
||||
db.prepare("DELETE FROM watchdog_jobs WHERE job_id = 'wd-cross'").run();
|
||||
db.prepare("DELETE FROM nodes WHERE rig_id = ?").run(RIG);
|
||||
db.prepare("DELETE FROM rigs WHERE id = ?").run(RIG);
|
||||
new EventBus(db).emit({ type: "session.detached", rigId: RIG, nodeId: "node-a", sessionName: A } as never);
|
||||
const rerun = await eraseQueue(db, { scope: { kind: "members", members: [A, B], rig: RIG }, apply: true, nowIso: NOW });
|
||||
expect(rerun.rig).toMatchObject({ found: false, erased: { events: 1 } });
|
||||
expect(rerun.clean).toBe(true);
|
||||
expect(payloads()).not.toContain(`"sessionName":"${A}"`);
|
||||
expect(payloads()).toContain(`"sessionName":"${OUT}"`);
|
||||
// The record is now a completion receipt without the rig's name, so an identical rerun (after a lost response,
|
||||
// say) still verifies and reports clean.
|
||||
expect(rerun.rig).toMatchObject({ completedAt: NOW });
|
||||
expect(db.prepare("SELECT rig_name, completed_at FROM queue_erased_rigs WHERE rig_id = ?").get(RIG)).toEqual({ rig_name: "", completed_at: NOW });
|
||||
const again = await eraseQueue(db, { scope: { kind: "members", members: [A, B], rig: RIG }, apply: true, nowIso: NOW });
|
||||
expect(again.rig).toMatchObject({ found: false, verified: true, completedAt: NOW, unknown: [] });
|
||||
expect(again.clean).toBe(true);
|
||||
});
|
||||
|
||||
it("K2: a rig pass that stops partway reports the records it already erased", async () => {
|
||||
db.exec("CREATE TEMP TRIGGER fail_wd BEFORE DELETE ON watchdog_jobs BEGIN SELECT RAISE(ABORT, 'disk I/O error (test)'); END;");
|
||||
const report = await eraseQueue(db, { scope: { kind: "members", members: [A, B], rig: RIG }, apply: true, nowIso: NOW });
|
||||
expect(report).toMatchObject({ complete: false, error: { stage: "rig" }, rig: { erased: { snapshots: 1, watchdog_history: 1 } } });
|
||||
expect(report.rig!.erased.events).toBeGreaterThanOrEqual(2);
|
||||
expect(db.prepare("SELECT COUNT(*) AS n FROM snapshots WHERE rig_id = ?").get(RIG)).toEqual({ n: 0 });
|
||||
});
|
||||
|
||||
it("restore control: after restoring a backup taken before the erase, the documented replay leaves no account content", async () => {
|
||||
await row("g-scrub", A, OUT, [{ state: "done", daysAgo: 40 }]); // another account's row, scrubbed by routine retention
|
||||
const backup = db.serialize();
|
||||
await eraseQueue(db, { scope: { kind: "members", members: [A, B], rig: RIG }, apply: true, nowIso: NOW });
|
||||
const routine = await eraseQueue(db, { scope: { kind: "routine", contentDays: 30, metadataDays: 0 }, apply: true, nowIso: NOW });
|
||||
const ledger = routine.qitems; // what a ledger keeps: ids with their actions
|
||||
// The restore: the database comes back as it was before both runs.
|
||||
db.close();
|
||||
open(new Database(backup));
|
||||
expect(everything()).toContain("secret-g-done body");
|
||||
// The documented replay: the retained membership and rig, then the ledger with its actions, then routine.
|
||||
const replay = await eraseQueue(db, { scope: { kind: "members", members: [A, B], rig: RIG }, apply: true, nowIso: NOW });
|
||||
await eraseQueue(db, { scope: { kind: "ids", entries: ledger }, apply: true, nowIso: NOW });
|
||||
await eraseQueue(db, { scope: { kind: "routine", contentDays: 30, metadataDays: 0 }, apply: true, nowIso: NOW });
|
||||
expect(replay.complete).toBe(true);
|
||||
const all = everything();
|
||||
for (const secret of ["secret-g-done", "secret-loose-inbox", "secret-rig-spec", "secret-snapshot", "secret-watchdog", "secret-g-scrub"]) {
|
||||
expect(all).not.toContain(secret);
|
||||
}
|
||||
expect(marker("g-done")).toMatchObject({ reason: "account-deletion" });
|
||||
// The scrubbed row was scrubbed again, not promoted to a deletion.
|
||||
expect(exists("g-scrub")).toBe(true);
|
||||
expect(marker("g-scrub")).toBeUndefined();
|
||||
});
|
||||
});
|
||||
|
||||
describe("clean", () => {
|
||||
it("is true only when the run finished and left nothing", async () => {
|
||||
await row("c-one", A, B, [{ state: "done", daysAgo: 2 }]);
|
||||
expect((await eraseQueue(db, { scope: { kind: "members", members: [A, B] }, apply: false, nowIso: NOW })).clean).toBe(true);
|
||||
await row("c-mixed", A, OUT, [{ state: "done", daysAgo: 2 }]);
|
||||
expect((await eraseQueue(db, { scope: { kind: "members", members: [A, B] }, apply: false, nowIso: NOW })).clean).toBe(false);
|
||||
const partial = await eraseQueue(db, { scope: { kind: "ids", entries: [{ qitemId: "c-one", action: "delete" }] }, apply: false, nowIso: NOW, batchSize: 1, maxBatches: 0 });
|
||||
expect([partial.complete, partial.clean]).toEqual([false, false]);
|
||||
});
|
||||
});
|
||||
|
||||
describe("round 1 controls", () => {
|
||||
it("J1: the events walk finds the same events whatever its window", async () => {
|
||||
await row("w-one", A, B, [{ state: "done", daysAgo: 40 }]);
|
||||
await row("w-two", A, B, [{ state: "done", daysAgo: 40 }]);
|
||||
const wide = await eraseQueue(db, { scope: { kind: "routine", contentDays: 30, metadataDays: 0 }, apply: false, nowIso: NOW });
|
||||
const narrow = await eraseQueue(db, { scope: { kind: "routine", contentDays: 30, metadataDays: 0 }, apply: true, nowIso: NOW, eventsWindow: 1 });
|
||||
expect(narrow.linked.events).toBe(wide.linked.events);
|
||||
expect(narrow.linked.events).toBeGreaterThanOrEqual(2);
|
||||
expect(payloads()).not.toContain("secret-w-");
|
||||
});
|
||||
|
||||
it("J2: a linked copy held with someone outside the account is kept and reported, and the run isn't clean", async () => {
|
||||
await row("o-own", A, B, [{ state: "done", daysAgo: 2 }]);
|
||||
db.prepare(`INSERT INTO outbox_entries (outbox_id, sender_session, destination_session, body, ts_dispatched, audit_pointer)
|
||||
VALUES ('o-mixed', ?, ?, 'secret-mixed-copy', ?, 'o-own'), ('o-foreign', ?, ?, 'secret-foreign-copy', ?, 'o-own')`).run(A, OUT, NOW, OUT, OUT, NOW);
|
||||
const report = await eraseQueue(db, { scope: { kind: "members", members: [A, B] }, apply: true, nowIso: NOW });
|
||||
expect(report.keptCopies).toEqual(expect.arrayContaining([
|
||||
{ table: "outbox_entries", id: "o-mixed", qitemId: "o-own", ownership: "mixed" },
|
||||
{ table: "outbox_entries", id: "o-foreign", qitemId: "o-own", ownership: "foreign" },
|
||||
]));
|
||||
expect(report.clean).toBe(false);
|
||||
const all = everything();
|
||||
expect(all).toContain("secret-mixed-copy");
|
||||
expect(all).toContain("secret-foreign-copy");
|
||||
expect(all).not.toContain("secret-o-own");
|
||||
});
|
||||
|
||||
it("J3: a scrub clears closure notes in every copy and a duplicate reader action's reason, and keeps a successor id", async () => {
|
||||
await row("c-canceled", A, B);
|
||||
repo.update({ qitemId: "c-canceled", actorSession: B, state: "done", closureReason: "canceled", closureTarget: "secret-closure-note" });
|
||||
await row("c-handed", A, B);
|
||||
repo.update({ qitemId: "c-handed", actorSession: B, state: "done", closureReason: "handed_off_to", closureTarget: B });
|
||||
db.prepare("INSERT INTO stream_items (stream_item_id, ts_emitted, stream_sort_key, source_session, body) VALUES ('s-dup', ?, 'k', ?, 'x')").run(NOW, A);
|
||||
recordReaderAction(db, { streamItemId: "s-dup", readerSession: A, action: "duplicate", duplicateOf: "c-canceled", reason: "secret-dup-reason" }, NOW);
|
||||
db.prepare("UPDATE queue_transitions SET ts = ? WHERE qitem_id IN ('c-canceled', 'c-handed')").run(ago(40));
|
||||
await eraseQueue(db, { scope: { kind: "routine", contentDays: 30, metadataDays: 0 }, apply: true, nowIso: NOW });
|
||||
const all = everything();
|
||||
expect(all).not.toContain("secret-closure-note");
|
||||
expect(all).not.toContain("secret-dup-reason");
|
||||
expect(db.prepare("SELECT closure_target AS t FROM queue_items WHERE qitem_id = 'c-handed'").get()).toEqual({ t: B });
|
||||
});
|
||||
|
||||
it("J4: content written to a scrubbed row after its scrub is erased by the next run, and a reclose counts from the new close", async () => {
|
||||
await row("n-row", A, B, [{ state: "done", daysAgo: 40 }]);
|
||||
await eraseQueue(db, { scope: { kind: "routine", contentDays: 30, metadataDays: 0 }, apply: true, nowIso: NOW });
|
||||
repo.update({ qitemId: "n-row", actorSession: B, transitionNote: "secret-later-note" });
|
||||
const again = await eraseQueue(db, { scope: { kind: "routine", contentDays: 30, metadataDays: 0 }, apply: true, nowIso: NOW });
|
||||
expect(again.qitems.map((q) => q.qitemId)).toEqual(["n-row"]);
|
||||
expect(everything()).not.toContain("secret-later-note");
|
||||
repo.update({ qitemId: "n-row", actorSession: B, state: "pending", reopen: true, transitionNote: "secret-reopened" });
|
||||
repo.update({ qitemId: "n-row", actorSession: B, state: "done", closureReason: "no-follow-on" });
|
||||
const soon = await eraseQueue(db, { scope: { kind: "routine", contentDays: 30, metadataDays: 0 }, apply: true, nowIso: NOW });
|
||||
expect(soon.qitems).toEqual([]); // closed again just now: inside the window
|
||||
const later = new Date(Date.now() + 31 * 86_400_000).toISOString();
|
||||
const aged = await eraseQueue(db, { scope: { kind: "routine", contentDays: 30, metadataDays: 0 }, apply: true, nowIso: later });
|
||||
expect(aged.qitems.map((q) => q.qitemId)).toEqual(["n-row"]);
|
||||
expect(everything()).not.toContain("secret-reopened");
|
||||
});
|
||||
|
||||
it("J5: a dry run and an apply agree when deleting a row frees a member's stream item", async () => {
|
||||
await row("s-row", A, B, [{ state: "done", daysAgo: 2 }]);
|
||||
db.prepare("INSERT INTO stream_items (stream_item_id, ts_emitted, stream_sort_key, source_session, body) VALUES ('s-act', ?, 'k', ?, 'secret-stream')").run(NOW, A);
|
||||
recordReaderAction(db, { streamItemId: "s-act", readerSession: B, action: "act", workQitemId: "s-row" }, NOW);
|
||||
const dry = await eraseQueue(db, { scope: { kind: "members", members: [A, B] }, apply: false, nowIso: NOW });
|
||||
const applied = await eraseQueue(db, { scope: { kind: "members", members: [A, B] }, apply: true, nowIso: NOW });
|
||||
expect({ ...applied, applied: false }).toEqual(dry);
|
||||
expect(dry.unlinked).toMatchObject({ streamItems: 1, referencedStreamItems: 0 });
|
||||
expect(everything()).not.toContain("secret-stream");
|
||||
});
|
||||
|
||||
it("J8: a failure after a committed batch returns only what committed, and the sweep's other passes still run", async () => {
|
||||
await row("f-1", A, B, [{ state: "done", daysAgo: 2 }]);
|
||||
await row("f-2", A, B, [{ state: "done", daysAgo: 2 }]);
|
||||
db.exec("CREATE TEMP TRIGGER fail_f2 BEFORE DELETE ON queue_items WHEN old.qitem_id = 'f-2' BEGIN SELECT RAISE(ABORT, 'disk I/O error (test)'); END;");
|
||||
const report = await eraseQueue(db, { scope: { kind: "members", members: [A, B] }, apply: true, nowIso: NOW, batchSize: 1 });
|
||||
expect(report).toMatchObject({ complete: false, clean: false, error: { stage: "rows", message: "disk I/O error (test)" } });
|
||||
expect(report.qitems.map((q) => q.qitemId)).toEqual(["f-1"]);
|
||||
expect([exists("f-1"), exists("f-2")]).toEqual([false, true]);
|
||||
expect(report.linked.events).toBeGreaterThan(0);
|
||||
expect(payloads()).not.toContain('"qitemId":"f-1"');
|
||||
expect(payloads()).toContain('"qitemId":"f-2"');
|
||||
db.exec("CREATE TEMP TRIGGER fail_scrub BEFORE UPDATE OF content_erased_at ON queue_items BEGIN SELECT RAISE(ABORT, 'disk I/O error (test)'); END;");
|
||||
db.prepare("UPDATE queue_transitions SET ts = ? WHERE qitem_id = 'f-2'").run(ago(40));
|
||||
const swept = await runQueueRetentionSweep(db, { nowIso: NOW, closedContentDays: 30, transitionsRetentionDays: 1 });
|
||||
expect(swept.closedError).toMatch(/disk I\/O error/);
|
||||
expect(swept.archivedQitems).toBeGreaterThan(0); // the transition archive still ran
|
||||
});
|
||||
});
|
||||
|
||||
describe("round 1 controls: the account rig after removal, and a restore across a close", () => {
|
||||
const RIG = "rig-r1-id";
|
||||
beforeEach(() => {
|
||||
db.prepare("INSERT INTO rigs (id, name) VALUES (?, 'n-acct')").run(RIG);
|
||||
db.prepare("INSERT INTO nodes (id, rig_id, logical_id) VALUES ('node-r1', ?, 'a.one')").run(RIG);
|
||||
new EventBus(db).emit({ type: "rig.imported", rigId: RIG, specName: "secret-r1-spec", specVersion: "1" } as never);
|
||||
});
|
||||
|
||||
it("N2: refuses a rig with a session the members don't list", async () => {
|
||||
db.prepare("INSERT INTO sessions (id, node_id, session_name) VALUES ('sess-x', 'node-r1', ?)").run(OUT);
|
||||
await expect(eraseQueue(db, { scope: { kind: "members", members: [A, B], rig: RIG }, apply: false, nowIso: NOW }))
|
||||
.rejects.toMatchObject({ code: "erase_rig_mismatch" });
|
||||
});
|
||||
|
||||
it("J6: after removal a name-bound workflow record is counted as left, and an unverified id erases nothing", async () => {
|
||||
db.pragma("foreign_keys = OFF");
|
||||
db.prepare(`INSERT INTO workflow_specs (spec_id, name, version, purpose, target_rig, roles_json, steps_json, coordination_terminal_turn_rule,
|
||||
source_path, source_hash, cached_at) VALUES ('spec-1', 's', '1', 'p', 'n-acct', '[]', '[]', 'hot_potato', '/x', 'h', ?)`).run(NOW);
|
||||
db.pragma("foreign_keys = ON");
|
||||
await eraseQueue(db, { scope: { kind: "members", members: [A, B], rig: RIG }, apply: true, nowIso: NOW });
|
||||
db.prepare("DELETE FROM nodes WHERE rig_id = ?").run(RIG);
|
||||
db.prepare("DELETE FROM rigs WHERE id = ?").run(RIG);
|
||||
new EventBus(db).emit({ type: "rig.deleted", rigId: RIG } as never);
|
||||
const rerun = await eraseQueue(db, { scope: { kind: "members", members: [A, B], rig: RIG }, apply: true, nowIso: NOW });
|
||||
expect(rerun.rig).toMatchObject({ found: false, verified: true, left: { workflow_specs: 1 }, unknown: [] });
|
||||
expect(rerun.clean).toBe(false);
|
||||
// Another removed rig's id, never verified with this membership: nothing keyed by it is erased.
|
||||
new EventBus(db).emit({ type: "rig.deleted", rigId: "rig-someone-else" } as never);
|
||||
const unverified = await eraseQueue(db, { scope: { kind: "members", members: [A, B], rig: "rig-someone-else" }, apply: true, nowIso: NOW });
|
||||
expect(unverified.rig).toMatchObject({ found: false, verified: false, left: { events: 1 }, unknown: ["workflow_instances", "workflow_specs"] });
|
||||
expect(unverified.clean).toBe(false);
|
||||
expect(db.prepare("SELECT COUNT(*) AS n FROM events WHERE rig_id = 'rig-someone-else'").get()).toEqual({ n: 1 });
|
||||
});
|
||||
|
||||
it("J7: a backup taken before a row closed is replayed from the ledger, and restored open work stays", async () => {
|
||||
const ledgerPath = join(mkdtempSync(join(tmpdir(), "erase-ledger-")), "ledger.jsonl");
|
||||
await row("l-row", A, B); // open when the backup is taken
|
||||
await row("l-open", A, B); // still open later
|
||||
const backup = db.serialize();
|
||||
repo.update({ qitemId: "l-row", actorSession: B, state: "done", closureReason: "no-follow-on" });
|
||||
db.prepare("UPDATE queue_transitions SET ts = ? WHERE qitem_id = 'l-row'").run(ago(40));
|
||||
await eraseQueue(db, { scope: { kind: "routine", contentDays: 30, metadataDays: 0 }, apply: true, nowIso: NOW, ledgerPath });
|
||||
const lines = readFileSync(ledgerPath, "utf8").trim().split("\n").map((line) => JSON.parse(line) as { qitemId: string; action: "scrub" });
|
||||
expect(lines).toEqual([{ qitemId: "l-row", action: "scrub", reason: "retention-content", erasedAt: NOW }]);
|
||||
db.close();
|
||||
open(new Database(backup));
|
||||
expect(everything()).toContain("secret-l-row body");
|
||||
await eraseQueue(db, { scope: { kind: "ids", entries: lines }, apply: true, nowIso: NOW });
|
||||
await eraseQueue(db, { scope: { kind: "routine", contentDays: 30, metadataDays: 0 }, apply: true, nowIso: NOW });
|
||||
const all = everything();
|
||||
expect(all).not.toContain("secret-l-row");
|
||||
expect(all).toContain("secret-l-open body");
|
||||
});
|
||||
});
|
||||
|
||||
describe("round 2 controls", () => {
|
||||
const routine = { kind: "routine", contentDays: 30, metadataDays: 0 } as const;
|
||||
|
||||
it("K1: a row reopened or taken up by a workflow after selection is kept and reported, with its events", async () => {
|
||||
await row("k-reopen", A, B, [{ state: "done", daysAgo: 100 }]);
|
||||
await row("k-workflow", A, B, [{ state: "done", daysAgo: 100 }]);
|
||||
await row("k-stays", A, B, [{ state: "done", daysAgo: 100 }]);
|
||||
// The run yields after each events window. With a window of 1, these writes land during the walk, after selection.
|
||||
const run = eraseQueue(db, { scope: { kind: "routine", contentDays: 30, metadataDays: 90 }, apply: true, nowIso: NOW, eventsWindow: 1 });
|
||||
await new Promise((resolve) => setImmediate(resolve));
|
||||
repo.update({ qitemId: "k-reopen", actorSession: B, state: "pending", reopen: true, transitionNote: "back to work" });
|
||||
db.pragma("foreign_keys = OFF");
|
||||
db.prepare(`INSERT INTO workflow_frontier_bindings (instance_id, packet_id, step_id, created_at) VALUES ('wf-k', 'k-workflow', 's1', ?)`).run(NOW);
|
||||
db.pragma("foreign_keys = ON");
|
||||
const report = await run;
|
||||
expect(report.qitems.map((q) => q.qitemId)).toEqual(["k-stays"]);
|
||||
expect(report.gaps).toEqual(expect.arrayContaining([{ code: "changed-during-run", qitemId: "k-reopen" },
|
||||
{ code: "workflow-reference", qitemId: "k-workflow" }]));
|
||||
expect(report.clean).toBe(false);
|
||||
expect(db.prepare("SELECT state, body FROM queue_items WHERE qitem_id = 'k-reopen'").get()).toEqual({ state: "pending", body: "secret-k-reopen body" });
|
||||
expect(marker("k-reopen")).toBeUndefined();
|
||||
expect(payloads()).toContain("secret-k-reopen");
|
||||
expect(payloads()).toContain("secret-k-workflow");
|
||||
expect([exists("k-workflow"), exists("k-stays")]).toEqual([true, false]);
|
||||
});
|
||||
|
||||
it("K2: a ledger that can't be written stops the run before anything changes", async () => {
|
||||
await row("l-one", A, B, [{ state: "done", daysAgo: 2 }]);
|
||||
const before = everything();
|
||||
const ledgerPath = join(mkdtempSync(join(tmpdir(), "erase-ledger-")), "missing", "ledger.jsonl");
|
||||
const report = await eraseQueue(db, { scope: { kind: "ids", entries: [{ qitemId: "l-one", action: "delete" }, { qitemId: "l-gone", action: "delete" }] },
|
||||
apply: true, nowIso: NOW, ledgerPath });
|
||||
expect(report).toMatchObject({ complete: false, clean: false, error: { stage: "ledger" }, qitems: [], alreadyAbsent: [] });
|
||||
expect(everything()).toBe(before);
|
||||
expect(payloads()).toContain("secret-l-one");
|
||||
});
|
||||
|
||||
it("K3, L3: an event that names a row but can't be read is reported with its row kept, on every rerun until it's resolved", async () => {
|
||||
await row("u-row", A, B, [{ state: "done", daysAgo: 2 }]);
|
||||
await row("u-other", A, B, [{ state: "done", daysAgo: 2 }]);
|
||||
const bad = db.prepare("INSERT INTO events (type, payload) VALUES ('queue.updated', ?)").run('{"qitemId":"u-row","summary":"secret-truncated');
|
||||
const dry = await eraseQueue(db, { scope: { kind: "members", members: [A, B] }, apply: false, nowIso: NOW });
|
||||
const report = await eraseQueue(db, { scope: { kind: "members", members: [A, B] }, apply: true, nowIso: NOW });
|
||||
expect({ ...report, applied: false }).toEqual(dry);
|
||||
expect(report.unreadableEvents).toEqual([{ seq: Number(bad.lastInsertRowid), qitemIds: ["u-row"] }]);
|
||||
expect(report.gaps).toEqual([{ code: "unreadable-event", qitemId: "u-row" }]);
|
||||
expect([report.complete, report.clean]).toEqual([true, false]);
|
||||
expect([exists("u-row"), exists("u-other")]).toEqual([true, false]);
|
||||
// An identical rerun (after a lost response, say) finds it again.
|
||||
const again = await eraseQueue(db, { scope: { kind: "members", members: [A, B] }, apply: true, nowIso: NOW });
|
||||
expect(again.unreadableEvents).toEqual(report.unreadableEvents);
|
||||
expect(again.clean).toBe(false);
|
||||
expect(payloads()).toContain("secret-truncated");
|
||||
// Once the event is resolved, the next run erases the row and is clean.
|
||||
db.prepare("DELETE FROM events WHERE seq = ?").run(bad.lastInsertRowid);
|
||||
const resolved = await eraseQueue(db, { scope: { kind: "members", members: [A, B] }, apply: true, nowIso: NOW });
|
||||
expect([resolved.clean, exists("u-row")]).toEqual([true, false]);
|
||||
});
|
||||
|
||||
it("K4: a successor reason keeps its target only when it names a session or a queue row", async () => {
|
||||
await row("t-prose", A, B);
|
||||
repo.update({ qitemId: "t-prose", actorSession: B, state: "done", closureReason: "handed_off_to", closureTarget: "secret-prose-target, see the thread" });
|
||||
await row("t-super", A, B);
|
||||
repo.update({ qitemId: "t-super", actorSession: B, state: "canceled", closureReason: "superseded", closureTarget: "qitem-successor-row" });
|
||||
db.prepare("UPDATE queue_transitions SET ts = ? WHERE qitem_id IN ('t-prose', 't-super')").run(ago(40));
|
||||
const report = await eraseQueue(db, { scope: routine, apply: true, nowIso: NOW });
|
||||
expect(report.qitems.map((q) => q.qitemId)).toEqual(["t-prose", "t-super"]);
|
||||
expect(everything()).not.toContain("secret-prose-target");
|
||||
expect(payloads()).not.toContain("secret-prose-target");
|
||||
expect(db.prepare("SELECT closure_target AS t FROM queue_items WHERE qitem_id = 't-super'").get()).toEqual({ t: "qitem-successor-row" });
|
||||
});
|
||||
|
||||
it("N1: a note archived while erasure was off still makes a scrubbed row eligible again", async () => {
|
||||
await row("v-row", A, B, [{ state: "done", daysAgo: 40 }]);
|
||||
await eraseQueue(db, { scope: routine, apply: true, nowIso: NOW });
|
||||
repo.update({ qitemId: "v-row", actorSession: B, transitionNote: "secret-archived-note" });
|
||||
const later = new Date(Date.now() + 2 * 86_400_000).toISOString();
|
||||
await runQueueRetentionSweep(db, { nowIso: later, transitionsRetentionDays: 1 });
|
||||
const note = "SELECT COUNT(*) AS n FROM %s WHERE transition_note = 'secret-archived-note'";
|
||||
expect([db.prepare(note.replace("%s", "queue_transitions")).get(), db.prepare(note.replace("%s", "queue_transitions_archive")).get()])
|
||||
.toEqual([{ n: 0 }, { n: 1 }]);
|
||||
const again = await eraseQueue(db, { scope: routine, apply: true, nowIso: later });
|
||||
expect(again.qitems.map((q) => q.qitemId)).toEqual(["v-row"]);
|
||||
expect(everything()).not.toContain("secret-archived-note");
|
||||
});
|
||||
|
||||
it("N2: a dry run and an apply count the same mixed entries when a kept copy points at an erased row", async () => {
|
||||
await row("m-row", A, B, [{ state: "done", daysAgo: 2 }]);
|
||||
db.prepare(`INSERT INTO outbox_entries (outbox_id, sender_session, destination_session, body, ts_dispatched, audit_pointer)
|
||||
VALUES ('o-m', ?, ?, 'secret-m-copy', ?, 'm-row')`).run(A, OUT, NOW);
|
||||
const dry = await eraseQueue(db, { scope: { kind: "members", members: [A, B] }, apply: false, nowIso: NOW });
|
||||
const applied = await eraseQueue(db, { scope: { kind: "members", members: [A, B] }, apply: true, nowIso: NOW });
|
||||
expect({ ...applied, applied: false }).toEqual(dry);
|
||||
expect(dry.keptCopies).toEqual([{ table: "outbox_entries", id: "o-m", qitemId: "m-row", ownership: "mixed" }]);
|
||||
expect(dry.unlinked).toMatchObject({ mixedEntries: 0 });
|
||||
});
|
||||
});
|
||||
|
||||
describe("round 3 controls", () => {
|
||||
// The run yields after each events window. With a window of 1, a write queued now lands during the walk, after
|
||||
// selection and before the batches.
|
||||
const duringWalk = () => new Promise((resolve) => setImmediate(resolve));
|
||||
|
||||
it("L1: a row rerouted out of the account after selection is kept with its events and reported", async () => {
|
||||
await row("l1-moved", A, B, [{ state: "done", daysAgo: 2 }]);
|
||||
await row("l1-stays", A, B, [{ state: "done", daysAgo: 2 }]);
|
||||
const run = eraseQueue(db, { scope: { kind: "members", members: [A, B] }, apply: true, nowIso: NOW, eventsWindow: 1 });
|
||||
await duringWalk();
|
||||
repo.routeToFallback("l1-moved", OUT, "owner unavailable");
|
||||
const report = await run;
|
||||
expect(report.qitems.map((q) => q.qitemId)).toEqual(["l1-stays"]);
|
||||
expect(report.gaps).toEqual([{ code: "changed-during-run", qitemId: "l1-moved" }]);
|
||||
expect(report.clean).toBe(false);
|
||||
expect(db.prepare("SELECT destination_session AS d, body FROM queue_items WHERE qitem_id = 'l1-moved'").get())
|
||||
.toEqual({ d: OUT, body: "secret-l1-moved body" });
|
||||
expect(marker("l1-moved")).toBeUndefined();
|
||||
expect(payloads()).toContain("secret-l1-moved");
|
||||
});
|
||||
|
||||
it("L2: an id absent at selection but created during the run is reported, never certified absent", async () => {
|
||||
await row("l2-old", A, B, [{ state: "done", daysAgo: 2 }]);
|
||||
const run = eraseQueue(db, { scope: { kind: "ids", entries: [{ qitemId: "l2-new", action: "delete" }, { qitemId: "l2-old", action: "delete" }] },
|
||||
apply: true, nowIso: NOW, eventsWindow: 1 });
|
||||
await duringWalk();
|
||||
await repo.create({ qitemId: "l2-new", sourceSession: A, destinationSession: B, body: "secret-l2-new body" });
|
||||
const report = await run;
|
||||
expect(report.qitems.map((q) => q.qitemId)).toEqual(["l2-old"]);
|
||||
expect(report.alreadyAbsent).toEqual([]);
|
||||
expect(report.gaps).toEqual([{ code: "changed-during-run", qitemId: "l2-new" }]);
|
||||
expect(report.clean).toBe(false);
|
||||
expect(marker("l2-new")).toBeUndefined();
|
||||
expect(everything()).toContain("secret-l2-new body");
|
||||
});
|
||||
});
|
||||
@@ -138,10 +138,15 @@ describe("config routes (User Settings v0)", () => {
|
||||
// + 1 acp.to (the ACP agent's receiver; default empty).
|
||||
// + 1 daemon.seat_poll_interval_ms (#2687 the seat pollers' cadence; startup-only, default 1000).
|
||||
// + 1 retention.events_days (the history-only event types of events; default 0 = keep).
|
||||
expect(Object.keys(body.settings).length).toBe(100);
|
||||
// + 2 retention.closed_content_days, retention.closed_metadata_days (closed-work erasure; default 0 = keep).
|
||||
// + 1 retention.erasure_ledger (queue erasure's replay ledger; default empty = none).
|
||||
expect(Object.keys(body.settings).length).toBe(103);
|
||||
expect(body.settings["launch.codex_login_shell"]).toMatchObject({ value: false, source: "default", defaultValue: false });
|
||||
expect(body.settings["transport.draft_delivery"]).toMatchObject({ value: "deliver", source: "default", defaultValue: "deliver" });
|
||||
expect(body.settings["retention.events_days"]).toMatchObject({ value: 0, source: "default", defaultValue: 0 });
|
||||
expect(body.settings["retention.closed_content_days"]).toMatchObject({ value: 0, source: "default", defaultValue: 0 });
|
||||
expect(body.settings["retention.closed_metadata_days"]).toMatchObject({ value: 0, source: "default", defaultValue: 0 });
|
||||
expect(body.settings["retention.erasure_ledger"]).toMatchObject({ value: "", source: "default", defaultValue: "" });
|
||||
expect(body.settings["specs.builtins_visible"]).toMatchObject({ value: true, source: "default", defaultValue: true });
|
||||
expect(body.settings["stream.intake"]).toMatchObject({ value: "queue.worker", source: "default", defaultValue: "queue.worker" });
|
||||
expect(body.settings["retention.stream_days"]).toMatchObject({ value: 30, source: "default", defaultValue: 30 });
|
||||
|
||||
@@ -196,6 +196,9 @@ describe("SettingsStore (User Settings v0)", () => {
|
||||
"retention.watchdog_days",
|
||||
"retention.usage_samples_days", // 51-08 A2 (plan-lock rev-1): the PM-ruled 14d telemetry window
|
||||
"retention.events_days", // the history-only event types of events
|
||||
"retention.closed_content_days", // closed-work erasure: content
|
||||
"retention.closed_metadata_days", // closed-work erasure: the row, keeping its id
|
||||
"retention.erasure_ledger", // queue erasure's replay ledger file
|
||||
"retention.watchdog_keep_per_job",
|
||||
"retention.stream_days", // slice 22 outcome 9: stream retention
|
||||
"retention.stream_max_items",
|
||||
@@ -268,6 +271,24 @@ describe("SettingsStore (User Settings v0)", () => {
|
||||
}
|
||||
});
|
||||
|
||||
it.each([
|
||||
["retention.closed_content_days", "closedContentDays", "OPENRIG_RETENTION_CLOSED_CONTENT_DAYS"],
|
||||
["retention.closed_metadata_days", "closedMetadataDays", "OPENRIG_RETENTION_CLOSED_METADATA_DAYS"],
|
||||
] as const)("%s defaults to 0 (keep), resolves file then env, and refuses negative or non-integer values", (key, fileKey, envKey) => {
|
||||
const store = new SettingsStore(configPath);
|
||||
expect(store.resolveOne(key)).toMatchObject({ value: 0, source: "default" });
|
||||
store.set(key, "30");
|
||||
expect(store.resolveOne(key)).toMatchObject({ value: 30, source: "file" });
|
||||
expect(JSON.parse(readFileSync(configPath, "utf-8")).retention[fileKey]).toBe(30);
|
||||
process.env[envKey] = "7";
|
||||
try {
|
||||
expect(store.resolveOne(key)).toMatchObject({ value: 7, source: "env" });
|
||||
} finally {
|
||||
delete process.env[envKey];
|
||||
}
|
||||
for (const raw of ["-1", "2.5", "30days"]) expect(() => store.set(key, raw)).toThrow(new RegExp(key.replace(".", "\\.")));
|
||||
});
|
||||
|
||||
it("retention.events_days defaults to 0 (keep), resolves file then env, accepts 0 and refuses negative or non-integer values", () => {
|
||||
const store = new SettingsStore(configPath);
|
||||
expect(store.resolveOne("retention.events_days")).toMatchObject({ value: 0, source: "default" });
|
||||
|
||||
@@ -60,7 +60,7 @@ const LABELS: Record<string, string> = {
|
||||
"queue.stuck_sweep_unclaimed_age_minutes": "Unclaimed age", "policies.idle_gate_qitem.auto_register": "Auto-registration",
|
||||
"snapshots.periodic.enabled": "Periodic snapshots", "snapshots.periodic.interval_seconds": "Snapshot interval",
|
||||
"snapshots.periodic.retention_keep": "Snapshots to retain", "ui.terminal.max_live_terminals": "Legacy web terminal limit",
|
||||
"retention.usage_samples_days": "Usage sample retention", "retention.events_days": "Event history retention (days)",
|
||||
"retention.usage_samples_days": "Usage sample retention", "retention.events_days": "Event history retention (days)", "retention.closed_content_days": "Closed work content retention (days)", "retention.closed_metadata_days": "Closed work row retention (days)", "retention.erasure_ledger": "Erasure ledger file",
|
||||
"retention.stream_days": "Stream retention (days)", "retention.stream_max_items": "Stream items kept",
|
||||
"transcripts.poll_interval_seconds": "Transcript refresh", "transcripts.lines": "Transcript lines",
|
||||
"ui.preview.refresh_interval_seconds": "Preview refresh", "ui.preview.max_pins": "Preview pins",
|
||||
|
||||
Reference in new issue
Block a user