Files
paperclip/scripts/test-skill-default-cadence.mjs
DottaandPaperclip 44c60c7221 Verify default skill sync through two real cold sleep cycles (#19191)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Automatic skill sources check upstream every five minutes.
> - The domain deadline contract can hand future work to a durable wake
alarm.
> - A saved deadline or a worker unit test does not prove that a stopped
tenant resumes the real producer.
> - This pull request checks the default cadence through the actual app
and a matching service checkout.
> - It checks two cold starts and three distinct persisted skill
revisions with real clocks.
> - The benefit is repeatable local evidence without changing production
timing or authority.

## Linked Issues or Issue Description

Refs: #19001, #17008, #17923.

This is a test and documentation follow-up to #19001. #19001 is now
merged; this PR targets master. Feature code remains in that merged
foundation. #17008 covers the earlier source worker change. #17923
covers separate skill-cache retention; this PR does not change that
behavior.

A scoped search of public PRs and issues found no duplicate
default-cadence cold-cycle acceptance test. No matching public issue was
found. The enhancement fields follow.

**Pre-submission checklist**

This improves checks for an existing feature. It adds no product
endpoint or execution authority. The public related-work search is
complete.

**What existing behavior does this improve?**

Verification of automatic skill-source updates across a durable idle
handoff and cold restart.

**Subsystem affected**

`server/` test fixtures and development documentation.

**Current behavior**

Domain tests cover claims, access and future deadlines. They do not
prove the full default-300-second path from ordinary source attachment
through signed tenant reports, durable alarms, stopped compute and a
changed persisted skill.

**Proposed behavior**

An opt-in launcher checks bootstrap first, then two full real-clock
cycles. Ordinary CI can run the transport controls without the host
fixture. The launcher-specific worker assertion runs only when prewire
is requested, and fails closed in that lane.

**Reason and benefit**

The test can detect an early refresh, a lost deadline, a stale proof, an
ACK after physical stop, a failed cold restart or a producer that does
not publish changed contents.

**Breaking changes**

None to product APIs, schema, cadence, authorization, claims or retry
policy. Existing maintenance-helper defaults stay unchanged.

**Additional context**

The opt-in host flow needs an authorized matching service checkout, its
normal compiled acceptance bridge, Node 22 and two owned PostgreSQL test
databases. The standalone controls do not need that service or
PostgreSQL. Roadmap review places this under existing Skills Manager and
Cloud deployment verification, not a new core feature.

## What Changed

- Add an opt-in default-cadence acceptance test and checked-in launcher.
- Reuse real server startup, managed tenant authority and the ordinary
project attachment route. Control only the remote Git snapshot boundary;
keep actual authorization, scanning, claims and publication.
- Use actual signed tenant routes and the matching service's gate, sleep
controller and PostgreSQL wake-alarm store. Stop and restart real app
children through a controlled local provider.
- Preload the Node transport guard before coordinator, worker and child
module work. Check worker-local and child PID, thread and source-hash
receipts. Keep caught unexpected transport attempts fatal.
- Require both persisted refresh times and actual upstream handler
timestamps to be at or after each original deadline.
- Join physical callbacks during bridge close, stop a late wake-created
child, and preserve the original cleanup budget. Add a real
IPC/HTTP-child counterfactual.
- Accept valid owned HTTP options paths without allowing Unix sockets or
unknown origins. Remove the unused App helper export.
- Add finite transport and launcher controls. Defer only the
launcher-specific parent assertion in vanilla CI. Keep all subprocess
controls default-runnable.
- Add optional child entry, preload and bounded pool settings to the
existing test helper. Preserve its defaults and the real observer test
body.
- Remove added explanatory comments under the contribution policy.
Preserve executable syntax, directives and literals.
- Document pinned setup, fresh owned databases, bootstrap and full
two-cycle commands in `doc/DEVELOPING.md`.

The diff against the intended base contains nine test/helper/launcher
files and this one document. It adds no production feature code or Cloud
source to the App repository.

## Verification

The latest two-file CI repair preserves both original failed jobs. A
newer master workflow introduced two required step inputs absent from
its extracted-step test environment; the faithful false/false inputs now
pass all five existing target-authorization cases, after an actual
missing-input counterfactual failed. A controlled socket case also
reproduces legal reuse of the released runner port; the fixture now
holds that reservation until the preview binds, releases it in finally,
and asserts distinct ports. All original retry/result/accounting
assertions and timeouts stay unchanged. The original Linux listener
owner remains unknown; the control proves this fixture collision
mechanism, not that original attribution. Fresh current-head full CI and
complete reviews are pending.


Current merge-integration candidate:
`f6f686103b7bfd0a4ab83085634c622af3896ecb`, a normal merge after #19001
landed, including actual master
`1c9767d323cf872a266962b9e6b4bd5cbcb57f1c`. All nine acceptance code
files are byte-identical to148a82e5; both parents' documentation
additions survive. Every other path equals actual master. The only
conflict was the old foundation's startup block; current master was
retained in full. Fresh exact-head CI and complete reviews are pending.
The local measurements below belong to their explicitly named revisions;
no fresh full614-second run is claimed.

Current candidate: `148a82e54c502759785407357001c6b50c1439dd`, normally
carrying the current base `8c912e1ba4fc66397a7341ade59373af8075068f`.
All nine acceptance code blobs remain identical to qualified repair
`e092746f4da181c8414c63be71474619a84b97e7`; documentation retains both
parents. The matching built service for the fresh bootstrap is
`389878364b42dd129835d263f91882c5774d5123`, using Node 22.23.4. The App
uses Node 24.22.2 and pnpm 9.15.4.

Use the exact controlled setup and commands in
[doc/DEVELOPING.md](doc/DEVELOPING.md#controlled-default-cadence-sleep-acceptance).
Fresh owned databases and the matching compiled service are
prerequisites; the launcher does not create a cluster or build
dependencies.

- **Current affected semantic check passed**, using actual native
TypeScript 7 and the current generated CI options/references (0.640
seconds). Normal declaration references passed in 11.726 seconds, with
source-matched SDK/shared/kernel/Runner prerequisites and the frozen
current lockfile.
- **Current guarded controls: 44 passed**, plus the same single explicit
real-PG observer deferral, in 4.790 seconds. The executing worker
checked its own global PID/thread/preload source hash and matching
receipt. Guard hash:
`7daf9ae7d2f26e8fcd389b1d560fac1d4e218a8304a2c58e49f646af108165bc`. No
unexpected attempts or PG sockets occurred. Vanilla controls on the
identical repair files passed 22 cases with one explicit launcher-parent
assertion deferral (2.044 seconds).
- **Real cleanup counterfactual:** unchanged old helper failed the
actual endpoint assertion in 1.568 seconds: its late HTTP child remained
reachable after close returned. The same fixed-helper case passed in
1.485 seconds: close waited, the child exited, the port was unreachable
and no IPC error occurred. The isolated driver records only the known
legacy closed-IPC error long enough to inspect cleanup; current helper
requires zero such errors. All recorded processes and ports were
independently checked empty.
- **Fresh current bootstrap passed once**, 16.159 seconds wall / 14.12
seconds Vitest / 12.237 seconds body. The ordinary enabled producer
persisted useful revision 1, cleared its lease and retained exactly
300,000 ms between success and original next deadline. A signed
current-generation report produced the matching
generation/operation-lock SQL ACK before physical child stop (ACK
06:54:02.020 UTC; stop signal 06:54:02.025 UTC). The alarm retained the
original D and D-120 eligibility. Both fixture databases ended with zero
sessions; the coordinator independently audited and normally removed
them. This bootstrap does not execute a due wake or the new v2/v3 timing
assertions.
- **Historical full two-cycle result remains 1/1 passed in 614.213
seconds wall**, at measured App
`74f7c012ca17ad0d7302d76403f5033affb1430b` and service
`2cb54f904031dcf69a32902cd853b43f74d6f301`. Two new processes published
distinct immutable v2/v3 at original D+115 ms and D+98 ms. Exact signed
SQL ACKs preceded both physical stops; real SQL D-120 claims caused both
cold starts. Features, claims and original business deadlines stayed
unchanged.
- Both historical 10.002-second quiet windows kept app SQL, upstream and
outgoing HTTP counts unchanged. Two awake PG sockets remained. Quiet
counts do not mean all connections closed.
- The stronger timing checks are additional assertions, not
retroactively executed results. Offline inspection of the original
receipts confirms persisted successes at D+115/+98 ms and earliest
upstream handler entries at D+44/+30 ms; all five requests in each
refresh group started after D. The exact new pre-sleep request counter
and dispatch-before-handler time were not independently recorded
historically.
- Source review found 19 critical automatic Git producer/runtime files
identical to the measured App. The current active cache helper change
does not run on automatic Git publication and adds no pool. The matching
service carries unchanged measured helper/sleep/controller/alarm seams
through its normal master merge. The current bootstrap and focused
controls qualify the changed cleanup/guard and parent integration. No
fresh current-head 614-second pass or whole-tree identity is claimed.
- Runtime is serialized: worker 1, per-pool limits at most 4, configured
App potential 10 + shared service pool 2 + one observer against server
max 40. Current bootstrap point samples observed at most 3 owned clients
plus the observer; those samples are not a continuous aggregate peak.
App/service guard peaks were 2 each, with zero remaining sockets at
cleanup.

Failures remain accounted for. The first historical bootstrap cleanup
masked an unknown primary error. Later attempts exposed missing
fixture-owned domain metadata and lost origin snapshots; test-only
ownership/cleanup corrections preserved the original assertions and
timeouts. The new causal driver first lacked a final observation, then
encountered sandbox bind EPERM, then exposed the old closed-IPC error;
only the final real endpoint failure above is labeled the cleanup
counterfactual. An offline dependency miss and a generator-cleared
ignored compiler config were retained and corrected without weakening
compiler options. The former inherited policy failure (397 findings) is
resolved by the genuine current-master base/scanner, with zero findings
and seven scanner tests; no unrelated UI site was edited and the gate
was not relaxed.

Historical real TCP observer coverage passed 1/1 in 3.688 seconds.
Comment-only and opt-in-selection carries remain documented in prior
evidence; self-hash changes are metadata exceptions with fresh
executing-worker checks.

The full workspace typecheck, test, build and browser commands were not
run locally for this test-only slice. The current base passes its full
exact-head CI and Greptile 5/5; this leaf now has all 56 exact-head
check runs terminal (54 success, one neutral, one skip), the additional
legacy security status passing, complete current Greptile 5/5 and
Ponytail 5/5 reviews, and all four review threads resolved across all
pages. No live provider or deployed host/database suspension was
exercised.
## Risks

- The host test requires a matching service checkout and compiled
bridge. An unrelated service revision is not a valid substitute.
- Controlled localhost Git, S3 and physical process operations do not
qualify deployed Railway or Neon suspension, external permissions or
provider uncertainty.
- The Node guard does not prove native subprocess transport isolation.
Only finite local Git metadata and Node version commands were needed.
- Two short SQL/HTTP quiet windows do not establish strict 90-second
host acceptance. Separate plugin-host LISTEN, MCP wake and other domain
gaps remain open.
- Explicit test URLs are destructive fixture inputs. Use fresh, owned
`_test` databases. The launcher does not start a cluster or choose a
database implicitly.
- Passing CI and review do not qualify deployed suspension; the current
proof boundaries above still apply.

> For core feature work, check `ROADMAP.md` first and discuss it in
`#dev` before opening the PR. This PR adds verification to the existing
feature in its base.

## Model Used

OpenAI `gpt-6.1-sol` assisted with implementation, repository review and
local code execution. OpenAI Codex, based on GPT-6, performed managing
review and integration. The context-window sizes and exact served
backend snapshots were not exposed.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change and contains no internal
Paperclip ticket id
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green — fresh merge-head gates pending
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups —
fresh merge-head review pending
- [x] I will address all Greptile and reviewer comments before
requesting merge


### Current master reconciliation (2026-11-06)

Merged actual master `e28158b353fd274f0322be76d62cd223e0df599f`
normally. All nine qualified acceptance code files and the held-port
repair remain byte-identical. Upstream independently landed the workflow
input fix in #19602; the fixture now uses that exact upstream version
once, avoiding duplicate object keys in a squash merge. All other master
changes are preserved. The earlier two-cycle local evidence remains
source-specific; no fresh 614-second run or deployed suspension is
claimed. This new head requires fresh CI and bot reviews.

### Actual combined-tree typecheck failure and repair

CI job `128156887548` failed with TS1117 at workflow-target.test.ts:22:
two independently landed master fixes declared the same `IMAGE_ONLY` and
`TESTOPS_REQUESTED` keys. The complete original log is retained. A
further normal master merge now removes the duplicate declarations,
retaining exactly one false/false pair and every original assertion. All
nine qualified acceptance code paths and the held-port repair remain
byte-identical; unrelated master changes are preserved. This is a
concrete compiler failure and source repair, with fresh exact-head
CI/reviews required.

---------

Co-authored-by: Paperclip <[email protected]>
2026-11-06 15:55:05 -06:00

59 lines
4.7 KiB
JavaScript

import { createRequire } from 'node:module';
import { spawn } from 'node:child_process';
import { existsSync, readFileSync, statSync } from 'node:fs';
import path from 'node:path';
import { fileURLToPath } from 'node:url';
const root = fileURLToPath(new URL('../', import.meta.url));
const controls = process.argv.slice(2).join(' ') === '--controls';
if (!controls && process.argv.length > 2) throw new Error('Use no arguments for the witness or --controls for finite PG0 controls');
const names = ['PAPERCLIP_SKILL_DEFAULT_CADENCE_TEST_DATABASE_URL', 'PAPERCLIP_SKILL_DEFAULT_CADENCE_CLOUD_TEST_DATABASE_URL'];
if (!controls && process.env.PAPERCLIP_MAINTENANCE_PG0_CONTROLS) throw new Error('PG0 controls flag is forbidden for the witness');
if (!controls) {
const urls = names.map(name => {
if (!process.env[name]) throw new Error(`${name} requires an explicit dedicated test database`);
let url; try { url = new URL(process.env[name]); } catch { throw new Error(`${name} requires a loopback dedicated test database`); }
if (!['postgres:', 'postgresql:'].includes(url.protocol) || !['127.0.0.1', 'localhost', '::1', '[::1]'].includes(url.hostname)
|| !/^[a-zA-Z0-9_]+_test$/.test(decodeURIComponent(url.pathname.slice(1)))
|| ['options', 'dbname', 'database'].some(key => url.searchParams.has(key))) throw new Error(`${name} requires a loopback dedicated test database`);
return url;
});
if (urls[0].pathname === urls[1].pathname) throw new Error('App and Cloud require distinct test databases');
const cloud = process.env.PAPERCLIP_SKILL_DEFAULT_CADENCE_CLOUD_CHECKOUT;
if (!cloud || !path.isAbsolute(cloud) || !existsSync(path.join(cloud, 'dist/sleep/fixtures/default-cadence-bridge.js'))) throw new Error('Set the exact compiled Cloud checkout');
if (!path.isAbsolute(process.env.PAPERCLIP_SKILL_DEFAULT_CADENCE_CLOUD_NODE || '')) throw new Error('Set the actual Node22 executable');
if (!/^[a-f0-9]{40}$/.test(process.env.PAPERCLIP_SKILL_DEFAULT_CADENCE_CLOUD_SHA || '')) throw new Error('Set exact Cloud source SHA');
}
const log = process.env.PAPERCLIP_SKILL_DEFAULT_CADENCE_GUARD_LOG;
if (!log || !path.isAbsolute(log) || !existsSync(path.dirname(log))) throw new Error('Set an absolute guard log in the existing evidence directory');
if (existsSync(log) && statSync(log).size !== 0) throw new Error('Use a fresh per-attempt guard log');
if (!process.version.startsWith('v24.')) throw new Error('Launch acceptance with the actual Node24 executable');
const require = createRequire(new URL('../server/package.json', import.meta.url));
const guard = path.join(root, 'scripts/fixtures/skill-default-cadence-network-guard.cjs');
require(guard);
const vitest = path.join(path.dirname(require.resolve('vitest/package.json')), 'vitest.mjs');
const env = { PATH: process.env.PATH, HOME: process.env.HOME, LANG: process.env.LANG, TMPDIR: process.env.TMPDIR,
NODE_ENV: 'test', NODE_OPTIONS: `--require=${guard}`, PAPERCLIP_SKILL_DEFAULT_CADENCE_GUARD_LOG: log };
if (controls) env.PAPERCLIP_MAINTENANCE_PG0_CONTROLS = '1';
if (!controls) {
for (const name of [...names, 'PAPERCLIP_SKILL_DEFAULT_CADENCE_CLOUD_CHECKOUT', 'PAPERCLIP_SKILL_DEFAULT_CADENCE_CLOUD_NODE',
'PAPERCLIP_SKILL_DEFAULT_CADENCE_CLOUD_SHA', 'PAPERCLIP_SKILL_DEFAULT_CADENCE_EVIDENCE', 'PAPERCLIP_SKILL_DEFAULT_CADENCE_STAGE',
'PAPERCLIP_TEST_POSTGRES_ADMIN_URL']) if (process.env[name]) env[name] = process.env[name];
env.PAPERCLIP_SKILL_DEFAULT_CADENCE_ACCEPTANCE = '1'; env.PAPERCLIP_MAINTENANCE_KEEP_ARTIFACTS = '1';
}
const files = controls ? ['server/src/__tests__/skill-default-cadence-prewire.test.ts', 'server/src/__tests__/maintenance-harness-resources.test.ts',
'server/src/__tests__/maintenance-postgres-proxy.test.ts', 'server/src/services/skill-source-idle-schedule.test.ts']
: ['server/src/__tests__/skill-source-default-cadence-acceptance.test.ts'];
const child = spawn(process.execPath, ['--require', guard, vitest, 'run', ...files,
'--maxWorkers=1', '--no-file-parallelism', ...(controls ? ['--testNamePattern=^(?!.*maintenance observer against PostgreSQL)'] : []), `--execArgv=--require=${guard}`], { cwd: root, env, stdio: 'inherit' });
for (const signal of ['SIGINT', 'SIGTERM']) process.on(signal, () => child.kill(signal));
child.once('error', error => { console.error(error); process.exitCode = 1; });
child.once('close', (code, signal) => {
const events = readFileSync(log, 'utf8').trim().split('\n').filter(Boolean).map(line => JSON.parse(line));
const violation = events.some(event => event.event === 'blocked' || event.unexpectedAttempts?.length);
if (violation) console.error('Default cadence guard recorded an unexpected attempt; caught exceptions do not qualify');
process.exitCode = violation ? 98 : signal ? 1 : code ?? 1;
});