mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-07 06:16:06 -10:00
perf(ci): let the public-install registry reuse warmed scoped metadata (#16838)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Every pull request runs the `ci / Build` check. This check was the slowest check in recent green runs: 432s in run 39254079691, 480s in run 39251304886 and 472s in run 39245233967. > - The slowest step in Build is "Verify built-in Grok from a clean public npm install" (162s and 207s). Its consumer `npm install` took 69s and 75s, even though a background step fills the npm cache first. > - The loopback registry in `scripts/public-install-registry.mjs` sends packages that it does not serve to npmjs with a 302 redirect. It escaped scoped names with `encodeURIComponent` (`%40scope%2Fname`). npm requests and caches the same metadata as `@scope%2fname`, and npm keys cached metadata by URL. > - Thus the consumer install could not use the scoped metadata that the warm-up had just put in the cache. It downloaded and parsed all 291 scoped packuments again (about 285 MB of full packument JSON) inside the 4-vCPU container. > - This pull request makes the redirect use the same escaping as npm. > - The benefit is a shorter Build check on every pull request. The installed bytes do not change. ## Linked Issues or Issue Description **Describe the bug** The consumer install in the public-install verifier does not use cached scoped package metadata. The warm-up step (`scripts/warm-public-install-npm-cache.mjs`) caches this metadata through direct npm requests. **Steps to reproduce** 1. Run `PAPERCLIP_PUBLIC_INSTALL_NPM_CACHE=/tmp/c node scripts/warm-public-install-npm-cache.mjs`. 2. Install the same dependency set through the loopback registry with `npm_config_cache=/tmp/c` and `--loglevel http`. 3. Count the `(cache miss)` lines. **Expected behavior** No cache misses for metadata that the warm-up already fetched. **Actual behavior** 291 cache misses. All of them are scoped packages that are requested as `https://registry.npmjs.org/%40scope%2Fname`. The cache holds them as `https://registry.npmjs.org/@scope%2fname`. Related: #16513 changes when the verifier waits for the warm-up. This pull request does not overlap with it, and the two changes add together. ## What Changed - `scripts/public-install-registry.mjs`: the 302 redirect for packages that the registry does not serve now uses npm's own escaping (`name.replace('/', '%2f')`, the same as npm-package-arg `escapedName`). The `packageName` check already allows only characters that need no other escaping. - `scripts/__tests__/public-install-registry.test.mjs`: the test now expects `@other%2fpkg` for a scoped redirect. A new assertion covers an unscoped redirect. ## Verification - `node --test scripts/__tests__/public-install-registry.test.mjs`: 3 of 3 pass. This includes the real npm install test that compares installed bytes from file installs and registry installs. - Local replay (Linux, npm 11.12.1). I filled a fresh cache with the warm-up script, then installed the same 62-dependency set (627 packages) through the registry: - Before: 291 cache misses, 28.3s. - After: 0 cache misses, 19.8s. - I also ran the real `scripts/public-install-registry.mjs` with a packed root package that depends on the full public dependency set. Result: 2 misses (only the loopback package and its tarball, as expected) and 1438 cache hits. - To compare on CI, look at `consumer_install_ms` in the `paperclip.public-npm-install.timing.v1` line of the Build log. Baseline: 68822 ms (run 39254079691) and 75175 ms (run 39251304886). ## Risks - Low risk. The redirect target is the same npmjs document. Only the URL escaping changes. npm still checks every tarball against the integrity in the packument. - If the npm cache holds stale scoped metadata, npm revalidates it the same way as it does for direct requests (`cache-control: max-age=300`). - The expected time saving comes from a local measurement. The real Build saving depends on the network and CPU of the runner. I expect about 10–20s per Build run. The CI timing line will confirm the number. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code as a Paperclip agent with tool use (shell, file edit). I did the root-cause analysis, the local replays and the code change with this model. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: devinfoley <[email protected]> Co-authored-by: Paperclip <[email protected]> Co-authored-by: Claude Opus 5.5 <[email protected]>
This commit is contained in:
2 files changed
+4
-2
No files matched your search
@@ -124,7 +124,9 @@ test('the registry exposes only known tarballs, redirects other packages to npm
|
||||
assert.equal(await (await fetch(metadata.versions['1.0.0'].dist.tarball)).text(), 'packed bytes');
|
||||
const external = await fetch(`${registry.url}/@other%2fpkg`, { redirect: 'manual' });
|
||||
assert.equal(external.status, 302);
|
||||
assert.equal(external.headers.get('location'), 'https://registry.npmjs.org/%40other%2Fpkg');
|
||||
assert.equal(external.headers.get('location'), 'https://registry.npmjs.org/@other%2fpkg');
|
||||
const unscoped = await fetch(`${registry.url}/other-pkg`, { redirect: 'manual' });
|
||||
assert.equal(unscoped.headers.get('location'), 'https://registry.npmjs.org/other-pkg');
|
||||
assert.equal((await fetch(`${registry.url}/-/900.tgz`)).status, 404);
|
||||
assert.equal((await fetch(`${registry.url}/some%2foutside%2ffile`)).status, 404);
|
||||
assert.equal((await fetch(`${registry.url}/@ci%2fpkg`, { method: 'POST' })).status, 405);
|
||||
|
||||
@@ -54,7 +54,7 @@ export async function startPackedRegistry(directory) {
|
||||
if (!packageName.test(path)) { response.writeHead(404).end(); return; }
|
||||
const entry = packages.get(path);
|
||||
if (!entry) {
|
||||
response.writeHead(302, { location: `https://registry.npmjs.org/${encodeURIComponent(path)}` }).end();
|
||||
response.writeHead(302, { location: `https://registry.npmjs.org/${path.replace('/', '%2f')}` }).end();
|
||||
return;
|
||||
}
|
||||
const { manifest, integrity } = entry;
|
||||
|
||||
Reference in new issue
Block a user