mirror of
https://github.com/QwenLM/qwen-code.git
synced 2026-11-07 06:19:43 -10:00
feat(skills): add file-backed archive installation kernel (#16421)
This commit is contained in:
1 parent
9b1ce46d77
commit
17ad9cdb04
12 files changed
+1542
-44
No files matched your search
@@ -0,0 +1,231 @@
|
||||
# Native Skill archive installation
|
||||
|
||||
[English](2026-10-29-skill-archive-installation.md) | [简体中文](2026-10-29-skill-archive-installation.zh-CN.md)
|
||||
|
||||
## Status and baseline
|
||||
|
||||
PR #15862 is being delivered as four dependent PRs, rebased onto main
|
||||
`7bd78683e63a50334fbcb124568101868048264c`.
|
||||
This document describes the complete design; each PR implements only its layer:
|
||||
|
||||
1. Archive kernel: bounded file intake and extraction, cancellable mutex waits,
|
||||
publication callbacks, backup preservation, and their tests. Existing HTTP
|
||||
routes still accept only the legacy sources at this stage.
|
||||
2. Daemon API: native upload and host-file routes, owner-scoped receipts,
|
||||
capability limits, runtime reconciliation, and route tests.
|
||||
3. TypeScript SDK: native transport, receipt lookup, public types, and transport
|
||||
tests. This layer depends on the daemon protocol from the preceding PR.
|
||||
4. Web Shell: File/Blob upload, uncertain-result lookup without retransmission,
|
||||
separate activation warnings, old-daemon fallback, and UI tests.
|
||||
|
||||
The stack preserves the implementation and prior review fixes at
|
||||
`780f1c6a08b493232adda97f235558121c4412b9`. Historical verification below is
|
||||
evidence for the original implementation; validation of each split is reported
|
||||
separately in its PR. The PRs must land in this order.
|
||||
|
||||
Implementation approved on 2026-10-29, with expanded limits restricted to the new raw/file sources. The implementation starts from main `06fbfc9951c0a0833e57eb84a9b3625bb84f2853`, which includes [PR #15030](https://github.com/QwenLM/qwen-code/pull/15030) at `2415b12329b1bb0ef1b5beb6bfc0bd0535b9ee7b`.
|
||||
|
||||
## Current behavior and remaining need
|
||||
|
||||
| Dimension | Current main | Remaining need |
|
||||
| ----------------- | ------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------- |
|
||||
| ZIP transport | Base64 JSON, at most 7,864,071 compressed bytes | Transfer a real 20 MiB archive without raising the shared JSON limit |
|
||||
| HTTP JSON body | 10 MiB | Keep this limit for existing JSON routes |
|
||||
| Extracted package | 64 MiB total, 32 MiB per file, 2000 files | Expand only the new raw/file archive sources; preserve all existing source policies |
|
||||
| Extraction | A complete compressed Buffer, followed by streamed entry writes and byte guards | Open a ZIP already on disk without buffering the entire archive |
|
||||
| Sources | GitHub URL, local folder, base64 ZIP | Add a daemon-host ZIP file reference |
|
||||
| Publication | Staging, validation, backup/rename/rollback, existing route mutex | Reuse the current path for new transports |
|
||||
| Identity | Existing name/path contract and manifest-name matching | Keep it unchanged in the first delivery |
|
||||
|
||||
Two caller situations require different entry points: a browser has a File that must be uploaded; an integration may already have uploaded the archive onto the daemon host. Requiring the second caller to read and base64-encode that file would repeat the transport work.
|
||||
|
||||
The required behavior is large-file transport with bounded memory. Maximum accepted package size and complete compatibility with an external installer are separate requirements. The first delivery will not claim that every package accepted by another system can be installed by Qwen.
|
||||
|
||||
## Reference lessons
|
||||
|
||||
The inspected Codex snapshot is `a29c7b97e88367044ae84c28c85451ec3e72af9f`.
|
||||
|
||||
| Reference | Useful lesson | Limit of the analogy |
|
||||
| ---------------------------- | ---------------------------------------------------- | ------------------------------------------------------------------------------------------ |
|
||||
| Python Skill installer | Acquire content before installation | It buffers the repository ZIP and does not establish a bounded-memory design |
|
||||
| Rust remote plugin installer | Account for compressed and expanded bytes separately | Its 100 MiB / 512 MiB numbers are not Qwen requirements; compressed data is still buffered |
|
||||
| Plugin sharing | Send bulk bytes separately from JSON metadata | A local daemon does not need the hosted upload-URL/finalize protocol |
|
||||
| Plugin store | Give staging and publication explicit ownership | It does not make a store rewrite necessary for a new ZIP input |
|
||||
|
||||
Existing application upload flows also show why an already-uploaded local file is a useful input. Product-specific chunking, object storage, publication, ownership policy, and error envelopes remain outside the native daemon protocol.
|
||||
|
||||
## Scope
|
||||
|
||||
This feature includes file-backed ZIP installation, raw HTTP upload, SDK and Web Shell integration, owned inputs, cancellable waiting, bounded attempt receipts, and focused publication/recovery safeguards. New archives use 200 MiB / 4096 entries; legacy-source limits, naming rules, GitHub fetching, folder copying, and the publisher structure remain unchanged.
|
||||
|
||||
Extending the new policy to legacy sources, redesigning identities or the publisher, cross-process locking, persistent transactions, automatic retries, and resumable uploads are outside this change.
|
||||
|
||||
## Installation contracts
|
||||
|
||||
### Already-uploaded ZIP
|
||||
|
||||
Extend the existing JSON source union:
|
||||
|
||||
```json
|
||||
{
|
||||
"name": "example-skill",
|
||||
"scope": "global",
|
||||
"attemptId": "c72732d3-7e2e-4b91-8c39-ec0f5ed45176",
|
||||
"source": { "type": "zip-file", "path": "/absolute/path/example.zip" }
|
||||
}
|
||||
```
|
||||
|
||||
The path belongs to the daemon host, not the browser. Apply the existing authentication, trust, runtime ownership, and name validation. Require an absolute regular file, reject a symlink source and special files, validate the opened handle, and retain compressed-byte bounds. Qwen borrows the source and never deletes or renames it.
|
||||
|
||||
Copy a borrowed ZIP into an installer-owned disk file before waiting for the mutation lock. Extract only from that copy. The caller must keep the source stable during acquisition; it no longer needs to retain it unchanged throughout lock waiting and extraction. See the lifecycle decisions below for the limits of copying and optional `source.sha256` verification.
|
||||
|
||||
### Browser ZIP
|
||||
|
||||
Use one raw request with `Content-Type: application/octet-stream`:
|
||||
|
||||
| Route | Ownership and activation |
|
||||
| ---------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------ |
|
||||
| `POST /workspace/config/skills/install-archive?name=...&attemptId=...` | Global configuration; preserve the existing global owner/trust resolver and reconciliation |
|
||||
| `POST /workspaces/:workspace/config/skills/install-archive?name=...&attemptId=...` | The selected trusted runtime, workspace scope; reconcile that runtime |
|
||||
|
||||
The route fixes scope. Resolve authentication, name, runtime, trust, content type, and admission before intake. Stream bytes into a private temporary file, enforcing the actual byte count with or without Content-Length. Reject HTTP content encoding. Unknown, untrusted, bootstrapping, draining, or removed runtimes retain their failure semantics and never fall back to the primary runtime.
|
||||
|
||||
After intake, use the same mutation mutex/key and installation path as existing requests. Keep the existing mutation result and activation behavior. The normal operation remains one synchronous POST, without upload sessions, a second commit request, resumable transfer, or a background-job service. Read-only attempt lookup confirms the outcome after a lost response.
|
||||
|
||||
The request owns and cleans its temporary file on every settled outcome. Do not use the generic workspace file-upload destination: it publishes a user file and has different scope and collision semantics. Reusing the shared SDK transport does not mean reusing a buffering HTTP parser.
|
||||
|
||||
### SDK and Web Shell
|
||||
|
||||
Add thin global and workspace archive-install methods. Generate an `attemptId` before sending new archive requests, retain it in client errors, and expose attempt lookup. Reuse existing binary-upload authentication, error handling, AbortSignal, and optional progress conventions. Preserve existing file-upload behavior.
|
||||
|
||||
Advertise `skill_archive_install`, `skill_zip_file_source`, and `maxSkillArchiveBytes` only where supported. Web Shell sends File/Blob directly when the raw capability is present. Otherwise, retain the 7,864,071-byte base64 path and report an upgrade requirement for larger input before uploading. Keep the current configuration/runtime split and activation handling.
|
||||
|
||||
The UI retains its busy state through installation and result lookup. Upload progress reaching 100% means intake has completed, not that installation succeeded. A lost response or timeout leaves the result unconfirmed until an attempt lookup supplies evidence. Do not automatically retry through another transport or treat an existing name/path as proof of this revision. Listing refresh remains useful for current inventory, independently of the historical attempt result.
|
||||
|
||||
### Approved source-policy and API boundaries
|
||||
|
||||
Only modern config installation routes accept `zip-file`; legacy runtime routes reject it with a config-route-required error. Old SDK runtime source types remain unchanged. Both new archive sources use 50 MiB compressed / 200 MiB retained expanded bytes / 4096 ZIP records. Ordinary resources have no separate cap below the remaining expanded budget. Validate all new-source entry paths/types before filtering, create retained files exclusively, then apply the 32 MiB limit to the effective root SKILL.md after the existing single-wrapper normalization and before reading it as text. Nested resource manifests remain ordinary resources. Do not introduce a full metadata planning framework.
|
||||
|
||||
SDK archive calls default to 130 seconds; timeoutMs=0 disables only the client timer. After uncertain completion, Web Shell makes one independent lookup with a 5-second timeout and offers a manual check if still running/unknown; never retransmit automatically. Committed content remains successful when activation fails, with a separate activation warning. Historical receipt reads follow config-read authorization and workspace identity, without entering the mutation coordinator or requiring the old generation to remain active. Removed/inaccessible owners never fall back to primary.
|
||||
|
||||
## Resource and lifecycle policy
|
||||
|
||||
| Budget | Fixed policy |
|
||||
| --------------------------------- | -------------------------------------------------------------------------------------------- |
|
||||
| New raw/file ZIP compressed bytes | 50 MiB |
|
||||
| Legacy base64 compressed bytes | 7,864,071 bytes, unchanged |
|
||||
| Expanded package | 200 MiB for new raw/file archives |
|
||||
| Effective root SKILL.md | 32 MiB, unchanged |
|
||||
| ZIP records | 4096 ZIP records for new sources, counted before filtering; legacy sources retain 2000 files |
|
||||
| Paths | Existing 512-character / 16-depth rules |
|
||||
| New local archive operations | At most two admitted per daemon; reject excess before intake |
|
||||
|
||||
50 MiB is the fixed transport ceiling, not a promise that every 50 MiB ZIP is installable. A ZIP must also satisfy all expanded-size, per-file, path, and manifest rules. Folder and GitHub policy remains unchanged.
|
||||
|
||||
Use one 120-second pre-publication budget covering intake/copying, lock waiting, and extraction; phases do not restart the clock. Check elapsed monotonic time again at publication admission. The SDK default is 130 seconds, but this is a client waiting budget, not proof of server rollback. Publication, rollback, and outstanding filesystem cleanup may outlast these budgets. The cancellation and receipt contract below explains that boundary.
|
||||
|
||||
A single owned archive plus candidate adds up to 250 MiB of temporary disk; two active operations add up to 500 MiB. Raw uploads are not recopied. Existing installations, backups, filesystem metadata, and crash leftovers are additional. Prefer passing an already-owned raw-upload file to the same internal file-opening extractor, without recopying it or exposing a public ownership-transfer flag. A borrowed file is copied once; its caller-owned original is outside the installer temporary-disk budget.
|
||||
|
||||
## Lifecycle decisions
|
||||
|
||||
### Borrowed-file changes
|
||||
|
||||
**Recommendation: an owned disk copy, normal change detection, and optional expected-content verification.** A read-only file descriptor binds the opened file identity, but another process can still modify or truncate that same file. A hard link has the same problem. Waiting for a lock while reading the borrowed ZIP later extends the exposure unnecessarily.
|
||||
|
||||
Open and validate a regular file handle without following the final symlink; check that the handle matches the validated input. Capture identity, size, and modification/change metadata, stream through a compressed-byte counter and SHA-256 hasher into a private exclusive file, then compare source metadata again. Fail on observed changes, a byte-count mismatch, or a limit violation. Close the writer and use only the owned copy for ZIP parsing and extraction. Once acquisition has finished, replacement or deletion of the original path must not affect this attempt; Qwen must never clean up that original path. A caller cannot infer completion of acquisition from elapsed time or upload progress: normally it retains the original until the request settles, unless server-reported phase evidence confirms acquisition has completed.
|
||||
|
||||
Copying is not an atomic snapshot of a concurrently written source. Before/after metadata checks detect ordinary races, not every hostile or undetectable edit. If the producer already knows the digest of the completed upload, it can supply optional `source.sha256`; reject a different acquired digest before extraction. This binds accepted bytes to the producer's expected content, not to a trusted author or a filesystem instant. Without that digest, the supported contract still requires a stable source during copying. The hash Qwen computes by itself identifies the acquired bytes and cannot prove what the producer intended.
|
||||
|
||||
| Option | Guarantee / cost | Decision |
|
||||
| ----------------------------------------------- | ---------------------------------------------------------------- | -------------------------------------------------------- |
|
||||
| Direct read-only handle | No copy; vulnerable to in-place edits throughout extraction | Not the default |
|
||||
| Hard link or advisory lock | Hard links share writes; locks require every writer to cooperate | Does not establish the required isolation |
|
||||
| Owned copy plus checks | Stable extraction input after acquisition; one bounded disk copy | Recommended |
|
||||
| Owned copy plus producer digest | Also verifies the expected archive bytes | Optional for integrations with an existing upload digest |
|
||||
| Filesystem snapshot or immutable upload service | Stronger source ownership, but platform/protocol dependencies | Separate future requirement |
|
||||
|
||||
The copy costs up to one 50 MiB local read/write and temporary file. Hash while copying, rather than performing another complete read. For raw HTTP input, the completed private upload file is already the owned input. This removes source lifetime dependence during potentially long lock waits.
|
||||
|
||||
Codex's remote bundle path first owns the downloaded bytes, then extracts into its own TempDir. Its store copies into staging and checks manifest bytes before publication. These are useful ownership patterns, but neither proves point-in-time consistency of a borrowed local ZIP or all files in a mutable source directory. [Remote acquisition](https://github.com/openai/codex/blob/a29c7b97e88367044ae84c28c85451ec3e72af9f/codex-rs/core-plugins/src/remote_bundle.rs#L257-L301), [store staging](https://github.com/openai/codex/blob/a29c7b97e88367044ae84c28c85451ec3e72af9f/codex-rs/core-plugins/src/store.rs#L555-L638).
|
||||
|
||||
### Cancellation while waiting for the lock
|
||||
|
||||
**Recommendation: add optional cancellation to the existing mutex wait, preserve its ordering, and establish one publication admission boundary.** Keep the existing registry/key shared by old and new installs/deletes. Do not introduce a second archive-only lock or move the current critical section as part of this work. Callers without a signal keep their behavior; the shared filesystem consumers need regression coverage. Immediate busy rejection would avoid queue cancellation but change existing waiting behavior; leaving a cancelled request queued would retain files/admission capacity and could execute unexpectedly. The additive waiter change is the recommended tradeoff.
|
||||
|
||||
A waiter cancelled before admission must promptly stop waiting, must never call the installation callback, and must detach its abort listener and release its owned input after real I/O stops. For the current Promise-tail implementation, a cancelled waiter's ordering link must remain behind its predecessor until that predecessor settles. The callback and package payload need not remain attached. Do not delete the key merely because the cancelled caller returned. With A holding the lock and B waiting, cancelling B must not allow C to overlap A. Cleanup of the map tail happens only once its ordering chain has actually settled and it is still the current tail.
|
||||
|
||||
Check the signal both before joining the queue and immediately after admission. During intake/extraction, cancellation stops actual streams and prevents further work; retain the lock and admission lease until active work settles. A race against the acquisition wait is valid only if the losing waiter is safely retired. Racing the entire installation Promise against a timer is insufficient because installation may continue after the client is told it was cancelled.
|
||||
|
||||
Immediately before the first destination-changing rename, synchronously check cancellation, deadline, and runtime eligibility and enter the publishing phase without an intervening await. Cancellation processed first prevents publication. Publication admitted first finishes commit or rollback without being interrupted by an HTTP abort or client deadline. Runtime-generation checks and filesystem failures still follow the existing consistency/rollback path; shielding is not permission to publish into a removed runtime.
|
||||
|
||||
| Phase | Cancellation semantics |
|
||||
| ---------------------------------- | --------------------------------------------------------------------------------- |
|
||||
| Intake / owned-copy preparation | Stop I/O, await closure, clean owned files; no publication |
|
||||
| Waiting for mutation lock | Retire this waiter; never invoke its callback; preserve other waiters' ordering |
|
||||
| Extracting / validating under lock | Stop real work, clean staging, then release the lock |
|
||||
| Publishing / restoring backup | Finish the consistent outcome; cancellation can no longer promise no installation |
|
||||
|
||||
A client AbortSignal expresses a cancellation request. Only server-observed disconnect/cancellation or its own deadline can enforce the fence; a proxy may delay disconnect propagation. A client-side AbortError alone is not cancellation confirmation. There is no separate cancellation endpoint; attempt lookup reports whether publication happened.
|
||||
|
||||
Codex's plugin synchronization gate has a separate 120-second acquisition timeout. Its enterprise-login flow explicitly races gate acquisition with cancellation and shares cancellation/commit admission state. Its goal writer keeps ownership of a lock through background I/O even if the RPC disappears. The latter two are different subsystems, cited as lifecycle patterns, not plugin API guarantees. [Plugin gate](https://github.com/openai/codex/blob/a29c7b97e88367044ae84c28c85451ec3e72af9f/codex-rs/core-plugins/src/manager.rs#L1723-L1735), [cancellable wait](https://github.com/openai/codex/blob/a29c7b97e88367044ae84c28c85451ec3e72af9f/codex-rs/app-server/src/request_processors/account_processor/enterprise_login.rs#L161-L174), [commit admission](https://github.com/openai/codex/blob/a29c7b97e88367044ae84c28c85451ec3e72af9f/codex-rs/app-server/src/request_processors/account_processor/enterprise_login.rs#L262-L293), [I/O ownership](https://github.com/openai/codex/blob/a29c7b97e88367044ae84c28c85451ec3e72af9f/codex-rs/app-server/src/request_processors/thread_goal_user_context.rs#L24-L50).
|
||||
|
||||
### Confirming the result after a timeout
|
||||
|
||||
**Recommendation: a client-generated attempt ID and a bounded, in-memory receipt per attempt.** This adds real protocol scope, but ordinary Skill inventory cannot distinguish an old installation, this attempt, and a later overwrite. A synchronous POST remains the normal API; receipts are for reconciliation after an uncertain response, not a task scheduler.
|
||||
|
||||
For new raw/file calls, generate a UUID `attemptId` before transmission: a JSON field for file sources and a query parameter for raw upload. The SDK preserves it even when no response arrives. After authorization and admission, reserve a receipt before reading the body. Bind it to the resolved global/workspace owner, runtime generation where applicable, target name, and acquired archive SHA-256 once available. Possession of an ID grants no additional access.
|
||||
|
||||
Read-only lookup routes are `GET /workspace/config/skills/install-attempts/:attemptId` and `GET /workspaces/:workspace/config/skills/install-attempts/:attemptId`. They obey the same ownership, authentication, and visibility boundary. An unavailable/removed runtime must not fall back to another owner. Query with a new bounded read request, not the already-aborted installation signal.
|
||||
|
||||
| Receipt result | What the client can conclude |
|
||||
| ------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `running` | The attempt is still active; include its phase, and do not resubmit |
|
||||
| `committed` | This attempt completed publication of the recorded archive bytes; runtime activation and current inventory remain separate |
|
||||
| `not_committed` | The attempt never published, or restoration of the prior state was confirmed; include the failure/cancellation reason |
|
||||
| `recovery_required` | Publication/rollback failed to establish a confirmed restored state; do not claim cancellation or retry automatically |
|
||||
| Unknown / no accessible record | No conclusion about success or failure; the request may not have arrived, the record may have expired, or the daemon/runtime may have changed |
|
||||
|
||||
Record `committed` at the installer’s final commit boundary, after its last possible generation-driven rollback decision, before returning through service/HTTP layers. Do not derive it from an HTTP 200 alone. Later runtime, activation, or refresh failures must not relabel committed content as absent; retain the committed outcome alongside those errors. Best-effort backup cleanup can continue while ownership of active work is retained. A receipt is historical: another install, deletion, or external edit can change the current directory afterwards. The archive digest identifies the input, not a verified hash of the current expanded tree. A checked lookup of current inventory answers a different question.
|
||||
|
||||
Keep active receipts until work settles; retain terminal outcomes for 10 minutes and a total capacity of 256 receipts. Reserve capacity before mutation and reject new attempts when full rather than evict an unexpired promised receipt. These are fixed defaults, not public configuration knobs. Duplicate IDs within the same owner/retention window must never execute again; return a conflict and direct the caller to lookup, even if a different payload is supplied. This is bounded duplicate suppression, not durable exactly-once execution or an automatic replay contract. New raw/file calls require the ID; existing-source requests retain their old contract.
|
||||
|
||||
A daemon restart loses these receipts. Unknown must stay unknown, never become “failed, safe to retry.” Do not add automatic retry, cross-restart deduplication, or persistence in the first delivery. A separate receipt file written after rename leaves a crash gap; a marker inside the candidate changes package metadata and does not cover later legacy/external writers; a durable transaction journal is a separate design. If recovery across restart is mandatory, this bounded receipt design is insufficient and that stronger requirement must be designed before promising it.
|
||||
|
||||
The focused rollback change distinguishes confirmed restoration from `recovery_required` and preserves recoverable backups across later installs. Failure to clean an unpublished candidate must not prevent restoration of the previous installation. Tests cover lost responses, late abort during commit, failed restore, later overwrite, duplicate IDs, receipt capacity/expiry, and restart/owner disappearance.
|
||||
|
||||
Codex defines `installAttemptId` as a client-generated correlation identifier and forwards it to the remote install request, checking the returned plugin identity and enabled state. The inspected client code does not establish a query-by-attempt endpoint, backend idempotency, or crash-durable results. The Qwen receipt above is this design’s addition, not a Codex feature claim. [Protocol definition](https://github.com/openai/codex/blob/a29c7b97e88367044ae84c28c85451ec3e72af9f/codex-rs/app-server-protocol/src/protocol/v2/plugin.rs#L956-L975), [remote request](https://github.com/openai/codex/blob/a29c7b97e88367044ae84c28c85451ec3e72af9f/codex-rs/core-plugins/src/remote.rs#L1641-L1683).
|
||||
|
||||
## Reuse and correctness boundaries
|
||||
|
||||
The current ZIP opening helper is the Buffer-specific boundary. The remaining extraction path consumes a ZIP reader, so a file-opening branch reuses lazy entry traversal, path checks, declared-size checks, actual-byte guards, wrapper flattening, manifest checks, and publication. Keep the owned PassThrough for stream teardown and the existing inflater.
|
||||
|
||||
Implementation validation found that yauzl 2.10's fd-slicer dependency marks a file stream destroyed before its buffered compressed bytes drain on current Node versions. A valid 70 MiB entry stalled after 60,071,690 output bytes with 58,163 compressed bytes stranded; the same archive completed through 2.10's Buffer input and yauzl 3.4's file input. The CLI therefore upgrades to yauzl 3.4, already present elsewhere in the repository, with the legacy installer suite retained as regression coverage. No custom file reader or inflater is introduced.
|
||||
|
||||
Do not move the mutation lock, add cross-process locking, rewrite folder copying, or build an archive framework merely to add the new input. New routes must participate in the existing lock; this design does not claim cross-daemon serialization.
|
||||
|
||||
A failed rollback reports `recovery_required` with retained paths. Subsequent installs must preserve abandoned backups; successful installs still remove the backup they created. This protection applies to every source, including legacy base64, directories and GitHub: after receipt expiry or process restart, an installer cannot safely distinguish disposable backups from the only recoverable copy. Fault-injection tests cover failed restoration followed by another install. No automatic recovery program or publisher rewrite is introduced.
|
||||
|
||||
## Validation and acceptance
|
||||
|
||||
The completed baseline used the global `qwen 0.22.0`, not the merged main bundle. Small ZIP and folder requests succeeded with identical bytes; a real 20 MiB ZIP in a 27,962,121-byte JSON request received 413; zip-file received 400 and raw upload received 404. That version lacks the modern configuration routes, so these results do not verify their current-main behavior. Temporary inputs were retained and the test daemon was cleaned up.
|
||||
|
||||
The unmodified main bundle at the pinned baseline was also verified: modern config routes installed small ZIP and folder inputs with identical bytes; the real 20 MiB base64 request returned 413, `zip-file` returned 400, and raw upload returned 404. This is the baseline for the implementation checks below:
|
||||
|
||||
- A real 20 MiB archive installed through both raw and file inputs, with byte-for-byte comparison and no whole-archive Buffer on the new paths.
|
||||
- New-source 50 MiB compressed / 200 MiB expanded / 4096-entry boundaries and effective-root 32 MiB manifest checks; preserve 64 MiB / 32 MiB / 2000-file boundaries for legacy sources.
|
||||
- Shared old/new path behavior for traversal, symlinks, duplicates, wrapper directories, manifests, GitHub fallback, and base64 compatibility.
|
||||
- Interrupted intake, malformed declared sizes, staging write failure, timeout while waiting for a lock, closed runtime generation, and replacement failure; real work settles, temporary files close/clean up, and the old installation is preserved where the operation has not committed.
|
||||
- Correct global and selected-runtime activation; no primary fallback in rejected workspace states.
|
||||
- SDK fetch/XHR behavior, old-daemon fallback, and unchanged generic file upload.
|
||||
- Measured memory and temporary disk during two large operations, plus rejection of a third.
|
||||
|
||||
The new 4096 limit counts every explicit ZIP record, including directories and ignored metadata; implicit parent directories are not counted again. Expanded bytes count only retained files. Keeping this policy on the new sources avoids tightening legacy directory layouts. Name compatibility and managed ownership need separate acceptance, not a claim based on upload success.
|
||||
|
||||
## Verification boundaries
|
||||
|
||||
Use the fixed policies above without adding configuration knobs. Measure memory and temporary disk, exercise cancellation and recovery faults, and run independent real HTTP and browser acceptance against the built bundle. Record the actual operating system and browser; source inspection and unit tests do not imply cross-platform or browser acceptance. The E2E report records execution evidence separately from this design.
|
||||
|
||||
## Sources
|
||||
|
||||
- [Merged Qwen installer](https://github.com/QwenLM/qwen-code/blob/2415b12329b1bb0ef1b5beb6bfc0bd0535b9ee7b/packages/cli/src/serve/workspace-skill-management.ts), [Skill routes](https://github.com/QwenLM/qwen-code/blob/2415b12329b1bb0ef1b5beb6bfc0bd0535b9ee7b/packages/cli/src/serve/routes/workspace-skills.ts), [JSON middleware](https://github.com/QwenLM/qwen-code/blob/2415b12329b1bb0ef1b5beb6bfc0bd0535b9ee7b/packages/cli/src/serve/server/error-handlers.ts).
|
||||
- [Codex Python installer](https://github.com/openai/codex/blob/a29c7b97e88367044ae84c28c85451ec3e72af9f/codex-rs/skills/src/assets/samples/skill-installer/scripts/install-skill-from-github.py), [remote bundle](https://github.com/openai/codex/blob/a29c7b97e88367044ae84c28c85451ec3e72af9f/codex-rs/core-plugins/src/remote_bundle.rs), [plugin store](https://github.com/openai/codex/blob/a29c7b97e88367044ae84c28c85451ec3e72af9f/codex-rs/core-plugins/src/store.rs), [plugin sharing](https://github.com/openai/codex/blob/a29c7b97e88367044ae84c28c85451ec3e72af9f/codex-rs/core-plugins/src/remote/share.rs).
|
||||
@@ -0,0 +1,225 @@
|
||||
# 原生 Skill 归档安装
|
||||
|
||||
[English](2026-10-29-skill-archive-installation.md) | [简体中文](2026-10-29-skill-archive-installation.zh-CN.md)
|
||||
|
||||
## 状态与基线
|
||||
|
||||
PR #15862 将按依赖顺序拆为四个 PR,基于 main
|
||||
`7bd78683e63a50334fbcb124568101868048264c`。本文描述完整设计,每个 PR 仅实现对应层:
|
||||
|
||||
1. 归档内核:有界文件接收和解压、可取消的互斥锁等待、发布回调、备份保留及对应测试。
|
||||
此阶段现有 HTTP 路由仍只接受旧来源。
|
||||
2. Daemon API:原生上传和主机文件路由、按所有者隔离的回执、能力限额、运行时刷新及路由测试。
|
||||
3. TypeScript SDK:原生传输、回执查询、公开类型及传输测试,依赖前一 PR 提供的 daemon 协议。
|
||||
4. Web Shell:File/Blob 上传、不重传的不确定结果查询、独立激活警告、旧 daemon 回退及 UI 测试。
|
||||
|
||||
拆分保留 `780f1c6a08b493232adda97f235558121c4412b9` 的实现和此前审查修复。
|
||||
下文历史验证属于原始实现的证据;各拆分版本的验证会分别记录在对应 PR 中。
|
||||
四个 PR 必须按此顺序合入。
|
||||
|
||||
方案于 2026-10-29 批准实施,扩容仅限新的 raw/文件来源。实施从 main `06fbfc9951c0a0833e57eb84a9b3625bb84f2853` 开始,其中已包含 [PR #15030](https://github.com/QwenLM/qwen-code/pull/15030) 的提交 `2415b12329b1bb0ef1b5beb6bfc0bd0535b9ee7b`。
|
||||
|
||||
## 现状与剩余需求
|
||||
|
||||
| 维度 | 当前 main | 剩余需求 |
|
||||
| ---------------- | -------------------------------------------- | ----------------------------------------------- |
|
||||
| ZIP 传输 | Base64 JSON,压缩内容最多 7,864,071 字节 | 不增大全局 JSON 上限,也能传输真实 20 MiB 归档 |
|
||||
| HTTP JSON 请求体 | 10 MiB | 保持现有 JSON 路由的限制 |
|
||||
| 解压后的包 | 总量 64 MiB、单文件 32 MiB、2000 个文件 | 仅扩容新的 raw/文件归档来源,保持所有旧来源政策 |
|
||||
| 解压 | 压缩包完整 Buffer,随后逐条目流式写入并计数 | 直接打开磁盘 ZIP,避免整个归档进入内存 |
|
||||
| 来源 | GitHub URL、本地目录、base64 ZIP | 增加 daemon 主机上的 ZIP 文件引用 |
|
||||
| 发布 | 暂存、验证、备份/重命名/回滚,现有路由互斥锁 | 新传输复用既有路径 |
|
||||
| 身份 | 现有名称/路径契约,manifest 名称必须匹配 | 首次交付保持不变 |
|
||||
|
||||
两种调用场景需要不同入口:浏览器只有需要上传的 File;集成方可能已经把 ZIP 上传到了 daemon 所在主机。要求后者再读取并编码为 base64,会重复传输工作。
|
||||
|
||||
要解决的是内存有界的大文件传输。最大可接受包规模、与外部安装器的完整兼容是另外两个需求。首次交付不声称其他系统接受的所有包都能由 Qwen 安装。
|
||||
|
||||
## 参考实现的启发
|
||||
|
||||
已检查的 Codex 快照为 `a29c7b97e88367044ae84c28c85451ec3e72af9f`。
|
||||
|
||||
| 参考实现 | 可借鉴的原则 | 类比边界 |
|
||||
| ------------------- | ------------------------------------ | ------------------------------------------------------ |
|
||||
| Python Skill 安装器 | 先获取内容,再安装 | 它会缓冲整个仓库 ZIP,不能据此证明内存有界 |
|
||||
| Rust 远程插件安装器 | 压缩量与展开量分别计数 | 100 MiB / 512 MiB 不是 Qwen 的需求;压缩数据仍被缓冲 |
|
||||
| 插件分享 | 大块二进制内容与 JSON 元数据分开传输 | 本地 daemon 不需要照搬托管服务的上传 URL/finalize 协议 |
|
||||
| 插件存储 | 明确暂存与发布阶段的所有权 | 新增 ZIP 输入不要求重写整个存储层 |
|
||||
|
||||
已有应用上传流程也说明,直接接收已落盘文件有实际价值。产品自己的分片、对象存储、发布、归属策略和错误封装继续留在原生 daemon 协议之外。
|
||||
|
||||
## 范围
|
||||
|
||||
本功能包含基于磁盘文件的 ZIP 安装、raw HTTP 上传、SDK 与 Web Shell 接入、自有输入、可取消等待、有界尝试回执,以及局部发布/恢复保护。新归档采用 200 MiB / 4096 条目;旧来源限额、名称规则、GitHub 获取、目录复制与发布结构保持不变。
|
||||
|
||||
将新政策扩展到旧来源、重新设计身份或发布器、跨进程锁、持久化事务、自动重试和断点续传不在本次范围内。
|
||||
|
||||
## 安装契约
|
||||
|
||||
### 已上传的 ZIP
|
||||
|
||||
扩展现有 JSON 来源联合类型:
|
||||
|
||||
```json
|
||||
{
|
||||
"name": "example-skill",
|
||||
"scope": "global",
|
||||
"attemptId": "c72732d3-7e2e-4b91-8c39-ec0f5ed45176",
|
||||
"source": { "type": "zip-file", "path": "/absolute/path/example.zip" }
|
||||
}
|
||||
```
|
||||
|
||||
路径属于 daemon 主机,不是浏览器本地路径。沿用现有认证、信任、运行时归属和名称验证。要求绝对路径的普通文件,拒绝作为来源的符号链接及特殊文件,验证打开的句柄,并限制压缩输入字节数。Qwen 借用该来源,不删除或重命名它。
|
||||
|
||||
在等待变更锁之前,将借用的 ZIP 复制到安装器自己的磁盘文件,解压只读取该副本。调用方需要在获取期间保持来源稳定,不再需要贯穿等待锁和解压全过程。下方生命周期决策说明复制的保证边界,以及可选的 `source.sha256` 验证。
|
||||
|
||||
### 浏览器 ZIP
|
||||
|
||||
使用一次 raw 请求,`Content-Type: application/octet-stream`:
|
||||
|
||||
| 路由 | 归属与激活 |
|
||||
| ---------------------------------------------------------------------------------- | -------------------------------------------------- |
|
||||
| `POST /workspace/config/skills/install-archive?name=...&attemptId=...` | 全局配置;保留现有全局所有者/信任解析和协调更新 |
|
||||
| `POST /workspaces/:workspace/config/skills/install-archive?name=...&attemptId=...` | 选中的可信运行时,workspace 范围;协调更新该运行时 |
|
||||
|
||||
路由固定 scope。在接收正文前完成认证、名称、运行时、信任、内容类型和并发准入检查。流式写入私有临时文件,无论有无 Content-Length 都按实际字节计数。拒绝 HTTP 内容编码。未知、不可信、启动中、排空中或已移除的运行时沿用原失败语义,绝不回退到 primary。
|
||||
|
||||
接收完成后进入现有安装请求使用的同一互斥锁、同一 key 和安装流程。沿用现有变更结果与激活行为。正常操作仍是一次同步 POST,不需要上传会话、第二次提交请求、断点续传或后台任务服务。只读尝试查询用于在响应丢失后确认操作结果。
|
||||
|
||||
请求持有临时文件,并在操作确实结束后的所有结果下清理它。不借用通用 workspace 文件上传的目标目录:该接口发布的是用户文件,归属和重名处理不同。复用 SDK 的传输逻辑不等于复用会缓冲整个请求体的 HTTP 解析器。
|
||||
|
||||
### SDK 与 Web Shell
|
||||
|
||||
为全局和 workspace 安装增加薄封装方法。发送新归档请求前生成 `attemptId`,在客户端错误中保留它,并提供尝试查询。复用已有二进制上传的认证、错误处理、AbortSignal 和可选进度约定,保持原文件上传行为不变。
|
||||
|
||||
仅在支持的服务端暴露 `skill_archive_install`、`skill_zip_file_source` 和 `maxSkillArchiveBytes`。Web Shell 在支持 raw 能力时直接发送 File/Blob,否则保留 7,864,071 字节的 base64 路径,超限时在上传前提示升级。保持现有配置/运行时划分及激活处理。
|
||||
|
||||
UI 在安装和查询结果期间保持忙碌状态。上传进度达到 100% 只代表接收完成,不代表安装成功。丢失响应或超时后,结果仍待确认,直到尝试查询提供证据。不自动换传输方式重试,也不把已有同名目录/路径当作本次版本已安装的证明。列表刷新用于了解当前清单,与历史尝试结果分别处理。
|
||||
|
||||
### 已批准的来源政策与 API 边界
|
||||
|
||||
只有现代 config 安装路由接受 `zip-file`;旧 runtime 路由返回要求使用 config 路由的错误。SDK 旧 runtime 来源类型保持不变。两种新归档来源采用 50 MiB 压缩 / 200 MiB 实际保留展开字节 / 4096 ZIP 记录。普通资源只受剩余展开总量约束。新来源条目在过滤前验证路径/类型,保留文件使用独占创建;在现有单层 wrapper 归一化后、读取文本前,对有效根 SKILL.md 应用 32 MiB 限额。嵌套资源 manifest 仍是普通资源。不引入全量元数据规划框架。
|
||||
|
||||
SDK 归档调用默认 130 秒;timeoutMs=0 仅禁用客户端计时器。结果不确定时,Web Shell 用 5 秒超时独立查询一次,仍在进行/未知则提供手动检查,绝不自动重传。激活失败不改变已提交内容的成功结果,另附激活警告。历史回执查询遵循配置读取的授权和 workspace 身份,不进入变更 coordinator,也不要求旧 generation 仍活跃。已移除或不可访问的所有者不回退 primary。
|
||||
|
||||
## 资源与生命周期政策
|
||||
|
||||
| 限制 | 固定政策 |
|
||||
| ---------------------------- | ---------------------------------------------------- |
|
||||
| 新 raw/文件 ZIP 的压缩字节数 | 50 MiB |
|
||||
| 旧 base64 的压缩字节数 | 7,864,071 字节,不变 |
|
||||
| 展开后的包总量 | 新 raw/文件归档为 200 MiB |
|
||||
| 有效根 SKILL.md | 32 MiB,不变 |
|
||||
| ZIP 条目数 | 新来源过滤前计数 4096 ZIP 记录;旧来源仍为 2000 文件 |
|
||||
| 路径 | 现有长度 512 / 深度 16 规则 |
|
||||
| 新本地归档操作 | 每个 daemon 最多同时准入两个;超额在接收前拒绝 |
|
||||
|
||||
50 MiB 是固定传输上限,不保证所有 50 MiB ZIP 都能安装。归档仍必须满足展开总量、单文件、路径和 manifest 规则。目录与 GitHub 政策保持不变。
|
||||
|
||||
采用统一的发布前 120 秒预算,涵盖接收/复制、等待锁和解压,各阶段不重新计时。进入发布时再次检查单调时钟的已用时间。SDK 默认等待时间为 130 秒,但它是客户端等待预算,不能证明服务端已回滚。发布、回滚及尚未结束的文件系统清理可能超过这些预算;下方取消与回执契约明确该边界。
|
||||
|
||||
一个自有归档及候选目录最多新增 250 MiB 临时磁盘,两个操作最多 500 MiB。raw 上传不再复制一遍。已有安装、备份、文件系统元数据和崩溃遗留另计。优先让已由请求持有的 raw 临时文件直接进入相同的内部文件解压流程,避免再次复制,也不暴露公共所有权转移开关。借用文件复制一次;调用方持有的原文件不计入安装器临时磁盘预算。
|
||||
|
||||
## 生命周期决策
|
||||
|
||||
### 借用文件被修改
|
||||
|
||||
**推荐:自有磁盘副本、常规变化检测,以及可选的预期内容校验。** 只读文件描述符绑定的是已打开文件的身份,其他进程仍能修改或截断同一个文件。硬链接也有同样问题。等到拿锁之后才读取借用 ZIP,会无谓延长暴露时间。
|
||||
|
||||
不跟随最终符号链接,打开并验证普通文件句柄,确认句柄与已验证输入一致。记录身份、大小和修改/变更元数据,经压缩字节计数及 SHA-256 哈希流式复制到私有、独占创建的文件,再次比较来源元数据。发现变化、读取字节数不匹配或超限时失败。关闭写句柄,此后 ZIP 解析和解压只使用自有副本。获取完成后,原路径被替换或删除不应影响本次尝试;Qwen 绝不能清理该原路径。调用方不能从经过的时间或上传进度推断获取已完成:通常应等请求结束再清理原文件,除非服务端报告的阶段已确认获取完成。
|
||||
|
||||
复制不等于对并发写入来源做原子快照。前后元数据检查能识别普通竞争,不能识别所有恶意或不可见修改。如果生产者已知完整上传的摘要,可以提供可选的 `source.sha256`;获取摘要不一致时,在解压前拒绝。这能把接受的字节绑定到生产者预期内容,但不能证明作者可信或对应某个文件系统瞬间。没有预期摘要时,支持契约仍要求复制期间来源稳定。Qwen 自己计算的哈希只能标识实际获取的字节,不能证明生产者原本想安装什么。
|
||||
|
||||
| 选项 | 保证与代价 | 决策 |
|
||||
| ---------------------------- | -------------------------------------------- | ---------------------------------- |
|
||||
| 直接只读句柄 | 无复制;整个解压期间都可能受原地修改影响 | 不作为默认 |
|
||||
| 硬链接或协作式锁 | 硬链接共享写入;锁要求所有写入者配合 | 不能建立所需隔离 |
|
||||
| 自有副本加检查 | 获取完成后解压输入稳定;增加一次有界磁盘复制 | 推荐 |
|
||||
| 自有副本加生产者摘要 | 额外验证归档符合预期字节 | 对已有上传摘要的集成方提供可选能力 |
|
||||
| 文件系统快照或不可变上传服务 | 更强来源所有权,但引入平台/协议依赖 | 独立的后续需求 |
|
||||
|
||||
复制增加最多一份 50 MiB 本地读写和临时文件。复制时同时算哈希,不再整包读取一遍。raw HTTP 输入完成后,私有上传文件本身已经是自有输入。这样消除了漫长等待锁期间对原文件生命周期的依赖。
|
||||
|
||||
Codex 的远程包路径先持有已下载字节,再解压进自有 TempDir;其存储层复制到暂存目录,并在发布前比较 manifest 字节。这些所有权方式值得参考,但不能证明借用本地 ZIP 或可变来源目录中所有文件具有某一时刻的一致性。[远程获取](https://github.com/openai/codex/blob/a29c7b97e88367044ae84c28c85451ec3e72af9f/codex-rs/core-plugins/src/remote_bundle.rs#L257-L301)、[存储暂存](https://github.com/openai/codex/blob/a29c7b97e88367044ae84c28c85451ec3e72af9f/codex-rs/core-plugins/src/store.rs#L555-L638)。
|
||||
|
||||
### 等待锁时取消
|
||||
|
||||
**推荐:为现有互斥锁等待增加可选取消能力,保持串行顺序,并设立一次发布准入边界。** 沿用新旧安装/删除共享的 registry/key。本次不另设归档专用锁,也不搬动现有临界区。不传 signal 的调用方保持原行为;共用该类的文件系统操作需要回归覆盖。直接返回忙碌可以省去排队取消,但会改变现有等待行为;让已取消请求继续排队则占用文件/准入名额,并可能意外执行。推荐付出小幅等待逻辑改动,保留既有行为。
|
||||
|
||||
在准入前被取消的等待者应及时退出等待,绝不调用安装回调,移除 abort 监听,并在真实 I/O 结束后释放自有输入。对当前 Promise-tail 实现,被取消等待者的顺序连接必须留在前驱之后,直到前驱结束;安装回调和包内容不必继续挂在链上。不能因为被取消的调用已经返回,就删除对应 key。A 持锁、B 等待时取消 B,不得让 C 与 A 并发。只有顺序链真正结束且仍是当前 tail,才能清理 map 中的 tail。
|
||||
|
||||
入队前和获准执行后都检查 signal。接收/解压期间,取消应停止实际流并阻止后续工作;在实际工作结束前继续持有锁和并发准入名额。只对等待锁做竞速,并正确撤销落败等待者,可以成立;对整个安装 Promise 与计时器做竞速不够,因为客户端被告知取消后,安装可能仍继续。
|
||||
|
||||
第一次修改目标目录的 rename 之前,同步检查取消、截止时间和运行时资格,中间不经过 await 就进入发布阶段。先处理取消则不得发布;先获准发布则完成提交或回滚,不受 HTTP abort 或客户端截止时间打断。运行时 generation 检查和文件系统失败仍走既有一致性/回滚路径;保护发布阶段不等于允许向已移除运行时发布。
|
||||
|
||||
| 阶段 | 取消语义 |
|
||||
| ------------------- | -------------------------------------------------- |
|
||||
| 接收 / 自有副本准备 | 停止 I/O,等待关闭,清理自有文件;不发布 |
|
||||
| 等待变更锁 | 撤销本等待者,绝不调用其回调,保持其他等待者的顺序 |
|
||||
| 持锁解压 / 验证 | 停止实际工作、清理暂存,再释放锁 |
|
||||
| 发布 / 恢复备份 | 完成一致的结果;取消不再能承诺没有安装 |
|
||||
|
||||
客户端 AbortSignal 表达的是取消请求。只有服务端观察到的断连/取消或自己的截止时间,才能阻止发布;代理可能延迟传播断连。客户端 AbortError 本身不是取消确认。首版不增加独立取消接口,通过尝试查询报告是否已发布。
|
||||
|
||||
Codex 插件同步 gate 对获取锁设置了独立的 120 秒超时。它的企业登录流程明确对获取 gate 与取消做竞速,并共享取消/提交准入状态。它的目标写入器即使 RPC 消失,也继续持锁完成后台 I/O。后两者属于不同子系统,仅作为生命周期设计参考,不代表插件 API 的保证。[插件 gate](https://github.com/openai/codex/blob/a29c7b97e88367044ae84c28c85451ec3e72af9f/codex-rs/core-plugins/src/manager.rs#L1723-L1735)、[可取消等待](https://github.com/openai/codex/blob/a29c7b97e88367044ae84c28c85451ec3e72af9f/codex-rs/app-server/src/request_processors/account_processor/enterprise_login.rs#L161-L174)、[提交准入](https://github.com/openai/codex/blob/a29c7b97e88367044ae84c28c85451ec3e72af9f/codex-rs/app-server/src/request_processors/account_processor/enterprise_login.rs#L262-L293)、[I/O 所有权](https://github.com/openai/codex/blob/a29c7b97e88367044ae84c28c85451ec3e72af9f/codex-rs/app-server/src/request_processors/thread_goal_user_context.rs#L24-L50)。
|
||||
|
||||
### 超时后确认结果
|
||||
|
||||
**推荐:客户端生成尝试 ID,每次尝试保留一份有界内存回执。** 这确实增加协议范围,但普通 Skill 清单无法区分旧安装、本次尝试和后续覆盖。正常 API 仍是同步 POST;回执用于响应不确定后的核对,不是任务调度系统。
|
||||
|
||||
新 raw/文件调用在传输前生成 UUID `attemptId`:文件来源放在 JSON 字段中,raw 上传放在 query 参数中。SDK 即使没收到响应也保留它。认证和准入后,在接收正文之前预留回执。回执绑定已解析的全局/workspace 所有者、适用时的运行时 generation、目标名称,以及获取后才有的归档 SHA-256。持有 ID 不增加任何访问权限。
|
||||
|
||||
只读查询路由为 `GET /workspace/config/skills/install-attempts/:attemptId` 和 `GET /workspaces/:workspace/config/skills/install-attempts/:attemptId`,遵守相同归属、认证与可见性边界。运行时不可用或已移除时,不得回退到其他所有者。查询使用新的、有超时限制的只读请求,不复用已经 abort 的安装 signal。
|
||||
|
||||
| 回执结果 | 客户端可以得出的结论 |
|
||||
| --------------------- | -------------------------------------------------------------------- |
|
||||
| `running` | 本次尝试仍活跃;包含阶段,不要重新提交 |
|
||||
| `committed` | 本次尝试已完成所记录归档字节的发布;运行时激活和当前清单另行判断 |
|
||||
| `not_committed` | 本次从未发布,或已确认恢复此前状态;附失败/取消原因 |
|
||||
| `recovery_required` | 发布/回滚未能确认恢复状态;不能声称已取消,也不自动重试 |
|
||||
| 未知 / 没有可访问记录 | 无法判断成败;请求可能未到达、记录可能过期,或 daemon/运行时发生变化 |
|
||||
|
||||
在安装器最终提交边界、最后一个可能导致 generation 回滚的决策之后、返回 service/HTTP 层之前记录 `committed`,不能仅从 HTTP 200 推导。后续运行时、激活或刷新失败不得把已提交内容标成不存在,应保留提交结果并附相应错误。尽力执行的备份清理可以继续,同时保留对尚未结束工作的所有权。回执表达历史事实:后来的安装、删除或外部编辑仍可能改变当前目录。归档摘要标识输入,不是当前展开目录的已验证哈希。读取当前清单回答的是另一个问题。
|
||||
|
||||
活跃回执保留到工作结束;终态保留 10 分钟,总容量 256 份。变更前预留容量,满时拒绝新尝试,不驱逐已承诺保留且尚未过期的记录。这些是固定默认值,不是公开配置开关。同一所有者/保留窗口内的重复 ID 绝不再次执行,即使提交了不同内容,也返回冲突并引导查询。这是有界的重复抑制,不是持久化的恰好执行一次或自动重放契约。新 raw/文件调用要求 ID;原有来源请求保持原契约。
|
||||
|
||||
daemon 重启会丢失这些回执。未知必须保持未知,不能变成“失败,可以安全重试”。首版不做自动重试、跨重启去重或持久化。rename 后另写回执文件存在崩溃窗口;在候选目录中放标记会改变包元数据,也覆盖不了后续旧路径/外部写入;持久化事务日志是独立设计。如果必须支持重启后的结果恢复,这个有界回执方案不够,需要先设计更强保证再作承诺。
|
||||
|
||||
局部回滚改动区分已确认恢复与 `recovery_required`,并确保后续安装仍保留可恢复备份。未发布候选目录的清理失败不能阻止恢复旧安装。测试覆盖响应丢失、提交中的迟到 abort、恢复失败、后续覆盖、重复 ID、回执容量/过期,以及重启/所有者消失。
|
||||
|
||||
Codex 把 `installAttemptId` 定义为客户端生成的关联标识,并将其转交远端安装请求,同时检查返回的插件身份和启用状态。已检查的客户端代码不能证明存在按尝试查询接口、服务端幂等或跨崩溃持久化结果。上述 Qwen 回执是我们的新增建议,不是对 Codex 功能的宣称。[协议定义](https://github.com/openai/codex/blob/a29c7b97e88367044ae84c28c85451ec3e72af9f/codex-rs/app-server-protocol/src/protocol/v2/plugin.rs#L956-L975)、[远端请求](https://github.com/openai/codex/blob/a29c7b97e88367044ae84c28c85451ec3e72af9f/codex-rs/core-plugins/src/remote.rs#L1641-L1683)。
|
||||
|
||||
## 复用与正确性边界
|
||||
|
||||
当前 ZIP 打开函数是依赖 Buffer 的边界;其余解压逻辑消费 ZIP reader。因此文件打开分支复用惰性条目遍历、路径检查、声明尺寸检查、实际字节计数、包装目录展开、manifest 检查和发布流程。保留自有 PassThrough 的流关闭处理及现有 inflater。
|
||||
|
||||
实施验证发现,yauzl 2.10 的 fd-slicer 依赖在当前 Node 版本下会提前把文件流标记为已销毁,阻止缓冲中的压缩字节继续流出。一个有效的 70 MiB 条目在输出 60,071,690 字节后挂起,尚有 58,163 压缩字节滞留;同一归档通过 2.10 的 Buffer 入口和 yauzl 3.4 的文件入口均完整结束。因此 CLI 升级到仓库其他位置已使用的 yauzl 3.4,并保留旧安装测试作为回归覆盖。不引入自定义文件 reader 或 inflater。
|
||||
|
||||
不要仅为新增输入就迁移锁位置、引入跨进程锁、重写目录复制或搭建归档框架。新路由必须参加既有互斥,本方案不宣称跨 daemon 串行化。
|
||||
|
||||
回滚失败返回 `recovery_required` 及保留路径。后续安装必须保留遗留备份;成功安装仍清理自己创建的备份。这项保护适用于所有来源(包括旧 base64、目录和 GitHub),因为内存回执过期或进程重启后,无法安全区分可删除的备份与唯一恢复副本。故障注入测试覆盖恢复失败后再次安装的序列。不引入自动恢复程序或发布器重写。
|
||||
|
||||
## 验证与验收
|
||||
|
||||
已完成的基线使用全局 `qwen 0.22.0`,不是合入后的 main bundle。小 ZIP 与目录请求成功且字节一致;真实 20 MiB ZIP 编成 27,962,121 字节 JSON 后返回 413;zip-file 返回 400,raw 上传返回 404。该版本没有现代配置路由,因此这些结果不能验证当前 main 的相应行为。临时输入得到保留,测试 daemon 已清理。
|
||||
|
||||
上述固定基线的未修改 main bundle 也已验证:现代 config 路由成功安装小 ZIP 和目录,内容逐字节一致;真实 20 MiB base64 请求返回 413,`zip-file` 返回 400,raw 上传返回 404。以下实施验收以此为基线:
|
||||
|
||||
- 真实 20 MiB 归档分别通过 raw 和文件来源安装,逐字节对比;新路径不读取整个归档为 Buffer。
|
||||
- 新来源 50 MiB 压缩 / 200 MiB 展开 / 4096 条目及有效根 manifest 32 MiB 边界;旧来源保持 64 MiB / 32 MiB / 2000 文件边界。
|
||||
- 新旧路径对穿越、符号链接、重复路径、包装目录、manifest、GitHub 回退和 base64 兼容的共同表现。
|
||||
- 接收中断、伪造声明尺寸、暂存写入失败、等待锁时超时、运行时 generation 关闭、替换失败;实际工作结束、临时文件关闭并清理,未提交时保留旧安装。
|
||||
- 全局与选中运行时的激活归属正确;拒绝 workspace 状态时没有 primary 回退。
|
||||
- SDK fetch/XHR 行为、旧 daemon 回退、通用文件上传不变。
|
||||
- 两个大操作期间的内存和临时磁盘实测,以及第三个操作被拒绝。
|
||||
|
||||
新 4096 限额统计所有显式 ZIP 记录,包括目录和忽略元数据;隐式父目录不重复计数。展开字节只统计保留文件。将新政策仅用于新来源,可以避免收紧旧目录布局。名称兼容和受管理内容归属也需要独立验收,不能从上传成功推导。
|
||||
|
||||
## 验证边界
|
||||
|
||||
采用上方固定政策,不增加配置开关。测量内存与临时磁盘,验证取消和恢复故障,并使用构建后的 bundle 执行独立真实 HTTP 与浏览器验收。记录实际操作系统与浏览器;源码检查和单元测试不代表跨平台或浏览器验收通过。执行证据单独记录在 E2E 报告中。
|
||||
|
||||
## 来源
|
||||
|
||||
- [合入后的 Qwen 安装器](https://github.com/QwenLM/qwen-code/blob/2415b12329b1bb0ef1b5beb6bfc0bd0535b9ee7b/packages/cli/src/serve/workspace-skill-management.ts)、[Skill 路由](https://github.com/QwenLM/qwen-code/blob/2415b12329b1bb0ef1b5beb6bfc0bd0535b9ee7b/packages/cli/src/serve/routes/workspace-skills.ts)、[JSON 中间件](https://github.com/QwenLM/qwen-code/blob/2415b12329b1bb0ef1b5beb6bfc0bd0535b9ee7b/packages/cli/src/serve/server/error-handlers.ts)。
|
||||
- [Codex Python 安装器](https://github.com/openai/codex/blob/a29c7b97e88367044ae84c28c85451ec3e72af9f/codex-rs/skills/src/assets/samples/skill-installer/scripts/install-skill-from-github.py)、[远程包](https://github.com/openai/codex/blob/a29c7b97e88367044ae84c28c85451ec3e72af9f/codex-rs/core-plugins/src/remote_bundle.rs)、[插件存储](https://github.com/openai/codex/blob/a29c7b97e88367044ae84c28c85451ec3e72af9f/codex-rs/core-plugins/src/store.rs)、[插件分享](https://github.com/openai/codex/blob/a29c7b97e88367044ae84c28c85451ec3e72af9f/codex-rs/core-plugins/src/remote/share.rs)。
|
||||
@@ -184,7 +184,6 @@ export const legacyCoreBarrelImports = [
|
||||
'packages/cli/src/serve/workspace-providers-status.ts',
|
||||
'packages/cli/src/serve/workspace-remember.ts',
|
||||
'packages/cli/src/serve/workspace-runtime-storage.ts',
|
||||
'packages/cli/src/serve/workspace-skill-management.ts',
|
||||
'packages/cli/src/services/BuiltinCommandLoader.ts',
|
||||
'packages/cli/src/services/BundledSkillLoader.ts',
|
||||
'packages/cli/src/services/CommandService.ts',
|
||||
|
||||
@@ -110,7 +110,7 @@
|
||||
"ws": "^8.18.0",
|
||||
"yaml": "^2.8.1",
|
||||
"yargs": "^17.7.2",
|
||||
"yauzl": "^2.10.0",
|
||||
"yauzl": "^3.4.0",
|
||||
"zod": "^3.23.8"
|
||||
},
|
||||
"devDependencies": {
|
||||
|
||||
@@ -0,0 +1,89 @@
|
||||
/**
|
||||
* @license
|
||||
* Copyright 2026 Qwen Team
|
||||
* SPDX-License-Identifier: Apache-2.0
|
||||
*/
|
||||
|
||||
import { describe, expect, it, vi } from 'vitest';
|
||||
import { PathMutexRegistry } from './path-mutex-registry.js';
|
||||
|
||||
describe('PathMutexRegistry cancellation', () => {
|
||||
it('retires a cancelled waiter without letting later callers bypass the owner', async () => {
|
||||
const locks = new PathMutexRegistry();
|
||||
const controller = new AbortController();
|
||||
let release!: () => void;
|
||||
const held = new Promise<void>((resolve) => {
|
||||
release = resolve;
|
||||
});
|
||||
const first = locks.runExclusive('target', () => held);
|
||||
const cancelledWork = vi.fn();
|
||||
const cancelled = locks.runExclusive(
|
||||
'target',
|
||||
cancelledWork,
|
||||
controller.signal,
|
||||
);
|
||||
controller.abort();
|
||||
await expect(cancelled).rejects.toMatchObject({ name: 'AbortError' });
|
||||
const laterWork = vi.fn().mockResolvedValue('done');
|
||||
const later = locks.runExclusive('target', laterWork);
|
||||
await new Promise((resolve) => setImmediate(resolve));
|
||||
expect(laterWork).not.toHaveBeenCalled();
|
||||
expect(cancelledWork).not.toHaveBeenCalled();
|
||||
release();
|
||||
await first;
|
||||
await expect(later).resolves.toBe('done');
|
||||
});
|
||||
|
||||
it('rejects an already aborted signal without calling the callback', async () => {
|
||||
const locks = new PathMutexRegistry();
|
||||
const callback = vi.fn();
|
||||
await expect(
|
||||
locks.runExclusive('target', callback, AbortSignal.abort()),
|
||||
).rejects.toMatchObject({ name: 'AbortError' });
|
||||
expect(callback).not.toHaveBeenCalled();
|
||||
await expect(
|
||||
locks.runExclusive('target', async () => 'next'),
|
||||
).resolves.toBe('next');
|
||||
});
|
||||
|
||||
it('keeps ownership until running work settles after cancellation', async () => {
|
||||
const locks = new PathMutexRegistry();
|
||||
const controller = new AbortController();
|
||||
let release!: () => void;
|
||||
let entered!: () => void;
|
||||
const ready = new Promise<void>((resolve) => {
|
||||
entered = resolve;
|
||||
});
|
||||
const first = locks.runExclusive(
|
||||
'target',
|
||||
async () => {
|
||||
entered();
|
||||
await new Promise<void>((resolve) => {
|
||||
release = resolve;
|
||||
});
|
||||
},
|
||||
controller.signal,
|
||||
);
|
||||
await ready;
|
||||
controller.abort();
|
||||
const nextWork = vi.fn().mockResolvedValue(undefined);
|
||||
const next = locks.runExclusive('target', nextWork);
|
||||
await new Promise((resolve) => setImmediate(resolve));
|
||||
expect(nextWork).not.toHaveBeenCalled();
|
||||
release();
|
||||
await Promise.all([first, next]);
|
||||
expect(nextWork).toHaveBeenCalledOnce();
|
||||
});
|
||||
|
||||
it('releases after a failed owner and permits unrelated keys', async () => {
|
||||
const locks = new PathMutexRegistry();
|
||||
const error = new Error('failed');
|
||||
await expect(
|
||||
locks.runExclusive('target', async () => {
|
||||
throw error;
|
||||
}),
|
||||
).rejects.toBe(error);
|
||||
await expect(locks.runExclusive('target', async () => 1)).resolves.toBe(1);
|
||||
await expect(locks.runExclusive('other', async () => 2)).resolves.toBe(2);
|
||||
});
|
||||
});
|
||||
@@ -7,7 +7,12 @@
|
||||
export class PathMutexRegistry {
|
||||
private readonly tails = new Map<string, Promise<void>>();
|
||||
|
||||
async runExclusive<T>(key: string, fn: () => Promise<T>): Promise<T> {
|
||||
async runExclusive<T>(
|
||||
key: string,
|
||||
fn: () => Promise<T>,
|
||||
signal?: AbortSignal,
|
||||
): Promise<T> {
|
||||
signal?.throwIfAborted();
|
||||
const previous = this.tails.get(key) ?? Promise.resolve();
|
||||
let release!: () => void;
|
||||
const current = new Promise<void>((resolve) => {
|
||||
@@ -15,12 +20,34 @@ export class PathMutexRegistry {
|
||||
});
|
||||
const tail = previous.catch(() => undefined).then(() => current);
|
||||
this.tails.set(key, tail);
|
||||
await previous.catch(() => undefined);
|
||||
// A cancelled waiter still follows its predecessor; deleting its tail
|
||||
// early would let the next caller bypass the current lock owner.
|
||||
void tail.then(() => {
|
||||
if (this.tails.get(key) === tail) this.tails.delete(key);
|
||||
});
|
||||
try {
|
||||
if (signal) {
|
||||
await new Promise<void>((resolve, reject) => {
|
||||
const onAbort = () => {
|
||||
signal.removeEventListener('abort', onAbort);
|
||||
reject(signal.reason);
|
||||
};
|
||||
signal.addEventListener('abort', onAbort, { once: true });
|
||||
void previous
|
||||
.catch(() => undefined)
|
||||
.then(() => {
|
||||
signal.removeEventListener('abort', onAbort);
|
||||
resolve();
|
||||
});
|
||||
if (signal.aborted) onAbort();
|
||||
});
|
||||
} else {
|
||||
await previous.catch(() => undefined);
|
||||
}
|
||||
signal?.throwIfAborted();
|
||||
return await fn();
|
||||
} finally {
|
||||
release();
|
||||
if (this.tails.get(key) === tail) this.tails.delete(key);
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,175 @@
|
||||
/**
|
||||
* @license
|
||||
* Copyright 2026 Qwen Team
|
||||
* SPDX-License-Identifier: Apache-2.0
|
||||
*/
|
||||
|
||||
import { createHash } from 'node:crypto';
|
||||
import {
|
||||
promises as fs,
|
||||
truncateSync,
|
||||
utimesSync,
|
||||
writeFileSync,
|
||||
} from 'node:fs';
|
||||
import * as os from 'node:os';
|
||||
import * as path from 'node:path';
|
||||
import { Readable } from 'node:stream';
|
||||
import { afterEach, describe, expect, it, vi } from 'vitest';
|
||||
import { acquireSkillArchive } from './skill-archive-input.js';
|
||||
import { MAX_SKILL_ARCHIVE_BYTES } from './workspace-skill-management.js';
|
||||
|
||||
const directories: string[] = [];
|
||||
afterEach(async () => {
|
||||
vi.restoreAllMocks();
|
||||
await Promise.all(
|
||||
directories
|
||||
.splice(0)
|
||||
.map((directory) => fs.rm(directory, { recursive: true, force: true })),
|
||||
);
|
||||
});
|
||||
async function fixture() {
|
||||
const directory = await fs.mkdtemp(
|
||||
path.join(os.tmpdir(), 'skill-input-test-'),
|
||||
);
|
||||
directories.push(directory);
|
||||
const file = path.join(directory, 'source.zip');
|
||||
await fs.writeFile(file, 'original');
|
||||
return file;
|
||||
}
|
||||
|
||||
describe('owned Skill archives', () => {
|
||||
it('keeps a verified independent copy after the borrowed path is replaced', async () => {
|
||||
const file = await fixture();
|
||||
const expected = createHash('sha256').update('original').digest('hex');
|
||||
const owned = await acquireSkillArchive(
|
||||
{ type: 'zip-file', path: file, sha256: expected.toUpperCase() },
|
||||
new AbortController().signal,
|
||||
);
|
||||
await fs.unlink(file);
|
||||
await fs.writeFile(file, 'replacement');
|
||||
expect(owned.sha256).toBe(expected);
|
||||
await expect(fs.readFile(owned.path, 'utf8')).resolves.toBe('original');
|
||||
await owned.dispose();
|
||||
await expect(fs.readFile(file, 'utf8')).resolves.toBe('replacement');
|
||||
await expect(fs.access(owned.path)).rejects.toThrow();
|
||||
});
|
||||
|
||||
it('rejects a digest mismatch without changing the input', async () => {
|
||||
const file = await fixture();
|
||||
const mkdtemp = vi.spyOn(fs, 'mkdtemp');
|
||||
await expect(
|
||||
acquireSkillArchive(
|
||||
{ type: 'zip-file', path: file, sha256: '0'.repeat(64) },
|
||||
new AbortController().signal,
|
||||
),
|
||||
).rejects.toMatchObject({ code: 'skill_archive_digest_mismatch' });
|
||||
await expect(fs.readFile(file, 'utf8')).resolves.toBe('original');
|
||||
await expect(
|
||||
fs.access(await mkdtemp.mock.results[0]!.value),
|
||||
).rejects.toThrow();
|
||||
});
|
||||
|
||||
it('rejects a symbolic link and a directory', async () => {
|
||||
const file = await fixture();
|
||||
const link = `${file}.link`;
|
||||
await fs.symlink(file, link);
|
||||
for (const input of [link, path.dirname(file)]) {
|
||||
await expect(
|
||||
acquireSkillArchive(
|
||||
{ type: 'zip-file', path: input },
|
||||
new AbortController().signal,
|
||||
),
|
||||
).rejects.toMatchObject({ code: 'invalid_skill_source' });
|
||||
}
|
||||
});
|
||||
|
||||
it.each(['truncated', 'modified'])(
|
||||
'rejects a source %s during copying and closes its handle',
|
||||
async (change) => {
|
||||
const file = await fixture();
|
||||
const mkdtemp = vi.spyOn(fs, 'mkdtemp');
|
||||
const open = fs.open.bind(fs);
|
||||
const handles: Array<Awaited<ReturnType<typeof fs.open>>> = [];
|
||||
vi.spyOn(fs, 'open').mockImplementation(async (...args) => {
|
||||
const handle = await open(...args);
|
||||
handles.push(handle);
|
||||
const stream = handle.createReadStream.bind(handle);
|
||||
vi.spyOn(handle, 'createReadStream').mockImplementation((options) => {
|
||||
const result = stream(options);
|
||||
result.once('data', () => {
|
||||
if (change === 'truncated') truncateSync(file, 0);
|
||||
else {
|
||||
writeFileSync(file, 'modified');
|
||||
utimesSync(file, 1, 1);
|
||||
}
|
||||
});
|
||||
result.pause();
|
||||
return result;
|
||||
});
|
||||
return handle;
|
||||
});
|
||||
await expect(
|
||||
acquireSkillArchive(
|
||||
{ type: 'zip-file', path: file },
|
||||
new AbortController().signal,
|
||||
),
|
||||
).rejects.toMatchObject({ code: 'skill_archive_changed' });
|
||||
expect(handles.every((handle) => handle.fd === -1)).toBe(true);
|
||||
await expect(
|
||||
fs.access(await mkdtemp.mock.results[0]!.value),
|
||||
).rejects.toThrow();
|
||||
},
|
||||
);
|
||||
|
||||
it('accepts exactly the raw limit and rejects one more actual byte', async () => {
|
||||
function bytes(size: number) {
|
||||
return Readable.from(
|
||||
(function* () {
|
||||
let remaining = size;
|
||||
while (remaining > 0) {
|
||||
const length = Math.min(remaining, 65536);
|
||||
remaining -= length;
|
||||
yield Buffer.alloc(length);
|
||||
}
|
||||
})(),
|
||||
);
|
||||
}
|
||||
const accepted = await acquireSkillArchive(
|
||||
bytes(MAX_SKILL_ARCHIVE_BYTES),
|
||||
new AbortController().signal,
|
||||
);
|
||||
expect((await fs.stat(accepted.path)).size).toBe(MAX_SKILL_ARCHIVE_BYTES);
|
||||
await accepted.dispose();
|
||||
const mkdtemp = vi.spyOn(fs, 'mkdtemp');
|
||||
await expect(
|
||||
acquireSkillArchive(
|
||||
bytes(MAX_SKILL_ARCHIVE_BYTES + 1),
|
||||
new AbortController().signal,
|
||||
),
|
||||
).rejects.toMatchObject({
|
||||
code: 'skill_archive_too_large',
|
||||
statusCode: 413,
|
||||
});
|
||||
await expect(
|
||||
fs.access(await mkdtemp.mock.results[0]!.value),
|
||||
).rejects.toThrow();
|
||||
});
|
||||
|
||||
it('stops a live receive on cancellation', async () => {
|
||||
const mkdtemp = vi.spyOn(fs, 'mkdtemp');
|
||||
const controller = new AbortController();
|
||||
const source = new Readable({
|
||||
read() {
|
||||
this.push(Buffer.alloc(1024));
|
||||
controller.abort();
|
||||
},
|
||||
});
|
||||
await expect(
|
||||
acquireSkillArchive(source, controller.signal),
|
||||
).rejects.toMatchObject({ name: 'AbortError' });
|
||||
source.destroy();
|
||||
await expect(
|
||||
fs.access(await mkdtemp.mock.results[0]!.value),
|
||||
).rejects.toThrow();
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,171 @@
|
||||
/**
|
||||
* @license
|
||||
* Copyright 2026 Qwen Team
|
||||
* SPDX-License-Identifier: Apache-2.0
|
||||
*/
|
||||
|
||||
import { createHash } from 'node:crypto';
|
||||
import { constants, createWriteStream, promises as fs } from 'node:fs';
|
||||
import type { FileHandle } from 'node:fs/promises';
|
||||
import * as os from 'node:os';
|
||||
import * as path from 'node:path';
|
||||
import { PassThrough, Transform, type Readable } from 'node:stream';
|
||||
import { pipeline } from 'node:stream/promises';
|
||||
import {
|
||||
MAX_SKILL_ARCHIVE_BYTES,
|
||||
WorkspaceSkillManagementError,
|
||||
} from './workspace-skill-management.js';
|
||||
|
||||
export interface SkillZipFileSource {
|
||||
type: 'zip-file';
|
||||
path: string;
|
||||
sha256?: string;
|
||||
}
|
||||
|
||||
export interface PreparedSkillArchive {
|
||||
path: string;
|
||||
sha256: string;
|
||||
dispose(): Promise<void>;
|
||||
}
|
||||
|
||||
export function validateSkillZipFileSource(source: SkillZipFileSource): void {
|
||||
if (!path.isAbsolute(source.path)) {
|
||||
throw new WorkspaceSkillManagementError(
|
||||
'invalid_skill_source',
|
||||
'ZIP path must be absolute on the daemon host',
|
||||
);
|
||||
}
|
||||
if (source.sha256 !== undefined && !/^[a-fA-F0-9]{64}$/.test(source.sha256)) {
|
||||
throw new WorkspaceSkillManagementError(
|
||||
'invalid_skill_source',
|
||||
'Expected SHA-256 must contain 64 hexadecimal digits',
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
export async function acquireSkillArchive(
|
||||
input: Readable | SkillZipFileSource,
|
||||
signal: AbortSignal,
|
||||
): Promise<PreparedSkillArchive> {
|
||||
signal.throwIfAborted();
|
||||
const fileSource = 'type' in input ? input : undefined;
|
||||
if (fileSource) validateSkillZipFileSource(fileSource);
|
||||
let handle: FileHandle | undefined;
|
||||
let directory: string | undefined;
|
||||
let source: Readable | undefined;
|
||||
const bridge = new PassThrough();
|
||||
const forwardError = (error: Error) => bridge.destroy(error);
|
||||
try {
|
||||
let before;
|
||||
if (fileSource) {
|
||||
const named = await fs.lstat(fileSource.path, { bigint: true });
|
||||
if (!named.isFile()) {
|
||||
throw new WorkspaceSkillManagementError(
|
||||
'invalid_skill_source',
|
||||
'ZIP source must be a regular file, not a link or special file',
|
||||
);
|
||||
}
|
||||
// NONBLOCK also prevents a raced-in FIFO from hanging the open forever.
|
||||
// Identity checks cover platforms without O_NOFOLLOW.
|
||||
handle = await fs.open(
|
||||
fileSource.path,
|
||||
constants.O_RDONLY |
|
||||
(constants.O_NOFOLLOW ?? 0) |
|
||||
(constants.O_NONBLOCK ?? 0),
|
||||
);
|
||||
before = await handle.stat({ bigint: true });
|
||||
if (
|
||||
!before.isFile() ||
|
||||
before.ino === 0n ||
|
||||
before.ino !== named.ino ||
|
||||
before.dev !== named.dev
|
||||
) {
|
||||
throw new WorkspaceSkillManagementError(
|
||||
'invalid_skill_source',
|
||||
'ZIP source changed while opening',
|
||||
);
|
||||
}
|
||||
if (before.size > BigInt(MAX_SKILL_ARCHIVE_BYTES)) {
|
||||
throw new WorkspaceSkillManagementError(
|
||||
'skill_archive_too_large',
|
||||
'ZIP archive exceeds 50 MiB',
|
||||
413,
|
||||
);
|
||||
}
|
||||
source = handle.createReadStream({ autoClose: false });
|
||||
} else {
|
||||
source = input as Readable;
|
||||
}
|
||||
signal.throwIfAborted();
|
||||
directory = await fs.mkdtemp(path.join(os.tmpdir(), 'qwen-skill-archive-'));
|
||||
const archivePath = path.join(directory, 'archive.zip');
|
||||
const hash = createHash('sha256');
|
||||
let bytes = 0;
|
||||
const guard = new Transform({
|
||||
transform(chunk: Buffer, _encoding, callback) {
|
||||
bytes += chunk.length;
|
||||
if (bytes > MAX_SKILL_ARCHIVE_BYTES) {
|
||||
callback(
|
||||
new WorkspaceSkillManagementError(
|
||||
'skill_archive_too_large',
|
||||
'ZIP archive exceeds 50 MiB',
|
||||
413,
|
||||
),
|
||||
);
|
||||
return;
|
||||
}
|
||||
hash.update(chunk);
|
||||
callback(null, chunk);
|
||||
},
|
||||
});
|
||||
source.on('error', forwardError);
|
||||
source.pipe(bridge);
|
||||
await pipeline(
|
||||
bridge,
|
||||
guard,
|
||||
createWriteStream(archivePath, { flags: 'wx', mode: 0o600 }),
|
||||
{ signal },
|
||||
);
|
||||
signal.throwIfAborted();
|
||||
if (handle && before) {
|
||||
const after = await handle.stat({ bigint: true });
|
||||
if (
|
||||
BigInt(bytes) !== before.size ||
|
||||
before.size !== after.size ||
|
||||
before.dev !== after.dev ||
|
||||
before.ino !== after.ino ||
|
||||
before.mtimeNs !== after.mtimeNs ||
|
||||
before.ctimeNs !== after.ctimeNs
|
||||
) {
|
||||
throw new WorkspaceSkillManagementError(
|
||||
'skill_archive_changed',
|
||||
'ZIP source changed while copying',
|
||||
);
|
||||
}
|
||||
}
|
||||
const sha256 = hash.digest('hex');
|
||||
if (fileSource?.sha256 && fileSource.sha256.toLowerCase() !== sha256) {
|
||||
throw new WorkspaceSkillManagementError(
|
||||
'skill_archive_digest_mismatch',
|
||||
'ZIP archive does not match the expected SHA-256',
|
||||
);
|
||||
}
|
||||
const ownedDirectory = directory;
|
||||
return {
|
||||
path: archivePath,
|
||||
sha256,
|
||||
dispose: () => fs.rm(ownedDirectory, { recursive: true, force: true }),
|
||||
};
|
||||
} catch (error) {
|
||||
if (directory) await fs.rm(directory, { recursive: true, force: true });
|
||||
throw error;
|
||||
} finally {
|
||||
if (source) {
|
||||
source.unpipe(bridge);
|
||||
source.removeListener('error', forwardError);
|
||||
if (fileSource) source.destroy();
|
||||
}
|
||||
bridge.destroy();
|
||||
await handle?.close();
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,421 @@
|
||||
/**
|
||||
* @license
|
||||
* Copyright 2026 Qwen Team
|
||||
* SPDX-License-Identifier: Apache-2.0
|
||||
*/
|
||||
|
||||
import { createWriteStream, promises as fs } from 'node:fs';
|
||||
import * as os from 'node:os';
|
||||
import * as path from 'node:path';
|
||||
import { Readable } from 'node:stream';
|
||||
import { pipeline } from 'node:stream/promises';
|
||||
import archiver from 'archiver';
|
||||
import { ZipFile, type Entry } from 'yauzl';
|
||||
import { afterEach, describe, expect, it, vi } from 'vitest';
|
||||
import {
|
||||
installWorkspaceSkill,
|
||||
MAX_SKILL_ARCHIVE_ENTRIES,
|
||||
MAX_SKILL_ARCHIVE_EXPANDED_BYTES,
|
||||
MAX_SKILL_ARCHIVE_MANIFEST_BYTES,
|
||||
WorkspaceSkillRecoveryError,
|
||||
} from './workspace-skill-management.js';
|
||||
|
||||
const directories: string[] = [];
|
||||
const manifest =
|
||||
'---\nname: example\ndescription: Example\n---\nInstructions.\n';
|
||||
async function workspace() {
|
||||
const directory = await fs.mkdtemp(
|
||||
path.join(os.tmpdir(), 'skill-archive-test-'),
|
||||
);
|
||||
directories.push(directory);
|
||||
return directory;
|
||||
}
|
||||
async function archive(
|
||||
files: Array<{ name: string; content?: string; bytes?: number }>,
|
||||
) {
|
||||
const file = path.join(await workspace(), 'source.zip');
|
||||
const zip = archiver('zip', { zlib: { level: 1 } });
|
||||
const written = pipeline(zip, createWriteStream(file));
|
||||
for (const entry of files) {
|
||||
const content =
|
||||
entry.bytes === undefined
|
||||
? (entry.content ?? '')
|
||||
: Readable.from(
|
||||
(function* () {
|
||||
let remaining = entry.bytes!;
|
||||
while (remaining > 0) {
|
||||
const size = Math.min(remaining, 65536);
|
||||
remaining -= size;
|
||||
yield Buffer.alloc(size);
|
||||
}
|
||||
})(),
|
||||
);
|
||||
zip.append(content, { name: entry.name });
|
||||
}
|
||||
await zip.finalize();
|
||||
await written;
|
||||
return file;
|
||||
}
|
||||
const request = (file: string) => ({
|
||||
name: 'example',
|
||||
scope: 'workspace' as const,
|
||||
source: { type: 'prepared-zip' as const, path: file },
|
||||
});
|
||||
afterEach(async () => {
|
||||
vi.restoreAllMocks();
|
||||
await Promise.all(
|
||||
directories
|
||||
.splice(0)
|
||||
.map((directory) => fs.rm(directory, { recursive: true, force: true })),
|
||||
);
|
||||
});
|
||||
|
||||
describe('file-backed Skill ZIP policy', () => {
|
||||
it('accepts repeated trailing directory separators in both ZIP sources', async () => {
|
||||
const file = await archive([
|
||||
{ name: 'SKILL.md', content: manifest },
|
||||
{ name: 'fooX/' },
|
||||
{ name: 'foo/a.txt', content: 'resource' },
|
||||
]);
|
||||
const bytes = await fs.readFile(file);
|
||||
const originalName = Buffer.from('fooX/');
|
||||
let replacements = 0;
|
||||
for (
|
||||
let offset = bytes.indexOf(originalName);
|
||||
offset !== -1;
|
||||
offset = bytes.indexOf(originalName, offset + originalName.length)
|
||||
) {
|
||||
bytes.write('foo//', offset);
|
||||
replacements += 1;
|
||||
}
|
||||
expect(replacements).toBe(2);
|
||||
await fs.writeFile(file, bytes);
|
||||
for (const source of [
|
||||
{ type: 'zip' as const, contentBase64: bytes.toString('base64') },
|
||||
{ type: 'prepared-zip' as const, path: file },
|
||||
]) {
|
||||
const result = await installWorkspaceSkill(await workspace(), {
|
||||
...request(file),
|
||||
source,
|
||||
});
|
||||
expect(
|
||||
await fs.readFile(
|
||||
path.join(path.dirname(result.installedPath!), 'foo/a.txt'),
|
||||
'utf8',
|
||||
),
|
||||
).toBe('resource');
|
||||
}
|
||||
});
|
||||
|
||||
it('restores the old installation even if cleaning an unpublished candidate fails', async () => {
|
||||
const root = await workspace();
|
||||
const file = await archive([{ name: 'SKILL.md', content: manifest }]);
|
||||
const installed = await installWorkspaceSkill(root, request(file));
|
||||
const rename = fs.rename.bind(fs);
|
||||
const remove = fs.rm.bind(fs);
|
||||
vi.spyOn(fs, 'rename').mockImplementation(async (from, to) => {
|
||||
if (String(from).includes('.installing-'))
|
||||
throw new Error('publication failed');
|
||||
return rename(from, to);
|
||||
});
|
||||
vi.spyOn(fs, 'rm').mockImplementation(async (target, options) => {
|
||||
if (String(target).includes('.installing-'))
|
||||
throw new Error('candidate cleanup failed');
|
||||
return remove(target, options);
|
||||
});
|
||||
await expect(installWorkspaceSkill(root, request(file))).rejects.toThrow(
|
||||
'publication failed',
|
||||
);
|
||||
expect(await fs.readFile(installed.installedPath!, 'utf8')).toBe(manifest);
|
||||
expect(
|
||||
(await fs.readdir(root, { recursive: true })).some((entry) =>
|
||||
entry.includes('.backup-'),
|
||||
),
|
||||
).toBe(false);
|
||||
});
|
||||
|
||||
it('stops a live entry stream before cancelled extraction returns and removes staging', async () => {
|
||||
const file = await archive([
|
||||
{ name: 'SKILL.md', content: manifest },
|
||||
{ name: 'asset.bin', bytes: 70 * 1024 * 1024 },
|
||||
]);
|
||||
const root = await workspace();
|
||||
const controller = new AbortController();
|
||||
const streams: Readable[] = [];
|
||||
let closed = false;
|
||||
const open = ZipFile.prototype.openReadStream;
|
||||
vi.spyOn(ZipFile.prototype, 'openReadStream').mockImplementation(function (
|
||||
this: ZipFile,
|
||||
entry: Entry,
|
||||
callback: (error: Error | null, stream: Readable) => void,
|
||||
) {
|
||||
return open.call(this, entry, (error, stream) => {
|
||||
if (stream && entry.fileName === 'asset.bin') {
|
||||
streams.push(stream);
|
||||
stream.once('data', () =>
|
||||
controller.abort(new Error('stop extraction')),
|
||||
);
|
||||
stream.once('close', () => {
|
||||
closed = true;
|
||||
});
|
||||
}
|
||||
callback(error, stream);
|
||||
});
|
||||
} as typeof ZipFile.prototype.openReadStream);
|
||||
await expect(
|
||||
installWorkspaceSkill(root, request(file), undefined, undefined, {
|
||||
signal: controller.signal,
|
||||
}),
|
||||
).rejects.toThrow('stop extraction');
|
||||
expect(streams).toHaveLength(1);
|
||||
expect(streams[0].destroyed).toBe(true);
|
||||
expect(closed).toBe(true);
|
||||
expect(
|
||||
(await fs.readdir(root, { recursive: true })).some(
|
||||
(entry) => entry.includes('.installing-') || entry.endsWith('SKILL.md'),
|
||||
),
|
||||
).toBe(false);
|
||||
expect((await fs.stat(file)).isFile()).toBe(true);
|
||||
});
|
||||
|
||||
it.each([
|
||||
['asset.bin', 'asset.bin'],
|
||||
['assets/', 'assets'],
|
||||
['assets/child.bin', 'assets'],
|
||||
])('rejects conflicting native paths %s and %s', async (first, second) => {
|
||||
const file = await archive([
|
||||
{ name: 'SKILL.md', content: manifest },
|
||||
{ name: first, content: first.endsWith('/') ? '' : 'first' },
|
||||
{ name: second, content: 'second' },
|
||||
]);
|
||||
await expect(
|
||||
installWorkspaceSkill(await workspace(), request(file)),
|
||||
).rejects.toMatchObject({ code: 'invalid_skill_package' });
|
||||
});
|
||||
|
||||
it('does not overwrite files through filesystem case aliases', async () => {
|
||||
const root = await workspace();
|
||||
await fs.writeFile(path.join(root, 'CaseProbe'), 'probe');
|
||||
const aliases = await fs.access(path.join(root, 'caseprobe')).then(
|
||||
() => true,
|
||||
() => false,
|
||||
);
|
||||
const file = await archive([
|
||||
{ name: 'SKILL.md', content: manifest },
|
||||
{ name: 'Asset.bin', content: 'first' },
|
||||
{ name: 'asset.bin', content: 'second' },
|
||||
]);
|
||||
const operation = installWorkspaceSkill(root, request(file));
|
||||
if (aliases)
|
||||
await expect(operation).rejects.toMatchObject({
|
||||
code: 'invalid_skill_package',
|
||||
});
|
||||
else {
|
||||
const result = await operation;
|
||||
const directory = path.dirname(result.installedPath!);
|
||||
expect(await fs.readFile(path.join(directory, 'Asset.bin'), 'utf8')).toBe(
|
||||
'first',
|
||||
);
|
||||
expect(await fs.readFile(path.join(directory, 'asset.bin'), 'utf8')).toBe(
|
||||
'second',
|
||||
);
|
||||
}
|
||||
});
|
||||
|
||||
it('installs resources above both legacy limits, including nested resource manifests', async () => {
|
||||
const file = await archive([
|
||||
{ name: 'wrapper/SKILL.md', content: manifest },
|
||||
{ name: 'wrapper/assets/large.bin', bytes: 70 * 1024 * 1024 },
|
||||
{ name: 'wrapper/assets/nested/SKILL.md', bytes: 33 * 1024 * 1024 },
|
||||
]);
|
||||
const root = await workspace();
|
||||
const result = await installWorkspaceSkill(root, request(file));
|
||||
const installed = path.dirname(result.installedPath!);
|
||||
expect((await fs.stat(path.join(installed, 'assets/large.bin'))).size).toBe(
|
||||
70 * 1024 * 1024,
|
||||
);
|
||||
expect(
|
||||
(await fs.stat(path.join(installed, 'assets/nested/SKILL.md'))).size,
|
||||
).toBe(33 * 1024 * 1024);
|
||||
await expect(
|
||||
installWorkspaceSkill(await workspace(), {
|
||||
name: 'example',
|
||||
scope: 'workspace',
|
||||
source: {
|
||||
type: 'zip',
|
||||
contentBase64: (await fs.readFile(file)).toString('base64'),
|
||||
},
|
||||
}),
|
||||
).rejects.toMatchObject({ code: 'skill_package_too_large' });
|
||||
});
|
||||
|
||||
it.each([0, 1])(
|
||||
'enforces the actual 200 MiB expanded ceiling at offset %s',
|
||||
async (over) => {
|
||||
const file = await archive([
|
||||
{ name: 'SKILL.md', content: manifest },
|
||||
{
|
||||
name: 'asset.bin',
|
||||
bytes:
|
||||
MAX_SKILL_ARCHIVE_EXPANDED_BYTES -
|
||||
Buffer.byteLength(manifest) +
|
||||
over,
|
||||
},
|
||||
]);
|
||||
const install = installWorkspaceSkill(await workspace(), request(file));
|
||||
if (over)
|
||||
await expect(install).rejects.toMatchObject({
|
||||
code: 'skill_package_too_large',
|
||||
statusCode: 413,
|
||||
});
|
||||
else
|
||||
await expect(install).resolves.toMatchObject({ skillName: 'example' });
|
||||
},
|
||||
);
|
||||
|
||||
it('checks the normalized root manifest size before reading it as text', async () => {
|
||||
const file = await archive([
|
||||
{ name: 'wrapper/SKILL.md', bytes: MAX_SKILL_ARCHIVE_MANIFEST_BYTES + 1 },
|
||||
]);
|
||||
const read = vi.spyOn(fs, 'readFile');
|
||||
await expect(
|
||||
installWorkspaceSkill(await workspace(), request(file)),
|
||||
).rejects.toMatchObject({ code: 'skill_package_too_large' });
|
||||
expect(
|
||||
read.mock.calls.some(([file]) => String(file).endsWith('/SKILL.md')),
|
||||
).toBe(false);
|
||||
});
|
||||
|
||||
it('accepts an exactly 32 MiB root manifest', async () => {
|
||||
const file = await archive([
|
||||
{
|
||||
name: 'SKILL.md',
|
||||
content:
|
||||
manifest +
|
||||
'x'.repeat(
|
||||
MAX_SKILL_ARCHIVE_MANIFEST_BYTES - Buffer.byteLength(manifest),
|
||||
),
|
||||
},
|
||||
]);
|
||||
await expect(
|
||||
installWorkspaceSkill(await workspace(), request(file)),
|
||||
).resolves.toMatchObject({ skillName: 'example' });
|
||||
});
|
||||
|
||||
it.each(['directories', 'metadata'])(
|
||||
'counts %s before filtering and does not count implicit parents',
|
||||
async (kind) => {
|
||||
const entries = [
|
||||
{ name: 'SKILL.md', content: manifest },
|
||||
{ name: 'implicit/parents/resource.txt', content: 'resource' },
|
||||
...Array.from({ length: MAX_SKILL_ARCHIVE_ENTRIES - 2 }, (_, i) => ({
|
||||
name:
|
||||
kind === 'directories'
|
||||
? `directory-${i}/`
|
||||
: `__MACOSX/ignored-${i}`,
|
||||
content: '',
|
||||
})),
|
||||
];
|
||||
await expect(
|
||||
installWorkspaceSkill(
|
||||
await workspace(),
|
||||
request(await archive(entries)),
|
||||
),
|
||||
).resolves.toMatchObject({ skillName: 'example' });
|
||||
entries.push({
|
||||
name: kind === 'directories' ? 'one-more/' : '__MACOSX/one-more',
|
||||
content: '',
|
||||
});
|
||||
await expect(
|
||||
installWorkspaceSkill(
|
||||
await workspace(),
|
||||
request(await archive(entries)),
|
||||
),
|
||||
).rejects.toMatchObject({ code: 'skill_too_many_files' });
|
||||
},
|
||||
);
|
||||
|
||||
it('does not inflate ignored metadata or charge it to the expanded budget', async () => {
|
||||
const file = await archive([
|
||||
{ name: 'SKILL.md', content: manifest },
|
||||
{ name: '__MACOSX/ignored', bytes: MAX_SKILL_ARCHIVE_EXPANDED_BYTES + 1 },
|
||||
]);
|
||||
const result = await installWorkspaceSkill(
|
||||
await workspace(),
|
||||
request(file),
|
||||
);
|
||||
await expect(
|
||||
fs.readdir(path.dirname(result.installedPath!)),
|
||||
).resolves.toEqual(['SKILL.md']);
|
||||
});
|
||||
|
||||
it('retains a failed rollback backup across a later installation', async () => {
|
||||
const root = await workspace();
|
||||
const file = await archive([{ name: 'SKILL.md', content: manifest }]);
|
||||
await installWorkspaceSkill(root, request(file));
|
||||
const rename = fs.rename.bind(fs);
|
||||
let published = false;
|
||||
const mocked = vi
|
||||
.spyOn(fs, 'rename')
|
||||
.mockImplementation(async (source, target) => {
|
||||
if (String(source).includes('.backup-'))
|
||||
throw new Error('restore failed');
|
||||
await rename(source, target);
|
||||
if (String(source).includes('.installing-')) published = true;
|
||||
});
|
||||
const committed = vi.fn();
|
||||
const error = await installWorkspaceSkill(
|
||||
root,
|
||||
request(file),
|
||||
undefined,
|
||||
() => {
|
||||
if (published)
|
||||
throw Object.assign(new Error('generation closed'), {
|
||||
name: 'WorkspaceGenerationClosedError',
|
||||
});
|
||||
},
|
||||
{ onCommitted: committed },
|
||||
).catch((error: unknown) => error);
|
||||
expect(error).toBeInstanceOf(WorkspaceSkillRecoveryError);
|
||||
expect(committed).not.toHaveBeenCalled();
|
||||
if (!(error instanceof WorkspaceSkillRecoveryError)) throw error;
|
||||
const backup = error.retainedPaths.find((file) =>
|
||||
file.includes('.backup-'),
|
||||
)!;
|
||||
await expect(
|
||||
fs.readFile(path.join(backup, 'SKILL.md'), 'utf8'),
|
||||
).resolves.toBe(manifest);
|
||||
mocked.mockRestore();
|
||||
await installWorkspaceSkill(root, request(file));
|
||||
await expect(
|
||||
fs.readFile(path.join(backup, 'SKILL.md'), 'utf8'),
|
||||
).resolves.toBe(manifest);
|
||||
});
|
||||
|
||||
it('finishes publication after admission and records commit before backup cleanup', async () => {
|
||||
const root = await workspace();
|
||||
const file = await archive([{ name: 'SKILL.md', content: manifest }]);
|
||||
await installWorkspaceSkill(root, request(file));
|
||||
const controller = new AbortController();
|
||||
const events: string[] = [];
|
||||
const committed = vi.fn(() => events.push('committed'));
|
||||
const rename = vi.spyOn(fs, 'rename');
|
||||
const rm = fs.rm.bind(fs);
|
||||
vi.spyOn(fs, 'rm').mockImplementation(async (...args) => {
|
||||
if (String(args[0]).includes('.backup-')) events.push('backup cleanup');
|
||||
return rm(...args);
|
||||
});
|
||||
await expect(
|
||||
installWorkspaceSkill(root, request(file), undefined, undefined, {
|
||||
signal: controller.signal,
|
||||
beforePublish: () => {
|
||||
expect(rename).not.toHaveBeenCalled();
|
||||
controller.abort();
|
||||
},
|
||||
onCommitted: committed,
|
||||
}),
|
||||
).resolves.toMatchObject({ skillName: 'example' });
|
||||
expect(committed).toHaveBeenCalledOnce();
|
||||
expect(events).toEqual(['committed', 'backup cleanup']);
|
||||
});
|
||||
});
|
||||
@@ -1136,6 +1136,7 @@ describe('workspace Skill management', () => {
|
||||
const nativeMkdir = fs.mkdir.bind(fs);
|
||||
const denied: string[] = [];
|
||||
const destroySpies: Array<MockInstance<(error?: Error) => Readable>> = [];
|
||||
const entryStreams: Readable[] = [];
|
||||
const nativeOpenReadStream = ZipFile.prototype.openReadStream;
|
||||
vi.spyOn(ZipFile.prototype, 'openReadStream').mockImplementation(
|
||||
function (
|
||||
@@ -1144,7 +1145,10 @@ describe('workspace Skill management', () => {
|
||||
callback: (error: Error | null, stream: Readable) => void,
|
||||
) {
|
||||
return nativeOpenReadStream.call(this, entry, (error, stream) => {
|
||||
if (stream) destroySpies.push(vi.spyOn(stream, 'destroy'));
|
||||
if (stream) {
|
||||
entryStreams.push(stream);
|
||||
destroySpies.push(vi.spyOn(stream, 'destroy'));
|
||||
}
|
||||
callback(error, stream);
|
||||
});
|
||||
} as typeof ZipFile.prototype.openReadStream,
|
||||
@@ -1187,7 +1191,8 @@ describe('workspace Skill management', () => {
|
||||
}
|
||||
}
|
||||
expect(destroySpies).toHaveLength(1);
|
||||
expect(destroySpies[0]).toHaveBeenCalledTimes(1);
|
||||
expect(destroySpies[0]).toHaveBeenCalled();
|
||||
expect(entryStreams[0].destroyed).toBe(true);
|
||||
await expectNoPartialInstall(workspace, 'zip-write-denied');
|
||||
},
|
||||
);
|
||||
@@ -1474,7 +1479,7 @@ describe('workspace Skill management', () => {
|
||||
).resolves.toEqual([]);
|
||||
});
|
||||
|
||||
it('removes legacy install artifacts before installing', async () => {
|
||||
it('removes abandoned staging but preserves recovery backups before installing', async () => {
|
||||
const workspace = await temporaryDirectory('qwen-skill-workspace-');
|
||||
const source = await temporaryDirectory('qwen-skill-source-');
|
||||
const baseDir = path.join(workspace, '.qwen', 'skills');
|
||||
@@ -1515,9 +1520,9 @@ describe('workspace Skill management', () => {
|
||||
source: { type: 'folder', path: source },
|
||||
});
|
||||
|
||||
await expect(fs.access(legacyBackup)).rejects.toThrow();
|
||||
await expect(fs.access(staleBackup)).rejects.toThrow();
|
||||
await expect(fs.access(datedBackup)).rejects.toThrow();
|
||||
await expect(fs.access(legacyBackup)).resolves.toBeUndefined();
|
||||
await expect(fs.access(staleBackup)).resolves.toBeUndefined();
|
||||
await expect(fs.access(datedBackup)).resolves.toBeUndefined();
|
||||
await expect(fs.access(randomStaging)).rejects.toThrow();
|
||||
await expect(fs.access(foreignBackup)).resolves.toBeUndefined();
|
||||
});
|
||||
|
||||
@@ -7,16 +7,24 @@ import { PassThrough, Transform, type Readable } from 'node:stream';
|
||||
import { pipeline } from 'node:stream/promises';
|
||||
import { promisify } from 'node:util';
|
||||
|
||||
import { SkillManager, Storage, type Config } from '@qwen-code/qwen-code-core';
|
||||
import type { Config } from '@qwen-code/qwen-code-core';
|
||||
import { SkillManager } from '@qwen-code/qwen-code-core/skills/skill-manager.js';
|
||||
import { Storage } from '@qwen-code/qwen-code-core/config/storage.js';
|
||||
import { sanitizeChildEnv } from '@qwen-code/qwen-code-core/utils/sanitize-child-env.js';
|
||||
import { fromBuffer, type Entry, type ZipFile } from 'yauzl';
|
||||
import {
|
||||
fromBuffer,
|
||||
open as openZipFile,
|
||||
type Entry,
|
||||
type ZipFile,
|
||||
} from 'yauzl';
|
||||
|
||||
export type WorkspaceSkillScope = 'workspace' | 'global';
|
||||
|
||||
export type WorkspaceSkillInstallSource =
|
||||
| { type: 'github'; url: string }
|
||||
| { type: 'folder'; path: string }
|
||||
| { type: 'zip'; contentBase64: string };
|
||||
| { type: 'zip'; contentBase64: string }
|
||||
| { type: 'prepared-zip'; path: string };
|
||||
|
||||
export interface WorkspaceSkillInstallRequest {
|
||||
name: string;
|
||||
@@ -31,12 +39,32 @@ export interface WorkspaceSkillMutationResult {
|
||||
deleted?: boolean;
|
||||
}
|
||||
|
||||
export interface WorkspaceSkillInstallOptions {
|
||||
signal?: AbortSignal;
|
||||
beforePublish?: () => void;
|
||||
onCommitted?: (result: WorkspaceSkillMutationResult) => void;
|
||||
}
|
||||
|
||||
// Largest raw ZIP that fits in the 10 MiB JSON body:
|
||||
// 3 * floor((10 * 1024 * 1024 - 330) / 4).
|
||||
export const MAX_ZIP_ARCHIVE_BYTES = 7_864_071;
|
||||
export const MAX_FILE_BYTES = 32 * 1024 * 1024;
|
||||
export const MAX_PACKAGE_BYTES = 64 * 1024 * 1024;
|
||||
export const MAX_FILES = 2000;
|
||||
export const MAX_SKILL_ARCHIVE_BYTES = 50 * 1024 * 1024;
|
||||
export const MAX_SKILL_ARCHIVE_EXPANDED_BYTES = 200 * 1024 * 1024;
|
||||
export const MAX_SKILL_ARCHIVE_ENTRIES = 4096;
|
||||
export const MAX_SKILL_ARCHIVE_MANIFEST_BYTES = 32 * 1024 * 1024;
|
||||
const LEGACY_ZIP_LIMITS = {
|
||||
fileBytes: MAX_FILE_BYTES,
|
||||
packageBytes: MAX_PACKAGE_BYTES,
|
||||
files: MAX_FILES,
|
||||
};
|
||||
const FILE_ZIP_LIMITS = {
|
||||
fileBytes: MAX_SKILL_ARCHIVE_EXPANDED_BYTES,
|
||||
packageBytes: MAX_SKILL_ARCHIVE_EXPANDED_BYTES,
|
||||
files: MAX_SKILL_ARCHIVE_ENTRIES,
|
||||
};
|
||||
const MAX_PATH_LENGTH = 512;
|
||||
const MAX_PATH_DEPTH = 16;
|
||||
// GitHub returns at most 1000 entries per Contents API listing and silently
|
||||
@@ -93,6 +121,20 @@ export class WorkspaceSkillManagementError extends Error {
|
||||
}
|
||||
}
|
||||
|
||||
export class WorkspaceSkillRecoveryError extends WorkspaceSkillManagementError {
|
||||
constructor(
|
||||
readonly retainedPaths: string[],
|
||||
cause: unknown,
|
||||
) {
|
||||
super(
|
||||
'skill_install_recovery_required',
|
||||
'Skill installation could not restore the previous state; recovery materials were retained',
|
||||
500,
|
||||
);
|
||||
this.cause = cause;
|
||||
}
|
||||
}
|
||||
|
||||
function skillError(code: string, message: string, statusCode = 400): never {
|
||||
throw new WorkspaceSkillManagementError(code, message, statusCode);
|
||||
}
|
||||
@@ -249,6 +291,7 @@ function beginStagedFile(
|
||||
relativePath: string,
|
||||
state: PackageState,
|
||||
tooManyMessage: string,
|
||||
maxFiles = MAX_FILES,
|
||||
): StagedFile | undefined {
|
||||
if (isPlatformMetadataPath(relativePath)) return undefined;
|
||||
const normalized = normalizeRelativePath(relativePath);
|
||||
@@ -264,7 +307,7 @@ function beginStagedFile(
|
||||
`Skill package has conflicting file and directory entries: ${normalized}`,
|
||||
);
|
||||
}
|
||||
if (state.count >= MAX_FILES) {
|
||||
if (state.count >= maxFiles) {
|
||||
skillError('skill_too_many_files', tooManyMessage);
|
||||
}
|
||||
return {
|
||||
@@ -717,12 +760,17 @@ async function downloadGitHubSkill(
|
||||
}
|
||||
}
|
||||
|
||||
function openZip(content: Buffer): Promise<ZipFile> {
|
||||
function openZip(content: Buffer | string): Promise<ZipFile> {
|
||||
return new Promise((resolve, reject) => {
|
||||
fromBuffer(content, { lazyEntries: true }, (error, zipFile) => {
|
||||
const callback = (error: Error | null, zipFile: ZipFile) => {
|
||||
if (error) reject(error);
|
||||
else resolve(zipFile);
|
||||
});
|
||||
};
|
||||
if (typeof content === 'string') {
|
||||
openZipFile(content, { lazyEntries: true }, callback);
|
||||
} else {
|
||||
fromBuffer(content, { lazyEntries: true }, callback);
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
@@ -740,10 +788,12 @@ async function writeZipEntry(
|
||||
entry: Entry,
|
||||
target: string,
|
||||
totalBytes: number,
|
||||
limits: typeof LEGACY_ZIP_LIMITS,
|
||||
signal?: AbortSignal,
|
||||
exclusive = false,
|
||||
): Promise<number> {
|
||||
const stream = await openZipStream(zipFile, entry);
|
||||
// yauzl keeps its own destroy() on a deflated entry's stream and never
|
||||
// emits 'close', so pipeline() must own a stream it can close itself.
|
||||
// An owned bridge keeps cancellation independent of yauzl's stream teardown.
|
||||
const source = new PassThrough();
|
||||
stream.on('error', (error) => source.destroy(error));
|
||||
stream.pipe(source);
|
||||
@@ -751,7 +801,7 @@ async function writeZipEntry(
|
||||
const guard = new Transform({
|
||||
transform(chunk: Buffer, _encoding, callback) {
|
||||
size += chunk.length;
|
||||
if (size > MAX_FILE_BYTES) {
|
||||
if (size > limits.fileBytes) {
|
||||
callback(
|
||||
new WorkspaceSkillManagementError(
|
||||
'skill_package_too_large',
|
||||
@@ -761,7 +811,7 @@ async function writeZipEntry(
|
||||
);
|
||||
return;
|
||||
}
|
||||
if (totalBytes + size > MAX_PACKAGE_BYTES) {
|
||||
if (totalBytes + size > limits.packageBytes) {
|
||||
callback(
|
||||
new WorkspaceSkillManagementError(
|
||||
'skill_package_too_large',
|
||||
@@ -775,7 +825,14 @@ async function writeZipEntry(
|
||||
},
|
||||
});
|
||||
try {
|
||||
await stageWrite(() => pipeline(source, guard, createWriteStream(target)));
|
||||
await stageWrite(() =>
|
||||
pipeline(
|
||||
source,
|
||||
guard,
|
||||
createWriteStream(target, { flags: exclusive ? 'wx' : 'w' }),
|
||||
{ signal },
|
||||
),
|
||||
);
|
||||
} finally {
|
||||
stream.destroy();
|
||||
source.destroy();
|
||||
@@ -784,24 +841,73 @@ async function writeZipEntry(
|
||||
}
|
||||
|
||||
async function filesFromZip(
|
||||
content: Buffer,
|
||||
content: Buffer | string,
|
||||
staging: string,
|
||||
state: PackageState,
|
||||
assertGenerationOpen?: () => void,
|
||||
signal?: AbortSignal,
|
||||
): Promise<void> {
|
||||
const nativeArchive = typeof content === 'string';
|
||||
const limits = nativeArchive ? FILE_ZIP_LIMITS : LEGACY_ZIP_LIMITS;
|
||||
signal?.throwIfAborted();
|
||||
const zipFile = await openZip(content);
|
||||
await new Promise<void>((resolve, reject) => {
|
||||
let settled = false;
|
||||
let writing: Promise<void> | undefined;
|
||||
let entries = 0;
|
||||
const paths = new Map<string, boolean>();
|
||||
const finish = (error?: unknown) => {
|
||||
if (settled) return;
|
||||
settled = true;
|
||||
signal?.removeEventListener('abort', onAbort);
|
||||
zipFile.close();
|
||||
if (error) reject(error);
|
||||
else resolve();
|
||||
// Closing the reader must not release the lock while a file write is live.
|
||||
void (writing ?? Promise.resolve())
|
||||
.catch(() => undefined)
|
||||
.then(() => {
|
||||
if (error) reject(error);
|
||||
else resolve();
|
||||
});
|
||||
};
|
||||
const onAbort = () => finish(signal?.reason);
|
||||
const writeEntry = async (entry: Entry) => {
|
||||
signal?.throwIfAborted();
|
||||
assertGenerationOpen?.();
|
||||
const mode = (entry.externalFileAttributes >>> 16) & 0xffff;
|
||||
const fileType = mode & 0xf000;
|
||||
if (nativeArchive) {
|
||||
entries += 1;
|
||||
if (entries > MAX_SKILL_ARCHIVE_ENTRIES) {
|
||||
skillError(
|
||||
'skill_too_many_files',
|
||||
'Skill ZIP contains too many entries (max 4096)',
|
||||
);
|
||||
}
|
||||
const directory = entry.fileName.endsWith('/') || fileType === 0x4000;
|
||||
const normalized = normalizeRelativePath(
|
||||
entry.fileName.replace(/\/+$/, ''),
|
||||
);
|
||||
if (fileType !== 0 && fileType !== 0x8000 && fileType !== 0x4000) {
|
||||
skillError(
|
||||
'invalid_skill_package',
|
||||
'Skill ZIP contains a link or special file',
|
||||
);
|
||||
}
|
||||
if (
|
||||
paths.has(normalized) ||
|
||||
[...paths].some(
|
||||
([existing, isDirectory]) =>
|
||||
(!isDirectory && normalized.startsWith(`${existing}/`)) ||
|
||||
(!directory && existing.startsWith(`${normalized}/`)),
|
||||
)
|
||||
) {
|
||||
skillError(
|
||||
'invalid_skill_package',
|
||||
`Skill ZIP has conflicting paths: ${normalized}`,
|
||||
);
|
||||
}
|
||||
paths.set(normalized, directory);
|
||||
}
|
||||
if (fileType === 0xa000) {
|
||||
throw new Error('Skill ZIP contains a symbolic link');
|
||||
}
|
||||
@@ -814,16 +920,17 @@ async function filesFromZip(
|
||||
entry.fileName,
|
||||
state,
|
||||
ZIP_TOO_MANY,
|
||||
limits.files,
|
||||
);
|
||||
if (!staged) return;
|
||||
if (entry.uncompressedSize > MAX_FILE_BYTES) {
|
||||
if (entry.uncompressedSize > limits.fileBytes) {
|
||||
skillError(
|
||||
'skill_package_too_large',
|
||||
'Skill ZIP entry exceeds the allowed size',
|
||||
413,
|
||||
);
|
||||
}
|
||||
if (state.totalBytes + entry.uncompressedSize > MAX_PACKAGE_BYTES) {
|
||||
if (state.totalBytes + entry.uncompressedSize > limits.packageBytes) {
|
||||
skillError(
|
||||
'skill_package_too_large',
|
||||
'Skill ZIP expands beyond the allowed size',
|
||||
@@ -836,16 +943,26 @@ async function filesFromZip(
|
||||
entry,
|
||||
staged.target,
|
||||
state.totalBytes,
|
||||
limits,
|
||||
signal,
|
||||
nativeArchive,
|
||||
);
|
||||
commitStagedFile(staged.normalized, written, state);
|
||||
};
|
||||
zipFile.on('error', finish);
|
||||
zipFile.on('entry', (entry: Entry) => {
|
||||
void writeEntry(entry).then(() => {
|
||||
if (settled) return;
|
||||
writing = writeEntry(entry);
|
||||
void writing.then(() => {
|
||||
if (!settled) zipFile.readEntry();
|
||||
}, finish);
|
||||
});
|
||||
zipFile.on('end', () => finish());
|
||||
signal?.addEventListener('abort', onAbort, { once: true });
|
||||
if (signal?.aborted) {
|
||||
onAbort();
|
||||
return;
|
||||
}
|
||||
zipFile.readEntry();
|
||||
});
|
||||
}
|
||||
@@ -1027,6 +1144,7 @@ async function stageSkillSource(
|
||||
staging: string,
|
||||
githubToken: string | undefined,
|
||||
assertGenerationOpen?: () => void,
|
||||
signal?: AbortSignal,
|
||||
): Promise<void> {
|
||||
const state: PackageState = { count: 0, totalBytes: 0, paths: new Set() };
|
||||
try {
|
||||
@@ -1042,14 +1160,18 @@ async function stageSkillSource(
|
||||
await copyFolder(source.path, staging, state, assertGenerationOpen);
|
||||
} else {
|
||||
await filesFromZip(
|
||||
decodeBase64(source.contentBase64),
|
||||
source.type === 'prepared-zip'
|
||||
? source.path
|
||||
: decodeBase64(source.contentBase64),
|
||||
staging,
|
||||
state,
|
||||
assertGenerationOpen,
|
||||
signal,
|
||||
);
|
||||
}
|
||||
await finalizeStagedPackage(staging, state, assertGenerationOpen);
|
||||
} catch (error) {
|
||||
signal?.throwIfAborted();
|
||||
if (error instanceof WorkspaceSkillManagementError) throw error;
|
||||
if (
|
||||
error instanceof Error &&
|
||||
@@ -1090,11 +1212,11 @@ async function removeInstallArtifacts(
|
||||
const locations = [
|
||||
{
|
||||
directory: baseDir,
|
||||
prefixes: [`.${skillName}.installing-`, `.${skillName}.backup-`],
|
||||
prefixes: [`.${skillName}.installing-`],
|
||||
},
|
||||
{
|
||||
directory: workDir,
|
||||
prefixes: [`${workPrefix}.installing-`, `${workPrefix}.backup-`],
|
||||
prefixes: [`${workPrefix}.installing-`],
|
||||
},
|
||||
];
|
||||
for (const { directory, prefixes } of locations) {
|
||||
@@ -1156,7 +1278,9 @@ export async function installWorkspaceSkill(
|
||||
request: WorkspaceSkillInstallRequest,
|
||||
githubToken?: string,
|
||||
assertGenerationOpen?: () => void,
|
||||
options: WorkspaceSkillInstallOptions = {},
|
||||
): Promise<WorkspaceSkillMutationResult> {
|
||||
options.signal?.throwIfAborted();
|
||||
assertGenerationOpen?.();
|
||||
const skillName = validateWorkspaceSkillName(request.name);
|
||||
rejectInstallArtifactSkillName(skillName);
|
||||
@@ -1203,8 +1327,20 @@ export async function installWorkspaceSkill(
|
||||
staging,
|
||||
githubToken,
|
||||
assertGenerationOpen,
|
||||
options.signal,
|
||||
);
|
||||
const skillFile = path.join(staging, 'SKILL.md');
|
||||
options.signal?.throwIfAborted();
|
||||
if (
|
||||
request.source.type === 'prepared-zip' &&
|
||||
(await fs.stat(skillFile)).size > MAX_SKILL_ARCHIVE_MANIFEST_BYTES
|
||||
) {
|
||||
skillError(
|
||||
'skill_package_too_large',
|
||||
'Root SKILL.md exceeds 32 MiB',
|
||||
413,
|
||||
);
|
||||
}
|
||||
let parsed: ReturnType<SkillManager['parseSkillContent']>;
|
||||
try {
|
||||
parsed = new SkillManager({} as Config).parseSkillContent(
|
||||
@@ -1224,6 +1360,9 @@ export async function installWorkspaceSkill(
|
||||
`Skill name "${parsed.name}" does not match requested name "${skillName}"`,
|
||||
);
|
||||
}
|
||||
options.signal?.throwIfAborted();
|
||||
assertGenerationOpen?.();
|
||||
options.beforePublish?.();
|
||||
if (existing) {
|
||||
assertGenerationOpen?.();
|
||||
liveInstallArtifacts.add(backup);
|
||||
@@ -1235,30 +1374,46 @@ export async function installWorkspaceSkill(
|
||||
committedStaging = true;
|
||||
assertGenerationOpen?.();
|
||||
} catch (error) {
|
||||
await fs
|
||||
.rm(committedStaging ? destination : staging, {
|
||||
let recoveryError: unknown;
|
||||
try {
|
||||
await fs.rm(committedStaging ? destination : staging, {
|
||||
recursive: true,
|
||||
force: true,
|
||||
})
|
||||
.catch(() => undefined);
|
||||
});
|
||||
} catch (cleanupError) {
|
||||
if (committedStaging) recoveryError = cleanupError;
|
||||
}
|
||||
if (movedExisting) {
|
||||
await fs.rename(backup, destination).catch(() => undefined);
|
||||
try {
|
||||
await fs.rename(backup, destination);
|
||||
recoveryError = undefined;
|
||||
} catch (restoreError) {
|
||||
recoveryError = restoreError;
|
||||
}
|
||||
}
|
||||
if (recoveryError !== undefined) {
|
||||
throw new WorkspaceSkillRecoveryError(
|
||||
[destination, ...(movedExisting ? [backup] : []), staging],
|
||||
new AggregateError([error, recoveryError]),
|
||||
);
|
||||
}
|
||||
throw error;
|
||||
} finally {
|
||||
liveInstallArtifacts.delete(staging);
|
||||
liveInstallArtifacts.delete(backup);
|
||||
}
|
||||
const result = {
|
||||
skillName,
|
||||
scope: request.scope,
|
||||
installedPath: path.join(destination, 'SKILL.md'),
|
||||
};
|
||||
options.onCommitted?.(result);
|
||||
if (movedExisting) {
|
||||
await fs
|
||||
.rm(backup, { recursive: true, force: true })
|
||||
.catch(() => undefined);
|
||||
}
|
||||
return {
|
||||
skillName,
|
||||
scope: request.scope,
|
||||
installedPath: path.join(destination, 'SKILL.md'),
|
||||
};
|
||||
return result;
|
||||
}
|
||||
|
||||
export async function deleteWorkspaceSkill(
|
||||
|
||||
Generated
+2
-2
@@ -856,8 +856,8 @@ importers:
|
||||
specifier: ^17.7.2
|
||||
version: 17.7.2
|
||||
yauzl:
|
||||
specifier: ^2.10.0
|
||||
version: 2.10.0
|
||||
specifier: ^3.4.0
|
||||
version: 3.4.0
|
||||
zod:
|
||||
specifier: ^3.23.8
|
||||
version: 3.25.76
|
||||
|
||||
Reference in new issue
Block a user