Nicholas TindleandClaude Opus 5.5
ccb658aa39
### Why / What / How **Why:** an ordinary skill save, from Otto, the skills API or a marketplace install or update, wrote a package one file at a time. It overwrote changed sibling files, deleted stale ones, then wrote `SKILL.md`. If a later step failed, the save failed and version history didn't advance. The workspace kept the new siblings, possibly minus files the old `SKILL.md` still names, next to the old root. The loader served that mix as an unversioned skill until a complete retry repaired it, and learning could import it into history as an untracked edit. Both review models flagged it on #14551 (Linear SECRT-2939). The old code's own comment said so: "Not a rollback: a file already here keeps the new bytes, so an upsert can fail mixed." **What:** a package write is now a single change. Readers see the whole old package or the whole new one. A failed or interrupted save leaves the previous package exactly as it was, and retrying applies it. **How:** - `WorkspaceManager.write_skill_package(root, files, removed_paths, scanned_checksums)` (`util/workspace.py`) works in two steps: 1. Checks the size limit for every file and the storage quota for the whole package (net of the rows it replaces or removes), scans what needs scanning, and stores every changed file as a new blob. 2. Calls `publish_workspace_skill_file` once. If anything fails, cancellation included (a `BaseException` handler, with the cleanup in `asyncio.shield`), it deletes the blobs it stored and re-raises. The one exception: before deleting, it checks whether the new root row exists, since a publication whose reply was lost (for example over the database manager) may have committed, and its rows name those blobs. Only after the commit does it delete the replaced blobs and their search embeddings, best-effort as before, and schedule embeddings for the new files. - `publish_workspace_skill_file` (`data/workspace_skill.py`) takes the package's `files` and `removed_paths` on `WorkspaceSkillWrite`. - It runs in the existing transaction under the skill-capacity advisory lock. The root checks (exists, owned, capacity) run first, so a refused root touches nothing. - It then retires every live row the package replaces or drops, creates the new rows, and replaces the root. Paths are unique among live rows only, so a retired row frees its path within the same transaction. - Package paths must sit inside the root's folder, can't be the root, can't contain `..`, and must be distinct. - The publication reports the rows it retired. - The capacity, owned and exists errors moved into `publication_refusal`, shared with `write_file`'s single-root path, which is otherwise unchanged. - `_write_skill` (`copilot/tools/skills.py`) still skips unchanged siblings (same checksum and mode) and lists the stale paths under the write lock as before. It sends everything as one package write. `_delete_paths` had no other caller and is gone. - User scoping in `data/`: every query in the new code filters by `workspaceId`, like the existing root publication. The caller is the user's own `WorkspaceManager`, which resolves and authorizes every path (including an expert scope's grants) before the DB call. - Out of scope: the generic workspace file tools can still write a single file inside a skill folder; that's an out-of-band edit, which learning records as such. ### Changes 🏗️ - `backend/data/workspace_skill.py`: package files on `WorkspaceSkillWrite`, `_swap_package_files`, `_require_package_paths`, `retired_files`, `publication_refusal`. - `backend/util/workspace.py`: `SkillPackageFile`, `write_skill_package` and its helpers; `write_file` uses `publication_refusal`. - `backend/copilot/tools/skills.py`: `_write_skill` writes one package; `_delete_paths` removed. - `docs/platform/skill-learning.md`: the package-write contract. - Tests: - `data/workspace_skill_test.py`: real Postgres; - `copilot/tools/skills_test.py`: the shared fake manager gains an all-or-nothing `write_skill_package`; - `skill_package_versions_test.py` and `skill_update_recovery_test.py`: they used to inject failures through `delete_file`, which no longer runs for stale files. They now fail the `SKILL.md` write, the last step of the swap, after the stale file is already gone. ### Agents and large language models used - Claude Code (Claude Opus 5.5) ### Checklist 📋 #### For code changes: - [x] I have clearly listed my changes in the PR description - [x] I have made a test plan - [x] I have tested my changes according to the test plan: - [x] `workspace_skill_test.py::test_a_package_write_swaps_every_file_or_none` (real Postgres and local storage): writes a package, then makes an update fail in two ways. - The second blob store fails. - `_replace_root` fails inside the transaction, after the siblings were swapped. Each time, the live files and their bytes match the previous package, and the blob count on disk is unchanged. The update then succeeds: the root, `a.md` and `c.md` are new, `b.md` is gone, and only the three live blobs remain. - [x] `util/workspace_skill_test.py::test_a_failed_package_write_drops_only_unpublished_blobs[cancelled|refused|rolled_back|reply_lost]`: - a cancel during the third blob deletes the first two and never publishes; - a refused or rolled-back publication deletes all three; - a failed call whose root row exists keeps them. The cancelled and lost-reply cases fail with the old handler. Capy found the cancellation leak. - [x] `workspace_skill_test.py::test_package_files_must_sit_inside_their_package[4 cases]` and `::test_a_refused_root_leaves_its_package_files_alone` (a new root at capacity leaves an existing sibling row live and creates nothing). - [x] `skills_test.py::test_a_failed_re_store_leaves_the_previous_package_as_it_was`: a re-save whose third file fails leaves the workspace byte-for-byte as before. It fails against dev's `skills.py`, where the changed `kept.md` keeps its new bytes. - [x] `skill_package_versions_test.py::test_a_failed_package_swap_keeps_publication_pending_until_retry` and `skill_update_recovery_test.py::test_a_failed_marketplace_update_preserves_baseline_and_retries[False|True]`: with the root write failing, the stale file is still there, the publication stays pending or the baseline is kept, and a retry applies cleanly. - [x] Suites: - the skill tool tests, `copilot/learning`, the `util/workspace*` tests, and the skills, skill-learning and workspace API tests pass (754); - `data/workspace_skill_test.py` plus every `data/skill_*_test.py` pass against Postgres (81); - pyright: 0 errors; isort and black are clean. - [x] Not run locally: `skills_learning_test.py` (local scanner) and the store/experts DB tests. Those need RabbitMQ and the service stack, and fail the same way on a clean tree here; CI covers them. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 (Claude Code) <[email protected]>
13 KiB
Nightly skill learning
AutoPilot and Experts can turn work they actually verified into reusable
skills. This page describes what the first release does, what it does not
do, and how to inspect and control it. The feature is off by default and
enabled per cohort with the dream-skill-learning-enabled flag. The dreaming
pipeline runs learning after its memory stages when this flag is enabled.
Learning also has an independent nightly job, so it can run without enabling
other memory stages. Both paths share the same lease and revision ledger.
What happens
- During work the chat services record a bounded reference to each completed user turn as a learning source revision: message references, typed tool outcomes (an exit code, a completed run, a passing validation), and the user's explicit, unqualified confirmations. The record includes the persisted user message that started the turn, so a confirmation is attributed to a real stored message and never to an invented reference. No model writes a lesson at this point, and recording never blocks a reply.
- Overnight (03:00 in the user's timezone, with a capped catch-up after downtime) a per-user job reviews eligible revisions. A source is eligible only when it still exists, still belongs to the same owner and Expert, is not excluded, learning is not paused for that Expert, and it carries a concrete outcome signal. Plans, unanswered questions, and unsupported "done" claims are skipped without a model call. Idle accounts cost nothing: with no new eligible work the pass exits before any model call.
- The reviewer model (routed and billed like ordinary work) looks for
work worth keeping: repeatable multi-step workflows, non-obvious error
recoveries, and user corrections that improve recurring work. Trivial
successes, generic advice, chat logs, and duplicate procedures are skipped.
It proposes one
bounded change: update an existing skill, create a meaningfully distinct
one, or skip. Deterministic checks then require a valid slug, the required
sections (
Steps,Verification), a concrete verification statement, and citations of evidence spans that carry a checked outcome. Rejected metadata is never persisted. An existing skill can be rewritten only when its complete body was included in the review. Skills outside the bounded body window still appear by name and description to prevent duplicate creation. - Publication re-checks everything live inside one database
transaction: source eligibility, epoch and revision, the skill's own
policy, and the head version (compare-and-swap). A concurrent human edit,
pause, or exclusion produces a conflict or a refusal, never an overwrite.
Creation also checks that the target file or current version has not
appeared since review. Malformed existing files are preserved.
The workspace
SKILL.mdis written afterwards, inside the registry's per-owner write lock and only while the head still names that version: a write resumed after a newer human correction is abandoned rather than placed on top of it. An interrupted write is finished (or abandoned) by the next run without a second model call. Reading the skill also retries its current pending write, even with background learning disabled; an owner use-pause still prevents loading it. - Retrieval pins the exact version a conversation loads. A load is recorded as a load, never as a success. An existing conversation sees an overnight update at its next user turn without starting a new chat.
Every completed write to a skill — overnight, during work, an owner edit, a restore, or an import — appends an immutable version with its origin and the contributing source revisions. A version restored from history inherits the restored content's sources.
Learned skills stay private to the account and Expert that did the work. Personal conversations teach the account's personal assistant; an Expert's conversation teaches only that Expert. The learning process never creates a marketplace listing or automatically shares a skill. Existing imported skills require the owner's decision before an automated change is applied.
Evidence and states
Evidence labels are concrete: Worked once in the source, Requested by user, Loaded N times · Outcome unknown, Reported as working by you. "Passed checks in N later uses" is reserved for recorded checks and is not produced by an owner's report.
Recorded verification can be retrieved outside the newest 120 messages. Up to 40 retained verification references and nearby context are fetched within the same owner and source revision. Complete proof takes priority within the 24,000-character evidence budget; missing or oversized proof is still refused.
| State | Meaning |
|---|---|
| Ready to use | Published and passed the applicable checks |
| Needs your decision | A proposal for a human-controlled skill or an uncertain match |
| Blocked by content check | A credential-like value was found; only the pattern class and an ordinal step are recorded |
| Paused (learning) | Future automated changes are off for the skill or Expert |
| Paused (not in use) | The owner stopped using the skill; history is kept |
| Archived | Superseded, stale, or invalidated by a revoked source |
| Deleted | The owner deleted the skill; its history is kept, archived |
Controls
Learned skills can contain a concise SKILL.md entrypoint together with
reusable scripts, reference material, fixtures, and templates. The reviewer
uses code and material visible in verified evidence, and records the limits
of what was checked. A simple procedure can remain a single Markdown file.
Supporting files are saved with the immutable version, so retrying a save,
applying an update, or restoring a version preserves the corresponding
package. Older versions created before file snapshots were introduced
retain their existing Markdown-only restore behavior.
The Changes and decision views show added, changed, and removed package files with bounded text previews. The instructions editor changes the entrypoint; supporting file changes shown in a proposal are retained when applying an edited alternative. New scripts are not executed automatically during dreaming. A later task loads the package into its working directory and runs the applicable checks.
Discovery triggers may contain up to 512 characters each. The per-turn index displays a shortened combined hint when necessary; the full phrases remain in the saved skill. Metadata and package constraints are checked before a version is committed. A temporary storage failure retries the saved package without another review; a permanent validation failure is settled instead of retried indefinitely.
- Expert page → Skills: a recent-change line, per-skill learning lines, and a detail sheet with Summary, Changes (before/after by step, with the raw Markdown diff on request), Sources, and History. The sheet offers Automatic improvements, Pause learning, Stop using this skill, Edit skill (applies immediately), Report an outcome (evidence for a later review), and Restore (a new version; future use only).
- Settings → Memory: a learning history filtered by scope, origin, and state, and one list of open decisions across Experts. Personal (AutoPilot) skills open the same detail sheet here, with the same controls, since they have no Expert page.
- Chat: a once-per-load "Skill loaded" chip naming the exact version and origin. It links to that exact skill and version: the Expert page for an Expert skill, Settings → Memory for a personal one. A conversation also offers "Exclude from learning".
Editing always starts from the current version (restore a historical one first to build on it). The version the editor saw is submitted with the edit and checked under the same write lock, so a change that landed while the owner was typing — overnight or from another tab — makes the save a conflict with the draft kept, never a silent overwrite of that change. A human edit turns automatic improvements off for that skill unless the editor keeps them on. An owner restore also turns them off and records a suppression for the undone behaviour, so the same change is not reapplied silently; a near-identical proposal becomes a labelled decision instead. Excluding a source makes every version that depends on it unavailable, restores the newest independent version when one exists, and otherwise pauses the skill with an explanation. That fallback restore leaves automatic improvements off but does not suppress the behavior: independent valid evidence can support a new proposal for the owner to decide. A source the viewer can no longer open shows Evidence not visible to you without a title or link.
Deleting a skill ends its history. Its versions stay in History, archived, after a Deleted entry, and excluding one of their sources later restores nothing. The deleted behaviour is suppressed as an undone change is, so learning never reapplies it and a near-identical proposal becomes a decision. A genuinely different skill can still be learned under the same name; it starts a new history with the default controls.
Adapters
The learner never inspects an application's schema. A source adapter
implements revalidate(source_id, revision, scope, approval_event_id) and
load_evidence(...); approval-aware kinds also expose an approval
checkpoint (event id, actor, approved revision). Publication is conditional
on that live answer; a static approved flag in a queued payload is not
enough. The first release ships the ordinary-chat adapter only.
Operators
GET /api/skill-learning/status reports the backlog, the oldest pending
source, the last review and last applied change, reviews still retrying,
and the last 30 days of review cost. Every review disposition — applied,
skipped, no novel procedure, deferred, conflict, blocked by content check,
provider error, budget exhausted, stale eligibility, inaccessible evidence,
suppressed — is kept per source revision in the review ledger.
POST /api/skill-learning/run runs the same bounded learning pass immediately
for the authenticated account. It preserves the normal billing, scope,
content, and ownership checks. A second run with no new evidence makes no
model calls. Runs renew their lease between sources and check it again after
each model review; losing the lease leaves the source pending without
publishing the proposed change.
Validated paid review output can be retried without another model call, including when the current budget cannot fund new calls. Its checkpoint is limited to 512 KiB and seven days; owner, source, approval, and target state are checked again before applying it. Uncacheable failures and ambiguous crashes consume at most three model-call reservations per source revision and policy. After that limit the revision is skipped; new verified evidence can be considered in a later revision. A reservation does not prove that the provider completed a call.
At the saved-skill limit (150 for the account or for each Expert), learning saves nothing that would need another saved slot: a new skill, or an edit that would move an installed copy into the saved budget. The revision is recorded as paused before any version exists, and its paid review is kept for the retry above, so it is saved once there is room. If another save takes the last slot between that check and the write, the new version waits as a pending write and is saved, without another review, once there is room. Improvements to skills already saved continue.
Owner-approved content-check exceptions are stored with the immutable version as exact match hashes, file paths, and occurrence counts. Restoring or improving a version preserves only retained approved occurrences. New, duplicated, relocated, and seeded values still require the applicable checks.
Every registry package write replaces the whole package in one change: every file is stored first, then the package's files are swapped in a single database transaction. A failed or interrupted write leaves the previous package exactly as it was; retry it to apply the change. Versioned pending writes stay unavailable until recovery completes. Repeating source exclusion also resumes any interrupted head repair, while preserving an owner use-pause.
Not in the first release
A morning digest, per-section edit protection, selective forgetting cascades, automatic sharing between Experts, skill merges or renames, age-based archiving, additional source adapters, and any automatic execution of a newly written recipe.