mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-07 06:16:06 -10:00
feat: add portable verification skills and refresh the feature map (#15983)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Contributors use the feature map to select verification recipes. > - The repository needs separate skills to create a verifier, maintain recipes, and run checks. > - Shared recipes must match current routes, controls, and prerequisites. > - This pull request adds portable verification skills and updates the existing map. > - The benefit is a repeatable verification workflow with clear ownership and evidence limits. ## Linked Issues or Issue Description Refs: #15288, which added the existing feature map. The related test cleanup in #15978 changes test ownership; this PR changes skill instructions and recipes. **Issue type** Missing documentation and outdated verification instructions. **Where is the issue?** The project-local skills in `.agents/skills/` and the recipes in `feature-map/`. **What's wrong?** The repository lacks the portable creation and maintenance skills. The verifier mixes live verification with map editing. Some recipes describe old controls or omit required flags, identities, and provider setup. **Suggested fix** Import the skills and their complete references from Paperclip's pstack fork. Keep the existing map in its current directory. Let the verifier read recipes and report drift. Let the maintenance skill update the map. ## What Changed - Add `create-verification-skill` and `maintain-verification-skill` from the portable fork of Poteto's pstack. Include invocation policies, all feature-map examples, self-contained attribution notices and the full MIT license in each skill folder. - Use the existing `feature-map/` directory and Poteto's recipe format. Omit the fork's separate runtime skill and runtime-skill dependencies because Paperclip supplies execution. - Add `verify-paperclip` with executable launch, readiness, onboarding, and cleanup helpers. Use isolated temporary data and separate evidence directories. - Make `verify-paperclip` consume the map and report recipe drift. Give map editing to `maintain-verification-skill`. - Review all 35 feature files and correct 30 recipes. Clarify routes, flags, current controls, persistence scope, and the limits of linked tests. - Remove the unrelated Conference Room flag change from the onboarding smoke helper. - Add a standard-library ownership test. It proves that doctor and stop reject a foreign checkout without stopping its live server. A delayed-exit fixture also proves cleanup waits for runtime exit. - Keep audit screenshots and scratch reports outside the repository. No product code, migrations, lockfile, or workflow changes. ## Verification - Passed after rebase: metadata validation for all three skills, Python syntax and executable checks, JavaScript syntax, 35-entry map/index consistency, relative links, license preservation, and `git diff --check`. - Ran one isolated live audit before rebase. Checked task persistence, document upload/download bytes, goal hierarchy, project edits, search, local process Agent Chat, question replies, queued-message discard/stop, and a harmless loopback MCP call. - Checked local settings and draft flows. Synthetic cost events reached a small budget limit and paused the test agent. These checks do not prove external model quality or provider billing. - Checked cleanup of the audit server, database, local MCP fixture, and workspace service inventory. - Passed: `pnpm -r typecheck`. - Greptile: 5/5 on commit `4344ac4df7`. The cleanup finding is fixed and its review thread is resolved. A fresh review is pending for the notice relocation. - Passed: `python3 -m unittest discover -s .agents/skills/verify-paperclip/tests -v`. Both tests pass: the foreign fixture stayed alive after rejection, and owned cleanup waited for delayed runtime exit. - Passed: `pnpm build`. - All GitHub CI checks were green on commit `4344ac4df7`. The cloud-auth browser shard passed after one retry for a disposed mocked API response. The notice relocation passed skill metadata, relative-link, and license-preservation checks; fresh CI/review is pending. - The full local `pnpm test:run` did not complete. It was interrupted after nearly 30 minutes once the complete CI suite passed on the same commit. Exit code: 130. The new helper safety tests passed locally. The local full-suite checklist item stays unchecked. - Full live coverage is incomplete. Authenticated multi-user checks and external provider journeys need their own identities, credentials, and targets. - The corrected onboarding helper could not be re-tested after removing the flag mutation. Two fresh launches failed before server startup. Direct PostgreSQL initialization reported exhausted shared-memory IDs or limits. The host had free disk space. No host-wide limits or unrelated instances were changed. ## Risks - Recipe drift can return as the UI changes. The map records verification scope, not proof of a passed journey. - The new helpers assume a source checkout with installed dependencies, Python, Node, and `lsof`. The onboarding helper also needs Playwright. - The local audit found observations that need separate investigation: an action test returned fixture data while the connection control showed Off and native profiles existed; a failed status-card setup task left the card showing Setting up. Workspace service start also did not reach a running state. This PR does not change product behavior or redefine those outcomes as success. - The main audit trace did not finalize during browser shutdown. Screenshots, text captures, and iteration results were retained. - Live evidence was captured before rebase. The onboarding helper re-test and full provider/auth qualification remain incomplete. ## Model Used OpenAI GPT-6 through Codex assisted with this change. The session does not expose the exact serving model ID, configured context-window size, or reasoning setting. Capabilities used: source analysis, read-only source subagents, file edits, shell commands, and browser automation. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [ ] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <[email protected]>
This commit is contained in:
1 parent
8227485372
commit
cc88b61c09
46 files changed
+1086
-48
No files matched your search
@@ -0,0 +1,46 @@
|
||||
# create-verification-skill attribution
|
||||
|
||||
This skill comes from Paperclip's portable pstack fork in
|
||||
`paperclipai/agent-plugins`, commit
|
||||
`410257663e0b344df656e604f1601aa1db95f7e4`, under `vendor/pstack/`.
|
||||
The fork imported pstack 0.15.15 by Lauren Tan (poteto) from
|
||||
[cursor/plugins](https://github.com/cursor/plugins/tree/ccb5507cec1546dc88135c1139c811e6c59115ba/pstack),
|
||||
upstream revision `ccb5507cec1546dc88135c1139c811e6c59115ba`.
|
||||
|
||||
Paperclip adapts the workflow to the existing repo-root `feature-map/`.
|
||||
It preserves the invocation policy in `agents/openai.yaml`, removes the
|
||||
Cursor-specific `disable-model-invocation` frontmatter field, and omits the
|
||||
separate runtime skill dependency because Paperclip supplies execution.
|
||||
No upstream Cursor built-in skills are redistributed.
|
||||
|
||||
The imported resources include `SKILL.md`, `agents/openai.yaml`, and all three
|
||||
files in `references/feature-map-example/`. Keep those examples with the skill
|
||||
when copying or refreshing it.
|
||||
|
||||
## License
|
||||
|
||||
The original license is reproduced verbatim below.
|
||||
|
||||
```text
|
||||
MIT License
|
||||
|
||||
Copyright (c) 2026 Lauren Tan
|
||||
|
||||
Permission is hereby granted, free of charge, to any person obtaining a copy
|
||||
of this software and associated documentation files (the "Software"), to deal
|
||||
in the Software without restriction, including without limitation the rights
|
||||
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
||||
copies of the Software, and to permit persons to whom the Software is
|
||||
furnished to do so, subject to the following conditions:
|
||||
|
||||
The above copyright notice and this permission notice shall be included in all
|
||||
copies or substantial portions of the Software.
|
||||
|
||||
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
||||
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
||||
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
||||
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
||||
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
||||
SOFTWARE.
|
||||
```
|
||||
@@ -0,0 +1,45 @@
|
||||
---
|
||||
name: create-verification-skill
|
||||
description: "Generate a project-local verification skill that drives your app the way a user does — any language, framework, or platform. Use for /create-verification-skill, \"make a control skill for this repo\", or when a project has no scripted way to prove UI/CLI/service behavior."
|
||||
---
|
||||
|
||||
# Create a verification skill
|
||||
|
||||
Every serious project needs a scripted way to drive the real app and prove behavior: launch it, exercise a feature the way a user would, and capture evidence. This skill generates that as a project-local skill (`.agents/skills/verify-<app>/`) tailored to the repo. You write the generator's output for the next agent, not for a human: it will be read cold, mid-task, by an agent that has never seen the app.
|
||||
|
||||
## 1. Interview the repo, not the user
|
||||
|
||||
Answer these from the codebase and only ask the user what you cannot observe:
|
||||
|
||||
- **Surface:** what does a user actually touch? A web UI, a CLI/TUI, a desktop app, an API, a mobile app, a library? A repo can have several; pick the primary one and note the rest.
|
||||
- **Run:** how does the app start locally? Prefer the repo's own documented dev command (package scripts, Makefile, README quickstart). Note ports, env vars, seed data, auth.
|
||||
- **Drive:** how can an agent interact with it programmatically? Existing harnesses first — Playwright/Cypress specs, expect scripts, PTY helpers, curl-able endpoints, a debug port. Only then pick a generic recipe: browser/CDP for web and Electron, a tmux/PTY harness for CLI/TUI, plain HTTP for services.
|
||||
- **Observe:** what evidence can be captured? Screenshots, terminal transcripts, response bodies, logs, exit codes, DB state.
|
||||
- **Isolate:** can two instances run side by side (ports, data dirs, profiles)? If not, say so in the generated skill: refusing to double-drive a shared instance beats corrupting the user's session.
|
||||
|
||||
If the checkout doesn't build or start as-is, fix that first (or report it precisely) before generating; a skill written against a broken base teaches wrong steps. When an irrelevant missing asset blocks startup (a static dir the API never serves, a sample config), the generated skill may create it, clearly marked as verification scaffolding, and remove it in cleanup.
|
||||
|
||||
## 2. Generate the skill
|
||||
|
||||
Write `.agents/skills/verify-<app>/SKILL.md` with YAML frontmatter (`name: verify-<app>` and a `description` that names the app, the surface, and when to reach for it — without frontmatter the skill never registers) and these sections, each grounded in what the interview actually found (no placeholders left):
|
||||
|
||||
- **Launch:** the exact command that starts the app for verification, and how to tell it's ready (a log line, a port answering, a prompt). Include teardown. For a short-lived CLI or TUI there is no server to keep alive: launch means build the binary (or install deps) once, then start each drive in its own isolated PTY or tmux session.
|
||||
- **Doctor:** one read-only check that answers "is this instance worth driving?" — process up, right version/build, port owned by us, auth valid. An agent runs this first whenever anything looks off.
|
||||
- **Drive:** the harness recipe with real selectors/commands from this repo, not examples. Prefer stable handles (ARIA labels, data attributes, prompt strings, route paths) over coordinates and tab order.
|
||||
- **Evidence:** what to capture for a proof and where it goes. State the proof standards: exercise the real user path, not internal setters or test-only endpoints; capture the action and the resulting state, not just the final screen; verify side effects (files written, rows inserted, messages sent) alongside what's visible; mocks only where a production boundary already isolates the external system. When the safe path is a dry-run or test mode, verify what it actually skips by observing (files, network, git refs) rather than trusting its name: some dry-runs still touch the network or open a browser.
|
||||
- **Cleanup:** how to tear down instances the run created. Never kill by process name; kill what you started. Cleanup removes instances and scratch state, never the evidence: proof artifacts survive the teardown, in a location the skill names.
|
||||
- **Helpers:** any script the skill ships is executable and its invocation is shown in the skill body. A helper the reader has to reverse-engineer is not a helper.
|
||||
|
||||
## 3. Seed the feature map
|
||||
|
||||
For Paperclip, use the existing repo-root `feature-map/` directory and its `README.md`; have the generated verification skill reference that map instead of creating a second map inside the skill directory. Reuse existing feature files and use Poteto's format below for new or updated entries. For other projects, create `.agents/skills/verify-<app>/features/README.md` plus one file per user-facing feature you can identify (aim for the top 3-5 to start, from routes, commands, menus, or docs). Follow the shape in [`references/feature-map-example/`](references/feature-map-example/), with a README index and one file per feature. Each file answers, from the user's point of view: what the feature is, how to reach it, how to drive it with the harness, and what observable end state proves it works. The four H2s are `Sub-features`, `How to get to it (user POV)`, `Driving it with <harness>`, and `Gotchas`. The map is the repo's maintained verification source; a proof that drives one convenient entry point is incomplete when the map lists others.
|
||||
|
||||
## 4. Prove the generated skill before handing it over
|
||||
|
||||
Run its own instructions end to end once: launch, doctor, drive ONE mapped feature (one is enough; the map exists so later runs can cover the rest), capture evidence, clean up. After cleanup, confirm the evidence still exists at the named location — a cleanup that eats the proof fails this step. Fix what fails, and run the generated cleanup after every failed iteration too, so broken attempts don't strand processes and ports. A generated skill that was never executed is a draft, not a deliverable.
|
||||
|
||||
## 5. Offer the maintenance loop
|
||||
|
||||
Point the user at `/maintain-verification-skill` for keeping the map honest as the app changes. Suggest a cadence only if they ask.
|
||||
|
||||
Origin and license: [NOTICE.md](NOTICE.md).
|
||||
@@ -0,0 +1,2 @@
|
||||
policy:
|
||||
allow_implicit_invocation: false
|
||||
@@ -0,0 +1,47 @@
|
||||
# Notes verification map
|
||||
|
||||
This directory is the maintained source for verifying the user-facing behavior of Notes. Read the index before driving the app, then use the matching feature file as the recipe.
|
||||
|
||||
## Baseline preconditions
|
||||
|
||||
- Launch Notes at `http://127.0.0.1:4173` with a disposable data directory.
|
||||
- Set `NOTES_DATA_DIR=/tmp/notes-verify-$RUN_ID` so concurrent runs do not share state.
|
||||
- Seed notes titled `Quarterly plan` and `Grocery list`.
|
||||
- Put `control-notes` and the `notes` CLI on `PATH`.
|
||||
- Run `control-notes doctor` and require the expected URL, data directory, and build revision.
|
||||
- Never drive an instance that was not started by this verification run.
|
||||
|
||||
## Driving conventions
|
||||
|
||||
- Start every recipe from the baseline state unless its preconditions say otherwise.
|
||||
- Prefer ARIA roles and accessible names over CSS selectors or DOM position.
|
||||
- Treat every command as literal. Keep quoted names and flags unchanged.
|
||||
- Run browser actions through `control-notes browser`.
|
||||
- Run terminal actions through `control-notes cli -- <command>`.
|
||||
- Restore seeded data after a mutation. Do not remove proof artifacts during cleanup.
|
||||
|
||||
## Proof and skip reporting
|
||||
|
||||
- Capture the user action and the resulting state, not only the final screen.
|
||||
- UI proof includes an ARIA snapshot and a screenshot with the app identity visible.
|
||||
- CLI proof includes the command, stdout, stderr, and exit code.
|
||||
- Mutation proof includes a read-only second view of the stored value.
|
||||
- Record the feature ID and entry point used with every artifact.
|
||||
- Report an unreachable path with the attempted command and the unmet precondition.
|
||||
- Do not report a skipped entry point as verified through a different path.
|
||||
|
||||
## Feature entry contract
|
||||
|
||||
Each feature file starts with an H1 title and one paragraph describing the user-visible behavior. It then uses exactly four H2 sections in this order.
|
||||
|
||||
1. `Sub-features` lists short IDs with one line for each behavior.
|
||||
2. `How to get to it (user POV)` lists every user entry point.
|
||||
3. `Driving it with <harness>` starts with `Preconditions:` and uses labeled bullets that pair each user action with an exact command and observable result.
|
||||
4. `Gotchas` lists traps that can waste or invalidate a verification run.
|
||||
|
||||
Keep implementation details out of the map. Name only user paths, stable handles, required state, commands, and observable proof.
|
||||
|
||||
## Features
|
||||
|
||||
- [Create a note](./create-note.md) covers browser and CLI creation, cancellation, persistence, and cleanup.
|
||||
- [Search notes](./search.md) covers toolbar, keyboard, and CLI search with matching, empty, and clear states.
|
||||
+39
@@ -0,0 +1,39 @@
|
||||
# Create a note
|
||||
|
||||
Create note lets a user save a titled note from the browser or CLI, cancel an unfinished draft, and confirm the saved note from a second user-facing view.
|
||||
|
||||
## Sub-features
|
||||
|
||||
- `create-open` opens a blank editor from each browser entry point.
|
||||
- `create-save` persists a title and body.
|
||||
- `create-cancel` discards an unfinished browser draft.
|
||||
- `create-cli` creates the same note shape from the terminal.
|
||||
|
||||
## How to get to it (user POV)
|
||||
|
||||
- Choose the `New note` button in the browser toolbar.
|
||||
- Press `n` in the browser while focus is outside an editable field.
|
||||
- Run `notes create --title <title> --body <body>` in a terminal.
|
||||
|
||||
## Driving it with control-notes
|
||||
|
||||
Preconditions:
|
||||
|
||||
- Notes is healthy at `http://127.0.0.1:4173`.
|
||||
- No note is titled `Release checklist`.
|
||||
- `control-notes doctor` reports the expected URL and disposable data directory.
|
||||
|
||||
- **Open editor.** Choose `New note`. Run `control-notes browser click --role button --name "New note"`. A form named `Note editor` appears with focus in the `Title` textbox.
|
||||
- **Enter content.** Type the title and body. Run `control-notes browser fill --role textbox --name "Title" --value "Release checklist"` and `control-notes browser fill --role textbox --name "Body" --value "Tag and publish"`. The `Save note` button becomes enabled.
|
||||
- **Save note.** Choose `Save note`. Run `control-notes browser click --role button --name "Save note"`. A status named `Note saved` appears and the heading reads `Release checklist`.
|
||||
- **Confirm persistence.** Return to the note list and reopen the note. Run `control-notes browser click --role link --name "All notes"` and `control-notes browser click --role link --name "Release checklist"`. The editor shows both saved values.
|
||||
- **Cancel draft.** Open a new note, enter `Discard me`, and choose `Cancel`. Run `control-notes browser click --role button --name "New note"`, `control-notes browser fill --role textbox --name "Title" --value "Discard me"`, and `control-notes browser click --role button --name "Cancel"`. The note list returns and has no `Discard me` link.
|
||||
- **CLI entry.** Create a second note. Run `control-notes cli -- notes create --title "CLI note" --body "Created from terminal" --format json`. Exit code `0` and stdout contain the new note ID and title.
|
||||
- **Proof.** Reopen both saved notes from `All notes`. Run `control-notes browser snapshot --aria --path artifacts/create-note/list.aria.txt` and `control-notes browser screenshot --path artifacts/create-note/list.png`. The artifacts show `Release checklist` and `CLI note`.
|
||||
|
||||
## Gotchas
|
||||
|
||||
- Pressing `n` while a textbox has focus types the character instead of opening a new editor.
|
||||
- Titles are trimmed on save. Assert the rendered title, not the draft input value.
|
||||
- A save status alone is insufficient proof. Reopen the note from the list.
|
||||
- Remove `Release checklist` and `CLI note` during fixture cleanup, but retain their proof artifacts.
|
||||
@@ -0,0 +1,45 @@
|
||||
# Search notes
|
||||
|
||||
Search lets a user find notes by title or body text, inspect a matching note, and distinguish no matches from an unavailable search.
|
||||
|
||||
## Sub-features
|
||||
|
||||
- `search-open` opens search from each supported browser entry point.
|
||||
- `search-match` returns title and body matches without changing note data.
|
||||
- `search-open-result` opens a result in the note editor.
|
||||
- `search-empty` shows a complete empty state for a query with no matches.
|
||||
- `search-clear` removes the query and restores the recent-notes view.
|
||||
- `search-cli` returns the same matching notes from the terminal.
|
||||
|
||||
## How to get to it (user POV)
|
||||
|
||||
- Choose the `Search` button in the browser toolbar.
|
||||
- Press `/` in the browser while focus is outside an editable field.
|
||||
- Run `notes search <query>` in a terminal.
|
||||
|
||||
## Driving it with control-notes
|
||||
|
||||
Preconditions:
|
||||
|
||||
- Notes is healthy at `http://127.0.0.1:4173`.
|
||||
- The disposable data directory contains `Quarterly plan` with body text `Draft budget`.
|
||||
- `control-notes doctor` reports the expected URL and data directory.
|
||||
|
||||
- **Toolbar entry.** Choose the `Search` button. Run `control-notes browser click --role button --name "Search"`. A dialog named `Search notes` appears with focus in its searchbox.
|
||||
- **Keyboard entry.** Close the dialog, focus the page, and press `/`. Run `control-notes browser press --key "/"`. The same dialog appears and the page does not insert a slash.
|
||||
- **Title match.** Type `quarterly`. Run `control-notes browser fill --role searchbox --name "Search notes" --value "quarterly"`. The `Search results` list contains `Quarterly plan` and does not contain `Grocery list`.
|
||||
- **Body match.** Replace the query with `budget`. Run `control-notes browser fill --role searchbox --name "Search notes" --value "budget"`. The result `Quarterly plan` remains visible with a body-match excerpt.
|
||||
- **Open result.** Choose `Quarterly plan`. Run `control-notes browser click --role link --name "Quarterly plan"`. The dialog closes and the editor heading reads `Quarterly plan`.
|
||||
- **Empty state.** Reopen search and enter `volcano`. Run `control-notes browser fill --role searchbox --name "Search notes" --value "volcano"`. A status named `No matching notes` appears after search completes.
|
||||
- **Clear query.** Choose `Clear search`. Run `control-notes browser click --role button --name "Clear search"`. The searchbox is empty and the `Recent notes` region replaces the result list.
|
||||
- **CLI match.** Search from the terminal. Run `control-notes cli -- notes search "quarterly" --format json`. Exit code `0` and stdout contain one object whose title is `Quarterly plan`.
|
||||
- **CLI miss.** Search for an absent value. Run `control-notes cli -- notes search "volcano" --format json`. Exit code `0` and stdout are `[]`.
|
||||
- **Proof.** Capture the populated result state. Run `control-notes browser snapshot --aria --path artifacts/search/results.aria.txt` and `control-notes browser screenshot --path artifacts/search/results.png`. Both artifacts identify Notes, the query, and `Quarterly plan`.
|
||||
|
||||
## Gotchas
|
||||
|
||||
- Pressing `/` while the editor or searchbox has focus inserts text instead of opening search.
|
||||
- Results update after a short debounce. Wait for the results list or empty status, not a fixed sleep.
|
||||
- Archived notes are excluded unless the user enables `Include archived`.
|
||||
- The CLI defaults to human-readable output. Use `--format json` for stable assertions.
|
||||
- Opening a result changes browser state. Reopen search before proving another query.
|
||||
@@ -0,0 +1,46 @@
|
||||
# maintain-verification-skill attribution
|
||||
|
||||
This skill comes from Paperclip's portable pstack fork in
|
||||
`paperclipai/agent-plugins`, commit
|
||||
`410257663e0b344df656e604f1601aa1db95f7e4`, under `vendor/pstack/`.
|
||||
The fork imported pstack 0.15.15 by Lauren Tan (poteto) from
|
||||
[cursor/plugins](https://github.com/cursor/plugins/tree/ccb5507cec1546dc88135c1139c811e6c59115ba/pstack),
|
||||
upstream revision `ccb5507cec1546dc88135c1139c811e6c59115ba`.
|
||||
|
||||
Paperclip adapts the workflow to the existing repo-root `feature-map/`.
|
||||
It preserves the invocation policy in `agents/openai.yaml`, removes the
|
||||
Cursor-specific `disable-model-invocation` frontmatter field, and omits the
|
||||
separate runtime skill dependency because Paperclip supplies execution.
|
||||
No upstream Cursor built-in skills are redistributed.
|
||||
|
||||
The imported resources include `SKILL.md` and `agents/openai.yaml`.
|
||||
The workflow uses the examples shipped with
|
||||
[`create-verification-skill`](../create-verification-skill/references/feature-map-example/).
|
||||
|
||||
## License
|
||||
|
||||
The original license is reproduced verbatim below.
|
||||
|
||||
```text
|
||||
MIT License
|
||||
|
||||
Copyright (c) 2026 Lauren Tan
|
||||
|
||||
Permission is hereby granted, free of charge, to any person obtaining a copy
|
||||
of this software and associated documentation files (the "Software"), to deal
|
||||
in the Software without restriction, including without limitation the rights
|
||||
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
||||
copies of the Software, and to permit persons to whom the Software is
|
||||
furnished to do so, subject to the following conditions:
|
||||
|
||||
The above copyright notice and this permission notice shall be included in all
|
||||
copies or substantial portions of the Software.
|
||||
|
||||
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
||||
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
||||
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
||||
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
||||
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
||||
SOFTWARE.
|
||||
```
|
||||
@@ -0,0 +1,40 @@
|
||||
---
|
||||
name: maintain-verification-skill
|
||||
description: "Periodic pass that keeps a project's verification skill and feature map honest: parallel source readers per feature, one live session driving every feature, at most one PR of proven corrections. Use for /maintain-verification-skill or \"audit the verify skill\"."
|
||||
---
|
||||
|
||||
# Maintain a verification skill
|
||||
|
||||
A feature map rots the moment the app changes. This skill is the upkeep loop for a skill generated by `/create-verification-skill` (or any project-local verification skill with a feature map). The unit of rigor is the feature, not every sentence: cover every feature file from source and exercise every feature live, without terminalising every bullet.
|
||||
|
||||
## Outcomes
|
||||
|
||||
Pick one, and say which:
|
||||
|
||||
- **clean** — every feature got source and live coverage; nothing worth shipping. No branch, no PR.
|
||||
- **changed** — one PR ships proven doc, harness, or map corrections.
|
||||
- **blocked** — coverage could not finish or a proven fix could not ship safely. Say exactly what blocked it.
|
||||
|
||||
## Edit scope
|
||||
|
||||
Only edit the verification skill's own directory (its SKILL.md and any harness scripts it owns) and its feature-map directory. For Paperclip, the map is the existing repo-root `feature-map/`; maintain it in place instead of creating or moving it into a skill directory. Use [Poteto's feature-map format](../create-verification-skill/references/feature-map-example/) for new or updated entries. Never edit product code during a run: a behavior the map describes that the app no longer does is either doc drift (fix the map) or a product regression (report it, don't paper over it in docs).
|
||||
|
||||
## Pass
|
||||
|
||||
0. **Locate the target.** Find the verification skill to maintain: the project-local skill whose body has launch/drive sections and a feature map (usually `.agents/skills/verify-*/`). Several candidates → ask which one; none → stop and point at `/create-verification-skill` instead of inventing a target.
|
||||
|
||||
1. **Index hygiene.** Read the feature map README and glob its sibling files. Fix missing, extra, duplicate, or dead entries. Lightweight; no generated inventory.
|
||||
|
||||
2. **Source wave.** One read-only subagent per feature file, launched concurrently. Each explains "how does this user-facing feature work?" from source, flags likely doc drift with citations, and returns one concise live-verification recipe. Children never drive the app and never edit files. Return shape: feature summary / source entry points / likely drift or none / one recipe.
|
||||
|
||||
3. **Reconcile.** Every feature file has a returned summary. Merge overlapping recipes into as few app states as practical. Spot-check cited drift; don't re-prove clean claims. Sweep recent churn for user-facing surfaces missing from the map — require a concrete source path before calling one missing.
|
||||
|
||||
4. **Live pass.** Required even when source looks clean. The coordinator owns all driving; follow the verification skill's own launch model — one long-lived instance driven serially for servers and UIs, or a fresh isolated session per drive for short-lived CLIs (the skill's Launch section decides, not this one). Exercise every feature at least once, and hold three invariants the whole pass, whatever the failure: (1) never drive an instance you haven't health-checked since it last did something surprising — doctor before first drive, doctor on each fresh session where sessions are the unit, doctor again after any failed drive, and where doctor can't see the failure (a wedged UI state on a healthy process), reset to a known state or relaunch rather than hoping; (2) evidence captured so far survives every cleanup, checked at its named location, not assumed; (3) nothing a drive started outlives that drive's usefulness — failed-iteration residue is cleaned whether the session is stuck, exited, or shared (for a shared instance, clean the residue, not the instance). A doctor failure caused by skill drift is drift: fix it under edit scope and retry once — restart whatever the fix invalidated, nothing more — before calling the pass `blocked`. A feature that can't be reached is `verified-unreachable` only with the concrete prerequisite (auth, entitlement, OS, external state) and the route attempted; if the map omits that prerequisite, that's drift. Any harness fix from triage gets re-driven live before it ships. Final teardown happens after the last drive of the run — including those re-proofs — so nothing outlives the run (evidence stays, per the skill).
|
||||
|
||||
5. **Triage.** Wrong or missing user-POV description → doc drift, fix it. Working behavior the harness can't drive → harness gap, fix it; a harness fix follows the same helpers rule as generation (scripts executable, invocation documented in the skill body). App behavior that's actually broken → product gap; record it for the user, keep it out of this PR.
|
||||
|
||||
6. **Ship or stop.** For changed: one PR of proven corrections, re-read every changed file first. For clean or blocked: no PR, report the outcome and the coverage honestly.
|
||||
|
||||
Keep concise run notes (features covered, unreachable prerequisites, confirmed drift, outcome) in a scratch location; don't commit them.
|
||||
|
||||
Origin and license: [NOTICE.md](NOTICE.md).
|
||||
@@ -0,0 +1,2 @@
|
||||
policy:
|
||||
allow_implicit_invocation: false
|
||||
@@ -0,0 +1,166 @@
|
||||
---
|
||||
name: verify-paperclip
|
||||
description: Launch an isolated Paperclip instance, drive its real web UI or CLI using the existing feature-map recipes, capture evidence, and clean up owned processes. Use to verify user-facing Paperclip behavior.
|
||||
---
|
||||
|
||||
# Verify Paperclip
|
||||
|
||||
Use the repo-root [feature map](../../../feature-map/README.md) to select
|
||||
verification recipes. Read the relevant feature file before driving it. Report
|
||||
any recipe drift with evidence; map updates belong to
|
||||
[maintain-verification-skill](../maintain-verification-skill/SKILL.md).
|
||||
Read [development instructions](../../../doc/DEVELOPING.md) for current commands.
|
||||
|
||||
## Launch
|
||||
|
||||
Run from the repository root with Node 24.11+, Python 3, `lsof`, and dependencies
|
||||
installed with `pnpm install --frozen-lockfile`. Browser driving uses the existing
|
||||
`@playwright/test` dependency. If Chromium is unavailable, run
|
||||
`node node_modules/@playwright/test/cli.js install chromium`, or select an installed
|
||||
browser with `PAPERCLIP_PLAYWRIGHT_CHANNEL=chrome`.
|
||||
|
||||
Create separate, unique data and evidence directories. Keep the evidence outside
|
||||
the data directory:
|
||||
|
||||
```sh
|
||||
VERIFY_DATA_DIR=$(mktemp -d "${TMPDIR:-/tmp}/paperclip-verify-XXXXXX")
|
||||
VERIFY_EVIDENCE_DIR=$(mktemp -d "${TMPDIR:-/tmp}/paperclip-proof-XXXXXX")
|
||||
python3 .agents/skills/verify-paperclip/scripts/launch.py \
|
||||
--data-dir "$VERIFY_DATA_DIR" --evidence-dir "$VERIFY_EVIDENCE_DIR"
|
||||
```
|
||||
|
||||
Keep this command in a foreground terminal/session. The helper runs the supported
|
||||
`onboard --yes --bind loopback --no-install-service --run` command, with isolated
|
||||
embedded PostgreSQL, a free loopback port, no inherited instance routing or cwd
|
||||
`.env`, and no automatic browser opening. It records `launch.json` in the evidence
|
||||
directory. Source startup builds its required shared/plugin packages and migrates
|
||||
the empty database; readiness can take longer on the first run.
|
||||
|
||||
This path starts an empty instance without model credentials. Create company and
|
||||
agent state through the journey being checked. A UI check that stops before model
|
||||
execution does not prove an agent run. For a preconfigured company/CEO and live
|
||||
model work, use the documented [test-drive](../../../doc/DEVELOPING.md#one-command-isolated-manual-test-drive)
|
||||
command with an authorized environment-backed credential and `--no-browser`.
|
||||
Record its printed data directory, URL, runtime PID and commit before driving;
|
||||
never use a fake credential to claim provider verification.
|
||||
|
||||
For Paperclip-managed reusable services, follow
|
||||
[dev-workspace run/verify/fix](../paperclip-dev-workspace-run-verify-fix/SKILL.md)
|
||||
instead of launching a second server. Its cloned-data/login requirements differ
|
||||
from this empty local instance.
|
||||
|
||||
## Doctor
|
||||
|
||||
After startup, and again after any surprising behavior, run:
|
||||
|
||||
```sh
|
||||
python3 .agents/skills/verify-paperclip/scripts/doctor.py \
|
||||
--evidence-dir "$VERIFY_EVIDENCE_DIR"
|
||||
```
|
||||
|
||||
This read-only check compares the recorded launcher, runtime PID, port owner,
|
||||
process cwd, configuration, served Git commit, health/bootstrap readiness, root
|
||||
HTML response, and company-list access. It writes `doctor.json` or exits nonzero.
|
||||
Do not trust a root 200 or CLI doctor output alone. The CLI doctor checks whether
|
||||
the port is available before startup; it is not proof of a running server's
|
||||
ownership. Browser readiness still requires the expected visible page. Reset or
|
||||
relaunch a wedged browser even when the process health passes.
|
||||
|
||||
## Drive
|
||||
|
||||
Use accessible roles, labels, and placeholders from the latest page state.
|
||||
Existing Playwright specs in `tests/e2e/` supply recipes and selectors; their
|
||||
configured servers and fixtures are separate from this live instance. Run a named
|
||||
suite through its own configuration only when that is the evidence needed.
|
||||
|
||||
The executable onboarding smoke drives `onboarding.md` / `web-wizard`: open
|
||||
`/onboarding` in the disposable self-hosted instance,
|
||||
enter a unique organization name, choose Continue, reach the agent-name field,
|
||||
reopen `/<company-prefix>/onboarding`, reload, and corroborate the saved company
|
||||
through read-only API checks:
|
||||
|
||||
```sh
|
||||
node .agents/skills/verify-paperclip/scripts/onboarding-smoke.mjs \
|
||||
--evidence-dir "$VERIFY_EVIDENCE_DIR"
|
||||
```
|
||||
|
||||
This covers company creation and setup resumption, with no mocked responses or
|
||||
provider calls. Hiring, first task execution, and the alternate CLI entry remain
|
||||
separate journeys. The script requires an empty instance and refuses a non-owned
|
||||
or non-loopback target. For other features, reuse the owned URL and follow the
|
||||
map's entry points. Company routes normally require the selected company's
|
||||
`issuePrefix`, for example `/<PREFIX>/goals`; use current navigation rather than
|
||||
guessing resource URLs. The unprefixed `/onboarding` route starts naming; use the
|
||||
company-prefixed onboarding route to resume its agent setup. Check relevant
|
||||
experimental gates and actor permissions.
|
||||
|
||||
For CLI checks, discover exact flags with
|
||||
`node cli/node_modules/tsx/dist/cli.mjs cli/src/index.ts <command> --help`, then
|
||||
select the disposable data directory/API/company explicitly. API reads can
|
||||
corroborate UI writes; API-created state does not prove a creation UI.
|
||||
|
||||
## Evidence
|
||||
|
||||
Record feature file, sub-feature and entry-point ID, commit/dirty state, owned URL,
|
||||
company/user, flags, runtime/harness, live versus simulated dependencies, command,
|
||||
exit code, expected and observed behavior, and evidence links. Report each entry
|
||||
as passed, failed, blocked with its prerequisite, or not run.
|
||||
|
||||
The smoke saves before/after/reload screenshots, visible-page text, a Playwright
|
||||
trace, and `result.json` with company ID and API corroboration. On failure it
|
||||
retains the trace, visible state, and failure result; rerun doctor before retrying.
|
||||
Preserve launch output and relevant application logs without secrets. Reopen or
|
||||
reload changed state and verify persistence/continuation, not just a success toast.
|
||||
One passing sub-feature does not certify the entire feature family or other
|
||||
surfaces. A full maintenance pass must exercise every mapped feature or report
|
||||
incomplete coverage.
|
||||
|
||||
For Paperclip-assigned work, attach important proof through
|
||||
[the artifact workflow](../../../doc/AGENT-ARTIFACTS.md). Keep runtime results in
|
||||
the evidence directory or issue, not in the map; `coverage.json` describes
|
||||
documentation coverage rather than live test health.
|
||||
|
||||
## Cleanup
|
||||
|
||||
Close the browser context you created (the smoke does this in `finally`). Stop the owned runtime with:
|
||||
|
||||
```sh
|
||||
python3 .agents/skills/verify-paperclip/scripts/stop.py \
|
||||
--evidence-dir "$VERIFY_EVIDENCE_DIR"
|
||||
```
|
||||
|
||||
The helper verifies the runtime's port, cwd, and launch ancestry before sending
|
||||
SIGTERM to that PID, then waits for runtime exit and checks that its HTTP
|
||||
listener and embedded database have stopped. Signaling the runtime PID lets the server drain before stopping
|
||||
PostgreSQL; a terminal group interrupt can reach the database first. Do this after
|
||||
failed drives as well as successful runs. If startup fails before a runtime record
|
||||
exists, stop only the foreground session you created and inspect its recorded data
|
||||
directory for residue. Never kill by process name or stop another checkout's server.
|
||||
|
||||
Verify the recorded HTTP port has no listener and the owned runtime PID has
|
||||
exited. Embedded PostgreSQL should stop with the server; inspect only the recorded
|
||||
data directory if shutdown is incomplete. Retain the data directory for diagnosis
|
||||
until no owned process uses it; remove only that exact temporary directory when
|
||||
it is no longer useful. Never delete the evidence directory during teardown.
|
||||
Confirm screenshots, trace, `result.json`, `launch.json`, and `doctor.json` still
|
||||
exist after shutdown. A failed launch leaves its evidence intact.
|
||||
|
||||
## Helpers
|
||||
|
||||
All helper invocations are shown above. `launch.py`, `doctor.py`, and `stop.py` use Python's
|
||||
standard library and the repo CLI; `onboarding-smoke.mjs` uses the repo's existing
|
||||
Playwright dependency. They introduce no new package dependency or test framework.
|
||||
|
||||
## Helper safety check
|
||||
|
||||
Run the standard-library ownership test before changing launch/doctor/stop helpers:
|
||||
|
||||
```sh
|
||||
python3 -m unittest discover -s .agents/skills/verify-paperclip/tests -v
|
||||
```
|
||||
|
||||
It starts an owned fixture in another temporary checkout and confirms that both
|
||||
doctor and stop reject its claimed port/PID without stopping the server. A second
|
||||
fixture closes its listener before exiting and proves that cleanup waits for
|
||||
runtime exit. These checks do
|
||||
not qualify Paperclip startup, database readiness, or onboarding.
|
||||
+60
@@ -0,0 +1,60 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Read-only identity and readiness check for an owned verification launch."""
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
import subprocess
|
||||
import sys
|
||||
import urllib.request
|
||||
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument('--evidence-dir', required=True, type=Path)
|
||||
args = parser.parse_args()
|
||||
evidence = args.evidence_dir.resolve()
|
||||
try:
|
||||
launch = json.loads((evidence / 'launch.json').read_text())
|
||||
repo = Path(launch['repo']).resolve()
|
||||
assert repo == Path(__file__).resolve().parents[4], 'Launch belongs to another checkout.'
|
||||
runtime = json.loads((Path(launch['dataDir']) / 'instances/default/runtime-info.json').read_text())
|
||||
assert runtime['dashboardUrl'] == launch['url'], 'Runtime URL changed.'
|
||||
assert runtime['port'] == launch['port'], 'Runtime port changed.'
|
||||
assert runtime['host'] == '127.0.0.1', 'Expected trusted loopback.'
|
||||
pid = runtime['pid']
|
||||
os.kill(pid, 0)
|
||||
listeners = subprocess.check_output(['lsof', '-nP', f'-iTCP:{launch["port"]}',
|
||||
'-sTCP:LISTEN', '-t'], text=True).split()
|
||||
assert set(listeners) == {str(pid)}, 'Port owner does not match runtime PID.'
|
||||
if sys.platform == 'darwin':
|
||||
rows = subprocess.check_output(['lsof', '-a', '-p', str(pid), '-d', 'cwd', '-Fn'], text=True).splitlines()
|
||||
cwd = Path(next(row[1:] for row in rows if row.startswith('n'))).resolve()
|
||||
else:
|
||||
cwd = Path(f'/proc/{pid}/cwd').resolve(strict=True)
|
||||
assert cwd == repo or repo in cwd.parents, 'Port owner cwd belongs to another checkout.'
|
||||
ancestor = pid
|
||||
seen = set()
|
||||
while ancestor != launch['launcherPid']:
|
||||
assert ancestor > 1 and ancestor not in seen, 'Runtime is not descended from this launch.'
|
||||
seen.add(ancestor)
|
||||
ancestor = int(subprocess.check_output(['ps', '-p', str(ancestor), '-o', 'ppid='], text=True).strip())
|
||||
config = json.loads((Path(launch['dataDir']) / 'instances/default/config.json').read_text())
|
||||
assert config['database']['mode'] == 'embedded-postgres', 'Expected isolated embedded database.'
|
||||
assert Path(config['database']['embeddedPostgresDataDir']).resolve().is_relative_to(Path(launch['dataDir'])), 'Database path escaped isolation.'
|
||||
with urllib.request.urlopen(launch['url'] + '/api/health', timeout=15) as response:
|
||||
health = json.load(response)
|
||||
assert health['status'] == 'ok' and health['bootstrapStatus'] == 'ready', 'Instance is not ready.'
|
||||
assert health['deploymentMode'] == 'local_trusted' and health['authReady'], 'Unexpected auth mode.'
|
||||
commit = subprocess.check_output(['git', '-C', str(repo), 'rev-parse', 'HEAD'], text=True).strip()
|
||||
assert health['commit'] == launch['commit'] == commit, 'Served commit differs from checkout/launch.'
|
||||
with urllib.request.urlopen(launch['url'], timeout=15) as response:
|
||||
assert response.status == 200 and 'text/html' in response.headers.get('Content-Type', ''), 'Root UI is unavailable.'
|
||||
with urllib.request.urlopen(launch['url'] + '/api/companies', timeout=15) as response:
|
||||
companies = json.load(response)
|
||||
assert isinstance(companies, list), 'Board company access failed.'
|
||||
result = {'status': 'passed', 'pid': pid, 'cwd': str(cwd), 'url': launch['url'],
|
||||
'health': health, 'companyIds': [company['id'] for company in companies]}
|
||||
(evidence / 'doctor.json').write_text(json.dumps(result, indent=2) + '\n')
|
||||
print(f'PASS: owned PID {pid}, {launch["url"]}, commit {commit}, ready board UI/API.')
|
||||
except Exception as error:
|
||||
(evidence / 'doctor-failure.json').write_text(json.dumps({'status': 'failed', 'error': str(error)}, indent=2) + '\n')
|
||||
raise SystemExit(f'Doctor failed: {error}')
|
||||
+52
@@ -0,0 +1,52 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Run the repository's credential-free isolated foreground onboarding path."""
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
import socket
|
||||
import subprocess
|
||||
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument('--data-dir', required=True, type=Path)
|
||||
parser.add_argument('--evidence-dir', required=True, type=Path)
|
||||
args = parser.parse_args()
|
||||
repo = Path(__file__).resolve().parents[4]
|
||||
data = args.data_dir.resolve()
|
||||
evidence = args.evidence_dir.resolve()
|
||||
if data == evidence or data in evidence.parents or evidence in data.parents:
|
||||
parser.error('Data and evidence directories must be separate.')
|
||||
data.mkdir(parents=True, exist_ok=True)
|
||||
evidence.mkdir(parents=True, exist_ok=True)
|
||||
if any(data.iterdir()) or (evidence / 'launch.json').exists():
|
||||
parser.error('Use a fresh empty data directory and a new evidence directory.')
|
||||
cli = repo / 'cli/node_modules/tsx/dist/cli.mjs'
|
||||
if not cli.is_file():
|
||||
parser.error('Install checkout dependencies first: pnpm install --frozen-lockfile')
|
||||
for port in range(3100, 4100):
|
||||
try:
|
||||
with socket.socket() as probe:
|
||||
probe.bind(('127.0.0.1', port))
|
||||
break
|
||||
except OSError:
|
||||
continue
|
||||
else:
|
||||
raise RuntimeError('No available verification port between 3100 and 4099.')
|
||||
excluded = {'DATABASE_URL', 'DATABASE_MIGRATION_URL', 'HOST', 'PORT', 'SERVE_UI',
|
||||
'BETTER_AUTH_URL', 'BETTER_AUTH_BASE_URL', 'NODE_OPTIONS'}
|
||||
env = {key: value for key, value in os.environ.items()
|
||||
if not key.startswith('PAPERCLIP_') and key not in excluded}
|
||||
env.update({'PAPERCLIP_HOME': str(data), 'PAPERCLIP_INSTANCE_ID': 'default',
|
||||
'PAPERCLIP_DISABLE_CWD_ENV_FILE': 'true', 'PAPERCLIP_NO_BROWSER': 'true',
|
||||
'PAPERCLIP_OPEN_ON_LISTEN': 'false', 'PAPERCLIP_UPDATE_CHECK': '0',
|
||||
'PAPERCLIP_TELEMETRY_DISABLED': '1', 'PAPERCLIP_DEPLOYMENT_MODE': 'local_trusted',
|
||||
'PAPERCLIP_DEPLOYMENT_EXPOSURE': 'private', 'PAPERCLIP_BIND': 'loopback',
|
||||
'HOST': '127.0.0.1', 'PORT': str(port)})
|
||||
record = {'dataDir': str(data), 'evidenceDir': str(evidence), 'repo': str(repo),
|
||||
'launcherPid': os.getpid(), 'port': port, 'url': f'http://127.0.0.1:{port}',
|
||||
'commit': subprocess.check_output(['git', '-C', str(repo), 'rev-parse', 'HEAD'], text=True).strip()}
|
||||
(evidence / 'launch.json').write_text(json.dumps(record, indent=2) + '\n')
|
||||
print(json.dumps(record), flush=True)
|
||||
os.chdir(repo)
|
||||
os.execvpe('node', ['node', str(cli), 'cli/src/index.ts', 'onboard', '--data-dir',
|
||||
str(data), '--yes', '--bind', 'loopback', '--no-install-service', '--run'], env)
|
||||
@@ -0,0 +1,79 @@
|
||||
#!/usr/bin/env node
|
||||
import assert from 'node:assert/strict';
|
||||
import fs from 'node:fs/promises';
|
||||
import path from 'node:path';
|
||||
import { createRequire } from 'node:module';
|
||||
import { fileURLToPath } from 'node:url';
|
||||
import { execFileSync } from 'node:child_process';
|
||||
|
||||
const repo = path.resolve(path.dirname(fileURLToPath(import.meta.url)), '../../../..');
|
||||
const require = createRequire(path.join(repo, 'package.json'));
|
||||
const { chromium, expect } = require('@playwright/test');
|
||||
assert.equal(process.argv[2], '--evidence-dir', 'Usage: onboarding-smoke.mjs --evidence-dir <path>');
|
||||
assert.ok(process.argv[3]);
|
||||
const evidence = path.resolve(process.argv[3]);
|
||||
execFileSync('python3', [path.join(repo, '.agents/skills/verify-paperclip/scripts/doctor.py'), '--evidence-dir', evidence], { stdio: 'inherit' });
|
||||
const launch = JSON.parse(await fs.readFile(path.join(evidence, 'launch.json'), 'utf8'));
|
||||
assert.equal(new URL(launch.url).hostname, '127.0.0.1');
|
||||
const browser = await chromium.launch({ headless: true, ...(process.env.PAPERCLIP_PLAYWRIGHT_CHANNEL ? { channel: process.env.PAPERCLIP_PLAYWRIGHT_CHANNEL } : {}) });
|
||||
const context = await browser.newContext({ viewport: { width: 1440, height: 1000 }, serviceWorkers: 'block' });
|
||||
const page = await context.newPage();
|
||||
const errors = [];
|
||||
page.on('pageerror', error => errors.push(error.message));
|
||||
const companyName = `Verification ${Date.now()}`;
|
||||
const result = { feature: 'onboarding.md', subFeatures: ['company', 'resume'], entryPoint: 'web-wizard', commit: launch.commit, url: launch.url, actor: 'local_trusted board', providerCalls: false, mockedResponses: false, companyName, status: 'failed' };
|
||||
await context.tracing.start({ screenshots: true, snapshots: true, sources: true });
|
||||
async function capture(name) {
|
||||
await page.screenshot({ path: path.join(evidence, `${name}.png`), fullPage: true });
|
||||
await fs.writeFile(path.join(evidence, `${name}.txt`), await page.locator('body').innerText());
|
||||
}
|
||||
try {
|
||||
const before = await page.request.get(`${launch.url}/api/companies`);
|
||||
assert.equal(before.ok(), true);
|
||||
assert.deepEqual(await before.json(), [], 'Smoke requires a fresh instance.');
|
||||
await page.goto(`${launch.url}/onboarding`);
|
||||
const start = page.getByRole('button', { name: /Start Onboarding|New Organization|Add Agent/ });
|
||||
if (await start.count()) await start.first().click();
|
||||
const create = page.getByRole('button', { name: /Build a new organization/ });
|
||||
if (await create.count()) await create.first().click();
|
||||
await expect(page.getByRole('heading', { name: 'What is the name of your organization?' })).toBeVisible({ timeout: 60_000 });
|
||||
await page.getByPlaceholder('e.g. Northwind Labs').fill(companyName);
|
||||
await capture('before-create');
|
||||
await page.getByRole('button', { name: /^Continue/ }).click();
|
||||
await expect(page.locator('#onboarding-agent-name')).toBeVisible({ timeout: 30_000 });
|
||||
await capture('after-create');
|
||||
const companiesResponse = await page.request.get(`${launch.url}/api/companies`);
|
||||
assert.equal(companiesResponse.ok(), true);
|
||||
const companies = await companiesResponse.json();
|
||||
const company = companies.find(item => item.name === companyName);
|
||||
assert.ok(company, 'UI-created company must be persisted.');
|
||||
result.companyId = company.id;
|
||||
const goalsResponse = await page.request.get(`${launch.url}/api/companies/${company.id}/goals`);
|
||||
assert.equal(goalsResponse.ok(), true);
|
||||
assert.deepEqual(await goalsResponse.json(), [], 'Naming must not invent a mission.');
|
||||
await page.goto(`${launch.url}/${company.issuePrefix}/onboarding`);
|
||||
await expect(page.locator('#onboarding-agent-name')).toBeVisible({ timeout: 30_000 });
|
||||
await capture('after-reopen');
|
||||
await page.reload();
|
||||
await expect(page.locator('#onboarding-agent-name')).toBeVisible({ timeout: 30_000 });
|
||||
await capture('after-reload');
|
||||
const persistedResponse = await page.request.get(`${launch.url}/api/companies`);
|
||||
assert.equal(persistedResponse.ok(), true);
|
||||
const persisted = await persistedResponse.json();
|
||||
assert.equal(persisted.length, 1, 'Resume must not create another company.');
|
||||
assert.equal(persisted[0].id, company.id);
|
||||
assert.deepEqual(errors, [], 'No browser exceptions.');
|
||||
result.status = 'passed';
|
||||
result.observed = 'Company persisted; onboarding resumed at the agent step after reload; no mission invented.';
|
||||
result.notRun = ['agent hiring', 'first task/model execution', 'cli-setup entry point'];
|
||||
} catch (error) {
|
||||
result.error = error.message;
|
||||
await capture('failure').catch(() => {});
|
||||
process.exitCode = 1;
|
||||
} finally {
|
||||
result.browserErrors = errors;
|
||||
await context.tracing.stop({ path: path.join(evidence, 'trace.zip') });
|
||||
await browser.close();
|
||||
await fs.writeFile(path.join(evidence, 'result.json'), `${JSON.stringify(result, null, 2)}\n`);
|
||||
console.log(JSON.stringify(result));
|
||||
}
|
||||
+65
@@ -0,0 +1,65 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Stop only an owned verification runtime and check that its database exits."""
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
import signal
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument('--evidence-dir', required=True, type=Path)
|
||||
args = parser.parse_args()
|
||||
evidence = args.evidence_dir.resolve()
|
||||
launch = json.loads((evidence / 'launch.json').read_text())
|
||||
repo = Path(launch['repo']).resolve()
|
||||
assert repo == Path(__file__).resolve().parents[4], 'Launch belongs to another checkout.'
|
||||
instance = Path(launch['dataDir']) / 'instances/default'
|
||||
runtime_file = instance / 'runtime-info.json'
|
||||
runtime = json.loads(runtime_file.read_text()) if runtime_file.exists() else None
|
||||
pid = None
|
||||
if runtime:
|
||||
assert runtime['port'] == launch['port'] and runtime['dashboardUrl'] == launch['url']
|
||||
pid = runtime['pid']
|
||||
owners = subprocess.run(['lsof', '-nP', f'-iTCP:{launch["port"]}', '-sTCP:LISTEN', '-t'], capture_output=True, text=True).stdout.split()
|
||||
assert set(owners) == {str(pid)}, 'Refusing to signal a different or unverified port owner.'
|
||||
if sys.platform == 'darwin':
|
||||
rows = subprocess.check_output(['lsof', '-a', '-p', str(pid), '-d', 'cwd', '-Fn'], text=True).splitlines()
|
||||
cwd = Path(next(row[1:] for row in rows if row.startswith('n'))).resolve()
|
||||
else:
|
||||
cwd = Path(f'/proc/{pid}/cwd').resolve(strict=True)
|
||||
assert cwd == repo or repo in cwd.parents, 'Refusing to signal another checkout.'
|
||||
ancestor = pid
|
||||
seen = set()
|
||||
while ancestor != launch['launcherPid']:
|
||||
assert ancestor > 1 and ancestor not in seen, 'Runtime is not descended from this launch.'
|
||||
seen.add(ancestor)
|
||||
ancestor = int(subprocess.check_output(['ps', '-p', str(ancestor), '-o', 'ppid='], text=True).strip())
|
||||
os.kill(pid, signal.SIGTERM)
|
||||
def runtime_exited():
|
||||
if pid is None:
|
||||
return True
|
||||
try:
|
||||
os.kill(pid, 0)
|
||||
except ProcessLookupError:
|
||||
return True
|
||||
# An exited child can remain a zombie until its launcher reaps it.
|
||||
state = subprocess.run(['ps', '-p', str(pid), '-o', 'stat='],
|
||||
capture_output=True, text=True)
|
||||
return state.returncode == 0 and state.stdout.strip().startswith('Z')
|
||||
|
||||
|
||||
deadline = time.monotonic() + 45
|
||||
while time.monotonic() < deadline:
|
||||
owners = subprocess.run(['lsof', '-nP', f'-iTCP:{launch["port"]}', '-sTCP:LISTEN', '-t'], capture_output=True, text=True).stdout.split()
|
||||
if not owners and not (instance / 'db/postmaster.pid').exists() and runtime_exited():
|
||||
result = {'status': 'passed', 'port': launch['port'], 'serverStopped': True,
|
||||
'databaseStopped': True, 'runtimeExited': True, 'evidenceRetained': sorted(p.name for p in evidence.iterdir())}
|
||||
(evidence / 'cleanup.json').write_text(json.dumps(result, indent=2) + '\n')
|
||||
print(f'PASS: owned runtime/database stopped; evidence retained at {evidence}.')
|
||||
break
|
||||
time.sleep(0.5)
|
||||
else:
|
||||
raise SystemExit('Shutdown incomplete; retain data and evidence and inspect only this owned instance.')
|
||||
@@ -0,0 +1,124 @@
|
||||
"""Check that verification helpers cannot take ownership of another checkout."""
|
||||
import json
|
||||
import os
|
||||
import time
|
||||
from pathlib import Path
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
import unittest
|
||||
import urllib.request
|
||||
|
||||
|
||||
class OwnershipSafety(unittest.TestCase):
|
||||
def test_foreign_checkout_server_remains_alive(self):
|
||||
repo = Path(__file__).resolve().parents[4]
|
||||
helpers = repo / '.agents/skills/verify-paperclip/scripts'
|
||||
with tempfile.TemporaryDirectory(prefix='paperclip-ownership-test-') as temp:
|
||||
root = Path(temp)
|
||||
foreign = root / 'another-checkout'
|
||||
foreign.mkdir()
|
||||
server = subprocess.Popen(
|
||||
[sys.executable, '-u', '-c',
|
||||
"from http.server import HTTPServer, BaseHTTPRequestHandler\n"
|
||||
"class Handler(BaseHTTPRequestHandler):\n"
|
||||
" def do_GET(self):\n"
|
||||
" self.send_response(200); self.end_headers(); self.wfile.write(b'foreign fixture alive')\n"
|
||||
" def log_message(self, *args): pass\n"
|
||||
"server = HTTPServer(('127.0.0.1', 0), Handler)\n"
|
||||
"print(server.server_port, flush=True)\n"
|
||||
"server.serve_forever()\n"],
|
||||
cwd=foreign, stdout=subprocess.PIPE, stderr=subprocess.PIPE,
|
||||
text=True,
|
||||
)
|
||||
try:
|
||||
port = int(server.stdout.readline().strip())
|
||||
url = f'http://127.0.0.1:{port}'
|
||||
data = root / 'verification-data'
|
||||
instance = data / 'instances/default'
|
||||
instance.mkdir(parents=True)
|
||||
evidence = root / 'evidence'
|
||||
evidence.mkdir()
|
||||
# The claimed PID really owns this port, but its cwd is foreign.
|
||||
(evidence / 'launch.json').write_text(json.dumps({
|
||||
'repo': str(repo), 'dataDir': str(data), 'port': port,
|
||||
'url': url, 'launcherPid': server.pid,
|
||||
}))
|
||||
(instance / 'runtime-info.json').write_text(json.dumps({
|
||||
'pid': server.pid, 'port': port, 'dashboardUrl': url,
|
||||
'host': '127.0.0.1',
|
||||
}))
|
||||
for helper in ['doctor.py', 'stop.py']:
|
||||
with self.subTest(helper=helper):
|
||||
result = subprocess.run(
|
||||
[sys.executable, str(helpers / helper),
|
||||
'--evidence-dir', str(evidence)],
|
||||
capture_output=True, text=True, timeout=10,
|
||||
)
|
||||
self.assertNotEqual(result.returncode, 0)
|
||||
self.assertIn('checkout', result.stdout + result.stderr)
|
||||
self.assertIsNone(server.poll(), 'Helper killed foreign server')
|
||||
with urllib.request.urlopen(url, timeout=5) as response:
|
||||
self.assertEqual(response.read(), b'foreign fixture alive')
|
||||
self.assertFalse((evidence / 'cleanup.json').exists())
|
||||
finally:
|
||||
server.terminate()
|
||||
try:
|
||||
server.communicate(timeout=5)
|
||||
except subprocess.TimeoutExpired:
|
||||
server.kill()
|
||||
server.communicate()
|
||||
|
||||
|
||||
def test_cleanup_waits_for_exit_after_listener_closes(self):
|
||||
repo = Path(__file__).resolve().parents[4]
|
||||
with tempfile.TemporaryDirectory(prefix='paperclip-delayed-exit-test-') as temp:
|
||||
root = Path(temp)
|
||||
instance = root / 'data/instances/default'
|
||||
instance.mkdir(parents=True)
|
||||
evidence = root / 'evidence'
|
||||
evidence.mkdir()
|
||||
marker = root / 'listener-closed'
|
||||
server = subprocess.Popen(
|
||||
[sys.executable, '-u', '-c',
|
||||
"import signal, socket, sys, time\n"
|
||||
"from pathlib import Path\n"
|
||||
"listener = socket.socket(); listener.bind(('127.0.0.1', 0)); listener.listen()\n"
|
||||
"def stop(signum, frame):\n"
|
||||
" listener.close(); Path(sys.argv[1]).write_text('closed')\n"
|
||||
" time.sleep(2); sys.exit(0)\n"
|
||||
"signal.signal(signal.SIGTERM, stop)\n"
|
||||
"print(listener.getsockname()[1], flush=True)\n"
|
||||
"signal.pause()\n", str(marker)],
|
||||
cwd=repo, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True,
|
||||
)
|
||||
try:
|
||||
port = int(server.stdout.readline().strip())
|
||||
url = f'http://127.0.0.1:{port}'
|
||||
(evidence / 'launch.json').write_text(json.dumps({
|
||||
'repo': str(repo), 'dataDir': str(root / 'data'),
|
||||
'port': port, 'url': url, 'launcherPid': os.getpid(),
|
||||
}))
|
||||
(instance / 'runtime-info.json').write_text(json.dumps({
|
||||
'pid': server.pid, 'port': port, 'dashboardUrl': url,
|
||||
}))
|
||||
started = time.monotonic()
|
||||
result = subprocess.run(
|
||||
[sys.executable, str(repo / '.agents/skills/verify-paperclip/scripts/stop.py'),
|
||||
'--evidence-dir', str(evidence)],
|
||||
capture_output=True, text=True, timeout=12,
|
||||
)
|
||||
self.assertEqual(result.returncode, 0, result.stdout + result.stderr)
|
||||
self.assertTrue(marker.exists(), 'Fixture did not close its listener')
|
||||
self.assertGreaterEqual(time.monotonic() - started, 2)
|
||||
self.assertEqual(server.poll(), 0, 'Cleanup returned before runtime exit')
|
||||
cleanup = json.loads((evidence / 'cleanup.json').read_text())
|
||||
self.assertTrue(cleanup['runtimeExited'])
|
||||
finally:
|
||||
if server.poll() is None:
|
||||
server.kill()
|
||||
server.communicate(timeout=5)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
unittest.main()
|
||||
+10
-4
@@ -9,7 +9,7 @@ Product behavior is still governed by [the implementation spec](../doc/SPEC-impl
|
||||
|
||||
The map is organized by user capability, not by source file. It covers **35 feature
|
||||
families** across the product, CLI, and operator workflows, checked against source
|
||||
on 2026-10-05. Experimental and developer-only surfaces are labeled explicitly.
|
||||
on 2026-10-12. Experimental and developer-only surfaces are labeled explicitly.
|
||||
Each recipe includes sub-features, current entry points, automated evidence,
|
||||
manual verification steps, and gotchas.
|
||||
|
||||
@@ -113,7 +113,10 @@ fixture integration. Record executed results separately from the map.
|
||||
persisted state or continuation. Use read-only API checks to corroborate UI
|
||||
evidence; creating state through the API does not prove the creation UI.
|
||||
|
||||
For agent-driven acceptance work, the existing
|
||||
For agent-driven acceptance work, start with the
|
||||
[Paperclip verification skill](../.agents/skills/verify-paperclip/SKILL.md) for an
|
||||
isolated launch, ownership/readiness checks, browser driving, evidence, and cleanup.
|
||||
It uses this directory as its feature map. The existing
|
||||
[dev-workspace run/verify skill](../.agents/skills/paperclip-dev-workspace-run-verify-fix/SKILL.md)
|
||||
and [evaluation skill](../.agents/skills/paperclip-evals/SKILL.md) describe runtime
|
||||
ownership and evidence handling. Reuse them; this map introduces no environment
|
||||
@@ -168,10 +171,13 @@ Each recipe starts with an H1 and a user-visible description, then exactly:
|
||||
|
||||
1. **Sub-features** — stable backticked IDs and observable behavior/states.
|
||||
2. **How to get to it (user POV)** — one H3 backticked entry-point ID per surface.
|
||||
3. **Driving it** — starts with `Preconditions:`; repeat each entry-point H3,
|
||||
3. **Driving it with <harness>** — starts with `Preconditions:`; repeat each entry-point H3,
|
||||
with `Automated:` and `Manual:` paragraphs. Name test scope and gaps honestly.
|
||||
4. **Gotchas** — misleading look-alikes, feature gates, and invalid evidence.
|
||||
|
||||
The format is inspired by
|
||||
New or updated entries follow
|
||||
[Poteto's feature-map examples](../.agents/skills/create-verification-skill/references/feature-map-example/).
|
||||
Existing entries with a `Driving it` heading can adopt the named harness when
|
||||
their recipes are updated. The original format was inspired by
|
||||
[Omnigent's feature map](https://github.com/omnigent-ai/omnigent/tree/91acfbbb59f6fc210ff95a9e9428aadd62e06582/feature-map).
|
||||
Paperclip's recipes and checks follow its own task model and test infrastructure.
|
||||
@@ -21,12 +21,15 @@ Open `/auth`; first-owner bootstrap uses `/board-claim/:token` when configured.
|
||||
### `invite-members`
|
||||
|
||||
Open `/invite/:token`; operators manage members in company settings and join requests at `/inbox/requests`.
|
||||
Create/revoke human links at Members → Invites
|
||||
(`/company/settings/members?tab=invites`). Human invitees receive the chosen
|
||||
role after sign-in; agent invitations still use join-request approval.
|
||||
|
||||
### `cli-authorization`
|
||||
|
||||
Start the CLI auth/connect flow and approve its `/cli-auth/:id` browser page; instance access is under Settings when visible.
|
||||
|
||||
## Driving it
|
||||
## Driving it with Playwright and the Paperclip CLI
|
||||
|
||||
Preconditions: follow the [baseline](./README.md#before-driving-a-journey). For login tests use authenticated mode and two test users; local-trusted mode does not prove session enforcement. Use two companies for isolation checks.
|
||||
|
||||
@@ -48,6 +51,10 @@ Automated: [board auth client](../cli/src/__tests__/board-auth.test.ts) and [tok
|
||||
|
||||
Manual: Authorize the intended CLI context, make a read-only company request, then revoke its token and retry. Verify failure is explicit and does not silently fall back to another identity.
|
||||
|
||||
Use `auth login`, `auth whoami`, and `auth revoke-current` with explicit context.
|
||||
Retry with the same bearer to prove revocation; deleting local credentials with
|
||||
logout alone does not establish that server revocation succeeded.
|
||||
|
||||
## Gotchas
|
||||
|
||||
- Settings visibility is not authorization; direct requests still need permission checks.
|
||||
|
||||
@@ -22,7 +22,7 @@ Open `/dashboard` or `/dashboard/live`.
|
||||
|
||||
Open `/activity`, `/activity/runs`, or `/activity/timeline` in the streamlined shell; legacy routes may redirect.
|
||||
|
||||
## Driving it
|
||||
## Driving it with Playwright and the Paperclip CLI
|
||||
|
||||
Preconditions: follow the [baseline](./README.md#before-driving-a-journey). Prepare known task mutations and completed/failed runs in a disposable company, recording their IDs and actors.
|
||||
|
||||
@@ -38,6 +38,11 @@ Automated: [activity service](../server/src/__tests__/activity-service.test.ts),
|
||||
|
||||
Manual: Find a known mutation by actor/resource and a known run by status. Change filters, load more, and follow the source links. Reload the chosen view and verify no company leakage or duplicate events. Inspect a routine run from its audit entry.
|
||||
|
||||
Audit-event Load more and run listings differ: Runs currently loads at most 200
|
||||
entries without pagination. The tested RoutineAuditActivity component has no
|
||||
registered activity-route host; inspect execution links actually offered by
|
||||
the routine rather than treating that component test as a reachable journey.
|
||||
|
||||
## Gotchas
|
||||
|
||||
- Audit/activity events, run-log rows, first-party Telemetry, and operator OpenTelemetry traces are separate data paths.
|
||||
|
||||
@@ -25,7 +25,7 @@ Ask for project work in Agent Chat and follow its created project/task links.
|
||||
|
||||
Open `/board-chat` only when its conference-room gate is enabled.
|
||||
|
||||
## Driving it
|
||||
## Driving it with Playwright and the Paperclip CLI
|
||||
|
||||
Preconditions: follow the [baseline](./README.md#before-driving-a-journey). Enable the applicable chat feature, use a configured test agent, and distinguish model replies from a mocked conversation fixture.
|
||||
|
||||
@@ -33,7 +33,10 @@ Preconditions: follow the [baseline](./README.md#before-driving-a-journey). Enab
|
||||
|
||||
Automated: [chat landing](../ui/src/pages/AgentChats.test.tsx) checks recent-chat routing; [conversation service](../server/src/__tests__/agent-conversations.test.ts) covers persisted conversation behavior.
|
||||
|
||||
Manual: Start a conversation, send a recognizable request, and inspect its reply. Navigate away and reopen the same history. Test a new conversation, unavailable historical agent, and an inaccessible saved conversation; verify no cross-user or cross-company history appears.
|
||||
Manual: Start a conversation, send a recognizable request, and inspect its reply.
|
||||
Navigate away and reopen the same history. Use `/new` for a fresh session within
|
||||
the same standing conversation. Test an unavailable historical agent and an
|
||||
inaccessible saved conversation; verify no cross-user or cross-company history appears.
|
||||
|
||||
### `project-handoff`
|
||||
|
||||
@@ -45,6 +48,10 @@ Manual: Ask the configured agent to create bounded project work, inspect the pro
|
||||
|
||||
Automated: [board chat gate](../server/src/__tests__/board-chat-route-feature-flag.test.ts) proves route gating, not a live conversation.
|
||||
|
||||
Preconditions for this entry point: `local_trusted` mode and an installed,
|
||||
authenticated `claude` CLI, in addition to the Conference Room flag. These are
|
||||
separate from Agent Chat's configured-agent prerequisites.
|
||||
|
||||
Manual: Check disabled access first, enable in the disposable instance, then send and reload a conversation. Inspect the actual responding agent and persisted history; report this result separately from Agent Chat.
|
||||
|
||||
## Gotchas
|
||||
|
||||
@@ -6,7 +6,8 @@ Implementation: [hiring](../ui/src/pages/NewAgent.tsx), [configuration](../ui/sr
|
||||
|
||||
## Sub-features
|
||||
|
||||
- `hire`: choose role, manager, adapter, model, and required account settings.
|
||||
- `hire`: choose identity, adapter, model, and required account settings; edit
|
||||
reporting relationships through the agent configuration afterward.
|
||||
- `instructions`: edit agent configuration and instruction files with durable revisions.
|
||||
- `organization`: inspect and change reporting relationships through supported controls.
|
||||
- `lifecycle`: pause/resume or retire an agent while preserving history.
|
||||
@@ -26,7 +27,7 @@ Open `/agents/:agentId` and the configuration/instruction surfaces it exposes.
|
||||
|
||||
Use the roster’s organization view; `/org` redirects to the roster in the streamlined shell.
|
||||
|
||||
## Driving it
|
||||
## Driving it with Playwright and the Paperclip CLI
|
||||
|
||||
Preconditions: follow the [baseline](./README.md#before-driving-a-journey). Use a disposable company and supported adapter; hiring may require a formal approval.
|
||||
|
||||
@@ -36,6 +37,11 @@ Automated: [new agent page](../ui/src/pages/NewAgent.test.tsx) and [configuratio
|
||||
|
||||
Manual: Hire an agent with a distinct name and role. Complete any required approval, reload the roster, and open its configuration. Confirm manager, adapter, and model persisted; run a small task to prove execution readiness.
|
||||
|
||||
The current hiring wizard assigns the first hire CEO and subsequent hires
|
||||
General reporting to the CEO. Change manager/role afterward through supported
|
||||
configuration. Process/HTTP agents can be created through API/CLI but are excluded
|
||||
from the hiring picker; API fixture creation does not prove this UI journey.
|
||||
|
||||
### `agent-configuration`
|
||||
|
||||
Automated: [instruction service](../server/src/__tests__/agent-instructions-service.test.ts) and [instruction revisions](../server/src/__tests__/agent-instruction-revisions.test.ts) cover file/revision contracts.
|
||||
|
||||
@@ -21,7 +21,7 @@ Use Activity → Costs/Budgets in the streamlined shell, `/costs` in the product
|
||||
|
||||
Open the governing budget configuration and any resulting budget incident card.
|
||||
|
||||
## Driving it
|
||||
## Driving it with Playwright and the Paperclip CLI
|
||||
|
||||
Preconditions: follow the [baseline](./README.md#before-driving-a-journey). Use synthetic low-cost events and small disposable budgets. Do not buy model usage merely to hit a threshold.
|
||||
|
||||
@@ -29,7 +29,10 @@ Preconditions: follow the [baseline](./README.md#before-driving-a-journey). Use
|
||||
|
||||
Automated: [costs service](../server/src/__tests__/costs-service.test.ts) covers cost contracts; [embedded costs UI](../ui/src/pages/Costs.test.tsx) has narrow Audit-host coverage.
|
||||
|
||||
Manual: Inspect a known fixture run and reconcile its recorded spend with the selected report period and resource. Switch company/project filters and reload. Check empty and unavailable cost data without treating missing data as zero spend.
|
||||
Manual: Inspect a known fixture run and reconcile recorded spend with the chosen
|
||||
period and resource. Switch the report period/company and inspect the **By
|
||||
project** breakdown; it is not a project-selector filter. Reload and check empty
|
||||
or unavailable cost data without treating missing data as zero spend.
|
||||
|
||||
### `budget-enforcement`
|
||||
|
||||
|
||||
@@ -21,7 +21,7 @@ Open `/cases` and switch between available grouped/tree views.
|
||||
|
||||
Open `/cases/:caseIdentifier` or a task’s linked case.
|
||||
|
||||
## Driving it
|
||||
## Driving it with Playwright and the Paperclip CLI
|
||||
|
||||
Preconditions: follow the [baseline](./README.md#before-driving-a-journey). Enable cases and prepare related records with different types/statuses plus an associated task.
|
||||
|
||||
@@ -35,9 +35,15 @@ Manual: Search for a child, verify its parent context remains understandable, ch
|
||||
|
||||
Automated: [case routes](../server/src/__tests__/cases-routes.test.ts) covers API behavior; [case fields](../ui/src/components/CaseFieldsPanel.test.tsx) covers field UI separately.
|
||||
|
||||
Manual: Edit an offered field, reload, inspect its revision/activity, and follow child and task links. Verify cross-company and invalid field changes fail clearly. Distinguish a pipeline-owned item from a stand-alone case.
|
||||
Manual: Change status or labels, reload, inspect activity, and follow child/task
|
||||
links. Edit a case document and inspect its document revisions separately.
|
||||
Arbitrary JSON fields are currently displayed without a general field editor.
|
||||
Verify cross-company and invalid changes fail clearly. Distinguish a
|
||||
pipeline-owned item from a stand-alone case.
|
||||
|
||||
## Gotchas
|
||||
|
||||
- A case is not a task execution; inspect linked tasks to prove work occurred.
|
||||
- Feature gating and case-type schema determine which actions are available.
|
||||
- Feature gating and offered controls determine available actions. The API
|
||||
accepts a string case type and arbitrary JSON fields; do not infer typed
|
||||
case-field validation or a field-editing UI from those values.
|
||||
@@ -22,9 +22,13 @@ Message the configured bot/address from its external provider and follow the cor
|
||||
|
||||
Open `/apps/chat/:endpointId/settings` and the endpoint’s other available tabs; identity confirmation uses `/chat-identity/confirm`.
|
||||
|
||||
## Driving it
|
||||
## Driving it with Playwright and provider clients
|
||||
|
||||
Preconditions: follow the [baseline](./README.md#before-driving-a-journey). Enable chat connectors and use a test provider account/channel with permission to send messages. Setup is covered in [connection setup](./connection-setup.md).
|
||||
Preconditions: follow the [baseline](./README.md#before-driving-a-journey). Use a
|
||||
test provider account/channel with permission to send messages. Gates differ:
|
||||
AgentMail bypasses the chat-connectors UI gate, GitHub uses
|
||||
`enableGitHubReviewBots`, and other chat providers use `enableChatConnectors`.
|
||||
Setup is covered in [connection setup](./connection-setup.md).
|
||||
|
||||
### `external-thread`
|
||||
|
||||
@@ -40,6 +44,9 @@ Manual: Inspect the bound agent and identity, recent activity, and delivery fail
|
||||
|
||||
## Gotchas
|
||||
|
||||
- Visibility flags do not pause existing delivery. Disable the disposable
|
||||
endpoint itself to test stopped traffic.
|
||||
|
||||
- One provider’s passing fixture does not qualify another provider or live delivery.
|
||||
- An accepted send request does not prove the recipient received the message or file.
|
||||
- Use [steering](./steering.md) and [questions](./questions-and-approvals.md) for their channel-specific interaction checks.
|
||||
@@ -25,10 +25,15 @@ Use issue subresource, asset/skill, and run/heartbeat commands exposed by the CL
|
||||
### `worktree-instance`
|
||||
|
||||
Use the documented worktree/test-drive commands in an owned checkout; inspect `paperclipai worktree --help` first.
|
||||
Inspect root help too: `worktree:make/list/merge-history/cleanup` are top-level
|
||||
commands rather than subcommands of `worktree`.
|
||||
|
||||
## Driving it
|
||||
## Driving it with the Paperclip CLI
|
||||
|
||||
Preconditions: follow the [baseline](./README.md#before-driving-a-journey). Use `--help` from the checked-out CLI to discover exact arguments. Select a disposable company and isolated context; do not inherit an unknown API target.
|
||||
Pass the server origin to `--api-base` (without appending `/api`); the commands
|
||||
already include the API path. `issue create` requires explicit `--company-id`
|
||||
even when a profile stores a company. A persona label is not authentication proof.
|
||||
|
||||
### `client-context`
|
||||
|
||||
|
||||
@@ -21,7 +21,7 @@ Use the company switcher or `/companies`, then enter company Settings.
|
||||
|
||||
Use `/company/export` and `/company/import` where enabled, or the corresponding CLI company commands.
|
||||
|
||||
## Driving it
|
||||
## Driving it with Playwright and the Paperclip CLI
|
||||
|
||||
Preconditions: follow the [baseline](./README.md#before-driving-a-journey). Use two disposable companies and a small package fixture. Record cloud-managed restrictions and visible settings gates.
|
||||
|
||||
@@ -39,6 +39,10 @@ Manual: Export a small company, inspect the package preview, and import into a d
|
||||
|
||||
## Gotchas
|
||||
|
||||
- Restore an archived company through `/companies` → its menu → **Unarchive**.
|
||||
Archiving pauses eligible agents and cancels active runs; restoration handles
|
||||
agents paused specifically for company archival.
|
||||
|
||||
- Package presence does not prove a runnable imported company; credentials and environment bindings need separate verification.
|
||||
- Company deletion is a distinct gated operation; do not use it as the archive test.
|
||||
- Settings hidden in a managed deployment should be recorded as unavailable, not bypassed.
|
||||
@@ -57,7 +57,7 @@ or open `/apps/chat/connect`. Complete that provider's endpoint setup, then
|
||||
open its settings (`/apps/chat/:endpointId/settings`). Follow identity linking
|
||||
as the intended sender before messaging the agent externally.
|
||||
|
||||
## Driving it
|
||||
## Driving it with Playwright and the Paperclip CLI
|
||||
|
||||
Preconditions: follow the [baseline](./README.md#before-driving-a-journey).
|
||||
Choose a disposable provider account/resource and record its auth method,
|
||||
@@ -138,6 +138,10 @@ Slack setup does not qualify Discord, Telegram, GitHub, email, or iMessage.
|
||||
|
||||
## Gotchas
|
||||
|
||||
- Private endpoint rejection depends on deployment: owned loopback MCP servers
|
||||
are supported in local-trusted/private instances. Public authenticated
|
||||
deployments apply the private-network restriction.
|
||||
|
||||
- A successful OAuth callback is not proof of saved connection access or task
|
||||
continuation. Verify all three checkpoints.
|
||||
- Personal identity, shared account access, and per-agent access are distinct.
|
||||
|
||||
@@ -21,7 +21,7 @@ Use `/design-guide`, registered `/ux-lab/...` routes, or Storybook; consult `ui/
|
||||
|
||||
Open `/tests/perf/long-thread` only in an appropriate development environment.
|
||||
|
||||
## Driving it
|
||||
## Driving it with Playwright
|
||||
|
||||
Preconditions: follow the [baseline](./README.md#before-driving-a-journey). Use a development/test environment. Confirm the route is registered for the current build before claiming the page is available.
|
||||
|
||||
@@ -35,9 +35,14 @@ Manual: Open the specific fixture state, inspect controls and layout, then repea
|
||||
|
||||
Automated: [task thread component](../ui/src/components/IssueChatThread.test.tsx) does not establish a performance budget. Timing/scroll measurements need a dedicated browser run.
|
||||
|
||||
Manual: Record browser/build/data size, exercise scrolling and composing in the fixture, and retain timing or trace evidence. Confirm a representative real task does not regress before extrapolating fixture measurements.
|
||||
Manual: Record browser/build/data size, exercise scrolling and rendering in the
|
||||
long-thread fixture, and retain timing or trace evidence. Its composer is
|
||||
disabled; test composing separately in a real task. Confirm representative
|
||||
real-task behavior before extrapolating fixture measurements.
|
||||
|
||||
## Gotchas
|
||||
|
||||
- An unmounted or obsolete lab file remains in the conservative page inventory but is not a shipped entry point.
|
||||
- Intended developer use is not a universal runtime gate: Design Guide and
|
||||
global lab/performance routes are also registered in production builds.
|
||||
- Screenshots of a lab do not prove live provider behavior, persistence, or authorization.
|
||||
@@ -21,12 +21,15 @@ Open a document from the task thread or document side panel.
|
||||
### `task-output`
|
||||
|
||||
Open an attachment, output card, or file link from a task.
|
||||
For direct uploads, use the task side panel's **Open a new tab → Artifacts →
|
||||
Upload files** or drop files anywhere in that panel. On mobile, open
|
||||
**More actions → Properties** first. Agent conversations do not expose this upload path.
|
||||
|
||||
### `artifact-library`
|
||||
|
||||
Open `/artifacts` and its grouping, search, and media controls.
|
||||
|
||||
## Driving it
|
||||
## Driving it with Playwright and the Paperclip CLI
|
||||
|
||||
Preconditions: follow the [baseline](./README.md#before-driving-a-journey). Use a task with a text document, a small attachment, and a generated work product. Use nonsensitive fixture files.
|
||||
|
||||
@@ -38,9 +41,15 @@ Manual: Edit a paragraph, save, reload, and compare revisions. Add an annotation
|
||||
|
||||
### `task-output`
|
||||
|
||||
Automated: [attachment routes](../server/src/__tests__/issue-attachment-routes.test.ts) and [work products](../server/src/__tests__/work-products.test.ts) cover data/access behavior; [artifact arrival browser suite](../tests/e2e/artifact-tab-arrival.spec.ts) targets the arrival experience.
|
||||
Automated: [attachment routes](../server/src/__tests__/issue-attachment-routes.test.ts) and [work products](../server/src/__tests__/work-products.test.ts) cover data/access behavior; [artifact arrival browser suite](../tests/e2e/artifact-tab-arrival.spec.ts) targets arrival. [Artifact uploads](../tests/e2e/artifact-uploads.spec.ts) drives file selection/drop, filename collisions, retry, and preservation of paused/unassigned tasks.
|
||||
|
||||
Manual: Upload a fixture and have the test task produce an output. Preview and download each, compare the contents, and follow the source task link. Reload and test an inaccessible user/company; verify missing workspace-only resources have an explicit state.
|
||||
Manual: Upload a small Markdown file and CSV through Artifacts. Markdown becomes
|
||||
a versioned task document; other files become attachments. Read the document,
|
||||
download the CSV and compare bytes, then reload. Upload the same Markdown filename
|
||||
again and verify a separate suffixed document without overwriting the original.
|
||||
On a paused unassigned task, confirm upload does not start work or add comments.
|
||||
Verify agent-produced work products separately using an eligible test run.
|
||||
Test an inaccessible user/company and explicit missing workspace-only resources.
|
||||
|
||||
### `artifact-library`
|
||||
|
||||
|
||||
@@ -17,12 +17,16 @@ Implementation: [environment settings](../ui/src/pages/CompanyEnvironments.tsx),
|
||||
### `environment-settings`
|
||||
|
||||
Open Settings → Environments, then create/edit a supported target.
|
||||
The canonical URL is `/company/settings/instance/environments`. Enable
|
||||
`enableEnvironments` first; keep `enableManagedSandboxOnly` off to inspect local
|
||||
targets. The catalog is instance-scoped and mutation requires a local operator
|
||||
or instance administrator.
|
||||
|
||||
### `remote-execution`
|
||||
|
||||
Choose the environment in agent/project/workspace configuration, then inspect the resulting task/run and workspace.
|
||||
|
||||
## Driving it
|
||||
## Driving it with Playwright and the Paperclip CLI
|
||||
|
||||
Preconditions: follow the [baseline](./README.md#before-driving-a-journey). Use an owned disposable target and a supported provider plugin. Record provider, target identity, account, and spending constraints before a live remote run.
|
||||
|
||||
@@ -32,6 +36,10 @@ Automated: [environment UI](../ui/src/pages/CompanyEnvironments.test.tsx), [driv
|
||||
|
||||
Manual: Save a disposable target, reload it, and run its available probe. Verify both success and missing-credential/reachability errors identify the target. Confirm unavailable providers are not offered as working run targets.
|
||||
|
||||
The local row has no UI Test button. Its API/CLI probe reports host/cwd
|
||||
availability only; it does not validate an adapter binary, workspace, or task
|
||||
execution. Remote-driver probes and a real bounded run need their own evidence.
|
||||
|
||||
### `remote-execution`
|
||||
|
||||
Automated: [execution target](../server/src/__tests__/environment-execution-target.test.ts) and [custom images](../server/src/__tests__/environment-custom-images-service.test.ts) cover contracts; [SSH live test](../server/src/__tests__/environment-live-ssh.test.ts) requires its own external setup.
|
||||
|
||||
@@ -6,7 +6,7 @@ Implementation: [gateways](../ui/src/pages/apps/gateways/GatewaysList.tsx), [gat
|
||||
|
||||
## Sub-features
|
||||
|
||||
- `gateway`: create/edit an endpoint and select its exposed apps/tools.
|
||||
- `gateway`: create/edit an endpoint backed by a tool access profile.
|
||||
- `profiles`: define reusable tool access configuration and inspect draft/detail state.
|
||||
- `client-connect`: copy the intended endpoint/configuration to an authorized test client.
|
||||
- `tokens`: create/revoke client credentials and verify their scope.
|
||||
@@ -22,10 +22,15 @@ Open `/apps/gateways`, then a gateway’s Overview, Apps/Tools, Tokens, and Acti
|
||||
|
||||
Open `/apps/advanced/profiles`, create a profile, or follow its detail/edit route.
|
||||
|
||||
## Driving it
|
||||
## Driving it with Playwright and an MCP client
|
||||
|
||||
Preconditions: follow the [baseline](./README.md#before-driving-a-journey). Use a fixture tool connection and test client. Record gateway/profile, token owner, transport, and selected company.
|
||||
|
||||
Gateway creation selects a profile. Maintain that profile's app/action selection
|
||||
through the Apps/Tools links; the gateway form is not a separate tool picker.
|
||||
Use an active organization member for human-scoped access, and check the actual
|
||||
endpoint authorization independently of whether the admin UI is visible.
|
||||
|
||||
### `gateway-client`
|
||||
|
||||
Automated: [gateway service](../server/src/__tests__/tool-gateway-service.test.ts), [connect client dialog](../ui/src/pages/apps/gateways/ConnectClientDialog.test.tsx), and [token panel](../ui/src/pages/apps/gateways/panels/TokensPanel.test.tsx) cover service/UI contracts.
|
||||
|
||||
@@ -20,7 +20,7 @@ Open `/goals`, create a goal, and enter `/goals/:goalId`.
|
||||
|
||||
Inspect the goal context from a project or task and compare explicit versus inherited context.
|
||||
|
||||
## Driving it
|
||||
## Driving it with Playwright and the Paperclip CLI
|
||||
|
||||
Preconditions: follow the [baseline](./README.md#before-driving-a-journey). Prepare two goals and a disposable project/task in the selected company.
|
||||
|
||||
@@ -28,7 +28,12 @@ Preconditions: follow the [baseline](./README.md#before-driving-a-journey). Prep
|
||||
|
||||
Automated: [project-goal validation](../server/src/__tests__/project-goal-validation.test.ts) covers relationship validation; [goal detail control](../ui/src/pages/GoalDetail.test.tsx) only tests the properties toggle, not the whole creation journey.
|
||||
|
||||
Manual: Create a parent and child goal, save owner/status changes, and reload. Link a project and task through the offered controls, follow their goal context, and verify a cross-company or invalid relationship is rejected.
|
||||
Manual: Choose **Add Goal/New Goal**, enter **Goal title**, select the level, and
|
||||
create a parent. Open it and choose **Sub Goal** to create a child with the
|
||||
parent preselected. Change status in Properties and reload; follow the Parent
|
||||
Goal link and inspect Sub-Goals. Owner is currently display-only in this UI.
|
||||
Link a project and task through the offered controls, follow their goal context,
|
||||
and verify a cross-company or invalid relationship is rejected.
|
||||
|
||||
### `task-goal-context`
|
||||
|
||||
|
||||
@@ -25,7 +25,7 @@ Open `/decisions`, then a queue at `/decisions/queues/:key` and its source item.
|
||||
|
||||
Open `/search` or the navigation search entry.
|
||||
|
||||
## Driving it
|
||||
## Driving it with Playwright and the Paperclip CLI
|
||||
|
||||
Preconditions: follow the [baseline](./README.md#before-driving-a-journey). Seed assigned, unread, blocked, and resolved tasks plus a pending decision. Record the combined Inbox/Tasks experimental setting.
|
||||
|
||||
@@ -49,6 +49,10 @@ Manual: Search a unique task title and content phrase, open the result, and comp
|
||||
|
||||
## Gotchas
|
||||
|
||||
- Combined Inbox/Tasks requires both `enableCombinedInboxTasks` and
|
||||
`enableStreamlinedUi`. [Standalone Inbox tests](../ui/src/pages/Inbox.test.tsx)
|
||||
render archive/read controls and toolbars; this is not live multi-user evidence.
|
||||
|
||||
- Personal archival is not task cancellation or completion.
|
||||
- Legacy Inbox routes can redirect to the combined task view; record which host actually rendered.
|
||||
- Search service tests do not establish the relevance quality of every query.
|
||||
@@ -22,7 +22,7 @@ Use `paperclipai install`, `run`, `service`, `doctor`, `configure`, `channels`,
|
||||
|
||||
Use currently visible instance settings for experiments/access and `paperclipai db:backup --help` for one-off backup. Some legacy general-settings URLs redirect.
|
||||
|
||||
## Driving it
|
||||
## Driving it with Playwright and the Paperclip CLI
|
||||
|
||||
Preconditions: follow the [baseline](./README.md#before-driving-a-journey). Use a disposable data directory/service identity. Do not update, restart, or restore over an unrelated live instance as a documentation test.
|
||||
|
||||
@@ -40,6 +40,11 @@ Manual: Change a harmless experimental setting, reload, and confirm the affected
|
||||
|
||||
## Gotchas
|
||||
|
||||
- Source-checkout update/check commands report manual update guidance; they do
|
||||
not qualify a managed installation. Rollback restores retained CLI payloads,
|
||||
not database migrations. Service status applies to its selected service
|
||||
identity, not automatically to an owned foreground dev server.
|
||||
|
||||
- A successful backup write does not prove restoration works.
|
||||
- Settings visibility and mutability differ for cloud-managed and self-hosted instances.
|
||||
- Keep the three data paths distinct: first-party Telemetry, operator-configured OpenTelemetry, and local run logs.
|
||||
@@ -22,7 +22,7 @@ Use starred/recent/sidebar links and direct company-prefixed URLs.
|
||||
|
||||
Use Settings → Profile, `/u/:userSlug`, and the announcement well when a publication is available.
|
||||
|
||||
## Driving it
|
||||
## Driving it with Playwright and the Paperclip CLI
|
||||
|
||||
Preconditions: follow the [baseline](./README.md#before-driving-a-journey). Use two test users and companies; record the selected UI shell and any client-local preference storage.
|
||||
|
||||
@@ -32,6 +32,11 @@ Automated: [sidebar](../ui/src/components/Sidebar.test.tsx) and [sidebar prefere
|
||||
|
||||
Manual: Star and reorder supported resources, collapse a section, navigate and reload. Switch company/user and inspect the intended preference scope. Open an unavailable resource and a legacy route; verify clear fallback or correct redirect without leaking another company’s state.
|
||||
|
||||
Record persistence separately: section collapse is in-memory and reopens after
|
||||
reload; global sidebar collapse is disabled. Streamlined starred projects sort
|
||||
alphabetically. Recents are browser-local by user/company; stars and supported
|
||||
ordering are saved server-side. Local-trusted mode does not prove two-user isolation.
|
||||
|
||||
### `profile-announcements`
|
||||
|
||||
Automated: [profile page](../ui/src/pages/ProfileSettings.test.tsx), [announcement hook](../ui/src/hooks/useAnnouncement.test.tsx), and [announcement routes](../server/src/__tests__/announcements-routes.test.ts) cover separate UI/persistence layers.
|
||||
|
||||
@@ -17,12 +17,15 @@ Implementation: [web wizard](../ui/src/components/OnboardingWizard.tsx), [CLI se
|
||||
### `web-wizard`
|
||||
|
||||
Open `/onboarding` or the onboarding launcher in an empty company context.
|
||||
Use `/<company-prefix>/onboarding` to resume an existing company's agent setup;
|
||||
the unprefixed route starts at organization naming on self-hosted instances.
|
||||
Cloud-managed instances reuse their existing company for agent setup.
|
||||
|
||||
### `cli-setup`
|
||||
|
||||
Use `paperclipai onboard --help` or `paperclipai run --help` to select an isolated data directory before setup.
|
||||
|
||||
## Driving it
|
||||
## Driving it with Playwright and the Paperclip CLI
|
||||
|
||||
Preconditions: follow the [baseline](./README.md#before-driving-a-journey). Use a new isolated data directory and record deployment mode. A configured adapter may still need account authentication.
|
||||
|
||||
@@ -30,7 +33,20 @@ Preconditions: follow the [baseline](./README.md#before-driving-a-journey). Use
|
||||
|
||||
Automated: [wizard tests](../ui/src/components/OnboardingWizard.test.tsx) cover mocked wizard behavior; [onboarding browser suite](../tests/e2e/onboarding.spec.ts) supplies its own test environment. Live model execution is a separate check.
|
||||
|
||||
Manual: Create a disposable company and first agent. Save and leave midway, reopen, and finish. Follow the created first task, verify the selected agent and company, and inspect its output or explicit authentication gate. Reload both company and task.
|
||||
- **Create company and resume.** Start a fresh empty instance and run doctor using
|
||||
the [verification skill](../.agents/skills/verify-paperclip/SKILL.md). Run
|
||||
`node .agents/skills/verify-paperclip/scripts/onboarding-smoke.mjs --evidence-dir "$VERIFY_EVIDENCE_DIR"`.
|
||||
The helper opens `/onboarding` in a fresh self-hosted instance, enters a unique
|
||||
name in `e.g. Northwind Labs`, and chooses `Continue`. The agent-name field must
|
||||
appear and the saved company must have no invented goal. Reopen the
|
||||
company-prefixed onboarding route and reload; the agent step must appear
|
||||
without creating another company. Screenshots and a trace capture the real UI action and result.
|
||||
This proves company creation/resumption, not hiring or a first model run.
|
||||
|
||||
Manual: Create a disposable company and first agent. Leave midway, reopen the
|
||||
company's onboarding route, and finish; drafts save automatically. Follow the
|
||||
created first task, verify the selected agent and company, and inspect its output
|
||||
or explicit authentication gate. Reload both company and task.
|
||||
|
||||
### `cli-setup`
|
||||
|
||||
|
||||
@@ -21,7 +21,7 @@ Open `/pipelines`, `/pipelines/:pipelineId`, and its Settings route.
|
||||
|
||||
Open `/pipelines/:pipelineId/items/:caseId`, `/review-queue`, or `/learnings`.
|
||||
|
||||
## Driving it
|
||||
## Driving it with Playwright and the Paperclip CLI
|
||||
|
||||
Preconditions: follow the [baseline](./README.md#before-driving-a-journey). Enable pipelines and prepare a disposable pipeline with at least two stages and a fixture agent. Record any associated case type.
|
||||
|
||||
@@ -39,6 +39,10 @@ Manual: Open a pending review, inspect its source output and revision, accept or
|
||||
|
||||
## Gotchas
|
||||
|
||||
- Learnings is filtered case-event history, such as review decisions and forced
|
||||
moves, not a separate authored learning-record system. The UI visibility flag
|
||||
is separate from API authorization.
|
||||
|
||||
- These routes are experimental and may redirect or deny access when disabled.
|
||||
- ReviewQueue and Learnings are hosted from the pipeline module; a page-file inventory alone misses them.
|
||||
- Pipeline items and free-standing [cases](./cases.md) share records but have different navigation contexts.
|
||||
@@ -21,7 +21,7 @@ Open Settings → Plugins where permitted, or the CLI plugin commands.
|
||||
|
||||
Open `/plugins/:pluginId`, a registered plugin route, or its contributed company/project settings surface.
|
||||
|
||||
## Driving it
|
||||
## Driving it with Playwright and the Paperclip CLI
|
||||
|
||||
Preconditions: follow the [baseline](./README.md#before-driving-a-journey). Use a known local fixture plugin with bounded capabilities. Record installed version and any declared external dependencies.
|
||||
|
||||
@@ -39,6 +39,11 @@ Manual: Open the contributed page by navigation and cold deep link, perform a ha
|
||||
|
||||
## Gotchas
|
||||
|
||||
- Installation/lifecycle is instance-level; configuration can be company-scoped.
|
||||
Settings lives at `/company/settings/instance/plugins`. npm installation and
|
||||
bundled-local installation are different controls. Developer file-watcher
|
||||
reload is not a lifecycle action in the plugin manager.
|
||||
|
||||
- Installing a plugin does not authorize every host capability or external action.
|
||||
- Plugin-provided routes are dynamic; this map covers the host lifecycle, not every third-party feature.
|
||||
- Environment-driver plugins also need the [environment](./execution-environments.md) checks.
|
||||
@@ -21,7 +21,7 @@ Open `/projects`, create a project, and enter its overview or Issues view.
|
||||
|
||||
Open `/projects/:projectId/configuration` and its repository/workspace settings.
|
||||
|
||||
## Driving it
|
||||
## Driving it with Playwright and the Paperclip CLI
|
||||
|
||||
Preconditions: follow the [baseline](./README.md#before-driving-a-journey). Use a disposable project and a small repository you can access; record checkout and credential ownership.
|
||||
|
||||
@@ -39,6 +39,13 @@ Manual: Attach a small repository, save, and create a project task. Inspect the
|
||||
|
||||
## Gotchas
|
||||
|
||||
- `/overview` now selects Configuration, not a distinct overview page. Personal
|
||||
join/leave and star controls have their own persistence and should be checked
|
||||
when affected; **Star/Unstar {project name}** is the visible star action.
|
||||
- Project name slugs change on rename and have no historical alias lookup.
|
||||
Reopen a renamed project through its stable UUID route, wait for the current
|
||||
Configuration form, and record the newly normalized URL before reloading.
|
||||
|
||||
- Project repository configuration and a provisioned execution workspace are different records.
|
||||
- See [workspaces](./workspaces.md), [goals](./goals.md), and [budgets](./budgets-costs.md) for those features.
|
||||
- Do not assume a deleted or archived project should erase its historical task evidence.
|
||||
@@ -20,9 +20,10 @@ Open `/routines`, create one, and inspect `/routines/:routineId` and its edit se
|
||||
|
||||
### `trigger-history`
|
||||
|
||||
Use the routine’s trigger controls and History/Audit links; lists also offer Run now.
|
||||
Use the routine's trigger controls and Runs/Recent Runs/Audit links; lists also
|
||||
offer Run now. **History** records configuration versions, not executions.
|
||||
|
||||
## Driving it
|
||||
## Driving it with Playwright and the Paperclip CLI
|
||||
|
||||
Preconditions: follow the [baseline](./README.md#before-driving-a-journey). Use a disposable routine, deterministic test agent, and harmless target. Record schedule timezone and enabled state.
|
||||
|
||||
@@ -32,6 +33,11 @@ Automated: [routine detail helper](../ui/src/pages/RoutineDetail.test.tsx) tests
|
||||
|
||||
Manual: Save an agent, request, project, and a required variable. Reload and inspect the saved definition. Run now with a distinguishable input, follow the resulting task, and verify its output and history entry. Check a missing required input.
|
||||
|
||||
Variables are detected from `{{name}}` placeholders; mark a detected variable
|
||||
Required and verify missing input prevents Run routine. Configuration can be
|
||||
checked without model credentials. A successful subprocess run alone does not
|
||||
complete the generated task or establish a useful task output.
|
||||
|
||||
### `trigger-history`
|
||||
|
||||
Automated: [trigger wizard](../ui/src/components/routine-triggers/TriggerWizard.test.tsx) is component coverage; [routine integration](../server/src/__tests__/routines-e2e.test.ts) is server integration rather than a provider-wide browser proof.
|
||||
|
||||
@@ -21,7 +21,7 @@ Open an agent’s model/adapter configuration or the instance adapter settings p
|
||||
|
||||
Open an agent run at `/agents/:agentId/runs/:runId` from the agent or task activity.
|
||||
|
||||
## Driving it
|
||||
## Driving it with Playwright and the Paperclip CLI
|
||||
|
||||
Preconditions: follow the [baseline](./README.md#before-driving-a-journey). Record adapter, model, account, native/legacy runtime, and environment. Capabilities vary; one harness result is not a matrix-wide qualification.
|
||||
|
||||
@@ -39,6 +39,10 @@ Manual: Start a disposable task, open its running transcript, and follow it to a
|
||||
|
||||
## Gotchas
|
||||
|
||||
- Model accounts are managed in Apps and inline agent account controls.
|
||||
`/company/settings/connections` configures decision/fast-response models,
|
||||
not the general model-account library.
|
||||
|
||||
- The run log is instance database data; it is not the opt-out first-party Telemetry system or operator-configured OpenTelemetry tracing.
|
||||
- Authentication/model discovery does not establish tool, image, steering, or resumption capability parity.
|
||||
- Use [recovery](./recovery.md) for interrupted runs and [execution environments](./execution-environments.md) for local/SSH/sandbox differences.
|
||||
@@ -22,7 +22,7 @@ Open company Settings → Secrets and the company/personal secret views.
|
||||
|
||||
Use the Proposals tab or Import from vault where available.
|
||||
|
||||
## Driving it
|
||||
## Driving it with Playwright and the Paperclip CLI
|
||||
|
||||
Preconditions: follow the [baseline](./README.md#before-driving-a-journey). Use dummy credentials in an isolated company, two user identities, and a test agent. Never print actual secret values in evidence.
|
||||
|
||||
@@ -40,6 +40,10 @@ Manual: Review a dummy proposal, reject one and accept another, then inspect the
|
||||
|
||||
## Gotchas
|
||||
|
||||
- Vault import currently targets eligible AWS Secrets Manager providers.
|
||||
Organization-secret checks in local-trusted mode do not establish two-user
|
||||
personal-secret ownership or actual external-vault import.
|
||||
|
||||
- A displayed reference name is not proof that the actor can resolve its value at runtime.
|
||||
- Company secrets and user-specific values have different ownership.
|
||||
- Redaction needs verification in errors, activity, and run logs as well as the form.
|
||||
@@ -21,12 +21,15 @@ Open `/skills`, Discover, or `/skills/sources`; project import is available thro
|
||||
### `studio`
|
||||
|
||||
Use New skill or `/skills/studio/:skillId`; writable copies and source-backed originals differ.
|
||||
Direct authoring starts at `/skills/studio/new`: fill **Name** and **Slug**, then
|
||||
**Create skill**. Authoring/save/reload requires no model credential; running a
|
||||
Studio test is separate.
|
||||
|
||||
### `agent-enablement`
|
||||
|
||||
Use skill policy/agent selection in the library or an agent’s Skills tab.
|
||||
|
||||
## Driving it
|
||||
## Driving it with Playwright and the Paperclip CLI
|
||||
|
||||
Preconditions: follow the [baseline](./README.md#before-driving-a-journey). Use a harmless skill fixture and test agent with skill support. Record whether the source is local, bundled, project, or remote.
|
||||
|
||||
@@ -50,6 +53,9 @@ Manual: Enable the skill for one test agent and leave another without access. Ru
|
||||
|
||||
## Gotchas
|
||||
|
||||
- The release picker requires `enableBetaSkills`. Do not invent an Instructions
|
||||
accessibility label for the Studio body editor; inspect its current handles.
|
||||
|
||||
- Installed does not mean enabled for every agent.
|
||||
- Read-only GitHub/bundled skills may require a writable copy rather than an in-place edit.
|
||||
- Studio interaction coverage is in [questions and approvals](./questions-and-approvals.md); a passing mock is not a live harness test.
|
||||
@@ -21,10 +21,14 @@ Open the task’s parent/child and dependency links in task detail.
|
||||
|
||||
Inspect the task’s execution policy and available review, pause, resume, or stop controls.
|
||||
|
||||
## Driving it
|
||||
## Driving it with Playwright and the Paperclip API
|
||||
|
||||
Preconditions: follow the [baseline](./README.md#before-driving-a-journey). Prepare a parent, child, and dependent task plus implementer/reviewer agents. Record execution policy and use a disposable tree.
|
||||
|
||||
Parent/dependency navigation and inherited tree holds can be checked on unassigned
|
||||
tasks. Agents are required for the continuation and signoff portion. A tree pause
|
||||
creates a hold; it does not change task statuses to a paused status.
|
||||
|
||||
### `task-relationships`
|
||||
|
||||
Automated: [dependency wakeups](../server/src/__tests__/issue-dependency-wakeups-routes.test.ts) checks server behavior; visual relationship navigation remains a manual check.
|
||||
|
||||
+11
-2
@@ -26,9 +26,12 @@ Open `/issues` or a project’s Issues view; legacy `/tasks` links redirect.
|
||||
|
||||
Open `/issues/:issueId` from a row, mention, or direct link.
|
||||
|
||||
## Driving it
|
||||
## Driving it with Playwright and the Paperclip CLI
|
||||
|
||||
Preconditions: follow the [baseline](./README.md#before-driving-a-journey). Use an invokable test agent and a project; record whether a real or simulated harness handles the request.
|
||||
Preconditions: follow the [baseline](./README.md#before-driving-a-journey). Use a
|
||||
disposable project. Persistence checks can assign work to Me or leave it
|
||||
unassigned; execution checks additionally need an invokable test agent. Record
|
||||
whether a real or simulated harness handles execution.
|
||||
|
||||
### `new-task`
|
||||
|
||||
@@ -36,6 +39,10 @@ Automated: [creation dialog](../ui/src/components/NewIssueDialog.test.tsx) is co
|
||||
|
||||
Manual: Create a uniquely named task with an agent and project. Open its resulting link and reload. Confirm text, attachment, assignment, and project persisted, and verify execution started only when the task and agent are eligible.
|
||||
|
||||
Creation is request-first: enter **Describe a task…**, choose the assignee and
|
||||
project, and submit **Create task**. A standalone title field is not always shown.
|
||||
Use a human/unassigned task to verify persistence without invoking a model.
|
||||
|
||||
### `task-list`
|
||||
|
||||
Automated: [task-list helpers](../ui/src/pages/Issues.test.tsx) cover search URLs, pagination, deduplication, and presentation constants. The suite does not render the page or exercise filtering, grouping, opening a task, and returning; those interactions still need a running-browser check.
|
||||
@@ -50,6 +57,8 @@ Manual: Add a comment and edit authorized properties. Reload and verify history,
|
||||
|
||||
## Gotchas
|
||||
|
||||
- Priority remains an API field, but ordinary priority UI editing is hidden.
|
||||
|
||||
- A status label alone does not prove that the requested work exists.
|
||||
- Single-assignee and atomic checkout rules still apply when tasks are created from different hosts.
|
||||
- Use [coordination](./task-coordination.md), [steering](./steering.md), and [documents](./documents-artifacts.md) for their distinct workflows.
|
||||
@@ -21,7 +21,7 @@ Use the CLI teams catalog/preview/install operations in the selected context.
|
||||
|
||||
The catalog components and install wizard are available in developer tests/design surfaces. No top-level `/teams` route is registered; use the CLI/API for the current installation workflow.
|
||||
|
||||
## Driving it
|
||||
## Driving it with the Paperclip CLI
|
||||
|
||||
Preconditions: follow the [baseline](./README.md#before-driving-a-journey). Use a disposable company, a small catalog entry, and explicit company context. Check `paperclipai teams --help` for current subcommands.
|
||||
|
||||
@@ -39,6 +39,10 @@ Manual: When changing or mounting these components, exercise preview, manager/so
|
||||
|
||||
## Gotchas
|
||||
|
||||
- Catalog preview records an audit event but installs no agents/projects. Use
|
||||
`teams browse/inspect/preview` with explicit company and origin API context;
|
||||
the current product has no `/teams` route.
|
||||
|
||||
- Source files do not prove that a screen is reachable in the current app.
|
||||
- Installed team configuration may still need credentials or environment setup.
|
||||
- Company package import is a separate [portability](./companies.md) workflow.
|
||||
@@ -22,7 +22,7 @@ Use the task’s workspace card or Files panel and follow its workspace link.
|
||||
|
||||
Open `/workspaces`, a project workspace, or `/execution-workspaces/:workspaceId` and its Services/Configuration/Runtime logs views.
|
||||
|
||||
## Driving it
|
||||
## Driving it with Playwright and the Paperclip CLI
|
||||
|
||||
Preconditions: follow the [baseline](./README.md#before-driving-a-journey). Enable isolated-workspace UI where needed. Use an owned disposable repository and environment; record workspace ID and branch.
|
||||
|
||||
@@ -40,6 +40,11 @@ Manual: Provision a disposable workspace, wait for readiness, start an available
|
||||
|
||||
## Gotchas
|
||||
|
||||
- UI visibility, default isolation policy, the local environment row, a workspace
|
||||
record, and a running service are distinct prerequisites. A fresh instance
|
||||
still needs an owned cwd/repository and runtime commands; `/workspaces` lists
|
||||
existing work. Reopening depends on lifecycle, not a generic detail action.
|
||||
|
||||
- A link or running badge does not prove the service is reachable from the user’s browser.
|
||||
- Workspace IDs, runtime leases, and repository paths are not interchangeable.
|
||||
- Use [recovery](./recovery.md) for divergence/export repair and [CLI operations](./cli-operations.md) for local worktree tooling.
|
||||
Reference in new issue
Block a user