e2e.md promised "unmatched calls fail the test" without qualification, but only
HTTP is gated: an unmatched subscription is deliberately answered with silence
and merely attached as a diagnostic, because a flow page opens ~15 of them and
erroring on the unmocked ones would poison the client. A typo'd subscription key
therefore surfaces as a UI timeout, which is the opposite of what the doc led a
contributor to expect. Say which half is gated and where the missed operations
show up.
list_detail_pages.md named /settings/api-tokens as the sole exception to the
list/detail pattern, so a reader looking to extend /settings/providers/:id would
grep for a use-provider-detail-navigation.ts that never existed — as would one
looking at /settings/prompts/:name. Both are single-entity forms; name them, and
add the two phantom hooks to the doc's own do-not-reintroduce table.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
--with-deps hardlinks node_modules in, and a hardlink cannot cross filesystems,
so on a host whose $TMPDIR is its own mount the flag was unusable: `cp -al`
fails and the ERR trap tears the half-built worktree down, leaving the caller
with an empty $SANDBOX. Reproduced on a real second filesystem (attached image,
st_dev 16777238 vs 16777234): the old script dies with "Cross-device link" at
rc=1, the new one probes with a real link first, reports the fallback on stderr
and builds the sandbox beside the repo. Same-filesystem hosts keep using
$TMPDIR untouched, and PENTAGI_SANDBOX_ROOT overrides both.
`clean` accepts a path under either root, so a sandbox built through the
fallback can still be removed by the containment check.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
`clean <path>` ran `git worktree remove --force || rm -rf` on whatever it was
handed. git exits 128 both for the main working tree and for any directory that
is not a worktree, so the rm fallback is what runs for every wrong argument, and
sitting left of `||` it never trips `set -e` — `clean <repo-root>` erased the
checkout including .git, and `clean <unrelated dir>` erased it silently at rc=0
because the git error was redirected away. Resolve the target and refuse
anything that is not a child of the sandbox root, and let git's refusal print.
The no-argument form was the other half: it wiped the whole root, which every
agent on the host shares, so the documented copy-paste flow with an unset
$SANDBOX took out concurrent live sandboxes. The sweep is now the explicit
`clean --all` and only removes what has been untouched for two hours; a bare
`clean` is an error.
Four of the six new cases fail against the old script.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The doc told readers to dispatch the workflow "with tier: stand", an input
fc15ec8 removed — workflow_dispatch takes none — and claimed the protected
Environment is what stops a fork or mislabeled run from reaching secrets. It is
not: the job's `if` (label name plus a non-fork head, added in 7b1ecfd) skips
before the Environment is ever evaluated. Describe the real two-gate model: the
`if` first, then the Environment for a run that clears it.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Verifying a claim about a gate means breaking something on purpose, and review
runs kept doing that in the working tree: probe specs left behind under src/ and
e2e/, tracked files edited mid-run, and once an `rm -rf` of a directory the run
had not created. Nothing survived to the commits, but only because each run
happened to clean up after itself.
`review-sandbox.sh` hands out a detached `git worktree` under $TMPDIR, optionally
carrying the uncommitted diff and a hardlinked node_modules so pnpm, vitest and
playwright run there. Verified by doing the damage on purpose inside one — an
edited world.ts, a stray probe spec, a deleted frontend/docs — and confirming
the real tree came back untouched.
node_modules is hardlinked rather than symlinked because pnpm treats a symlinked
modules dir as foreign and tries to purge and reinstall it, which would reach
through the link into the real tree; it aborted here only for want of a TTY.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
`DetailNavigationToolbar` was documented as returning `null` when
`controller.itemsEmpty`. It has no such branch — it always renders the
fragment, and `itemsEmpty` is read nowhere outside the module's own tests.
Describe what it actually does: mount straight away and let the buttons show
their disabled/"–/0" state until items land.
"Prev/Next runs the same matcher the list filter uses" is also untrue. The
toolbar's `createTextMatcher` normalizes NFKD and strips combining marks;
`DataTable`'s `globalFilterFn` only lowercases. Searching `cafe` hides a `café`
row from the table while Prev/Next still steps onto it. Record the divergence
rather than the intent.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The B1-B3/B6/B7 pass guarded the settings detail pages' error branch with
`&& !data`, but each detail page has an `if (loading)` branch that runs FIRST,
and it was left unguarded — so the fix it was meant to deliver never applied.
The queries are cache-and-network, so a background revalidation (a list→detail
navigation into a warm cache, or a post-save refetchQueries) reports loading
true with cached data present and blanks the edit form to the full-page spinner
before the guarded error branch is ever reached.
- settings-prompt.tsx / settings-provider.tsx: `if (loading)` -> `&& !data`
- template.tsx spinner: `if (!isNew && isLoadingTemplate)` -> `&& !template`,
which also realigns it with knowledge.tsx (fixed in 28ab3d2 to gate on the
entity, not raw loading) — the two had silently diverged.
- docs/list_detail_pages.md: the "canonical render gate" recipe still taught the
unguarded `if (isLoading)` it tells new pages to copy; both branches now gate.
settings-provider.test gains the loading-with-cached-data case (revert -> red);
the detail loading branch had zero coverage before. Found by the adversarial
review of the previous fix pass — the guard I applied was one line short.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The comment on AppHeaderActions claimed the component pins itself to the right.
It does not — the sibling content block takes `flex-1` and pushes it there — and
the rule it stated was an instruction to call sites, not a fact about the twelve
characters below it. The convention now lives in docs/list_detail_pages.md, next
to the detail-page recipe that needs it.
Two more went the same way, by making the code carry the point instead: the
knowledge header's two ids are now `documentId` and `routeKnowledgeId`, so
there is nothing to warn about, and the separator that doubles up is gated on a
named `hasViewRow`. In the pager spec the injected delay is held by an assertion
that counts the samples taken before the sibling appears — removing the delay
now fails the test instead of quietly emptying it.
Across the e2e specs, six comments that only explained why a test exists are
gone and five more are down to one line.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The intro left /settings/prompts out of the list pages built on these
pieces and implied every one of them has a detail page; api-tokens edits
in place. The removed/renamed table pointed at stale mentions of the
deleted writer hook in two files that no longer contain any.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The stand tier's fork/secret protection is the protected Environment's reviewer
gate, not the job's label condition — a fork PR can carry the label but blocks
on a human before any secret. Say so, note the tier now lives in its own
workflow, and document the E2E_STAND_* secrets and their E2E_* tool names.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The probes carry `transition-colors`, so reading the computed background right
after hover() sampled a colour still interpolating away from the rest state —
the half of the gate that exists to catch hover regressions could not catch one.
Disabling the transition on the probes fixes it: verified by giving one variant a
deliberately unreadable hover fill, which the assertion now reports as "badge
blue on hover" and previously let through.
Key the probe list off the variant union so a new badge variant fails to compile
until it is probed, rather than silently going unmeasured.
Also correct two route-manifest ownerships that misdirect diff-scoping: the flows
provider is mounted by the detail route as well as the list, and the provider
detail page was attributed to the list route that never renders it.
Doc fixes: the route sweep is tagged @cross, not @coverage; the licenses link in
CONTRIBUTING resolved outside the repository; and the frontend prettierignore
duplicated entries prettier already reads from gitignore.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The suite tags every spec but the docs only hinted at "@smoke, …". List the real
tags (@smoke/@flows/@crud/@coverage/@cross/@visual/@real/@stand) with their meaning
and how to filter, and spell out the two conventions the mock gate reserves:
@quarantine to isolate a flaky spec (excluded via --grep-invert) and @generated for
recorder-derived cassettes.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The base compose file pins fixed network names, so any network left un-overridden
in this e2e override joins the stack to the developer's live observability and
langfuse networks. Give all three networks e2e-scoped names. Add a healthcheck to
the mock-LLM service and make pentagi depend on service_healthy, so a crash-looping
mock can never read as a started dependency and silently mask itself.
Update frontend/docs/e2e.md to match reality: the stand tier runs @stand only
(flow-run drives a real paid agent and stays local), every mock run rebuilds its
own bundle by design, and the trend/affected-routes notes.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The co-located components/shared/README.md documented hooks that no longer
exist. `usePagination` was never a hook, and the writer `useTableQueryFilter`
was replaced by the unified `useTableState`, so the copy-paste recipe it shipped
would not compile. Detail pages had likewise moved from spreading a `toolbarProps`
object to passing a `controller` prop.
Rewrite it as frontend/docs/list_detail_pages.md, matching the flat
per-subsystem layout backend/docs already uses: exact signatures for every
hook, component and storage helper, the wiring the five live list pages
actually use, and an explicit table of removed names so they are not
reintroduced. Drop the source-adjacent README — no other directory under
frontend/src or backend/pkg carries one.
Fix the comments that still named the removed hook, plus two claims in them
that contradict the code: the controlled filter is URL-backed only (DataTable
persists sorting, column visibility, page size and search columns, never the
filter), and `busy` on InlineEditInput disables the Save and Cancel buttons
while leaving the input editable.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
schema-compat set NODE_TLS_REJECT_UNAUTHORIZED=0 process-wide (CodeQL
js/disabling-certificate-validation, high). A real stand has a valid
cert; only the local self-signed Tier-2 stack needs it, so the operator
now opts in via their own shell env, never in code.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Lays the Phase-3 substrate on top of the three tiers.
- schema-compat pre-flight (e2e/tools/schema-compat.mjs): introspects a
target backend's live GraphQL schema and validates every frontend
operation against it, so deploy skew (a renamed/removed field) fails
once, readably, instead of as dozens of red specs. Verified against the
live Tier-2 backend both ways: 105 operations pass, an injected bogus
field is caught with the exact location.
- trend aggregation (trend.mjs): turns a run's results.json into one JSONL
record (p50/p95 spec duration, slowest three, pass/flaky/fail) so slow
regressions are visible, not just green/red; CI appends it to a
90-day-retained artifact.
- diff-scoping (affected-routes.ts + affected.ts): maps a diff to the
manifest routes it touches via each route's owning sources — the
substrate for selective runs and for scoping the exploratory agent.
Pure mapping fn, unit-tested (backend-only → none; shared infra → all;
owned → that route).
- CI: an e2e-stand job (label-gated + a protected Environment whose
reviewers approve before secrets are exposed; schema-compat runs first),
a trend step, and an LLM-independent @stand smoke.
- docs: the stand tier, trend/affected tools, and the LLM advisory recipe
(playwright-mcp + init-agents + guardrails) — the recipe the
deterministic tiers plug into, not a bespoke bot.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
18 baselines (9 manifest routes × light/dark) compared only inside
mcr.microsoft.com/playwright:v<version>-noble, so pixels are identical on
every machine and CI; the wrapper derives the tag from the installed
@playwright/test version, making the pin drift-proof, and a CI guard
fails loudly if the workflow's container tag falls behind.
macOS hosts cannot run the visual project directly (parallel darwin
baselines) nor mount their node_modules into the container (native vite
binaries): the wrapper builds dist on the host and the container serves
it with a dependency-free static server — route mocks intercept API
calls before the network, so no proxy is needed. The xterm canvas is
masked (SwiftShader pixels are driver-dependent). Determinism proven by
back-to-back container runs. The e2e-visual CI job is advisory, never a
required check.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
specs/real/** now exercise the actual agent loop end to end: create flow
-> image/language/title -> tool-call-ID sampling -> subtask_list plan ->
done barrier -> subtask_patch refine -> report_result — with messages
streamed over the real GraphQL websocket and the flow settling in
Waiting. The mock LLM is ~150 lines of Node driven by a deterministic
first-match transcript; the custom provider env seam means zero backend
changes.
The stack is fully isolated from a developer machine:
- own compose project/network/ports (8444/5433), coexists with a dev stack
- --env-file /dev/null so the developer's .env (live keys, DOCKER_HOST,
config paths) never leaks into the e2e backend
- flow ids seeded from 90001: sandbox containers are named
pentagi-terminal-<flowId> on the shared docker daemon, so the range
keeps them clear of dev flows and makes runner cleanup unambiguous
(down -v cannot remove them — the backend spawns them outside compose)
- pentagi healthcheck via busybox wget (the alpine image has no curl),
since the base service has none and up --wait returns too early
Auth for live tiers moved to the canonical setup-project + storageState;
the localStorage seed is now mock-tier-only (a forged client session has
no cookie behind it and the first 401 wipes it). CI gains a nightly/
dispatch-only e2e-local job.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The default tier runs the production bundle (vite build + preview)
against a cassette-driven mock of the whole API surface — GraphQL over
HTTP, graphql-transport-ws subscriptions, and REST — so the suite needs
no backend, no keys, and no secrets, and fork PRs can run it.
- mock engine: (operationName, variables) matching with sequenced
entries and world flags (login flips /info guest->user without
call-order coupling); unmatched calls answer 501 and fail the test,
so nothing leaks through the vite preview proxy to a live backend
- ws mock follows the graphql-transport-ws contract the app's client
needs: immediate ack, nothing before ack, streams stay open, delta
cursors survive reconnects, drops use retryable close codes
- clock and timezone pinned on the mock tier: formatDate branches on
isToday/isThisYear, so unpinned cassette dates rot within a day
- cassettes are TS modules typed against the generated GraphQL types,
so schema/operation drift fails the existing tsc gate at compile time
- CI: fork-safe e2e.yml (read-only token, no secrets, no write steps)
plus e2e-report.yml posting a sticky PR comment via workflow_run,
resolving fork PRs by head SHA
- smoke specs (login redirect, form login, authenticated /flows render)
green on the mock tier; e2e/ wired into tsc, eslint, and prettier
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sergey Kozyrenko