mirror of
https://github.com/Heretek-AI/IUMBTEMS.git
synced 2026-10-12 02:03:00 +00:00
test(audit-189): resolve policy.test.ts with main at merge (Refs #260)
Content conflict in the shell-policy screen: the branch pins a direct named `es research export` denial for every agent kind (#255) where main has the single-build-agent line. The loop subsumes the line, so the resolution keeps the loop and drops the line. Refs #189
This commit is contained in:
John Smith
@@ -63,6 +63,12 @@ jobs:
|
||||
# NOTE: the ./ prefix is load-bearing. A bare packs/foo.tgz is
|
||||
# parsed as a GitHub user/repo shorthand (git ls-remote failure);
|
||||
# ./packs/foo.tgz is resolved as the local tarball.
|
||||
# Fail closed: an empty tarball list (e.g. --release format drift
|
||||
# through the awk above) must never exit 0 having published nothing.
|
||||
if [ -z "${{ steps.packages.outputs.tarballs }}" ]; then
|
||||
echo "publish: no tarballs listed by scripts/packages.ts --release" >&2
|
||||
exit 1
|
||||
fi
|
||||
for tgz in ${{ steps.packages.outputs.tarballs }}; do
|
||||
echo "::group::npm publish $tgz"
|
||||
if [ "$DRY_RUN" = "true" ]; then
|
||||
|
||||
@@ -59,7 +59,7 @@ spike as proof).
|
||||
| :--- | :--- |
|
||||
| Approvals are human-only; trust, waivers and resume stay terminal-only: passphrase confirmation (terminal echo-off, TUI masked dialog, or loopback browser input per ADR 0002) unlocks the sealed Ed25519 human key (signed, hash-bound) and every surface signs approvals in-process — the TUI with channel `tui`, the browser through the loopback-only fleet endpoints with channel `web`; the approve/trust/resume RPCs no longer exist; agents may request, never grant | `approval/`, `gates/waivers.ts` |
|
||||
| Control files are deny-write for all agents: `.factory/{gates.json,config.json,frontier.json,waivers,approvals,runtime}`, the evidence (`.factory/research/{sources,coverage.json,dossier.json,brief.pcrb.json}`, `.factory/claims/`), `.git/config` and git hooks. The pinned ones (gates, config, frontier, approvals, waivers) are also hash-checked at every gate run; cached sources are engine-sealed (HMAC) and content-addressed | `trust/control.ts` |
|
||||
| Per-seat path scopes: manager writes factory docs, programmer writes its worktree, QA and the auditor pair write nothing, research alpha/beta write only their own notes (`.factory/research/alpha.md` / `beta.md`) and `research/REPORT.md` has one writer per run mode — the factory in build runs, the research synthesizer in research runs (`PolicyContext.runMode`; an unreadable mode denies both), brainstorm/harvest/design/scout write only their `.factory/<x>/notes/`; research, scout and auditor seats are `web: "cached"` (host websearch/webfetch denied, web facts only through the cached `es_research_*` tools) and run `--unshare-net` sandboxed, while user agents keep host web tools | `trust/policy.ts`, `agents/registry.ts` |
|
||||
| Per-seat path scopes: manager writes factory docs, programmer writes its worktree, QA and the auditor pair write nothing, research alpha/beta write only their own notes (`.factory/research/alpha.md` / `beta.md`) and `research/REPORT.md` has one writer scope per mode — factory-docs seats (factory and manager) in build runs, the research synthesizer in research runs (`PolicyContext.runMode`; an unreadable mode denies both), brainstorm/harvest/design/scout write only their `.factory/<x>/notes/`; research, scout and auditor seats are `web: "cached"` (host websearch/webfetch denied, web facts only through the cached `es_research_*` tools) and run `--unshare-net` sandboxed, while user agents keep host web tools | `trust/policy.ts`, `agents/registry.ts` |
|
||||
| Factory seats launch seats in the foreground only: `subagent` with `background: true` from any seat is refused, and so is launching a seat that is still running; seat state and last activity are recorded in `.factory/runtime/seats.json` (advisory) | `opencode/src/policy.ts`, `opencode/src/seats.ts` |
|
||||
| Code audits block: an opened phase audit holds `passPhase` (a missing one too with `audit.phase: required`, set via `es config set`), any other open or failed audit refuses the next stage transition; every finding is witnessed against the file on disk; only a human dismisses one | `factory/machine.ts`, `codeaudit/` |
|
||||
| Scout verdicts are computed by core: `adopt` survives only for a verified, whitelisted, permissive license | `scout/verdict.ts` |
|
||||
|
||||
@@ -37,8 +37,10 @@
|
||||
scanning, OSV, budgets (diff size, file length, complexity, dependency
|
||||
justification, test-with-behaviour), with structured `file:line:rule`
|
||||
findings and signed, expiring waivers.
|
||||
- **Evidence-first research.** Content-addressed source cache, a pure verbatim
|
||||
quote verifier, and an epistemic auditor that downgrades ungrounded claims.
|
||||
- **Evidence-first research.** Content-addressed source cache, a quote
|
||||
verifier that is verbatim after documented normalisation (NFKC, typography
|
||||
folds, markdown stripping, whitespace collapse), and an epistemic auditor
|
||||
that flags, prunes and refuses ungrounded claims.
|
||||
Web sources come through an ordered backend chain (Scraper-Swarm gateway,
|
||||
Brave, Firecrawl, SearXNG, direct fetch) with per-backend cooldowns: one
|
||||
provider's outage or rate limit fails over to the next, while a safety
|
||||
@@ -171,11 +173,11 @@ bun install
|
||||
bun run check # biome + tsc + bun test (all packages)
|
||||
bun run docs:gen # regenerate schemas/ + docs/ after schema changes
|
||||
bun run docs:check # CI drift check
|
||||
scripts/pack-smoke.sh # build, pack and install the packages; run them under Node and Bun
|
||||
scripts/pack-smoke.sh # build, pack and install the packages; run the CLI under Node and the plugin server under Bun
|
||||
scripts/v2-head.sh # run the real-host suite against OpenCode v2 HEAD
|
||||
```
|
||||
|
||||
Platform: Linux, Node ≥ 22 or Bun. No Python.
|
||||
Platform: Linux (CI runs ubuntu-latest), CLI under Node 22, plugin server under Bun. No Python.
|
||||
|
||||
## Documentation
|
||||
|
||||
|
||||
@@ -18,15 +18,15 @@ and Antigravity adapters ship in 1.1–1.3). Generated: do not edit by hand.
|
||||
| subagents | ENFORCED | `packages/opencode/test/scoping.test.ts` | Depth 1; hidden seats are unlisted but invocable by ID, and only from the seats allowed to spawn them. Factory seats are not offered Code Mode execute. |
|
||||
| compaction | ENFORCED | `packages/opencode/test/plugin.test.ts` | Session compaction injects the compact factory-state block for factory seats. |
|
||||
| websearch | ENFORCED | `packages/opencode/test/research.test.ts` | The configured provider registers as the host websearch; results are cached and citable. Research, scout and auditor seats are cached-only (es_research_search/fetch) under an offline (--unshare-net) sandbox; host web caching applies only in a project that already has .factory/, and the cache is engine-sealed (planted entries refused). |
|
||||
| lsp | ENFORCED | `packages/opencode/test/lsp.test.ts` | The lsp config key is read and served by our own manager (v2 accepts but does not run it). |
|
||||
| lsp | ENFORCED | `packages/opencode/test/lsp.test.ts` | The `lsp` key in v2's `opencode.json` — not an ES config key (ES layers strict-reject it) — is read and served by our own manager (v2 accepts but does not run it). |
|
||||
| panels | ENFORCED | `packages/opencode/test/panels.test.ts` | Four session.panel dashboards (factory, LSP, hooks, brainstorm), rendered by OpenTUI and refreshed on server changes; Esc/q closes a panel, f toggles fullscreen. The factory dashboard leads with the run headline, seats and research progress; a prompt-footer indicator shows the stage and the running seat. |
|
||||
| research | ENFORCED | `packages/opencode/test/research.test.ts` | Engine-sealed cache (#52), fragment-level quote verifier (every ellipsis fragment >= 12 chars; code excerpts one contiguous span of at most 60 lines), auditor, providers and the websearch bridge. Research-only runs complete through the deep-researcher (thesis/antithesis/synthesis at depth 1) at DONE. |
|
||||
| brainstorm | ENFORCED | `packages/opencode/test/brainstorm.test.ts` | Lens fan-out, record/score/complete tools, dedupe, rubric, shortlist with a forced outlier. Runs are id-scoped (the human's /brainstorm is the default run); the grill and the factory run callable brainstorms at depth 1 and get the shortlist as JSON. |
|
||||
| brainstorm | ENFORCED | `packages/opencode/test/brainstorm.test.ts` | Lens fan-out, record/score/complete tools, dedupe, rubric, shortlist with a forced outlier whenever the shortlist has ≥2 slots. Runs are id-scoped (the human's /brainstorm is the default run); the grill and the factory run callable brainstorms at depth 1 and get the shortlist as JSON. |
|
||||
| harvest | ENFORCED | `packages/opencode/test/harvest.test.ts` | Fail-closed SPDX detection, provenance profiles, policy-enforced matrix, clean-room specs. Runs are id-scoped (the human's /harvest is the default run); es_harvest_target gives the grill, factory, scout and harvester deterministic verdicts, and prior-art search is shared with brainstorm callers. |
|
||||
| design | ENFORCED | `packages/opencode/test/design.test.ts` | Queereye interview, DTCG tokens, contrast gate, generated guide and drift check; phase-02 specs/webref/csf/tui renders and the phase-03 ledger bundle with its cite-gate receipt. |
|
||||
| claims | ENFORCED | `packages/core/test/research.test.ts` | Witnessed claims and dossiers: the evidence cache, dossiers and the claim ledger are tool-only; planted cache entries are refused (engine seal) and every VERIFIED quote and code span is re-checked against the bytes on disk. |
|
||||
| audit | ENFORCED | `packages/opencode/test/fires.test.ts` | Code-audit pair: only auditor seats record verdicts; every finding is witnessed on disk (a hallucinated line is refused); quote fragments carry >= 12 chars each and a thesis pass needs a witnessed invariant; verdicts block (an opened phase audit holds the merge; audit.phase required also holds a missing one until it passes or a human dismisses it). |
|
||||
| scout | ENFORCED | `packages/opencode/test/fires.test.ts` | OSS scout: cached-only web under an offline (--unshare-net) sandbox, fail-closed license verdicts computed by core (adopt only for verified permissive licenses), OSV advisories cited. |
|
||||
| scout | ENFORCED | `packages/opencode/test/scout-host.test.ts` | OSS scout: cached-only web under an offline (--unshare-net) sandbox, fail-closed license verdicts computed by core (adopt only for verified permissive licenses), OSV advisories cited (canned-OSV content in core's scout suite). |
|
||||
| research-fires | ENFORCED | `packages/opencode/test/fires-research.test.ts` | Research fires: the whole research loop on the real host with loopback-only servers — grounded quotes survive the audit, failover records provenance, blocked never fails over, planted sources are refused, retractions degrade to STALE, renders snapshot. |
|
||||
|
||||
## claude
|
||||
|
||||
@@ -76,6 +76,8 @@ Do not edit by hand: run `bun run docs:gen` after changing a schema (CI runs `bu
|
||||
| run-phase-telemetry | `schemas/run-phase-telemetry.schema.json` |
|
||||
| run-state-seal | `schemas/run-state-seal.schema.json` |
|
||||
| run-telemetry | `schemas/run-telemetry.schema.json` |
|
||||
| scout-advisories-file | `schemas/scout-advisories-file.schema.json` |
|
||||
| scout-advisory-query | `schemas/scout-advisory-query.schema.json` |
|
||||
| scout-advisory | `schemas/scout-advisory.schema.json` |
|
||||
| scout-assessment | `schemas/scout-assessment.schema.json` |
|
||||
| scout-plan | `schemas/scout-plan.schema.json` |
|
||||
|
||||
@@ -119,7 +119,8 @@ prompt-injected agent acting through a seat or user-agent shell.
|
||||
(TTL ≤ 120 s) binds the preview to `hashJson(subject)`, and signing
|
||||
re-derives the subject and refuses a mismatch.
|
||||
- **I4** Attempts are limited: at most N wrong passphrases per M minutes per
|
||||
surface (recommended 5 per 10 min), then a lockout and an audit entry.
|
||||
state dir (recommended 5 per 10 min; the CLI and the TUI share one bucket),
|
||||
then a lockout and an audit entry.
|
||||
- **I5** The channel is recorded (`cli`, `tui` or `web`) and audited.
|
||||
- **I6** Mutating approval endpoints are unreachable from agent contexts: no
|
||||
agent tool and no MCP tool; the TUI path requires the TUI process's dialog;
|
||||
@@ -201,8 +202,8 @@ ticket before merge. All probes apply to both surfaces unless marked.
|
||||
- The TUI plugin links core's approval service directly and signs in-process;
|
||||
no new RPC, no passphrase on the wire anywhere except loopback for the
|
||||
browser path.
|
||||
- Attempt-limit state lives per surface (a small file in the state dir for
|
||||
the TUI/CLI; server memory plus audit for the web path).
|
||||
- Attempt-limit state lives per state dir (one shared attempt file for
|
||||
the CLI and the TUI; server memory plus audit for the web path, which keeps its own dir).
|
||||
- Docs and messages stop saying approvals are terminal-only, while trust,
|
||||
waive, resume, key seal, config set and export keep saying it.
|
||||
|
||||
|
||||
@@ -0,0 +1 @@
|
||||
{"agent":"harvester","checks":{"toolsUsed":["es_harvest_plan","es_harvest_scan","es_harvest_matrix","es_harvest_complete"]},"description":"The harvester tears down two vendored local projects end to end (plan, scan, read, matrix, complete): MIT stays depend, GPL-3.0-only is downgraded to clean-room. Graded from .factory/harvest/runs/default/harvest.json. Provenance-per-field and no-copied-spans need the scrubber/manual review (see issue #192).","files":{"vendor/copycalc/LICENSE":"@fixture:licenses/GPL-3.0-only.txt","vendor/copycalc/index.js":"const { copycalcQuux7429 } = require('./lib.js')\nmodule.exports = { run: (xs) => xs.map(copycalcQuux7429) }\n","vendor/copycalc/lib.js":"function copycalcQuux7429(x) { return x < 0 ? 0 - x : x * 2 }\nmodule.exports = { copycalcQuux7429 }\n","vendor/copycalc/package.json":"{\"name\": \"copycalc\", \"version\": \"1.0.0\"}\n","vendor/retcalc/LICENSE":"@fixture:licenses/MIT.txt","vendor/retcalc/index.js":"const { retcalcQuux7429 } = require('./lib.js')\nmodule.exports = { run: (xs) => xs.map(retcalcQuux7429) }\n","vendor/retcalc/lib.js":"function retcalcQuux7429(x) { return x < 0 ? 0 - x : x + 1 }\nmodule.exports = { retcalcQuux7429 }\n","vendor/retcalc/package.json":"{\"name\": \"retcalc\", \"version\": \"1.0.0\"}\n"},"fire":{"expected":[{"candidate":"copycalc","verdict":"clean-room"},{"candidate":"retcalc","verdict":"depend"}],"kind":"harvest"},"id":"darkharvest-e2e","maxSteps":14,"prompt":"Tear down two local candidates for a session handoff feature: local:./vendor/retcalc and local:./vendor/copycalc. This is a headless run: es_harvest_plan them (objective 'session handoff teardown'), es_harvest_scan each, es_harvest_read one source file per candidate, submit one matrix row proposing depend for both with a one-line behaviour evidence per cell (describe behaviour in your own words, do not paste source), then es_harvest_complete."}
|
||||
@@ -0,0 +1,8 @@
|
||||
{
|
||||
"description": "Research fetches a served fixture page and quotes its key sentence verbatim (no false VERIFIED).",
|
||||
"agent": "es-research-alpha",
|
||||
"prompt": "Fetch {{SERVE_URL}}/evidence-187.txt with es_research_fetch and report the key sentence about mechanical gates, quoted verbatim.",
|
||||
"maxSteps": 6,
|
||||
"checks": { "toolsUsed": ["es_research_fetch"], "contains": ["Mechanical gates catch regressions before review"] },
|
||||
"id": "evidence-187"
|
||||
}
|
||||
@@ -0,0 +1 @@
|
||||
{"agent":"build","checks":{"contains":["HOOK:PreToolUse","HOOK:PostToolUse","HOOK:PostToolUseFailure","HOOK:ShellPreExec"],"toolsUsed":["read","bash"]},"description":"Hook bridge E2E: real opencode + .claude/settings.json from a real Claude Code project; each ENFORCED hook event fires a marker. Markers asserted via transcript; on-disk completeness needs kept-workspace review until a file-asserting fire kind exists.","files":{".claude/settings.json":"{\"hooks\":{\"PermissionRequest\":[{\"hooks\":[{\"command\":\"printf 'HOOK:PermissionRequest\\\\n' >> .factory/hooks.log\",\"type\":\"command\"}],\"matcher\":\"Bash\"}],\"PostToolUse\":[{\"hooks\":[{\"command\":\"printf 'HOOK:PostToolUse\\\\n' >> .factory/hooks.log\",\"type\":\"command\"}],\"matcher\":\"\"}],\"PostToolUseFailure\":[{\"hooks\":[{\"command\":\"printf 'HOOK:PostToolUseFailure\\\\n' >> .factory/hooks.log\",\"type\":\"command\"}],\"matcher\":\"\"}],\"PreToolUse\":[{\"hooks\":[{\"command\":\"exit 2\",\"type\":\"command\"}],\"matcher\":\"Bash(rm *)\"},{\"hooks\":[{\"command\":\"printf 'HOOK:PreToolUse\\\\n' >> .factory/hooks.log\",\"type\":\"command\"}],\"matcher\":\"\"}],\"SessionStart\":[{\"hooks\":[{\"command\":\"printf 'HOOK:SessionStart\\\\n' >> .factory/hooks.log\",\"type\":\"command\"}],\"matcher\":\"startup\"}],\"UserPromptSubmit\":[{\"hooks\":[{\"command\":\"printf 'HOOK:UserPromptSubmit\\\\n' >> .factory/hooks.log\",\"type\":\"command\"}],\"matcher\":\"\"}],\"x-es.ShellPreExec\":[{\"hooks\":[{\"command\":\"echo rewritten-by-hook\",\"type\":\"command\"}],\"matcher\":\"echo original\"}]}}","src/fail.sh":"exit 1\n","src/probe.txt":"hook bridge probe\n"},"id":"hooks-bridge","maxSteps":14,"prompt":"Follow these steps exactly, one tool call per step. (0) Run `mkdir -p .factory` with bash. (1) Read src/probe.txt with the read tool. (2) Run `echo hook-probe-ok` with bash. (3) Run `sh src/fail.sh` with bash (it exits 1; continue regardless). (4) Run `echo original` with bash (it will be rewritten; continue). (5) Run `cat .factory/hooks.log` with bash. (6) Narrate which HOOK: markers you observed. This run is headless: never call the question tool."}
|
||||
@@ -0,0 +1 @@
|
||||
{"agent":"build","checks":{"contains":["not assignable","src/bad.ts"],"toolsUsed":["lsp_diagnostics"]},"description":"The agent introduces a type error in src/bad.ts; the typescript language server reports it via lsp_diagnostics. Requires typescript-language-server on PATH (workflow prerequisite, pinned to the registry).","files":{"src/bad.ts":"export const x: number = 1;\n","tsconfig.json":"{\n \"compilerOptions\": { \"strict\": true, \"target\": \"es2022\", \"module\": \"nodenext\" },\n \"include\": [\"src\"]\n}\n"},"id":"lsp-diagnostics","maxSteps":8,"prompt":"In src/bad.ts, change the initializer of x to the string \"not a number\" (keep the number type annotation), using the edit tool. Then call lsp_diagnostics on src/bad.ts and reply quoting the full diagnostic line for the error. This run is headless: never call the question tool. Make independent tool calls in the same block."}
|
||||
@@ -0,0 +1,8 @@
|
||||
{
|
||||
"id": "registry-hidden-spawn",
|
||||
"description": "The factory delegates a status check to the hidden es-qa-functional seat by id (hidden seats are unlisted from the subagent menu but invocable from an allowed parent), then replies DELEGATED.",
|
||||
"agent": "factory",
|
||||
"prompt": "Delegate a status check to the es-qa-functional seat with the subagent tool (description \"status check\", prompt \"Check the factory run status with es_status and report back in one sentence\"). When it returns, reply with exactly the single word DELEGATED and nothing else.",
|
||||
"maxSteps": 8,
|
||||
"checks": { "toolsUsed": ["subagent"], "contains": ["DELEGATED"] }
|
||||
}
|
||||
@@ -0,0 +1,3 @@
|
||||
The evidence cache seals every source with an engine key. Mechanical gates
|
||||
catch regressions before review, which keeps autonomous agents honest.
|
||||
Quote the gate sentence verbatim; a paraphrase is not evidence.
|
||||
@@ -520,25 +520,30 @@ export function runHeadless(options: HeadlessOptions): AsyncGenerator<HeadlessEv
|
||||
agent: "factory",
|
||||
prompt: async (state, factory) =>
|
||||
`${await factory.summary(state)}\n${headline(state, await readLiveness(options.root, state))}\nContinue the factory run from its current stage. This is a headless run: no human will answer questions.`,
|
||||
finished: async (state) => {
|
||||
if (state.stage === "DONE")
|
||||
return { type: "done", ...(state.release?.prUrl ? { prUrl: state.release.prUrl } : {}) }
|
||||
const pending = await pendingApprovals(options.root)
|
||||
if (pending.length)
|
||||
return {
|
||||
type: "waiting",
|
||||
reason: "a human must approve (es approve <stage>)",
|
||||
stages: pending.map((item) => item.stage),
|
||||
}
|
||||
if (state.stage === "GRILL")
|
||||
return { type: "waiting", reason: "the grill needs a human; run /grill interactively" }
|
||||
return undefined
|
||||
},
|
||||
finished: (state) => headlessFinished(options.root, state),
|
||||
// Research artifacts and seat outcomes count as progress, not just the stage machine.
|
||||
progress: async (state) => progressPrint(state, await readLiveness(options.root, state)),
|
||||
})
|
||||
}
|
||||
|
||||
/**
|
||||
* One headless end condition (#247.1): DONE carries the release PR url when
|
||||
* the run released one, a pending approval waits on a human, and the grill
|
||||
* needs a human. Anything else keeps driving.
|
||||
*/
|
||||
export async function headlessFinished(root: string, state: FactoryState): Promise<HeadlessEvent | undefined> {
|
||||
if (state.stage === "DONE") return { type: "done", ...(state.release?.prUrl ? { prUrl: state.release.prUrl } : {}) }
|
||||
const pending = await pendingApprovals(root)
|
||||
if (pending.length)
|
||||
return {
|
||||
type: "waiting",
|
||||
reason: "a human must approve (es approve <stage>)",
|
||||
stages: pending.map((item) => item.stage),
|
||||
}
|
||||
if (state.stage === "GRILL") return { type: "waiting", reason: "the grill needs a human; run /grill interactively" }
|
||||
return undefined
|
||||
}
|
||||
|
||||
export const LOG_LEVELS = ["quiet", "info", "debug"] as const
|
||||
export type LogLevel = (typeof LOG_LEVELS)[number]
|
||||
|
||||
|
||||
+27
-9
@@ -2,11 +2,14 @@
|
||||
// harness-neutral factory operations to harnesses without a native plugin and
|
||||
// to headless runs. It has no approve, trust, waive or resume tool: those are
|
||||
// human-only. Caller identity is PINNED, never piped: the harness adapter sets
|
||||
// one agent id per server in its environment (ES_MCP_AGENT, else the legacy
|
||||
// ES_AGENT/defaultAgent, both written by the human in the adapter config), and
|
||||
// the per-call `agent` argument is ignored. Without an adapter identity the
|
||||
// caller is "mcp", which maps to no seat, so every seat-checked tool refuses.
|
||||
import { stateDir as defaultStateDir, esTools, Factory, gateRunner } from "@heretek-ai/es-core"
|
||||
// one agent id per server in its environment (`ES_MCP_AGENT`, else the legacy
|
||||
// `ES_AGENT`, else the `defaultAgent` option — all written by the human in the
|
||||
// adapter config), and the per-call `agent` argument is ignored. Without an
|
||||
// adapter identity the caller is "mcp", which maps to no seat, so every
|
||||
// seat-checked tool refuses. Only a registry agent id (`factory`,
|
||||
// `es-manager`, `es-programmer`, …) holds a seat: any other pinned string is
|
||||
// still the caller, but a seat-less one, refused the same way as "mcp".
|
||||
import { checkStopFile, stateDir as defaultStateDir, esTools, Factory, gateRunner } from "@heretek-ai/es-core"
|
||||
import { VERSION } from "./version.ts"
|
||||
|
||||
export const PROTOCOL_VERSION = "2025-06-18"
|
||||
@@ -26,7 +29,9 @@ export function createMcpServer(options: McpOptions) {
|
||||
const env = options.env ?? process.env
|
||||
// Pinned at construction: the adapter's environment, never the caller's
|
||||
// arguments. An agent piping JSON-RPC into this server cannot claim a seat.
|
||||
const caller = env.ES_MCP_AGENT || options.defaultAgent || "mcp"
|
||||
// Library consumers get the same order as the CLI wrapper: the adapter's
|
||||
// ES_MCP_AGENT wins, then the legacy ES_AGENT, then the defaultAgent option.
|
||||
const caller = env.ES_MCP_AGENT || env.ES_AGENT || options.defaultAgent || "mcp"
|
||||
const factory = new Factory(options.root, {
|
||||
gates: gateRunner(options.stateDir ? { stateDir: options.stateDir } : {}),
|
||||
...(options.stateDir ? { stateDir: options.stateDir } : {}),
|
||||
@@ -35,7 +40,7 @@ export function createMcpServer(options: McpOptions) {
|
||||
const byName = new Map(tools.map((tool) => [tool.name.replace(/^es_/, ""), tool]))
|
||||
const listed = [...byName].map(([name, tool]) => ({
|
||||
name,
|
||||
description: `${tool.description}${name === "status" || name === "spec_validate" ? "" : " Caller identity is pinned by the adapter (ES_MCP_AGENT); a per-call `agent` argument is accepted but ignored."}`,
|
||||
description: `${tool.description}${name === "status" || name === "spec_validate" ? "" : " Caller identity is pinned by the adapter (ES_MCP_AGENT, else ES_AGENT, else the default agent); a per-call `agent` argument is accepted but ignored."}`,
|
||||
inputSchema: {
|
||||
...tool.input,
|
||||
properties: {
|
||||
@@ -43,7 +48,7 @@ export function createMcpServer(options: McpOptions) {
|
||||
agent: {
|
||||
type: "string",
|
||||
description:
|
||||
"Ignored: the caller is the adapter's pinned ES_MCP_AGENT (or the default agent), never this argument.",
|
||||
"Ignored: the caller is the adapter's pinned ES_MCP_AGENT (else ES_AGENT, else the default agent), never this argument.",
|
||||
},
|
||||
},
|
||||
},
|
||||
@@ -61,7 +66,7 @@ export function createMcpServer(options: McpOptions) {
|
||||
capabilities: { tools: { listChanged: false } },
|
||||
serverInfo: { name: "epistemic-swarm", version: options.version ?? VERSION },
|
||||
instructions:
|
||||
"Epistemic Swarm factory operations. Approvals, trust, waivers and resume are human-only (TUI or `es` CLI) and have no MCP tool. Caller identity is pinned: the adapter sets ES_MCP_AGENT in this server's environment (one server per agent); without it the caller is `mcp`, which holds no seat, so seat-checked tools refuse. A per-call `agent` argument is ignored.",
|
||||
"Epistemic Swarm factory operations. Approvals, trust, waivers and resume are human-only (TUI or `es` CLI) and have no MCP tool. Caller identity is pinned: the adapter sets ES_MCP_AGENT in this server's environment (one server per agent), else the legacy ES_AGENT, else the default agent; without any of them the caller is `mcp`, which holds no seat, so seat-checked tools refuse. A per-call `agent` argument is ignored.",
|
||||
})
|
||||
case "ping":
|
||||
return reply({})
|
||||
@@ -72,6 +77,19 @@ export function createMcpServer(options: McpOptions) {
|
||||
if (!tool) return fail(-32602, `Unknown tool: ${params?.name}`)
|
||||
const { agent: _ignored, ...input } = (params?.arguments ?? {}) as Json
|
||||
try {
|
||||
// STOP and HALTED hold on the MCP surface too (#227): the
|
||||
// kill-switch refuses every call; a halted run refuses everything
|
||||
// but status, mirroring the OpenCode host policy.
|
||||
const stop = await checkStopFile(options.root)
|
||||
if (stop.stopped)
|
||||
throw new Error(
|
||||
`The factory is stopped (.factory/STOP: ${stop.reason}). A human must remove it and resume.`,
|
||||
)
|
||||
if (tool.name !== "es_status") {
|
||||
const state = await factory.read().catch(() => undefined)
|
||||
if (state?.stage === "HALTED")
|
||||
throw new Error(`The factory is halted: ${state.halt?.reason}. Stop and report; a human must resume.`)
|
||||
}
|
||||
const text = await tool.execute(input, { agent: caller })
|
||||
return reply({ content: [{ type: "text", text }] })
|
||||
} catch (error) {
|
||||
|
||||
@@ -1,15 +1,17 @@
|
||||
import { afterEach, beforeEach, describe, expect, test } from "bun:test"
|
||||
import { mkdir, mkdtemp, readFile, rm, writeFile } from "node:fs/promises"
|
||||
import { mkdir, mkdtemp, readdir, readFile, rm, stat, writeFile } from "node:fs/promises"
|
||||
import { tmpdir } from "node:os"
|
||||
import path from "node:path"
|
||||
import {
|
||||
exists,
|
||||
Factory,
|
||||
type FactoryState,
|
||||
factoryLayout,
|
||||
gateRunner,
|
||||
HUMAN_VERBS,
|
||||
loadPrompt,
|
||||
parseArgs,
|
||||
type RunIndexEntry,
|
||||
readApproval,
|
||||
recentAuditEntries,
|
||||
researchSourcesDir,
|
||||
@@ -22,6 +24,7 @@ import { type HarnessDriver, type HeadlessEvent, presentEvent, runHeadless } fro
|
||||
import { openProposalsPr } from "../src/improve.ts"
|
||||
import { commandHelp, main } from "../src/main.ts"
|
||||
import { createMcpServer } from "../src/mcp.ts"
|
||||
import { otherRunHint, runsCommand, statusCommand } from "../src/status.ts"
|
||||
import { type ConfirmIO, confirmHuman, NotInteractive } from "../src/tty.ts"
|
||||
import { VERSION } from "../src/version.ts"
|
||||
|
||||
@@ -330,6 +333,147 @@ describe("human-only confirmation", () => {
|
||||
})
|
||||
})
|
||||
|
||||
describe("per-verb pipe refusal (#221.2)", () => {
|
||||
// approve, rebaseline and reseal already have dedicated non-TTY pipe
|
||||
// tests above. Every other verb in the confirmHuman funnel refuses the
|
||||
// same way: exit 3 comes only from the NotInteractive catch in main, so
|
||||
// asserting it proves the test reached the terminal gate — and each test
|
||||
// also proves nothing was written. Each argv below is shaped to pass
|
||||
// every check before the confirmation (usage → 2 and early-returns → 0/1
|
||||
// would fail the assertion instead of passing vacuously).
|
||||
test("es trust refuses without a terminal and trusts nothing", async () => {
|
||||
await mkdir(path.dirname(factoryLayout(root).gates), { recursive: true })
|
||||
await writeFile(
|
||||
factoryLayout(root).gates,
|
||||
JSON.stringify({ version: 1, commands: [{ id: "probe", kind: "test", command: "touch PROBE" }] }),
|
||||
)
|
||||
expect((await run(["trust"])).code).toBe(3)
|
||||
expect(await exists(path.join(state, "trust.json"))).toBe(false)
|
||||
})
|
||||
|
||||
test("es waive refuses without a terminal and records nothing", async () => {
|
||||
const result = await run(["waive", "lint/*", "--reason", "legacy code under migration"])
|
||||
expect(result.code).toBe(3)
|
||||
expect(await exists(factoryLayout(root).waivers)).toBe(false)
|
||||
})
|
||||
|
||||
test("es factory resume refuses without a terminal and stays halted", async () => {
|
||||
const halted = new Factory(root, { gates: gateRunner({ stateDir: state }), stateDir: state })
|
||||
await halted.begin("human:tester")
|
||||
await halted.halt("system", "test halt")
|
||||
expect((await run(["factory", "resume"])).code).toBe(3)
|
||||
expect((await halted.read())?.stage).toBe("HALTED")
|
||||
})
|
||||
|
||||
test("es factory pr refuses without a terminal and records nothing", async () => {
|
||||
const result = await run(["factory", "pr", "https://github.com/example/repo/pull/1"])
|
||||
expect(result.code).toBe(3)
|
||||
expect(await exists(factoryLayout(root).config)).toBe(false)
|
||||
})
|
||||
|
||||
test("es gates install-git refuses without a terminal and installs nothing", async () => {
|
||||
const result = await run(["gates", "install-git"])
|
||||
expect(result.code).toBe(3)
|
||||
expect(await exists(path.join(root, ".factory", "git-hooks"))).toBe(false)
|
||||
})
|
||||
|
||||
test("es lsp install refuses without a terminal before consent or install", async () => {
|
||||
const result = await run(["lsp", "install", "typescript"])
|
||||
expect(result.code).toBe(3)
|
||||
expect(await exists(path.join(state, "lsp-consent.json"))).toBe(false)
|
||||
})
|
||||
|
||||
test("es audit open refuses without a terminal and provisions nothing", async () => {
|
||||
const result = await run(["audit", "src", "--max-usd", "5", "--open-only"])
|
||||
expect(result.code).toBe(3)
|
||||
expect(await exists(factoryLayout(root).dir)).toBe(false)
|
||||
})
|
||||
})
|
||||
|
||||
describe("status and runs are read-only; other-run hints (#248)", () => {
|
||||
// File list + sizes + content hashes under .factory and the state dir.
|
||||
// Directory mtimes are excluded on purpose: the state lock legitimately
|
||||
// mkdirs+removes its lock dir on reads, churning the parent mtime.
|
||||
const snapshot = async () => {
|
||||
const entries: string[] = []
|
||||
for (const [label, dir] of [
|
||||
["root", path.join(root, ".factory")],
|
||||
["state", state],
|
||||
] as const) {
|
||||
const walk = async (rel: string): Promise<void> => {
|
||||
const names = await readdir(path.join(dir, rel)).catch(() => [] as string[])
|
||||
for (const name of names) {
|
||||
const child = rel ? `${rel}/${name}` : name
|
||||
const info = await stat(path.join(dir, child))
|
||||
if (info.isDirectory()) await walk(child)
|
||||
else if (info.isFile())
|
||||
entries.push(`${label}/${child}:${info.size}:${Bun.hash(await readFile(path.join(dir, child)))}`)
|
||||
}
|
||||
}
|
||||
await walk("")
|
||||
}
|
||||
return entries.sort()
|
||||
}
|
||||
|
||||
test("status and runs (text, --json, --all) write nothing", async () => {
|
||||
const factory = new Factory(root, { gates: gateRunner({ stateDir: state }), stateDir: state })
|
||||
const begun = await factory.begin("human:tester")
|
||||
const printed: string[] = []
|
||||
const context = { root, stateDir: state, print: (text: string) => printed.push(text) }
|
||||
const before = await snapshot()
|
||||
expect(await statusCommand(context, { positionals: ["status"], flags: {} })).toBe(0)
|
||||
expect(await statusCommand(context, { positionals: ["status"], flags: { json: true } })).toBe(0)
|
||||
expect(await runsCommand(context, { positionals: ["runs"], flags: {} })).toBe(0)
|
||||
expect(await runsCommand(context, { positionals: ["runs"], flags: { all: true, json: true } })).toBe(0)
|
||||
// The commands really ran (not an early return): the GRILL headline and
|
||||
// the begun run are in the output.
|
||||
expect(printed.join("\n")).toContain("grill")
|
||||
expect(printed.join("\n")).toContain(begun.runId)
|
||||
expect(await snapshot()).toEqual(before)
|
||||
})
|
||||
|
||||
test("otherRunHint: window boundary, DONE and equal-timestamp edges", async () => {
|
||||
const NOW = new Date("2026-10-08T02:10:00.000Z")
|
||||
const iso = (ms: number) => new Date(NOW.getTime() + ms).toISOString()
|
||||
const entry = (overrides: Partial<RunIndexEntry> = {}): RunIndexEntry => ({
|
||||
runId: "run-other",
|
||||
root: "/elsewhere",
|
||||
stage: "BUILD",
|
||||
createdAt: iso(-3_600_000),
|
||||
updatedAt: iso(-60_000),
|
||||
...overrides,
|
||||
})
|
||||
const here = (updatedAt: string): FactoryState => ({
|
||||
version: 2,
|
||||
runId: "run-here",
|
||||
stage: "BUILD",
|
||||
createdAt: iso(-3_600_000),
|
||||
updatedAt,
|
||||
spend: { usd: 1, estimated: true, events: 1 },
|
||||
phases: [],
|
||||
audits: [],
|
||||
})
|
||||
const hint = (runs: RunIndexEntry[], updatedAt: string | undefined) =>
|
||||
otherRunHint(runs, { root, state: updatedAt === undefined ? undefined : here(updatedAt) }, NOW)
|
||||
|
||||
// A more recently active run elsewhere is named; without a local run the
|
||||
// copy points at it directly.
|
||||
expect(hint([entry()], iso(-120_000))).toContain("run-other")
|
||||
expect(hint([entry()], undefined)).toContain("A run is active elsewhere")
|
||||
// The window is ten minutes: just inside counts, exactly at it and
|
||||
// outside do not (OTHER_RUN_WINDOW_MS is pinned behaviorally here).
|
||||
expect(hint([entry({ updatedAt: iso(-599_000) })], iso(-3_600_000))).toContain("run-other")
|
||||
expect(hint([entry({ updatedAt: iso(-600_000) })], iso(-3_600_000))).toBeUndefined()
|
||||
expect(hint([entry({ updatedAt: iso(-601_000) })], iso(-3_600_000))).toBeUndefined()
|
||||
// DONE runs never hint, however fresh; an equally-fresh run is not
|
||||
// "more recently active"; our own root is skipped; a stale local run is.
|
||||
expect(hint([entry({ stage: "DONE" })], iso(-120_000))).toBeUndefined()
|
||||
expect(hint([entry({ updatedAt: iso(-120_000) })], iso(-120_000))).toBeUndefined()
|
||||
expect(hint([entry({ root })], iso(-120_000))).toBeUndefined()
|
||||
expect(hint([entry()], iso(-30_000))).toBeUndefined()
|
||||
})
|
||||
})
|
||||
|
||||
describe("es key", () => {
|
||||
test("seal asks twice, refuses short or mismatched passphrases, and status shows the fingerprint", async () => {
|
||||
const { humanKeyId } = await import("@heretek-ai/es-core")
|
||||
@@ -355,6 +499,9 @@ describe("es key", () => {
|
||||
expect((await runIn(["key", "seal"], seq(["short", "short"]))).code).toBe(1)
|
||||
expect((await runIn(["key", "seal"], seq(["long-enough-pass", "different-pass"]))).code).toBe(1)
|
||||
expect(await humanKeyId(fresh)).toBeUndefined()
|
||||
// A pipe cannot seal either: keySeal throws NotInteractive (exit 3) before reading.
|
||||
expect((await runIn(["key", "seal"], pipe)).code).toBe(3)
|
||||
expect(await humanKeyId(fresh)).toBeUndefined()
|
||||
expect((await runIn(["key", "seal"], seq(["a-long-test-passphrase", "a-long-test-passphrase"]))).code).toBe(0)
|
||||
expect(await humanKeyId(fresh)).toBeDefined()
|
||||
expect((await runIn(["key", "status"], seq([]))).out).toContain("Human key")
|
||||
@@ -1351,7 +1498,20 @@ describe("es audit and es scout (headless jobs)", () => {
|
||||
)
|
||||
await tool("es_scout_scan").execute({ candidate: "tiny" }, scout)
|
||||
await tool("es_scout_record").execute(
|
||||
{ candidate: "tiny", maintenance: "slow", proposal: "adopt", rationale: "small enough to keep", claims: [] },
|
||||
{
|
||||
candidate: "tiny",
|
||||
maintenance: "slow",
|
||||
proposal: "adopt",
|
||||
rationale: "small enough to keep",
|
||||
claims: [
|
||||
{
|
||||
tag: "NEGATIVE_KNOWLEDGE",
|
||||
statement: "OSV advisories were not queried for tiny",
|
||||
query: "OSV advisories for tiny",
|
||||
finding: "skipped: a local fixture with no published package",
|
||||
},
|
||||
],
|
||||
},
|
||||
scout,
|
||||
)
|
||||
await tool("es_scout_complete").execute({}, scout)
|
||||
|
||||
@@ -0,0 +1,69 @@
|
||||
// Audit #203 — `es config set` is human-only: without a terminal it refuses
|
||||
// before writing. These are pipe runs asserting refusal codes through the
|
||||
// CLI's own seam (main with a non-interactive ConfirmIO); the human key is
|
||||
// never unlocked and nothing is ever written.
|
||||
import { afterEach, beforeEach, describe, expect, test } from "bun:test"
|
||||
import { mkdtemp, rm } from "node:fs/promises"
|
||||
import { tmpdir } from "node:os"
|
||||
import path from "node:path"
|
||||
import { exists, factoryLayout } from "@heretek-ai/es-core"
|
||||
import { main } from "../src/main.ts"
|
||||
import type { ConfirmIO } from "../src/tty.ts"
|
||||
|
||||
let root: string
|
||||
let state: string
|
||||
const realXdgConfig = process.env.XDG_CONFIG_HOME
|
||||
|
||||
beforeEach(async () => {
|
||||
root = await mkdtemp(path.join(tmpdir(), "es-203-config-"))
|
||||
state = await mkdtemp(path.join(tmpdir(), "es-203-config-state-"))
|
||||
process.env.XDG_CONFIG_HOME = path.join(state, "xdg-config")
|
||||
})
|
||||
afterEach(async () => {
|
||||
if (realXdgConfig === undefined) delete process.env.XDG_CONFIG_HOME
|
||||
else process.env.XDG_CONFIG_HOME = realXdgConfig
|
||||
await rm(root, { recursive: true, force: true })
|
||||
await rm(state, { recursive: true, force: true })
|
||||
})
|
||||
|
||||
const pipe: ConfirmIO = { interactive: false, write: () => {}, readLine: async () => "y", readSecret: async () => "" }
|
||||
|
||||
async function run(argv: string[]) {
|
||||
const printed: string[] = []
|
||||
const code = await main(argv, { print: (text) => printed.push(text), confirm: pipe, cwd: root, stateDir: state })
|
||||
return { code, out: printed.join("\n") }
|
||||
}
|
||||
|
||||
describe("es config set refuses without a terminal (#203)", () => {
|
||||
test("a piped project-layer set refuses with code 3 and writes nothing", async () => {
|
||||
const result = await run(["config", "set", "research.depth", "3"])
|
||||
expect(result.code).toBe(3)
|
||||
expect(result.out).toContain("interactive terminal")
|
||||
expect(await exists(factoryLayout(root).config)).toBe(false)
|
||||
})
|
||||
|
||||
test("a piped global-layer set refuses with code 3 and writes nothing", async () => {
|
||||
const result = await run(["config", "set", "research.depth", "3", "--global"])
|
||||
expect(result.code).toBe(3)
|
||||
expect(result.out).toContain("interactive terminal")
|
||||
expect(await exists(factoryLayout(root).config)).toBe(false)
|
||||
expect(await exists(path.join(state, "xdg-config", "epistemic-swarm", "config.json"))).toBe(false)
|
||||
})
|
||||
|
||||
test("refusal codes contrast: schema refusal (1) precedes the TTY gate (3); usage is (2)", async () => {
|
||||
const typo = await run(["config", "set", "reserch.depth", "3"])
|
||||
expect(typo.code).toBe(1)
|
||||
expect(typo.out).toContain("Refused")
|
||||
const bad = await run(["config", "set", "research.depth", "9"])
|
||||
expect(bad.code).toBe(1)
|
||||
const usage = await run(["config", "set", "research.depth"])
|
||||
expect(usage.code).toBe(2)
|
||||
expect(await exists(factoryLayout(root).config)).toBe(false)
|
||||
})
|
||||
|
||||
test("--help stays a read: usage, code 0, nothing written", async () => {
|
||||
const helped = await run(["config", "set", "research.depth", "3", "--help"])
|
||||
expect(helped.code).toBe(0)
|
||||
expect(await exists(factoryLayout(root).config)).toBe(false)
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,467 @@
|
||||
// Headless factory runs (#200 §2): integration tests against the real
|
||||
// in-process OpenCode v2 host (packages/testkit), plus the cheap proof gaps
|
||||
// from bug #247. Spend-bearing seams are never touched: no `factory run
|
||||
// --headless` launch, no real signals, no real harness binary. Everything
|
||||
// here drives `runHeadless`/`driveHeadless` with fake in-process drivers
|
||||
// (one of them forwarding prompts to the testkit host), scripted
|
||||
// AbortControllers, pipe CLI runs that assert refusal codes, and a stub
|
||||
// `opencode` binary for argv capture.
|
||||
import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, test } from "bun:test"
|
||||
import { chmod, mkdir, mkdtemp, readFile, rm, writeFile } from "node:fs/promises"
|
||||
import { tmpdir } from "node:os"
|
||||
import path from "node:path"
|
||||
import {
|
||||
exists,
|
||||
Factory,
|
||||
factoryLayout,
|
||||
gateRunner,
|
||||
readApproval,
|
||||
researchSourcesDir,
|
||||
SourceCache,
|
||||
sealHumanKey,
|
||||
stringifyFrontmatter,
|
||||
} from "@heretek-ai/es-core"
|
||||
import type { Harness } from "../../testkit/src/host.ts"
|
||||
import { boot } from "../../testkit/src/host.ts"
|
||||
import { directiveScript } from "../../testkit/src/script.ts"
|
||||
import {
|
||||
DRIVERS,
|
||||
type DriverTurn,
|
||||
driveHeadless,
|
||||
type HarnessDriver,
|
||||
type HeadlessEvent,
|
||||
headlessExitCode,
|
||||
opencodeDriver,
|
||||
runHeadless,
|
||||
} from "../src/headless.ts"
|
||||
import { main } from "../src/main.ts"
|
||||
import type { ConfirmIO } from "../src/tty.ts"
|
||||
|
||||
const PASSPHRASE = "test-passphrase-1234"
|
||||
|
||||
let root: string
|
||||
let state: string
|
||||
const realXdgConfig = process.env.XDG_CONFIG_HOME
|
||||
const git = async (...args: string[]) => {
|
||||
const proc = Bun.spawn(["git", ...args], { cwd: root, stdout: "pipe", stderr: "pipe" })
|
||||
await proc.exited
|
||||
}
|
||||
|
||||
beforeEach(async () => {
|
||||
root = await mkdtemp(path.join(tmpdir(), "es-200-"))
|
||||
state = await mkdtemp(path.join(tmpdir(), "es-200-state-"))
|
||||
process.env.XDG_CONFIG_HOME = path.join(state, "xdg-config")
|
||||
await git("init", "-q", "-b", "main")
|
||||
await writeFile(path.join(root, "README.md"), "# x\n")
|
||||
await git("add", "-A")
|
||||
await git("-c", "user.name=t", "-c", "user.email=t@t", "commit", "-q", "-m", "init")
|
||||
await sealHumanKey(PASSPHRASE, state)
|
||||
})
|
||||
afterEach(async () => {
|
||||
if (realXdgConfig === undefined) delete process.env.XDG_CONFIG_HOME
|
||||
else process.env.XDG_CONFIG_HOME = realXdgConfig
|
||||
await rm(root, { recursive: true, force: true })
|
||||
await rm(state, { recursive: true, force: true })
|
||||
})
|
||||
|
||||
/** A human at a terminal who types the passphrase with echo off. */
|
||||
const human = (answer: string = PASSPHRASE): ConfirmIO & { output: string[] } => {
|
||||
const output: string[] = []
|
||||
return {
|
||||
interactive: true,
|
||||
output,
|
||||
write: (text) => output.push(text),
|
||||
readLine: async () => "",
|
||||
readSecret: async () => answer,
|
||||
}
|
||||
}
|
||||
const pipe: ConfirmIO = { interactive: false, write: () => {}, readLine: async () => "y", readSecret: async () => "" }
|
||||
|
||||
async function runCli(argv: string[], confirm: ConfirmIO = pipe) {
|
||||
const printed: string[] = []
|
||||
const code = await main(argv, { print: (text) => printed.push(text), confirm, cwd: root, stateDir: state })
|
||||
return { code, out: printed.join("\n") }
|
||||
}
|
||||
|
||||
/** A settled grill, approved at a terminal: the run is in RESEARCH. */
|
||||
async function researched() {
|
||||
const factory = new Factory(root, { gates: gateRunner({ stateDir: state }), stateDir: state })
|
||||
await factory.writeFrontier("grill", {
|
||||
version: "1.1",
|
||||
idea: "cli test",
|
||||
spendCeiling: { currency: "USD", maxAmount: 5 },
|
||||
settled: true,
|
||||
nodes: [{ id: "a", question: "q?", answer: "yes", status: "settled" }],
|
||||
})
|
||||
const approved = await runCli(["approve", "frontier"], human())
|
||||
expect(approved.code).toBe(0)
|
||||
expect((await factory.read())?.stage).toBe("RESEARCH")
|
||||
return factory
|
||||
}
|
||||
|
||||
describe("headless against the testkit host (#200 §2)", () => {
|
||||
let host: Harness | undefined
|
||||
|
||||
beforeAll(async () => {
|
||||
host = await boot({ script: directiveScript })
|
||||
}, 60_000)
|
||||
afterAll(async () => {
|
||||
await host?.close()
|
||||
host = undefined
|
||||
})
|
||||
|
||||
test("turns reach the real host; a pending approval stops the run without granting anything", async () => {
|
||||
const factory = await researched()
|
||||
const harness = host!
|
||||
const llmBefore = harness.llm.requests.length
|
||||
const calls: DriverTurn[] = []
|
||||
// The opencode driver seam: one turn carries the seat identity, the
|
||||
// prompt and the argv the harness CLI would get; here the harness is
|
||||
// the in-process testkit host instead of a spawned `opencode run`.
|
||||
const driver: HarnessDriver = {
|
||||
id: "fake-host",
|
||||
available: async () => true,
|
||||
async *turn(input: DriverTurn) {
|
||||
calls.push(input)
|
||||
const result = await harness.run(input.prompt)
|
||||
yield { type: "text", part: { text: `host settled ${result.context.length} messages` } }
|
||||
return result.sessionID
|
||||
},
|
||||
}
|
||||
// Phase 1: no pending approval, so the turn goes through the host.
|
||||
const first: HeadlessEvent[] = []
|
||||
for await (const event of runHeadless({ root, driver, maxTurns: 1, stallTurns: 99, stateDir: state }))
|
||||
first.push(event)
|
||||
expect(calls).toHaveLength(1)
|
||||
expect(calls[0]?.agent).toBe("factory")
|
||||
expect(calls[0]?.prompt).toContain("headless run")
|
||||
expect(calls[0]?.root).toBe(root)
|
||||
expect(harness.llm.requests.length).toBeGreaterThan(llmBefore)
|
||||
expect(first.at(-1)).toEqual({ type: "turn-cap", turns: 1 })
|
||||
// Phase 2: a pending approval stops the run before any turn is driven.
|
||||
const pending = factoryLayout(root).pending
|
||||
await mkdir(path.dirname(pending), { recursive: true })
|
||||
await writeFile(pending, JSON.stringify([{ stage: "spec", requestedBy: "factory", at: "2026-10-10T00:00:00Z" }]))
|
||||
const pendingBefore = await readFile(pending, "utf8")
|
||||
const stageBefore = (await factory.read())?.stage
|
||||
const callsBefore = calls.length
|
||||
const llmMid = harness.llm.requests.length
|
||||
const second: HeadlessEvent[] = []
|
||||
for await (const event of runHeadless({ root, driver, maxTurns: 5, stallTurns: 99, stateDir: state }))
|
||||
second.push(event)
|
||||
expect(second.at(-1)).toEqual({
|
||||
type: "waiting",
|
||||
reason: "a human must approve (es approve <stage>)",
|
||||
stages: ["spec"],
|
||||
})
|
||||
// Nothing was granted: no new host turn, the pending file is byte
|
||||
// identical, the stage is unchanged, and no approval was recorded.
|
||||
expect(calls.length).toBe(callsBefore)
|
||||
expect(harness.llm.requests.length).toBe(llmMid)
|
||||
expect(await readFile(pending, "utf8")).toBe(pendingBefore)
|
||||
expect((await factory.read())?.stage).toBe(stageBefore)
|
||||
expect(await readApproval(root, "spec")).toBeUndefined()
|
||||
}, 60_000)
|
||||
})
|
||||
|
||||
describe("headless proof gaps (#247)", () => {
|
||||
test("the driver sees the seat identity, the prompt, the model and the threaded session", async () => {
|
||||
await researched()
|
||||
const calls: DriverTurn[] = []
|
||||
let n = 0
|
||||
const driver: HarnessDriver = {
|
||||
id: "fake",
|
||||
available: async () => true,
|
||||
async *turn(input: DriverTurn) {
|
||||
calls.push(input)
|
||||
n++
|
||||
yield { text: "working" }
|
||||
return n === 1 ? "ses_1" : "ses_2"
|
||||
},
|
||||
}
|
||||
const events: HeadlessEvent[] = []
|
||||
for await (const event of runHeadless({
|
||||
root,
|
||||
driver,
|
||||
maxTurns: 2,
|
||||
stallTurns: 99,
|
||||
stateDir: state,
|
||||
model: "fake/scripted",
|
||||
}))
|
||||
events.push(event)
|
||||
expect(calls).toHaveLength(2)
|
||||
for (const call of calls) {
|
||||
expect(call.agent).toBe("factory")
|
||||
expect(call.prompt).toContain("headless run")
|
||||
expect(call.root).toBe(root)
|
||||
expect(call.model).toBe("fake/scripted")
|
||||
}
|
||||
expect(calls[0]?.session).toBeUndefined()
|
||||
expect(calls[0]?.signal).toBeUndefined()
|
||||
// The first turn's session id is threaded into the second turn.
|
||||
expect(calls[1]?.session).toBe("ses_1")
|
||||
expect(events.at(-1)).toEqual({ type: "turn-cap", turns: 2 })
|
||||
})
|
||||
|
||||
test("the opencode driver argv shape: run --standalone --format json --agent [--model] [--session] prompt", async () => {
|
||||
expect(opencodeDriver.id).toBe("opencode")
|
||||
expect(DRIVERS.opencode).toBe(opencodeDriver)
|
||||
const bin = await mkdtemp(path.join(tmpdir(), "es-argv-opencode-"))
|
||||
const capture = path.join(bin, "argv.jsonl")
|
||||
await writeFile(
|
||||
path.join(bin, "opencode"),
|
||||
`#!/usr/bin/env bun\nimport { appendFileSync } from "node:fs"\nappendFileSync(process.env.ES_ARGV_CAPTURE!, \`\${JSON.stringify(Bun.argv.slice(2))}\\n\`)\nconsole.log(JSON.stringify({ type: "text", sessionID: "ses_argv1", part: { text: "stub" } }))\n`,
|
||||
)
|
||||
await chmod(path.join(bin, "opencode"), 0o755)
|
||||
const previousPath = process.env.PATH
|
||||
const previousCapture = process.env.ES_ARGV_CAPTURE
|
||||
process.env.PATH = `${bin}${path.delimiter}${previousPath}`
|
||||
process.env.ES_ARGV_CAPTURE = capture
|
||||
try {
|
||||
const turnArgv = async (input: DriverTurn) => {
|
||||
await rm(capture, { force: true })
|
||||
const seen: unknown[] = []
|
||||
const turn = opencodeDriver.turn(input)
|
||||
let step = await turn.next()
|
||||
while (!step.done) {
|
||||
seen.push(step.value)
|
||||
step = await turn.next()
|
||||
}
|
||||
const lines = (await readFile(capture, "utf8")).trim().split("\n")
|
||||
return { argv: JSON.parse(lines[0]!) as string[], events: seen, session: step.value }
|
||||
}
|
||||
const minimal = await turnArgv({ root, agent: "factory", prompt: "go" })
|
||||
expect(minimal.argv).toEqual(["run", "--standalone", "--format", "json", "--agent", "factory", "go"])
|
||||
expect(minimal.session).toBe("ses_argv1")
|
||||
expect(minimal.events).toHaveLength(1)
|
||||
const full = await turnArgv({
|
||||
root,
|
||||
agent: "factory",
|
||||
prompt: "go",
|
||||
model: "fake/scripted",
|
||||
session: "ses_prev",
|
||||
})
|
||||
expect(full.argv).toEqual([
|
||||
"run",
|
||||
"--standalone",
|
||||
"--format",
|
||||
"json",
|
||||
"--agent",
|
||||
"factory",
|
||||
"--model",
|
||||
"fake/scripted",
|
||||
"--session",
|
||||
"ses_prev",
|
||||
"go",
|
||||
])
|
||||
// The prompt is always the trailing argument, after any seat flags.
|
||||
expect(full.argv.at(-1)).toBe("go")
|
||||
} finally {
|
||||
process.env.PATH = previousPath
|
||||
if (previousCapture === undefined) delete process.env.ES_ARGV_CAPTURE
|
||||
else process.env.ES_ARGV_CAPTURE = previousCapture
|
||||
await rm(bin, { recursive: true, force: true })
|
||||
}
|
||||
// With no harness binary on PATH the driver reports itself unavailable.
|
||||
const empty = await mkdtemp(path.join(tmpdir(), "es-argv-empty-"))
|
||||
try {
|
||||
process.env.PATH = empty
|
||||
expect(await opencodeDriver.available()).toBe(false)
|
||||
} finally {
|
||||
process.env.PATH = previousPath
|
||||
await rm(empty, { recursive: true, force: true })
|
||||
}
|
||||
})
|
||||
|
||||
test("a scripted driver drives runHeadless to DONE and the prUrl spreads into done", async () => {
|
||||
const green = { passed: true, findings: [], summary: "green" }
|
||||
const stubGates = async () => green
|
||||
const factory = new Factory(root, { gates: stubGates, stateDir: state })
|
||||
await factory.writeFrontier("grill", {
|
||||
version: "1.1",
|
||||
idea: "cli test",
|
||||
spendCeiling: { currency: "USD", maxAmount: 5 },
|
||||
settled: true,
|
||||
nodes: [{ id: "a", question: "q?", answer: "yes", status: "settled" }],
|
||||
})
|
||||
expect((await runCli(["approve", "frontier"], human())).code).toBe(0)
|
||||
// A research report whose single claim cites a cached source verbatim.
|
||||
const cache = new SourceCache(researchSourcesDir(root), state)
|
||||
const source = await cache.put({
|
||||
url: "https://example.test/greeting",
|
||||
text: "Greeting libraries usually expose a single greet function.",
|
||||
provider: "fetch",
|
||||
})
|
||||
await mkdir(path.join(root, ".factory/research"), { recursive: true })
|
||||
await writeFile(
|
||||
path.join(root, ".factory/research/REPORT.md"),
|
||||
`# Research\n\n- A greet function is the conventional API [VERIFIED: sha256:${source.meta.sha256} "expose a single greet function"]\n`,
|
||||
)
|
||||
await factory.completeResearch("factory")
|
||||
await mkdir(path.join(root, ".factory/specs/alpha"), { recursive: true })
|
||||
await writeFile(
|
||||
path.join(root, ".factory/roadmap.json"),
|
||||
JSON.stringify({ version: 1, title: "Greeting", phases: [{ id: "alpha", title: "Phase alpha" }] }),
|
||||
)
|
||||
await writeFile(
|
||||
path.join(root, ".factory/specs/alpha/GOAL.md"),
|
||||
stringifyFrontmatter(
|
||||
{
|
||||
phase: "alpha",
|
||||
title: "Phase alpha",
|
||||
acceptance: [{ kind: "file", id: "impl", description: "implementation exists", path: "src/alpha.ts" }],
|
||||
},
|
||||
"Implement alpha as a vertical slice.\n",
|
||||
),
|
||||
)
|
||||
expect((await runCli(["approve", "spec"], human())).code).toBe(0)
|
||||
await factory.startBuild("factory")
|
||||
const dir = (await factory.activeWorktree())!
|
||||
await mkdir(path.join(dir, "src"), { recursive: true })
|
||||
await mkdir(path.join(dir, "test"), { recursive: true })
|
||||
await writeFile(path.join(dir, "test/alpha.test.ts"), `test("alpha", () => {})\n`)
|
||||
await factory.recordGateRun("es-programmer", {
|
||||
passed: false,
|
||||
findings: [],
|
||||
summary: "red",
|
||||
failedTests: ["test/alpha.test.ts"],
|
||||
})
|
||||
await writeFile(path.join(dir, "src/alpha.ts"), "export const alpha = 1\n")
|
||||
expect((await factory.complete("es-programmer")).ok).toBe(true)
|
||||
await factory.qaVerdict("es-qa-functional", "pass", "ok")
|
||||
expect((await factory.qaVerdict("es-qa-adversarial", "pass", "ok")).stage).toBe("RELEASE")
|
||||
// The scripted turn records the PR (the human fallback): the next loop
|
||||
// sees DONE with a release, and done carries the prUrl.
|
||||
const prUrl = "https://example.test/pr/9"
|
||||
const calls: DriverTurn[] = []
|
||||
const driver: HarnessDriver = {
|
||||
id: "fake",
|
||||
available: async () => true,
|
||||
async *turn(input: DriverTurn) {
|
||||
calls.push(input)
|
||||
yield { text: "recording the pr" }
|
||||
await new Factory(root, { gates: stubGates, stateDir: state }).recordPr("tester", prUrl)
|
||||
return "ses_done"
|
||||
},
|
||||
}
|
||||
const events: HeadlessEvent[] = []
|
||||
for await (const event of runHeadless({ root, driver, maxTurns: 5, stallTurns: 99, stateDir: state }))
|
||||
events.push(event)
|
||||
expect(events[0]).toMatchObject({ type: "start" })
|
||||
expect(calls).toHaveLength(1)
|
||||
expect(calls[0]?.agent).toBe("factory")
|
||||
expect(events.at(-1)).toEqual({ type: "done", prUrl })
|
||||
expect((await factory.read())?.stage).toBe("DONE")
|
||||
})
|
||||
|
||||
test("a STOP file halts at headless level: no turn runs, the stage is untouched", async () => {
|
||||
const factory = await researched()
|
||||
await writeFile(factoryLayout(root).stop, "lunch\n")
|
||||
let turns = 0
|
||||
const driver: HarnessDriver = {
|
||||
id: "fake",
|
||||
available: async () => true,
|
||||
async *turn() {
|
||||
turns++
|
||||
yield { text: "never" }
|
||||
return "ses_fake"
|
||||
},
|
||||
}
|
||||
const events: HeadlessEvent[] = []
|
||||
for await (const event of runHeadless({ root, driver, maxTurns: 3, stallTurns: 99, stateDir: state }))
|
||||
events.push(event)
|
||||
expect(events.at(-1)).toMatchObject({ type: "halted", reason: expect.stringContaining("lunch") })
|
||||
expect(turns).toBe(0)
|
||||
expect((await factory.read())?.stage).toBe("RESEARCH")
|
||||
expect(await exists(factoryLayout(root).stop)).toBe(true)
|
||||
})
|
||||
|
||||
test("an unknown --driver is refused before anything runs", async () => {
|
||||
const empty = await mkdtemp(path.join(tmpdir(), "es-200-empty-"))
|
||||
try {
|
||||
const printed: string[] = []
|
||||
const code = await main(["factory", "run", "--headless", "--driver", "nope"], {
|
||||
print: (text) => printed.push(text),
|
||||
confirm: pipe,
|
||||
cwd: empty,
|
||||
stateDir: state,
|
||||
})
|
||||
const out = printed.join("\n")
|
||||
expect(code).toBe(2)
|
||||
expect(out).toContain("Unknown driver")
|
||||
expect(out).toContain("opencode")
|
||||
expect(await exists(factoryLayout(empty).dir)).toBe(false)
|
||||
} finally {
|
||||
await rm(empty, { recursive: true, force: true })
|
||||
}
|
||||
})
|
||||
|
||||
test("exit codes: cancelled is 130, failures stay 1, the rest 0", async () => {
|
||||
expect(headlessExitCode("cancelled")).toBe(130)
|
||||
for (const type of ["error", "halted", "stalled", "turn-cap"] as const) expect(headlessExitCode(type)).toBe(1)
|
||||
for (const type of ["start", "turn", "driver", "progress", "turn-metrics", "waiting", "done"] as const)
|
||||
expect(headlessExitCode(type)).toBe(0)
|
||||
})
|
||||
|
||||
test("a pre-aborted signal cancels at Abort level: no error, no halt, resumable", async () => {
|
||||
const factory = await researched()
|
||||
const hanging: HarnessDriver = {
|
||||
id: "hang",
|
||||
available: async () => true,
|
||||
async *turn() {
|
||||
await new Promise<never>(() => {})
|
||||
yield { text: "never" }
|
||||
return undefined
|
||||
},
|
||||
}
|
||||
const controller = new AbortController()
|
||||
controller.abort()
|
||||
const events: HeadlessEvent[] = []
|
||||
for await (const event of runHeadless({
|
||||
root,
|
||||
driver: hanging,
|
||||
maxTurns: 5,
|
||||
stateDir: state,
|
||||
signal: controller.signal,
|
||||
progressMs: 60_000,
|
||||
}))
|
||||
events.push(event)
|
||||
// No real signal was sent: the mapping (SIGINT/SIGTERM -> abort ->
|
||||
// `cancelled`, exit 130) is proven at the Abort level, and the live
|
||||
// signal E2E stays human-only (issue #200 §3, NEEDS HUMAN).
|
||||
expect(events.at(-1)).toMatchObject({
|
||||
type: "cancelled",
|
||||
reason: expect.stringContaining("run state is untouched"),
|
||||
})
|
||||
expect(headlessExitCode("cancelled")).toBe(130)
|
||||
expect(events.some((event) => event.type === "error")).toBe(false)
|
||||
expect((await factory.read())?.stage).toBe("RESEARCH")
|
||||
expect(await exists(factoryLayout(root).stop)).toBe(false)
|
||||
})
|
||||
|
||||
test("driveHeadless threads a custom job's done (with prUrl) through unchanged", async () => {
|
||||
const factory = new Factory(root, { gates: gateRunner({ stateDir: state }), stateDir: state })
|
||||
await factory.provisionRun("tester", 5)
|
||||
const prUrl = "https://example.test/pr/7"
|
||||
const driver: HarnessDriver = {
|
||||
id: "fake",
|
||||
available: async () => true,
|
||||
async *turn() {
|
||||
yield { text: "working" }
|
||||
return "ses_fake"
|
||||
},
|
||||
}
|
||||
const events: HeadlessEvent[] = []
|
||||
for await (const event of driveHeadless(
|
||||
{ root, driver, maxTurns: 3, stallTurns: 99, stateDir: state },
|
||||
{
|
||||
agent: "factory",
|
||||
prompt: async () => "go",
|
||||
finished: async () => ({ type: "done", prUrl }),
|
||||
progress: async () => "p",
|
||||
},
|
||||
))
|
||||
events.push(event)
|
||||
expect(events.at(-1)).toEqual({ type: "done", prUrl })
|
||||
})
|
||||
})
|
||||
@@ -1,16 +1,19 @@
|
||||
// Headless fleet probes (#120): per-turn tool metrics carry duration and seat,
|
||||
// and stay out of the default human output.
|
||||
import { afterEach, beforeEach, describe, expect, test } from "bun:test"
|
||||
import { mkdtemp, rm } from "node:fs/promises"
|
||||
import { mkdir, mkdtemp, rm, writeFile } from "node:fs/promises"
|
||||
import { tmpdir } from "node:os"
|
||||
import path from "node:path"
|
||||
import { Factory, gateRunner } from "@heretek-ai/es-core"
|
||||
import { Factory, type FactoryState, factoryLayout, gateRunner } from "@heretek-ai/es-core"
|
||||
import {
|
||||
driveHeadless,
|
||||
type HarnessDriver,
|
||||
type HeadlessEvent,
|
||||
HeadlessJsonlSchema,
|
||||
headlessFinished,
|
||||
opencodeDriver,
|
||||
presentEvent,
|
||||
runHeadless,
|
||||
toJsonl,
|
||||
} from "../src/headless.ts"
|
||||
|
||||
@@ -117,3 +120,190 @@ describe("turn-metrics human output (#120)", () => {
|
||||
expect(presentEvent(metrics, "debug")).toEqual(metrics)
|
||||
})
|
||||
})
|
||||
|
||||
describe("runHeadless end conditions (#247)", () => {
|
||||
test("a research run drives to done, and done carries no prUrl without a release", async () => {
|
||||
const dir = await mkdtemp(path.join(tmpdir(), "es-headless-done-"))
|
||||
const dirState = await mkdtemp(path.join(tmpdir(), "es-headless-done-state-"))
|
||||
try {
|
||||
const factory = new Factory(dir, { gates: gateRunner({ stateDir: dirState }), stateDir: dirState })
|
||||
await factory.beginResearchRun({ objective: "what queue", ceilingUSD: 5 })
|
||||
const seen: Array<{ agent: string; prompt: string }> = []
|
||||
const driver: HarnessDriver = {
|
||||
id: "fake",
|
||||
available: async () => true,
|
||||
async *turn(input) {
|
||||
seen.push({ agent: input.agent, prompt: input.prompt })
|
||||
yield { text: "working" }
|
||||
// The coordinator flow, played against the real tools and gate.
|
||||
const { researchTools } = await import("@heretek-ai/es-core")
|
||||
const stubFetch = (async () =>
|
||||
new Response("<p>Queues decouple workers from producers durably.</p>", {
|
||||
headers: { "content-type": "text/html" },
|
||||
})) as unknown as typeof fetch
|
||||
const tools = researchTools({
|
||||
root: dir,
|
||||
fetch: stubFetch,
|
||||
stateDir: dirState,
|
||||
policy: async () => ({ root: dir }),
|
||||
})
|
||||
const fetched = await tools
|
||||
.find((tool) => tool.name === "es_research_fetch")!
|
||||
.execute({ url: "https://192.0.2.1/queues" }, { agent: "es-research-alpha" })
|
||||
const sha = /sha256:([0-9a-f]{64})/.exec(fetched)?.[1]
|
||||
await writeFile(
|
||||
path.join(dir, ".factory/research/REPORT.md"),
|
||||
`# Report\n\n- Queues decouple workers [VERIFIED: sha256:${sha} "decouple workers from producers"]\n`,
|
||||
)
|
||||
await new Factory(dir, { gates: gateRunner({ stateDir: dirState }), stateDir: dirState }).completeResearch(
|
||||
"deep-researcher",
|
||||
)
|
||||
return "ses_fake"
|
||||
},
|
||||
}
|
||||
const events: HeadlessEvent[] = []
|
||||
for await (const event of runHeadless({ root: dir, driver, maxTurns: 5, stallTurns: 99, stateDir: dirState }))
|
||||
events.push(event)
|
||||
// The driver saw the factory seat identity and the headless prompt (#247.2).
|
||||
expect(seen).toHaveLength(1)
|
||||
expect(seen[0]?.agent).toBe("factory")
|
||||
expect(seen[0]?.prompt).toContain("headless run")
|
||||
expect(events[0]).toMatchObject({ type: "start", driver: "fake" })
|
||||
expect(events.filter((event) => event.type === "turn")).toHaveLength(1)
|
||||
const done = events.at(-1)!
|
||||
expect(done).toEqual({ type: "done" })
|
||||
expect("prUrl" in done).toBe(false)
|
||||
} finally {
|
||||
await rm(dir, { recursive: true, force: true })
|
||||
await rm(dirState, { recursive: true, force: true })
|
||||
}
|
||||
})
|
||||
|
||||
test("headlessFinished maps DONE (with prUrl spread), pending approvals and the grill", async () => {
|
||||
const dir = await mkdtemp(path.join(tmpdir(), "es-headless-finished-"))
|
||||
try {
|
||||
const done = (release?: FactoryState["release"]) =>
|
||||
({ stage: "DONE", ...(release ? { release } : {}) }) as unknown as FactoryState
|
||||
// No release: no prUrl key. With one: the url spreads onto done (#247.1).
|
||||
expect(await headlessFinished(dir, done())).toEqual({ type: "done" })
|
||||
expect(await headlessFinished(dir, done({ prUrl: "https://example.test/pr/9" }))).toEqual({
|
||||
type: "done",
|
||||
prUrl: "https://example.test/pr/9",
|
||||
})
|
||||
// A pending approval waits on a human, with its stages.
|
||||
await mkdir(path.dirname(factoryLayout(dir).pending), { recursive: true })
|
||||
await writeFile(factoryLayout(dir).pending, JSON.stringify([{ stage: "spec", requestedBy: "factory", at: "x" }]))
|
||||
expect(await headlessFinished(dir, { stage: "RESEARCH" } as unknown as FactoryState)).toEqual({
|
||||
type: "waiting",
|
||||
reason: "a human must approve (es approve <stage>)",
|
||||
stages: ["spec"],
|
||||
})
|
||||
await writeFile(factoryLayout(dir).pending, "[]")
|
||||
// The grill (no pending) waits on a human; a mid-build stage keeps driving.
|
||||
expect(await headlessFinished(dir, { stage: "GRILL" } as unknown as FactoryState)).toMatchObject({
|
||||
type: "waiting",
|
||||
reason: expect.stringContaining("grill"),
|
||||
})
|
||||
expect(await headlessFinished(dir, { stage: "RESEARCH" } as unknown as FactoryState)).toBeUndefined()
|
||||
} finally {
|
||||
await rm(dir, { recursive: true, force: true })
|
||||
}
|
||||
})
|
||||
|
||||
test("a STOP file halts at headless level without granting anything", async () => {
|
||||
const dir = await mkdtemp(path.join(tmpdir(), "es-headless-stop-"))
|
||||
const dirState = await mkdtemp(path.join(tmpdir(), "es-headless-stop-state-"))
|
||||
try {
|
||||
await writeFile(path.join(dir, "README.md"), "# x\n")
|
||||
await new Factory(dir, { gates: gateRunner({ stateDir: dirState }), stateDir: dirState }).provisionRun(
|
||||
"tester",
|
||||
5,
|
||||
)
|
||||
await mkdir(path.dirname(factoryLayout(dir).stop), { recursive: true })
|
||||
await writeFile(factoryLayout(dir).stop, "lunch\n")
|
||||
const driver: HarnessDriver = {
|
||||
id: "fake",
|
||||
available: async () => true,
|
||||
async *turn() {
|
||||
yield { text: "unused" }
|
||||
return "ses_fake"
|
||||
},
|
||||
}
|
||||
const halted: HeadlessEvent[] = []
|
||||
for await (const event of runHeadless({ root: dir, driver, maxTurns: 3, stateDir: dirState })) halted.push(event)
|
||||
expect(halted[0]).toMatchObject({ type: "start" })
|
||||
expect(halted.at(-1)).toMatchObject({ type: "halted", reason: expect.stringContaining("STOP file") })
|
||||
expect(halted.some((event) => event.type === "turn")).toBe(false)
|
||||
} finally {
|
||||
await rm(dir, { recursive: true, force: true })
|
||||
await rm(dirState, { recursive: true, force: true })
|
||||
}
|
||||
})
|
||||
|
||||
test("an unknown --driver is refused before anything launches", async () => {
|
||||
const { main } = await import("../src/main.ts")
|
||||
const printed: string[] = []
|
||||
const code = await main(["factory", "run", "--headless", "--driver", "nope"], {
|
||||
print: (text) => printed.push(text),
|
||||
confirm: { interactive: false, write: () => {}, readLine: async () => "y", readSecret: async () => "" },
|
||||
cwd: root,
|
||||
stateDir: state,
|
||||
})
|
||||
expect(code).toBe(2)
|
||||
expect(printed.join("\n")).toContain("Unknown driver")
|
||||
})
|
||||
})
|
||||
|
||||
describe("opencode driver argv shape (#247.2)", () => {
|
||||
test("run --standalone --format json --agent, with model/session/prompt passthrough", async () => {
|
||||
const { chmod } = await import("node:fs/promises")
|
||||
const bin = await mkdtemp(path.join(tmpdir(), "es-driver-argv-"))
|
||||
const previous = process.env.PATH
|
||||
try {
|
||||
await writeFile(
|
||||
path.join(bin, "opencode"),
|
||||
`#!/bin/sh\nprintf '%s\\n' "$@" > "$ES_ARGV_LOG"\necho '{"type":"text","sessionID":"ses_argv","part":{"text":"hi"}}'\n`,
|
||||
)
|
||||
await chmod(path.join(bin, "opencode"), 0o755)
|
||||
process.env.PATH = `${bin}${path.delimiter}${previous}`
|
||||
const argvOf = async (input: Parameters<typeof opencodeDriver.turn>[0]): Promise<string[]> => {
|
||||
const log = path.join(bin, `argv-${Math.random().toString(36).slice(2)}.txt`)
|
||||
process.env.ES_ARGV_LOG = log
|
||||
try {
|
||||
const turn = opencodeDriver.turn(input)
|
||||
let step = await turn.next()
|
||||
while (!step.done) step = await turn.next()
|
||||
expect(step.value).toBe("ses_argv")
|
||||
return (await Bun.file(log).text()).split("\n").filter(Boolean)
|
||||
} finally {
|
||||
delete process.env.ES_ARGV_LOG
|
||||
}
|
||||
}
|
||||
expect(await argvOf({ root, agent: "factory", prompt: "go" })).toEqual([
|
||||
"run",
|
||||
"--standalone",
|
||||
"--format",
|
||||
"json",
|
||||
"--agent",
|
||||
"factory",
|
||||
"go",
|
||||
])
|
||||
expect(await argvOf({ root, agent: "factory", prompt: "again", model: "prov/model", session: "ses_1" })).toEqual([
|
||||
"run",
|
||||
"--standalone",
|
||||
"--format",
|
||||
"json",
|
||||
"--agent",
|
||||
"factory",
|
||||
"--model",
|
||||
"prov/model",
|
||||
"--session",
|
||||
"ses_1",
|
||||
"again",
|
||||
])
|
||||
} finally {
|
||||
process.env.PATH = previous
|
||||
await rm(bin, { recursive: true, force: true })
|
||||
}
|
||||
})
|
||||
})
|
||||
|
||||
@@ -0,0 +1,282 @@
|
||||
// #204 integration through the CLI: harvest + distill over several
|
||||
// unit-constructed run dirs (one with a broken audit chain); scrubbed
|
||||
// output and the code-disposes gate dropping fabricated evidence with
|
||||
// reasons.
|
||||
//
|
||||
// Unit-constructed only: no real factory runs, no timing/perf assertions,
|
||||
// no --open-pr (human TTY; existing pipe-refusal coverage stands).
|
||||
import { describe, expect, test } from "bun:test"
|
||||
import { appendFile, mkdir, mkdtemp, readdir, readFile, rm, stat, writeFile } from "node:fs/promises"
|
||||
import { tmpdir } from "node:os"
|
||||
import path from "node:path"
|
||||
import {
|
||||
appendAuditEntry,
|
||||
disposedWithReasons,
|
||||
disposeReason,
|
||||
distillProposals,
|
||||
factoryLayout,
|
||||
TelemetrySchema,
|
||||
undisposed,
|
||||
} from "@heretek-ai/es-core"
|
||||
import { main } from "../src/main.ts"
|
||||
import type { ConfirmIO } from "../src/tty.ts"
|
||||
|
||||
const pipe: ConfirmIO = { interactive: false, write: () => {}, readLine: async () => "y", readSecret: async () => "" }
|
||||
|
||||
const GH_A = "ghp_cliHarvest001AAA"
|
||||
const GH_BROKEN = "ghu_cliBroken999aaa"
|
||||
const SK_ANT = "sk-ant-cliplanted0000Z"
|
||||
const AKIA = "AKIAIOSFODNN7EXAMPLE"
|
||||
const BEARER = "eyJhbGciOiJIUzI1NiJ9"
|
||||
const GATE_RULE = "lint/no-unused-vars"
|
||||
|
||||
async function writeGateSummary(root: string, id: string): Promise<void> {
|
||||
const dir = path.join(factoryLayout(root).runs, id, "gates", "2026-10-04T00-30-full")
|
||||
await mkdir(dir, { recursive: true })
|
||||
await writeFile(
|
||||
path.join(dir, "summary.json"),
|
||||
`${JSON.stringify(
|
||||
{
|
||||
passed: false,
|
||||
findings: [
|
||||
{ file: "src/x.ts", line: 3, rule: GATE_RULE, severity: "error", message: "unused", check: "lint" },
|
||||
{ file: "src/y.ts", line: 9, rule: GATE_RULE, severity: "error", message: "unused", check: "lint" },
|
||||
],
|
||||
},
|
||||
null,
|
||||
2,
|
||||
)}\n`,
|
||||
)
|
||||
}
|
||||
|
||||
async function writeAuditRecords(root: string): Promise<void> {
|
||||
const dir = factoryLayout(root).audits
|
||||
await mkdir(path.join(dir, "audit-01"), { recursive: true })
|
||||
const finding = (title: string, detail: string) => ({
|
||||
kind: "invariant",
|
||||
title,
|
||||
severity: "high",
|
||||
file: "packages/core/src/trust/policy.ts",
|
||||
lines: [59, 75],
|
||||
excerpt: "export function evaluateWrite(context, agentId, target)",
|
||||
detail,
|
||||
holds: false,
|
||||
})
|
||||
await writeFile(
|
||||
path.join(dir, "audit-01", "records.json"),
|
||||
`${JSON.stringify(
|
||||
[
|
||||
{
|
||||
audit: "audit-01",
|
||||
round: 1,
|
||||
seat: "auditor-thesis",
|
||||
by: "auditor",
|
||||
verdict: "fail",
|
||||
notes: "notes",
|
||||
findings: [
|
||||
finding("control files are deny-write", "the programmer seat can write a control file path"),
|
||||
finding("control files are deny-write again", "a second invariant detail here"),
|
||||
],
|
||||
at: "2026-10-04T00:00:00.000Z",
|
||||
},
|
||||
],
|
||||
null,
|
||||
2,
|
||||
)}\n`,
|
||||
)
|
||||
}
|
||||
|
||||
async function writeChain(root: string, haltReason: string): Promise<void> {
|
||||
await appendAuditEntry(root, { actor: "system", action: "stage.build" })
|
||||
await appendAuditEntry(root, {
|
||||
actor: "agent:qa",
|
||||
action: "qa.fail",
|
||||
payload: { phase: "p1", reason: "lint gate red" },
|
||||
})
|
||||
await appendAuditEntry(root, { actor: "system", action: "stage.halted", payload: { reason: haltReason } })
|
||||
}
|
||||
|
||||
async function makeRun(parent: string, id: string, haltReason: string, broken: boolean): Promise<string> {
|
||||
const root = path.join(parent, id)
|
||||
await writeGateSummary(root, id)
|
||||
await writeAuditRecords(root)
|
||||
await writeChain(root, haltReason)
|
||||
if (broken) await appendFile(factoryLayout(root).audit, "not-json\n")
|
||||
return root
|
||||
}
|
||||
|
||||
async function writeEvals(dir: string, failures: { id: string; failures: string[] }[]): Promise<void> {
|
||||
await mkdir(dir, { recursive: true })
|
||||
await writeFile(
|
||||
path.join(dir, "agg.json"),
|
||||
`${JSON.stringify(
|
||||
{
|
||||
at: "2026-10-04T00-00-00-000Z",
|
||||
model: "test/model",
|
||||
results: failures.map((entry, index) => ({
|
||||
id: entry.id,
|
||||
pass: entry.failures.length === 0,
|
||||
failures: entry.failures,
|
||||
steps: 4 + index,
|
||||
})),
|
||||
},
|
||||
null,
|
||||
2,
|
||||
)}\n`,
|
||||
)
|
||||
}
|
||||
|
||||
async function walkFiles(dir: string): Promise<string[]> {
|
||||
const out: string[] = []
|
||||
const visit = async (current: string): Promise<void> => {
|
||||
for (const name of await readdir(current)) {
|
||||
const full = path.join(current, name)
|
||||
const info = await stat(full).catch(() => undefined)
|
||||
if (info?.isDirectory()) await visit(full)
|
||||
else out.push(full)
|
||||
}
|
||||
}
|
||||
await visit(dir)
|
||||
return out.sort()
|
||||
}
|
||||
|
||||
describe("issue #204: es improve harvest + distill over several runs incl. a broken chain", () => {
|
||||
test("harvest reports the broken chain and scrubs planted tokens; distill writes all three kinds, drops fabrications", async () => {
|
||||
const parent = await mkdtemp(path.join(tmpdir(), "es-204-cli-"))
|
||||
const root = await mkdtemp(path.join(tmpdir(), "es-204-cli-root-"))
|
||||
const state = await mkdtemp(path.join(tmpdir(), "es-204-cli-state-"))
|
||||
try {
|
||||
const run = async (argv: string[]) => {
|
||||
const printed: string[] = []
|
||||
const code = await main(argv, {
|
||||
print: (text) => printed.push(text),
|
||||
confirm: pipe,
|
||||
cwd: root,
|
||||
stateDir: state,
|
||||
})
|
||||
return { code, out: printed.join("\n") }
|
||||
}
|
||||
const runA = await makeRun(parent, "run-a", `halt A: GITHUB_TOKEN=${GH_A}`, false)
|
||||
const runB = await makeRun(parent, "run-b", `halt B: Bearer ${BEARER}`, false)
|
||||
const runBroken = await makeRun(parent, "run-broken", `halt C: ${GH_BROKEN}`, true)
|
||||
const brokenFile = factoryLayout(runBroken).audit
|
||||
const before = await readFile(brokenFile, "utf8")
|
||||
const evals = path.join(parent, "evals")
|
||||
await writeEvals(evals, [
|
||||
{ id: "case-x", failures: ["gate lint red", "gate lint red", `leaked ${SK_ANT}`] },
|
||||
{ id: "case-y", failures: ["gate lint red", `key ${AKIA} here`] },
|
||||
{ id: "case-ok", failures: [] },
|
||||
])
|
||||
const telemetryFile = path.join(root, "telemetry.json")
|
||||
const harvested = await run([
|
||||
"improve",
|
||||
"harvest",
|
||||
"--runs",
|
||||
`${runA},${runB},${runBroken}`,
|
||||
"--evals",
|
||||
evals,
|
||||
"--out",
|
||||
telemetryFile,
|
||||
])
|
||||
expect(harvested.code).toBe(0)
|
||||
expect(harvested.out).toContain("3 run(s) and 3 eval(s)")
|
||||
const raw = await readFile(telemetryFile, "utf8")
|
||||
const telemetry = TelemetrySchema.parse(JSON.parse(raw))
|
||||
expect(telemetry.runs.map((entry) => entry.id)).toEqual(["run-a", "run-b", "run-broken"])
|
||||
for (const secret of [GH_A, GH_BROKEN, SK_ANT, AKIA, BEARER]) expect(raw).not.toContain(secret)
|
||||
expect(raw).toContain("GITHUB_TOKEN=")
|
||||
expect(raw).toContain("Bearer")
|
||||
const brokenTelemetry = telemetry.runs.find((entry) => entry.id === "run-broken")!
|
||||
expect(brokenTelemetry.auditChain.valid).toBe(false)
|
||||
expect(typeof brokenTelemetry.auditChain.error).toBe("string")
|
||||
// Reported, never repaired.
|
||||
expect(await readFile(brokenFile, "utf8")).toBe(before)
|
||||
const proposalsDir = path.join(root, "proposals")
|
||||
const distilled = await run([
|
||||
"improve",
|
||||
"distill",
|
||||
"--telemetry",
|
||||
telemetryFile,
|
||||
"--out",
|
||||
proposalsDir,
|
||||
"--min-occurrences",
|
||||
"2",
|
||||
"--min-runs",
|
||||
"2",
|
||||
])
|
||||
expect(distilled.code).toBe(0)
|
||||
expect(distilled.out).toContain("Nothing was applied")
|
||||
expect(distilled.out).toContain("prompt-guidance")
|
||||
expect(distilled.out).toContain("gate-tuning")
|
||||
expect(distilled.out).toContain("domain-pack")
|
||||
const files = await walkFiles(proposalsDir)
|
||||
expect(files.some((file) => path.basename(file) === "gates.json")).toBe(false)
|
||||
expect(files.some((file) => path.basename(file) === "proposal.json")).toBe(true)
|
||||
// Fabricated evidence against the CLI-harvested telemetry is dropped with reasons.
|
||||
const proposals = await distillProposals(telemetry, { minOccurrences: 2, minRuns: 2 })
|
||||
expect(undisposed(proposals, telemetry)).toEqual(proposals)
|
||||
const [first] = proposals
|
||||
const forged = { ...first!, cluster: { ...first!.cluster, evidence: ["run x: invented quote"] } }
|
||||
expect(disposeReason(forged, telemetry).some((reason) => reason.includes("verbatim"))).toBe(true)
|
||||
expect(undisposed([forged], telemetry)).toEqual([])
|
||||
expect(disposedWithReasons([forged], telemetry)[0]!.reasons.length).toBeGreaterThan(0)
|
||||
} finally {
|
||||
await rm(parent, { recursive: true, force: true })
|
||||
await rm(root, { recursive: true, force: true })
|
||||
await rm(state, { recursive: true, force: true })
|
||||
}
|
||||
})
|
||||
|
||||
test("a hallucinated model numeral is disposed by the gate with its reason in output and dropped.json", async () => {
|
||||
const parent = await mkdtemp(path.join(tmpdir(), "es-204-cli-model-"))
|
||||
const root = await mkdtemp(path.join(tmpdir(), "es-204-cli-model-root-"))
|
||||
const state = await mkdtemp(path.join(tmpdir(), "es-204-cli-model-state-"))
|
||||
try {
|
||||
const run = async (argv: string[]) => {
|
||||
const printed: string[] = []
|
||||
const code = await main(argv, {
|
||||
print: (text) => printed.push(text),
|
||||
confirm: pipe,
|
||||
cwd: root,
|
||||
stateDir: state,
|
||||
})
|
||||
return { code, out: printed.join("\n") }
|
||||
}
|
||||
const runA = await makeRun(parent, "run-a", "halt A", false)
|
||||
const evals = path.join(parent, "evals")
|
||||
await writeEvals(evals, [
|
||||
{ id: "case-x", failures: ["fails like gpt-4 would", "fails like gpt-4 would"] },
|
||||
{ id: "case-y", failures: ["fails like gpt-4 would"] },
|
||||
])
|
||||
const telemetryFile = path.join(root, "telemetry.json")
|
||||
expect((await run(["improve", "harvest", "--runs", runA, "--evals", evals, "--out", telemetryFile])).code).toBe(0)
|
||||
const proposalsDir = path.join(root, "proposals")
|
||||
const distilled = await run([
|
||||
"improve",
|
||||
"distill",
|
||||
"--telemetry",
|
||||
telemetryFile,
|
||||
"--out",
|
||||
proposalsDir,
|
||||
"--min-occurrences",
|
||||
"3",
|
||||
"--min-runs",
|
||||
"2",
|
||||
])
|
||||
expect(distilled.code).toBe(0)
|
||||
expect(distilled.out).toContain("Disposed by the gate")
|
||||
expect(distilled.out).toContain("hallucinated model numeral")
|
||||
const dropped = JSON.parse(await readFile(path.join(proposalsDir, "dropped.json"), "utf8")) as Array<{
|
||||
reasons: string[]
|
||||
}>
|
||||
expect(dropped.length).toBeGreaterThan(0)
|
||||
expect(
|
||||
dropped.some((entry) => entry.reasons.some((reason) => reason.includes("hallucinated model numeral"))),
|
||||
).toBe(true)
|
||||
} finally {
|
||||
await rm(parent, { recursive: true, force: true })
|
||||
await rm(root, { recursive: true, force: true })
|
||||
await rm(state, { recursive: true, force: true })
|
||||
}
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,217 @@
|
||||
// MCP caller-identity pinning (#230, #232): the adapter pins one agent id per
|
||||
// server in the server environment — `ES_MCP_AGENT`, else the legacy
|
||||
// `ES_AGENT`, else the `defaultAgent` option — and the per-call `agent`
|
||||
// argument is ignored. Without an adapter identity the caller is "mcp", which
|
||||
// maps to no seat, so every seat-checked tool refuses. A pinned string that is
|
||||
// not a registry agent id (e.g. `programmer`) pins to a seat-less identity:
|
||||
// the caller string is still pinned, it just holds no seat.
|
||||
import { afterEach, beforeEach, describe, expect, test } from "bun:test"
|
||||
import { mkdtemp, readFile, rm, writeFile } from "node:fs/promises"
|
||||
import { tmpdir } from "node:os"
|
||||
import path from "node:path"
|
||||
import { Factory, gateRunner, HUMAN_VERBS, readApproval } from "@heretek-ai/es-core"
|
||||
import { createMcpServer } from "../src/mcp.ts"
|
||||
|
||||
let root: string
|
||||
let state: string
|
||||
|
||||
beforeEach(async () => {
|
||||
root = await mkdtemp(path.join(tmpdir(), "es-mcp-id-"))
|
||||
state = await mkdtemp(path.join(tmpdir(), "es-mcp-id-state-"))
|
||||
const proc = Bun.spawn(["git", "init", "-q", "-b", "main"], { cwd: root, stdout: "pipe", stderr: "pipe" })
|
||||
await proc.exited
|
||||
await writeFile(path.join(root, "README.md"), "# x\n")
|
||||
})
|
||||
|
||||
afterEach(async () => {
|
||||
await rm(root, { recursive: true, force: true })
|
||||
await rm(state, { recursive: true, force: true })
|
||||
})
|
||||
|
||||
const call = (server: ReturnType<typeof createMcpServer>, name: string, args: Record<string, unknown> = {}, id = 1) =>
|
||||
server.handle({ jsonrpc: "2.0", id, method: "tools/call", params: { name, arguments: args } })
|
||||
|
||||
describe("mcp identity pin order (mcp.ts:29)", () => {
|
||||
test("the legacy ES_AGENT pins the caller when ES_MCP_AGENT is absent", async () => {
|
||||
const server = createMcpServer({ root, stateDir: state, env: { ES_AGENT: "factory" } })
|
||||
const refused = await call(server, "complete", {})
|
||||
expect(refused?.result.isError).toBe(true)
|
||||
expect(refused?.result.content[0].text).toContain("called by factory")
|
||||
})
|
||||
|
||||
test("ES_MCP_AGENT beats ES_AGENT beats defaultAgent", async () => {
|
||||
const top = createMcpServer({
|
||||
root,
|
||||
stateDir: state,
|
||||
env: { ES_MCP_AGENT: "es-qa-functional", ES_AGENT: "factory" },
|
||||
defaultAgent: "factory",
|
||||
})
|
||||
const pinned = await call(top, "complete", { agent: "factory" })
|
||||
expect(pinned?.result.isError).toBe(true)
|
||||
expect(pinned?.result.content[0].text).toContain("called by es-qa-functional")
|
||||
|
||||
const legacy = createMcpServer({
|
||||
root,
|
||||
stateDir: state,
|
||||
env: { ES_AGENT: "factory" },
|
||||
defaultAgent: "es-qa-functional",
|
||||
})
|
||||
const viaLegacy = await call(legacy, "complete", {})
|
||||
expect(viaLegacy?.result.isError).toBe(true)
|
||||
expect(viaLegacy?.result.content[0].text).toContain("called by factory")
|
||||
|
||||
const fallback = createMcpServer({ root, stateDir: state, env: {}, defaultAgent: "factory" })
|
||||
const viaDefault = await call(fallback, "complete", {})
|
||||
expect(viaDefault?.result.isError).toBe(true)
|
||||
expect(viaDefault?.result.content[0].text).toContain("called by factory")
|
||||
})
|
||||
|
||||
test("a pinned string without a registry seat stays seat-less but pinned", async () => {
|
||||
const server = createMcpServer({ root, stateDir: state, env: { ES_MCP_AGENT: "programmer" } })
|
||||
const refused = await call(server, "complete", {})
|
||||
expect(refused?.result.isError).toBe(true)
|
||||
// Registry ids are `es-programmer`/`es-manager`/`factory`: the literal
|
||||
// `programmer` holds no seat, yet the caller string is still the pin.
|
||||
expect(refused?.result.content[0].text).toContain("called by programmer")
|
||||
})
|
||||
})
|
||||
|
||||
describe("as-mcp refusal across every seat-gated tool", () => {
|
||||
// Every tool whose execute path opens with requireSeat, called with an
|
||||
// empty adapter environment (caller "mcp", no seat).
|
||||
const seatGated: Array<{ tool: string; args: Record<string, unknown> }> = [
|
||||
{ tool: "frontier_write", args: { frontier: {} } },
|
||||
{ tool: "research_complete", args: {} },
|
||||
{ tool: "build_start", args: {} },
|
||||
{ tool: "qa_verdict", args: { verdict: "pass", notes: "n" } },
|
||||
{ tool: "tiebreak", args: { verdict: "pass", notes: "n" } },
|
||||
{ tool: "replan", args: { notes: "n" } },
|
||||
{ tool: "release", args: {} },
|
||||
{ tool: "complete", args: {} },
|
||||
]
|
||||
for (const { tool, args } of seatGated) {
|
||||
test(`${tool} refuses the unpinned caller as mcp`, async () => {
|
||||
const server = createMcpServer({ root, stateDir: state, env: {} })
|
||||
const refused = await call(server, tool, args)
|
||||
expect(refused?.result.isError).toBe(true)
|
||||
expect(refused?.result.content[0].text).toContain("called by mcp")
|
||||
})
|
||||
}
|
||||
})
|
||||
|
||||
describe("no MCP tool grants a human-only verb (derived from HUMAN_VERBS)", () => {
|
||||
test("the coarse tool list is exactly the 12 read/request/seat tools", async () => {
|
||||
const server = createMcpServer({ root, stateDir: state })
|
||||
const list = await server.handle({ jsonrpc: "2.0", id: 1, method: "tools/list" })
|
||||
const names = list?.result.tools.map((tool: { name: string }) => tool.name).sort()
|
||||
// A future human-only verb gaining an MCP tool changes this list and fails loudly here.
|
||||
expect(names).toEqual(
|
||||
[
|
||||
"build_start",
|
||||
"complete",
|
||||
"frontier_write",
|
||||
"gates_run",
|
||||
"qa_verdict",
|
||||
"release",
|
||||
"replan",
|
||||
"request_approval",
|
||||
"research_complete",
|
||||
"spec_validate",
|
||||
"status",
|
||||
"tiebreak",
|
||||
].sort(),
|
||||
)
|
||||
})
|
||||
|
||||
test("no listed tool (except the request-only approval) matches a HUMAN_VERBS rule", async () => {
|
||||
const server = createMcpServer({ root, stateDir: state })
|
||||
const list = await server.handle({ jsonrpc: "2.0", id: 1, method: "tools/list" })
|
||||
const names = list?.result.tools.map((tool: { name: string }) => tool.name) as string[]
|
||||
const humanOnly = (verb: string, sub: string | undefined): boolean => {
|
||||
const rule = HUMAN_VERBS.find(([name]) => name === verb)
|
||||
if (!rule) return false
|
||||
const [, predicate] = rule
|
||||
return predicate === undefined ? true : predicate(sub)
|
||||
}
|
||||
for (const name of names) {
|
||||
if (name === "request_approval") continue // request-only: proven below, never grants.
|
||||
const tokens = name.split(/[_-]/)
|
||||
const matched = HUMAN_VERBS.some(([verb, predicate]) => {
|
||||
if (!tokens.includes(verb)) return false
|
||||
if (predicate === undefined) return true
|
||||
return tokens.some((token) => token !== verb && predicate(token))
|
||||
})
|
||||
expect([name, matched, humanOnly("never-a-verb", undefined)]).toEqual([name, false, false])
|
||||
}
|
||||
})
|
||||
|
||||
test("request_approval records a request without granting anything", async () => {
|
||||
const factory = new Factory(root, { gates: gateRunner({ stateDir: state }), stateDir: state })
|
||||
await factory.writeFrontier("grill", {
|
||||
version: "1.1",
|
||||
idea: "mcp test",
|
||||
spendCeiling: { currency: "USD", maxAmount: 3 },
|
||||
settled: true,
|
||||
nodes: [{ id: "a", question: "q?", answer: "yes", status: "settled" }],
|
||||
})
|
||||
const server = createMcpServer({ root, stateDir: state, env: { ES_MCP_AGENT: "factory" } })
|
||||
const requested = await call(server, "request_approval", { stage: "frontier" })
|
||||
expect(requested?.result.isError).toBeUndefined()
|
||||
expect(requested?.result.content[0].text).toContain("Requested human approval")
|
||||
// Request-only: the approval file is still absent (a human grants it elsewhere).
|
||||
expect(await readApproval(root, "frontier")).toBeUndefined()
|
||||
})
|
||||
})
|
||||
|
||||
describe("mcp identity over real stdio", () => {
|
||||
test("serveStdio pins ES_MCP_AGENT: a piped agent claim is refused as the pin", async () => {
|
||||
const proc = Bun.spawn(["bun", path.join(import.meta.dir, "../src/main.ts"), "mcp", "--cwd", root], {
|
||||
stdin: "pipe",
|
||||
stdout: "pipe",
|
||||
stderr: "pipe",
|
||||
// The literal `programmer` holds no registry seat (#232.1), so the
|
||||
// seat check itself names the pin — the strongest stdio proof.
|
||||
env: { ...process.env, ES_MCP_AGENT: "programmer" },
|
||||
})
|
||||
proc.stdin.write(
|
||||
`${JSON.stringify({ jsonrpc: "2.0", id: 1, method: "tools/call", params: { name: "complete", arguments: { agent: "factory" } } })}\n`,
|
||||
)
|
||||
proc.stdin.end()
|
||||
const lines = (await new Response(proc.stdout).text())
|
||||
.trim()
|
||||
.split("\n")
|
||||
.map((line) => JSON.parse(line))
|
||||
await proc.exited
|
||||
expect(lines.map((line) => line.id)).toEqual([1])
|
||||
expect(lines[0].result.isError).toBe(true)
|
||||
// The per-call `agent: "factory"` is ignored: the refusal names the pin.
|
||||
expect(lines[0].result.content[0].text).toContain("called by programmer")
|
||||
expect(await readFile(path.join(root, "README.md"), "utf8")).toContain("# x")
|
||||
})
|
||||
})
|
||||
|
||||
describe("mcp STOP/HALTED enforcement (#227)", () => {
|
||||
test("a STOP file refuses tool calls on the MCP surface", async () => {
|
||||
await Bun.write(path.join(root, ".factory/STOP"), "halt\n")
|
||||
try {
|
||||
const server = createMcpServer({ root, stateDir: state, env: { ES_AGENT: "factory" } })
|
||||
const refused = await call(server, "status", {})
|
||||
expect(refused?.result.isError).toBe(true)
|
||||
expect(refused?.result.content[0].text).toContain("stopped")
|
||||
} finally {
|
||||
await rm(path.join(root, ".factory/STOP"), { force: true })
|
||||
}
|
||||
})
|
||||
|
||||
test("a HALTED run refuses non-status tool calls on the MCP surface", async () => {
|
||||
const factory = new Factory(root, { gates: gateRunner({ stateDir: state }), stateDir: state })
|
||||
await factory.begin("human:tester")
|
||||
await factory.halt("human:tester", "spend ceiling")
|
||||
const server = createMcpServer({ root, stateDir: state, env: { ES_AGENT: "factory" } })
|
||||
const refused = await call(server, "complete", {})
|
||||
expect(refused?.result.isError).toBe(true)
|
||||
expect(refused?.result.content[0].text).toContain("halted")
|
||||
const status = await call(server, "status", {})
|
||||
expect(status?.result.isError).not.toBe(true)
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,131 @@
|
||||
// Trust pinning verbs stay terminal-only (#181 §2, #221 per-verb pipe gaps).
|
||||
// Pipe (non-interactive) runs of every human-gated trust verb refuse with code
|
||||
// 3 and record nothing; the shell policy denies the same verbs for every
|
||||
// agent. No test here runs a real human verb: the only confirms are pipes.
|
||||
import { afterEach, beforeEach, describe, expect, test } from "bun:test"
|
||||
import { mkdir, mkdtemp, rm, writeFile } from "node:fs/promises"
|
||||
import { tmpdir } from "node:os"
|
||||
import path from "node:path"
|
||||
import {
|
||||
evaluateShell,
|
||||
Factory,
|
||||
factoryLayout,
|
||||
gateRunner,
|
||||
invokesHumanOnly,
|
||||
sealHumanKey,
|
||||
verifyControl,
|
||||
} from "@heretek-ai/es-core"
|
||||
import { main } from "../src/main.ts"
|
||||
import type { ConfirmIO } from "../src/tty.ts"
|
||||
|
||||
const PASSPHRASE = "test-passphrase-181"
|
||||
|
||||
let root: string
|
||||
let state: string
|
||||
const realXdgConfig = process.env.XDG_CONFIG_HOME
|
||||
const git = async (...args: string[]) => {
|
||||
const proc = Bun.spawn(["git", ...args], { cwd: root, stdout: "pipe", stderr: "pipe" })
|
||||
await proc.exited
|
||||
}
|
||||
|
||||
beforeEach(async () => {
|
||||
root = await mkdtemp(path.join(tmpdir(), "es-trust-pinning-"))
|
||||
state = await mkdtemp(path.join(tmpdir(), "es-trust-pinning-state-"))
|
||||
process.env.XDG_CONFIG_HOME = path.join(state, "xdg-config")
|
||||
await git("init", "-q", "-b", "main")
|
||||
await writeFile(path.join(root, "README.md"), "# x\n")
|
||||
await git("add", "-A")
|
||||
await git("-c", "user.name=t", "-c", "user.email=t@t", "commit", "-q", "-m", "init")
|
||||
await sealHumanKey(PASSPHRASE, state)
|
||||
})
|
||||
afterEach(async () => {
|
||||
if (realXdgConfig === undefined) delete process.env.XDG_CONFIG_HOME
|
||||
else process.env.XDG_CONFIG_HOME = realXdgConfig
|
||||
await rm(root, { recursive: true, force: true })
|
||||
await rm(state, { recursive: true, force: true })
|
||||
})
|
||||
|
||||
const pipe: ConfirmIO = { interactive: false, write: () => {}, readLine: async () => "y", readSecret: async () => "" }
|
||||
const run = async (argv: string[], confirm: ConfirmIO = pipe) => {
|
||||
const printed: string[] = []
|
||||
const code = await main(argv, { print: (text) => printed.push(text), confirm, cwd: root, stateDir: state })
|
||||
return { code, out: printed.join("\n") }
|
||||
}
|
||||
const trustFile = () => path.join(state, "trust.json")
|
||||
const waiverDir = () => factoryLayout(root).waivers
|
||||
|
||||
describe("trust verbs refuse a pipe (#221)", () => {
|
||||
test("es trust needs a terminal and trusts nothing", async () => {
|
||||
await mkdir(factoryLayout(root).dir, { recursive: true })
|
||||
await writeFile(
|
||||
factoryLayout(root).gates,
|
||||
JSON.stringify({
|
||||
version: 1,
|
||||
commands: [{ id: "probe", kind: "test", command: "touch RAN" }],
|
||||
security: { gitleaks: "off", osv: "off" },
|
||||
}),
|
||||
)
|
||||
const result = await run(["trust"])
|
||||
expect(result.code).toBe(3)
|
||||
expect(result.out).toContain("interactive terminal")
|
||||
expect(await Bun.file(trustFile()).exists()).toBe(false)
|
||||
})
|
||||
|
||||
test("es waive needs a terminal and records nothing", async () => {
|
||||
const result = await run(["waive", "security/*", "--reason", "legacy code under migration"])
|
||||
expect(result.code).toBe(3)
|
||||
expect(result.out).toContain("interactive terminal")
|
||||
expect(await Bun.file(path.join(waiverDir(), "security-abc.json")).exists()).toBe(false)
|
||||
const { readdir } = await import("node:fs/promises")
|
||||
expect(await readdir(waiverDir()).catch(() => [])).toEqual([])
|
||||
})
|
||||
|
||||
test("es rebaseline needs a terminal and keeps the drift", async () => {
|
||||
const { rebaseline } = await import("@heretek-ai/es-core")
|
||||
await rebaseline(root, "test")
|
||||
await mkdir(factoryLayout(root).dir, { recursive: true })
|
||||
await writeFile(factoryLayout(root).config, '{"pr":"off"}\n')
|
||||
expect((await verifyControl(root)).clean).toBe(false)
|
||||
const result = await run(["rebaseline"])
|
||||
expect(result.code).toBe(3)
|
||||
expect(result.out).toContain("interactive terminal")
|
||||
expect((await verifyControl(root)).clean).toBe(false)
|
||||
})
|
||||
|
||||
test("es reseal --sign needs a terminal and keeps the mismatch", async () => {
|
||||
const factory = new Factory(root, { gates: gateRunner({ stateDir: state }), stateDir: state })
|
||||
await factory.begin("human:tester")
|
||||
const stateFile = factoryLayout(root).state
|
||||
await writeFile(stateFile, `${await Bun.file(stateFile).text()} `)
|
||||
const result = await run(["reseal", "--sign"])
|
||||
expect(result.code).toBe(3)
|
||||
expect(result.out).toContain("interactive terminal")
|
||||
await expect(factory.read()).rejects.toThrow("signature mismatch")
|
||||
})
|
||||
})
|
||||
|
||||
describe("agent shells cannot reach human-gated trust verbs", () => {
|
||||
test("the policy denies trust, waive, rebaseline and reseal for every agent", () => {
|
||||
for (const command of [
|
||||
"es trust",
|
||||
"es waive lint/check --reason legacy",
|
||||
"es rebaseline",
|
||||
"es reseal --sign",
|
||||
"es approve frontier",
|
||||
]) {
|
||||
expect([command, invokesHumanOnly(command)]).toEqual([command, true])
|
||||
for (const agent of [undefined, "build", "factory", "es-programmer"]) {
|
||||
const decision = evaluateShell({ root, stateDir: state }, agent, command, { sandboxAvailable: true })
|
||||
expect([command, agent, decision.effect]).toEqual([command, agent, "deny"])
|
||||
if (decision.effect === "deny") expect(decision.reason).toContain("human-only")
|
||||
}
|
||||
}
|
||||
})
|
||||
|
||||
test("reads keep working: help and status stay allowed", () => {
|
||||
expect(invokesHumanOnly("es trust --help")).toBe(false)
|
||||
expect(evaluateShell({ root, stateDir: state }, "build", "es status", { sandboxAvailable: true }).effect).toBe(
|
||||
"allow",
|
||||
)
|
||||
})
|
||||
})
|
||||
@@ -6,8 +6,7 @@
|
||||
"PEER_REVIEWED": 1.0,
|
||||
"TECHNICAL_DOCUMENTATION": 0.8,
|
||||
"PRIMARY_PRESS": 0.7,
|
||||
"PREPRINT": 0.3,
|
||||
"SECONDARY_MEDIA": 0.2
|
||||
"PREPRINT": 0.3
|
||||
},
|
||||
"negBonus": 0.5,
|
||||
"rejectPenalty": 2.5,
|
||||
|
||||
@@ -6,8 +6,7 @@
|
||||
"PEER_REVIEWED": 0.8,
|
||||
"TECHNICAL_DOCUMENTATION": 0.7,
|
||||
"PRIMARY_PRESS": 1.0,
|
||||
"PREPRINT": 0.2,
|
||||
"SECONDARY_MEDIA": 0.3
|
||||
"PREPRINT": 0.2
|
||||
},
|
||||
"negBonus": 0.5,
|
||||
"rejectPenalty": 2.5,
|
||||
|
||||
@@ -6,8 +6,7 @@
|
||||
"PEER_REVIEWED": 0.9,
|
||||
"TECHNICAL_DOCUMENTATION": 1.0,
|
||||
"PRIMARY_PRESS": 0.6,
|
||||
"PREPRINT": 0.5,
|
||||
"SECONDARY_MEDIA": 0.2
|
||||
"PREPRINT": 0.5
|
||||
},
|
||||
"negBonus": 0.75,
|
||||
"rejectPenalty": 3.0,
|
||||
|
||||
@@ -1,8 +1,8 @@
|
||||
---
|
||||
id: brainstormer
|
||||
version: 3
|
||||
version: 4
|
||||
seat: brainstormer
|
||||
description: Fans out divergent lenses on an idea and distills a diversified shortlist.
|
||||
description: Fans out divergent lenses (eight-lens registry; six by default, four lean for callable runs) on an idea and distills a diversified shortlist.
|
||||
---
|
||||
You are the **brainstormer** agent of Epistemic Swarm. Your job is divergent, lateral ideation — never bug fixes, never implementation plans. You orchestrate lenses and distil their output; the mechanics (caps, dedupe, ranking, shortlist) are enforced by tools, not by you.
|
||||
|
||||
@@ -11,7 +11,7 @@ You are the **brainstormer** agent of Epistemic Swarm. Your job is divergent, la
|
||||
2. Launch every lens subagent **in parallel** with the subagent tool: all the calls in one message, in the foreground (background launches are refused). Call them by exact ID (`es-lens-inversion`, `es-lens-constraint-removal`, `es-lens-scamper`, `es-lens-cross-domain`, `es-lens-extreme-user`, `es-lens-scale-shift`, and so on). Give each the brief and how many ideas you expect.
|
||||
3. Record each lens's output with `es_brainstorm_record`, one call per lens. It rejects cap violations and collapses near-duplicates; mention convergences when lenses independently reach the same idea.
|
||||
4. Launch `es-brainstorm-critic` with the surviving idea IDs and their texts. The critic scores every idea with `es_brainstorm_score` (1-5 on novelty, upside, feasibility, fit).
|
||||
5. Call `es_brainstorm_complete`. It re-checks coverage and scores, ranks the ideas, picks the diversified shortlist with one forced outlier slot, and writes the run's `brainstorm.json` and `BRAINSTORM.md`. Report the shortlist to the human, briefly.
|
||||
5. Call `es_brainstorm_complete`. It re-checks coverage and scores, ranks the ideas, picks the diversified shortlist with one forced outlier slot (whenever the shortlist has ≥2 slots), and writes the run's `brainstorm.json` and `BRAINSTORM.md`. Report the shortlist to the human, briefly.
|
||||
|
||||
## Prior art yourself
|
||||
You can run `es_harvest_prior_art` on the surviving ideas before completing: it finds related projects, you judge each relation (`novel`/`similar`/`existing`), and you pass the list to `es_brainstorm_complete` as `priorArt`. Only recorded searches count — the tool refuses URLs no search returned.
|
||||
|
||||
@@ -1,13 +1,13 @@
|
||||
---
|
||||
id: factory
|
||||
version: 9
|
||||
version: 10
|
||||
seat: factory
|
||||
description: Orchestrates the build factory from research to release.
|
||||
---
|
||||
You are the **factory** agent of Epistemic Swarm, an AI build factory. You orchestrate; you do not write product code yourself.
|
||||
|
||||
## How the factory works
|
||||
The run moves through stages enforced by code, not by you: GRILL → RESEARCH → SPEC → BUILD ⇄ QA → RELEASE → DONE. HALTED can happen at any time. Call `es_status` first and whenever you are unsure; its `<factory-state>` block is the truth. If it says there is no run, stop and tell the human to run `/grill` first. Every `es_*` tool refuses an out-of-order step and tells you why. Read the refusal and follow it instead of retrying blindly.
|
||||
The run moves through stages enforced by code, not by you: GRILL → RESEARCH → SPEC → BUILD ⇄ QA → RELEASE → DONE. HALTED can happen at any time. Call `es_status` first and whenever you are unsure; its `<factory-state>` block is the truth. If it says there is no run, stop and tell the human to run `/grill` first. The phase-transition tools refuse an out-of-order step and tell you why. Read the refusal and follow it instead of retrying blindly.
|
||||
|
||||
Two checkpoints belong to the human and only the human: approving the frontier (end of grill) and approving the spec (roadmap plus every GOAL.md). You cannot approve anything. When a checkpoint is due, call `es_request_approval` and tell the human to run `/es-approve` in the TUI (or `es approve <stage>` in a terminal). Then stop and wait.
|
||||
|
||||
|
||||
@@ -1,10 +1,10 @@
|
||||
---
|
||||
name: brainstorm
|
||||
description: Divergent lens fan-out with deterministic dedupe, rubric scoring, and a diversified shortlist with a forced outlier.
|
||||
description: Divergent lens fan-out (eight-lens registry; six by default, four lean for callable runs) with deterministic dedupe, rubric scoring, and a diversified shortlist with a forced outlier whenever the shortlist has ≥2 slots.
|
||||
---
|
||||
# Brainstorm
|
||||
|
||||
Lateral ideation on an idea, run as a fan-out of divergent lenses and distilled mechanically. Never bug fixes, never implementation plans.
|
||||
Lateral ideation on an idea, run as a fan-out of divergent lenses (eight in the registry; six by default, four lean for callable runs) and distilled mechanically. Never bug fixes, never implementation plans.
|
||||
|
||||
## When to use
|
||||
- The human runs `/brainstorm <idea>` or asks for ideas, directions or alternatives.
|
||||
@@ -15,7 +15,7 @@ Lateral ideation on an idea, run as a fan-out of divergent lenses and distilled
|
||||
2. Launch every lens subagent in parallel, by exact ID (`es-lens-<lens>`), each with the brief. Fast tier; one lens each: inversion, constraint-removal, scamper, cross-domain, extreme-user, scale-shift, failure-first, adjacent-possible.
|
||||
3. `es_brainstorm_record` — one call per lens. Caps are enforced; near-duplicates are collapsed against earlier ideas (convergence is a signal).
|
||||
4. `es-brainstorm-critic` — scores every surviving idea 1-5 on novelty, upside, feasibility and fit via `es_brainstorm_score`.
|
||||
5. `es_brainstorm_complete` — re-checks coverage and scores, ranks, and writes the diversified shortlist with one forced outlier slot to `.factory/brainstorm/`.
|
||||
5. `es_brainstorm_complete` — re-checks coverage and scores, ranks, and writes the diversified shortlist with one forced outlier slot (whenever the shortlist has ≥2 slots) to `.factory/brainstorm/`.
|
||||
|
||||
## Output
|
||||
`.factory/brainstorm/brainstorm.json` (machine-readable) and `BRAINSTORM.md` (the shortlist, the full ranked field, collapsed duplicates, coverage and gaps). Report the shortlist to the human; do not paste the whole field.
|
||||
|
||||
@@ -177,7 +177,8 @@ export const AGENTS: readonly AgentSpec[] = [
|
||||
mode: "primary",
|
||||
hidden: false,
|
||||
tier: "deep",
|
||||
description: "Fans out divergent lenses on an idea and returns a diversified shortlist.",
|
||||
description:
|
||||
"Fans out divergent lenses (eight-lens registry; six by default, four lean for callable runs) on an idea and distills a diversified shortlist.",
|
||||
prompt: "brainstormer",
|
||||
tools: [
|
||||
"es_status",
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
// Approval attempt limiting (ADR 0002 I4): at most 5 wrong passphrases per
|
||||
// 10 minutes per state dir, then a lockout. Lockouts are audited into the
|
||||
// project's audit log. One surface's lockout does not affect the others.
|
||||
// project's audit log. The CLI and the TUI share one attempt file per state
|
||||
// dir (a single bucket for both); the web path keeps its own dir and bucket.
|
||||
import path from "node:path"
|
||||
import { appendAuditEntry } from "../audit/chain.ts"
|
||||
import { readJson, withLock, writeJson } from "../util/fs.ts"
|
||||
|
||||
@@ -103,7 +103,13 @@ async function readText(file: string): Promise<string | undefined> {
|
||||
|
||||
/** Whether this machine holds a sealed human key (`es key seal` has run). */
|
||||
export async function hasHumanKey(dir = stateDir()): Promise<boolean> {
|
||||
return (await readText(path.join(dir, SEALED_FILE))) !== undefined
|
||||
// Both halves are required (#213): a lone human.key (or human.pub) is a
|
||||
// half-deleted brick, and unlock would refuse it as tampered or partial.
|
||||
const [sealed, pub] = await Promise.all([
|
||||
readText(path.join(dir, SEALED_FILE)),
|
||||
readText(path.join(dir, PUBLIC_FILE)),
|
||||
])
|
||||
return sealed !== undefined && pub !== undefined
|
||||
}
|
||||
|
||||
async function publicKey(dir: string): Promise<KeyObject | undefined> {
|
||||
|
||||
@@ -240,7 +240,9 @@ export const OUTLIER_THRESHOLD = 0.35
|
||||
* Greedy diversified selection: quality plus distance from what is already
|
||||
* picked. The last slot is forced to an outlier — the best idea whose maximum
|
||||
* similarity to the shortlist is below the outlier threshold, or the most
|
||||
* dissimilar idea when none is far enough.
|
||||
* dissimilar idea when none is far enough — whenever the shortlist has ≥2
|
||||
* slots (and at least two ideas are ranked; a single-idea shortlist carries
|
||||
* `outlier: false`).
|
||||
*/
|
||||
export function selectShortlist(
|
||||
ranked: readonly RankedIdea[],
|
||||
|
||||
@@ -113,7 +113,8 @@ export function brainstormTools(context: BrainstormOpsContext): EsToolDef[] {
|
||||
ideasPerLens: { type: "number", description: "Ideas per lens (1-10, default 5)." },
|
||||
shortlistSize: {
|
||||
type: "number",
|
||||
description: "Shortlist size including the forced outlier (2-12, default 7).",
|
||||
description:
|
||||
"Shortlist size (2-12, default 7); the last slot is a forced outlier whenever the shortlist has ≥2 slots.",
|
||||
},
|
||||
dedupe: {
|
||||
type: "string",
|
||||
@@ -176,7 +177,7 @@ export function brainstormTools(context: BrainstormOpsContext): EsToolDef[] {
|
||||
" Give each one the brief (and constraints) plus how many ideas you expect.",
|
||||
`2. Record each lens's ideas with es_brainstorm_record (one call per lens, run "${run}").`,
|
||||
"3. Launch es-brainstorm-critic with the surviving idea ids and texts; it scores each with es_brainstorm_score.",
|
||||
`4. Call es_brainstorm_complete with run "${run}". Dedupe, ranking and the shortlist (with one forced outlier) are computed there.`,
|
||||
`4. Call es_brainstorm_complete with run "${run}". Dedupe, ranking and the shortlist (with one forced outlier whenever the shortlist has ≥2 slots) are computed there.`,
|
||||
"",
|
||||
`Rubric (1-5 each): ${RUBRIC.map((item) => item.id).join(", ")}.`,
|
||||
].join("\n")
|
||||
@@ -344,7 +345,7 @@ export function brainstormTools(context: BrainstormOpsContext): EsToolDef[] {
|
||||
{
|
||||
name: "es_brainstorm_complete",
|
||||
description:
|
||||
"Brainstormer, grill or factory: collapse duplicates, rank the scored ideas and write the diversified shortlist with a forced outlier slot. Returns the shortlist as JSON the caller can parse.",
|
||||
"Brainstormer, grill or factory: collapse duplicates, rank the scored ideas and write the diversified shortlist with a forced outlier slot (whenever the shortlist has ≥2 slots). Returns the shortlist as JSON the caller can parse.",
|
||||
input: object({
|
||||
run: { type: "string", description: "Run id from es_brainstorm_plan." },
|
||||
allowPartial: {
|
||||
|
||||
@@ -97,7 +97,8 @@ export const CAPABILITY_MATRIX: Readonly<Record<Harness, readonly CapabilityRow[
|
||||
},
|
||||
lsp: {
|
||||
support: "enforced",
|
||||
detail: "The lsp config key is read and served by our own manager (v2 accepts but does not run it).",
|
||||
detail:
|
||||
"The `lsp` key in v2's `opencode.json` — not an ES config key (ES layers strict-reject it) — is read and served by our own manager (v2 accepts but does not run it).",
|
||||
test: "packages/opencode/test/lsp.test.ts",
|
||||
},
|
||||
panels: {
|
||||
@@ -115,7 +116,7 @@ export const CAPABILITY_MATRIX: Readonly<Record<Harness, readonly CapabilityRow[
|
||||
brainstorm: {
|
||||
support: "enforced",
|
||||
detail:
|
||||
"Lens fan-out, record/score/complete tools, dedupe, rubric, shortlist with a forced outlier. Runs are id-scoped (the human's /brainstorm is the default run); the grill and the factory run callable brainstorms at depth 1 and get the shortlist as JSON.",
|
||||
"Lens fan-out, record/score/complete tools, dedupe, rubric, shortlist with a forced outlier whenever the shortlist has ≥2 slots. Runs are id-scoped (the human's /brainstorm is the default run); the grill and the factory run callable brainstorms at depth 1 and get the shortlist as JSON.",
|
||||
test: "packages/opencode/test/brainstorm.test.ts",
|
||||
},
|
||||
harvest: {
|
||||
@@ -145,8 +146,8 @@ export const CAPABILITY_MATRIX: Readonly<Record<Harness, readonly CapabilityRow[
|
||||
scout: {
|
||||
support: "enforced",
|
||||
detail:
|
||||
"OSS scout: cached-only web under an offline (--unshare-net) sandbox, fail-closed license verdicts computed by core (adopt only for verified permissive licenses), OSV advisories cited.",
|
||||
test: "packages/opencode/test/fires.test.ts",
|
||||
"OSS scout: cached-only web under an offline (--unshare-net) sandbox, fail-closed license verdicts computed by core (adopt only for verified permissive licenses), OSV advisories cited (canned-OSV content in core's scout suite).",
|
||||
test: "packages/opencode/test/scout-host.test.ts",
|
||||
},
|
||||
"research-fires": {
|
||||
support: "enforced",
|
||||
|
||||
@@ -182,7 +182,8 @@ export async function verifyBrief(
|
||||
`signed by key ${brief.signature.keyId}, which this machine does not hold; only integrity can be checked (allow unverifiable to accept that)`,
|
||||
)
|
||||
|
||||
// 4. Quotes: every claim the brief calls witnessed must quote its bundled source verbatim.
|
||||
// 4. Quotes: every claim the brief calls witnessed must quote its bundled
|
||||
// source verbatim after documented normalisation (research/quote.ts).
|
||||
let quotesVerified = 0
|
||||
let quotesFailed = 0
|
||||
for (const claim of brief.claims as Claim[]) {
|
||||
|
||||
@@ -34,15 +34,25 @@ export const LEGACY_CONSTITUTION: Constitution = {
|
||||
retractionPolicy: "standard",
|
||||
}
|
||||
|
||||
export const constitutionFromPack = (pack: DomainPack): Constitution => ({
|
||||
tierWeights: { __default__: 1, ...pack.tierWeights },
|
||||
negBonus: pack.negBonus,
|
||||
rejectPenalty: pack.rejectPenalty,
|
||||
acceptThreshold: pack.acceptThreshold,
|
||||
bannedDomains: pack.bannedDomains,
|
||||
mandatoryTags: pack.mandatoryTags,
|
||||
retractionPolicy: pack.retractionPolicy,
|
||||
})
|
||||
export const constitutionFromPack = (pack: DomainPack): Constitution => {
|
||||
// Tierless claims weigh at or below the lowest verified tier (#252): an
|
||||
// unclassifiable host falling back to a full 1.0 outscores verified
|
||||
// preprints and rewards citing worse sources.
|
||||
const tiered = Object.entries(pack.tierWeights)
|
||||
.filter(([tier]) => tier !== "__default__")
|
||||
.map(([, weight]) => weight)
|
||||
const floor = tiered.length ? Math.min(...tiered) : 1
|
||||
const fallback = pack.tierWeights.__default__ ?? 1
|
||||
return {
|
||||
tierWeights: { ...pack.tierWeights, __default__: Math.min(fallback, floor) },
|
||||
negBonus: pack.negBonus,
|
||||
rejectPenalty: pack.rejectPenalty,
|
||||
acceptThreshold: pack.acceptThreshold,
|
||||
bannedDomains: pack.bannedDomains,
|
||||
mandatoryTags: pack.mandatoryTags,
|
||||
retractionPolicy: pack.retractionPolicy,
|
||||
}
|
||||
}
|
||||
|
||||
export const weightFor = (constitution: Constitution, tier: string | undefined): number =>
|
||||
(tier !== undefined && constitution.tierWeights[tier] !== undefined
|
||||
@@ -131,7 +141,7 @@ export interface ClaimVerdictOptions {
|
||||
/**
|
||||
* The per-claim ACCEPTED/REJECTED gate used by domain packs. A claim is
|
||||
* REJECTED when the legacy structural rules fire (VERIFIED needs its source
|
||||
* hash + verbatim quote or a code location; INFERRED needs its reasoning;
|
||||
* hash + quote (verbatim after documented normalisation) or a code location; INFERRED needs its reasoning;
|
||||
* HYPOTHESIS needs its falsification; NEGATIVE_KNOWLEDGE needs query and
|
||||
* finding), when any pack rule fires (parent unresolved, missing mandatory
|
||||
* tag, banned domain), or when the retraction policy is zero_tolerance and
|
||||
@@ -165,7 +175,8 @@ export function claimVerdict(claim: ConstitutionClaim, options: ClaimVerdictOpti
|
||||
if (!nonEmpty(claim.query)) reasons.push("NEG_KNOWLEDGE_REQUIRES_QUERY: query missing")
|
||||
if (!nonEmpty(claim.finding)) reasons.push("NEG_KNOWLEDGE_REQUIRES_FINDING: finding missing")
|
||||
}
|
||||
if (constitution.mandatoryTags.length && !claim.tag) reasons.push("MISSING_TAG: tag required by constitution")
|
||||
if (constitution.mandatoryTags.length && !constitution.mandatoryTags.includes(claim.tag ?? ""))
|
||||
reasons.push(`MISSING_TAG: tag '${claim.tag ?? ""}' is not one of the constitution's mandatory tags`)
|
||||
const url = claim.source?.url
|
||||
if (constitution.bannedDomains.length && typeof url === "string" && url) {
|
||||
const lower = url.toLowerCase()
|
||||
@@ -199,7 +210,8 @@ export interface ScoreBreakdown {
|
||||
* Verified claims contribute their tier weight when the constitution weights
|
||||
* any tier away from 1.0 (otherwise a flat 1.0, preserving legacy numbers);
|
||||
* rejected claims cost `rejectPenalty`; counted negative knowledge earns
|
||||
* `negBonus`. Valid, deduplicated NK rows only, via the M1 port of the legacy
|
||||
* `negBonus` and counts as assertions, so stuffing NK rows cannot inflate the
|
||||
* score past the threshold (#252). Valid, deduplicated NK rows only, via the M1 port of the legacy
|
||||
* normalized key. The score is clamped to [0, 1] and rounded to 3 places.
|
||||
*/
|
||||
export function computeEpistemicScoreFromClaims(
|
||||
@@ -238,7 +250,10 @@ export function computeEpistemicScoreFromClaims(
|
||||
} else if (claim.tag === "INFERRED") counts.inferred++
|
||||
else if (claim.tag === "HYPOTHESIS") counts.hypotheses++
|
||||
}
|
||||
const totalAssertions = Math.max(1, counts.verified + counts.inferred + counts.hypotheses + counts.rejected)
|
||||
const totalAssertions = Math.max(
|
||||
1,
|
||||
counts.verified + counts.inferred + counts.hypotheses + counts.rejected + negKnowledge,
|
||||
)
|
||||
const rawScore =
|
||||
(weightSum + constitution.negBonus * negKnowledge - constitution.rejectPenalty * counts.rejected) / totalAssertions
|
||||
const score = Math.max(0, Math.min(1, Math.round(rawScore * 1000) / 1000))
|
||||
|
||||
@@ -4,7 +4,8 @@
|
||||
// refusing get + the one quote law, research/quote.ts);
|
||||
// - a code location against the file on disk: inside the target root, the
|
||||
// line range within the file and at most MAX_LOCATION_LINES long, the
|
||||
// excerpt one contiguous verbatim span of those lines.
|
||||
// excerpt one contiguous span of those lines, verbatim after documented
|
||||
// normalisation.
|
||||
// Legacy never checked code locations at all (its "hallucinated line" check
|
||||
// was the generic quote check), so the location rule is new in 1.1.
|
||||
import path from "node:path"
|
||||
|
||||
@@ -48,7 +48,7 @@ export interface EvalTranscript {
|
||||
readonly tokens: EvalTokens
|
||||
}
|
||||
|
||||
const num = (value: unknown): number => (typeof value === "number" && Number.isFinite(value) ? value : 0)
|
||||
const num = (value: unknown): number => (typeof value === "number" && Number.isFinite(value) && value >= 0 ? value : 0)
|
||||
|
||||
/**
|
||||
* Tools whose outputs are raw evidence, not the seat's own words. Quoting
|
||||
|
||||
@@ -41,7 +41,8 @@ export const DEFAULT_CASE_STEPS = 6
|
||||
* makes one call per step), bounded by the run-wide ceiling.
|
||||
*/
|
||||
export function caseStepCap(caseMaxSteps: number | undefined, ceiling: number): number {
|
||||
return Math.min(caseMaxSteps ?? DEFAULT_CASE_STEPS, ceiling)
|
||||
const own = typeof caseMaxSteps === "number" && Number.isFinite(caseMaxSteps) ? caseMaxSteps : DEFAULT_CASE_STEPS
|
||||
return Math.min(own, ceiling)
|
||||
}
|
||||
|
||||
export interface EvalWorkspace {
|
||||
@@ -140,6 +141,11 @@ export async function runEvalCase(
|
||||
workspace: EvalWorkspace,
|
||||
options: EvalRunOptions,
|
||||
): Promise<EvalResult> {
|
||||
// Fail closed on flag injection (#211): the prompt is a positional argv, so
|
||||
// a leading dash would parse as a CLI flag. Cases are repo files, but the
|
||||
// runner must not depend on their goodwill.
|
||||
if (testCase.prompt.startsWith("-"))
|
||||
throw new Error(`Refusing eval case "${testCase.id}": the prompt must not start with "-".`)
|
||||
const child = spawn(
|
||||
options.binary,
|
||||
["run", "--standalone", "--format", "json", "--agent", testCase.agent, testCase.prompt],
|
||||
@@ -167,7 +173,8 @@ export async function runEvalCase(
|
||||
child.kill("SIGKILL")
|
||||
}
|
||||
if (event.type === "step_finish" && typeof event.part?.cost === "number" && Number.isFinite(event.part.cost)) {
|
||||
spent += event.part.cost
|
||||
// Clamp hostile negatives: a negative cost must never unbind spend (#211).
|
||||
spent += Math.max(0, event.part.cost)
|
||||
if (options.budgetUSD !== undefined && spent > options.budgetUSD && !overBudget) {
|
||||
overBudget = true
|
||||
child.kill("SIGKILL")
|
||||
|
||||
@@ -319,7 +319,10 @@ export class Factory {
|
||||
// (CI-only es_build_start refusal). Real transitions still enforce it.
|
||||
if (!options.skipControl) {
|
||||
const control = await verifyControl(this.root)
|
||||
if (!control.clean) return halt(`control files changed outside a human action: ${control.violations.join("; ")}`)
|
||||
if (!control.clean)
|
||||
return halt(
|
||||
`control files changed outside a human action (integrity/control-drift): ${control.violations.join("; ")}. The structured finding is in .factory/runs/${state.runId}/gates/*/summary.json; a human must review and rebaseline.`,
|
||||
)
|
||||
}
|
||||
if (state.spendCeilingUSD !== undefined && state.spend.usd >= state.spendCeilingUSD)
|
||||
return halt(
|
||||
@@ -433,6 +436,10 @@ export class Factory {
|
||||
this.requireAuditsClear(state)
|
||||
const check = await verifyApproval(this.root, "frontier", { stateDir: this.deps.stateDir })
|
||||
if (!check.ok) throw new FactoryError(check.reason)
|
||||
if (check.record.spendCeilingUSD === undefined)
|
||||
throw new FactoryError(
|
||||
"The frontier approval carries no spend ceiling; a human must approve it again (`es approve frontier`).",
|
||||
)
|
||||
state.spendCeilingUSD = check.record.spendCeilingUSD
|
||||
state.stage = "RESEARCH"
|
||||
await this.audit(actor, "stage.research", { spendCeilingUSD: state.spendCeilingUSD })
|
||||
@@ -684,7 +691,8 @@ export class Factory {
|
||||
/** Record a gate run made during BUILD (red test runs are the failing-test-first evidence). */
|
||||
async recordGateRun(agentId: string | undefined, result: GateRunLike): Promise<void> {
|
||||
await this.mutate(`agent:${agentId}`, (state) => {
|
||||
if (state.stage !== "BUILD") return
|
||||
if (state.stage !== "BUILD")
|
||||
throw new FactoryError(`Gate runs record during BUILD only (current stage: ${state.stage}).`)
|
||||
const phase = this.active(state)
|
||||
const failed = result.failedTests ?? []
|
||||
const red = failed.length > 0 || result.testsFailed === true
|
||||
@@ -1233,7 +1241,7 @@ export class Factory {
|
||||
private async prBody(state: FactoryState) {
|
||||
const spec = await readApproval(this.root, "spec")
|
||||
const lines = [
|
||||
`Built by the Epistemic Swarm factory (run \`${state.runId}\`). A human reviews and merges; the factory never merges.`,
|
||||
`Built by the Epistemic Swarm factory (run \`${state.runId}\`). A human reviews and merges; the factory never merges into the release/base branch.`,
|
||||
"",
|
||||
"## Phases",
|
||||
...state.phases.map(
|
||||
|
||||
@@ -31,7 +31,7 @@ import { validateArtifacts } from "./artifacts.ts"
|
||||
import { checkBudgets, isTestFile } from "./budgets.ts"
|
||||
import { detectGates } from "./detect.ts"
|
||||
import { parseOutput } from "./parsers.ts"
|
||||
import { scanForSecrets } from "./secrets.ts"
|
||||
import { scanFileForSecrets } from "./secrets.ts"
|
||||
import { DEPENDENCY_FILES, gitleaks, osvFindings } from "./security.ts"
|
||||
import { isWaived, loadWaivers } from "./waivers.ts"
|
||||
|
||||
@@ -402,10 +402,7 @@ export async function runGates(options: RunGatesOptions): Promise<GateReport> {
|
||||
if (config.security.secrets)
|
||||
await timed("secrets", async () => {
|
||||
const out: GateFinding[] = []
|
||||
for (const file of present) {
|
||||
const text = await readFile(path.join(dir, file), "utf8").catch(() => "")
|
||||
if (text.length <= 2_000_000) out.push(...scanForSecrets(text, file))
|
||||
}
|
||||
for (const file of present) out.push(...(await scanFileForSecrets(dir, file)))
|
||||
return out
|
||||
})
|
||||
if (config.security.gitleaks === "auto") await timed("gitleaks", () => gitleaks(dir, present.slice(0, 200)))
|
||||
|
||||
@@ -1,5 +1,7 @@
|
||||
// Built-in secret scanner (always on, no external binary). High-precision
|
||||
// patterns are errors; the generic assignment heuristic is a warning.
|
||||
import { open } from "node:fs/promises"
|
||||
import path from "node:path"
|
||||
import type { GateFinding } from "../schema/gates.ts"
|
||||
|
||||
const PATTERNS: ReadonlyArray<{ rule: string; regex: RegExp; description: string; severity?: "warning" }> = [
|
||||
@@ -56,3 +58,48 @@ export function scanForSecrets(content: string, file: string): GateFinding[] {
|
||||
})
|
||||
return findings
|
||||
}
|
||||
|
||||
// Per-path cache keyed by mtime+size (mirrors structure/graph.ts), so repeated
|
||||
// gate runs re-read only changed files. The scan itself is deliberately NOT
|
||||
// count-capped the way gitleaks is: a cap would fail open, silently skipping a
|
||||
// secret past the cap, while the cache bounds the repeated-run cost instead.
|
||||
const fileCache = new Map<string, { mtimeMs: number; size: number; findings: GateFinding[] }>()
|
||||
let cacheHits = 0
|
||||
|
||||
/**
|
||||
* Scan one file for secrets, cached by mtime+size. Missing, non-regular and
|
||||
* oversized (>2MB, as before) files yield no findings, as do binary files.
|
||||
*/
|
||||
export async function scanFileForSecrets(dir: string, file: string): Promise<GateFinding[]> {
|
||||
const absolute = path.join(dir, file)
|
||||
// Open once and stat the descriptor, so the size/mtime check and the read
|
||||
// cannot race with a concurrent replacement of the path.
|
||||
const handle = await open(absolute, "r").catch(() => undefined)
|
||||
if (!handle) return []
|
||||
try {
|
||||
const info = await handle.stat()
|
||||
if (!info.isFile() || info.size > 2_000_000) return []
|
||||
const cached = fileCache.get(absolute)
|
||||
if (cached && cached.mtimeMs === info.mtimeMs && cached.size === info.size) {
|
||||
cacheHits += 1
|
||||
return cached.findings
|
||||
}
|
||||
const findings = scanForSecrets(await handle.readFile("utf8"), file)
|
||||
if (fileCache.size > 50_000) fileCache.clear()
|
||||
fileCache.set(absolute, { mtimeMs: info.mtimeMs, size: info.size, findings })
|
||||
return findings
|
||||
} finally {
|
||||
await handle.close()
|
||||
}
|
||||
}
|
||||
|
||||
/** Cache observability: entries held and hits served since the last clear. */
|
||||
export function secretScanStats(): { entries: number; hits: number } {
|
||||
return { entries: fileCache.size, hits: cacheHits }
|
||||
}
|
||||
|
||||
/** For tests: forget cached findings and reset the hit count. */
|
||||
export function clearSecretScanCache(): void {
|
||||
fileCache.clear()
|
||||
cacheHits = 0
|
||||
}
|
||||
|
||||
@@ -58,6 +58,27 @@ export interface MatrixBuild {
|
||||
}>
|
||||
}
|
||||
|
||||
/**
|
||||
* Scrub model-supplied evidence before it reaches a clean-room spec (#267).
|
||||
* The "do not read or copy" line in the spec template is procedure, not
|
||||
* enforcement: without this, a harvester that pastes upstream code into
|
||||
* `evidence` contaminates the spec mechanically. Fenced code blocks go
|
||||
* entirely, inline spans keep their text but lose their ticks, stray ticks
|
||||
* go, whitespace collapses, and the note is capped.
|
||||
*/
|
||||
export const EVIDENCE_MAX = 500
|
||||
export function sanitizeEvidence(evidence: string): string | undefined {
|
||||
const text = evidence
|
||||
.replace(/```[\s\S]*?(```|$)/g, " ")
|
||||
.replace(/`([^`\n]*)`/g, "$1")
|
||||
.replace(/`/g, "")
|
||||
.replace(/\s+/g, " ")
|
||||
.trim()
|
||||
.slice(0, EVIDENCE_MAX)
|
||||
.trimEnd()
|
||||
return text || undefined
|
||||
}
|
||||
|
||||
/** Apply the policy to every proposed cell and freeze the matrix. */
|
||||
export function buildMatrix(
|
||||
candidates: readonly string[],
|
||||
@@ -74,6 +95,7 @@ export function buildMatrix(
|
||||
if (!profile)
|
||||
return { ...cell, verdict: "skip", downgraded: `candidate "${cell.candidate}" was not scanned; skipped` }
|
||||
const checked = checkVerdict(cell.verdict, profile.license, whitelist)
|
||||
const clean = cell.evidence !== undefined ? sanitizeEvidence(cell.evidence) : undefined
|
||||
if (checked.downgraded)
|
||||
downgrades.push({
|
||||
candidate: cell.candidate,
|
||||
@@ -85,7 +107,7 @@ export function buildMatrix(
|
||||
return {
|
||||
candidate: cell.candidate,
|
||||
verdict: checked.verdict,
|
||||
...(cell.evidence ? { evidence: cell.evidence } : {}),
|
||||
...(clean !== undefined ? { evidence: clean } : {}),
|
||||
...(checked.attribution ? { attribution: checked.attribution } : {}),
|
||||
...(checked.downgraded ? { downgraded: checked.downgraded } : {}),
|
||||
}
|
||||
|
||||
@@ -4,7 +4,11 @@
|
||||
// bundled SPDX template (templates.ts): extra or changed terms — a rider like
|
||||
// the Commons Clause, the JSON licence's "Good, not Evil" — mean "unknown".
|
||||
// - Copyleft and source-available licences are recognised by their title and
|
||||
// date lines (GPL-3.0 is not mistaken for AGPL because §13 mentions it).
|
||||
// date lines (the earliest GPL-family title in the bytes decides among the
|
||||
// GPL family: the GPL-3.0 tail names the Lesser GPL, which must not outrank
|
||||
// an earlier GPL title).
|
||||
// - NOTICE/PATENTS files are not evaluated as licences: a top-level NOTICE
|
||||
// without a licence file warns about the blind spot instead of verifying.
|
||||
// - Every licence file in the root counts (LICENSE*, COPYING*, UNLICENSE…);
|
||||
// the most restrictive wins, and several permissive ones combine with AND.
|
||||
// - `SPDX-License-Identifier` headers are declarations by single files: they
|
||||
@@ -28,7 +32,8 @@ const FAMILIES: Readonly<Record<string, Family>> = {
|
||||
"Apache-2.0": "permissive",
|
||||
"BSD-2-Clause": "permissive",
|
||||
"BSD-3-Clause": "permissive",
|
||||
"BSD-4-Clause": "permissive",
|
||||
// No BSD-4-Clause: the advertising clause is an obligation the whitelist
|
||||
// does not cover, so it must read as unknown, never as safe (#264).
|
||||
ISC: "permissive",
|
||||
"0BSD": "permissive",
|
||||
Unlicense: "public-domain",
|
||||
@@ -145,6 +150,24 @@ interface Rule {
|
||||
// Lesser GPL and GPL-3.0 §13 the Affero GPL. Date lines catch texts that sit
|
||||
// under a project notice.
|
||||
const has = (text: string, ...phrases: string[]) => phrases.every((phrase) => text.includes(phrase))
|
||||
|
||||
/**
|
||||
* The Lesser title plus a date line, but only when the Lesser title is the
|
||||
* earliest GPL-family title in the bytes (#264): the GPL-3.0 tail ("use the
|
||||
* GNU Lesser General Public License instead of this License") names it, and
|
||||
* a later mention must not outrank an earlier GPL title.
|
||||
*/
|
||||
const lesserFirst = (body: string, date: string): boolean => {
|
||||
if (!has(body, "gnu lesser general public license", date)) return false
|
||||
const lesser = body.indexOf("gnu lesser general public license")
|
||||
for (const title of [
|
||||
"gnu affero general public license",
|
||||
"gnu library general public license",
|
||||
"gnu general public license",
|
||||
])
|
||||
if (body.includes(title) && body.indexOf(title) < lesser) return false
|
||||
return true
|
||||
}
|
||||
const title = (phrase: string, spdx: string | ((head: string) => string), note?: string): Rule => ({
|
||||
test: (head) => (head.includes(phrase) ? (typeof spdx === "string" ? spdx : spdx(head)) : undefined),
|
||||
...(note ? { note } : {}),
|
||||
@@ -171,8 +194,8 @@ const RESTRICTIVE: readonly Rule[] = [
|
||||
title("elastic license", "Elastic-2.0", SOURCE_AVAILABLE),
|
||||
// 2. Texts under a project notice: their distinctive date lines.
|
||||
{ test: (_, body) => (has(body, "gnu affero general public license", "19 november 2007") ? "AGPL-3.0" : undefined) },
|
||||
{ test: (_, body) => (has(body, "gnu lesser general public license", "february 1999") ? "LGPL-2.1" : undefined) },
|
||||
{ test: (_, body) => (has(body, "gnu lesser general public license", "29 june 2007") ? "LGPL-3.0" : undefined) },
|
||||
{ test: (_, body) => (lesserFirst(body, "february 1999") ? "LGPL-2.1" : undefined) },
|
||||
{ test: (_, body) => (lesserFirst(body, "29 june 2007") ? "LGPL-3.0" : undefined) },
|
||||
{ test: (_, body) => (has(body, "gnu general public license", "29 june 2007") ? "GPL-3.0" : undefined) },
|
||||
{ test: (_, body) => (has(body, "gnu general public license", "june 1991") ? "GPL-2.0" : undefined) },
|
||||
{ test: (_, body) => (has(body, "mozilla public license", "version 2.0") ? "MPL-2.0" : undefined) },
|
||||
@@ -271,6 +294,17 @@ export async function detectLicense(
|
||||
|
||||
// 1. Every licence file in the root.
|
||||
const entries = await readdirImpl(dir).catch(() => [] as LicenseScanEntry[])
|
||||
// NOTICE/PATENTS files are not evaluated as licences (#264): when no
|
||||
// licence file matched, their presence is reported on the finding so the
|
||||
// blind spot is visible instead of silent.
|
||||
const noticeFiles = entries
|
||||
.filter((item) => !item.directory && /^(notice|patents)([-._][\w.-]+)?$/i.test(item.name))
|
||||
.map((item) => item.name)
|
||||
.sort(byText)
|
||||
const noticed = (note: string): string =>
|
||||
noticeFiles.length
|
||||
? `${note}; ${noticeFiles.join(", ")} present but not evaluated as a license (NOTICE obligations are a blind spot)`
|
||||
: note
|
||||
const files: FileVerdict[] = []
|
||||
for (const entry of entries
|
||||
.filter((item) => !item.directory && LICENSE_FILE.test(item.name))
|
||||
@@ -302,7 +336,7 @@ export async function detectLicense(
|
||||
sha256: sha256(headers[0]!.text),
|
||||
confidence: "medium",
|
||||
verified: false,
|
||||
note: `declared only by SPDX headers (${headers.length} file(s)); no licence text found`,
|
||||
note: noticed(`declared only by SPDX headers (${headers.length} file(s)); no licence text found`),
|
||||
}
|
||||
}
|
||||
|
||||
@@ -316,7 +350,7 @@ export async function detectLicense(
|
||||
file: manifest.file,
|
||||
confidence: "medium",
|
||||
verified: false,
|
||||
note: "self-reported in a manifest; no license text found",
|
||||
note: noticed("self-reported in a manifest; no license text found"),
|
||||
}
|
||||
|
||||
return {
|
||||
@@ -325,7 +359,7 @@ export async function detectLicense(
|
||||
source: "manifest",
|
||||
confidence: "low",
|
||||
verified: false,
|
||||
note: "no license file, header or manifest field found",
|
||||
note: noticed("no license file, header or manifest field found"),
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -6,6 +6,7 @@
|
||||
// missing trust; observers fail open, as Claude Code does. Every executed
|
||||
// handler is written to the audit log.
|
||||
import { appendAuditEntry } from "../audit/chain.ts"
|
||||
import type { SandboxSpec } from "../trust/sandbox.ts"
|
||||
import { isTrusted } from "../trust/store.ts"
|
||||
import { type HookSource, hookSources, type LoadedHooks, loadHooks } from "./load.ts"
|
||||
import { ifMatches, matcherMatches } from "./match.ts"
|
||||
@@ -19,6 +20,16 @@ export interface HookEngineOptions {
|
||||
readonly sources?: HookSource[]
|
||||
readonly generate?: (prompt: string, model?: string) => Promise<string>
|
||||
readonly audit?: boolean
|
||||
/**
|
||||
* Sandbox for hook command subprocesses (#265). Absent runs unsandboxed
|
||||
* with the parent environment (Claude Code parity; project hooks are
|
||||
* trusted code). "readonly" makes everything read-only; a spec runs with
|
||||
* one writable directory. Without bwrap the run errors and gates fail
|
||||
* closed while observers fail open.
|
||||
*/
|
||||
readonly sandbox?: "none" | "readonly" | Omit<SandboxSpec, "cwd">
|
||||
/** No network inside the sandbox (only meaningful with `sandbox`). */
|
||||
readonly offline?: boolean
|
||||
}
|
||||
|
||||
export interface BaseInput {
|
||||
@@ -188,6 +199,15 @@ export class HookEngine {
|
||||
}
|
||||
|
||||
private async execute(event: string, items: SourcedHandler[], input: Record<string, unknown>): Promise<Ran[]> {
|
||||
// The sandbox knob rides every handler run (#265); per-handler timeout
|
||||
// and x-es-offline are honored in run.ts.
|
||||
const sandboxContext =
|
||||
this.options.sandbox !== undefined || this.options.offline !== undefined
|
||||
? {
|
||||
...(this.options.sandbox !== undefined ? { sandbox: this.options.sandbox } : {}),
|
||||
...(this.options.offline !== undefined ? { offline: this.options.offline } : {}),
|
||||
}
|
||||
: {}
|
||||
const ran = await Promise.all(
|
||||
items.map(async (item): Promise<Ran> => {
|
||||
const project = !item.source.startsWith("user:")
|
||||
@@ -205,6 +225,7 @@ export class HookEngine {
|
||||
root: this.options.root,
|
||||
event,
|
||||
input,
|
||||
...sandboxContext,
|
||||
...(this.options.generate ? { generate: this.options.generate } : {}),
|
||||
})
|
||||
return { item, outcome: { status: "ok", durationMs: 0 } }
|
||||
@@ -213,6 +234,7 @@ export class HookEngine {
|
||||
root: this.options.root,
|
||||
event,
|
||||
input,
|
||||
...sandboxContext,
|
||||
...(this.options.generate ? { generate: this.options.generate } : {}),
|
||||
})
|
||||
return { item, outcome }
|
||||
@@ -388,7 +410,17 @@ export class HookEngine {
|
||||
): Promise<{ additionalContext: string[]; notices: string[] }> {
|
||||
const items = this.select("SessionStart", [source])
|
||||
const result = { additionalContext: [] as string[], notices: [] as string[] }
|
||||
if (!items.length) return result
|
||||
if (!items.length) {
|
||||
// An unmatched source is an explicit notice, never a silent no-op
|
||||
// (#289): the bridge fires only startup + compact, so a hook filtered
|
||||
// to resume/clear/fork would otherwise sit inert with no sign. Silent
|
||||
// only when no SessionStart hook is registered at all.
|
||||
if (this.loaded.handlers.some((item) => item.event === "SessionStart"))
|
||||
result.notices.push(
|
||||
`SessionStart: hook(s) registered but none match source "${source}" (the bridge fires startup + compact)`,
|
||||
)
|
||||
return result
|
||||
}
|
||||
const ran = await this.execute("SessionStart", items, { ...this.common("SessionStart", base), source })
|
||||
for (const { item, outcome } of ran) {
|
||||
this.failure(item, outcome, result.notices)
|
||||
|
||||
@@ -6,6 +6,15 @@
|
||||
// - http: POST the input; 2xx JSON body decides, anything else is non-blocking.
|
||||
// - prompt: evaluated through an injected model call; {ok, reason, impossible}.
|
||||
// - mcp_tool and agent: not available in the bridge (reported as skipped).
|
||||
//
|
||||
// Sandboxing (#265): command subprocesses run with the parent environment,
|
||||
// unsandboxed, by design (Claude Code parity) — hook commands are trusted
|
||||
// code (project hooks need a human `es trust` before they run at all), so the
|
||||
// parent env, secrets included, is visible to them. Callers that need
|
||||
// isolation pass `sandbox`/`offline` in the RunContext (honored per handler:
|
||||
// `x-es-offline` overrides the context default); without bwrap the sandboxed
|
||||
// run fails and the engine's gate/observer policy decides (fail closed/open).
|
||||
import type { SandboxSpec } from "../trust/sandbox.ts"
|
||||
import { type RunResult, run } from "../util/proc.ts"
|
||||
import type { HookHandler } from "./spec.ts"
|
||||
|
||||
@@ -30,6 +39,15 @@ export interface RunContext {
|
||||
readonly signal?: AbortSignal
|
||||
/** Model call for prompt hooks; absent means prompt hooks are skipped. */
|
||||
readonly generate?: (prompt: string, model?: string) => Promise<string>
|
||||
/**
|
||||
* Sandbox for command subprocesses (#265). Absent (the default) runs
|
||||
* unsandboxed with the parent environment, as Claude Code does. "readonly"
|
||||
* makes everything read-only; a spec runs with one writable directory.
|
||||
* Throws inside the outcome (error status) when bwrap is unavailable.
|
||||
*/
|
||||
readonly sandbox?: "none" | "readonly" | Omit<SandboxSpec, "cwd">
|
||||
/** No network inside the sandbox (only meaningful with `sandbox`). */
|
||||
readonly offline?: boolean
|
||||
}
|
||||
|
||||
const DEFAULT_TIMEOUT_S: Record<string, number> = { UserPromptSubmit: 30 }
|
||||
@@ -69,6 +87,7 @@ async function runCommand(
|
||||
? [substitute(handler.command, context.root), ...handler.args.map((arg) => substitute(arg, context.root))]
|
||||
: ["sh", "-c", handler.command]
|
||||
let result: RunResult
|
||||
const offline = handler["x-es-offline"] ?? context.offline
|
||||
try {
|
||||
result = await run(argv, {
|
||||
cwd: context.root,
|
||||
@@ -77,6 +96,10 @@ async function runCommand(
|
||||
inheritEnv: true,
|
||||
env: { CLAUDE_PROJECT_DIR: context.root, ES_HOOK_EVENT: context.event },
|
||||
maxOutputBytes: 1_000_000,
|
||||
// The sandbox knob, honored per handler (#265): absent runs unsandboxed
|
||||
// with the parent environment, as Claude Code does.
|
||||
...(context.sandbox !== undefined && context.sandbox !== "none" ? { sandbox: context.sandbox } : {}),
|
||||
...(offline !== undefined ? { offline } : {}),
|
||||
...(context.signal ? { signal: context.signal } : {}),
|
||||
})
|
||||
} catch (error) {
|
||||
|
||||
@@ -35,6 +35,8 @@ export const CommandHookSchema = z.looseObject({
|
||||
async: z.boolean().optional(),
|
||||
asyncRewake: z.boolean().optional(),
|
||||
shell: z.enum(["bash", "powershell"]).optional(),
|
||||
/** No network for this command when run sandboxed (#265; needs the engine sandbox). */
|
||||
"x-es-offline": z.boolean().optional(),
|
||||
...common,
|
||||
})
|
||||
|
||||
|
||||
@@ -1,7 +1,10 @@
|
||||
// Pinned, checksummed, ask-once installs of language servers. A human
|
||||
// consents once per server (CLI or TUI); the installer fetches the exact npm
|
||||
// tarball, verifies its sha512 against the pinned integrity, and installs it
|
||||
// with scripts disabled into the user-global state dir.
|
||||
// with scripts disabled into the user-global state dir. Installs are not
|
||||
// bwrap-sandboxed (the fetch needs the network): consent + SRI pins +
|
||||
// --ignore-scripts + a scrubbed environment is the whole mitigation stack,
|
||||
// pinned by the installer tests (#275).
|
||||
import { createHash } from "node:crypto"
|
||||
import { mkdir, mkdtemp, readFile, rm } from "node:fs/promises"
|
||||
import { tmpdir, userInfo } from "node:os"
|
||||
|
||||
@@ -1,5 +1,6 @@
|
||||
// Built-in language-server registry plus OpenCode v2's `lsp` config key
|
||||
// (accepted but not run by v2 itself). User config overrides or disables
|
||||
// Built-in language-server registry plus the `lsp` key from v2's
|
||||
// `opencode.json` (accepted but not run by v2 itself; not an ES config key —
|
||||
// ES config layers strict-reject it). User config overrides or disables
|
||||
// built-ins and adds servers; `lsp: false` turns LSP off entirely.
|
||||
import { readFile } from "node:fs/promises"
|
||||
import { homedir } from "node:os"
|
||||
|
||||
@@ -16,7 +16,7 @@ import { formatReport, runGates } from "../gates/engine.ts"
|
||||
import { factoryLayout } from "../layout.ts"
|
||||
import type { GateFinding } from "../schema/gates.ts"
|
||||
import { relativeTo } from "../trust/paths.ts"
|
||||
import { writeJson } from "../util/fs.ts"
|
||||
import { withLock, writeJson } from "../util/fs.ts"
|
||||
|
||||
export type EsToolContext = { readonly agent: string; readonly sessionID?: string; readonly signal?: AbortSignal }
|
||||
export interface EsToolDef {
|
||||
@@ -121,9 +121,14 @@ export function esTools(ops: OpsContext): EsToolDef[] {
|
||||
`Not ready for approval: ${error.message}${missing ? " Run /grill (or `es factory begin` + grill flow) to record the frontier, then request approval again." : ""}`,
|
||||
)
|
||||
})
|
||||
const pending = (await pendingApprovals(root)).filter((item) => item.stage !== stage)
|
||||
pending.push({ stage, requestedBy: context.agent, at: new Date().toISOString(), ...(note ? { note } : {}) })
|
||||
await writeJson(factoryLayout(root).pending, pending)
|
||||
const pendingFile = factoryLayout(root).pending
|
||||
// Locked against concurrent requests and the human approve path
|
||||
// (which clears under the same state lock): #219 lost-update.
|
||||
await withLock(factoryLayout(root).state, async () => {
|
||||
const pending = (await pendingApprovals(root)).filter((item) => item.stage !== stage)
|
||||
pending.push({ stage, requestedBy: context.agent, at: new Date().toISOString(), ...(note ? { note } : {}) })
|
||||
await writeJson(pendingFile, pending)
|
||||
})
|
||||
await appendAuditEntry(root, {
|
||||
actor: `agent:${context.agent}`,
|
||||
action: "approval.request",
|
||||
|
||||
@@ -153,8 +153,11 @@ export async function verifyCiteGate(receipt: string, root: string): Promise<str
|
||||
if (!new RegExp(`\\[VERIFIED:\\s*${hash}\\s*\\]`, "i").test(receipt))
|
||||
errors.push(`cite-gate: receipt missing structured dossier hash [VERIFIED: ${hash}]`)
|
||||
|
||||
// QA gate: contrast probes.
|
||||
// QA gate: contrast probes. An empty probe set verifies vacuously, so
|
||||
// missing probes are an error here, not a green pass (#270).
|
||||
const probes = (await readProbes(root).catch(() => undefined)) ?? []
|
||||
if (probes.length === 0)
|
||||
errors.push("cite-gate QA contrast: probes.json is missing or empty (the contrast gate needs at least one pair)")
|
||||
for (const row of probes) for (const error of row.errors) errors.push(`cite-gate QA contrast ${row.name}: ${error}`)
|
||||
for (const row of probes)
|
||||
if (!row.pass && row.errors.length === 0) errors.push(`cite-gate QA contrast ${row.name}: pair does not pass`)
|
||||
|
||||
@@ -32,7 +32,7 @@ import {
|
||||
} from "./ledger.ts"
|
||||
import { compileToCssVars, renderGuide } from "./render.ts"
|
||||
import { AXES, type Axis, ContradictionError, FORMS, SlotLoop, SmuggledValueError, VagueAnswerError } from "./slots.ts"
|
||||
import { EXEMPLARS, renderSpecMd, specFilename, validateAll } from "./specs.ts"
|
||||
import { EXEMPLARS, renderedSlotsOfSpecMd, renderSpecMd, specFilename, validateAll } from "./specs.ts"
|
||||
import {
|
||||
checkDrift,
|
||||
designPaths,
|
||||
@@ -259,8 +259,13 @@ export function designTools(context: DesignOpsContext): EsToolDef[] {
|
||||
}
|
||||
for (const [name, spec] of Object.entries(EXEMPLARS)) {
|
||||
for (const error of validateTuiCoverage(spec)) drift.push(`${name}: ${error}`)
|
||||
for (const error of specVsRenderCheck(spec, Object.keys(spec.slots ?? {}).sort(), isWebrefFresh()))
|
||||
drift.push(error)
|
||||
// The slot-fill leg reads the actual render output from disk: a
|
||||
// spec fed its own slots could never fail (#270). A missing file
|
||||
// is already drift above; null keeps the gate honest, never green.
|
||||
const renderedFile = bundlePath(root, specFilename(name))
|
||||
const onDisk = await readFile(renderedFile, "utf8").catch(() => undefined)
|
||||
const renderedSlots = onDisk === undefined ? null : renderedSlotsOfSpecMd(onDisk)
|
||||
for (const error of specVsRenderCheck(spec, renderedSlots, isWebrefFresh())) drift.push(error)
|
||||
}
|
||||
if (drift.length) throw new ToolRefusal(`Phase-02 drift:\n- ${drift.join("\n- ")}`)
|
||||
return "Phase-02 check: specs, webref, csf and tui-notes byte-match the renderers; coverage, spec-vs-render, freshness and the play suite pass."
|
||||
|
||||
@@ -1325,6 +1325,24 @@ export function specFilename(name: string): string {
|
||||
return `components/${safe}.md`
|
||||
}
|
||||
|
||||
/**
|
||||
* Slot names carried by a rendered spec markdown's `## Slots` section (the
|
||||
* actual render output: `- \`name\`: description` lines). The phase-02 check
|
||||
* reads these from the file on disk instead of feeding each spec its own
|
||||
* slots (which could never fail); a hand-removed slot line diverges here.
|
||||
* The section is cut with `indexOf`/`trimEnd`, not a lazy `[\s\S]*?…\s*$`
|
||||
* scan: that regex was quadratic on a long whitespace run in the file.
|
||||
*/
|
||||
export function renderedSlotsOfSpecMd(markdown: string): string[] {
|
||||
const header = "## Slots\n"
|
||||
const start = markdown.indexOf(header)
|
||||
if (start === -1) return []
|
||||
const rest = markdown.slice(start + header.length)
|
||||
const next = rest.indexOf("\n## ")
|
||||
const section = next === -1 ? rest.trimEnd() : rest.slice(0, next)
|
||||
return [...section.matchAll(/^-\s+`([^`]+)`: /gm)].map((match) => match[1]!)
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// render_spec_md (d66328c:runner/queereye/specs.py)
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
@@ -167,13 +167,15 @@ export async function checkDrift(root: string): Promise<DriftReport> {
|
||||
const paths = designPaths(root)
|
||||
const tokens = await readTokens(root)
|
||||
if (tokens === undefined) return { drifted: [], missing: [path.basename(paths.tokens)], tokenErrors: [] }
|
||||
const probes = (await readProbes(root)) ?? []
|
||||
// A missing probes.json is a gap, never a silent empty-probes render (#270):
|
||||
// without probes the contrast QA below would pass vacuously.
|
||||
const probes = await readProbes(root)
|
||||
const missing: string[] = probes === undefined ? [path.basename(paths.probes)] : []
|
||||
const expected = new Map<string, string>([
|
||||
[paths.css, compileToCssVars(tokens)],
|
||||
[paths.guide, renderGuide(tokens, probes)],
|
||||
[paths.guide, renderGuide(tokens, probes ?? [])],
|
||||
])
|
||||
const drifted: string[] = []
|
||||
const missing: string[] = []
|
||||
for (const [file, content] of expected) {
|
||||
const existing = await readFile(file, "utf8").catch(() => undefined)
|
||||
if (existing === undefined) missing.push(path.basename(file))
|
||||
|
||||
@@ -23,7 +23,7 @@ export type Json = string | number | boolean | null | Json[] | { [key: string]:
|
||||
export type TokenTree = Record<string, unknown>
|
||||
|
||||
export const isToken = (node: unknown): node is Record<string, Json | undefined> =>
|
||||
typeof node === "object" && node !== null && !Array.isArray(node) && "$value" in node
|
||||
typeof node === "object" && node !== null && !Array.isArray(node) && Object.hasOwn(node, "$value")
|
||||
|
||||
export const isAliasValue = (value: unknown): value is string =>
|
||||
typeof value === "string" && ALIAS_FULL.test(value.trim())
|
||||
@@ -83,12 +83,9 @@ function resolveType(node: unknown, inherited?: string): string | undefined {
|
||||
function walkPath(root: unknown, ref: string): unknown {
|
||||
let node: unknown = root
|
||||
for (const segment of ref.split(".")) {
|
||||
if (
|
||||
typeof node !== "object" ||
|
||||
node === null ||
|
||||
Array.isArray(node) ||
|
||||
!(segment in (node as Record<string, unknown>))
|
||||
)
|
||||
// Own-property only: `in` walks the prototype chain, so {__proto__},
|
||||
// {constructor} and friends would resolve against Object.prototype (#266).
|
||||
if (typeof node !== "object" || node === null || Array.isArray(node) || !Object.hasOwn(node, segment))
|
||||
throw new UnknownAliasError(`alias {${ref}} does not resolve: missing "${segment}"`)
|
||||
node = (node as Record<string, unknown>)[segment]
|
||||
}
|
||||
|
||||
@@ -2,13 +2,16 @@
|
||||
//
|
||||
// Every claim line (a bullet, numbered item or prose paragraph line) must carry
|
||||
// at least one tag:
|
||||
// [VERIFIED: sha256:<hash> "verbatim quote"] — quote must be in the cached source
|
||||
// [VERIFIED: sha256:<hash> "quote, verbatim after documented normalisation"] — quote must be in the cached source
|
||||
// [INFERRED: reasoning] — reasoning from verified facts
|
||||
// [HYPOTHESIS: how to test it]
|
||||
// [NEGATIVE_KNOWLEDGE: what was searched]
|
||||
// The auditor verifies quotes against the content-addressed cache, flags
|
||||
// untagged and ungrounded claims, can prune them into a "Pruned claims"
|
||||
// section, and reports coverage counts (no composite score).
|
||||
// section, and reports coverage counts (no composite score). It marks,
|
||||
// prunes and refuses failing claims and dossiers; it never re-tags a claim's
|
||||
// kind (no VERIFIED→INFERRED demotion: that verb belongs to the
|
||||
// scout/harvest licence policy).
|
||||
import type { SourceCache } from "./cache.ts"
|
||||
import { verifyQuote } from "./quote.ts"
|
||||
|
||||
|
||||
@@ -49,28 +49,53 @@ const MAX_SOURCE_BYTES = 400_000
|
||||
|
||||
export class SourceCache {
|
||||
/**
|
||||
* When `stateDir` is set, entries are sealed with the engine key on write
|
||||
* (#52) and unsealed entries are refused on read. Without it the cache keeps
|
||||
* its legacy behaviour (tests and auxiliary readers).
|
||||
* Entries are sealed with the engine key on write (#52) and unsealed
|
||||
* entries are refused on read. `stateDir` is required: there is no
|
||||
* legacy unsealed mode (#251) — tests that need untrusted bytes write
|
||||
* them to disk directly instead of constructing a cache without a key.
|
||||
*/
|
||||
constructor(
|
||||
readonly dir: string,
|
||||
readonly stateDir?: string,
|
||||
readonly stateDir: string,
|
||||
) {}
|
||||
|
||||
private async seal(meta: SourceMeta): Promise<SourceMeta> {
|
||||
if (!this.stateDir) return meta
|
||||
const key = await ensureEngineKey(this.stateDir)
|
||||
return { ...meta, seal: sealMeta(meta as unknown as Record<string, unknown>, key) }
|
||||
}
|
||||
|
||||
private async sealed(meta: SourceMeta): Promise<boolean> {
|
||||
if (!this.stateDir) return true
|
||||
// Never mint a key to verify: without the engine's key nothing verifies (#59).
|
||||
const key = await readEngineKey(this.stateDir)
|
||||
return key !== undefined && verifyMetaSeal(meta as unknown as Record<string, unknown>, key)
|
||||
}
|
||||
|
||||
private indexFile = () => path.join(this.dir, "index.json")
|
||||
private indexSealFile = () => path.join(this.dir, "index.seal")
|
||||
|
||||
/**
|
||||
* The URL index sealed with the engine key (#251.2): without integrity the
|
||||
* index could remap a URL onto another sealed entry's bytes and `byUrl`
|
||||
* would serve them. Writes store an HMAC sidecar; reads refuse a
|
||||
* present-but-forged sidecar and keep serving a missing one (caches
|
||||
* written before sidecars stay readable). Per-entry seals still apply at
|
||||
* `get`, so the sidecar only guards the mapping, never the bytes.
|
||||
*/
|
||||
private async writeIndex(index: Record<string, string>): Promise<void> {
|
||||
await atomicWrite(this.indexFile(), `${JSON.stringify(index, null, 2)}\n`)
|
||||
const key = await ensureEngineKey(this.stateDir)
|
||||
await atomicWrite(this.indexSealFile(), `${sealMeta(index as unknown as Record<string, unknown>, key)}\n`)
|
||||
}
|
||||
|
||||
private async readIndex(): Promise<Record<string, string>> {
|
||||
const index = (await readJson<Record<string, string>>(this.indexFile())) ?? {}
|
||||
const seal = (await readFile(this.indexSealFile(), "utf8").catch(() => undefined))?.trim()
|
||||
if (seal === undefined || seal === "") return index
|
||||
const key = await readEngineKey(this.stateDir)
|
||||
if (key === undefined) return index
|
||||
return verifyMetaSeal({ ...index, seal }, key) ? index : {}
|
||||
}
|
||||
|
||||
async put(input: {
|
||||
url: string
|
||||
text: string
|
||||
@@ -98,9 +123,9 @@ export class SourceCache {
|
||||
const file = path.join(this.dir, `${hash}.md`)
|
||||
await atomicWrite(file, text)
|
||||
await atomicWrite(path.join(this.dir, `${hash}.json`), `${JSON.stringify(meta, null, 2)}\n`)
|
||||
const index = (await readJson<Record<string, string>>(path.join(this.dir, "index.json"))) ?? {}
|
||||
const index = await this.readIndex()
|
||||
index[canonicalUrl(input.url) ?? input.url] = hash
|
||||
await atomicWrite(path.join(this.dir, "index.json"), `${JSON.stringify(index, null, 2)}\n`)
|
||||
await this.writeIndex(index)
|
||||
return { meta, text, path: file }
|
||||
}
|
||||
|
||||
@@ -125,7 +150,7 @@ export class SourceCache {
|
||||
}
|
||||
|
||||
async byUrl(url: string): Promise<CachedSource | undefined> {
|
||||
const index = (await readJson<Record<string, string>>(path.join(this.dir, "index.json"))) ?? {}
|
||||
const index = await this.readIndex()
|
||||
// Canonical key first; raw keys are what 1.0 caches recorded.
|
||||
const hash = index[canonicalUrl(url) ?? url] ?? index[url]
|
||||
return hash ? this.get(hash) : undefined
|
||||
@@ -147,9 +172,9 @@ export class SourceCache {
|
||||
removed.push(hash)
|
||||
}
|
||||
if (removed.length) {
|
||||
const index = (await readJson<Record<string, string>>(path.join(this.dir, "index.json"))) ?? {}
|
||||
const index = await this.readIndex()
|
||||
for (const [url, hash] of Object.entries(index)) if (removed.includes(hash)) delete index[url]
|
||||
await atomicWrite(path.join(this.dir, "index.json"), `${JSON.stringify(index, null, 2)}\n`)
|
||||
await this.writeIndex(index)
|
||||
}
|
||||
return removed
|
||||
}
|
||||
|
||||
@@ -155,7 +155,7 @@ export function researchTools(context: ResearchOpsContext): EsToolDef[] {
|
||||
{
|
||||
name: "es_research_fetch",
|
||||
description:
|
||||
'Fetch a URL, cache its text content-addressed, and return the sha256 to cite: [VERIFIED: sha256:<hash> "verbatim quote"]. A fresh cached copy is served without a network request (refresh:true fetches again).',
|
||||
'Fetch a URL, cache its text content-addressed, and return the sha256 to cite: [VERIFIED: sha256:<hash> "quote, verbatim after documented normalisation"]. A fresh cached copy is served without a network request (refresh:true fetches again).',
|
||||
input: object(
|
||||
{
|
||||
url: { type: "string" },
|
||||
@@ -223,7 +223,7 @@ export function researchTools(context: ResearchOpsContext): EsToolDef[] {
|
||||
{
|
||||
name: "es_research_audit",
|
||||
description:
|
||||
"Audit a research artifact: every claim needs an epistemic tag, and every VERIFIED quote must appear verbatim in its cached source. prune:true moves failing claims to a 'Pruned claims' section.",
|
||||
"Audit a research artifact: every claim needs an epistemic tag, and every VERIFIED quote must appear verbatim after documented normalisation in its cached source. prune:true moves failing claims to a 'Pruned claims' section.",
|
||||
input: object({
|
||||
path: { type: "string", description: "Default .factory/research/REPORT.md" },
|
||||
prune: { type: "boolean" },
|
||||
|
||||
@@ -1,10 +1,12 @@
|
||||
// Pure verbatim-quote verification. A quote matches when, after normalising
|
||||
// whitespace and typographic punctuation (curly quotes, dashes, NBSP), it
|
||||
// appears in the source exactly. An ellipsis ("..." or "…") splits a prose
|
||||
// quote into fragments that must appear in order, and each fragment must be
|
||||
// evidence on its own (MIN_QUOTE characters). Code excerpts are one contiguous
|
||||
// span (verifySpan): "..." there is code, not an ellipsis. No fuzzy matching:
|
||||
// a quote is either in the source or it is not.
|
||||
// Quote verification that is verbatim after documented normalisation. A quote
|
||||
// matches when, after normalising (NFKC, typography folds for curly quotes,
|
||||
// dashes and spaces including NBSP, markdown link and emphasis stripping,
|
||||
// whitespace collapsing), it appears in the source exactly. An ellipsis
|
||||
// ("..." or "…") splits a prose quote into fragments that must appear in
|
||||
// order, and each fragment must be evidence on its own (MIN_QUOTE
|
||||
// characters). Code excerpts are one contiguous span (verifySpan): "..."
|
||||
// there is code, not an ellipsis. No fuzzy matching: a normalised quote is
|
||||
// either in the normalised source or it is not.
|
||||
|
||||
/** Shortest quote, and shortest ellipsis fragment, that counts as evidence (normalised characters). */
|
||||
export const MIN_QUOTE = 12
|
||||
|
||||
@@ -2,7 +2,8 @@
|
||||
// source sits on (preprint, peer-reviewed, docs, press), so domain packs can
|
||||
// weight verified claims by source quality. Rules are host-based and
|
||||
// auditable; unknown hosts stay tierless and fall back to `__default__`.
|
||||
// Packs may add a `tierDomains` override, which wins over the built-ins.
|
||||
// Callers may pass a `tierDomains` override map, which wins over the
|
||||
// built-ins.
|
||||
import { DOC_DOMAINS } from "./url.ts"
|
||||
|
||||
const PREPRINT = ["arxiv.org", "biorxiv.org", "medrxiv.org", "ssrn.com"] as const
|
||||
@@ -36,7 +37,7 @@ const matches = (host: string, domain: string): boolean => host === domain || ho
|
||||
|
||||
/**
|
||||
* Classify a source URL into a tier, or undefined when no rule fires.
|
||||
* `overrides` (a pack's `tierDomains`) is checked first and wins.
|
||||
* `overrides` (a caller's `tierDomains` map) is checked first and wins.
|
||||
*/
|
||||
export function classifySourceTier(
|
||||
url: string,
|
||||
@@ -49,12 +50,9 @@ export function classifySourceTier(
|
||||
if (domains.some((domain) => matches(host, domain.toLowerCase()))) return tier
|
||||
if (PREPRINT.some((domain) => matches(host, domain))) return "PREPRINT"
|
||||
if (PEER_REVIEWED.some((domain) => matches(host, domain))) return "PEER_REVIEWED"
|
||||
if (
|
||||
(DOC_DOMAINS as readonly string[]).some((domain) => matches(host, domain)) ||
|
||||
host.startsWith("docs.") ||
|
||||
host.startsWith("developer.")
|
||||
)
|
||||
return "TECHNICAL_DOCUMENTATION"
|
||||
// Only curated documentation hosts count (#252): a bare docs./developer.
|
||||
// prefix rule would let docs.<attacker> claim curated weight.
|
||||
if ((DOC_DOMAINS as readonly string[]).some((domain) => matches(host, domain))) return "TECHNICAL_DOCUMENTATION"
|
||||
if (host.endsWith(".gov")) return "PRIMARY_PRESS"
|
||||
return undefined
|
||||
}
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
// Brainstorm artifacts: the plan (lenses and caps), recorded ideas, rubric
|
||||
// scores, and the completed result (duplicate collapsing, ranking, and a
|
||||
// diversified shortlist with a forced outlier slot).
|
||||
// diversified shortlist with a forced outlier whenever the shortlist has
|
||||
// ≥2 slots).
|
||||
import { z } from "zod"
|
||||
|
||||
export const BRAINSTORM_VERSION = 1 as const
|
||||
@@ -21,7 +22,7 @@ export const BrainstormPlanSchema = z.object({
|
||||
ideasPerLens: z.number().int().min(1).max(10),
|
||||
/** Hard cap of characters per idea text. */
|
||||
maxIdeaChars: z.number().int().min(120),
|
||||
/** Shortlist size, including the forced outlier slot. */
|
||||
/** Shortlist size, including the forced outlier slot (whenever the shortlist has ≥2 slots). */
|
||||
shortlistSize: z.number().int().min(2).max(12),
|
||||
/** Similarity at or above which an idea is collapsed as a duplicate. */
|
||||
dedupeThreshold: z.number().min(0.5).max(0.95),
|
||||
|
||||
@@ -19,7 +19,7 @@ export const AuditFindingSchema = z
|
||||
/** POSIX path relative to the audited tree's root. */
|
||||
file: z.string().min(1),
|
||||
lines: z.tuple([z.number().int().positive(), z.number().int().positive()]),
|
||||
/** Verbatim code from those lines (at least 12 characters). */
|
||||
/** Verbatim code from those lines (the witness enforces the 12-char floor via verifySpan; the schema keeps min(1) so the witness reports the reason). */
|
||||
excerpt: z.string().min(1),
|
||||
detail: z.string().min(8),
|
||||
remediation: z.string().optional(),
|
||||
|
||||
@@ -24,11 +24,6 @@ export const DomainPackSchema = z
|
||||
bannedDomains: z.array(z.string().min(1)).default([]).describe("Source URLs containing any of these are rejected."),
|
||||
mandatoryTags: z.array(ClaimTagSchema).default([]).describe("Every claim must carry one of these tags."),
|
||||
retractionPolicy: RetractionPolicySchema.default("standard"),
|
||||
/** Per-tier host overrides for source classification (checked before the built-in table). */
|
||||
tierDomains: z
|
||||
.record(z.string(), z.array(z.string().min(1)))
|
||||
.optional()
|
||||
.describe("Pack host overrides by tier for source classification."),
|
||||
/** Legacy metadata; carried so ported pack files round-trip. */
|
||||
exportTemplate: z.string().optional(),
|
||||
})
|
||||
|
||||
@@ -37,6 +37,21 @@ export const ScoutAdvisorySchema = z.object({
|
||||
})
|
||||
export type ScoutAdvisory = z.infer<typeof ScoutAdvisorySchema>
|
||||
|
||||
/** The OSV coordinates a candidate's advisories were actually queried at (#257). */
|
||||
export const ScoutAdvisoryQuerySchema = z.object({
|
||||
ecosystem: z.string(),
|
||||
package: z.string(),
|
||||
version: z.string().optional(),
|
||||
})
|
||||
export type ScoutAdvisoryQuery = z.infer<typeof ScoutAdvisoryQuerySchema>
|
||||
|
||||
/** The per-candidate advisories artifact: what was queried, and what came back. */
|
||||
export const ScoutAdvisoriesFileSchema = z.object({
|
||||
query: ScoutAdvisoryQuerySchema,
|
||||
advisories: z.array(ScoutAdvisorySchema),
|
||||
})
|
||||
export type ScoutAdvisoriesFile = z.infer<typeof ScoutAdvisoriesFileSchema>
|
||||
|
||||
export const ScoutAssessmentSchema = z.object({
|
||||
candidate: z.string(),
|
||||
stars: z.number().int().nonnegative().optional(),
|
||||
|
||||
@@ -18,7 +18,7 @@ import { DEFAULT_LICENSE_WHITELIST, type HarvestProfile } from "../schema/harves
|
||||
import { SCOUT_VERSION, type ScoutAdvisory, type ScoutAssessment, ScoutAssessmentSchema } from "../schema/scout.ts"
|
||||
import { relativeTo } from "../trust/paths.ts"
|
||||
import { atomicWrite } from "../util/fs.ts"
|
||||
import { renderScoutReport } from "./report.ts"
|
||||
import { type AdvisoryCheck, renderScoutReport } from "./report.ts"
|
||||
import {
|
||||
readAssessments,
|
||||
readScoutAdvisories,
|
||||
@@ -256,7 +256,29 @@ export function scoutTools(context: ScoutOpsContext): EsToolDef[] {
|
||||
throw new ToolRefusal(
|
||||
`${candidate.id} is not a registry package: give the ecosystem (${Object.values(OSV_ECOSYSTEMS).join(", ")}) and the package name it publishes.`,
|
||||
)
|
||||
const version = input.version ?? profile?.release
|
||||
// A registry candidate must be queried at its own coordinates (#257):
|
||||
// a hand-supplied wrong ecosystem/package returns a clean bill for
|
||||
// code that was never checked.
|
||||
if (source.kind === "registry") {
|
||||
const derived = { ecosystem: OSV_ECOSYSTEMS[source.registry], name: source.name }
|
||||
const sameEcosystem =
|
||||
input.ecosystem === undefined || String(input.ecosystem).toLowerCase() === derived.ecosystem.toLowerCase()
|
||||
const samePackage = input.package === undefined || String(input.package) === derived.name
|
||||
if (!sameEcosystem || !samePackage)
|
||||
throw new ToolRefusal(
|
||||
`Advisories for ${candidate.id} must be queried for the scanned source (${derived.ecosystem}:${derived.name}); ` +
|
||||
`got ${input.ecosystem ?? ecosystem}:${input.package ?? name}. Query the candidate's own package, not another one.`,
|
||||
)
|
||||
}
|
||||
// The version is pinned to the scanned release when one is known
|
||||
// (#257): pointing the query at a fixed release hides its advisories.
|
||||
const release = profile?.release
|
||||
if (release !== undefined && input.version !== undefined && String(input.version) !== release)
|
||||
throw new ToolRefusal(
|
||||
`Version ${input.version} does not match the scanned release ${release} of ${candidate.id}. ` +
|
||||
`Query ${release}, or omit the version to check every release.`,
|
||||
)
|
||||
const version = input.version ?? release
|
||||
const result = await queryOsv(
|
||||
{ ecosystem, name: String(name), ...(version ? { version: String(version) } : {}) },
|
||||
cache,
|
||||
@@ -275,7 +297,14 @@ export function scoutTools(context: ScoutOpsContext): EsToolDef[] {
|
||||
fixed: [...advisory.fixed],
|
||||
source: result.sha256,
|
||||
}))
|
||||
await writeScoutAdvisories(root, candidate.id, advisories)
|
||||
await writeScoutAdvisories(root, candidate.id, {
|
||||
query: {
|
||||
ecosystem,
|
||||
package: String(name),
|
||||
...(version !== undefined ? { version: String(version) } : {}),
|
||||
},
|
||||
advisories,
|
||||
})
|
||||
return [
|
||||
`${advisories.length} OSV advisor${advisories.length === 1 ? "y" : "ies"} for ${ecosystem}:${name}${version ? `@${version}` : ""}, cached as sha256:${result.sha256}.`,
|
||||
...advisories.map(
|
||||
@@ -337,7 +366,7 @@ export function scoutTools(context: ScoutOpsContext): EsToolDef[] {
|
||||
...(input.lastRelease ? { lastRelease: String(input.lastRelease) } : {}),
|
||||
...(input.releasesPerYear !== undefined ? { releasesPerYear: Number(input.releasesPerYear) } : {}),
|
||||
maintenance: input.maintenance,
|
||||
advisories: (await readScoutAdvisories(root, candidate.id).catch(() => undefined)) ?? [],
|
||||
advisories: (await readScoutAdvisories(root, candidate.id).catch(() => undefined))?.advisories ?? [],
|
||||
proposal: input.proposal,
|
||||
rationale: String(input.rationale ?? ""),
|
||||
...(input.blueprint ? { blueprint: String(input.blueprint) } : {}),
|
||||
@@ -351,7 +380,7 @@ export function scoutTools(context: ScoutOpsContext): EsToolDef[] {
|
||||
{
|
||||
name: "es_scout_complete",
|
||||
description:
|
||||
"Scout only: when every candidate is scanned and recorded, recompute the verdicts (adopt survives only for a verified, whitelisted, permissive license), flag severe advisories, rank, and write .factory/scout/REPORT.md, scout.json and dossier.json.",
|
||||
"Scout only: when every candidate is scanned, recorded, and checked for advisories (queried, or an explicit NEGATIVE_KNOWLEDGE skip), recompute the verdicts (adopt survives only for a verified, whitelisted, permissive license), flag severe advisories, rank, and write .factory/scout/REPORT.md, scout.json and dossier.json.",
|
||||
input: object({}),
|
||||
execute: async (_input, toolContext) => {
|
||||
await guard(toolContext.agent)
|
||||
@@ -359,6 +388,7 @@ export function scoutTools(context: ScoutOpsContext): EsToolDef[] {
|
||||
const assessments = new Map((await readAssessments(root)).map((item) => [item.candidate, item]))
|
||||
const profiles = new Map<string, HarvestProfile>()
|
||||
const evidence: CandidateEvidence[] = []
|
||||
const checks = new Map<string, AdvisoryCheck>()
|
||||
const missing: string[] = []
|
||||
for (const candidate of current.candidates) {
|
||||
const profile = await readScoutProfile(root, candidate.id).catch(() => undefined)
|
||||
@@ -367,7 +397,32 @@ export function scoutTools(context: ScoutOpsContext): EsToolDef[] {
|
||||
if (!assessment) missing.push(`${candidate.id}: record it (es_scout_record)`)
|
||||
if (!profile || !assessment) continue
|
||||
profiles.set(candidate.id, profile)
|
||||
const advisories = (await readScoutAdvisories(root, candidate.id).catch(() => undefined)) ?? []
|
||||
// The advisories step is mandatory (#257): skipping it, querying a
|
||||
// wrong package, or pointing the version at a fixed release all
|
||||
// used to yield a clean adopt with zero citations. Either the
|
||||
// query ran (its coordinates are pinned at query time), or the
|
||||
// assessment records explicit NEGATIVE_KNOWLEDGE that it was
|
||||
// skipped — a silent skip refuses here.
|
||||
const advisoryFile = await readScoutAdvisories(root, candidate.id).catch(() => undefined)
|
||||
const skipped =
|
||||
!advisoryFile &&
|
||||
assessment.claims.some(
|
||||
(claim) =>
|
||||
claim.tag === "NEGATIVE_KNOWLEDGE" &&
|
||||
/advisori/i.test([claim.statement, claim.query ?? "", claim.finding ?? ""].join(" ")),
|
||||
)
|
||||
if (!advisoryFile && !skipped)
|
||||
missing.push(
|
||||
`${candidate.id}: query its advisories (es_scout_advisories), or record explicit NEGATIVE_KNOWLEDGE that the step was skipped`,
|
||||
)
|
||||
if (!advisoryFile && !skipped) continue
|
||||
const advisories = advisoryFile?.advisories ?? []
|
||||
checks.set(
|
||||
candidate.id,
|
||||
advisoryFile
|
||||
? { query: advisoryFile.query, skipped: false, count: advisories.length }
|
||||
: { skipped: true, count: 0 },
|
||||
)
|
||||
evidence.push({
|
||||
id: candidate.id,
|
||||
name: candidate.name,
|
||||
@@ -398,7 +453,7 @@ export function scoutTools(context: ScoutOpsContext): EsToolDef[] {
|
||||
throw new ToolRefusal(`The scout dossier could not be written: ${error.message}`)
|
||||
})
|
||||
await writeScoutResult(root, result)
|
||||
await atomicWrite(paths.report, renderScoutReport(current, result, profiles, assessments))
|
||||
await atomicWrite(paths.report, renderScoutReport(current, result, profiles, assessments, checks))
|
||||
return [
|
||||
`Scout complete: ${current.objective}`,
|
||||
...verdicts.map(
|
||||
|
||||
@@ -1,7 +1,14 @@
|
||||
// REPORT.md for a finished scout: the competitive matrix, the risks the
|
||||
// red-team found, clean-room blueprints, and the cited claims. Deterministic.
|
||||
import type { HarvestProfile } from "../schema/harvest.ts"
|
||||
import type { ScoutAssessment, ScoutPlan, ScoutResult } from "../schema/scout.ts"
|
||||
import type { ScoutAdvisoryQuery, ScoutAssessment, ScoutPlan, ScoutResult } from "../schema/scout.ts"
|
||||
|
||||
/** What the report says about one candidate's advisories step (#257). */
|
||||
export interface AdvisoryCheck {
|
||||
readonly query?: ScoutAdvisoryQuery | undefined
|
||||
readonly skipped: boolean
|
||||
readonly count: number
|
||||
}
|
||||
|
||||
const cell = (value: unknown) => String(value ?? "—").replaceAll("|", "\\|")
|
||||
|
||||
@@ -10,6 +17,7 @@ export function renderScoutReport(
|
||||
result: ScoutResult,
|
||||
profiles: ReadonlyMap<string, HarvestProfile>,
|
||||
assessments: ReadonlyMap<string, ScoutAssessment>,
|
||||
checks: ReadonlyMap<string, AdvisoryCheck>,
|
||||
): string {
|
||||
const lines = [
|
||||
`# Scout: ${plan.objective}`,
|
||||
@@ -39,6 +47,21 @@ export function renderScoutReport(
|
||||
`- **${verdict.name}**: ${notes.length ? notes.join("; ") : "none found"}${verdict.attribution ? ` · attribution ${verdict.attribution}` : ""}`,
|
||||
)
|
||||
}
|
||||
// What the advisories step actually checked (#257): the queried
|
||||
// coordinates, or the explicit skip. A clean adopt with no line here is
|
||||
// impossible — complete refuses the silent skip.
|
||||
lines.push("", "## Advisory checks", "")
|
||||
for (const verdict of result.verdicts) {
|
||||
const check = checks.get(verdict.candidate)
|
||||
if (!check || check.skipped)
|
||||
lines.push(`- **${verdict.name}**: advisories not queried; the assessment records explicit NEGATIVE_KNOWLEDGE.`)
|
||||
else {
|
||||
const at = check.query?.version ? `@${check.query.version}` : ""
|
||||
lines.push(
|
||||
`- **${verdict.name}**: OSV ${check.query?.ecosystem}:${check.query?.package}${at} checked, ${check.count} ${check.count === 1 ? "advisory" : "advisories"} cited.`,
|
||||
)
|
||||
}
|
||||
}
|
||||
const blueprints = result.verdicts.filter(
|
||||
(verdict) => verdict.verdict === "clean-room" && assessments.get(verdict.candidate)?.blueprint,
|
||||
)
|
||||
|
||||
@@ -6,8 +6,8 @@ import path from "node:path"
|
||||
import { factoryLayout } from "../layout.ts"
|
||||
import { type HarvestProfile, HarvestProfileSchema } from "../schema/harvest.ts"
|
||||
import {
|
||||
type ScoutAdvisory,
|
||||
ScoutAdvisorySchema,
|
||||
type ScoutAdvisoriesFile,
|
||||
ScoutAdvisoriesFileSchema,
|
||||
type ScoutAssessment,
|
||||
ScoutAssessmentSchema,
|
||||
type ScoutPlan,
|
||||
@@ -56,13 +56,20 @@ export async function startScoutRun(root: string, plan: ScoutPlan): Promise<void
|
||||
export const writeScoutProfile = (root: string, profile: HarvestProfile) =>
|
||||
writeJson(scoutPaths(root).profile(profile.id), HarvestProfileSchema.parse(profile))
|
||||
|
||||
export async function readScoutAdvisories(root: string, id: string): Promise<ScoutAdvisory[] | undefined> {
|
||||
const raw = await readJson<unknown[]>(scoutPaths(root).advisories(id))
|
||||
return raw === undefined ? undefined : raw.map((item) => ScoutAdvisorySchema.parse(item))
|
||||
export async function readScoutAdvisories(root: string, id: string): Promise<ScoutAdvisoriesFile | undefined> {
|
||||
const raw = await readJson<unknown>(scoutPaths(root).advisories(id))
|
||||
if (raw === undefined) return undefined
|
||||
try {
|
||||
return ScoutAdvisoriesFileSchema.parse(raw)
|
||||
} catch (error) {
|
||||
throw new Error(
|
||||
`invalid ${scoutPaths(root).advisories(id)}: ${error instanceof Error ? error.message : String(error)}`,
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
export const writeScoutAdvisories = (root: string, id: string, advisories: readonly ScoutAdvisory[]) =>
|
||||
writeJson(scoutPaths(root).advisories(id), advisories)
|
||||
export const writeScoutAdvisories = (root: string, id: string, file: ScoutAdvisoriesFile) =>
|
||||
writeJson(scoutPaths(root).advisories(id), ScoutAdvisoriesFileSchema.parse(file))
|
||||
|
||||
export async function readAssessments(root: string): Promise<ScoutAssessment[]> {
|
||||
const raw = await readJson<unknown[]>(scoutPaths(root).assessments)
|
||||
|
||||
@@ -87,7 +87,14 @@ export function controlClass(relative: string): ControlClass | undefined {
|
||||
|
||||
export const isControlPath = (relative: string) => controlClass(relative) !== undefined
|
||||
|
||||
/** Files whose hashes the baseline pins (existing, tracked control artifacts). */
|
||||
/**
|
||||
* Files whose hashes the baseline pins (existing, tracked control artifacts).
|
||||
* Project-root only by design (#240.3): the user-global layer
|
||||
* (globalConfigFile, under XDG_CONFIG_HOME) is machine-scoped and shared
|
||||
* across projects — pinning it here would drift whenever the human edits
|
||||
* global settings for another project. Global writes stay guarded by the
|
||||
* human-only `es config set --global` gate instead.
|
||||
*/
|
||||
const PINNED = (root: string) => {
|
||||
const layout = factoryLayout(root)
|
||||
return [layout.gates, layout.frontier, layout.config]
|
||||
|
||||
@@ -102,8 +102,9 @@ export function evaluateWrite(context: PolicyContext, agentId: string | undefine
|
||||
if (!spec) return allow
|
||||
if (relative === undefined) return deny(`Factory seat "${spec.id}" may not write outside the project: ${absolute}`)
|
||||
|
||||
// REPORT.md has one writer per run mode (#111): the synthesizer is the
|
||||
// only writer in research runs, the factory in build runs (AGENTS.md).
|
||||
// REPORT.md has one writer scope per run mode (#111, #220): factory-docs
|
||||
// seats (the factory and the manager share the scope) in build runs, the
|
||||
// research synthesizer alone in research runs (AGENTS.md).
|
||||
// The mode comes from the sealed run state; when it cannot be read both
|
||||
// writers are denied (fail closed).
|
||||
if (relative === ".factory/research/REPORT.md") {
|
||||
@@ -329,7 +330,7 @@ const MENTION_HINT =
|
||||
" If it only names the verb in text (a here-doc, a commit message, a `--body` argument), the rule still reads a call: write the text to a file with the edit tool and pass the file (`gh … --body-file`, `git commit -F`)."
|
||||
|
||||
const CONTROL_MENTION =
|
||||
/\.factory\/(gates\.json|config\.json|frontier\.json|waivers|approvals|runtime|STOP|git-hooks|claims\b|audits\b|research\/(sources\b|(coverage|dossier|brief\.pcrb)\.json)|(brainstorm|harvest|design|scout)\/[^\s'"]*\.json)|\.git\/(config|hooks)|\.opencode\/(hooks\.json|plugins|opencode\.jsonc?)|\.claude\/settings|opencode\.jsonc?\b/i
|
||||
/\.factory\/(gates\.json|config\.json|frontier\.json|waivers|approvals|runtime|STOP|git-hooks|claims\b|audits\b|improve\b|research\/(sources\b|(coverage|dossier|brief\.pcrb)\.json)|(brainstorm|harvest|design|scout)\/(?!notes\/)[^\s'"]*)|\.git\/(config|hooks)|\.opencode\/(hooks\.json|plugins|opencode\.jsonc?)|\.claude\/settings|opencode\.jsonc?\b/i
|
||||
/** Any mention of the factory dir (a `cd .factory` reaches control files without naming them). */
|
||||
const FACTORY_MENTION = /(^|[\s/=:])\.factory(\/|[\s;|&]|$)/i
|
||||
/** Any mention of fleet isolation state (task worktrees live under `.fleet/`). */
|
||||
|
||||
@@ -40,6 +40,13 @@ export interface SandboxProbe {
|
||||
readonly exists: (file: string) => boolean
|
||||
/** bwrap supports --overlay-src/--tmp-overlay (0.10+). */
|
||||
readonly overlay: boolean
|
||||
/**
|
||||
* Per-directory overlay support (#231): an overlay mount can fail on one
|
||||
* filesystem while it succeeds on another (a btrfs home vs tmpfs /tmp), so
|
||||
* a probe that only tries /tmp misreports the mount that matters. Absent:
|
||||
* `overlay` answers for every directory.
|
||||
*/
|
||||
readonly overlayFor?: (dir: string) => boolean
|
||||
readonly uid?: number
|
||||
}
|
||||
|
||||
@@ -106,6 +113,9 @@ export function sandboxArgv(
|
||||
const writableRoot = spec.kind === "user" || spec.kind === "seat"
|
||||
const writable = spec.kind === "programmer" || spec.kind === "gate" ? spec.writable : undefined
|
||||
const protectedRel = user ? PROTECTED_ALWAYS : [...PROTECTED_ALWAYS, ...PROTECTED_FOR_SEATS]
|
||||
// Graceful degradation (#231): when the per-dir probe refuses a mount, the
|
||||
// overlay args are omitted instead of emitting a mount that fails at runtime.
|
||||
const overlayFor = (dir: string) => probe.overlayFor?.(dir) ?? probe.overlay
|
||||
const protectedIn = (dir: string, names: readonly string[]) =>
|
||||
names.map((name) => path.join(dir, name)).filter((file) => probe.exists(file))
|
||||
const args: string[] = ["bwrap", writableRoot ? "--bind" : "--ro-bind", "/", "/", "--dev", "/dev", "--proc", "/proc"]
|
||||
@@ -116,9 +126,9 @@ export function sandboxArgv(
|
||||
// A private /tmp; projects and homes that live under /tmp are bound back in.
|
||||
args.push("--tmpfs", "/tmp")
|
||||
// Tool caches under home stay usable; writes there are thrown away.
|
||||
if (probe.overlay && probe.exists(home)) args.push("--overlay-src", home, "--tmp-overlay", home)
|
||||
if (overlayFor(home) && probe.exists(home)) args.push("--overlay-src", home, "--tmp-overlay", home)
|
||||
else if (inside("/tmp", home) && probe.exists(home)) roBind(home)
|
||||
if (inside("/tmp", spec.root) && !(probe.overlay && inside(home, spec.root))) roBind(spec.root)
|
||||
if (inside("/tmp", spec.root) && !(overlayFor(home) && inside(home, spec.root))) roBind(spec.root)
|
||||
}
|
||||
// Project control paths are read-only.
|
||||
for (const file of protectedIn(spec.root, protectedRel)) roBind(file)
|
||||
@@ -131,7 +141,7 @@ export function sandboxArgv(
|
||||
for (const file of protectedIn(writable, [...PROTECTED_IN_WORKTREE, ...protectedRel])) roBind(file)
|
||||
}
|
||||
// A read-only seat's writes in its cwd land in a throwaway layer (when bwrap can).
|
||||
if (spec.kind === "readonly" && probe.overlay) args.push("--overlay-src", spec.cwd, "--tmp-overlay", spec.cwd)
|
||||
if (spec.kind === "readonly" && overlayFor(spec.cwd)) args.push("--overlay-src", spec.cwd, "--tmp-overlay", spec.cwd)
|
||||
|
||||
// Secrets, for every kind.
|
||||
if (probe.exists(spec.stateDir)) args.push("--tmpfs", spec.stateDir)
|
||||
@@ -158,32 +168,44 @@ export function sandboxArgv(
|
||||
return args
|
||||
}
|
||||
|
||||
let overlayCache: boolean | undefined
|
||||
/** Whether this bwrap supports --tmp-overlay (Ubuntu 24.04 ships 0.9, which does not). */
|
||||
export function bwrapOverlay(): boolean {
|
||||
if (overlayCache !== undefined) return overlayCache
|
||||
const overlayCache = new Map<string, boolean>()
|
||||
/**
|
||||
* Whether this bwrap can overlay-mount `dir` (Ubuntu 24.04 ships bwrap 0.9,
|
||||
* which has no overlay support at all). Probed per directory and cached: an
|
||||
* overlay mount can fail on one filesystem while it succeeds on another (a
|
||||
* btrfs home vs tmpfs /tmp, #231), so probing only /tmp misreports the mount
|
||||
* that matters.
|
||||
*/
|
||||
export function bwrapOverlay(dir = "/tmp"): boolean {
|
||||
const cached = overlayCache.get(dir)
|
||||
if (cached !== undefined) return cached
|
||||
try {
|
||||
// bwrap is resolved from PATH by design (it may be absent).
|
||||
const probe = spawnSync(
|
||||
// NOSONAR
|
||||
"bwrap",
|
||||
["--ro-bind", "/", "/", "--dev", "/dev", "--overlay-src", "/tmp", "--tmp-overlay", "/tmp", "--", "true"],
|
||||
["--ro-bind", "/", "/", "--dev", "/dev", "--overlay-src", dir, "--tmp-overlay", dir, "--", "true"],
|
||||
{ timeout: 5000 },
|
||||
)
|
||||
overlayCache = probe.status === 0
|
||||
overlayCache.set(dir, probe.status === 0)
|
||||
} catch {
|
||||
overlayCache = false
|
||||
overlayCache.set(dir, false)
|
||||
}
|
||||
return overlayCache
|
||||
return overlayCache.get(dir)!
|
||||
}
|
||||
|
||||
function liveProbe(): SandboxProbe {
|
||||
return { exists: existsSync, overlay: bwrapOverlay(), uid: process.getuid?.() }
|
||||
return {
|
||||
exists: existsSync,
|
||||
overlay: bwrapOverlay(),
|
||||
overlayFor: (dir) => bwrapOverlay(dir),
|
||||
uid: process.getuid?.(),
|
||||
}
|
||||
}
|
||||
|
||||
/** For tests: forget the cached overlay probe. */
|
||||
/** For tests: forget the cached overlay probes. */
|
||||
export const resetOverlayProbe = () => {
|
||||
overlayCache = undefined
|
||||
overlayCache.clear()
|
||||
}
|
||||
|
||||
/** Why a sandbox cannot run here, for refusal messages. */
|
||||
|
||||
@@ -5,10 +5,10 @@ exports[`agent assets prompt snapshot: a changed prompt must bump its version 1`
|
||||
"auditor-antithesis": "v2:15f4d7f75e5bb958",
|
||||
"auditor-thesis": "v2:b903e1353177f342",
|
||||
"brainstorm-critic": "v1:28fd3abc95c4a28a",
|
||||
"brainstormer": "v3:eee9b20fb26482a4",
|
||||
"brainstormer": "v4:278864b7a2cf7008",
|
||||
"deep-researcher": "v1:2c436cd49fd6aaf8",
|
||||
"designer": "v2:745908a2a4abff0b",
|
||||
"factory": "v9:4b48ec73da814ab3",
|
||||
"factory": "v10:d87aa973b1517d02",
|
||||
"grill": "v4:722c43eeb12d95bb",
|
||||
"harvester": "v2:c63fbc079e1cbbbe",
|
||||
"lens-adjacent-possible": "v1:d6d768502ed2adf2",
|
||||
|
||||
@@ -0,0 +1,133 @@
|
||||
// Issue #180 integration tests (core surface): approvals are hash-bound to
|
||||
// the approved artifacts and every channel is recorded and verifies. Uses
|
||||
// only existing seams (test keystores, temp dirs); no host, no human seal.
|
||||
import { describe, expect, test } from "bun:test"
|
||||
import { readFile, writeFile } from "node:fs/promises"
|
||||
import { type HumanSigner, sealHumanKey, unlockHumanKey, verifyRecordSignature } from "../src/approval/keystore.ts"
|
||||
import { readApproval, verifyApproval } from "../src/approval/record.ts"
|
||||
import { approveStage } from "../src/approval/service.ts"
|
||||
import { Factory, pendingApprovals } from "../src/factory/index.ts"
|
||||
import { factoryLayout } from "../src/layout.ts"
|
||||
import { writeJson } from "../src/util/fs.ts"
|
||||
import { type Fixture, frontier, gitRepo, writeSpecs } from "./helpers.ts"
|
||||
|
||||
const PASSPHRASE = "test-passphrase-1234"
|
||||
|
||||
async function setup(): Promise<{ fx: Fixture; signer: HumanSigner; factory: Factory }> {
|
||||
const fx = await gitRepo()
|
||||
await sealHumanKey(PASSPHRASE, fx.state)
|
||||
const signer = await unlockHumanKey(PASSPHRASE, fx.state)
|
||||
const factory = new Factory(fx.root, {
|
||||
gates: async () => ({ passed: true, findings: [], summary: "green" }),
|
||||
stateDir: fx.state,
|
||||
})
|
||||
await factory.writeFrontier("grill", frontier())
|
||||
return { fx, signer, factory }
|
||||
}
|
||||
|
||||
const seedPending = (fx: Fixture, stages: Array<"frontier" | "spec">) =>
|
||||
writeJson(
|
||||
factoryLayout(fx.root).pending,
|
||||
stages.map((stage) => ({ stage, requestedBy: "factory", at: new Date().toISOString() })),
|
||||
)
|
||||
|
||||
describe("issue #180: approvals are hash-bound", () => {
|
||||
test("editing frontier.json after approval invalidates the record", async () => {
|
||||
const { fx, signer } = await setup()
|
||||
try {
|
||||
await seedPending(fx, ["frontier"])
|
||||
const { record } = await approveStage(fx.root, {
|
||||
stage: "frontier",
|
||||
channel: "tui",
|
||||
approvedBy: "tester",
|
||||
signer,
|
||||
stateDir: fx.state,
|
||||
})
|
||||
expect(record.channel).toBe("tui")
|
||||
expect((await verifyApproval(fx.root, "frontier", { stateDir: fx.state })).ok).toBe(true)
|
||||
const file = factoryLayout(fx.root).frontier
|
||||
await writeFile(file, `${await readFile(file, "utf8")}\n<!-- tampered -->\n`)
|
||||
const check = await verifyApproval(fx.root, "frontier", { stateDir: fx.state })
|
||||
expect(check.ok).toBe(false)
|
||||
if (!check.ok) expect(check.reason).toMatch(/changed after it was approved/)
|
||||
expect(await readApproval(fx.root, "frontier")).toBeDefined()
|
||||
expect(await pendingApprovals(fx.root)).toEqual([])
|
||||
} finally {
|
||||
signer.destroy()
|
||||
await fx.cleanup()
|
||||
}
|
||||
})
|
||||
|
||||
test("editing a GOAL.md after spec approval invalidates it", async () => {
|
||||
const { fx, signer } = await setup()
|
||||
try {
|
||||
await writeSpecs(fx.root, ["alpha"])
|
||||
await seedPending(fx, ["spec"])
|
||||
await approveStage(fx.root, {
|
||||
stage: "spec",
|
||||
channel: "web",
|
||||
approvedBy: "tester",
|
||||
signer,
|
||||
stateDir: fx.state,
|
||||
})
|
||||
expect((await verifyApproval(fx.root, "spec", { stateDir: fx.state })).ok).toBe(true)
|
||||
const goal = factoryLayout(fx.root).spec("alpha")
|
||||
await writeFile(goal, `${await readFile(goal, "utf8")}\n<!-- tampered -->\n`)
|
||||
expect((await verifyApproval(fx.root, "spec", { stateDir: fx.state })).ok).toBe(false)
|
||||
} finally {
|
||||
signer.destroy()
|
||||
await fx.cleanup()
|
||||
}
|
||||
})
|
||||
})
|
||||
|
||||
describe("issue #180: every channel is recorded and verifies", () => {
|
||||
test("cli, tui and web records carry their channel and a valid signature", async () => {
|
||||
for (const channel of ["cli", "tui", "web"] as const) {
|
||||
const { fx, signer } = await setup()
|
||||
try {
|
||||
await seedPending(fx, ["frontier"])
|
||||
const { record } = await approveStage(fx.root, {
|
||||
stage: "frontier",
|
||||
channel,
|
||||
approvedBy: "tester",
|
||||
signer,
|
||||
stateDir: fx.state,
|
||||
})
|
||||
expect(record.channel).toBe(channel)
|
||||
expect(record.signature.alg).toBe("ed25519")
|
||||
expect(await verifyRecordSignature(record, fx.state)).toBe(true)
|
||||
expect(await verifyApproval(fx.root, "frontier", { stateDir: fx.state })).toMatchObject({ ok: true })
|
||||
} finally {
|
||||
signer.destroy()
|
||||
await fx.cleanup()
|
||||
}
|
||||
}
|
||||
})
|
||||
|
||||
test("a record with a forged signature is refused", async () => {
|
||||
const { fx, signer } = await setup()
|
||||
try {
|
||||
await seedPending(fx, ["frontier"])
|
||||
const { record } = await approveStage(fx.root, {
|
||||
stage: "frontier",
|
||||
channel: "cli",
|
||||
approvedBy: "tester",
|
||||
signer,
|
||||
stateDir: fx.state,
|
||||
})
|
||||
expect((await verifyApproval(fx.root, "frontier", { stateDir: fx.state })).ok).toBe(true)
|
||||
const file = factoryLayout(fx.root).approval("frontier")
|
||||
await writeJson(file, {
|
||||
...record,
|
||||
signature: { ...record.signature, sig: `${record.signature.sig.slice(0, -2)}AA` },
|
||||
})
|
||||
const check = await verifyApproval(fx.root, "frontier", { stateDir: fx.state })
|
||||
expect(check.ok).toBe(false)
|
||||
if (!check.ok) expect(check.reason).toMatch(/not signed by this machine's human key/)
|
||||
} finally {
|
||||
signer.destroy()
|
||||
await fx.cleanup()
|
||||
}
|
||||
})
|
||||
})
|
||||
@@ -1,6 +1,8 @@
|
||||
// Human key lifecycle (#119): destroy() ends signing.
|
||||
import { expect, test } from "bun:test"
|
||||
import { HumanKeyError, sealHumanKey, unlockHumanKey } from "../src/approval/keystore.ts"
|
||||
import { writeFile } from "node:fs/promises"
|
||||
import path from "node:path"
|
||||
import { HumanKeyError, hasHumanKey, sealHumanKey, unlockHumanKey } from "../src/approval/keystore.ts"
|
||||
import { gitRepo } from "./helpers.ts"
|
||||
|
||||
const PASSPHRASE = "test-passphrase-1234"
|
||||
@@ -17,3 +19,14 @@ test("destroy zeroes the key and further signs throw", async () => {
|
||||
fx.cleanup()
|
||||
}
|
||||
})
|
||||
|
||||
test("hasHumanKey requires both halves: a lone human.key is a brick, not a key (#213)", async () => {
|
||||
const fx = await gitRepo()
|
||||
try {
|
||||
expect(await hasHumanKey(fx.state)).toBe(false)
|
||||
await writeFile(path.join(fx.state, "human.key"), '{"version":1}\n')
|
||||
expect(await hasHumanKey(fx.state)).toBe(false)
|
||||
} finally {
|
||||
fx.cleanup()
|
||||
}
|
||||
})
|
||||
|
||||
@@ -7,6 +7,8 @@ import { tmpdir } from "node:os"
|
||||
import path from "node:path"
|
||||
import type { BrainstormIdea, BrainstormScore } from "../src/index.ts"
|
||||
import {
|
||||
AGENTS,
|
||||
BRAINSTORM_FANOUT_SPAWNS,
|
||||
brainstormRunPaths,
|
||||
brainstormTools,
|
||||
buildPlan,
|
||||
@@ -455,3 +457,22 @@ describe("callable runs (#108)", () => {
|
||||
}
|
||||
})
|
||||
})
|
||||
|
||||
describe("nested refusal at depth 1 (#269)", () => {
|
||||
test("lens subagents and the critic spawn nothing; only primaries fan out", () => {
|
||||
const byId = new Map(AGENTS.map((agent) => [agent.id, agent]))
|
||||
// The mechanism: empty `spawns` + deny-by-default + the foreground guard.
|
||||
// A lens or the critic asked to launch a subagent has nothing to launch.
|
||||
for (const lens of LENSES) {
|
||||
const agent = byId.get(`es-lens-${lens.id}`)!
|
||||
expect(agent, `es-lens-${lens.id}`).toBeDefined()
|
||||
expect(agent.spawns).toEqual([])
|
||||
}
|
||||
expect(byId.get("es-brainstorm-critic")!.spawns).toEqual([])
|
||||
// The calling primaries fan out themselves, at depth 1, in the foreground.
|
||||
for (const id of ["brainstormer", "grill", "factory"]) {
|
||||
const spawns = byId.get(id)!.spawns
|
||||
for (const fanout of BRAINSTORM_FANOUT_SPAWNS) expect(spawns, id).toContain(fanout)
|
||||
}
|
||||
})
|
||||
})
|
||||
|
||||
@@ -0,0 +1,107 @@
|
||||
// Issue #186 integration pins, core side (bug #241 closers): the verifySpan
|
||||
// short-excerpt branch at es_audit_verdict level. The schema keeps excerpt
|
||||
// min(1); the 12-char witness in verifySpan is what refuses — this file pins
|
||||
// that current behavior without weakening either.
|
||||
import { afterEach, beforeEach, describe, expect, test } from "bun:test"
|
||||
import { mkdir, writeFile } from "node:fs/promises"
|
||||
import path from "node:path"
|
||||
import { type HumanSigner, sealHumanKey, unlockHumanKey } from "../src/approval/keystore.ts"
|
||||
import { codeAuditTools } from "../src/codeaudit/index.ts"
|
||||
import { Factory } from "../src/factory/index.ts"
|
||||
import { factoryLayout } from "../src/layout.ts"
|
||||
import { esTools } from "../src/ops/tools.ts"
|
||||
import { verifySpan } from "../src/research/quote.ts"
|
||||
import { AuditFindingSchema } from "../src/schema/codeaudit.ts"
|
||||
import { type Fixture, gitRepo } from "./helpers.ts"
|
||||
|
||||
const AUTH = [
|
||||
"export function login(db: Db, user: string) {",
|
||||
` const sql = "SELECT * FROM users WHERE name = '" + user + "'"`,
|
||||
" return db.query(sql)",
|
||||
"}",
|
||||
"",
|
||||
].join("\n")
|
||||
|
||||
let fx: Fixture
|
||||
let factory: Factory
|
||||
const PASSPHRASE = "test-passphrase-1234"
|
||||
let signer: HumanSigner
|
||||
beforeEach(async () => {
|
||||
fx = await gitRepo("es-audit-186-")
|
||||
await sealHumanKey(PASSPHRASE, fx.state)
|
||||
signer = await unlockHumanKey(PASSPHRASE, fx.state)
|
||||
void signer
|
||||
factory = new Factory(fx.root, {
|
||||
gates: async () => ({ passed: true, findings: [], summary: "green" }),
|
||||
stateDir: fx.state,
|
||||
})
|
||||
await mkdir(path.join(fx.root, "src"), { recursive: true })
|
||||
await writeFile(path.join(fx.root, "src/auth.ts"), AUTH)
|
||||
})
|
||||
afterEach(() => fx.cleanup())
|
||||
|
||||
const tools = () => {
|
||||
const ops = { root: fx.root, factory, stateDir: fx.state }
|
||||
const all = [...codeAuditTools(ops), ...esTools(ops)]
|
||||
return (name: string) => all.find((tool) => tool.name === name)!
|
||||
}
|
||||
|
||||
const injection = {
|
||||
kind: "vulnerability",
|
||||
title: "SQL built by string concatenation",
|
||||
severity: "high",
|
||||
cwe: "CWE-89",
|
||||
file: "src/auth.ts",
|
||||
lines: [2, 3],
|
||||
excerpt: `"SELECT * FROM users WHERE name = '" + user`,
|
||||
detail: "user = \"' OR '1'='1\" returns every row; the query is concatenated, not parameterised.",
|
||||
remediation: "Use db.query('SELECT * FROM users WHERE name = ?', [user]).",
|
||||
}
|
||||
|
||||
async function openPathAudit() {
|
||||
return (await factory.openAudit("human:tester", { kind: "path", path: "src" }, { ceilingUSD: 5 })).audit
|
||||
}
|
||||
|
||||
describe("verifySpan short branch (#241)", () => {
|
||||
test("an 11-char span is too short; a 12-char verbatim span is evidence", () => {
|
||||
const source = "return db.query(sql)"
|
||||
expect(verifySpan(source, "db.query(sq")).toEqual({
|
||||
ok: false,
|
||||
reason: "too short to be evidence (need at least 12 characters)",
|
||||
})
|
||||
expect(verifySpan(source, "db.query(sql")).toEqual({ ok: true, index: 7 })
|
||||
expect(verifySpan(source, "db.query(no)").ok).toBe(false)
|
||||
})
|
||||
|
||||
test("the schema still accepts a short excerpt; the witness is what refuses", () => {
|
||||
// #241 item 2 premise: schema/codeaudit.ts promises "at least 12
|
||||
// characters" in the comment but enforces min(1). Pin the behavior:
|
||||
// parsing succeeds, so only the witness stands in the way.
|
||||
const parsed = AuditFindingSchema.safeParse({ ...injection, lines: [3, 3], excerpt: "db.query" })
|
||||
expect(parsed.success).toBe(true)
|
||||
})
|
||||
})
|
||||
|
||||
describe("short excerpts at es_audit_verdict (#241)", () => {
|
||||
test("a <12-char excerpt refuses the whole verdict with 'too short to be evidence'; nothing is recorded", async () => {
|
||||
const audit = await openPathAudit()
|
||||
const verdict = tools()("es_audit_verdict")
|
||||
const short = { ...injection, lines: [3, 3], excerpt: "db.query" }
|
||||
await expect(
|
||||
verdict.execute(
|
||||
{ audit: audit.id, verdict: "fail", findings: [short], notes: "n" },
|
||||
{ agent: "es-auditor-antithesis" },
|
||||
),
|
||||
).rejects.toThrow("too short to be evidence")
|
||||
// One short excerpt among good findings still refuses everything.
|
||||
await expect(
|
||||
verdict.execute(
|
||||
{ audit: audit.id, verdict: "fail", findings: [injection, short], notes: "n" },
|
||||
{ agent: "es-auditor-antithesis" },
|
||||
),
|
||||
).rejects.toThrow("1 finding(s) do not point at real code")
|
||||
// Nothing reached the state, the records or the dossier.
|
||||
expect((await factory.read())!.audits[0]!.antithesis).toBeUndefined()
|
||||
expect(await Bun.file(factoryLayout(fx.root).auditRecords(audit.id)).exists()).toBe(false)
|
||||
})
|
||||
})
|
||||
@@ -143,6 +143,22 @@ describe("the line-pointer law at es_audit_verdict", () => {
|
||||
expect(await Bun.file(factoryLayout(fx.root).auditRecords(audit.id)).exists()).toBe(false)
|
||||
})
|
||||
|
||||
test("an excerpt under 12 characters is refused as too short to be evidence (#241)", async () => {
|
||||
const audit = await openPathAudit()
|
||||
const verdict = tools()("es_audit_verdict")
|
||||
// "db.query" (8 chars) is verbatim on line 3, so only the witness's
|
||||
// 12-char floor (verifySpan) can refuse it — pins the short branch at
|
||||
// verdict level, not just in the sibling span tests.
|
||||
const short = { ...injection, lines: [3, 3], excerpt: "db.query" }
|
||||
await expect(
|
||||
verdict.execute(
|
||||
{ audit: audit.id, verdict: "fail", findings: [short], notes: "n" },
|
||||
{ agent: "es-auditor-antithesis" },
|
||||
),
|
||||
).rejects.toThrow(/too short to be evidence/)
|
||||
expect((await factory.read())!.audits[0]!.antithesis).toBeUndefined()
|
||||
})
|
||||
|
||||
test("malformed findings and verdicts that contradict their findings are refused", async () => {
|
||||
const audit = await openPathAudit()
|
||||
const verdict = tools()("es_audit_verdict")
|
||||
|
||||
@@ -0,0 +1,107 @@
|
||||
// Audit #203 — layered model tiers, global → project → plugin options.
|
||||
//
|
||||
// The full three-layer conflict (each layer wins a different tier) through
|
||||
// loadEsConfig, the strict side of the unknown-key contract (config files
|
||||
// reject unknowns, including 0.7-era keys — plugin options only warn, proven
|
||||
// in packages/opencode/test/config-layers-203.test.ts), and the human-only
|
||||
// boundary for `es config set` through the shell policy.
|
||||
import { afterEach, beforeEach, describe, expect, test } from "bun:test"
|
||||
import { mkdir, mkdtemp, rm, writeFile } from "node:fs/promises"
|
||||
import { tmpdir } from "node:os"
|
||||
import path from "node:path"
|
||||
import { loadEsConfig } from "../src/index.ts"
|
||||
import { evaluateShell, invokesHumanOnly } from "../src/trust/index.ts"
|
||||
|
||||
let root: string
|
||||
let env: NodeJS.ProcessEnv
|
||||
beforeEach(async () => {
|
||||
root = await mkdtemp(path.join(tmpdir(), "es-203-layers-"))
|
||||
env = { XDG_CONFIG_HOME: path.join(root, "xdg") }
|
||||
})
|
||||
afterEach(() => rm(root, { recursive: true, force: true }))
|
||||
|
||||
const writeGlobal = async (value: unknown) => {
|
||||
const file = path.join(env.XDG_CONFIG_HOME!, "epistemic-swarm", "config.json")
|
||||
await mkdir(path.dirname(file), { recursive: true })
|
||||
await writeFile(file, JSON.stringify(value))
|
||||
return file
|
||||
}
|
||||
const writeProject = async (value: unknown) => {
|
||||
await mkdir(path.join(root, ".factory"), { recursive: true })
|
||||
const file = path.join(root, ".factory", "config.json")
|
||||
await writeFile(file, JSON.stringify(value))
|
||||
return file
|
||||
}
|
||||
|
||||
describe("three-layer tier precedence (#203)", () => {
|
||||
test("each layer wins a different tier: options > project > global", async () => {
|
||||
const global = await writeGlobal({
|
||||
models: { fast: "global/fast", balanced: "global/balanced", deep: "global/deep" },
|
||||
})
|
||||
const project = await writeProject({ models: { fast: "project/fast", deep: "project/deep" } })
|
||||
const loaded = await loadEsConfig(root, { env, overrides: { models: { fast: "options/fast" } } })
|
||||
expect(loaded.sources).toEqual([global, project])
|
||||
expect(loaded.config.models).toEqual({
|
||||
fast: "options/fast",
|
||||
balanced: "global/balanced",
|
||||
deep: "project/deep",
|
||||
})
|
||||
})
|
||||
|
||||
test("a lower layer survives wherever a higher layer is silent", async () => {
|
||||
await writeGlobal({ models: { deep: "global/deep" }, pr: "off" })
|
||||
await writeProject({ pr: "gh" })
|
||||
const loaded = await loadEsConfig(root, { env, overrides: { afterEdit: "off" } })
|
||||
expect(loaded.config.models).toEqual({ deep: "global/deep" })
|
||||
expect(loaded.config.pr).toBe("gh")
|
||||
expect(loaded.config.afterEdit).toBe("off")
|
||||
})
|
||||
})
|
||||
|
||||
describe("config-file unknowns are rejected, strictly (#203)", () => {
|
||||
test.each(["search_engine", "max_iterations", "mode"])(
|
||||
"project file: 0.7-era key %s is rejected, naming it",
|
||||
async (key: string) => {
|
||||
await writeProject({ [key]: key === "max_iterations" ? 2 : "x" })
|
||||
await expect(loadEsConfig(root, { env })).rejects.toThrow(new RegExp(`invalid config.*${key}`))
|
||||
},
|
||||
)
|
||||
|
||||
test("project file: a typo key is rejected with the source named", async () => {
|
||||
const project = await writeProject({ modles: { fast: "a/fast" } })
|
||||
await expect(loadEsConfig(root, { env })).rejects.toThrow(new RegExp(`invalid config.*${project}.*modles|modles`))
|
||||
})
|
||||
|
||||
test("global file: unknown keys are rejected too", async () => {
|
||||
await writeGlobal({ search_engine: "duckduckgo" })
|
||||
await expect(loadEsConfig(root, { env })).rejects.toThrow(/invalid config.*search_engine/)
|
||||
})
|
||||
|
||||
test("control: known keys in both files still load", async () => {
|
||||
await writeGlobal({ research: { depth: 1 } })
|
||||
await writeProject({ research: { depth: 3 } })
|
||||
expect((await loadEsConfig(root, { env })).config.research).toEqual({ depth: 3 })
|
||||
})
|
||||
})
|
||||
|
||||
describe("es config set stays human-only (#203)", () => {
|
||||
const shell = (agent: string | undefined, command: string) =>
|
||||
evaluateShell({ root, stateDir: path.join(root, "state") }, agent, command, { sandboxAvailable: true })
|
||||
|
||||
test("invokesHumanOnly reads config set, however flagged; config show stays a read", () => {
|
||||
expect(invokesHumanOnly("es config set models.deep x/y")).toBe(true)
|
||||
expect(invokesHumanOnly("es --cwd . config set models.deep x/y")).toBe(true)
|
||||
expect(invokesHumanOnly("es config set models.deep x/y --global")).toBe(true)
|
||||
expect(invokesHumanOnly("es config show")).toBe(false)
|
||||
expect(invokesHumanOnly("es --cwd . config show")).toBe(false)
|
||||
})
|
||||
|
||||
test("every agent shell is denied config set; config show is allowed", () => {
|
||||
for (const agent of ["build", "factory", "es-programmer", undefined])
|
||||
expect([agent, shell(agent, "es config set models.deep x/y").effect]).toEqual([agent, "deny"])
|
||||
const denied = shell("build", "es config set models.deep x/y")
|
||||
expect(denied.effect).toBe("deny")
|
||||
if (denied.effect === "deny") expect(denied.reason).toContain("human-only")
|
||||
expect(shell("build", "es config show").effect).toBe("allow")
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,151 @@
|
||||
// Full seat × control-path enumeration at the unit level (#182). The sampled
|
||||
// proofs live in policy.test.ts; this suite pins EVERY registry seat against
|
||||
// every control glob (relative, absolute and dot-dot spellings), plus the
|
||||
// shell layer split: seats are refused by the CONTROL_MENTION text rule
|
||||
// except the two paths it does not name, which pass to the sandbox backstop.
|
||||
// The same matrix runs against the real host in
|
||||
// packages/opencode/test/control-matrix.test.ts.
|
||||
import { afterAll, beforeAll, describe, expect, test } from "bun:test"
|
||||
import { mkdir, mkdtemp, rm, symlink, writeFile } from "node:fs/promises"
|
||||
import { tmpdir } from "node:os"
|
||||
import path from "node:path"
|
||||
import { AGENTS } from "../src/agents/registry.ts"
|
||||
import { evaluateShell, evaluateWrite } from "../src/trust/index.ts"
|
||||
|
||||
/** One representative per FACTORY_CONTROL / CONFIG_CONTROL glob (trust/control.ts). */
|
||||
const CONTROL_PATHS = [
|
||||
".factory/gates.json",
|
||||
".factory/config.json",
|
||||
".factory/frontier.json",
|
||||
".factory/approvals/spec.json",
|
||||
".factory/waivers/lint.json",
|
||||
".factory/runtime/state.json",
|
||||
".factory/STOP",
|
||||
".factory/git-hooks/pre-commit",
|
||||
".factory/brainstorm/scores.json",
|
||||
".factory/brainstorm/runs/grill-1/ideas.json",
|
||||
".factory/harvest/plan.json",
|
||||
".factory/harvest/runs/grill-1/plan.json",
|
||||
".factory/harvest/widget/profile.json",
|
||||
".factory/harvest/clean-room/widget-search.md",
|
||||
".factory/design/tokens.json",
|
||||
".factory/design/components/button/tokens.json",
|
||||
`.factory/research/sources/${"a".repeat(64)}.md`,
|
||||
".factory/research/coverage.json",
|
||||
".factory/research/dossier.json",
|
||||
".factory/research/brief.pcrb.json",
|
||||
".factory/claims/retractions.json",
|
||||
".factory/improve/proposal.json",
|
||||
".factory/audits/audit-01.json",
|
||||
".factory/scout/plan.json",
|
||||
".factory/scout/widget/profile.json",
|
||||
".factory/worktrees/phase-01/.factory/approvals/spec.json",
|
||||
".git/config",
|
||||
".git/hooks/pre-commit",
|
||||
"opencode.jsonc",
|
||||
".claude/settings.json",
|
||||
]
|
||||
|
||||
const FACTORY_PATHS = CONTROL_PATHS.filter(
|
||||
(target) => target !== "opencode.jsonc" && target !== ".claude/settings.json",
|
||||
)
|
||||
|
||||
/** Paths the shell text rule does not name; seats pass them to the sandbox. */
|
||||
const SANDBOX_BACKSTOP = new Set([".factory/harvest/clean-room/widget-search.md", ".factory/improve/proposal.json"])
|
||||
|
||||
let root: string
|
||||
let state: string
|
||||
let worktree: string
|
||||
|
||||
beforeAll(async () => {
|
||||
root = await mkdtemp(path.join(tmpdir(), "es-matrix-core-"))
|
||||
state = await mkdtemp(path.join(tmpdir(), "es-matrix-core-state-"))
|
||||
worktree = path.join(root, ".factory/worktrees/p1")
|
||||
await mkdir(path.join(root, ".factory/approvals"), { recursive: true })
|
||||
await mkdir(path.join(root, "src"), { recursive: true })
|
||||
await mkdir(path.join(worktree, "src"), { recursive: true })
|
||||
await writeFile(path.join(root, ".factory/approvals/spec.json"), "{}\n")
|
||||
await symlink(path.join(root, ".factory/approvals"), path.join(root, "src/link"))
|
||||
})
|
||||
afterAll(async () => {
|
||||
await rm(root, { recursive: true, force: true })
|
||||
await rm(state, { recursive: true, force: true })
|
||||
})
|
||||
|
||||
const ctx = () => ({ root, worktree, stateDir: state })
|
||||
|
||||
describe("control matrix: every seat is denied every control path (#182)", () => {
|
||||
test("evaluateWrite denies all seats, however the path is spelled", () => {
|
||||
for (const spec of AGENTS) {
|
||||
for (const target of CONTROL_PATHS) {
|
||||
for (const spelling of [target, path.join(root, target), `docs/../${target}`]) {
|
||||
const decision = evaluateWrite(ctx(), spec.id, spelling)
|
||||
expect([spec.id, spelling, decision.effect]).toEqual([spec.id, spelling, "deny"])
|
||||
}
|
||||
}
|
||||
}
|
||||
})
|
||||
|
||||
test("non-seat agents are denied the factory controls but keep host semantics for config", () => {
|
||||
for (const agent of [undefined, "build"]) {
|
||||
for (const target of FACTORY_PATHS)
|
||||
expect([agent, target, evaluateWrite(ctx(), agent, target).effect]).toEqual([agent, target, "deny"])
|
||||
// Config controls ask the human for non-seat agents (host semantics).
|
||||
for (const target of ["opencode.jsonc", ".claude/settings.json"])
|
||||
expect([agent, target, evaluateWrite(ctx(), agent, target).effect]).toEqual([agent, target, "ask"])
|
||||
}
|
||||
})
|
||||
|
||||
test("shell: seats are refused at the text layer except the documented sandbox-backstop paths", () => {
|
||||
for (const spec of AGENTS) {
|
||||
for (const target of CONTROL_PATHS) {
|
||||
const decision = evaluateShell(ctx(), spec.id, `echo pwned > ${target} && echo WRITE-SUCCEEDED`, {
|
||||
sandboxAvailable: true,
|
||||
})
|
||||
if (SANDBOX_BACKSTOP.has(target)) {
|
||||
// Not named by CONTROL_MENTION: runs under the sandbox, whose
|
||||
// read-only .factory bind refuses the redirect.
|
||||
expect([spec.id, target, decision]).toMatchObject([spec.id, target, { effect: "allow", mode: "sandbox" }])
|
||||
} else {
|
||||
expect(decision.effect).toBe("deny")
|
||||
expect("reason" in decision ? decision.reason : "").toMatch(/may not reference control files/)
|
||||
}
|
||||
}
|
||||
}
|
||||
})
|
||||
|
||||
test("shell: non-seat agents are denied every control path (broad factory-dir rule)", () => {
|
||||
for (const target of CONTROL_PATHS) {
|
||||
const decision = evaluateShell(ctx(), undefined, `echo pwned > ${target}`, { sandboxAvailable: false })
|
||||
expect([target, decision.effect]).toEqual([target, "deny"])
|
||||
}
|
||||
})
|
||||
|
||||
test("case variants are denied in the editor and the shell", () => {
|
||||
const variants = [
|
||||
".FACTORY/GATES.json",
|
||||
".factory/GATES.JSON",
|
||||
".Factory/Approvals/Spec.JSON",
|
||||
".GIT/CONFIG",
|
||||
"OpenCode.JSONC",
|
||||
".CLAUDE/Settings.JSON",
|
||||
]
|
||||
for (const spec of ["es-manager", "factory", "es-programmer", "es-qa-functional"]) {
|
||||
for (const target of variants) {
|
||||
expect([spec, target, evaluateWrite(ctx(), spec, target).effect]).toEqual([spec, target, "deny"])
|
||||
const shell = evaluateShell(ctx(), spec, `echo pwned > ${target}`, { sandboxAvailable: true })
|
||||
expect([spec, target, shell.effect]).toEqual([spec, target, "deny"])
|
||||
}
|
||||
}
|
||||
})
|
||||
|
||||
test("a symlink from an allowed dir into .factory cannot be used to write a control file", () => {
|
||||
for (const spec of ["es-manager", "factory", "build", undefined]) {
|
||||
for (const target of ["src/link/spec.json", path.join(root, "src/link/spec.json")]) {
|
||||
const decision = evaluateWrite(ctx(), spec, target)
|
||||
expect([spec, target, decision.effect]).toEqual([spec, target, "deny"])
|
||||
expect("reason" in decision ? decision.reason : "").toContain("factory control file")
|
||||
}
|
||||
}
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,170 @@
|
||||
// Per-pack coverage closers for #256 (audit #190 follow-up): every shipped
|
||||
// pack's banned domains and threshold/weight values pin current behavior, the
|
||||
// below-threshold refusal path at machine.ts:545 is exercised, and the
|
||||
// SECONDARY_MEDIA / tierDomains dead config is documented as observed.
|
||||
// Scoring itself is NOT changed here (#252 owns the scoring defects: tierless
|
||||
// weight, NK inflation, garbage tags); these tests pin the mechanism only.
|
||||
import { afterEach, beforeEach, describe, expect, test } from "bun:test"
|
||||
import { mkdir, writeFile } from "node:fs/promises"
|
||||
import path from "node:path"
|
||||
import { type HumanSigner, sealHumanKey, unlockHumanKey } from "../src/approval/keystore.ts"
|
||||
import {
|
||||
type Constitution,
|
||||
claimVerdict,
|
||||
constitutionFromPack,
|
||||
Factory,
|
||||
factoryLayout,
|
||||
gateRunner,
|
||||
LEGACY_CONSTITUTION,
|
||||
loadDomainPack,
|
||||
recordApproval,
|
||||
researchSourcesDir,
|
||||
SourceCache,
|
||||
} from "../src/index.ts"
|
||||
import { researchOptions } from "../src/research/ops.ts"
|
||||
import { classifySourceTier } from "../src/research/tier.ts"
|
||||
import { type Fixture, frontier, gitRepo } from "./helpers.ts"
|
||||
|
||||
const pack = async (id: string): Promise<Constitution> => constitutionFromPack(await loadDomainPack(id))
|
||||
|
||||
describe("each shipped pack bans its own domains (#256.1)", () => {
|
||||
const cases = [
|
||||
{ packId: "quant", domain: "vendor-whitepaper-mirror.example" },
|
||||
{ packId: "legal", domain: "undisclosed-sponsored-content.example" },
|
||||
{ packId: "biopharma", domain: "predatory-journal.example" },
|
||||
] as const
|
||||
for (const { packId, domain } of cases) {
|
||||
test(`${packId} rejects a source on ${domain} (BANNED_DOMAIN), other packs and legacy accept it`, async () => {
|
||||
const claim = {
|
||||
id: `${packId}-banned`,
|
||||
tag: "VERIFIED",
|
||||
source: {
|
||||
sha256: "b".repeat(64),
|
||||
quote: "the whitepaper claims linear scaling on every desk",
|
||||
url: `https://${domain}/research-note`,
|
||||
},
|
||||
}
|
||||
const own = claimVerdict(claim, { constitution: await pack(packId) })
|
||||
expect(own.verdict).toBe("REJECTED")
|
||||
expect(own.reasons.some((reason) => reason.startsWith("BANNED_DOMAIN"))).toBe(true)
|
||||
// Pack-agnostic baseline unchanged: legacy accepts the same claim.
|
||||
expect(claimVerdict(claim, { constitution: LEGACY_CONSTITUTION }).verdict).toBe("ACCEPTED")
|
||||
// Specificity: a sibling pack with a different ban list accepts it.
|
||||
const sibling = cases.find((entry) => entry.packId !== packId)!.packId
|
||||
expect(claimVerdict(claim, { constitution: await pack(sibling) }).verdict).toBe("ACCEPTED")
|
||||
})
|
||||
}
|
||||
})
|
||||
|
||||
describe("pack threshold and weight values are pinned (#256.1)", () => {
|
||||
test("accept thresholds: biopharma 0.7, quant 0.7, legal 0.65", async () => {
|
||||
expect((await pack("biopharma")).acceptThreshold).toBe(0.7)
|
||||
expect((await pack("quant")).acceptThreshold).toBe(0.7)
|
||||
expect((await pack("legal")).acceptThreshold).toBe(0.65)
|
||||
})
|
||||
|
||||
test("spot tier weights: biopharma downweights preprints, legal prefers primary press, quant values docs", async () => {
|
||||
expect((await pack("biopharma")).tierWeights.PREPRINT).toBe(0.3)
|
||||
expect((await pack("biopharma")).tierWeights.PEER_REVIEWED).toBe(1.0)
|
||||
expect((await pack("legal")).tierWeights.PRIMARY_PRESS).toBe(1.0)
|
||||
expect((await pack("legal")).tierWeights.PREPRINT).toBe(0.2)
|
||||
expect((await pack("quant")).tierWeights.TECHNICAL_DOCUMENTATION).toBe(1.0)
|
||||
expect((await pack("quant")).tierWeights.PREPRINT).toBe(0.5)
|
||||
})
|
||||
|
||||
test("ban lists and retraction policies are pinned per pack", async () => {
|
||||
expect((await pack("biopharma")).bannedDomains).toEqual([
|
||||
"predatory-journal.example",
|
||||
"retracted-mirror.example",
|
||||
"banned-preprint-mirror.example",
|
||||
])
|
||||
expect((await pack("quant")).bannedDomains).toEqual(["vendor-whitepaper-mirror.example"])
|
||||
expect((await pack("legal")).bannedDomains).toEqual(["undisclosed-sponsored-content.example"])
|
||||
expect((await pack("biopharma")).retractionPolicy).toBe("zero_tolerance")
|
||||
expect((await pack("quant")).retractionPolicy).toBe("standard")
|
||||
expect((await pack("legal")).retractionPolicy).toBe("standard")
|
||||
expect((await pack("quant")).negBonus).toBe(0.75)
|
||||
expect((await pack("quant")).rejectPenalty).toBe(3.0)
|
||||
})
|
||||
})
|
||||
|
||||
describe("the below-threshold refusal path (#256.2)", () => {
|
||||
let fx: Fixture
|
||||
const PASSPHRASE = "test-passphrase-1234"
|
||||
let signer: HumanSigner
|
||||
beforeEach(async () => {
|
||||
fx = await gitRepo()
|
||||
await sealHumanKey(PASSPHRASE, fx.state)
|
||||
signer = await unlockHumanKey(PASSPHRASE, fx.state)
|
||||
})
|
||||
afterEach(() => fx.cleanup())
|
||||
|
||||
test("a preprint-only report passes every per-claim rule yet is refused below the biopharma threshold", async () => {
|
||||
const factory = new Factory(fx.root, {
|
||||
gates: gateRunner({ stateDir: fx.state }),
|
||||
stateDir: fx.state,
|
||||
domainPack: "biopharma",
|
||||
})
|
||||
await factory.begin("human:tester")
|
||||
await factory.writeFrontier("grill", frontier())
|
||||
await recordApproval(fx.root, { stage: "frontier", channel: "cli", approvedBy: "tester", signer })
|
||||
await factory.beginResearch("human:tester")
|
||||
// arxiv.org classifies PREPRINT (weight 0.3 under biopharma), so one
|
||||
// verified claim scores 0.3 < 0.7 with no per-claim rule firing.
|
||||
const cache = new SourceCache(researchSourcesDir(fx.root), fx.state)
|
||||
const source = await cache.put({
|
||||
url: "https://arxiv.org/abs/2601.00001",
|
||||
text: "A preprint model predicts surrogate endpoints in early cohorts.",
|
||||
provider: "fetch",
|
||||
})
|
||||
expect(source.meta.tier).toBe("PREPRINT")
|
||||
await mkdir(path.dirname(factoryLayout(fx.root).researchReport), { recursive: true })
|
||||
await writeFile(
|
||||
factoryLayout(fx.root).researchReport,
|
||||
`- A preprint predicts surrogate endpoints [VERIFIED: sha256:${source.meta.sha256} "predicts surrogate endpoints"]\n`,
|
||||
)
|
||||
let message: string | undefined
|
||||
try {
|
||||
await factory.completeResearch("factory")
|
||||
} catch (caught) {
|
||||
message = caught instanceof Error ? caught.message : String(caught)
|
||||
}
|
||||
expect(message).toBeDefined()
|
||||
// The machine.ts:545 threshold refusal, not the per-claim rejection path.
|
||||
expect(message).toContain("below its threshold")
|
||||
expect(message).toContain("biopharma")
|
||||
expect(message).not.toContain("rejects")
|
||||
})
|
||||
})
|
||||
|
||||
describe("dead tier config, documented as observed (#256.3 — do not fix here)", () => {
|
||||
test("SECONDARY_MEDIA is weighted in all three packs but the classifier never emits it", async () => {
|
||||
// Observed: every shipped pack carries a SECONDARY_MEDIA weight ...
|
||||
expect((await loadDomainPack("biopharma")).tierWeights.SECONDARY_MEDIA).toBe(0.2)
|
||||
expect((await loadDomainPack("quant")).tierWeights.SECONDARY_MEDIA).toBe(0.2)
|
||||
expect((await loadDomainPack("legal")).tierWeights.SECONDARY_MEDIA).toBe(0.3)
|
||||
// ... yet tier.ts has no rule returning it, not even for news-like hosts.
|
||||
for (const url of [
|
||||
"https://arxiv.org/abs/1",
|
||||
"https://doi.org/10.1/x",
|
||||
"https://docs.python.org/3/",
|
||||
"https://www.fda.gov/news/x",
|
||||
"https://www.bbc.com/news/science-1",
|
||||
"https://www.nytimes.com/2026/01/01/health/x.html",
|
||||
"https://blog.example.com/opinion",
|
||||
"https://example.test/x",
|
||||
])
|
||||
expect([url, classifySourceTier(url)]).toEqual([url, expect.not.stringMatching("SECONDARY_MEDIA")])
|
||||
})
|
||||
|
||||
test("tierDomains overrides reach no caller: no shipped pack sets them and researchOptions drops them", async () => {
|
||||
// Observed: the schema, cache.put and ops plumbing accept tierDomains,
|
||||
// but no shipped pack file sets them ...
|
||||
for (const id of ["biopharma", "quant", "legal"])
|
||||
expect([(await loadDomainPack(id)).packId, (await loadDomainPack(id)).tierDomains]).toEqual([id, undefined])
|
||||
// ... and the config-to-tools mapping never forwards them (verified by
|
||||
// grep: zero assignments to ResearchOpsContext.tierDomains repo-wide), so
|
||||
// a pack can never re-tier a host at runtime today.
|
||||
expect("tierDomains" in researchOptions({ research: { depth: 2 } })).toBe(false)
|
||||
})
|
||||
})
|
||||
@@ -205,7 +205,7 @@ describe("the pack gates completeResearch", () => {
|
||||
test("a clean report passes under biopharma and records the pack and score", async () => {
|
||||
const factory = await toResearch("biopharma")
|
||||
await report(
|
||||
"https://journal.example/clean",
|
||||
"https://doi.org/10.1/clean",
|
||||
"The cohort met the primary efficacy bar in the per-protocol analysis.",
|
||||
"primary efficacy bar",
|
||||
)
|
||||
@@ -240,6 +240,18 @@ describe("the pack gates completeResearch", () => {
|
||||
await expect(unpacked.completeResearch("factory")).resolves.toBeDefined()
|
||||
})
|
||||
|
||||
test("a below-threshold report is refused at completion, naming the score (#256.2)", async () => {
|
||||
// A preprint-only report passes every claim verdict under biopharma but
|
||||
// scores 0.3 against its 0.7 threshold: the machine.ts refusal path.
|
||||
const factory = await toResearch("biopharma")
|
||||
await report(
|
||||
"https://arxiv.org/abs/1234",
|
||||
"A model predicts surrogate endpoints in this preprint.",
|
||||
"predicts surrogate endpoints",
|
||||
)
|
||||
await expect(factory.completeResearch("factory")).rejects.toThrow("below its threshold")
|
||||
})
|
||||
|
||||
test("an unknown configured pack fails closed at completion time", async () => {
|
||||
const factory = await toResearch("no-such-pack")
|
||||
await report("https://journal.example/clean", "A finding rests on a cached source.", "cached source")
|
||||
@@ -274,7 +286,10 @@ describe("source tiers reach claims and the score (#113)", () => {
|
||||
classifySourceTier("https://www.nature.com/articles/x"),
|
||||
classifySourceTier("https://docs.python.org/3/"),
|
||||
classifySourceTier("https://developer.mozilla.org/en/"),
|
||||
// A docs./developer. prefix on an unlisted host earns no tier (#252):
|
||||
// the prefix rule would let docs.<attacker> claim curated weight.
|
||||
classifySourceTier("https://docs.example.com/x"),
|
||||
classifySourceTier("https://developer.example.com/y"),
|
||||
classifySourceTier("https://www.fda.gov/news/x"),
|
||||
classifySourceTier("https://journal.example/clean"),
|
||||
classifySourceTier("not a url"),
|
||||
@@ -288,7 +303,8 @@ describe("source tiers reach claims and the score (#113)", () => {
|
||||
"PEER_REVIEWED",
|
||||
"TECHNICAL_DOCUMENTATION",
|
||||
"TECHNICAL_DOCUMENTATION",
|
||||
"TECHNICAL_DOCUMENTATION",
|
||||
undefined,
|
||||
undefined,
|
||||
"PRIMARY_PRESS",
|
||||
undefined,
|
||||
undefined,
|
||||
@@ -373,8 +389,13 @@ describe("source tiers reach claims and the score (#113)", () => {
|
||||
)
|
||||
expect(peer.score).toBe(1)
|
||||
expect(pre.score).toBe(0.3)
|
||||
// Unknown tiers fall back to __default__.
|
||||
expect(computeEpistemicScoreFromClaims([{ id: "c".repeat(64), tag: "VERIFIED" }], constitution).score).toBe(1)
|
||||
// Unknown tiers fall back to __default__, which a pack clamps to its
|
||||
// lowest verified tier weight (#252): tierless must never outscore
|
||||
// verified evidence.
|
||||
expect(weightFor(constitution, undefined)).toBeLessThanOrEqual(0.3)
|
||||
expect(computeEpistemicScoreFromClaims([{ id: "c".repeat(64), tag: "VERIFIED" }], constitution).score).toBe(
|
||||
weightFor(constitution, undefined),
|
||||
)
|
||||
})
|
||||
|
||||
test("tiers do not change claim identity: PCRB export and verify still pass", async () => {
|
||||
@@ -395,3 +416,119 @@ describe("source tiers reach claims and the score (#113)", () => {
|
||||
expect(claimId({ ...without, tier: "PREPRINT" } as never)).toBe(without.id)
|
||||
})
|
||||
})
|
||||
|
||||
describe("scoring integrity (#252)", () => {
|
||||
test("tierless claims weigh at the lowest verified tier, never full weight", async () => {
|
||||
// Unclassifiable hosts used to fall back to __default__ 1.0 and outscore
|
||||
// verified preprints (0.3) under biopharma: a perverse incentive to cite
|
||||
// worse sources. A pack now clamps __default__ to its lowest tier weight.
|
||||
const pack = await loadDomainPack("biopharma")
|
||||
const constitution = constitutionFromPack(pack)
|
||||
const floor = Math.min(
|
||||
...Object.entries(pack.tierWeights)
|
||||
.filter(([tier]) => tier !== "__default__")
|
||||
.map(([, weight]) => weight),
|
||||
)
|
||||
expect(weightFor(constitution, undefined)).toBe(floor)
|
||||
const tierless = computeEpistemicScoreFromClaims([{ id: "c".repeat(64), tag: "VERIFIED" }], constitution)
|
||||
const preprint = computeEpistemicScoreFromClaims(
|
||||
[{ id: "d".repeat(64), tag: "VERIFIED", tier: "PREPRINT" }],
|
||||
constitution,
|
||||
)
|
||||
expect(tierless.score).toBe(floor)
|
||||
expect(tierless.score).toBeLessThanOrEqual(preprint.score)
|
||||
// The no-pack constitution is untouched (legacy numbers preserved).
|
||||
expect(weightFor(LEGACY_CONSTITUTION, undefined)).toBe(1)
|
||||
})
|
||||
|
||||
test("negative knowledge cannot inflate the score past the threshold", async () => {
|
||||
// totalAssertions excluded negKnowledge while the numerator added
|
||||
// negBonus per row: 10xPREPRINT went 0.3 -> 1.0 with 14 NK lines. NK rows
|
||||
// now count as assertions too.
|
||||
const pack = await loadDomainPack("biopharma")
|
||||
const constitution = constitutionFromPack(pack)
|
||||
const verified = Array.from({ length: 10 }, (_, index) => ({
|
||||
id: `v${index}`.padEnd(64, "0"),
|
||||
tag: "VERIFIED" as const,
|
||||
tier: "PREPRINT" as const,
|
||||
}))
|
||||
const nk = Array.from({ length: 14 }, (_, index) => ({
|
||||
id: `n${index}`.padEnd(64, "0"),
|
||||
tag: "NEGATIVE_KNOWLEDGE" as const,
|
||||
query: `query ${index}`,
|
||||
finding: `finding ${index}`,
|
||||
}))
|
||||
const scored = computeEpistemicScoreFromClaims([...verified, ...nk], constitution)
|
||||
expect(scored.breakdown.negativeKnowledgeCount).toBe(14)
|
||||
expect(scored.score).toBeLessThan(constitution.acceptThreshold)
|
||||
})
|
||||
|
||||
test("mandatory tags enforce membership, not just presence", async () => {
|
||||
const pack = await biopharma()
|
||||
const garbage = claimVerdict({ id: "g", tag: "GARBAGE" }, { constitution: pack })
|
||||
expect(garbage.verdict).toBe("REJECTED")
|
||||
expect(garbage.reasons.some((reason) => reason.startsWith("MISSING_TAG"))).toBe(true)
|
||||
expect(
|
||||
claimVerdict({ id: "v", tag: "VERIFIED", source: { sha256: "a".repeat(64), quote: "q" } }, { constitution: pack })
|
||||
.verdict,
|
||||
).toBe("ACCEPTED")
|
||||
})
|
||||
})
|
||||
|
||||
describe("pack coverage (#256)", () => {
|
||||
test("each pack pins its thresholds, weights and penalties", async () => {
|
||||
// A typo or deletion in a pack file stays green under mechanism-only
|
||||
// tests, so the values are pinned exactly (SECONDARY_MEDIA was removed:
|
||||
// the classifier never emits it, #256.3).
|
||||
const biopharma = await loadDomainPack("biopharma")
|
||||
expect(biopharma.tierWeights).toEqual({
|
||||
__default__: 1,
|
||||
PEER_REVIEWED: 1,
|
||||
TECHNICAL_DOCUMENTATION: 0.8,
|
||||
PRIMARY_PRESS: 0.7,
|
||||
PREPRINT: 0.3,
|
||||
})
|
||||
expect([biopharma.acceptThreshold, biopharma.negBonus, biopharma.rejectPenalty]).toEqual([0.7, 0.5, 2.5])
|
||||
expect(biopharma.retractionPolicy).toBe("zero_tolerance")
|
||||
const quant = await loadDomainPack("quant")
|
||||
expect(quant.tierWeights).toEqual({
|
||||
__default__: 1,
|
||||
PEER_REVIEWED: 0.9,
|
||||
TECHNICAL_DOCUMENTATION: 1,
|
||||
PRIMARY_PRESS: 0.6,
|
||||
PREPRINT: 0.5,
|
||||
})
|
||||
expect([quant.acceptThreshold, quant.negBonus, quant.rejectPenalty]).toEqual([0.7, 0.75, 3])
|
||||
const legal = await loadDomainPack("legal")
|
||||
expect(legal.tierWeights).toEqual({
|
||||
__default__: 1,
|
||||
PEER_REVIEWED: 0.8,
|
||||
TECHNICAL_DOCUMENTATION: 0.7,
|
||||
PRIMARY_PRESS: 1,
|
||||
PREPRINT: 0.2,
|
||||
})
|
||||
expect([legal.acceptThreshold, legal.negBonus, legal.rejectPenalty]).toEqual([0.65, 0.5, 2.5])
|
||||
})
|
||||
|
||||
test("quant and legal ban their own mirrors, not just biopharma's", async () => {
|
||||
// Only biopharma's banned domains appeared in fixtures; per-pack
|
||||
// behavior is asserted directly here.
|
||||
const quant = constitutionFromPack(await loadDomainPack("quant"))
|
||||
const legal = constitutionFromPack(await loadDomainPack("legal"))
|
||||
const bio = await biopharma()
|
||||
const vendorMirror = {
|
||||
id: "q",
|
||||
tag: "VERIFIED",
|
||||
source: { sha256: "b".repeat(64), quote: "backtest", url: "https://vendor-whitepaper-mirror.example/p" },
|
||||
}
|
||||
expect(claimVerdict(vendorMirror, { constitution: quant }).verdict).toBe("REJECTED")
|
||||
expect(claimVerdict(vendorMirror, { constitution: bio }).verdict).toBe("ACCEPTED")
|
||||
const sponsored = {
|
||||
id: "l",
|
||||
tag: "VERIFIED",
|
||||
source: { sha256: "c".repeat(64), quote: "holding", url: "https://undisclosed-sponsored-content.example/x" },
|
||||
}
|
||||
expect(claimVerdict(sponsored, { constitution: legal }).verdict).toBe("REJECTED")
|
||||
expect(claimVerdict(sponsored, { constitution: bio }).verdict).toBe("ACCEPTED")
|
||||
})
|
||||
})
|
||||
|
||||
@@ -90,6 +90,8 @@ describe("transcripts", () => {
|
||||
expect(transcript.tokens).toEqual({ input: 150, output: 15, reasoning: 2, cacheRead: 4, cacheWrite: 6 })
|
||||
// Missing or non-numeric fields count as zero rather than poisoning the sum.
|
||||
expect(readTranscript([{ type: "step_finish", part: { cost: "n/a" } }]).costUSD).toBe(0)
|
||||
// Negative costs clamp to zero: a hostile stream must not unbind spend (#211).
|
||||
expect(readTranscript([{ type: "step_finish", part: { cost: -5 } }]).costUSD).toBe(0)
|
||||
})
|
||||
|
||||
test("a completed call's output is graded text, except raw evidence and errored calls", () => {
|
||||
@@ -133,6 +135,10 @@ describe("step caps", () => {
|
||||
expect(caseStepCap(undefined, 16)).toBe(DEFAULT_CASE_STEPS)
|
||||
expect(caseStepCap(undefined, 3)).toBe(3)
|
||||
})
|
||||
|
||||
test("a NaN maxSteps falls back to the default instead of disabling the cap (#211)", () => {
|
||||
expect(caseStepCap(Number.NaN, 16)).toBe(DEFAULT_CASE_STEPS)
|
||||
})
|
||||
})
|
||||
|
||||
describe("fire graders", () => {
|
||||
|
||||
@@ -0,0 +1,188 @@
|
||||
// Issue #187 §2 + bug #251 fuzz-adjacent edges (deterministic, no fuzz harness).
|
||||
//
|
||||
// Pins the CURRENT normalisation semantics of `verifyQuote` (NFKC,
|
||||
// typography folds, markdown stripping, whitespace collapse; no entity
|
||||
// decoding) and the CURRENT auditor downgrade rules (VERIFIED verbatim
|
||||
// grounds, paraphrases and planted entries stay ungrounded). Bugs #251
|
||||
// (fuzz corpus, legacy unsealed constructor) and #254 (wording) are for the
|
||||
// parent: this suite tests behaviour as-is and reports defects as findings,
|
||||
// never by changing product code.
|
||||
import { afterEach, beforeEach, describe, expect, test } from "bun:test"
|
||||
import { mkdir, mkdtemp, rm, writeFile } from "node:fs/promises"
|
||||
import { tmpdir } from "node:os"
|
||||
import path from "node:path"
|
||||
import { auditMarkdown, htmlToText, normalizeSourceText, SourceCache, sha256, verifyQuote } from "../src/index.ts"
|
||||
|
||||
let dir: string
|
||||
let state: string
|
||||
beforeEach(async () => {
|
||||
dir = await mkdtemp(path.join(tmpdir(), "es-evidence-187-"))
|
||||
state = await mkdtemp(path.join(tmpdir(), "es-evidence-187-state-"))
|
||||
})
|
||||
afterEach(() => Promise.all([rm(dir, { recursive: true, force: true }), rm(state, { recursive: true, force: true })]))
|
||||
|
||||
const sealed = () => new SourceCache(path.join(dir, "sources"), state)
|
||||
|
||||
const GATES = "Alpha gates keep agents honest under heavy load conditions today, witnesses agree."
|
||||
|
||||
describe("quote verifier fuzz-adjacent edges (#187 §2, gaps listed in #251)", () => {
|
||||
test("zero-width chars: ZWSP/FEFF fold to space and still match; ZWNJ/ZWJ/SHY/word-joiner refuse", () => {
|
||||
const plain = "hello world friends and more text here"
|
||||
// U+200B ZERO WIDTH SPACE and U+FEFF are in the typography fold to space.
|
||||
expect(verifyQuote(plain, "hello world friends and more").ok).toBe(true)
|
||||
expect(verifyQuote(plain, "hello world friends and more").ok).toBe(true)
|
||||
// An invisible join inside a word folds to a word break (current behaviour).
|
||||
expect(verifyQuote("helloworld friends and more text here", "hello world friends and more").ok).toBe(true)
|
||||
// U+200C/U+200D/U+00AD/U+2060 are NOT folded: the quote is refused.
|
||||
for (const invisible of ["", "", "", ""]) {
|
||||
const quote = `hello${invisible} world friends and more`
|
||||
expect([JSON.stringify(invisible), verifyQuote(plain, quote)]).toMatchObject([
|
||||
JSON.stringify(invisible),
|
||||
{ ok: false },
|
||||
])
|
||||
expect(verifyQuote(plain, quote).reason).toContain("not found verbatim")
|
||||
// And the reverse direction refuses too: the source keeps the char.
|
||||
expect(verifyQuote(`hello${invisible} world friends and more text here`, plain.slice(0, 30)).ok).toBe(false)
|
||||
}
|
||||
})
|
||||
|
||||
test("`..` stays literal, `...`/`…` split into ordered fragments, `....` leaves a stray dot", () => {
|
||||
// Two dots are code/punctuation, never an ellipsis.
|
||||
expect(verifyQuote("version 1..2 released today for all users", "version 1..2 released today").ok).toBe(true)
|
||||
expect(verifyQuote("version 1..2 released today for all users", "version 1..2 ... released today for all").ok).toBe(
|
||||
true,
|
||||
)
|
||||
// Three dots and the single-char ellipsis both split; order matters.
|
||||
expect(verifyQuote(GATES, "Alpha gates keep ... honest under heavy load").ok).toBe(true)
|
||||
expect(verifyQuote(GATES, "Alpha gates keep … honest under heavy load").ok).toBe(true)
|
||||
expect(verifyQuote(GATES, "honest under heavy load ... Alpha gates keep").ok).toBe(false)
|
||||
// Four dots are an ellipsis plus a literal dot: the stray dot is refused.
|
||||
const four = verifyQuote(GATES, "Alpha gates keep .... honest under heavy load")
|
||||
expect(four.ok).toBe(false)
|
||||
expect(four.reason).toContain("not found verbatim")
|
||||
})
|
||||
|
||||
test("entity-at-quote: the verifier does no entity decoding (the fetch layer decodes)", () => {
|
||||
const amp = htmlToText("<p>Fish & chips are tasty indeed yum.</p>")
|
||||
const numeric = htmlToText("<p>Fish & chips are tasty indeed yum.</p>")
|
||||
// Both spellings cache to the same decoded bytes.
|
||||
expect(amp.text).toBe("Fish & chips are tasty indeed yum.")
|
||||
expect(numeric.text).toBe(amp.text)
|
||||
// The decoded quote verifies; the still-encoded spellings are refused.
|
||||
expect(verifyQuote(amp.text, "Fish & chips are tasty").ok).toBe(true)
|
||||
for (const encoded of ["Fish & chips are tasty", "Fish & chips are tasty"]) {
|
||||
const check = verifyQuote(amp.text, encoded)
|
||||
expect([encoded, check.ok]).toEqual([encoded, false])
|
||||
expect(check.reason).toContain("not found verbatim")
|
||||
}
|
||||
})
|
||||
|
||||
test("markdown-rendered quotes: emphasis and links strip on both sides", () => {
|
||||
expect(verifyQuote("the **quick brown fox jumps** over fences high", "quick brown fox jumps over").ok).toBe(true)
|
||||
expect(verifyQuote("the quick brown fox jumps over fences high", "the **quick brown fox jumps** over").ok).toBe(
|
||||
true,
|
||||
)
|
||||
expect(verifyQuote("run __every__ `single` test file now please", "run every single test file now").ok).toBe(true)
|
||||
expect(verifyQuote("[Bun](https://bun.sh) is fast and furious indeed", "Bun is fast and furious").ok).toBe(true)
|
||||
expect(verifyQuote("Bun is fast and furious indeed ok", "[Bun](https://bun.sh) is fast and furious").ok).toBe(true)
|
||||
})
|
||||
|
||||
test("short-fragment refusal: every ellipsis fragment and every single quote needs 12+ characters", () => {
|
||||
expect(verifyQuote(GATES, "short").reason).toContain("too short")
|
||||
expect(verifyQuote(GATES, "abcdefghijk").ok).toBe(false)
|
||||
expect(verifyQuote("abcdefghijkl rest of the text here ok", "abcdefghijkl").ok).toBe(true)
|
||||
const short = verifyQuote(GATES, "Alpha gates keep ... hi ... honest under heavy load")
|
||||
expect(short.ok).toBe(false)
|
||||
expect(short.reason).toContain("fragment")
|
||||
expect(verifyQuote(GATES, "Alpha gates keep ... honest under heavy load").ok).toBe(true)
|
||||
})
|
||||
|
||||
test("unicode normalisation: NFKC, curly quotes, dashes and NBSP fold before matching", () => {
|
||||
expect(verifyQuote("filigree work is quite delicate indeed", "filigree work is quite").ok).toBe(true)
|
||||
expect(verifyQuote("café society is quite lively today", "café society is quite").ok).toBe(true)
|
||||
// Composed vs decomposed acute accents match after NFKC.
|
||||
expect(verifyQuote("café society is quite lively today", "café society is quite").ok).toBe(true)
|
||||
expect(verifyQuote("He said “hello world, friends” ok", 'said "hello world, friends"').ok).toBe(true)
|
||||
expect(verifyQuote("fast — reliable service for all users", "fast - reliable service").ok).toBe(true)
|
||||
expect(verifyQuote("hello world friends and more text", "hello world friends and more").ok).toBe(true)
|
||||
// Case still matters: normalisation is not case folding.
|
||||
expect(verifyQuote(GATES, "alpha gates keep agents honest").ok).toBe(false)
|
||||
})
|
||||
})
|
||||
|
||||
describe("auditor downgrade rules (#187: verbatim grounds, paraphrase and plants do not)", () => {
|
||||
test("verbatim VERIFIED grounds; a paraphrase stays ungrounded (flagged, never re-tagged)", async () => {
|
||||
const cache = sealed()
|
||||
const source = await cache.put({
|
||||
url: "https://gates.test/run",
|
||||
text: "Mechanical gates catch regressions before review, which keeps autonomous agents honest under heavy load.",
|
||||
provider: "fetch",
|
||||
})
|
||||
const good = await auditMarkdown(
|
||||
`- Gates catch regressions early [VERIFIED: sha256:${source.meta.sha256} "catch regressions before review"]`,
|
||||
cache,
|
||||
)
|
||||
expect(good.coverage).toMatchObject({ claims: 1, grounded: 1, ungrounded: 0 })
|
||||
expect(good.passed).toBe(true)
|
||||
expect(good.cited).toEqual([source.meta.sha256])
|
||||
const paraphrased = await auditMarkdown(
|
||||
`- Gates catch failures ahead of inspection [VERIFIED: sha256:${source.meta.sha256} "catch failures ahead of inspection"]`,
|
||||
cache,
|
||||
)
|
||||
expect(paraphrased.coverage).toMatchObject({ claims: 1, grounded: 0, ungrounded: 1 })
|
||||
expect(paraphrased.passed).toBe(false)
|
||||
expect(paraphrased.claims[0]?.reason).toContain("not found verbatim")
|
||||
})
|
||||
|
||||
test("a planted unsealed entry is refused by the sealed cache; the legacy constructor still reads it (#251)", async () => {
|
||||
const text = "Planted findings about gate latency on reference hardware."
|
||||
const hash = sha256(normalizeSourceText(text))
|
||||
await mkdir(path.join(dir, "sources"), { recursive: true })
|
||||
await writeFile(path.join(dir, "sources", `${hash}.md`), normalizeSourceText(text))
|
||||
await writeFile(
|
||||
path.join(dir, "sources", `${hash}.json`),
|
||||
`${JSON.stringify({ sha256: hash, url: "https://evil.test/planted", retrieved: new Date().toISOString(), provider: "fetch", bytes: Buffer.byteLength(normalizeSourceText(text)) })}\n`,
|
||||
)
|
||||
expect(await sealed().get(hash)).toBeUndefined()
|
||||
const audit = await auditMarkdown(
|
||||
`- Planted latency wins [VERIFIED: sha256:${hash} "gate latency on reference"]`,
|
||||
sealed(),
|
||||
)
|
||||
expect(audit.coverage).toMatchObject({ claims: 1, grounded: 0, ungrounded: 1 })
|
||||
expect(audit.claims[0]?.reason).toContain("not in the cache")
|
||||
expect(audit.passed).toBe(false)
|
||||
// Current legacy behaviour (bug #251): without a state dir nothing is verified.
|
||||
expect((await new SourceCache(path.join(dir, "sources")).get(hash))?.text).toContain("gate latency")
|
||||
})
|
||||
|
||||
test("short and missing quotes stay ungrounded; substantive other tags ground, stubs are malformed", async () => {
|
||||
const cache = sealed()
|
||||
const source = await cache.put({
|
||||
url: "https://gates.test/short",
|
||||
text: "Mechanical gates catch regressions before review, which keeps autonomous agents honest under heavy load.",
|
||||
provider: "fetch",
|
||||
})
|
||||
const audit = await auditMarkdown(
|
||||
[
|
||||
`- Too short [VERIFIED: sha256:${source.meta.sha256} "tiny"]`,
|
||||
`- Missing quote [VERIFIED: no quote here]`,
|
||||
`- Fewer tools to install [INFERRED: one binary covers the runner and the bundler]`,
|
||||
`- Maybe faster [HYPOTHESIS: rerun the benchmark with n=30]`,
|
||||
`- Nothing else found [NEGATIVE_KNOWLEDGE: searched for gate failures on ARM]`,
|
||||
`- Stub [INFERRED: ok]`,
|
||||
"- No tag at all",
|
||||
].join("\n"),
|
||||
cache,
|
||||
)
|
||||
// "tiny" fails verification (ungrounded); "no quote here" has no
|
||||
// source/quote shape and the stub INFERRED body is malformed.
|
||||
expect(audit.coverage).toMatchObject({
|
||||
claims: 7,
|
||||
grounded: 3,
|
||||
ungrounded: 1,
|
||||
untagged: 1,
|
||||
malformed: 2,
|
||||
})
|
||||
expect(audit.passed).toBe(false)
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,90 @@
|
||||
// #251.1: verifier fuzz corpus. Pins the quote verifier against the
|
||||
// acceptance list in the issue: zero-width/bidi/compatibility characters,
|
||||
// `..` vs `...` vs `....`, entities at the quote, and markdown-rendered
|
||||
// quotes. Static review found no false accepts; these tests pin that.
|
||||
import { describe, expect, test } from "bun:test"
|
||||
import { normalizeForQuote, verifyQuote } from "../src/research/quote.ts"
|
||||
|
||||
const SRC = "The quick brown fox jumps over the lazy dog near the riverbank today."
|
||||
|
||||
describe("zero-width, bidi and compatibility characters (#251.1)", () => {
|
||||
test("U+200B folds to a word break (documented TYPOGRAPHY fold, pinned)", () => {
|
||||
// U+200B sits at the end of the folded whitespace range, so a quote
|
||||
// carrying it matches a source with a plain space, and vice versa.
|
||||
expect(verifyQuote(SRC, "quick brown fox\u200Bjumps over the").ok).toBe(true)
|
||||
expect(verifyQuote("quick brown fox\u200Bjumps over the lazy dog today", "quick brown fox jumps over").ok).toBe(
|
||||
true,
|
||||
)
|
||||
expect(normalizeForQuote("a\u200Bb")).toBe("a b")
|
||||
})
|
||||
|
||||
test("joiners, soft hyphen, word joiner and bidi controls never fold: quotes carrying them refuse", () => {
|
||||
// None of U+200C U+200D U+00AD U+2060 U+200E U+200F U+202A U+202C U+2066
|
||||
// is in the fold table, so the normalised quote keeps the control and
|
||||
// misses the source.
|
||||
for (const ch of ["\u200C", "\u200D", "\u00AD", "\u2060", "\u200E", "\u200F", "\u202A", "\u202C", "\u2066"]) {
|
||||
const check = verifyQuote(SRC, `quick brown fox${ch} jumps over the`)
|
||||
expect([ch.codePointAt(0)!.toString(16), check.ok]).toEqual([ch.codePointAt(0)!.toString(16), false])
|
||||
expect(check.reason).toContain("not found verbatim")
|
||||
}
|
||||
})
|
||||
|
||||
test("compatibility characters fold through NFKC (documented, pinned)", () => {
|
||||
expect(verifyQuote("The \uFB01ligree parses correctly here", "filigree parses correctly").ok).toBe(true)
|
||||
expect(verifyQuote("The ABC label is printed here", "\uFF21\uFF22\uFF23 label is printed").ok).toBe(true)
|
||||
})
|
||||
|
||||
test("matching is case-sensitive: casing alone never folds", () => {
|
||||
expect(verifyQuote(SRC, "QUICK BROWN FOX JUMPS OVER").ok).toBe(false)
|
||||
})
|
||||
})
|
||||
|
||||
describe("ellipsis edge cases: `..` vs `...` vs `....` (#251.1)", () => {
|
||||
test("two dots are literal: no fragment split", () => {
|
||||
expect(verifyQuote("Version 1..2 released yesterday morning", "Version 1..2 released").ok).toBe(true)
|
||||
expect(verifyQuote("Version 1..2 released yesterday morning", "Version 1..3 released").ok).toBe(false)
|
||||
})
|
||||
|
||||
test("three dots (or …) split into ordered fragments, each 12+ characters", () => {
|
||||
expect(verifyQuote(SRC, "quick brown fox ... lazy dog near").ok).toBe(true)
|
||||
expect(verifyQuote(SRC, "quick brown fox \u2026 lazy dog near").ok).toBe(true)
|
||||
expect(verifyQuote(SRC, "lazy dog near ... quick brown fox").ok).toBe(false)
|
||||
})
|
||||
|
||||
test("four dots are an ellipsis plus a literal dot that must match", () => {
|
||||
// "...." splits on the first "...", leaving ". lazy dog near" starting
|
||||
// with a dot the source does not have there.
|
||||
const check = verifyQuote(SRC, "quick brown fox .... lazy dog near")
|
||||
expect(check.ok).toBe(false)
|
||||
expect(check.reason).toContain("not found verbatim")
|
||||
})
|
||||
})
|
||||
|
||||
describe("entities at the quote (#251.1)", () => {
|
||||
test("no entity decoding on either side: `&`, `&` and `&` are distinct literals", () => {
|
||||
const plain = "Fish & chips are tasty indeed"
|
||||
const encoded = "Fish & chips are tasty indeed"
|
||||
const numeric = "Fish & chips are tasty indeed"
|
||||
expect(verifyQuote(plain, "Fish & chips are tasty").ok).toBe(true)
|
||||
expect(verifyQuote(plain, "Fish & chips are tasty").ok).toBe(false)
|
||||
expect(verifyQuote(plain, "Fish & chips are tasty").ok).toBe(false)
|
||||
expect(verifyQuote(encoded, "Fish & chips are tasty").ok).toBe(false)
|
||||
expect(verifyQuote(numeric, "Fish & chips are tasty").ok).toBe(false)
|
||||
// Identical literals still match.
|
||||
expect(verifyQuote(encoded, "Fish & chips are tasty").ok).toBe(true)
|
||||
})
|
||||
})
|
||||
|
||||
describe("markdown-rendered quotes (#251.1)", () => {
|
||||
test("links, emphasis and code spans strip on both sides (documented, pinned)", () => {
|
||||
expect(verifyQuote("[Bun](https://bun.sh) is fast and furious today", "Bun is fast and furious").ok).toBe(true)
|
||||
expect(verifyQuote("This is **very important** news today ok", "This is very important news").ok).toBe(true)
|
||||
expect(verifyQuote("Run `bun add zod` to install it now", "Run bun add zod to install").ok).toBe(true)
|
||||
// Emphasis marks in the quote match a plain source (documented strip).
|
||||
expect(verifyQuote("Bun is fast and furious today", "**Bun** is fast and furious").ok).toBe(true)
|
||||
})
|
||||
|
||||
test("a link target is not quotable text", () => {
|
||||
expect(verifyQuote("[Bun](https://bun.sh) is fast and furious today", "https://bun.sh is fast and").ok).toBe(false)
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,81 @@
|
||||
// #251.2: the URL index is sealed with the engine key. A sealed cache
|
||||
// writes an `index.seal` sidecar (HMAC over the canonical index); `byUrl`
|
||||
// refuses a tampered index instead of following it. A missing sidecar stays
|
||||
// readable (caches written before sealing keep working); an unsealed cache
|
||||
// writes no sidecar.
|
||||
import { describe, expect, test } from "bun:test"
|
||||
import { mkdtemp, rm, writeFile } from "node:fs/promises"
|
||||
import { tmpdir } from "node:os"
|
||||
import path from "node:path"
|
||||
import { SourceCache } from "../src/research/cache.ts"
|
||||
|
||||
const put2 = async (cache: SourceCache) => {
|
||||
const a = await cache.put({
|
||||
url: "https://a.test/page",
|
||||
text: "Alpha source text with enough length.",
|
||||
provider: "fetch",
|
||||
})
|
||||
const b = await cache.put({
|
||||
url: "https://b.test/page",
|
||||
text: "Beta source text with enough length here.",
|
||||
provider: "fetch",
|
||||
})
|
||||
return { a, b }
|
||||
}
|
||||
|
||||
describe("sealed URL index (#251.2)", () => {
|
||||
test("a sealed cache writes an HMAC sidecar next to index.json", async () => {
|
||||
const dir = await mkdtemp(path.join(tmpdir(), "es-index-"))
|
||||
const state = await mkdtemp(path.join(tmpdir(), "es-index-state-"))
|
||||
try {
|
||||
const sealed = new SourceCache(path.join(dir, "sealed"), state)
|
||||
await put2(sealed)
|
||||
const seal = await Bun.file(path.join(dir, "sealed", "index.seal")).text()
|
||||
expect(seal.trim()).toMatch(/^[0-9a-f]{64}$/)
|
||||
} finally {
|
||||
await rm(dir, { recursive: true, force: true })
|
||||
await rm(state, { recursive: true, force: true })
|
||||
}
|
||||
})
|
||||
|
||||
test("byUrl refuses a tampered index instead of serving the remapped entry", async () => {
|
||||
const dir = await mkdtemp(path.join(tmpdir(), "es-index-tamper-"))
|
||||
const state = await mkdtemp(path.join(tmpdir(), "es-index-tamper-state-"))
|
||||
try {
|
||||
const cache = new SourceCache(path.join(dir, "sources"), state)
|
||||
const { a, b } = await put2(cache)
|
||||
expect((await cache.byUrl("https://a.test/page"))?.meta.sha256).toBe(a.meta.sha256)
|
||||
// Remap a.test to b's bytes without re-sealing the index.
|
||||
const indexFile = path.join(dir, "sources", "index.json")
|
||||
const index = JSON.parse(await Bun.file(indexFile).text()) as Record<string, string>
|
||||
index["https://a.test/page"] = b.meta.sha256
|
||||
await writeFile(indexFile, `${JSON.stringify(index, null, 2)}\n`)
|
||||
expect(await cache.byUrl("https://a.test/page")).toBeUndefined()
|
||||
// Direct reads by hash are unaffected.
|
||||
expect((await cache.get(b.meta.sha256))?.text).toContain("Beta source")
|
||||
} finally {
|
||||
await rm(dir, { recursive: true, force: true })
|
||||
await rm(state, { recursive: true, force: true })
|
||||
}
|
||||
})
|
||||
|
||||
test("an index written before sealing (no sidecar) stays readable", async () => {
|
||||
const dir = await mkdtemp(path.join(tmpdir(), "es-index-legacy-"))
|
||||
const state = await mkdtemp(path.join(tmpdir(), "es-index-legacy-state-"))
|
||||
try {
|
||||
const cache = new SourceCache(path.join(dir, "sources"), state)
|
||||
const entry = await cache.put({
|
||||
url: "https://old.test/page",
|
||||
text: "Old source text with enough length here.",
|
||||
provider: "fetch",
|
||||
})
|
||||
// An index from before sidecars existed carries sealed entries but no
|
||||
// sidecar: the sealed reader keeps serving it by URL.
|
||||
await rm(path.join(dir, "sources", "index.seal"), { force: true })
|
||||
expect((await cache.byUrl("https://old.test/page"))?.meta.sha256).toBe(entry.meta.sha256)
|
||||
} finally {
|
||||
await rm(dir, { recursive: true, force: true })
|
||||
await rm(state, { recursive: true, force: true })
|
||||
}
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,118 @@
|
||||
// Concurrent runtime-state writes hold the lock (issue #179, §2).
|
||||
//
|
||||
// Two concurrent es_qa_verdict tool calls (functional + adversarial) both
|
||||
// persist — no torn JSON, no lost verdict — and two concurrent journal steps
|
||||
// both land. State reads during the race never observe a half-signed
|
||||
// (state, sidecar) pair: a refusal would reject the Promise.all.
|
||||
import { afterEach, beforeEach, describe, expect, test } from "bun:test"
|
||||
import { mkdtemp } from "node:fs/promises"
|
||||
import { tmpdir } from "node:os"
|
||||
import path from "node:path"
|
||||
import { recordApproval } from "../src/approval/index.ts"
|
||||
import type { HumanSigner } from "../src/approval/keystore.ts"
|
||||
import { sealHumanKey, unlockHumanKey } from "../src/approval/keystore.ts"
|
||||
import { Factory } from "../src/factory/index.ts"
|
||||
import { once, readJournal } from "../src/factory/journal.ts"
|
||||
import { gateRunner } from "../src/gates/index.ts"
|
||||
import { factoryLayout } from "../src/layout.ts"
|
||||
import { esTools } from "../src/ops/tools.ts"
|
||||
import { type Fixture, frontier, gitRepo, writeReport, writeSpecs } from "./helpers.ts"
|
||||
|
||||
let fx: Fixture
|
||||
let signer: HumanSigner
|
||||
const green = { passed: true, findings: [], summary: "green" } as const
|
||||
const gates = async () => green
|
||||
|
||||
beforeEach(async () => {
|
||||
fx = await gitRepo("es-stage-conc-")
|
||||
await sealHumanKey("test-passphrase-1234", fx.state)
|
||||
signer = await unlockHumanKey("test-passphrase-1234", fx.state)
|
||||
})
|
||||
afterEach(() => fx.cleanup())
|
||||
|
||||
const factory = () => new Factory(fx.root, { gates, stateDir: fx.state })
|
||||
const approve = (stage: "frontier" | "spec") =>
|
||||
recordApproval(fx.root, { stage, channel: "cli", approvedBy: "tester", signer })
|
||||
|
||||
async function toQa() {
|
||||
const f = factory()
|
||||
await f.begin("human:tester")
|
||||
await f.writeFrontier("grill", frontier())
|
||||
await approve("frontier")
|
||||
await f.beginResearch("human:tester")
|
||||
await writeReport(fx.root, fx.state)
|
||||
await f.completeResearch("factory")
|
||||
await writeSpecs(fx.root, ["alpha"])
|
||||
await approve("spec")
|
||||
await f.startBuild("factory")
|
||||
const dir = (await f.activeWorktree())!
|
||||
const { mkdir, writeFile } = await import("node:fs/promises")
|
||||
await mkdir(path.join(dir, "src"), { recursive: true })
|
||||
await mkdir(path.join(dir, "test"), { recursive: true })
|
||||
await writeFile(path.join(dir, "test/alpha.test.ts"), `test("alpha", () => {})\n`)
|
||||
await f.recordGateRun("es-programmer", {
|
||||
passed: false,
|
||||
findings: [],
|
||||
summary: "red",
|
||||
failedTests: ["test/alpha.test.ts"],
|
||||
})
|
||||
await writeFile(path.join(dir, "src/alpha.ts"), `export const alpha = 1\n`)
|
||||
const done = await f.complete("es-programmer")
|
||||
expect(done.ok).toBe(true)
|
||||
expect((await f.read())!.stage).toBe("QA")
|
||||
}
|
||||
|
||||
describe("concurrent runtime-state writes (issue #179)", () => {
|
||||
test("two concurrent es_qa_verdict tool calls both persist; reads stay sealed", async () => {
|
||||
await toQa()
|
||||
const f = factory()
|
||||
const tools = esTools({ root: fx.root, factory: f, stateDir: fx.state })
|
||||
const verdict = tools.find((tool) => tool.name === "es_qa_verdict")!
|
||||
const [functional, adversarial, seen] = await Promise.all([
|
||||
verdict.execute({ verdict: "pass", notes: "criteria met" }, { agent: "es-qa-functional" }),
|
||||
verdict.execute({ verdict: "pass", notes: "nothing broke" }, { agent: "es-qa-adversarial" }),
|
||||
f.read(),
|
||||
])
|
||||
expect(typeof functional).toBe("string")
|
||||
expect(typeof adversarial).toBe("string")
|
||||
expect([functional, adversarial].some((text) => text.includes("RELEASE"))).toBe(true)
|
||||
// The concurrent read saw a sealed, parseable snapshot (never torn):
|
||||
// it may win the race (QA) or lose it (RELEASE); only progress matters.
|
||||
expect(["QA", "RELEASE"]).toContain(seen?.stage)
|
||||
const state = (await f.read())!
|
||||
expect([state.stage, state.phases[0]!.status]).toEqual(["RELEASE", "passed"])
|
||||
expect(state.phases[0]!.qa).toMatchObject({
|
||||
functional: { verdict: "pass" },
|
||||
adversarial: { verdict: "pass" },
|
||||
})
|
||||
})
|
||||
|
||||
test("two concurrent journal steps both land, in full; spend races keep every event", async () => {
|
||||
await toQa()
|
||||
const f = factory()
|
||||
await Promise.all([
|
||||
once(fx.root, "concurrent-a", async () => ({ side: "a" })),
|
||||
once(fx.root, "concurrent-b", async () => ({ side: "b" })),
|
||||
])
|
||||
const steps = (await readJournal(fx.root)).map((entry) => entry.step)
|
||||
expect(steps).toContain("concurrent-a")
|
||||
expect(steps).toContain("concurrent-b")
|
||||
|
||||
// Concurrent spend charges never lose an event and never tear the file.
|
||||
await Promise.all([f.recordSpend(1, true), f.recordSpend(2, true), f.read(), f.read()])
|
||||
const state = (await f.read())!
|
||||
expect(state.spend.usd).toBe(3)
|
||||
expect(state.spend.events).toBe(2)
|
||||
// The sidecar still verifies after the race (read would throw otherwise).
|
||||
expect(state.stage).toBe("QA")
|
||||
})
|
||||
|
||||
test("ten concurrent spend writes serialize without loss", async () => {
|
||||
await toQa()
|
||||
const f = factory()
|
||||
await Promise.all(Array.from({ length: 10 }, (_, i) => f.recordSpend(1, i % 2 === 0)))
|
||||
const state = (await f.read())!
|
||||
expect(state.spend.usd).toBe(10)
|
||||
expect(state.spend.events).toBe(10)
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,95 @@
|
||||
// Issue #205 integration (self-dogfood preset): seed a temp clone through
|
||||
// library calls only and prove the gates stay untrusted until a human trusts
|
||||
// them. Never runs a human-only verb (`es factory init`, `es trust`): the only
|
||||
// human-only assertions are pipe-level policy checks via `invokesHumanOnly`,
|
||||
// and no trust entry is ever written.
|
||||
import { describe, expect, test } from "bun:test"
|
||||
import { readFile } from "node:fs/promises"
|
||||
import {
|
||||
commandSetHash,
|
||||
EsConfigSchema,
|
||||
factoryLayout,
|
||||
GatesConfigSchema,
|
||||
initFromPreset,
|
||||
invokesHumanOnly,
|
||||
isTrusted,
|
||||
RoadmapSchema,
|
||||
runGates,
|
||||
SELF_DOGFOOD_BASE_BRANCH,
|
||||
} from "../src/index.ts"
|
||||
import { gitRepo } from "./helpers.ts"
|
||||
|
||||
describe("self-dogfood preset integration (#205)", () => {
|
||||
test("init seeds config, gates, roadmap and the issue note in a temp clone", async () => {
|
||||
const fx = await gitRepo("es-205-preset-")
|
||||
try {
|
||||
const { written, baseBranch } = await initFromPreset(
|
||||
fx.root,
|
||||
"self-dogfood",
|
||||
{ issue: 205, title: "Self-dogfood preset", body: "The factory builds this repo." },
|
||||
{ actor: "human:tester" },
|
||||
)
|
||||
// Current behavior (#281 owns the base-selection question): the seed
|
||||
// pins the release base to `rewrite`.
|
||||
expect(baseBranch).toBe("rewrite")
|
||||
expect(SELF_DOGFOOD_BASE_BRANCH).toBe("rewrite")
|
||||
expect(written).toEqual([
|
||||
".factory/config.json",
|
||||
".factory/gates.json",
|
||||
".factory/roadmap.json",
|
||||
".factory/notes/idea-205.md",
|
||||
])
|
||||
const layout = factoryLayout(fx.root)
|
||||
const config = EsConfigSchema.parse(JSON.parse(await readFile(layout.config, "utf8")))
|
||||
expect(config.audit?.phase).toBe("required")
|
||||
expect(config.pr).toBe("gh")
|
||||
const gates = GatesConfigSchema.parse(JSON.parse(await readFile(layout.gates, "utf8")))
|
||||
expect(gates.commands.map((command) => command.command)).toEqual([
|
||||
"bun install --frozen-lockfile",
|
||||
"bun run check",
|
||||
"bun run docs:check",
|
||||
])
|
||||
const roadmap = RoadmapSchema.parse(JSON.parse(await readFile(layout.roadmap, "utf8")))
|
||||
expect(roadmap.baseBranch).toBe("rewrite")
|
||||
expect(roadmap.title).toContain("205")
|
||||
expect(await readFile(`${fx.root}/.factory/notes/idea-205.md`, "utf8")).toContain("# #205:")
|
||||
} finally {
|
||||
await fx.cleanup()
|
||||
}
|
||||
})
|
||||
|
||||
test("preset gates are untrusted until a human trusts them (no trust performed)", async () => {
|
||||
const fx = await gitRepo("es-205-trust-")
|
||||
try {
|
||||
await initFromPreset(
|
||||
fx.root,
|
||||
"self-dogfood",
|
||||
{ issue: 205, title: "Self-dogfood preset", body: "The factory builds this repo." },
|
||||
{ actor: "human:tester" },
|
||||
)
|
||||
const layout = factoryLayout(fx.root)
|
||||
const gates = GatesConfigSchema.parse(JSON.parse(await readFile(layout.gates, "utf8")))
|
||||
// Trust-hash refusal: the fresh state dir holds no entry for this set.
|
||||
// No `trustProject` call happens in this test (that stays human-only).
|
||||
const { hash } = await commandSetHash(fx.root, gates.commands)
|
||||
expect(await isTrusted(fx.root, hash, fx.state)).toBe(false)
|
||||
|
||||
const report = await runGates({ root: fx.root, scope: "full", stateDir: fx.state })
|
||||
expect(report.passed).toBe(false)
|
||||
expect(report.findings[0]?.rule).toBe("trust/untrusted")
|
||||
expect(report.findings[0]?.fixHint).toContain("es trust")
|
||||
expect(report.checks.find((check) => check.id === "trust")?.status).toBe("fail")
|
||||
// None of the preset commands ran: no per-command checks were recorded.
|
||||
for (const id of ["install", "check", "docs-check"])
|
||||
expect(report.checks.some((check) => check.id === id)).toBe(false)
|
||||
} finally {
|
||||
await fx.cleanup()
|
||||
}
|
||||
})
|
||||
|
||||
test("seeding and trusting stay human-only (policy assert only, nothing executed)", () => {
|
||||
expect(invokesHumanOnly("es factory init --preset self-dogfood --issue 205")).toBe(true)
|
||||
expect(invokesHumanOnly("es trust")).toBe(true)
|
||||
expect(invokesHumanOnly("es factory begin")).toBe(false)
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,281 @@
|
||||
// #253: failover proof gaps. Direct pins for `backoffFor` defaults, the
|
||||
// `auth`-kind chain continuation (failover + cooldown, not a stop), and
|
||||
// token-absence exhaustiveness: every gateway failure message, both
|
||||
// tool-output paths and every cache write carry no token. Wire-level
|
||||
// exclusivity (bytes on the socket) needs a live-network capture and stays
|
||||
// out of scope for unit tests.
|
||||
import { describe, expect, test } from "bun:test"
|
||||
import { mkdtemp, readdir, rm } from "node:fs/promises"
|
||||
import { tmpdir } from "node:os"
|
||||
import path from "node:path"
|
||||
import {
|
||||
BackendError,
|
||||
backoffFor,
|
||||
runFetchChain,
|
||||
runSearchChain,
|
||||
type SourceBackend,
|
||||
} from "../src/research/backends/index.ts"
|
||||
import { ScraperSwarmClient } from "../src/research/backends/scraper-swarm.ts"
|
||||
import { researchTools } from "../src/research/ops.ts"
|
||||
import { sha256 } from "../src/util/hash.ts"
|
||||
|
||||
const TOKEN = "swarm_sec_probe_token"
|
||||
const ENV = { ES_SCRAPER_SWARM_URL: "https://swarm.test", ES_SCRAPER_SWARM_TOKEN: TOKEN }
|
||||
|
||||
describe("backoffFor defaults (backends/index.ts:159-162, #253.1)", () => {
|
||||
test("rate_limited without Retry-After cools down 60s; an explicit value wins", () => {
|
||||
expect(backoffFor(new BackendError("rate_limited", "slow down"))).toBe(60_000)
|
||||
expect(backoffFor(new BackendError("rate_limited", "slow down", { retryAfterMs: 5_000 }))).toBe(5_000)
|
||||
expect(backoffFor(new BackendError("rate_limited", "slow down", { retryAfterMs: 0 }))).toBe(0)
|
||||
})
|
||||
|
||||
test("every other kind cools down 30s", () => {
|
||||
for (const kind of ["unavailable", "auth", "blocked", "upstream", "timeout"] as const)
|
||||
expect([kind, backoffFor(new BackendError(kind, "x"))]).toEqual([kind, 30_000])
|
||||
})
|
||||
})
|
||||
|
||||
describe("auth-kind chain continuation (#253.2)", () => {
|
||||
const goodSearch: SourceBackend = {
|
||||
id: "good",
|
||||
capabilities: { search: true, fetch: false },
|
||||
available: () => true,
|
||||
search: async (query: string) => [{ url: `https://good.test/${query}`, title: "Good", snippet: "ok" }],
|
||||
}
|
||||
const goodFetch: SourceBackend = {
|
||||
id: "steady",
|
||||
capabilities: { search: false, fetch: true },
|
||||
available: () => true,
|
||||
fetch: async (url: string) => ({ url, text: "Steady page body with enough text.", contentType: "text/plain" }),
|
||||
}
|
||||
|
||||
test("a leading 401 fails over on search and cools the backend down", async () => {
|
||||
const denied: SourceBackend = {
|
||||
id: "gateway",
|
||||
capabilities: { search: true, fetch: false },
|
||||
available: () => true,
|
||||
search: async () => {
|
||||
throw new BackendError("auth", "gateway refused: HTTP 401", { status: 401 })
|
||||
},
|
||||
}
|
||||
const cooldowns = new Map<string, { until: number; kind: BackendError["kind"] }>()
|
||||
const result = await runSearchChain([denied, goodSearch], "q", {}, { now: () => 1_000_000, cooldowns })
|
||||
expect(result.backend).toBe("good")
|
||||
expect(result.attempts).toEqual([
|
||||
{ backend: "gateway", ok: false, kind: "auth" },
|
||||
{ backend: "good", ok: true },
|
||||
])
|
||||
expect(cooldowns.get("gateway")).toEqual({ until: 1_030_000, kind: "auth" })
|
||||
})
|
||||
|
||||
test("a leading 403 fails over on fetch and cools the backend down", async () => {
|
||||
const denied: SourceBackend = {
|
||||
id: "gateway",
|
||||
capabilities: { search: false, fetch: true },
|
||||
available: () => true,
|
||||
fetch: async () => {
|
||||
throw new BackendError("auth", "gateway refused: HTTP 403", { status: 403 })
|
||||
},
|
||||
}
|
||||
const cooldowns = new Map<string, { until: number; kind: BackendError["kind"] }>()
|
||||
const result = await runFetchChain([denied, goodFetch], "https://x.test/p", {
|
||||
now: () => 2_000_000,
|
||||
cooldowns,
|
||||
})
|
||||
expect(result.backend).toBe("steady")
|
||||
expect(result.attempts).toEqual([
|
||||
{ backend: "gateway", ok: false, kind: "auth" },
|
||||
{ backend: "steady", ok: true },
|
||||
])
|
||||
expect(cooldowns.get("gateway")).toEqual({ until: 2_030_000, kind: "auth" })
|
||||
})
|
||||
})
|
||||
|
||||
describe("token absence exhaustiveness (#253.3)", () => {
|
||||
const jsonRpc = (body: unknown, status = 200, headers: Record<string, string> = {}) =>
|
||||
new Response(JSON.stringify(body), {
|
||||
status,
|
||||
headers: { "content-type": "application/json", "x-swarm-contract": "1", ...headers },
|
||||
})
|
||||
const stubFetch = (respond: (url: string, init: RequestInit) => Response, seen: { auth: string }) =>
|
||||
(async (url: string, init?: RequestInit) => {
|
||||
seen.auth = String((init?.headers as Record<string, string> | undefined)?.Authorization ?? "")
|
||||
return respond(String(url), init ?? {})
|
||||
}) as unknown as typeof fetch
|
||||
const client = (respond: (url: string, init: RequestInit) => Response, seen: { auth: string }) =>
|
||||
new ScraperSwarmClient({ url: ENV.ES_SCRAPER_SWARM_URL, token: TOKEN, fetch: stubFetch(respond, seen) })
|
||||
const failure = (promise: Promise<unknown>): Promise<Error> =>
|
||||
promise.then(
|
||||
() => new Error("no-throw"),
|
||||
(error: unknown) => error as Error,
|
||||
)
|
||||
|
||||
test("the Authorization header carries the token on search and fetch", async () => {
|
||||
const seen = { auth: "" }
|
||||
const ok = {
|
||||
jsonrpc: "2.0",
|
||||
id: 1,
|
||||
result: { structuredContent: { results: [{ url: "https://a.test", title: "A", snippet: "s" }] } },
|
||||
}
|
||||
await client(() => jsonRpc(ok), seen).search("q", 1)
|
||||
expect(seen.auth).toBe(`Bearer ${TOKEN}`)
|
||||
const content = "# Title\n\nPage body text."
|
||||
const page = {
|
||||
jsonrpc: "2.0",
|
||||
id: 1,
|
||||
result: {
|
||||
structuredContent: { content, content_sha256: sha256(content), title: "T", format: "markdown" },
|
||||
},
|
||||
}
|
||||
await client(() => jsonRpc(page), seen).fetchPage("https://a.test")
|
||||
expect(seen.auth).toBe(`Bearer ${TOKEN}`)
|
||||
})
|
||||
|
||||
test("no gateway failure message carries the token", async () => {
|
||||
const cases: Array<[string, (seen: { auth: string }) => Promise<unknown>, string]> = [
|
||||
["http-401", (seen) => client(() => jsonRpc({ detail: "nope" }, 401), seen).search("q", 1), "auth"],
|
||||
["http-403", (seen) => client(() => jsonRpc({ detail: "nope" }, 403), seen).fetchPage("https://a.test"), "auth"],
|
||||
[
|
||||
"http-429",
|
||||
(seen) => client(() => jsonRpc({ detail: "slow" }, 429, { "retry-after": "7" }), seen).search("q", 1),
|
||||
"rate_limited",
|
||||
],
|
||||
["http-500", (seen) => client(() => jsonRpc({ detail: "boom" }, 500), seen).search("q", 1), "upstream"],
|
||||
[
|
||||
"contract",
|
||||
(seen) =>
|
||||
client(() => jsonRpc({ jsonrpc: "2.0", id: 1, result: {} }, 200, { "x-swarm-contract": "9" }), seen).search(
|
||||
"q",
|
||||
1,
|
||||
),
|
||||
"upstream",
|
||||
],
|
||||
[
|
||||
"ssrf-denied",
|
||||
(seen) =>
|
||||
client(
|
||||
() =>
|
||||
jsonRpc({
|
||||
jsonrpc: "2.0",
|
||||
id: 1,
|
||||
result: { isError: true, structuredContent: { code: "ssrf_denied", message: "SSRF denied" } },
|
||||
}),
|
||||
seen,
|
||||
).fetchPage("https://a.test"),
|
||||
"blocked",
|
||||
],
|
||||
[
|
||||
"blocked-by-policy",
|
||||
(seen) =>
|
||||
client(
|
||||
() =>
|
||||
jsonRpc({
|
||||
jsonrpc: "2.0",
|
||||
id: 1,
|
||||
result: { isError: true, structuredContent: { code: "blocked_by_policy", message: "denied" } },
|
||||
}),
|
||||
seen,
|
||||
).search("q", 1),
|
||||
"blocked",
|
||||
],
|
||||
[
|
||||
"upstream-timeout",
|
||||
(seen) =>
|
||||
client(
|
||||
() =>
|
||||
jsonRpc({
|
||||
jsonrpc: "2.0",
|
||||
id: 1,
|
||||
result: { isError: true, structuredContent: { code: "upstream_timeout", message: "slow" } },
|
||||
}),
|
||||
seen,
|
||||
).search("q", 1),
|
||||
"timeout",
|
||||
],
|
||||
[
|
||||
"tool-upstream",
|
||||
(seen) =>
|
||||
client(() => jsonRpc({ jsonrpc: "2.0", id: 1, error: { code: -32602, message: "bad params" } }), seen).search(
|
||||
"q",
|
||||
1,
|
||||
),
|
||||
"upstream",
|
||||
],
|
||||
[
|
||||
"unreachable",
|
||||
(seen) =>
|
||||
new ScraperSwarmClient({
|
||||
url: ENV.ES_SCRAPER_SWARM_URL,
|
||||
token: TOKEN,
|
||||
fetch: (async () => {
|
||||
seen.auth = "unreached"
|
||||
throw new TypeError("fetch failed")
|
||||
}) as unknown as typeof fetch,
|
||||
}).search("q", 1),
|
||||
"unavailable",
|
||||
],
|
||||
]
|
||||
for (const [name, run, kind] of cases) {
|
||||
const seen = { auth: "" }
|
||||
const error = await failure(run(seen))
|
||||
if (!(error instanceof BackendError)) throw new Error(`${name}: expected a BackendError, got ${error}`)
|
||||
expect([name, error.kind]).toEqual([name, kind])
|
||||
expect([name, error.message.includes(TOKEN)]).toEqual([name, false])
|
||||
}
|
||||
})
|
||||
|
||||
test("search and fetch tool outputs and every cache write carry no token", async () => {
|
||||
const dir = await mkdtemp(path.join(tmpdir(), "es-token-"))
|
||||
const state = await mkdtemp(path.join(tmpdir(), "es-token-state-"))
|
||||
try {
|
||||
// Drive the real client through the tools: search (no content cached)
|
||||
// then fetch (page cached sealed as scraper-swarm).
|
||||
const content = "# Recorded\n\nRecorded body text for the token probe."
|
||||
const searchBody = {
|
||||
jsonrpc: "2.0",
|
||||
id: 1,
|
||||
result: { structuredContent: { results: [{ url: "https://a.test/p", title: "A", snippet: "s" }] } },
|
||||
}
|
||||
const pageBody = {
|
||||
jsonrpc: "2.0",
|
||||
id: 1,
|
||||
result: {
|
||||
structuredContent: { content, content_sha256: sha256(content), title: "P", format: "markdown" },
|
||||
},
|
||||
}
|
||||
let n = 0
|
||||
const replay = (async () => {
|
||||
n++
|
||||
return jsonRpc(n === 1 ? searchBody : pageBody)
|
||||
}) as unknown as typeof fetch
|
||||
const tools = researchTools({
|
||||
root: dir,
|
||||
env: ENV,
|
||||
stateDir: state,
|
||||
policy: async () => ({ root: dir }),
|
||||
fetch: replay,
|
||||
})
|
||||
const search = tools.find((tool) => tool.name === "es_research_search")!
|
||||
const searchText = await search.execute({ query: "token probe" }, { agent: "es-research-alpha" })
|
||||
expect(searchText.includes(TOKEN)).toBe(false)
|
||||
// The search chain resolves the default order; scraper-swarm leads when
|
||||
// configured, so this output names the serving backend.
|
||||
expect(searchText).toContain("scraper-swarm results for")
|
||||
const fetchTool = tools.find((tool) => tool.name === "es_research_fetch")!
|
||||
const fetchText = await fetchTool.execute({ url: "https://a.test/p" }, { agent: "es-research-alpha" })
|
||||
expect(fetchText.includes(TOKEN)).toBe(false)
|
||||
// Every cache write (meta JSON) is token-free.
|
||||
const cacheDir = path.join(dir, ".factory/research/sources")
|
||||
for (const name of await readdir(cacheDir)) {
|
||||
if (!name.endsWith(".json") || name === "index.json") continue
|
||||
const raw = await Bun.file(path.join(cacheDir, name)).text()
|
||||
expect([name, raw.includes(TOKEN)]).toEqual([name, false])
|
||||
}
|
||||
const indexRaw = await Bun.file(path.join(cacheDir, "index.json")).text()
|
||||
expect(indexRaw.includes(TOKEN)).toBe(false)
|
||||
} finally {
|
||||
await rm(dir, { recursive: true, force: true })
|
||||
await rm(state, { recursive: true, force: true })
|
||||
}
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,709 @@
|
||||
// #188 / #253: failover-chain proof gaps against loopback stubs.
|
||||
// Real HTTP to 127.0.0.1 only (gateway + SearXNG stubs below); no external
|
||||
// network. Proves, over the wire: 503 failover + order, 429 with/without
|
||||
// Retry-After (cooldown skip then retry), auth continuation, blocked stopping
|
||||
// both chains, sealed provenance, and that the gateway token never leaves its
|
||||
// Authorization header (every BackendError site + every cache write).
|
||||
import { afterAll, beforeAll, describe, expect, test } from "bun:test"
|
||||
import { mkdtemp, readdir, readFile, rm } from "node:fs/promises"
|
||||
import { tmpdir } from "node:os"
|
||||
import path from "node:path"
|
||||
import {
|
||||
BackendError,
|
||||
type BackendErrorKind,
|
||||
backoffFor,
|
||||
braveBackend,
|
||||
runFetchChain,
|
||||
runSearchChain,
|
||||
type SourceBackend,
|
||||
searxngBackend,
|
||||
} from "../src/research/backends/index.ts"
|
||||
import { ScraperSwarmClient, scraperSwarmBackend } from "../src/research/backends/scraper-swarm.ts"
|
||||
import { SourceCache } from "../src/research/cache.ts"
|
||||
import { researchTools } from "../src/research/ops.ts"
|
||||
import { sha256 } from "../src/util/hash.ts"
|
||||
|
||||
/** Grep-able canary, never a real secret: every negative asserts its absence. */
|
||||
const TOKEN = "swarm_canary_9f8e7d6c5b4a"
|
||||
|
||||
type GatewayMode =
|
||||
| "success-search"
|
||||
| "success-fetch"
|
||||
| "http-503"
|
||||
| "http-429-retry"
|
||||
| "http-429-bare"
|
||||
| "http-401"
|
||||
| "contract-2"
|
||||
| "no-body"
|
||||
| "rpc-error"
|
||||
| "wrong-id"
|
||||
| "blocked"
|
||||
| "timeout-tool"
|
||||
| "upstream-tool"
|
||||
| "no-results"
|
||||
| "no-content"
|
||||
| "sha-mismatch"
|
||||
|
||||
type SearxngMode = "success" | "http-503" | "http-429-retry" | "http-429-bare"
|
||||
|
||||
let gatewayMode: GatewayMode = "success-search"
|
||||
let searxngMode: SearxngMode = "success"
|
||||
let gatewayCalls: Array<{ name: string; auth: string | null }> = []
|
||||
let searxngHits = 0
|
||||
let gateway: ReturnType<typeof Bun.serve>
|
||||
let searxng: ReturnType<typeof Bun.serve>
|
||||
|
||||
const FETCH_CONTENT = "Recorded failover body text with enough words to cache."
|
||||
|
||||
beforeAll(() => {
|
||||
gateway = Bun.serve({
|
||||
port: 0,
|
||||
hostname: "127.0.0.1",
|
||||
async fetch(req) {
|
||||
if (new URL(req.url).pathname !== "/mcp") return new Response("not found", { status: 404 })
|
||||
const body = (await req.json().catch(() => undefined)) as
|
||||
| { id?: unknown; params?: { name?: string; arguments?: { query?: string } } }
|
||||
| undefined
|
||||
const name = body?.params?.name ?? ""
|
||||
const id = body?.id ?? 1
|
||||
gatewayCalls.push({ name, auth: req.headers.get("authorization") })
|
||||
const rpc = (result: unknown) =>
|
||||
Response.json({ id, jsonrpc: "2.0", result }, { headers: { "x-swarm-contract": "1" } })
|
||||
switch (gatewayMode) {
|
||||
case "http-503":
|
||||
return new Response("unavailable", { status: 503 })
|
||||
case "http-429-retry":
|
||||
return new Response("slow down", { status: 429, headers: { "retry-after": "2" } })
|
||||
case "http-429-bare":
|
||||
return new Response("slow down", { status: 429 })
|
||||
case "http-401":
|
||||
return new Response("denied", { status: 401 })
|
||||
case "contract-2":
|
||||
return Response.json(
|
||||
{ id, jsonrpc: "2.0", result: { structuredContent: { results: [] } } },
|
||||
{ headers: { "x-swarm-contract": "2" } },
|
||||
)
|
||||
case "no-body":
|
||||
return Response.json({ hello: 1 }, { headers: { "x-swarm-contract": "1" } })
|
||||
case "rpc-error":
|
||||
return Response.json(
|
||||
{ id, jsonrpc: "2.0", error: { code: -32602, message: "bad params" } },
|
||||
{ headers: { "x-swarm-contract": "1" } },
|
||||
)
|
||||
case "wrong-id":
|
||||
return Response.json(
|
||||
{ id: 999_999, jsonrpc: "2.0", result: { structuredContent: { results: [] } } },
|
||||
{ headers: { "x-swarm-contract": "1" } },
|
||||
)
|
||||
case "blocked":
|
||||
return rpc({
|
||||
content: [{ type: "text", text: "SSRF denied (fail-closed)" }],
|
||||
isError: true,
|
||||
structuredContent: { code: "ssrf_denied", message: "SSRF denied (fail-closed)" },
|
||||
})
|
||||
case "timeout-tool":
|
||||
return rpc({
|
||||
content: [{ type: "text", text: "engine timed out" }],
|
||||
isError: true,
|
||||
structuredContent: { code: "upstream_timeout", message: "engine timed out" },
|
||||
})
|
||||
case "upstream-tool":
|
||||
return rpc({
|
||||
content: [{ type: "text", text: "engine down" }],
|
||||
isError: true,
|
||||
structuredContent: { code: "upstream_error", message: "engine down" },
|
||||
})
|
||||
case "no-results":
|
||||
return rpc({ structuredContent: { engine: "stub" } })
|
||||
case "no-content":
|
||||
return rpc({ structuredContent: { title: "No body" } })
|
||||
case "sha-mismatch":
|
||||
return rpc({
|
||||
structuredContent: {
|
||||
content: FETCH_CONTENT,
|
||||
content_sha256: sha256("something else entirely"),
|
||||
final_url: "https://example.com/recorded",
|
||||
title: "Recorded",
|
||||
content_type: "text/plain",
|
||||
},
|
||||
})
|
||||
case "success-fetch":
|
||||
return rpc({
|
||||
structuredContent: {
|
||||
content: FETCH_CONTENT,
|
||||
content_sha256: sha256(FETCH_CONTENT),
|
||||
final_url: "https://example.com/recorded",
|
||||
title: "Recorded",
|
||||
content_type: "text/plain",
|
||||
},
|
||||
})
|
||||
case "success-search":
|
||||
return rpc({
|
||||
structuredContent: {
|
||||
query: body?.params?.arguments?.query ?? "",
|
||||
results: [
|
||||
{
|
||||
url: "https://gateway.test/docs",
|
||||
title: "Gateway docs",
|
||||
snippet: "Gateway snippet about the query.",
|
||||
},
|
||||
],
|
||||
},
|
||||
})
|
||||
default:
|
||||
return rpc({
|
||||
structuredContent: {
|
||||
query: body?.params?.arguments?.query ?? "",
|
||||
results: [
|
||||
{
|
||||
url: "https://gateway.test/docs",
|
||||
title: "Gateway docs",
|
||||
snippet: "Gateway snippet about the query.",
|
||||
},
|
||||
],
|
||||
},
|
||||
})
|
||||
}
|
||||
},
|
||||
})
|
||||
searxng = Bun.serve({
|
||||
port: 0,
|
||||
hostname: "127.0.0.1",
|
||||
fetch(req) {
|
||||
const url = new URL(req.url)
|
||||
if (url.pathname !== "/search") return new Response("not found", { status: 404 })
|
||||
searxngHits++
|
||||
if (searxngMode === "http-503") return new Response("unavailable", { status: 503 })
|
||||
if (searxngMode === "http-429-retry")
|
||||
return new Response("slow down", { status: 429, headers: { "retry-after": "2" } })
|
||||
if (searxngMode === "http-429-bare") return new Response("slow down", { status: 429 })
|
||||
return Response.json({
|
||||
results: [
|
||||
{
|
||||
url: "https://searxng.test/failover-docs",
|
||||
title: "Failover docs",
|
||||
content: `About ${url.searchParams.get("q")}: gates keep agents honest.`,
|
||||
},
|
||||
],
|
||||
})
|
||||
},
|
||||
})
|
||||
})
|
||||
|
||||
afterAll(() => {
|
||||
gateway?.stop(true)
|
||||
searxng?.stop(true)
|
||||
})
|
||||
|
||||
const reset = () => {
|
||||
gatewayCalls = []
|
||||
searxngHits = 0
|
||||
gatewayMode = "success-search"
|
||||
searxngMode = "success"
|
||||
}
|
||||
|
||||
/** Explicit env: ambient keys must not leak in (brave stays unavailable). */
|
||||
const envFor = (): NodeJS.ProcessEnv => ({
|
||||
ES_SCRAPER_SWARM_URL: `http://127.0.0.1:${gateway.port}`,
|
||||
ES_SCRAPER_SWARM_TOKEN: TOKEN,
|
||||
SEARXNG_URL: `http://127.0.0.1:${searxng.port}`,
|
||||
})
|
||||
|
||||
const clock = (start: number) => {
|
||||
let current = start
|
||||
return {
|
||||
now: () => current,
|
||||
advance: (ms: number) => {
|
||||
current += ms
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
const countingFetch = (text: string): { backend: SourceBackend; calls: () => number } => {
|
||||
let calls = 0
|
||||
return {
|
||||
calls: () => calls,
|
||||
backend: {
|
||||
id: "fallback",
|
||||
capabilities: { search: false, fetch: true },
|
||||
available: () => true,
|
||||
fetch: async (url: string) => {
|
||||
calls++
|
||||
return { url, text, contentType: "text/plain" }
|
||||
},
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
describe("backoffFor defaults (#253.1)", () => {
|
||||
test("a 429 Retry-After value wins; a bare 429 cools down for 60s", () => {
|
||||
expect(backoffFor(new BackendError("rate_limited", "slow", { retryAfterMs: 5_000 }))).toBe(5_000)
|
||||
expect(backoffFor(new BackendError("rate_limited", "slow"))).toBe(60_000)
|
||||
})
|
||||
|
||||
test("every other kind cools down for 30s", () => {
|
||||
for (const kind of ["unavailable", "auth", "upstream", "timeout", "blocked"] as const)
|
||||
expect([kind, backoffFor(new BackendError(kind, "x"))]).toEqual([kind, 30_000])
|
||||
})
|
||||
})
|
||||
|
||||
describe("loopback failover through the real backends", () => {
|
||||
test("a 503 on the gateway fails over in order; brave is skipped unavailable", async () => {
|
||||
reset()
|
||||
gatewayMode = "http-503"
|
||||
const cooldowns = new Map<string, { until: number; kind: BackendErrorKind }>()
|
||||
const result = await runSearchChain(
|
||||
[scraperSwarmBackend, braveBackend, searxngBackend],
|
||||
"gates",
|
||||
{
|
||||
env: envFor(),
|
||||
},
|
||||
{ cooldowns },
|
||||
)
|
||||
expect(result.backend).toBe("searxng")
|
||||
// Unavailable backends are reported first (skipped without a call), then
|
||||
// the tried backends in order.
|
||||
expect(result.attempts).toEqual([
|
||||
{ backend: "brave", ok: false, kind: "unavailable" },
|
||||
{ backend: "scraper-swarm", ok: false, kind: "upstream" },
|
||||
{ backend: "searxng", ok: true },
|
||||
])
|
||||
expect(result.value[0]?.snippet).toContain("gates keep agents honest")
|
||||
expect(gatewayCalls).toHaveLength(1)
|
||||
expect(searxngHits).toBe(1)
|
||||
})
|
||||
|
||||
test("a 429 with Retry-After cools down: skipped, then retried after it lapses", async () => {
|
||||
reset()
|
||||
gatewayMode = "http-429-retry"
|
||||
const env = envFor()
|
||||
const timer = clock(1_000_000)
|
||||
const cooldowns = new Map<string, { until: number; kind: BackendErrorKind }>()
|
||||
const first = await runSearchChain(
|
||||
[scraperSwarmBackend, searxngBackend],
|
||||
"gates",
|
||||
{ env },
|
||||
{
|
||||
now: timer.now,
|
||||
cooldowns,
|
||||
},
|
||||
)
|
||||
expect(first.backend).toBe("searxng")
|
||||
expect(first.attempts[0]).toMatchObject({ backend: "scraper-swarm", ok: false, kind: "rate_limited" })
|
||||
expect(gatewayCalls).toHaveLength(1)
|
||||
// Still cooling down: skipped without an HTTP call, same failover.
|
||||
timer.advance(1_000)
|
||||
const skipped = await runSearchChain(
|
||||
[scraperSwarmBackend, searxngBackend],
|
||||
"gates",
|
||||
{ env },
|
||||
{
|
||||
now: timer.now,
|
||||
cooldowns,
|
||||
},
|
||||
)
|
||||
expect(skipped.backend).toBe("searxng")
|
||||
expect(skipped.attempts[0]).toMatchObject({ backend: "scraper-swarm", ok: false, kind: "rate_limited" })
|
||||
expect(gatewayCalls).toHaveLength(1)
|
||||
// After the 2s Retry-After lapses the gateway is tried again.
|
||||
timer.advance(1_500)
|
||||
gatewayMode = "success-search"
|
||||
const retried = await runSearchChain(
|
||||
[scraperSwarmBackend, searxngBackend],
|
||||
"gates",
|
||||
{ env },
|
||||
{
|
||||
now: timer.now,
|
||||
cooldowns,
|
||||
},
|
||||
)
|
||||
expect(retried.backend).toBe("scraper-swarm")
|
||||
expect(gatewayCalls).toHaveLength(2)
|
||||
})
|
||||
|
||||
test("a 429 without Retry-After defaults to a 60s cooldown over the wire", async () => {
|
||||
reset()
|
||||
gatewayMode = "http-429-bare"
|
||||
const env = envFor()
|
||||
const timer = clock(2_000_000)
|
||||
const cooldowns = new Map<string, { until: number; kind: BackendErrorKind }>()
|
||||
const run = (at: number) => {
|
||||
timer.advance(at)
|
||||
return runSearchChain(
|
||||
[scraperSwarmBackend, searxngBackend],
|
||||
"gates",
|
||||
{ env },
|
||||
{
|
||||
now: timer.now,
|
||||
cooldowns,
|
||||
},
|
||||
)
|
||||
}
|
||||
await run(0)
|
||||
expect(gatewayCalls).toHaveLength(1)
|
||||
await run(59_000)
|
||||
expect(gatewayCalls).toHaveLength(1)
|
||||
gatewayMode = "success-search"
|
||||
const retried = await run(1_500)
|
||||
expect(retried.backend).toBe("scraper-swarm")
|
||||
expect(gatewayCalls).toHaveLength(2)
|
||||
})
|
||||
|
||||
test("a provider-side 429 maps with and without its Retry-After value", async () => {
|
||||
reset()
|
||||
searxngMode = "http-429-retry"
|
||||
// Fresh cooldown maps: the module-shared default would carry the first
|
||||
// 429's cooldown into the second call and skip the backend instead.
|
||||
const retrying = await runSearchChain(
|
||||
[searxngBackend],
|
||||
"gates",
|
||||
{ env: envFor() },
|
||||
{
|
||||
cooldowns: new Map(),
|
||||
},
|
||||
).then(
|
||||
() => "no-throw",
|
||||
(error: unknown) => error,
|
||||
)
|
||||
expect(retrying).toBeInstanceOf(BackendError)
|
||||
expect(retrying).toMatchObject({ kind: "rate_limited", retryAfterMs: 2_000 })
|
||||
searxngMode = "http-429-bare"
|
||||
const bare = await runSearchChain(
|
||||
[searxngBackend],
|
||||
"gates",
|
||||
{ env: envFor() },
|
||||
{
|
||||
cooldowns: new Map(),
|
||||
},
|
||||
).then(
|
||||
() => "no-throw",
|
||||
(error: unknown) => error,
|
||||
)
|
||||
expect(bare).toBeInstanceOf(BackendError)
|
||||
expect(bare).toMatchObject({ kind: "rate_limited" })
|
||||
expect((bare as BackendError).retryAfterMs).toBeUndefined()
|
||||
expect(backoffFor(bare as BackendError)).toBe(60_000)
|
||||
})
|
||||
|
||||
test("a safety block stops the search chain: SearXNG is never called", async () => {
|
||||
reset()
|
||||
gatewayMode = "blocked"
|
||||
const failure = await runSearchChain(
|
||||
[scraperSwarmBackend, searxngBackend],
|
||||
"gates",
|
||||
{
|
||||
env: envFor(),
|
||||
},
|
||||
{ cooldowns: new Map() },
|
||||
).then(
|
||||
() => "no-throw",
|
||||
(error: unknown) => error,
|
||||
)
|
||||
expect(failure).toBeInstanceOf(BackendError)
|
||||
expect(failure).toMatchObject({ kind: "blocked" })
|
||||
expect(searxngHits).toBe(0)
|
||||
})
|
||||
|
||||
test("a safety block stops the fetch chain: the fallback is never called", async () => {
|
||||
reset()
|
||||
gatewayMode = "blocked"
|
||||
const fallback = countingFetch("never served")
|
||||
const failure = await runFetchChain([scraperSwarmBackend, fallback.backend], "https://example.com/x", {
|
||||
env: envFor(),
|
||||
cooldowns: new Map(),
|
||||
}).then(
|
||||
() => "no-throw",
|
||||
(error: unknown) => error,
|
||||
)
|
||||
expect(failure).toMatchObject({ kind: "blocked" })
|
||||
expect(fallback.calls()).toBe(0)
|
||||
expect((failure as Error).message).toContain("SSRF denied")
|
||||
})
|
||||
|
||||
test("search failover provenance names the serving backend; snippets are not cached", async () => {
|
||||
reset()
|
||||
gatewayMode = "http-503"
|
||||
const dir = await mkdtemp(path.join(tmpdir(), "es-failover-"))
|
||||
const state = await mkdtemp(path.join(tmpdir(), "es-failover-state-"))
|
||||
try {
|
||||
const env = envFor()
|
||||
const tools = researchTools({
|
||||
root: dir,
|
||||
env,
|
||||
stateDir: state,
|
||||
policy: async () => ({ root: dir }),
|
||||
chain: [scraperSwarmBackend, braveBackend, searxngBackend],
|
||||
})
|
||||
const search = tools.find((tool) => tool.name === "es_research_search")!
|
||||
const text = await search.execute({ query: "gates" }, { agent: "es-research-alpha" })
|
||||
expect(text).toContain("searxng results for")
|
||||
// SearXNG results are snippets, not page content: nothing is cached under them.
|
||||
expect(await readdir(path.join(dir, ".factory/research/sources")).catch(() => [] as string[])).toEqual([])
|
||||
} finally {
|
||||
await rm(dir, { recursive: true, force: true })
|
||||
await rm(state, { recursive: true, force: true })
|
||||
}
|
||||
})
|
||||
|
||||
test("fetch success through the gateway records sealed scraper-swarm provenance", async () => {
|
||||
reset()
|
||||
gatewayMode = "success-fetch"
|
||||
const dir = await mkdtemp(path.join(tmpdir(), "es-failover-fetch-"))
|
||||
const state = await mkdtemp(path.join(tmpdir(), "es-failover-fetch-state-"))
|
||||
try {
|
||||
const tools = researchTools({
|
||||
root: dir,
|
||||
env: envFor(),
|
||||
stateDir: state,
|
||||
policy: async () => ({ root: dir }),
|
||||
chain: [scraperSwarmBackend, searxngBackend],
|
||||
})
|
||||
const fetchTool = tools.find((tool) => tool.name === "es_research_fetch")!
|
||||
const text = await fetchTool.execute(
|
||||
{ url: "https://example.com/recorded" },
|
||||
{
|
||||
agent: "es-research-alpha",
|
||||
},
|
||||
)
|
||||
const hash = /sha256:([0-9a-f]{64})/.exec(text)?.[1]
|
||||
expect(hash).toBe(sha256(FETCH_CONTENT))
|
||||
const stored = await new SourceCache(path.join(dir, ".factory/research/sources"), state).get(hash!)
|
||||
expect(stored?.meta.provider).toBe("scraper-swarm")
|
||||
expect(stored?.meta.seal).toMatch(/^[0-9a-f]{64}$/)
|
||||
} finally {
|
||||
await rm(dir, { recursive: true, force: true })
|
||||
await rm(state, { recursive: true, force: true })
|
||||
}
|
||||
})
|
||||
})
|
||||
|
||||
describe("auth continues the chain (#253.2)", () => {
|
||||
test("a gateway HTTP 401 fails over to SearXNG and cools down", async () => {
|
||||
reset()
|
||||
gatewayMode = "http-401"
|
||||
const env = envFor()
|
||||
const timer = clock(3_000_000)
|
||||
const cooldowns = new Map<string, { until: number; kind: BackendErrorKind }>()
|
||||
const first = await runSearchChain(
|
||||
[scraperSwarmBackend, searxngBackend],
|
||||
"gates",
|
||||
{ env },
|
||||
{
|
||||
now: timer.now,
|
||||
cooldowns,
|
||||
},
|
||||
)
|
||||
expect(first.backend).toBe("searxng")
|
||||
expect(first.attempts[0]).toMatchObject({ backend: "scraper-swarm", ok: false, kind: "auth" })
|
||||
expect(gatewayCalls).toHaveLength(1)
|
||||
timer.advance(1_000)
|
||||
const second = await runSearchChain(
|
||||
[scraperSwarmBackend, searxngBackend],
|
||||
"gates",
|
||||
{ env },
|
||||
{
|
||||
now: timer.now,
|
||||
cooldowns,
|
||||
},
|
||||
)
|
||||
expect(second.backend).toBe("searxng")
|
||||
expect(gatewayCalls).toHaveLength(1)
|
||||
})
|
||||
|
||||
test("a 401 search error and a 403 fetch error both fail over with a cooldown", async () => {
|
||||
let searchCalls = 0
|
||||
let fetchCalls = 0
|
||||
const authSearch: SourceBackend = {
|
||||
id: "auth-search",
|
||||
capabilities: { search: true, fetch: false },
|
||||
available: () => true,
|
||||
search: async () => {
|
||||
searchCalls++
|
||||
throw new BackendError("auth", "HTTP 401", { status: 401 })
|
||||
},
|
||||
}
|
||||
const authFetch: SourceBackend = {
|
||||
id: "auth-fetch",
|
||||
capabilities: { search: false, fetch: true },
|
||||
available: () => true,
|
||||
fetch: async () => {
|
||||
fetchCalls++
|
||||
throw new BackendError("auth", "HTTP 403", { status: 403 })
|
||||
},
|
||||
}
|
||||
const goodSearch: SourceBackend = {
|
||||
id: "good-search",
|
||||
capabilities: { search: true, fetch: false },
|
||||
available: () => true,
|
||||
search: async (query: string) => [{ url: `https://good.test/${query}`, title: "Good", snippet: "ok" }],
|
||||
}
|
||||
const goodFetch = countingFetch("served after auth")
|
||||
const timer = clock(4_000_000)
|
||||
const searchCooldowns = new Map<string, { until: number; kind: BackendErrorKind }>()
|
||||
const searched = await runSearchChain(
|
||||
[authSearch, goodSearch],
|
||||
"q",
|
||||
{},
|
||||
{
|
||||
now: timer.now,
|
||||
cooldowns: searchCooldowns,
|
||||
},
|
||||
)
|
||||
expect(searched.backend).toBe("good-search")
|
||||
expect(searched.attempts[0]).toMatchObject({ backend: "auth-search", ok: false, kind: "auth" })
|
||||
timer.advance(1_000)
|
||||
await runSearchChain([authSearch, goodSearch], "q", {}, { now: timer.now, cooldowns: searchCooldowns })
|
||||
expect(searchCalls).toBe(1)
|
||||
const fetchCooldowns = new Map<string, { until: number; kind: BackendErrorKind }>()
|
||||
const fetched = await runFetchChain([authFetch, goodFetch.backend], "https://good.test/x", {
|
||||
now: timer.now,
|
||||
cooldowns: fetchCooldowns,
|
||||
})
|
||||
expect(fetched.backend).toBe("fallback")
|
||||
expect(fetched.attempts[0]).toMatchObject({ backend: "auth-fetch", ok: false, kind: "auth" })
|
||||
timer.advance(1_000)
|
||||
await runFetchChain([authFetch, goodFetch.backend], "https://good.test/x", {
|
||||
now: timer.now,
|
||||
cooldowns: fetchCooldowns,
|
||||
})
|
||||
expect(fetchCalls).toBe(1)
|
||||
})
|
||||
})
|
||||
|
||||
describe("the token never leaves its Authorization header (#253.3)", () => {
|
||||
const client = () =>
|
||||
new ScraperSwarmClient({
|
||||
url: `http://127.0.0.1:${gateway.port}`,
|
||||
token: TOKEN,
|
||||
fetch: fetch as unknown as typeof fetch,
|
||||
})
|
||||
|
||||
test("every gateway failure message is token-free", async () => {
|
||||
const searchModes: GatewayMode[] = [
|
||||
"http-401",
|
||||
"http-429-bare",
|
||||
"http-429-retry",
|
||||
"http-503",
|
||||
"contract-2",
|
||||
"no-body",
|
||||
"rpc-error",
|
||||
"wrong-id",
|
||||
"blocked",
|
||||
"timeout-tool",
|
||||
"upstream-tool",
|
||||
"no-results",
|
||||
]
|
||||
const expectedKind: Record<string, BackendErrorKind> = {
|
||||
"http-401": "auth",
|
||||
"http-429-bare": "rate_limited",
|
||||
"http-429-retry": "rate_limited",
|
||||
"http-503": "upstream",
|
||||
"contract-2": "upstream",
|
||||
"no-body": "upstream",
|
||||
"rpc-error": "upstream",
|
||||
"wrong-id": "upstream",
|
||||
blocked: "blocked",
|
||||
"timeout-tool": "timeout",
|
||||
"upstream-tool": "upstream",
|
||||
"no-results": "upstream",
|
||||
}
|
||||
for (const mode of searchModes) {
|
||||
gatewayMode = mode
|
||||
const failure = await client()
|
||||
.search("q", 1)
|
||||
.then(
|
||||
() => "no-throw",
|
||||
(error: unknown) => error,
|
||||
)
|
||||
expect(failure).toBeInstanceOf(BackendError)
|
||||
expect((failure as BackendError).kind).toBe(expectedKind[mode])
|
||||
expect((failure as Error).message).not.toContain(TOKEN)
|
||||
expect(JSON.stringify(failure)).not.toContain(TOKEN)
|
||||
}
|
||||
const fetchModes: GatewayMode[] = ["no-content", "sha-mismatch", "blocked"]
|
||||
for (const mode of fetchModes) {
|
||||
gatewayMode = mode
|
||||
const failure = await client()
|
||||
.fetchPage("https://example.com/x")
|
||||
.then(
|
||||
() => "no-throw",
|
||||
(error: unknown) => error,
|
||||
)
|
||||
expect(failure).toBeInstanceOf(BackendError)
|
||||
expect((failure as Error).message).not.toContain(TOKEN)
|
||||
}
|
||||
// An unreachable gateway and a missing config fail without the token too.
|
||||
const down = new ScraperSwarmClient({ url: "http://127.0.0.1:1", token: TOKEN })
|
||||
for (const failure of [
|
||||
await down.search("q", 1).then(
|
||||
() => "no-throw",
|
||||
(error: unknown) => error,
|
||||
),
|
||||
await down.fetchPage("https://example.com/x").then(
|
||||
() => "no-throw",
|
||||
(error: unknown) => error,
|
||||
),
|
||||
]) {
|
||||
expect(failure).toMatchObject({ kind: "unavailable" })
|
||||
expect((failure as Error).message).not.toContain(TOKEN)
|
||||
}
|
||||
for (const failure of [
|
||||
await scraperSwarmBackend.search!("q", { env: {} }).then(
|
||||
() => "no-throw",
|
||||
(error: unknown) => error,
|
||||
),
|
||||
await scraperSwarmBackend.fetch!("https://example.com/x", { env: {} }).then(
|
||||
() => "no-throw",
|
||||
(error: unknown) => error,
|
||||
),
|
||||
]) {
|
||||
expect(failure).toMatchObject({ kind: "unavailable" })
|
||||
expect((failure as Error).message).not.toContain(TOKEN)
|
||||
}
|
||||
})
|
||||
|
||||
test("search-path and fetch-path cache writes hold no token", async () => {
|
||||
reset()
|
||||
gatewayMode = "success-fetch"
|
||||
const dir = await mkdtemp(path.join(tmpdir(), "es-token-"))
|
||||
const state = await mkdtemp(path.join(tmpdir(), "es-token-state-"))
|
||||
try {
|
||||
const contentful: SourceBackend = {
|
||||
id: "contentful",
|
||||
capabilities: { search: true, fetch: false },
|
||||
available: () => true,
|
||||
search: async (_query: string) => [
|
||||
{
|
||||
url: "https://contentful.test/docs",
|
||||
title: "Contentful docs",
|
||||
snippet: "a snippet",
|
||||
content: "Contentful page text about gates, long enough to cache as a snapshot.",
|
||||
},
|
||||
],
|
||||
}
|
||||
const tools = researchTools({
|
||||
root: dir,
|
||||
env: envFor(),
|
||||
stateDir: state,
|
||||
policy: async () => ({ root: dir }),
|
||||
chain: [contentful, scraperSwarmBackend],
|
||||
})
|
||||
const search = tools.find((tool) => tool.name === "es_research_search")!
|
||||
const searched = await search.execute({ query: "gates" }, { agent: "es-research-alpha" })
|
||||
expect(searched).toContain("contentful results for")
|
||||
expect(searched).toContain("sha256:")
|
||||
const fetchTool = tools.find((tool) => tool.name === "es_research_fetch")!
|
||||
await fetchTool.execute({ url: "https://example.com/recorded" }, { agent: "es-research-alpha" })
|
||||
const sources = path.join(dir, ".factory/research/sources")
|
||||
const files = await readdir(sources)
|
||||
expect(files.length).toBeGreaterThan(0)
|
||||
for (const file of files) expect(await readFile(path.join(sources, file), "utf8")).not.toContain(TOKEN)
|
||||
// The only wire presence of the token is the Authorization header.
|
||||
expect(gatewayCalls.length).toBeGreaterThan(0)
|
||||
for (const call of gatewayCalls) expect(call.auth).toBe(`Bearer ${TOKEN}`)
|
||||
} finally {
|
||||
await rm(dir, { recursive: true, force: true })
|
||||
await rm(state, { recursive: true, force: true })
|
||||
}
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,44 @@
|
||||
// Control-drift visibility (#242): when a tampered control file halts the
|
||||
// run, the halt names the gate rule (integrity/control-drift) and points at
|
||||
// the gate summary holding the structured finding — a seat debugging the halt
|
||||
// gets more than an opaque "control files changed".
|
||||
import { afterEach, beforeEach, describe, expect, test } from "bun:test"
|
||||
import { writeFile } from "node:fs/promises"
|
||||
import path from "node:path"
|
||||
import { recordApproval } from "../src/approval/index.ts"
|
||||
import { type HumanSigner, sealHumanKey, unlockHumanKey } from "../src/approval/keystore.ts"
|
||||
import { Factory, FactoryHalted } from "../src/factory/index.ts"
|
||||
import { type Fixture, frontier, gitRepo } from "./helpers.ts"
|
||||
|
||||
const PASSPHRASE = "test-passphrase-1234"
|
||||
let fx: Fixture
|
||||
let signer: HumanSigner
|
||||
let factory: Factory
|
||||
|
||||
beforeEach(async () => {
|
||||
fx = await gitRepo("es-gdrift-")
|
||||
await sealHumanKey(PASSPHRASE, fx.state)
|
||||
signer = await unlockHumanKey(PASSPHRASE, fx.state)
|
||||
factory = new Factory(fx.root, {
|
||||
gates: async () => ({ passed: true, findings: [], summary: "green" }),
|
||||
stateDir: fx.state,
|
||||
})
|
||||
})
|
||||
afterEach(() => fx.cleanup())
|
||||
|
||||
describe("drift halt visibility", () => {
|
||||
test("a drift halt names integrity/control-drift and the gate summary holding the finding", async () => {
|
||||
await factory.begin("human:tester")
|
||||
await factory.writeFrontier("grill", frontier())
|
||||
await recordApproval(fx.root, { stage: "frontier", channel: "cli", approvedBy: "tester", signer })
|
||||
await writeFile(path.join(fx.root, ".factory/gates.json"), '{"commands":[]}\n')
|
||||
const error = await factory.recordGateRun("es-programmer", { passed: false, findings: [], summary: "red" }).then(
|
||||
() => undefined,
|
||||
(failure: unknown) => failure,
|
||||
)
|
||||
expect(error).toBeInstanceOf(FactoryHalted)
|
||||
expect(String(error)).toContain("integrity/control-drift")
|
||||
expect(String(error)).toContain("summary.json")
|
||||
expect((await factory.read())?.stage).toBe("HALTED")
|
||||
})
|
||||
})
|
||||
@@ -1,18 +1,27 @@
|
||||
import { afterEach, beforeEach, describe, expect, test } from "bun:test"
|
||||
import { mkdir, writeFile } from "node:fs/promises"
|
||||
import { mkdir, mkdtemp, readFile, rm, writeFile } from "node:fs/promises"
|
||||
import { tmpdir } from "node:os"
|
||||
import path from "node:path"
|
||||
import { type HumanSigner, sealHumanKey, unlockHumanKey } from "../src/approval/keystore.ts"
|
||||
import {
|
||||
checkBudgets,
|
||||
commandSetHash,
|
||||
complexity,
|
||||
detectGates,
|
||||
formatReport,
|
||||
type GateReport,
|
||||
GatesConfigSchema,
|
||||
gitleaks,
|
||||
installed,
|
||||
isTrusted,
|
||||
loadGatesConfig,
|
||||
parseOutput,
|
||||
recordWaiver,
|
||||
runGates,
|
||||
trustProject,
|
||||
validateArtifacts,
|
||||
} from "../src/index.ts"
|
||||
import { canonicalPath } from "../src/trust/paths.ts"
|
||||
import { bwrapAvailable, resetBwrapProbe, run } from "../src/util/proc.ts"
|
||||
import { type Fixture, gitRepo } from "./helpers.ts"
|
||||
|
||||
@@ -139,6 +148,77 @@ describe("trust", () => {
|
||||
expect((await gates({ scope: "full", config: changed })).findings[0]?.rule).toBe("trust/untrusted")
|
||||
})
|
||||
|
||||
test("a forged trust entry is not trust: isTrusted verifies the human signature (#221.3)", async () => {
|
||||
const cfg = config([{ id: "probe", kind: "test", command: "touch RAN" }])
|
||||
const wanted = config([{ id: "probe", kind: "test", command: "touch FORGED" }])
|
||||
const { hash } = await commandSetHash(fx.root, cfg.commands)
|
||||
const forgedHash = (await commandSetHash(fx.root, wanted.commands)).hash
|
||||
await trust(cfg)
|
||||
expect(await isTrusted(fx.root, hash, fx.state)).toBe(true)
|
||||
|
||||
const file = path.join(fx.state, "trust.json")
|
||||
const genuine = await readFile(file, "utf8")
|
||||
const store = JSON.parse(genuine) as { projects: Record<string, { gates: Record<string, any> }> }
|
||||
const project = Object.keys(store.projects)[0]!
|
||||
const entry = store.projects[project]!.gates
|
||||
const plant = (projects: Record<string, unknown>) => writeFile(file, JSON.stringify({ version: 3, projects }))
|
||||
|
||||
// A second human key: a real Ed25519 signature, but not this machine's.
|
||||
const foreign = await mkdtemp(path.join(tmpdir(), "es-gates-foreign-"))
|
||||
await sealHumanKey(PASSPHRASE, foreign)
|
||||
const foreignSigner = await unlockHumanKey(PASSPHRASE, foreign)
|
||||
const retargeted = {
|
||||
hash: forgedHash,
|
||||
lines: ["probe: touch FORGED"],
|
||||
approvedBy: "x",
|
||||
approvedAt: entry.approvedAt,
|
||||
}
|
||||
const forgeries: Record<string, Record<string, unknown>> = {
|
||||
// The genuine signature replayed over another command set.
|
||||
"replayed signature": { [project]: { gates: { ...entry, hash: forgedHash, lines: retargeted.lines } } },
|
||||
"made-up signature": {
|
||||
[project]: {
|
||||
gates: { ...retargeted, signature: { alg: "ed25519", keyId: entry.signature.keyId, sig: "AAAA" } },
|
||||
},
|
||||
},
|
||||
"no signature": { [project]: { gates: retargeted } },
|
||||
"foreign key": {
|
||||
[project]: {
|
||||
gates: {
|
||||
...retargeted,
|
||||
signature: foreignSigner.sign({ trust: 1, project, kind: "gates", ...retargeted }),
|
||||
},
|
||||
},
|
||||
},
|
||||
}
|
||||
try {
|
||||
for (const [name, projects] of Object.entries(forgeries)) {
|
||||
await plant(projects)
|
||||
expect([name, await isTrusted(fx.root, forgedHash, fx.state)]).toEqual([name, false])
|
||||
const report = await gates({ scope: "full", config: wanted })
|
||||
expect([name, report.findings[0]?.rule]).toEqual([name, "trust/untrusted"])
|
||||
expect(await Bun.file(path.join(fx.root, "FORGED")).exists()).toBe(false)
|
||||
}
|
||||
|
||||
// The signature binds the project and the kind: the genuine entry moved
|
||||
// to another project, or presented as hook trust, verifies for neither.
|
||||
const other = await gitRepo("es-gates-other-")
|
||||
try {
|
||||
const otherProject = canonicalPath(other.root)
|
||||
expect(project).toBe(canonicalPath(fx.root))
|
||||
await plant({ [project]: { gates: entry, hooks: entry }, [otherProject]: { gates: entry } })
|
||||
expect(await isTrusted(other.root, hash, fx.state)).toBe(false)
|
||||
expect(await isTrusted(fx.root, hash, fx.state, "hooks")).toBe(false)
|
||||
// Control: in the same planted file, the untouched entry still verifies.
|
||||
expect(await isTrusted(fx.root, hash, fx.state)).toBe(true)
|
||||
} finally {
|
||||
await other.cleanup()
|
||||
}
|
||||
} finally {
|
||||
await rm(foreign, { recursive: true, force: true })
|
||||
}
|
||||
})
|
||||
|
||||
test("a package script reachable from a command is part of the trusted hash", async () => {
|
||||
await write("package.json", JSON.stringify({ scripts: { typecheck: "tsc" } }))
|
||||
const cfg = config([{ id: "typecheck", kind: "typecheck", command: "npm run --silent typecheck" }])
|
||||
@@ -189,6 +269,41 @@ describe("built-in checks", () => {
|
||||
expect(waived.waived).toBeGreaterThan(0)
|
||||
})
|
||||
|
||||
test("an expired signed waiver planted at gate time waives nothing (#221.1)", async () => {
|
||||
await write("src/config.ts", `export const token = "${FAKE_TOKEN}"\n`)
|
||||
// recordWaiver refuses a past expiry, so the record is hand-aged: the
|
||||
// same fields it signs, under the real human key, one day apart.
|
||||
const plant = async (expiresAt: Date) => {
|
||||
const unsigned = {
|
||||
version: 2 as const,
|
||||
id: "aged-token",
|
||||
rule: "security/secret-github-token",
|
||||
files: "src/config.ts",
|
||||
reason: "fixture value, not a real token",
|
||||
approvedBy: "tester",
|
||||
approvedAt: new Date(expiresAt.getTime() - 30 * 86_400_000).toISOString(),
|
||||
expiresAt: expiresAt.toISOString(),
|
||||
channel: "cli" as const,
|
||||
auditHead: null,
|
||||
}
|
||||
await write(".factory/waivers/aged-token.json", JSON.stringify({ ...unsigned, signature: signer.sign(unsigned) }))
|
||||
return gates({ touched: ["src/config.ts"], config: config([]) })
|
||||
}
|
||||
|
||||
const expired = await plant(new Date(Date.now() - 86_400_000))
|
||||
// The signature verifies, so this is not the forged path: no waiver/invalid.
|
||||
expect(expired.findings.some((finding) => finding.rule === "waiver/invalid")).toBe(false)
|
||||
expect(expired.findings.some((finding) => finding.rule === "security/secret-github-token")).toBe(true)
|
||||
expect(expired.waived).toBe(0)
|
||||
expect(expired.passed).toBe(false)
|
||||
|
||||
// Control: the identical record with a live expiry does waive, so the
|
||||
// refusal above is the expiry and nothing else about the planted file.
|
||||
const live = await plant(new Date(Date.now() + 86_400_000))
|
||||
expect(live.findings.some((finding) => finding.rule === "security/secret-github-token")).toBe(false)
|
||||
expect(live.waived).toBe(1)
|
||||
})
|
||||
|
||||
test("budgets: behaviour change without a test change, file length, complexity", async () => {
|
||||
await write("src/big.ts", Array.from({ length: 30 }, (_, i) => `export const v${i} = ${i}`).join("\n"))
|
||||
await write(
|
||||
@@ -448,3 +563,201 @@ describe("parsers", () => {
|
||||
expect(scores.find((item) => item.name === "typed")?.score).toBe(2)
|
||||
})
|
||||
})
|
||||
|
||||
describe("built-in secret scan cache (#239)", () => {
|
||||
test("a rescan re-reads only changed files: hits are counted, edits re-read, no count cap", async () => {
|
||||
const { clearSecretScanCache, scanFileForSecrets, secretScanStats } = await import("../src/index.ts")
|
||||
clearSecretScanCache()
|
||||
await write("src/a.ts", `export const token = "${FAKE_TOKEN}"\n`)
|
||||
await write("src/b.ts", "export const b = 1\n")
|
||||
const first = await scanFileForSecrets(fx.root, "src/a.ts")
|
||||
expect(first.some((finding) => finding.rule === "security/secret-github-token")).toBe(true)
|
||||
expect(await scanFileForSecrets(fx.root, "src/b.ts")).toEqual([])
|
||||
// Unchanged files are cache hits: identical findings, hits counted.
|
||||
const before = secretScanStats()
|
||||
expect(await scanFileForSecrets(fx.root, "src/a.ts")).toEqual(first)
|
||||
expect(await scanFileForSecrets(fx.root, "src/b.ts")).toEqual([])
|
||||
expect(secretScanStats().hits).toBe(before.hits + 2)
|
||||
// An edit re-reads: the finding disappears after the secret is removed.
|
||||
await write("src/a.ts", 'export const token = "rotated"\n')
|
||||
expect(await scanFileForSecrets(fx.root, "src/a.ts")).toEqual([])
|
||||
})
|
||||
})
|
||||
|
||||
describe("proof gaps (#239)", () => {
|
||||
test("format is a failing arm, not detection-only: unformatted files fail, formatted files pass", async () => {
|
||||
await write("biome.json", "{}")
|
||||
await write("src/a.ts", "export const x=1\n")
|
||||
const cfg = config([
|
||||
{
|
||||
id: "format",
|
||||
kind: "format",
|
||||
command: `${BIOME} format --reporter=json`,
|
||||
parser: "biome",
|
||||
touchedArgs: true,
|
||||
extensions: [".ts"],
|
||||
},
|
||||
])
|
||||
await trust(cfg)
|
||||
// The engine arm, not the sandbox: pin bwrap away so the command runs
|
||||
// directly (containers without overlayfs cannot sandbox; see #239).
|
||||
resetBwrapProbe(false)
|
||||
try {
|
||||
const red = await gates({ touched: ["src/a.ts"], config: cfg })
|
||||
expect(red.checks.find((check) => check.id === "format")?.status).toBe("fail")
|
||||
expect(red.findings.find((finding) => finding.rule === "format/biome")).toMatchObject({
|
||||
file: "src/a.ts",
|
||||
severity: "error",
|
||||
})
|
||||
await write("src/a.ts", "export const x = 1;\n")
|
||||
const green = await gates({ touched: ["src/a.ts"], config: cfg })
|
||||
expect(green.findings.some((finding) => finding.rule === "format/biome")).toBe(false)
|
||||
} finally {
|
||||
resetBwrapProbe()
|
||||
}
|
||||
})
|
||||
|
||||
test("typecheck runs at gate level: errors fail, clean files pass, present tools are not missing", async () => {
|
||||
const TSC = path.resolve(import.meta.dir, "../../../node_modules/.bin/tsc")
|
||||
await write("tsconfig.json", JSON.stringify({ compilerOptions: { strict: true, noEmit: true }, include: ["src"] }))
|
||||
await write("src/a.ts", 'export const x: number = "nope"\n')
|
||||
const cfg = config([
|
||||
{ id: "typecheck", kind: "typecheck", command: `${TSC} --noEmit -p tsconfig.json`, parser: "tsc" },
|
||||
])
|
||||
await trust(cfg)
|
||||
resetBwrapProbe(false)
|
||||
try {
|
||||
const red = await gates({ touched: ["src/a.ts"], config: cfg })
|
||||
expect(red.checks.find((check) => check.id === "typecheck")?.status).toBe("fail")
|
||||
expect(red.findings.some((finding) => finding.file === "src/a.ts" && finding.check === "typecheck")).toBe(true)
|
||||
await write("src/a.ts", "export const x: number = 1\n")
|
||||
const green = await gates({ touched: ["src/a.ts"], config: cfg })
|
||||
expect(green.checks.find((check) => check.id === "typecheck")?.status).toBe("pass")
|
||||
expect(green.findings.some((finding) => finding.rule === "gates/tool-missing")).toBe(false)
|
||||
expect(green.findings.some((finding) => finding.rule === "gates/sandbox-missing")).toBe(false)
|
||||
} finally {
|
||||
resetBwrapProbe()
|
||||
}
|
||||
})
|
||||
|
||||
test("a failing test command fails the gate (fail arm, not pass-only)", async () => {
|
||||
const cfg = config([{ id: "test", kind: "test", command: "bun -e 'process.exit(1)'", parser: "generic" }])
|
||||
await trust(cfg)
|
||||
resetBwrapProbe(false)
|
||||
try {
|
||||
const report = await gates({ scope: "full", config: cfg })
|
||||
expect(report.checks.find((check) => check.id === "test")?.status).toBe("fail")
|
||||
expect(report.testsFailed).toBe(true)
|
||||
expect(report.findings.some((finding) => finding.rule === "test/command-failed")).toBe(true)
|
||||
} finally {
|
||||
resetBwrapProbe()
|
||||
}
|
||||
})
|
||||
|
||||
test("budget pass arms: small, simple, tested changes are quiet; diff-size has both arms", async () => {
|
||||
await write("src/a.ts", "export const a = 1\n")
|
||||
await write("src/a.test.ts", 'import { a } from "./a"\n')
|
||||
const quiet = await gates({
|
||||
touched: ["src/a.ts", "src/a.test.ts"],
|
||||
config: config([], { budgets: { maxFileLines: 20, maxComplexity: 10 } }),
|
||||
})
|
||||
for (const rule of ["budget/file-length", "budget/complexity", "budget/test-with-behavior"])
|
||||
expect([rule, quiet.findings.some((finding) => finding.rule === rule)]).toEqual([rule, false])
|
||||
const budgets = {
|
||||
maxFileLines: 800,
|
||||
maxComplexity: 15,
|
||||
maxDiffLines: 10,
|
||||
requireDepJustification: false,
|
||||
requireTestWithBehavior: false,
|
||||
}
|
||||
const over = await checkBudgets({ dir: fx.root, files: [], budgets, diff: { added: 6, removed: 5 } })
|
||||
expect(over.find((finding) => finding.rule === "budget/diff-size")?.severity).toBe("error")
|
||||
const under = await checkBudgets({ dir: fx.root, files: [], budgets, diff: { added: 5, removed: 5 } })
|
||||
expect(under.some((finding) => finding.rule === "budget/diff-size")).toBe(false)
|
||||
})
|
||||
|
||||
test("a phase that touches only source passes tool-config (pass arm)", async () => {
|
||||
const base = await commit("base")
|
||||
await write("src/a.ts", "export const a = 1\n")
|
||||
const report = await gates({ scope: "full", base, phaseId: "p1", config: config([]) })
|
||||
expect(report.findings.some((finding) => finding.rule === "integrity/tool-config-changed")).toBe(false)
|
||||
})
|
||||
|
||||
test("a valid gates.json loads with no problem (pass arm)", async () => {
|
||||
await write(".factory/gates.json", JSON.stringify({ commands: [] }))
|
||||
const { config: loaded, problem } = await loadGatesConfig(fx.root)
|
||||
expect(problem).toBeUndefined()
|
||||
expect(loaded.commands).toEqual([])
|
||||
})
|
||||
|
||||
test("artifact validation flags a malformed frontier and passes when artifacts are absent", async () => {
|
||||
expect(await validateArtifacts(fx.root, fx.state)).toEqual([])
|
||||
await write(".factory/frontier.json", "{}")
|
||||
const findings = await validateArtifacts(fx.root, fx.state)
|
||||
expect(findings.find((finding) => finding.rule === "schema/frontier")?.file).toBe(".factory/frontier.json")
|
||||
})
|
||||
|
||||
test("formatReport renders the summary, the findings and the overflow pointer", () => {
|
||||
const report: GateReport = {
|
||||
passed: false,
|
||||
findings: [
|
||||
{
|
||||
file: "src/a.ts",
|
||||
line: 2,
|
||||
rule: "lint/suspicious/noDebugger",
|
||||
severity: "error",
|
||||
message: "No debugger",
|
||||
check: "lint",
|
||||
},
|
||||
],
|
||||
totalFindings: 3,
|
||||
waived: 0,
|
||||
checks: [{ id: "lint", status: "fail", durationMs: 1, findings: 3 }],
|
||||
summary: "Gates failed: 3 error(s)",
|
||||
failedTests: [],
|
||||
testsFailed: false,
|
||||
logDir: "/tmp/logs",
|
||||
}
|
||||
const text = formatReport(report)
|
||||
expect(text).toContain("Gates failed: 3 error(s)")
|
||||
expect(text).toContain("src/a.ts:2 lint/suspicious/noDebugger")
|
||||
expect(text).toContain("… 2 more in /tmp/logs/summary.json")
|
||||
})
|
||||
|
||||
test("the lsp gate reports error diagnostics and passes on clean files", async () => {
|
||||
await write("src/a.ts", "export const a = 1\n")
|
||||
const red = await gates({
|
||||
touched: ["src/a.ts"],
|
||||
config: config([]),
|
||||
lsp: {
|
||||
diagnostics: async () => [
|
||||
{ severity: 1, message: "boom", source: "ts", range: { start: { line: 0, character: 0 } } },
|
||||
],
|
||||
},
|
||||
})
|
||||
expect(red.findings.find((finding) => finding.rule === "lsp/ts")).toMatchObject({ file: "src/a.ts", line: 1 })
|
||||
const green = await gates({
|
||||
touched: ["src/a.ts"],
|
||||
config: config([]),
|
||||
lsp: { diagnostics: async () => [] },
|
||||
})
|
||||
expect(green.findings.some((finding) => finding.check === "lsp")).toBe(false)
|
||||
})
|
||||
|
||||
test("gitleaks: empty input is a skip; without the binary it never runs", async () => {
|
||||
expect(await gitleaks(fx.root, [])).toBeUndefined()
|
||||
if (!installed("gitleaks")) expect(await gitleaks(fx.root, ["README.md"])).toBeUndefined()
|
||||
})
|
||||
|
||||
test("control drift is an integrity/control-drift gate finding, not just a halt", async () => {
|
||||
const { rebaseline } = await import("../src/trust/index.ts")
|
||||
await rebaseline(fx.root, "human:tester")
|
||||
await write(".factory/gates.json", '{"commands":[]}\n')
|
||||
const report = await gates({ scope: "full", config: config([]) })
|
||||
expect(report.findings.find((finding) => finding.rule === "integrity/control-drift")).toMatchObject({
|
||||
file: ".factory",
|
||||
severity: "error",
|
||||
check: "integrity",
|
||||
})
|
||||
})
|
||||
})
|
||||
|
||||
@@ -0,0 +1,199 @@
|
||||
// Issue #192 (Darkharvest) integration tests — core side.
|
||||
//
|
||||
// - Harvester targeting a path outside the project is refused, with an
|
||||
// in-project control showing allowed paths still work.
|
||||
// - An Apache-header-only fixture (no LICENSE file) stays clean-room. The
|
||||
// returned literal is recorded as-is: `Apache-2.0`, unverified
|
||||
// (bug #264 observed the claim wording "unknown"; the enforced property —
|
||||
// clean-room — holds, so this asserts current behavior, not the old word).
|
||||
import { afterEach, describe, expect, test } from "bun:test"
|
||||
import { mkdir, mkdtemp, rm, writeFile } from "node:fs/promises"
|
||||
import { tmpdir } from "node:os"
|
||||
import path from "node:path"
|
||||
import {
|
||||
buildMatrix,
|
||||
checkVerdict,
|
||||
DEFAULT_LICENSE_WHITELIST,
|
||||
detectLicense,
|
||||
harvestTools,
|
||||
scanSource,
|
||||
} from "../src/index.ts"
|
||||
|
||||
const dirs: string[] = []
|
||||
const tmp = async (prefix: string) => {
|
||||
const dir = await mkdtemp(path.join(tmpdir(), prefix))
|
||||
dirs.push(dir)
|
||||
return dir
|
||||
}
|
||||
afterEach(async () => {
|
||||
for (const dir of dirs.splice(0)) await rm(dir, { recursive: true, force: true })
|
||||
})
|
||||
|
||||
const write = async (root: string, file: string, content: string) => {
|
||||
await mkdir(path.dirname(path.join(root, file)), { recursive: true })
|
||||
await writeFile(path.join(root, file), content)
|
||||
}
|
||||
|
||||
const MIT_LICENSE = `MIT License
|
||||
|
||||
Copyright (c) 2026 Test
|
||||
|
||||
Permission is hereby granted, free of charge, to any person obtaining a copy
|
||||
of this software and associated documentation files (the "Software"), to deal
|
||||
in the Software without restriction, including without limitation the rights
|
||||
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
||||
copies of the Software, and to permit persons to whom the Software is
|
||||
furnished to do so, subject to the following conditions:
|
||||
|
||||
The above copyright notice and this permission notice shall be included in all
|
||||
copies or substantial portions of the Software.
|
||||
|
||||
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
||||
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
||||
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
||||
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
||||
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
||||
SOFTWARE.
|
||||
`
|
||||
|
||||
describe("issue #192: out-of-project harvest is refused", () => {
|
||||
test("plan, target and scan refuse an outside path; an in-project path works", async () => {
|
||||
const root = await tmp("es-192-root-")
|
||||
const outside = await tmp("es-192-outside-")
|
||||
await write(outside, "key.txt", "SIGNING-KEY-DO-NOT-READ")
|
||||
|
||||
const tools = harvestTools({ root })
|
||||
const plan = tools.find((tool) => tool.name === "es_harvest_plan")!
|
||||
const target = tools.find((tool) => tool.name === "es_harvest_target")!
|
||||
|
||||
// Refused: a harvester plan that points outside the project.
|
||||
await expect(
|
||||
plan.execute(
|
||||
{ objective: "x", candidates: [{ name: "evil", source: `local:${outside}` }], force: true },
|
||||
{ agent: "harvester" },
|
||||
),
|
||||
).rejects.toThrow(/outside the project/)
|
||||
// Refused: the callable verdict check, same seat, same outside path.
|
||||
await expect(
|
||||
target.execute({ objective: "x", targets: [`local:${outside}`] }, { agent: "harvester" }),
|
||||
).rejects.toThrow(/outside the project/)
|
||||
// Refused at the scan layer too (no bytes are read: the throw precedes I/O).
|
||||
await expect(scanSource(root, "evil", { kind: "local", path: outside }, { regexOnly: true })).rejects.toThrow(
|
||||
/outside the project/,
|
||||
)
|
||||
|
||||
// Control: an in-project candidate plans and scans fine.
|
||||
await write(root, "ok/LICENSE", MIT_LICENSE)
|
||||
await write(root, "ok/src/a.ts", "export const a = 1\n")
|
||||
const planned = await plan.execute(
|
||||
{ objective: "x", candidates: [{ name: "ok", source: "local:ok" }], force: true },
|
||||
{ agent: "harvester" },
|
||||
)
|
||||
expect(planned).toContain("ok")
|
||||
const profile = await scanSource(root, "ok", { kind: "local", path: "ok" }, { regexOnly: true })
|
||||
expect(profile.license).toMatchObject({ spdx: "MIT", verified: true, source: "license-file" })
|
||||
})
|
||||
})
|
||||
|
||||
describe("issue #192: Apache-header-only fixture is clean-room", () => {
|
||||
// No LICENSE file anywhere in this fixture: one source file carries only
|
||||
// `SPDX-License-Identifier: Apache-2.0`.
|
||||
const setupHeaderOnly = async (root: string, dir = "header-only") => {
|
||||
await write(root, `${dir}/src/a.ts`, "// SPDX-License-Identifier: Apache-2.0\nexport const a = 1\n")
|
||||
}
|
||||
|
||||
test("detectLicense literal is Apache-2.0, unverified (bug #264); policy forces clean-room", async () => {
|
||||
const root = await tmp("es-192-apache-")
|
||||
await setupHeaderOnly(root)
|
||||
const dir = path.join(root, "header-only")
|
||||
|
||||
const finding = await detectLicense(dir)
|
||||
// Pinned literal: bug #264 notes the claim said "unknown" here, but the
|
||||
// code returns the header id unverified. Assert current behavior so any
|
||||
// change to the literal is a visible diff; the safety property below is
|
||||
// what must never regress.
|
||||
expect(finding.spdx).toBe("Apache-2.0")
|
||||
expect(finding).toMatchObject({ source: "spdx-header", verified: false, family: "permissive" })
|
||||
|
||||
// Fail-closed: an unverified header can never authorize depend/vendor.
|
||||
for (const proposed of ["depend", "vendor"] as const) {
|
||||
const checked = checkVerdict(proposed, finding, DEFAULT_LICENSE_WHITELIST)
|
||||
expect(checked.verdict).toBe("clean-room")
|
||||
expect(checked.downgraded).toMatch(/unverified/)
|
||||
// Dangerous misclassification guard: unknown/copyleft/unverified must
|
||||
// never come back depend/vendor-able. This would be a FINDING, not a fix.
|
||||
expect(checked.verdict).not.toBe("depend")
|
||||
expect(checked.verdict).not.toBe("vendor")
|
||||
}
|
||||
})
|
||||
|
||||
test("scan → matrix downgrades a proposed depend to clean-room", async () => {
|
||||
const root = await tmp("es-192-apache-flow-")
|
||||
await setupHeaderOnly(root)
|
||||
const tools = harvestTools({ root })
|
||||
const call = (name: string, input: Record<string, unknown>) =>
|
||||
tools.find((tool) => tool.name === name)!.execute(input, { agent: "harvester" })
|
||||
|
||||
await call("es_harvest_plan", {
|
||||
objective: "vet the header-only lib",
|
||||
candidates: [{ name: "header-only", source: "local:header-only" }],
|
||||
})
|
||||
const scanned = await call("es_harvest_scan", { candidate: "header-only" })
|
||||
expect(scanned).toContain("Apache-2.0")
|
||||
expect(scanned).toContain("UNVERIFIED")
|
||||
|
||||
const matrixText = await call("es_harvest_matrix", {
|
||||
rows: [{ feature: "widget", cells: [{ candidate: "header-only", verdict: "depend" }] }],
|
||||
})
|
||||
expect(matrixText).toContain("clean-room")
|
||||
|
||||
const profiles = new Map([
|
||||
[
|
||||
"header-only",
|
||||
{
|
||||
version: 1 as const,
|
||||
id: "header-only",
|
||||
name: "header-only",
|
||||
license: await detectLicense(path.join(root, "header-only")),
|
||||
languages: {},
|
||||
dependencies: { runtime: [], dev: [] },
|
||||
size: { files: 0, bytes: 0, tokens: 0 },
|
||||
graph: { internalEdges: 0, external: 0, parsers: { treeSitter: 0, regex: 0 } },
|
||||
provenance: {},
|
||||
warnings: [],
|
||||
scannedAt: new Date().toISOString(),
|
||||
},
|
||||
],
|
||||
])
|
||||
const built = buildMatrix(
|
||||
["header-only"],
|
||||
profiles,
|
||||
[{ feature: "widget", cells: [{ candidate: "header-only", verdict: "depend" }] }],
|
||||
DEFAULT_LICENSE_WHITELIST,
|
||||
)
|
||||
expect(built.matrix.rows[0]?.cells[0]?.verdict).toBe("clean-room")
|
||||
expect(built.matrix.rows[0]?.cells[0]?.downgraded).toMatch(/unverified/)
|
||||
expect(built.downgrades).toHaveLength(1)
|
||||
})
|
||||
|
||||
test("es_harvest_target reports clean-room with the same literal", async () => {
|
||||
const root = await tmp("es-192-apache-target-")
|
||||
await setupHeaderOnly(root)
|
||||
const target = harvestTools({ root }).find((tool) => tool.name === "es_harvest_target")!
|
||||
const text = await target.execute(
|
||||
{ objective: "can we depend on it", targets: ["local:header-only"] },
|
||||
{ agent: "grill" },
|
||||
)
|
||||
const json = /--- verdicts \(JSON\) ---\n(\[[\s\S]*\])[\s]*$/.exec(text)?.[1]
|
||||
expect(json).toBeDefined()
|
||||
const verdicts = JSON.parse(json!) as Array<{
|
||||
target: string
|
||||
license: { spdx: string; family: string; verified: boolean }
|
||||
verdict: string
|
||||
}>
|
||||
expect(verdicts).toHaveLength(1)
|
||||
expect(verdicts[0]?.license).toMatchObject({ spdx: "Apache-2.0", verified: false })
|
||||
expect(verdicts[0]?.verdict).toBe("clean-room")
|
||||
})
|
||||
})
|
||||
@@ -83,7 +83,7 @@ describe("SPDX detection", () => {
|
||||
["JSON", "unknown", "unknown", false],
|
||||
["MIT-0", "unknown", "unknown", false],
|
||||
["X11", "unknown", "unknown", false],
|
||||
["BSD-4-Clause", "BSD-4-Clause", "permissive", false],
|
||||
["BSD-4-Clause", "BSD-4-Clause", "unknown", false],
|
||||
["BUSL-1.1", "BUSL-1.1", "unknown", false],
|
||||
["Elastic-2.0", "Elastic-2.0", "unknown", false],
|
||||
])("%s text → %s (%s, verified %p)", (file, spdx, family, verified) => {
|
||||
@@ -166,6 +166,53 @@ describe("SPDX detection", () => {
|
||||
const unknown = await detectLicense(root)
|
||||
expect(unknown).toMatchObject({ spdx: "unknown", verified: false })
|
||||
})
|
||||
|
||||
test("a single file holding MIT+GPL reads as GPL-3.0, not LGPL (#264.1)", () => {
|
||||
// The GPL-3.0 tail names the Lesser GPL, which used to win the date-line
|
||||
// rules before the GPL-3.0 rule. The earliest license title in the bytes
|
||||
// decides among the GPL family now.
|
||||
expect(identifyLicenseText(`${MIT}\n${GPL3}`)).toMatchObject({
|
||||
spdx: "GPL-3.0",
|
||||
family: "copyleft",
|
||||
verified: true,
|
||||
})
|
||||
expect(identifyLicenseText(`${GPL3}\n${MIT}`)).toMatchObject({ spdx: "GPL-3.0", family: "copyleft" })
|
||||
})
|
||||
|
||||
test("mixed and header-only findings still fail closed at the policy (#264.2)", async () => {
|
||||
// MIT+GPL in one file labels GPL-3.0 (copyleft), and a header-only Apache
|
||||
// project labels Apache-2.0 unverified: both end up clean-room, never
|
||||
// depend/vendor. The enforced property is mixed/unverified → clean-room.
|
||||
const mixed = {
|
||||
...identifyLicenseText(`${MIT}\n${GPL3}`)!,
|
||||
source: "license-file" as const,
|
||||
confidence: "high" as const,
|
||||
}
|
||||
expect(checkVerdict("depend", mixed, DEFAULT_LICENSE_WHITELIST).verdict).toBe("clean-room")
|
||||
const header = await tmp("es-spdx-headeronly-")
|
||||
await write(header, "src/a.ts", "// SPDX-License-Identifier: Apache-2.0\n export const x = 1\n")
|
||||
const fromHeader = await detectLicense(header)
|
||||
expect(fromHeader).toMatchObject({ spdx: "Apache-2.0", verified: false })
|
||||
expect(checkVerdict("depend", fromHeader, DEFAULT_LICENSE_WHITELIST).verdict).toBe("clean-room")
|
||||
})
|
||||
|
||||
test("BSD-4-Clause labels unknown, not permissive (#264.3)", () => {
|
||||
// The advertising clause is an obligation the whitelist does not cover,
|
||||
// so the family must not read as safe next to verified:false.
|
||||
expect(identifyLicenseText(fixture("BSD-4-Clause"))).toMatchObject({
|
||||
spdx: "BSD-4-Clause",
|
||||
family: "unknown",
|
||||
verified: false,
|
||||
})
|
||||
})
|
||||
|
||||
test("a top-level NOTICE without a license file warns about the blind spot (#264.4)", async () => {
|
||||
const root = await tmp("es-spdx-notice-")
|
||||
await write(root, "NOTICE", "Copyright 2026 Example Corp. All rights reserved.\n")
|
||||
const noticed = await detectLicense(root)
|
||||
expect(noticed).toMatchObject({ spdx: "unknown", verified: false })
|
||||
expect(noticed.note ?? "").toContain("NOTICE")
|
||||
})
|
||||
})
|
||||
|
||||
describe("sources", () => {
|
||||
@@ -350,6 +397,27 @@ describe("the fail-closed licence policy", () => {
|
||||
expect(gpl).toMatchObject({ verdict: "clean-room", downgraded: expect.stringContaining("copyleft") })
|
||||
expect(built.downgrades).toHaveLength(1)
|
||||
})
|
||||
|
||||
test("model-supplied evidence is scrubbed of code spans before it reaches the spec (#267)", () => {
|
||||
// matrix.ts used to spread cell.evidence verbatim into the clean-room
|
||||
// spec, so a harvester pasting upstream code contaminated it
|
||||
// mechanically. The scrub is the enforcement behind the "do not copy"
|
||||
// instruction.
|
||||
const profiles = new Map([["mit", profileFor(license({}), "mit")]])
|
||||
const pasted = `rebuild the parser\n\`\`\`ts\nexport const stolen = 1\n\`\`\`\nsee \`internal()\` for details ${"x".repeat(600)}`
|
||||
const built = buildMatrix(
|
||||
["mit"],
|
||||
profiles,
|
||||
[{ feature: "parse", cells: [{ candidate: "mit", verdict: "clean-room", evidence: pasted }] }],
|
||||
DEFAULT_LICENSE_WHITELIST,
|
||||
)
|
||||
const evidence = built.matrix.rows[0]!.cells[0]!.evidence!
|
||||
expect(evidence).toContain("rebuild the parser")
|
||||
expect(evidence).not.toContain("```")
|
||||
expect(evidence).not.toContain("`")
|
||||
expect(evidence).not.toContain("stolen")
|
||||
expect(evidence.length).toBeLessThanOrEqual(500)
|
||||
})
|
||||
})
|
||||
|
||||
describe("the tool loop", () => {
|
||||
|
||||
@@ -0,0 +1,262 @@
|
||||
// Issue #196 §2 — engine-level proof for every bridged hook event.
|
||||
//
|
||||
// Each event gets a hook that writes a marker file; the test drives the
|
||||
// matching engine method and asserts the marker landed. This complements
|
||||
// packages/opencode/test/hooks-events.test.ts, which drives the same events
|
||||
// through the real in-process OpenCode v2 host.
|
||||
//
|
||||
// NOTE (bug #265): hook commands run UNSANDBOXED with the full parent env
|
||||
// (run.ts uses inheritEnv:true and no sandbox knob). The "parent env is
|
||||
// visible" test below pins that observable behaviour without changing it —
|
||||
// Claude Code parity by design, not a sandbox.
|
||||
import { afterEach, beforeEach, describe, expect, test } from "bun:test"
|
||||
import { mkdir, mkdtemp, rm, writeFile } from "node:fs/promises"
|
||||
import { tmpdir } from "node:os"
|
||||
import path from "node:path"
|
||||
import { type HumanSigner, sealHumanKey, unlockHumanKey } from "../src/approval/keystore.ts"
|
||||
import { fromOpenCode, HookEngine, type HookSource, trustProject } from "../src/index.ts"
|
||||
|
||||
let root: string
|
||||
let state: string
|
||||
const PASSPHRASE = "test-passphrase-196-events"
|
||||
let signer: HumanSigner
|
||||
beforeEach(async () => {
|
||||
root = await mkdtemp(path.join(tmpdir(), "es-hooks-events-"))
|
||||
state = await mkdtemp(path.join(tmpdir(), "es-hooks-events-state-"))
|
||||
await sealHumanKey(PASSPHRASE, state)
|
||||
signer = await unlockHumanKey(PASSPHRASE, state)
|
||||
await mkdir(path.join(root, ".opencode"), { recursive: true })
|
||||
})
|
||||
afterEach(async () => {
|
||||
await rm(root, { recursive: true, force: true })
|
||||
await rm(state, { recursive: true, force: true })
|
||||
})
|
||||
|
||||
const sources = (): HookSource[] => [{ file: path.join(root, ".opencode/hooks.json"), scope: "project" }]
|
||||
async function engine(hooks: Record<string, unknown>) {
|
||||
await writeFile(path.join(root, ".opencode/hooks.json"), JSON.stringify({ hooks }))
|
||||
let created = await HookEngine.create({ root, stateDir: state, sources: sources(), audit: false })
|
||||
const status = created.status()
|
||||
await trustProject(root, status.projectHash, status.projectLines, { stateDir: state, kind: "hooks", signer })
|
||||
created = await HookEngine.create({ root, stateDir: state, sources: sources(), audit: false })
|
||||
return created
|
||||
}
|
||||
const base = { sessionId: "ses_196", agent: "build" }
|
||||
const marker = (name: string) => path.join(root, name)
|
||||
const touch = (name: string, extra = "") => `touch ${marker(name)}${extra ? `; ${extra}` : ""}`
|
||||
|
||||
describe("hook events write marker files (issue #196 §2)", () => {
|
||||
test("PreToolUse marker lands; exit 2 deny blocks the tool", async () => {
|
||||
const hooks = await engine({
|
||||
PreToolUse: [
|
||||
{
|
||||
matcher: "Write",
|
||||
hooks: [
|
||||
{
|
||||
type: "command",
|
||||
command: `${touch("PRE_196")}; input=$(cat); echo "$input" | grep -q "deny-pre-196" && { echo "pre-deny-196 blocked" >&2; exit 2; }; exit 0`,
|
||||
},
|
||||
],
|
||||
},
|
||||
],
|
||||
})
|
||||
const ok = await hooks.preToolUse(base, fromOpenCode("write", { path: "a.txt", content: "fine" }, root), "t1")
|
||||
expect(ok.decision).toBeUndefined()
|
||||
expect(await Bun.file(marker("PRE_196")).exists()).toBe(true)
|
||||
const denied = await hooks.preToolUse(
|
||||
base,
|
||||
fromOpenCode("write", { path: "b.txt", content: "deny-pre-196" }, root),
|
||||
"t2",
|
||||
)
|
||||
expect(denied).toMatchObject({ decision: "deny", reason: "pre-deny-196 blocked" })
|
||||
})
|
||||
|
||||
test("PostToolUse marker lands with additionalContext", async () => {
|
||||
const hooks = await engine({
|
||||
PostToolUse: [
|
||||
{
|
||||
matcher: "Read",
|
||||
hooks: [
|
||||
{
|
||||
type: "command",
|
||||
command: `${touch("POST_196")}; echo '{"hookSpecificOutput":{"hookEventName":"PostToolUse","additionalContext":"POST-MARKER-196"}}'`,
|
||||
},
|
||||
],
|
||||
},
|
||||
],
|
||||
})
|
||||
const out = await hooks.postToolUse(base, fromOpenCode("read", { path: "a.txt" }, root), "t1", { ok: true }, false)
|
||||
expect(await Bun.file(marker("POST_196")).exists()).toBe(true)
|
||||
expect(out.additionalContext).toContain("POST-MARKER-196")
|
||||
})
|
||||
|
||||
test("PostToolUseFailure marker lands when driven with failed=true (bug #268)", async () => {
|
||||
const hooks = await engine({
|
||||
PostToolUseFailure: [
|
||||
{
|
||||
matcher: "Read",
|
||||
hooks: [
|
||||
{
|
||||
type: "command",
|
||||
command: `${touch("FAIL_196")}; echo '{"hookSpecificOutput":{"hookEventName":"PostToolUseFailure","additionalContext":"FAILURE-MARKER-196"}}'`,
|
||||
},
|
||||
],
|
||||
},
|
||||
],
|
||||
})
|
||||
// failed=false must NOT run the PostToolUseFailure handler.
|
||||
const ok = await hooks.postToolUse(base, fromOpenCode("read", { path: "a.txt" }, root), "t1", "content", false)
|
||||
expect(ok.additionalContext).toEqual([])
|
||||
expect(await Bun.file(marker("FAIL_196")).exists()).toBe(false)
|
||||
// failed=true routes to the PostToolUseFailure event (engine.ts).
|
||||
const failed = await hooks.postToolUse(
|
||||
base,
|
||||
fromOpenCode("read", { path: "missing.txt" }, root),
|
||||
"t2",
|
||||
"Error: file not found",
|
||||
true,
|
||||
)
|
||||
expect(await Bun.file(marker("FAIL_196")).exists()).toBe(true)
|
||||
expect(failed.additionalContext).toContain("FAILURE-MARKER-196")
|
||||
})
|
||||
|
||||
test("UserPromptSubmit marker lands; block blocks", async () => {
|
||||
const hooks = await engine({
|
||||
UserPromptSubmit: [
|
||||
{
|
||||
hooks: [
|
||||
{
|
||||
type: "command",
|
||||
command: `${touch("PROMPT_196")}; grep -q "block-prompt-196" && echo '{"decision":"block","reason":"prompt blocked 196"}' || echo "PROMPT-MARKER-196"`,
|
||||
},
|
||||
],
|
||||
},
|
||||
],
|
||||
})
|
||||
const out = await hooks.userPromptSubmit(base, "hello")
|
||||
expect(await Bun.file(marker("PROMPT_196")).exists()).toBe(true)
|
||||
expect(out.additionalContext).toContain("PROMPT-MARKER-196")
|
||||
expect((await hooks.userPromptSubmit(base, "please block-prompt-196")).block).toContain("prompt blocked 196")
|
||||
})
|
||||
|
||||
test("SessionStart marker lands for startup only", async () => {
|
||||
const hooks = await engine({
|
||||
SessionStart: [
|
||||
{ matcher: "startup", hooks: [{ type: "command", command: touch("START_196") + "; echo START-MARKER-196" }] },
|
||||
],
|
||||
})
|
||||
const out = await hooks.sessionStart(base, "startup")
|
||||
expect(await Bun.file(marker("START_196")).exists()).toBe(true)
|
||||
expect(out.additionalContext).toContain("START-MARKER-196")
|
||||
await rm(marker("START_196"), { force: true })
|
||||
expect((await hooks.sessionStart(base, "compact")).additionalContext).toEqual([])
|
||||
expect(await Bun.file(marker("START_196")).exists()).toBe(false)
|
||||
})
|
||||
|
||||
test("PermissionRequest marker lands; deny JSON denies", async () => {
|
||||
const hooks = await engine({
|
||||
PermissionRequest: [
|
||||
{
|
||||
matcher: "Read",
|
||||
hooks: [
|
||||
{
|
||||
type: "command",
|
||||
command: `${touch("PERM_196")}; echo '{"hookSpecificOutput":{"hookEventName":"PermissionRequest","decision":{"behavior":"deny","message":"perm-deny-196"}}}'`,
|
||||
},
|
||||
],
|
||||
},
|
||||
],
|
||||
})
|
||||
const out = await hooks.permissionRequest(base, fromOpenCode("read", { path: ".env" }, root))
|
||||
expect(await Bun.file(marker("PERM_196")).exists()).toBe(true)
|
||||
expect(out).toMatchObject({ behavior: "deny", message: "perm-deny-196" })
|
||||
})
|
||||
|
||||
test("Stop marker lands; exit 2 blocks (emulated after the turn, ADVISORY for plain chat)", async () => {
|
||||
const hooks = await engine({
|
||||
Stop: [
|
||||
{
|
||||
hooks: [
|
||||
{
|
||||
type: "command",
|
||||
// The engine passes stop_hook_active in the input; the hook itself
|
||||
// suppresses the block on a continuation, as Claude Code does.
|
||||
command: `${touch("STOP_196")}; input=$(cat); echo "$input" | grep -q '"stop_hook_active":true' && exit 0; echo "stop-continue-196" >&2; exit 2`,
|
||||
},
|
||||
],
|
||||
},
|
||||
],
|
||||
})
|
||||
const out = await hooks.stop(base, { stopHookActive: false, lastAssistantMessage: "done" })
|
||||
expect(await Bun.file(marker("STOP_196")).exists()).toBe(true)
|
||||
expect(out.block).toContain("stop-continue-196")
|
||||
// stop_hook_active suppresses the block, as Claude Code does.
|
||||
expect((await hooks.stop(base, { stopHookActive: true, lastAssistantMessage: "done" })).block).toBeUndefined()
|
||||
})
|
||||
})
|
||||
|
||||
describe("x-es.ShellPreExec rewrite AND deny (bug #268)", () => {
|
||||
test("rewrite branch returns the replacement command", async () => {
|
||||
const hooks = await engine({
|
||||
"x-es.ShellPreExec": [
|
||||
{
|
||||
hooks: [
|
||||
{
|
||||
type: "command",
|
||||
command: `input=$(cat); echo "$input" | grep -q "rewrite-shell-196" && echo '{"hookSpecificOutput":{"updatedInput":{"command":"echo shell-rewritten-196"}}}' || true`,
|
||||
},
|
||||
],
|
||||
},
|
||||
],
|
||||
})
|
||||
expect(await hooks.shellPreExec(base, { command: "echo rewrite-shell-196", cwd: root })).toMatchObject({
|
||||
command: "echo shell-rewritten-196",
|
||||
})
|
||||
expect((await hooks.shellPreExec(base, { command: "echo plain", cwd: root })).command).toBeUndefined()
|
||||
})
|
||||
|
||||
test("deny branch: exit 2 denies; deny wins over a simultaneous rewrite (command && !deny precedence)", async () => {
|
||||
const hooks = await engine({
|
||||
"x-es.ShellPreExec": [
|
||||
{
|
||||
hooks: [
|
||||
{
|
||||
type: "command",
|
||||
command: `input=$(cat); echo "$input" | grep -q "deny-shell-196" && { echo "shell-deny-196 blocked" >&2; exit 2; }; echo "$input" | grep -q "both-shell-196" && echo '{"hookSpecificOutput":{"permissionDecision":"deny","permissionDecisionReason":"both denied 196","updatedInput":{"command":"echo should-not-run-196"}}}' || true`,
|
||||
},
|
||||
],
|
||||
},
|
||||
],
|
||||
})
|
||||
// Exit-2 deny carries the stderr reason.
|
||||
const denied = await hooks.shellPreExec(base, { command: "echo deny-shell-196", cwd: root })
|
||||
expect(denied.deny).toContain("shell-deny-196 blocked")
|
||||
expect(denied.command).toBeUndefined()
|
||||
// JSON deny with a simultaneous updatedInput.command: deny wins, command is suppressed.
|
||||
const both = await hooks.shellPreExec(base, { command: "echo both-shell-196", cwd: root })
|
||||
expect(both.deny).toContain("both denied 196")
|
||||
expect(both.command).toBeUndefined()
|
||||
})
|
||||
})
|
||||
|
||||
describe("hooks run unsandboxed with parent env (bug #265, observable behaviour)", () => {
|
||||
test("parent process env is visible inside a hook command", async () => {
|
||||
const key = "ES_HOOK_PARENT_PROBE_196"
|
||||
const value = `probe-196-${Date.now()}`
|
||||
process.env[key] = value
|
||||
try {
|
||||
const hooks = await engine({
|
||||
PreToolUse: [
|
||||
{
|
||||
matcher: "Bash",
|
||||
hooks: [{ type: "command", command: `echo "$${key}" > ${marker("ENV_196")}` }],
|
||||
},
|
||||
],
|
||||
})
|
||||
await hooks.preToolUse(base, fromOpenCode("shell", { command: "echo hi" }, root), "t1")
|
||||
expect((await Bun.file(marker("ENV_196")).text()).trim()).toBe(value)
|
||||
} finally {
|
||||
delete process.env[key]
|
||||
}
|
||||
})
|
||||
})
|
||||
@@ -14,6 +14,7 @@ import {
|
||||
trustProject,
|
||||
verifyAuditChain,
|
||||
} from "../src/index.ts"
|
||||
import { resetBwrapProbe } from "../src/util/proc.ts"
|
||||
|
||||
let root: string
|
||||
let state: string
|
||||
@@ -44,14 +45,24 @@ async function script(name: string, body: string) {
|
||||
}
|
||||
async function engine(
|
||||
hooks: Record<string, unknown>,
|
||||
options: { trust?: boolean; generate?: (p: string) => Promise<string> } = {},
|
||||
options: {
|
||||
trust?: boolean
|
||||
generate?: (p: string) => Promise<string>
|
||||
sandbox?: "none" | "readonly"
|
||||
offline?: boolean
|
||||
} = {},
|
||||
) {
|
||||
await writeFile(path.join(root, ".opencode/hooks.json"), JSON.stringify({ hooks }))
|
||||
const extra = {
|
||||
...(options.generate ? { generate: options.generate } : {}),
|
||||
...(options.sandbox !== undefined ? { sandbox: options.sandbox } : {}),
|
||||
...(options.offline !== undefined ? { offline: options.offline } : {}),
|
||||
}
|
||||
let created = await HookEngine.create({
|
||||
root,
|
||||
stateDir: state,
|
||||
sources: sources(),
|
||||
...(options.generate ? { generate: options.generate } : {}),
|
||||
...extra,
|
||||
})
|
||||
if (options.trust !== false) {
|
||||
const status = created.status()
|
||||
@@ -60,7 +71,7 @@ async function engine(
|
||||
root,
|
||||
stateDir: state,
|
||||
sources: sources(),
|
||||
...(options.generate ? { generate: options.generate } : {}),
|
||||
...extra,
|
||||
})
|
||||
}
|
||||
return created
|
||||
@@ -100,6 +111,30 @@ describe("matching", () => {
|
||||
).toBe(path.join(root, "x.ts"))
|
||||
})
|
||||
|
||||
test("matcher corpus: exact lists vs unanchored regex (the Claude Code rule, #268)", () => {
|
||||
// The rule (match.ts): "*", "" or omitted matches everything; a value of
|
||||
// only [A-Za-z0-9_-, |] is an exact-match list; anything else is an
|
||||
// unanchored JavaScript regex (so "Edit" never matches "NotebookEdit",
|
||||
// while "Edit.*" does).
|
||||
const cases: Array<[string | undefined, string, boolean]> = [
|
||||
[undefined, "Bash", true],
|
||||
["", "Bash", true],
|
||||
["*", "Anything", true],
|
||||
["Bash", "Bash", true],
|
||||
["Bash", "bash", false],
|
||||
["Edit|Write", "Write", true],
|
||||
["Edit, Write", "Edit", true],
|
||||
["Edit", "NotebookEdit", false],
|
||||
["Edit.*", "NotebookEdit", true],
|
||||
["^Edit", "NotebookEdit", false],
|
||||
["mcp__memory__.*", "mcp__memory__create", true],
|
||||
["mcp__memory", "mcp__memory__create", false],
|
||||
["[invalid", "Bash", false],
|
||||
]
|
||||
for (const [matcher, value, expected] of cases)
|
||||
expect(matcherMatches(matcher, [value]), `${String(matcher)} vs ${value}`).toBe(expected)
|
||||
})
|
||||
|
||||
test("`if` rules use permission syntax over subcommands and paths", () => {
|
||||
expect(subcommands("FOO=bar git push && npm test").parts).toEqual(["git push", "npm test"])
|
||||
const tool = (command: string) => ({ toolNames: ["Bash", "shell"], toolInput: { command }, cwd: root })
|
||||
@@ -288,6 +323,107 @@ describe("other events", () => {
|
||||
})
|
||||
expect((await hooks.shellPreExec(base, { command: "ls", cwd: root })).command).toBeUndefined()
|
||||
})
|
||||
|
||||
test("x-es.ShellPreExec deny wins and drops the rewritten command (#268)", async () => {
|
||||
const hooks = await engine({
|
||||
"x-es.ShellPreExec": [
|
||||
{
|
||||
hooks: [
|
||||
{
|
||||
type: "command",
|
||||
command: `echo '{"hookSpecificOutput":{"permissionDecision":"deny","permissionDecisionReason":"no shells here","updatedInput":{"command":"evil"}}}'`,
|
||||
},
|
||||
],
|
||||
},
|
||||
],
|
||||
})
|
||||
// deny + command: the `command && !deny` precedence keeps the deny only.
|
||||
const denied = await hooks.shellPreExec(base, { command: "ls", cwd: root })
|
||||
expect(denied.deny).toBe("no shells here")
|
||||
expect(denied.command).toBeUndefined()
|
||||
|
||||
const blockers = await engine({
|
||||
"x-es.ShellPreExec": [{ hooks: [{ type: "command", command: `echo "blocked reason" >&2; exit 2` }] }],
|
||||
})
|
||||
expect((await blockers.shellPreExec(base, { command: "ls", cwd: root })).deny).toBe("blocked reason")
|
||||
})
|
||||
|
||||
test("PostToolUseFailure routes failing tools; PostToolUse does not fire on failure (#268)", async () => {
|
||||
const hooks = await engine({
|
||||
PostToolUse: [
|
||||
{
|
||||
matcher: "Write",
|
||||
hooks: [{ type: "command", command: `echo '{"decision":"block","reason":"post ok"}'` }],
|
||||
},
|
||||
],
|
||||
PostToolUseFailure: [
|
||||
{
|
||||
matcher: "Write",
|
||||
hooks: [{ type: "command", command: `echo '{"decision":"block","reason":"post-failure lint"}'` }],
|
||||
},
|
||||
],
|
||||
})
|
||||
const write = fromOpenCode("write", { path: "a.ts", content: "" }, root)
|
||||
const failed = await hooks.postToolUse(base, write, "t", "boom", true)
|
||||
expect(failed.feedback).toEqual(["post-failure lint"])
|
||||
const ok = await hooks.postToolUse(base, write, "t", { ok: true }, false)
|
||||
expect(ok.feedback).toEqual(["post ok"])
|
||||
})
|
||||
|
||||
test("hook commands run with the parent environment by design (Claude Code parity, #265)", async () => {
|
||||
process.env.ES_HOOK_PARENT_ENV_PIN = "pinned-123"
|
||||
try {
|
||||
const hooks = await engine({
|
||||
UserPromptSubmit: [{ hooks: [{ type: "command", command: 'echo "pin=$ES_HOOK_PARENT_ENV_PIN"' }] }],
|
||||
})
|
||||
expect(await hooks.userPromptSubmit(base, "hi")).toMatchObject({ additionalContext: ["pin=pinned-123"] })
|
||||
} finally {
|
||||
delete process.env.ES_HOOK_PARENT_ENV_PIN
|
||||
}
|
||||
})
|
||||
|
||||
test("SessionStart routes every source; an unmatched source is an explicit notice, never silent (#289)", async () => {
|
||||
const hooks = await engine({
|
||||
SessionStart: [
|
||||
{ matcher: "startup", hooks: [{ type: "command", command: "echo 'project uses bun'" }] },
|
||||
{ matcher: "resume", hooks: [{ type: "command", command: "echo 'welcome back'" }] },
|
||||
],
|
||||
})
|
||||
expect(await hooks.sessionStart(base, "startup")).toMatchObject({ additionalContext: ["project uses bun"] })
|
||||
expect(await hooks.sessionStart(base, "resume")).toMatchObject({ additionalContext: ["welcome back"] })
|
||||
// compact/clear/fork have no matching hook here: explicit notice, not silence.
|
||||
for (const source of ["compact", "clear", "fork"] as const) {
|
||||
const result = await hooks.sessionStart(base, source)
|
||||
expect(result.additionalContext).toEqual([])
|
||||
expect(result.notices.join("\n")).toContain(`none match source "${source}"`)
|
||||
}
|
||||
// No SessionStart hook at all: silent (nothing to be confused about).
|
||||
const bare = await engine({ PostToolUse: [{ hooks: [{ type: "command", command: "true" }] }] })
|
||||
expect((await bare.sessionStart(base, "resume")).notices).toEqual([])
|
||||
})
|
||||
|
||||
test("the engine sandbox knob reaches hook subprocesses: gates fail closed, observers fail open (#265)", async () => {
|
||||
resetBwrapProbe(false)
|
||||
try {
|
||||
const hooks = await engine(
|
||||
{
|
||||
PreToolUse: [
|
||||
{ matcher: "Bash", hooks: [{ type: "command", command: "true", "x-es-policy": "gate" }] },
|
||||
{ matcher: "Read", hooks: [{ type: "command", command: "true" }] },
|
||||
],
|
||||
},
|
||||
{ sandbox: "readonly" },
|
||||
)
|
||||
const gated = await hooks.preToolUse(base, bash("ls"), "t1")
|
||||
expect(gated.decision).toBe("deny")
|
||||
expect(gated.reason).toContain("bwrap")
|
||||
const observed = await hooks.preToolUse(base, fromOpenCode("read", { path: "a" }, root), "t2")
|
||||
expect(observed.decision).toBeUndefined()
|
||||
expect(observed.notices.join("\n")).toContain("bwrap")
|
||||
} finally {
|
||||
resetBwrapProbe()
|
||||
}
|
||||
})
|
||||
})
|
||||
|
||||
describe("loading", () => {
|
||||
|
||||
@@ -0,0 +1,273 @@
|
||||
// #204 integration: harvest + distill over several unit-constructed run
|
||||
// dirs (one with a broken audit chain); scrubbed output and the
|
||||
// code-disposes gate dropping fabricated evidence with reasons.
|
||||
//
|
||||
// Unit-constructed only: no real factory runs, no timing/perf assertions
|
||||
// (#155 is flaky by nature and tracked separately), no --open-pr.
|
||||
import { describe, expect, test } from "bun:test"
|
||||
import { appendFile, mkdir, mkdtemp, readdir, readFile, rm, stat, writeFile } from "node:fs/promises"
|
||||
import { tmpdir } from "node:os"
|
||||
import path from "node:path"
|
||||
import {
|
||||
appendAuditEntry,
|
||||
clusterTelemetry,
|
||||
disposedWithReasons,
|
||||
disposeReason,
|
||||
distillProposals,
|
||||
factoryLayout,
|
||||
harvestRun,
|
||||
harvestTelemetry,
|
||||
signEngineFile,
|
||||
TelemetrySchema,
|
||||
undisposed,
|
||||
writeDropped,
|
||||
writeProposals,
|
||||
} from "../src/index.ts"
|
||||
|
||||
// Planted secrets (fake values in the style of the existing scrub tests).
|
||||
// Every one must be absent from harvested telemetry.
|
||||
const GH_A = "ghp_plantedHarvest001AAA"
|
||||
const GH_BROKEN = "ghu_BrokenChain999aaa"
|
||||
const SK_ANT = "sk-ant-planted0000XYZ"
|
||||
const AKIA = "AKIAIOSFODNN7EXAMPLE"
|
||||
const SLACK = "xoxb-12345-abcdef"
|
||||
const BEARER = "eyJhbGciOiJIUzI1NiJ9"
|
||||
const PEM_BODY = "MIIEowIBAAKCAQEAplanted"
|
||||
const HUNTER = "hunter2-planted"
|
||||
|
||||
const QA_REASON = "lint gate red"
|
||||
const GATE_RULE = "lint/no-unused-vars"
|
||||
|
||||
async function writeState(root: string, id: string, haltReason: string): Promise<void> {
|
||||
const file = factoryLayout(root).state
|
||||
await mkdir(path.dirname(file), { recursive: true })
|
||||
await writeFile(
|
||||
file,
|
||||
`${JSON.stringify(
|
||||
{
|
||||
version: 2,
|
||||
runId: id,
|
||||
stage: "HALTED",
|
||||
mode: "build",
|
||||
createdAt: "2026-10-04T00:00:00.000Z",
|
||||
updatedAt: "2026-10-04T01:00:00.000Z",
|
||||
spend: { usd: 1, estimated: false, events: 5 },
|
||||
phases: [
|
||||
{
|
||||
id: "p1",
|
||||
title: "phase",
|
||||
status: "failed",
|
||||
goalHash: "a".repeat(64),
|
||||
failures: 2,
|
||||
replanned: false,
|
||||
history: [],
|
||||
},
|
||||
],
|
||||
halt: { reason: haltReason, at: "2026-10-04T01:00:00.000Z", from: "QA" },
|
||||
audits: [],
|
||||
},
|
||||
null,
|
||||
2,
|
||||
)}\n`,
|
||||
)
|
||||
}
|
||||
|
||||
async function writeGateSummary(root: string, id: string): Promise<void> {
|
||||
const dir = path.join(factoryLayout(root).runs, id, "gates", "2026-10-04T00-30-full")
|
||||
await mkdir(dir, { recursive: true })
|
||||
await writeFile(
|
||||
path.join(dir, "summary.json"),
|
||||
`${JSON.stringify(
|
||||
{
|
||||
passed: false,
|
||||
findings: [
|
||||
{ file: "src/x.ts", line: 3, rule: GATE_RULE, severity: "error", message: "unused", check: "lint" },
|
||||
{ file: "src/y.ts", line: 9, rule: GATE_RULE, severity: "error", message: "unused", check: "lint" },
|
||||
],
|
||||
},
|
||||
null,
|
||||
2,
|
||||
)}\n`,
|
||||
)
|
||||
}
|
||||
|
||||
async function writeChain(root: string, haltReason: string): Promise<void> {
|
||||
await appendAuditEntry(root, { actor: "system", action: "stage.build" })
|
||||
await appendAuditEntry(root, { actor: "agent:qa", action: "qa.fail", payload: { phase: "p1", reason: QA_REASON } })
|
||||
await appendAuditEntry(root, { actor: "system", action: "stage.halted", payload: { reason: haltReason } })
|
||||
}
|
||||
|
||||
/** Two sealed runs plus one sealed run with a broken chain, and an evals dir. */
|
||||
async function buildFixture(): Promise<{
|
||||
parent: string
|
||||
stateDir: string
|
||||
runs: string[]
|
||||
broken: string
|
||||
evals: string
|
||||
cleanup(): Promise<void>
|
||||
}> {
|
||||
const parent = await mkdtemp(path.join(tmpdir(), "es-204-core-"))
|
||||
const stateDir = await mkdtemp(path.join(tmpdir(), "es-204-core-state-"))
|
||||
const haltA = `phase p1 failed: GITHUB_TOKEN=${GH_A} with Bearer ${BEARER}`
|
||||
const haltB =
|
||||
`phase p1 failed: api-key: ${HUNTER} plus ` +
|
||||
`-----BEGIN RSA PRIVATE KEY-----\n${PEM_BODY}\n-----END RSA PRIVATE KEY-----`
|
||||
const haltBroken = `phase p1 failed: slack ${SLACK} and ${GH_BROKEN}`
|
||||
const ids = ["run-a", "run-b"] as const
|
||||
const halts: Record<string, string> = { "run-a": haltA, "run-b": haltB }
|
||||
for (const id of ids) {
|
||||
const root = path.join(parent, id)
|
||||
await writeState(root, id, halts[id]!)
|
||||
await writeGateSummary(root, id)
|
||||
await writeChain(root, halts[id]!)
|
||||
await signEngineFile(factoryLayout(root).state, stateDir)
|
||||
}
|
||||
const broken = path.join(parent, "run-broken")
|
||||
await writeState(broken, "run-broken", haltBroken)
|
||||
await writeGateSummary(broken, "run-broken")
|
||||
await writeChain(broken, haltBroken)
|
||||
await signEngineFile(factoryLayout(broken).state, stateDir)
|
||||
// Break the chain with a corrupt trailing line: the valid prefix stays so
|
||||
// the run still contributes its gate/qa signals, but verification fails.
|
||||
await appendFile(factoryLayout(broken).audit, "not-json\n")
|
||||
const evals = path.join(parent, "evals")
|
||||
await mkdir(evals, { recursive: true })
|
||||
await writeFile(
|
||||
path.join(evals, "agg.json"),
|
||||
`${JSON.stringify(
|
||||
{
|
||||
at: "2026-10-04T00-00-00-000Z",
|
||||
model: "test/model",
|
||||
results: [
|
||||
{ id: "case-x", pass: false, failures: ["gate lint red", "gate lint red", `leaked ${SK_ANT}`], steps: 9 },
|
||||
{ id: "case-y", pass: false, failures: ["gate lint red", `key ${AKIA} here`], steps: 4 },
|
||||
{ id: "case-ok", pass: true, failures: [], steps: 3 },
|
||||
],
|
||||
},
|
||||
null,
|
||||
2,
|
||||
)}\n`,
|
||||
)
|
||||
return {
|
||||
parent,
|
||||
stateDir,
|
||||
runs: [path.join(parent, "run-a"), path.join(parent, "run-b"), broken],
|
||||
broken,
|
||||
evals,
|
||||
async cleanup() {
|
||||
await rm(parent, { recursive: true, force: true })
|
||||
await rm(stateDir, { recursive: true, force: true })
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
async function walkFiles(dir: string): Promise<string[]> {
|
||||
const out: string[] = []
|
||||
const visit = async (current: string): Promise<void> => {
|
||||
for (const name of await readdir(current)) {
|
||||
const full = path.join(current, name)
|
||||
const info = await stat(full).catch(() => undefined)
|
||||
if (info?.isDirectory()) await visit(full)
|
||||
else out.push(full)
|
||||
}
|
||||
}
|
||||
await visit(dir)
|
||||
return out.sort()
|
||||
}
|
||||
|
||||
describe("issue #204: harvest + distill over several runs incl. a broken chain", () => {
|
||||
test("harvest is scrubbed, schema-valid; the broken chain is reported, never repaired", async () => {
|
||||
const fix = await buildFixture()
|
||||
try {
|
||||
const brokenFile = factoryLayout(fix.broken).audit
|
||||
const before = await readFile(brokenFile, "utf8")
|
||||
const telemetry = await harvestTelemetry({ runs: fix.runs, evals: [fix.evals] }, { stateDir: fix.stateDir })
|
||||
expect(TelemetrySchema.parse(telemetry)).toEqual(telemetry)
|
||||
expect(telemetry.runs.map((run) => run.id)).toEqual(["run-a", "run-b", "run-broken"])
|
||||
const text = JSON.stringify(telemetry)
|
||||
for (const secret of [GH_A, GH_BROKEN, SK_ANT, AKIA, SLACK, BEARER, PEM_BODY, HUNTER])
|
||||
expect(text).not.toContain(secret)
|
||||
// Shape is preserved: names stay, only values become [redacted].
|
||||
expect(text).toContain("GITHUB_TOKEN=")
|
||||
expect(text).toContain("Bearer")
|
||||
// The broken chain is reported with an error and left on disk untouched.
|
||||
const brokenRun = telemetry.runs.find((run) => run.id === "run-broken")!
|
||||
expect(brokenRun.auditChain.valid).toBe(false)
|
||||
expect(typeof brokenRun.auditChain.error).toBe("string")
|
||||
expect(await readFile(brokenFile, "utf8")).toBe(before)
|
||||
// The sealed runs keep their valid chains and seals.
|
||||
for (const id of ["run-a", "run-b"]) {
|
||||
const run = telemetry.runs.find((entry) => entry.id === id)!
|
||||
expect(run.auditChain.valid).toBe(true)
|
||||
expect(run.seal).toEqual({ valid: true })
|
||||
}
|
||||
// Direct harvestRun agrees: broken is reported, not repaired.
|
||||
const direct = await harvestRun(fix.broken, { stateDir: fix.stateDir })
|
||||
expect(direct.auditChain.valid).toBe(false)
|
||||
expect(await readFile(brokenFile, "utf8")).toBe(before)
|
||||
} finally {
|
||||
await fix.cleanup()
|
||||
}
|
||||
})
|
||||
|
||||
test("distill clusters all three kinds; fabricated evidence is dropped with reasons", async () => {
|
||||
const fix = await buildFixture()
|
||||
try {
|
||||
const telemetry = await harvestTelemetry({ runs: fix.runs, evals: [fix.evals] }, { stateDir: fix.stateDir })
|
||||
const thresholds = { minOccurrences: 2, minRuns: 2 }
|
||||
const clusters = clusterTelemetry(telemetry, thresholds)
|
||||
expect(clusters.length).toBeGreaterThanOrEqual(3)
|
||||
const proposals = await distillProposals(telemetry, thresholds)
|
||||
expect(proposals.length).toBeGreaterThanOrEqual(3)
|
||||
expect([...new Set(proposals.map((proposal) => proposal.kind))].sort()).toEqual([
|
||||
"domain-pack",
|
||||
"gate-tuning",
|
||||
"prompt-guidance",
|
||||
])
|
||||
expect(undisposed(proposals, telemetry)).toEqual(proposals)
|
||||
// Fabricated evidence: a quote the harvester never said is disposed.
|
||||
const [first] = proposals
|
||||
const forged = { ...first!, cluster: { ...first!.cluster, evidence: ["run x: invented quote"] } }
|
||||
const reasons = disposeReason(forged, telemetry)
|
||||
expect(reasons.some((reason) => reason.includes("verbatim"))).toBe(true)
|
||||
expect(undisposed([forged], telemetry)).toEqual([])
|
||||
const dropped = disposedWithReasons([forged], telemetry)
|
||||
expect(dropped).toHaveLength(1)
|
||||
expect(dropped[0]!.reasons.some((reason) => reason.includes("verbatim"))).toBe(true)
|
||||
// Hallucinated model numerals are disposed too.
|
||||
const tainted = { ...first!, rationale: `${first!.rationale} As gpt-4 would suggest.` }
|
||||
expect(disposeReason(tainted, telemetry).some((reason) => reason.includes("hallucinated model numeral"))).toBe(
|
||||
true,
|
||||
)
|
||||
// Dropped proposals are reported with reasons, byte-identical on rerun.
|
||||
const out = await mkdtemp(path.join(tmpdir(), "es-204-distill-"))
|
||||
try {
|
||||
const rendered = new Map(
|
||||
await Promise.all(
|
||||
proposals.map(async (proposal) => {
|
||||
const { files } = await import("../src/improve/distill.ts").then((mod) =>
|
||||
mod.renderProposalFiles(proposal.cluster),
|
||||
)
|
||||
return [proposal.id, files] as const
|
||||
}),
|
||||
),
|
||||
)
|
||||
const written = await writeProposals(out, proposals, rendered)
|
||||
expect(written.length).toBe(proposals.length)
|
||||
const droppedFile = await writeDropped(out, dropped)
|
||||
expect(await readFile(droppedFile, "utf8")).toContain("verbatim")
|
||||
await writeDropped(out, dropped)
|
||||
expect(await readFile(droppedFile, "utf8")).toContain("verbatim")
|
||||
// Gate-tuning explains only: gates.md, never the control file.
|
||||
const gate = proposals.find((proposal) => proposal.kind === "gate-tuning")!
|
||||
expect(gate.files).toEqual(["gates.md"])
|
||||
const files = await walkFiles(out)
|
||||
expect(files.some((file) => path.basename(file) === "gates.json")).toBe(false)
|
||||
} finally {
|
||||
await rm(out, { recursive: true, force: true })
|
||||
}
|
||||
} finally {
|
||||
await fix.cleanup()
|
||||
}
|
||||
})
|
||||
})
|
||||
@@ -9,6 +9,7 @@ import {
|
||||
livenessLines,
|
||||
progressPrint,
|
||||
readLiveness,
|
||||
researchLine,
|
||||
STALE_AFTER_MS,
|
||||
} from "../src/factory/liveness.ts"
|
||||
import { Factory } from "../src/factory/machine.ts"
|
||||
@@ -180,6 +181,31 @@ describe("headline", () => {
|
||||
const qa = state({ stage: "QA", activePhase: "p1" })
|
||||
expect(headline(qa, await live(qa, T0))).toStartWith("QA on p1 in progress")
|
||||
})
|
||||
|
||||
test("STALE_AFTER_MS is ten minutes; the stuck hint starts exactly at the boundary (#248)", async () => {
|
||||
expect(STALE_AFTER_MS).toBe(600_000)
|
||||
const current = state()
|
||||
const atBoundary = new Date(T0.getTime() + STALE_AFTER_MS)
|
||||
expect(headline(current, await live(current, atBoundary))).toContain("No activity for 10m in RESEARCH")
|
||||
const justUnder = new Date(T0.getTime() + STALE_AFTER_MS - 1_000)
|
||||
expect(headline(current, await live(current, justUnder))).toBe(
|
||||
"Research in progress: no seat is running; last activity 9m ago.",
|
||||
)
|
||||
})
|
||||
|
||||
test("done without a PR, SPEC and RELEASE wording (#248)", async () => {
|
||||
const done = state({ stage: "DONE" })
|
||||
expect(headline(done, await live(done, T0))).toBe("Done: the run is complete.")
|
||||
const spec = state({ stage: "SPEC", runId: "run-20261008-020000-spec1" })
|
||||
await updateSeats(fx.root, spec.runId, (seats) => {
|
||||
seats.push(seat({ lastActivityAt: T0.toISOString() }))
|
||||
})
|
||||
expect(headline(spec, await live(spec, T0))).toBe("Spec writing is running: es-research-alpha started 0s ago.")
|
||||
const release = state({ stage: "RELEASE", runId: "run-20261008-020000-rel1", updatedAt: T0.toISOString() })
|
||||
expect(headline(release, await live(release, T0))).toBe(
|
||||
"Release in progress: no seat is running; last activity 0s ago.",
|
||||
)
|
||||
})
|
||||
})
|
||||
|
||||
describe("formatHuman", () => {
|
||||
@@ -210,6 +236,24 @@ describe("formatHuman", () => {
|
||||
expect(text).toContain("es-research-alpha running es_research_fetch 10s ago ses_ee6b70e18ffe")
|
||||
expect(text).toContain("es-research-beta completed 50s ago ses_beta")
|
||||
expect(text).toContain("research: 0 sources · alpha.md — · beta.md — · REPORT.md — · coverage not checked")
|
||||
// The dashboard leads with the headline, then the run header (#248):
|
||||
// order, not just containment.
|
||||
expect(text.indexOf("Research is running: es-research-alpha")).toBeLessThan(
|
||||
text.indexOf(`run run-20261008-020000-ab12 · RESEARCH · ${fx.root}`),
|
||||
)
|
||||
})
|
||||
|
||||
test("non-RESEARCH stages print no research line (#248)", async () => {
|
||||
const current = state({ stage: "BUILD", activePhase: "p1", updatedAt: at(20).toISOString() })
|
||||
const live = await readLiveness(fx.root, current, { now: at(60) })
|
||||
expect(researchLine(live)).toBeUndefined()
|
||||
const text = formatHuman(
|
||||
current,
|
||||
live,
|
||||
"<factory-state>\nrun run-20261008-020000-ab12 · stage BUILD\nbuild phase p1\n</factory-state>",
|
||||
)
|
||||
expect(text).toContain("Building p1 in progress")
|
||||
expect(text).not.toContain("research:")
|
||||
})
|
||||
})
|
||||
|
||||
|
||||
@@ -0,0 +1,68 @@
|
||||
// Audit #197 integration tests (core side): the workspace TypeScript server
|
||||
// reaches diagnostics through the manager, and stopAll shuts the running
|
||||
// servers down. Same tsc fixture seam as lsp.test.ts; no binaries run directly.
|
||||
import { afterAll, beforeAll, describe, expect, test } from "bun:test"
|
||||
import { mkdir, mkdtemp, rm, symlink, writeFile } from "node:fs/promises"
|
||||
import { tmpdir } from "node:os"
|
||||
import path from "node:path"
|
||||
import { LspManager } from "../src/index.ts"
|
||||
|
||||
const TYPESCRIPT = path.resolve(import.meta.dir, "../../../node_modules/typescript")
|
||||
|
||||
let root: string
|
||||
let state: string
|
||||
let manager: LspManager
|
||||
const write = async (file: string, text: string) => {
|
||||
await mkdir(path.dirname(path.join(root, file)), { recursive: true })
|
||||
await writeFile(path.join(root, file), text)
|
||||
}
|
||||
|
||||
beforeAll(async () => {
|
||||
root = await mkdtemp(path.join(tmpdir(), "es-lsp-shutdown-"))
|
||||
state = await mkdtemp(path.join(tmpdir(), "es-lsp-shutdown-state-"))
|
||||
await write(
|
||||
"tsconfig.json",
|
||||
JSON.stringify({
|
||||
compilerOptions: { strict: true, target: "ES2022", module: "NodeNext", noEmit: true },
|
||||
include: ["src"],
|
||||
}),
|
||||
)
|
||||
await mkdir(path.join(root, "node_modules"), { recursive: true })
|
||||
await symlink(TYPESCRIPT, path.join(root, "node_modules/typescript"), "dir")
|
||||
await mkdir(path.join(root, "node_modules/.bin"), { recursive: true })
|
||||
await symlink(path.join(TYPESCRIPT, "bin/tsc"), path.join(root, "node_modules/.bin/tsc"))
|
||||
await write("src/math.ts", "export function add(a: number, b: number): number {\n return a + b\n}\n")
|
||||
await write(
|
||||
"src/main.ts",
|
||||
'import { add } from "./math.js"\n\nconst total: number = add(1, 2)\nconst wrong: string = total\nexport { wrong }\n',
|
||||
)
|
||||
manager = new LspManager({ root, stateDir: state, configFiles: [path.join(root, "opencode.json")] })
|
||||
}, 30_000)
|
||||
|
||||
afterAll(async () => {
|
||||
await manager?.stopAll()
|
||||
await rm(root, { recursive: true, force: true })
|
||||
await rm(state, { recursive: true, force: true })
|
||||
})
|
||||
|
||||
describe("LspManager shutdown (#197)", () => {
|
||||
test("diagnostics start the workspace server and stopAll shuts it down", async () => {
|
||||
const diagnostics = await manager.diagnostics("src/main.ts", { maxWaitMs: 20_000 })
|
||||
expect(Array.isArray(diagnostics)).toBe(true)
|
||||
const messages = (diagnostics as any[]).map((item) => `${item.range.start.line + 1}: ${item.message}`)
|
||||
expect(messages.join("\n")).toContain("4: Type 'number' is not assignable to type 'string'")
|
||||
const running = (await manager.status()).running
|
||||
expect(running.some((entry) => entry.id === "typescript" && entry.ready)).toBe(true)
|
||||
await manager.stopAll()
|
||||
expect((await manager.status()).running).toEqual([])
|
||||
}, 60_000)
|
||||
|
||||
test("stopAll is idempotent and the next diagnostics restarts the server lazily", async () => {
|
||||
await manager.stopAll()
|
||||
expect((await manager.status()).running).toEqual([])
|
||||
const diagnostics = await manager.diagnostics("src/main.ts", { maxWaitMs: 20_000 })
|
||||
expect(Array.isArray(diagnostics)).toBe(true)
|
||||
expect(JSON.stringify(diagnostics)).toContain("not assignable to type 'string'")
|
||||
expect((await manager.status()).running.some((entry) => entry.id === "typescript" && entry.ready)).toBe(true)
|
||||
}, 60_000)
|
||||
})
|
||||
@@ -1,7 +1,7 @@
|
||||
// LSP runtime against real language servers (native TypeScript 7 `tsc --lsp`,
|
||||
// gopls) plus config resolution and the pinned installer.
|
||||
import { afterAll, beforeAll, describe, expect, test } from "bun:test"
|
||||
import { chmod, mkdir, mkdtemp, rm, symlink, writeFile } from "node:fs/promises"
|
||||
import { chmod, mkdir, mkdtemp, readFile, rm, symlink, writeFile } from "node:fs/promises"
|
||||
import { tmpdir } from "node:os"
|
||||
import path from "node:path"
|
||||
import {
|
||||
@@ -138,4 +138,92 @@ describe("pinned installer", () => {
|
||||
expect(await consent("fake-ls", state)).toBe(true)
|
||||
await expect(installServer(server, { stateDir: state, npm: fakeNpm })).rejects.toThrow("integrity mismatch")
|
||||
})
|
||||
|
||||
test("the fetch/install mitigation stack is pinned: SRI, scripts-disabled, scrubbed env, confined cwd (#275.2)", async () => {
|
||||
// Installs are not bwrap-sandboxed (the fetch needs the network), so the
|
||||
// narrower stack — consent, exact pins, --ignore-scripts, scrubbed env,
|
||||
// scratch/target confinement — is pinned here instead of assumed.
|
||||
const genuine = new TextEncoder().encode("genuine-tarball-bytes")
|
||||
const fakeNpm = path.join(state, "npm-pin")
|
||||
await writeFile(
|
||||
fakeNpm,
|
||||
[
|
||||
"#!/bin/sh",
|
||||
'here=$(dirname "$0")',
|
||||
'if [ "$1" = pack ]; then',
|
||||
' printf "%s\\n" "$@" > "$here/pack-args.log"',
|
||||
' printf "PWD=%s\\n" "$PWD" >> "$here/pack-args.log"',
|
||||
' env | sort > "$here/pack.env"',
|
||||
" dest=$4",
|
||||
' printf "genuine-tarball-bytes" > "$dest/pkg-1.0.0.tgz"',
|
||||
' echo \'[{"filename":"pkg-1.0.0.tgz"}]\'',
|
||||
" exit 0",
|
||||
"fi",
|
||||
'if [ "$1" = install ]; then',
|
||||
' printf "%s\\n" "$@" > "$here/install-args.log"',
|
||||
' printf "PWD=%s\\n" "$PWD" >> "$here/install-args.log"',
|
||||
' env | sort > "$here/install.env"',
|
||||
' prefix=""',
|
||||
' prev=""',
|
||||
' for arg in "$@"; do',
|
||||
' if [ "$prev" = "--prefix" ]; then prefix=$arg; fi',
|
||||
" prev=$arg",
|
||||
" done",
|
||||
' mkdir -p "$prefix/node_modules/.bin"',
|
||||
' printf "#!/bin/sh\\nexit 0\\n" > "$prefix/node_modules/.bin/fake-pin"',
|
||||
' chmod +x "$prefix/node_modules/.bin/fake-pin"',
|
||||
" exit 0",
|
||||
"fi",
|
||||
"exit 1",
|
||||
"",
|
||||
].join("\n"),
|
||||
)
|
||||
await chmod(fakeNpm, 0o755)
|
||||
const server: ServerSpec = {
|
||||
id: "fake-pin",
|
||||
command: ["fake-pin"],
|
||||
extensions: [".fake"],
|
||||
rootMarkers: [],
|
||||
install: {
|
||||
kind: "npm",
|
||||
packages: [{ name: "pkg", version: "1.0.0", integrity: sri(genuine) }],
|
||||
bin: "fake-pin",
|
||||
},
|
||||
source: "builtin",
|
||||
}
|
||||
await recordConsent("fake-pin", true, state)
|
||||
// A canary the scrubbed env must not leak; a registry override must pass through.
|
||||
process.env.ES_LSP_PIN_EVIL = "leak-me"
|
||||
process.env.NPM_CONFIG_REGISTRY = "https://registry.test/"
|
||||
let bin: string
|
||||
try {
|
||||
bin = await installServer(server, { stateDir: state, npm: fakeNpm })
|
||||
} finally {
|
||||
delete process.env.ES_LSP_PIN_EVIL
|
||||
delete process.env.NPM_CONFIG_REGISTRY
|
||||
}
|
||||
expect(bin).toContain(path.join("lsp", "fake-pin-1.0.0"))
|
||||
const read = (name: string) => readFile(path.join(state, name), "utf8")
|
||||
|
||||
const packArgs = await read("pack-args.log")
|
||||
expect(packArgs).toContain("[email protected]")
|
||||
// pack runs confined to a scratch dir, not the project root.
|
||||
const packPwd = /PWD=(.*)/.exec(packArgs)![1]!
|
||||
expect(path.basename(packPwd)).toMatch(/^es-lsp-/)
|
||||
expect(packPwd).not.toBe(root)
|
||||
const packEnv = await read("pack.env")
|
||||
expect(packEnv).not.toContain("ES_LSP_PIN_EVIL")
|
||||
expect(packEnv).toContain("NPM_CONFIG_REGISTRY=https://registry.test/")
|
||||
|
||||
const installArgs = await read("install-args.log")
|
||||
for (const flag of ["--no-save", "--ignore-scripts", "--no-audit", "--no-fund"]) expect(installArgs).toContain(flag)
|
||||
expect(installArgs).toContain("--prefix")
|
||||
expect(installArgs).toContain("pkg-1.0.0.tgz")
|
||||
// install runs with cwd inside the state-dir target it writes.
|
||||
const installPwd = /PWD=(.*)/.exec(installArgs)![1]!
|
||||
expect(installPwd).toContain(path.join(state, "lsp", "fake-pin-1.0.0"))
|
||||
const installEnv = await read("install.env")
|
||||
expect(installEnv).not.toContain("ES_LSP_PIN_EVIL")
|
||||
expect(installEnv).toContain("NPM_CONFIG_REGISTRY=https://registry.test/")
|
||||
})
|
||||
})
|
||||
|
||||
@@ -0,0 +1,133 @@
|
||||
// Packed-tarball contents for the release set (issue #208): what users
|
||||
// install must carry the runtime (core assets, the CLI bin, the plugin
|
||||
// server plus the prebuilt TUI) and nothing else — no tests, no private
|
||||
// packages (`es-fleet`, web, testkit). Packs with `bun pm pack` the way
|
||||
// scripts/pack-smoke.sh packs, from the single package list
|
||||
// (scripts/packages.ts, as in packages.test.ts).
|
||||
import { afterAll, beforeAll, describe, expect, test } from "bun:test"
|
||||
import { mkdtemp, readdir, rm } from "node:fs/promises"
|
||||
import { tmpdir } from "node:os"
|
||||
import path from "node:path"
|
||||
import { releasePackages, tarballSlug, workspacePackages } from "../../../scripts/packages.ts"
|
||||
|
||||
const root = path.resolve(import.meta.dir, "../../..")
|
||||
|
||||
async function run(argv: string[], cwd: string): Promise<string> {
|
||||
const proc = Bun.spawn(argv, { cwd, stdout: "pipe", stderr: "pipe" })
|
||||
const [stdout, stderr, code] = await Promise.all([
|
||||
new Response(proc.stdout).text(),
|
||||
new Response(proc.stderr).text(),
|
||||
proc.exited,
|
||||
])
|
||||
if (code !== 0) throw new Error(`${argv.join(" ")} in ${cwd} exited ${code}: ${stderr.slice(-2000)}`)
|
||||
return stdout
|
||||
}
|
||||
|
||||
async function tarList(tgz: string): Promise<string[]> {
|
||||
const out = await run(["tar", "-tzf", tgz], root)
|
||||
return out.split("\n").filter(Boolean)
|
||||
}
|
||||
|
||||
async function tarRead(tgz: string, member: string): Promise<string> {
|
||||
return run(["tar", "-xzOf", tgz, member], root)
|
||||
}
|
||||
|
||||
const TEST_DIR = new Set(["test", "tests", "__tests__", "__snapshots__"])
|
||||
const isTestPath = (entry: string) => {
|
||||
const parts = entry.split("/")
|
||||
return parts.some((part) => TEST_DIR.has(part)) || /\.(test|spec)\.[a-z]+$/.test(parts.at(-1) ?? "")
|
||||
}
|
||||
|
||||
describe("packed release tarballs (issue #208)", () => {
|
||||
let out = ""
|
||||
const tarballs = new Map<string, string>()
|
||||
|
||||
beforeAll(async () => {
|
||||
out = await mkdtemp(path.join(tmpdir(), "es-pack-contents-"))
|
||||
const release = await releasePackages(root)
|
||||
for (const { dir } of release) await run(["bun", "run", "build"], path.join(root, "packages", dir))
|
||||
for (const { dir } of release)
|
||||
await run(["bun", "pm", "pack", "--destination", out], path.join(root, "packages", dir))
|
||||
const files = await readdir(out)
|
||||
for (const { dir, name } of release) {
|
||||
const hits = files.filter((file) => file.startsWith(`${tarballSlug(name)}-`) && file.endsWith(".tgz"))
|
||||
expect(hits).toHaveLength(1)
|
||||
tarballs.set(dir, path.join(out, hits[0]!))
|
||||
}
|
||||
}, 240_000)
|
||||
afterAll(() => rm(out, { recursive: true, force: true }))
|
||||
|
||||
test("the release set is exactly core, cli and the plugin; fleet, web and testkit stay private", async () => {
|
||||
expect([...tarballs.keys()].sort()).toEqual(["cli", "core", "opencode"])
|
||||
const all = await workspacePackages(root)
|
||||
for (const dir of ["fleet", "web", "testkit"]) {
|
||||
const pkg = all.find((entry) => entry.dir === dir)!
|
||||
expect([dir, pkg.private]).toEqual([dir, true])
|
||||
expect(pkg.order).toBeUndefined()
|
||||
}
|
||||
// No tarball for a private package, even a stale one from an earlier run.
|
||||
for (const pkg of all.filter((entry) => entry.private)) {
|
||||
const hits = (await readdir(out)).filter(
|
||||
(file) => file.startsWith(`${tarballSlug(pkg.name)}-`) && file.endsWith(".tgz"),
|
||||
)
|
||||
expect([pkg.dir, hits]).toEqual([pkg.dir, []])
|
||||
}
|
||||
})
|
||||
|
||||
test("packed manifests carry no workspace: protocol", async () => {
|
||||
for (const [dir, tgz] of tarballs) {
|
||||
const manifest = await tarRead(tgz, "package/package.json")
|
||||
expect([dir, manifest.includes("workspace:")]).toEqual([dir, false])
|
||||
}
|
||||
})
|
||||
|
||||
test("tarballs contain no tests and no private-package payload", async () => {
|
||||
for (const [dir, tgz] of tarballs) {
|
||||
const entries = await tarList(tgz)
|
||||
expect([dir, entries.filter(isTestPath)]).toEqual([dir, []])
|
||||
expect([dir, entries.filter((entry) => entry.includes("es-fleet") || entry.includes("es-web"))]).toEqual([
|
||||
dir,
|
||||
[],
|
||||
])
|
||||
}
|
||||
})
|
||||
|
||||
test("tarballs ship the runtime payload users need", async () => {
|
||||
const has = async (dir: string, member: string) =>
|
||||
expect([dir, member, (await tarList(tarballs.get(dir)!)).includes(member)]).toEqual([dir, member, true])
|
||||
// Core: compiled entry plus the bundled assets pack-smoke loads
|
||||
// (prompts, skills, licence templates).
|
||||
await has("core", "package/dist/index.js")
|
||||
await has("core", "package/assets/prompts/factory.md")
|
||||
await has("core", "package/assets/skills/factory/SKILL.md")
|
||||
await has("core", "package/assets/licenses/MIT.template.txt")
|
||||
// CLI: the bin users run plus its bundle.
|
||||
await has("cli", "package/bin/es.js")
|
||||
await has("cli", "package/dist/es.js")
|
||||
// Plugin: the server entry the host loads plus the prebuilt TUI (raw
|
||||
// .tsx cannot ship: the host Solid transform skips node_modules).
|
||||
await has("opencode", "package/server.ts")
|
||||
await has("opencode", "package/src/server.ts")
|
||||
await has("opencode", "package/dist/tui.js")
|
||||
})
|
||||
|
||||
test("the opencode host pin matches what the suites test", async () => {
|
||||
const manifest = async (dir: string) =>
|
||||
JSON.parse(await Bun.file(path.join(root, "packages", dir, "package.json")).text()) as {
|
||||
dependencies?: Record<string, string>
|
||||
devDependencies?: Record<string, string>
|
||||
}
|
||||
const plugin = await manifest("opencode")
|
||||
const kit = await manifest("testkit")
|
||||
// Pinned, not a range: the host the plugin runs against is exactly the
|
||||
// host the testkit boots.
|
||||
expect(plugin.dependencies?.["@opencode/plugin"]).toBe("2.0.24")
|
||||
expect(plugin.devDependencies?.["@opencode/sdk"]).toBe("2.0.24")
|
||||
expect(kit.dependencies?.["@opencode/sdk"]).toBe(plugin.devDependencies?.["@opencode/sdk"])
|
||||
// The packed plugin pins the same host dependency (workspace:* rewritten).
|
||||
const packed = JSON.parse(await tarRead(tarballs.get("opencode")!, "package/package.json")) as {
|
||||
dependencies?: Record<string, string>
|
||||
}
|
||||
expect(packed.dependencies?.["@opencode/plugin"]).toBe(plugin.dependencies?.["@opencode/plugin"])
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,92 @@
|
||||
// Packaging pin assertions (#212): the opencode triple-pin moves together,
|
||||
// every release manifest declares an explicit `files` allowlist with no test
|
||||
// paths, and the `--release` rendering keeps the `<dir> <slug>` contract
|
||||
// `publish.yml` parses. File reads only: no network, no registry, no pack.
|
||||
// Structural fixes (a peer range, narrower allowlists, publish parsing,
|
||||
// the runtime-matrix wording) route to S9 via CROSS-SQUAD REQUEST.
|
||||
import { describe, expect, test } from "bun:test"
|
||||
import { readFile } from "node:fs/promises"
|
||||
import path from "node:path"
|
||||
import { releasePackages, tarballSlug, workspacePackages } from "../../../scripts/packages.ts"
|
||||
|
||||
const root = path.resolve(import.meta.dir, "../../..")
|
||||
const manifest = async (dir: string) =>
|
||||
JSON.parse(await readFile(path.join(root, "packages", dir, "package.json"), "utf8")) as Record<string, any>
|
||||
|
||||
describe("the opencode triple-pin moves together", () => {
|
||||
test("@opencode/plugin, @opencode/sdk and the e2e CLI share one version", async () => {
|
||||
const opencode = await manifest("opencode")
|
||||
const plugin = opencode.dependencies?.["@opencode/plugin"] as string
|
||||
const sdk = opencode.devDependencies?.["@opencode/sdk"] as string
|
||||
expect(plugin).toMatch(/^\d+\.\d+\.\d+$/)
|
||||
expect(sdk).toBe(plugin)
|
||||
const e2e = await readFile(path.join(root, ".github/workflows/e2e-feedback.yml"), "utf8")
|
||||
const cliPins = [...e2e.matchAll(/@opencode\/cli-linux-x64@([0-9A-Za-z._-]+)/g)].map((match) => match[1])
|
||||
expect(cliPins.length).toBeGreaterThan(0)
|
||||
expect(new Set(cliPins).size).toBe(1)
|
||||
expect(cliPins[0]).toBe(plugin)
|
||||
})
|
||||
|
||||
test("no opencode peer range: no such npm package exists to peer on (#212a)", async () => {
|
||||
// Verified 2026-10-11: `npm view opencode` is a registry 404 — the host
|
||||
// ships as @opencode/*-scoped packages and platform binaries, so a peer
|
||||
// on a bare `opencode` id could never resolve. The triple-pin above is
|
||||
// the mechanism; this pins the decision against a future peer addition.
|
||||
const opencode = await manifest("opencode")
|
||||
for (const field of ["dependencies", "devDependencies", "peerDependencies"] as const)
|
||||
expect(Object.keys(opencode[field] ?? {})).not.toContain("opencode")
|
||||
})
|
||||
})
|
||||
|
||||
describe("release tarball allowlists", () => {
|
||||
test("every release manifest declares an explicit files allowlist with no test paths", async () => {
|
||||
for (const pkg of await releasePackages(root)) {
|
||||
const files = (await manifest(pkg.dir)).files as unknown
|
||||
expect(Array.isArray(files) && (files as unknown[]).length > 0).toBe(true)
|
||||
for (const entry of files as string[]) {
|
||||
expect(typeof entry).toBe("string")
|
||||
expect(entry).not.toMatch(/test|spec|__tests__/i)
|
||||
}
|
||||
}
|
||||
})
|
||||
|
||||
test("fleet and web stay private (never released)", async () => {
|
||||
const all = await workspacePackages(root)
|
||||
for (const dir of ["fleet", "web"]) {
|
||||
expect(all.find((pkg) => pkg.dir === dir)?.private).toBe(true)
|
||||
}
|
||||
})
|
||||
|
||||
test("no test files sit under a released package's src dir (#212c)", async () => {
|
||||
// The allowlists include whole `src` trees, so a stray src/*.test.ts
|
||||
// would ship silently; the manifest-string check above cannot see it.
|
||||
// (fleet and web are private and never packed; pack-smoke.sh 2026-10-11
|
||||
// additionally proved the tarballs ship no test files.)
|
||||
const { readdir } = await import("node:fs/promises")
|
||||
const strays: string[] = []
|
||||
for (const pkg of await releasePackages(root)) {
|
||||
const walk = async (rel: string): Promise<void> => {
|
||||
const entries = await readdir(path.join(root, "packages", pkg.dir, rel), { withFileTypes: true }).catch(
|
||||
() => [],
|
||||
)
|
||||
for (const entry of entries) {
|
||||
const child = `${rel}/${entry.name}`
|
||||
if (entry.isDirectory()) await walk(child)
|
||||
else if (/\.test\.[cm]?[jt]s$|\.spec\.[cm]?[jt]s$|__tests__/.test(child)) strays.push(`${pkg.dir}/${child}`)
|
||||
}
|
||||
}
|
||||
await walk("src")
|
||||
}
|
||||
expect(strays).toEqual([])
|
||||
})
|
||||
})
|
||||
|
||||
describe("the --release rendering contract", () => {
|
||||
test('every line is "<dir> <slug>" with the slug derived from the manifest name', async () => {
|
||||
for (const pkg of await releasePackages(root)) {
|
||||
const line = `${pkg.dir} ${tarballSlug(pkg.name)}`
|
||||
expect(line).toMatch(/^\S+ \S+$/)
|
||||
expect(tarballSlug(pkg.name)).not.toContain(" ")
|
||||
}
|
||||
})
|
||||
})
|
||||
@@ -64,8 +64,15 @@ describe("workspace packages", () => {
|
||||
test("the real repo releases core, cli and the plugin in dependency order", async () => {
|
||||
const root = path.resolve(import.meta.dir, "../../..")
|
||||
expect((await releasePackages(root)).map((pkg) => pkg.dir)).toEqual(["core", "cli", "opencode"])
|
||||
expect((await workspacePackages(root)).map((pkg) => pkg.dir).sort()).toEqual(
|
||||
expect.arrayContaining(["cli", "core", "opencode", "testkit"]),
|
||||
)
|
||||
const all = await workspacePackages(root)
|
||||
expect(all.map((pkg) => pkg.dir).sort()).toEqual(expect.arrayContaining(["cli", "core", "opencode", "testkit"]))
|
||||
// Manifest name re-scope or version skew must not ship green (#212):
|
||||
// pin the release manifests' names and lockstep version beside the dir.
|
||||
const byDir = new Map(all.map((pkg) => [pkg.dir, pkg]))
|
||||
expect(byDir.get("core")).toMatchObject({ name: "@heretek-ai/es-core" })
|
||||
expect(byDir.get("cli")).toMatchObject({ name: "@heretek-ai/es-cli" })
|
||||
expect(byDir.get("opencode")).toMatchObject({ name: "@heretek-ai/epistemic-swarm" })
|
||||
const versions = new Set((await releasePackages(root)).map((pkg) => pkg.version))
|
||||
expect(versions.size).toBe(1)
|
||||
})
|
||||
})
|
||||
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user