Heretek-AI/IUMBTEMS
1
0
Fork 0
mirror of https://github.com/Heretek-AI/IUMBTEMS.git synced 2026-10-12 02:03:00 +00:00
Code Issues Packages Projects Releases Wiki Activity

test(audit-201): union mcp-identity.test.ts with main at merge (Refs #233)

Browse Source 
Add/add conflict with the remediation branch: both sides created
packages/cli/test/mcp-identity.test.ts. Union keeps every test from
both suites except three: the two exact-duplicate list/stdio tests
(subsumed by stronger twins) and the audit side's "server ignores
legacy ES_AGENT" assertion, which pins pre-remediation behavior — the
remediation pin order (ES_MCP_AGENT, else ES_AGENT, else defaultAgent)
reads it, proven by the pin-order test kept above.

mcp-identity.test.ts 32/32, biome clean.

Refs #201, Refs #230, Refs #232
This commit is contained in:
John Smith
2026-10-10 22:37:48 -04:00
parent a4eebe5406 7bbd42d8dc
commit e6fa2e943e
182 changed files with 15154 additions and 623 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.github/workflows/publish.yml
+6
View File
@@ -63,6 +63,12 @@ jobs:
# NOTE: the ./ prefix is load-bearing. A bare packs/foo.tgz is
# parsed as a GitHub user/repo shorthand (git ls-remote failure);
# ./packs/foo.tgz is resolved as the local tarball.
# Fail closed: an empty tarball list (e.g. --release format drift
# through the awk above) must never exit 0 having published nothing.
if [ -z "${{ steps.packages.outputs.tarballs }}" ]; then
echo "publish: no tarballs listed by scripts/packages.ts --release" >&2
exit 1
fi
for tgz in ${{ steps.packages.outputs.tarballs }}; do
echo "::group::npm publish $tgz"
if [ "$DRY_RUN" = "true" ]; then
AGENTS.md
+1 -1
View File
@@ -59,7 +59,7 @@ spike as proof).
| :--- | :--- |
| Approvals are human-only; trust, waivers and resume stay terminal-only: passphrase confirmation (terminal echo-off, TUI masked dialog, or loopback browser input per ADR 0002) unlocks the sealed Ed25519 human key (signed, hash-bound) and every surface signs approvals in-process — the TUI with channel `tui`, the browser through the loopback-only fleet endpoints with channel `web`; the approve/trust/resume RPCs no longer exist; agents may request, never grant | `approval/`, `gates/waivers.ts` |
| Control files are deny-write for all agents: `.factory/{gates.json,config.json,frontier.json,waivers,approvals,runtime}`, the evidence (`.factory/research/{sources,coverage.json,dossier.json,brief.pcrb.json}`, `.factory/claims/`), `.git/config` and git hooks. The pinned ones (gates, config, frontier, approvals, waivers) are also hash-checked at every gate run; cached sources are engine-sealed (HMAC) and content-addressed | `trust/control.ts` |
| Per-seat path scopes: manager writes factory docs, programmer writes its worktree, QA and the auditor pair write nothing, research alpha/beta write only their own notes (`.factory/research/alpha.md` / `beta.md`) and `research/REPORT.md` has one writer per run mode — the factory in build runs, the research synthesizer in research runs (`PolicyContext.runMode`; an unreadable mode denies both), brainstorm/harvest/design/scout write only their `.factory/<x>/notes/`; research, scout and auditor seats are `web: "cached"` (host websearch/webfetch denied, web facts only through the cached `es_research_*` tools) and run `--unshare-net` sandboxed, while user agents keep host web tools | `trust/policy.ts`, `agents/registry.ts` |
| Per-seat path scopes: manager writes factory docs, programmer writes its worktree, QA and the auditor pair write nothing, research alpha/beta write only their own notes (`.factory/research/alpha.md` / `beta.md`) and `research/REPORT.md` has one writer scope per mode — factory-docs seats (factory and manager) in build runs, the research synthesizer in research runs (`PolicyContext.runMode`; an unreadable mode denies both), brainstorm/harvest/design/scout write only their `.factory/<x>/notes/`; research, scout and auditor seats are `web: "cached"` (host websearch/webfetch denied, web facts only through the cached `es_research_*` tools) and run `--unshare-net` sandboxed, while user agents keep host web tools | `trust/policy.ts`, `agents/registry.ts` |
| Factory seats launch seats in the foreground only: `subagent` with `background: true` from any seat is refused, and so is launching a seat that is still running; seat state and last activity are recorded in `.factory/runtime/seats.json` (advisory) | `opencode/src/policy.ts`, `opencode/src/seats.ts` |
| Code audits block: an opened phase audit holds `passPhase` (a missing one too with `audit.phase: required`, set via `es config set`), any other open or failed audit refuses the next stage transition; every finding is witnessed against the file on disk; only a human dismisses one | `factory/machine.ts`, `codeaudit/` |
| Scout verdicts are computed by core: `adopt` survives only for a verified, whitelisted, permissive license | `scout/verdict.ts` |
README.md
+6 -4
View File
@@ -37,8 +37,10 @@
scanning, OSV, budgets (diff size, file length, complexity, dependency
justification, test-with-behaviour), with structured `file:line:rule`
findings and signed, expiring waivers.
- **Evidence-first research.** Content-addressed source cache, a pure verbatim
quote verifier, and an epistemic auditor that downgrades ungrounded claims.
- **Evidence-first research.** Content-addressed source cache, a quote
verifier that is verbatim after documented normalisation (NFKC, typography
folds, markdown stripping, whitespace collapse), and an epistemic auditor
that flags, prunes and refuses ungrounded claims.
Web sources come through an ordered backend chain (Scraper-Swarm gateway,
Brave, Firecrawl, SearXNG, direct fetch) with per-backend cooldowns: one
provider's outage or rate limit fails over to the next, while a safety
@@ -171,11 +173,11 @@ bun install
bun run check # biome + tsc + bun test (all packages)
bun run docs:gen # regenerate schemas/ + docs/ after schema changes
bun run docs:check # CI drift check
scripts/pack-smoke.sh # build, pack and install the packages; run them under Node and Bun
scripts/pack-smoke.sh # build, pack and install the packages; run the CLI under Node and the plugin server under Bun
scripts/v2-head.sh # run the real-host suite against OpenCode v2 HEAD
```
Platform: Linux, Node ≥ 22 or Bun. No Python.
Platform: Linux (CI runs ubuntu-latest), CLI under Node 22, plugin server under Bun. No Python.
## Documentation
docs/CAPABILITIES.md
+3 -3
View File
@@ -18,15 +18,15 @@ and Antigravity adapters ship in 1.1–1.3). Generated: do not edit by hand.
| subagents | ENFORCED | `packages/opencode/test/scoping.test.ts` | Depth 1; hidden seats are unlisted but invocable by ID, and only from the seats allowed to spawn them. Factory seats are not offered Code Mode execute. |
| compaction | ENFORCED | `packages/opencode/test/plugin.test.ts` | Session compaction injects the compact factory-state block for factory seats. |
| websearch | ENFORCED | `packages/opencode/test/research.test.ts` | The configured provider registers as the host websearch; results are cached and citable. Research, scout and auditor seats are cached-only (es_research_search/fetch) under an offline (--unshare-net) sandbox; host web caching applies only in a project that already has .factory/, and the cache is engine-sealed (planted entries refused). |
| lsp | ENFORCED | `packages/opencode/test/lsp.test.ts` | The lsp config key is read and served by our own manager (v2 accepts but does not run it). |
| lsp | ENFORCED | `packages/opencode/test/lsp.test.ts` | The `lsp` key in v2's `opencode.json` — not an ES config key (ES layers strict-reject it) — is read and served by our own manager (v2 accepts but does not run it). |
| panels | ENFORCED | `packages/opencode/test/panels.test.ts` | Four session.panel dashboards (factory, LSP, hooks, brainstorm), rendered by OpenTUI and refreshed on server changes; Esc/q closes a panel, f toggles fullscreen. The factory dashboard leads with the run headline, seats and research progress; a prompt-footer indicator shows the stage and the running seat. |
| research | ENFORCED | `packages/opencode/test/research.test.ts` | Engine-sealed cache (#52), fragment-level quote verifier (every ellipsis fragment >= 12 chars; code excerpts one contiguous span of at most 60 lines), auditor, providers and the websearch bridge. Research-only runs complete through the deep-researcher (thesis/antithesis/synthesis at depth 1) at DONE. |
| brainstorm | ENFORCED | `packages/opencode/test/brainstorm.test.ts` | Lens fan-out, record/score/complete tools, dedupe, rubric, shortlist with a forced outlier. Runs are id-scoped (the human's /brainstorm is the default run); the grill and the factory run callable brainstorms at depth 1 and get the shortlist as JSON. |
| brainstorm | ENFORCED | `packages/opencode/test/brainstorm.test.ts` | Lens fan-out, record/score/complete tools, dedupe, rubric, shortlist with a forced outlier whenever the shortlist has ≥2 slots. Runs are id-scoped (the human's /brainstorm is the default run); the grill and the factory run callable brainstorms at depth 1 and get the shortlist as JSON. |
| harvest | ENFORCED | `packages/opencode/test/harvest.test.ts` | Fail-closed SPDX detection, provenance profiles, policy-enforced matrix, clean-room specs. Runs are id-scoped (the human's /harvest is the default run); es_harvest_target gives the grill, factory, scout and harvester deterministic verdicts, and prior-art search is shared with brainstorm callers. |
| design | ENFORCED | `packages/opencode/test/design.test.ts` | Queereye interview, DTCG tokens, contrast gate, generated guide and drift check; phase-02 specs/webref/csf/tui renders and the phase-03 ledger bundle with its cite-gate receipt. |
| claims | ENFORCED | `packages/core/test/research.test.ts` | Witnessed claims and dossiers: the evidence cache, dossiers and the claim ledger are tool-only; planted cache entries are refused (engine seal) and every VERIFIED quote and code span is re-checked against the bytes on disk. |
| audit | ENFORCED | `packages/opencode/test/fires.test.ts` | Code-audit pair: only auditor seats record verdicts; every finding is witnessed on disk (a hallucinated line is refused); quote fragments carry >= 12 chars each and a thesis pass needs a witnessed invariant; verdicts block (an opened phase audit holds the merge; audit.phase required also holds a missing one until it passes or a human dismisses it). |
| scout | ENFORCED | `packages/opencode/test/fires.test.ts` | OSS scout: cached-only web under an offline (--unshare-net) sandbox, fail-closed license verdicts computed by core (adopt only for verified permissive licenses), OSV advisories cited. |
| scout | ENFORCED | `packages/opencode/test/scout-host.test.ts` | OSS scout: cached-only web under an offline (--unshare-net) sandbox, fail-closed license verdicts computed by core (adopt only for verified permissive licenses), OSV advisories cited (canned-OSV content in core's scout suite). |
| research-fires | ENFORCED | `packages/opencode/test/fires-research.test.ts` | Research fires: the whole research loop on the real host with loopback-only servers — grounded quotes survive the audit, failover records provenance, blocked never fails over, planted sources are refused, retractions degrade to STALE, renders snapshot. |
## claude
docs/SCHEMAS.md
+2
View File
@@ -76,6 +76,8 @@ Do not edit by hand: run `bun run docs:gen` after changing a schema (CI runs `bu
| run-phase-telemetry | `schemas/run-phase-telemetry.schema.json` |
| run-state-seal | `schemas/run-state-seal.schema.json` |
| run-telemetry | `schemas/run-telemetry.schema.json` |
| scout-advisories-file | `schemas/scout-advisories-file.schema.json` |
| scout-advisory-query | `schemas/scout-advisory-query.schema.json` |
| scout-advisory | `schemas/scout-advisory.schema.json` |
| scout-assessment | `schemas/scout-assessment.schema.json` |
| scout-plan | `schemas/scout-plan.schema.json` |
docs/adr/0002-approvals-on-every-surface.md
+4 -3
View File
@@ -119,7 +119,8 @@ prompt-injected agent acting through a seat or user-agent shell.
(TTL ≤ 120 s) binds the preview to `hashJson(subject)`, and signing
re-derives the subject and refuses a mismatch.
- **I4** Attempts are limited: at most N wrong passphrases per M minutes per
surface (recommended 5 per 10 min), then a lockout and an audit entry.
state dir (recommended 5 per 10 min; the CLI and the TUI share one bucket),
then a lockout and an audit entry.
- **I5** The channel is recorded (`cli`, `tui` or `web`) and audited.
- **I6** Mutating approval endpoints are unreachable from agent contexts: no
agent tool and no MCP tool; the TUI path requires the TUI process's dialog;
@@ -201,8 +202,8 @@ ticket before merge. All probes apply to both surfaces unless marked.
- The TUI plugin links core's approval service directly and signs in-process;
no new RPC, no passphrase on the wire anywhere except loopback for the
browser path.
- Attempt-limit state lives per surface (a small file in the state dir for
the TUI/CLI; server memory plus audit for the web path).
- Attempt-limit state lives per state dir (one shared attempt file for
the CLI and the TUI; server memory plus audit for the web path, which keeps its own dir).
- Docs and messages stop saying approvals are terminal-only, while trust,
waive, resume, key seal, config set and export keep saying it.
evals/cases/darkharvest-e2e.json
+1
View File
@@ -0,0 +1 @@
{"agent":"harvester","checks":{"toolsUsed":["es_harvest_plan","es_harvest_scan","es_harvest_matrix","es_harvest_complete"]},"description":"The harvester tears down two vendored local projects end to end (plan, scan, read, matrix, complete): MIT stays depend, GPL-3.0-only is downgraded to clean-room. Graded from .factory/harvest/runs/default/harvest.json. Provenance-per-field and no-copied-spans need the scrubber/manual review (see issue #192).","files":{"vendor/copycalc/LICENSE":"@fixture:licenses/GPL-3.0-only.txt","vendor/copycalc/index.js":"const { copycalcQuux7429 } = require('./lib.js')\nmodule.exports = { run: (xs) => xs.map(copycalcQuux7429) }\n","vendor/copycalc/lib.js":"function copycalcQuux7429(x) { return x < 0 ? 0 - x : x * 2 }\nmodule.exports = { copycalcQuux7429 }\n","vendor/copycalc/package.json":"{\"name\": \"copycalc\", \"version\": \"1.0.0\"}\n","vendor/retcalc/LICENSE":"@fixture:licenses/MIT.txt","vendor/retcalc/index.js":"const { retcalcQuux7429 } = require('./lib.js')\nmodule.exports = { run: (xs) => xs.map(retcalcQuux7429) }\n","vendor/retcalc/lib.js":"function retcalcQuux7429(x) { return x < 0 ? 0 - x : x + 1 }\nmodule.exports = { retcalcQuux7429 }\n","vendor/retcalc/package.json":"{\"name\": \"retcalc\", \"version\": \"1.0.0\"}\n"},"fire":{"expected":[{"candidate":"copycalc","verdict":"clean-room"},{"candidate":"retcalc","verdict":"depend"}],"kind":"harvest"},"id":"darkharvest-e2e","maxSteps":14,"prompt":"Tear down two local candidates for a session handoff feature: local:./vendor/retcalc and local:./vendor/copycalc. This is a headless run: es_harvest_plan them (objective 'session handoff teardown'), es_harvest_scan each, es_harvest_read one source file per candidate, submit one matrix row proposing depend for both with a one-line behaviour evidence per cell (describe behaviour in your own words, do not paste source), then es_harvest_complete."}
evals/cases/evidence-187.json
+8
View File
@@ -0,0 +1,8 @@
{
"description": "Research fetches a served fixture page and quotes its key sentence verbatim (no false VERIFIED).",
"agent": "es-research-alpha",
"prompt": "Fetch {{SERVE_URL}}/evidence-187.txt with es_research_fetch and report the key sentence about mechanical gates, quoted verbatim.",
"maxSteps": 6,
"checks": { "toolsUsed": ["es_research_fetch"], "contains": ["Mechanical gates catch regressions before review"] },
"id": "evidence-187"
}
evals/cases/hooks-bridge.json
+1
View File
@@ -0,0 +1 @@
{"agent":"build","checks":{"contains":["HOOK:PreToolUse","HOOK:PostToolUse","HOOK:PostToolUseFailure","HOOK:ShellPreExec"],"toolsUsed":["read","bash"]},"description":"Hook bridge E2E: real opencode + .claude/settings.json from a real Claude Code project; each ENFORCED hook event fires a marker. Markers asserted via transcript; on-disk completeness needs kept-workspace review until a file-asserting fire kind exists.","files":{".claude/settings.json":"{\"hooks\":{\"PermissionRequest\":[{\"hooks\":[{\"command\":\"printf 'HOOK:PermissionRequest\\\\n' >> .factory/hooks.log\",\"type\":\"command\"}],\"matcher\":\"Bash\"}],\"PostToolUse\":[{\"hooks\":[{\"command\":\"printf 'HOOK:PostToolUse\\\\n' >> .factory/hooks.log\",\"type\":\"command\"}],\"matcher\":\"\"}],\"PostToolUseFailure\":[{\"hooks\":[{\"command\":\"printf 'HOOK:PostToolUseFailure\\\\n' >> .factory/hooks.log\",\"type\":\"command\"}],\"matcher\":\"\"}],\"PreToolUse\":[{\"hooks\":[{\"command\":\"exit 2\",\"type\":\"command\"}],\"matcher\":\"Bash(rm *)\"},{\"hooks\":[{\"command\":\"printf 'HOOK:PreToolUse\\\\n' >> .factory/hooks.log\",\"type\":\"command\"}],\"matcher\":\"\"}],\"SessionStart\":[{\"hooks\":[{\"command\":\"printf 'HOOK:SessionStart\\\\n' >> .factory/hooks.log\",\"type\":\"command\"}],\"matcher\":\"startup\"}],\"UserPromptSubmit\":[{\"hooks\":[{\"command\":\"printf 'HOOK:UserPromptSubmit\\\\n' >> .factory/hooks.log\",\"type\":\"command\"}],\"matcher\":\"\"}],\"x-es.ShellPreExec\":[{\"hooks\":[{\"command\":\"echo rewritten-by-hook\",\"type\":\"command\"}],\"matcher\":\"echo original\"}]}}","src/fail.sh":"exit 1\n","src/probe.txt":"hook bridge probe\n"},"id":"hooks-bridge","maxSteps":14,"prompt":"Follow these steps exactly, one tool call per step. (0) Run `mkdir -p .factory` with bash. (1) Read src/probe.txt with the read tool. (2) Run `echo hook-probe-ok` with bash. (3) Run `sh src/fail.sh` with bash (it exits 1; continue regardless). (4) Run `echo original` with bash (it will be rewritten; continue). (5) Run `cat .factory/hooks.log` with bash. (6) Narrate which HOOK: markers you observed. This run is headless: never call the question tool."}
evals/cases/lsp-diagnostics.json
+1
View File
@@ -0,0 +1 @@
{"agent":"build","checks":{"contains":["not assignable","src/bad.ts"],"toolsUsed":["lsp_diagnostics"]},"description":"The agent introduces a type error in src/bad.ts; the typescript language server reports it via lsp_diagnostics. Requires typescript-language-server on PATH (workflow prerequisite, pinned to the registry).","files":{"src/bad.ts":"export const x: number = 1;\n","tsconfig.json":"{\n \"compilerOptions\": { \"strict\": true, \"target\": \"es2022\", \"module\": \"nodenext\" },\n \"include\": [\"src\"]\n}\n"},"id":"lsp-diagnostics","maxSteps":8,"prompt":"In src/bad.ts, change the initializer of x to the string \"not a number\" (keep the number type annotation), using the edit tool. Then call lsp_diagnostics on src/bad.ts and reply quoting the full diagnostic line for the error. This run is headless: never call the question tool. Make independent tool calls in the same block."}
evals/cases/registry-hidden-spawn.json
+8
View File
@@ -0,0 +1,8 @@
{
"id": "registry-hidden-spawn",
"description": "The factory delegates a status check to the hidden es-qa-functional seat by id (hidden seats are unlisted from the subagent menu but invocable from an allowed parent), then replies DELEGATED.",
"agent": "factory",
"prompt": "Delegate a status check to the es-qa-functional seat with the subagent tool (description \"status check\", prompt \"Check the factory run status with es_status and report back in one sentence\"). When it returns, reply with exactly the single word DELEGATED and nothing else.",
"maxSteps": 8,
"checks": { "toolsUsed": ["subagent"], "contains": ["DELEGATED"] }
}
evals/serve/evidence-187.txt
+3
View File
@@ -0,0 +1,3 @@
The evidence cache seals every source with an engine key. Mechanical gates
catch regressions before review, which keeps autonomous agents honest.
Quote the gate sentence verbatim; a paraphrase is not evidence.
packages/cli/src/headless.ts
+19 -14
View File
@@ -520,25 +520,30 @@ export function runHeadless(options: HeadlessOptions): AsyncGenerator<HeadlessEv
agent: "factory",
prompt: async (state, factory) =>
`${await factory.summary(state)}\n${headline(state, await readLiveness(options.root, state))}\nContinue the factory run from its current stage. This is a headless run: no human will answer questions.`,
finished: async (state) => {
if (state.stage === "DONE")
return { type: "done", ...(state.release?.prUrl ? { prUrl: state.release.prUrl } : {}) }
const pending = await pendingApprovals(options.root)
if (pending.length)
return {
type: "waiting",
reason: "a human must approve (es approve <stage>)",
stages: pending.map((item) => item.stage),
}
if (state.stage === "GRILL")
return { type: "waiting", reason: "the grill needs a human; run /grill interactively" }
return undefined
},
finished: (state) => headlessFinished(options.root, state),
// Research artifacts and seat outcomes count as progress, not just the stage machine.
progress: async (state) => progressPrint(state, await readLiveness(options.root, state)),
})
}
/**
* One headless end condition (#247.1): DONE carries the release PR url when
* the run released one, a pending approval waits on a human, and the grill
* needs a human. Anything else keeps driving.
*/
export async function headlessFinished(root: string, state: FactoryState): Promise<HeadlessEvent | undefined> {
if (state.stage === "DONE") return { type: "done", ...(state.release?.prUrl ? { prUrl: state.release.prUrl } : {}) }
const pending = await pendingApprovals(root)
if (pending.length)
return {
type: "waiting",
reason: "a human must approve (es approve <stage>)",
stages: pending.map((item) => item.stage),
}
if (state.stage === "GRILL") return { type: "waiting", reason: "the grill needs a human; run /grill interactively" }
return undefined
}
export const LOG_LEVELS = ["quiet", "info", "debug"] as const
export type LogLevel = (typeof LOG_LEVELS)[number]
packages/cli/src/mcp.ts
+27 -9
View File
@@ -2,11 +2,14 @@
// harness-neutral factory operations to harnesses without a native plugin and
// to headless runs. It has no approve, trust, waive or resume tool: those are
// human-only. Caller identity is PINNED, never piped: the harness adapter sets
// one agent id per server in its environment (ES_MCP_AGENT, else the legacy
// ES_AGENT/defaultAgent, both written by the human in the adapter config), and
// the per-call `agent` argument is ignored. Without an adapter identity the
// caller is "mcp", which maps to no seat, so every seat-checked tool refuses.
import { stateDir as defaultStateDir, esTools, Factory, gateRunner } from "@heretek-ai/es-core"
// one agent id per server in its environment (`ES_MCP_AGENT`, else the legacy
// `ES_AGENT`, else the `defaultAgent` option — all written by the human in the
// adapter config), and the per-call `agent` argument is ignored. Without an
// adapter identity the caller is "mcp", which maps to no seat, so every
// seat-checked tool refuses. Only a registry agent id (`factory`,
// `es-manager`, `es-programmer`, …) holds a seat: any other pinned string is
// still the caller, but a seat-less one, refused the same way as "mcp".
import { checkStopFile, stateDir as defaultStateDir, esTools, Factory, gateRunner } from "@heretek-ai/es-core"
import { VERSION } from "./version.ts"
export const PROTOCOL_VERSION = "2025-06-18"
@@ -26,7 +29,9 @@ export function createMcpServer(options: McpOptions) {
const env = options.env ?? process.env
// Pinned at construction: the adapter's environment, never the caller's
// arguments. An agent piping JSON-RPC into this server cannot claim a seat.
const caller = env.ES_MCP_AGENT || options.defaultAgent || "mcp"
// Library consumers get the same order as the CLI wrapper: the adapter's
// ES_MCP_AGENT wins, then the legacy ES_AGENT, then the defaultAgent option.
const caller = env.ES_MCP_AGENT || env.ES_AGENT || options.defaultAgent || "mcp"
const factory = new Factory(options.root, {
gates: gateRunner(options.stateDir ? { stateDir: options.stateDir } : {}),
...(options.stateDir ? { stateDir: options.stateDir } : {}),
@@ -35,7 +40,7 @@ export function createMcpServer(options: McpOptions) {
const byName = new Map(tools.map((tool) => [tool.name.replace(/^es_/, ""), tool]))
const listed = [...byName].map(([name, tool]) => ({
name,
description: `${tool.description}${name === "status" || name === "spec_validate" ? "" : " Caller identity is pinned by the adapter (ES_MCP_AGENT); a per-call `agent` argument is accepted but ignored."}`,
description: `${tool.description}${name === "status" || name === "spec_validate" ? "" : " Caller identity is pinned by the adapter (ES_MCP_AGENT, else ES_AGENT, else the default agent); a per-call `agent` argument is accepted but ignored."}`,
inputSchema: {
...tool.input,
properties: {
@@ -43,7 +48,7 @@ export function createMcpServer(options: McpOptions) {
agent: {
type: "string",
description:
"Ignored: the caller is the adapter's pinned ES_MCP_AGENT (or the default agent), never this argument.",
"Ignored: the caller is the adapter's pinned ES_MCP_AGENT (else ES_AGENT, else the default agent), never this argument.",
},
},
},
@@ -61,7 +66,7 @@ export function createMcpServer(options: McpOptions) {
capabilities: { tools: { listChanged: false } },
serverInfo: { name: "epistemic-swarm", version: options.version ?? VERSION },
instructions:
"Epistemic Swarm factory operations. Approvals, trust, waivers and resume are human-only (TUI or `es` CLI) and have no MCP tool. Caller identity is pinned: the adapter sets ES_MCP_AGENT in this server's environment (one server per agent); without it the caller is `mcp`, which holds no seat, so seat-checked tools refuse. A per-call `agent` argument is ignored.",
"Epistemic Swarm factory operations. Approvals, trust, waivers and resume are human-only (TUI or `es` CLI) and have no MCP tool. Caller identity is pinned: the adapter sets ES_MCP_AGENT in this server's environment (one server per agent), else the legacy ES_AGENT, else the default agent; without any of them the caller is `mcp`, which holds no seat, so seat-checked tools refuse. A per-call `agent` argument is ignored.",
})
case "ping":
return reply({})
@@ -72,6 +77,19 @@ export function createMcpServer(options: McpOptions) {
if (!tool) return fail(-32602, `Unknown tool: ${params?.name}`)
const { agent: _ignored, ...input } = (params?.arguments ?? {}) as Json
try {
// STOP and HALTED hold on the MCP surface too (#227): the
// kill-switch refuses every call; a halted run refuses everything
// but status, mirroring the OpenCode host policy.
const stop = await checkStopFile(options.root)
if (stop.stopped)
throw new Error(
`The factory is stopped (.factory/STOP: ${stop.reason}). A human must remove it and resume.`,
)
if (tool.name !== "es_status") {
const state = await factory.read().catch(() => undefined)
if (state?.stage === "HALTED")
throw new Error(`The factory is halted: ${state.halt?.reason}. Stop and report; a human must resume.`)
}
const text = await tool.execute(input, { agent: caller })
return reply({ content: [{ type: "text", text }] })
} catch (error) {
packages/cli/test/cli.test.ts
+162 -2
View File
@@ -1,15 +1,17 @@
import { afterEach, beforeEach, describe, expect, test } from "bun:test"
import { mkdir, mkdtemp, readFile, rm, writeFile } from "node:fs/promises"
import { mkdir, mkdtemp, readdir, readFile, rm, stat, writeFile } from "node:fs/promises"
import { tmpdir } from "node:os"
import path from "node:path"
import {
exists,
Factory,
type FactoryState,
factoryLayout,
gateRunner,
HUMAN_VERBS,
loadPrompt,
parseArgs,
type RunIndexEntry,
readApproval,
recentAuditEntries,
researchSourcesDir,
@@ -22,6 +24,7 @@ import { type HarnessDriver, type HeadlessEvent, presentEvent, runHeadless } fro
import { openProposalsPr } from "../src/improve.ts"
import { commandHelp, main } from "../src/main.ts"
import { createMcpServer } from "../src/mcp.ts"
import { otherRunHint, runsCommand, statusCommand } from "../src/status.ts"
import { type ConfirmIO, confirmHuman, NotInteractive } from "../src/tty.ts"
import { VERSION } from "../src/version.ts"
@@ -330,6 +333,147 @@ describe("human-only confirmation", () => {
})
})
describe("per-verb pipe refusal (#221.2)", () => {
// approve, rebaseline and reseal already have dedicated non-TTY pipe
// tests above. Every other verb in the confirmHuman funnel refuses the
// same way: exit 3 comes only from the NotInteractive catch in main, so
// asserting it proves the test reached the terminal gate — and each test
// also proves nothing was written. Each argv below is shaped to pass
// every check before the confirmation (usage → 2 and early-returns → 0/1
// would fail the assertion instead of passing vacuously).
test("es trust refuses without a terminal and trusts nothing", async () => {
await mkdir(path.dirname(factoryLayout(root).gates), { recursive: true })
await writeFile(
factoryLayout(root).gates,
JSON.stringify({ version: 1, commands: [{ id: "probe", kind: "test", command: "touch PROBE" }] }),
)
expect((await run(["trust"])).code).toBe(3)
expect(await exists(path.join(state, "trust.json"))).toBe(false)
})
test("es waive refuses without a terminal and records nothing", async () => {
const result = await run(["waive", "lint/*", "--reason", "legacy code under migration"])
expect(result.code).toBe(3)
expect(await exists(factoryLayout(root).waivers)).toBe(false)
})
test("es factory resume refuses without a terminal and stays halted", async () => {
const halted = new Factory(root, { gates: gateRunner({ stateDir: state }), stateDir: state })
await halted.begin("human:tester")
await halted.halt("system", "test halt")
expect((await run(["factory", "resume"])).code).toBe(3)
expect((await halted.read())?.stage).toBe("HALTED")
})
test("es factory pr refuses without a terminal and records nothing", async () => {
const result = await run(["factory", "pr", "https://github.com/example/repo/pull/1"])
expect(result.code).toBe(3)
expect(await exists(factoryLayout(root).config)).toBe(false)
})
test("es gates install-git refuses without a terminal and installs nothing", async () => {
const result = await run(["gates", "install-git"])
expect(result.code).toBe(3)
expect(await exists(path.join(root, ".factory", "git-hooks"))).toBe(false)
})
test("es lsp install refuses without a terminal before consent or install", async () => {
const result = await run(["lsp", "install", "typescript"])
expect(result.code).toBe(3)
expect(await exists(path.join(state, "lsp-consent.json"))).toBe(false)
})
test("es audit open refuses without a terminal and provisions nothing", async () => {
const result = await run(["audit", "src", "--max-usd", "5", "--open-only"])
expect(result.code).toBe(3)
expect(await exists(factoryLayout(root).dir)).toBe(false)
})
})
describe("status and runs are read-only; other-run hints (#248)", () => {
// File list + sizes + content hashes under .factory and the state dir.
// Directory mtimes are excluded on purpose: the state lock legitimately
// mkdirs+removes its lock dir on reads, churning the parent mtime.
const snapshot = async () => {
const entries: string[] = []
for (const [label, dir] of [
["root", path.join(root, ".factory")],
["state", state],
] as const) {
const walk = async (rel: string): Promise<void> => {
const names = await readdir(path.join(dir, rel)).catch(() => [] as string[])
for (const name of names) {
const child = rel ? `${rel}/${name}` : name
const info = await stat(path.join(dir, child))
if (info.isDirectory()) await walk(child)
else if (info.isFile())
entries.push(`${label}/${child}:${info.size}:${Bun.hash(await readFile(path.join(dir, child)))}`)
}
}
await walk("")
}
return entries.sort()
}
test("status and runs (text, --json, --all) write nothing", async () => {
const factory = new Factory(root, { gates: gateRunner({ stateDir: state }), stateDir: state })
const begun = await factory.begin("human:tester")
const printed: string[] = []
const context = { root, stateDir: state, print: (text: string) => printed.push(text) }
const before = await snapshot()
expect(await statusCommand(context, { positionals: ["status"], flags: {} })).toBe(0)
expect(await statusCommand(context, { positionals: ["status"], flags: { json: true } })).toBe(0)
expect(await runsCommand(context, { positionals: ["runs"], flags: {} })).toBe(0)
expect(await runsCommand(context, { positionals: ["runs"], flags: { all: true, json: true } })).toBe(0)
// The commands really ran (not an early return): the GRILL headline and
// the begun run are in the output.
expect(printed.join("\n")).toContain("grill")
expect(printed.join("\n")).toContain(begun.runId)
expect(await snapshot()).toEqual(before)
})
test("otherRunHint: window boundary, DONE and equal-timestamp edges", async () => {
const NOW = new Date("2026-10-08T02:10:00.000Z")
const iso = (ms: number) => new Date(NOW.getTime() + ms).toISOString()
const entry = (overrides: Partial<RunIndexEntry> = {}): RunIndexEntry => ({
runId: "run-other",
root: "/elsewhere",
stage: "BUILD",
createdAt: iso(-3_600_000),
updatedAt: iso(-60_000),
...overrides,
})
const here = (updatedAt: string): FactoryState => ({
version: 2,
runId: "run-here",
stage: "BUILD",
createdAt: iso(-3_600_000),
updatedAt,
spend: { usd: 1, estimated: true, events: 1 },
phases: [],
audits: [],
})
const hint = (runs: RunIndexEntry[], updatedAt: string | undefined) =>
otherRunHint(runs, { root, state: updatedAt === undefined ? undefined : here(updatedAt) }, NOW)
// A more recently active run elsewhere is named; without a local run the
// copy points at it directly.
expect(hint([entry()], iso(-120_000))).toContain("run-other")
expect(hint([entry()], undefined)).toContain("A run is active elsewhere")
// The window is ten minutes: just inside counts, exactly at it and
// outside do not (OTHER_RUN_WINDOW_MS is pinned behaviorally here).
expect(hint([entry({ updatedAt: iso(-599_000) })], iso(-3_600_000))).toContain("run-other")
expect(hint([entry({ updatedAt: iso(-600_000) })], iso(-3_600_000))).toBeUndefined()
expect(hint([entry({ updatedAt: iso(-601_000) })], iso(-3_600_000))).toBeUndefined()
// DONE runs never hint, however fresh; an equally-fresh run is not
// "more recently active"; our own root is skipped; a stale local run is.
expect(hint([entry({ stage: "DONE" })], iso(-120_000))).toBeUndefined()
expect(hint([entry({ updatedAt: iso(-120_000) })], iso(-120_000))).toBeUndefined()
expect(hint([entry({ root })], iso(-120_000))).toBeUndefined()
expect(hint([entry()], iso(-30_000))).toBeUndefined()
})
})
describe("es key", () => {
test("seal asks twice, refuses short or mismatched passphrases, and status shows the fingerprint", async () => {
const { humanKeyId } = await import("@heretek-ai/es-core")
@@ -355,6 +499,9 @@ describe("es key", () => {
expect((await runIn(["key", "seal"], seq(["short", "short"]))).code).toBe(1)
expect((await runIn(["key", "seal"], seq(["long-enough-pass", "different-pass"]))).code).toBe(1)
expect(await humanKeyId(fresh)).toBeUndefined()
// A pipe cannot seal either: keySeal throws NotInteractive (exit 3) before reading.
expect((await runIn(["key", "seal"], pipe)).code).toBe(3)
expect(await humanKeyId(fresh)).toBeUndefined()
expect((await runIn(["key", "seal"], seq(["a-long-test-passphrase", "a-long-test-passphrase"]))).code).toBe(0)
expect(await humanKeyId(fresh)).toBeDefined()
expect((await runIn(["key", "status"], seq([]))).out).toContain("Human key")
@@ -1351,7 +1498,20 @@ describe("es audit and es scout (headless jobs)", () => {
)
await tool("es_scout_scan").execute({ candidate: "tiny" }, scout)
await tool("es_scout_record").execute(
{ candidate: "tiny", maintenance: "slow", proposal: "adopt", rationale: "small enough to keep", claims: [] },
{
candidate: "tiny",
maintenance: "slow",
proposal: "adopt",
rationale: "small enough to keep",
claims: [
{
tag: "NEGATIVE_KNOWLEDGE",
statement: "OSV advisories were not queried for tiny",
query: "OSV advisories for tiny",
finding: "skipped: a local fixture with no published package",
},
],
},
scout,
)
await tool("es_scout_complete").execute({}, scout)
packages/cli/test/config-203.test.ts
+69
View File
@@ -0,0 +1,69 @@
// Audit #203 — `es config set` is human-only: without a terminal it refuses
// before writing. These are pipe runs asserting refusal codes through the
// CLI's own seam (main with a non-interactive ConfirmIO); the human key is
// never unlocked and nothing is ever written.
import { afterEach, beforeEach, describe, expect, test } from "bun:test"
import { mkdtemp, rm } from "node:fs/promises"
import { tmpdir } from "node:os"
import path from "node:path"
import { exists, factoryLayout } from "@heretek-ai/es-core"
import { main } from "../src/main.ts"
import type { ConfirmIO } from "../src/tty.ts"
let root: string
let state: string
const realXdgConfig = process.env.XDG_CONFIG_HOME
beforeEach(async () => {
root = await mkdtemp(path.join(tmpdir(), "es-203-config-"))
state = await mkdtemp(path.join(tmpdir(), "es-203-config-state-"))
process.env.XDG_CONFIG_HOME = path.join(state, "xdg-config")
})
afterEach(async () => {
if (realXdgConfig === undefined) delete process.env.XDG_CONFIG_HOME
else process.env.XDG_CONFIG_HOME = realXdgConfig
await rm(root, { recursive: true, force: true })
await rm(state, { recursive: true, force: true })
})
const pipe: ConfirmIO = { interactive: false, write: () => {}, readLine: async () => "y", readSecret: async () => "" }
async function run(argv: string[]) {
const printed: string[] = []
const code = await main(argv, { print: (text) => printed.push(text), confirm: pipe, cwd: root, stateDir: state })
return { code, out: printed.join("\n") }
}
describe("es config set refuses without a terminal (#203)", () => {
test("a piped project-layer set refuses with code 3 and writes nothing", async () => {
const result = await run(["config", "set", "research.depth", "3"])
expect(result.code).toBe(3)
expect(result.out).toContain("interactive terminal")
expect(await exists(factoryLayout(root).config)).toBe(false)
})
test("a piped global-layer set refuses with code 3 and writes nothing", async () => {
const result = await run(["config", "set", "research.depth", "3", "--global"])
expect(result.code).toBe(3)
expect(result.out).toContain("interactive terminal")
expect(await exists(factoryLayout(root).config)).toBe(false)
expect(await exists(path.join(state, "xdg-config", "epistemic-swarm", "config.json"))).toBe(false)
})
test("refusal codes contrast: schema refusal (1) precedes the TTY gate (3); usage is (2)", async () => {
const typo = await run(["config", "set", "reserch.depth", "3"])
expect(typo.code).toBe(1)
expect(typo.out).toContain("Refused")
const bad = await run(["config", "set", "research.depth", "9"])
expect(bad.code).toBe(1)
const usage = await run(["config", "set", "research.depth"])
expect(usage.code).toBe(2)
expect(await exists(factoryLayout(root).config)).toBe(false)
})
test("--help stays a read: usage, code 0, nothing written", async () => {
const helped = await run(["config", "set", "research.depth", "3", "--help"])
expect(helped.code).toBe(0)
expect(await exists(factoryLayout(root).config)).toBe(false)
})
})
packages/cli/test/headless-200.test.ts
+467
View File
@@ -0,0 +1,467 @@
// Headless factory runs (#200 §2): integration tests against the real
// in-process OpenCode v2 host (packages/testkit), plus the cheap proof gaps
// from bug #247. Spend-bearing seams are never touched: no `factory run
// --headless` launch, no real signals, no real harness binary. Everything
// here drives `runHeadless`/`driveHeadless` with fake in-process drivers
// (one of them forwarding prompts to the testkit host), scripted
// AbortControllers, pipe CLI runs that assert refusal codes, and a stub
// `opencode` binary for argv capture.
import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, test } from "bun:test"
import { chmod, mkdir, mkdtemp, readFile, rm, writeFile } from "node:fs/promises"
import { tmpdir } from "node:os"
import path from "node:path"
import {
exists,
Factory,
factoryLayout,
gateRunner,
readApproval,
researchSourcesDir,
SourceCache,
sealHumanKey,
stringifyFrontmatter,
} from "@heretek-ai/es-core"
import type { Harness } from "../../testkit/src/host.ts"
import { boot } from "../../testkit/src/host.ts"
import { directiveScript } from "../../testkit/src/script.ts"
import {
DRIVERS,
type DriverTurn,
driveHeadless,
type HarnessDriver,
type HeadlessEvent,
headlessExitCode,
opencodeDriver,
runHeadless,
} from "../src/headless.ts"
import { main } from "../src/main.ts"
import type { ConfirmIO } from "../src/tty.ts"
const PASSPHRASE = "test-passphrase-1234"
let root: string
let state: string
const realXdgConfig = process.env.XDG_CONFIG_HOME
const git = async (...args: string[]) => {
const proc = Bun.spawn(["git", ...args], { cwd: root, stdout: "pipe", stderr: "pipe" })
await proc.exited
}
beforeEach(async () => {
root = await mkdtemp(path.join(tmpdir(), "es-200-"))
state = await mkdtemp(path.join(tmpdir(), "es-200-state-"))
process.env.XDG_CONFIG_HOME = path.join(state, "xdg-config")
await git("init", "-q", "-b", "main")
await writeFile(path.join(root, "README.md"), "# x\n")
await git("add", "-A")
await git("-c", "user.name=t", "-c", "user.email=t@t", "commit", "-q", "-m", "init")
await sealHumanKey(PASSPHRASE, state)
})
afterEach(async () => {
if (realXdgConfig === undefined) delete process.env.XDG_CONFIG_HOME
else process.env.XDG_CONFIG_HOME = realXdgConfig
await rm(root, { recursive: true, force: true })
await rm(state, { recursive: true, force: true })
})
/** A human at a terminal who types the passphrase with echo off. */
const human = (answer: string = PASSPHRASE): ConfirmIO & { output: string[] } => {
const output: string[] = []
return {
interactive: true,
output,
write: (text) => output.push(text),
readLine: async () => "",
readSecret: async () => answer,
}
}
const pipe: ConfirmIO = { interactive: false, write: () => {}, readLine: async () => "y", readSecret: async () => "" }
async function runCli(argv: string[], confirm: ConfirmIO = pipe) {
const printed: string[] = []
const code = await main(argv, { print: (text) => printed.push(text), confirm, cwd: root, stateDir: state })
return { code, out: printed.join("\n") }
}
/** A settled grill, approved at a terminal: the run is in RESEARCH. */
async function researched() {
const factory = new Factory(root, { gates: gateRunner({ stateDir: state }), stateDir: state })
await factory.writeFrontier("grill", {
version: "1.1",
idea: "cli test",
spendCeiling: { currency: "USD", maxAmount: 5 },
settled: true,
nodes: [{ id: "a", question: "q?", answer: "yes", status: "settled" }],
})
const approved = await runCli(["approve", "frontier"], human())
expect(approved.code).toBe(0)
expect((await factory.read())?.stage).toBe("RESEARCH")
return factory
}
describe("headless against the testkit host (#200 §2)", () => {
let host: Harness | undefined
beforeAll(async () => {
host = await boot({ script: directiveScript })
}, 60_000)
afterAll(async () => {
await host?.close()
host = undefined
})
test("turns reach the real host; a pending approval stops the run without granting anything", async () => {
const factory = await researched()
const harness = host!
const llmBefore = harness.llm.requests.length
const calls: DriverTurn[] = []
// The opencode driver seam: one turn carries the seat identity, the
// prompt and the argv the harness CLI would get; here the harness is
// the in-process testkit host instead of a spawned `opencode run`.
const driver: HarnessDriver = {
id: "fake-host",
available: async () => true,
async *turn(input: DriverTurn) {
calls.push(input)
const result = await harness.run(input.prompt)
yield { type: "text", part: { text: `host settled ${result.context.length} messages` } }
return result.sessionID
},
}
// Phase 1: no pending approval, so the turn goes through the host.
const first: HeadlessEvent[] = []
for await (const event of runHeadless({ root, driver, maxTurns: 1, stallTurns: 99, stateDir: state }))
first.push(event)
expect(calls).toHaveLength(1)
expect(calls[0]?.agent).toBe("factory")
expect(calls[0]?.prompt).toContain("headless run")
expect(calls[0]?.root).toBe(root)
expect(harness.llm.requests.length).toBeGreaterThan(llmBefore)
expect(first.at(-1)).toEqual({ type: "turn-cap", turns: 1 })
// Phase 2: a pending approval stops the run before any turn is driven.
const pending = factoryLayout(root).pending
await mkdir(path.dirname(pending), { recursive: true })
await writeFile(pending, JSON.stringify([{ stage: "spec", requestedBy: "factory", at: "2026-10-10T00:00:00Z" }]))
const pendingBefore = await readFile(pending, "utf8")
const stageBefore = (await factory.read())?.stage
const callsBefore = calls.length
const llmMid = harness.llm.requests.length
const second: HeadlessEvent[] = []
for await (const event of runHeadless({ root, driver, maxTurns: 5, stallTurns: 99, stateDir: state }))
second.push(event)
expect(second.at(-1)).toEqual({
type: "waiting",
reason: "a human must approve (es approve <stage>)",
stages: ["spec"],
})
// Nothing was granted: no new host turn, the pending file is byte
// identical, the stage is unchanged, and no approval was recorded.
expect(calls.length).toBe(callsBefore)
expect(harness.llm.requests.length).toBe(llmMid)
expect(await readFile(pending, "utf8")).toBe(pendingBefore)
expect((await factory.read())?.stage).toBe(stageBefore)
expect(await readApproval(root, "spec")).toBeUndefined()
}, 60_000)
})
describe("headless proof gaps (#247)", () => {
test("the driver sees the seat identity, the prompt, the model and the threaded session", async () => {
await researched()
const calls: DriverTurn[] = []
let n = 0
const driver: HarnessDriver = {
id: "fake",
available: async () => true,
async *turn(input: DriverTurn) {
calls.push(input)
n++
yield { text: "working" }
return n === 1 ? "ses_1" : "ses_2"
},
}
const events: HeadlessEvent[] = []
for await (const event of runHeadless({
root,
driver,
maxTurns: 2,
stallTurns: 99,
stateDir: state,
model: "fake/scripted",
}))
events.push(event)
expect(calls).toHaveLength(2)
for (const call of calls) {
expect(call.agent).toBe("factory")
expect(call.prompt).toContain("headless run")
expect(call.root).toBe(root)
expect(call.model).toBe("fake/scripted")
}
expect(calls[0]?.session).toBeUndefined()
expect(calls[0]?.signal).toBeUndefined()
// The first turn's session id is threaded into the second turn.
expect(calls[1]?.session).toBe("ses_1")
expect(events.at(-1)).toEqual({ type: "turn-cap", turns: 2 })
})
test("the opencode driver argv shape: run --standalone --format json --agent [--model] [--session] prompt", async () => {
expect(opencodeDriver.id).toBe("opencode")
expect(DRIVERS.opencode).toBe(opencodeDriver)
const bin = await mkdtemp(path.join(tmpdir(), "es-argv-opencode-"))
const capture = path.join(bin, "argv.jsonl")
await writeFile(
path.join(bin, "opencode"),
`#!/usr/bin/env bun\nimport { appendFileSync } from "node:fs"\nappendFileSync(process.env.ES_ARGV_CAPTURE!, \`\${JSON.stringify(Bun.argv.slice(2))}\\n\`)\nconsole.log(JSON.stringify({ type: "text", sessionID: "ses_argv1", part: { text: "stub" } }))\n`,
)
await chmod(path.join(bin, "opencode"), 0o755)
const previousPath = process.env.PATH
const previousCapture = process.env.ES_ARGV_CAPTURE
process.env.PATH = `${bin}${path.delimiter}${previousPath}`
process.env.ES_ARGV_CAPTURE = capture
try {
const turnArgv = async (input: DriverTurn) => {
await rm(capture, { force: true })
const seen: unknown[] = []
const turn = opencodeDriver.turn(input)
let step = await turn.next()
while (!step.done) {
seen.push(step.value)
step = await turn.next()
}
const lines = (await readFile(capture, "utf8")).trim().split("\n")
return { argv: JSON.parse(lines[0]!) as string[], events: seen, session: step.value }
}
const minimal = await turnArgv({ root, agent: "factory", prompt: "go" })
expect(minimal.argv).toEqual(["run", "--standalone", "--format", "json", "--agent", "factory", "go"])
expect(minimal.session).toBe("ses_argv1")
expect(minimal.events).toHaveLength(1)
const full = await turnArgv({
root,
agent: "factory",
prompt: "go",
model: "fake/scripted",
session: "ses_prev",
})
expect(full.argv).toEqual([
"run",
"--standalone",
"--format",
"json",
"--agent",
"factory",
"--model",
"fake/scripted",
"--session",
"ses_prev",
"go",
])
// The prompt is always the trailing argument, after any seat flags.
expect(full.argv.at(-1)).toBe("go")
} finally {
process.env.PATH = previousPath
if (previousCapture === undefined) delete process.env.ES_ARGV_CAPTURE
else process.env.ES_ARGV_CAPTURE = previousCapture
await rm(bin, { recursive: true, force: true })
}
// With no harness binary on PATH the driver reports itself unavailable.
const empty = await mkdtemp(path.join(tmpdir(), "es-argv-empty-"))
try {
process.env.PATH = empty
expect(await opencodeDriver.available()).toBe(false)
} finally {
process.env.PATH = previousPath
await rm(empty, { recursive: true, force: true })
}
})
test("a scripted driver drives runHeadless to DONE and the prUrl spreads into done", async () => {
const green = { passed: true, findings: [], summary: "green" }
const stubGates = async () => green
const factory = new Factory(root, { gates: stubGates, stateDir: state })
await factory.writeFrontier("grill", {
version: "1.1",
idea: "cli test",
spendCeiling: { currency: "USD", maxAmount: 5 },
settled: true,
nodes: [{ id: "a", question: "q?", answer: "yes", status: "settled" }],
})
expect((await runCli(["approve", "frontier"], human())).code).toBe(0)
// A research report whose single claim cites a cached source verbatim.
const cache = new SourceCache(researchSourcesDir(root), state)
const source = await cache.put({
url: "https://example.test/greeting",
text: "Greeting libraries usually expose a single greet function.",
provider: "fetch",
})
await mkdir(path.join(root, ".factory/research"), { recursive: true })
await writeFile(
path.join(root, ".factory/research/REPORT.md"),
`# Research\n\n- A greet function is the conventional API [VERIFIED: sha256:${source.meta.sha256} "expose a single greet function"]\n`,
)
await factory.completeResearch("factory")
await mkdir(path.join(root, ".factory/specs/alpha"), { recursive: true })
await writeFile(
path.join(root, ".factory/roadmap.json"),
JSON.stringify({ version: 1, title: "Greeting", phases: [{ id: "alpha", title: "Phase alpha" }] }),
)
await writeFile(
path.join(root, ".factory/specs/alpha/GOAL.md"),
stringifyFrontmatter(
{
phase: "alpha",
title: "Phase alpha",
acceptance: [{ kind: "file", id: "impl", description: "implementation exists", path: "src/alpha.ts" }],
},
"Implement alpha as a vertical slice.\n",
),
)
expect((await runCli(["approve", "spec"], human())).code).toBe(0)
await factory.startBuild("factory")
const dir = (await factory.activeWorktree())!
await mkdir(path.join(dir, "src"), { recursive: true })
await mkdir(path.join(dir, "test"), { recursive: true })
await writeFile(path.join(dir, "test/alpha.test.ts"), `test("alpha", () => {})\n`)
await factory.recordGateRun("es-programmer", {
passed: false,
findings: [],
summary: "red",
failedTests: ["test/alpha.test.ts"],
})
await writeFile(path.join(dir, "src/alpha.ts"), "export const alpha = 1\n")
expect((await factory.complete("es-programmer")).ok).toBe(true)
await factory.qaVerdict("es-qa-functional", "pass", "ok")
expect((await factory.qaVerdict("es-qa-adversarial", "pass", "ok")).stage).toBe("RELEASE")
// The scripted turn records the PR (the human fallback): the next loop
// sees DONE with a release, and done carries the prUrl.
const prUrl = "https://example.test/pr/9"
const calls: DriverTurn[] = []
const driver: HarnessDriver = {
id: "fake",
available: async () => true,
async *turn(input: DriverTurn) {
calls.push(input)
yield { text: "recording the pr" }
await new Factory(root, { gates: stubGates, stateDir: state }).recordPr("tester", prUrl)
return "ses_done"
},
}
const events: HeadlessEvent[] = []
for await (const event of runHeadless({ root, driver, maxTurns: 5, stallTurns: 99, stateDir: state }))
events.push(event)
expect(events[0]).toMatchObject({ type: "start" })
expect(calls).toHaveLength(1)
expect(calls[0]?.agent).toBe("factory")
expect(events.at(-1)).toEqual({ type: "done", prUrl })
expect((await factory.read())?.stage).toBe("DONE")
})
test("a STOP file halts at headless level: no turn runs, the stage is untouched", async () => {
const factory = await researched()
await writeFile(factoryLayout(root).stop, "lunch\n")
let turns = 0
const driver: HarnessDriver = {
id: "fake",
available: async () => true,
async *turn() {
turns++
yield { text: "never" }
return "ses_fake"
},
}
const events: HeadlessEvent[] = []
for await (const event of runHeadless({ root, driver, maxTurns: 3, stallTurns: 99, stateDir: state }))
events.push(event)
expect(events.at(-1)).toMatchObject({ type: "halted", reason: expect.stringContaining("lunch") })
expect(turns).toBe(0)
expect((await factory.read())?.stage).toBe("RESEARCH")
expect(await exists(factoryLayout(root).stop)).toBe(true)
})
test("an unknown --driver is refused before anything runs", async () => {
const empty = await mkdtemp(path.join(tmpdir(), "es-200-empty-"))
try {
const printed: string[] = []
const code = await main(["factory", "run", "--headless", "--driver", "nope"], {
print: (text) => printed.push(text),
confirm: pipe,
cwd: empty,
stateDir: state,
})
const out = printed.join("\n")
expect(code).toBe(2)
expect(out).toContain("Unknown driver")
expect(out).toContain("opencode")
expect(await exists(factoryLayout(empty).dir)).toBe(false)
} finally {
await rm(empty, { recursive: true, force: true })
}
})
test("exit codes: cancelled is 130, failures stay 1, the rest 0", async () => {
expect(headlessExitCode("cancelled")).toBe(130)
for (const type of ["error", "halted", "stalled", "turn-cap"] as const) expect(headlessExitCode(type)).toBe(1)
for (const type of ["start", "turn", "driver", "progress", "turn-metrics", "waiting", "done"] as const)
expect(headlessExitCode(type)).toBe(0)
})
test("a pre-aborted signal cancels at Abort level: no error, no halt, resumable", async () => {
const factory = await researched()
const hanging: HarnessDriver = {
id: "hang",
available: async () => true,
async *turn() {
await new Promise<never>(() => {})
yield { text: "never" }
return undefined
},
}
const controller = new AbortController()
controller.abort()
const events: HeadlessEvent[] = []
for await (const event of runHeadless({
root,
driver: hanging,
maxTurns: 5,
stateDir: state,
signal: controller.signal,
progressMs: 60_000,
}))
events.push(event)
// No real signal was sent: the mapping (SIGINT/SIGTERM -> abort ->
// `cancelled`, exit 130) is proven at the Abort level, and the live
// signal E2E stays human-only (issue #200 §3, NEEDS HUMAN).
expect(events.at(-1)).toMatchObject({
type: "cancelled",
reason: expect.stringContaining("run state is untouched"),
})
expect(headlessExitCode("cancelled")).toBe(130)
expect(events.some((event) => event.type === "error")).toBe(false)
expect((await factory.read())?.stage).toBe("RESEARCH")
expect(await exists(factoryLayout(root).stop)).toBe(false)
})
test("driveHeadless threads a custom job's done (with prUrl) through unchanged", async () => {
const factory = new Factory(root, { gates: gateRunner({ stateDir: state }), stateDir: state })
await factory.provisionRun("tester", 5)
const prUrl = "https://example.test/pr/7"
const driver: HarnessDriver = {
id: "fake",
available: async () => true,
async *turn() {
yield { text: "working" }
return "ses_fake"
},
}
const events: HeadlessEvent[] = []
for await (const event of driveHeadless(
{ root, driver, maxTurns: 3, stallTurns: 99, stateDir: state },
{
agent: "factory",
prompt: async () => "go",
finished: async () => ({ type: "done", prUrl }),
progress: async () => "p",
},
))
events.push(event)
expect(events.at(-1)).toEqual({ type: "done", prUrl })
})
})
packages/cli/test/headless.test.ts
+192 -2
View File
@@ -1,16 +1,19 @@
// Headless fleet probes (#120): per-turn tool metrics carry duration and seat,
// and stay out of the default human output.
import { afterEach, beforeEach, describe, expect, test } from "bun:test"
import { mkdtemp, rm } from "node:fs/promises"
import { mkdir, mkdtemp, rm, writeFile } from "node:fs/promises"
import { tmpdir } from "node:os"
import path from "node:path"
import { Factory, gateRunner } from "@heretek-ai/es-core"
import { Factory, type FactoryState, factoryLayout, gateRunner } from "@heretek-ai/es-core"
import {
driveHeadless,
type HarnessDriver,
type HeadlessEvent,
HeadlessJsonlSchema,
headlessFinished,
opencodeDriver,
presentEvent,
runHeadless,
toJsonl,
} from "../src/headless.ts"
@@ -117,3 +120,190 @@ describe("turn-metrics human output (#120)", () => {
expect(presentEvent(metrics, "debug")).toEqual(metrics)
})
})
describe("runHeadless end conditions (#247)", () => {
test("a research run drives to done, and done carries no prUrl without a release", async () => {
const dir = await mkdtemp(path.join(tmpdir(), "es-headless-done-"))
const dirState = await mkdtemp(path.join(tmpdir(), "es-headless-done-state-"))
try {
const factory = new Factory(dir, { gates: gateRunner({ stateDir: dirState }), stateDir: dirState })
await factory.beginResearchRun({ objective: "what queue", ceilingUSD: 5 })
const seen: Array<{ agent: string; prompt: string }> = []
const driver: HarnessDriver = {
id: "fake",
available: async () => true,
async *turn(input) {
seen.push({ agent: input.agent, prompt: input.prompt })
yield { text: "working" }
// The coordinator flow, played against the real tools and gate.
const { researchTools } = await import("@heretek-ai/es-core")
const stubFetch = (async () =>
new Response("<p>Queues decouple workers from producers durably.</p>", {
headers: { "content-type": "text/html" },
})) as unknown as typeof fetch
const tools = researchTools({
root: dir,
fetch: stubFetch,
stateDir: dirState,
policy: async () => ({ root: dir }),
})
const fetched = await tools
.find((tool) => tool.name === "es_research_fetch")!
.execute({ url: "https://192.0.2.1/queues" }, { agent: "es-research-alpha" })
const sha = /sha256:([0-9a-f]{64})/.exec(fetched)?.[1]
await writeFile(
path.join(dir, ".factory/research/REPORT.md"),
`# Report\n\n- Queues decouple workers [VERIFIED: sha256:${sha} "decouple workers from producers"]\n`,
)
await new Factory(dir, { gates: gateRunner({ stateDir: dirState }), stateDir: dirState }).completeResearch(
"deep-researcher",
)
return "ses_fake"
},
}
const events: HeadlessEvent[] = []
for await (const event of runHeadless({ root: dir, driver, maxTurns: 5, stallTurns: 99, stateDir: dirState }))
events.push(event)
// The driver saw the factory seat identity and the headless prompt (#247.2).
expect(seen).toHaveLength(1)
expect(seen[0]?.agent).toBe("factory")
expect(seen[0]?.prompt).toContain("headless run")
expect(events[0]).toMatchObject({ type: "start", driver: "fake" })
expect(events.filter((event) => event.type === "turn")).toHaveLength(1)
const done = events.at(-1)!
expect(done).toEqual({ type: "done" })
expect("prUrl" in done).toBe(false)
} finally {
await rm(dir, { recursive: true, force: true })
await rm(dirState, { recursive: true, force: true })
}
})
test("headlessFinished maps DONE (with prUrl spread), pending approvals and the grill", async () => {
const dir = await mkdtemp(path.join(tmpdir(), "es-headless-finished-"))
try {
const done = (release?: FactoryState["release"]) =>
({ stage: "DONE", ...(release ? { release } : {}) }) as unknown as FactoryState
// No release: no prUrl key. With one: the url spreads onto done (#247.1).
expect(await headlessFinished(dir, done())).toEqual({ type: "done" })
expect(await headlessFinished(dir, done({ prUrl: "https://example.test/pr/9" }))).toEqual({
type: "done",
prUrl: "https://example.test/pr/9",
})
// A pending approval waits on a human, with its stages.
await mkdir(path.dirname(factoryLayout(dir).pending), { recursive: true })
await writeFile(factoryLayout(dir).pending, JSON.stringify([{ stage: "spec", requestedBy: "factory", at: "x" }]))
expect(await headlessFinished(dir, { stage: "RESEARCH" } as unknown as FactoryState)).toEqual({
type: "waiting",
reason: "a human must approve (es approve <stage>)",
stages: ["spec"],
})
await writeFile(factoryLayout(dir).pending, "[]")
// The grill (no pending) waits on a human; a mid-build stage keeps driving.
expect(await headlessFinished(dir, { stage: "GRILL" } as unknown as FactoryState)).toMatchObject({
type: "waiting",
reason: expect.stringContaining("grill"),
})
expect(await headlessFinished(dir, { stage: "RESEARCH" } as unknown as FactoryState)).toBeUndefined()
} finally {
await rm(dir, { recursive: true, force: true })
}
})
test("a STOP file halts at headless level without granting anything", async () => {
const dir = await mkdtemp(path.join(tmpdir(), "es-headless-stop-"))
const dirState = await mkdtemp(path.join(tmpdir(), "es-headless-stop-state-"))
try {
await writeFile(path.join(dir, "README.md"), "# x\n")
await new Factory(dir, { gates: gateRunner({ stateDir: dirState }), stateDir: dirState }).provisionRun(
"tester",
5,
)
await mkdir(path.dirname(factoryLayout(dir).stop), { recursive: true })
await writeFile(factoryLayout(dir).stop, "lunch\n")
const driver: HarnessDriver = {
id: "fake",
available: async () => true,
async *turn() {
yield { text: "unused" }
return "ses_fake"
},
}
const halted: HeadlessEvent[] = []
for await (const event of runHeadless({ root: dir, driver, maxTurns: 3, stateDir: dirState })) halted.push(event)
expect(halted[0]).toMatchObject({ type: "start" })
expect(halted.at(-1)).toMatchObject({ type: "halted", reason: expect.stringContaining("STOP file") })
expect(halted.some((event) => event.type === "turn")).toBe(false)
} finally {
await rm(dir, { recursive: true, force: true })
await rm(dirState, { recursive: true, force: true })
}
})
test("an unknown --driver is refused before anything launches", async () => {
const { main } = await import("../src/main.ts")
const printed: string[] = []
const code = await main(["factory", "run", "--headless", "--driver", "nope"], {
print: (text) => printed.push(text),
confirm: { interactive: false, write: () => {}, readLine: async () => "y", readSecret: async () => "" },
cwd: root,
stateDir: state,
})
expect(code).toBe(2)
expect(printed.join("\n")).toContain("Unknown driver")
})
})
describe("opencode driver argv shape (#247.2)", () => {
test("run --standalone --format json --agent, with model/session/prompt passthrough", async () => {
const { chmod } = await import("node:fs/promises")
const bin = await mkdtemp(path.join(tmpdir(), "es-driver-argv-"))
const previous = process.env.PATH
try {
await writeFile(
path.join(bin, "opencode"),
`#!/bin/sh\nprintf '%s\\n' "$@" > "$ES_ARGV_LOG"\necho '{"type":"text","sessionID":"ses_argv","part":{"text":"hi"}}'\n`,
)
await chmod(path.join(bin, "opencode"), 0o755)
process.env.PATH = `${bin}${path.delimiter}${previous}`
const argvOf = async (input: Parameters<typeof opencodeDriver.turn>[0]): Promise<string[]> => {
const log = path.join(bin, `argv-${Math.random().toString(36).slice(2)}.txt`)
process.env.ES_ARGV_LOG = log
try {
const turn = opencodeDriver.turn(input)
let step = await turn.next()
while (!step.done) step = await turn.next()
expect(step.value).toBe("ses_argv")
return (await Bun.file(log).text()).split("\n").filter(Boolean)
} finally {
delete process.env.ES_ARGV_LOG
}
}
expect(await argvOf({ root, agent: "factory", prompt: "go" })).toEqual([
"run",
"--standalone",
"--format",
"json",
"--agent",
"factory",
"go",
])
expect(await argvOf({ root, agent: "factory", prompt: "again", model: "prov/model", session: "ses_1" })).toEqual([
"run",
"--standalone",
"--format",
"json",
"--agent",
"factory",
"--model",
"prov/model",
"--session",
"ses_1",
"again",
])
} finally {
process.env.PATH = previous
await rm(bin, { recursive: true, force: true })
}
})
})
packages/cli/test/improve-204.test.ts
+282
View File
@@ -0,0 +1,282 @@
// #204 integration through the CLI: harvest + distill over several
// unit-constructed run dirs (one with a broken audit chain); scrubbed
// output and the code-disposes gate dropping fabricated evidence with
// reasons.
//
// Unit-constructed only: no real factory runs, no timing/perf assertions,
// no --open-pr (human TTY; existing pipe-refusal coverage stands).
import { describe, expect, test } from "bun:test"
import { appendFile, mkdir, mkdtemp, readdir, readFile, rm, stat, writeFile } from "node:fs/promises"
import { tmpdir } from "node:os"
import path from "node:path"
import {
appendAuditEntry,
disposedWithReasons,
disposeReason,
distillProposals,
factoryLayout,
TelemetrySchema,
undisposed,
} from "@heretek-ai/es-core"
import { main } from "../src/main.ts"
import type { ConfirmIO } from "../src/tty.ts"
const pipe: ConfirmIO = { interactive: false, write: () => {}, readLine: async () => "y", readSecret: async () => "" }
const GH_A = "ghp_cliHarvest001AAA"
const GH_BROKEN = "ghu_cliBroken999aaa"
const SK_ANT = "sk-ant-cliplanted0000Z"
const AKIA = "AKIAIOSFODNN7EXAMPLE"
const BEARER = "eyJhbGciOiJIUzI1NiJ9"
const GATE_RULE = "lint/no-unused-vars"
async function writeGateSummary(root: string, id: string): Promise<void> {
const dir = path.join(factoryLayout(root).runs, id, "gates", "2026-10-04T00-30-full")
await mkdir(dir, { recursive: true })
await writeFile(
path.join(dir, "summary.json"),
`${JSON.stringify(
{
passed: false,
findings: [
{ file: "src/x.ts", line: 3, rule: GATE_RULE, severity: "error", message: "unused", check: "lint" },
{ file: "src/y.ts", line: 9, rule: GATE_RULE, severity: "error", message: "unused", check: "lint" },
],
},
null,
2,
)}\n`,
)
}
async function writeAuditRecords(root: string): Promise<void> {
const dir = factoryLayout(root).audits
await mkdir(path.join(dir, "audit-01"), { recursive: true })
const finding = (title: string, detail: string) => ({
kind: "invariant",
title,
severity: "high",
file: "packages/core/src/trust/policy.ts",
lines: [59, 75],
excerpt: "export function evaluateWrite(context, agentId, target)",
detail,
holds: false,
})
await writeFile(
path.join(dir, "audit-01", "records.json"),
`${JSON.stringify(
[
{
audit: "audit-01",
round: 1,
seat: "auditor-thesis",
by: "auditor",
verdict: "fail",
notes: "notes",
findings: [
finding("control files are deny-write", "the programmer seat can write a control file path"),
finding("control files are deny-write again", "a second invariant detail here"),
],
at: "2026-10-04T00:00:00.000Z",
},
],
null,
2,
)}\n`,
)
}
async function writeChain(root: string, haltReason: string): Promise<void> {
await appendAuditEntry(root, { actor: "system", action: "stage.build" })
await appendAuditEntry(root, {
actor: "agent:qa",
action: "qa.fail",
payload: { phase: "p1", reason: "lint gate red" },
})
await appendAuditEntry(root, { actor: "system", action: "stage.halted", payload: { reason: haltReason } })
}
async function makeRun(parent: string, id: string, haltReason: string, broken: boolean): Promise<string> {
const root = path.join(parent, id)
await writeGateSummary(root, id)
await writeAuditRecords(root)
await writeChain(root, haltReason)
if (broken) await appendFile(factoryLayout(root).audit, "not-json\n")
return root
}
async function writeEvals(dir: string, failures: { id: string; failures: string[] }[]): Promise<void> {
await mkdir(dir, { recursive: true })
await writeFile(
path.join(dir, "agg.json"),
`${JSON.stringify(
{
at: "2026-10-04T00-00-00-000Z",
model: "test/model",
results: failures.map((entry, index) => ({
id: entry.id,
pass: entry.failures.length === 0,
failures: entry.failures,
steps: 4 + index,
})),
},
null,
2,
)}\n`,
)
}
async function walkFiles(dir: string): Promise<string[]> {
const out: string[] = []
const visit = async (current: string): Promise<void> => {
for (const name of await readdir(current)) {
const full = path.join(current, name)
const info = await stat(full).catch(() => undefined)
if (info?.isDirectory()) await visit(full)
else out.push(full)
}
}
await visit(dir)
return out.sort()
}
describe("issue #204: es improve harvest + distill over several runs incl. a broken chain", () => {
test("harvest reports the broken chain and scrubs planted tokens; distill writes all three kinds, drops fabrications", async () => {
const parent = await mkdtemp(path.join(tmpdir(), "es-204-cli-"))
const root = await mkdtemp(path.join(tmpdir(), "es-204-cli-root-"))
const state = await mkdtemp(path.join(tmpdir(), "es-204-cli-state-"))
try {
const run = async (argv: string[]) => {
const printed: string[] = []
const code = await main(argv, {
print: (text) => printed.push(text),
confirm: pipe,
cwd: root,
stateDir: state,
})
return { code, out: printed.join("\n") }
}
const runA = await makeRun(parent, "run-a", `halt A: GITHUB_TOKEN=${GH_A}`, false)
const runB = await makeRun(parent, "run-b", `halt B: Bearer ${BEARER}`, false)
const runBroken = await makeRun(parent, "run-broken", `halt C: ${GH_BROKEN}`, true)
const brokenFile = factoryLayout(runBroken).audit
const before = await readFile(brokenFile, "utf8")
const evals = path.join(parent, "evals")
await writeEvals(evals, [
{ id: "case-x", failures: ["gate lint red", "gate lint red", `leaked ${SK_ANT}`] },
{ id: "case-y", failures: ["gate lint red", `key ${AKIA} here`] },
{ id: "case-ok", failures: [] },
])
const telemetryFile = path.join(root, "telemetry.json")
const harvested = await run([
"improve",
"harvest",
"--runs",
`${runA},${runB},${runBroken}`,
"--evals",
evals,
"--out",
telemetryFile,
])
expect(harvested.code).toBe(0)
expect(harvested.out).toContain("3 run(s) and 3 eval(s)")
const raw = await readFile(telemetryFile, "utf8")
const telemetry = TelemetrySchema.parse(JSON.parse(raw))
expect(telemetry.runs.map((entry) => entry.id)).toEqual(["run-a", "run-b", "run-broken"])
for (const secret of [GH_A, GH_BROKEN, SK_ANT, AKIA, BEARER]) expect(raw).not.toContain(secret)
expect(raw).toContain("GITHUB_TOKEN=")
expect(raw).toContain("Bearer")
const brokenTelemetry = telemetry.runs.find((entry) => entry.id === "run-broken")!
expect(brokenTelemetry.auditChain.valid).toBe(false)
expect(typeof brokenTelemetry.auditChain.error).toBe("string")
// Reported, never repaired.
expect(await readFile(brokenFile, "utf8")).toBe(before)
const proposalsDir = path.join(root, "proposals")
const distilled = await run([
"improve",
"distill",
"--telemetry",
telemetryFile,
"--out",
proposalsDir,
"--min-occurrences",
"2",
"--min-runs",
"2",
])
expect(distilled.code).toBe(0)
expect(distilled.out).toContain("Nothing was applied")
expect(distilled.out).toContain("prompt-guidance")
expect(distilled.out).toContain("gate-tuning")
expect(distilled.out).toContain("domain-pack")
const files = await walkFiles(proposalsDir)
expect(files.some((file) => path.basename(file) === "gates.json")).toBe(false)
expect(files.some((file) => path.basename(file) === "proposal.json")).toBe(true)
// Fabricated evidence against the CLI-harvested telemetry is dropped with reasons.
const proposals = await distillProposals(telemetry, { minOccurrences: 2, minRuns: 2 })
expect(undisposed(proposals, telemetry)).toEqual(proposals)
const [first] = proposals
const forged = { ...first!, cluster: { ...first!.cluster, evidence: ["run x: invented quote"] } }
expect(disposeReason(forged, telemetry).some((reason) => reason.includes("verbatim"))).toBe(true)
expect(undisposed([forged], telemetry)).toEqual([])
expect(disposedWithReasons([forged], telemetry)[0]!.reasons.length).toBeGreaterThan(0)
} finally {
await rm(parent, { recursive: true, force: true })
await rm(root, { recursive: true, force: true })
await rm(state, { recursive: true, force: true })
}
})
test("a hallucinated model numeral is disposed by the gate with its reason in output and dropped.json", async () => {
const parent = await mkdtemp(path.join(tmpdir(), "es-204-cli-model-"))
const root = await mkdtemp(path.join(tmpdir(), "es-204-cli-model-root-"))
const state = await mkdtemp(path.join(tmpdir(), "es-204-cli-model-state-"))
try {
const run = async (argv: string[]) => {
const printed: string[] = []
const code = await main(argv, {
print: (text) => printed.push(text),
confirm: pipe,
cwd: root,
stateDir: state,
})
return { code, out: printed.join("\n") }
}
const runA = await makeRun(parent, "run-a", "halt A", false)
const evals = path.join(parent, "evals")
await writeEvals(evals, [
{ id: "case-x", failures: ["fails like gpt-4 would", "fails like gpt-4 would"] },
{ id: "case-y", failures: ["fails like gpt-4 would"] },
])
const telemetryFile = path.join(root, "telemetry.json")
expect((await run(["improve", "harvest", "--runs", runA, "--evals", evals, "--out", telemetryFile])).code).toBe(0)
const proposalsDir = path.join(root, "proposals")
const distilled = await run([
"improve",
"distill",
"--telemetry",
telemetryFile,
"--out",
proposalsDir,
"--min-occurrences",
"3",
"--min-runs",
"2",
])
expect(distilled.code).toBe(0)
expect(distilled.out).toContain("Disposed by the gate")
expect(distilled.out).toContain("hallucinated model numeral")
const dropped = JSON.parse(await readFile(path.join(proposalsDir, "dropped.json"), "utf8")) as Array<{
reasons: string[]
}>
expect(dropped.length).toBeGreaterThan(0)
expect(
dropped.some((entry) => entry.reasons.some((reason) => reason.includes("hallucinated model numeral"))),
).toBe(true)
} finally {
await rm(parent, { recursive: true, force: true })
await rm(root, { recursive: true, force: true })
await rm(state, { recursive: true, force: true })
}
})
})
packages/cli/test/mcp-identity.test.ts
+158 -35
View File
@@ -1,27 +1,42 @@
// MCP caller-identity pinning (#201, #230, #232): the adapter pins one agent
// id per server in the server environment — `ES_MCP_AGENT`, else the legacy
// `ES_AGENT`, else the `defaultAgent` option — and the per-call `agent`
// argument is ignored. Without an adapter identity the caller is "mcp", which
// maps to no seat, so every seat-checked tool refuses. A pinned string that is
// not a registry agent id (e.g. `programmer`) pins to a seat-less identity:
// the caller string is still pinned, it just holds no seat.
// Union of the audit (#201) and remediation pin suites at the #290 merge:
// the audit side proves the tool list, the HUMAN_VERBS screen, request-only
// approval, the fallback order and stdio wiring; the remediation side proves
// the pin order, seat-less pins, grant-recording and STOP/HALTED enforcement.
// Dropped in the union (not kept): the audit side's "server ignores legacy
// ES_AGENT" test (asserted pre-remediation behavior; the remediation pin
// order reads ES_AGENT, proven by the pin-order test kept above) and the two
// exact-duplicate list/stdio tests subsumed by stronger twins.
import { afterEach, beforeEach, describe, expect, test } from "bun:test"
import { mkdtemp, rm } from "node:fs/promises"
import { mkdtemp, rm, writeFile } from "node:fs/promises"
import { tmpdir } from "node:os"
import path from "node:path"
import { exists, factoryLayout, HUMAN_VERBS } from "@heretek-ai/es-core"
import { exists, Factory, factoryLayout, gateRunner, HUMAN_VERBS, readApproval } from "@heretek-ai/es-core"
import { createMcpServer, PROTOCOL_VERSION } from "../src/mcp.ts"
let root: string
let state: string
const realXdgConfig = process.env.XDG_CONFIG_HOME
beforeEach(async () => {
root = await mkdtemp(path.join(tmpdir(), "es-mcp-identity-"))
state = await mkdtemp(path.join(tmpdir(), "es-mcp-identity-state-"))
process.env.XDG_CONFIG_HOME = path.join(state, "xdg-config")
root = await mkdtemp(path.join(tmpdir(), "es-mcp-id-"))
state = await mkdtemp(path.join(tmpdir(), "es-mcp-id-state-"))
const proc = Bun.spawn(["git", "init", "-q", "-b", "main"], { cwd: root, stdout: "pipe", stderr: "pipe" })
await proc.exited
await writeFile(path.join(root, "README.md"), "# x\n")
})
afterEach(async () => {
if (realXdgConfig === undefined) delete process.env.XDG_CONFIG_HOME
else process.env.XDG_CONFIG_HOME = realXdgConfig
await rm(root, { recursive: true, force: true })
await rm(state, { recursive: true, force: true })
})
const call = (server: ReturnType<typeof createMcpServer>, name: string, args: Record<string, unknown>, id = 1) =>
const call = (server: ReturnType<typeof createMcpServer>, name: string, args: Record<string, unknown> = {}, id = 1) =>
server.handle({ jsonrpc: "2.0", id, method: "tools/call", params: { name, arguments: args } })
const toolNames = async (server: ReturnType<typeof createMcpServer>): Promise<string[]> => {
@@ -42,27 +57,52 @@ const SEAT_GATED: Readonly<Record<string, { readonly args: Record<string, unknow
release: { args: {}, spoof: "factory" },
}
describe("mcp identity pinning (#201, proof gaps #230)", () => {
test("lists exactly the 12 coarse tools (a new tool must face the HUMAN_VERBS screen below)", async () => {
const server = createMcpServer({ root, stateDir: state, env: {} })
expect((await toolNames(server)).sort()).toEqual(
[
"status",
"frontier_write",
"request_approval",
"research_complete",
"spec_validate",
"build_start",
"gates_run",
"complete",
"qa_verdict",
"tiebreak",
"replan",
"release",
].sort(),
)
describe("mcp identity pin order (mcp.ts:29)", () => {
test("the legacy ES_AGENT pins the caller when ES_MCP_AGENT is absent", async () => {
const server = createMcpServer({ root, stateDir: state, env: { ES_AGENT: "factory" } })
const refused = await call(server, "complete", {})
expect(refused?.result.isError).toBe(true)
expect(refused?.result.content[0].text).toContain("called by factory")
})
test("ES_MCP_AGENT beats ES_AGENT beats defaultAgent", async () => {
const top = createMcpServer({
root,
stateDir: state,
env: { ES_MCP_AGENT: "es-qa-functional", ES_AGENT: "factory" },
defaultAgent: "factory",
})
const pinned = await call(top, "complete", { agent: "factory" })
expect(pinned?.result.isError).toBe(true)
expect(pinned?.result.content[0].text).toContain("called by es-qa-functional")
const legacy = createMcpServer({
root,
stateDir: state,
env: { ES_AGENT: "factory" },
defaultAgent: "es-qa-functional",
})
const viaLegacy = await call(legacy, "complete", {})
expect(viaLegacy?.result.isError).toBe(true)
expect(viaLegacy?.result.content[0].text).toContain("called by factory")
const fallback = createMcpServer({ root, stateDir: state, env: {}, defaultAgent: "factory" })
const viaDefault = await call(fallback, "complete", {})
expect(viaDefault?.result.isError).toBe(true)
expect(viaDefault?.result.content[0].text).toContain("called by factory")
})
test("a pinned string without a registry seat stays seat-less but pinned", async () => {
const server = createMcpServer({ root, stateDir: state, env: { ES_MCP_AGENT: "programmer" } })
const refused = await call(server, "complete", {})
expect(refused?.result.isError).toBe(true)
// Registry ids are `es-programmer`/`es-manager`/`factory`: the literal
// `programmer` holds no seat, yet the caller string is still the pin.
expect(refused?.result.content[0].text).toContain("called by programmer")
})
})
describe("mcp identity pinning (#201, proof gaps #230)", () => {
test("no MCP tool grants a human-only action (derived from HUMAN_VERBS, not a fixed regex)", async () => {
const server = createMcpServer({ root, stateDir: state, env: {} })
const names = await toolNames(server)
@@ -125,13 +165,6 @@ describe("mcp identity pinning (#201, proof gaps #230)", () => {
expect(scoped?.result.content[0].text).not.toContain("called by")
})
test("createMcpServer itself does not read legacy ES_AGENT (only main.ts maps it to defaultAgent)", async () => {
const server = createMcpServer({ root, stateDir: state, env: { ES_AGENT: "factory" } })
const refused = await call(server, "complete", {})
expect(refused?.result.isError).toBe(true)
expect(refused?.result.content[0].text).toContain("called by mcp")
})
for (const [name, { args, spoof }] of Object.entries(SEAT_GATED)) {
test(`as mcp refuses ${name}: isError + called by mcp`, async () => {
const server = createMcpServer({ root, stateDir: state, env: {} })
@@ -227,3 +260,93 @@ describe("mcp identity pinning (#201, proof gaps #230)", () => {
}, 60_000)
})
})
describe("no MCP tool grants a human-only verb (derived from HUMAN_VERBS)", () => {
test("the coarse tool list is exactly the 12 read/request/seat tools", async () => {
const server = createMcpServer({ root, stateDir: state })
const list = await server.handle({ jsonrpc: "2.0", id: 1, method: "tools/list" })
const names = list?.result.tools.map((tool: { name: string }) => tool.name).sort()
// A future human-only verb gaining an MCP tool changes this list and fails loudly here.
expect(names).toEqual(
[
"build_start",
"complete",
"frontier_write",
"gates_run",
"qa_verdict",
"release",
"replan",
"request_approval",
"research_complete",
"spec_validate",
"status",
"tiebreak",
].sort(),
)
})
test("no listed tool (except the request-only approval) matches a HUMAN_VERBS rule", async () => {
const server = createMcpServer({ root, stateDir: state })
const list = await server.handle({ jsonrpc: "2.0", id: 1, method: "tools/list" })
const names = list?.result.tools.map((tool: { name: string }) => tool.name) as string[]
const humanOnly = (verb: string, sub: string | undefined): boolean => {
const rule = HUMAN_VERBS.find(([name]) => name === verb)
if (!rule) return false
const [, predicate] = rule
return predicate === undefined ? true : predicate(sub)
}
for (const name of names) {
if (name === "request_approval") continue // request-only: proven below, never grants.
const tokens = name.split(/[_-]/)
const matched = HUMAN_VERBS.some(([verb, predicate]) => {
if (!tokens.includes(verb)) return false
if (predicate === undefined) return true
return tokens.some((token) => token !== verb && predicate(token))
})
expect([name, matched, humanOnly("never-a-verb", undefined)]).toEqual([name, false, false])
}
})
test("request_approval records a request without granting anything", async () => {
const factory = new Factory(root, { gates: gateRunner({ stateDir: state }), stateDir: state })
await factory.writeFrontier("grill", {
version: "1.1",
idea: "mcp test",
spendCeiling: { currency: "USD", maxAmount: 3 },
settled: true,
nodes: [{ id: "a", question: "q?", answer: "yes", status: "settled" }],
})
const server = createMcpServer({ root, stateDir: state, env: { ES_MCP_AGENT: "factory" } })
const requested = await call(server, "request_approval", { stage: "frontier" })
expect(requested?.result.isError).toBeUndefined()
expect(requested?.result.content[0].text).toContain("Requested human approval")
// Request-only: the approval file is still absent (a human grants it elsewhere).
expect(await readApproval(root, "frontier")).toBeUndefined()
})
})
describe("mcp STOP/HALTED enforcement (#227)", () => {
test("a STOP file refuses tool calls on the MCP surface", async () => {
await Bun.write(path.join(root, ".factory/STOP"), "halt\n")
try {
const server = createMcpServer({ root, stateDir: state, env: { ES_AGENT: "factory" } })
const refused = await call(server, "status", {})
expect(refused?.result.isError).toBe(true)
expect(refused?.result.content[0].text).toContain("stopped")
} finally {
await rm(path.join(root, ".factory/STOP"), { force: true })
}
})
test("a HALTED run refuses non-status tool calls on the MCP surface", async () => {
const factory = new Factory(root, { gates: gateRunner({ stateDir: state }), stateDir: state })
await factory.begin("human:tester")
await factory.halt("human:tester", "spend ceiling")
const server = createMcpServer({ root, stateDir: state, env: { ES_AGENT: "factory" } })
const refused = await call(server, "complete", {})
expect(refused?.result.isError).toBe(true)
expect(refused?.result.content[0].text).toContain("halted")
const status = await call(server, "status", {})
expect(status?.result.isError).not.toBe(true)
})
})
packages/cli/test/trust-pinning.test.ts
+131
View File
@@ -0,0 +1,131 @@
// Trust pinning verbs stay terminal-only (#181 §2, #221 per-verb pipe gaps).
// Pipe (non-interactive) runs of every human-gated trust verb refuse with code
// 3 and record nothing; the shell policy denies the same verbs for every
// agent. No test here runs a real human verb: the only confirms are pipes.
import { afterEach, beforeEach, describe, expect, test } from "bun:test"
import { mkdir, mkdtemp, rm, writeFile } from "node:fs/promises"
import { tmpdir } from "node:os"
import path from "node:path"
import {
evaluateShell,
Factory,
factoryLayout,
gateRunner,
invokesHumanOnly,
sealHumanKey,
verifyControl,
} from "@heretek-ai/es-core"
import { main } from "../src/main.ts"
import type { ConfirmIO } from "../src/tty.ts"
const PASSPHRASE = "test-passphrase-181"
let root: string
let state: string
const realXdgConfig = process.env.XDG_CONFIG_HOME
const git = async (...args: string[]) => {
const proc = Bun.spawn(["git", ...args], { cwd: root, stdout: "pipe", stderr: "pipe" })
await proc.exited
}
beforeEach(async () => {
root = await mkdtemp(path.join(tmpdir(), "es-trust-pinning-"))
state = await mkdtemp(path.join(tmpdir(), "es-trust-pinning-state-"))
process.env.XDG_CONFIG_HOME = path.join(state, "xdg-config")
await git("init", "-q", "-b", "main")
await writeFile(path.join(root, "README.md"), "# x\n")
await git("add", "-A")
await git("-c", "user.name=t", "-c", "user.email=t@t", "commit", "-q", "-m", "init")
await sealHumanKey(PASSPHRASE, state)
})
afterEach(async () => {
if (realXdgConfig === undefined) delete process.env.XDG_CONFIG_HOME
else process.env.XDG_CONFIG_HOME = realXdgConfig
await rm(root, { recursive: true, force: true })
await rm(state, { recursive: true, force: true })
})
const pipe: ConfirmIO = { interactive: false, write: () => {}, readLine: async () => "y", readSecret: async () => "" }
const run = async (argv: string[], confirm: ConfirmIO = pipe) => {
const printed: string[] = []
const code = await main(argv, { print: (text) => printed.push(text), confirm, cwd: root, stateDir: state })
return { code, out: printed.join("\n") }
}
const trustFile = () => path.join(state, "trust.json")
const waiverDir = () => factoryLayout(root).waivers
describe("trust verbs refuse a pipe (#221)", () => {
test("es trust needs a terminal and trusts nothing", async () => {
await mkdir(factoryLayout(root).dir, { recursive: true })
await writeFile(
factoryLayout(root).gates,
JSON.stringify({
version: 1,
commands: [{ id: "probe", kind: "test", command: "touch RAN" }],
security: { gitleaks: "off", osv: "off" },
}),
)
const result = await run(["trust"])
expect(result.code).toBe(3)
expect(result.out).toContain("interactive terminal")
expect(await Bun.file(trustFile()).exists()).toBe(false)
})
test("es waive needs a terminal and records nothing", async () => {
const result = await run(["waive", "security/*", "--reason", "legacy code under migration"])
expect(result.code).toBe(3)
expect(result.out).toContain("interactive terminal")
expect(await Bun.file(path.join(waiverDir(), "security-abc.json")).exists()).toBe(false)
const { readdir } = await import("node:fs/promises")
expect(await readdir(waiverDir()).catch(() => [])).toEqual([])
})
test("es rebaseline needs a terminal and keeps the drift", async () => {
const { rebaseline } = await import("@heretek-ai/es-core")
await rebaseline(root, "test")
await mkdir(factoryLayout(root).dir, { recursive: true })
await writeFile(factoryLayout(root).config, '{"pr":"off"}\n')
expect((await verifyControl(root)).clean).toBe(false)
const result = await run(["rebaseline"])
expect(result.code).toBe(3)
expect(result.out).toContain("interactive terminal")
expect((await verifyControl(root)).clean).toBe(false)
})
test("es reseal --sign needs a terminal and keeps the mismatch", async () => {
const factory = new Factory(root, { gates: gateRunner({ stateDir: state }), stateDir: state })
await factory.begin("human:tester")
const stateFile = factoryLayout(root).state
await writeFile(stateFile, `${await Bun.file(stateFile).text()} `)
const result = await run(["reseal", "--sign"])
expect(result.code).toBe(3)
expect(result.out).toContain("interactive terminal")
await expect(factory.read()).rejects.toThrow("signature mismatch")
})
})
describe("agent shells cannot reach human-gated trust verbs", () => {
test("the policy denies trust, waive, rebaseline and reseal for every agent", () => {
for (const command of [
"es trust",
"es waive lint/check --reason legacy",
"es rebaseline",
"es reseal --sign",
"es approve frontier",
]) {
expect([command, invokesHumanOnly(command)]).toEqual([command, true])
for (const agent of [undefined, "build", "factory", "es-programmer"]) {
const decision = evaluateShell({ root, stateDir: state }, agent, command, { sandboxAvailable: true })
expect([command, agent, decision.effect]).toEqual([command, agent, "deny"])
if (decision.effect === "deny") expect(decision.reason).toContain("human-only")
}
}
})
test("reads keep working: help and status stay allowed", () => {
expect(invokesHumanOnly("es trust --help")).toBe(false)
expect(evaluateShell({ root, stateDir: state }, "build", "es status", { sandboxAvailable: true }).effect).toBe(
"allow",
)
})
})
packages/core/assets/domain-packs/biopharma.json
+1 -2
View File
@@ -6,8 +6,7 @@
"PEER_REVIEWED": 1.0,
"TECHNICAL_DOCUMENTATION": 0.8,
"PRIMARY_PRESS": 0.7,
"PREPRINT": 0.3,
"SECONDARY_MEDIA": 0.2
"PREPRINT": 0.3
},
"negBonus": 0.5,
"rejectPenalty": 2.5,
packages/core/assets/domain-packs/legal.json
+1 -2
View File
@@ -6,8 +6,7 @@
"PEER_REVIEWED": 0.8,
"TECHNICAL_DOCUMENTATION": 0.7,
"PRIMARY_PRESS": 1.0,
"PREPRINT": 0.2,
"SECONDARY_MEDIA": 0.3
"PREPRINT": 0.2
},
"negBonus": 0.5,
"rejectPenalty": 2.5,
packages/core/assets/domain-packs/quant.json
+1 -2
View File
@@ -6,8 +6,7 @@
"PEER_REVIEWED": 0.9,
"TECHNICAL_DOCUMENTATION": 1.0,
"PRIMARY_PRESS": 0.6,
"PREPRINT": 0.5,
"SECONDARY_MEDIA": 0.2
"PREPRINT": 0.5
},
"negBonus": 0.75,
"rejectPenalty": 3.0,
packages/core/assets/prompts/brainstormer.md
+3 -3
View File
@@ -1,8 +1,8 @@
---
id: brainstormer
version: 3
version: 4
seat: brainstormer
description: Fans out divergent lenses on an idea and distills a diversified shortlist.
description: Fans out divergent lenses (eight-lens registry; six by default, four lean for callable runs) on an idea and distills a diversified shortlist.
---
You are the **brainstormer** agent of Epistemic Swarm. Your job is divergent, lateral ideation — never bug fixes, never implementation plans. You orchestrate lenses and distil their output; the mechanics (caps, dedupe, ranking, shortlist) are enforced by tools, not by you.
@@ -11,7 +11,7 @@ You are the **brainstormer** agent of Epistemic Swarm. Your job is divergent, la
2. Launch every lens subagent **in parallel** with the subagent tool: all the calls in one message, in the foreground (background launches are refused). Call them by exact ID (`es-lens-inversion`, `es-lens-constraint-removal`, `es-lens-scamper`, `es-lens-cross-domain`, `es-lens-extreme-user`, `es-lens-scale-shift`, and so on). Give each the brief and how many ideas you expect.
3. Record each lens's output with `es_brainstorm_record`, one call per lens. It rejects cap violations and collapses near-duplicates; mention convergences when lenses independently reach the same idea.
4. Launch `es-brainstorm-critic` with the surviving idea IDs and their texts. The critic scores every idea with `es_brainstorm_score` (1-5 on novelty, upside, feasibility, fit).
5. Call `es_brainstorm_complete`. It re-checks coverage and scores, ranks the ideas, picks the diversified shortlist with one forced outlier slot, and writes the run's `brainstorm.json` and `BRAINSTORM.md`. Report the shortlist to the human, briefly.
5. Call `es_brainstorm_complete`. It re-checks coverage and scores, ranks the ideas, picks the diversified shortlist with one forced outlier slot (whenever the shortlist has ≥2 slots), and writes the run's `brainstorm.json` and `BRAINSTORM.md`. Report the shortlist to the human, briefly.
## Prior art yourself
You can run `es_harvest_prior_art` on the surviving ideas before completing: it finds related projects, you judge each relation (`novel`/`similar`/`existing`), and you pass the list to `es_brainstorm_complete` as `priorArt`. Only recorded searches count — the tool refuses URLs no search returned.
packages/core/assets/prompts/factory.md
+2 -2
View File
@@ -1,13 +1,13 @@
---
id: factory
version: 9
version: 10
seat: factory
description: Orchestrates the build factory from research to release.
---
You are the **factory** agent of Epistemic Swarm, an AI build factory. You orchestrate; you do not write product code yourself.
## How the factory works
The run moves through stages enforced by code, not by you: GRILL → RESEARCH → SPEC → BUILD ⇄ QA → RELEASE → DONE. HALTED can happen at any time. Call `es_status` first and whenever you are unsure; its `<factory-state>` block is the truth. If it says there is no run, stop and tell the human to run `/grill` first. Every `es_*` tool refuses an out-of-order step and tells you why. Read the refusal and follow it instead of retrying blindly.
The run moves through stages enforced by code, not by you: GRILL → RESEARCH → SPEC → BUILD ⇄ QA → RELEASE → DONE. HALTED can happen at any time. Call `es_status` first and whenever you are unsure; its `<factory-state>` block is the truth. If it says there is no run, stop and tell the human to run `/grill` first. The phase-transition tools refuse an out-of-order step and tell you why. Read the refusal and follow it instead of retrying blindly.
Two checkpoints belong to the human and only the human: approving the frontier (end of grill) and approving the spec (roadmap plus every GOAL.md). You cannot approve anything. When a checkpoint is due, call `es_request_approval` and tell the human to run `/es-approve` in the TUI (or `es approve <stage>` in a terminal). Then stop and wait.
packages/core/assets/skills/brainstorm/SKILL.md
+3 -3
View File
@@ -1,10 +1,10 @@
---
name: brainstorm
description: Divergent lens fan-out with deterministic dedupe, rubric scoring, and a diversified shortlist with a forced outlier.
description: Divergent lens fan-out (eight-lens registry; six by default, four lean for callable runs) with deterministic dedupe, rubric scoring, and a diversified shortlist with a forced outlier whenever the shortlist has ≥2 slots.
---
# Brainstorm
Lateral ideation on an idea, run as a fan-out of divergent lenses and distilled mechanically. Never bug fixes, never implementation plans.
Lateral ideation on an idea, run as a fan-out of divergent lenses (eight in the registry; six by default, four lean for callable runs) and distilled mechanically. Never bug fixes, never implementation plans.
## When to use
- The human runs `/brainstorm <idea>` or asks for ideas, directions or alternatives.
@@ -15,7 +15,7 @@ Lateral ideation on an idea, run as a fan-out of divergent lenses and distilled
2. Launch every lens subagent in parallel, by exact ID (`es-lens-<lens>`), each with the brief. Fast tier; one lens each: inversion, constraint-removal, scamper, cross-domain, extreme-user, scale-shift, failure-first, adjacent-possible.
3. `es_brainstorm_record` — one call per lens. Caps are enforced; near-duplicates are collapsed against earlier ideas (convergence is a signal).
4. `es-brainstorm-critic` — scores every surviving idea 1-5 on novelty, upside, feasibility and fit via `es_brainstorm_score`.
5. `es_brainstorm_complete` — re-checks coverage and scores, ranks, and writes the diversified shortlist with one forced outlier slot to `.factory/brainstorm/`.
5. `es_brainstorm_complete` — re-checks coverage and scores, ranks, and writes the diversified shortlist with one forced outlier slot (whenever the shortlist has ≥2 slots) to `.factory/brainstorm/`.
## Output
`.factory/brainstorm/brainstorm.json` (machine-readable) and `BRAINSTORM.md` (the shortlist, the full ranked field, collapsed duplicates, coverage and gaps). Report the shortlist to the human; do not paste the whole field.
packages/core/src/agents/registry.ts
+2 -1
View File
@@ -177,7 +177,8 @@ export const AGENTS: readonly AgentSpec[] = [
mode: "primary",
hidden: false,
tier: "deep",
description: "Fans out divergent lenses on an idea and returns a diversified shortlist.",
description:
"Fans out divergent lenses (eight-lens registry; six by default, four lean for callable runs) on an idea and distills a diversified shortlist.",
prompt: "brainstormer",
tools: [
"es_status",
packages/core/src/approval/attempts.ts
+2 -1
View File
@@ -1,6 +1,7 @@
// Approval attempt limiting (ADR 0002 I4): at most 5 wrong passphrases per
// 10 minutes per state dir, then a lockout. Lockouts are audited into the
// project's audit log. One surface's lockout does not affect the others.
// project's audit log. The CLI and the TUI share one attempt file per state
// dir (a single bucket for both); the web path keeps its own dir and bucket.
import path from "node:path"
import { appendAuditEntry } from "../audit/chain.ts"
import { readJson, withLock, writeJson } from "../util/fs.ts"
packages/core/src/approval/keystore.ts
+7 -1
View File
@@ -103,7 +103,13 @@ async function readText(file: string): Promise<string | undefined> {
/** Whether this machine holds a sealed human key (`es key seal` has run). */
export async function hasHumanKey(dir = stateDir()): Promise<boolean> {
return (await readText(path.join(dir, SEALED_FILE))) !== undefined
// Both halves are required (#213): a lone human.key (or human.pub) is a
// half-deleted brick, and unlock would refuse it as tampered or partial.
const [sealed, pub] = await Promise.all([
readText(path.join(dir, SEALED_FILE)),
readText(path.join(dir, PUBLIC_FILE)),
])
return sealed !== undefined && pub !== undefined
}
async function publicKey(dir: string): Promise<KeyObject | undefined> {
packages/core/src/brainstorm/engine.ts
+3 -1
View File
@@ -240,7 +240,9 @@ export const OUTLIER_THRESHOLD = 0.35
* Greedy diversified selection: quality plus distance from what is already
* picked. The last slot is forced to an outlier — the best idea whose maximum
* similarity to the shortlist is below the outlier threshold, or the most
* dissimilar idea when none is far enough.
* dissimilar idea when none is far enough — whenever the shortlist has ≥2
* slots (and at least two ideas are ranked; a single-idea shortlist carries
* `outlier: false`).
*/
export function selectShortlist(
ranked: readonly RankedIdea[],
packages/core/src/brainstorm/ops.ts
+4 -3
View File
@@ -113,7 +113,8 @@ export function brainstormTools(context: BrainstormOpsContext): EsToolDef[] {
ideasPerLens: { type: "number", description: "Ideas per lens (1-10, default 5)." },
shortlistSize: {
type: "number",
description: "Shortlist size including the forced outlier (2-12, default 7).",
description:
"Shortlist size (2-12, default 7); the last slot is a forced outlier whenever the shortlist has ≥2 slots.",
},
dedupe: {
type: "string",
@@ -176,7 +177,7 @@ export function brainstormTools(context: BrainstormOpsContext): EsToolDef[] {
" Give each one the brief (and constraints) plus how many ideas you expect.",
`2. Record each lens's ideas with es_brainstorm_record (one call per lens, run "${run}").`,
"3. Launch es-brainstorm-critic with the surviving idea ids and texts; it scores each with es_brainstorm_score.",
`4. Call es_brainstorm_complete with run "${run}". Dedupe, ranking and the shortlist (with one forced outlier) are computed there.`,
`4. Call es_brainstorm_complete with run "${run}". Dedupe, ranking and the shortlist (with one forced outlier whenever the shortlist has ≥2 slots) are computed there.`,
"",
`Rubric (1-5 each): ${RUBRIC.map((item) => item.id).join(", ")}.`,
].join("\n")
@@ -344,7 +345,7 @@ export function brainstormTools(context: BrainstormOpsContext): EsToolDef[] {
{
name: "es_brainstorm_complete",
description:
"Brainstormer, grill or factory: collapse duplicates, rank the scored ideas and write the diversified shortlist with a forced outlier slot. Returns the shortlist as JSON the caller can parse.",
"Brainstormer, grill or factory: collapse duplicates, rank the scored ideas and write the diversified shortlist with a forced outlier slot (whenever the shortlist has ≥2 slots). Returns the shortlist as JSON the caller can parse.",
input: object({
run: { type: "string", description: "Run id from es_brainstorm_plan." },
allowPartial: {
packages/core/src/capabilities.ts
+5 -4
View File
@@ -97,7 +97,8 @@ export const CAPABILITY_MATRIX: Readonly<Record<Harness, readonly CapabilityRow[
},
lsp: {
support: "enforced",
detail: "The lsp config key is read and served by our own manager (v2 accepts but does not run it).",
detail:
"The `lsp` key in v2's `opencode.json` — not an ES config key (ES layers strict-reject it) — is read and served by our own manager (v2 accepts but does not run it).",
test: "packages/opencode/test/lsp.test.ts",
},
panels: {
@@ -115,7 +116,7 @@ export const CAPABILITY_MATRIX: Readonly<Record<Harness, readonly CapabilityRow[
brainstorm: {
support: "enforced",
detail:
"Lens fan-out, record/score/complete tools, dedupe, rubric, shortlist with a forced outlier. Runs are id-scoped (the human's /brainstorm is the default run); the grill and the factory run callable brainstorms at depth 1 and get the shortlist as JSON.",
"Lens fan-out, record/score/complete tools, dedupe, rubric, shortlist with a forced outlier whenever the shortlist has ≥2 slots. Runs are id-scoped (the human's /brainstorm is the default run); the grill and the factory run callable brainstorms at depth 1 and get the shortlist as JSON.",
test: "packages/opencode/test/brainstorm.test.ts",
},
harvest: {
@@ -145,8 +146,8 @@ export const CAPABILITY_MATRIX: Readonly<Record<Harness, readonly CapabilityRow[
scout: {
support: "enforced",
detail:
"OSS scout: cached-only web under an offline (--unshare-net) sandbox, fail-closed license verdicts computed by core (adopt only for verified permissive licenses), OSV advisories cited.",
test: "packages/opencode/test/fires.test.ts",
"OSS scout: cached-only web under an offline (--unshare-net) sandbox, fail-closed license verdicts computed by core (adopt only for verified permissive licenses), OSV advisories cited (canned-OSV content in core's scout suite).",
test: "packages/opencode/test/scout-host.test.ts",
},
"research-fires": {
support: "enforced",
packages/core/src/claims/brief.ts
+2 -1
View File
@@ -182,7 +182,8 @@ export async function verifyBrief(
`signed by key ${brief.signature.keyId}, which this machine does not hold; only integrity can be checked (allow unverifiable to accept that)`,
)
// 4. Quotes: every claim the brief calls witnessed must quote its bundled source verbatim.
// 4. Quotes: every claim the brief calls witnessed must quote its bundled
// source verbatim after documented normalisation (research/quote.ts).
let quotesVerified = 0
let quotesFailed = 0
for (const claim of brief.claims as Claim[]) {
packages/core/src/claims/constitution.ts
+28 -13
View File
@@ -34,15 +34,25 @@ export const LEGACY_CONSTITUTION: Constitution = {
retractionPolicy: "standard",
}
export const constitutionFromPack = (pack: DomainPack): Constitution => ({
tierWeights: { __default__: 1, ...pack.tierWeights },
negBonus: pack.negBonus,
rejectPenalty: pack.rejectPenalty,
acceptThreshold: pack.acceptThreshold,
bannedDomains: pack.bannedDomains,
mandatoryTags: pack.mandatoryTags,
retractionPolicy: pack.retractionPolicy,
})
export const constitutionFromPack = (pack: DomainPack): Constitution => {
// Tierless claims weigh at or below the lowest verified tier (#252): an
// unclassifiable host falling back to a full 1.0 outscores verified
// preprints and rewards citing worse sources.
const tiered = Object.entries(pack.tierWeights)
.filter(([tier]) => tier !== "__default__")
.map(([, weight]) => weight)
const floor = tiered.length ? Math.min(...tiered) : 1
const fallback = pack.tierWeights.__default__ ?? 1
return {
tierWeights: { ...pack.tierWeights, __default__: Math.min(fallback, floor) },
negBonus: pack.negBonus,
rejectPenalty: pack.rejectPenalty,
acceptThreshold: pack.acceptThreshold,
bannedDomains: pack.bannedDomains,
mandatoryTags: pack.mandatoryTags,
retractionPolicy: pack.retractionPolicy,
}
}
export const weightFor = (constitution: Constitution, tier: string | undefined): number =>
(tier !== undefined && constitution.tierWeights[tier] !== undefined
@@ -131,7 +141,7 @@ export interface ClaimVerdictOptions {
/**
* The per-claim ACCEPTED/REJECTED gate used by domain packs. A claim is
* REJECTED when the legacy structural rules fire (VERIFIED needs its source
* hash + verbatim quote or a code location; INFERRED needs its reasoning;
* hash + quote (verbatim after documented normalisation) or a code location; INFERRED needs its reasoning;
* HYPOTHESIS needs its falsification; NEGATIVE_KNOWLEDGE needs query and
* finding), when any pack rule fires (parent unresolved, missing mandatory
* tag, banned domain), or when the retraction policy is zero_tolerance and
@@ -165,7 +175,8 @@ export function claimVerdict(claim: ConstitutionClaim, options: ClaimVerdictOpti
if (!nonEmpty(claim.query)) reasons.push("NEG_KNOWLEDGE_REQUIRES_QUERY: query missing")
if (!nonEmpty(claim.finding)) reasons.push("NEG_KNOWLEDGE_REQUIRES_FINDING: finding missing")
}
if (constitution.mandatoryTags.length && !claim.tag) reasons.push("MISSING_TAG: tag required by constitution")
if (constitution.mandatoryTags.length && !constitution.mandatoryTags.includes(claim.tag ?? ""))
reasons.push(`MISSING_TAG: tag '${claim.tag ?? ""}' is not one of the constitution's mandatory tags`)
const url = claim.source?.url
if (constitution.bannedDomains.length && typeof url === "string" && url) {
const lower = url.toLowerCase()
@@ -199,7 +210,8 @@ export interface ScoreBreakdown {
* Verified claims contribute their tier weight when the constitution weights
* any tier away from 1.0 (otherwise a flat 1.0, preserving legacy numbers);
* rejected claims cost `rejectPenalty`; counted negative knowledge earns
* `negBonus`. Valid, deduplicated NK rows only, via the M1 port of the legacy
* `negBonus` and counts as assertions, so stuffing NK rows cannot inflate the
* score past the threshold (#252). Valid, deduplicated NK rows only, via the M1 port of the legacy
* normalized key. The score is clamped to [0, 1] and rounded to 3 places.
*/
export function computeEpistemicScoreFromClaims(
@@ -238,7 +250,10 @@ export function computeEpistemicScoreFromClaims(
} else if (claim.tag === "INFERRED") counts.inferred++
else if (claim.tag === "HYPOTHESIS") counts.hypotheses++
}
const totalAssertions = Math.max(1, counts.verified + counts.inferred + counts.hypotheses + counts.rejected)
const totalAssertions = Math.max(
1,
counts.verified + counts.inferred + counts.hypotheses + counts.rejected + negKnowledge,
)
const rawScore =
(weightSum + constitution.negBonus * negKnowledge - constitution.rejectPenalty * counts.rejected) / totalAssertions
const score = Math.max(0, Math.min(1, Math.round(rawScore * 1000) / 1000))
packages/core/src/claims/witness.ts
+2 -1
View File
@@ -4,7 +4,8 @@
// refusing get + the one quote law, research/quote.ts);
// - a code location against the file on disk: inside the target root, the
// line range within the file and at most MAX_LOCATION_LINES long, the
// excerpt one contiguous verbatim span of those lines.
// excerpt one contiguous span of those lines, verbatim after documented
// normalisation.
// Legacy never checked code locations at all (its "hallucinated line" check
// was the generic quote check), so the location rule is new in 1.1.
import path from "node:path"
packages/core/src/evals/grade.ts
+1 -1
View File
@@ -48,7 +48,7 @@ export interface EvalTranscript {
readonly tokens: EvalTokens
}
const num = (value: unknown): number => (typeof value === "number" && Number.isFinite(value) ? value : 0)
const num = (value: unknown): number => (typeof value === "number" && Number.isFinite(value) && value >= 0 ? value : 0)
/**
* Tools whose outputs are raw evidence, not the seat's own words. Quoting
packages/core/src/evals/run.ts
+9 -2
View File
@@ -41,7 +41,8 @@ export const DEFAULT_CASE_STEPS = 6
* makes one call per step), bounded by the run-wide ceiling.
*/
export function caseStepCap(caseMaxSteps: number | undefined, ceiling: number): number {
return Math.min(caseMaxSteps ?? DEFAULT_CASE_STEPS, ceiling)
const own = typeof caseMaxSteps === "number" && Number.isFinite(caseMaxSteps) ? caseMaxSteps : DEFAULT_CASE_STEPS
return Math.min(own, ceiling)
}
export interface EvalWorkspace {
@@ -140,6 +141,11 @@ export async function runEvalCase(
workspace: EvalWorkspace,
options: EvalRunOptions,
): Promise<EvalResult> {
// Fail closed on flag injection (#211): the prompt is a positional argv, so
// a leading dash would parse as a CLI flag. Cases are repo files, but the
// runner must not depend on their goodwill.
if (testCase.prompt.startsWith("-"))
throw new Error(`Refusing eval case "${testCase.id}": the prompt must not start with "-".`)
const child = spawn(
options.binary,
["run", "--standalone", "--format", "json", "--agent", testCase.agent, testCase.prompt],
@@ -167,7 +173,8 @@ export async function runEvalCase(
child.kill("SIGKILL")
}
if (event.type === "step_finish" && typeof event.part?.cost === "number" && Number.isFinite(event.part.cost)) {
spent += event.part.cost
// Clamp hostile negatives: a negative cost must never unbind spend (#211).
spent += Math.max(0, event.part.cost)
if (options.budgetUSD !== undefined && spent > options.budgetUSD && !overBudget) {
overBudget = true
child.kill("SIGKILL")
packages/core/src/factory/machine.ts
+11 -3
View File
@@ -319,7 +319,10 @@ export class Factory {
// (CI-only es_build_start refusal). Real transitions still enforce it.
if (!options.skipControl) {
const control = await verifyControl(this.root)
if (!control.clean) return halt(`control files changed outside a human action: ${control.violations.join("; ")}`)
if (!control.clean)
return halt(
`control files changed outside a human action (integrity/control-drift): ${control.violations.join("; ")}. The structured finding is in .factory/runs/${state.runId}/gates/*/summary.json; a human must review and rebaseline.`,
)
}
if (state.spendCeilingUSD !== undefined && state.spend.usd >= state.spendCeilingUSD)
return halt(
@@ -433,6 +436,10 @@ export class Factory {
this.requireAuditsClear(state)
const check = await verifyApproval(this.root, "frontier", { stateDir: this.deps.stateDir })
if (!check.ok) throw new FactoryError(check.reason)
if (check.record.spendCeilingUSD === undefined)
throw new FactoryError(
"The frontier approval carries no spend ceiling; a human must approve it again (`es approve frontier`).",
)
state.spendCeilingUSD = check.record.spendCeilingUSD
state.stage = "RESEARCH"
await this.audit(actor, "stage.research", { spendCeilingUSD: state.spendCeilingUSD })
@@ -684,7 +691,8 @@ export class Factory {
/** Record a gate run made during BUILD (red test runs are the failing-test-first evidence). */
async recordGateRun(agentId: string | undefined, result: GateRunLike): Promise<void> {
await this.mutate(`agent:${agentId}`, (state) => {
if (state.stage !== "BUILD") return
if (state.stage !== "BUILD")
throw new FactoryError(`Gate runs record during BUILD only (current stage: ${state.stage}).`)
const phase = this.active(state)
const failed = result.failedTests ?? []
const red = failed.length > 0 || result.testsFailed === true
@@ -1233,7 +1241,7 @@ export class Factory {
private async prBody(state: FactoryState) {
const spec = await readApproval(this.root, "spec")
const lines = [
`Built by the Epistemic Swarm factory (run \`${state.runId}\`). A human reviews and merges; the factory never merges.`,
`Built by the Epistemic Swarm factory (run \`${state.runId}\`). A human reviews and merges; the factory never merges into the release/base branch.`,
"",
"## Phases",
...state.phases.map(
packages/core/src/gates/engine.ts
+2 -5
View File
@@ -31,7 +31,7 @@ import { validateArtifacts } from "./artifacts.ts"
import { checkBudgets, isTestFile } from "./budgets.ts"
import { detectGates } from "./detect.ts"
import { parseOutput } from "./parsers.ts"
import { scanForSecrets } from "./secrets.ts"
import { scanFileForSecrets } from "./secrets.ts"
import { DEPENDENCY_FILES, gitleaks, osvFindings } from "./security.ts"
import { isWaived, loadWaivers } from "./waivers.ts"
@@ -402,10 +402,7 @@ export async function runGates(options: RunGatesOptions): Promise<GateReport> {
if (config.security.secrets)
await timed("secrets", async () => {
const out: GateFinding[] = []
for (const file of present) {
const text = await readFile(path.join(dir, file), "utf8").catch(() => "")
if (text.length <= 2_000_000) out.push(...scanForSecrets(text, file))
}
for (const file of present) out.push(...(await scanFileForSecrets(dir, file)))
return out
})
if (config.security.gitleaks === "auto") await timed("gitleaks", () => gitleaks(dir, present.slice(0, 200)))
packages/core/src/gates/secrets.ts
+47
View File
@@ -1,5 +1,7 @@
// Built-in secret scanner (always on, no external binary). High-precision
// patterns are errors; the generic assignment heuristic is a warning.
import { open } from "node:fs/promises"
import path from "node:path"
import type { GateFinding } from "../schema/gates.ts"
const PATTERNS: ReadonlyArray<{ rule: string; regex: RegExp; description: string; severity?: "warning" }> = [
@@ -56,3 +58,48 @@ export function scanForSecrets(content: string, file: string): GateFinding[] {
})
return findings
}
// Per-path cache keyed by mtime+size (mirrors structure/graph.ts), so repeated
// gate runs re-read only changed files. The scan itself is deliberately NOT
// count-capped the way gitleaks is: a cap would fail open, silently skipping a
// secret past the cap, while the cache bounds the repeated-run cost instead.
const fileCache = new Map<string, { mtimeMs: number; size: number; findings: GateFinding[] }>()
let cacheHits = 0
/**
* Scan one file for secrets, cached by mtime+size. Missing, non-regular and
* oversized (>2MB, as before) files yield no findings, as do binary files.
*/
export async function scanFileForSecrets(dir: string, file: string): Promise<GateFinding[]> {
const absolute = path.join(dir, file)
// Open once and stat the descriptor, so the size/mtime check and the read
// cannot race with a concurrent replacement of the path.
const handle = await open(absolute, "r").catch(() => undefined)
if (!handle) return []
try {
const info = await handle.stat()
if (!info.isFile() || info.size > 2_000_000) return []
const cached = fileCache.get(absolute)
if (cached && cached.mtimeMs === info.mtimeMs && cached.size === info.size) {
cacheHits += 1
return cached.findings
}
const findings = scanForSecrets(await handle.readFile("utf8"), file)
if (fileCache.size > 50_000) fileCache.clear()
fileCache.set(absolute, { mtimeMs: info.mtimeMs, size: info.size, findings })
return findings
} finally {
await handle.close()
}
}
/** Cache observability: entries held and hits served since the last clear. */
export function secretScanStats(): { entries: number; hits: number } {
return { entries: fileCache.size, hits: cacheHits }
}
/** For tests: forget cached findings and reset the hit count. */
export function clearSecretScanCache(): void {
fileCache.clear()
cacheHits = 0
}
packages/core/src/harvest/matrix.ts
+23 -1
View File
@@ -58,6 +58,27 @@ export interface MatrixBuild {
}>
}
/**
* Scrub model-supplied evidence before it reaches a clean-room spec (#267).
* The "do not read or copy" line in the spec template is procedure, not
* enforcement: without this, a harvester that pastes upstream code into
* `evidence` contaminates the spec mechanically. Fenced code blocks go
* entirely, inline spans keep their text but lose their ticks, stray ticks
* go, whitespace collapses, and the note is capped.
*/
export const EVIDENCE_MAX = 500
export function sanitizeEvidence(evidence: string): string | undefined {
const text = evidence
.replace(/```[\s\S]*?(```|$)/g, " ")
.replace(/`([^`\n]*)`/g, "$1")
.replace(/`/g, "")
.replace(/\s+/g, " ")
.trim()
.slice(0, EVIDENCE_MAX)
.trimEnd()
return text || undefined
}
/** Apply the policy to every proposed cell and freeze the matrix. */
export function buildMatrix(
candidates: readonly string[],
@@ -74,6 +95,7 @@ export function buildMatrix(
if (!profile)
return { ...cell, verdict: "skip", downgraded: `candidate "${cell.candidate}" was not scanned; skipped` }
const checked = checkVerdict(cell.verdict, profile.license, whitelist)
const clean = cell.evidence !== undefined ? sanitizeEvidence(cell.evidence) : undefined
if (checked.downgraded)
downgrades.push({
candidate: cell.candidate,
@@ -85,7 +107,7 @@ export function buildMatrix(
return {
candidate: cell.candidate,
verdict: checked.verdict,
...(cell.evidence ? { evidence: cell.evidence } : {}),
...(clean !== undefined ? { evidence: clean } : {}),
...(checked.attribution ? { attribution: checked.attribution } : {}),
...(checked.downgraded ? { downgraded: checked.downgraded } : {}),
}
packages/core/src/harvest/spdx.ts
+41 -7
View File
@@ -4,7 +4,11 @@
// bundled SPDX template (templates.ts): extra or changed terms — a rider like
// the Commons Clause, the JSON licence's "Good, not Evil" — mean "unknown".
// - Copyleft and source-available licences are recognised by their title and
// date lines (GPL-3.0 is not mistaken for AGPL because §13 mentions it).
// date lines (the earliest GPL-family title in the bytes decides among the
// GPL family: the GPL-3.0 tail names the Lesser GPL, which must not outrank
// an earlier GPL title).
// - NOTICE/PATENTS files are not evaluated as licences: a top-level NOTICE
// without a licence file warns about the blind spot instead of verifying.
// - Every licence file in the root counts (LICENSE*, COPYING*, UNLICENSE…);
// the most restrictive wins, and several permissive ones combine with AND.
// - `SPDX-License-Identifier` headers are declarations by single files: they
@@ -28,7 +32,8 @@ const FAMILIES: Readonly<Record<string, Family>> = {
"Apache-2.0": "permissive",
"BSD-2-Clause": "permissive",
"BSD-3-Clause": "permissive",
"BSD-4-Clause": "permissive",
// No BSD-4-Clause: the advertising clause is an obligation the whitelist
// does not cover, so it must read as unknown, never as safe (#264).
ISC: "permissive",
"0BSD": "permissive",
Unlicense: "public-domain",
@@ -145,6 +150,24 @@ interface Rule {
// Lesser GPL and GPL-3.0 §13 the Affero GPL. Date lines catch texts that sit
// under a project notice.
const has = (text: string, ...phrases: string[]) => phrases.every((phrase) => text.includes(phrase))
/**
* The Lesser title plus a date line, but only when the Lesser title is the
* earliest GPL-family title in the bytes (#264): the GPL-3.0 tail ("use the
* GNU Lesser General Public License instead of this License") names it, and
* a later mention must not outrank an earlier GPL title.
*/
const lesserFirst = (body: string, date: string): boolean => {
if (!has(body, "gnu lesser general public license", date)) return false
const lesser = body.indexOf("gnu lesser general public license")
for (const title of [
"gnu affero general public license",
"gnu library general public license",
"gnu general public license",
])
if (body.includes(title) && body.indexOf(title) < lesser) return false
return true
}
const title = (phrase: string, spdx: string | ((head: string) => string), note?: string): Rule => ({
test: (head) => (head.includes(phrase) ? (typeof spdx === "string" ? spdx : spdx(head)) : undefined),
...(note ? { note } : {}),
@@ -171,8 +194,8 @@ const RESTRICTIVE: readonly Rule[] = [
title("elastic license", "Elastic-2.0", SOURCE_AVAILABLE),
// 2. Texts under a project notice: their distinctive date lines.
{ test: (_, body) => (has(body, "gnu affero general public license", "19 november 2007") ? "AGPL-3.0" : undefined) },
{ test: (_, body) => (has(body, "gnu lesser general public license", "february 1999") ? "LGPL-2.1" : undefined) },
{ test: (_, body) => (has(body, "gnu lesser general public license", "29 june 2007") ? "LGPL-3.0" : undefined) },
{ test: (_, body) => (lesserFirst(body, "february 1999") ? "LGPL-2.1" : undefined) },
{ test: (_, body) => (lesserFirst(body, "29 june 2007") ? "LGPL-3.0" : undefined) },
{ test: (_, body) => (has(body, "gnu general public license", "29 june 2007") ? "GPL-3.0" : undefined) },
{ test: (_, body) => (has(body, "gnu general public license", "june 1991") ? "GPL-2.0" : undefined) },
{ test: (_, body) => (has(body, "mozilla public license", "version 2.0") ? "MPL-2.0" : undefined) },
@@ -271,6 +294,17 @@ export async function detectLicense(
// 1. Every licence file in the root.
const entries = await readdirImpl(dir).catch(() => [] as LicenseScanEntry[])
// NOTICE/PATENTS files are not evaluated as licences (#264): when no
// licence file matched, their presence is reported on the finding so the
// blind spot is visible instead of silent.
const noticeFiles = entries
.filter((item) => !item.directory && /^(notice|patents)([-._][\w.-]+)?$/i.test(item.name))
.map((item) => item.name)
.sort(byText)
const noticed = (note: string): string =>
noticeFiles.length
? `${note}; ${noticeFiles.join(", ")} present but not evaluated as a license (NOTICE obligations are a blind spot)`
: note
const files: FileVerdict[] = []
for (const entry of entries
.filter((item) => !item.directory && LICENSE_FILE.test(item.name))
@@ -302,7 +336,7 @@ export async function detectLicense(
sha256: sha256(headers[0]!.text),
confidence: "medium",
verified: false,
note: `declared only by SPDX headers (${headers.length} file(s)); no licence text found`,
note: noticed(`declared only by SPDX headers (${headers.length} file(s)); no licence text found`),
}
}
@@ -316,7 +350,7 @@ export async function detectLicense(
file: manifest.file,
confidence: "medium",
verified: false,
note: "self-reported in a manifest; no license text found",
note: noticed("self-reported in a manifest; no license text found"),
}
return {
@@ -325,7 +359,7 @@ export async function detectLicense(
source: "manifest",
confidence: "low",
verified: false,
note: "no license file, header or manifest field found",
note: noticed("no license file, header or manifest field found"),
}
}
packages/core/src/hooks/engine.ts
+33 -1
View File
@@ -6,6 +6,7 @@
// missing trust; observers fail open, as Claude Code does. Every executed
// handler is written to the audit log.
import { appendAuditEntry } from "../audit/chain.ts"
import type { SandboxSpec } from "../trust/sandbox.ts"
import { isTrusted } from "../trust/store.ts"
import { type HookSource, hookSources, type LoadedHooks, loadHooks } from "./load.ts"
import { ifMatches, matcherMatches } from "./match.ts"
@@ -19,6 +20,16 @@ export interface HookEngineOptions {
readonly sources?: HookSource[]
readonly generate?: (prompt: string, model?: string) => Promise<string>
readonly audit?: boolean
/**
* Sandbox for hook command subprocesses (#265). Absent runs unsandboxed
* with the parent environment (Claude Code parity; project hooks are
* trusted code). "readonly" makes everything read-only; a spec runs with
* one writable directory. Without bwrap the run errors and gates fail
* closed while observers fail open.
*/
readonly sandbox?: "none" | "readonly" | Omit<SandboxSpec, "cwd">
/** No network inside the sandbox (only meaningful with `sandbox`). */
readonly offline?: boolean
}
export interface BaseInput {
@@ -188,6 +199,15 @@ export class HookEngine {
}
private async execute(event: string, items: SourcedHandler[], input: Record<string, unknown>): Promise<Ran[]> {
// The sandbox knob rides every handler run (#265); per-handler timeout
// and x-es-offline are honored in run.ts.
const sandboxContext =
this.options.sandbox !== undefined || this.options.offline !== undefined
? {
...(this.options.sandbox !== undefined ? { sandbox: this.options.sandbox } : {}),
...(this.options.offline !== undefined ? { offline: this.options.offline } : {}),
}
: {}
const ran = await Promise.all(
items.map(async (item): Promise<Ran> => {
const project = !item.source.startsWith("user:")
@@ -205,6 +225,7 @@ export class HookEngine {
root: this.options.root,
event,
input,
...sandboxContext,
...(this.options.generate ? { generate: this.options.generate } : {}),
})
return { item, outcome: { status: "ok", durationMs: 0 } }
@@ -213,6 +234,7 @@ export class HookEngine {
root: this.options.root,
event,
input,
...sandboxContext,
...(this.options.generate ? { generate: this.options.generate } : {}),
})
return { item, outcome }
@@ -388,7 +410,17 @@ export class HookEngine {
): Promise<{ additionalContext: string[]; notices: string[] }> {
const items = this.select("SessionStart", [source])
const result = { additionalContext: [] as string[], notices: [] as string[] }
if (!items.length) return result
if (!items.length) {
// An unmatched source is an explicit notice, never a silent no-op
// (#289): the bridge fires only startup + compact, so a hook filtered
// to resume/clear/fork would otherwise sit inert with no sign. Silent
// only when no SessionStart hook is registered at all.
if (this.loaded.handlers.some((item) => item.event === "SessionStart"))
result.notices.push(
`SessionStart: hook(s) registered but none match source "${source}" (the bridge fires startup + compact)`,
)
return result
}
const ran = await this.execute("SessionStart", items, { ...this.common("SessionStart", base), source })
for (const { item, outcome } of ran) {
this.failure(item, outcome, result.notices)
packages/core/src/hooks/run.ts
+23
View File
@@ -6,6 +6,15 @@
// - http: POST the input; 2xx JSON body decides, anything else is non-blocking.
// - prompt: evaluated through an injected model call; {ok, reason, impossible}.
// - mcp_tool and agent: not available in the bridge (reported as skipped).
//
// Sandboxing (#265): command subprocesses run with the parent environment,
// unsandboxed, by design (Claude Code parity) — hook commands are trusted
// code (project hooks need a human `es trust` before they run at all), so the
// parent env, secrets included, is visible to them. Callers that need
// isolation pass `sandbox`/`offline` in the RunContext (honored per handler:
// `x-es-offline` overrides the context default); without bwrap the sandboxed
// run fails and the engine's gate/observer policy decides (fail closed/open).
import type { SandboxSpec } from "../trust/sandbox.ts"
import { type RunResult, run } from "../util/proc.ts"
import type { HookHandler } from "./spec.ts"
@@ -30,6 +39,15 @@ export interface RunContext {
readonly signal?: AbortSignal
/** Model call for prompt hooks; absent means prompt hooks are skipped. */
readonly generate?: (prompt: string, model?: string) => Promise<string>
/**
* Sandbox for command subprocesses (#265). Absent (the default) runs
* unsandboxed with the parent environment, as Claude Code does. "readonly"
* makes everything read-only; a spec runs with one writable directory.
* Throws inside the outcome (error status) when bwrap is unavailable.
*/
readonly sandbox?: "none" | "readonly" | Omit<SandboxSpec, "cwd">
/** No network inside the sandbox (only meaningful with `sandbox`). */
readonly offline?: boolean
}
const DEFAULT_TIMEOUT_S: Record<string, number> = { UserPromptSubmit: 30 }
@@ -69,6 +87,7 @@ async function runCommand(
? [substitute(handler.command, context.root), ...handler.args.map((arg) => substitute(arg, context.root))]
: ["sh", "-c", handler.command]
let result: RunResult
const offline = handler["x-es-offline"] ?? context.offline
try {
result = await run(argv, {
cwd: context.root,
@@ -77,6 +96,10 @@ async function runCommand(
inheritEnv: true,
env: { CLAUDE_PROJECT_DIR: context.root, ES_HOOK_EVENT: context.event },
maxOutputBytes: 1_000_000,
// The sandbox knob, honored per handler (#265): absent runs unsandboxed
// with the parent environment, as Claude Code does.
...(context.sandbox !== undefined && context.sandbox !== "none" ? { sandbox: context.sandbox } : {}),
...(offline !== undefined ? { offline } : {}),
...(context.signal ? { signal: context.signal } : {}),
})
} catch (error) {
packages/core/src/hooks/spec.ts
+2
View File
@@ -35,6 +35,8 @@ export const CommandHookSchema = z.looseObject({
async: z.boolean().optional(),
asyncRewake: z.boolean().optional(),
shell: z.enum(["bash", "powershell"]).optional(),
/** No network for this command when run sandboxed (#265; needs the engine sandbox). */
"x-es-offline": z.boolean().optional(),
...common,
})
packages/core/src/lsp/install.ts
+4 -1
View File
@@ -1,7 +1,10 @@
// Pinned, checksummed, ask-once installs of language servers. A human
// consents once per server (CLI or TUI); the installer fetches the exact npm
// tarball, verifies its sha512 against the pinned integrity, and installs it
// with scripts disabled into the user-global state dir.
// with scripts disabled into the user-global state dir. Installs are not
// bwrap-sandboxed (the fetch needs the network): consent + SRI pins +
// --ignore-scripts + a scrubbed environment is the whole mitigation stack,
// pinned by the installer tests (#275).
import { createHash } from "node:crypto"
import { mkdir, mkdtemp, readFile, rm } from "node:fs/promises"
import { tmpdir, userInfo } from "node:os"
packages/core/src/lsp/registry.ts
+3 -2
View File
@@ -1,5 +1,6 @@
// Built-in language-server registry plus OpenCode v2's `lsp` config key
// (accepted but not run by v2 itself). User config overrides or disables
// Built-in language-server registry plus the `lsp` key from v2's
// `opencode.json` (accepted but not run by v2 itself; not an ES config key —
// ES config layers strict-reject it). User config overrides or disables
// built-ins and adds servers; `lsp: false` turns LSP off entirely.
import { readFile } from "node:fs/promises"
import { homedir } from "node:os"
packages/core/src/ops/tools.ts
+9 -4
View File
@@ -16,7 +16,7 @@ import { formatReport, runGates } from "../gates/engine.ts"
import { factoryLayout } from "../layout.ts"
import type { GateFinding } from "../schema/gates.ts"
import { relativeTo } from "../trust/paths.ts"
import { writeJson } from "../util/fs.ts"
import { withLock, writeJson } from "../util/fs.ts"
export type EsToolContext = { readonly agent: string; readonly sessionID?: string; readonly signal?: AbortSignal }
export interface EsToolDef {
@@ -121,9 +121,14 @@ export function esTools(ops: OpsContext): EsToolDef[] {
`Not ready for approval: ${error.message}${missing ? " Run /grill (or `es factory begin` + grill flow) to record the frontier, then request approval again." : ""}`,
)
})
const pending = (await pendingApprovals(root)).filter((item) => item.stage !== stage)
pending.push({ stage, requestedBy: context.agent, at: new Date().toISOString(), ...(note ? { note } : {}) })
await writeJson(factoryLayout(root).pending, pending)
const pendingFile = factoryLayout(root).pending
// Locked against concurrent requests and the human approve path
// (which clears under the same state lock): #219 lost-update.
await withLock(factoryLayout(root).state, async () => {
const pending = (await pendingApprovals(root)).filter((item) => item.stage !== stage)
pending.push({ stage, requestedBy: context.agent, at: new Date().toISOString(), ...(note ? { note } : {}) })
await writeJson(pendingFile, pending)
})
await appendAuditEntry(root, {
actor: `agent:${context.agent}`,
action: "approval.request",
packages/core/src/queereye/bundle.ts
+4 -1
View File
@@ -153,8 +153,11 @@ export async function verifyCiteGate(receipt: string, root: string): Promise<str
if (!new RegExp(`\\[VERIFIED:\\s*${hash}\\s*\\]`, "i").test(receipt))
errors.push(`cite-gate: receipt missing structured dossier hash [VERIFIED: ${hash}]`)
// QA gate: contrast probes.
// QA gate: contrast probes. An empty probe set verifies vacuously, so
// missing probes are an error here, not a green pass (#270).
const probes = (await readProbes(root).catch(() => undefined)) ?? []
if (probes.length === 0)
errors.push("cite-gate QA contrast: probes.json is missing or empty (the contrast gate needs at least one pair)")
for (const row of probes) for (const error of row.errors) errors.push(`cite-gate QA contrast ${row.name}: ${error}`)
for (const row of probes)
if (!row.pass && row.errors.length === 0) errors.push(`cite-gate QA contrast ${row.name}: pair does not pass`)
packages/core/src/queereye/ops.ts
+8 -3
View File
@@ -32,7 +32,7 @@ import {
} from "./ledger.ts"
import { compileToCssVars, renderGuide } from "./render.ts"
import { AXES, type Axis, ContradictionError, FORMS, SlotLoop, SmuggledValueError, VagueAnswerError } from "./slots.ts"
import { EXEMPLARS, renderSpecMd, specFilename, validateAll } from "./specs.ts"
import { EXEMPLARS, renderedSlotsOfSpecMd, renderSpecMd, specFilename, validateAll } from "./specs.ts"
import {
checkDrift,
designPaths,
@@ -259,8 +259,13 @@ export function designTools(context: DesignOpsContext): EsToolDef[] {
}
for (const [name, spec] of Object.entries(EXEMPLARS)) {
for (const error of validateTuiCoverage(spec)) drift.push(`${name}: ${error}`)
for (const error of specVsRenderCheck(spec, Object.keys(spec.slots ?? {}).sort(), isWebrefFresh()))
drift.push(error)
// The slot-fill leg reads the actual render output from disk: a
// spec fed its own slots could never fail (#270). A missing file
// is already drift above; null keeps the gate honest, never green.
const renderedFile = bundlePath(root, specFilename(name))
const onDisk = await readFile(renderedFile, "utf8").catch(() => undefined)
const renderedSlots = onDisk === undefined ? null : renderedSlotsOfSpecMd(onDisk)
for (const error of specVsRenderCheck(spec, renderedSlots, isWebrefFresh())) drift.push(error)
}
if (drift.length) throw new ToolRefusal(`Phase-02 drift:\n- ${drift.join("\n- ")}`)
return "Phase-02 check: specs, webref, csf and tui-notes byte-match the renderers; coverage, spec-vs-render, freshness and the play suite pass."
packages/core/src/queereye/specs.ts
+18
View File
@@ -1325,6 +1325,24 @@ export function specFilename(name: string): string {
return `components/${safe}.md`
}
/**
* Slot names carried by a rendered spec markdown's `## Slots` section (the
* actual render output: `- \`name\`: description` lines). The phase-02 check
* reads these from the file on disk instead of feeding each spec its own
* slots (which could never fail); a hand-removed slot line diverges here.
* The section is cut with `indexOf`/`trimEnd`, not a lazy `[\s\S]*?…\s*$`
* scan: that regex was quadratic on a long whitespace run in the file.
*/
export function renderedSlotsOfSpecMd(markdown: string): string[] {
const header = "## Slots\n"
const start = markdown.indexOf(header)
if (start === -1) return []
const rest = markdown.slice(start + header.length)
const next = rest.indexOf("\n## ")
const section = next === -1 ? rest.trimEnd() : rest.slice(0, next)
return [...section.matchAll(/^-\s+`([^`]+)`: /gm)].map((match) => match[1]!)
}
// ---------------------------------------------------------------------------
// render_spec_md (d66328c:runner/queereye/specs.py)
// ---------------------------------------------------------------------------
packages/core/src/queereye/store.ts
+5 -3
View File
@@ -167,13 +167,15 @@ export async function checkDrift(root: string): Promise<DriftReport> {
const paths = designPaths(root)
const tokens = await readTokens(root)
if (tokens === undefined) return { drifted: [], missing: [path.basename(paths.tokens)], tokenErrors: [] }
const probes = (await readProbes(root)) ?? []
// A missing probes.json is a gap, never a silent empty-probes render (#270):
// without probes the contrast QA below would pass vacuously.
const probes = await readProbes(root)
const missing: string[] = probes === undefined ? [path.basename(paths.probes)] : []
const expected = new Map<string, string>([
[paths.css, compileToCssVars(tokens)],
[paths.guide, renderGuide(tokens, probes)],
[paths.guide, renderGuide(tokens, probes ?? [])],
])
const drifted: string[] = []
const missing: string[] = []
for (const [file, content] of expected) {
const existing = await readFile(file, "utf8").catch(() => undefined)
if (existing === undefined) missing.push(path.basename(file))
packages/core/src/queereye/tokens.ts
+4 -7
View File
@@ -23,7 +23,7 @@ export type Json = string | number | boolean | null | Json[] | { [key: string]:
export type TokenTree = Record<string, unknown>
export const isToken = (node: unknown): node is Record<string, Json | undefined> =>
typeof node === "object" && node !== null && !Array.isArray(node) && "$value" in node
typeof node === "object" && node !== null && !Array.isArray(node) && Object.hasOwn(node, "$value")
export const isAliasValue = (value: unknown): value is string =>
typeof value === "string" && ALIAS_FULL.test(value.trim())
@@ -83,12 +83,9 @@ function resolveType(node: unknown, inherited?: string): string | undefined {
function walkPath(root: unknown, ref: string): unknown {
let node: unknown = root
for (const segment of ref.split(".")) {
if (
typeof node !== "object" ||
node === null ||
Array.isArray(node) ||
!(segment in (node as Record<string, unknown>))
)
// Own-property only: `in` walks the prototype chain, so {__proto__},
// {constructor} and friends would resolve against Object.prototype (#266).
if (typeof node !== "object" || node === null || Array.isArray(node) || !Object.hasOwn(node, segment))
throw new UnknownAliasError(`alias {${ref}} does not resolve: missing "${segment}"`)
node = (node as Record<string, unknown>)[segment]
}
packages/core/src/research/auditor.ts
+5 -2
View File
@@ -2,13 +2,16 @@
//
// Every claim line (a bullet, numbered item or prose paragraph line) must carry
// at least one tag:
// [VERIFIED: sha256:<hash> "verbatim quote"] — quote must be in the cached source
// [VERIFIED: sha256:<hash> "quote, verbatim after documented normalisation"] — quote must be in the cached source
// [INFERRED: reasoning] — reasoning from verified facts
// [HYPOTHESIS: how to test it]
// [NEGATIVE_KNOWLEDGE: what was searched]
// The auditor verifies quotes against the content-addressed cache, flags
// untagged and ungrounded claims, can prune them into a "Pruned claims"
// section, and reports coverage counts (no composite score).
// section, and reports coverage counts (no composite score). It marks,
// prunes and refuses failing claims and dossiers; it never re-tags a claim's
// kind (no VERIFIED→INFERRED demotion: that verb belongs to the
// scout/harvest licence policy).
import type { SourceCache } from "./cache.ts"
import { verifyQuote } from "./quote.ts"
packages/core/src/research/cache.ts
+36 -11
View File
@@ -49,28 +49,53 @@ const MAX_SOURCE_BYTES = 400_000
export class SourceCache {
/**
* When `stateDir` is set, entries are sealed with the engine key on write
* (#52) and unsealed entries are refused on read. Without it the cache keeps
* its legacy behaviour (tests and auxiliary readers).
* Entries are sealed with the engine key on write (#52) and unsealed
* entries are refused on read. `stateDir` is required: there is no
* legacy unsealed mode (#251) — tests that need untrusted bytes write
* them to disk directly instead of constructing a cache without a key.
*/
constructor(
readonly dir: string,
readonly stateDir?: string,
readonly stateDir: string,
) {}
private async seal(meta: SourceMeta): Promise<SourceMeta> {
if (!this.stateDir) return meta
const key = await ensureEngineKey(this.stateDir)
return { ...meta, seal: sealMeta(meta as unknown as Record<string, unknown>, key) }
}
private async sealed(meta: SourceMeta): Promise<boolean> {
if (!this.stateDir) return true
// Never mint a key to verify: without the engine's key nothing verifies (#59).
const key = await readEngineKey(this.stateDir)
return key !== undefined && verifyMetaSeal(meta as unknown as Record<string, unknown>, key)
}
private indexFile = () => path.join(this.dir, "index.json")
private indexSealFile = () => path.join(this.dir, "index.seal")
/**
* The URL index sealed with the engine key (#251.2): without integrity the
* index could remap a URL onto another sealed entry's bytes and `byUrl`
* would serve them. Writes store an HMAC sidecar; reads refuse a
* present-but-forged sidecar and keep serving a missing one (caches
* written before sidecars stay readable). Per-entry seals still apply at
* `get`, so the sidecar only guards the mapping, never the bytes.
*/
private async writeIndex(index: Record<string, string>): Promise<void> {
await atomicWrite(this.indexFile(), `${JSON.stringify(index, null, 2)}\n`)
const key = await ensureEngineKey(this.stateDir)
await atomicWrite(this.indexSealFile(), `${sealMeta(index as unknown as Record<string, unknown>, key)}\n`)
}
private async readIndex(): Promise<Record<string, string>> {
const index = (await readJson<Record<string, string>>(this.indexFile())) ?? {}
const seal = (await readFile(this.indexSealFile(), "utf8").catch(() => undefined))?.trim()
if (seal === undefined || seal === "") return index
const key = await readEngineKey(this.stateDir)
if (key === undefined) return index
return verifyMetaSeal({ ...index, seal }, key) ? index : {}
}
async put(input: {
url: string
text: string
@@ -98,9 +123,9 @@ export class SourceCache {
const file = path.join(this.dir, `${hash}.md`)
await atomicWrite(file, text)
await atomicWrite(path.join(this.dir, `${hash}.json`), `${JSON.stringify(meta, null, 2)}\n`)
const index = (await readJson<Record<string, string>>(path.join(this.dir, "index.json"))) ?? {}
const index = await this.readIndex()
index[canonicalUrl(input.url) ?? input.url] = hash
await atomicWrite(path.join(this.dir, "index.json"), `${JSON.stringify(index, null, 2)}\n`)
await this.writeIndex(index)
return { meta, text, path: file }
}
@@ -125,7 +150,7 @@ export class SourceCache {
}
async byUrl(url: string): Promise<CachedSource | undefined> {
const index = (await readJson<Record<string, string>>(path.join(this.dir, "index.json"))) ?? {}
const index = await this.readIndex()
// Canonical key first; raw keys are what 1.0 caches recorded.
const hash = index[canonicalUrl(url) ?? url] ?? index[url]
return hash ? this.get(hash) : undefined
@@ -147,9 +172,9 @@ export class SourceCache {
removed.push(hash)
}
if (removed.length) {
const index = (await readJson<Record<string, string>>(path.join(this.dir, "index.json"))) ?? {}
const index = await this.readIndex()
for (const [url, hash] of Object.entries(index)) if (removed.includes(hash)) delete index[url]
await atomicWrite(path.join(this.dir, "index.json"), `${JSON.stringify(index, null, 2)}\n`)
await this.writeIndex(index)
}
return removed
}
packages/core/src/research/ops.ts
+2 -2
View File
@@ -155,7 +155,7 @@ export function researchTools(context: ResearchOpsContext): EsToolDef[] {
{
name: "es_research_fetch",
description:
'Fetch a URL, cache its text content-addressed, and return the sha256 to cite: [VERIFIED: sha256:<hash> "verbatim quote"]. A fresh cached copy is served without a network request (refresh:true fetches again).',
'Fetch a URL, cache its text content-addressed, and return the sha256 to cite: [VERIFIED: sha256:<hash> "quote, verbatim after documented normalisation"]. A fresh cached copy is served without a network request (refresh:true fetches again).',
input: object(
{
url: { type: "string" },
@@ -223,7 +223,7 @@ export function researchTools(context: ResearchOpsContext): EsToolDef[] {
{
name: "es_research_audit",
description:
"Audit a research artifact: every claim needs an epistemic tag, and every VERIFIED quote must appear verbatim in its cached source. prune:true moves failing claims to a 'Pruned claims' section.",
"Audit a research artifact: every claim needs an epistemic tag, and every VERIFIED quote must appear verbatim after documented normalisation in its cached source. prune:true moves failing claims to a 'Pruned claims' section.",
input: object({
path: { type: "string", description: "Default .factory/research/REPORT.md" },
prune: { type: "boolean" },
packages/core/src/research/quote.ts
+9 -7
View File
@@ -1,10 +1,12 @@
// Pure verbatim-quote verification. A quote matches when, after normalising
// whitespace and typographic punctuation (curly quotes, dashes, NBSP), it
// appears in the source exactly. An ellipsis ("..." or "…") splits a prose
// quote into fragments that must appear in order, and each fragment must be
// evidence on its own (MIN_QUOTE characters). Code excerpts are one contiguous
// span (verifySpan): "..." there is code, not an ellipsis. No fuzzy matching:
// a quote is either in the source or it is not.
// Quote verification that is verbatim after documented normalisation. A quote
// matches when, after normalising (NFKC, typography folds for curly quotes,
// dashes and spaces including NBSP, markdown link and emphasis stripping,
// whitespace collapsing), it appears in the source exactly. An ellipsis
// ("..." or "…") splits a prose quote into fragments that must appear in
// order, and each fragment must be evidence on its own (MIN_QUOTE
// characters). Code excerpts are one contiguous span (verifySpan): "..."
// there is code, not an ellipsis. No fuzzy matching: a normalised quote is
// either in the normalised source or it is not.
/** Shortest quote, and shortest ellipsis fragment, that counts as evidence (normalised characters). */
export const MIN_QUOTE = 12
packages/core/src/research/tier.ts
+6 -8
View File
@@ -2,7 +2,8 @@
// source sits on (preprint, peer-reviewed, docs, press), so domain packs can
// weight verified claims by source quality. Rules are host-based and
// auditable; unknown hosts stay tierless and fall back to `__default__`.
// Packs may add a `tierDomains` override, which wins over the built-ins.
// Callers may pass a `tierDomains` override map, which wins over the
// built-ins.
import { DOC_DOMAINS } from "./url.ts"
const PREPRINT = ["arxiv.org", "biorxiv.org", "medrxiv.org", "ssrn.com"] as const
@@ -36,7 +37,7 @@ const matches = (host: string, domain: string): boolean => host === domain || ho
/**
* Classify a source URL into a tier, or undefined when no rule fires.
* `overrides` (a pack's `tierDomains`) is checked first and wins.
* `overrides` (a caller's `tierDomains` map) is checked first and wins.
*/
export function classifySourceTier(
url: string,
@@ -49,12 +50,9 @@ export function classifySourceTier(
if (domains.some((domain) => matches(host, domain.toLowerCase()))) return tier
if (PREPRINT.some((domain) => matches(host, domain))) return "PREPRINT"
if (PEER_REVIEWED.some((domain) => matches(host, domain))) return "PEER_REVIEWED"
if (
(DOC_DOMAINS as readonly string[]).some((domain) => matches(host, domain)) ||
host.startsWith("docs.") ||
host.startsWith("developer.")
)
return "TECHNICAL_DOCUMENTATION"
// Only curated documentation hosts count (#252): a bare docs./developer.
// prefix rule would let docs.<attacker> claim curated weight.
if ((DOC_DOMAINS as readonly string[]).some((domain) => matches(host, domain))) return "TECHNICAL_DOCUMENTATION"
if (host.endsWith(".gov")) return "PRIMARY_PRESS"
return undefined
}
packages/core/src/schema/brainstorm.ts
+3 -2
View File
@@ -1,6 +1,7 @@
// Brainstorm artifacts: the plan (lenses and caps), recorded ideas, rubric
// scores, and the completed result (duplicate collapsing, ranking, and a
// diversified shortlist with a forced outlier slot).
// diversified shortlist with a forced outlier whenever the shortlist has
// ≥2 slots).
import { z } from "zod"
export const BRAINSTORM_VERSION = 1 as const
@@ -21,7 +22,7 @@ export const BrainstormPlanSchema = z.object({
ideasPerLens: z.number().int().min(1).max(10),
/** Hard cap of characters per idea text. */
maxIdeaChars: z.number().int().min(120),
/** Shortlist size, including the forced outlier slot. */
/** Shortlist size, including the forced outlier slot (whenever the shortlist has ≥2 slots). */
shortlistSize: z.number().int().min(2).max(12),
/** Similarity at or above which an idea is collapsed as a duplicate. */
dedupeThreshold: z.number().min(0.5).max(0.95),
packages/core/src/schema/codeaudit.ts
+1 -1
View File
@@ -19,7 +19,7 @@ export const AuditFindingSchema = z
/** POSIX path relative to the audited tree's root. */
file: z.string().min(1),
lines: z.tuple([z.number().int().positive(), z.number().int().positive()]),
/** Verbatim code from those lines (at least 12 characters). */
/** Verbatim code from those lines (the witness enforces the 12-char floor via verifySpan; the schema keeps min(1) so the witness reports the reason). */
excerpt: z.string().min(1),
detail: z.string().min(8),
remediation: z.string().optional(),
packages/core/src/schema/domain.ts
-5
View File
@@ -24,11 +24,6 @@ export const DomainPackSchema = z
bannedDomains: z.array(z.string().min(1)).default([]).describe("Source URLs containing any of these are rejected."),
mandatoryTags: z.array(ClaimTagSchema).default([]).describe("Every claim must carry one of these tags."),
retractionPolicy: RetractionPolicySchema.default("standard"),
/** Per-tier host overrides for source classification (checked before the built-in table). */
tierDomains: z
.record(z.string(), z.array(z.string().min(1)))
.optional()
.describe("Pack host overrides by tier for source classification."),
/** Legacy metadata; carried so ported pack files round-trip. */
exportTemplate: z.string().optional(),
})
packages/core/src/schema/scout.ts
+15
View File
@@ -37,6 +37,21 @@ export const ScoutAdvisorySchema = z.object({
})
export type ScoutAdvisory = z.infer<typeof ScoutAdvisorySchema>
/** The OSV coordinates a candidate's advisories were actually queried at (#257). */
export const ScoutAdvisoryQuerySchema = z.object({
ecosystem: z.string(),
package: z.string(),
version: z.string().optional(),
})
export type ScoutAdvisoryQuery = z.infer<typeof ScoutAdvisoryQuerySchema>
/** The per-candidate advisories artifact: what was queried, and what came back. */
export const ScoutAdvisoriesFileSchema = z.object({
query: ScoutAdvisoryQuerySchema,
advisories: z.array(ScoutAdvisorySchema),
})
export type ScoutAdvisoriesFile = z.infer<typeof ScoutAdvisoriesFileSchema>
export const ScoutAssessmentSchema = z.object({
candidate: z.string(),
stars: z.number().int().nonnegative().optional(),
packages/core/src/scout/ops.ts
+62 -7
View File
@@ -18,7 +18,7 @@ import { DEFAULT_LICENSE_WHITELIST, type HarvestProfile } from "../schema/harves
import { SCOUT_VERSION, type ScoutAdvisory, type ScoutAssessment, ScoutAssessmentSchema } from "../schema/scout.ts"
import { relativeTo } from "../trust/paths.ts"
import { atomicWrite } from "../util/fs.ts"
import { renderScoutReport } from "./report.ts"
import { type AdvisoryCheck, renderScoutReport } from "./report.ts"
import {
readAssessments,
readScoutAdvisories,
@@ -256,7 +256,29 @@ export function scoutTools(context: ScoutOpsContext): EsToolDef[] {
throw new ToolRefusal(
`${candidate.id} is not a registry package: give the ecosystem (${Object.values(OSV_ECOSYSTEMS).join(", ")}) and the package name it publishes.`,
)
const version = input.version ?? profile?.release
// A registry candidate must be queried at its own coordinates (#257):
// a hand-supplied wrong ecosystem/package returns a clean bill for
// code that was never checked.
if (source.kind === "registry") {
const derived = { ecosystem: OSV_ECOSYSTEMS[source.registry], name: source.name }
const sameEcosystem =
input.ecosystem === undefined || String(input.ecosystem).toLowerCase() === derived.ecosystem.toLowerCase()
const samePackage = input.package === undefined || String(input.package) === derived.name
if (!sameEcosystem || !samePackage)
throw new ToolRefusal(
`Advisories for ${candidate.id} must be queried for the scanned source (${derived.ecosystem}:${derived.name}); ` +
`got ${input.ecosystem ?? ecosystem}:${input.package ?? name}. Query the candidate's own package, not another one.`,
)
}
// The version is pinned to the scanned release when one is known
// (#257): pointing the query at a fixed release hides its advisories.
const release = profile?.release
if (release !== undefined && input.version !== undefined && String(input.version) !== release)
throw new ToolRefusal(
`Version ${input.version} does not match the scanned release ${release} of ${candidate.id}. ` +
`Query ${release}, or omit the version to check every release.`,
)
const version = input.version ?? release
const result = await queryOsv(
{ ecosystem, name: String(name), ...(version ? { version: String(version) } : {}) },
cache,
@@ -275,7 +297,14 @@ export function scoutTools(context: ScoutOpsContext): EsToolDef[] {
fixed: [...advisory.fixed],
source: result.sha256,
}))
await writeScoutAdvisories(root, candidate.id, advisories)
await writeScoutAdvisories(root, candidate.id, {
query: {
ecosystem,
package: String(name),
...(version !== undefined ? { version: String(version) } : {}),
},
advisories,
})
return [
`${advisories.length} OSV advisor${advisories.length === 1 ? "y" : "ies"} for ${ecosystem}:${name}${version ? `@${version}` : ""}, cached as sha256:${result.sha256}.`,
...advisories.map(
@@ -337,7 +366,7 @@ export function scoutTools(context: ScoutOpsContext): EsToolDef[] {
...(input.lastRelease ? { lastRelease: String(input.lastRelease) } : {}),
...(input.releasesPerYear !== undefined ? { releasesPerYear: Number(input.releasesPerYear) } : {}),
maintenance: input.maintenance,
advisories: (await readScoutAdvisories(root, candidate.id).catch(() => undefined)) ?? [],
advisories: (await readScoutAdvisories(root, candidate.id).catch(() => undefined))?.advisories ?? [],
proposal: input.proposal,
rationale: String(input.rationale ?? ""),
...(input.blueprint ? { blueprint: String(input.blueprint) } : {}),
@@ -351,7 +380,7 @@ export function scoutTools(context: ScoutOpsContext): EsToolDef[] {
{
name: "es_scout_complete",
description:
"Scout only: when every candidate is scanned and recorded, recompute the verdicts (adopt survives only for a verified, whitelisted, permissive license), flag severe advisories, rank, and write .factory/scout/REPORT.md, scout.json and dossier.json.",
"Scout only: when every candidate is scanned, recorded, and checked for advisories (queried, or an explicit NEGATIVE_KNOWLEDGE skip), recompute the verdicts (adopt survives only for a verified, whitelisted, permissive license), flag severe advisories, rank, and write .factory/scout/REPORT.md, scout.json and dossier.json.",
input: object({}),
execute: async (_input, toolContext) => {
await guard(toolContext.agent)
@@ -359,6 +388,7 @@ export function scoutTools(context: ScoutOpsContext): EsToolDef[] {
const assessments = new Map((await readAssessments(root)).map((item) => [item.candidate, item]))
const profiles = new Map<string, HarvestProfile>()
const evidence: CandidateEvidence[] = []
const checks = new Map<string, AdvisoryCheck>()
const missing: string[] = []
for (const candidate of current.candidates) {
const profile = await readScoutProfile(root, candidate.id).catch(() => undefined)
@@ -367,7 +397,32 @@ export function scoutTools(context: ScoutOpsContext): EsToolDef[] {
if (!assessment) missing.push(`${candidate.id}: record it (es_scout_record)`)
if (!profile || !assessment) continue
profiles.set(candidate.id, profile)
const advisories = (await readScoutAdvisories(root, candidate.id).catch(() => undefined)) ?? []
// The advisories step is mandatory (#257): skipping it, querying a
// wrong package, or pointing the version at a fixed release all
// used to yield a clean adopt with zero citations. Either the
// query ran (its coordinates are pinned at query time), or the
// assessment records explicit NEGATIVE_KNOWLEDGE that it was
// skipped — a silent skip refuses here.
const advisoryFile = await readScoutAdvisories(root, candidate.id).catch(() => undefined)
const skipped =
!advisoryFile &&
assessment.claims.some(
(claim) =>
claim.tag === "NEGATIVE_KNOWLEDGE" &&
/advisori/i.test([claim.statement, claim.query ?? "", claim.finding ?? ""].join(" ")),
)
if (!advisoryFile && !skipped)
missing.push(
`${candidate.id}: query its advisories (es_scout_advisories), or record explicit NEGATIVE_KNOWLEDGE that the step was skipped`,
)
if (!advisoryFile && !skipped) continue
const advisories = advisoryFile?.advisories ?? []
checks.set(
candidate.id,
advisoryFile
? { query: advisoryFile.query, skipped: false, count: advisories.length }
: { skipped: true, count: 0 },
)
evidence.push({
id: candidate.id,
name: candidate.name,
@@ -398,7 +453,7 @@ export function scoutTools(context: ScoutOpsContext): EsToolDef[] {
throw new ToolRefusal(`The scout dossier could not be written: ${error.message}`)
})
await writeScoutResult(root, result)
await atomicWrite(paths.report, renderScoutReport(current, result, profiles, assessments))
await atomicWrite(paths.report, renderScoutReport(current, result, profiles, assessments, checks))
return [
`Scout complete: ${current.objective}`,
...verdicts.map(
packages/core/src/scout/report.ts
+24 -1
View File
@@ -1,7 +1,14 @@
// REPORT.md for a finished scout: the competitive matrix, the risks the
// red-team found, clean-room blueprints, and the cited claims. Deterministic.
import type { HarvestProfile } from "../schema/harvest.ts"
import type { ScoutAssessment, ScoutPlan, ScoutResult } from "../schema/scout.ts"
import type { ScoutAdvisoryQuery, ScoutAssessment, ScoutPlan, ScoutResult } from "../schema/scout.ts"
/** What the report says about one candidate's advisories step (#257). */
export interface AdvisoryCheck {
readonly query?: ScoutAdvisoryQuery | undefined
readonly skipped: boolean
readonly count: number
}
const cell = (value: unknown) => String(value ?? "—").replaceAll("|", "\\|")
@@ -10,6 +17,7 @@ export function renderScoutReport(
result: ScoutResult,
profiles: ReadonlyMap<string, HarvestProfile>,
assessments: ReadonlyMap<string, ScoutAssessment>,
checks: ReadonlyMap<string, AdvisoryCheck>,
): string {
const lines = [
`# Scout: ${plan.objective}`,
@@ -39,6 +47,21 @@ export function renderScoutReport(
`- **${verdict.name}**: ${notes.length ? notes.join("; ") : "none found"}${verdict.attribution ? ` · attribution ${verdict.attribution}` : ""}`,
)
}
// What the advisories step actually checked (#257): the queried
// coordinates, or the explicit skip. A clean adopt with no line here is
// impossible — complete refuses the silent skip.
lines.push("", "## Advisory checks", "")
for (const verdict of result.verdicts) {
const check = checks.get(verdict.candidate)
if (!check || check.skipped)
lines.push(`- **${verdict.name}**: advisories not queried; the assessment records explicit NEGATIVE_KNOWLEDGE.`)
else {
const at = check.query?.version ? `@${check.query.version}` : ""
lines.push(
`- **${verdict.name}**: OSV ${check.query?.ecosystem}:${check.query?.package}${at} checked, ${check.count} ${check.count === 1 ? "advisory" : "advisories"} cited.`,
)
}
}
const blueprints = result.verdicts.filter(
(verdict) => verdict.verdict === "clean-room" && assessments.get(verdict.candidate)?.blueprint,
)
packages/core/src/scout/store.ts
+14 -7
View File
@@ -6,8 +6,8 @@ import path from "node:path"
import { factoryLayout } from "../layout.ts"
import { type HarvestProfile, HarvestProfileSchema } from "../schema/harvest.ts"
import {
type ScoutAdvisory,
ScoutAdvisorySchema,
type ScoutAdvisoriesFile,
ScoutAdvisoriesFileSchema,
type ScoutAssessment,
ScoutAssessmentSchema,
type ScoutPlan,
@@ -56,13 +56,20 @@ export async function startScoutRun(root: string, plan: ScoutPlan): Promise<void
export const writeScoutProfile = (root: string, profile: HarvestProfile) =>
writeJson(scoutPaths(root).profile(profile.id), HarvestProfileSchema.parse(profile))
export async function readScoutAdvisories(root: string, id: string): Promise<ScoutAdvisory[] | undefined> {
const raw = await readJson<unknown[]>(scoutPaths(root).advisories(id))
return raw === undefined ? undefined : raw.map((item) => ScoutAdvisorySchema.parse(item))
export async function readScoutAdvisories(root: string, id: string): Promise<ScoutAdvisoriesFile | undefined> {
const raw = await readJson<unknown>(scoutPaths(root).advisories(id))
if (raw === undefined) return undefined
try {
return ScoutAdvisoriesFileSchema.parse(raw)
} catch (error) {
throw new Error(
`invalid ${scoutPaths(root).advisories(id)}: ${error instanceof Error ? error.message : String(error)}`,
)
}
}
export const writeScoutAdvisories = (root: string, id: string, advisories: readonly ScoutAdvisory[]) =>
writeJson(scoutPaths(root).advisories(id), advisories)
export const writeScoutAdvisories = (root: string, id: string, file: ScoutAdvisoriesFile) =>
writeJson(scoutPaths(root).advisories(id), ScoutAdvisoriesFileSchema.parse(file))
export async function readAssessments(root: string): Promise<ScoutAssessment[]> {
const raw = await readJson<unknown[]>(scoutPaths(root).assessments)
packages/core/src/trust/control.ts
+8 -1
View File
@@ -87,7 +87,14 @@ export function controlClass(relative: string): ControlClass | undefined {
export const isControlPath = (relative: string) => controlClass(relative) !== undefined
/** Files whose hashes the baseline pins (existing, tracked control artifacts). */
/**
* Files whose hashes the baseline pins (existing, tracked control artifacts).
* Project-root only by design (#240.3): the user-global layer
* (globalConfigFile, under XDG_CONFIG_HOME) is machine-scoped and shared
* across projects — pinning it here would drift whenever the human edits
* global settings for another project. Global writes stay guarded by the
* human-only `es config set --global` gate instead.
*/
const PINNED = (root: string) => {
const layout = factoryLayout(root)
return [layout.gates, layout.frontier, layout.config]
packages/core/src/trust/policy.ts
+4 -3
View File
@@ -102,8 +102,9 @@ export function evaluateWrite(context: PolicyContext, agentId: string | undefine
if (!spec) return allow
if (relative === undefined) return deny(`Factory seat "${spec.id}" may not write outside the project: ${absolute}`)
// REPORT.md has one writer per run mode (#111): the synthesizer is the
// only writer in research runs, the factory in build runs (AGENTS.md).
// REPORT.md has one writer scope per run mode (#111, #220): factory-docs
// seats (the factory and the manager share the scope) in build runs, the
// research synthesizer alone in research runs (AGENTS.md).
// The mode comes from the sealed run state; when it cannot be read both
// writers are denied (fail closed).
if (relative === ".factory/research/REPORT.md") {
@@ -329,7 +330,7 @@ const MENTION_HINT =
" If it only names the verb in text (a here-doc, a commit message, a `--body` argument), the rule still reads a call: write the text to a file with the edit tool and pass the file (`gh … --body-file`, `git commit -F`)."
const CONTROL_MENTION =
/\.factory\/(gates\.json|config\.json|frontier\.json|waivers|approvals|runtime|STOP|git-hooks|claims\b|audits\b|research\/(sources\b|(coverage|dossier|brief\.pcrb)\.json)|(brainstorm|harvest|design|scout)\/[^\s'"]*\.json)|\.git\/(config|hooks)|\.opencode\/(hooks\.json|plugins|opencode\.jsonc?)|\.claude\/settings|opencode\.jsonc?\b/i
/\.factory\/(gates\.json|config\.json|frontier\.json|waivers|approvals|runtime|STOP|git-hooks|claims\b|audits\b|improve\b|research\/(sources\b|(coverage|dossier|brief\.pcrb)\.json)|(brainstorm|harvest|design|scout)\/(?!notes\/)[^\s'"]*)|\.git\/(config|hooks)|\.opencode\/(hooks\.json|plugins|opencode\.jsonc?)|\.claude\/settings|opencode\.jsonc?\b/i
/** Any mention of the factory dir (a `cd .factory` reaches control files without naming them). */
const FACTORY_MENTION = /(^|[\s/=:])\.factory(\/|[\s;|&]|$)/i
/** Any mention of fleet isolation state (task worktrees live under `.fleet/`). */
packages/core/src/trust/sandbox.ts
+36 -14
View File
@@ -40,6 +40,13 @@ export interface SandboxProbe {
readonly exists: (file: string) => boolean
/** bwrap supports --overlay-src/--tmp-overlay (0.10+). */
readonly overlay: boolean
/**
* Per-directory overlay support (#231): an overlay mount can fail on one
* filesystem while it succeeds on another (a btrfs home vs tmpfs /tmp), so
* a probe that only tries /tmp misreports the mount that matters. Absent:
* `overlay` answers for every directory.
*/
readonly overlayFor?: (dir: string) => boolean
readonly uid?: number
}
@@ -106,6 +113,9 @@ export function sandboxArgv(
const writableRoot = spec.kind === "user" || spec.kind === "seat"
const writable = spec.kind === "programmer" || spec.kind === "gate" ? spec.writable : undefined
const protectedRel = user ? PROTECTED_ALWAYS : [...PROTECTED_ALWAYS, ...PROTECTED_FOR_SEATS]
// Graceful degradation (#231): when the per-dir probe refuses a mount, the
// overlay args are omitted instead of emitting a mount that fails at runtime.
const overlayFor = (dir: string) => probe.overlayFor?.(dir) ?? probe.overlay
const protectedIn = (dir: string, names: readonly string[]) =>
names.map((name) => path.join(dir, name)).filter((file) => probe.exists(file))
const args: string[] = ["bwrap", writableRoot ? "--bind" : "--ro-bind", "/", "/", "--dev", "/dev", "--proc", "/proc"]
@@ -116,9 +126,9 @@ export function sandboxArgv(
// A private /tmp; projects and homes that live under /tmp are bound back in.
args.push("--tmpfs", "/tmp")
// Tool caches under home stay usable; writes there are thrown away.
if (probe.overlay && probe.exists(home)) args.push("--overlay-src", home, "--tmp-overlay", home)
if (overlayFor(home) && probe.exists(home)) args.push("--overlay-src", home, "--tmp-overlay", home)
else if (inside("/tmp", home) && probe.exists(home)) roBind(home)
if (inside("/tmp", spec.root) && !(probe.overlay && inside(home, spec.root))) roBind(spec.root)
if (inside("/tmp", spec.root) && !(overlayFor(home) && inside(home, spec.root))) roBind(spec.root)
}
// Project control paths are read-only.
for (const file of protectedIn(spec.root, protectedRel)) roBind(file)
@@ -131,7 +141,7 @@ export function sandboxArgv(
for (const file of protectedIn(writable, [...PROTECTED_IN_WORKTREE, ...protectedRel])) roBind(file)
}
// A read-only seat's writes in its cwd land in a throwaway layer (when bwrap can).
if (spec.kind === "readonly" && probe.overlay) args.push("--overlay-src", spec.cwd, "--tmp-overlay", spec.cwd)
if (spec.kind === "readonly" && overlayFor(spec.cwd)) args.push("--overlay-src", spec.cwd, "--tmp-overlay", spec.cwd)
// Secrets, for every kind.
if (probe.exists(spec.stateDir)) args.push("--tmpfs", spec.stateDir)
@@ -158,32 +168,44 @@ export function sandboxArgv(
return args
}
let overlayCache: boolean | undefined
/** Whether this bwrap supports --tmp-overlay (Ubuntu 24.04 ships 0.9, which does not). */
export function bwrapOverlay(): boolean {
if (overlayCache !== undefined) return overlayCache
const overlayCache = new Map<string, boolean>()
/**
* Whether this bwrap can overlay-mount `dir` (Ubuntu 24.04 ships bwrap 0.9,
* which has no overlay support at all). Probed per directory and cached: an
* overlay mount can fail on one filesystem while it succeeds on another (a
* btrfs home vs tmpfs /tmp, #231), so probing only /tmp misreports the mount
* that matters.
*/
export function bwrapOverlay(dir = "/tmp"): boolean {
const cached = overlayCache.get(dir)
if (cached !== undefined) return cached
try {
// bwrap is resolved from PATH by design (it may be absent).
const probe = spawnSync(
// NOSONAR
"bwrap",
["--ro-bind", "/", "/", "--dev", "/dev", "--overlay-src", "/tmp", "--tmp-overlay", "/tmp", "--", "true"],
["--ro-bind", "/", "/", "--dev", "/dev", "--overlay-src", dir, "--tmp-overlay", dir, "--", "true"],
{ timeout: 5000 },
)
overlayCache = probe.status === 0
overlayCache.set(dir, probe.status === 0)
} catch {
overlayCache = false
overlayCache.set(dir, false)
}
return overlayCache
return overlayCache.get(dir)!
}
function liveProbe(): SandboxProbe {
return { exists: existsSync, overlay: bwrapOverlay(), uid: process.getuid?.() }
return {
exists: existsSync,
overlay: bwrapOverlay(),
overlayFor: (dir) => bwrapOverlay(dir),
uid: process.getuid?.(),
}
}
/** For tests: forget the cached overlay probe. */
/** For tests: forget the cached overlay probes. */
export const resetOverlayProbe = () => {
overlayCache = undefined
overlayCache.clear()
}
/** Why a sandbox cannot run here, for refusal messages. */
packages/core/test/__snapshots__/assets.test.ts.snap
+2 -2
View File
@@ -5,10 +5,10 @@ exports[`agent assets prompt snapshot: a changed prompt must bump its version 1`
"auditor-antithesis": "v2:15f4d7f75e5bb958",
"auditor-thesis": "v2:b903e1353177f342",
"brainstorm-critic": "v1:28fd3abc95c4a28a",
"brainstormer": "v3:eee9b20fb26482a4",
"brainstormer": "v4:278864b7a2cf7008",
"deep-researcher": "v1:2c436cd49fd6aaf8",
"designer": "v2:745908a2a4abff0b",
"factory": "v9:4b48ec73da814ab3",
"factory": "v10:d87aa973b1517d02",
"grill": "v4:722c43eeb12d95bb",
"harvester": "v2:c63fbc079e1cbbbe",
"lens-adjacent-possible": "v1:d6d768502ed2adf2",
packages/core/test/approval-180.test.ts
+133
View File
@@ -0,0 +1,133 @@
// Issue #180 integration tests (core surface): approvals are hash-bound to
// the approved artifacts and every channel is recorded and verifies. Uses
// only existing seams (test keystores, temp dirs); no host, no human seal.
import { describe, expect, test } from "bun:test"
import { readFile, writeFile } from "node:fs/promises"
import { type HumanSigner, sealHumanKey, unlockHumanKey, verifyRecordSignature } from "../src/approval/keystore.ts"
import { readApproval, verifyApproval } from "../src/approval/record.ts"
import { approveStage } from "../src/approval/service.ts"
import { Factory, pendingApprovals } from "../src/factory/index.ts"
import { factoryLayout } from "../src/layout.ts"
import { writeJson } from "../src/util/fs.ts"
import { type Fixture, frontier, gitRepo, writeSpecs } from "./helpers.ts"
const PASSPHRASE = "test-passphrase-1234"
async function setup(): Promise<{ fx: Fixture; signer: HumanSigner; factory: Factory }> {
const fx = await gitRepo()
await sealHumanKey(PASSPHRASE, fx.state)
const signer = await unlockHumanKey(PASSPHRASE, fx.state)
const factory = new Factory(fx.root, {
gates: async () => ({ passed: true, findings: [], summary: "green" }),
stateDir: fx.state,
})
await factory.writeFrontier("grill", frontier())
return { fx, signer, factory }
}
const seedPending = (fx: Fixture, stages: Array<"frontier" | "spec">) =>
writeJson(
factoryLayout(fx.root).pending,
stages.map((stage) => ({ stage, requestedBy: "factory", at: new Date().toISOString() })),
)
describe("issue #180: approvals are hash-bound", () => {
test("editing frontier.json after approval invalidates the record", async () => {
const { fx, signer } = await setup()
try {
await seedPending(fx, ["frontier"])
const { record } = await approveStage(fx.root, {
stage: "frontier",
channel: "tui",
approvedBy: "tester",
signer,
stateDir: fx.state,
})
expect(record.channel).toBe("tui")
expect((await verifyApproval(fx.root, "frontier", { stateDir: fx.state })).ok).toBe(true)
const file = factoryLayout(fx.root).frontier
await writeFile(file, `${await readFile(file, "utf8")}\n<!-- tampered -->\n`)
const check = await verifyApproval(fx.root, "frontier", { stateDir: fx.state })
expect(check.ok).toBe(false)
if (!check.ok) expect(check.reason).toMatch(/changed after it was approved/)
expect(await readApproval(fx.root, "frontier")).toBeDefined()
expect(await pendingApprovals(fx.root)).toEqual([])
} finally {
signer.destroy()
await fx.cleanup()
}
})
test("editing a GOAL.md after spec approval invalidates it", async () => {
const { fx, signer } = await setup()
try {
await writeSpecs(fx.root, ["alpha"])
await seedPending(fx, ["spec"])
await approveStage(fx.root, {
stage: "spec",
channel: "web",
approvedBy: "tester",
signer,
stateDir: fx.state,
})
expect((await verifyApproval(fx.root, "spec", { stateDir: fx.state })).ok).toBe(true)
const goal = factoryLayout(fx.root).spec("alpha")
await writeFile(goal, `${await readFile(goal, "utf8")}\n<!-- tampered -->\n`)
expect((await verifyApproval(fx.root, "spec", { stateDir: fx.state })).ok).toBe(false)
} finally {
signer.destroy()
await fx.cleanup()
}
})
})
describe("issue #180: every channel is recorded and verifies", () => {
test("cli, tui and web records carry their channel and a valid signature", async () => {
for (const channel of ["cli", "tui", "web"] as const) {
const { fx, signer } = await setup()
try {
await seedPending(fx, ["frontier"])
const { record } = await approveStage(fx.root, {
stage: "frontier",
channel,
approvedBy: "tester",
signer,
stateDir: fx.state,
})
expect(record.channel).toBe(channel)
expect(record.signature.alg).toBe("ed25519")
expect(await verifyRecordSignature(record, fx.state)).toBe(true)
expect(await verifyApproval(fx.root, "frontier", { stateDir: fx.state })).toMatchObject({ ok: true })
} finally {
signer.destroy()
await fx.cleanup()
}
}
})
test("a record with a forged signature is refused", async () => {
const { fx, signer } = await setup()
try {
await seedPending(fx, ["frontier"])
const { record } = await approveStage(fx.root, {
stage: "frontier",
channel: "cli",
approvedBy: "tester",
signer,
stateDir: fx.state,
})
expect((await verifyApproval(fx.root, "frontier", { stateDir: fx.state })).ok).toBe(true)
const file = factoryLayout(fx.root).approval("frontier")
await writeJson(file, {
...record,
signature: { ...record.signature, sig: `${record.signature.sig.slice(0, -2)}AA` },
})
const check = await verifyApproval(fx.root, "frontier", { stateDir: fx.state })
expect(check.ok).toBe(false)
if (!check.ok) expect(check.reason).toMatch(/not signed by this machine's human key/)
} finally {
signer.destroy()
await fx.cleanup()
}
})
})
packages/core/test/approval-keystore.test.ts
+14 -1
View File
@@ -1,6 +1,8 @@
// Human key lifecycle (#119): destroy() ends signing.
import { expect, test } from "bun:test"
import { HumanKeyError, sealHumanKey, unlockHumanKey } from "../src/approval/keystore.ts"
import { writeFile } from "node:fs/promises"
import path from "node:path"
import { HumanKeyError, hasHumanKey, sealHumanKey, unlockHumanKey } from "../src/approval/keystore.ts"
import { gitRepo } from "./helpers.ts"
const PASSPHRASE = "test-passphrase-1234"
@@ -17,3 +19,14 @@ test("destroy zeroes the key and further signs throw", async () => {
fx.cleanup()
}
})
test("hasHumanKey requires both halves: a lone human.key is a brick, not a key (#213)", async () => {
const fx = await gitRepo()
try {
expect(await hasHumanKey(fx.state)).toBe(false)
await writeFile(path.join(fx.state, "human.key"), '{"version":1}\n')
expect(await hasHumanKey(fx.state)).toBe(false)
} finally {
fx.cleanup()
}
})
packages/core/test/brainstorm.test.ts
+21
View File
@@ -7,6 +7,8 @@ import { tmpdir } from "node:os"
import path from "node:path"
import type { BrainstormIdea, BrainstormScore } from "../src/index.ts"
import {
AGENTS,
BRAINSTORM_FANOUT_SPAWNS,
brainstormRunPaths,
brainstormTools,
buildPlan,
@@ -455,3 +457,22 @@ describe("callable runs (#108)", () => {
}
})
})
describe("nested refusal at depth 1 (#269)", () => {
test("lens subagents and the critic spawn nothing; only primaries fan out", () => {
const byId = new Map(AGENTS.map((agent) => [agent.id, agent]))
// The mechanism: empty `spawns` + deny-by-default + the foreground guard.
// A lens or the critic asked to launch a subagent has nothing to launch.
for (const lens of LENSES) {
const agent = byId.get(`es-lens-${lens.id}`)!
expect(agent, `es-lens-${lens.id}`).toBeDefined()
expect(agent.spawns).toEqual([])
}
expect(byId.get("es-brainstorm-critic")!.spawns).toEqual([])
// The calling primaries fan out themselves, at depth 1, in the foreground.
for (const id of ["brainstormer", "grill", "factory"]) {
const spawns = byId.get(id)!.spawns
for (const fanout of BRAINSTORM_FANOUT_SPAWNS) expect(spawns, id).toContain(fanout)
}
})
})
packages/core/test/codeaudit-186.test.ts
+107
View File
@@ -0,0 +1,107 @@
// Issue #186 integration pins, core side (bug #241 closers): the verifySpan
// short-excerpt branch at es_audit_verdict level. The schema keeps excerpt
// min(1); the 12-char witness in verifySpan is what refuses — this file pins
// that current behavior without weakening either.
import { afterEach, beforeEach, describe, expect, test } from "bun:test"
import { mkdir, writeFile } from "node:fs/promises"
import path from "node:path"
import { type HumanSigner, sealHumanKey, unlockHumanKey } from "../src/approval/keystore.ts"
import { codeAuditTools } from "../src/codeaudit/index.ts"
import { Factory } from "../src/factory/index.ts"
import { factoryLayout } from "../src/layout.ts"
import { esTools } from "../src/ops/tools.ts"
import { verifySpan } from "../src/research/quote.ts"
import { AuditFindingSchema } from "../src/schema/codeaudit.ts"
import { type Fixture, gitRepo } from "./helpers.ts"
const AUTH = [
"export function login(db: Db, user: string) {",
` const sql = "SELECT * FROM users WHERE name = '" + user + "'"`,
" return db.query(sql)",
"}",
"",
].join("\n")
let fx: Fixture
let factory: Factory
const PASSPHRASE = "test-passphrase-1234"
let signer: HumanSigner
beforeEach(async () => {
fx = await gitRepo("es-audit-186-")
await sealHumanKey(PASSPHRASE, fx.state)
signer = await unlockHumanKey(PASSPHRASE, fx.state)
void signer
factory = new Factory(fx.root, {
gates: async () => ({ passed: true, findings: [], summary: "green" }),
stateDir: fx.state,
})
await mkdir(path.join(fx.root, "src"), { recursive: true })
await writeFile(path.join(fx.root, "src/auth.ts"), AUTH)
})
afterEach(() => fx.cleanup())
const tools = () => {
const ops = { root: fx.root, factory, stateDir: fx.state }
const all = [...codeAuditTools(ops), ...esTools(ops)]
return (name: string) => all.find((tool) => tool.name === name)!
}
const injection = {
kind: "vulnerability",
title: "SQL built by string concatenation",
severity: "high",
cwe: "CWE-89",
file: "src/auth.ts",
lines: [2, 3],
excerpt: `"SELECT * FROM users WHERE name = '" + user`,
detail: "user = \"' OR '1'='1\" returns every row; the query is concatenated, not parameterised.",
remediation: "Use db.query('SELECT * FROM users WHERE name = ?', [user]).",
}
async function openPathAudit() {
return (await factory.openAudit("human:tester", { kind: "path", path: "src" }, { ceilingUSD: 5 })).audit
}
describe("verifySpan short branch (#241)", () => {
test("an 11-char span is too short; a 12-char verbatim span is evidence", () => {
const source = "return db.query(sql)"
expect(verifySpan(source, "db.query(sq")).toEqual({
ok: false,
reason: "too short to be evidence (need at least 12 characters)",
})
expect(verifySpan(source, "db.query(sql")).toEqual({ ok: true, index: 7 })
expect(verifySpan(source, "db.query(no)").ok).toBe(false)
})
test("the schema still accepts a short excerpt; the witness is what refuses", () => {
// #241 item 2 premise: schema/codeaudit.ts promises "at least 12
// characters" in the comment but enforces min(1). Pin the behavior:
// parsing succeeds, so only the witness stands in the way.
const parsed = AuditFindingSchema.safeParse({ ...injection, lines: [3, 3], excerpt: "db.query" })
expect(parsed.success).toBe(true)
})
})
describe("short excerpts at es_audit_verdict (#241)", () => {
test("a <12-char excerpt refuses the whole verdict with 'too short to be evidence'; nothing is recorded", async () => {
const audit = await openPathAudit()
const verdict = tools()("es_audit_verdict")
const short = { ...injection, lines: [3, 3], excerpt: "db.query" }
await expect(
verdict.execute(
{ audit: audit.id, verdict: "fail", findings: [short], notes: "n" },
{ agent: "es-auditor-antithesis" },
),
).rejects.toThrow("too short to be evidence")
// One short excerpt among good findings still refuses everything.
await expect(
verdict.execute(
{ audit: audit.id, verdict: "fail", findings: [injection, short], notes: "n" },
{ agent: "es-auditor-antithesis" },
),
).rejects.toThrow("1 finding(s) do not point at real code")
// Nothing reached the state, the records or the dossier.
expect((await factory.read())!.audits[0]!.antithesis).toBeUndefined()
expect(await Bun.file(factoryLayout(fx.root).auditRecords(audit.id)).exists()).toBe(false)
})
})
packages/core/test/codeaudit.test.ts
+16
View File
@@ -143,6 +143,22 @@ describe("the line-pointer law at es_audit_verdict", () => {
expect(await Bun.file(factoryLayout(fx.root).auditRecords(audit.id)).exists()).toBe(false)
})
test("an excerpt under 12 characters is refused as too short to be evidence (#241)", async () => {
const audit = await openPathAudit()
const verdict = tools()("es_audit_verdict")
// "db.query" (8 chars) is verbatim on line 3, so only the witness's
// 12-char floor (verifySpan) can refuse it — pins the short branch at
// verdict level, not just in the sibling span tests.
const short = { ...injection, lines: [3, 3], excerpt: "db.query" }
await expect(
verdict.execute(
{ audit: audit.id, verdict: "fail", findings: [short], notes: "n" },
{ agent: "es-auditor-antithesis" },
),
).rejects.toThrow(/too short to be evidence/)
expect((await factory.read())!.audits[0]!.antithesis).toBeUndefined()
})
test("malformed findings and verdicts that contradict their findings are refused", async () => {
const audit = await openPathAudit()
const verdict = tools()("es_audit_verdict")
packages/core/test/config-layers-203.test.ts
+107
View File
@@ -0,0 +1,107 @@
// Audit #203 — layered model tiers, global → project → plugin options.
//
// The full three-layer conflict (each layer wins a different tier) through
// loadEsConfig, the strict side of the unknown-key contract (config files
// reject unknowns, including 0.7-era keys — plugin options only warn, proven
// in packages/opencode/test/config-layers-203.test.ts), and the human-only
// boundary for `es config set` through the shell policy.
import { afterEach, beforeEach, describe, expect, test } from "bun:test"
import { mkdir, mkdtemp, rm, writeFile } from "node:fs/promises"
import { tmpdir } from "node:os"
import path from "node:path"
import { loadEsConfig } from "../src/index.ts"
import { evaluateShell, invokesHumanOnly } from "../src/trust/index.ts"
let root: string
let env: NodeJS.ProcessEnv
beforeEach(async () => {
root = await mkdtemp(path.join(tmpdir(), "es-203-layers-"))
env = { XDG_CONFIG_HOME: path.join(root, "xdg") }
})
afterEach(() => rm(root, { recursive: true, force: true }))
const writeGlobal = async (value: unknown) => {
const file = path.join(env.XDG_CONFIG_HOME!, "epistemic-swarm", "config.json")
await mkdir(path.dirname(file), { recursive: true })
await writeFile(file, JSON.stringify(value))
return file
}
const writeProject = async (value: unknown) => {
await mkdir(path.join(root, ".factory"), { recursive: true })
const file = path.join(root, ".factory", "config.json")
await writeFile(file, JSON.stringify(value))
return file
}
describe("three-layer tier precedence (#203)", () => {
test("each layer wins a different tier: options > project > global", async () => {
const global = await writeGlobal({
models: { fast: "global/fast", balanced: "global/balanced", deep: "global/deep" },
})
const project = await writeProject({ models: { fast: "project/fast", deep: "project/deep" } })
const loaded = await loadEsConfig(root, { env, overrides: { models: { fast: "options/fast" } } })
expect(loaded.sources).toEqual([global, project])
expect(loaded.config.models).toEqual({
fast: "options/fast",
balanced: "global/balanced",
deep: "project/deep",
})
})
test("a lower layer survives wherever a higher layer is silent", async () => {
await writeGlobal({ models: { deep: "global/deep" }, pr: "off" })
await writeProject({ pr: "gh" })
const loaded = await loadEsConfig(root, { env, overrides: { afterEdit: "off" } })
expect(loaded.config.models).toEqual({ deep: "global/deep" })
expect(loaded.config.pr).toBe("gh")
expect(loaded.config.afterEdit).toBe("off")
})
})
describe("config-file unknowns are rejected, strictly (#203)", () => {
test.each(["search_engine", "max_iterations", "mode"])(
"project file: 0.7-era key %s is rejected, naming it",
async (key: string) => {
await writeProject({ [key]: key === "max_iterations" ? 2 : "x" })
await expect(loadEsConfig(root, { env })).rejects.toThrow(new RegExp(`invalid config.*${key}`))
},
)
test("project file: a typo key is rejected with the source named", async () => {
const project = await writeProject({ modles: { fast: "a/fast" } })
await expect(loadEsConfig(root, { env })).rejects.toThrow(new RegExp(`invalid config.*${project}.*modles|modles`))
})
test("global file: unknown keys are rejected too", async () => {
await writeGlobal({ search_engine: "duckduckgo" })
await expect(loadEsConfig(root, { env })).rejects.toThrow(/invalid config.*search_engine/)
})
test("control: known keys in both files still load", async () => {
await writeGlobal({ research: { depth: 1 } })
await writeProject({ research: { depth: 3 } })
expect((await loadEsConfig(root, { env })).config.research).toEqual({ depth: 3 })
})
})
describe("es config set stays human-only (#203)", () => {
const shell = (agent: string | undefined, command: string) =>
evaluateShell({ root, stateDir: path.join(root, "state") }, agent, command, { sandboxAvailable: true })
test("invokesHumanOnly reads config set, however flagged; config show stays a read", () => {
expect(invokesHumanOnly("es config set models.deep x/y")).toBe(true)
expect(invokesHumanOnly("es --cwd . config set models.deep x/y")).toBe(true)
expect(invokesHumanOnly("es config set models.deep x/y --global")).toBe(true)
expect(invokesHumanOnly("es config show")).toBe(false)
expect(invokesHumanOnly("es --cwd . config show")).toBe(false)
})
test("every agent shell is denied config set; config show is allowed", () => {
for (const agent of ["build", "factory", "es-programmer", undefined])
expect([agent, shell(agent, "es config set models.deep x/y").effect]).toEqual([agent, "deny"])
const denied = shell("build", "es config set models.deep x/y")
expect(denied.effect).toBe("deny")
if (denied.effect === "deny") expect(denied.reason).toContain("human-only")
expect(shell("build", "es config show").effect).toBe("allow")
})
})
packages/core/test/control-matrix.test.ts
+151
View File
@@ -0,0 +1,151 @@
// Full seat × control-path enumeration at the unit level (#182). The sampled
// proofs live in policy.test.ts; this suite pins EVERY registry seat against
// every control glob (relative, absolute and dot-dot spellings), plus the
// shell layer split: seats are refused by the CONTROL_MENTION text rule
// except the two paths it does not name, which pass to the sandbox backstop.
// The same matrix runs against the real host in
// packages/opencode/test/control-matrix.test.ts.
import { afterAll, beforeAll, describe, expect, test } from "bun:test"
import { mkdir, mkdtemp, rm, symlink, writeFile } from "node:fs/promises"
import { tmpdir } from "node:os"
import path from "node:path"
import { AGENTS } from "../src/agents/registry.ts"
import { evaluateShell, evaluateWrite } from "../src/trust/index.ts"
/** One representative per FACTORY_CONTROL / CONFIG_CONTROL glob (trust/control.ts). */
const CONTROL_PATHS = [
".factory/gates.json",
".factory/config.json",
".factory/frontier.json",
".factory/approvals/spec.json",
".factory/waivers/lint.json",
".factory/runtime/state.json",
".factory/STOP",
".factory/git-hooks/pre-commit",
".factory/brainstorm/scores.json",
".factory/brainstorm/runs/grill-1/ideas.json",
".factory/harvest/plan.json",
".factory/harvest/runs/grill-1/plan.json",
".factory/harvest/widget/profile.json",
".factory/harvest/clean-room/widget-search.md",
".factory/design/tokens.json",
".factory/design/components/button/tokens.json",
`.factory/research/sources/${"a".repeat(64)}.md`,
".factory/research/coverage.json",
".factory/research/dossier.json",
".factory/research/brief.pcrb.json",
".factory/claims/retractions.json",
".factory/improve/proposal.json",
".factory/audits/audit-01.json",
".factory/scout/plan.json",
".factory/scout/widget/profile.json",
".factory/worktrees/phase-01/.factory/approvals/spec.json",
".git/config",
".git/hooks/pre-commit",
"opencode.jsonc",
".claude/settings.json",
]
const FACTORY_PATHS = CONTROL_PATHS.filter(
(target) => target !== "opencode.jsonc" && target !== ".claude/settings.json",
)
/** Paths the shell text rule does not name; seats pass them to the sandbox. */
const SANDBOX_BACKSTOP = new Set([".factory/harvest/clean-room/widget-search.md", ".factory/improve/proposal.json"])
let root: string
let state: string
let worktree: string
beforeAll(async () => {
root = await mkdtemp(path.join(tmpdir(), "es-matrix-core-"))
state = await mkdtemp(path.join(tmpdir(), "es-matrix-core-state-"))
worktree = path.join(root, ".factory/worktrees/p1")
await mkdir(path.join(root, ".factory/approvals"), { recursive: true })
await mkdir(path.join(root, "src"), { recursive: true })
await mkdir(path.join(worktree, "src"), { recursive: true })
await writeFile(path.join(root, ".factory/approvals/spec.json"), "{}\n")
await symlink(path.join(root, ".factory/approvals"), path.join(root, "src/link"))
})
afterAll(async () => {
await rm(root, { recursive: true, force: true })
await rm(state, { recursive: true, force: true })
})
const ctx = () => ({ root, worktree, stateDir: state })
describe("control matrix: every seat is denied every control path (#182)", () => {
test("evaluateWrite denies all seats, however the path is spelled", () => {
for (const spec of AGENTS) {
for (const target of CONTROL_PATHS) {
for (const spelling of [target, path.join(root, target), `docs/../${target}`]) {
const decision = evaluateWrite(ctx(), spec.id, spelling)
expect([spec.id, spelling, decision.effect]).toEqual([spec.id, spelling, "deny"])
}
}
}
})
test("non-seat agents are denied the factory controls but keep host semantics for config", () => {
for (const agent of [undefined, "build"]) {
for (const target of FACTORY_PATHS)
expect([agent, target, evaluateWrite(ctx(), agent, target).effect]).toEqual([agent, target, "deny"])
// Config controls ask the human for non-seat agents (host semantics).
for (const target of ["opencode.jsonc", ".claude/settings.json"])
expect([agent, target, evaluateWrite(ctx(), agent, target).effect]).toEqual([agent, target, "ask"])
}
})
test("shell: seats are refused at the text layer except the documented sandbox-backstop paths", () => {
for (const spec of AGENTS) {
for (const target of CONTROL_PATHS) {
const decision = evaluateShell(ctx(), spec.id, `echo pwned > ${target} && echo WRITE-SUCCEEDED`, {
sandboxAvailable: true,
})
if (SANDBOX_BACKSTOP.has(target)) {
// Not named by CONTROL_MENTION: runs under the sandbox, whose
// read-only .factory bind refuses the redirect.
expect([spec.id, target, decision]).toMatchObject([spec.id, target, { effect: "allow", mode: "sandbox" }])
} else {
expect(decision.effect).toBe("deny")
expect("reason" in decision ? decision.reason : "").toMatch(/may not reference control files/)
}
}
}
})
test("shell: non-seat agents are denied every control path (broad factory-dir rule)", () => {
for (const target of CONTROL_PATHS) {
const decision = evaluateShell(ctx(), undefined, `echo pwned > ${target}`, { sandboxAvailable: false })
expect([target, decision.effect]).toEqual([target, "deny"])
}
})
test("case variants are denied in the editor and the shell", () => {
const variants = [
".FACTORY/GATES.json",
".factory/GATES.JSON",
".Factory/Approvals/Spec.JSON",
".GIT/CONFIG",
"OpenCode.JSONC",
".CLAUDE/Settings.JSON",
]
for (const spec of ["es-manager", "factory", "es-programmer", "es-qa-functional"]) {
for (const target of variants) {
expect([spec, target, evaluateWrite(ctx(), spec, target).effect]).toEqual([spec, target, "deny"])
const shell = evaluateShell(ctx(), spec, `echo pwned > ${target}`, { sandboxAvailable: true })
expect([spec, target, shell.effect]).toEqual([spec, target, "deny"])
}
}
})
test("a symlink from an allowed dir into .factory cannot be used to write a control file", () => {
for (const spec of ["es-manager", "factory", "build", undefined]) {
for (const target of ["src/link/spec.json", path.join(root, "src/link/spec.json")]) {
const decision = evaluateWrite(ctx(), spec, target)
expect([spec, target, decision.effect]).toEqual([spec, target, "deny"])
expect("reason" in decision ? decision.reason : "").toContain("factory control file")
}
}
})
})
packages/core/test/domainpacks-packs.test.ts
+170
View File
@@ -0,0 +1,170 @@
// Per-pack coverage closers for #256 (audit #190 follow-up): every shipped
// pack's banned domains and threshold/weight values pin current behavior, the
// below-threshold refusal path at machine.ts:545 is exercised, and the
// SECONDARY_MEDIA / tierDomains dead config is documented as observed.
// Scoring itself is NOT changed here (#252 owns the scoring defects: tierless
// weight, NK inflation, garbage tags); these tests pin the mechanism only.
import { afterEach, beforeEach, describe, expect, test } from "bun:test"
import { mkdir, writeFile } from "node:fs/promises"
import path from "node:path"
import { type HumanSigner, sealHumanKey, unlockHumanKey } from "../src/approval/keystore.ts"
import {
type Constitution,
claimVerdict,
constitutionFromPack,
Factory,
factoryLayout,
gateRunner,
LEGACY_CONSTITUTION,
loadDomainPack,
recordApproval,
researchSourcesDir,
SourceCache,
} from "../src/index.ts"
import { researchOptions } from "../src/research/ops.ts"
import { classifySourceTier } from "../src/research/tier.ts"
import { type Fixture, frontier, gitRepo } from "./helpers.ts"
const pack = async (id: string): Promise<Constitution> => constitutionFromPack(await loadDomainPack(id))
describe("each shipped pack bans its own domains (#256.1)", () => {
const cases = [
{ packId: "quant", domain: "vendor-whitepaper-mirror.example" },
{ packId: "legal", domain: "undisclosed-sponsored-content.example" },
{ packId: "biopharma", domain: "predatory-journal.example" },
] as const
for (const { packId, domain } of cases) {
test(`${packId} rejects a source on ${domain} (BANNED_DOMAIN), other packs and legacy accept it`, async () => {
const claim = {
id: `${packId}-banned`,
tag: "VERIFIED",
source: {
sha256: "b".repeat(64),
quote: "the whitepaper claims linear scaling on every desk",
url: `https://${domain}/research-note`,
},
}
const own = claimVerdict(claim, { constitution: await pack(packId) })
expect(own.verdict).toBe("REJECTED")
expect(own.reasons.some((reason) => reason.startsWith("BANNED_DOMAIN"))).toBe(true)
// Pack-agnostic baseline unchanged: legacy accepts the same claim.
expect(claimVerdict(claim, { constitution: LEGACY_CONSTITUTION }).verdict).toBe("ACCEPTED")
// Specificity: a sibling pack with a different ban list accepts it.
const sibling = cases.find((entry) => entry.packId !== packId)!.packId
expect(claimVerdict(claim, { constitution: await pack(sibling) }).verdict).toBe("ACCEPTED")
})
}
})
describe("pack threshold and weight values are pinned (#256.1)", () => {
test("accept thresholds: biopharma 0.7, quant 0.7, legal 0.65", async () => {
expect((await pack("biopharma")).acceptThreshold).toBe(0.7)
expect((await pack("quant")).acceptThreshold).toBe(0.7)
expect((await pack("legal")).acceptThreshold).toBe(0.65)
})
test("spot tier weights: biopharma downweights preprints, legal prefers primary press, quant values docs", async () => {
expect((await pack("biopharma")).tierWeights.PREPRINT).toBe(0.3)
expect((await pack("biopharma")).tierWeights.PEER_REVIEWED).toBe(1.0)
expect((await pack("legal")).tierWeights.PRIMARY_PRESS).toBe(1.0)
expect((await pack("legal")).tierWeights.PREPRINT).toBe(0.2)
expect((await pack("quant")).tierWeights.TECHNICAL_DOCUMENTATION).toBe(1.0)
expect((await pack("quant")).tierWeights.PREPRINT).toBe(0.5)
})
test("ban lists and retraction policies are pinned per pack", async () => {
expect((await pack("biopharma")).bannedDomains).toEqual([
"predatory-journal.example",
"retracted-mirror.example",
"banned-preprint-mirror.example",
])
expect((await pack("quant")).bannedDomains).toEqual(["vendor-whitepaper-mirror.example"])
expect((await pack("legal")).bannedDomains).toEqual(["undisclosed-sponsored-content.example"])
expect((await pack("biopharma")).retractionPolicy).toBe("zero_tolerance")
expect((await pack("quant")).retractionPolicy).toBe("standard")
expect((await pack("legal")).retractionPolicy).toBe("standard")
expect((await pack("quant")).negBonus).toBe(0.75)
expect((await pack("quant")).rejectPenalty).toBe(3.0)
})
})
describe("the below-threshold refusal path (#256.2)", () => {
let fx: Fixture
const PASSPHRASE = "test-passphrase-1234"
let signer: HumanSigner
beforeEach(async () => {
fx = await gitRepo()
await sealHumanKey(PASSPHRASE, fx.state)
signer = await unlockHumanKey(PASSPHRASE, fx.state)
})
afterEach(() => fx.cleanup())
test("a preprint-only report passes every per-claim rule yet is refused below the biopharma threshold", async () => {
const factory = new Factory(fx.root, {
gates: gateRunner({ stateDir: fx.state }),
stateDir: fx.state,
domainPack: "biopharma",
})
await factory.begin("human:tester")
await factory.writeFrontier("grill", frontier())
await recordApproval(fx.root, { stage: "frontier", channel: "cli", approvedBy: "tester", signer })
await factory.beginResearch("human:tester")
// arxiv.org classifies PREPRINT (weight 0.3 under biopharma), so one
// verified claim scores 0.3 < 0.7 with no per-claim rule firing.
const cache = new SourceCache(researchSourcesDir(fx.root), fx.state)
const source = await cache.put({
url: "https://arxiv.org/abs/2601.00001",
text: "A preprint model predicts surrogate endpoints in early cohorts.",
provider: "fetch",
})
expect(source.meta.tier).toBe("PREPRINT")
await mkdir(path.dirname(factoryLayout(fx.root).researchReport), { recursive: true })
await writeFile(
factoryLayout(fx.root).researchReport,
`- A preprint predicts surrogate endpoints [VERIFIED: sha256:${source.meta.sha256} "predicts surrogate endpoints"]\n`,
)
let message: string | undefined
try {
await factory.completeResearch("factory")
} catch (caught) {
message = caught instanceof Error ? caught.message : String(caught)
}
expect(message).toBeDefined()
// The machine.ts:545 threshold refusal, not the per-claim rejection path.
expect(message).toContain("below its threshold")
expect(message).toContain("biopharma")
expect(message).not.toContain("rejects")
})
})
describe("dead tier config, documented as observed (#256.3 — do not fix here)", () => {
test("SECONDARY_MEDIA is weighted in all three packs but the classifier never emits it", async () => {
// Observed: every shipped pack carries a SECONDARY_MEDIA weight ...
expect((await loadDomainPack("biopharma")).tierWeights.SECONDARY_MEDIA).toBe(0.2)
expect((await loadDomainPack("quant")).tierWeights.SECONDARY_MEDIA).toBe(0.2)
expect((await loadDomainPack("legal")).tierWeights.SECONDARY_MEDIA).toBe(0.3)
// ... yet tier.ts has no rule returning it, not even for news-like hosts.
for (const url of [
"https://arxiv.org/abs/1",
"https://doi.org/10.1/x",
"https://docs.python.org/3/",
"https://www.fda.gov/news/x",
"https://www.bbc.com/news/science-1",
"https://www.nytimes.com/2026/01/01/health/x.html",
"https://blog.example.com/opinion",
"https://example.test/x",
])
expect([url, classifySourceTier(url)]).toEqual([url, expect.not.stringMatching("SECONDARY_MEDIA")])
})
test("tierDomains overrides reach no caller: no shipped pack sets them and researchOptions drops them", async () => {
// Observed: the schema, cache.put and ops plumbing accept tierDomains,
// but no shipped pack file sets them ...
for (const id of ["biopharma", "quant", "legal"])
expect([(await loadDomainPack(id)).packId, (await loadDomainPack(id)).tierDomains]).toEqual([id, undefined])
// ... and the config-to-tools mapping never forwards them (verified by
// grep: zero assignments to ResearchOpsContext.tierDomains repo-wide), so
// a pack can never re-tier a host at runtime today.
expect("tierDomains" in researchOptions({ research: { depth: 2 } })).toBe(false)
})
})
packages/core/test/domainpacks.test.ts
+141 -4
View File
@@ -205,7 +205,7 @@ describe("the pack gates completeResearch", () => {
test("a clean report passes under biopharma and records the pack and score", async () => {
const factory = await toResearch("biopharma")
await report(
"https://journal.example/clean",
"https://doi.org/10.1/clean",
"The cohort met the primary efficacy bar in the per-protocol analysis.",
"primary efficacy bar",
)
@@ -240,6 +240,18 @@ describe("the pack gates completeResearch", () => {
await expect(unpacked.completeResearch("factory")).resolves.toBeDefined()
})
test("a below-threshold report is refused at completion, naming the score (#256.2)", async () => {
// A preprint-only report passes every claim verdict under biopharma but
// scores 0.3 against its 0.7 threshold: the machine.ts refusal path.
const factory = await toResearch("biopharma")
await report(
"https://arxiv.org/abs/1234",
"A model predicts surrogate endpoints in this preprint.",
"predicts surrogate endpoints",
)
await expect(factory.completeResearch("factory")).rejects.toThrow("below its threshold")
})
test("an unknown configured pack fails closed at completion time", async () => {
const factory = await toResearch("no-such-pack")
await report("https://journal.example/clean", "A finding rests on a cached source.", "cached source")
@@ -274,7 +286,10 @@ describe("source tiers reach claims and the score (#113)", () => {
classifySourceTier("https://www.nature.com/articles/x"),
classifySourceTier("https://docs.python.org/3/"),
classifySourceTier("https://developer.mozilla.org/en/"),
// A docs./developer. prefix on an unlisted host earns no tier (#252):
// the prefix rule would let docs.<attacker> claim curated weight.
classifySourceTier("https://docs.example.com/x"),
classifySourceTier("https://developer.example.com/y"),
classifySourceTier("https://www.fda.gov/news/x"),
classifySourceTier("https://journal.example/clean"),
classifySourceTier("not a url"),
@@ -288,7 +303,8 @@ describe("source tiers reach claims and the score (#113)", () => {
"PEER_REVIEWED",
"TECHNICAL_DOCUMENTATION",
"TECHNICAL_DOCUMENTATION",
"TECHNICAL_DOCUMENTATION",
undefined,
undefined,
"PRIMARY_PRESS",
undefined,
undefined,
@@ -373,8 +389,13 @@ describe("source tiers reach claims and the score (#113)", () => {
)
expect(peer.score).toBe(1)
expect(pre.score).toBe(0.3)
// Unknown tiers fall back to __default__.
expect(computeEpistemicScoreFromClaims([{ id: "c".repeat(64), tag: "VERIFIED" }], constitution).score).toBe(1)
// Unknown tiers fall back to __default__, which a pack clamps to its
// lowest verified tier weight (#252): tierless must never outscore
// verified evidence.
expect(weightFor(constitution, undefined)).toBeLessThanOrEqual(0.3)
expect(computeEpistemicScoreFromClaims([{ id: "c".repeat(64), tag: "VERIFIED" }], constitution).score).toBe(
weightFor(constitution, undefined),
)
})
test("tiers do not change claim identity: PCRB export and verify still pass", async () => {
@@ -395,3 +416,119 @@ describe("source tiers reach claims and the score (#113)", () => {
expect(claimId({ ...without, tier: "PREPRINT" } as never)).toBe(without.id)
})
})
describe("scoring integrity (#252)", () => {
test("tierless claims weigh at the lowest verified tier, never full weight", async () => {
// Unclassifiable hosts used to fall back to __default__ 1.0 and outscore
// verified preprints (0.3) under biopharma: a perverse incentive to cite
// worse sources. A pack now clamps __default__ to its lowest tier weight.
const pack = await loadDomainPack("biopharma")
const constitution = constitutionFromPack(pack)
const floor = Math.min(
...Object.entries(pack.tierWeights)
.filter(([tier]) => tier !== "__default__")
.map(([, weight]) => weight),
)
expect(weightFor(constitution, undefined)).toBe(floor)
const tierless = computeEpistemicScoreFromClaims([{ id: "c".repeat(64), tag: "VERIFIED" }], constitution)
const preprint = computeEpistemicScoreFromClaims(
[{ id: "d".repeat(64), tag: "VERIFIED", tier: "PREPRINT" }],
constitution,
)
expect(tierless.score).toBe(floor)
expect(tierless.score).toBeLessThanOrEqual(preprint.score)
// The no-pack constitution is untouched (legacy numbers preserved).
expect(weightFor(LEGACY_CONSTITUTION, undefined)).toBe(1)
})
test("negative knowledge cannot inflate the score past the threshold", async () => {
// totalAssertions excluded negKnowledge while the numerator added
// negBonus per row: 10xPREPRINT went 0.3 -> 1.0 with 14 NK lines. NK rows
// now count as assertions too.
const pack = await loadDomainPack("biopharma")
const constitution = constitutionFromPack(pack)
const verified = Array.from({ length: 10 }, (_, index) => ({
id: `v${index}`.padEnd(64, "0"),
tag: "VERIFIED" as const,
tier: "PREPRINT" as const,
}))
const nk = Array.from({ length: 14 }, (_, index) => ({
id: `n${index}`.padEnd(64, "0"),
tag: "NEGATIVE_KNOWLEDGE" as const,
query: `query ${index}`,
finding: `finding ${index}`,
}))
const scored = computeEpistemicScoreFromClaims([...verified, ...nk], constitution)
expect(scored.breakdown.negativeKnowledgeCount).toBe(14)
expect(scored.score).toBeLessThan(constitution.acceptThreshold)
})
test("mandatory tags enforce membership, not just presence", async () => {
const pack = await biopharma()
const garbage = claimVerdict({ id: "g", tag: "GARBAGE" }, { constitution: pack })
expect(garbage.verdict).toBe("REJECTED")
expect(garbage.reasons.some((reason) => reason.startsWith("MISSING_TAG"))).toBe(true)
expect(
claimVerdict({ id: "v", tag: "VERIFIED", source: { sha256: "a".repeat(64), quote: "q" } }, { constitution: pack })
.verdict,
).toBe("ACCEPTED")
})
})
describe("pack coverage (#256)", () => {
test("each pack pins its thresholds, weights and penalties", async () => {
// A typo or deletion in a pack file stays green under mechanism-only
// tests, so the values are pinned exactly (SECONDARY_MEDIA was removed:
// the classifier never emits it, #256.3).
const biopharma = await loadDomainPack("biopharma")
expect(biopharma.tierWeights).toEqual({
__default__: 1,
PEER_REVIEWED: 1,
TECHNICAL_DOCUMENTATION: 0.8,
PRIMARY_PRESS: 0.7,
PREPRINT: 0.3,
})
expect([biopharma.acceptThreshold, biopharma.negBonus, biopharma.rejectPenalty]).toEqual([0.7, 0.5, 2.5])
expect(biopharma.retractionPolicy).toBe("zero_tolerance")
const quant = await loadDomainPack("quant")
expect(quant.tierWeights).toEqual({
__default__: 1,
PEER_REVIEWED: 0.9,
TECHNICAL_DOCUMENTATION: 1,
PRIMARY_PRESS: 0.6,
PREPRINT: 0.5,
})
expect([quant.acceptThreshold, quant.negBonus, quant.rejectPenalty]).toEqual([0.7, 0.75, 3])
const legal = await loadDomainPack("legal")
expect(legal.tierWeights).toEqual({
__default__: 1,
PEER_REVIEWED: 0.8,
TECHNICAL_DOCUMENTATION: 0.7,
PRIMARY_PRESS: 1,
PREPRINT: 0.2,
})
expect([legal.acceptThreshold, legal.negBonus, legal.rejectPenalty]).toEqual([0.65, 0.5, 2.5])
})
test("quant and legal ban their own mirrors, not just biopharma's", async () => {
// Only biopharma's banned domains appeared in fixtures; per-pack
// behavior is asserted directly here.
const quant = constitutionFromPack(await loadDomainPack("quant"))
const legal = constitutionFromPack(await loadDomainPack("legal"))
const bio = await biopharma()
const vendorMirror = {
id: "q",
tag: "VERIFIED",
source: { sha256: "b".repeat(64), quote: "backtest", url: "https://vendor-whitepaper-mirror.example/p" },
}
expect(claimVerdict(vendorMirror, { constitution: quant }).verdict).toBe("REJECTED")
expect(claimVerdict(vendorMirror, { constitution: bio }).verdict).toBe("ACCEPTED")
const sponsored = {
id: "l",
tag: "VERIFIED",
source: { sha256: "c".repeat(64), quote: "holding", url: "https://undisclosed-sponsored-content.example/x" },
}
expect(claimVerdict(sponsored, { constitution: legal }).verdict).toBe("REJECTED")
expect(claimVerdict(sponsored, { constitution: bio }).verdict).toBe("ACCEPTED")
})
})
packages/core/test/evals.test.ts
+6
View File
@@ -90,6 +90,8 @@ describe("transcripts", () => {
expect(transcript.tokens).toEqual({ input: 150, output: 15, reasoning: 2, cacheRead: 4, cacheWrite: 6 })
// Missing or non-numeric fields count as zero rather than poisoning the sum.
expect(readTranscript([{ type: "step_finish", part: { cost: "n/a" } }]).costUSD).toBe(0)
// Negative costs clamp to zero: a hostile stream must not unbind spend (#211).
expect(readTranscript([{ type: "step_finish", part: { cost: -5 } }]).costUSD).toBe(0)
})
test("a completed call's output is graded text, except raw evidence and errored calls", () => {
@@ -133,6 +135,10 @@ describe("step caps", () => {
expect(caseStepCap(undefined, 16)).toBe(DEFAULT_CASE_STEPS)
expect(caseStepCap(undefined, 3)).toBe(3)
})
test("a NaN maxSteps falls back to the default instead of disabling the cap (#211)", () => {
expect(caseStepCap(Number.NaN, 16)).toBe(DEFAULT_CASE_STEPS)
})
})
describe("fire graders", () => {
packages/core/test/evidence-187.test.ts
+188
View File
@@ -0,0 +1,188 @@
// Issue #187 §2 + bug #251 fuzz-adjacent edges (deterministic, no fuzz harness).
//
// Pins the CURRENT normalisation semantics of `verifyQuote` (NFKC,
// typography folds, markdown stripping, whitespace collapse; no entity
// decoding) and the CURRENT auditor downgrade rules (VERIFIED verbatim
// grounds, paraphrases and planted entries stay ungrounded). Bugs #251
// (fuzz corpus, legacy unsealed constructor) and #254 (wording) are for the
// parent: this suite tests behaviour as-is and reports defects as findings,
// never by changing product code.
import { afterEach, beforeEach, describe, expect, test } from "bun:test"
import { mkdir, mkdtemp, rm, writeFile } from "node:fs/promises"
import { tmpdir } from "node:os"
import path from "node:path"
import { auditMarkdown, htmlToText, normalizeSourceText, SourceCache, sha256, verifyQuote } from "../src/index.ts"
let dir: string
let state: string
beforeEach(async () => {
dir = await mkdtemp(path.join(tmpdir(), "es-evidence-187-"))
state = await mkdtemp(path.join(tmpdir(), "es-evidence-187-state-"))
})
afterEach(() => Promise.all([rm(dir, { recursive: true, force: true }), rm(state, { recursive: true, force: true })]))
const sealed = () => new SourceCache(path.join(dir, "sources"), state)
const GATES = "Alpha gates keep agents honest under heavy load conditions today, witnesses agree."
describe("quote verifier fuzz-adjacent edges (#187 §2, gaps listed in #251)", () => {
test("zero-width chars: ZWSP/FEFF fold to space and still match; ZWNJ/ZWJ/SHY/word-joiner refuse", () => {
const plain = "hello world friends and more text here"
// U+200B ZERO WIDTH SPACE and U+FEFF are in the typography fold to space.
expect(verifyQuote(plain, "hello​ world friends and more").ok).toBe(true)
expect(verifyQuote(plain, "hello world friends and more").ok).toBe(true)
// An invisible join inside a word folds to a word break (current behaviour).
expect(verifyQuote("hello​world friends and more text here", "hello world friends and more").ok).toBe(true)
// U+200C/U+200D/U+00AD/U+2060 are NOT folded: the quote is refused.
for (const invisible of ["‌", "‍", "­", "⁠"]) {
const quote = `hello${invisible} world friends and more`
expect([JSON.stringify(invisible), verifyQuote(plain, quote)]).toMatchObject([
JSON.stringify(invisible),
{ ok: false },
])
expect(verifyQuote(plain, quote).reason).toContain("not found verbatim")
// And the reverse direction refuses too: the source keeps the char.
expect(verifyQuote(`hello${invisible} world friends and more text here`, plain.slice(0, 30)).ok).toBe(false)
}
})
test("`..` stays literal, `...`/`…` split into ordered fragments, `....` leaves a stray dot", () => {
// Two dots are code/punctuation, never an ellipsis.
expect(verifyQuote("version 1..2 released today for all users", "version 1..2 released today").ok).toBe(true)
expect(verifyQuote("version 1..2 released today for all users", "version 1..2 ... released today for all").ok).toBe(
true,
)
// Three dots and the single-char ellipsis both split; order matters.
expect(verifyQuote(GATES, "Alpha gates keep ... honest under heavy load").ok).toBe(true)
expect(verifyQuote(GATES, "Alpha gates keep … honest under heavy load").ok).toBe(true)
expect(verifyQuote(GATES, "honest under heavy load ... Alpha gates keep").ok).toBe(false)
// Four dots are an ellipsis plus a literal dot: the stray dot is refused.
const four = verifyQuote(GATES, "Alpha gates keep .... honest under heavy load")
expect(four.ok).toBe(false)
expect(four.reason).toContain("not found verbatim")
})
test("entity-at-quote: the verifier does no entity decoding (the fetch layer decodes)", () => {
const amp = htmlToText("<p>Fish &amp; chips are tasty indeed yum.</p>")
const numeric = htmlToText("<p>Fish &#38; chips are tasty indeed yum.</p>")
// Both spellings cache to the same decoded bytes.
expect(amp.text).toBe("Fish & chips are tasty indeed yum.")
expect(numeric.text).toBe(amp.text)
// The decoded quote verifies; the still-encoded spellings are refused.
expect(verifyQuote(amp.text, "Fish & chips are tasty").ok).toBe(true)
for (const encoded of ["Fish &amp; chips are tasty", "Fish &#38; chips are tasty"]) {
const check = verifyQuote(amp.text, encoded)
expect([encoded, check.ok]).toEqual([encoded, false])
expect(check.reason).toContain("not found verbatim")
}
})
test("markdown-rendered quotes: emphasis and links strip on both sides", () => {
expect(verifyQuote("the **quick brown fox jumps** over fences high", "quick brown fox jumps over").ok).toBe(true)
expect(verifyQuote("the quick brown fox jumps over fences high", "the **quick brown fox jumps** over").ok).toBe(
true,
)
expect(verifyQuote("run __every__ `single` test file now please", "run every single test file now").ok).toBe(true)
expect(verifyQuote("[Bun](https://bun.sh) is fast and furious indeed", "Bun is fast and furious").ok).toBe(true)
expect(verifyQuote("Bun is fast and furious indeed ok", "[Bun](https://bun.sh) is fast and furious").ok).toBe(true)
})
test("short-fragment refusal: every ellipsis fragment and every single quote needs 12+ characters", () => {
expect(verifyQuote(GATES, "short").reason).toContain("too short")
expect(verifyQuote(GATES, "abcdefghijk").ok).toBe(false)
expect(verifyQuote("abcdefghijkl rest of the text here ok", "abcdefghijkl").ok).toBe(true)
const short = verifyQuote(GATES, "Alpha gates keep ... hi ... honest under heavy load")
expect(short.ok).toBe(false)
expect(short.reason).toContain("fragment")
expect(verifyQuote(GATES, "Alpha gates keep ... honest under heavy load").ok).toBe(true)
})
test("unicode normalisation: NFKC, curly quotes, dashes and NBSP fold before matching", () => {
expect(verifyQuote("filigree work is quite delicate indeed", "filigree work is quite").ok).toBe(true)
expect(verifyQuote("café society is quite lively today", "café society is quite").ok).toBe(true)
// Composed vs decomposed acute accents match after NFKC.
expect(verifyQuote("café society is quite lively today", "café society is quite").ok).toBe(true)
expect(verifyQuote("He said “hello world, friends” ok", 'said "hello world, friends"').ok).toBe(true)
expect(verifyQuote("fast — reliable service for all users", "fast - reliable service").ok).toBe(true)
expect(verifyQuote("hello world friends and more text", "hello world friends and more").ok).toBe(true)
// Case still matters: normalisation is not case folding.
expect(verifyQuote(GATES, "alpha gates keep agents honest").ok).toBe(false)
})
})
describe("auditor downgrade rules (#187: verbatim grounds, paraphrase and plants do not)", () => {
test("verbatim VERIFIED grounds; a paraphrase stays ungrounded (flagged, never re-tagged)", async () => {
const cache = sealed()
const source = await cache.put({
url: "https://gates.test/run",
text: "Mechanical gates catch regressions before review, which keeps autonomous agents honest under heavy load.",
provider: "fetch",
})
const good = await auditMarkdown(
`- Gates catch regressions early [VERIFIED: sha256:${source.meta.sha256} "catch regressions before review"]`,
cache,
)
expect(good.coverage).toMatchObject({ claims: 1, grounded: 1, ungrounded: 0 })
expect(good.passed).toBe(true)
expect(good.cited).toEqual([source.meta.sha256])
const paraphrased = await auditMarkdown(
`- Gates catch failures ahead of inspection [VERIFIED: sha256:${source.meta.sha256} "catch failures ahead of inspection"]`,
cache,
)
expect(paraphrased.coverage).toMatchObject({ claims: 1, grounded: 0, ungrounded: 1 })
expect(paraphrased.passed).toBe(false)
expect(paraphrased.claims[0]?.reason).toContain("not found verbatim")
})
test("a planted unsealed entry is refused by the sealed cache; the legacy constructor still reads it (#251)", async () => {
const text = "Planted findings about gate latency on reference hardware."
const hash = sha256(normalizeSourceText(text))
await mkdir(path.join(dir, "sources"), { recursive: true })
await writeFile(path.join(dir, "sources", `${hash}.md`), normalizeSourceText(text))
await writeFile(
path.join(dir, "sources", `${hash}.json`),
`${JSON.stringify({ sha256: hash, url: "https://evil.test/planted", retrieved: new Date().toISOString(), provider: "fetch", bytes: Buffer.byteLength(normalizeSourceText(text)) })}\n`,
)
expect(await sealed().get(hash)).toBeUndefined()
const audit = await auditMarkdown(
`- Planted latency wins [VERIFIED: sha256:${hash} "gate latency on reference"]`,
sealed(),
)
expect(audit.coverage).toMatchObject({ claims: 1, grounded: 0, ungrounded: 1 })
expect(audit.claims[0]?.reason).toContain("not in the cache")
expect(audit.passed).toBe(false)
// Current legacy behaviour (bug #251): without a state dir nothing is verified.
expect((await new SourceCache(path.join(dir, "sources")).get(hash))?.text).toContain("gate latency")
})
test("short and missing quotes stay ungrounded; substantive other tags ground, stubs are malformed", async () => {
const cache = sealed()
const source = await cache.put({
url: "https://gates.test/short",
text: "Mechanical gates catch regressions before review, which keeps autonomous agents honest under heavy load.",
provider: "fetch",
})
const audit = await auditMarkdown(
[
`- Too short [VERIFIED: sha256:${source.meta.sha256} "tiny"]`,
`- Missing quote [VERIFIED: no quote here]`,
`- Fewer tools to install [INFERRED: one binary covers the runner and the bundler]`,
`- Maybe faster [HYPOTHESIS: rerun the benchmark with n=30]`,
`- Nothing else found [NEGATIVE_KNOWLEDGE: searched for gate failures on ARM]`,
`- Stub [INFERRED: ok]`,
"- No tag at all",
].join("\n"),
cache,
)
// "tiny" fails verification (ungrounded); "no quote here" has no
// source/quote shape and the stub INFERRED body is malformed.
expect(audit.coverage).toMatchObject({
claims: 7,
grounded: 3,
ungrounded: 1,
untagged: 1,
malformed: 2,
})
expect(audit.passed).toBe(false)
})
})
packages/core/test/evidence-fuzz.test.ts
+90
View File
@@ -0,0 +1,90 @@
// #251.1: verifier fuzz corpus. Pins the quote verifier against the
// acceptance list in the issue: zero-width/bidi/compatibility characters,
// `..` vs `...` vs `....`, entities at the quote, and markdown-rendered
// quotes. Static review found no false accepts; these tests pin that.
import { describe, expect, test } from "bun:test"
import { normalizeForQuote, verifyQuote } from "../src/research/quote.ts"
const SRC = "The quick brown fox jumps over the lazy dog near the riverbank today."
describe("zero-width, bidi and compatibility characters (#251.1)", () => {
test("U+200B folds to a word break (documented TYPOGRAPHY fold, pinned)", () => {
// U+200B sits at the end of the folded whitespace range, so a quote
// carrying it matches a source with a plain space, and vice versa.
expect(verifyQuote(SRC, "quick brown fox\u200Bjumps over the").ok).toBe(true)
expect(verifyQuote("quick brown fox\u200Bjumps over the lazy dog today", "quick brown fox jumps over").ok).toBe(
true,
)
expect(normalizeForQuote("a\u200Bb")).toBe("a b")
})
test("joiners, soft hyphen, word joiner and bidi controls never fold: quotes carrying them refuse", () => {
// None of U+200C U+200D U+00AD U+2060 U+200E U+200F U+202A U+202C U+2066
// is in the fold table, so the normalised quote keeps the control and
// misses the source.
for (const ch of ["\u200C", "\u200D", "\u00AD", "\u2060", "\u200E", "\u200F", "\u202A", "\u202C", "\u2066"]) {
const check = verifyQuote(SRC, `quick brown fox${ch} jumps over the`)
expect([ch.codePointAt(0)!.toString(16), check.ok]).toEqual([ch.codePointAt(0)!.toString(16), false])
expect(check.reason).toContain("not found verbatim")
}
})
test("compatibility characters fold through NFKC (documented, pinned)", () => {
expect(verifyQuote("The \uFB01ligree parses correctly here", "filigree parses correctly").ok).toBe(true)
expect(verifyQuote("The ABC label is printed here", "\uFF21\uFF22\uFF23 label is printed").ok).toBe(true)
})
test("matching is case-sensitive: casing alone never folds", () => {
expect(verifyQuote(SRC, "QUICK BROWN FOX JUMPS OVER").ok).toBe(false)
})
})
describe("ellipsis edge cases: `..` vs `...` vs `....` (#251.1)", () => {
test("two dots are literal: no fragment split", () => {
expect(verifyQuote("Version 1..2 released yesterday morning", "Version 1..2 released").ok).toBe(true)
expect(verifyQuote("Version 1..2 released yesterday morning", "Version 1..3 released").ok).toBe(false)
})
test("three dots (or …) split into ordered fragments, each 12+ characters", () => {
expect(verifyQuote(SRC, "quick brown fox ... lazy dog near").ok).toBe(true)
expect(verifyQuote(SRC, "quick brown fox \u2026 lazy dog near").ok).toBe(true)
expect(verifyQuote(SRC, "lazy dog near ... quick brown fox").ok).toBe(false)
})
test("four dots are an ellipsis plus a literal dot that must match", () => {
// "...." splits on the first "...", leaving ". lazy dog near" starting
// with a dot the source does not have there.
const check = verifyQuote(SRC, "quick brown fox .... lazy dog near")
expect(check.ok).toBe(false)
expect(check.reason).toContain("not found verbatim")
})
})
describe("entities at the quote (#251.1)", () => {
test("no entity decoding on either side: `&`, `&amp;` and `&#38;` are distinct literals", () => {
const plain = "Fish & chips are tasty indeed"
const encoded = "Fish &amp; chips are tasty indeed"
const numeric = "Fish &#38; chips are tasty indeed"
expect(verifyQuote(plain, "Fish & chips are tasty").ok).toBe(true)
expect(verifyQuote(plain, "Fish &amp; chips are tasty").ok).toBe(false)
expect(verifyQuote(plain, "Fish &#38; chips are tasty").ok).toBe(false)
expect(verifyQuote(encoded, "Fish & chips are tasty").ok).toBe(false)
expect(verifyQuote(numeric, "Fish & chips are tasty").ok).toBe(false)
// Identical literals still match.
expect(verifyQuote(encoded, "Fish &amp; chips are tasty").ok).toBe(true)
})
})
describe("markdown-rendered quotes (#251.1)", () => {
test("links, emphasis and code spans strip on both sides (documented, pinned)", () => {
expect(verifyQuote("[Bun](https://bun.sh) is fast and furious today", "Bun is fast and furious").ok).toBe(true)
expect(verifyQuote("This is **very important** news today ok", "This is very important news").ok).toBe(true)
expect(verifyQuote("Run `bun add zod` to install it now", "Run bun add zod to install").ok).toBe(true)
// Emphasis marks in the quote match a plain source (documented strip).
expect(verifyQuote("Bun is fast and furious today", "**Bun** is fast and furious").ok).toBe(true)
})
test("a link target is not quotable text", () => {
expect(verifyQuote("[Bun](https://bun.sh) is fast and furious today", "https://bun.sh is fast and").ok).toBe(false)
})
})
packages/core/test/evidence-seal.test.ts
+81
View File
@@ -0,0 +1,81 @@
// #251.2: the URL index is sealed with the engine key. A sealed cache
// writes an `index.seal` sidecar (HMAC over the canonical index); `byUrl`
// refuses a tampered index instead of following it. A missing sidecar stays
// readable (caches written before sealing keep working); an unsealed cache
// writes no sidecar.
import { describe, expect, test } from "bun:test"
import { mkdtemp, rm, writeFile } from "node:fs/promises"
import { tmpdir } from "node:os"
import path from "node:path"
import { SourceCache } from "../src/research/cache.ts"
const put2 = async (cache: SourceCache) => {
const a = await cache.put({
url: "https://a.test/page",
text: "Alpha source text with enough length.",
provider: "fetch",
})
const b = await cache.put({
url: "https://b.test/page",
text: "Beta source text with enough length here.",
provider: "fetch",
})
return { a, b }
}
describe("sealed URL index (#251.2)", () => {
test("a sealed cache writes an HMAC sidecar next to index.json", async () => {
const dir = await mkdtemp(path.join(tmpdir(), "es-index-"))
const state = await mkdtemp(path.join(tmpdir(), "es-index-state-"))
try {
const sealed = new SourceCache(path.join(dir, "sealed"), state)
await put2(sealed)
const seal = await Bun.file(path.join(dir, "sealed", "index.seal")).text()
expect(seal.trim()).toMatch(/^[0-9a-f]{64}$/)
} finally {
await rm(dir, { recursive: true, force: true })
await rm(state, { recursive: true, force: true })
}
})
test("byUrl refuses a tampered index instead of serving the remapped entry", async () => {
const dir = await mkdtemp(path.join(tmpdir(), "es-index-tamper-"))
const state = await mkdtemp(path.join(tmpdir(), "es-index-tamper-state-"))
try {
const cache = new SourceCache(path.join(dir, "sources"), state)
const { a, b } = await put2(cache)
expect((await cache.byUrl("https://a.test/page"))?.meta.sha256).toBe(a.meta.sha256)
// Remap a.test to b's bytes without re-sealing the index.
const indexFile = path.join(dir, "sources", "index.json")
const index = JSON.parse(await Bun.file(indexFile).text()) as Record<string, string>
index["https://a.test/page"] = b.meta.sha256
await writeFile(indexFile, `${JSON.stringify(index, null, 2)}\n`)
expect(await cache.byUrl("https://a.test/page")).toBeUndefined()
// Direct reads by hash are unaffected.
expect((await cache.get(b.meta.sha256))?.text).toContain("Beta source")
} finally {
await rm(dir, { recursive: true, force: true })
await rm(state, { recursive: true, force: true })
}
})
test("an index written before sealing (no sidecar) stays readable", async () => {
const dir = await mkdtemp(path.join(tmpdir(), "es-index-legacy-"))
const state = await mkdtemp(path.join(tmpdir(), "es-index-legacy-state-"))
try {
const cache = new SourceCache(path.join(dir, "sources"), state)
const entry = await cache.put({
url: "https://old.test/page",
text: "Old source text with enough length here.",
provider: "fetch",
})
// An index from before sidecars existed carries sealed entries but no
// sidecar: the sealed reader keeps serving it by URL.
await rm(path.join(dir, "sources", "index.seal"), { force: true })
expect((await cache.byUrl("https://old.test/page"))?.meta.sha256).toBe(entry.meta.sha256)
} finally {
await rm(dir, { recursive: true, force: true })
await rm(state, { recursive: true, force: true })
}
})
})
packages/core/test/factory-concurrency.test.ts
+118
View File
@@ -0,0 +1,118 @@
// Concurrent runtime-state writes hold the lock (issue #179, §2).
//
// Two concurrent es_qa_verdict tool calls (functional + adversarial) both
// persist — no torn JSON, no lost verdict — and two concurrent journal steps
// both land. State reads during the race never observe a half-signed
// (state, sidecar) pair: a refusal would reject the Promise.all.
import { afterEach, beforeEach, describe, expect, test } from "bun:test"
import { mkdtemp } from "node:fs/promises"
import { tmpdir } from "node:os"
import path from "node:path"
import { recordApproval } from "../src/approval/index.ts"
import type { HumanSigner } from "../src/approval/keystore.ts"
import { sealHumanKey, unlockHumanKey } from "../src/approval/keystore.ts"
import { Factory } from "../src/factory/index.ts"
import { once, readJournal } from "../src/factory/journal.ts"
import { gateRunner } from "../src/gates/index.ts"
import { factoryLayout } from "../src/layout.ts"
import { esTools } from "../src/ops/tools.ts"
import { type Fixture, frontier, gitRepo, writeReport, writeSpecs } from "./helpers.ts"
let fx: Fixture
let signer: HumanSigner
const green = { passed: true, findings: [], summary: "green" } as const
const gates = async () => green
beforeEach(async () => {
fx = await gitRepo("es-stage-conc-")
await sealHumanKey("test-passphrase-1234", fx.state)
signer = await unlockHumanKey("test-passphrase-1234", fx.state)
})
afterEach(() => fx.cleanup())
const factory = () => new Factory(fx.root, { gates, stateDir: fx.state })
const approve = (stage: "frontier" | "spec") =>
recordApproval(fx.root, { stage, channel: "cli", approvedBy: "tester", signer })
async function toQa() {
const f = factory()
await f.begin("human:tester")
await f.writeFrontier("grill", frontier())
await approve("frontier")
await f.beginResearch("human:tester")
await writeReport(fx.root, fx.state)
await f.completeResearch("factory")
await writeSpecs(fx.root, ["alpha"])
await approve("spec")
await f.startBuild("factory")
const dir = (await f.activeWorktree())!
const { mkdir, writeFile } = await import("node:fs/promises")
await mkdir(path.join(dir, "src"), { recursive: true })
await mkdir(path.join(dir, "test"), { recursive: true })
await writeFile(path.join(dir, "test/alpha.test.ts"), `test("alpha", () => {})\n`)
await f.recordGateRun("es-programmer", {
passed: false,
findings: [],
summary: "red",
failedTests: ["test/alpha.test.ts"],
})
await writeFile(path.join(dir, "src/alpha.ts"), `export const alpha = 1\n`)
const done = await f.complete("es-programmer")
expect(done.ok).toBe(true)
expect((await f.read())!.stage).toBe("QA")
}
describe("concurrent runtime-state writes (issue #179)", () => {
test("two concurrent es_qa_verdict tool calls both persist; reads stay sealed", async () => {
await toQa()
const f = factory()
const tools = esTools({ root: fx.root, factory: f, stateDir: fx.state })
const verdict = tools.find((tool) => tool.name === "es_qa_verdict")!
const [functional, adversarial, seen] = await Promise.all([
verdict.execute({ verdict: "pass", notes: "criteria met" }, { agent: "es-qa-functional" }),
verdict.execute({ verdict: "pass", notes: "nothing broke" }, { agent: "es-qa-adversarial" }),
f.read(),
])
expect(typeof functional).toBe("string")
expect(typeof adversarial).toBe("string")
expect([functional, adversarial].some((text) => text.includes("RELEASE"))).toBe(true)
// The concurrent read saw a sealed, parseable snapshot (never torn):
// it may win the race (QA) or lose it (RELEASE); only progress matters.
expect(["QA", "RELEASE"]).toContain(seen?.stage)
const state = (await f.read())!
expect([state.stage, state.phases[0]!.status]).toEqual(["RELEASE", "passed"])
expect(state.phases[0]!.qa).toMatchObject({
functional: { verdict: "pass" },
adversarial: { verdict: "pass" },
})
})
test("two concurrent journal steps both land, in full; spend races keep every event", async () => {
await toQa()
const f = factory()
await Promise.all([
once(fx.root, "concurrent-a", async () => ({ side: "a" })),
once(fx.root, "concurrent-b", async () => ({ side: "b" })),
])
const steps = (await readJournal(fx.root)).map((entry) => entry.step)
expect(steps).toContain("concurrent-a")
expect(steps).toContain("concurrent-b")
// Concurrent spend charges never lose an event and never tear the file.
await Promise.all([f.recordSpend(1, true), f.recordSpend(2, true), f.read(), f.read()])
const state = (await f.read())!
expect(state.spend.usd).toBe(3)
expect(state.spend.events).toBe(2)
// The sidecar still verifies after the race (read would throw otherwise).
expect(state.stage).toBe("QA")
})
test("ten concurrent spend writes serialize without loss", async () => {
await toQa()
const f = factory()
await Promise.all(Array.from({ length: 10 }, (_, i) => f.recordSpend(1, i % 2 === 0)))
const state = (await f.read())!
expect(state.spend.usd).toBe(10)
expect(state.spend.events).toBe(10)
})
})
packages/core/test/factory-preset-205.test.ts
+95
View File
@@ -0,0 +1,95 @@
// Issue #205 integration (self-dogfood preset): seed a temp clone through
// library calls only and prove the gates stay untrusted until a human trusts
// them. Never runs a human-only verb (`es factory init`, `es trust`): the only
// human-only assertions are pipe-level policy checks via `invokesHumanOnly`,
// and no trust entry is ever written.
import { describe, expect, test } from "bun:test"
import { readFile } from "node:fs/promises"
import {
commandSetHash,
EsConfigSchema,
factoryLayout,
GatesConfigSchema,
initFromPreset,
invokesHumanOnly,
isTrusted,
RoadmapSchema,
runGates,
SELF_DOGFOOD_BASE_BRANCH,
} from "../src/index.ts"
import { gitRepo } from "./helpers.ts"
describe("self-dogfood preset integration (#205)", () => {
test("init seeds config, gates, roadmap and the issue note in a temp clone", async () => {
const fx = await gitRepo("es-205-preset-")
try {
const { written, baseBranch } = await initFromPreset(
fx.root,
"self-dogfood",
{ issue: 205, title: "Self-dogfood preset", body: "The factory builds this repo." },
{ actor: "human:tester" },
)
// Current behavior (#281 owns the base-selection question): the seed
// pins the release base to `rewrite`.
expect(baseBranch).toBe("rewrite")
expect(SELF_DOGFOOD_BASE_BRANCH).toBe("rewrite")
expect(written).toEqual([
".factory/config.json",
".factory/gates.json",
".factory/roadmap.json",
".factory/notes/idea-205.md",
])
const layout = factoryLayout(fx.root)
const config = EsConfigSchema.parse(JSON.parse(await readFile(layout.config, "utf8")))
expect(config.audit?.phase).toBe("required")
expect(config.pr).toBe("gh")
const gates = GatesConfigSchema.parse(JSON.parse(await readFile(layout.gates, "utf8")))
expect(gates.commands.map((command) => command.command)).toEqual([
"bun install --frozen-lockfile",
"bun run check",
"bun run docs:check",
])
const roadmap = RoadmapSchema.parse(JSON.parse(await readFile(layout.roadmap, "utf8")))
expect(roadmap.baseBranch).toBe("rewrite")
expect(roadmap.title).toContain("205")
expect(await readFile(`${fx.root}/.factory/notes/idea-205.md`, "utf8")).toContain("# #205:")
} finally {
await fx.cleanup()
}
})
test("preset gates are untrusted until a human trusts them (no trust performed)", async () => {
const fx = await gitRepo("es-205-trust-")
try {
await initFromPreset(
fx.root,
"self-dogfood",
{ issue: 205, title: "Self-dogfood preset", body: "The factory builds this repo." },
{ actor: "human:tester" },
)
const layout = factoryLayout(fx.root)
const gates = GatesConfigSchema.parse(JSON.parse(await readFile(layout.gates, "utf8")))
// Trust-hash refusal: the fresh state dir holds no entry for this set.
// No `trustProject` call happens in this test (that stays human-only).
const { hash } = await commandSetHash(fx.root, gates.commands)
expect(await isTrusted(fx.root, hash, fx.state)).toBe(false)
const report = await runGates({ root: fx.root, scope: "full", stateDir: fx.state })
expect(report.passed).toBe(false)
expect(report.findings[0]?.rule).toBe("trust/untrusted")
expect(report.findings[0]?.fixHint).toContain("es trust")
expect(report.checks.find((check) => check.id === "trust")?.status).toBe("fail")
// None of the preset commands ran: no per-command checks were recorded.
for (const id of ["install", "check", "docs-check"])
expect(report.checks.some((check) => check.id === id)).toBe(false)
} finally {
await fx.cleanup()
}
})
test("seeding and trusting stay human-only (policy assert only, nothing executed)", () => {
expect(invokesHumanOnly("es factory init --preset self-dogfood --issue 205")).toBe(true)
expect(invokesHumanOnly("es trust")).toBe(true)
expect(invokesHumanOnly("es factory begin")).toBe(false)
})
})
packages/core/test/failover-gaps.test.ts
+281
View File
@@ -0,0 +1,281 @@
// #253: failover proof gaps. Direct pins for `backoffFor` defaults, the
// `auth`-kind chain continuation (failover + cooldown, not a stop), and
// token-absence exhaustiveness: every gateway failure message, both
// tool-output paths and every cache write carry no token. Wire-level
// exclusivity (bytes on the socket) needs a live-network capture and stays
// out of scope for unit tests.
import { describe, expect, test } from "bun:test"
import { mkdtemp, readdir, rm } from "node:fs/promises"
import { tmpdir } from "node:os"
import path from "node:path"
import {
BackendError,
backoffFor,
runFetchChain,
runSearchChain,
type SourceBackend,
} from "../src/research/backends/index.ts"
import { ScraperSwarmClient } from "../src/research/backends/scraper-swarm.ts"
import { researchTools } from "../src/research/ops.ts"
import { sha256 } from "../src/util/hash.ts"
const TOKEN = "swarm_sec_probe_token"
const ENV = { ES_SCRAPER_SWARM_URL: "https://swarm.test", ES_SCRAPER_SWARM_TOKEN: TOKEN }
describe("backoffFor defaults (backends/index.ts:159-162, #253.1)", () => {
test("rate_limited without Retry-After cools down 60s; an explicit value wins", () => {
expect(backoffFor(new BackendError("rate_limited", "slow down"))).toBe(60_000)
expect(backoffFor(new BackendError("rate_limited", "slow down", { retryAfterMs: 5_000 }))).toBe(5_000)
expect(backoffFor(new BackendError("rate_limited", "slow down", { retryAfterMs: 0 }))).toBe(0)
})
test("every other kind cools down 30s", () => {
for (const kind of ["unavailable", "auth", "blocked", "upstream", "timeout"] as const)
expect([kind, backoffFor(new BackendError(kind, "x"))]).toEqual([kind, 30_000])
})
})
describe("auth-kind chain continuation (#253.2)", () => {
const goodSearch: SourceBackend = {
id: "good",
capabilities: { search: true, fetch: false },
available: () => true,
search: async (query: string) => [{ url: `https://good.test/${query}`, title: "Good", snippet: "ok" }],
}
const goodFetch: SourceBackend = {
id: "steady",
capabilities: { search: false, fetch: true },
available: () => true,
fetch: async (url: string) => ({ url, text: "Steady page body with enough text.", contentType: "text/plain" }),
}
test("a leading 401 fails over on search and cools the backend down", async () => {
const denied: SourceBackend = {
id: "gateway",
capabilities: { search: true, fetch: false },
available: () => true,
search: async () => {
throw new BackendError("auth", "gateway refused: HTTP 401", { status: 401 })
},
}
const cooldowns = new Map<string, { until: number; kind: BackendError["kind"] }>()
const result = await runSearchChain([denied, goodSearch], "q", {}, { now: () => 1_000_000, cooldowns })
expect(result.backend).toBe("good")
expect(result.attempts).toEqual([
{ backend: "gateway", ok: false, kind: "auth" },
{ backend: "good", ok: true },
])
expect(cooldowns.get("gateway")).toEqual({ until: 1_030_000, kind: "auth" })
})
test("a leading 403 fails over on fetch and cools the backend down", async () => {
const denied: SourceBackend = {
id: "gateway",
capabilities: { search: false, fetch: true },
available: () => true,
fetch: async () => {
throw new BackendError("auth", "gateway refused: HTTP 403", { status: 403 })
},
}
const cooldowns = new Map<string, { until: number; kind: BackendError["kind"] }>()
const result = await runFetchChain([denied, goodFetch], "https://x.test/p", {
now: () => 2_000_000,
cooldowns,
})
expect(result.backend).toBe("steady")
expect(result.attempts).toEqual([
{ backend: "gateway", ok: false, kind: "auth" },
{ backend: "steady", ok: true },
])
expect(cooldowns.get("gateway")).toEqual({ until: 2_030_000, kind: "auth" })
})
})
describe("token absence exhaustiveness (#253.3)", () => {
const jsonRpc = (body: unknown, status = 200, headers: Record<string, string> = {}) =>
new Response(JSON.stringify(body), {
status,
headers: { "content-type": "application/json", "x-swarm-contract": "1", ...headers },
})
const stubFetch = (respond: (url: string, init: RequestInit) => Response, seen: { auth: string }) =>
(async (url: string, init?: RequestInit) => {
seen.auth = String((init?.headers as Record<string, string> | undefined)?.Authorization ?? "")
return respond(String(url), init ?? {})
}) as unknown as typeof fetch
const client = (respond: (url: string, init: RequestInit) => Response, seen: { auth: string }) =>
new ScraperSwarmClient({ url: ENV.ES_SCRAPER_SWARM_URL, token: TOKEN, fetch: stubFetch(respond, seen) })
const failure = (promise: Promise<unknown>): Promise<Error> =>
promise.then(
() => new Error("no-throw"),
(error: unknown) => error as Error,
)
test("the Authorization header carries the token on search and fetch", async () => {
const seen = { auth: "" }
const ok = {
jsonrpc: "2.0",
id: 1,
result: { structuredContent: { results: [{ url: "https://a.test", title: "A", snippet: "s" }] } },
}
await client(() => jsonRpc(ok), seen).search("q", 1)
expect(seen.auth).toBe(`Bearer ${TOKEN}`)
const content = "# Title\n\nPage body text."
const page = {
jsonrpc: "2.0",
id: 1,
result: {
structuredContent: { content, content_sha256: sha256(content), title: "T", format: "markdown" },
},
}
await client(() => jsonRpc(page), seen).fetchPage("https://a.test")
expect(seen.auth).toBe(`Bearer ${TOKEN}`)
})
test("no gateway failure message carries the token", async () => {
const cases: Array<[string, (seen: { auth: string }) => Promise<unknown>, string]> = [
["http-401", (seen) => client(() => jsonRpc({ detail: "nope" }, 401), seen).search("q", 1), "auth"],
["http-403", (seen) => client(() => jsonRpc({ detail: "nope" }, 403), seen).fetchPage("https://a.test"), "auth"],
[
"http-429",
(seen) => client(() => jsonRpc({ detail: "slow" }, 429, { "retry-after": "7" }), seen).search("q", 1),
"rate_limited",
],
["http-500", (seen) => client(() => jsonRpc({ detail: "boom" }, 500), seen).search("q", 1), "upstream"],
[
"contract",
(seen) =>
client(() => jsonRpc({ jsonrpc: "2.0", id: 1, result: {} }, 200, { "x-swarm-contract": "9" }), seen).search(
"q",
1,
),
"upstream",
],
[
"ssrf-denied",
(seen) =>
client(
() =>
jsonRpc({
jsonrpc: "2.0",
id: 1,
result: { isError: true, structuredContent: { code: "ssrf_denied", message: "SSRF denied" } },
}),
seen,
).fetchPage("https://a.test"),
"blocked",
],
[
"blocked-by-policy",
(seen) =>
client(
() =>
jsonRpc({
jsonrpc: "2.0",
id: 1,
result: { isError: true, structuredContent: { code: "blocked_by_policy", message: "denied" } },
}),
seen,
).search("q", 1),
"blocked",
],
[
"upstream-timeout",
(seen) =>
client(
() =>
jsonRpc({
jsonrpc: "2.0",
id: 1,
result: { isError: true, structuredContent: { code: "upstream_timeout", message: "slow" } },
}),
seen,
).search("q", 1),
"timeout",
],
[
"tool-upstream",
(seen) =>
client(() => jsonRpc({ jsonrpc: "2.0", id: 1, error: { code: -32602, message: "bad params" } }), seen).search(
"q",
1,
),
"upstream",
],
[
"unreachable",
(seen) =>
new ScraperSwarmClient({
url: ENV.ES_SCRAPER_SWARM_URL,
token: TOKEN,
fetch: (async () => {
seen.auth = "unreached"
throw new TypeError("fetch failed")
}) as unknown as typeof fetch,
}).search("q", 1),
"unavailable",
],
]
for (const [name, run, kind] of cases) {
const seen = { auth: "" }
const error = await failure(run(seen))
if (!(error instanceof BackendError)) throw new Error(`${name}: expected a BackendError, got ${error}`)
expect([name, error.kind]).toEqual([name, kind])
expect([name, error.message.includes(TOKEN)]).toEqual([name, false])
}
})
test("search and fetch tool outputs and every cache write carry no token", async () => {
const dir = await mkdtemp(path.join(tmpdir(), "es-token-"))
const state = await mkdtemp(path.join(tmpdir(), "es-token-state-"))
try {
// Drive the real client through the tools: search (no content cached)
// then fetch (page cached sealed as scraper-swarm).
const content = "# Recorded\n\nRecorded body text for the token probe."
const searchBody = {
jsonrpc: "2.0",
id: 1,
result: { structuredContent: { results: [{ url: "https://a.test/p", title: "A", snippet: "s" }] } },
}
const pageBody = {
jsonrpc: "2.0",
id: 1,
result: {
structuredContent: { content, content_sha256: sha256(content), title: "P", format: "markdown" },
},
}
let n = 0
const replay = (async () => {
n++
return jsonRpc(n === 1 ? searchBody : pageBody)
}) as unknown as typeof fetch
const tools = researchTools({
root: dir,
env: ENV,
stateDir: state,
policy: async () => ({ root: dir }),
fetch: replay,
})
const search = tools.find((tool) => tool.name === "es_research_search")!
const searchText = await search.execute({ query: "token probe" }, { agent: "es-research-alpha" })
expect(searchText.includes(TOKEN)).toBe(false)
// The search chain resolves the default order; scraper-swarm leads when
// configured, so this output names the serving backend.
expect(searchText).toContain("scraper-swarm results for")
const fetchTool = tools.find((tool) => tool.name === "es_research_fetch")!
const fetchText = await fetchTool.execute({ url: "https://a.test/p" }, { agent: "es-research-alpha" })
expect(fetchText.includes(TOKEN)).toBe(false)
// Every cache write (meta JSON) is token-free.
const cacheDir = path.join(dir, ".factory/research/sources")
for (const name of await readdir(cacheDir)) {
if (!name.endsWith(".json") || name === "index.json") continue
const raw = await Bun.file(path.join(cacheDir, name)).text()
expect([name, raw.includes(TOKEN)]).toEqual([name, false])
}
const indexRaw = await Bun.file(path.join(cacheDir, "index.json")).text()
expect(indexRaw.includes(TOKEN)).toBe(false)
} finally {
await rm(dir, { recursive: true, force: true })
await rm(state, { recursive: true, force: true })
}
})
})
packages/core/test/failover-loopback.test.ts
+709
View File
@@ -0,0 +1,709 @@
// #188 / #253: failover-chain proof gaps against loopback stubs.
// Real HTTP to 127.0.0.1 only (gateway + SearXNG stubs below); no external
// network. Proves, over the wire: 503 failover + order, 429 with/without
// Retry-After (cooldown skip then retry), auth continuation, blocked stopping
// both chains, sealed provenance, and that the gateway token never leaves its
// Authorization header (every BackendError site + every cache write).
import { afterAll, beforeAll, describe, expect, test } from "bun:test"
import { mkdtemp, readdir, readFile, rm } from "node:fs/promises"
import { tmpdir } from "node:os"
import path from "node:path"
import {
BackendError,
type BackendErrorKind,
backoffFor,
braveBackend,
runFetchChain,
runSearchChain,
type SourceBackend,
searxngBackend,
} from "../src/research/backends/index.ts"
import { ScraperSwarmClient, scraperSwarmBackend } from "../src/research/backends/scraper-swarm.ts"
import { SourceCache } from "../src/research/cache.ts"
import { researchTools } from "../src/research/ops.ts"
import { sha256 } from "../src/util/hash.ts"
/** Grep-able canary, never a real secret: every negative asserts its absence. */
const TOKEN = "swarm_canary_9f8e7d6c5b4a"
type GatewayMode =
| "success-search"
| "success-fetch"
| "http-503"
| "http-429-retry"
| "http-429-bare"
| "http-401"
| "contract-2"
| "no-body"
| "rpc-error"
| "wrong-id"
| "blocked"
| "timeout-tool"
| "upstream-tool"
| "no-results"
| "no-content"
| "sha-mismatch"
type SearxngMode = "success" | "http-503" | "http-429-retry" | "http-429-bare"
let gatewayMode: GatewayMode = "success-search"
let searxngMode: SearxngMode = "success"
let gatewayCalls: Array<{ name: string; auth: string | null }> = []
let searxngHits = 0
let gateway: ReturnType<typeof Bun.serve>
let searxng: ReturnType<typeof Bun.serve>
const FETCH_CONTENT = "Recorded failover body text with enough words to cache."
beforeAll(() => {
gateway = Bun.serve({
port: 0,
hostname: "127.0.0.1",
async fetch(req) {
if (new URL(req.url).pathname !== "/mcp") return new Response("not found", { status: 404 })
const body = (await req.json().catch(() => undefined)) as
| { id?: unknown; params?: { name?: string; arguments?: { query?: string } } }
| undefined
const name = body?.params?.name ?? ""
const id = body?.id ?? 1
gatewayCalls.push({ name, auth: req.headers.get("authorization") })
const rpc = (result: unknown) =>
Response.json({ id, jsonrpc: "2.0", result }, { headers: { "x-swarm-contract": "1" } })
switch (gatewayMode) {
case "http-503":
return new Response("unavailable", { status: 503 })
case "http-429-retry":
return new Response("slow down", { status: 429, headers: { "retry-after": "2" } })
case "http-429-bare":
return new Response("slow down", { status: 429 })
case "http-401":
return new Response("denied", { status: 401 })
case "contract-2":
return Response.json(
{ id, jsonrpc: "2.0", result: { structuredContent: { results: [] } } },
{ headers: { "x-swarm-contract": "2" } },
)
case "no-body":
return Response.json({ hello: 1 }, { headers: { "x-swarm-contract": "1" } })
case "rpc-error":
return Response.json(
{ id, jsonrpc: "2.0", error: { code: -32602, message: "bad params" } },
{ headers: { "x-swarm-contract": "1" } },
)
case "wrong-id":
return Response.json(
{ id: 999_999, jsonrpc: "2.0", result: { structuredContent: { results: [] } } },
{ headers: { "x-swarm-contract": "1" } },
)
case "blocked":
return rpc({
content: [{ type: "text", text: "SSRF denied (fail-closed)" }],
isError: true,
structuredContent: { code: "ssrf_denied", message: "SSRF denied (fail-closed)" },
})
case "timeout-tool":
return rpc({
content: [{ type: "text", text: "engine timed out" }],
isError: true,
structuredContent: { code: "upstream_timeout", message: "engine timed out" },
})
case "upstream-tool":
return rpc({
content: [{ type: "text", text: "engine down" }],
isError: true,
structuredContent: { code: "upstream_error", message: "engine down" },
})
case "no-results":
return rpc({ structuredContent: { engine: "stub" } })
case "no-content":
return rpc({ structuredContent: { title: "No body" } })
case "sha-mismatch":
return rpc({
structuredContent: {
content: FETCH_CONTENT,
content_sha256: sha256("something else entirely"),
final_url: "https://example.com/recorded",
title: "Recorded",
content_type: "text/plain",
},
})
case "success-fetch":
return rpc({
structuredContent: {
content: FETCH_CONTENT,
content_sha256: sha256(FETCH_CONTENT),
final_url: "https://example.com/recorded",
title: "Recorded",
content_type: "text/plain",
},
})
case "success-search":
return rpc({
structuredContent: {
query: body?.params?.arguments?.query ?? "",
results: [
{
url: "https://gateway.test/docs",
title: "Gateway docs",
snippet: "Gateway snippet about the query.",
},
],
},
})
default:
return rpc({
structuredContent: {
query: body?.params?.arguments?.query ?? "",
results: [
{
url: "https://gateway.test/docs",
title: "Gateway docs",
snippet: "Gateway snippet about the query.",
},
],
},
})
}
},
})
searxng = Bun.serve({
port: 0,
hostname: "127.0.0.1",
fetch(req) {
const url = new URL(req.url)
if (url.pathname !== "/search") return new Response("not found", { status: 404 })
searxngHits++
if (searxngMode === "http-503") return new Response("unavailable", { status: 503 })
if (searxngMode === "http-429-retry")
return new Response("slow down", { status: 429, headers: { "retry-after": "2" } })
if (searxngMode === "http-429-bare") return new Response("slow down", { status: 429 })
return Response.json({
results: [
{
url: "https://searxng.test/failover-docs",
title: "Failover docs",
content: `About ${url.searchParams.get("q")}: gates keep agents honest.`,
},
],
})
},
})
})
afterAll(() => {
gateway?.stop(true)
searxng?.stop(true)
})
const reset = () => {
gatewayCalls = []
searxngHits = 0
gatewayMode = "success-search"
searxngMode = "success"
}
/** Explicit env: ambient keys must not leak in (brave stays unavailable). */
const envFor = (): NodeJS.ProcessEnv => ({
ES_SCRAPER_SWARM_URL: `http://127.0.0.1:${gateway.port}`,
ES_SCRAPER_SWARM_TOKEN: TOKEN,
SEARXNG_URL: `http://127.0.0.1:${searxng.port}`,
})
const clock = (start: number) => {
let current = start
return {
now: () => current,
advance: (ms: number) => {
current += ms
},
}
}
const countingFetch = (text: string): { backend: SourceBackend; calls: () => number } => {
let calls = 0
return {
calls: () => calls,
backend: {
id: "fallback",
capabilities: { search: false, fetch: true },
available: () => true,
fetch: async (url: string) => {
calls++
return { url, text, contentType: "text/plain" }
},
},
}
}
describe("backoffFor defaults (#253.1)", () => {
test("a 429 Retry-After value wins; a bare 429 cools down for 60s", () => {
expect(backoffFor(new BackendError("rate_limited", "slow", { retryAfterMs: 5_000 }))).toBe(5_000)
expect(backoffFor(new BackendError("rate_limited", "slow"))).toBe(60_000)
})
test("every other kind cools down for 30s", () => {
for (const kind of ["unavailable", "auth", "upstream", "timeout", "blocked"] as const)
expect([kind, backoffFor(new BackendError(kind, "x"))]).toEqual([kind, 30_000])
})
})
describe("loopback failover through the real backends", () => {
test("a 503 on the gateway fails over in order; brave is skipped unavailable", async () => {
reset()
gatewayMode = "http-503"
const cooldowns = new Map<string, { until: number; kind: BackendErrorKind }>()
const result = await runSearchChain(
[scraperSwarmBackend, braveBackend, searxngBackend],
"gates",
{
env: envFor(),
},
{ cooldowns },
)
expect(result.backend).toBe("searxng")
// Unavailable backends are reported first (skipped without a call), then
// the tried backends in order.
expect(result.attempts).toEqual([
{ backend: "brave", ok: false, kind: "unavailable" },
{ backend: "scraper-swarm", ok: false, kind: "upstream" },
{ backend: "searxng", ok: true },
])
expect(result.value[0]?.snippet).toContain("gates keep agents honest")
expect(gatewayCalls).toHaveLength(1)
expect(searxngHits).toBe(1)
})
test("a 429 with Retry-After cools down: skipped, then retried after it lapses", async () => {
reset()
gatewayMode = "http-429-retry"
const env = envFor()
const timer = clock(1_000_000)
const cooldowns = new Map<string, { until: number; kind: BackendErrorKind }>()
const first = await runSearchChain(
[scraperSwarmBackend, searxngBackend],
"gates",
{ env },
{
now: timer.now,
cooldowns,
},
)
expect(first.backend).toBe("searxng")
expect(first.attempts[0]).toMatchObject({ backend: "scraper-swarm", ok: false, kind: "rate_limited" })
expect(gatewayCalls).toHaveLength(1)
// Still cooling down: skipped without an HTTP call, same failover.
timer.advance(1_000)
const skipped = await runSearchChain(
[scraperSwarmBackend, searxngBackend],
"gates",
{ env },
{
now: timer.now,
cooldowns,
},
)
expect(skipped.backend).toBe("searxng")
expect(skipped.attempts[0]).toMatchObject({ backend: "scraper-swarm", ok: false, kind: "rate_limited" })
expect(gatewayCalls).toHaveLength(1)
// After the 2s Retry-After lapses the gateway is tried again.
timer.advance(1_500)
gatewayMode = "success-search"
const retried = await runSearchChain(
[scraperSwarmBackend, searxngBackend],
"gates",
{ env },
{
now: timer.now,
cooldowns,
},
)
expect(retried.backend).toBe("scraper-swarm")
expect(gatewayCalls).toHaveLength(2)
})
test("a 429 without Retry-After defaults to a 60s cooldown over the wire", async () => {
reset()
gatewayMode = "http-429-bare"
const env = envFor()
const timer = clock(2_000_000)
const cooldowns = new Map<string, { until: number; kind: BackendErrorKind }>()
const run = (at: number) => {
timer.advance(at)
return runSearchChain(
[scraperSwarmBackend, searxngBackend],
"gates",
{ env },
{
now: timer.now,
cooldowns,
},
)
}
await run(0)
expect(gatewayCalls).toHaveLength(1)
await run(59_000)
expect(gatewayCalls).toHaveLength(1)
gatewayMode = "success-search"
const retried = await run(1_500)
expect(retried.backend).toBe("scraper-swarm")
expect(gatewayCalls).toHaveLength(2)
})
test("a provider-side 429 maps with and without its Retry-After value", async () => {
reset()
searxngMode = "http-429-retry"
// Fresh cooldown maps: the module-shared default would carry the first
// 429's cooldown into the second call and skip the backend instead.
const retrying = await runSearchChain(
[searxngBackend],
"gates",
{ env: envFor() },
{
cooldowns: new Map(),
},
).then(
() => "no-throw",
(error: unknown) => error,
)
expect(retrying).toBeInstanceOf(BackendError)
expect(retrying).toMatchObject({ kind: "rate_limited", retryAfterMs: 2_000 })
searxngMode = "http-429-bare"
const bare = await runSearchChain(
[searxngBackend],
"gates",
{ env: envFor() },
{
cooldowns: new Map(),
},
).then(
() => "no-throw",
(error: unknown) => error,
)
expect(bare).toBeInstanceOf(BackendError)
expect(bare).toMatchObject({ kind: "rate_limited" })
expect((bare as BackendError).retryAfterMs).toBeUndefined()
expect(backoffFor(bare as BackendError)).toBe(60_000)
})
test("a safety block stops the search chain: SearXNG is never called", async () => {
reset()
gatewayMode = "blocked"
const failure = await runSearchChain(
[scraperSwarmBackend, searxngBackend],
"gates",
{
env: envFor(),
},
{ cooldowns: new Map() },
).then(
() => "no-throw",
(error: unknown) => error,
)
expect(failure).toBeInstanceOf(BackendError)
expect(failure).toMatchObject({ kind: "blocked" })
expect(searxngHits).toBe(0)
})
test("a safety block stops the fetch chain: the fallback is never called", async () => {
reset()
gatewayMode = "blocked"
const fallback = countingFetch("never served")
const failure = await runFetchChain([scraperSwarmBackend, fallback.backend], "https://example.com/x", {
env: envFor(),
cooldowns: new Map(),
}).then(
() => "no-throw",
(error: unknown) => error,
)
expect(failure).toMatchObject({ kind: "blocked" })
expect(fallback.calls()).toBe(0)
expect((failure as Error).message).toContain("SSRF denied")
})
test("search failover provenance names the serving backend; snippets are not cached", async () => {
reset()
gatewayMode = "http-503"
const dir = await mkdtemp(path.join(tmpdir(), "es-failover-"))
const state = await mkdtemp(path.join(tmpdir(), "es-failover-state-"))
try {
const env = envFor()
const tools = researchTools({
root: dir,
env,
stateDir: state,
policy: async () => ({ root: dir }),
chain: [scraperSwarmBackend, braveBackend, searxngBackend],
})
const search = tools.find((tool) => tool.name === "es_research_search")!
const text = await search.execute({ query: "gates" }, { agent: "es-research-alpha" })
expect(text).toContain("searxng results for")
// SearXNG results are snippets, not page content: nothing is cached under them.
expect(await readdir(path.join(dir, ".factory/research/sources")).catch(() => [] as string[])).toEqual([])
} finally {
await rm(dir, { recursive: true, force: true })
await rm(state, { recursive: true, force: true })
}
})
test("fetch success through the gateway records sealed scraper-swarm provenance", async () => {
reset()
gatewayMode = "success-fetch"
const dir = await mkdtemp(path.join(tmpdir(), "es-failover-fetch-"))
const state = await mkdtemp(path.join(tmpdir(), "es-failover-fetch-state-"))
try {
const tools = researchTools({
root: dir,
env: envFor(),
stateDir: state,
policy: async () => ({ root: dir }),
chain: [scraperSwarmBackend, searxngBackend],
})
const fetchTool = tools.find((tool) => tool.name === "es_research_fetch")!
const text = await fetchTool.execute(
{ url: "https://example.com/recorded" },
{
agent: "es-research-alpha",
},
)
const hash = /sha256:([0-9a-f]{64})/.exec(text)?.[1]
expect(hash).toBe(sha256(FETCH_CONTENT))
const stored = await new SourceCache(path.join(dir, ".factory/research/sources"), state).get(hash!)
expect(stored?.meta.provider).toBe("scraper-swarm")
expect(stored?.meta.seal).toMatch(/^[0-9a-f]{64}$/)
} finally {
await rm(dir, { recursive: true, force: true })
await rm(state, { recursive: true, force: true })
}
})
})
describe("auth continues the chain (#253.2)", () => {
test("a gateway HTTP 401 fails over to SearXNG and cools down", async () => {
reset()
gatewayMode = "http-401"
const env = envFor()
const timer = clock(3_000_000)
const cooldowns = new Map<string, { until: number; kind: BackendErrorKind }>()
const first = await runSearchChain(
[scraperSwarmBackend, searxngBackend],
"gates",
{ env },
{
now: timer.now,
cooldowns,
},
)
expect(first.backend).toBe("searxng")
expect(first.attempts[0]).toMatchObject({ backend: "scraper-swarm", ok: false, kind: "auth" })
expect(gatewayCalls).toHaveLength(1)
timer.advance(1_000)
const second = await runSearchChain(
[scraperSwarmBackend, searxngBackend],
"gates",
{ env },
{
now: timer.now,
cooldowns,
},
)
expect(second.backend).toBe("searxng")
expect(gatewayCalls).toHaveLength(1)
})
test("a 401 search error and a 403 fetch error both fail over with a cooldown", async () => {
let searchCalls = 0
let fetchCalls = 0
const authSearch: SourceBackend = {
id: "auth-search",
capabilities: { search: true, fetch: false },
available: () => true,
search: async () => {
searchCalls++
throw new BackendError("auth", "HTTP 401", { status: 401 })
},
}
const authFetch: SourceBackend = {
id: "auth-fetch",
capabilities: { search: false, fetch: true },
available: () => true,
fetch: async () => {
fetchCalls++
throw new BackendError("auth", "HTTP 403", { status: 403 })
},
}
const goodSearch: SourceBackend = {
id: "good-search",
capabilities: { search: true, fetch: false },
available: () => true,
search: async (query: string) => [{ url: `https://good.test/${query}`, title: "Good", snippet: "ok" }],
}
const goodFetch = countingFetch("served after auth")
const timer = clock(4_000_000)
const searchCooldowns = new Map<string, { until: number; kind: BackendErrorKind }>()
const searched = await runSearchChain(
[authSearch, goodSearch],
"q",
{},
{
now: timer.now,
cooldowns: searchCooldowns,
},
)
expect(searched.backend).toBe("good-search")
expect(searched.attempts[0]).toMatchObject({ backend: "auth-search", ok: false, kind: "auth" })
timer.advance(1_000)
await runSearchChain([authSearch, goodSearch], "q", {}, { now: timer.now, cooldowns: searchCooldowns })
expect(searchCalls).toBe(1)
const fetchCooldowns = new Map<string, { until: number; kind: BackendErrorKind }>()
const fetched = await runFetchChain([authFetch, goodFetch.backend], "https://good.test/x", {
now: timer.now,
cooldowns: fetchCooldowns,
})
expect(fetched.backend).toBe("fallback")
expect(fetched.attempts[0]).toMatchObject({ backend: "auth-fetch", ok: false, kind: "auth" })
timer.advance(1_000)
await runFetchChain([authFetch, goodFetch.backend], "https://good.test/x", {
now: timer.now,
cooldowns: fetchCooldowns,
})
expect(fetchCalls).toBe(1)
})
})
describe("the token never leaves its Authorization header (#253.3)", () => {
const client = () =>
new ScraperSwarmClient({
url: `http://127.0.0.1:${gateway.port}`,
token: TOKEN,
fetch: fetch as unknown as typeof fetch,
})
test("every gateway failure message is token-free", async () => {
const searchModes: GatewayMode[] = [
"http-401",
"http-429-bare",
"http-429-retry",
"http-503",
"contract-2",
"no-body",
"rpc-error",
"wrong-id",
"blocked",
"timeout-tool",
"upstream-tool",
"no-results",
]
const expectedKind: Record<string, BackendErrorKind> = {
"http-401": "auth",
"http-429-bare": "rate_limited",
"http-429-retry": "rate_limited",
"http-503": "upstream",
"contract-2": "upstream",
"no-body": "upstream",
"rpc-error": "upstream",
"wrong-id": "upstream",
blocked: "blocked",
"timeout-tool": "timeout",
"upstream-tool": "upstream",
"no-results": "upstream",
}
for (const mode of searchModes) {
gatewayMode = mode
const failure = await client()
.search("q", 1)
.then(
() => "no-throw",
(error: unknown) => error,
)
expect(failure).toBeInstanceOf(BackendError)
expect((failure as BackendError).kind).toBe(expectedKind[mode])
expect((failure as Error).message).not.toContain(TOKEN)
expect(JSON.stringify(failure)).not.toContain(TOKEN)
}
const fetchModes: GatewayMode[] = ["no-content", "sha-mismatch", "blocked"]
for (const mode of fetchModes) {
gatewayMode = mode
const failure = await client()
.fetchPage("https://example.com/x")
.then(
() => "no-throw",
(error: unknown) => error,
)
expect(failure).toBeInstanceOf(BackendError)
expect((failure as Error).message).not.toContain(TOKEN)
}
// An unreachable gateway and a missing config fail without the token too.
const down = new ScraperSwarmClient({ url: "http://127.0.0.1:1", token: TOKEN })
for (const failure of [
await down.search("q", 1).then(
() => "no-throw",
(error: unknown) => error,
),
await down.fetchPage("https://example.com/x").then(
() => "no-throw",
(error: unknown) => error,
),
]) {
expect(failure).toMatchObject({ kind: "unavailable" })
expect((failure as Error).message).not.toContain(TOKEN)
}
for (const failure of [
await scraperSwarmBackend.search!("q", { env: {} }).then(
() => "no-throw",
(error: unknown) => error,
),
await scraperSwarmBackend.fetch!("https://example.com/x", { env: {} }).then(
() => "no-throw",
(error: unknown) => error,
),
]) {
expect(failure).toMatchObject({ kind: "unavailable" })
expect((failure as Error).message).not.toContain(TOKEN)
}
})
test("search-path and fetch-path cache writes hold no token", async () => {
reset()
gatewayMode = "success-fetch"
const dir = await mkdtemp(path.join(tmpdir(), "es-token-"))
const state = await mkdtemp(path.join(tmpdir(), "es-token-state-"))
try {
const contentful: SourceBackend = {
id: "contentful",
capabilities: { search: true, fetch: false },
available: () => true,
search: async (_query: string) => [
{
url: "https://contentful.test/docs",
title: "Contentful docs",
snippet: "a snippet",
content: "Contentful page text about gates, long enough to cache as a snapshot.",
},
],
}
const tools = researchTools({
root: dir,
env: envFor(),
stateDir: state,
policy: async () => ({ root: dir }),
chain: [contentful, scraperSwarmBackend],
})
const search = tools.find((tool) => tool.name === "es_research_search")!
const searched = await search.execute({ query: "gates" }, { agent: "es-research-alpha" })
expect(searched).toContain("contentful results for")
expect(searched).toContain("sha256:")
const fetchTool = tools.find((tool) => tool.name === "es_research_fetch")!
await fetchTool.execute({ url: "https://example.com/recorded" }, { agent: "es-research-alpha" })
const sources = path.join(dir, ".factory/research/sources")
const files = await readdir(sources)
expect(files.length).toBeGreaterThan(0)
for (const file of files) expect(await readFile(path.join(sources, file), "utf8")).not.toContain(TOKEN)
// The only wire presence of the token is the Authorization header.
expect(gatewayCalls.length).toBeGreaterThan(0)
for (const call of gatewayCalls) expect(call.auth).toBe(`Bearer ${TOKEN}`)
} finally {
await rm(dir, { recursive: true, force: true })
await rm(state, { recursive: true, force: true })
}
})
})
packages/core/test/gates-drift.test.ts
+44
View File
@@ -0,0 +1,44 @@
// Control-drift visibility (#242): when a tampered control file halts the
// run, the halt names the gate rule (integrity/control-drift) and points at
// the gate summary holding the structured finding — a seat debugging the halt
// gets more than an opaque "control files changed".
import { afterEach, beforeEach, describe, expect, test } from "bun:test"
import { writeFile } from "node:fs/promises"
import path from "node:path"
import { recordApproval } from "../src/approval/index.ts"
import { type HumanSigner, sealHumanKey, unlockHumanKey } from "../src/approval/keystore.ts"
import { Factory, FactoryHalted } from "../src/factory/index.ts"
import { type Fixture, frontier, gitRepo } from "./helpers.ts"
const PASSPHRASE = "test-passphrase-1234"
let fx: Fixture
let signer: HumanSigner
let factory: Factory
beforeEach(async () => {
fx = await gitRepo("es-gdrift-")
await sealHumanKey(PASSPHRASE, fx.state)
signer = await unlockHumanKey(PASSPHRASE, fx.state)
factory = new Factory(fx.root, {
gates: async () => ({ passed: true, findings: [], summary: "green" }),
stateDir: fx.state,
})
})
afterEach(() => fx.cleanup())
describe("drift halt visibility", () => {
test("a drift halt names integrity/control-drift and the gate summary holding the finding", async () => {
await factory.begin("human:tester")
await factory.writeFrontier("grill", frontier())
await recordApproval(fx.root, { stage: "frontier", channel: "cli", approvedBy: "tester", signer })
await writeFile(path.join(fx.root, ".factory/gates.json"), '{"commands":[]}\n')
const error = await factory.recordGateRun("es-programmer", { passed: false, findings: [], summary: "red" }).then(
() => undefined,
(failure: unknown) => failure,
)
expect(error).toBeInstanceOf(FactoryHalted)
expect(String(error)).toContain("integrity/control-drift")
expect(String(error)).toContain("summary.json")
expect((await factory.read())?.stage).toBe("HALTED")
})
})
packages/core/test/gates.test.ts
+314 -1
View File
@@ -1,18 +1,27 @@
import { afterEach, beforeEach, describe, expect, test } from "bun:test"
import { mkdir, writeFile } from "node:fs/promises"
import { mkdir, mkdtemp, readFile, rm, writeFile } from "node:fs/promises"
import { tmpdir } from "node:os"
import path from "node:path"
import { type HumanSigner, sealHumanKey, unlockHumanKey } from "../src/approval/keystore.ts"
import {
checkBudgets,
commandSetHash,
complexity,
detectGates,
formatReport,
type GateReport,
GatesConfigSchema,
gitleaks,
installed,
isTrusted,
loadGatesConfig,
parseOutput,
recordWaiver,
runGates,
trustProject,
validateArtifacts,
} from "../src/index.ts"
import { canonicalPath } from "../src/trust/paths.ts"
import { bwrapAvailable, resetBwrapProbe, run } from "../src/util/proc.ts"
import { type Fixture, gitRepo } from "./helpers.ts"
@@ -139,6 +148,77 @@ describe("trust", () => {
expect((await gates({ scope: "full", config: changed })).findings[0]?.rule).toBe("trust/untrusted")
})
test("a forged trust entry is not trust: isTrusted verifies the human signature (#221.3)", async () => {
const cfg = config([{ id: "probe", kind: "test", command: "touch RAN" }])
const wanted = config([{ id: "probe", kind: "test", command: "touch FORGED" }])
const { hash } = await commandSetHash(fx.root, cfg.commands)
const forgedHash = (await commandSetHash(fx.root, wanted.commands)).hash
await trust(cfg)
expect(await isTrusted(fx.root, hash, fx.state)).toBe(true)
const file = path.join(fx.state, "trust.json")
const genuine = await readFile(file, "utf8")
const store = JSON.parse(genuine) as { projects: Record<string, { gates: Record<string, any> }> }
const project = Object.keys(store.projects)[0]!
const entry = store.projects[project]!.gates
const plant = (projects: Record<string, unknown>) => writeFile(file, JSON.stringify({ version: 3, projects }))
// A second human key: a real Ed25519 signature, but not this machine's.
const foreign = await mkdtemp(path.join(tmpdir(), "es-gates-foreign-"))
await sealHumanKey(PASSPHRASE, foreign)
const foreignSigner = await unlockHumanKey(PASSPHRASE, foreign)
const retargeted = {
hash: forgedHash,
lines: ["probe: touch FORGED"],
approvedBy: "x",
approvedAt: entry.approvedAt,
}
const forgeries: Record<string, Record<string, unknown>> = {
// The genuine signature replayed over another command set.
"replayed signature": { [project]: { gates: { ...entry, hash: forgedHash, lines: retargeted.lines } } },
"made-up signature": {
[project]: {
gates: { ...retargeted, signature: { alg: "ed25519", keyId: entry.signature.keyId, sig: "AAAA" } },
},
},
"no signature": { [project]: { gates: retargeted } },
"foreign key": {
[project]: {
gates: {
...retargeted,
signature: foreignSigner.sign({ trust: 1, project, kind: "gates", ...retargeted }),
},
},
},
}
try {
for (const [name, projects] of Object.entries(forgeries)) {
await plant(projects)
expect([name, await isTrusted(fx.root, forgedHash, fx.state)]).toEqual([name, false])
const report = await gates({ scope: "full", config: wanted })
expect([name, report.findings[0]?.rule]).toEqual([name, "trust/untrusted"])
expect(await Bun.file(path.join(fx.root, "FORGED")).exists()).toBe(false)
}
// The signature binds the project and the kind: the genuine entry moved
// to another project, or presented as hook trust, verifies for neither.
const other = await gitRepo("es-gates-other-")
try {
const otherProject = canonicalPath(other.root)
expect(project).toBe(canonicalPath(fx.root))
await plant({ [project]: { gates: entry, hooks: entry }, [otherProject]: { gates: entry } })
expect(await isTrusted(other.root, hash, fx.state)).toBe(false)
expect(await isTrusted(fx.root, hash, fx.state, "hooks")).toBe(false)
// Control: in the same planted file, the untouched entry still verifies.
expect(await isTrusted(fx.root, hash, fx.state)).toBe(true)
} finally {
await other.cleanup()
}
} finally {
await rm(foreign, { recursive: true, force: true })
}
})
test("a package script reachable from a command is part of the trusted hash", async () => {
await write("package.json", JSON.stringify({ scripts: { typecheck: "tsc" } }))
const cfg = config([{ id: "typecheck", kind: "typecheck", command: "npm run --silent typecheck" }])
@@ -189,6 +269,41 @@ describe("built-in checks", () => {
expect(waived.waived).toBeGreaterThan(0)
})
test("an expired signed waiver planted at gate time waives nothing (#221.1)", async () => {
await write("src/config.ts", `export const token = "${FAKE_TOKEN}"\n`)
// recordWaiver refuses a past expiry, so the record is hand-aged: the
// same fields it signs, under the real human key, one day apart.
const plant = async (expiresAt: Date) => {
const unsigned = {
version: 2 as const,
id: "aged-token",
rule: "security/secret-github-token",
files: "src/config.ts",
reason: "fixture value, not a real token",
approvedBy: "tester",
approvedAt: new Date(expiresAt.getTime() - 30 * 86_400_000).toISOString(),
expiresAt: expiresAt.toISOString(),
channel: "cli" as const,
auditHead: null,
}
await write(".factory/waivers/aged-token.json", JSON.stringify({ ...unsigned, signature: signer.sign(unsigned) }))
return gates({ touched: ["src/config.ts"], config: config([]) })
}
const expired = await plant(new Date(Date.now() - 86_400_000))
// The signature verifies, so this is not the forged path: no waiver/invalid.
expect(expired.findings.some((finding) => finding.rule === "waiver/invalid")).toBe(false)
expect(expired.findings.some((finding) => finding.rule === "security/secret-github-token")).toBe(true)
expect(expired.waived).toBe(0)
expect(expired.passed).toBe(false)
// Control: the identical record with a live expiry does waive, so the
// refusal above is the expiry and nothing else about the planted file.
const live = await plant(new Date(Date.now() + 86_400_000))
expect(live.findings.some((finding) => finding.rule === "security/secret-github-token")).toBe(false)
expect(live.waived).toBe(1)
})
test("budgets: behaviour change without a test change, file length, complexity", async () => {
await write("src/big.ts", Array.from({ length: 30 }, (_, i) => `export const v${i} = ${i}`).join("\n"))
await write(
@@ -448,3 +563,201 @@ describe("parsers", () => {
expect(scores.find((item) => item.name === "typed")?.score).toBe(2)
})
})
describe("built-in secret scan cache (#239)", () => {
test("a rescan re-reads only changed files: hits are counted, edits re-read, no count cap", async () => {
const { clearSecretScanCache, scanFileForSecrets, secretScanStats } = await import("../src/index.ts")
clearSecretScanCache()
await write("src/a.ts", `export const token = "${FAKE_TOKEN}"\n`)
await write("src/b.ts", "export const b = 1\n")
const first = await scanFileForSecrets(fx.root, "src/a.ts")
expect(first.some((finding) => finding.rule === "security/secret-github-token")).toBe(true)
expect(await scanFileForSecrets(fx.root, "src/b.ts")).toEqual([])
// Unchanged files are cache hits: identical findings, hits counted.
const before = secretScanStats()
expect(await scanFileForSecrets(fx.root, "src/a.ts")).toEqual(first)
expect(await scanFileForSecrets(fx.root, "src/b.ts")).toEqual([])
expect(secretScanStats().hits).toBe(before.hits + 2)
// An edit re-reads: the finding disappears after the secret is removed.
await write("src/a.ts", 'export const token = "rotated"\n')
expect(await scanFileForSecrets(fx.root, "src/a.ts")).toEqual([])
})
})
describe("proof gaps (#239)", () => {
test("format is a failing arm, not detection-only: unformatted files fail, formatted files pass", async () => {
await write("biome.json", "{}")
await write("src/a.ts", "export const x=1\n")
const cfg = config([
{
id: "format",
kind: "format",
command: `${BIOME} format --reporter=json`,
parser: "biome",
touchedArgs: true,
extensions: [".ts"],
},
])
await trust(cfg)
// The engine arm, not the sandbox: pin bwrap away so the command runs
// directly (containers without overlayfs cannot sandbox; see #239).
resetBwrapProbe(false)
try {
const red = await gates({ touched: ["src/a.ts"], config: cfg })
expect(red.checks.find((check) => check.id === "format")?.status).toBe("fail")
expect(red.findings.find((finding) => finding.rule === "format/biome")).toMatchObject({
file: "src/a.ts",
severity: "error",
})
await write("src/a.ts", "export const x = 1;\n")
const green = await gates({ touched: ["src/a.ts"], config: cfg })
expect(green.findings.some((finding) => finding.rule === "format/biome")).toBe(false)
} finally {
resetBwrapProbe()
}
})
test("typecheck runs at gate level: errors fail, clean files pass, present tools are not missing", async () => {
const TSC = path.resolve(import.meta.dir, "../../../node_modules/.bin/tsc")
await write("tsconfig.json", JSON.stringify({ compilerOptions: { strict: true, noEmit: true }, include: ["src"] }))
await write("src/a.ts", 'export const x: number = "nope"\n')
const cfg = config([
{ id: "typecheck", kind: "typecheck", command: `${TSC} --noEmit -p tsconfig.json`, parser: "tsc" },
])
await trust(cfg)
resetBwrapProbe(false)
try {
const red = await gates({ touched: ["src/a.ts"], config: cfg })
expect(red.checks.find((check) => check.id === "typecheck")?.status).toBe("fail")
expect(red.findings.some((finding) => finding.file === "src/a.ts" && finding.check === "typecheck")).toBe(true)
await write("src/a.ts", "export const x: number = 1\n")
const green = await gates({ touched: ["src/a.ts"], config: cfg })
expect(green.checks.find((check) => check.id === "typecheck")?.status).toBe("pass")
expect(green.findings.some((finding) => finding.rule === "gates/tool-missing")).toBe(false)
expect(green.findings.some((finding) => finding.rule === "gates/sandbox-missing")).toBe(false)
} finally {
resetBwrapProbe()
}
})
test("a failing test command fails the gate (fail arm, not pass-only)", async () => {
const cfg = config([{ id: "test", kind: "test", command: "bun -e 'process.exit(1)'", parser: "generic" }])
await trust(cfg)
resetBwrapProbe(false)
try {
const report = await gates({ scope: "full", config: cfg })
expect(report.checks.find((check) => check.id === "test")?.status).toBe("fail")
expect(report.testsFailed).toBe(true)
expect(report.findings.some((finding) => finding.rule === "test/command-failed")).toBe(true)
} finally {
resetBwrapProbe()
}
})
test("budget pass arms: small, simple, tested changes are quiet; diff-size has both arms", async () => {
await write("src/a.ts", "export const a = 1\n")
await write("src/a.test.ts", 'import { a } from "./a"\n')
const quiet = await gates({
touched: ["src/a.ts", "src/a.test.ts"],
config: config([], { budgets: { maxFileLines: 20, maxComplexity: 10 } }),
})
for (const rule of ["budget/file-length", "budget/complexity", "budget/test-with-behavior"])
expect([rule, quiet.findings.some((finding) => finding.rule === rule)]).toEqual([rule, false])
const budgets = {
maxFileLines: 800,
maxComplexity: 15,
maxDiffLines: 10,
requireDepJustification: false,
requireTestWithBehavior: false,
}
const over = await checkBudgets({ dir: fx.root, files: [], budgets, diff: { added: 6, removed: 5 } })
expect(over.find((finding) => finding.rule === "budget/diff-size")?.severity).toBe("error")
const under = await checkBudgets({ dir: fx.root, files: [], budgets, diff: { added: 5, removed: 5 } })
expect(under.some((finding) => finding.rule === "budget/diff-size")).toBe(false)
})
test("a phase that touches only source passes tool-config (pass arm)", async () => {
const base = await commit("base")
await write("src/a.ts", "export const a = 1\n")
const report = await gates({ scope: "full", base, phaseId: "p1", config: config([]) })
expect(report.findings.some((finding) => finding.rule === "integrity/tool-config-changed")).toBe(false)
})
test("a valid gates.json loads with no problem (pass arm)", async () => {
await write(".factory/gates.json", JSON.stringify({ commands: [] }))
const { config: loaded, problem } = await loadGatesConfig(fx.root)
expect(problem).toBeUndefined()
expect(loaded.commands).toEqual([])
})
test("artifact validation flags a malformed frontier and passes when artifacts are absent", async () => {
expect(await validateArtifacts(fx.root, fx.state)).toEqual([])
await write(".factory/frontier.json", "{}")
const findings = await validateArtifacts(fx.root, fx.state)
expect(findings.find((finding) => finding.rule === "schema/frontier")?.file).toBe(".factory/frontier.json")
})
test("formatReport renders the summary, the findings and the overflow pointer", () => {
const report: GateReport = {
passed: false,
findings: [
{
file: "src/a.ts",
line: 2,
rule: "lint/suspicious/noDebugger",
severity: "error",
message: "No debugger",
check: "lint",
},
],
totalFindings: 3,
waived: 0,
checks: [{ id: "lint", status: "fail", durationMs: 1, findings: 3 }],
summary: "Gates failed: 3 error(s)",
failedTests: [],
testsFailed: false,
logDir: "/tmp/logs",
}
const text = formatReport(report)
expect(text).toContain("Gates failed: 3 error(s)")
expect(text).toContain("src/a.ts:2 lint/suspicious/noDebugger")
expect(text).toContain("… 2 more in /tmp/logs/summary.json")
})
test("the lsp gate reports error diagnostics and passes on clean files", async () => {
await write("src/a.ts", "export const a = 1\n")
const red = await gates({
touched: ["src/a.ts"],
config: config([]),
lsp: {
diagnostics: async () => [
{ severity: 1, message: "boom", source: "ts", range: { start: { line: 0, character: 0 } } },
],
},
})
expect(red.findings.find((finding) => finding.rule === "lsp/ts")).toMatchObject({ file: "src/a.ts", line: 1 })
const green = await gates({
touched: ["src/a.ts"],
config: config([]),
lsp: { diagnostics: async () => [] },
})
expect(green.findings.some((finding) => finding.check === "lsp")).toBe(false)
})
test("gitleaks: empty input is a skip; without the binary it never runs", async () => {
expect(await gitleaks(fx.root, [])).toBeUndefined()
if (!installed("gitleaks")) expect(await gitleaks(fx.root, ["README.md"])).toBeUndefined()
})
test("control drift is an integrity/control-drift gate finding, not just a halt", async () => {
const { rebaseline } = await import("../src/trust/index.ts")
await rebaseline(fx.root, "human:tester")
await write(".factory/gates.json", '{"commands":[]}\n')
const report = await gates({ scope: "full", config: config([]) })
expect(report.findings.find((finding) => finding.rule === "integrity/control-drift")).toMatchObject({
file: ".factory",
severity: "error",
check: "integrity",
})
})
})
packages/core/test/harvest-192.test.ts
+199
View File
@@ -0,0 +1,199 @@
// Issue #192 (Darkharvest) integration tests — core side.
//
// - Harvester targeting a path outside the project is refused, with an
// in-project control showing allowed paths still work.
// - An Apache-header-only fixture (no LICENSE file) stays clean-room. The
// returned literal is recorded as-is: `Apache-2.0`, unverified
// (bug #264 observed the claim wording "unknown"; the enforced property —
// clean-room — holds, so this asserts current behavior, not the old word).
import { afterEach, describe, expect, test } from "bun:test"
import { mkdir, mkdtemp, rm, writeFile } from "node:fs/promises"
import { tmpdir } from "node:os"
import path from "node:path"
import {
buildMatrix,
checkVerdict,
DEFAULT_LICENSE_WHITELIST,
detectLicense,
harvestTools,
scanSource,
} from "../src/index.ts"
const dirs: string[] = []
const tmp = async (prefix: string) => {
const dir = await mkdtemp(path.join(tmpdir(), prefix))
dirs.push(dir)
return dir
}
afterEach(async () => {
for (const dir of dirs.splice(0)) await rm(dir, { recursive: true, force: true })
})
const write = async (root: string, file: string, content: string) => {
await mkdir(path.dirname(path.join(root, file)), { recursive: true })
await writeFile(path.join(root, file), content)
}
const MIT_LICENSE = `MIT License
Copyright (c) 2026 Test
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
`
describe("issue #192: out-of-project harvest is refused", () => {
test("plan, target and scan refuse an outside path; an in-project path works", async () => {
const root = await tmp("es-192-root-")
const outside = await tmp("es-192-outside-")
await write(outside, "key.txt", "SIGNING-KEY-DO-NOT-READ")
const tools = harvestTools({ root })
const plan = tools.find((tool) => tool.name === "es_harvest_plan")!
const target = tools.find((tool) => tool.name === "es_harvest_target")!
// Refused: a harvester plan that points outside the project.
await expect(
plan.execute(
{ objective: "x", candidates: [{ name: "evil", source: `local:${outside}` }], force: true },
{ agent: "harvester" },
),
).rejects.toThrow(/outside the project/)
// Refused: the callable verdict check, same seat, same outside path.
await expect(
target.execute({ objective: "x", targets: [`local:${outside}`] }, { agent: "harvester" }),
).rejects.toThrow(/outside the project/)
// Refused at the scan layer too (no bytes are read: the throw precedes I/O).
await expect(scanSource(root, "evil", { kind: "local", path: outside }, { regexOnly: true })).rejects.toThrow(
/outside the project/,
)
// Control: an in-project candidate plans and scans fine.
await write(root, "ok/LICENSE", MIT_LICENSE)
await write(root, "ok/src/a.ts", "export const a = 1\n")
const planned = await plan.execute(
{ objective: "x", candidates: [{ name: "ok", source: "local:ok" }], force: true },
{ agent: "harvester" },
)
expect(planned).toContain("ok")
const profile = await scanSource(root, "ok", { kind: "local", path: "ok" }, { regexOnly: true })
expect(profile.license).toMatchObject({ spdx: "MIT", verified: true, source: "license-file" })
})
})
describe("issue #192: Apache-header-only fixture is clean-room", () => {
// No LICENSE file anywhere in this fixture: one source file carries only
// `SPDX-License-Identifier: Apache-2.0`.
const setupHeaderOnly = async (root: string, dir = "header-only") => {
await write(root, `${dir}/src/a.ts`, "// SPDX-License-Identifier: Apache-2.0\nexport const a = 1\n")
}
test("detectLicense literal is Apache-2.0, unverified (bug #264); policy forces clean-room", async () => {
const root = await tmp("es-192-apache-")
await setupHeaderOnly(root)
const dir = path.join(root, "header-only")
const finding = await detectLicense(dir)
// Pinned literal: bug #264 notes the claim said "unknown" here, but the
// code returns the header id unverified. Assert current behavior so any
// change to the literal is a visible diff; the safety property below is
// what must never regress.
expect(finding.spdx).toBe("Apache-2.0")
expect(finding).toMatchObject({ source: "spdx-header", verified: false, family: "permissive" })
// Fail-closed: an unverified header can never authorize depend/vendor.
for (const proposed of ["depend", "vendor"] as const) {
const checked = checkVerdict(proposed, finding, DEFAULT_LICENSE_WHITELIST)
expect(checked.verdict).toBe("clean-room")
expect(checked.downgraded).toMatch(/unverified/)
// Dangerous misclassification guard: unknown/copyleft/unverified must
// never come back depend/vendor-able. This would be a FINDING, not a fix.
expect(checked.verdict).not.toBe("depend")
expect(checked.verdict).not.toBe("vendor")
}
})
test("scan → matrix downgrades a proposed depend to clean-room", async () => {
const root = await tmp("es-192-apache-flow-")
await setupHeaderOnly(root)
const tools = harvestTools({ root })
const call = (name: string, input: Record<string, unknown>) =>
tools.find((tool) => tool.name === name)!.execute(input, { agent: "harvester" })
await call("es_harvest_plan", {
objective: "vet the header-only lib",
candidates: [{ name: "header-only", source: "local:header-only" }],
})
const scanned = await call("es_harvest_scan", { candidate: "header-only" })
expect(scanned).toContain("Apache-2.0")
expect(scanned).toContain("UNVERIFIED")
const matrixText = await call("es_harvest_matrix", {
rows: [{ feature: "widget", cells: [{ candidate: "header-only", verdict: "depend" }] }],
})
expect(matrixText).toContain("clean-room")
const profiles = new Map([
[
"header-only",
{
version: 1 as const,
id: "header-only",
name: "header-only",
license: await detectLicense(path.join(root, "header-only")),
languages: {},
dependencies: { runtime: [], dev: [] },
size: { files: 0, bytes: 0, tokens: 0 },
graph: { internalEdges: 0, external: 0, parsers: { treeSitter: 0, regex: 0 } },
provenance: {},
warnings: [],
scannedAt: new Date().toISOString(),
},
],
])
const built = buildMatrix(
["header-only"],
profiles,
[{ feature: "widget", cells: [{ candidate: "header-only", verdict: "depend" }] }],
DEFAULT_LICENSE_WHITELIST,
)
expect(built.matrix.rows[0]?.cells[0]?.verdict).toBe("clean-room")
expect(built.matrix.rows[0]?.cells[0]?.downgraded).toMatch(/unverified/)
expect(built.downgrades).toHaveLength(1)
})
test("es_harvest_target reports clean-room with the same literal", async () => {
const root = await tmp("es-192-apache-target-")
await setupHeaderOnly(root)
const target = harvestTools({ root }).find((tool) => tool.name === "es_harvest_target")!
const text = await target.execute(
{ objective: "can we depend on it", targets: ["local:header-only"] },
{ agent: "grill" },
)
const json = /--- verdicts \(JSON\) ---\n(\[[\s\S]*\])[\s]*$/.exec(text)?.[1]
expect(json).toBeDefined()
const verdicts = JSON.parse(json!) as Array<{
target: string
license: { spdx: string; family: string; verified: boolean }
verdict: string
}>
expect(verdicts).toHaveLength(1)
expect(verdicts[0]?.license).toMatchObject({ spdx: "Apache-2.0", verified: false })
expect(verdicts[0]?.verdict).toBe("clean-room")
})
})
packages/core/test/harvest.test.ts
+69 -1
View File
@@ -83,7 +83,7 @@ describe("SPDX detection", () => {
["JSON", "unknown", "unknown", false],
["MIT-0", "unknown", "unknown", false],
["X11", "unknown", "unknown", false],
["BSD-4-Clause", "BSD-4-Clause", "permissive", false],
["BSD-4-Clause", "BSD-4-Clause", "unknown", false],
["BUSL-1.1", "BUSL-1.1", "unknown", false],
["Elastic-2.0", "Elastic-2.0", "unknown", false],
])("%s text → %s (%s, verified %p)", (file, spdx, family, verified) => {
@@ -166,6 +166,53 @@ describe("SPDX detection", () => {
const unknown = await detectLicense(root)
expect(unknown).toMatchObject({ spdx: "unknown", verified: false })
})
test("a single file holding MIT+GPL reads as GPL-3.0, not LGPL (#264.1)", () => {
// The GPL-3.0 tail names the Lesser GPL, which used to win the date-line
// rules before the GPL-3.0 rule. The earliest license title in the bytes
// decides among the GPL family now.
expect(identifyLicenseText(`${MIT}\n${GPL3}`)).toMatchObject({
spdx: "GPL-3.0",
family: "copyleft",
verified: true,
})
expect(identifyLicenseText(`${GPL3}\n${MIT}`)).toMatchObject({ spdx: "GPL-3.0", family: "copyleft" })
})
test("mixed and header-only findings still fail closed at the policy (#264.2)", async () => {
// MIT+GPL in one file labels GPL-3.0 (copyleft), and a header-only Apache
// project labels Apache-2.0 unverified: both end up clean-room, never
// depend/vendor. The enforced property is mixed/unverified → clean-room.
const mixed = {
...identifyLicenseText(`${MIT}\n${GPL3}`)!,
source: "license-file" as const,
confidence: "high" as const,
}
expect(checkVerdict("depend", mixed, DEFAULT_LICENSE_WHITELIST).verdict).toBe("clean-room")
const header = await tmp("es-spdx-headeronly-")
await write(header, "src/a.ts", "// SPDX-License-Identifier: Apache-2.0\n export const x = 1\n")
const fromHeader = await detectLicense(header)
expect(fromHeader).toMatchObject({ spdx: "Apache-2.0", verified: false })
expect(checkVerdict("depend", fromHeader, DEFAULT_LICENSE_WHITELIST).verdict).toBe("clean-room")
})
test("BSD-4-Clause labels unknown, not permissive (#264.3)", () => {
// The advertising clause is an obligation the whitelist does not cover,
// so the family must not read as safe next to verified:false.
expect(identifyLicenseText(fixture("BSD-4-Clause"))).toMatchObject({
spdx: "BSD-4-Clause",
family: "unknown",
verified: false,
})
})
test("a top-level NOTICE without a license file warns about the blind spot (#264.4)", async () => {
const root = await tmp("es-spdx-notice-")
await write(root, "NOTICE", "Copyright 2026 Example Corp. All rights reserved.\n")
const noticed = await detectLicense(root)
expect(noticed).toMatchObject({ spdx: "unknown", verified: false })
expect(noticed.note ?? "").toContain("NOTICE")
})
})
describe("sources", () => {
@@ -350,6 +397,27 @@ describe("the fail-closed licence policy", () => {
expect(gpl).toMatchObject({ verdict: "clean-room", downgraded: expect.stringContaining("copyleft") })
expect(built.downgrades).toHaveLength(1)
})
test("model-supplied evidence is scrubbed of code spans before it reaches the spec (#267)", () => {
// matrix.ts used to spread cell.evidence verbatim into the clean-room
// spec, so a harvester pasting upstream code contaminated it
// mechanically. The scrub is the enforcement behind the "do not copy"
// instruction.
const profiles = new Map([["mit", profileFor(license({}), "mit")]])
const pasted = `rebuild the parser\n\`\`\`ts\nexport const stolen = 1\n\`\`\`\nsee \`internal()\` for details ${"x".repeat(600)}`
const built = buildMatrix(
["mit"],
profiles,
[{ feature: "parse", cells: [{ candidate: "mit", verdict: "clean-room", evidence: pasted }] }],
DEFAULT_LICENSE_WHITELIST,
)
const evidence = built.matrix.rows[0]!.cells[0]!.evidence!
expect(evidence).toContain("rebuild the parser")
expect(evidence).not.toContain("```")
expect(evidence).not.toContain("`")
expect(evidence).not.toContain("stolen")
expect(evidence.length).toBeLessThanOrEqual(500)
})
})
describe("the tool loop", () => {
packages/core/test/hooks-events.test.ts
+262
View File
@@ -0,0 +1,262 @@
// Issue #196 §2 — engine-level proof for every bridged hook event.
//
// Each event gets a hook that writes a marker file; the test drives the
// matching engine method and asserts the marker landed. This complements
// packages/opencode/test/hooks-events.test.ts, which drives the same events
// through the real in-process OpenCode v2 host.
//
// NOTE (bug #265): hook commands run UNSANDBOXED with the full parent env
// (run.ts uses inheritEnv:true and no sandbox knob). The "parent env is
// visible" test below pins that observable behaviour without changing it —
// Claude Code parity by design, not a sandbox.
import { afterEach, beforeEach, describe, expect, test } from "bun:test"
import { mkdir, mkdtemp, rm, writeFile } from "node:fs/promises"
import { tmpdir } from "node:os"
import path from "node:path"
import { type HumanSigner, sealHumanKey, unlockHumanKey } from "../src/approval/keystore.ts"
import { fromOpenCode, HookEngine, type HookSource, trustProject } from "../src/index.ts"
let root: string
let state: string
const PASSPHRASE = "test-passphrase-196-events"
let signer: HumanSigner
beforeEach(async () => {
root = await mkdtemp(path.join(tmpdir(), "es-hooks-events-"))
state = await mkdtemp(path.join(tmpdir(), "es-hooks-events-state-"))
await sealHumanKey(PASSPHRASE, state)
signer = await unlockHumanKey(PASSPHRASE, state)
await mkdir(path.join(root, ".opencode"), { recursive: true })
})
afterEach(async () => {
await rm(root, { recursive: true, force: true })
await rm(state, { recursive: true, force: true })
})
const sources = (): HookSource[] => [{ file: path.join(root, ".opencode/hooks.json"), scope: "project" }]
async function engine(hooks: Record<string, unknown>) {
await writeFile(path.join(root, ".opencode/hooks.json"), JSON.stringify({ hooks }))
let created = await HookEngine.create({ root, stateDir: state, sources: sources(), audit: false })
const status = created.status()
await trustProject(root, status.projectHash, status.projectLines, { stateDir: state, kind: "hooks", signer })
created = await HookEngine.create({ root, stateDir: state, sources: sources(), audit: false })
return created
}
const base = { sessionId: "ses_196", agent: "build" }
const marker = (name: string) => path.join(root, name)
const touch = (name: string, extra = "") => `touch ${marker(name)}${extra ? `; ${extra}` : ""}`
describe("hook events write marker files (issue #196 §2)", () => {
test("PreToolUse marker lands; exit 2 deny blocks the tool", async () => {
const hooks = await engine({
PreToolUse: [
{
matcher: "Write",
hooks: [
{
type: "command",
command: `${touch("PRE_196")}; input=$(cat); echo "$input" | grep -q "deny-pre-196" && { echo "pre-deny-196 blocked" >&2; exit 2; }; exit 0`,
},
],
},
],
})
const ok = await hooks.preToolUse(base, fromOpenCode("write", { path: "a.txt", content: "fine" }, root), "t1")
expect(ok.decision).toBeUndefined()
expect(await Bun.file(marker("PRE_196")).exists()).toBe(true)
const denied = await hooks.preToolUse(
base,
fromOpenCode("write", { path: "b.txt", content: "deny-pre-196" }, root),
"t2",
)
expect(denied).toMatchObject({ decision: "deny", reason: "pre-deny-196 blocked" })
})
test("PostToolUse marker lands with additionalContext", async () => {
const hooks = await engine({
PostToolUse: [
{
matcher: "Read",
hooks: [
{
type: "command",
command: `${touch("POST_196")}; echo '{"hookSpecificOutput":{"hookEventName":"PostToolUse","additionalContext":"POST-MARKER-196"}}'`,
},
],
},
],
})
const out = await hooks.postToolUse(base, fromOpenCode("read", { path: "a.txt" }, root), "t1", { ok: true }, false)
expect(await Bun.file(marker("POST_196")).exists()).toBe(true)
expect(out.additionalContext).toContain("POST-MARKER-196")
})
test("PostToolUseFailure marker lands when driven with failed=true (bug #268)", async () => {
const hooks = await engine({
PostToolUseFailure: [
{
matcher: "Read",
hooks: [
{
type: "command",
command: `${touch("FAIL_196")}; echo '{"hookSpecificOutput":{"hookEventName":"PostToolUseFailure","additionalContext":"FAILURE-MARKER-196"}}'`,
},
],
},
],
})
// failed=false must NOT run the PostToolUseFailure handler.
const ok = await hooks.postToolUse(base, fromOpenCode("read", { path: "a.txt" }, root), "t1", "content", false)
expect(ok.additionalContext).toEqual([])
expect(await Bun.file(marker("FAIL_196")).exists()).toBe(false)
// failed=true routes to the PostToolUseFailure event (engine.ts).
const failed = await hooks.postToolUse(
base,
fromOpenCode("read", { path: "missing.txt" }, root),
"t2",
"Error: file not found",
true,
)
expect(await Bun.file(marker("FAIL_196")).exists()).toBe(true)
expect(failed.additionalContext).toContain("FAILURE-MARKER-196")
})
test("UserPromptSubmit marker lands; block blocks", async () => {
const hooks = await engine({
UserPromptSubmit: [
{
hooks: [
{
type: "command",
command: `${touch("PROMPT_196")}; grep -q "block-prompt-196" && echo '{"decision":"block","reason":"prompt blocked 196"}' || echo "PROMPT-MARKER-196"`,
},
],
},
],
})
const out = await hooks.userPromptSubmit(base, "hello")
expect(await Bun.file(marker("PROMPT_196")).exists()).toBe(true)
expect(out.additionalContext).toContain("PROMPT-MARKER-196")
expect((await hooks.userPromptSubmit(base, "please block-prompt-196")).block).toContain("prompt blocked 196")
})
test("SessionStart marker lands for startup only", async () => {
const hooks = await engine({
SessionStart: [
{ matcher: "startup", hooks: [{ type: "command", command: touch("START_196") + "; echo START-MARKER-196" }] },
],
})
const out = await hooks.sessionStart(base, "startup")
expect(await Bun.file(marker("START_196")).exists()).toBe(true)
expect(out.additionalContext).toContain("START-MARKER-196")
await rm(marker("START_196"), { force: true })
expect((await hooks.sessionStart(base, "compact")).additionalContext).toEqual([])
expect(await Bun.file(marker("START_196")).exists()).toBe(false)
})
test("PermissionRequest marker lands; deny JSON denies", async () => {
const hooks = await engine({
PermissionRequest: [
{
matcher: "Read",
hooks: [
{
type: "command",
command: `${touch("PERM_196")}; echo '{"hookSpecificOutput":{"hookEventName":"PermissionRequest","decision":{"behavior":"deny","message":"perm-deny-196"}}}'`,
},
],
},
],
})
const out = await hooks.permissionRequest(base, fromOpenCode("read", { path: ".env" }, root))
expect(await Bun.file(marker("PERM_196")).exists()).toBe(true)
expect(out).toMatchObject({ behavior: "deny", message: "perm-deny-196" })
})
test("Stop marker lands; exit 2 blocks (emulated after the turn, ADVISORY for plain chat)", async () => {
const hooks = await engine({
Stop: [
{
hooks: [
{
type: "command",
// The engine passes stop_hook_active in the input; the hook itself
// suppresses the block on a continuation, as Claude Code does.
command: `${touch("STOP_196")}; input=$(cat); echo "$input" | grep -q '"stop_hook_active":true' && exit 0; echo "stop-continue-196" >&2; exit 2`,
},
],
},
],
})
const out = await hooks.stop(base, { stopHookActive: false, lastAssistantMessage: "done" })
expect(await Bun.file(marker("STOP_196")).exists()).toBe(true)
expect(out.block).toContain("stop-continue-196")
// stop_hook_active suppresses the block, as Claude Code does.
expect((await hooks.stop(base, { stopHookActive: true, lastAssistantMessage: "done" })).block).toBeUndefined()
})
})
describe("x-es.ShellPreExec rewrite AND deny (bug #268)", () => {
test("rewrite branch returns the replacement command", async () => {
const hooks = await engine({
"x-es.ShellPreExec": [
{
hooks: [
{
type: "command",
command: `input=$(cat); echo "$input" | grep -q "rewrite-shell-196" && echo '{"hookSpecificOutput":{"updatedInput":{"command":"echo shell-rewritten-196"}}}' || true`,
},
],
},
],
})
expect(await hooks.shellPreExec(base, { command: "echo rewrite-shell-196", cwd: root })).toMatchObject({
command: "echo shell-rewritten-196",
})
expect((await hooks.shellPreExec(base, { command: "echo plain", cwd: root })).command).toBeUndefined()
})
test("deny branch: exit 2 denies; deny wins over a simultaneous rewrite (command && !deny precedence)", async () => {
const hooks = await engine({
"x-es.ShellPreExec": [
{
hooks: [
{
type: "command",
command: `input=$(cat); echo "$input" | grep -q "deny-shell-196" && { echo "shell-deny-196 blocked" >&2; exit 2; }; echo "$input" | grep -q "both-shell-196" && echo '{"hookSpecificOutput":{"permissionDecision":"deny","permissionDecisionReason":"both denied 196","updatedInput":{"command":"echo should-not-run-196"}}}' || true`,
},
],
},
],
})
// Exit-2 deny carries the stderr reason.
const denied = await hooks.shellPreExec(base, { command: "echo deny-shell-196", cwd: root })
expect(denied.deny).toContain("shell-deny-196 blocked")
expect(denied.command).toBeUndefined()
// JSON deny with a simultaneous updatedInput.command: deny wins, command is suppressed.
const both = await hooks.shellPreExec(base, { command: "echo both-shell-196", cwd: root })
expect(both.deny).toContain("both denied 196")
expect(both.command).toBeUndefined()
})
})
describe("hooks run unsandboxed with parent env (bug #265, observable behaviour)", () => {
test("parent process env is visible inside a hook command", async () => {
const key = "ES_HOOK_PARENT_PROBE_196"
const value = `probe-196-${Date.now()}`
process.env[key] = value
try {
const hooks = await engine({
PreToolUse: [
{
matcher: "Bash",
hooks: [{ type: "command", command: `echo "$${key}" > ${marker("ENV_196")}` }],
},
],
})
await hooks.preToolUse(base, fromOpenCode("shell", { command: "echo hi" }, root), "t1")
expect((await Bun.file(marker("ENV_196")).text()).trim()).toBe(value)
} finally {
delete process.env[key]
}
})
})
packages/core/test/hooks.test.ts
+139 -3
View File
@@ -14,6 +14,7 @@ import {
trustProject,
verifyAuditChain,
} from "../src/index.ts"
import { resetBwrapProbe } from "../src/util/proc.ts"
let root: string
let state: string
@@ -44,14 +45,24 @@ async function script(name: string, body: string) {
}
async function engine(
hooks: Record<string, unknown>,
options: { trust?: boolean; generate?: (p: string) => Promise<string> } = {},
options: {
trust?: boolean
generate?: (p: string) => Promise<string>
sandbox?: "none" | "readonly"
offline?: boolean
} = {},
) {
await writeFile(path.join(root, ".opencode/hooks.json"), JSON.stringify({ hooks }))
const extra = {
...(options.generate ? { generate: options.generate } : {}),
...(options.sandbox !== undefined ? { sandbox: options.sandbox } : {}),
...(options.offline !== undefined ? { offline: options.offline } : {}),
}
let created = await HookEngine.create({
root,
stateDir: state,
sources: sources(),
...(options.generate ? { generate: options.generate } : {}),
...extra,
})
if (options.trust !== false) {
const status = created.status()
@@ -60,7 +71,7 @@ async function engine(
root,
stateDir: state,
sources: sources(),
...(options.generate ? { generate: options.generate } : {}),
...extra,
})
}
return created
@@ -100,6 +111,30 @@ describe("matching", () => {
).toBe(path.join(root, "x.ts"))
})
test("matcher corpus: exact lists vs unanchored regex (the Claude Code rule, #268)", () => {
// The rule (match.ts): "*", "" or omitted matches everything; a value of
// only [A-Za-z0-9_-, |] is an exact-match list; anything else is an
// unanchored JavaScript regex (so "Edit" never matches "NotebookEdit",
// while "Edit.*" does).
const cases: Array<[string | undefined, string, boolean]> = [
[undefined, "Bash", true],
["", "Bash", true],
["*", "Anything", true],
["Bash", "Bash", true],
["Bash", "bash", false],
["Edit|Write", "Write", true],
["Edit, Write", "Edit", true],
["Edit", "NotebookEdit", false],
["Edit.*", "NotebookEdit", true],
["^Edit", "NotebookEdit", false],
["mcp__memory__.*", "mcp__memory__create", true],
["mcp__memory", "mcp__memory__create", false],
["[invalid", "Bash", false],
]
for (const [matcher, value, expected] of cases)
expect(matcherMatches(matcher, [value]), `${String(matcher)} vs ${value}`).toBe(expected)
})
test("`if` rules use permission syntax over subcommands and paths", () => {
expect(subcommands("FOO=bar git push && npm test").parts).toEqual(["git push", "npm test"])
const tool = (command: string) => ({ toolNames: ["Bash", "shell"], toolInput: { command }, cwd: root })
@@ -288,6 +323,107 @@ describe("other events", () => {
})
expect((await hooks.shellPreExec(base, { command: "ls", cwd: root })).command).toBeUndefined()
})
test("x-es.ShellPreExec deny wins and drops the rewritten command (#268)", async () => {
const hooks = await engine({
"x-es.ShellPreExec": [
{
hooks: [
{
type: "command",
command: `echo '{"hookSpecificOutput":{"permissionDecision":"deny","permissionDecisionReason":"no shells here","updatedInput":{"command":"evil"}}}'`,
},
],
},
],
})
// deny + command: the `command && !deny` precedence keeps the deny only.
const denied = await hooks.shellPreExec(base, { command: "ls", cwd: root })
expect(denied.deny).toBe("no shells here")
expect(denied.command).toBeUndefined()
const blockers = await engine({
"x-es.ShellPreExec": [{ hooks: [{ type: "command", command: `echo "blocked reason" >&2; exit 2` }] }],
})
expect((await blockers.shellPreExec(base, { command: "ls", cwd: root })).deny).toBe("blocked reason")
})
test("PostToolUseFailure routes failing tools; PostToolUse does not fire on failure (#268)", async () => {
const hooks = await engine({
PostToolUse: [
{
matcher: "Write",
hooks: [{ type: "command", command: `echo '{"decision":"block","reason":"post ok"}'` }],
},
],
PostToolUseFailure: [
{
matcher: "Write",
hooks: [{ type: "command", command: `echo '{"decision":"block","reason":"post-failure lint"}'` }],
},
],
})
const write = fromOpenCode("write", { path: "a.ts", content: "" }, root)
const failed = await hooks.postToolUse(base, write, "t", "boom", true)
expect(failed.feedback).toEqual(["post-failure lint"])
const ok = await hooks.postToolUse(base, write, "t", { ok: true }, false)
expect(ok.feedback).toEqual(["post ok"])
})
test("hook commands run with the parent environment by design (Claude Code parity, #265)", async () => {
process.env.ES_HOOK_PARENT_ENV_PIN = "pinned-123"
try {
const hooks = await engine({
UserPromptSubmit: [{ hooks: [{ type: "command", command: 'echo "pin=$ES_HOOK_PARENT_ENV_PIN"' }] }],
})
expect(await hooks.userPromptSubmit(base, "hi")).toMatchObject({ additionalContext: ["pin=pinned-123"] })
} finally {
delete process.env.ES_HOOK_PARENT_ENV_PIN
}
})
test("SessionStart routes every source; an unmatched source is an explicit notice, never silent (#289)", async () => {
const hooks = await engine({
SessionStart: [
{ matcher: "startup", hooks: [{ type: "command", command: "echo 'project uses bun'" }] },
{ matcher: "resume", hooks: [{ type: "command", command: "echo 'welcome back'" }] },
],
})
expect(await hooks.sessionStart(base, "startup")).toMatchObject({ additionalContext: ["project uses bun"] })
expect(await hooks.sessionStart(base, "resume")).toMatchObject({ additionalContext: ["welcome back"] })
// compact/clear/fork have no matching hook here: explicit notice, not silence.
for (const source of ["compact", "clear", "fork"] as const) {
const result = await hooks.sessionStart(base, source)
expect(result.additionalContext).toEqual([])
expect(result.notices.join("\n")).toContain(`none match source "${source}"`)
}
// No SessionStart hook at all: silent (nothing to be confused about).
const bare = await engine({ PostToolUse: [{ hooks: [{ type: "command", command: "true" }] }] })
expect((await bare.sessionStart(base, "resume")).notices).toEqual([])
})
test("the engine sandbox knob reaches hook subprocesses: gates fail closed, observers fail open (#265)", async () => {
resetBwrapProbe(false)
try {
const hooks = await engine(
{
PreToolUse: [
{ matcher: "Bash", hooks: [{ type: "command", command: "true", "x-es-policy": "gate" }] },
{ matcher: "Read", hooks: [{ type: "command", command: "true" }] },
],
},
{ sandbox: "readonly" },
)
const gated = await hooks.preToolUse(base, bash("ls"), "t1")
expect(gated.decision).toBe("deny")
expect(gated.reason).toContain("bwrap")
const observed = await hooks.preToolUse(base, fromOpenCode("read", { path: "a" }, root), "t2")
expect(observed.decision).toBeUndefined()
expect(observed.notices.join("\n")).toContain("bwrap")
} finally {
resetBwrapProbe()
}
})
})
describe("loading", () => {
packages/core/test/improve-204.test.ts
+273
View File
@@ -0,0 +1,273 @@
// #204 integration: harvest + distill over several unit-constructed run
// dirs (one with a broken audit chain); scrubbed output and the
// code-disposes gate dropping fabricated evidence with reasons.
//
// Unit-constructed only: no real factory runs, no timing/perf assertions
// (#155 is flaky by nature and tracked separately), no --open-pr.
import { describe, expect, test } from "bun:test"
import { appendFile, mkdir, mkdtemp, readdir, readFile, rm, stat, writeFile } from "node:fs/promises"
import { tmpdir } from "node:os"
import path from "node:path"
import {
appendAuditEntry,
clusterTelemetry,
disposedWithReasons,
disposeReason,
distillProposals,
factoryLayout,
harvestRun,
harvestTelemetry,
signEngineFile,
TelemetrySchema,
undisposed,
writeDropped,
writeProposals,
} from "../src/index.ts"
// Planted secrets (fake values in the style of the existing scrub tests).
// Every one must be absent from harvested telemetry.
const GH_A = "ghp_plantedHarvest001AAA"
const GH_BROKEN = "ghu_BrokenChain999aaa"
const SK_ANT = "sk-ant-planted0000XYZ"
const AKIA = "AKIAIOSFODNN7EXAMPLE"
const SLACK = "xoxb-12345-abcdef"
const BEARER = "eyJhbGciOiJIUzI1NiJ9"
const PEM_BODY = "MIIEowIBAAKCAQEAplanted"
const HUNTER = "hunter2-planted"
const QA_REASON = "lint gate red"
const GATE_RULE = "lint/no-unused-vars"
async function writeState(root: string, id: string, haltReason: string): Promise<void> {
const file = factoryLayout(root).state
await mkdir(path.dirname(file), { recursive: true })
await writeFile(
file,
`${JSON.stringify(
{
version: 2,
runId: id,
stage: "HALTED",
mode: "build",
createdAt: "2026-10-04T00:00:00.000Z",
updatedAt: "2026-10-04T01:00:00.000Z",
spend: { usd: 1, estimated: false, events: 5 },
phases: [
{
id: "p1",
title: "phase",
status: "failed",
goalHash: "a".repeat(64),
failures: 2,
replanned: false,
history: [],
},
],
halt: { reason: haltReason, at: "2026-10-04T01:00:00.000Z", from: "QA" },
audits: [],
},
null,
2,
)}\n`,
)
}
async function writeGateSummary(root: string, id: string): Promise<void> {
const dir = path.join(factoryLayout(root).runs, id, "gates", "2026-10-04T00-30-full")
await mkdir(dir, { recursive: true })
await writeFile(
path.join(dir, "summary.json"),
`${JSON.stringify(
{
passed: false,
findings: [
{ file: "src/x.ts", line: 3, rule: GATE_RULE, severity: "error", message: "unused", check: "lint" },
{ file: "src/y.ts", line: 9, rule: GATE_RULE, severity: "error", message: "unused", check: "lint" },
],
},
null,
2,
)}\n`,
)
}
async function writeChain(root: string, haltReason: string): Promise<void> {
await appendAuditEntry(root, { actor: "system", action: "stage.build" })
await appendAuditEntry(root, { actor: "agent:qa", action: "qa.fail", payload: { phase: "p1", reason: QA_REASON } })
await appendAuditEntry(root, { actor: "system", action: "stage.halted", payload: { reason: haltReason } })
}
/** Two sealed runs plus one sealed run with a broken chain, and an evals dir. */
async function buildFixture(): Promise<{
parent: string
stateDir: string
runs: string[]
broken: string
evals: string
cleanup(): Promise<void>
}> {
const parent = await mkdtemp(path.join(tmpdir(), "es-204-core-"))
const stateDir = await mkdtemp(path.join(tmpdir(), "es-204-core-state-"))
const haltA = `phase p1 failed: GITHUB_TOKEN=${GH_A} with Bearer ${BEARER}`
const haltB =
`phase p1 failed: api-key: ${HUNTER} plus ` +
`-----BEGIN RSA PRIVATE KEY-----\n${PEM_BODY}\n-----END RSA PRIVATE KEY-----`
const haltBroken = `phase p1 failed: slack ${SLACK} and ${GH_BROKEN}`
const ids = ["run-a", "run-b"] as const
const halts: Record<string, string> = { "run-a": haltA, "run-b": haltB }
for (const id of ids) {
const root = path.join(parent, id)
await writeState(root, id, halts[id]!)
await writeGateSummary(root, id)
await writeChain(root, halts[id]!)
await signEngineFile(factoryLayout(root).state, stateDir)
}
const broken = path.join(parent, "run-broken")
await writeState(broken, "run-broken", haltBroken)
await writeGateSummary(broken, "run-broken")
await writeChain(broken, haltBroken)
await signEngineFile(factoryLayout(broken).state, stateDir)
// Break the chain with a corrupt trailing line: the valid prefix stays so
// the run still contributes its gate/qa signals, but verification fails.
await appendFile(factoryLayout(broken).audit, "not-json\n")
const evals = path.join(parent, "evals")
await mkdir(evals, { recursive: true })
await writeFile(
path.join(evals, "agg.json"),
`${JSON.stringify(
{
at: "2026-10-04T00-00-00-000Z",
model: "test/model",
results: [
{ id: "case-x", pass: false, failures: ["gate lint red", "gate lint red", `leaked ${SK_ANT}`], steps: 9 },
{ id: "case-y", pass: false, failures: ["gate lint red", `key ${AKIA} here`], steps: 4 },
{ id: "case-ok", pass: true, failures: [], steps: 3 },
],
},
null,
2,
)}\n`,
)
return {
parent,
stateDir,
runs: [path.join(parent, "run-a"), path.join(parent, "run-b"), broken],
broken,
evals,
async cleanup() {
await rm(parent, { recursive: true, force: true })
await rm(stateDir, { recursive: true, force: true })
},
}
}
async function walkFiles(dir: string): Promise<string[]> {
const out: string[] = []
const visit = async (current: string): Promise<void> => {
for (const name of await readdir(current)) {
const full = path.join(current, name)
const info = await stat(full).catch(() => undefined)
if (info?.isDirectory()) await visit(full)
else out.push(full)
}
}
await visit(dir)
return out.sort()
}
describe("issue #204: harvest + distill over several runs incl. a broken chain", () => {
test("harvest is scrubbed, schema-valid; the broken chain is reported, never repaired", async () => {
const fix = await buildFixture()
try {
const brokenFile = factoryLayout(fix.broken).audit
const before = await readFile(brokenFile, "utf8")
const telemetry = await harvestTelemetry({ runs: fix.runs, evals: [fix.evals] }, { stateDir: fix.stateDir })
expect(TelemetrySchema.parse(telemetry)).toEqual(telemetry)
expect(telemetry.runs.map((run) => run.id)).toEqual(["run-a", "run-b", "run-broken"])
const text = JSON.stringify(telemetry)
for (const secret of [GH_A, GH_BROKEN, SK_ANT, AKIA, SLACK, BEARER, PEM_BODY, HUNTER])
expect(text).not.toContain(secret)
// Shape is preserved: names stay, only values become [redacted].
expect(text).toContain("GITHUB_TOKEN=")
expect(text).toContain("Bearer")
// The broken chain is reported with an error and left on disk untouched.
const brokenRun = telemetry.runs.find((run) => run.id === "run-broken")!
expect(brokenRun.auditChain.valid).toBe(false)
expect(typeof brokenRun.auditChain.error).toBe("string")
expect(await readFile(brokenFile, "utf8")).toBe(before)
// The sealed runs keep their valid chains and seals.
for (const id of ["run-a", "run-b"]) {
const run = telemetry.runs.find((entry) => entry.id === id)!
expect(run.auditChain.valid).toBe(true)
expect(run.seal).toEqual({ valid: true })
}
// Direct harvestRun agrees: broken is reported, not repaired.
const direct = await harvestRun(fix.broken, { stateDir: fix.stateDir })
expect(direct.auditChain.valid).toBe(false)
expect(await readFile(brokenFile, "utf8")).toBe(before)
} finally {
await fix.cleanup()
}
})
test("distill clusters all three kinds; fabricated evidence is dropped with reasons", async () => {
const fix = await buildFixture()
try {
const telemetry = await harvestTelemetry({ runs: fix.runs, evals: [fix.evals] }, { stateDir: fix.stateDir })
const thresholds = { minOccurrences: 2, minRuns: 2 }
const clusters = clusterTelemetry(telemetry, thresholds)
expect(clusters.length).toBeGreaterThanOrEqual(3)
const proposals = await distillProposals(telemetry, thresholds)
expect(proposals.length).toBeGreaterThanOrEqual(3)
expect([...new Set(proposals.map((proposal) => proposal.kind))].sort()).toEqual([
"domain-pack",
"gate-tuning",
"prompt-guidance",
])
expect(undisposed(proposals, telemetry)).toEqual(proposals)
// Fabricated evidence: a quote the harvester never said is disposed.
const [first] = proposals
const forged = { ...first!, cluster: { ...first!.cluster, evidence: ["run x: invented quote"] } }
const reasons = disposeReason(forged, telemetry)
expect(reasons.some((reason) => reason.includes("verbatim"))).toBe(true)
expect(undisposed([forged], telemetry)).toEqual([])
const dropped = disposedWithReasons([forged], telemetry)
expect(dropped).toHaveLength(1)
expect(dropped[0]!.reasons.some((reason) => reason.includes("verbatim"))).toBe(true)
// Hallucinated model numerals are disposed too.
const tainted = { ...first!, rationale: `${first!.rationale} As gpt-4 would suggest.` }
expect(disposeReason(tainted, telemetry).some((reason) => reason.includes("hallucinated model numeral"))).toBe(
true,
)
// Dropped proposals are reported with reasons, byte-identical on rerun.
const out = await mkdtemp(path.join(tmpdir(), "es-204-distill-"))
try {
const rendered = new Map(
await Promise.all(
proposals.map(async (proposal) => {
const { files } = await import("../src/improve/distill.ts").then((mod) =>
mod.renderProposalFiles(proposal.cluster),
)
return [proposal.id, files] as const
}),
),
)
const written = await writeProposals(out, proposals, rendered)
expect(written.length).toBe(proposals.length)
const droppedFile = await writeDropped(out, dropped)
expect(await readFile(droppedFile, "utf8")).toContain("verbatim")
await writeDropped(out, dropped)
expect(await readFile(droppedFile, "utf8")).toContain("verbatim")
// Gate-tuning explains only: gates.md, never the control file.
const gate = proposals.find((proposal) => proposal.kind === "gate-tuning")!
expect(gate.files).toEqual(["gates.md"])
const files = await walkFiles(out)
expect(files.some((file) => path.basename(file) === "gates.json")).toBe(false)
} finally {
await rm(out, { recursive: true, force: true })
}
} finally {
await fix.cleanup()
}
})
})
packages/core/test/liveness.test.ts
+44
View File
@@ -9,6 +9,7 @@ import {
livenessLines,
progressPrint,
readLiveness,
researchLine,
STALE_AFTER_MS,
} from "../src/factory/liveness.ts"
import { Factory } from "../src/factory/machine.ts"
@@ -180,6 +181,31 @@ describe("headline", () => {
const qa = state({ stage: "QA", activePhase: "p1" })
expect(headline(qa, await live(qa, T0))).toStartWith("QA on p1 in progress")
})
test("STALE_AFTER_MS is ten minutes; the stuck hint starts exactly at the boundary (#248)", async () => {
expect(STALE_AFTER_MS).toBe(600_000)
const current = state()
const atBoundary = new Date(T0.getTime() + STALE_AFTER_MS)
expect(headline(current, await live(current, atBoundary))).toContain("No activity for 10m in RESEARCH")
const justUnder = new Date(T0.getTime() + STALE_AFTER_MS - 1_000)
expect(headline(current, await live(current, justUnder))).toBe(
"Research in progress: no seat is running; last activity 9m ago.",
)
})
test("done without a PR, SPEC and RELEASE wording (#248)", async () => {
const done = state({ stage: "DONE" })
expect(headline(done, await live(done, T0))).toBe("Done: the run is complete.")
const spec = state({ stage: "SPEC", runId: "run-20261008-020000-spec1" })
await updateSeats(fx.root, spec.runId, (seats) => {
seats.push(seat({ lastActivityAt: T0.toISOString() }))
})
expect(headline(spec, await live(spec, T0))).toBe("Spec writing is running: es-research-alpha started 0s ago.")
const release = state({ stage: "RELEASE", runId: "run-20261008-020000-rel1", updatedAt: T0.toISOString() })
expect(headline(release, await live(release, T0))).toBe(
"Release in progress: no seat is running; last activity 0s ago.",
)
})
})
describe("formatHuman", () => {
@@ -210,6 +236,24 @@ describe("formatHuman", () => {
expect(text).toContain("es-research-alpha running es_research_fetch 10s ago ses_ee6b70e18ffe")
expect(text).toContain("es-research-beta completed 50s ago ses_beta")
expect(text).toContain("research: 0 sources · alpha.md — · beta.md — · REPORT.md — · coverage not checked")
// The dashboard leads with the headline, then the run header (#248):
// order, not just containment.
expect(text.indexOf("Research is running: es-research-alpha")).toBeLessThan(
text.indexOf(`run run-20261008-020000-ab12 · RESEARCH · ${fx.root}`),
)
})
test("non-RESEARCH stages print no research line (#248)", async () => {
const current = state({ stage: "BUILD", activePhase: "p1", updatedAt: at(20).toISOString() })
const live = await readLiveness(fx.root, current, { now: at(60) })
expect(researchLine(live)).toBeUndefined()
const text = formatHuman(
current,
live,
"<factory-state>\nrun run-20261008-020000-ab12 · stage BUILD\nbuild phase p1\n</factory-state>",
)
expect(text).toContain("Building p1 in progress")
expect(text).not.toContain("research:")
})
})
packages/core/test/lsp-shutdown.test.ts
+68
View File
@@ -0,0 +1,68 @@
// Audit #197 integration tests (core side): the workspace TypeScript server
// reaches diagnostics through the manager, and stopAll shuts the running
// servers down. Same tsc fixture seam as lsp.test.ts; no binaries run directly.
import { afterAll, beforeAll, describe, expect, test } from "bun:test"
import { mkdir, mkdtemp, rm, symlink, writeFile } from "node:fs/promises"
import { tmpdir } from "node:os"
import path from "node:path"
import { LspManager } from "../src/index.ts"
const TYPESCRIPT = path.resolve(import.meta.dir, "../../../node_modules/typescript")
let root: string
let state: string
let manager: LspManager
const write = async (file: string, text: string) => {
await mkdir(path.dirname(path.join(root, file)), { recursive: true })
await writeFile(path.join(root, file), text)
}
beforeAll(async () => {
root = await mkdtemp(path.join(tmpdir(), "es-lsp-shutdown-"))
state = await mkdtemp(path.join(tmpdir(), "es-lsp-shutdown-state-"))
await write(
"tsconfig.json",
JSON.stringify({
compilerOptions: { strict: true, target: "ES2022", module: "NodeNext", noEmit: true },
include: ["src"],
}),
)
await mkdir(path.join(root, "node_modules"), { recursive: true })
await symlink(TYPESCRIPT, path.join(root, "node_modules/typescript"), "dir")
await mkdir(path.join(root, "node_modules/.bin"), { recursive: true })
await symlink(path.join(TYPESCRIPT, "bin/tsc"), path.join(root, "node_modules/.bin/tsc"))
await write("src/math.ts", "export function add(a: number, b: number): number {\n return a + b\n}\n")
await write(
"src/main.ts",
'import { add } from "./math.js"\n\nconst total: number = add(1, 2)\nconst wrong: string = total\nexport { wrong }\n',
)
manager = new LspManager({ root, stateDir: state, configFiles: [path.join(root, "opencode.json")] })
}, 30_000)
afterAll(async () => {
await manager?.stopAll()
await rm(root, { recursive: true, force: true })
await rm(state, { recursive: true, force: true })
})
describe("LspManager shutdown (#197)", () => {
test("diagnostics start the workspace server and stopAll shuts it down", async () => {
const diagnostics = await manager.diagnostics("src/main.ts", { maxWaitMs: 20_000 })
expect(Array.isArray(diagnostics)).toBe(true)
const messages = (diagnostics as any[]).map((item) => `${item.range.start.line + 1}: ${item.message}`)
expect(messages.join("\n")).toContain("4: Type 'number' is not assignable to type 'string'")
const running = (await manager.status()).running
expect(running.some((entry) => entry.id === "typescript" && entry.ready)).toBe(true)
await manager.stopAll()
expect((await manager.status()).running).toEqual([])
}, 60_000)
test("stopAll is idempotent and the next diagnostics restarts the server lazily", async () => {
await manager.stopAll()
expect((await manager.status()).running).toEqual([])
const diagnostics = await manager.diagnostics("src/main.ts", { maxWaitMs: 20_000 })
expect(Array.isArray(diagnostics)).toBe(true)
expect(JSON.stringify(diagnostics)).toContain("not assignable to type 'string'")
expect((await manager.status()).running.some((entry) => entry.id === "typescript" && entry.ready)).toBe(true)
}, 60_000)
})
packages/core/test/lsp.test.ts
+89 -1
View File
@@ -1,7 +1,7 @@
// LSP runtime against real language servers (native TypeScript 7 `tsc --lsp`,
// gopls) plus config resolution and the pinned installer.
import { afterAll, beforeAll, describe, expect, test } from "bun:test"
import { chmod, mkdir, mkdtemp, rm, symlink, writeFile } from "node:fs/promises"
import { chmod, mkdir, mkdtemp, readFile, rm, symlink, writeFile } from "node:fs/promises"
import { tmpdir } from "node:os"
import path from "node:path"
import {
@@ -138,4 +138,92 @@ describe("pinned installer", () => {
expect(await consent("fake-ls", state)).toBe(true)
await expect(installServer(server, { stateDir: state, npm: fakeNpm })).rejects.toThrow("integrity mismatch")
})
test("the fetch/install mitigation stack is pinned: SRI, scripts-disabled, scrubbed env, confined cwd (#275.2)", async () => {
// Installs are not bwrap-sandboxed (the fetch needs the network), so the
// narrower stack — consent, exact pins, --ignore-scripts, scrubbed env,
// scratch/target confinement — is pinned here instead of assumed.
const genuine = new TextEncoder().encode("genuine-tarball-bytes")
const fakeNpm = path.join(state, "npm-pin")
await writeFile(
fakeNpm,
[
"#!/bin/sh",
'here=$(dirname "$0")',
'if [ "$1" = pack ]; then',
' printf "%s\\n" "$@" > "$here/pack-args.log"',
' printf "PWD=%s\\n" "$PWD" >> "$here/pack-args.log"',
' env | sort > "$here/pack.env"',
" dest=$4",
' printf "genuine-tarball-bytes" > "$dest/pkg-1.0.0.tgz"',
' echo \'[{"filename":"pkg-1.0.0.tgz"}]\'',
" exit 0",
"fi",
'if [ "$1" = install ]; then',
' printf "%s\\n" "$@" > "$here/install-args.log"',
' printf "PWD=%s\\n" "$PWD" >> "$here/install-args.log"',
' env | sort > "$here/install.env"',
' prefix=""',
' prev=""',
' for arg in "$@"; do',
' if [ "$prev" = "--prefix" ]; then prefix=$arg; fi',
" prev=$arg",
" done",
' mkdir -p "$prefix/node_modules/.bin"',
' printf "#!/bin/sh\\nexit 0\\n" > "$prefix/node_modules/.bin/fake-pin"',
' chmod +x "$prefix/node_modules/.bin/fake-pin"',
" exit 0",
"fi",
"exit 1",
"",
].join("\n"),
)
await chmod(fakeNpm, 0o755)
const server: ServerSpec = {
id: "fake-pin",
command: ["fake-pin"],
extensions: [".fake"],
rootMarkers: [],
install: {
kind: "npm",
packages: [{ name: "pkg", version: "1.0.0", integrity: sri(genuine) }],
bin: "fake-pin",
},
source: "builtin",
}
await recordConsent("fake-pin", true, state)
// A canary the scrubbed env must not leak; a registry override must pass through.
process.env.ES_LSP_PIN_EVIL = "leak-me"
process.env.NPM_CONFIG_REGISTRY = "https://registry.test/"
let bin: string
try {
bin = await installServer(server, { stateDir: state, npm: fakeNpm })
} finally {
delete process.env.ES_LSP_PIN_EVIL
delete process.env.NPM_CONFIG_REGISTRY
}
expect(bin).toContain(path.join("lsp", "fake-pin-1.0.0"))
const read = (name: string) => readFile(path.join(state, name), "utf8")
const packArgs = await read("pack-args.log")
expect(packArgs).toContain("[email protected]")
// pack runs confined to a scratch dir, not the project root.
const packPwd = /PWD=(.*)/.exec(packArgs)![1]!
expect(path.basename(packPwd)).toMatch(/^es-lsp-/)
expect(packPwd).not.toBe(root)
const packEnv = await read("pack.env")
expect(packEnv).not.toContain("ES_LSP_PIN_EVIL")
expect(packEnv).toContain("NPM_CONFIG_REGISTRY=https://registry.test/")
const installArgs = await read("install-args.log")
for (const flag of ["--no-save", "--ignore-scripts", "--no-audit", "--no-fund"]) expect(installArgs).toContain(flag)
expect(installArgs).toContain("--prefix")
expect(installArgs).toContain("pkg-1.0.0.tgz")
// install runs with cwd inside the state-dir target it writes.
const installPwd = /PWD=(.*)/.exec(installArgs)![1]!
expect(installPwd).toContain(path.join(state, "lsp", "fake-pin-1.0.0"))
const installEnv = await read("install.env")
expect(installEnv).not.toContain("ES_LSP_PIN_EVIL")
expect(installEnv).toContain("NPM_CONFIG_REGISTRY=https://registry.test/")
})
})
packages/core/test/pack-contents.test.ts
+133
View File
@@ -0,0 +1,133 @@
// Packed-tarball contents for the release set (issue #208): what users
// install must carry the runtime (core assets, the CLI bin, the plugin
// server plus the prebuilt TUI) and nothing else — no tests, no private
// packages (`es-fleet`, web, testkit). Packs with `bun pm pack` the way
// scripts/pack-smoke.sh packs, from the single package list
// (scripts/packages.ts, as in packages.test.ts).
import { afterAll, beforeAll, describe, expect, test } from "bun:test"
import { mkdtemp, readdir, rm } from "node:fs/promises"
import { tmpdir } from "node:os"
import path from "node:path"
import { releasePackages, tarballSlug, workspacePackages } from "../../../scripts/packages.ts"
const root = path.resolve(import.meta.dir, "../../..")
async function run(argv: string[], cwd: string): Promise<string> {
const proc = Bun.spawn(argv, { cwd, stdout: "pipe", stderr: "pipe" })
const [stdout, stderr, code] = await Promise.all([
new Response(proc.stdout).text(),
new Response(proc.stderr).text(),
proc.exited,
])
if (code !== 0) throw new Error(`${argv.join(" ")} in ${cwd} exited ${code}: ${stderr.slice(-2000)}`)
return stdout
}
async function tarList(tgz: string): Promise<string[]> {
const out = await run(["tar", "-tzf", tgz], root)
return out.split("\n").filter(Boolean)
}
async function tarRead(tgz: string, member: string): Promise<string> {
return run(["tar", "-xzOf", tgz, member], root)
}
const TEST_DIR = new Set(["test", "tests", "__tests__", "__snapshots__"])
const isTestPath = (entry: string) => {
const parts = entry.split("/")
return parts.some((part) => TEST_DIR.has(part)) || /\.(test|spec)\.[a-z]+$/.test(parts.at(-1) ?? "")
}
describe("packed release tarballs (issue #208)", () => {
let out = ""
const tarballs = new Map<string, string>()
beforeAll(async () => {
out = await mkdtemp(path.join(tmpdir(), "es-pack-contents-"))
const release = await releasePackages(root)
for (const { dir } of release) await run(["bun", "run", "build"], path.join(root, "packages", dir))
for (const { dir } of release)
await run(["bun", "pm", "pack", "--destination", out], path.join(root, "packages", dir))
const files = await readdir(out)
for (const { dir, name } of release) {
const hits = files.filter((file) => file.startsWith(`${tarballSlug(name)}-`) && file.endsWith(".tgz"))
expect(hits).toHaveLength(1)
tarballs.set(dir, path.join(out, hits[0]!))
}
}, 240_000)
afterAll(() => rm(out, { recursive: true, force: true }))
test("the release set is exactly core, cli and the plugin; fleet, web and testkit stay private", async () => {
expect([...tarballs.keys()].sort()).toEqual(["cli", "core", "opencode"])
const all = await workspacePackages(root)
for (const dir of ["fleet", "web", "testkit"]) {
const pkg = all.find((entry) => entry.dir === dir)!
expect([dir, pkg.private]).toEqual([dir, true])
expect(pkg.order).toBeUndefined()
}
// No tarball for a private package, even a stale one from an earlier run.
for (const pkg of all.filter((entry) => entry.private)) {
const hits = (await readdir(out)).filter(
(file) => file.startsWith(`${tarballSlug(pkg.name)}-`) && file.endsWith(".tgz"),
)
expect([pkg.dir, hits]).toEqual([pkg.dir, []])
}
})
test("packed manifests carry no workspace: protocol", async () => {
for (const [dir, tgz] of tarballs) {
const manifest = await tarRead(tgz, "package/package.json")
expect([dir, manifest.includes("workspace:")]).toEqual([dir, false])
}
})
test("tarballs contain no tests and no private-package payload", async () => {
for (const [dir, tgz] of tarballs) {
const entries = await tarList(tgz)
expect([dir, entries.filter(isTestPath)]).toEqual([dir, []])
expect([dir, entries.filter((entry) => entry.includes("es-fleet") || entry.includes("es-web"))]).toEqual([
dir,
[],
])
}
})
test("tarballs ship the runtime payload users need", async () => {
const has = async (dir: string, member: string) =>
expect([dir, member, (await tarList(tarballs.get(dir)!)).includes(member)]).toEqual([dir, member, true])
// Core: compiled entry plus the bundled assets pack-smoke loads
// (prompts, skills, licence templates).
await has("core", "package/dist/index.js")
await has("core", "package/assets/prompts/factory.md")
await has("core", "package/assets/skills/factory/SKILL.md")
await has("core", "package/assets/licenses/MIT.template.txt")
// CLI: the bin users run plus its bundle.
await has("cli", "package/bin/es.js")
await has("cli", "package/dist/es.js")
// Plugin: the server entry the host loads plus the prebuilt TUI (raw
// .tsx cannot ship: the host Solid transform skips node_modules).
await has("opencode", "package/server.ts")
await has("opencode", "package/src/server.ts")
await has("opencode", "package/dist/tui.js")
})
test("the opencode host pin matches what the suites test", async () => {
const manifest = async (dir: string) =>
JSON.parse(await Bun.file(path.join(root, "packages", dir, "package.json")).text()) as {
dependencies?: Record<string, string>
devDependencies?: Record<string, string>
}
const plugin = await manifest("opencode")
const kit = await manifest("testkit")
// Pinned, not a range: the host the plugin runs against is exactly the
// host the testkit boots.
expect(plugin.dependencies?.["@opencode/plugin"]).toBe("2.0.24")
expect(plugin.devDependencies?.["@opencode/sdk"]).toBe("2.0.24")
expect(kit.dependencies?.["@opencode/sdk"]).toBe(plugin.devDependencies?.["@opencode/sdk"])
// The packed plugin pins the same host dependency (workspace:* rewritten).
const packed = JSON.parse(await tarRead(tarballs.get("opencode")!, "package/package.json")) as {
dependencies?: Record<string, string>
}
expect(packed.dependencies?.["@opencode/plugin"]).toBe(plugin.dependencies?.["@opencode/plugin"])
})
})
packages/core/test/pack-pins.test.ts
+92
View File
@@ -0,0 +1,92 @@
// Packaging pin assertions (#212): the opencode triple-pin moves together,
// every release manifest declares an explicit `files` allowlist with no test
// paths, and the `--release` rendering keeps the `<dir> <slug>` contract
// `publish.yml` parses. File reads only: no network, no registry, no pack.
// Structural fixes (a peer range, narrower allowlists, publish parsing,
// the runtime-matrix wording) route to S9 via CROSS-SQUAD REQUEST.
import { describe, expect, test } from "bun:test"
import { readFile } from "node:fs/promises"
import path from "node:path"
import { releasePackages, tarballSlug, workspacePackages } from "../../../scripts/packages.ts"
const root = path.resolve(import.meta.dir, "../../..")
const manifest = async (dir: string) =>
JSON.parse(await readFile(path.join(root, "packages", dir, "package.json"), "utf8")) as Record<string, any>
describe("the opencode triple-pin moves together", () => {
test("@opencode/plugin, @opencode/sdk and the e2e CLI share one version", async () => {
const opencode = await manifest("opencode")
const plugin = opencode.dependencies?.["@opencode/plugin"] as string
const sdk = opencode.devDependencies?.["@opencode/sdk"] as string
expect(plugin).toMatch(/^\d+\.\d+\.\d+$/)
expect(sdk).toBe(plugin)
const e2e = await readFile(path.join(root, ".github/workflows/e2e-feedback.yml"), "utf8")
const cliPins = [...e2e.matchAll(/@opencode\/cli-linux-x64@([0-9A-Za-z._-]+)/g)].map((match) => match[1])
expect(cliPins.length).toBeGreaterThan(0)
expect(new Set(cliPins).size).toBe(1)
expect(cliPins[0]).toBe(plugin)
})
test("no opencode peer range: no such npm package exists to peer on (#212a)", async () => {
// Verified 2026-10-11: `npm view opencode` is a registry 404 — the host
// ships as @opencode/*-scoped packages and platform binaries, so a peer
// on a bare `opencode` id could never resolve. The triple-pin above is
// the mechanism; this pins the decision against a future peer addition.
const opencode = await manifest("opencode")
for (const field of ["dependencies", "devDependencies", "peerDependencies"] as const)
expect(Object.keys(opencode[field] ?? {})).not.toContain("opencode")
})
})
describe("release tarball allowlists", () => {
test("every release manifest declares an explicit files allowlist with no test paths", async () => {
for (const pkg of await releasePackages(root)) {
const files = (await manifest(pkg.dir)).files as unknown
expect(Array.isArray(files) && (files as unknown[]).length > 0).toBe(true)
for (const entry of files as string[]) {
expect(typeof entry).toBe("string")
expect(entry).not.toMatch(/test|spec|__tests__/i)
}
}
})
test("fleet and web stay private (never released)", async () => {
const all = await workspacePackages(root)
for (const dir of ["fleet", "web"]) {
expect(all.find((pkg) => pkg.dir === dir)?.private).toBe(true)
}
})
test("no test files sit under a released package's src dir (#212c)", async () => {
// The allowlists include whole `src` trees, so a stray src/*.test.ts
// would ship silently; the manifest-string check above cannot see it.
// (fleet and web are private and never packed; pack-smoke.sh 2026-10-11
// additionally proved the tarballs ship no test files.)
const { readdir } = await import("node:fs/promises")
const strays: string[] = []
for (const pkg of await releasePackages(root)) {
const walk = async (rel: string): Promise<void> => {
const entries = await readdir(path.join(root, "packages", pkg.dir, rel), { withFileTypes: true }).catch(
() => [],
)
for (const entry of entries) {
const child = `${rel}/${entry.name}`
if (entry.isDirectory()) await walk(child)
else if (/\.test\.[cm]?[jt]s$|\.spec\.[cm]?[jt]s$|__tests__/.test(child)) strays.push(`${pkg.dir}/${child}`)
}
}
await walk("src")
}
expect(strays).toEqual([])
})
})
describe("the --release rendering contract", () => {
test('every line is "<dir> <slug>" with the slug derived from the manifest name', async () => {
for (const pkg of await releasePackages(root)) {
const line = `${pkg.dir} ${tarballSlug(pkg.name)}`
expect(line).toMatch(/^\S+ \S+$/)
expect(tarballSlug(pkg.name)).not.toContain(" ")
}
})
})
packages/core/test/packages.test.ts
+10 -3
View File
@@ -64,8 +64,15 @@ describe("workspace packages", () => {
test("the real repo releases core, cli and the plugin in dependency order", async () => {
const root = path.resolve(import.meta.dir, "../../..")
expect((await releasePackages(root)).map((pkg) => pkg.dir)).toEqual(["core", "cli", "opencode"])
expect((await workspacePackages(root)).map((pkg) => pkg.dir).sort()).toEqual(
expect.arrayContaining(["cli", "core", "opencode", "testkit"]),
)
const all = await workspacePackages(root)
expect(all.map((pkg) => pkg.dir).sort()).toEqual(expect.arrayContaining(["cli", "core", "opencode", "testkit"]))
// Manifest name re-scope or version skew must not ship green (#212):
// pin the release manifests' names and lockstep version beside the dir.
const byDir = new Map(all.map((pkg) => [pkg.dir, pkg]))
expect(byDir.get("core")).toMatchObject({ name: "@heretek-ai/es-core" })
expect(byDir.get("cli")).toMatchObject({ name: "@heretek-ai/es-cli" })
expect(byDir.get("opencode")).toMatchObject({ name: "@heretek-ai/epistemic-swarm" })
const versions = new Set((await releasePackages(root)).map((pkg) => pkg.version))
expect(versions.size).toBe(1)
})
})

Some files were not shown because too many files have changed in this diff Show More