From 5d79387505ebae1ec8d800ee3ffd901920a9d8be Mon Sep 17 00:00:00 2001 From: morluto <76467478+morluto@users.noreply.github.com> Date: Sat, 10 Oct 2026 05:28:13 +0800 Subject: [PATCH] docs: prune repeated contributor and tool design instructions --- AGENTS.md | 74 ++++++---------- CONTRIBUTING.md | 176 ++++++++++++++------------------------ docs/testing.md | 201 ++++++++++---------------------------------- docs/tool-design.md | 175 +++++++++++++++++--------------------- 4 files changed, 209 insertions(+), 417 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 334d1333c..15d3984ff 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -2,74 +2,48 @@ ## Product and Authority -REA is a local-only reverse-engineering tool exposed through a CLI and MCP server. Keep observed, derived, inferred, and unknown results distinct. Shared application workflows must provide equivalent CLI/MCP behavior, including Evidence provenance and session retention. Provider support and setup live in [README.md](README.md#choosing-a-deep-analysis-provider) and the linked provider guides. +REA is a local-only reverse-engineering tool with shared CLI/MCP workflows. Distinguish observations, derivations, inferences, and unknowns; preserve Evidence provenance and session semantics through both adapters. Provider support and setup live in [README.md](README.md#choosing-a-deep-analysis-provider) and its linked guides. -Preserve provider authority boundaries: Ghidra uses an ephemeral analysis database without modifying executable bytes or controlling a GUI; Windows Ghidra P0 has no mutation authority. IDA analysis is read-only, and an attached GUI database is never saved or closed. +Ghidra uses ephemeral databases without modifying executable bytes or controlling a GUI; Windows Ghidra P0 has no mutation authority. IDA analysis is read-only; never save or close an attached GUI database. -Configuration changes must be additive and idempotent, with backups. Installers must not install or upgrade Homebrew, Node.js, npm, Java, Ghidra, or other unrelated software. Ghidra and IDA are bring-your-own. `rea setup` must print its planned changes and require approval before writing files or installing Hopper. +Configuration changes must be additive and idempotent, with backups. Installers must not install or upgrade unrelated software, including Homebrew, Node.js, npm, Java, or Ghidra. Ghidra and IDA are bring-your-own. `rea setup` must show its planned changes and require approval before writing files or installing Hopper. -Preserve caller-selected inputs, captured output, URLs, paths, digests, mismatch locations, and analysis metadata. Redact transport authentication credentials and values the caller explicitly marks sensitive; do not infer secrecy from names or text patterns, or persist the entire ambient environment merely because a child inherits it. +Preserve caller-selected inputs and local evidence, including output, URLs, paths, digests, and metadata. Redact transport authentication credentials and explicitly marked sensitive values; do not infer secrecy from names or text patterns. Do not persist the ambient environment merely because a child inherits it. -## Architecture and Ownership +## Architecture and Cleanup -REA is a layered ESM TypeScript application. Dependencies flow inward; see [docs/architecture.mermaid](docs/architecture.mermaid). +Dependencies flow inward: domain semantics and contracts, provider adapters, shared application workflows, then CLI/MCP adapters. Keep engine protocols and provider-specific code out of domain/application layers. See [docs/architecture.mermaid](docs/architecture.mermaid) when changing composition. -- `src/domain/` owns pure provider-neutral semantics; `src/contracts/` owns caller-visible schemas and the canonical tool inventory. -- Provider directories own engine protocols and parsing. Keep provider-specific code out of domain and application layers; `bridge/` contains provider-side adapters. -- `src/application/` owns shared workflows and Evidence/session semantics. `src/server/` translates MCP requests; `src/cli.ts` and `src/main.ts` are the CLI and MCP entry points. -- `src/process/` owns shared lifecycle primitives. Reuse its supervision and identity checks; a PID and executable pathname alone do not establish ownership after exit or reuse. -- `src/config/` owns configuration parsing. Keep user-facing setup and environment documentation in [README.md](README.md) and the relevant provider guide. +Give each interpretation and resource a clear owner. Preserve identity, omission, uncertainty, and source evidence across boundaries. Retain ownership of resources whose cleanup failed. Reuse `src/process/` supervision and identity primitives; a PID and executable path alone do not establish ownership after exit or reuse. -Assess brittleness during implementation, debugging, review, and maintenance. Give each interpretation and resource a clear owner. Preserve identity, omission, uncertainty, and source evidence across boundaries. Retain ownership of resources whose cleanup failed. Fix the producing representation or ownership rule before adding downstream repairs. +Assess brittleness in touched code during implementation, debugging, review, and maintenance. Fix the producing representation or ownership rule before adding downstream repairs. Normalize once at the owning boundary using format-aware parsers. Remove superseded representations, aliases, wrappers, and redundant tests after verifying affected consumers. Introduce abstractions only when they remove observed complexity. -Simplify duplicated rules, scattered defaults, caller-specific exceptions, and unclear invariants in touched code. Prefer standard format-aware parsers and existing shared helpers; normalize once at the owning boundary. Remove superseded representations, aliases, wrappers, and tests when their replacement is verified through affected consumers. Keep refactors focused on the task and introduce abstractions only when they remove observed complexity. +Parse unknown boundary values and model expected failures with the tagged error algebra and `Result`. Preserve partial facts and meaningful failure reasons. Keep identity and lookup values separate from display formatting; preserve source values and explicit unknowns when normalization loses information. Absolute paths, file URLs, and HTTP paths have different semantics. -## Development and Generated Files +Validate advertised JSON Schemas after SDK conversion against their declared dialect and actual clients. Bind handlers to named contracts with exact types; catalog ordering must never select an operation or schema. -Use the Node.js version in [.nvmrc](.nvmrc) and npm version in `package.json#packageManager`. Confirm the active toolchain before installing dependencies. In Cursor Cloud, the image may put an older Node.js first on `PATH`; prepend `/usr/local/bin`, where environment setup installs the pinned toolchain. +## Development + +Use [.nvmrc](.nvmrc) and `package.json#packageManager` for the pinned toolchain. Cursor Cloud may place older Node.js first on `PATH`; environment setup installs the pinned toolchain under `/usr/local/bin`. - `npm ci`: install locked dependencies. -- `npm run build:cached`: compile the runtime and bundle its authored skill; test catalogs and documentation artifacts have separate tasks. -- `npm run test:local -- PATH...`: run source tests without building; explicit paths run regardless of Git status. -- `npm run test:focused -- PATH...`: run exact test files with their required runtime and generated artifacts. -- `npm run check:changed`: run static checks and source tests affected since the merge base, defaulting to `origin/main`. -- `npm run check:fast`: run cached typecheck and lint checks; also runs before push. -- `npm run check:pr`: complete local deterministic gate and generated-document checks for broad changes. Routine iterations need focused tests and relevant checks; CI owns full coverage. -- `npm run docs:generate` / `npm run docs:check`: generate / validate documentation artifacts. +- `npm run build:cached`: build the runtime and packaged skill. +- `npm run test:local -- PATH...`: source tests without building. +- `npm run test:focused -- PATH...`: exact tests with their required artifacts. +- `npm run check:fast`: cached typecheck/lint and the pre-push check. +- `npm run docs:check`: validate generated documentation. -`docs/public/product-catalog.json`, `docs/verification/managed-conformance-*.json`, and `skills/` are ignored build outputs. Edit source contracts and authored instructions in `skill-src/`; never commit derived catalog digests or portable conformance projections. `.cache/mcp-tool-catalog.json` is test metadata generated by `npm run mcp-catalog:generate`; source checking and runtime builds do not consume it. CI retains generated outputs as artifacts rather than pushing them to feature branches. +Use relevant checks for the change. The complete local gate, `npm run check:pr`, is optional for broad changes or CI diagnosis. See [CONTRIBUTING.md](CONTRIBUTING.md) for development, generated-file ownership, and release conventions. -See [docs/testing.md](docs/testing.md) for verification lanes and [CONTRIBUTING.md](CONTRIBUTING.md) for contribution checks. Keep each lane's prerequisites limited to its claim; separate host-native acceptance from optional cross-target toolchains and preflight required commands. JavaScript analysis and deterministic tests do not require Hopper, Ghidra, or IDA. +`docs/public/product-catalog.json`, `docs/verification/managed-conformance-*.json`, and `skills/` are ignored outputs. Edit source contracts and `skill-src/`, then regenerate as needed. `.cache/mcp-tool-catalog.json` is test metadata; runtime builds and source checking do not consume it. Never commit binaries, provider project documents, credentials, `dist/`, `node_modules/`, or local planning artifacts. -## Coding and Boundary Contracts +## Tool and Test Changes -Use ESM TypeScript, two-space indentation, and committed Oxfmt formatting. Preserve compiler strictness, including `exactOptionalPropertyTypes`, `noUncheckedIndexedAccess`, and `verbatimModuleSyntax`. Use `camelCase` for values/functions, `PascalCase` for classes/types, and `UPPER_SNAKE_CASE` for constants. Exported APIs require concise JSDoc. Avoid `any`, unchecked casts, non-null assertions, import-time I/O, and floating promises. Model expected failures with the tagged error algebra and `Result`. +For tool design, follow [docs/tool-design.md](docs/tool-design.md): start from the analyst question, reuse existing contracts, preserve caller choices, and return useful evidence inline. Limits must follow real format, protocol, authority, or resource constraints. Account for representation expansion before allocation; derive completeness from examined coverage. -Parse unknown values at MCP, environment, and subprocess boundaries. Trace the producer's actual representation through parsing, normalization, authorization, serialization, and the consumer result. Establish affected callers from code paths; inspect adjacent representations and failure paths when changing a boundary. +For test changes or provider claims, use [docs/testing.md](docs/testing.md). Prefer real public CLI/MCP workflows, then production-boundary integration, then producer goldens. Keep module tests for distinct semantics stronger workflows cannot reliably reproduce. Preserve malformed-input, cancellation, cleanup, permission, capacity, and target-identity coverage when pruning. Report unverified coverage explicitly; simulated providers do not establish real-provider behavior. -Keep identity, provenance, matching, and path-resolution values separate from display formatting. Absolute filesystem paths, file URLs, and HTTP paths have different semantics. Interpret provider metadata according to its producer; preserve source values and explicit unknowns when normalization loses information. Display placeholders must not feed lookup or selection. - -Preserve partial facts without inventing missing metadata or requiring another provider's full representation. Inspect installed APIs and representative producer objects before claiming a capability is unavailable. Preserve meaningful failure reasons and actionable recovery guidance: malformed input, unsupported targets, unavailable providers, and host permission denial are distinct. Collected observations must survive execution and cleanup failures. - -Validate advertised input and output JSON Schemas against their declared dialect after SDK conversion, and verify client compatibility at the consuming boundary. Bind handlers to named contracts with exact input/output types; catalog ordering must never select an operation or schema. - -## Tool Design and Verification - -Start from the analyst question and inspect existing contracts before adding a tool. Prefer reusable primitives and ordinary commands/scripts for experiments; add composed workflows for observed recurring needs. Leave meaningful target, action, capture, and output choices to the caller. Remove ignored settings and repeated approval declarations; derive built-in lifecycle behavior and default optional metadata. - -Return useful evidence inline, complete by default, with identity, source locations, relevant limitations, and truthful authority/lifecycle effects. Limits must follow real format, protocol, authority, or resource constraints; account for representation expansion and products of bounded dimensions before allocation. Derive completeness from what was examined and exhausted. Follow [docs/tool-design.md](docs/tool-design.md) for naming, composition, contract design, and representative CLI/MCP usability evaluation. Update canonical contracts and generated artifacts together. - -Prefer full public CLI/MCP end-to-end workflows with real providers, then production-boundary integration tests, then goldens from real producers. Keep focused module tests for distinct failures or semantics that stronger workflows cannot reliably reproduce. Delete redundant assertions when stronger coverage proves the same claim; preserve distinct malformed-input, cancellation, cleanup, permission, capacity, and target-identity cases. Follow [docs/testing.md](docs/testing.md) for pruning criteria, test classification, fixtures, and verification commands. - -Use Vitest, `*.test.ts`, production seams rather than module mocks, and the pinned MCP client SDK. Treat advertised examples as executable contracts using representative producer data. Provider/platform and packaged-artifact claims require their real workflows; capability probes and simulated transports prove only their narrower scope. Report unverified coverage explicitly. - -Keep tool catalogs complete and self-describing; prefer capability/session-scoped availability over schema truncation. Serialized bytes alone do not establish agent usability or model context cost. - -## Commits and Pull Requests - -Use Conventional Commit subjects, including PR titles for squash merges. Release Please always bumps minor; mark breaking changes with `!` or a `BREAKING CHANGE:` footer. Describe behavior/contract changes, relevant validation, linked issues, and real-provider verification scope; include sanitized CLI/MCP examples for schema changes. - -Never commit binaries, provider project documents, credentials, `dist/`, `node_modules/`, or local planning artifacts such as `.codex/`. Pre-commit formats and lints staged source. +Use Conventional Commit subjects and PR titles. Mark breaking changes with `!` or a `BREAKING CHANGE:` footer. Describe behavior changes, relevant validation, and real-provider verification scope. diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index cce65e730..c60c1e9e2 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -1,29 +1,14 @@ # Contributing to REA -REA welcomes focused bug fixes, documentation improvements, tests, and reverse-engineering workflow enhancements. Open an issue before a large contract or architecture change so its scope can be agreed before implementation. +REA welcomes focused bug fixes, documentation improvements, tests, and reverse-engineering workflows. For large contract or architecture changes, explain the intended scope and link the relevant issue or design discussion. ## Choosing a contribution -Small contributions are welcome when they solve a concrete problem, including -broken commands, misleading documentation, and missing regression coverage. -Explain the problem, keep the diff focused, and share how you checked the change. - -Please avoid standalone cosmetic changes without a clear benefit, such as fixing -a typo in an internal comment, changing capitalization or punctuation in -already-clear prose, or reformatting unchanged code. Include related polish in a -useful fix when practical. For cosmetic-only suggestions, please open an issue -instead of a standalone PR so they can be collected and addressed together. -Corrections to confusing explanations or incorrect instructions are welcome. - -Avoid speculative cleanup without a concrete problem to solve. Keep related -corrections in one PR. - -Before requesting review, read the final diff and run the relevant checks. Be clear -about anything you couldn't test. You should be able to explain the changes you -submit. - -We may decline changes when the benefit doesn't justify the review or maintenance -work. +Explain the concrete problem, keep related corrections together, and report +verification and unavailable coverage. Contributions should justify their review +and maintenance cost. Keep cosmetic polish with useful fixes; confusing or +incorrect user instructions warrant standalone corrections. Collect other +cosmetic-only suggestions in an issue rather than separate PRs. For capability organization and provider composition, follow the incremental [migration guide](docs/capability-migration.md). Run `npm run verify:test-discovery` @@ -33,7 +18,10 @@ When adding or changing an MCP tool, follow the [tool design guide](docs/tool-de ## Development setup -REA development requires Node.js 24.18.x and npm 11.16.x (pinned toolchain via `nvm use`; the supported runtime range is Node.js ^22.19 || ^24.11 || >=26, as the README badge states). Real-Hopper verification additionally requires either macOS 12+ or an officially supported Linux host (Ubuntu 24.04+, Fedora 41+, 64-bit Arch, or CachyOS) and an installed Hopper application. Linux demo verification uses its own private Xvfb display and does not require a desktop session. Run `nvm use` before installing dependencies. +Use the development toolchain pinned in `.nvmrc` and +`package.json#packageManager`. Run `nvm use` before installing dependencies; +installed-package runtime support is documented in +[installation](docs/installation.md#start-setup). ```bash npm ci @@ -48,7 +36,8 @@ catalogs, documentation projections, or managed verification evidence. Turbo caches deterministic builds and static checks across Git worktrees. After a package, lockfile, or managed-skill version change, run `npm run metadata:generate` before building. -Keep dependencies flowing inward through the existing domain, contracts, provider, application, server, and adapter layers. Parse unknown values at process and protocol boundaries, model expected failures with `Result`, and preserve the canonical tool inventory defined by `TOOL_CONTRACTS` unless a deliberate contract change updates every verifier, generated catalog artifact, and snapshot. Keep tool discovery complete and report capability- and session-scoped availability through `binary_session`. +Repository invariants live in [AGENTS.md](AGENTS.md). Keep provider-specific +protocols in adapters and shared CLI/MCP workflows in the application layer. ## Documentation website @@ -74,112 +63,77 @@ The separate GitHub Pages host uses only the manual ## Development feedback and PR verification -For an ordinary edit, run the relevant regression and cached static checks: +Select checks for the behavior changed. For a source edit: ```bash npm run test:focused -- src/config.test.ts npm run check:fast ``` -`test:local` selects dirty source tests without building; explicit source test -paths run even on a clean tree. `check:changed` adds source tests affected since -the branch merge base with `origin/main`. Use `npm run test:changed -- --base -REVISION` to choose another base. Changed-test selection follows the import -graph and is feedback, not complete correctness evidence. Inspect relevant -boundary and provider behavior explicitly; see [docs/testing.md](docs/testing.md). +`test:local` runs source tests without building. `check:changed` combines cached +static checks with source tests affected since the merge base with `origin/main`. +Import-graph selection can miss runtime registration, bridges, and generated data; +select relevant boundary tests explicitly. See [testing](docs/testing.md#developer-commands) +for commands, test placement, pruning, and real-provider prerequisites. -Before handing off a PR, run the relevant tests, `npm run check`, and -`npm run docs:check` when contracts or generated metadata change. Record which -checks ran. CI owns complete deterministic tests and aggregate coverage. Use -`npm run check:pr` for a deliberate full local gate on broad changes or when -investigating CI failures; it is not required after each edit, rebase, or push. -Packaging, setup, installation, or distribution changes also require -`npm run verify:package` and `npm pack --dry-run`. Provider behavior changes -require the matching real-provider `verify:*` lane. +Before requesting review, run relevant tests and `npm run check`; add +`npm run docs:check` when contracts or generated metadata change. Record the +checks and unavailable coverage. CI owns the full deterministic suite and +coverage. `npm run check:pr` is an optional full local gate for broad changes or +CI diagnosis. Packaging/setup/distribution changes additionally require +`npm run verify:package` and `npm pack --dry-run`; provider changes require the +matching real-provider lane. State which host/provider workflows ran. -`check:fast` runs cached source/test typecheck and lint without compiling or -running generators. `npm run test:prepare` prepares all artifacts consumed by -the complete suite; focused tests prepare only their runtime and declared -artifact dependencies. The test-only MCP catalog lives at -`.cache/mcp-tool-catalog.json` and can be regenerated independently with -`npm run mcp-catalog:generate`. +Pre-commit formats and lints staged files; pre-push runs `check:fast`. -Build and generation commands hold one SQLite exclusive transaction around -their whole Turbo graph, including cache restoration. The persistent database -in `.cache/rea-command-locks/` is a lock identity, not a stale-file marker; -the operating system releases ownership when the command supervisor exits. -On POSIX, that supervisor retains the transaction and process-group identity -through cancellation, including loss of its parent runner. Windows cancellation -waits for the command to complete. Do not remove the database while commands -are running. +### Generated files and command ownership -`check` adds formatting, dead-code, and package-metadata freshness checks. -Formatting uses Oxfmt and the committed `.oxfmtrc.json`; generated sources use -the same configuration. Pre-commit formats and lints staged files; pre-push runs -`check:fast`. -`docs:check` builds and validates generated metadata for the current checkout. -The product catalog (`docs/public/product-catalog.json`), portable managed -conformance projections (`docs/verification/managed-conformance-*.json`), and -packaged skill (`skills/`) are ignored generated outputs. `docs:generate` -prepares documentation and managed evidence; `evidence:generate` prepares only -the managed commitment and its runtime/skill dependencies. Edit skill -instructions and references in `skill-src/`; the skill task adds metadata to -the packaged copy without rewriting authored files. The generated manifest -commits to that exact packaged skill bundle. It is a portable projection of -the deterministic managed verifier, not a record of optional real-provider runs. -CI validates these outputs and retains them as artifacts instead of pushing -generated-only commits onto feature branches. Reviewed source metadata such as -`src/generatedPackageMetadata.ts` and `docs/error-contract.schema.json` remains -tracked and checked for freshness. Do not commit ignored generated outputs. -The build-generated product catalog contains documented facts and their provider -identity, rather than full runtime schema hashes. The managed skill contains -instructions and inventory metadata; doctor compares its installed files with -the canonical bundle. Schema-only fixes should not change these outputs or the -skill commitment in the conformance manifest. Runtime schema identity remains -available through doctor and `binary_session`. -Real-provider execution remains uncached; deterministic builds use Turbo. +`docs:generate` prepares documentation and portable managed evidence; +`evidence:generate` prepares only the managed commitment and its runtime/skill +dependencies. The ignored outputs are `docs/public/product-catalog.json`, +`docs/verification/managed-conformance-*.json`, and `skills/`. Edit authored +skill instructions in `skill-src/`. CI validates generated outputs and retains +them as artifacts; do not commit them. Reviewed source metadata, including +`src/generatedPackageMetadata.ts` and `docs/error-contract.schema.json`, remains +tracked and checked for freshness. -Local `npm test` runs every deterministic Vitest project without coverage or -retries. CI runs four coverage shards and merges JUnit and timing reports. -CI cancels superseded PR runs and skips package, Windows, and full test lanes -for documentation-only PRs. Coverage thresholds remain in `vitest.config.ts`. +The packaged skill contains instructions and inventory metadata, not runtime +schema hashes. Its conformance commitment covers that exact bundle and portable +verification; it does not attest optional real-provider runs. Schema-only changes +should not alter that commitment. Doctor and `binary_session` expose runtime +schema identity. -Tests that need a temporary directory must use -`createTestTempDirectory` from `tests/fixtures/temporaryDirectory.ts`. The -helper binds exact-path, awaited cleanup to the current Vitest case, including -failure and timeout completion. Run `npm run verify:test-temp-hygiene` to build -REA, execute the complete suite under a fresh `TMPDIR`, and reject any remaining -REA-owned temporary path. Never add a glob cleanup for shared `/tmp/rea-*` -content. +Build/generation commands hold a SQLite transaction around the entire Turbo +graph, including cache restoration. `.cache/rea-command-locks/` databases identify +locks, not stale files; never remove them while commands run. POSIX cancellation +retains process-group ownership until settlement; Windows waits for completion. +Real-provider execution is uncached. -Set `REA_LOG_LEVEL` to `trace`, `debug`, `info`, `warn`, `error`, `fatal`, or -`silent` to control structured JSON diagnostics. MCP mode defaults to `info` and -always writes logs to stderr so the stdio protocol remains intact. One-shot CLI -logging is opt-in and writes to stdout when a level is configured, preserving -machine-readable command output by default. Request arguments, bridge -authentication tokens, and environment data are redacted. +### Temporary files and diagnostics -Changes that claim real Hopper behavior must also be tested against the -source-owned, digest-bound conformance manifest. The verifier builds the -platform-native fixtures before starting Hopper: +Tests use `createTestTempDirectory` from `tests/fixtures/temporaryDirectory.ts` +for exact-path, awaited cleanup, including failures and timeouts. +`npm run verify:test-temp-hygiene` runs the suite under a fresh `TMPDIR` and +rejects remaining REA-owned paths. Never glob-clean shared `/tmp/rea-*` content. -```bash -npm run verify:hopper -``` +`REA_LOG_LEVEL` selects `trace`, `debug`, `info`, `warn`, `error`, `fatal`, or +`silent`. MCP defaults to `info` on stderr; one-shot CLI logging is opt-in on +stdout. Request arguments, bridge tokens, and environment data are redacted +from diagnostic logs. -On a self-hosted Linux runner with the setup-installed Xvfb dependencies, use: +## Real Hopper changes -```bash -npm run verify:hopper:linux -``` +Run `npm run verify:hopper` on a supported macOS host or +`npm run verify:hopper:linux` on the supported Linux demo runner. These build +native fixtures and verify the installed engine through CLI/MCP; package or +simulated-provider checks do not establish engine behavior. See +[real-toolchain verification](docs/testing.md#end-to-end-integration-and-golden-evidence) +for prerequisites and scope, and [Hopper setup](docs/installation.md#hopper) +for supported hosts and unattended startup. -Set `REA_HOPPER_CONFORMANCE_MANIFEST_PATH` only to verify another source-built -manifest. The normal commands use `build/conformance/manifest.json`; generated -fixtures and manifests remain ignored and must not be committed. - -The macOS and Linux real-Hopper workflows remain separate so a successful mock or package test cannot be reported as platform-runtime proof. Pull requests changing setup, launch, bridge, or Hopper behavior must state which real workflows ran and why either workflow was unavailable. - -Describe the behavior change and verification performed in the pull request. Never commit binaries, Hopper documents, credentials, `dist/`, `node_modules/`, or local planning artifacts. +Use the default `build/conformance/manifest.json` unless explicitly verifying +another source-built manifest with `REA_HOPPER_CONFORMANCE_MANIFEST_PATH`. +Generated fixtures and manifests remain ignored. ## Maintainer release checklist diff --git a/docs/testing.md b/docs/testing.md index c0846b184..9730f027b 100644 --- a/docs/testing.md +++ b/docs/testing.md @@ -1,76 +1,36 @@ # Testing REA -Prefer evidence in this order: full end-to-end workflows through the public -CLI or MCP with real production providers and no mocked dependencies, -integration across production data/API boundaries, then golden regressions -from real captured inputs. Keep a focused module test only for a distinct -failure or semantic case that stronger workflows cannot reliably reproduce. -A test's path, compiled imports, or suite name does not establish its depth. +Use this guide when choosing coverage, pruning tests, or running a verification +lane. For commands, jump to [developer commands](#developer-commands); for engine +prerequisites, use [real-toolchain lanes](#real-toolchain-verification-lanes). -Delete tests that only mirror getters, serialization helpers, enum mappings, -fixture helpers, prescribed call sequences, arbitrary counts or snapshots, -or cache object identity without a caller-visible contract. Do not retain a -weaker test merely because it is fast when a stronger workflow already proves -the same claim. Keep format-specific malformed representations and actual -capacity, cleanup, permission and target-identity regressions. -Prefer representative workflows over a Cartesian matrix when rows cross the -same boundary. Retain combinations that exercise a distinct interaction, such as -filtering within a full-output envelope. Keep distinct command handlers, -producer representations, failure reasons, and lifecycle states covered. -Consolidate their shared setup without hiding the inputs or expected evidence. +Prefer real public CLI/MCP workflows, then production-boundary integration, +then goldens captured from real producers. Keep module tests for distinct +failures or semantics that stronger workflows cannot reliably reproduce. +Paths, compiled imports, and suite names do not establish test depth. -Tests are grouped into Vitest projects so ownership, allowed dependencies and -runtime cost are visible from their paths. Before pruning a distinct boundary -regression, identify the replacement scenario and its assertions; passing -success journeys do not replace malformed input, cancellation, permission or -cleanup coverage. Trivial helper checks do not need a replacement. If only -tests consume a production module, trace CLI/MCP and verifier imports, -including imports of compiled files, and remove abandoned scaffolding with its -tests when it has no runtime consumer. +Prune tests that mirror implementation details or duplicate a stronger claim. +Before deleting a distinct regression, identify the replacement scenario and +assertions. Preserve malformed formats, cancellation, cleanup, permissions, +capacity, target identity, and combinations with distinct interactions. +Trivial helper assertions need no replacement. Remove abandoned production +scaffolding only after checking runtime and verifier consumers, including +compiled imports. -When a test expects a `Result` to succeed, throw its error before asserting the -returned value. An `.ok` assertion immediately before the same throwing guard -adds no coverage and hides the typed failure. A removed parser matrix can leave -one golden malformed-input case and a real filesystem lifecycle that proves -configure, update, readback, and removal; an add-only journey does not cover the -whole lifecycle. +Use representative scenarios rather than Cartesian matrices of equivalent +cases. Share setup without hiding inputs or expected evidence. For successful +`Result` values, throw the typed error before asserting the value; a preceding +`.ok` assertion adds no coverage. Protocol fixtures must reject unmodeled +commands and preserve actual producer reply shapes. -Protocol fixtures must model the commands they support and their producer reply -shapes. Reject unmodeled commands instead of returning a generic success; an -empty result is appropriate only for a supported command with no result fields. +Report unavailable authority as a named skip. Await owned cleanup before a +verifier reports success. Native process inspectors are released at Vitest +worker teardown; fork termination does not run Node exit hooks, and per-file +teardown would retire inspectors still needed by other files. -Keep transport-only in-memory MCP sessions in the MCP boundary lane and direct -filesystem workflows in the filesystem boundary lane. Real process-tree capture -belongs in the serial process boundary lane, including captures invoked through -MCP. Report unavailable capture authority as a named skip rather than returning -from the test without assertions. Reserve acceptance paths for journeys -through the compiled public CLI or MCP runtime. Calling a compiled private -workflow or injecting a provider still establishes its narrower integration -claim, even if a directory name calls it acceptance. - -Emit a verifier's final success report only after awaited cleanup. Observe final -process lineage after closing its owned resources. - -Owned JavaScript semantic graphs capture and schema-parse caller input before -returning an iterator. Whole-graph hashing, batched integrity checks and sealing -run through the application cancellation driver; only full completion grants -the private immutable-graph proof. Regression coverage checks canonical IDs, -validation diagnostics, nested caller mutations across yields, cancellation -before result validation and subsequent SDK requests. Single-file parsing, -schema cloning and canonical sorting remain synchronous phases; measure them -separately when assessing cancellation latency on real applications. - -The Vitest runner releases cached native process inspectors during awaited -worker teardown. Fork termination does not run Node's normal exit hooks; a -per-file teardown would instead retire shared inspectors before later files -can use them. - -Measure slow files before removing capacity regressions. Optimize repeated -process startup, fixture construction, and production algorithms when they -dominate runtime. Keep at least one input beyond each formerly failing size or -depth; smaller fixtures do not establish that truncation or stack exhaustion -has been fixed. Record the toolchain, coverage mode, and cache conditions when -comparing durations. +Measure slow cases before pruning capacity coverage. Retain inputs beyond the +formerly failing size/depth and distinguish cold runs, warm builds, coverage, +and toolchain conditions when comparing timings. ## Behavioral depths @@ -93,12 +53,6 @@ registries but do not cross production filesystem, process, socket, or browser b Boundary tests cross one production boundary. Only acceptance tests assemble the complete runtime or invoke the compiled product surface. -Binary session, registry, lazy-client and snapshot module cases live under -`src/application/binary/`. Their composition cases stay in -`tests/composition/analysis-sessions/`; snapshot persistence and actual SDK calls -stay in the filesystem and MCP boundary lanes. The shared injected session fixture -serves several capabilities and remains under `tests/fixtures/`. - Focused immutable builders and recording ports shared by one test family live beside their production owner as `src/**/*.fixture.ts`. They are typechecked with the suite and excluded from package builds; broader runtime and provider @@ -165,29 +119,6 @@ an empty versioned scan or a suffix that only repeats the original path. It runs in macOS CI. -Apple artifact verifiers live in `scripts/verify/apple/`, with the macOS bundle -builder under `scripts/fixtures/apple/` and NIB byte fixtures beside the decoder -in `src/artifacts/apple/`. The npm entrypoints are unchanged. Format-specific -Swift/XIB/asset sources and goldens retain their locations; real Apple workflows -resolve them from the verifier file URL and run in the macOS CI lane. - -Portable native semantics and their tests live in `src/domain/native/`; shared -analyst workflows and service-lane tests live in `src/application/native/`. -Named native contracts live in `src/contracts/native/`. Provider protocol and -host UI tests retain their adapter/boundary lanes, and real verifier command -names remain unchanged. -The existing Apple CI job also runs the host Swift-demangling CLI/MCP regression -suites, including option-like symbols, carriage returns and multiline rejection. - -Process semantics/tests live in `src/domain/process/`; PTY capture implementation -and forked helper tests live in `src/process/capture/`. Evidence projection and -file workflows live in `src/application/process/`; producer-backed Evidence/host -cases run in the serial `tests/boundary/process/` lane. Installed-package probes -load the compiled capture capability owner, including the missing-optional-module -case. Real capture tests preserve actual descendant and cleanup checks. -The existing Apple job also exercises the relocated filesystem snapshot identity, -cancellation and descriptor cleanup regressions on macOS. - Prepare the native inspector before cases that measure producer output or exit behavior; keep startup deadlines and cancellation in distinct cases. Run real process-capture verification separately from package or build checks. On macOS, @@ -198,8 +129,7 @@ MCP SDK transport tests with recording providers remain integration tests. They are useful for schema drift and failure projection but do not prove that Hopper, Ghidra or another substituted engine works. `verify:package` proves packaging/install behavior and fake-provider integration; use the corresponding -real-provider lanes for engine claims. Packed-bridge checks verify shipped files -and Python syntax without deleted mock fixtures or source-spelling assertions. +real-provider lanes for engine claims. Packed-bridge checks verify shipped files and Python syntax. Real Apple dispatch and Interface Builder verifiers currently prove format integration through production readers. @@ -274,7 +204,7 @@ Do not call handcrafted utility output or synthetic binary builders real-data goldens. Keep unsupported binary layouts and malformed boundaries as targeted regressions until a real fixture establishes equivalent coverage. -`verify:browser` exercises source-map failure isolation and expanded-output limits through the compiled CLI and stdio MCP with real Chrome. It also submits five 2 MiB source-map annotations, checks retained script identities and explicit map omissions, and closes its owned fixture target. These public workflows replace redundant metadata-admission assertions; replacement and document-reset boundaries remain focused producer regressions. +`verify:browser` exercises source-map failure isolation and expanded-output limits through the compiled CLI and stdio MCP with real Chrome. It also submits five 2 MiB source-map annotations, checks retained script identities and explicit map omissions, and closes its owned fixture target. Replacement and document-reset behavior has separate producer-boundary coverage. `verify:browser` also captures a source-owned noise canvas as a real PNG above 8 MiB through the CLI and stdio MCP, with complete byte/digest parity and real PNG @@ -399,9 +329,7 @@ Instruction inspection and containing-function lookup also agree across hexadecimal case, leading zeros, and encoded default address-space spellings. The same source-acquisition workflow exercises missing and directory-replaced inputs, plus real read-permission denial on a non-root host. Root runs report -that permission-denial check as unverified. A focused filesystem integration -check retains selected-platform routing and exclusive creation; two copy/digest -checks duplicated by the real workflow have been removed. +that permission-denial check as unverified. Filesystem integration separately checks selected-platform routing and exclusive creation. Namespace annotation probes compile a separate host C fixture with C++ ABI symbols, avoiding a C++ runtime prerequisite. Real Ghidra demangling supplies duplicate leaf names in two top-level namespaces and a nested namespace. The @@ -416,8 +344,7 @@ Successful-result and oversized-error delivery constraints must identify their successfully retained Evidence records; export must recover every source byte, complete annotation and original error diagnostic, with CLI parity -and an unchanged executable. The focused formatter check retains only the -separate case where recording was not acknowledged. +and an unchanged executable. Formatter integration separately checks unacknowledged Evidence recording. Long ordinary procedure names and encoded address-space selectors must produce normal validation errors without exhausting Java's regex stack or losing the private bridge connection. Each rejection is followed by a real provider lookup; @@ -457,14 +384,9 @@ the CLI and stdio MCP, including special filenames, unresolved discovery locations, and independently resolved loaded scripts. Double-quote filenames are tested on POSIX only because Windows does not support them. -Inspector adapters live in `src/inspector/`, with loopback fixtures under -`tests/fixtures/inspector/` and forked producer boundaries under -`tests/boundary/inspector/`. The real lane is owned by -`scripts/verify/inspector/runtime-observation.mjs` and -`tests/conformance/inspector/`. Browser and Inspector deliberately share the -existing browser CDP transport/value and file-location helpers. Node fixture -success proves this Inspector workflow on the tested host; it does not prove -Electron GUI behavior or another engine. +Node Inspector verification does not establish Electron GUI or other engine +behavior. Browser and Inspector share CDP transport/value and file-location +helpers; their producer workflows remain distinct. ## Android APK analysis @@ -486,14 +408,6 @@ See [Android analysis](android-analysis.md) for boundaries and resource budgets. Authenticated IPA and macOS application inventory projection is documented in [Apple application analysis](apple-application-analysis.md). -The lane and owned-process cancellation helper live in `scripts/verify/android/`. -The explicit downloader and fixed manifest live in `scripts/fixtures/android/`; -their default remains repository-root `_reference/apk-integration/`. Producer -fixtures live in `tests/fixtures/android/`, boundary cases in -`tests/boundary/android/`, and inventory projection composition cases in -`tests/composition/android/`. Real engine success and synthetic protocol success -are separate proof levels. - Synthetic producer regressions run independently: ```sh @@ -627,13 +541,6 @@ installation, CLI/MCP discovery, target-free analysis, configuration backups/recovery, Evidence and owned lifecycle; Inspector checks execute source-owned loopback targets and special filename cases. -Each native package host runs package and Inspector checks on the same runner -after a single dependency installation. macOS uses one OS version with one -native baseline job per architecture. Windows runs Inspector with its curated -capability/package checks before switching Node versions for directory-identity -regressions. Inspector steps retain a ten-minute deadline; Apple artifact checks -retain their own macOS 15 arm64 job for Xcode-dependent workflows. - These native baseline checks complement the Linux source-test shards and the separate Apple-artifact and real-provider lanes. Actual Hopper, Ghidra, IDA, browser and managed-tool claims require their corresponding verification lanes. @@ -703,23 +610,15 @@ broad changes or diagnosing CI, rather than after every edit. Package/install changes additionally need package verification; provider changes need actual provider evidence. -Each Vitest project uses up to two workers, bounded by available host parallelism. -Default-group projects can run together. Only `process-boundary` runs in the -later sequence group and serializes its files because process-tree observations -share host sampling resources. Tests that launch the real process harness belong -in `tests/boundary/process/`, including CLI command and environment tests. -Acceptance and process-global files retain isolated -forks but are not serialized; isolation does not imply serial scheduling. CI uses -the same per-project worker bound. -The pure domain/contracts and recording-port service projects share one worker -module context because their tests own no mutable runtime resources. MCP -boundary files also share the immutable server module graph while creating and -closing independent in-memory sessions. Adapter, composition, acceptance, -process-global, and other boundary projects retain per-file isolation. -Build and documentation writers use checkout-local locks for their respective -output files. `npm test` delegates its build to the build lock; the Vitest suite -does not hold a test-command lock. `check:pr` runs its test task before starting -generated-document validation. +Vitest projects use up to two workers, bounded by host parallelism. +`process-boundary` runs later with serial files because process-tree sampling +shares host resources. Acceptance and process-global cases use isolated forks +without serial scheduling. Pure domain/service and MCP boundary projects share +immutable module graphs; each MCP session still owns its resources. Other +projects retain per-file isolation. See `vitest.config.ts` for current settings. + +Build/documentation writers hold checkout-local output locks. Tests do not +hold a broad command lock; `check:pr` finishes tests before document validation. Vitest and Node persistent compile caches are deliberately not enabled by default. To evaluate repeated local runs, opt in for both cold and warm @@ -735,13 +634,9 @@ instrumentation remains equivalent. ## Coverage and timing -CI owns coverage. The aggregate floors are 65% statements, 60% branches, 60% -functions, and 68% lines. `src/domain/**` must reach 80% statements, 75% -branches, 75% functions, and 80% lines. `src/contracts/**` must reach 85% -statements, 80% branches, 80% functions, and 85% lines. Thresholds are -glob-specific rather than per-file and are never updated automatically. -Coverage does not replace named boundary, acceptance, or real-provider scenario -matrices. +CI owns coverage; aggregate and domain/contract thresholds are maintained in +`vitest.config.ts`. They are glob-specific and never updated automatically. +Coverage does not replace named boundary or real-provider scenarios. CI runs four native Vitest shards without retries. Each shard emits a blob report; the merge job produces aggregate coverage plus JUnit and JSON timing @@ -815,12 +710,6 @@ unknown chunks, depth limits and extracted child digests. The optional Neither optional toolchain is a base-lane prerequisite. See [firmware analysis](firmware-analysis.md) for limits and unverified formats. -The real entrypoint is `scripts/verify/firmware/analysis.mjs`; the fixture runner -and unchanged Python producer belong together in `scripts/fixtures/firmware/`. -Producer fixtures and boundaries remain in `tests/fixtures/firmware/` and -`tests/boundary/firmware/`. Source-fixture generation proves the runner and -independent oracle; actual Binwalk/Unblob CLI/MCP proof requires selected tools. - ### JavaScript source recovery Build once with `npm run build:cached`, then run `npm run verify:javascript:recovery`. diff --git a/docs/tool-design.md b/docs/tool-design.md index ed8c09aaf..934722850 100644 --- a/docs/tool-design.md +++ b/docs/tool-design.md @@ -1,112 +1,87 @@ # MCP tool design -Design tools around analyst tasks. A provider API is an implementation detail; -expose it directly when it gives agents a useful, reusable capability that -cannot be expressed through an existing contract. Do not impose caller-facing -limits merely to make work fit an assumed agent budget. Keep real input-format, -protocol, authority, and resource-safety constraints, and report their effects. - -## Choose the tool shape - -| Shape | Use it for | Contract should return | -| --------------------- | -------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------- | -| `inspect` | Facts about one explicit target, address, object, or resource | Relevant fields, source locations, and facet-level availability | -| `search` / `list` | Finding candidate targets or entities | Stable ordering and useful result context; expose pagination when the result format or caller's query needs it | -| `trace` | Relationships across code, metadata, UI resources, or observations | Typed edges, supporting evidence, and unresolved paths; identify any genuine traversal boundary | -| `compare` | Two explicitly identified artifacts, versions, or evidence sets | Paired identity, comparable coverage, and deltas with evidence | -| `workflow` | A distinct analyst outcome that benefits from composition inside REA | A useful inline result, contributing evidence, and partial/unavailable facets | -| `observe` / `capture` | A question that requires runtime behavior | Required authority, launch/attach behavior, real operational constraints, lifecycle, and cleanup status | - -These are task shapes, not required prefixes. Name a tool for the action and -object agents reason about; keep the name distinct from nearby alternatives. - -## Prefer primitives; compose workflows - -Let agents compose experiments with ordinary commands, scripts, and local -fixture servers. A custom orchestration language, replay engine, or separate -prepare/execute plan needs an observed requirement those primitives cannot -satisfy. Expose useful inspection directly; a plan-only tool without an executor -does not establish runtime behavior. - -Start with a primitive when one call can report a reusable fact about one -identified object or relationship, such as an instruction decode, a type -layout, a reference, a dispatch target, or a resource graph. The same primitive -should be useful across different applications or analyst questions. A -format-specific decoder can still be a primitive when it describes a reusable -format rather than one application's product behavior. - -Add a workflow when repeated analysis shows that callers need the same -multi-source result and REA can join the evidence without hiding important -choices or uncertainty. A useful check is whether the workflow remains -meaningful for different applications that share the relevant evidence types. -If its purpose depends on one application's business rules or product concepts, -keep that interpretation outside the general tool contract and expose the -underlying evidence through reusable primitives. - ## Decide whether to add a tool -1. State the user intent and the smallest result that answers it. -2. Inspect the canonical tool contracts, current provider capabilities, and - representative CLI/MCP traces. Record the nearest competing tool. -3. Extend an existing contract when the intent and result are the same. Add a - batch or composed workflow when it materially improves a demonstrated task; - do not force agents through extra calls or collapse distinct tasks into one - tool. Add a tool when it answers a distinct analyst question or has a - materially different authority or result contract. -4. Keep public tool names and result semantics provider-neutral. Put engine- - specific parsing and protocol handling in adapters. Advertise exact support - per provider; do not create parallel tools just because engines differ. -5. Keep prompts optional and concise. They may point out useful tools, but must - not prescribe a call sequence when the task can be answered directly. +Design around the analyst question. Inspect the current canonical contracts, +provider capabilities, and nearest existing tool before changing the surface. +Extend a tool when intent and result contract match; add one for a distinct +outcome or materially different authority. Public names and semantics remain +provider-neutral, with exact coverage reported per provider. -Avoid opaque mode flags and mega-tools that combine unrelated discovery, -execution, and mutation. Prefer a useful direct tool call over a tool sequence -or model-authored loop when one call can answer the question. +Keep the catalog complete; report capability/session availability instead of +truncating schemas. Serialized bytes alone do not establish agent usability. + +## Choose the tool shape + +| Shape | Use it for | Result | +| --------------------- | ------------------------------------------- | ------------------------------------------------------------------------- | +| `inspect` | Facts about one target, object, or resource | Relevant fields, locations, and facet availability | +| `search` / `list` | Candidate discovery | Stable ordering and context; pagination when the query or format needs it | +| `trace` | Relationships across evidence | Typed edges, supporting facts, and unresolved paths | +| `compare` | Explicitly paired artifacts or observations | Identity, comparable coverage, and supported deltas | +| `workflow` | A recurring outcome needing composition | Inline answer, contributing evidence, and partial facets | +| `observe` / `capture` | Runtime behavior | Authority, launch/attach effects, lifecycle, and cleanup status | + +These are task shapes, not mandatory prefixes. Name tools for the action and +object callers reason about. Avoid unrelated discovery/execution/mutation modes +inside one tool. + +## Prefer primitives; compose workflows + +A primitive reports a reusable fact about an identified object or relationship. +A format-specific decoder qualifies; one application's business interpretation +belongs outside the general contract. Expose provider APIs when they add a +useful capability that existing tools cannot express. + +Compose a workflow when repeated use demonstrates a multi-source outcome REA +can join without hiding caller choices or uncertainty. Batch homogeneous reads +when that removes repeated work while preserving per-item outcomes. Keep prompts +optional and avoid prescribing call sequences that a direct tool can replace. + +Agents can compose experiments with ordinary commands, scripts, and local +fixture servers. Custom orchestration languages, replay engines, or separate +prepare/execute plans need an observed requirement those primitives cannot +satisfy. A plan without execution does not establish runtime behavior. ## Define the contract -- Use strict object inputs with explicit required fields and enums. Add numeric - bounds only when the format, protocol, authority, or a measured resource - constraint requires them. -- Return complete results by default. Make real pagination, truncation, partial - failure, cancellation, and unavailable states explicit. Missing evidence is - unknown, not empty or false. -- Return task-oriented results inline with artifact identity, provider/version, - addresses or resource paths, Evidence references, confidence, and relevant - limitations. Do not require agents to fetch a resource or dereference an - opaque Evidence link to understand a tool result. -- Separate observed facts from derived and inferred edges. Cite the evidence - supporting every important relationship; preserve unresolved edges. -- Declare read-only, mutation, process, filesystem, network, and UI effects - truthfully in the contract and enforce the target and lifecycle named in the - request. -- Keep provider-specific types out of provider-neutral domain and application - layers. Normalize supported provider results without implying equal coverage. +- Use strict inputs with explicit target identity and required fields. Preserve + meaningful target, action, capture, and output choices; remove ignored options + and repeated permission declarations. +- Return complete evidence inline, with artifact/provider identity, source + locations, Evidence references, and relevant confidence and limitations. + Missing coverage is unknown, not absence. Avoid extra lookups solely to + understand the answer or its provenance. +- Keep observed, derived, and inferred relationships distinct. Preserve + unresolved edges and partial provider facts instead of requiring every + provider to supply the same metadata. +- Declare process, filesystem, network, UI, and mutation effects truthfully; + enforce the requested target and lifecycle. +- Add limits only for real format, protocol, authority, or measured resource + constraints. Report pagination, truncation, cancellation, and partial failure. + An assumed agent budget is insufficient justification for a limit. ## Implement and evaluate -- Put shared user workflows in the application layer; keep MCP translation in - the server adapter and CLI behavior aligned with the same workflow. Share - Evidence provenance, unknown projection, and eligible snapshot binding through - application/session owners. Matching result payloads alone do not establish - equivalent retention or replay behavior. -- Derive each facet's coverage from examined records, unsupported representations, - and exhaustiveness. Keep issues structured until presentation; do not select a - facet by matching diagnostic prose or infer completeness from an empty failure - list. Preserve available observations independently of cleanup success. -- Measure the resource consumed before choosing a limit. Value alternatives, - retained records, trace-frame products, output bytes, and temporary-file growth - need different accounting. Enforce a justified budget before the costly step - and report its effect; an input-byte or graph-node cap does not bound every stage. -- Update the canonical contract inventory, output schemas, examples, generated - catalog, and relevant docs together. -- Verify valid, malformed, boundary, unavailable, cancellation, and partial - results at the owning boundaries. Use real-provider checks for claims that - depend on external analysis engines. -- Evaluate representative broad and direct task prompts on the intended MCP - host when tool selection is the question. Record the host, model-facing tool - catalog, selected tool, arguments, errors, retries, and task outcome. - Schema validity or word-overlap heuristics do not prove discoverability. +Put provider protocols/parsing in adapters and shared CLI/MCP workflows in the +application layer. Share Evidence provenance, unknown projection, and session +retention semantics; matching payloads alone do not establish parity. -See [mcp-contracts.md](mcp-contracts.md) for shipped runtime behavior and -[testing.md](testing.md) for test boundaries and verification lanes. +Measure the resource before choosing a budget and enforce it before the costly +step. Account for value alternatives, trace-frame products, retained records, +serialization expansion, and temporary-file growth separately. Derive facet +completeness from examined records and exhaustiveness, rather than diagnostic +prose or an empty failure list. Preserve observations independently of cleanup +success. + +Update canonical contracts, output schemas, examples, generated artifacts, and +relevant docs together. Verify converted advertised schemas and success, +malformed, unavailable, cancellation, and partial outcomes through affected +consumers. Real-engine claims require the matching provider lane; see +[testing.md](testing.md). + +When discoverability changes, evaluate representative broad, direct, and negative +tasks on the intended host. Record host/model/catalog, selected tools, arguments, +errors, retries, and task outcomes. Schema validity and keyword heuristics do not +establish successful tool selection. See [mcp-contracts.md](mcp-contracts.md) for +shipped runtime behavior.