diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index c60c1e9e2..07c71c4c5 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -10,9 +10,12 @@ and maintenance cost. Keep cosmetic polish with useful fixes; confusing or incorrect user instructions warrant standalone corrections. Collect other cosmetic-only suggestions in an issue rather than separate PRs. -For capability organization and provider composition, follow the incremental -[migration guide](docs/capability-migration.md). Run `npm run verify:test-discovery` -after adding or moving tests. +For capability organization and provider composition, use the +[architecture map](docs/architecture.mermaid). Production factories in +`src/composition/` construct fresh providers without starting engines or acquiring +targets; optional adapter failures must preserve successful peers. Run +`npm run verify:test-discovery` after adding or moving tests. Resolved import +boundaries are checked by `npm run verify:module-boundaries`, included in lint. When adding or changing an MCP tool, follow the [tool design guide](docs/tool-design.md) and preserve the canonical contracts and generated catalog. diff --git a/docs/.vitepress/config.ts b/docs/.vitepress/config.ts index 314362009..e1c184993 100644 --- a/docs/.vitepress/config.ts +++ b/docs/.vitepress/config.ts @@ -55,7 +55,7 @@ export default defineConfig({ { text: "Ghidra on Windows", link: "/windows-ghidra-p0" }, { text: "Ghidra DOS analysis", link: "/ghidra-dos" }, { text: "Ghidra NativeAOT", link: "/ghidra-nativeaot" }, - { text: "Provider evaluation", link: "/provider-evaluation" }, + { text: "Ghidra semantics", link: "/ghidra-provider" }, ], }, { @@ -66,7 +66,6 @@ export default defineConfig({ { text: "Tool design", link: "/tool-design" }, { text: "Testing", link: "/testing" }, { text: "Releasing", link: "/releasing" }, - { text: "Architecture decisions", link: "/adr/README" }, ], }, ], diff --git a/docs/adr/0001-provider-selection-and-analysis-profiles.md b/docs/adr/0001-provider-selection-and-analysis-profiles.md deleted file mode 100644 index 588fb3ebc..000000000 --- a/docs/adr/0001-provider-selection-and-analysis-profiles.md +++ /dev/null @@ -1,572 +0,0 @@ -# ADR-0001: Provider selection and analysis profiles - -- Status: Accepted -- Date: 2026-07-15 -- Implementation status: The provider registry, deterministic selection, - target binding, analysis-profile commitment, and snapshot/Evidence migration - are implemented. Ghidra discovery, target/profile resolution, doctor checks, - the private headless-session foundation, thirteen read-only inventory/load-image - capabilities, and twelve function-analysis capabilities are implemented with - real cross-format conformance. Linux/macOS also supports atomic session - function annotations; experimental Windows x64 P0 uses bundled native controls - and admits only the 25 read-only operations. See the - [release boundary](../installation.md#released-package-and-main). - -## Context - -Before this decision, REA composed the artifact, native macOS, and Hopper -providers with `CompositeProvider`. That class intentionally requires every -operation to have one route. Hopper and Ghidra will both implement the deep -static-analysis operation family, so putting both into that composite would -either fail at construction or make array order an undeclared selection policy. - -The active `BinarySession` also owned one composite client. `open_binary` had no -provider selector, and the domain `BinaryTarget` contained Hopper loader -arguments. Analysis snapshot v1 committed those loader arguments and each -cached entry's provider identity, but it did not commit the selected analysis -engine, engine version, or complete analysis configuration. Two analyses of the -same bytes could therefore disagree because of language, compiler, loader, -analyzer, or provider-version differences while appearing target-compatible. - -The decision addressed the earlier representations of -[`CompositeProvider`](https://github.com/morluto/rea/blob/main/src/application/binary/CompositeProvider.ts), -[runtime composition](https://github.com/morluto/rea/blob/main/src/composition/binary.ts), -[`BinarySession`](https://github.com/morluto/rea/blob/main/src/application/binary/BinarySession.ts), -[`BinaryTarget`](https://github.com/morluto/rea/blob/main/src/domain/binaryTarget.ts), -[analysis snapshot](https://github.com/morluto/rea/blob/main/src/domain/analysisSnapshot.ts), and -[session lifecycle inputs](https://github.com/morluto/rea/blob/main/src/contracts/sessionLifecycleInputs.ts). - -The next deep provider must not weaken the existing product invariants: - -- stable provider-neutral CLI and MCP tool names; -- exact provenance and explicit unavailable or degraded capabilities; -- CLI and MCP parity; -- no silent provider fallback; -- provider-specific code outside the domain and application models; -- local, actionable diagnostics with only genuine secrets redacted; and -- snapshot reuse only when semantic analysis inputs are identical. - -## Decision drivers - -1. Hopper and Ghidra need to declare overlapping operations without ambiguous - dispatch. -2. A target must have one explainable deep-analysis producer for its lifetime. -3. Target-free discovery must remain useful and must not start an analysis - process. -4. Existing Hopper-only configurations must keep working when no selector is - supplied. -5. Adding a second installed provider must not silently change results. -6. Snapshot and Evidence compatibility must include every setting that can - materially change analysis. -7. Provider-specific open settings must not leak into `BinaryTarget` or become - generic application fields. - -## Terminology - -- **Deep provider**: an engine such as Hopper or Ghidra that implements the - overlapping static-analysis operation family. -- **Auxiliary provider**: a provider with a disjoint operation family, such as - artifact inventory or native macOS inspection. -- **Candidate**: a registered deep provider before target selection. -- **Binding**: the selected deep provider and analysis profile attached to one - active target. -- **Analysis profile**: a canonical commitment to the selected provider, - provider version, and normalized semantic analysis settings. -- **Available**: the adapter and its required local runtime can be used on this - host. -- **Supported**: the candidate accepts the parsed target and supplied open - options. Availability and target support are separate observations. - -## Decision - -### 1. Separate disjoint composition from overlapping selection - -`CompositeProvider` remains the mechanism for operation families whose routes -are disjoint. It must continue rejecting duplicate operation declarations. It -must not acquire priority, racing, fallback, or provider-selection behavior. - -A new `AnalysisProviderRegistry` owns the configured deep-provider candidates. -Provider IDs are unique; duplicate IDs are a configuration error. The registry -may hold multiple candidates that declare the same operation. It performs -candidate discovery, target-support evaluation, option parsing, profile -resolution, and selection, but it does not choose a different provider for each -request. - -After selection, the session composes exactly one bound deep provider with the -auxiliary disjoint families: - -```text - target-free discovery - | - AnalysisProviderRegistry - / \ - Hopper Ghidra - \ / - one selected binding - | - +-------------------------+-------------------------+ - | | | - selected deep family artifact family native family - +-------------------------+-------------------------+ - | - BinarySession -``` - -An auxiliary operation continues to report its actual auxiliary provider as -provenance. A deep operation always reports the provider in the active binding. -The synthetic composite identity is never evidence that two engines jointly -produced one observation. - -Operation ownership is fixed by family: - -| Operation family | Executor | -| ------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------- | -| Deep direct operations and provider-native function analysis | The one bound Hopper or Ghidra provider | -| Application-enhanced deep workflows | Application composition over the bound provider's declared operations | -| Native macOS inspection | The disjoint native provider | -| Artifact inventory and extraction | The disjoint artifact provider | -| Browser and Electron observation | Their independently authorized CDP providers; they do not participate in deep-provider selection | -| Session lifecycle, Evidence, comparisons, and reconstruction | The application/session layer using the exact provider provenance recorded by their source operations | - -Selecting a deep provider selects the whole deep family. If that provider marks -an operation unavailable, REA returns the unavailability; it does not borrow -that one operation from another deep provider. Application-enhanced workflows -run only when the bound provider supplies their declared dependencies. - -### 2. Discover candidates without launching analysis - -Each registered candidate exposes three bounded adapter-owned observations: - -1. identity and declared capability descriptors; -2. host availability, including an actionable rejection when unavailable; and -3. target support plus normalized profile resolution when a target is known. - -Target-free discovery may inspect configuration, manifests, executable -metadata, and documented version files. It must not import a target, create a -project, open a GUI, or start the long-lived analysis process. Any short-lived -version probe added later must be explicitly bounded and truthfully declared as -a process effect. - -Candidate output is sorted by provider ID for determinism. Sort order is not -selection priority. Availability and support use stable codes plus a human -diagnostic: - -| Code | Meaning | -| ---------------------------- | --------------------------------------------------------------- | -| `not_configured` | Required provider location or configuration is absent | -| `executable_missing` | The configured provider entry point is missing or not runnable | -| `runtime_missing` | A required provider runtime, such as Java, is unavailable | -| `unsupported_host` | The host platform or architecture is unsupported | -| `unsupported_version` | The discovered provider or runtime version is unsupported | -| `target_kind_unsupported` | The provider does not accept this target kind | -| `target_format_unsupported` | The provider does not accept this target format | -| `architecture_unsupported` | The selected target architecture is unsupported | -| `target_role_unsupported` | The provider rejects this executable image role | -| `managed_target_unsupported` | The provider does not admit managed code through this deep path | -| `open_options_invalid` | The selected adapter rejected its open options | -| `version_unresolved` | A concrete analysis-engine version could not be committed | - -Diagnostics retain useful local paths, observed versions, architectures, and -mismatch details. Credentials, authorization headers, license secrets, and -other genuine secrets remain redacted. - -### 3. Use one deterministic selection algorithm - -The caller-visible selectors are: - -- MCP `open_binary.provider_id`; -- `rea ... --provider ` on every one-shot command that needs - deep analysis; and -- `REA_ANALYSIS_PROVIDER=`, defaulting to `auto`. - -The literal value `auto` is reserved and cannot be a provider ID. Precedence is -request or CLI flag, then environment configuration, then the default `auto`. -An explicit request for `auto` overrides a concrete environment preference and -re-enters automatic selection. - -Selection occurs after provider-neutral target parsing and before a deep client -is created: - -1. A concrete requested provider ID must exist, be available, support the - target, accept its options, and resolve a concrete version and profile. Any - failure returns a typed selection error; another provider is never tried. -2. `auto` filters candidates by availability and target support. -3. Exactly one usable candidate is selected with source - `auto-single-candidate`. -4. More than one usable candidate is an `ambiguous` selection error. The - diagnostic lists the candidates and asks for `provider_id`, `--provider`, or - `REA_ANALYSIS_PROVIDER`. REA does not prefer Hopper, Ghidra, registration - order, the fastest startup, or the first health response. -5. When no deep candidate is usable, an automatic open may still establish an - unbound target so disjoint artifact or native operations remain available. - Deep operations are then explicitly unavailable with the candidate - rejections. A concrete request, or a concrete environment preference for a - deep-eligible target, remains a hard requirement and fails instead of - becoming unbound. -6. A target that no deep adapter declares support for, such as an artifact-only - target, is `not_applicable` rather than a failed deep analysis. Supplying a - concrete provider in that open request is still an actionable unsupported- - target error. A global environment preference does not make an artifact-only - target require a deep provider. - -This preserves target opening for non-Hopper capabilities on hosts without a -deep engine while ensuring that installing a second usable deep engine never -changes analysis silently. - -### 4. Bind once and never fail over mid-session - -An active deep binding contains: - -```text -canonical target identity -selected provider identity and concrete version -selection source: request | environment | auto-single-candidate -analysis profile commitment -owned provider client, once started -``` - -The binding is immutable until `close_binary` or an explicit open/switch -transaction. Provider client creation may remain lazy, but laziness cannot -change which provider was selected. - -- Opening the same canonical target without a new selector keeps the existing - binding. -- Opening the same target with the same provider and profile is idempotent. -- A different explicit provider or a profile-changing option is a real switch: - drain active calls, close owned resources, resolve the new binding, and start - it as one serialized transaction. -- If a switch fails, restore the prior target and binding on a best-effort - transactional path and return the original switch error. -- Provider process exit, timeout, cancellation, capability failure, or adapter - error never triggers fallback. The binding stays visible and unavailable - until the caller explicitly switches or closes it. -- A snapshot never chooses or changes the provider. Selection happens first; - snapshot compatibility is checked against the resulting binding. - -Calls already admitted to one binding complete or cancel against that binding. -No request can cross a concurrent target or provider transition. - -### 5. Keep provider open options adapter-owned - -`BinaryTarget` retains only provider-neutral source identity and classification: -canonical path, source digest, generic kind and format, selected architecture, -and available architectures. Hopper loader arguments leave the domain model. -Provider-owned databases use a generic analysis-database classification; their -adapter determines support. A `.hop` analysis database is therefore accepted only -by Hopper without teaching the domain how to open it. Ghidra v1 continues to use -private temporary projects and does not open user projects. - -The generic transport may carry one bounded JSON `provider_options` object. -Only the selected adapter parses its keys and reports `open_options_invalid` at -that boundary. If this object is exposed over MCP, the CLI parity surface is -`--provider-options-json` with the same JSON schema and limits. Unknown keys, -wrong-provider options, excessive depth or size, and non-JSON values fail -closed. - -Environment settings that are inherently adapter-specific, such as the Hopper -launcher or future Ghidra installation path, remain in adapter configuration. -They do not become `BinaryTarget` fields. The application layer transports an -opaque, already-bounded JSON value and the generic commitment described below; -it does not interpret Hopper or Ghidra option names. - -### 6. Commit a canonical analysis profile - -Before creating a deep client, the selected adapter normalizes every semantic -input that can affect analysis. Examples include Hopper loader selection and -architecture flags, or Ghidra language ID, compiler specification, import mode, -analyzer set, and analyzer options. Defaults are made explicit, unordered sets -are sorted, and implicit host-dependent choices are resolved. - -The adapter returns this provider-neutral envelope: - -```json -{ - "provider": { "id": "ghidra", "version": "" }, - "parameters": {}, - "digest": "" -} -``` - -`parameters` is bounded JSON interpreted only by that provider adapter. It -contains analysis semantics, not credentials, authorization data, temporary -paths, installation paths, process IDs, timestamps, or other incidental host -state. The generic domain validates the envelope and digest but treats the -parameters as opaque data. - -A selected deep binding requires a concrete provider version. If the adapter -cannot determine it, the candidate is unavailable with `version_unresolved`; -REA does not persist a cache that merely commits `null` or "whatever is -installed". Provider identity, normalized parameters, or provider version -changes produce a different digest. - -The same target, configuration, and provider must resolve the same profile over -CLI and MCP. Profile resolution must be side-effect-free and complete before -the provider imports the target. - -### 7. Make snapshots provider- and profile-exact - -Analysis snapshots replace Hopper loader arguments with an explicit binding -commitment: - -```text -target: digest + generic format/kind + selected architecture -binding: provider identity + analysis profile commitment -entries: exact immutable queries -evidence_bundle: retained Evidence records -``` - -The snapshot target identity, canonical snapshot content digest, and every -query ID commit the provider ID, concrete provider version, and profile digest. -A cached entry is reusable only when all of the following match exactly: - -- target digest, generic format and kind, and selected architecture; -- provider ID and provider version; -- profile digest; -- operation and canonical parameters; and -- the existing cacheability and immutability rules. - -Provider or profile switches select a separate cache partition and cannot reuse -entries from the previous binding. Auxiliary operations are cacheable only if -their provider defines an equivalent explicit profile commitment; otherwise -they remain Evidence but not snapshot entries. - -Snapshot v1 cannot be parsed as the current snapshot format. Its `loader_args` -cannot prove which engine version, defaults, or analyzer settings produced the -entries, so snapshots must be recaptured before they can be used as a cache. -An embedded Evidence bundle can still be imported separately through the -Evidence import flow; it is never promoted into an analysis cache. - -### 8. Commit concrete or explicitly unknown analysis profiles in Evidence - -Every Evidence record includes `analysis_profile`, either the canonical profile -envelope above or `null` when no profile was observed. Deep-analysis observations -commit the resolved provider ID, version, and effective configuration in this -profile. The field participates in the Evidence semantic ID. - -An explicitly unknown profile can be displayed and cited, but cannot establish -profile compatibility, seed snapshot replay, or support an "unchanged" -conclusion that depends on identical analysis semantics. Snapshot query entries -must have matching profile-bound Evidence; imports reject missing or mismatched -observations instead of silently dropping query entries. Both `entries` and -`workflow_entries` are required arrays, including when empty. - -Different providers or profile digests are never snapshot-compatible. A -derived comparison may compare cross-provider normalized facts only when its -own contract explicitly defines that semantic comparison. Exact pseudocode, -provider text, or provider-specific metadata is not treated as equivalent. A -missing or incompatible profile yields unknown or incompatible, never equal. - -### 9. Expose selection truth in status and diagnostics - -`binary_session`, `rea providers`, and `rea capabilities` expose the same -candidate and binding model. The target-free form does not start providers and -uses `unknown` for target support: - -```json -{ - "analysis_provider_binding": null, - "analysis_provider_candidates": [ - { - "provider": { - "id": "hopper", - "name": "Hopper Disassembler", - "version": "" - }, - "availability": { - "status": "available", - "code": null, - "reason": null, - "diagnostics": {} - }, - "target_support": { - "status": "unknown", - "code": null, - "reason": "No target is open." - }, - "selected": false, - "capabilities": [] - } - ] -} -``` - -With a binding, `analysis_provider_binding` includes the selected identity, -selection source, and complete profile commitment. Every candidate reports -`supported`, `unsupported`, or `unknown` target support and whether it was -selected. Candidate capability descriptors preserve supported, unsupported, -and degraded truth independently. - -An open target also exposes `analysis_run`. Its run ID is allocated by the -session before client construction and propagated into selected Hopper or -Ghidra bridge handshake and process environment. Dynamic clients retain -provider-attributed lineage snapshots only after revalidating launcher -parent/command and every observed launcher or descendant run token. `not_observed` -means no dynamic provider has supplied a snapshot. Once one or more start, -`snapshots` preserves every provider observation; each remains explicitly -`unavailable` or `verified` and records its `observed_at` time. Retained -snapshots are not live inventories, and a point-in-time observation never proves -historical absence. - -Status exposes `analysis_provider_candidates` for configured deep providers and -`analysis_provider_binding` for the selected concrete provider and profile. -`capabilities` retains effective operation availability and exact provider -provenance, including auxiliary operation families. There are no synthetic -flat `provider` or `providers` aliases. Provider prompt completion uses the -actual selectable candidates. - -Selection failures extend the tagged `ProviderSelectionError` with a stable -reason: - -- `unknown_provider`; -- `provider_unavailable`; -- `target_unsupported`; -- `ambiguous`; -- `invalid_options`. - -The error includes the requested ID or `auto`, candidate IDs, rejection codes, -and actionable local diagnostics. `AnalysisCapabilityUnavailableError` remains -the error for an operation the selected provider cannot supply. -`ProviderAdapterError` and more specific provider errors remain failures of the -already selected provider. None authorizes fallback. - -A snapshot whose provider or profile differs from the selected binding returns -`EvidenceIntegrityError` with stable `profile_mismatch` details. It is an -integrity incompatibility after selection, not permission to select the -snapshot's provider. - -### 10. Preserve CLI and MCP parity - -- `open_binary.provider_id` and one-shot `--provider` use the same parser and - selection service. -- `provider_id: "auto"`, `--provider auto`, and - `REA_ANALYSIS_PROVIDER=auto` have identical auto semantics. -- Startup targets opened from environment configuration use the same service. -- `rea providers`, `rea capabilities`, and target-free `binary_session` use the - same candidate snapshot and reason codes. -- Guided prompt `provider_id` context is a preference to forward into the open - operation; it does not create a hidden routing channel. -- Tool names and normalized output contracts remain provider-neutral. REA does - not add `ghidra_decompile` or `hopper_decompile` tools. - -## Provider and snapshot representations - -| Behavior or data | Contract | -| -------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------- | -| No selector, one usable deep provider | `auto` selects the sole candidate | -| No usable deep provider | Automatic open may remain unbound for auxiliary operations; deep capabilities explain their unavailability | -| Several usable deep providers without a preference | Open fails with `ambiguous`; installation order never becomes policy | -| Hopper environment settings | Adapter configuration committed into the typed analysis profile, including effective loader arguments | -| `open_binary` output | Provider-neutral target identity only; no `loaderArgs` projection | -| `.hop` input | Analysis database accepted only by Hopper | -| Snapshots | Concrete provider/profile binding, required primitive and workflow entry arrays, and matching authenticated Evidence | -| Evidence without an observed profile | Explicit `analysis_profile: null`; ineligible for profile-dependent replay | -| Snapshot save receipt | `primitive_entries`, `workflow_entries`, and `evidence_records` counts without a duplicate `entries` alias | -| Provider failure after selection | Returned from that provider, without transparent retry through another engine | - -## Required behavior scenarios - -| Scenario | Result | -| ---------------------------------------------------------- | --------------------------------------------------------------------------- | -| Hopper is the only usable candidate, selector omitted | Hopper is bound by `auto-single-candidate` | -| Ghidra is the only usable candidate, selector omitted | Ghidra is bound by `auto-single-candidate` | -| Both are usable, selector omitted | Typed `ambiguous` error; neither provider starts | -| Environment selects Ghidra, request omits a selector | Ghidra is required and bound or the open fails with its rejection | -| Environment selects Ghidra, request selects Hopper | Request wins; Hopper is required | -| Environment selects Ghidra, request selects `auto` | Auto policy runs and is ambiguous when both are usable | -| Unknown concrete provider ID | Typed `unknown_provider`; no target transition | -| Selected provider rejects the target or options | Typed rejection from that candidate only | -| Selected provider times out after binding | Binding remains selected and unavailable; no fallback | -| Same target is reopened without a selector | Existing binding remains unchanged | -| Same bytes are opened with a different profile | Real provider switch and a separate snapshot partition | -| Snapshot provider or profile differs from the binding | Typed `EvidenceIntegrityError` with `profile_mismatch`; nothing is imported | -| Artifact-only target has no deep candidate | Target opens unbound; disjoint artifact capabilities remain usable | -| Environment prefers Ghidra for an artifact-only target | Deep selection is `not_applicable`; auxiliary operations remain usable | -| Concrete deep provider is requested for artifact-only data | Typed `target_unsupported`; the request is not silently ignored | - -## Consequences - -Positive consequences: - -- Hopper and Ghidra can coexist without ambiguous operation routes. -- Provider choice is reproducible, visible, and stable for a target. -- Installation of another provider cannot silently change analysis. -- Snapshot and Evidence identities become analysis-configuration aware. -- Provider adapters retain ownership of their option schemas and diagnostics. -- Auxiliary capabilities remain usable when no deep engine is installed. - -Costs and tradeoffs: - -- Multi-provider hosts must choose when more than one candidate is usable. -- Snapshots must be recaptured when their provider or profile binding changes. -- Providers must expose deterministic availability, support, version, and - profile resolution before they can be admitted. -- Status and error contracts gain additive structured metadata. -- Hopper needs an adapter-owned version/profile implementation before profile- - exact snapshot persistence is complete. - -## Rejected alternatives - -### Let `CompositeProvider` choose the first duplicate route - -Array order is not a product policy, becomes fragile during wiring changes, and -hides ambiguity from callers. - -### Race providers and keep the first healthy response - -Machine load would determine provenance and results. It also starts resources -the operator did not select and makes cancellation and cleanup harder. - -### Fall back after a provider error - -Mid-session fallback can combine incompatible engine state and evidence under -one apparent target. Explicit close or switch is required instead. - -### Add provider-prefixed tools - -Parallel Hopper and Ghidra tool families would leak adapters into the product -contract and prevent shared workflows from using capability metadata. - -### Store raw provider options in `BinaryTarget` - -That repeats the current Hopper coupling and makes every generic consumer learn -provider schemas. Only canonical profile commitments cross the adapter boundary. - -### Treat matching target bytes and provider ID as sufficient for snapshots - -Provider versions, analyzer defaults, loaders, languages, and compiler models -can change results without changing either value. - -### Silently upgrade snapshot v1 - -The missing engine version and analysis settings cannot be reconstructed from -Hopper loader arguments. A fabricated profile would turn unknown provenance -into false certainty. - -## Implementation progress - -The first six implementation stages are shipped: - -1. Provider-specific target state was replaced by generic profile and snapshot - v2 commitments while preserving Hopper execution behavior. -2. The registry, selectors, candidate status, binding lifecycle, and - overlapping-provider production seams were added. -3. Proven process-lifecycle primitives were extracted for shared use. -4. Ghidra availability, target/profile resolution, doctor checks, and a private - read-only headless session were admitted without declaring operations. -5. Program identity, procedures, strings, symbols, memory blocks, - address/name resolution, containing-procedure resolution, and search - were admitted with exact wire schemas and real debug/stripped ELF - conformance. -6. Function metadata, persistent decompilation, assembly, resolved - callers/callees, typed references, xrefs, CFG, and complete function dossiers - were admitted. A serial per-Program queue keeps API access ordered without a - fixed queue length or per-operation deadline. Real conformance covers - x86-64 and AArch64 ELF, PE, Mach-O, stripped symbols, targetless indirect - calls, cancellation, and cleanup. - -Future stages may deepen format and indirect-flow coverage, but must continue -to compare normalized semantics rather than provider-specific pseudocode or -assembly text. Linux and macOS now admit atomic function-name and entry-comment -edits in the ephemeral database, with readback, rollback, and cache invalidation. -Executable bytes remain unchanged. Windows P0 retains its read-only boundary; -GUI and other unimplemented mutations remain unavailable. diff --git a/docs/adr/0002-controlled-replay-authority-and-sandbox.md b/docs/adr/0002-controlled-replay-authority-and-sandbox.md deleted file mode 100644 index 6f5ae25de..000000000 --- a/docs/adr/0002-controlled-replay-authority-and-sandbox.md +++ /dev/null @@ -1,551 +0,0 @@ -# ADR-0002: Controlled JavaScript replay authority and sandbox policy - -- Status: Superseded; the controlled JavaScript replay tool was removed -- Date: 2026-07-16 -- Implementation status: The implementation was removed. The design below is - retained as historical context and does not describe a supported REA tool. - -> Historical design note: The tool, sandbox backend, and plan/execute workflow -> described here are no longer part of REA. For runtime behavior, run probes -> directly against the application and use the available browser, Electron, or -> process capture workflows. - -## Context - -REA can reconstruct JavaScript and Electron artifacts, observe an existing -browser or Electron page through passive CDP, capture an explicitly approved -process scenario, and correlate those observations without executing bundle -code in the REA server. The next research workflow needs to execute selected, -extracted JavaScript modules with controlled inputs and deterministic stubs so -that parsers, sanitizers, serializers, and similar pure logic can be compared -across versions. - -That workflow crosses a qualitatively different boundary. Source recovered -from an application is untrusted code. It can intentionally or accidentally -read files, contact services, spawn processes, exhaust resources, corrupt the -analysis process, forge protocol output, or interact with another same-user -process. A JavaScript realm or Node.js `vm` context is useful for constructing -an API surface, but it is not a security boundary. Node.js also documents its -permission model as protection against accidental access by trusted code, not -as containment for malicious code. - -Existing authorities cannot be widened to cover this behavior: - -- `browser_observe` and `electron_observe` authorize passive attachment to an - already-running, operator-owned target. They do not authorize evaluation, - navigation, input, or target lifecycle changes. -- `process_capture` authorizes one declared host process scenario and explicitly - states that it is not a sandbox. It cannot silently become permission to run - recovered code. -- static artifact reads authorize inspection of bytes, not execution of those - bytes. -- the Evidence authority `controlled-replay` already describes observations - from an approved process experiment. Reusing that epistemic authority is - correct, but reusing the process permission capability is not. - -The implementation following this ADR must fail closed on hosts where REA -cannot establish the promised operating-system boundary. It must not copy the -research `vm.runInContext` scripts into the production server and call that -isolation. - -## Decision drivers - -1. Passive observation must never imply authority to execute application code. -2. The exact code, inputs, stubs, runtime, sandbox policy, and limits approved - by the operator must be the ones executed. -3. Recovered code must not run in the MCP server, CLI adapter, or any provider - process. -4. A sandbox failure must make the capability unavailable, not select a weaker - execution mode. -5. The default policy must provide no host network and no writable host path. -6. Time, memory, process count, temporary storage, protocol, and output must be - bounded independently. -7. Results must distinguish an observed return, an observed exception or - termination, a derived comparison, and an unresolved application-level - claim. -8. A reproducer must commit enough provenance to detect every relevant change - without committing proprietary application source to the repository. -9. CLI and MCP must expose the same plan, approval, execution, cancellation, - failure, and Evidence semantics. - -## Terminology - -- **Passive observation**: bounded reads from an existing browser or Electron - target without evaluating code, driving input, navigating, or controlling - its lifecycle. -- **Controlled replay**: execution of operator-selected code in a declared, - isolated environment with committed inputs and limits. This is an Evidence - authority, not a claim that the real application behaved identically. -- **Extracted-module replay**: the only admitted JavaScript replay mode in v1. - REA evaluates a finite manifest of module bytes and explicit stubs without - launching the application. -- **Application interaction**: launching or driving the original application, - browser, renderer, or Electron main process. This is not extracted-module - replay and is not authorized by this ADR. -- **Replay plan**: the canonical, content-addressed description of one proposed - run before execution. -- **Replay worker**: the disposable Node.js process inside the OS sandbox. It - is never the REA server process. -- **Sandbox backend**: the OS mechanisms and adapter that establish namespaces, - resource controls, filesystem visibility, and owned cleanup. - -## Threat model - -### Protected assets - -The boundary protects: - -- the REA MCP/CLI process, its memory, credentials, sockets, and Evidence - ledger; -- user files outside the approved source reads and all host files from writes; -- other same-user processes and their control interfaces; -- host network interfaces, loopback services, Unix sockets, and external - services; -- process, memory, CPU, temporary-storage, output, and protocol capacity; -- the integrity of the replay plan, result, Evidence provenance, and exported - reproducer; and -- proprietary module bytes, which remain local and are never repository - fixtures. - -### Adversaries and failure sources - -REA treats all of the following as untrusted: - -- recovered module source, including transitive modules and source maps; -- caller-provided JavaScript stubs; -- structured test inputs and generated fuzz cases; -- returned values, exceptions, console data, and worker protocol messages; -- source paths, symbolic links, and files that can change between planning and - execution; and -- a worker that crashes, hangs, forks, allocates aggressively, or exits while - descendants remain alive. - -An operator-controlled sandbox executable, kernel, Node.js runtime, and REA -installation are trusted computing base. A hostile host administrator, kernel -or sandbox vulnerability, hardware side channel, and denial of service by an -already-compromised same-user process are out of scope. These exclusions do not -permit REA to omit the per-run resource controls in this decision. - -### Required mitigations - -The implementation must address: - -| Threat | Required control | -| -------------------------------- | --------------------------------------------------------------------------------------------------------------- | -| Server-process compromise | Separate worker process inside the OS sandbox; no in-process fallback | -| Host file reads | Empty sandbox root; only a minimal read-only runtime closure; module bytes enter over a private bounded channel | -| Host file writes | No writable bind mounts; size-limited private tmpfs only | -| Network access | Separate network namespace with no host or external connectivity; no allowlist mode in v1 | -| Host process control | PID/user/IPC/UTS isolation, no inherited control sockets, new session, dropped capabilities | -| Kernel escape/inspection surface | Architecture-checked seccomp denial policy in addition to namespaces, capabilities, and cgroups | -| Fork, memory, or CPU exhaustion | Per-run cgroup v2 task, memory, swap, and CPU limits plus an outer wall-clock deadline | -| Output/protocol exhaustion | Independent byte/message/depth limits enforced by the parent before parsing or retaining output | -| Symlink and file-change races | Canonical authorization, descriptor-backed reads, digesting, staging, and exact plan revalidation | -| Undeclared dependencies | Closed module manifest; exact imports only; missing/dynamic imports fail rather than use host resolution | -| Nondeterministic globals | Versioned clock, PRNG, locale, timezone, platform, and API stubs; undeclared globals are unavailable | -| Orphan descendants | Owned PID namespace and cgroup kill followed by an observable empty-cgroup cleanup check | -| Forged result claims | Framed worker protocol, strict schemas, parent-computed commitments, and bounded stderr kept separate | -| Secret leakage | Empty environment, no credential inheritance, bounded local results, and existing genuine-secret redaction | - -## Decision - -### 1. Add a distinct `javascript_replay` permission capability - -JavaScript replay receives a new permission capability named -`javascript_replay`. It does not inherit, imply, or satisfy -`process_capture`, `browser_observe`, `electron_observe`, artifact-read, or -workspace permissions. Grants for any of those capabilities do not satisfy a -JavaScript replay request, and a JavaScript replay grant does not authorize -them. - -The administrator ceiling is disabled by default. Its eventual configuration -must commit: - -- canonical roots from which module and optional stub files may be read; -- the exact Node.js and sandbox-backend executables; -- filesystem mount authority needed only to construct the private sandbox; -- network mode `none`; and -- no inherited environment names. - -Configuration establishes only the maximum authority. Every execution also -requires an active permission grant and literal per-call approval bound to the -exact replay plan. Project, session, and administrator grants may satisfy the -permission-policy layer, but none replace that per-run approval. - -Reproducer export is a separate host write after the sandbox has stopped. This -ADR's root-scoped `evidence_write` authority was removed; current exports use -the path supplied by the caller, while replay still waits for complete sandbox -cleanup before writing. - -### 2. Admit extracted modules only - -The v1 mode executes a finite, closed manifest of JavaScript modules. The -manifest identifies one entry module and every admitted dependency by a stable -sandbox alias and SHA-256 digest. Imports resolve only to another manifest -entry or to an explicitly declared, versioned stub. Bare package lookup, -ambient `node_modules`, import maps from the host, network imports, native -addons, WASI, FFI, child processes, workers, inspector activation, and implicit -dynamic import are unavailable. - -Modules and stubs are read and hashed by the authorized parent, then copied as -bytes through a bounded private channel. Original host paths are not mounted -into the sandbox. The worker cannot reopen the operator's source path. - -Source maps may supply locations, but source-map source code is not executed -unless its bytes are separately present in the approved manifest. A source-map -name or Webpack module ordinal is never execution authority. - -REA does not launch, attach to, navigate, click, type into, or terminate the -user's application under this capability. A full application process remains -an independently approved Process Capture experiment; browser and Electron -targets remain passive. Any future mode that drives an application or permits -host/allowlisted network requires a new ADR and a distinct permission scope. - -### 3. Use a two-phase, content-bound approval flow - -One public replay surface supports a plan phase and an execute phase with the -same schema through CLI and MCP. - -The plan phase: - -1. validates all caller input and hard limits before file access; -2. obtains authority to read every selected module or stub path; -3. canonicalizes and opens each file without following an unapproved identity; -4. reads bounded bytes, computes digests, and closes the source descriptors; -5. resolves and probes the exact runtime and sandbox backend without running - application code; -6. normalizes stubs, inputs, generator settings, and limits; and -7. returns the complete proposed effects and a canonical `plan_digest`. - -The plan must disclose at least: - -- module aliases, canonical local paths, byte counts, and SHA-256 digests; -- input and stub identities, provenance, and digests; -- runtime and sandbox executable paths, versions, and executable digests; -- sandbox policy version and digest; -- network `none`, visible read-only runtime paths, and private writable tmpfs - size; -- time, CPU, memory, swap, task, temporary-storage, input, output, protocol, - case-count, and serialization limits; -- deterministic clock, PRNG seed/algorithm, locale, timezone, and platform - projection; and -- expected Evidence and cleanup behavior. - -Execution requires the exact `plan_digest`. REA -rebuilds the plan immediately before admission. Any changed file, runtime, -sandbox executable, policy, stub, input, limit, or environment commitment -returns `plan_stale` before application code starts. Approval is not transferable -to a different plan and cannot be represented by a bare Boolean alone. - -Interactive CLI may print the plan and ask for confirmation. Non-interactive -CLI and MCP must perform the same explicit plan/execute exchange; they may not -auto-approve because the caller requested JSON output or because permission -elicitation is supported. - -### 4. Require an OS sandbox; fail closed elsewhere - -The first production backend is Linux-only. It must combine: - -- an empty mount namespace rooted in private tmpfs; -- a new user namespace with all capabilities dropped and further nested user - namespaces disabled; -- new PID, network, IPC, UTS, and cgroup namespaces; -- a new terminal session and parent-death behavior; -- no host D-Bus, agent, browser, Docker, Wayland/X11, SSH, GPG, audio, device, - or application sockets; -- a minimal, read-only runtime closure rather than `/`, the user's home, - working directory, or host `/tmp`; -- a fresh `/proc` for the sandbox PID namespace and a minimal private `/dev`; -- a parent-generated, architecture-checked seccomp policy that denies at least - namespace or mount changes, `ptrace` and cross-process memory inspection, - BPF/perf access, kernel keyring and module control, reboot/swap controls, and - the `TIOCSTI` terminal injection path; -- cgroup v2 `memory.max`, `memory.swap.max=0`, `pids.max`, and bounded CPU - controls owned for the one run; and -- an outer REA supervisor that can kill the complete cgroup and verify it is - empty. - -The seccomp program is passed by descriptor, has a version and digest committed -to the replay plan, validates the syscall architecture before the syscall -number, and never permits `ptrace`-based mediation. Seccomp reduces exposed -kernel surface; it is not described as a complete sandbox by itself. - -The reference backend for the implementation PR is Bubblewrap for namespace, -mount, and seccomp construction plus a delegated cgroup v2 manager for resource -enforcement. REA must feature-probe the actual required operations; version -text alone is insufficient. A setuid Bubblewrap installation, a backend that -silently skips a requested namespace or filter, unavailable user namespaces, -missing cgroup delegation, or inability to prove cleanup makes the capability -unavailable. REA setup must not install or upgrade Bubblewrap, systemd, Node.js, -or kernel components. - -macOS, Windows, Linux hosts without every required feature, containers that -block the required namespaces, and alternative backends are unavailable in -v1. They must return precise diagnostics and remediation, not a process-only or -`vm`-only degraded mode. - -Node.js runs with its permission model enabled and without grants for network, -filesystem, child processes, workers, native addons, WASI, FFI, or inspector. -That layer is defense in depth. It is never cited as the OS security boundary, -because Node.js explicitly does not promise containment of malicious code. - -### 5. Make the worker disposable and deterministic by construction - -Each approved run receives a new sandbox, runtime root, worker process, module -loader, and deterministic state. No module cache, realm, timer, file, global, -or child survives into another run. Within a multi-case run, every case gets a -fresh realm and fresh module instances; case order and generator state are -committed. - -The worker exposes a small versioned set of language primitives and explicitly -selected stubs. It does not expose `process`, `require`, host `console`, timers, -`fetch`, sockets, filesystem APIs, `Buffer`, worker APIs, native bindings, or -ambient environment variables. Date/time, randomness, locale, timezone, and -platform values come only from committed deterministic providers. An -unsupported global or import produces an observed exception; REA does not -substitute a convenient host implementation. - -Caller-provided stubs are executable untrusted code and receive the same -sandbox treatment as recovered modules. Built-in stubs have stable IDs, -versions, canonical source digests, and documented semantics. No undeclared -fallback stub is allowed. - -Inputs are bounded canonical JSON. Results are projected without invoking -getters, proxies, `toJSON`, or arbitrary coercion: only admitted primitives, -arrays, and plain data properties are serialized. Cycles, accessors, functions, -symbols, unsupported prototypes, excessive depth, excessive nodes, and -oversized values become explicit serialization outcomes. Worker stdout is the -framed protocol only; bounded diagnostic stderr and bounded, opt-in console -observations remain separate. - -### 6. Enforce independent resource ceilings - -Every limit has a conservative product default. The v1 plan and result commit: - -- an outer wall-clock deadline for the complete disposable run; -- cgroup memory, swap, task, and CPU ceilings; -- Node/V8 heap ceiling as defense in depth; -- private tmpfs bytes; -- serialized worker-protocol bytes, aggregate module bytes, and case-input - bytes; preset-generated cases also have a fixed preflight count ceiling; -- stdout, stderr, exception, stack, and aggregate retained-output bytes; and -- structured-result depth and node count. - -The parent enforces protocol and output limits while bytes arrive, before JSON -parse or Evidence retention. The worker cannot request wider limits. A limit -hit terminates the complete run cgroup and is an observed termination, not a -partial successful return. - -### 7. Preserve exact provenance and the right Evidence authority - -A run that reaches worker admission produces Evidence with: - -- provider `rea-javascript-replay` and a concrete implementation version; -- operation `run_controlled_replay`; -- authority `controlled-replay`; -- confidence `observed` for the exact returned value, thrown exception, - sandbox denial, limit termination, signal, exit, or crash that REA directly - observed; -- environment isolation `container`, the host platform/architecture, runtime - identity, sandbox-backend identity, and sandbox-policy digest; -- the replay plan digest, module and stub digests, canonical input or generated - case digests, deterministic providers, and every effective limit; -- bounded raw input, output, and exception data when retained, plus their - canonical digests and truncation state; -- start/admission/termination/cleanup states; and -- explicit limitations for incomplete output, nondeterminism not checked, - sandbox diagnostics, or cleanup that could not be proven. - -The host module path remains an actionable local diagnostic and Evidence -subject location, while the worker sees only a stable alias. Module source is -not embedded in ordinary Evidence. A separately approved reproducer may copy -module/stub bytes or may retain path-plus-digest references; its manifest says -which mode was used. Repository tests use source-owned fixtures, and real-app -verification stores only digests and expected structural facts. - -Derived differential results use confidence `derived` and link every source -run Evidence ID. A repeat with the same commitment can establish -`repeat-confirmed` for those runs; a single deterministic plan is only -`committed-not-confirmed`. Neither state proves behavior in the original -application. - -When replay observations are added to the JavaScript Application Graph, their -graph-fact authority is `controlled-replay`. A claim that the real renderer, -preload, main process, or remote service behaves the same remains inferred or -unknown until compatible runtime authority exists. Static inference, passive -CDP observation, controlled replay, and real application interaction are never -promoted into one another. - -### 8. Treat post-admission failures as observations - -Pre-admission failures return a typed error and create no claim that module code -ran. Once the worker is admitted, REA retains a bounded execution record even -when the requested function does not return normally. - -| Condition | Caller result and Evidence semantics | -| ------------------------------------------------ | ---------------------------------------------------------------------------------------------------------- | -| Disabled capability or missing grant | `permission_required`/`capability_unavailable`; no worker admission and no replay observation | -| Sandbox/runtime probe failure | `capability_unavailable` with exact failed feature; no module execution | -| Plan digest mismatch or changed source | `plan_stale`; no module execution | -| Sandbox setup fails before worker admission | `execution_failure`; no controlled-replay claim about the module | -| Module returns admitted data | Observed return value with complete or explicitly limited coverage | -| Module throws or result serialization rejects | Successful replay observation with outcome `exception` or `serialization_error`; application claim unknown | -| Node permission or sandbox denies an operation | Observed denial/termination when attributable; no claim that all attempted side effects were enumerated | -| Timeout, memory, CPU, task, tmpfs, or output hit | Observed limit termination; requested functional result unknown | -| Signal, runtime crash, or malformed protocol | Observed termination plus bounded diagnostics; functional result unknown | -| Caller cancellation after admission | Kill full cgroup, record cancellation/cleanup if possible, and project the public cancellation contract | -| Cleanup proves an empty cgroup and removed root | `cleanup: complete` | -| Cleanup cannot prove emptiness or removal | `cleanup_incomplete`, name every residual resource, and do not report success | - -An exception is data, not an adapter failure. A sandbox setup error is not -evidence that the module crashed. A timeout is not evidence that the function -would never return. An output limit is not evidence that the truncated prefix -equals the full result. These distinctions are part of the public contract. - -Cancellation and cleanup must be idempotent. REA first stops admission and -input, kills the owned cgroup, waits for an empty-cgroup observation, removes -the private runtime root, closes protocol descriptors, and releases timers. -Failure at one cleanup step does not skip the remaining steps. - -### 9. Keep replay records reproducible without overstating determinism - -The canonical replay manifest commits: - -```text -plan structure and policy identity -REA/provider/runtime/sandbox executable identities and digests -module aliases, dependency graph, bytes digests, and source references -stub IDs, versions, source digests, and caller-provided stub digests -canonical inputs or generator algorithm, seed, case order, and case digests -clock, PRNG, timezone, locale, platform, and exposed-global commitments -all resource and serialization limits -network and filesystem projections -expected entry export and invocation convention -``` - -The result manifest adds canonical output/exception/termination digests, -coverage, dropped-data counts, cleanup state, and source Evidence links. -Canonical digests are computed by the trusted parent; worker-provided digests -are ignored. - -Reproduction requires exact manifest compatibility. A runtime, sandbox policy, -built-in stub, module byte, generated case, limit, architecture, or platform -change is a different experiment. Cross-environment comparison may still be a -derived investigation, but it is never an exact replay. - -### 10. Expose availability and policy truth before execution - -`rea doctor`, capability inventory, and the planning response must report the -JavaScript replay capability without executing application code. Diagnostics -name the exact missing executable, unsupported host, failed namespace feature, -cgroup delegation failure, setuid rejection, runtime mismatch, or cleanup-probe -failure. Genuine secrets remain redacted; local executable paths, versions, -digests, and kernel feature failures remain visible. - -The capability must be absent from executable tool routing until implemented. -Accepting this ADR does not increase the current tool count and does not add a -hidden experimental flag. - -## Required implementation and verification gate - -The implementation PR cannot merge until all of the following are true: - -1. `javascript_replay` exists in configuration, permission policy, project - grants, policy explanation, CLI, MCP, docs, and capability inventory with - fail-closed defaults. -2. Plan and execute phases have schema-identical CLI/MCP behavior and an exact - plan-digest stale check. -3. No replay-disabled static, browser, Electron, session, setup, doctor, or - application workflow can reach module execution. -4. The worker never imports into the REA server process and no adapter offers a - `vm`-only fallback. -5. The Linux backend feature-probes namespaces, seccomp installation and - denial, delegated cgroup v2 limits, and the exact admitted architecture. - Every plan content-addresses the worker, seccomp filter, Node executable, - ELF loader, shared-library closure, and sandbox executables. -6. Source-owned fixtures cover parser, sanitizer, clipboard, exception, - ambient side-effect attempts, memory pressure, and busy loops. The worker - exposes no filesystem, environment, network, process, worker, native-addon, - timer, inspector, or ambient package-resolution API. -7. Tests and the real verifier prove bounded protocol/diagnostics, timeout and - OOM cgroup termination, and observed cgroup cleanup. -8. Evidence tests cover observed return/exception/termination, derived - comparison with source Evidence links, stale admission, cancellation - projection, and explicit limitations on application-level claims. -9. Package verification exercises the target-free public contract. The - dedicated real-Linux workflow exercises the sandbox backend; environments - without delegated user namespaces or cgroups report truthful unavailability - rather than selecting a weaker backend. -10. A real operator-local Notion-like parser or sanitizer benchmark can run - without committing its source, and exports only approved local reproducer - material. - -## Rejected alternatives - -### Reuse `browser_observe`, `electron_observe`, or `process_capture` - -Rejected because those grants authorize different effects. Reuse would turn a -passive or unsandboxed process permission into recovered-code execution. - -### Execute in the MCP server with `vm.runInContext` - -Rejected because a VM context is not a security boundary and a crash, resource -attack, or escape would compromise the long-lived session and its credentials. - -### Use only the Node.js permission model - -Rejected because Node.js documents it as a seat belt for trusted code and -explicitly does not guarantee containment of malicious code. It remains a -defense-in-depth layer inside the OS sandbox. - -### Ship a process-only fallback on unsupported hosts - -Rejected because it changes the security claim while preserving the same tool -name. Unsupported hosts return unavailable. - -### Permit network allowlists in v1 - -Rejected because DNS, redirects, proxies, credentials, service identity, and -response nondeterminism would substantially expand the threat and Evidence -model. The v1 namespace has no host or external network. - -### Mount the approved project root read-only - -Rejected because one approved module does not imply authority to disclose -every file in its project. The parent stages only exact, digested bytes. - -### Treat exceptions, crashes, or limit hits as ordinary tool errors only - -Rejected because that discards the principal behavioral observation and makes -negative results impossible to reproduce. Post-admission termination is -retained with explicit functional uncertainty. - -## Consequences - -- JavaScript replay is Linux-only at first and may be unavailable on containers - or distributions that disable user namespaces or cgroup delegation. -- The implementation requires more host probing and lifecycle code than a - simple Node worker, but the public security claim remains stable. -- Plans require two calls in non-interactive environments and become stale when - any committed input changes. -- Network-dependent modules and full application interactions are excluded; - their absence is explicit rather than emulated implicitly. -- Results are reproducible for the committed isolated experiment, not proof of - production behavior. -- Adding another sandbox backend, network mode, or application-interaction mode - requires a separate reviewed decision and conformance suite. - -## References - -- [Node.js permissions](https://nodejs.org/api/permissions.html) documents that - the permission model does not protect against malicious code and describes - its known filesystem and cross-process limitations. -- [Bubblewrap README](https://github.com/containers/bubblewrap/blob/main/README.md) - describes its namespace construction and makes clear that the caller's - arguments, not Bubblewrap alone, define the sandbox security model. -- [Linux seccomp filter documentation](https://www.kernel.org/doc/html/latest/userspace-api/seccomp_filter.html) - explains that syscall filtering reduces kernel surface but is not a complete - sandbox by itself. -- [ADR-0001](0001-provider-selection-and-analysis-profiles.md) establishes the - existing rule that provider selection, Evidence provenance, and capability - unavailability remain explicit and never silently fall back. diff --git a/docs/adr/0003-managed-code-evidence-and-provider-boundary.md b/docs/adr/0003-managed-code-evidence-and-provider-boundary.md deleted file mode 100644 index bd3895fb4..000000000 --- a/docs/adr/0003-managed-code-evidence-and-provider-boundary.md +++ /dev/null @@ -1,452 +0,0 @@ -# ADR-0003: Managed-code evidence and provider boundary - -> Historical design record: The non-executing runtime-correlation planning -> tool described in earlier revisions was removed. This ADR's current -> implementation status below covers only the static managed tools that ship. - -- Status: Superseded for runtime planning; static evidence boundary remains in force -- Date: 2026-07-16 -- Implementation status: Read-only PE/CLI triage and exact identity are shipped - through `inspect_managed_artifact` / `rea inspect-managed-artifact`. Bounded - metadata, signatures, method bodies, raw CIL hashes, limited - decoded-instruction-tuple v1 hashes, exception regions, call edges, and - field-access anchors are shipped through `inspect_managed_members`; `rea - inspect-managed-members` provides CLI parity. Static member comparison and build-local - token remapping are shipped through `compare_managed_members` / - `rea compare-managed-members`. Declaration-only managed/native boundary - inventory for ModuleRef, ImplMap/PInvoke, ReadyToRun indicators, and non-IL - method flags is shipped through `inspect_managed_native_boundaries` / - `rea inspect-managed-native-boundaries`. Managed/native export or function - Evidence matching is shipped through `verify_managed_native_boundaries` / - `rea verify-managed-native-boundaries`; it verifies only supplied P/Invoke - export/function observations and does not map managed tokens to native - addresses. Decompiler reconstruction import is - shipped through `import_managed_reconstruction` / - `rea import-managed-reconstruction`; it records user-supplied C#/IL/pseudocode - as analyst inference only after exact static member locks match. Managed - inspection does not execute assemblies or observe CLR internals. Managed static graph - projection is shipped through - `project_managed_application_graph` / - `rea project-managed-application-graph`; it projects authenticated managed - Evidence into the application graph without changing managed tokens into - native addresses. Native-body bridge mapping and an actual runtime executor - remain future contracts. - -## Context - -Professional C# reverse engineering is not one decompiler operation. A file -described as “.NET” can be ordinary Common Intermediate Language (CIL), -ReadyToRun code containing both CIL and native implementations, a single-file -deployment, C++/CLI mixed mode, Unity Mono, Unity IL2CPP, or NativeAOT. Those -forms have different sources of truth and different native-analysis needs. -Choosing a decompiler before classifying the artifact can therefore produce a -plausible but false model. - -The durable static facts in an ordinary managed PE are its PE/CLI headers, -metadata tables, signatures, method bodies, exception regions, and CIL. A C# -decompiler reconstructs source-like control flow and language constructs from -those facts. That reconstruction is valuable, but it is not the original -source and cannot replace the underlying metadata and CIL evidence. Obfuscation -makes names especially weak while tokens remain coordinates in only one build. - -REA already separates one selected deep native provider from disjoint auxiliary -providers under [ADR-0001](0001-provider-selection-and-analysis-profiles.md). -It also separates static evidence, passive observation, process capture, and -interactive observation authorities. Managed analysis must fit those boundaries -without requiring a CLR to inspect an artifact, loading target code into the -REA process, or treating a managed token as a native address. - -The design is informed by exact-build, reflection-free metadata extraction, -hash-locked method manifests, independent behavior models, and managed/native -boundary analysis performed against a local C# application. The application, -its decompiler output, and its runtime observations are research inputs only; -they are not distributable REA fixtures. - -## Decision drivers - -1. Classify the deployment before selecting an analysis route. -2. Make metadata and CIL independently inspectable without a CLR or target - execution. -3. Preserve exact-build provenance while enabling structural comparison across - builds whose MVIDs, tokens, names, and layout differ. -4. Compose managed and native analysis without introducing another ambiguous - deep-provider candidate. -5. Keep decompiled C# useful but epistemically subordinate to canonical bytes. -6. Analyze Linux, macOS, and Windows managed artifacts through the same local - CLI/MCP contracts on REA's supported hosts. -7. Keep setup additive and prevent managed analysis from installing .NET, - ILSpy, an SDK, or another unrelated toolchain. -8. Prove semantics with source-built conformance artifacts and compact, - operator-local real-application checks. - -## Terminology - -- **Managed artifact**: a PE, bundle component, or related metadata artifact - that carries a CLI header, CLI metadata, CIL, or another managed-runtime - commitment. -- **Classification vector**: independently observed container, runtime family, - implementation form, and architecture. It is not one guessed label. -- **Canonical static observation**: a bounded value decoded directly from the - target bytes under a versioned parser contract. -- **Reconstruction**: source-like output, including decompiled C#, derived from - canonical observations. -- **Structural inference**: a relationship or cross-version match derived from - signatures, APIs, constants, control/data-flow shape, or other observations. -- **Validation**: an independently checked proposition, for example a - source-built oracle assertion or a behavior-model corpus result. -- **Runtime observation**: a separately authorized fact obtained by attaching, - loading, debugging, instrumenting, or otherwise executing code. - -## Decision - -### 1. Classify before analyzing - -Managed triage returns a classification vector and the observations supporting -each dimension. Absence of a marker produces `unknown` or `not_applicable`, not -an invented runtime family. - -| Dimension | Initial values | -| ------------------- | ------------------------------------------------------------------------------------------------------------ | -| Container | PE file, single-file bundle, bundle component, Unity metadata pair, native image, malformed, unknown | -| Runtime family | .NET Framework, modern .NET, Unity Mono, Unity IL2CPP, NativeAOT, mixed CLR/native, unknown | -| Implementation form | CIL, CIL plus ReadyToRun, native only, C++/CLI mixed, metadata only, unsupported, unknown | -| Architecture | PE machine plus CLI flags; component-specific architecture for bundles; conflicting or unknown when required | - -Target-framework attributes, assembly references, metadata stream versions, -bundle manifests, ReadyToRun directories, and native headers are observations. -They may support a runtime-family inference, but no single string is treated as -authoritative when the other bytes conflict. - -Classification selects these routes: - -| Observed form | Managed route | Native route and limitation | -| --------------------------------- | ------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------- | -| Ordinary Framework or modern CIL | Full metadata, signatures, method bodies, and CIL | Optional only for declared interop boundaries | -| Obfuscated CIL | Same canonical route; names receive no special trust | Optional for declared interop boundaries | -| Unity Mono assembly | Same canonical route with Unity context retained | Optional for engine/native boundaries | -| ReadyToRun | Metadata/CIL where present; implementation availability is per method | Selected Hopper/Ghidra analyzes native sections; CIL absence is reported per method | -| C++/CLI mixed mode | Metadata/CIL for managed members and explicit mixed-mode boundary observations | Selected Hopper/Ghidra analyzes native bodies; REA never converts tokens directly to native addresses | -| Single-file deployment | Inventory the bundle, then analyze each authenticated managed component | Selected Hopper/Ghidra analyzes the host/native components; unsupported compression remains explicit | -| Unity IL2CPP | Pair and authenticate available Unity metadata; do not claim canonical CIL | Selected Hopper/Ghidra analyzes the generated native binary; semantic recovery is degraded until supported | -| NativeAOT | Report any independently present managed metadata, otherwise no CIL claim | Treat as a native target with NativeAOT provenance; managed source reconstruction is unavailable | -| Malformed or unsupported artifact | Return bounded partial observations and typed unavailable/unknown scopes | Do not silently retry by loading or executing the file | - -Classification is byte-driven and bounded. File extensions, product names, -process names, and an operator's description are hints only. - -### 2. Implement managed static analysis as a disjoint auxiliary provider - -The initial producer is a REA-owned auxiliary provider with ID -`rea-dotnet-static`. It owns a disjoint managed-artifact operation family. It is -not registered as a Hopper/Ghidra deep-provider candidate because it does not -compete for the same operation names and must be able to coexist with the one -selected native provider. - -```text - one opened artifact/session - | - SessionProviderRouter - / \ - rea-dotnet-static auxiliary selected deep binding - PE/CLI metadata and CIL Hopper or Ghidra - \ / - explicit managed/native links - | - Evidence and workflows -``` - -An ordinary managed artifact can therefore remain unbound to a deep native -provider while managed operations work. ReadyToRun, C++/CLI, IL2CPP, NativeAOT, -and interop workflows may additionally require a caller-selected native -binding. Native-provider failure never causes managed evidence to be relabeled, -and managed-parser failure never causes an implicit native fallback. - -Provider mechanics live under a provider directory such as `src/dotnet/`. -Provider-neutral identities, evidence states, and bounded values may live in -the domain/contracts layers. Application code composes results but does not -parse PE headers, metadata tables, signatures, CIL, ReadyToRun headers, or -provider-specific output. - -### 3. Make the production static path byte-only and execution-free - -`rea-dotnet-static` reads approved local bytes and implements the required -PE/CLI metadata and CIL subset in TypeScript. It does not call any reflection -API and does not require the .NET runtime. In particular, static inspection -must never: - -- call `Assembly.Load`, `Assembly.LoadFrom`, `ReflectionOnlyLoad`, or an - equivalent runtime loader; -- resolve target dependencies by executing target-controlled resolver code; -- run module initializers, constructors, entry points, build tasks, plugins, or - generated helpers; -- start the target, attach to a process, contact a service, or inherit target - configuration and credentials; or -- interpret a decompiler, CLR, or native-provider diagnostic as an observation - from the artifact bytes. - -The parser uses checked offsets and additions, explicit endianness, bounded -table/heap/body counts, cycle-safe traversal, and typed partial results. A bad -row, signature, body, resource, or stream does not authorize an unbounded retry -or a CLR load. Complete-coverage claims require every applicable bounded region -to have been admitted; otherwise coverage is partial, unknown, or unavailable -with exact offsets and reasons. - -The caller selects the local input path directly. Local artifact -paths, digests, MVIDs, tokens, malformed offsets, and analysis metadata are -actionable diagnostics and are not secrets. Credentials, authorization data, -and genuine secrets remain redacted. - -### 4. Bind every finding to exact artifact and producer identity - -Every managed result carries a commitment with at least: - -```text -artifact: canonical path + byte length + SHA-256 -component: container identity + component path/name + component SHA-256, when applicable -classification: container + runtime family + implementation form + architecture -assembly/module: simple name + version + culture + public-key identity + module name + MVID -producer: provider ID + exact version + analysis-affecting parameters + profile digest -``` - -An assembly-qualified name does not replace the artifact digest or MVID. -Neither an MVID nor an Authenticode signature proves who authored the behavior. -Bundle components retain both their container and component commitments so that -bytes extracted from another host cannot be silently substituted. - -Method observations add: - -```text -declaring type + normalized CLI signature -build-local metadata token -RVA/file extent when present -raw CIL byte length + SHA-256 when the body is admitted -decoded-instruction-tuple SHA-256 when CIL is valid -separate max-stack, locals-token, and exception-region observations -``` - -The shipped `il_sha256` commits only the raw CIL byte stream identified by the -method header's `il_size`. It does not commit the method header, local-signature -blob, alignment padding, or extra sections such as exception clauses. Those -facts remain separate observations and the enclosing artifact SHA-256 binds -them to the exact file. A digest over the complete method body is not shipped. - -The caller-visible field remains named `normalized_il_sha256`, but its shipped -v1 meaning is deliberately narrower than a canonical semantic CIL identity. -For a completely decoded method, REA serializes the instruction-order array of -`[opcode, operand_kind, operand]` tuples with JavaScript `JSON.stringify`, -encodes that text as UTF-8, and reports its lowercase SHA-256. The tuple -projection is defined here; the member result carries only the described -normalized digest. - -Operands retain the current decoder projection: metadata and user-string -operands are build-local lowercase hexadecimal tokens, scalar branch operands -are decimal target CIL byte offsets, and `switch` retains only its decimal case -count. Constants and variable indexes are decimal strings; an operand-free -instruction uses kind `none` and value `null`. Opcode names preserve distinct -short forms and prefix instructions, but instruction offsets, branch target -instruction identity, switch targets, resolved token identities, method -headers, locals, and exception regions are not part of this digest. A partial -or malformed decode has no `normalized_il_sha256`. - -Consequently v1 is a deterministic decoded-tuple fingerprint, not proof of -semantic equivalence. In particular, methods with the same opcodes and switch -case counts can share it despite different switch targets, and token movement -can change it without a semantic change. Cross-build comparison may use it as -one unique-only structural tier while retaining artifact/MVID commitments and -ambiguity; it must not describe the digest itself as token-remapped canonical -CIL. Canonical operand resolution, complete control-flow operands, -exception/local commitments, and an explicit algorithm version require a new -normalized-CIL schema. - -Metadata tokens are reported because they are precise coordinates inside one -module. Cross-version identity never consists of a token alone. Matching tries -exact CIL/signature identity, then an exact declared-type/name/signature tuple, -then structural shape; a name by itself never selects a pair. Partial or -malformed bodies leave body facets unknown after a signature tuple pairs the -methods. Matches cite both build commitments, both local tokens, and all -competing candidates. V1 cannot cite a separate matching-algorithm version -because none is exposed. - -### 5. Separate five evidence layers - -Managed workflows retain these layers independently: - -| Layer | Examples | Permitted claim | -| ---------------------- | --------------------------------------------------------------------- | --------------------------------------------------------------- | -| Static observation | Metadata row, signature, CIL opcode, literal, exception region | “These bounded bytes decode to this value.” | -| Reconstruction | Decompiled C#, async/iterator source shape, expression simplification | “This is one source-like representation of cited observations.” | -| Structural inference | Likely caller, field flow, behavior slice, cross-build method match | “These cited structures support this relationship.” | -| Independent validation | Synthetic oracle, corpus result, second parser agreement | “This declared proposition passed the named independent check.” | -| Runtime observation | Exact-build attach/load/instrumentation event | “This separately authorized experiment observed this event.” | - -Each layer has its own producer, authority, state, confidence, coverage, and -limitations. A readable C# method is never emitted as original source or exact -semantic proof. A validator does not upgrade unrelated claims. Runtime -agreement does not retroactively turn a static inference into an observation. - -Negative results are scope-qualified. For example, “no declared P/Invoke rows -were found in the admitted metadata tables” does not prove the application has -no native, dynamically resolved, generated, server-side, or protected path. - -### 6. Admit ILSpy only as an optional reconstruction and conformance tool - -REA's canonical static operations do not depend on ILSpy, a .NET SDK/runtime, -dnlib, or Mono.Cecil. The following review was current on the decision date: - -| Tool | Reviewed coordinate | License | Runtime/host observation | Decision | -| ---------------------------- | ------------------- | ------- | --------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------ | -| `System.Reflection.Metadata` | 10.0.10 | MIT | Library targets .NET Framework 4.6.2, .NET 8/9/10, and .NET Standard 2.0 | Pinned independent metadata oracle for conformance; not shipped or invoked by production parsing | -| `ICSharpCode.Decompiler` | 10.1.1.8388 | MIT | Library targets .NET Standard 2.0; output is reconstruction | Pinned reconstruction/differential oracle candidate | -| `ilspycmd` | 9.1.0.7988 | MIT | Cross-platform .NET 8 tool; newer 10.1.1.8388 requires .NET 10 | Initial optional BYO reconstruction coordinate because it matches the source research baseline | -| dnlib | 4.5.0 | MIT | Cross-platform library; also exposes assembly-writing APIs | Differential test candidate only; its writable model is outside the production trust boundary | -| Mono.Cecil | 0.11.6 | MIT | Cross-platform .NET Framework/.NET Standard library with read/write support | Differential test candidate only; not a production dependency | - -Coordinates and package hashes used by real verification are checked into a -source-owned lock manifest when that verifier is implemented. “Latest” is -never a reproducibility coordinate. A version change requires conformance and -an explicit profile/version change. - -Artifact platform and REA host support are separate. The byte-only parser must -inspect PE/CLI artifacts produced for Windows on a supported Linux or macOS REA -host; that is not a claim that REA itself currently runs on Windows. Managed -library/tool candidates above are cross-platform at their reviewed targets, but -native composition remains limited by the selected Hopper/Ghidra host and -format support. Real verification runs the canonical path on Linux and macOS -and may run pinned oracle checks on Windows without turning that development -check into a public Windows-host claim. - -If a caller-visible C# reconstruction operation is admitted, it is a distinct -optional capability. It accepts only an exact supported BYO `ilspycmd`, records -the tool and decompiler version, runs in an owned bounded process without -loading target code into REA, and links output to the canonical method -observation. Setup and installers do not download or install ILSpy, .NET, an -SDK, or a runtime. Doctor may inspect an explicitly configured executable and -report exact actionable incompatibility. Missing reconstruction never removes -canonical metadata/CIL capabilities. - -This decision does not make a decompiler process a security sandbox. The -initial public path can defer C# reconstruction while canonical contracts and -process isolation are proven. - -### 7. Use source-built conformance and operator-local real targets - -The repository stores source, build descriptions, expected semantic facts, -and tool lock manifests, but not compiled managed binaries. An isolated build -step creates fixtures outside source directories and records their SHA-256 and -MVID before verification. At minimum, the corpus covers: - -- Framework-compatible and modern .NET metadata; -- AnyCPU, x86, and x64 PE/CLI flags; -- generics, nested types, overloaded signatures, properties, events, resources, - attributes, and constants; -- exceptions, filters, prefixes, switch branches, P/Invoke, async and iterator - state machines; -- obfuscated and Unicode names without assigning semantics from names; -- malformed headers, tables, heaps, signatures, bodies, and exception regions; - and -- two semantically related builds with different MVIDs, tokens, layout, and - selected names. - -Expected facts are checked directly and against pinned -`System.Reflection.Metadata`; ILSpy, dnlib, or Mono.Cecil may add differential -coverage but do not vote by majority. Disagreement is a failing or explicitly -unsupported case, not silently normalized output. - -The real-application benchmark is an operator-local, opt-in osu! check derived -from the research workflow. It accepts an explicit path plus a compact -expectation manifest; verifies target SHA-256, MVID, architecture, and expected -assembly identity before analysis; and emits only bounded identities, tokens, -signatures, IL lengths/hashes, match states, and assertion results. It does not -emit full IL, decompiled source, runtime logs, user data, credentials, or -service/account material. No proprietary binary, hash-locked application -manifest, or derived dump is committed to REA. - -### 8. Keep runtime behavior separate from static managed analysis - -No managed runtime planning or execution tool ships. Managed inspection does -not execute assemblies or establish which methods ran. Where supported, direct -process capture can record target-level inputs and outputs, but it does not -prove CLR method execution or resolution. Internal managed behavior remains -unknown until independently observed with suitable runtime evidence. - -## Public contract consequences - -- Managed static operations will have CLI and MCP parity and produce the same - Evidence commitments. -- Adding managed operations changes the caller-visible tool inventory and - generated product catalog only in the implementation PR that admits them. -- Partial parsing includes admitted/dropped counts, exact limitations, and - coverage; it never returns a partial list labeled complete. -- Managed addresses distinguish file offsets, RVAs, native virtual addresses, - metadata tokens, and CIL offsets. They are not interchangeable. -- Cache compatibility is exact on artifact/component commitment, member-result - fields, parser profile, and producer version. For shipped v1, - `normalized_il_sha256` has no independent algorithm-version field. -- Provider-neutral workflows may compose canonical managed findings with the - one native binding, but every source operation retains its actual producer. - -## Alternatives rejected - -### Make ILSpy the only managed provider - -This would make canonical metadata/IL inventory depend on a separately -installed CLR and conflate source reconstruction with byte observations. It -would also make tool-version drift part of ordinary inventory. ILSpy remains a -valuable optional reconstruction and conformance tool instead. - -### Use `Assembly.Load` or reflection-only loading - -Both delegate parsing and dependency behavior to a CLR and make static support -runtime-dependent. Ordinary loading can execute target code; reflection-only -loading still establishes a CLR load boundary and is unavailable across modern -runtimes in a uniform way. Neither is required to decode ECMA-335 bytes. - -### Register the managed analyzer as another deep candidate - -That would force callers to choose managed analysis instead of Hopper/Ghidra -for ReadyToRun and mixed-mode artifacts even though both are needed. Disjoint -auxiliary composition represents the real ownership boundary. - -### Treat tokens, names, or decompiler output as durable identity - -Tokens and obfuscated names change across builds, while decompiler output -changes across tools and versions. They remain useful build-local coordinates -or reconstruction labels but are insufficient for exact or structural -cross-version identity. - -### Package a .NET runtime or mutable managed library in REA - -This would expand installer effects, package size, patching obligations, and -the target-parsing attack surface. The initial byte parser keeps canonical -support within REA's existing Node runtime; external managed tools remain BYO -or development-only oracles. - -## Rollout gates - -1. Add read-only triage and identity with malformed-input bounds. Shipped. -2. Add metadata, signatures, method bodies, deterministic decoded-CIL tuple - fingerprints, and exact artifact Evidence. Shipped with the v1 limitations - above; full normalized-CIL semantics remain planned. -3. Add obfuscation-resistant slices and cross-build match states. Shipped. -4. Compose explicit managed/native boundaries with Hopper/Ghidra. Declaration - inventory and native export/function Evidence matching shipped; native-body - bridge mapping remains planned. -5. Add source-built, pinned real-tool, package, CLI, and MCP conformance. - Source-owned conformance is shipped; pinned external real-tool checks remain - planned. -6. Project authenticated managed static Evidence into the application graph. - Shipped. - -Every gate must keep unsupported ReadyToRun, NativeAOT, IL2CPP, mixed-mode, or -bundle cases explicit rather than broadening claims to make a fixture pass. - -## References - -- [ECMA-335: Common Language Infrastructure](https://ecma-international.org/publications-and-standards/standards/ecma-335/) -- [.NET ReadyToRun deployment overview](https://learn.microsoft.com/en-us/dotnet/core/deploying/ready-to-run) -- [.NET Native AOT deployment overview](https://learn.microsoft.com/en-us/dotnet/core/deploying/native-aot/) -- [.NET single-file deployment overview](https://learn.microsoft.com/en-us/dotnet/core/deploying/single-file/overview) -- [`System.Reflection.Metadata` package](https://www.nuget.org/packages/System.Reflection.Metadata/10.0.10) -- [ILSpy repository and releases](https://github.com/icsharpcode/ILSpy) -- [dnlib repository](https://github.com/0xd4d/dnlib) -- [Mono.Cecil repository](https://github.com/jbevain/cecil) diff --git a/docs/adr/README.md b/docs/adr/README.md deleted file mode 100644 index 7130daa78..000000000 --- a/docs/adr/README.md +++ /dev/null @@ -1,11 +0,0 @@ -# Architecture decision records - -Accepted decisions describe the architecture that implementation PRs must -follow. Acceptance does not by itself mean the behavior is shipped; each record -states its implementation status separately. - -| ADR | Status | Implementation | -| ------------------------------------------------------------------------------------------------------------------ | -------------------- | ------------------------------------------------------------------------------------------------------------------------------------- | -| [0001: Provider selection and analysis profiles](0001-provider-selection-and-analysis-profiles.md) | Accepted | Provider selection and 25 read-only Ghidra operations on Linux/macOS and experimental Windows x64 P0; Linux/macOS session annotations | -| [0002: Controlled JavaScript replay authority and sandbox policy](0002-controlled-replay-authority-and-sandbox.md) | Superseded | Historical design; the controlled JavaScript replay tool was removed | -| [0003: Managed-code evidence and provider boundary](0003-managed-code-evidence-and-provider-boundary.md) | Partially superseded | Seven static managed tools remain; runtime planning removed | diff --git a/docs/capability-migration.md b/docs/capability-migration.md deleted file mode 100644 index 1a757dd82..000000000 --- a/docs/capability-migration.md +++ /dev/null @@ -1,477 +0,0 @@ -# Incremental capability migration - -This working guide tracks the migration in #740. Update it as concrete domains -migrate. Existing public contracts and domain-specific ports remain authoritative; -there is no new universal provider interface or fixed module manifest. - -## Ownership to preserve - -| Responsibility | Current owner | Migration direction | -| --------------------------------------------- | ----------------------------------- | ----------------------------------------------------- | -| Analyst semantics and Evidence representation | domain | Keep pure and provider-neutral | -| Named public input/output contracts | contracts | Keep exact CLI/MCP meaning and complete discovery | -| Producer parsing and external tool protocols | provider adapters | Keep behind the relevant typed port | -| Shared analyst workflows | application | Share between CLI and MCP | -| Concrete provider construction | composition, MCP startup | Keep fresh typed factories at the production boundary | -| Investigation Evidence and Unknowns | InvestigationRecords/EvidenceLedger | Consume narrow read/write/atomic record ports | -| Target/profile snapshots and invalidation | BinarySessionRecords/BinarySession | Keep binary-owned | -| Subprocess ownership and host primitives | process/windows | Reuse; preserve cancellation and cleanup | -| Public translation | CLI/MCP adapters | Delegate to shared workflows and named contracts | - -Retain exact Evidence IDs, provenance, detached reads, atomic Evidence/Unknown -mutations and optimistic revisions. Preserve current close and failed-cleanup -record clearing while extracting ownership; #721 owns any retention-policy change. -Each CLI invocation and MCP connection keeps its own state. Provider registration -must not launch engines or install prerequisites. - -## Migrate before generalizing - -1. Record the actual callers, state, environment, protocol and verification owner - of the capability being migrated. -2. Move concrete construction behind its existing typed port. Preserve injected - ports, selected environment, queues, failure state and cleanup behavior. -3. Test both caller paths and the applicable package/real-provider boundary. -4. Use independently migrated capabilities to identify repeated metadata and - composition needs. Derive a small source declaration only when those needs - are established; keep unlike target/lifecycle semantics separate. -5. Move a coherent capability's files, fixtures and verification lanes in a - mechanical PR. Update all path-sensitive consumers in that same PR. - -Pure helpers do not need an interface each. A capture-format adapter or typed -factory may be sufficient. Avoid generic JSON execution, external MCP catalog -passthrough, eager provider acquisition, new permission fields or a multi-package -rewrite. Optional loading should preserve successful peers and report the failed -capability without hiding tools from the catalog. - -## Checks before and after a move - -`npm run verify:test-discovery` compares actual Vitest file discovery with every -repository-owned `*.test.ts` under src and tests, including newly added files. The -check requires Git, includes untracked non-ignored tests, and excludes removed -paths during a move. It -rejects missing files and overlapping project ownership. It only lists files; -it does not replace behavior tests or real-provider verification. - -Check imports, project isolation and coverage, development-test selection, -lint exceptions, catalog/generator imports, npm and CI verification entrypoints, -fixture ownership, and installed-package bridge/native path resolution. -Domain and contracts must not import application workflows, caller adapters or -provider implementations. Production composition is a deliberate outer boundary. -Other existing adapter imports will migrate incrementally; do not widen a lint -exception to conceal new protocol code inside application workflows. - -`npm run verify:module-boundaries` parses repository-owned source imports, -reexports, type imports and literal dynamic imports, then checks their resolved -source layer. Nested names such as `domain/android` are domain code; `src/android` -is the provider adapter. The check runs with lint, including cached static gates. -It covers tracked and untracked non-ignored TypeScript source and skips cached -paths removed during a move. Pass exact source paths for a focused check. - -Pure source and its tests keep inward dependencies. Application/server provider -construction guards cover the migrated deep, Android, firmware, JavaScript recovery and observation -implementations. Application workflows cannot import composition, including the -former binary runtime entrypoint; shared browser capture/export helpers remain admitted while their -ownership is reviewed. Test lane restrictions still use Oxlint. Producer-dependent -browser and managed comparison tests live in boundary lanes with their original -fixtures and assertions. This is a development guard, not runtime authorization. - -Select verification by the claim changed: schema/composition, package, real -engine or host-native workflow. Record host/target prerequisites and source-owned -fixtures. Run heavy lanes serially with bounded workers/heaps and reuse existing -engines. Recording ports do not prove real provider or platform support. - -## Extension examples to validate during migration - -An implementation of an existing port should require a provider adapter, typed -production factory selection, exact producer parsing/provenance and matching -verification. Existing workflow meaning and public contracts should stay stable. - -A new analyst capability deliberately adds semantics, named contracts, a typed -port/workflow when needed, both caller adapters, availability, documentation and -verification. Its integration is broader than swapping a provider. The migration -should eliminate incidental duplicate wiring, not erase those design decisions. - -Update these examples with real migrated entrypoints before closing #740. - -## First production factory migration - -`src/composition/android.ts` and `src/composition/firmware.ts` construct providers -behind the existing AndroidAnalysisPort and FirmwareAnalysisPort. CLI registrars -pass their selected environment; MCP uses the process environment unless a caller -injects a port. Each factory call returns a fresh instance and acquires no process, -workspace or toolchain. Queues, cleanup-failure state and per-request cleanup stay -inside the provider. The existing launcher seams remain available to boundary -fixtures, which now exercise the production factories. - -Application workflows and MCP registration must not import these concrete -providers. Composition may import their implementations. Binary composition also -uses this outer boundary after its direct-analysis callers were migrated. -This concrete pilot does not introduce a shared provider interface or module -manifest. - -## Independent optional observation adapters - -Browser observation, browser scenarios, Electron observation, Electron scenarios -and V8 Inspector observation now have separate factories in `src/composition/`. -CLI commands reuse them; MCP startup dynamically imports each factory independently. -`OptionalObservationProviders` describes these five existing typed ports only. It -is provisional startup wiring, not an extensible plugin manifest or a universal -provider interface. - -A failed import or constructor retains the successful peers. The complete tool -catalog remains advertised. Session availability reports the failed adapter and -its actual reason, and affected handlers return the existing capability-unavailable -error. Explicit loading failures take precedence over configured availability. -Passive Electron and active Electron failures are independent; static JavaScript -analysis and runtime reconciliation remain usable without either adapter. CLI -Electron commands also load only their selected runtime factory at execution. - -Source identities are available without importing the optional implementations. -Registration acquires no engines, probes no endpoints, changes no permissions and -installs nothing. Core transport failures and shutdown still use their existing -lifecycle. Fault-injection coverage uses the production loading/startup path and -actual MCP SDK calls; recording ports establish composition behavior, while the -real Inspector lane separately establishes runtime behavior. - -## Investigation record ownership pilot - -`src/application/investigation/InvestigationRecords.ts` owns one existing -EvidenceLedger per runtime. The production session composition constructs that -owner explicitly; the direct BinarySession constructor retains a fresh default -for compatibility. There is no second ledger, global store, persistence or new -retention policy. - -The ledger, Unknown Evidence helper and the ledger's two original behavior tests -now live beside the owner in `src/application/investigation/`. Bundle validation -and record-owner consumers follow the new paths; retained-reference access still -uses the narrow record ports. The move preserves implementation bodies and test -assertions. Binary snapshot/cache files remain with their existing owner. - -EvidenceReader, EvidenceWriter and EvidenceUnknownWriter describe existing -read/write/atomic callback needs. UnknownRegistryPort preserves optimistic -revisions and consistency verification. Non-binary MCP registrars use these -record types without depending on the binary execution/lifecycle surface. -BinarySessionPort retains its compatible record methods through composition. - -BinarySessionRecords remains the snapshot owner and compatibility facade. It -passes the actual active target explicitly for recordUnknown/updateUnknown; -recordEvidenceWithUnknown keeps its target-free mutation subject. Snapshot import -merges records without notifying midway, then emits after the cache commits. -Bundle import retains the full change delta so Unknown-only changes notify even -when recordsAdded is zero. Clearing resets records, cache and invalidation before -notification; observer failures remain best effort after a committed mutation. - -Close and failed-cleanup clearing, ordinary target-switch retention, detached -reads, Evidence identities and deterministic bundle transfer keep their existing -behavior. Snapshots still require an active target and concrete analysis profile; -extracting investigation ownership does not make them a target-free record store. -#721 remains responsible for any future lifetime-policy change. - -## Inspector adapter ownership - -`src/inspector/` owns V8 Inspector discovery, passive capture, script/target -location interpretation and its lightweight provider identity. The composition -factory and optional loader use this owner. Browser CDP connection, endpoint/value -and Electron file-location helpers remain deliberate shared adapter utilities at -their existing paths; no provider wire protocol moves into domain or process. - -Loopback fixtures and producer boundaries now live under -`tests/fixtures/inspector/` and `tests/boundary/inspector/`. The socket-backed -unresolved-target reconciliation test moves from composition into the forked -boundary lane with its original assertions. Adapter tests retain fork isolation. -`npm run verify:inspector` keeps its public development entrypoint and delegates -to `scripts/verify/inspector/runtime-observation.mjs`, using the real Node fixture -under `tests/conformance/inspector/`. Catalog imports, source guards and test -discovery follow the new paths. Observation authority, lifecycle, producer -interpretation and caller-visible contracts stay unchanged. - -## Auxiliary source declaration pilot - -Artifact, macOS native and managed static metadata now belongs to each adapter's -`*ProviderMetadata.ts` module. The implementation and -`src/composition/auxiliaryAnalysisProviders.ts` reuse the same identity and exact -capability descriptors. The declarations reference existing AnalysisProvider -loaders; they cover these three disjoint auxiliary implementations only. Deep -candidate selection and the five typed observation ports remain separate. - -The binary runtime reads these source declarations while auxiliary construction -stays lazy. The separate managed-only entrypoint retains direct construction for -its execution-free workflow. The MCP catalog generator reads declarations without -constructing implementations, using its canonical macOS platform; actual runtime -host restrictions remain explicit. -Discovery and binary operation-kind routing now consume canonical source contracts -rather than their generated JSON artifact. No declaration imports its generated -output. Generated catalog facts and public schemas stay unchanged. - -All observation registrars resolve contracts by name. An isolated compiled-process -test reverses presentation arrays and performs actual MCP SDK calls to verify -advertised schemas and handler meaning. Reordering metadata must not select a -different operation, schema, provider or deep-provider priority. - -## Android layer and verification ownership - -Android workflows and their existing typed port live in `src/application/android/`, -pure APK/inventory semantics in `src/domain/android/`, and the named inspection -contracts in `src/contracts/android/`. The headless JADX producer remains in -`src/android/`; `src/composition/android.ts` supplies its fresh factory to both -caller adapters. Public names, result meaning and JVM/queue/cleanup limits are -unchanged. Inventory-only application projection remains execution-free. - -The real lane and signal-cleanup helper belong to `scripts/verify/android/`; its -explicit downloader and fixed manifest belong to `scripts/fixtures/android/`. -`verify:android` and `fixtures:android` keep their command names and repository-root -`_reference/apk-integration/` fixture default. Synthetic producer fixtures stay in -`tests/fixtures/android/`, with matching boundary and composition ownership. Real -JADX CLI/MCP parity requires the fixed APK and existing audited JAR; synthetic -protocol/cancellation success does not establish that engine or an unverified host. - -## Deep-provider source declarations - -Hopper, Ghidra and IDA declare their implemented analyst operations in their -own `*ProviderCapabilities.ts` modules. Hopper's inventory is typed against the -existing direct-operation contracts; Ghidra reuses its inventory/function adapter -operations; IDA retains its read-only operation family. Frozen operation arrays -feed the existing capability builders without importing generated catalog data. - -The catalog generator reads these source declarations and joins operation names -to canonical `TOOL_CONTRACTS`. A missing contract identifies the declaring provider -and operation. Public schemas and descriptions remain owned by contracts; the -declarations describe implementation coverage. Provider adapters must not consume -the generated MCP catalog that is derived from them. The resolved module guard -checks this edge while admitting canonical source contracts, generated package -metadata and the existing application snapshot-cache consumer. - -Exact provider effects and lifecycle differences remain in their adapters: -Hopper's GUI and mutation effects, Ghidra's experimental Windows authority and -mutation restrictions, and IDA's attached/headless effects and live observations. -This source increment introduces no common provider lifecycle or new interface. - -## JavaScript application ownership - -`src/application/javascript/` owns JavaScript/Electron artifact analysis and -graph projection, feature/version/export/source comparisons, runtime Evidence -and reconciliation, observation ports, and source recovery workflows. Pure -analysis and public schemas keep their existing domain/contract ownership; -shared artifact inventory, investigation records and execution types remain -shared. CLI, MCP and production factories import the owning workflow directly. - -`src/domain/javascript/` owns static parsing and semantic IR, application and -semantic graphs, Electron facts, runtime reconciliation, recovery result -semantics, and source-to-bundle comparisons. Shared Evidence, digests, ordering, -comparison semantics and historical-source graphs stay at their existing owners. -Native handoffs link exact-subject Evidence without selecting or executing a -provider. - -`src/contracts/javascript/` owns the JavaScript/Electron observation and recovery -contracts, JavaScript-specific workflow requests, and their examples. The shared -`applicationToolContracts.ts` aggregate stays at the contract root because it also -owns reconstruction and Android/Apple projection operations. Canonical tool -aggregation, named handler binding and SDK schema presentation remain unchanged. -Catalog source paths, compiled verifier/SDK fixtures and recovery CI filters -follow the domain/contract owners. - -`scripts/verify/javascript/` groups application-workflow, runtime-reconciliation, -large-digest and real-recovery entrypoints. Their existing npm commands retain -their invocation semantics. Recovery fixture compilation lives beside its -unchanged sources under `scripts/fixtures/javascript-recovery/`. Its default -REA entrypoint still resolves to `scripts/rea.mjs`. Managed conformance imports -the relocated compiled workflow; owning CI source filters follow the new paths. - -`src/artifacts/javascript/` owns directory/ASAR reader construction and relevant -file acquisition, including root/nested byte verification, UTF-8 diagnostics and -nested reader cleanup. Its narrow factory returns the existing `ArtifactReader`; -the application retains input admission, inventory, graph construction and its -original root-reader `finally` close. Pure file facts live in -`src/domain/javascript/javascriptArtifactFiles.ts`, and the shared immutable -inventory snapshot lives in `src/domain/artifactInventorySnapshot.ts`. Inventory -options and integrity policy remain application-owned. Shared stream hashing -lives in `src/artifacts/ArtifactHash.ts`, preserving complete byte counts and the -bounded classification prefix. Acquisition/hash guards reject imports from -application, composition or caller adapters; JavaScript workflows use the factory -instead of importing concrete ASAR/directory reader classes. - -ZIP, native-DMG and Mach-O inventory reader selection keep their existing -lifecycles. Runtime ports retain their distinct Inspector, passive Electron and -active Electron effects; shipped Electron hooks remain at their existing -provider boundary. - -## Managed capability ownership - -`src/domain/managed/` owns managed artifact and member semantics, reconstruction -import, native-boundary verification and application graph projection. -`src/application/managed/` owns their shared Evidence workflows, and -`src/contracts/managed/` owns their named contracts and examples. Producer -metadata/IL parsing stays in `src/dotnet/`; target resolution, Evidence, ordering -and investigation records retain their shared owners. Path-based member -comparison preserves its actual target/byte-digest admission and existing parser -helper. - -`scripts/verify/managed/` groups the portable conformance entrypoint and its -support, manifest, oracle and completion-report helpers. Its PE byte fixture -builder and declaration live in `scripts/fixtures/managed/`, shared with the -installed-package lane and existing managed conformance tests. `verify:managed` -keeps its command name and optional caller-cwd manifest resolution. Compiled -catalog and generator imports follow the owners; document-check cache inputs -include the relocated runtime verifier and fixture files. - -Portable byte fixtures, optional operator-local manifests, actual ILSpy oracles -and real Ghidra NativeAOT checks retain their distinct proof levels and -prerequisites. The existing NativeAOT fixture/engine lane keeps its owner. This -move preserves implementations, Evidence identities and generated commitments. - -## Binary application ownership - -`src/application/binary/` owns the active session, deep-provider registry and -evaluation, operation routing, disjoint provider composition, lazy auxiliary -clients, cancellation and client cleanup. Its records facade and snapshot -cache/files remain bound to the exact target and analysis profile. Investigation -records still have their separate owner in `src/application/investigation/`. - -`AnalysisProvider.ts` stays shared because its identity/execution types also serve -nonbinary ports. `BinaryTargetResolver.ts` stays shared because Android, firmware, -managed and artifact workflows use its target parsing. Production wiring lives -in `src/composition/binary.ts`; application workflows gain no outward composition -dependency. - -The three colocated application tests move with their owner. Composition cases -remain in `tests/composition/analysis-sessions/`; filesystem and SDK cases retain -their boundary lanes. The shared `tests/fixtures/binarySession.ts` factory supports -several capability families. Existing recursive application globs discover the -moved tests; bridge assets and real-verifier entrypoints remain at their owners. - -## Firmware layer and verification ownership - -Firmware workflows and their existing typed port live in -`src/application/firmware/`, pure request/result semantics in -`src/domain/firmware/`, and named contracts in `src/contracts/firmware/`. -`src/firmware/` keeps producer commands, report parsing, source copies, publication -and cleanup; `src/composition/firmware.ts` supplies a fresh typed factory. - -The real lane belongs to `scripts/verify/firmware/analysis.mjs`; the fixture runner -and its unchanged Python producer belong together in `scripts/fixtures/firmware/`. -The npm command names and selected `REA_FIRMWARE_FIXTURE_ROOT` are unchanged. The -runner retains its working-directory-relative default; the verifier retains its -repository-root default. Both point to `_reference/firmware-integration/generated` -when invoked through the documented npm commands. Optional ext4/Ghidra lanes keep -their own prerequisites. Source-fixture generation and synthetic producer tests -do not establish real Binwalk/Unblob analysis or another host's execution support. - -## Apple artifact capability ownership - -`src/domain/apple/` owns Apple application anatomy/projection, asset catalog -facts, Interface Builder graphs, keyed archive semantics and plist values. -`src/application/apple/AppleApplicationService.ts` retains the shared workflow -that projects authenticated IPA/macOS inventory Evidence. - -`src/artifacts/apple/` owns the actual producer boundary: directory acquisition, -assetutil invocation, plist/NIB decoding, encoded view-parent interpretation and -keyed archive inspection. These helpers are used by ArtifactProvider, retain -its existing typed analysis boundary, injected assetutil seam and original reader -cleanup, and do not require a new port for each helper. Producer code no longer -lives in the application layer. Guards reject application/composition/caller -imports from this owner while admitting pure facts and shared artifact readers. -Adapter-source tests retain their assertions in the existing forked adapter lane. - -`scripts/verify/apple/` groups archive, asset catalog, Interface Builder, -Objective-C/Swift dispatch and macOS bundle verification. The macOS bundle fixture -builder lives in `scripts/fixtures/apple/`; NIB byte fixtures stay beside their -artifact decoder. Existing format-specific conformance sources and goldens keep -their locations, and relocated verifiers resolve them against their own file -URLs. Public npm entrypoints and CLI/MCP contracts stay stable; generated-document -cache inputs include the relocated runtime scripts and fixture builder. - -Portable decoder and injected producer tests establish their format/boundary -claims. Actual Swift/Xcode, assetutil, native NIB, bundle signing and DMG lifecycle -claims require the real Apple lane. Native binary API/value-flow semantics and -shared cross-domain contract aggregates keep their separate owners. - -## Artifact inventory and extraction ownership - -`src/artifacts/inventory/` owns format classification, reader selection, inventory -policy, canonical scanning and artifact graph construction. The shared artifact -provider and JavaScript reconstruction workflow consume this owner. Snapshot -semantics remain in `src/domain/artifactInventorySnapshot.ts`. - -`src/artifacts/extraction/` owns member selection, integrity checks and transactional -materialization through the existing readers and safe output tree. Reader cleanup, -cancellation, rollback, archive containment and caller-visible results retain their -existing behavior. Acquisition guards reject imports from these producer owners -into application workflows, production composition or CLI/MCP adapters. - -`src/application/artifacts/ArtifactExtractionDestination.ts` owns the caller's -default temporary output-root choice for both CLI and MCP. The adapters pass this -destination to the existing provider operation; format handling and member writes -remain in the artifact owner. - -Add format-specific acquisition behavior beside the inventory readers and exercise -its actual producer representation in filesystem boundary tests. Add caller -workflow behavior in the application owner and verify both adapters. The installed -package lane checks inventory identity and extraction through CLI and MCP; the -real Apple bundle lane separately establishes DMG mount and detach behavior. - -## Binary production composition closure - -`src/composition/binary.ts` is the shared production owner for deep candidate -construction, lazy auxiliary composition and the managed-only session factory. -It preserves the existing constructor bodies, selection policy and fresh session -per invocation. Factory registration acquires no engine or target. - -`DirectAnalysis` and `DirectAnalysisStatus` receive these two existing factories -through `DirectAnalysisDependencies`. They retain configuration parsing, selected -status environment, cancellation, snapshot replay/binding, Evidence projection -and finally-close. The dependency uses the existing BinarySession type, including -its replay policy; it adds no universal backend or lifecycle contract. - -`src/composition/directAnalysis.ts` binds the production factories for CLI and -one-shot managed MCP callers. Binding creates no session and stores no retained -analysis state. MCP startup constructs its connection session through the same -binary factory. Source and compiled fixtures use these entrypoints. The former -application runtime path is removed, and the source guard no longer admits its -outward provider/composition imports. - -## Native analyst capability ownership - -`src/domain/native/` owns portable native inspection, instruction/data types, -load-image facts, API boundaries, annotations, metadata recovery, UI observation -semantics and value/investigation graphs. `src/application/native/` owns shared -API projection, call routes, dispatch inspection and UI/value tracing. Its tests -keep the existing service lane. `src/contracts/native/` owns the named native -adapter contracts; shared official/enhanced aggregates remain at their owners. - -Producer interpretation, external commands and host UI lifecycle remain in -`src/native/`, Hopper, Ghidra and IDA. The shared function dossier still lives in -`domain/hopperValues.ts`: its historical name does not make its meaning exclusive -to Hopper. This move retains existing typed ports, exact result schemas, -observed/derived/unknown distinctions, profiles and Evidence links. - -Source and compiled verifier imports follow the new owners. Host UI and real -native-value verifier locations are retained until their own proof lanes migrate; -interactive macOS UI success and real dependency tracing require their respective -host/engine workflows. Portable source fixtures and injected call-route tests -establish their narrower boundaries, without expanding provider/platform support. - -## Process capture and analyst workflow ownership - -`src/domain/process/` owns scenario/capture validation, portable process trees, -trace specifications, observations, comparisons and the canonical process Evidence -identity. The standalone contract example lives in `src/contracts/process/`; -shared session contracts retain their broader investigation owner. - -`src/process/capture/` owns the actual PTY boundary, terminal rendering, sampling, -selected child environment, filesystem snapshots/effects, runtime path admission, -event journal, settlement and resource cleanup. It reuses `src/process/` ownership -primitives and the Windows host substrate. Its identity import follows the -existing pure declaration directly. A source guard rejects outward application, -composition and caller imports; no new provider or lifecycle interface is added. - -`src/application/process/` retains Evidence projection and file-backed CLI -capture/comparison workflows. Capture helper tests use the forked adapter lane; -Evidence/host integration tests use the serial process-boundary lane. Their -fixtures and assertions remain unchanged. Installed capability-probe URLs follow -the new compiled adapter path, including the optional-dependency failure probe. - -Windows PTY capture retains its existing unavailable outcome because descendant -cleanup is not yet verified. Native Job Objects and a PTY binary do not establish -that capture workflow. Real POSIX capture/cleanup and installed terminal/PTY module -resolution require their actual host and package lanes; portable comparison -fixtures establish their separate evidence semantics. diff --git a/docs/cli.md b/docs/cli.md index 51a45c462..a80339cf4 100644 --- a/docs/cli.md +++ b/docs/cli.md @@ -80,7 +80,7 @@ or close. Provider failures are returned with their original reason. For an `ambiguous` selection error, choose from `details.candidate_ids`; for `provider_unavailable`, run `rea doctor --provider ID --json` to diagnose the selected engine. See [task readiness](installation.md#check-readiness-for-your-task) -and [provider selection](adr/0001-provider-selection-and-analysis-profiles.md). +and [provider selection](mcp-contracts.md#identity-and-discovery). The session reports work still in progress through `analysis_activity`. A client timeout can end its wait while the provider continues analyzing. diff --git a/docs/ghidra-provider.md b/docs/ghidra-provider.md new file mode 100644 index 000000000..0ba061177 --- /dev/null +++ b/docs/ghidra-provider.md @@ -0,0 +1,72 @@ +# Ghidra provider semantics + +REA imports one target into an ephemeral Ghidra database. Linux/macOS support +read-only analysis and atomic function-name/entry-comment edits; Windows P0 is +read-only. GUI controls and persistent database mutation are unavailable. +Executable bytes remain unchanged. + +Use [installation](installation.md#ghidra) for supported hosts, prerequisites, +configuration, startup deadlines, resource controls, and cleanup recovery. +[Windows P0](windows-ghidra-p0.md) has its own admission and lifecycle boundary; +[NativeAOT recovery](ghidra-nativeaot.md) is an optional, separately verified +extension. Check the [release boundary](installation.md#released-package-and-main) +when using npm. + +Provider selection and profile identity follow +[MCP contracts](mcp-contracts.md#identity-and-discovery). One deep +provider stays bound to the target; runtime failure never selects another +provider automatically. The contracts below describe Ghidra observations, +not equivalence with another engine's pseudocode or analysis database. + +## Admitted inventory semantics + +| Concern | Ghidra contract | +| ---------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Program identity | One `analyzeHeadless` import produces exactly one Program; `list_documents` therefore returns exactly one name. | +| Addresses | Default memory uses lowercase `0x` hexadecimal. Non-default and external spaces use `:0x` and remain round-trippable. The handshake commits image base and default address-space name. | +| Symbols | `list_names` includes address-bearing memory and external symbols, including dynamic symbols, while excluding variable and no-address namespace records. Each item reports primary, dynamic, external, symbol type, and source facts. | +| Procedures | Both non-external and external functions are listed. A local thunk remains distinct from its resolved target; exact and qualified name lookup fails on ambiguity rather than guessing. | +| Strings | Only Ghidra-defined string `Data` is observed. Items report charset, byte length, and whether a required null terminator is missing. The API cannot distinguish a present terminator from fixed/Pascal layouts when no terminator is missing, so that state is named `present_or_not_required`. | +| Memory | Memory-block end addresses are exclusive. Read/write/execute, initialization, overlay, address space, and image base are direct Ghidra observations. | +| Inventory | Procedure, symbol, and string listings plus searches return the complete matching collection in one response; callers do not provide offsets or result-count limits. | +| Search | Literal and Java-regex searches scan the complete immutable inventory and return all matching entries inline. Search waits for a reply or caller cancellation; there are no caller-supplied offsets or result-count limits. | +| Analysis state | The socket is exposed only after default auto-analysis completes. Established operations wait for their reply or caller cancellation; there is no fixed response-size ceiling. | + +## Admitted function-analysis semantics + +| Concern | Ghidra contract | +| ----------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Decompiler lifetime | One persistent `DecompInterface` is opened for the imported Program and disposed during bridge shutdown. Native decompilation has no fixed per-function deadline. External functions or functions without bodies return `null`; cancellation and native failure remain distinct. | +| Serialization | A FIFO sends one Program request at a time. Caller cancellation removes queued work promptly and is passed to active socket waits. This is an adapter safety commitment, not a claim that every Ghidra API is thread-safe. | +| Function identity | Every function result carries the entry address and Ghidra FunctionManager classification for external, thunk, and resolved thunk target. These are observations, not proof that unresolved targetless calls have been recovered. | +| Assembly and pseudocode | Assembly is complete Ghidra Listing text; pseudocode is Ghidra decompiler output. Neither is original source, and cross-provider comparison never treats Hopper and Ghidra text as equal or unequal semantic facts. | +| Instruction fast path | `read_function_instructions` returns every raw Listing instruction for the requested function without invoking the decompiler or whole-program name/string inventories. | +| Calls and references | Callers/callees contain only resolved functions. Reference edges preserve exact ReferenceManager type and call/jump/data/read/write/indirect/computed/conditional/terminal/external facts. Targetless computed flow remains unknown. Synthetic entry-point references without actionable memory sources are omitted explicitly. | +| CFG | Dossiers use `BasicBlockModel` and retain only non-call successors inside the function body. CFG topology is address-normalized for comparison; provider-specific block construction remains a declared difference. | +| P-code value flow | Function dossiers include up to 3,000 high-p-code operations, at most 64 inputs per operation, and 12,000 def-use edges. Results identify memory reads/writes, branches, and calls by p-code opcode, and mark decompiler-dead operations. Large results report truncation, an omitted-operation lower bound, and known omitted-input/edge counts. The dossier is an intra-function graph; `trace_native_values` composes derived parameter/argument and return/output dependencies across resolved calls. It does not resolve memory aliasing, call side effects, runtime behavior, persistent state, or business meaning. | +| Result extent | Function instruction scans and native API boundary observations remain complete. P-code flow is the explicit bounded exception; its count fields distinguish exact retained counts from lower-bound or known-omitted counts. Caller cancellation and provider failures remain distinct. | + +`npm run verify:ghidra` compiles the versioned C oracle into debug and stripped +host-native targets (x86-64/AArch64 ELF on Linux or Mach-O on macOS), plus a native +DWARF 4 type-layout object. It proves the admitted operations, external functions, resolved thunks, exports, stripped-name +behavior, direct and targetless indirect calls, typed references, strings/xrefs, +multi-block CFG, bounded p-code def-use/effect links, semantic enhanced workflows, cancellation, startup deadlines, +serialized concurrency, malformed-target rejection, profile identity, and +process/project cleanup against real Ghidra 12.1.4. Unit fixtures separately +cover startup deadlines, process exit, queued cancellation, and malformed wire +output. + +`npm run verify:ghidra:cross-format` adds AArch64 ELF, x86-64 PE, and x86-64 +Mach-O target coverage. It requires `clang`, LLD, and `lld-link`; set +`REA_CLANG`, `REA_LLD`, or `REA_LLD_LINK` to select them. The verifier checks +these tools before compilation. Keeping this matrix separate lets Linux +host/provider acceptance run with only the host compiler. + +`npm run verify:ghidra:windows` and its `-- --x86` variant check the source-owned native x86 and x86-64 PE +fixture, all 25 read-only operations, target/snapshot/import SHA-256 linkage, +and project, endpoint, process, and runtime cleanup. The independent native +lane checks DACLs, handle admission, cancellation, and Job Object lifecycle. +`npm run verify:ghidra:windows:package` packs and installs REA into an isolated +prefix, then checks ordinary-user CLI/MCP operations against canonical contracts. +All lanes require matching native controls; package startup alone establishes +only package compatibility. diff --git a/docs/managed-code-analysis.md b/docs/managed-code-analysis.md index 92bc23060..74e11e55e 100644 --- a/docs/managed-code-analysis.md +++ b/docs/managed-code-analysis.md @@ -1,4 +1,4 @@ -# Managed-code analysis and planned extensions +# Managed-code analysis REA inspects .NET PE/CLI artifacts without loading or executing their code. It can identify an assembly, inspect metadata and CIL, compare members across @@ -20,12 +20,11 @@ The seven shipped tools are: Each has a matching CLI command: replace underscores with hyphens and prefix the name with `rea`, for example `rea inspect-managed-artifact`. -Native-body bridge mapping remains planned; managed runtime execution is not -part of the current tool set. +Native-body bridge mapping and managed runtime execution are outside the +current tool set. -This guide describes the implementation and verification of -[ADR-0003](adr/0003-managed-code-evidence-and-provider-boundary.md). The canonical -tool inventory is the [build-generated catalog](mcp-contracts.md#generated-catalog). +The canonical tool inventory is the +[build-generated catalog](mcp-contracts.md#generated-catalog). ## Shipped scope @@ -36,10 +35,8 @@ NativeAOT identity from an ordinary native PE. Inputs without admitted CLI metadata do not become managed assemblies through naming or routing guesses. Inspect separately obtained components explicitly and preserve their identities. -The broader deployment classification and native-body mapping below are design -goals from ADR-0003, not claims that every row has an implemented parser. A -valid marker can establish a candidate native boundary, not recovered native -semantics or runtime behavior. +A valid implementation marker establishes a candidate native boundary, not +recovered native semantics or runtime behavior. ## Analysis objective @@ -57,40 +54,7 @@ collapsing them: The ordinary workflow ends at question four. Static analysis never loads or executes the target. -## Planned deployment classification - -The intended extended classifier proceeds from outermost authenticated bytes inward: - -```text -source path - -> canonical path, size, SHA-256 - -> container inventory - -> PE/native header and architecture - -> CLI header and metadata root - -> deployment/runtime markers - -> per-component and per-method implementation availability - -> managed-only, native-only, composed, degraded, or unsupported route -``` - -The planned result is a vector rather than one label. For example, a single-file modern -.NET deployment can contain a native host, ordinary CIL assemblies, and -ReadyToRun components. Each component receives its own digest, classification, -coverage, and route while retaining the outer bundle commitment. - -| Case | Required positive observations | Claims that remain unavailable or inferred | -| --------------------- | ---------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------- | -| .NET Framework | Valid CLI metadata plus bounded Framework-specific target/reference evidence | Exact installed CLR and runtime behavior | -| Modern .NET | Valid CLI metadata plus bounded target/reference/runtime-config evidence | Exact runtime selected on another host | -| Unity Mono | Valid managed assembly plus authenticated Unity context when supplied | Engine behavior or native integration not represented by metadata | -| ReadyToRun | Valid managed metadata plus authenticated ReadyToRun header/sections | Native implementation semantics until a selected deep provider analyzes them | -| C++/CLI | Valid CLI metadata plus mixed-mode PE/header evidence | Mapping managed declarations to native implementations without explicit bridge evidence | -| Single-file | Valid authenticated bundle inventory and component extents | Components that are compressed or encoded by an unsupported bundle version | -| Unity IL2CPP | Authenticated native image/metadata pairing and supported metadata version | Canonical CIL and source-equivalent C# | -| NativeAOT | Native image plus bounded NativeAOT evidence | Ordinary CLI metadata/CIL unless independently present | -| Obfuscated assembly | Same positive byte observations as its underlying deployment form | Meaning inferred only from names | -| Malformed/unsupported | Exact admitted regions and failure locations | Completeness, successfully skipped rows, or semantics beyond the admitted parser boundary | - -### NativeAOT metadata recovery +## NativeAOT metadata recovery NativeAOT removes the ordinary CIL method bodies, so a CIL decompiler cannot reconstruct those bodies as C# source. The executable remains a native target @@ -110,28 +74,17 @@ lists supported OS/architecture targets and deployment limitations, while its [native library guide](https://learn.microsoft.com/en-us/dotnet/core/deploying/native-aot/libraries) describes exported shared-library entry points. -The [Ghidra NativeAOT analyzer](https://github.com/Washi1337/ghidra-nativeaot) -is a useful optional companion for recovering ReadyToRun metadata. It -rehydrates the `DEHYDRATED_DATA` section and annotates method tables, type -relationships, vtable slots, frozen objects, and strings. It does not recover -original C# method bodies. It is a separate Ghidra extension and interactive -metadata browser; REA does not currently install or invoke it through its -headless bridge. Its README describes ReadyToRun header discovery, including -symbol-based and heuristic paths, so a missed header or unsupported binary -should remain an explicit limitation rather than a failed claim about the -binary's contents. +REA supports an optional headless adapter for the Ghidra NativeAOT analyzer. +Supply a locally built `REA_GHIDRA_NATIVEAOT_JAR`; REA does not install the +extension or its toolchain. The verified boundary is .NET 8.0.22 RTR 9.1, +x86-64 ELF and native Windows PE targets analyzed on Linux x64 with Ghidra +12.1.4 and JDK 21. Other layouts, architectures, and hosts are unsupported. +See [NativeAOT recovery](ghidra-nativeaot.md) for build/configuration, recovered +metadata, derived-memory provenance, and the real verification lane. -The upstream analyzer currently gates its own analysis to x86-64. NativeAOT's -broader platform matrix does not establish REA coverage: PE/COFF, ELF, Mach-O, -shared libraries, target architectures, and runtime metadata versions each need -their own provider verification. REA's experimental Windows Ghidra boundary is -limited to admitted native x86 and x86-64 PE applications, so it does not currently -establish coverage for PE DLLs. Mobile NativeAOT targets should remain -experimental until REA has a matching provider and package-level verification. - -For REA's native analysis, preserve the same artifact path and digest used by -`inspect_managed_artifact`, and report recovered type data as provider -observations. Linking those types to native function addresses requires verified +For NativeAOT analysis, preserve the selected native artifact's path and digest +and report recovered type data as provider observations. Linking those types +to native function addresses requires verified provider evidence; names or vtable similarity alone do not prove that a native function implements a specific managed method. @@ -143,43 +96,18 @@ matches the normalized artifact SHA-256. A valid Evidence ID alone does not establish that those two identities agree. Evidence without a subject remains usable, with no subject digest available to cross-check. -Every planned operation returns a provider result and Evidence with four -commitment groups: +Artifact inspection returns the selected path, byte length and SHA-256, observed +PE/CLI classification, assembly and module identities, and metadata coverage. +Member inspection binds build-local tokens, signatures, row offsets, method +bodies, and decoded relationships to that artifact and MVID. Missing or malformed +facts remain explicit in coverage and per-member status. -### Artifact commitment +CLI `#GUID` heap values retain their exact 16-byte GUID text. REA does not impose +RFC 4122 UUID version or variant bits on MVID, EncId, or EncBaseId because +ECMA-335 metadata does not require those bit patterns. Tokens and RVAs are +build-local coordinates, not durable cross-build identities or native addresses. -- canonical local path, byte length, and SHA-256; -- outer-container identity and digest when inspecting a component; -- component name/path, byte extent, and SHA-256; -- observed PE machine, CLI flags, and conflicts; -- classification vector, supporting observations, and unresolved dimensions. - -### Module commitment - -- assembly simple name, version, culture, public-key/token state, flags, and - hash algorithm; -- module name, generation, and MVID; -- target framework and runtime/version strings with their exact metadata - locations; -- provider identity, analysis-affecting parameters, and their profile digest. - -CLI `#GUID` heap values are committed as their exact 16-byte GUID text. REA -does not impose RFC 4122 UUID version or variant bits on MVID, EncId, or -EncBaseId because ECMA-335 metadata does not require those bit patterns. - -### Entity commitment - -- table kind and build-local token; -- declaring scope and normalized CLI signature; -- row/heap/body locations expressed as typed file offsets, RVAs, or CIL - offsets; -- exact raw bytes or bounded value digest; -- raw CIL digest plus separately reported method-header, locals-token, and - exception-region observations where available; -- a deterministic decoded-instruction-tuple digest for completely decoded - CIL, with the limitations below. - -### Shipped decoded-CIL fingerprint +## Decoded-CIL fingerprint `inspect_managed_members` exposes two method CIL hashes with different byte boundaries: @@ -208,9 +136,7 @@ reproducible decoded-tuple fingerprint, not a canonical semantic CIL identity: it does not resolve tokens across MVIDs, does not commit full control flow or exception semantics, and does not replace the raw CIL digest. Cross-build comparison uses the value only as one unique-only structural tier and reports -ambiguity when that evidence is insufficient. A future normalized-CIL -commitment would need canonical token/literal identities, -complete branch and switch targets, and explicit locals/exception semantics. +ambiguity when that evidence is insufficient. For example, a decoded `nop; ret` body serializes exactly as `[["nop","none",null],["ret","none",null]]` and has digest @@ -218,73 +144,10 @@ For example, a decoded `nop; ret` body serializes exactly as This vector specifies the tuple order, `null` handling, absence of whitespace, UTF-8 encoding, and lowercase hexadecimal output. -### Epistemic commitment +## Member comparison -- authority: static bytes, reconstruction, structural inference, independent - validation, native provider, or future runtime observation; -- state: observed, inferred, unknown, or unavailable; -- confidence independent of authority; -- coverage and admitted/dropped counts; -- exact supporting Evidence IDs and actionable limitations. - -A semantic label such as `score_submission_gate` can annotate an entity, but it -is never the entity's identity. The same applies to an obfuscated name, token, -RVA, or decompiler-generated local name. - -## Planned static capabilities - -The caller-visible grouping and exact tool names are decided with the contracts -that implement them. The underlying capability slices are: - -1. **Triage and inventory**: container/component classification, assembly and - module identity, references, files, exported types, resources, attributes, - and runtime markers. -2. **Member inventory**: file-backed types, methods, fields, properties, events, - interfaces, nesting, generics, and normalized signatures. -3. **Method inspection**: headers, max stack, locals, CIL instructions, - constants, metadata operands, exception regions, and implementation flags. -4. **Relationships**: declared overrides, interface implementations, member - references, direct CIL callers/callees, field access, construction, and - interop declarations. Dynamic dispatch remains qualified. -5. **Structural search and comparison**: signature/API/constant/flow anchors, - exact and normalized hashes, competing candidates, and explicit - exact/structural/missing/ambiguous states. -6. **Managed/native composition**: P/Invoke, unmanaged exports, COM/mixed-mode - declarations, ReadyToRun/native bodies, single-file hosts, and authenticated - IL2CPP pairs linked to selected Hopper/Ghidra evidence. The shipped - verification workflow checks P/Invoke declarations against supplied native - export or function Evidence; native-body, thunk, and token-to-address bridge - mapping remain unavailable without explicit provider-supported evidence. -7. **Application graph projection**: managed assembly/module/type/method/field, - P/Invoke, and native-implementation declarations enter the same - Evidence-backed application graph vocabulary as JavaScript/Electron - findings while preserving managed static-analysis authority. - -Inventory lists preserve deterministic order and include every fact available -from the admitted PE/CLI streams. Malformed ranges and unsupported metadata are -reported as coverage issues instead of being silently omitted. Search exposes -scanned, matched, returned, and dropped counts. An unresolved indirect call or -a failed signature decode is not silently omitted from a completeness claim. - -## Obfuscation-resistant method slices - -Names are one weak feature among many. A behavior slice records the smallest -bounded set of observations needed to investigate a proposition: - -- normalized declaring/member signatures and inheritance/interface shape; -- framework and application API references; -- exact strings, numeric constants, enum-like values, and serialization keys; -- branch, switch, exception, loop, call, and field-flow shape; -- construction and generic-instantiation sites; -- raw CIL and decoded-tuple commitments, plus separate header, locals, and - exception observations; -- callers, callees, shared fields, and competing candidates; and -- limitations from reflection, dynamic dispatch, generated code, protection, - native transitions, or incomplete coverage. - -The workflow produces separate observation, inference, validation, and unknown -tables. It may say that a method is a strong candidate for a role; it cannot -turn that role into the method's durable identity. +Names and semantic labels do not establish durable member identity. Comparison +preserves observed differences, structural inferences, and unknowns separately. Cross-version matching first prefers exact CIL/signature identity. When that does not match, it pairs an exact declared type, method name, and raw signature @@ -310,16 +173,14 @@ Tokens are always remapped through observed structure. A caller cannot carry ## Managed/native boundary rules -| Boundary | Managed observation | Native observation | Permitted link | -| ----------------------- | ----------------------------------------------------------------------- | ------------------------------------------------------------------------------- | -------------------------------------------------------------------- | -| P/Invoke | Module, entry point, charset/calling-convention flags, declaring method | Import/export/symbol/function evidence from the selected deep provider | Exact declared-name/module link or qualified resolution inference | -| COM | Interop attributes, GUIDs, imported interfaces, method signatures | Native registration/vtable evidence when independently available | Identifier/signature inference with explicit environment limitations | -| C++/CLI | Managed declaration and implementation flags | Native body/function evidence | Only a provider-supported bridge observation; never token-as-address | -| ReadyToRun | Component/header and per-method CIL/native availability | Native section/function evidence | Authenticated image mapping with format/profile version | -| Unmanaged export | Export metadata/attribute when present | PE export and native thunk/function | Exact export identity plus provider-qualified address | -| Single-file host | Bundle entry/component identity | Host and native component evidence | Outer bundle plus component digest/extent commitment | -| Unity IL2CPP | Supported metadata entity with authenticated pairing | Generated native function/type evidence | Versioned IL2CPP mapping only; no invented CIL | -| Runtime-resolved native | API/constant/data-flow candidate | Loaded-module/symbol observation from an independently collected runtime source | Static candidate remains inference until separately observed | +`verify_managed_native_boundaries` compares declared P/Invoke module and entry +point identities with supplied native export or function evidence. Name/module +matches establish declaration links; they do not establish that a call executed. + +Metadata tokens are not native addresses. Native body mapping for C++/CLI, +ReadyToRun, NativeAOT, and IL2CPP requires separately verified format and provider +support. The managed static tools do not recover those mappings. The optional +[Ghidra NativeAOT extension](ghidra-nativeaot.md) has its own supported boundary. Declared import inventory supports a bounded positive claim. Its absence does not exclude dynamic resolution, generated code, protected code, native helpers, @@ -333,16 +194,12 @@ fixtures and expected semantic facts. An optional real ILSpy lane runs only when its executable is explicitly supplied; the default lane does not establish independent pinned-oracle parity. Production inspection needs none of those tools. -- A pinned `System.Reflection.Metadata` differential oracle is planned; it is - not part of the current verifier. - `ICSharpCode.Decompiler`/`ilspycmd` can supply reconstruction inference. A BYO reconstruction import records the supplied version and remains non-canonical. When `REA_ILSPY_CMD_PATH` points to an absolute runnable `ilspycmd`, `verify:managed` runs a source-owned real ILSpy oracle and imports its C# output as reconstruction inference against exact static member Evidence. -- dnlib and Mono.Cecil are potential future differential oracles. Their mutation APIs - are not exposed or included in the production parsing boundary. - The package contains no .NET runtime, SDK, ILSpy installation, proprietary assembly, or compiled conformance fixture. - Setup never installs or upgrades those tools. Doctor may inspect an explicit @@ -360,29 +217,15 @@ they do not claim that the REA application itself supports Windows. Native ReadyToRun, C++/CLI, NativeAOT, or IL2CPP coverage is additionally constrained by the selected Hopper/Ghidra host and format matrix. -## Source-built conformance corpus +## Conformance verification The current lane uses source-owned byte-built fixtures and malformed-input regressions, with optional operator-supplied applications and ILSpy verification. -The table below describes the desired expanded corpus; it is not a report that -all compiler-generated forms have been independently verified. -Fixture sources are intentionally small and behavior-focused. Build outputs -are generated outside tracked fixture directories and must not remain after -verification or packaging. - -| Corpus slice | Required properties | -| ---------------------- | ------------------------------------------------------------------------------------------ | -| Identity | Assembly/module/version/culture/public key, references, attributes, resources, MVID | -| Signatures | Nested/generic types, arrays, pointers, by-ref, function pointers where supported, varargs | -| CIL | Short/long branches, switch, prefixes, constants, locals, calls, fields, boxing, tokens | -| Exceptions | Catch, finally, fault/filter where the compiler/toolchain supports them | -| Generated structure | Async and iterator state machines, lambdas, closures, properties, events | -| Interop | P/Invoke flags, module/name mapping, managed/unmanaged implementation metadata | -| Architecture | AnyCPU, preferred-32-bit where applicable, x86, x64 | -| Obfuscation resistance | Unicode/meaningless/duplicate-looking names and structure-preserving renames | -| Cross-build comparison | At least two builds with changed MVID/token order/layout and known same/changed behaviors | -| Malformed/adversarial | Truncated/overflowing headers, streams, tables, heaps, signatures, bodies, and EH sections | +Run the deterministic lane with `npm run verify:managed`. See +[testing](testing.md#real-toolchain-verification-lanes) for prerequisites and +coverage. Generated outputs belong outside tracked fixture directories and must +not remain after verification or packaging. Verification compares semantic facts, not decompiler text. It checks resource ceilings, deterministic ordering, complete cleanup, CLI/MCP parity, Evidence @@ -492,32 +335,7 @@ evidence is collected. A direct process capture can retain declared inputs, outputs, filesystem and protocol activity for a target run; it does not prove which managed methods executed or how the CLR resolved them. -## Delivery sequence - -The managed-code track advances as reviewable pull requests: - -1. accepted evidence/provider boundary (this document and ADR); -2. read-only artifact triage and exact identity (shipped); -3. file-backed metadata, signatures, method bodies, raw CIL hashes, and the - decoded-instruction-tuple fingerprint (shipped; complete normalized-CIL - semantics remain planned); -4. obfuscation-resistant slices and cross-version comparison (shipped for - static member observations); -5. decompiler reconstruction import (shipped as analyst inference; REA does - not run ILSpy/dnSpy, and metadata/CIL remain canonical); -6. managed/native composition and truthful deployment degradation (declaration - inventory and native export/function Evidence matching shipped; native-body - bridge mapping remains planned); -7. source-built managed conformance and package/CLI verification (source-owned - PE/CLI corpus shipped through `npm run verify:managed`; optional BYO - `ilspycmd` real-tool oracle shipped through `REA_ILSPY_CMD_PATH`; dnSpy and - pinned Windows checks remain planned); and -8. managed static Evidence projection into the application graph (shipped). - -Each implementation PR updates generated product facts only for behavior it -actually ships and states which real-tool checks were performed. - -### Method metadata during build comparison +## Method metadata during build comparison Managed member comparisons report a `metadata` dimension when an observed MethodDef flags or implementation flags value differs, including accessibility diff --git a/docs/provider-evaluation.md b/docs/provider-evaluation.md deleted file mode 100644 index 8aa655e52..000000000 --- a/docs/provider-evaluation.md +++ /dev/null @@ -1,219 +0,0 @@ -# Static-analysis provider evaluation - -This guide describes repository main. Check the [released package boundary](installation.md#released-package-and-main) when using npm. - -Ghidra read-only analysis is available on Linux x64/arm64 and macOS x64/arm64. -Install a Ghidra 12.1.x release and the 64-bit full JDK that release declares, -then configure REA to use them. Current 12.1 releases require JDK 21 or newer -and set no maximum. The bridge is verified with Ghidra 12.1.4 and JDK 21. -Each installation needs the matching native decompiler. - -REA imports one target into a temporary project and exposes 25 read-only -operations after analysis completes. They cover inventories, search, -decompilation, assembly, function metadata, resolved calls, references, control -flow, instructions, and recovered types. Results identify the provider version, -analysis settings, target digest, and any unavailable or incomplete facts. -On Linux and macOS, `annotate_native_function` additionally edits names and entry -comments atomically, verifies readback, and returns refreshed function analysis. -Changes live only in the session database and leave executable bytes unchanged. -Ghidra does not expose GUI controls; Windows P0 remains read-only. - -Windows x64 P0 supports native x86 and x86-64 PE applications on local NTFS with -bundled Job Object ownership, protected runtime DACLs, and handle-based path -admission. Real ordinary-user verification covers all 25 read-only operations -through the packaged CLI and MCP, target integrity, and cleanup. This remains -an experimental boundary; see [Windows Ghidra P0](windows-ghidra-p0.md) and -[issue #527](https://github.com/morluto/rea/issues/527). - -[ADR-0001](adr/0001-provider-selection-and-analysis-profiles.md) fixes the -provider registry, deterministic selection, target binding, analysis profile, -snapshot migration, and compatibility semantics that implementation must -follow. - -## Approved Ghidra v1 boundary - -- Keep the existing provider-neutral CLI and MCP tool names. -- Bind one deep-analysis provider to a target for the target's lifetime; never - fail over silently between Hopper and Ghidra. -- Use an existing Ghidra installation and a compatible Java runtime. Setup must - not install or upgrade Ghidra or Java. -- Run Ghidra headlessly in an owned process with a private temporary project, - startup and cleanup deadlines, caller cancellation, and complete replies. -- Prefer a packaged Java bridge loaded through Ghidra's script path. PyGhidra - remains useful for prototypes but is not a mandatory production dependency. -- Treat authenticated `ping` and `shutdown` as lifecycle proof only. Inventory, - xref, CFG, and decompilation claims require separately admitted operation - contracts and real-provider conformance. -- Implement read-only inventories, assembly, decompilation, function metadata, - calls, references, containment, and complete inventory search first. -- Report GUI cursor/navigation and persistent mutation operations as unavailable - until their semantics and project ownership are explicitly designed. -- Verify real claims on Linux with at least two distinct source-owned binaries. - Verify the Windows host claim with the deterministic source-owned x86 and x86-64 PE - fixtures and all admitted operations; compare normalized semantic facts rather - than provider-specific pseudocode text. - -## Shipped foundation boundary - -`GHIDRA_INSTALL_DIR` must identify an extracted Ghidra 12.1.x release. Optional -`JAVA_HOME` must identify a 64-bit full JDK inside that installation's -`application.java.min` and `application.java.max` (JDK 21 or newer, with no -maximum, when the release leaves those at the 12.1 defaults). Otherwise doctor -probes `java`/`javac` or `java.exe`/`javac.exe` from `PATH`. Supported Linux and macOS -hosts accept compatible ELF, PE, and Mach-O executable targets. Host admission -and target compatibility are separate checks. Windows x64 P0 admits only native, -non-managed, non-DLL x86 and x86-64 PE applications on fixed local NTFS, with the packaged -native authority available before launch. - -The launcher creates one ephemeral runtime root with project, -home/cache/config/data/temp, logs, descriptor, endpoint, target snapshot, and -ownership manifest. -It passes `-readOnly` and `-deleteProject`, and uses Ghidra's default analysis -and resource settings; inherited Java option injection variables are cleared. -On Linux, the mode-0600 descriptor carries the random token without -exposing it in argv or environment and the Java bridge binds a mode-restricted -Unix socket. macOS uses the same local transport. The bridge reports actual -Ghidra/language/compiler/analysis/import-digest metadata and accepts only -authenticated `ping`, `shutdown`, thirteen read-only inventory/load-image methods, -and twelve function-analysis methods. Linux/macOS also admits atomic function -annotation changes in the ephemeral database. Close, cancellation, timeout, malformed protocol, -or process exit stops the owned process resources, closes the socket, and -removes the runtime root. - -The experimental Windows transport uses authenticated IPv4 loopback with a -token-free endpoint record. The bundled native authority establishes Job Object -ownership, protected runtime DACLs, and handle-based path admission before -launch. If the bundle or an admission constraint is unavailable, the provider -remains unavailable; a `taskkill` fallback does not establish that authority. - -The provider catalog lists 25 read-only Ghidra operations and the Linux/macOS -session annotation operation. GUI cursor/navigation and persistent mutations -remain unavailable; Windows P0 admits no database mutation. The router therefore -reports them unavailable instead of borrowing Hopper semantics or inferring -capability from a successful import. - -## Admitted inventory semantics - -| Concern | Ghidra contract | -| ---------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| Program identity | One `analyzeHeadless` import produces exactly one Program; `list_documents` therefore returns exactly one name. | -| Addresses | Default memory uses lowercase `0x` hexadecimal. Non-default and external spaces use `:0x` and remain round-trippable. The handshake commits image base and default address-space name. | -| Symbols | `list_names` includes address-bearing memory and external symbols, including dynamic symbols, while excluding variable and no-address namespace records. Each item reports primary, dynamic, external, symbol type, and source facts. | -| Procedures | Both non-external and external functions are listed. A local thunk remains distinct from its resolved target; exact and qualified name lookup fails on ambiguity rather than guessing. | -| Strings | Only Ghidra-defined string `Data` is observed. Items report charset, byte length, and whether a required null terminator is missing. The API cannot distinguish a present terminator from fixed/Pascal layouts when no terminator is missing, so that state is named `present_or_not_required`. | -| Memory | Memory-block end addresses are exclusive. Read/write/execute, initialization, overlay, address space, and image base are direct Ghidra observations. | -| Inventory | Procedure, symbol, and string listings plus searches return the complete matching collection in one response; callers do not provide offsets or result-count limits. | -| Search | Literal and Java-regex searches scan the complete immutable inventory and return all matching entries inline. Search waits for a reply or caller cancellation; there are no caller-supplied offsets or result-count limits. | -| Analysis state | The socket is exposed only after default auto-analysis completes. Established operations wait for their reply or caller cancellation; there is no fixed response-size ceiling. | - -## Admitted function-analysis semantics - -| Concern | Ghidra contract | -| ----------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| Decompiler lifetime | One persistent `DecompInterface` is opened for the imported Program and disposed during bridge shutdown. Native decompilation has no fixed per-function deadline. External functions or functions without bodies return `null`; cancellation and native failure remain distinct. | -| Serialization | A FIFO sends one Program request at a time. Caller cancellation removes queued work promptly and is passed to active socket waits. This is an adapter safety commitment, not a claim that every Ghidra API is thread-safe. | -| Function identity | Every function result carries the entry address and Ghidra FunctionManager classification for external, thunk, and resolved thunk target. These are observations, not proof that unresolved targetless calls have been recovered. | -| Assembly and pseudocode | Assembly is complete Ghidra Listing text; pseudocode is Ghidra decompiler output. Neither is original source, and cross-provider comparison never treats Hopper and Ghidra text as equal or unequal semantic facts. | -| Instruction fast path | `read_function_instructions` returns every raw Listing instruction for the requested function without invoking the decompiler or whole-program name/string inventories. | -| Calls and references | Callers/callees contain only resolved functions. Reference edges preserve exact ReferenceManager type and call/jump/data/read/write/indirect/computed/conditional/terminal/external facts. Targetless computed flow remains unknown. Synthetic entry-point references without actionable memory sources are omitted explicitly. | -| CFG | Dossiers use `BasicBlockModel` and retain only non-call successors inside the function body. CFG topology is address-normalized for comparison; provider-specific block construction remains a declared difference. | -| P-code value flow | Function dossiers include up to 3,000 high-p-code operations, at most 64 inputs per operation, and 12,000 def-use edges. Results identify memory reads/writes, branches, and calls by p-code opcode, and mark decompiler-dead operations. Large results report truncation, an omitted-operation lower bound, and known omitted-input/edge counts. The dossier is an intra-function graph; `trace_native_values` composes derived parameter/argument and return/output dependencies across resolved calls. It does not resolve memory aliasing, call side effects, runtime behavior, persistent state, or business meaning. | -| Result extent | Function instruction scans and native API boundary observations remain complete. P-code flow is the explicit bounded exception; its count fields distinguish exact retained counts from lower-bound or known-omitted counts. Caller cancellation and provider failures remain distinct. | - -`npm run verify:ghidra` compiles the versioned C oracle into debug and stripped -host-native targets (x86-64/AArch64 ELF on Linux or Mach-O on macOS), plus a native -DWARF 4 type-layout object. It proves the admitted operations, external functions, resolved thunks, exports, stripped-name -behavior, direct and targetless indirect calls, typed references, strings/xrefs, -multi-block CFG, bounded p-code def-use/effect links, semantic enhanced workflows, cancellation, startup deadlines, -serialized concurrency, malformed-target rejection, profile identity, and -process/project cleanup against real Ghidra 12.1.4. Unit fixtures separately -cover startup deadlines, process exit, queued cancellation, and malformed wire -output. - -`npm run verify:ghidra:cross-format` adds AArch64 ELF, x86-64 PE, and x86-64 -Mach-O target coverage. It requires `clang`, LLD, and `lld-link`; set -`REA_CLANG`, `REA_LLD`, or `REA_LLD_LINK` to select them. The verifier checks -these tools before compilation. Keeping this matrix separate lets Linux -host/provider acceptance run with only the host compiler. - -`npm run verify:ghidra:windows` and its `-- --x86` variant check the source-owned native x86 and x86-64 PE -fixture, all 25 read-only operations, target/snapshot/import SHA-256 linkage, -and project, endpoint, process, and runtime cleanup. The independent native -lane checks DACLs, handle admission, cancellation, and Job Object lifecycle. -`npm run verify:ghidra:windows:package` packs and installs REA into an isolated -prefix, then checks ordinary-user CLI/MCP operations against canonical contracts. -All lanes require matching native controls; package startup alone establishes -only package compatibility. - -## Shared provider-process foundation - -`src/process/` now provides the mechanisms that a long-lived Hopper or Ghidra -adapter shares: POSIX run-token-authenticated process-group ownership, -ephemeral temporary runtime roots, one absolute startup deadline, correlated -request cancellation and lifecycle-deadline cleanup, bounded stdout and stderr retention with -exact byte counts, process-exit diagnostics, and bounded TERM-to-KILL shutdown. -Reusable fixtures exercise exit, timeout, cancellation, graceful termination, -forced termination, double-close, spawn failure, and resource release. Windows -Ghidra uses its native Job Object, private runtime, and path-admission boundary; -generic process-tree termination alone cannot establish provider availability. - -The foundation does not define a bridge schema, socket framing, health payload, -analysis model, or shutdown acknowledgement. Hopper keeps its authenticated -NDJSON-over-Unix-socket protocol in `src/hopper/`; Ghidra has a separate strict -NDJSON protocol implemented by the packaged Java `HeadlessScript` and reuses -only the generic process mechanisms. - -## Shortlist - -| Provider | License / automation surface | What it brings | REA fit and blockers | -| ------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| [Ghidra](https://github.com/NationalSecurityAgency/ghidra) | Apache-2.0 source license; `analyzeHeadless`, Java APIs, and PyGhidra | Static analysis, multiple processors and formats, scripting, and project/database workflows | 25 read-only operations shipped on Linux/macOS with installation checks, temporary projects, a serial API queue, and real ELF/PE/Mach-O conformance. Experimental Windows x64 P0 supports native x86 and x86-64 PE on local NTFS with bundled native controls and ordinary-user CLI/MCP verification; Linux/macOS additionally support atomic session function annotations; Windows mutation and all Ghidra GUI operations are unavailable. | -| [Rizin](https://github.com/rizinorg/rizin) / [rz-pipe](https://github.com/rizinorg/rz-pipe) | Rizin repository contains LGPL-3.0 and GPL-3.0 components; `rizin`, `rz-bin`, and language bridges through `rzpipe` | Portable CLI analysis, disassembly/debugging, many architectures and file formats, JSON command output | Good candidate for a process-backed Linux provider and fast metadata fallback. License/component inventory must be preserved; command output needs version-pinned parsers and semantic conformance before evidence is trusted. | -| [LIEF](https://github.com/lief-project/LIEF) | Apache-2.0; C++, Python, and other bindings | Deterministic parsing and modification of ELF, PE, Mach-O, COFF, and related executable formats; headers, sections, symbols, relocations, and functions | Best near-term complement, not a decompiler replacement. It can cover format metadata and artifact evidence without a long-lived analysis process; function semantics, pseudocode, CFG, and cross-reference parity remain out of scope unless separately demonstrated. | -| [Binary Ninja](https://docs.binary.ninja/dev/index.html) | API/documentation components are MIT, while the analysis product is licensed by edition; commercial, Ultimate, or Headless license is required for headless automation | Python/Core/C++/Rust APIs, headless loading, IL layers, function analysis, plugins, and configurable analysis | Strong technical fit for a native provider, especially function dossiers. Commercial licensing, license-secret handling, native runtime packaging, and multithreaded lifecycle rules are material deployment blockers. | - -## Recommended order - -1. Use the implemented explicit provider registry and target binding without - changing Hopper behavior. -2. Maintain the admitted Ghidra function boundary through the shared - conformance corpus and add formats or semantics only after real proof. -3. Connect Electron/native-add-on application findings to the selected native - analysis provider without introducing provider-prefixed tools. -4. Evaluate LIEF or Rizin later as complementary metadata/disassembly providers, - and Binary Ninja as an optional licensed provider, using the same admission - gate. - -## Required admission gate - -Before implementation, an adapter proposal must provide: - -- a capability matrix covering supported, unsupported, and degraded results; -- provider identity, version, analysis profile, target digest, authority, - limitations, and deterministic locations in every Evidence record; -- bounded subprocess or library lifetime, cancellation, timeouts, and cleanup; -- actionable local diagnostics that retain paths, digests, mismatch locations, - and provider metadata while redacting credentials, authorization headers, - license secrets, and other genuine secrets; -- the same source-owned fixture corpus across architectures and formats; -- repeatable checks for function identity, addresses, strings, names, xrefs, - CFG, and pseudocode wherever those capabilities are claimed. - -These gates turn the accepted direction into verified capabilities rather than -prematurely claiming equivalence or redistribution support. - -## Primary-source notes - -- Ghidra 12.1.4 documents headless batch mode and its Java runtime requirements - in its [release-specific Getting Started guide](https://github.com/NationalSecurityAgency/ghidra/blob/Ghidra_12.1.4_build/GhidraDocs/GettingStarted.md); - the [official release](https://github.com/NationalSecurityAgency/ghidra/releases/tag/Ghidra_12.1.4_build) - supplies the corresponding distribution and checksums. -- Rizin lists supported formats, architectures, tools, and `rzpipe` bridges in - its [repository README](https://github.com/rizinorg/rizin); `rz-pipe` documents - JSON command transport and pipe backends. -- LIEF's [format tutorial](https://lief.re/doc/stable/tutorials/01_play_with_formats.html) - and [Python API](https://lief.re/doc/latest/api/binary_abstraction/python.html) - document parsing and format-agnostic binary access. -- Binary Ninja documents its [headless automation model](https://docs.binary.ninja/dev/batch.html), - [edition licenses](https://docs.binary.ninja/about/license.html), and the - [MIT license for API/documentation components](https://docs.binary.ninja/about/open-source.html). diff --git a/docs/roadmap.md b/docs/roadmap.md index 0df3a67c0..75d9f4679 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -1,125 +1,35 @@ # REA roadmap -## Tool direction +The [platform tracker](https://github.com/morluto/rea/issues/32), +[process/Hopper tracker](https://github.com/morluto/rea/issues/48), and +[browser tracker](https://github.com/morluto/rea/issues/39) track capability and +verification work. Available tools and setup are documented in the +[investigation guides](https://rea.tools/guides/), +[installation guide](installation.md), and +[generated catalog](mcp-contracts.md#generated-catalog). -REA provides inspection, capture, and comparison tools that return evidence -directly. Agents choose commands, paths, browser actions, scripts, and local -fixture servers, then compose experiments from those results. CLI and MCP must -expose equivalent behavior and preserve observations, inferences, and unknowns. +## Priorities -[#555](https://github.com/morluto/rea/pull/555) removed REA permission grants, -scope ceilings, elicitation, and repeated approval fields. -[#572](https://github.com/morluto/rea/pull/572) removed the replay engines, -Node characterization prepare/execute flow, and plan-only managed runtime -correlation tool. Those systems are retired roadmap items. Existing evidence -authority labels remain readable as provenance; they do not imply a current -replay executor. +- Expand native architecture, type, and indirect-call verification. +- Connect static findings and runtime observations across application layers. +- Improve obfuscated managed-code comparisons and verified native links. +- Extend process, protocol, filesystem, reconnect, and build-comparison coverage. +- Evaluate additional providers and targets when an analyst workflow demonstrates + a capability gap. -Local operations use the current user's OS permissions. Input and protocol -validation, provider prerequisites, target identity, cancellation, and owned -resource cleanup remain part of each tool's contract. Setup still discloses -installation changes and requires approval before writing or installing. +## Admission criteria -## Planned work +New capabilities must return useful evidence through shared CLI/MCP workflows, +preserve observations and unknowns, and declare actual authority and lifecycle +constraints. Agents compose experiments using ordinary commands, scripts, and +fixture servers; custom orchestration needs a demonstrated requirement. -The [website guides](https://rea.tools/guides/) and -[generated catalog](product-catalog.json) describe available workflows. -Current development priorities are: +Provider and platform claims require their corresponding +[real verification lanes](testing.md#real-toolchain-verification-lanes). +See [tool design](tool-design.md) for choosing a tool boundary and +[Ghidra semantics](ghidra-provider.md) for the existing deep-provider contract. -- Keep generated metadata and narrative documentation aligned with tools, - providers, setup options and releases. -- Expand native architecture, type and indirect-call verification across - Hopper and Ghidra. -- Connect additional static extractors and runtime observations to - cross-layer feature traces. -- Improve obfuscated .NET comparisons and links between managed findings - and verified native analysis. -- Extend process, protocol, filesystem, reconnect and build-comparison - coverage, plus browser and Electron scenario actions. -- Evaluate native runtime observation through LLDB, Frida, system logs and - API tracing, and additional tools and targets such as Binary Ninja, Rizin, - LIEF, Windows-native workflows, mobile applications and firmware. - -Provider additions and platform support require the corresponding real -verification lanes; see [testing](testing.md) and -[provider evaluation](provider-evaluation.md). - -## Remaining evidence and provider work - -The [platform roadmap](https://github.com/morluto/rea/issues/32), -[process and Hopper fidelity tracker](https://github.com/morluto/rea/issues/48), -and [browser tracker](https://github.com/morluto/rea/issues/39) track remaining -capabilities and proof. Their scope follows direct tools and agent-composed -experiments; custom replay languages and grant systems are not completion gates. - -[Inline Evidence composition](https://github.com/morluto/rea/pull/585) now returns -the complete record accepted by comparison tools. Managed inspection uses an -explicit PE/CLI target independently of the native provider selection. Comparisons -preserve unknowns when observation is incomplete, and PTY diagnostics retain the -actual loader or startup failure. Browser cleanup reports incomplete teardown -alongside the primary failure; transient documents invalidate only the affected -frame's WebMCP registrations. Real-provider and platform claims still require -their corresponding verification lanes. - -## Shipped behavior - -REA setup lets you select agent integration and optional Hopper installation. -It installs the bundled workflow, configures detected agents, and can save -verified paths for an existing Ghidra installation. It configures Claude Code, -Claude Desktop, Codex, Cursor, Gemini CLI, Windsurf, Devin, OpenCode, Antigravity, -GitHub Copilot CLI, Command Code, VS Code, Grok Build, and OMP using each client's configuration format. Grok Bot is detected locally, but its connector is added from the Grok Bot chat because the signed-in account store is not a data-directory file. - -Ghidra analysis supports Linux x64/arm64 and macOS x64/arm64 with Ghidra 12.1.x and the -64-bit full JDK that installation declares. Current 12.1 releases require JDK 21 -or newer and set no maximum; the bridge is verified with Ghidra 12.1.4 and JDK 21. -Each installation requires the matching native decompiler. The -adapter exposes thirteen inventory operations and twelve function-analysis -operations, for 25 read-only operations total. Linux and macOS additionally support -atomic function-name and entry-comment edits with refreshed analysis; metadata -is discarded on close and executable bytes stay unchanged. Approved setup saves verified -installation paths in agent configurations without installing or changing -Ghidra or Java. - -Real-provider verification uses host-native debug and stripped fixtures plus -a native type-layout object. A separate lane covers AArch64 ELF, PE, and -Mach-O cross-target fixtures. See [testing](testing.md) for the prerequisites -and exact commands. - -Experimental Windows x64 P0 now supplies bundled Job Object ownership, -protected runtime DACLs, and handle-based admission for native x86 and x86-64 PE -applications on local NTFS. Real ordinary-user CLI/MCP verification covers -all 25 read-only operations and cleanup. The [Windows P0 guide](windows-ghidra-p0.md) -describes this boundary and unverified broader coverage. - -## Ghidra maintenance boundary - -Extend formats or semantics only with normalized provider-neutral contracts and -real source-owned conformance. Hopper and Ghidra pseudocode or assembly text is -not expected to match; unresolved targetless flow remains unknown. Automatic -Ghidra acquisition, if ever added, remains a separately planned and approved -related-tool change; setup must never install Java. - -Maintain Windows P0 with independent DACL readback, handle-based admission, -private authenticated IPC, and Job Object assignment before provider execution. -Process capture, Hopper, and broad filesystem-sensitive workflows remain -separate Windows projects; P0 does not imply their parity. - -## Capability-selective setup - -Setup already lets users select agent integration and Hopper installation. -Once REA supports installing additional analysis tools, it can expand these -choices to match the work a user wants to do. It may ask whether the operator wants to investigate -native binaries, websites, mobile applications, firmware, or runtime protocols, -then propose only the tools needed for the selected work. - -That future installer must preserve the current safety boundary: - -- detect and reuse existing tools; -- disclose every source, destination, license, command, and system effect; -- prefer user-local installation; -- install only explicitly selected toolchains; -- require tool-specific authorization for unattended system changes; -- verify each installed tool before reporting readiness. - -No placeholder tool choices or speculative installer registry are implemented -until another supported toolchain makes the abstraction concrete. +Setup remains additive, reuses existing tools, and requires approval for disclosed +configuration and Hopper installation changes. Additional installer choices need +an implemented toolchain and explicit scope; setup does not install or upgrade +unrelated runtimes or dependencies.