morluto/rea
1
0
Fork 0
mirror of https://github.com/morluto/rea.git synced 2026-10-11 21:39:55 +00:00
Code Issues Packages Projects Releases Wiki Activity

docs: remove retired architecture and migration artifacts

Browse Source
This commit is contained in:
morluto
2026-10-10 05:41:45 +08:00
parent 7738c1f6a9
commit 7a0c800851
12 changed files with 153 additions and 2633 deletions
Download Patch File Download Diff File Expand all files Collapse all files
CONTRIBUTING.md
+6 -3
View File
@@ -10,9 +10,12 @@ and maintenance cost. Keep cosmetic polish with useful fixes; confusing or
incorrect user instructions warrant standalone corrections. Collect other
cosmetic-only suggestions in an issue rather than separate PRs.
For capability organization and provider composition, follow the incremental
[migration guide](docs/capability-migration.md). Run `npm run verify:test-discovery`
after adding or moving tests.
For capability organization and provider composition, use the
[architecture map](docs/architecture.mermaid). Production factories in
`src/composition/` construct fresh providers without starting engines or acquiring
targets; optional adapter failures must preserve successful peers. Run
`npm run verify:test-discovery` after adding or moving tests. Resolved import
boundaries are checked by `npm run verify:module-boundaries`, included in lint.
When adding or changing an MCP tool, follow the [tool design guide](docs/tool-design.md) and preserve the canonical contracts and generated catalog.
docs/.vitepress/config.ts
+1 -2
View File
@@ -55,7 +55,7 @@ export default defineConfig({
{ text: "Ghidra on Windows", link: "/windows-ghidra-p0" },
{ text: "Ghidra DOS analysis", link: "/ghidra-dos" },
{ text: "Ghidra NativeAOT", link: "/ghidra-nativeaot" },
{ text: "Provider evaluation", link: "/provider-evaluation" },
{ text: "Ghidra semantics", link: "/ghidra-provider" },
],
},
{
@@ -66,7 +66,6 @@ export default defineConfig({
{ text: "Tool design", link: "/tool-design" },
{ text: "Testing", link: "/testing" },
{ text: "Releasing", link: "/releasing" },
{ text: "Architecture decisions", link: "/adr/README" },
],
},
],
docs/adr/0001-provider-selection-and-analysis-profiles.md
-572
View File
@@ -1,572 +0,0 @@
# ADR-0001: Provider selection and analysis profiles
- Status: Accepted
- Date: 2026-07-15
- Implementation status: The provider registry, deterministic selection,
target binding, analysis-profile commitment, and snapshot/Evidence migration
are implemented. Ghidra discovery, target/profile resolution, doctor checks,
the private headless-session foundation, thirteen read-only inventory/load-image
capabilities, and twelve function-analysis capabilities are implemented with
real cross-format conformance. Linux/macOS also supports atomic session
function annotations; experimental Windows x64 P0 uses bundled native controls
and admits only the 25 read-only operations. See the
[release boundary](../installation.md#released-package-and-main).
## Context
Before this decision, REA composed the artifact, native macOS, and Hopper
providers with `CompositeProvider`. That class intentionally requires every
operation to have one route. Hopper and Ghidra will both implement the deep
static-analysis operation family, so putting both into that composite would
either fail at construction or make array order an undeclared selection policy.
The active `BinarySession` also owned one composite client. `open_binary` had no
provider selector, and the domain `BinaryTarget` contained Hopper loader
arguments. Analysis snapshot v1 committed those loader arguments and each
cached entry's provider identity, but it did not commit the selected analysis
engine, engine version, or complete analysis configuration. Two analyses of the
same bytes could therefore disagree because of language, compiler, loader,
analyzer, or provider-version differences while appearing target-compatible.
The decision addressed the earlier representations of
[`CompositeProvider`](https://github.com/morluto/rea/blob/main/src/application/binary/CompositeProvider.ts),
[runtime composition](https://github.com/morluto/rea/blob/main/src/composition/binary.ts),
[`BinarySession`](https://github.com/morluto/rea/blob/main/src/application/binary/BinarySession.ts),
[`BinaryTarget`](https://github.com/morluto/rea/blob/main/src/domain/binaryTarget.ts),
[analysis snapshot](https://github.com/morluto/rea/blob/main/src/domain/analysisSnapshot.ts), and
[session lifecycle inputs](https://github.com/morluto/rea/blob/main/src/contracts/sessionLifecycleInputs.ts).
The next deep provider must not weaken the existing product invariants:
- stable provider-neutral CLI and MCP tool names;
- exact provenance and explicit unavailable or degraded capabilities;
- CLI and MCP parity;
- no silent provider fallback;
- provider-specific code outside the domain and application models;
- local, actionable diagnostics with only genuine secrets redacted; and
- snapshot reuse only when semantic analysis inputs are identical.
## Decision drivers
1. Hopper and Ghidra need to declare overlapping operations without ambiguous
dispatch.
2. A target must have one explainable deep-analysis producer for its lifetime.
3. Target-free discovery must remain useful and must not start an analysis
process.
4. Existing Hopper-only configurations must keep working when no selector is
supplied.
5. Adding a second installed provider must not silently change results.
6. Snapshot and Evidence compatibility must include every setting that can
materially change analysis.
7. Provider-specific open settings must not leak into `BinaryTarget` or become
generic application fields.
## Terminology
- **Deep provider**: an engine such as Hopper or Ghidra that implements the
overlapping static-analysis operation family.
- **Auxiliary provider**: a provider with a disjoint operation family, such as
artifact inventory or native macOS inspection.
- **Candidate**: a registered deep provider before target selection.
- **Binding**: the selected deep provider and analysis profile attached to one
active target.
- **Analysis profile**: a canonical commitment to the selected provider,
provider version, and normalized semantic analysis settings.
- **Available**: the adapter and its required local runtime can be used on this
host.
- **Supported**: the candidate accepts the parsed target and supplied open
options. Availability and target support are separate observations.
## Decision
### 1. Separate disjoint composition from overlapping selection
`CompositeProvider` remains the mechanism for operation families whose routes
are disjoint. It must continue rejecting duplicate operation declarations. It
must not acquire priority, racing, fallback, or provider-selection behavior.
A new `AnalysisProviderRegistry` owns the configured deep-provider candidates.
Provider IDs are unique; duplicate IDs are a configuration error. The registry
may hold multiple candidates that declare the same operation. It performs
candidate discovery, target-support evaluation, option parsing, profile
resolution, and selection, but it does not choose a different provider for each
request.
After selection, the session composes exactly one bound deep provider with the
auxiliary disjoint families:
```text
target-free discovery
|
AnalysisProviderRegistry
/ \
Hopper Ghidra
\ /
one selected binding
|
+-------------------------+-------------------------+
| | |
selected deep family artifact family native family
+-------------------------+-------------------------+
|
BinarySession
```
An auxiliary operation continues to report its actual auxiliary provider as
provenance. A deep operation always reports the provider in the active binding.
The synthetic composite identity is never evidence that two engines jointly
produced one observation.
Operation ownership is fixed by family:
| Operation family | Executor |
| ------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------- |
| Deep direct operations and provider-native function analysis | The one bound Hopper or Ghidra provider |
| Application-enhanced deep workflows | Application composition over the bound provider's declared operations |
| Native macOS inspection | The disjoint native provider |
| Artifact inventory and extraction | The disjoint artifact provider |
| Browser and Electron observation | Their independently authorized CDP providers; they do not participate in deep-provider selection |
| Session lifecycle, Evidence, comparisons, and reconstruction | The application/session layer using the exact provider provenance recorded by their source operations |
Selecting a deep provider selects the whole deep family. If that provider marks
an operation unavailable, REA returns the unavailability; it does not borrow
that one operation from another deep provider. Application-enhanced workflows
run only when the bound provider supplies their declared dependencies.
### 2. Discover candidates without launching analysis
Each registered candidate exposes three bounded adapter-owned observations:
1. identity and declared capability descriptors;
2. host availability, including an actionable rejection when unavailable; and
3. target support plus normalized profile resolution when a target is known.
Target-free discovery may inspect configuration, manifests, executable
metadata, and documented version files. It must not import a target, create a
project, open a GUI, or start the long-lived analysis process. Any short-lived
version probe added later must be explicitly bounded and truthfully declared as
a process effect.
Candidate output is sorted by provider ID for determinism. Sort order is not
selection priority. Availability and support use stable codes plus a human
diagnostic:
| Code | Meaning |
| ---------------------------- | --------------------------------------------------------------- |
| `not_configured` | Required provider location or configuration is absent |
| `executable_missing` | The configured provider entry point is missing or not runnable |
| `runtime_missing` | A required provider runtime, such as Java, is unavailable |
| `unsupported_host` | The host platform or architecture is unsupported |
| `unsupported_version` | The discovered provider or runtime version is unsupported |
| `target_kind_unsupported` | The provider does not accept this target kind |
| `target_format_unsupported` | The provider does not accept this target format |
| `architecture_unsupported` | The selected target architecture is unsupported |
| `target_role_unsupported` | The provider rejects this executable image role |
| `managed_target_unsupported` | The provider does not admit managed code through this deep path |
| `open_options_invalid` | The selected adapter rejected its open options |
| `version_unresolved` | A concrete analysis-engine version could not be committed |
Diagnostics retain useful local paths, observed versions, architectures, and
mismatch details. Credentials, authorization headers, license secrets, and
other genuine secrets remain redacted.
### 3. Use one deterministic selection algorithm
The caller-visible selectors are:
- MCP `open_binary.provider_id`;
- `rea ... --provider <provider-id|auto>` on every one-shot command that needs
deep analysis; and
- `REA_ANALYSIS_PROVIDER=<provider-id|auto>`, defaulting to `auto`.
The literal value `auto` is reserved and cannot be a provider ID. Precedence is
request or CLI flag, then environment configuration, then the default `auto`.
An explicit request for `auto` overrides a concrete environment preference and
re-enters automatic selection.
Selection occurs after provider-neutral target parsing and before a deep client
is created:
1. A concrete requested provider ID must exist, be available, support the
target, accept its options, and resolve a concrete version and profile. Any
failure returns a typed selection error; another provider is never tried.
2. `auto` filters candidates by availability and target support.
3. Exactly one usable candidate is selected with source
`auto-single-candidate`.
4. More than one usable candidate is an `ambiguous` selection error. The
diagnostic lists the candidates and asks for `provider_id`, `--provider`, or
`REA_ANALYSIS_PROVIDER`. REA does not prefer Hopper, Ghidra, registration
order, the fastest startup, or the first health response.
5. When no deep candidate is usable, an automatic open may still establish an
unbound target so disjoint artifact or native operations remain available.
Deep operations are then explicitly unavailable with the candidate
rejections. A concrete request, or a concrete environment preference for a
deep-eligible target, remains a hard requirement and fails instead of
becoming unbound.
6. A target that no deep adapter declares support for, such as an artifact-only
target, is `not_applicable` rather than a failed deep analysis. Supplying a
concrete provider in that open request is still an actionable unsupported-
target error. A global environment preference does not make an artifact-only
target require a deep provider.
This preserves target opening for non-Hopper capabilities on hosts without a
deep engine while ensuring that installing a second usable deep engine never
changes analysis silently.
### 4. Bind once and never fail over mid-session
An active deep binding contains:
```text
canonical target identity
selected provider identity and concrete version
selection source: request | environment | auto-single-candidate
analysis profile commitment
owned provider client, once started
```
The binding is immutable until `close_binary` or an explicit open/switch
transaction. Provider client creation may remain lazy, but laziness cannot
change which provider was selected.
- Opening the same canonical target without a new selector keeps the existing
binding.
- Opening the same target with the same provider and profile is idempotent.
- A different explicit provider or a profile-changing option is a real switch:
drain active calls, close owned resources, resolve the new binding, and start
it as one serialized transaction.
- If a switch fails, restore the prior target and binding on a best-effort
transactional path and return the original switch error.
- Provider process exit, timeout, cancellation, capability failure, or adapter
error never triggers fallback. The binding stays visible and unavailable
until the caller explicitly switches or closes it.
- A snapshot never chooses or changes the provider. Selection happens first;
snapshot compatibility is checked against the resulting binding.
Calls already admitted to one binding complete or cancel against that binding.
No request can cross a concurrent target or provider transition.
### 5. Keep provider open options adapter-owned
`BinaryTarget` retains only provider-neutral source identity and classification:
canonical path, source digest, generic kind and format, selected architecture,
and available architectures. Hopper loader arguments leave the domain model.
Provider-owned databases use a generic analysis-database classification; their
adapter determines support. A `.hop` analysis database is therefore accepted only
by Hopper without teaching the domain how to open it. Ghidra v1 continues to use
private temporary projects and does not open user projects.
The generic transport may carry one bounded JSON `provider_options` object.
Only the selected adapter parses its keys and reports `open_options_invalid` at
that boundary. If this object is exposed over MCP, the CLI parity surface is
`--provider-options-json` with the same JSON schema and limits. Unknown keys,
wrong-provider options, excessive depth or size, and non-JSON values fail
closed.
Environment settings that are inherently adapter-specific, such as the Hopper
launcher or future Ghidra installation path, remain in adapter configuration.
They do not become `BinaryTarget` fields. The application layer transports an
opaque, already-bounded JSON value and the generic commitment described below;
it does not interpret Hopper or Ghidra option names.
### 6. Commit a canonical analysis profile
Before creating a deep client, the selected adapter normalizes every semantic
input that can affect analysis. Examples include Hopper loader selection and
architecture flags, or Ghidra language ID, compiler specification, import mode,
analyzer set, and analyzer options. Defaults are made explicit, unordered sets
are sorted, and implicit host-dependent choices are resolved.
The adapter returns this provider-neutral envelope:
```json
{
"provider": { "id": "ghidra", "version": "<concrete version>" },
"parameters": {},
"digest": "<sha256 of RFC 8785 canonical JSON excluding digest>"
}
```
`parameters` is bounded JSON interpreted only by that provider adapter. It
contains analysis semantics, not credentials, authorization data, temporary
paths, installation paths, process IDs, timestamps, or other incidental host
state. The generic domain validates the envelope and digest but treats the
parameters as opaque data.
A selected deep binding requires a concrete provider version. If the adapter
cannot determine it, the candidate is unavailable with `version_unresolved`;
REA does not persist a cache that merely commits `null` or "whatever is
installed". Provider identity, normalized parameters, or provider version
changes produce a different digest.
The same target, configuration, and provider must resolve the same profile over
CLI and MCP. Profile resolution must be side-effect-free and complete before
the provider imports the target.
### 7. Make snapshots provider- and profile-exact
Analysis snapshots replace Hopper loader arguments with an explicit binding
commitment:
```text
target: digest + generic format/kind + selected architecture
binding: provider identity + analysis profile commitment
entries: exact immutable queries
evidence_bundle: retained Evidence records
```
The snapshot target identity, canonical snapshot content digest, and every
query ID commit the provider ID, concrete provider version, and profile digest.
A cached entry is reusable only when all of the following match exactly:
- target digest, generic format and kind, and selected architecture;
- provider ID and provider version;
- profile digest;
- operation and canonical parameters; and
- the existing cacheability and immutability rules.
Provider or profile switches select a separate cache partition and cannot reuse
entries from the previous binding. Auxiliary operations are cacheable only if
their provider defines an equivalent explicit profile commitment; otherwise
they remain Evidence but not snapshot entries.
Snapshot v1 cannot be parsed as the current snapshot format. Its `loader_args`
cannot prove which engine version, defaults, or analyzer settings produced the
entries, so snapshots must be recaptured before they can be used as a cache.
An embedded Evidence bundle can still be imported separately through the
Evidence import flow; it is never promoted into an analysis cache.
### 8. Commit concrete or explicitly unknown analysis profiles in Evidence
Every Evidence record includes `analysis_profile`, either the canonical profile
envelope above or `null` when no profile was observed. Deep-analysis observations
commit the resolved provider ID, version, and effective configuration in this
profile. The field participates in the Evidence semantic ID.
An explicitly unknown profile can be displayed and cited, but cannot establish
profile compatibility, seed snapshot replay, or support an "unchanged"
conclusion that depends on identical analysis semantics. Snapshot query entries
must have matching profile-bound Evidence; imports reject missing or mismatched
observations instead of silently dropping query entries. Both `entries` and
`workflow_entries` are required arrays, including when empty.
Different providers or profile digests are never snapshot-compatible. A
derived comparison may compare cross-provider normalized facts only when its
own contract explicitly defines that semantic comparison. Exact pseudocode,
provider text, or provider-specific metadata is not treated as equivalent. A
missing or incompatible profile yields unknown or incompatible, never equal.
### 9. Expose selection truth in status and diagnostics
`binary_session`, `rea providers`, and `rea capabilities` expose the same
candidate and binding model. The target-free form does not start providers and
uses `unknown` for target support:
```json
{
"analysis_provider_binding": null,
"analysis_provider_candidates": [
{
"provider": {
"id": "hopper",
"name": "Hopper Disassembler",
"version": "<detected version>"
},
"availability": {
"status": "available",
"code": null,
"reason": null,
"diagnostics": {}
},
"target_support": {
"status": "unknown",
"code": null,
"reason": "No target is open."
},
"selected": false,
"capabilities": []
}
]
}
```
With a binding, `analysis_provider_binding` includes the selected identity,
selection source, and complete profile commitment. Every candidate reports
`supported`, `unsupported`, or `unknown` target support and whether it was
selected. Candidate capability descriptors preserve supported, unsupported,
and degraded truth independently.
An open target also exposes `analysis_run`. Its run ID is allocated by the
session before client construction and propagated into selected Hopper or
Ghidra bridge handshake and process environment. Dynamic clients retain
provider-attributed lineage snapshots only after revalidating launcher
parent/command and every observed launcher or descendant run token. `not_observed`
means no dynamic provider has supplied a snapshot. Once one or more start,
`snapshots` preserves every provider observation; each remains explicitly
`unavailable` or `verified` and records its `observed_at` time. Retained
snapshots are not live inventories, and a point-in-time observation never proves
historical absence.
Status exposes `analysis_provider_candidates` for configured deep providers and
`analysis_provider_binding` for the selected concrete provider and profile.
`capabilities` retains effective operation availability and exact provider
provenance, including auxiliary operation families. There are no synthetic
flat `provider` or `providers` aliases. Provider prompt completion uses the
actual selectable candidates.
Selection failures extend the tagged `ProviderSelectionError` with a stable
reason:
- `unknown_provider`;
- `provider_unavailable`;
- `target_unsupported`;
- `ambiguous`;
- `invalid_options`.
The error includes the requested ID or `auto`, candidate IDs, rejection codes,
and actionable local diagnostics. `AnalysisCapabilityUnavailableError` remains
the error for an operation the selected provider cannot supply.
`ProviderAdapterError` and more specific provider errors remain failures of the
already selected provider. None authorizes fallback.
A snapshot whose provider or profile differs from the selected binding returns
`EvidenceIntegrityError` with stable `profile_mismatch` details. It is an
integrity incompatibility after selection, not permission to select the
snapshot's provider.
### 10. Preserve CLI and MCP parity
- `open_binary.provider_id` and one-shot `--provider` use the same parser and
selection service.
- `provider_id: "auto"`, `--provider auto`, and
`REA_ANALYSIS_PROVIDER=auto` have identical auto semantics.
- Startup targets opened from environment configuration use the same service.
- `rea providers`, `rea capabilities`, and target-free `binary_session` use the
same candidate snapshot and reason codes.
- Guided prompt `provider_id` context is a preference to forward into the open
operation; it does not create a hidden routing channel.
- Tool names and normalized output contracts remain provider-neutral. REA does
not add `ghidra_decompile` or `hopper_decompile` tools.
## Provider and snapshot representations
| Behavior or data | Contract |
| -------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------- |
| No selector, one usable deep provider | `auto` selects the sole candidate |
| No usable deep provider | Automatic open may remain unbound for auxiliary operations; deep capabilities explain their unavailability |
| Several usable deep providers without a preference | Open fails with `ambiguous`; installation order never becomes policy |
| Hopper environment settings | Adapter configuration committed into the typed analysis profile, including effective loader arguments |
| `open_binary` output | Provider-neutral target identity only; no `loaderArgs` projection |
| `.hop` input | Analysis database accepted only by Hopper |
| Snapshots | Concrete provider/profile binding, required primitive and workflow entry arrays, and matching authenticated Evidence |
| Evidence without an observed profile | Explicit `analysis_profile: null`; ineligible for profile-dependent replay |
| Snapshot save receipt | `primitive_entries`, `workflow_entries`, and `evidence_records` counts without a duplicate `entries` alias |
| Provider failure after selection | Returned from that provider, without transparent retry through another engine |
## Required behavior scenarios
| Scenario | Result |
| ---------------------------------------------------------- | --------------------------------------------------------------------------- |
| Hopper is the only usable candidate, selector omitted | Hopper is bound by `auto-single-candidate` |
| Ghidra is the only usable candidate, selector omitted | Ghidra is bound by `auto-single-candidate` |
| Both are usable, selector omitted | Typed `ambiguous` error; neither provider starts |
| Environment selects Ghidra, request omits a selector | Ghidra is required and bound or the open fails with its rejection |
| Environment selects Ghidra, request selects Hopper | Request wins; Hopper is required |
| Environment selects Ghidra, request selects `auto` | Auto policy runs and is ambiguous when both are usable |
| Unknown concrete provider ID | Typed `unknown_provider`; no target transition |
| Selected provider rejects the target or options | Typed rejection from that candidate only |
| Selected provider times out after binding | Binding remains selected and unavailable; no fallback |
| Same target is reopened without a selector | Existing binding remains unchanged |
| Same bytes are opened with a different profile | Real provider switch and a separate snapshot partition |
| Snapshot provider or profile differs from the binding | Typed `EvidenceIntegrityError` with `profile_mismatch`; nothing is imported |
| Artifact-only target has no deep candidate | Target opens unbound; disjoint artifact capabilities remain usable |
| Environment prefers Ghidra for an artifact-only target | Deep selection is `not_applicable`; auxiliary operations remain usable |
| Concrete deep provider is requested for artifact-only data | Typed `target_unsupported`; the request is not silently ignored |
## Consequences
Positive consequences:
- Hopper and Ghidra can coexist without ambiguous operation routes.
- Provider choice is reproducible, visible, and stable for a target.
- Installation of another provider cannot silently change analysis.
- Snapshot and Evidence identities become analysis-configuration aware.
- Provider adapters retain ownership of their option schemas and diagnostics.
- Auxiliary capabilities remain usable when no deep engine is installed.
Costs and tradeoffs:
- Multi-provider hosts must choose when more than one candidate is usable.
- Snapshots must be recaptured when their provider or profile binding changes.
- Providers must expose deterministic availability, support, version, and
profile resolution before they can be admitted.
- Status and error contracts gain additive structured metadata.
- Hopper needs an adapter-owned version/profile implementation before profile-
exact snapshot persistence is complete.
## Rejected alternatives
### Let `CompositeProvider` choose the first duplicate route
Array order is not a product policy, becomes fragile during wiring changes, and
hides ambiguity from callers.
### Race providers and keep the first healthy response
Machine load would determine provenance and results. It also starts resources
the operator did not select and makes cancellation and cleanup harder.
### Fall back after a provider error
Mid-session fallback can combine incompatible engine state and evidence under
one apparent target. Explicit close or switch is required instead.
### Add provider-prefixed tools
Parallel Hopper and Ghidra tool families would leak adapters into the product
contract and prevent shared workflows from using capability metadata.
### Store raw provider options in `BinaryTarget`
That repeats the current Hopper coupling and makes every generic consumer learn
provider schemas. Only canonical profile commitments cross the adapter boundary.
### Treat matching target bytes and provider ID as sufficient for snapshots
Provider versions, analyzer defaults, loaders, languages, and compiler models
can change results without changing either value.
### Silently upgrade snapshot v1
The missing engine version and analysis settings cannot be reconstructed from
Hopper loader arguments. A fabricated profile would turn unknown provenance
into false certainty.
## Implementation progress
The first six implementation stages are shipped:
1. Provider-specific target state was replaced by generic profile and snapshot
v2 commitments while preserving Hopper execution behavior.
2. The registry, selectors, candidate status, binding lifecycle, and
overlapping-provider production seams were added.
3. Proven process-lifecycle primitives were extracted for shared use.
4. Ghidra availability, target/profile resolution, doctor checks, and a private
read-only headless session were admitted without declaring operations.
5. Program identity, procedures, strings, symbols, memory blocks,
address/name resolution, containing-procedure resolution, and search
were admitted with exact wire schemas and real debug/stripped ELF
conformance.
6. Function metadata, persistent decompilation, assembly, resolved
callers/callees, typed references, xrefs, CFG, and complete function dossiers
were admitted. A serial per-Program queue keeps API access ordered without a
fixed queue length or per-operation deadline. Real conformance covers
x86-64 and AArch64 ELF, PE, Mach-O, stripped symbols, targetless indirect
calls, cancellation, and cleanup.
Future stages may deepen format and indirect-flow coverage, but must continue
to compare normalized semantics rather than provider-specific pseudocode or
assembly text. Linux and macOS now admit atomic function-name and entry-comment
edits in the ephemeral database, with readback, rollback, and cache invalidation.
Executable bytes remain unchanged. Windows P0 retains its read-only boundary;
GUI and other unimplemented mutations remain unavailable.
docs/adr/0002-controlled-replay-authority-and-sandbox.md
-551
View File
@@ -1,551 +0,0 @@
# ADR-0002: Controlled JavaScript replay authority and sandbox policy
- Status: Superseded; the controlled JavaScript replay tool was removed
- Date: 2026-07-16
- Implementation status: The implementation was removed. The design below is
retained as historical context and does not describe a supported REA tool.
> Historical design note: The tool, sandbox backend, and plan/execute workflow
> described here are no longer part of REA. For runtime behavior, run probes
> directly against the application and use the available browser, Electron, or
> process capture workflows.
## Context
REA can reconstruct JavaScript and Electron artifacts, observe an existing
browser or Electron page through passive CDP, capture an explicitly approved
process scenario, and correlate those observations without executing bundle
code in the REA server. The next research workflow needs to execute selected,
extracted JavaScript modules with controlled inputs and deterministic stubs so
that parsers, sanitizers, serializers, and similar pure logic can be compared
across versions.
That workflow crosses a qualitatively different boundary. Source recovered
from an application is untrusted code. It can intentionally or accidentally
read files, contact services, spawn processes, exhaust resources, corrupt the
analysis process, forge protocol output, or interact with another same-user
process. A JavaScript realm or Node.js `vm` context is useful for constructing
an API surface, but it is not a security boundary. Node.js also documents its
permission model as protection against accidental access by trusted code, not
as containment for malicious code.
Existing authorities cannot be widened to cover this behavior:
- `browser_observe` and `electron_observe` authorize passive attachment to an
already-running, operator-owned target. They do not authorize evaluation,
navigation, input, or target lifecycle changes.
- `process_capture` authorizes one declared host process scenario and explicitly
states that it is not a sandbox. It cannot silently become permission to run
recovered code.
- static artifact reads authorize inspection of bytes, not execution of those
bytes.
- the Evidence authority `controlled-replay` already describes observations
from an approved process experiment. Reusing that epistemic authority is
correct, but reusing the process permission capability is not.
The implementation following this ADR must fail closed on hosts where REA
cannot establish the promised operating-system boundary. It must not copy the
research `vm.runInContext` scripts into the production server and call that
isolation.
## Decision drivers
1. Passive observation must never imply authority to execute application code.
2. The exact code, inputs, stubs, runtime, sandbox policy, and limits approved
by the operator must be the ones executed.
3. Recovered code must not run in the MCP server, CLI adapter, or any provider
process.
4. A sandbox failure must make the capability unavailable, not select a weaker
execution mode.
5. The default policy must provide no host network and no writable host path.
6. Time, memory, process count, temporary storage, protocol, and output must be
bounded independently.
7. Results must distinguish an observed return, an observed exception or
termination, a derived comparison, and an unresolved application-level
claim.
8. A reproducer must commit enough provenance to detect every relevant change
without committing proprietary application source to the repository.
9. CLI and MCP must expose the same plan, approval, execution, cancellation,
failure, and Evidence semantics.
## Terminology
- **Passive observation**: bounded reads from an existing browser or Electron
target without evaluating code, driving input, navigating, or controlling
its lifecycle.
- **Controlled replay**: execution of operator-selected code in a declared,
isolated environment with committed inputs and limits. This is an Evidence
authority, not a claim that the real application behaved identically.
- **Extracted-module replay**: the only admitted JavaScript replay mode in v1.
REA evaluates a finite manifest of module bytes and explicit stubs without
launching the application.
- **Application interaction**: launching or driving the original application,
browser, renderer, or Electron main process. This is not extracted-module
replay and is not authorized by this ADR.
- **Replay plan**: the canonical, content-addressed description of one proposed
run before execution.
- **Replay worker**: the disposable Node.js process inside the OS sandbox. It
is never the REA server process.
- **Sandbox backend**: the OS mechanisms and adapter that establish namespaces,
resource controls, filesystem visibility, and owned cleanup.
## Threat model
### Protected assets
The boundary protects:
- the REA MCP/CLI process, its memory, credentials, sockets, and Evidence
ledger;
- user files outside the approved source reads and all host files from writes;
- other same-user processes and their control interfaces;
- host network interfaces, loopback services, Unix sockets, and external
services;
- process, memory, CPU, temporary-storage, output, and protocol capacity;
- the integrity of the replay plan, result, Evidence provenance, and exported
reproducer; and
- proprietary module bytes, which remain local and are never repository
fixtures.
### Adversaries and failure sources
REA treats all of the following as untrusted:
- recovered module source, including transitive modules and source maps;
- caller-provided JavaScript stubs;
- structured test inputs and generated fuzz cases;
- returned values, exceptions, console data, and worker protocol messages;
- source paths, symbolic links, and files that can change between planning and
execution; and
- a worker that crashes, hangs, forks, allocates aggressively, or exits while
descendants remain alive.
An operator-controlled sandbox executable, kernel, Node.js runtime, and REA
installation are trusted computing base. A hostile host administrator, kernel
or sandbox vulnerability, hardware side channel, and denial of service by an
already-compromised same-user process are out of scope. These exclusions do not
permit REA to omit the per-run resource controls in this decision.
### Required mitigations
The implementation must address:
| Threat | Required control |
| -------------------------------- | --------------------------------------------------------------------------------------------------------------- |
| Server-process compromise | Separate worker process inside the OS sandbox; no in-process fallback |
| Host file reads | Empty sandbox root; only a minimal read-only runtime closure; module bytes enter over a private bounded channel |
| Host file writes | No writable bind mounts; size-limited private tmpfs only |
| Network access | Separate network namespace with no host or external connectivity; no allowlist mode in v1 |
| Host process control | PID/user/IPC/UTS isolation, no inherited control sockets, new session, dropped capabilities |
| Kernel escape/inspection surface | Architecture-checked seccomp denial policy in addition to namespaces, capabilities, and cgroups |
| Fork, memory, or CPU exhaustion | Per-run cgroup v2 task, memory, swap, and CPU limits plus an outer wall-clock deadline |
| Output/protocol exhaustion | Independent byte/message/depth limits enforced by the parent before parsing or retaining output |
| Symlink and file-change races | Canonical authorization, descriptor-backed reads, digesting, staging, and exact plan revalidation |
| Undeclared dependencies | Closed module manifest; exact imports only; missing/dynamic imports fail rather than use host resolution |
| Nondeterministic globals | Versioned clock, PRNG, locale, timezone, platform, and API stubs; undeclared globals are unavailable |
| Orphan descendants | Owned PID namespace and cgroup kill followed by an observable empty-cgroup cleanup check |
| Forged result claims | Framed worker protocol, strict schemas, parent-computed commitments, and bounded stderr kept separate |
| Secret leakage | Empty environment, no credential inheritance, bounded local results, and existing genuine-secret redaction |
## Decision
### 1. Add a distinct `javascript_replay` permission capability
JavaScript replay receives a new permission capability named
`javascript_replay`. It does not inherit, imply, or satisfy
`process_capture`, `browser_observe`, `electron_observe`, artifact-read, or
workspace permissions. Grants for any of those capabilities do not satisfy a
JavaScript replay request, and a JavaScript replay grant does not authorize
them.
The administrator ceiling is disabled by default. Its eventual configuration
must commit:
- canonical roots from which module and optional stub files may be read;
- the exact Node.js and sandbox-backend executables;
- filesystem mount authority needed only to construct the private sandbox;
- network mode `none`; and
- no inherited environment names.
Configuration establishes only the maximum authority. Every execution also
requires an active permission grant and literal per-call approval bound to the
exact replay plan. Project, session, and administrator grants may satisfy the
permission-policy layer, but none replace that per-run approval.
Reproducer export is a separate host write after the sandbox has stopped. This
ADR's root-scoped `evidence_write` authority was removed; current exports use
the path supplied by the caller, while replay still waits for complete sandbox
cleanup before writing.
### 2. Admit extracted modules only
The v1 mode executes a finite, closed manifest of JavaScript modules. The
manifest identifies one entry module and every admitted dependency by a stable
sandbox alias and SHA-256 digest. Imports resolve only to another manifest
entry or to an explicitly declared, versioned stub. Bare package lookup,
ambient `node_modules`, import maps from the host, network imports, native
addons, WASI, FFI, child processes, workers, inspector activation, and implicit
dynamic import are unavailable.
Modules and stubs are read and hashed by the authorized parent, then copied as
bytes through a bounded private channel. Original host paths are not mounted
into the sandbox. The worker cannot reopen the operator's source path.
Source maps may supply locations, but source-map source code is not executed
unless its bytes are separately present in the approved manifest. A source-map
name or Webpack module ordinal is never execution authority.
REA does not launch, attach to, navigate, click, type into, or terminate the
user's application under this capability. A full application process remains
an independently approved Process Capture experiment; browser and Electron
targets remain passive. Any future mode that drives an application or permits
host/allowlisted network requires a new ADR and a distinct permission scope.
### 3. Use a two-phase, content-bound approval flow
One public replay surface supports a plan phase and an execute phase with the
same schema through CLI and MCP.
The plan phase:
1. validates all caller input and hard limits before file access;
2. obtains authority to read every selected module or stub path;
3. canonicalizes and opens each file without following an unapproved identity;
4. reads bounded bytes, computes digests, and closes the source descriptors;
5. resolves and probes the exact runtime and sandbox backend without running
application code;
6. normalizes stubs, inputs, generator settings, and limits; and
7. returns the complete proposed effects and a canonical `plan_digest`.
The plan must disclose at least:
- module aliases, canonical local paths, byte counts, and SHA-256 digests;
- input and stub identities, provenance, and digests;
- runtime and sandbox executable paths, versions, and executable digests;
- sandbox policy version and digest;
- network `none`, visible read-only runtime paths, and private writable tmpfs
size;
- time, CPU, memory, swap, task, temporary-storage, input, output, protocol,
case-count, and serialization limits;
- deterministic clock, PRNG seed/algorithm, locale, timezone, and platform
projection; and
- expected Evidence and cleanup behavior.
Execution requires the exact `plan_digest`. REA
rebuilds the plan immediately before admission. Any changed file, runtime,
sandbox executable, policy, stub, input, limit, or environment commitment
returns `plan_stale` before application code starts. Approval is not transferable
to a different plan and cannot be represented by a bare Boolean alone.
Interactive CLI may print the plan and ask for confirmation. Non-interactive
CLI and MCP must perform the same explicit plan/execute exchange; they may not
auto-approve because the caller requested JSON output or because permission
elicitation is supported.
### 4. Require an OS sandbox; fail closed elsewhere
The first production backend is Linux-only. It must combine:
- an empty mount namespace rooted in private tmpfs;
- a new user namespace with all capabilities dropped and further nested user
namespaces disabled;
- new PID, network, IPC, UTS, and cgroup namespaces;
- a new terminal session and parent-death behavior;
- no host D-Bus, agent, browser, Docker, Wayland/X11, SSH, GPG, audio, device,
or application sockets;
- a minimal, read-only runtime closure rather than `/`, the user's home,
working directory, or host `/tmp`;
- a fresh `/proc` for the sandbox PID namespace and a minimal private `/dev`;
- a parent-generated, architecture-checked seccomp policy that denies at least
namespace or mount changes, `ptrace` and cross-process memory inspection,
BPF/perf access, kernel keyring and module control, reboot/swap controls, and
the `TIOCSTI` terminal injection path;
- cgroup v2 `memory.max`, `memory.swap.max=0`, `pids.max`, and bounded CPU
controls owned for the one run; and
- an outer REA supervisor that can kill the complete cgroup and verify it is
empty.
The seccomp program is passed by descriptor, has a version and digest committed
to the replay plan, validates the syscall architecture before the syscall
number, and never permits `ptrace`-based mediation. Seccomp reduces exposed
kernel surface; it is not described as a complete sandbox by itself.
The reference backend for the implementation PR is Bubblewrap for namespace,
mount, and seccomp construction plus a delegated cgroup v2 manager for resource
enforcement. REA must feature-probe the actual required operations; version
text alone is insufficient. A setuid Bubblewrap installation, a backend that
silently skips a requested namespace or filter, unavailable user namespaces,
missing cgroup delegation, or inability to prove cleanup makes the capability
unavailable. REA setup must not install or upgrade Bubblewrap, systemd, Node.js,
or kernel components.
macOS, Windows, Linux hosts without every required feature, containers that
block the required namespaces, and alternative backends are unavailable in
v1. They must return precise diagnostics and remediation, not a process-only or
`vm`-only degraded mode.
Node.js runs with its permission model enabled and without grants for network,
filesystem, child processes, workers, native addons, WASI, FFI, or inspector.
That layer is defense in depth. It is never cited as the OS security boundary,
because Node.js explicitly does not promise containment of malicious code.
### 5. Make the worker disposable and deterministic by construction
Each approved run receives a new sandbox, runtime root, worker process, module
loader, and deterministic state. No module cache, realm, timer, file, global,
or child survives into another run. Within a multi-case run, every case gets a
fresh realm and fresh module instances; case order and generator state are
committed.
The worker exposes a small versioned set of language primitives and explicitly
selected stubs. It does not expose `process`, `require`, host `console`, timers,
`fetch`, sockets, filesystem APIs, `Buffer`, worker APIs, native bindings, or
ambient environment variables. Date/time, randomness, locale, timezone, and
platform values come only from committed deterministic providers. An
unsupported global or import produces an observed exception; REA does not
substitute a convenient host implementation.
Caller-provided stubs are executable untrusted code and receive the same
sandbox treatment as recovered modules. Built-in stubs have stable IDs,
versions, canonical source digests, and documented semantics. No undeclared
fallback stub is allowed.
Inputs are bounded canonical JSON. Results are projected without invoking
getters, proxies, `toJSON`, or arbitrary coercion: only admitted primitives,
arrays, and plain data properties are serialized. Cycles, accessors, functions,
symbols, unsupported prototypes, excessive depth, excessive nodes, and
oversized values become explicit serialization outcomes. Worker stdout is the
framed protocol only; bounded diagnostic stderr and bounded, opt-in console
observations remain separate.
### 6. Enforce independent resource ceilings
Every limit has a conservative product default. The v1 plan and result commit:
- an outer wall-clock deadline for the complete disposable run;
- cgroup memory, swap, task, and CPU ceilings;
- Node/V8 heap ceiling as defense in depth;
- private tmpfs bytes;
- serialized worker-protocol bytes, aggregate module bytes, and case-input
bytes; preset-generated cases also have a fixed preflight count ceiling;
- stdout, stderr, exception, stack, and aggregate retained-output bytes; and
- structured-result depth and node count.
The parent enforces protocol and output limits while bytes arrive, before JSON
parse or Evidence retention. The worker cannot request wider limits. A limit
hit terminates the complete run cgroup and is an observed termination, not a
partial successful return.
### 7. Preserve exact provenance and the right Evidence authority
A run that reaches worker admission produces Evidence with:
- provider `rea-javascript-replay` and a concrete implementation version;
- operation `run_controlled_replay`;
- authority `controlled-replay`;
- confidence `observed` for the exact returned value, thrown exception,
sandbox denial, limit termination, signal, exit, or crash that REA directly
observed;
- environment isolation `container`, the host platform/architecture, runtime
identity, sandbox-backend identity, and sandbox-policy digest;
- the replay plan digest, module and stub digests, canonical input or generated
case digests, deterministic providers, and every effective limit;
- bounded raw input, output, and exception data when retained, plus their
canonical digests and truncation state;
- start/admission/termination/cleanup states; and
- explicit limitations for incomplete output, nondeterminism not checked,
sandbox diagnostics, or cleanup that could not be proven.
The host module path remains an actionable local diagnostic and Evidence
subject location, while the worker sees only a stable alias. Module source is
not embedded in ordinary Evidence. A separately approved reproducer may copy
module/stub bytes or may retain path-plus-digest references; its manifest says
which mode was used. Repository tests use source-owned fixtures, and real-app
verification stores only digests and expected structural facts.
Derived differential results use confidence `derived` and link every source
run Evidence ID. A repeat with the same commitment can establish
`repeat-confirmed` for those runs; a single deterministic plan is only
`committed-not-confirmed`. Neither state proves behavior in the original
application.
When replay observations are added to the JavaScript Application Graph, their
graph-fact authority is `controlled-replay`. A claim that the real renderer,
preload, main process, or remote service behaves the same remains inferred or
unknown until compatible runtime authority exists. Static inference, passive
CDP observation, controlled replay, and real application interaction are never
promoted into one another.
### 8. Treat post-admission failures as observations
Pre-admission failures return a typed error and create no claim that module code
ran. Once the worker is admitted, REA retains a bounded execution record even
when the requested function does not return normally.
| Condition | Caller result and Evidence semantics |
| ------------------------------------------------ | ---------------------------------------------------------------------------------------------------------- |
| Disabled capability or missing grant | `permission_required`/`capability_unavailable`; no worker admission and no replay observation |
| Sandbox/runtime probe failure | `capability_unavailable` with exact failed feature; no module execution |
| Plan digest mismatch or changed source | `plan_stale`; no module execution |
| Sandbox setup fails before worker admission | `execution_failure`; no controlled-replay claim about the module |
| Module returns admitted data | Observed return value with complete or explicitly limited coverage |
| Module throws or result serialization rejects | Successful replay observation with outcome `exception` or `serialization_error`; application claim unknown |
| Node permission or sandbox denies an operation | Observed denial/termination when attributable; no claim that all attempted side effects were enumerated |
| Timeout, memory, CPU, task, tmpfs, or output hit | Observed limit termination; requested functional result unknown |
| Signal, runtime crash, or malformed protocol | Observed termination plus bounded diagnostics; functional result unknown |
| Caller cancellation after admission | Kill full cgroup, record cancellation/cleanup if possible, and project the public cancellation contract |
| Cleanup proves an empty cgroup and removed root | `cleanup: complete` |
| Cleanup cannot prove emptiness or removal | `cleanup_incomplete`, name every residual resource, and do not report success |
An exception is data, not an adapter failure. A sandbox setup error is not
evidence that the module crashed. A timeout is not evidence that the function
would never return. An output limit is not evidence that the truncated prefix
equals the full result. These distinctions are part of the public contract.
Cancellation and cleanup must be idempotent. REA first stops admission and
input, kills the owned cgroup, waits for an empty-cgroup observation, removes
the private runtime root, closes protocol descriptors, and releases timers.
Failure at one cleanup step does not skip the remaining steps.
### 9. Keep replay records reproducible without overstating determinism
The canonical replay manifest commits:
```text
plan structure and policy identity
REA/provider/runtime/sandbox executable identities and digests
module aliases, dependency graph, bytes digests, and source references
stub IDs, versions, source digests, and caller-provided stub digests
canonical inputs or generator algorithm, seed, case order, and case digests
clock, PRNG, timezone, locale, platform, and exposed-global commitments
all resource and serialization limits
network and filesystem projections
expected entry export and invocation convention
```
The result manifest adds canonical output/exception/termination digests,
coverage, dropped-data counts, cleanup state, and source Evidence links.
Canonical digests are computed by the trusted parent; worker-provided digests
are ignored.
Reproduction requires exact manifest compatibility. A runtime, sandbox policy,
built-in stub, module byte, generated case, limit, architecture, or platform
change is a different experiment. Cross-environment comparison may still be a
derived investigation, but it is never an exact replay.
### 10. Expose availability and policy truth before execution
`rea doctor`, capability inventory, and the planning response must report the
JavaScript replay capability without executing application code. Diagnostics
name the exact missing executable, unsupported host, failed namespace feature,
cgroup delegation failure, setuid rejection, runtime mismatch, or cleanup-probe
failure. Genuine secrets remain redacted; local executable paths, versions,
digests, and kernel feature failures remain visible.
The capability must be absent from executable tool routing until implemented.
Accepting this ADR does not increase the current tool count and does not add a
hidden experimental flag.
## Required implementation and verification gate
The implementation PR cannot merge until all of the following are true:
1. `javascript_replay` exists in configuration, permission policy, project
grants, policy explanation, CLI, MCP, docs, and capability inventory with
fail-closed defaults.
2. Plan and execute phases have schema-identical CLI/MCP behavior and an exact
plan-digest stale check.
3. No replay-disabled static, browser, Electron, session, setup, doctor, or
application workflow can reach module execution.
4. The worker never imports into the REA server process and no adapter offers a
`vm`-only fallback.
5. The Linux backend feature-probes namespaces, seccomp installation and
denial, delegated cgroup v2 limits, and the exact admitted architecture.
Every plan content-addresses the worker, seccomp filter, Node executable,
ELF loader, shared-library closure, and sandbox executables.
6. Source-owned fixtures cover parser, sanitizer, clipboard, exception,
ambient side-effect attempts, memory pressure, and busy loops. The worker
exposes no filesystem, environment, network, process, worker, native-addon,
timer, inspector, or ambient package-resolution API.
7. Tests and the real verifier prove bounded protocol/diagnostics, timeout and
OOM cgroup termination, and observed cgroup cleanup.
8. Evidence tests cover observed return/exception/termination, derived
comparison with source Evidence links, stale admission, cancellation
projection, and explicit limitations on application-level claims.
9. Package verification exercises the target-free public contract. The
dedicated real-Linux workflow exercises the sandbox backend; environments
without delegated user namespaces or cgroups report truthful unavailability
rather than selecting a weaker backend.
10. A real operator-local Notion-like parser or sanitizer benchmark can run
without committing its source, and exports only approved local reproducer
material.
## Rejected alternatives
### Reuse `browser_observe`, `electron_observe`, or `process_capture`
Rejected because those grants authorize different effects. Reuse would turn a
passive or unsandboxed process permission into recovered-code execution.
### Execute in the MCP server with `vm.runInContext`
Rejected because a VM context is not a security boundary and a crash, resource
attack, or escape would compromise the long-lived session and its credentials.
### Use only the Node.js permission model
Rejected because Node.js documents it as a seat belt for trusted code and
explicitly does not guarantee containment of malicious code. It remains a
defense-in-depth layer inside the OS sandbox.
### Ship a process-only fallback on unsupported hosts
Rejected because it changes the security claim while preserving the same tool
name. Unsupported hosts return unavailable.
### Permit network allowlists in v1
Rejected because DNS, redirects, proxies, credentials, service identity, and
response nondeterminism would substantially expand the threat and Evidence
model. The v1 namespace has no host or external network.
### Mount the approved project root read-only
Rejected because one approved module does not imply authority to disclose
every file in its project. The parent stages only exact, digested bytes.
### Treat exceptions, crashes, or limit hits as ordinary tool errors only
Rejected because that discards the principal behavioral observation and makes
negative results impossible to reproduce. Post-admission termination is
retained with explicit functional uncertainty.
## Consequences
- JavaScript replay is Linux-only at first and may be unavailable on containers
or distributions that disable user namespaces or cgroup delegation.
- The implementation requires more host probing and lifecycle code than a
simple Node worker, but the public security claim remains stable.
- Plans require two calls in non-interactive environments and become stale when
any committed input changes.
- Network-dependent modules and full application interactions are excluded;
their absence is explicit rather than emulated implicitly.
- Results are reproducible for the committed isolated experiment, not proof of
production behavior.
- Adding another sandbox backend, network mode, or application-interaction mode
requires a separate reviewed decision and conformance suite.
## References
- [Node.js permissions](https://nodejs.org/api/permissions.html) documents that
the permission model does not protect against malicious code and describes
its known filesystem and cross-process limitations.
- [Bubblewrap README](https://github.com/containers/bubblewrap/blob/main/README.md)
describes its namespace construction and makes clear that the caller's
arguments, not Bubblewrap alone, define the sandbox security model.
- [Linux seccomp filter documentation](https://www.kernel.org/doc/html/latest/userspace-api/seccomp_filter.html)
explains that syscall filtering reduces kernel surface but is not a complete
sandbox by itself.
- [ADR-0001](0001-provider-selection-and-analysis-profiles.md) establishes the
existing rule that provider selection, Evidence provenance, and capability
unavailability remain explicit and never silently fall back.
docs/adr/0003-managed-code-evidence-and-provider-boundary.md
-452
View File
@@ -1,452 +0,0 @@
# ADR-0003: Managed-code evidence and provider boundary
> Historical design record: The non-executing runtime-correlation planning
> tool described in earlier revisions was removed. This ADR's current
> implementation status below covers only the static managed tools that ship.
- Status: Superseded for runtime planning; static evidence boundary remains in force
- Date: 2026-07-16
- Implementation status: Read-only PE/CLI triage and exact identity are shipped
through `inspect_managed_artifact` / `rea inspect-managed-artifact`. Bounded
metadata, signatures, method bodies, raw CIL hashes, limited
decoded-instruction-tuple v1 hashes, exception regions, call edges, and
field-access anchors are shipped through `inspect_managed_members`; `rea
inspect-managed-members` provides CLI parity. Static member comparison and build-local
token remapping are shipped through `compare_managed_members` /
`rea compare-managed-members`. Declaration-only managed/native boundary
inventory for ModuleRef, ImplMap/PInvoke, ReadyToRun indicators, and non-IL
method flags is shipped through `inspect_managed_native_boundaries` /
`rea inspect-managed-native-boundaries`. Managed/native export or function
Evidence matching is shipped through `verify_managed_native_boundaries` /
`rea verify-managed-native-boundaries`; it verifies only supplied P/Invoke
export/function observations and does not map managed tokens to native
addresses. Decompiler reconstruction import is
shipped through `import_managed_reconstruction` /
`rea import-managed-reconstruction`; it records user-supplied C#/IL/pseudocode
as analyst inference only after exact static member locks match. Managed
inspection does not execute assemblies or observe CLR internals. Managed static graph
projection is shipped through
`project_managed_application_graph` /
`rea project-managed-application-graph`; it projects authenticated managed
Evidence into the application graph without changing managed tokens into
native addresses. Native-body bridge mapping and an actual runtime executor
remain future contracts.
## Context
Professional C# reverse engineering is not one decompiler operation. A file
described as “.NET” can be ordinary Common Intermediate Language (CIL),
ReadyToRun code containing both CIL and native implementations, a single-file
deployment, C++/CLI mixed mode, Unity Mono, Unity IL2CPP, or NativeAOT. Those
forms have different sources of truth and different native-analysis needs.
Choosing a decompiler before classifying the artifact can therefore produce a
plausible but false model.
The durable static facts in an ordinary managed PE are its PE/CLI headers,
metadata tables, signatures, method bodies, exception regions, and CIL. A C#
decompiler reconstructs source-like control flow and language constructs from
those facts. That reconstruction is valuable, but it is not the original
source and cannot replace the underlying metadata and CIL evidence. Obfuscation
makes names especially weak while tokens remain coordinates in only one build.
REA already separates one selected deep native provider from disjoint auxiliary
providers under [ADR-0001](0001-provider-selection-and-analysis-profiles.md).
It also separates static evidence, passive observation, process capture, and
interactive observation authorities. Managed analysis must fit those boundaries
without requiring a CLR to inspect an artifact, loading target code into the
REA process, or treating a managed token as a native address.
The design is informed by exact-build, reflection-free metadata extraction,
hash-locked method manifests, independent behavior models, and managed/native
boundary analysis performed against a local C# application. The application,
its decompiler output, and its runtime observations are research inputs only;
they are not distributable REA fixtures.
## Decision drivers
1. Classify the deployment before selecting an analysis route.
2. Make metadata and CIL independently inspectable without a CLR or target
execution.
3. Preserve exact-build provenance while enabling structural comparison across
builds whose MVIDs, tokens, names, and layout differ.
4. Compose managed and native analysis without introducing another ambiguous
deep-provider candidate.
5. Keep decompiled C# useful but epistemically subordinate to canonical bytes.
6. Analyze Linux, macOS, and Windows managed artifacts through the same local
CLI/MCP contracts on REA's supported hosts.
7. Keep setup additive and prevent managed analysis from installing .NET,
ILSpy, an SDK, or another unrelated toolchain.
8. Prove semantics with source-built conformance artifacts and compact,
operator-local real-application checks.
## Terminology
- **Managed artifact**: a PE, bundle component, or related metadata artifact
that carries a CLI header, CLI metadata, CIL, or another managed-runtime
commitment.
- **Classification vector**: independently observed container, runtime family,
implementation form, and architecture. It is not one guessed label.
- **Canonical static observation**: a bounded value decoded directly from the
target bytes under a versioned parser contract.
- **Reconstruction**: source-like output, including decompiled C#, derived from
canonical observations.
- **Structural inference**: a relationship or cross-version match derived from
signatures, APIs, constants, control/data-flow shape, or other observations.
- **Validation**: an independently checked proposition, for example a
source-built oracle assertion or a behavior-model corpus result.
- **Runtime observation**: a separately authorized fact obtained by attaching,
loading, debugging, instrumenting, or otherwise executing code.
## Decision
### 1. Classify before analyzing
Managed triage returns a classification vector and the observations supporting
each dimension. Absence of a marker produces `unknown` or `not_applicable`, not
an invented runtime family.
| Dimension | Initial values |
| ------------------- | ------------------------------------------------------------------------------------------------------------ |
| Container | PE file, single-file bundle, bundle component, Unity metadata pair, native image, malformed, unknown |
| Runtime family | .NET Framework, modern .NET, Unity Mono, Unity IL2CPP, NativeAOT, mixed CLR/native, unknown |
| Implementation form | CIL, CIL plus ReadyToRun, native only, C++/CLI mixed, metadata only, unsupported, unknown |
| Architecture | PE machine plus CLI flags; component-specific architecture for bundles; conflicting or unknown when required |
Target-framework attributes, assembly references, metadata stream versions,
bundle manifests, ReadyToRun directories, and native headers are observations.
They may support a runtime-family inference, but no single string is treated as
authoritative when the other bytes conflict.
Classification selects these routes:
| Observed form | Managed route | Native route and limitation |
| --------------------------------- | ------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------- |
| Ordinary Framework or modern CIL | Full metadata, signatures, method bodies, and CIL | Optional only for declared interop boundaries |
| Obfuscated CIL | Same canonical route; names receive no special trust | Optional for declared interop boundaries |
| Unity Mono assembly | Same canonical route with Unity context retained | Optional for engine/native boundaries |
| ReadyToRun | Metadata/CIL where present; implementation availability is per method | Selected Hopper/Ghidra analyzes native sections; CIL absence is reported per method |
| C++/CLI mixed mode | Metadata/CIL for managed members and explicit mixed-mode boundary observations | Selected Hopper/Ghidra analyzes native bodies; REA never converts tokens directly to native addresses |
| Single-file deployment | Inventory the bundle, then analyze each authenticated managed component | Selected Hopper/Ghidra analyzes the host/native components; unsupported compression remains explicit |
| Unity IL2CPP | Pair and authenticate available Unity metadata; do not claim canonical CIL | Selected Hopper/Ghidra analyzes the generated native binary; semantic recovery is degraded until supported |
| NativeAOT | Report any independently present managed metadata, otherwise no CIL claim | Treat as a native target with NativeAOT provenance; managed source reconstruction is unavailable |
| Malformed or unsupported artifact | Return bounded partial observations and typed unavailable/unknown scopes | Do not silently retry by loading or executing the file |
Classification is byte-driven and bounded. File extensions, product names,
process names, and an operator's description are hints only.
### 2. Implement managed static analysis as a disjoint auxiliary provider
The initial producer is a REA-owned auxiliary provider with ID
`rea-dotnet-static`. It owns a disjoint managed-artifact operation family. It is
not registered as a Hopper/Ghidra deep-provider candidate because it does not
compete for the same operation names and must be able to coexist with the one
selected native provider.
```text
one opened artifact/session
|
SessionProviderRouter
/ \
rea-dotnet-static auxiliary selected deep binding
PE/CLI metadata and CIL Hopper or Ghidra
\ /
explicit managed/native links
|
Evidence and workflows
```
An ordinary managed artifact can therefore remain unbound to a deep native
provider while managed operations work. ReadyToRun, C++/CLI, IL2CPP, NativeAOT,
and interop workflows may additionally require a caller-selected native
binding. Native-provider failure never causes managed evidence to be relabeled,
and managed-parser failure never causes an implicit native fallback.
Provider mechanics live under a provider directory such as `src/dotnet/`.
Provider-neutral identities, evidence states, and bounded values may live in
the domain/contracts layers. Application code composes results but does not
parse PE headers, metadata tables, signatures, CIL, ReadyToRun headers, or
provider-specific output.
### 3. Make the production static path byte-only and execution-free
`rea-dotnet-static` reads approved local bytes and implements the required
PE/CLI metadata and CIL subset in TypeScript. It does not call any reflection
API and does not require the .NET runtime. In particular, static inspection
must never:
- call `Assembly.Load`, `Assembly.LoadFrom`, `ReflectionOnlyLoad`, or an
equivalent runtime loader;
- resolve target dependencies by executing target-controlled resolver code;
- run module initializers, constructors, entry points, build tasks, plugins, or
generated helpers;
- start the target, attach to a process, contact a service, or inherit target
configuration and credentials; or
- interpret a decompiler, CLR, or native-provider diagnostic as an observation
from the artifact bytes.
The parser uses checked offsets and additions, explicit endianness, bounded
table/heap/body counts, cycle-safe traversal, and typed partial results. A bad
row, signature, body, resource, or stream does not authorize an unbounded retry
or a CLR load. Complete-coverage claims require every applicable bounded region
to have been admitted; otherwise coverage is partial, unknown, or unavailable
with exact offsets and reasons.
The caller selects the local input path directly. Local artifact
paths, digests, MVIDs, tokens, malformed offsets, and analysis metadata are
actionable diagnostics and are not secrets. Credentials, authorization data,
and genuine secrets remain redacted.
### 4. Bind every finding to exact artifact and producer identity
Every managed result carries a commitment with at least:
```text
artifact: canonical path + byte length + SHA-256
component: container identity + component path/name + component SHA-256, when applicable
classification: container + runtime family + implementation form + architecture
assembly/module: simple name + version + culture + public-key identity + module name + MVID
producer: provider ID + exact version + analysis-affecting parameters + profile digest
```
An assembly-qualified name does not replace the artifact digest or MVID.
Neither an MVID nor an Authenticode signature proves who authored the behavior.
Bundle components retain both their container and component commitments so that
bytes extracted from another host cannot be silently substituted.
Method observations add:
```text
declaring type + normalized CLI signature
build-local metadata token
RVA/file extent when present
raw CIL byte length + SHA-256 when the body is admitted
decoded-instruction-tuple SHA-256 when CIL is valid
separate max-stack, locals-token, and exception-region observations
```
The shipped `il_sha256` commits only the raw CIL byte stream identified by the
method header's `il_size`. It does not commit the method header, local-signature
blob, alignment padding, or extra sections such as exception clauses. Those
facts remain separate observations and the enclosing artifact SHA-256 binds
them to the exact file. A digest over the complete method body is not shipped.
The caller-visible field remains named `normalized_il_sha256`, but its shipped
v1 meaning is deliberately narrower than a canonical semantic CIL identity.
For a completely decoded method, REA serializes the instruction-order array of
`[opcode, operand_kind, operand]` tuples with JavaScript `JSON.stringify`,
encodes that text as UTF-8, and reports its lowercase SHA-256. The tuple
projection is defined here; the member result carries only the described
normalized digest.
Operands retain the current decoder projection: metadata and user-string
operands are build-local lowercase hexadecimal tokens, scalar branch operands
are decimal target CIL byte offsets, and `switch` retains only its decimal case
count. Constants and variable indexes are decimal strings; an operand-free
instruction uses kind `none` and value `null`. Opcode names preserve distinct
short forms and prefix instructions, but instruction offsets, branch target
instruction identity, switch targets, resolved token identities, method
headers, locals, and exception regions are not part of this digest. A partial
or malformed decode has no `normalized_il_sha256`.
Consequently v1 is a deterministic decoded-tuple fingerprint, not proof of
semantic equivalence. In particular, methods with the same opcodes and switch
case counts can share it despite different switch targets, and token movement
can change it without a semantic change. Cross-build comparison may use it as
one unique-only structural tier while retaining artifact/MVID commitments and
ambiguity; it must not describe the digest itself as token-remapped canonical
CIL. Canonical operand resolution, complete control-flow operands,
exception/local commitments, and an explicit algorithm version require a new
normalized-CIL schema.
Metadata tokens are reported because they are precise coordinates inside one
module. Cross-version identity never consists of a token alone. Matching tries
exact CIL/signature identity, then an exact declared-type/name/signature tuple,
then structural shape; a name by itself never selects a pair. Partial or
malformed bodies leave body facets unknown after a signature tuple pairs the
methods. Matches cite both build commitments, both local tokens, and all
competing candidates. V1 cannot cite a separate matching-algorithm version
because none is exposed.
### 5. Separate five evidence layers
Managed workflows retain these layers independently:
| Layer | Examples | Permitted claim |
| ---------------------- | --------------------------------------------------------------------- | --------------------------------------------------------------- |
| Static observation | Metadata row, signature, CIL opcode, literal, exception region | “These bounded bytes decode to this value.” |
| Reconstruction | Decompiled C#, async/iterator source shape, expression simplification | “This is one source-like representation of cited observations.” |
| Structural inference | Likely caller, field flow, behavior slice, cross-build method match | “These cited structures support this relationship.” |
| Independent validation | Synthetic oracle, corpus result, second parser agreement | “This declared proposition passed the named independent check.” |
| Runtime observation | Exact-build attach/load/instrumentation event | “This separately authorized experiment observed this event.” |
Each layer has its own producer, authority, state, confidence, coverage, and
limitations. A readable C# method is never emitted as original source or exact
semantic proof. A validator does not upgrade unrelated claims. Runtime
agreement does not retroactively turn a static inference into an observation.
Negative results are scope-qualified. For example, “no declared P/Invoke rows
were found in the admitted metadata tables” does not prove the application has
no native, dynamically resolved, generated, server-side, or protected path.
### 6. Admit ILSpy only as an optional reconstruction and conformance tool
REA's canonical static operations do not depend on ILSpy, a .NET SDK/runtime,
dnlib, or Mono.Cecil. The following review was current on the decision date:
| Tool | Reviewed coordinate | License | Runtime/host observation | Decision |
| ---------------------------- | ------------------- | ------- | --------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------ |
| `System.Reflection.Metadata` | 10.0.10 | MIT | Library targets .NET Framework 4.6.2, .NET 8/9/10, and .NET Standard 2.0 | Pinned independent metadata oracle for conformance; not shipped or invoked by production parsing |
| `ICSharpCode.Decompiler` | 10.1.1.8388 | MIT | Library targets .NET Standard 2.0; output is reconstruction | Pinned reconstruction/differential oracle candidate |
| `ilspycmd` | 9.1.0.7988 | MIT | Cross-platform .NET 8 tool; newer 10.1.1.8388 requires .NET 10 | Initial optional BYO reconstruction coordinate because it matches the source research baseline |
| dnlib | 4.5.0 | MIT | Cross-platform library; also exposes assembly-writing APIs | Differential test candidate only; its writable model is outside the production trust boundary |
| Mono.Cecil | 0.11.6 | MIT | Cross-platform .NET Framework/.NET Standard library with read/write support | Differential test candidate only; not a production dependency |
Coordinates and package hashes used by real verification are checked into a
source-owned lock manifest when that verifier is implemented. “Latest” is
never a reproducibility coordinate. A version change requires conformance and
an explicit profile/version change.
Artifact platform and REA host support are separate. The byte-only parser must
inspect PE/CLI artifacts produced for Windows on a supported Linux or macOS REA
host; that is not a claim that REA itself currently runs on Windows. Managed
library/tool candidates above are cross-platform at their reviewed targets, but
native composition remains limited by the selected Hopper/Ghidra host and
format support. Real verification runs the canonical path on Linux and macOS
and may run pinned oracle checks on Windows without turning that development
check into a public Windows-host claim.
If a caller-visible C# reconstruction operation is admitted, it is a distinct
optional capability. It accepts only an exact supported BYO `ilspycmd`, records
the tool and decompiler version, runs in an owned bounded process without
loading target code into REA, and links output to the canonical method
observation. Setup and installers do not download or install ILSpy, .NET, an
SDK, or a runtime. Doctor may inspect an explicitly configured executable and
report exact actionable incompatibility. Missing reconstruction never removes
canonical metadata/CIL capabilities.
This decision does not make a decompiler process a security sandbox. The
initial public path can defer C# reconstruction while canonical contracts and
process isolation are proven.
### 7. Use source-built conformance and operator-local real targets
The repository stores source, build descriptions, expected semantic facts,
and tool lock manifests, but not compiled managed binaries. An isolated build
step creates fixtures outside source directories and records their SHA-256 and
MVID before verification. At minimum, the corpus covers:
- Framework-compatible and modern .NET metadata;
- AnyCPU, x86, and x64 PE/CLI flags;
- generics, nested types, overloaded signatures, properties, events, resources,
attributes, and constants;
- exceptions, filters, prefixes, switch branches, P/Invoke, async and iterator
state machines;
- obfuscated and Unicode names without assigning semantics from names;
- malformed headers, tables, heaps, signatures, bodies, and exception regions;
and
- two semantically related builds with different MVIDs, tokens, layout, and
selected names.
Expected facts are checked directly and against pinned
`System.Reflection.Metadata`; ILSpy, dnlib, or Mono.Cecil may add differential
coverage but do not vote by majority. Disagreement is a failing or explicitly
unsupported case, not silently normalized output.
The real-application benchmark is an operator-local, opt-in osu! check derived
from the research workflow. It accepts an explicit path plus a compact
expectation manifest; verifies target SHA-256, MVID, architecture, and expected
assembly identity before analysis; and emits only bounded identities, tokens,
signatures, IL lengths/hashes, match states, and assertion results. It does not
emit full IL, decompiled source, runtime logs, user data, credentials, or
service/account material. No proprietary binary, hash-locked application
manifest, or derived dump is committed to REA.
### 8. Keep runtime behavior separate from static managed analysis
No managed runtime planning or execution tool ships. Managed inspection does
not execute assemblies or establish which methods ran. Where supported, direct
process capture can record target-level inputs and outputs, but it does not
prove CLR method execution or resolution. Internal managed behavior remains
unknown until independently observed with suitable runtime evidence.
## Public contract consequences
- Managed static operations will have CLI and MCP parity and produce the same
Evidence commitments.
- Adding managed operations changes the caller-visible tool inventory and
generated product catalog only in the implementation PR that admits them.
- Partial parsing includes admitted/dropped counts, exact limitations, and
coverage; it never returns a partial list labeled complete.
- Managed addresses distinguish file offsets, RVAs, native virtual addresses,
metadata tokens, and CIL offsets. They are not interchangeable.
- Cache compatibility is exact on artifact/component commitment, member-result
fields, parser profile, and producer version. For shipped v1,
`normalized_il_sha256` has no independent algorithm-version field.
- Provider-neutral workflows may compose canonical managed findings with the
one native binding, but every source operation retains its actual producer.
## Alternatives rejected
### Make ILSpy the only managed provider
This would make canonical metadata/IL inventory depend on a separately
installed CLR and conflate source reconstruction with byte observations. It
would also make tool-version drift part of ordinary inventory. ILSpy remains a
valuable optional reconstruction and conformance tool instead.
### Use `Assembly.Load` or reflection-only loading
Both delegate parsing and dependency behavior to a CLR and make static support
runtime-dependent. Ordinary loading can execute target code; reflection-only
loading still establishes a CLR load boundary and is unavailable across modern
runtimes in a uniform way. Neither is required to decode ECMA-335 bytes.
### Register the managed analyzer as another deep candidate
That would force callers to choose managed analysis instead of Hopper/Ghidra
for ReadyToRun and mixed-mode artifacts even though both are needed. Disjoint
auxiliary composition represents the real ownership boundary.
### Treat tokens, names, or decompiler output as durable identity
Tokens and obfuscated names change across builds, while decompiler output
changes across tools and versions. They remain useful build-local coordinates
or reconstruction labels but are insufficient for exact or structural
cross-version identity.
### Package a .NET runtime or mutable managed library in REA
This would expand installer effects, package size, patching obligations, and
the target-parsing attack surface. The initial byte parser keeps canonical
support within REA's existing Node runtime; external managed tools remain BYO
or development-only oracles.
## Rollout gates
1. Add read-only triage and identity with malformed-input bounds. Shipped.
2. Add metadata, signatures, method bodies, deterministic decoded-CIL tuple
fingerprints, and exact artifact Evidence. Shipped with the v1 limitations
above; full normalized-CIL semantics remain planned.
3. Add obfuscation-resistant slices and cross-build match states. Shipped.
4. Compose explicit managed/native boundaries with Hopper/Ghidra. Declaration
inventory and native export/function Evidence matching shipped; native-body
bridge mapping remains planned.
5. Add source-built, pinned real-tool, package, CLI, and MCP conformance.
Source-owned conformance is shipped; pinned external real-tool checks remain
planned.
6. Project authenticated managed static Evidence into the application graph.
Shipped.
Every gate must keep unsupported ReadyToRun, NativeAOT, IL2CPP, mixed-mode, or
bundle cases explicit rather than broadening claims to make a fixture pass.
## References
- [ECMA-335: Common Language Infrastructure](https://ecma-international.org/publications-and-standards/standards/ecma-335/)
- [.NET ReadyToRun deployment overview](https://learn.microsoft.com/en-us/dotnet/core/deploying/ready-to-run)
- [.NET Native AOT deployment overview](https://learn.microsoft.com/en-us/dotnet/core/deploying/native-aot/)
- [.NET single-file deployment overview](https://learn.microsoft.com/en-us/dotnet/core/deploying/single-file/overview)
- [`System.Reflection.Metadata` package](https://www.nuget.org/packages/System.Reflection.Metadata/10.0.10)
- [ILSpy repository and releases](https://github.com/icsharpcode/ILSpy)
- [dnlib repository](https://github.com/0xd4d/dnlib)
- [Mono.Cecil repository](https://github.com/jbevain/cecil)
docs/adr/README.md
-11
View File
@@ -1,11 +0,0 @@
# Architecture decision records
Accepted decisions describe the architecture that implementation PRs must
follow. Acceptance does not by itself mean the behavior is shipped; each record
states its implementation status separately.
| ADR | Status | Implementation |
| ------------------------------------------------------------------------------------------------------------------ | -------------------- | ------------------------------------------------------------------------------------------------------------------------------------- |
| [0001: Provider selection and analysis profiles](0001-provider-selection-and-analysis-profiles.md) | Accepted | Provider selection and 25 read-only Ghidra operations on Linux/macOS and experimental Windows x64 P0; Linux/macOS session annotations |
| [0002: Controlled JavaScript replay authority and sandbox policy](0002-controlled-replay-authority-and-sandbox.md) | Superseded | Historical design; the controlled JavaScript replay tool was removed |
| [0003: Managed-code evidence and provider boundary](0003-managed-code-evidence-and-provider-boundary.md) | Partially superseded | Seven static managed tools remain; runtime planning removed |
docs/capability-migration.md
-477
View File
@@ -1,477 +0,0 @@
# Incremental capability migration
This working guide tracks the migration in #740. Update it as concrete domains
migrate. Existing public contracts and domain-specific ports remain authoritative;
there is no new universal provider interface or fixed module manifest.
## Ownership to preserve
| Responsibility | Current owner | Migration direction |
| --------------------------------------------- | ----------------------------------- | ----------------------------------------------------- |
| Analyst semantics and Evidence representation | domain | Keep pure and provider-neutral |
| Named public input/output contracts | contracts | Keep exact CLI/MCP meaning and complete discovery |
| Producer parsing and external tool protocols | provider adapters | Keep behind the relevant typed port |
| Shared analyst workflows | application | Share between CLI and MCP |
| Concrete provider construction | composition, MCP startup | Keep fresh typed factories at the production boundary |
| Investigation Evidence and Unknowns | InvestigationRecords/EvidenceLedger | Consume narrow read/write/atomic record ports |
| Target/profile snapshots and invalidation | BinarySessionRecords/BinarySession | Keep binary-owned |
| Subprocess ownership and host primitives | process/windows | Reuse; preserve cancellation and cleanup |
| Public translation | CLI/MCP adapters | Delegate to shared workflows and named contracts |
Retain exact Evidence IDs, provenance, detached reads, atomic Evidence/Unknown
mutations and optimistic revisions. Preserve current close and failed-cleanup
record clearing while extracting ownership; #721 owns any retention-policy change.
Each CLI invocation and MCP connection keeps its own state. Provider registration
must not launch engines or install prerequisites.
## Migrate before generalizing
1. Record the actual callers, state, environment, protocol and verification owner
of the capability being migrated.
2. Move concrete construction behind its existing typed port. Preserve injected
ports, selected environment, queues, failure state and cleanup behavior.
3. Test both caller paths and the applicable package/real-provider boundary.
4. Use independently migrated capabilities to identify repeated metadata and
composition needs. Derive a small source declaration only when those needs
are established; keep unlike target/lifecycle semantics separate.
5. Move a coherent capability's files, fixtures and verification lanes in a
mechanical PR. Update all path-sensitive consumers in that same PR.
Pure helpers do not need an interface each. A capture-format adapter or typed
factory may be sufficient. Avoid generic JSON execution, external MCP catalog
passthrough, eager provider acquisition, new permission fields or a multi-package
rewrite. Optional loading should preserve successful peers and report the failed
capability without hiding tools from the catalog.
## Checks before and after a move
`npm run verify:test-discovery` compares actual Vitest file discovery with every
repository-owned `*.test.ts` under src and tests, including newly added files. The
check requires Git, includes untracked non-ignored tests, and excludes removed
paths during a move. It
rejects missing files and overlapping project ownership. It only lists files;
it does not replace behavior tests or real-provider verification.
Check imports, project isolation and coverage, development-test selection,
lint exceptions, catalog/generator imports, npm and CI verification entrypoints,
fixture ownership, and installed-package bridge/native path resolution.
Domain and contracts must not import application workflows, caller adapters or
provider implementations. Production composition is a deliberate outer boundary.
Other existing adapter imports will migrate incrementally; do not widen a lint
exception to conceal new protocol code inside application workflows.
`npm run verify:module-boundaries` parses repository-owned source imports,
reexports, type imports and literal dynamic imports, then checks their resolved
source layer. Nested names such as `domain/android` are domain code; `src/android`
is the provider adapter. The check runs with lint, including cached static gates.
It covers tracked and untracked non-ignored TypeScript source and skips cached
paths removed during a move. Pass exact source paths for a focused check.
Pure source and its tests keep inward dependencies. Application/server provider
construction guards cover the migrated deep, Android, firmware, JavaScript recovery and observation
implementations. Application workflows cannot import composition, including the
former binary runtime entrypoint; shared browser capture/export helpers remain admitted while their
ownership is reviewed. Test lane restrictions still use Oxlint. Producer-dependent
browser and managed comparison tests live in boundary lanes with their original
fixtures and assertions. This is a development guard, not runtime authorization.
Select verification by the claim changed: schema/composition, package, real
engine or host-native workflow. Record host/target prerequisites and source-owned
fixtures. Run heavy lanes serially with bounded workers/heaps and reuse existing
engines. Recording ports do not prove real provider or platform support.
## Extension examples to validate during migration
An implementation of an existing port should require a provider adapter, typed
production factory selection, exact producer parsing/provenance and matching
verification. Existing workflow meaning and public contracts should stay stable.
A new analyst capability deliberately adds semantics, named contracts, a typed
port/workflow when needed, both caller adapters, availability, documentation and
verification. Its integration is broader than swapping a provider. The migration
should eliminate incidental duplicate wiring, not erase those design decisions.
Update these examples with real migrated entrypoints before closing #740.
## First production factory migration
`src/composition/android.ts` and `src/composition/firmware.ts` construct providers
behind the existing AndroidAnalysisPort and FirmwareAnalysisPort. CLI registrars
pass their selected environment; MCP uses the process environment unless a caller
injects a port. Each factory call returns a fresh instance and acquires no process,
workspace or toolchain. Queues, cleanup-failure state and per-request cleanup stay
inside the provider. The existing launcher seams remain available to boundary
fixtures, which now exercise the production factories.
Application workflows and MCP registration must not import these concrete
providers. Composition may import their implementations. Binary composition also
uses this outer boundary after its direct-analysis callers were migrated.
This concrete pilot does not introduce a shared provider interface or module
manifest.
## Independent optional observation adapters
Browser observation, browser scenarios, Electron observation, Electron scenarios
and V8 Inspector observation now have separate factories in `src/composition/`.
CLI commands reuse them; MCP startup dynamically imports each factory independently.
`OptionalObservationProviders` describes these five existing typed ports only. It
is provisional startup wiring, not an extensible plugin manifest or a universal
provider interface.
A failed import or constructor retains the successful peers. The complete tool
catalog remains advertised. Session availability reports the failed adapter and
its actual reason, and affected handlers return the existing capability-unavailable
error. Explicit loading failures take precedence over configured availability.
Passive Electron and active Electron failures are independent; static JavaScript
analysis and runtime reconciliation remain usable without either adapter. CLI
Electron commands also load only their selected runtime factory at execution.
Source identities are available without importing the optional implementations.
Registration acquires no engines, probes no endpoints, changes no permissions and
installs nothing. Core transport failures and shutdown still use their existing
lifecycle. Fault-injection coverage uses the production loading/startup path and
actual MCP SDK calls; recording ports establish composition behavior, while the
real Inspector lane separately establishes runtime behavior.
## Investigation record ownership pilot
`src/application/investigation/InvestigationRecords.ts` owns one existing
EvidenceLedger per runtime. The production session composition constructs that
owner explicitly; the direct BinarySession constructor retains a fresh default
for compatibility. There is no second ledger, global store, persistence or new
retention policy.
The ledger, Unknown Evidence helper and the ledger's two original behavior tests
now live beside the owner in `src/application/investigation/`. Bundle validation
and record-owner consumers follow the new paths; retained-reference access still
uses the narrow record ports. The move preserves implementation bodies and test
assertions. Binary snapshot/cache files remain with their existing owner.
EvidenceReader, EvidenceWriter and EvidenceUnknownWriter describe existing
read/write/atomic callback needs. UnknownRegistryPort preserves optimistic
revisions and consistency verification. Non-binary MCP registrars use these
record types without depending on the binary execution/lifecycle surface.
BinarySessionPort retains its compatible record methods through composition.
BinarySessionRecords remains the snapshot owner and compatibility facade. It
passes the actual active target explicitly for recordUnknown/updateUnknown;
recordEvidenceWithUnknown keeps its target-free mutation subject. Snapshot import
merges records without notifying midway, then emits after the cache commits.
Bundle import retains the full change delta so Unknown-only changes notify even
when recordsAdded is zero. Clearing resets records, cache and invalidation before
notification; observer failures remain best effort after a committed mutation.
Close and failed-cleanup clearing, ordinary target-switch retention, detached
reads, Evidence identities and deterministic bundle transfer keep their existing
behavior. Snapshots still require an active target and concrete analysis profile;
extracting investigation ownership does not make them a target-free record store.
#721 remains responsible for any future lifetime-policy change.
## Inspector adapter ownership
`src/inspector/` owns V8 Inspector discovery, passive capture, script/target
location interpretation and its lightweight provider identity. The composition
factory and optional loader use this owner. Browser CDP connection, endpoint/value
and Electron file-location helpers remain deliberate shared adapter utilities at
their existing paths; no provider wire protocol moves into domain or process.
Loopback fixtures and producer boundaries now live under
`tests/fixtures/inspector/` and `tests/boundary/inspector/`. The socket-backed
unresolved-target reconciliation test moves from composition into the forked
boundary lane with its original assertions. Adapter tests retain fork isolation.
`npm run verify:inspector` keeps its public development entrypoint and delegates
to `scripts/verify/inspector/runtime-observation.mjs`, using the real Node fixture
under `tests/conformance/inspector/`. Catalog imports, source guards and test
discovery follow the new paths. Observation authority, lifecycle, producer
interpretation and caller-visible contracts stay unchanged.
## Auxiliary source declaration pilot
Artifact, macOS native and managed static metadata now belongs to each adapter's
`*ProviderMetadata.ts` module. The implementation and
`src/composition/auxiliaryAnalysisProviders.ts` reuse the same identity and exact
capability descriptors. The declarations reference existing AnalysisProvider
loaders; they cover these three disjoint auxiliary implementations only. Deep
candidate selection and the five typed observation ports remain separate.
The binary runtime reads these source declarations while auxiliary construction
stays lazy. The separate managed-only entrypoint retains direct construction for
its execution-free workflow. The MCP catalog generator reads declarations without
constructing implementations, using its canonical macOS platform; actual runtime
host restrictions remain explicit.
Discovery and binary operation-kind routing now consume canonical source contracts
rather than their generated JSON artifact. No declaration imports its generated
output. Generated catalog facts and public schemas stay unchanged.
All observation registrars resolve contracts by name. An isolated compiled-process
test reverses presentation arrays and performs actual MCP SDK calls to verify
advertised schemas and handler meaning. Reordering metadata must not select a
different operation, schema, provider or deep-provider priority.
## Android layer and verification ownership
Android workflows and their existing typed port live in `src/application/android/`,
pure APK/inventory semantics in `src/domain/android/`, and the named inspection
contracts in `src/contracts/android/`. The headless JADX producer remains in
`src/android/`; `src/composition/android.ts` supplies its fresh factory to both
caller adapters. Public names, result meaning and JVM/queue/cleanup limits are
unchanged. Inventory-only application projection remains execution-free.
The real lane and signal-cleanup helper belong to `scripts/verify/android/`; its
explicit downloader and fixed manifest belong to `scripts/fixtures/android/`.
`verify:android` and `fixtures:android` keep their command names and repository-root
`_reference/apk-integration/` fixture default. Synthetic producer fixtures stay in
`tests/fixtures/android/`, with matching boundary and composition ownership. Real
JADX CLI/MCP parity requires the fixed APK and existing audited JAR; synthetic
protocol/cancellation success does not establish that engine or an unverified host.
## Deep-provider source declarations
Hopper, Ghidra and IDA declare their implemented analyst operations in their
own `*ProviderCapabilities.ts` modules. Hopper's inventory is typed against the
existing direct-operation contracts; Ghidra reuses its inventory/function adapter
operations; IDA retains its read-only operation family. Frozen operation arrays
feed the existing capability builders without importing generated catalog data.
The catalog generator reads these source declarations and joins operation names
to canonical `TOOL_CONTRACTS`. A missing contract identifies the declaring provider
and operation. Public schemas and descriptions remain owned by contracts; the
declarations describe implementation coverage. Provider adapters must not consume
the generated MCP catalog that is derived from them. The resolved module guard
checks this edge while admitting canonical source contracts, generated package
metadata and the existing application snapshot-cache consumer.
Exact provider effects and lifecycle differences remain in their adapters:
Hopper's GUI and mutation effects, Ghidra's experimental Windows authority and
mutation restrictions, and IDA's attached/headless effects and live observations.
This source increment introduces no common provider lifecycle or new interface.
## JavaScript application ownership
`src/application/javascript/` owns JavaScript/Electron artifact analysis and
graph projection, feature/version/export/source comparisons, runtime Evidence
and reconciliation, observation ports, and source recovery workflows. Pure
analysis and public schemas keep their existing domain/contract ownership;
shared artifact inventory, investigation records and execution types remain
shared. CLI, MCP and production factories import the owning workflow directly.
`src/domain/javascript/` owns static parsing and semantic IR, application and
semantic graphs, Electron facts, runtime reconciliation, recovery result
semantics, and source-to-bundle comparisons. Shared Evidence, digests, ordering,
comparison semantics and historical-source graphs stay at their existing owners.
Native handoffs link exact-subject Evidence without selecting or executing a
provider.
`src/contracts/javascript/` owns the JavaScript/Electron observation and recovery
contracts, JavaScript-specific workflow requests, and their examples. The shared
`applicationToolContracts.ts` aggregate stays at the contract root because it also
owns reconstruction and Android/Apple projection operations. Canonical tool
aggregation, named handler binding and SDK schema presentation remain unchanged.
Catalog source paths, compiled verifier/SDK fixtures and recovery CI filters
follow the domain/contract owners.
`scripts/verify/javascript/` groups application-workflow, runtime-reconciliation,
large-digest and real-recovery entrypoints. Their existing npm commands retain
their invocation semantics. Recovery fixture compilation lives beside its
unchanged sources under `scripts/fixtures/javascript-recovery/`. Its default
REA entrypoint still resolves to `scripts/rea.mjs`. Managed conformance imports
the relocated compiled workflow; owning CI source filters follow the new paths.
`src/artifacts/javascript/` owns directory/ASAR reader construction and relevant
file acquisition, including root/nested byte verification, UTF-8 diagnostics and
nested reader cleanup. Its narrow factory returns the existing `ArtifactReader`;
the application retains input admission, inventory, graph construction and its
original root-reader `finally` close. Pure file facts live in
`src/domain/javascript/javascriptArtifactFiles.ts`, and the shared immutable
inventory snapshot lives in `src/domain/artifactInventorySnapshot.ts`. Inventory
options and integrity policy remain application-owned. Shared stream hashing
lives in `src/artifacts/ArtifactHash.ts`, preserving complete byte counts and the
bounded classification prefix. Acquisition/hash guards reject imports from
application, composition or caller adapters; JavaScript workflows use the factory
instead of importing concrete ASAR/directory reader classes.
ZIP, native-DMG and Mach-O inventory reader selection keep their existing
lifecycles. Runtime ports retain their distinct Inspector, passive Electron and
active Electron effects; shipped Electron hooks remain at their existing
provider boundary.
## Managed capability ownership
`src/domain/managed/` owns managed artifact and member semantics, reconstruction
import, native-boundary verification and application graph projection.
`src/application/managed/` owns their shared Evidence workflows, and
`src/contracts/managed/` owns their named contracts and examples. Producer
metadata/IL parsing stays in `src/dotnet/`; target resolution, Evidence, ordering
and investigation records retain their shared owners. Path-based member
comparison preserves its actual target/byte-digest admission and existing parser
helper.
`scripts/verify/managed/` groups the portable conformance entrypoint and its
support, manifest, oracle and completion-report helpers. Its PE byte fixture
builder and declaration live in `scripts/fixtures/managed/`, shared with the
installed-package lane and existing managed conformance tests. `verify:managed`
keeps its command name and optional caller-cwd manifest resolution. Compiled
catalog and generator imports follow the owners; document-check cache inputs
include the relocated runtime verifier and fixture files.
Portable byte fixtures, optional operator-local manifests, actual ILSpy oracles
and real Ghidra NativeAOT checks retain their distinct proof levels and
prerequisites. The existing NativeAOT fixture/engine lane keeps its owner. This
move preserves implementations, Evidence identities and generated commitments.
## Binary application ownership
`src/application/binary/` owns the active session, deep-provider registry and
evaluation, operation routing, disjoint provider composition, lazy auxiliary
clients, cancellation and client cleanup. Its records facade and snapshot
cache/files remain bound to the exact target and analysis profile. Investigation
records still have their separate owner in `src/application/investigation/`.
`AnalysisProvider.ts` stays shared because its identity/execution types also serve
nonbinary ports. `BinaryTargetResolver.ts` stays shared because Android, firmware,
managed and artifact workflows use its target parsing. Production wiring lives
in `src/composition/binary.ts`; application workflows gain no outward composition
dependency.
The three colocated application tests move with their owner. Composition cases
remain in `tests/composition/analysis-sessions/`; filesystem and SDK cases retain
their boundary lanes. The shared `tests/fixtures/binarySession.ts` factory supports
several capability families. Existing recursive application globs discover the
moved tests; bridge assets and real-verifier entrypoints remain at their owners.
## Firmware layer and verification ownership
Firmware workflows and their existing typed port live in
`src/application/firmware/`, pure request/result semantics in
`src/domain/firmware/`, and named contracts in `src/contracts/firmware/`.
`src/firmware/` keeps producer commands, report parsing, source copies, publication
and cleanup; `src/composition/firmware.ts` supplies a fresh typed factory.
The real lane belongs to `scripts/verify/firmware/analysis.mjs`; the fixture runner
and its unchanged Python producer belong together in `scripts/fixtures/firmware/`.
The npm command names and selected `REA_FIRMWARE_FIXTURE_ROOT` are unchanged. The
runner retains its working-directory-relative default; the verifier retains its
repository-root default. Both point to `_reference/firmware-integration/generated`
when invoked through the documented npm commands. Optional ext4/Ghidra lanes keep
their own prerequisites. Source-fixture generation and synthetic producer tests
do not establish real Binwalk/Unblob analysis or another host's execution support.
## Apple artifact capability ownership
`src/domain/apple/` owns Apple application anatomy/projection, asset catalog
facts, Interface Builder graphs, keyed archive semantics and plist values.
`src/application/apple/AppleApplicationService.ts` retains the shared workflow
that projects authenticated IPA/macOS inventory Evidence.
`src/artifacts/apple/` owns the actual producer boundary: directory acquisition,
assetutil invocation, plist/NIB decoding, encoded view-parent interpretation and
keyed archive inspection. These helpers are used by ArtifactProvider, retain
its existing typed analysis boundary, injected assetutil seam and original reader
cleanup, and do not require a new port for each helper. Producer code no longer
lives in the application layer. Guards reject application/composition/caller
imports from this owner while admitting pure facts and shared artifact readers.
Adapter-source tests retain their assertions in the existing forked adapter lane.
`scripts/verify/apple/` groups archive, asset catalog, Interface Builder,
Objective-C/Swift dispatch and macOS bundle verification. The macOS bundle fixture
builder lives in `scripts/fixtures/apple/`; NIB byte fixtures stay beside their
artifact decoder. Existing format-specific conformance sources and goldens keep
their locations, and relocated verifiers resolve them against their own file
URLs. Public npm entrypoints and CLI/MCP contracts stay stable; generated-document
cache inputs include the relocated runtime scripts and fixture builder.
Portable decoder and injected producer tests establish their format/boundary
claims. Actual Swift/Xcode, assetutil, native NIB, bundle signing and DMG lifecycle
claims require the real Apple lane. Native binary API/value-flow semantics and
shared cross-domain contract aggregates keep their separate owners.
## Artifact inventory and extraction ownership
`src/artifacts/inventory/` owns format classification, reader selection, inventory
policy, canonical scanning and artifact graph construction. The shared artifact
provider and JavaScript reconstruction workflow consume this owner. Snapshot
semantics remain in `src/domain/artifactInventorySnapshot.ts`.
`src/artifacts/extraction/` owns member selection, integrity checks and transactional
materialization through the existing readers and safe output tree. Reader cleanup,
cancellation, rollback, archive containment and caller-visible results retain their
existing behavior. Acquisition guards reject imports from these producer owners
into application workflows, production composition or CLI/MCP adapters.
`src/application/artifacts/ArtifactExtractionDestination.ts` owns the caller's
default temporary output-root choice for both CLI and MCP. The adapters pass this
destination to the existing provider operation; format handling and member writes
remain in the artifact owner.
Add format-specific acquisition behavior beside the inventory readers and exercise
its actual producer representation in filesystem boundary tests. Add caller
workflow behavior in the application owner and verify both adapters. The installed
package lane checks inventory identity and extraction through CLI and MCP; the
real Apple bundle lane separately establishes DMG mount and detach behavior.
## Binary production composition closure
`src/composition/binary.ts` is the shared production owner for deep candidate
construction, lazy auxiliary composition and the managed-only session factory.
It preserves the existing constructor bodies, selection policy and fresh session
per invocation. Factory registration acquires no engine or target.
`DirectAnalysis` and `DirectAnalysisStatus` receive these two existing factories
through `DirectAnalysisDependencies`. They retain configuration parsing, selected
status environment, cancellation, snapshot replay/binding, Evidence projection
and finally-close. The dependency uses the existing BinarySession type, including
its replay policy; it adds no universal backend or lifecycle contract.
`src/composition/directAnalysis.ts` binds the production factories for CLI and
one-shot managed MCP callers. Binding creates no session and stores no retained
analysis state. MCP startup constructs its connection session through the same
binary factory. Source and compiled fixtures use these entrypoints. The former
application runtime path is removed, and the source guard no longer admits its
outward provider/composition imports.
## Native analyst capability ownership
`src/domain/native/` owns portable native inspection, instruction/data types,
load-image facts, API boundaries, annotations, metadata recovery, UI observation
semantics and value/investigation graphs. `src/application/native/` owns shared
API projection, call routes, dispatch inspection and UI/value tracing. Its tests
keep the existing service lane. `src/contracts/native/` owns the named native
adapter contracts; shared official/enhanced aggregates remain at their owners.
Producer interpretation, external commands and host UI lifecycle remain in
`src/native/`, Hopper, Ghidra and IDA. The shared function dossier still lives in
`domain/hopperValues.ts`: its historical name does not make its meaning exclusive
to Hopper. This move retains existing typed ports, exact result schemas,
observed/derived/unknown distinctions, profiles and Evidence links.
Source and compiled verifier imports follow the new owners. Host UI and real
native-value verifier locations are retained until their own proof lanes migrate;
interactive macOS UI success and real dependency tracing require their respective
host/engine workflows. Portable source fixtures and injected call-route tests
establish their narrower boundaries, without expanding provider/platform support.
## Process capture and analyst workflow ownership
`src/domain/process/` owns scenario/capture validation, portable process trees,
trace specifications, observations, comparisons and the canonical process Evidence
identity. The standalone contract example lives in `src/contracts/process/`;
shared session contracts retain their broader investigation owner.
`src/process/capture/` owns the actual PTY boundary, terminal rendering, sampling,
selected child environment, filesystem snapshots/effects, runtime path admission,
event journal, settlement and resource cleanup. It reuses `src/process/` ownership
primitives and the Windows host substrate. Its identity import follows the
existing pure declaration directly. A source guard rejects outward application,
composition and caller imports; no new provider or lifecycle interface is added.
`src/application/process/` retains Evidence projection and file-backed CLI
capture/comparison workflows. Capture helper tests use the forked adapter lane;
Evidence/host integration tests use the serial process-boundary lane. Their
fixtures and assertions remain unchanged. Installed capability-probe URLs follow
the new compiled adapter path, including the optional-dependency failure probe.
Windows PTY capture retains its existing unavailable outcome because descendant
cleanup is not yet verified. Native Job Objects and a PTY binary do not establish
that capture workflow. Real POSIX capture/cleanup and installed terminal/PTY module
resolution require their actual host and package lanes; portable comparison
fixtures establish their separate evidence semantics.
docs/cli.md
+1 -1
View File
@@ -80,7 +80,7 @@ or close. Provider failures are returned with their original reason. For an
`ambiguous` selection error, choose from `details.candidate_ids`; for
`provider_unavailable`, run `rea doctor --provider ID --json` to diagnose the
selected engine. See [task readiness](installation.md#check-readiness-for-your-task)
and [provider selection](adr/0001-provider-selection-and-analysis-profiles.md).
and [provider selection](mcp-contracts.md#identity-and-discovery).
The session reports work still in progress through `analysis_activity`.
A client timeout can end its wait while the provider continues analyzing.
docs/ghidra-provider.md
+72
View File
@@ -0,0 +1,72 @@
# Ghidra provider semantics
REA imports one target into an ephemeral Ghidra database. Linux/macOS support
read-only analysis and atomic function-name/entry-comment edits; Windows P0 is
read-only. GUI controls and persistent database mutation are unavailable.
Executable bytes remain unchanged.
Use [installation](installation.md#ghidra) for supported hosts, prerequisites,
configuration, startup deadlines, resource controls, and cleanup recovery.
[Windows P0](windows-ghidra-p0.md) has its own admission and lifecycle boundary;
[NativeAOT recovery](ghidra-nativeaot.md) is an optional, separately verified
extension. Check the [release boundary](installation.md#released-package-and-main)
when using npm.
Provider selection and profile identity follow
[MCP contracts](mcp-contracts.md#identity-and-discovery). One deep
provider stays bound to the target; runtime failure never selects another
provider automatically. The contracts below describe Ghidra observations,
not equivalence with another engine's pseudocode or analysis database.
## Admitted inventory semantics
| Concern | Ghidra contract |
| ---------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Program identity | One `analyzeHeadless` import produces exactly one Program; `list_documents` therefore returns exactly one name. |
| Addresses | Default memory uses lowercase `0x` hexadecimal. Non-default and external spaces use `<percent-encoded-space>:0x<hex>` and remain round-trippable. The handshake commits image base and default address-space name. |
| Symbols | `list_names` includes address-bearing memory and external symbols, including dynamic symbols, while excluding variable and no-address namespace records. Each item reports primary, dynamic, external, symbol type, and source facts. |
| Procedures | Both non-external and external functions are listed. A local thunk remains distinct from its resolved target; exact and qualified name lookup fails on ambiguity rather than guessing. |
| Strings | Only Ghidra-defined string `Data` is observed. Items report charset, byte length, and whether a required null terminator is missing. The API cannot distinguish a present terminator from fixed/Pascal layouts when no terminator is missing, so that state is named `present_or_not_required`. |
| Memory | Memory-block end addresses are exclusive. Read/write/execute, initialization, overlay, address space, and image base are direct Ghidra observations. |
| Inventory | Procedure, symbol, and string listings plus searches return the complete matching collection in one response; callers do not provide offsets or result-count limits. |
| Search | Literal and Java-regex searches scan the complete immutable inventory and return all matching entries inline. Search waits for a reply or caller cancellation; there are no caller-supplied offsets or result-count limits. |
| Analysis state | The socket is exposed only after default auto-analysis completes. Established operations wait for their reply or caller cancellation; there is no fixed response-size ceiling. |
## Admitted function-analysis semantics
| Concern | Ghidra contract |
| ----------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Decompiler lifetime | One persistent `DecompInterface` is opened for the imported Program and disposed during bridge shutdown. Native decompilation has no fixed per-function deadline. External functions or functions without bodies return `null`; cancellation and native failure remain distinct. |
| Serialization | A FIFO sends one Program request at a time. Caller cancellation removes queued work promptly and is passed to active socket waits. This is an adapter safety commitment, not a claim that every Ghidra API is thread-safe. |
| Function identity | Every function result carries the entry address and Ghidra FunctionManager classification for external, thunk, and resolved thunk target. These are observations, not proof that unresolved targetless calls have been recovered. |
| Assembly and pseudocode | Assembly is complete Ghidra Listing text; pseudocode is Ghidra decompiler output. Neither is original source, and cross-provider comparison never treats Hopper and Ghidra text as equal or unequal semantic facts. |
| Instruction fast path | `read_function_instructions` returns every raw Listing instruction for the requested function without invoking the decompiler or whole-program name/string inventories. |
| Calls and references | Callers/callees contain only resolved functions. Reference edges preserve exact ReferenceManager type and call/jump/data/read/write/indirect/computed/conditional/terminal/external facts. Targetless computed flow remains unknown. Synthetic entry-point references without actionable memory sources are omitted explicitly. |
| CFG | Dossiers use `BasicBlockModel` and retain only non-call successors inside the function body. CFG topology is address-normalized for comparison; provider-specific block construction remains a declared difference. |
| P-code value flow | Function dossiers include up to 3,000 high-p-code operations, at most 64 inputs per operation, and 12,000 def-use edges. Results identify memory reads/writes, branches, and calls by p-code opcode, and mark decompiler-dead operations. Large results report truncation, an omitted-operation lower bound, and known omitted-input/edge counts. The dossier is an intra-function graph; `trace_native_values` composes derived parameter/argument and return/output dependencies across resolved calls. It does not resolve memory aliasing, call side effects, runtime behavior, persistent state, or business meaning. |
| Result extent | Function instruction scans and native API boundary observations remain complete. P-code flow is the explicit bounded exception; its count fields distinguish exact retained counts from lower-bound or known-omitted counts. Caller cancellation and provider failures remain distinct. |
`npm run verify:ghidra` compiles the versioned C oracle into debug and stripped
host-native targets (x86-64/AArch64 ELF on Linux or Mach-O on macOS), plus a native
DWARF 4 type-layout object. It proves the admitted operations, external functions, resolved thunks, exports, stripped-name
behavior, direct and targetless indirect calls, typed references, strings/xrefs,
multi-block CFG, bounded p-code def-use/effect links, semantic enhanced workflows, cancellation, startup deadlines,
serialized concurrency, malformed-target rejection, profile identity, and
process/project cleanup against real Ghidra 12.1.4. Unit fixtures separately
cover startup deadlines, process exit, queued cancellation, and malformed wire
output.
`npm run verify:ghidra:cross-format` adds AArch64 ELF, x86-64 PE, and x86-64
Mach-O target coverage. It requires `clang`, LLD, and `lld-link`; set
`REA_CLANG`, `REA_LLD`, or `REA_LLD_LINK` to select them. The verifier checks
these tools before compilation. Keeping this matrix separate lets Linux
host/provider acceptance run with only the host compiler.
`npm run verify:ghidra:windows` and its `-- --x86` variant check the source-owned native x86 and x86-64 PE
fixture, all 25 read-only operations, target/snapshot/import SHA-256 linkage,
and project, endpoint, process, and runtime cleanup. The independent native
lane checks DACLs, handle admission, cancellation, and Job Object lifecycle.
`npm run verify:ghidra:windows:package` packs and installs REA into an isolated
prefix, then checks ordinary-user CLI/MCP operations against canonical contracts.
All lanes require matching native controls; package startup alone establishes
only package compatibility.
docs/managed-code-analysis.md
+46 -228
View File
@@ -1,4 +1,4 @@
# Managed-code analysis and planned extensions
# Managed-code analysis
REA inspects .NET PE/CLI artifacts without loading or executing their code.
It can identify an assembly, inspect metadata and CIL, compare members across
@@ -20,12 +20,11 @@ The seven shipped tools are:
Each has a matching CLI command: replace underscores with hyphens and prefix
the name with `rea`, for example `rea inspect-managed-artifact`.
Native-body bridge mapping remains planned; managed runtime execution is not
part of the current tool set.
Native-body bridge mapping and managed runtime execution are outside the
current tool set.
This guide describes the implementation and verification of
[ADR-0003](adr/0003-managed-code-evidence-and-provider-boundary.md). The canonical
tool inventory is the [build-generated catalog](mcp-contracts.md#generated-catalog).
The canonical tool inventory is the
[build-generated catalog](mcp-contracts.md#generated-catalog).
## Shipped scope
@@ -36,10 +35,8 @@ NativeAOT identity from an ordinary native PE. Inputs without admitted CLI
metadata do not become managed assemblies through naming or routing guesses.
Inspect separately obtained components explicitly and preserve their identities.
The broader deployment classification and native-body mapping below are design
goals from ADR-0003, not claims that every row has an implemented parser. A
valid marker can establish a candidate native boundary, not recovered native
semantics or runtime behavior.
A valid implementation marker establishes a candidate native boundary, not
recovered native semantics or runtime behavior.
## Analysis objective
@@ -57,40 +54,7 @@ collapsing them:
The ordinary workflow ends at question four. Static analysis never loads or
executes the target.
## Planned deployment classification
The intended extended classifier proceeds from outermost authenticated bytes inward:
```text
source path
-> canonical path, size, SHA-256
-> container inventory
-> PE/native header and architecture
-> CLI header and metadata root
-> deployment/runtime markers
-> per-component and per-method implementation availability
-> managed-only, native-only, composed, degraded, or unsupported route
```
The planned result is a vector rather than one label. For example, a single-file modern
.NET deployment can contain a native host, ordinary CIL assemblies, and
ReadyToRun components. Each component receives its own digest, classification,
coverage, and route while retaining the outer bundle commitment.
| Case | Required positive observations | Claims that remain unavailable or inferred |
| --------------------- | ---------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------- |
| .NET Framework | Valid CLI metadata plus bounded Framework-specific target/reference evidence | Exact installed CLR and runtime behavior |
| Modern .NET | Valid CLI metadata plus bounded target/reference/runtime-config evidence | Exact runtime selected on another host |
| Unity Mono | Valid managed assembly plus authenticated Unity context when supplied | Engine behavior or native integration not represented by metadata |
| ReadyToRun | Valid managed metadata plus authenticated ReadyToRun header/sections | Native implementation semantics until a selected deep provider analyzes them |
| C++/CLI | Valid CLI metadata plus mixed-mode PE/header evidence | Mapping managed declarations to native implementations without explicit bridge evidence |
| Single-file | Valid authenticated bundle inventory and component extents | Components that are compressed or encoded by an unsupported bundle version |
| Unity IL2CPP | Authenticated native image/metadata pairing and supported metadata version | Canonical CIL and source-equivalent C# |
| NativeAOT | Native image plus bounded NativeAOT evidence | Ordinary CLI metadata/CIL unless independently present |
| Obfuscated assembly | Same positive byte observations as its underlying deployment form | Meaning inferred only from names |
| Malformed/unsupported | Exact admitted regions and failure locations | Completeness, successfully skipped rows, or semantics beyond the admitted parser boundary |
### NativeAOT metadata recovery
## NativeAOT metadata recovery
NativeAOT removes the ordinary CIL method bodies, so a CIL decompiler cannot
reconstruct those bodies as C# source. The executable remains a native target
@@ -110,28 +74,17 @@ lists supported OS/architecture targets and deployment limitations, while its
[native library guide](https://learn.microsoft.com/en-us/dotnet/core/deploying/native-aot/libraries)
describes exported shared-library entry points.
The [Ghidra NativeAOT analyzer](https://github.com/Washi1337/ghidra-nativeaot)
is a useful optional companion for recovering ReadyToRun metadata. It
rehydrates the `DEHYDRATED_DATA` section and annotates method tables, type
relationships, vtable slots, frozen objects, and strings. It does not recover
original C# method bodies. It is a separate Ghidra extension and interactive
metadata browser; REA does not currently install or invoke it through its
headless bridge. Its README describes ReadyToRun header discovery, including
symbol-based and heuristic paths, so a missed header or unsupported binary
should remain an explicit limitation rather than a failed claim about the
binary's contents.
REA supports an optional headless adapter for the Ghidra NativeAOT analyzer.
Supply a locally built `REA_GHIDRA_NATIVEAOT_JAR`; REA does not install the
extension or its toolchain. The verified boundary is .NET 8.0.22 RTR 9.1,
x86-64 ELF and native Windows PE targets analyzed on Linux x64 with Ghidra
12.1.4 and JDK 21. Other layouts, architectures, and hosts are unsupported.
See [NativeAOT recovery](ghidra-nativeaot.md) for build/configuration, recovered
metadata, derived-memory provenance, and the real verification lane.
The upstream analyzer currently gates its own analysis to x86-64. NativeAOT's
broader platform matrix does not establish REA coverage: PE/COFF, ELF, Mach-O,
shared libraries, target architectures, and runtime metadata versions each need
their own provider verification. REA's experimental Windows Ghidra boundary is
limited to admitted native x86 and x86-64 PE applications, so it does not currently
establish coverage for PE DLLs. Mobile NativeAOT targets should remain
experimental until REA has a matching provider and package-level verification.
For REA's native analysis, preserve the same artifact path and digest used by
`inspect_managed_artifact`, and report recovered type data as provider
observations. Linking those types to native function addresses requires verified
For NativeAOT analysis, preserve the selected native artifact's path and digest
and report recovered type data as provider observations. Linking those types
to native function addresses requires verified
provider evidence; names or vtable similarity alone do not prove that a native
function implements a specific managed method.
@@ -143,43 +96,18 @@ matches the normalized artifact SHA-256. A valid Evidence ID alone does not
establish that those two identities agree. Evidence without a subject remains
usable, with no subject digest available to cross-check.
Every planned operation returns a provider result and Evidence with four
commitment groups:
Artifact inspection returns the selected path, byte length and SHA-256, observed
PE/CLI classification, assembly and module identities, and metadata coverage.
Member inspection binds build-local tokens, signatures, row offsets, method
bodies, and decoded relationships to that artifact and MVID. Missing or malformed
facts remain explicit in coverage and per-member status.
### Artifact commitment
CLI `#GUID` heap values retain their exact 16-byte GUID text. REA does not impose
RFC 4122 UUID version or variant bits on MVID, EncId, or EncBaseId because
ECMA-335 metadata does not require those bit patterns. Tokens and RVAs are
build-local coordinates, not durable cross-build identities or native addresses.
- canonical local path, byte length, and SHA-256;
- outer-container identity and digest when inspecting a component;
- component name/path, byte extent, and SHA-256;
- observed PE machine, CLI flags, and conflicts;
- classification vector, supporting observations, and unresolved dimensions.
### Module commitment
- assembly simple name, version, culture, public-key/token state, flags, and
hash algorithm;
- module name, generation, and MVID;
- target framework and runtime/version strings with their exact metadata
locations;
- provider identity, analysis-affecting parameters, and their profile digest.
CLI `#GUID` heap values are committed as their exact 16-byte GUID text. REA
does not impose RFC 4122 UUID version or variant bits on MVID, EncId, or
EncBaseId because ECMA-335 metadata does not require those bit patterns.
### Entity commitment
- table kind and build-local token;
- declaring scope and normalized CLI signature;
- row/heap/body locations expressed as typed file offsets, RVAs, or CIL
offsets;
- exact raw bytes or bounded value digest;
- raw CIL digest plus separately reported method-header, locals-token, and
exception-region observations where available;
- a deterministic decoded-instruction-tuple digest for completely decoded
CIL, with the limitations below.
### Shipped decoded-CIL fingerprint
## Decoded-CIL fingerprint
`inspect_managed_members` exposes two method CIL hashes with different byte
boundaries:
@@ -208,9 +136,7 @@ reproducible decoded-tuple fingerprint, not a canonical semantic CIL identity:
it does not resolve tokens across MVIDs, does not commit full control flow or
exception semantics, and does not replace the raw CIL digest. Cross-build
comparison uses the value only as one unique-only structural tier and reports
ambiguity when that evidence is insufficient. A future normalized-CIL
commitment would need canonical token/literal identities,
complete branch and switch targets, and explicit locals/exception semantics.
ambiguity when that evidence is insufficient.
For example, a decoded `nop; ret` body serializes exactly as
`[["nop","none",null],["ret","none",null]]` and has digest
@@ -218,73 +144,10 @@ For example, a decoded `nop; ret` body serializes exactly as
This vector specifies the tuple order, `null` handling, absence of whitespace,
UTF-8 encoding, and lowercase hexadecimal output.
### Epistemic commitment
## Member comparison
- authority: static bytes, reconstruction, structural inference, independent
validation, native provider, or future runtime observation;
- state: observed, inferred, unknown, or unavailable;
- confidence independent of authority;
- coverage and admitted/dropped counts;
- exact supporting Evidence IDs and actionable limitations.
A semantic label such as `score_submission_gate` can annotate an entity, but it
is never the entity's identity. The same applies to an obfuscated name, token,
RVA, or decompiler-generated local name.
## Planned static capabilities
The caller-visible grouping and exact tool names are decided with the contracts
that implement them. The underlying capability slices are:
1. **Triage and inventory**: container/component classification, assembly and
module identity, references, files, exported types, resources, attributes,
and runtime markers.
2. **Member inventory**: file-backed types, methods, fields, properties, events,
interfaces, nesting, generics, and normalized signatures.
3. **Method inspection**: headers, max stack, locals, CIL instructions,
constants, metadata operands, exception regions, and implementation flags.
4. **Relationships**: declared overrides, interface implementations, member
references, direct CIL callers/callees, field access, construction, and
interop declarations. Dynamic dispatch remains qualified.
5. **Structural search and comparison**: signature/API/constant/flow anchors,
exact and normalized hashes, competing candidates, and explicit
exact/structural/missing/ambiguous states.
6. **Managed/native composition**: P/Invoke, unmanaged exports, COM/mixed-mode
declarations, ReadyToRun/native bodies, single-file hosts, and authenticated
IL2CPP pairs linked to selected Hopper/Ghidra evidence. The shipped
verification workflow checks P/Invoke declarations against supplied native
export or function Evidence; native-body, thunk, and token-to-address bridge
mapping remain unavailable without explicit provider-supported evidence.
7. **Application graph projection**: managed assembly/module/type/method/field,
P/Invoke, and native-implementation declarations enter the same
Evidence-backed application graph vocabulary as JavaScript/Electron
findings while preserving managed static-analysis authority.
Inventory lists preserve deterministic order and include every fact available
from the admitted PE/CLI streams. Malformed ranges and unsupported metadata are
reported as coverage issues instead of being silently omitted. Search exposes
scanned, matched, returned, and dropped counts. An unresolved indirect call or
a failed signature decode is not silently omitted from a completeness claim.
## Obfuscation-resistant method slices
Names are one weak feature among many. A behavior slice records the smallest
bounded set of observations needed to investigate a proposition:
- normalized declaring/member signatures and inheritance/interface shape;
- framework and application API references;
- exact strings, numeric constants, enum-like values, and serialization keys;
- branch, switch, exception, loop, call, and field-flow shape;
- construction and generic-instantiation sites;
- raw CIL and decoded-tuple commitments, plus separate header, locals, and
exception observations;
- callers, callees, shared fields, and competing candidates; and
- limitations from reflection, dynamic dispatch, generated code, protection,
native transitions, or incomplete coverage.
The workflow produces separate observation, inference, validation, and unknown
tables. It may say that a method is a strong candidate for a role; it cannot
turn that role into the method's durable identity.
Names and semantic labels do not establish durable member identity. Comparison
preserves observed differences, structural inferences, and unknowns separately.
Cross-version matching first prefers exact CIL/signature identity. When that
does not match, it pairs an exact declared type, method name, and raw signature
@@ -310,16 +173,14 @@ Tokens are always remapped through observed structure. A caller cannot carry
## Managed/native boundary rules
| Boundary | Managed observation | Native observation | Permitted link |
| ----------------------- | ----------------------------------------------------------------------- | ------------------------------------------------------------------------------- | -------------------------------------------------------------------- |
| P/Invoke | Module, entry point, charset/calling-convention flags, declaring method | Import/export/symbol/function evidence from the selected deep provider | Exact declared-name/module link or qualified resolution inference |
| COM | Interop attributes, GUIDs, imported interfaces, method signatures | Native registration/vtable evidence when independently available | Identifier/signature inference with explicit environment limitations |
| C++/CLI | Managed declaration and implementation flags | Native body/function evidence | Only a provider-supported bridge observation; never token-as-address |
| ReadyToRun | Component/header and per-method CIL/native availability | Native section/function evidence | Authenticated image mapping with format/profile version |
| Unmanaged export | Export metadata/attribute when present | PE export and native thunk/function | Exact export identity plus provider-qualified address |
| Single-file host | Bundle entry/component identity | Host and native component evidence | Outer bundle plus component digest/extent commitment |
| Unity IL2CPP | Supported metadata entity with authenticated pairing | Generated native function/type evidence | Versioned IL2CPP mapping only; no invented CIL |
| Runtime-resolved native | API/constant/data-flow candidate | Loaded-module/symbol observation from an independently collected runtime source | Static candidate remains inference until separately observed |
`verify_managed_native_boundaries` compares declared P/Invoke module and entry
point identities with supplied native export or function evidence. Name/module
matches establish declaration links; they do not establish that a call executed.
Metadata tokens are not native addresses. Native body mapping for C++/CLI,
ReadyToRun, NativeAOT, and IL2CPP requires separately verified format and provider
support. The managed static tools do not recover those mappings. The optional
[Ghidra NativeAOT extension](ghidra-nativeaot.md) has its own supported boundary.
Declared import inventory supports a bounded positive claim. Its absence does
not exclude dynamic resolution, generated code, protected code, native helpers,
@@ -333,16 +194,12 @@ fixtures and expected semantic facts. An optional real ILSpy lane runs only when
its executable is explicitly supplied; the default lane does not establish
independent pinned-oracle parity. Production inspection needs none of those tools.
- A pinned `System.Reflection.Metadata` differential oracle is planned; it is
not part of the current verifier.
- `ICSharpCode.Decompiler`/`ilspycmd` can supply reconstruction inference.
A BYO reconstruction import records the supplied version
and remains non-canonical. When `REA_ILSPY_CMD_PATH` points to an absolute
runnable `ilspycmd`, `verify:managed` runs a source-owned real ILSpy oracle
and imports its C# output as reconstruction inference against exact static
member Evidence.
- dnlib and Mono.Cecil are potential future differential oracles. Their mutation APIs
are not exposed or included in the production parsing boundary.
- The package contains no .NET runtime, SDK, ILSpy installation, proprietary
assembly, or compiled conformance fixture.
- Setup never installs or upgrades those tools. Doctor may inspect an explicit
@@ -360,29 +217,15 @@ they do not claim that the REA application itself supports Windows. Native
ReadyToRun, C++/CLI, NativeAOT, or IL2CPP coverage is additionally constrained
by the selected Hopper/Ghidra host and format matrix.
## Source-built conformance corpus
## Conformance verification
The current lane uses source-owned byte-built fixtures and malformed-input
regressions, with optional operator-supplied applications and ILSpy verification.
The table below describes the desired expanded corpus; it is not a report that
all compiler-generated forms have been independently verified.
Fixture sources are intentionally small and behavior-focused. Build outputs
are generated outside tracked fixture directories and must not remain after
verification or packaging.
| Corpus slice | Required properties |
| ---------------------- | ------------------------------------------------------------------------------------------ |
| Identity | Assembly/module/version/culture/public key, references, attributes, resources, MVID |
| Signatures | Nested/generic types, arrays, pointers, by-ref, function pointers where supported, varargs |
| CIL | Short/long branches, switch, prefixes, constants, locals, calls, fields, boxing, tokens |
| Exceptions | Catch, finally, fault/filter where the compiler/toolchain supports them |
| Generated structure | Async and iterator state machines, lambdas, closures, properties, events |
| Interop | P/Invoke flags, module/name mapping, managed/unmanaged implementation metadata |
| Architecture | AnyCPU, preferred-32-bit where applicable, x86, x64 |
| Obfuscation resistance | Unicode/meaningless/duplicate-looking names and structure-preserving renames |
| Cross-build comparison | At least two builds with changed MVID/token order/layout and known same/changed behaviors |
| Malformed/adversarial | Truncated/overflowing headers, streams, tables, heaps, signatures, bodies, and EH sections |
Run the deterministic lane with `npm run verify:managed`. See
[testing](testing.md#real-toolchain-verification-lanes) for prerequisites and
coverage. Generated outputs belong outside tracked fixture directories and must
not remain after verification or packaging.
Verification compares semantic facts, not decompiler text. It checks resource
ceilings, deterministic ordering, complete cleanup, CLI/MCP parity, Evidence
@@ -492,32 +335,7 @@ evidence is collected. A direct process capture can retain declared inputs,
outputs, filesystem and protocol activity for a target run; it does not prove
which managed methods executed or how the CLR resolved them.
## Delivery sequence
The managed-code track advances as reviewable pull requests:
1. accepted evidence/provider boundary (this document and ADR);
2. read-only artifact triage and exact identity (shipped);
3. file-backed metadata, signatures, method bodies, raw CIL hashes, and the
decoded-instruction-tuple fingerprint (shipped; complete normalized-CIL
semantics remain planned);
4. obfuscation-resistant slices and cross-version comparison (shipped for
static member observations);
5. decompiler reconstruction import (shipped as analyst inference; REA does
not run ILSpy/dnSpy, and metadata/CIL remain canonical);
6. managed/native composition and truthful deployment degradation (declaration
inventory and native export/function Evidence matching shipped; native-body
bridge mapping remains planned);
7. source-built managed conformance and package/CLI verification (source-owned
PE/CLI corpus shipped through `npm run verify:managed`; optional BYO
`ilspycmd` real-tool oracle shipped through `REA_ILSPY_CMD_PATH`; dnSpy and
pinned Windows checks remain planned); and
8. managed static Evidence projection into the application graph (shipped).
Each implementation PR updates generated product facts only for behavior it
actually ships and states which real-tool checks were performed.
### Method metadata during build comparison
## Method metadata during build comparison
Managed member comparisons report a `metadata` dimension when an observed
MethodDef flags or implementation flags value differs, including accessibility
docs/provider-evaluation.md
-219
View File
@@ -1,219 +0,0 @@
# Static-analysis provider evaluation
This guide describes repository main. Check the [released package boundary](installation.md#released-package-and-main) when using npm.
Ghidra read-only analysis is available on Linux x64/arm64 and macOS x64/arm64.
Install a Ghidra 12.1.x release and the 64-bit full JDK that release declares,
then configure REA to use them. Current 12.1 releases require JDK 21 or newer
and set no maximum. The bridge is verified with Ghidra 12.1.4 and JDK 21.
Each installation needs the matching native decompiler.
REA imports one target into a temporary project and exposes 25 read-only
operations after analysis completes. They cover inventories, search,
decompilation, assembly, function metadata, resolved calls, references, control
flow, instructions, and recovered types. Results identify the provider version,
analysis settings, target digest, and any unavailable or incomplete facts.
On Linux and macOS, `annotate_native_function` additionally edits names and entry
comments atomically, verifies readback, and returns refreshed function analysis.
Changes live only in the session database and leave executable bytes unchanged.
Ghidra does not expose GUI controls; Windows P0 remains read-only.
Windows x64 P0 supports native x86 and x86-64 PE applications on local NTFS with
bundled Job Object ownership, protected runtime DACLs, and handle-based path
admission. Real ordinary-user verification covers all 25 read-only operations
through the packaged CLI and MCP, target integrity, and cleanup. This remains
an experimental boundary; see [Windows Ghidra P0](windows-ghidra-p0.md) and
[issue #527](https://github.com/morluto/rea/issues/527).
[ADR-0001](adr/0001-provider-selection-and-analysis-profiles.md) fixes the
provider registry, deterministic selection, target binding, analysis profile,
snapshot migration, and compatibility semantics that implementation must
follow.
## Approved Ghidra v1 boundary
- Keep the existing provider-neutral CLI and MCP tool names.
- Bind one deep-analysis provider to a target for the target's lifetime; never
fail over silently between Hopper and Ghidra.
- Use an existing Ghidra installation and a compatible Java runtime. Setup must
not install or upgrade Ghidra or Java.
- Run Ghidra headlessly in an owned process with a private temporary project,
startup and cleanup deadlines, caller cancellation, and complete replies.
- Prefer a packaged Java bridge loaded through Ghidra's script path. PyGhidra
remains useful for prototypes but is not a mandatory production dependency.
- Treat authenticated `ping` and `shutdown` as lifecycle proof only. Inventory,
xref, CFG, and decompilation claims require separately admitted operation
contracts and real-provider conformance.
- Implement read-only inventories, assembly, decompilation, function metadata,
calls, references, containment, and complete inventory search first.
- Report GUI cursor/navigation and persistent mutation operations as unavailable
until their semantics and project ownership are explicitly designed.
- Verify real claims on Linux with at least two distinct source-owned binaries.
Verify the Windows host claim with the deterministic source-owned x86 and x86-64 PE
fixtures and all admitted operations; compare normalized semantic facts rather
than provider-specific pseudocode text.
## Shipped foundation boundary
`GHIDRA_INSTALL_DIR` must identify an extracted Ghidra 12.1.x release. Optional
`JAVA_HOME` must identify a 64-bit full JDK inside that installation's
`application.java.min` and `application.java.max` (JDK 21 or newer, with no
maximum, when the release leaves those at the 12.1 defaults). Otherwise doctor
probes `java`/`javac` or `java.exe`/`javac.exe` from `PATH`. Supported Linux and macOS
hosts accept compatible ELF, PE, and Mach-O executable targets. Host admission
and target compatibility are separate checks. Windows x64 P0 admits only native,
non-managed, non-DLL x86 and x86-64 PE applications on fixed local NTFS, with the packaged
native authority available before launch.
The launcher creates one ephemeral runtime root with project,
home/cache/config/data/temp, logs, descriptor, endpoint, target snapshot, and
ownership manifest.
It passes `-readOnly` and `-deleteProject`, and uses Ghidra's default analysis
and resource settings; inherited Java option injection variables are cleared.
On Linux, the mode-0600 descriptor carries the random token without
exposing it in argv or environment and the Java bridge binds a mode-restricted
Unix socket. macOS uses the same local transport. The bridge reports actual
Ghidra/language/compiler/analysis/import-digest metadata and accepts only
authenticated `ping`, `shutdown`, thirteen read-only inventory/load-image methods,
and twelve function-analysis methods. Linux/macOS also admits atomic function
annotation changes in the ephemeral database. Close, cancellation, timeout, malformed protocol,
or process exit stops the owned process resources, closes the socket, and
removes the runtime root.
The experimental Windows transport uses authenticated IPv4 loopback with a
token-free endpoint record. The bundled native authority establishes Job Object
ownership, protected runtime DACLs, and handle-based path admission before
launch. If the bundle or an admission constraint is unavailable, the provider
remains unavailable; a `taskkill` fallback does not establish that authority.
The provider catalog lists 25 read-only Ghidra operations and the Linux/macOS
session annotation operation. GUI cursor/navigation and persistent mutations
remain unavailable; Windows P0 admits no database mutation. The router therefore
reports them unavailable instead of borrowing Hopper semantics or inferring
capability from a successful import.
## Admitted inventory semantics
| Concern | Ghidra contract |
| ---------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Program identity | One `analyzeHeadless` import produces exactly one Program; `list_documents` therefore returns exactly one name. |
| Addresses | Default memory uses lowercase `0x` hexadecimal. Non-default and external spaces use `<percent-encoded-space>:0x<hex>` and remain round-trippable. The handshake commits image base and default address-space name. |
| Symbols | `list_names` includes address-bearing memory and external symbols, including dynamic symbols, while excluding variable and no-address namespace records. Each item reports primary, dynamic, external, symbol type, and source facts. |
| Procedures | Both non-external and external functions are listed. A local thunk remains distinct from its resolved target; exact and qualified name lookup fails on ambiguity rather than guessing. |
| Strings | Only Ghidra-defined string `Data` is observed. Items report charset, byte length, and whether a required null terminator is missing. The API cannot distinguish a present terminator from fixed/Pascal layouts when no terminator is missing, so that state is named `present_or_not_required`. |
| Memory | Memory-block end addresses are exclusive. Read/write/execute, initialization, overlay, address space, and image base are direct Ghidra observations. |
| Inventory | Procedure, symbol, and string listings plus searches return the complete matching collection in one response; callers do not provide offsets or result-count limits. |
| Search | Literal and Java-regex searches scan the complete immutable inventory and return all matching entries inline. Search waits for a reply or caller cancellation; there are no caller-supplied offsets or result-count limits. |
| Analysis state | The socket is exposed only after default auto-analysis completes. Established operations wait for their reply or caller cancellation; there is no fixed response-size ceiling. |
## Admitted function-analysis semantics
| Concern | Ghidra contract |
| ----------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Decompiler lifetime | One persistent `DecompInterface` is opened for the imported Program and disposed during bridge shutdown. Native decompilation has no fixed per-function deadline. External functions or functions without bodies return `null`; cancellation and native failure remain distinct. |
| Serialization | A FIFO sends one Program request at a time. Caller cancellation removes queued work promptly and is passed to active socket waits. This is an adapter safety commitment, not a claim that every Ghidra API is thread-safe. |
| Function identity | Every function result carries the entry address and Ghidra FunctionManager classification for external, thunk, and resolved thunk target. These are observations, not proof that unresolved targetless calls have been recovered. |
| Assembly and pseudocode | Assembly is complete Ghidra Listing text; pseudocode is Ghidra decompiler output. Neither is original source, and cross-provider comparison never treats Hopper and Ghidra text as equal or unequal semantic facts. |
| Instruction fast path | `read_function_instructions` returns every raw Listing instruction for the requested function without invoking the decompiler or whole-program name/string inventories. |
| Calls and references | Callers/callees contain only resolved functions. Reference edges preserve exact ReferenceManager type and call/jump/data/read/write/indirect/computed/conditional/terminal/external facts. Targetless computed flow remains unknown. Synthetic entry-point references without actionable memory sources are omitted explicitly. |
| CFG | Dossiers use `BasicBlockModel` and retain only non-call successors inside the function body. CFG topology is address-normalized for comparison; provider-specific block construction remains a declared difference. |
| P-code value flow | Function dossiers include up to 3,000 high-p-code operations, at most 64 inputs per operation, and 12,000 def-use edges. Results identify memory reads/writes, branches, and calls by p-code opcode, and mark decompiler-dead operations. Large results report truncation, an omitted-operation lower bound, and known omitted-input/edge counts. The dossier is an intra-function graph; `trace_native_values` composes derived parameter/argument and return/output dependencies across resolved calls. It does not resolve memory aliasing, call side effects, runtime behavior, persistent state, or business meaning. |
| Result extent | Function instruction scans and native API boundary observations remain complete. P-code flow is the explicit bounded exception; its count fields distinguish exact retained counts from lower-bound or known-omitted counts. Caller cancellation and provider failures remain distinct. |
`npm run verify:ghidra` compiles the versioned C oracle into debug and stripped
host-native targets (x86-64/AArch64 ELF on Linux or Mach-O on macOS), plus a native
DWARF 4 type-layout object. It proves the admitted operations, external functions, resolved thunks, exports, stripped-name
behavior, direct and targetless indirect calls, typed references, strings/xrefs,
multi-block CFG, bounded p-code def-use/effect links, semantic enhanced workflows, cancellation, startup deadlines,
serialized concurrency, malformed-target rejection, profile identity, and
process/project cleanup against real Ghidra 12.1.4. Unit fixtures separately
cover startup deadlines, process exit, queued cancellation, and malformed wire
output.
`npm run verify:ghidra:cross-format` adds AArch64 ELF, x86-64 PE, and x86-64
Mach-O target coverage. It requires `clang`, LLD, and `lld-link`; set
`REA_CLANG`, `REA_LLD`, or `REA_LLD_LINK` to select them. The verifier checks
these tools before compilation. Keeping this matrix separate lets Linux
host/provider acceptance run with only the host compiler.
`npm run verify:ghidra:windows` and its `-- --x86` variant check the source-owned native x86 and x86-64 PE
fixture, all 25 read-only operations, target/snapshot/import SHA-256 linkage,
and project, endpoint, process, and runtime cleanup. The independent native
lane checks DACLs, handle admission, cancellation, and Job Object lifecycle.
`npm run verify:ghidra:windows:package` packs and installs REA into an isolated
prefix, then checks ordinary-user CLI/MCP operations against canonical contracts.
All lanes require matching native controls; package startup alone establishes
only package compatibility.
## Shared provider-process foundation
`src/process/` now provides the mechanisms that a long-lived Hopper or Ghidra
adapter shares: POSIX run-token-authenticated process-group ownership,
ephemeral temporary runtime roots, one absolute startup deadline, correlated
request cancellation and lifecycle-deadline cleanup, bounded stdout and stderr retention with
exact byte counts, process-exit diagnostics, and bounded TERM-to-KILL shutdown.
Reusable fixtures exercise exit, timeout, cancellation, graceful termination,
forced termination, double-close, spawn failure, and resource release. Windows
Ghidra uses its native Job Object, private runtime, and path-admission boundary;
generic process-tree termination alone cannot establish provider availability.
The foundation does not define a bridge schema, socket framing, health payload,
analysis model, or shutdown acknowledgement. Hopper keeps its authenticated
NDJSON-over-Unix-socket protocol in `src/hopper/`; Ghidra has a separate strict
NDJSON protocol implemented by the packaged Java `HeadlessScript` and reuses
only the generic process mechanisms.
## Shortlist
| Provider | License / automation surface | What it brings | REA fit and blockers |
| ------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| [Ghidra](https://github.com/NationalSecurityAgency/ghidra) | Apache-2.0 source license; `analyzeHeadless`, Java APIs, and PyGhidra | Static analysis, multiple processors and formats, scripting, and project/database workflows | 25 read-only operations shipped on Linux/macOS with installation checks, temporary projects, a serial API queue, and real ELF/PE/Mach-O conformance. Experimental Windows x64 P0 supports native x86 and x86-64 PE on local NTFS with bundled native controls and ordinary-user CLI/MCP verification; Linux/macOS additionally support atomic session function annotations; Windows mutation and all Ghidra GUI operations are unavailable. |
| [Rizin](https://github.com/rizinorg/rizin) / [rz-pipe](https://github.com/rizinorg/rz-pipe) | Rizin repository contains LGPL-3.0 and GPL-3.0 components; `rizin`, `rz-bin`, and language bridges through `rzpipe` | Portable CLI analysis, disassembly/debugging, many architectures and file formats, JSON command output | Good candidate for a process-backed Linux provider and fast metadata fallback. License/component inventory must be preserved; command output needs version-pinned parsers and semantic conformance before evidence is trusted. |
| [LIEF](https://github.com/lief-project/LIEF) | Apache-2.0; C++, Python, and other bindings | Deterministic parsing and modification of ELF, PE, Mach-O, COFF, and related executable formats; headers, sections, symbols, relocations, and functions | Best near-term complement, not a decompiler replacement. It can cover format metadata and artifact evidence without a long-lived analysis process; function semantics, pseudocode, CFG, and cross-reference parity remain out of scope unless separately demonstrated. |
| [Binary Ninja](https://docs.binary.ninja/dev/index.html) | API/documentation components are MIT, while the analysis product is licensed by edition; commercial, Ultimate, or Headless license is required for headless automation | Python/Core/C++/Rust APIs, headless loading, IL layers, function analysis, plugins, and configurable analysis | Strong technical fit for a native provider, especially function dossiers. Commercial licensing, license-secret handling, native runtime packaging, and multithreaded lifecycle rules are material deployment blockers. |
## Recommended order
1. Use the implemented explicit provider registry and target binding without
changing Hopper behavior.
2. Maintain the admitted Ghidra function boundary through the shared
conformance corpus and add formats or semantics only after real proof.
3. Connect Electron/native-add-on application findings to the selected native
analysis provider without introducing provider-prefixed tools.
4. Evaluate LIEF or Rizin later as complementary metadata/disassembly providers,
and Binary Ninja as an optional licensed provider, using the same admission
gate.
## Required admission gate
Before implementation, an adapter proposal must provide:
- a capability matrix covering supported, unsupported, and degraded results;
- provider identity, version, analysis profile, target digest, authority,
limitations, and deterministic locations in every Evidence record;
- bounded subprocess or library lifetime, cancellation, timeouts, and cleanup;
- actionable local diagnostics that retain paths, digests, mismatch locations,
and provider metadata while redacting credentials, authorization headers,
license secrets, and other genuine secrets;
- the same source-owned fixture corpus across architectures and formats;
- repeatable checks for function identity, addresses, strings, names, xrefs,
CFG, and pseudocode wherever those capabilities are claimed.
These gates turn the accepted direction into verified capabilities rather than
prematurely claiming equivalence or redistribution support.
## Primary-source notes
- Ghidra 12.1.4 documents headless batch mode and its Java runtime requirements
in its [release-specific Getting Started guide](https://github.com/NationalSecurityAgency/ghidra/blob/Ghidra_12.1.4_build/GhidraDocs/GettingStarted.md);
the [official release](https://github.com/NationalSecurityAgency/ghidra/releases/tag/Ghidra_12.1.4_build)
supplies the corresponding distribution and checksums.
- Rizin lists supported formats, architectures, tools, and `rzpipe` bridges in
its [repository README](https://github.com/rizinorg/rizin); `rz-pipe` documents
JSON command transport and pipe backends.
- LIEF's [format tutorial](https://lief.re/doc/stable/tutorials/01_play_with_formats.html)
and [Python API](https://lief.re/doc/latest/api/binary_abstraction/python.html)
document parsing and format-agnostic binary access.
- Binary Ninja documents its [headless automation model](https://docs.binary.ninja/dev/batch.html),
[edition licenses](https://docs.binary.ninja/about/license.html), and the
[MIT license for API/documentation components](https://docs.binary.ninja/about/open-source.html).
docs/roadmap.md
+27 -117
View File
@@ -1,125 +1,35 @@
# REA roadmap
## Tool direction
The [platform tracker](https://github.com/morluto/rea/issues/32),
[process/Hopper tracker](https://github.com/morluto/rea/issues/48), and
[browser tracker](https://github.com/morluto/rea/issues/39) track capability and
verification work. Available tools and setup are documented in the
[investigation guides](https://rea.tools/guides/),
[installation guide](installation.md), and
[generated catalog](mcp-contracts.md#generated-catalog).
REA provides inspection, capture, and comparison tools that return evidence
directly. Agents choose commands, paths, browser actions, scripts, and local
fixture servers, then compose experiments from those results. CLI and MCP must
expose equivalent behavior and preserve observations, inferences, and unknowns.
## Priorities
[#555](https://github.com/morluto/rea/pull/555) removed REA permission grants,
scope ceilings, elicitation, and repeated approval fields.
[#572](https://github.com/morluto/rea/pull/572) removed the replay engines,
Node characterization prepare/execute flow, and plan-only managed runtime
correlation tool. Those systems are retired roadmap items. Existing evidence
authority labels remain readable as provenance; they do not imply a current
replay executor.
- Expand native architecture, type, and indirect-call verification.
- Connect static findings and runtime observations across application layers.
- Improve obfuscated managed-code comparisons and verified native links.
- Extend process, protocol, filesystem, reconnect, and build-comparison coverage.
- Evaluate additional providers and targets when an analyst workflow demonstrates
a capability gap.
Local operations use the current user's OS permissions. Input and protocol
validation, provider prerequisites, target identity, cancellation, and owned
resource cleanup remain part of each tool's contract. Setup still discloses
installation changes and requires approval before writing or installing.
## Admission criteria
## Planned work
New capabilities must return useful evidence through shared CLI/MCP workflows,
preserve observations and unknowns, and declare actual authority and lifecycle
constraints. Agents compose experiments using ordinary commands, scripts, and
fixture servers; custom orchestration needs a demonstrated requirement.
The [website guides](https://rea.tools/guides/) and
[generated catalog](product-catalog.json) describe available workflows.
Current development priorities are:
Provider and platform claims require their corresponding
[real verification lanes](testing.md#real-toolchain-verification-lanes).
See [tool design](tool-design.md) for choosing a tool boundary and
[Ghidra semantics](ghidra-provider.md) for the existing deep-provider contract.
- Keep generated metadata and narrative documentation aligned with tools,
providers, setup options and releases.
- Expand native architecture, type and indirect-call verification across
Hopper and Ghidra.
- Connect additional static extractors and runtime observations to
cross-layer feature traces.
- Improve obfuscated .NET comparisons and links between managed findings
and verified native analysis.
- Extend process, protocol, filesystem, reconnect and build-comparison
coverage, plus browser and Electron scenario actions.
- Evaluate native runtime observation through LLDB, Frida, system logs and
API tracing, and additional tools and targets such as Binary Ninja, Rizin,
LIEF, Windows-native workflows, mobile applications and firmware.
Provider additions and platform support require the corresponding real
verification lanes; see [testing](testing.md) and
[provider evaluation](provider-evaluation.md).
## Remaining evidence and provider work
The [platform roadmap](https://github.com/morluto/rea/issues/32),
[process and Hopper fidelity tracker](https://github.com/morluto/rea/issues/48),
and [browser tracker](https://github.com/morluto/rea/issues/39) track remaining
capabilities and proof. Their scope follows direct tools and agent-composed
experiments; custom replay languages and grant systems are not completion gates.
[Inline Evidence composition](https://github.com/morluto/rea/pull/585) now returns
the complete record accepted by comparison tools. Managed inspection uses an
explicit PE/CLI target independently of the native provider selection. Comparisons
preserve unknowns when observation is incomplete, and PTY diagnostics retain the
actual loader or startup failure. Browser cleanup reports incomplete teardown
alongside the primary failure; transient documents invalidate only the affected
frame's WebMCP registrations. Real-provider and platform claims still require
their corresponding verification lanes.
## Shipped behavior
REA setup lets you select agent integration and optional Hopper installation.
It installs the bundled workflow, configures detected agents, and can save
verified paths for an existing Ghidra installation. It configures Claude Code,
Claude Desktop, Codex, Cursor, Gemini CLI, Windsurf, Devin, OpenCode, Antigravity,
GitHub Copilot CLI, Command Code, VS Code, Grok Build, and OMP using each client's configuration format. Grok Bot is detected locally, but its connector is added from the Grok Bot chat because the signed-in account store is not a data-directory file.
Ghidra analysis supports Linux x64/arm64 and macOS x64/arm64 with Ghidra 12.1.x and the
64-bit full JDK that installation declares. Current 12.1 releases require JDK 21
or newer and set no maximum; the bridge is verified with Ghidra 12.1.4 and JDK 21.
Each installation requires the matching native decompiler. The
adapter exposes thirteen inventory operations and twelve function-analysis
operations, for 25 read-only operations total. Linux and macOS additionally support
atomic function-name and entry-comment edits with refreshed analysis; metadata
is discarded on close and executable bytes stay unchanged. Approved setup saves verified
installation paths in agent configurations without installing or changing
Ghidra or Java.
Real-provider verification uses host-native debug and stripped fixtures plus
a native type-layout object. A separate lane covers AArch64 ELF, PE, and
Mach-O cross-target fixtures. See [testing](testing.md) for the prerequisites
and exact commands.
Experimental Windows x64 P0 now supplies bundled Job Object ownership,
protected runtime DACLs, and handle-based admission for native x86 and x86-64 PE
applications on local NTFS. Real ordinary-user CLI/MCP verification covers
all 25 read-only operations and cleanup. The [Windows P0 guide](windows-ghidra-p0.md)
describes this boundary and unverified broader coverage.
## Ghidra maintenance boundary
Extend formats or semantics only with normalized provider-neutral contracts and
real source-owned conformance. Hopper and Ghidra pseudocode or assembly text is
not expected to match; unresolved targetless flow remains unknown. Automatic
Ghidra acquisition, if ever added, remains a separately planned and approved
related-tool change; setup must never install Java.
Maintain Windows P0 with independent DACL readback, handle-based admission,
private authenticated IPC, and Job Object assignment before provider execution.
Process capture, Hopper, and broad filesystem-sensitive workflows remain
separate Windows projects; P0 does not imply their parity.
## Capability-selective setup
Setup already lets users select agent integration and Hopper installation.
Once REA supports installing additional analysis tools, it can expand these
choices to match the work a user wants to do. It may ask whether the operator wants to investigate
native binaries, websites, mobile applications, firmware, or runtime protocols,
then propose only the tools needed for the selected work.
That future installer must preserve the current safety boundary:
- detect and reuse existing tools;
- disclose every source, destination, license, command, and system effect;
- prefer user-local installation;
- install only explicitly selected toolchains;
- require tool-specific authorization for unattended system changes;
- verify each installed tool before reporting readiness.
No placeholder tool choices or speculative installer registry are implemented
until another supported toolchain makes the abstraction concrete.
Setup remains additive, reuses existing tools, and requires approval for disclosed
configuration and Hopper installation changes. Additional installer choices need
an implemented toolchain and explicit scope; setup does not install or upgrade
unrelated runtimes or dependencies.