Compare commits

...
Author SHA1 Message Date
morluto 89bf190ea2 Merge remote-tracking branch 'origin/main' into codex/address-context-recovery
# Conflicts:
#	src/application/AnalysisContextQueries.ts
2026-11-02 00:55:18 +08:00
Kaoru0822andClaude Opus 5.5 44b6384009 fix(native): find procedures by the secondary names exact lookup accepts (#8520)
* fix(native): find procedures by the secondary names exact lookup accepts

search_procedures matched only each procedure's primary name, while exact
lookup also accepts Kuna inventory aliases (versioned-symbol and generated
sub_ names) and Ghidra symbols at an exact function entry. A search for a
name procedure_address resolves returned nothing, and trace_feature
silently missed the procedure.

Both providers now match the same names their exact lookup accepts. A
procedure still appears once under its primary name; the optional
matched_aliases field lists the secondary names that matched, so aliases
never become procedures of their own and list_procedures is unchanged.
The shared domain helper owns that row rule for Kuna and for the Ghidra
host-side Unicode folding pass. The Ghidra analysis profile records the
new search semantics so older snapshots cannot replay alias-blind results.

Fixes #8330

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* fix(ghidra): match leaf spellings of procedure names in search

procedure_address accepts each entry symbol's leaf or namespace-qualified
spelling, but procedure search tested only qualified names, so a regex
such as ^audit_alias$ missed a secondary symbol ns::audit_alias (and ^same$
missed a primary alpha::same) that exact lookup selects.

Lookup and search now share one bridge definition of the names that select
a procedure, so they cannot drift. Matching secondary symbols are still
reported by their qualified names; because a leaf ends its qualified
spelling, the host's Unicode literal pass stays exact over those names.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

---------

Co-authored-by: Claude Opus 5.5 <[email protected]>
2026-11-01 08:53:29 -08:00
Kaoru0822andClaude Opus 5.5 3f61303939 fix(session): clear provider-inferred unavailability after recovery (#8518)
* fix(session): clear provider-inferred unavailability after recovery

A provider adapter or process failure marked every routed operation of
that provider unavailable, but a later success cleared only the operation
that succeeded. Sibling operations, and composed tools built on them, kept
advertising the stale failure until each was retried individually.

Move runtime availability into one owner that records whether each entry
was observed for the operation or inferred from another operation's
provider-wide failure. A success clears its own failure and that provider's
inferred entries while keeping failures observed on other operations or
providers and declared refusals. Outcomes are ordered by dispatch to the
provider client, so an older in-flight success no longer erases a newer
failure while a request queued behind a failure still clears it.

Fixes #8404

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* fix(session): keep newer successes when older failures complete last

Runtime availability kept one merged entry per operation and forgot
successful dispatches. An older provider-wide failure that completed after
a newer success re-inferred unavailability for siblings the newer success
had already proved working, and a superseded refusal could survive its own
later success.

Record only the newest failure and newest success dispatch per operation
and per provider, and project unavailability from them: an operation is
unavailable while its own newest failure is newer than its own newest
success, or, as inferred, while its provider's newest provider-wide failure
is newer than any success from that provider. The projection no longer
depends on reply completion order.

The recovery tests now bound held-reply acknowledgements, release and join
every held call in finally, and compare remediation with the owning
formatters instead of copying their wording.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

---------

Co-authored-by: Claude Opus 5.5 <[email protected]>
2026-11-01 08:53:11 -08:00
Kaoru0822andClaude Opus 5.5 35ae7232bf fix(enhanced): stop call graph traversals at unconfirmed provider cleanup (#8516)
* fix(enhanced): stop call graph traversals at unconfirmed provider cleanup

A relationship read whose provider could not confirm cleanup of resources it
owns was projected into a graph node or call-path failure, the traversal kept
dispatching queued reads, and get_call_graph / trace_call_path returned
success with the cleanup_incomplete status buried inside the result.

Both traversals now read relationships through one shared owner,
readCallRelationships, which classifies a cancelled read or a read with
unconfirmed owned cleanup as terminal. On such a cleanup stop the workflow
returns the provider's own error as the outcome, so its cleanup owner, retry
binding and live resolution stay intact, and keeps the work admitted before
the stop as the workflow's normal result shape with a caveat naming the
stopped procedure and the unexpanded count. The composed-workflow layer
turns that admitted result into partial workflow Evidence on the failure,
and the MCP adapter retains it through the shared partial-error delivery.
Ordinary read failures still produce successful partial results.

Fixes #8369

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* fix(enhanced): retain every carried partial Evidence on enhanced failures

Enhanced-tool failures used the shared partial-error delivery only when this
PR's workflow partial was attached. A failing read that already carried its
own partial Evidence, and analyze_function dossier failures, were projected
inline and never recorded with the session writer, so readback and bundle
export could not recover them. Every enhanced execution failure now goes
through one delivery owner that uses toPartialErrorToolResult with the
session writer; it still falls back to inline when no writer or no valid
Evidence exists.

Tests compare the traversal caveat against its owning formatter instead of
copying the sentence.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

---------

Co-authored-by: Claude Opus 5.5 <[email protected]>
2026-11-01 08:52:53 -08:00
Kaoru0822andClaude Opus 5.5 76cd23d2e8 fix(process): remove unused Darwin helper cache type export (#8511)
Scheduled deep static checks on main fail knip because the
DarwinHelperCache type alias added in #8318 has no consumer. Remove
the alias; callers already use createDarwinHelperCache's inferred
return type.

Co-authored-by: Claude Opus 5.5 <[email protected]>
2026-11-01 08:52:42 -08:00
morluto 3e6a5cb20a fix!: preserve address context input errors (#8419)
* fix: retain address context cleanup failures

* fix!: preserve address context input errors

BREAKING CHANGE: Strict clients that cached the previous full inspect_address_context output schema must refresh tools/list and accept the typed error field on unavailable facets.

* test: use session contract for address context
2026-11-02 00:52:27 +08:00
morluto f4c00055d5 Merge remote-tracking branch 'origin/main' into codex/address-context-recovery
# Conflicts:
#	src/application/AnalysisContextQueries.ts
#	tests/boundary/mcp/addressContextBookmarks.test.ts
2026-11-02 00:50:20 +08:00
morluto 1d343a88e5 feat(kuna): verify release 1.988 (#8509) 2026-11-02 00:25:06 +08:00
morluto df25bc8068 fix(ghidra): retain relocation symbol observations (#8531)
* fix(ghidra): retain relocation symbol observations

* test(ghidra): verify named relocation symbols through CLI

* fix(ghidra): clarify verification and supervise test commands

* test(ghidra): share load verification guidance
2026-11-01 23:53:56 +08:00
ShuaixinHuang aa48d494e9 fix(pe): enforce certificate name string alphabets (#8494)
* fix(pe): enforce certificate name string alphabets

* test(mcp): assert structured absolute-path diagnostics

* test(mcp): assert typed selector and stdio diagnostics

* fix(pe): bound IA5 name scans before validation
2026-11-01 21:24:01 +08:00
ShuaixinHuang 32c253ba31 fix(electron): preserve application argv during active capture (#8492) 2026-11-01 21:23:46 +08:00
morluto bafe3784cd fix(native): report dispatch target applicability (#8400)
* fix(native): report dispatch target applicability

* fix(native): clarify unavailable mode recovery

* test(kuna): avoid overlapping verifier imports

* docs(native): explain mode-specific discovery

* fix(native): preserve unbound provider recovery
2026-11-01 21:17:55 +08:00
morluto d684125eed docs(providers): verify prerequisite recovery and sibling failures 2026-11-01 21:14:27 +08:00
morluto 53363166f0 fix(evidence)!: validate native view parent predicates (#8451)
Reject native view sources whose predicate disagrees with the selected operation. Use the original predicate for existing Evidence; do not alter a record without recomputing its identity.
2026-11-01 20:27:34 +08:00
morluto 3842809c37 docs(setup): clarify Hopper failures and installer verification 2026-11-01 20:27:15 +08:00
gensokyo-dev e5e0dc5542 fix(ci): include hidden VitePress configuration in CodeQL (#8486) 2026-11-01 19:05:37 +08:00
2SinceJuly 1aa6beb5dc fix(decompile): name the differing resume identity (#8465)
Resume refused every identity mismatch with one generic message and
generic input-schema recovery. Report the differing target, profile,
layout, or provider facts with the journal and current values so callers
can choose the matching provider/profile or a new output directory.

Fixes #8349
2026-11-01 18:42:27 +08:00
gensokyo-dev 71746351d3 fix(ci): preserve the package verifier HTTP boundary (#8483) 2026-11-01 18:36:21 +08:00
morluto 293db94dd0 docs: prioritize product contracts over skill guidance 2026-11-01 18:29:34 +08:00
morluto 60c0cf977a docs: clarify recovery for valid lookup failures 2026-11-01 17:53:37 +08:00
morluto 7c755f2147 fix: retain address context cleanup failures (#8417) 2026-11-01 17:24:01 +08:00
morluto 1c1c8f6286 fix(snapshot): report binding mismatches (#8432)
* fix(snapshot): report binding mismatches

* test(snapshot): assert structured mismatch recovery

* docs(snapshot): explain profile comparison recovery

* fix(snapshot): include profile in recovery advice

* docs(snapshot): explain fresh-session profile discovery

* refactor(snapshot): derive admission from binding comparison
2026-11-01 17:23:11 +08:00
gensokyo-dev a6caa79bf4 ci(codeql): separate production scans from test fixtures (#8450) 2026-11-01 15:24:29 +08:00
gensokyo-dev e18594b247 refactor(composition): unify capability bindings and cleanup ownership (#8441)
Bind typed service creation, observed availability and retryable cleanup in one composition owner. Reuse selected CLI/MCP factories and canonical contract groups without copying provider operation inventories.

Closes #8429. Refs #740.

Validated paired real application workflows, actual local Codex CLI Terra/high use, complete Evidence and Unknown checkpoint parity, compiler-backed import boundaries, SDK schemas and final real mixed routing. Automated review findings were fixed at their producing owners. Existing exact-base CodeQL alerts are documented on the PR under the owner-authorized unrelated-check exception; no global alerts or security rules were changed.

Assisted-by: Suika (gensokyo-skills:suika-scatter-gather-planning)
2026-11-01 13:30:21 +08:00
morluto a3f0b22d2c test: compare context guidance with its owner 2026-11-01 09:48:52 +08:00
morluto 8b715f2559 fix: guide address-context comment recovery 2026-11-01 09:45:28 +08:00
morluto 6e9ee97a69 fix!: preserve feature traces when owner lookup is unavailable (#8368)
* fix!: preserve feature traces when owner lookup is unavailable

* test: strengthen feature trace cancellation assertions

* fix: preserve feature trace provider readiness errors

* test: verify all feature trace reference sources

* test: verify IDA guidance at its owning boundary

* style: format feature trace regression
2026-11-01 05:38:40 +08:00
gensokyo-dev f72638f6b1 feat(investigation)!: retain records across engine cleanup (#8361)
Separate engine cleanup, investigation reset and terminal disposal in the shared application owner. Keep stable Evidence IDs and full Unknown revision history through engine close and target changes; serialize reset with all record writers and cleanup owners, optionally acknowledging a complete checkpoint before clearing. Bound retained representation without eviction.

Validation: 43 focused files / 533 distinct tests, source/type/lint/schema/documentation checks, nine genuine source cases across eight families, real Ghidra session identity and cleanup, and nine actual local Codex CLI terra/high runs with independent full-result and checkpoint audits. See docs/verification/investigation-lifecycle.md for measurements, partial attempts and verification limits.

Fixes #8347.

BREAKING CHANGE: close_binary retains investigation records. Use reset_investigation to clear them. Cross-process recovery requires an explicitly acknowledged Evidence bundle checkpoint.

Assisted-by: Suika (gensokyo-skills:suika-scatter-gather-planning)
2026-10-31 21:34:45 +00:00
Kaoru0822andClaude Opus 5.5 d54a4661da fix(ghidra): compare the DOS verifier's CLI window before its own rename (#8339)
The DOS verifier compared a fresh CLI session with an MCP session that
its address-boundary helper had renamed and restored, so the entry's
symbol source differed (imported vs user_defined). Compare the CLI with
the pre-edit MCP window, then assert the restoration changes only the
symbol provenance.

Fixes #8189

Co-authored-by: Claude Opus 5.5 <[email protected]>
2026-10-31 11:13:04 -07:00
Kaoru0822andClaude Opus 5.5 d2518a19f8 feat(native): label binary overview counts with their inventory scope (#8338)
* feat(native): label binary overview counts with their inventory scope

`binary_overview` described its counts as "exhaustive", and its result was
a pair of bare scalars. An agent could read them as a discovery of every
function and string in the file, or as comparable across providers. They
are exact totals of the rows each provider's inventories return.

The shared `projectBinaryOverview` now adds one provider-neutral
`inventory_coverage` record, so Ghidra's direct summary and the
listing fallback carry the same record:
`{ scope: "provider_inventory", discovery_coverage: "unknown" }`.
The record reuses the `discovery_coverage` vocabulary that
`list_procedures({detail:"summary"})` already reports, now owned by one
domain schema. The tool description states the scope and that counts are
not comparable across providers. Counts, segment omission semantics and
limitations are unchanged.

Fixes #8287

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* fix(native): label overviews in the workflow so older snapshots stay valid

Analysis snapshots replay the projected inventory summary. A snapshot saved
before inventory_coverage existed would then fail the overview output schema.
The record describes what overview counts mean, whoever produced them, so
the binary_overview workflow now adds it on both its provider-summary and
listing paths. Provider values and projectBinaryOverview are unchanged, and
the composed overview's linked atomic record stays as the provider
reported it.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* fix(native): refresh composed overviews from before inventory_coverage

Bump the binary_overview workflow revision to 5, so snapshot and CLI replay
recompute a revision-4 composed overview from its atomic record. Replay
would otherwise return the cached overview without its count labels. The
revision-4 case joins the existing refresh scenario. The atomic Evidence
is not rewritten.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

---------

Co-authored-by: Claude Opus 5.5 <[email protected]>
2026-10-31 11:12:52 -07:00
Kaoru0822andClaude Opus 5.5 a36c9dd9c8 fix(kuna): omit Mach-O data symbols from the procedure inventory (#8335)
* fix(kuna): omit Mach-O data symbols from the procedure inventory

Kuna's `functions --json` registers every Mach-O export as a function
entry, so globals in `__DATA,__data`, constants in `__TEXT,__const` and
`__mh_execute_header` were listed, counted and decompiled as procedures.

The Kuna session now classifies each inventory entry with the target's own
section table, read by the shared `inspect_macho` load-command reader from
the header bytes kept while its private copy is hashed. Entries in sections
that declare no instructions, or in the header and load commands, are data:
listings and searches omit them and report per-region counts, and
procedure operations refuse them by name or address while `address_name`
and `xrefs` still accept them. Instruction sections, symbol stubs and
symbol-pointer import slots are kept; addresses outside every section stay
with an explicit limitation, an unreadable section table keeps the whole
inventory unclassified, and ELF inventories are unchanged. Composed
workflows such as `decompile_all` and `binary_overview` follow the listing.

Fixes #8168

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* fix(kuna): keep an entry's classification caveat on direct answers

Resolution now returns the selected entry's Mach-O placement and the
caveat it leaves open, built by the same constructor as the listings: the
one-entry unplaced wording, or the unreadable section table's reason.
Pseudocode, assembly, analyze_function (including the dossier's own
limitations), procedure_address, call queries by name and address_name
carry it; code and import-slot entries carry none.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

---------

Co-authored-by: Claude Opus 5.5 <[email protected]>
2026-10-31 11:12:41 -07:00
Kaoru0822andClaude Opus 5.5 8a6ddec83a fix(mcp): name the close_binary retry for retained cleanup (#8329)
* fix(mcp): name the close_binary retry for retained cleanup

A close_binary that retired its target but could not confirm cleanup
returned cleanup_incomplete with retryable: false and only the generic
"Review the reported local resources" remediation, although the session
keeps every failed owner and the next close_binary on the same connection
retries them. Agents could abandon the connection that owns the only retry.

The MCP close tool now passes its retry operation to the session. The
session binds that typed AnalysisCleanupRetry to a failed retirement only
while it retains the cleanup for its next close; the close tool binds it to
process-capture failures its own scope retains and to compositions whose
session part was offered. Projection then reports
recovery.kind=retry_cleanup and appends a conditional instruction: keep the
connection, let the condition settle or resolve it, call close_binary
without snapshot_path and check the result. The hint disappears once that
cleanup resolves. retryable keeps describing the failed operation.

One-shot CLI closes, open_binary, analysis tools and failed snapshot saves
(which keep the target open) offer no retry. Cleanup composition copies the
owner's remediation without a retry offer, so the text appears once.

Fixes #8211

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* test: derive close retry guidance from its owning formatters

The new close_binary retry assertions froze ProviderCleanupError's
explanatory sentence, so a harmless wording edit would break them. The MCP
test now compares delivered remediation with analysisErrorRemediationAction
applied to the delivered typed error plus cleanupRetryRemediation. The
domain tests use the same formatters, check that unbound and released errors
carry no retry text, and check that a bound composition adds the offer once.
Recovery fields, cleanup state and retry behavior keep separate assertions.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

---------

Co-authored-by: Claude Opus 5.5 <[email protected]>
2026-10-31 11:12:30 -07:00
Kaoru0822andClaude Opus 5.5 c534e249d1 perf(process): reuse a verified Darwin ownership helper across processes (#8318)
* perf(process): reuse a verified Darwin ownership helper across processes

Every REA process on macOS compiled the Swift ownership helper into a fresh
temporary root and then paid macOS's first-launch assessment of that new
executable: 408 ms + 325 ms of a 1.4 s `rea inspect-macho /bin/ls`.

With the default xcrun compiler and module cache, the reader now keeps one
content-addressed copy under ~/Library/Caches/rea/process-helper/<key>/.
The key covers the helper sources, toolchain selection, architecture and
Darwin release. A copy is executed only after the cache root, entry and
files are verified as current-user owned, not group- or world-writable,
not links, and digest-matched to the entry manifest. Malformed entries
owned by the user are removed and rebuilt; linked or foreign entries are
left untouched and the per-process compile runs as before. Any cache
failure falls back to that compile, so the cache never fails an ownership
operation.

A miss compiles exactly as before, then publishes a copy through an atomic
rename, keeping the first concurrent publisher, and runs the shared copy.
That way its first-launch assessment is paid once. Per-run roots keep their
existing cleanup; published entries are never removed by close or exit.
An explicit REA_DARWIN_SWIFTC_PATH or REA_DARWIN_MODULE_CACHE_PATH keeps the
per-process compile and writes nothing to the cache. `rea uninstall
--purge-data` removes the namespace.

Fixes #2483

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* fix(process): harden the shared Darwin helper cache

Address review findings on the shared ownership-helper cache:

- Verify every real parent of the cache root: each must belong to the user
  or root and not be writable by others (sticky root-owned directories
  excepted), so no other user can rename a verified entry. Execute through
  the real path.
- Open manifest and helper through the owned non-blocking regular-file
  reader, so a FIFO cannot block preparation, and retain handles whose close
  fails.
- Key entries on the swiftc and SDK that xcrun itself resolves, including
  their real paths, sizes and modification times. DEVELOPER_DIR, TOOLCHAINS,
  xcode-select and in-place updates therefore select new entries.
- Never remove or replace an invalid or published entry. Removal could race
  another process that has verified or is running that path, so such an
  entry falls back to the per-process compile.
- Give the cache a close() that retries retained staging removals and
  handle closes and reports ProviderCleanupError while any remain. The
  reader closes it with its per-run root.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* fix(process): keep cache lookups owned and fall back from unlaunchable helpers

Address the second round of review on the shared ownership-helper cache:

- Pass the preparation signal to both xcrun toolchain lookups, and wait for
  every started lookup before reporting the first failure. Cancellation and
  close then stop a stalled lookup instead of waiting out its deadline.
  A failed or cancelled key computation is retried by the next lookup.
- When the verified shared copy cannot launch (EACCES, EPERM, ENOEXEC,
  ENOENT or an unrequested SIGKILL, for example from a noexec home volume),
  the reader stops using the cache. It reuses its own compiled helper or
  compiles one, and retries the read once.
- Anchor the cache's returned cleanup errors as cleanupRoot does, so every
  error returned while cleanup was pending resolves after a retry succeeds.
- Build the cache path with appendCallerPath, so a home with symlinks and
  `..` keeps the caller's meaning instead of being normalized lexically.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* fix(process): keep concurrent fallbacks and dual cleanup failures

Each helper read now records before awaiting whether it launched the shared
copy, so a concurrent read's fallback cannot hide its own. When the per-run
root and the cache's retained resources both fail one close, the returned
error composes both. It stays incomplete until both owners recover.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* fix(process): refuse shared helpers that an ACL lets others write

macOS ACLs can grant writes that the POSIX mode bits do not show. On
Darwin, lookup and publication list the real ancestors, the entry and its
files with `ls -lde`, and refuse sharing when any ACL entry allows a
write-class right. The default `group:everyone deny delete` entries on
macOS home folders are unaffected. The unlaunchable-helper test now removes
the execute bit from a verified copy, which refuses launch on both Linux
and macOS.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

---------

Co-authored-by: Claude Opus 5.5 <[email protected]>
2026-10-31 11:12:02 -07:00
Kaoru0822andClaude Opus 5.5 41e7333cb5 feat(guidance): distinguish focused matches from broad parent sections (#8305)
* feat(guidance): distinguish focused matches from broad parent sections

Guidance search treated a parent section, including every descendant, as one
match. Terms scattered across unrelated subsections made a 49K-character root
the only result for "switch provider annotations", and every ancestor of a
focused hit was listed alongside it.

The section parser now records each section's own text (heading and body
before its first subsection) and its parent. Search keeps the same admission
rule but classifies each match by match_scope: "section" when every term is
in the section's own text, "subsections" when the terms are only spread
across subsections. Ancestors whose terms all sit inside one subsection are
no longer listed, section matches rank first, and subsections matches carry
own_text_terms plus bounded greedy pointers to the subsections holding the
remaining terms. Every match reports content_length so callers can judge a
read before making it. Read still returns the complete section.

Own text is tokenized once per request instead of each expanded subtree.

Fixes #8290

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* test(guidance): require the scattered-match checks to run

Count the subsections matches the real-bundle test inspects and fail when
none were checked, and document that a parent whose own text holds every
term stays listed beside its matching subsection.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

---------

Co-authored-by: Claude Opus 5.5 <[email protected]>
2026-10-31 11:11:05 -07:00
morluto 60149282fc test: strengthen integration oracles and verification (#8295)
* test: strengthen integration oracles and verification

* test: bound lifecycle gates and strengthen workflow checks

* test: verify complete paged and extracted results

* test: preserve bytecode and cancellation evidence

* test: strengthen retained evidence and fixture cleanup

* test: require cross-projection bridge identities

* test: retain fixture owners through cleanup failures

* test: verify graph examples and project selectors

* test: tighten provider and lifecycle regressions

* test: consolidate hex coverage and bound cutter cleanup

* test: bound subprocess output waits and strengthen graph checks

* test: check mcp recovery and preserve cleanup owners

* test: verify structured mcp argument failures

* test: preserve artifact cleanup and gdb limit evidence

* test: preserve evidence identities and bound provider gates

* test: retain browser and comparison cleanup owners

* test: bound startup and strengthen fixture oracles

* test: bound lifecycle gates and verify recovered evidence

* test: verify public outcomes and lifecycle ownership

* test: assert precise evidence and release lifecycle gates

* test: verify selected facts and bound cleanup waits

* test: check retained evidence and decoded records

* test: verify final reports and retain cleanup ownership

* test: bound reviewed startup gates and retain cleanup

* test: verify PE errors and preserve MCP cleanup

* test: preserve MCP fixture cleanup ownership

* test: release provider gates before late startup
2026-11-01 00:15:28 +08:00
Kaoru0822andClaude Opus 5.5 2d5dec26c3 fix(ghidra): assemble the Swift verifier's unresolved-jump fixture before linking (#8325)
swiftc rejects assembly sources as inputs, so verify:ghidra:swift failed
during fixture preparation before any provider check. Assemble both
assembly fixtures to objects with xcrun clang and link only objects.

Fixes #8188

Co-authored-by: Claude Opus 5.5 <[email protected]>
2026-10-31 09:01:06 -07:00
Kaoru0822andClaude Opus 5.5 da0e9669a8 fix(decompile): keep provider no-pseudocode facts on batch items (#8323)
* fix(decompile): keep provider no-pseudocode facts on batch items

A batch_decompile item whose provider answered without pseudocode was
reported as unreadable output with doctor guidance, and the provider's
reason, such as a Kuna export failure for that entry, survived only in the
batch-wide limitations. The workflow read only the observation's result.

Read each item's complete observation. A null answer is now a
capability_unavailable item naming the provider, and every item answered
without usable text keeps that observation's own limitations, so a
batch-only Evidence export retains the association. Empty or non-text
pseudocode stays unreadable output; typed provider failures and successful
items are unchanged, and the batch Evidence still aggregates every caveat.
The batch workflow revision changes so earlier snapshots are not replayed
with the previous item semantics.

Fixes #8226

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* fix(decompile): match null-pseudocode guidance to reported caveats

A null answer without caveats told the caller to read item limitations
that the batch omits when the list is empty. The guidance now depends on
whether that observation reported caveats, and says the provider gave no
further explanation when it did not.

The batch tests now select their contracts by name from the enhanced and
session families instead of the full catalog.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

---------

Co-authored-by: Claude Opus 5.5 <[email protected]>
2026-10-31 09:00:48 -07:00
Kaoru0822andClaude Opus 5.5 73ff6ca0c8 fix(native): separate value-trace page omission from traversal truncation (#8309)
trace_native_values set one truncated flag for two different facts: the
returned page omits retained nodes, and a budget stopped or narrowed the
traversal. A last page of a complete trace and a node- or edge-capped trace
both reported truncated: true with next_offset null and no unknown naming the
cap, so a caller could not tell whether to follow pages, raise a budget, or
treat the examined graph as exhausted.

Results now carry page_truncated for page omission and a traversal object
that is complete, or truncated with the observed reasons in schema order:
max_depth, max_functions, max_call_sites, max_nodes, max_edges,
node_payload_bytes, provider_pcode_truncated and ambiguous_call_candidates.
truncated keeps its previous meaning as the union of both.

A new budget owner admits nodes and edges and records every cause together
with a located unknown, only after a budget rejects work; an exactly filled
budget is no longer truncation. Depth and ambiguous-candidate limits are
resolved after traversal, so a destination analyzed or queued through a
shallower path is no longer reported as depth-limited. Once an edge is
rejected, remaining call sites in that procedure are not resolved.

Fixes #8283

Co-authored-by: Claude Opus 5.5 <[email protected]>
2026-10-31 08:59:58 -07:00
Kaoru0822andClaude Opus 5.5 c287a73276 fix(ghidra): report unbacked file-offset addresses without schema repair (#8308)
* fix(ghidra): report unbacked file-offset addresses without schema repair

address_to_file_offset projected a well-formed address in uninitialized
or unmapped Ghidra memory as generic invalid input: the headline said the
arguments were invalid and recovery pointed at the input schema, while the
real reason survived only in a nested issue.

The bridge now gives each outcome its own code (address_unmapped,
address_uninitialized, file_mapping_absent, file_mapping_ambiguous,
file_offset_unrepresentable) instead of folding them into invalid_request.
The Ghidra projection keeps the bridge reason as the headline, points to
inspect_native_load_image, records the caller's exact address and the
file_mapping outcome, and scopes diagnostics to the request.
AnalysisInputError gains conformsToInputContract so a conforming request
omits recovery.kind=inspect_input_contract. Malformed addresses and other
bridge refusals keep their existing projection.

Fixes #8244

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* test(ghidra): match fixture block ends to bridge conventions

The fake bridge treated .text's exclusive list_segments end (0x1100) as an
inclusive bound, so address_to_file_offset mapped an address outside the
block. Give the fixture's blocks one owner (start and size) and derive each
reply from the production bridge convention: exclusive list_segments ends,
inclusive byte-search coverage, half-open containment for file offsets.

Add an MCP boundary test that reads the block edges from list_segments and
search_bytes, maps the last file-backed byte and refuses the first address
past each block with its outcome code.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

---------

Co-authored-by: Claude Opus 5.5 <[email protected]>
2026-10-31 08:59:00 -07:00
Kaoru0822andClaude Opus 5.5 7c9ed94259 fix(ghidra): decide batch names independently of request order (#8306)
* fix(ghidra): decide batch names independently of request order

The Ghidra bridge applied set_addresses_names in JSON member order, each
address in its own transaction, so a reordered but equivalent map could give
a shared name, or a name released by another selected address, to a
different address.

The bridge now resolves every destination (namespace and leaf name) against
the names held before the batch. Addresses that request the same destination,
or a name that another selected address holds, are rejected without applying;
the remaining edits are independent and apply as before. Both naming
operations report each failure's canonical address, reason and related
addresses. The adapter projects the bridge reply to the unchanged boolean
results and keeps the reply as raw_result only when an address failed.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* fix(ghidra): report each contested batch name once

A batch conflict listed every other requester on each failure, so k
addresses competing for one name produced k(k-1) related addresses, and
the adapter retained the whole reply, booleans included, as raw_result.

The bridge now reports each contested destination once in a conflicts
table with its namespace, name, requesting and holding addresses, and
each conflict failure refers to it by index. Conflicts are ordered by
their first requester, so indexes do not depend on member order. Single
and batch naming share one planner and reply layout. The adapter
validates the table in linear time and keeps only failures and conflicts
in raw_result.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

---------

Co-authored-by: Claude Opus 5.5 <[email protected]>
2026-10-31 08:53:11 -07:00
gensokyo-dev a989c0e458 refactor(evidence): unify retained inputs across domains (#8299)
Declare logical Evidence sources once and resolve exact retained inputs through
shared application workflows across 20 composition tools. Preserve domain-owned
authentication, source closure, caller choices, coverage and resource admission.

Validation includes public CLI/MCP regressions, local Codex terra/high trials
and fresh-import real-artifact workflows. Measurements, failed attempts and
coverage limits are recorded in docs/verification/evidence-input-composition-2026-10-31.md.

Fixes #8293.
Fixes #8281.

Assisted-by: Suika (gensokyo-skills:suika-scatter-gather-planning)
2026-10-31 14:45:21 +00:00
morluto 23de10bb28 fix: streamline tests and preserve Darwin process ownership (#8275)
* test: strengthen suite guards and regression oracles

* test: share JSON client registration contract

* test: reuse fixtures and reduce repeated setup

* fix(process): preserve ambiguous Darwin token ownership

* test(process): bound child readiness waits
2026-10-31 16:43:04 +08:00
Entrpiandmorluto b43c329df7 feat(artifacts): inventory and extract UF2 and DfuSe firmware containers (#8230)
* feat(artifacts): inventory and extract UF2 and DfuSe regions

open_binary refused UF2 and DfuSe firmware containers as an unsupported
format, so inspect_artifact and extract_artifact could not show which
bytes a drag-and-drop or DFU update writes at which addresses.

UF2 is recognized by a complete first block with all three magic
numbers. Valid main-flash payloads are joined into address-contiguous
regions per familyID, named family-0x<id>/0x<start>-0x<end>.bin, and
known IDs are named from a small table copied from Microsoft's
uf2families.json at a pinned commit. Numbering is counted per familyID
and block count: absent, repeated, out-of-order and out-of-range block
numbers are reported, and absent blocks leave coverage unknown.
Not-main-flash blocks are excluded and reported; file-container blocks
are reported and leave coverage unknown; invalid blocks and a partial
final block are excluded with their position. A rewrite of one address
with different bytes fails closed, naming both block spans.

DfuSe is recognized only by its prefix together with a DFU 1.1a suffix.
Each element becomes one region at its file range with its target's
alternate setting and name. Structural faults fail closed with a byte
offset; overlapping elements stay separate and are reported. A suffix
CRC mismatch follows integrity_policy: fail refuses with a dfu-crc32
integrity error, and record-and-continue records a root contradiction
that extraction also carries.

HEX, UF2 and DfuSe share one retained-region reader and one run
merger, so the HEX decoder and reader lose their private copies.

* docs(artifacts): document UF2 and DfuSe images and their esptool lane

Describe recognition, region names, metadata, numbering, failures, the
DfuSe suffix CRC under integrity_policy, and limits, and link the page
from the CLI and firmware guides and the agent skill reference.

npm run verify:uf2 runs esptool merge-bin --format uf2 on generated
inputs at known addresses, with and without MD5, with 256-byte
payloads, and on a concatenation of two chips' outputs. Inspection and
extraction through the built CLI must return every input at its address
under the familyID esptool itself reports.

* fix(artifacts): accept both DfuSe image size conventions

UM0391 calls DFUImageSize the file length, and dfu-util's dfuse-pack.py
writes that value, while ST's DfuFileMgr, MicroPython's dfu.py and pydfu
leave the 16-byte suffix out. The decoder accepted only the second form,
so it refused dfuse-pack.py output as malformed.

Accept either value and keep the declared one in image_size; any other
value still fails with the suffix offset and the file length. The tests
pin both producers' output for one image, CRC included, from their
build() functions transcribed and run under Python 3.

* fix(artifacts): keep reused UF2 block numbers from hiding missing blocks

esptool numbers every merged input from 0, so two inputs with the same
block count share one numbering group. A block lost from one input left
its number present in the other, and the file reported complete
coverage. Each block number now records the target addresses it was
seen at: equal address counts are complete copies, a resend to the same
address is one block, and unequal counts make coverage unknown with a
limitation and ambiguous_numbering_group_count. sequence_count becomes
numbering_group_count, and region block positions become
first_file_block and last_file_block.

Review follow-ups in the same change:
- compare_artifacts compares a DfuSe element's alternate setting and
  target name, so bytes moved to another device memory are changed;
- the shared reader keeps no region bytes outside its map after close;
- one domain type replaces four identical decoder failure types;
- the UF2 decoder returns a typed limit instead of throwing on bytes
  beyond its admitted size, and caps retained regions, not runs;
- DfuSe caps elements with bytes only, names empty elements and
  element-less targets, drops the constant bVersion and bcdDFU fields,
  and says whether the CRC matches when DFUImageSize fits neither
  convention;
- classification evidence carries a suffix only when the probe saw the
  end, and root classification reads the tail only for a DfuSe prefix;
- the UF2 domain region owns family_name, and the root records the
  family registry revision; the table's MIT notice is preserved;
- docs pin the producer revisions (dfu-util ee6b393d and f9a537c9,
  MicroPython cb99ca98), give the MD5 structure's real meaning, and
  describe container contradictions in the MCP contract guide;
- tests and the esptool lane add the equal-count inputs and a block
  lost from one of them, accepted boundaries beside each refusal, a
  DfuSe root larger than the classification prefix, and CLI refusals.

* feat(artifacts): name the UF2 families that LibreTiny builds for

LibreTiny writes UF2 for Realtek, Beken and Bouffalo Lab modules. The
Microsoft registry revision REA already cites (cf3bd7f9) lists eight of
those families: RTL8710A, RTL8710B, RTL8720C, RTL8720D, BK7231U, BK7231N,
BK7251 and BL602. Their regions now carry those names. LN882H, which
that revision omits, keeps its familyID with no name.

* fix(artifacts): count DfuSe declarations without bytes instead of listing them

Empty DfuSe elements have no entry, so DFUSE_MAX_ELEMENTS does not cap
them, and the target walk kept a description string for each one. At 8
bytes per element, a file under the 128 MiB ceiling can declare about
16.7 million of them. The walk now counts every declaration without bytes
and builds only the first NAMED_ELEMENTS descriptions, which is all that
the limitation shows. The limitation text and empty_element_count do not
change. The guide now says that both DfuSe limitations name the first four.

On this host, decoding 4,000,000 empty elements (a 32 MB file) ran out of
memory under a 128 MB V8 heap before this change, and it peaked at 1.59 GB
RSS with the default heap. It now decodes in 114 ms with a 220 MB peak.

---------

Co-authored-by: morluto <[email protected]>
2026-10-31 16:27:00 +08:00
22e5aa97af feat(artifacts): inspect classic iNES container regions (#2816)
* feat(artifacts): inspect classic iNES container regions

* fix(artifacts): preserve iNES source disappearance as integrity failure

* fix(artifacts): revalidate iNES source before parsing header

* fix(artifacts): validate admitted roots before classification

* fix(artifacts): normalize source loss during root hashing

* fix(artifacts): bind iNES extraction admission to inventory

* test(artifacts): clarify non-executing inspection wording

* fix: preserve binary plist detection with extended magic probe

* test(browser): wait for popup document loads before capture

* fix(artifacts): honor cancellation across source validation

* fix(artifacts): prefer artifact magic over implicit database suffix

* fix(artifacts): select extraction readers from scanned format

* fix(artifacts): complete prefix reads and share ZIP format admission

* fix(browser): budget scenario cleanup for content drain and shutdown

* test(artifacts): admit valid universal inputs before routing

* test(artifacts): verify iNES workflows on macOS

* test(ci): own iNES macOS coverage and portable fixture paths

* fix: preserve cleanup retries and avoid duplicate inventory parsing

* fix(contracts): share ZIP target formats across MCP lifecycle schemas

* fix(target): preserve typed cancellation during descriptor admission

* refactor(artifacts): use JSON-safe inventory producer output directly

* test(artifacts): share file read setup and satisfy CI checks

* fix(artifacts): classify changed admission as integrity failure

* ci: run Mach-O summary delivery coverage on macOS

* test(javascript): align parameter provenance with main contract

* test(browser): await discoverable fixture pages before runtime verification

* fix(artifacts): pass target metadata to format detection

* fix(artifacts): bind extraction to opened target

* fix: repair the static-check failures on the default branch (#2895)

* fix: repair the static-check failures on the default branch

The default branch fails its own required `Static checks` job three ways, so
every pull request reports that job as failed regardless of its own change.

- `npm run format:check` reports `src/domain/functionComparison.test.ts` and
  `src/ida/IdaOperationRunner.ts`.
- `npm run lint` reports a duplicate `import { join } from "node:path"` in
  `tests/boundary/mcp/ghidraInventoryMcp.test.ts` (lines 4 and 17).
- `npm run typecheck` reports `format: "devin"` missing from the client format
  union in `src/application/SupportedClients.ts`, and an unchecked index in
  `tests/boundary/cli/clientProfileLifecycle.test.ts`, whose pair array is
  inferred as `string[][]`.

Apply the pinned formatter, drop the duplicate import, list the `"devin"` format
that its own definition already uses, and mark the pair array `as const`.

Validation: `npm run check:fast` and `npm run format:check` pass.

* fix(setup): keep Devin on the JSON configuration dialect

---------

Co-authored-by: 吴佳浩 <[email protected]>
Co-authored-by: morluto <[email protected]>
(cherry picked from commit df3d96da80)

* fix(artifacts): bind session inventories to target identity

* test(browser): bound fixture readiness acknowledgements

---------

Co-authored-by: 64579525 <[email protected]>
Co-authored-by: 吴佳浩Alben <[email protected]>
Co-authored-by: 吴佳浩 <[email protected]>
2026-10-31 15:49:16 +08:00
Entrpi e669365c3b feat(ghidra): admit RISC-V ilp32 ELF with a derived calling convention (#8229)
* feat(ghidra): record the RISC-V float-ABI calling-convention decision

Decision D3 (#7477), provisional: admit ELF32 soft-float (ilp32) RISC-V
images through a lane that keeps the pinned RISCV:LE:32:default language and
corrects the calling convention in the bridge.

This commit is the contract only; admission is unchanged.

- The domain owns the e_flags float-ABI vocabulary and parses
  .riscv.attributes and Tag_RISCV_arch into a declared, absent or
  unavailable ISA declaration.
- The Ghidra profile owns the lane table (class and float ABI), lane ids
  built from fields, the ilp32-prototype-v1 policy and its __rea_ilp32
  prototype, and the extensions Ghidra 12.1.4 decodes with p-code.
- The session handshake accepts the bridge's calling-convention report.
- docs/ghidra-riscv.md keeps the decision record and per-ABI policies for
  ilp32 and ilp32f; a test keeps it in step with the tables.
- tests/conformance/c/riscv-float-abi.c is the source the policies were
  measured on.

* feat(ghidra): admit RISC-V ilp32 ELF with a derived calling convention

ESP32-C2, C3, C5, C6, C61, H2 and H21 builds are ELF32 soft-float (ilp32)
images, which the RISC-V lane refused at its double-float check. Ghidra
12.1.4's generic RV32 compiler specification models ilp32d, so its stock
prototype would read float and double arguments from fa0-fa7.

- Target resolution reads the one SHT_RISCV_ATTRIBUTES section through the
  same bounded ELF declaration reader as range validation, and records
  Tag_RISCV_arch as declared, absent or unavailable.
- Admission selects a lane from the class and float-ABI table. The soft-float
  lane also requires an RV32I Tag_RISCV_arch that declares only extensions
  Ghidra decodes with p-code; the refusal names any others (Zcb, Zcmp, Zcmt,
  Zb*, xesp*).
- Soft-float profiles commit float_abi, arch, riscv_calling_convention
  ilp32-prototype-v1 and the lane elf32-exec-le-soft-float-generic-v1.
  Double-float profiles are unchanged.
- ReaGhidraRiscvCallingConvention.java runs after analysis. It derives
  __rea_ilp32 from the default prototype without floating-point entries and
  rules, with fs0-fs11 killed by calls, self-checks its storage and assigns
  it to every non-thunk function, external function and function
  definition. Its report is written last.
- The provider client requires that report exactly when the profile commits
  the policy, before any operation, and keeps it as session limitations and
  caller-project history.

ELF32 single-float (ilp32f), quad-float and ELF64 soft- or single-float
images stay refused, now with a reason built from the lane table.

* test(ghidra): verify RISC-V ilp32 against real Ghidra

verify:ghidra:riscv now also builds tests/conformance/c/riscv-float-abi.c
with ESP32-C3's -march=rv32imc_zicsr_zifencei -mabi=ilp32, typed with DWARF
and linked again with --strip-all. Every allocated section of the stripped
image must keep its address and bytes.

- riscv-checks.mjs holds the ilp32 storage of each fixture function together
  with the words the compiler's call site writes there. An LLVM objdump
  reading of the caller must find every word at its stated location: an
  unaligned named double, the a7/stack split, two floats in an integer pair,
  a large structure by reference, aligned variadic pairs and an indirect
  call.
- Through CLI and stdio MCP, the soft-float builds must report the soft-float
  lane and ilp32-prototype-v1, __rea_ilp32 with exactly that storage (one
  integer word per location when stripped), caller pseudocode that passes the
  source constants and names no floating-point register, the prototype
  outcome in retained Evidence, and decoded fence.i and csrr.
- The RV32 double-float image relabelled as soft-float, refused before, is
  admitted through the new lane. RV64 soft-float, single-float and unsupported
  bits are refused with their reasons.
- riscv-checks.test.ts runs the checks offline against recorded LLVM and
  Ghidra output, including Ghidra's stock ilp32d prototypes as a negative
  control.

* fix(ghidra): pass RISC-V ilp32 values over 8 bytes by reference

The psABI passes any value wider than 2 x XLEN by reference, scalars
included. __rea_ilp32 converted only structures and unions over 8 bytes to
pointers, so a long double (16 bytes) was joined into a1-a4. Measured on
Ghidra 12.1.4, rea_riscv_abi_quad(long double value, int sel) read value
from a4:4,a3:4,a2:4,a1:4 and sel from a5, where clang passes a pointer in
a1 and sel in a2.

- ReaGhidraRiscvCallingConvention.java now has one derivation driven by a
  per-policy row (floating-point entry size, preserved floating-point
  registers). Every policy inserts a rule before the input list's final
  join that converts any value of 9 bytes or more to a pointer; a return
  that size already becomes a caller buffer. The self-check requires a
  16-byte value to pass by reference.
- The fixture gains rea_riscv_abi_quad. The call-site oracle now checks the
  words the by-reference pointers point at, for the large structure and the
  long double, and the recorded data comes from an LLVM 19 build.
- docs/ghidra-riscv.md: the decision record and the ilp32 policy row.

ilp32-prototype-v1 keeps its id: no release carries it yet.

* test(ghidra): check RISC-V lane limitations by their facts

The limitation tests froze a full explanatory sentence, the note count and
array positions, so a harmless rewording failed them: the pattern Greptile
flagged on #7486. Assert the facts the notes must carry instead: the pinned
language, the static-linkage boundary, one note naming the declared ABI,
the ABI the pinned language models and the prototype REA applies, and the
declared ISA.

* test(ghidra): check return storage in stripped RISC-V ilp32 builds

The stripped branch of assertRiscvIlp32Storage compared parameter storage
only, so an inferred return that lost the high word of a register pair,
such as rea_riscv_abi_wide at a0:4 instead of a1:4,a0:4, still passed.

Stripped returns must now use the typed build's storage, or none when the
result goes through caller memory. Recovered type names are still not
compared: a test accepts other type names and refuses the dropped high word.

* docs(ghidra): record decision D3 as decided in #7477
2026-10-31 15:48:46 +08:00
Entrpi e6d0618979 feat(artifacts): inventory and extract ESP flash layouts (#8228)
* feat(artifacts): inventory and extract ESP32 firmware images

ESP32-family app and bootloader images (magic 0xE9) had no reader, so
open_binary refused OTA downloads, app partitions and bootloaders.

Add the esp-image container format behind inspect_artifact and
extract_artifact. A pure decoder reads each byte once and keeps only
fixed-size fields. Derived facts cover the chip, from a table cited to
ESP-IDF v6.1; segment classification as ESP-IDF's should_map and
should_load; checksum and appended SHA-256 results; app and bootloader
descriptors; Secure Boot v2 signature blocks, observed and unverified;
and undescribed trailing byte ranges. Truncation and undescribed bytes
make coverage unknown. ESP8266 images are refused with a specific
reason, and ciphertext stays unrecognized.

Add the format to every target, Evidence, snapshot, live-summary and
artifact vocabulary, document it, and add an optional verify:esp-images
lane that compares esptool-built images with esptool image-info and
espsecure signature-info-v2.

Refs #3615

* test(artifacts): run the ESP image lane on esptool v5.6.0

The lane compared descriptor fields only when esptool image-info printed
one, so a descriptor that REA reports and esptool does not print passed
without comparison. Compare the descriptor kind in both directions.

Record that the lane passed for all 13 images with esptool and espsecure
v5.6.0, the version the layout facts cite, as well as with 4.12.0.

* refactor(artifacts): define ESP image layout sizes once

The decoder and the derived facts each defined the segment-header,
appended-digest and flash-sector sizes, the erased-byte scan and the
offset formatter. Define them once with the other fixed-layout structures
and import them in both places. The signature slot count is the one
constant that drives the slot scan.

Correct the observedEnd comment: it is the end of the last chunk read.

* fix(artifacts): report ESP-IDF's segment length limit by chip

ESP-IDF's verify_segment_header rejects a segment of 16 MiB or more
unless the chip maps flash through 32-bit cache addresses
(SOC_SPI_MEM_SUPPORT_CACHE_32BIT_ADDR_MAP: ESP32-S3, P4, C5, C61, H4 and
S31 in v6.1). Record that capability in the chip table and add a
limitation for such segments; unknown chips get none.

`loaded` repeats should_load. It does not mean the bootloader accepts the
segment: verify_load_addresses still rejects one outside the chip's RAM,
which REA does not model. Say so in the docs, the skill reference and the
inventory limitation.

* test(artifacts): check ESP lane tools first and verify extraction

The lane started work before checking its tools, so a missing RISC-V
target, ld.lld, esptool or espsecure surfaced as a raw spawn or command
error. Check each one before building and name the variable to set. Warn
when esptool is not v5.6.0, the release the layout facts cite.

Assert each segment's class for the ESP32-C3 app, its variants and the
bootloader, as the Xtensa image already did. Extract the app and the
signed image through the CLI and compare every file with the bytes at
esptool's segment offsets or in the appended signature sector; the
source's SHA-256 must be unchanged and equal the reported root_sha256.

* docs(artifacts): state what the ESP image lane does not establish

The Verification section listed the lane's checks but not its gaps: only
ESP32-C3 and ESP32 images come from a real producer, no real project
build, OTA download or flash dump is inspected, no signature is verified,
and the lane does not run in CI.

Record the timing of the final lane run on esptool v5.6.0.

* fix(artifacts): compare ESP segment declarations

Segment entries hash only their data, so compare_artifacts reported a
segment that moved to another load address with the same bytes as
unchanged; only the image itself changed. Compare each segment's
load_addr and data_len as its declaration, as Node SEA and PyInstaller
members are compared, and count a changed declaration as a metadata
change, also when content is unavailable.

Header offsets are layout and the classification is derived, so neither
is compared.

* test(artifacts): run ESP lane commands as owned process trees

The lane ran its tools through execFile. A timeout killed only the direct
child, so a linker or assembler that Clang or GCC had started could keep
running while the lane removed its workspace. Run every command through
createVerifierCommands, check the tools inside withVerifierCleanup, and
remove the workspace only after commands.close() has stopped every tree;
a failed close keeps the workspace.

Only a nonzero exit now counts as a refusal or an expected esptool
failure; timeouts and spawn failures propagate. Supervising each command
roughly doubles the lane's run time.

* test(artifacts): cancel ESP decoding after its first decoded chunk

The cancellation test requested cancel or close on setImmediate, which
can land while the file is still being opened and checked, so it never
showed that decoding had started. It also passed against a reader that
ignored both requests while decoding. Acknowledge the decoder's first
chunk, then cancel or close, join the operation, and assert that no
later chunk reaches the decoder and that the reader ends closed.

* docs(artifacts): record which ESP metadata comparison ignores

Segments are compared by their declaration. The image's and the
signature blocks' format_metadata are not: every field comes from their
own bytes, which content comparison already covers, and a block's
image_digest_matches follows the image's content.

* feat(artifacts): inventory and extract ESP flash layouts

A whole-flash dump of an ESP32-family chip was read as its bootloader
image, with the partition table, apps and data left as undescribed bytes,
or refused when it began with erased flash. Read it as an esp-flash-image
container instead.

The partition table is decoded as esp_partition_table_verify accepts it
and is found at the caller's partition_table_offset, at the default
0x8000, or by scanning 0x1000-aligned sectors for a table whose MD5
verifies, which the inventory reports as derived. The bootloader is
located at its chip's ROM bootloader offset. Each partition is an entry
with its declaration in flat format_metadata; the bootloader and app
partitions carry ESP image facts from the existing streaming decoder; the
otadata records are reported with a selection derived without rollback or
anti-rollback, and the running partition stays unknown. One unreadable
partition does not hide the others, and extraction writes every partition,
nvs_keys included, as ordinary data.

Flash layouts are detected before single ESP images. The scan runs only
when an ESP image sits at a ROM bootloader offset, and is refused when a
bootloader at another chip's offset shows that the dump does not start at
flash address 0. open_binary gains flash_offset and partition_table_offset,
which require format esp-flash-image and are part of the snapshot target
identity; the CLI takes them with --target-format esp-flash-image.
Comparison treats a partition's label, type, subtype, range and flags as
its declaration.

* test(artifacts): verify ESP flash layouts with gen_esp32part, merge-bin and otatool

Add the optional verify:esp-flash lane. It builds default and relocated
partition tables with ESP-IDF's gen_esp32part.py, assembles dumps with
esptool merge-bin, including one written from flash 0x1000, and writes
OTA records with otatool through flash_file_tool.py, which replaces
parttool's esptool calls with reads and writes of the dump file.

It compares REA's bootloader and app facts with esptool image-info, every
partition with gen_esp32part.py's decoding of the extracted table, and the
OTA records and derived slot with otatool read_otadata after each switch.
It also checks extracted bytes and source integrity, the scan and caller
table offsets, flash_offset on the 0x1000 dump, a corrupted app beside
valid partitions and a truncated dump. ESP32 linker scripts place the
conformance bootloader and app in ESP32 address ranges.

* fix(artifacts): read ESP flash layouts as the bootloader does

Read the partition table as bootloader_utility_load_partition_table
does: the last factory, otadata and same-subtype OTA declaration wins,
and a limitation names repeats. Plan each otadata record within its
partition and the dump. An otadata partition smaller than two sectors
selects none, the bootloader's INVALID_INDEX; a record the dump lacks
leaves the selection unknown while the other record stays reported. The
fallbacks follow the bootloader's indexes: ota_0-fallback and
factory-fallback name no partition when the table declares none, and the
assumptions add GPIO holds and deep-sleep wake.

Refuse a dump whose bootloader contradicts its origin before the default
table offset is tried, whether flash_offset was omitted or given, and keep
the ESP image reader's ESP8266 refusal. A table never lies inside the
located bootloader image, so single app images are no longer scanned. An
explicit esp-flash-image selection is decided before other container
signatures, and the native commands' --target-format no longer takes it.
Say when no bootloader confirms the default origin.

Exempt the primary bootloader and partition_table entries of ESP-IDF 5.3
from placement limits, report every region inside a longer one, and name
partitions by index when labels repeat ignoring case. Keep an undecodable
label's bytes in label_hex and compare them. Word undescribed ranges as
starting at a byte that is not erased, and treat a short erased region as
erased.

Add an MCP compare_artifacts test for partition declarations, reader
identity and close tests, scan edge cases, the 0x2000 ROM offset and a
literal otatool CRC.

* test(artifacts): verify ESP primary layout entries and the otadata fallback

Build a table with the primary bootloader and partition_table entries
that gen_esp32part.py fills from --primary-bootloader-offset, and check
that REA reads it with complete coverage, no overlap or misplacement
limitation, and a table that agrees with gen_esp32part's decoding. Erase
the otadata with otatool after the slot switches and check the records
against read_otadata and the factory fallback.

* fix(artifacts): keep ESP flash coverage complete for a dump cut short

An esptool merge-bin dump ends after the last part it wrote, usually inside
its last partition. Such a partition made the layout's coverage unknown, so
artifact comparison of two merge-bin dumps reported every unchanged
partition as unknown.

The partition table names every partition whatever the file's length, so a
partition cut short or left out by the end of the file hides no entry. As
the ESP image reader does for a cut-short segment, its entry holds the bytes
the file has and an entry limitation gives the declared size; a partition
with no bytes stays in partitions_outside_dump. Coverage stays complete
unless undescribed bytes remain or no bootloader is located. An MCP
compare_artifacts test of two dumps that end inside ota_0 now classifies
every partition, and fails on the old rule.

* fix(artifacts): report written bytes in ESP partitions, erased slots included

An app slot is erased for ESP-IDF's bootloader when its image header bytes
are erased flash, and REA reported only that status. A slot can still hold
data further in: a dump read from a device held a complete older app image
1 MiB into an OTA slot reported as erased, with no limitation.

Each partition entry now records where the bytes the dump holds are
written, relative to the partition: written_offset, the first byte that is
not 0xFF, written_end, just past the last, and written_bytes, how many are
not 0xFF. They are null for a partition outside the dump. The status stays
erased, which is the bootloader's view, and a limitation names an erased
app slot that holds written bytes and says they are not read as an image.
The reader reads every partition byte once more to find them.

* fix(artifacts): derive the ESP boot selection for every partition table

The boot selection lived inside otadata, so a table without an otadata
partition, the common factory-only layout, had none. ESP-IDF's
bootloader_utility_get_selected_boot_partition returns the factory index
when the table declares no otadata partition.

Move it to a top-level boot_selection that every layout has. Its basis
names the bootloader's branch: otadata, factory-fallback and ota_0-fallback
from the records, no-otadata for a table without otadata, none for the
bootloader's INVALID_INDEX, and unknown when the dump lacks a record the
bootloader reads, which replaces the null selection. running_partition
moves with it, and otadata keeps its partition index and records. The
assumptions name what each branch leaves out: no records assumption
without otadata, and none for none or unknown. The real-tool lane checks
no-otadata on a table gen_esp32part built without otadata.

* fix(artifacts): say when an ESP image is a flash dump read from 0x1000

An ESP32 dump read from flash 0x1000, as read-flash 0x1000 or merge-bin
--target-offset 0x1000 writes it, opened automatically as its bootloader
image with a megabyte of undescribed bytes and no next probe. The flash
reader refuses that origin, but nothing told an agent which flash_offset to
pass.

espFlashDumpOrigin owns the inference: an image of a chip whose ROM loads
the bootloader from flash R, at file offset F, is the bootloader of a dump
read from R - F when a partition table sits where that origin puts 0x8000.
The ESP image reader asks it for a non-app image at file offset 0, so the
result does not depend on how the reader was created, and records
flash_dump {flash_offset, table_file_offset}, null otherwise, with a
limitation that says to reopen the file as esp-flash-image with that
flash_offset. inspect_artifact proposes the same open_binary call as a next
probe, and the flash reader's misplaced-origin refusal names the offset.

* perf(artifacts): summarize nested ESP images in the flash layout summary view

The summary view returned an ESP flash layout's root facts whole, so each
app partition and the bootloader carried every segment and signature
block, about half of a dump's summary.

The summary view now gives each nested image a summary instead of its
facts: status, chip, entry point, segment count, load ranges (each mapped,
loaded or unclassified segment's address and declared length), the
checksum and appended SHA-256 statuses, coverage and limitations. Complete
detail keeps the full facts, every segment included, and the retained
Evidence is unchanged. On a 16 MiB device dump the summary's root falls
from 5,141 to 2,657 bytes, and the summary Evidence from 10,038 to 7,554.

* test(artifacts): expect the misplaced image the ESP bootloader search reports

8a748d3cc added misplacedImage to the bootloader search, which the
locator uses to name the origin a misplaced bootloader implies, but one
test still compared the search result without it. Expect the image's
chip and file offset, and no image for an unknown chip.

* fix(artifacts): give every ESP partition from the dump's start an entry

After ecbc1a772 kept coverage complete for a dump cut short, a partition
wholly past the end of the file still had no entry, only a
partitions_outside_dump row. Comparing a merge-bin dump with a full device
read then reported that partition as added or removed, as if one table
did not declare it.

Every partition from the dump's first byte on is now an entry holding the
bytes the file has: none for one past the end, with a limitation saying
the file holds none of its bytes, no image facts and null written fields.
partitions_outside_dump keeps only partitions that begin before the dump's
first byte, and coverage is unknown when one does, since it has no entry.
Two dumps that lack the same partition now compare it as unchanged, and a
dump against a longer one reports the partition as changed; an MCP test
of a merge-bin dump against a full one checks that no path is added or
removed.

Unlike the ESP image reader, which marks a truncated image's coverage
unknown because later segments and signature blocks may be hidden, the
flash reader can list every partition from the table whatever the file's
length; only the entry limitation follows the image reader.

* fix(artifacts): select no ESP app from a partition table the bootloader rejects

boot_selection followed bootloader_utility_get_selected_boot_partition but
not the table check before it. For a table whose MD5 does not match, or
that has a second MD5 entry, esp_partition_table_verify fails, the
bootloader selects no app and resets. REA still derived a slot from the
otadata records, although the table's own limitation said the bootloader
rejects it.

The selection now has basis none, with no assumptions, for a table the
decoder marks invalid. It also follows two more branches of the
bootloader: an otadata partition at offset 0, which the bootloader takes
for none, selects the factory index as no-otadata, and a table without an
OTA app selects its fallback even when the dump lacks a record, since the
bootloader then ignores the records.

The bootloader also rejects a table whose partitions end past the flash
size in its header. REA does not model that check; the format guide says
so.

* fix(artifacts): require the ESP dump origin's table to follow the image

espFlashDumpOrigin named a later flash origin whenever a partition table
decoded where that origin puts 0x8000. Bytes inside the image's own
segments could be read that way, so an ESP32 bootloader whose segment data
held table-shaped bytes at file offset 0x7000 was reported as a dump read
from 0x1000.

The table must now start at or after the image's end, the rule the locator
already applies to the default table. The ESP image reader passes the
image_end it decoded, or the file size for a truncated image; the flash
reader's refusal passes the end of the misplaced image's segments. The
bootloader search drops its misplaced flag, which only restated
misplacedImage, and the next probe formats offsets with offsetText.

Reader tests cover the 0x1000 and 0x2000 ROM offsets, a standalone
bootloader, bytes that begin like a table, an app with a table at 0x7000
and a table inside a segment; without the boundary check, three of them
fail.

* fix(artifacts): give the ESP flash summary its own kind and each image's identity

The summary view's ESP flash root kept kind esp-flash-image, so a consumer
bound to kind could not tell an image's summary from its full facts, and
the advertised schema held two objects with the same kind const. It is now
esp-flash-image-summary.

The per-image summary also dropped the descriptor, which is what tells two
app slots apart, and named each image only by its partition index, which
the summary view, having no entry rows, could not resolve. Each nested
image now carries its entry path from the layout plan, in both views. The
summary adds the descriptor kind with its identifying fields (an app's
project_name, version, idf_ver, secure_version and app_elf_sha256; a
bootloader's version, idf_ver and secure_version), the signature block
count and each load range's classification. Its enums and fields derive
from the schemas that own them.

An MCP test reads a dump with an app in ota_0 and an erased ota_1 that
holds older bytes: it checks both summaries, that the summary's inventory
Evidence is the one the complete view returns with full facts, and the
erased slot's written range.

* test(artifacts): cover cancel and close during the ESP written-bytes scan

The written-bytes pass reads every byte of each partition the dump holds,
checking that the reader is open before each chunk, but only app decoding
had a cancellation and close test. A data partition, which only that pass
reads, now holds 9 MiB; once the scan has examined its first chunk, a
cancel rejects preparation with the caller's reason and a close with
unavailable, and the scan examines no further chunk.

The erased-slot test now asserts the whole set of erased-slot limitations,
so factory, ota_0 and nvs, which the bootloader reads or which are not app
slots, are shown to have none.

* docs(artifacts): re-time the ESP flash lane and state only observed causes

The lane ran again after its assertions changed; both esptool runs passed,
and the timed run's figures replace the earlier ones. The erased-slot
paragraph no longer names a cause for the older bytes, which REA cannot
observe, and the skill reference states that every table has a
boot_selection in its own sentence, with the none basis for a table the
bootloader rejects.

* fix(artifacts): retire ESP flash members as the ESP image reader does

#7498's maintainer commits made the ESP image reader keep a member owned
until its consumer closes it, retire fully buffered bytes when
cancellation arrives after production, and refuse to finish an inventory
paused across close. The ESP flash reader was copied from the earlier
reader. Main's shared stored-member lifecycle cases failed two of eight
against it: a paused inventory completed after close, and buffered member
bytes stayed readable after cancellation.

Give the flash reader the same lifecycle and run the shared cases against
it.

* docs(artifacts): re-time the ESP flash lane after merging main

The lane passed again with esptool v5.6.0 and 4.12.0 on the merge of upstream main. Record the v5.6.0 run's time and peak resident set.

* fix(artifacts): read ESP image Evidence recorded before flash_dump

The flash layout work made `flash_dump` a required field of the
`esp-image` root metadata. Evidence recorded before it has no such
field, so newer REA rejected records that were valid when they were
made: `compare_artifacts` and `inspect_analysis_view` reported
`evidence_integrity_mismatch` for them.

The field is now optional when read. A record without it parses as it
is, so the graph digest it commits to still matches; nothing fills the
field in. The image reader still writes it for every image, and the
inspection probe reads a missing field as no hint.

A regression test recommits an image inventory without the field and
passes it inline to `compare_artifacts` and `inspect_analysis_view`.
Against a server built with the required field, `compare_artifacts`
rejects it.

* fix(artifacts): stop the ESP flash gap scan when the reader closes

The scan for the first written byte in each undescribed gap checked
only the caller's signal. `close()` marks the reader closed and waits
for preparation, so it waited while the rest of a long erased gap was
read.

The gap scan now takes the reader's open check, as the written-bytes
scan and the image decoder do, and checks it before each chunk. Tests
cancel or close the reader after the scan's first chunk of a 9 MiB
erased gap and confirm that no later chunk is examined.

* fix(artifacts): name ESP partitions with Windows device labels by index

The safe-label rule accepted labels such as `NUL`, `con` or `COM1`, so
extraction wrote `partitions/NUL.bin`. Windows reserves those names
before any extension, and ordinary Windows tools cannot use such a
file.

A label that is a Windows device name (`CON`, `PRN`, `AUX`, `NUL`,
`COM0`-`COM9` and `LPT0`-`LPT9`, any case, alone or before a dot) now
takes the existing index fallback. The partition metadata and the
entry's source path keep the label.

* fix(artifacts): read ESP flash partitions' written bytes in one pass

Each partition got its own scan of the bytes the dump holds, so
partitions declared over the same bytes read them again and again.
Overlaps only add limitations, so a crafted table reached this normally:
94 overlapping partitions over a 32 MiB dump would examine about 3 GiB
before the first entry.

The reader now reads the bytes the partitions cover once. Partition
boundaries cut them into segments; one pass counts each segment's
written bytes and bounds, and each partition's extent is the sum of the
segments inside it. Every partition keeps its own declaration, extent
and limitations. The pass checks the reader's open state before each
chunk, as the per-partition scan did.

A test declares 60 partitions over one MiB, 40 over all of it and 20
over its first half, and checks each partition's extent and the
overlap limitations. The reader reads 1,113,256 bytes of the
1,114,112-byte dump; one scan per partition read 52,493,480.

* test(artifacts): fail the ESP lanes on failed commands and refusals

`otatool` turned a nonzero exit into returned text, which its callers
discarded, and `inspect()` returned `{ refused }`, which the flash lane
never checked. A failed command then surfaced later as a slot
assertion, an output-parser error or a TypeError, hiding its cause.

Commands the lanes need to succeed now reject with the output they
exited with: `otatool` and `inspect()` do so through a shared
`failedCommand`. An expected refusal is opt-in: `refusal()` returns
REA's output and fails the lane when REA opens the file. The ESP image
lane's two ESP8266 cases use it.

Checked by calling `otatool` on a file without a table, `inspect()` on a
file REA refuses and `refusal()` on a file REA opens; each rejects with
its cause.

* fix(artifacts): require a partition entry in an ESP partition table

The decoder counted the slot, not the partitions, so a lone MD5 entry
over the empty prefix, followed by the terminator, decoded as a table
with no partitions. Default and caller-selected detection then accepted
that empty layout, although the guide requires one partition.

A table now needs one partition entry before its terminator. An MD5
entry alone is refused as `entry 1 at 0x20 ends the table before any
partition entry`. The cases cover the decoder and both automatic and
caller-selected location.

* fix(artifacts): decode an image shared by ESP app partitions once

Each app partition got its own decode, so app partitions declared over
the same bytes decoded and hashed the same image again and again, the
same pattern the written-bytes scan had. With 60 app partitions over
one 960 KiB image the reader read 64,029,736 bytes of a 1,114,112-byte
dump.

Regions with the same file range now share one decode; each keeps its
own copy of the facts. Partitions with different ranges still decode
separately, because truncation can change the facts: at most one decode
per partition the table holds, each within its partition.

A test declares 60 app partitions over one image, 40 holding all of it
and 20 cutting it short, and expects three decodes: the bootloader, the
whole image and the cut one. The 60-partition dump now reads
2,161,864 bytes, as a dump with one app partition does.
2026-10-31 15:24:32 +08:00
Entrpi 091892dfed test(ghidra): expect carved MIPS copies to be refused at target opening (#8227)
Since 651afa445 target opening refuses every ELF declaration beyond EOF,
including carved copies, and ac6e0e258 moved the carved MIPS boundary tests
to that contract. The real-Ghidra MIPS lane still expected a carved image to
resolve with section-table limitations and be refused later by Ghidra
admission, so verify:ghidra:mips failed at its first carved fixture.

The lane now requires open_binary and the CLI to refuse the carved copy as
target_unavailable (file_format_rejected) with the exact declared
section-table range and captured size. The carve and that expectation live in
the shared MIPS checks, and a boundary regression applies them to the
production resolver and error projection, so a later contract change fails
without real Ghidra. The MIPS and PSP guides describe the current contract.
2026-10-31 15:24:17 +08:00
Machen John 1dc97d8cec fix(ghidra): report a missing existing-project program as a configuration error (#8222)
* fix(ghidra): report a missing existing-project program as a configuration error

When REA_GHIDRA_PROJECT_PATH selects a project that has no root program
with the selected name, Ghidra's headless analyzer aborts with
"Requested project program file(s) not found: <name>". REA reported this
as a generic execution_failure with rea doctor advice, and the reason was
visible only in the captured diagnostics.

Recognize that refusal in the operation-scoped launcher output, matching
only the exact name REA passed to -process, and return a
ConfigurationError for REA_GHIDRA_PROGRAM_NAME. The message says whether
the name was configured or defaulted from the target file name, and the
remediation explains how to select the exact root program. The selected
project is unchanged and there is no fallback to a fresh import. Other
launch failures keep their existing projection.

Fixes #8216

* test(ghidra): assert missing-program guidance through its formatter

Compare the projected setting constraint and remediation with
ghidraMissingProgramGuidance instead of freezing explanatory phrases,
following docs/testing.md. The program name and configured-or-defaulted
origin are still checked independently.
2026-10-31 15:24:00 +08:00
Jonnalagadda Sai Amarthya 7ba3f558d4 fix(windows): keep polling owned process output after a failed poll (#8163)
* fix(windows): keep polling owned process output after a failed poll

* test(windows): pin the failed-poll retry delay
2026-10-31 15:23:13 +08:00
morluto 245a581818 feat(kuna): verify release 1.986 and guide version selection (#8187)
* feat(kuna): verify release 1.986 and guide version selection

* test(kuna): cover verified release doctor guidance
2026-10-31 14:58:56 +08:00
morluto ec107022fa docs: guide audits of existing investigation tools (#8272) 2026-10-31 14:38:13 +08:00
gensokyo-dev b349e3c708 fix(evidence): give missing retained closures actionable recovery (#8270)
* fix(evidence): give missing retained closures actionable recovery

* refactor(evidence): assign retained lookup context at resolution
2026-10-31 14:30:30 +08:00
gensokyo-dev d9e01e5af2 feat(evidence): aggregate exact retained comparison references (#8267) 2026-10-31 13:59:13 +08:00
gensokyo-dev 6e2404b869 fix(comparison)!: preserve same-entry correspondence across renames (#8263)
* fix(comparison): preserve same-entry correspondence across renames

* docs(comparison): align rename guidance and focused contract checks
2026-10-31 12:53:42 +08:00
morluto 356cc50475 docs: guide design and verification of provider workflows (#8261)
* docs: guide provider workflow design and annotation comparisons

* docs: connect provider tool design to workflow verification
2026-10-31 12:15:15 +08:00
ShuaixinHuang d48e472fa6 fix(mcp): preserve repeated HTTP host allowlists (#8164) 2026-10-31 03:13:07 +08:00
Abdelrahman Teimaandgensokyo-dev 47fa6ab379 feat(electron): report custom URL protocol registrations
Expose static Electron protocol call facts through the shared application graph and existing CLI/MCP Evidence views. Preserve canonical case-insensitive scheme identity, original spellings, exact source provenance and explicit unknowns; keep registration success and runtime routing unproven.

Integration fixes also enforce module-only privileged API ownership, reuse the shared finding-key owner, and align the owning type documentation with the Evidence semantics. Validation and bounded real Obsidian CLI/MCP measurements are recorded in the PR.

Co-authored-by: UnknownHacker1 <[email protected]>
Co-authored-by: gensokyo-dev <[email protected]>
2026-10-31 02:23:47 +08:00
Harsh Bhardwaj 69bf61f7c7 fix(browser): preserve event coverage for failed requests (#8158) 2026-10-31 01:36:41 +08:00
Machen John ca7d1c3750 fix(ghidra): match startup-timeout guidance to the project mode (#8114)
In fresh-import mode, the startup-timeout remediation now points to
REA_GHIDRA_PROJECT_PATH for reusing a completed analysis, next to the
existing REA_GHIDRA_STARTUP_TIMEOUT_MS and REA_GHIDRA_DISABLED_ANALYZERS
advice.

In existing-project mode, the message no longer says Ghidra was
importing and analyzing the target, and the remediation no longer
suggests REA_GHIDRA_DISABLED_ANALYZERS, which configuration validation
rejects together with REA_GHIDRA_PROJECT_PATH.

Refs #3751
2026-10-31 01:32:06 +08:00
吴佳浩Alben e5c46dff03 fix(javascript): emit no socket for a literal out-of-range port (#8137)
* fix(javascript): emit no socket for a literal out-of-range port

`net` and `tls` validate a numeric port before creating a socket, so a literal
port Node rejects throws synchronously and no socket exists — but the analysis
reported an `acquire` operation.

Measured on Node 24.22.1:

    net.connect(0) / 80 / 65535        -> Socket created
    net.connect(65536)                 -> THREW ERR_SOCKET_BAD_PORT
    net.connect(-1) / 1.5              -> THREW ERR_SOCKET_BAD_PORT
    net.connect(NaN) / Infinity        -> THREW ERR_SOCKET_BAD_PORT
    net.connect(1e400)                 -> THREW ERR_SOCKET_BAD_PORT
    net.connect(0n) / true / false     -> THREW ERR_INVALID_ARG_TYPE
    net.connect(null)                  -> THREW ERR_INVALID_ARG_TYPE
    net.connect("")                    -> THREW ERR_SOCKET_BAD_PORT
    net.connect("/tmp/x.sock")         -> Socket created (Unix socket path)
    net.connect({ port: 80 })          -> Socket created
    net.connect(undefined) / {} / []   -> THREW ERR_MISSING_ARGS (undecided)
    --- tls mirrors net ---
    tls.connect(80)                    -> Socket created
    tls.connect(65536) / -1 / NaN      -> THREW ERR_SOCKET_BAD_PORT

A string is a Unix socket path and an object may carry a path, so only an
exactly readable non-port literal is decided. `undefined` and an empty object
throw `ERR_MISSING_ARGS`, which is a missing option rather than a bad port, so
they stay undecided. `NaN` and `Infinity` are identifiers rather than literals
and are decided only through `isUnmodifiedGlobalReference`.

* chore: retrigger title and readiness checks after the title was corrected

* fix(javascript): place the port check away from the fs flag check

`#8125` adds an fs flag guard at the top of `collectAcquisitions` and this branch
added a socket port guard in the same spot, so the two conflicted on every
rebase. The port check now filters the completed candidate list from
`collectJavaScriptSemanticResources` and its helpers sit at the end of the file,
which cannot collide with the fs flag guard.

* fix(javascript): drop the redundant in-loop port guard

The caller now filters the completed candidate list, so the loop guard became
dead code. Removing it also keeps this branch from editing the same lines as the
fs flag guard in #8125.
2026-10-31 01:23:50 +08:00
1059 changed files with 61550 additions and 12368 deletions

No files matched your search

+15 -1
View File
@@ -1,6 +1,6 @@
---
name: rea-tool-design
description: Design or change REA investigation tools, CLI/MCP contracts, provider capabilities, and Evidence semantics.
description: Audit, investigate, design, or change REA investigation tools, CLI/MCP contracts, provider capabilities, and Evidence semantics.
---
# REA tool design
@@ -10,6 +10,20 @@ contract semantics, provider ownership, and discoverability. Inspect the current
contract and nearest existing tool relevant to the analyst question; source,
documentation, and live catalogs may describe different versions.
For provider-backed work, exercise the analyst's task through REA and compare
relevant native provider and independent artifact observations where available.
Check applicable partial results, recovery and session boundaries; report what
was not exercised. For failure guidance, follow [design recovery around the
failure](../../../docs/tool-design.md#design-recovery-around-the-failure).
Use [workflow verification](../../../docs/testing.md#provider-backed-workflow-verification)
to select evidence for the proposed contract instead of seeking provider API
parity.
For an audit, follow [audit existing tools](../../../docs/tool-design.md#audit-existing-tools):
scope representative tasks, exercise the live workflow, and distinguish
reproduced bugs, supported limitations, design opportunities and unverified
claims. Report working alternatives and evidence limits as well as findings.
For a design request, return the proposed boundary, input/result shape, Evidence
semantics, nearest alternative, and verification approach. Implement when
requested, preserving the user's scope and existing authorization.
@@ -2,7 +2,7 @@
name: reverse-engineer-anything
description: Reverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis.
metadata:
version: "55"
version: "59"
---
# REA
@@ -401,9 +401,18 @@ call `analyze_javascript_application` or `inspect_binary_layout` with
a flag: its default reply is the summary, and the complete dossier stays
retained under `parent_evidence_id` for facet paging. Pass that ID as a
`retained-evidence` reference to `compare_functions`, or inside
`match_native_functions` `left_observations`/`right_observations`; request
`"detail": "complete"` only when the dossier must travel inline, as
`build_call_path` requires.
`match_native_functions` `left_observations`/`right_observations` or
`build_call_path.functions`; request `"detail": "complete"` only when the
dossier must travel inline.
Use the same exact-reference form in other advertised logical Evidence inputs,
including optional sources, arrays and nested sources across domains. Each
source can independently be inline or retained; preserve its source identity,
coverage, partial facts and unknowns. A summary or inspection page has its own
Evidence ID: use `normalized_result.parent_evidence_id` when the consumer needs
the complete source. For portable CLI follow-ups, commands advertising
`--evidence-bundle` resolve references from that selected exported bundle.
See [Evidence composition](references/evidence-workflows.md) for scope and
recovery; file selectors are supported only where explicitly advertised.
Page module identities and select the returned `node_id` before requesting an
item; a single file path can represent several graph roles. Do not repeat an
identical analysis or rejected call without a changed input or resolved blocker.
@@ -505,4 +514,8 @@ as complete while required questions remain open.
Explain findings in plain language and tie them to returned evidence. When the
user asks to build something, use normal coding tools and separate observed
behavior from design choices. Close an opened native session with
`close_binary` when the investigation is complete.
`close_binary` when its engine is no longer needed. Evidence references and Unknown
history remain in memory on the same MCP connection. End the investigation with
`reset_investigation`; supply `save_path` to checkpoint the complete bundle before
clearing. Import the saved bundle to restore original references and Unknown
history in a new process. Retention is bounded and is not automatic persistence.
@@ -11,6 +11,44 @@ returned inline. When complete `inspect_binary_layout` or
`inspect_analysis_view` for a summary, one section/symbol/module, or a stable
page instead of repeating the producer.
For any logical Evidence source that advertises retained references, pass
`{kind: "retained-evidence", evidence_id}` instead of copying the complete
record back into a call. The same contract covers individual, optional, array
and nested sources across native, process, JavaScript, managed, mobile and WASM
workflows. Sources can independently mix inline records and exact references.
Resolution preserves the source's identity, provenance, coverage, partial facts
and unknowns; it neither reruns analysis nor makes incomplete facts complete.
Keep domain role and identity requirements: a native dossier cannot substitute
for a managed member observation, for example.
When starting from a summary or inspection page, use its
`normalized_result.parent_evidence_id` to reference the complete source.
The view's own `evidence_id` names
only that view. `compare_artifacts` explicitly supports artifact summary inputs
and resolves their parents; do not assume every consumer does. When a workflow
follows source links, keep those linked records available too.
MCP references belong to the current connection's ledger. If the ID is known,
pass it directly; no discovery call or provider setup is needed. If it is missing,
import the exported bundle or supply complete inline Evidence. Do not retry an
unchanged reference or rerun a producer merely to move an existing record.
Export before closing the session when the source must survive it. For CLI JSON
workflows that advertise `--evidence-bundle`, the same reference JSON resolves
against one explicitly selected authenticated bundle:
```bash
rea trace-application-feature request.json --evidence-bundle /absolute/path/evidence.json --json
```
This includes JavaScript traces/comparisons and runtime reconciliation, managed
reconstruction/native verification/graph projection, Android/Apple/Harmony graph
projection, WASM interface comparison and retained analysis views. The managed
member comparison CLI still takes two assembly paths; process comparison takes
two capture Evidence JSON paths. Check command help for the installed version.
An input bundle grants no implicit file lookup: `evidence-file` and historical
graph file selectors are accepted only by contracts advertising them. See the
[CLI reuse guide](https://github.com/morluto/rea/blob/main/docs/cli.md#import-export-and-compare-evidence).
Completed `inspect_artifact` inventory also supports a summary and
`view: {kind: "inventory", collection: "occurrences", logical_paths: ["package.json", "main.js"]}`.
Omit `logical_paths` to page all available occurrences; `[]` selects none.
@@ -56,11 +94,12 @@ or absence:
the complete dossier.
- `compare_bundles` compares canonical bundle membership and unknown history.
- `find_changed_behavior` combines existing comparisons; static differences
remain candidates, not causal proof.
remain candidates, not causal proof. Its `comparisons` accept exact references.
- `build_call_path` needs explicit function dossiers from one artifact/provider;
an incomplete frontier makes absence unknown.
pass complete parent IDs in `functions`. An incomplete frontier makes absence
unknown.
- `correlate_static_and_runtime` uses explicit mappings; matching patterns are
hypotheses, not causality.
hypotheses, not causality. Both comparison arrays accept exact references.
- `verify_reconstruction` evaluates a finite typed specification. A pass covers
only declared comparable claims, not global equivalence.
@@ -236,10 +236,19 @@ This notation is
a template: replace it with the actual returned ID. Versions before 4.1.0 accept only full
inline Evidence. Use the exact ID
returned by the producer on the same MCP connection. Resolution does not run
analysis or select a provider; findings remain inline. Native observation
arrays still take full Evidence. `close_binary` clears retained references;
export a bundle before closing, or supply portable inline Evidence in another
connection. A missing reference includes its ID and recovery guidance.
analysis or select a provider; findings remain inline. For summary or inspection
replies, select the complete `normalized_result.parent_evidence_id`. Native
observation arrays and runtime reconciliation's `static_layers[].analysis` and
`runtime_observations[]` also accept exact references where advertised. Each
source can independently be inline or retained, preserving identity, coverage
and unknowns. CLI workflows advertising `--evidence-bundle` resolve these inputs
from the explicitly selected bundle. `close_binary` releases the engine while
retaining references and complete Unknown history on the same MCP connection.
`reset_investigation` explicitly clears them; optional `save_path` checkpoints the
complete bundle before clearing. Import that bundle in another connection to
restore its original references. Retention is bounded, in-memory and never silently
evicted; `binary_session.investigation` reports the budget. A missing reference
includes its ID and recovery guidance.
For exact file/native metadata or processing gaps, use
`view: {kind: "javascript-facts", where: {paths: ["package.json"]}, property_keys: ["name", "operation", "analysis_status"], limit: 32}`.
@@ -1,8 +1,11 @@
# Native, managed, and packaged artifacts
For release comparisons, export `list_procedures` Evidence from each target before
closing it. Close clears session records: import both bundles after collecting the
two targets, or supply portable inline Evidence. `match_native_functions` pairs unique exact names and, when supplied,
For release comparisons on one MCP connection, retain `list_procedures` Evidence
from each target and reuse its exact IDs after target switches or `close_binary`.
For a handoff or new process, export the complete bundle and import it there,
including its source closure. `reset_investigation` explicitly clears retained
records; optional `save_path` checkpoints them before clearing.
`match_native_functions` pairs unique exact names and, when supplied,
complete observed body bytes. Reuse optional same-context `list_names`,
`procedure_info`, `read_bytes` and existing `analyze_function` dossiers as
`left_observations`/`right_observations`; an MCP `analyze_function` summary
@@ -13,6 +16,13 @@ Select columns, batch IDs and page individual facts; aliases remain unresolved.
Unpaired observations do not prove additions/removals. Existing matched dossier
pairs include inputs for `compare_functions`; otherwise analyze just chosen pairs.
Aggregate completed comparisons with `find_changed_behavior`, selecting each
comparison's exact `retained-evidence` ID instead of resending full records.
Inline comparison Evidence remains accepted. References belong to the current
server: after handoff, import the original bundle including its source closure.
Do not substitute dossier IDs or summary views. Static differences remain
behavior candidates, with runtime behavior unknown unless separately observed.
For a matching representation refusal, inspect `details.limit` and raise
`native.matching_representation_bytes` with `update_analysis_settings` before
retrying the unchanged MCP request. CLI launch seed:
@@ -578,6 +588,16 @@ REA. Symlinks and encrypted entries are inventory facts, not extractable files.
The result reports the materialized directory; callers do not choose the
destination. No separate permission grant is needed.
For classic iNES cartridges, `open_binary` recognizes the byte signature without
an executable architecture. `inspect_artifact` retains header facts and each
PRG/CHR file unit's offset, length and SHA-256; its nested inventory's
`ines.reconstruction` verifies the actual ordered region bytes against the
whole-file digest. Extraction includes the original header, optional trainer
and uninterpreted trailing bytes. These file units are not mapper/CPU bank
maps. NES 2.0, UNIF, dirty legacy headers and truncated payloads are explicitly
refused. No emulator or 6502 analysis is performed. See the
[iNES guide](https://github.com/morluto/rea/blob/main/docs/ines-containers.md).
Native DMG traversal automatically uses a read-only mount on supported macOS
hosts, with an owned temporary mount directory and cleanup. It needs no approval
flag. Unsupported hosts or mount failures remain explicit limitations; do not
@@ -606,10 +626,27 @@ semantics are unsupported. `m68k` requires an even entry and decodes the 68040
superset of the 68000 family; Amiga/Atari containers and vector-table entry
discovery are unsupported. Windows Ghidra P0 admits raw images only for `6502`. See [raw binary guide](https://github.com/morluto/rea/blob/main/docs/raw-binary.md).
Intel HEX, S-record, UF2 and DfuSe files are containers, not raw images:
`inspect_artifact` lists address regions (UF2 per familyID, DfuSe per element
with its alternate setting), and `extract_artifact` writes each region. Open an
extracted region as `raw-binary` with its start address as `entry_point`. UF2
numbering gaps leave coverage unknown; a DfuSe suffix CRC mismatch follows
`integrity_policy`. See [UF2 and DfuSe](https://github.com/morluto/rea/blob/main/docs/uf2-dfuse.md).
### Function annotations in Ghidra
Fresh-import sessions admit annotations; existing-project reuse is read-only.
Use `annotate_native_function` for one function name and/or entry comments, with at least one explicit change. Review its annotation readback and refreshed dossier inline. Changes are atomic and session-scoped; empty comments clear them, omitted fields preserve them. Later MCP calls observe edits until close. CLI `annotate-native-function` returns the updated analysis before discarding the session. Original executable bytes are unchanged; edits invalidate immutable snapshots. Windows P0 does not admit database mutations. For naming only, `set_address_name` / `set_addresses_names` also work with Ghidra: they name function entries or data addresses (primary labels) and return compact per-address success booleans instead of a dossier.
Use `annotate_native_function` for one function name and/or entry comments, with at least one explicit change. Review its annotation readback and refreshed dossier inline. Changes are atomic and session-scoped; empty comments clear them, omitted fields preserve them. Later MCP calls observe edits until close. CLI `annotate-native-function` returns the updated analysis before discarding the session. Original executable bytes are unchanged; edits invalidate immutable snapshots. Windows P0 does not admit database mutations. For naming only, `set_address_name` / `set_addresses_names` also work with Ghidra: they name function entries or data addresses (primary labels) and return compact per-address success booleans instead of a dossier. Ghidra rejects batch addresses that request the same name or a name another selected address holds; `raw_result.failures` gives each failure's reason.
For `compare_functions`, obtain `analyze_function` Evidence before and after the
edit while the MCP session is open; annotation Evidence is not a comparison
input. Export the needed Evidence before close if comparison will happen later.
Authenticated same-artifact coordinates preserve renamed entries as
`matched / same-entry`, with the canonical `entry_address` inline; name and
exact-text changes remain separate dimensions. This requires compatible
architecture, provider-owned mapping and selected-image commitments. Equal
numeric addresses alone cannot establish correspondence; inspect retained
source Evidence and coverage when the match remains ambiguous.
## Offline PE resource inventory
@@ -637,7 +674,9 @@ For exact core-module import/export differences, use `compare_wasm_interfaces`
with two complete inspection Evidence records (or current-session retained
references) and explicit scope. Preserve unknown types/duplicate names; static
interface equality is not runtime compatibility. CLI accepts complete inline
Evidence in `rea compare-wasm-interfaces input.json --json`.
Evidence in `rea compare-wasm-interfaces input.json --json`, or exact references
with `--evidence-bundle /absolute/path/evidence.json`. Use the complete parent
ID when the inspection returned a summary.
## Godot PCK
@@ -696,6 +735,32 @@ recognized. Segments are not disassembled, and RISC-V or Xtensa segments
cannot be opened as raw-binary. See
[ESP images](https://github.com/morluto/rea/blob/main/docs/esp-image.md).
## ESP flash images
A whole-flash dump opens as `esp-flash-image`. `root_format_metadata` holds
`partition_table` (its `method`: `default`, `caller` or a derived
`scan`), `bootloader` and `apps` ESP image facts, the `otadata` records,
`partitions_outside_dump` and `boot_selection`. Every table has a
`boot_selection`: its `basis` names the bootloader's branch (`no-otadata`
without otadata, `none` when the bootloader rejects the table, `unknown`
when a record is missing), and it lists its `assumptions`;
`running_partition` is always `unknown`. Each image names its entry
`path`. With `detail: "summary"` the root kind is `esp-flash-image-summary`
and each image is a compact `summary` (chip, entry point, segment count,
classified load ranges, signature block count, descriptor identity,
integrity); complete detail keeps every segment. Partition rows
(`partitions/<label>.bin`) carry the declared `type`, `subtype`, names,
`flash_address`, `size` and flags; `encrypted` is a storage flag, not
proof of ciphertext. `written_offset`/`written_end` bound the bytes that
are not erased: an app slot whose facts say `erased` can still hold an older
image, and a limitation says so. For a dump read from another flash address, call
`open_binary` with `format: "esp-flash-image"` and `flash_offset`; an
`esp-image` whose `flash_dump` is set names that offset in a next probe; a
relocated table can be named with `partition_table_offset`. No table found
means unknown, often flash encryption. Extraction writes `nvs_keys` as
ordinary data. See
[ESP flash images](https://github.com/morluto/rea/blob/main/docs/esp-flash-image.md).
## Bun standalone executables
A `bun build --compile` executable stores its module graph in Mach-O
+21
View File
@@ -0,0 +1,21 @@
name: REA production code
# Match the existing default-setup scan's local-input threat model.
threat-models: [local]
# Retain root traversal, including future production roots. Hidden directories
# need their own root; use a directory so Python can share this configuration.
paths:
- .
- docs/.vitepress
# Keep the default query suite and all production roots, including bridge/,
# install.sh, runtime scripts, release controllers, and GitHub Actions.
# Exclude test inputs at extraction time so they cannot taint production callers.
paths-ignore:
- tests
- scripts/fixtures
- "**/*.test.*"
- "**/*.fixture.*"
- "**/test_*.py"
- "**/*_test.py"
+7
View File
@@ -779,6 +779,12 @@ jobs:
npm run verify:legacy-simulator
npm run verify:apple-dispatch
npm run verify:interface-builder
- name: Verify portable iNES CLI, MCP and source boundaries
run: >-
npm run test:focused --
tests/acceptance/inesContainer.test.ts
tests/boundary/filesystem/inesArtifact.test.ts
tests/boundary/filesystem/inesCancellation.test.ts
- name: Run native Apple and process boundary regressions
run: >-
npm run test:focused --
@@ -799,6 +805,7 @@ jobs:
tests/boundary/cli/nativeDemangleCli.test.ts
tests/boundary/cli/nativeDispatchPropertyAttributes.test.ts
tests/boundary/mcp/nativeDemangleMcp.test.ts
tests/boundary/mcp/nativeMachoSummaryMcp.test.ts
tests/boundary/providers/native/nativeSignatureTargetBinding.test.ts
tests/boundary/cli/nativeSignatureCli.test.ts
tests/boundary/cli/nativeMachoArm64_32Cli.test.ts
+61
View File
@@ -0,0 +1,61 @@
name: CodeQL production
on:
push:
branches: [main]
pull_request:
branches: [main]
schedule:
- cron: "37 5 * * 1"
workflow_dispatch:
permissions:
contents: read
concurrency:
group: codeql-production-${{ github.event.pull_request.number || github.ref }}
cancel-in-progress: true
jobs:
analyze:
# A repository administrator must disable default setup before enabling
# this workflow; default setup rejects advanced CodeQL result uploads.
if: vars.REA_CODEQL_ADVANCED == 'true'
name: Analyze production (${{ matrix.language }})
runs-on: ubuntu-latest
timeout-minutes: 45
permissions:
contents: read
security-events: write
strategy:
fail-fast: false
max-parallel: 1
matrix:
language: [javascript-typescript, python, actions]
steps:
- uses: actions/checkout@d23441a48e516b6c34aea4fa41551a30e30af803 # v6.1.0
with:
persist-credentials: false
- name: Initialize production analysis
uses: github/codeql-action/init@9b3254085494b55a0f7c4c2b3d2d52cc712ce370 # v4
with:
languages: ${{ matrix.language }}
build-mode: none
config-file: .github/codeql/production.yml
threads: 2
ram: 4096
- name: Analyze production code
uses: github/codeql-action/analyze@9b3254085494b55a0f7c4c2b3d2d52cc712ce370 # v4
with:
category: /language:${{ matrix.language }}
output: ${{ runner.temp }}/codeql-production-results
threads: 2
ram: 4096
- name: Retain complete analysis evidence
if: always()
uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2
with:
name: codeql-production-${{ matrix.language }}
path: ${{ runner.temp }}/codeql-production-results
if-no-files-found: error
retention-days: 7
+5
View File
@@ -53,3 +53,8 @@ jobs:
env:
REA_BROWSER_EXECUTABLE: /usr/bin/google-chrome
REA_BROWSER_NO_SANDBOX: "true"
# The preceding verifier builds the runtime used by this request journey.
- run: dbus-run-session -- node scripts/verify-browser-request.mjs
env:
REA_BROWSER_EXECUTABLE: /usr/bin/google-chrome
REA_BROWSER_NO_SANDBOX: "true"
+1 -1
View File
@@ -54,7 +54,7 @@ jobs:
mkdir -p .cache
rm -f "$report"
node scripts/run-exact-tests.mjs run --reporter=default --reporter=json --outputFile.json="$report" \
tests/boundary/captures/fieldDiscovery.test.ts src/process/LinuxResourceLimits.test.ts tests/boundary/captures/captureProvider.test.ts tests/boundary/captures/captureDescriptorCleanup.test.ts tests/boundary/pcap/pcapProvider.test.ts tests/boundary/pcap/pcapProviderCrossPlatform.test.ts tests/boundary/captures/resourceLimits.test.ts tests/boundary/captures/packetOutput.test.ts tests/boundary/mcp/packetCaptureMcp.test.ts tests/boundary/cli/captureCleanup.test.ts
tests/boundary/captures/fieldDiscovery.test.ts src/process/LinuxResourceLimits.test.ts tests/boundary/captures/captureProvider.test.ts tests/boundary/captures/captureDescriptorCleanup.test.ts tests/boundary/pcap/pcapProvider.test.ts tests/boundary/pcap/pcapProviderCrossPlatform.test.ts tests/boundary/captures/resourceLimits.test.ts tests/boundary/captures/packetOutput.test.ts tests/boundary/mcp/packetCaptureMcp.test.ts tests/boundary/cli/captureCleanup.test.ts tests/acceptance/pcapVerifier.test.ts
node scripts/ci/require-provider-tests.mjs packet "$report"
- run: node scripts/verify-packet-capture.mjs
env:
+13
View File
@@ -28,6 +28,10 @@ Assess brittleness in touched code during implementation, debugging, review, and
When fixing a bug, inspect the producing boundary and affected consumers for duplicated interpretation or unclear ownership. Complete any directly related refactor in the same change and remove superseded code. Passing regression tests alone does not establish that the brittleness is resolved. If related cleanup remains, explain what remains and why.
After confirming a bug, search sibling adapters, consumers and similar workflows
for the same failure. Verify suspected matches, record working comparisons, and
group findings by root cause instead of filing duplicate reports.
When addressing a review thread, inspect the producing boundary and affected consumers for brittleness. Complete directly related refactoring in the same change. Before resolving the thread, explain the root cause, structural fix, validation, and any remaining cleanup.
Parse unknown boundary values and model expected failures with the tagged error algebra and `Result`. Preserve partial facts and meaningful failure reasons. Keep identity and lookup values separate from display formatting; preserve source values and explicit unknowns when normalization loses information. Absolute paths, file URLs, and HTTP paths have different semantics.
@@ -75,6 +79,9 @@ Use [.nvmrc](.nvmrc) and `package.json#packageManager` for the pinned toolchain.
Use relevant checks for the change. The complete local gate, `npm run check:pr`, is optional for broad changes or CI diagnosis. See [CONTRIBUTING.md](CONTRIBUTING.md) for development, generated-file ownership, and release conventions.
When changing installers or setup recovery, follow
[Checking installers and publisher downloads](docs/testing.md#checking-installers-and-publisher-downloads).
Coordinate expensive checks across agents. If a test unexpectedly consumes heavy CPU, stop its owned process family and profile a bounded representative case before retrying. Inspect production work, fixture generation, subprocesses, and repeated setup; preserve regression coverage when optimizing. Follow the [test resource guidance](docs/testing.md#developer-commands).
`docs/public/product-catalog.json`, `docs/verification/managed-conformance-*.json`, and `skills/` are ignored outputs. Edit source contracts and `.agents/skills/reverse-engineer-anything/`, then regenerate as needed. `.cache/mcp-tool-catalog.json` is test metadata; runtime builds and source checking do not consume it. Never commit binaries, provider project documents, credentials, `dist/`, `node_modules/`, or local planning artifacts.
@@ -83,6 +90,12 @@ Coordinate expensive checks across agents. If a test unexpectedly consumes heavy
For tool design, follow [docs/tool-design.md](docs/tool-design.md): start from the analyst question, reuse existing contracts, preserve caller choices, and return useful evidence inline. Limits must follow real format, protocol, authority, or resource constraints. Account for representation expansion before allocation; derive completeness from examined coverage.
For user- or agent-facing tool problems, prioritize fixing the owning behavior,
contract, and actionable result or recovery at the point of use. Document the
result in product docs when explanation is needed. Add skill guidance only for
workflow judgment or context that the tool cannot reasonably convey; do not use
a skill as the primary repair for a discoverability or contract defect.
For test changes or provider claims, use [docs/testing.md](docs/testing.md). Select applicable [boundary regressions](docs/testing.md#boundary-regressions) for parser, inventory, snapshot, and projection fixes. Prefer real public CLI/MCP workflows, then production-boundary integration, then producer goldens. Keep module tests for distinct semantics stronger workflows cannot reliably reproduce. Preserve malformed-input, cancellation, cleanup, permission, capacity, and target-identity coverage when pruning. Report unverified coverage explicitly; simulated providers do not establish real-provider behavior.
Use Conventional Commit subjects and PR titles. Mark breaking changes with `!` or a `BREAKING CHANGE:` footer. Describe behavior changes, relevant validation, and real-provider verification scope.
+3 -4
View File
@@ -16,10 +16,9 @@ import inventory. For current provider wiring, start with
[`src/composition/binary.ts`](src/composition/binary.ts) for deep candidates and
the other `src/composition/` factories for tool services, including
[`gpu.ts`](src/composition/gpu.ts) and [`model.ts`](src/composition/model.ts).
Live Frida and Cutter adapters are constructed directly in
[`src/cli/fridaCommands.ts`](src/cli/fridaCommands.ts),
[`src/cli/cutterCommands.ts`](src/cli/cutterCommands.ts), and
[`src/server/createServer.ts`](src/server/createServer.ts).
Live Frida, Cutter, GDB and Rizin construction also uses shared composition
factories. For service ownership, availability and catalog declarations, follow
the [capability composition guide](docs/capability-composition.md).
Production factories construct fresh providers without starting engines or
acquiring targets; optional adapter failures must preserve successful peers. Run
`npm run verify:test-discovery` after adding or moving tests. Resolved import
+2 -1
View File
@@ -17,6 +17,7 @@ notices supplied with the software you actually obtain or redistribute.
| ------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| npm dependencies | [Complete inventory](licenses/npm-dependencies.txt) of every entry in `package-lock.json`, including transitive, optional platform, development, and website dependencies. [Runtime notices](licenses/npm-runtime-notices.txt) preserve the copyright, license, and NOTICE documents from integrity-verified release archives, plus the MIT notice for Rolldown's generated runtime helpers. Development dependencies are inventoried but are not all distributed in REA's npm package. |
| pefile ordinal tables | Import-hash ordinal names are derived from [pefile 2024.8.26](https://github.com/erocarrera/pefile/tree/4b3b1e2e568a88d4f1897d694d684f23d9e270c4), copyright 2004–2024 Ero Carrera. Preserve the [MIT license](licenses/pefile-2024.8.26-LICENSE.txt). The Python package is not a runtime dependency. |
| UF2 family names | UF2 family IDs and short names are copied from [microsoft/uf2 `utils/uf2families.json`](https://github.com/microsoft/uf2/blob/cf3bd7f9c274b949528f8cd6a19eef2502a687d2/utils/uf2families.json), copyright Microsoft Corporation. Preserve the [MIT license](licenses/microsoft-uf2-MIT.txt). |
| NativeAOT parser | REA's derived parser uses the MIT-licensed [ghidra-nativeaot](https://github.com/Washi1337/ghidra-nativeaot/tree/effeb734fc570c32650f88b159608979dc7b423e) work of Washi and the NativeAOT layout definitions in [dotnet/runtime](https://github.com/dotnet/runtime/tree/a3b2d40328be0be3f8dd43f0e5cc925b8827b044). Preserve [Washi's MIT notice](licenses/nativeaot-parser-MIT.txt) and the [.NET Foundation MIT notice](licenses/dotnet-runtime-MIT.txt). |
| EVM analysis | `[email protected]`, MIT, copyright 2023 Maxim Andreev. The published npm archive omits its license document; REA retains the [original upstream license](third_party/evmole/LICENSE) and [provenance](third_party/evmole/README.md). It is also included in the runtime notices. |
| Windows native addon | The release addon is compiled with MinGW-w64 and statically links GCC runtime libraries. Preserve the [MinGW-w64 runtime notices](licenses/MinGW-w64-runtime.txt), [GPLv3 text](licenses/GPL-3.0.txt), and [GCC Runtime Library Exception 3.1](licenses/GCC-Runtime-Library-Exception-3.1.txt). The exception governs eligible compiled combinations; it does not relicense the runtime itself under MIT. The Node-API build headers are from Node 22.19.0; its [original license and component notices](licenses/Node-22.19.0-LICENSE.txt) are retained. Node itself is supplied by the user. Snapshot provenance and release-toolchain review requirements are in [licenses/README.md](licenses/README.md). |
@@ -59,7 +60,7 @@ pin inspection sources; they are not included in the npm package.
| ghidra-nativeaot | [MIT](https://github.com/Washi1337/ghidra-nativeaot/blob/effeb734fc570c32650f88b159608979dc7b423e/LICENSE.md) | Pinned inspection source; derived parser attribution above. |
| JADX | [Apache-2.0](https://github.com/skylot/jadx/blob/v1.5.6/LICENSE) | The audited jadx-headless-mcp source uses JADX 1.5.6 for Android decompilation; the engine JAR is separately supplied. |
| jadx-headless-mcp | [Apache-2.0](https://github.com/1013503897/jadx-headless-mcp/blob/5844800a486d2248fa949b74c9896285f5b54de2/LICENSE) | v0.7.1; pinned submodule and user-supplied JAR. |
| Kuna | [Apache-2.0](https://github.com/Noelo-Lab/kuna/blob/v1.833/LICENSE), with its [NOTICE](https://github.com/Noelo-Lab/kuna/blob/v1.833/NOTICE) | Bring your own `kuna`; REA verified 1.833 and never downloads, installs or bundles it. REA invokes it as a separate process. |
| Kuna | [Apache-2.0](https://github.com/Noelo-Lab/kuna/blob/v1.988/LICENSE), with its [NOTICE](https://github.com/Noelo-Lab/kuna/blob/v1.988/NOTICE) | Bring your own `kuna`; REA verified 1.988 and never downloads, installs or bundles it. REA invokes it as a separate process. |
| Apktool | [Apache-2.0](https://github.com/iBotPeaches/Apktool/blob/master/LICENSE.md) | User-supplied Android resource decoder. |
| Binwalk | [MIT](https://github.com/ReFirmLabs/binwalk/blob/4fdab3d464d97b68e0af9088df3f9e2e1545b21c/LICENSE) | v3.1.0; pinned submodule and user-supplied firmware engine. |
| Unblob | [MIT](https://github.com/onekey-sec/unblob/blob/1fcc7a0a584a70a96c31f5a276c20944d199a089/LICENSE) | 26.6.4; pinned submodule and user-supplied firmware engine. |
+264 -69
View File
@@ -40,6 +40,7 @@ import java.util.Locale;
import java.util.Map;
import java.util.Set;
import java.util.TreeMap;
import java.util.TreeSet;
import java.util.regex.Pattern;
import java.util.regex.PatternSyntaxException;
@@ -548,6 +549,11 @@ public final class ReaGhidraBridge extends HeadlessScript {
String softFloat = currentProgram.getOptions(ghidra.program.model.listing.Program.PROGRAM_INFO)
.getString("REA MIPS soft-float storage", null);
if (softFloat != null) result.add("mips_soft_float_storage", JsonParser.parseString(softFloat));
// Written by ReaGhidraRiscvCallingConvention after analysis; REA requires it
// exactly when the committed profile declares a RISC-V calling-convention policy.
String riscvCallingConvention = currentProgram.getOptions(ghidra.program.model.listing.Program.PROGRAM_INFO)
.getString("REA RISC-V calling convention", null);
if (riscvCallingConvention != null) result.add("riscv_calling_convention", JsonParser.parseString(riscvCallingConvention));
return result;
}
@@ -687,6 +693,7 @@ public final class ReaGhidraBridge extends HeadlessScript {
}
row.addProperty("status", relocation.getStatus().toString());
row.addProperty("type", relocation.getType());
row.add("symbol", GSON.toJsonTree(relocation.getSymbolName()));
row.add("values", GSON.toJsonTree(relocation.getValues() == null ? new long[0] : relocation.getValues()));
byte[] original = relocation.getBytes();
row.add("original_bytes_hex", original == null ? JsonNull.INSTANCE : GSON.toJsonTree(bytesHex(original)));
@@ -922,17 +929,33 @@ public final class ReaGhidraBridge extends HeadlessScript {
requireKeys(params, Set.of("document", "address"));
requireDocument(params);
Address address = requireAddress(params, "address");
String at = canonicalAddress(address);
// A well-formed address can still lack one exact source offset. Each
// outcome has its own code, so REA reports the reason rather than
// schema repair. Initialization, not the loaded flag, decides whether
// source bytes can exist: file-backed overlays are not loaded.
MemoryBlock block = currentProgram.getMemory().getBlock(address);
if (block == null || !block.isInitialized())
throw new RequestFailure("invalid_request", "Address has no initialized file-backed memory: " + canonicalAddress(address));
if (block == null)
throw new RequestFailure("address_unmapped",
"Address " + at + " is outside every Ghidra memory block");
if (!block.isInitialized())
throw new RequestFailure("address_uninitialized",
"Address " + at + " is in uninitialized Ghidra memory block " + block.getName());
TargetFileOffset mapping = targetFileOffset(block, address);
if (mapping.mappings() != 1)
throw new RequestFailure("invalid_request", "Address has " + mapping.mappings() + " authoritative target-file mappings: " + canonicalAddress(address));
if (mapping.mappings() == 0)
throw new RequestFailure("file_mapping_absent",
"Address " + at + " in initialized Ghidra memory block " + block.getName() +
" has no authoritative mapping to the admitted target file");
if (mapping.mappings() > 1)
throw new RequestFailure("file_mapping_ambiguous",
"Address " + at + " has " + mapping.mappings() +
" authoritative mappings to the admitted target file");
Long fileOffset = mapping.offset();
if (fileOffset == null)
throw new RequestFailure("invalid_request", "Address lacks a safe file offset: " + canonicalAddress(address));
throw new RequestFailure("file_offset_unrepresentable",
"Address " + at + " maps to an admitted target-file offset beyond the exact JSON integer range");
JsonObject result = new JsonObject();
result.addProperty("address", canonicalAddress(address));
result.addProperty("address", at);
result.addProperty("file_offset", fileOffset);
return result;
}
@@ -1931,41 +1954,62 @@ public final class ReaGhidraBridge extends HeadlessScript {
return symbol == null ? JsonNull.INSTANCE : GSON.toJsonTree(symbol.getName(true));
}
// Analyst naming: a function entry renames the function, other addresses get a primary label.
private boolean applyAddressName(Address address, String name) throws Exception {
// Analyst naming: a function entry renames the function; another mapped
// address renames its non-dynamic primary label or receives a new global
// label. Resolve that destination without editing the program.
private AddressNameTarget addressNameTarget(Address address, String name)
throws ghidra.util.exception.InvalidInputException {
if (!currentProgram.getMemory().contains(address))
throw new RequestFailure("invalid_request", "Address is not mapped: " + canonicalAddress(address));
validateAnnotationText(name, "name");
Symbol edited = editedNameSymbol(address);
return edited == null
? new AddressNameTarget(address, currentProgram.getGlobalNamespace(), name)
: new AddressNameTarget(address, edited.getParentNamespace(), symbolLeafName(edited, name));
}
private Symbol editedNameSymbol(Address address) {
Function function = currentProgram.getFunctionManager().getFunctionAt(address);
if (function != null) return function.getSymbol();
Symbol primary = currentProgram.getSymbolTable().getPrimarySymbol(address);
return primary == null || primary.isDynamic() ? null : primary;
}
// Every address whose label or function symbol holds the destination.
private List<Address> addressNameHolders(AddressNameTarget target) {
Set<Address> holders = new TreeSet<>();
for (Symbol symbol : currentProgram.getSymbolTable().getLabelOrFunctionSymbols(target.leaf(), target.namespace()))
holders.add(symbol.getAddress());
return List.copyOf(holders);
}
// Apply one planned destination in its own transaction. Null means the
// edit committed with matching primary-symbol readback.
private String applyAddressName(AddressNameTarget target) {
Address address = target.address();
int transaction = currentProgram.startTransaction("REA address name");
boolean commit = false;
try {
Function function = currentProgram.getFunctionManager().getFunctionAt(address);
SymbolTable table = currentProgram.getSymbolTable();
Symbol primary = function == null ? table.getPrimarySymbol(address) : function.getSymbol();
Namespace namespace = primary == null
? currentProgram.getGlobalNamespace()
: primary.getParentNamespace();
String expected = primary == null ? name : symbolLeafName(primary, name);
for (Symbol symbol : table.getLabelOrFunctionSymbols(expected, namespace)) {
if (!address.equals(symbol.getAddress())) return false;
}
Symbol edited = editedNameSymbol(address);
if (function != null) {
function.setName(expected, SourceType.USER_DEFINED);
function.setName(target.leaf(), SourceType.USER_DEFINED);
} else if (edited != null) {
edited.setName(target.leaf(), SourceType.USER_DEFINED);
} else {
if (primary != null && !primary.isDynamic()) {
primary.setName(expected, SourceType.USER_DEFINED);
} else {
Symbol created = table.createLabel(address, expected, SourceType.USER_DEFINED);
created.setPrimary();
}
Symbol created = currentProgram.getSymbolTable().createLabel(address, target.leaf(), SourceType.USER_DEFINED);
created.setPrimary();
}
invalidateAnalysisCaches();
Symbol readback = currentProgram.getSymbolTable().getPrimarySymbol(address);
commit = readback != null && expected.equals(readback.getName());
return commit;
commit = readback != null && target.leaf().equals(readback.getName());
return commit ? null : "readback_mismatch";
}
catch (ghidra.util.exception.InvalidInputException | ghidra.util.exception.DuplicateNameException exception) {
return false;
catch (ghidra.util.exception.InvalidInputException exception) {
return "invalid_name";
}
catch (ghidra.util.exception.DuplicateNameException exception) {
return "duplicate_name";
}
finally {
currentProgram.endTransaction(transaction, commit);
@@ -1975,7 +2019,17 @@ public final class ReaGhidraBridge extends HeadlessScript {
private JsonElement setAddressName(JsonObject params) throws Exception {
requireKeys(params, Set.of("document", "address", "name"));
requireDocument(params);
return GSON.toJsonTree(applyAddressName(requireAddress(params, "address"), requireString(params, "name")));
String key = requireString(params, "address");
Address address = requireAddress(params, "address");
String name = requireString(params, "name");
// A single edit keeps request errors for unmapped addresses and
// unrepresentable text; other outcomes share the batch representation.
if (!currentProgram.getMemory().contains(address))
throw new RequestFailure("invalid_request", "Address is not mapped: " + canonicalAddress(address));
validateAnnotationText(name, "name");
Map<String, String> requested = new LinkedHashMap<>();
requested.put(key, name);
return applyAddressNames(requested);
}
private JsonElement setAddressesNames(JsonObject params) throws Exception {
@@ -1983,34 +2037,129 @@ public final class ReaGhidraBridge extends HeadlessScript {
requireDocument(params);
if (!params.has("names") || !params.get("names").isJsonObject())
throw new RequestFailure("invalid_request", "names must be an object of address -> name");
Map<String, String> requested = new LinkedHashMap<>();
for (Map.Entry<String, JsonElement> entry : params.getAsJsonObject("names").entrySet())
requested.put(entry.getKey(), entry.getValue().getAsString());
return applyAddressNames(requested);
}
// Decide every request from the names held before any edit, so object
// member order never selects which address receives a name. Reply with
// per-key results and reasons; a contested destination is listed once in
// conflicts and failures refer to it by index.
private JsonObject applyAddressNames(Map<String, String> requested) {
Map<String, Address> addressesByKey = new LinkedHashMap<>();
Map<Address, List<AddressNameRequest>> requestsByAddress = new LinkedHashMap<>();
JsonObject result = new JsonObject();
for (Map.Entry<String, JsonElement> entry : params.getAsJsonObject("names").entrySet()) {
for (Map.Entry<String, String> entry : requested.entrySet()) {
Address address = null;
try {
Address address = parseReaAddress(entry.getKey());
address = parseReaAddress(entry.getKey());
requestsByAddress.computeIfAbsent(address, ignored -> new ArrayList<>())
.add(new AddressNameRequest(entry.getKey(), address, entry.getValue().getAsString()));
.add(new AddressNameRequest(entry.getKey(), address, entry.getValue()));
}
catch (RequestFailure failure) {
result.addProperty(entry.getKey(), false);
// The key keeps its own invalid_address outcome below.
}
addressesByKey.put(entry.getKey(), address);
}
Map<Address, AddressNameFailure> failures = new HashMap<>();
Map<AddressNameDestination, List<AddressNameTarget>> destinations = new LinkedHashMap<>();
for (Map.Entry<Address, List<AddressNameRequest>> selected : requestsByAddress.entrySet()) {
Address address = selected.getKey();
String name = selected.getValue().get(0).name;
if (selected.getValue().stream().anyMatch(request -> !request.name.equals(name))) {
failures.put(address, new AddressNameFailure("conflicting_alias_names", null));
continue;
}
if (!currentProgram.getMemory().contains(address)) {
failures.put(address, new AddressNameFailure("unmapped_address", null));
continue;
}
try {
AddressNameTarget target = addressNameTarget(address, name);
destinations.computeIfAbsent(target.destination(), ignored -> new ArrayList<>()).add(target);
}
catch (RequestFailure | ghidra.util.exception.InvalidInputException exception) {
failures.put(address, new AddressNameFailure("invalid_name", null));
}
}
for (List<AddressNameRequest> requests : requestsByAddress.values()) {
String name = requests.get(0).name;
boolean conflictingAliasNames = requests.stream()
.anyMatch(request -> !request.name.equals(name));
boolean success = false;
if (!conflictingAliasNames) {
try {
success = applyAddressName(requests.get(0).address, name);
}
catch (RequestFailure failure) {
// Keep per-address failures in the existing batch result shape.
}
// Reject a destination that several selected addresses request, or
// that another address holds, instead of letting order pick a winner.
List<AddressNameConflict> conflicts = new ArrayList<>();
List<AddressNameTarget> independent = new ArrayList<>();
for (List<AddressNameTarget> requesters : destinations.values()) {
AddressNameTarget first = requesters.get(0);
List<Address> holders = addressNameHolders(first);
String reason;
if (requesters.size() > 1) {
reason = "duplicate_requested_name";
} else if (holders.stream().allMatch(first.address()::equals)) {
independent.add(first);
continue;
} else if (holders.stream().anyMatch(holder ->
!holder.equals(first.address()) && !requestsByAddress.containsKey(holder))) {
reason = "name_held_by_unselected_address";
} else {
reason = "name_held_by_selected_address";
}
for (AddressNameRequest request : requests)
result.addProperty(request.inputAddress, success);
AddressNameConflict conflict = new AddressNameConflict(
first.namespace(),
first.leaf(),
requesters.stream().map(AddressNameTarget::address).sorted().toList(),
holders
);
conflicts.add(conflict);
for (AddressNameTarget target : requesters)
failures.put(target.address(), new AddressNameFailure(reason, conflict));
}
// Independent edits have distinct destinations that no other address
// holds, so their results do not depend on application order.
for (AddressNameTarget target : independent) {
String reason = applyAddressName(target);
if (reason != null) failures.put(target.address(), new AddressNameFailure(reason, null));
}
// Requesters belong to exactly one destination, so ordering conflicts
// by their first requester gives indexes independent of member order.
conflicts.sort(Comparator.comparing((AddressNameConflict conflict) -> conflict.requesters().get(0)));
Map<AddressNameConflict, Integer> indexes = new IdentityHashMap<>();
JsonArray contested = new JsonArray();
for (AddressNameConflict conflict : conflicts) {
indexes.put(conflict, contested.size());
JsonObject value = new JsonObject();
if (conflict.namespace().isGlobal()) value.add("namespace", JsonNull.INSTANCE);
else value.addProperty("namespace", conflict.namespace().getName(true));
value.addProperty("name", conflict.leaf());
value.add("requested_by", canonicalAddresses(conflict.requesters()));
value.add("held_by", canonicalAddresses(conflict.holders()));
contested.add(value);
}
JsonObject results = new JsonObject();
JsonObject failed = new JsonObject();
for (Map.Entry<String, Address> entry : addressesByKey.entrySet()) {
Address address = entry.getValue();
AddressNameFailure failure = address == null
? new AddressNameFailure("invalid_address", null)
: failures.get(address);
results.addProperty(entry.getKey(), failure == null);
if (failure == null) continue;
JsonObject value = new JsonObject();
if (address == null) value.add("address", JsonNull.INSTANCE);
else value.addProperty("address", canonicalAddress(address));
value.addProperty("reason", failure.reason());
if (failure.conflict() == null) value.add("conflict", JsonNull.INSTANCE);
else value.addProperty("conflict", indexes.get(failure.conflict()));
failed.add(entry.getKey(), value);
}
JsonObject result = new JsonObject();
result.add("results", results);
result.add("failures", failed);
result.add("conflicts", contested);
return result;
}
private static JsonArray canonicalAddresses(List<Address> addresses) {
JsonArray result = new JsonArray();
for (Address address : addresses) result.add(canonicalAddress(address));
return result;
}
@@ -2126,9 +2275,6 @@ public final class ReaGhidraBridge extends HeadlessScript {
throw new RequestFailure("invalid_request", "mode must be literal or regex");
}
boolean caseSensitive = requireBoolean(params, "case_sensitive");
List<InventoryItem> inventory = procedureSearch
? procedures().stream().map(FunctionEntry::item).toList()
: strings();
ValueMatcher matcher;
if (mode.equals("literal") && !caseSensitive) {
if (!params.has("literal_ascii_candidate"))
@@ -2142,7 +2288,26 @@ public final class ReaGhidraBridge extends HeadlessScript {
: regexMatcher(expression, caseSensitive);
}
JsonArray items = new JsonArray();
for (InventoryItem item : inventory) {
if (procedureSearch) {
// Search the names exact lookup accepts, keeping one row per
// procedure under its primary name.
for (FunctionEntry entry : procedures()) {
monitor.checkCancelled();
ProcedureNameMatch match = matchProcedureNames(entry, matcher);
if (!match.matched()) {
continue;
}
JsonObject item = resultItem(entry.item, null);
if (!match.aliases().isEmpty()) {
JsonArray aliases = new JsonArray();
for (String alias : match.aliases()) aliases.add(alias);
item.add("matched_aliases", aliases);
}
items.add(item);
}
return items;
}
for (InventoryItem item : strings()) {
monitor.checkCancelled();
if (!matcher.matches(item.value)) {
continue;
@@ -2152,6 +2317,36 @@ public final class ReaGhidraBridge extends HeadlessScript {
return items;
}
/**
* The one definition of the names that select a procedure, shared by exact
* lookup and search: its primary name and every other symbol at its exact
* entry, each in leaf or namespace-qualified spelling. A label inside the
* body never names the procedure. Matching secondary symbols are reported
* by their qualified names, sorted.
*/
private ProcedureNameMatch matchProcedureNames(FunctionEntry entry, ValueMatcher matcher) throws Exception {
Function function = entry.function;
boolean primary = spellingMatches(function.getName(), entry.item.value, matcher);
Set<String> aliases = null;
for (Symbol symbol : currentProgram.getSymbolTable().getSymbols(function.getEntryPoint())) {
monitor.checkCancelled();
String qualified = symbol.getName(true);
if (!spellingMatches(symbol.getName(), qualified, matcher)) continue;
if (qualified.equals(entry.item.value)) {
primary = true;
continue;
}
if (aliases == null) aliases = new TreeSet<>();
aliases.add(qualified);
}
return new ProcedureNameMatch(primary, aliases == null ? List.of() : List.copyOf(aliases));
}
/** A leaf is the suffix of its qualified spelling, so a literal found in it is found in both. */
private static boolean spellingMatches(String leaf, String qualified, ValueMatcher matcher) {
return matcher.matches(qualified) || (!leaf.equals(qualified) && matcher.matches(leaf));
}
private List<InventoryItem> names() throws Exception {
if (nameInventory != null) {
return nameInventory;
@@ -2254,26 +2449,13 @@ public final class ReaGhidraBridge extends HeadlessScript {
// exact symbol name, even when every character is a hexadecimal digit.
boolean explicitAddress = isExplicitReaAddress(value);
if (!explicitAddress) {
// Imported labels (for example Mach-O _main) can share a function
// entry without becoming its primary symbol; search accepts the same.
ValueMatcher selector = value::equals;
List<Function> matches = new ArrayList<>();
for (FunctionEntry entry : procedures()) {
monitor.checkCancelled();
Function function = entry.function;
Symbol symbol = function.getSymbol();
String qualified = symbol == null ? function.getName() : symbol.getName(true);
boolean matchesName = function.getName().equals(value) || qualified.equals(value);
if (!matchesName) {
// Imported labels (for example Mach-O _main) can share a
// function entry without becoming its primary symbol. Only
// exact entries qualify; an interior label is not a selector.
for (Symbol entrySymbol : currentProgram.getSymbolTable().getSymbols(function.getEntryPoint())) {
monitor.checkCancelled();
if (entrySymbol.getName().equals(value) || entrySymbol.getName(true).equals(value)) {
matchesName = true;
break;
}
}
}
if (matchesName) matches.add(function);
if (matchProcedureNames(entry, selector).matched()) matches.add(entry.function);
}
if (matches.size() > 1)
throw new RequestFailure("ambiguous", "Ghidra procedure name is ambiguous: " + value +
@@ -4981,7 +5163,20 @@ public final class ReaGhidraBridge extends HeadlessScript {
private record InventoryItem(Address address, String value, JsonObject facts, JsonObject symbol) {}
private record AddressNameRequest(String inputAddress, Address address, String name) {}
private record AddressNameDestination(long namespace, String leaf) {}
private record AddressNameTarget(Address address, Namespace namespace, String leaf) {
AddressNameDestination destination() {
return new AddressNameDestination(namespace.getID(), leaf);
}
}
private record AddressNameConflict(Namespace namespace, String leaf, List<Address> requesters, List<Address> holders) {}
private record AddressNameFailure(String reason, AddressNameConflict conflict) {}
private record FunctionEntry(Function function, InventoryItem item) {}
private record ProcedureNameMatch(boolean primary, List<String> aliases) {
boolean matched() {
return primary || !aliases.isEmpty();
}
}
private record FunctionAnalysisObservation(JsonObject dossier, List<String> limitations, JsonObject relationshipCoverage, boolean decompilerFailed) {}
private record CallBoundaryObservation(List<String> limitations, Integer unrepresentedCalleeCalls, JsonArray unresolvedCalls) {}
private record SwiftStringLiteralDecode(JsonArray literals, JsonArray omissions) {}
@@ -0,0 +1,283 @@
import ghidra.app.script.GhidraScript;
import ghidra.program.database.SpecExtension;
import ghidra.program.model.data.DataType;
import ghidra.program.model.data.DoubleDataType;
import ghidra.program.model.data.Float16DataType;
import ghidra.program.model.data.FloatDataType;
import ghidra.program.model.data.FunctionDefinition;
import ghidra.program.model.data.IntegerDataType;
import ghidra.program.model.data.StructureDataType;
import ghidra.program.model.lang.CompilerSpec;
import ghidra.program.model.lang.Language;
import ghidra.program.model.lang.PrototypeModel;
import ghidra.program.model.lang.Register;
import ghidra.program.model.listing.Function;
import ghidra.program.model.listing.Parameter;
import ghidra.program.model.listing.Program;
import ghidra.program.model.listing.VariableStorage;
import ghidra.program.model.pcode.Varnode;
import ghidra.program.model.pcode.XmlEncode;
import java.io.StringReader;
import java.io.StringWriter;
import java.util.ArrayList;
import java.util.Iterator;
import java.util.List;
import java.util.regex.Pattern;
import javax.xml.parsers.DocumentBuilderFactory;
import javax.xml.transform.OutputKeys;
import javax.xml.transform.TransformerFactory;
import javax.xml.transform.dom.DOMSource;
import javax.xml.transform.stream.StreamResult;
import org.w3c.dom.Document;
import org.w3c.dom.Element;
import org.w3c.dom.Node;
import org.w3c.dom.NodeList;
import org.xml.sax.InputSource;
/**
* Apply the committed RISC-V calling-convention policy after analysis. Ghidra's generic RV32
* compiler specification models ilp32d. Each policy derives a program-specific prototype from the
* program's own default prototype, as its {@link Policy} row and docs/ghidra-riscv.md state. The
* script installs the prototype as a specification extension and assigns it, with dynamic
* storage, to every non-thunk function, external function and function definition.
*/
public final class ReaGhidraRiscvCallingConvention extends GhidraScript {
static final String REPORT_KEY = "REA RISC-V calling convention";
/** RISC-V floating-point registers by ABI name: ft0-ft11, fs0-fs11 and fa0-fa7. */
private static final Pattern FLOAT_REGISTER = Pattern.compile("f[ast]\\d+");
/**
* What a policy changes in the default ilp32d prototype. Every policy also passes values of 9
* bytes or more, such as {@code long double}, by reference.
*
* @param floatEntryBytes the largest value a floating-point register may pass or return; 0
* removes those entries and every rule that assigns floating-point storage
* @param preservesFloatRegisters whether fs0-fs11 stay preserved across calls
*/
private record Policy(
String id, String prototype, int floatEntryBytes, boolean preservesFloatRegisters) {}
private static final List<Policy> POLICIES =
List.of(new Policy("ilp32-prototype-v1", "__rea_ilp32", 0, false));
@Override
public void run() throws Exception {
String[] arguments = getScriptArgs();
Policy policy =
POLICIES.stream()
.filter(candidate -> arguments.length == 1 && candidate.id().equals(arguments[0]))
.findFirst()
.orElseThrow(
() ->
new IllegalArgumentException(
"REA RISC-V calling convention requires one of the policies "
+ POLICIES.stream().map(Policy::id).toList()));
Language language = currentProgram.getLanguage();
if (!language.getProcessor().toString().equals("RISCV")
|| currentProgram.getDefaultPointerSize() != 4)
throw new IllegalStateException(
"REA " + policy.id() + " requires a 32-bit RISC-V program, not "
+ language.getLanguageID());
CompilerSpec spec = currentProgram.getCompilerSpec();
new SpecExtension(currentProgram)
.addReplaceCompilerSpecExtension(
derive(spec.getDefaultCallingConvention(), spec, language, policy), monitor);
PrototypeModel model =
currentProgram.getCompilerSpec().getCallingConvention(policy.prototype());
if (model == null)
throw new IllegalStateException("REA " + policy.prototype() + " was not installed");
selfCheck(model, policy, language);
int functions = 0;
List<Function> callables = new ArrayList<>();
currentProgram.getFunctionManager().getFunctions(true).forEach(callables::add);
currentProgram.getFunctionManager().getExternalFunctions().forEach(callables::add);
for (Function function : callables) {
// A thunk's signature is its target's; the target is in this list.
if (function.isThunk()) continue;
if (function.hasCustomVariableStorage()) function.setCustomVariableStorage(false);
function.setCallingConvention(policy.prototype());
if (policy.floatEntryBytes() == 0) {
List<VariableStorage> storage = new ArrayList<>();
storage.add(function.getReturn().getVariableStorage());
for (Parameter parameter : function.getParameters())
storage.add(parameter.getVariableStorage());
// An untyped function has no assigned storage yet; the decompiler applies the
// prototype when it recovers one.
requireIntegerStorage(
storage.toArray(VariableStorage[]::new), function.getName(true), true);
}
functions += 1;
}
int definitions = 0;
Iterator<DataType> types = currentProgram.getDataTypeManager().getAllDataTypes();
while (types.hasNext()) {
if (!(types.next() instanceof FunctionDefinition definition)) continue;
definition.setCallingConvention(policy.prototype());
definitions += 1;
}
// Written last: any failure above leaves no report, and REA refuses the session.
currentProgram.getOptions(Program.PROGRAM_INFO).setString(
REPORT_KEY,
"{\"policy\":\"" + policy.id() + "\",\"functions\":" + functions
+ ",\"definitions\":" + definitions + "}");
println(
"REA RISC-V " + policy.prototype() + ": functions=" + functions
+ " definitions=" + definitions);
}
/** The default ilp32d prototype changed as the policy states. */
private static String derive(
PrototypeModel base, CompilerSpec spec, Language language, Policy policy) throws Exception {
XmlEncode encoder = new XmlEncode();
base.encode(encoder, spec.getPcodeInjectLibrary());
DocumentBuilderFactory factory = DocumentBuilderFactory.newInstance();
factory.setFeature("http://apache.org/xml/features/disallow-doctype-decl", true);
Document document =
factory.newDocumentBuilder().parse(new InputSource(new StringReader(encoder.toString())));
Element root = document.getDocumentElement();
if (!root.getTagName().equals("prototype"))
throw new IllegalStateException("Unexpected default prototype encoding " + root.getTagName());
root.setAttribute("name", policy.prototype());
List<Element> floatEntries = new ArrayList<>();
NodeList entries = root.getElementsByTagName("pentry");
for (int index = 0; index < entries.getLength(); index += 1) {
Element entry = (Element) entries.item(index);
if (entry.getAttribute("storage").equals("float")) floatEntries.add(entry);
}
if (floatEntries.isEmpty())
throw new IllegalStateException("Default prototype has no floating-point entries");
if (policy.floatEntryBytes() == 0) {
List<Node> removed = new ArrayList<>(floatEntries);
NodeList rules = root.getElementsByTagName("rule");
for (int index = 0; index < rules.getLength(); index += 1) {
Element rule = (Element) rules.item(index);
if (usesFloatStorage(rule)) removed.add(rule);
}
if (removed.size() == floatEntries.size())
throw new IllegalStateException("Default prototype has no floating-point rules to remove");
for (Node node : removed) node.getParentNode().removeChild(node);
} else
for (Element entry : floatEntries)
entry.setAttribute("maxsize", Integer.toString(policy.floatEntryBytes()));
// Values over 2 x XLEN go by reference. Returns need no rule: a value that the return
// registers cannot hold already becomes a caller buffer passed in a0.
NodeList inputs = root.getElementsByTagName("input");
if (inputs.getLength() != 1)
throw new IllegalStateException("Default prototype has no single input list");
Element input = (Element) inputs.item(0);
Element last = null;
for (Node child = input.getFirstChild(); child != null; child = child.getNextSibling())
if (child instanceof Element element && element.getTagName().equals("rule")) last = element;
if (last == null || !joinsAnyValue(last))
throw new IllegalStateException("Default prototype's input list does not end with a join");
Element byReference = document.createElement("rule");
Element datatype = document.createElement("datatype");
datatype.setAttribute("name", "any");
datatype.setAttribute("minsize", "9");
byReference.appendChild(datatype);
byReference.appendChild(document.createElement("convert_to_ptr"));
input.insertBefore(byReference, last);
if (!policy.preservesFloatRegisters()) {
NodeList killedLists = root.getElementsByTagName("killedbycall");
Element killed = killedLists.getLength() == 0 ? null : (Element) killedLists.item(0);
if (killed == null) {
killed = document.createElement("killedbycall");
root.appendChild(killed);
}
List<Element> moved = new ArrayList<>();
NodeList preservedLists = root.getElementsByTagName("unaffected");
for (int list = 0; list < preservedLists.getLength(); list += 1) {
NodeList preserved = preservedLists.item(list).getChildNodes();
for (int index = 0; index < preserved.getLength(); index += 1) {
if (preserved.item(index) instanceof Element varnode
&& isFloatRegister(varnode, language))
moved.add(varnode);
}
}
for (Element varnode : moved) {
varnode.getParentNode().removeChild(varnode);
killed.appendChild(varnode);
}
}
StringWriter xml = new StringWriter();
var transformer = TransformerFactory.newInstance().newTransformer();
transformer.setOutputProperty(OutputKeys.OMIT_XML_DECLARATION, "yes");
transformer.transform(new DOMSource(document), new StreamResult(xml));
return xml.toString();
}
/** The installed prototype places each class of value as its policy states. */
private void selfCheck(PrototypeModel model, Policy policy, Language language) {
StructureDataType pair = new StructureDataType("rea_float_pair", 0);
pair.add(FloatDataType.dataType);
pair.add(FloatDataType.dataType);
VariableStorage[] storage =
model.getStorageLocations(
currentProgram,
new DataType[] {
DoubleDataType.dataType, FloatDataType.dataType, DoubleDataType.dataType,
IntegerDataType.dataType, pair, Float16DataType.dataType
},
false);
VariableStorage quad = storage[storage.length - 1];
if (quad.size() != currentProgram.getDefaultPointerSize() || quad.getRegister() == null)
throw new IllegalStateException(
"REA " + policy.prototype() + " does not pass a 16-byte value by reference: " + quad);
if (policy.floatEntryBytes() == 0)
requireIntegerStorage(storage, policy.prototype() + " self-check", false);
if (policy.preservesFloatRegisters()) return;
for (Varnode preserved : model.getUnaffectedList()) {
Register register = language.getRegister(preserved.getAddress(), preserved.getSize());
if (register != null && FLOAT_REGISTER.matcher(register.getName()).matches())
throw new IllegalStateException(
"REA " + policy.prototype() + " still preserves " + register.getName());
}
}
private static boolean joinsAnyValue(Element rule) {
List<Element> children = new ArrayList<>();
for (Node child = rule.getFirstChild(); child != null; child = child.getNextSibling())
if (child instanceof Element element) children.add(element);
return children.size() == 2
&& children.get(0).getTagName().equals("datatype")
&& children.get(0).getAttribute("name").equals("any")
&& children.get(1).getTagName().equals("join");
}
/** Whether a rule's action, at any depth, assigns floating-point storage. */
private static boolean usesFloatStorage(Element element) {
NodeList children = element.getChildNodes();
for (int index = 0; index < children.getLength(); index += 1) {
if (!(children.item(index) instanceof Element child)) continue;
if (child.getAttribute("storage").equals("float") || usesFloatStorage(child)) return true;
}
return false;
}
private static boolean isFloatRegister(Element varnode, Language language) {
if (!varnode.getTagName().equals("varnode")
|| !varnode.getAttribute("space").equals("register"))
return false;
Register register =
language.getRegister(
language.getAddressFactory().getRegisterSpace()
.getAddress(Long.decode(varnode.getAttribute("offset"))),
Integer.parseInt(varnode.getAttribute("size")));
return register != null && FLOAT_REGISTER.matcher(register.getName()).matches();
}
private static void requireIntegerStorage(
VariableStorage[] storage, String owner, boolean allowUnassigned) {
for (VariableStorage item : storage) {
if (item.isBadStorage() || (!allowUnassigned && item.isUnassignedStorage()))
throw new IllegalStateException("REA RISC-V storage unresolved for " + owner);
List<Register> registers = item.getRegisters();
if (registers == null) continue; // stack-only storage
for (Register register : registers) {
if (FLOAT_REGISTER.matcher(register.getName()).matches())
throw new IllegalStateException(
"REA RISC-V storage still uses " + register.getName() + " for " + owner);
}
}
}
}
+35 -12
View File
@@ -161,8 +161,20 @@ private func unavailableIdentity(pid: Int32, reason: String) -> ProcessIdentityO
)
}
enum ProcargsMalformedStage: String, CaseIterable {
case header
case argumentCount = "argument_count"
case executablePath = "executable_path"
case pointerSize = "pointer_size"
case pathPadding = "path_padding"
case arguments
case terminator
case trailingEntries = "trailing_entries"
}
enum RunTokenReadError: Error {
case invalidBuffer
case invalidBufferAt(ProcargsMalformedStage)
case duplicateToken
case invalidTokenEncoding
case environmentUnavailable
@@ -244,6 +256,8 @@ func processRunTokenObservation(
return unavailable(pid: pid, reason: "apple_vector_unavailable")
case .invalidBuffer:
return unavailable(pid: pid, reason: "malformed_procargs")
case .invalidBufferAt(let stage):
return unavailable(pid: pid, reason: "malformed_procargs_\(stage.rawValue)")
}
}
}
@@ -302,27 +316,30 @@ func parseProcessEnvironment(
in bytes: [UInt8],
pointerSize: Int = MemoryLayout<UnsafeRawPointer>.size
) throws -> ParsedRunToken {
guard bytes.count >= MemoryLayout<Int32>.size else { throw RunTokenReadError.invalidBuffer }
guard bytes.count >= MemoryLayout<Int32>.size else { throw RunTokenReadError.invalidBufferAt(.header) }
let argc = bytes.withUnsafeBytes { $0.loadUnaligned(as: Int32.self).littleEndian }
guard argc >= 0, Int(argc) <= bytes.count else { throw RunTokenReadError.invalidBuffer }
guard argc >= 0, Int(argc) <= bytes.count else { throw RunTokenReadError.invalidBufferAt(.argumentCount) }
var offset = MemoryLayout<Int32>.size
guard let executablePath = readString(in: bytes, offset: &offset) else {
throw RunTokenReadError.invalidBuffer
throw RunTokenReadError.invalidBufferAt(.executablePath)
}
guard pointerSize == 4 || pointerSize == 8 else { throw RunTokenReadError.invalidBuffer }
guard pointerSize == 4 || pointerSize == 8 else { throw RunTokenReadError.invalidBufferAt(.pointerSize) }
let pathRecordSize = Array("executable_path=".utf8).count + executablePath.count + 1
let padding = (pointerSize - pathRecordSize % pointerSize) % pointerSize
guard offset + padding <= bytes.count,
bytes[offset..<(offset + padding)].allSatisfy({ $0 == 0 }) else {
throw RunTokenReadError.invalidBuffer
throw RunTokenReadError.invalidBufferAt(.pathPadding)
}
offset += padding
let tokenPrefix = Array("REA_PROCESS_RUN_ID=".utf8)
var tokenInArgument = false
for _ in 0..<argc {
guard readString(in: bytes, offset: &offset, allowEmpty: true) != nil else {
throw RunTokenReadError.invalidBuffer
guard let argument = readString(in: bytes, offset: &offset, allowEmpty: true) else {
throw RunTokenReadError.invalidBufferAt(.arguments)
}
tokenInArgument = tokenInArgument || argument.starts(with: tokenPrefix)
}
guard bytes.last == 0 else { throw RunTokenReadError.invalidBuffer }
guard bytes.last == 0 else { throw RunTokenReadError.invalidBufferAt(.terminator) }
if offset == bytes.count {
throw RunTokenReadError.environmentUnavailable
}
@@ -340,7 +357,7 @@ func parseProcessEnvironment(
continue
}
guard let entry = readString(in: bytes, offset: &offset) else {
throw RunTokenReadError.invalidBuffer
throw RunTokenReadError.invalidBufferAt(.trailingEntries)
}
trailingEntries.append(entry)
}
@@ -361,22 +378,28 @@ func parseProcessEnvironment(
throw RunTokenReadError.appleVectorUnavailable
}
if trailingEntries[appleVectorStart...].contains(where: {
$0.starts(with: Array("REA_PROCESS_RUN_ID=".utf8))
$0.starts(with: tokenPrefix)
}) {
throw RunTokenReadError.ambiguousEnvironmentBoundary
}
let environmentEntries = trailingEntries[..<appleVectorStart]
var found: String?
for entry in environmentEntries {
if entry.starts(with: Array("REA_PROCESS_RUN_ID=".utf8)) {
if entry.starts(with: tokenPrefix) {
guard found == nil else { throw RunTokenReadError.duplicateToken }
let value = entry.dropFirst("REA_PROCESS_RUN_ID=".utf8.count)
let value = entry.dropFirst(tokenPrefix.count)
guard let decoded = String(bytes: value, encoding: .utf8) else {
throw RunTokenReadError.invalidTokenEncoding
}
found = decoded
}
}
// A changed process title can overwrite the original argument boundaries,
// causing a real environment entry to be consumed as an argument. A real
// argument can have this form too, so it proves neither ownership nor absence.
if tokenInArgument {
throw RunTokenReadError.ambiguousEnvironmentBoundary
}
return ParsedRunToken(runId: found, environmentEntryCount: environmentEntries.count)
}
+1
View File
@@ -70,6 +70,7 @@ export default defineConfig({
{ text: "Apple applications", link: "/apple-application-analysis" },
{ text: "Desktop installations", link: "/desktop-applications" },
{ text: "Firmware", link: "/firmware-analysis" },
{ text: "iNES containers", link: "/ines-containers" },
{ text: "SQLite databases", link: "/sqlite-databases" },
{ text: "Packet captures", link: "/packet-captures" },
{ text: "Signal recordings", link: "/signal-recordings" },
+5 -5
View File
@@ -8,7 +8,7 @@ graph TD
end
subgraph "Production Composition"
COMPOSITION["src/composition/<br/>fresh typed provider and session factories<br/>one-shot binding and independent optional loading<br/>static WASM/EVM descriptors: named contracts, factories, retryable lifetimes"]
COMPOSITION["src/composition/<br/>fresh typed provider and session factories<br/>one-shot binding and independent optional loading<br/>typed capability bindings: construction, availability and retryable lifetimes<br/>pure capability declarations feed provider catalogs"]
end
subgraph "Adapters"
@@ -23,6 +23,7 @@ graph TD
SESSION["SessionProviderRouter + BinarySession<br/>one immutable deep binding per target<br/>disjoint auxiliary routing"]
REGISTRY["AnalysisProviderRegistry<br/>sorted deep candidates<br/>deterministic selection, no fallback<br/>ProviderInspectionScope: prerequisite observation lifetime"]
RECORDS["src/application/investigation/<br/>composed InvestigationRecords + narrow ports<br/>EvidenceLedger and Unknown ownership"]
LIFETIMES["CapabilityScope<br/>host-owned service close capabilities<br/>released peers removed; failed cleanup retained for retry"]
SNAPSHOT["BinarySessionRecords + AnalysisSnapshotCache<br/>binary target/profile snapshots and session file ownership<br/>record coordination and post-commit notifications"]
JS_RECON["src/application/javascript/ + src/artifacts/javascript/<br/>JavaScript artifact reconstruction<br/>directory/ASAR reads<br/>AST-only bundle projection"]
end
@@ -99,6 +100,7 @@ graph TD
MAIN --> COMPOSITION
CLI --> COMPOSITION
SERVER --> COMPOSITION
COMPOSITION --> LIFETIMES
COMPOSITION --> SESSION
APP --> FIRMWARE
FIRMWARE --> PROCESS
@@ -149,8 +151,7 @@ graph TD
OBJDUMP --> DOMAIN
COMPOSITION --> RIZIN
RIZIN --> DOMAIN
CLI --> GDB
SERVER --> GDB
COMPOSITION --> GDB
GDB --> PROCESS
GDB --> DOMAIN
HOPPER --> HOPPER_BRIDGE
@@ -206,8 +207,7 @@ graph TD
NATIVE_READERS --> DOMAIN
COMPOSITION --> FORMAT_READERS
FORMAT_READERS --> DOMAIN
SERVER --> RUNTIME
CLI --> RUNTIME
COMPOSITION --> RUNTIME
RUNTIME --> DOMAIN
PROCESS --> FILESYSTEM
AUX --> FILESYSTEM
+107
View File
@@ -0,0 +1,107 @@
# Capability composition and ownership
REA keeps domain contracts and shared workflows separate from provider adapters
and CLI/MCP translation. A capability can have several provider identities; a
provider can contribute operations to several capabilities. Those associations
are source facts, while host availability is an observation.
## Source declarations and runtime bindings
`src/composition/capabilityDeclarations.ts` contains static provider identities,
named contract-group references and precise operation subsets. Complete groups
refer to `TOOL_CONTRACT_GROUPS` in the existing contract owner; family definitions
and provider catalogs read the same arrays, so a new group operation is included
without updating a copied provider list. Metadata imports only the group ID type
and pure metadata leaves; it must not load factories, schemas, engines or
generated catalogs. Catalog generation resolves groups and subsets against the
canonical contracts and rejects unknown groups/operations and conflicting
identities. New provider provenance must come from the implementation
that actually produces the Evidence, including format-specific capture adapters
and derived workflows.
`src/composition/capabilityServices.ts` binds typed service factories to those
capability keys. Each runtime entry owns construction, cleanup and any available
host/configuration facts or live probes. A service entry requires a close
capability; entries adapting managers retain their existing `closeAll` operation.
The host maps entries without erasing their individual service/port types. There
is no generic JSON operation dispatcher.
These two declarations have different loading needs. Static facts can be read
without loading a schema or provider graph. The full runtime composition is
materialized only when the MCP runtime needs it. One-shot CLI adapters import the
selected factory rather than constructing a complete capability host. CLI and MCP
reuse the same provider/service factories and application workflows, while
keeping their respective argument translation and lifetime.
## Resource lifetime
`CapabilityScope` registers a module's actual close capability when its instance
is composed. There is no independent server provider-close list to update.
Concurrent shutdown calls join the same cleanup attempt. A successful peer is
removed from pending cleanup; failed peers remain owned and are retried on the
next close. Synchronous throws and rejections without a reason remain failures,
and cannot prevent other peers from attempting cleanup. Shutdown seals further
ownership registration.
The host distinguishes these lifetimes:
- Host-owned service instances are released by that host.
- Caller-owned optional observations remain borrowed by SDK server instances;
their process-level owner closes them, including discarded discovery probes.
- CLI invocations use their existing finalizers to join cleanup and preserve the
original observation, cancellation and cleanup errors.
- Binary engines, bundle readers and investigation records retain their existing
owners. The `BinarySession` transition/drain barrier still coordinates target
changes, reset and disposal. A module refactor must not create independent
barriers that allow late work to repopulate reset records.
An application workflow may borrow a shared resource scope, such as byte entropy
using the host's artifact scope. Its wrapper does not acquire a second ownership
claim over the borrowed scope.
## Availability and registration
Module bindings provide host support, selected configuration and precisely typed
live availability probes through `CapabilityAvailabilityFacts`. Constructors do
not run these probes or launch engines. Session discovery invokes the existing
provider-owned probes only for advertised families; live failures remain fresh
observations. Caller policy, selected environments and injected providers retain
their existing precedence.
MCP registration remains in the MCP adapter. It consumes composed services,
record ports, admission and logging; it must not add fallback constructors that
can escape the host owner. Named contracts bind handlers and schemas. The complete
tool-family registrar map is checked by TypeScript, and catalog order never
selects a handler or schema.
## Extending a capability
1. Add or reuse the domain contract and application workflow, with explicit
Evidence provenance, coverage, unknowns and caller choices.
2. Put concrete construction in `src/composition/` and keep its default factory
lazy. Reference adapter-owned pure identity metadata from both the provider
and its static capability declaration.
3. Add the runtime binding if a host-owned service is needed. Bind cleanup and
availability/probes there, preserving borrowed versus owned resources.
4. Add the CLI/MCP argument adapters and their named registrations. A one-shot
command uses the shared selected factory and existing finalization helpers.
5. Run source-boundary, discovery, SDK-schema and relevant public workflow checks.
Exercise partial cleanup, retry, cancellation and caller-owned peers at the
applicable production seam; use actual applications/providers for routing and
complete Evidence verification. Recording ports do not prove real providers.
The constructor boundary guard covers the centralized Frida, Cutter, GDB and
Rizin constructors without rejecting legitimate type-only injected interfaces.
Other settled pure/provider boundaries continue checking type imports too. The
shared parser records module-edge erasure once. Whole-declaration type imports
are erased; inline type bindings retain their runtime module edge under
`verbatimModuleSyntax`. Static dynamic imports remain runtime edges too.
Tests belong with their semantic owner and must be
found exactly once; provider roots need the corresponding adapter discovery lane.
This is an internal composition contract. External plugin distribution, automatic
provider installation and a universal provider interface are separate concerns.
The [initial verification report](verification/capability-composition-2026-11-01.md)
records the executed public workflows, cleanup boundaries, resource measurements
and remaining provider qualification gaps for this composition increment.
+93 -8
View File
@@ -281,6 +281,18 @@ The inventory lists each segment with its ESP-IDF load classification, the
checksum and SHA-256 results, the descriptor and any Secure Boot v2 signature
blocks. See [ESP images](esp-image.md) for recognition, coverage and refusals.
Inspect an ESP flash dump, or one read from a flash address other than 0:
```sh
rea inspect-artifact flash.bin --json
rea inspect-artifact dump.bin --target-format esp-flash-image --flash-offset 0x1000 --json
rea extract-artifact flash.bin --json
```
The inventory lists the bootloader, the partition table and each partition,
with ESP image facts for app partitions and the OTA records. See
[ESP flash images](esp-flash-image.md) for table location, inputs and limits.
Inspect a Node.js SEA carrier without running it:
```sh
@@ -317,6 +329,16 @@ overlap and archive expansion checks. The native streaming decoder keeps the
existing 128 MiB Zstandard window cap; frames requiring a larger window are
refused. Deprecated method 20 and other unregistered ZIP codecs remain unsupported.
Inventory and extract UF2 or DfuSe regions:
```sh
rea inspect-artifact ./firmware.uf2 --json
rea extract-artifact ./firmware.dfu --json
```
See [UF2 and DfuSe images](uf2-dfuse.md) for recognition, families, numbering,
the DfuSe suffix CRC and limits.
Inventory and extract U-Boot FIT images:
```sh
@@ -593,6 +615,30 @@ cached result. `close_binary` accepts `snapshot_path` and optional
`overwrite: true` to save before releasing provider resources. A failed save
leaves the session open so the caller can resolve the output failure.
When a validated snapshot does not match the selected binding, the error's
`details.binding_mismatch` reports confirmed target identities, provider
identities, or profile digests that differ. `snapshot` is the saved value and
`selected` is the requested value. Multiple differences can be reported. A
`profile_comparison: "not_compared"` means the profile was not available at
that boundary; it does not establish a match. A profileless snapshot can report
`workflow_scope: "incompatible"` even when its target and profile match.
Choose a matching target, provider, and profile, or analyze without the snapshot
and save a new one for the intended binding.
For a profile mismatch, the saved snapshot contains
`binding.analysis_profile.parameters`. If `binary_session` reports an active
profile whose digest matches `binding_mismatch.profile_digest.selected`, its
`analysis_provider_binding.analysis_profile.parameters` can be compared with
those saved parameters. A rejected switch leaves the previous session active,
so its profile may be different from the attempted selection; check the digest
before using it for this comparison. A configuration difference does not by
itself establish why analysis results differ.
If no target is open and you intend to inspect the selected binding, open that
target without the snapshot, then check its profile digest in `binary_session`
before comparing parameters. This may start the selected provider; the saved
snapshot remains a separate file.
If a selected snapshot's file descriptor cannot close, `open_binary` reports
`cleanup_incomplete`. `binary_session` reports `cleanup_pending` even when no
target opened. Retry `close_binary` to release the retained descriptor before
@@ -651,12 +697,45 @@ with `details.reason: path-limit` and `boundary: filesystem-path`.
For application traces and version comparisons, see
[JavaScript application workflows](javascript-application-workflows.md).
MCP [retained Evidence references](mcp-contracts.md#retained-application-evidence-inputs)
belong to one connection. Separate CLI calls can consume full saved records.
`inspect-analysis-view REQUEST.json --evidence-bundle BUNDLE.json` resolves
the request's `retained-evidence` source ID from that authenticated bundle,
including bundles exported from an MCP session. The same view JSON works for
every supported parent kind; no target is reopened and no record is selected
by file order. Omit `--evidence-bundle` when the request supplies inline Evidence.
belong to one connection. Separate CLI calls can consume full saved records or
resolve exact retained IDs from an explicitly selected local bundle:
```bash
rea trace-application-feature request.json --evidence-bundle /absolute/path/evidence.json --json
```
Here `request.json` uses the same logical Evidence input as MCP, for example
`"application": {"kind":"retained-evidence","evidence_id":"ev_<64 lowercase hex characters>"}`.
Inline records and references may be mixed in the request's declared source
fields, including arrays and nested inputs. The bundle is read and authenticated
once for that command; no target is reopened, no producer is rerun, and no
record is selected by file order. Source identity, provenance, coverage and
partial or unknown facts agree with the complete inline form. When starting
from summary or inspection Evidence, select its
`normalized_result.parent_evidence_id` for a workflow requiring complete source
facts. Omit `--evidence-bundle` when all sources are inline.
The JSON workflow commands supporting this option are:
- `trace-application-feature`, `trace-javascript-semantics`,
`compare-application-versions`, `compare-javascript-export-shapes`,
`compare-source-to-bundle`, and `reconcile-javascript-runtime`.
- `import-managed-reconstruction`, `verify-managed-native-boundaries`, and
`project-managed-application-graph`.
- `project-android-application-graph`, `project-apple-application-graph`, and
`project-harmony-application-graph`.
- `compare-wasm-interfaces` and `inspect-analysis-view`.
For `inspect-analysis-view`, the request's source reference selects a parent
from the bundle for every supported view kind. Bundles exported from MCP work
for these commands. Its inline source form cannot also select a bundle.
A missing ID is refused with `details.reason: "missing"`;
supplying a bundle does not make incompatible Evidence valid for a workflow.
`compare-managed-members` keeps its two assembly paths, and
`compare-process-captures` keeps its two capture Evidence JSON paths. The bundle
option above selects an input; producer options that write a new bundle, such
as `inspect-managed-members --evidence-bundle`, retain their existing meaning.
File selectors remain available only on contracts that explicitly support them.
Compare two artifact inventories with `compare-artifacts`. It takes the same
`left` and `right` input as MCP `compare_artifacts`: complete Evidence, or an
@@ -669,7 +748,7 @@ rea inspect-artifact new.zip --json > new.json
rea compare-artifacts '{"left":{"kind":"evidence-file","path":"old.json"},"right":{"kind":"evidence-file","path":"new.json"}}' --json
```
A one-shot command keeps no session Evidence, so it refuses a
A one-shot `compare-artifacts` command keeps no session Evidence, so it refuses a
`retained-evidence` source with `details.reason: "missing"`. Select the record
from a saved bundle instead.
For an artifact summary selected from a bundle, include its complete parent
@@ -853,7 +932,13 @@ rea mcp --http --port 8765
`--host` and `--port` default to `MCP_HOST` and `MCP_PORT` from the environment,
then to `127.0.0.1` and an ephemeral port. A bind outside loopback requires
`MCP_ALLOWED_HOSTS` or `--allowed-hosts`, a comma-separated list of client host
names or IP addresses without URLs or ports. For example:
names or IP addresses without URLs or ports.
Repeat `--allowed-hosts` to append names, or put multiple names in one
comma-separated value. All occurrences are validated together and deduplicated
after hostname normalization; the explicit list replaces `MCP_ALLOWED_HOSTS`.
For example:
```sh
rea mcp --http --host 0.0.0.0 --allowed-hosts 127.0.0.1,rea.lan
+6 -2
View File
@@ -316,8 +316,12 @@ request to change it.
`runtime-configuration:` check. Re-run `rea setup` for that client when the
setting is on the registration allowlist, then reconnect so the new process
sees it.
- `invalid_request`: correct the reported fields from the current schema or
CLI help. Do not repeat the unchanged call.
- `invalid_request`: follow the specific recovery and reported issue. Inspect
the schema or CLI help when the input violates the contract or the response
asks for `inspect_input_contract`. A valid selector can instead require a
different choice, such as selecting a reported address; an inventory miss
does not establish that the target lacks the item. Do not repeat the
unchanged call when the response says it cannot succeed unchanged.
A restart is required to adopt a changed launch environment or registration.
The curated runtime settings above require no REA restart. It is not the first step for a transient
+8
View File
@@ -147,6 +147,14 @@ window-targeted renderer reload/crash actions, and synthetic `open-url` or
WebContents identities, process metrics, and IPC channel and value-shape metadata.
Payload values are not retained.
For the exact Playwright launch sequence, the active hook removes only REA's
bootstrap arguments from the application-visible `process.argv`. The vector keeps
the selected entry point and every caller argument, in order, including repeated
hook paths, `.asar` filenames, and debug-like options. The private launch handoff
is consumed before application code runs. If the observed sequence cannot be
matched exactly, argv stays unchanged and capture coverage reports `hook_conflict`;
REA does not guess an entry point or remove arguments by filename patterns.
`window_index` selects from Playwright's current Page list at the start of each action. REA resolves that Page to its corresponding BrowserWindow once; the action's `target` uses the BrowserWindow's capture-scoped `window_id`, matching the window inventory. Reordering either enumeration after selection does not retarget the action. Handle cleanup failures retain the completed action records and remain owned for provider cleanup retry.
Run the real fixture verifier with an operator-provided Electron runtime:
+206
View File
@@ -52,6 +52,15 @@
},
"required": ["kind", "operation", "timeout_ms"],
"additionalProperties": false
},
{
"type": "object",
"properties": {
"kind": { "type": "string", "const": "retry_cleanup" },
"operation": { "type": "string", "minLength": 1 }
},
"required": ["kind", "operation"],
"additionalProperties": false
}
]
},
@@ -115,6 +124,15 @@
},
"required": ["kind", "operation", "timeout_ms"],
"additionalProperties": false
},
{
"type": "object",
"properties": {
"kind": { "type": "string", "const": "retry_cleanup" },
"operation": { "type": "string", "minLength": 1 }
},
"required": ["kind", "operation"],
"additionalProperties": false
}
]
},
@@ -178,6 +196,15 @@
},
"required": ["kind", "operation", "timeout_ms"],
"additionalProperties": false
},
{
"type": "object",
"properties": {
"kind": { "type": "string", "const": "retry_cleanup" },
"operation": { "type": "string", "minLength": 1 }
},
"required": ["kind", "operation"],
"additionalProperties": false
}
]
},
@@ -241,6 +268,15 @@
},
"required": ["kind", "operation", "timeout_ms"],
"additionalProperties": false
},
{
"type": "object",
"properties": {
"kind": { "type": "string", "const": "retry_cleanup" },
"operation": { "type": "string", "minLength": 1 }
},
"required": ["kind", "operation"],
"additionalProperties": false
}
]
},
@@ -304,6 +340,15 @@
},
"required": ["kind", "operation", "timeout_ms"],
"additionalProperties": false
},
{
"type": "object",
"properties": {
"kind": { "type": "string", "const": "retry_cleanup" },
"operation": { "type": "string", "minLength": 1 }
},
"required": ["kind", "operation"],
"additionalProperties": false
}
]
},
@@ -367,6 +412,15 @@
},
"required": ["kind", "operation", "timeout_ms"],
"additionalProperties": false
},
{
"type": "object",
"properties": {
"kind": { "type": "string", "const": "retry_cleanup" },
"operation": { "type": "string", "minLength": 1 }
},
"required": ["kind", "operation"],
"additionalProperties": false
}
]
},
@@ -430,6 +484,15 @@
},
"required": ["kind", "operation", "timeout_ms"],
"additionalProperties": false
},
{
"type": "object",
"properties": {
"kind": { "type": "string", "const": "retry_cleanup" },
"operation": { "type": "string", "minLength": 1 }
},
"required": ["kind", "operation"],
"additionalProperties": false
}
]
},
@@ -493,6 +556,15 @@
},
"required": ["kind", "operation", "timeout_ms"],
"additionalProperties": false
},
{
"type": "object",
"properties": {
"kind": { "type": "string", "const": "retry_cleanup" },
"operation": { "type": "string", "minLength": 1 }
},
"required": ["kind", "operation"],
"additionalProperties": false
}
]
},
@@ -556,6 +628,15 @@
},
"required": ["kind", "operation", "timeout_ms"],
"additionalProperties": false
},
{
"type": "object",
"properties": {
"kind": { "type": "string", "const": "retry_cleanup" },
"operation": { "type": "string", "minLength": 1 }
},
"required": ["kind", "operation"],
"additionalProperties": false
}
]
},
@@ -619,6 +700,15 @@
},
"required": ["kind", "operation", "timeout_ms"],
"additionalProperties": false
},
{
"type": "object",
"properties": {
"kind": { "type": "string", "const": "retry_cleanup" },
"operation": { "type": "string", "minLength": 1 }
},
"required": ["kind", "operation"],
"additionalProperties": false
}
]
},
@@ -682,6 +772,15 @@
},
"required": ["kind", "operation", "timeout_ms"],
"additionalProperties": false
},
{
"type": "object",
"properties": {
"kind": { "type": "string", "const": "retry_cleanup" },
"operation": { "type": "string", "minLength": 1 }
},
"required": ["kind", "operation"],
"additionalProperties": false
}
]
},
@@ -728,6 +827,32 @@
"unpacked"
],
"additionalProperties": {}
},
{
"type": "object",
"properties": {
"logical_path": { "type": "string" },
"algorithm": { "type": "string", "const": "dfu-crc32" },
"declared_crc32": {
"type": "string",
"pattern": "^[a-f0-9]{8}$"
},
"calculated_crc32": {
"anyOf": [
{ "type": "string", "pattern": "^[a-f0-9]{8}$" },
{ "type": "null" }
]
},
"unpacked": { "type": "boolean" }
},
"required": [
"logical_path",
"algorithm",
"declared_crc32",
"calculated_crc32",
"unpacked"
],
"additionalProperties": {}
}
]
}
@@ -793,6 +918,15 @@
},
"required": ["kind", "operation", "timeout_ms"],
"additionalProperties": false
},
{
"type": "object",
"properties": {
"kind": { "type": "string", "const": "retry_cleanup" },
"operation": { "type": "string", "minLength": 1 }
},
"required": ["kind", "operation"],
"additionalProperties": false
}
]
},
@@ -856,6 +990,15 @@
},
"required": ["kind", "operation", "timeout_ms"],
"additionalProperties": false
},
{
"type": "object",
"properties": {
"kind": { "type": "string", "const": "retry_cleanup" },
"operation": { "type": "string", "minLength": 1 }
},
"required": ["kind", "operation"],
"additionalProperties": false
}
]
},
@@ -919,6 +1062,15 @@
},
"required": ["kind", "operation", "timeout_ms"],
"additionalProperties": false
},
{
"type": "object",
"properties": {
"kind": { "type": "string", "const": "retry_cleanup" },
"operation": { "type": "string", "minLength": 1 }
},
"required": ["kind", "operation"],
"additionalProperties": false
}
]
},
@@ -982,6 +1134,15 @@
},
"required": ["kind", "operation", "timeout_ms"],
"additionalProperties": false
},
{
"type": "object",
"properties": {
"kind": { "type": "string", "const": "retry_cleanup" },
"operation": { "type": "string", "minLength": 1 }
},
"required": ["kind", "operation"],
"additionalProperties": false
}
]
},
@@ -1045,6 +1206,15 @@
},
"required": ["kind", "operation", "timeout_ms"],
"additionalProperties": false
},
{
"type": "object",
"properties": {
"kind": { "type": "string", "const": "retry_cleanup" },
"operation": { "type": "string", "minLength": 1 }
},
"required": ["kind", "operation"],
"additionalProperties": false
}
]
},
@@ -1108,6 +1278,15 @@
},
"required": ["kind", "operation", "timeout_ms"],
"additionalProperties": false
},
{
"type": "object",
"properties": {
"kind": { "type": "string", "const": "retry_cleanup" },
"operation": { "type": "string", "minLength": 1 }
},
"required": ["kind", "operation"],
"additionalProperties": false
}
]
},
@@ -1171,6 +1350,15 @@
},
"required": ["kind", "operation", "timeout_ms"],
"additionalProperties": false
},
{
"type": "object",
"properties": {
"kind": { "type": "string", "const": "retry_cleanup" },
"operation": { "type": "string", "minLength": 1 }
},
"required": ["kind", "operation"],
"additionalProperties": false
}
]
},
@@ -1234,6 +1422,15 @@
},
"required": ["kind", "operation", "timeout_ms"],
"additionalProperties": false
},
{
"type": "object",
"properties": {
"kind": { "type": "string", "const": "retry_cleanup" },
"operation": { "type": "string", "minLength": 1 }
},
"required": ["kind", "operation"],
"additionalProperties": false
}
]
},
@@ -1297,6 +1494,15 @@
},
"required": ["kind", "operation", "timeout_ms"],
"additionalProperties": false
},
{
"type": "object",
"properties": {
"kind": { "type": "string", "const": "retry_cleanup" },
"operation": { "type": "string", "minLength": 1 }
},
"required": ["kind", "operation"],
"additionalProperties": false
}
]
},
+348
View File
@@ -0,0 +1,348 @@
# ESP flash images
REA reads a dump of an ESP32-family SPI flash, such as the output of
`esptool read-flash 0 ALL dump.bin` or `esptool merge-bin`, through
`inspect_artifact` and `extract_artifact`. The reader is static: no device,
esptool or ESP-IDF is needed, and no code is run.
The container format is `esp-flash-image`. Entries are the bootloader region,
the partition table sector and one file per partition. The bootloader and
each app partition are also decoded as [ESP images](esp-image.md), so their
headers, segments, integrity and descriptors appear in the root
`format_metadata`. Layout facts are cited to ESP-IDF v6.1
(`esp_flash_partitions.h`, `flash_partitions.c`, `gen_esp32part.py`,
`bootloader_common_loader.c`, `bootloader_utility.c` and each chip's
`esp_rom_caps.h`).
## Recognition
A file is a flash image when REA finds a partition table in the entry
structure ESP-IDF's bootloader verifies (`esp_partition_table_verify`):
32-byte `0x50AA` entries, at most one `0xEBEB` MD5 entry over the entries
before it, and a terminating entry within `0xC00` bytes. ESP-IDF accepts a
table with no partition entries, but REA requires one, so that neither
erased flash nor a lone MD5 entry is read as a table. A table whose MD5 does not match, or that has a second MD5 entry, is
still read, with `md5.status` `invalid` and a limitation, because the
bootloader would refuse it. The table is located in this order:
1. at the caller's `partition_table_offset`, when given; no scan follows;
2. at the default `0x8000`;
3. by scanning every `0x1000`-aligned flash sector for a table whose MD5
verifies. This location is derived, and the inventory says so.
A table never lies inside the located bootloader image, so the default
offset and the scan are tried only after the image's last segment. Without
a caller selection, the scan runs only when an ESP image sits at a ROM
bootloader offset, and a single app or bootloader image ends at its last
segment, so neither ordinary files nor single images are scanned. A C3 or
S3 dump begins with its bootloader, so flash layouts are detected before a
file is read as one ESP image.
Unless the caller names the table with `partition_table_offset`, REA
refuses two kinds of dump before looking for a table:
- an image of a known chip at another chip's ROM offset, such as an ESP32
bootloader at the start of a dump read from `0x1000`. The dump's origin
is not the flash address REA assumed, given or default, and a table found
there would place every partition at the wrong file offset; pass the
`flash_offset` where the dump starts. When a partition table sits where
the image's own ROM offset puts `0x8000`, after the image's segments, the
refusal names that offset.
Without a selected format such a dump opens as its bootloader image, whose
[ESP image](esp-image.md) facts say how to reopen it.
- an ESP8266 image at a ROM offset. ESP8266 dumps can hold the same table
format, and they keep the [ESP image](esp-image.md) reader's ESP8266
refusal.
When no table is found the layout is not recognized. This is an unknown
outcome, not a malformed file: flash encryption, which encrypts the
bootloader and the table, is one cause. When a table is found without a
located bootloader and `flash_offset` is omitted, a limitation says that
each partition's file offset assumes the dump starts at flash `0x0`.
## Caller inputs
| Input | Meaning |
| ------------------------- | ----------------------------------------------------------------------------- |
| `format: esp-flash-image` | Read the file as a flash dump; with no table found, refuse it |
| `flash_offset` | Flash address (`0x…`) of the dump's first byte; default 0 |
| `partition_table_offset` | Flash address (`0x…`, a multiple of `0x1000`) of the table; disables the scan |
MCP takes these on `open_binary`; the CLI takes `--target-format
esp-flash-image`, `--flash-offset` and `--partition-table-offset` on
`inspect-artifact` and `extract-artifact`. The offsets require the format, and
the format takes no `architecture` or `entry_point`. The inputs are part of
the target's identity, so reopening the same bytes with other inputs reads
them again instead of reusing the earlier reading. The `open_binary` summary
reports them as `flash_offset` and `partition_table_offset`, null when
omitted.
## Layout facts
The root `format_metadata` (`esp-flash-image`) records:
- `flash_offset`: its `value` and whether the `caller` gave it or it is the
`default`;
- `chip`: from the bootloader's `chip_id`, or null without a located
bootloader;
- `bootloader`: the region from the chip's ROM bootloader offset to the
table, with its ESP image facts;
- `partition_table`: its flash address, the `method` that found it
(`default`, `caller` or `scan`), the partition count and the MD5 `status`
(`valid`, `invalid` or `absent`) with the stored and computed digests;
- `apps`: ESP image facts for each app partition the dump holds; app
partitions declared over the same bytes share one decode;
- `otadata`: the otadata partition's records, or null without one;
- `boot_selection`: the app the bootloader would select, for every table;
- `partitions_outside_dump`: declared partitions that begin before the
dump's first byte, which have no entry;
- `undescribed`: ranges outside every region, each from a byte that is not
erased flash to the end of its gap.
The ROM loads the bootloader from `ESP_ROM_BOOTLOADER_OFFSET_FLASH`:
`0x1000` for ESP32 and ESP32-S2; `0x0` for ESP32-S3, C2, C3, C6, C61, H2 and
H21; and `0x2000` for ESP32-C5, H4, P4 and S31. A bootloader is located only
where its own `chip_id` says the ROM loads it.
### Partitions
Each partition entry is `partitions/<label>.bin` when the label is a safe
name of up to 16 letters, digits, `.`, `_` and `-` that starts with a
letter, digit or underscore and has at least one letter or underscore, is
not a Windows device name such as `NUL` or `COM1`, alone or before a dot,
and no other label repeats it, ignoring case as some file systems do; it is
`partitions/<index>.bin` otherwise. `source_path` keeps the label, the
table's own name for the partition. Its `format_metadata`
(`esp-flash-partition`) records the table `index`, `label` (null when not
UTF-8), `label_hex` (the label's bytes when it is not UTF-8, else null),
`type`, `subtype`, ESP-IDF's `type_name` and `subtype_name`,
`flash_address`, `size`, the `encrypted` and `readonly` flags and any
`unknown_flags`. The `encrypted` flag says how a device with flash
encryption stores the partition; it does not establish that the dump's
bytes are ciphertext.
Each partition entry also records where the bytes the dump holds are
written, relative to the partition's start: `written_offset`, the first
byte that is not erased flash (`0xFF`), `written_end`, just past the last
such byte, and `written_bytes`, how many bytes are not `0xFF`. A partition
erased throughout has null offsets and `written_bytes` `0`; a partition the
file holds none of has all three null. The range bounds the written
bytes and does not say whether erased flash lies between them; extract the
partition to read them. REA reads the bytes the partitions cover once, so
overlapping partitions share that read however many the table declares.
ESP-IDF 5.3 and later may declare primary `bootloader` and `partition_table`
entries over the bootloader region and the table sector. They are
partitions like any other, whose bytes are also `bootloader.bin` and
`partition-table.bin`, and they are exempt from the rule that partitions
follow the table, as in `gen_esp32part.py`.
Every partition from the dump's first byte on is an entry holding the bytes
the file has. One that the file holds only in part, as when an `esptool
merge-bin` dump ends inside its last partition, has an entry limitation that
gives the size the table declares; one that lies past the end of the file is
an entry with no bytes and a limitation, and has no image facts. A partition
that begins before the dump's first byte has no entry and is listed in
`partitions_outside_dump`. One unreadable partition does not hide the
others: partitions outside the dump or overlapping another are each reported
while the rest stay readable.
### App partitions and the bootloader
Each app partition and the bootloader region carry their entry's `path`
and ESP image `facts`:
`status` is `esp-image`, `erased` (the image header bytes are erased flash,
as in an empty OTA slot) or `unrecognized` (ciphertext or other data). For an
image they hold the [ESP image](esp-image.md) root facts, `segments`,
`signature_blocks`, `coverage` and `limitations`, with offsets relative to
the region. Extract a partition and inspect it as an ESP image for its
segments as entries.
The summary view (`inspect_artifact` with `detail: "summary"`, or the
inventory summary of `inspect_analysis_view`) reports the root facts under
their own kind, `esp-flash-image-summary`, and gives each of these images a
`summary` instead of its `facts`: `status`, `chip`, `entry_addr`,
`segment_count`, `load_ranges` (the load address, declared length and
classification of each segment that is mapped, loaded or unclassified),
`signature_block_count`, the `descriptor` kind with the fields that tell
builds apart (an app's `project_name`, `version`, `idf_ver`,
`secure_version` and `app_elf_sha256`; a bootloader's `version`, `idf_ver`
and `secure_version`), the `checksum` and `appended_sha256` statuses,
`coverage` and `limitations`. The default complete detail keeps the full
facts, every segment included, and the summary's `inventory_evidence_ids`
name the Evidence that holds them.
`erased` is the bootloader's view: with no image header at the start of the
slot, it finds no app there. An erased slot can still hold data further in,
even a complete app image. When it does, its entry's `written_offset` and
`written_end` say where, and a limitation says that the slot holds bytes the
bootloader does not read; REA does not look for images in them.
### OTA data
The otadata partition holds two `esp_ota_select_entry_t` records, one at the
start of each 4 KiB sector. Like the bootloader, REA reads the last otadata
partition the table declares, and the last factory app and OTA app of each
subtype; a limitation names any repeated declaration. REA reports each
record the partition and the dump hold: its `ota_seq`, `seq_label`,
`ota_state` and its name, `crc`, whether the CRC matches (CRC-32 of
`ota_seq` seeded with `0xFFFFFFFF`) and whether the bootloader would treat
it as valid.
### Boot selection
`boot_selection` is present for every table. It is derived as
`bootloader_utility_get_selected_boot_partition` does without app rollback
or anti-rollback, and `basis` names its branch:
- `otadata`: the valid record with the higher `ota_seq` (record 0 on a tie)
selects slot `(ota_seq - 1) % ota_app_count`, indexed by OTA subtype;
- `factory-fallback`: no record is usable, or the table has no OTA app, and
a factory app is declared; or records are set but none is valid, as after
a CRC mismatch, which selects the factory index whether or not a factory
app is declared;
- `ota_0-fallback`: no record is usable, or the table has no OTA app, and no
factory app is declared, so the bootloader selects boot index 0;
- `no-otadata`: the table declares no otadata partition, or one at offset 0,
which the bootloader takes for none, so it selects the factory index, as in
the common factory-only layout;
- `none`: the bootloader rejects the table, for an MD5 mismatch or a second
MD5 entry, or cannot read an otadata partition smaller than two 4 KiB
sectors, and resets without selecting an app;
- `unknown`: the dump lacks an otadata record the bootloader reads.
Without an OTA app the records do not matter, so a dump that lacks them
still selects a fallback. REA does not compare the partitions with the
flash size in the bootloader header, which the bootloader also checks.
`active_record` and `slot` are set for `otadata` and `slot` for
`ota_0-fallback`. `partition_index` is null when the table declares no
partition for the selected slot or fallback; the bootloader then tries the
other apps. `assumptions` list what the derivation leaves out: records used
as written (for the branches that read them), anti-rollback, load fallback,
and factory-reset and test-app GPIO holds and a wake from deep sleep; they
are empty for `none` and `unknown`. `running_partition` is always
`unknown`: at boot the bootloader may first mark a pending-verify record
aborted, check the app's secure version against eFuse, and fall back to
another partition when an app fails to load.
## Coverage and extraction
Erased flash (`0xFF`) between regions is described. In a gap between
regions, the first byte that is not erased starts an `undescribed` range to
the end of the gap, and makes coverage `unknown`; the rest of that range is
not examined. Coverage is also `unknown` when no bootloader is located, or
when a declared partition begins before the dump's first byte, since it has
no entry. A partition cut short or left out by the end of the file does not
make it `unknown`: its entry holds the bytes the file has, none for one past
the end, and its limitation says what the file lacks. Coverage describes the
inventory of the file, not the observation of every declared flash byte.
So artifact comparison classifies every partition entry of two dumps. Where
both lack the same bytes, as two `merge-bin` dumps of the same layout do,
the entries are equal; a dump against a longer one reports each partition
it cuts short or lacks as `changed`, and that entry's limitation says the
file did not hold the bytes.
`extract_artifact` writes every entry as an ordinary file, `nvs_keys`
partitions included. On a device without flash encryption an `nvs_keys`
partition holds the NVS encryption keys in plain text, and a dump of it
contains them. REA decrypts nothing, recovers no keys and does not interpret
NVS.
## Comparison
Artifact comparison uses each partition's `label`, `label_hex`, `type`,
`subtype`, `flash_address`, `size` and flags as its declaration, so a
partition that moves, changes type or flags, or is renamed with unchanged
bytes is `changed` in `metadata`. Partitions pair by path. The root facts
are not compared as a declaration: they derive from the dump's bytes, which
content comparison covers, and from the caller's `flash_offset` and
`partition_table_offset`, which are part of the target's identity and
decide each partition's byte range.
## Limits
- A flash dump inside another container, such as a ZIP, is classified from
its first bytes and so reads as an ESP image, not as a flash layout; extract
it first.
- ESP8266 dumps are refused; Secure Boot v1 bootloader digests and
flash-encrypted layouts are not read.
- Partitions are not checked against the flash chip size, which the
bootloader also verifies.
- NVS, FAT, SPIFFS and other data partitions are extracted, not decoded.
- An image nested in a partition is decoded for its facts; its segments are
not inventory entries.
## Verification
Synthetic fixtures with computed MD5s and CRCs, and one CRC otatool wrote,
cover the table rules, every way the table is located, the scan's chunk and
sector edges, the misplaced-origin guard with the origin omitted or given,
the ESP8266 refusal, the `0x0`, `0x1000` and `0x2000` ROM bootloader
offsets, single images that are not scanned, overlaps, primary entries,
truncation and partitions outside the dump, index names for unsafe labels,
repeated declarations, the otadata states and records, and each branch of
the bootloader's selection. CLI and stdio MCP tests exercise the inputs and
their identity and extraction. MCP `compare_artifacts` tests compare
partition declarations, labels that are not UTF-8 included, two dumps that
end inside their last partition, and such a dump against a full one.
The optional real lane builds its dumps with esptool and ESP-IDF's own tools:
```sh
REA_RISCV_CLANG=/path/to/clang REA_RISCV_LLD=/path/to/ld.lld \
REA_ESPTOOL_PYTHON=/path/to/python-with-esptool \
REA_ESP_IDF_PATH=/path/to/esp-idf \
npm run verify:esp-flash
```
It converts the conformance ELFs with esptool, builds a default, a
relocated and a primary-entry partition table with `gen_esp32part.py`,
assembles dumps with `esptool merge-bin`, one of them written from flash
`0x1000` with `--target-offset`, and writes OTA records with `otatool`
through `tests/conformance/esp/flash_file_tool.py`. That script replaces
parttool's esptool calls with reads and writes of the dump file, so
otatool's own OTA logic runs unchanged. The lane then checks:
- the bootloader and app facts against `esptool image-info`;
- every partition against `gen_esp32part.py`'s decoding of the table REA
extracted;
- both OTA records against `otatool read_otadata` after switching to slot 0,
then to slot 1, then erasing the otadata, and the derived selection after
each: slot 0, slot 1, then the factory fallback;
- that each extracted file is the dump's bytes at its range and that the
source is unchanged;
- a table with the primary `bootloader` and `partition_table` entries that
`--primary-bootloader-offset` fills, read with complete coverage and no
overlap or misplacement limitation; it has no otadata, so its boot
selection is `no-otadata` with the factory app;
- the relocated table, found by scanning and named with
`partition_table_offset`, and the dump written from `0x1000`, read with
`flash_offset` and, without it, left an ESP image whose `flash_dump`
names `flash_offset` `0x1000`;
- an app with a flipped byte beside valid partitions, and a truncated dump,
read with complete coverage and its missing partitions as entries with no
bytes.
The lane ran on macOS arm64 with Node 24.22.1, Homebrew Clang and LLD 21.1.8
and ESP-IDF v5.5.2's `gen_esp32part.py`, `otatool.py` and `parttool.py`,
once with esptool v5.6.0 and once with 4.12.0. Its `gen_esp32part.py` is
identical to v6.1's, and its otatool and parttool differ only in import style
and esptool command spelling. All checks passed in both runs. With v5.6.0 and
the build already cached, the run took 24.1 s elapsed, 16.9 s user CPU and
6.3 s system CPU under `/usr/bin/time -l`; the largest single process peaked
at a 278 MiB resident set, and the combined peak of all processes was not
measured.
The lane does not establish:
- dumps read from a device: its dumps come from `merge-bin`;
- chips other than ESP32-C3 and ESP32;
- `ota_state` and record validity, which `read_otadata` does not print, or
OTA states the bootloader writes itself, app rollback or anti-rollback;
- selection with gaps or repeated OTA subtypes, which otatool does not
write;
- flash-encrypted dumps, which are not recognized by design.
It does not run in CI.
+20 -3
View File
@@ -223,9 +223,26 @@ first byte that is not erased starts a range to the end of that region. A
slot that is neither a block nor erased is reported whole. Adjacent ranges
are merged.
A whole-flash dump of a chip whose bootloader is at offset 0, such as
ESP32-C3 or ESP32-S3, is recognized as that bootloader image; the partition
table, apps and data follow as undescribed bytes.
A whole-flash dump whose partition table REA finds is read as an
[ESP flash image](esp-flash-image.md). Without one, a dump of a chip whose
bootloader is at offset 0, such as ESP32-C3 or ESP32-S3, is recognized as that
bootloader image, and the partition table, apps and data follow as
undescribed bytes.
A dump read from a later flash address, such as an ESP32 dump read from
`0x1000` with `read-flash 0x1000 …` or `merge-bin --target-offset 0x1000`,
opens as its bootloader image. REA says so when the file shows it: the image
at file offset 0 is not an app, its chip's ROM loads the bootloader from a
later address (`0x1000` for ESP32 and S2, `0x2000` for C5, H4, P4 and S31),
and a partition table sits where that origin puts `0x8000` (file offset
`0x7000` for `0x1000`), after the image ends. Then `flash_dump` records
`flash_offset` and
`table_file_offset`, a limitation says to reopen the file with
`format: "esp-flash-image"` and that `flash_offset`, and `inspect_artifact`
lists the same `open_binary` call as a next probe. `flash_dump` is null for
every other image. Evidence that REA recorded before it reported the field
omits it; REA still reads and compares such Evidence, and the omission says
nothing about where the file was read from.
## Limits
+20
View File
@@ -300,6 +300,17 @@ v2 signature blocks. `extract-artifact` / `extract_artifact` writes one file
per segment and signature block. Reading is portable and needs no external
tool. See [ESP images](esp-image.md).
## Inspect ESP flash dumps
Use the same tools on a whole-flash dump, such as `esptool read-flash 0 ALL`
output. The inventory reports the bootloader, the partition table and how it
was found, each partition with its declared type, flags and digest, ESP image
facts for the bootloader and each app partition, and the OTA records with a
derived selection; the running partition stays unknown. Extraction writes
each partition, `nvs_keys` included, as an ordinary file. For a dump that
does not start at flash address 0, select `esp-flash-image` with
`flash_offset`. See [ESP flash images](esp-flash-image.md).
## Inspect extracted devicetrees
Use [`inspect_devicetree`](devicetree.md) / `rea inspect-devicetree` on an
@@ -329,6 +340,15 @@ payloads stay as stored. Intel HEX/S-record and FIT images are separate
readers and are not covered here. See [U-Boot legacy uImage](uimage.md) for
component padding, preserved names, resource policies and verification scope.
## Inspect UF2 and DfuSe images
Use `inspect-artifact` / `inspect_artifact` on a UF2 file for each familyID's
address regions and its block numbering, or on a DfuSe file for its targets,
elements and DFU suffix, then `extract-artifact` / `extract_artifact` for one
raw file per region or element. A DfuSe suffix CRC mismatch follows
`integrity_policy`. Decoding is portable and requires no external tool. See
[UF2 and DfuSe images](uf2-dfuse.md).
## Inspect U-Boot FIT images
Use `inspect-artifact` / `inspect_artifact` on a U-Boot FIT (`.itb`) for its
+20 -15
View File
@@ -39,9 +39,9 @@ bridge-reported `language_id` after import. A caller-selected
pin; combining the pinned profile with an environment override is a typed
refusal. A header that declares an ABI field but carries no record is still
refused. Two further boundaries apply: an inspection that could not examine
every ABI declaration source (for example a carved ELF whose section table
extends past the captured bytes without a program-header record) does not
establish the recordless admission, and an `EF_MIPS_ARCH_1` or
every ABI declaration source does not establish the recordless admission
(target opening already refuses a carved ELF whose section table extends past
the captured bytes, as described below), and an `EF_MIPS_ARCH_1` or
`EF_MIPS_ARCH_32` header that _declares_ an ABI is refused rather than
relabeled, because the declared-ABI lane is verified for
`EF_MIPS_ARCH_32R2` headers only. The recordless MIPS-I lane pins the same
@@ -77,13 +77,16 @@ argument ABI and unknown nonzero MSA values remain unsupported. Without a
separate ABI flags record, FP attribute values are admitted only when absent,
`ANY` (0), double precision (1), or FPXX (5), the values covered by the
recordless profile. Parsed values are retained in the MIPS analysis profile.
For carved ELFs whose section table extends
beyond the captured file, REA examines every complete captured section header
and records the unavailable headers as a limitation. The provider refuses an
incomplete ABI declaration inspection even when a program-header ABI record is
valid, because an unavailable section header could carry an unsupported or
conflicting declaration. A missing program-header record does not make ABI
declarations in uncaptured sections absent or supply a default ABI.
A carved ELF whose section header table extends beyond the captured file is
refused when the target is opened, before provider selection, as is every ELF
declaration beyond EOF ([installation](installation.md#ghidra)). The refusal
names the declared range and the file size. It applies even when a
program-header ABI record was captured: an uncaptured section header could
carry an unsupported or conflicting declaration, so the captured bytes are not
analyzed as a complete smaller program. The provider also refuses any ABI
declaration inspection that reports unexamined headers. A missing
program-header record does not make ABI declarations in uncaptured sections
absent or supply a default ABI.
The field layout follows
[`Elf_MIPS_ABIFlags_v0`](https://github.com/bminor/glibc/blob/master/elf/elf.h).
@@ -257,11 +260,13 @@ observations and does not relabel the upstream result as verified.
CLI/MCP discovery and decompilation, direct calls, target/profile/Evidence
identity, snapshot round-trip and final process cleanup remain covered.
For both byte orders, the lane also carves away the section table while retaining
all program-header payloads. It checks the actual Ghidra language, entry-address
decompilation, distinct artifact digest and missing-section limitations through
CLI Evidence and an MCP-produced snapshot. Caller-selected Ghidra heap limits
are passed to these child workflows.
For both byte orders, the lane also cuts the image one byte before its section
header table, after checking that every program header and payload stays
captured. `open_binary` and the CLI must both refuse that copy before provider
selection as `target_unavailable` with constraint `file_format_rejected`,
naming the declared section-table range and the captured size, and leave it
unchanged. Caller-selected Ghidra heap limits are passed to the lane's CLI and
MCP child workflows.
For soft-float fixtures the lane also checks, through `inspect_native_api`, the
exact parameter and return types and storage of float, double, mixed (pair
alignment and stack spill) and struct-returning functions; that a caller's
+7 -1
View File
@@ -31,7 +31,8 @@ operation-scoped JVM `OutOfMemoryError` observations retain their separate resou
classification. Automatic kernel or cgroup OOM attribution is not implemented.
After the authenticated startup handshake, rejected inputs (`invalid_request`,
`not_found`, `ambiguous`, or an invalid function name) use
`not_found`, `ambiguous`, an invalid function name, or an address without one
exact file offset) use
`diagnostics.diagnostic_scope: "request"` when their active output checkpoint is
available. They preserve the active request's
`failure_output`, target/profile identity and runtime/log coordinates, while
@@ -171,6 +172,11 @@ have no file backing. ELF mappings are observed
without independent format-specific load-image verification, as the result's
status and limitations state.
Relocation rows retain Ghidra's `symbol` value, including an empty string or
null when returned by Ghidra. It is a provider observation, not proof of the
runtime-resolved address. Older Evidence may not have recorded this field;
a relocation page requesting it reports that uncertainty.
For large relocation inventories, select MCP `inspect_native_load_image` with
`detail: "summary"`. It returns verification state, observed identity and counts,
while retaining complete Evidence. Use `normalized_result.parent_evidence_id`
+3 -3
View File
@@ -85,9 +85,9 @@ node scripts/verify-real-ghidra-mips.mjs --psp-missing-extension
npm run verify:ghidra:mips
```
The generic verifier also exercises a section-truncated firmware fixture. That
check uses generic MIPS symbols and language expectations and runs only in generic
mode; the PSP report explicitly records carved-PSP coverage as not run.
The generic verifier also checks that a section-truncated firmware copy is
refused when the target is opened. That check runs only in generic mode; the
PSP report explicitly records carved-PSP coverage as not run.
The maintained PSP fixture builds from `tests/conformance/psp/`, never executes,
and is independent of commercial game data. PSP GNU tools supply ELF/ABI,
+264 -40
View File
@@ -9,29 +9,43 @@ lane shape as [generic MIPS support](ghidra-mips.md).
ELF `EM_RISCV` (243) is retained as the provider-neutral `riscv` family. The
target also retains the raw ELF declarations that decide a lane: class, byte
order, `e_type`, and `e_flags`. Recognition does not mean every provider or
RISC-V variant is supported. Existing PE, managed and Mach-O architecture sets
are unchanged. Headerless inputs use the separate [raw-binary admission
order, `e_type`, `e_flags`, and the ISA string that `.riscv.attributes`
declares. Recognition does not mean every provider or RISC-V variant is
supported. Existing PE, managed and Mach-O architecture sets are unchanged.
Headerless inputs use the separate [raw-binary admission
boundary](raw-binary.md); raw RISC-V images are not supported.
The Ghidra lane admits little-endian ELF32 and ELF64 `ET_EXEC` images that
declare `EF_RISCV_FLOAT_ABI_DOUBLE` in `e_flags`, with no `EF_RISCV_RVE`,
`EF_RISCV_TSO`, or any bit outside `EF_RISCV_RVC | EF_RISCV_FLOAT_ABI`. These are
header declarations, not proof that every instruction conforms to that ISA.
The Ghidra lane admits little-endian `ET_EXEC` images with no `EF_RISCV_RVE`,
`EF_RISCV_TSO`, or any bit outside `EF_RISCV_RVC | EF_RISCV_FLOAT_ABI`, in one
of three lanes:
- ELF64 or ELF32 images that declare `EF_RISCV_FLOAT_ABI_DOUBLE` use Ghidra's
stock double-float model unchanged. Their `.riscv.attributes` is not read for
admission.
- ELF32 images that declare the soft-float ABI (`e_flags` float-ABI bits `0`,
ilp32) use REA's [ilp32 calling-convention policy](#ilp32). They must also
declare an RV32I base and only [decoded extensions](#isa-declarations) in
`Tag_RISCV_arch`.
These are header declarations, not proof that every instruction conforms to
that ISA.
Provider selection returns the specific unsupported constraint rather than
guessing a compatible profile:
| Declaration | Outcome |
| ----------------------------------------------- | ---------------------------------------------------------------------------------------- |
| Little-endian ELF64 `ET_EXEC`, double-float ABI | admitted, `RISCV:LE:64:default` + `gcc` |
| Little-endian ELF32 `ET_EXEC`, double-float ABI | admitted, `RISCV:LE:32:default` + `gcc` |
| Big-endian | refused; Ghidra 12.1.4 ships no big-endian RISC-V language |
| `ET_REL`, `ET_DYN`, other `e_type` | refused with the declared value |
| `EF_RISCV_RVE` | refused; the embedded 16-register profile (x0–x15) needs its own processor specification |
| `EF_RISCV_TSO` | refused; the total-store-order memory model is unverified here |
| Soft, single, or quad float ABI | refused; the pinned generic compiler specification models double-float |
| Reserved or newer `e_flags` bits | refused with the offending bits named |
| Declaration | Outcome |
| -------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------- |
| Little-endian ELF64 `ET_EXEC`, double-float ABI | admitted, `RISCV:LE:64:default` + `gcc`, lane `elf64-exec-le-double-float-generic-v1` |
| Little-endian ELF32 `ET_EXEC`, double-float ABI | admitted, `RISCV:LE:32:default` + `gcc`, lane `elf32-exec-le-double-float-generic-v1` |
| Little-endian ELF32 `ET_EXEC`, soft-float ABI, decoded extensions only | admitted, `RISCV:LE:32:default` + `gcc` with the `__rea_ilp32` prototype, lane `elf32-exec-le-soft-float-generic-v1` |
| ELF32 soft-float ABI without `Tag_RISCV_arch`, or with any other extension | refused; the reason names the extensions outside the decoded set |
| ELF32 single-float ABI (ilp32f) | refused until its [written policy](#ilp32f) is implemented and verified |
| ELF64 soft- or single-float ABI (lp64, lp64f), or the quad-float ABI | refused; no verified calling-convention policy |
| Big-endian | refused; Ghidra 12.1.4 ships no big-endian RISC-V language |
| `ET_REL`, `ET_DYN`, other `e_type` | refused with the declared value |
| `EF_RISCV_RVE` | refused; the embedded 16-register profile (x0–x15) needs its own processor specification |
| `EF_RISCV_TSO` | refused; the total-store-order memory model is unverified here |
| Reserved or newer `e_flags` bits | refused with the offending bits named |
`e_type` is the only load-shape declaration this lane reads. A fixed-address
`ET_EXEC` declaration does **not** establish static linkage: `PT_INTERP` and
@@ -58,15 +72,193 @@ with the `gcc` compiler specification per class, records
pin without changing the requested override. An override that matches the pin is
accepted and recorded as `configured-v1`.
The soft-float lane pins the same `RISCV:LE:32:default` language. Every RV32
language in Ghidra 12.1.4, including the deprecated soft-float `RV32I`, `RV32IC`
and `RV32IMC` variants in `old/riscv_deprecated.ldefs`, uses the same
`riscv.ilp32d.sla` decoder; only the compiler specification differs. REA keeps
the generic language and corrects the calling convention instead.
## Calling conventions
Ghidra's generic RV32 compiler specification, `riscv32-fp.cspec`, models the
ilp32d ABI. Its default prototype passes `float` and `double` values in
`fa0`–`fa7` and returns them in `fa0`, and it passes a structure of one or two
floating-point members in floating-point registers. An ilp32 image passes all
of these in integer registers. Measured on Ghidra 12.1.4 with an ilp32 build of
the fixture in `tests/conformance/c/riscv-float-abi.c`, the stock model
decompiles `rea_riscv_abi_pick(1.5f, 2.5f, sel)` as
`rea_riscv_abi_pick(in_fa0,in_fa1,0x3fc00000)`.
### Decision record: D3, RISC-V float ABIs
Decided in [#7477](https://github.com/morluto/rea/issues/7477).
- **Choice.**
- Admit little-endian ELF32 `ET_EXEC` images that declare the soft-float ABI
(ilp32) through a new lane, `elf32-exec-le-soft-float-generic-v1`. Lane ids
are built from their fields: `elf<class>-exec-le-<float ABI>-float-generic-v1`.
- Keep the pinned `RISCV:LE:32:default` language and `gcc` compiler
specification. After analysis, `ReaGhidraRiscvCallingConvention.java`
derives the program-specific `__rea_ilp32` prototype from the program's
default prototype as the [ilp32 policy](#ilp32) states. It installs the
prototype through a Ghidra specification extension and assigns it, with
dynamic storage, to every non-thunk function, every external function and
every function definition. Thunks follow their targets. The script fails,
and writes no report, if the derived prototype or any function still
places a value in a floating-point register, or if the prototype does not
pass a 16-byte value by reference.
- One derivation serves every policy. A policy row says how it changes the
default prototype's floating-point entries and preserved registers; every
policy passes values of 9 bytes or more by reference.
- The profile commits `riscv_calling_convention: ilp32-prototype-v1`. The
bridge reports the policy and its function and definition counts, and REA
requires that report exactly when the profile commits the policy.
- Admission requires a `Tag_RISCV_arch` declaration with an RV32I base and
only extensions that Ghidra 12.1.4's generic RV32 language implements with
p-code ([ISA declarations](#isa-declarations)).
- Soft-float profiles add `float_abi` and `arch` to `riscv_elf`. Double-float
profiles keep their exact parameters, so their digests, sessions and
snapshots are unchanged.
- ilp32f keeps its [written policy](#ilp32f) and stays refused until its
prototype is implemented and verified. Quad-float, RVE, TSO, and ELF64
soft- or single-float images stay refused.
- **Alternatives** (and what switching to each would take):
- Pin the deprecated soft-float languages (`RISCV:LE:32:RV32IMC` and others,
`riscv32.cspec`): REA's language catalog would have to read `old/`, the
variants cover only I, IC and IMC, Ghidra marks them deprecated, and no
stock specification models ilp32f.
- Assign `__rea_ilp32` only to callables whose types use floating-point
values, as the MIPS soft-float lane does: typed storage is the same, but
prototypes Ghidra infers for untyped code would keep the ilp32d model. In
an ilp32 image that contains F or D instructions, that model can still
claim `fa0`. Switching is one predicate in the Java script and a new
policy id.
- Admit with a limitation and no correction: a reduced-capability contract
that needs its own review, because arguments, returns and aggregates
become unverified.
- Keep refusing: the lane rows become refusals again.
- Admit images without `Tag_RISCV_arch`, or refuse a list of known
extensions instead of admitting a list: a declaration that REA cannot
check would be treated as compatible. Widening the decoded set later is
additive.
- Add `float_abi` and `arch` to double-float profiles as well: one shape for
every lane, but every existing RISC-V session and snapshot would be
invalidated.
- Ship ilp32 and ilp32f together (the question in #7477): ilp32f needs a
float-and-integer structure rule that Ghidra's model lacks, so it ships
separately with its own fixtures.
- **Semantic owner and consumers.**
- Owners:
- `src/domain/riscvIsaDeclaration.ts` owns the float-ABI vocabulary of
`e_flags` and the parsing of `.riscv.attributes` and `Tag_RISCV_arch`.
- `GHIDRA_RISCV_LANES`, `GHIDRA_RISCV_CALLING_CONVENTIONS` and
`GHIDRA_RISCV32_DECODED_EXTENSIONS` (`src/ghidra/GhidraRiscvProfile.ts`)
own admission, lane ids and the policy table.
- `bridge/ghidra/ReaGhidraRiscvCallingConvention.java` owns how each policy
becomes a Ghidra prototype.
- Consumers: target resolution, Ghidra provider admission, the analysis
profile, the provider client and launcher, the session handshake, project
history, the real-Ghidra lane and this page. Later work reuses the table:
ilp32f adds a row, and RISC-V ESP images (F6) select a row through D5.
- **Costs of changing it.**
- **Code:** S. A policy or lane is one table row plus the Java derivation.
- **Compatibility:** the soft-float lane id, `riscv_elf.float_abi`,
`riscv_elf.arch` and `riscv_calling_convention` are persisted in sessions,
snapshots and Evidence once released. A changed derivation ships as a new
policy id, such as `ilp32-prototype-v2`, which invalidates soft-float
snapshots only.
- **Migration:** none for admitted targets; double-float profiles are pinned
by tests. Previously refused soft-float targets become admitted.
- **Verification:** rerun `npm run verify:ghidra:riscv`.
### ilp32
Policy `ilp32-prototype-v1`, prototype `__rea_ilp32`. XLEN is 32 bits and no
floating-point register passes a value.
| Topic | psABI rule | `__rea_ilp32` |
| -------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------- |
| Integer and pointer values | `a0`–`a7`, then the stack in 4-byte-aligned slots | The default integer and stack entries, unchanged |
| `float` | One integer register, or a stack slot | The `fa0`–`fa7` entries and the `float` rule are removed, so a `float` takes the next integer register |
| `double` and 8-byte values | A register pair with the low word in the lower register. Named arguments need no aligned pair. With one register left, the low word goes in `a7` and the high word on the stack | The default `join` rule with stack spill: Ghidra writes `a2:4,a1:4` for a pair and `Stack[0x0]:4,a7:4` for the split |
| Aggregates up to 8 bytes | One or two integer registers, laid out as in memory, also when every member is floating-point | The homogeneous-float-aggregate rule is removed; the `join` rule applies |
| Aggregates over 8 bytes | By reference: the caller passes the address of a copy | The default rule that converts structures and unions of 9 bytes or more to pointers |
| Return values | As the first argument of the same type: `a0`, or `a0` and `a1`. Larger values go to a caller buffer whose address is an implicit first argument in `a0` | The `fa0`/`fa1` return entries and floating-point rules are removed; Ghidra's hidden-return fallback supplies the pointer in `a0` |
| Variadic arguments | The integer convention. A variadic argument with 8-byte alignment and size takes an aligned (even-numbered) register pair, or the stack; after the first stack argument, the rest go on the stack | The default variadic rule (`join` with alignment), unchanged |
| Preserved registers | `s0`–`s11` and `sp`; `gp` and `tp` are not allocated. No floating-point register is preserved | The default preserved list without `fs0`–`fs11`, which join the killed-by-call list |
| Scalars over 8 bytes | By reference (`long double` is 16 bytes), and returned through a caller buffer | A rule before the final join converts any value of 9 bytes or more to a pointer; a return that size already goes to a caller buffer |
Ghidra recovers the arguments of a variadic call from the registers the caller
writes, not from their types. A padding register that still holds an earlier
value can appear as an argument, and arguments after an unwritten register are
omitted.
### ilp32f
Written policy for `EF_RISCV_FLOAT_ABI_SINGLE` on ELF32. REA does not admit it
yet. ABI_FLEN is 32 bits.
| Topic | psABI rule |
| ------------------------------------------------------- | ------------------------------------------------------------------------------------------------ |
| `float` | `fa0`–`fa7` while one is free, then the integer convention |
| `double` and other values wider than 32 bits | The integer convention, as in ilp32 |
| Structure of one or two floats (after flattening) | One floating-point register per member when enough are free, otherwise the integer convention |
| Structure of one float and one integer, in either order | One floating-point and one integer register when both are free, otherwise the integer convention |
| Unions | Never flattened; the integer convention |
| Return values | As the first argument: `fa0`, `fa0` and `fa1`, or `fa0` and `a0` |
| Variadic arguments | The integer convention, as in ilp32 |
| Preserved registers | `s0`–`s11`, `sp`, and `fs0`–`fs11` for values of 32 bits or fewer |
The planned prototype limits the default floating-point entries to 4 bytes and
keeps the shared rule that passes values of 9 bytes or more by reference.
Observed on Ghidra 12.1.4, a `double` then takes an integer pair and a structure
of two floats takes `fa0` and `fa1`. Ghidra's model has no rule for a structure
of one float and one integer: it joins `struct { float; int; }` into `a1:a0`
instead of `fa0` and `a0`. The ilp32f policy needs that rule and its own
fixtures before it is admitted.
### ilp32d and lp64d
The double-float lanes use Ghidra's stock `riscv32-fp.cspec` and
`riscv64-fp.cspec` models unchanged. This page does not verify them. They share
the float-and-integer structure gap above.
## ISA declarations
The target reader finds the one `SHT_RISCV_ATTRIBUTES` section and reads
`Tag_RISCV_arch`, such as `rv32i2p1_m2p0_c2p0_zicsr2p0_zifencei2p0`. It
retains the string, the base and the extension names, or an unavailable reason
when the section is malformed, declares other vendors or scopes, or uses a
non-canonical string. A missing section or tag is recorded as absent.
Calling-convention lanes admit an image only when it declares an RV32I base
and only extensions that Ghidra 12.1.4's generic RV32 language implements with
p-code: `m`, `a`, `f`, `d`, `c`, `zicsr`, `zifencei`, and the subset names
toolchains record, `zmmul`, `zaamo`, `zalrsc`, `zca`, `zcf` and `zcd`. Any
other extension is refused. In particular:
- Ghidra 12.1.4 decodes no Zcb, Zcmp or Zcmt instruction.
- Its bit-manipulation, cryptography and packed-SIMD files follow draft
encodings and mark nearly every instruction unimplemented.
- Vendor extensions, such as Espressif's `xesploop`, `xespv` and `xespdsp`,
decode only as custom-opcode placeholders.
ESP-IDF builds ESP32-C2 and C3 images as `rv32imc_zicsr_zifencei` and ESP32-C5,
C6, C61, H2 and H21 images as `rv32imac_zicsr_zifencei_zaamo_zalrsc`, all ilp32.
Those base strings are admitted. Builds that add `_zba_zbb_zbs`,
`_zcb_zcmp_zcmt` or an `xesp` extension are refused.
## Evidence
The lane identity is `elf32-exec-le-double-float-generic-v1` or
`elf64-exec-le-double-float-generic-v1`, and `riscv_elf` retains `elf_class`,
`byte_order`, `type`, and the raw `flags` alongside the read-only booleans
`float_abi_double`, `rvc_declared`, `rve_declared`, and `tso_declared`. Those
fields report declarations the header makes; none of them is evidence of the
original compiler, of instruction conformance, or of a floating-point calling
convention beyond the ABI the pinned language models.
The lane identity is one of the lane ids above, and `riscv_elf` retains
`elf_class`, `byte_order`, `type`, and the raw `flags` alongside the read-only
booleans `float_abi_double`, `rvc_declared`, `rve_declared`, and
`tso_declared`. Soft-float profiles also retain `float_abi: soft`, the declared
`arch` string, and `riscv_calling_convention: ilp32-prototype-v1`. Those fields
report declarations the header and attributes make, and the policy REA applied;
none of them is evidence of the original compiler or of instruction
conformance. Session Evidence states the prototype and how many functions and
definitions use it.
## Verification and limits
@@ -76,9 +268,9 @@ tree ships no host decompiler binary, so `Decompiler/os/<platform>/decompile` an
[installation](installation.md#ghidra).
Run `npm run verify:ghidra:riscv` with an existing Ghidra 12.1.4 installation,
JDK 21+, and a RISC-V-capable Clang, LLD and LLVM readelf. The verifier acquires
no tools and never executes target code. Select installed tools explicitly when
they are not on `PATH`:
JDK 21+, and a RISC-V-capable Clang, LLD, LLVM readelf and LLVM objdump. The
verifier acquires no tools and never executes target code. Select installed
tools explicitly when they are not on `PATH`:
```sh
GHIDRA_INSTALL_DIR=/path/to/ghidra_12.1.4_PUBLIC \
@@ -86,20 +278,52 @@ JAVA_HOME=/path/to/jdk-21 \
REA_RISCV_CLANG=/path/to/llvm/bin/clang \
REA_RISCV_LLD=/path/to/lld/bin/ld.lld \
REA_RISCV_READELF=/path/to/llvm/bin/llvm-readelf \
REA_RISCV_OBJDUMP=/path/to/llvm/bin/llvm-objdump \
npm run verify:ghidra:riscv
```
The lane builds the committed freestanding C and assembly sources into RV32 and
RV64 `ET_EXEC` fixtures. Independent LLVM readelf observations establish ELF
declarations and symbol addresses. Public CLI and stdio MCP workflows verify
function enumeration, decompilation, pinned language/compiler identity, decoded
probe bytes, global data, and direct calls. It validates advertised MCP output
schemas and both JSON representations, saves and reopens snapshots, rejects
header-mutated soft-float and unsupported high-bit declarations, and checks
unchanged source bytes and owned cleanup. Failures retain the diagnostic
workspace; successful runs remove it and report the observed process lineage.
The lane builds four `ET_EXEC` fixtures from committed freestanding C and
assembly sources:
Module tests cover admission and malformed headers. Call-graph completeness,
string inventories, dynamic linkage, Andes or RVE/TSO images, and big-endian
targets are **not** established by this lane. `.riscv.attributes` ISA extensions
are not inspected; the generic language pin remains an analysis convention.
- RV32 and RV64 double-float builds (`rv{32,64}imafdc`) of `riscv.c`.
- An RV32 soft-float build of `riscv-float-abi.c` with ESP32-C3's
`-march=rv32imc_zicsr_zifencei -mabi=ilp32`, typed with DWARF.
- The same objects linked with `--strip-all`. Every allocated section keeps
its address and bytes, so the typed build's symbols locate its functions.
Independent LLVM readelf observations establish ELF declarations,
`Tag_RISCV_arch` and symbol addresses. Public CLI and stdio MCP workflows verify
function enumeration, decompilation, pinned language/compiler identity, the
committed lane and policy, decoded probe bytes, global data, and direct calls.
They validate advertised MCP output schemas and both JSON representations, save
and reopen snapshots, and check unchanged source bytes and owned cleanup.
For the soft-float builds the lane also checks:
- `fence.i` and `csrr` decode, as ESP32-C3's Zifencei and Zicsr declare.
- An LLVM objdump reading of `rea_riscv_abi_caller` finds each call-site word
where the [ilp32 policy](#ilp32) puts it: an unaligned `double` in `a1`/`a2`,
the `a7`/stack split, two floats in an integer pair, a large structure and a
`long double` passed by reference (the pointed-to copies hold the source
values), aligned variadic pairs, and an indirect call.
- `inspect_native_api` reports `__rea_ilp32` with exactly that storage for the
typed build. The stripped build has one integer word per parameter location
and returns in the typed build's registers, or in none when the result goes
through caller memory.
- The caller's pseudocode passes the source constants through that storage and
names no floating-point register.
- Retained Evidence states the prototype outcome; the stripped build has no
function definitions.
Header-mutated copies check admission. The RV32 double-float image relabelled as
soft-float (`e_flags` `0x1`), refused before this lane, is admitted through the
soft-float lane over CLI and MCP. The soft-float image relabelled as
single-float, RV64 soft-float, and unsupported high bits are refused with their
reasons. Failures retain the diagnostic workspace; successful runs remove it and
report the observed process lineage.
Module tests cover admission, malformed headers and attribute sections, the
handshake, and the conformance checks against recorded LLVM and Ghidra output.
Call-graph completeness, string inventories, dynamic linkage, Andes or RVE/TSO
images, big-endian targets, ilp32f images, and decoding beyond the probes are
**not** established by this lane.
+14 -14
View File
@@ -47,20 +47,20 @@ For runtime instrumentation, see the [Frida guide](./frida-instrumentation.md).
## Choose a guide
| Task | Guides |
| --------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Identify a desktop installation | [Runtime candidates, evidence and follow-up files](desktop-applications.md) |
| Inspect native functions, references, or types | [Native binaries](native-investigation.md), [Go metadata](go-binaries.md), [Rust dependencies](rust-binaries.md), [CPython pyc files](python-pyc.md), [PE certificates](pe-certificates.md), [PE imports](pe-imports.md), [PE TLS/load config](pe-load-config.md), [ELF notes](elf-notes.md), [Linux kernel modules](kernel-modules.md), [Apple IPS](apple-ips.md), [Erlang BEAM](erlang-beam.md), [offline ELF layout](binary-diagnostics.md), [recorded crashes](recorded-crashes.md) |
| Inspect native functions, references, or types | [Native binaries](native-investigation.md), [Go metadata](go-binaries.md), [Rust dependencies](rust-binaries.md), [CPython pyc files](python-pyc.md), [PE certificates](pe-certificates.md), [PE imports](pe-imports.md), [PE TLS/load config](pe-load-config.md), [ELF notes](elf-notes.md), [Apple IPS](apple-ips.md), [Erlang BEAM](erlang-beam.md), [offline ELF layout](binary-diagnostics.md), [recorded crashes](recorded-crashes.md) |
| Select a native provider or import format | [Ghidra semantics](ghidra-provider.md), [Windows](windows-ghidra-p0.md), [DOS](ghidra-dos.md), [raw bytes](raw-binary.md), [GameCube DOL](ghidra-dol.md), [MIPS](ghidra-mips.md), [PSP](ghidra-psp.md), [NativeAOT](ghidra-nativeaot.md), [IDA](ida-provider.md), [Kuna](kuna-provider.md), [JEB](jeb-analysis.md) |
| Analyze JavaScript or Electron artifacts | [Static analysis](javascript-artifact-reconstruction.md), [tracing and comparisons](javascript-application-workflows.md), [application graph](javascript-application-graph.md), [source recovery](javascript-recovery.md) |
| Compare static JavaScript with runtime observations | [Runtime attribution](javascript-runtime-observation.md), [reconciliation](javascript-runtime-reconciliation.md), [Electron observation](electron-observation.md), [Electron fuses](electron-fuses.md) |
| Inspect assemblies, packages, or firmware | [Godot PCK](godot-pck.md), [UnityFS](unityfs.md), [U-Boot uImage](uimage.md), [.NET](managed-code-analysis.md), [Python packages](python-packaged-applications.md), [Android](android-analysis.md), [APK signatures](apk-signature.md), [Apktool resources](apktool-resource-analysis.md), [ADB acquisition](adb-device-analysis.md), [Apple](apple-application-analysis.md), [Harmony](harmony-analysis.md), [firmware](firmware-analysis.md), [ESP images](esp-image.md), [WASM](wasm-artifacts.md), [EVM](evm-bytecode.md) |
| Observe or compare a browser scenario | [CDP observation](browser-observation.md), [scenarios](browser-scenario-contract.md), [runtime evidence](web-runtime.md), [historical captures](web-network-captures.md) |
| Serve recorded behavior as an agent environment | [Agent environments](agent-environments.md) |
| Trace captured website source | [Source locations](web-source-location.md), [module traces](website-module-trace.md), [script export](website-script-export.md) |
| Capture and compare a command | [Process capture](process-capture.md) |
| Use CLI/MCP and retain evidence | [CLI](cli.md), [MCP contracts](mcp-contracts.md), [agent prompts](mcp-prompts.md), [obligation ledgers](reconstruction-obligation-ledgers.md) |
| Task | Guides |
| --------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Identify a desktop installation | [Runtime candidates, evidence and follow-up files](desktop-applications.md) |
| Inspect native functions, references, or types | [Native binaries](native-investigation.md), [Go metadata](go-binaries.md), [Rust dependencies](rust-binaries.md), [CPython pyc files](python-pyc.md), [PE certificates](pe-certificates.md), [PE imports](pe-imports.md), [PE TLS/load config](pe-load-config.md), [ELF notes](elf-notes.md), [Linux kernel modules](kernel-modules.md), [Apple IPS](apple-ips.md), [Erlang BEAM](erlang-beam.md), [offline ELF layout](binary-diagnostics.md), [recorded crashes](recorded-crashes.md) |
| Inspect native functions, references, or types | [Native binaries](native-investigation.md), [Go metadata](go-binaries.md), [Rust dependencies](rust-binaries.md), [CPython pyc files](python-pyc.md), [PE certificates](pe-certificates.md), [PE imports](pe-imports.md), [PE TLS/load config](pe-load-config.md), [ELF notes](elf-notes.md), [Apple IPS](apple-ips.md), [Erlang BEAM](erlang-beam.md), [offline ELF layout](binary-diagnostics.md), [recorded crashes](recorded-crashes.md) |
| Select a native provider or import format | [Ghidra semantics](ghidra-provider.md), [Windows](windows-ghidra-p0.md), [DOS](ghidra-dos.md), [raw bytes](raw-binary.md), [GameCube DOL](ghidra-dol.md), [MIPS](ghidra-mips.md), [PSP](ghidra-psp.md), [NativeAOT](ghidra-nativeaot.md), [IDA](ida-provider.md), [Kuna](kuna-provider.md), [JEB](jeb-analysis.md) |
| Analyze JavaScript or Electron artifacts | [Static analysis](javascript-artifact-reconstruction.md), [tracing and comparisons](javascript-application-workflows.md), [application graph](javascript-application-graph.md), [source recovery](javascript-recovery.md) |
| Compare static JavaScript with runtime observations | [Runtime attribution](javascript-runtime-observation.md), [reconciliation](javascript-runtime-reconciliation.md), [Electron observation](electron-observation.md), [Electron fuses](electron-fuses.md) |
| Inspect assemblies, packages, or firmware | [Godot PCK](godot-pck.md), [UnityFS](unityfs.md), [U-Boot uImage](uimage.md), [.NET](managed-code-analysis.md), [Python packages](python-packaged-applications.md), [Android](android-analysis.md), [APK signatures](apk-signature.md), [Apktool resources](apktool-resource-analysis.md), [ADB acquisition](adb-device-analysis.md), [Apple](apple-application-analysis.md), [Harmony](harmony-analysis.md), [firmware](firmware-analysis.md), [ESP images](esp-image.md), [ESP flash images](esp-flash-image.md), [WASM](wasm-artifacts.md), [EVM](evm-bytecode.md) |
| Observe or compare a browser scenario | [CDP observation](browser-observation.md), [scenarios](browser-scenario-contract.md), [runtime evidence](web-runtime.md), [historical captures](web-network-captures.md) |
| Serve recorded behavior as an agent environment | [Agent environments](agent-environments.md) |
| Trace captured website source | [Source locations](web-source-location.md), [module traces](website-module-trace.md), [script export](website-script-export.md) |
| Capture and compare a command | [Process capture](process-capture.md) |
| Use CLI/MCP and retain evidence | [CLI](cli.md), [MCP contracts](mcp-contracts.md), [agent prompts](mcp-prompts.md), [obligation ledgers](reconstruction-obligation-ledgers.md) |
For development, use [testing](testing.md), [client verification](client-verification.md),
[tool design](tool-design.md), [admission limits and recovery](design/limits-and-recovery.md),
+72
View File
@@ -0,0 +1,72 @@
# Classic iNES container inspection
REA recognizes classic iNES from the `NES` + `0x1a` byte signature, independently
of the file extension. This read-only container inspection uses portable
Node.js file reads; it needs neither Ghidra nor an emulator.
```bash
rea inspect-artifact ./cartridge.nes --json
rea extract-artifact ./cartridge.nes --json
```
Through MCP, call `open_binary` with the local path, then `inspect_artifact`.
Use `extract_artifact` to materialize the regions in the fresh directory named
by its result. The caller owns that successful extraction directory.
## Header and file regions
Inspection returns `container-header` and `reconstruction` observations inline.
The single inventory Evidence under `substeps[].evidence.normalized_result`
contains the complete graph and an `ines` object:
- `ines.header`: original 16-byte `header_hex`, PRG/CHR file-unit counts, mapper,
raw flags 6/7, trainer and battery flags, and the declared mirroring mode.
- `occurrences[]`: each region's path, `source_location.offset` and
`source_location.length` in the original container. Follow `artifact_id` to
the node's observed SHA-256 and byte count.
- `ines.reconstruction`: ordered region paths, concatenated byte count,
SHA-256 and `verified: true`. REA accumulates the concatenation digest during
the same physical-order region scan that produces node hashes, then compares
it with the original whole-file digest. Final source freshness is also checked.
No reconstructed file is written during inspection.
Regions are `header.bin`, optional `trainer.bin` (512 bytes),
`prg/000.bin` onward (16 KiB each), `chr/000.bin` onward (8 KiB each), and
optional `trailing.bin`. Every input byte is retained, including unused header
fields and uninterpreted trailing data. Zero CHR-ROM units create no CHR region;
REA does not fabricate CHR-RAM contents.
After extraction, concatenate the materialized files in
`ines.reconstruction.region_order` from the inspection inventory. Their bytes
reproduce the input exactly. Graph path order is lexical and is not physical
file order; use the explicit region order or source offsets.
## Supported boundary
This increment supports classic iNES with a complete 16-byte header, nonzero
PRG count, complete declared trainer/PRG/CHR payload, and zero reserved bytes
12–15. Dirty legacy headers are refused because their upper mapper bits are
ambiguous; REA does not silently repair them. NES 2.0 and UNIF signatures,
unsupported header variants and truncated payloads are explicitly refused.
An arbitrary file named `.nes` is not admitted as iNES.
The PRG/CHR units describe file storage, not mapper bank sizes or CPU windows.
The mapper number and flags are header declarations, not proof of cartridge
hardware or runtime behavior. Bank switching, 6502/65816 disassembly, emulator
sessions and interpretation of console-specific trailing data remain outside
this lane. Raw-binary Ghidra analysis is a separate workflow. This container
increment addresses the first milestone of [#2072](https://github.com/morluto/rea/issues/2072);
the related [#1641](https://github.com/morluto/rea/issues/1641) tracks 6502/65816
native analysis with explicit load addresses and provider limitations. Extracted
PRG units still need caller-selected CPU windows before static code analysis;
their physical file offsets do not establish those addresses.
Source-owned byte fixtures verify original offsets and hashes, extraction and
byte-exact reconstruction, CLI/stdio MCP parity, malformed/unsupported inputs,
failed target-switch recovery, cancellation and source identity. These checks
do not establish emulator behavior or compatibility with every historical ROM.
The portable CLI/stdio MCP acceptance and source-identity checks run in Linux
and macOS CI. The same workflows have also been executed on native Windows x64.
Format references: [iNES](https://www.nesdev.org/wiki/INES) and
[NES 2.0](https://www.nesdev.org/wiki/NES_2.0).
+37 -3
View File
@@ -675,6 +675,20 @@ an existing Hopper launcher is selected through `HOPPER_LAUNCHER_PATH`. Ghidra
and IDA have their own provider-specific host requirements; Windows Ghidra uses
the [experimental P0 boundary](windows-ghidra-p0.md).
Bazzite uses RPM packages, but REA does not install Hopper automatically on it.
Setup skips the Hopper offer because Bazzite is outside the supported host list,
regardless of whether Xvfb is installed. Bazzite documents
[containers for installing Linux packages](https://docs.bazzite.gg/Installing_and_Managing_Software/Containers/).
If using a container, check its distribution and architecture against the list
above. Install REA and Hopper in that container, give it access to the files you
want to analyze, and configure your agent to launch REA there. Verify installation
and analysis before relying on that workflow.
`rea doctor` checks health; it does not offer to install Hopper. Run `rea setup`
for installation. Setup cannot prompt when its output is redirected or during a
container image build. To approve Hopper installation without prompts, run
`rea setup --yes --install-hopper`.
If the Linux package download receives an HTTP failure, setup includes the
observed HTTP status in its `download_failed` diagnostic. The status does not
establish the underlying cause or guarantee that retrying will succeed.
@@ -728,6 +742,12 @@ Rotating signing subkeys remain bound to that primary identity.
The feed row itself is not PGP-signed; doctor labels its display version as
`reported_release_version`, separately from the authenticated launcher identity.
Even a correctly signed package can contain a launcher whose hash disagrees with
the feed. REA stops before installing that package. When reporting this error,
include the release version, package format and both hashes from the diagnostic.
Each format needs its own check: a matching Ubuntu DEB does not establish that
the Fedora RPM matches.
The approval plan includes the caller-selected cache location. Approved setup
retains the signed archive, public key and release metadata under
`${XDG_CACHE_HOME:-$HOME/.cache}/rea/hopper-releases/<launcher-sha256>`.
@@ -761,6 +781,16 @@ host mount and the rest of `/tmp` remain unchanged; this fallback never invokes
`sudo`. `rea doctor --provider hopper --json` reports the selected
strategy and both host and effective mount facts.
You do not need to start a separate `Xvfb :99` server. REA starts and manages its
own display. After installation, run `rea doctor --provider hopper --json` and
check `hopper-demo-runtime` for display and Python problems. Setup also runs these
health checks: installing the package is only one step toward a working provider.
If an image build ends with `exit status 1`, read the setup output immediately
before that summary to find the failing step. On minimal Ubuntu images, run
`apt-get update` before setup if an earlier layer deleted the package indexes.
REA's package installation commands do not refresh those indexes.
Hopper also embeds Python; the Python executable and standard library alone
are insufficient on Ubuntu. Approved setup checks the system interpreter
at `/usr/bin/python3` and, when necessary, installs its matching
@@ -971,7 +1001,9 @@ including while a shutdown request is in flight. A helper exit alone is
insufficient, and an application that never authenticated has no verified
process identity. If no matching bridge becomes available, inspect and close
the reported document in Hopper before ending the owning REA connection.
Export the Evidence bundle before `close_binary` clears session records.
`close_binary` preserves Evidence and complete Unknown history in memory on the
same connection. Export the bundle before ending the REA process to resume later;
`reset_investigation(save_path)` checkpoints and explicitly clears those records.
`binary_session.cleanup_pending` remains true while provider cleanup is retained,
and provider health remains visible even with `open: false`; that flag alone
does not establish successful cleanup.
@@ -1225,8 +1257,10 @@ bridge, and health readiness. Large binaries can need more: set
milliseconds in the server environment. An absent setting uses the default;
empty, invalid or out-of-range supplied values fail configuration validation.
When the deadline elapses, the `provider_timeout` error names
`REA_GHIDRA_STARTUP_TIMEOUT_MS` with the deadline that was applied and suggests
`REA_GHIDRA_DISABLED_ANALYZERS` for analyzers that dominate the time.
`REA_GHIDRA_STARTUP_TIMEOUT_MS` with the deadline that was applied. In
fresh-import mode it also suggests `REA_GHIDRA_DISABLED_ANALYZERS` for analyzers
that dominate the time and `REA_GHIDRA_PROJECT_PATH` for reusing a completed
analysis; in existing-project mode it omits that import and analyzer advice.
REA parses the supplied configuration environment and
passes the deadline to each provider client; a startup failure is returned by the query that triggered it,
without exposing partial analysis. This deadline is separate from MCP transport
+10 -4
View File
@@ -622,8 +622,10 @@ or process workflows.
## CLI and verification
All six CLI commands accept inline JSON or a path to a JSON file. The CLI
returns an Evidence record directly. Put the full records in a later CLI input;
a separate CLI process has no retained MCP connection state.
returns an Evidence record directly. Later workflow inputs can contain full
records, or exact retained references when `--evidence-bundle PATH` selects an
authenticated exported bundle. A separate CLI process has no retained MCP
connection state. See [CLI Evidence reuse](cli.md#import-export-and-compare-evidence).
File inputs must be regular files; symlinks to regular files are accepted.
Directories, named pipes and device files produce an input error before JSON
@@ -646,8 +648,12 @@ rea trace-application-feature ./trace-input.json --json
MCP analysis returns the complete Evidence record, with the operation result
in `normalized_result` and its identity in `evidence_id`. If the connected server advertises retained references, reuse its
exact returned ID as `{"kind":"retained-evidence","evidence_id":"RETURNED_ID"}`
in `application`, or `left`/`right` for comparisons. `RETURNED_ID` is a template,
not a literal valid ID. Native Evidence arrays still use complete records.
in `application`, or `left`/`right` for comparisons. For summary or inspection
replies, use `normalized_result.parent_evidence_id` to select the complete source.
`RETURNED_ID` is a template, not a literal valid ID. Native-observation arrays
and `reconcile_javascript_runtime`'s nested static layers and runtime-observation
array also accept references where advertised, independently mixed with inline
records. Exact identity, coverage and partial or unknown facts are preserved.
References are scoped to one connection; `close_binary` clears them. Export a
bundle before closing and import it on another connection, or supply the full
inline Evidence there. Versions before 4.1.0 accept full inline Evidence only; installing
+17 -4
View File
@@ -1,6 +1,6 @@
# Kuna provider
[Kuna](https://github.com/Noelo-Lab/kuna) is an Apache-2.0 decompiler distributed as one `kuna` executable. REA uses a `kuna` you supply. It does not download, install, update or bundle Kuna. REA verified this provider with Kuna 1.833; it accepts other releases and reports them as unverified in `rea doctor` and in each analysis profile.
[Kuna](https://github.com/Noelo-Lab/kuna) is an Apache-2.0 decompiler distributed as one `kuna` executable. For a new installation, use Kuna 1.988, REA's recommended verified release. REA uses a `kuna` you supply; it does not download, install, update or bundle Kuna. REA accepts other releases, which it reports as unverified in `rea doctor` and in each analysis profile. The verified-release flag compares the reported version; it does not authenticate the executable. The 1.988 real-provider checks ran on macOS arm64; Linux and other hosts have not been verified with that release.
Kuna is static: it reads the selected image and never executes it.
@@ -38,7 +38,7 @@ Windows is not supported yet. Supported targets are ELF and thin Mach-O executab
| Tool | Source |
| ----------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------- |
| `list_procedures`, `search_procedures`, `procedure_address` | Kuna's callable inventory (`kuna functions --json`), including import slots it does not decompile |
| `list_procedures`, `search_procedures`, `procedure_address` | Kuna's callable inventory (`kuna functions --json`), including import slots but not [Mach-O data](#mach-o-data-entries) |
| `address_name` | Function entries, then the data labels Kuna's export assigns |
| `procedure_pseudo_code`, `batch_decompile` | The project export, or a single-function run while the export builds |
| `procedure_assembly` | The project export's labeled listing |
@@ -47,6 +47,8 @@ Windows is not supported yet. Supported targets are ELF and thin Mach-O executab
| `xrefs` | Exact-address incoming reference sites, with forwarding aliases filtered explicitly |
| `analyze_function` | Pseudocode and assembly from the export, plus call relationships; typed references, comments, strings and CFG remain unavailable |
Kuna's inventory gives some entries aliases, such as ELF versioned-symbol names and generated `sub_` names. Name selectors accept an alias that names one entry and list every entry a shared alias names. `search_procedures` matches aliases too: each entry appears once under its primary name, with the matching aliases in `matched_aliases`. `list_procedures` shows primary names only.
Typed procedure references, basic blocks, raw-memory tools, annotations and type recovery are not available from this provider. Use another provider for those operations.
## How answers are produced
@@ -62,7 +64,7 @@ The first text or assembly request in a session looks for this target's export i
- Assembly and data names wait for the export; progress notifications report the wait every ten seconds. A session does not rebuild a failed export: requests that need it fail with `retryable: false`, the export's failure, its captured Kuna output and what to change before reopening the target. A setting such as `REA_KUNA_PROJECT_TIMEOUT_MS` or `REA_KUNA_JOBS` is read when REA starts, so changing it also needs a restart. Pseudocode keeps using single-function runs.
- A single-function answer says so in its limitations. It lacks the export's whole-program prototypes, so a later request can return different text for the same function.
Pseudocode, assembly, `analyze_function` and data-name answers record where their text came from in `raw_result.source`: `single-function`, or `project-export` with the cache directory. `procedure_pseudo_code` and `procedure_assembly` answers from the export also name the file, byte span and the file's SHA-256. Combined `analyze_function` answers retain both facet records in `raw_result.source.files`, each with its file, digest and byte span. Both texts come from one export; losing that project retries the complete report. Listings and procedure lookups carry no `raw_result`. Agents can open or search the export's `.c` and `.asm` there directly when that is cheaper than calling a tool per function. Text read that way is target-derived, like any decompiler output, but it bypasses the untrusted-content marker that REA's tool results carry.
Pseudocode, assembly, `analyze_function` and data-name answers record where their text came from in `raw_result.source`: `single-function`, or `project-export` with the cache directory. `procedure_pseudo_code` and `procedure_assembly` answers from the export also name the file, byte span and the file's SHA-256. Combined `analyze_function` answers retain both facet records in `raw_result.source.files`, each with its file, digest and byte span. Both texts come from one export; losing that project retries the complete report. Listings and procedure lookups carry no `raw_result`; omitted [Mach-O data entries](#mach-o-data-entries) are reported in their limitations. Agents can open or search the export's `.c` and `.asm` there directly when that is cheaper than calling a tool per function. Text read that way is target-derived, like any decompiler output, but it bypasses the untrusted-content marker that REA's tool results carry.
The background export belongs to the session. `close_binary`, or the end of a CLI command, cancels it unless another session still waits for it. A one-shot CLI decompile therefore answers from a single-function run; any request that needs the export, such as `procedure_assembly` or `analyze_function`, keeps it.
@@ -72,6 +74,17 @@ Regex procedure and string search uses ECMAScript Unicode matching in a cancella
Single-function subprocess output and each cached source facet share a 16 MiB buffered-text capacity. Cached spans are checked before allocation; larger text remains in the local export and the error identifies the file and range. `analyze_function` with `detail: "complete"` (the CLI default) returns the complete dossier inline, which can still be large; later retained views do not remove that first transfer. The MCP default retains the dossier and returns its summary, so page the facets you need through `inspect_analysis_view`, or prefer the individual text tools for a focused question.
## Mach-O data entries
Kuna's inventory also lists Mach-O exports that are data, such as globals in `__DATA,__data`, constants in `__TEXT,__const` and `__mh_execute_header`. REA classifies each entry with the target's own section table, the same load-command facts `inspect_macho` reports, read from the leading bytes kept while the session's private copy is hashed:
- A section with `S_ATTR_PURE_INSTRUCTIONS` or `S_ATTR_SOME_INSTRUCTIONS`, or of type `S_SYMBOL_STUBS`, holds code; the entry stays.
- A lazy or non-lazy symbol-pointer section (`__got`, `__la_symbol_ptr`) holds import slots; the entry stays.
- Any other section, or the header and load commands before the first section, holds data. Listings and searches omit the entry and say how many they omitted in each region. Pseudocode, assembly, `analyze_function` and `procedure_address` refuse it by name or address, and call queries refuse its name, naming its region; call queries by exact address stay validated by Kuna's graph. `address_name` and `xrefs` still accept it.
- An address outside every section and the header stays, with a limitation saying REA cannot classify it. If the section table cannot be read, every entry stays and listings say why. Answers about one such entry, including pseudocode, assembly, `analyze_function`, `procedure_address`, call queries by name and `address_name`, carry the same limitation; code and import slots carry none.
A section's attributes are the format's declaration, not proof of what runs there, and a kept entry is not proven to be a function. ELF inventories are kept as Kuna reports them. On a Mach-O target, the first listing or lookup establishes the private copy that Kuna commands also use, once per session, even when a warm export answers it.
## Strings
`list_strings` and `search_strings` share one lazy `kuna strings --json --encoding all --min-length 5 --termination any --no-xrefs` scan of the digest-checked private input. Skipping references avoids program lifting, a call graph and a C export. REA validates the input, scan policy, every row and the total before caching any facts. Encoding, character count, byte length and terminator facts must agree; supplementary UTF-8 characters count as Unicode scalars. Malformed output is an error, not an empty search. REA parses the JSON as Kuna writes it instead of buffering it, so the scan has no fixed size: Kuna writes about 290 bytes per row, 31 MB for a 20 MB Ghostty binary and 179 MB (611k rows) for an arm64 Firefox `XUL`. Rows are validated one at a time, and the first contradiction stops Kuna. Text REA does not keep is dropped unassembled; kept text and retained rows are admitted against the REA process's live heap headroom as they arrive; a refusal is a `memory` resource constraint with `retryable: false`, and closing other targets or starting REA with a larger Node.js heap makes room. String scans use the whole-program `REA_KUNA_PROJECT_TIMEOUT_MS` deadline; `REA_KUNA_TIMEOUT_MS` applies to single-function decompilation. The parsed inventory belongs to one selected target/executable profile and expires on session close; Kuna diagnostics beyond their 8 MiB buffer are retained as a failure until close, while other failures are not cached.
@@ -137,7 +150,7 @@ the manifest's file identity as well as the export files. These refusal and
migration cases are verified with controlled Kuna-format exports; they do not
establish behavior of an independently installed Kuna release.
Builds publish by one rename, so a reader sees a complete export or none. If later staging retirement fails, the typed cleanup error retains the completed export's key, directory and manifest in `published_export`. The session adopts that verified project; later queries reuse its immutable files while `project_export.cleanup_incomplete` reports the retained owner's live state. `close_binary` retries only the remaining cleanup, and a successful retry resolves prior cleanup errors. A failed export followed by failed staging cleanup retains both failures; it does not invent a publication receipt. A session keeps the export it uses from eviction only within its own REA process. If another REA process sharing the cache evicts it, the session rebuilds it; meanwhile pseudocode falls back to single-function runs. A damaged or edited export is removed and rebuilt. REA retains the file state verified with each digest, checks it before warm-cache reuse, and checks open descriptors before and after reading text. Assembly indexes are checked against the same state; a changed file makes the whole report retry rather than mixing facets from different exports. Staging moves to `trash/` for removal only after REA proves its Kuna process stopped. Staging of a process it cannot prove stopped is never swept; REA keeps it owned, retries the stop on `close_binary`, and reports it until the stop is proven. Failed file removal also stays bound to its export and is retried by `close_binary`; closing does not report success while that cleanup remains. `rea doctor --provider kuna` reports the cache directory without creating it.
Builds publish by one rename, so a reader sees a complete export or none. If later staging retirement fails, the typed cleanup error retains the completed export's key, directory and manifest in `published_export`. The session adopts that verified project; later queries reuse its immutable files while `project_export.cleanup_incomplete` reports the retained owner's live state. `close_binary` retries only the remaining cleanup, and a successful retry resolves prior cleanup errors. A failed export followed by failed staging cleanup retains both failures; it does not invent a publication receipt. A session keeps the export it uses from eviction only within its own REA process. If another REA process sharing the cache evicts it, the session rebuilds it; meanwhile pseudocode falls back to single-function runs. A damaged or edited export is removed and rebuilt. REA retains the file state verified with each digest, checks it before warm-cache reuse, and checks open descriptors before and after reading text. Assembly indexes are checked against the same state; a changed file makes the whole report retry rather than mixing facets from different exports. Staging moves to `trash/` for removal only after REA proves its Kuna process stopped. Staging of a process it cannot prove stopped is never swept; REA keeps it owned, retries the stop on `close_binary`, and reports it until the stop is proven; that MCP error names the same-connection retry as `recovery.kind=retry_cleanup` ([retained cleanup retry](mcp-contracts.md#retained-cleanup-retry)). Failed file removal also stays bound to its export and is retried by `close_binary`; closing does not report success while that cleanup remains. `rea doctor --provider kuna` reports the cache directory without creating it.
## Verify with a real Kuna
+132 -29
View File
@@ -53,8 +53,9 @@ reject the summary record. Give `compare_functions` the retained dossier as
`{"kind": "retained-evidence", "evidence_id": "<parent_evidence_id>"}`; give
`match_native_functions` the same reference inside `left_observations` or
`right_observations`, since its `left` and `right` take `list_procedures`
Evidence. Request `detail: "complete"` for `build_call_path`, which accepts
only inline complete dossiers.
Evidence. `build_call_path.functions` also accepts retained references to the
complete dossiers. Request `detail: "complete"` only when the record must travel
inline.
Saved records from the pre-6.6 identity projection that omitted
`relationship_coverage: null` from its digest remain incompatible, even when
@@ -73,7 +74,9 @@ Ghidra inventories, measured load-image observations and function replies use
`raw_result: null` when their useful facts are already preserved in
`normalized_result`, limitations, locations and relationship coverage. Distinct
overview details, file-offset selection roles and database types supplemented
with derived metadata keep their upstream representation. Native load-image
with derived metadata keep their upstream representation. When a Ghidra naming
edit is not applied, `raw_result` keeps the bridge's `failures` and shared
`conflicts` table without a second copy of the result booleans. Native load-image
attestation continues to use the bridge reply locally before Evidence
projection. Read function results from `normalized_result`; older retained
Evidence keeps its original representation and identity.
@@ -158,7 +161,9 @@ for the error code, target details, partial observations, remediation, and the
optional bounded `recovery` next step. `recovery.kind=inspect_input_contract`
names the rejected operation: re-read that operation's advertised input schema
(call `tools/list` if the catalog may be stale), correct the listed fields, and
do not repeat the unchanged call. `recovery.kind=inspect_timeout_state` names
do not repeat the unchanged call. A well-formed request the selected target
cannot answer, such as an address without one exact file offset, omits this
hint; follow its `remediation.action`. `recovery.kind=inspect_timeout_state` names
the operation and the REA server/provider deadline. Inspect partial Evidence
and reported cleanup state, then follow the returned remediation. Change a
request field or REA_* setting only when the returned remediation explicitly
@@ -168,6 +173,9 @@ recommendation to change it. When it names no setting, reduce supported work;
do not invent one. Raising only the MCP
client's deadline cannot extend this budget. These hints never include the
complete catalog or caller arguments.
`recovery.kind=retry_cleanup` appears only on a `close_binary`
`cleanup_incomplete` error whose retained resources the session will retry; see
[retained cleanup](#retained-cleanup-retry).
Failures omit `structuredContent`, since each advertised `outputSchema`
describes successful results. Successful replies retain their schema-validated
structured data and matching text projection. Oversized errors are retained
@@ -202,8 +210,13 @@ unresolved relationship reads. Cancellation remains a tool error.
`trace_feature` validates each reference's containing-procedure resolution using
the same contract as navigation context. Explicit absence remains a successful
negative observation; malformed provider replies return `unreadable_output`
for `resolve_containing_procedure` before successful-result projection.
negative observation. If the provider reports a non-retryable capability refusal
attributed to the owner lookup without incomplete cleanup, the reference remains visible
with `containing_procedure.status: "unavailable"` and a residual unknown. That
unknown does not imply references were truncated.
Malformed provider replies return `unreadable_output` for
`resolve_containing_procedure` before successful-result projection;
cancellation and incomplete cleanup remain errors.
REA prepares complete MCP results within the pinned stdio client's 10 MiB
receive-buffer budget, including both text and structured representations and
@@ -516,7 +529,11 @@ names, the native API record, and high-pcode facets without starting a provider.
and limit default to 0 and 64; pseudocode uses UTF-16 code units and never
splits surrogate pairs, while other pages count rows. Unavailable facts and
provider limitations remain explicit. Source is
an exact same-session retained reference or portable inline Evidence. Views are
an exact same-session retained reference, portable inline Evidence, or an
Evidence record or bundle file. Native function, inventory, and observation
views require the `rea.analysis` predicate; function-matching views require
`rea.native-function-matching`. A mismatched predicate returns an
`invalid_request` identifying `source.predicate_type`. Views are
a summary, a layout mitigations or linkage facet, one section/symbol/module, or
a stable page with a caller-selected positive `limit`. Module pages include
JavaScript assets, bundled modules, and source modules. Select an exact
@@ -623,10 +640,24 @@ counts, and whether the decoded graph is incomplete. Each page repeats
budget (XLD.app: 6.7 MB), so select `detail: "summary"` on the first call.
Delivery detail never reaches the decoder or changes the source Evidence ID.
`trace_application_feature`, `trace_javascript_semantics`,
`compare_application_versions`, `compare_source_to_bundle`, and
`compare_javascript_export_shapes` accept complete inline application Evidence
or an exact reference to a record already retained by the current connection:
Logical Evidence source fields share one input contract across domains:
complete inline Evidence or an exact reference to a record already retained by
the current connection. This applies to individual sources, optional sources,
arrays and nested sources declared by the tool's input schema:
| Workflows | Retained-reference inputs |
| ----------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------- |
| `compare_functions`, `compare_process_captures`, `compare_wasm_interfaces` | `left`, `right` |
| `match_native_functions` | `left`, `right`, `left_observations[]`, `right_observations[]` |
| `trace_application_feature`, `trace_javascript_semantics`, `compare_source_to_bundle` | `application`; feature tracing also accepts `native_observations[]` |
| `compare_application_versions`, `compare_javascript_export_shapes` | `left`, `right`; version comparison also accepts both native-observation arrays |
| `reconcile_javascript_runtime` | `static_layers[].analysis`, `runtime_observations[]` |
| `compare_managed_members`, `verify_managed_native_boundaries`, `import_managed_reconstruction` | Comparison `left`/`right`; verification `managed_boundaries`/`native_observations[]`; reconstruction `static_members` |
| `project_managed_application_graph` | `managed_artifact`, `managed_members`, `managed_native_boundaries` |
| `project_android_application_graph`, `project_apple_application_graph`, `project_harmony_application_graph` | `inventory_evidence[]` |
| `build_call_path`, `find_changed_behavior`, `correlate_static_and_runtime` | `functions[]`, `comparisons[]`, or `static_comparisons[]`/`runtime_comparisons[]`, respectively |
For example:
```json
{
@@ -642,27 +673,46 @@ or an exact reference to a record already retained by the current connection:
```
Use the `evidence_id` returned by `analyze_javascript_application` (or another
compatible application-graph producer). Comparisons accept this form in `left`
and `right`; each side can independently be inline or retained. Native
observation arrays continue to take complete inline Evidence. Results and their
Evidence remain complete inline, and both input forms pass the same semantic,
identity, authority, and provenance checks without running the producer again.
compatible application-graph producer). When the producer returned a summary
or inspection page, use its `normalized_result.parent_evidence_id` to select
the complete source, rather than the view's own `evidence_id`. Each source can
independently be inline or retained, including within one array. Both forms use
the same application workflow and owner validation, preserve exact source
identity, provenance, authority, coverage and partial or unknown facts, and do
not run the producer again. A reference cannot supply facts absent from that
record. Workflows that follow source links also require those linked records
in the ledger. File references are accepted only where the tool explicitly
advertises a file selector; a retained ID does not authorize a file read.
References belong to the current connection's Evidence ledger. Opening another
target or restarting the current target preserves retained records, including
when provider cleanup fails; records remain available for export and retry.
`close_binary` clears them, even when no binary is active. A fresh connection
has its own ledger. A missing reference reports its exact ID and
`close_binary` releases the engine and replay cache while preserving these records,
even when no binary is active or cleanup fails. `reset_investigation` explicitly
clears all records and Unknown revisions after draining earlier work; a healthy
active engine remains open. Its optional absolute `save_path` saves a complete
canonical bundle before clearing. Failed save, cancellation observed before clearing or incomplete cleanup
preserves records, and a published file is reported even if later cleanup fails.
A fresh connection has its own in-memory ledger; import the saved bundle to restore
original Evidence IDs and complete Unknown history. A missing reference reports its exact ID and
`details.reason: "missing"`; the server cannot infer
whether it was never recorded, cleared, or retained by another connection.
Supply complete inline Evidence, repeat its producer, or import an exported
Evidence bundle before referencing that imported record. Export a bundle before
closing if the investigation needs it later.
Evidence bundle before referencing that imported record. Export a bundle or use
`reset_investigation(save_path)` before ending the process if the investigation
needs it later. Retention has no automatic persistence or silent eviction.
`binary_session.investigation` reports record/revision/head counts, estimated
retained representation bytes and the fixed allowance derived from initial V8
heap headroom. A capacity refusal preserves the existing investigation; checkpoint
and reset it, or start with a larger Node heap. These byte estimates are not RSS.
CLI application workflows continue to read portable inline Evidence from files
and run the same analysis workflows; a standalone CLI invocation cannot resolve
another MCP connection's retained records. No additional lookup call, provider
selection, or approval step is required for a same-session follow-up.
CLI workflows that advertise `--evidence-bundle` can resolve the same reference
JSON from one explicitly selected local bundle, including an MCP export. They
also accept portable inline Evidence and use the same application workflows.
A standalone CLI invocation cannot resolve another MCP connection's ledger
without that exported bundle. See [CLI Evidence reuse](cli.md#import-export-and-compare-evidence)
for the supported commands. No additional lookup call, provider selection, or
approval step is required for a same-session follow-up.
CLI `inspect-artifact` continues to return complete portable Evidence. Save that
JSON to a file and use `inspect-analysis-view` with inline source Evidence for
later inventory summaries or pages; a one-shot CLI session cannot retain a
@@ -692,6 +742,19 @@ requirements. Compare the union of group names with `tools/list` when checking
that a client received the full catalog. An operation absent from all groups is
missing from the complete inventory; it is not an implicit unavailable entry.
Availability also reflects the active provider client's dispatched requests. A
refusal or provider failure observed for an operation keeps that operation
unavailable until a request for it dispatched later succeeds. A provider adapter
or process failure also marks that provider's other declared-available
operations unavailable until any request to that provider dispatched later
succeeds; their remediation states that the status was inferred rather than
observed for them. Recovery is not proof that each operation works, and failures
observed on other operations or providers remain. Outcomes are ordered by
dispatch to the provider client rather than reply completion, so a late reply
from an older request does not override a newer outcome. Request deadlines and
cancellations neither mark nor clear availability. A new provider client starts
without these outcomes.
`detail` selects the projection and the result echoes it. The default
`summary` keeps open state, target identity, the binding, current-route
`capabilities`, run, activity and health status, and each provider candidate's
@@ -715,6 +778,13 @@ finish without reading relationships, even when neither traversal mode is
available. Discovery does not block that direct invocation or replace its typed
partial/error outcomes.
`trace_native_ui_action` reports authored-UI and native-seed modes separately.
Apple dispatch metadata requires a Mach-O target; another provider may supply
the symbol-only alternative on other formats. If neither mode is available,
read each mode's missing operations and remediation: a remedy for an authored
app-bundle route does not by itself describe the native-seed route. Availability
does not guarantee that every selector or provider operation will succeed.
With complete detail, the `effects` and `annotations` arrays list the true
members of their fixed vocabularies. A known member omitted from either array is false. This applies
only to these closed flag sets; it does not make an omitted tool, provider, mode,
@@ -834,6 +904,20 @@ inspection fails or is cancelled before a provider client exists. If no client
remains, no provider run or health observation is available; follow the cleanup
error's reported resources.
### Retained cleanup retry
A `close_binary` that retires the target clears engine replay state while
retaining investigation Evidence and Unknown history, including when cleanup fails. When the session keeps every reported owner for the next
`close_binary` on the same connection, its `cleanup_incomplete` error carries
`recovery: {"kind": "retry_cleanup", "operation": "close_binary"}`.
Keep the connection open, let the reported condition settle or resolve it, call
`close_binary` without `snapshot_path`, and check the result. The retry does not
rerun the failed analysis and can report `cleanup_incomplete` again.
`retryable` still describes the failed operation and stays `false`. A failed
`snapshot_path` save returns before retirement, keeps the target open and has
no `retry_cleanup` hint. `open_binary`, analysis tools and one-shot CLI commands
do not report this hint either, even when their cleanup remains pending.
## Initial target at startup
MCP normally starts without a target; use `open_binary` to select one during the
@@ -1094,6 +1178,15 @@ mapping. A successful start-address mapping does not establish that an entire
range is contiguous in the original file, or that its bytes equal analysis
memory after loader fixups. Uninitialized, missing, and ambiguous mappings fail
explicitly; preserve the reported failure rather than guessing an offset.
For Ghidra, such a well-formed address returns `invalid_request` whose message
states the bridge's reason and whose remediation points to
`inspect_native_load_image`. It carries no `inspect_input_contract` recovery.
`details.issues[0].observed` keeps the caller's exact `address` and a
`file_mapping` outcome: `unmapped` (outside every memory block),
`uninitialized`, `absent` (no authoritative target-file source), `ambiguous`,
or `unrepresentable` (beyond the exact JSON integer range). Initialization, not
the loaded flag, decides whether source bytes can exist, so an initialized
file-backed overlay still maps.
For constructor-populated pointer or resource slots, a static zero is not proof
of a later null value. Inspect the slot's `xrefs` and the writer's assembly or
@@ -1216,11 +1309,11 @@ If the client timed out or cancelled during startup, the pending query may have
been interrupted. Reissuing `open_binary` for the same active target can reuse
the current client, so it is not a fresh-start recovery. On the same connection:
1. Keep any useful inline Evidence, or export a bundle before closing if retained
records are needed. `binary_session` can show the selected binding and its
1. Keep useful inline Evidence or use the retained investigation references;
export a bundle before ending the process to resume later. `binary_session` can show the selected binding and its
recorded state; target selection alone does not prove engine readiness.
2. Call `close_binary` and check its result. It drains owned work, closes the
provider, and clears retained session records. A `cleanup_incomplete` result
provider and engine replay state while preserving investigation Evidence and Unknown history. A `cleanup_incomplete` result
must be addressed according to its reported owned resources before retrying.
3. Call `open_binary` with the same caller-selected path and
`provider_id: "ghidra"`, then retry the first query with an appropriate client
@@ -1271,7 +1364,9 @@ cursor outside a procedure is represented as
procedure resolution returns typed unreadable output; provider errors never
become absence from their message text. `inspect_address_context` requires an explicit address and
returns name, procedure, comment, inline-comment, and bookmark facets;
unsupported facets are local `unavailable` outcomes. Use `current_document`,
unsupported facets are local `unavailable` outcomes. A facet whose input is
rejected includes its canonical `error` projection, including any structured
issues, alongside the other facet outcomes. Use `current_document`,
`current_address`, and `current_procedure` for direct single-field lookups; use
the aggregate tools when you need related context together.
@@ -1317,12 +1412,20 @@ bytes and returns the `integrity_contradictions` records for the extracted
files. An active ASAR unpacked entry whose companion file is absent cannot be
materialized; extraction fails as `unavailable` with that logical path.
A container can also declare a checksum over its own bytes; today that is the
DFU suffix CRC of a DfuSe file. A mismatch is a contradiction of the root
occurrence (logical path `.`), recorded as a `dfu-crc32` pair. Under
`record-and-continue` the root becomes `mismatched` while its elements stay
listed and extractable as observed-untrusted bytes, and `extract_artifact`
returns the root record with the extracted files' records.
If native DMG child traversal is unavailable after carrier identification,
the typed failure retains a revalidated root-only inventory under
`details.partial_observation.inventory`, with unknown child coverage and the
producer's diagnostics/provenance. It does not borrow or detach an existing
attachment. Native checksum verification failures still refuse traversal;
`record-and-continue` recovery applies to structured entry contradictions.
`record-and-continue` recovery applies to structured entry and container
contradictions.
Occurrences use normalized logical paths for identity and lookup. Optional
`source_path` retains the original path within the owning reader when Unicode
+14 -2
View File
@@ -24,13 +24,25 @@ CLI equivalent is `rea guidance --query "exact JavaScript module"`, followed
by `rea guidance --action read --locator LOCATOR`. Search is limited to the
packaged managed skill files, works without providers or a target, and returns
deterministically paged matches. Every non-generic query word must occur in the
section; generic-only queries return no matches. Results are documentation; locators
section or its subsections; generic-only queries return no matches. Results are documentation; locators
are bound to the installed bundle and report stale or unavailable guidance without
returning cached section content. A no-match result says nothing about analysis
capability availability.
Each match reports its `match_scope`. A `section` match has every term in its
own text, meaning its heading and body before its first subsection. A
`subsections` match has the terms only spread across subsections, with no
single subsection holding them all. Its `own_text_terms` lists the terms in its
own text, and `subsections` lists up to 8 subsection locators, chosen greedily,
whose own text holds the rest; `omitted_subsections` counts any more that were
needed. Section matches rank first. A parent is not listed when one subsection
holds every term, because that subsection is already a match, unless the
parent's own text also holds them all. `content_length`
is the size of the section that read returns, so callers can decide before
reading a broad parent.
Search returns at most 20 matches per page with 500-code-unit context excerpts.
Read retrieves the complete selected section. Admission permits 1 MiB per file,
Read retrieves the complete selected section, including its subsections. Admission permits 1 MiB per file,
4 MiB per bundle, 4,096 sections, and 25,165,824 expanded UTF-16 code units before
section text is materialized. Unreadable or oversized bundles return
`bundle_unavailable`, with no partial index or inferred absence. Cleanup failures
+72 -3
View File
@@ -68,7 +68,10 @@ function renamed to `dead` remains selectable by name. Use the names returned
by the inventory, including their namespaces and any platform symbol prefix.
Ghidra also accepts secondary symbols at a function entry, such as an imported
`_main` label. A label inside a function is not a procedure-name selector; use
its address when selecting the containing function.
its address when selecting the containing function. `search_procedures` matches
the same names, in leaf or qualified spelling: the function appears once with
its primary name as `value`, and `matched_aliases` lists the qualified names of
the secondary symbols that matched.
Overloads can share a fully qualified name; ambiguity errors return every
matching entry address so the caller can select the intended function directly.
@@ -185,6 +188,13 @@ observed entry label. An interior address therefore remains visible without
being mistaken for the function entry. Identity failures are reported as
`procedure.status: "unknown"`; label failures leave a null name and `name_error`.
Neither failure discards usable pseudocode, and names are not parsed from it.
A provider answer without pseudocode is an error item that keeps that
observation's own `limitations`, so a reason the provider reported for that
entry, such as a Kuna export failure or a missing body, stays with the item.
Null pseudocode is `capability_unavailable`; empty or non-text pseudocode is
`unreadable_output`. Provider failures keep their typed error. Successful items
do not repeat limitations; the batch Evidence aggregates every observation's
limitations by provider and operation.
Resolved entries also appear in the composed Evidence locations. New batch
Evidence commits the revised workflow profile; retained historical Evidence
remains valid. This tool
@@ -422,6 +432,21 @@ establish canonical function identity. A successful seed without a native def-us
model still returns its procedure and an explicit coverage unknown. A new trace
can retry failed selectors.
Paging and traversal completeness are separate facts. `page_truncated` means the
page omits retained nodes, and the edges they source, before `offset` or after
`next_offset`. Every page repeats `traversal`: `complete`, or `truncated` with
`reasons` naming each observed cause: the `max_depth`, `max_functions`,
`max_call_sites`, `max_nodes` or `max_edges` budget, the fixed
`node_payload_bytes` allowance, `provider_pcode_truncated`, or
`ambiguous_call_candidates`. Each cause also has a located unknown. A budget is
reported only after it rejects work, so an exactly filled budget is not
truncation; a depth or candidate limit is reported only when no other path
analyzes or queues that destination. `total_nodes` and `total_edges` count
retained rows. `truncated` remains true for either kind of omission. Failed
analyses and unresolved calls stay in `unknowns` and do not truncate the
traversal. Each page reruns analysis; pages are not pinned to one snapshot
across session edits.
The raw Java `is_dead` flag can remain set after Ghidra decodes a live block;
`block_membership` reports actual syntax-tree membership separately.
@@ -1117,11 +1142,55 @@ unpaired Unicode surrogate; a rejection identifies the field and UTF-16 index
and leaves every annotation unchanged. CRLF, supplementary Unicode characters,
and combining characters are preserved. Windows P0 remains read-only. Ghidra has no
GUI authority, and REA never falls back automatically to Hopper.
For a before-and-after `compare_functions` in MCP, call `analyze_function`
before and after the edit while the session remains open; the annotation Evidence
itself is not an `analyze_function` input. If comparing after close, export the
needed Evidence before closing and import it into the later session. A one-shot
CLI annotation returns its edited dossier before its temporary session is
discarded; a later CLI invocation with the same fresh-import settings starts from
the original executable. Comparison preserves correspondence across renames
when authenticated artifact identity, architecture and provider-owned mapping
and slice commitments establish the same canonical entry. The result reports
`matched`, `method: "same-entry"` and `entry_address`; name and exact-text changes
remain separate dimensions. Equal numeric addresses without applicable coordinate
commitments do not establish same-entry correspondence. Inspect the retained
source Evidence and coverage when correspondence remains ambiguous.
To aggregate already-retained comparison results, pass their exact IDs to
`find_changed_behavior`, for example:
```javascript
await call("find_changed_behavior", {
comparisons: comparisons.map(({ evidence_id }) => ({
kind: "retained-evidence",
evidence_id,
})),
});
```
These IDs select comparison Evidence, not function dossiers or summary views.
Complete inline comparisons remain accepted and can be mixed with references.
The current server must retain the comparisons and their original source closure;
import the complete source bundle after a session handoff. Missing or damaged
records are explicit failures. A missing required retained record reports its exact
`evidence_id` with `details.reference_scope: "retained-closure"`. Import the
originating bundle with its linked sources, or re-create those observations in the
current investigation, before retrying; resending the inline comparison alone
does not restore its closure. Aggregating static differences does not establish
observed runtime change.
`set_address_name` and `set_addresses_names` give Ghidra sessions the same
compact naming results as Hopper: a local function entry renames that function,
and any other mapped address, such as a global, receives a new or renamed primary
label. A batch applies each address separately and reports one success boolean
per address. Use them to name many globals and helpers cheaply; use
label. A batch decides every address from the names held before it: addresses
that request the same name in one namespace, or a name another selected address
holds, are rejected without applying them, so JSON member order never selects a
winner. Every other address applies separately, and each address reports one
success boolean. `raw_result.failures` gives each failed address's canonical
address and reason. A contested name appears once in `raw_result.conflicts`,
with its namespace, requesting addresses and holding addresses, and conflict
failures refer to it by index. To move a label between
addresses, rename its holder first and assign the name in a later call. Use
them to name many globals and helpers cheaply; use
`annotate_native_function` when you also want comments and a refreshed dossier.
Ghidra name edits preserve the existing namespace. Supply either a leaf name
such as `renamed` or a fully qualified name in that namespace, such as
+27
View File
@@ -652,6 +652,33 @@ that directory: it outlives each run, so the first compilation into a new
directory pays a cold-cache build and later runs reuse it. An absent or empty
value keeps the shared default.
With the default `xcrun` compiler and module cache, REA also keeps one verified
copy of the compiled helper in `~/Library/Caches/rea/process-helper/`, so later
REA processes skip both the compile and macOS's first-launch assessment of a
new executable. Each entry is keyed by the helper source, the `swiftc` and SDK
that `xcrun --find swiftc` and `xcrun --show-sdk-path` resolve (their real
paths, sizes and modification times, so `DEVELOPER_DIR`, `TOOLCHAINS`,
`xcode-select` and in-place toolchain updates all select a new entry), the CPU
architecture and the Darwin release. Before running an entry, REA requires the
cache directory, the entry and its files to be owned by the current user and
not writable by group or others, every real parent directory to be owned by
the current user or root and not writable by others (sticky root-owned
directories excepted), no ACL on any of them to allow another principal a
write-class right (the default `deny delete` entries are fine), the helper and
manifest to be regular files rather than
links or FIFOs, and the helper's SHA-256 to match the manifest. Anything else is
never executed and never removed, because another REA process may still be
verifying or running that path; REA then compiles a per-process helper as
before. A published entry is never replaced. If the cache is unavailable, for
example under a read-only home directory, the ownership operation proceeds
with the per-process compile. These checks reject corruption, loosened
permissions and other users, but they do not defend against another process
running as the same user, which could already modify REA itself. Setting
`REA_DARWIN_SWIFTC_PATH` or `REA_DARWIN_MODULE_CACHE_PATH` keeps the per-process
compile and writes nothing to this cache. REA does not prune old entries; each
is about 240 KiB. Delete the directory while no REA process is running, or run
`rea uninstall --purge-data`, to remove them.
`xcrun` also maintains a separate tool-lookup cache. Its `--no-cache` option
refreshes entries and does not guarantee a write-free lookup. To avoid that
lookup while compiling REA's helper, set `REA_DARWIN_SWIFTC_PATH` to the absolute
+161 -3
View File
@@ -365,6 +365,41 @@ Do not report the warm result as a cold-suite improvement, and do not enable
the cache in coverage or benchmark CI without first showing that its
instrumentation remains equivalent.
## CodeQL production scope
[The production configuration](../.github/codeql/production.yml) excludes
`tests/`, `scripts/fixtures/`, `*.test.*`, `*.fixture.*`, `test_*.py`, and `*_test.py` inputs
during extraction. It retains the default queries and the existing `local`
threat model. Production source, bridges, runtime scripts, installation and
release controllers, and Actions remain in scope for the configured languages.
Verification scripts remain included unless they match an explicit test path;
excluding the entire `scripts/` directory would hide shipped entrypoints.
The `paths` list retains repository-root traversal and explicitly includes
`docs/.vitepress/`, which JavaScript extraction otherwise skips as a hidden
directory. Keep directory entries here: the shared Python extractor does not
accept individual files as traversal roots. When changing this list, compare the
complete production file inventory as well as alert counts.
These exclusions do not change which behavioral tests CI executes. CodeQL's
JavaScript/TypeScript, Python, and Actions lanes do not add shell, Java, or Swift
analysis; their existing checks remain responsible for those languages.
[The production workflow](../.github/workflows/codeql.yml) is inactive until a
repository administrator disables CodeQL **default setup** and sets the Actions
repository variable `REA_CODEQL_ADVANCED` to `true`. Disable default setup first:
it rejects advanced CodeQL uploads. Then enable the variable and manually run
**CodeQL production** on `main`; verify all three analyses complete before
relying on the new scope. While the variable is unset, the existing default
setup continues scanning. Adding the configuration file alone does not change
default setup in this user-owned repository.
Each analysis uses two threads and a 4096 MiB CodeQL memory budget; language
jobs run one at a time. Complete SARIF reports are retained for seven days.
For a scope comparison, use the same source SHA, CodeQL bundle, default query
suite, and `local` threat model on both runs. Change only extraction paths, inspect
the extracted-file inventory and diagnostics, and retain both complete reports.
Alert counts do not establish a false-positive rate or a speedup. Report elapsed
time, CPU time, and process-family memory with their measurement limits.
## Choose verification for the change
`test:local` is fast source-level feedback. For the strongest practical
@@ -382,6 +417,83 @@ matching real-provider lane.
| Client configuration or schema conversion | [Native client lane](client-verification.md) | Distinguish loopback models from live services |
| Agent investigation quality | [Agent evaluation](#agent-evaluation-and-conformance-records) | Review transcript and scenario-specific evidence |
### Provider-backed workflow verification
For a new or changed provider-backed tool, select tests from the analyst's
question and the facts the contract claims, not from an upstream command list.
Use the existing workflow as a baseline and identify what the proposed tool
adds. Choose the relevant cases below; do not require every provider or a
Cartesian matrix when it would not change the claim.
When auditing existing tools, compare selected analyst workflows with the cases
their existing tests and real-provider lanes actually exercise. Run relevant
public workflows and focused checks; report untested scenarios rather than
inferring that a green test proves them or that no prior testing occurred. A fix
should add a regression for the demonstrated failure at its owning boundary and
affected consumer.
- Use a controlled target with independently known facts and a meaningful
counterexample. Compare native provider output with the target or a
format-aware oracle; a provider label, a mock, or another REA projection is
not independent proof of the underlying fact.
- Exercise the affected public CLI/MCP workflow, including handoffs between
calls. Assert the requested result, caller-selected inputs, source identity,
evidence and coverage. Check that unavailable, partial and empty outcomes
retain their different meanings across the producing and consuming boundaries.
- Where relevant, continue after a mixed success, failed request, cancellation
or close/reopen. Verify the next supported recovery and preservation of prior
results and resource ownership. Use the applicable lifecycle regressions below.
- For lookup failures, distinguish malformed input from a valid missing or
ambiguous selector. Check the returned candidate and input identity, follow
an actionable recovery where possible, and verify that an inventory miss is
not presented as artifact-wide absence. Test schema-repair advice separately.
- Record the provider version, host, fixture, actual entrypoint, controlled
faults and skipped or unavailable coverage. A simulated provider tests the
integration boundary; a scripted sequence tests that sequence, not an agent's
ability to discover it. Run the matching real-provider and agent lanes when
their respective claims require them.
For prerequisite recovery, include a case where only the wrong alternative is
configured: Unblob cannot enable Binwalk inspection, and Binwalk cannot enable
Unblob extraction. Check the final readiness or capability output retains the
producer's repair instructions, including which launch environment must change.
When feasible, follow the advice and verify that the same operation now works.
Report unexecuted recovery steps explicitly.
### Checking installers and publisher downloads
When changing setup or investigating an install failure, check the stages affected
by the change:
1. **Does REA support installation on this host?** Check the distribution and CPU
architecture, including an unsupported case. Recognizing RPM or DEB packages
does not mean REA can install on every related distribution. Check prompts in
a terminal and with redirected output; image builds cannot answer interactive
prompts. `doctor` checks health; `setup` installs.
2. **Does the downloaded package match the publisher's metadata?** Record the
release version, package format, URL and check date. Verify the signature and
compare the launcher hash inside that package with the feed and installed
launcher. Check each format you claim works: a passing DEB check says nothing
about the RPM. Old fixtures cannot catch a newly published package mismatch.
Test both an invalid signature and a valid signature with a mismatched launcher
hash; setup must reject either before running the package manager.
3. **Can download failures be diagnosed?** Use REA's request headers and timeout.
Check whether the setup result identifies the failed URL, HTTP status or
timeout, and next action; report missing details. Use controlled failures for
repeatable tests. A fast download on one network does not rule out a timeout
elsewhere, and curl with different headers may get a different response.
4. **Does the installed provider run?** Check package-manager prerequisites and
run doctor for the selected provider. For Linux Hopper, check Python and REA's
private Xvfb display separately from package installation. Where relevant,
test an unusable `/tmp/.X11-unix` and a blocked user/mount namespace fallback.
Starting an extra X server does not test the display REA creates and cleans up.
For a full Linux Hopper setup check, run `npm run verify:setup:linux` on a clean
root Linux runner. Set `REA_VERIFY_SERVER_COMMAND` to the installed REA command.
This checks setup, doctor, CLI/MCP analysis and repeated setup on that runner.
Report which host, package format and steps you tested. Inspecting a Linux package
on macOS verifies package contents, not a working Linux installation.
## Test design and pruning
Prefer real public CLI/MCP workflows, then production-boundary integration,
@@ -538,6 +650,27 @@ and toolchain conditions when comparing timings.
## Lifecycle regressions
Investigation lifetime changes exercise public MCP target A/B switches, target-free
close and close cleanup failure/retry, with exact retained IDs and complete Unknown
revision histories. Verify engine replay/cache removal separately from retained
facts. An explicit reset must preserve a healthy engine and existing metadata
invalidation, drain an admitted signal-ignoring late result, block later work,
and prevent stale replay/source bindings from restoring old facts. Cover checkpoint
overwrite refusal, publication followed by cancellation or staging cleanup failure,
cleanup retry, then fresh-process bundle import with identical original references
and Unknown history. Capacity admission tests preserve the complete pre-state on
failed record/import/Unknown transactions, count duplicates once, account for
metadata size deltas and release budget on reset. Controlled providers and cleanup
seams establish ordering and failure semantics; real application/provider runs are
reported separately.
Hold successful and error replies at the MCP transport boundary while reset
queues behind their original admission. Verify checkpoint contents and publication
order, including SDK output validation failure, handler failure and cancellation
before or during send. Handler cleanup must settle even when the SDK suppresses
the reply. An outgoing server request with the same numeric ID as an incoming
tool request must not acknowledge that tool's response.
For changes to resource acquisition, readiness, cancellation, or cleanup, select
the affected owners and applicable scenarios below. Include temporary snapshots
and one-shot commands. Use public CLI/MCP workflows where practical and a
@@ -1029,7 +1162,7 @@ prove behavior on the verified Ghidra 12.1.4 and JDK 21 build.
| `npm run verify:ghidra` | Linux x64/arm64 ELF or macOS x64/arm64 Mach-O | Host C compiler, Ghidra 12.1.4, and full JDK 21 |
| `npm run verify:ghidra:seeds` | POSIX; raw x86 fixture; seed ordering, BSS labels, analyzer readback, not-applicable and unknown analyzer names | Ghidra 12.1.4 and full JDK 21 |
| `npm run verify:ghidra:swift` | macOS x64/arm64 Mach-O | Host Swift compiler, Ghidra 12.1.4, and full JDK 21 |
| `npm run verify:ghidra:riscv` | Linux/macOS x64/arm64 host; source-owned RV32/RV64 ELF through CLI and MCP | RISC-V Clang, LLD, LLVM readelf, Ghidra 12.1.4, and full JDK 21 |
| `npm run verify:ghidra:riscv` | Linux/macOS x64/arm64 host; source-owned RV32/RV64 double-float and RV32 ilp32 ELF through CLI and MCP | RISC-V Clang, LLD, LLVM readelf and objdump, Ghidra 12.1.4, and full JDK 21 |
| `npm run verify:ghidra:switch` | Linux x64 ELF; GCC/Clang optimized and stripped switch fixtures | GCC, Clang, GNU nm/objdump/strip, Ghidra 12.1.4, and full JDK 21 |
| `npm run verify:ghidra:aarch64-jump-table` | Any supported Ghidra host; AArch64 ELF; signed/unsigned byte/halfword tables; host ARM64 Mach-O | Clang with AArch64 target support, Ghidra 12.1.4, and full JDK 21 |
| `npm run verify:ghidra:cross-format` | Any supported Ghidra host; also analyzes AArch64 ELF, x86-64 PE, and x86-64 Mach-O | `clang`, LLD, and `lld-link` in addition to host-lane prerequisites |
@@ -1438,6 +1571,16 @@ cached byte spans and a Ghidra comparison. See the [Linux report](https://github
for measured scope and limitations. Windows, PE and universal Mach-O remain
unverified by this lane.
Kuna 1.986 passed the fixture and a separate small, source-owned Mach-O with
symbols and stripped on macOS arm64, including the Ghidra comparison and owned
cleanup. Linux and other hosts have not been verified with 1.986; this check is
not a repeat of the larger 1.833 matrix.
Kuna 1.988 passed the real-provider lane on macOS arm64 with the generated
Mach-O fixture and caller-built symbol-bearing and stripped x86-64 ELF fixtures.
Linux and other hosts have not been verified with 1.988; this is not a repeat
of the larger 1.833 matrix.
## DOS Ghidra analysis
`npm run verify:ghidra:dos` requires the supported Ghidra and JDK installation
@@ -1508,8 +1651,9 @@ real-provider coverage.
For source-identity boundaries that a normal single-file import cannot create,
`tests/conformance/ghidra/ReaFileOffsetSourceProbe.java` exercises the production
bridge against real Ghidra `FileBytes`. It rejects auxiliary and partial sources,
checks the original target's round trip, and retains ambiguity when duplicate
target bytes have distinct loaded mappings. Injected decompiler failures and
keeps distinct outcome codes for uninitialized and unmapped addresses, checks
the original target's round trip, and retains ambiguity when duplicate target
bytes have distinct loaded mappings. Injected decompiler failures and
cancellation verify that completed source/function resolution facts and backend
failure codes survive; these injections do not establish natural engine failures.
On Windows, after creating the
@@ -2001,6 +2145,20 @@ tool before building and warns when esptool is not the pinned v5.6.0.
`REA_XTENSA_ESP32_GCC` adds an ESP32 Xtensa build. See
[ESP images](esp-image.md).
`npm run verify:uf2` needs only a Python that runs esptool, named by
`REA_ESPTOOL_PYTHON` (default `python3`). It writes inputs at known addresses,
builds UF2 files with esptool's `merge-bin --format uf2` (v4 `merge_bin`) with
and without MD5 and with 256-byte payloads, and concatenates two chips'
outputs. Inspection and extraction through the built CLI must return each input
at its address under the familyID esptool itself reports. It is not part of CI.
See [UF2 and DfuSe images](uf2-dfuse.md).
`npm run verify:esp-flash` needs the same tools and an ESP-IDF checkout
(`REA_ESP_IDF_PATH`) for `gen_esp32part.py` and `otatool.py`. It builds
partition tables, assembles dumps with `esptool merge-bin`, writes OTA records
with otatool through `tests/conformance/esp/flash_file_tool.py`, and compares
REA's flash facts with `image-info`, the decoded table and
`otatool read_otadata`. See [ESP flash images](esp-flash-image.md).
### JavaScript source recovery
Build once with `npm run build:cached`, then run `npm run verify:javascript:recovery`.
+180
View File
@@ -48,6 +48,100 @@ fixture servers. Custom orchestration languages, replay engines, or separate
prepare/execute plans need an observed requirement those primitives cannot
satisfy. A plan without execution does not establish runtime behavior.
## Exercise the analyst workflow
Before defining a new provider-backed tool, state the analyst's question, what
they already have, the intended result and authority, and what must remain
unknown. Try the nearest existing REA workflow and the relevant native provider
or ordinary script path when available. Identify the specific gap and what a
new or extended contract would add; upstream capability alone does not require
a new tool or equal support from every provider.
Exercise the task through REA as a whole and compare relevant provider behavior
with an independent artifact or format observation where possible. Use the
caller's selected target and authority. Record the provider version, target and
observed coverage, and distinguish an unsupported operation from an empty
result or an untested path.
Include the boundary cases that could change the answer: provider labels versus
source facts, addresses versus file locations, partial results, and a useful
recovery after failure or session close when those apply to the task. Check that
composed results keep each observation's identity, limitations and source
association. Existing real-provider lanes and focused tests establish their
selected scenarios; they do not by themselves establish a different workflow or
provider equivalence. If a provider cannot be exercised, report that limitation
and continue with the available evidence. Measure agent tool selection only with
an actual agent evaluation; a scripted tool sequence does not establish it. Use
[testing guidance](testing.md#provider-backed-workflow-verification) to select
the cases that establish the proposed contract.
## Audit existing tools
Scope an audit by analyst tasks and the caller's target, providers and authority.
Record the source revision and, when used, the running server, catalog and
provider versions; a connected server can differ from the current checkout.
Inspect the live CLI help or MCP descriptions and schemas alongside the owning
contracts and documentation. Choose representative tasks and their nearest REA
and native alternatives rather than counting upstream commands.
Exercise the selected public workflows, including the steps before and after a
tool call. Check task selection, input choices, returned facts and their source,
partial or unsupported outcomes, and applicable session and recovery behavior.
Compare native provider output and an independent artifact observation where
available. Inspect existing tests and real-provider lanes for the scenario they
actually cover; a passing test does not establish a different task or host.
Use [workflow verification](testing.md#provider-backed-workflow-verification)
for counterexamples and lifecycle controls relevant to the audit.
Report working routes as well as findings. Separate a reproduced contract bug,
an intentional limitation, a design opportunity and an unverified hypothesis;
include the smallest useful reproduction, impact and coverage limits. Do not
claim autonomous-agent selection from a scripted trace. Check related reports
before proposing another issue, and avoid treating lack of API parity as a bug.
## Design recovery around the failure
Distinguish a request that violates its input contract from a valid selection
that the current target, provider inventory, or project cannot resolve. The
same `invalid_request` code can represent either. Suggest schema inspection
only when the input contract is the problem; for a valid lookup, preserve the
caller's exact selector, the observed reason and any candidates, and give a
next step that addresses that outcome. An inventory miss does not prove that
the artifact lacks the requested item.
For prerequisite failures, identify what the requested operation actually needs.
A missing executable, a wrong path, an incompatible host or version, denied
permission, and a download timeout need different repairs. Preserve the observed
failure instead of calling all of them "not installed".
Derive the required engine from the operation's execution contract. For example,
`inspect_firmware_regions` needs Binwalk; `extract_firmware` needs Unblob.
Configuring the other tool cannot enable the requested operation. An installed
JDK outside the selected PATH may need `JAVA_HOME`, not another installation.
Say where a setting must change: a shell export does not change an already
running MCP server's environment.
Keep repair instructions with the contract that owns the failure, and preserve
them through provider readiness, capability discovery, doctor and CLI/MCP output.
An error description is not a repair instruction. Do not make each consumer
guess the fix from diagnostic text or invent another list of prerequisites.
Keep recovery within the selected target, provider, project and authority. If
another context could help, describe it as an explicit choice and state any
relevant loss of state; do not switch or select a candidate automatically.
Preserve uncertainty, partial observations, and session or cleanup state. A
suggested retry must be safe and explain any required change or wait.
Verify recovery through the public workflow: reproduce the failure, follow
the suggested next step when feasible, and check the result and session state.
Include a malformed-input control so schema guidance remains available when it
is actually needed. Compare CLI and MCP when both expose the operation.
For each confirmed bug, check sibling adapters and other consumers of the same
facts. Reproduce suspected matches and retain a working comparison where one
exists. Group reports and fixes around the shared cause; similar wording alone
does not establish another instance.
## Define the contract
- Use strict inputs with explicit target identity and required fields. Preserve
@@ -83,6 +177,51 @@ satisfy. A plan without execution does not establish runtime behavior.
ceiling and its recovery as described in
[admission limits and recovery](design/limits-and-recovery.md).
### Compose declared Evidence inputs
Declare each logical Evidence source with the shared `evidenceInputSchema`:
complete inline Evidence or an exact
`{kind: "retained-evidence", evidence_id}` reference. Use that declaration in
single, optional, array and nested source fields across domains. Keep complete
record schemas for stored Evidence, bundle records and pure semantic inputs;
request alternatives do not change those representations. Ordinary IDs in
parameters or result data are not implicitly source selectors.
Derive borrowing request validation and source resolution from the canonical
request schema with `getEvidenceInputContract` and `resolveEvidenceRequest`.
The shared application entrypoint selects references for CLI and MCP before the
format owner admits, authenticates and interprets their records. Preserve
admission before copying large source payloads. Do not maintain a second list
of source paths or resolve references only in an adapter. A resolved reference
has the same identity, caller-selected observations, provenance, coverage and
partial or unknown facts as its complete inline record; it never reruns a
producer or upgrades incomplete evidence.
Keep Evidence-bearing request unions outside skipping `default` wrappers;
use `prefault` when a default must pass through branch selection. Catalog
conformance rejects unsupported source containers rather than silently losing
their references.
Return authenticated source records alongside derived workflow Evidence when
the adapter must retain them. Keep links to those exact sources and require the
source closure when the workflow follows their links. Missing references and
incompatible workflow roles are explicit failures; do not infer a substitute
from ledger order, display names or the active target. A summary or inspection
page has its own identity: consumers needing complete source facts use its
`normalized_result.parent_evidence_id`, unless their contract explicitly accepts
that view.
New provider-owned source observations require a retention owner. Existing
caller-owned inputs can also compose on a fixed port without a recorder;
when a recorder is available, preserve its acceptance or rejection and retain
completed sources through cancellation.
MCP supplies the current ledger lookup. CLI commands that advertise
`--evidence-bundle` use one explicitly selected, authenticated bundle for the
command's lifetime. Keep cleanup with that owner. A retained reference does not
grant filesystem access: file selectors remain separate, explicit contracts
where supported. See [retained Evidence inputs](mcp-contracts.md#retained-application-evidence-inputs)
and [CLI reuse](cli.md#import-export-and-compare-evidence).
### Interpret boundary facts once
The format or protocol reader owns interpretation. Pass its parsed facts and
@@ -132,6 +271,47 @@ artifact can execute successfully.
## Own the lifecycle
The investigation owns authenticated Evidence and complete Unknown revision
history independently of an engine. `close_binary` drains earlier admitted work,
releases engine resources and clears replay/invalidation state; investigation
references survive, including target-free closes and cleanup failures. A target
switch does not make historical observations current for its new engine.
`reset_investigation` is the explicit destructive boundary. It drains earlier
analysis, imports, exports and Unknown operations, then clears investigation and
replay/source bindings while retaining a healthy active engine. Metadata mutation
invalidation remains in force until that engine is recreated. Later admitted work
waits for the transition; earlier late results cannot repopulate cleared history.
Runtime disposal seals new acquisitions and releases records after successful
owned engine/file cleanup; failed cleanup keeps the owner available for retry.
Records are retained in memory under a fixed per-owner representation allowance
derived from one quarter of initial available V8 heap. Admission accounts for
Evidence, Unknown history and indexes before atomic commit; metadata replacement
uses its size delta and duplicates do not accumulate. There is no silent eviction.
`binary_session.investigation` reports committed counts, estimated representation
bytes and the allowance; these estimates are not process RSS measurements. Live
authentication headroom is checked separately. On refusal, export a checkpoint,
reset, and import only the required investigation, or start a process with a
larger heap. Retention does not promise automatic persistence.
An MCP request retains its original session admission until its handler cleanup
and response send settle. A reset queued behind that request therefore cannot
invalidate Evidence before its reply is sent. Nested work borrows the same
admission. Cancellation before sending releases a suppressed reply after handler
cleanup; cancellation after sending starts still waits for transport settlement.
Lifecycle tools drain earlier admissions without acquiring one themselves.
`reset_investigation(save_path)` saves the complete cross-domain canonical bundle
once before clearing. It awaits file sync, atomic publication and staging cleanup;
failed save, incomplete cleanup or cancellation observed before reset commit
preserves all records. Publication
can precede cancellation or cleanup failure; the error retains its committed file
receipt. Import that bundle to restore original Evidence IDs and all Unknown
revisions in a new process. Engine replay snapshots remain a separate target and
profile-bound mechanism. This checkpoint contract does not promise recovery from
an uncheckpointed process exit or parent-directory loss after a host crash.
`SafeOutputTree` publishes only after file readback, nested-directory fsync
from children to parents, and root-parent fsync on hosts that support directory
syncing. Windows skips directory fsync. Failed writes retain a separate pending
+224
View File
@@ -0,0 +1,224 @@
# UF2 and DfuSe images
REA inventories UF2 (Microsoft's USB Flashing Format) and DfuSe (ST's DFU
update container, `.dfu`) files through `inspect-artifact` /
`inspect_artifact`, and materializes their decoded bytes through
`extract-artifact` / `extract_artifact`. UF2 is the drag-and-drop format of the
RP2040 and RP2350 bootloaders, of TinyUF2 on ESP32-S2 and ESP32-S3, and of
`esptool merge-bin --format uf2`. Decoding is static and needs no external tool.
```sh
rea inspect-artifact ./firmware.uf2 --json
rea inspect-artifact ./firmware.dfu --detail summary --evidence_output evidence.json --json
rea extract-artifact ./firmware.uf2 --json
```
For MCP, open the file with `open_binary`, then call `inspect_artifact`. Summary
delivery retains the complete Evidence; use `parent_evidence_id` with
`inspect_analysis_view` to read selected regions and fields. `extract_artifact`
returns a fresh `output_root` containing one raw file per region.
## Recognition
A file is UF2 only when its first 512 bytes are a complete block with all three
magic numbers: `0x0A324655` and `0x9E5D5157` at the start and `0x0AB16F30` at
the end. A file is DfuSe only when it begins with `DfuSe` and format version 1
and ends with a 16-byte DFU 1.1a suffix: `bcdDFU` `0x011A`, the `UFD` signature
and `bLength` 16. The suffix CRC is not part of recognition. The file name does
not matter, and a file with only one of the DfuSe markers stays unrecognized.
The container formats are `uf2` and `dfuse`.
## UF2 regions and metadata
Each entry is one address-contiguous region of one family's main-flash
payloads, named `family-0x<familyID>/0x<start>-0x<end>.bin` with an inclusive
end address, for example `family-0xe48bff56/0x10000000-0x1000ffff.bin`. Blocks
without the familyID flag (`0x2000`) go under `no-family/`. Families are
separate address spaces: a multi-family file, such as a Pico SDK image for both
RP2040 and RP2350, has a region per family even at the same addresses. Gaps are
never filled.
Region `format_metadata` (`uf2-region`) records `family_id` (or `null`),
`family_name`, `start_address`, `end_address`, `length`, `block_count`, and
`first_file_block`/`last_file_block`: the zero-based positions in the file of
the contributing blocks, not their UF2 block numbers. `family_name` is REA's
label, not file content. It comes from a small table copied from Microsoft's
`uf2families.json` at commit
[`cf3bd7f9`](https://github.com/microsoft/uf2/blob/cf3bd7f9c274b949528f8cd6a19eef2502a687d2/utils/uf2families.json),
which the root's `family_name_registry` names. The table covers the ESP32
family, ESP8266, RP2040, the RP2350 and RP2XXX download families, STM32F4,
nRF52840, SAMD21 and SAMD51. It also covers the families that LibreTiny builds
for and the registry lists: RTL8710A, RTL8710B, RTL8720C, RTL8720D, BK7231U,
BK7231N, BK7251 and BL602. Other IDs, LibreTiny's LN882H among them, keep their
value with `family_name: null`. A familyID names a target family; it does not
establish the CPU state, memory map or entry point.
Root `format_metadata` (`uf2`) records:
| Field | Meaning |
| ---------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------- |
| `block_count`, `trailing_bytes` | Complete 512-byte blocks, and bytes after the last complete block |
| `invalid_block_count` | Blocks with a wrong magic number, a payload over 476 bytes, or a payload that wraps past 4 GiB |
| `flash_block_count` | Valid main-flash blocks, the only source of region bytes |
| `not_main_flash_block_count`, `file_container_block_count` | Blocks with flag `0x1`, and blocks with flag `0x1000` |
| `md5_block_count`, `extension_tag_block_count` | Blocks that declare an MD5 flash region (`0x4000`) or extension tags (`0x8000`) |
| `family_count`, `family_name_registry` | Distinct familyIDs among main-flash blocks, and the registry revision of `family_name` |
| `numbering_group_count` | Groups of valid blocks that share a familyID (or none) and a declared block count |
| `missing_block_count` | Block numbers below a group's block count that no valid block carries |
| `ambiguous_numbering_group_count` | Groups whose numbers appear at unequal counts of target addresses |
| `repeated_block_count`, `out_of_order_block_count` | Blocks whose number already appeared in their group, and blocks numbered at or below the previous block of their group |
| `block_number_out_of_range_count` | Blocks numbered at or above their block count, or with a block count of 0 |
| `region_count`, `data_bytes`, `duplicate_bytes` | Regions, unique decoded bytes, and bytes rewritten with identical values |
Numbering is grouped by familyID because a bootloader ignores every field of
another family's block, including its numbering. Within a group, a block number
seen at several target addresses belongs to several copies of a sequence:
esptool numbers each merged input file from 0, so inputs with equal block
counts share a group, and each number then appears once per input. A block sent
twice to the same address is one block. A group is complete when every number
below its block count is present at the same count of addresses. If some numbers
appear at more addresses than others, a copy may have lost a block, so coverage
is unknown and a limitation names the group.
## UF2 facts, limitations and failures
| Situation | Outcome |
| ------------------------------------------------------------- | ---------------------------------------------------------------------------------------------- |
| Blocks out of address order | Sorted into regions |
| Blocks numbered at or below the previous block of their group | Counted in `out_of_order_block_count`; the spec lets blocks arrive in any order |
| An address written twice with the same value in one family | Accepted and counted in `duplicate_bytes` |
| An address written twice with different values in one family | Fails, naming the family, the address and the block spans of both writes |
| The same addresses in different families | Separate regions; never a conflict |
| Numbered blocks absent from a group | Regions are returned; coverage is unknown and a limitation names the groups |
| Reused numbers at unequal address counts | Regions are returned; coverage is unknown and a limitation names the groups |
| An invalid block | Excluded; coverage is unknown and a limitation names the first one and its offset |
| A partial final block | Excluded; coverage is unknown and the file may be truncated |
| Block numbers at or above the block count | Payloads are kept; coverage is unknown |
| Not-main-flash blocks (`0x1`) | Excluded from regions; their count and payload byte count are reported; coverage is unaffected |
| File-container blocks (`0x1000`) | Excluded from regions; their files are not reconstructed, so coverage is unknown |
| MD5 (`0x4000`) and extension-tag (`0x8000`) blocks | Counted and reported as a limitation; neither is decoded or verified |
Unlike a malformed Intel HEX record, which fails the whole file, an invalid UF2
block is excluded and reported: each block carries its own framing and address,
so its neighbours stay readable.
The MD5 structure is not checked because it is not a checksum of the block's
bytes in the file. It names a region of device flash, usually aligned to erase
pages, whose MD5 a flashing tool compares to skip writing. Producers also
disagree on its place: the specification puts it in the last 24 bytes of the
data area, while esptool v4.12.0 and v5.6.0 write it right after a shorter final
payload, so those 24 bytes hold zeros or part of that structure.
## DfuSe regions and metadata
Each element is one entry at its own file range, named
`target-<t>/element-<e>-0x<start>-0x<end>.bin` with zero-based target and
element indexes, for example `target-0/element-0-0x08000000-0x0800ffff.bin`.
Elements are never merged. Each occurrence's `source_location` is the byte range
of the element data in the file.
Element `format_metadata` (`dfuse-element`) records `target_index`,
`alternate_setting`, `target_name`, `element_index`, `start_address`,
`end_address` and `length`. `target_name` is `null` when `bTargetNamed` is 0;
otherwise it is `szTargetName` up to its first NUL, one character per byte.
`compare_artifacts` treats an element's alternate setting and target name as
its declaration, so the same bytes moved to another device memory are
`changed` in `metadata`.
Root `format_metadata` (`dfuse`) records:
| Field | Meaning |
| ---------------------------------------------------- | ------------------------------------------------------------------------------- |
| `image_size`, `target_count` | `DFUImageSize` and `bTargets` from the prefix |
| `element_count`, `empty_element_count`, `data_bytes` | All elements, elements of 0 bytes (which have no entry), and element data bytes |
| `overlapping_element_count` | Elements that overlap an earlier element of the same alternate setting |
| `device_version`, `product_id`, `vendor_id` | `bcdDevice`, `idProduct` and `idVendor` from the suffix |
| `suffix_crc`, `computed_crc` | The stored `dwCRC`, and the same CRC computed over every byte before it |
Recognition already requires `bVersion` 1 and `bcdDFU` `0x011A`, so they are
not repeated. The DFU suffix CRC is CRC-32 without the final inversion, as
dfu-util stores and checks it.
Producers disagree on `DFUImageSize`. UM0391 calls it the file length, and
dfu-util's `dfuse-pack.py` writes that value since commit `ee6b393d`
(2020-10-08). Earlier `dfuse-pack.py` and MicroPython's `tools/dfu.py`
(checked at `cb99ca98`) write the length before the suffix, and an ST community
report describes ST's DfuFileMgr the same way; REA has not run DfuFileMgr.
dfu-util's own parser ignores the field. REA accepts either value and keeps the
declared one in `image_size`.
## DfuSe facts, limitations and failures
| Situation | Outcome |
| ---------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------- |
| `dwCRC` differs from the computed CRC | Follows `integrity_policy`, below |
| A target without the `Target` signature | Fails with its byte offset |
| `DFUImageSize` that equals neither the file length nor the bytes before the suffix | Fails with the declared size, the suffix offset, the file length and whether the suffix CRC matches |
| A target size or an element size that disagrees with the bytes that hold it | Fails with the byte offset and the declared and available sizes |
| An element whose address range wraps past `0xFFFFFFFF` | Fails with its byte offset |
| Overlapping elements of one alternate setting | Kept as separate entries; a limitation names the first four; coverage is unaffected |
| The same addresses in different alternate settings | Separate entries; an alternate setting selects a device memory |
| Elements of 0 bytes, and targets with no elements | No entry; counted; a limitation names the first four, with address and alternate setting |
A CRC mismatch is an integrity contradiction, never proof of a cause. With the
default `integrity_policy` `fail`, inspection and extraction fail with
`artifact_integrity_mismatch`, whose details carry `algorithm: "dfu-crc32"`,
`declared_crc32` and `calculated_crc32`. With `record-and-continue`
(CLI `--integrity-policy record-and-continue`), the root occurrence becomes
`mismatched`, `integrity_contradictions` records the pair for logical path `.`,
and extraction carries that contradiction, because every element came from the
contradicted container. The root occurrence has no `declared_integrity`: that
field holds entry digests, so the declared CRC appears in the contradiction and,
as a number, in `suffix_crc`. Under `fail`, the elements are hashed before the
mismatch is raised.
## Limits
Each file must be at most 128 MiB. The UF2 decoder reserves at most 476 bytes
per complete block before reading, merges payloads into regions of at most the
same total size, and keeps one numbering entry per valid block, so its peak is
about twice the decoded bytes. It retains at most 65,536 regions as entries; the
order of blocks does not count against that bound. The DfuSe decoder holds the
file once, serves elements as views of it, and retains at most 65,536 elements
with bytes as entries; empty elements are only counted. These are fixed
resource bounds, not format limits, and fail as limits (`file-size` and
`capacity`) with no runtime override. Configurable ceilings are tracked in
[#6176](https://github.com/morluto/rea/issues/6176).
Decoded regions are cached only for the reader's lifetime, and each open checks
the admitted carrier identity again. These readers share the Intel HEX
reader's region streams, cancellation and close behavior.
## Analyzing a region
Inventory does not establish the CPU, byte order or memory map, or an entry
point. To disassemble a region, extract it and open the region file with
`open_binary` using format `raw-binary`, an architecture you choose, and
`entry_point` set to the region start address; raw-binary maps file offset 0 at
`entry_point`. Each region is opened separately, so references between regions
are not resolved.
## Verification
Fixtures written from the UF2 specification and from ST UM0391 cover block and
element layouts, numbering groups and reused numbers, families, rewrites and
conflicts, invalid and partial blocks, not-main-flash and file-container blocks,
structural faults, overlaps, both integrity policies, accepted boundaries beside
each refusal, resource refusals, inventory, extraction, comparison and reader
identity checks. The reader lifecycle (cancellation, close and stream
ownership) is covered by the Intel HEX reader tests, which exercise the shared
reader. Two DfuSe fixtures are byte-identical, CRC included, to what the
`build()` functions of dfu-util's `dfuse-pack.py` (at `f9a537c9`) and
MicroPython's `tools/dfu.py` (at `cb99ca98`) produce for the same image, one for
each `DFUImageSize` convention. Those functions were transcribed and run under
Python 3; the scripts themselves were not run. CLI and stdio MCP tests exercise
the same artifact provider and advertised schemas.
`npm run verify:uf2` runs esptool's `merge-bin --format uf2` on generated
inputs at known addresses, with and without MD5, with 256-byte payloads, with
two inputs of equal block count, and on a concatenation of two chips' outputs.
Inspection and extraction must return every input at its address under
esptool's own familyID. It passed with esptool v4.12.0 and v5.6.0 on macOS. Set
`REA_ESPTOOL_PYTHON` to a Python that runs `-m esptool`. Pico SDK, picotool,
`uf2conv.py`, TinyUF2 and real DfuSe producers remain unverified, and no device
accepted these files in REA's tests.
+2
View File
@@ -50,6 +50,7 @@ harm_rank,location,trigger,current_copy,user_risk,proposed_replacement,implement
36,src/main/reportMcpTransportError.ts:reportMcpTransportError,MCP error has no known validation or disconnection identity,MCP reported an error with unknown connection state. Check whether the client is still connected before following connection-loss recovery.,Connection state remains unknown,Keep current copy,implemented,injected unknown-error recovery regression
37,src/main.ts:serveStdio throw,MCP connection cannot be created,Failed to start MCP stdio transport,Exposes transport implementation and gives no recovery,"Retry once. If it fails again, tell the user REA could not start its MCP connection and ask them to restart the MCP client, then run `rea doctor`.",implemented,verified by injected transport startup-throw runtime test
38,src/main.ts:requestShutdown catch,Connection or analysis session close rejects,Failed to shut down cleanly,Does not explain duplicate-process risk or recovery,REA could not close cleanly. End the REA process before starting it again.,implemented,verified by injected handle-close rejection runtime test
38.5,src/domain/analysisErrorRecovery.ts:cleanupRetryRemediation,close_binary returns cleanup_incomplete while the session retains the failed cleanup owners,"Provider cleanup could not be fully confirmed. Review the reported local resources before opening another target. (retryable: false, no recovery)",Agent could abandon the connection that owns the only cleanup retry,"Append: This connection retains that cleanup. Keep it open; after the reported condition settles or is resolved, call close_binary without snapshot_path and check its result. That call retries the retained cleanup without rerunning the failed analysis, preserves investigation Evidence and Unknown history, and can report cleanup_incomplete again. Add recovery.kind=retry_cleanup naming close_binary; keep retryable false",implemented,MCP close_binary with the real KunaProvider and a fake Kuna whose export stop stays unproven; process-capture MCP retry; session tests for an unoffered one-shot close and for failed and completed snapshot saves; one-shot IDA CLI cleanup reports no hint; real Kuna not run
39,src/main.ts:entrypoint catch,Unhandled rejection escapes run(),Unexpected server startup failure,Vague and lacks diagnostic recovery,"Do not repeat this startup. Tell the user REA could not start and ask them to restart the MCP client, then run `rea doctor`.",implemented,verified by injected entrypoint rejection test
40,src/application/Update.ts:unknown-install-method,Running package is not owned by detectable global npm installation,Update manually with: npm install --global rea-agents@latest,Old global-npm advice does not identify the owning installation,"Update this installation with the package manager or source checkout that owns it. For a one-off latest invocation, use npx rea-agents@latest.",implemented,36 Update tests passed; public CLI unowned-installation refusal verified with isolated package-manager fixtures
41,src/application/Update.ts:install,npm global install subprocess exits unsuccessfully,npm install --global rea-agents@latest failed.,Old registry/global-install advice omits the actual version prefix and installer failure,"REA could not install {version} in {prefix}: {failure}. Repair this npm installation, then rerun rea update.",implemented,36 Update tests passed; public CLI owned npm fixture failure preserved version prefix and installer reason
@@ -119,6 +120,7 @@ Then rerun this installer.",implemented,verified by shell branch inspection
90.5,@modelcontextprotocol/server:tool input validation,Caller sends arguments that violate a declared tool schema,"Canonical invalid_request with the SDK diagnostic plus recovery.kind=inspect_input_contract and a bounded instruction to read the advertised schema or call tools/list",Keeps field/type correction while telling the agent how to recover; diagnostics over 4 KiB stay on the existing retained-error path,Normalize ordinary validation failures and retain oversized diagnostics,implemented,verified by real in-memory and stdio MCP client/server calls with wrong argument types
90.6,incur:unknown CLI command,Caller enters an unknown REA command,"'COMMAND' is not a command for 'rea'. Suggested command: rea --help",Low risk; identifies input and exact discovery action,Keep framework-generated copy,unchanged,verified by real packaged dispatcher invocation
90.7,src/ghidra/GhidraProviderClient.ts:projectPrimarySessionError,One procedure's decompile reaches REA_GHIDRA_DECOMPILE_TIMEOUT_SECONDS,"Ghidra decompilation of {entry} exceeded the {N}-second limit set by REA_GHIDRA_DECOMPILE_TIMEOUT_SECONDS. Only this procedure's decompilation was stopped; the Ghidra session remains usable.",Low risk; names the setting and keeps the session,Keep current copy,implemented,provider projection test and real-provider lane
90.8,src/ghidra/GhidraFileOffsetFailure.ts:ghidraFileOffsetUnavailable,"address_to_file_offset receives a well-formed address that is unmapped, uninitialized, without an authoritative target-file source, ambiguous or beyond the exact JSON integer range",{bridge reason}. REA establishes no single exact original-file offset for this well-formed address and does not derive or choose one.,Previous copy called the input invalid and directed schema repair for a valid address,"Keep the bridge reason as the headline, point to inspect_native_load_image, omit recovery.kind=inspect_input_contract and record details.issues[0].observed.file_mapping",implemented,provider projection tests; MCP and CLI analysis path through the real Ghidra client with a simulated bridge; real Ghidra not run
91,src/application/binaryDiagnostics/ByteEntropyService.ts (issues); src/domain/analysisErrorPresentation.ts (recovery),Invalid file or window,Expected a regular file.; Selected file could not be found.; Selected byte window starts beyond the end of the file.; Selected byte window extends beyond the end of the file.; File size exceeds exact numeric offset representation.,Invalid selection cannot produce complete entropy Evidence,"Read the declared input schema or CLI usage for inspect_byte_entropy; if using MCP and the catalog may be stale, call tools/list. Correct the listed fields, then retry. Do not repeat the unchanged call.",unchanged,Executed CLI/stdio MCP: path/offset/length refusals and tiny + system-binary controls; shared canonical recovery; service/MCP tests check classification and fields; remaining listed errors source-inspected
92,src/domain/analysisView/byteEntropyView.ts,Non-summary entropy view,Byte entropy Evidence supports the summary view; retrieve parent Evidence for the complete block series.,Unsupported view selection,Use the summary view or retrieve complete parent Evidence,implemented,typed input rejection
93,src/windows/WindowsNativeLoader.ts:load refused,Windows refuses to load the digest-verified native addon (for example Smart App Control or App Control policy),"REA Windows native controls are unavailable: Windows refused to load the verified native/windows/build/rea-windows-x64.node: {OS error} The file matched the package manifest, so restoring or reinstalling it will not help. ...",Previous artifact copy told users to restore or rebuild files that were already intact,"Name the OS refusal, say restoring cannot help, state that Ghidra and JADX need native controls while setup can still register clients, and point to a policy-allowed host or a maintainer-signed build",implemented,simulated loader refusal unit test; real Smart App Control block not reproduced
1 harm_rank location trigger current_copy user_risk proposed_replacement implementation_status verification_result
50 36 src/main/reportMcpTransportError.ts:reportMcpTransportError MCP error has no known validation or disconnection identity MCP reported an error with unknown connection state. Check whether the client is still connected before following connection-loss recovery. Connection state remains unknown Keep current copy implemented injected unknown-error recovery regression
51 37 src/main.ts:serveStdio throw MCP connection cannot be created Failed to start MCP stdio transport Exposes transport implementation and gives no recovery Retry once. If it fails again, tell the user REA could not start its MCP connection and ask them to restart the MCP client, then run `rea doctor`. implemented verified by injected transport startup-throw runtime test
52 38 src/main.ts:requestShutdown catch Connection or analysis session close rejects Failed to shut down cleanly Does not explain duplicate-process risk or recovery REA could not close cleanly. End the REA process before starting it again. implemented verified by injected handle-close rejection runtime test
53 38.5 src/domain/analysisErrorRecovery.ts:cleanupRetryRemediation close_binary returns cleanup_incomplete while the session retains the failed cleanup owners Provider cleanup could not be fully confirmed. Review the reported local resources before opening another target. (retryable: false, no recovery) Agent could abandon the connection that owns the only cleanup retry Append: This connection retains that cleanup. Keep it open; after the reported condition settles or is resolved, call close_binary without snapshot_path and check its result. That call retries the retained cleanup without rerunning the failed analysis, preserves investigation Evidence and Unknown history, and can report cleanup_incomplete again. Add recovery.kind=retry_cleanup naming close_binary; keep retryable false implemented MCP close_binary with the real KunaProvider and a fake Kuna whose export stop stays unproven; process-capture MCP retry; session tests for an unoffered one-shot close and for failed and completed snapshot saves; one-shot IDA CLI cleanup reports no hint; real Kuna not run
54 39 src/main.ts:entrypoint catch Unhandled rejection escapes run() Unexpected server startup failure Vague and lacks diagnostic recovery Do not repeat this startup. Tell the user REA could not start and ask them to restart the MCP client, then run `rea doctor`. implemented verified by injected entrypoint rejection test
55 40 src/application/Update.ts:unknown-install-method Running package is not owned by detectable global npm installation Update manually with: npm install --global rea-agents@latest Old global-npm advice does not identify the owning installation Update this installation with the package manager or source checkout that owns it. For a one-off latest invocation, use npx rea-agents@latest. implemented 36 Update tests passed; public CLI unowned-installation refusal verified with isolated package-manager fixtures
56 41 src/application/Update.ts:install npm global install subprocess exits unsuccessfully npm install --global rea-agents@latest failed. Old registry/global-install advice omits the actual version prefix and installer failure REA could not install {version} in {prefix}: {failure}. Repair this npm installation, then rerun rea update. implemented 36 Update tests passed; public CLI owned npm fixture failure preserved version prefix and installer reason
120
121
122
123
124
125
126
@@ -0,0 +1,246 @@
# Capability composition verification — 2026-11-01
Scope: issue #8429, the service composition/ownership increment of #740. Baseline
`6e9ee97a69cb16c7150d8b550e705cba055ee4f0`; implementation `bd9bf76d8`, followed
by type-only binding cleanup `161c3c146`. This changes existing composition,
metadata ownership and cleanup bookkeeping; no new provider/target authority.
`BinarySession` keeps its existing transition/admission barrier.
## Executed source and boundary checks
Full typecheck, module boundaries, architecture guards, test discovery,
production reachability, documentation consistency and repository formatting
passed. Changed-file typed lint passed with one worker and `GOMEMLIMIT=512MiB`:
39.622 s elapsed, 39.261 s process-family CPU, 2,945,368,064 B kernel peak.
Initial whole-repository typed and ordinary lint attempts hit the configured
3 GiB early stop; no OOM occurred. Whole-repository lint is not locally
qualified; changed-file checks and the final ownership-boundary typed checks
passed separately.
Relevant executed groups: 216 parser/boundary/metadata/provider-owner tests;
5 scope/factory tests; 27 MCP availability/composition tests; 28 deferred
startup/protocol/publication tests; 50 CLI factory/cleanup/target-input tests;
16 SDK schema tests; 4 compiled-verifier metadata link tests; and investigation
lifecycle, unknown-history and transport cleanup regressions. These groups
contain overlapping repeated checks and are not a unique total. The final
Android transport group passed all 4 cases, including construction failure and
failed cleanup retry. The moved Frida test body was preserved byte-for-byte;
its discovery owner changed to the adapter lane.
The old manual provider-close list was temporarily restored as a negative
control: the new successful-peer cleanup regression failed, then the current
source was restored. Synchronous close throws, undefined rejection reasons,
concurrent close joining, borrowed optional observations and failed-owner retry
are controlled tests, rather than claims of natural failures in real engines.
An initial new transport assertion omitted the existing outer
`OwnedMcpServers` AggregateError; its expected nesting was corrected. Other
preserved harness failures were missing SDK catalog preparation and two wrong
CLI test paths; corrected commands passed. One formatting issue was corrected.
Ordinary Knip reports one unused exported `DarwinHelperCache` type at
`src/process/DarwinHelperCache.ts:465`. An independent run on the clean pinned
baseline reports the identical sole issue; that file is unchanged here.
## Real applications and complete-result parity
Every case used actual public CLI/MCP routes. Inputs were hashed before/after,
with no source mutation. Both arms used pinned Node 24.22.1/npm 11.16.0 and
fresh isolated processes/configuration; OS page caches were uncontrolled.
| Domain | Genuine application/artifact | Independent fact oracle |
| ---------- | ------------------------------------------------------------------- | ----------------------------------------------- |
| Go | installed GitHub CLI `/usr/bin/gh`, Go 1.19.6 | ELF Go build-info bytes/module data |
| Managed | SDK 8.0.427 `NuGet.CommandLine.XPlat.dll`, selected Program members | independent dnfile metadata/CIL facts |
| JavaScript | installed agent-browser 0.27, selected entry source | original source declarations/imports |
| Android | NewPipe 0.29.1 APK inventory/application graph | complete ZIP member census and 9 member digests |
| WASM | actual CSS WASM artifact with WABT 1.0.42 | independent raw sections/types/exports |
All five full public CLI Evidence results matched across arms. Explicit Go/JS
per-arm bundle output paths were verified separately, and their complete saved
parent Evidence matched. MCP performed 117 calls per arm, including six fresh
producer calls, retained derivation, repeated close, checkpoint/reset, expected
missing references, same-process restore and actual fresh-process restore.
Complete producer bundles, full derived Evidence, exact requests/IDs and
complete checkpoints matched across arms. The analyst Unknown revision 1→2
history and every canonical mutation/history record remained authenticated;
fresh re-exported checkpoints were byte-identical. Fresh consumers ran no
producer calls. Ten independent domain fact assessments passed.
| Completed workflow | Baseline elapsed / CPU (s) | Feature elapsed / CPU (s) | Baseline / feature kernel peak (B) |
| ------------------------ | -------------------------- | ------------------------- | ---------------------------------- |
| Five CLI tasks | 17.675 / 16.268 | 22.192 / 20.667 | 393,936,896 / 390,504,448 |
| Five MCP lifecycle tasks | 68.270 / 65.500 | 61.201 / 58.041 | 731,680,768 / 776,331,264 |
Times are single observations including setup and all calls, not a causal
speedup/regression claim. Fresh-process tasks are reported separately from
same-process/restored consumers; neither is a controlled cold OS-cache run.
All tasks ran serially under a one-CPU, 4 GiB hard/3 GiB early-stop, no-swap,
192-task guard. CPU and peak memory are kernel process-family counters at
payload exit while the observer remained alive; all completed tasks verified
owned cleanup and no OOM. Sampling/RSS is not substituted for these counters.
Scripted checks do not incur model tokens; model usage is recorded separately.
## Native engine and Codex CLI
Real Ghidra 12.1.4/JDK 21 on `/usr/bin/jq` passed 37 public MCP calls per
arm. Two new light address queries before/after healthy investigation reset
returned identical Evidence. The same MCP client retained the same owned JVM
PID, start time, process group and public run identity. Three retained native
compositions used authenticated historical dossiers; these dossiers were not
regenerated. Full derived results, original bundles and Unknown histories
matched across arms and survived same/fresh-process restore.
A mixed-routing task passed 58 calls per arm, with explicit `jq → APK → managed`
metadata target switches and five new static producer families through the final
registry. These metadata switches do not imply launching every target engine.
Complete originals, derived results and checkpoint history matched across arms.
| Completed task | Baseline elapsed / CPU (s) | Feature elapsed / CPU (s) | Baseline / feature kernel peak (B) |
| --------------------------- | -------------------------- | ------------------------- | ---------------------------------- |
| Ghidra/native lifecycle | 38.196 / 36.804 | 41.741 / 40.369 | 1,111,789,568 / 1,098,362,880 |
| Mixed target/domain routing | 27.557 / 26.984 | 27.255 / 26.527 | 1,039,253,504 / 1,037,910,016 |
The native task includes a fresh engine start; its second query is a healthy
same-engine observation. Historical dossier reuse and fresh process consumers
are separate phases. These are single observations with uncontrolled OS caches.
Local `codex-cli 0.165.0` ran one actual `client-code` trial with the requested
`gpt-6.1-terra` alias, reasoning `high`, and 19 selected tools. The resolved
server-side model identity is not inferred. It completed 32 MCP calls across
JavaScript, managed, Android and WASM: five new producer calls, retained
compositions, four expected missing-reference controls, close/reset/save/import
and Unknown revision 1→2. There were no unexpected MCP errors or timeout.
The model's JS self-check remained explicitly **partial/unverified**: it used
`JSON.stringify`, whose property ordering changed after canonical import.
Independent full structural comparisons of all three actual results and the
baseline proved identical fields, values, array order and Evidence ID. The
first ordering difference was only the keys of
`graph.nodes[0].observations[0].properties`. This distinction is retained in the
assessment; the model's self-check is not relabeled a complete pass.
An actual new, target-free MCP process replayed the model's checkpoint in
9 calls. Complete Evidence and Unknown revision/mutation histories authenticated,
and its 18,968,472 B export was byte-identical (17 Evidence records,
2 Unknown revisions, 1 head), SHA-256
`165188d74298f0173c5730973ecb5354a1b966ca54ec46ee177efa049ebbc755`.
All four derived results and their complete producer records matched the
independent baseline. The copied authentication profile was removed; runtime,
production source and selected input hashes remained unchanged.
Actual client usage for the single attempt: **336,490 input tokens**, of which
**295,680 were cached**; **3,659 output tokens** and **250 reasoning output
tokens** were reported. Cached input is a subset, not an additional token total.
Baseline model usage was not measured; neither token savings nor a speedup is
claimed. Scripted source-fact and fresh-process verifiers incur no model API
tokens and are separate from this model usage.
| Completed verification | Elapsed / process-family CPU (s) | Kernel family peak (B) |
| ----------------------------------- | -------------------------------- | ---------------------- |
| Actual Codex CLI trial | 74.644 / 26.818 | 1,061,597,184 |
| Independent fresh checkpoint replay | 6.142 / 5.934 | 757,895,168 |
Manual final-answer review verified the reported IDs, counts, digests, coverage
and Unknown state against complete baseline values, all actual model calls and
the independent replay. The outcome is supported with the explicit model JS
self-check limitation resolved independently. Maximum observed in-flight MCP
calls was one; no target command, web or file-change tool items occurred.
Factual-assessment SHA-256:
`33451102e5faf57f11c627481899f16c4fc47fa5cba5f12cd2ed781a3de54ccf`.
Final immutable input/runtime audit SHA-256:
`73abce493182a7b433b4b78737720640ceb41d6df435cb0fec915468a0a304c8`.
## Catalog and build identity
Both compact/full SDK catalogs are byte-identical to baseline: 234 tools,
6 prompts, 180 CLI commands. The provider projection changes from 63 to 67 rows
only for source-proven web-source/module tracing, Harmony and QLOG provenance
corrections. All other product sections are unchanged. Its expected provider
SHA-256 is `4234ed15d7e53f34599b59dfba1f678369c0ae95b2135c7ba3414d944ab156b6`.
The 66 static declarations have 49 runtime keys and 17 explicit source-owned
workflow/engine groups; 70 bindings union to 67 identities. Static imports reach
44 pure modules without schema, engine or factory runtime imports.
The baseline runtime is `0e94962f64f3f532cc96b2432916f328f60058e97cd0e8e25f5c3e6ed3cdec10`
(2,670 files). Static pairs used clean `bd9bf76d8`, runtime
`62018a023b185dd9c6839799ed1fe4f085810161c723e508fc817d2fccdb59e3`
(2,713 files). Final type-only cleanup `161c3c146` rebuilt to
`d7be1d17dc6b5c23f33468398656ac4d3961380368be1f70dc767f0ba1fac6bf`.
The complete earlier manifest is cryptographically reconstructed by removing
one emitted trailing comma in `dist/composition/capabilityServices.js`'s
existing named import; every other runtime file is byte-identical. This is not
a claim that the two runtime hashes are identical. Native/mixed/model cases use
the final runtime.
Raw inputs, outputs, commands, hashes, recorded failed checks and resource receipts
remain outside Git under
`/home/yann/yann/.rea-worktree-artifacts/20261101/capability-module-ownership/`.
Static-pair audit receipt SHA-256:
`801707569577c7cec848334cf72bfab26a7ff4a95c2a87872d71d78eef7765a3`.
## Review corrections and final qualification
Two actionable review findings were fixed at their producing owners:
- `980991573`: import-edge facts now distinguish fully erased declarations from
inline type bindings, which still load their module under
`verbatimModuleSyntax`. Compiler-backed parser/boundary regressions passed;
the old parser fails the new expectations. A separate three-file probe through
the actually selected compiler emitted both inline runtime edges and erased
whole-declaration/type-query edges, without compiling the repository.
- `de6a61636`: the existing contract owner exposes 47 named canonical groups.
Thirty-seven full provider bindings reference those shared arrays, while
33 deliberate provider-subset/engine selections remain precise. The actual
ADB metadata now includes a new valid canonical ADB operation without an edit
to the provider declaration; reverting to the copied production lists fails
the regression. Unknown groups, unknown names and identity conflicts fail
catalog generation.
Final full typecheck, module boundaries, test discovery and changed-file typed
lint passed. Post-review source regressions passed 228 cases and SDK/MCP
regressions passed 34; the two strengthened boundary/metadata files passed
31 cases separately. These overlap earlier groups. Product catalog SHA-256
remains exactly `e942c72047db651939dd47a31042e6818caaca7470092d63645801954a72b523`;
both SDK catalogs remain byte-identical to the original baseline. Static metadata
retains its pure loading boundary; canonical group IDs are imported only as types.
The final implementation freeze is `de6a61636f3f84862cb4263c9ea219b7d2d4de36`,
with 2,713 runtime files, SHA-256
`be1d60c7dcca57eeb7ad09d682efd172c7f1679215e796a291b29eb52e29bda1`.
Earlier model/engine measurements remain attributed to their executed revisions;
they are not relabeled as runs on this later freeze.
A new actual mixed-routing task on this final freeze passed all 58 calls:
32.970 s elapsed, 32.180282 s process-family CPU, 1,042,538,496 B kernel peak.
Independent five-domain source facts, complete producer records and all five
full derived results/requests/IDs matched baseline and the prior qualified
implementation. Same-process and true fresh-process restore preserved the full
Unknown 1→2 mutation history and byte-identical 18,978,040 B checkpoint
(20 Evidence records, 2 Unknown revisions, 1 head), SHA-256
`0316bd6271c6d401a18c10d552b2005d39c98f3bbbc5945ae4823875af13fd5f`.
Owned cleanup, runtime/production-source/input hashes and no OOM were verified.
Append-only final-routing receipt SHA-256:
`9c410c2c31bf38d7edc569deb95b93b7f9a2cf68a7b9a7f84d3ec924f0ad75cb`.
This bounded requalification adds no model invocation or repeated native
analysis query; earlier token measurements remain unchanged.
CodeQL's six reported alert IDs (215, 490, 491, 493, 494, 495) all have open
instances on exact-base main `6e9ee97`. All 24 SARIF paths were reviewed and
independently checked: the new factory-default `process.env` sources retain the
existing command/directory selection and explicit local authority. The shared
filesystem/process sink files are byte-identical to base; Cutter's sink line
shift only removes its old identity constant. No source suppression or global
alert dismissal is included. The PR records the exact-base instance evidence
and the owner's authorized exception for this unrelated existing-alert check.
## Qualification limits
NewPipe coverage is static inventory/graph, not real JADX decompilation: JADX,
apktool and aapt2 were unavailable. No genuine local GGUF/safetensors target was
available. Real Hopper, IDA, JEB, Cutter GUI, GDB/Rizin debugger and Frida target
attachment are not qualified by these application cases; relevant factory,
protocol and cleanup tests use controlled seams. Historical native dossiers
are explicitly labeled reused evidence, not newly reproduced function analysis.
Retained records remain bounded by existing investigation policies; this does
not promise unlimited retention or a new durability backend. Further module
qualification and `BinarySession` simplification remain under parent #740.
File diff suppressed because it is too large. Load diff
@@ -0,0 +1,263 @@
# Evidence input composition verification
The shared declaration and request resolver cover 20 individual-Evidence
composition tools. Artifact file sources, analysis views and environment export
keep their explicit source owners. Catalog conformance checks those boundaries
and rejects undeclared record inputs or unsupported source containers.
This record accompanies [#8293](https://github.com/morluto/rea/issues/8293) and
the real Ghidra input-capacity regression
[#8281](https://github.com/morluto/rea/issues/8281).
The [machine-readable receipt](evidence-input-composition-2026-10-31.json)
retains all fourteen model attempts, artifact identities, oracle provenance,
tool profiles, actual usage, resource measurements and limitations. It also
records the thirteen subsequent scripted public-transport attempts below.
## Runtime and measurement scope
Tests used Node 24.22.1 and local Codex CLI 0.165.0, explicitly requesting
`gpt-6.1-terra` with reasoning `high`. This identifies the requested alias;
the resolved server model is not inferred. Both dirty candidate builds were
based on `23de10bb2849cdce8ddbe70504aa7cc57de75a98`. Their complete built-file
fingerprints are recorded in the receipt. The later build includes the shared
reference-format hint and authenticated source/type-boundary corrections.
Subsequent cleanup reused the existing pure semantic-query input schema and
moved SDK tests into their boundary project. Full and compact catalog bytes
remained identical to the later model build; 22 affected checks were rerun.
Heavy tasks ran serially in owned Linux scopes: one CPU, 4 GiB hard memory,
3 GiB early stop, no swap and at most 192 tasks. CPU and peak memory are kernel
process-family counters at payload exit while the observer remains alive;
they are not final whole-scope counters. Cleanup was verified for every model
attempt and CLI phase. No OOM event occurred.
Each model task starts a fresh MCP process and isolated client profile. Native
tasks start a fresh Ghidra database, then reuse its two retained dossiers.
System/filesystem caches were not cleared; cold filesystem costs are unmeasured.
Initial primary tasks used the complete catalog. Later scoped tasks are labeled
`diagnostic/catalog-selection`; their different tools, guidance and selected
facts prevent a controlled speed or token comparison.
## Real application results
| Domain | Real artifact | Reviewed outcome |
| ---------------- | ------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Native | Installed jq ELF, real Ghidra 12.1.4 | Each selected function analyzed once; complete parent references supported a direct one-hop path and function comparison. Independent `objdump` corroborated the selected call. Nonexhaustive frontier and behavioral uncertainty remained visible. |
| Managed | Shipped NuGet.CommandLine.XPlat.dll, assembly 6.11.2.1 | Artifact/member parent references projected a managed graph. The existing distinct-Evidence rule refused self-comparison. Digest, MVID, owner/token and selected CIL facts matched an independent metadata reader; its parser warnings limit that oracle's scope. |
| JavaScript | Obsidian 1.14.4 ASAR | Partial: the producer hit its existing 300-second per-source timeout. The retained parent still supported a semantic trace. A bidirectional feature trace exceeded 10 MiB and had no supported retained summary. Tracked separately in [#8296](https://github.com/morluto/rea/issues/8296). |
| JavaScript | Shipped agent-browser 0.27.0 CLI entrypoint | The final scoped task analyzed the exact selected source once, then reused its parent for feature and semantic traces. Independent source facts confirmed builtin imports/call locations. Unselected sources remained unknown; directory and file digests were distinguished. |
| WASM | Shipped VS Code Tree-sitter CSS module | One real WABT inspection supplied retained references for export comparison and a portable bundle. An independent raw WASM reader confirmed seven imports, three exports and `tree_sitter_css: () -> i32`. Static interface equality did not imply runtime compatibility. |
| Artifact / Apple | Rectangle 2.0.4 release ZIP | One retained inventory supported Apple projection and artifact comparison. Independent ZIP and Mach-O observations corroborated paths/content. Symlink metadata gaps remained unknown. |
| Android | NewPipe 0.29.1 release APK | One retained inventory supported Android projection. Independent ZIP observations confirmed manifest, three DEX files and four ABI library members/digests. JNI path-pair hypotheses stayed inferred. |
| Harmony | Shark 1.0.8 release APP and its real inner HAP | Both retained inventories supported Harmony projection. Separate ZIP/JSON checks preserved outer ZIP, APP and HAP identities, Stage/Ark/resource path observations and shallow APP coverage. Signature naming was not treated as cryptographic verification. |
| Process | Two controlled real jq CLI captures | Authentic captures were imported and compared by references, then their retained comparison was aggregated. MCP output deep-equaled the existing inline CLI result. Terminal change, settlement differences, filesystem/process unknowns and source closure were preserved. |
The first supplemental agent-browser model task wandered into unrelated
inventory tools and failed to perform the requested producer analysis. It was
stopped and retained as an interrupted failure, including cumulative usage as
a lower bound. The successful later scoped task does not erase that attempt.
The missing-reference task produced its expected typed failure without opening
a target, repeating a request or inventing Evidence.
## Actual model task costs
Input counts include schemas, accumulated context, extra calls and retries.
Cached input is a subset of total input. Reasoning is included in output;
its separate subset is retained in JSON. These are completed task costs, not
serialized-byte estimates. The initial partial/interrupted attempts and their
costs also remain in the receipt.
| Later task | Calls / failed | Input tokens | Cached input | Output tokens | Task elapsed (s) | Family CPU (s) | Peak bytes |
| ------------------------------ | -------------: | -----------: | -----------: | ------------: | ---------------: | -------------: | ------------: |
| jq, complete catalog | 11 / 0 | 599,933 | 535,424 | 1,894 | 80.931 | 38.326466 | 1,269,936,128 |
| agent-browser, scoped | 12 / 1 | 619,589 | 563,712 | 2,519 | 54.521 | 9.472096 | 903,847,936 |
| NewPipe, scoped | 5 / 0 | 179,635 | 150,528 | 1,750 | 40.835 | 10.080132 | 704,409,600 |
| Shark APP, scoped | 6 / 0 | 221,571 | 197,376 | 1,339 | 32.437 | 6.377128 | 560,021,504 |
| Shark HAP, scoped | 5 / 0 | 184,452 | 152,576 | 1,212 | 31.108 | 6.012587 | 580,714,496 |
| WASM plus export, scoped | 4 / 0 | 161,702 | 141,056 | 1,033 | 32.320 | 9.036127 | 700,076,032 |
| Imported process chain, scoped | 3 / 0 | 89,101 | 73,984 | 725 | 26.911 | 5.575060 | 531,283,968 |
The later jq task made no malformed reference calls, while the initial task
omitted the required `kind` twice. It selected a different callee and made three
guidance calls; its longer elapsed time and different facts preclude attributing
a general improvement to the new hint.
## Equivalent portable CLI comparison
The actual WASM model task exported a 13,925,066-byte authenticated bundle.
Both fresh CLI processes selected its same complete inspection parent and
compared exports against that parent. Their 3,732-byte outputs were byte-identical
and deep-equal to the model's MCP comparison:
`ev_b32346965c153a067abff9698aeba1ccbace34c78ac2071ac18311250b68c6da`.
Bundle, selected source and runtime hashes stayed unchanged. Neither CLI arm
started a decoder or reanalyzed the module.
| Input | Request bytes | Elapsed including scope (s) | Family CPU (s) | Peak bytes |
| ---------------------------------- | ------------: | --------------------------: | -------------: | ----------: |
| Complete inline pair | 27,823,017 | 2.832 | 2.612440 | 370,515,968 |
| Retained pair plus selected bundle | 261 | 2.632 | 2.385397 | 310,009,856 |
The reference arm still reads/authenticates the existing bundle. This single
ordered pair ran inline first with shared uncontrolled host caches. It establishes
equivalent completed results and records observed costs; it does not establish
a general speedup. Common upstream inspection/export costs appear in the model
table. Preparation additionally cost 1.119 s / 0.875255 CPU s / 316,796,928 peak
bytes; verification cost 0.518 s / 0.282755 CPU s / 42,881,024 peak bytes.
The optional tokenizer was unavailable, so request-token savings are unmeasured.
Request bytes are not presented as model tokens.
## Fresh-connection public transport matrix
The real transport matrix completed all nine supported domain lanes on built revision `7e10c1cc9` (`63746cf7b340358ac5ce609761ea3198ae3a62e24df52f62ed5be16893940021`). It used public stdio MCP and CLI, real artifacts and authentic exported Evidence; no model or fixture provider ran.
| Lane | Attempts | Scope seconds, all attempts | CPU seconds, all attempts | Max kernel peak MiB | CLI calls, including negatives |
| ----------- | -------: | --------------------------: | ------------------------: | ------------------: | -----------------------------: |
| native | 1 | 35.915 | 34.674174 | 927.99 | 0 |
| javascript | 3 | 34.324 | 33.094918 | 658.91 | 8 |
| managed | 2 | 37.341 | 36.221694 | 694.65 | 9 |
| android | 1 | 20.208 | 19.569397 | 696.40 | 4 |
| harmony-app | 1 | 16.714 | 15.950327 | 495.68 | 4 |
| harmony-hap | 1 | 16.294 | 15.886932 | 508.47 | 4 |
| apple | 1 | 25.453 | 24.824482 | 642.90 | 8 |
| wasm | 2 | 22.648 | 22.027649 | 791.67 | 4 |
| process | 1 | 9.056 | 8.586552 | 472.58 | 2 |
Producer/export costs from the failed JS/managed harness attempts are included. Three harness assumptions and one actual SDK transport-capacity failure are retained individually. Every payload cleanup and kernel observation verified; no OOM or increased allowance.
Fresh imported references match complete CLI inline and selected-bundle outputs where those command contracts support them. Small APP/process/correlation inputs also match admitted MCP inline. Larger MCP inline forms hit the unchanged 1024-element guard. Native and selected managed-member comparisons have no equivalent Evidence-input CLI form; process accepts two complete capture files, and artifact comparison uses explicit evidence-file selectors.
The 27,823,016-byte WASM inline arguments exceeded the default 10,485,760-byte stdio frame buffer and closed the connection before application admission. The resumed attempt reused the same authentic bundle: fresh MCP references and both supported complete CLI forms agree. No WABT replay or limit increase.
Process capture comparison and behavior aggregation preserve changed/unknown facts. Correlation with genuine native comparison Evidence remains unknown with one unresolved branch under an explicit noncausal hypothesis.
These are scripted transport-task costs with uncontrolled caches, not a before/after benchmark or model-token measurement. Earlier WASM inspection/process capture costs are linked in the prior verification report. Later source-only diagnostic fixes have separate focused validation.
The accompanying JSON retains all thirteen attempts, exact kernel counters,
source/result identities and hashes, CLI arm receipts, failures and limits.
These scripted costs do not add to the model-token table.
## Review follow-up
An additional local terra/high source-only review found that the refactored
process CLI read later selected files before authenticating the first capture.
The shared owner now exposes authenticated source facts and their comparison;
CLI authenticates each source before reading the next, while MCP uses the same
readers and comparator. Two public CLI regressions cover invalid left/right
records with missing later files. The original real captures were compared
again without target execution and retained the exact comparison Evidence ID.
Greptile found a fixed-server WASM regression: caller-supplied complete inputs
incorrectly required a recorder. Shared retention now explicitly recognizes
caller-owned inputs. Provider-produced sources still require a recorder;
available recorders, their rejections and cancellation sources remain honored.
A fixture-backed public fixed-server MCP regression and ownership/cancellation
checks cover that distinction. These follow-up regressions are not additional
real-provider measurements. The related nine-file run passed 103 checks;
subsequent schema cleanup and CLI checks passed ten more selected checks.
Type, lint, Knip and architecture/module-boundary checks were repeated.
The fourteen application-analysis attempts above are separate from source-only
review; that CLI review did not report usable model-token counts. Its cleanup
was verified, with 57.860 s task elapsed, 4.720926 s family CPU and 427,364,352
peak bytes. Existing captured model measurements remain tied to their recorded
runtime fingerprints, preceding these review corrections.
The additional seven-file boundary run passed 161 checks after the CLI's JSON
parser was corrected to keep public command names in malformed-input diagnostics.
It includes 17 fixed-server cases for six JavaScript and three mobile consumers,
direct inline analysis views, missing references, wrong source roles, summary
retention and default input admission. These use complete authored fixtures;
they do not substitute for the real-artifact matrix. The run included a runtime
build and cost 154.106 s / 153.592029 CPU s / 1,884,516,352 peak bytes. Subsequent
type, touched-file lint and module checks passed, with no OOM or cleanup gap.
Native function and process workflows send malformed complete source records to
the canonical Evidence owner after resource admission. Schema or identity failures return
`evidence_integrity_mismatch` in both MCP schema profiles, including the original
`relationship_coverage` diagnostics. Other workflows preserve their format-owner
error projections. The directional-coverage regression still
checks every malformed case and the advertised full schema. Invalid request
envelopes and reference grammar remain request errors; no duplicate source parse
was introduced solely to reproduce the former SDK error classification.
A later review reported a possible null-source dereference in reconciliation.
On `70db1e816`, the public request has no identity refinement; source authentication
runs before identity-only validation. A public fixed-server MCP probe passed two
valid controls and rejected 24 malformed source cases across full/compact profiles
with structured `invalid_request` and accurate paths. Both complete schemas also
rejected each of twelve cases without throwing. No provider ran. Cleanup was
verified: 3.334 s elapsed, 3.155635 CPU s and 287,404,032 peak bytes, with no OOM.
This authored-fixture probe disproves the reported path on that revision;
it does not establish real passive-runtime behavior. No production change was
needed for that finding. The accompanying JSON records its observations.
CI later exposed the corresponding process-depth assertion against the old SDK
error envelope. Its regression now checks the canonical integrity error while
preserving the public left-source, `parameters` and exact nesting-limit diagnostic.
A normal comparison must still succeed afterward and provider startup is forbidden.
The iterative JSON depth guard still rejects before recursive cloning or hashing;
no second source parser was added. Runtime code was unchanged. Six related MCP
files passed 104 checks, including source identity, native/process diagnostics,
investigation composition and session lifecycle: 64.384 s elapsed, 64.066692 CPU s
and 1,476,808,704 peak bytes. Cleanup was verified with no OOM.
A retained-closure acceptance assertion also assumed that a complete inline
comparison must already exist in the ledger. The shared workflow intentionally
uses that supplied record: a missing dependency is reported for inline input,
while an unresolved selector reports the comparison itself. Both still require
the full referenced closure. Importing the originating bundle restores both
forms to complete, equal Evidence. Five relevant acceptance files passed 17
checks, covering default-budget aggregation, artifact coverage, replay and public
local contracts: 123.152 s elapsed, 121.276327 CPU s and 1,466,314,752 peak bytes.
Cleanup was verified with no OOM; runtime code was unchanged.
The HTTP admission-edge regression now checks the same source-owner semantics.
An admitted array of 1,019 malformed carriers fails at its first source, while
1,020 carriers exceed the unchanged 1,024-element envelope allowance. Responses
were 703 and 824 bytes, respectively; both stayed below 4 KiB and the server kept
serving. The shared resolver additionally preserves `input_path` on owner errors
when absent. Original error identity, owner diagnostics and existing paths remain
intact, with no second source parse. The rebuilt nine-file gate passed 75 checks,
including all 41 ADB CLI cases, resolver, native references and HTTP/stdio admission:
98.459 s elapsed, 98.133579 CPU s and 1,605,656,576 peak bytes; cleanup verified,
no OOM.
CI also terminated one unchanged ADB CLI child at its helper's existing 20-second
deadline. Two isolated exact 17-extra CLI probes returned the expected argument
validation failure in 1.970 s with a fresh Node compile cache and 1.170 s reusing
it. Neither ran ADB or reached a device; the 20-second deadline was unchanged.
Both local scopes cleaned up with no OOM. The rebuilt full ADB CLI file passed,
including that case in 1.597 s. The CI delay was not reproduced; these observations
do not establish baseline flakiness or a controlled before/after startup result.
After the diagnostic change, the untouched genuine jq/Ghidra bundle was imported
again into a fresh target-free stdio server. Call path, function comparison and
native matching returned complete public values equal to their original matrix
goldens, including IDs, provenance, coverage and unknowns. Seven calls included
no producer, target opening or Ghidra execution. The built runtime fingerprint
was `1bcdcd29df3b90f2cfa61aad29d647dbfb22176d0bca880c7a537507fdeab271`;
source bundle and runtime stayed unchanged. The replay cost 5.543 s / 5.263196 CPU s
/ 447,791,104 peak bytes, with verified cleanup and no OOM. This is an additional
scripted replay, separate from the thirteen-attempt matrix and fourteen model
attempts. Its full receipt and the timeout probes are included in JSON.
## Regression and coverage limits
The focused regression run passed 508 checks covering source integrity, exact
IDs, mixed/optional/nested selections, default constraints, converted full and
compact SDK schemas, source closure, caller choices, CLI bundle immutability,
capacity admission and late cancellation/source retention. SDK boundary tests
are kept in the MCP boundary project; pure declaration checks stay in contracts.
Full-library lint exceeded the local early-stop limit even with one worker.
The same selected 6,105 files were checked in sequential batches of 400;
peak family memory was 741,650,432 bytes. Two new test grouping errors were
corrected and those files rerun. Failed lint/driver attempts remain in local
receipts. Process-driver path/ANSI oracle corrections reused the original
authentic captures.
No real Hopper, IDA, Windows capture, Android device, Harmony runtime, passive
browser reconciliation or managed/native runtime equivalence is established
by these tests. There is no controlled main-versus-candidate model cohort,
no cold filesystem measurement and no claim that every provider facet was
independently verified.
@@ -0,0 +1,298 @@
# Investigation record lifetime verification
Verified on Linux with Node v24.22.1 and local Codex CLI 0.165.0 for
[#8347](https://github.com/morluto/rea/issues/8347). The final production revision is
`40ec9fe79b221b7550bf6a652f55389ea0b7e54f`, including main
`d54a4661da20382ad8456564a08569fdd24131c5`. Its runtime manifest SHA-256 is
`f4e55abdc5cb9fc2a86efcec5c410f2b5bc3eb5a7109a8d5ab88f07f8511c48b`.
Earlier measurements below identify their older revisions explicitly. Documentation
was being updated during final qualification; production sources and runtime
remained unchanged.
The [MCP contract](../mcp-contracts.md) separates engine cleanup, investigation
reset and explicit checkpointing. Automatic journaling remains tracked in
[#4809](https://github.com/morluto/rea/issues/4809).
## Response publication qualification
The request-publication correction is pinned to
`40ec9fe79b221b7550bf6a652f55389ea0b7e54f`. An MCP request now retains its
originating session admission through handler cleanup and physical send
settlement. Nested operations borrow that admission, avoiding reentry behind a
queued reset. Cancellation before a suppressed send and during an already-started
send have separate completion rules. Outgoing server requests cannot acknowledge
an incoming tool response merely because their IDs coincide.
All nine genuine saved-source workflows were repeated: 313 MCP calls and 19 CLI
checks, 114.210 s guarded elapsed, 109.411889 s CPU and 1,055,035,392 B maximum
family peak. Complete original public results and source identities matched
through close, checkpoint refusal, reset and same/fresh-process recovery; all
Unknown histories were preserved. No producer replay, source/runtime changes,
OOM or unresolved owned cleanup occurred. These runs have uncontrolled OS caches
and are not a paired timing comparison with the earlier cohorts below.
A new local Codex CLI task requested `gpt-6.1-terra`/`high` with the scoped
four-domain profile. It completed 25 MCP calls, including four expected
missing-reference controls, with no producer replay. Usage was 232,677 input
tokens including the 199,552 cached subset, 3,613 output tokens and 195 separately
reported reasoning tokens; elapsed was 61.466 s, local CPU 11.031356 s and peak
806,146,048 B. Requested aliases do not establish the server-resolved model
identity, and remote model compute is unmeasured.
An independent fresh stdio process authenticated that task's 21-record,
three-revision checkpoint: 16,138,657 B, SHA-256
`a4395246473397cdc9040cb643e3175fd6f8b24db40597f8b680f0e065643df5`.
Nine calls reproduced all four complete original results and source records and
reexported byte-identical bundle bytes. This took 6.353 s elapsed, 6.060161 s CPU
and 592,621,568 B peak, with verified cleanup and no OOM.
Independent full-result/source/history authentication and manual answer review
supported the model's stated conclusions. No task or model retry was needed in
this cohort; separate checker and report-aggregation CPU was not measured.
The actual Ghidra healthy-engine control also passed on this runtime: 49 calls,
51.115 s elapsed, 50.000976 s CPU and 1,124,110,336 B peak. The verifier-owned JVM
PID/start identity/process group, public analysis run and MCP client were the same
across reset. Two explicit `address_name` queries returned identical Evidence;
no dossier, inventory or decompile producer was replayed. Full checkpoint/history
recovery and owned cleanup passed, with no OOM. This observes retained-engine
behavior rather than a before/after speedup.
Across all three model cohorts, nine actual runs plus one prelaunch failure used
2,771,744 input tokens including the 2,468,608 cached subset, 23,334 output tokens
and 1,035 separately reported reasoning tokens. Guarded elapsed totals 480.694 s
and local CPU 87.103671 s; maximum family peak is 922,148,864 B. All earlier
failed/partial attempts remain included. These are execution costs, not a paired
model-token savings claim.
All nine controlled success/error publication cases passed. Reinstating only the
old admission factory made both successful-publication cases fail at the original
admission boundary; the corrected source was restored and all nine passed again.
The final boundary run used 12.872 s elapsed, 12.647482 s CPU and 941,723,648 B
peak. Existing lifecycle/deferred/shutdown tests passed 79/79; the full HTTP suite
passed 74/74. These controlled seams establish ordering, separately from real
application behavior.
CI also exposed the 1500-file JavaScript timeout fixture's retention headroom.
At a 512 MiB parent heap, the original timeout exceeded the 10 MiB wire budget;
fallback authentication projected 771,325,184 B against 411,666,312 B available
and correctly refused. The transport/timeout fixture now uses a finite 1536 MiB
parent heap while retaining all 1500 files and every recovery/query assertion.
Its worker remains at 128 MiB; the separate memory-pressure case retains its
512 MiB parent. All three tests passed in 23.556 s elapsed, 22.989758 s CPU and
1,064,308,736 B peak, including a public assertion that retained records fit the
fixed allowance. No production limit was raised for this fixture.
## Earlier review qualification
Review corrections were qualified at clean
`6d875b3ff73d40b365a713cf4ba69a10cca257b7`, with runtime manifest SHA-256
`328f05dc2d7b114c33dc0406e488c3046a43e1c12187e15ad9263be65df7cbaa`.
SDK error Evidence retention and reply publication now share session admission;
reset retries the MCP artifact scope alongside process cleanup before saving or
clearing. Controlled public MCP regressions cover both publication orderings,
queued cancellation, terminal shutdown, ordinary export cleanup failure, mixed
cleanup outcomes and scope reuse. Reinstating the former wiring made four of the
five new regression cases fail; all five pass with the correction. Controlled
waits also use bounded acknowledgements and release/join held work in `finally`.
The nine genuine saved-source cases below were repeated against this runtime:
**313 MCP calls and 19 CLI checks**, 118.510 s guarded elapsed, 114.859544 s CPU,
and 1,065,390,080 B maximum kernel-family peak. Every complete original result,
ID, source, coverage and Unknown revision chain matched through close, checkpoint
collision refusal, reset, same-process import and fresh-process recovery. No
producer or target open was replayed in this cohort. Input/checkpoint/runtime
immutability and owned cleanup were verified, with no OOM. These normal workflows
complement the controlled error/cleanup timing regressions; they do not reproduce
those timing faults themselves.
One further local Codex CLI task requested `gpt-6.1-terra`/`high` with the
scoped four-family code-mode profile. Its 25 MCP calls included four expected
missing-reference controls and no unexpected model errors or producer replay.
Independent full-result/source/history authentication and answer review supported
the answer within its stated scope; it did not claim to query every other
Unknown. The run used 58.269 s elapsed, 9.837083 s local CPU and 885,448,704 B peak,
with 239,931 input tokens including 215,168 cached, 3,721 output tokens and 277
separately reported reasoning tokens.
A real fresh stdio client restored that same new checkpoint: 21 records and
three Unknown revisions in 16,138,749 B, SHA-256
`760fc74951abc292a3ff2fe1771391ac0d4345416b2345dfa4e85cbb6ce96181`.
Nine calls reproduced all four complete original results and source records,
then reexported byte-identical bundle bytes. This took 5.443 s elapsed,
5.253212 s CPU and 595,218,432 B peak, with verified cleanup and no OOM.
Two external-checker failures were preserved and corrected against actual event
ordinals and query filter/paging semantics; neither required model or backend
replay. Separate checker CPU and remote model compute were not measured.
Across those two earlier cohorts, eight actual model runs plus one prelaunch failure used
2,539,067 input tokens including the 2,269,056 cached subset, 19,721 output tokens,
and 840 separately reported reasoning tokens. Guarded elapsed totals 419.228 s
and local CPU 76.072315 s; maximum family peak remains 922,148,864 B. Initial
failed/partial attempts below remain included. This is execution accounting,
not a paired model-token performance comparison.
That runtime also repeated the actual Ghidra healthy-engine control:
49 calls, 44.333 s elapsed, 43.348651 s CPU and 1,093,464,064 B peak. The same
verifier-owned JVM PID/start identity/process group, public analysis run and MCP
client survived reset; two `address_name` queries returned identical Evidence.
Startup-inclusive and warm requests took 24,953.573 ms and 6.762 ms respectively,
without dossier, inventory or decompile replay. These are startup/reuse
observations rather than a PR before/after speedup. Full checkpoint/history
recovery and owned cleanup passed, with no OOM.
## Initial main-integration workflows
Main was integrated in `4e20698b7b9883f659e445c059c011aa0e5de834`, with runtime
manifest SHA-256
`ee23cbaf0ecbf8d896fb5dc7a88317e2ab8eb77bd3a4467bf5793c3471ca9c7d`.
Nine genuine saved-source cases covered eight target families:
| Family | Actual source | Saved-source bundle SHA-256 |
| ---------- | ---------------------- | ------------------------------------------------------------------ |
| Native | jq observations | `0a3adad6ab39bfe29320e1f070a8e598382d2f17c4d6c19079bd6ff16dfe0937` |
| JavaScript | agent-browser 0.27.0 | `66232b6eb01b3dbf435949a1cd94cb2532b7f989b91cdf952a105b3cee231bda` |
| Managed | NuGet CLI, SDK 8.0.427 | `11ac63842299c0ac641a320ec9b2a5161b454a8a322d57724fd5f2ffb2fddfa4` |
| Android | NewPipe 0.29.1 | `fa499d0464a40ddc2f5159c893d68b8f50aa8e6d2b0080de22f2f3e5fc326825` |
| Harmony | Shark 1.0.8 APP | `3a526ab0113f80ada26ff5a0631fc95dac91485946f20584fc5a76f5ee8eb880` |
| Harmony | Shark inner HAP | `3431c0a347f331c3effacf7984ecb890f3fef46d8b00b03b50980023b065cd4a` |
| Apple | Rectangle 2.0.4 | `8daa6286b28aadf5ba0d41038046805e5c69874eaa0bc5daf0e265360fcae79d` |
| WASM | tree-sitter-css | `acb0cb6c3e0d520f83144e106e171c3ecebba70c5c0498bbb0113185a3e4bf95` |
| Process | Controlled jq captures | `55da0f130dc2265aae54261d49949471e5c929163c2b7314b42e95df3325b739` |
Each case imported authenticated observations, composed exact retained references,
closed without an active target, refused an existing checkpoint destination,
saved and reset, checked missing references, and restored into the same process
and a fresh public stdio MCP process. Complete original public results, source
links, coverage and Evidence IDs matched throughout. Every case included an
explicit analyst question with revisions 1→2; complete revision histories and
mutation Evidence authenticated after recovery. Bundle hashes above identify
the saved observations rather than release binaries. The actual jq target was
`/usr/bin/jq`, SHA-256
`90b1eef4498f31836342898fc6e97cfeb8a197c4366773a0fb120ae02a45ea52`.
That cohort completed **313 MCP calls and 19 public CLI checks**, without
source producer replay. Summed guarded elapsed was 119.445 s and CPU was
115.040362 s; maximum kernel cgroup-family peak was 1,047,826,432 B. All pinned
input/runtime hashes stayed unchanged, owned cleanup was verified, and no resource
stop or OOM occurred.
A separate actual Ghidra 12.1.4 control reset while its jq engine remained open.
The same verifier-owned JVM PID/start identity/process group, reported analysis
run and MCP client survived reset. Two identical `address_name` queries at
`0x104500` returned the same Evidence. The first request including startup took
21,023.963 ms; the post-reset request took 6.672 ms. This observes startup versus
reuse, rather than a PR before/after speedup. No function dossiers or procedure
inventories were regenerated. The complete close/checkpoint/import/fresh-process
control used 49 calls, 39.351 s elapsed, 38.222523 s CPU and 1,094,340,608 B peak;
reset acknowledged 15 records and three Unknown revisions with `retained_target`.
Earlier actual-provider controls also passed Ghidra→Rectangle provider-family
switching and Rectangle→NewPipe target switching, with reset retaining the healthy
second target. Those pre-sync runs preserved their complete original results and
histories; they are separate from the latest-runtime qualification above.
## Equivalent completed-task measurements
These pairs used exact baseline `60149282fcfc05ac32fbc94434c107710120da6f` and
implementation `3c9fb7f44a4496d18d893beb903c5f2ba68c06b4`, before the later main
integration. The baseline exported, closed and imported before reference
composition; the candidate closed and reused references directly. Both returned
the same complete original result for the same authentic inputs.
| Saved source | MCP calls, before→after | Elapsed s | CPU s | Kernel-family peak B |
| ------------ | ----------------------- | ----------- | ----------------- | ----------------------- |
| Native | 9→7 | 5.846→4.938 | 5.609106→4.760475 | 582,672,384→511,090,688 |
| JavaScript | 8→6 | 3.529→3.333 | 3.371675→3.163932 | 394,170,368→387,428,352 |
| WASM | 7→5 | 4.236→3.731 | 4.035871→3.520731 | 658,063,360→492,548,096 |
Each row is one sequential pair in fresh isolated processes. OS caches were
uncontrolled; builds and fingerprinting were measured separately. These samples
establish two fewer necessary calls, rather than statistical speedups or model
token savings.
## Initial Codex tasks
Local Codex requested `gpt-6.1-terra` with `high` reasoning. Native used the full
advertised surface without an enabled-tool filter or exact-name guide, both
before and after the latest main integration. JavaScript, packages and the mixed
managed/WASM/process/Harmony task used explicit scoped diagnostic profiles. The
mixed task used the existing experimental client code mode; its development
warning was retained.
All four task scopes passed independent canonical bundle/history authentication,
complete original-result comparison and final-answer review. These conclusions
were checked separately from process exit and token accounting. Native's final
12 calls preserved the original comparison, acknowledged a 14-record/three-revision
checkpoint, observed the expected missing-reference control, and restored both
Unknown heads. The mixed model used 25 calls; a separate latest-runtime fresh
stdio verifier used nine calls to restore its same 21-record/three-revision
checkpoint, reproduce all four complete originals, and reexport byte-identical
bundle bytes.
The initial cohort table includes **all attempts**, partial runs and the native task repeated
after main integration:
| Task/profile | Input tokens (cached subset) | Output tokens | Elapsed s / CPU s | Kernel-family peak B |
| ----------------------------------- | ---------------------------- | ------------- | ------------------- | -------------------- |
| Native, full discovery | 841,133 (769,536) | 4,326 | 101.845 / 18.383645 | 888,786,944 |
| JavaScript, scoped | 531,546 (452,224) | 3,421 | 88.236 / 14.456771 | 922,148,864 |
| Rectangle→NewPipe, scoped | 629,186 (567,424) | 4,384 | 108.062 / 22.589319 | 878,800,896 |
| Four-family mixed, scoped code mode | 297,271 (264,704) | 3,869 | 62.816 / 10.805497 | 914,604,032 |
Seven actual model runs plus one prelaunch failure consumed 2,299,136 input tokens
including the 2,053,888 cached subset, and 16,000 output tokens. Input already
includes cached input. Reported reasoning output was 563 tokens, recorded
separately. Canonical verification costs are separate from client/model costs.
The latest SDK catalog had 234 tools and 3,919,838 JSON bytes; those representation
bytes are not model tokens. No paired baseline model cohort ran, so these counters
do not establish causal whole-task token savings.
Three failed or partial attempts were preserved:
- Native's first attempt stopped before model/MCP launch because of a redundant
template-only harness argument. Its 0.718 s/0.473821 CPU s remains included.
- JavaScript saved its six-record/two-revision checkpoint before the isolated
client denied `binary_session`. The harness approval was corrected; a new
process restored the same checkpoint and identical trace without another
producer or reset. Joint accounting includes 16 client attempts, 15 served
calls and the client denial.
- The package model selected an unsupported generic view for a derived Apple
projection and stopped before checkpointing. Its continuation rehydrated
original authenticated saved inputs after matching current IDs and observed
summaries/projections, then used the correct projectors. Two inventory producers
ran once total; one additional NewPipe metadata reopen restored active state.
This qualifies explicit saved-source recovery. The first run's unsaved full
inventory body was not directly transported or checkpointed.
## Resource and regression boundaries
The genuine multi-domain/model workflows used `--max-old-space-size=512` for
MCP/CLI Node children; retention had no quota override. The controlled timeout
fixture's separate parent allowance is documented above. An earlier cumulative
default-policy control admitted all nine genuine
bundles: 34 records, 87,532,230 estimated representation bytes against its observed
150,244,580-byte allowance. Exact duplicates did not grow committed state. This
corpus did not reach default capacity; injected-policy regressions establish
atomic refusal, metadata replacement accounting and budget release on reset.
Local checks passed across 43 focused source/public-boundary files, 533 distinct
tests using each file's latest passing run, and 93 Python tests. The main
integration specifically reran eight lifecycle/recovery files with 61 passing
tests, including retained candidate cleanup failure/recovery and healthy-client
preservation. Typecheck, bounded lint, formatting, module/architecture guards,
documentation and production reachability passed. Controlled providers and
failure seams qualify cancellation, late work and cleanup orderings separately
from actual-provider observations.
Heavy tasks ran serially with one CPU, 4 GiB hard memory, 3 GiB early stop, zero
swap and 192 tasks maximum. Kernel CPU and memory.peak were observed at payload
exit while a cgroup observer remained alive. They include drivers/helpers and the
observer, exclude preflight outside the guard, and are distinct from process RSS
and retained-representation estimates. Failed model attempts, partial results,
assessments and raw receipts remain local; binaries and authentication data are
excluded from the repository.
Explicit acknowledged checkpoints qualify cross-process recovery. Uncheckpointed
process loss, automatic journaling, parent-directory power-loss recovery,
Windows, IDA/Hopper and a second actual deep engine were not qualified here.
+9 -5
View File
@@ -189,11 +189,13 @@ typed interfaces.
`compare_wasm_interfaces` accepts two complete `inspect_wasm_artifact` Evidence
records, or exact retained references in the current MCP session, and required
`scope: "imports" | "exports" | "both"`. It reads no files and needs no active
WABT toolchain. CLI uses portable inline Evidence in one JSON document:
`scope: "imports" | "exports" | "both"`. It uses producer-owned typed facts
without rereading the artifact and needs no active WABT toolchain. CLI accepts
portable inline Evidence in one JSON document, or
resolves the same retained-reference inputs from an explicit saved bundle:
```bash
rea compare-wasm-interfaces comparison-input.json --json
rea compare-wasm-interfaces comparison-input.json --evidence-bundle /absolute/path/evidence.json --json
```
```json
@@ -204,8 +206,10 @@ rea compare-wasm-interfaces comparison-input.json --json
}
```
The references above are MCP-only; put complete Evidence objects in the CLI
input. Resolve summary delivery to its retained complete parent before comparing.
Omit `--evidence-bundle` when the CLI input contains complete inline records.
Resolve summary delivery to its complete `normalized_result.parent_evidence_id`
before comparing. References preserve that exact source identity and coverage;
the bundle supplies no facts absent from the selected record.
Results retain both input Evidence IDs, artifact paths/digests/lengths and every
selected occurrence. Unique names are added, removed, changed or unchanged;
duplicate names and unsupported signatures are unknown. Incomplete or historical
+1
View File
@@ -9,6 +9,7 @@ upstream texts; they do not change [REA's MIT license](../LICENSE).
| `npm-dependencies.txt` | Every package path in the root lockfile, with its exact version, declared license, scope, download URL, and integrity. Includes development and optional platform entries. |
| `npm-runtime-notices.txt` | Original license, copyright, and NOTICE texts collected from integrity-verified locked runtime npm archives across all platforms; also the license of Rolldown's generated runtime helpers. The archive manifest's name and version are checked. |
| `protobuf-es-v2.16.0-LICENSE.txt` | Original Apache license omitted from the protobuf-es npm archive, preserved from its v2.16.0 source commit. Its embedded Google BSD notice is retained in `npm-runtime-notices.txt`. |
| `microsoft-uf2-MIT.txt` | Unchanged `LICENSE.txt` from microsoft/uf2 revision `cf3bd7f9c274b949528f8cd6a19eef2502a687d2`, for the UF2 family names copied from its `utils/uf2families.json`. |
| `nativeaot-parser-MIT.txt` | Original Washi notice for derived NativeAOT parser code; source revision is recorded in the parser and the root notices. |
| `dotnet-runtime-MIT.txt` | Original .NET Foundation MIT text from the NativeAOT reference revision `a3b2d40328be0be3f8dd43f0e5cc925b8827b044`. |
| `ghidra-12.1.4-{LICENSE,NOTICE,DISCLAIMER}.txt` | Unchanged top-level documents from `Ghidra_12.1.4_build`. Ghidra is not bundled; these reference copies do not replace its component and GPL support-program notices. The US-government-specific disclaimer applies to Ghidra, not to REA. |
+25
View File
@@ -0,0 +1,25 @@
Microsoft UF2
The MIT License (MIT)
Copyright (c) Microsoft Corporation
All rights reserved.
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
+5
View File
@@ -78,5 +78,10 @@
"file": "licenses/pefile-2024.8.26-LICENSE.txt",
"source": "https://raw.githubusercontent.com/erocarrera/pefile/4b3b1e2e568a88d4f1897d694d684f23d9e270c4/LICENSE",
"sha256": "b44409c067c6da52bfb54bb6624fb11a7b52157c3a13ae1400232f8196e86ad3"
},
{
"file": "licenses/microsoft-uf2-MIT.txt",
"source": "https://raw.githubusercontent.com/microsoft/uf2/cf3bd7f9c274b949528f8cd6a19eef2502a687d2/LICENSE.txt",
"sha256": "c581accf1e5449482b70ac5fcdfb171599317ccff559cf0794d771671486f800"
}
]
+4 -1
View File
@@ -65,6 +65,7 @@
"bridge/ghidra/ReaGhidraAnalysisOptions.java",
"bridge/ghidra/ReaGhidraNoReturnFix.java",
"bridge/ghidra/ReaGhidraMipsSoftFloat.java",
"bridge/ghidra/ReaGhidraRiscvCallingConvention.java",
"bridge/ghidra/ReaGhidraPrepareRaw.java",
"bridge/ghidra/ReaGhidraPrepareDol.java",
"bridge/android/ReaJadxBridge.java",
@@ -238,9 +239,11 @@
"verify:javascript:retained-facts": "npm run build:cached && node scripts/verify/javascript/retained-facts.mjs",
"verify:application-workflows": "npm run build:cached && node scripts/verify/javascript/application-workflows.mjs",
"verify:managed": "npm run build:cached && node scripts/verify/managed/conformance.mjs",
"verify:uf2": "npm run build:cached && node scripts/verify/firmware/uf2-esptool.mjs",
"verify:firmware": "npm run build:cached && node scripts/verify/firmware/analysis.mjs",
"verify:devicetree": "npm run build:cached && node scripts/verify/firmware/devicetree.mjs",
"verify:esp-images": "npm run build:cached && node scripts/verify/firmware/esp-image.mjs",
"verify:esp-flash": "npm run build:cached && node scripts/verify/firmware/esp-flash.mjs",
"fixtures:firmware": "node scripts/fixtures/firmware/generate.mjs",
"verify:android": "npm run build:cached && node scripts/verify/android/analysis.mjs",
"verify:jeb": "npm run build:cached && node scripts/verify-jeb.mjs",
@@ -288,7 +291,7 @@
"verify:native-ui:permissions": "npm run build:cached && node scripts/verify-native-ui.mjs --allow-permission-denial",
"test:focused": "node scripts/run-development-tests.mjs focused",
"test:sqlite": "npm run test:prepare && npm run test:sqlite:run",
"test:sqlite:run": "vitest run src/application/sqlite/ tests/boundary/sqlite/ tests/acceptance/sqliteDatabaseCli.test.ts tests/boundary/mcp/serverLifecycleMcp.test.ts",
"test:sqlite:run": "node scripts/run-exact-tests.mjs run tests/acceptance/sqliteDatabaseCli.test.ts tests/boundary/mcp/serverLifecycleMcp.test.ts && vitest run src/application/sqlite/ tests/boundary/sqlite/",
"build:windows-native": "node scripts/build-windows-native.mjs",
"verify:windows-native:artifact": "node scripts/verify-windows-native-artifact.mjs",
"prepublishOnly": "npm run verify:windows-native:artifact",
+4 -1
View File
@@ -12,6 +12,7 @@ import { planTestSelection } from "./test-selection.mjs";
// Actionlint validates workflow syntax only. The semantic suite asserts these
// workflows' release, publication and self-hosted runner safeguards.
const GUARDED_WORKFLOWS = new Set([
".github/workflows/codeql.yml",
".github/workflows/pr-title.yml",
".github/workflows/pr-readiness.yml",
".github/workflows/ci-full-pr.yml",
@@ -275,10 +276,12 @@ if (tier === "reused") {
for (const owner of owners) selected[owner] = true;
if (
file.startsWith(".github/workflows/") ||
file.startsWith(".github/codeql/") ||
file === ".github/actionlint.yaml"
) {
selected.workflows = true;
if (GUARDED_WORKFLOWS.has(file)) selected.code = true;
if (GUARDED_WORKFLOWS.has(file) || file.startsWith(".github/codeql/"))
selected.code = true;
if (file !== ".github/workflows/ci.yml") continue;
}
// The Notes example is also the source-owned agent evaluation fixture.
+3
View File
@@ -19,6 +19,9 @@ const descriptors = [
);
const requirements = {
packet: {
"tests/acceptance/pcapVerifier.test.ts": [
"verifies complete and unresolved HTTP coverage with real Wireshark tools",
],
"tests/boundary/captures/resourceLimits.test.ts": [
...resources,
"reports a real kernel file-limit termination with its effective ceiling",
+24
View File
@@ -28,6 +28,8 @@ const PATH_SCOPES = {
"tests/boundary/cli/captureCleanup.test.ts",
"tests/boundary/mcp/packetCaptureMcp.test.ts",
"tests/boundary/pcap/**",
"tests/acceptance/pcapVerifier.test.ts",
"tests/fixtures/pcap/offlineHttpCapture.ts",
"src/captures/**",
"src/pcap/**",
"src/domain/pcap/**",
@@ -43,6 +45,7 @@ const PATH_SCOPES = {
"src/server/registerCaptureTools.ts",
...CAPTURE_SHARED_INPUTS,
"scripts/verify-packet-capture.mjs",
"scripts/verify-pcap.mjs",
".github/workflows/real-packet-capture.yml",
],
signal_recording: [
@@ -125,6 +128,12 @@ const PATH_SCOPES = {
],
// Apple artifact CLI/MCP lane on macOS, including its focused regressions.
apple: [
"src/artifacts/InesArtifactReader.ts",
"src/domain/inesContainer.ts",
"tests/fixtures/ines.ts",
"tests/acceptance/inesContainer.test.ts",
"tests/boundary/filesystem/inesArtifact.test.ts",
"tests/boundary/filesystem/inesCancellation.test.ts",
"src/hopper/BridgeLauncher.ts",
"src/domain/hopperErrors.ts",
"tests/boundary/providers/hopper/hopperMacLaunchArguments.test.ts",
@@ -152,6 +161,7 @@ const PATH_SCOPES = {
"tests/boundary/cli/nativeDemangleCli.test.ts",
"tests/boundary/cli/nativeDispatchPropertyAttributes.test.ts",
"tests/boundary/mcp/nativeDemangleMcp.test.ts",
"tests/boundary/mcp/nativeMachoSummaryMcp.test.ts",
"tests/boundary/providers/native/nativeSignatureTargetBinding.test.ts",
"tests/boundary/cli/nativeSignatureCli.test.ts",
"tests/boundary/cli/nativeMachoArm64_32Cli.test.ts",
@@ -344,20 +354,30 @@ const PATH_SCOPES = {
".github/workflows/real-browser.yml",
"scripts/verify-real-browser.mjs",
"scripts/verify-browser-network.mjs",
"scripts/verify-browser-request.mjs",
"scripts/verify-browser-scripts.mjs",
"scripts/verify-browser-modules.mjs",
"scripts/verify-browser-dom-destinations.mjs",
"scripts/lib/browser-*.mjs",
"scripts/fixtures/browser-network-site.mjs",
"scripts/fixtures/browser-request-site.mjs",
"scripts/fixtures/browser-script-site.mjs",
"scripts/fixtures/browser-module-site.mjs",
"scripts/fixtures/browser-verifier-site.mjs",
"scripts/fixtures/page-cdp-proxy.mjs",
"src/browser/**",
"src/domain/browser*.ts",
"src/domain/webCaptureDiffSchemas.ts",
"src/domain/webScreenshot.ts",
"src/domain/webScript*.ts",
"src/domain/webModule*.ts",
"src/domain/analysisView/environment*.ts",
"src/domain/analysisView/{EnvironmentManifest*,EnvironmentState*}.ts",
"src/domain/analysisView/{scenarioCaptureRecords,recordedResponseBody,recordedExchangeBody,recordedMediaType,webNetworkRouteTemplates,WebNetworkProjectionAccumulator}.ts",
"src/application/BrowserScenario*.ts",
"src/application/environment/**",
"src/application/WebModule*.ts",
"src/composition/browserScenario.ts",
"src/composition/webModules.ts",
"src/artifacts/readStableArtifact.ts",
"src/artifacts/ArtifactPaths.ts",
@@ -367,6 +387,8 @@ const PATH_SCOPES = {
"src/server/registerWebModuleTool.ts",
"src/server/createServer.ts",
"src/cli/webModuleCommands.ts",
"src/cliBrowserScenarioCommands.ts",
"src/cli/environmentCommands.ts",
"src/cli.ts",
"src/cliCommandNames.ts",
"src/contracts/toolEffects.ts",
@@ -557,6 +579,8 @@ const PATH_SCOPES = {
".github/workflows/real-web-source-map.yml",
"scripts/verify-browser-source-maps.mjs",
"scripts/lib/browser-source-map-e2e.mjs",
"scripts/lib/browser-nested-source-map-e2e.mjs",
"scripts/lib/browser-runtime-fixture-lifecycle.mjs",
"scripts/fixtures/browser-source-map-site.mjs",
"src/javascript/sourceMaps/**",
"src/browser/assets/**",
+54
View File
@@ -28,6 +28,7 @@ const setAdd = bindBuiltinMethod(Set.prototype.add);
const hasOwnProperty = bindBuiltinMethod(Object.prototype.hasOwnProperty);
const { isProxy } = require("node:util").types;
const { createHash } = require("node:crypto");
const { EventEmitter: NodeEventEmitter } = require("node:events");
const nodeEventEmitterMethods = Object.create(null);
for (const method of [
@@ -1241,6 +1242,59 @@ const boundaries = createElectronActiveBoundaryPatches({
shape,
});
// Admit only this provider's exact launch tail and Playwright's owned prefix.
// Caller arguments remain opaque, including repeated hook paths and debug flags.
const restoreApplicationArgv = () => {
const handoff = process.env.REA_ELECTRON_CAPTURE_ARGV;
if (handoff === undefined) return;
delete process.env.REA_ELECTRON_CAPTURE_ARGV;
const plan = JSON.parse(handoff);
if (
plan === null ||
typeof plan !== "object" ||
!Number.isSafeInteger(plan.length) ||
plan.length < 3 ||
typeof plan.sha256 !== "string" ||
!/^[a-f0-9]{64}$/u.test(plan.sha256)
)
throw new Error("Capture argument handoff is not this hook's launch plan");
const tailIndex = process.argv.length - plan.length;
const prefixIndex = process.argv[1] === "--no-sandbox" ? 2 : 1;
if (
tailIndex !== prefixIndex + 2 ||
process.argv[prefixIndex] !== "--inspect=0" ||
process.argv[prefixIndex + 1] !== "--remote-debugging-port=0" ||
process.argv[tailIndex] !== "-r" ||
process.argv[tailIndex + 1] !== __filename ||
typeof process.argv[tailIndex + 2] !== "string" ||
process.argv[tailIndex + 2].length === 0
)
throw new Error("Observed argv does not match Playwright's capture prefix");
const tail = sliceArguments(process.argv, tailIndex);
if (
createHash("sha256").update(JSON.stringify(tail)).digest("hex") !==
plan.sha256
)
throw new Error(
"Observed argv does not match the capture argument handoff",
);
for (let index = 2; index < tail.length; index++)
process.argv[index - 1] = tail[index];
process.argv.length = tail.length - 1;
};
try {
restoreApplicationArgv();
} catch (cause) {
globalThis.__reaElectronActiveHookError = true;
recordRuntime(
"error",
cause instanceof Error ? cause.message : String(cause),
"failed",
{ target: "active-hook-arguments", error: true },
);
}
try {
const electron = require("electron");
patchEmitter(electron.app, (event, args) =>
@@ -94,6 +94,12 @@ export async function startBrowserRequestSite() {
writes.push({
method: request.method,
path: url.pathname,
query: url.search,
body,
contentType: request.headers["content-type"],
cookie:
request.headers.cookie?.includes(`csrftoken=${requestFixtureCsrf}`) ??
false,
csrf: request.headers["x-csrftoken"] === requestFixtureCsrf,
bearer:
request.headers.authorization === `Bearer ${requestFixtureToken}`,
+55 -10
View File
@@ -17,17 +17,28 @@ import { compareBrowserScenarios } from "../../dist/domain/browserScenarioDiff.j
import { browserScenarioCaptureSchema } from "../../dist/domain/browserScenarioCapture.js";
import { parseEvidence } from "../../dist/domain/evidence.js";
// Readiness includes descendant document loads, so complete network capture
// does not depend on whether an inline script runs before an iframe finishes.
const document = (path) => {
const icon = '<link rel="icon" href="data:,">';
if (path.startsWith("/nested"))
return `<script>throw new Error("nested popup failure")</script>
<script>opener.postMessage("nested-ready", "*")</script>`;
return `${icon}<script>throw new Error("nested popup failure")</script>
<script>addEventListener("load", () => opener.postMessage("nested-ready", "*"))</script>`;
if (path.startsWith("/popup"))
return `<script>throw new Error("popup failure")</script>
<script>onmessage = e => {
if (e.data === "nested-ready") opener.postMessage("popup-ready", "*");
}; window.open("/nested");</script><iframe src="/child"></iframe>`;
if (path.startsWith("/child")) return "child frame";
return `<script>onmessage = e => {
return `${icon}<script>throw new Error("popup failure")</script>
<script>
let loaded = false, nested = false;
const ready = () => {
if (loaded && nested) opener.postMessage("popup-ready", "*");
};
addEventListener("load", () => { loaded = true; ready(); });
onmessage = e => {
if (e.data === "nested-ready") { nested = true; ready(); }
};
window.open("/nested");
</script><iframe src="/child"></iframe>`;
if (path.startsWith("/child")) return `${icon}child frame`;
return `${icon}<script>onmessage = e => {
if (e.data === "popup-ready") document.body.dataset.ready = "true";
}</script><button onclick="window.open('/redirect')">Open popup</button>`;
};
@@ -75,7 +86,25 @@ const assertNetwork = (result, origin) => {
1,
`${kind} ${path} must be captured exactly once`,
);
assert.equal(result.completeness.equality_eligible, true);
assert.equal(
result.completeness.equality_eligible,
true,
JSON.stringify({
limitations: result.limitations,
unfinished: result.events.items.filter(
(event) => event.kind === "request-unfinished",
),
}),
);
assert.equal(
result.events.items.filter(
(event) =>
event.kind === "request-finished" &&
event.url?.url === origin + "/child",
).length,
1,
"child document must finish before network capture ends",
);
assert.equal(compareWithSelf(result).overall_status, "unchanged");
};
@@ -180,13 +209,27 @@ const verifyTransports = async (executable, origin) => {
/** Verify popup timing, selection, coverage, and shared-context isolation on Chromium. */
export async function verifyPopupEventCoverage(executable) {
let slowChild = true;
let childTimer;
const childStarted = Promise.withResolvers();
const server = createServer((request, response) => {
if (request.url === "/redirect") {
response.writeHead(302, { Location: "/popup" });
response.end();
} else {
response.setHeader("Content-Type", "text/html");
response.end(document(request.url ?? "/"));
if (slowChild && request.url === "/child") {
// Keep one response open while the nested page can announce readiness.
// This exposes premature capture cutoff even on fast local browsers.
response.writeHead(200);
response.write(document("/child"));
childStarted.resolve();
childTimer = setTimeout(() => response.end(), 1500);
} else if (slowChild && request.url === "/nested") {
void childStarted.promise.then(() => response.end(document("/nested")));
} else {
response.end(document(request.url ?? "/"));
}
}
});
server.listen(0, "127.0.0.1");
@@ -199,6 +242,7 @@ export async function verifyPopupEventCoverage(executable) {
await capture(scenarioFor(executable, origin, ["network"])),
origin,
);
slowChild = false;
const errors = await capture(
scenarioFor(executable, origin, ["page-errors"]),
);
@@ -240,6 +284,7 @@ export async function verifyPopupEventCoverage(executable) {
stdio_mcp: true,
};
} finally {
clearTimeout(childTimer);
server.closeAllConnections();
await new Promise((resolve, reject) =>
server.close((error) =>
+17 -4
View File
@@ -67,10 +67,13 @@ const writeSession = (executable, origin) =>
read("list-1", "/api/items"),
read("list-2", "/api/items"),
read("item-1", "/api/items/1"),
write("create-1", "POST", "/api/items", {
json: { title: "Scratch item 1" },
status: 201,
}),
{
...write("create-1", "POST", "/api/items", {
json: { title: "Scratch item 1" },
status: 201,
}),
query: [{ name: "org", value: { source: "literal", value: "1" } }],
},
read("created-1", "/api/items/3"),
read("list-created-1", "/api/items"),
write("create-2", "POST", "/api/items", {
@@ -337,6 +340,16 @@ export async function verifyBrowserRequestSteps(
assertRequestsAttributed(scenario, capture);
assert.equal(site.writes.length, 6);
assert.ok(site.writes.every(({ csrf, bearer }) => csrf && bearer));
assert.deepEqual(site.writes[0], {
method: "POST",
path: "/api/items",
query: "?org=1",
body: { title: "Scratch item 1" },
contentType: "application/json",
cookie: true,
csrf: true,
bearer: true,
});
assertPageValuesRedacted(evidence, capture);
const capturePath = join(root.path, "items.capture.json");
@@ -11,6 +11,8 @@ import { cleanupOwnedProcessGroup } from "../../dist/process/ProcessOwnership.js
import { ArtifactResourceScope } from "../../dist/artifacts/ArtifactResourceScope.js";
import { AnalysisCleanupLease } from "../../dist/artifacts/AnalysisCleanupLease.js";
import { withVerifierCleanup } from "./verifier-cleanup.mjs";
import { discoverCdpEndpoint } from "../../dist/browser/CdpEndpoint.js";
import { BrowserObservationError } from "../../dist/domain/browserObservationError.js";
import { CdpConnection } from "../../dist/browser/CdpConnection.js";
import { importVerifierSource } from "./verifier-source-loader.mjs";
@@ -26,6 +28,7 @@ const { BROWSER_DEVTOOLS_STARTUP_TIMEOUT_MS, waitForBrowserDevtoolsPort } =
export async function startRuntimeFixtureBrowser(executable, initialUrl) {
await requireOrdinaryWindowsVerifierCaller();
const profile = await mkdtemp(join(tmpdir(), "rea-real-web-runtime-"));
const deadline = performance.now() + BROWSER_DEVTOOLS_STARTUP_TIMEOUT_MS;
let supervisor;
let stopOwnedBrowser;
const resources = new ArtifactResourceScope();
@@ -94,7 +97,13 @@ export async function startRuntimeFixtureBrowser(executable, initialUrl) {
stderr: () => supervisor.snapshot().stderr.text,
// Match the broader real-browser verifier's allowance for cold Chrome
// startup on shared runners; early process failures still reject at once.
timeoutMs: BROWSER_DEVTOOLS_STARTUP_TIMEOUT_MS,
timeoutMs: Math.max(1, deadline - performance.now()),
});
await waitForRuntimeFixturePage({
child: owned.process,
port,
origin: initialUrl,
deadline,
});
return { port, profile, process: owned.process, close };
} catch (cause) {
@@ -104,6 +113,67 @@ export async function startRuntimeFixtureBrowser(executable, initialUrl) {
}
}
async function waitForRuntimeFixturePage({ child, port, origin, deadline }) {
const controller = new AbortController();
let lastFailure;
const onExit = () =>
controller.abort(
new Error(
"Owned runtime fixture exited before CDP discovery became ready",
{ cause: lastFailure },
),
);
const onError = (cause) => controller.abort(cause);
const timer = setTimeout(
() =>
controller.abort(
new Error(
"Owned runtime fixture page was not discoverable within its startup budget",
{ cause: lastFailure },
),
),
Math.max(1, deadline - performance.now()),
);
child.once("exit", onExit);
child.once("error", onError);
try {
for (;;) {
if (child.exitCode !== null || child.signalCode !== null) onExit();
controller.signal.throwIfAborted();
try {
const discovery = await discoverCdpEndpoint(
`http://127.0.0.1:${port}`,
"list_browser_targets",
controller.signal,
);
if (
discovery.targets.some(
(target) =>
target.type === "page" &&
(target.url === origin || target.url.startsWith(`${origin}/`)),
)
)
return;
} catch (cause) {
controller.signal.throwIfAborted();
// Only transient startup failures are retryable. Invalid protocol
// responses and payload-limit failures must remain visible immediately.
if (
!(cause instanceof BrowserObservationError) ||
!["timeout", "endpoint_unreachable"].includes(cause.reason)
)
throw cause;
lastFailure = cause;
}
await new Promise((resolve) => setTimeout(resolve, 25));
}
} finally {
clearTimeout(timer);
child.removeListener("exit", onExit);
child.removeListener("error", onError);
}
}
function createOwnedBrowserStop(supervisor, profile) {
let connection;
return async () => {
+57
View File
@@ -0,0 +1,57 @@
interface ProviderBinding {
readonly identity: {
readonly id: string;
readonly name: string;
readonly version: string | null;
};
readonly operations: readonly string[] | { readonly group: string };
}
interface ProviderCatalogSources {
readonly toolContracts: {
readonly TOOL_CONTRACTS: readonly { readonly name: string }[];
};
readonly toolFamilies: {
readonly TOOL_CONTRACT_GROUPS: Readonly<
Record<string, readonly { readonly name: string }[]>
>;
};
readonly capabilityDeclarations: {
readonly CAPABILITY_DECLARATIONS: Readonly<
Record<string, { readonly providers: readonly ProviderBinding[] }>
>;
};
}
export function providerCatalog(sources: ProviderCatalogSources): {
readonly id: string;
readonly name: string;
readonly version: string | null;
readonly capabilities: readonly string[];
}[];
export function toolFamilyCatalog(sources: {
readonly toolFamilies: {
readonly TOOL_FAMILY_DEFINITIONS: readonly {
readonly id: string;
readonly surface: string;
readonly contracts: readonly { readonly name: string }[];
}[];
};
readonly toolContracts: {
readonly TOOL_CONTRACTS: readonly unknown[];
};
readonly catalogIdentity: {
readonly FULL_CATALOG_IDENTITY: {
readonly counts: { readonly mcp_tools: number };
};
};
}): {
readonly total: number;
readonly families: readonly {
readonly id: string;
readonly surface: string;
readonly count: number;
readonly tools: readonly string[];
}[];
};
+44 -403
View File
@@ -23,414 +23,55 @@ export const toolFamilyCatalog = (sources) => {
return { total: familyTotal, families };
};
/** Build provider identities with their sorted capability names. */
/** Derive provider capabilities from the same static declarations used by composition. */
export const providerCatalog = (sources) => {
const contractsByName = new Map(
sources.toolContracts.TOOL_CONTRACTS.map((contract) => [
contract.name,
contract,
]),
const contractNames = new Set(
sources.toolContracts.TOOL_CONTRACTS.map(({ name }) => name),
);
const declaredContracts = (providerId, operations) =>
operations.map((operation) => {
const contract = contractsByName.get(operation);
if (contract === undefined)
const providers = new Map();
for (const declaration of Object.values(
sources.capabilityDeclarations.CAPABILITY_DECLARATIONS,
)) {
for (const { identity, operations } of declaration.providers) {
let provider = providers.get(identity.id);
if (provider === undefined) {
provider = { ...identity, capabilities: new Set() };
providers.set(identity.id, provider);
} else if (
provider.name !== identity.name ||
provider.version !== identity.version
) {
throw new TypeError(
`Provider ${providerId} declares an unknown analyst operation: ${operation}`,
`Provider ${identity.id} has conflicting source identities`,
);
return contract;
});
const {
applicationContracts,
reconciliationContracts,
observationContracts,
activeContracts,
fuseContracts,
} = electronContractSlices(sources);
return [
{
identity: sources.cutterProvider.CUTTER_PROVIDER_IDENTITY,
contracts: sources.cutterContracts.CUTTER_TOOL_CONTRACTS,
},
{
identity: sources.objdumpProvider.OBJDUMP_PROVIDER_IDENTITY,
contracts:
sources.reverseEngineeringContracts.REVERSE_ENGINEERING_TOOL_CONTRACTS.filter(
({ name }) => name === "inspect_with_objdump",
),
},
{
identity: sources.rizinProvider.RIZIN_PROVIDER_IDENTITY,
contracts:
sources.reverseEngineeringContracts.REVERSE_ENGINEERING_TOOL_CONTRACTS.filter(
({ name }) => name === "execute_rizin_command",
),
},
{
identity: sources.gdbProvider.GDB_PROVIDER_IDENTITY,
contracts: sources.gdbContracts.GDB_TOOL_CONTRACTS,
},
{
identity: sources.rizinDebugProvider.RIZIN_DEBUGGER_PROVIDER_IDENTITY,
contracts: sources.rizinDebugContracts.RIZIN_DEBUG_TOOL_CONTRACTS,
},
{
identity: sources.fridaProvider.FRIDA_PROVIDER_IDENTITY,
contracts: sources.fridaContracts.FRIDA_TOOL_CONTRACTS,
},
{
identity: sources.hopperProvider.HOPPER_PROVIDER_IDENTITY,
contracts: declaredContracts(
"hopper",
sources.hopperProvider.HOPPER_OPERATIONS,
}
let selectedOperations = operations;
if (!Array.isArray(operations)) {
const contracts =
sources.toolFamilies.TOOL_CONTRACT_GROUPS[operations.group];
if (contracts === undefined)
throw new TypeError(
`Provider ${identity.id} declares an unknown contract group: ${operations.group}`,
);
selectedOperations = contracts.map(({ name }) => name);
}
for (const operation of selectedOperations) {
if (!contractNames.has(operation))
throw new TypeError(
`Provider ${identity.id} declares an unknown analyst operation: ${operation}`,
);
provider.capabilities.add(operation);
}
}
}
return [...providers.values()]
.map(({ id, name, version, capabilities }) => ({
id,
name,
version,
capabilities: [...capabilities].sort((left, right) =>
left.localeCompare(right),
),
},
{
identity: sources.ghidraProvider.GHIDRA_PROVIDER_IDENTITY,
contracts: declaredContracts(
"ghidra",
sources.ghidraProvider.GHIDRA_OPERATIONS,
),
},
{
identity: sources.idaProvider.IDA_PROVIDER_IDENTITY,
contracts: declaredContracts("ida", sources.idaProvider.IDA_OPERATIONS),
},
{
identity: sources.kunaProvider.KUNA_PROVIDER_IDENTITY,
contracts: declaredContracts(
"kuna",
sources.kunaProvider.KUNA_OPERATIONS,
),
},
{
identity: sources.nativeProvider.NATIVE_MACOS_PROVIDER_IDENTITY,
contracts: sources.nativeContracts.NATIVE_TOOL_CONTRACTS,
},
{
identity: sources.pwntoolsProvider.PWNTOOLS_PROVIDER_IDENTITY,
contracts:
sources.binaryDiagnosticsContracts.BINARY_DIAGNOSTICS_TOOL_CONTRACTS.filter(
({ name }) =>
name === "inspect_binary_layout" ||
name === "inspect_recorded_crash",
),
},
{
identity: sources.artifactProviders.PE_RESOURCES_PROVIDER,
contracts:
sources.binaryDiagnosticsContracts.BINARY_DIAGNOSTICS_TOOL_CONTRACTS.filter(
({ name }) => name === "inspect_pe_resources",
),
},
{
identity: sources.artifactProviders.PE_CERTIFICATES_PROVIDER,
contracts:
sources.binaryDiagnosticsContracts.BINARY_DIAGNOSTICS_TOOL_CONTRACTS.filter(
({ name }) => name === "inspect_pe_certificates",
),
},
{
identity: sources.artifactProviders.PE_IMPORTS_PROVIDER,
contracts:
sources.binaryDiagnosticsContracts.BINARY_DIAGNOSTICS_TOOL_CONTRACTS.filter(
({ name }) => name === "inspect_pe_imports",
),
},
{
identity: sources.artifactProviders.DEVICETREE_PROVIDER,
contracts:
sources.binaryDiagnosticsContracts.BINARY_DIAGNOSTICS_TOOL_CONTRACTS.filter(
({ name }) => name === "inspect_devicetree",
),
},
{
identity: sources.artifactProviders.PE_LOAD_CONFIG_PROVIDER,
contracts:
sources.binaryDiagnosticsContracts.BINARY_DIAGNOSTICS_TOOL_CONTRACTS.filter(
({ name }) => name === "inspect_pe_load_config",
),
},
{
identity: sources.artifactProviders.ELF_NOTES_PROVIDER,
contracts:
sources.binaryDiagnosticsContracts.BINARY_DIAGNOSTICS_TOOL_CONTRACTS.filter(
({ name }) => name === "inspect_elf_notes",
),
},
{
identity: sources.artifactProviders.APPLE_IPS_PROVIDER,
contracts:
sources.binaryDiagnosticsContracts.BINARY_DIAGNOSTICS_TOOL_CONTRACTS.filter(
({ name }) => name === "inspect_apple_ips",
),
},
{
identity: sources.artifactProviders.KERNEL_MODULE_PROVIDER,
contracts:
sources.binaryDiagnosticsContracts.BINARY_DIAGNOSTICS_TOOL_CONTRACTS.filter(
({ name }) => name === "inspect_kernel_module",
),
},
{
identity: sources.artifactProviders.BYTE_ENTROPY_PROVIDER,
contracts:
sources.binaryDiagnosticsContracts.BINARY_DIAGNOSTICS_TOOL_CONTRACTS.filter(
({ name }) => name === "inspect_byte_entropy",
),
},
{
identity: sources.artifactProviders.NATIVE_EXPORTS_PROVIDER,
contracts:
sources.binaryDiagnosticsContracts.BINARY_DIAGNOSTICS_TOOL_CONTRACTS.filter(
({ name }) => name === "inspect_native_exports",
),
},
{
identity: sources.captureProvider.TSHARK_PROVIDER_IDENTITY,
contracts: sources.captureContracts.CAPTURE_TOOL_CONTRACTS,
},
{
identity: sources.signalProvider.SIGROK_PROVIDER_IDENTITY,
contracts: sources.signalContracts.SIGNAL_TOOL_CONTRACTS,
},
{
identity: sources.wasmProvider.WABT_PROVIDER_IDENTITY,
contracts: sources.wasmContracts.WASM_TOOL_CONTRACTS.filter(
({ kind }) => kind === "artifact-provider",
),
},
{
identity: sources.apkSignatureProvider.APK_SIGNATURE_PROVIDER_IDENTITY,
contracts: sources.apkSignatureContracts.APK_SIGNATURE_TOOL_CONTRACTS,
},
{
identity: sources.evmProvider.EVMOLE_PROVIDER_IDENTITY,
contracts: sources.evmContracts.EVM_TOOL_CONTRACTS,
},
{
identity: sources.goProvider.GO_BINARY_PROVIDER_IDENTITY,
contracts: sources.goContracts.GO_TOOL_CONTRACTS,
},
{
identity: sources.gpuProvider.GPU_CODE_OBJECTS_PROVIDER_IDENTITY,
contracts: sources.gpuContracts.GPU_TOOL_CONTRACTS,
},
{
identity: sources.il2cppProvider.IL2CPP_METADATA_PROVIDER_IDENTITY,
contracts: sources.il2cppContracts.IL2CPP_TOOL_CONTRACTS,
},
{
identity: sources.modelProvider.MODEL_ARTIFACT_PROVIDER_IDENTITY,
contracts: sources.modelContracts.MODEL_TOOL_CONTRACTS,
},
{
identity: sources.rustProvider.RUST_BINARY_PROVIDER_IDENTITY,
contracts: sources.rustContracts.RUST_TOOL_CONTRACTS,
},
{
identity: sources.pythonPycProvider.PYTHON_PYC_PROVIDER_IDENTITY,
contracts: sources.pythonPycContracts.PYTHON_PYC_TOOL_CONTRACTS,
},
{
identity: sources.erlangProvider.ERLANG_BEAM_PROVIDER_IDENTITY,
contracts: sources.erlangContracts.ERLANG_BEAM_TOOL_CONTRACTS,
},
{
identity: sources.electronFuseProvider.ELECTRON_FUSE_PROVIDER_IDENTITY,
contracts: fuseContracts,
},
{
identity: sources.sqliteDomain.SQLITE_PROVIDER_IDENTITY,
contracts: sources.sqliteContracts.SQLITE_TOOL_CONTRACTS,
},
{
identity: sources.artifactProviders.ARTIFACT_GRAPH_PROVIDER,
contracts: [
...sources.artifactContracts.ARTIFACT_TOOL_CONTRACTS,
...sources.desktopContracts.DESKTOP_APPLICATION_TOOL_CONTRACTS,
],
},
{
identity: sources.artifactProviders.MANAGED_STATIC_PROVIDER,
contracts: sources.managedContracts.MANAGED_TOOL_CONTRACTS,
},
{
identity: sources.artifactProviders.MANAGED_WORKFLOW_PROVIDER,
contracts:
sources.managedWorkflowContracts.MANAGED_WORKFLOW_TOOL_CONTRACTS,
},
{
identity: sources.firmwareProvider.BINWALK_PROVIDER_IDENTITY,
contracts: sources.firmwareContracts.FIRMWARE_TOOL_CONTRACTS.filter(
({ name }) => name === "inspect_firmware_regions",
),
},
{
identity: sources.firmwareProvider.UNBLOB_PROVIDER_IDENTITY,
contracts: sources.firmwareContracts.FIRMWARE_TOOL_CONTRACTS.filter(
({ name }) => name === "extract_firmware",
),
},
{
identity: sources.androidProvider.JADX_PROVIDER_IDENTITY,
contracts: sources.androidContracts.ANDROID_TOOL_CONTRACTS,
},
{
identity: sources.pcapProvider.PCAP_PROVIDER_IDENTITY,
contracts: sources.pcapContracts.PCAP_TOOL_CONTRACTS,
},
{
identity: sources.flutterProvider.FLUTTER_PROVIDER_IDENTITY,
contracts: sources.flutterContracts.FLUTTER_TOOL_CONTRACTS,
},
{
identity: sources.apktoolProvider.APKTOOL_PROVIDER_IDENTITY,
contracts: sources.apktoolContracts.APKTOOL_TOOL_CONTRACTS,
},
{
identity: sources.jebProvider.JEB_PROVIDER_IDENTITY,
contracts: sources.jebContracts.JEB_TOOL_CONTRACTS,
},
{
identity: sources.adbProvider.ADB_PROVIDER_IDENTITY,
contracts: sources.adbContracts.ADB_TOOL_CONTRACTS,
},
{
identity: sources.webNetworkCaptureProvider.HAR_CAPTURE_PROVIDER_IDENTITY,
contracts:
sources.webNetworkCaptureContracts.WEB_NETWORK_CAPTURE_TOOL_CONTRACTS,
},
{
identity: sources.webNetworkCaptureProvider.SAZ_CAPTURE_PROVIDER_IDENTITY,
contracts:
sources.webNetworkCaptureContracts.WEB_NETWORK_CAPTURE_TOOL_CONTRACTS,
},
{
identity:
sources.webNetworkCaptureProvider.MITMPROXY_CAPTURE_PROVIDER_IDENTITY,
contracts:
sources.webNetworkCaptureContracts.WEB_NETWORK_CAPTURE_TOOL_CONTRACTS,
},
{
identity: sources.browserIdentities.CDP_BROWSER_PROVIDER_IDENTITY,
contracts: [
...sources.browserContracts.BROWSER_TOOL_CONTRACTS,
...sources.webRuntimeContracts.WEB_RUNTIME_TOOL_CONTRACTS,
],
},
{
identity:
sources.browserIdentities.PLAYWRIGHT_BROWSER_SCENARIO_PROVIDER_IDENTITY,
contracts:
sources.browserScenarioContracts.BROWSER_SCENARIO_TOOL_CONTRACTS,
},
{
identity: sources.browserIdentities.CDP_ELECTRON_PROVIDER_IDENTITY,
contracts: observationContracts,
},
{
identity:
sources.browserIdentities.PLAYWRIGHT_ELECTRON_ACTIVE_PROVIDER_IDENTITY,
contracts: activeContracts,
},
{
identity: sources.v8InspectorProvider.V8_INSPECTOR_PROVIDER_IDENTITY,
contracts:
sources.javascriptRuntimeObservationContracts
.JAVASCRIPT_RUNTIME_OBSERVATION_TOOL_CONTRACTS,
},
{
identity: sources.artifactProviders.JAVASCRIPT_APPLICATION_PROVIDER,
contracts: applicationContracts,
},
{
identity:
sources.artifactProviders.JAVASCRIPT_RUNTIME_RECONCILIATION_PROVIDER,
contracts: reconciliationContracts,
},
{
identity:
sources.artifactProviders.JAVASCRIPT_APPLICATION_WORKFLOW_PROVIDER,
contracts: sources.applicationContracts.APPLICATION_TOOL_CONTRACTS.filter(
({ name }) =>
name !== "project_android_application_graph" &&
name !== "project_apple_application_graph",
),
},
{
identity: sources.artifactProviders.ANALYSIS_VIEW_PROVIDER,
contracts: sources.analysisViewContracts.ANALYSIS_VIEW_TOOL_CONTRACTS,
},
{
identity: sources.artifactProviders.ANDROID_APPLICATION_PROVIDER,
contracts: sources.applicationContracts.APPLICATION_TOOL_CONTRACTS.filter(
({ name }) => name === "project_android_application_graph",
),
},
{
identity: sources.artifactProviders.APPLE_APPLICATION_PROVIDER,
contracts: sources.applicationContracts.APPLICATION_TOOL_CONTRACTS.filter(
({ name }) => name === "project_apple_application_graph",
),
},
{
identity: sources.artifactProviders.WEB_SCRIPT_EXPORT_PROVIDER,
contracts: sources.webScriptContracts.WEB_SCRIPT_TOOL_CONTRACTS,
},
{
identity: sources.javascriptRecoveryProvider.WAKARU_PROVIDER_IDENTITY,
contracts:
sources.javascriptRecoveryContracts.JAVASCRIPT_RECOVERY_TOOL_CONTRACTS,
},
]
.map(({ identity, contracts }) => ({
id: identity.id,
name: identity.name,
version: identity.version,
capabilities: names(contracts),
}))
.sort((left, right) => left.id.localeCompare(right.id));
};
const electronContractSlices = (sources) => {
const applicationContracts =
sources.electronContracts.ELECTRON_TOOL_CONTRACTS.filter(
({ name }) => name === "analyze_javascript_application",
);
const reconciliationContracts =
sources.electronContracts.ELECTRON_TOOL_CONTRACTS.filter(
({ name }) => name === "reconcile_javascript_runtime",
);
const observationContracts =
sources.electronContracts.ELECTRON_TOOL_CONTRACTS.filter(
({ name }) =>
name !== "analyze_javascript_application" &&
name !== "reconcile_javascript_runtime" &&
name !== "capture_electron_scenario" &&
name !== "inspect_electron_fuses",
);
const activeContracts =
sources.electronContracts.ELECTRON_TOOL_CONTRACTS.filter(
({ name }) => name === "capture_electron_scenario",
);
const fuseContracts =
sources.electronContracts.ELECTRON_TOOL_CONTRACTS.filter(
({ name }) => name === "inspect_electron_fuses",
);
if (
applicationContracts.length !== 1 ||
reconciliationContracts.length !== 1 ||
observationContracts.length !== 2 ||
activeContracts.length !== 1 ||
fuseContracts.length !== 1
)
throw new Error("Electron provider capability ownership drifted");
return {
applicationContracts,
reconciliationContracts,
observationContracts,
activeContracts,
fuseContracts,
};
};
+1 -124
View File
@@ -39,131 +39,8 @@ const SOURCE_PATHS = {
cli: "dist/cli.js",
toolContracts: "dist/contracts/toolContracts.js",
toolFamilies: "dist/contracts/toolFamilies.js",
binaryDiagnosticsContracts:
"dist/contracts/native/binaryDiagnosticsToolContracts.js",
cutterContracts: "dist/contracts/cutterToolContracts.js",
cutterProvider: "dist/cutter/CutterBridgeClient.js",
reverseEngineeringContracts:
"dist/contracts/reverseEngineeringToolContracts.js",
gdbContracts: "dist/contracts/gdbToolContracts.js",
gdbProvider: "dist/gdb/GdbSessionManager.js",
rizinDebugContracts: "dist/contracts/rizinDebugToolContracts.js",
objdumpProvider: "dist/objdump/ObjdumpCommand.js",
rizinProvider: "dist/rizin/RizinCommand.js",
rizinDebugProvider: "dist/rizin/RizinDebugSessionManager.js",
fridaContracts: "dist/contracts/fridaToolContracts.js",
fridaProvider: "dist/frida/FridaInstrumentationManager.js",
pwntoolsProvider: "dist/native/pwntools/PwntoolsRelease.js",
nativeContracts: "dist/contracts/native/nativeToolContracts.js",
captureContracts: "dist/contracts/captures/captureToolContracts.js",
captureProvider: "dist/captures/TsharkCaptureProvider.js",
signalContracts: "dist/contracts/signals/signalToolContracts.js",
signalProvider: "dist/signals/SigrokRecordingProvider.js",
wasmContracts: "dist/contracts/wasm/wasmToolContracts.js",
wasmProvider: "dist/wasm/WabtRelease.js",
apkSignatureContracts:
"dist/contracts/apkSignature/apkSignatureToolContracts.js",
apkSignatureProvider: "dist/apkSignature/ApkSignatureRelease.js",
evmContracts: "dist/contracts/evm/evmToolContracts.js",
sqliteContracts: "dist/contracts/sqlite/sqliteToolContracts.js",
sqliteDomain: "dist/domain/sqlite/sqliteDatabase.js",
evmProvider: "dist/evm/EvmoleRelease.js",
goContracts: "dist/contracts/go/goToolContracts.js",
goProvider: "dist/go/GoBinaryProvider.js",
gpuContracts: "dist/contracts/gpu/gpuToolContracts.js",
gpuProvider: "dist/gpu/GpuCodeObjectsProvider.js",
modelContracts: "dist/contracts/model/modelToolContracts.js",
modelProvider: "dist/model/ModelArtifactProvider.js",
il2cppContracts: "dist/contracts/il2cpp/il2cppToolContracts.js",
il2cppProvider: "dist/il2cpp/Il2cppMetadataProvider.js",
rustContracts: "dist/contracts/rust/rustToolContracts.js",
rustProvider: "dist/rust/RustBinaryProvider.js",
pythonPycContracts: "dist/contracts/python/pythonPycToolContracts.js",
pythonPycProvider: "dist/python/PythonPycProvider.js",
erlangContracts: "dist/contracts/erlang/beamToolContracts.js",
erlangProvider: "dist/erlang/BeamProvider.js",
artifactContracts: "dist/contracts/artifactToolContracts.js",
desktopContracts: "dist/contracts/desktopApplicationToolContracts.js",
managedContracts: "dist/contracts/managed/managedToolContracts.js",
firmwareContracts: "dist/contracts/firmware/firmwareToolContracts.js",
firmwareProvider: "dist/firmware/FirmwareRelease.js",
androidContracts: "dist/contracts/android/androidToolContracts.js",
flutterContracts: "dist/contracts/flutter/flutterToolContracts.js",
flutterProvider: "dist/flutter/FlutterBuildProvider.js",
androidProvider: "dist/android/JadxRelease.js",
apktoolContracts: "dist/contracts/apktool/apktoolToolContracts.js",
pcapContracts: "dist/contracts/pcap/pcapToolContracts.js",
pcapProvider: "dist/pcap/PcapProvider.js",
apktoolProvider: "dist/apktool/ApktoolProvider.js",
jebContracts: "dist/contracts/jeb/jebToolContracts.js",
jebProvider: "dist/jeb/JebProvider.js",
adbContracts: "dist/contracts/adb/adbToolContracts.js",
adbProvider: "dist/adb/AdbProvider.js",
managedWorkflowContracts:
"dist/contracts/managed/managedWorkflowToolContracts.js",
browserContracts: "dist/contracts/browserToolContracts.js",
webRuntimeContracts: "dist/contracts/webRuntimeToolContracts.js",
webNetworkCaptureContracts:
"dist/contracts/webNetworkCaptureToolContracts.js",
webNetworkCaptureProvider: "dist/browser/history/CaptureRelease.js",
browserScenarioContracts: "dist/contracts/browserScenarioToolContracts.js",
electronContracts: "dist/contracts/javascript/electronToolContracts.js",
electronFuseProvider: "dist/electron/ElectronFuseProvider.js",
javascriptRuntimeObservationContracts:
"dist/contracts/javascript/javascriptRuntimeObservationToolContracts.js",
applicationContracts: "dist/contracts/applicationToolContracts.js",
analysisViewContracts: "dist/contracts/analysisViewToolContracts.js",
webScriptContracts: "dist/contracts/webScriptToolContracts.js",
javascriptRecoveryContracts:
"dist/contracts/javascript/javascriptRecoveryToolContracts.js",
javascriptRecoveryProvider: "dist/javascript/recovery/WakaruRelease.js",
capabilityDeclarations: "dist/composition/capabilityDeclarations.js",
supportedClients: "dist/application/SupportedClients.js",
hopperProvider: "dist/hopper/HopperProviderCapabilities.js",
ghidraProvider: "dist/ghidra/GhidraProviderCapabilities.js",
idaProvider: "dist/ida/IdaProviderCapabilities.js",
kunaProvider: "dist/kuna/KunaProviderCapabilities.js",
nativeProvider: "dist/native/NativeMacOSProviderMetadata.js",
artifactProviders: "dist/domain/investigationProviders.js",
browserIdentities: "dist/browser/providerIdentities.js",
v8InspectorProvider: "dist/inspector/providerIdentity.js",
evidence: "dist/domain/evidence.js",
evidenceBundle: "dist/domain/evidenceBundle.js",
processCapture: "dist/domain/process/processCapture.js",
analysisSnapshot: "dist/domain/analysisSnapshot.js",
artifactGraph: "dist/domain/artifactGraph.js",
browserObservation: "dist/domain/browserObservation.js",
browserScenario: "dist/domain/browserScenario.js",
browserScenarioCapture: "dist/domain/browserScenarioCapture.js",
browserScenarioDiff: "dist/domain/browserScenarioDiff.js",
browserSession: "dist/domain/browserSession.js",
electronObservation: "dist/domain/javascript/electronObservation.js",
electronActiveObservation:
"dist/domain/javascript/electronActiveObservation.js",
javascriptRuntimeObservation:
"dist/domain/javascript/javascriptRuntimeObservation.js",
webBundleAnalysis: "dist/domain/webBundleAnalysis.js",
webCaptureDiff: "dist/domain/webCaptureDiff.js",
managedArtifact: "dist/domain/managed/managedArtifact.js",
managedComparison: "dist/domain/managed/managedMemberComparison.js",
managedNativeVerification: "dist/domain/managed/managedNativeVerification.js",
webMcpDiscovery: "dist/domain/webMcpDiscovery.js",
webScreenshot: "dist/domain/webScreenshot.js",
javascriptApplicationGraph:
"dist/domain/javascript/javascriptApplicationGraph.js",
javascriptApplicationAnalysis:
"dist/domain/javascript/javascriptApplicationAnalysis.js",
javascriptSemanticGraph: "dist/domain/javascript/javascriptSemanticGraph.js",
javascriptSemanticQuery:
"dist/domain/javascript/javascriptSemanticQuerySchemas.js",
javascriptSemanticTrace:
"dist/domain/javascript/javascriptSemanticTraceSchemas.js",
javascriptFeatureTrace:
"dist/domain/javascript/javascriptFeatureTraceSchemas.js",
javascriptVersionComparison:
"dist/domain/javascript/javascriptApplicationVersionComparisonSchemas.js",
reconstructionVerification:
"dist/domain/reconstructionVerificationSchemas.js",
residualUnknown: "dist/domain/residualUnknown.js",
};
/** Load every compiled source module referenced by the catalog. */
+2
View File
@@ -2,6 +2,8 @@
export interface ImportSpecifierFact {
readonly specifier: string;
readonly line: number;
/** True only when TypeScript erases this module edge under verbatimModuleSyntax. */
readonly typeOnly: boolean;
}
/** Static string specifier on one import or re-export node, when it has one. */
+21 -1
View File
@@ -26,6 +26,22 @@ export const importSpecifier = (node) => {
}
};
/** Whether TypeScript erases this module edge under verbatimModuleSyntax. */
const isTypeOnly = (node) => {
switch (node.type) {
case "TSImportType":
return true;
case "ImportDeclaration":
case "TSImportEqualsDeclaration":
return node.importKind === "type";
case "ExportNamedDeclaration":
case "ExportAllDeclaration":
return node.exportKind === "type";
default:
return false;
}
};
/**
* Static import and re-export specifiers in source order.
* Comments and string literals are not imports. Non-string dynamic imports
@@ -37,7 +53,11 @@ export const readImportSpecifiers = (source) => {
traverseFast(tree, (node) => {
const specifier = importSpecifier(node);
if (specifier === undefined) return;
specifiers.push({ specifier, line: node.loc?.start.line ?? 1 });
specifiers.push({
specifier,
line: node.loc?.start.line ?? 1,
typeOnly: isTypeOnly(node),
});
});
return specifiers;
};
+40 -1
View File
@@ -21,6 +21,8 @@ export const verifyKunaReferences = async ({
address,
expectedCaller,
nativeIncoming,
verifyNamedTrace = false,
minimumTraceReferences = 1,
}) => {
const before = await Promise.all([
entries(join(cacheRoot, "projects")),
@@ -80,7 +82,44 @@ export const verifyKunaReferences = async ({
direction: "forward",
});
assert.equal(path.value.goal_status, "reached");
composed = { graph_ms: graph.ms, path_ms: path.ms };
let traceMs;
if (verifyNamedTrace) {
const trace = await call("trace_feature", { query: name });
traceMs = trace.ms;
assert.ok(
trace.value.matches.some(
(match) => match.type === "procedure" && match.address === address,
),
`${name} was not retained in its feature trace`,
);
assert.ok(
incoming.value.length >= minimumTraceReferences,
`${name} fixture did not produce the expected incoming references`,
);
const references = trace.value.references.filter(
(row) => row.target_address === address,
);
assert.deepEqual(
references.map((row) => row.source_address).sort(),
[...incoming.value].sort(),
`${name} lost or duplicated an observed incoming reference`,
);
for (const reference of references) {
assert.equal(reference.containing_procedure.status, "unavailable");
assert.equal(reference.containing_procedure.provider_id, "kuna");
assert.ok(
trace.value.residual_unknowns.some((unknown) =>
unknown.includes(reference.source_address),
),
);
}
assert.equal(trace.value.truncated, false);
}
composed = {
graph_ms: graph.ms,
path_ms: path.ms,
...(traceMs === undefined ? {} : { feature_trace_ms: traceMs }),
};
}
assert.deepEqual(
await Promise.all([
+11 -3
View File
@@ -24,6 +24,8 @@ const MIGRATED_PROVIDER_ROOTS = new Set([
"inspector",
"javascript",
]);
const CENTRALIZED_CONSTRUCTOR_OWNER =
/^src\/(?:frida\/FridaInstrumentationManager|cutter\/CutterBridge(?:Client|Service)|gdb\/GdbSessionManager|rizin\/RizinDebugSessionManager)\.(?:js|ts)$/u;
const repositoryPath = (root, path) =>
relative(root, path).replaceAll("\\", "/");
@@ -35,7 +37,7 @@ const sourceTarget = (file, specifier) => {
: undefined;
};
const failedBoundary = (file, target) => {
const failedBoundary = (file, target, typeOnly) => {
if (!file.startsWith("src/") || !target.startsWith("src/")) return undefined;
const layer = file.split("/")[1];
const targetLayer = target.split("/")[1];
@@ -75,6 +77,12 @@ const failedBoundary = (file, target) => {
}
if (layer === "application" && targetLayer === "composition")
return "application-composition";
if (
(layer === "application" || layer === "server") &&
!typeOnly &&
CENTRALIZED_CONSTRUCTOR_OWNER.test(target)
)
return "provider-construction";
if (
(layer === "application" || layer === "server") &&
(MIGRATED_PROVIDER_ROOTS.has(targetLayer) ||
@@ -92,11 +100,11 @@ export const inspectModuleBoundaries = (file, source, root) => {
const absoluteFile = resolve(root, file);
const importer = repositoryPath(root, absoluteFile);
const violations = [];
for (const { specifier, line } of readImportSpecifiers(source)) {
for (const { specifier, line, typeOnly } of readImportSpecifiers(source)) {
const resolved = sourceTarget(absoluteFile, specifier);
if (resolved === undefined) continue;
const target = repositoryPath(root, resolved);
const boundary = failedBoundary(importer, target);
const boundary = failedBoundary(importer, target, typeOnly);
if (boundary !== undefined)
violations.push({ file: importer, line, specifier, target, boundary });
}
+8 -2
View File
@@ -46,7 +46,7 @@ export async function verifyGhidraBoundaries(
}
let successfulCalls = 0;
let rejectedCalls = 0;
const call = async (name, args = {}) => {
const checkedReply = async (name, args = {}) => {
const reply = await client.callTool({ name, arguments: args }, options);
const result = requireMcpOperationResult(reply, name);
const validate = validators.get(name);
@@ -66,8 +66,13 @@ export async function verifyGhidraBoundaries(
`${name} omitted defined-string coverage`,
);
successfulCalls++;
return result;
return { reply, result };
};
const call = async (name, args = {}) =>
(await checkedReply(name, args)).result;
// Validated complete Evidence for checks outside the operation result.
const evidence = async (name, args = {}) =>
(await checkedReply(name, args)).reply.structuredContent;
const invalid = async (name, args, diagnostic) => {
const reply = await client.callTool({ name, arguments: args }, options);
assert.equal(
@@ -760,6 +765,7 @@ export async function verifyGhidraBoundaries(
});
await verifyGhidraNamespaceAnnotations({
call,
evidence,
reject: invalid,
target,
entrypoint,
+39
View File
@@ -87,11 +87,50 @@ export async function verifyGhidraEntryAliases({
(await call("procedure_info", { procedure: alias.value })).entrypoint,
alias.address,
);
// Search matches the names exact lookup accepts, as one procedure row
// under its primary name.
const escaped = alias.value.replace(/[.*+?^${}()|[\]\\]/gu, "\\$&");
for (const query of [
{ pattern: alias.value },
{ pattern: `^${escaped}$`, mode: "regex", case_sensitive: true },
]) {
assert.deepEqual(
(await call("search_procedures", query))
.filter(({ address }) => address === alias.address)
.map(({ value, matched_aliases }) => ({ value, matched_aliases })),
[{ value: canonical.value, matched_aliases: [alias.value] }],
`Procedure search missed the entry alias for ${JSON.stringify(query)}`,
);
}
assert.ok(
(await call("list_procedures")).every(
(row) => row.matched_aliases === undefined,
),
"Listings must not report search matches",
);
const traced = await call("trace_feature", { query: alias.value });
assert.ok(
traced.matches.some(
({ type, address, matched_aliases }) =>
type === "procedure" &&
address === alias.address &&
matched_aliases?.includes(alias.value) === true,
),
"trace_feature missed the procedure its entry alias names",
);
const interior = names.find(
({ value }) => value === `${prefix}rea_interior`,
);
assert.ok(interior);
assert.notEqual(interior.address, alias.address);
assert.ok(
(await call("search_procedures", { pattern: interior.value })).every(
({ address, matched_aliases = [] }) =>
address !== interior.address &&
!matched_aliases.includes(interior.value),
),
"An interior label must not match procedure search",
);
assert.equal(
interior.symbol.type,
"label",
+140 -23
View File
@@ -12,6 +12,7 @@ import { inspectVerifierGhidraInstallation } from "./ghidra-installation-inspect
/** Exercise qualified annotation names against real demangled C++ ABI symbols. */
export async function verifyGhidraNamespaceAnnotations({
call,
evidence,
reject,
target,
entrypoint,
@@ -62,11 +63,26 @@ export async function verifyGhidraNamespaceAnnotations({
assert.ok(symbol, `Missing demangled fixture global ${namespace}::value`);
return { namespace, address: symbol.address };
});
await verifyDuplicateAddressNames(call, procedures, names);
await verifyDuplicateAddressNames(call, evidence, procedures, names);
const ambiguous = await reject("procedure_address", { procedure: "same" });
assert.equal(ambiguous.code, "invalid_request");
for (const { address } of entries)
assert.ok(JSON.stringify(ambiguous.details.issues).includes(address));
// Search accepts the leaf spelling exact lookup accepts, keeping each
// procedure as one row under its qualified primary name.
const leafMatches = await call("search_procedures", {
pattern: "^same$",
mode: "regex",
case_sensitive: true,
});
for (const { namespace, address } of entries)
assert.deepEqual(
leafMatches
.filter((row) => row.address === address)
.map(({ value }) => value),
[`${namespace}::same`],
`Leaf-name search missed ${namespace}::same`,
);
for (const { namespace, address } of entries) {
const leaf = await call("annotate_native_function", {
procedure: address,
@@ -136,6 +152,21 @@ export async function verifyGhidraNamespaceAnnotations({
}
await verifyAddressNames(call, [entries[0], entries[2]], "function");
await verifyAddressNames(call, globals, "label");
// Equal leaf names in different namespaces are distinct destinations.
const shared = "rea_namespace_shared";
assert.deepEqual(
await call("set_addresses_names", {
names: Object.fromEntries(
entries.map(({ address }) => [address, shared]),
),
}),
Object.fromEntries(entries.map(({ address }) => [address, true])),
);
for (const { namespace, address } of entries)
assert.equal(
await call("address_name", { address }),
`${namespace}::${shared}`,
);
for (const { namespace, address } of entries) {
assert.equal(
await call("set_address_name", {
@@ -183,7 +214,7 @@ export async function verifyGhidraNamespaceAnnotations({
}
}
async function verifyDuplicateAddressNames(call, procedures, names) {
async function verifyDuplicateAddressNames(call, evidence, procedures, names) {
const functionEntries = [
"rea_duplicate_function_a",
"rea_duplicate_function_b",
@@ -231,32 +262,118 @@ async function verifyDuplicateAddressNames(call, procedures, names) {
"A rejected duplicate leaves the other address unchanged",
);
const readback = async (ownerName, message) => {
assert.equal(
await call("address_name", { address: owner.address }),
ownerName,
message,
);
assert.equal(
await call("address_name", { address: duplicate.address }),
duplicate.original,
message,
);
};
// Object member order is not an ordering choice: each batch is decided
// from the names held before it, so both orders must agree.
const batchName = `rea_batch_duplicate_${kind}`;
const moved = `${name}_moved`;
const competing = "duplicate_requested_name";
const selected = "name_held_by_selected_address";
for (const [requested, expected, ownerName, reasons, message] of [
[
{ [owner.address]: batchName, [duplicate.address]: batchName },
{ [owner.address]: false, [duplicate.address]: false },
name,
{ [owner.address]: competing, [duplicate.address]: competing },
"Competing requests for one name apply neither address",
],
[
{ [owner.address]: duplicate.original, [duplicate.address]: name },
{ [owner.address]: false, [duplicate.address]: false },
name,
{ [owner.address]: selected, [duplicate.address]: selected },
"A swap between selected addresses is not ordered implicitly",
],
[
{ [duplicate.address]: name, [owner.address]: moved },
{ [owner.address]: true, [duplicate.address]: false },
moved,
{ [duplicate.address]: selected },
"A name held by another selected address is not taken by member order",
],
[
{ [duplicate.address]: name },
{ [duplicate.address]: false },
name,
{ [duplicate.address]: "name_held_by_unselected_address" },
"A name held by an unselected address is not moved",
],
]) {
const facts = [];
for (const ordered of [
requested,
Object.fromEntries(Object.entries(requested).reverse()),
]) {
const record = await evidence("set_addresses_names", {
names: ordered,
});
assert.deepEqual(record.normalized_result, expected, message);
const { failures, conflicts } = record.raw_result;
assert.deepEqual(
Object.fromEntries(
Object.entries(failures).map(([key, { reason }]) => [key, reason]),
),
reasons,
message,
);
// Each contested global name is reported once and names its requester.
for (const [key, failure] of Object.entries(failures)) {
const conflict = conflicts[failure.conflict];
assert.equal(conflict.namespace, null, message);
assert.equal(conflict.name, requested[key], message);
assert.ok(conflict.requested_by.includes(failure.address), message);
}
assert.equal(
conflicts.length,
new Set(Object.keys(failures).map((key) => requested[key])).size,
message,
);
facts.push(record.raw_result);
await readback(ownerName, message);
if (ownerName !== name)
assert.equal(
await call("set_address_name", { address: owner.address, name }),
true,
);
}
assert.deepEqual(facts[1], facts[0], message);
}
assert.deepEqual(
await call("set_addresses_names", {
names: {
[owner.address]: batchName,
[duplicate.address]: batchName,
},
}),
{ [owner.address]: true, [duplicate.address]: false },
(await call("list_names")).filter((item) => item.value === batchName),
[],
"A rejected competing name has no owner",
);
// An explicit later call may take a name released by an earlier one.
assert.equal(
await call("set_address_name", { address: owner.address, name: moved }),
true,
);
assert.equal(
await call("address_name", { address: owner.address }),
batchName,
);
assert.equal(
await call("address_name", { address: duplicate.address }),
duplicate.original,
"A failed batch member rolls back only its own address",
);
const owners = (await call("list_names")).filter(
(item) => item.value === batchName && item.symbol.primary,
);
assert.deepEqual(
owners.map(({ address }) => address),
[owner.address],
await call("set_address_name", { address: duplicate.address, name }),
true,
);
// Ghidra symbol names are case-sensitive, so case variants do not compete.
const cased = {
[owner.address]: `${batchName}_Case`,
[duplicate.address]: `${batchName}_case`,
};
assert.deepEqual(await call("set_addresses_names", { names: cased }), {
[owner.address]: true,
[duplicate.address]: true,
});
for (const [address, expected] of Object.entries(cased))
assert.equal(await call("address_name", { address }), expected);
}
}
@@ -0,0 +1,92 @@
import assert from "node:assert/strict";
import { createHash } from "node:crypto";
import { readFile } from "node:fs/promises";
import { join } from "node:path";
import { fileURLToPath } from "node:url";
import { parseEvidence } from "../../dist/domain/evidence.js";
import { PrivateRuntimeRoot } from "../../dist/process/PrivateRuntimeRoot.js";
import { withVerifierCleanup } from "./verifier-cleanup.mjs";
import { createVerifierCommands } from "./verify-commands.mjs";
const source = fileURLToPath(
new URL(
"../../tests/conformance/ghidra/relocation-symbol.c",
import.meta.url,
),
);
/** Verify named and unnamed ELF relocations through the shipped CLI and Ghidra. */
export async function verifyGhidraRelocationSymbols({ clang, lld, env }) {
const commands = await createVerifierCommands();
let root;
let commandsClosed = false;
return withVerifierCleanup(async () => {
root = await PrivateRuntimeRoot.create({ prefix: "rea-relocation-proof-" });
const object = join(root.path, "relocation-symbol.o");
const target = join(root.path, "relocation-symbol.so");
const commandEnv = {
...env,
TMPDIR: root.path,
TMP: root.path,
TEMP: root.path,
};
const run = (command, args) =>
commands.exec(command, args, {
env: commandEnv,
timeout: 120000,
maxBuffer: 16 * 1024 * 1024,
});
await run(clang, [
"--target=x86_64-linux-gnu",
"-fPIC",
"-O0",
"-c",
source,
"-o",
object,
]);
await run(lld, ["-shared", object, "-o", target]);
const hash = async () =>
createHash("sha256")
.update(await readFile(target))
.digest("hex");
const before = await hash();
const { stdout } = await run(process.execPath, [
"scripts/rea.mjs",
"inspect-native-load-image",
target,
"--provider",
"ghidra",
"--json",
]);
const evidence = parseEvidence(JSON.parse(stdout));
assert.equal(evidence.subject?.digest.sha256, before);
assert.equal(evidence.provider.id, "ghidra");
const report = evidence.normalized_result;
assert.equal(report.status, "unsupported");
const relocations = report.observations.relocations;
assert.ok(
relocations.some(({ symbol }) => symbol === "rea_external"),
"The Ghidra relocation must retain its imported symbol",
);
assert.ok(
relocations.some(({ symbol }) => symbol === ""),
"The provider's empty symbol must remain distinct from an unrecorded field",
);
assert.equal(await hash(), before);
return {
source,
target_sha256: before,
named_symbol: "rea_external",
empty_symbol: true,
};
}, [
async () => {
await commands.close();
commandsClosed = true;
},
async () => {
if (commandsClosed && root !== undefined) await root.close();
},
]);
}
+1 -1
View File
@@ -1,5 +1,5 @@
import assert from "node:assert/strict";
import { HOPPER_PROVIDER_IDENTITY } from "../../dist/hopper/HopperProviderCapabilities.js";
import { HOPPER_PROVIDER_IDENTITY } from "../../dist/hopper/HopperProviderMetadata.js";
import {
requireMcpEvidenceResult,
requireEvidenceProvider,
+1 -1
View File
@@ -1,4 +1,4 @@
import { HOPPER_PROVIDER_IDENTITY } from "../../dist/hopper/HopperProviderCapabilities.js";
import { HOPPER_PROVIDER_IDENTITY } from "../../dist/hopper/HopperProviderMetadata.js";
/** Verify the selected Hopper candidate and concrete provider binding. */
export const requireHopperSelection = (status, expected) => {
+1 -1
View File
@@ -19,7 +19,7 @@ export const SOURCE_TEST_PROJECTS = [
group: "source",
include: [
"src/*.test.ts",
"src/{android,apkSignature,apktool,jeb,adb,artifacts,browser,config,cutter,dotnet,electron,erlang,evm,firmware,flutter,pcap,ghidra,go,gpu,hopper,ida,inspector,javascript,main,native,process,providers,python,reference,rust,server,sqlite,wasm,windows}/**/*.test.ts",
"src/{android,apkSignature,apktool,jeb,adb,artifacts,browser,config,cutter,dotnet,electron,erlang,evm,firmware,flutter,frida,pcap,ghidra,go,gpu,hopper,ida,inspector,javascript,main,native,process,providers,python,reference,rust,server,sqlite,wasm,windows}/**/*.test.ts",
],
},
{
+50 -24
View File
@@ -1,6 +1,7 @@
import { spawn } from "node:child_process";
import { stat } from "node:fs/promises";
import { resolve } from "node:path";
import { fileURLToPath } from "node:url";
import {
developmentTestPlan,
@@ -10,6 +11,7 @@ import {
let child;
let forceTimer;
let shutdownSignal;
let childOwnsTree = false;
const signalExitCodes = { SIGINT: 130, SIGTERM: 143 };
for (const signal of Object.keys(signalExitCodes))
process.on(signal, () => {
@@ -17,8 +19,10 @@ for (const signal of Object.keys(signalExitCodes))
shutdownSignal = signal;
if (child !== undefined && child.exitCode === null) {
child.kill(signal);
forceTimer = setTimeout(() => child?.kill("SIGKILL"), 1_000);
forceTimer.unref?.();
if (!childOwnsTree) {
forceTimer = setTimeout(() => child?.kill("SIGKILL"), 1_000);
forceTimer.unref?.();
}
}
});
@@ -55,28 +59,38 @@ try {
throw new Error(
"Test prerequisites require the locked dependency installation",
);
const prepared = await run(process.execPath, [
"scripts/run-exclusive.mjs",
"artifacts",
const prepared = await run(
process.execPath,
resolve("scripts/run-turbo.mjs"),
"run",
...new Set([
...(plan.needsBuild ? ["artifacts:skills"] : []),
...plan.artifactTasks,
]),
]);
[
"scripts/run-exclusive.mjs",
"artifacts",
process.execPath,
resolve("scripts/run-turbo.mjs"),
"run",
...new Set([
...(plan.needsBuild ? ["artifacts:skills"] : []),
...plan.artifactTasks,
]),
],
false,
"existing",
);
if (prepared.code !== 0) process.exitCode = prepared.code;
}
if (!process.exitCode && !shutdownSignal) {
const tested = await run(process.execPath, [
resolve(
request.paths.length > 0
? "scripts/run-exact-tests.mjs"
: "node_modules/vitest/vitest.mjs",
),
...plan.vitestArguments,
]);
const tested = await run(
process.execPath,
[
resolve(
request.paths.length > 0
? "scripts/run-exact-tests.mjs"
: "node_modules/vitest/vitest.mjs",
),
...plan.vitestArguments,
],
false,
"test",
);
process.exitCode = tested.code;
}
}
@@ -86,15 +100,27 @@ try {
}
if (shutdownSignal) process.exitCode = signalExitCodes[shutdownSignal];
function run(executable, arguments_, capture = false) {
function run(executable, arguments_, capture = false, supervision) {
if (shutdownSignal)
return Promise.resolve({
code: signalExitCodes[shutdownSignal],
output: "",
});
child = spawn(executable, arguments_, {
stdio: capture ? ["ignore", "pipe", "pipe"] : "inherit",
});
// Source-only tests do not require the built Windows Job Object artifact.
const wrap = supervision === "test" && process.platform !== "win32";
childOwnsTree = supervision === "existing" || wrap;
child = spawn(
wrap ? process.execPath : executable,
wrap
? [
fileURLToPath(new URL("./run-exclusive.mjs", import.meta.url)),
"--unlocked",
executable,
...arguments_,
]
: arguments_,
{ stdio: capture ? ["ignore", "pipe", "pipe"] : "inherit" },
);
let output = "";
if (capture) {
child.stdout.on("data", (data) => (output += data.toString()));
+16 -14
View File
@@ -33,7 +33,7 @@ if (!process.connected) {
process.exitCode = signalExitCodes.SIGTERM;
} else if (
typeof lockName !== "string" ||
!/^[a-z][a-z0-9-]*$/u.test(lockName) ||
!(lockName === "--unlocked" || /^[a-z][a-z0-9-]*$/u.test(lockName)) ||
command.length === 0
) {
console.error("Usage: run-exclusive.mjs <lock-name> <command> [args...]");
@@ -45,19 +45,21 @@ if (!process.connected) {
async function run() {
try {
await mkdir(lockRoot, { recursive: true });
// SQLite owns acquisition and OS-backed lock release, including process
// death. Keep the database pathname stable; deleting it would split owners.
database = new DatabaseSync(join(lockRoot, `${lockName}.sqlite`));
try {
database.exec("BEGIN EXCLUSIVE");
} catch (cause) {
if (isBusyError(cause))
throw new Error(
`Another ${lockName} command is already running. Wait for it to finish.`,
{ cause },
);
throw cause;
if (lockName !== "--unlocked") {
await mkdir(lockRoot, { recursive: true });
// SQLite owns acquisition and OS-backed lock release, including process
// death. Keep the database pathname stable; deleting it would split owners.
database = new DatabaseSync(join(lockRoot, `${lockName}.sqlite`));
try {
database.exec("BEGIN EXCLUSIVE");
} catch (cause) {
if (isBusyError(cause))
throw new Error(
`Another ${lockName} command is already running. Wait for it to finish.`,
{ cause },
);
throw cause;
}
}
if (shutdownSignal === undefined) {
const result = await runCommand();
+27 -12
View File
@@ -15,8 +15,8 @@ def fail(code, **details):
raise SystemExit(1)
class BridgeResult(unittest.TextTestResult):
"""Record non-skipped test and subtest outcomes for required families."""
class ExecutedResult(unittest.TextTestResult):
"""Record tests with a non-skipped test or subtest outcome."""
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
@@ -25,10 +25,7 @@ class BridgeResult(unittest.TextTestResult):
self.has_subtests = False
def record(self, test):
parts = test.id().split(".")
if len(parts) > 2 and parts[0] == "bridge":
if parts[1] in REQUIRED_BRIDGE_FAMILIES:
self.completed.add(parts[1])
self.completed.add(test.id())
def startTest(self, test):
self.active_test = test
@@ -65,11 +62,16 @@ def run_bridge(modules):
# The script directory is sys.path[0]; imports must resolve from the checkout.
sys.path.insert(0, str(root))
suite = unittest.defaultTestLoader.loadTestsFromNames(modules)
result = unittest.TextTestRunner(verbosity=2, resultclass=BridgeResult).run(suite)
result = unittest.TextTestRunner(verbosity=2, resultclass=ExecutedResult).run(suite)
completed_families = {
parts[1]
for name in result.completed
if len(parts := name.split(".")) > 2 and parts[0] == "bridge"
}
missing = [
family
for family in REQUIRED_BRIDGE_FAMILIES
if family not in result.completed
if family not in completed_families
]
if missing:
fail("no_executed_bridge_tests", families=missing)
@@ -77,6 +79,19 @@ def run_bridge(modules):
raise SystemExit(1)
def run_analysis_efficiency():
# Match the checkout-root import path of `python -m unittest discover`.
sys.path.insert(0, str(root))
suite = unittest.defaultTestLoader.discover(
str(root / "tests/support/analysis-efficiency"), pattern="test_*.py"
)
result = unittest.TextTestRunner(verbosity=2, resultclass=ExecutedResult).run(suite)
if not result.wasSuccessful():
raise SystemExit(1)
if not result.completed:
fail("no_executed_analysis_efficiency_tests")
bridge_root = root / "bridge"
bridge_tests = sorted(
path for path in bridge_root.rglob("test_*.py") if path.is_file()
@@ -95,6 +110,9 @@ modules = [
if sys.argv[1:] == ["--bridge"]:
run_bridge(modules)
raise SystemExit(0)
if sys.argv[1:] == ["--analysis-efficiency"]:
run_analysis_efficiency()
raise SystemExit(0)
if sys.argv[1:]:
fail("invalid_python_test_arguments")
bridge_result = subprocess.run(
@@ -103,10 +121,7 @@ bridge_result = subprocess.run(
if bridge_result.returncode:
sys.exit(bridge_result.returncode)
result = subprocess.run(
[
sys.executable, "-m", "unittest", "discover",
"-s", "tests/support/analysis-efficiency", "-p", "test_*.py", "-v",
],
[sys.executable, __file__, "--analysis-efficiency"],
cwd=root,
check=False,
)
+14
View File
@@ -0,0 +1,14 @@
import type { PackageVerifierCommands } from "./lib/verify-package-commands.mjs";
export function verifyPackageUpdate(options: {
readonly prefix: string;
readonly packageRoot: string;
readonly tarball: string;
readonly workspace: string;
readonly environment: NodeJS.ProcessEnv;
readonly commands: Pick<PackageVerifierCommands, "exec">;
}): Promise<{
readonly installed_version: string;
readonly exact_release: true;
readonly maintenance_read_only: true;
}>;
+13 -3
View File
@@ -35,13 +35,20 @@ export async function verifyPackageUpdate({
response.destroy();
return;
}
response.writeHead(500);
response.writeHead(500, {
"Content-Type": "text/plain; charset=utf-8",
"X-Content-Type-Options": "nosniff",
});
response.end(String(error));
});
});
let registry;
async function serveRegistry(request, response) {
const pathname = new URL(request.url, registry).pathname;
// A leading // belongs to an origin-form HTTP path, not a URL authority.
const requested = new URL(
request.url.startsWith("/") ? `${registry}${request.url}` : request.url,
);
const { pathname } = requested;
if (pathname === "/rea-agents") {
response.setHeader("Content-Type", "application/json");
response.end(
@@ -65,7 +72,10 @@ export async function verifyPackageUpdate({
response.setHeader("Content-Type", "application/json");
response.end("{}");
} else {
const upstream = await fetch(`https://registry.npmjs.org${request.url}`);
const upstreamUrl = new URL("https://registry.npmjs.org");
upstreamUrl.pathname = pathname;
upstreamUrl.search = requested.search;
const upstream = await fetch(upstreamUrl);
const body = Buffer.from(await upstream.arrayBuffer());
response.writeHead(upstream.status, {
"Content-Type":
+1 -1
View File
@@ -510,7 +510,7 @@ try {
);
const { GHIDRA_OPERATIONS } = await import(
pathToFileURL(
join(packageRoot, "dist/ghidra/GhidraProviderCapabilities.js"),
join(packageRoot, "dist/ghidra/GhidraProviderMetadata.js"),
)
);
const expected = GHIDRA_OPERATIONS.filter(
+3 -1
View File
@@ -79,7 +79,9 @@ await withVerifierCleanup(async () => {
(entry) =>
entry.response_address_count > entry.response_addresses.length,
);
assert.equal(result.coverage, partial ? "partial" : "complete");
// Count discrepancies prove partial coverage, but matching counts do not
// rule out unresolved HTTP targets or unavailable DNS identity extraction.
if (partial) assert.equal(result.coverage, "partial");
assert.deepEqual(
await hashVerifierFile(path, resources),
identity,
+101 -1
View File
@@ -3,6 +3,7 @@ import assert from "node:assert/strict";
import {
requireMcpLifecycleResult,
requireMcpOutput,
requireMcpToolError,
} from "./lib/mcp-verifier-results.mjs";
import { execFile } from "node:child_process";
import { createHash } from "node:crypto";
@@ -15,6 +16,10 @@ import { Client } from "@modelcontextprotocol/client";
import { StdioClientTransport } from "@modelcontextprotocol/client/stdio";
import { Ajv2020 } from "ajv/dist/2020.js";
import { parseEvidence } from "../dist/domain/evidence.js";
import {
ADDRESS_CONTEXT_FUNCTION_COMMENT_RECOVERY,
ADDRESS_CONTEXT_UNAVAILABLE_RECOVERY,
} from "../dist/application/AnalysisContextQueries.js";
import { functionDossierSchema } from "../dist/domain/native/functionDossier.js";
import { nativeInstructionSchema } from "../dist/domain/native/nativeInstruction.js";
import { inspectVerifierGhidraInstallation } from "./lib/ghidra-installation-inspection.mjs";
@@ -183,7 +188,8 @@ try {
item.includes("did not attempt a decode"),
),
);
const advertised = (await client.listTools()).tools.find(
const tools = (await client.listTools()).tools;
const advertised = tools.find(
(tool) => tool.name === "inspect_native_instruction",
);
assert.ok(advertised?.outputSchema);
@@ -198,6 +204,34 @@ try {
nativeInstructionSchema.safeParse({ ...unlisted, status: "undecodable" })
.success,
);
const contextDefinition = tools.find(
(tool) => tool.name === "inspect_address_context",
);
assert.ok(contextDefinition?.outputSchema);
const documents = await call("list_documents");
assert.equal(documents.length, 1);
const document = documents[0];
for (const input of [
{ address: fixture.entry, document: `${document}-not-a-program` },
{ address: "not-an-address", document },
]) {
const direct = requireMcpToolError(
await client.callTool({ name: "address_name", arguments: input }),
);
const response = await client.callTool(
{ name: "inspect_address_context", arguments: input },
{ timeout: 240000, toolDefinition: contextDefinition },
);
const context = requireMcpLifecycleResult(
response,
"inspect_address_context",
);
assert.equal(context.name.state, "unavailable");
assert.equal(context.name.error.code, "invalid_request");
assert.deepEqual(context.name.error.details.issues, direct.details.issues);
assert.equal(context.name.reason, direct.message);
assert.equal(context.name.remediation, direct.remediation.action);
}
assert.equal(
(await call("resolve_native_call_targets", { address: unlistedAddress }))
.status,
@@ -246,6 +280,70 @@ try {
assert.equal(updated.annotations.inline_comment, changes.inline_comment);
assert.equal(updated.dossier.procedure.name, changes.name);
assert.ok(updated.dossier.pseudocode.includes(changes.name));
// Exercise the documented recovery through the public MCP catalog on real
// Ghidra, rather than assuming the provider exposes direct comment methods.
const contextCall = async (name, args) => {
const definition = tools.find((tool) => tool.name === name);
assert.ok(definition?.outputSchema, `${name} must be advertised`);
const response = await client.callTool(
{ name, arguments: args },
{ timeout: 240000, toolDefinition: definition },
);
return requireMcpOutput(response, name);
};
const context = (
await contextCall("inspect_address_context", {
address: fixture.entry,
})
).result;
assert.equal(context.procedure.state, "available");
assert.equal(context.procedure.value.found, true);
for (const facet of [context.comment, context.inline_comment]) {
assert.equal(facet.state, "unavailable");
assert.equal(facet.remediation, ADDRESS_CONTEXT_FUNCTION_COMMENT_RECOVERY);
}
assert.equal(
context.bookmarks.remediation,
ADDRESS_CONTEXT_UNAVAILABLE_RECOVERY,
);
const summary = parseEvidence(
await contextCall("analyze_function", {
procedure: context.procedure.value.procedure.address,
}),
).normalized_result;
const comments = parseEvidence(
await contextCall("inspect_analysis_view", {
source: {
kind: "retained-evidence",
evidence_id: summary.parent_evidence_id,
},
view: { kind: "native", facet: "comments" },
}),
).normalized_result;
for (const [kind, text] of [
["comment", changes.comment],
["inline", changes.inline_comment],
])
assert.ok(
comments.item.some(
(item) =>
item.address === context.procedure.value.query_address &&
item.kind === kind &&
item.text === text,
),
);
const outside = (
await contextCall("inspect_address_context", {
address: unlistedAddress,
})
).result;
assert.equal(outside.procedure.state, "available");
assert.equal(outside.procedure.value.found, false);
assert.equal(outside.comment.state, "unavailable");
assert.equal(
outside.comment.remediation,
ADDRESS_CONTEXT_UNAVAILABLE_RECOVERY,
);
assert.equal(
await call("procedure_address", { procedure: changes.name }),
fixture.entry,
@@ -356,7 +454,9 @@ try {
fresh_decompiler: true,
clear_comments: true,
cli_mcp_parity: true,
address_context_comment_recovery: true,
},
context_input_errors_preserved: true,
original_unchanged: true,
address_truncation_rejected: true,
limitations: image.limitations,
+25 -2
View File
@@ -125,6 +125,13 @@ try {
assert.equal(image.module_bytes, fixture.module_bytes);
assert.equal(image.entry.linear_address, expected.entry);
assert.ok(image.checks.every((check) => check.matched));
assert.ok(image.observations.relocations.length > 0);
assert.ok(
image.observations.relocations.every(
({ symbol }) => symbol === null || typeof symbol === "string",
),
"Each real Ghidra relocation must record its symbol observation",
);
const cliImage = await cliEvidence("inspect-native-load-image");
// Snapshot filenames are session-specific. Every measured byte/range/check is stable.
const stableImage = (value) => ({
@@ -378,17 +385,33 @@ try {
arguments: { procedure: "0xffffffffffffffff" },
});
assert.equal(invalid.isError, true, "Unmapped address did not fail");
// The CLI reads a fresh session, so compare it with MCP before the boundary
// helper's temporary rename and restoration.
const entryWindow = await call("read_function_instructions", {
procedure: identities.entry.address,
});
const cliWindow = await cliEvidence("instructions", identities.entry.address);
assert.deepEqual(cliWindow.normalized_result, entryWindow);
await verifyGhidraAddressBoundaries(
client,
identities.entry.address,
cliEvidence,
);
const cliWindow = await cliEvidence("instructions", identities.entry.address);
const entrySymbol = entryWindow.procedure.symbol;
assert.ok(entrySymbol != null, "The DOS entry needs observed symbol facts");
// Restoring the name is itself a user edit, so Ghidra records user provenance.
assert.deepEqual(
cliWindow.normalized_result,
await call("read_function_instructions", {
procedure: identities.entry.address,
}),
{
...entryWindow,
procedure: {
...entryWindow.procedure,
symbol: { ...entrySymbol, source: "user_defined" },
},
},
"Restoring the entry name changed more than its symbol provenance",
);
assert.equal(
digest(await readFile(targetPath)),
+30 -52
View File
@@ -2,7 +2,14 @@
import assert from "node:assert/strict";
import { execFile } from "node:child_process";
import { createHash } from "node:crypto";
import { mkdtemp, mkdir, readFile, rm, writeFile } from "node:fs/promises";
import {
mkdtemp,
mkdir,
readFile,
realpath,
rm,
writeFile,
} from "node:fs/promises";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { fileURLToPath } from "node:url";
@@ -18,6 +25,8 @@ import {
assertMipsProbe,
assertMipsGlobal,
assertMipsSoftFloatStorage,
assertMipsCarvedTargetRefusal,
carveMipsElfBeforeSectionTable,
} from "../tests/conformance/ghidra/mips-checks.mjs";
import { parseEvidence } from "../dist/domain/evidence.js";
import { parseAnalysisSnapshot } from "../dist/domain/analysisSnapshot.js";
@@ -122,16 +131,6 @@ async function verifySoftFloat(call, symbols, byteOrder) {
);
return { storage: boundaries };
}
function assertIncompleteMipsInspection(error) {
assert.equal(error.code, "capability_unavailable");
assert.equal(error.details.selection_reason, "target_unsupported");
const rejection = error.details.rejections.find(
(candidate) => candidate.provider_id === "ghidra",
);
assert.equal(rejection?.code, "architecture_unsupported");
assert.match(rejection.reason, /ABI declaration inspection is incomplete/u);
assert.match(rejection.reason, /section header table/u);
}
const reports = [];
let primaryFailure;
@@ -617,45 +616,20 @@ try {
// PSP keeps its independent Allegrex/A-B-A checks; carved PSP is not verified.
let carved;
if (!psp) {
// A program-header ABI record survives carving, but GNU declarations in
// uncaptured section headers remain unknown: both adapters must refuse.
const read16 = (offset) =>
byteOrder === "little"
? sourceBytes.readUInt16LE(offset)
: sourceBytes.readUInt16BE(offset);
const read32 = (offset) =>
byteOrder === "little"
? sourceBytes.readUInt32LE(offset)
: sourceBytes.readUInt32BE(offset);
const sectionOffset = read32(32);
const cut = sectionOffset - 1;
const programOffset = read32(28);
const programCount = read16(44);
assert.equal(read16(42), 32);
assert.ok(cut >= programOffset + programCount * 32);
for (let i = 0; i < programCount; i += 1) {
const offset = programOffset + i * 32;
assert.ok(read32(offset + 4) + read32(offset + 16) <= cut);
}
const carvedBytes = sourceBytes.subarray(0, cut);
// The carved copy keeps every program header and payload but declares
// a section table it did not capture. Both adapters refuse it when
// opening the target; it is not analyzed as a smaller program.
const carve = carveMipsElfBeforeSectionTable(sourceBytes, byteOrder);
const carvedPath = join(
workspace,
`${byteOrder}-${floatAbi}-carved.elf`,
);
const carvedSha256 = createHash("sha256")
.update(carvedBytes)
.update(carve.bytes)
.digest("hex");
await writeFile(carvedPath, carvedBytes);
const carvedResolved = await parseBinaryTarget(carvedPath);
if (!carvedResolved.ok) throw carvedResolved.error;
assert.deepEqual(
carvedResolved.value.mips.abiFlags,
independent.mips.abiFlags,
);
assert.match(
carvedResolved.value.mips.limitations.join(" "),
/section header table/u,
);
await writeFile(carvedPath, carve.bytes);
// Target errors name the canonical path that was opened.
const openedPath = await realpath(carvedPath);
const carvedError = requireMcpToolError(
await client.callTool(
{
@@ -665,7 +639,7 @@ try {
{ timeout: 60000 },
),
);
assertIncompleteMipsInspection(carvedError);
assertMipsCarvedTargetRefusal(carvedError, openedPath, carve.reason);
let carvedCliError;
try {
await execute(
@@ -688,16 +662,20 @@ try {
carvedCliError &&
Number.isInteger(carvedCliError.code) &&
carvedCliError.code !== 0,
"Expected an incomplete-ABI refusal, not a timeout or success",
"Expected a target-opening refusal, not a timeout or success",
);
assertIncompleteMipsInspection(JSON.parse(carvedCliError.stdout));
assert.deepEqual(await readFile(carvedPath), carvedBytes);
assertMipsCarvedTargetRefusal(
JSON.parse(carvedCliError.stdout),
openedPath,
carve.reason,
);
assert.deepEqual(await readFile(carvedPath), carve.bytes);
carved = {
sha256: carvedSha256,
source_bytes: carvedBytes.length,
section_header_offset: sectionOffset,
status: "expected-incomplete-abi-refusal",
limitations: carvedResolved.value.mips.limitations,
source_bytes: carve.bytes.length,
section_header_offset: carve.sectionOffset,
status: "expected-target-refusal",
reason: carve.reason,
cli_exit: carvedCliError.code,
mcp_error: carvedError,
};
+285 -91
View File
@@ -23,7 +23,17 @@ import { parseAnalysisSnapshot } from "../dist/domain/analysisSnapshot.js";
import { parseEvidence } from "../dist/domain/evidence.js";
import { functionDossierSchema } from "../dist/domain/native/functionDossier.js";
import { inspectVerifierGhidraInstallation } from "./lib/ghidra-installation-inspection.mjs";
import { buildRiscvFixture } from "../tests/conformance/ghidra/riscv-fixture.mjs";
import {
buildRiscvFixture,
stripRiscvFixture,
} from "../tests/conformance/ghidra/riscv-fixture.mjs";
import {
RISCV_ILP32_ABI,
assertRiscvIlp32CallSites,
assertRiscvIlp32Caller,
assertRiscvIlp32Storage,
inspectRiscvCallSites,
} from "../tests/conformance/ghidra/riscv-checks.mjs";
import {
mcpTextValue,
requireMcpLifecycleResult,
@@ -82,6 +92,29 @@ transport.stderr?.on("data", (chunk) => {
stderr = (stderr + chunk.toString()).slice(-65536);
});
/** The profile a fixture's lane must commit, from independent declarations. */
const expectedLane = (fixture) => {
const soft = fixture.riscv.flags === 1;
return {
lane: `elf${fixture.riscv.elfClass}-exec-le-${soft ? "soft" : "double"}-float-generic-v1`,
callingConvention: soft ? "ilp32-prototype-v1" : undefined,
riscvElf: {
elf_class: fixture.riscv.elfClass,
byte_order: "little",
type: 2,
flags: fixture.riscv.flags,
float_abi_double: !soft,
...(soft ? { float_abi: "soft", arch: fixture.riscv.isa.arch } : {}),
rvc_declared: true,
rve_declared: false,
tso_declared: false,
},
};
};
const OUTCOME =
/RISC-V calling-convention outcome: [1-9]\d* functions and (\d+) function definitions use __rea_ilp32\./u;
const reports = [];
let failure;
try {
@@ -101,41 +134,55 @@ try {
: undefined;
validators.set(tool.name, existing ?? ajv.compile(tool.outputSchema));
}
for (const bits of [32, 64]) {
const fixture = await buildRiscvFixture(workspace, bits);
const typedSoft = await buildRiscvFixture(workspace, 32, "soft");
const callSites = await inspectRiscvCallSites(typedSoft.path);
assertRiscvIlp32CallSites(callSites.calls);
const fixtures = [
await buildRiscvFixture(workspace, 32),
await buildRiscvFixture(workspace, 64),
typedSoft,
await stripRiscvFixture(workspace, typedSoft),
];
for (const fixture of fixtures) {
const bits = fixture.riscv.elfClass;
const soft = fixture.riscv.flags === 1;
const stripped = fixture.stripped === true;
const expected = expectedLane(fixture);
const resolved = await parseBinaryTarget(fixture.path);
if (!resolved.ok) throw resolved.error;
assert.equal(resolved.value.architecture, "riscv");
assert.equal(resolved.value.sha256, fixture.sha256);
assert.deepEqual(resolved.value.riscv, fixture.riscv);
const expectedLanguage = `RISCV:LE:${bits}:default`;
const assertProfile = (profile) => {
const assertProfile = (profile, want = expected) => {
assert.equal(profile.provider.id, "ghidra");
assert.equal(profile.provider.version, installation.providerVersion);
const parameters = profile.parameters;
assert.equal(parameters.language_id, expectedLanguage);
assert.equal(parameters.compiler_spec_id, "gcc");
assert.equal(parameters.language_selection, "provider-pinned-v1");
assert.equal(
parameters.riscv_support_lane,
`elf${bits}-exec-le-double-float-generic-v1`,
);
assert.deepEqual(parameters.riscv_elf, {
elf_class: bits,
byte_order: "little",
type: 2,
flags: 5,
float_abi_double: true,
rvc_declared: true,
rve_declared: false,
tso_declared: false,
});
assert.equal(parameters.riscv_support_lane, want.lane);
assert.equal(parameters.riscv_calling_convention, want.callingConvention);
assert.deepEqual(parameters.riscv_elf, want.riscvElf);
};
const assertEvidence = (evidence) => {
assert.equal(evidence.provider.id, "ghidra");
assert.equal(evidence.subject.architecture, "riscv");
assert.equal(evidence.subject.digest.sha256, fixture.sha256);
assertProfile(evidence.analysis_profile);
// Composed workflows such as inspect_native_api commit the Ghidra
// profile as their upstream profile. The residual-unknown registry
// records what they could not establish, such as the types of a
// stripped function's parameters, without a profile of its own.
if (evidence.provider.id === "rea-workflow")
assertProfile(
evidence.analysis_profile.parameters.upstream_analysis_profile,
);
else if (evidence.provider.id === "rea-unknown-registry")
assert.equal(evidence.analysis_profile, null);
else {
assert.equal(evidence.provider.id, "ghidra");
assertProfile(evidence.analysis_profile);
}
};
const call = async (name, args = {}) => {
const reply = await client.callTool(
@@ -176,13 +223,32 @@ try {
assert.equal(source.modified_sha256, fixture.sha256);
}
const procedures = await call("list_procedures");
for (const name of [
"rea_riscv_entry",
"rea_riscv_leaf",
"rea_riscv_float",
"rea_riscv_probe",
]) {
const procedure = procedures.find((value) => value.value === name);
const names = soft
? [
"rea_riscv_entry",
"rea_riscv_probe",
"rea_riscv_isa_probe",
"rea_riscv_abi_caller",
...Object.keys(RISCV_ILP32_ABI),
]
: [
"rea_riscv_entry",
"rea_riscv_leaf",
"rea_riscv_float",
"rea_riscv_probe",
];
for (const name of [...names, "rea_riscv_global"])
assert.ok(
fixture.symbols[name],
`${name} missing from the readelf symbols`,
);
for (const name of names) {
// A stripped image has no names; the typed twin's addresses locate it.
const procedure = procedures.find((value) =>
stripped
? BigInt(value.address) === BigInt(fixture.symbols[name])
: value.value === name,
);
assert.ok(procedure, `${name} missing from Ghidra inventory`);
assert.equal(BigInt(procedure.address), BigInt(fixture.symbols[name]));
}
@@ -203,6 +269,22 @@ try {
.toString("hex"),
probe.bytes,
);
const isaProbe = [];
if (soft) {
// ESP32-C3's -march declares Zifencei and Zicsr; both must decode.
for (const [offset, bytes, mnemonic] of [
[0n, "0f100000", /^fence\.i$/iu],
[4n, "f32200b0", /^csrr/iu],
]) {
const instruction = await call("inspect_native_instruction", {
address: `0x${(BigInt(fixture.symbols.rea_riscv_isa_probe) + offset).toString(16)}`,
});
assert.equal(instruction.status, "decoded");
assert.equal(instruction.bytes, bytes);
assert.match(instruction.mnemonic, mnemonic);
isaProbe.push(instruction.mnemonic);
}
}
const global = await call("read_bytes", {
address: fixture.symbols.rea_riscv_global,
length: 4,
@@ -211,11 +293,14 @@ try {
const callees = await call("procedure_callees", {
procedure: fixture.symbols.rea_riscv_entry,
});
for (const name of ["rea_riscv_leaf", "rea_riscv_probe"])
for (const name of soft
? ["rea_riscv_abi_caller", "rea_riscv_isa_probe", "rea_riscv_probe"]
: ["rea_riscv_leaf", "rea_riscv_probe"])
assert.ok(
callees.some(
(address) => BigInt(address) === BigInt(fixture.symbols[name]),
),
`${name} missing from the entry's callees`,
);
const dossier = functionDossierSchema.parse(
await call("analyze_function", {
@@ -224,60 +309,50 @@ try {
}),
);
assert.ok(dossier.pseudocode.trim().length > 0);
const floatCode = await call("procedure_pseudo_code", {
procedure: fixture.symbols.rea_riscv_float,
});
assert.ok(typeof floatCode === "string" && floatCode.trim().length > 0);
assert.match(floatCode, /\bdouble\b/u);
const refusals = [];
for (const [flags, reason] of [
[1, "EF_RISCV_FLOAT_ABI_DOUBLE"],
[0x80000005, "(0x80000000)"],
]) {
// Header-mutated copies test admission; they are not new compiler claims.
const rejectedPath = join(
workspace,
`rv${bits}-flags-${flags.toString(16)}.elf`,
);
const rejectedBytes = Buffer.from(sourceBytes);
rejectedBytes.writeUInt32LE(flags, bits === 32 ? 36 : 48);
await writeFile(rejectedPath, rejectedBytes);
const reply = await client.callTool(
{
name: "open_binary",
arguments: { path: rejectedPath, provider_id: "ghidra" },
},
{ timeout: 60000 },
);
const error = requireMcpToolError(reply);
assert.ok(JSON.stringify(error).includes(reason), JSON.stringify(error));
await assert.rejects(
execute(
process.execPath,
[
entrypoint,
"function",
rejectedPath,
"rea_riscv_entry",
"--provider",
"ghidra",
"--json",
],
{
env,
timeout: 60000,
maxBuffer: 1024 * 1024,
},
),
(cause) =>
Number.isInteger(cause.code) &&
cause.code !== 0 &&
`${cause.stdout}\n${cause.stderr}`.includes(reason),
);
refusals.push({ flags, reason });
if (!stripped) {
const floatCode = await call("procedure_pseudo_code", {
procedure:
fixture.symbols[soft ? "rea_riscv_abi_wide" : "rea_riscv_float"],
});
assert.ok(typeof floatCode === "string" && floatCode.trim().length > 0);
assert.match(floatCode, /\bdouble\b/u);
}
const snapshotPath = join(workspace, `rv${bits}.snapshot.json`);
let abi;
if (soft) {
const boundaries = {};
for (const name of Object.keys(RISCV_ILP32_ABI))
boundaries[name] = (
await call("inspect_native_api", { procedure: fixture.symbols[name] })
).boundary;
assertRiscvIlp32Storage(boundaries, { stripped });
const caller = functionDossierSchema.parse(
await call("analyze_function", {
procedure: fixture.symbols.rea_riscv_abi_caller,
detail: "complete",
}),
);
assertRiscvIlp32Caller(caller.pseudocode, fixture.symbols, {
stripped,
});
abi = {
storage: Object.fromEntries(
Object.entries(boundaries).map(([name, boundary]) => [
name,
{
return: boundary.return_type.storage,
parameters: boundary.parameters.map(
(parameter) => parameter.storage,
),
},
]),
),
};
}
const snapshotPath = join(
workspace,
`${fixture.path.split("/").at(-1)}.snapshot.json`,
);
const saved = await call("close_binary", { snapshot_path: snapshotPath });
assert.ok(saved.evidence_records > 0);
const snapshot = parseAnalysisSnapshot(
@@ -288,6 +363,19 @@ try {
assertProfile(snapshot.binding.analysis_profile);
for (const record of snapshot.evidence_bundle.records)
assertEvidence(parseEvidence(record));
const outcomes = snapshot.evidence_bundle.records
.flatMap((record) => parseEvidence(record).limitations)
.map((limitation) => OUTCOME.exec(limitation))
.filter((match) => match !== null);
assert.equal(
outcomes.length > 0,
soft,
"ilp32 prototype outcome missing from, or unexpected in, retained Evidence",
);
// DWARF supplies the typed build's function definitions; stripping removes them.
if (soft)
for (const match of outcomes)
assert.equal(Number(match[1]) > 0, !stripped, match[0]);
await call("open_binary", {
path: fixture.path,
provider_id: "ghidra",
@@ -297,8 +385,11 @@ try {
await call("close_binary");
for (const [command, arguments_] of [
["search", ["rea_riscv_", "--kind", "procedures"]],
["function", ["rea_riscv_entry"]],
...(stripped ? [] : [["search", ["rea_riscv_", "--kind", "procedures"]]]),
[
"function",
[stripped ? fixture.symbols.rea_riscv_entry : "rea_riscv_entry"],
],
]) {
const cli = await execute(
process.execPath,
@@ -316,12 +407,7 @@ try {
const evidence = parseEvidence(JSON.parse(cli.stdout));
assertEvidence(evidence);
if (command === "search")
for (const name of [
"rea_riscv_entry",
"rea_riscv_leaf",
"rea_riscv_float",
"rea_riscv_probe",
])
for (const name of names)
assert.ok(
evidence.normalized_result.some((value) => value.value === name),
);
@@ -333,6 +419,92 @@ try {
assert.ok(cliDossier.pseudocode.trim().length > 0);
}
}
const headers = [];
// Header-mutated copies test admission; they are not new compiler claims.
for (const [flags, outcome] of stripped
? []
: soft
? [
[0x3, "single-float ABI (ilp32f)"],
[0x80000001, "(0x80000000)"],
]
: bits === 32
? [
[0x1, "admitted"],
[0x80000005, "(0x80000000)"],
]
: [
[0x1, "soft-float ABI (lp64)"],
[0x80000005, "(0x80000000)"],
]) {
const mutatedPath = join(
workspace,
`rv${bits}${soft ? "-soft" : ""}-flags-${flags.toString(16)}.elf`,
);
const mutatedBytes = Buffer.from(sourceBytes);
mutatedBytes.writeUInt32LE(flags, bits === 32 ? 36 : 48);
await writeFile(mutatedPath, mutatedBytes);
const reply = await client.callTool(
{
name: "open_binary",
arguments: { path: mutatedPath, provider_id: "ghidra" },
},
{ timeout: 240000 },
);
const cliArguments = [
entrypoint,
"function",
mutatedPath,
"rea_riscv_entry",
"--provider",
"ghidra",
"--json",
];
const cliOptions = { env, timeout: 240000, maxBuffer: 16 * 1024 * 1024 };
if (outcome === "admitted") {
// The double-float image now declares ilp32: refused before, admitted
// now through the soft-float lane with the image's own ISA string.
assert.equal(
requireMcpLifecycleResult(reply, "open_binary").sha256,
createHash("sha256").update(mutatedBytes).digest("hex"),
);
requireMcpLifecycleResult(
await client.callTool(
{ name: "close_binary", arguments: {} },
{ timeout: 240000 },
),
"close_binary",
);
const want = {
lane: "elf32-exec-le-soft-float-generic-v1",
callingConvention: "ilp32-prototype-v1",
riscvElf: {
...expected.riscvElf,
flags,
float_abi_double: false,
float_abi: "soft",
arch: fixture.riscv.isa.arch,
},
};
const cli = await execute(process.execPath, cliArguments, cliOptions);
assertProfile(
parseEvidence(JSON.parse(cli.stdout)).analysis_profile,
want,
);
headers.push({ flags, outcome, lane: want.lane });
continue;
}
const error = requireMcpToolError(reply);
assert.ok(JSON.stringify(error).includes(outcome), JSON.stringify(error));
await assert.rejects(
execute(process.execPath, cliArguments, cliOptions),
(cause) =>
Number.isInteger(cause.code) &&
cause.code !== 0 &&
`${cause.stdout}\n${cause.stderr}`.includes(outcome),
);
headers.push({ flags, outcome: "refused", reason: outcome });
}
assert.equal(
createHash("sha256")
.update(await readFile(fixture.path))
@@ -340,16 +512,38 @@ try {
fixture.sha256,
);
reports.push({
...fixture,
path: fixture.path,
sha256: fixture.sha256,
riscv: fixture.riscv,
stripped,
toolchain: fixture.toolchain,
sources: fixture.sources,
language: expectedLanguage,
lane: expected.lane,
calling_convention: expected.callingConvention ?? null,
procedures: procedures.length,
refusals,
headers,
...(isaProbe.length === 0 ? {} : { isa_probe: isaProbe }),
...(abi === undefined
? {}
: {
abi,
prototype_outcome: outcomes.map((match) => match[0]),
}),
cli: true,
stdio_mcp: true,
snapshot_replay: true,
original_unchanged: true,
});
}
reports.push({
call_sites: {
fixture: typedSoft.path,
command: callSites.command,
version: callSites.version,
calls: callSites.calls.map((value) => value.target),
},
});
} catch (cause) {
failure = cause;
}
+23 -24
View File
@@ -59,18 +59,17 @@ const source = fileURLToPath(
const literalSource = fileURLToPath(
new URL("../tests/conformance/swift/string-literals.swift", import.meta.url),
);
const unresolvedJumpSource = fileURLToPath(
new URL(
"../tests/conformance/swift/unresolved-string-jump.s",
import.meta.url,
),
);
const binary = join(root, "swift-inventory");
const literalBinary = join(root, "swift-literals");
const literalAssembly = fileURLToPath(
new URL("../tests/conformance/swift/decoder-safeguards.S", import.meta.url),
// swiftc rejects assembly inputs, so each source is assembled to an object first.
const literalObjects = ["decoder-safeguards.S", "unresolved-string-jump.s"].map(
(name) => ({
source: fileURLToPath(
new URL(`../tests/conformance/swift/${name}`, import.meta.url),
),
object: join(root, name.replace(/\.[sS]$/u, ".o")),
}),
);
const literalObject = join(root, "decoder-safeguards.o");
const reports = [];
let literalReport;
let decoderCleaned = true;
@@ -81,20 +80,21 @@ try {
await mkdir(join(root, "copied"));
const copied = join(root, "copied", "swift-inventory");
await copyFile(binary, copied);
await execute(
"xcrun",
[
"clang",
"-target",
"arm64-apple-macosx13.0",
"-c",
literalAssembly,
"-o",
literalObject,
],
{ timeout: 60_000 },
);
// Decoding covers AArch64 register pairs, so build that slice on any Mac host.
for (const { source: assembly, object } of literalObjects)
await execute(
"xcrun",
[
"clang",
"-target",
"arm64-apple-macosx13.0",
"-c",
assembly,
"-o",
object,
],
{ timeout: 60_000 },
);
await execute(
"swiftc",
[
@@ -102,8 +102,7 @@ try {
"-target",
"arm64-apple-macosx13.0",
literalSource,
literalObject,
unresolvedJumpSource,
...literalObjects.map(({ object }) => object),
"-o",
literalBinary,
],
+3 -3
View File
@@ -8,20 +8,20 @@ import { fileURLToPath } from "node:url";
import { parseConfig } from "../dist/config/parseConfig.js";
import { resolveGhidraAnalysisProfile } from "../dist/ghidra/GhidraAnalysisProfile.js";
import { GhidraClient } from "../dist/ghidra/GhidraClient.js";
import { GHIDRA_FUNCTION_OPERATIONS } from "../dist/ghidra/GhidraFunctionOperations.js";
import {
GHIDRA_FUNCTION_OPERATIONS,
parseGhidraFunctionInput,
parseGhidraFunctionResult,
} from "../dist/ghidra/GhidraFunctionValues.js";
import { GHIDRA_INVENTORY_OPERATIONS } from "../dist/ghidra/GhidraInventoryOperations.js";
import {
GHIDRA_INVENTORY_OPERATIONS,
parseGhidraInventoryInput,
parseGhidraInventoryResult,
} from "../dist/ghidra/GhidraInventoryValues.js";
import { inspectVerifierGhidraInstallation } from "./lib/ghidra-installation-inspection.mjs";
import { SUPPORTED_GHIDRA_VERSION } from "../dist/ghidra/GhidraReleasePolicy.js";
import { GhidraHeadlessLauncher } from "../dist/ghidra/GhidraLauncher.js";
import { GHIDRA_PROVIDER_IDENTITY } from "../dist/ghidra/GhidraProviderCapabilities.js";
import { GHIDRA_PROVIDER_IDENTITY } from "../dist/ghidra/GhidraProviderMetadata.js";
import {
WINDOWS_P0_NATIVE_PE_ARCHITECTURES,
windowsP0Capabilities,
+11 -1
View File
@@ -13,7 +13,7 @@ import { GhidraClient } from "../dist/ghidra/GhidraClient.js";
import { inspectVerifierGhidraInstallation } from "./lib/ghidra-installation-inspection.mjs";
import { SUPPORTED_GHIDRA_VERSION } from "../dist/ghidra/GhidraReleasePolicy.js";
import { GhidraHeadlessLauncher } from "../dist/ghidra/GhidraLauncher.js";
import { GHIDRA_PROVIDER_IDENTITY } from "../dist/ghidra/GhidraProviderCapabilities.js";
import { GHIDRA_PROVIDER_IDENTITY } from "../dist/ghidra/GhidraProviderMetadata.js";
import {
assertCleanup,
@@ -32,6 +32,7 @@ import {
} from "./verify-real-ghidra-inventory.mjs";
import { completeVerifierRun, createVerifierRun } from "./lib/verifier-run.mjs";
import { hasTypedSwitchEvidence } from "./verify-real-ghidra-switch-assertions.mjs";
import { verifyGhidraRelocationSymbols } from "./lib/real-ghidra-relocation-symbols.mjs";
import {
verifyDenseDefaultReturn,
@@ -281,6 +282,14 @@ try {
}
await writeFile(malformedPath, Buffer.from("not-a-binary\n", "utf8"));
const relocationSymbols = crossFormat
? await verifyGhidraRelocationSymbols({
clang,
lld,
env: process.env,
})
: null;
const debug = await verifyTarget(debugPath, "debug", expectedNativeTarget);
assertDebugFixture(debug);
const layout = await verifyTarget(
@@ -331,6 +340,7 @@ try {
? (crossResults[0]?.native_api_cli ?? null)
: jumpTable.native_api_cli,
native_value_e2e: debug.native_values,
relocation_symbols: relocationSymbols,
custom_target: custom === null ? null : summary(custom),
cleanup: "complete",
};
+1 -1
View File
@@ -11,7 +11,7 @@ import { AnalysisCleanupLease } from "../dist/artifacts/AnalysisCleanupLease.js"
import { resolveHopperAnalysisProfile } from "../dist/hopper/HopperAnalysisProfile.js";
import { HopperApplicationLauncher } from "../dist/hopper/BridgeLauncher.js";
import { HopperClient } from "../dist/hopper/HopperClient.js";
import { HOPPER_PROVIDER_IDENTITY } from "../dist/hopper/HopperProviderCapabilities.js";
import { HOPPER_PROVIDER_IDENTITY } from "../dist/hopper/HopperProviderMetadata.js";
import { loadRealHopperFixtureTargets } from "./lib/real-hopper-fixture.mjs";
// These options belong to the native client, below the CLI/MCP tool surface.
+1 -1
View File
@@ -11,7 +11,7 @@ import {
requireFunctionDossier,
requirePseudocode,
} from "./lib/real-hopper-assertions.mjs";
import { HOPPER_PROVIDER_IDENTITY } from "../dist/hopper/HopperProviderCapabilities.js";
import { HOPPER_PROVIDER_IDENTITY } from "../dist/hopper/HopperProviderMetadata.js";
import { HOPPER_TARGET_LEASE_DIRECTORY } from "../dist/process/HopperLease.js";
import { REA_WORKFLOW_PROVIDER } from "../dist/domain/investigationProviders.js";
import { loadRealHopperFixtureTargets } from "./lib/real-hopper-fixture.mjs";
+48 -7
View File
@@ -35,6 +35,7 @@ import {
verifyKunaStrings,
} from "./lib/kuna-string-verifier.mjs";
import { nativeKunaIncomingSites } from "./lib/kuna-native-reference-verifier.mjs";
import { MULTI_MODE_REMEDIATION } from "../dist/application/CapabilityInventory.js";
import {
assertVerifierCleanupComplete,
completeVerifierRun,
@@ -115,7 +116,9 @@ __attribute__((noinline)) int rea_classify(int c) {
}
int main(int argc, char **argv) {
if (argc < 2) return 2;
printf("%u %d\n", rea_checksum((const unsigned char *)argv[1], strlen(argv[1])),
size_t length = strlen(argv[1]);
printf("%u %u %d\n", rea_checksum((const unsigned char *)argv[1], length),
rea_checksum((const unsigned char *)argv[1], length ? length - 1 : 0),
rea_classify(argv[1][0]));
return 0;
}
@@ -282,8 +285,12 @@ const quality = (text) => ({
const verifyKuna = async (
path,
functions,
expectedCaller,
expectedStrings = [],
{
expectedCaller,
expectedStrings = [],
verifyNamedTrace = false,
minimumTraceReferences = {},
} = {},
) => {
const sha256 = createHash("sha256")
.update(await readFile(path))
@@ -292,6 +299,36 @@ const verifyKuna = async (
const answers = [];
let coldFailure;
try {
const status = await cold.call("binary_session", { detail: "complete" });
if (status.value.format === "elf") {
const availability = status.value.tool_availability;
const dispatch = availability.find(
({ name }) => name === "inspect_native_dispatch_metadata",
);
const trace = availability.find(
({ name }) => name === "trace_native_ui_action",
);
assert.equal(dispatch?.available, false);
assert.equal(
trace?.modes?.find(({ name }) => name === "native_seed_call_trace")
?.available,
false,
);
if (process.platform === "darwin") {
const authored = trace?.modes?.find(
({ name }) => name === "authored_ui_dispatch",
);
const nativeSeed = trace?.modes?.find(
({ name }) => name === "native_seed_call_trace",
);
assert.equal(trace?.remediation, MULTI_MODE_REMEDIATION);
assert.notEqual(authored?.remediation, nativeSeed?.remediation);
assert.ok(
authored?.missing_operations.includes("decode_interface_builder"),
);
assert.ok(nativeSeed?.missing_operations.includes("list_names"));
}
}
answers.push({
strings: await verifyKunaStrings({
call: cold.call,
@@ -320,6 +357,8 @@ const verifyKuna = async (
name,
address,
expectedCaller,
verifyNamedTrace,
minimumTraceReferences: minimumTraceReferences[name],
nativeIncoming: (targetAddress) =>
nativeKunaIncomingSites({
executable: kuna,
@@ -549,10 +588,12 @@ try {
for (const binary of [built, stripped])
results.push({
binary: binary.slice(workspace.length + 1),
kuna: await verifyKuna(binary, [...fixtureFunctions], fixtureMain, [
"_φ( °-°)/ REA unicode marker",
"REA wide marker",
]),
kuna: await verifyKuna(binary, [...fixtureFunctions], {
expectedCaller: fixtureMain,
expectedStrings: ["_φ( °-°)/ REA unicode marker", "REA wide marker"],
verifyNamedTrace: binary === built,
minimumTraceReferences: { rea_checksum: 2 },
}),
});
// Caller-built matrix.
+136
View File
@@ -0,0 +1,136 @@
import assert from "node:assert/strict";
import { access, readFile } from "node:fs/promises";
import { join } from "node:path";
import {
execute,
failedCommand,
options,
source,
spell,
tool,
} from "./esp-image-tools.mjs";
// Optional verification tools only; the runtime reader starts no tool.
const idf = process.env.REA_ESP_IDF_PATH;
const python = process.env.REA_ESPTOOL_PYTHON ?? "python3";
const generator = () =>
join(idf, "components", "partition_table", "gen_esp32part.py");
const environment = () => ({ ...process.env, IDF_PATH: idf });
/** Check the ESP-IDF checkout before any work and report its version. */
export const checkEspIdf = async () => {
if (idf === undefined)
throw new Error(
"verify:esp-flash needs an ESP-IDF checkout for gen_esp32part.py and otatool.py; set REA_ESP_IDF_PATH",
);
try {
await access(generator());
await access(join(idf, "components", "app_update", "otatool.py"));
const version = await readFile(
join(idf, "tools", "cmake", "version.cmake"),
"utf8",
);
const part = (name) =>
new RegExp(`IDF_VERSION_${name} (\\d+)`, "u").exec(version)?.[1];
return `v${part("MAJOR")}.${part("MINOR")}.${part("PATCH")}`;
} catch (failure) {
throw new Error(
`verify:esp-flash needs gen_esp32part.py, otatool.py and tools/cmake/version.cmake under REA_ESP_IDF_PATH (${idf}). ${failure.message}`,
{ cause: failure },
);
}
};
/** Build a binary table from CSV with ESP-IDF's generator. */
export const buildTable = (csv, output, flags = []) =>
execute(python, ["-I", generator(), "--quiet", ...flags, csv, output], {
...options,
env: environment(),
});
const bytesOf = (text) => {
const value = text.trim();
if (value.endsWith("M")) return Number(value.slice(0, -1)) * 0x10_0000;
if (value.endsWith("K")) return Number(value.slice(0, -1)) * 0x400;
return Number(value);
};
/** ESP-IDF's own reading of a binary table, as the CSV it prints. */
export const decodeTable = async (binary) => {
const { stdout } = await execute(python, ["-I", generator(), binary], {
...options,
env: environment(),
});
return stdout
.split("\n")
.filter((line) => line.trim() !== "" && !line.startsWith("#"))
.map((line) => {
const [name, type, subtype, offset, size, flags = ""] = line
.split(",")
.map((field) => field.trim());
return {
name,
type,
subtype,
offset: bytesOf(offset),
size: bytesOf(size),
encrypted: flags.split(":").includes("encrypted"),
readonly: flags.split(":").includes("readonly"),
};
});
};
/** Run otatool against a dump file through the conformance shim. */
export const otatool = async (image, flashOffset, args) => {
const run = await execute(
python,
[
"-I",
source("flash_file_tool.py"),
image,
`0x${flashOffset.toString(16)}`,
"otatool",
...args,
],
{ ...options, env: environment() },
).then(
({ stdout, stderr }) => `${stdout}${stderr}`,
failedCommand(`otatool ${args.join(" ")}`),
);
return run;
};
/** The two records otatool read_otadata prints. */
export const readOtadata = async (image, flashOffset, tableOffset) => {
const text = await otatool(image, flashOffset, [
"--partition-table-offset",
`0x${tableOffset.toString(16)}`,
"read_otadata",
]);
const row =
/Firmware:\s+0x([0-9a-f]{8})\s+0x([0-9a-f]{8})\s*\|\s*0x([0-9a-f]{8})\s+0x([0-9a-f]{8})/u.exec(
text,
);
assert.ok(row, `Unrecognized read_otadata output: ${text}`);
return [
{ seq: Number.parseInt(row[1], 16), crc: Number.parseInt(row[2], 16) },
{ seq: Number.parseInt(row[3], 16), crc: Number.parseInt(row[4], 16) },
];
};
/** Assemble a flash image with esptool merge-bin. */
export const mergeBin = (chip, output, parts, flags = []) =>
tool("esptool", [
"--chip",
chip,
spell("merge_bin", "merge-bin"),
"-o",
output,
...flags,
...parts.flatMap(({ at, file }) => [`0x${at.toString(16)}`, file]),
]);
export const padTo = (size) => [
spell("--fill-flash-size", "--pad-to-size"),
size,
];
+419
View File
@@ -0,0 +1,419 @@
#!/usr/bin/env node
import assert from "node:assert/strict";
import { createHash } from "node:crypto";
import { mkdtemp, readFile, rm, writeFile } from "node:fs/promises";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { withVerifierCleanup } from "../../lib/verifier-cleanup.mjs";
import {
assertAgrees,
buildElf,
checkTools,
commands,
elf2image,
extract,
imageInfo,
inspect,
} from "./esp-image-tools.mjs";
import {
buildTable,
checkEspIdf,
decodeTable,
mergeBin,
otatool,
padTo,
readOtadata,
} from "./esp-flash-tools.mjs";
let workspace;
const path = (name) => join(workspace, name);
const sha256 = (bytes) => createHash("sha256").update(bytes).digest("hex");
const DEFAULT_CSV = `# Name, Type, SubType, Offset, Size, Flags
nvs,data,nvs,0x9000,0x4000,
otadata,data,ota,0xd000,0x2000,
phy_init,data,phy,0xf000,0x1000,
factory,app,factory,0x10000,0x20000,
ota_0,app,ota_0,0x30000,0x20000,
ota_1,app,ota_1,0x50000,0x20000,
keys,data,nvs_keys,0x70000,0x1000,encrypted
`;
// Blank offsets: gen_esp32part places each partition after the moved table.
const RELOCATED_CSV = `# Name, Type, SubType, Offset, Size, Flags
nvs,data,nvs,,0x4000,
otadata,data,ota,,0x2000,
factory,app,factory,,0x20000,
`;
// ESP-IDF 5.3 entries over the bootloader region and the table sector;
// gen_esp32part fills their ranges from --primary-bootloader-offset.
const PRIMARY_CSV = `# Name, Type, SubType, Offset, Size, Flags
PrimaryBTLDR,bootloader,primary,N/A,N/A,
PrimaryPrtTable,partition_table,primary,N/A,N/A,
nvs,data,nvs,0x9000,0x4000,
factory,app,factory,0x10000,0x20000,
`;
/** Nested ESP image facts, shaped like a top-level inventory for assertAgrees. */
const nested = (facts) => ({
inventory: { manifest: { root_format: "esp-image" } },
root: facts.image,
segments: facts.segments.map((segment) => ({ format_metadata: segment })),
});
const appFacts = (rea, label) => {
const row = rea.inventory.occurrences.find(
({ format_metadata: metadata }) =>
metadata?.kind === "esp-flash-partition" && metadata.label === label,
);
return rea.root.apps.find(
({ partition_index: index }) => index === row?.format_metadata.index,
)?.facts;
};
/** Every declared partition agrees with ESP-IDF's decoding of the table REA extracted. */
const assertTableAgrees = async (name, rea, extraction) => {
const decoded = await decodeTable(
join(extraction.output_root, "partition-table.bin"),
);
const declared = [
...rea.inventory.occurrences
.filter(({ format_metadata: m }) => m?.kind === "esp-flash-partition")
.map(({ format_metadata: m }) => m),
...rea.root.partitions_outside_dump,
].sort((left, right) => left.index - right.index);
assert.deepEqual(
declared.map((p) => ({
name: p.label,
type: p.type_name,
subtype: p.subtype_name,
offset: p.flash_address,
size: p.size,
encrypted: p.encrypted,
readonly: p.readonly,
})),
decoded,
`${name} table`,
);
};
/** Each extracted file is the dump's bytes at its range; the source is unchanged. */
const assertExtraction = async (name, file, rea, flags = []) => {
const bytes = await readFile(file);
const extraction = await extract(file, workspace, flags);
assert.equal(extraction.containment_verified, true, name);
assert.equal(
extraction.manifest.root_sha256,
sha256(bytes),
`${name} source`,
);
const rows = rea.inventory.occurrences.filter(
({ logical_path }) => logical_path !== ".",
);
assert.equal(extraction.artifacts.length, rows.length, `${name} entries`);
for (const { logical_path: entry, source_location: range } of rows) {
const written = await readFile(join(extraction.output_root, entry));
assert.ok(
written.equals(bytes.subarray(range.offset, range.offset + range.length)),
`${name} ${entry}`,
);
}
assert.equal(sha256(await readFile(file)), sha256(bytes), `${name} source`);
return extraction;
};
const build = async () => {
const app = await buildElf("app.c", "app.ld", path("app.elf"));
const boot = await buildElf(
"bootloader.c",
"bootloader.ld",
path("boot.elf"),
);
const descriptor = ["--elf-sha256-offset", "0xb0"];
await elf2image("esp32c3", app, path("app.bin"), descriptor);
await elf2image("esp32c3", boot, path("boot.bin"));
// esptool converts the RISC-V ELF for any chip; these use ESP32 addresses.
const app32 = await buildElf(
"app.c",
"esp32-app-riscv.ld",
path("app32.elf"),
);
const boot32 = await buildElf(
"bootloader.c",
"esp32-bootloader.ld",
path("boot32.elf"),
);
await elf2image("esp32", app32, path("app32.bin"), descriptor);
await elf2image("esp32", boot32, path("boot32.bin"));
await writeFile(path("default.csv"), DEFAULT_CSV);
await writeFile(path("relocated.csv"), RELOCATED_CSV);
await writeFile(path("primary.csv"), PRIMARY_CSV);
await buildTable(path("default.csv"), path("default-table.bin"));
await buildTable(path("relocated.csv"), path("relocated-table.bin"), [
"--offset",
"0xa000",
]);
await buildTable(path("primary.csv"), path("primary-table.bin"), [
"--primary-bootloader-offset",
"0x0",
]);
};
const assertOtaAgrees = async (name, file, rea) => {
const records = await readOtadata(file, 0, 0x8000);
assert.deepEqual(
rea.root.otadata.records.map(({ ota_seq: seq, crc }) => ({ seq, crc })),
records,
`${name} otadata`,
);
};
const defaultLayout = async (report) => {
const file = path("c3.bin");
await mergeBin(
"esp32c3",
file,
[
{ at: 0x0, file: path("boot.bin") },
{ at: 0x8000, file: path("default-table.bin") },
{ at: 0x1_0000, file: path("app.bin") },
{ at: 0x3_0000, file: path("app.bin") },
],
padTo("1MB"),
);
await otatool(file, 0, ["switch_ota_partition", "--slot", "0"]);
const rea = await inspect(file);
assert.equal(rea.inventory.manifest.root_format, "esp-flash-image");
assert.equal(rea.inventory.coverage.status, "complete");
assert.deepEqual(rea.root.undescribed, []);
assert.equal(rea.root.partition_table.method, "default");
assert.equal(rea.root.partition_table.md5.status, "valid");
assertAgrees(
"c3 bootloader",
await imageInfo(path("boot.bin")),
nested(rea.root.bootloader.facts),
);
const appOracle = await imageInfo(path("app.bin"));
for (const label of ["factory", "ota_0"])
assertAgrees(`c3 ${label}`, appOracle, nested(appFacts(rea, label)));
assert.equal(appFacts(rea, "ota_1").status, "erased");
await assertOtaAgrees("c3 slot 0", file, rea);
assert.deepEqual(
[rea.root.boot_selection.slot, rea.root.boot_selection.basis],
[0, "otadata"],
);
const extraction = await assertExtraction("c3", file, rea);
await assertTableAgrees("c3", rea, extraction);
// A second switch writes the other record; the higher sequence selects ota_1.
await otatool(file, 0, ["switch_ota_partition", "--slot", "1"]);
const switched = await inspect(file);
await assertOtaAgrees("c3 slot 1", file, switched);
assert.equal(switched.root.boot_selection.slot, 1);
assert.equal(switched.root.boot_selection.running_partition, "unknown");
// Erased records leave the bootloader's factory fallback.
await otatool(file, 0, ["erase_otadata"]);
const erased = await inspect(file);
await assertOtaAgrees("c3 erased", file, erased);
const factory = erased.inventory.occurrences.find(
({ format_metadata: m }) => m?.label === "factory",
);
assert.deepEqual(
[
erased.root.boot_selection.basis,
erased.root.boot_selection.partition_index,
],
["factory-fallback", factory?.format_metadata.index],
);
report.cases.push({
name: "c3.bin",
partitions: extraction.artifacts.length - 2,
selected: [0, 1, "factory-fallback"],
});
};
const primaryEntries = async (report) => {
const file = path("primary.bin");
await mergeBin(
"esp32c3",
file,
[
{ at: 0x0, file: path("boot.bin") },
{ at: 0x8000, file: path("primary-table.bin") },
{ at: 0x1_0000, file: path("app.bin") },
],
padTo("1MB"),
);
const rea = await inspect(file);
assert.equal(rea.inventory.coverage.status, "complete");
assert.deepEqual(
rea.inventory.limitations.filter((text) =>
/overlaps|before the end of the partition table/u.test(text),
),
[],
"primary entries are not misplaced",
);
const extraction = await assertExtraction("primary", file, rea);
await assertTableAgrees("primary", rea, extraction);
// Without otadata the bootloader selects the factory index.
const factory = rea.inventory.occurrences.find(
({ format_metadata: m }) => m?.label === "factory",
);
assert.deepEqual(
[
rea.root.otadata,
rea.root.boot_selection.basis,
rea.root.boot_selection.partition_index,
],
[null, "no-otadata", factory?.format_metadata.index],
);
report.cases.push({
name: "primary.bin",
partitions: extraction.artifacts.length - 2,
});
};
const relocated = async (report) => {
const [, , factory] = await decodeTable(path("relocated-table.bin"));
const file = path("relocated.bin");
await mergeBin(
"esp32c3",
file,
[
{ at: 0x0, file: path("boot.bin") },
{ at: 0xa000, file: path("relocated-table.bin") },
{ at: factory.offset, file: path("app.bin") },
],
padTo("1MB"),
);
const scanned = await inspect(file);
assert.deepEqual(
[
scanned.root.partition_table.flash_address,
scanned.root.partition_table.method,
],
[0xa000, "scan"],
);
const caller = [
"--target-format",
"esp-flash-image",
"--partition-table-offset",
"0xa000",
];
const selected = await inspect(file, caller);
assert.equal(selected.root.partition_table.method, "caller");
await assertTableAgrees(
"relocated",
selected,
await assertExtraction("relocated", file, selected, caller),
);
report.cases.push({
name: "relocated.bin",
table: "0xa000",
methods: ["scan", "caller"],
});
};
const esp32Origin = async (report) => {
const parts = [
{ at: 0x1000, file: path("boot32.bin") },
{ at: 0x8000, file: path("default-table.bin") },
{ at: 0x1_0000, file: path("app32.bin") },
];
const whole = path("esp32.bin");
await mergeBin("esp32", whole, parts, padTo("1MB"));
const rea = await inspect(whole);
assert.deepEqual(
[rea.root.chip.target, rea.root.bootloader.flash_address],
["esp32", 0x1000],
);
assertAgrees(
"esp32 factory",
await imageInfo(path("app32.bin")),
nested(appFacts(rea, "factory")),
);
// esptool writes the same parts as a dump that starts at flash 0x1000.
const partial = path("esp32-from-0x1000.bin");
await mergeBin("esp32", partial, parts, ["--target-offset", "0x1000"]);
const guessed = await inspect(partial);
assert.equal(
guessed.inventory.manifest.root_format,
"esp-image",
"no origin, no layout",
);
// The table at file offset 0x7000 shows where the dump was read from.
assert.deepEqual(guessed.root.flash_dump, {
flash_offset: 0x1000,
table_file_offset: 0x7000,
});
const flags = [
"--target-format",
"esp-flash-image",
"--flash-offset",
"0x1000",
];
const origin = await inspect(partial, flags);
assert.deepEqual(origin.root.flash_offset, {
value: 0x1000,
source: "caller",
});
await assertExtraction("esp32 from 0x1000", partial, origin, flags);
report.cases.push({
name: "esp32-from-0x1000.bin",
automatic: "esp-image",
flash_offset: "0x1000",
});
};
const damaged = async (report) => {
const bytes = await readFile(path("c3.bin"));
const corrupt = Buffer.from(bytes);
// Flip a byte of the factory app's first segment data.
corrupt[0x1_0000 + 0x18 + 8] ^= 0x01;
await writeFile(path("corrupt.bin"), corrupt);
const rea = await inspect(path("corrupt.bin"));
assert.equal(appFacts(rea, "factory").image.checksum.status, "invalid");
assert.equal(appFacts(rea, "ota_0").image.checksum.status, "valid");
await writeFile(path("truncated.bin"), bytes.subarray(0, 0x4_0000));
const cut = await inspect(path("truncated.bin"));
// Partitions past the end of the file are entries holding no bytes.
assert.equal(cut.inventory.coverage.status, "complete");
assert.deepEqual(cut.root.partitions_outside_dump, []);
const empty = cut.inventory.occurrences
.filter(({ declared_size: size }) => size === 0)
.map(({ format_metadata: m }) => m?.label);
assert.deepEqual(empty, ["keys", "ota_1"]);
assert.ok(
cut.inventory.occurrences.some(
({ logical_path }) => logical_path === "partitions/factory.bin",
),
);
report.cases.push(
{ name: "corrupt.bin", factory: "invalid" },
{ name: "truncated.bin", without_bytes: empty },
);
};
await withVerifierCleanup(async () => {
const tools = await checkTools("verify:esp-flash");
const espIdf = await checkEspIdf();
workspace = await mkdtemp(join(tmpdir(), "rea-esp-flash-proof-"));
const report = {
esptool: tools.esptoolVersion,
esp_idf: espIdf,
compiler: tools.compilerVersion,
cases: [],
};
await build();
await defaultLayout(report);
await primaryEntries(report);
await relocated(report);
await esp32Origin(report);
await damaged(report);
console.log(JSON.stringify(report, null, 2));
}, [
// Remove outputs only after every owned command tree has stopped.
async () => {
await commands.close();
if (workspace !== undefined)
await rm(workspace, { recursive: true, force: true });
},
]);
+33 -15
View File
@@ -22,6 +22,13 @@ export const exitOutput = (failure) => {
if (typeof failure?.code !== "number" || failure.code === 0) throw failure;
return `${failure.snapshot.stdout.text}${failure.snapshot.stderr.text}`;
};
/** Reject a command that had to succeed, with the output it exited with. */
export const failedCommand = (label) => (failure) => {
throw new Error(`${label} failed: ${exitOutput(failure)}`, {
cause: failure,
});
};
export const source = (name) =>
fileURLToPath(
new URL(`../../../tests/conformance/esp/${name}`, import.meta.url),
@@ -41,13 +48,16 @@ export const tool = async (module, args) => {
return `${stdout}${stderr}`;
};
/** The npm script whose tools are checked, named in each failure. */
let lane = "verify:esp-images";
/** Check one tool before any work; a failure names the variable to set. */
const prerequisite = async (needed, variable, check) => {
try {
return await check();
} catch (failure) {
throw new Error(
`verify:esp-images needs ${needed}; set ${variable}. ${failure.message}`,
`${lane} needs ${needed}; set ${variable}. ${failure.message}`,
{ cause: failure },
);
}
@@ -63,7 +73,8 @@ export const spell = (v4, later) => {
};
/** Check every tool before any work and report their versions. */
export const checkTools = async () => {
export const checkTools = async (script = "verify:esp-images") => {
lane = script;
const compilerVersion = await prerequisite(
`a Clang with the RISC-V target (tried ${compiler})`,
"REA_RISCV_CLANG",
@@ -98,7 +109,7 @@ export const checkTools = async () => {
);
if (version[0] !== PINNED_ESPTOOL)
console.warn(
`verify:esp-images: esptool ${version[0]} is not the pinned ${PINNED_ESPTOOL}, whose sources the reader's layout facts cite.`,
`${lane}: esptool ${version[0]} is not the pinned ${PINNED_ESPTOOL}, whose sources the reader's layout facts cite.`,
);
const xtensaVersion =
xtensa === undefined
@@ -201,19 +212,20 @@ export const imageInfo = async (file, chip) => {
};
};
/** REA's complete inline inventory for one file through the built CLI. */
export const inspect = async (file) => {
const result = await execute(
const inspectArtifact = (file, flags) =>
execute(
process.execPath,
[entrypoint, "inspect-artifact", file, "--json"],
[entrypoint, "inspect-artifact", file, ...flags, "--json"],
{ ...options, env: { ...process.env, REA_LOG_LEVEL: "silent" } },
).then(
({ stdout }) => ({ ok: true, stdout }),
(failure) => ({ ok: false, stdout: exitOutput(failure) }),
);
if (!result.ok) return { refused: result.stdout };
const inventory = JSON.parse(result.stdout).normalized_result.substeps[0]
.evidence.normalized_result;
/** REA's complete inline inventory for one file through the built CLI. */
export const inspect = async (file, flags = []) => {
const { stdout } = await inspectArtifact(file, flags).catch(
failedCommand(`inspect-artifact ${file}`),
);
const inventory =
JSON.parse(stdout).normalized_result.substeps[0].evidence.normalized_result;
const rows = inventory.occurrences.filter((row) => row.logical_path !== ".");
return {
inventory,
@@ -228,6 +240,12 @@ export const inspect = async (file) => {
};
};
/** REA's output for a file it must refuse; opening the file fails the lane. */
export const refusal = (file, flags = []) =>
inspectArtifact(file, flags).then(() => {
throw new Error(`inspect-artifact opened ${file}, which it must refuse`);
}, exitOutput);
const oracleDescriptorKind = (oracle) => {
if (oracle.project !== undefined) return "app";
if (oracle.bootloaderVersion !== undefined) return "bootloader";
@@ -235,10 +253,10 @@ const oracleDescriptorKind = (oracle) => {
};
/** Extract every entry through the built CLI into a new directory in `root`. */
export const extract = async (file, root) => {
export const extract = async (file, root, flags = []) => {
const { stdout } = await execute(
process.execPath,
[entrypoint, "extract-artifact", file, "--json"],
[entrypoint, "extract-artifact", file, ...flags, "--json"],
{
...options,
env: {
+3 -3
View File
@@ -17,6 +17,7 @@ import {
imageInfo,
inspect,
options,
refusal,
source,
spell,
tool,
@@ -392,7 +393,7 @@ await withVerifierCleanup(async () => {
const v3Oracle = await imageInfo(path("esp8266-v3.bin"), "esp8266");
assert.ok((v3Oracle.segments[0]?.address ?? 0) >= 0x4020_0000);
assert.match(
(await inspect(path("esp8266-v3.bin"))).refused ?? "",
await refusal(path("esp8266-v3.bin")),
/ESP8266 firmware images are not supported/u,
);
report.images.push({ name: "esp8266-v3.bin", refused: true });
@@ -400,9 +401,8 @@ await withVerifierCleanup(async () => {
// ESP8266 v1: esptool falls back to ESP8266; REA refuses it by name.
const esp8266Image = path("esp8266-0x00000.bin");
assert.equal((await imageInfo(esp8266Image)).detected, "ESP8266");
const refused = await inspect(esp8266Image);
assert.match(
refused.refused ?? "",
await refusal(esp8266Image),
/ESP8266 firmware images are not supported/u,
);
report.images.push({ name: "esp8266-0x00000.bin", refused: true });
+284
View File
@@ -0,0 +1,284 @@
#!/usr/bin/env node
import assert from "node:assert/strict";
import { mkdtemp, readFile, rm, writeFile } from "node:fs/promises";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { fileURLToPath } from "node:url";
import { createVerifierCommands } from "../../lib/verify-commands.mjs";
import { withVerifierCleanup } from "../../lib/verifier-cleanup.mjs";
// Optional verification tool only; the UF2 reader starts no tool.
const python = process.env.REA_ESPTOOL_PYTHON ?? "python3";
const entrypoint = fileURLToPath(new URL("../../rea.mjs", import.meta.url));
const options = { maxBuffer: 16 * 1024 * 1024, timeout: 120_000 };
/** Every tool runs as an owned process tree that close() stops. */
const commands = await createVerifierCommands();
let workspace;
const path = (name) => join(workspace, name);
/** Deterministic input bytes; lengths include one that is not word-aligned. */
const input = (length, seed) =>
Buffer.from(Array.from({ length }, (_, index) => (index * 13 + seed) & 0xff));
// `otadata` matches the table's block count, so esptool's per-input numbering
// from 0 puts both inputs in one numbering group.
const FILES = {
bootloader: input(1000, 1),
table: input(3072, 2),
otadata: input(3072, 4),
app: input(501, 3),
};
const CASES = [
{
name: "esp32s3.uf2",
chip: "esp32s3",
layout: [
[0x0, "bootloader"],
[0x8000, "table"],
[0xd000, "otadata"],
[0x10000, "app"],
],
extra: [],
md5: true,
payload: 452,
},
{
name: "esp32s2-tinyuf2.uf2",
chip: "esp32s2",
layout: [
[0x1000, "bootloader"],
[0x8000, "table"],
],
extra: ["--chunk-size", "256", "--md5-disable"],
md5: false,
payload: 256,
},
];
const tool = async (args) => {
const { stdout, stderr } = await commands.exec(
python,
["-m", "esptool", ...args],
options,
);
return `${stdout}${stderr}`;
};
/** esptool's own family ID for a chip: the oracle REA's table must match. */
const familyOf = async (chip) => {
const { stdout } = await commands.exec(
python,
[
"-c",
"import sys, esptool.targets as t; print(t.CHIP_DEFS[sys.argv[1]].UF2_FAMILY_ID)",
chip,
],
options,
);
return Number(stdout.trim());
};
const rea = async (command, file) => {
const { stdout } = await commands.exec(
process.execPath,
[entrypoint, command, file, "--json"],
{
...options,
env: {
...process.env,
REA_LOG_LEVEL: "silent",
TMPDIR: workspace,
TMP: workspace,
TEMP: workspace,
},
},
);
return JSON.parse(stdout).normalized_result;
};
const familyDirectory = (familyId) =>
`family-0x${familyId.toString(16).padStart(8, "0")}`;
const hex32 = (value) => `0x${value.toString(16).padStart(8, "0")}`;
const blocksOf = (entry, name) => Math.ceil(FILES[name].length / entry.payload);
const inventoryOf = async (file) =>
(await rea("inspect-artifact", path(file))).substeps[0].evidence
.normalized_result;
const rootOf = (inventory) =>
inventory.occurrences.find((row) => row.logical_path === ".").format_metadata;
/** Write the inputs, merge each case with esptool, and concatenate the cases. */
const produce = async (v5) => {
for (const [name, bytes] of Object.entries(FILES))
await writeFile(path(`${name}.bin`), bytes);
for (const entry of CASES) {
await tool([
"--chip",
entry.chip,
v5 ? "merge-bin" : "merge_bin",
"--format",
"uf2",
...entry.extra,
"-o",
path(entry.name),
...entry.layout.flatMap(([address, name]) => [
hex32(address),
path(`${name}.bin`),
]),
]);
entry.familyId = await familyOf(entry.chip);
}
// The UF2 specification makes a multi-family file by concatenation.
await writeFile(
path("concatenated.uf2"),
Buffer.concat(
await Promise.all(CASES.map((entry) => readFile(path(entry.name)))),
),
);
};
/** Root facts a complete file from these cases must report. */
const expectedRoot = (included) => {
const blocks = included.flatMap((entry) =>
entry.layout.map(([, name]) => ({
group: `${entry.chip}:${String(blocksOf(entry, name))}`,
count: blocksOf(entry, name),
md5: entry.md5,
})),
);
const total = blocks.reduce((sum, { count }) => sum + count, 0);
// Inputs of one chip with equal block counts form one numbering group,
// and every input after the first in a group repeats its numbers.
const groups = new Map();
for (const { group, count } of blocks)
groups.set(group, [...(groups.get(group) ?? []), count]);
return {
block_count: total,
invalid_block_count: 0,
flash_block_count: total,
md5_block_count: blocks
.filter(({ md5 }) => md5)
.reduce((sum, { count }) => sum + count, 0),
family_count: included.length,
numbering_group_count: groups.size,
missing_block_count: 0,
ambiguous_numbering_group_count: 0,
repeated_block_count: [...groups.values()].reduce(
(sum, counts) => sum + (counts.length - 1) * (counts[0] ?? 0),
0,
),
duplicate_bytes: 0,
};
};
/** Every input must come back at its address under esptool's familyID. */
const check = async (file, included) => {
const inventory = await inventoryOf(file);
assert.equal(inventory.manifest.root_format, "uf2", file);
assert.equal(inventory.coverage.status, "complete", file);
const root = rootOf(inventory);
const expected = expectedRoot(included);
assert.deepEqual(
Object.fromEntries(Object.keys(expected).map((key) => [key, root[key]])),
expected,
file,
);
const inputs = included.flatMap((entry) =>
entry.layout.map(([address, name]) => ({
path: `${familyDirectory(entry.familyId)}/${hex32(address)}-${hex32(address + FILES[name].length - 1)}.bin`,
familyId: entry.familyId,
bytes: FILES[name],
})),
);
const regions = inventory.occurrences.filter(
(row) => row.format_metadata?.kind === "uf2-region",
);
assert.deepEqual(
regions.map((row) => row.logical_path).sort(),
inputs.map((region) => region.path).sort(),
file,
);
for (const region of regions)
assert.equal(
region.format_metadata.family_id,
inputs.find((row) => row.path === region.logical_path).familyId,
`${file} ${region.logical_path}`,
);
const extracted = await rea("extract-artifact", path(file));
assert.equal(extracted.containment_verified, true, file);
for (const region of inputs)
assert.equal(
Buffer.compare(
await readFile(join(extracted.output_root, region.path)),
region.bytes,
),
0,
`${file} ${region.path}`,
);
return {
file,
families: included.map((entry) => ({
chip: entry.chip,
family_id: hex32(entry.familyId),
family_name: regions.find(
(row) => row.format_metadata.family_id === entry.familyId,
).format_metadata.family_name,
})),
blocks: root.block_count,
regions: regions.length,
md5_blocks: root.md5_block_count,
};
};
/**
* Losing one block of one of two equal-count inputs must not read as
* complete: the other input still carries every block number.
*/
const checkLostBlock = async () => {
const [s3] = CASES;
const position = s3.layout.findIndex(([, name]) => name === "otadata");
const lostIndex =
s3.layout
.slice(0, position)
.reduce((sum, [, name]) => sum + blocksOf(s3, name), 0) + 1;
const whole = await readFile(path(s3.name));
await writeFile(
path("lost-block.uf2"),
Buffer.concat([
whole.subarray(0, lostIndex * 512),
whole.subarray((lostIndex + 1) * 512),
]),
);
const inventory = await inventoryOf("lost-block.uf2");
const root = rootOf(inventory);
assert.equal(inventory.coverage.status, "unknown", "lost-block.uf2");
assert.equal(root.ambiguous_numbering_group_count, 1, "lost-block.uf2");
assert.equal(root.missing_block_count, 0, "lost-block.uf2");
return {
file: "lost-block.uf2",
removed_block: lostIndex,
coverage: inventory.coverage.status,
ambiguous_numbering_groups: root.ambiguous_numbering_group_count,
};
};
await withVerifierCleanup(async () => {
const versionText = await tool(["version"]);
const version = /\bv?(\d+)\.(\d+)\.(\d+)\b/u.exec(versionText);
assert.ok(version, `Unrecognized esptool version: ${versionText}`);
workspace = await mkdtemp(join(tmpdir(), "rea-uf2-esptool-"));
await produce(Number(version[1]) >= 5);
const report = { esptool: version[0], cases: [] };
for (const entry of CASES)
report.cases.push(await check(entry.name, [entry]));
report.cases.push(await check("concatenated.uf2", CASES));
report.cases.push(await checkLostBlock());
process.stdout.write(`${JSON.stringify(report, null, 2)}\n`);
}, [
() => commands.close(),
() =>
workspace === undefined
? undefined
: rm(workspace, { recursive: true, force: true }),
]);
@@ -60,13 +60,13 @@ const reconciled = reconcileJavaScriptRuntimeEvidence({
runtime_observations: runtimeObservations,
});
if (!reconciled.ok) throw reconciled.error;
const result = reconciled.value.normalized_result;
const result = reconciled.value.evidence.normalized_result;
process.stdout.write(
`${JSON.stringify({
verifier_run: await completeVerifierRun(verifierRun),
reconciliation_id: result.reconciliation_id,
evidence_id: reconciled.value.evidence_id,
evidence_id: reconciled.value.evidence.evidence_id,
graph_id: result.graph.graph_id,
layers: result.static_layers.map((layer) => ({
role: layer.role,
+1 -7
View File
@@ -1,3 +1,4 @@
import { ADB_PROVIDER_IDENTITY } from "./AdbProviderIdentity.js";
import {
absoluteCallerPath,
appendCallerPath,
@@ -152,13 +153,6 @@ import {
type AdbCommandContext,
} from "./AdbDeviceOperations.js";
/** Stable identity for the ADB device-acquisition provider. */
export const ADB_PROVIDER_IDENTITY = {
id: "adb",
name: "Android Debug Bridge",
version: null,
} as const;
const SHELL_TIMEOUT_MS = 30_000;
const PACKAGE_LIST_TIMEOUT_MS = 60_000;
const PACKAGE_LIST_MAX_OUTPUT_BYTES = 8 * 1024 * 1024;
+6
View File
@@ -0,0 +1,6 @@
/** Stable identity for the ADB device-acquisition provider. */
export const ADB_PROVIDER_IDENTITY = {
id: "adb",
name: "Android Debug Bridge",
version: null,
} as const;
@@ -11,9 +11,6 @@ import {
APK_SIGNATURE_V3_ID,
LEAF_CERTIFICATE,
LEAF_SHA256,
LEAF_SUBJECT,
OID_ECDSA_SHA256,
OID_SHA256,
OPENSSL_BLOCK_ISSUER_SERIAL,
ROOT_CERTIFICATE,
ROOT_SHA256,
@@ -149,18 +146,6 @@ const reportOf = async (path: string) => {
};
describe("ApkSignatureProvider without the Android SDK", () => {
it("retains the declared digest's signature algorithm ID and bytes", async () => {
const report = await reportOf(
await write(
"algorithm-id.apk",
syntheticSigned([{ id: V2, value: bogusV2Value() }]),
),
);
expect(report.schemes.v2?.signers?.[0]?.digests[0]).toMatchObject({
signature_algorithm_id: 0x0103,
declared_digest_hex: "ab".repeat(32),
});
});
it("reports an unsigned ZIP as having no scheme", async () => {
const r = await reportOf(
await write(
@@ -174,53 +159,6 @@ describe("ApkSignatureProvider without the Android SDK", () => {
expect(r.posture.v1_only).toBe(false);
});
it("detects v1-only signing and names the signer certificate its block declares", async () => {
for (const stored of [false, true]) {
const r = await reportOf(
await write(
`v1-${String(stored)}.apk`,
await zipWith([
{ name: "classes.dex", data: "x" },
{ name: "META-INF/MANIFEST.MF", data: "Manifest-Version: 1.0" },
{ name: "META-INF/CERT.SF", data: "Signature-Version: 1.0" },
{
name: "META-INF/CERT.EC",
data: OPENSSL_BLOCK_ISSUER_SERIAL,
stored,
},
]),
),
);
expect(r.schemes.v1.present).toBe(true);
expect(r.posture.v1_only).toBe(true);
expect(r.posture.strongest_scheme).toBe("v1");
expect(r.posture.v1_signers_match_block_signers).toBeNull();
const [signer] = r.schemes.v1.signers;
expect(signer).toMatchObject({
signature_file: "META-INF/CERT.SF",
block_file: "META-INF/CERT.EC",
certificate: {
sha256_fingerprint: LEAF_SHA256,
subject: LEAF_SUBJECT,
public_key_algorithm: "ec",
},
certificates: [
{ sha256_fingerprint: LEAF_SHA256, subject: LEAF_SUBJECT },
{ sha256_fingerprint: ROOT_SHA256 },
],
signer_infos: [
{
identified_by: "issuer-and-serial",
certificate_sha256: LEAF_SHA256,
digest_algorithm_oid: OID_SHA256,
signature_algorithm_oid: OID_ECDSA_SHA256,
},
],
decode_error: null,
});
}
});
it("decodes a v2 signer, keeps an unparseable certificate null, and flags a digest mismatch", async () => {
const r = await reportOf(
await write(
Loaded 100 of 1059 files, more files were not shown because too many files have changed in this diff. Show more