mirror of
https://github.com/morluto/rea.git
synced 2026-11-07 05:56:19 -10:00
Compare commits
25
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
626c7b422a | ||
|
|
2d5dec26c3 | ||
|
|
da0e9669a8 | ||
|
|
73ff6ca0c8 | ||
|
|
c287a73276 | ||
|
|
7c9ed94259 | ||
|
|
a989c0e458 | ||
|
|
678cd82042 | ||
|
|
23de10bb28 | ||
|
|
b43c329df7 | ||
|
|
22e5aa97af | ||
|
|
e669365c3b | ||
|
|
e6d0618979 | ||
|
|
091892dfed | ||
|
|
1dc97d8cec | ||
|
|
7ba3f558d4 | ||
|
|
245a581818 | ||
|
|
ec107022fa | ||
|
|
b349e3c708 | ||
|
|
d9e01e5af2 | ||
|
|
6e2404b869 | ||
|
|
356cc50475 | ||
|
|
d48e472fa6 | ||
|
|
47fa6ab379 | ||
|
|
69bf61f7c7 |
No files matched your search
@@ -1,6 +1,6 @@
|
||||
---
|
||||
name: rea-tool-design
|
||||
description: Design or change REA investigation tools, CLI/MCP contracts, provider capabilities, and Evidence semantics.
|
||||
description: Audit, investigate, design, or change REA investigation tools, CLI/MCP contracts, provider capabilities, and Evidence semantics.
|
||||
---
|
||||
|
||||
# REA tool design
|
||||
@@ -10,6 +10,18 @@ contract semantics, provider ownership, and discoverability. Inspect the current
|
||||
contract and nearest existing tool relevant to the analyst question; source,
|
||||
documentation, and live catalogs may describe different versions.
|
||||
|
||||
For provider-backed work, exercise the analyst's task through REA and compare
|
||||
relevant native provider and independent artifact observations where available.
|
||||
Check applicable partial results, recovery and session boundaries; report what
|
||||
was not exercised. Use [workflow verification](../../../docs/testing.md#provider-backed-workflow-verification)
|
||||
to select evidence for the proposed contract instead of seeking provider API
|
||||
parity.
|
||||
|
||||
For an audit, follow [audit existing tools](../../../docs/tool-design.md#audit-existing-tools):
|
||||
scope representative tasks, exercise the live workflow, and distinguish
|
||||
reproduced bugs, supported limitations, design opportunities and unverified
|
||||
claims. Report working alternatives and evidence limits as well as findings.
|
||||
|
||||
For a design request, return the proposed boundary, input/result shape, Evidence
|
||||
semantics, nearest alternative, and verification approach. Implement when
|
||||
requested, preserving the user's scope and existing authorization.
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
name: reverse-engineer-anything
|
||||
description: Reverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis.
|
||||
metadata:
|
||||
version: "55"
|
||||
version: "59"
|
||||
---
|
||||
|
||||
# REA
|
||||
@@ -401,9 +401,18 @@ call `analyze_javascript_application` or `inspect_binary_layout` with
|
||||
a flag: its default reply is the summary, and the complete dossier stays
|
||||
retained under `parent_evidence_id` for facet paging. Pass that ID as a
|
||||
`retained-evidence` reference to `compare_functions`, or inside
|
||||
`match_native_functions` `left_observations`/`right_observations`; request
|
||||
`"detail": "complete"` only when the dossier must travel inline, as
|
||||
`build_call_path` requires.
|
||||
`match_native_functions` `left_observations`/`right_observations` or
|
||||
`build_call_path.functions`; request `"detail": "complete"` only when the
|
||||
dossier must travel inline.
|
||||
Use the same exact-reference form in other advertised logical Evidence inputs,
|
||||
including optional sources, arrays and nested sources across domains. Each
|
||||
source can independently be inline or retained; preserve its source identity,
|
||||
coverage, partial facts and unknowns. A summary or inspection page has its own
|
||||
Evidence ID: use `normalized_result.parent_evidence_id` when the consumer needs
|
||||
the complete source. For portable CLI follow-ups, commands advertising
|
||||
`--evidence-bundle` resolve references from that selected exported bundle.
|
||||
See [Evidence composition](references/evidence-workflows.md) for scope and
|
||||
recovery; file selectors are supported only where explicitly advertised.
|
||||
Page module identities and select the returned `node_id` before requesting an
|
||||
item; a single file path can represent several graph roles. Do not repeat an
|
||||
identical analysis or rejected call without a changed input or resolved blocker.
|
||||
|
||||
@@ -11,6 +11,44 @@ returned inline. When complete `inspect_binary_layout` or
|
||||
`inspect_analysis_view` for a summary, one section/symbol/module, or a stable
|
||||
page instead of repeating the producer.
|
||||
|
||||
For any logical Evidence source that advertises retained references, pass
|
||||
`{kind: "retained-evidence", evidence_id}` instead of copying the complete
|
||||
record back into a call. The same contract covers individual, optional, array
|
||||
and nested sources across native, process, JavaScript, managed, mobile and WASM
|
||||
workflows. Sources can independently mix inline records and exact references.
|
||||
Resolution preserves the source's identity, provenance, coverage, partial facts
|
||||
and unknowns; it neither reruns analysis nor makes incomplete facts complete.
|
||||
Keep domain role and identity requirements: a native dossier cannot substitute
|
||||
for a managed member observation, for example.
|
||||
|
||||
When starting from a summary or inspection page, use its
|
||||
`normalized_result.parent_evidence_id` to reference the complete source.
|
||||
The view's own `evidence_id` names
|
||||
only that view. `compare_artifacts` explicitly supports artifact summary inputs
|
||||
and resolves their parents; do not assume every consumer does. When a workflow
|
||||
follows source links, keep those linked records available too.
|
||||
|
||||
MCP references belong to the current connection's ledger. If the ID is known,
|
||||
pass it directly; no discovery call or provider setup is needed. If it is missing,
|
||||
import the exported bundle or supply complete inline Evidence. Do not retry an
|
||||
unchanged reference or rerun a producer merely to move an existing record.
|
||||
Export before closing the session when the source must survive it. For CLI JSON
|
||||
workflows that advertise `--evidence-bundle`, the same reference JSON resolves
|
||||
against one explicitly selected authenticated bundle:
|
||||
|
||||
```bash
|
||||
rea trace-application-feature request.json --evidence-bundle /absolute/path/evidence.json --json
|
||||
```
|
||||
|
||||
This includes JavaScript traces/comparisons and runtime reconciliation, managed
|
||||
reconstruction/native verification/graph projection, Android/Apple/Harmony graph
|
||||
projection, WASM interface comparison and retained analysis views. The managed
|
||||
member comparison CLI still takes two assembly paths; process comparison takes
|
||||
two capture Evidence JSON paths. Check command help for the installed version.
|
||||
An input bundle grants no implicit file lookup: `evidence-file` and historical
|
||||
graph file selectors are accepted only by contracts advertising them. See the
|
||||
[CLI reuse guide](https://github.com/morluto/rea/blob/main/docs/cli.md#import-export-and-compare-evidence).
|
||||
|
||||
Completed `inspect_artifact` inventory also supports a summary and
|
||||
`view: {kind: "inventory", collection: "occurrences", logical_paths: ["package.json", "main.js"]}`.
|
||||
Omit `logical_paths` to page all available occurrences; `[]` selects none.
|
||||
@@ -56,11 +94,12 @@ or absence:
|
||||
the complete dossier.
|
||||
- `compare_bundles` compares canonical bundle membership and unknown history.
|
||||
- `find_changed_behavior` combines existing comparisons; static differences
|
||||
remain candidates, not causal proof.
|
||||
remain candidates, not causal proof. Its `comparisons` accept exact references.
|
||||
- `build_call_path` needs explicit function dossiers from one artifact/provider;
|
||||
an incomplete frontier makes absence unknown.
|
||||
pass complete parent IDs in `functions`. An incomplete frontier makes absence
|
||||
unknown.
|
||||
- `correlate_static_and_runtime` uses explicit mappings; matching patterns are
|
||||
hypotheses, not causality.
|
||||
hypotheses, not causality. Both comparison arrays accept exact references.
|
||||
- `verify_reconstruction` evaluates a finite typed specification. A pass covers
|
||||
only declared comparable claims, not global equivalence.
|
||||
|
||||
|
||||
@@ -236,8 +236,13 @@ This notation is
|
||||
a template: replace it with the actual returned ID. Versions before 4.1.0 accept only full
|
||||
inline Evidence. Use the exact ID
|
||||
returned by the producer on the same MCP connection. Resolution does not run
|
||||
analysis or select a provider; findings remain inline. Native observation
|
||||
arrays still take full Evidence. `close_binary` clears retained references;
|
||||
analysis or select a provider; findings remain inline. For summary or inspection
|
||||
replies, select the complete `normalized_result.parent_evidence_id`. Native
|
||||
observation arrays and runtime reconciliation's `static_layers[].analysis` and
|
||||
`runtime_observations[]` also accept exact references where advertised. Each
|
||||
source can independently be inline or retained, preserving identity, coverage
|
||||
and unknowns. CLI workflows advertising `--evidence-bundle` resolve these inputs
|
||||
from the explicitly selected bundle. `close_binary` clears retained references;
|
||||
export a bundle before closing, or supply portable inline Evidence in another
|
||||
connection. A missing reference includes its ID and recovery guidance.
|
||||
|
||||
|
||||
@@ -13,6 +13,13 @@ Select columns, batch IDs and page individual facts; aliases remain unresolved.
|
||||
Unpaired observations do not prove additions/removals. Existing matched dossier
|
||||
pairs include inputs for `compare_functions`; otherwise analyze just chosen pairs.
|
||||
|
||||
Aggregate completed comparisons with `find_changed_behavior`, selecting each
|
||||
comparison's exact `retained-evidence` ID instead of resending full records.
|
||||
Inline comparison Evidence remains accepted. References belong to the current
|
||||
server: after handoff, import the original bundle including its source closure.
|
||||
Do not substitute dossier IDs or summary views. Static differences remain
|
||||
behavior candidates, with runtime behavior unknown unless separately observed.
|
||||
|
||||
For a matching representation refusal, inspect `details.limit` and raise
|
||||
`native.matching_representation_bytes` with `update_analysis_settings` before
|
||||
retrying the unchanged MCP request. CLI launch seed:
|
||||
@@ -578,6 +585,16 @@ REA. Symlinks and encrypted entries are inventory facts, not extractable files.
|
||||
The result reports the materialized directory; callers do not choose the
|
||||
destination. No separate permission grant is needed.
|
||||
|
||||
For classic iNES cartridges, `open_binary` recognizes the byte signature without
|
||||
an executable architecture. `inspect_artifact` retains header facts and each
|
||||
PRG/CHR file unit's offset, length and SHA-256; its nested inventory's
|
||||
`ines.reconstruction` verifies the actual ordered region bytes against the
|
||||
whole-file digest. Extraction includes the original header, optional trainer
|
||||
and uninterpreted trailing bytes. These file units are not mapper/CPU bank
|
||||
maps. NES 2.0, UNIF, dirty legacy headers and truncated payloads are explicitly
|
||||
refused. No emulator or 6502 analysis is performed. See the
|
||||
[iNES guide](https://github.com/morluto/rea/blob/main/docs/ines-containers.md).
|
||||
|
||||
Native DMG traversal automatically uses a read-only mount on supported macOS
|
||||
hosts, with an owned temporary mount directory and cleanup. It needs no approval
|
||||
flag. Unsupported hosts or mount failures remain explicit limitations; do not
|
||||
@@ -606,10 +623,27 @@ semantics are unsupported. `m68k` requires an even entry and decodes the 68040
|
||||
superset of the 68000 family; Amiga/Atari containers and vector-table entry
|
||||
discovery are unsupported. Windows Ghidra P0 admits raw images only for `6502`. See [raw binary guide](https://github.com/morluto/rea/blob/main/docs/raw-binary.md).
|
||||
|
||||
Intel HEX, S-record, UF2 and DfuSe files are containers, not raw images:
|
||||
`inspect_artifact` lists address regions (UF2 per familyID, DfuSe per element
|
||||
with its alternate setting), and `extract_artifact` writes each region. Open an
|
||||
extracted region as `raw-binary` with its start address as `entry_point`. UF2
|
||||
numbering gaps leave coverage unknown; a DfuSe suffix CRC mismatch follows
|
||||
`integrity_policy`. See [UF2 and DfuSe](https://github.com/morluto/rea/blob/main/docs/uf2-dfuse.md).
|
||||
|
||||
### Function annotations in Ghidra
|
||||
|
||||
Fresh-import sessions admit annotations; existing-project reuse is read-only.
|
||||
Use `annotate_native_function` for one function name and/or entry comments, with at least one explicit change. Review its annotation readback and refreshed dossier inline. Changes are atomic and session-scoped; empty comments clear them, omitted fields preserve them. Later MCP calls observe edits until close. CLI `annotate-native-function` returns the updated analysis before discarding the session. Original executable bytes are unchanged; edits invalidate immutable snapshots. Windows P0 does not admit database mutations. For naming only, `set_address_name` / `set_addresses_names` also work with Ghidra: they name function entries or data addresses (primary labels) and return compact per-address success booleans instead of a dossier.
|
||||
Use `annotate_native_function` for one function name and/or entry comments, with at least one explicit change. Review its annotation readback and refreshed dossier inline. Changes are atomic and session-scoped; empty comments clear them, omitted fields preserve them. Later MCP calls observe edits until close. CLI `annotate-native-function` returns the updated analysis before discarding the session. Original executable bytes are unchanged; edits invalidate immutable snapshots. Windows P0 does not admit database mutations. For naming only, `set_address_name` / `set_addresses_names` also work with Ghidra: they name function entries or data addresses (primary labels) and return compact per-address success booleans instead of a dossier. Ghidra rejects batch addresses that request the same name or a name another selected address holds; `raw_result.failures` gives each failure's reason.
|
||||
|
||||
For `compare_functions`, obtain `analyze_function` Evidence before and after the
|
||||
edit while the MCP session is open; annotation Evidence is not a comparison
|
||||
input. Export the needed Evidence before close if comparison will happen later.
|
||||
Authenticated same-artifact coordinates preserve renamed entries as
|
||||
`matched / same-entry`, with the canonical `entry_address` inline; name and
|
||||
exact-text changes remain separate dimensions. This requires compatible
|
||||
architecture, provider-owned mapping and selected-image commitments. Equal
|
||||
numeric addresses alone cannot establish correspondence; inspect retained
|
||||
source Evidence and coverage when the match remains ambiguous.
|
||||
|
||||
## Offline PE resource inventory
|
||||
|
||||
@@ -637,7 +671,9 @@ For exact core-module import/export differences, use `compare_wasm_interfaces`
|
||||
with two complete inspection Evidence records (or current-session retained
|
||||
references) and explicit scope. Preserve unknown types/duplicate names; static
|
||||
interface equality is not runtime compatibility. CLI accepts complete inline
|
||||
Evidence in `rea compare-wasm-interfaces input.json --json`.
|
||||
Evidence in `rea compare-wasm-interfaces input.json --json`, or exact references
|
||||
with `--evidence-bundle /absolute/path/evidence.json`. Use the complete parent
|
||||
ID when the inspection returned a summary.
|
||||
|
||||
## Godot PCK
|
||||
|
||||
@@ -696,6 +732,32 @@ recognized. Segments are not disassembled, and RISC-V or Xtensa segments
|
||||
cannot be opened as raw-binary. See
|
||||
[ESP images](https://github.com/morluto/rea/blob/main/docs/esp-image.md).
|
||||
|
||||
## ESP flash images
|
||||
|
||||
A whole-flash dump opens as `esp-flash-image`. `root_format_metadata` holds
|
||||
`partition_table` (its `method`: `default`, `caller` or a derived
|
||||
`scan`), `bootloader` and `apps` ESP image facts, the `otadata` records,
|
||||
`partitions_outside_dump` and `boot_selection`. Every table has a
|
||||
`boot_selection`: its `basis` names the bootloader's branch (`no-otadata`
|
||||
without otadata, `none` when the bootloader rejects the table, `unknown`
|
||||
when a record is missing), and it lists its `assumptions`;
|
||||
`running_partition` is always `unknown`. Each image names its entry
|
||||
`path`. With `detail: "summary"` the root kind is `esp-flash-image-summary`
|
||||
and each image is a compact `summary` (chip, entry point, segment count,
|
||||
classified load ranges, signature block count, descriptor identity,
|
||||
integrity); complete detail keeps every segment. Partition rows
|
||||
(`partitions/<label>.bin`) carry the declared `type`, `subtype`, names,
|
||||
`flash_address`, `size` and flags; `encrypted` is a storage flag, not
|
||||
proof of ciphertext. `written_offset`/`written_end` bound the bytes that
|
||||
are not erased: an app slot whose facts say `erased` can still hold an older
|
||||
image, and a limitation says so. For a dump read from another flash address, call
|
||||
`open_binary` with `format: "esp-flash-image"` and `flash_offset`; an
|
||||
`esp-image` whose `flash_dump` is set names that offset in a next probe; a
|
||||
relocated table can be named with `partition_table_offset`. No table found
|
||||
means unknown, often flash encryption. Extraction writes `nvs_keys` as
|
||||
ordinary data. See
|
||||
[ESP flash images](https://github.com/morluto/rea/blob/main/docs/esp-flash-image.md).
|
||||
|
||||
## Bun standalone executables
|
||||
|
||||
A `bun build --compile` executable stores its module graph in Mach-O
|
||||
|
||||
@@ -779,6 +779,12 @@ jobs:
|
||||
npm run verify:legacy-simulator
|
||||
npm run verify:apple-dispatch
|
||||
npm run verify:interface-builder
|
||||
- name: Verify portable iNES CLI, MCP and source boundaries
|
||||
run: >-
|
||||
npm run test:focused --
|
||||
tests/acceptance/inesContainer.test.ts
|
||||
tests/boundary/filesystem/inesArtifact.test.ts
|
||||
tests/boundary/filesystem/inesCancellation.test.ts
|
||||
- name: Run native Apple and process boundary regressions
|
||||
run: >-
|
||||
npm run test:focused --
|
||||
@@ -793,12 +799,14 @@ jobs:
|
||||
tests/boundary/filesystem/installScript.test.ts
|
||||
tests/boundary/filesystem/nibNativeHierarchy.test.ts
|
||||
tests/boundary/filesystem/interfaceBuilderBundle.test.ts
|
||||
tests/acceptance/nativeUiHandlerLocation.test.ts
|
||||
tests/boundary/cli/nativePlistCli.test.ts
|
||||
tests/boundary/cli/nativePlistPrototypeKeyCli.test.ts
|
||||
tests/boundary/cli/nativePlistSelectionFailureCli.test.ts
|
||||
tests/boundary/cli/nativeDemangleCli.test.ts
|
||||
tests/boundary/cli/nativeDispatchPropertyAttributes.test.ts
|
||||
tests/boundary/mcp/nativeDemangleMcp.test.ts
|
||||
tests/boundary/mcp/nativeMachoSummaryMcp.test.ts
|
||||
tests/boundary/providers/native/nativeSignatureTargetBinding.test.ts
|
||||
tests/boundary/cli/nativeSignatureCli.test.ts
|
||||
tests/boundary/cli/nativeMachoArm64_32Cli.test.ts
|
||||
|
||||
@@ -17,6 +17,7 @@ notices supplied with the software you actually obtain or redistribute.
|
||||
| ------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| npm dependencies | [Complete inventory](licenses/npm-dependencies.txt) of every entry in `package-lock.json`, including transitive, optional platform, development, and website dependencies. [Runtime notices](licenses/npm-runtime-notices.txt) preserve the copyright, license, and NOTICE documents from integrity-verified release archives, plus the MIT notice for Rolldown's generated runtime helpers. Development dependencies are inventoried but are not all distributed in REA's npm package. |
|
||||
| pefile ordinal tables | Import-hash ordinal names are derived from [pefile 2024.8.26](https://github.com/erocarrera/pefile/tree/4b3b1e2e568a88d4f1897d694d684f23d9e270c4), copyright 2004–2024 Ero Carrera. Preserve the [MIT license](licenses/pefile-2024.8.26-LICENSE.txt). The Python package is not a runtime dependency. |
|
||||
| UF2 family names | UF2 family IDs and short names are copied from [microsoft/uf2 `utils/uf2families.json`](https://github.com/microsoft/uf2/blob/cf3bd7f9c274b949528f8cd6a19eef2502a687d2/utils/uf2families.json), copyright Microsoft Corporation. Preserve the [MIT license](licenses/microsoft-uf2-MIT.txt). |
|
||||
| NativeAOT parser | REA's derived parser uses the MIT-licensed [ghidra-nativeaot](https://github.com/Washi1337/ghidra-nativeaot/tree/effeb734fc570c32650f88b159608979dc7b423e) work of Washi and the NativeAOT layout definitions in [dotnet/runtime](https://github.com/dotnet/runtime/tree/a3b2d40328be0be3f8dd43f0e5cc925b8827b044). Preserve [Washi's MIT notice](licenses/nativeaot-parser-MIT.txt) and the [.NET Foundation MIT notice](licenses/dotnet-runtime-MIT.txt). |
|
||||
| EVM analysis | `[email protected]`, MIT, copyright 2023 Maxim Andreev. The published npm archive omits its license document; REA retains the [original upstream license](third_party/evmole/LICENSE) and [provenance](third_party/evmole/README.md). It is also included in the runtime notices. |
|
||||
| Windows native addon | The release addon is compiled with MinGW-w64 and statically links GCC runtime libraries. Preserve the [MinGW-w64 runtime notices](licenses/MinGW-w64-runtime.txt), [GPLv3 text](licenses/GPL-3.0.txt), and [GCC Runtime Library Exception 3.1](licenses/GCC-Runtime-Library-Exception-3.1.txt). The exception governs eligible compiled combinations; it does not relicense the runtime itself under MIT. The Node-API build headers are from Node 22.19.0; its [original license and component notices](licenses/Node-22.19.0-LICENSE.txt) are retained. Node itself is supplied by the user. Snapshot provenance and release-toolchain review requirements are in [licenses/README.md](licenses/README.md). |
|
||||
@@ -59,7 +60,7 @@ pin inspection sources; they are not included in the npm package.
|
||||
| ghidra-nativeaot | [MIT](https://github.com/Washi1337/ghidra-nativeaot/blob/effeb734fc570c32650f88b159608979dc7b423e/LICENSE.md) | Pinned inspection source; derived parser attribution above. |
|
||||
| JADX | [Apache-2.0](https://github.com/skylot/jadx/blob/v1.5.6/LICENSE) | The audited jadx-headless-mcp source uses JADX 1.5.6 for Android decompilation; the engine JAR is separately supplied. |
|
||||
| jadx-headless-mcp | [Apache-2.0](https://github.com/1013503897/jadx-headless-mcp/blob/5844800a486d2248fa949b74c9896285f5b54de2/LICENSE) | v0.7.1; pinned submodule and user-supplied JAR. |
|
||||
| Kuna | [Apache-2.0](https://github.com/Noelo-Lab/kuna/blob/v1.833/LICENSE), with its [NOTICE](https://github.com/Noelo-Lab/kuna/blob/v1.833/NOTICE) | Bring your own `kuna`; REA verified 1.833 and never downloads, installs or bundles it. REA invokes it as a separate process. |
|
||||
| Kuna | [Apache-2.0](https://github.com/Noelo-Lab/kuna/blob/v1.986/LICENSE), with its [NOTICE](https://github.com/Noelo-Lab/kuna/blob/v1.986/NOTICE) | Bring your own `kuna`; REA verified 1.986 and never downloads, installs or bundles it. REA invokes it as a separate process. |
|
||||
| Apktool | [Apache-2.0](https://github.com/iBotPeaches/Apktool/blob/master/LICENSE.md) | User-supplied Android resource decoder. |
|
||||
| Binwalk | [MIT](https://github.com/ReFirmLabs/binwalk/blob/4fdab3d464d97b68e0af9088df3f9e2e1545b21c/LICENSE) | v3.1.0; pinned submodule and user-supplied firmware engine. |
|
||||
| Unblob | [MIT](https://github.com/onekey-sec/unblob/blob/1fcc7a0a584a70a96c31f5a276c20944d199a089/LICENSE) | 26.6.4; pinned submodule and user-supplied firmware engine. |
|
||||
|
||||
@@ -40,6 +40,7 @@ import java.util.Locale;
|
||||
import java.util.Map;
|
||||
import java.util.Set;
|
||||
import java.util.TreeMap;
|
||||
import java.util.TreeSet;
|
||||
import java.util.regex.Pattern;
|
||||
import java.util.regex.PatternSyntaxException;
|
||||
|
||||
@@ -548,6 +549,11 @@ public final class ReaGhidraBridge extends HeadlessScript {
|
||||
String softFloat = currentProgram.getOptions(ghidra.program.model.listing.Program.PROGRAM_INFO)
|
||||
.getString("REA MIPS soft-float storage", null);
|
||||
if (softFloat != null) result.add("mips_soft_float_storage", JsonParser.parseString(softFloat));
|
||||
// Written by ReaGhidraRiscvCallingConvention after analysis; REA requires it
|
||||
// exactly when the committed profile declares a RISC-V calling-convention policy.
|
||||
String riscvCallingConvention = currentProgram.getOptions(ghidra.program.model.listing.Program.PROGRAM_INFO)
|
||||
.getString("REA RISC-V calling convention", null);
|
||||
if (riscvCallingConvention != null) result.add("riscv_calling_convention", JsonParser.parseString(riscvCallingConvention));
|
||||
return result;
|
||||
}
|
||||
|
||||
@@ -922,17 +928,33 @@ public final class ReaGhidraBridge extends HeadlessScript {
|
||||
requireKeys(params, Set.of("document", "address"));
|
||||
requireDocument(params);
|
||||
Address address = requireAddress(params, "address");
|
||||
String at = canonicalAddress(address);
|
||||
// A well-formed address can still lack one exact source offset. Each
|
||||
// outcome has its own code, so REA reports the reason rather than
|
||||
// schema repair. Initialization, not the loaded flag, decides whether
|
||||
// source bytes can exist: file-backed overlays are not loaded.
|
||||
MemoryBlock block = currentProgram.getMemory().getBlock(address);
|
||||
if (block == null || !block.isInitialized())
|
||||
throw new RequestFailure("invalid_request", "Address has no initialized file-backed memory: " + canonicalAddress(address));
|
||||
if (block == null)
|
||||
throw new RequestFailure("address_unmapped",
|
||||
"Address " + at + " is outside every Ghidra memory block");
|
||||
if (!block.isInitialized())
|
||||
throw new RequestFailure("address_uninitialized",
|
||||
"Address " + at + " is in uninitialized Ghidra memory block " + block.getName());
|
||||
TargetFileOffset mapping = targetFileOffset(block, address);
|
||||
if (mapping.mappings() != 1)
|
||||
throw new RequestFailure("invalid_request", "Address has " + mapping.mappings() + " authoritative target-file mappings: " + canonicalAddress(address));
|
||||
if (mapping.mappings() == 0)
|
||||
throw new RequestFailure("file_mapping_absent",
|
||||
"Address " + at + " in initialized Ghidra memory block " + block.getName() +
|
||||
" has no authoritative mapping to the admitted target file");
|
||||
if (mapping.mappings() > 1)
|
||||
throw new RequestFailure("file_mapping_ambiguous",
|
||||
"Address " + at + " has " + mapping.mappings() +
|
||||
" authoritative mappings to the admitted target file");
|
||||
Long fileOffset = mapping.offset();
|
||||
if (fileOffset == null)
|
||||
throw new RequestFailure("invalid_request", "Address lacks a safe file offset: " + canonicalAddress(address));
|
||||
throw new RequestFailure("file_offset_unrepresentable",
|
||||
"Address " + at + " maps to an admitted target-file offset beyond the exact JSON integer range");
|
||||
JsonObject result = new JsonObject();
|
||||
result.addProperty("address", canonicalAddress(address));
|
||||
result.addProperty("address", at);
|
||||
result.addProperty("file_offset", fileOffset);
|
||||
return result;
|
||||
}
|
||||
@@ -1931,41 +1953,62 @@ public final class ReaGhidraBridge extends HeadlessScript {
|
||||
return symbol == null ? JsonNull.INSTANCE : GSON.toJsonTree(symbol.getName(true));
|
||||
}
|
||||
|
||||
// Analyst naming: a function entry renames the function, other addresses get a primary label.
|
||||
private boolean applyAddressName(Address address, String name) throws Exception {
|
||||
// Analyst naming: a function entry renames the function; another mapped
|
||||
// address renames its non-dynamic primary label or receives a new global
|
||||
// label. Resolve that destination without editing the program.
|
||||
private AddressNameTarget addressNameTarget(Address address, String name)
|
||||
throws ghidra.util.exception.InvalidInputException {
|
||||
if (!currentProgram.getMemory().contains(address))
|
||||
throw new RequestFailure("invalid_request", "Address is not mapped: " + canonicalAddress(address));
|
||||
validateAnnotationText(name, "name");
|
||||
Symbol edited = editedNameSymbol(address);
|
||||
return edited == null
|
||||
? new AddressNameTarget(address, currentProgram.getGlobalNamespace(), name)
|
||||
: new AddressNameTarget(address, edited.getParentNamespace(), symbolLeafName(edited, name));
|
||||
}
|
||||
|
||||
private Symbol editedNameSymbol(Address address) {
|
||||
Function function = currentProgram.getFunctionManager().getFunctionAt(address);
|
||||
if (function != null) return function.getSymbol();
|
||||
Symbol primary = currentProgram.getSymbolTable().getPrimarySymbol(address);
|
||||
return primary == null || primary.isDynamic() ? null : primary;
|
||||
}
|
||||
|
||||
// Every address whose label or function symbol holds the destination.
|
||||
private List<Address> addressNameHolders(AddressNameTarget target) {
|
||||
Set<Address> holders = new TreeSet<>();
|
||||
for (Symbol symbol : currentProgram.getSymbolTable().getLabelOrFunctionSymbols(target.leaf(), target.namespace()))
|
||||
holders.add(symbol.getAddress());
|
||||
return List.copyOf(holders);
|
||||
}
|
||||
|
||||
// Apply one planned destination in its own transaction. Null means the
|
||||
// edit committed with matching primary-symbol readback.
|
||||
private String applyAddressName(AddressNameTarget target) {
|
||||
Address address = target.address();
|
||||
int transaction = currentProgram.startTransaction("REA address name");
|
||||
boolean commit = false;
|
||||
try {
|
||||
Function function = currentProgram.getFunctionManager().getFunctionAt(address);
|
||||
SymbolTable table = currentProgram.getSymbolTable();
|
||||
Symbol primary = function == null ? table.getPrimarySymbol(address) : function.getSymbol();
|
||||
Namespace namespace = primary == null
|
||||
? currentProgram.getGlobalNamespace()
|
||||
: primary.getParentNamespace();
|
||||
String expected = primary == null ? name : symbolLeafName(primary, name);
|
||||
for (Symbol symbol : table.getLabelOrFunctionSymbols(expected, namespace)) {
|
||||
if (!address.equals(symbol.getAddress())) return false;
|
||||
}
|
||||
Symbol edited = editedNameSymbol(address);
|
||||
if (function != null) {
|
||||
function.setName(expected, SourceType.USER_DEFINED);
|
||||
function.setName(target.leaf(), SourceType.USER_DEFINED);
|
||||
} else if (edited != null) {
|
||||
edited.setName(target.leaf(), SourceType.USER_DEFINED);
|
||||
} else {
|
||||
if (primary != null && !primary.isDynamic()) {
|
||||
primary.setName(expected, SourceType.USER_DEFINED);
|
||||
} else {
|
||||
Symbol created = table.createLabel(address, expected, SourceType.USER_DEFINED);
|
||||
created.setPrimary();
|
||||
}
|
||||
Symbol created = currentProgram.getSymbolTable().createLabel(address, target.leaf(), SourceType.USER_DEFINED);
|
||||
created.setPrimary();
|
||||
}
|
||||
invalidateAnalysisCaches();
|
||||
Symbol readback = currentProgram.getSymbolTable().getPrimarySymbol(address);
|
||||
commit = readback != null && expected.equals(readback.getName());
|
||||
return commit;
|
||||
commit = readback != null && target.leaf().equals(readback.getName());
|
||||
return commit ? null : "readback_mismatch";
|
||||
}
|
||||
catch (ghidra.util.exception.InvalidInputException | ghidra.util.exception.DuplicateNameException exception) {
|
||||
return false;
|
||||
catch (ghidra.util.exception.InvalidInputException exception) {
|
||||
return "invalid_name";
|
||||
}
|
||||
catch (ghidra.util.exception.DuplicateNameException exception) {
|
||||
return "duplicate_name";
|
||||
}
|
||||
finally {
|
||||
currentProgram.endTransaction(transaction, commit);
|
||||
@@ -1975,7 +2018,17 @@ public final class ReaGhidraBridge extends HeadlessScript {
|
||||
private JsonElement setAddressName(JsonObject params) throws Exception {
|
||||
requireKeys(params, Set.of("document", "address", "name"));
|
||||
requireDocument(params);
|
||||
return GSON.toJsonTree(applyAddressName(requireAddress(params, "address"), requireString(params, "name")));
|
||||
String key = requireString(params, "address");
|
||||
Address address = requireAddress(params, "address");
|
||||
String name = requireString(params, "name");
|
||||
// A single edit keeps request errors for unmapped addresses and
|
||||
// unrepresentable text; other outcomes share the batch representation.
|
||||
if (!currentProgram.getMemory().contains(address))
|
||||
throw new RequestFailure("invalid_request", "Address is not mapped: " + canonicalAddress(address));
|
||||
validateAnnotationText(name, "name");
|
||||
Map<String, String> requested = new LinkedHashMap<>();
|
||||
requested.put(key, name);
|
||||
return applyAddressNames(requested);
|
||||
}
|
||||
|
||||
private JsonElement setAddressesNames(JsonObject params) throws Exception {
|
||||
@@ -1983,34 +2036,129 @@ public final class ReaGhidraBridge extends HeadlessScript {
|
||||
requireDocument(params);
|
||||
if (!params.has("names") || !params.get("names").isJsonObject())
|
||||
throw new RequestFailure("invalid_request", "names must be an object of address -> name");
|
||||
Map<String, String> requested = new LinkedHashMap<>();
|
||||
for (Map.Entry<String, JsonElement> entry : params.getAsJsonObject("names").entrySet())
|
||||
requested.put(entry.getKey(), entry.getValue().getAsString());
|
||||
return applyAddressNames(requested);
|
||||
}
|
||||
|
||||
// Decide every request from the names held before any edit, so object
|
||||
// member order never selects which address receives a name. Reply with
|
||||
// per-key results and reasons; a contested destination is listed once in
|
||||
// conflicts and failures refer to it by index.
|
||||
private JsonObject applyAddressNames(Map<String, String> requested) {
|
||||
Map<String, Address> addressesByKey = new LinkedHashMap<>();
|
||||
Map<Address, List<AddressNameRequest>> requestsByAddress = new LinkedHashMap<>();
|
||||
JsonObject result = new JsonObject();
|
||||
for (Map.Entry<String, JsonElement> entry : params.getAsJsonObject("names").entrySet()) {
|
||||
for (Map.Entry<String, String> entry : requested.entrySet()) {
|
||||
Address address = null;
|
||||
try {
|
||||
Address address = parseReaAddress(entry.getKey());
|
||||
address = parseReaAddress(entry.getKey());
|
||||
requestsByAddress.computeIfAbsent(address, ignored -> new ArrayList<>())
|
||||
.add(new AddressNameRequest(entry.getKey(), address, entry.getValue().getAsString()));
|
||||
.add(new AddressNameRequest(entry.getKey(), address, entry.getValue()));
|
||||
}
|
||||
catch (RequestFailure failure) {
|
||||
result.addProperty(entry.getKey(), false);
|
||||
// The key keeps its own invalid_address outcome below.
|
||||
}
|
||||
addressesByKey.put(entry.getKey(), address);
|
||||
}
|
||||
Map<Address, AddressNameFailure> failures = new HashMap<>();
|
||||
Map<AddressNameDestination, List<AddressNameTarget>> destinations = new LinkedHashMap<>();
|
||||
for (Map.Entry<Address, List<AddressNameRequest>> selected : requestsByAddress.entrySet()) {
|
||||
Address address = selected.getKey();
|
||||
String name = selected.getValue().get(0).name;
|
||||
if (selected.getValue().stream().anyMatch(request -> !request.name.equals(name))) {
|
||||
failures.put(address, new AddressNameFailure("conflicting_alias_names", null));
|
||||
continue;
|
||||
}
|
||||
if (!currentProgram.getMemory().contains(address)) {
|
||||
failures.put(address, new AddressNameFailure("unmapped_address", null));
|
||||
continue;
|
||||
}
|
||||
try {
|
||||
AddressNameTarget target = addressNameTarget(address, name);
|
||||
destinations.computeIfAbsent(target.destination(), ignored -> new ArrayList<>()).add(target);
|
||||
}
|
||||
catch (RequestFailure | ghidra.util.exception.InvalidInputException exception) {
|
||||
failures.put(address, new AddressNameFailure("invalid_name", null));
|
||||
}
|
||||
}
|
||||
for (List<AddressNameRequest> requests : requestsByAddress.values()) {
|
||||
String name = requests.get(0).name;
|
||||
boolean conflictingAliasNames = requests.stream()
|
||||
.anyMatch(request -> !request.name.equals(name));
|
||||
boolean success = false;
|
||||
if (!conflictingAliasNames) {
|
||||
try {
|
||||
success = applyAddressName(requests.get(0).address, name);
|
||||
}
|
||||
catch (RequestFailure failure) {
|
||||
// Keep per-address failures in the existing batch result shape.
|
||||
}
|
||||
// Reject a destination that several selected addresses request, or
|
||||
// that another address holds, instead of letting order pick a winner.
|
||||
List<AddressNameConflict> conflicts = new ArrayList<>();
|
||||
List<AddressNameTarget> independent = new ArrayList<>();
|
||||
for (List<AddressNameTarget> requesters : destinations.values()) {
|
||||
AddressNameTarget first = requesters.get(0);
|
||||
List<Address> holders = addressNameHolders(first);
|
||||
String reason;
|
||||
if (requesters.size() > 1) {
|
||||
reason = "duplicate_requested_name";
|
||||
} else if (holders.stream().allMatch(first.address()::equals)) {
|
||||
independent.add(first);
|
||||
continue;
|
||||
} else if (holders.stream().anyMatch(holder ->
|
||||
!holder.equals(first.address()) && !requestsByAddress.containsKey(holder))) {
|
||||
reason = "name_held_by_unselected_address";
|
||||
} else {
|
||||
reason = "name_held_by_selected_address";
|
||||
}
|
||||
for (AddressNameRequest request : requests)
|
||||
result.addProperty(request.inputAddress, success);
|
||||
AddressNameConflict conflict = new AddressNameConflict(
|
||||
first.namespace(),
|
||||
first.leaf(),
|
||||
requesters.stream().map(AddressNameTarget::address).sorted().toList(),
|
||||
holders
|
||||
);
|
||||
conflicts.add(conflict);
|
||||
for (AddressNameTarget target : requesters)
|
||||
failures.put(target.address(), new AddressNameFailure(reason, conflict));
|
||||
}
|
||||
// Independent edits have distinct destinations that no other address
|
||||
// holds, so their results do not depend on application order.
|
||||
for (AddressNameTarget target : independent) {
|
||||
String reason = applyAddressName(target);
|
||||
if (reason != null) failures.put(target.address(), new AddressNameFailure(reason, null));
|
||||
}
|
||||
// Requesters belong to exactly one destination, so ordering conflicts
|
||||
// by their first requester gives indexes independent of member order.
|
||||
conflicts.sort(Comparator.comparing((AddressNameConflict conflict) -> conflict.requesters().get(0)));
|
||||
Map<AddressNameConflict, Integer> indexes = new IdentityHashMap<>();
|
||||
JsonArray contested = new JsonArray();
|
||||
for (AddressNameConflict conflict : conflicts) {
|
||||
indexes.put(conflict, contested.size());
|
||||
JsonObject value = new JsonObject();
|
||||
if (conflict.namespace().isGlobal()) value.add("namespace", JsonNull.INSTANCE);
|
||||
else value.addProperty("namespace", conflict.namespace().getName(true));
|
||||
value.addProperty("name", conflict.leaf());
|
||||
value.add("requested_by", canonicalAddresses(conflict.requesters()));
|
||||
value.add("held_by", canonicalAddresses(conflict.holders()));
|
||||
contested.add(value);
|
||||
}
|
||||
JsonObject results = new JsonObject();
|
||||
JsonObject failed = new JsonObject();
|
||||
for (Map.Entry<String, Address> entry : addressesByKey.entrySet()) {
|
||||
Address address = entry.getValue();
|
||||
AddressNameFailure failure = address == null
|
||||
? new AddressNameFailure("invalid_address", null)
|
||||
: failures.get(address);
|
||||
results.addProperty(entry.getKey(), failure == null);
|
||||
if (failure == null) continue;
|
||||
JsonObject value = new JsonObject();
|
||||
if (address == null) value.add("address", JsonNull.INSTANCE);
|
||||
else value.addProperty("address", canonicalAddress(address));
|
||||
value.addProperty("reason", failure.reason());
|
||||
if (failure.conflict() == null) value.add("conflict", JsonNull.INSTANCE);
|
||||
else value.addProperty("conflict", indexes.get(failure.conflict()));
|
||||
failed.add(entry.getKey(), value);
|
||||
}
|
||||
JsonObject result = new JsonObject();
|
||||
result.add("results", results);
|
||||
result.add("failures", failed);
|
||||
result.add("conflicts", contested);
|
||||
return result;
|
||||
}
|
||||
|
||||
private static JsonArray canonicalAddresses(List<Address> addresses) {
|
||||
JsonArray result = new JsonArray();
|
||||
for (Address address : addresses) result.add(canonicalAddress(address));
|
||||
return result;
|
||||
}
|
||||
|
||||
@@ -4981,6 +5129,14 @@ public final class ReaGhidraBridge extends HeadlessScript {
|
||||
|
||||
private record InventoryItem(Address address, String value, JsonObject facts, JsonObject symbol) {}
|
||||
private record AddressNameRequest(String inputAddress, Address address, String name) {}
|
||||
private record AddressNameDestination(long namespace, String leaf) {}
|
||||
private record AddressNameTarget(Address address, Namespace namespace, String leaf) {
|
||||
AddressNameDestination destination() {
|
||||
return new AddressNameDestination(namespace.getID(), leaf);
|
||||
}
|
||||
}
|
||||
private record AddressNameConflict(Namespace namespace, String leaf, List<Address> requesters, List<Address> holders) {}
|
||||
private record AddressNameFailure(String reason, AddressNameConflict conflict) {}
|
||||
private record FunctionEntry(Function function, InventoryItem item) {}
|
||||
private record FunctionAnalysisObservation(JsonObject dossier, List<String> limitations, JsonObject relationshipCoverage, boolean decompilerFailed) {}
|
||||
private record CallBoundaryObservation(List<String> limitations, Integer unrepresentedCalleeCalls, JsonArray unresolvedCalls) {}
|
||||
|
||||
@@ -0,0 +1,283 @@
|
||||
import ghidra.app.script.GhidraScript;
|
||||
import ghidra.program.database.SpecExtension;
|
||||
import ghidra.program.model.data.DataType;
|
||||
import ghidra.program.model.data.DoubleDataType;
|
||||
import ghidra.program.model.data.Float16DataType;
|
||||
import ghidra.program.model.data.FloatDataType;
|
||||
import ghidra.program.model.data.FunctionDefinition;
|
||||
import ghidra.program.model.data.IntegerDataType;
|
||||
import ghidra.program.model.data.StructureDataType;
|
||||
import ghidra.program.model.lang.CompilerSpec;
|
||||
import ghidra.program.model.lang.Language;
|
||||
import ghidra.program.model.lang.PrototypeModel;
|
||||
import ghidra.program.model.lang.Register;
|
||||
import ghidra.program.model.listing.Function;
|
||||
import ghidra.program.model.listing.Parameter;
|
||||
import ghidra.program.model.listing.Program;
|
||||
import ghidra.program.model.listing.VariableStorage;
|
||||
import ghidra.program.model.pcode.Varnode;
|
||||
import ghidra.program.model.pcode.XmlEncode;
|
||||
import java.io.StringReader;
|
||||
import java.io.StringWriter;
|
||||
import java.util.ArrayList;
|
||||
import java.util.Iterator;
|
||||
import java.util.List;
|
||||
import java.util.regex.Pattern;
|
||||
import javax.xml.parsers.DocumentBuilderFactory;
|
||||
import javax.xml.transform.OutputKeys;
|
||||
import javax.xml.transform.TransformerFactory;
|
||||
import javax.xml.transform.dom.DOMSource;
|
||||
import javax.xml.transform.stream.StreamResult;
|
||||
import org.w3c.dom.Document;
|
||||
import org.w3c.dom.Element;
|
||||
import org.w3c.dom.Node;
|
||||
import org.w3c.dom.NodeList;
|
||||
import org.xml.sax.InputSource;
|
||||
|
||||
/**
|
||||
* Apply the committed RISC-V calling-convention policy after analysis. Ghidra's generic RV32
|
||||
* compiler specification models ilp32d. Each policy derives a program-specific prototype from the
|
||||
* program's own default prototype, as its {@link Policy} row and docs/ghidra-riscv.md state. The
|
||||
* script installs the prototype as a specification extension and assigns it, with dynamic
|
||||
* storage, to every non-thunk function, external function and function definition.
|
||||
*/
|
||||
public final class ReaGhidraRiscvCallingConvention extends GhidraScript {
|
||||
static final String REPORT_KEY = "REA RISC-V calling convention";
|
||||
/** RISC-V floating-point registers by ABI name: ft0-ft11, fs0-fs11 and fa0-fa7. */
|
||||
private static final Pattern FLOAT_REGISTER = Pattern.compile("f[ast]\\d+");
|
||||
|
||||
/**
|
||||
* What a policy changes in the default ilp32d prototype. Every policy also passes values of 9
|
||||
* bytes or more, such as {@code long double}, by reference.
|
||||
*
|
||||
* @param floatEntryBytes the largest value a floating-point register may pass or return; 0
|
||||
* removes those entries and every rule that assigns floating-point storage
|
||||
* @param preservesFloatRegisters whether fs0-fs11 stay preserved across calls
|
||||
*/
|
||||
private record Policy(
|
||||
String id, String prototype, int floatEntryBytes, boolean preservesFloatRegisters) {}
|
||||
|
||||
private static final List<Policy> POLICIES =
|
||||
List.of(new Policy("ilp32-prototype-v1", "__rea_ilp32", 0, false));
|
||||
|
||||
@Override
|
||||
public void run() throws Exception {
|
||||
String[] arguments = getScriptArgs();
|
||||
Policy policy =
|
||||
POLICIES.stream()
|
||||
.filter(candidate -> arguments.length == 1 && candidate.id().equals(arguments[0]))
|
||||
.findFirst()
|
||||
.orElseThrow(
|
||||
() ->
|
||||
new IllegalArgumentException(
|
||||
"REA RISC-V calling convention requires one of the policies "
|
||||
+ POLICIES.stream().map(Policy::id).toList()));
|
||||
Language language = currentProgram.getLanguage();
|
||||
if (!language.getProcessor().toString().equals("RISCV")
|
||||
|| currentProgram.getDefaultPointerSize() != 4)
|
||||
throw new IllegalStateException(
|
||||
"REA " + policy.id() + " requires a 32-bit RISC-V program, not "
|
||||
+ language.getLanguageID());
|
||||
CompilerSpec spec = currentProgram.getCompilerSpec();
|
||||
new SpecExtension(currentProgram)
|
||||
.addReplaceCompilerSpecExtension(
|
||||
derive(spec.getDefaultCallingConvention(), spec, language, policy), monitor);
|
||||
PrototypeModel model =
|
||||
currentProgram.getCompilerSpec().getCallingConvention(policy.prototype());
|
||||
if (model == null)
|
||||
throw new IllegalStateException("REA " + policy.prototype() + " was not installed");
|
||||
selfCheck(model, policy, language);
|
||||
int functions = 0;
|
||||
List<Function> callables = new ArrayList<>();
|
||||
currentProgram.getFunctionManager().getFunctions(true).forEach(callables::add);
|
||||
currentProgram.getFunctionManager().getExternalFunctions().forEach(callables::add);
|
||||
for (Function function : callables) {
|
||||
// A thunk's signature is its target's; the target is in this list.
|
||||
if (function.isThunk()) continue;
|
||||
if (function.hasCustomVariableStorage()) function.setCustomVariableStorage(false);
|
||||
function.setCallingConvention(policy.prototype());
|
||||
if (policy.floatEntryBytes() == 0) {
|
||||
List<VariableStorage> storage = new ArrayList<>();
|
||||
storage.add(function.getReturn().getVariableStorage());
|
||||
for (Parameter parameter : function.getParameters())
|
||||
storage.add(parameter.getVariableStorage());
|
||||
// An untyped function has no assigned storage yet; the decompiler applies the
|
||||
// prototype when it recovers one.
|
||||
requireIntegerStorage(
|
||||
storage.toArray(VariableStorage[]::new), function.getName(true), true);
|
||||
}
|
||||
functions += 1;
|
||||
}
|
||||
int definitions = 0;
|
||||
Iterator<DataType> types = currentProgram.getDataTypeManager().getAllDataTypes();
|
||||
while (types.hasNext()) {
|
||||
if (!(types.next() instanceof FunctionDefinition definition)) continue;
|
||||
definition.setCallingConvention(policy.prototype());
|
||||
definitions += 1;
|
||||
}
|
||||
// Written last: any failure above leaves no report, and REA refuses the session.
|
||||
currentProgram.getOptions(Program.PROGRAM_INFO).setString(
|
||||
REPORT_KEY,
|
||||
"{\"policy\":\"" + policy.id() + "\",\"functions\":" + functions
|
||||
+ ",\"definitions\":" + definitions + "}");
|
||||
println(
|
||||
"REA RISC-V " + policy.prototype() + ": functions=" + functions
|
||||
+ " definitions=" + definitions);
|
||||
}
|
||||
|
||||
/** The default ilp32d prototype changed as the policy states. */
|
||||
private static String derive(
|
||||
PrototypeModel base, CompilerSpec spec, Language language, Policy policy) throws Exception {
|
||||
XmlEncode encoder = new XmlEncode();
|
||||
base.encode(encoder, spec.getPcodeInjectLibrary());
|
||||
DocumentBuilderFactory factory = DocumentBuilderFactory.newInstance();
|
||||
factory.setFeature("http://apache.org/xml/features/disallow-doctype-decl", true);
|
||||
Document document =
|
||||
factory.newDocumentBuilder().parse(new InputSource(new StringReader(encoder.toString())));
|
||||
Element root = document.getDocumentElement();
|
||||
if (!root.getTagName().equals("prototype"))
|
||||
throw new IllegalStateException("Unexpected default prototype encoding " + root.getTagName());
|
||||
root.setAttribute("name", policy.prototype());
|
||||
List<Element> floatEntries = new ArrayList<>();
|
||||
NodeList entries = root.getElementsByTagName("pentry");
|
||||
for (int index = 0; index < entries.getLength(); index += 1) {
|
||||
Element entry = (Element) entries.item(index);
|
||||
if (entry.getAttribute("storage").equals("float")) floatEntries.add(entry);
|
||||
}
|
||||
if (floatEntries.isEmpty())
|
||||
throw new IllegalStateException("Default prototype has no floating-point entries");
|
||||
if (policy.floatEntryBytes() == 0) {
|
||||
List<Node> removed = new ArrayList<>(floatEntries);
|
||||
NodeList rules = root.getElementsByTagName("rule");
|
||||
for (int index = 0; index < rules.getLength(); index += 1) {
|
||||
Element rule = (Element) rules.item(index);
|
||||
if (usesFloatStorage(rule)) removed.add(rule);
|
||||
}
|
||||
if (removed.size() == floatEntries.size())
|
||||
throw new IllegalStateException("Default prototype has no floating-point rules to remove");
|
||||
for (Node node : removed) node.getParentNode().removeChild(node);
|
||||
} else
|
||||
for (Element entry : floatEntries)
|
||||
entry.setAttribute("maxsize", Integer.toString(policy.floatEntryBytes()));
|
||||
// Values over 2 x XLEN go by reference. Returns need no rule: a value that the return
|
||||
// registers cannot hold already becomes a caller buffer passed in a0.
|
||||
NodeList inputs = root.getElementsByTagName("input");
|
||||
if (inputs.getLength() != 1)
|
||||
throw new IllegalStateException("Default prototype has no single input list");
|
||||
Element input = (Element) inputs.item(0);
|
||||
Element last = null;
|
||||
for (Node child = input.getFirstChild(); child != null; child = child.getNextSibling())
|
||||
if (child instanceof Element element && element.getTagName().equals("rule")) last = element;
|
||||
if (last == null || !joinsAnyValue(last))
|
||||
throw new IllegalStateException("Default prototype's input list does not end with a join");
|
||||
Element byReference = document.createElement("rule");
|
||||
Element datatype = document.createElement("datatype");
|
||||
datatype.setAttribute("name", "any");
|
||||
datatype.setAttribute("minsize", "9");
|
||||
byReference.appendChild(datatype);
|
||||
byReference.appendChild(document.createElement("convert_to_ptr"));
|
||||
input.insertBefore(byReference, last);
|
||||
if (!policy.preservesFloatRegisters()) {
|
||||
NodeList killedLists = root.getElementsByTagName("killedbycall");
|
||||
Element killed = killedLists.getLength() == 0 ? null : (Element) killedLists.item(0);
|
||||
if (killed == null) {
|
||||
killed = document.createElement("killedbycall");
|
||||
root.appendChild(killed);
|
||||
}
|
||||
List<Element> moved = new ArrayList<>();
|
||||
NodeList preservedLists = root.getElementsByTagName("unaffected");
|
||||
for (int list = 0; list < preservedLists.getLength(); list += 1) {
|
||||
NodeList preserved = preservedLists.item(list).getChildNodes();
|
||||
for (int index = 0; index < preserved.getLength(); index += 1) {
|
||||
if (preserved.item(index) instanceof Element varnode
|
||||
&& isFloatRegister(varnode, language))
|
||||
moved.add(varnode);
|
||||
}
|
||||
}
|
||||
for (Element varnode : moved) {
|
||||
varnode.getParentNode().removeChild(varnode);
|
||||
killed.appendChild(varnode);
|
||||
}
|
||||
}
|
||||
StringWriter xml = new StringWriter();
|
||||
var transformer = TransformerFactory.newInstance().newTransformer();
|
||||
transformer.setOutputProperty(OutputKeys.OMIT_XML_DECLARATION, "yes");
|
||||
transformer.transform(new DOMSource(document), new StreamResult(xml));
|
||||
return xml.toString();
|
||||
}
|
||||
|
||||
/** The installed prototype places each class of value as its policy states. */
|
||||
private void selfCheck(PrototypeModel model, Policy policy, Language language) {
|
||||
StructureDataType pair = new StructureDataType("rea_float_pair", 0);
|
||||
pair.add(FloatDataType.dataType);
|
||||
pair.add(FloatDataType.dataType);
|
||||
VariableStorage[] storage =
|
||||
model.getStorageLocations(
|
||||
currentProgram,
|
||||
new DataType[] {
|
||||
DoubleDataType.dataType, FloatDataType.dataType, DoubleDataType.dataType,
|
||||
IntegerDataType.dataType, pair, Float16DataType.dataType
|
||||
},
|
||||
false);
|
||||
VariableStorage quad = storage[storage.length - 1];
|
||||
if (quad.size() != currentProgram.getDefaultPointerSize() || quad.getRegister() == null)
|
||||
throw new IllegalStateException(
|
||||
"REA " + policy.prototype() + " does not pass a 16-byte value by reference: " + quad);
|
||||
if (policy.floatEntryBytes() == 0)
|
||||
requireIntegerStorage(storage, policy.prototype() + " self-check", false);
|
||||
if (policy.preservesFloatRegisters()) return;
|
||||
for (Varnode preserved : model.getUnaffectedList()) {
|
||||
Register register = language.getRegister(preserved.getAddress(), preserved.getSize());
|
||||
if (register != null && FLOAT_REGISTER.matcher(register.getName()).matches())
|
||||
throw new IllegalStateException(
|
||||
"REA " + policy.prototype() + " still preserves " + register.getName());
|
||||
}
|
||||
}
|
||||
|
||||
private static boolean joinsAnyValue(Element rule) {
|
||||
List<Element> children = new ArrayList<>();
|
||||
for (Node child = rule.getFirstChild(); child != null; child = child.getNextSibling())
|
||||
if (child instanceof Element element) children.add(element);
|
||||
return children.size() == 2
|
||||
&& children.get(0).getTagName().equals("datatype")
|
||||
&& children.get(0).getAttribute("name").equals("any")
|
||||
&& children.get(1).getTagName().equals("join");
|
||||
}
|
||||
|
||||
/** Whether a rule's action, at any depth, assigns floating-point storage. */
|
||||
private static boolean usesFloatStorage(Element element) {
|
||||
NodeList children = element.getChildNodes();
|
||||
for (int index = 0; index < children.getLength(); index += 1) {
|
||||
if (!(children.item(index) instanceof Element child)) continue;
|
||||
if (child.getAttribute("storage").equals("float") || usesFloatStorage(child)) return true;
|
||||
}
|
||||
return false;
|
||||
}
|
||||
|
||||
private static boolean isFloatRegister(Element varnode, Language language) {
|
||||
if (!varnode.getTagName().equals("varnode")
|
||||
|| !varnode.getAttribute("space").equals("register"))
|
||||
return false;
|
||||
Register register =
|
||||
language.getRegister(
|
||||
language.getAddressFactory().getRegisterSpace()
|
||||
.getAddress(Long.decode(varnode.getAttribute("offset"))),
|
||||
Integer.parseInt(varnode.getAttribute("size")));
|
||||
return register != null && FLOAT_REGISTER.matcher(register.getName()).matches();
|
||||
}
|
||||
|
||||
private static void requireIntegerStorage(
|
||||
VariableStorage[] storage, String owner, boolean allowUnassigned) {
|
||||
for (VariableStorage item : storage) {
|
||||
if (item.isBadStorage() || (!allowUnassigned && item.isUnassignedStorage()))
|
||||
throw new IllegalStateException("REA RISC-V storage unresolved for " + owner);
|
||||
List<Register> registers = item.getRegisters();
|
||||
if (registers == null) continue; // stack-only storage
|
||||
for (Register register : registers) {
|
||||
if (FLOAT_REGISTER.matcher(register.getName()).matches())
|
||||
throw new IllegalStateException(
|
||||
"REA RISC-V storage still uses " + register.getName() + " for " + owner);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -161,8 +161,20 @@ private func unavailableIdentity(pid: Int32, reason: String) -> ProcessIdentityO
|
||||
)
|
||||
}
|
||||
|
||||
enum ProcargsMalformedStage: String, CaseIterable {
|
||||
case header
|
||||
case argumentCount = "argument_count"
|
||||
case executablePath = "executable_path"
|
||||
case pointerSize = "pointer_size"
|
||||
case pathPadding = "path_padding"
|
||||
case arguments
|
||||
case terminator
|
||||
case trailingEntries = "trailing_entries"
|
||||
}
|
||||
|
||||
enum RunTokenReadError: Error {
|
||||
case invalidBuffer
|
||||
case invalidBufferAt(ProcargsMalformedStage)
|
||||
case duplicateToken
|
||||
case invalidTokenEncoding
|
||||
case environmentUnavailable
|
||||
@@ -244,6 +256,8 @@ func processRunTokenObservation(
|
||||
return unavailable(pid: pid, reason: "apple_vector_unavailable")
|
||||
case .invalidBuffer:
|
||||
return unavailable(pid: pid, reason: "malformed_procargs")
|
||||
case .invalidBufferAt(let stage):
|
||||
return unavailable(pid: pid, reason: "malformed_procargs_\(stage.rawValue)")
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -302,27 +316,30 @@ func parseProcessEnvironment(
|
||||
in bytes: [UInt8],
|
||||
pointerSize: Int = MemoryLayout<UnsafeRawPointer>.size
|
||||
) throws -> ParsedRunToken {
|
||||
guard bytes.count >= MemoryLayout<Int32>.size else { throw RunTokenReadError.invalidBuffer }
|
||||
guard bytes.count >= MemoryLayout<Int32>.size else { throw RunTokenReadError.invalidBufferAt(.header) }
|
||||
let argc = bytes.withUnsafeBytes { $0.loadUnaligned(as: Int32.self).littleEndian }
|
||||
guard argc >= 0, Int(argc) <= bytes.count else { throw RunTokenReadError.invalidBuffer }
|
||||
guard argc >= 0, Int(argc) <= bytes.count else { throw RunTokenReadError.invalidBufferAt(.argumentCount) }
|
||||
var offset = MemoryLayout<Int32>.size
|
||||
guard let executablePath = readString(in: bytes, offset: &offset) else {
|
||||
throw RunTokenReadError.invalidBuffer
|
||||
throw RunTokenReadError.invalidBufferAt(.executablePath)
|
||||
}
|
||||
guard pointerSize == 4 || pointerSize == 8 else { throw RunTokenReadError.invalidBuffer }
|
||||
guard pointerSize == 4 || pointerSize == 8 else { throw RunTokenReadError.invalidBufferAt(.pointerSize) }
|
||||
let pathRecordSize = Array("executable_path=".utf8).count + executablePath.count + 1
|
||||
let padding = (pointerSize - pathRecordSize % pointerSize) % pointerSize
|
||||
guard offset + padding <= bytes.count,
|
||||
bytes[offset..<(offset + padding)].allSatisfy({ $0 == 0 }) else {
|
||||
throw RunTokenReadError.invalidBuffer
|
||||
throw RunTokenReadError.invalidBufferAt(.pathPadding)
|
||||
}
|
||||
offset += padding
|
||||
let tokenPrefix = Array("REA_PROCESS_RUN_ID=".utf8)
|
||||
var tokenInArgument = false
|
||||
for _ in 0..<argc {
|
||||
guard readString(in: bytes, offset: &offset, allowEmpty: true) != nil else {
|
||||
throw RunTokenReadError.invalidBuffer
|
||||
guard let argument = readString(in: bytes, offset: &offset, allowEmpty: true) else {
|
||||
throw RunTokenReadError.invalidBufferAt(.arguments)
|
||||
}
|
||||
tokenInArgument = tokenInArgument || argument.starts(with: tokenPrefix)
|
||||
}
|
||||
guard bytes.last == 0 else { throw RunTokenReadError.invalidBuffer }
|
||||
guard bytes.last == 0 else { throw RunTokenReadError.invalidBufferAt(.terminator) }
|
||||
if offset == bytes.count {
|
||||
throw RunTokenReadError.environmentUnavailable
|
||||
}
|
||||
@@ -340,7 +357,7 @@ func parseProcessEnvironment(
|
||||
continue
|
||||
}
|
||||
guard let entry = readString(in: bytes, offset: &offset) else {
|
||||
throw RunTokenReadError.invalidBuffer
|
||||
throw RunTokenReadError.invalidBufferAt(.trailingEntries)
|
||||
}
|
||||
trailingEntries.append(entry)
|
||||
}
|
||||
@@ -361,22 +378,28 @@ func parseProcessEnvironment(
|
||||
throw RunTokenReadError.appleVectorUnavailable
|
||||
}
|
||||
if trailingEntries[appleVectorStart...].contains(where: {
|
||||
$0.starts(with: Array("REA_PROCESS_RUN_ID=".utf8))
|
||||
$0.starts(with: tokenPrefix)
|
||||
}) {
|
||||
throw RunTokenReadError.ambiguousEnvironmentBoundary
|
||||
}
|
||||
let environmentEntries = trailingEntries[..<appleVectorStart]
|
||||
var found: String?
|
||||
for entry in environmentEntries {
|
||||
if entry.starts(with: Array("REA_PROCESS_RUN_ID=".utf8)) {
|
||||
if entry.starts(with: tokenPrefix) {
|
||||
guard found == nil else { throw RunTokenReadError.duplicateToken }
|
||||
let value = entry.dropFirst("REA_PROCESS_RUN_ID=".utf8.count)
|
||||
let value = entry.dropFirst(tokenPrefix.count)
|
||||
guard let decoded = String(bytes: value, encoding: .utf8) else {
|
||||
throw RunTokenReadError.invalidTokenEncoding
|
||||
}
|
||||
found = decoded
|
||||
}
|
||||
}
|
||||
// A changed process title can overwrite the original argument boundaries,
|
||||
// causing a real environment entry to be consumed as an argument. A real
|
||||
// argument can have this form too, so it proves neither ownership nor absence.
|
||||
if tokenInArgument {
|
||||
throw RunTokenReadError.ambiguousEnvironmentBoundary
|
||||
}
|
||||
return ParsedRunToken(runId: found, environmentEntryCount: environmentEntries.count)
|
||||
}
|
||||
|
||||
|
||||
@@ -70,6 +70,7 @@ export default defineConfig({
|
||||
{ text: "Apple applications", link: "/apple-application-analysis" },
|
||||
{ text: "Desktop installations", link: "/desktop-applications" },
|
||||
{ text: "Firmware", link: "/firmware-analysis" },
|
||||
{ text: "iNES containers", link: "/ines-containers" },
|
||||
{ text: "SQLite databases", link: "/sqlite-databases" },
|
||||
{ text: "Packet captures", link: "/packet-captures" },
|
||||
{ text: "Signal recordings", link: "/signal-recordings" },
|
||||
|
||||
+69
-8
@@ -281,6 +281,18 @@ The inventory lists each segment with its ESP-IDF load classification, the
|
||||
checksum and SHA-256 results, the descriptor and any Secure Boot v2 signature
|
||||
blocks. See [ESP images](esp-image.md) for recognition, coverage and refusals.
|
||||
|
||||
Inspect an ESP flash dump, or one read from a flash address other than 0:
|
||||
|
||||
```sh
|
||||
rea inspect-artifact flash.bin --json
|
||||
rea inspect-artifact dump.bin --target-format esp-flash-image --flash-offset 0x1000 --json
|
||||
rea extract-artifact flash.bin --json
|
||||
```
|
||||
|
||||
The inventory lists the bootloader, the partition table and each partition,
|
||||
with ESP image facts for app partitions and the OTA records. See
|
||||
[ESP flash images](esp-flash-image.md) for table location, inputs and limits.
|
||||
|
||||
Inspect a Node.js SEA carrier without running it:
|
||||
|
||||
```sh
|
||||
@@ -317,6 +329,16 @@ overlap and archive expansion checks. The native streaming decoder keeps the
|
||||
existing 128 MiB Zstandard window cap; frames requiring a larger window are
|
||||
refused. Deprecated method 20 and other unregistered ZIP codecs remain unsupported.
|
||||
|
||||
Inventory and extract UF2 or DfuSe regions:
|
||||
|
||||
```sh
|
||||
rea inspect-artifact ./firmware.uf2 --json
|
||||
rea extract-artifact ./firmware.dfu --json
|
||||
```
|
||||
|
||||
See [UF2 and DfuSe images](uf2-dfuse.md) for recognition, families, numbering,
|
||||
the DfuSe suffix CRC and limits.
|
||||
|
||||
Inventory and extract U-Boot FIT images:
|
||||
|
||||
```sh
|
||||
@@ -651,12 +673,45 @@ with `details.reason: path-limit` and `boundary: filesystem-path`.
|
||||
For application traces and version comparisons, see
|
||||
[JavaScript application workflows](javascript-application-workflows.md).
|
||||
MCP [retained Evidence references](mcp-contracts.md#retained-application-evidence-inputs)
|
||||
belong to one connection. Separate CLI calls can consume full saved records.
|
||||
`inspect-analysis-view REQUEST.json --evidence-bundle BUNDLE.json` resolves
|
||||
the request's `retained-evidence` source ID from that authenticated bundle,
|
||||
including bundles exported from an MCP session. The same view JSON works for
|
||||
every supported parent kind; no target is reopened and no record is selected
|
||||
by file order. Omit `--evidence-bundle` when the request supplies inline Evidence.
|
||||
belong to one connection. Separate CLI calls can consume full saved records or
|
||||
resolve exact retained IDs from an explicitly selected local bundle:
|
||||
|
||||
```bash
|
||||
rea trace-application-feature request.json --evidence-bundle /absolute/path/evidence.json --json
|
||||
```
|
||||
|
||||
Here `request.json` uses the same logical Evidence input as MCP, for example
|
||||
`"application": {"kind":"retained-evidence","evidence_id":"ev_<64 lowercase hex characters>"}`.
|
||||
Inline records and references may be mixed in the request's declared source
|
||||
fields, including arrays and nested inputs. The bundle is read and authenticated
|
||||
once for that command; no target is reopened, no producer is rerun, and no
|
||||
record is selected by file order. Source identity, provenance, coverage and
|
||||
partial or unknown facts agree with the complete inline form. When starting
|
||||
from summary or inspection Evidence, select its
|
||||
`normalized_result.parent_evidence_id` for a workflow requiring complete source
|
||||
facts. Omit `--evidence-bundle` when all sources are inline.
|
||||
|
||||
The JSON workflow commands supporting this option are:
|
||||
|
||||
- `trace-application-feature`, `trace-javascript-semantics`,
|
||||
`compare-application-versions`, `compare-javascript-export-shapes`,
|
||||
`compare-source-to-bundle`, and `reconcile-javascript-runtime`.
|
||||
- `import-managed-reconstruction`, `verify-managed-native-boundaries`, and
|
||||
`project-managed-application-graph`.
|
||||
- `project-android-application-graph`, `project-apple-application-graph`, and
|
||||
`project-harmony-application-graph`.
|
||||
- `compare-wasm-interfaces` and `inspect-analysis-view`.
|
||||
|
||||
For `inspect-analysis-view`, the request's source reference selects a parent
|
||||
from the bundle for every supported view kind. Bundles exported from MCP work
|
||||
for these commands. Its inline source form cannot also select a bundle.
|
||||
A missing ID is refused with `details.reason: "missing"`;
|
||||
supplying a bundle does not make incompatible Evidence valid for a workflow.
|
||||
`compare-managed-members` keeps its two assembly paths, and
|
||||
`compare-process-captures` keeps its two capture Evidence JSON paths. The bundle
|
||||
option above selects an input; producer options that write a new bundle, such
|
||||
as `inspect-managed-members --evidence-bundle`, retain their existing meaning.
|
||||
File selectors remain available only on contracts that explicitly support them.
|
||||
|
||||
Compare two artifact inventories with `compare-artifacts`. It takes the same
|
||||
`left` and `right` input as MCP `compare_artifacts`: complete Evidence, or an
|
||||
@@ -669,7 +724,7 @@ rea inspect-artifact new.zip --json > new.json
|
||||
rea compare-artifacts '{"left":{"kind":"evidence-file","path":"old.json"},"right":{"kind":"evidence-file","path":"new.json"}}' --json
|
||||
```
|
||||
|
||||
A one-shot command keeps no session Evidence, so it refuses a
|
||||
A one-shot `compare-artifacts` command keeps no session Evidence, so it refuses a
|
||||
`retained-evidence` source with `details.reason: "missing"`. Select the record
|
||||
from a saved bundle instead.
|
||||
For an artifact summary selected from a bundle, include its complete parent
|
||||
@@ -853,7 +908,13 @@ rea mcp --http --port 8765
|
||||
`--host` and `--port` default to `MCP_HOST` and `MCP_PORT` from the environment,
|
||||
then to `127.0.0.1` and an ephemeral port. A bind outside loopback requires
|
||||
`MCP_ALLOWED_HOSTS` or `--allowed-hosts`, a comma-separated list of client host
|
||||
names or IP addresses without URLs or ports. For example:
|
||||
names or IP addresses without URLs or ports.
|
||||
|
||||
Repeat `--allowed-hosts` to append names, or put multiple names in one
|
||||
comma-separated value. All occurrences are validated together and deduplicated
|
||||
after hostname normalization; the explicit list replaces `MCP_ALLOWED_HOSTS`.
|
||||
|
||||
For example:
|
||||
|
||||
```sh
|
||||
rea mcp --http --host 0.0.0.0 --allowed-hosts 127.0.0.1,rea.lan
|
||||
|
||||
@@ -728,6 +728,32 @@
|
||||
"unpacked"
|
||||
],
|
||||
"additionalProperties": {}
|
||||
},
|
||||
{
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"logical_path": { "type": "string" },
|
||||
"algorithm": { "type": "string", "const": "dfu-crc32" },
|
||||
"declared_crc32": {
|
||||
"type": "string",
|
||||
"pattern": "^[a-f0-9]{8}$"
|
||||
},
|
||||
"calculated_crc32": {
|
||||
"anyOf": [
|
||||
{ "type": "string", "pattern": "^[a-f0-9]{8}$" },
|
||||
{ "type": "null" }
|
||||
]
|
||||
},
|
||||
"unpacked": { "type": "boolean" }
|
||||
},
|
||||
"required": [
|
||||
"logical_path",
|
||||
"algorithm",
|
||||
"declared_crc32",
|
||||
"calculated_crc32",
|
||||
"unpacked"
|
||||
],
|
||||
"additionalProperties": {}
|
||||
}
|
||||
]
|
||||
}
|
||||
|
||||
@@ -0,0 +1,348 @@
|
||||
# ESP flash images
|
||||
|
||||
REA reads a dump of an ESP32-family SPI flash, such as the output of
|
||||
`esptool read-flash 0 ALL dump.bin` or `esptool merge-bin`, through
|
||||
`inspect_artifact` and `extract_artifact`. The reader is static: no device,
|
||||
esptool or ESP-IDF is needed, and no code is run.
|
||||
|
||||
The container format is `esp-flash-image`. Entries are the bootloader region,
|
||||
the partition table sector and one file per partition. The bootloader and
|
||||
each app partition are also decoded as [ESP images](esp-image.md), so their
|
||||
headers, segments, integrity and descriptors appear in the root
|
||||
`format_metadata`. Layout facts are cited to ESP-IDF v6.1
|
||||
(`esp_flash_partitions.h`, `flash_partitions.c`, `gen_esp32part.py`,
|
||||
`bootloader_common_loader.c`, `bootloader_utility.c` and each chip's
|
||||
`esp_rom_caps.h`).
|
||||
|
||||
## Recognition
|
||||
|
||||
A file is a flash image when REA finds a partition table in the entry
|
||||
structure ESP-IDF's bootloader verifies (`esp_partition_table_verify`):
|
||||
32-byte `0x50AA` entries, at most one `0xEBEB` MD5 entry over the entries
|
||||
before it, and a terminating entry within `0xC00` bytes. ESP-IDF accepts a
|
||||
table with no partition entries, but REA requires one, so that neither
|
||||
erased flash nor a lone MD5 entry is read as a table. A table whose MD5 does not match, or that has a second MD5 entry, is
|
||||
still read, with `md5.status` `invalid` and a limitation, because the
|
||||
bootloader would refuse it. The table is located in this order:
|
||||
|
||||
1. at the caller's `partition_table_offset`, when given; no scan follows;
|
||||
2. at the default `0x8000`;
|
||||
3. by scanning every `0x1000`-aligned flash sector for a table whose MD5
|
||||
verifies. This location is derived, and the inventory says so.
|
||||
|
||||
A table never lies inside the located bootloader image, so the default
|
||||
offset and the scan are tried only after the image's last segment. Without
|
||||
a caller selection, the scan runs only when an ESP image sits at a ROM
|
||||
bootloader offset, and a single app or bootloader image ends at its last
|
||||
segment, so neither ordinary files nor single images are scanned. A C3 or
|
||||
S3 dump begins with its bootloader, so flash layouts are detected before a
|
||||
file is read as one ESP image.
|
||||
|
||||
Unless the caller names the table with `partition_table_offset`, REA
|
||||
refuses two kinds of dump before looking for a table:
|
||||
|
||||
- an image of a known chip at another chip's ROM offset, such as an ESP32
|
||||
bootloader at the start of a dump read from `0x1000`. The dump's origin
|
||||
is not the flash address REA assumed, given or default, and a table found
|
||||
there would place every partition at the wrong file offset; pass the
|
||||
`flash_offset` where the dump starts. When a partition table sits where
|
||||
the image's own ROM offset puts `0x8000`, after the image's segments, the
|
||||
refusal names that offset.
|
||||
Without a selected format such a dump opens as its bootloader image, whose
|
||||
[ESP image](esp-image.md) facts say how to reopen it.
|
||||
- an ESP8266 image at a ROM offset. ESP8266 dumps can hold the same table
|
||||
format, and they keep the [ESP image](esp-image.md) reader's ESP8266
|
||||
refusal.
|
||||
|
||||
When no table is found the layout is not recognized. This is an unknown
|
||||
outcome, not a malformed file: flash encryption, which encrypts the
|
||||
bootloader and the table, is one cause. When a table is found without a
|
||||
located bootloader and `flash_offset` is omitted, a limitation says that
|
||||
each partition's file offset assumes the dump starts at flash `0x0`.
|
||||
|
||||
## Caller inputs
|
||||
|
||||
| Input | Meaning |
|
||||
| ------------------------- | ----------------------------------------------------------------------------- |
|
||||
| `format: esp-flash-image` | Read the file as a flash dump; with no table found, refuse it |
|
||||
| `flash_offset` | Flash address (`0x…`) of the dump's first byte; default 0 |
|
||||
| `partition_table_offset` | Flash address (`0x…`, a multiple of `0x1000`) of the table; disables the scan |
|
||||
|
||||
MCP takes these on `open_binary`; the CLI takes `--target-format
|
||||
esp-flash-image`, `--flash-offset` and `--partition-table-offset` on
|
||||
`inspect-artifact` and `extract-artifact`. The offsets require the format, and
|
||||
the format takes no `architecture` or `entry_point`. The inputs are part of
|
||||
the target's identity, so reopening the same bytes with other inputs reads
|
||||
them again instead of reusing the earlier reading. The `open_binary` summary
|
||||
reports them as `flash_offset` and `partition_table_offset`, null when
|
||||
omitted.
|
||||
|
||||
## Layout facts
|
||||
|
||||
The root `format_metadata` (`esp-flash-image`) records:
|
||||
|
||||
- `flash_offset`: its `value` and whether the `caller` gave it or it is the
|
||||
`default`;
|
||||
- `chip`: from the bootloader's `chip_id`, or null without a located
|
||||
bootloader;
|
||||
- `bootloader`: the region from the chip's ROM bootloader offset to the
|
||||
table, with its ESP image facts;
|
||||
- `partition_table`: its flash address, the `method` that found it
|
||||
(`default`, `caller` or `scan`), the partition count and the MD5 `status`
|
||||
(`valid`, `invalid` or `absent`) with the stored and computed digests;
|
||||
- `apps`: ESP image facts for each app partition the dump holds; app
|
||||
partitions declared over the same bytes share one decode;
|
||||
- `otadata`: the otadata partition's records, or null without one;
|
||||
- `boot_selection`: the app the bootloader would select, for every table;
|
||||
- `partitions_outside_dump`: declared partitions that begin before the
|
||||
dump's first byte, which have no entry;
|
||||
- `undescribed`: ranges outside every region, each from a byte that is not
|
||||
erased flash to the end of its gap.
|
||||
|
||||
The ROM loads the bootloader from `ESP_ROM_BOOTLOADER_OFFSET_FLASH`:
|
||||
`0x1000` for ESP32 and ESP32-S2; `0x0` for ESP32-S3, C2, C3, C6, C61, H2 and
|
||||
H21; and `0x2000` for ESP32-C5, H4, P4 and S31. A bootloader is located only
|
||||
where its own `chip_id` says the ROM loads it.
|
||||
|
||||
### Partitions
|
||||
|
||||
Each partition entry is `partitions/<label>.bin` when the label is a safe
|
||||
name of up to 16 letters, digits, `.`, `_` and `-` that starts with a
|
||||
letter, digit or underscore and has at least one letter or underscore, is
|
||||
not a Windows device name such as `NUL` or `COM1`, alone or before a dot,
|
||||
and no other label repeats it, ignoring case as some file systems do; it is
|
||||
`partitions/<index>.bin` otherwise. `source_path` keeps the label, the
|
||||
table's own name for the partition. Its `format_metadata`
|
||||
(`esp-flash-partition`) records the table `index`, `label` (null when not
|
||||
UTF-8), `label_hex` (the label's bytes when it is not UTF-8, else null),
|
||||
`type`, `subtype`, ESP-IDF's `type_name` and `subtype_name`,
|
||||
`flash_address`, `size`, the `encrypted` and `readonly` flags and any
|
||||
`unknown_flags`. The `encrypted` flag says how a device with flash
|
||||
encryption stores the partition; it does not establish that the dump's
|
||||
bytes are ciphertext.
|
||||
|
||||
Each partition entry also records where the bytes the dump holds are
|
||||
written, relative to the partition's start: `written_offset`, the first
|
||||
byte that is not erased flash (`0xFF`), `written_end`, just past the last
|
||||
such byte, and `written_bytes`, how many bytes are not `0xFF`. A partition
|
||||
erased throughout has null offsets and `written_bytes` `0`; a partition the
|
||||
file holds none of has all three null. The range bounds the written
|
||||
bytes and does not say whether erased flash lies between them; extract the
|
||||
partition to read them. REA reads the bytes the partitions cover once, so
|
||||
overlapping partitions share that read however many the table declares.
|
||||
|
||||
ESP-IDF 5.3 and later may declare primary `bootloader` and `partition_table`
|
||||
entries over the bootloader region and the table sector. They are
|
||||
partitions like any other, whose bytes are also `bootloader.bin` and
|
||||
`partition-table.bin`, and they are exempt from the rule that partitions
|
||||
follow the table, as in `gen_esp32part.py`.
|
||||
|
||||
Every partition from the dump's first byte on is an entry holding the bytes
|
||||
the file has. One that the file holds only in part, as when an `esptool
|
||||
merge-bin` dump ends inside its last partition, has an entry limitation that
|
||||
gives the size the table declares; one that lies past the end of the file is
|
||||
an entry with no bytes and a limitation, and has no image facts. A partition
|
||||
that begins before the dump's first byte has no entry and is listed in
|
||||
`partitions_outside_dump`. One unreadable partition does not hide the
|
||||
others: partitions outside the dump or overlapping another are each reported
|
||||
while the rest stay readable.
|
||||
|
||||
### App partitions and the bootloader
|
||||
|
||||
Each app partition and the bootloader region carry their entry's `path`
|
||||
and ESP image `facts`:
|
||||
`status` is `esp-image`, `erased` (the image header bytes are erased flash,
|
||||
as in an empty OTA slot) or `unrecognized` (ciphertext or other data). For an
|
||||
image they hold the [ESP image](esp-image.md) root facts, `segments`,
|
||||
`signature_blocks`, `coverage` and `limitations`, with offsets relative to
|
||||
the region. Extract a partition and inspect it as an ESP image for its
|
||||
segments as entries.
|
||||
|
||||
The summary view (`inspect_artifact` with `detail: "summary"`, or the
|
||||
inventory summary of `inspect_analysis_view`) reports the root facts under
|
||||
their own kind, `esp-flash-image-summary`, and gives each of these images a
|
||||
`summary` instead of its `facts`: `status`, `chip`, `entry_addr`,
|
||||
`segment_count`, `load_ranges` (the load address, declared length and
|
||||
classification of each segment that is mapped, loaded or unclassified),
|
||||
`signature_block_count`, the `descriptor` kind with the fields that tell
|
||||
builds apart (an app's `project_name`, `version`, `idf_ver`,
|
||||
`secure_version` and `app_elf_sha256`; a bootloader's `version`, `idf_ver`
|
||||
and `secure_version`), the `checksum` and `appended_sha256` statuses,
|
||||
`coverage` and `limitations`. The default complete detail keeps the full
|
||||
facts, every segment included, and the summary's `inventory_evidence_ids`
|
||||
name the Evidence that holds them.
|
||||
|
||||
`erased` is the bootloader's view: with no image header at the start of the
|
||||
slot, it finds no app there. An erased slot can still hold data further in,
|
||||
even a complete app image. When it does, its entry's `written_offset` and
|
||||
`written_end` say where, and a limitation says that the slot holds bytes the
|
||||
bootloader does not read; REA does not look for images in them.
|
||||
|
||||
### OTA data
|
||||
|
||||
The otadata partition holds two `esp_ota_select_entry_t` records, one at the
|
||||
start of each 4 KiB sector. Like the bootloader, REA reads the last otadata
|
||||
partition the table declares, and the last factory app and OTA app of each
|
||||
subtype; a limitation names any repeated declaration. REA reports each
|
||||
record the partition and the dump hold: its `ota_seq`, `seq_label`,
|
||||
`ota_state` and its name, `crc`, whether the CRC matches (CRC-32 of
|
||||
`ota_seq` seeded with `0xFFFFFFFF`) and whether the bootloader would treat
|
||||
it as valid.
|
||||
|
||||
### Boot selection
|
||||
|
||||
`boot_selection` is present for every table. It is derived as
|
||||
`bootloader_utility_get_selected_boot_partition` does without app rollback
|
||||
or anti-rollback, and `basis` names its branch:
|
||||
|
||||
- `otadata`: the valid record with the higher `ota_seq` (record 0 on a tie)
|
||||
selects slot `(ota_seq - 1) % ota_app_count`, indexed by OTA subtype;
|
||||
- `factory-fallback`: no record is usable, or the table has no OTA app, and
|
||||
a factory app is declared; or records are set but none is valid, as after
|
||||
a CRC mismatch, which selects the factory index whether or not a factory
|
||||
app is declared;
|
||||
- `ota_0-fallback`: no record is usable, or the table has no OTA app, and no
|
||||
factory app is declared, so the bootloader selects boot index 0;
|
||||
- `no-otadata`: the table declares no otadata partition, or one at offset 0,
|
||||
which the bootloader takes for none, so it selects the factory index, as in
|
||||
the common factory-only layout;
|
||||
- `none`: the bootloader rejects the table, for an MD5 mismatch or a second
|
||||
MD5 entry, or cannot read an otadata partition smaller than two 4 KiB
|
||||
sectors, and resets without selecting an app;
|
||||
- `unknown`: the dump lacks an otadata record the bootloader reads.
|
||||
|
||||
Without an OTA app the records do not matter, so a dump that lacks them
|
||||
still selects a fallback. REA does not compare the partitions with the
|
||||
flash size in the bootloader header, which the bootloader also checks.
|
||||
|
||||
`active_record` and `slot` are set for `otadata` and `slot` for
|
||||
`ota_0-fallback`. `partition_index` is null when the table declares no
|
||||
partition for the selected slot or fallback; the bootloader then tries the
|
||||
other apps. `assumptions` list what the derivation leaves out: records used
|
||||
as written (for the branches that read them), anti-rollback, load fallback,
|
||||
and factory-reset and test-app GPIO holds and a wake from deep sleep; they
|
||||
are empty for `none` and `unknown`. `running_partition` is always
|
||||
`unknown`: at boot the bootloader may first mark a pending-verify record
|
||||
aborted, check the app's secure version against eFuse, and fall back to
|
||||
another partition when an app fails to load.
|
||||
|
||||
## Coverage and extraction
|
||||
|
||||
Erased flash (`0xFF`) between regions is described. In a gap between
|
||||
regions, the first byte that is not erased starts an `undescribed` range to
|
||||
the end of the gap, and makes coverage `unknown`; the rest of that range is
|
||||
not examined. Coverage is also `unknown` when no bootloader is located, or
|
||||
when a declared partition begins before the dump's first byte, since it has
|
||||
no entry. A partition cut short or left out by the end of the file does not
|
||||
make it `unknown`: its entry holds the bytes the file has, none for one past
|
||||
the end, and its limitation says what the file lacks. Coverage describes the
|
||||
inventory of the file, not the observation of every declared flash byte.
|
||||
|
||||
So artifact comparison classifies every partition entry of two dumps. Where
|
||||
both lack the same bytes, as two `merge-bin` dumps of the same layout do,
|
||||
the entries are equal; a dump against a longer one reports each partition
|
||||
it cuts short or lacks as `changed`, and that entry's limitation says the
|
||||
file did not hold the bytes.
|
||||
|
||||
`extract_artifact` writes every entry as an ordinary file, `nvs_keys`
|
||||
partitions included. On a device without flash encryption an `nvs_keys`
|
||||
partition holds the NVS encryption keys in plain text, and a dump of it
|
||||
contains them. REA decrypts nothing, recovers no keys and does not interpret
|
||||
NVS.
|
||||
|
||||
## Comparison
|
||||
|
||||
Artifact comparison uses each partition's `label`, `label_hex`, `type`,
|
||||
`subtype`, `flash_address`, `size` and flags as its declaration, so a
|
||||
partition that moves, changes type or flags, or is renamed with unchanged
|
||||
bytes is `changed` in `metadata`. Partitions pair by path. The root facts
|
||||
are not compared as a declaration: they derive from the dump's bytes, which
|
||||
content comparison covers, and from the caller's `flash_offset` and
|
||||
`partition_table_offset`, which are part of the target's identity and
|
||||
decide each partition's byte range.
|
||||
|
||||
## Limits
|
||||
|
||||
- A flash dump inside another container, such as a ZIP, is classified from
|
||||
its first bytes and so reads as an ESP image, not as a flash layout; extract
|
||||
it first.
|
||||
- ESP8266 dumps are refused; Secure Boot v1 bootloader digests and
|
||||
flash-encrypted layouts are not read.
|
||||
- Partitions are not checked against the flash chip size, which the
|
||||
bootloader also verifies.
|
||||
- NVS, FAT, SPIFFS and other data partitions are extracted, not decoded.
|
||||
- An image nested in a partition is decoded for its facts; its segments are
|
||||
not inventory entries.
|
||||
|
||||
## Verification
|
||||
|
||||
Synthetic fixtures with computed MD5s and CRCs, and one CRC otatool wrote,
|
||||
cover the table rules, every way the table is located, the scan's chunk and
|
||||
sector edges, the misplaced-origin guard with the origin omitted or given,
|
||||
the ESP8266 refusal, the `0x0`, `0x1000` and `0x2000` ROM bootloader
|
||||
offsets, single images that are not scanned, overlaps, primary entries,
|
||||
truncation and partitions outside the dump, index names for unsafe labels,
|
||||
repeated declarations, the otadata states and records, and each branch of
|
||||
the bootloader's selection. CLI and stdio MCP tests exercise the inputs and
|
||||
their identity and extraction. MCP `compare_artifacts` tests compare
|
||||
partition declarations, labels that are not UTF-8 included, two dumps that
|
||||
end inside their last partition, and such a dump against a full one.
|
||||
|
||||
The optional real lane builds its dumps with esptool and ESP-IDF's own tools:
|
||||
|
||||
```sh
|
||||
REA_RISCV_CLANG=/path/to/clang REA_RISCV_LLD=/path/to/ld.lld \
|
||||
REA_ESPTOOL_PYTHON=/path/to/python-with-esptool \
|
||||
REA_ESP_IDF_PATH=/path/to/esp-idf \
|
||||
npm run verify:esp-flash
|
||||
```
|
||||
|
||||
It converts the conformance ELFs with esptool, builds a default, a
|
||||
relocated and a primary-entry partition table with `gen_esp32part.py`,
|
||||
assembles dumps with `esptool merge-bin`, one of them written from flash
|
||||
`0x1000` with `--target-offset`, and writes OTA records with `otatool`
|
||||
through `tests/conformance/esp/flash_file_tool.py`. That script replaces
|
||||
parttool's esptool calls with reads and writes of the dump file, so
|
||||
otatool's own OTA logic runs unchanged. The lane then checks:
|
||||
|
||||
- the bootloader and app facts against `esptool image-info`;
|
||||
- every partition against `gen_esp32part.py`'s decoding of the table REA
|
||||
extracted;
|
||||
- both OTA records against `otatool read_otadata` after switching to slot 0,
|
||||
then to slot 1, then erasing the otadata, and the derived selection after
|
||||
each: slot 0, slot 1, then the factory fallback;
|
||||
- that each extracted file is the dump's bytes at its range and that the
|
||||
source is unchanged;
|
||||
- a table with the primary `bootloader` and `partition_table` entries that
|
||||
`--primary-bootloader-offset` fills, read with complete coverage and no
|
||||
overlap or misplacement limitation; it has no otadata, so its boot
|
||||
selection is `no-otadata` with the factory app;
|
||||
- the relocated table, found by scanning and named with
|
||||
`partition_table_offset`, and the dump written from `0x1000`, read with
|
||||
`flash_offset` and, without it, left an ESP image whose `flash_dump`
|
||||
names `flash_offset` `0x1000`;
|
||||
- an app with a flipped byte beside valid partitions, and a truncated dump,
|
||||
read with complete coverage and its missing partitions as entries with no
|
||||
bytes.
|
||||
|
||||
The lane ran on macOS arm64 with Node 24.22.1, Homebrew Clang and LLD 21.1.8
|
||||
and ESP-IDF v5.5.2's `gen_esp32part.py`, `otatool.py` and `parttool.py`,
|
||||
once with esptool v5.6.0 and once with 4.12.0. Its `gen_esp32part.py` is
|
||||
identical to v6.1's, and its otatool and parttool differ only in import style
|
||||
and esptool command spelling. All checks passed in both runs. With v5.6.0 and
|
||||
the build already cached, the run took 24.1 s elapsed, 16.9 s user CPU and
|
||||
6.3 s system CPU under `/usr/bin/time -l`; the largest single process peaked
|
||||
at a 278 MiB resident set, and the combined peak of all processes was not
|
||||
measured.
|
||||
|
||||
The lane does not establish:
|
||||
|
||||
- dumps read from a device: its dumps come from `merge-bin`;
|
||||
- chips other than ESP32-C3 and ESP32;
|
||||
- `ota_state` and record validity, which `read_otadata` does not print, or
|
||||
OTA states the bootloader writes itself, app rollback or anti-rollback;
|
||||
- selection with gaps or repeated OTA subtypes, which otatool does not
|
||||
write;
|
||||
- flash-encrypted dumps, which are not recognized by design.
|
||||
|
||||
It does not run in CI.
|
||||
+20
-3
@@ -223,9 +223,26 @@ first byte that is not erased starts a range to the end of that region. A
|
||||
slot that is neither a block nor erased is reported whole. Adjacent ranges
|
||||
are merged.
|
||||
|
||||
A whole-flash dump of a chip whose bootloader is at offset 0, such as
|
||||
ESP32-C3 or ESP32-S3, is recognized as that bootloader image; the partition
|
||||
table, apps and data follow as undescribed bytes.
|
||||
A whole-flash dump whose partition table REA finds is read as an
|
||||
[ESP flash image](esp-flash-image.md). Without one, a dump of a chip whose
|
||||
bootloader is at offset 0, such as ESP32-C3 or ESP32-S3, is recognized as that
|
||||
bootloader image, and the partition table, apps and data follow as
|
||||
undescribed bytes.
|
||||
|
||||
A dump read from a later flash address, such as an ESP32 dump read from
|
||||
`0x1000` with `read-flash 0x1000 …` or `merge-bin --target-offset 0x1000`,
|
||||
opens as its bootloader image. REA says so when the file shows it: the image
|
||||
at file offset 0 is not an app, its chip's ROM loads the bootloader from a
|
||||
later address (`0x1000` for ESP32 and S2, `0x2000` for C5, H4, P4 and S31),
|
||||
and a partition table sits where that origin puts `0x8000` (file offset
|
||||
`0x7000` for `0x1000`), after the image ends. Then `flash_dump` records
|
||||
`flash_offset` and
|
||||
`table_file_offset`, a limitation says to reopen the file with
|
||||
`format: "esp-flash-image"` and that `flash_offset`, and `inspect_artifact`
|
||||
lists the same `open_binary` call as a next probe. `flash_dump` is null for
|
||||
every other image. Evidence that REA recorded before it reported the field
|
||||
omits it; REA still reads and compares such Evidence, and the omission says
|
||||
nothing about where the file was read from.
|
||||
|
||||
## Limits
|
||||
|
||||
|
||||
@@ -300,6 +300,17 @@ v2 signature blocks. `extract-artifact` / `extract_artifact` writes one file
|
||||
per segment and signature block. Reading is portable and needs no external
|
||||
tool. See [ESP images](esp-image.md).
|
||||
|
||||
## Inspect ESP flash dumps
|
||||
|
||||
Use the same tools on a whole-flash dump, such as `esptool read-flash 0 ALL`
|
||||
output. The inventory reports the bootloader, the partition table and how it
|
||||
was found, each partition with its declared type, flags and digest, ESP image
|
||||
facts for the bootloader and each app partition, and the OTA records with a
|
||||
derived selection; the running partition stays unknown. Extraction writes
|
||||
each partition, `nvs_keys` included, as an ordinary file. For a dump that
|
||||
does not start at flash address 0, select `esp-flash-image` with
|
||||
`flash_offset`. See [ESP flash images](esp-flash-image.md).
|
||||
|
||||
## Inspect extracted devicetrees
|
||||
|
||||
Use [`inspect_devicetree`](devicetree.md) / `rea inspect-devicetree` on an
|
||||
@@ -329,6 +340,15 @@ payloads stay as stored. Intel HEX/S-record and FIT images are separate
|
||||
readers and are not covered here. See [U-Boot legacy uImage](uimage.md) for
|
||||
component padding, preserved names, resource policies and verification scope.
|
||||
|
||||
## Inspect UF2 and DfuSe images
|
||||
|
||||
Use `inspect-artifact` / `inspect_artifact` on a UF2 file for each familyID's
|
||||
address regions and its block numbering, or on a DfuSe file for its targets,
|
||||
elements and DFU suffix, then `extract-artifact` / `extract_artifact` for one
|
||||
raw file per region or element. A DfuSe suffix CRC mismatch follows
|
||||
`integrity_policy`. Decoding is portable and requires no external tool. See
|
||||
[UF2 and DfuSe images](uf2-dfuse.md).
|
||||
|
||||
## Inspect U-Boot FIT images
|
||||
|
||||
Use `inspect-artifact` / `inspect_artifact` on a U-Boot FIT (`.itb`) for its
|
||||
|
||||
+20
-15
@@ -39,9 +39,9 @@ bridge-reported `language_id` after import. A caller-selected
|
||||
pin; combining the pinned profile with an environment override is a typed
|
||||
refusal. A header that declares an ABI field but carries no record is still
|
||||
refused. Two further boundaries apply: an inspection that could not examine
|
||||
every ABI declaration source (for example a carved ELF whose section table
|
||||
extends past the captured bytes without a program-header record) does not
|
||||
establish the recordless admission, and an `EF_MIPS_ARCH_1` or
|
||||
every ABI declaration source does not establish the recordless admission
|
||||
(target opening already refuses a carved ELF whose section table extends past
|
||||
the captured bytes, as described below), and an `EF_MIPS_ARCH_1` or
|
||||
`EF_MIPS_ARCH_32` header that _declares_ an ABI is refused rather than
|
||||
relabeled, because the declared-ABI lane is verified for
|
||||
`EF_MIPS_ARCH_32R2` headers only. The recordless MIPS-I lane pins the same
|
||||
@@ -77,13 +77,16 @@ argument ABI and unknown nonzero MSA values remain unsupported. Without a
|
||||
separate ABI flags record, FP attribute values are admitted only when absent,
|
||||
`ANY` (0), double precision (1), or FPXX (5), the values covered by the
|
||||
recordless profile. Parsed values are retained in the MIPS analysis profile.
|
||||
For carved ELFs whose section table extends
|
||||
beyond the captured file, REA examines every complete captured section header
|
||||
and records the unavailable headers as a limitation. The provider refuses an
|
||||
incomplete ABI declaration inspection even when a program-header ABI record is
|
||||
valid, because an unavailable section header could carry an unsupported or
|
||||
conflicting declaration. A missing program-header record does not make ABI
|
||||
declarations in uncaptured sections absent or supply a default ABI.
|
||||
A carved ELF whose section header table extends beyond the captured file is
|
||||
refused when the target is opened, before provider selection, as is every ELF
|
||||
declaration beyond EOF ([installation](installation.md#ghidra)). The refusal
|
||||
names the declared range and the file size. It applies even when a
|
||||
program-header ABI record was captured: an uncaptured section header could
|
||||
carry an unsupported or conflicting declaration, so the captured bytes are not
|
||||
analyzed as a complete smaller program. The provider also refuses any ABI
|
||||
declaration inspection that reports unexamined headers. A missing
|
||||
program-header record does not make ABI declarations in uncaptured sections
|
||||
absent or supply a default ABI.
|
||||
The field layout follows
|
||||
[`Elf_MIPS_ABIFlags_v0`](https://github.com/bminor/glibc/blob/master/elf/elf.h).
|
||||
|
||||
@@ -257,11 +260,13 @@ observations and does not relabel the upstream result as verified.
|
||||
|
||||
CLI/MCP discovery and decompilation, direct calls, target/profile/Evidence
|
||||
identity, snapshot round-trip and final process cleanup remain covered.
|
||||
For both byte orders, the lane also carves away the section table while retaining
|
||||
all program-header payloads. It checks the actual Ghidra language, entry-address
|
||||
decompilation, distinct artifact digest and missing-section limitations through
|
||||
CLI Evidence and an MCP-produced snapshot. Caller-selected Ghidra heap limits
|
||||
are passed to these child workflows.
|
||||
For both byte orders, the lane also cuts the image one byte before its section
|
||||
header table, after checking that every program header and payload stays
|
||||
captured. `open_binary` and the CLI must both refuse that copy before provider
|
||||
selection as `target_unavailable` with constraint `file_format_rejected`,
|
||||
naming the declared section-table range and the captured size, and leave it
|
||||
unchanged. Caller-selected Ghidra heap limits are passed to the lane's CLI and
|
||||
MCP child workflows.
|
||||
For soft-float fixtures the lane also checks, through `inspect_native_api`, the
|
||||
exact parameter and return types and storage of float, double, mixed (pair
|
||||
alignment and stack spill) and struct-returning functions; that a caller's
|
||||
|
||||
@@ -31,7 +31,8 @@ operation-scoped JVM `OutOfMemoryError` observations retain their separate resou
|
||||
classification. Automatic kernel or cgroup OOM attribution is not implemented.
|
||||
|
||||
After the authenticated startup handshake, rejected inputs (`invalid_request`,
|
||||
`not_found`, `ambiguous`, or an invalid function name) use
|
||||
`not_found`, `ambiguous`, an invalid function name, or an address without one
|
||||
exact file offset) use
|
||||
`diagnostics.diagnostic_scope: "request"` when their active output checkpoint is
|
||||
available. They preserve the active request's
|
||||
`failure_output`, target/profile identity and runtime/log coordinates, while
|
||||
|
||||
+3
-3
@@ -85,9 +85,9 @@ node scripts/verify-real-ghidra-mips.mjs --psp-missing-extension
|
||||
npm run verify:ghidra:mips
|
||||
```
|
||||
|
||||
The generic verifier also exercises a section-truncated firmware fixture. That
|
||||
check uses generic MIPS symbols and language expectations and runs only in generic
|
||||
mode; the PSP report explicitly records carved-PSP coverage as not run.
|
||||
The generic verifier also checks that a section-truncated firmware copy is
|
||||
refused when the target is opened. That check runs only in generic mode; the
|
||||
PSP report explicitly records carved-PSP coverage as not run.
|
||||
|
||||
The maintained PSP fixture builds from `tests/conformance/psp/`, never executes,
|
||||
and is independent of commercial game data. PSP GNU tools supply ELF/ABI,
|
||||
|
||||
+264
-40
@@ -9,29 +9,43 @@ lane shape as [generic MIPS support](ghidra-mips.md).
|
||||
|
||||
ELF `EM_RISCV` (243) is retained as the provider-neutral `riscv` family. The
|
||||
target also retains the raw ELF declarations that decide a lane: class, byte
|
||||
order, `e_type`, and `e_flags`. Recognition does not mean every provider or
|
||||
RISC-V variant is supported. Existing PE, managed and Mach-O architecture sets
|
||||
are unchanged. Headerless inputs use the separate [raw-binary admission
|
||||
order, `e_type`, `e_flags`, and the ISA string that `.riscv.attributes`
|
||||
declares. Recognition does not mean every provider or RISC-V variant is
|
||||
supported. Existing PE, managed and Mach-O architecture sets are unchanged.
|
||||
Headerless inputs use the separate [raw-binary admission
|
||||
boundary](raw-binary.md); raw RISC-V images are not supported.
|
||||
|
||||
The Ghidra lane admits little-endian ELF32 and ELF64 `ET_EXEC` images that
|
||||
declare `EF_RISCV_FLOAT_ABI_DOUBLE` in `e_flags`, with no `EF_RISCV_RVE`,
|
||||
`EF_RISCV_TSO`, or any bit outside `EF_RISCV_RVC | EF_RISCV_FLOAT_ABI`. These are
|
||||
header declarations, not proof that every instruction conforms to that ISA.
|
||||
The Ghidra lane admits little-endian `ET_EXEC` images with no `EF_RISCV_RVE`,
|
||||
`EF_RISCV_TSO`, or any bit outside `EF_RISCV_RVC | EF_RISCV_FLOAT_ABI`, in one
|
||||
of three lanes:
|
||||
|
||||
- ELF64 or ELF32 images that declare `EF_RISCV_FLOAT_ABI_DOUBLE` use Ghidra's
|
||||
stock double-float model unchanged. Their `.riscv.attributes` is not read for
|
||||
admission.
|
||||
- ELF32 images that declare the soft-float ABI (`e_flags` float-ABI bits `0`,
|
||||
ilp32) use REA's [ilp32 calling-convention policy](#ilp32). They must also
|
||||
declare an RV32I base and only [decoded extensions](#isa-declarations) in
|
||||
`Tag_RISCV_arch`.
|
||||
|
||||
These are header declarations, not proof that every instruction conforms to
|
||||
that ISA.
|
||||
|
||||
Provider selection returns the specific unsupported constraint rather than
|
||||
guessing a compatible profile:
|
||||
|
||||
| Declaration | Outcome |
|
||||
| ----------------------------------------------- | ---------------------------------------------------------------------------------------- |
|
||||
| Little-endian ELF64 `ET_EXEC`, double-float ABI | admitted, `RISCV:LE:64:default` + `gcc` |
|
||||
| Little-endian ELF32 `ET_EXEC`, double-float ABI | admitted, `RISCV:LE:32:default` + `gcc` |
|
||||
| Big-endian | refused; Ghidra 12.1.4 ships no big-endian RISC-V language |
|
||||
| `ET_REL`, `ET_DYN`, other `e_type` | refused with the declared value |
|
||||
| `EF_RISCV_RVE` | refused; the embedded 16-register profile (x0–x15) needs its own processor specification |
|
||||
| `EF_RISCV_TSO` | refused; the total-store-order memory model is unverified here |
|
||||
| Soft, single, or quad float ABI | refused; the pinned generic compiler specification models double-float |
|
||||
| Reserved or newer `e_flags` bits | refused with the offending bits named |
|
||||
| Declaration | Outcome |
|
||||
| -------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------- |
|
||||
| Little-endian ELF64 `ET_EXEC`, double-float ABI | admitted, `RISCV:LE:64:default` + `gcc`, lane `elf64-exec-le-double-float-generic-v1` |
|
||||
| Little-endian ELF32 `ET_EXEC`, double-float ABI | admitted, `RISCV:LE:32:default` + `gcc`, lane `elf32-exec-le-double-float-generic-v1` |
|
||||
| Little-endian ELF32 `ET_EXEC`, soft-float ABI, decoded extensions only | admitted, `RISCV:LE:32:default` + `gcc` with the `__rea_ilp32` prototype, lane `elf32-exec-le-soft-float-generic-v1` |
|
||||
| ELF32 soft-float ABI without `Tag_RISCV_arch`, or with any other extension | refused; the reason names the extensions outside the decoded set |
|
||||
| ELF32 single-float ABI (ilp32f) | refused until its [written policy](#ilp32f) is implemented and verified |
|
||||
| ELF64 soft- or single-float ABI (lp64, lp64f), or the quad-float ABI | refused; no verified calling-convention policy |
|
||||
| Big-endian | refused; Ghidra 12.1.4 ships no big-endian RISC-V language |
|
||||
| `ET_REL`, `ET_DYN`, other `e_type` | refused with the declared value |
|
||||
| `EF_RISCV_RVE` | refused; the embedded 16-register profile (x0–x15) needs its own processor specification |
|
||||
| `EF_RISCV_TSO` | refused; the total-store-order memory model is unverified here |
|
||||
| Reserved or newer `e_flags` bits | refused with the offending bits named |
|
||||
|
||||
`e_type` is the only load-shape declaration this lane reads. A fixed-address
|
||||
`ET_EXEC` declaration does **not** establish static linkage: `PT_INTERP` and
|
||||
@@ -58,15 +72,193 @@ with the `gcc` compiler specification per class, records
|
||||
pin without changing the requested override. An override that matches the pin is
|
||||
accepted and recorded as `configured-v1`.
|
||||
|
||||
The soft-float lane pins the same `RISCV:LE:32:default` language. Every RV32
|
||||
language in Ghidra 12.1.4, including the deprecated soft-float `RV32I`, `RV32IC`
|
||||
and `RV32IMC` variants in `old/riscv_deprecated.ldefs`, uses the same
|
||||
`riscv.ilp32d.sla` decoder; only the compiler specification differs. REA keeps
|
||||
the generic language and corrects the calling convention instead.
|
||||
|
||||
## Calling conventions
|
||||
|
||||
Ghidra's generic RV32 compiler specification, `riscv32-fp.cspec`, models the
|
||||
ilp32d ABI. Its default prototype passes `float` and `double` values in
|
||||
`fa0`–`fa7` and returns them in `fa0`, and it passes a structure of one or two
|
||||
floating-point members in floating-point registers. An ilp32 image passes all
|
||||
of these in integer registers. Measured on Ghidra 12.1.4 with an ilp32 build of
|
||||
the fixture in `tests/conformance/c/riscv-float-abi.c`, the stock model
|
||||
decompiles `rea_riscv_abi_pick(1.5f, 2.5f, sel)` as
|
||||
`rea_riscv_abi_pick(in_fa0,in_fa1,0x3fc00000)`.
|
||||
|
||||
### Decision record: D3, RISC-V float ABIs
|
||||
|
||||
Decided in [#7477](https://github.com/morluto/rea/issues/7477).
|
||||
|
||||
- **Choice.**
|
||||
- Admit little-endian ELF32 `ET_EXEC` images that declare the soft-float ABI
|
||||
(ilp32) through a new lane, `elf32-exec-le-soft-float-generic-v1`. Lane ids
|
||||
are built from their fields: `elf<class>-exec-le-<float ABI>-float-generic-v1`.
|
||||
- Keep the pinned `RISCV:LE:32:default` language and `gcc` compiler
|
||||
specification. After analysis, `ReaGhidraRiscvCallingConvention.java`
|
||||
derives the program-specific `__rea_ilp32` prototype from the program's
|
||||
default prototype as the [ilp32 policy](#ilp32) states. It installs the
|
||||
prototype through a Ghidra specification extension and assigns it, with
|
||||
dynamic storage, to every non-thunk function, every external function and
|
||||
every function definition. Thunks follow their targets. The script fails,
|
||||
and writes no report, if the derived prototype or any function still
|
||||
places a value in a floating-point register, or if the prototype does not
|
||||
pass a 16-byte value by reference.
|
||||
- One derivation serves every policy. A policy row says how it changes the
|
||||
default prototype's floating-point entries and preserved registers; every
|
||||
policy passes values of 9 bytes or more by reference.
|
||||
- The profile commits `riscv_calling_convention: ilp32-prototype-v1`. The
|
||||
bridge reports the policy and its function and definition counts, and REA
|
||||
requires that report exactly when the profile commits the policy.
|
||||
- Admission requires a `Tag_RISCV_arch` declaration with an RV32I base and
|
||||
only extensions that Ghidra 12.1.4's generic RV32 language implements with
|
||||
p-code ([ISA declarations](#isa-declarations)).
|
||||
- Soft-float profiles add `float_abi` and `arch` to `riscv_elf`. Double-float
|
||||
profiles keep their exact parameters, so their digests, sessions and
|
||||
snapshots are unchanged.
|
||||
- ilp32f keeps its [written policy](#ilp32f) and stays refused until its
|
||||
prototype is implemented and verified. Quad-float, RVE, TSO, and ELF64
|
||||
soft- or single-float images stay refused.
|
||||
- **Alternatives** (and what switching to each would take):
|
||||
- Pin the deprecated soft-float languages (`RISCV:LE:32:RV32IMC` and others,
|
||||
`riscv32.cspec`): REA's language catalog would have to read `old/`, the
|
||||
variants cover only I, IC and IMC, Ghidra marks them deprecated, and no
|
||||
stock specification models ilp32f.
|
||||
- Assign `__rea_ilp32` only to callables whose types use floating-point
|
||||
values, as the MIPS soft-float lane does: typed storage is the same, but
|
||||
prototypes Ghidra infers for untyped code would keep the ilp32d model. In
|
||||
an ilp32 image that contains F or D instructions, that model can still
|
||||
claim `fa0`. Switching is one predicate in the Java script and a new
|
||||
policy id.
|
||||
- Admit with a limitation and no correction: a reduced-capability contract
|
||||
that needs its own review, because arguments, returns and aggregates
|
||||
become unverified.
|
||||
- Keep refusing: the lane rows become refusals again.
|
||||
- Admit images without `Tag_RISCV_arch`, or refuse a list of known
|
||||
extensions instead of admitting a list: a declaration that REA cannot
|
||||
check would be treated as compatible. Widening the decoded set later is
|
||||
additive.
|
||||
- Add `float_abi` and `arch` to double-float profiles as well: one shape for
|
||||
every lane, but every existing RISC-V session and snapshot would be
|
||||
invalidated.
|
||||
- Ship ilp32 and ilp32f together (the question in #7477): ilp32f needs a
|
||||
float-and-integer structure rule that Ghidra's model lacks, so it ships
|
||||
separately with its own fixtures.
|
||||
- **Semantic owner and consumers.**
|
||||
- Owners:
|
||||
- `src/domain/riscvIsaDeclaration.ts` owns the float-ABI vocabulary of
|
||||
`e_flags` and the parsing of `.riscv.attributes` and `Tag_RISCV_arch`.
|
||||
- `GHIDRA_RISCV_LANES`, `GHIDRA_RISCV_CALLING_CONVENTIONS` and
|
||||
`GHIDRA_RISCV32_DECODED_EXTENSIONS` (`src/ghidra/GhidraRiscvProfile.ts`)
|
||||
own admission, lane ids and the policy table.
|
||||
- `bridge/ghidra/ReaGhidraRiscvCallingConvention.java` owns how each policy
|
||||
becomes a Ghidra prototype.
|
||||
- Consumers: target resolution, Ghidra provider admission, the analysis
|
||||
profile, the provider client and launcher, the session handshake, project
|
||||
history, the real-Ghidra lane and this page. Later work reuses the table:
|
||||
ilp32f adds a row, and RISC-V ESP images (F6) select a row through D5.
|
||||
- **Costs of changing it.**
|
||||
- **Code:** S. A policy or lane is one table row plus the Java derivation.
|
||||
- **Compatibility:** the soft-float lane id, `riscv_elf.float_abi`,
|
||||
`riscv_elf.arch` and `riscv_calling_convention` are persisted in sessions,
|
||||
snapshots and Evidence once released. A changed derivation ships as a new
|
||||
policy id, such as `ilp32-prototype-v2`, which invalidates soft-float
|
||||
snapshots only.
|
||||
- **Migration:** none for admitted targets; double-float profiles are pinned
|
||||
by tests. Previously refused soft-float targets become admitted.
|
||||
- **Verification:** rerun `npm run verify:ghidra:riscv`.
|
||||
|
||||
### ilp32
|
||||
|
||||
Policy `ilp32-prototype-v1`, prototype `__rea_ilp32`. XLEN is 32 bits and no
|
||||
floating-point register passes a value.
|
||||
|
||||
| Topic | psABI rule | `__rea_ilp32` |
|
||||
| -------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| Integer and pointer values | `a0`–`a7`, then the stack in 4-byte-aligned slots | The default integer and stack entries, unchanged |
|
||||
| `float` | One integer register, or a stack slot | The `fa0`–`fa7` entries and the `float` rule are removed, so a `float` takes the next integer register |
|
||||
| `double` and 8-byte values | A register pair with the low word in the lower register. Named arguments need no aligned pair. With one register left, the low word goes in `a7` and the high word on the stack | The default `join` rule with stack spill: Ghidra writes `a2:4,a1:4` for a pair and `Stack[0x0]:4,a7:4` for the split |
|
||||
| Aggregates up to 8 bytes | One or two integer registers, laid out as in memory, also when every member is floating-point | The homogeneous-float-aggregate rule is removed; the `join` rule applies |
|
||||
| Aggregates over 8 bytes | By reference: the caller passes the address of a copy | The default rule that converts structures and unions of 9 bytes or more to pointers |
|
||||
| Return values | As the first argument of the same type: `a0`, or `a0` and `a1`. Larger values go to a caller buffer whose address is an implicit first argument in `a0` | The `fa0`/`fa1` return entries and floating-point rules are removed; Ghidra's hidden-return fallback supplies the pointer in `a0` |
|
||||
| Variadic arguments | The integer convention. A variadic argument with 8-byte alignment and size takes an aligned (even-numbered) register pair, or the stack; after the first stack argument, the rest go on the stack | The default variadic rule (`join` with alignment), unchanged |
|
||||
| Preserved registers | `s0`–`s11` and `sp`; `gp` and `tp` are not allocated. No floating-point register is preserved | The default preserved list without `fs0`–`fs11`, which join the killed-by-call list |
|
||||
| Scalars over 8 bytes | By reference (`long double` is 16 bytes), and returned through a caller buffer | A rule before the final join converts any value of 9 bytes or more to a pointer; a return that size already goes to a caller buffer |
|
||||
|
||||
Ghidra recovers the arguments of a variadic call from the registers the caller
|
||||
writes, not from their types. A padding register that still holds an earlier
|
||||
value can appear as an argument, and arguments after an unwritten register are
|
||||
omitted.
|
||||
|
||||
### ilp32f
|
||||
|
||||
Written policy for `EF_RISCV_FLOAT_ABI_SINGLE` on ELF32. REA does not admit it
|
||||
yet. ABI_FLEN is 32 bits.
|
||||
|
||||
| Topic | psABI rule |
|
||||
| ------------------------------------------------------- | ------------------------------------------------------------------------------------------------ |
|
||||
| `float` | `fa0`–`fa7` while one is free, then the integer convention |
|
||||
| `double` and other values wider than 32 bits | The integer convention, as in ilp32 |
|
||||
| Structure of one or two floats (after flattening) | One floating-point register per member when enough are free, otherwise the integer convention |
|
||||
| Structure of one float and one integer, in either order | One floating-point and one integer register when both are free, otherwise the integer convention |
|
||||
| Unions | Never flattened; the integer convention |
|
||||
| Return values | As the first argument: `fa0`, `fa0` and `fa1`, or `fa0` and `a0` |
|
||||
| Variadic arguments | The integer convention, as in ilp32 |
|
||||
| Preserved registers | `s0`–`s11`, `sp`, and `fs0`–`fs11` for values of 32 bits or fewer |
|
||||
|
||||
The planned prototype limits the default floating-point entries to 4 bytes and
|
||||
keeps the shared rule that passes values of 9 bytes or more by reference.
|
||||
Observed on Ghidra 12.1.4, a `double` then takes an integer pair and a structure
|
||||
of two floats takes `fa0` and `fa1`. Ghidra's model has no rule for a structure
|
||||
of one float and one integer: it joins `struct { float; int; }` into `a1:a0`
|
||||
instead of `fa0` and `a0`. The ilp32f policy needs that rule and its own
|
||||
fixtures before it is admitted.
|
||||
|
||||
### ilp32d and lp64d
|
||||
|
||||
The double-float lanes use Ghidra's stock `riscv32-fp.cspec` and
|
||||
`riscv64-fp.cspec` models unchanged. This page does not verify them. They share
|
||||
the float-and-integer structure gap above.
|
||||
|
||||
## ISA declarations
|
||||
|
||||
The target reader finds the one `SHT_RISCV_ATTRIBUTES` section and reads
|
||||
`Tag_RISCV_arch`, such as `rv32i2p1_m2p0_c2p0_zicsr2p0_zifencei2p0`. It
|
||||
retains the string, the base and the extension names, or an unavailable reason
|
||||
when the section is malformed, declares other vendors or scopes, or uses a
|
||||
non-canonical string. A missing section or tag is recorded as absent.
|
||||
|
||||
Calling-convention lanes admit an image only when it declares an RV32I base
|
||||
and only extensions that Ghidra 12.1.4's generic RV32 language implements with
|
||||
p-code: `m`, `a`, `f`, `d`, `c`, `zicsr`, `zifencei`, and the subset names
|
||||
toolchains record, `zmmul`, `zaamo`, `zalrsc`, `zca`, `zcf` and `zcd`. Any
|
||||
other extension is refused. In particular:
|
||||
|
||||
- Ghidra 12.1.4 decodes no Zcb, Zcmp or Zcmt instruction.
|
||||
- Its bit-manipulation, cryptography and packed-SIMD files follow draft
|
||||
encodings and mark nearly every instruction unimplemented.
|
||||
- Vendor extensions, such as Espressif's `xesploop`, `xespv` and `xespdsp`,
|
||||
decode only as custom-opcode placeholders.
|
||||
|
||||
ESP-IDF builds ESP32-C2 and C3 images as `rv32imc_zicsr_zifencei` and ESP32-C5,
|
||||
C6, C61, H2 and H21 images as `rv32imac_zicsr_zifencei_zaamo_zalrsc`, all ilp32.
|
||||
Those base strings are admitted. Builds that add `_zba_zbb_zbs`,
|
||||
`_zcb_zcmp_zcmt` or an `xesp` extension are refused.
|
||||
|
||||
## Evidence
|
||||
|
||||
The lane identity is `elf32-exec-le-double-float-generic-v1` or
|
||||
`elf64-exec-le-double-float-generic-v1`, and `riscv_elf` retains `elf_class`,
|
||||
`byte_order`, `type`, and the raw `flags` alongside the read-only booleans
|
||||
`float_abi_double`, `rvc_declared`, `rve_declared`, and `tso_declared`. Those
|
||||
fields report declarations the header makes; none of them is evidence of the
|
||||
original compiler, of instruction conformance, or of a floating-point calling
|
||||
convention beyond the ABI the pinned language models.
|
||||
The lane identity is one of the lane ids above, and `riscv_elf` retains
|
||||
`elf_class`, `byte_order`, `type`, and the raw `flags` alongside the read-only
|
||||
booleans `float_abi_double`, `rvc_declared`, `rve_declared`, and
|
||||
`tso_declared`. Soft-float profiles also retain `float_abi: soft`, the declared
|
||||
`arch` string, and `riscv_calling_convention: ilp32-prototype-v1`. Those fields
|
||||
report declarations the header and attributes make, and the policy REA applied;
|
||||
none of them is evidence of the original compiler or of instruction
|
||||
conformance. Session Evidence states the prototype and how many functions and
|
||||
definitions use it.
|
||||
|
||||
## Verification and limits
|
||||
|
||||
@@ -76,9 +268,9 @@ tree ships no host decompiler binary, so `Decompiler/os/<platform>/decompile` an
|
||||
[installation](installation.md#ghidra).
|
||||
|
||||
Run `npm run verify:ghidra:riscv` with an existing Ghidra 12.1.4 installation,
|
||||
JDK 21+, and a RISC-V-capable Clang, LLD and LLVM readelf. The verifier acquires
|
||||
no tools and never executes target code. Select installed tools explicitly when
|
||||
they are not on `PATH`:
|
||||
JDK 21+, and a RISC-V-capable Clang, LLD, LLVM readelf and LLVM objdump. The
|
||||
verifier acquires no tools and never executes target code. Select installed
|
||||
tools explicitly when they are not on `PATH`:
|
||||
|
||||
```sh
|
||||
GHIDRA_INSTALL_DIR=/path/to/ghidra_12.1.4_PUBLIC \
|
||||
@@ -86,20 +278,52 @@ JAVA_HOME=/path/to/jdk-21 \
|
||||
REA_RISCV_CLANG=/path/to/llvm/bin/clang \
|
||||
REA_RISCV_LLD=/path/to/lld/bin/ld.lld \
|
||||
REA_RISCV_READELF=/path/to/llvm/bin/llvm-readelf \
|
||||
REA_RISCV_OBJDUMP=/path/to/llvm/bin/llvm-objdump \
|
||||
npm run verify:ghidra:riscv
|
||||
```
|
||||
|
||||
The lane builds the committed freestanding C and assembly sources into RV32 and
|
||||
RV64 `ET_EXEC` fixtures. Independent LLVM readelf observations establish ELF
|
||||
declarations and symbol addresses. Public CLI and stdio MCP workflows verify
|
||||
function enumeration, decompilation, pinned language/compiler identity, decoded
|
||||
probe bytes, global data, and direct calls. It validates advertised MCP output
|
||||
schemas and both JSON representations, saves and reopens snapshots, rejects
|
||||
header-mutated soft-float and unsupported high-bit declarations, and checks
|
||||
unchanged source bytes and owned cleanup. Failures retain the diagnostic
|
||||
workspace; successful runs remove it and report the observed process lineage.
|
||||
The lane builds four `ET_EXEC` fixtures from committed freestanding C and
|
||||
assembly sources:
|
||||
|
||||
Module tests cover admission and malformed headers. Call-graph completeness,
|
||||
string inventories, dynamic linkage, Andes or RVE/TSO images, and big-endian
|
||||
targets are **not** established by this lane. `.riscv.attributes` ISA extensions
|
||||
are not inspected; the generic language pin remains an analysis convention.
|
||||
- RV32 and RV64 double-float builds (`rv{32,64}imafdc`) of `riscv.c`.
|
||||
- An RV32 soft-float build of `riscv-float-abi.c` with ESP32-C3's
|
||||
`-march=rv32imc_zicsr_zifencei -mabi=ilp32`, typed with DWARF.
|
||||
- The same objects linked with `--strip-all`. Every allocated section keeps
|
||||
its address and bytes, so the typed build's symbols locate its functions.
|
||||
|
||||
Independent LLVM readelf observations establish ELF declarations,
|
||||
`Tag_RISCV_arch` and symbol addresses. Public CLI and stdio MCP workflows verify
|
||||
function enumeration, decompilation, pinned language/compiler identity, the
|
||||
committed lane and policy, decoded probe bytes, global data, and direct calls.
|
||||
They validate advertised MCP output schemas and both JSON representations, save
|
||||
and reopen snapshots, and check unchanged source bytes and owned cleanup.
|
||||
|
||||
For the soft-float builds the lane also checks:
|
||||
|
||||
- `fence.i` and `csrr` decode, as ESP32-C3's Zifencei and Zicsr declare.
|
||||
- An LLVM objdump reading of `rea_riscv_abi_caller` finds each call-site word
|
||||
where the [ilp32 policy](#ilp32) puts it: an unaligned `double` in `a1`/`a2`,
|
||||
the `a7`/stack split, two floats in an integer pair, a large structure and a
|
||||
`long double` passed by reference (the pointed-to copies hold the source
|
||||
values), aligned variadic pairs, and an indirect call.
|
||||
- `inspect_native_api` reports `__rea_ilp32` with exactly that storage for the
|
||||
typed build. The stripped build has one integer word per parameter location
|
||||
and returns in the typed build's registers, or in none when the result goes
|
||||
through caller memory.
|
||||
- The caller's pseudocode passes the source constants through that storage and
|
||||
names no floating-point register.
|
||||
- Retained Evidence states the prototype outcome; the stripped build has no
|
||||
function definitions.
|
||||
|
||||
Header-mutated copies check admission. The RV32 double-float image relabelled as
|
||||
soft-float (`e_flags` `0x1`), refused before this lane, is admitted through the
|
||||
soft-float lane over CLI and MCP. The soft-float image relabelled as
|
||||
single-float, RV64 soft-float, and unsupported high bits are refused with their
|
||||
reasons. Failures retain the diagnostic workspace; successful runs remove it and
|
||||
report the observed process lineage.
|
||||
|
||||
Module tests cover admission, malformed headers and attribute sections, the
|
||||
handshake, and the conformance checks against recorded LLVM and Ghidra output.
|
||||
Call-graph completeness, string inventories, dynamic linkage, Andes or RVE/TSO
|
||||
images, big-endian targets, ilp32f images, and decoding beyond the probes are
|
||||
**not** established by this lane.
|
||||
+14
-14
@@ -47,20 +47,20 @@ For runtime instrumentation, see the [Frida guide](./frida-instrumentation.md).
|
||||
|
||||
## Choose a guide
|
||||
|
||||
| Task | Guides |
|
||||
| --------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| Identify a desktop installation | [Runtime candidates, evidence and follow-up files](desktop-applications.md) |
|
||||
| Inspect native functions, references, or types | [Native binaries](native-investigation.md), [Go metadata](go-binaries.md), [Rust dependencies](rust-binaries.md), [CPython pyc files](python-pyc.md), [PE certificates](pe-certificates.md), [PE imports](pe-imports.md), [PE TLS/load config](pe-load-config.md), [ELF notes](elf-notes.md), [Linux kernel modules](kernel-modules.md), [Apple IPS](apple-ips.md), [Erlang BEAM](erlang-beam.md), [offline ELF layout](binary-diagnostics.md), [recorded crashes](recorded-crashes.md) |
|
||||
| Inspect native functions, references, or types | [Native binaries](native-investigation.md), [Go metadata](go-binaries.md), [Rust dependencies](rust-binaries.md), [CPython pyc files](python-pyc.md), [PE certificates](pe-certificates.md), [PE imports](pe-imports.md), [PE TLS/load config](pe-load-config.md), [ELF notes](elf-notes.md), [Apple IPS](apple-ips.md), [Erlang BEAM](erlang-beam.md), [offline ELF layout](binary-diagnostics.md), [recorded crashes](recorded-crashes.md) |
|
||||
| Select a native provider or import format | [Ghidra semantics](ghidra-provider.md), [Windows](windows-ghidra-p0.md), [DOS](ghidra-dos.md), [raw bytes](raw-binary.md), [GameCube DOL](ghidra-dol.md), [MIPS](ghidra-mips.md), [PSP](ghidra-psp.md), [NativeAOT](ghidra-nativeaot.md), [IDA](ida-provider.md), [Kuna](kuna-provider.md), [JEB](jeb-analysis.md) |
|
||||
| Analyze JavaScript or Electron artifacts | [Static analysis](javascript-artifact-reconstruction.md), [tracing and comparisons](javascript-application-workflows.md), [application graph](javascript-application-graph.md), [source recovery](javascript-recovery.md) |
|
||||
| Compare static JavaScript with runtime observations | [Runtime attribution](javascript-runtime-observation.md), [reconciliation](javascript-runtime-reconciliation.md), [Electron observation](electron-observation.md), [Electron fuses](electron-fuses.md) |
|
||||
| Inspect assemblies, packages, or firmware | [Godot PCK](godot-pck.md), [UnityFS](unityfs.md), [U-Boot uImage](uimage.md), [.NET](managed-code-analysis.md), [Python packages](python-packaged-applications.md), [Android](android-analysis.md), [APK signatures](apk-signature.md), [Apktool resources](apktool-resource-analysis.md), [ADB acquisition](adb-device-analysis.md), [Apple](apple-application-analysis.md), [Harmony](harmony-analysis.md), [firmware](firmware-analysis.md), [ESP images](esp-image.md), [WASM](wasm-artifacts.md), [EVM](evm-bytecode.md) |
|
||||
| Observe or compare a browser scenario | [CDP observation](browser-observation.md), [scenarios](browser-scenario-contract.md), [runtime evidence](web-runtime.md), [historical captures](web-network-captures.md) |
|
||||
| Serve recorded behavior as an agent environment | [Agent environments](agent-environments.md) |
|
||||
| Trace captured website source | [Source locations](web-source-location.md), [module traces](website-module-trace.md), [script export](website-script-export.md) |
|
||||
| Capture and compare a command | [Process capture](process-capture.md) |
|
||||
| Use CLI/MCP and retain evidence | [CLI](cli.md), [MCP contracts](mcp-contracts.md), [agent prompts](mcp-prompts.md), [obligation ledgers](reconstruction-obligation-ledgers.md) |
|
||||
| Task | Guides |
|
||||
| --------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| Identify a desktop installation | [Runtime candidates, evidence and follow-up files](desktop-applications.md) |
|
||||
| Inspect native functions, references, or types | [Native binaries](native-investigation.md), [Go metadata](go-binaries.md), [Rust dependencies](rust-binaries.md), [CPython pyc files](python-pyc.md), [PE certificates](pe-certificates.md), [PE imports](pe-imports.md), [PE TLS/load config](pe-load-config.md), [ELF notes](elf-notes.md), [Linux kernel modules](kernel-modules.md), [Apple IPS](apple-ips.md), [Erlang BEAM](erlang-beam.md), [offline ELF layout](binary-diagnostics.md), [recorded crashes](recorded-crashes.md) |
|
||||
| Inspect native functions, references, or types | [Native binaries](native-investigation.md), [Go metadata](go-binaries.md), [Rust dependencies](rust-binaries.md), [CPython pyc files](python-pyc.md), [PE certificates](pe-certificates.md), [PE imports](pe-imports.md), [PE TLS/load config](pe-load-config.md), [ELF notes](elf-notes.md), [Apple IPS](apple-ips.md), [Erlang BEAM](erlang-beam.md), [offline ELF layout](binary-diagnostics.md), [recorded crashes](recorded-crashes.md) |
|
||||
| Select a native provider or import format | [Ghidra semantics](ghidra-provider.md), [Windows](windows-ghidra-p0.md), [DOS](ghidra-dos.md), [raw bytes](raw-binary.md), [GameCube DOL](ghidra-dol.md), [MIPS](ghidra-mips.md), [PSP](ghidra-psp.md), [NativeAOT](ghidra-nativeaot.md), [IDA](ida-provider.md), [Kuna](kuna-provider.md), [JEB](jeb-analysis.md) |
|
||||
| Analyze JavaScript or Electron artifacts | [Static analysis](javascript-artifact-reconstruction.md), [tracing and comparisons](javascript-application-workflows.md), [application graph](javascript-application-graph.md), [source recovery](javascript-recovery.md) |
|
||||
| Compare static JavaScript with runtime observations | [Runtime attribution](javascript-runtime-observation.md), [reconciliation](javascript-runtime-reconciliation.md), [Electron observation](electron-observation.md), [Electron fuses](electron-fuses.md) |
|
||||
| Inspect assemblies, packages, or firmware | [Godot PCK](godot-pck.md), [UnityFS](unityfs.md), [U-Boot uImage](uimage.md), [.NET](managed-code-analysis.md), [Python packages](python-packaged-applications.md), [Android](android-analysis.md), [APK signatures](apk-signature.md), [Apktool resources](apktool-resource-analysis.md), [ADB acquisition](adb-device-analysis.md), [Apple](apple-application-analysis.md), [Harmony](harmony-analysis.md), [firmware](firmware-analysis.md), [ESP images](esp-image.md), [ESP flash images](esp-flash-image.md), [WASM](wasm-artifacts.md), [EVM](evm-bytecode.md) |
|
||||
| Observe or compare a browser scenario | [CDP observation](browser-observation.md), [scenarios](browser-scenario-contract.md), [runtime evidence](web-runtime.md), [historical captures](web-network-captures.md) |
|
||||
| Serve recorded behavior as an agent environment | [Agent environments](agent-environments.md) |
|
||||
| Trace captured website source | [Source locations](web-source-location.md), [module traces](website-module-trace.md), [script export](website-script-export.md) |
|
||||
| Capture and compare a command | [Process capture](process-capture.md) |
|
||||
| Use CLI/MCP and retain evidence | [CLI](cli.md), [MCP contracts](mcp-contracts.md), [agent prompts](mcp-prompts.md), [obligation ledgers](reconstruction-obligation-ledgers.md) |
|
||||
|
||||
For development, use [testing](testing.md), [client verification](client-verification.md),
|
||||
[tool design](tool-design.md), [admission limits and recovery](design/limits-and-recovery.md),
|
||||
|
||||
@@ -0,0 +1,72 @@
|
||||
# Classic iNES container inspection
|
||||
|
||||
REA recognizes classic iNES from the `NES` + `0x1a` byte signature, independently
|
||||
of the file extension. This read-only container inspection uses portable
|
||||
Node.js file reads; it needs neither Ghidra nor an emulator.
|
||||
|
||||
```bash
|
||||
rea inspect-artifact ./cartridge.nes --json
|
||||
rea extract-artifact ./cartridge.nes --json
|
||||
```
|
||||
|
||||
Through MCP, call `open_binary` with the local path, then `inspect_artifact`.
|
||||
Use `extract_artifact` to materialize the regions in the fresh directory named
|
||||
by its result. The caller owns that successful extraction directory.
|
||||
|
||||
## Header and file regions
|
||||
|
||||
Inspection returns `container-header` and `reconstruction` observations inline.
|
||||
The single inventory Evidence under `substeps[].evidence.normalized_result`
|
||||
contains the complete graph and an `ines` object:
|
||||
|
||||
- `ines.header`: original 16-byte `header_hex`, PRG/CHR file-unit counts, mapper,
|
||||
raw flags 6/7, trainer and battery flags, and the declared mirroring mode.
|
||||
- `occurrences[]`: each region's path, `source_location.offset` and
|
||||
`source_location.length` in the original container. Follow `artifact_id` to
|
||||
the node's observed SHA-256 and byte count.
|
||||
- `ines.reconstruction`: ordered region paths, concatenated byte count,
|
||||
SHA-256 and `verified: true`. REA accumulates the concatenation digest during
|
||||
the same physical-order region scan that produces node hashes, then compares
|
||||
it with the original whole-file digest. Final source freshness is also checked.
|
||||
No reconstructed file is written during inspection.
|
||||
|
||||
Regions are `header.bin`, optional `trainer.bin` (512 bytes),
|
||||
`prg/000.bin` onward (16 KiB each), `chr/000.bin` onward (8 KiB each), and
|
||||
optional `trailing.bin`. Every input byte is retained, including unused header
|
||||
fields and uninterpreted trailing data. Zero CHR-ROM units create no CHR region;
|
||||
REA does not fabricate CHR-RAM contents.
|
||||
|
||||
After extraction, concatenate the materialized files in
|
||||
`ines.reconstruction.region_order` from the inspection inventory. Their bytes
|
||||
reproduce the input exactly. Graph path order is lexical and is not physical
|
||||
file order; use the explicit region order or source offsets.
|
||||
|
||||
## Supported boundary
|
||||
|
||||
This increment supports classic iNES with a complete 16-byte header, nonzero
|
||||
PRG count, complete declared trainer/PRG/CHR payload, and zero reserved bytes
|
||||
12–15. Dirty legacy headers are refused because their upper mapper bits are
|
||||
ambiguous; REA does not silently repair them. NES 2.0 and UNIF signatures,
|
||||
unsupported header variants and truncated payloads are explicitly refused.
|
||||
An arbitrary file named `.nes` is not admitted as iNES.
|
||||
|
||||
The PRG/CHR units describe file storage, not mapper bank sizes or CPU windows.
|
||||
The mapper number and flags are header declarations, not proof of cartridge
|
||||
hardware or runtime behavior. Bank switching, 6502/65816 disassembly, emulator
|
||||
sessions and interpretation of console-specific trailing data remain outside
|
||||
this lane. Raw-binary Ghidra analysis is a separate workflow. This container
|
||||
increment addresses the first milestone of [#2072](https://github.com/morluto/rea/issues/2072);
|
||||
the related [#1641](https://github.com/morluto/rea/issues/1641) tracks 6502/65816
|
||||
native analysis with explicit load addresses and provider limitations. Extracted
|
||||
PRG units still need caller-selected CPU windows before static code analysis;
|
||||
their physical file offsets do not establish those addresses.
|
||||
|
||||
Source-owned byte fixtures verify original offsets and hashes, extraction and
|
||||
byte-exact reconstruction, CLI/stdio MCP parity, malformed/unsupported inputs,
|
||||
failed target-switch recovery, cancellation and source identity. These checks
|
||||
do not establish emulator behavior or compatibility with every historical ROM.
|
||||
The portable CLI/stdio MCP acceptance and source-identity checks run in Linux
|
||||
and macOS CI. The same workflows have also been executed on native Windows x64.
|
||||
|
||||
Format references: [iNES](https://www.nesdev.org/wiki/INES) and
|
||||
[NES 2.0](https://www.nesdev.org/wiki/NES_2.0).
|
||||
@@ -622,8 +622,10 @@ or process workflows.
|
||||
## CLI and verification
|
||||
|
||||
All six CLI commands accept inline JSON or a path to a JSON file. The CLI
|
||||
returns an Evidence record directly. Put the full records in a later CLI input;
|
||||
a separate CLI process has no retained MCP connection state.
|
||||
returns an Evidence record directly. Later workflow inputs can contain full
|
||||
records, or exact retained references when `--evidence-bundle PATH` selects an
|
||||
authenticated exported bundle. A separate CLI process has no retained MCP
|
||||
connection state. See [CLI Evidence reuse](cli.md#import-export-and-compare-evidence).
|
||||
|
||||
File inputs must be regular files; symlinks to regular files are accepted.
|
||||
Directories, named pipes and device files produce an input error before JSON
|
||||
@@ -646,8 +648,12 @@ rea trace-application-feature ./trace-input.json --json
|
||||
MCP analysis returns the complete Evidence record, with the operation result
|
||||
in `normalized_result` and its identity in `evidence_id`. If the connected server advertises retained references, reuse its
|
||||
exact returned ID as `{"kind":"retained-evidence","evidence_id":"RETURNED_ID"}`
|
||||
in `application`, or `left`/`right` for comparisons. `RETURNED_ID` is a template,
|
||||
not a literal valid ID. Native Evidence arrays still use complete records.
|
||||
in `application`, or `left`/`right` for comparisons. For summary or inspection
|
||||
replies, use `normalized_result.parent_evidence_id` to select the complete source.
|
||||
`RETURNED_ID` is a template, not a literal valid ID. Native-observation arrays
|
||||
and `reconcile_javascript_runtime`'s nested static layers and runtime-observation
|
||||
array also accept references where advertised, independently mixed with inline
|
||||
records. Exact identity, coverage and partial or unknown facts are preserved.
|
||||
References are scoped to one connection; `close_binary` clears them. Export a
|
||||
bundle before closing and import it on another connection, or supply the full
|
||||
inline Evidence there. Versions before 4.1.0 accept full inline Evidence only; installing
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# Kuna provider
|
||||
|
||||
[Kuna](https://github.com/Noelo-Lab/kuna) is an Apache-2.0 decompiler distributed as one `kuna` executable. REA uses a `kuna` you supply. It does not download, install, update or bundle Kuna. REA verified this provider with Kuna 1.833; it accepts other releases and reports them as unverified in `rea doctor` and in each analysis profile.
|
||||
[Kuna](https://github.com/Noelo-Lab/kuna) is an Apache-2.0 decompiler distributed as one `kuna` executable. For a new installation, use Kuna 1.986, REA's recommended verified release. REA uses a `kuna` you supply; it does not download, install, update or bundle Kuna. REA accepts other releases, which it reports as unverified in `rea doctor` and in each analysis profile. The 1.986 real-provider checks ran on macOS arm64; Linux and other hosts have not been verified with that release.
|
||||
|
||||
Kuna is static: it reads the selected image and never executes it.
|
||||
|
||||
|
||||
+62
-18
@@ -53,8 +53,9 @@ reject the summary record. Give `compare_functions` the retained dossier as
|
||||
`{"kind": "retained-evidence", "evidence_id": "<parent_evidence_id>"}`; give
|
||||
`match_native_functions` the same reference inside `left_observations` or
|
||||
`right_observations`, since its `left` and `right` take `list_procedures`
|
||||
Evidence. Request `detail: "complete"` for `build_call_path`, which accepts
|
||||
only inline complete dossiers.
|
||||
Evidence. `build_call_path.functions` also accepts retained references to the
|
||||
complete dossiers. Request `detail: "complete"` only when the record must travel
|
||||
inline.
|
||||
|
||||
Saved records from the pre-6.6 identity projection that omitted
|
||||
`relationship_coverage: null` from its digest remain incompatible, even when
|
||||
@@ -73,7 +74,9 @@ Ghidra inventories, measured load-image observations and function replies use
|
||||
`raw_result: null` when their useful facts are already preserved in
|
||||
`normalized_result`, limitations, locations and relationship coverage. Distinct
|
||||
overview details, file-offset selection roles and database types supplemented
|
||||
with derived metadata keep their upstream representation. Native load-image
|
||||
with derived metadata keep their upstream representation. When a Ghidra naming
|
||||
edit is not applied, `raw_result` keeps the bridge's `failures` and shared
|
||||
`conflicts` table without a second copy of the result booleans. Native load-image
|
||||
attestation continues to use the bridge reply locally before Evidence
|
||||
projection. Read function results from `normalized_result`; older retained
|
||||
Evidence keeps its original representation and identity.
|
||||
@@ -158,7 +161,9 @@ for the error code, target details, partial observations, remediation, and the
|
||||
optional bounded `recovery` next step. `recovery.kind=inspect_input_contract`
|
||||
names the rejected operation: re-read that operation's advertised input schema
|
||||
(call `tools/list` if the catalog may be stale), correct the listed fields, and
|
||||
do not repeat the unchanged call. `recovery.kind=inspect_timeout_state` names
|
||||
do not repeat the unchanged call. A well-formed request the selected target
|
||||
cannot answer, such as an address without one exact file offset, omits this
|
||||
hint; follow its `remediation.action`. `recovery.kind=inspect_timeout_state` names
|
||||
the operation and the REA server/provider deadline. Inspect partial Evidence
|
||||
and reported cleanup state, then follow the returned remediation. Change a
|
||||
request field or REA_* setting only when the returned remediation explicitly
|
||||
@@ -623,10 +628,24 @@ counts, and whether the decoded graph is incomplete. Each page repeats
|
||||
budget (XLD.app: 6.7 MB), so select `detail: "summary"` on the first call.
|
||||
Delivery detail never reaches the decoder or changes the source Evidence ID.
|
||||
|
||||
`trace_application_feature`, `trace_javascript_semantics`,
|
||||
`compare_application_versions`, `compare_source_to_bundle`, and
|
||||
`compare_javascript_export_shapes` accept complete inline application Evidence
|
||||
or an exact reference to a record already retained by the current connection:
|
||||
Logical Evidence source fields share one input contract across domains:
|
||||
complete inline Evidence or an exact reference to a record already retained by
|
||||
the current connection. This applies to individual sources, optional sources,
|
||||
arrays and nested sources declared by the tool's input schema:
|
||||
|
||||
| Workflows | Retained-reference inputs |
|
||||
| ----------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------- |
|
||||
| `compare_functions`, `compare_process_captures`, `compare_wasm_interfaces` | `left`, `right` |
|
||||
| `match_native_functions` | `left`, `right`, `left_observations[]`, `right_observations[]` |
|
||||
| `trace_application_feature`, `trace_javascript_semantics`, `compare_source_to_bundle` | `application`; feature tracing also accepts `native_observations[]` |
|
||||
| `compare_application_versions`, `compare_javascript_export_shapes` | `left`, `right`; version comparison also accepts both native-observation arrays |
|
||||
| `reconcile_javascript_runtime` | `static_layers[].analysis`, `runtime_observations[]` |
|
||||
| `compare_managed_members`, `verify_managed_native_boundaries`, `import_managed_reconstruction` | Comparison `left`/`right`; verification `managed_boundaries`/`native_observations[]`; reconstruction `static_members` |
|
||||
| `project_managed_application_graph` | `managed_artifact`, `managed_members`, `managed_native_boundaries` |
|
||||
| `project_android_application_graph`, `project_apple_application_graph`, `project_harmony_application_graph` | `inventory_evidence[]` |
|
||||
| `build_call_path`, `find_changed_behavior`, `correlate_static_and_runtime` | `functions[]`, `comparisons[]`, or `static_comparisons[]`/`runtime_comparisons[]`, respectively |
|
||||
|
||||
For example:
|
||||
|
||||
```json
|
||||
{
|
||||
@@ -642,11 +661,16 @@ or an exact reference to a record already retained by the current connection:
|
||||
```
|
||||
|
||||
Use the `evidence_id` returned by `analyze_javascript_application` (or another
|
||||
compatible application-graph producer). Comparisons accept this form in `left`
|
||||
and `right`; each side can independently be inline or retained. Native
|
||||
observation arrays continue to take complete inline Evidence. Results and their
|
||||
Evidence remain complete inline, and both input forms pass the same semantic,
|
||||
identity, authority, and provenance checks without running the producer again.
|
||||
compatible application-graph producer). When the producer returned a summary
|
||||
or inspection page, use its `normalized_result.parent_evidence_id` to select
|
||||
the complete source, rather than the view's own `evidence_id`. Each source can
|
||||
independently be inline or retained, including within one array. Both forms use
|
||||
the same application workflow and owner validation, preserve exact source
|
||||
identity, provenance, authority, coverage and partial or unknown facts, and do
|
||||
not run the producer again. A reference cannot supply facts absent from that
|
||||
record. Workflows that follow source links also require those linked records
|
||||
in the ledger. File references are accepted only where the tool explicitly
|
||||
advertises a file selector; a retained ID does not authorize a file read.
|
||||
|
||||
References belong to the current connection's Evidence ledger. Opening another
|
||||
target or restarting the current target preserves retained records, including
|
||||
@@ -659,10 +683,13 @@ Supply complete inline Evidence, repeat its producer, or import an exported
|
||||
Evidence bundle before referencing that imported record. Export a bundle before
|
||||
closing if the investigation needs it later.
|
||||
|
||||
CLI application workflows continue to read portable inline Evidence from files
|
||||
and run the same analysis workflows; a standalone CLI invocation cannot resolve
|
||||
another MCP connection's retained records. No additional lookup call, provider
|
||||
selection, or approval step is required for a same-session follow-up.
|
||||
CLI workflows that advertise `--evidence-bundle` can resolve the same reference
|
||||
JSON from one explicitly selected local bundle, including an MCP export. They
|
||||
also accept portable inline Evidence and use the same application workflows.
|
||||
A standalone CLI invocation cannot resolve another MCP connection's ledger
|
||||
without that exported bundle. See [CLI Evidence reuse](cli.md#import-export-and-compare-evidence)
|
||||
for the supported commands. No additional lookup call, provider selection, or
|
||||
approval step is required for a same-session follow-up.
|
||||
CLI `inspect-artifact` continues to return complete portable Evidence. Save that
|
||||
JSON to a file and use `inspect-analysis-view` with inline source Evidence for
|
||||
later inventory summaries or pages; a one-shot CLI session cannot retain a
|
||||
@@ -1094,6 +1121,15 @@ mapping. A successful start-address mapping does not establish that an entire
|
||||
range is contiguous in the original file, or that its bytes equal analysis
|
||||
memory after loader fixups. Uninitialized, missing, and ambiguous mappings fail
|
||||
explicitly; preserve the reported failure rather than guessing an offset.
|
||||
For Ghidra, such a well-formed address returns `invalid_request` whose message
|
||||
states the bridge's reason and whose remediation points to
|
||||
`inspect_native_load_image`. It carries no `inspect_input_contract` recovery.
|
||||
`details.issues[0].observed` keeps the caller's exact `address` and a
|
||||
`file_mapping` outcome: `unmapped` (outside every memory block),
|
||||
`uninitialized`, `absent` (no authoritative target-file source), `ambiguous`,
|
||||
or `unrepresentable` (beyond the exact JSON integer range). Initialization, not
|
||||
the loaded flag, decides whether source bytes can exist, so an initialized
|
||||
file-backed overlay still maps.
|
||||
|
||||
For constructor-populated pointer or resource slots, a static zero is not proof
|
||||
of a later null value. Inspect the slot's `xrefs` and the writer's assembly or
|
||||
@@ -1317,12 +1353,20 @@ bytes and returns the `integrity_contradictions` records for the extracted
|
||||
files. An active ASAR unpacked entry whose companion file is absent cannot be
|
||||
materialized; extraction fails as `unavailable` with that logical path.
|
||||
|
||||
A container can also declare a checksum over its own bytes; today that is the
|
||||
DFU suffix CRC of a DfuSe file. A mismatch is a contradiction of the root
|
||||
occurrence (logical path `.`), recorded as a `dfu-crc32` pair. Under
|
||||
`record-and-continue` the root becomes `mismatched` while its elements stay
|
||||
listed and extractable as observed-untrusted bytes, and `extract_artifact`
|
||||
returns the root record with the extracted files' records.
|
||||
|
||||
If native DMG child traversal is unavailable after carrier identification,
|
||||
the typed failure retains a revalidated root-only inventory under
|
||||
`details.partial_observation.inventory`, with unknown child coverage and the
|
||||
producer's diagnostics/provenance. It does not borrow or detach an existing
|
||||
attachment. Native checksum verification failures still refuse traversal;
|
||||
`record-and-continue` recovery applies to structured entry contradictions.
|
||||
`record-and-continue` recovery applies to structured entry and container
|
||||
contradictions.
|
||||
|
||||
Occurrences use normalized logical paths for identity and lookup. Optional
|
||||
`source_path` retains the original path within the owning reader when Unicode
|
||||
|
||||
@@ -185,6 +185,13 @@ observed entry label. An interior address therefore remains visible without
|
||||
being mistaken for the function entry. Identity failures are reported as
|
||||
`procedure.status: "unknown"`; label failures leave a null name and `name_error`.
|
||||
Neither failure discards usable pseudocode, and names are not parsed from it.
|
||||
A provider answer without pseudocode is an error item that keeps that
|
||||
observation's own `limitations`, so a reason the provider reported for that
|
||||
entry, such as a Kuna export failure or a missing body, stays with the item.
|
||||
Null pseudocode is `capability_unavailable`; empty or non-text pseudocode is
|
||||
`unreadable_output`. Provider failures keep their typed error. Successful items
|
||||
do not repeat limitations; the batch Evidence aggregates every observation's
|
||||
limitations by provider and operation.
|
||||
Resolved entries also appear in the composed Evidence locations. New batch
|
||||
Evidence commits the revised workflow profile; retained historical Evidence
|
||||
remains valid. This tool
|
||||
@@ -371,7 +378,9 @@ bounded typed call references. Direct references, resolved indirect references,
|
||||
ambiguous candidates, inferred untyped provider callees and targetless call
|
||||
sites remain distinct. A static route does not establish runtime reachability.
|
||||
Swift/closure dispatch requires resolved static references; unavailable ABI
|
||||
forms remain unknown.
|
||||
forms remain unknown. A handler node's `location` pairs the implementation
|
||||
address with its `implementation_file_offset`, or an unknown offset; the method
|
||||
record's own location remains in the node's Evidence.
|
||||
|
||||
Incomplete reference kinds do not discard known typed calls or partial
|
||||
provider call references. A `provider_call` with `observed` resolution retains
|
||||
@@ -422,6 +431,21 @@ establish canonical function identity. A successful seed without a native def-us
|
||||
model still returns its procedure and an explicit coverage unknown. A new trace
|
||||
can retry failed selectors.
|
||||
|
||||
Paging and traversal completeness are separate facts. `page_truncated` means the
|
||||
page omits retained nodes, and the edges they source, before `offset` or after
|
||||
`next_offset`. Every page repeats `traversal`: `complete`, or `truncated` with
|
||||
`reasons` naming each observed cause: the `max_depth`, `max_functions`,
|
||||
`max_call_sites`, `max_nodes` or `max_edges` budget, the fixed
|
||||
`node_payload_bytes` allowance, `provider_pcode_truncated`, or
|
||||
`ambiguous_call_candidates`. Each cause also has a located unknown. A budget is
|
||||
reported only after it rejects work, so an exactly filled budget is not
|
||||
truncation; a depth or candidate limit is reported only when no other path
|
||||
analyzes or queues that destination. `total_nodes` and `total_edges` count
|
||||
retained rows. `truncated` remains true for either kind of omission. Failed
|
||||
analyses and unresolved calls stay in `unknowns` and do not truncate the
|
||||
traversal. Each page reruns analysis; pages are not pinned to one snapshot
|
||||
across session edits.
|
||||
|
||||
The raw Java `is_dead` flag can remain set after Ghidra decodes a live block;
|
||||
`block_membership` reports actual syntax-tree membership separately.
|
||||
|
||||
@@ -813,6 +837,11 @@ complete observation.
|
||||
non-generic, non-resilient class vtable descriptors. Records include exact virtual addresses/file offsets and
|
||||
artifact evidence. Vtable indexes are metadata word offsets; witness indexes
|
||||
start after the conformance header. Names unavailable in metadata remain null.
|
||||
An Objective-C implementation's `location` is its method-list entry; the
|
||||
reader maps `implementation_address` separately into
|
||||
`implementation_file_offset`, which uses the same file coordinates as
|
||||
`location.file_offset`, including a universal slice's container offset.
|
||||
Unmapped implementations and symbol-only records keep it null.
|
||||
The selected Mach-O header must match the requested CPU family for both thin
|
||||
files and universal slices before metadata decoding. CPU subtypes and
|
||||
capability bits remain source facts; an `arm64` request may select an
|
||||
@@ -1117,11 +1146,55 @@ unpaired Unicode surrogate; a rejection identifies the field and UTF-16 index
|
||||
and leaves every annotation unchanged. CRLF, supplementary Unicode characters,
|
||||
and combining characters are preserved. Windows P0 remains read-only. Ghidra has no
|
||||
GUI authority, and REA never falls back automatically to Hopper.
|
||||
For a before-and-after `compare_functions` in MCP, call `analyze_function`
|
||||
before and after the edit while the session remains open; the annotation Evidence
|
||||
itself is not an `analyze_function` input. If comparing after close, export the
|
||||
needed Evidence before closing and import it into the later session. A one-shot
|
||||
CLI annotation returns its edited dossier before its temporary session is
|
||||
discarded; a later CLI invocation with the same fresh-import settings starts from
|
||||
the original executable. Comparison preserves correspondence across renames
|
||||
when authenticated artifact identity, architecture and provider-owned mapping
|
||||
and slice commitments establish the same canonical entry. The result reports
|
||||
`matched`, `method: "same-entry"` and `entry_address`; name and exact-text changes
|
||||
remain separate dimensions. Equal numeric addresses without applicable coordinate
|
||||
commitments do not establish same-entry correspondence. Inspect the retained
|
||||
source Evidence and coverage when correspondence remains ambiguous.
|
||||
To aggregate already-retained comparison results, pass their exact IDs to
|
||||
`find_changed_behavior`, for example:
|
||||
|
||||
```javascript
|
||||
await call("find_changed_behavior", {
|
||||
comparisons: comparisons.map(({ evidence_id }) => ({
|
||||
kind: "retained-evidence",
|
||||
evidence_id,
|
||||
})),
|
||||
});
|
||||
```
|
||||
|
||||
These IDs select comparison Evidence, not function dossiers or summary views.
|
||||
Complete inline comparisons remain accepted and can be mixed with references.
|
||||
The current server must retain the comparisons and their original source closure;
|
||||
import the complete source bundle after a session handoff. Missing or damaged
|
||||
records are explicit failures. A missing required retained record reports its exact
|
||||
`evidence_id` with `details.reference_scope: "retained-closure"`. Import the
|
||||
originating bundle with its linked sources, or re-create those observations in the
|
||||
current investigation, before retrying; resending the inline comparison alone
|
||||
does not restore its closure. Aggregating static differences does not establish
|
||||
observed runtime change.
|
||||
|
||||
`set_address_name` and `set_addresses_names` give Ghidra sessions the same
|
||||
compact naming results as Hopper: a local function entry renames that function,
|
||||
and any other mapped address, such as a global, receives a new or renamed primary
|
||||
label. A batch applies each address separately and reports one success boolean
|
||||
per address. Use them to name many globals and helpers cheaply; use
|
||||
label. A batch decides every address from the names held before it: addresses
|
||||
that request the same name in one namespace, or a name another selected address
|
||||
holds, are rejected without applying them, so JSON member order never selects a
|
||||
winner. Every other address applies separately, and each address reports one
|
||||
success boolean. `raw_result.failures` gives each failed address's canonical
|
||||
address and reason. A contested name appears once in `raw_result.conflicts`,
|
||||
with its namespace, requesting addresses and holding addresses, and conflict
|
||||
failures refer to it by index. To move a label between
|
||||
addresses, rename its holder first and assign the name in a later call. Use
|
||||
them to name many globals and helpers cheaply; use
|
||||
`annotate_native_function` when you also want comments and a refreshed dossier.
|
||||
Ghidra name edits preserve the existing namespace. Supply either a leaf name
|
||||
such as `renamed` or a fully qualified name in that namespace, such as
|
||||
|
||||
+55
-3
@@ -382,6 +382,38 @@ matching real-provider lane.
|
||||
| Client configuration or schema conversion | [Native client lane](client-verification.md) | Distinguish loopback models from live services |
|
||||
| Agent investigation quality | [Agent evaluation](#agent-evaluation-and-conformance-records) | Review transcript and scenario-specific evidence |
|
||||
|
||||
### Provider-backed workflow verification
|
||||
|
||||
For a new or changed provider-backed tool, select tests from the analyst's
|
||||
question and the facts the contract claims, not from an upstream command list.
|
||||
Use the existing workflow as a baseline and identify what the proposed tool
|
||||
adds. Choose the relevant cases below; do not require every provider or a
|
||||
Cartesian matrix when it would not change the claim.
|
||||
|
||||
When auditing existing tools, compare selected analyst workflows with the cases
|
||||
their existing tests and real-provider lanes actually exercise. Run relevant
|
||||
public workflows and focused checks; report untested scenarios rather than
|
||||
inferring that a green test proves them or that no prior testing occurred. A fix
|
||||
should add a regression for the demonstrated failure at its owning boundary and
|
||||
affected consumer.
|
||||
|
||||
- Use a controlled target with independently known facts and a meaningful
|
||||
counterexample. Compare native provider output with the target or a
|
||||
format-aware oracle; a provider label, a mock, or another REA projection is
|
||||
not independent proof of the underlying fact.
|
||||
- Exercise the affected public CLI/MCP workflow, including handoffs between
|
||||
calls. Assert the requested result, caller-selected inputs, source identity,
|
||||
evidence and coverage. Check that unavailable, partial and empty outcomes
|
||||
retain their different meanings across the producing and consuming boundaries.
|
||||
- Where relevant, continue after a mixed success, failed request, cancellation
|
||||
or close/reopen. Verify the next supported recovery and preservation of prior
|
||||
results and resource ownership. Use the applicable lifecycle regressions below.
|
||||
- Record the provider version, host, fixture, actual entrypoint, controlled
|
||||
faults and skipped or unavailable coverage. A simulated provider tests the
|
||||
integration boundary; a scripted sequence tests that sequence, not an agent's
|
||||
ability to discover it. Run the matching real-provider and agent lanes when
|
||||
their respective claims require them.
|
||||
|
||||
## Test design and pruning
|
||||
|
||||
Prefer real public CLI/MCP workflows, then production-boundary integration,
|
||||
@@ -1029,7 +1061,7 @@ prove behavior on the verified Ghidra 12.1.4 and JDK 21 build.
|
||||
| `npm run verify:ghidra` | Linux x64/arm64 ELF or macOS x64/arm64 Mach-O | Host C compiler, Ghidra 12.1.4, and full JDK 21 |
|
||||
| `npm run verify:ghidra:seeds` | POSIX; raw x86 fixture; seed ordering, BSS labels, analyzer readback, not-applicable and unknown analyzer names | Ghidra 12.1.4 and full JDK 21 |
|
||||
| `npm run verify:ghidra:swift` | macOS x64/arm64 Mach-O | Host Swift compiler, Ghidra 12.1.4, and full JDK 21 |
|
||||
| `npm run verify:ghidra:riscv` | Linux/macOS x64/arm64 host; source-owned RV32/RV64 ELF through CLI and MCP | RISC-V Clang, LLD, LLVM readelf, Ghidra 12.1.4, and full JDK 21 |
|
||||
| `npm run verify:ghidra:riscv` | Linux/macOS x64/arm64 host; source-owned RV32/RV64 double-float and RV32 ilp32 ELF through CLI and MCP | RISC-V Clang, LLD, LLVM readelf and objdump, Ghidra 12.1.4, and full JDK 21 |
|
||||
| `npm run verify:ghidra:switch` | Linux x64 ELF; GCC/Clang optimized and stripped switch fixtures | GCC, Clang, GNU nm/objdump/strip, Ghidra 12.1.4, and full JDK 21 |
|
||||
| `npm run verify:ghidra:aarch64-jump-table` | Any supported Ghidra host; AArch64 ELF; signed/unsigned byte/halfword tables; host ARM64 Mach-O | Clang with AArch64 target support, Ghidra 12.1.4, and full JDK 21 |
|
||||
| `npm run verify:ghidra:cross-format` | Any supported Ghidra host; also analyzes AArch64 ELF, x86-64 PE, and x86-64 Mach-O | `clang`, LLD, and `lld-link` in addition to host-lane prerequisites |
|
||||
@@ -1438,6 +1470,11 @@ cached byte spans and a Ghidra comparison. See the [Linux report](https://github
|
||||
for measured scope and limitations. Windows, PE and universal Mach-O remain
|
||||
unverified by this lane.
|
||||
|
||||
Kuna 1.986 passed the fixture and a separate small, source-owned Mach-O with
|
||||
symbols and stripped on macOS arm64, including the Ghidra comparison and owned
|
||||
cleanup. Linux and other hosts have not been verified with 1.986; this check is
|
||||
not a repeat of the larger 1.833 matrix.
|
||||
|
||||
## DOS Ghidra analysis
|
||||
|
||||
`npm run verify:ghidra:dos` requires the supported Ghidra and JDK installation
|
||||
@@ -1508,8 +1545,9 @@ real-provider coverage.
|
||||
For source-identity boundaries that a normal single-file import cannot create,
|
||||
`tests/conformance/ghidra/ReaFileOffsetSourceProbe.java` exercises the production
|
||||
bridge against real Ghidra `FileBytes`. It rejects auxiliary and partial sources,
|
||||
checks the original target's round trip, and retains ambiguity when duplicate
|
||||
target bytes have distinct loaded mappings. Injected decompiler failures and
|
||||
keeps distinct outcome codes for uninitialized and unmapped addresses, checks
|
||||
the original target's round trip, and retains ambiguity when duplicate target
|
||||
bytes have distinct loaded mappings. Injected decompiler failures and
|
||||
cancellation verify that completed source/function resolution facts and backend
|
||||
failure codes survive; these injections do not establish natural engine failures.
|
||||
On Windows, after creating the
|
||||
@@ -2001,6 +2039,20 @@ tool before building and warns when esptool is not the pinned v5.6.0.
|
||||
`REA_XTENSA_ESP32_GCC` adds an ESP32 Xtensa build. See
|
||||
[ESP images](esp-image.md).
|
||||
|
||||
`npm run verify:uf2` needs only a Python that runs esptool, named by
|
||||
`REA_ESPTOOL_PYTHON` (default `python3`). It writes inputs at known addresses,
|
||||
builds UF2 files with esptool's `merge-bin --format uf2` (v4 `merge_bin`) with
|
||||
and without MD5 and with 256-byte payloads, and concatenates two chips'
|
||||
outputs. Inspection and extraction through the built CLI must return each input
|
||||
at its address under the familyID esptool itself reports. It is not part of CI.
|
||||
See [UF2 and DfuSe images](uf2-dfuse.md).
|
||||
`npm run verify:esp-flash` needs the same tools and an ESP-IDF checkout
|
||||
(`REA_ESP_IDF_PATH`) for `gen_esp32part.py` and `otatool.py`. It builds
|
||||
partition tables, assembles dumps with `esptool merge-bin`, writes OTA records
|
||||
with otatool through `tests/conformance/esp/flash_file_tool.py`, and compares
|
||||
REA's flash facts with `image-info`, the decoded table and
|
||||
`otatool read_otadata`. See [ESP flash images](esp-flash-image.md).
|
||||
|
||||
### JavaScript source recovery
|
||||
|
||||
Build once with `npm run build:cached`, then run `npm run verify:javascript:recovery`.
|
||||
|
||||
@@ -48,6 +48,57 @@ fixture servers. Custom orchestration languages, replay engines, or separate
|
||||
prepare/execute plans need an observed requirement those primitives cannot
|
||||
satisfy. A plan without execution does not establish runtime behavior.
|
||||
|
||||
## Exercise the analyst workflow
|
||||
|
||||
Before defining a new provider-backed tool, state the analyst's question, what
|
||||
they already have, the intended result and authority, and what must remain
|
||||
unknown. Try the nearest existing REA workflow and the relevant native provider
|
||||
or ordinary script path when available. Identify the specific gap and what a
|
||||
new or extended contract would add; upstream capability alone does not require
|
||||
a new tool or equal support from every provider.
|
||||
|
||||
Exercise the task through REA as a whole and compare relevant provider behavior
|
||||
with an independent artifact or format observation where possible. Use the
|
||||
caller's selected target and authority. Record the provider version, target and
|
||||
observed coverage, and distinguish an unsupported operation from an empty
|
||||
result or an untested path.
|
||||
|
||||
Include the boundary cases that could change the answer: provider labels versus
|
||||
source facts, addresses versus file locations, partial results, and a useful
|
||||
recovery after failure or session close when those apply to the task. Check that
|
||||
composed results keep each observation's identity, limitations and source
|
||||
association. Existing real-provider lanes and focused tests establish their
|
||||
selected scenarios; they do not by themselves establish a different workflow or
|
||||
provider equivalence. If a provider cannot be exercised, report that limitation
|
||||
and continue with the available evidence. Measure agent tool selection only with
|
||||
an actual agent evaluation; a scripted tool sequence does not establish it. Use
|
||||
[testing guidance](testing.md#provider-backed-workflow-verification) to select
|
||||
the cases that establish the proposed contract.
|
||||
|
||||
## Audit existing tools
|
||||
|
||||
Scope an audit by analyst tasks and the caller's target, providers and authority.
|
||||
Record the source revision and, when used, the running server, catalog and
|
||||
provider versions; a connected server can differ from the current checkout.
|
||||
Inspect the live CLI help or MCP descriptions and schemas alongside the owning
|
||||
contracts and documentation. Choose representative tasks and their nearest REA
|
||||
and native alternatives rather than counting upstream commands.
|
||||
|
||||
Exercise the selected public workflows, including the steps before and after a
|
||||
tool call. Check task selection, input choices, returned facts and their source,
|
||||
partial or unsupported outcomes, and applicable session and recovery behavior.
|
||||
Compare native provider output and an independent artifact observation where
|
||||
available. Inspect existing tests and real-provider lanes for the scenario they
|
||||
actually cover; a passing test does not establish a different task or host.
|
||||
Use [workflow verification](testing.md#provider-backed-workflow-verification)
|
||||
for counterexamples and lifecycle controls relevant to the audit.
|
||||
|
||||
Report working routes as well as findings. Separate a reproduced contract bug,
|
||||
an intentional limitation, a design opportunity and an unverified hypothesis;
|
||||
include the smallest useful reproduction, impact and coverage limits. Do not
|
||||
claim autonomous-agent selection from a scripted trace. Check related reports
|
||||
before proposing another issue, and avoid treating lack of API parity as a bug.
|
||||
|
||||
## Define the contract
|
||||
|
||||
- Use strict inputs with explicit target identity and required fields. Preserve
|
||||
@@ -83,6 +134,51 @@ satisfy. A plan without execution does not establish runtime behavior.
|
||||
ceiling and its recovery as described in
|
||||
[admission limits and recovery](design/limits-and-recovery.md).
|
||||
|
||||
### Compose declared Evidence inputs
|
||||
|
||||
Declare each logical Evidence source with the shared `evidenceInputSchema`:
|
||||
complete inline Evidence or an exact
|
||||
`{kind: "retained-evidence", evidence_id}` reference. Use that declaration in
|
||||
single, optional, array and nested source fields across domains. Keep complete
|
||||
record schemas for stored Evidence, bundle records and pure semantic inputs;
|
||||
request alternatives do not change those representations. Ordinary IDs in
|
||||
parameters or result data are not implicitly source selectors.
|
||||
|
||||
Derive borrowing request validation and source resolution from the canonical
|
||||
request schema with `getEvidenceInputContract` and `resolveEvidenceRequest`.
|
||||
The shared application entrypoint selects references for CLI and MCP before the
|
||||
format owner admits, authenticates and interprets their records. Preserve
|
||||
admission before copying large source payloads. Do not maintain a second list
|
||||
of source paths or resolve references only in an adapter. A resolved reference
|
||||
has the same identity, caller-selected observations, provenance, coverage and
|
||||
partial or unknown facts as its complete inline record; it never reruns a
|
||||
producer or upgrades incomplete evidence.
|
||||
Keep Evidence-bearing request unions outside skipping `default` wrappers;
|
||||
use `prefault` when a default must pass through branch selection. Catalog
|
||||
conformance rejects unsupported source containers rather than silently losing
|
||||
their references.
|
||||
|
||||
Return authenticated source records alongside derived workflow Evidence when
|
||||
the adapter must retain them. Keep links to those exact sources and require the
|
||||
source closure when the workflow follows their links. Missing references and
|
||||
incompatible workflow roles are explicit failures; do not infer a substitute
|
||||
from ledger order, display names or the active target. A summary or inspection
|
||||
page has its own identity: consumers needing complete source facts use its
|
||||
`normalized_result.parent_evidence_id`, unless their contract explicitly accepts
|
||||
that view.
|
||||
|
||||
New provider-owned source observations require a retention owner. Existing
|
||||
caller-owned inputs can also compose on a fixed port without a recorder;
|
||||
when a recorder is available, preserve its acceptance or rejection and retain
|
||||
completed sources through cancellation.
|
||||
|
||||
MCP supplies the current ledger lookup. CLI commands that advertise
|
||||
`--evidence-bundle` use one explicitly selected, authenticated bundle for the
|
||||
command's lifetime. Keep cleanup with that owner. A retained reference does not
|
||||
grant filesystem access: file selectors remain separate, explicit contracts
|
||||
where supported. See [retained Evidence inputs](mcp-contracts.md#retained-application-evidence-inputs)
|
||||
and [CLI reuse](cli.md#import-export-and-compare-evidence).
|
||||
|
||||
### Interpret boundary facts once
|
||||
|
||||
The format or protocol reader owns interpretation. Pass its parsed facts and
|
||||
|
||||
@@ -0,0 +1,224 @@
|
||||
# UF2 and DfuSe images
|
||||
|
||||
REA inventories UF2 (Microsoft's USB Flashing Format) and DfuSe (ST's DFU
|
||||
update container, `.dfu`) files through `inspect-artifact` /
|
||||
`inspect_artifact`, and materializes their decoded bytes through
|
||||
`extract-artifact` / `extract_artifact`. UF2 is the drag-and-drop format of the
|
||||
RP2040 and RP2350 bootloaders, of TinyUF2 on ESP32-S2 and ESP32-S3, and of
|
||||
`esptool merge-bin --format uf2`. Decoding is static and needs no external tool.
|
||||
|
||||
```sh
|
||||
rea inspect-artifact ./firmware.uf2 --json
|
||||
rea inspect-artifact ./firmware.dfu --detail summary --evidence_output evidence.json --json
|
||||
rea extract-artifact ./firmware.uf2 --json
|
||||
```
|
||||
|
||||
For MCP, open the file with `open_binary`, then call `inspect_artifact`. Summary
|
||||
delivery retains the complete Evidence; use `parent_evidence_id` with
|
||||
`inspect_analysis_view` to read selected regions and fields. `extract_artifact`
|
||||
returns a fresh `output_root` containing one raw file per region.
|
||||
|
||||
## Recognition
|
||||
|
||||
A file is UF2 only when its first 512 bytes are a complete block with all three
|
||||
magic numbers: `0x0A324655` and `0x9E5D5157` at the start and `0x0AB16F30` at
|
||||
the end. A file is DfuSe only when it begins with `DfuSe` and format version 1
|
||||
and ends with a 16-byte DFU 1.1a suffix: `bcdDFU` `0x011A`, the `UFD` signature
|
||||
and `bLength` 16. The suffix CRC is not part of recognition. The file name does
|
||||
not matter, and a file with only one of the DfuSe markers stays unrecognized.
|
||||
The container formats are `uf2` and `dfuse`.
|
||||
|
||||
## UF2 regions and metadata
|
||||
|
||||
Each entry is one address-contiguous region of one family's main-flash
|
||||
payloads, named `family-0x<familyID>/0x<start>-0x<end>.bin` with an inclusive
|
||||
end address, for example `family-0xe48bff56/0x10000000-0x1000ffff.bin`. Blocks
|
||||
without the familyID flag (`0x2000`) go under `no-family/`. Families are
|
||||
separate address spaces: a multi-family file, such as a Pico SDK image for both
|
||||
RP2040 and RP2350, has a region per family even at the same addresses. Gaps are
|
||||
never filled.
|
||||
|
||||
Region `format_metadata` (`uf2-region`) records `family_id` (or `null`),
|
||||
`family_name`, `start_address`, `end_address`, `length`, `block_count`, and
|
||||
`first_file_block`/`last_file_block`: the zero-based positions in the file of
|
||||
the contributing blocks, not their UF2 block numbers. `family_name` is REA's
|
||||
label, not file content. It comes from a small table copied from Microsoft's
|
||||
`uf2families.json` at commit
|
||||
[`cf3bd7f9`](https://github.com/microsoft/uf2/blob/cf3bd7f9c274b949528f8cd6a19eef2502a687d2/utils/uf2families.json),
|
||||
which the root's `family_name_registry` names. The table covers the ESP32
|
||||
family, ESP8266, RP2040, the RP2350 and RP2XXX download families, STM32F4,
|
||||
nRF52840, SAMD21 and SAMD51. It also covers the families that LibreTiny builds
|
||||
for and the registry lists: RTL8710A, RTL8710B, RTL8720C, RTL8720D, BK7231U,
|
||||
BK7231N, BK7251 and BL602. Other IDs, LibreTiny's LN882H among them, keep their
|
||||
value with `family_name: null`. A familyID names a target family; it does not
|
||||
establish the CPU state, memory map or entry point.
|
||||
|
||||
Root `format_metadata` (`uf2`) records:
|
||||
|
||||
| Field | Meaning |
|
||||
| ---------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------- |
|
||||
| `block_count`, `trailing_bytes` | Complete 512-byte blocks, and bytes after the last complete block |
|
||||
| `invalid_block_count` | Blocks with a wrong magic number, a payload over 476 bytes, or a payload that wraps past 4 GiB |
|
||||
| `flash_block_count` | Valid main-flash blocks, the only source of region bytes |
|
||||
| `not_main_flash_block_count`, `file_container_block_count` | Blocks with flag `0x1`, and blocks with flag `0x1000` |
|
||||
| `md5_block_count`, `extension_tag_block_count` | Blocks that declare an MD5 flash region (`0x4000`) or extension tags (`0x8000`) |
|
||||
| `family_count`, `family_name_registry` | Distinct familyIDs among main-flash blocks, and the registry revision of `family_name` |
|
||||
| `numbering_group_count` | Groups of valid blocks that share a familyID (or none) and a declared block count |
|
||||
| `missing_block_count` | Block numbers below a group's block count that no valid block carries |
|
||||
| `ambiguous_numbering_group_count` | Groups whose numbers appear at unequal counts of target addresses |
|
||||
| `repeated_block_count`, `out_of_order_block_count` | Blocks whose number already appeared in their group, and blocks numbered at or below the previous block of their group |
|
||||
| `block_number_out_of_range_count` | Blocks numbered at or above their block count, or with a block count of 0 |
|
||||
| `region_count`, `data_bytes`, `duplicate_bytes` | Regions, unique decoded bytes, and bytes rewritten with identical values |
|
||||
|
||||
Numbering is grouped by familyID because a bootloader ignores every field of
|
||||
another family's block, including its numbering. Within a group, a block number
|
||||
seen at several target addresses belongs to several copies of a sequence:
|
||||
esptool numbers each merged input file from 0, so inputs with equal block
|
||||
counts share a group, and each number then appears once per input. A block sent
|
||||
twice to the same address is one block. A group is complete when every number
|
||||
below its block count is present at the same count of addresses. If some numbers
|
||||
appear at more addresses than others, a copy may have lost a block, so coverage
|
||||
is unknown and a limitation names the group.
|
||||
|
||||
## UF2 facts, limitations and failures
|
||||
|
||||
| Situation | Outcome |
|
||||
| ------------------------------------------------------------- | ---------------------------------------------------------------------------------------------- |
|
||||
| Blocks out of address order | Sorted into regions |
|
||||
| Blocks numbered at or below the previous block of their group | Counted in `out_of_order_block_count`; the spec lets blocks arrive in any order |
|
||||
| An address written twice with the same value in one family | Accepted and counted in `duplicate_bytes` |
|
||||
| An address written twice with different values in one family | Fails, naming the family, the address and the block spans of both writes |
|
||||
| The same addresses in different families | Separate regions; never a conflict |
|
||||
| Numbered blocks absent from a group | Regions are returned; coverage is unknown and a limitation names the groups |
|
||||
| Reused numbers at unequal address counts | Regions are returned; coverage is unknown and a limitation names the groups |
|
||||
| An invalid block | Excluded; coverage is unknown and a limitation names the first one and its offset |
|
||||
| A partial final block | Excluded; coverage is unknown and the file may be truncated |
|
||||
| Block numbers at or above the block count | Payloads are kept; coverage is unknown |
|
||||
| Not-main-flash blocks (`0x1`) | Excluded from regions; their count and payload byte count are reported; coverage is unaffected |
|
||||
| File-container blocks (`0x1000`) | Excluded from regions; their files are not reconstructed, so coverage is unknown |
|
||||
| MD5 (`0x4000`) and extension-tag (`0x8000`) blocks | Counted and reported as a limitation; neither is decoded or verified |
|
||||
|
||||
Unlike a malformed Intel HEX record, which fails the whole file, an invalid UF2
|
||||
block is excluded and reported: each block carries its own framing and address,
|
||||
so its neighbours stay readable.
|
||||
|
||||
The MD5 structure is not checked because it is not a checksum of the block's
|
||||
bytes in the file. It names a region of device flash, usually aligned to erase
|
||||
pages, whose MD5 a flashing tool compares to skip writing. Producers also
|
||||
disagree on its place: the specification puts it in the last 24 bytes of the
|
||||
data area, while esptool v4.12.0 and v5.6.0 write it right after a shorter final
|
||||
payload, so those 24 bytes hold zeros or part of that structure.
|
||||
|
||||
## DfuSe regions and metadata
|
||||
|
||||
Each element is one entry at its own file range, named
|
||||
`target-<t>/element-<e>-0x<start>-0x<end>.bin` with zero-based target and
|
||||
element indexes, for example `target-0/element-0-0x08000000-0x0800ffff.bin`.
|
||||
Elements are never merged. Each occurrence's `source_location` is the byte range
|
||||
of the element data in the file.
|
||||
|
||||
Element `format_metadata` (`dfuse-element`) records `target_index`,
|
||||
`alternate_setting`, `target_name`, `element_index`, `start_address`,
|
||||
`end_address` and `length`. `target_name` is `null` when `bTargetNamed` is 0;
|
||||
otherwise it is `szTargetName` up to its first NUL, one character per byte.
|
||||
`compare_artifacts` treats an element's alternate setting and target name as
|
||||
its declaration, so the same bytes moved to another device memory are
|
||||
`changed` in `metadata`.
|
||||
|
||||
Root `format_metadata` (`dfuse`) records:
|
||||
|
||||
| Field | Meaning |
|
||||
| ---------------------------------------------------- | ------------------------------------------------------------------------------- |
|
||||
| `image_size`, `target_count` | `DFUImageSize` and `bTargets` from the prefix |
|
||||
| `element_count`, `empty_element_count`, `data_bytes` | All elements, elements of 0 bytes (which have no entry), and element data bytes |
|
||||
| `overlapping_element_count` | Elements that overlap an earlier element of the same alternate setting |
|
||||
| `device_version`, `product_id`, `vendor_id` | `bcdDevice`, `idProduct` and `idVendor` from the suffix |
|
||||
| `suffix_crc`, `computed_crc` | The stored `dwCRC`, and the same CRC computed over every byte before it |
|
||||
|
||||
Recognition already requires `bVersion` 1 and `bcdDFU` `0x011A`, so they are
|
||||
not repeated. The DFU suffix CRC is CRC-32 without the final inversion, as
|
||||
dfu-util stores and checks it.
|
||||
|
||||
Producers disagree on `DFUImageSize`. UM0391 calls it the file length, and
|
||||
dfu-util's `dfuse-pack.py` writes that value since commit `ee6b393d`
|
||||
(2020-10-08). Earlier `dfuse-pack.py` and MicroPython's `tools/dfu.py`
|
||||
(checked at `cb99ca98`) write the length before the suffix, and an ST community
|
||||
report describes ST's DfuFileMgr the same way; REA has not run DfuFileMgr.
|
||||
dfu-util's own parser ignores the field. REA accepts either value and keeps the
|
||||
declared one in `image_size`.
|
||||
|
||||
## DfuSe facts, limitations and failures
|
||||
|
||||
| Situation | Outcome |
|
||||
| ---------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------- |
|
||||
| `dwCRC` differs from the computed CRC | Follows `integrity_policy`, below |
|
||||
| A target without the `Target` signature | Fails with its byte offset |
|
||||
| `DFUImageSize` that equals neither the file length nor the bytes before the suffix | Fails with the declared size, the suffix offset, the file length and whether the suffix CRC matches |
|
||||
| A target size or an element size that disagrees with the bytes that hold it | Fails with the byte offset and the declared and available sizes |
|
||||
| An element whose address range wraps past `0xFFFFFFFF` | Fails with its byte offset |
|
||||
| Overlapping elements of one alternate setting | Kept as separate entries; a limitation names the first four; coverage is unaffected |
|
||||
| The same addresses in different alternate settings | Separate entries; an alternate setting selects a device memory |
|
||||
| Elements of 0 bytes, and targets with no elements | No entry; counted; a limitation names the first four, with address and alternate setting |
|
||||
|
||||
A CRC mismatch is an integrity contradiction, never proof of a cause. With the
|
||||
default `integrity_policy` `fail`, inspection and extraction fail with
|
||||
`artifact_integrity_mismatch`, whose details carry `algorithm: "dfu-crc32"`,
|
||||
`declared_crc32` and `calculated_crc32`. With `record-and-continue`
|
||||
(CLI `--integrity-policy record-and-continue`), the root occurrence becomes
|
||||
`mismatched`, `integrity_contradictions` records the pair for logical path `.`,
|
||||
and extraction carries that contradiction, because every element came from the
|
||||
contradicted container. The root occurrence has no `declared_integrity`: that
|
||||
field holds entry digests, so the declared CRC appears in the contradiction and,
|
||||
as a number, in `suffix_crc`. Under `fail`, the elements are hashed before the
|
||||
mismatch is raised.
|
||||
|
||||
## Limits
|
||||
|
||||
Each file must be at most 128 MiB. The UF2 decoder reserves at most 476 bytes
|
||||
per complete block before reading, merges payloads into regions of at most the
|
||||
same total size, and keeps one numbering entry per valid block, so its peak is
|
||||
about twice the decoded bytes. It retains at most 65,536 regions as entries; the
|
||||
order of blocks does not count against that bound. The DfuSe decoder holds the
|
||||
file once, serves elements as views of it, and retains at most 65,536 elements
|
||||
with bytes as entries; empty elements are only counted. These are fixed
|
||||
resource bounds, not format limits, and fail as limits (`file-size` and
|
||||
`capacity`) with no runtime override. Configurable ceilings are tracked in
|
||||
[#6176](https://github.com/morluto/rea/issues/6176).
|
||||
|
||||
Decoded regions are cached only for the reader's lifetime, and each open checks
|
||||
the admitted carrier identity again. These readers share the Intel HEX
|
||||
reader's region streams, cancellation and close behavior.
|
||||
|
||||
## Analyzing a region
|
||||
|
||||
Inventory does not establish the CPU, byte order or memory map, or an entry
|
||||
point. To disassemble a region, extract it and open the region file with
|
||||
`open_binary` using format `raw-binary`, an architecture you choose, and
|
||||
`entry_point` set to the region start address; raw-binary maps file offset 0 at
|
||||
`entry_point`. Each region is opened separately, so references between regions
|
||||
are not resolved.
|
||||
|
||||
## Verification
|
||||
|
||||
Fixtures written from the UF2 specification and from ST UM0391 cover block and
|
||||
element layouts, numbering groups and reused numbers, families, rewrites and
|
||||
conflicts, invalid and partial blocks, not-main-flash and file-container blocks,
|
||||
structural faults, overlaps, both integrity policies, accepted boundaries beside
|
||||
each refusal, resource refusals, inventory, extraction, comparison and reader
|
||||
identity checks. The reader lifecycle (cancellation, close and stream
|
||||
ownership) is covered by the Intel HEX reader tests, which exercise the shared
|
||||
reader. Two DfuSe fixtures are byte-identical, CRC included, to what the
|
||||
`build()` functions of dfu-util's `dfuse-pack.py` (at `f9a537c9`) and
|
||||
MicroPython's `tools/dfu.py` (at `cb99ca98`) produce for the same image, one for
|
||||
each `DFUImageSize` convention. Those functions were transcribed and run under
|
||||
Python 3; the scripts themselves were not run. CLI and stdio MCP tests exercise
|
||||
the same artifact provider and advertised schemas.
|
||||
|
||||
`npm run verify:uf2` runs esptool's `merge-bin --format uf2` on generated
|
||||
inputs at known addresses, with and without MD5, with 256-byte payloads, with
|
||||
two inputs of equal block count, and on a concatenation of two chips' outputs.
|
||||
Inspection and extraction must return every input at its address under
|
||||
esptool's own familyID. It passed with esptool v4.12.0 and v5.6.0 on macOS. Set
|
||||
`REA_ESPTOOL_PYTHON` to a Python that runs `-m esptool`. Pico SDK, picotool,
|
||||
`uf2conv.py`, TinyUF2 and real DfuSe producers remain unverified, and no device
|
||||
accepted these files in REA's tests.
|
||||
@@ -119,6 +119,7 @@ Then rerun this installer.",implemented,verified by shell branch inspection
|
||||
90.5,@modelcontextprotocol/server:tool input validation,Caller sends arguments that violate a declared tool schema,"Canonical invalid_request with the SDK diagnostic plus recovery.kind=inspect_input_contract and a bounded instruction to read the advertised schema or call tools/list",Keeps field/type correction while telling the agent how to recover; diagnostics over 4 KiB stay on the existing retained-error path,Normalize ordinary validation failures and retain oversized diagnostics,implemented,verified by real in-memory and stdio MCP client/server calls with wrong argument types
|
||||
90.6,incur:unknown CLI command,Caller enters an unknown REA command,"'COMMAND' is not a command for 'rea'. Suggested command: rea --help",Low risk; identifies input and exact discovery action,Keep framework-generated copy,unchanged,verified by real packaged dispatcher invocation
|
||||
90.7,src/ghidra/GhidraProviderClient.ts:projectPrimarySessionError,One procedure's decompile reaches REA_GHIDRA_DECOMPILE_TIMEOUT_SECONDS,"Ghidra decompilation of {entry} exceeded the {N}-second limit set by REA_GHIDRA_DECOMPILE_TIMEOUT_SECONDS. Only this procedure's decompilation was stopped; the Ghidra session remains usable.",Low risk; names the setting and keeps the session,Keep current copy,implemented,provider projection test and real-provider lane
|
||||
90.8,src/ghidra/GhidraFileOffsetFailure.ts:ghidraFileOffsetUnavailable,"address_to_file_offset receives a well-formed address that is unmapped, uninitialized, without an authoritative target-file source, ambiguous or beyond the exact JSON integer range",{bridge reason}. REA establishes no single exact original-file offset for this well-formed address and does not derive or choose one.,Previous copy called the input invalid and directed schema repair for a valid address,"Keep the bridge reason as the headline, point to inspect_native_load_image, omit recovery.kind=inspect_input_contract and record details.issues[0].observed.file_mapping",implemented,provider projection tests; MCP and CLI analysis path through the real Ghidra client with a simulated bridge; real Ghidra not run
|
||||
91,src/application/binaryDiagnostics/ByteEntropyService.ts (issues); src/domain/analysisErrorPresentation.ts (recovery),Invalid file or window,Expected a regular file.; Selected file could not be found.; Selected byte window starts beyond the end of the file.; Selected byte window extends beyond the end of the file.; File size exceeds exact numeric offset representation.,Invalid selection cannot produce complete entropy Evidence,"Read the declared input schema or CLI usage for inspect_byte_entropy; if using MCP and the catalog may be stale, call tools/list. Correct the listed fields, then retry. Do not repeat the unchanged call.",unchanged,Executed CLI/stdio MCP: path/offset/length refusals and tiny + system-binary controls; shared canonical recovery; service/MCP tests check classification and fields; remaining listed errors source-inspected
|
||||
92,src/domain/analysisView/byteEntropyView.ts,Non-summary entropy view,Byte entropy Evidence supports the summary view; retrieve parent Evidence for the complete block series.,Unsupported view selection,Use the summary view or retrieve complete parent Evidence,implemented,typed input rejection
|
||||
93,src/windows/WindowsNativeLoader.ts:load refused,Windows refuses to load the digest-verified native addon (for example Smart App Control or App Control policy),"REA Windows native controls are unavailable: Windows refused to load the verified native/windows/build/rea-windows-x64.node: {OS error} The file matched the package manifest, so restoring or reinstalling it will not help. ...",Previous artifact copy told users to restore or rebuild files that were already intact,"Name the OS refusal, say restoring cannot help, state that Ghidra and JADX need native controls while setup can still register clients, and point to a policy-allowed host or a maintainer-signed build",implemented,simulated loader refusal unit test; real Smart App Control block not reproduced
|
||||
|
File diff suppressed because it is too large.
Load diff
@@ -0,0 +1,263 @@
|
||||
# Evidence input composition verification
|
||||
|
||||
The shared declaration and request resolver cover 20 individual-Evidence
|
||||
composition tools. Artifact file sources, analysis views and environment export
|
||||
keep their explicit source owners. Catalog conformance checks those boundaries
|
||||
and rejects undeclared record inputs or unsupported source containers.
|
||||
|
||||
This record accompanies [#8293](https://github.com/morluto/rea/issues/8293) and
|
||||
the real Ghidra input-capacity regression
|
||||
[#8281](https://github.com/morluto/rea/issues/8281).
|
||||
The [machine-readable receipt](evidence-input-composition-2026-10-31.json)
|
||||
retains all fourteen model attempts, artifact identities, oracle provenance,
|
||||
tool profiles, actual usage, resource measurements and limitations. It also
|
||||
records the thirteen subsequent scripted public-transport attempts below.
|
||||
|
||||
## Runtime and measurement scope
|
||||
|
||||
Tests used Node 24.22.1 and local Codex CLI 0.165.0, explicitly requesting
|
||||
`gpt-6.1-terra` with reasoning `high`. This identifies the requested alias;
|
||||
the resolved server model is not inferred. Both dirty candidate builds were
|
||||
based on `23de10bb2849cdce8ddbe70504aa7cc57de75a98`. Their complete built-file
|
||||
fingerprints are recorded in the receipt. The later build includes the shared
|
||||
reference-format hint and authenticated source/type-boundary corrections.
|
||||
Subsequent cleanup reused the existing pure semantic-query input schema and
|
||||
moved SDK tests into their boundary project. Full and compact catalog bytes
|
||||
remained identical to the later model build; 22 affected checks were rerun.
|
||||
|
||||
Heavy tasks ran serially in owned Linux scopes: one CPU, 4 GiB hard memory,
|
||||
3 GiB early stop, no swap and at most 192 tasks. CPU and peak memory are kernel
|
||||
process-family counters at payload exit while the observer remains alive;
|
||||
they are not final whole-scope counters. Cleanup was verified for every model
|
||||
attempt and CLI phase. No OOM event occurred.
|
||||
|
||||
Each model task starts a fresh MCP process and isolated client profile. Native
|
||||
tasks start a fresh Ghidra database, then reuse its two retained dossiers.
|
||||
System/filesystem caches were not cleared; cold filesystem costs are unmeasured.
|
||||
Initial primary tasks used the complete catalog. Later scoped tasks are labeled
|
||||
`diagnostic/catalog-selection`; their different tools, guidance and selected
|
||||
facts prevent a controlled speed or token comparison.
|
||||
|
||||
## Real application results
|
||||
|
||||
| Domain | Real artifact | Reviewed outcome |
|
||||
| ---------------- | ------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| Native | Installed jq ELF, real Ghidra 12.1.4 | Each selected function analyzed once; complete parent references supported a direct one-hop path and function comparison. Independent `objdump` corroborated the selected call. Nonexhaustive frontier and behavioral uncertainty remained visible. |
|
||||
| Managed | Shipped NuGet.CommandLine.XPlat.dll, assembly 6.11.2.1 | Artifact/member parent references projected a managed graph. The existing distinct-Evidence rule refused self-comparison. Digest, MVID, owner/token and selected CIL facts matched an independent metadata reader; its parser warnings limit that oracle's scope. |
|
||||
| JavaScript | Obsidian 1.14.4 ASAR | Partial: the producer hit its existing 300-second per-source timeout. The retained parent still supported a semantic trace. A bidirectional feature trace exceeded 10 MiB and had no supported retained summary. Tracked separately in [#8296](https://github.com/morluto/rea/issues/8296). |
|
||||
| JavaScript | Shipped agent-browser 0.27.0 CLI entrypoint | The final scoped task analyzed the exact selected source once, then reused its parent for feature and semantic traces. Independent source facts confirmed builtin imports/call locations. Unselected sources remained unknown; directory and file digests were distinguished. |
|
||||
| WASM | Shipped VS Code Tree-sitter CSS module | One real WABT inspection supplied retained references for export comparison and a portable bundle. An independent raw WASM reader confirmed seven imports, three exports and `tree_sitter_css: () -> i32`. Static interface equality did not imply runtime compatibility. |
|
||||
| Artifact / Apple | Rectangle 2.0.4 release ZIP | One retained inventory supported Apple projection and artifact comparison. Independent ZIP and Mach-O observations corroborated paths/content. Symlink metadata gaps remained unknown. |
|
||||
| Android | NewPipe 0.29.1 release APK | One retained inventory supported Android projection. Independent ZIP observations confirmed manifest, three DEX files and four ABI library members/digests. JNI path-pair hypotheses stayed inferred. |
|
||||
| Harmony | Shark 1.0.8 release APP and its real inner HAP | Both retained inventories supported Harmony projection. Separate ZIP/JSON checks preserved outer ZIP, APP and HAP identities, Stage/Ark/resource path observations and shallow APP coverage. Signature naming was not treated as cryptographic verification. |
|
||||
| Process | Two controlled real jq CLI captures | Authentic captures were imported and compared by references, then their retained comparison was aggregated. MCP output deep-equaled the existing inline CLI result. Terminal change, settlement differences, filesystem/process unknowns and source closure were preserved. |
|
||||
|
||||
The first supplemental agent-browser model task wandered into unrelated
|
||||
inventory tools and failed to perform the requested producer analysis. It was
|
||||
stopped and retained as an interrupted failure, including cumulative usage as
|
||||
a lower bound. The successful later scoped task does not erase that attempt.
|
||||
The missing-reference task produced its expected typed failure without opening
|
||||
a target, repeating a request or inventing Evidence.
|
||||
|
||||
## Actual model task costs
|
||||
|
||||
Input counts include schemas, accumulated context, extra calls and retries.
|
||||
Cached input is a subset of total input. Reasoning is included in output;
|
||||
its separate subset is retained in JSON. These are completed task costs, not
|
||||
serialized-byte estimates. The initial partial/interrupted attempts and their
|
||||
costs also remain in the receipt.
|
||||
|
||||
| Later task | Calls / failed | Input tokens | Cached input | Output tokens | Task elapsed (s) | Family CPU (s) | Peak bytes |
|
||||
| ------------------------------ | -------------: | -----------: | -----------: | ------------: | ---------------: | -------------: | ------------: |
|
||||
| jq, complete catalog | 11 / 0 | 599,933 | 535,424 | 1,894 | 80.931 | 38.326466 | 1,269,936,128 |
|
||||
| agent-browser, scoped | 12 / 1 | 619,589 | 563,712 | 2,519 | 54.521 | 9.472096 | 903,847,936 |
|
||||
| NewPipe, scoped | 5 / 0 | 179,635 | 150,528 | 1,750 | 40.835 | 10.080132 | 704,409,600 |
|
||||
| Shark APP, scoped | 6 / 0 | 221,571 | 197,376 | 1,339 | 32.437 | 6.377128 | 560,021,504 |
|
||||
| Shark HAP, scoped | 5 / 0 | 184,452 | 152,576 | 1,212 | 31.108 | 6.012587 | 580,714,496 |
|
||||
| WASM plus export, scoped | 4 / 0 | 161,702 | 141,056 | 1,033 | 32.320 | 9.036127 | 700,076,032 |
|
||||
| Imported process chain, scoped | 3 / 0 | 89,101 | 73,984 | 725 | 26.911 | 5.575060 | 531,283,968 |
|
||||
|
||||
The later jq task made no malformed reference calls, while the initial task
|
||||
omitted the required `kind` twice. It selected a different callee and made three
|
||||
guidance calls; its longer elapsed time and different facts preclude attributing
|
||||
a general improvement to the new hint.
|
||||
|
||||
## Equivalent portable CLI comparison
|
||||
|
||||
The actual WASM model task exported a 13,925,066-byte authenticated bundle.
|
||||
Both fresh CLI processes selected its same complete inspection parent and
|
||||
compared exports against that parent. Their 3,732-byte outputs were byte-identical
|
||||
and deep-equal to the model's MCP comparison:
|
||||
`ev_b32346965c153a067abff9698aeba1ccbace34c78ac2071ac18311250b68c6da`.
|
||||
Bundle, selected source and runtime hashes stayed unchanged. Neither CLI arm
|
||||
started a decoder or reanalyzed the module.
|
||||
|
||||
| Input | Request bytes | Elapsed including scope (s) | Family CPU (s) | Peak bytes |
|
||||
| ---------------------------------- | ------------: | --------------------------: | -------------: | ----------: |
|
||||
| Complete inline pair | 27,823,017 | 2.832 | 2.612440 | 370,515,968 |
|
||||
| Retained pair plus selected bundle | 261 | 2.632 | 2.385397 | 310,009,856 |
|
||||
|
||||
The reference arm still reads/authenticates the existing bundle. This single
|
||||
ordered pair ran inline first with shared uncontrolled host caches. It establishes
|
||||
equivalent completed results and records observed costs; it does not establish
|
||||
a general speedup. Common upstream inspection/export costs appear in the model
|
||||
table. Preparation additionally cost 1.119 s / 0.875255 CPU s / 316,796,928 peak
|
||||
bytes; verification cost 0.518 s / 0.282755 CPU s / 42,881,024 peak bytes.
|
||||
The optional tokenizer was unavailable, so request-token savings are unmeasured.
|
||||
Request bytes are not presented as model tokens.
|
||||
|
||||
## Fresh-connection public transport matrix
|
||||
|
||||
The real transport matrix completed all nine supported domain lanes on built revision `7e10c1cc9` (`63746cf7b340358ac5ce609761ea3198ae3a62e24df52f62ed5be16893940021`). It used public stdio MCP and CLI, real artifacts and authentic exported Evidence; no model or fixture provider ran.
|
||||
|
||||
| Lane | Attempts | Scope seconds, all attempts | CPU seconds, all attempts | Max kernel peak MiB | CLI calls, including negatives |
|
||||
| ----------- | -------: | --------------------------: | ------------------------: | ------------------: | -----------------------------: |
|
||||
| native | 1 | 35.915 | 34.674174 | 927.99 | 0 |
|
||||
| javascript | 3 | 34.324 | 33.094918 | 658.91 | 8 |
|
||||
| managed | 2 | 37.341 | 36.221694 | 694.65 | 9 |
|
||||
| android | 1 | 20.208 | 19.569397 | 696.40 | 4 |
|
||||
| harmony-app | 1 | 16.714 | 15.950327 | 495.68 | 4 |
|
||||
| harmony-hap | 1 | 16.294 | 15.886932 | 508.47 | 4 |
|
||||
| apple | 1 | 25.453 | 24.824482 | 642.90 | 8 |
|
||||
| wasm | 2 | 22.648 | 22.027649 | 791.67 | 4 |
|
||||
| process | 1 | 9.056 | 8.586552 | 472.58 | 2 |
|
||||
|
||||
Producer/export costs from the failed JS/managed harness attempts are included. Three harness assumptions and one actual SDK transport-capacity failure are retained individually. Every payload cleanup and kernel observation verified; no OOM or increased allowance.
|
||||
|
||||
Fresh imported references match complete CLI inline and selected-bundle outputs where those command contracts support them. Small APP/process/correlation inputs also match admitted MCP inline. Larger MCP inline forms hit the unchanged 1024-element guard. Native and selected managed-member comparisons have no equivalent Evidence-input CLI form; process accepts two complete capture files, and artifact comparison uses explicit evidence-file selectors.
|
||||
|
||||
The 27,823,016-byte WASM inline arguments exceeded the default 10,485,760-byte stdio frame buffer and closed the connection before application admission. The resumed attempt reused the same authentic bundle: fresh MCP references and both supported complete CLI forms agree. No WABT replay or limit increase.
|
||||
|
||||
Process capture comparison and behavior aggregation preserve changed/unknown facts. Correlation with genuine native comparison Evidence remains unknown with one unresolved branch under an explicit noncausal hypothesis.
|
||||
|
||||
These are scripted transport-task costs with uncontrolled caches, not a before/after benchmark or model-token measurement. Earlier WASM inspection/process capture costs are linked in the prior verification report. Later source-only diagnostic fixes have separate focused validation.
|
||||
|
||||
The accompanying JSON retains all thirteen attempts, exact kernel counters,
|
||||
source/result identities and hashes, CLI arm receipts, failures and limits.
|
||||
These scripted costs do not add to the model-token table.
|
||||
|
||||
## Review follow-up
|
||||
|
||||
An additional local terra/high source-only review found that the refactored
|
||||
process CLI read later selected files before authenticating the first capture.
|
||||
The shared owner now exposes authenticated source facts and their comparison;
|
||||
CLI authenticates each source before reading the next, while MCP uses the same
|
||||
readers and comparator. Two public CLI regressions cover invalid left/right
|
||||
records with missing later files. The original real captures were compared
|
||||
again without target execution and retained the exact comparison Evidence ID.
|
||||
|
||||
Greptile found a fixed-server WASM regression: caller-supplied complete inputs
|
||||
incorrectly required a recorder. Shared retention now explicitly recognizes
|
||||
caller-owned inputs. Provider-produced sources still require a recorder;
|
||||
available recorders, their rejections and cancellation sources remain honored.
|
||||
A fixture-backed public fixed-server MCP regression and ownership/cancellation
|
||||
checks cover that distinction. These follow-up regressions are not additional
|
||||
real-provider measurements. The related nine-file run passed 103 checks;
|
||||
subsequent schema cleanup and CLI checks passed ten more selected checks.
|
||||
Type, lint, Knip and architecture/module-boundary checks were repeated.
|
||||
|
||||
The fourteen application-analysis attempts above are separate from source-only
|
||||
review; that CLI review did not report usable model-token counts. Its cleanup
|
||||
was verified, with 57.860 s task elapsed, 4.720926 s family CPU and 427,364,352
|
||||
peak bytes. Existing captured model measurements remain tied to their recorded
|
||||
runtime fingerprints, preceding these review corrections.
|
||||
|
||||
The additional seven-file boundary run passed 161 checks after the CLI's JSON
|
||||
parser was corrected to keep public command names in malformed-input diagnostics.
|
||||
It includes 17 fixed-server cases for six JavaScript and three mobile consumers,
|
||||
direct inline analysis views, missing references, wrong source roles, summary
|
||||
retention and default input admission. These use complete authored fixtures;
|
||||
they do not substitute for the real-artifact matrix. The run included a runtime
|
||||
build and cost 154.106 s / 153.592029 CPU s / 1,884,516,352 peak bytes. Subsequent
|
||||
type, touched-file lint and module checks passed, with no OOM or cleanup gap.
|
||||
|
||||
Native function and process workflows send malformed complete source records to
|
||||
the canonical Evidence owner after resource admission. Schema or identity failures return
|
||||
`evidence_integrity_mismatch` in both MCP schema profiles, including the original
|
||||
`relationship_coverage` diagnostics. Other workflows preserve their format-owner
|
||||
error projections. The directional-coverage regression still
|
||||
checks every malformed case and the advertised full schema. Invalid request
|
||||
envelopes and reference grammar remain request errors; no duplicate source parse
|
||||
was introduced solely to reproduce the former SDK error classification.
|
||||
|
||||
A later review reported a possible null-source dereference in reconciliation.
|
||||
On `70db1e816`, the public request has no identity refinement; source authentication
|
||||
runs before identity-only validation. A public fixed-server MCP probe passed two
|
||||
valid controls and rejected 24 malformed source cases across full/compact profiles
|
||||
with structured `invalid_request` and accurate paths. Both complete schemas also
|
||||
rejected each of twelve cases without throwing. No provider ran. Cleanup was
|
||||
verified: 3.334 s elapsed, 3.155635 CPU s and 287,404,032 peak bytes, with no OOM.
|
||||
This authored-fixture probe disproves the reported path on that revision;
|
||||
it does not establish real passive-runtime behavior. No production change was
|
||||
needed for that finding. The accompanying JSON records its observations.
|
||||
|
||||
CI later exposed the corresponding process-depth assertion against the old SDK
|
||||
error envelope. Its regression now checks the canonical integrity error while
|
||||
preserving the public left-source, `parameters` and exact nesting-limit diagnostic.
|
||||
A normal comparison must still succeed afterward and provider startup is forbidden.
|
||||
The iterative JSON depth guard still rejects before recursive cloning or hashing;
|
||||
no second source parser was added. Runtime code was unchanged. Six related MCP
|
||||
files passed 104 checks, including source identity, native/process diagnostics,
|
||||
investigation composition and session lifecycle: 64.384 s elapsed, 64.066692 CPU s
|
||||
and 1,476,808,704 peak bytes. Cleanup was verified with no OOM.
|
||||
|
||||
A retained-closure acceptance assertion also assumed that a complete inline
|
||||
comparison must already exist in the ledger. The shared workflow intentionally
|
||||
uses that supplied record: a missing dependency is reported for inline input,
|
||||
while an unresolved selector reports the comparison itself. Both still require
|
||||
the full referenced closure. Importing the originating bundle restores both
|
||||
forms to complete, equal Evidence. Five relevant acceptance files passed 17
|
||||
checks, covering default-budget aggregation, artifact coverage, replay and public
|
||||
local contracts: 123.152 s elapsed, 121.276327 CPU s and 1,466,314,752 peak bytes.
|
||||
Cleanup was verified with no OOM; runtime code was unchanged.
|
||||
|
||||
The HTTP admission-edge regression now checks the same source-owner semantics.
|
||||
An admitted array of 1,019 malformed carriers fails at its first source, while
|
||||
1,020 carriers exceed the unchanged 1,024-element envelope allowance. Responses
|
||||
were 703 and 824 bytes, respectively; both stayed below 4 KiB and the server kept
|
||||
serving. The shared resolver additionally preserves `input_path` on owner errors
|
||||
when absent. Original error identity, owner diagnostics and existing paths remain
|
||||
intact, with no second source parse. The rebuilt nine-file gate passed 75 checks,
|
||||
including all 41 ADB CLI cases, resolver, native references and HTTP/stdio admission:
|
||||
98.459 s elapsed, 98.133579 CPU s and 1,605,656,576 peak bytes; cleanup verified,
|
||||
no OOM.
|
||||
|
||||
CI also terminated one unchanged ADB CLI child at its helper's existing 20-second
|
||||
deadline. Two isolated exact 17-extra CLI probes returned the expected argument
|
||||
validation failure in 1.970 s with a fresh Node compile cache and 1.170 s reusing
|
||||
it. Neither ran ADB or reached a device; the 20-second deadline was unchanged.
|
||||
Both local scopes cleaned up with no OOM. The rebuilt full ADB CLI file passed,
|
||||
including that case in 1.597 s. The CI delay was not reproduced; these observations
|
||||
do not establish baseline flakiness or a controlled before/after startup result.
|
||||
|
||||
After the diagnostic change, the untouched genuine jq/Ghidra bundle was imported
|
||||
again into a fresh target-free stdio server. Call path, function comparison and
|
||||
native matching returned complete public values equal to their original matrix
|
||||
goldens, including IDs, provenance, coverage and unknowns. Seven calls included
|
||||
no producer, target opening or Ghidra execution. The built runtime fingerprint
|
||||
was `1bcdcd29df3b90f2cfa61aad29d647dbfb22176d0bca880c7a537507fdeab271`;
|
||||
source bundle and runtime stayed unchanged. The replay cost 5.543 s / 5.263196 CPU s
|
||||
/ 447,791,104 peak bytes, with verified cleanup and no OOM. This is an additional
|
||||
scripted replay, separate from the thirteen-attempt matrix and fourteen model
|
||||
attempts. Its full receipt and the timeout probes are included in JSON.
|
||||
|
||||
## Regression and coverage limits
|
||||
|
||||
The focused regression run passed 508 checks covering source integrity, exact
|
||||
IDs, mixed/optional/nested selections, default constraints, converted full and
|
||||
compact SDK schemas, source closure, caller choices, CLI bundle immutability,
|
||||
capacity admission and late cancellation/source retention. SDK boundary tests
|
||||
are kept in the MCP boundary project; pure declaration checks stay in contracts.
|
||||
|
||||
Full-library lint exceeded the local early-stop limit even with one worker.
|
||||
The same selected 6,105 files were checked in sequential batches of 400;
|
||||
peak family memory was 741,650,432 bytes. Two new test grouping errors were
|
||||
corrected and those files rerun. Failed lint/driver attempts remain in local
|
||||
receipts. Process-driver path/ANSI oracle corrections reused the original
|
||||
authentic captures.
|
||||
|
||||
No real Hopper, IDA, Windows capture, Android device, Harmony runtime, passive
|
||||
browser reconciliation or managed/native runtime equivalence is established
|
||||
by these tests. There is no controlled main-versus-candidate model cohort,
|
||||
no cold filesystem measurement and no claim that every provider facet was
|
||||
independently verified.
|
||||
@@ -189,11 +189,13 @@ typed interfaces.
|
||||
|
||||
`compare_wasm_interfaces` accepts two complete `inspect_wasm_artifact` Evidence
|
||||
records, or exact retained references in the current MCP session, and required
|
||||
`scope: "imports" | "exports" | "both"`. It reads no files and needs no active
|
||||
WABT toolchain. CLI uses portable inline Evidence in one JSON document:
|
||||
`scope: "imports" | "exports" | "both"`. It uses producer-owned typed facts
|
||||
without rereading the artifact and needs no active WABT toolchain. CLI accepts
|
||||
portable inline Evidence in one JSON document, or
|
||||
resolves the same retained-reference inputs from an explicit saved bundle:
|
||||
|
||||
```bash
|
||||
rea compare-wasm-interfaces comparison-input.json --json
|
||||
rea compare-wasm-interfaces comparison-input.json --evidence-bundle /absolute/path/evidence.json --json
|
||||
```
|
||||
|
||||
```json
|
||||
@@ -204,8 +206,10 @@ rea compare-wasm-interfaces comparison-input.json --json
|
||||
}
|
||||
```
|
||||
|
||||
The references above are MCP-only; put complete Evidence objects in the CLI
|
||||
input. Resolve summary delivery to its retained complete parent before comparing.
|
||||
Omit `--evidence-bundle` when the CLI input contains complete inline records.
|
||||
Resolve summary delivery to its complete `normalized_result.parent_evidence_id`
|
||||
before comparing. References preserve that exact source identity and coverage;
|
||||
the bundle supplies no facts absent from the selected record.
|
||||
Results retain both input Evidence IDs, artifact paths/digests/lengths and every
|
||||
selected occurrence. Unique names are added, removed, changed or unchanged;
|
||||
duplicate names and unsupported signatures are unknown. Incomplete or historical
|
||||
|
||||
@@ -9,6 +9,7 @@ upstream texts; they do not change [REA's MIT license](../LICENSE).
|
||||
| `npm-dependencies.txt` | Every package path in the root lockfile, with its exact version, declared license, scope, download URL, and integrity. Includes development and optional platform entries. |
|
||||
| `npm-runtime-notices.txt` | Original license, copyright, and NOTICE texts collected from integrity-verified locked runtime npm archives across all platforms; also the license of Rolldown's generated runtime helpers. The archive manifest's name and version are checked. |
|
||||
| `protobuf-es-v2.16.0-LICENSE.txt` | Original Apache license omitted from the protobuf-es npm archive, preserved from its v2.16.0 source commit. Its embedded Google BSD notice is retained in `npm-runtime-notices.txt`. |
|
||||
| `microsoft-uf2-MIT.txt` | Unchanged `LICENSE.txt` from microsoft/uf2 revision `cf3bd7f9c274b949528f8cd6a19eef2502a687d2`, for the UF2 family names copied from its `utils/uf2families.json`. |
|
||||
| `nativeaot-parser-MIT.txt` | Original Washi notice for derived NativeAOT parser code; source revision is recorded in the parser and the root notices. |
|
||||
| `dotnet-runtime-MIT.txt` | Original .NET Foundation MIT text from the NativeAOT reference revision `a3b2d40328be0be3f8dd43f0e5cc925b8827b044`. |
|
||||
| `ghidra-12.1.4-{LICENSE,NOTICE,DISCLAIMER}.txt` | Unchanged top-level documents from `Ghidra_12.1.4_build`. Ghidra is not bundled; these reference copies do not replace its component and GPL support-program notices. The US-government-specific disclaimer applies to Ghidra, not to REA. |
|
||||
|
||||
@@ -0,0 +1,25 @@
|
||||
Microsoft UF2
|
||||
|
||||
The MIT License (MIT)
|
||||
|
||||
Copyright (c) Microsoft Corporation
|
||||
|
||||
All rights reserved.
|
||||
|
||||
Permission is hereby granted, free of charge, to any person obtaining a copy
|
||||
of this software and associated documentation files (the "Software"), to deal
|
||||
in the Software without restriction, including without limitation the rights
|
||||
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
||||
copies of the Software, and to permit persons to whom the Software is
|
||||
furnished to do so, subject to the following conditions:
|
||||
|
||||
The above copyright notice and this permission notice shall be included in all
|
||||
copies or substantial portions of the Software.
|
||||
|
||||
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
||||
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
||||
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
||||
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
||||
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
||||
SOFTWARE.
|
||||
@@ -78,5 +78,10 @@
|
||||
"file": "licenses/pefile-2024.8.26-LICENSE.txt",
|
||||
"source": "https://raw.githubusercontent.com/erocarrera/pefile/4b3b1e2e568a88d4f1897d694d684f23d9e270c4/LICENSE",
|
||||
"sha256": "b44409c067c6da52bfb54bb6624fb11a7b52157c3a13ae1400232f8196e86ad3"
|
||||
},
|
||||
{
|
||||
"file": "licenses/microsoft-uf2-MIT.txt",
|
||||
"source": "https://raw.githubusercontent.com/microsoft/uf2/cf3bd7f9c274b949528f8cd6a19eef2502a687d2/LICENSE.txt",
|
||||
"sha256": "c581accf1e5449482b70ac5fcdfb171599317ccff559cf0794d771671486f800"
|
||||
}
|
||||
]
|
||||
@@ -65,6 +65,7 @@
|
||||
"bridge/ghidra/ReaGhidraAnalysisOptions.java",
|
||||
"bridge/ghidra/ReaGhidraNoReturnFix.java",
|
||||
"bridge/ghidra/ReaGhidraMipsSoftFloat.java",
|
||||
"bridge/ghidra/ReaGhidraRiscvCallingConvention.java",
|
||||
"bridge/ghidra/ReaGhidraPrepareRaw.java",
|
||||
"bridge/ghidra/ReaGhidraPrepareDol.java",
|
||||
"bridge/android/ReaJadxBridge.java",
|
||||
@@ -238,9 +239,11 @@
|
||||
"verify:javascript:retained-facts": "npm run build:cached && node scripts/verify/javascript/retained-facts.mjs",
|
||||
"verify:application-workflows": "npm run build:cached && node scripts/verify/javascript/application-workflows.mjs",
|
||||
"verify:managed": "npm run build:cached && node scripts/verify/managed/conformance.mjs",
|
||||
"verify:uf2": "npm run build:cached && node scripts/verify/firmware/uf2-esptool.mjs",
|
||||
"verify:firmware": "npm run build:cached && node scripts/verify/firmware/analysis.mjs",
|
||||
"verify:devicetree": "npm run build:cached && node scripts/verify/firmware/devicetree.mjs",
|
||||
"verify:esp-images": "npm run build:cached && node scripts/verify/firmware/esp-image.mjs",
|
||||
"verify:esp-flash": "npm run build:cached && node scripts/verify/firmware/esp-flash.mjs",
|
||||
"fixtures:firmware": "node scripts/fixtures/firmware/generate.mjs",
|
||||
"verify:android": "npm run build:cached && node scripts/verify/android/analysis.mjs",
|
||||
"verify:jeb": "npm run build:cached && node scripts/verify-jeb.mjs",
|
||||
|
||||
@@ -125,6 +125,12 @@ const PATH_SCOPES = {
|
||||
],
|
||||
// Apple artifact CLI/MCP lane on macOS, including its focused regressions.
|
||||
apple: [
|
||||
"src/artifacts/InesArtifactReader.ts",
|
||||
"src/domain/inesContainer.ts",
|
||||
"tests/fixtures/ines.ts",
|
||||
"tests/acceptance/inesContainer.test.ts",
|
||||
"tests/boundary/filesystem/inesArtifact.test.ts",
|
||||
"tests/boundary/filesystem/inesCancellation.test.ts",
|
||||
"src/hopper/BridgeLauncher.ts",
|
||||
"src/domain/hopperErrors.ts",
|
||||
"tests/boundary/providers/hopper/hopperMacLaunchArguments.test.ts",
|
||||
@@ -146,12 +152,18 @@ const PATH_SCOPES = {
|
||||
"tests/boundary/filesystem/installScript.test.ts",
|
||||
"tests/boundary/filesystem/nibNativeHierarchy.test.ts",
|
||||
"tests/boundary/filesystem/interfaceBuilderBundle.test.ts",
|
||||
// Compiled UI handler joins: metadata reader, graph and trace workflow.
|
||||
"src/domain/native/objcSwiftMetadata.ts",
|
||||
"src/domain/native/nativeInvestigationGraph.ts",
|
||||
"src/application/native/NativeUiActionTrace.ts",
|
||||
"tests/acceptance/nativeUiHandlerLocation.test.ts",
|
||||
"tests/boundary/cli/nativePlistCli.test.ts",
|
||||
"tests/boundary/cli/nativePlistPrototypeKeyCli.test.ts",
|
||||
"tests/boundary/cli/nativePlistSelectionFailureCli.test.ts",
|
||||
"tests/boundary/cli/nativeDemangleCli.test.ts",
|
||||
"tests/boundary/cli/nativeDispatchPropertyAttributes.test.ts",
|
||||
"tests/boundary/mcp/nativeDemangleMcp.test.ts",
|
||||
"tests/boundary/mcp/nativeMachoSummaryMcp.test.ts",
|
||||
"tests/boundary/providers/native/nativeSignatureTargetBinding.test.ts",
|
||||
"tests/boundary/cli/nativeSignatureCli.test.ts",
|
||||
"tests/boundary/cli/nativeMachoArm64_32Cli.test.ts",
|
||||
|
||||
@@ -17,17 +17,28 @@ import { compareBrowserScenarios } from "../../dist/domain/browserScenarioDiff.j
|
||||
import { browserScenarioCaptureSchema } from "../../dist/domain/browserScenarioCapture.js";
|
||||
import { parseEvidence } from "../../dist/domain/evidence.js";
|
||||
|
||||
// Readiness includes descendant document loads, so complete network capture
|
||||
// does not depend on whether an inline script runs before an iframe finishes.
|
||||
const document = (path) => {
|
||||
const icon = '<link rel="icon" href="data:,">';
|
||||
if (path.startsWith("/nested"))
|
||||
return `<script>throw new Error("nested popup failure")</script>
|
||||
<script>opener.postMessage("nested-ready", "*")</script>`;
|
||||
return `${icon}<script>throw new Error("nested popup failure")</script>
|
||||
<script>addEventListener("load", () => opener.postMessage("nested-ready", "*"))</script>`;
|
||||
if (path.startsWith("/popup"))
|
||||
return `<script>throw new Error("popup failure")</script>
|
||||
<script>onmessage = e => {
|
||||
if (e.data === "nested-ready") opener.postMessage("popup-ready", "*");
|
||||
}; window.open("/nested");</script><iframe src="/child"></iframe>`;
|
||||
if (path.startsWith("/child")) return "child frame";
|
||||
return `<script>onmessage = e => {
|
||||
return `${icon}<script>throw new Error("popup failure")</script>
|
||||
<script>
|
||||
let loaded = false, nested = false;
|
||||
const ready = () => {
|
||||
if (loaded && nested) opener.postMessage("popup-ready", "*");
|
||||
};
|
||||
addEventListener("load", () => { loaded = true; ready(); });
|
||||
onmessage = e => {
|
||||
if (e.data === "nested-ready") { nested = true; ready(); }
|
||||
};
|
||||
window.open("/nested");
|
||||
</script><iframe src="/child"></iframe>`;
|
||||
if (path.startsWith("/child")) return `${icon}child frame`;
|
||||
return `${icon}<script>onmessage = e => {
|
||||
if (e.data === "popup-ready") document.body.dataset.ready = "true";
|
||||
}</script><button onclick="window.open('/redirect')">Open popup</button>`;
|
||||
};
|
||||
@@ -75,7 +86,25 @@ const assertNetwork = (result, origin) => {
|
||||
1,
|
||||
`${kind} ${path} must be captured exactly once`,
|
||||
);
|
||||
assert.equal(result.completeness.equality_eligible, true);
|
||||
assert.equal(
|
||||
result.completeness.equality_eligible,
|
||||
true,
|
||||
JSON.stringify({
|
||||
limitations: result.limitations,
|
||||
unfinished: result.events.items.filter(
|
||||
(event) => event.kind === "request-unfinished",
|
||||
),
|
||||
}),
|
||||
);
|
||||
assert.equal(
|
||||
result.events.items.filter(
|
||||
(event) =>
|
||||
event.kind === "request-finished" &&
|
||||
event.url?.url === origin + "/child",
|
||||
).length,
|
||||
1,
|
||||
"child document must finish before network capture ends",
|
||||
);
|
||||
assert.equal(compareWithSelf(result).overall_status, "unchanged");
|
||||
};
|
||||
|
||||
@@ -180,13 +209,27 @@ const verifyTransports = async (executable, origin) => {
|
||||
|
||||
/** Verify popup timing, selection, coverage, and shared-context isolation on Chromium. */
|
||||
export async function verifyPopupEventCoverage(executable) {
|
||||
let slowChild = true;
|
||||
let childTimer;
|
||||
const childStarted = Promise.withResolvers();
|
||||
const server = createServer((request, response) => {
|
||||
if (request.url === "/redirect") {
|
||||
response.writeHead(302, { Location: "/popup" });
|
||||
response.end();
|
||||
} else {
|
||||
response.setHeader("Content-Type", "text/html");
|
||||
response.end(document(request.url ?? "/"));
|
||||
if (slowChild && request.url === "/child") {
|
||||
// Keep one response open while the nested page can announce readiness.
|
||||
// This exposes premature capture cutoff even on fast local browsers.
|
||||
response.writeHead(200);
|
||||
response.write(document("/child"));
|
||||
childStarted.resolve();
|
||||
childTimer = setTimeout(() => response.end(), 1500);
|
||||
} else if (slowChild && request.url === "/nested") {
|
||||
void childStarted.promise.then(() => response.end(document("/nested")));
|
||||
} else {
|
||||
response.end(document(request.url ?? "/"));
|
||||
}
|
||||
}
|
||||
});
|
||||
server.listen(0, "127.0.0.1");
|
||||
@@ -199,6 +242,7 @@ export async function verifyPopupEventCoverage(executable) {
|
||||
await capture(scenarioFor(executable, origin, ["network"])),
|
||||
origin,
|
||||
);
|
||||
slowChild = false;
|
||||
const errors = await capture(
|
||||
scenarioFor(executable, origin, ["page-errors"]),
|
||||
);
|
||||
@@ -240,6 +284,7 @@ export async function verifyPopupEventCoverage(executable) {
|
||||
stdio_mcp: true,
|
||||
};
|
||||
} finally {
|
||||
clearTimeout(childTimer);
|
||||
server.closeAllConnections();
|
||||
await new Promise((resolve, reject) =>
|
||||
server.close((error) =>
|
||||
|
||||
@@ -11,6 +11,8 @@ import { cleanupOwnedProcessGroup } from "../../dist/process/ProcessOwnership.js
|
||||
import { ArtifactResourceScope } from "../../dist/artifacts/ArtifactResourceScope.js";
|
||||
import { AnalysisCleanupLease } from "../../dist/artifacts/AnalysisCleanupLease.js";
|
||||
import { withVerifierCleanup } from "./verifier-cleanup.mjs";
|
||||
import { discoverCdpEndpoint } from "../../dist/browser/CdpEndpoint.js";
|
||||
import { BrowserObservationError } from "../../dist/domain/browserObservationError.js";
|
||||
import { CdpConnection } from "../../dist/browser/CdpConnection.js";
|
||||
import { importVerifierSource } from "./verifier-source-loader.mjs";
|
||||
|
||||
@@ -26,6 +28,7 @@ const { BROWSER_DEVTOOLS_STARTUP_TIMEOUT_MS, waitForBrowserDevtoolsPort } =
|
||||
export async function startRuntimeFixtureBrowser(executable, initialUrl) {
|
||||
await requireOrdinaryWindowsVerifierCaller();
|
||||
const profile = await mkdtemp(join(tmpdir(), "rea-real-web-runtime-"));
|
||||
const deadline = performance.now() + BROWSER_DEVTOOLS_STARTUP_TIMEOUT_MS;
|
||||
let supervisor;
|
||||
let stopOwnedBrowser;
|
||||
const resources = new ArtifactResourceScope();
|
||||
@@ -94,7 +97,13 @@ export async function startRuntimeFixtureBrowser(executable, initialUrl) {
|
||||
stderr: () => supervisor.snapshot().stderr.text,
|
||||
// Match the broader real-browser verifier's allowance for cold Chrome
|
||||
// startup on shared runners; early process failures still reject at once.
|
||||
timeoutMs: BROWSER_DEVTOOLS_STARTUP_TIMEOUT_MS,
|
||||
timeoutMs: Math.max(1, deadline - performance.now()),
|
||||
});
|
||||
await waitForRuntimeFixturePage({
|
||||
child: owned.process,
|
||||
port,
|
||||
origin: initialUrl,
|
||||
deadline,
|
||||
});
|
||||
return { port, profile, process: owned.process, close };
|
||||
} catch (cause) {
|
||||
@@ -104,6 +113,67 @@ export async function startRuntimeFixtureBrowser(executable, initialUrl) {
|
||||
}
|
||||
}
|
||||
|
||||
async function waitForRuntimeFixturePage({ child, port, origin, deadline }) {
|
||||
const controller = new AbortController();
|
||||
let lastFailure;
|
||||
const onExit = () =>
|
||||
controller.abort(
|
||||
new Error(
|
||||
"Owned runtime fixture exited before CDP discovery became ready",
|
||||
{ cause: lastFailure },
|
||||
),
|
||||
);
|
||||
const onError = (cause) => controller.abort(cause);
|
||||
const timer = setTimeout(
|
||||
() =>
|
||||
controller.abort(
|
||||
new Error(
|
||||
"Owned runtime fixture page was not discoverable within its startup budget",
|
||||
{ cause: lastFailure },
|
||||
),
|
||||
),
|
||||
Math.max(1, deadline - performance.now()),
|
||||
);
|
||||
child.once("exit", onExit);
|
||||
child.once("error", onError);
|
||||
try {
|
||||
for (;;) {
|
||||
if (child.exitCode !== null || child.signalCode !== null) onExit();
|
||||
controller.signal.throwIfAborted();
|
||||
try {
|
||||
const discovery = await discoverCdpEndpoint(
|
||||
`http://127.0.0.1:${port}`,
|
||||
"list_browser_targets",
|
||||
controller.signal,
|
||||
);
|
||||
if (
|
||||
discovery.targets.some(
|
||||
(target) =>
|
||||
target.type === "page" &&
|
||||
(target.url === origin || target.url.startsWith(`${origin}/`)),
|
||||
)
|
||||
)
|
||||
return;
|
||||
} catch (cause) {
|
||||
controller.signal.throwIfAborted();
|
||||
// Only transient startup failures are retryable. Invalid protocol
|
||||
// responses and payload-limit failures must remain visible immediately.
|
||||
if (
|
||||
!(cause instanceof BrowserObservationError) ||
|
||||
!["timeout", "endpoint_unreachable"].includes(cause.reason)
|
||||
)
|
||||
throw cause;
|
||||
lastFailure = cause;
|
||||
}
|
||||
await new Promise((resolve) => setTimeout(resolve, 25));
|
||||
}
|
||||
} finally {
|
||||
clearTimeout(timer);
|
||||
child.removeListener("exit", onExit);
|
||||
child.removeListener("error", onError);
|
||||
}
|
||||
}
|
||||
|
||||
function createOwnedBrowserStop(supervisor, profile) {
|
||||
let connection;
|
||||
return async () => {
|
||||
|
||||
@@ -46,7 +46,7 @@ export async function verifyGhidraBoundaries(
|
||||
}
|
||||
let successfulCalls = 0;
|
||||
let rejectedCalls = 0;
|
||||
const call = async (name, args = {}) => {
|
||||
const checkedReply = async (name, args = {}) => {
|
||||
const reply = await client.callTool({ name, arguments: args }, options);
|
||||
const result = requireMcpOperationResult(reply, name);
|
||||
const validate = validators.get(name);
|
||||
@@ -66,8 +66,13 @@ export async function verifyGhidraBoundaries(
|
||||
`${name} omitted defined-string coverage`,
|
||||
);
|
||||
successfulCalls++;
|
||||
return result;
|
||||
return { reply, result };
|
||||
};
|
||||
const call = async (name, args = {}) =>
|
||||
(await checkedReply(name, args)).result;
|
||||
// Validated complete Evidence for checks outside the operation result.
|
||||
const evidence = async (name, args = {}) =>
|
||||
(await checkedReply(name, args)).reply.structuredContent;
|
||||
const invalid = async (name, args, diagnostic) => {
|
||||
const reply = await client.callTool({ name, arguments: args }, options);
|
||||
assert.equal(
|
||||
@@ -760,6 +765,7 @@ export async function verifyGhidraBoundaries(
|
||||
});
|
||||
await verifyGhidraNamespaceAnnotations({
|
||||
call,
|
||||
evidence,
|
||||
reject: invalid,
|
||||
target,
|
||||
entrypoint,
|
||||
|
||||
@@ -12,6 +12,7 @@ import { inspectVerifierGhidraInstallation } from "./ghidra-installation-inspect
|
||||
/** Exercise qualified annotation names against real demangled C++ ABI symbols. */
|
||||
export async function verifyGhidraNamespaceAnnotations({
|
||||
call,
|
||||
evidence,
|
||||
reject,
|
||||
target,
|
||||
entrypoint,
|
||||
@@ -62,7 +63,7 @@ export async function verifyGhidraNamespaceAnnotations({
|
||||
assert.ok(symbol, `Missing demangled fixture global ${namespace}::value`);
|
||||
return { namespace, address: symbol.address };
|
||||
});
|
||||
await verifyDuplicateAddressNames(call, procedures, names);
|
||||
await verifyDuplicateAddressNames(call, evidence, procedures, names);
|
||||
const ambiguous = await reject("procedure_address", { procedure: "same" });
|
||||
assert.equal(ambiguous.code, "invalid_request");
|
||||
for (const { address } of entries)
|
||||
@@ -136,6 +137,21 @@ export async function verifyGhidraNamespaceAnnotations({
|
||||
}
|
||||
await verifyAddressNames(call, [entries[0], entries[2]], "function");
|
||||
await verifyAddressNames(call, globals, "label");
|
||||
// Equal leaf names in different namespaces are distinct destinations.
|
||||
const shared = "rea_namespace_shared";
|
||||
assert.deepEqual(
|
||||
await call("set_addresses_names", {
|
||||
names: Object.fromEntries(
|
||||
entries.map(({ address }) => [address, shared]),
|
||||
),
|
||||
}),
|
||||
Object.fromEntries(entries.map(({ address }) => [address, true])),
|
||||
);
|
||||
for (const { namespace, address } of entries)
|
||||
assert.equal(
|
||||
await call("address_name", { address }),
|
||||
`${namespace}::${shared}`,
|
||||
);
|
||||
for (const { namespace, address } of entries) {
|
||||
assert.equal(
|
||||
await call("set_address_name", {
|
||||
@@ -183,7 +199,7 @@ export async function verifyGhidraNamespaceAnnotations({
|
||||
}
|
||||
}
|
||||
|
||||
async function verifyDuplicateAddressNames(call, procedures, names) {
|
||||
async function verifyDuplicateAddressNames(call, evidence, procedures, names) {
|
||||
const functionEntries = [
|
||||
"rea_duplicate_function_a",
|
||||
"rea_duplicate_function_b",
|
||||
@@ -231,32 +247,118 @@ async function verifyDuplicateAddressNames(call, procedures, names) {
|
||||
"A rejected duplicate leaves the other address unchanged",
|
||||
);
|
||||
|
||||
const readback = async (ownerName, message) => {
|
||||
assert.equal(
|
||||
await call("address_name", { address: owner.address }),
|
||||
ownerName,
|
||||
message,
|
||||
);
|
||||
assert.equal(
|
||||
await call("address_name", { address: duplicate.address }),
|
||||
duplicate.original,
|
||||
message,
|
||||
);
|
||||
};
|
||||
// Object member order is not an ordering choice: each batch is decided
|
||||
// from the names held before it, so both orders must agree.
|
||||
const batchName = `rea_batch_duplicate_${kind}`;
|
||||
const moved = `${name}_moved`;
|
||||
const competing = "duplicate_requested_name";
|
||||
const selected = "name_held_by_selected_address";
|
||||
for (const [requested, expected, ownerName, reasons, message] of [
|
||||
[
|
||||
{ [owner.address]: batchName, [duplicate.address]: batchName },
|
||||
{ [owner.address]: false, [duplicate.address]: false },
|
||||
name,
|
||||
{ [owner.address]: competing, [duplicate.address]: competing },
|
||||
"Competing requests for one name apply neither address",
|
||||
],
|
||||
[
|
||||
{ [owner.address]: duplicate.original, [duplicate.address]: name },
|
||||
{ [owner.address]: false, [duplicate.address]: false },
|
||||
name,
|
||||
{ [owner.address]: selected, [duplicate.address]: selected },
|
||||
"A swap between selected addresses is not ordered implicitly",
|
||||
],
|
||||
[
|
||||
{ [duplicate.address]: name, [owner.address]: moved },
|
||||
{ [owner.address]: true, [duplicate.address]: false },
|
||||
moved,
|
||||
{ [duplicate.address]: selected },
|
||||
"A name held by another selected address is not taken by member order",
|
||||
],
|
||||
[
|
||||
{ [duplicate.address]: name },
|
||||
{ [duplicate.address]: false },
|
||||
name,
|
||||
{ [duplicate.address]: "name_held_by_unselected_address" },
|
||||
"A name held by an unselected address is not moved",
|
||||
],
|
||||
]) {
|
||||
const facts = [];
|
||||
for (const ordered of [
|
||||
requested,
|
||||
Object.fromEntries(Object.entries(requested).reverse()),
|
||||
]) {
|
||||
const record = await evidence("set_addresses_names", {
|
||||
names: ordered,
|
||||
});
|
||||
assert.deepEqual(record.normalized_result, expected, message);
|
||||
const { failures, conflicts } = record.raw_result;
|
||||
assert.deepEqual(
|
||||
Object.fromEntries(
|
||||
Object.entries(failures).map(([key, { reason }]) => [key, reason]),
|
||||
),
|
||||
reasons,
|
||||
message,
|
||||
);
|
||||
// Each contested global name is reported once and names its requester.
|
||||
for (const [key, failure] of Object.entries(failures)) {
|
||||
const conflict = conflicts[failure.conflict];
|
||||
assert.equal(conflict.namespace, null, message);
|
||||
assert.equal(conflict.name, requested[key], message);
|
||||
assert.ok(conflict.requested_by.includes(failure.address), message);
|
||||
}
|
||||
assert.equal(
|
||||
conflicts.length,
|
||||
new Set(Object.keys(failures).map((key) => requested[key])).size,
|
||||
message,
|
||||
);
|
||||
facts.push(record.raw_result);
|
||||
await readback(ownerName, message);
|
||||
if (ownerName !== name)
|
||||
assert.equal(
|
||||
await call("set_address_name", { address: owner.address, name }),
|
||||
true,
|
||||
);
|
||||
}
|
||||
assert.deepEqual(facts[1], facts[0], message);
|
||||
}
|
||||
assert.deepEqual(
|
||||
await call("set_addresses_names", {
|
||||
names: {
|
||||
[owner.address]: batchName,
|
||||
[duplicate.address]: batchName,
|
||||
},
|
||||
}),
|
||||
{ [owner.address]: true, [duplicate.address]: false },
|
||||
(await call("list_names")).filter((item) => item.value === batchName),
|
||||
[],
|
||||
"A rejected competing name has no owner",
|
||||
);
|
||||
// An explicit later call may take a name released by an earlier one.
|
||||
assert.equal(
|
||||
await call("set_address_name", { address: owner.address, name: moved }),
|
||||
true,
|
||||
);
|
||||
assert.equal(
|
||||
await call("address_name", { address: owner.address }),
|
||||
batchName,
|
||||
);
|
||||
assert.equal(
|
||||
await call("address_name", { address: duplicate.address }),
|
||||
duplicate.original,
|
||||
"A failed batch member rolls back only its own address",
|
||||
);
|
||||
const owners = (await call("list_names")).filter(
|
||||
(item) => item.value === batchName && item.symbol.primary,
|
||||
);
|
||||
assert.deepEqual(
|
||||
owners.map(({ address }) => address),
|
||||
[owner.address],
|
||||
await call("set_address_name", { address: duplicate.address, name }),
|
||||
true,
|
||||
);
|
||||
// Ghidra symbol names are case-sensitive, so case variants do not compete.
|
||||
const cased = {
|
||||
[owner.address]: `${batchName}_Case`,
|
||||
[duplicate.address]: `${batchName}_case`,
|
||||
};
|
||||
assert.deepEqual(await call("set_addresses_names", { names: cased }), {
|
||||
[owner.address]: true,
|
||||
[duplicate.address]: true,
|
||||
});
|
||||
for (const [address, expected] of Object.entries(cased))
|
||||
assert.equal(await call("address_name", { address }), expected);
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
+27
-12
@@ -15,8 +15,8 @@ def fail(code, **details):
|
||||
raise SystemExit(1)
|
||||
|
||||
|
||||
class BridgeResult(unittest.TextTestResult):
|
||||
"""Record non-skipped test and subtest outcomes for required families."""
|
||||
class ExecutedResult(unittest.TextTestResult):
|
||||
"""Record tests with a non-skipped test or subtest outcome."""
|
||||
|
||||
def __init__(self, *args, **kwargs):
|
||||
super().__init__(*args, **kwargs)
|
||||
@@ -25,10 +25,7 @@ class BridgeResult(unittest.TextTestResult):
|
||||
self.has_subtests = False
|
||||
|
||||
def record(self, test):
|
||||
parts = test.id().split(".")
|
||||
if len(parts) > 2 and parts[0] == "bridge":
|
||||
if parts[1] in REQUIRED_BRIDGE_FAMILIES:
|
||||
self.completed.add(parts[1])
|
||||
self.completed.add(test.id())
|
||||
|
||||
def startTest(self, test):
|
||||
self.active_test = test
|
||||
@@ -65,11 +62,16 @@ def run_bridge(modules):
|
||||
# The script directory is sys.path[0]; imports must resolve from the checkout.
|
||||
sys.path.insert(0, str(root))
|
||||
suite = unittest.defaultTestLoader.loadTestsFromNames(modules)
|
||||
result = unittest.TextTestRunner(verbosity=2, resultclass=BridgeResult).run(suite)
|
||||
result = unittest.TextTestRunner(verbosity=2, resultclass=ExecutedResult).run(suite)
|
||||
completed_families = {
|
||||
parts[1]
|
||||
for name in result.completed
|
||||
if len(parts := name.split(".")) > 2 and parts[0] == "bridge"
|
||||
}
|
||||
missing = [
|
||||
family
|
||||
for family in REQUIRED_BRIDGE_FAMILIES
|
||||
if family not in result.completed
|
||||
if family not in completed_families
|
||||
]
|
||||
if missing:
|
||||
fail("no_executed_bridge_tests", families=missing)
|
||||
@@ -77,6 +79,19 @@ def run_bridge(modules):
|
||||
raise SystemExit(1)
|
||||
|
||||
|
||||
def run_analysis_efficiency():
|
||||
# Match the checkout-root import path of `python -m unittest discover`.
|
||||
sys.path.insert(0, str(root))
|
||||
suite = unittest.defaultTestLoader.discover(
|
||||
str(root / "tests/support/analysis-efficiency"), pattern="test_*.py"
|
||||
)
|
||||
result = unittest.TextTestRunner(verbosity=2, resultclass=ExecutedResult).run(suite)
|
||||
if not result.wasSuccessful():
|
||||
raise SystemExit(1)
|
||||
if not result.completed:
|
||||
fail("no_executed_analysis_efficiency_tests")
|
||||
|
||||
|
||||
bridge_root = root / "bridge"
|
||||
bridge_tests = sorted(
|
||||
path for path in bridge_root.rglob("test_*.py") if path.is_file()
|
||||
@@ -95,6 +110,9 @@ modules = [
|
||||
if sys.argv[1:] == ["--bridge"]:
|
||||
run_bridge(modules)
|
||||
raise SystemExit(0)
|
||||
if sys.argv[1:] == ["--analysis-efficiency"]:
|
||||
run_analysis_efficiency()
|
||||
raise SystemExit(0)
|
||||
if sys.argv[1:]:
|
||||
fail("invalid_python_test_arguments")
|
||||
bridge_result = subprocess.run(
|
||||
@@ -103,10 +121,7 @@ bridge_result = subprocess.run(
|
||||
if bridge_result.returncode:
|
||||
sys.exit(bridge_result.returncode)
|
||||
result = subprocess.run(
|
||||
[
|
||||
sys.executable, "-m", "unittest", "discover",
|
||||
"-s", "tests/support/analysis-efficiency", "-p", "test_*.py", "-v",
|
||||
],
|
||||
[sys.executable, __file__, "--analysis-efficiency"],
|
||||
cwd=root,
|
||||
check=False,
|
||||
)
|
||||
|
||||
@@ -2,7 +2,14 @@
|
||||
import assert from "node:assert/strict";
|
||||
import { execFile } from "node:child_process";
|
||||
import { createHash } from "node:crypto";
|
||||
import { mkdtemp, mkdir, readFile, rm, writeFile } from "node:fs/promises";
|
||||
import {
|
||||
mkdtemp,
|
||||
mkdir,
|
||||
readFile,
|
||||
realpath,
|
||||
rm,
|
||||
writeFile,
|
||||
} from "node:fs/promises";
|
||||
import { tmpdir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
import { fileURLToPath } from "node:url";
|
||||
@@ -18,6 +25,8 @@ import {
|
||||
assertMipsProbe,
|
||||
assertMipsGlobal,
|
||||
assertMipsSoftFloatStorage,
|
||||
assertMipsCarvedTargetRefusal,
|
||||
carveMipsElfBeforeSectionTable,
|
||||
} from "../tests/conformance/ghidra/mips-checks.mjs";
|
||||
import { parseEvidence } from "../dist/domain/evidence.js";
|
||||
import { parseAnalysisSnapshot } from "../dist/domain/analysisSnapshot.js";
|
||||
@@ -122,16 +131,6 @@ async function verifySoftFloat(call, symbols, byteOrder) {
|
||||
);
|
||||
return { storage: boundaries };
|
||||
}
|
||||
function assertIncompleteMipsInspection(error) {
|
||||
assert.equal(error.code, "capability_unavailable");
|
||||
assert.equal(error.details.selection_reason, "target_unsupported");
|
||||
const rejection = error.details.rejections.find(
|
||||
(candidate) => candidate.provider_id === "ghidra",
|
||||
);
|
||||
assert.equal(rejection?.code, "architecture_unsupported");
|
||||
assert.match(rejection.reason, /ABI declaration inspection is incomplete/u);
|
||||
assert.match(rejection.reason, /section header table/u);
|
||||
}
|
||||
|
||||
const reports = [];
|
||||
let primaryFailure;
|
||||
@@ -617,45 +616,20 @@ try {
|
||||
// PSP keeps its independent Allegrex/A-B-A checks; carved PSP is not verified.
|
||||
let carved;
|
||||
if (!psp) {
|
||||
// A program-header ABI record survives carving, but GNU declarations in
|
||||
// uncaptured section headers remain unknown: both adapters must refuse.
|
||||
const read16 = (offset) =>
|
||||
byteOrder === "little"
|
||||
? sourceBytes.readUInt16LE(offset)
|
||||
: sourceBytes.readUInt16BE(offset);
|
||||
const read32 = (offset) =>
|
||||
byteOrder === "little"
|
||||
? sourceBytes.readUInt32LE(offset)
|
||||
: sourceBytes.readUInt32BE(offset);
|
||||
const sectionOffset = read32(32);
|
||||
const cut = sectionOffset - 1;
|
||||
const programOffset = read32(28);
|
||||
const programCount = read16(44);
|
||||
assert.equal(read16(42), 32);
|
||||
assert.ok(cut >= programOffset + programCount * 32);
|
||||
for (let i = 0; i < programCount; i += 1) {
|
||||
const offset = programOffset + i * 32;
|
||||
assert.ok(read32(offset + 4) + read32(offset + 16) <= cut);
|
||||
}
|
||||
const carvedBytes = sourceBytes.subarray(0, cut);
|
||||
// The carved copy keeps every program header and payload but declares
|
||||
// a section table it did not capture. Both adapters refuse it when
|
||||
// opening the target; it is not analyzed as a smaller program.
|
||||
const carve = carveMipsElfBeforeSectionTable(sourceBytes, byteOrder);
|
||||
const carvedPath = join(
|
||||
workspace,
|
||||
`${byteOrder}-${floatAbi}-carved.elf`,
|
||||
);
|
||||
const carvedSha256 = createHash("sha256")
|
||||
.update(carvedBytes)
|
||||
.update(carve.bytes)
|
||||
.digest("hex");
|
||||
await writeFile(carvedPath, carvedBytes);
|
||||
const carvedResolved = await parseBinaryTarget(carvedPath);
|
||||
if (!carvedResolved.ok) throw carvedResolved.error;
|
||||
assert.deepEqual(
|
||||
carvedResolved.value.mips.abiFlags,
|
||||
independent.mips.abiFlags,
|
||||
);
|
||||
assert.match(
|
||||
carvedResolved.value.mips.limitations.join(" "),
|
||||
/section header table/u,
|
||||
);
|
||||
await writeFile(carvedPath, carve.bytes);
|
||||
// Target errors name the canonical path that was opened.
|
||||
const openedPath = await realpath(carvedPath);
|
||||
const carvedError = requireMcpToolError(
|
||||
await client.callTool(
|
||||
{
|
||||
@@ -665,7 +639,7 @@ try {
|
||||
{ timeout: 60000 },
|
||||
),
|
||||
);
|
||||
assertIncompleteMipsInspection(carvedError);
|
||||
assertMipsCarvedTargetRefusal(carvedError, openedPath, carve.reason);
|
||||
let carvedCliError;
|
||||
try {
|
||||
await execute(
|
||||
@@ -688,16 +662,20 @@ try {
|
||||
carvedCliError &&
|
||||
Number.isInteger(carvedCliError.code) &&
|
||||
carvedCliError.code !== 0,
|
||||
"Expected an incomplete-ABI refusal, not a timeout or success",
|
||||
"Expected a target-opening refusal, not a timeout or success",
|
||||
);
|
||||
assertIncompleteMipsInspection(JSON.parse(carvedCliError.stdout));
|
||||
assert.deepEqual(await readFile(carvedPath), carvedBytes);
|
||||
assertMipsCarvedTargetRefusal(
|
||||
JSON.parse(carvedCliError.stdout),
|
||||
openedPath,
|
||||
carve.reason,
|
||||
);
|
||||
assert.deepEqual(await readFile(carvedPath), carve.bytes);
|
||||
carved = {
|
||||
sha256: carvedSha256,
|
||||
source_bytes: carvedBytes.length,
|
||||
section_header_offset: sectionOffset,
|
||||
status: "expected-incomplete-abi-refusal",
|
||||
limitations: carvedResolved.value.mips.limitations,
|
||||
source_bytes: carve.bytes.length,
|
||||
section_header_offset: carve.sectionOffset,
|
||||
status: "expected-target-refusal",
|
||||
reason: carve.reason,
|
||||
cli_exit: carvedCliError.code,
|
||||
mcp_error: carvedError,
|
||||
};
|
||||
|
||||
@@ -23,7 +23,17 @@ import { parseAnalysisSnapshot } from "../dist/domain/analysisSnapshot.js";
|
||||
import { parseEvidence } from "../dist/domain/evidence.js";
|
||||
import { functionDossierSchema } from "../dist/domain/native/functionDossier.js";
|
||||
import { inspectVerifierGhidraInstallation } from "./lib/ghidra-installation-inspection.mjs";
|
||||
import { buildRiscvFixture } from "../tests/conformance/ghidra/riscv-fixture.mjs";
|
||||
import {
|
||||
buildRiscvFixture,
|
||||
stripRiscvFixture,
|
||||
} from "../tests/conformance/ghidra/riscv-fixture.mjs";
|
||||
import {
|
||||
RISCV_ILP32_ABI,
|
||||
assertRiscvIlp32CallSites,
|
||||
assertRiscvIlp32Caller,
|
||||
assertRiscvIlp32Storage,
|
||||
inspectRiscvCallSites,
|
||||
} from "../tests/conformance/ghidra/riscv-checks.mjs";
|
||||
import {
|
||||
mcpTextValue,
|
||||
requireMcpLifecycleResult,
|
||||
@@ -82,6 +92,29 @@ transport.stderr?.on("data", (chunk) => {
|
||||
stderr = (stderr + chunk.toString()).slice(-65536);
|
||||
});
|
||||
|
||||
/** The profile a fixture's lane must commit, from independent declarations. */
|
||||
const expectedLane = (fixture) => {
|
||||
const soft = fixture.riscv.flags === 1;
|
||||
return {
|
||||
lane: `elf${fixture.riscv.elfClass}-exec-le-${soft ? "soft" : "double"}-float-generic-v1`,
|
||||
callingConvention: soft ? "ilp32-prototype-v1" : undefined,
|
||||
riscvElf: {
|
||||
elf_class: fixture.riscv.elfClass,
|
||||
byte_order: "little",
|
||||
type: 2,
|
||||
flags: fixture.riscv.flags,
|
||||
float_abi_double: !soft,
|
||||
...(soft ? { float_abi: "soft", arch: fixture.riscv.isa.arch } : {}),
|
||||
rvc_declared: true,
|
||||
rve_declared: false,
|
||||
tso_declared: false,
|
||||
},
|
||||
};
|
||||
};
|
||||
|
||||
const OUTCOME =
|
||||
/RISC-V calling-convention outcome: [1-9]\d* functions and (\d+) function definitions use __rea_ilp32\./u;
|
||||
|
||||
const reports = [];
|
||||
let failure;
|
||||
try {
|
||||
@@ -101,41 +134,55 @@ try {
|
||||
: undefined;
|
||||
validators.set(tool.name, existing ?? ajv.compile(tool.outputSchema));
|
||||
}
|
||||
for (const bits of [32, 64]) {
|
||||
const fixture = await buildRiscvFixture(workspace, bits);
|
||||
|
||||
const typedSoft = await buildRiscvFixture(workspace, 32, "soft");
|
||||
const callSites = await inspectRiscvCallSites(typedSoft.path);
|
||||
assertRiscvIlp32CallSites(callSites.calls);
|
||||
const fixtures = [
|
||||
await buildRiscvFixture(workspace, 32),
|
||||
await buildRiscvFixture(workspace, 64),
|
||||
typedSoft,
|
||||
await stripRiscvFixture(workspace, typedSoft),
|
||||
];
|
||||
for (const fixture of fixtures) {
|
||||
const bits = fixture.riscv.elfClass;
|
||||
const soft = fixture.riscv.flags === 1;
|
||||
const stripped = fixture.stripped === true;
|
||||
const expected = expectedLane(fixture);
|
||||
const resolved = await parseBinaryTarget(fixture.path);
|
||||
if (!resolved.ok) throw resolved.error;
|
||||
assert.equal(resolved.value.architecture, "riscv");
|
||||
assert.equal(resolved.value.sha256, fixture.sha256);
|
||||
assert.deepEqual(resolved.value.riscv, fixture.riscv);
|
||||
const expectedLanguage = `RISCV:LE:${bits}:default`;
|
||||
const assertProfile = (profile) => {
|
||||
const assertProfile = (profile, want = expected) => {
|
||||
assert.equal(profile.provider.id, "ghidra");
|
||||
assert.equal(profile.provider.version, installation.providerVersion);
|
||||
const parameters = profile.parameters;
|
||||
assert.equal(parameters.language_id, expectedLanguage);
|
||||
assert.equal(parameters.compiler_spec_id, "gcc");
|
||||
assert.equal(parameters.language_selection, "provider-pinned-v1");
|
||||
assert.equal(
|
||||
parameters.riscv_support_lane,
|
||||
`elf${bits}-exec-le-double-float-generic-v1`,
|
||||
);
|
||||
assert.deepEqual(parameters.riscv_elf, {
|
||||
elf_class: bits,
|
||||
byte_order: "little",
|
||||
type: 2,
|
||||
flags: 5,
|
||||
float_abi_double: true,
|
||||
rvc_declared: true,
|
||||
rve_declared: false,
|
||||
tso_declared: false,
|
||||
});
|
||||
assert.equal(parameters.riscv_support_lane, want.lane);
|
||||
assert.equal(parameters.riscv_calling_convention, want.callingConvention);
|
||||
assert.deepEqual(parameters.riscv_elf, want.riscvElf);
|
||||
};
|
||||
const assertEvidence = (evidence) => {
|
||||
assert.equal(evidence.provider.id, "ghidra");
|
||||
assert.equal(evidence.subject.architecture, "riscv");
|
||||
assert.equal(evidence.subject.digest.sha256, fixture.sha256);
|
||||
assertProfile(evidence.analysis_profile);
|
||||
// Composed workflows such as inspect_native_api commit the Ghidra
|
||||
// profile as their upstream profile. The residual-unknown registry
|
||||
// records what they could not establish, such as the types of a
|
||||
// stripped function's parameters, without a profile of its own.
|
||||
if (evidence.provider.id === "rea-workflow")
|
||||
assertProfile(
|
||||
evidence.analysis_profile.parameters.upstream_analysis_profile,
|
||||
);
|
||||
else if (evidence.provider.id === "rea-unknown-registry")
|
||||
assert.equal(evidence.analysis_profile, null);
|
||||
else {
|
||||
assert.equal(evidence.provider.id, "ghidra");
|
||||
assertProfile(evidence.analysis_profile);
|
||||
}
|
||||
};
|
||||
const call = async (name, args = {}) => {
|
||||
const reply = await client.callTool(
|
||||
@@ -176,13 +223,32 @@ try {
|
||||
assert.equal(source.modified_sha256, fixture.sha256);
|
||||
}
|
||||
const procedures = await call("list_procedures");
|
||||
for (const name of [
|
||||
"rea_riscv_entry",
|
||||
"rea_riscv_leaf",
|
||||
"rea_riscv_float",
|
||||
"rea_riscv_probe",
|
||||
]) {
|
||||
const procedure = procedures.find((value) => value.value === name);
|
||||
const names = soft
|
||||
? [
|
||||
"rea_riscv_entry",
|
||||
"rea_riscv_probe",
|
||||
"rea_riscv_isa_probe",
|
||||
"rea_riscv_abi_caller",
|
||||
...Object.keys(RISCV_ILP32_ABI),
|
||||
]
|
||||
: [
|
||||
"rea_riscv_entry",
|
||||
"rea_riscv_leaf",
|
||||
"rea_riscv_float",
|
||||
"rea_riscv_probe",
|
||||
];
|
||||
for (const name of [...names, "rea_riscv_global"])
|
||||
assert.ok(
|
||||
fixture.symbols[name],
|
||||
`${name} missing from the readelf symbols`,
|
||||
);
|
||||
for (const name of names) {
|
||||
// A stripped image has no names; the typed twin's addresses locate it.
|
||||
const procedure = procedures.find((value) =>
|
||||
stripped
|
||||
? BigInt(value.address) === BigInt(fixture.symbols[name])
|
||||
: value.value === name,
|
||||
);
|
||||
assert.ok(procedure, `${name} missing from Ghidra inventory`);
|
||||
assert.equal(BigInt(procedure.address), BigInt(fixture.symbols[name]));
|
||||
}
|
||||
@@ -203,6 +269,22 @@ try {
|
||||
.toString("hex"),
|
||||
probe.bytes,
|
||||
);
|
||||
const isaProbe = [];
|
||||
if (soft) {
|
||||
// ESP32-C3's -march declares Zifencei and Zicsr; both must decode.
|
||||
for (const [offset, bytes, mnemonic] of [
|
||||
[0n, "0f100000", /^fence\.i$/iu],
|
||||
[4n, "f32200b0", /^csrr/iu],
|
||||
]) {
|
||||
const instruction = await call("inspect_native_instruction", {
|
||||
address: `0x${(BigInt(fixture.symbols.rea_riscv_isa_probe) + offset).toString(16)}`,
|
||||
});
|
||||
assert.equal(instruction.status, "decoded");
|
||||
assert.equal(instruction.bytes, bytes);
|
||||
assert.match(instruction.mnemonic, mnemonic);
|
||||
isaProbe.push(instruction.mnemonic);
|
||||
}
|
||||
}
|
||||
const global = await call("read_bytes", {
|
||||
address: fixture.symbols.rea_riscv_global,
|
||||
length: 4,
|
||||
@@ -211,11 +293,14 @@ try {
|
||||
const callees = await call("procedure_callees", {
|
||||
procedure: fixture.symbols.rea_riscv_entry,
|
||||
});
|
||||
for (const name of ["rea_riscv_leaf", "rea_riscv_probe"])
|
||||
for (const name of soft
|
||||
? ["rea_riscv_abi_caller", "rea_riscv_isa_probe", "rea_riscv_probe"]
|
||||
: ["rea_riscv_leaf", "rea_riscv_probe"])
|
||||
assert.ok(
|
||||
callees.some(
|
||||
(address) => BigInt(address) === BigInt(fixture.symbols[name]),
|
||||
),
|
||||
`${name} missing from the entry's callees`,
|
||||
);
|
||||
const dossier = functionDossierSchema.parse(
|
||||
await call("analyze_function", {
|
||||
@@ -224,60 +309,50 @@ try {
|
||||
}),
|
||||
);
|
||||
assert.ok(dossier.pseudocode.trim().length > 0);
|
||||
const floatCode = await call("procedure_pseudo_code", {
|
||||
procedure: fixture.symbols.rea_riscv_float,
|
||||
});
|
||||
assert.ok(typeof floatCode === "string" && floatCode.trim().length > 0);
|
||||
assert.match(floatCode, /\bdouble\b/u);
|
||||
|
||||
const refusals = [];
|
||||
for (const [flags, reason] of [
|
||||
[1, "EF_RISCV_FLOAT_ABI_DOUBLE"],
|
||||
[0x80000005, "(0x80000000)"],
|
||||
]) {
|
||||
// Header-mutated copies test admission; they are not new compiler claims.
|
||||
const rejectedPath = join(
|
||||
workspace,
|
||||
`rv${bits}-flags-${flags.toString(16)}.elf`,
|
||||
);
|
||||
const rejectedBytes = Buffer.from(sourceBytes);
|
||||
rejectedBytes.writeUInt32LE(flags, bits === 32 ? 36 : 48);
|
||||
await writeFile(rejectedPath, rejectedBytes);
|
||||
const reply = await client.callTool(
|
||||
{
|
||||
name: "open_binary",
|
||||
arguments: { path: rejectedPath, provider_id: "ghidra" },
|
||||
},
|
||||
{ timeout: 60000 },
|
||||
);
|
||||
const error = requireMcpToolError(reply);
|
||||
assert.ok(JSON.stringify(error).includes(reason), JSON.stringify(error));
|
||||
await assert.rejects(
|
||||
execute(
|
||||
process.execPath,
|
||||
[
|
||||
entrypoint,
|
||||
"function",
|
||||
rejectedPath,
|
||||
"rea_riscv_entry",
|
||||
"--provider",
|
||||
"ghidra",
|
||||
"--json",
|
||||
],
|
||||
{
|
||||
env,
|
||||
timeout: 60000,
|
||||
maxBuffer: 1024 * 1024,
|
||||
},
|
||||
),
|
||||
(cause) =>
|
||||
Number.isInteger(cause.code) &&
|
||||
cause.code !== 0 &&
|
||||
`${cause.stdout}\n${cause.stderr}`.includes(reason),
|
||||
);
|
||||
refusals.push({ flags, reason });
|
||||
if (!stripped) {
|
||||
const floatCode = await call("procedure_pseudo_code", {
|
||||
procedure:
|
||||
fixture.symbols[soft ? "rea_riscv_abi_wide" : "rea_riscv_float"],
|
||||
});
|
||||
assert.ok(typeof floatCode === "string" && floatCode.trim().length > 0);
|
||||
assert.match(floatCode, /\bdouble\b/u);
|
||||
}
|
||||
const snapshotPath = join(workspace, `rv${bits}.snapshot.json`);
|
||||
let abi;
|
||||
if (soft) {
|
||||
const boundaries = {};
|
||||
for (const name of Object.keys(RISCV_ILP32_ABI))
|
||||
boundaries[name] = (
|
||||
await call("inspect_native_api", { procedure: fixture.symbols[name] })
|
||||
).boundary;
|
||||
assertRiscvIlp32Storage(boundaries, { stripped });
|
||||
const caller = functionDossierSchema.parse(
|
||||
await call("analyze_function", {
|
||||
procedure: fixture.symbols.rea_riscv_abi_caller,
|
||||
detail: "complete",
|
||||
}),
|
||||
);
|
||||
assertRiscvIlp32Caller(caller.pseudocode, fixture.symbols, {
|
||||
stripped,
|
||||
});
|
||||
abi = {
|
||||
storage: Object.fromEntries(
|
||||
Object.entries(boundaries).map(([name, boundary]) => [
|
||||
name,
|
||||
{
|
||||
return: boundary.return_type.storage,
|
||||
parameters: boundary.parameters.map(
|
||||
(parameter) => parameter.storage,
|
||||
),
|
||||
},
|
||||
]),
|
||||
),
|
||||
};
|
||||
}
|
||||
|
||||
const snapshotPath = join(
|
||||
workspace,
|
||||
`${fixture.path.split("/").at(-1)}.snapshot.json`,
|
||||
);
|
||||
const saved = await call("close_binary", { snapshot_path: snapshotPath });
|
||||
assert.ok(saved.evidence_records > 0);
|
||||
const snapshot = parseAnalysisSnapshot(
|
||||
@@ -288,6 +363,19 @@ try {
|
||||
assertProfile(snapshot.binding.analysis_profile);
|
||||
for (const record of snapshot.evidence_bundle.records)
|
||||
assertEvidence(parseEvidence(record));
|
||||
const outcomes = snapshot.evidence_bundle.records
|
||||
.flatMap((record) => parseEvidence(record).limitations)
|
||||
.map((limitation) => OUTCOME.exec(limitation))
|
||||
.filter((match) => match !== null);
|
||||
assert.equal(
|
||||
outcomes.length > 0,
|
||||
soft,
|
||||
"ilp32 prototype outcome missing from, or unexpected in, retained Evidence",
|
||||
);
|
||||
// DWARF supplies the typed build's function definitions; stripping removes them.
|
||||
if (soft)
|
||||
for (const match of outcomes)
|
||||
assert.equal(Number(match[1]) > 0, !stripped, match[0]);
|
||||
await call("open_binary", {
|
||||
path: fixture.path,
|
||||
provider_id: "ghidra",
|
||||
@@ -297,8 +385,11 @@ try {
|
||||
await call("close_binary");
|
||||
|
||||
for (const [command, arguments_] of [
|
||||
["search", ["rea_riscv_", "--kind", "procedures"]],
|
||||
["function", ["rea_riscv_entry"]],
|
||||
...(stripped ? [] : [["search", ["rea_riscv_", "--kind", "procedures"]]]),
|
||||
[
|
||||
"function",
|
||||
[stripped ? fixture.symbols.rea_riscv_entry : "rea_riscv_entry"],
|
||||
],
|
||||
]) {
|
||||
const cli = await execute(
|
||||
process.execPath,
|
||||
@@ -316,12 +407,7 @@ try {
|
||||
const evidence = parseEvidence(JSON.parse(cli.stdout));
|
||||
assertEvidence(evidence);
|
||||
if (command === "search")
|
||||
for (const name of [
|
||||
"rea_riscv_entry",
|
||||
"rea_riscv_leaf",
|
||||
"rea_riscv_float",
|
||||
"rea_riscv_probe",
|
||||
])
|
||||
for (const name of names)
|
||||
assert.ok(
|
||||
evidence.normalized_result.some((value) => value.value === name),
|
||||
);
|
||||
@@ -333,6 +419,92 @@ try {
|
||||
assert.ok(cliDossier.pseudocode.trim().length > 0);
|
||||
}
|
||||
}
|
||||
const headers = [];
|
||||
// Header-mutated copies test admission; they are not new compiler claims.
|
||||
for (const [flags, outcome] of stripped
|
||||
? []
|
||||
: soft
|
||||
? [
|
||||
[0x3, "single-float ABI (ilp32f)"],
|
||||
[0x80000001, "(0x80000000)"],
|
||||
]
|
||||
: bits === 32
|
||||
? [
|
||||
[0x1, "admitted"],
|
||||
[0x80000005, "(0x80000000)"],
|
||||
]
|
||||
: [
|
||||
[0x1, "soft-float ABI (lp64)"],
|
||||
[0x80000005, "(0x80000000)"],
|
||||
]) {
|
||||
const mutatedPath = join(
|
||||
workspace,
|
||||
`rv${bits}${soft ? "-soft" : ""}-flags-${flags.toString(16)}.elf`,
|
||||
);
|
||||
const mutatedBytes = Buffer.from(sourceBytes);
|
||||
mutatedBytes.writeUInt32LE(flags, bits === 32 ? 36 : 48);
|
||||
await writeFile(mutatedPath, mutatedBytes);
|
||||
const reply = await client.callTool(
|
||||
{
|
||||
name: "open_binary",
|
||||
arguments: { path: mutatedPath, provider_id: "ghidra" },
|
||||
},
|
||||
{ timeout: 240000 },
|
||||
);
|
||||
const cliArguments = [
|
||||
entrypoint,
|
||||
"function",
|
||||
mutatedPath,
|
||||
"rea_riscv_entry",
|
||||
"--provider",
|
||||
"ghidra",
|
||||
"--json",
|
||||
];
|
||||
const cliOptions = { env, timeout: 240000, maxBuffer: 16 * 1024 * 1024 };
|
||||
if (outcome === "admitted") {
|
||||
// The double-float image now declares ilp32: refused before, admitted
|
||||
// now through the soft-float lane with the image's own ISA string.
|
||||
assert.equal(
|
||||
requireMcpLifecycleResult(reply, "open_binary").sha256,
|
||||
createHash("sha256").update(mutatedBytes).digest("hex"),
|
||||
);
|
||||
requireMcpLifecycleResult(
|
||||
await client.callTool(
|
||||
{ name: "close_binary", arguments: {} },
|
||||
{ timeout: 240000 },
|
||||
),
|
||||
"close_binary",
|
||||
);
|
||||
const want = {
|
||||
lane: "elf32-exec-le-soft-float-generic-v1",
|
||||
callingConvention: "ilp32-prototype-v1",
|
||||
riscvElf: {
|
||||
...expected.riscvElf,
|
||||
flags,
|
||||
float_abi_double: false,
|
||||
float_abi: "soft",
|
||||
arch: fixture.riscv.isa.arch,
|
||||
},
|
||||
};
|
||||
const cli = await execute(process.execPath, cliArguments, cliOptions);
|
||||
assertProfile(
|
||||
parseEvidence(JSON.parse(cli.stdout)).analysis_profile,
|
||||
want,
|
||||
);
|
||||
headers.push({ flags, outcome, lane: want.lane });
|
||||
continue;
|
||||
}
|
||||
const error = requireMcpToolError(reply);
|
||||
assert.ok(JSON.stringify(error).includes(outcome), JSON.stringify(error));
|
||||
await assert.rejects(
|
||||
execute(process.execPath, cliArguments, cliOptions),
|
||||
(cause) =>
|
||||
Number.isInteger(cause.code) &&
|
||||
cause.code !== 0 &&
|
||||
`${cause.stdout}\n${cause.stderr}`.includes(outcome),
|
||||
);
|
||||
headers.push({ flags, outcome: "refused", reason: outcome });
|
||||
}
|
||||
assert.equal(
|
||||
createHash("sha256")
|
||||
.update(await readFile(fixture.path))
|
||||
@@ -340,16 +512,38 @@ try {
|
||||
fixture.sha256,
|
||||
);
|
||||
reports.push({
|
||||
...fixture,
|
||||
path: fixture.path,
|
||||
sha256: fixture.sha256,
|
||||
riscv: fixture.riscv,
|
||||
stripped,
|
||||
toolchain: fixture.toolchain,
|
||||
sources: fixture.sources,
|
||||
language: expectedLanguage,
|
||||
lane: expected.lane,
|
||||
calling_convention: expected.callingConvention ?? null,
|
||||
procedures: procedures.length,
|
||||
refusals,
|
||||
headers,
|
||||
...(isaProbe.length === 0 ? {} : { isa_probe: isaProbe }),
|
||||
...(abi === undefined
|
||||
? {}
|
||||
: {
|
||||
abi,
|
||||
prototype_outcome: outcomes.map((match) => match[0]),
|
||||
}),
|
||||
cli: true,
|
||||
stdio_mcp: true,
|
||||
snapshot_replay: true,
|
||||
original_unchanged: true,
|
||||
});
|
||||
}
|
||||
reports.push({
|
||||
call_sites: {
|
||||
fixture: typedSoft.path,
|
||||
command: callSites.command,
|
||||
version: callSites.version,
|
||||
calls: callSites.calls.map((value) => value.target),
|
||||
},
|
||||
});
|
||||
} catch (cause) {
|
||||
failure = cause;
|
||||
}
|
||||
|
||||
@@ -59,18 +59,17 @@ const source = fileURLToPath(
|
||||
const literalSource = fileURLToPath(
|
||||
new URL("../tests/conformance/swift/string-literals.swift", import.meta.url),
|
||||
);
|
||||
const unresolvedJumpSource = fileURLToPath(
|
||||
new URL(
|
||||
"../tests/conformance/swift/unresolved-string-jump.s",
|
||||
import.meta.url,
|
||||
),
|
||||
);
|
||||
const binary = join(root, "swift-inventory");
|
||||
const literalBinary = join(root, "swift-literals");
|
||||
const literalAssembly = fileURLToPath(
|
||||
new URL("../tests/conformance/swift/decoder-safeguards.S", import.meta.url),
|
||||
// swiftc rejects assembly inputs, so each source is assembled to an object first.
|
||||
const literalObjects = ["decoder-safeguards.S", "unresolved-string-jump.s"].map(
|
||||
(name) => ({
|
||||
source: fileURLToPath(
|
||||
new URL(`../tests/conformance/swift/${name}`, import.meta.url),
|
||||
),
|
||||
object: join(root, name.replace(/\.[sS]$/u, ".o")),
|
||||
}),
|
||||
);
|
||||
const literalObject = join(root, "decoder-safeguards.o");
|
||||
const reports = [];
|
||||
let literalReport;
|
||||
let decoderCleaned = true;
|
||||
@@ -81,20 +80,21 @@ try {
|
||||
await mkdir(join(root, "copied"));
|
||||
const copied = join(root, "copied", "swift-inventory");
|
||||
await copyFile(binary, copied);
|
||||
await execute(
|
||||
"xcrun",
|
||||
[
|
||||
"clang",
|
||||
"-target",
|
||||
"arm64-apple-macosx13.0",
|
||||
"-c",
|
||||
literalAssembly,
|
||||
"-o",
|
||||
literalObject,
|
||||
],
|
||||
{ timeout: 60_000 },
|
||||
);
|
||||
// Decoding covers AArch64 register pairs, so build that slice on any Mac host.
|
||||
for (const { source: assembly, object } of literalObjects)
|
||||
await execute(
|
||||
"xcrun",
|
||||
[
|
||||
"clang",
|
||||
"-target",
|
||||
"arm64-apple-macosx13.0",
|
||||
"-c",
|
||||
assembly,
|
||||
"-o",
|
||||
object,
|
||||
],
|
||||
{ timeout: 60_000 },
|
||||
);
|
||||
await execute(
|
||||
"swiftc",
|
||||
[
|
||||
@@ -102,8 +102,7 @@ try {
|
||||
"-target",
|
||||
"arm64-apple-macosx13.0",
|
||||
literalSource,
|
||||
literalObject,
|
||||
unresolvedJumpSource,
|
||||
...literalObjects.map(({ object }) => object),
|
||||
"-o",
|
||||
literalBinary,
|
||||
],
|
||||
|
||||
@@ -0,0 +1,136 @@
|
||||
import assert from "node:assert/strict";
|
||||
import { access, readFile } from "node:fs/promises";
|
||||
import { join } from "node:path";
|
||||
import {
|
||||
execute,
|
||||
failedCommand,
|
||||
options,
|
||||
source,
|
||||
spell,
|
||||
tool,
|
||||
} from "./esp-image-tools.mjs";
|
||||
|
||||
// Optional verification tools only; the runtime reader starts no tool.
|
||||
const idf = process.env.REA_ESP_IDF_PATH;
|
||||
const python = process.env.REA_ESPTOOL_PYTHON ?? "python3";
|
||||
const generator = () =>
|
||||
join(idf, "components", "partition_table", "gen_esp32part.py");
|
||||
const environment = () => ({ ...process.env, IDF_PATH: idf });
|
||||
|
||||
/** Check the ESP-IDF checkout before any work and report its version. */
|
||||
export const checkEspIdf = async () => {
|
||||
if (idf === undefined)
|
||||
throw new Error(
|
||||
"verify:esp-flash needs an ESP-IDF checkout for gen_esp32part.py and otatool.py; set REA_ESP_IDF_PATH",
|
||||
);
|
||||
try {
|
||||
await access(generator());
|
||||
await access(join(idf, "components", "app_update", "otatool.py"));
|
||||
const version = await readFile(
|
||||
join(idf, "tools", "cmake", "version.cmake"),
|
||||
"utf8",
|
||||
);
|
||||
const part = (name) =>
|
||||
new RegExp(`IDF_VERSION_${name} (\\d+)`, "u").exec(version)?.[1];
|
||||
return `v${part("MAJOR")}.${part("MINOR")}.${part("PATCH")}`;
|
||||
} catch (failure) {
|
||||
throw new Error(
|
||||
`verify:esp-flash needs gen_esp32part.py, otatool.py and tools/cmake/version.cmake under REA_ESP_IDF_PATH (${idf}). ${failure.message}`,
|
||||
{ cause: failure },
|
||||
);
|
||||
}
|
||||
};
|
||||
|
||||
/** Build a binary table from CSV with ESP-IDF's generator. */
|
||||
export const buildTable = (csv, output, flags = []) =>
|
||||
execute(python, ["-I", generator(), "--quiet", ...flags, csv, output], {
|
||||
...options,
|
||||
env: environment(),
|
||||
});
|
||||
|
||||
const bytesOf = (text) => {
|
||||
const value = text.trim();
|
||||
if (value.endsWith("M")) return Number(value.slice(0, -1)) * 0x10_0000;
|
||||
if (value.endsWith("K")) return Number(value.slice(0, -1)) * 0x400;
|
||||
return Number(value);
|
||||
};
|
||||
|
||||
/** ESP-IDF's own reading of a binary table, as the CSV it prints. */
|
||||
export const decodeTable = async (binary) => {
|
||||
const { stdout } = await execute(python, ["-I", generator(), binary], {
|
||||
...options,
|
||||
env: environment(),
|
||||
});
|
||||
return stdout
|
||||
.split("\n")
|
||||
.filter((line) => line.trim() !== "" && !line.startsWith("#"))
|
||||
.map((line) => {
|
||||
const [name, type, subtype, offset, size, flags = ""] = line
|
||||
.split(",")
|
||||
.map((field) => field.trim());
|
||||
return {
|
||||
name,
|
||||
type,
|
||||
subtype,
|
||||
offset: bytesOf(offset),
|
||||
size: bytesOf(size),
|
||||
encrypted: flags.split(":").includes("encrypted"),
|
||||
readonly: flags.split(":").includes("readonly"),
|
||||
};
|
||||
});
|
||||
};
|
||||
|
||||
/** Run otatool against a dump file through the conformance shim. */
|
||||
export const otatool = async (image, flashOffset, args) => {
|
||||
const run = await execute(
|
||||
python,
|
||||
[
|
||||
"-I",
|
||||
source("flash_file_tool.py"),
|
||||
image,
|
||||
`0x${flashOffset.toString(16)}`,
|
||||
"otatool",
|
||||
...args,
|
||||
],
|
||||
{ ...options, env: environment() },
|
||||
).then(
|
||||
({ stdout, stderr }) => `${stdout}${stderr}`,
|
||||
failedCommand(`otatool ${args.join(" ")}`),
|
||||
);
|
||||
return run;
|
||||
};
|
||||
|
||||
/** The two records otatool read_otadata prints. */
|
||||
export const readOtadata = async (image, flashOffset, tableOffset) => {
|
||||
const text = await otatool(image, flashOffset, [
|
||||
"--partition-table-offset",
|
||||
`0x${tableOffset.toString(16)}`,
|
||||
"read_otadata",
|
||||
]);
|
||||
const row =
|
||||
/Firmware:\s+0x([0-9a-f]{8})\s+0x([0-9a-f]{8})\s*\|\s*0x([0-9a-f]{8})\s+0x([0-9a-f]{8})/u.exec(
|
||||
text,
|
||||
);
|
||||
assert.ok(row, `Unrecognized read_otadata output: ${text}`);
|
||||
return [
|
||||
{ seq: Number.parseInt(row[1], 16), crc: Number.parseInt(row[2], 16) },
|
||||
{ seq: Number.parseInt(row[3], 16), crc: Number.parseInt(row[4], 16) },
|
||||
];
|
||||
};
|
||||
|
||||
/** Assemble a flash image with esptool merge-bin. */
|
||||
export const mergeBin = (chip, output, parts, flags = []) =>
|
||||
tool("esptool", [
|
||||
"--chip",
|
||||
chip,
|
||||
spell("merge_bin", "merge-bin"),
|
||||
"-o",
|
||||
output,
|
||||
...flags,
|
||||
...parts.flatMap(({ at, file }) => [`0x${at.toString(16)}`, file]),
|
||||
]);
|
||||
|
||||
export const padTo = (size) => [
|
||||
spell("--fill-flash-size", "--pad-to-size"),
|
||||
size,
|
||||
];
|
||||
@@ -0,0 +1,419 @@
|
||||
#!/usr/bin/env node
|
||||
import assert from "node:assert/strict";
|
||||
import { createHash } from "node:crypto";
|
||||
import { mkdtemp, readFile, rm, writeFile } from "node:fs/promises";
|
||||
import { tmpdir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
import { withVerifierCleanup } from "../../lib/verifier-cleanup.mjs";
|
||||
import {
|
||||
assertAgrees,
|
||||
buildElf,
|
||||
checkTools,
|
||||
commands,
|
||||
elf2image,
|
||||
extract,
|
||||
imageInfo,
|
||||
inspect,
|
||||
} from "./esp-image-tools.mjs";
|
||||
import {
|
||||
buildTable,
|
||||
checkEspIdf,
|
||||
decodeTable,
|
||||
mergeBin,
|
||||
otatool,
|
||||
padTo,
|
||||
readOtadata,
|
||||
} from "./esp-flash-tools.mjs";
|
||||
|
||||
let workspace;
|
||||
const path = (name) => join(workspace, name);
|
||||
const sha256 = (bytes) => createHash("sha256").update(bytes).digest("hex");
|
||||
|
||||
const DEFAULT_CSV = `# Name, Type, SubType, Offset, Size, Flags
|
||||
nvs,data,nvs,0x9000,0x4000,
|
||||
otadata,data,ota,0xd000,0x2000,
|
||||
phy_init,data,phy,0xf000,0x1000,
|
||||
factory,app,factory,0x10000,0x20000,
|
||||
ota_0,app,ota_0,0x30000,0x20000,
|
||||
ota_1,app,ota_1,0x50000,0x20000,
|
||||
keys,data,nvs_keys,0x70000,0x1000,encrypted
|
||||
`;
|
||||
// Blank offsets: gen_esp32part places each partition after the moved table.
|
||||
const RELOCATED_CSV = `# Name, Type, SubType, Offset, Size, Flags
|
||||
nvs,data,nvs,,0x4000,
|
||||
otadata,data,ota,,0x2000,
|
||||
factory,app,factory,,0x20000,
|
||||
`;
|
||||
// ESP-IDF 5.3 entries over the bootloader region and the table sector;
|
||||
// gen_esp32part fills their ranges from --primary-bootloader-offset.
|
||||
const PRIMARY_CSV = `# Name, Type, SubType, Offset, Size, Flags
|
||||
PrimaryBTLDR,bootloader,primary,N/A,N/A,
|
||||
PrimaryPrtTable,partition_table,primary,N/A,N/A,
|
||||
nvs,data,nvs,0x9000,0x4000,
|
||||
factory,app,factory,0x10000,0x20000,
|
||||
`;
|
||||
|
||||
/** Nested ESP image facts, shaped like a top-level inventory for assertAgrees. */
|
||||
const nested = (facts) => ({
|
||||
inventory: { manifest: { root_format: "esp-image" } },
|
||||
root: facts.image,
|
||||
segments: facts.segments.map((segment) => ({ format_metadata: segment })),
|
||||
});
|
||||
|
||||
const appFacts = (rea, label) => {
|
||||
const row = rea.inventory.occurrences.find(
|
||||
({ format_metadata: metadata }) =>
|
||||
metadata?.kind === "esp-flash-partition" && metadata.label === label,
|
||||
);
|
||||
return rea.root.apps.find(
|
||||
({ partition_index: index }) => index === row?.format_metadata.index,
|
||||
)?.facts;
|
||||
};
|
||||
|
||||
/** Every declared partition agrees with ESP-IDF's decoding of the table REA extracted. */
|
||||
const assertTableAgrees = async (name, rea, extraction) => {
|
||||
const decoded = await decodeTable(
|
||||
join(extraction.output_root, "partition-table.bin"),
|
||||
);
|
||||
const declared = [
|
||||
...rea.inventory.occurrences
|
||||
.filter(({ format_metadata: m }) => m?.kind === "esp-flash-partition")
|
||||
.map(({ format_metadata: m }) => m),
|
||||
...rea.root.partitions_outside_dump,
|
||||
].sort((left, right) => left.index - right.index);
|
||||
assert.deepEqual(
|
||||
declared.map((p) => ({
|
||||
name: p.label,
|
||||
type: p.type_name,
|
||||
subtype: p.subtype_name,
|
||||
offset: p.flash_address,
|
||||
size: p.size,
|
||||
encrypted: p.encrypted,
|
||||
readonly: p.readonly,
|
||||
})),
|
||||
decoded,
|
||||
`${name} table`,
|
||||
);
|
||||
};
|
||||
|
||||
/** Each extracted file is the dump's bytes at its range; the source is unchanged. */
|
||||
const assertExtraction = async (name, file, rea, flags = []) => {
|
||||
const bytes = await readFile(file);
|
||||
const extraction = await extract(file, workspace, flags);
|
||||
assert.equal(extraction.containment_verified, true, name);
|
||||
assert.equal(
|
||||
extraction.manifest.root_sha256,
|
||||
sha256(bytes),
|
||||
`${name} source`,
|
||||
);
|
||||
const rows = rea.inventory.occurrences.filter(
|
||||
({ logical_path }) => logical_path !== ".",
|
||||
);
|
||||
assert.equal(extraction.artifacts.length, rows.length, `${name} entries`);
|
||||
for (const { logical_path: entry, source_location: range } of rows) {
|
||||
const written = await readFile(join(extraction.output_root, entry));
|
||||
assert.ok(
|
||||
written.equals(bytes.subarray(range.offset, range.offset + range.length)),
|
||||
`${name} ${entry}`,
|
||||
);
|
||||
}
|
||||
assert.equal(sha256(await readFile(file)), sha256(bytes), `${name} source`);
|
||||
return extraction;
|
||||
};
|
||||
|
||||
const build = async () => {
|
||||
const app = await buildElf("app.c", "app.ld", path("app.elf"));
|
||||
const boot = await buildElf(
|
||||
"bootloader.c",
|
||||
"bootloader.ld",
|
||||
path("boot.elf"),
|
||||
);
|
||||
const descriptor = ["--elf-sha256-offset", "0xb0"];
|
||||
await elf2image("esp32c3", app, path("app.bin"), descriptor);
|
||||
await elf2image("esp32c3", boot, path("boot.bin"));
|
||||
// esptool converts the RISC-V ELF for any chip; these use ESP32 addresses.
|
||||
const app32 = await buildElf(
|
||||
"app.c",
|
||||
"esp32-app-riscv.ld",
|
||||
path("app32.elf"),
|
||||
);
|
||||
const boot32 = await buildElf(
|
||||
"bootloader.c",
|
||||
"esp32-bootloader.ld",
|
||||
path("boot32.elf"),
|
||||
);
|
||||
await elf2image("esp32", app32, path("app32.bin"), descriptor);
|
||||
await elf2image("esp32", boot32, path("boot32.bin"));
|
||||
await writeFile(path("default.csv"), DEFAULT_CSV);
|
||||
await writeFile(path("relocated.csv"), RELOCATED_CSV);
|
||||
await writeFile(path("primary.csv"), PRIMARY_CSV);
|
||||
await buildTable(path("default.csv"), path("default-table.bin"));
|
||||
await buildTable(path("relocated.csv"), path("relocated-table.bin"), [
|
||||
"--offset",
|
||||
"0xa000",
|
||||
]);
|
||||
await buildTable(path("primary.csv"), path("primary-table.bin"), [
|
||||
"--primary-bootloader-offset",
|
||||
"0x0",
|
||||
]);
|
||||
};
|
||||
|
||||
const assertOtaAgrees = async (name, file, rea) => {
|
||||
const records = await readOtadata(file, 0, 0x8000);
|
||||
assert.deepEqual(
|
||||
rea.root.otadata.records.map(({ ota_seq: seq, crc }) => ({ seq, crc })),
|
||||
records,
|
||||
`${name} otadata`,
|
||||
);
|
||||
};
|
||||
|
||||
const defaultLayout = async (report) => {
|
||||
const file = path("c3.bin");
|
||||
await mergeBin(
|
||||
"esp32c3",
|
||||
file,
|
||||
[
|
||||
{ at: 0x0, file: path("boot.bin") },
|
||||
{ at: 0x8000, file: path("default-table.bin") },
|
||||
{ at: 0x1_0000, file: path("app.bin") },
|
||||
{ at: 0x3_0000, file: path("app.bin") },
|
||||
],
|
||||
padTo("1MB"),
|
||||
);
|
||||
await otatool(file, 0, ["switch_ota_partition", "--slot", "0"]);
|
||||
const rea = await inspect(file);
|
||||
assert.equal(rea.inventory.manifest.root_format, "esp-flash-image");
|
||||
assert.equal(rea.inventory.coverage.status, "complete");
|
||||
assert.deepEqual(rea.root.undescribed, []);
|
||||
assert.equal(rea.root.partition_table.method, "default");
|
||||
assert.equal(rea.root.partition_table.md5.status, "valid");
|
||||
assertAgrees(
|
||||
"c3 bootloader",
|
||||
await imageInfo(path("boot.bin")),
|
||||
nested(rea.root.bootloader.facts),
|
||||
);
|
||||
const appOracle = await imageInfo(path("app.bin"));
|
||||
for (const label of ["factory", "ota_0"])
|
||||
assertAgrees(`c3 ${label}`, appOracle, nested(appFacts(rea, label)));
|
||||
assert.equal(appFacts(rea, "ota_1").status, "erased");
|
||||
await assertOtaAgrees("c3 slot 0", file, rea);
|
||||
assert.deepEqual(
|
||||
[rea.root.boot_selection.slot, rea.root.boot_selection.basis],
|
||||
[0, "otadata"],
|
||||
);
|
||||
const extraction = await assertExtraction("c3", file, rea);
|
||||
await assertTableAgrees("c3", rea, extraction);
|
||||
// A second switch writes the other record; the higher sequence selects ota_1.
|
||||
await otatool(file, 0, ["switch_ota_partition", "--slot", "1"]);
|
||||
const switched = await inspect(file);
|
||||
await assertOtaAgrees("c3 slot 1", file, switched);
|
||||
assert.equal(switched.root.boot_selection.slot, 1);
|
||||
assert.equal(switched.root.boot_selection.running_partition, "unknown");
|
||||
// Erased records leave the bootloader's factory fallback.
|
||||
await otatool(file, 0, ["erase_otadata"]);
|
||||
const erased = await inspect(file);
|
||||
await assertOtaAgrees("c3 erased", file, erased);
|
||||
const factory = erased.inventory.occurrences.find(
|
||||
({ format_metadata: m }) => m?.label === "factory",
|
||||
);
|
||||
assert.deepEqual(
|
||||
[
|
||||
erased.root.boot_selection.basis,
|
||||
erased.root.boot_selection.partition_index,
|
||||
],
|
||||
["factory-fallback", factory?.format_metadata.index],
|
||||
);
|
||||
report.cases.push({
|
||||
name: "c3.bin",
|
||||
partitions: extraction.artifacts.length - 2,
|
||||
selected: [0, 1, "factory-fallback"],
|
||||
});
|
||||
};
|
||||
|
||||
const primaryEntries = async (report) => {
|
||||
const file = path("primary.bin");
|
||||
await mergeBin(
|
||||
"esp32c3",
|
||||
file,
|
||||
[
|
||||
{ at: 0x0, file: path("boot.bin") },
|
||||
{ at: 0x8000, file: path("primary-table.bin") },
|
||||
{ at: 0x1_0000, file: path("app.bin") },
|
||||
],
|
||||
padTo("1MB"),
|
||||
);
|
||||
const rea = await inspect(file);
|
||||
assert.equal(rea.inventory.coverage.status, "complete");
|
||||
assert.deepEqual(
|
||||
rea.inventory.limitations.filter((text) =>
|
||||
/overlaps|before the end of the partition table/u.test(text),
|
||||
),
|
||||
[],
|
||||
"primary entries are not misplaced",
|
||||
);
|
||||
const extraction = await assertExtraction("primary", file, rea);
|
||||
await assertTableAgrees("primary", rea, extraction);
|
||||
// Without otadata the bootloader selects the factory index.
|
||||
const factory = rea.inventory.occurrences.find(
|
||||
({ format_metadata: m }) => m?.label === "factory",
|
||||
);
|
||||
assert.deepEqual(
|
||||
[
|
||||
rea.root.otadata,
|
||||
rea.root.boot_selection.basis,
|
||||
rea.root.boot_selection.partition_index,
|
||||
],
|
||||
[null, "no-otadata", factory?.format_metadata.index],
|
||||
);
|
||||
report.cases.push({
|
||||
name: "primary.bin",
|
||||
partitions: extraction.artifacts.length - 2,
|
||||
});
|
||||
};
|
||||
|
||||
const relocated = async (report) => {
|
||||
const [, , factory] = await decodeTable(path("relocated-table.bin"));
|
||||
const file = path("relocated.bin");
|
||||
await mergeBin(
|
||||
"esp32c3",
|
||||
file,
|
||||
[
|
||||
{ at: 0x0, file: path("boot.bin") },
|
||||
{ at: 0xa000, file: path("relocated-table.bin") },
|
||||
{ at: factory.offset, file: path("app.bin") },
|
||||
],
|
||||
padTo("1MB"),
|
||||
);
|
||||
const scanned = await inspect(file);
|
||||
assert.deepEqual(
|
||||
[
|
||||
scanned.root.partition_table.flash_address,
|
||||
scanned.root.partition_table.method,
|
||||
],
|
||||
[0xa000, "scan"],
|
||||
);
|
||||
const caller = [
|
||||
"--target-format",
|
||||
"esp-flash-image",
|
||||
"--partition-table-offset",
|
||||
"0xa000",
|
||||
];
|
||||
const selected = await inspect(file, caller);
|
||||
assert.equal(selected.root.partition_table.method, "caller");
|
||||
await assertTableAgrees(
|
||||
"relocated",
|
||||
selected,
|
||||
await assertExtraction("relocated", file, selected, caller),
|
||||
);
|
||||
report.cases.push({
|
||||
name: "relocated.bin",
|
||||
table: "0xa000",
|
||||
methods: ["scan", "caller"],
|
||||
});
|
||||
};
|
||||
|
||||
const esp32Origin = async (report) => {
|
||||
const parts = [
|
||||
{ at: 0x1000, file: path("boot32.bin") },
|
||||
{ at: 0x8000, file: path("default-table.bin") },
|
||||
{ at: 0x1_0000, file: path("app32.bin") },
|
||||
];
|
||||
const whole = path("esp32.bin");
|
||||
await mergeBin("esp32", whole, parts, padTo("1MB"));
|
||||
const rea = await inspect(whole);
|
||||
assert.deepEqual(
|
||||
[rea.root.chip.target, rea.root.bootloader.flash_address],
|
||||
["esp32", 0x1000],
|
||||
);
|
||||
assertAgrees(
|
||||
"esp32 factory",
|
||||
await imageInfo(path("app32.bin")),
|
||||
nested(appFacts(rea, "factory")),
|
||||
);
|
||||
// esptool writes the same parts as a dump that starts at flash 0x1000.
|
||||
const partial = path("esp32-from-0x1000.bin");
|
||||
await mergeBin("esp32", partial, parts, ["--target-offset", "0x1000"]);
|
||||
const guessed = await inspect(partial);
|
||||
assert.equal(
|
||||
guessed.inventory.manifest.root_format,
|
||||
"esp-image",
|
||||
"no origin, no layout",
|
||||
);
|
||||
// The table at file offset 0x7000 shows where the dump was read from.
|
||||
assert.deepEqual(guessed.root.flash_dump, {
|
||||
flash_offset: 0x1000,
|
||||
table_file_offset: 0x7000,
|
||||
});
|
||||
const flags = [
|
||||
"--target-format",
|
||||
"esp-flash-image",
|
||||
"--flash-offset",
|
||||
"0x1000",
|
||||
];
|
||||
const origin = await inspect(partial, flags);
|
||||
assert.deepEqual(origin.root.flash_offset, {
|
||||
value: 0x1000,
|
||||
source: "caller",
|
||||
});
|
||||
await assertExtraction("esp32 from 0x1000", partial, origin, flags);
|
||||
report.cases.push({
|
||||
name: "esp32-from-0x1000.bin",
|
||||
automatic: "esp-image",
|
||||
flash_offset: "0x1000",
|
||||
});
|
||||
};
|
||||
|
||||
const damaged = async (report) => {
|
||||
const bytes = await readFile(path("c3.bin"));
|
||||
const corrupt = Buffer.from(bytes);
|
||||
// Flip a byte of the factory app's first segment data.
|
||||
corrupt[0x1_0000 + 0x18 + 8] ^= 0x01;
|
||||
await writeFile(path("corrupt.bin"), corrupt);
|
||||
const rea = await inspect(path("corrupt.bin"));
|
||||
assert.equal(appFacts(rea, "factory").image.checksum.status, "invalid");
|
||||
assert.equal(appFacts(rea, "ota_0").image.checksum.status, "valid");
|
||||
await writeFile(path("truncated.bin"), bytes.subarray(0, 0x4_0000));
|
||||
const cut = await inspect(path("truncated.bin"));
|
||||
// Partitions past the end of the file are entries holding no bytes.
|
||||
assert.equal(cut.inventory.coverage.status, "complete");
|
||||
assert.deepEqual(cut.root.partitions_outside_dump, []);
|
||||
const empty = cut.inventory.occurrences
|
||||
.filter(({ declared_size: size }) => size === 0)
|
||||
.map(({ format_metadata: m }) => m?.label);
|
||||
assert.deepEqual(empty, ["keys", "ota_1"]);
|
||||
assert.ok(
|
||||
cut.inventory.occurrences.some(
|
||||
({ logical_path }) => logical_path === "partitions/factory.bin",
|
||||
),
|
||||
);
|
||||
report.cases.push(
|
||||
{ name: "corrupt.bin", factory: "invalid" },
|
||||
{ name: "truncated.bin", without_bytes: empty },
|
||||
);
|
||||
};
|
||||
|
||||
await withVerifierCleanup(async () => {
|
||||
const tools = await checkTools("verify:esp-flash");
|
||||
const espIdf = await checkEspIdf();
|
||||
workspace = await mkdtemp(join(tmpdir(), "rea-esp-flash-proof-"));
|
||||
const report = {
|
||||
esptool: tools.esptoolVersion,
|
||||
esp_idf: espIdf,
|
||||
compiler: tools.compilerVersion,
|
||||
cases: [],
|
||||
};
|
||||
await build();
|
||||
await defaultLayout(report);
|
||||
await primaryEntries(report);
|
||||
await relocated(report);
|
||||
await esp32Origin(report);
|
||||
await damaged(report);
|
||||
console.log(JSON.stringify(report, null, 2));
|
||||
}, [
|
||||
// Remove outputs only after every owned command tree has stopped.
|
||||
async () => {
|
||||
await commands.close();
|
||||
if (workspace !== undefined)
|
||||
await rm(workspace, { recursive: true, force: true });
|
||||
},
|
||||
]);
|
||||
@@ -22,6 +22,13 @@ export const exitOutput = (failure) => {
|
||||
if (typeof failure?.code !== "number" || failure.code === 0) throw failure;
|
||||
return `${failure.snapshot.stdout.text}${failure.snapshot.stderr.text}`;
|
||||
};
|
||||
|
||||
/** Reject a command that had to succeed, with the output it exited with. */
|
||||
export const failedCommand = (label) => (failure) => {
|
||||
throw new Error(`${label} failed: ${exitOutput(failure)}`, {
|
||||
cause: failure,
|
||||
});
|
||||
};
|
||||
export const source = (name) =>
|
||||
fileURLToPath(
|
||||
new URL(`../../../tests/conformance/esp/${name}`, import.meta.url),
|
||||
@@ -41,13 +48,16 @@ export const tool = async (module, args) => {
|
||||
return `${stdout}${stderr}`;
|
||||
};
|
||||
|
||||
/** The npm script whose tools are checked, named in each failure. */
|
||||
let lane = "verify:esp-images";
|
||||
|
||||
/** Check one tool before any work; a failure names the variable to set. */
|
||||
const prerequisite = async (needed, variable, check) => {
|
||||
try {
|
||||
return await check();
|
||||
} catch (failure) {
|
||||
throw new Error(
|
||||
`verify:esp-images needs ${needed}; set ${variable}. ${failure.message}`,
|
||||
`${lane} needs ${needed}; set ${variable}. ${failure.message}`,
|
||||
{ cause: failure },
|
||||
);
|
||||
}
|
||||
@@ -63,7 +73,8 @@ export const spell = (v4, later) => {
|
||||
};
|
||||
|
||||
/** Check every tool before any work and report their versions. */
|
||||
export const checkTools = async () => {
|
||||
export const checkTools = async (script = "verify:esp-images") => {
|
||||
lane = script;
|
||||
const compilerVersion = await prerequisite(
|
||||
`a Clang with the RISC-V target (tried ${compiler})`,
|
||||
"REA_RISCV_CLANG",
|
||||
@@ -98,7 +109,7 @@ export const checkTools = async () => {
|
||||
);
|
||||
if (version[0] !== PINNED_ESPTOOL)
|
||||
console.warn(
|
||||
`verify:esp-images: esptool ${version[0]} is not the pinned ${PINNED_ESPTOOL}, whose sources the reader's layout facts cite.`,
|
||||
`${lane}: esptool ${version[0]} is not the pinned ${PINNED_ESPTOOL}, whose sources the reader's layout facts cite.`,
|
||||
);
|
||||
const xtensaVersion =
|
||||
xtensa === undefined
|
||||
@@ -201,19 +212,20 @@ export const imageInfo = async (file, chip) => {
|
||||
};
|
||||
};
|
||||
|
||||
/** REA's complete inline inventory for one file through the built CLI. */
|
||||
export const inspect = async (file) => {
|
||||
const result = await execute(
|
||||
const inspectArtifact = (file, flags) =>
|
||||
execute(
|
||||
process.execPath,
|
||||
[entrypoint, "inspect-artifact", file, "--json"],
|
||||
[entrypoint, "inspect-artifact", file, ...flags, "--json"],
|
||||
{ ...options, env: { ...process.env, REA_LOG_LEVEL: "silent" } },
|
||||
).then(
|
||||
({ stdout }) => ({ ok: true, stdout }),
|
||||
(failure) => ({ ok: false, stdout: exitOutput(failure) }),
|
||||
);
|
||||
if (!result.ok) return { refused: result.stdout };
|
||||
const inventory = JSON.parse(result.stdout).normalized_result.substeps[0]
|
||||
.evidence.normalized_result;
|
||||
|
||||
/** REA's complete inline inventory for one file through the built CLI. */
|
||||
export const inspect = async (file, flags = []) => {
|
||||
const { stdout } = await inspectArtifact(file, flags).catch(
|
||||
failedCommand(`inspect-artifact ${file}`),
|
||||
);
|
||||
const inventory =
|
||||
JSON.parse(stdout).normalized_result.substeps[0].evidence.normalized_result;
|
||||
const rows = inventory.occurrences.filter((row) => row.logical_path !== ".");
|
||||
return {
|
||||
inventory,
|
||||
@@ -228,6 +240,12 @@ export const inspect = async (file) => {
|
||||
};
|
||||
};
|
||||
|
||||
/** REA's output for a file it must refuse; opening the file fails the lane. */
|
||||
export const refusal = (file, flags = []) =>
|
||||
inspectArtifact(file, flags).then(() => {
|
||||
throw new Error(`inspect-artifact opened ${file}, which it must refuse`);
|
||||
}, exitOutput);
|
||||
|
||||
const oracleDescriptorKind = (oracle) => {
|
||||
if (oracle.project !== undefined) return "app";
|
||||
if (oracle.bootloaderVersion !== undefined) return "bootloader";
|
||||
@@ -235,10 +253,10 @@ const oracleDescriptorKind = (oracle) => {
|
||||
};
|
||||
|
||||
/** Extract every entry through the built CLI into a new directory in `root`. */
|
||||
export const extract = async (file, root) => {
|
||||
export const extract = async (file, root, flags = []) => {
|
||||
const { stdout } = await execute(
|
||||
process.execPath,
|
||||
[entrypoint, "extract-artifact", file, "--json"],
|
||||
[entrypoint, "extract-artifact", file, ...flags, "--json"],
|
||||
{
|
||||
...options,
|
||||
env: {
|
||||
|
||||
@@ -17,6 +17,7 @@ import {
|
||||
imageInfo,
|
||||
inspect,
|
||||
options,
|
||||
refusal,
|
||||
source,
|
||||
spell,
|
||||
tool,
|
||||
@@ -392,7 +393,7 @@ await withVerifierCleanup(async () => {
|
||||
const v3Oracle = await imageInfo(path("esp8266-v3.bin"), "esp8266");
|
||||
assert.ok((v3Oracle.segments[0]?.address ?? 0) >= 0x4020_0000);
|
||||
assert.match(
|
||||
(await inspect(path("esp8266-v3.bin"))).refused ?? "",
|
||||
await refusal(path("esp8266-v3.bin")),
|
||||
/ESP8266 firmware images are not supported/u,
|
||||
);
|
||||
report.images.push({ name: "esp8266-v3.bin", refused: true });
|
||||
@@ -400,9 +401,8 @@ await withVerifierCleanup(async () => {
|
||||
// ESP8266 v1: esptool falls back to ESP8266; REA refuses it by name.
|
||||
const esp8266Image = path("esp8266-0x00000.bin");
|
||||
assert.equal((await imageInfo(esp8266Image)).detected, "ESP8266");
|
||||
const refused = await inspect(esp8266Image);
|
||||
assert.match(
|
||||
refused.refused ?? "",
|
||||
await refusal(esp8266Image),
|
||||
/ESP8266 firmware images are not supported/u,
|
||||
);
|
||||
report.images.push({ name: "esp8266-0x00000.bin", refused: true });
|
||||
|
||||
@@ -0,0 +1,284 @@
|
||||
#!/usr/bin/env node
|
||||
import assert from "node:assert/strict";
|
||||
import { mkdtemp, readFile, rm, writeFile } from "node:fs/promises";
|
||||
import { tmpdir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
import { fileURLToPath } from "node:url";
|
||||
import { createVerifierCommands } from "../../lib/verify-commands.mjs";
|
||||
import { withVerifierCleanup } from "../../lib/verifier-cleanup.mjs";
|
||||
|
||||
// Optional verification tool only; the UF2 reader starts no tool.
|
||||
const python = process.env.REA_ESPTOOL_PYTHON ?? "python3";
|
||||
const entrypoint = fileURLToPath(new URL("../../rea.mjs", import.meta.url));
|
||||
const options = { maxBuffer: 16 * 1024 * 1024, timeout: 120_000 };
|
||||
/** Every tool runs as an owned process tree that close() stops. */
|
||||
const commands = await createVerifierCommands();
|
||||
|
||||
let workspace;
|
||||
const path = (name) => join(workspace, name);
|
||||
|
||||
/** Deterministic input bytes; lengths include one that is not word-aligned. */
|
||||
const input = (length, seed) =>
|
||||
Buffer.from(Array.from({ length }, (_, index) => (index * 13 + seed) & 0xff));
|
||||
|
||||
// `otadata` matches the table's block count, so esptool's per-input numbering
|
||||
// from 0 puts both inputs in one numbering group.
|
||||
const FILES = {
|
||||
bootloader: input(1000, 1),
|
||||
table: input(3072, 2),
|
||||
otadata: input(3072, 4),
|
||||
app: input(501, 3),
|
||||
};
|
||||
|
||||
const CASES = [
|
||||
{
|
||||
name: "esp32s3.uf2",
|
||||
chip: "esp32s3",
|
||||
layout: [
|
||||
[0x0, "bootloader"],
|
||||
[0x8000, "table"],
|
||||
[0xd000, "otadata"],
|
||||
[0x10000, "app"],
|
||||
],
|
||||
extra: [],
|
||||
md5: true,
|
||||
payload: 452,
|
||||
},
|
||||
{
|
||||
name: "esp32s2-tinyuf2.uf2",
|
||||
chip: "esp32s2",
|
||||
layout: [
|
||||
[0x1000, "bootloader"],
|
||||
[0x8000, "table"],
|
||||
],
|
||||
extra: ["--chunk-size", "256", "--md5-disable"],
|
||||
md5: false,
|
||||
payload: 256,
|
||||
},
|
||||
];
|
||||
|
||||
const tool = async (args) => {
|
||||
const { stdout, stderr } = await commands.exec(
|
||||
python,
|
||||
["-m", "esptool", ...args],
|
||||
options,
|
||||
);
|
||||
return `${stdout}${stderr}`;
|
||||
};
|
||||
|
||||
/** esptool's own family ID for a chip: the oracle REA's table must match. */
|
||||
const familyOf = async (chip) => {
|
||||
const { stdout } = await commands.exec(
|
||||
python,
|
||||
[
|
||||
"-c",
|
||||
"import sys, esptool.targets as t; print(t.CHIP_DEFS[sys.argv[1]].UF2_FAMILY_ID)",
|
||||
chip,
|
||||
],
|
||||
options,
|
||||
);
|
||||
return Number(stdout.trim());
|
||||
};
|
||||
|
||||
const rea = async (command, file) => {
|
||||
const { stdout } = await commands.exec(
|
||||
process.execPath,
|
||||
[entrypoint, command, file, "--json"],
|
||||
{
|
||||
...options,
|
||||
env: {
|
||||
...process.env,
|
||||
REA_LOG_LEVEL: "silent",
|
||||
TMPDIR: workspace,
|
||||
TMP: workspace,
|
||||
TEMP: workspace,
|
||||
},
|
||||
},
|
||||
);
|
||||
return JSON.parse(stdout).normalized_result;
|
||||
};
|
||||
|
||||
const familyDirectory = (familyId) =>
|
||||
`family-0x${familyId.toString(16).padStart(8, "0")}`;
|
||||
const hex32 = (value) => `0x${value.toString(16).padStart(8, "0")}`;
|
||||
const blocksOf = (entry, name) => Math.ceil(FILES[name].length / entry.payload);
|
||||
const inventoryOf = async (file) =>
|
||||
(await rea("inspect-artifact", path(file))).substeps[0].evidence
|
||||
.normalized_result;
|
||||
const rootOf = (inventory) =>
|
||||
inventory.occurrences.find((row) => row.logical_path === ".").format_metadata;
|
||||
|
||||
/** Write the inputs, merge each case with esptool, and concatenate the cases. */
|
||||
const produce = async (v5) => {
|
||||
for (const [name, bytes] of Object.entries(FILES))
|
||||
await writeFile(path(`${name}.bin`), bytes);
|
||||
for (const entry of CASES) {
|
||||
await tool([
|
||||
"--chip",
|
||||
entry.chip,
|
||||
v5 ? "merge-bin" : "merge_bin",
|
||||
"--format",
|
||||
"uf2",
|
||||
...entry.extra,
|
||||
"-o",
|
||||
path(entry.name),
|
||||
...entry.layout.flatMap(([address, name]) => [
|
||||
hex32(address),
|
||||
path(`${name}.bin`),
|
||||
]),
|
||||
]);
|
||||
entry.familyId = await familyOf(entry.chip);
|
||||
}
|
||||
// The UF2 specification makes a multi-family file by concatenation.
|
||||
await writeFile(
|
||||
path("concatenated.uf2"),
|
||||
Buffer.concat(
|
||||
await Promise.all(CASES.map((entry) => readFile(path(entry.name)))),
|
||||
),
|
||||
);
|
||||
};
|
||||
|
||||
/** Root facts a complete file from these cases must report. */
|
||||
const expectedRoot = (included) => {
|
||||
const blocks = included.flatMap((entry) =>
|
||||
entry.layout.map(([, name]) => ({
|
||||
group: `${entry.chip}:${String(blocksOf(entry, name))}`,
|
||||
count: blocksOf(entry, name),
|
||||
md5: entry.md5,
|
||||
})),
|
||||
);
|
||||
const total = blocks.reduce((sum, { count }) => sum + count, 0);
|
||||
// Inputs of one chip with equal block counts form one numbering group,
|
||||
// and every input after the first in a group repeats its numbers.
|
||||
const groups = new Map();
|
||||
for (const { group, count } of blocks)
|
||||
groups.set(group, [...(groups.get(group) ?? []), count]);
|
||||
return {
|
||||
block_count: total,
|
||||
invalid_block_count: 0,
|
||||
flash_block_count: total,
|
||||
md5_block_count: blocks
|
||||
.filter(({ md5 }) => md5)
|
||||
.reduce((sum, { count }) => sum + count, 0),
|
||||
family_count: included.length,
|
||||
numbering_group_count: groups.size,
|
||||
missing_block_count: 0,
|
||||
ambiguous_numbering_group_count: 0,
|
||||
repeated_block_count: [...groups.values()].reduce(
|
||||
(sum, counts) => sum + (counts.length - 1) * (counts[0] ?? 0),
|
||||
0,
|
||||
),
|
||||
duplicate_bytes: 0,
|
||||
};
|
||||
};
|
||||
|
||||
/** Every input must come back at its address under esptool's familyID. */
|
||||
const check = async (file, included) => {
|
||||
const inventory = await inventoryOf(file);
|
||||
assert.equal(inventory.manifest.root_format, "uf2", file);
|
||||
assert.equal(inventory.coverage.status, "complete", file);
|
||||
const root = rootOf(inventory);
|
||||
const expected = expectedRoot(included);
|
||||
assert.deepEqual(
|
||||
Object.fromEntries(Object.keys(expected).map((key) => [key, root[key]])),
|
||||
expected,
|
||||
file,
|
||||
);
|
||||
const inputs = included.flatMap((entry) =>
|
||||
entry.layout.map(([address, name]) => ({
|
||||
path: `${familyDirectory(entry.familyId)}/${hex32(address)}-${hex32(address + FILES[name].length - 1)}.bin`,
|
||||
familyId: entry.familyId,
|
||||
bytes: FILES[name],
|
||||
})),
|
||||
);
|
||||
const regions = inventory.occurrences.filter(
|
||||
(row) => row.format_metadata?.kind === "uf2-region",
|
||||
);
|
||||
assert.deepEqual(
|
||||
regions.map((row) => row.logical_path).sort(),
|
||||
inputs.map((region) => region.path).sort(),
|
||||
file,
|
||||
);
|
||||
for (const region of regions)
|
||||
assert.equal(
|
||||
region.format_metadata.family_id,
|
||||
inputs.find((row) => row.path === region.logical_path).familyId,
|
||||
`${file} ${region.logical_path}`,
|
||||
);
|
||||
const extracted = await rea("extract-artifact", path(file));
|
||||
assert.equal(extracted.containment_verified, true, file);
|
||||
for (const region of inputs)
|
||||
assert.equal(
|
||||
Buffer.compare(
|
||||
await readFile(join(extracted.output_root, region.path)),
|
||||
region.bytes,
|
||||
),
|
||||
0,
|
||||
`${file} ${region.path}`,
|
||||
);
|
||||
return {
|
||||
file,
|
||||
families: included.map((entry) => ({
|
||||
chip: entry.chip,
|
||||
family_id: hex32(entry.familyId),
|
||||
family_name: regions.find(
|
||||
(row) => row.format_metadata.family_id === entry.familyId,
|
||||
).format_metadata.family_name,
|
||||
})),
|
||||
blocks: root.block_count,
|
||||
regions: regions.length,
|
||||
md5_blocks: root.md5_block_count,
|
||||
};
|
||||
};
|
||||
|
||||
/**
|
||||
* Losing one block of one of two equal-count inputs must not read as
|
||||
* complete: the other input still carries every block number.
|
||||
*/
|
||||
const checkLostBlock = async () => {
|
||||
const [s3] = CASES;
|
||||
const position = s3.layout.findIndex(([, name]) => name === "otadata");
|
||||
const lostIndex =
|
||||
s3.layout
|
||||
.slice(0, position)
|
||||
.reduce((sum, [, name]) => sum + blocksOf(s3, name), 0) + 1;
|
||||
const whole = await readFile(path(s3.name));
|
||||
await writeFile(
|
||||
path("lost-block.uf2"),
|
||||
Buffer.concat([
|
||||
whole.subarray(0, lostIndex * 512),
|
||||
whole.subarray((lostIndex + 1) * 512),
|
||||
]),
|
||||
);
|
||||
const inventory = await inventoryOf("lost-block.uf2");
|
||||
const root = rootOf(inventory);
|
||||
assert.equal(inventory.coverage.status, "unknown", "lost-block.uf2");
|
||||
assert.equal(root.ambiguous_numbering_group_count, 1, "lost-block.uf2");
|
||||
assert.equal(root.missing_block_count, 0, "lost-block.uf2");
|
||||
return {
|
||||
file: "lost-block.uf2",
|
||||
removed_block: lostIndex,
|
||||
coverage: inventory.coverage.status,
|
||||
ambiguous_numbering_groups: root.ambiguous_numbering_group_count,
|
||||
};
|
||||
};
|
||||
|
||||
await withVerifierCleanup(async () => {
|
||||
const versionText = await tool(["version"]);
|
||||
const version = /\bv?(\d+)\.(\d+)\.(\d+)\b/u.exec(versionText);
|
||||
assert.ok(version, `Unrecognized esptool version: ${versionText}`);
|
||||
workspace = await mkdtemp(join(tmpdir(), "rea-uf2-esptool-"));
|
||||
await produce(Number(version[1]) >= 5);
|
||||
const report = { esptool: version[0], cases: [] };
|
||||
for (const entry of CASES)
|
||||
report.cases.push(await check(entry.name, [entry]));
|
||||
report.cases.push(await check("concatenated.uf2", CASES));
|
||||
report.cases.push(await checkLostBlock());
|
||||
process.stdout.write(`${JSON.stringify(report, null, 2)}\n`);
|
||||
}, [
|
||||
() => commands.close(),
|
||||
() =>
|
||||
workspace === undefined
|
||||
? undefined
|
||||
: rm(workspace, { recursive: true, force: true }),
|
||||
]);
|
||||
@@ -60,13 +60,13 @@ const reconciled = reconcileJavaScriptRuntimeEvidence({
|
||||
runtime_observations: runtimeObservations,
|
||||
});
|
||||
if (!reconciled.ok) throw reconciled.error;
|
||||
const result = reconciled.value.normalized_result;
|
||||
const result = reconciled.value.evidence.normalized_result;
|
||||
|
||||
process.stdout.write(
|
||||
`${JSON.stringify({
|
||||
verifier_run: await completeVerifierRun(verifierRun),
|
||||
reconciliation_id: result.reconciliation_id,
|
||||
evidence_id: reconciled.value.evidence_id,
|
||||
evidence_id: reconciled.value.evidence.evidence_id,
|
||||
graph_id: result.graph.graph_id,
|
||||
layers: result.static_layers.map((layer) => ({
|
||||
role: layer.role,
|
||||
|
||||
@@ -4,6 +4,10 @@ import {
|
||||
} from "../domain/artifactNameSelection.js";
|
||||
import { readFileHandleRange } from "../filesystem/readFileHandleRange.js";
|
||||
import { readPeCoffHeader } from "../domain/peCoffHeader.js";
|
||||
import {
|
||||
nesContainerFormat,
|
||||
parseInesLayout,
|
||||
} from "../domain/inesContainer.js";
|
||||
import {
|
||||
regularFileCancellation,
|
||||
RegularFileCleanupFailure,
|
||||
@@ -22,18 +26,13 @@ import { jsonValueSchema } from "../domain/jsonValue.js";
|
||||
import { projectAnalysisError } from "../domain/analysisErrorProjection.js";
|
||||
import { constants, type Stats } from "node:fs";
|
||||
import { createHash } from "node:crypto";
|
||||
import {
|
||||
access,
|
||||
lstat,
|
||||
realpath,
|
||||
stat,
|
||||
type FileHandle,
|
||||
} from "node:fs/promises";
|
||||
import { access, realpath, stat, type FileHandle } from "node:fs/promises";
|
||||
import { extname } from "node:path";
|
||||
|
||||
import { BinaryTargetError } from "../domain/configurationErrors.js";
|
||||
import {
|
||||
sameRegularFileState,
|
||||
assertStableRegularFileIdentity,
|
||||
RegularFileChangedError,
|
||||
type StableRegularFileDescriptor,
|
||||
} from "../filesystem/RegularFile.js";
|
||||
import { AnalysisCancelledError } from "../domain/analysisErrorCore.js";
|
||||
@@ -45,7 +44,11 @@ import type {
|
||||
} from "../domain/binaryTargetTypes.js";
|
||||
import { err, ok, type Result } from "../domain/result.js";
|
||||
import { readMipsElfMetadata } from "./MipsElfMetadata.js";
|
||||
import { validateElfDeclaredRanges } from "../domain/elfDeclaredRanges.js";
|
||||
import {
|
||||
validateElfDeclaredRanges,
|
||||
type ElfDeclarationRead,
|
||||
} from "../domain/elfDeclaredRanges.js";
|
||||
import { readRiscvIsaDeclaration } from "../domain/riscvIsaDeclaration.js";
|
||||
import {
|
||||
resolveAppleBundleExecutable,
|
||||
type ResolvedAppBundle,
|
||||
@@ -83,7 +86,18 @@ import { hasGodotPckHeader } from "../domain/godotPckFormat.js";
|
||||
import { hasUImageMagic } from "../domain/uimageFormat.js";
|
||||
import { hasUnityFsHeader } from "../domain/unityfsFormat.js";
|
||||
import { detectHexRecordEncoding } from "../domain/firmware/hexRecords.js";
|
||||
import { hasUf2Block } from "../domain/firmware/uf2.js";
|
||||
import {
|
||||
DFUSE_SUFFIX_BYTES,
|
||||
hasDfuSePrefix,
|
||||
isDfuSeFile,
|
||||
} from "../domain/firmware/dfuse.js";
|
||||
import { hasFitImagesNode } from "../domain/firmware/fitImage.js";
|
||||
import {
|
||||
locateEspFlashLayout,
|
||||
type EspFlashSelection,
|
||||
} from "../domain/firmware/espFlashLocation.js";
|
||||
import { espFlashSelectionError } from "../domain/firmware/espFlashSelection.js";
|
||||
import {
|
||||
ESP8266_REFUSAL,
|
||||
recognizeEspImage,
|
||||
@@ -136,6 +150,8 @@ export interface BinaryTargetResolverOptions {
|
||||
readonly formatHint?: ExecutableFormatHint;
|
||||
readonly architecture?: BinaryArchitecture;
|
||||
readonly entryPoint?: string;
|
||||
/** Inputs for format=esp-flash-image; rejected with any other format. */
|
||||
readonly espFlash?: EspFlashSelection;
|
||||
readonly signal?: AbortSignal;
|
||||
readonly resources?: ArtifactResourceScope;
|
||||
}
|
||||
@@ -273,6 +289,18 @@ const readOpenedBinaryTarget = async (
|
||||
const hostArchitecture = options.hostArchitecture ?? process.arch;
|
||||
const { targetKind, formatHint, signal, architecture, entryPoint } = options;
|
||||
throwIfTargetResolutionCancelled(signal);
|
||||
// Flash inputs pair with one format; check before any hint branch.
|
||||
const flashPairing = espFlashSelectionError({
|
||||
format: formatHint,
|
||||
flashOffset: options.espFlash?.flashOffset,
|
||||
partitionTableOffset: options.espFlash?.partitionTableOffset,
|
||||
});
|
||||
if (flashPairing !== undefined)
|
||||
return err(
|
||||
new BinaryTargetError(path, flashPairing, {
|
||||
constraint: "load_selection_rejected",
|
||||
}),
|
||||
);
|
||||
if (formatHint === "raw-binary") {
|
||||
if (targetKind !== undefined && targetKind !== "executable")
|
||||
return err(
|
||||
@@ -360,6 +388,45 @@ const readOpenedBinaryTarget = async (
|
||||
dol: admitted.value.header,
|
||||
});
|
||||
}
|
||||
if (formatHint === "esp-flash-image") {
|
||||
if (targetKind !== undefined && targetKind !== "archive")
|
||||
return err(
|
||||
new BinaryTargetError(
|
||||
path,
|
||||
"ESP flash image format requires an archive target kind",
|
||||
{ constraint: "load_selection_rejected" },
|
||||
),
|
||||
);
|
||||
const selection = options.espFlash ?? {};
|
||||
// An encrypted dump has no readable table: unknown, not malformed.
|
||||
const located = await locateEspFlashLayout(
|
||||
metadata.size,
|
||||
(position, length) => readHandleRange(handle, position, length, signal),
|
||||
selection,
|
||||
{ explicit: true, ...(signal === undefined ? {} : { signal }) },
|
||||
);
|
||||
if (!located.ok)
|
||||
return err(
|
||||
new BinaryTargetError(
|
||||
path,
|
||||
`ESP flash image layout is not recognized: ${located.error}`,
|
||||
{ constraint: "file_format_rejected" },
|
||||
),
|
||||
);
|
||||
return ok({
|
||||
path,
|
||||
sourcePath,
|
||||
artifactSelectedPath: selectedPath,
|
||||
sha256: await sha256Handle(handle, path, metadata, signal),
|
||||
kind: "archive",
|
||||
format: "esp-flash-image",
|
||||
espFlash: {
|
||||
selected: true,
|
||||
flashOffset: selection.flashOffset ?? null,
|
||||
partitionTableOffset: selection.partitionTableOffset ?? null,
|
||||
},
|
||||
});
|
||||
}
|
||||
if (formatHint === "dos-com") {
|
||||
if (targetKind !== undefined && targetKind !== "executable")
|
||||
return err(
|
||||
@@ -393,10 +460,13 @@ const readOpenedBinaryTarget = async (
|
||||
const artifactFormat =
|
||||
targetKind === "database"
|
||||
? undefined
|
||||
: await detectArtifactFormat(classificationPath, source, signal);
|
||||
: await detectArtifactFormat(classificationPath, source, signal, path);
|
||||
if (
|
||||
targetKind === "database" ||
|
||||
(targetKind === undefined && path.toLowerCase().endsWith(".hop"))
|
||||
(targetKind === undefined &&
|
||||
artifactFormat !== "ines" &&
|
||||
artifactFormat !== "plist" &&
|
||||
path.toLowerCase().endsWith(".hop"))
|
||||
)
|
||||
return ok({
|
||||
path,
|
||||
@@ -412,6 +482,17 @@ const readOpenedBinaryTarget = async (
|
||||
artifactSelectedPath: selectedPath,
|
||||
sha256: await sha256Handle(handle, path, metadata, signal),
|
||||
};
|
||||
if (artifactFormat === "esp-flash-image")
|
||||
return ok({
|
||||
...identity,
|
||||
kind: "archive",
|
||||
format: artifactFormat,
|
||||
espFlash: {
|
||||
selected: false,
|
||||
flashOffset: null,
|
||||
partitionTableOffset: null,
|
||||
},
|
||||
});
|
||||
return isArchiveFormat(artifactFormat)
|
||||
? ok({ ...identity, kind: "archive", format: artifactFormat })
|
||||
: ok({ ...identity, kind: "artifact", format: artifactFormat });
|
||||
@@ -647,6 +728,7 @@ const detectArtifactFormat = async (
|
||||
path: string,
|
||||
source: StableRegularFileDescriptor,
|
||||
signal?: AbortSignal,
|
||||
sourcePath = path,
|
||||
): Promise<
|
||||
| Exclude<
|
||||
BinaryTarget["format"],
|
||||
@@ -667,6 +749,16 @@ const detectArtifactFormat = async (
|
||||
const lower = path.toLowerCase();
|
||||
// Fill the prefix so a short host read cannot hide a signature.
|
||||
const magic = Buffer.from(await readHandleRange(handle, 0, 4096, signal));
|
||||
try {
|
||||
await assertStableRegularFileIdentity(handle, sourcePath, initial, {
|
||||
symlinks: "reject",
|
||||
...(signal === undefined ? {} : { signal }),
|
||||
});
|
||||
} catch (cause: unknown) {
|
||||
throwIfTargetResolutionCancelled(signal);
|
||||
throw cause;
|
||||
}
|
||||
throwIfTargetResolutionCancelled(signal);
|
||||
if (
|
||||
magic.length >= 8 &&
|
||||
magic.subarray(0, 8).equals(Buffer.from([0, 0x61, 0x73, 0x6d, 1, 0, 0, 0]))
|
||||
@@ -676,12 +768,45 @@ const detectArtifactFormat = async (
|
||||
"WebAssembly v1 artifacts are not supported by the native opening route",
|
||||
{ constraint: "wasm_requires_static_inspection" },
|
||||
);
|
||||
if (nesContainerFormat(magic) !== undefined) {
|
||||
const parsed = parseInesLayout(magic, initial.size);
|
||||
throwIfTargetResolutionCancelled(signal);
|
||||
if (!parsed.ok)
|
||||
throw new BinaryTargetError(path, parsed.error, {
|
||||
constraint: "file_format_rejected",
|
||||
});
|
||||
return "ines";
|
||||
}
|
||||
if (hasZipSignature(magic)) return zipRootPackageFormat(lower);
|
||||
if (hasGodotPckHeader(magic)) return "godot-pck";
|
||||
if (hasUnityFsHeader(magic)) return "unityfs";
|
||||
if (hasUImageMagic(magic)) return "uimage";
|
||||
const hexRecords = detectHexRecordEncoding(magic);
|
||||
if (hexRecords !== undefined) return hexRecords;
|
||||
if (hasUf2Block(magic)) return "uf2";
|
||||
if (
|
||||
hasDfuSePrefix(magic) &&
|
||||
initial.size >= DFUSE_SUFFIX_BYTES &&
|
||||
isDfuSeFile(
|
||||
magic,
|
||||
await readHandleRange(
|
||||
handle,
|
||||
initial.size - DFUSE_SUFFIX_BYTES,
|
||||
DFUSE_SUFFIX_BYTES,
|
||||
signal,
|
||||
),
|
||||
initial.size,
|
||||
)
|
||||
)
|
||||
return "dfuse";
|
||||
// A C3 or S3 dump begins with its bootloader, so a layout comes first.
|
||||
const flash = await locateEspFlashLayout(
|
||||
initial.size,
|
||||
(position, length) => readHandleRange(handle, position, length, signal),
|
||||
{},
|
||||
{ explicit: false, ...(signal === undefined ? {} : { signal }) },
|
||||
);
|
||||
if (flash.ok) return "esp-flash-image";
|
||||
const esp = recognizeEspImage(magic, initial.size);
|
||||
if (esp === "esp8266")
|
||||
throw new BinaryTargetError(path, ESP8266_REFUSAL, {
|
||||
@@ -758,6 +883,7 @@ const isArchiveFormat = (
|
||||
isZipPackageFormat(format) ||
|
||||
[
|
||||
"asar",
|
||||
"ines",
|
||||
"godot-pck",
|
||||
"unityfs",
|
||||
"uimage",
|
||||
@@ -766,6 +892,9 @@ const isArchiveFormat = (
|
||||
"intel-hex",
|
||||
"srec",
|
||||
"esp-image",
|
||||
"uf2",
|
||||
"dfuse",
|
||||
"esp-flash-image",
|
||||
"fit",
|
||||
].includes(format);
|
||||
|
||||
@@ -843,12 +972,21 @@ const sha256Handle = async (
|
||||
throw new BinaryTargetError(path, "target grew while being read");
|
||||
}
|
||||
throwIfTargetResolutionCancelled(signal);
|
||||
const [opened, currentPath] = await Promise.all([handle.stat(), lstat(path)]);
|
||||
if (
|
||||
!sameRegularFileState(initial, opened) ||
|
||||
!sameRegularFileState(initial, currentPath) ||
|
||||
position !== initial.size
|
||||
)
|
||||
try {
|
||||
await assertStableRegularFileIdentity(handle, path, initial, {
|
||||
symlinks: "reject",
|
||||
...(signal === undefined ? {} : { signal }),
|
||||
});
|
||||
} catch (cause: unknown) {
|
||||
throwIfTargetResolutionCancelled(signal);
|
||||
if (cause instanceof RegularFileChangedError)
|
||||
throw new BinaryTargetError(path, "target changed while being read", {
|
||||
cause,
|
||||
});
|
||||
throw cause;
|
||||
}
|
||||
throwIfTargetResolutionCancelled(signal);
|
||||
if (position !== initial.size)
|
||||
throw new BinaryTargetError(path, "target changed while being read");
|
||||
return hash.digest("hex");
|
||||
};
|
||||
@@ -1029,8 +1167,26 @@ const readElfMetadata = async (
|
||||
const { fileSize, hostArchitecture, signal } = options;
|
||||
const parsed = parseExecutableHeader(bytes, hostArchitecture, fileSize);
|
||||
if (!parsed.ok || parsed.value.format !== "elf") return parsed;
|
||||
const ranges = await readElfDeclaredRanges(handle, bytes, fileSize, signal);
|
||||
const ranges = await readElfDeclarations(
|
||||
handle,
|
||||
bytes,
|
||||
validateElfDeclaredRanges(bytes, fileSize),
|
||||
signal,
|
||||
);
|
||||
if (!ranges.ok) return ranges;
|
||||
if (parsed.value.architecture === "riscv")
|
||||
return ok({
|
||||
...parsed.value,
|
||||
riscv: {
|
||||
...parsed.value.riscv,
|
||||
isa: await readElfDeclarations(
|
||||
handle,
|
||||
bytes,
|
||||
readRiscvIsaDeclaration(bytes, fileSize),
|
||||
signal,
|
||||
),
|
||||
},
|
||||
});
|
||||
if (parsed.value.architecture !== "mips" || parsed.value.mips.elfClass !== 32)
|
||||
return parsed;
|
||||
const declarations = await readMipsElfMetadata(
|
||||
@@ -1055,15 +1211,18 @@ const readElfMetadata = async (
|
||||
});
|
||||
};
|
||||
|
||||
/** Borrow the admitted descriptor; every request is at most one 64 KiB table chunk. */
|
||||
const readElfDeclaredRanges = async (
|
||||
/**
|
||||
* Satisfy one ELF declaration reader from the admitted descriptor. Every
|
||||
* request is at most one 64 KiB chunk, and the reader consumes each chunk
|
||||
* before its next request reuses the window.
|
||||
*/
|
||||
const readElfDeclarations = async <T>(
|
||||
handle: FileHandle,
|
||||
prefix: Buffer,
|
||||
fileSize: number,
|
||||
steps: Generator<ElfDeclarationRead, T, Buffer>,
|
||||
signal?: AbortSignal,
|
||||
): Promise<Result<null, string>> => {
|
||||
const ranges = validateElfDeclaredRanges(prefix, fileSize);
|
||||
let next = ranges.next();
|
||||
): Promise<T> => {
|
||||
let next = steps.next();
|
||||
let window: Buffer | undefined;
|
||||
while (!next.done) {
|
||||
throwIfTargetResolutionCancelled(signal);
|
||||
@@ -1088,7 +1247,7 @@ const readElfDeclaredRanges = async (
|
||||
}
|
||||
bytes = bytes.subarray(0, length);
|
||||
}
|
||||
next = ranges.next(bytes);
|
||||
next = steps.next(bytes);
|
||||
}
|
||||
return next.value;
|
||||
};
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
import type { EspFlashSelection } from "../domain/firmware/espFlashLocation.js";
|
||||
import type { ExecutableFormatHint } from "../domain/dosCom.js";
|
||||
import { dispatchQueuedInterrupts } from "../process/queuedInterrupts.js";
|
||||
import type {
|
||||
@@ -133,6 +134,7 @@ export const runDirectAnalysis = async (
|
||||
readonly formatHint?: ExecutableFormatHint;
|
||||
readonly architecture?: BinaryArchitecture;
|
||||
readonly entryPoint?: string;
|
||||
readonly espFlash?: EspFlashSelection;
|
||||
} = {},
|
||||
): Promise<Result<Evidence, JsonValue>> =>
|
||||
withProcessCancellation(options.signal, (signal) =>
|
||||
@@ -140,25 +142,50 @@ export const runDirectAnalysis = async (
|
||||
logger: options.logger ?? silentLogger,
|
||||
snapshotPath: options.snapshotPath,
|
||||
signal,
|
||||
...(options.formatHint === undefined
|
||||
? {}
|
||||
: { formatHint: options.formatHint }),
|
||||
...(options.architecture === undefined
|
||||
? {}
|
||||
: { architecture: options.architecture }),
|
||||
...(options.entryPoint === undefined
|
||||
? {}
|
||||
: { entryPoint: options.entryPoint }),
|
||||
...targetSelection(options),
|
||||
...(options.providerId === undefined
|
||||
? {}
|
||||
: { providerId: options.providerId }),
|
||||
}),
|
||||
);
|
||||
|
||||
/** How the caller asks an artifact target to be read. */
|
||||
export interface ArtifactTargetSelection {
|
||||
readonly formatHint?: ExecutableFormatHint;
|
||||
readonly espFlash?: EspFlashSelection;
|
||||
}
|
||||
|
||||
/** The caller's target interpretation, forwarded unchanged to every open. */
|
||||
const targetSelection = (options: {
|
||||
readonly formatHint?: ExecutableFormatHint;
|
||||
readonly architecture?: BinaryArchitecture;
|
||||
readonly entryPoint?: string;
|
||||
readonly espFlash?: EspFlashSelection;
|
||||
}) => ({
|
||||
...(options.formatHint === undefined
|
||||
? {}
|
||||
: { formatHint: options.formatHint }),
|
||||
...(options.architecture === undefined
|
||||
? {}
|
||||
: { architecture: options.architecture }),
|
||||
...(options.entryPoint === undefined
|
||||
? {}
|
||||
: { entryPoint: options.entryPoint }),
|
||||
...(options.espFlash === undefined ? {} : { espFlash: options.espFlash }),
|
||||
});
|
||||
|
||||
/** Execute one provider-native semantic operation with atomic provenance. */
|
||||
export const runProviderAnalysis = async (
|
||||
dependencies: DirectAnalysisDependencies,
|
||||
...[path, tool, arguments_, logger = silentLogger, signal, delivery]:
|
||||
...[
|
||||
path,
|
||||
tool,
|
||||
arguments_,
|
||||
logger = silentLogger,
|
||||
signal,
|
||||
delivery,
|
||||
selection,
|
||||
]:
|
||||
| readonly [
|
||||
path: string,
|
||||
tool: NativeToolName | ArtifactAnalysisOperation | ManagedToolName,
|
||||
@@ -177,6 +204,15 @@ export const runProviderAnalysis = async (
|
||||
readonly evidenceBundlePath?: string;
|
||||
},
|
||||
]
|
||||
| readonly [
|
||||
path: string,
|
||||
tool: ArtifactAnalysisOperation,
|
||||
arguments_: Readonly<Record<string, JsonValue>>,
|
||||
logger: Logger | undefined,
|
||||
signal: AbortSignal | undefined,
|
||||
delivery: undefined,
|
||||
selection: ArtifactTargetSelection,
|
||||
]
|
||||
): Promise<Result<Evidence, JsonValue>> => {
|
||||
if (delivery !== undefined && tool !== "inspect_managed_members")
|
||||
return err(
|
||||
@@ -248,6 +284,7 @@ export const runProviderAnalysis = async (
|
||||
snapshotPath: undefined,
|
||||
signal: operationSignal,
|
||||
evidenceParameters,
|
||||
...targetSelection(selection ?? {}),
|
||||
...(ARTIFACT_ANALYSIS_OPERATION_SET.has(tool)
|
||||
? { createSession: dependencies.createArtifactBinarySession }
|
||||
: NATIVE_INSPECTION_OPERATIONS.has(tool)
|
||||
@@ -328,6 +365,7 @@ const runAnalysis = async (
|
||||
readonly formatHint?: ExecutableFormatHint;
|
||||
readonly architecture?: BinaryArchitecture;
|
||||
readonly entryPoint?: string;
|
||||
readonly espFlash?: EspFlashSelection;
|
||||
},
|
||||
): Promise<Result<Evidence, JsonValue>> => {
|
||||
const { logger, signal, snapshotPath } = options;
|
||||
@@ -374,15 +412,7 @@ const runAnalysis = async (
|
||||
path,
|
||||
snapshotPath,
|
||||
signal,
|
||||
...(options.formatHint === undefined
|
||||
? {}
|
||||
: { formatHint: options.formatHint }),
|
||||
...(options.architecture === undefined
|
||||
? {}
|
||||
: { architecture: options.architecture }),
|
||||
...(options.entryPoint === undefined
|
||||
? {}
|
||||
: { entryPoint: options.entryPoint }),
|
||||
...targetSelection(options),
|
||||
});
|
||||
if (!prepared.ok) return err(prepared.error);
|
||||
const { snapshot } = prepared.value;
|
||||
@@ -391,15 +421,7 @@ const runAnalysis = async (
|
||||
const preview = await session.previewTarget(prepared.value.target, {
|
||||
signal,
|
||||
snapshot,
|
||||
...(options.formatHint === undefined
|
||||
? {}
|
||||
: { formatHint: options.formatHint }),
|
||||
...(options.architecture === undefined
|
||||
? {}
|
||||
: { architecture: options.architecture }),
|
||||
...(options.entryPoint === undefined
|
||||
? {}
|
||||
: { entryPoint: options.entryPoint }),
|
||||
...targetSelection(options),
|
||||
...(options.providerId === undefined
|
||||
? {}
|
||||
: { providerId: options.providerId }),
|
||||
@@ -438,15 +460,7 @@ const runAnalysis = async (
|
||||
}
|
||||
const openOptions = {
|
||||
signal,
|
||||
...(options.formatHint === undefined
|
||||
? {}
|
||||
: { formatHint: options.formatHint }),
|
||||
...(options.architecture === undefined
|
||||
? {}
|
||||
: { architecture: options.architecture }),
|
||||
...(options.entryPoint === undefined
|
||||
? {}
|
||||
: { entryPoint: options.entryPoint }),
|
||||
...targetSelection(options),
|
||||
...(snapshot === undefined ? {} : { snapshot }),
|
||||
...(options.providerId === undefined
|
||||
? {}
|
||||
@@ -706,6 +720,7 @@ const prepareSnapshot = async (options: {
|
||||
readonly formatHint?: ExecutableFormatHint;
|
||||
readonly architecture?: BinaryArchitecture;
|
||||
readonly entryPoint?: string;
|
||||
readonly espFlash?: EspFlashSelection;
|
||||
readonly snapshotPath: string | undefined;
|
||||
readonly signal: AbortSignal;
|
||||
}): Promise<
|
||||
@@ -724,15 +739,7 @@ const prepareSnapshot = async (options: {
|
||||
if (!loaded.ok) return loaded;
|
||||
const target = await session.resolveTarget(path, {
|
||||
signal: options.signal,
|
||||
...(options.formatHint === undefined
|
||||
? {}
|
||||
: { formatHint: options.formatHint }),
|
||||
...(options.architecture === undefined
|
||||
? {}
|
||||
: { architecture: options.architecture }),
|
||||
...(options.entryPoint === undefined
|
||||
? {}
|
||||
: { entryPoint: options.entryPoint }),
|
||||
...targetSelection(options),
|
||||
});
|
||||
if (!target.ok) return target;
|
||||
if (!snapshotMatchesTarget(loaded.value.target, target.value))
|
||||
|
||||
@@ -17,6 +17,7 @@ import {
|
||||
} from "../contracts/enhancedInputs.js";
|
||||
import {
|
||||
AnalysisCancelledError,
|
||||
AnalysisCapabilityUnavailableError,
|
||||
AnalysisOutputError,
|
||||
} from "../domain/analysisErrorCore.js";
|
||||
import { projectAnalysisError } from "../domain/analysisErrorProjection.js";
|
||||
@@ -100,8 +101,53 @@ type BatchDecompileItem = {
|
||||
readonly status: "ok" | "error";
|
||||
readonly pseudocode?: string;
|
||||
readonly error?: ReturnType<typeof projectAnalysisError>;
|
||||
/** Caveats of this item's own observation when it supplied no pseudocode. */
|
||||
readonly limitations?: string[];
|
||||
};
|
||||
|
||||
/**
|
||||
* Guidance for a batch item whose provider returned null pseudocode, by
|
||||
* whether that observation reported caveats for the item to carry.
|
||||
*/
|
||||
export const NULL_PSEUDOCODE_GUIDANCE = {
|
||||
reported: {
|
||||
userMessage:
|
||||
"The provider returned no pseudocode for this function; this item's limitations hold that observation's caveats, including any reason the provider reported.",
|
||||
remediationAction:
|
||||
"Read this item's limitations for the provider's reason before retrying; another provider may decompile this entry.",
|
||||
},
|
||||
none: {
|
||||
userMessage:
|
||||
"The provider returned no pseudocode for this function and reported no caveat or reason for it.",
|
||||
remediationAction:
|
||||
"The provider gave no further explanation; another provider may decompile this entry.",
|
||||
},
|
||||
} as const;
|
||||
|
||||
/**
|
||||
* The item error for a provider answer without usable text. Null is the
|
||||
* provider's observation that it has no pseudocode for this function; empty
|
||||
* or non-text pseudocode is unreadable output.
|
||||
*/
|
||||
const missingPseudocodeError = (
|
||||
providerId: string,
|
||||
pseudocode: JsonValue,
|
||||
caveats: keyof typeof NULL_PSEUDOCODE_GUIDANCE,
|
||||
): AnalysisError =>
|
||||
pseudocode === null
|
||||
? new AnalysisCapabilityUnavailableError(
|
||||
providerId,
|
||||
"procedure_pseudo_code",
|
||||
"provider returned null pseudocode",
|
||||
NULL_PSEUDOCODE_GUIDANCE[caveats],
|
||||
)
|
||||
: new AnalysisOutputError(
|
||||
"procedure_pseudo_code",
|
||||
typeof pseudocode === "string"
|
||||
? "provider returned empty pseudocode"
|
||||
: "provider returned non-text pseudocode",
|
||||
);
|
||||
|
||||
const isAnalysisCancellation = (
|
||||
error: AnalysisError,
|
||||
): error is AnalysisCancelledError => error._tag === "AnalysisCancelledError";
|
||||
@@ -491,40 +537,43 @@ export class EnhancedTools {
|
||||
): Promise<Result<BatchDecompileItem[], AnalysisCancelledError>> {
|
||||
const items: BatchDecompileItem[] = [];
|
||||
for (const { address, procedure, providerProcedure } of ready) {
|
||||
const result = await this.#call(
|
||||
const observed = await this.#observe(
|
||||
"procedure_pseudo_code",
|
||||
{ procedure: providerProcedure },
|
||||
signal,
|
||||
procedure.status === "resolved" ? "provider" : undefined,
|
||||
);
|
||||
if (!result.ok && isAnalysisCancellation(result.error))
|
||||
return err(result.error);
|
||||
const item: BatchDecompileItem = !result.ok
|
||||
? {
|
||||
address,
|
||||
procedure,
|
||||
status: "error",
|
||||
error: projectAnalysisError(result.error),
|
||||
}
|
||||
: typeof result.value !== "string" || result.value.length === 0
|
||||
? {
|
||||
address,
|
||||
procedure,
|
||||
status: "error",
|
||||
error: projectAnalysisError(
|
||||
new AnalysisOutputError(
|
||||
"procedure_pseudo_code",
|
||||
"provider returned empty pseudocode",
|
||||
),
|
||||
),
|
||||
}
|
||||
: {
|
||||
address,
|
||||
procedure,
|
||||
status: "ok",
|
||||
pseudocode: result.value,
|
||||
};
|
||||
items.push(item);
|
||||
if (!observed.ok) {
|
||||
if (isAnalysisCancellation(observed.error)) return err(observed.error);
|
||||
items.push({
|
||||
address,
|
||||
procedure,
|
||||
status: "error",
|
||||
error: projectAnalysisError(observed.error),
|
||||
});
|
||||
continue;
|
||||
}
|
||||
const { result: pseudocode, provider } = observed.value;
|
||||
if (typeof pseudocode === "string" && pseudocode.length > 0) {
|
||||
items.push({ address, procedure, status: "ok", pseudocode });
|
||||
continue;
|
||||
}
|
||||
// Only this observation can explain its missing text, so its caveats
|
||||
// stay with the item; the batch Evidence still aggregates them.
|
||||
const limitations = [...new Set(observed.value.limitations)];
|
||||
items.push({
|
||||
address,
|
||||
procedure,
|
||||
status: "error",
|
||||
error: projectAnalysisError(
|
||||
missingPseudocodeError(
|
||||
provider.id,
|
||||
pseudocode,
|
||||
limitations.length === 0 ? "none" : "reported",
|
||||
),
|
||||
),
|
||||
...(limitations.length === 0 ? {} : { limitations }),
|
||||
});
|
||||
}
|
||||
return ok(items);
|
||||
}
|
||||
|
||||
@@ -28,6 +28,7 @@ describe("exact Evidence input resolution", () => {
|
||||
expect(projected.remediation.action).toContain(
|
||||
"complete inline Evidence",
|
||||
);
|
||||
expect(projected.details).not.toHaveProperty("reference_scope");
|
||||
}
|
||||
});
|
||||
|
||||
|
||||
@@ -3,9 +3,20 @@ import type {
|
||||
EvidenceSourceInput,
|
||||
} from "../domain/evidenceReference.js";
|
||||
import type { Evidence } from "../domain/evidence.js";
|
||||
import type { z } from "zod";
|
||||
import {
|
||||
getEvidenceInputContract,
|
||||
type EvidenceInputPath,
|
||||
type EvidenceRequestFrameInput,
|
||||
type ResolvedEvidenceRequestInput,
|
||||
} from "../domain/evidenceInputContract.js";
|
||||
import { retainedEvidenceReferenceSchema } from "../domain/evidenceReference.js";
|
||||
import type { AnalysisError } from "../domain/analysisErrorBase.js";
|
||||
import { AnalysisCancelledError } from "../domain/analysisErrorCore.js";
|
||||
import { EvidenceReferenceError } from "../domain/evidenceErrors.js";
|
||||
import { err, ok, type Result } from "../domain/result.js";
|
||||
import { analysisInputErrorFromIssues } from "../domain/inputIssueProjection.js";
|
||||
import { parseAnalysisInput } from "./AnalysisInput.js";
|
||||
import {
|
||||
readEvidenceFileSource,
|
||||
type EvidenceFileReadOptions,
|
||||
@@ -14,6 +25,139 @@ import {
|
||||
/** Read-only lookup supplied by the current investigation's Evidence ledger. */
|
||||
export type EvidenceLookup = (evidenceId: string) => Evidence | undefined;
|
||||
|
||||
export interface EvidenceRequestResolutionOptions {
|
||||
readonly signal?: AbortSignal;
|
||||
readonly referenceScope?: "retained-closure";
|
||||
/** Optional format-owner admission, after lookup and before source use. */
|
||||
readonly readEvidence?: (
|
||||
input: unknown,
|
||||
path: EvidenceInputPath,
|
||||
) => Result<Evidence, AnalysisError>;
|
||||
}
|
||||
|
||||
interface EvidenceRequestResolution<Input> {
|
||||
readonly input: Input;
|
||||
readonly sources: readonly unknown[];
|
||||
}
|
||||
|
||||
/** An admitting owner may rely on complete Evidence at each declared source. */
|
||||
export function resolveEvidenceRequest<Schema extends z.ZodType>(
|
||||
operation: string,
|
||||
rawInput: unknown,
|
||||
schema: Schema,
|
||||
lookup: EvidenceLookup | undefined,
|
||||
options: EvidenceRequestResolutionOptions & {
|
||||
readonly readEvidence: NonNullable<
|
||||
EvidenceRequestResolutionOptions["readEvidence"]
|
||||
>;
|
||||
},
|
||||
): Result<
|
||||
EvidenceRequestResolution<ResolvedEvidenceRequestInput<z.output<Schema>>>,
|
||||
AnalysisError
|
||||
>;
|
||||
/** Source carriers remain unknown until their format owner admits them. */
|
||||
export function resolveEvidenceRequest<Schema extends z.ZodType>(
|
||||
operation: string,
|
||||
rawInput: unknown,
|
||||
schema: Schema,
|
||||
lookup?: EvidenceLookup,
|
||||
options?: EvidenceRequestResolutionOptions,
|
||||
): Result<
|
||||
EvidenceRequestResolution<EvidenceRequestFrameInput<z.output<Schema>>>,
|
||||
AnalysisError
|
||||
>;
|
||||
/** Resolve only schema-declared Evidence fields, sharing one lookup per exact ID. */
|
||||
export function resolveEvidenceRequest(
|
||||
operation: string,
|
||||
rawInput: unknown,
|
||||
schema: z.ZodType,
|
||||
lookup?: EvidenceLookup,
|
||||
options: EvidenceRequestResolutionOptions = {},
|
||||
): Result<EvidenceRequestResolution<unknown>, AnalysisError> {
|
||||
const contract = getEvidenceInputContract(schema);
|
||||
const parsed = parseAnalysisInput(
|
||||
operation,
|
||||
rawInput,
|
||||
contract.frameSchema,
|
||||
options.signal,
|
||||
);
|
||||
if (!parsed.ok) return parsed;
|
||||
const records = new Map<string, Evidence | undefined>();
|
||||
const read: EvidenceLookup = (id) => {
|
||||
if (!records.has(id)) records.set(id, lookup?.(id));
|
||||
return records.get(id);
|
||||
};
|
||||
const sources = new Set<unknown>();
|
||||
const result = contract.resolve<AnalysisError>(
|
||||
parsed.value,
|
||||
(input, path) => {
|
||||
if (options.signal?.aborted)
|
||||
return err(
|
||||
new AnalysisCancelledError(operation, {
|
||||
cause: options.signal.reason,
|
||||
}),
|
||||
);
|
||||
let selected = input;
|
||||
if (typeof input === "object" && input !== null && "kind" in input) {
|
||||
const reference = retainedEvidenceReferenceSchema.safeParse(input);
|
||||
if (!reference.success)
|
||||
return err(
|
||||
analysisInputErrorFromIssues(
|
||||
operation,
|
||||
reference.error.issues.map((issue) => ({
|
||||
...issue,
|
||||
path: [...path, ...issue.path],
|
||||
})),
|
||||
rawInput,
|
||||
{ cause: reference.error },
|
||||
),
|
||||
);
|
||||
const found = resolveEvidenceInput(
|
||||
reference.data,
|
||||
read,
|
||||
options.referenceScope,
|
||||
);
|
||||
if (!found.ok)
|
||||
return err(
|
||||
new EvidenceReferenceError(
|
||||
found.error.evidenceId,
|
||||
found.error.reason,
|
||||
{
|
||||
cause: found.error,
|
||||
expected: found.error.expected,
|
||||
actual: found.error.actual,
|
||||
referenceScope: found.error.referenceScope,
|
||||
diagnostics: { input_path: [...path] },
|
||||
},
|
||||
),
|
||||
);
|
||||
selected = found.value;
|
||||
}
|
||||
if (options.readEvidence !== undefined) {
|
||||
const admitted = options.readEvidence(selected, path);
|
||||
if (!admitted.ok) {
|
||||
if (admitted.error.diagnostics?.input_path === undefined)
|
||||
admitted.error.retainDiagnostics({ input_path: [...path] });
|
||||
return admitted;
|
||||
}
|
||||
selected = admitted.value;
|
||||
}
|
||||
sources.add(selected);
|
||||
return ok(selected);
|
||||
},
|
||||
);
|
||||
if (result.ok && options.signal?.aborted)
|
||||
return err(
|
||||
new AnalysisCancelledError(operation, { cause: options.signal.reason }),
|
||||
);
|
||||
return result.ok
|
||||
? ok({
|
||||
input: result.value,
|
||||
sources: [...sources],
|
||||
})
|
||||
: result;
|
||||
}
|
||||
|
||||
/** Normalize each selected record using its file context and the current ledger. */
|
||||
export interface EvidenceSourceResolutionOptions extends EvidenceFileReadOptions {
|
||||
readonly operation: string;
|
||||
@@ -24,70 +168,33 @@ export interface EvidenceSourceResolutionOptions extends EvidenceFileReadOptions
|
||||
) => Result<Evidence, AnalysisError>;
|
||||
}
|
||||
|
||||
/** Resolve an exact reference without selecting a target or running analysis. */
|
||||
/** Resolve an exact reference; callers declare when recovery requires retained closure. */
|
||||
export const resolveEvidenceInput = (
|
||||
input: EvidenceInput,
|
||||
lookup: EvidenceLookup | undefined,
|
||||
referenceScope?: EvidenceReferenceError["referenceScope"],
|
||||
): Result<Evidence, EvidenceReferenceError> => {
|
||||
if (!("kind" in input)) return ok(input);
|
||||
const record = lookup?.(input.evidence_id);
|
||||
if (record === undefined)
|
||||
return err(
|
||||
new EvidenceReferenceError(
|
||||
input.evidence_id,
|
||||
"missing",
|
||||
"Evidence available to the selected source",
|
||||
null,
|
||||
),
|
||||
new EvidenceReferenceError(input.evidence_id, "missing", {
|
||||
expected: "Evidence available to the selected source",
|
||||
actual: null,
|
||||
referenceScope,
|
||||
}),
|
||||
);
|
||||
if (record.evidence_id !== input.evidence_id)
|
||||
return err(
|
||||
new EvidenceReferenceError(
|
||||
input.evidence_id,
|
||||
"identity_mismatch",
|
||||
input.evidence_id,
|
||||
record.evidence_id,
|
||||
),
|
||||
new EvidenceReferenceError(input.evidence_id, "identity_mismatch", {
|
||||
expected: input.evidence_id,
|
||||
actual: record.evidence_id,
|
||||
}),
|
||||
);
|
||||
return ok(record);
|
||||
};
|
||||
|
||||
/** Resolve an application's Evidence before its existing semantic validation. */
|
||||
export const resolveApplicationEvidenceRequest = <
|
||||
T extends { readonly application: EvidenceInput },
|
||||
>(
|
||||
input: T,
|
||||
lookup: EvidenceLookup | undefined,
|
||||
): Result<
|
||||
Omit<T, "application"> & { readonly application: Evidence },
|
||||
EvidenceReferenceError
|
||||
> => {
|
||||
const application = resolveEvidenceInput(input.application, lookup);
|
||||
if (!application.ok) return application;
|
||||
return ok({ ...input, application: application.value });
|
||||
};
|
||||
|
||||
/** Resolve explicitly paired Evidence without guessing another session's record. */
|
||||
export const resolvePairedEvidenceRequest = <
|
||||
T extends { readonly left: EvidenceInput; readonly right: EvidenceInput },
|
||||
>(
|
||||
input: T,
|
||||
lookup: EvidenceLookup | undefined,
|
||||
): Result<
|
||||
Omit<T, "left" | "right"> & {
|
||||
readonly left: Evidence;
|
||||
readonly right: Evidence;
|
||||
},
|
||||
EvidenceReferenceError
|
||||
> => {
|
||||
const left = resolveEvidenceInput(input.left, lookup);
|
||||
if (!left.ok) return left;
|
||||
const right = resolveEvidenceInput(input.right, lookup);
|
||||
if (!right.ok) return right;
|
||||
return ok({ ...input, left: left.value, right: right.value });
|
||||
};
|
||||
|
||||
/** Resolve paired sources like resolvePairedEvidenceRequest; Evidence files use the shared file reader. */
|
||||
/** Resolve explicit paired sources; Evidence files use the shared file reader. */
|
||||
export const resolvePairedEvidenceSources = async <
|
||||
T extends {
|
||||
readonly left: EvidenceSourceInput;
|
||||
|
||||
@@ -0,0 +1,375 @@
|
||||
import { describe, expect, expectTypeOf, it } from "vitest";
|
||||
import { z } from "zod";
|
||||
|
||||
import { createEvidence, type Evidence } from "../domain/evidence.js";
|
||||
import {
|
||||
borrowEvidenceArray,
|
||||
borrowEvidenceInput,
|
||||
evidenceInputSchema,
|
||||
} from "../domain/evidenceReference.js";
|
||||
import { getEvidenceInputContract } from "../domain/evidenceInputContract.js";
|
||||
import { readImmutableEvidence } from "../domain/evidenceValidation.js";
|
||||
import { AnalysisInputError } from "../domain/analysisErrorCore.js";
|
||||
import {
|
||||
EvidenceIntegrityError,
|
||||
EvidenceReferenceError,
|
||||
} from "../domain/evidenceErrors.js";
|
||||
import { err } from "../domain/result.js";
|
||||
import {
|
||||
resolveEvidenceRequest,
|
||||
type EvidenceRequestResolutionOptions,
|
||||
} from "./EvidenceInputResolver.js";
|
||||
|
||||
const evidence = createEvidence(
|
||||
undefined,
|
||||
{
|
||||
id: "test-evidence-input",
|
||||
name: "Evidence input regression",
|
||||
version: "1",
|
||||
},
|
||||
{
|
||||
operation: "test_observation",
|
||||
parameters: {},
|
||||
result: { value: 42 },
|
||||
},
|
||||
);
|
||||
const reference = {
|
||||
kind: "retained-evidence" as const,
|
||||
evidence_id: evidence.evidence_id,
|
||||
};
|
||||
|
||||
describe("schema-owned Evidence request resolution", () => {
|
||||
it("resolves declared single, nested and array sources once per exact ID", () => {
|
||||
const schema = z.strictObject({
|
||||
source: evidenceInputSchema.describe("Owner-described source"),
|
||||
layers: z.array(z.strictObject({ analysis: evidenceInputSchema })),
|
||||
optional: evidenceInputSchema.optional(),
|
||||
ordinary: z.unknown(),
|
||||
});
|
||||
let lookups = 0;
|
||||
const ordinary = { kind: "retained-evidence", evidence_id: "display-data" };
|
||||
const result = resolveEvidenceRequest(
|
||||
"test",
|
||||
{
|
||||
source: reference,
|
||||
layers: [{ analysis: reference }, { analysis: evidence }],
|
||||
ordinary,
|
||||
},
|
||||
schema,
|
||||
() => {
|
||||
lookups += 1;
|
||||
return evidence;
|
||||
},
|
||||
);
|
||||
if (!result.ok) throw result.error;
|
||||
expectTypeOf(result.value.input.source).toBeUnknown();
|
||||
expectTypeOf(result.value.input.layers).toEqualTypeOf<
|
||||
{ analysis: unknown }[]
|
||||
>();
|
||||
expect(lookups).toBe(1);
|
||||
expect(result.value.input.source).toBe(evidence);
|
||||
expect(result.value.input.layers.map(({ analysis }) => analysis)).toEqual([
|
||||
evidence,
|
||||
evidence,
|
||||
]);
|
||||
expect(result.value.input).not.toHaveProperty("optional");
|
||||
expect(result.value.input.ordinary).toBe(ordinary);
|
||||
expect(result.value.sources).toEqual([evidence]);
|
||||
});
|
||||
|
||||
it("borrows inline values until the format owner authenticates them", () => {
|
||||
const schema = z.strictObject({ source: evidenceInputSchema });
|
||||
const opaque = Object.defineProperty({}, "normalized_result", {
|
||||
get: () => {
|
||||
throw new Error("premature payload access");
|
||||
},
|
||||
});
|
||||
const result = resolveEvidenceRequest("test", { source: opaque }, schema);
|
||||
if (!result.ok) throw result.error;
|
||||
expectTypeOf(result.value.input.source).toBeUnknown();
|
||||
expect(result.value.input.source).toBe(opaque);
|
||||
expect(result.value.sources).toEqual([opaque]);
|
||||
const admitted = resolveEvidenceRequest(
|
||||
"test",
|
||||
{ source: evidence },
|
||||
schema,
|
||||
undefined,
|
||||
{ readEvidence: readImmutableEvidence },
|
||||
);
|
||||
if (!admitted.ok) throw admitted.error;
|
||||
expectTypeOf(admitted.value.input.source).toEqualTypeOf<Evidence>();
|
||||
expect(admitted.value.input.source).toEqual(evidence);
|
||||
expect(admitted.value.sources).toEqual([admitted.value.input.source]);
|
||||
const optionalAdmission: EvidenceRequestResolutionOptions = {};
|
||||
const borrowed = resolveEvidenceRequest(
|
||||
"test",
|
||||
{ source: null },
|
||||
schema,
|
||||
undefined,
|
||||
optionalAdmission,
|
||||
);
|
||||
if (!borrowed.ok) throw borrowed.error;
|
||||
expectTypeOf(borrowed.value.input.source).toBeUnknown();
|
||||
expect(borrowed.value.input.source).toBeNull();
|
||||
const rejected = resolveEvidenceRequest(
|
||||
"test",
|
||||
{ source: null },
|
||||
schema,
|
||||
undefined,
|
||||
{ readEvidence: readImmutableEvidence },
|
||||
);
|
||||
expect(rejected.ok).toBe(false);
|
||||
if (!rejected.ok)
|
||||
expect(rejected.error.diagnostics).toEqual({ input_path: ["source"] });
|
||||
});
|
||||
|
||||
it.each([{ ownerPath: undefined }, { ownerPath: ["payload", 2] }])(
|
||||
"preserves source-owner failures and existing paths: %j",
|
||||
({ ownerPath }) => {
|
||||
const failure = new EvidenceIntegrityError("Invalid source", {
|
||||
diagnostics: {
|
||||
owner: "fixture",
|
||||
...(ownerPath === undefined ? {} : { input_path: ownerPath }),
|
||||
},
|
||||
});
|
||||
let reads = 0;
|
||||
const result = resolveEvidenceRequest(
|
||||
"test",
|
||||
{ layers: [{ source: {} }] },
|
||||
z.strictObject({
|
||||
layers: z.array(z.strictObject({ source: evidenceInputSchema })),
|
||||
}),
|
||||
undefined,
|
||||
{
|
||||
readEvidence: () => {
|
||||
reads += 1;
|
||||
return err(failure);
|
||||
},
|
||||
},
|
||||
);
|
||||
if (result.ok) throw new Error("Invalid source was accepted");
|
||||
expect(result.error).toBe(failure);
|
||||
expect(reads).toBe(1);
|
||||
expect(failure.diagnostics).toEqual({
|
||||
owner: "fixture",
|
||||
input_path: ownerPath ?? ["layers", 0, "source"],
|
||||
});
|
||||
},
|
||||
);
|
||||
|
||||
it("preserves optional/default/nullable source selection and envelope constraints", () => {
|
||||
const schema = z.strictObject({
|
||||
sources: z.array(evidenceInputSchema).min(1).default([reference]),
|
||||
absent: evidenceInputSchema.exactOptional(),
|
||||
nullable: evidenceInputSchema.nullable(),
|
||||
limit: z.int().positive(),
|
||||
});
|
||||
const result = resolveEvidenceRequest(
|
||||
"test",
|
||||
{
|
||||
nullable: null,
|
||||
limit: 1,
|
||||
},
|
||||
schema,
|
||||
() => evidence,
|
||||
);
|
||||
if (!result.ok) throw result.error;
|
||||
expectTypeOf(result.value.input.limit).toEqualTypeOf<number>();
|
||||
expect(result.value.input.sources).toEqual([evidence]);
|
||||
expect(result.value.input.nullable).toBeNull();
|
||||
expect(result.value.input).not.toHaveProperty("absent");
|
||||
const rejected = resolveEvidenceRequest(
|
||||
"test",
|
||||
{
|
||||
sources: [],
|
||||
nullable: null,
|
||||
limit: 0,
|
||||
},
|
||||
schema,
|
||||
);
|
||||
if (rejected.ok) throw new Error("invalid envelope was admitted");
|
||||
expect(rejected.error).toBeInstanceOf(AnalysisInputError);
|
||||
});
|
||||
});
|
||||
|
||||
describe("Evidence request branches and source diagnostics", () => {
|
||||
it("preserves selected union defaults and overwrites without running them twice", () => {
|
||||
let refinements = 0;
|
||||
const schema = z
|
||||
.union([
|
||||
z.strictObject({
|
||||
kind: z.literal("selected"),
|
||||
source: evidenceInputSchema,
|
||||
label: z.string().overwrite((value) => value + "!"),
|
||||
}),
|
||||
z.strictObject({ kind: z.literal("plain"), value: z.string() }),
|
||||
])
|
||||
.superRefine(() => {
|
||||
refinements += 1;
|
||||
});
|
||||
const result = resolveEvidenceRequest(
|
||||
"test",
|
||||
{
|
||||
kind: "selected",
|
||||
source: reference,
|
||||
label: "once",
|
||||
},
|
||||
schema,
|
||||
() => evidence,
|
||||
);
|
||||
if (!result.ok) throw result.error;
|
||||
expect(result.value.input).toEqual({
|
||||
kind: "selected",
|
||||
source: evidence,
|
||||
label: "once!",
|
||||
});
|
||||
expect(refinements).toBe(1);
|
||||
const plain = resolveEvidenceRequest(
|
||||
"test",
|
||||
{
|
||||
kind: "plain",
|
||||
value: "literal",
|
||||
},
|
||||
schema,
|
||||
);
|
||||
if (!plain.ok) throw plain.error;
|
||||
expect(plain.value.sources).toEqual([]);
|
||||
});
|
||||
|
||||
it("preserves nested optional malformed carriers for typed owner rejection", () => {
|
||||
const schema = z.strictObject({
|
||||
sources: z.array(
|
||||
z.strictObject({ source: evidenceInputSchema.optional() }),
|
||||
),
|
||||
});
|
||||
const result = resolveEvidenceRequest(
|
||||
"test",
|
||||
{
|
||||
sources: [{ source: null }],
|
||||
},
|
||||
schema,
|
||||
undefined,
|
||||
{ readEvidence: readImmutableEvidence },
|
||||
);
|
||||
expect(result.ok).toBe(false);
|
||||
});
|
||||
|
||||
it("reports malformed reference paths without treating arbitrary objects as refs", () => {
|
||||
const schema = z.strictObject({
|
||||
layers: z.array(
|
||||
z.strictObject({
|
||||
source: evidenceInputSchema,
|
||||
}),
|
||||
),
|
||||
});
|
||||
const result = resolveEvidenceRequest(
|
||||
"test",
|
||||
{
|
||||
layers: [{ source: { kind: "retained-evidence" } }],
|
||||
},
|
||||
schema,
|
||||
);
|
||||
if (result.ok) throw new Error("missing ID was accepted");
|
||||
expect(result.error).toBeInstanceOf(AnalysisInputError);
|
||||
expect(result.error).toMatchObject({
|
||||
issues: [
|
||||
expect.objectContaining({
|
||||
path: ["layers", 0, "source", "evidence_id"],
|
||||
}),
|
||||
],
|
||||
});
|
||||
});
|
||||
|
||||
it("retains the exact missing/incorrect identity and its source path", () => {
|
||||
const schema = z.strictObject({ source: evidenceInputSchema });
|
||||
const missing = resolveEvidenceRequest(
|
||||
"test",
|
||||
{ source: reference },
|
||||
schema,
|
||||
undefined,
|
||||
{ referenceScope: "retained-closure" },
|
||||
);
|
||||
if (missing.ok) throw new Error("missing reference was accepted");
|
||||
expect(missing.error).toBeInstanceOf(EvidenceReferenceError);
|
||||
expect(missing.error).toMatchObject({
|
||||
reason: "missing",
|
||||
referenceScope: "retained-closure",
|
||||
diagnostics: { input_path: ["source"] },
|
||||
});
|
||||
const different = { ...reference, evidence_id: `ev_${"f".repeat(64)}` };
|
||||
const mismatched = resolveEvidenceRequest(
|
||||
"test",
|
||||
{ source: different },
|
||||
schema,
|
||||
() => evidence,
|
||||
);
|
||||
if (mismatched.ok) throw new Error("mismatched record was accepted");
|
||||
expect(mismatched.error).toMatchObject({ reason: "identity_mismatch" });
|
||||
});
|
||||
});
|
||||
|
||||
describe("Evidence request borrowing and cancellation", () => {
|
||||
it("keeps borrowed SDK selections untouched before service capacity admission", () => {
|
||||
const schema = z.strictObject({
|
||||
source: borrowEvidenceInput(),
|
||||
observations: borrowEvidenceArray().default([]),
|
||||
});
|
||||
const observations = [reference, evidence];
|
||||
const parsed = getEvidenceInputContract(schema).frameSchema.parse({
|
||||
source: evidence,
|
||||
observations,
|
||||
});
|
||||
expect(parsed.source).toBe(evidence);
|
||||
expect(parsed.observations).toBe(observations);
|
||||
const result = resolveEvidenceRequest(
|
||||
"test",
|
||||
parsed,
|
||||
schema,
|
||||
() => evidence,
|
||||
);
|
||||
if (!result.ok) throw result.error;
|
||||
expect(result.value.input.observations).toEqual([evidence, evidence]);
|
||||
expect(observations[0]).toBe(reference);
|
||||
});
|
||||
|
||||
it("honors cancellation before parsing or resolving selected sources", () => {
|
||||
const controller = new AbortController();
|
||||
const reason = new Error("cancelled by caller");
|
||||
controller.abort(reason);
|
||||
let lookups = 0;
|
||||
const result = resolveEvidenceRequest(
|
||||
"test",
|
||||
{ source: reference },
|
||||
z.strictObject({ source: evidenceInputSchema }),
|
||||
() => {
|
||||
lookups += 1;
|
||||
return evidence;
|
||||
},
|
||||
{ signal: controller.signal },
|
||||
);
|
||||
if (result.ok) throw new Error("cancelled request was admitted");
|
||||
expect(result.error.cause).toBe(reason);
|
||||
expect(lookups).toBe(0);
|
||||
});
|
||||
|
||||
it("observes cancellation raised by the final source owner", () => {
|
||||
const controller = new AbortController();
|
||||
const reason = new Error("cancelled during source admission");
|
||||
const result = resolveEvidenceRequest(
|
||||
"test",
|
||||
{ source: reference },
|
||||
z.strictObject({ source: evidenceInputSchema }),
|
||||
() => evidence,
|
||||
{
|
||||
signal: controller.signal,
|
||||
readEvidence: (input) => {
|
||||
const read = readImmutableEvidence(input);
|
||||
controller.abort(reason);
|
||||
return read;
|
||||
},
|
||||
},
|
||||
);
|
||||
if (result.ok) throw new Error("late source cancellation was lost");
|
||||
expect(result.error.cause).toBe(reason);
|
||||
});
|
||||
});
|
||||
@@ -1,3 +1,8 @@
|
||||
import {
|
||||
resolveEvidenceRequest,
|
||||
type EvidenceLookup,
|
||||
} from "./EvidenceInputResolver.js";
|
||||
import type { ExecutionOptions } from "./AnalysisProvider.js";
|
||||
import type { AnalysisInputError } from "../domain/analysisErrorCore.js";
|
||||
import type { AnalysisError } from "../domain/analysisErrorBase.js";
|
||||
import {
|
||||
@@ -8,7 +13,10 @@ import {
|
||||
} from "../domain/evidence.js";
|
||||
import type { JsonValue } from "../domain/jsonValue.js";
|
||||
import type { ParsedInventorySet } from "../domain/artifactInventoryEvidence.js";
|
||||
import { readApplicationInventoryProjection } from "../domain/mobileApplicationGraphProjection.js";
|
||||
import {
|
||||
applicationInventoryProjectionInputSchema,
|
||||
readApplicationInventoryProjection,
|
||||
} from "../domain/mobileApplicationGraphProjection.js";
|
||||
import { ok, type Result } from "../domain/result.js";
|
||||
|
||||
interface InventoryProjectionResult {
|
||||
@@ -26,6 +34,8 @@ interface InventoryProjectionOptions<
|
||||
Output extends InventoryProjectionResult & JsonValue,
|
||||
> {
|
||||
readonly rawInput: unknown;
|
||||
readonly lookup?: EvidenceLookup;
|
||||
readonly execution?: Pick<ExecutionOptions, "signal">;
|
||||
readonly project: (
|
||||
input: ParsedInventorySet,
|
||||
) => Result<Output, AnalysisInputError>;
|
||||
@@ -41,8 +51,16 @@ export const projectInventoryEvidence = <
|
||||
>(
|
||||
options: InventoryProjectionOptions<Output>,
|
||||
): Result<InventoryProjectionEvidenceResult, AnalysisError> => {
|
||||
const source = readApplicationInventoryProjection(
|
||||
const request = resolveEvidenceRequest(
|
||||
options.operation,
|
||||
options.rawInput,
|
||||
applicationInventoryProjectionInputSchema,
|
||||
options.lookup,
|
||||
options.execution,
|
||||
);
|
||||
if (!request.ok) return request;
|
||||
const source = readApplicationInventoryProjection(
|
||||
request.value.input,
|
||||
options.operation,
|
||||
);
|
||||
if (!source.ok) return source;
|
||||
|
||||
@@ -38,6 +38,12 @@ const APPLICATION_NODE_POLICIES: Readonly<
|
||||
"negative",
|
||||
"malformed",
|
||||
]),
|
||||
// Scheme links reach the running application through the OS or a session.
|
||||
"url-scheme": policy("electron", "url-scheme", "packaged-process", [
|
||||
"positive",
|
||||
"negative",
|
||||
"malformed",
|
||||
]),
|
||||
endpoint: policy("protocol", "endpoint", "protocol", [
|
||||
"positive",
|
||||
"negative",
|
||||
|
||||
@@ -13,11 +13,9 @@ export const workflowAnalysisProfile = (
|
||||
createAnalysisProfile(REA_WORKFLOW_PROVIDER, {
|
||||
upstream_analysis_profile: upstream,
|
||||
source_limitations: "provider-operation-caveats-v1",
|
||||
...(operation === "binary_overview"
|
||||
...(operation === "binary_overview" || operation === "batch_decompile"
|
||||
? { workflow_revision: 4 }
|
||||
: operation === "batch_decompile" ||
|
||||
operation === "get_call_graph" ||
|
||||
operation === "trace_feature"
|
||||
: operation === "get_call_graph" || operation === "trace_feature"
|
||||
? { workflow_revision: 3 }
|
||||
: { workflow_revision: 1 }),
|
||||
});
|
||||
@@ -1,3 +1,5 @@
|
||||
import type { EvidenceLookup } from "../EvidenceInputResolver.js";
|
||||
import type { ExecutionOptions } from "../AnalysisProvider.js";
|
||||
import type { AnalysisError } from "../../domain/analysisErrorBase.js";
|
||||
import { projectAndroidInventory } from "../../domain/android/androidApplication.js";
|
||||
import type { Result } from "../../domain/result.js";
|
||||
@@ -12,9 +14,13 @@ const OPERATION = "project_android_application_graph" as const;
|
||||
/** Project authenticated APK inventory Evidence into Android application Evidence. */
|
||||
export const projectAndroidApplicationEvidence = (
|
||||
rawInput: unknown,
|
||||
lookup?: EvidenceLookup,
|
||||
execution?: Pick<ExecutionOptions, "signal">,
|
||||
): Result<InventoryProjectionEvidenceResult, AnalysisError> => {
|
||||
return projectInventoryEvidence({
|
||||
rawInput,
|
||||
...(lookup === undefined ? {} : { lookup }),
|
||||
...(execution === undefined ? {} : { execution }),
|
||||
project: projectAndroidInventory,
|
||||
operation: OPERATION,
|
||||
predicateType: "rea.android-application-graph",
|
||||
|
||||
@@ -1,3 +1,5 @@
|
||||
import type { EvidenceLookup } from "../EvidenceInputResolver.js";
|
||||
import type { ExecutionOptions } from "../AnalysisProvider.js";
|
||||
import type { AnalysisError } from "../../domain/analysisErrorBase.js";
|
||||
import { projectAppleInventory } from "../../domain/apple/appleApplication.js";
|
||||
import type { Result } from "../../domain/result.js";
|
||||
@@ -12,9 +14,13 @@ const OPERATION = "project_apple_application_graph" as const;
|
||||
/** Project authenticated IPA or macOS app inventory Evidence into Apple application evidence. */
|
||||
export const projectAppleApplicationEvidence = (
|
||||
rawInput: unknown,
|
||||
lookup?: EvidenceLookup,
|
||||
execution?: Pick<ExecutionOptions, "signal">,
|
||||
): Result<InventoryProjectionEvidenceResult, AnalysisError> => {
|
||||
return projectInventoryEvidence({
|
||||
rawInput,
|
||||
...(lookup === undefined ? {} : { lookup }),
|
||||
...(execution === undefined ? {} : { execution }),
|
||||
project: projectAppleInventory,
|
||||
operation: OPERATION,
|
||||
predicateType: "rea.apple-application-graph",
|
||||
|
||||
@@ -7,7 +7,8 @@ import { AnalysisError } from "../../domain/analysisErrorBase.js";
|
||||
import { AnalysisInputError } from "../../domain/analysisErrorCore.js";
|
||||
import { inspectAnalysisViewValidated } from "../analysisView/AnalysisViewService.js";
|
||||
import type { EvidenceFileReadOptions } from "../analysisView/evidenceFileSource.js";
|
||||
import type { ArtifactComparisonInput } from "../../contracts/sessionToolSchemas.js";
|
||||
import { artifactComparisonInputSchema } from "../../contracts/artifactComparisonContracts.js";
|
||||
import { parseAnalysisInput } from "../AnalysisInput.js";
|
||||
import { artifactInventorySummaryViewResultSchema } from "../../domain/analysisView/analysisView.js";
|
||||
import {
|
||||
compareArtifacts,
|
||||
@@ -46,11 +47,18 @@ export interface ArtifactComparisonOutcome {
|
||||
* then resolve a summary to its exact retained parent inventory.
|
||||
*/
|
||||
export const resolveArtifactComparisonSources = async (
|
||||
input: ArtifactComparisonInput,
|
||||
input: unknown,
|
||||
evidenceById: EvidenceLookup | undefined,
|
||||
options: EvidenceFileReadOptions = {},
|
||||
): Promise<Result<ArtifactComparisonSources, AnalysisError>> => {
|
||||
return resolvePairedEvidenceSources(input, evidenceById, {
|
||||
const request = parseAnalysisInput(
|
||||
OPERATION,
|
||||
input,
|
||||
artifactComparisonInputSchema,
|
||||
options.signal,
|
||||
);
|
||||
if (!request.ok) return request;
|
||||
return resolvePairedEvidenceSources(request.value, evidenceById, {
|
||||
...options,
|
||||
operation: OPERATION,
|
||||
resolveEvidence: (candidate, related, side) => {
|
||||
@@ -130,12 +138,10 @@ export const resolveArtifactComparisonEvidence = (
|
||||
const parent = evidenceById?.(parentId);
|
||||
if (parent === undefined)
|
||||
return err(
|
||||
new EvidenceReferenceError(
|
||||
parentId,
|
||||
"missing",
|
||||
"retained complete artifact inventory Evidence",
|
||||
null,
|
||||
),
|
||||
new EvidenceReferenceError(parentId, "missing", {
|
||||
expected: "retained complete artifact inventory Evidence",
|
||||
actual: null,
|
||||
}),
|
||||
);
|
||||
const selected = resolveEvidenceInput(
|
||||
{ kind: "retained-evidence", evidence_id: parentId },
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
import type { EspFlashSelection } from "../../domain/firmware/espFlashLocation.js";
|
||||
import type { ExecutableFormatHint } from "../../domain/dosCom.js";
|
||||
import type {
|
||||
BinaryArchitecture,
|
||||
@@ -38,6 +39,8 @@ export interface BinarySessionOpenOptions {
|
||||
readonly formatHint?: ExecutableFormatHint;
|
||||
readonly architecture?: BinaryArchitecture;
|
||||
readonly entryPoint?: string;
|
||||
/** Inputs for format=esp-flash-image. */
|
||||
readonly espFlash?: EspFlashSelection;
|
||||
readonly snapshot?: AnalysisSnapshot;
|
||||
/**
|
||||
* Load the snapshot inside the open's turn instead of before it, so calls
|
||||
@@ -101,6 +104,7 @@ export const resolveSessionOpen = async (
|
||||
...(options.entryPoint === undefined
|
||||
? {}
|
||||
: { entryPoint: options.entryPoint }),
|
||||
...(options.espFlash === undefined ? {} : { espFlash: options.espFlash }),
|
||||
...(options.signal === undefined ? {} : { signal: options.signal }),
|
||||
});
|
||||
if (!parsed.ok) return parsed;
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
import type { EspFlashSelection } from "../../domain/firmware/espFlashLocation.js";
|
||||
import type { InvestigationRecordPort } from "../investigation/InvestigationRecordPort.js";
|
||||
import type { ExecutableFormatHint } from "../../domain/dosCom.js";
|
||||
import type {
|
||||
@@ -64,6 +65,7 @@ export interface BinarySessionPort
|
||||
readonly formatHint?: ExecutableFormatHint;
|
||||
readonly architecture?: BinaryArchitecture;
|
||||
readonly entryPoint?: string;
|
||||
readonly espFlash?: EspFlashSelection;
|
||||
readonly snapshot?: AnalysisSnapshot;
|
||||
readonly loadSnapshot?: AnalysisSnapshotLoader;
|
||||
readonly providerId?: AnalysisProviderSelector;
|
||||
|
||||
@@ -13,4 +13,10 @@ export const binaryTargetSummary = (target: BinaryTarget) =>
|
||||
...(target.format === "raw-binary" || target.format === "gamecube-dol"
|
||||
? { entry_point: target.entryPoint }
|
||||
: {}),
|
||||
...(target.format === "esp-flash-image"
|
||||
? {
|
||||
flash_offset: target.espFlash.flashOffset,
|
||||
partition_table_offset: target.espFlash.partitionTableOffset,
|
||||
}
|
||||
: {}),
|
||||
});
|
||||
@@ -117,14 +117,22 @@ const TARGETS: TargetFixtures = {
|
||||
hsp: { ...archive, format: "hsp" },
|
||||
"app-pack": { ...archive, format: "app-pack" },
|
||||
asar: { ...archive, format: "asar" },
|
||||
ines: { ...archive, format: "ines" },
|
||||
"godot-pck": { ...archive, format: "godot-pck" },
|
||||
unityfs: { ...archive, format: "unityfs" },
|
||||
uimage: { ...archive, format: "uimage" },
|
||||
uf2: { ...archive, format: "uf2" },
|
||||
dfuse: { ...archive, format: "dfuse" },
|
||||
dmg: { ...archive, format: "dmg" },
|
||||
pkg: { ...archive, format: "pkg" },
|
||||
"intel-hex": { ...archive, format: "intel-hex" },
|
||||
srec: { ...archive, format: "srec" },
|
||||
"esp-image": { ...archive, format: "esp-image" },
|
||||
"esp-flash-image": {
|
||||
...archive,
|
||||
format: "esp-flash-image",
|
||||
espFlash: { selected: true, flashOffset: null, partitionTableOffset: null },
|
||||
},
|
||||
fit: { ...archive, format: "fit" },
|
||||
plist: { ...artifact, format: "plist" },
|
||||
javascript: { ...artifact, format: "javascript" },
|
||||
@@ -169,9 +177,13 @@ describe("target format vocabularies", () => {
|
||||
);
|
||||
|
||||
it.each(executableFormatHintSchema.options)(
|
||||
"admits the caller-selected %s hint only as an executable target format",
|
||||
"admits the caller-selected %s hint only as its own target format",
|
||||
(hint) => {
|
||||
expect(TARGETS[hint]).toMatchObject({ format: hint, kind: "executable" });
|
||||
// An ESP flash dump is the one caller-selected format read as an archive.
|
||||
expect(TARGETS[hint]).toMatchObject({
|
||||
format: hint,
|
||||
kind: hint === "esp-flash-image" ? "archive" : "executable",
|
||||
});
|
||||
},
|
||||
);
|
||||
});
|
||||
@@ -1,3 +1,5 @@
|
||||
import type { EvidenceLookup } from "../EvidenceInputResolver.js";
|
||||
import type { ExecutionOptions } from "../AnalysisProvider.js";
|
||||
import type { AnalysisError } from "../../domain/analysisErrorBase.js";
|
||||
import { projectHarmonyInventory } from "../../domain/harmony/harmonyApplication.js";
|
||||
import type { Result } from "../../domain/result.js";
|
||||
@@ -12,9 +14,13 @@ const OPERATION = "project_harmony_application_graph" as const;
|
||||
/** Project authenticated HarmonyOS package inventory Evidence into application evidence. */
|
||||
export const projectHarmonyApplicationEvidence = (
|
||||
rawInput: unknown,
|
||||
lookup?: EvidenceLookup,
|
||||
execution?: Pick<ExecutionOptions, "signal">,
|
||||
): Result<InventoryProjectionEvidenceResult, AnalysisError> => {
|
||||
return projectInventoryEvidence({
|
||||
rawInput,
|
||||
...(lookup === undefined ? {} : { lookup }),
|
||||
...(execution === undefined ? {} : { execution }),
|
||||
project: projectHarmonyInventory,
|
||||
operation: OPERATION,
|
||||
predicateType: "rea.harmony-application-graph",
|
||||
|
||||
@@ -0,0 +1,7 @@
|
||||
import type { Evidence } from "../../domain/evidence.js";
|
||||
|
||||
/** Completed workflow observation and the authenticated sources it consumed. */
|
||||
export interface EvidenceWorkflowExecution {
|
||||
readonly evidence: Evidence;
|
||||
readonly sources: readonly Evidence[];
|
||||
}
|
||||
@@ -0,0 +1,149 @@
|
||||
import { functionComparisonInputSchema } from "../../domain/functionComparisonSchemas.js";
|
||||
import { FUNCTION_COMPARISON_PROVIDER } from "../../domain/comparisonEvidenceIdentity.js";
|
||||
import { compareFunctions } from "../../domain/functionComparison.js";
|
||||
import { parseFunctionEvidence } from "../../domain/functionDossierEvidence.js";
|
||||
import { createEvidence } from "../../domain/evidence.js";
|
||||
import { uniqueSorted } from "../../domain/canonicalOrdering.js";
|
||||
import type { RecordUnknownInput } from "../../domain/residualUnknown.js";
|
||||
import { AnalysisError } from "../../domain/analysisErrorBase.js";
|
||||
import { err, ok, type Result } from "../../domain/result.js";
|
||||
import {
|
||||
resolveEvidenceRequest,
|
||||
type EvidenceLookup,
|
||||
} from "../EvidenceInputResolver.js";
|
||||
import type { EvidenceWorkflowExecution } from "./EvidenceWorkflowExecution.js";
|
||||
|
||||
export interface FunctionComparisonExecution extends EvidenceWorkflowExecution {
|
||||
readonly unknown: RecordUnknownInput | undefined;
|
||||
}
|
||||
|
||||
/** Resolve selected records once, then compare their producer-owned typed facts. */
|
||||
export const compareFunctionEvidence = (
|
||||
input: unknown,
|
||||
lookup?: EvidenceLookup,
|
||||
signal?: AbortSignal,
|
||||
): Result<FunctionComparisonExecution, AnalysisError> => {
|
||||
const selected = resolveEvidenceRequest(
|
||||
"compare_functions",
|
||||
input,
|
||||
functionComparisonInputSchema,
|
||||
lookup,
|
||||
signal === undefined ? {} : { signal },
|
||||
);
|
||||
if (!selected.ok) return selected;
|
||||
const left = parseFunctionEvidence(selected.value.input.left, {
|
||||
operation: "compare_functions",
|
||||
path: ["left"],
|
||||
});
|
||||
if (!left.ok) return left;
|
||||
const right = parseFunctionEvidence(selected.value.input.right, {
|
||||
operation: "compare_functions",
|
||||
path: ["right"],
|
||||
});
|
||||
if (!right.ok) return right;
|
||||
try {
|
||||
const comparison = compareFunctions(left.value, right.value);
|
||||
const leftEvidence = left.value.evidence;
|
||||
const rightEvidence = right.value.evidence;
|
||||
const leftIds = [leftEvidence.evidence_id];
|
||||
const rightIds = [rightEvidence.evidence_id];
|
||||
const evidence = createEvidence(undefined, FUNCTION_COMPARISON_PROVIDER, {
|
||||
predicateType: "rea.function-comparison",
|
||||
operation: "compare_functions",
|
||||
parameters: {
|
||||
left_evidence_id: leftEvidence.evidence_id,
|
||||
right_evidence_id: rightEvidence.evidence_id,
|
||||
},
|
||||
result: comparison,
|
||||
confidence: "derived",
|
||||
authority: "analyst-inference",
|
||||
limitations: comparison.limitations,
|
||||
evidenceLinks: [...leftIds, ...rightIds],
|
||||
});
|
||||
return ok({
|
||||
evidence,
|
||||
sources: [leftEvidence, rightEvidence],
|
||||
unknown: comparisonUnknownInput(
|
||||
comparison,
|
||||
leftIds,
|
||||
rightIds,
|
||||
evidence.evidence_id,
|
||||
),
|
||||
});
|
||||
} catch (cause: unknown) {
|
||||
if (cause instanceof AnalysisError) return err(cause);
|
||||
throw cause;
|
||||
}
|
||||
};
|
||||
|
||||
const comparisonUnknownInput = (
|
||||
comparison: ReturnType<typeof compareFunctions>,
|
||||
leftIds: readonly string[],
|
||||
rightIds: readonly string[],
|
||||
comparisonEvidenceId: string,
|
||||
): RecordUnknownInput | undefined =>
|
||||
functionUnknownInput({
|
||||
status: comparison.status,
|
||||
dimensions: comparison.dimensions
|
||||
.filter(({ status }) =>
|
||||
comparison.status === "unknown"
|
||||
? status === "unknown"
|
||||
: status === "changed",
|
||||
)
|
||||
.map(({ dimension }) => dimension),
|
||||
leftIds,
|
||||
rightIds,
|
||||
comparisonEvidenceId,
|
||||
});
|
||||
|
||||
const functionUnknownInput = ({
|
||||
status,
|
||||
dimensions,
|
||||
leftIds,
|
||||
rightIds,
|
||||
comparisonEvidenceId,
|
||||
}: {
|
||||
status: ReturnType<typeof compareFunctions>["status"];
|
||||
dimensions: readonly ReturnType<
|
||||
typeof compareFunctions
|
||||
>["dimensions"][number]["dimension"][];
|
||||
leftIds: readonly string[];
|
||||
rightIds: readonly string[];
|
||||
comparisonEvidenceId: string;
|
||||
}): RecordUnknownInput | undefined => {
|
||||
if (status === "unchanged") return undefined;
|
||||
const supportingIds = [
|
||||
...new Set([...leftIds, ...rightIds, comparisonEvidenceId]),
|
||||
];
|
||||
const contradictingIds = rightIds.filter((id) => !leftIds.includes(id));
|
||||
const contradictory = status === "changed";
|
||||
return {
|
||||
question: `Function comparison is ${status} (comparison ${comparisonEvidenceId})`,
|
||||
severity: status === "changed" ? "medium" : "high",
|
||||
domain: "function-comparison",
|
||||
comparison_conclusion_scope: {
|
||||
comparison_evidence_id: comparisonEvidenceId,
|
||||
dimensions: uniqueSorted([
|
||||
...(status === "unknown" || dimensions.length === 0
|
||||
? ["overall" as const]
|
||||
: []),
|
||||
...dimensions,
|
||||
]),
|
||||
},
|
||||
supporting_evidence_ids: contradictory
|
||||
? [...leftIds, comparisonEvidenceId]
|
||||
: supportingIds,
|
||||
contradicting_evidence_ids: contradictory ? contradictingIds : [],
|
||||
required_authority: "shipped-artifact",
|
||||
required_confidence: "observed",
|
||||
required_environment: null,
|
||||
recommended_probes: [
|
||||
{
|
||||
operation: "analyze_function",
|
||||
rationale:
|
||||
"Capture complete dossiers for both functions under the same target context.",
|
||||
},
|
||||
],
|
||||
relationships: [],
|
||||
};
|
||||
};
|
||||
@@ -2,6 +2,8 @@ import { expect, it } from "vitest";
|
||||
|
||||
import { createEvidence, type Evidence } from "../../domain/evidence.js";
|
||||
import { createEvidenceBundle } from "../../domain/evidenceBundle.js";
|
||||
import { projectAnalysisError } from "../../domain/analysisErrorProjection.js";
|
||||
import { analysisErrorProjectionSchema } from "../../contracts/errorSchemas.js";
|
||||
import { InvestigationRecords } from "./InvestigationRecords.js";
|
||||
import { resolveInvestigationEvidenceClosure } from "./InvestigationEvidence.js";
|
||||
|
||||
@@ -64,12 +66,28 @@ it("keeps a selected transitive closure distinct from unrelated records and reje
|
||||
records.clear();
|
||||
const added = records.recordEvidence(root);
|
||||
if (!added.ok) throw added.error;
|
||||
expect(
|
||||
resolveInvestigationEvidenceClosure(records, [root.evidence_id]),
|
||||
).toMatchObject({
|
||||
const missing = resolveInvestigationEvidenceClosure(records, [
|
||||
root.evidence_id,
|
||||
]);
|
||||
expect(missing).toMatchObject({
|
||||
ok: false,
|
||||
error: { _tag: "EvidenceIntegrityError" },
|
||||
error: {
|
||||
_tag: "EvidenceIntegrityError",
|
||||
reason: "missing",
|
||||
referenceScope: "retained-closure",
|
||||
},
|
||||
});
|
||||
if (missing.ok) throw new Error("missing source closure was resolved");
|
||||
const projected = projectAnalysisError(missing.error);
|
||||
expect(projected).toMatchObject({
|
||||
details: {
|
||||
evidence_id: leaf.evidence_id,
|
||||
reference_scope: "retained-closure",
|
||||
reason: "missing",
|
||||
},
|
||||
});
|
||||
expect(projected.message).toContain(leaf.evidence_id);
|
||||
expect(analysisErrorProjectionSchema.parse(projected)).toEqual(projected);
|
||||
});
|
||||
|
||||
it("rejects a lookup that returns another semantic Evidence identity", () => {
|
||||
@@ -86,6 +104,7 @@ it("rejects a lookup that returns another semantic Evidence identity", () => {
|
||||
reason: "identity_mismatch",
|
||||
evidenceId: selectedId,
|
||||
actual: record.evidence_id,
|
||||
referenceScope: undefined,
|
||||
},
|
||||
});
|
||||
});
|
||||
@@ -2,14 +2,22 @@ import { resolveEvidenceInput } from "../EvidenceInputResolver.js";
|
||||
import type { EvidenceReader } from "./InvestigationRecordPort.js";
|
||||
import { uniqueSorted } from "../../domain/canonicalOrdering.js";
|
||||
import type { EvidenceIntegrityError } from "../../domain/evidenceErrors.js";
|
||||
import type { Evidence } from "../../domain/evidence.js";
|
||||
import { ok, type Result } from "../../domain/result.js";
|
||||
|
||||
/** Follow exact session references without exporting unrelated records or Unknowns. */
|
||||
export const resolveInvestigationEvidenceClosure = (
|
||||
reader: Pick<EvidenceReader, "evidenceForAnalysis">,
|
||||
seedIds: readonly string[],
|
||||
sources: readonly Evidence[] = [],
|
||||
): Result<string[], EvidenceIntegrityError> => {
|
||||
const lookup = reader.evidenceForAnalysis.bind(reader);
|
||||
// Supplied authenticated records are part of this investigation too. A
|
||||
// portable inline source need not already have been inserted in the ledger.
|
||||
const supplied = new Map(
|
||||
sources.map((source) => [source.evidence_id, source]),
|
||||
);
|
||||
const lookup = (id: string) =>
|
||||
supplied.get(id) ?? reader.evidenceForAnalysis(id);
|
||||
const visited = new Set<string>();
|
||||
const pending = [...seedIds];
|
||||
while (pending.length > 0) {
|
||||
@@ -18,6 +26,7 @@ export const resolveInvestigationEvidenceClosure = (
|
||||
const evidence = resolveEvidenceInput(
|
||||
{ kind: "retained-evidence", evidence_id: evidenceId },
|
||||
lookup,
|
||||
"retained-closure",
|
||||
);
|
||||
if (!evidence.ok) return evidence;
|
||||
visited.add(evidenceId);
|
||||
|
||||
@@ -0,0 +1,252 @@
|
||||
import {
|
||||
callPathRequestSchema,
|
||||
staticRuntimeCorrelationRequestSchema,
|
||||
} from "../../contracts/investigationInputContracts.js";
|
||||
import {
|
||||
changedBehaviorInputSchema,
|
||||
findChangedBehavior,
|
||||
} from "../../domain/changedBehavior.js";
|
||||
import { buildCallPath } from "../../domain/callPath.js";
|
||||
import { correlateStaticAndRuntime } from "../../domain/staticRuntimeCorrelation.js";
|
||||
import { createEvidence, type Evidence } from "../../domain/evidence.js";
|
||||
import { readImmutableEvidence } from "../../domain/evidenceValidation.js";
|
||||
import { AnalysisError } from "../../domain/analysisErrorBase.js";
|
||||
import {
|
||||
CALL_PATH_PROVIDER,
|
||||
CHANGED_BEHAVIOR_PROVIDER,
|
||||
STATIC_RUNTIME_PROVIDER,
|
||||
} from "../../domain/investigationProviders.js";
|
||||
import type { RecordUnknownInput } from "../../domain/residualUnknown.js";
|
||||
import { err, ok, type Result } from "../../domain/result.js";
|
||||
import { resolveEvidenceRequest } from "../EvidenceInputResolver.js";
|
||||
import type { EvidenceReader } from "./InvestigationRecordPort.js";
|
||||
import type { EvidenceWorkflowExecution } from "./EvidenceWorkflowExecution.js";
|
||||
import { resolveInvestigationEvidenceClosure } from "./InvestigationEvidence.js";
|
||||
import { workflowUnknownInput } from "./WorkflowUnknown.js";
|
||||
|
||||
export interface InvestigationWorkflowExecution extends EvidenceWorkflowExecution {
|
||||
readonly unknown: RecordUnknownInput | undefined;
|
||||
}
|
||||
|
||||
type Reader = Pick<EvidenceReader, "evidenceForAnalysis">;
|
||||
type UnknownDescription = {
|
||||
readonly question: string;
|
||||
readonly domain: string;
|
||||
readonly requiredAuthority: "shipped-artifact" | "controlled-replay" | null;
|
||||
readonly requiredConfidence: "observed" | "derived";
|
||||
readonly probes: readonly {
|
||||
readonly operation: string;
|
||||
readonly rationale: string;
|
||||
}[];
|
||||
};
|
||||
|
||||
const finish = (
|
||||
evidence: Evidence,
|
||||
sources: readonly Evidence[],
|
||||
unresolved: boolean,
|
||||
description: UnknownDescription,
|
||||
): Result<InvestigationWorkflowExecution, AnalysisError> => {
|
||||
if (!unresolved) return ok({ evidence, sources, unknown: undefined });
|
||||
const unknown = workflowUnknownInput(evidence.evidence_id, {
|
||||
question: description.question,
|
||||
severity: "high",
|
||||
domain: description.domain,
|
||||
required_authority: description.requiredAuthority,
|
||||
required_confidence: description.requiredConfidence,
|
||||
required_environment: null,
|
||||
recommended_probes: [...description.probes],
|
||||
relationships: [],
|
||||
});
|
||||
return unknown.ok
|
||||
? ok({ evidence, sources, unknown: unknown.value })
|
||||
: unknown;
|
||||
};
|
||||
|
||||
/** Compose exact existing dossiers without starting a provider or resending payloads. */
|
||||
export const buildCallPathEvidence = (
|
||||
input: unknown,
|
||||
reader: Reader,
|
||||
signal?: AbortSignal,
|
||||
): Result<InvestigationWorkflowExecution, AnalysisError> => {
|
||||
const selected = resolveEvidenceRequest(
|
||||
"build_call_path",
|
||||
input,
|
||||
callPathRequestSchema,
|
||||
reader.evidenceForAnalysis.bind(reader),
|
||||
{
|
||||
...(signal === undefined ? {} : { signal }),
|
||||
referenceScope: "retained-closure",
|
||||
readEvidence: readImmutableEvidence,
|
||||
},
|
||||
);
|
||||
if (!selected.ok) return selected;
|
||||
try {
|
||||
const result = buildCallPath(selected.value.input);
|
||||
const sources = selected.value.input.functions;
|
||||
const closure = resolveInvestigationEvidenceClosure(
|
||||
reader,
|
||||
result.evidence_links,
|
||||
sources,
|
||||
);
|
||||
if (!closure.ok) return closure;
|
||||
const evidence = createEvidence(undefined, CALL_PATH_PROVIDER, {
|
||||
predicateType: "rea.call-path",
|
||||
operation: "build_call_path",
|
||||
parameters: {
|
||||
start: selected.value.input.start.address,
|
||||
goal: selected.value.input.goal.address,
|
||||
},
|
||||
result,
|
||||
confidence: "derived",
|
||||
authority: "analyst-inference",
|
||||
limitations: result.limitations,
|
||||
evidenceLinks: closure.value,
|
||||
});
|
||||
return finish(evidence, sources, result.status === "unknown", {
|
||||
question:
|
||||
"Can the requested call path be established from complete analysis?",
|
||||
domain: "call-path",
|
||||
requiredAuthority: "shipped-artifact",
|
||||
requiredConfidence: "derived",
|
||||
probes: [
|
||||
{
|
||||
operation: "analyze_function",
|
||||
rationale:
|
||||
"Collect complete callee dossiers for the reported frontier addresses.",
|
||||
},
|
||||
],
|
||||
});
|
||||
} catch (cause: unknown) {
|
||||
if (cause instanceof AnalysisError) return err(cause);
|
||||
throw cause;
|
||||
}
|
||||
};
|
||||
|
||||
/** Aggregate caller-selected comparisons and preserve their exact source closure. */
|
||||
export const findChangedBehaviorEvidence = (
|
||||
input: unknown,
|
||||
reader: Reader,
|
||||
signal?: AbortSignal,
|
||||
): Result<InvestigationWorkflowExecution, AnalysisError> => {
|
||||
const selected = resolveEvidenceRequest(
|
||||
"find_changed_behavior",
|
||||
input,
|
||||
changedBehaviorInputSchema,
|
||||
reader.evidenceForAnalysis.bind(reader),
|
||||
{
|
||||
...(signal === undefined ? {} : { signal }),
|
||||
referenceScope: "retained-closure",
|
||||
readEvidence: readImmutableEvidence,
|
||||
},
|
||||
);
|
||||
if (!selected.ok) return selected;
|
||||
try {
|
||||
const { result, comparisonEvidenceIds } = findChangedBehavior(
|
||||
selected.value.input,
|
||||
);
|
||||
const sources = selected.value.input.comparisons;
|
||||
const closure = resolveInvestigationEvidenceClosure(
|
||||
reader,
|
||||
result.evidence_links,
|
||||
sources,
|
||||
);
|
||||
if (!closure.ok) return closure;
|
||||
const evidence = createEvidence(undefined, CHANGED_BEHAVIOR_PROVIDER, {
|
||||
predicateType: "rea.changed-behavior",
|
||||
operation: "find_changed_behavior",
|
||||
parameters: { comparison_evidence_ids: [...comparisonEvidenceIds] },
|
||||
result,
|
||||
confidence: "derived",
|
||||
authority: "analyst-inference",
|
||||
limitations: result.limitations,
|
||||
evidenceLinks: closure.value,
|
||||
});
|
||||
return finish(
|
||||
evidence,
|
||||
sources,
|
||||
result.behavior_status === "unknown" ||
|
||||
result.behavior_status === "truncated",
|
||||
{
|
||||
question:
|
||||
"Did both versions behave the same under a complete controlled replay?",
|
||||
domain: "changed-behavior",
|
||||
requiredAuthority: "controlled-replay",
|
||||
requiredConfidence: "observed",
|
||||
probes: [
|
||||
{
|
||||
operation: "capture_process_scenario",
|
||||
rationale:
|
||||
"Capture both versions under the same bounded scenario and environment.",
|
||||
},
|
||||
],
|
||||
},
|
||||
);
|
||||
} catch (cause: unknown) {
|
||||
if (cause instanceof AnalysisError) return err(cause);
|
||||
throw cause;
|
||||
}
|
||||
};
|
||||
|
||||
/** Keep correlation hypotheses explicit while resolving record transport centrally. */
|
||||
export const correlateStaticAndRuntimeEvidence = (
|
||||
input: unknown,
|
||||
reader: Reader,
|
||||
signal?: AbortSignal,
|
||||
): Result<InvestigationWorkflowExecution, AnalysisError> => {
|
||||
const selected = resolveEvidenceRequest(
|
||||
"correlate_static_and_runtime",
|
||||
input,
|
||||
staticRuntimeCorrelationRequestSchema,
|
||||
reader.evidenceForAnalysis.bind(reader),
|
||||
{
|
||||
...(signal === undefined ? {} : { signal }),
|
||||
referenceScope: "retained-closure",
|
||||
readEvidence: readImmutableEvidence,
|
||||
},
|
||||
);
|
||||
if (!selected.ok) return selected;
|
||||
try {
|
||||
const result = correlateStaticAndRuntime(selected.value.input);
|
||||
const sources = [
|
||||
...selected.value.input.static_comparisons,
|
||||
...selected.value.input.runtime_comparisons,
|
||||
];
|
||||
const closure = resolveInvestigationEvidenceClosure(
|
||||
reader,
|
||||
result.evidence_links,
|
||||
sources,
|
||||
);
|
||||
if (!closure.ok) return closure;
|
||||
const evidence = createEvidence(undefined, STATIC_RUNTIME_PROVIDER, {
|
||||
predicateType: "rea.static-runtime-correlation",
|
||||
operation: "correlate_static_and_runtime",
|
||||
parameters: { mapping_count: selected.value.input.mappings.length },
|
||||
result,
|
||||
confidence: "inferred",
|
||||
authority: "analyst-inference",
|
||||
limitations: result.limitations,
|
||||
evidenceLinks: closure.value,
|
||||
});
|
||||
return finish(
|
||||
evidence,
|
||||
sources,
|
||||
result.status === "unknown" || result.status === "truncated",
|
||||
{
|
||||
question: "Does runtime behavior match the available static analysis?",
|
||||
domain: "static-runtime-correlation",
|
||||
requiredAuthority: null,
|
||||
requiredConfidence: "derived",
|
||||
probes: [
|
||||
{
|
||||
operation: "capture_process_scenario",
|
||||
rationale:
|
||||
"Repeat runtime observations and complete the mapped static comparison Evidence.",
|
||||
},
|
||||
],
|
||||
},
|
||||
);
|
||||
} catch (cause: unknown) {
|
||||
if (cause instanceof AnalysisError) return err(cause);
|
||||
throw cause;
|
||||
}
|
||||
};
|
||||
@@ -8,14 +8,20 @@ import {
|
||||
withEvidenceSources,
|
||||
} from "./recordEvidenceSources.js";
|
||||
|
||||
it.each([false, true])(
|
||||
"reports source recording failure without replacing earlier cancellation (cancelled=%s)",
|
||||
(cancelled) => {
|
||||
const source = createEvidence(
|
||||
undefined,
|
||||
{ id: "fixture", name: "Fixture", version: "1" },
|
||||
{ operation: "health", parameters: {}, result: "historical" },
|
||||
);
|
||||
const source = createEvidence(
|
||||
undefined,
|
||||
{ id: "fixture", name: "Fixture", version: "1" },
|
||||
{ operation: "health", parameters: {}, result: "historical" },
|
||||
);
|
||||
|
||||
it.each([
|
||||
{ cancelled: false, sourceOwner: undefined },
|
||||
{ cancelled: true, sourceOwner: undefined },
|
||||
{ cancelled: false, sourceOwner: "caller" as const },
|
||||
{ cancelled: true, sourceOwner: "caller" as const },
|
||||
])(
|
||||
"reports source recording failure without replacing earlier cancellation (cancelled=$cancelled, owner=$sourceOwner)",
|
||||
({ cancelled, sourceOwner }) => {
|
||||
const primary = new AnalysisCancelledError("analyze_function", {
|
||||
cause: "caller",
|
||||
});
|
||||
@@ -24,9 +30,73 @@ it.each([false, true])(
|
||||
"analyze_function",
|
||||
() => err(failure),
|
||||
withEvidenceSources(cancelled ? err(primary) : ok("complete"), [source]),
|
||||
sourceOwner === undefined ? undefined : { sourceOwner },
|
||||
);
|
||||
expect(result.ok).toBe(false);
|
||||
if (result.ok) throw new Error("Expected recording failure");
|
||||
expect(result.error).toBe(cancelled ? primary : failure);
|
||||
expect(primary.secondaryFailures).toEqual(cancelled ? [failure] : []);
|
||||
},
|
||||
);
|
||||
|
||||
it.each([false, true])(
|
||||
"requires retention for provider sources without a writer (cancelled=%s)",
|
||||
(cancelled) => {
|
||||
const primary = new AnalysisCancelledError("analyze_function", {
|
||||
cause: "caller",
|
||||
});
|
||||
const result = retainAnalysisResultSources(
|
||||
"analyze_function",
|
||||
undefined,
|
||||
withEvidenceSources(cancelled ? err(primary) : ok("complete"), [source]),
|
||||
);
|
||||
if (result.ok)
|
||||
throw new Error("Provider sources lost their retention owner");
|
||||
if (cancelled) {
|
||||
expect(result.error).toBe(primary);
|
||||
expect(primary.secondaryFailures).toMatchObject([
|
||||
{ _tag: "AnalysisCapabilityUnavailableError" },
|
||||
]);
|
||||
} else
|
||||
expect(result.error).toMatchObject({
|
||||
_tag: "AnalysisCapabilityUnavailableError",
|
||||
});
|
||||
},
|
||||
);
|
||||
|
||||
it.each([false, true])(
|
||||
"preserves caller sources without a writer, including late cancellation (cancelled=%s)",
|
||||
(cancelled) => {
|
||||
const primary = new AnalysisCancelledError("compare_wasm_interfaces", {
|
||||
cause: "caller",
|
||||
});
|
||||
const outcome = withEvidenceSources(
|
||||
cancelled ? err(primary) : ok("complete"),
|
||||
[source],
|
||||
);
|
||||
const result = retainAnalysisResultSources(
|
||||
"compare_wasm_interfaces",
|
||||
undefined,
|
||||
outcome,
|
||||
{ sourceOwner: "caller" },
|
||||
);
|
||||
expect(result).toBe(outcome);
|
||||
expect(outcome.sourceEvidence).toEqual([source]);
|
||||
expect(primary.secondaryFailures).toEqual([]);
|
||||
},
|
||||
);
|
||||
|
||||
it("still records caller sources when an explicit writer exists", () => {
|
||||
const recorded: string[] = [];
|
||||
const result = retainAnalysisResultSources(
|
||||
"compare_wasm_interfaces",
|
||||
(evidence) => {
|
||||
recorded.push(evidence.evidence_id);
|
||||
return ok("added");
|
||||
},
|
||||
withEvidenceSources(ok("complete"), [source]),
|
||||
{ sourceOwner: "caller" },
|
||||
);
|
||||
expect(result).toMatchObject({ ok: true, value: "complete" });
|
||||
expect(recorded).toEqual([source.evidence_id]);
|
||||
});
|
||||
@@ -18,15 +18,18 @@ export const withEvidenceSources = <
|
||||
): Outcome & { readonly sourceEvidence?: readonly Evidence[] } =>
|
||||
sources.length === 0 ? result : { ...result, sourceEvidence: sources };
|
||||
|
||||
/** Never deliver links to records a plain-port adapter cannot retain. */
|
||||
/** Retain producer sources; caller-owned records remain available without a writer. */
|
||||
export const retainAnalysisResultSources = <Value>(
|
||||
operation: string,
|
||||
recordEvidence: EvidenceWriter["recordEvidence"] | undefined,
|
||||
result: AnalysisResultWithSources<Value>,
|
||||
options: { readonly sourceOwner?: "caller" } = {},
|
||||
): Result<Value, AnalysisError> => {
|
||||
const sources = result.sourceEvidence ?? [];
|
||||
const retained =
|
||||
sources.length > 0 && recordEvidence === undefined
|
||||
sources.length > 0 &&
|
||||
recordEvidence === undefined &&
|
||||
options.sourceOwner !== "caller"
|
||||
? err(
|
||||
new AnalysisCapabilityUnavailableError(
|
||||
"rea",
|
||||
|
||||
@@ -220,6 +220,10 @@ export const summarizeElectronBoundaries = (
|
||||
moduleKind: finding.module_kind,
|
||||
}),
|
||||
).length,
|
||||
protocol_registrations: javascript.reduce(
|
||||
(count, { electron }) => count + electron.protocols.length,
|
||||
0,
|
||||
),
|
||||
};
|
||||
};
|
||||
|
||||
|
||||
@@ -2,6 +2,7 @@ import type { ElectronIpcContext } from "./ElectronBoundaryAnalysis.js";
|
||||
import type { JavaScriptArtifactGraphContext } from "./JavaScriptArtifactGraphContext.js";
|
||||
import { addElectronIpcBoundaries } from "./ElectronBoundaryGraphIpc.js";
|
||||
import { addElectronNativeBoundaries } from "./ElectronBoundaryGraphNative.js";
|
||||
import { addElectronProtocolBoundaries } from "./ElectronBoundaryGraphProtocols.js";
|
||||
import { addElectronWindowBoundaries } from "./ElectronBoundaryGraphWindows.js";
|
||||
import { addElectronCallPaths } from "./ElectronBoundaryGraphCallPaths.js";
|
||||
|
||||
@@ -13,5 +14,6 @@ export const addElectronBoundaries = (
|
||||
addElectronWindowBoundaries(context);
|
||||
addElectronIpcBoundaries(context, ipc);
|
||||
addElectronNativeBoundaries(context);
|
||||
addElectronProtocolBoundaries(context);
|
||||
addElectronCallPaths(context, ipc);
|
||||
};
|
||||
@@ -0,0 +1,129 @@
|
||||
import { javascriptSourceRangeKey } from "../../domain/javascript/javascriptSourceCoordinates.js";
|
||||
import type { ElectronProtocolFinding } from "../../domain/javascript/electronStaticAnalysisTypes.js";
|
||||
import type { JavaScriptArtifactFile } from "../../domain/javascript/javascriptArtifactFiles.js";
|
||||
import {
|
||||
artifactLocalIdentity,
|
||||
javascriptAnalysisCoverage,
|
||||
type JavaScriptArtifactGraphContext,
|
||||
type JavaScriptArtifactGraphCoverage,
|
||||
} from "./JavaScriptArtifactGraphContext.js";
|
||||
import { astObservationEvidence } from "./JavaScriptArtifactGraphEvidence.js";
|
||||
import {
|
||||
addElectronInferenceEdge,
|
||||
electronFindingSourceNode,
|
||||
electronObservationIdentity,
|
||||
} from "./ElectronBoundaryGraphContext.js";
|
||||
|
||||
interface ProtocolInput {
|
||||
readonly context: JavaScriptArtifactGraphContext;
|
||||
readonly file: JavaScriptArtifactFile;
|
||||
readonly value: ElectronProtocolFinding;
|
||||
readonly coverage: JavaScriptArtifactGraphCoverage;
|
||||
}
|
||||
|
||||
const URL_SCHEME = /^[A-Za-z][A-Za-z0-9+.-]*$/u;
|
||||
|
||||
const REGISTRATION_LIMITATION =
|
||||
"A registration call is present in source; successful registration, control-flow reachability and operating-system or session routing are not established.";
|
||||
|
||||
/** Project custom URL scheme registrations as schemes the application claims. */
|
||||
export const addElectronProtocolBoundaries = (
|
||||
context: JavaScriptArtifactGraphContext,
|
||||
): void => {
|
||||
for (const analyzed of context.analysis.files) {
|
||||
const { file, javascript } = analyzed;
|
||||
if (javascript === null) continue;
|
||||
const coverage = javascriptAnalysisCoverage(javascript);
|
||||
for (const value of javascript.electron.protocols)
|
||||
addProtocolRegistration({ context, file, value, coverage });
|
||||
}
|
||||
};
|
||||
|
||||
const addProtocolRegistration = (input: ProtocolInput): void => {
|
||||
const { context, file, value, coverage } = input;
|
||||
const source = electronFindingSourceNode(
|
||||
context,
|
||||
file,
|
||||
value.module_key,
|
||||
value.module_runtime,
|
||||
);
|
||||
if (source === undefined) return;
|
||||
const literal = value.scheme.status === "literal";
|
||||
const raw = value.scheme.value;
|
||||
// RFC 3986: ALPHA *( ALPHA / DIGIT / "+" / "-" / "." ).
|
||||
const scheme =
|
||||
literal && typeof raw === "string" && URL_SCHEME.test(raw)
|
||||
? raw.toLowerCase()
|
||||
: null;
|
||||
// A non-string or malformed literal is observed exactly but names no scheme.
|
||||
const invalid = literal && scheme === null;
|
||||
const facts = {
|
||||
scheme,
|
||||
scheme_expression: value.scheme.expression,
|
||||
scheme_literal: literal ? raw : null,
|
||||
resolution:
|
||||
scheme !== null ? "literal" : invalid ? "invalid-literal" : "dynamic",
|
||||
api: value.api,
|
||||
role: value.role,
|
||||
session: value.session,
|
||||
enabled_privileges: value.enabled_privileges,
|
||||
};
|
||||
const limitations =
|
||||
scheme !== null
|
||||
? [REGISTRATION_LIMITATION]
|
||||
: invalid
|
||||
? [
|
||||
REGISTRATION_LIMITATION,
|
||||
"The scheme argument is a literal that is not a valid URL scheme, so no URL can target it as written.",
|
||||
]
|
||||
: [
|
||||
REGISTRATION_LIMITATION,
|
||||
"The scheme expression is dynamic; its runtime value remains unknown.",
|
||||
];
|
||||
// Scheme names are case-insensitive; retain their spelling separately from
|
||||
// canonical identity. One node per scheme across the artifact, so one both
|
||||
// privileged and handled is one entry point with several registrations.
|
||||
const node = context.accumulator.addNode({
|
||||
kind: "url-scheme",
|
||||
identity:
|
||||
scheme === null
|
||||
? artifactLocalIdentity(
|
||||
file.sha256,
|
||||
"electron-dynamic-url-scheme",
|
||||
// Identical bytes at two paths are separate occurrences whose
|
||||
// expressions can resolve differently.
|
||||
`${file.path}:${value.api}:${javascriptSourceRangeKey(value.location)}`,
|
||||
)
|
||||
: electronObservationIdentity(
|
||||
context,
|
||||
"electron-url-scheme",
|
||||
`literal:${scheme}`,
|
||||
),
|
||||
observations: [
|
||||
{
|
||||
label: scheme ?? value.scheme.expression ?? String(value.scheme.value),
|
||||
properties: facts,
|
||||
evidence: astObservationEvidence({
|
||||
sha256: file.sha256,
|
||||
path: file.path,
|
||||
range: value.location,
|
||||
operation: "observe-protocol-registration",
|
||||
coverage,
|
||||
limitations,
|
||||
}),
|
||||
},
|
||||
],
|
||||
});
|
||||
addElectronInferenceEdge(context, {
|
||||
source,
|
||||
target: node,
|
||||
file,
|
||||
range: value.location,
|
||||
coverage,
|
||||
// Granting privileges exposes the scheme; the other roles claim its links.
|
||||
relation: value.role === "privileged-scheme" ? "exposes" : "handles",
|
||||
operation: "map-protocol-registration",
|
||||
properties: facts,
|
||||
limitations,
|
||||
});
|
||||
};
|
||||
@@ -1,39 +1,39 @@
|
||||
import type { ExecutionOptions } from "../AnalysisProvider.js";
|
||||
import {
|
||||
resolveEvidenceInput,
|
||||
resolveEvidenceRequest,
|
||||
type EvidenceLookup,
|
||||
} from "../EvidenceInputResolver.js";
|
||||
import { retainedEvidenceReferenceSchema } from "../../domain/evidenceReference.js";
|
||||
import { parseAnalysisInput } from "../AnalysisInput.js";
|
||||
import {
|
||||
compareApplicationVersionsRequestSchema,
|
||||
compareJavaScriptExportShapesRequestSchema,
|
||||
compareSourceToBundleCliInputSchema,
|
||||
compareSourceToBundleRequestSchema,
|
||||
traceApplicationFeatureRequestSchema,
|
||||
} from "../../contracts/javascript/applicationWorkflowInputContracts.js";
|
||||
import type { EvidenceWorkflowExecution } from "../investigation/EvidenceWorkflowExecution.js";
|
||||
import type { ArtifactResourceScope } from "../../artifacts/ArtifactResourceScope.js";
|
||||
import type { ResolvedSourceToBundleReference } from "./SourceToBundleReference.js";
|
||||
import { z } from "zod";
|
||||
|
||||
import type { AnalysisError } from "../../domain/analysisErrorBase.js";
|
||||
import type { Evidence } from "../../domain/evidence.js";
|
||||
import { analysisInputErrorFromIssues } from "../../domain/inputIssueProjection.js";
|
||||
import { compareJavaScriptApplicationVersions } from "../../domain/javascript/javascriptApplicationVersionComparison.js";
|
||||
import {
|
||||
compareApplicationVersionsInputSchema,
|
||||
applicationVersionIdentitiesSchema,
|
||||
type ApplicationVersionComparisonResult,
|
||||
} from "../../domain/javascript/javascriptApplicationVersionComparisonSchemas.js";
|
||||
import { compareJavaScriptExportShapes } from "../../domain/javascript/javascriptExportShapeComparison.js";
|
||||
import {
|
||||
compareJavaScriptExportShapesInputSchema,
|
||||
javaScriptExportShapeIdentitiesSchema,
|
||||
type JavaScriptExportShapeComparisonResult,
|
||||
} from "../../domain/javascript/javascriptExportShapeComparisonSchemas.js";
|
||||
import { traceApplicationFeature } from "../../domain/javascript/javascriptFeatureTrace.js";
|
||||
import {
|
||||
traceApplicationFeatureInputSchema,
|
||||
applicationTraceIdentitiesSchema,
|
||||
type ApplicationFeatureTraceResult,
|
||||
} from "../../domain/javascript/javascriptFeatureTraceSchemas.js";
|
||||
import { err, ok, type Result } from "../../domain/result.js";
|
||||
import { compareSourceToBundle } from "../../domain/javascript/sourceToBundleComparison.js";
|
||||
import {
|
||||
compareSourceToBundleInputSchema,
|
||||
type SourceToBundleComparisonResult,
|
||||
} from "../../domain/javascript/sourceToBundleComparisonSchemas.js";
|
||||
import type { SourceToBundleComparisonResult } from "../../domain/javascript/sourceToBundleComparisonSchemas.js";
|
||||
import {
|
||||
applicationGraphEvidenceInputError,
|
||||
parseApplicationGraphEvidence,
|
||||
@@ -48,68 +48,32 @@ import {
|
||||
} from "./JavaScriptApplicationWorkflowEvidence.js";
|
||||
|
||||
/** Completed workflow facts and authenticated sources retained for adapter delivery. */
|
||||
export interface ApplicationWorkflowExecution<Projection> {
|
||||
readonly evidence: Evidence;
|
||||
export interface ApplicationWorkflowExecution<
|
||||
Projection,
|
||||
> extends EvidenceWorkflowExecution {
|
||||
readonly result: Projection;
|
||||
readonly sources: readonly Evidence[];
|
||||
}
|
||||
|
||||
// Validate the request frame without cloning source payloads before their owner reads them.
|
||||
const featureTraceFrameSchema = z.strictObject({
|
||||
...traceApplicationFeatureInputSchema.shape,
|
||||
application: z.unknown().nonoptional(),
|
||||
native_observations: z.array(z.unknown()).default([]),
|
||||
});
|
||||
const versionComparisonFrameSchema = z.strictObject({
|
||||
...compareApplicationVersionsInputSchema.shape,
|
||||
left: z.unknown().nonoptional(),
|
||||
right: z.unknown().nonoptional(),
|
||||
left_native_observations: z.array(z.unknown()).default([]),
|
||||
right_native_observations: z.array(z.unknown()).default([]),
|
||||
});
|
||||
const exportShapeFrameSchema = z.strictObject({
|
||||
...compareJavaScriptExportShapesInputSchema.shape,
|
||||
left: z.unknown().nonoptional(),
|
||||
right: z.unknown().nonoptional(),
|
||||
});
|
||||
|
||||
const resolveWorkflowEvidence = (
|
||||
raw: unknown,
|
||||
lookup: EvidenceLookup | undefined,
|
||||
): Result<unknown, AnalysisError> => {
|
||||
const reference = retainedEvidenceReferenceSchema.safeParse(raw);
|
||||
return reference.success
|
||||
? resolveEvidenceInput(reference.data, lookup)
|
||||
: ok(raw);
|
||||
};
|
||||
|
||||
/** Resolve both selections before authenticating either graph, retaining diagnostic inputs. */
|
||||
const resolveApplicationGraphPair = (
|
||||
/** Authenticate the resolved graph pair at the owning domain boundary. */
|
||||
const parseApplicationGraphPair = (
|
||||
operation: string,
|
||||
input: { readonly left: unknown; readonly right: unknown },
|
||||
lookup: EvidenceLookup | undefined,
|
||||
): Result<
|
||||
{
|
||||
readonly left: ApplicationGraphEvidenceSource;
|
||||
readonly right: ApplicationGraphEvidenceSource;
|
||||
readonly selected: { readonly left: unknown; readonly right: unknown };
|
||||
},
|
||||
AnalysisError
|
||||
> => {
|
||||
const selectedLeft = resolveWorkflowEvidence(input.left, lookup);
|
||||
if (!selectedLeft.ok) return selectedLeft;
|
||||
const selectedRight = resolveWorkflowEvidence(input.right, lookup);
|
||||
if (!selectedRight.ok) return selectedRight;
|
||||
const left = parseApplicationGraphEvidence(selectedLeft.value, ["left"]);
|
||||
const left = parseApplicationGraphEvidence(input.left, ["left"]);
|
||||
if (!left.ok)
|
||||
return err(applicationGraphEvidenceInputError(operation, left.error));
|
||||
const right = parseApplicationGraphEvidence(selectedRight.value, ["right"]);
|
||||
const right = parseApplicationGraphEvidence(input.right, ["right"]);
|
||||
if (!right.ok)
|
||||
return err(applicationGraphEvidenceInputError(operation, right.error));
|
||||
return ok({
|
||||
left: left.value,
|
||||
right: right.value,
|
||||
selected: { left: selectedLeft.value, right: selectedRight.value },
|
||||
});
|
||||
};
|
||||
|
||||
@@ -117,20 +81,21 @@ const resolveApplicationGraphPair = (
|
||||
export const traceApplicationFeatureEvidence = (
|
||||
rawInput: unknown,
|
||||
lookup?: EvidenceLookup,
|
||||
options: Pick<ExecutionOptions, "signal"> = {},
|
||||
): Result<
|
||||
ApplicationWorkflowExecution<ApplicationFeatureTraceResult>,
|
||||
AnalysisError
|
||||
> => {
|
||||
const operation = "trace_application_feature";
|
||||
const request = parseAnalysisInput(
|
||||
const request = resolveEvidenceRequest(
|
||||
operation,
|
||||
rawInput,
|
||||
featureTraceFrameSchema,
|
||||
traceApplicationFeatureRequestSchema,
|
||||
lookup,
|
||||
options,
|
||||
);
|
||||
if (!request.ok) return request;
|
||||
const selected = resolveWorkflowEvidence(request.value.application, lookup);
|
||||
if (!selected.ok) return selected;
|
||||
const input = { ...request.value, application: selected.value };
|
||||
const input = request.value.input;
|
||||
const sourceResult = parseApplicationGraphEvidence(input.application, [
|
||||
"application",
|
||||
]);
|
||||
@@ -189,21 +154,24 @@ export const traceApplicationFeatureEvidence = (
|
||||
export const compareApplicationVersionsEvidence = (
|
||||
rawInput: unknown,
|
||||
lookup?: EvidenceLookup,
|
||||
options: Pick<ExecutionOptions, "signal"> = {},
|
||||
): Result<
|
||||
ApplicationWorkflowExecution<ApplicationVersionComparisonResult>,
|
||||
AnalysisError
|
||||
> => {
|
||||
const operation = "compare_application_versions";
|
||||
const request = parseAnalysisInput(
|
||||
const request = resolveEvidenceRequest(
|
||||
operation,
|
||||
rawInput,
|
||||
versionComparisonFrameSchema,
|
||||
compareApplicationVersionsRequestSchema,
|
||||
lookup,
|
||||
options,
|
||||
);
|
||||
if (!request.ok) return request;
|
||||
const pair = resolveApplicationGraphPair(operation, request.value, lookup);
|
||||
const input = request.value.input;
|
||||
const pair = parseApplicationGraphPair(operation, input);
|
||||
if (!pair.ok) return pair;
|
||||
const { left, right } = pair.value;
|
||||
const input = { ...request.value, ...pair.value.selected };
|
||||
const leftNativeResult = parseNativeApplicationEvidence(
|
||||
input.left_native_observations,
|
||||
["left_native_observations"],
|
||||
@@ -262,10 +230,47 @@ export const compareApplicationVersionsEvidence = (
|
||||
});
|
||||
};
|
||||
|
||||
/** Compare source and bundle inputs already parsed by a trusted adapter. */
|
||||
export const compareSourceToBundleEvidence = (
|
||||
input: ResolvedSourceToBundleReference &
|
||||
Pick<z.output<typeof compareSourceToBundleInputSchema>, "application">,
|
||||
/** Select the application record and historical graph through their shared owners. */
|
||||
export const executeSourceToBundleComparison = async (
|
||||
rawInput: unknown,
|
||||
schema:
|
||||
| typeof compareSourceToBundleCliInputSchema
|
||||
| typeof compareSourceToBundleRequestSchema,
|
||||
lookup: EvidenceLookup | undefined,
|
||||
resources: ArtifactResourceScope,
|
||||
options: Pick<ExecutionOptions, "signal"> = {},
|
||||
): Promise<
|
||||
Result<
|
||||
ApplicationWorkflowExecution<SourceToBundleComparisonResult>,
|
||||
AnalysisError
|
||||
>
|
||||
> => {
|
||||
const selected = resolveEvidenceRequest(
|
||||
"compare_source_to_bundle",
|
||||
rawInput,
|
||||
schema,
|
||||
lookup,
|
||||
options,
|
||||
);
|
||||
if (!selected.ok) return selected;
|
||||
const { resolveSourceToBundleReference } =
|
||||
await import("./SourceToBundleReference.js");
|
||||
const reference = await resolveSourceToBundleReference(
|
||||
selected.value.input.reference,
|
||||
options.signal,
|
||||
resources,
|
||||
);
|
||||
return reference.ok
|
||||
? compareSourceToBundleEvidence({
|
||||
...reference.value,
|
||||
application: selected.value.input.application,
|
||||
})
|
||||
: reference;
|
||||
};
|
||||
|
||||
/** Compare an owner-parsed historical graph with an unadmitted application source. */
|
||||
const compareSourceToBundleEvidence = (
|
||||
input: ResolvedSourceToBundleReference & { readonly application: unknown },
|
||||
): Result<
|
||||
ApplicationWorkflowExecution<SourceToBundleComparisonResult>,
|
||||
AnalysisError
|
||||
@@ -307,21 +312,24 @@ export const compareSourceToBundleEvidence = (
|
||||
export const compareJavaScriptExportShapesEvidence = (
|
||||
rawInput: unknown,
|
||||
lookup?: EvidenceLookup,
|
||||
options: Pick<ExecutionOptions, "signal"> = {},
|
||||
): Result<
|
||||
ApplicationWorkflowExecution<JavaScriptExportShapeComparisonResult>,
|
||||
AnalysisError
|
||||
> => {
|
||||
const operation = "compare_javascript_export_shapes";
|
||||
const request = parseAnalysisInput(
|
||||
const request = resolveEvidenceRequest(
|
||||
operation,
|
||||
rawInput,
|
||||
exportShapeFrameSchema,
|
||||
compareJavaScriptExportShapesRequestSchema,
|
||||
lookup,
|
||||
options,
|
||||
);
|
||||
if (!request.ok) return request;
|
||||
const pair = resolveApplicationGraphPair(operation, request.value, lookup);
|
||||
const input = request.value.input;
|
||||
const pair = parseApplicationGraphPair(operation, input);
|
||||
if (!pair.ok) return pair;
|
||||
const { left, right } = pair.value;
|
||||
const input = { ...request.value, ...pair.value.selected };
|
||||
const identities = javaScriptExportShapeIdentitiesSchema.safeParse({
|
||||
left: left.evidence,
|
||||
right: right.evidence,
|
||||
|
||||
@@ -668,7 +668,8 @@ const findingCount = (analysis: JavaScriptStaticAnalysis): number =>
|
||||
analysis.electron.ipc.length +
|
||||
analysis.electron.sender_validations.length +
|
||||
analysis.electron.utility_processes.length +
|
||||
analysis.electron.native_addon_bindings.length;
|
||||
analysis.electron.native_addon_bindings.length +
|
||||
analysis.electron.protocols.length;
|
||||
|
||||
const optionalString = (value: unknown): string | null =>
|
||||
typeof value === "string" && value.length > 0 ? value : null;
|
||||
|
||||
@@ -366,7 +366,8 @@ const graphLimitations = (
|
||||
javascript.electron.ipc.length +
|
||||
javascript.electron.sender_validations.length +
|
||||
javascript.electron.utility_processes.length +
|
||||
javascript.electron.native_addon_bindings.length),
|
||||
javascript.electron.native_addon_bindings.length +
|
||||
javascript.electron.protocols.length),
|
||||
0,
|
||||
);
|
||||
return [
|
||||
|
||||
@@ -0,0 +1,195 @@
|
||||
import { expect, it, vi } from "vitest";
|
||||
|
||||
import { createEvidence, type Evidence } from "../../domain/evidence.js";
|
||||
import {
|
||||
JAVASCRIPT_APPLICATION_VERSION_COMPARISON_EXAMPLE,
|
||||
JAVASCRIPT_FEATURE_TRACE_EXAMPLE,
|
||||
} from "../../contracts/javascript/javascriptApplicationWorkflowExamples.js";
|
||||
import { JAVASCRIPT_EXPORT_SHAPE_COMPARISON_EXAMPLE } from "../../contracts/javascript/javascriptExportShapeComparisonExample.js";
|
||||
import { JAVASCRIPT_RUNTIME_RECONCILIATION_EXAMPLE } from "../../contracts/javascript/javascriptRuntimeReconciliationExample.js";
|
||||
import {
|
||||
compareApplicationVersionsEvidence,
|
||||
compareJavaScriptExportShapesEvidence,
|
||||
traceApplicationFeatureEvidence,
|
||||
} from "./JavaScriptApplicationWorkflowService.js";
|
||||
import { reconcileJavaScriptRuntimeEvidence } from "./JavaScriptRuntimeReconciliationService.js";
|
||||
import { traceJavaScriptSemanticsEvidence } from "./JavaScriptSemanticTraceService.js";
|
||||
|
||||
const reference = ({ evidence_id }: Evidence) => ({
|
||||
kind: "retained-evidence" as const,
|
||||
evidence_id,
|
||||
});
|
||||
|
||||
const native = createEvidence(
|
||||
{ path: "/example/addon.node", sha256: "e".repeat(64), format: "elf" },
|
||||
{ id: "fixture", name: "Native fixture", version: "1" },
|
||||
{ operation: "procedure_info", parameters: {}, result: {} },
|
||||
);
|
||||
|
||||
it("resolves feature native arrays without changing source identity or unknowns", () => {
|
||||
const input = {
|
||||
...JAVASCRIPT_FEATURE_TRACE_EXAMPLE,
|
||||
native_observations: [native],
|
||||
};
|
||||
const inline = traceApplicationFeatureEvidence(input);
|
||||
const records = new Map(
|
||||
[input.application, native].map((record) => [record.evidence_id, record]),
|
||||
);
|
||||
const lookup = vi.fn((id: string) => records.get(id));
|
||||
const retained = traceApplicationFeatureEvidence(
|
||||
{
|
||||
...input,
|
||||
application: reference(input.application),
|
||||
native_observations: [reference(native)],
|
||||
},
|
||||
lookup,
|
||||
);
|
||||
if (!inline.ok) throw inline.error;
|
||||
if (!retained.ok) throw retained.error;
|
||||
expect(retained.value.evidence.evidence_id).toBe(
|
||||
inline.value.evidence.evidence_id,
|
||||
);
|
||||
expect(retained.value.result).toEqual(inline.value.result);
|
||||
expect(retained.value.sources.map(({ evidence_id }) => evidence_id)).toEqual([
|
||||
input.application.evidence_id,
|
||||
native.evidence_id,
|
||||
]);
|
||||
expect(lookup).toHaveBeenCalledTimes(2);
|
||||
});
|
||||
|
||||
it("resolves per-side native observations with one lookup per repeated exact ID", () => {
|
||||
const input = {
|
||||
...JAVASCRIPT_APPLICATION_VERSION_COMPARISON_EXAMPLE,
|
||||
left_native_observations: [native],
|
||||
right_native_observations: [native],
|
||||
};
|
||||
const inline = compareApplicationVersionsEvidence(input);
|
||||
const records = new Map(
|
||||
[input.left, input.right, native].map((record) => [
|
||||
record.evidence_id,
|
||||
record,
|
||||
]),
|
||||
);
|
||||
const lookup = vi.fn((id: string) => records.get(id));
|
||||
const retained = compareApplicationVersionsEvidence(
|
||||
{
|
||||
left: reference(input.left),
|
||||
right: reference(input.right),
|
||||
left_native_observations: [reference(native)],
|
||||
right_native_observations: [reference(native)],
|
||||
},
|
||||
lookup,
|
||||
);
|
||||
if (!inline.ok) throw inline.error;
|
||||
if (!retained.ok) throw retained.error;
|
||||
expect(retained.value.evidence.evidence_id).toBe(
|
||||
inline.value.evidence.evidence_id,
|
||||
);
|
||||
expect(retained.value.result).toEqual(inline.value.result);
|
||||
expect(lookup).toHaveBeenCalledTimes(3);
|
||||
});
|
||||
|
||||
it("uses retained graphs for exact export-shape comparison", () => {
|
||||
const input = JAVASCRIPT_EXPORT_SHAPE_COMPARISON_EXAMPLE;
|
||||
const inline = compareJavaScriptExportShapesEvidence(input);
|
||||
const records = new Map(
|
||||
[input.left, input.right].map((record) => [record.evidence_id, record]),
|
||||
);
|
||||
const retained = compareJavaScriptExportShapesEvidence(
|
||||
{
|
||||
...input,
|
||||
left: reference(input.left),
|
||||
right: reference(input.right),
|
||||
},
|
||||
(id) => records.get(id),
|
||||
);
|
||||
if (!inline.ok) throw inline.error;
|
||||
if (!retained.ok) throw retained.error;
|
||||
expect(retained.value.evidence.evidence_id).toBe(
|
||||
inline.value.evidence.evidence_id,
|
||||
);
|
||||
});
|
||||
|
||||
it("resolves nested static layers and runtime arrays before source-owned reconciliation", () => {
|
||||
const input = JAVASCRIPT_RUNTIME_RECONCILIATION_EXAMPLE;
|
||||
const records = new Map(
|
||||
[
|
||||
...input.static_layers.map(({ analysis }) => analysis),
|
||||
...input.runtime_observations,
|
||||
].map((record) => [record.evidence_id, record]),
|
||||
);
|
||||
const inline = reconcileJavaScriptRuntimeEvidence(input);
|
||||
const retained = reconcileJavaScriptRuntimeEvidence(
|
||||
{
|
||||
...input,
|
||||
static_layers: input.static_layers.map((layer) => ({
|
||||
...layer,
|
||||
analysis: reference(layer.analysis),
|
||||
})),
|
||||
runtime_observations: input.runtime_observations.map(reference),
|
||||
},
|
||||
(id) => records.get(id),
|
||||
);
|
||||
if (!inline.ok) throw inline.error;
|
||||
if (!retained.ok) throw retained.error;
|
||||
expect(retained.value.evidence.evidence_id).toBe(
|
||||
inline.value.evidence.evidence_id,
|
||||
);
|
||||
expect(
|
||||
retained.value.sources.map(({ evidence_id }) => evidence_id).sort(),
|
||||
).toEqual([...records.keys()].sort());
|
||||
});
|
||||
|
||||
it("preserves semantic-trace unknown coverage through retained sources", () => {
|
||||
const application = JAVASCRIPT_FEATURE_TRACE_EXAMPLE.application;
|
||||
const input = {
|
||||
application,
|
||||
query: {
|
||||
seed: { kind: "property", name: "unobserved" },
|
||||
direction: "ownership",
|
||||
},
|
||||
};
|
||||
const inline = traceJavaScriptSemanticsEvidence(input);
|
||||
const retained = traceJavaScriptSemanticsEvidence(
|
||||
{ ...input, application: reference(application) },
|
||||
(id) => (id === application.evidence_id ? application : undefined),
|
||||
);
|
||||
if (!inline.ok) throw inline.error;
|
||||
if (!retained.ok) throw retained.error;
|
||||
expect(retained.value.result).toEqual(inline.value.result);
|
||||
expect(retained.value.sources).toEqual([application]);
|
||||
});
|
||||
|
||||
it("reports missing nested references before reconciliation without guessing a source", () => {
|
||||
const layer = JAVASCRIPT_RUNTIME_RECONCILIATION_EXAMPLE.static_layers[0];
|
||||
if (layer === undefined) throw new Error("Expected an application layer");
|
||||
const result = reconcileJavaScriptRuntimeEvidence({
|
||||
...JAVASCRIPT_RUNTIME_RECONCILIATION_EXAMPLE,
|
||||
static_layers: [{ ...layer, analysis: reference(layer.analysis) }],
|
||||
});
|
||||
expect(result).toMatchObject({
|
||||
ok: false,
|
||||
error: { _tag: "EvidenceIntegrityError" },
|
||||
});
|
||||
});
|
||||
|
||||
it("honors cancellation before reading any retained graph", () => {
|
||||
const controller = new AbortController();
|
||||
const reason = { request: "cancel before source resolution" };
|
||||
controller.abort(reason);
|
||||
const application = JAVASCRIPT_FEATURE_TRACE_EXAMPLE.application;
|
||||
const lookup = vi.fn(() => application);
|
||||
const result = traceApplicationFeatureEvidence(
|
||||
{
|
||||
...JAVASCRIPT_FEATURE_TRACE_EXAMPLE,
|
||||
application: reference(application),
|
||||
},
|
||||
lookup,
|
||||
{ signal: controller.signal },
|
||||
);
|
||||
expect(result).toMatchObject({
|
||||
ok: false,
|
||||
error: { _tag: "AnalysisCancelledError", cause: reason },
|
||||
});
|
||||
expect(lookup).not.toHaveBeenCalled();
|
||||
});
|
||||
@@ -1,14 +1,21 @@
|
||||
import type { ExecutionOptions } from "../AnalysisProvider.js";
|
||||
import { resolve } from "node:path";
|
||||
import { z } from "zod";
|
||||
import type { z } from "zod";
|
||||
import {
|
||||
resolveEvidenceRequest,
|
||||
type EvidenceLookup,
|
||||
} from "../EvidenceInputResolver.js";
|
||||
import type { EvidenceWorkflowExecution } from "../investigation/EvidenceWorkflowExecution.js";
|
||||
import type { EvidenceRequestFrameInput } from "../../domain/evidenceInputContract.js";
|
||||
import { reconcileJavaScriptRuntimeRequestSchema } from "../../contracts/javascript/applicationWorkflowInputContracts.js";
|
||||
|
||||
import type { AnalysisError } from "../../domain/analysisErrorBase.js";
|
||||
import type { Evidence } from "../../domain/evidence.js";
|
||||
import { analysisInputErrorFromIssues } from "../../domain/inputIssueProjection.js";
|
||||
import {
|
||||
readRuntimeReconciliationInput,
|
||||
reconcileJavaScriptRuntime,
|
||||
} from "../../domain/javascript/javascriptRuntimeReconciliation.js";
|
||||
import { reconcileJavaScriptRuntimeInputSchema } from "../../domain/javascript/javascriptRuntimeReconciliationSchemas.js";
|
||||
import { runtimeReconciliationIdentitiesSchema } from "../../domain/javascript/javascriptRuntimeReconciliationSchemas.js";
|
||||
import { err, ok, type Result } from "../../domain/result.js";
|
||||
import { createJavaScriptRuntimeReconciliationEvidence } from "./JavaScriptRuntimeReconciliationEvidence.js";
|
||||
|
||||
@@ -17,32 +24,56 @@ const OPERATION = "reconcile_javascript_runtime" as const;
|
||||
/** Derive a combined JAG from verified static and passive-runtime Evidence. */
|
||||
export const reconcileJavaScriptRuntimeEvidence = (
|
||||
rawInput: unknown,
|
||||
): Result<Evidence, AnalysisError> => {
|
||||
const parsed = reconcileJavaScriptRuntimeInputSchema.safeParse(rawInput);
|
||||
if (!parsed.success)
|
||||
return err(
|
||||
analysisInputErrorFromIssues(OPERATION, parsed.error.issues, rawInput),
|
||||
);
|
||||
return reconcileJavaScriptRuntimeEvidenceValidated(parsed.data);
|
||||
};
|
||||
|
||||
/** Reconcile input already parsed by a trusted adapter boundary. */
|
||||
export const reconcileJavaScriptRuntimeEvidenceValidated = (
|
||||
parsedInput: z.output<typeof reconcileJavaScriptRuntimeInputSchema>,
|
||||
): Result<Evidence, AnalysisError> => {
|
||||
lookup?: EvidenceLookup,
|
||||
options: Pick<ExecutionOptions, "signal"> = {},
|
||||
): Result<EvidenceWorkflowExecution, AnalysisError> => {
|
||||
const selected = resolveEvidenceRequest(
|
||||
OPERATION,
|
||||
rawInput,
|
||||
reconcileJavaScriptRuntimeRequestSchema,
|
||||
lookup,
|
||||
options,
|
||||
);
|
||||
if (!selected.ok) return selected;
|
||||
const parsed = readRuntimeReconciliationInput(
|
||||
normalizeRuntimeMappingRoots(parsedInput),
|
||||
normalizeRuntimeMappingRoots(selected.value.input),
|
||||
);
|
||||
if (!parsed.ok) return parsed;
|
||||
const identities = runtimeReconciliationIdentitiesSchema.safeParse({
|
||||
static_layers: parsed.value.layers.map(({ role, evidence }) => ({
|
||||
role,
|
||||
analysis: evidence,
|
||||
})),
|
||||
runtime_observations: parsed.value.captures.map(({ evidence }) => evidence),
|
||||
});
|
||||
if (!identities.success)
|
||||
return err(
|
||||
analysisInputErrorFromIssues(
|
||||
OPERATION,
|
||||
identities.error.issues,
|
||||
rawInput,
|
||||
),
|
||||
);
|
||||
const result = reconcileJavaScriptRuntime(parsed.value);
|
||||
return ok(
|
||||
createJavaScriptRuntimeReconciliationEvidence(parsed.value.input, result),
|
||||
);
|
||||
return ok({
|
||||
evidence: createJavaScriptRuntimeReconciliationEvidence(
|
||||
parsed.value.input,
|
||||
result,
|
||||
),
|
||||
sources: [
|
||||
...parsed.value.layers.map(({ evidence }) => evidence),
|
||||
...parsed.value.captures.map(({ evidence }) => evidence),
|
||||
],
|
||||
});
|
||||
};
|
||||
|
||||
type RuntimeReconciliationFrame = EvidenceRequestFrameInput<
|
||||
z.output<typeof reconcileJavaScriptRuntimeRequestSchema>
|
||||
>;
|
||||
|
||||
const normalizeRuntimeMappingRoots = (
|
||||
input: z.output<typeof reconcileJavaScriptRuntimeInputSchema>,
|
||||
): z.output<typeof reconcileJavaScriptRuntimeInputSchema> => ({
|
||||
input: RuntimeReconciliationFrame,
|
||||
): RuntimeReconciliationFrame => ({
|
||||
...input,
|
||||
static_layers: input.static_layers.map((layer) => ({
|
||||
...layer,
|
||||
|
||||
@@ -1,26 +1,41 @@
|
||||
import { z } from "zod";
|
||||
|
||||
import type { ExecutionOptions } from "../AnalysisProvider.js";
|
||||
import {
|
||||
resolveEvidenceRequest,
|
||||
type EvidenceLookup,
|
||||
} from "../EvidenceInputResolver.js";
|
||||
import { traceJavaScriptSemanticsRequestSchema } from "../../contracts/javascript/applicationWorkflowInputContracts.js";
|
||||
import { AnalysisInputError } from "../../domain/analysisErrorCore.js";
|
||||
import type { AnalysisError } from "../../domain/analysisErrorBase.js";
|
||||
import {
|
||||
applicationGraphEvidenceInputError,
|
||||
parseApplicationGraphEvidence,
|
||||
} from "./JavaScriptApplicationEvidenceGraph.js";
|
||||
import type { Evidence } from "../../domain/evidence.js";
|
||||
import { queryJavaScriptSemanticGraph } from "../../domain/javascript/javascriptSemanticQuery.js";
|
||||
import {
|
||||
traceJavaScriptSemanticsInputSchema,
|
||||
type JavaScriptSemanticTraceResult,
|
||||
} from "../../domain/javascript/javascriptSemanticTraceSchemas.js";
|
||||
import type { JavaScriptSemanticTraceResult } from "../../domain/javascript/javascriptSemanticTraceSchemas.js";
|
||||
import { err, ok, type Result } from "../../domain/result.js";
|
||||
import { createJavaScriptSemanticTraceEvidence } from "./JavaScriptApplicationWorkflowEvidence.js";
|
||||
import type { ApplicationWorkflowExecution } from "./JavaScriptApplicationWorkflowService.js";
|
||||
|
||||
const OPERATION = "trace_javascript_semantics" as const;
|
||||
|
||||
/** Trace semantic relations from input already parsed by a trusted adapter. */
|
||||
export const traceJavaScriptSemanticsEvidenceValidated = (
|
||||
input: z.output<typeof traceJavaScriptSemanticsInputSchema>,
|
||||
): Result<Evidence, AnalysisError> => {
|
||||
/** Resolve and authenticate one application source before tracing semantic relations. */
|
||||
export const traceJavaScriptSemanticsEvidence = (
|
||||
rawInput: unknown,
|
||||
lookup?: EvidenceLookup,
|
||||
options: Pick<ExecutionOptions, "signal"> = {},
|
||||
): Result<
|
||||
ApplicationWorkflowExecution<JavaScriptSemanticTraceResult>,
|
||||
AnalysisError
|
||||
> => {
|
||||
const request = resolveEvidenceRequest(
|
||||
OPERATION,
|
||||
rawInput,
|
||||
traceJavaScriptSemanticsRequestSchema,
|
||||
lookup,
|
||||
options,
|
||||
);
|
||||
if (!request.ok) return request;
|
||||
const input = request.value.input;
|
||||
const sourceResult = parseApplicationGraphEvidence(input.application, [
|
||||
"application",
|
||||
]);
|
||||
@@ -36,7 +51,7 @@ export const traceJavaScriptSemanticsEvidenceValidated = (
|
||||
path: ["application"],
|
||||
reason: "invalid_value",
|
||||
expected:
|
||||
"inline Evidence from analyze_javascript_application with a semantic graph",
|
||||
"Evidence from analyze_javascript_application with a semantic graph",
|
||||
},
|
||||
]),
|
||||
);
|
||||
@@ -47,16 +62,15 @@ export const traceJavaScriptSemanticsEvidenceValidated = (
|
||||
evidence_links: [source.evidence.evidence_id],
|
||||
};
|
||||
const { allowed_relations, ...queryParameters } = input.query;
|
||||
return ok(
|
||||
createJavaScriptSemanticTraceEvidence(
|
||||
{
|
||||
application_evidence_id: source.evidence.evidence_id,
|
||||
query:
|
||||
allowed_relations === undefined
|
||||
? queryParameters
|
||||
: { ...queryParameters, allowed_relations },
|
||||
},
|
||||
result,
|
||||
),
|
||||
const evidence = createJavaScriptSemanticTraceEvidence(
|
||||
{
|
||||
application_evidence_id: source.evidence.evidence_id,
|
||||
query:
|
||||
allowed_relations === undefined
|
||||
? queryParameters
|
||||
: { ...queryParameters, allowed_relations },
|
||||
},
|
||||
result,
|
||||
);
|
||||
return ok({ evidence, result, sources: [source.evidence] });
|
||||
};
|
||||
@@ -1,3 +1,8 @@
|
||||
import {
|
||||
resolveEvidenceRequest,
|
||||
type EvidenceLookup,
|
||||
} from "../EvidenceInputResolver.js";
|
||||
import type { ExecutionOptions } from "../AnalysisProvider.js";
|
||||
import type { AnalysisError } from "../../domain/analysisErrorBase.js";
|
||||
import {
|
||||
createEvidence,
|
||||
@@ -5,6 +10,7 @@ import {
|
||||
type EvidenceSubjectTarget,
|
||||
} from "../../domain/evidence.js";
|
||||
import {
|
||||
managedApplicationGraphReferenceInputSchema,
|
||||
projectManagedGraph,
|
||||
readManagedGraphProjection,
|
||||
type ManagedApplicationGraphResult,
|
||||
@@ -19,12 +25,30 @@ const OPERATION = "project_managed_application_graph" as const;
|
||||
/** Project authenticated managed Evidence into a provider-neutral graph. */
|
||||
export const projectManagedApplicationGraphEvidence = (
|
||||
rawInput: unknown,
|
||||
lookup?: EvidenceLookup,
|
||||
options: Pick<ExecutionOptions, "signal"> = {},
|
||||
): Result<Evidence, AnalysisError> => {
|
||||
const request = readManagedGraphProjection(rawInput);
|
||||
const request = resolveManagedGraphEvidenceInput(rawInput, lookup, options);
|
||||
if (!request.ok) return request;
|
||||
return ok(projectManagedGraphEvidence(request.value));
|
||||
};
|
||||
|
||||
/** Resolve selected records before the managed owner authenticates and joins them. */
|
||||
export const resolveManagedGraphEvidenceInput = (
|
||||
rawInput: unknown,
|
||||
lookup?: EvidenceLookup,
|
||||
options: Pick<ExecutionOptions, "signal"> = {},
|
||||
): Result<ManagedGraphProjection, AnalysisError> => {
|
||||
const request = resolveEvidenceRequest(
|
||||
OPERATION,
|
||||
rawInput,
|
||||
managedApplicationGraphReferenceInputSchema,
|
||||
lookup,
|
||||
options,
|
||||
);
|
||||
return request.ok ? readManagedGraphProjection(request.value.input) : request;
|
||||
};
|
||||
|
||||
/** Project the workflow owner's authenticated, identity-checked source facts. */
|
||||
export const projectManagedGraphEvidence = (
|
||||
request: ManagedGraphProjection,
|
||||
|
||||
@@ -1,8 +1,12 @@
|
||||
import { parseAnalysisInput } from "../AnalysisInput.js";
|
||||
import { z } from "zod";
|
||||
import {
|
||||
resolveEvidenceRequest,
|
||||
type EvidenceLookup,
|
||||
} from "../EvidenceInputResolver.js";
|
||||
import type { ExecutionOptions } from "../AnalysisProvider.js";
|
||||
|
||||
import {
|
||||
compareManagedMembers,
|
||||
compareManagedMembersRequestSchema,
|
||||
type ManagedMemberComparisonResult,
|
||||
} from "../../domain/managed/managedMemberComparison.js";
|
||||
import { readManagedMemberEvidence } from "../../domain/managed/managedMemberComparisonMatch.js";
|
||||
@@ -31,11 +35,6 @@ import {
|
||||
MANAGED_WORKFLOW_PROVIDER,
|
||||
} from "../../domain/investigationProviders.js";
|
||||
|
||||
const managedMemberComparisonRequestSchema = z.strictObject({
|
||||
left: z.unknown(),
|
||||
right: z.unknown(),
|
||||
});
|
||||
|
||||
export interface ManagedMemberComparisonExecution {
|
||||
readonly evidence: Evidence;
|
||||
readonly summary: ManagedMemberComparisonResult["summary"];
|
||||
@@ -45,17 +44,21 @@ export interface ManagedMemberComparisonExecution {
|
||||
/** Authenticate both source records, then compare their parsed observations. */
|
||||
export const executeManagedMemberComparison = (
|
||||
input: unknown,
|
||||
lookup?: EvidenceLookup,
|
||||
options: Pick<ExecutionOptions, "signal"> = {},
|
||||
): Result<ManagedMemberComparisonExecution, AnalysisError> => {
|
||||
const operation = "compare_managed_members";
|
||||
const request = parseAnalysisInput(
|
||||
const request = resolveEvidenceRequest(
|
||||
operation,
|
||||
input,
|
||||
managedMemberComparisonRequestSchema,
|
||||
compareManagedMembersRequestSchema,
|
||||
lookup,
|
||||
options,
|
||||
);
|
||||
if (!request.ok) return request;
|
||||
const left = readManagedMemberEvidence(request.value.left, "left");
|
||||
const left = readManagedMemberEvidence(request.value.input.left, "left");
|
||||
if (!left.ok) return left;
|
||||
const right = readManagedMemberEvidence(request.value.right, "right");
|
||||
const right = readManagedMemberEvidence(request.value.input.right, "right");
|
||||
if (!right.ok) return right;
|
||||
if (left.value.evidence.evidence_id === right.value.evidence.evidence_id)
|
||||
return err(
|
||||
|
||||
@@ -1,3 +1,8 @@
|
||||
import {
|
||||
resolveEvidenceRequest,
|
||||
type EvidenceLookup,
|
||||
} from "../EvidenceInputResolver.js";
|
||||
import type { ExecutionOptions } from "../AnalysisProvider.js";
|
||||
import type { AnalysisError } from "../../domain/analysisErrorBase.js";
|
||||
import { createEvidence, type Evidence } from "../../domain/evidence.js";
|
||||
import { ok, type Result } from "../../domain/result.js";
|
||||
@@ -9,7 +14,10 @@ import {
|
||||
readManagedNativeVerificationSource,
|
||||
type ManagedNativeVerificationSource,
|
||||
} from "../../domain/managed/managedNativeVerificationSource.js";
|
||||
import type { ManagedNativeVerificationResult } from "../../domain/managed/managedNativeVerificationSchemas.js";
|
||||
import {
|
||||
managedNativeVerificationRequestSchema,
|
||||
type ManagedNativeVerificationResult,
|
||||
} from "../../domain/managed/managedNativeVerificationSchemas.js";
|
||||
import { MANAGED_WORKFLOW_PROVIDER } from "../../domain/investigationProviders.js";
|
||||
|
||||
const OPERATION = "verify_managed_native_boundaries" as const;
|
||||
@@ -24,8 +32,18 @@ export interface ManagedNativeVerificationExecution {
|
||||
/** Resolve untrusted source Evidence, then execute over the trusted parsed source. */
|
||||
export const executeManagedNativeBoundaries = (
|
||||
rawInput: unknown,
|
||||
lookup?: EvidenceLookup,
|
||||
options: Pick<ExecutionOptions, "signal"> = {},
|
||||
): Result<ManagedNativeVerificationExecution, AnalysisError> => {
|
||||
const source = readManagedNativeVerificationSource(rawInput);
|
||||
const request = resolveEvidenceRequest(
|
||||
OPERATION,
|
||||
rawInput,
|
||||
managedNativeVerificationRequestSchema,
|
||||
lookup,
|
||||
options,
|
||||
);
|
||||
if (!request.ok) return request;
|
||||
const source = readManagedNativeVerificationSource(request.value.input);
|
||||
if (!source.ok) return source;
|
||||
const result = verifyManagedNativeVerificationSource(source.value);
|
||||
const evidence = createManagedNativeVerificationEvidence(
|
||||
|
||||
@@ -10,21 +10,29 @@ import {
|
||||
} from "../../domain/managed/managedReconstruction.js";
|
||||
import { ok, type Result } from "../../domain/result.js";
|
||||
import { MANAGED_WORKFLOW_PROVIDER } from "../../domain/investigationProviders.js";
|
||||
import { parseAnalysisInput } from "../AnalysisInput.js";
|
||||
import {
|
||||
resolveEvidenceRequest,
|
||||
type EvidenceLookup,
|
||||
} from "../EvidenceInputResolver.js";
|
||||
import type { ExecutionOptions } from "../AnalysisProvider.js";
|
||||
|
||||
const OPERATION = "import_managed_reconstruction" as const;
|
||||
|
||||
/** Validate the request envelope and authenticate its managed Evidence source once. */
|
||||
export const resolveManagedReconstructionEvidenceInput = (
|
||||
rawInput: unknown,
|
||||
lookup?: EvidenceLookup,
|
||||
options: Pick<ExecutionOptions, "signal"> = {},
|
||||
): Result<ResolvedManagedReconstructionImport, AnalysisError> => {
|
||||
const parsed = parseAnalysisInput(
|
||||
const parsed = resolveEvidenceRequest(
|
||||
OPERATION,
|
||||
rawInput,
|
||||
managedReconstructionImportRequestSchema,
|
||||
lookup,
|
||||
options,
|
||||
);
|
||||
if (!parsed.ok) return parsed;
|
||||
return resolveManagedReconstructionImport(parsed.value);
|
||||
return resolveManagedReconstructionImport(parsed.value.input);
|
||||
};
|
||||
|
||||
/** Execute from the source facts returned by the resolver. */
|
||||
@@ -38,8 +46,14 @@ export const importManagedReconstructionEvidenceResolved = (
|
||||
/** Validate the raw request envelope, then authenticate and import its source. */
|
||||
export const importManagedReconstructionEvidence = (
|
||||
rawInput: unknown,
|
||||
lookup?: EvidenceLookup,
|
||||
options: Pick<ExecutionOptions, "signal"> = {},
|
||||
): Result<Evidence, AnalysisError> => {
|
||||
const resolved = resolveManagedReconstructionEvidenceInput(rawInput);
|
||||
const resolved = resolveManagedReconstructionEvidenceInput(
|
||||
rawInput,
|
||||
lookup,
|
||||
options,
|
||||
);
|
||||
if (!resolved.ok) return resolved;
|
||||
return importManagedReconstructionEvidenceResolved(resolved.value);
|
||||
};
|
||||
|
||||
@@ -130,12 +130,11 @@ export const loadNativeMatchingBundleSources = async ({
|
||||
await admission.checkpoint(128);
|
||||
const entry = index.get(id);
|
||||
if (entry === undefined)
|
||||
throw new EvidenceReferenceError(
|
||||
id,
|
||||
"missing",
|
||||
"Complete Evidence closure in the supplied bundles or inline inputs",
|
||||
null,
|
||||
);
|
||||
throw new EvidenceReferenceError(id, "missing", {
|
||||
expected:
|
||||
"Complete Evidence closure in the supplied bundles or inline inputs",
|
||||
actual: null,
|
||||
});
|
||||
if (entry.file >= 0) files.add(entry.file);
|
||||
for (const linked of entry.links) needed.add(linked);
|
||||
}
|
||||
|
||||
@@ -0,0 +1,217 @@
|
||||
import {
|
||||
createAnalysisExecution,
|
||||
type AnalysisOperationPort,
|
||||
} from "../AnalysisProvider.js";
|
||||
import { ghidraFunctionDossier } from "../../domain/ghidraValues.fixture.js";
|
||||
import { functionDossierSchema } from "../../domain/native/functionDossier.js";
|
||||
import { jsonValueSchema } from "../../domain/jsonValue.js";
|
||||
import { ok } from "../../domain/result.js";
|
||||
|
||||
/** Simulated provider seam for value traces; not a real decompiler observation. */
|
||||
export const valueTraceSubject = {
|
||||
path: "/fixture",
|
||||
sha256: "a".repeat(64),
|
||||
format: "mach-o" as const,
|
||||
architecture: "arm64" as const,
|
||||
};
|
||||
export const valueTraceProvider = {
|
||||
id: "fixture",
|
||||
name: "Fixture",
|
||||
version: "1",
|
||||
};
|
||||
export const pcodeScalar = {
|
||||
kind: "constant",
|
||||
size_bytes: 4,
|
||||
location: null,
|
||||
constant_hex: "3",
|
||||
};
|
||||
|
||||
/** A live one-input, one-output operation; a CALL passes one argument. */
|
||||
export const pcodeOperation = (
|
||||
id: string,
|
||||
address: string,
|
||||
opcode: string,
|
||||
) => ({
|
||||
id,
|
||||
address,
|
||||
sequence: 0,
|
||||
opcode,
|
||||
is_dead: true,
|
||||
block_membership: "member",
|
||||
inputs: [pcodeScalar, pcodeScalar],
|
||||
output: pcodeScalar,
|
||||
});
|
||||
|
||||
export const nativeFlowDossier = (
|
||||
base: ReturnType<typeof functionDossierSchema.parse>,
|
||||
facts: {
|
||||
address: string;
|
||||
name: string;
|
||||
operations: ReturnType<typeof pcodeOperation>[];
|
||||
parameters: { ordinal: number; name: string; data_type: string }[];
|
||||
},
|
||||
) => {
|
||||
const { address, name, operations, parameters } = facts;
|
||||
return jsonValueSchema.parse({
|
||||
...base,
|
||||
procedure: {
|
||||
address,
|
||||
name,
|
||||
classification: null,
|
||||
body: {
|
||||
available: false,
|
||||
reason: "The provider did not report complete function body ranges.",
|
||||
},
|
||||
signature: null,
|
||||
locals: [],
|
||||
},
|
||||
native_value_flow: {
|
||||
available: true,
|
||||
provenance: "ghidra-high-pcode",
|
||||
operations,
|
||||
def_use: [],
|
||||
effects: [],
|
||||
parameters,
|
||||
parameter_uses: [],
|
||||
truncated: false,
|
||||
omitted_operations_lower_bound: 0,
|
||||
known_omitted_inputs: 0,
|
||||
known_omitted_edges: 0,
|
||||
limitations: [],
|
||||
},
|
||||
});
|
||||
};
|
||||
|
||||
interface StaticProcedure {
|
||||
readonly operations: ReturnType<typeof pcodeOperation>[];
|
||||
readonly parameters?: { ordinal: number; name: string; data_type: string }[];
|
||||
readonly def_use?: { definition: string; use: string; input_index: number }[];
|
||||
readonly pcodeTruncated?: boolean;
|
||||
}
|
||||
|
||||
interface GraphOptions {
|
||||
readonly sessionSubject?: boolean;
|
||||
}
|
||||
|
||||
/** A static call graph whose call sites resolve to the listed destinations. */
|
||||
export const staticValueGraph = (
|
||||
procedures: Readonly<Record<string, StaticProcedure>>,
|
||||
calls: Readonly<
|
||||
Record<
|
||||
string,
|
||||
readonly { destination: string; status: "direct" | "candidate" }[]
|
||||
>
|
||||
>,
|
||||
options: GraphOptions = {},
|
||||
) => {
|
||||
// A production session binds an omitted subject to its opened target.
|
||||
const subject =
|
||||
options.sessionSubject === true ? {} : { subject: valueTraceSubject };
|
||||
const base = functionDossierSchema.parse(ghidraFunctionDossier());
|
||||
const analyzed: string[] = [];
|
||||
const resolved: string[] = [];
|
||||
const port: AnalysisOperationPort = {
|
||||
execute: async (operation, parameters) => {
|
||||
if (operation === "resolve_native_call_targets") {
|
||||
const address = String(parameters.address);
|
||||
resolved.push(address);
|
||||
const targets = calls[address] ?? [];
|
||||
const ambiguous = targets.some(({ status }) => status === "candidate");
|
||||
return ok(
|
||||
createAnalysisExecution(
|
||||
{
|
||||
call_site: address,
|
||||
procedure: ambiguous ? null : (targets[0]?.destination ?? null),
|
||||
status: ambiguous
|
||||
? "ambiguous"
|
||||
: targets.length === 0
|
||||
? "unresolved"
|
||||
: "direct",
|
||||
mechanism: ambiguous ? "computed" : "direct",
|
||||
targets: targets.map(({ destination, status }) => ({
|
||||
address: destination,
|
||||
procedure: destination,
|
||||
status,
|
||||
basis: "provider-reference",
|
||||
references: [],
|
||||
})),
|
||||
limitations: [],
|
||||
},
|
||||
valueTraceProvider,
|
||||
subject,
|
||||
),
|
||||
);
|
||||
}
|
||||
if (operation !== "analyze_function")
|
||||
throw new Error(`Unexpected native trace operation: ${operation}`);
|
||||
const address = String(parameters.procedure);
|
||||
analyzed.push(address);
|
||||
const facts = procedures[address];
|
||||
if (facts === undefined) throw new Error(`Unexpected ${address}`);
|
||||
const dossier = functionDossierSchema.parse(
|
||||
nativeFlowDossier(base, {
|
||||
address,
|
||||
name: address,
|
||||
operations: facts.operations,
|
||||
parameters: facts.parameters ?? [],
|
||||
}),
|
||||
);
|
||||
if (!dossier.native_value_flow?.available)
|
||||
throw new Error("Invalid fixture");
|
||||
dossier.native_value_flow.def_use = facts.def_use ?? [];
|
||||
if (facts.pcodeTruncated === true) {
|
||||
dossier.native_value_flow.truncated = true;
|
||||
dossier.native_value_flow.omitted_operations_lower_bound = 1;
|
||||
}
|
||||
return ok(createAnalysisExecution(dossier, valueTraceProvider, subject));
|
||||
},
|
||||
};
|
||||
return { port, analyzed, resolved };
|
||||
};
|
||||
|
||||
const copies = (address: string, count: number) =>
|
||||
Array.from({ length: count }, (_, index) => ({
|
||||
...pcodeOperation(`op-${String(index)}`, address, "COPY"),
|
||||
sequence: index,
|
||||
}));
|
||||
const parameter = [{ ordinal: 0, name: "value", data_type: "/int" }];
|
||||
|
||||
/**
|
||||
* 0x1000 calls 0x2000 and 0x3000, and 0x2000 also calls 0x3000. With
|
||||
* max_depth 1, each one-argument call binds to its callee's single parameter:
|
||||
* 11 nodes and 9 edges, reached in breadth-first order.
|
||||
*/
|
||||
export const valueTraceDiamond = (options: GraphOptions = {}) =>
|
||||
staticValueGraph(
|
||||
{
|
||||
"0x1000": {
|
||||
operations: [
|
||||
pcodeOperation("call-b", "0x1004", "CALL"),
|
||||
pcodeOperation("call-c", "0x1008", "CALL"),
|
||||
...copies("0x1000", 2),
|
||||
],
|
||||
def_use: [{ definition: "op-0", use: "op-1", input_index: 0 }],
|
||||
},
|
||||
"0x2000": {
|
||||
operations: [
|
||||
pcodeOperation("call-c", "0x2004", "CALL"),
|
||||
pcodeOperation("x", "0x2000", "COPY"),
|
||||
],
|
||||
parameters: parameter,
|
||||
},
|
||||
"0x3000": {
|
||||
operations: copies("0x3000", 3),
|
||||
parameters: parameter,
|
||||
def_use: [
|
||||
{ definition: "op-0", use: "op-1", input_index: 0 },
|
||||
{ definition: "op-1", use: "op-2", input_index: 0 },
|
||||
],
|
||||
},
|
||||
},
|
||||
{
|
||||
"0x1004": [{ destination: "0x2000", status: "direct" }],
|
||||
"0x1008": [{ destination: "0x3000", status: "direct" }],
|
||||
"0x2004": [{ destination: "0x3000", status: "direct" }],
|
||||
},
|
||||
options,
|
||||
);
|
||||
@@ -17,30 +17,15 @@ import {
|
||||
} from "../../domain/analysisErrorCore.js";
|
||||
import { NoBinaryOpenError } from "../../domain/configurationErrors.js";
|
||||
import { ok, err } from "../../domain/result.js";
|
||||
|
||||
const subject = {
|
||||
path: "/fixture",
|
||||
sha256: "a".repeat(64),
|
||||
format: "mach-o" as const,
|
||||
architecture: "arm64" as const,
|
||||
};
|
||||
const provider = { id: "fixture", name: "Fixture", version: "1" };
|
||||
const scalar = {
|
||||
kind: "constant",
|
||||
size_bytes: 4,
|
||||
location: null,
|
||||
constant_hex: "3",
|
||||
};
|
||||
const node = (id: string, address: string, opcode: string) => ({
|
||||
id,
|
||||
address,
|
||||
sequence: 0,
|
||||
opcode,
|
||||
is_dead: true,
|
||||
block_membership: "member",
|
||||
inputs: [scalar, scalar],
|
||||
output: scalar,
|
||||
});
|
||||
import {
|
||||
nativeFlowDossier,
|
||||
pcodeOperation as node,
|
||||
pcodeScalar as scalar,
|
||||
staticValueGraph,
|
||||
valueTraceDiamond,
|
||||
valueTraceProvider as provider,
|
||||
valueTraceSubject as subject,
|
||||
} from "./NativeValueTrace.fixture.js";
|
||||
const analysis: AnalysisOperationPort = {
|
||||
execute: async (operation, parameters) => {
|
||||
if (operation === "resolve_native_call_targets")
|
||||
@@ -247,7 +232,8 @@ describe("native value parameter-use node membership", () => {
|
||||
kind: "parameter",
|
||||
}),
|
||||
],
|
||||
truncated: true,
|
||||
page_truncated: true,
|
||||
traversal: { status: "complete" },
|
||||
});
|
||||
expect(includedResult.edges).toHaveLength(parameterUseCount);
|
||||
|
||||
@@ -265,8 +251,16 @@ describe("native value parameter-use node membership", () => {
|
||||
total_edges: 0,
|
||||
nodes: [expect.objectContaining({ kind: "operation" })],
|
||||
edges: [],
|
||||
truncated: true,
|
||||
page_truncated: true,
|
||||
traversal: { status: "truncated", reasons: ["max_nodes"] },
|
||||
});
|
||||
expect(omittedResult.unknowns).toEqual([
|
||||
{
|
||||
procedure: procedureAddress,
|
||||
address: null,
|
||||
reason: "Node budget reached",
|
||||
},
|
||||
]);
|
||||
});
|
||||
|
||||
it.each([3, 4])(
|
||||
@@ -336,46 +330,6 @@ describe("native value parameter-use node membership", () => {
|
||||
);
|
||||
});
|
||||
|
||||
const nativeFlowDossier = (
|
||||
base: ReturnType<typeof functionDossierSchema.parse>,
|
||||
facts: {
|
||||
address: string;
|
||||
name: string;
|
||||
operations: ReturnType<typeof node>[];
|
||||
parameters: { ordinal: number; name: string; data_type: string }[];
|
||||
},
|
||||
) => {
|
||||
const { address, name, operations, parameters } = facts;
|
||||
return jsonValueSchema.parse({
|
||||
...base,
|
||||
procedure: {
|
||||
address,
|
||||
name,
|
||||
classification: null,
|
||||
body: {
|
||||
available: false,
|
||||
reason: "The provider did not report complete function body ranges.",
|
||||
},
|
||||
signature: null,
|
||||
locals: [],
|
||||
},
|
||||
native_value_flow: {
|
||||
available: true,
|
||||
provenance: "ghidra-high-pcode",
|
||||
operations,
|
||||
def_use: [],
|
||||
effects: [],
|
||||
parameters,
|
||||
parameter_uses: [],
|
||||
truncated: false,
|
||||
omitted_operations_lower_bound: 0,
|
||||
known_omitted_inputs: 0,
|
||||
known_omitted_edges: 0,
|
||||
limitations: [],
|
||||
},
|
||||
});
|
||||
};
|
||||
|
||||
const callBindingAnalysis = (input: {
|
||||
readonly callerAddress: string;
|
||||
readonly calleeAddress: string;
|
||||
@@ -521,7 +475,12 @@ describe("native value failed-callee traversal", () => {
|
||||
.filter(({ kind }) => kind === "call")
|
||||
.map(({ source, target }) => ({ source, target })),
|
||||
).toEqual(expectedCalls);
|
||||
expect(trace.truncated).toBe(false);
|
||||
// A failed callee is an explicit unknown, not a traversal budget stop.
|
||||
expect(trace).toMatchObject({
|
||||
page_truncated: false,
|
||||
traversal: { status: "complete" },
|
||||
truncated: false,
|
||||
});
|
||||
}
|
||||
const capped = await traceNativeValues(port, {
|
||||
procedure: "0x1000",
|
||||
@@ -533,7 +492,16 @@ describe("native value failed-callee traversal", () => {
|
||||
expect(
|
||||
cappedTrace.edges.map(({ source, target }) => ({ source, target })),
|
||||
).toEqual(expectedCalls.slice(0, 2));
|
||||
expect(cappedTrace.truncated).toBe(true);
|
||||
expect(cappedTrace).toMatchObject({
|
||||
page_truncated: false,
|
||||
traversal: { status: "truncated", reasons: ["max_edges"] },
|
||||
truncated: true,
|
||||
});
|
||||
expect(cappedTrace.unknowns).toContainEqual({
|
||||
procedure: "0x1000",
|
||||
address: null,
|
||||
reason: "Edge budget reached",
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
@@ -647,7 +615,7 @@ describe("native value call-binding fan-out", () => {
|
||||
expect(completeResult).toMatchObject({
|
||||
total_nodes: callCount + 2,
|
||||
total_edges: callCount * 3,
|
||||
truncated: false,
|
||||
traversal: { status: "complete" },
|
||||
unknowns: [],
|
||||
});
|
||||
expect(edgeShape(completeResult.edges)).toEqual(expectedEdges);
|
||||
@@ -661,7 +629,8 @@ describe("native value call-binding fan-out", () => {
|
||||
expect(cappedResult).toMatchObject({
|
||||
total_nodes: callCount + 2,
|
||||
total_edges: callCount * 3 - 1,
|
||||
truncated: true,
|
||||
traversal: { status: "truncated", reasons: ["max_edges"] },
|
||||
unknowns: [budgetUnknown(callerAddress, "Edge budget reached")],
|
||||
});
|
||||
expect(edgeShape(cappedResult.edges)).toEqual(
|
||||
expectedEdges.slice(0, callCount * 3 - 1),
|
||||
@@ -676,12 +645,15 @@ describe("native value call-binding fan-out", () => {
|
||||
expect(nodeCappedResult).toMatchObject({
|
||||
total_nodes: callCount + 1,
|
||||
total_edges: callCount * 2,
|
||||
truncated: true,
|
||||
unknowns: Array.from({ length: callCount }, () => ({
|
||||
procedure: callerAddress,
|
||||
address: null,
|
||||
reason: `Parameter bindings for ${calleeAddress} are unavailable or omitted`,
|
||||
})),
|
||||
traversal: { status: "truncated", reasons: ["max_nodes"] },
|
||||
unknowns: [
|
||||
budgetUnknown(calleeAddress, "Node budget reached"),
|
||||
...Array.from({ length: callCount }, () => ({
|
||||
procedure: callerAddress,
|
||||
address: null,
|
||||
reason: `Parameter bindings for ${calleeAddress} are unavailable or omitted`,
|
||||
})),
|
||||
],
|
||||
});
|
||||
expect(edgeShape(nodeCappedResult.edges)).toEqual(
|
||||
expectedEdges.filter(({ kind }) => kind !== "argument-binding"),
|
||||
@@ -698,6 +670,8 @@ describe("bounded native value dependency composition", () => {
|
||||
expect(bounded.ok).toBe(true);
|
||||
if (bounded.ok)
|
||||
expect(bounded.value).toMatchObject({
|
||||
page_truncated: false,
|
||||
traversal: { status: "truncated", reasons: ["max_functions"] },
|
||||
truncated: true,
|
||||
decompilations: 1,
|
||||
});
|
||||
@@ -707,7 +681,12 @@ describe("bounded native value dependency composition", () => {
|
||||
});
|
||||
expect(page.ok).toBe(true);
|
||||
if (page.ok)
|
||||
expect(page.value).toMatchObject({ total_nodes: 3, next_offset: 1 });
|
||||
expect(page.value).toMatchObject({
|
||||
total_nodes: 3,
|
||||
next_offset: 1,
|
||||
page_truncated: true,
|
||||
traversal: { status: "complete" },
|
||||
});
|
||||
const ambiguous = await traceNativeValues(
|
||||
{
|
||||
execute: async (operation, parameters, options) => {
|
||||
@@ -744,6 +723,10 @@ describe("bounded native value dependency composition", () => {
|
||||
expect(ambiguous.value).toMatchObject({
|
||||
decompilations: 1,
|
||||
edges: [expect.objectContaining({ status: "candidate" })],
|
||||
traversal: {
|
||||
status: "truncated",
|
||||
reasons: ["ambiguous_call_candidates"],
|
||||
},
|
||||
});
|
||||
});
|
||||
it("propagates cancellation and rejects cross-target dependencies", async () => {
|
||||
@@ -851,11 +834,19 @@ describe("native value node byte budgets", () => {
|
||||
);
|
||||
if (!result.ok) throw result.error;
|
||||
expect(nativeValueTraceSchema.parse(result.value)).toMatchObject({
|
||||
traversal: { status: "truncated", reasons: ["node_payload_bytes"] },
|
||||
truncated: true,
|
||||
total_nodes: 0,
|
||||
total_edges: 0,
|
||||
nodes: [],
|
||||
edges: [],
|
||||
unknowns: [
|
||||
{
|
||||
procedure: "0x1000",
|
||||
address: null,
|
||||
reason: "Node payload budget reached",
|
||||
},
|
||||
],
|
||||
});
|
||||
});
|
||||
|
||||
@@ -889,6 +880,7 @@ describe("native value node byte budgets", () => {
|
||||
if (!result.ok) throw result.error;
|
||||
const trace = nativeValueTraceSchema.parse(result.value);
|
||||
expect(trace).toMatchObject({
|
||||
traversal: { status: "truncated", reasons: ["node_payload_bytes"] },
|
||||
truncated: true,
|
||||
total_nodes: 2,
|
||||
total_edges: 0,
|
||||
@@ -929,6 +921,7 @@ describe("native value node byte budgets", () => {
|
||||
if (!result.ok) throw result.error;
|
||||
const trace = nativeValueTraceSchema.parse(result.value);
|
||||
expect(trace).toMatchObject({
|
||||
traversal: { status: "truncated", reasons: ["node_payload_bytes"] },
|
||||
truncated: true,
|
||||
total_nodes: 2,
|
||||
total_edges: 1,
|
||||
@@ -974,7 +967,10 @@ describe("native value byte-bound call bindings", () => {
|
||||
);
|
||||
if (!result.ok) throw result.error;
|
||||
const trace = nativeValueTraceSchema.parse(result.value);
|
||||
expect(trace.truncated).toBe(true);
|
||||
expect(trace).toMatchObject({
|
||||
traversal: { status: "truncated", reasons: ["node_payload_bytes"] },
|
||||
truncated: true,
|
||||
});
|
||||
expect(trace.nodes.map(({ kind }) => kind)).toEqual([
|
||||
"operation",
|
||||
"operation",
|
||||
@@ -1041,3 +1037,293 @@ describe("native value validation failure provenance", () => {
|
||||
},
|
||||
);
|
||||
});
|
||||
|
||||
const budgetUnknown = (procedure: string, reason: string) => ({
|
||||
procedure,
|
||||
address: null,
|
||||
reason,
|
||||
});
|
||||
|
||||
const traced = async (
|
||||
port: AnalysisOperationPort,
|
||||
request: Record<string, unknown>,
|
||||
) => {
|
||||
const result = await traceNativeValues(port, request);
|
||||
if (!result.ok) throw result.error;
|
||||
return nativeValueTraceSchema.parse(result.value);
|
||||
};
|
||||
|
||||
const omittedBindings = (callee: string, callers: readonly string[]) =>
|
||||
callers.map((procedure) =>
|
||||
budgetUnknown(
|
||||
procedure,
|
||||
`Parameter bindings for ${callee} are unavailable or omitted`,
|
||||
),
|
||||
);
|
||||
const edgeKey = (edge: { source: string; target: string; kind: string }) =>
|
||||
`${edge.kind} ${edge.source} ${edge.target}`;
|
||||
|
||||
describe("native value trace page omission", () => {
|
||||
it("assembles every page of a complete trace without reporting traversal truncation", async () => {
|
||||
const { port } = valueTraceDiamond();
|
||||
const request = { procedure: "0x1000", max_depth: 1 };
|
||||
const complete = await traced(port, request);
|
||||
expect(complete).toMatchObject({
|
||||
total_nodes: 11,
|
||||
total_edges: 9,
|
||||
next_offset: null,
|
||||
page_truncated: false,
|
||||
traversal: { status: "complete" },
|
||||
truncated: false,
|
||||
unknowns: [],
|
||||
});
|
||||
const pages: (typeof complete)[] = [];
|
||||
for (let offset: number | null = 0; offset !== null;) {
|
||||
const page = await traced(port, { ...request, offset, limit: 2 });
|
||||
pages.push(page);
|
||||
offset = page.next_offset;
|
||||
}
|
||||
expect(pages).toHaveLength(6);
|
||||
for (const page of pages)
|
||||
expect(page).toMatchObject({
|
||||
total_nodes: 11,
|
||||
total_edges: 9,
|
||||
page_truncated: true,
|
||||
traversal: { status: "complete" },
|
||||
truncated: true,
|
||||
unknowns: [],
|
||||
});
|
||||
expect(pages.at(-1)).toMatchObject({ next_offset: null, nodes: [{}] });
|
||||
expect(pages.flatMap(({ nodes }) => nodes)).toEqual(complete.nodes);
|
||||
expect(pages.flatMap(({ edges }) => edges.map(edgeKey)).sort()).toEqual(
|
||||
complete.edges.map(edgeKey).sort(),
|
||||
);
|
||||
|
||||
expect(await traced(port, { ...request, limit: 11 })).toMatchObject({
|
||||
next_offset: null,
|
||||
page_truncated: false,
|
||||
truncated: false,
|
||||
});
|
||||
});
|
||||
|
||||
it("keeps a page beyond the retained nodes separate from traversal truncation", async () => {
|
||||
const { port } = valueTraceDiamond();
|
||||
expect(
|
||||
await traced(port, { procedure: "0x1000", max_depth: 1, offset: 100 }),
|
||||
).toMatchObject({
|
||||
nodes: [],
|
||||
edges: [],
|
||||
total_nodes: 11,
|
||||
next_offset: null,
|
||||
page_truncated: true,
|
||||
traversal: { status: "complete" },
|
||||
});
|
||||
});
|
||||
|
||||
it("does not report an exactly filled node or edge budget as truncation", async () => {
|
||||
const { port } = valueTraceDiamond();
|
||||
expect(
|
||||
await traced(port, {
|
||||
procedure: "0x1000",
|
||||
max_depth: 1,
|
||||
max_nodes: 11,
|
||||
max_edges: 9,
|
||||
}),
|
||||
).toMatchObject({
|
||||
total_nodes: 11,
|
||||
total_edges: 9,
|
||||
traversal: { status: "complete" },
|
||||
truncated: false,
|
||||
unknowns: [],
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
describe("native value traversal budgets", () => {
|
||||
it.each([
|
||||
{
|
||||
label: "node",
|
||||
budgets: { max_nodes: 9 },
|
||||
reasons: ["max_nodes"],
|
||||
unknowns: [
|
||||
{ procedure: "0x3000", address: null, reason: "Node budget reached" },
|
||||
...omittedBindings("0x3000", ["0x1000", "0x2000"]),
|
||||
],
|
||||
nodes: 9,
|
||||
},
|
||||
{
|
||||
label: "edge",
|
||||
budgets: { max_edges: 2 },
|
||||
reasons: ["max_edges"],
|
||||
unknowns: [
|
||||
{ procedure: "0x1000", address: null, reason: "Edge budget reached" },
|
||||
...omittedBindings("0x2000", ["0x1000"]),
|
||||
],
|
||||
nodes: 4,
|
||||
},
|
||||
{
|
||||
label: "node and edge",
|
||||
budgets: { max_nodes: 9, max_edges: 4 },
|
||||
reasons: ["max_nodes", "max_edges"],
|
||||
unknowns: [
|
||||
{ procedure: "0x3000", address: null, reason: "Node budget reached" },
|
||||
{ procedure: "0x3000", address: null, reason: "Edge budget reached" },
|
||||
...omittedBindings("0x3000", ["0x1000", "0x2000"]),
|
||||
],
|
||||
nodes: 9,
|
||||
},
|
||||
])(
|
||||
"names the observed $label budget on every page, including the last",
|
||||
async ({ budgets, reasons, unknowns, nodes }) => {
|
||||
const { port } = valueTraceDiamond();
|
||||
const request = { procedure: "0x1000", max_depth: 1, ...budgets };
|
||||
const unpaged = await traced(port, request);
|
||||
expect(unpaged).toMatchObject({
|
||||
total_nodes: nodes,
|
||||
next_offset: null,
|
||||
page_truncated: false,
|
||||
traversal: { status: "truncated", reasons },
|
||||
truncated: true,
|
||||
unknowns,
|
||||
});
|
||||
const last = await traced(port, {
|
||||
...request,
|
||||
offset: nodes - 1,
|
||||
limit: 3,
|
||||
});
|
||||
expect(last).toMatchObject({
|
||||
nodes: [{ id: unpaged.nodes.at(-1)?.id }],
|
||||
next_offset: null,
|
||||
page_truncated: true,
|
||||
traversal: { status: "truncated", reasons },
|
||||
truncated: true,
|
||||
unknowns,
|
||||
});
|
||||
},
|
||||
);
|
||||
|
||||
it("stops resolving call sites once the edge budget rejects a call", async () => {
|
||||
const { port, resolved, analyzed } = valueTraceDiamond();
|
||||
const trace = await traced(port, {
|
||||
procedure: "0x1000",
|
||||
max_depth: 1,
|
||||
max_edges: 1,
|
||||
});
|
||||
expect(trace).toMatchObject({
|
||||
traversal: { status: "truncated", reasons: ["max_edges"] },
|
||||
unknowns: [
|
||||
{ procedure: "0x1000", address: null, reason: "Edge budget reached" },
|
||||
],
|
||||
});
|
||||
expect(trace.edges.map(({ kind }) => kind)).toEqual(["def-use"]);
|
||||
expect(resolved).toEqual(["0x1004"]);
|
||||
expect(analyzed).toEqual(["0x1000"]);
|
||||
});
|
||||
});
|
||||
|
||||
describe("native value traversal limits and policies", () => {
|
||||
it("reports a depth limit only when no shallower path analyzes the destination", async () => {
|
||||
const shared = valueTraceDiamond();
|
||||
expect(
|
||||
await traced(shared.port, { procedure: "0x1000", max_depth: 1 }),
|
||||
).toMatchObject({ traversal: { status: "complete" }, unknowns: [] });
|
||||
// 0x2000 reaches 0x3000 at the depth limit before the seed's own call does.
|
||||
expect(shared.analyzed).toEqual(["0x1000", "0x2000", "0x3000"]);
|
||||
|
||||
expect(
|
||||
await traced(valueTraceDiamond().port, {
|
||||
procedure: "0x1000",
|
||||
max_depth: 0,
|
||||
}),
|
||||
).toMatchObject({
|
||||
traversal: { status: "truncated", reasons: ["max_depth"] },
|
||||
unknowns: [
|
||||
{
|
||||
procedure: "0x1000",
|
||||
address: "0x1004",
|
||||
reason: "Call depth budget reached",
|
||||
},
|
||||
{
|
||||
procedure: "0x1000",
|
||||
address: "0x1008",
|
||||
reason: "Call depth budget reached",
|
||||
},
|
||||
...omittedBindings("0x2000", ["0x1000"]),
|
||||
...omittedBindings("0x3000", ["0x1000"]),
|
||||
],
|
||||
});
|
||||
});
|
||||
|
||||
it("attributes a queued destination to the decompilation budget, not depth", async () => {
|
||||
expect(
|
||||
await traced(valueTraceDiamond().port, {
|
||||
procedure: "0x1000",
|
||||
max_depth: 1,
|
||||
max_functions: 2,
|
||||
}),
|
||||
).toMatchObject({
|
||||
traversal: { status: "truncated", reasons: ["max_functions"] },
|
||||
unknowns: [
|
||||
{
|
||||
procedure: "0x3000",
|
||||
address: null,
|
||||
reason: "decompilation budget reached",
|
||||
},
|
||||
...omittedBindings("0x3000", ["0x1000", "0x2000"]),
|
||||
],
|
||||
});
|
||||
});
|
||||
|
||||
it("names call-site, provider p-code and ambiguous-candidate limits", async () => {
|
||||
expect(
|
||||
await traced(valueTraceDiamond().port, {
|
||||
procedure: "0x1000",
|
||||
max_depth: 1,
|
||||
max_call_sites: 1,
|
||||
}),
|
||||
).toMatchObject({
|
||||
traversal: { status: "truncated", reasons: ["max_call_sites"] },
|
||||
unknowns: [
|
||||
{
|
||||
procedure: "0x1000",
|
||||
address: "0x1008",
|
||||
reason: "Call-site resolution budget reached",
|
||||
},
|
||||
{
|
||||
procedure: "0x2000",
|
||||
address: "0x2004",
|
||||
reason: "Call-site resolution budget reached",
|
||||
},
|
||||
],
|
||||
});
|
||||
|
||||
const limited = staticValueGraph(
|
||||
{
|
||||
"0x1000": {
|
||||
operations: [node("call", "0x1004", "CALL")],
|
||||
pcodeTruncated: true,
|
||||
},
|
||||
},
|
||||
{ "0x1004": [{ destination: "0x2000", status: "candidate" }] },
|
||||
);
|
||||
expect(await traced(limited.port, { procedure: "0x1000" })).toMatchObject({
|
||||
traversal: {
|
||||
status: "truncated",
|
||||
reasons: ["provider_pcode_truncated", "ambiguous_call_candidates"],
|
||||
},
|
||||
unknowns: [
|
||||
{
|
||||
procedure: "0x1000",
|
||||
address: null,
|
||||
reason: "Provider p-code was truncated",
|
||||
},
|
||||
{
|
||||
procedure: "0x1000",
|
||||
address: "0x1004",
|
||||
reason: "Ambiguous call candidates were not traversed",
|
||||
},
|
||||
],
|
||||
});
|
||||
expect(limited.analyzed).toEqual(["0x1000"]);
|
||||
});
|
||||
});
|
||||
@@ -12,13 +12,11 @@ import {
|
||||
import { err, ok } from "../../domain/result.js";
|
||||
import type { AnalysisOperationPort } from "../AnalysisProvider.js";
|
||||
import type { EnhancedResult } from "../EnhancedToolTypes.js";
|
||||
import { jsonParts } from "../../domain/jsonSerialization.js";
|
||||
import {
|
||||
renderSourceLimitations,
|
||||
type SourceLimitation,
|
||||
} from "../SourceLimitations.js";
|
||||
|
||||
const NODE_PAYLOAD_BYTES = 8 * 1024 * 1024;
|
||||
import { createValueTraceBudget } from "./NativeValueTraceBudget.js";
|
||||
|
||||
/** Compose bounded high-p-code def-use graphs across exactly referenced call destinations. */
|
||||
export const traceNativeValues = async (
|
||||
@@ -27,8 +25,6 @@ export const traceNativeValues = async (
|
||||
signal?: AbortSignal,
|
||||
): EnhancedResult => {
|
||||
const input = nativeValueTraceInputSchema.parse(parameters);
|
||||
const nodes: typeof nativeValueTraceSchema._output.nodes = [];
|
||||
const edges: typeof nativeValueTraceSchema._output.edges = [];
|
||||
const procedures: typeof nativeValueTraceSchema._output.procedures = [];
|
||||
const unknowns: typeof nativeValueTraceSchema._output.unknowns = [];
|
||||
const sourceLimitations: SourceLimitation[] = [];
|
||||
@@ -43,34 +39,19 @@ export const traceNativeValues = async (
|
||||
const pending = [{ procedure: input.procedure, depth: 0 }];
|
||||
const attempted = new Set<string>();
|
||||
const visited = new Set<string>();
|
||||
// Calls not followed because of depth or candidate policy. They narrow the
|
||||
// traversal only if no other path analyzes or queues the destination.
|
||||
const unexpanded: {
|
||||
caller: string;
|
||||
address: string;
|
||||
destination: string;
|
||||
reason: "max_depth" | "ambiguous_call_candidates";
|
||||
}[] = [];
|
||||
const budget = createValueTraceBudget(input, unknowns);
|
||||
const { nodes, edges } = budget;
|
||||
let targetSha256: string | undefined;
|
||||
let truncated = false;
|
||||
let decompilations = 0;
|
||||
let callSites = 0;
|
||||
let bytes = 0;
|
||||
let nodeBudgetReached = false;
|
||||
const retainNode = (value: (typeof nodes)[number]): boolean => {
|
||||
if (nodeBudgetReached) return false;
|
||||
if (nodes.length >= input.max_nodes) {
|
||||
truncated = true;
|
||||
nodeBudgetReached = true;
|
||||
return false;
|
||||
}
|
||||
let size = 0;
|
||||
// Reuse bounded JSON encoding to count escapes and UTF-8 expansion
|
||||
// before allocating a complete serialized node that might not fit.
|
||||
for (const part of jsonParts(value)) {
|
||||
size += Buffer.byteLength(part);
|
||||
if (size > NODE_PAYLOAD_BYTES - bytes) {
|
||||
truncated = true;
|
||||
nodeBudgetReached = true;
|
||||
return false;
|
||||
}
|
||||
}
|
||||
bytes += size;
|
||||
nodes.push(value);
|
||||
return true;
|
||||
};
|
||||
while (pending.length > 0) {
|
||||
if (signal?.aborted)
|
||||
return err(
|
||||
@@ -86,12 +67,14 @@ export const traceNativeValues = async (
|
||||
)
|
||||
continue;
|
||||
if (decompilations >= input.max_functions) {
|
||||
truncated = true;
|
||||
unknowns.push({
|
||||
budget.narrow("max_functions", {
|
||||
procedure: next.procedure,
|
||||
address: null,
|
||||
reason: "decompilation budget reached",
|
||||
});
|
||||
// Keep the unanalyzed selector queued so later accounting attributes
|
||||
// it to this budget rather than to a depth limit.
|
||||
pending.unshift(next);
|
||||
break;
|
||||
}
|
||||
// An exact selector is attempted once per trace, even on failure. Only
|
||||
@@ -163,14 +146,12 @@ export const traceNativeValues = async (
|
||||
});
|
||||
continue;
|
||||
}
|
||||
if (flow.truncated) {
|
||||
truncated = true;
|
||||
unknowns.push({
|
||||
if (flow.truncated)
|
||||
budget.narrow("provider_pcode_truncated", {
|
||||
procedure: address,
|
||||
address: null,
|
||||
reason: "Provider p-code was truncated",
|
||||
});
|
||||
}
|
||||
const ids = new Set<string>();
|
||||
// Keep generated parameter identity independent of which operations fit
|
||||
// the budget, while preserving every provider-supplied operation ID.
|
||||
@@ -191,7 +172,7 @@ export const traceNativeValues = async (
|
||||
kind: "operation" as const,
|
||||
parameter: null,
|
||||
};
|
||||
if (!retainNode(node)) break;
|
||||
if (!budget.retainNode(node)) break;
|
||||
ids.add(operation.id);
|
||||
}
|
||||
for (const parameter of flow.parameters) {
|
||||
@@ -201,7 +182,7 @@ export const traceNativeValues = async (
|
||||
localId = `${parameterId}#${suffix}`;
|
||||
const id = qualified(localId);
|
||||
if (
|
||||
!retainNode({
|
||||
!budget.retainNode({
|
||||
id,
|
||||
procedure: address,
|
||||
evidence_id: evidenceId,
|
||||
@@ -216,33 +197,37 @@ export const traceNativeValues = async (
|
||||
for (const use of flow.parameter_uses) {
|
||||
const source = parameterIds.get(use.ordinal);
|
||||
if (!ids.has(use.use) || source === undefined) continue;
|
||||
if (edges.length >= input.max_edges) {
|
||||
truncated = true;
|
||||
if (
|
||||
!budget.retainEdge(
|
||||
{
|
||||
source,
|
||||
target: qualified(use.use),
|
||||
input_index: use.input_index,
|
||||
kind: "parameter-use",
|
||||
status: "derived",
|
||||
evidence_id: evidenceId,
|
||||
},
|
||||
address,
|
||||
)
|
||||
)
|
||||
break;
|
||||
}
|
||||
edges.push({
|
||||
source,
|
||||
target: qualified(use.use),
|
||||
input_index: use.input_index,
|
||||
kind: "parameter-use",
|
||||
status: "derived",
|
||||
evidence_id: evidenceId,
|
||||
});
|
||||
}
|
||||
for (const edge of flow.def_use) {
|
||||
if (!ids.has(edge.definition) || !ids.has(edge.use)) continue;
|
||||
if (edges.length >= input.max_edges) {
|
||||
truncated = true;
|
||||
if (
|
||||
!budget.retainEdge(
|
||||
{
|
||||
source: qualified(edge.definition),
|
||||
target: qualified(edge.use),
|
||||
input_index: edge.input_index,
|
||||
kind: "def-use",
|
||||
status: "derived",
|
||||
evidence_id: evidenceId,
|
||||
},
|
||||
address,
|
||||
)
|
||||
)
|
||||
break;
|
||||
}
|
||||
edges.push({
|
||||
source: qualified(edge.definition),
|
||||
target: qualified(edge.use),
|
||||
input_index: edge.input_index,
|
||||
kind: "def-use",
|
||||
status: "derived",
|
||||
evidence_id: evidenceId,
|
||||
});
|
||||
}
|
||||
for (const operation of flow.operations) {
|
||||
if (
|
||||
@@ -250,9 +235,10 @@ export const traceNativeValues = async (
|
||||
!["CALL", "CALLIND"].includes(operation.opcode)
|
||||
)
|
||||
continue;
|
||||
// A full edge budget cannot retain this call or expand its destination.
|
||||
if (budget.edgesExhausted()) break;
|
||||
if (callSites >= input.max_call_sites) {
|
||||
truncated = true;
|
||||
unknowns.push({
|
||||
budget.narrow("max_call_sites", {
|
||||
procedure: address,
|
||||
address: operation.address,
|
||||
reason: "Call-site resolution budget reached",
|
||||
@@ -305,19 +291,21 @@ export const traceNativeValues = async (
|
||||
reason: `Call target ${targets.data.status}`,
|
||||
});
|
||||
for (const target of targets.data.targets) {
|
||||
if (edges.length >= input.max_edges) {
|
||||
truncated = true;
|
||||
break;
|
||||
}
|
||||
const destination = target.procedure ?? target.address;
|
||||
edges.push({
|
||||
source: qualified(operation.id),
|
||||
target: destination,
|
||||
input_index: null,
|
||||
kind: "call",
|
||||
status: target.status,
|
||||
evidence_id: evidenceId,
|
||||
});
|
||||
if (
|
||||
!budget.retainEdge(
|
||||
{
|
||||
source: qualified(operation.id),
|
||||
target: destination,
|
||||
input_index: null,
|
||||
kind: "call",
|
||||
status: target.status,
|
||||
evidence_id: evidenceId,
|
||||
},
|
||||
address,
|
||||
)
|
||||
)
|
||||
break;
|
||||
if (target.status !== "candidate")
|
||||
callBindings.push({
|
||||
caller: address,
|
||||
@@ -329,27 +317,20 @@ export const traceNativeValues = async (
|
||||
});
|
||||
if (next.depth < input.max_depth && target.status !== "candidate")
|
||||
pending.push({ procedure: destination, depth: next.depth + 1 });
|
||||
else if (!visited.has(destination)) {
|
||||
truncated = true;
|
||||
unknowns.push({
|
||||
procedure: address,
|
||||
else
|
||||
unexpanded.push({
|
||||
caller: address,
|
||||
address: operation.address,
|
||||
destination,
|
||||
reason:
|
||||
target.status === "candidate"
|
||||
? "Ambiguous call candidates were not traversed"
|
||||
: "Call depth budget reached",
|
||||
? "ambiguous_call_candidates"
|
||||
: "max_depth",
|
||||
});
|
||||
}
|
||||
}
|
||||
}
|
||||
if (
|
||||
truncated &&
|
||||
(nodeBudgetReached ||
|
||||
nodes.length >= input.max_nodes ||
|
||||
edges.length >= input.max_edges ||
|
||||
bytes >= NODE_PAYLOAD_BYTES)
|
||||
)
|
||||
break;
|
||||
// Remaining work is attributed to the rejecting budget recorded above.
|
||||
if (budget.exhausted()) break;
|
||||
}
|
||||
if (targetSha256 === undefined)
|
||||
return err(
|
||||
@@ -358,6 +339,26 @@ export const traceNativeValues = async (
|
||||
"No seed function could be analyzed",
|
||||
),
|
||||
);
|
||||
// A destination queued at a lower depth is omitted by the budget that ended
|
||||
// traversal, and one already attempted was not narrowed by this call.
|
||||
const queued = new Set(pending.map(({ procedure }) => procedure));
|
||||
for (const skipped of unexpanded) {
|
||||
const { destination } = skipped;
|
||||
if (
|
||||
visited.has(destination) ||
|
||||
attempted.has(destination) ||
|
||||
queued.has(destination)
|
||||
)
|
||||
continue;
|
||||
budget.narrow(skipped.reason, {
|
||||
procedure: skipped.caller,
|
||||
address: skipped.address,
|
||||
reason:
|
||||
skipped.reason === "ambiguous_call_candidates"
|
||||
? "Ambiguous call candidates were not traversed"
|
||||
: "Call depth budget reached",
|
||||
});
|
||||
}
|
||||
const parametersByProcedure = new Map<string, typeof nodes>();
|
||||
const returnsByProcedure = new Map<string, typeof nodes>();
|
||||
for (const node of nodes) {
|
||||
@@ -392,37 +393,45 @@ export const traceNativeValues = async (
|
||||
});
|
||||
continue;
|
||||
}
|
||||
if (edges.length >= input.max_edges) {
|
||||
truncated = true;
|
||||
if (
|
||||
!budget.retainEdge(
|
||||
{
|
||||
source: binding.call,
|
||||
target: node.id,
|
||||
input_index: ordinal + 1,
|
||||
kind: "argument-binding",
|
||||
status: "derived",
|
||||
evidence_id: binding.evidence_id,
|
||||
},
|
||||
binding.caller,
|
||||
)
|
||||
)
|
||||
break;
|
||||
}
|
||||
edges.push({
|
||||
source: binding.call,
|
||||
target: node.id,
|
||||
input_index: ordinal + 1,
|
||||
kind: "argument-binding",
|
||||
status: "derived",
|
||||
evidence_id: binding.evidence_id,
|
||||
});
|
||||
}
|
||||
if (binding.output)
|
||||
for (const node of returnsByProcedure.get(binding.callee) ?? []) {
|
||||
if (edges.length >= input.max_edges) {
|
||||
truncated = true;
|
||||
if (
|
||||
!budget.retainEdge(
|
||||
{
|
||||
source: node.id,
|
||||
target: binding.call,
|
||||
input_index: 1,
|
||||
kind: "return-binding",
|
||||
status: "derived",
|
||||
evidence_id: node.evidence_id,
|
||||
},
|
||||
binding.caller,
|
||||
)
|
||||
)
|
||||
break;
|
||||
}
|
||||
edges.push({
|
||||
source: node.id,
|
||||
target: binding.call,
|
||||
input_index: 1,
|
||||
kind: "return-binding",
|
||||
status: "derived",
|
||||
evidence_id: node.evidence_id,
|
||||
});
|
||||
}
|
||||
}
|
||||
const page = nodes.slice(input.offset, input.offset + input.limit);
|
||||
const pageIds = new Set(page.map(({ id }) => id));
|
||||
// Every edge source is a retained node, so a page holding every node holds
|
||||
// every edge; traversal truncation is independent of the requested page.
|
||||
const pageTruncated = page.length !== nodes.length;
|
||||
const traversal = budget.traversal();
|
||||
const result = nativeValueTraceSchema.parse({
|
||||
seed: input.procedure,
|
||||
target_sha256: targetSha256,
|
||||
@@ -437,11 +446,13 @@ export const traceNativeValues = async (
|
||||
input.offset + page.length < nodes.length
|
||||
? input.offset + page.length
|
||||
: null,
|
||||
truncated: truncated || page.length !== nodes.length,
|
||||
page_truncated: pageTruncated,
|
||||
traversal,
|
||||
truncated: pageTruncated || traversal.status === "truncated",
|
||||
limitations: [
|
||||
"Static decompiler def-use graphs cover recovered block members; when membership is unavailable they exclude operations marked dead. Logical CALL argument positions bind to recovered HighParam ordinals and RETURN values bind to recovered CALL outputs; these are decompiler-derived dependencies, not original source or runtime observations. Missing parameters, variadic arguments and alias dependencies remain unknown.",
|
||||
"LOAD and STORE are memory operations; object fields, persistent state and RNG semantics are not inferred from names or pseudocode.",
|
||||
"Traversal caps depth, decompilations, call-site resolutions, nodes, edges and node payloads (8388608 bytes, 8 MiB). This release has no runtime override for the 8388608-byte node-payload allowance. Complete serialized output is bounded to 33554432 bytes (32 MiB) and that output allowance has no runtime override. Pagination reruns analysis and returns edges whose source is on the node page; endpoints outside the page retain stable identities.",
|
||||
"Traversal caps depth, decompilations, call-site resolutions, nodes, edges and node payloads (8388608 bytes, 8 MiB); traversal.reasons names each cap or policy this trace reached. This release has no runtime override for the 8388608-byte node-payload allowance. Complete serialized output is bounded to 33554432 bytes (32 MiB) and that output allowance has no runtime override. Pagination reruns analysis and returns edges whose source is on the node page; endpoints outside the page retain stable identities.",
|
||||
...renderSourceLimitations(sourceLimitations),
|
||||
],
|
||||
});
|
||||
|
||||
@@ -0,0 +1,98 @@
|
||||
import type { z } from "zod";
|
||||
import { jsonParts } from "../../domain/jsonSerialization.js";
|
||||
import {
|
||||
nativeValueTraceTruncationReasonSchema,
|
||||
type nativeValueTraceSchema,
|
||||
type NativeValueTraceTruncationReason,
|
||||
} from "../../domain/native/nativeValueTrace.js";
|
||||
|
||||
type Trace = z.output<typeof nativeValueTraceSchema>;
|
||||
type TraceNode = Trace["nodes"][number];
|
||||
type TraceEdge = Trace["edges"][number];
|
||||
type TraceUnknown = Trace["unknowns"][number];
|
||||
|
||||
/** Retained node JSON allowance; this release has no runtime override. */
|
||||
const NODE_PAYLOAD_BYTES = 8 * 1024 * 1024;
|
||||
|
||||
const rowBudgetReasons = {
|
||||
max_nodes: "Node budget reached",
|
||||
node_payload_bytes: "Node payload budget reached",
|
||||
max_edges: "Edge budget reached",
|
||||
} as const;
|
||||
|
||||
/**
|
||||
* Own row admission and every observed cause of a narrowed traversal. Each
|
||||
* cause is recorded with a located unknown, and a row budget is recorded only
|
||||
* after it rejects a row, so an exactly filled budget is not truncation.
|
||||
*/
|
||||
export const createValueTraceBudget = (
|
||||
limits: { readonly max_nodes: number; readonly max_edges: number },
|
||||
unknowns: TraceUnknown[],
|
||||
) => {
|
||||
const nodes: TraceNode[] = [];
|
||||
const edges: TraceEdge[] = [];
|
||||
const causes = new Set<NativeValueTraceTruncationReason>();
|
||||
let bytes = 0;
|
||||
let nodesExhausted = false;
|
||||
let edgesExhausted = false;
|
||||
const narrow = (
|
||||
reason: NativeValueTraceTruncationReason,
|
||||
unknown: TraceUnknown,
|
||||
): false => {
|
||||
causes.add(reason);
|
||||
unknowns.push(unknown);
|
||||
return false;
|
||||
};
|
||||
const reject = (procedure: string, reason: keyof typeof rowBudgetReasons) =>
|
||||
narrow(reason, {
|
||||
procedure,
|
||||
address: null,
|
||||
reason: rowBudgetReasons[reason],
|
||||
});
|
||||
return {
|
||||
nodes,
|
||||
edges,
|
||||
narrow,
|
||||
retainNode(value: TraceNode): boolean {
|
||||
if (nodesExhausted) return false;
|
||||
if (nodes.length >= limits.max_nodes) {
|
||||
nodesExhausted = true;
|
||||
return reject(value.procedure, "max_nodes");
|
||||
}
|
||||
let size = 0;
|
||||
// Reuse bounded JSON encoding to count escapes and UTF-8 expansion
|
||||
// before allocating a complete serialized node that might not fit.
|
||||
for (const part of jsonParts(value)) {
|
||||
size += Buffer.byteLength(part);
|
||||
if (size > NODE_PAYLOAD_BYTES - bytes) {
|
||||
nodesExhausted = true;
|
||||
return reject(value.procedure, "node_payload_bytes");
|
||||
}
|
||||
}
|
||||
bytes += size;
|
||||
nodes.push(value);
|
||||
return true;
|
||||
},
|
||||
retainEdge(value: TraceEdge, procedure: string): boolean {
|
||||
if (edgesExhausted) return false;
|
||||
if (edges.length >= limits.max_edges) {
|
||||
edgesExhausted = true;
|
||||
return reject(procedure, "max_edges");
|
||||
}
|
||||
edges.push(value);
|
||||
return true;
|
||||
},
|
||||
/** A rejected edge means no later call edge or expansion can be retained. */
|
||||
edgesExhausted: () => edgesExhausted,
|
||||
/** Traversal stops after the procedure in which a row budget rejected work. */
|
||||
exhausted: () => nodesExhausted || edgesExhausted,
|
||||
traversal: (): Trace["traversal"] => {
|
||||
const reasons = nativeValueTraceTruncationReasonSchema.options.filter(
|
||||
(reason) => causes.has(reason),
|
||||
);
|
||||
return reasons.length === 0
|
||||
? { status: "complete" }
|
||||
: { status: "truncated", reasons };
|
||||
},
|
||||
};
|
||||
};
|
||||
@@ -9,21 +9,24 @@ import { projectAnalysisError } from "../../domain/analysisErrorProjection.js";
|
||||
import type { AnalysisErrorProjection } from "../../domain/analysisErrorProjection.js";
|
||||
import { ProviderAdapterError } from "../../domain/providerAdapterError.js";
|
||||
import { describeValidationFailure } from "../../domain/validationFailure.js";
|
||||
import { PROCESS_PROVIDER } from "../../domain/process/processEvidenceProvider.js";
|
||||
import {
|
||||
analysisInputErrorFromIssues,
|
||||
projectInputIssues,
|
||||
} from "../../domain/inputIssueProjection.js";
|
||||
import { processTraceSpecificationSchema } from "../../domain/process/processTraceSpecification.js";
|
||||
import { processScenarioSchema } from "../../domain/process/processScenario.js";
|
||||
import { compareProcessCaptureEvidence } from "../../domain/process/processComparisonEvidence.js";
|
||||
import {
|
||||
compareParsedProcessEvidence,
|
||||
readProcessComparisonSource,
|
||||
} from "./ProcessComparisonService.js";
|
||||
import { createProgressReporter } from "../ProgressReporter.js";
|
||||
import { captureProcessScenario } from "../../process/capture/ProcessHarness.js";
|
||||
import { ProcessCaptureResourceScope } from "../../process/capture/ProcessCaptureLifecycle.js";
|
||||
import { ProcessCaptureError } from "../../process/capture/ProcessCaptureError.js";
|
||||
import { PROCESS_PROVIDER } from "../../domain/process/processEvidenceProvider.js";
|
||||
import {
|
||||
createProcessCaptureEvidence,
|
||||
parseProcessCaptureEvidence,
|
||||
type ProcessCaptureEvidenceFailure,
|
||||
} from "./ProcessEvidence.js";
|
||||
import { readJsonFile } from "../JsonFiles.js";
|
||||
import { ArtifactReaderFailure } from "../../artifacts/ArtifactReader.js";
|
||||
@@ -99,14 +102,30 @@ export const compareProcessEvidenceFiles = async (
|
||||
signal?: AbortSignal,
|
||||
) =>
|
||||
withProcessInputResources("compare_process_captures", async (resources) => {
|
||||
const left = parseCaptureEvidence(
|
||||
await readJson(leftPath, "compare_process_captures", signal, resources),
|
||||
const leftInput = await readJson(
|
||||
leftPath,
|
||||
"compare_process_captures",
|
||||
signal,
|
||||
resources,
|
||||
);
|
||||
const left = readProcessComparisonSource(
|
||||
"left",
|
||||
leftInput,
|
||||
projectCaptureFileFailure,
|
||||
);
|
||||
const right = parseCaptureEvidence(
|
||||
await readJson(rightPath, "compare_process_captures", signal, resources),
|
||||
if (!left.ok) throw left.error;
|
||||
const rightInput = await readJson(
|
||||
rightPath,
|
||||
"compare_process_captures",
|
||||
signal,
|
||||
resources,
|
||||
);
|
||||
const right = readProcessComparisonSource(
|
||||
"right",
|
||||
rightInput,
|
||||
projectCaptureFileFailure,
|
||||
);
|
||||
if (!right.ok) throw right.error;
|
||||
const traceSpecification =
|
||||
traceSpecPath === undefined
|
||||
? undefined
|
||||
@@ -118,34 +137,38 @@ export const compareProcessEvidenceFiles = async (
|
||||
resources,
|
||||
),
|
||||
);
|
||||
return compareProcessCaptureEvidence(
|
||||
left,
|
||||
right,
|
||||
const result = compareParsedProcessEvidence(
|
||||
left.value,
|
||||
right.value,
|
||||
traceSpecification === undefined ? {} : { traceSpecification },
|
||||
).evidence;
|
||||
);
|
||||
if (!result.ok) throw result.error;
|
||||
return result.value.evidence;
|
||||
});
|
||||
|
||||
const parseCaptureEvidence = (input: unknown, side: "left" | "right") => {
|
||||
const parsed = parseProcessCaptureEvidence(input);
|
||||
if (parsed.ok) return parsed.value;
|
||||
const failure = parsed.error;
|
||||
/** Preserve CLI file diagnostics while sharing all source interpretation. */
|
||||
const projectCaptureFileFailure = (
|
||||
failure: ProcessCaptureEvidenceFailure,
|
||||
side: "left" | "right",
|
||||
input: unknown,
|
||||
): AnalysisError => {
|
||||
switch (failure.kind) {
|
||||
case "invalid-evidence":
|
||||
throw analysisInputErrorFromIssues(
|
||||
return analysisInputErrorFromIssues(
|
||||
"compare_process_captures",
|
||||
failure.cause.issues,
|
||||
input,
|
||||
{ cause: failure.cause },
|
||||
);
|
||||
case "wrong-workflow":
|
||||
throw processInputFailure(
|
||||
return processInputFailure(
|
||||
"compare_process_captures",
|
||||
"Capture evidence is not from the current process-capture workflow. Create new capture evidence, then try again.",
|
||||
[side],
|
||||
"invalid_value",
|
||||
);
|
||||
case "unsupported-provider":
|
||||
throw new AnalysisInputError("compare_process_captures", undefined, [
|
||||
return new AnalysisInputError("compare_process_captures", undefined, [
|
||||
{
|
||||
path: [side, "provider"],
|
||||
reason: "invalid_value",
|
||||
@@ -153,7 +176,7 @@ const parseCaptureEvidence = (input: unknown, side: "left" | "right") => {
|
||||
},
|
||||
]);
|
||||
case "invalid-capture":
|
||||
throw invalidCaptureEvidence(failure.cause, side);
|
||||
return invalidCaptureEvidence(failure.cause, side);
|
||||
}
|
||||
};
|
||||
|
||||
|
||||
@@ -0,0 +1,188 @@
|
||||
import { processComparisonInputSchema } from "../../contracts/processComparisonContracts.js";
|
||||
import { AnalysisError } from "../../domain/analysisErrorBase.js";
|
||||
import { AnalysisInputError } from "../../domain/analysisErrorCore.js";
|
||||
import { describeValidationFailure } from "../../domain/validationFailure.js";
|
||||
import { EvidenceIntegrityError } from "../../domain/evidenceErrors.js";
|
||||
import {
|
||||
PROCESS_COMPARISON_DIMENSIONS,
|
||||
type ProcessCaptureComparison,
|
||||
} from "../../domain/process/processComparison.js";
|
||||
import {
|
||||
compareProcessCaptureEvidence,
|
||||
type ParsedProcessCaptureEvidence,
|
||||
} from "../../domain/process/processComparisonEvidence.js";
|
||||
import { PROCESS_PROVIDER } from "../../domain/process/processEvidenceProvider.js";
|
||||
import type { RecordUnknownInput } from "../../domain/residualUnknown.js";
|
||||
import { err, ok, type Result } from "../../domain/result.js";
|
||||
import {
|
||||
parseProcessCaptureEvidence,
|
||||
type ProcessCaptureEvidenceFailure,
|
||||
} from "./ProcessEvidence.js";
|
||||
import {
|
||||
resolveEvidenceRequest,
|
||||
type EvidenceLookup,
|
||||
} from "../EvidenceInputResolver.js";
|
||||
import type { EvidenceWorkflowExecution } from "../investigation/EvidenceWorkflowExecution.js";
|
||||
|
||||
interface ProcessComparisonExecution extends EvidenceWorkflowExecution {
|
||||
readonly unknown: RecordUnknownInput | undefined;
|
||||
}
|
||||
|
||||
/** Shared exact-source selection and process-capture interpretation for CLI/MCP. */
|
||||
export const compareProcessEvidence = (
|
||||
input: unknown,
|
||||
lookup?: EvidenceLookup,
|
||||
options: {
|
||||
readonly signal?: AbortSignal;
|
||||
readonly now?: () => number;
|
||||
} = {},
|
||||
): Result<ProcessComparisonExecution, AnalysisError> => {
|
||||
const selected = resolveEvidenceRequest(
|
||||
"compare_process_captures",
|
||||
input,
|
||||
processComparisonInputSchema,
|
||||
lookup,
|
||||
options,
|
||||
);
|
||||
if (!selected.ok) return selected;
|
||||
const request = selected.value.input;
|
||||
const left = readProcessComparisonSource("left", request.left);
|
||||
if (!left.ok) return left;
|
||||
const right = readProcessComparisonSource("right", request.right);
|
||||
if (!right.ok) return right;
|
||||
return compareParsedProcessEvidence(left.value, right.value, {
|
||||
...(request.max_capture_age_ms === undefined
|
||||
? {}
|
||||
: { maxCaptureAgeMs: request.max_capture_age_ms }),
|
||||
...(request.trace_spec === undefined
|
||||
? {}
|
||||
: { traceSpecification: request.trace_spec }),
|
||||
...(options.now === undefined ? {} : { now: options.now }),
|
||||
});
|
||||
};
|
||||
|
||||
/** Compare authenticated facts without reinterpreting or copying source carriers. */
|
||||
export const compareParsedProcessEvidence = (
|
||||
left: ParsedProcessCaptureEvidence,
|
||||
right: ParsedProcessCaptureEvidence,
|
||||
options: Parameters<typeof compareProcessCaptureEvidence>[2] = {},
|
||||
): Result<ProcessComparisonExecution, AnalysisError> => {
|
||||
try {
|
||||
const { comparison, evidence } = compareProcessCaptureEvidence(
|
||||
left,
|
||||
right,
|
||||
options,
|
||||
);
|
||||
return ok({
|
||||
evidence,
|
||||
sources: [left.record, right.record],
|
||||
unknown: comparisonUnknownInput(
|
||||
{
|
||||
left_evidence_id: left.record.evidence_id,
|
||||
right_evidence_id: right.record.evidence_id,
|
||||
comparison_evidence_id: evidence.evidence_id,
|
||||
},
|
||||
comparison,
|
||||
),
|
||||
});
|
||||
} catch (cause: unknown) {
|
||||
if (cause instanceof AnalysisError) return err(cause);
|
||||
throw cause;
|
||||
}
|
||||
};
|
||||
|
||||
/** Authenticate each selected capture before a caller admits its next source. */
|
||||
export const readProcessComparisonSource = (
|
||||
side: "left" | "right",
|
||||
value: unknown,
|
||||
projectFailure?: (
|
||||
failure: ProcessCaptureEvidenceFailure,
|
||||
side: "left" | "right",
|
||||
input: unknown,
|
||||
) => AnalysisError,
|
||||
): Result<ParsedProcessCaptureEvidence, AnalysisError> => {
|
||||
const operation = "compare_process_captures";
|
||||
const parsed = parseProcessCaptureEvidence(value);
|
||||
if (parsed.ok) return parsed;
|
||||
const failure = parsed.error;
|
||||
if (projectFailure !== undefined)
|
||||
return err(projectFailure(failure, side, value));
|
||||
switch (failure.kind) {
|
||||
case "invalid-evidence":
|
||||
return err(
|
||||
new EvidenceIntegrityError(`${side} Evidence failed validation`, {
|
||||
cause: failure.cause,
|
||||
userMessage: `The ${side} Evidence failed validation (${describeValidationFailure(failure.cause)}). Supply unmodified capture Evidence from capture_process_scenario.`,
|
||||
}),
|
||||
);
|
||||
case "wrong-workflow":
|
||||
return err(
|
||||
new AnalysisInputError(operation, undefined, [
|
||||
{
|
||||
path: [side],
|
||||
reason: "invalid_value",
|
||||
message: `Expected process capture Evidence (operation ${failure.expected.operation}, predicate ${failure.expected.predicate}); the ${side} Evidence has operation ${failure.actual.operation} and predicate ${failure.actual.predicate}.`,
|
||||
},
|
||||
]),
|
||||
);
|
||||
case "unsupported-provider":
|
||||
return err(
|
||||
new AnalysisInputError(operation, undefined, [
|
||||
{
|
||||
path: [side, "provider"],
|
||||
reason: "invalid_value",
|
||||
message: `The ${side} Evidence provider ${failure.provider.id} v${failure.provider.version} (${failure.provider.name}) is not the current or supported legacy ${PROCESS_PROVIDER.id} v${PROCESS_PROVIDER.version} process capture provider.`,
|
||||
},
|
||||
]),
|
||||
);
|
||||
case "invalid-capture":
|
||||
return err(
|
||||
new EvidenceIntegrityError(
|
||||
`${side} process capture failed validation`,
|
||||
{
|
||||
cause: failure.cause,
|
||||
userMessage: `The ${side} Evidence has an invalid process capture result (${describeValidationFailure(failure.cause)}). Recreate the capture with capture_process_scenario.`,
|
||||
},
|
||||
),
|
||||
);
|
||||
}
|
||||
};
|
||||
|
||||
const comparisonUnknownInput = (
|
||||
parsed: {
|
||||
readonly left_evidence_id: string;
|
||||
readonly right_evidence_id: string;
|
||||
readonly comparison_evidence_id: string;
|
||||
},
|
||||
comparison: ProcessCaptureComparison,
|
||||
): RecordUnknownInput | undefined => {
|
||||
if (comparison.status === "unchanged") return undefined;
|
||||
const differingScopes = PROCESS_COMPARISON_DIMENSIONS.filter(
|
||||
(dimension) => comparison[dimension] !== "unchanged",
|
||||
).join(", ");
|
||||
const contradictory =
|
||||
comparison.status === "changed" &&
|
||||
parsed.left_evidence_id !== parsed.right_evidence_id;
|
||||
// Unknown identity uses the question, not its supporting records. Bind the
|
||||
// question to this comparison so distinct capture pairs and policies coexist.
|
||||
return {
|
||||
question: `${contradictory ? "Process captures disagree" : `Process comparison is ${comparison.status}`} across: ${differingScopes} (comparison ${parsed.comparison_evidence_id})`,
|
||||
severity: "high",
|
||||
domain: "process-comparison",
|
||||
supporting_evidence_ids: contradictory
|
||||
? [parsed.left_evidence_id]
|
||||
: [...new Set([parsed.left_evidence_id, parsed.right_evidence_id])],
|
||||
contradicting_evidence_ids: contradictory ? [parsed.right_evidence_id] : [],
|
||||
required_authority: "controlled-replay",
|
||||
required_confidence: "observed",
|
||||
required_environment: null,
|
||||
recommended_probes: [
|
||||
{
|
||||
operation: "capture_process_scenario",
|
||||
rationale:
|
||||
"Repeat both scenarios under the same controlled environment.",
|
||||
},
|
||||
],
|
||||
relationships: [],
|
||||
};
|
||||
};
|
||||
@@ -4,6 +4,8 @@ import { createEvidence, parseEvidence } from "../../domain/evidence.js";
|
||||
import { projectAnalysisError } from "../../domain/analysisErrorProjection.js";
|
||||
import { wasmInterfaceComparisonSchema } from "../../contracts/wasm/wasmComparisonContracts.js";
|
||||
import { compareWasmInterfaces } from "./WasmInterfaceComparisonService.js";
|
||||
import { retainAnalysisResultSources } from "../investigation/recordEvidenceSources.js";
|
||||
import { ok } from "../../domain/result.js";
|
||||
import {
|
||||
WASM_INTERFACE_REPRESENTATION_BYTES,
|
||||
wasmInterfaceEntryRepresentationBytes,
|
||||
@@ -57,7 +59,9 @@ const compare = async (
|
||||
) => {
|
||||
const result = await compareWasmInterfaces({ left, right, scope: "both" });
|
||||
if (!result.ok) throw result.error;
|
||||
return wasmInterfaceComparisonSchema.parse(result.value.normalized_result);
|
||||
return wasmInterfaceComparisonSchema.parse(
|
||||
result.value.evidence.normalized_result,
|
||||
);
|
||||
};
|
||||
|
||||
it("reports all five comparison states, including duplicate ambiguity and explicit selected scope", async () => {
|
||||
@@ -93,7 +97,7 @@ it("reports all five comparison states, including duplicate ambiguity and explic
|
||||
scope: "imports",
|
||||
});
|
||||
if (!selected.ok) throw selected.error;
|
||||
expect(selected.value.normalized_result).toMatchObject({
|
||||
expect(selected.value.evidence.normalized_result).toMatchObject({
|
||||
changes: [],
|
||||
coverage: "complete-within-inputs",
|
||||
});
|
||||
@@ -180,7 +184,7 @@ it("resolves exact retained references and permits cancellation between comparis
|
||||
id === left.evidence_id ? left : right,
|
||||
);
|
||||
if (!result.ok) throw result.error;
|
||||
expect(result.value.evidence_links).toEqual([
|
||||
expect(result.value.evidence.evidence_links).toEqual([
|
||||
left.evidence_id,
|
||||
right.evidence_id,
|
||||
]);
|
||||
@@ -315,6 +319,18 @@ it("retains the complete comparison when cancelled at the final handoff", async
|
||||
if (result.ok) throw new Error("Expected cancellation during final handoff");
|
||||
expect(result.error).toMatchObject({ _tag: "AnalysisCancelledError" });
|
||||
expect(result.error.cause).toBe(reason);
|
||||
expect(result.sourceEvidence).toEqual([left, right]);
|
||||
const retainedSources: string[] = [];
|
||||
const retained = retainAnalysisResultSources(
|
||||
"compare_wasm_interfaces",
|
||||
(source) => {
|
||||
retainedSources.push(source.evidence_id);
|
||||
return ok("added");
|
||||
},
|
||||
result,
|
||||
);
|
||||
expect(retained.ok).toBe(false);
|
||||
expect(retainedSources).toEqual([left.evidence_id, right.evidence_id]);
|
||||
const partial = parseEvidence(result.error.partialObservation);
|
||||
expect(partial).toMatchObject({
|
||||
operation: "compare_wasm_interfaces",
|
||||
|
||||
@@ -22,16 +22,20 @@ import {
|
||||
} from "../../domain/analysisErrorCore.js";
|
||||
import { fixedAllowanceOptions } from "../../domain/fixedAllowanceRemediation.js";
|
||||
import { AnalysisError } from "../../domain/analysisErrorBase.js";
|
||||
import { analysisInputErrorFromIssues } from "../../domain/inputIssueProjection.js";
|
||||
import { err, ok, type Result } from "../../domain/result.js";
|
||||
import {
|
||||
resolvePairedEvidenceRequest,
|
||||
resolveEvidenceRequest,
|
||||
type EvidenceLookup,
|
||||
} from "../EvidenceInputResolver.js";
|
||||
import type { ExecutionOptions } from "../AnalysisProvider.js";
|
||||
import type { z } from "zod";
|
||||
import { safeParseEvidence } from "../../domain/evidenceValidation.js";
|
||||
import { completeProviderEvidence } from "../ProviderOperationExecution.js";
|
||||
import type { EvidenceWorkflowExecution } from "../investigation/EvidenceWorkflowExecution.js";
|
||||
import {
|
||||
withEvidenceSources,
|
||||
type AnalysisResultWithSources,
|
||||
} from "../investigation/recordEvidenceSources.js";
|
||||
|
||||
const OPERATION = "compare_wasm_interfaces";
|
||||
const parseInspection = (
|
||||
@@ -81,17 +85,18 @@ export const compareWasmInterfaces = async (
|
||||
rawInput: unknown,
|
||||
options?: ExecutionOptions,
|
||||
lookup?: EvidenceLookup,
|
||||
): Promise<Result<Evidence, AnalysisError>> => {
|
||||
): Promise<AnalysisResultWithSources<EvidenceWorkflowExecution>> => {
|
||||
if (options?.signal?.aborted)
|
||||
return err(
|
||||
new AnalysisCancelledError(OPERATION, { cause: options.signal.reason }),
|
||||
);
|
||||
const parsed = compareWasmInterfacesInputSchema.safeParse(rawInput);
|
||||
if (!parsed.success)
|
||||
return err(
|
||||
analysisInputErrorFromIssues(OPERATION, parsed.error.issues, rawInput),
|
||||
);
|
||||
const resolved = resolvePairedEvidenceRequest(parsed.data, lookup);
|
||||
const resolved = resolveEvidenceRequest(
|
||||
OPERATION,
|
||||
rawInput,
|
||||
compareWasmInterfacesInputSchema,
|
||||
lookup,
|
||||
options,
|
||||
);
|
||||
if (!resolved.ok) return resolved;
|
||||
const assertActive = () => {
|
||||
if (options?.signal?.aborted)
|
||||
@@ -102,13 +107,13 @@ export const compareWasmInterfaces = async (
|
||||
try {
|
||||
await setImmediate();
|
||||
assertActive();
|
||||
const parsedLeft = parseInspection(resolved.value.left);
|
||||
const parsedLeft = parseInspection(resolved.value.input.left);
|
||||
if (!parsedLeft.ok) return parsedLeft;
|
||||
const parsedRight = parseInspection(resolved.value.right);
|
||||
const parsedRight = parseInspection(resolved.value.input.right);
|
||||
if (!parsedRight.ok) return parsedRight;
|
||||
const left = parsedLeft.value;
|
||||
const right = parsedRight.value;
|
||||
const scope = parsed.data.scope;
|
||||
const scope = resolved.value.input.scope;
|
||||
const selected = (entry: WasmInterfaceEntry): boolean =>
|
||||
scope === "both" ||
|
||||
entry.direction === (scope === "imports" ? "import" : "export");
|
||||
@@ -270,7 +275,12 @@ export const compareWasmInterfaces = async (
|
||||
evidenceLinks: [left.evidence.evidence_id, right.evidence.evidence_id],
|
||||
});
|
||||
await setImmediate();
|
||||
return completeProviderEvidence(evidence, options?.signal);
|
||||
const completed = completeProviderEvidence(evidence, options?.signal);
|
||||
const sources = [left.evidence, right.evidence];
|
||||
return withEvidenceSources(
|
||||
completed.ok ? ok({ evidence: completed.value, sources }) : completed,
|
||||
sources,
|
||||
);
|
||||
} catch (cause: unknown) {
|
||||
if (cause instanceof AnalysisError) return err(cause);
|
||||
throw cause;
|
||||
|
||||
@@ -19,13 +19,18 @@ import {
|
||||
observeMzWindowsHeader,
|
||||
type MzWindowsHeaderSignatureEvidence,
|
||||
} from "../domain/dosMz.js";
|
||||
import { DFUSE_SUFFIX_BYTES } from "../domain/firmware/dfuse.js";
|
||||
|
||||
/** Initial bytes required by the other bounded format classifiers. */
|
||||
export const ARTIFACT_CLASSIFICATION_PREFIX_BYTES = 8_192;
|
||||
/** Final bytes kept for formats identified by a trailer: the DFU suffix. */
|
||||
const ARTIFACT_CLASSIFICATION_SUFFIX_BYTES = DFUSE_SUFFIX_BYTES;
|
||||
|
||||
/** Retained prefix plus bounded format-specific observations from the stream. */
|
||||
export interface ArtifactClassificationEvidence {
|
||||
readonly prefix: Buffer;
|
||||
/** The last bytes of the stream, when the producer observed its end. */
|
||||
readonly suffix?: Buffer;
|
||||
readonly asarHeader: FinalAsarHeaderProbeObservation;
|
||||
readonly fatSliceHeaders?: FatSliceHeaderEvidence;
|
||||
readonly mzWindowsHeaderSignature?: MzWindowsHeaderSignatureEvidence;
|
||||
@@ -76,6 +81,7 @@ const observeFatSliceHeaders = (input: {
|
||||
|
||||
const finishClassificationEvidence = (
|
||||
prefix: Buffer,
|
||||
suffix: Buffer | undefined,
|
||||
archiveBytes: number,
|
||||
finishAsar: (archiveBytes: number) => FinalAsarHeaderProbeObservation,
|
||||
sliceHeadersByOffset: Map<number, MachoHeaderPrefixFacts>,
|
||||
@@ -106,6 +112,7 @@ const finishClassificationEvidence = (
|
||||
: undefined;
|
||||
return {
|
||||
prefix,
|
||||
...(suffix === undefined ? {} : { suffix }),
|
||||
asarHeader: finishAsar(archiveBytes),
|
||||
...(fatSliceHeaders === undefined ? {} : { fatSliceHeaders }),
|
||||
...(mzWindowsHeaderSignature === undefined
|
||||
@@ -132,6 +139,18 @@ export const createArtifactClassificationProbe = (
|
||||
let mzWindowsHeaderSignature: MzWindowsHeaderSignatureEvidence | undefined;
|
||||
let mzWindowsHeaderOffset: number | undefined;
|
||||
let mzHeaderChecked = false;
|
||||
let suffix = Buffer.alloc(0);
|
||||
const observeSuffix = (bytes: Uint8Array): void => {
|
||||
const joined =
|
||||
bytes.length >= ARTIFACT_CLASSIFICATION_SUFFIX_BYTES
|
||||
? bytes
|
||||
: Buffer.concat([suffix, bytes]);
|
||||
suffix = Buffer.from(
|
||||
joined.subarray(
|
||||
Math.max(0, joined.length - ARTIFACT_CLASSIFICATION_SUFFIX_BYTES),
|
||||
),
|
||||
);
|
||||
};
|
||||
const observeSignatureWindow = (
|
||||
offset: number,
|
||||
sourceOffset: number,
|
||||
@@ -258,11 +277,14 @@ export const createArtifactClassificationProbe = (
|
||||
}
|
||||
if (mzWindowsHeaderOffset !== undefined)
|
||||
observeSignatureWindow(mzWindowsHeaderOffset, streamOffset, bytes);
|
||||
observeSuffix(bytes);
|
||||
observedBytes += bytes.length;
|
||||
},
|
||||
finish(archiveBytes: number): ArtifactClassificationEvidence {
|
||||
return finishClassificationEvidence(
|
||||
retainedPrefix(),
|
||||
// Bytes the probe did not observe could still follow its last write.
|
||||
observedBytes === archiveBytes ? suffix : undefined,
|
||||
archiveBytes,
|
||||
(bytes) => asar.finish(bytes),
|
||||
fatSliceHeadersByOffset,
|
||||
|
||||
@@ -26,18 +26,20 @@ export const abortIfNeeded = (signal?: AbortSignal): void => {
|
||||
);
|
||||
};
|
||||
|
||||
/** Hash every stream byte and retain its bounded classification prefix. */
|
||||
/** Hash every stream byte, optionally feeding the same bytes to a scan aggregate. */
|
||||
export const hashReadable = async (
|
||||
stream: Readable,
|
||||
signal?: AbortSignal,
|
||||
integrityAlgorithm?: "sha256" | "md5",
|
||||
asarHeaderAdmission?: AsarHeaderAdmission,
|
||||
observeChunk?: (chunk: Buffer) => void,
|
||||
): Promise<HashResult> => {
|
||||
const completed = await hashReadableToCompletion(
|
||||
stream,
|
||||
signal,
|
||||
integrityAlgorithm,
|
||||
asarHeaderAdmission,
|
||||
observeChunk,
|
||||
);
|
||||
abortIfNeeded(signal);
|
||||
return completed;
|
||||
@@ -51,6 +53,7 @@ export const hashReadableToCompletion = async (
|
||||
signal?: AbortSignal,
|
||||
integrityAlgorithm?: "sha256" | "md5",
|
||||
asarHeaderAdmission?: AsarHeaderAdmission,
|
||||
observeChunk?: (chunk: Buffer) => void,
|
||||
): Promise<HashResult> => {
|
||||
const hash = createHash("sha256");
|
||||
const md5 = integrityAlgorithm === "md5" ? createHash("md5") : undefined;
|
||||
@@ -61,6 +64,7 @@ export const hashReadableToCompletion = async (
|
||||
const chunk = streamChunkToBuffer(raw);
|
||||
bytes += chunk.length;
|
||||
hash.update(chunk);
|
||||
observeChunk?.(chunk);
|
||||
md5?.update(chunk);
|
||||
probe.write(chunk);
|
||||
}
|
||||
|
||||
Loaded 100 of 408 files, more files were not shown because too many files have changed in this diff.
Show more
Reference in new issue
Block a user