diff --git a/docs/ghidra-nativeaot.md b/docs/ghidra-nativeaot.md index 90dbf99b4..d9505a226 100644 --- a/docs/ghidra-nativeaot.md +++ b/docs/ghidra-nativeaot.md @@ -1,16 +1,25 @@ # Optional NativeAOT metadata recovery -REA integrates [Washi1337/ghidra-nativeaot](https://github.com/Washi1337/ghidra-nativeaot) -through an optional headless adapter. The original source is an opt-in submodule, -pinned to `effeb734fc570c32650f88b159608979dc7b423e`; its MIT license and existing -source notices are retained. No upstream source or algorithms are rewritten. +REA has a built-in read-only snapshot parser for x64 PE targets with .NET 8 RTR +9.1 metadata. It parses the exact admitted Ghidra session snapshot and uses an +in-memory overlay for rehydrated metadata; it does not define Ghidra DataTypes, +labels, functions, or memory bytes. The hydration semantics are adapted from +[Washi1337/ghidra-nativeaot](https://github.com/Washi1337/ghidra-nativeaot) at +`effeb734fc570c32650f88b159608979dc7b423e` (MIT), cross-checked against +.NET's `MethodTable.h` at runtime commit +`a3b2d40328be0be3f8dd43f0e5cc925b8827b044` (MIT). The required notice ships in +[`licenses/nativeaot-parser-MIT.txt`](../licenses/nativeaot-parser-MIT.txt). -The initial tested layout is .NET runtime **8.0.22**, NativeAOT RTR **9.1**, -x86-64 ELF and native Windows PE **targets analyzed on Linux x64**, using -Ghidra **12.1.4** and JDK **21**. The build accepts any Ghidra 12.1.x release and -javac 21 or newer, and it compiles with `--release 21`. Real verification of -this layout remains the Ghidra 12.1.4 and JDK 21 pair. Other NativeAOT layouts, -architectures, and hosts are unsupported. +The optional headless Ghidra adapter remains available for database annotations +and broader recovery. It uses the same pinned upstream commit as an opt-in +submodule and is a separate, more capable workflow. + +The parser currently handles x64 PE32+ and RTR 9.1, including bounded +DEHYDRATED_DATA processing, MethodTable relationships, virtual slots, and +frozen strings. Other RTR versions and PE architectures are reported as +unsupported or partial. The matching .NET 8.0.22 fixture is source-built and +the parser's Windows-host Ghidra workflow has not yet had a real Windows-host +run. This does not establish Windows Ghidra import or decompiler behavior. Windows Ghidra P0 has no database mutation authority. A PE/CLI or ReadyToRun assembly should use `inspect_managed_artifact`; it is not NativeAOT. @@ -43,42 +52,62 @@ arbitrary caller-supplied bytes. The bounded loader accepts regular JARs up to ## Use existing tools -Open the executable with provider `ghidra`. `inspect_native_load_image` returns -optional `observations.metadata_recovery` inline, including the detected header, -layout, recovery status, type category paths/addresses, diagnostics, annotation -coverage and derived-memory SHA. The enclosing `unsupported` load-image status -for PE/ELF concerns independent DOS loader verification; metadata recovery has -its own status. It is not a failed import. +Open the executable with provider `ghidra`. For an x64 PE, `inspect_native_load_image` +returns `observations.metadata_recovery` inline from the captured source snapshot, +including the RTR header, MethodTable addresses, frozen strings, derived overlay +identity, and examined/recovered counts. Its enclosing `unsupported` load-image +status means independent PE load-image verification was not performed; it does +not mean the metadata parser failed or the executable failed to load. -Pass one returned type path or its typed address to `inspect_native_data_type`. -Its optional `metadata_recovery` includes related/base type, implemented -interfaces and virtual slots with targets and known procedure names. Follow -these with `analyze_function`, string searches and ordinary cross-references. -For CLI use: +Pass a returned MethodTable address to `inspect_native_data_type`. A direct PE +query also derives the snapshot metadata from the authenticated Ghidra session +handshake, so it does not require a preceding load-image call. In a persistent +MCP session, load-image followed by type inspection reuses that same derivation. +The database result remains `unavailable` when Ghidra has no DataType at that +address; its `metadata_recovery.source` is `read-only-derived-overlay` and +carries related type, interface, slot, and target observations. Generated +address identities do not claim original managed names. The parser does not +merge frozen literals into Ghidra's database string inventory. Follow slot +targets with `analyze_function` and ordinary cross-references. + +The built-in CLI path uses a MethodTable address returned by the parser or +another observation: ```sh rea inspect-native-load-image ./NativeAotFixture --provider ghidra --json -rea inspect-native-data-type ./NativeAotFixture --type /NativeAOT/MethodTables/Class_ADDRESS_MT --provider ghidra --json +rea inspect-native-data-type ./NativeAotFixture --address 0xMETHODTABLE_ADDRESS --provider ghidra --json rea function ./NativeAotFixture 0xADDRESS --provider ghidra --json ``` -No new MCP workflow or approval flag is required. Configuration opts into the -adapter; without it ordinary Ghidra profiles/results remain unchanged. +The generated `/NativeAOT/...` category path belongs to the optional headless +adapter. Use `--type /NativeAOT/MethodTables/Class_ADDRESS_MT` only when that +adapter is configured and has created the corresponding Ghidra DataType. +Built-in PE analysis includes its parser contract and resource policy in the +Ghidra analysis profile; non-PE profiles do not include that contract. ## Evidence boundaries -Discovery prefers an imported RTR symbol, otherwise the upstream initialized, -non-executable-memory signature heuristic. REA rejects ambiguous candidates, -unknown layout versions, malformed directory rows, unsupported flags and -unmapped ranges before recovery. It requires the verified dehydrated layout. +The built-in parser uses an initialized non-executable PE signature scan and +requires a unique RTR 9.1 header and exactly one DEHYDRATED_DATA row. Rehydrated +bytes live only in a parser-owned overlay, identified by virtual address, size, +and SHA-256; the caller's source Buffer remains unchanged. Limits derive from +captured section extents and format field bounds. Separate resource policies +admit up to 128 MiB of captured PE bytes, 64 MiB-equivalent parser work, 64 MiB +of exact parser-owned overlay/index Buffers, and 4 MiB of serialized NativeAOT +report JSON. These are local parser limits, independent of MCP delivery; the +buffer counter does not claim to cap V8 heap or total process memory. Rows that +do not fit the report budget are omitted with explicit partial status and +coverage. Coverage counts describe the pointer scan, recovered MethodTables, +and frozen-string candidates examined, not every possible runtime type or +object. Cancellation is honored during snapshot I/O and checked immediately +before and after parsing; the synchronous parser call itself is bounded but +cannot be interrupted mid-call. Parser output never claims that a Ghidra +database type was defined. -The adapter performs rehydration, method-table recovery and frozen-object -annotation in one ephemeral Program transaction. Fatal failure/cancellation -rolls it back. Recovery may change analysis-memory bytes, never the original -executable file. Derived ranges have a digest and **no original file offset**; -they are not captured runtime state. Candidate/committed instance annotation -counts distinguish partial frozen-object coverage. Coverage concerns the -recovered candidate set, not every possible object or every runtime type. +The optional adapter performs its annotations in an ephemeral Program +transaction. That workflow may change analysis-memory bytes, never the original +executable file. It is unavailable on Windows P0, which has no database +mutation authority. REA preserves pre-recovery calling conventions and uses the loaded compiler specification default for newly created methods. The upstream universal diff --git a/docs/ghidra-provider.md b/docs/ghidra-provider.md index b1e87a188..a79afecbd 100644 --- a/docs/ghidra-provider.md +++ b/docs/ghidra-provider.md @@ -8,9 +8,9 @@ Executable bytes remain unchanged. Use [installation](installation.md#ghidra) for supported hosts, prerequisites, configuration, startup deadlines, resource controls, and cleanup recovery. [Windows P0](windows-ghidra-p0.md) has its own admission and lifecycle boundary; -[NativeAOT recovery](ghidra-nativeaot.md) is an optional, separately verified -extension. Check the [release boundary](installation.md#released-package-and-main) -when using npm. +[NativeAOT observations](ghidra-nativeaot.md) include a built-in read-only PE +snapshot parser and a separate optional annotation extension. Check the +[release boundary](installation.md#released-package-and-main) when using npm. See [import language and analysis seeds](installation.md#ghidra-import-language-and-analysis-seeds) to override the header-detected language or seed analysis. diff --git a/docs/javascript-artifact-reconstruction.md b/docs/javascript-artifact-reconstruction.md index 6446432d9..02373d625 100644 --- a/docs/javascript-artifact-reconstruction.md +++ b/docs/javascript-artifact-reconstruction.md @@ -247,6 +247,46 @@ while locally shadowed receivers can still contribute network endpoint candidates. A template recovered with a parser error and no cooked value stays dynamic; its raw spelling is not treated as a valid JavaScript string. +RPC client syntax also contributes network endpoint candidates, in both +`analyze_javascript_application` and `analyze_web_bundle`, distinguished by +`mechanism`: + +- `trpc:procedure:` records the static member path between the root + expression and the method, such as `invoice.list` in + `root.invoice.list.useQuery(...)`. tRPC proxies keep these property names + through identifier minification. React hooks and TanStack option factories + (`useQuery`, `useMutation`, `useInfiniteQuery`, `useSubscription`, suspense + and prefetch variants, `queryOptions`, `infiniteQueryOptions`, + `mutationOptions`, `subscriptionOptions`) are recognized directly. The + vanilla `query` and `mutate` methods are recognized on a root bound to + `createTRPCClient` or `createTRPCProxyClient` in the same lexical binding, or on a + path of at least two segments in a source containing `TRPCClientError`, + `trpc-accept`, or `/trpc`. Factory bindings must have one unconditional + initializer and no binding assignments; shadowed locals do not inherit client + identity. `subscribe` additionally requires a tRPC observer + argument (`onData`, `onError`, `onStarted`, `onStopped`, `onComplete`, or + `onConnectionStateChange`). `this` roots, browser globals, and any path + containing an RTK Query `endpoints` segment are never procedures; cache-key + helpers such as `queryKey` issue no request and are excluded. +- `trpc:link:` records the literal `url` option of `httpLink`, + `httpBatchLink`, `httpBatchStreamLink`, and `httpSubscriptionLink` when the + link's callee name survives bundling, including `(0, module.httpBatchLink)` + import-call wrappers. +- `graphql:query`, `graphql:mutation`, and `graphql:subscription` record a + named operation found at the start of a string or cooked template literal — + the name must be followed by variable definitions (`($`), a directive, or a + selection set starting with a field name — or in a precompiled + `OperationDefinition` document object. A graphql-tag `loc.source.body` copy is + not reported twice. Anonymous operations are not reported. + +A procedure path is the server-side procedure name, not a resolved URL; the +transport URL depends on the link configuration and batching. When the proxy is +reached through a member expression, such as a webpack module binding +(`l.S.invoice.list`) or a property holding the proxy (`o.api.post.byId`), the +recorded path keeps those leading carrier segments; the procedure path is its +suffix. Utility calls such as `useUtils().invoice.list.fetch()`, `useQueries`, +and links whose callee was renamed by a minifier are not recognized. + Each recovered bundle module retains the exact factory-source digest. A complete bounded AST also receives a `babel-ast-v1` structural fingerprint that ignores ordinary identifier names while retaining syntax, literals, operators, object diff --git a/docs/testing.md b/docs/testing.md index f1d43ee3a..300bf473e 100644 --- a/docs/testing.md +++ b/docs/testing.md @@ -672,6 +672,13 @@ boundaries and budgets. ## Optional NativeAOT Ghidra analysis +The source parser lane is independent of the optional annotation JAR. Its +synthetic RTR 9.1 PE golden checks captured-byte digest admission, rehydration +into an immutable parser overlay, MethodTable/slot relationships, and frozen +literal extraction; provider tests exercise `inspect_native_load_image` and +address-based `inspect_native_data_type` with the database result still +unavailable. These tests do not establish Windows Ghidra host behavior. + This lane is separate from the default native lane. `build:fixtures:nativeaot` requires an existing .NET SDK 8.0.416, the platform NativeAOT compiler/linker and runtime pack 8.0.22. It builds benign sources into ignored `_reference/` and diff --git a/docs/windows-ghidra-p0.md b/docs/windows-ghidra-p0.md index 5f50d832c..7d72056f6 100644 --- a/docs/windows-ghidra-p0.md +++ b/docs/windows-ghidra-p0.md @@ -21,9 +21,12 @@ additional permission flag or degraded mode is required. - the 25 read-only Ghidra inventory, memory, and function-analysis operations. Loaded memory reads and file offsets preserve Ghidra's observed source mappings. -PE load-image inspection returns measurements with an explicit unsupported -attestation status; independent format-specific verification currently supports -DOS MZ only. +PE load-image inspection returns an explicit unsupported attestation status; +independent load-image verification currently supports DOS MZ only. For x64 PE +NativeAOT RTR 9.1, the read-only snapshot parser can report metadata through +`inspect_native_load_image` and `inspect_native_data_type` without a JAR or +database mutation. A real Windows-host NativeAOT Ghidra run is still required +to establish the complete host workflow. This boundary does not establish general Windows feature parity. Ghidra GUI state, annotations, other target architectures and formats, and Hopper-only diff --git a/licenses/nativeaot-parser-MIT.txt b/licenses/nativeaot-parser-MIT.txt new file mode 100644 index 000000000..cd793e5ba --- /dev/null +++ b/licenses/nativeaot-parser-MIT.txt @@ -0,0 +1,28 @@ +The MIT License (MIT) +===================== + +Copyright © 2023-2025 Washi + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the “Software”), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. + +The read-only NativeAOT parser adapts format and hydration semantics from +Washi1337/ghidra-nativeaot commit +effeb734fc570c32650f88b159608979dc7b423e (MIT). MethodTable field layout was +cross-checked against .NET runtime src/coreclr/nativeaot/Runtime/inc/MethodTable.h +at dotnet/runtime commit a3b2d40328be0be3f8dd43f0e5cc925b8827b044 (MIT). diff --git a/package.json b/package.json index 0592256db..f96e53d19 100644 --- a/package.json +++ b/package.json @@ -81,7 +81,8 @@ "bridge/pwndbg/launch_impl.py", "bridge/pwndbg/core_context.py", "third_party/pwndbg/README.md", - "third_party/pwndbg/LICENSE.md" + "third_party/pwndbg/LICENSE.md", + "licenses/nativeaot-parser-MIT.txt" ], "scripts": { "release:notes": "node scripts/release-notes.mjs", diff --git a/scripts/ci/plan.mjs b/scripts/ci/plan.mjs index 40ec03a6b..46ee8d039 100644 --- a/scripts/ci/plan.mjs +++ b/scripts/ci/plan.mjs @@ -14,6 +14,24 @@ const flags = [ "workflows", ...Object.keys(SCOPES), ]; +// Actionlint validates workflow syntax only. The semantic suite asserts these +// workflows' release, publication and self-hosted runner safeguards. +const GUARDED_WORKFLOWS = new Set([ + ".github/workflows/release.yml", + ".github/workflows/real-ghidra-windows.yml", +]); +// Sources compiled by build-conformance-fixtures.mjs. The readiness journey +// analyzes the built C fixture as well as its own applications. +const FIXTURE_SOURCES = [ + "tests/conformance/c/**", + "tests/conformance/napi/**", + "tests/conformance/objc/**", + "tests/conformance/swift/**", + "tests/conformance/versions/**", +]; +const READINESS_SOURCES = ["tests/conformance/readiness/**"]; +const matchesAny = (file, patterns) => + patterns.some((pattern) => matchesGlob(file, pattern)); const selected = Object.fromEntries(flags.map((flag) => [flag, false])); const reasons = []; const full = (reason) => { @@ -122,6 +140,7 @@ if ( file === ".github/actionlint.yaml" ) { selected.workflows = true; + if (GUARDED_WORKFLOWS.has(file)) selected.code = true; if (file !== ".github/workflows/ci.yml") continue; } if (owners.length === 1 && owners[0] === "website") continue; @@ -131,6 +150,11 @@ if ( continue; } selected.code = true; + if (matchesAny(file, FIXTURE_SOURCES)) { + selected.fixtures = true; + selected.readiness = true; + } + if (matchesAny(file, READINESS_SOURCES)) selected.readiness = true; if (file.startsWith("tests/")) continue; if ( /^(src\/process\/|src\/filesystem\/|src\/contracts\/tool|src\/server\/createServer|src\/cli\.ts|scripts\/ci\/)/.test( diff --git a/scripts/lib/browser-scenario-storage-e2e.mjs b/scripts/lib/browser-scenario-storage-e2e.mjs index bcf71b1b2..601a10053 100644 --- a/scripts/lib/browser-scenario-storage-e2e.mjs +++ b/scripts/lib/browser-scenario-storage-e2e.mjs @@ -99,6 +99,14 @@ const scenarioFor = (browser, origin) => capture: { after_each_step: ["dom"], at_end: ["dom"] }, }); +const raceWithTimeout = (operation, timeoutMs, onTimeout = () => undefined) => { + const controller = new AbortController(); + return Promise.race([ + operation, + delay(timeoutMs, undefined, { signal: controller.signal }).then(onTimeout), + ]).finally(() => controller.abort()); +}; + const storagePage = (kind, port) => { if (kind === "main") return `