a91412aaf9
* feat(decompile): write function pseudocode to a directory and return a manifest Add decompile_all and `rea decompile-all` so a caller can write one provider pseudocode file per function, or a filtered set, and receive only a small manifest. The result stays bounded when the program is larger than a local model's tool-result cap. Resume continues an export for the same target digest, provider, analysis profile, and layout. Matching files are skipped, conflicting files stay unchanged, and one function's failure does not stop the rest. Cancellation leaves files already written. Requesting dwarf layout reports that compilation-unit paths are unavailable. Simulated-provider tests cover the manifest, resume, filters, limits, cancellation, and the failure cap. MCP registration, CLI coverage, schema validity, the product catalog, check:fast, and docs:check passed. Real Ghidra was not exercised. Refs #6090 * fix(decompile): keep unknown export identity unresumable and recover torn journals Resume treated two missing target or profile digests as a match, so an export of one binary could continue into another. Resume now requires both digests, and a decompiler observation without the export's profile, or with a different target, is recorded as profile_changed. A crash during a journal append left a torn final line that made the export unresumable. Text after the last newline is now treated as uncommitted and truncated before the next append, and stale temporary files are removed. An inventory row with an empty address aborted the whole export because its failure record failed the journal schema. Such rows are now excluded and counted in the limitations. A limited resume retried the same recorded failures first and never reached new functions. Functions not yet attempted now run before retries. File path reservations come from every journal line, so an output_io record no longer releases a path. The index is rebuilt from the journal in numeric address order, and journal close and cleanup failures return a Result. The cancellation test now cancels after a file is committed and checks that file survives and is skipped on resume. Redundant snapshot-replay checks are removed, the CLI limit uses the contract constant, and --snapshot no longer claims decompile-all updates it. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01AoJZDPcrgmMsCqLG4LfiFb * fix(decompile): refuse resume of a missing export and repair journals by byte offset A second review round found that resume against a directory that does not exist silently started a full new export and reported it as resumed. Resume now requires an existing export directory. The torn-line repair measured the committed journal from decoded text, so damaged UTF-8 could move the truncation point past the end of the file. The journal is now read as bytes and cut at the last newline byte, and an intact journal is not truncated, so a concurrent append is kept. A per-function observation for a different target is now recorded as target_changed instead of profile_changed. A failure to remove a partly created output directory is reported instead of hidden. The journal handle is closed on every path again, including an unexpected throw. The path allocator no longer gives up after 1,000 collisions and reuses a taken name. Tests now use createTestTempDirectory, assert journal lines against the journal schema, and cover resume of a missing directory, resume with an unknown analysis profile, and a target-mismatched observation. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01AoJZDPcrgmMsCqLG4LfiFb * fix(decompile): keep a function's first file name across failed retries Reserving every journal path stopped other addresses from taking a path, but an address looked up only its latest record. After its file went missing and a retry failed without a path, the next resume moved the function to a hash-suffixed name and orphaned the original. Each address now keeps the first path the journal gave it. A matched commit still ran the decompiler but did not count toward limit. The limit now counts every function taken on after skipping verified files. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01AoJZDPcrgmMsCqLG4LfiFb * fix(decompile): never follow a journal symlink and check procedure_info identity Torn-line repair truncates the journal, and the journal was opened through any symlink in the export directory, so repair could truncate a file outside it. The journal must now be a regular file and is opened without following symlinks where the platform supports it. Body ranges from procedure_info were written without checking that the observation belonged to this export or described the requested function. An observation with a different provider, profile or target, or another entry point, now leaves the body unavailable with that reason. File, journal and index writes now use FileHandle.writeFile, which writes the whole buffer instead of trusting one write call. The stale temporary sweep is removed: a matching file name does not prove this export created the file. The inverted-range test now uses one address space so it reaches the inversion check. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01AoJZDPcrgmMsCqLG4LfiFb * fix(decompile): bound returned caveats and keep verified checksums Ghidra names each terminal call's address in its per-function caveats, so decompile_all copied one new caveat per function into the returned Evidence and the answer grew with the program. The returned Evidence now keeps the eight most often observed distinct per-function caveats, grouped by provider and text as they are rendered, plus a count of the rest. Inventory caveats are unchanged, and other workflows are not bounded. Each function keeps the caveats of the observations it used: a written file in its header, now including procedure_info caveats, and a failed attempt in its journal and index record. A procedure_info observation for another target or profile is foreign and contributes none. A failure record for a function that was already written replaced the record holding its checksum, so after a transient read error a resume decompiled that function again. The journal now keeps the last successful write separately from the latest attempt. Resume verifies against that write and records it again once the file verifies. A caller abort that arrives while the export finishes now returns cancellation with the caller's reason instead of a success manifest, as #6143 does for other late finalization. A publish, index or close failure stays the primary error. Also make the missing-journal resume error say what to do, and add boundary tests: one runs decompile-all through the real CLI against a fake Kuna to check flag mapping, the relative output directory, filters and limit; one runs decompile_all through production MCP registration to check that the caveat bound is applied. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01AoJZDPcrgmMsCqLG4LfiFb * fix(decompile): keep each function's complete caveats in the index The returned Evidence keeps eight distinct per-function caveats and counts the rest, pointing readers at the export. A written function's file header truncates its caveats after 4,000 characters, and its journal record kept none, so a large function with many terminal calls could lose caveats from both the answer and the export. Every journal record, written or failed, now keeps the complete caveats of the observations its attempt used, and decompile-index.json carries them. The header stays a readable summary and, when truncated, says that the index keeps the complete list. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01AoJZDPcrgmMsCqLG4LfiFb * fix(decompile): report external functions instead of failing them Ghidra classifies imports as external functions with an empty body and returns null pseudocode for them, as its limitations state. decompile_all recorded each one as an invalid_pseudocode failure ("not text"), so a dynamically linked program reported every import as a decompiler failure: 34 of 116 functions in /usr/bin/xxd. procedure_info owns the external classification. When it classifies a function as external, the export now records status "external", counts it in counts.external, writes no file and requests no pseudocode. The record keeps the body fact as observed. The journal header pins the target, provider and profile, so a resume skips a recorded external function without asking the provider again; a limit below the external count still leaves nothing remaining. A null pseudocode result for a function whose classification is not external is reported as no_pseudocode. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01AoJZDPcrgmMsCqLG4LfiFb * fix(decompile): report finalization failures after a cancelled export When the caller's abort stopped the publish loop, decompile_all returned that cancellation before checking the index write and the journal close, which had already run. A failure to finalize the export was hidden, so the caller could not tell the saved index or journal was incomplete. Publish failures stay primary, then index and close failures. A cancellation counts as the caller's only when the caller's signal is aborted, and is then reported after a successful finalization with the caller's own reason. A provider's own cancellation, without a caller abort, stays a publish failure like any other. Tests cover an index failure and a close failure after the caller stops the loop, a stopped loop with a successful finalization keeping the caller's reason, and a provider cancellation staying primary over an index failure. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01AoJZDPcrgmMsCqLG4LfiFb * fix(decompile): remove a created output directory that cannot be secured createPrivateDirectory ran mkdir and chmod as one step. When mkdir succeeded and chmod failed, it returned an error before #create reached its cleanup, leaving an empty directory behind. A fresh export then refused the existing directory, and a resume refused it for having no journal, so the caller had to remove it by hand. mkdir and chmod are now separate. A mkdir failure created nothing, so nothing is removed. A chmod failure after this call created the directory removes it with a non-recursive rmdir: it is still empty, so a path another process filled or replaced meanwhile is refused rather than deleted, and that refusal is reported. A filesystem boundary test fails chmod for the new root and checks the directory is gone and a retry succeeds. A module test confirms that a function written earlier and now classified as external still reports a changed file at its path as output_exists. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01AoJZDPcrgmMsCqLG4LfiFb * refactor(process): share the socket lease with owners other than Hopper The socket lease that keeps two REA processes from opening the same Hopper document lived only in HopperLease, with Hopper error messages built in. A decompile export needs the same guarantee for its output directory. OwnedLease now holds the lease mechanism unchanged: a listening local socket that the operating system ends with its process, owner identity answered by the live holder, and recovery of a stale socket serialized through a recovery directory. Callers map a tagged LeaseFailure to their own error. HopperLease keeps its exports and maps each failure to the same HopperStartError message as before. On Windows the lease is a named pipe, which needs no stale recovery. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01AoJZDPcrgmMsCqLG4LfiFb * fix(decompile): hold the output directory per run and stop on repeated write failures Two runs could resume the same output directory at once. Each read the journal once, and the index each wrote reflected only its own view, so the last writer could drop the other run's records. A run's torn-line repair could also truncate the other run's append. DecompileExport now holds a lease on the output directory, named by its device and inode, from before the journal is read (or, for a new export, from just after the directory is created and before its journal exists) until the journal is closed. A second run is refused as a retryable capacity constraint that reports the holder's process ID. The lease ends with its process, so a crashed run leaves no lock to clear. The lease is released on every open failure, and a failed release is reported. An export also kept decompiling every remaining function when its files had stopped being writable mid-run, for example on a full disk, failing each one with output_io. After five consecutive functions whose files could not be written or read, the export now stops with an output error that carries the counts and the last failures. A verified file or any other outcome ends a run of failures. Committed files and records stay, the index is still written, and a later resume continues. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01AoJZDPcrgmMsCqLG4LfiFb * docs(native): record directory export verification Record matched real Ghidra outputs, bounded caveat and cleanup regressions, actual catalog-inclusive context estimates, runtime measurements and coverage limits for the pinned integration. * fix(native): bound export summaries and reserve portable paths Use one case-folded reservation owner for allocation and resumed journals. Budget inline failures and provider caveats by encoded JSON bytes while retaining complete records in the output index. Align the tool contract with the collector order and verify SDK result-window behavior. * docs(native): refresh final export measurements --------- Co-authored-by: Claude Opus 5.5 <[email protected]> Co-authored-by: gensokyo-dev <[email protected]>
15 KiB
Native directory decompilation verification — 2026-10-23
The completed analyst task is to export the bound provider's function inventory as one local pseudocode file per non-external function, with names, body ranges, provider/profile identity and the limitations of each used observation. The agent receives a manifest rather than the pseudocode. A function decompilation failure remains a recorded partial outcome and does not end the export.
Artifact, provider and correctness
The real target was the installed ELF x86-64 /usr/bin/sha256sum (60,368 bytes),
SHA-256 6cd7c6bfc81d645ba13b927e31651a1466092a28ed0bd2632e82f8b27882b25e.
It was read and analyzed statically, never executed. Both variants used the
existing Ghidra 12.1.4 installation and JDK 21.0.12.1+1, Node 24.22.1, the locked
npm dependencies, GHIDRA_HEADLESS_MAXMEM=1G and
GHIDRA_HEADLESS_JAVA_OPTIONS=-XX:ActiveProcessorCount=2.
The upstream analysis-profile digest was
ae80ece8c83379eddabbdf379bbffca5da84c7df7fa7c159f45fa810468f85d9.
Baseline main was 5d2c8c2b78d2552da0e13a11d3e2678efb69810d.
The feature branch integrates original PR head
d3a207038cfc00d58a97b86b06af1368839bec99 on that base, with the maintenance
changes described below. The frozen integration is c566cace04d24f8ccb3d54353ababf95b96c8830,
including refreshed main bded9c628de23643aab81f98959b6a473318a240. The review fixes were verified on d3e0e96821a09a447785d100f3d85139b6e77e96. Each variant ran through the public production
scripts/rea.mjs mcp entrypoint and the installed MCP SDK client.
The baseline called list_procedures, then procedure_info for each addressed
entry and procedure_pseudo_code for each non-external entry. A local verifier
used the returned Evidence to materialize the same function files and retain an
index. It reused the feature's pure file-header renderer to keep file encoding
equal; baseline analysis itself used only the existing public primitive tools.
The existing batch_decompile item format does not retain per-function caveat
associations; its workflow limitations aggregate across items. It would not
provide equivalent per-function caveat provenance here, so the baseline uses
primitive observations. The feature called decompile_all once per output directory. Both retained the
complete MCP responses locally. A second export used a new output directory in
the same session. The feature additionally resumed the verified warm export.
Both cold and warm exports examined the same 298 provider inventory entries: 236 written, 61 external, one failed. Addresses, statuses, body ranges, function names, pseudocode SHA-256 and complete emitted-file SHA-256 matched for every entry in both rounds. The target's bytes stayed unchanged. This proves export preservation of the provider's observed results; it does not establish that the provider found every actual function or that its pseudocode is original source.
Ghidra failed at 0x10a510 with Overlapping input varnodes; the failure code
was execution_failure. The export still wrote 68 functions at later numeric
addresses. Resume skipped every successful/external entry and retried the one
failed entry without rewriting the successful files. Complete per-function
caveats stayed in the index, and no pseudocode was returned in the manifest.
Measurements
| Completed workflow | Before: public primitives + local files | After: directory workflow |
|---|---|---|
| Cold export: MCP calls | 536 | 1 |
| Cold export: request + response token estimate | 1,375,776 | 3,800 |
| Cold export: elapsed | 50.940 s | 39.155 s |
| Cold export: sampled family RSS peak | 1754.5 MiB | 1709.3 MiB |
| Warm export: MCP calls | 536 | 1 |
| Warm export: request + response token estimate | 1,377,486 | 3,800 |
| Warm export: elapsed | 8.566 s | 8.058 s |
| Warm export: sampled family RSS peak | 1779.4 MiB | 1710.6 MiB |
The identical two-export task, including the full catalog discovery, open and
close, used 3,723,191 → 981,913 estimated tokens (73.63% fewer).
The extra verified-resume call is excluded from that comparison. The catalog
itself grew from 969,665 to
974,049 estimated tokens. All 220 existing tool definitions
were identical; only decompile_all was added. The per-export request/reply
reduction is about 99.72%; applying that percentage to the entire task would
overstate the benefit. A client which injects the entire catalog still pays its
large fixed context cost.
The full verifier process family took 85.697 → 62.034 s elapsed, 111.166 → 68.185 s waited CPU, and 1779.4 → 1713.7 MiB sampled peak RSS. The after run additionally includes verified resume (0.266 s). These compare the retained baseline with the final review-fix run; no general CPU or elapsed-time speedup is claimed. Earlier maintenance trials ranged 50–83 s elapsed and 65–118 s CPU; they used earlier source revisions and are retained separately. Full-run figures include SDK, verifier formatting/hash checks and token estimation as well as REA and its provider. Per-phase sampled CPU was 70.19 → 43.16 s cold and 6.15 → 4.42 s warm, subject to delayed child accounting. The integration decision rests on equivalent results with far fewer agent-facing calls and estimated context, plus bounded auxiliary accumulators.
Cold/warm here describe primitive analysis reuse inside one provider session. Host filesystem caches were not flushed, and the baseline ran first. This is a paired functional measurement, not a statistical speedup claim. Provider startup and host load varied: the independent CLI runs below took longer than the MCP runs. No claim is made about cold OS caches or other binaries.
Token counts use tokenx 1.3.0 estimates of actual tool arguments and complete
MCP call results, including both textual and structured projections. They are
not model-specific BPE counts or actual model usage. The completed-task total
also includes the actual tools/list response, open and close calls, and both
exports. MCP handshake framing, orchestrator prompts, reasoning, subsequent
agent reading of files and unknown model tokenizer costs are unmeasured.
Process-family RSS was sampled from /proc every 50 ms. Shared pages can be
counted more than once, and short peaks can be missed. Full-run CPU uses waited
child usage; per-phase CPU is a sampled /proc delta and can include delayed
accounting of exited descendants. Limits were two CPU cores, a 2 GiB V8 heap per
Node process, 1 GiB configured Java heap, 6 GiB sampled family RSS, 300 sampled
family CPU seconds and 600 elapsed seconds. These are verifier controls, not a
kernel memory limit. All reported completed runs exited zero without a resource
stop. No reduction in Java's analysis memory is claimed.
The first baseline verifier attempt stopped at the real per-function Ghidra error because its harness initially required every primitive call to succeed. It took 74.325 s elapsed and 101.569 s waited CPU, with no resource stop. The completed baseline was corrected to retain that same failure and continue, matching the requested partial-result semantics. The failed trial is retained separately and is excluded from the paired completed-export table; no provider fix or disappearance of that failure is claimed.
Ownership, reuse and resource limits
The export owner parses a resumed journal once and updates its latest records
only after append and sync complete. The shared function-record schema owns
path/status validation for both journal and index; the append retains the
parsed snapshot, including nested fields, rather than the caller's mutable input. Index generation consumes those committed
facts instead of reading and parsing the whole journal again. Resume identity
covers target SHA-256, provider identity/version, analysis-profile digest and
layout. Unknown target/profile digests prevent resume. Files are checked afresh
with the shared stable-file hash owner and bounded reads, rather than trusting
an old checksum as current. decompile_all remains ineligible for snapshot
replay: filesystem publication must run for the actual request.
Journal descriptors, verified staging areas and the directory lease have one cleanup owner. Failed close/removal retains that owner under the session; the lease is released after child cleanup and initialization cleanup finish. Initialization removes verified owned entries rather than recursively deleting unexpected files. Function path reservations use case-folded ASCII lookup keys while preserving the original address identity and chosen filename; collisions receive a stable address digest suffix. Resume refuses conflicting legacy reservations. These cases are exercised with the real filesystem on Linux; a real Windows/macOS filesystem was not exercised. A failed lease acquisition preserves even an empty directory and reports its created path, because another holder may be preparing it.
The inline failure accumulator retains at most 32 rows within 2,048 JSON-encoded UTF-8 bytes. It omits display names before omitting complete address/code/reason rows. Recent output-I/O diagnostics retain at most five rows within the same byte budget. The index retains all complete function records. Per-item caveat collection retains at most eight first-observed provider/text groups within an 8,000-byte JSON-encoded UTF-8 text budget, instead of retaining all distinct caveats to rank them. Omitted observations are counted without retaining their text; this count is not a distinct-caveat count. Complete caveats remain in the owning index.
Inventory and latest journal/index state still scale with the number of functions and their retained metadata. Larger exports, long-running journal histories and worst-case metadata expansion were not capacity-benchmarked; this report makes no 100,000-function memory guarantee. The attempt limit is a positive safe integer and does not allocate according to its numeric value.
Validation
npm run check passed after the review fixes on d3e0e9682: type checking,
lint, formatting, dependency/metadata checks, module boundaries and architecture
guards. npm run docs:check also passed, including nine README conformance cases. After the main refresh, 226 tests passed across ten files, including the
four Ghidra profile suites and the six CLI/MCP/workflow/schema files. Earlier
Hopper lease and session-cleanup coverage below is additional; repeated tests
are not counted as new cases.
- The focused integration run passed 112 tests across 11 files; four macOS-specific real-Hopper startup cases were skipped on Linux. It includes public CLI/MCP behavior, actual SDK schema conversion/dialect/client checks, the shared lease extraction, session cleanup retry and descriptor ownership.
- After the final caveat/cleanup changes, 75 tests passed across the four affected files: 48 export cases, 20 workflow-Evidence cases, six filesystem ownership cases and one MCP caveat case. This adds five distinct cases to the earlier integration coverage; overlapping executions are not summed.
- Follow-up review coverage passed 72 tests across the workflow and MCP files, plus 22 filesystem/CLI/schema cases. This includes portable filename reservations, conflicting legacy journals, encoded caveat budgets and long failure names. Against the published pre-fix implementation, all three new regression cases failed; the complete SDK result for the long-name case was 485,937 bytes, above the 51,200-byte tool window. The repaired case retains useful failures inline and verifies complete names and caveats in the index.
- Original-PR controls reproduce the uppercase-input refusal, valid-journal initialization cleanup race, lost failed-close ownership, unexpected-file deletion, and the empty-root lease-acquisition race. These controls use the original production implementation, not the repaired one. An additional production selection control returns zero matching aliases before the fix and the selected alias after it. A separate negative control of the maintenance index-reuse change reproduced journal/index disagreement after mutation of the append input; retaining the parsed snapshot resolves it.
- Three real Ghidra CLI calls passed: uppercase address range plus limit 1 wrote one of two functions; resume with limit 100001 skipped the verified file and wrote the other; an unmatched name selected zero and reported 298 excluded inventory entries. Both output files matched the full MCP export. The CLI group took 166.128 s elapsed, 120.848 s waited CPU and 1257.0 MiB sampled peak family RSS, without a resource stop.
Raw measurements and verifier scripts remain in the maintainer workspace at
/tmp/rea-maintenance-20261023/6170/, including completed and failed attempts,
full MCP exchanges, all function outputs and equivalence checks. They are local
verification artifacts, not shipped provider projects or committed binaries.
Real-provider coverage here is Linux x86-64 Ghidra only. Hopper, IDA, Windows and macOS
were not exercised with real providers by this verification.
CI attribution
The first maintenance CI run used synthetic merge
798091d99f5c9f434d832981e0c8d28a5ec4aabc and exact base
9fd78a74f84a8e924bd58e7deaf5934bfb24f53c. All 54 observed failing
assertions reproduced on both revisions with identical failure diagnostics
after normalizing only their checkout paths: 11 source cases and 43 built
cases. Forty built failures came from cancelled-shard logs and were absent
from the aggregate report; cancelled coverage is not claimed as executed.
The real-browser job also failed its stable-capture checkpoint. A bounded
checkpoint-only execution of the production verifier reproduced the same
dom_structure: unchanged, scripts: unknown result on exact base and
merge, with cleanup retained. Local Chromium was 155.0.8059.79 on Debian 12;
the CI Chrome version was not measured. This does not establish a passing
complete browser workflow. The public CI attribution
records the controls and separates these baseline failures from the three
actionable feature review findings fixed above.