Compare commits

...
Author SHA1 Message Date
morluto c2077c9f3b fix(native): reconcile current main formatting 2026-11-07 23:17:06 +08:00
morluto ac30fbd658 style(test): format CLI help default annotations (#9280) 2026-11-07 23:02:44 +08:00
Nianze Wuandmorluto 5d21bddb18 fix(desktop): read JSON with deeply nested overwritten values (#9277)
* fix(desktop): read JSON with deeply nested overwritten values

* fix(desktop): bound exported JSON construction by view depth

* fix(desktop): preserve deep JSON numeric facts within bounded memory

---------

Co-authored-by: morluto <[email protected]>
2026-11-07 22:57:15 +08:00
morlutoandCursor Agent 2e2f981ccf test: assert absolute-path facts without frozen sentences (#9231)
* test: assert absolute-path facts without frozen sentences

Sibling path tests now require the field path, invalid_format, and regex
expectation, and they record that each negative case ran. NUL rejection
follows the argument that contains the byte instead of searching for the
word NUL.

* test: retain actionable NUL path diagnostics

---------

Co-authored-by: Cursor Agent <[email protected]>
2026-11-07 22:46:17 +08:00
morluto 1693e074b0 fix(native): reconcile procedure reference views with main 2026-11-07 22:40:05 +08:00
Heo Hao f8bb792f15 fix(cli): stop restating schema defaults in help descriptions (#9278)
The help renderer appends "(default: X)" whenever the option field declares a
schema default, so descriptions that also spell the default out print it twice:

  --block-size <value>  Power-of-two block size from 256 to 65536 bytes
                        (default 4096) (default: 4096)

Remove the hand-written default from the eleven options that own a schema
default across inspect-byte-entropy, inspect-devicetree,
inspect-packet-capture, inspect-packet-fields, inspect-signal-decoders and
decode-signal-recording, matching the rule in cliDecimalField's contract that
the domain field owns bounds, defaults and help.

Options without a schema default keep their prose: --match_limit, --line_limit,
--limit and --packet_limit are nullable or optional, so the renderer emits no
annotation and the description is the only place the effective value appears.

Add a boundary test asserting each such option states its default exactly once,
so a restated default cannot come back unnoticed.
2026-11-07 22:38:03 +08:00
Cursor Agent 9975a60381 ci: rerun after the binary-layout toolchain timeout
The required check failed because the toolchain install exceeded 15 minutes.
The same job completed on the previous head, and this commit does not change
the procedure-reference behavior.
2026-11-04 22:50:57 +00:00
Cursor Agent e57e4c72e7 test(native): catch the string-length endpoint comparison
Use a 200,000-character multibyte selected procedure and a shorter ASCII
endpoint so the old JavaScript length check drops the selected row.
2026-11-04 22:29:15 +00:00
Cursor Agent b436f44b58 test(cli): count procedure-references as a dedicated command
The MCP tool has a CLI command, so it is no longer an MCP-only exception.
2026-11-04 22:06:40 +00:00
Cursor Agent 025f10a848 fix(native): compare endpoint size with UTF-8 identity length
Keep one running maximum of stored identity bytes. A multibyte selected
procedure is recognized when its JSON no longer fits the page.
2026-11-04 21:45:13 +00:00
Cursor Agent 9ebef9c8d1 fix(native): describe procedure-reference direction and coverage
Avoid execution wording in the tool description, and give the CLI direction
option the help text the catalog requires.
2026-11-04 21:39:04 +00:00
Cursor Agent 255865c727 fix(native): reject an oversized procedure endpoint before canonicalizing it
When the JSON size already exceeds every stored identity, skip the canonical
string and keep the row out of the page.
2026-11-04 21:22:45 +00:00
Cursor Agent acad91bb54 fix(native): measure a procedure endpoint once before its comparison key
Size the candidate before building one canonical identity, and reuse that
key for the selected-procedure and shared-endpoint checks.
2026-11-04 21:00:44 +00:00
Cursor Agent 5833bb7c9b Merge origin/main into codex/procedure-reference-stacked
Keep procedure-reference pages beside the landed instruction-reference view
and the value-flow parameter description.
2026-11-04 20:38:25 +00:00
Cursor Agent b7ebd60341 fix(native): count procedure-reference endpoints before keeping a row
Include the shared endpoint table in the page budget, with JSON punctuation,
and store a new procedure only when its row is accepted.
2026-11-04 20:12:32 +00:00
Cursor Agent 42e8fa871b fix(native): document shared procedure endpoints and the required selector
Describe kind shared and items.endpoints instead of inline procedure copies.
State that procedure-references requires a positional selector or --procedure.
2026-11-04 20:08:07 +00:00
Cursor Agent 1a4986718c merge: page procedure references without repeated scans
Bring the branch onto main. Unfiltered pages use the collection length
and only the requested window. Repeated procedure identities are shared
once per page. The CLI accepts --procedure, and procedure_references
uses the shared analysis Evidence predicate.
2026-11-04 19:57:10 +00:00
Cursor Agent 5c283971bc Merge origin/main into codex/procedure-reference-stacked
Retarget onto main after the feature-trace squash. Keep procedure-reference
pages with the landed feature-trace and batch-decompile view text.
2026-11-04 19:29:08 +00:00
Cursor Agent 7b0f6299bf Merge codex/native-feature-trace-views into codex/procedure-reference-stacked
Keep the procedure-reference description with the updated feature-trace text
and the batch-decompile unavailable status example from main.
2026-11-04 18:11:15 +00:00
Cursor Agent 6ffa15c4d6 Merge origin/main into codex/native-feature-trace-views
Keep the native feature-trace view description and main's batch-decompile
status example that includes unavailable rows.
2026-11-04 18:10:11 +00:00
Cursor Agent 27b90d5d59 ci: rerun pull request checks
The merge of origin/main did not start the CI workflow for this head.
2026-11-04 18:06:30 +00:00
Cursor Agent 090dacd526 Merge remote-tracking branch 'origin/main' into codex/native-feature-trace-views 2026-11-04 17:52:17 +00:00
Cursor Agent b403205338 Merge remote-tracking branch 'origin/main' into codex/procedure-reference-stacked 2026-11-04 17:51:21 +00:00
Cursor Agent 7d956fa9bf Merge branch 'codex/native-feature-trace-views' into codex/procedure-reference-stacked 2026-11-04 17:50:42 +00:00
Cursor Agent 51cfd08d45 fix(cli): report result sources when Evidence publication fails
A trace keeps linked sources on the result. Use those records for the
undelivered source closure as well as the bundle, so a refused output
does not report them as not captured.
2026-11-04 17:50:36 +00:00
Cursor Agent fb5f1ec805 Merge codex/native-feature-trace-views into codex/procedure-reference-stacked
Update the stacked branch with the base that now contains origin/main. Keep
procedure-reference pages together with the landed feature-trace, value-trace,
and batch-decompile views.
2026-11-04 17:11:23 +00:00
Cursor Agent bb73c66baf Merge origin/main into codex/native-feature-trace-views
Keep one-shot Evidence bundle capture and linked source closure together,
including snapshot replay and immutable bundle publication.
2026-11-04 17:08:31 +00:00
morluto 1fc86c45aa docs(mcp): guide trace reference classification 2026-11-04 10:15:18 +08:00
morluto de9d468211 docs(mcp): clarify imported residual questions 2026-11-04 10:02:56 +08:00
morluto ec05b9de27 merge: integrate main into native trace views 2026-11-03 19:31:43 +08:00
morluto 0fabb2f7b8 feat(native): page retained procedure references 2026-11-02 03:19:29 +08:00
morluto 2a0415cd5b test(snapshot): verify linked foreign evidence stays scoped 2026-11-02 02:56:23 +08:00
morluto 673d971381 fix(snapshot): retain linked source evidence for replay 2026-11-02 02:54:41 +08:00
morluto 4809afeef5 feat(native): page retained feature traces 2026-11-02 02:43:25 +08:00
34 changed files with 1894 additions and 230 deletions

No files matched your search

+2 -2
View File
@@ -310,8 +310,8 @@ entry it does not read. The module reads the central directory itself, as REA
does, and shows `zipfile` only the records it lists, reading each entry in
bounded chunks no further than a byte past its declared size. A JSON file
holding a number JavaScript cannot read exactly, such as an integer above 2^53,
or nested more than 500 levels deep, gets an `error` instead of a view, as in
REA, while text that is not one JSON
or whose retained value nests more than 500 levels deep, gets an `error` instead of a view, as in
REA. Duplicate keys keep their last value before the depth check. Text that is not one JSON
document, such as JSON Lines, stays text. Format detection follows REA's
rules. Files larger than 64 MiB get an `error` instead of a view
and are not read. XML is read with Python's `expat` twice: with namespace
+30
View File
@@ -97,6 +97,36 @@ body are excluded. `procedure_references` returns internal and external edges;
The reference collector's semantics participate in the analysis profile, so
older snapshots cannot replay the former incomplete results under this profile.
For a bounded handoff, request `procedure_references` with `detail: "summary"`.
The summary retains the complete observation and reports the selected procedure,
direction and counts. Page it with `inspect_analysis_view`, using the summary's
`normalized_result.parent_evidence_id` and a view such as
`{"kind":"native-procedure-references","collection":"references","source_address":"0x1234","offset":0,"limit":64}`.
The optional exact source and target filters apply before paging. A row's
`source_index` identifies it in the original retained collection, including
duplicates. Endpoint `kind: "selected-procedure"` refers to the full procedure
in the page summary; `kind: "null"` preserves the original null without asserting
why it was unavailable. `kind: "shared"` carries an index into `items.endpoints`,
where each other procedure object is stored once per page. Use collection
`unresolved_calls` for paged unresolved call sites and their original reasons.
Page coverage describes retained rows, not complete provider discovery or runtime
execution. A 1 MiB projection allowance, also used by other bounded analysis views, can
shorten a requested page; `byte_limited: true` and `coverage.next_offset` identify the
continuation. A single row or summary that cannot fit returns a resource error;
retrieve or export the complete parent Evidence. This allowance has no runtime
override. Omit detail to receive the complete observation inline.
For a one-shot CLI handoff, save the complete Evidence and its linked sources:
```bash
rea procedure-references /absolute/path/to/program my_function --provider ghidra \
--direction incoming --detail summary --evidence_output /absolute/path/to/references.json
```
Use `rea inspect-analysis-view` with that bundle as an `evidence-file` source to
page it without reopening the target. This command requires a provider supporting
`procedure_references`; Kuna currently reports that capability unavailable.
Ghidra regex searches use Java Pattern semantics. If compiling or matching a
pattern exhausts the engine's stack, REA returns a resource constraint and
preserves the active database and annotations. Retry in literal mode or simplify
+1
View File
@@ -97,6 +97,7 @@ type DirectAnalysisTool =
| "address_to_file_offset"
| "binary_overview"
| "procedure_pseudo_code"
| "procedure_references"
| "read_function_instructions"
| "inspect_native_instruction"
| "inspect_native_data_type"
+6 -8
View File
@@ -266,12 +266,10 @@ export const registerBinaryDiagnosticsCommands = (
// "0x10" or "1e2".
blockSize: cliDecimalField(
inspectByteEntropyInputSchema.shape.block_size,
).describe(
"Power-of-two block size from 256 to 65536 bytes (default 4096)",
),
).describe("Power-of-two block size from 256 to 65536 bytes"),
offset: cliDecimalField(
inspectByteEntropyInputSchema.shape.offset,
).describe("Window start as a decimal file byte offset (default 0)"),
).describe("Window start as a decimal file byte offset"),
length: cliDecimalField(
inspectByteEntropyInputSchema.shape.length,
).describe("Window length in bytes; omit to extend through EOF"),
@@ -429,16 +427,16 @@ export const registerBinaryDiagnosticsCommands = (
options: z.object({
maxFileBytes: cliDecimalField(
inspectDevicetreeInputSchema.shape.max_file_bytes,
).describe("Maximum FDT file size in bytes (default 4194304)"),
).describe("Maximum FDT file size in bytes"),
maxNodes: cliDecimalField(
inspectDevicetreeInputSchema.shape.max_nodes,
).describe("Maximum number of device-tree nodes (default 8192)"),
).describe("Maximum number of device-tree nodes"),
maxProperties: cliDecimalField(
inspectDevicetreeInputSchema.shape.max_properties,
).describe("Maximum number of device-tree properties (default 32768)"),
).describe("Maximum number of device-tree properties"),
maxDepth: cliDecimalField(
inspectDevicetreeInputSchema.shape.max_depth,
).describe("Maximum device-tree nesting depth (default 64)"),
).describe("Maximum device-tree nesting depth"),
...analysisEvidenceDeliveryOptions,
}),
run: ({ args, options }) =>
+62
View File
@@ -140,6 +140,68 @@ export const registerCoreNativeCommands = (
}),
),
});
cli.command(CLI_COMMANDS.procedureReferences, {
description: `Inspect incoming or outgoing references for one native procedure. ${addressOrProcedureGuidance} ${equalSelectorGuidance}`,
args: z.object({
path: localPathStringSchema.describe("Local executable path"),
procedure: z
.string()
.min(1)
.optional()
.describe(
`Procedure name or provider address. ${addressOrProcedureGuidance}`,
),
}),
options: z.object({
procedure: z
.string()
.min(1)
.optional()
.describe(
`Procedure name or provider address; use --procedure=<value> for leading dashes. ${addressOrProcedureGuidance}`,
),
direction: z
.enum(["incoming", "outgoing"])
.default("outgoing")
.describe("Reference direction. Defaults to outgoing."),
snapshot: snapshotSelectionOption,
...analysisEvidenceDeliveryOptions,
...nativeOpenOptionFields,
}),
run: ({ args, options }) =>
withCommandCancellation((signal) =>
logCliCommand(logger, CLI_COMMANDS.procedureReferences, async () => {
const delivery = {
providerId: "auto",
operation: "procedure_references",
signal,
detail: options.detail,
evidenceOutput: options.evidence_output,
};
const invalid = cliEvidenceDeliveryInputError(delivery);
if (invalid !== undefined) return projectAnalysisError(invalid);
const result = await runDirectAnalysis(
args.path,
"procedure_references",
{
procedure: resolveCliAnalysisSelector(
args.procedure,
options.procedure,
"procedure",
"procedure_references",
),
direction: options.direction,
},
{
...directAnalysisOptionsFromCli(logger, options),
signal,
includeEvidenceSources: options.evidence_output !== undefined,
},
);
return publishCliAnalysisEvidence(result, delivery);
}),
),
});
cli.command(CLI_COMMANDS.readBytes, {
description:
"Read provider memory bytes; Ghidra memory may include loader fixups and differ from file bytes",
+1
View File
@@ -35,6 +35,7 @@ export const CLI_COMMANDS = Object.freeze({
annotateNativeFunction: "annotate-native-function",
inspectNativeApi: "inspect-native-api",
instructions: "instructions",
procedureReferences: "procedure-references",
search: "search",
importReferenceSource: "import-reference-source",
inspectArtifact: "inspect-artifact",
+1 -1
View File
@@ -123,7 +123,7 @@ describe("CLI JSON path normalization", () => {
expect.objectContaining({
path: ["executable_path"],
code: "invalid_format",
message: expect.stringContaining("absolute local filesystem path"),
format: "regex",
}),
]);
});
+1 -1
View File
@@ -17,7 +17,7 @@ export const ANALYSIS_VIEW_TOOL_CONTRACTS = [
...toolContractMetadata("inspect_analysis_view"),
kind: "application",
description:
'For inspect_api_contract parents, graphql-contract pages retained SDL types/fields, operation definitions, field references, fragment-spreads or diagnostics with exact filters and selected columns. Use javascript-routes collections to query route patterns, source-linked occurrences, source status, unknowns or summary counts from retained JavaScript application or web-bundle Evidence. Use web-bundle collections endpoints or inferences to page endpoint observations and inferences from retained analyze_web_bundle Evidence; summary returns target URL, origin, and counts. For retained trace_feature Evidence, kind native-feature-trace pages matches, references or unknowns; matches filter exact address, references filter exact source_address and target_address, source_index identifies each original row, and list_unknowns pages durable question heads. Inspect selected views of supported completed Evidence without repeating analysis. For procedure_pseudo_code Evidence, summary reports text availability and the original selection; kind native with facet pseudocode pages the original UTF-16 text. Other native facets require analyze_function. Choose a view compatible with the parent operation described below. Command summary views preserve ordered GDB records or opaque objdump stdout/stderr once, source parameters, completion facts and parent identity. They do not establish current target state. Historical capture summaries return identity and counts without bodies or archive inventories. Capture views page retained records: view {kind: "capture", detail: "preview" | "complete", offset: 0, limit: 20}; preview selects recorded headline fields and binary descriptors, complete returns exact records with all sidecars. Offsets follow the selected parent record order, preserving original producer ordinals. Paging never reopens the source or reruns the decoder. Native export tables use kind native-exports, collection exports/issues, exact names or original indices, selected fields and matching-row pagination; parent Evidence plus source_index identifies each row. Source coverage stays separate from page coverage. BEAM views use kind beam, collections atoms/exports/imports/lambdas/chunks, exact names/indices/fields and matching-row pages; import modules intersects function names. source_index identifies original parent-and-collection rows, preserving every arity and raw atom index. beam.counts preserves null missing tables versus decoded empty tables; runtime absence remains unknown. PYC views use kind python-pyc, collection code_objects, names/indices/fields and paging; source_index scopes every same-name alternative to the parent, nested_code keeps code-constant references, and pyc.coverage preserves source coverage. Go metadata uses kind go, collection dependencies/settings, exact paths/keys or indices, fields and pagination. Row index refers to the original collection; empty selections return no rows, and dependency coverage remains unknown. Rust metadata uses kind rust, collection packages/edges, exact names/versions or original indices, selected fields, edge from_indices/to_indices and matching-row pagination. Embedded-record coverage is separate from unknown runtime dependencies. For inspect_pcap_capture parents, kind pcap pages endpoints, dns-queries, tls-sni, protocols, http-uris or unattributed-addresses. Select an exact query for dns-addresses, or a tshark/capinfos stream for UTF-16 diagnostics text. Page totals count retained facts; source coverage remains explicit. Qlog summaries keep profile/scope and counts; kind qlog pages traces, streams, loss-declarations, incomplete-records, contributions, ranges or final-sizes. Except traces, select trace_index; stream-specific facets also require exact stream_id and direction. Facts retain all-recorded-events-before-selection scope, and interval overlap or endpoint loss declarations do not prove retransmission or network loss. The api-surface view groups retained HAR requests by method, origin and format-rule path template; templated segments are inferences. It reads no headers or bodies and reports no query values. The environment-manifest view projects the same HAR requests on the primary origin (most JSON-like records) into a replayable tool surface: view {kind: "environment-manifest", template_rules: ["after-uid-literal", "opaque-token", "sibling-variance"], include_query_parameter_names: true}. Each tool has a deterministic name, a path template and parameters from the four format rules plus the selected opt-in rules, observed query parameter names, a value-free body shape for POST/PUT/PATCH, status and content-type sets and record ordinals; records are pointers into the parent, excluded groups carry a reason and three fixed unknowns are always stated. Templates, parameters and names are inferences; the view copies no header, cookie or body values and refuses parents inspected with sensitive values. A capture_browser_scenario parent also supports summary and bLine truncated
'Use native-procedure-references to page retained procedure reference or unresolved-call rows. Endpoint kind selected-procedure refers to the summary procedure, null preserves the original null, and shared carries an index into items.endpoints, where each other procedure is stored once per page. Exact source_address and target_address filters apply to reference rows. For inspect_api_contract parents, graphql-contract pages retained SDL types/fields, operation definitions, field references, fragment-spreads or diagnostics with exact filters and selected columns. Use javascript-routes collections to query route patterns, source-linked occurrences, source status, unknowns or summary counts from retained JavaScript application or web-bundle Evidence. Use web-bundle collections endpoints or inferences to page endpoint observations and inferences from retained analyze_web_bundle Evidence; summary returns target URL, origin, and counts. For retained trace_feature Evidence, kind native-feature-trace pages matches, references or unknowns; matches filter exact address, references filter exact source_address and target_address, source_index identifies each original row, and list_unknowns pages durable question heads. Inspect selected views of supported completed Evidence without repeating analysis. For procedure_pseudo_code Evidence, summary reports text availability and the original selection; kind native with facet pseudocode pages the original UTF-16 text. Other native facets require analyze_function. Choose a view compatible with the parent operation described below. Command summary views preserve ordered GDB records or opaque objdump stdout/stderr once, source parameters, completion facts and parent identity. They do not establish current target state. Historical capture summaries return identity and counts without bodies or archive inventories. Capture views page retained records: view {kind: "capture", detail: "preview" | "complete", offset: 0, limit: 20}; preview selects recorded headline fields and binary descriptors, complete returns exact records with all sidecars. Offsets follow the selected parent record order, preserving original producer ordinals. Paging never reopens the source or reruns the decoder. Native export tables use kind native-exports, collection exports/issues, exact names or original indices, selected fields and matching-row pagination; parent Evidence plus source_index identifies each row. Source coverage stays separate from page coverage. BEAM views use kind beam, collections atoms/exports/imports/lambdas/chunks, exact names/indices/fields and matching-row pages; import modules intersects function names. source_index identifies original parent-and-collection rows, preserving every arity and raw atom index. beam.counts preserves null missing tables versus decoded empty tables; runtime absence remains unknown. PYC views use kind python-pyc, collection code_objects, names/indices/fields and paging; source_index scopes every same-name alternative to the parent, nested_code keeps code-constant references, and pyc.coverage preserves source coverage. Go metadata uses kind go, collection dependencies/settings, exact paths/keys or indices, fields and pagination. Row index refers to the original collection; empty selections return no rows, and dependency coverage remains unknown. Rust metadata uses kind rust, collection packages/edges, exact names/versions or original indices, selected fields, edge from_indices/to_indices and matching-row pagination. Embedded-record coverage is separate from unknown runtime dependencies. For inspect_pcap_capture parents, kind pcap pages endpoints, dns-queries, tls-sni, protocols, http-uris or unattributed-addresses. Select an exact query for dns-addresses, or a tshark/capinfos stream for UTF-16 diagnostics text. Page totals count retained facts; source coverage remains explicit. Qlog summaries keep profile/scope and counts; kind qlog pages traces, streams, loss-declarations, incomplete-records, contributions, ranges or final-sizes. Except traces, select trace_index; stream-specific facets also require exact stream_id and direction. Facts retain all-recorded-events-before-selection scope, and interval overlap or endpoint loss declarations do not prove retransmission or network loss. The api-surface view groups retained HAR requests by method, origin and format-rule path template; templated segments are inferences. It reads no headers or bodies and reports no query values. The environment-manifest view projects the same HAR requests on the primary origin (most JSON-like records) into a replayable tool surface: view {kind: "environment-manifest", template_rules: ["after-uid-literal", "opaque-token", "sibling-variance"], include_query_parameter_names: true}. Each tool has a deterministic name, a path template and parameters from the four format rules plus the selected opt-in rules, observed query parameter names, a value-free body shape for POST/PUT/PATCH, statuLine truncated
inputSchema: inspectAnalysisViewInputSchema,
outputSchema: evidenceResultOf(analysisViewResultSchema),
examples: [
+3 -1
View File
@@ -198,12 +198,14 @@ export const OFFICIAL_TOOL_CONTRACTS = [
),
official(
"procedure_references",
"Return every raw incoming or outgoing reference edge for one procedure. Endpoint procedures are resolved only from provider containment. Ghidra preserves detailed reference flags. Hopper preserves native CallReference classifications and exact endpoints while leaving unavailable detailed flags explicit.",
"Return incoming or outgoing reference edges for one procedure. Ghidra preserves detailed reference flags; Hopper preserves native CallReference classifications and exact endpoints. Select detail summary to retain complete Evidence and page references or unresolved calls using inspect_analysis_view kind native-procedure-references. Page coverage describes retained observations, not complete discovery or proof the target ran.",
z.object({
procedure,
direction: z.enum(["incoming", "outgoing"]).default("outgoing"),
document,
detail: analysisDetailSchema,
}),
{ evidenceParameters: "omit-detail" },
),
official(
"procedure_pseudo_code",
+5 -13
View File
@@ -1,3 +1,4 @@
import { procedureReferencesSchema } from "../domain/native/procedureReferences.js";
import {
nativeAddressListSchema,
nativeObservationSourceSchemas,
@@ -11,6 +12,7 @@ import {
nativeFunctionSummaryViewResultSchema,
nativeInventorySummaryViewResultSchema,
nativeFeatureTraceSummaryViewResultSchema,
nativeProcedureReferencesSummaryViewResultSchema,
batchDecompileSummaryViewResultSchema,
nativeLoadImageSummaryViewResultSchema,
artifactInventorySummaryViewResultSchema,
@@ -33,11 +35,6 @@ import {
analysisBookmarkSchema,
nativeInventoryRowsSchemas,
} from "../domain/native/nativeInventory.js";
import {
referenceEdgeSchema,
unresolvedCallSchema,
} from "../domain/native/nativeReference.js";
import { procedureIdentitySchema } from "../domain/native/procedureInfo.js";
import { z } from "zod";
import {
evidenceBundleExportReceiptSchema,
@@ -229,14 +226,9 @@ export const officialOutputSchemas = {
),
}),
),
procedure_references: evidenceResultOf(
z.object({
procedure: procedureIdentitySchema,
direction: z.enum(["incoming", "outgoing"]),
reference_kinds_available: z.boolean(),
unresolved_calls: z.array(unresolvedCallSchema),
references: z.array(referenceEdgeSchema),
}),
procedure_references: analysisDetailOutputSchemaOf(
procedureReferencesSchema,
nativeProcedureReferencesSummaryViewResultSchema,
),
procedure_pseudo_code: analysisDetailOutputSchemaOf(
nullableText,
+5
View File
@@ -1,3 +1,4 @@
import { projectNativeProcedureReferencesView } from "./nativeProcedureReferencesView.js";
import { projectPlaywrightTraceView } from "./playwrightTraceView.js";
import { projectApiContractView } from "./apiContractView.js";
import {
@@ -153,6 +154,7 @@ import {
il2cppMetadataSummaryViewResultSchema,
nativeInventorySummaryViewResultSchema,
nativeFeatureTraceSummaryViewResultSchema,
nativeProcedureReferencesSummaryViewResultSchema,
nativeLoadImageSummaryViewResultSchema,
nativeFunctionMatchingSummaryViewResultSchema,
nativeFunctionSummaryViewResultSchema,
@@ -229,6 +231,7 @@ export {
modelArtifactPageViewResultSchema,
il2cppMetadataSummaryViewResultSchema,
nativeInventorySummaryViewResultSchema,
nativeProcedureReferencesSummaryViewResultSchema,
nativeLoadImageSummaryViewResultSchema,
nativeFunctionMatchingSummaryViewResultSchema,
nativeFunctionSummaryViewResultSchema,
@@ -335,6 +338,7 @@ const retainedViewProjectors = new Map<string, RetainedViewProjector>([
...NATIVE_OBSERVATION_OPERATIONS.map((operation) =>
retain(operation, projectNativeObservationView),
),
retain("procedure_references", projectNativeProcedureReferencesView),
retain("trace_native_values", projectNativeValueTraceView),
retain("inspect_erlang_beam", projectErlangBeamView),
retain("inspect_devicetree", projectDevicetreeView),
@@ -475,6 +479,7 @@ export const projectAnalysisView = (
parent.operation === "match_native_functions"
? NATIVE_FUNCTION_MATCHING_PREDICATE
: parent.operation === "analyze_function" ||
parent.operation === "procedure_references" ||
parent.operation === "procedure_pseudo_code" ||
parent.operation === "inspect_native_instruction" ||
parent.operation === "trace_feature" ||
@@ -1,3 +1,4 @@
import { nativeProcedureReferencesViewRequestSchema } from "./nativeProcedureReferencesView.js";
import { graphqlContractViewRequestSchema } from "../apiContract/graphqlContractQuery.js";
import { playwrightTraceViewRequestSchema } from "../playwrightTrace/playwrightTraceQuery.js";
import { apiContractViewRequestSchema } from "../apiContract/apiContractQuery.js";
@@ -94,6 +95,7 @@ export const analysisViewRequestSchema = z.discriminatedUnion("kind", [
javascriptFactsViewRequestSchema,
nativeViewSchema,
nativeInventoryViewRequestSchema,
nativeProcedureReferencesViewRequestSchema,
nativeLoadImageViewRequestSchema,
artifactInventoryViewRequestSchema,
wasmViewRequestSchema,
@@ -14,6 +14,11 @@ import {
apiContractViewContextSchema,
apiContractViewItemsSchema,
} from "./apiContractView.js";
import {
procedureReferenceViewRequests,
procedureReferenceViewItems,
procedureReferenceSummarySchema,
} from "./nativeProcedureReferencesView.js";
import {
analysisViewCoverageSchema,
nullableCollectionViewCoverageSchema,
@@ -387,6 +392,7 @@ const viewResultBase = {
...NATIVE_OBSERVATION_OPERATIONS,
"trace_native_values",
"match_native_functions",
"procedure_references",
]),
parent_digest: digestSchema,
view_digest: digestSchema,
@@ -921,6 +927,39 @@ const nativeFeatureTracePageViewResultSchema = z
nativeFeatureTracePage("unknowns"),
]),
);
export const nativeProcedureReferencesSummaryViewResultSchema = z.strictObject({
...summaryViewResultBase,
...nativeViewSourceContext,
parent_operation: z.literal("procedure_references"),
artifact: artifactSchema.nullable(),
summary: procedureReferenceSummarySchema,
});
const procedureReferencePage = <
Collection extends keyof typeof procedureReferenceViewRequests,
>(
collection: Collection,
) =>
z.strictObject({
...viewResultBase,
...nativeViewSourceContext,
kind: z.literal("native-procedure-references"),
parent_operation: z.literal("procedure_references"),
artifact: artifactSchema.nullable(),
view: procedureReferenceViewRequests[collection],
summary: procedureReferenceSummarySchema,
items: procedureReferenceViewItems[collection],
byte_limited: z.boolean(),
});
const procedureReferencePageViewResultSchema = z
.object({ kind: z.literal("native-procedure-references") })
.passthrough()
.pipe(
z.union([
procedureReferencePage("references"),
procedureReferencePage("unresolved_calls"),
]),
);
/** Compact provider load observations retain the independent verification state. */
export const nativeLoadImageSummaryViewResultSchema = z.strictObject({
@@ -1314,6 +1353,7 @@ export const analysisViewResultSchema = z.discriminatedUnion("kind", [
gpuCodeObjectsSummaryViewResultSchema,
nativeInventorySummaryViewResultSchema,
nativeFeatureTraceSummaryViewResultSchema,
nativeProcedureReferencesSummaryViewResultSchema,
nativeLoadImageSummaryViewResultSchema,
nativeFunctionMatchingSummaryViewResultSchema,
nativeFunctionSummaryViewResultSchema,
@@ -1353,6 +1393,7 @@ export const analysisViewResultSchema = z.discriminatedUnion("kind", [
rows: z.array(nativeInventoryViewRowSchema),
}),
nativeFeatureTracePageViewResultSchema,
procedureReferencePageViewResultSchema,
z.strictObject({
...viewResultBase,
kind: z.literal("native-load-image"),
@@ -0,0 +1,141 @@
import { expect, it } from "vitest";
import type { AnalysisViewParent } from "./analysisViewParent.js";
import {
procedureReferenceViewRequests,
projectNativeProcedureReferencesView,
} from "./nativeProcedureReferencesView.js";
const PAGE_BYTES = 1024 * 1024;
const selected = {
address: "0x1000",
name: "selected",
classification: null,
body: { available: false as const, reason: "not reported" },
};
const parentFor = (
references: readonly Record<string, unknown>[],
): AnalysisViewParent => ({
evidenceId: `ev_${"a".repeat(64)}`,
operation: "procedure_references",
predicateType: "rea.analysis",
limitations: [],
parameters: {},
normalizedResult: {
procedure: selected,
direction: "outgoing",
reference_kinds_available: false,
unresolved_calls: [],
references,
},
});
const edge = (
sourceAddress: string,
target: Record<string, unknown> | null,
) => ({
source_address: sourceAddress,
target_address: "0x2000",
source_procedure: selected,
target_procedure: target,
kind: { available: false as const, reason: "not reported" },
});
const viewFor = (limit: number) =>
procedureReferenceViewRequests.references.parse({
kind: "native-procedure-references",
collection: "references",
offset: 0,
limit,
});
const pageBytes = (value: unknown) => Buffer.byteLength(JSON.stringify(value));
it("omits an endpoint that no accepted row uses", () => {
const projected = projectNativeProcedureReferencesView(
parentFor([
edge("0x1001", null),
edge("0x1002", {
...selected,
address: "0x3000",
name: "x".repeat(1_048_000),
}),
]),
viewFor(8),
);
if (!projected.ok) throw projected.error;
if (projected.value.kind !== "native-procedure-references")
throw new Error("Expected a procedure-reference page");
if (projected.value.items.collection !== "references")
throw new Error("Expected reference rows");
expect(projected.value.byte_limited).toBe(true);
expect(projected.value.items.rows).toHaveLength(1);
expect(projected.value.items.endpoints).toEqual([]);
expect(pageBytes(projected.value)).toBeLessThanOrEqual(PAGE_BYTES);
});
it("recognizes a multibyte selected procedure after the page is partly full", () => {
const multibyte = {
...selected,
name: "雪".repeat(200_000),
};
const filler = {
...selected,
address: "0x3000",
name: "y".repeat(150_000),
};
const projected = projectNativeProcedureReferencesView(
{
...parentFor([edge("0x1001", filler), edge("0x1002", multibyte)]),
normalizedResult: {
procedure: multibyte,
direction: "outgoing",
reference_kinds_available: false,
unresolved_calls: [],
references: [
{
...edge("0x1001", filler),
source_procedure: null,
},
{
...edge("0x1002", multibyte),
source_procedure: null,
},
],
},
},
viewFor(8),
);
if (!projected.ok) throw projected.error;
if (projected.value.kind !== "native-procedure-references")
throw new Error("Expected a procedure-reference page");
if (projected.value.items.collection !== "references")
throw new Error("Expected reference rows");
expect(projected.value.items.rows).toHaveLength(2);
expect(projected.value.items.rows[1]?.target_procedure).toEqual({
kind: "selected-procedure",
});
expect(projected.value.items.endpoints).toHaveLength(1);
});
it("counts distinct shared endpoints inside the page allowance", () => {
const projected = projectNativeProcedureReferencesView(
parentFor(
Array.from({ length: 40 }, (_, index) =>
edge(`0x${(0x2000 + index).toString(16)}`, {
...selected,
address: `0x${(0x8000 + index).toString(16)}`,
name: "endpoint".repeat(4_000),
}),
),
),
viewFor(40),
);
if (!projected.ok) throw projected.error;
if (projected.value.kind !== "native-procedure-references")
throw new Error("Expected a procedure-reference page");
if (projected.value.items.collection !== "references")
throw new Error("Expected reference rows");
expect(projected.value.byte_limited).toBe(true);
expect(projected.value.items.endpoints).toHaveLength(
projected.value.items.rows.length,
);
expect(pageBytes(projected.value)).toBeLessThanOrEqual(PAGE_BYTES);
});
@@ -0,0 +1,391 @@
import { z } from "zod";
import { canonicalJson } from "../comparisonSemantics.js";
import { jsonValueSchema } from "../jsonValue.js";
import { procedureIdentitySchema } from "../native/procedureInfo.js";
import {
referenceEdgeSchema,
unresolvedCallSchema,
} from "../native/nativeReference.js";
import { procedureReferencesSchema } from "../native/procedureReferences.js";
import { err, ok } from "../result.js";
import {
createAnalysisViewResultReader,
type AnalysisViewParent,
} from "./analysisViewParent.js";
import {
assertOwnedView,
type AdmittedView,
type SummaryAnalysisView,
} from "./analysisViewShape.js";
import { analysisViewInputError } from "./analysisViewErrors.js";
import { AnalysisResourceConstraintError } from "../analysisErrorCore.js";
import type { JsonValue } from "../jsonValue.js";
import { jsonValueByteLength } from "../jsonValueByteLength.js";
import {
boundedViewMessages,
completeWithinViewCoverage,
pageViewCoverage,
} from "./analysisViewCoverage.js";
const page = {
kind: z.literal("native-procedure-references"),
offset: z.int().nonnegative().default(0),
limit: z.int().positive().default(64),
};
export const procedureReferenceViewRequests = {
references: z.strictObject({
...page,
collection: z.literal("references"),
source_address: z
.string()
.min(1)
.optional()
.describe("Match an exact retained source address."),
target_address: z
.string()
.min(1)
.optional()
.describe("Match an exact retained target address."),
}),
unresolved_calls: z.strictObject({
...page,
collection: z.literal("unresolved_calls"),
}),
} as const;
export const nativeProcedureReferencesViewRequestSchema = z.discriminatedUnion(
"collection",
[
procedureReferenceViewRequests.references,
procedureReferenceViewRequests.unresolved_calls,
],
);
const endpointSchema = z.discriminatedUnion("kind", [
z.strictObject({ kind: z.literal("null") }),
z.strictObject({ kind: z.literal("selected-procedure") }),
z.strictObject({
kind: z.literal("shared"),
index: z.int().nonnegative(),
}),
]);
const rowSchema = referenceEdgeSchema.extend({
source_procedure: endpointSchema,
target_procedure: endpointSchema,
source_index: z.int().nonnegative(),
});
export const procedureReferenceViewItems = {
references: z.strictObject({
collection: z.literal("references"),
endpoints: z.array(procedureIdentitySchema),
rows: z.array(rowSchema),
}),
unresolved_calls: z.strictObject({
collection: z.literal("unresolved_calls"),
rows: z.array(
z.strictObject({
source_index: z.int().nonnegative(),
value: unresolvedCallSchema,
}),
),
}),
} as const;
export const procedureReferenceSummarySchema = z.strictObject({
procedure: procedureIdentitySchema,
direction: z.enum(["incoming", "outgoing"]),
reference_kinds_available: z.boolean(),
reference_count: z.int().nonnegative(),
unresolved_call_count: z.int().nonnegative(),
});
const readSource = createAnalysisViewResultReader(procedureReferencesSchema);
// The analysis-view projection allowance leaves room for Evidence and MCP
// encoding expansion. Count JSON without serializing the full page.
const PAGE_BYTES = 1024 * 1024;
const DIGEST_BYTES = jsonValueByteLength({ view_digest: "0".repeat(64) }) - 1;
/** Page the retained result without re-querying the provider or changing source coverage. */
export const projectNativeProcedureReferencesView = (
parent: AnalysisViewParent,
view: AdmittedView,
) => {
if (view.kind !== "summary" && view.kind !== "native-procedure-references")
return err(
analysisViewInputError([
{
path: ["view"],
reason: "invalid_value",
message:
"Procedure-reference Evidence supports summary and native-procedure-references views.",
},
]),
);
assertOwnedView<
| SummaryAnalysisView
| z.output<typeof nativeProcedureReferencesViewRequestSchema>
>(view, "summary", "native-procedure-references");
const parsed = readSource(parent);
if (!parsed.ok) return parsed;
const source = parsed.value;
const summary = {
procedure: source.procedure,
direction: source.direction,
reference_kinds_available: source.reference_kinds_available,
reference_count: source.references.length,
unresolved_call_count: source.unresolved_calls.length,
};
const shared = {
parent_evidence_id: parent.evidenceId,
parent_operation: "procedure_references" as const,
parent_digest: parent.evidenceId.slice(3),
artifact: parent.artifact ?? null,
source_provider: parent.provider ?? null,
source_analysis_profile_digest: parent.analysisProfile?.digest ?? null,
source_parameters: jsonValueSchema.parse(parent.parameters ?? null),
limitations: boundedViewMessages(
parent.limitations,
parent.evidenceId,
"limitations",
),
unknowns: [
"Page coverage describes retained references, not provider discovery completeness or runtime execution.",
],
};
const tooLarge = () =>
err(
new AnalysisResourceConstraintError(
"inspect_analysis_view",
"transport",
"The selected procedure-reference view cannot fit the analysis-view byte budget.",
{ parent_evidence_id: parent.evidenceId, limit_bytes: PAGE_BYTES },
{
retryable: false,
remediationAction:
"An individual row or the page context exceeds the fixed 1,048,576-byte projection allowance. Use the complete input Evidence or its local bundle, or retrieve or export the retained parent by the reported ID. If a CLI failure includes a partial observation, that observation is the completed parent.",
},
),
);
if (view.kind === "summary") {
const result = {
...shared,
kind: "summary" as const,
view,
summary,
coverage: completeWithinViewCoverage(1, 1),
};
return jsonValueByteLength(result as JsonValue, PAGE_BYTES - DIGEST_BYTES) >
PAGE_BYTES - DIGEST_BYTES
? tooLarge()
: ok(result);
}
const common = {
...shared,
kind: "native-procedure-references" as const,
view,
summary,
};
const header = {
...common,
items:
view.collection === "references"
? { collection: view.collection, endpoints: [], rows: [] }
: { collection: view.collection, rows: [] },
byte_limited: false,
coverage: {
status: "complete-within-view",
examined: Number.MAX_SAFE_INTEGER,
total: Number.MAX_SAFE_INTEGER,
next_offset: Number.MAX_SAFE_INTEGER,
exhausted: false,
},
};
let bytes =
DIGEST_BYTES +
jsonValueByteLength(header as JsonValue, PAGE_BYTES - DIGEST_BYTES);
if (bytes > PAGE_BYTES) return tooLarge();
const referenceRows: z.infer<typeof rowSchema>[] = [];
const unresolvedRows: {
source_index: number;
value: z.infer<typeof unresolvedCallSchema>;
}[] = [];
let byteLimited = false;
const offer = (row: JsonValue, accepted: number, extra = 0) => {
const size =
jsonValueByteLength(row, PAGE_BYTES - bytes) +
(accepted > 0 ? 1 : 0) +
extra;
if (bytes + size > PAGE_BYTES) {
byteLimited = true;
return false;
}
bytes += size;
return true;
};
if (view.collection === "unresolved_calls") {
const calls = source.unresolved_calls;
const end = Math.min(calls.length, view.offset + view.limit);
for (let index = view.offset; index < end; index += 1) {
const value = calls[index];
if (value === undefined) break;
const row = { source_index: index, value };
if (!offer(row, unresolvedRows.length)) break;
unresolvedRows.push(row);
}
if (byteLimited && unresolvedRows.length === 0) return tooLarge();
return ok({
...common,
view,
items: { collection: "unresolved_calls" as const, rows: unresolvedRows },
byte_limited: byteLimited,
coverage: pageViewCoverage(
view.offset,
unresolvedRows.length,
calls.length,
),
});
}
const selected = canonicalJson(source.procedure);
const endpoints: z.infer<typeof procedureIdentitySchema>[] = [];
const endpointIndexes = new Map<string, number>();
let longestIdentityBytes = Buffer.byteLength(selected);
const planEndpoints = (
sourceProcedure: z.infer<typeof procedureIdentitySchema> | null,
targetProcedure: z.infer<typeof procedureIdentitySchema> | null,
) => {
const pending: z.infer<typeof procedureIdentitySchema>[] = [];
const pendingKeys: string[] = [];
let extra = 0;
let overflow = false;
const resolve = (value: z.infer<typeof procedureIdentitySchema> | null) => {
if (value === null) return { kind: "null" as const };
const comma = endpoints.length + pending.length > 0 ? 1 : 0;
const measured = jsonValueByteLength(
value as JsonValue,
Math.max(0, PAGE_BYTES - bytes - extra),
);
const size = measured + comma;
// `measured` is a UTF-8 lower bound. It can skip canonicalization only
// when it already exceeds every stored identity's UTF-8 length.
if (
bytes + extra + size > PAGE_BYTES &&
measured > longestIdentityBytes
) {
overflow = true;
return undefined;
}
const key = canonicalJson(value);
if (key === selected) return { kind: "selected-procedure" as const };
const existing = endpointIndexes.get(key);
if (existing !== undefined)
return { kind: "shared" as const, index: existing };
const pendingIndex = pendingKeys.indexOf(key);
if (pendingIndex !== -1)
return {
kind: "shared" as const,
index: endpoints.length + pendingIndex,
};
if (bytes + extra + size > PAGE_BYTES) {
overflow = true;
return undefined;
}
extra += size;
pending.push(value);
pendingKeys.push(key);
const keyBytes = Buffer.byteLength(key);
if (keyBytes > longestIdentityBytes) longestIdentityBytes = keyBytes;
return {
kind: "shared" as const,
index: endpoints.length + pending.length - 1,
};
};
const resolvedSource = resolve(sourceProcedure);
const resolvedTarget = resolve(targetProcedure);
if (
overflow ||
resolvedSource === undefined ||
resolvedTarget === undefined
)
return { overflow: true as const };
return {
overflow: false as const,
resolvedSource,
resolvedTarget,
pending,
pendingKeys,
extra,
};
};
const filtered =
view.source_address !== undefined || view.target_address !== undefined;
const references = source.references;
const visit = function* () {
if (!filtered) {
const end = Math.min(references.length, view.offset + view.limit);
for (let index = view.offset; index < end; index += 1) {
const reference = references[index];
if (reference !== undefined) yield [index, reference] as const;
}
return;
}
let seen = 0;
for (const [index, reference] of references.entries()) {
if (
(view.source_address !== undefined &&
reference.source_address !== view.source_address) ||
(view.target_address !== undefined &&
reference.target_address !== view.target_address)
)
continue;
if (seen >= view.offset && seen < view.offset + view.limit)
yield [index, reference] as const;
seen += 1;
}
};
let filteredTotal = 0;
if (filtered) {
for (const reference of references) {
if (
(view.source_address === undefined ||
reference.source_address === view.source_address) &&
(view.target_address === undefined ||
reference.target_address === view.target_address)
)
filteredTotal += 1;
}
}
for (const [index, reference] of visit()) {
const planned = planEndpoints(
reference.source_procedure,
reference.target_procedure,
);
if (planned.overflow) {
byteLimited = true;
break;
}
const row = {
...reference,
source_index: index,
source_procedure: planned.resolvedSource,
target_procedure: planned.resolvedTarget,
};
if (!offer(row as JsonValue, referenceRows.length, planned.extra)) break;
for (const [offset, value] of planned.pending.entries()) {
endpointIndexes.set(planned.pendingKeys[offset]!, endpoints.length);
endpoints.push(value);
}
referenceRows.push(row);
}
if (byteLimited && referenceRows.length === 0) return tooLarge();
return ok({
...common,
view,
items: {
collection: "references" as const,
endpoints,
rows: referenceRows,
},
byte_limited: byteLimited,
coverage: pageViewCoverage(
view.offset,
referenceRows.length,
filtered ? filteredTotal : references.length,
),
});
};
+1 -1
View File
@@ -51,7 +51,7 @@ const captureInputShape = z.strictObject({
.int()
.nonnegative()
.default(0)
.describe("Skip this many matching packets after filtering; default 0"),
.describe("Skip this many matching packets after filtering"),
match_limit: z
.number()
.int()
+2 -4
View File
@@ -5,9 +5,7 @@ export const inspectPacketFieldsInputSchema = z.strictObject({
kind: z
.enum(["protocols", "fields"])
.default("protocols")
.describe(
"Return protocol registrations or field definitions; defaults to protocols",
),
.describe("Return protocol registrations or field definitions"),
protocol: z
.string()
.min(1)
@@ -30,7 +28,7 @@ export const inspectPacketFieldsInputSchema = z.strictObject({
.nonnegative()
.max(Number.MAX_SAFE_INTEGER)
.default(0)
.describe("Number of matching records to skip; defaults to 0"),
.describe("Number of matching records to skip"),
limit: z
.number()
.int()
+3 -142
View File
@@ -1,3 +1,4 @@
import { PYTHON_JSON_READER } from "./appExportPythonJson.js";
import { PYTHON_XML_READER } from "./appExportPythonXml.js";
import { PYTHON_ZIP_HELPERS } from "./appExportPythonZip.js";
@@ -12,6 +13,7 @@ export const PYTHON_READY_CHECK = `
export const PYTHON_IMPORTS = `from __future__ import annotations
from array import array
import contextlib
import copy
import hashlib
@@ -57,11 +59,6 @@ MAX_PARSED_BYTES = 4 * 1024 * 1024
PARSED_TOO_LARGE = "larger than the 4 MiB limit for JSON and key file views"
MAX_JSON_VIEW_DEPTH = 500
JSON_TOO_DEEP = f"nested deeper than the {MAX_JSON_VIEW_DEPTH} levels a JSON view keeps"
#: A JSON token after any whitespace, by group: an opening, a closing, a comma, a colon, a string's quote, or a number
#: or literal. Strings are read by json's scanner, since a pattern repeating per escape holds memory for each.
JSON_TOKEN = re.compile(r'[ \t\n\r]*(?:([\[{])|([\]}])|(,)|(:)|(")|(-?(?:0|[1-9][0-9]*)(?:\.[0-9]+)?(?:[eE][+-]?[0-9]+)?'
r"|true|false|null))")
JSON_SPACE = re.compile(r"[ \t\n\r]*")
def _respelled(text: str, home: Path) -> str:
@@ -254,143 +251,7 @@ def _keyfile(text: str) -> dict:
${PYTHON_XML_READER}
def _not_json(constant: str):
raise ValueError(f"{constant} is not JSON")
class _Inexact(Exception):
"""A JSON number that REA's state view refuses, since reading it would change it."""
def _digits(text: str) -> str:
return re.split(r"[eE]", text.lstrip("-"))[0].replace(".", "").strip("0") or "0"
def _js_string(value: float) -> str:
"""How JavaScript's String() writes a finite double: an integer below 1e21 in full, any other in shortest form."""
return str(int(value)) if value.is_integer() and abs(value) < 1e21 else repr(value)
def _safe_number(text: str) -> bool:
"""lossless-json's isSafeNumber: an integer spelling within 2^53, or a float spelling that keeps its digits."""
if re.fullmatch(r"-?[0-9]+", text):
return len(text.lstrip("-")) <= 16 and abs(int(text)) <= MAX_SAFE_INTEGER
value = float(text)
if not math.isfinite(value):
return False
written = _js_string(value)
return written == text or _digits(written) == _digits(text)
def _exact_number(text: str):
"""A JSON number read only when reading it changes nothing, as REA's state view rules: its spelling and its
written-back form both safe, and not -0."""
value = float(text)
if (not _safe_number(text) or not math.isfinite(value) or not _safe_number(_js_string(value))
or (value == 0 and math.copysign(1, value) < 0)):
raise _Inexact(text)
return int(text) if re.fullmatch(r"-?[0-9]+", text) else value
_INEXACT = object()
_LEVEL_END = object()
def _number_or_mark(spelling: str):
"""A JSON number read exactly, or a mark for one REA's state view refuses."""
try:
return _exact_number(spelling)
except _Inexact:
return _INEXACT
def _holds_inexact(value) -> bool:
"""Whether a parsed JSON value still holds a marked number once duplicate keys kept their last value; one
iterator per nesting level, so no child list is copied."""
levels = [iter((value,))]
while levels:
item = next(levels[-1], _LEVEL_END)
if item is _LEVEL_END:
levels.pop()
elif item is _INEXACT:
return True
elif isinstance(item, dict):
levels.append(iter(item.values()))
elif isinstance(item, list):
levels.append(iter(item))
return False
def _deeper_than(root, limit: int) -> bool:
"""Whether a parsed JSON value nests deeper than a limit, as REA checks it: a value's depth is the number of
iterators open above it, one per nesting level, so no child list is copied."""
levels = [iter((root,))]
while levels:
item = next(levels[-1], _LEVEL_END)
if item is _LEVEL_END:
levels.pop()
elif len(levels) - 1 > limit:
return True
elif isinstance(item, dict):
levels.append(iter(item.values()))
elif isinstance(item, list):
levels.append(iter(item))
return False
def _is_json(text: str) -> bool:
"""Whether text is one JSON value as REA's state view reads it, checked a token at a time without building it or
recursing: it holds a byte per open container and one string at a time, so text too deep for Python's parser or
too large to parse into a view is still told apart from other text."""
containers, expect, empty, at = bytearray(), "value", False, 0
while True:
token = JSON_TOKEN.match(text, at)
if token is None:
break
kind, at = token.lastindex, token.end()
if kind == 5:
try:
at = json.decoder.scanstring(text, at)[1]
except ValueError:
return False
if kind == 1 and expect == "value":
containers.append(ord(text[at - 1]))
expect, empty = ("value" if text[at - 1] == "[" else "key"), True
# A closing bracket's code is its opening's plus two.
elif kind == 2 and (expect == "after" or empty) and containers and containers[-1] == ord(text[at - 1]) - 2:
containers.pop()
expect, empty = "after", False
elif kind == 3 and expect == "after" and containers:
expect = "value" if containers[-1] == ord("[") else "key"
elif kind == 4 and expect == "colon":
expect = "value"
elif kind >= 5 and (expect == "value" or (kind == 5 and expect == "key")):
expect, empty = ("colon" if expect == "key" else "after"), False
else:
return False
return expect == "after" and not containers and JSON_SPACE.fullmatch(text, at) is not None
def _json_view(text: str, size: int):
"""A JSON text's format and view, or why it has none, as REA reads it; None when it is not JSON. Text over the
parsed-view limit, or nested past Python's recursion limit, is told apart without building its value."""
if size > MAX_PARSED_BYTES:
return {"format": "json", "view": None, "error": PARSED_TOO_LARGE} if _is_json(text) else None
# The whole text must parse, and duplicate keys settle, before an inexact number makes it JSON with an error.
try:
view = json.loads(text, parse_constant=_not_json, parse_float=_number_or_mark, parse_int=_number_or_mark)
except RecursionError:
# Nesting too deep for Python to parse is far deeper than REA's view keeps, once the rest of it is JSON too.
return {"format": "json", "view": None, "error": JSON_TOO_DEEP} if _is_json(text) else None
except ValueError:
return None
if _holds_inexact(view):
return {"format": "json", "view": None,
"error": "a number in it cannot be read exactly, such as an integer above 2^53"}
if _deeper_than(view, MAX_JSON_VIEW_DEPTH):
return {"format": "json", "view": None, "error": JSON_TOO_DEEP}
return {"format": "json", "view": view, "truncated": False}
${PYTHON_JSON_READER}
def _text_view(name: str, text: str, size: int) -> dict:
"""The format REA's state view detects for decoded text of size bytes, and its view; a JSON or key file over the
+231
View File
@@ -0,0 +1,231 @@
/** JSON parsing and bounded retained-view checks for exported desktop environments. */
export const PYTHON_JSON_READER = String.raw`
#: A JSON token after any whitespace, by group: an opening, a closing, a comma, a colon, a string's quote, or a number
#: or literal. Strings are read by json's scanner, since a pattern repeating per escape holds memory for each.
JSON_TOKEN = re.compile(r'[ \t\n\r]*(?:([\[{])|([\]}])|(,)|(:)|(")|(-?(?:0|[1-9][0-9]*)(?:\.[0-9]+)?(?:[eE][+-]?[0-9]+)?'
r"|true|false|null))")
JSON_SPACE = re.compile(r"[ \t\n\r]*")
class _Inexact(Exception):
"""A JSON number that REA's state view refuses, since reading it would change it."""
def _digits(text: str) -> str:
return re.split(r"[eE]", text.lstrip("-"))[0].replace(".", "").strip("0") or "0"
def _js_string(value: float) -> str:
"""How JavaScript's String() writes a finite double: an integer below 1e21 in full, any other in shortest form."""
return str(int(value)) if value.is_integer() and abs(value) < 1e21 else repr(value)
def _safe_number(text: str) -> bool:
"""lossless-json's isSafeNumber: an integer spelling within 2^53, or a float spelling that keeps its digits."""
if re.fullmatch(r"-?[0-9]+", text):
return len(text.lstrip("-")) <= 16 and abs(int(text)) <= MAX_SAFE_INTEGER
value = float(text)
if not math.isfinite(value):
return False
written = _js_string(value)
return written == text or _digits(written) == _digits(text)
def _exact_number(text: str):
"""A JSON number read only when reading it changes nothing, as REA's state view rules: its spelling and its
written-back form both safe, and not -0."""
value = float(text)
if (not _safe_number(text) or not math.isfinite(value) or not _safe_number(_js_string(value))
or (value == 0 and math.copysign(1, value) < 0)):
raise _Inexact(text)
return int(text) if re.fullmatch(r"-?[0-9]+", text) else value
_INEXACT = object()
_DEEP = object()
_DEEP_INEXACT = object()
_LEVEL_END = object()
_INVALID_JSON = object()
JSON_ATOMS = {"true": True, "false": False, "null": None}
def _number_or_mark(spelling: str):
"""A JSON number read exactly, or a mark for one REA's state view refuses."""
try:
return _exact_number(spelling)
except _Inexact:
return _INEXACT
def _holds_inexact(value) -> bool:
"""Whether a parsed JSON value still holds a marked number once duplicate keys kept their last value; one
iterator per nesting level, so no child list is copied."""
levels = [iter((value,))]
while levels:
item = next(levels[-1], _LEVEL_END)
if item is _LEVEL_END:
levels.pop()
elif item is _INEXACT or item is _DEEP_INEXACT:
return True
elif isinstance(item, dict):
levels.append(iter(item.values()))
elif isinstance(item, list):
levels.append(iter(item))
return False
def _deeper_than(root, limit: int) -> bool:
"""Whether a parsed JSON value nests deeper than a limit, as REA checks it: a value's depth is the number of
iterators open above it, one per nesting level, so no child list is copied."""
levels = [iter((root,))]
while levels:
item = next(levels[-1], _LEVEL_END)
if item is _LEVEL_END:
levels.pop()
elif item is _DEEP or item is _DEEP_INEXACT:
return True
elif len(levels) - 1 > limit:
return True
elif isinstance(item, dict):
levels.append(iter(item.values()))
elif isinstance(item, list):
levels.append(iter(item))
return False
def _read_json(text: str, build: bool = True):
"""One JSON value, or _INVALID_JSON, read without recursion so duplicate keys settle before depth is checked.
Deep containers are summarized rather than built; exact-number marks still follow last-key-wins semantics.
With build=False, only syntax is checked: one byte per open container and one decoded string at a time."""
containers, expect, empty, at = bytearray(), "value", False, 0
values, keys, root = [], [], _INVALID_JSON
deep_bad, deep_key_at, bad_keys = bytearray(), array("I"), {}
def keep(value) -> None:
nonlocal root
if not values:
root = value
elif containers[-1] == ord("["):
values[-1].append(value)
else:
values[-1][keys[-1]] = value
def record_deep(bad: bool) -> None:
"""Summarize a finished child; keep only object keys with live inexact values."""
depth = len(deep_bad) - 1
if containers[-1] == ord("["):
deep_bad[-1] |= 2 | bad
return
marked = bad_keys.get(depth)
if bad or marked is not None:
key, _ = json.decoder.scanstring(text, deep_key_at[-1])
if bad:
if marked is None:
marked = key
elif isinstance(marked, str):
if marked != key:
marked = {marked, key}
else:
marked.add(key)
else:
if isinstance(marked, str):
if marked == key:
marked = None
else:
marked.discard(key)
if len(marked) == 1:
marked = next(iter(marked))
if marked is None:
bad_keys.pop(depth, None)
else:
bad_keys[depth] = marked
deep_bad[-1] = 2 | (marked is not None)
else:
deep_bad[-1] |= 2
while True:
token = JSON_TOKEN.match(text, at)
if token is None:
break
kind, at = token.lastindex, token.end()
if kind == 5:
string_at = at
try:
value, at = json.decoder.scanstring(text, at)
except ValueError:
return _INVALID_JSON
if kind == 1 and expect == "value":
if build:
if deep_bad or len(containers) >= MAX_JSON_VIEW_DEPTH:
deep_bad.append(0)
deep_key_at.append(0)
else:
value = [] if text[at - 1] == "[" else {}
keep(value)
values.append(value)
keys.append(None)
containers.append(ord(text[at - 1]))
expect, empty = ("value" if text[at - 1] == "[" else "key"), True
# A closing bracket's code is its opening's plus two.
elif kind == 2 and (expect == "after" or empty) and containers and containers[-1] == ord(text[at - 1]) - 2:
opening = containers.pop()
if build:
if deep_bad:
state = deep_bad.pop()
bad = bool(state & 1)
deep_key_at.pop()
bad_keys.pop(len(deep_bad), None)
if deep_bad:
record_deep(bad)
else:
keep((_DEEP_INEXACT if bad else _DEEP) if state & 2 else ([] if opening == ord("[") else {}))
else:
values.pop()
keys.pop()
expect, empty = "after", False
elif kind == 3 and expect == "after" and containers:
expect = "value" if containers[-1] == ord("[") else "key"
elif kind == 4 and expect == "colon":
expect = "value"
elif kind >= 5 and (expect == "value" or (kind == 5 and expect == "key")):
if build:
if expect == "key":
if deep_bad:
deep_key_at[-1] = string_at
else:
keys[-1] = value
else:
if kind == 6:
value = JSON_ATOMS.get(token[6], _INVALID_JSON)
if value is _INVALID_JSON:
value = _number_or_mark(token[6])
if deep_bad:
record_deep(value is _INEXACT)
else:
keep(value)
expect, empty = ("colon" if expect == "key" else "after"), False
else:
return _INVALID_JSON
if expect != "after" or containers or JSON_SPACE.fullmatch(text, at) is None:
return _INVALID_JSON
return root if build else True
def _json_view(text: str, size: int):
"""A JSON text's format and view, or why it has none, as REA reads it; None when it is not JSON. Text over the
parsed-view limit is told apart without building its value."""
if size > MAX_PARSED_BYTES:
valid = _read_json(text, build=False) is not _INVALID_JSON
return {"format": "json", "view": None, "error": PARSED_TOO_LARGE} if valid else None
# The whole text must parse, and duplicate keys settle, before an inexact number makes it JSON with an error.
view = _read_json(text)
if view is _INVALID_JSON:
return None
if _holds_inexact(view):
return {"format": "json", "view": None,
"error": "a number in it cannot be read exactly, such as an integer above 2^53"}
if _deeper_than(view, MAX_JSON_VIEW_DEPTH):
return {"format": "json", "view": None, "error": JSON_TOO_DEEP}
return {"format": "json", "view": view, "truncated": False}
`;
+20 -1
View File
@@ -21,11 +21,20 @@ const PATHS = [
];
it("matches the host path module for absolute local paths", () => {
const checked: string[] = [];
for (const path of PATHS) {
checked.push(path);
expect(absoluteLocalPathStringSchema.safeParse(path).success).toBe(
isAbsolute(path),
);
}
expect(checked).toEqual(
expect.arrayContaining([
"/absolute/path",
"relative/path",
"file:///absolute/path",
]),
);
expect(absoluteLocalPathStringSchema.safeParse("").success).toBe(false);
expect(
absoluteLocalPathStringSchema.safeParse("/absolute/with\0nul").success,
@@ -40,8 +49,18 @@ it("keeps the Windows absolute-path pattern valid in every client mode", () => {
() => new RegExp(windowsAbsolutePathPattern.source, flags),
flags,
).not.toThrow();
for (const path of PATHS)
const checked: string[] = [];
for (const path of PATHS) {
checked.push(path);
expect(windowsAbsolutePathPattern.test(path), path).toBe(
win32.isAbsolute(path),
);
}
expect(checked).toEqual(
expect.arrayContaining([
"C:\\absolute\\path",
"C:relative/path",
"/absolute/path",
]),
);
});
+15
View File
@@ -0,0 +1,15 @@
import { z } from "zod";
import { procedureIdentitySchema } from "./procedureInfo.js";
import {
referenceEdgeSchema,
unresolvedCallSchema,
} from "./nativeReference.js";
/** Complete provider-observed references and unresolved calls for one procedure. */
export const procedureReferencesSchema = z.object({
procedure: procedureIdentitySchema,
direction: z.enum(["incoming", "outgoing"]),
reference_kinds_available: z.boolean(),
unresolved_calls: z.array(unresolvedCallSchema),
references: z.array(referenceEdgeSchema),
});
+1 -3
View File
@@ -24,9 +24,7 @@ export const inspectSignalDecodersInputSchema = z.strictObject({
.nonnegative()
.max(Number.MAX_SAFE_INTEGER)
.default(0)
.describe(
"Number of matching decoders or help lines to skip; defaults to 0",
),
.describe("Number of matching decoders or help lines to skip"),
limit: z
.number()
.int()
+1 -1
View File
@@ -31,7 +31,7 @@ const signalInputShape = z.strictObject({
.int()
.nonnegative()
.default(0)
.describe("Skip this many literal LF-separated output lines; default 0"),
.describe("Skip this many literal LF-separated output lines"),
line_limit: z
.number()
.int()
@@ -255,11 +255,11 @@ cliTest(
code: "invalid_request",
details: {
issues: [
{
expect.objectContaining({
path: ["executable_path"],
reason: "invalid_format",
message: expect.stringContaining("absolute local filesystem path"),
},
expected: "regex",
}),
],
},
});
@@ -4,6 +4,7 @@ import { join } from "node:path";
import { fileURLToPath } from "node:url";
import { promisify } from "node:util";
import { expect, it } from "vitest";
import { z } from "zod";
import { toolContract } from "../../../src/contracts/toolContracts.js";
import { createEvidence, parseEvidence } from "../../../src/domain/evidence.js";
@@ -329,21 +330,84 @@ const pathRequests: readonly [string, Record<string, unknown>][] = [
["start_gdb_session", { inferior_tty: nul }],
];
const nulIssuePaths = (
value: unknown,
path: readonly (string | number)[] = [],
): (string | number)[][] => {
if (typeof value === "string") return value.includes("\0") ? [[...path]] : [];
if (Array.isArray(value))
return value.flatMap((item, index) =>
nulIssuePaths(item, [...path, index]),
);
if (typeof value === "object" && value !== null)
return Object.entries(value).flatMap(([key, item]) =>
nulIssuePaths(item, [...path, key]),
);
return [];
};
it("rejects OS-invalid local inputs through the real advertised MCP contracts", async () => {
const { call } = await connectLocalToolsMcp();
const checked: string[] = [];
for (const [name, arguments_] of pathRequests) {
const response = await call(name, arguments_);
expect(response.isError, name).toBe(true);
expect(JSON.stringify(response.content), name).toContain("NUL");
const paths = nulIssuePaths(arguments_);
expect(paths.length, name).toBeGreaterThan(0);
const error = parseMcpToolError(await call(name, arguments_)).error;
expect(error, name).toMatchObject({
code: "invalid_request",
details: { operation: name },
});
const issues = z
.object({
issues: z.array(
z.object({
path: z.array(z.union([z.string(), z.number()])),
reason: z.string(),
message: z.string(),
expected: z.unknown().optional(),
}),
),
})
.parse(error.details).issues;
expect(issues, name).toEqual(
expect.arrayContaining(
paths.map((path) =>
expect.objectContaining({
path,
reason: "invalid_format",
expected: "regex",
message: expect.stringMatching(/\bNUL\b|null byte|zero byte/i),
}),
),
),
);
checked.push(name);
}
expect(checked).toEqual(
expect.arrayContaining([
"extract_firmware",
"inspect_wasm_artifact",
"demangle_swift",
"capture_electron_scenario",
]),
);
const relative = await call("extract_firmware", {
path: "/tmp/firmware.bin",
output_directory: "relative-output",
});
expect(relative.isError).toBe(true);
expect(JSON.stringify(relative.content)).toContain(
"absolute local filesystem path",
);
expect(parseMcpToolError(relative).error).toMatchObject({
code: "invalid_request",
details: {
operation: "extract_firmware",
issues: [
expect.objectContaining({
path: ["output_directory"],
reason: "invalid_format",
expected: "regex",
}),
],
},
});
}, 60_000);
const ignoredOptions = [
@@ -0,0 +1,87 @@
import { describe, expect, it } from "vitest";
import { createCli } from "../../../src/cli.js";
type CliInstance = ReturnType<typeof createCli>;
const serve = async (
cli: CliInstance,
argv: readonly string[],
): Promise<{ readonly stdout: string; readonly exitCode: number }> => {
let stdout = "";
let exitCode = 0;
await cli.serve([...argv], {
env: {},
exit: (code) => {
exitCode = code;
},
stdout: (text) => {
stdout += text;
},
});
return { stdout, exitCode };
};
/**
* Options whose owning field already declares a schema default. The help
* renderer appends that default, so the description must not restate it.
*/
const singleDefaultOptions: readonly {
readonly command: string;
readonly options: readonly string[];
}[] = [
{ command: "inspect-byte-entropy", options: ["--block-size", "--offset"] },
{
command: "inspect-devicetree",
options: [
"--max-file-bytes",
"--max-nodes",
"--max-properties",
"--max-depth",
],
},
{
command: "decode-signal-recording",
options: ["--line_offset", "--line_limit"],
},
{
command: "inspect-packet-capture",
options: ["--match_offset", "--match_limit"],
},
{
command: "inspect-packet-fields",
options: ["--kind", "--offset"],
},
{ command: "inspect-signal-decoders", options: ["--offset"] },
];
const optionLine = (help: string, option: string): string | undefined => {
const line = help
.split(/\r?\n/u)
.find((candidate) => candidate.trimStart().startsWith(`${option} `));
return line?.trim();
};
describe("CLI help default annotations", () => {
it("states each schema-owned option default exactly once", async () => {
const cli = createCli({});
const repeated: string[] = [];
for (const { command, options } of singleDefaultOptions) {
const { stdout, exitCode } = await serve(cli, [command, "--help"]);
expect(exitCode, command).toBe(0);
for (const option of options) {
const line = optionLine(stdout, option);
if (line === undefined) {
repeated.push(`${command} ${option}: option line missing`);
continue;
}
const statements = line.match(/default/giu)?.length ?? 0;
if (statements !== 1)
repeated.push(
`${command} ${option}: ${statements} default statements`,
);
}
}
expect(repeated).toEqual([]);
});
});
@@ -50,7 +50,6 @@ const MCP_TOOLS_WITHOUT_DEDICATED_CLI = Object.freeze([
"procedure_callees",
"procedure_callers",
"procedure_info",
"procedure_references",
"resolve_containing_procedure",
"set_address_name",
"set_addresses_names",
@@ -0,0 +1,153 @@
import { readFile, stat, writeFile } from "node:fs/promises";
import { join } from "node:path";
import { Cli } from "incur";
import pino from "pino";
import { expect, it } from "vitest";
import {
createAnalysisExecution,
type AnalysisProvider,
} from "../../../src/application/AnalysisProvider.js";
import { runDirectAnalysis } from "../../../src/application/DirectAnalysis.js";
import type { DirectAnalysisDependencies } from "../../../src/application/DirectAnalysisDependencies.js";
import { registerCoreNativeCommands } from "../../../src/cli/coreNativeCommands.js";
import { parseConfig } from "../../../src/config/parseConfig.js";
import { createAnalysisProfile } from "../../../src/domain/analysisProfile.js";
import { createEvidence } from "../../../src/domain/evidence.js";
import { parseEvidenceBundle } from "../../../src/domain/evidenceBundle.js";
import { ok } from "../../../src/domain/result.js";
import { createTestBinarySession } from "../../fixtures/binarySession.js";
import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js";
const identity = { id: "fixture", name: "Fixture", version: "1" } as const;
const effects = {
mutatesArtifact: false,
launchesProcess: false,
mayShowUi: false,
mayAccessNetwork: false,
mayWriteFilesystem: false,
changesPermissions: false,
requiresRoot: false,
};
const procedure = {
address: "0x1000",
name: "selected",
classification: null,
body: { available: false as const, reason: "unavailable" },
};
const result = {
procedure,
direction: "incoming" as const,
reference_kinds_available: false,
references: [],
unresolved_calls: [{ address: "0x1010", reason: "unknown target" }],
};
it.each([false, true])(
"exports linked source Evidence or refuses missing closure (missing=%s)",
async (missing) => {
const root = await createTestTempDirectory("rea-procedure-references-cli-");
const path = join(root, "fixture.hop"),
output = join(root, "evidence.json");
await writeFile(path, "synthetic analysis database");
const history = createEvidence(undefined, identity, {
operation: "health",
parameters: {},
result: "project history",
...(missing ? { evidenceLinks: ["ev_" + "a".repeat(64)] } : {}),
});
const provider: AnalysisProvider = {
identity: () => identity,
capabilities: () => [
{
provider: identity,
operation: "procedure_references",
available: true,
reason: null,
cachePolicy: "live",
limitations: [],
effects,
},
],
resolveAnalysisProfile: async () =>
ok({ profile: createAnalysisProfile(identity, { fixture: true }) }),
createClient: () => ({
execute: async () =>
ok(
createAnalysisExecution(result, identity, {
sourceEvidence: [history],
}),
),
close: async () => ok(null),
}),
};
const createSession = () => createTestBinarySession(provider);
const dependencies: DirectAnalysisDependencies = {
readConfiguration: () => parseConfig({}),
createBinarySession: createSession,
createNativeDispatchBinarySession: createSession,
createArtifactBinarySession: createSession,
createNativeBinarySession: createSession,
createManagedBinarySession: createSession,
};
const cli = Cli.create("procedure-references", { sync: false });
registerCoreNativeCommands(
cli,
pino({ enabled: false }),
(target, tool, args, options) =>
runDirectAnalysis(dependencies, target, tool, args, options),
);
let stdout = "";
await cli.serve(
[
"procedure-references",
path,
"selected",
"--provider",
"fixture",
"--direction",
"incoming",
"--detail",
"summary",
"--evidence_output",
output,
"--json",
],
{
env: {},
stdout: (text) => {
stdout += text;
},
exit: () => undefined,
},
);
const response = JSON.parse(stdout);
if (missing) {
expect(response).toMatchObject({
code: "evidence_integrity_mismatch",
details: { reason: "missing", evidence_id: "ev_" + "a".repeat(64) },
});
await expect(stat(output)).rejects.toMatchObject({ code: "ENOENT" });
} else {
const bundle = parseEvidenceBundle(
JSON.parse(await readFile(output, "utf8")),
);
const parent = bundle.records.find(
(record) => record.operation === "procedure_references",
);
expect(response.summary.normalized_result).toMatchObject({
parent_evidence_id: parent?.evidence_id,
summary: {
reference_count: 0,
unresolved_call_count: 1,
direction: "incoming",
},
});
expect(parent?.evidence_links).toEqual([history.evidence_id]);
expect(
bundle.records.some(
(record) => record.evidence_id === history.evidence_id,
),
).toBe(true);
}
},
);
@@ -13,7 +13,7 @@ interface PathInputCase {
readonly field: string;
readonly valid: Readonly<Record<string, unknown>>;
readonly invalid: Readonly<Record<string, unknown>>;
readonly issuePath?: readonly string[];
readonly issuePath: readonly string[];
}
const createMcp = async (
@@ -104,7 +104,13 @@ const inputCases = (
field: string,
valid: Readonly<Record<string, unknown>>,
invalid: Readonly<Record<string, unknown>>,
): PathInputCase => ({ name, field, valid, invalid });
): PathInputCase => ({
name,
field,
valid,
invalid,
issuePath: field.split("."),
});
return [
pathCase("open_binary", "path", { path: absolute }, { path: relative }),
@@ -246,8 +252,7 @@ it("advertises full host-absolute path constraints after SDK schema conversion",
const ajv = new Ajv2020({ strict: false, validateFormats: false });
const absolute = resolve("rea-absolute-input");
const relative = "relative-input";
for (const { name, field, valid, invalid } of [
const schemaCases = [
...inputCases(absolute, relative),
{
name: "inspect_go_binary",
@@ -276,7 +281,9 @@ it("advertises full host-absolute path constraints after SDK schema conversion",
right_bundle_path: absolute,
},
},
]) {
];
const checked: string[] = [];
for (const { name, field, valid, invalid } of schemaCases) {
const schema = listed.tools.find(
(tool) => tool.name === name,
)?.inputSchema;
@@ -288,7 +295,16 @@ it("advertises full host-absolute path constraints after SDK schema conversion",
validate(valid),
`${name}.${field}: ${JSON.stringify(validate.errors)}`,
).toBe(true);
checked.push(`${name}.${field}`);
}
expect(checked).toEqual(
expect.arrayContaining([
"open_binary.path",
"extract_firmware.output_directory",
"inspect_go_binary.path NUL",
"compare_bundles.left_bundle_path empty",
]),
);
expect(firmwareCalls).toEqual([]);
} finally {
await close();
@@ -299,19 +315,39 @@ it("rejects relative paths through in-memory MCP handlers", async () => {
const firmwareCalls: string[] = [];
const { client, close } = await createMcp(firmwareCalls);
try {
for (const { name, field, invalid, issuePath } of inputCases(
resolve("rea-absolute-input"),
"relative-input",
)) {
const cases = inputCases(resolve("rea-absolute-input"), "relative-input");
const checked: string[] = [];
for (const { name, field, invalid, issuePath } of cases) {
const result = await client.callTool({ name, arguments: invalid });
expect(result.isError, `${name}.${field}`).toBe(true);
expect(JSON.stringify(result), `${name}.${field}`).toContain("absolute");
if (issuePath !== undefined)
expect(parseMcpToolError(result).error.details).toMatchObject({
operation: name,
issues: [{ path: issuePath, reason: "invalid_format" }],
});
expect(parseMcpToolError(result).error, `${name}.${field}`).toMatchObject(
{
code: "invalid_request",
details: {
operation: name,
issues: [
expect.objectContaining({
path: issuePath,
reason: "invalid_format",
expected: "regex",
}),
],
},
},
);
checked.push(`${name}.${field}`);
}
expect(checked).toEqual(
expect.arrayContaining([
"open_binary.path",
"open_binary.snapshot_path",
"extract_firmware.path",
"extract_firmware.output_directory",
"inspect_analysis_view.source.path",
"capture_browser_scenario.browser.executable_path",
"capture_electron_scenario.application_root",
]),
);
expect(firmwareCalls).toEqual([]);
} finally {
await close();
@@ -0,0 +1,297 @@
import { join } from "node:path";
import { expect, it } from "vitest";
import { createEvidence, parseEvidence } from "../../../src/domain/evidence.js";
import { analysisViewResultSchema } from "../../../src/domain/analysisView/analysisView.js";
import { createApplicationMcpHarness } from "../../fixtures/applicationMcpHarness.js";
import { createTestBinarySession } from "../../fixtures/binarySession.js";
import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js";
import { parseMcpToolError } from "../../fixtures/mcpToolError.js";
const selected = {
address: "0x1000",
name: "selected",
classification: null,
body: { available: false as const, reason: "not reported" },
};
const other = { ...selected, address: "0x2000", name: "other" };
const edge = {
source_address: "0x1001",
target_address: "0x2000",
source_procedure: selected,
target_procedure: other,
kind: { available: false as const, reason: "not reported" },
};
const fixture = {
procedure: selected,
direction: "outgoing" as const,
reference_kinds_available: false,
references: [
edge,
{ ...edge, source_address: "0x1002", target_procedure: null },
edge,
],
unresolved_calls: [{ address: "0x1003", reason: "indirect destination" }],
};
const setup = async (profile: "full" | "compact") => {
const session = createTestBinarySession(() => {
throw new Error("View opened provider");
});
return {
...(await createApplicationMcpHarness({
session,
clientIdentity: { name: "procedure-reference-view", version: "1" },
serverOptions: {
environment: {
...process.env,
REA_MCP_INPUT_SCHEMA_PROFILE: profile,
REA_MCP_OUTPUT_SCHEMA_PROFILE: profile,
},
},
})),
session,
};
};
it.each(["full", "compact"] as const)(
"pages procedure reference Evidence across a %s offline handoff",
async (profile) => {
const parent = createEvidence(
undefined,
{ id: "fixture", name: "Fixture", version: "1" },
{
operation: "procedure_references",
parameters: { procedure: "selected", direction: "outgoing" },
result: fixture,
limitations: ["fixture coverage"],
},
);
const first = await setup(profile);
const source = {
kind: "retained-evidence",
evidence_id: parent.evidence_id,
};
const invoke = async (
client: typeof first.client,
name: string,
args: Record<string, unknown>,
) => {
const reply = await client.callTool({ name, arguments: args });
expect(reply.isError, JSON.stringify(reply)).not.toBe(true);
return reply.structuredContent;
};
const view = async (
client: typeof first.client,
request: Record<string, unknown>,
) =>
analysisViewResultSchema.parse(
parseEvidence(
await invoke(client, "inspect_analysis_view", {
source,
view: { kind: "native-procedure-references", ...request },
}),
).normalized_result,
);
const path = join(
await createTestTempDirectory("rea-procedure-reference-view-"),
"bundle.json",
);
try {
expect(first.session.recordEvidence(parent).ok).toBe(true);
const firstPage = await view(first.client, {
collection: "references",
source_address: "0x1001",
target_address: "0x2000",
offset: 0,
limit: 1,
});
const secondPage = await view(first.client, {
collection: "references",
source_address: "0x1001",
target_address: "0x2000",
offset: 1,
limit: 1,
});
expect(firstPage).toMatchObject({
parent_evidence_id: parent.evidence_id,
summary: {
procedure: selected,
direction: "outgoing",
reference_count: 3,
unresolved_call_count: 1,
},
coverage: { total: 2, next_offset: 1 },
items: {
rows: [
{
source_index: 0,
source_procedure: { kind: "selected-procedure" },
target_procedure: { kind: "shared", index: 0 },
kind: edge.kind,
},
],
},
});
expect(secondPage).toMatchObject({
coverage: { total: 2, next_offset: null },
items: { rows: [{ source_index: 2 }] },
});
expect(
analysisViewResultSchema.safeParse({
...firstPage,
items: { collection: "unresolved_calls", rows: [] },
}).success,
).toBe(false);
expect(
await view(first.client, {
collection: "unresolved_calls",
offset: 0,
limit: 1,
}),
).toMatchObject({
items: {
rows: [{ source_index: 0, value: fixture.unresolved_calls[0] }],
},
});
expect(
await view(first.client, {
collection: "references",
source_address: "0xmissing",
}),
).toMatchObject({ coverage: { status: "empty", total: 0 } });
const invalid = await first.client.callTool({
name: "inspect_analysis_view",
arguments: {
source,
view: {
kind: "native-procedure-references",
collection: "unresolved_calls",
source_address: "0x1000",
},
},
});
expect(invalid.isError).toBe(true);
await invoke(first.client, "export_evidence_bundle", { path });
} finally {
await first.close();
}
const second = await setup(profile);
try {
await invoke(second.client, "import_evidence_bundle", { path });
expect(
await view(second.client, {
collection: "references",
offset: 1,
limit: 1,
}),
).toMatchObject({
items: {
rows: [{ source_index: 1, target_procedure: { kind: "null" } }],
},
});
} finally {
await second.close();
}
},
);
it("bounds a requested page by bytes and keeps oversized-row recovery available", async () => {
const largeReason = "observed reference limitation ".repeat(90);
const rows = Array.from({ length: 900 }, (_, index) => ({
...edge,
source_address: `0x${(0x1000 + index).toString(16)}`,
kind: { available: false as const, reason: largeReason },
}));
const provider = { id: "fixture", name: "Fixture", version: "1" };
const parent = createEvidence(undefined, provider, {
operation: "procedure_references",
parameters: {},
result: { ...fixture, references: rows },
});
const oversize = createEvidence(undefined, provider, {
operation: "procedure_references",
parameters: {},
result: {
...fixture,
references: [
{
...edge,
target_procedure: { ...other, name: "x".repeat(1_100_000) },
},
],
},
});
const { session, client, close } = await setup("compact");
const call = (
id: string,
collection: "references" | "unresolved_calls",
offset: number,
) =>
client.callTool({
name: "inspect_analysis_view",
arguments: {
source: { kind: "retained-evidence", evidence_id: id },
view: {
kind: "native-procedure-references",
collection,
offset,
limit: 900,
},
},
});
try {
expect(session.recordEvidence(parent).ok).toBe(true);
expect(session.recordEvidence(oversize).ok).toBe(true);
const first = await call(parent.evidence_id, "references", 0);
expect(first.isError, JSON.stringify(first)).not.toBe(true);
const page = analysisViewResultSchema.parse(
parseEvidence(first.structuredContent).normalized_result,
);
if (
page.kind !== "native-procedure-references" ||
page.items.collection !== "references"
)
throw new Error("Expected references page");
expect(page.byte_limited).toBe(true);
expect(page.coverage.total).toBe(rows.length);
const nextOffset = page.coverage.next_offset;
expect(nextOffset).toBe(page.items.rows.length);
if (nextOffset === null) throw new Error("Expected continuation");
const next = await call(parent.evidence_id, "references", nextOffset);
expect(next.isError, JSON.stringify(next)).not.toBe(true);
const nextPage = analysisViewResultSchema.parse(
parseEvidence(next.structuredContent).normalized_result,
);
if (
nextPage.kind !== "native-procedure-references" ||
nextPage.items.collection !== "references"
)
throw new Error("Expected references continuation");
expect(nextPage.items.rows[0]?.source_index).toBe(nextOffset);
const refused = await call(oversize.evidence_id, "references", 0);
expect(refused.isError).toBe(true);
expect(parseMcpToolError(refused)).toMatchObject({
error: {
code: "resource_constraint",
remediation: {
action: expect.stringContaining("export the retained parent"),
},
details: {
reported_limits: {
parent_evidence_id: oversize.evidence_id,
limit_bytes: 1_048_576,
},
},
},
});
const otherPage = await call(oversize.evidence_id, "unresolved_calls", 0);
expect(otherPage.isError, JSON.stringify(otherPage)).not.toBe(true);
expect(
parseEvidence(otherPage.structuredContent).normalized_result,
).toMatchObject({
items: { rows: [{ value: fixture.unresolved_calls[0] }] },
});
} finally {
await close();
}
});
+26 -4
View File
@@ -709,8 +709,19 @@ describe("session filesystem path boundaries over MCP", () => {
name: "export_evidence_bundle",
arguments: { path: "relative-bundle.json" },
});
expect(exported.isError, JSON.stringify(exported.content)).toBe(true);
expect(JSON.stringify(exported.content)).toContain("absolute");
expect(parseMcpToolError(exported).error).toMatchObject({
code: "invalid_request",
details: {
operation: "export_evidence_bundle",
issues: [
expect.objectContaining({
path: ["path"],
reason: "invalid_format",
expected: "regex",
}),
],
},
});
const opened = await mcp.callTool({
name: "open_binary",
@@ -721,8 +732,19 @@ describe("session filesystem path boundaries over MCP", () => {
name: "close_binary",
arguments: { snapshot_path: "relative-analysis.json" },
});
expect(closed.isError, JSON.stringify(closed.content)).toBe(true);
expect(JSON.stringify(closed.content)).toContain("absolute");
expect(parseMcpToolError(closed).error).toMatchObject({
code: "invalid_request",
details: {
operation: "close_binary",
issues: [
expect.objectContaining({
path: ["snapshot_path"],
reason: "invalid_format",
expected: "regex",
}),
],
},
});
await mcp.callTool({ name: "close_binary", arguments: {} });
}, 10_000);
});
+22 -14
View File
@@ -383,7 +383,7 @@ it("attributes rejected WASM paths to their selected fields before provider work
},
});
onTestFinished(harness.close);
for (const [input, paths] of [
const relativeCases = [
[{ path: "relative.wasm" }, [["path"]]],
[
{ path: value.artifact.path, glue_paths: ["relative.js"] },
@@ -412,7 +412,9 @@ it("attributes rejected WASM paths to their selected fields before provider work
},
[["path"], ["glue_paths", 0], ["candidate_paths", 0]],
],
] as const) {
] as const;
const checked: string[] = [];
for (const [input, paths] of relativeCases) {
const response = await harness.client.callTool({
name: "inspect_wasm_artifact",
arguments: input,
@@ -423,14 +425,19 @@ it("attributes rejected WASM paths to their selected fields before provider work
.object({ issues: z.array(z.unknown()) })
.parse(failure.details);
expect(issues).toEqual(
paths.map((path) => ({
path,
reason: "invalid_format",
expected: "regex",
message: "must be an absolute local filesystem path",
})),
paths.map((path) =>
expect.objectContaining({
path,
reason: "invalid_format",
expected: "regex",
}),
),
);
checked.push(paths.map((path) => path.join(".")).join("+"));
}
expect(checked).toEqual(
expect.arrayContaining(["path", "glue_paths.0", "candidate_paths.0"]),
);
expect(inspect).not.toHaveBeenCalled();
const valid = await harness.client.callTool({
name: "inspect_wasm_artifact",
@@ -760,12 +767,13 @@ it("rejects malformed paths, refuses WASM summary delivery before provider work
expect(
z.object({ issues: z.array(z.unknown()) }).parse(rejected.details).issues,
).toEqual(
[["path"], ["glue_paths", 0]].map((path) => ({
path,
reason: "invalid_format",
expected: "regex",
message: "must be an absolute local filesystem path",
})),
[["path"], ["glue_paths", 0]].map((path) =>
expect.objectContaining({
path,
reason: "invalid_format",
expected: "regex",
}),
),
);
expect(inspect).not.toHaveBeenCalled();
for (const input of [
@@ -0,0 +1,205 @@
import { execFileSync } from "node:child_process";
import { writeFile } from "node:fs/promises";
import { join } from "node:path";
import { expect, it } from "vitest";
import {
PYTHON_HELPERS,
PYTHON_IMPORTS,
} from "../../../src/domain/desktop/appExportPython.js";
import {
detectStateFormat,
textualView,
} from "../../../src/domain/desktop/stateFormats.js";
import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js";
const arrays = "[".repeat(1500) + "1" + "]".repeat(1500);
const objects = '{"a":'.repeat(1500) + "1" + "}".repeat(1500);
const cases = [
["discarded arrays", `{"value":${arrays},"value":"done"}`, true],
["discarded objects", `{"value":${objects},"value":"done"}`, true],
["escaped duplicate key", `{"val\\u0075e":${arrays},"value":"done"}`, true],
["retained deep value", `{"value":${arrays}}`, false],
[
"retained deep inexact number",
`{"value":${"[".repeat(600)}9007199254740993${"]".repeat(600)}}`,
false,
],
[
"deep overwritten inexact number",
`{"value":${"[".repeat(600)}{"x":9007199254740993,"x":1}${"]".repeat(600)}}`,
false,
],
[
"deep empty-key overwrite",
`{"value":${"[".repeat(600)}{"":9007199254740993,"":1}${"]".repeat(600)}}`,
false,
],
[
"deep multiple inexact keys with one overwritten",
`{"value":${"[".repeat(600)}{"a":9007199254740993,"b":9007199254740993,"a":1}${"]".repeat(600)}}`,
false,
],
[
"deep multiple inexact keys all overwritten",
`{"value":${"[".repeat(600)}{"a":9007199254740993,"b":9007199254740993,"a":1,"b":2}${"]".repeat(600)}}`,
false,
],
[
"deep inexact key replaced by inexact",
`{"value":${"[".repeat(600)}{"a":9007199254740993,"a":9007199254740995}${"]".repeat(600)}}`,
false,
],
[
"deep escaped duplicate overwritten",
`{"value":${"[".repeat(600)}{"\\u0061":9007199254740993,"a":1}${"]".repeat(600)}}`,
false,
],
[
"discarded deep inexact subtree",
`{"value":${"[".repeat(600)}9007199254740993${"]".repeat(600)},"value":"done"}`,
true,
],
["last admitted depth", "[".repeat(500) + "1" + "]".repeat(500), true],
["empty container at limit", "[".repeat(501) + "]".repeat(501), true],
["empty container over limit", "[".repeat(502) + "]".repeat(502), false],
["trailing comma", `{"value":${arrays},"value":"done",}`, false],
["missing value", `{"value":${arrays},"value":}`, false],
["wrong closing bracket", `{"value":${arrays},"value":"done"]`, false],
] as const;
it.each(cases)("reads JSON with %s", async (_name, text, readable) => {
const root = await createTestTempDirectory("desktop-json-depth-");
const script = join(root, "read.py");
const imports = PYTHON_IMPORTS.split("\n")
.filter((line) => !line.includes("agentenv_protocol"))
.join("\n");
await writeFile(
script,
`${imports}${PYTHON_HELPERS}
text = json.load(sys.stdin)
print(json.dumps(_text_view("state.json", text, len(text.encode("utf-8")))))
`,
);
const output = JSON.parse(
execFileSync(
process.platform === "win32" ? "python" : "python3",
["-I", "-B", "-X", "utf8", script],
{ input: JSON.stringify(text), encoding: "utf8" },
),
) as { format: string; view: unknown; error?: string };
const bytes = Buffer.from(text);
const format = detectStateFormat("state.json", bytes);
if (format !== "json" && format !== "text")
throw new Error(`Unexpected native format ${format}`);
const native = textualView(format, bytes);
expect(format === "json" && native.ok).toBe(readable);
expect(output.format).toBe(format);
if (native.ok) {
expect(output.error).toBeUndefined();
// Compare JSON values on their wire form, independent of object prototypes.
expect(JSON.stringify(output.view)).toBe(JSON.stringify(native.value));
if (
readable &&
(_name.startsWith("discarded") || _name === "escaped duplicate key")
)
expect(output.view).toEqual({ value: "done" });
} else {
expect(output.view).toBeNull();
expect(output.error).toBe(native.error);
}
});
it("bounds construction before refusing retained excessive JSON depth", async () => {
const root = await createTestTempDirectory("desktop-json-construction-");
const script = join(root, "bounded.py");
const imports = PYTHON_IMPORTS.split("\n")
.filter((line) => !line.includes("agentenv_protocol"))
.join("\n");
await writeFile(
script,
`${imports}${PYTHON_HELPERS}
import tracemalloc
text = "[" * 20_000 + "0" + "]" * 20_000
recursion_limit = sys.getrecursionlimit()
tracemalloc.start()
entry = _json_view(text, len(text))
peak = tracemalloc.get_traced_memory()[1]
tracemalloc.stop()
print(json.dumps({"entry": entry, "peak": peak, "recursion_unchanged": sys.getrecursionlimit() == recursion_limit}))
`,
);
const output = JSON.parse(
execFileSync(
process.platform === "win32" ? "python" : "python3",
["-I", "-B", "-X", "utf8", script],
{ encoding: "utf8" },
),
) as { entry: unknown; peak: number; recursion_unchanged: boolean };
expect(output.entry).toEqual({
format: "json",
view: null,
error: "nested deeper than the 500 levels a JSON view keeps",
});
expect(output.recursion_unchanged).toBe(true);
// Python heap only, excluding source construction and interpreter/OS memory.
// A value retained only to refuse it must not allocate every input container.
expect(output.peak).toBeLessThan(1024 * 1024);
});
it("bounds allocations when a very deep value is overwritten", async () => {
const root = await createTestTempDirectory("desktop-json-memory-");
const script = join(root, "read.py");
const imports = PYTHON_IMPORTS.split("\n")
.filter((line) => !line.includes("agentenv_protocol"))
.join("\n");
await writeFile(
script,
`${imports}\nimport tracemalloc\n${PYTHON_HELPERS}
text = '{"value":' + '[' * 200000 + '1' + ']' * 200000 + ',"value":"done"}'
tracemalloc.start()
result = _json_view(text, len(text))
_, peak = tracemalloc.get_traced_memory()
print(json.dumps({"view": result["view"], "peak": peak}))
`,
);
const measured = JSON.parse(
execFileSync(
process.platform === "win32" ? "python" : "python3",
["-I", "-B", script],
{ encoding: "utf8", timeout: 30000 },
),
) as { view: unknown; peak: number };
expect(measured.view).toEqual({ value: "done" });
expect(measured.peak).toBeLessThan(8 * 1024 * 1024);
}, 40000);
it("bounds a wide container already beyond the retained view depth", async () => {
const root = await createTestTempDirectory("desktop-json-wide-depth-");
const script = join(root, "read.py");
const imports = PYTHON_IMPORTS.split("\n")
.filter((line) => !line.includes("agentenv_protocol"))
.join("\n");
await writeFile(
script,
`${imports}\nimport tracemalloc\n${PYTHON_HELPERS}
text = '[' * 501 + ','.join(['"abc"'] * 600000) + ']' * 501
tracemalloc.start()
result = _json_view(text, len(text))
_, peak = tracemalloc.get_traced_memory()
print(json.dumps({"error": result.get("error"), "peak": peak}))
`,
);
const measured = JSON.parse(
execFileSync(
process.platform === "win32" ? "python" : "python3",
["-I", "-B", script],
{ encoding: "utf8", timeout: 30000 },
),
) as { error: string; peak: number };
expect(measured.error).toBe(
"nested deeper than the 500 levels a JSON view keeps",
);
expect(measured.peak).toBeLessThan(8 * 1024 * 1024);
}, 40000);
@@ -34,7 +34,7 @@ const execPython = (
try {
writeFileSync(script, args[codeIndex + 1]!, "utf8");
return execFileSync(
"python3",
process.platform === "win32" ? "python" : "python3",
[
"-X",
"utf8",
@@ -375,15 +375,16 @@ it("checks text for JSON the way REA does, token by token, without parsing it",
"-I",
"-B",
"-c",
`${imports}${PYTHON_HELPERS}\nprint(json.dumps([_is_json(text) for text in json.load(sys.stdin)]))`,
`${imports}${PYTHON_HELPERS}\ntexts = json.load(sys.stdin)\nprint(json.dumps({"syntax": [_read_json(text, build=False) is not _INVALID_JSON for text in texts], "built": [_read_json(text) is not _INVALID_JSON for text in texts]}))`,
],
{ input: JSON.stringify(texts), encoding: "utf8" },
),
) as boolean[];
) as { syntax: boolean[]; built: boolean[] };
const parsed = texts.map((text) => safeParseJson(text).ok);
expect(texts.map((text) => isJsonText(text))).toEqual(parsed);
expect(python).toEqual(parsed);
expect(python.syntax).toEqual(parsed);
expect(python.built).toEqual(parsed);
expect(parsed.filter(Boolean)).toHaveLength(10);
});
@@ -445,9 +446,13 @@ print(json.dumps([_text_view("settings.ini", text, len(text.encode()))["view"]["
const root = await createTestTempDirectory("rea-keyfile-python-");
const script = join(root, "fixture.py");
await writeFile(script, program, "utf8");
const output = execFileSync("python3", ["-I", "-B", "-X", "utf8", script], {
input: JSON.stringify(inputs),
encoding: "utf8",
});
const output = execFileSync(
process.platform === "win32" ? "python" : "python3",
["-I", "-B", "-X", "utf8", script],
{
input: JSON.stringify(inputs),
encoding: "utf8",
},
);
expect(JSON.parse(output)).toEqual(inputs.map(keyfileView));
});