Compare commits

...
Author SHA1 Message Date
morluto 5ce9776a19 fix(ghidra): reconcile current main formatting 2026-11-07 23:16:20 +08:00
morluto ac30fbd658 style(test): format CLI help default annotations (#9280) 2026-11-07 23:02:44 +08:00
Nianze Wuandmorluto 5d21bddb18 fix(desktop): read JSON with deeply nested overwritten values (#9277)
* fix(desktop): read JSON with deeply nested overwritten values

* fix(desktop): bound exported JSON construction by view depth

* fix(desktop): preserve deep JSON numeric facts within bounded memory

---------

Co-authored-by: morluto <[email protected]>
2026-11-07 22:57:15 +08:00
morlutoandCursor Agent 2e2f981ccf test: assert absolute-path facts without frozen sentences (#9231)
* test: assert absolute-path facts without frozen sentences

Sibling path tests now require the field path, invalid_format, and regex
expectation, and they record that each negative case ran. NUL rejection
follows the argument that contains the byte instead of searching for the
word NUL.

* test: retain actionable NUL path diagnostics

---------

Co-authored-by: Cursor Agent <[email protected]>
2026-11-07 22:46:17 +08:00
morluto 5b2bb57247 fix(ghidra): reconcile overlay coverage with main 2026-11-07 22:37:54 +08:00
Cursor Agent 3fc27b45cd docs(ghidra): match the raw lane's overlay coverage
The raw Ghidra lane checks a manually created overlay block through MCP.
Say that, and do not claim an overlapping-load ELF, CLI search, or snapshot
cursor handoff.
2026-11-04 19:58:40 +00:00
Cursor Agent b9db66256a merge: report loaded overlays separately from unloaded blocks
Keep contiguous initialized-block search and list loaded overlays in
skipped_overlays. Unloaded blocks, including unloaded overlays, stay in
skipped_not_loaded.
2026-11-04 19:42:48 +00:00
morluto 331f2dd827 test(ghidra): classify search start as an address bound 2026-11-02 08:19:58 +08:00
morluto 2a2fdfc0db test(ghidra): avoid freezing byte search help wording 2026-11-02 07:53:26 +08:00
morluto 5fad84347a test(ghidra): exercise overlay search cursor handoff 2026-11-02 07:43:14 +08:00
morluto 027aea38a2 fix(cli): explain overlay byte search scope 2026-11-02 07:36:39 +08:00
morluto f584290933 fix(ghidra)!: distinguish skipped overlays in byte search
BREAKING CHANGE: Byte search results require skipped_overlays; loaded overlays no longer appear in skipped_not_loaded. The Ghidra profile changes so prior snapshots are not replayed.
2026-11-02 07:31:31 +08:00
30 changed files with 763 additions and 218 deletions

No files matched your search

+9 -1
View File
@@ -876,6 +876,7 @@ public final class ReaGhidraBridge extends HeadlessScript {
JsonArray blocks = new JsonArray();
JsonArray skipped = new JsonArray();
JsonArray notLoaded = new JsonArray();
JsonArray overlays = new JsonArray();
List<ByteSearchBlock> eligible = new ArrayList<>();
for (MemoryBlock block : ordered) {
monitor.checkCancelled();
@@ -883,10 +884,16 @@ public final class ReaGhidraBridge extends HeadlessScript {
Address high = block.getEnd();
Address bound = start == null ? end : start;
if (bound != null && !bound.getAddressSpace().equals(low.getAddressSpace())) continue;
if (bound == null && (!block.isLoaded() || block.isOverlay())) {
// Without bounds, search loaded non-overlay memory. Unloaded file
// parts and loaded overlays stay reachable through explicit bounds.
if (bound == null && !block.isLoaded()) {
notLoaded.add(byteSearchCoverage(block, low, high));
continue;
}
if (bound == null && block.isOverlay()) {
overlays.add(byteSearchCoverage(block, low, high));
continue;
}
if (start != null && low.compareTo(start) < 0) low = start;
if (end != null && high.compareTo(end) > 0) high = end;
if (low.compareTo(high) > 0) continue;
@@ -960,6 +967,7 @@ public final class ReaGhidraBridge extends HeadlessScript {
searched.addProperty("bytes", covered);
searched.add("skipped_uninitialized", skipped);
searched.add("skipped_not_loaded", notLoaded);
searched.add("skipped_overlays", overlays);
JsonObject result = new JsonObject();
result.add("matches", matches);
result.add("searched", searched);
+2 -2
View File
@@ -310,8 +310,8 @@ entry it does not read. The module reads the central directory itself, as REA
does, and shows `zipfile` only the records it lists, reading each entry in
bounded chunks no further than a byte past its declared size. A JSON file
holding a number JavaScript cannot read exactly, such as an integer above 2^53,
or nested more than 500 levels deep, gets an `error` instead of a view, as in
REA, while text that is not one JSON
or whose retained value nests more than 500 levels deep, gets an `error` instead of a view, as in
REA. Duplicate keys keep their last value before the depth check. Text that is not one JSON
document, such as JSON Lines, stays text. Format detection follows REA's
rules. Files larger than 64 MiB get an `error` instead of a view
and are not read. XML is read with Python's `expat` twice: with namespace
+11 -4
View File
@@ -335,10 +335,17 @@ must be constrained.
Optional `start` and `end` addresses are inclusive and select one address space.
A match must end at or before `end`, so `end` set inside a match excludes it.
Without bounds, loaded memory blocks are eligible, in Ghidra address order.
Blocks that are not loaded (file parts such as `.symtab`) and overlay blocks are
listed in `searched.skipped_not_loaded` and are not searched; pass `start` or
`end` in their address space to search them.
Without bounds, loaded non-overlay memory blocks are eligible, in Ghidra address
order, including across adjacent initialized blocks. Blocks that are not loaded
(file parts such as `.symtab`) are listed in `searched.skipped_not_loaded`;
loaded overlay blocks are listed separately in `searched.skipped_overlays`.
Neither is searched by default; pass `start` or `end` in its address space to
search it. A block that is both unloaded and an overlay is listed once under
`skipped_not_loaded`.
The separate overlay field changes the byte-search result contract and the
Ghidra analysis profile. Refresh cached MCP schemas; snapshots from the older
profile are not replayed. Previously exported Evidence remains importable as
historical data with its original result shape.
Overlapping matches are included, including across adjacent eligible initialized
blocks in the same address space. Gaps and excluded or uninitialized blocks break
a search run. Each match reports its address, starting segment, unique source file
+1 -1
View File
@@ -531,7 +531,7 @@ the entry changes without `provider_id`, preservation of a session annotation
and the target/provider receipt after a rejected open, unchanged source bytes,
and owned cleanup. Six-byte x86 and x86_64 controls also verify supported compiler
specifications, independent mappings, unchanged bytes, and decompilation of a
known return value. AArch64 remains covered by deterministic tests rather than
known return value. On a disposable Ghidra project, MCP `search_bytes` also checks a manually created loaded `overlay_test` block: an unbounded search lists it in `skipped_overlays` and not in `skipped_not_loaded`, and bounds in that overlay space find the pattern. This lane does not build an overlapping-load ELF, does not call CLI `search-bytes`, and does not page an overlay search across a snapshot reopen. AArch64 remains covered by deterministic tests rather than
this real-provider lane. See [raw binary analysis](raw-binary.md).
`npm run verify:ghidra:address-context` runs on Linux with a caller-installed
+6 -1
View File
@@ -760,7 +760,12 @@ async function verifyCrossBlockProject(target, targetSha256) {
!all.matches.some((match) => match.address.startsWith("overlay_test:")),
);
assert.ok(
all.searched.skipped_not_loaded.some(
!all.searched.skipped_not_loaded.some(
(block) => block.segment === "overlay_test",
),
);
assert.ok(
all.searched.skipped_overlays.some(
(block) => block.segment === "overlay_test",
),
);
+14 -3
View File
@@ -51,15 +51,26 @@ export const registerCoreNativeCommands = (
): void => {
registerAnnotationCommand(cli, logger, runDirectAnalysis);
cli.command(CLI_COMMANDS.searchBytes, {
description: "Find masked byte patterns in initialized provider memory",
description:
"Find masked bytes; unbounded searches skip unloaded blocks and overlays",
args: z.object({
path: localPathStringSchema.describe("Local executable path"),
pattern: bytePatternFields.pattern,
}),
options: z.object({
mask: bytePatternFields.mask,
start: z.string().optional().describe("Inclusive start address"),
end: z.string().optional().describe("Inclusive end address"),
start: z
.string()
.optional()
.describe(
"Inclusive start address; use an overlay address space to search it",
),
end: z
.string()
.optional()
.describe(
"Inclusive end address; use an overlay address space to search it",
),
cursor: z
.string()
.optional()
+1 -1
View File
@@ -123,7 +123,7 @@ describe("CLI JSON path normalization", () => {
expect.objectContaining({
path: ["executable_path"],
code: "invalid_format",
message: expect.stringContaining("absolute local filesystem path"),
format: "regex",
}),
]);
});
+1 -1
View File
@@ -11,7 +11,7 @@ export const HOPPER_MEMORY_TOOL_DEFINITIONS = [
{
name: "search_bytes",
description:
"Find masked byte signatures in initialized analyzed memory. Hex byte pairs and ?? wildcards are supported; an optional equal-length hex mask selects bits. Bounds are inclusive. Matches are ordered by provider address and may span adjacent eligible initialized blocks; gaps and excluded blocks end a contiguous range. A match’s segment and file offset describe its starting address. Coverage describes this page; reuse the same query with next_cursor to continue. Uninitialized blocks are listed separately. Ghidra only; Hopper is unavailable.",
"Find masked byte signatures in initialized analyzed memory. Hex byte pairs and ?? wildcards are supported; an optional equal-length hex mask selects bits. Bounds are inclusive. Without bounds, unloaded blocks and loaded overlays are skipped and listed separately; select an overlay address space with start or end to search it. Matches are ordered by provider address and may span adjacent eligible initialized blocks; gaps and excluded blocks end a contiguous range. A match’s segment and file offset describe its starting address. Coverage describes this page; reuse the same query with next_cursor to continue. Uninitialized blocks are listed separately. Ghidra only; Hopper is unavailable.",
inputSchema: z
.strictObject({
...bytePatternFields,
+3 -142
View File
@@ -1,3 +1,4 @@
import { PYTHON_JSON_READER } from "./appExportPythonJson.js";
import { PYTHON_XML_READER } from "./appExportPythonXml.js";
import { PYTHON_ZIP_HELPERS } from "./appExportPythonZip.js";
@@ -12,6 +13,7 @@ export const PYTHON_READY_CHECK = `
export const PYTHON_IMPORTS = `from __future__ import annotations
from array import array
import contextlib
import copy
import hashlib
@@ -57,11 +59,6 @@ MAX_PARSED_BYTES = 4 * 1024 * 1024
PARSED_TOO_LARGE = "larger than the 4 MiB limit for JSON and key file views"
MAX_JSON_VIEW_DEPTH = 500
JSON_TOO_DEEP = f"nested deeper than the {MAX_JSON_VIEW_DEPTH} levels a JSON view keeps"
#: A JSON token after any whitespace, by group: an opening, a closing, a comma, a colon, a string's quote, or a number
#: or literal. Strings are read by json's scanner, since a pattern repeating per escape holds memory for each.
JSON_TOKEN = re.compile(r'[ \t\n\r]*(?:([\[{])|([\]}])|(,)|(:)|(")|(-?(?:0|[1-9][0-9]*)(?:\.[0-9]+)?(?:[eE][+-]?[0-9]+)?'
r"|true|false|null))")
JSON_SPACE = re.compile(r"[ \t\n\r]*")
def _respelled(text: str, home: Path) -> str:
@@ -254,143 +251,7 @@ def _keyfile(text: str) -> dict:
${PYTHON_XML_READER}
def _not_json(constant: str):
raise ValueError(f"{constant} is not JSON")
class _Inexact(Exception):
"""A JSON number that REA's state view refuses, since reading it would change it."""
def _digits(text: str) -> str:
return re.split(r"[eE]", text.lstrip("-"))[0].replace(".", "").strip("0") or "0"
def _js_string(value: float) -> str:
"""How JavaScript's String() writes a finite double: an integer below 1e21 in full, any other in shortest form."""
return str(int(value)) if value.is_integer() and abs(value) < 1e21 else repr(value)
def _safe_number(text: str) -> bool:
"""lossless-json's isSafeNumber: an integer spelling within 2^53, or a float spelling that keeps its digits."""
if re.fullmatch(r"-?[0-9]+", text):
return len(text.lstrip("-")) <= 16 and abs(int(text)) <= MAX_SAFE_INTEGER
value = float(text)
if not math.isfinite(value):
return False
written = _js_string(value)
return written == text or _digits(written) == _digits(text)
def _exact_number(text: str):
"""A JSON number read only when reading it changes nothing, as REA's state view rules: its spelling and its
written-back form both safe, and not -0."""
value = float(text)
if (not _safe_number(text) or not math.isfinite(value) or not _safe_number(_js_string(value))
or (value == 0 and math.copysign(1, value) < 0)):
raise _Inexact(text)
return int(text) if re.fullmatch(r"-?[0-9]+", text) else value
_INEXACT = object()
_LEVEL_END = object()
def _number_or_mark(spelling: str):
"""A JSON number read exactly, or a mark for one REA's state view refuses."""
try:
return _exact_number(spelling)
except _Inexact:
return _INEXACT
def _holds_inexact(value) -> bool:
"""Whether a parsed JSON value still holds a marked number once duplicate keys kept their last value; one
iterator per nesting level, so no child list is copied."""
levels = [iter((value,))]
while levels:
item = next(levels[-1], _LEVEL_END)
if item is _LEVEL_END:
levels.pop()
elif item is _INEXACT:
return True
elif isinstance(item, dict):
levels.append(iter(item.values()))
elif isinstance(item, list):
levels.append(iter(item))
return False
def _deeper_than(root, limit: int) -> bool:
"""Whether a parsed JSON value nests deeper than a limit, as REA checks it: a value's depth is the number of
iterators open above it, one per nesting level, so no child list is copied."""
levels = [iter((root,))]
while levels:
item = next(levels[-1], _LEVEL_END)
if item is _LEVEL_END:
levels.pop()
elif len(levels) - 1 > limit:
return True
elif isinstance(item, dict):
levels.append(iter(item.values()))
elif isinstance(item, list):
levels.append(iter(item))
return False
def _is_json(text: str) -> bool:
"""Whether text is one JSON value as REA's state view reads it, checked a token at a time without building it or
recursing: it holds a byte per open container and one string at a time, so text too deep for Python's parser or
too large to parse into a view is still told apart from other text."""
containers, expect, empty, at = bytearray(), "value", False, 0
while True:
token = JSON_TOKEN.match(text, at)
if token is None:
break
kind, at = token.lastindex, token.end()
if kind == 5:
try:
at = json.decoder.scanstring(text, at)[1]
except ValueError:
return False
if kind == 1 and expect == "value":
containers.append(ord(text[at - 1]))
expect, empty = ("value" if text[at - 1] == "[" else "key"), True
# A closing bracket's code is its opening's plus two.
elif kind == 2 and (expect == "after" or empty) and containers and containers[-1] == ord(text[at - 1]) - 2:
containers.pop()
expect, empty = "after", False
elif kind == 3 and expect == "after" and containers:
expect = "value" if containers[-1] == ord("[") else "key"
elif kind == 4 and expect == "colon":
expect = "value"
elif kind >= 5 and (expect == "value" or (kind == 5 and expect == "key")):
expect, empty = ("colon" if expect == "key" else "after"), False
else:
return False
return expect == "after" and not containers and JSON_SPACE.fullmatch(text, at) is not None
def _json_view(text: str, size: int):
"""A JSON text's format and view, or why it has none, as REA reads it; None when it is not JSON. Text over the
parsed-view limit, or nested past Python's recursion limit, is told apart without building its value."""
if size > MAX_PARSED_BYTES:
return {"format": "json", "view": None, "error": PARSED_TOO_LARGE} if _is_json(text) else None
# The whole text must parse, and duplicate keys settle, before an inexact number makes it JSON with an error.
try:
view = json.loads(text, parse_constant=_not_json, parse_float=_number_or_mark, parse_int=_number_or_mark)
except RecursionError:
# Nesting too deep for Python to parse is far deeper than REA's view keeps, once the rest of it is JSON too.
return {"format": "json", "view": None, "error": JSON_TOO_DEEP} if _is_json(text) else None
except ValueError:
return None
if _holds_inexact(view):
return {"format": "json", "view": None,
"error": "a number in it cannot be read exactly, such as an integer above 2^53"}
if _deeper_than(view, MAX_JSON_VIEW_DEPTH):
return {"format": "json", "view": None, "error": JSON_TOO_DEEP}
return {"format": "json", "view": view, "truncated": False}
${PYTHON_JSON_READER}
def _text_view(name: str, text: str, size: int) -> dict:
"""The format REA's state view detects for decoded text of size bytes, and its view; a JSON or key file over the
+231
View File
@@ -0,0 +1,231 @@
/** JSON parsing and bounded retained-view checks for exported desktop environments. */
export const PYTHON_JSON_READER = String.raw`
#: A JSON token after any whitespace, by group: an opening, a closing, a comma, a colon, a string's quote, or a number
#: or literal. Strings are read by json's scanner, since a pattern repeating per escape holds memory for each.
JSON_TOKEN = re.compile(r'[ \t\n\r]*(?:([\[{])|([\]}])|(,)|(:)|(")|(-?(?:0|[1-9][0-9]*)(?:\.[0-9]+)?(?:[eE][+-]?[0-9]+)?'
r"|true|false|null))")
JSON_SPACE = re.compile(r"[ \t\n\r]*")
class _Inexact(Exception):
"""A JSON number that REA's state view refuses, since reading it would change it."""
def _digits(text: str) -> str:
return re.split(r"[eE]", text.lstrip("-"))[0].replace(".", "").strip("0") or "0"
def _js_string(value: float) -> str:
"""How JavaScript's String() writes a finite double: an integer below 1e21 in full, any other in shortest form."""
return str(int(value)) if value.is_integer() and abs(value) < 1e21 else repr(value)
def _safe_number(text: str) -> bool:
"""lossless-json's isSafeNumber: an integer spelling within 2^53, or a float spelling that keeps its digits."""
if re.fullmatch(r"-?[0-9]+", text):
return len(text.lstrip("-")) <= 16 and abs(int(text)) <= MAX_SAFE_INTEGER
value = float(text)
if not math.isfinite(value):
return False
written = _js_string(value)
return written == text or _digits(written) == _digits(text)
def _exact_number(text: str):
"""A JSON number read only when reading it changes nothing, as REA's state view rules: its spelling and its
written-back form both safe, and not -0."""
value = float(text)
if (not _safe_number(text) or not math.isfinite(value) or not _safe_number(_js_string(value))
or (value == 0 and math.copysign(1, value) < 0)):
raise _Inexact(text)
return int(text) if re.fullmatch(r"-?[0-9]+", text) else value
_INEXACT = object()
_DEEP = object()
_DEEP_INEXACT = object()
_LEVEL_END = object()
_INVALID_JSON = object()
JSON_ATOMS = {"true": True, "false": False, "null": None}
def _number_or_mark(spelling: str):
"""A JSON number read exactly, or a mark for one REA's state view refuses."""
try:
return _exact_number(spelling)
except _Inexact:
return _INEXACT
def _holds_inexact(value) -> bool:
"""Whether a parsed JSON value still holds a marked number once duplicate keys kept their last value; one
iterator per nesting level, so no child list is copied."""
levels = [iter((value,))]
while levels:
item = next(levels[-1], _LEVEL_END)
if item is _LEVEL_END:
levels.pop()
elif item is _INEXACT or item is _DEEP_INEXACT:
return True
elif isinstance(item, dict):
levels.append(iter(item.values()))
elif isinstance(item, list):
levels.append(iter(item))
return False
def _deeper_than(root, limit: int) -> bool:
"""Whether a parsed JSON value nests deeper than a limit, as REA checks it: a value's depth is the number of
iterators open above it, one per nesting level, so no child list is copied."""
levels = [iter((root,))]
while levels:
item = next(levels[-1], _LEVEL_END)
if item is _LEVEL_END:
levels.pop()
elif item is _DEEP or item is _DEEP_INEXACT:
return True
elif len(levels) - 1 > limit:
return True
elif isinstance(item, dict):
levels.append(iter(item.values()))
elif isinstance(item, list):
levels.append(iter(item))
return False
def _read_json(text: str, build: bool = True):
"""One JSON value, or _INVALID_JSON, read without recursion so duplicate keys settle before depth is checked.
Deep containers are summarized rather than built; exact-number marks still follow last-key-wins semantics.
With build=False, only syntax is checked: one byte per open container and one decoded string at a time."""
containers, expect, empty, at = bytearray(), "value", False, 0
values, keys, root = [], [], _INVALID_JSON
deep_bad, deep_key_at, bad_keys = bytearray(), array("I"), {}
def keep(value) -> None:
nonlocal root
if not values:
root = value
elif containers[-1] == ord("["):
values[-1].append(value)
else:
values[-1][keys[-1]] = value
def record_deep(bad: bool) -> None:
"""Summarize a finished child; keep only object keys with live inexact values."""
depth = len(deep_bad) - 1
if containers[-1] == ord("["):
deep_bad[-1] |= 2 | bad
return
marked = bad_keys.get(depth)
if bad or marked is not None:
key, _ = json.decoder.scanstring(text, deep_key_at[-1])
if bad:
if marked is None:
marked = key
elif isinstance(marked, str):
if marked != key:
marked = {marked, key}
else:
marked.add(key)
else:
if isinstance(marked, str):
if marked == key:
marked = None
else:
marked.discard(key)
if len(marked) == 1:
marked = next(iter(marked))
if marked is None:
bad_keys.pop(depth, None)
else:
bad_keys[depth] = marked
deep_bad[-1] = 2 | (marked is not None)
else:
deep_bad[-1] |= 2
while True:
token = JSON_TOKEN.match(text, at)
if token is None:
break
kind, at = token.lastindex, token.end()
if kind == 5:
string_at = at
try:
value, at = json.decoder.scanstring(text, at)
except ValueError:
return _INVALID_JSON
if kind == 1 and expect == "value":
if build:
if deep_bad or len(containers) >= MAX_JSON_VIEW_DEPTH:
deep_bad.append(0)
deep_key_at.append(0)
else:
value = [] if text[at - 1] == "[" else {}
keep(value)
values.append(value)
keys.append(None)
containers.append(ord(text[at - 1]))
expect, empty = ("value" if text[at - 1] == "[" else "key"), True
# A closing bracket's code is its opening's plus two.
elif kind == 2 and (expect == "after" or empty) and containers and containers[-1] == ord(text[at - 1]) - 2:
opening = containers.pop()
if build:
if deep_bad:
state = deep_bad.pop()
bad = bool(state & 1)
deep_key_at.pop()
bad_keys.pop(len(deep_bad), None)
if deep_bad:
record_deep(bad)
else:
keep((_DEEP_INEXACT if bad else _DEEP) if state & 2 else ([] if opening == ord("[") else {}))
else:
values.pop()
keys.pop()
expect, empty = "after", False
elif kind == 3 and expect == "after" and containers:
expect = "value" if containers[-1] == ord("[") else "key"
elif kind == 4 and expect == "colon":
expect = "value"
elif kind >= 5 and (expect == "value" or (kind == 5 and expect == "key")):
if build:
if expect == "key":
if deep_bad:
deep_key_at[-1] = string_at
else:
keys[-1] = value
else:
if kind == 6:
value = JSON_ATOMS.get(token[6], _INVALID_JSON)
if value is _INVALID_JSON:
value = _number_or_mark(token[6])
if deep_bad:
record_deep(value is _INEXACT)
else:
keep(value)
expect, empty = ("colon" if expect == "key" else "after"), False
else:
return _INVALID_JSON
if expect != "after" or containers or JSON_SPACE.fullmatch(text, at) is None:
return _INVALID_JSON
return root if build else True
def _json_view(text: str, size: int):
"""A JSON text's format and view, or why it has none, as REA reads it; None when it is not JSON. Text over the
parsed-view limit is told apart without building its value."""
if size > MAX_PARSED_BYTES:
valid = _read_json(text, build=False) is not _INVALID_JSON
return {"format": "json", "view": None, "error": PARSED_TOO_LARGE} if valid else None
# The whole text must parse, and duplicate keys settle, before an inexact number makes it JSON with an error.
view = _read_json(text)
if view is _INVALID_JSON:
return None
if _holds_inexact(view):
return {"format": "json", "view": None,
"error": "a number in it cannot be read exactly, such as an integer above 2^53"}
if _deeper_than(view, MAX_JSON_VIEW_DEPTH):
return {"format": "json", "view": None, "error": JSON_TOO_DEEP}
return {"format": "json", "view": view, "truncated": False}
`;
+20 -1
View File
@@ -21,11 +21,20 @@ const PATHS = [
];
it("matches the host path module for absolute local paths", () => {
const checked: string[] = [];
for (const path of PATHS) {
checked.push(path);
expect(absoluteLocalPathStringSchema.safeParse(path).success).toBe(
isAbsolute(path),
);
}
expect(checked).toEqual(
expect.arrayContaining([
"/absolute/path",
"relative/path",
"file:///absolute/path",
]),
);
expect(absoluteLocalPathStringSchema.safeParse("").success).toBe(false);
expect(
absoluteLocalPathStringSchema.safeParse("/absolute/with\0nul").success,
@@ -40,8 +49,18 @@ it("keeps the Windows absolute-path pattern valid in every client mode", () => {
() => new RegExp(windowsAbsolutePathPattern.source, flags),
flags,
).not.toThrow();
for (const path of PATHS)
const checked: string[] = [];
for (const path of PATHS) {
checked.push(path);
expect(windowsAbsolutePathPattern.test(path), path).toBe(
win32.isAbsolute(path),
);
}
expect(checked).toEqual(
expect.arrayContaining([
"C:\\absolute\\path",
"C:relative/path",
"/absolute/path",
]),
);
});
+2
View File
@@ -22,6 +22,7 @@ it.each(["0x20", "overlay:0x10", "invalid"])(
bytes: 16,
skipped_uninitialized: [],
skipped_not_loaded: [],
skipped_overlays: [],
},
complete: true,
next_cursor: null,
@@ -52,6 +53,7 @@ it.each(["", "overlay:"])(
bytes: 16,
skipped_uninitialized: [],
skipped_not_loaded: [],
skipped_overlays: [],
},
complete: true,
next_cursor: null,
+15 -2
View File
@@ -165,9 +165,22 @@ export const byteSearchResultSchema = z
blocks: z.array(coverageBlock),
bytes: z.int().nonnegative().max(Number.MAX_SAFE_INTEGER),
skipped_uninitialized: z.array(coverageBlock),
skipped_not_loaded: z.array(coverageBlock),
skipped_not_loaded: z
.array(coverageBlock)
.describe(
"Unloaded blocks skipped by an unbounded search, including unloaded overlays.",
),
skipped_overlays: z
.array(coverageBlock)
.describe(
"Loaded overlay blocks skipped by an unbounded search; select their address space with a bound.",
),
}),
complete: z.boolean(),
complete: z
.boolean()
.describe(
"No further matches remain in the eligible selection; skipped blocks were not searched.",
),
next_cursor: z.string().min(1).nullable(),
})
.superRefine((value, context) => {
+1
View File
@@ -70,6 +70,7 @@ const sharedAnalysisSemantics = {
instruction_flow_evidence: "decoded-return-pcode-v1",
instruction_status: "unlisted-address-without-decode-v1",
instruction_references: "exact-address-incoming-without-synthetic-entry-v3",
byte_search_coverage: "separate-loaded-overlays-v1",
function_boundary_observations: "ghidra-function-boundary-limitations-v4",
} as const;
+10
View File
@@ -367,6 +367,14 @@ const byteSearch = (space = "") => ({
bytes: 16,
},
],
skipped_overlays: [
{
segment: "overlay",
start: `${space}0x0040`,
end: `${space}0x004f`,
bytes: 16,
},
],
},
complete: true,
next_cursor: null as string | null,
@@ -392,6 +400,8 @@ it.each([
["uninitialized end", ["searched", "skipped_uninitialized", 0, "end"]],
["unloaded start", ["searched", "skipped_not_loaded", 0, "start"]],
["unloaded end", ["searched", "skipped_not_loaded", 0, "end"]],
["overlay start", ["searched", "skipped_overlays", 0, "start"]],
["overlay end", ["searched", "skipped_overlays", 0, "end"]],
] as const)(
"rejects noncanonical %s at its owning address field",
(_label, path) => {
+1
View File
@@ -445,6 +445,7 @@ const resultSchemas = {
blocks: z.array(canonicalByteCoverage),
skipped_uninitialized: z.array(canonicalByteCoverage),
skipped_not_loaded: z.array(canonicalByteCoverage),
skipped_overlays: z.array(canonicalByteCoverage),
}),
}),
inspect_native_load_image: nativeLoadImageObservationSchema,
+1 -1
View File
@@ -138,7 +138,7 @@ export const limitationsFor = (
case "search_bytes":
return [
...common,
"Searches contiguous eligible initialized loader-adjusted memory in provider address order, including across adjacent blocks. Bounds and page coverage ends are inclusive. Uninitialized blocks are listed separately and break search runs. Overlapping matches are included. A cursor resumes the same target, profile and query. Cancellation discards the active search and returns no successful partial page.",
"Searches contiguous eligible initialized loader-adjusted memory in provider address order, including across adjacent blocks. Bounds and page coverage ends are inclusive. Unbounded searches separately list skipped unloaded blocks and loaded overlays; use an overlay address-space bound to search it. Uninitialized blocks are listed separately and break search runs. Overlapping matches are included. A cursor resumes the same target, profile and query. Cancellation discards the active search and returns no successful partial page.",
];
case "address_to_file_offset":
return [
+3 -2
View File
@@ -11,9 +11,9 @@ import { GhidraProvider } from "./GhidraProvider.js";
// bit-field base-type revisions that prevent reuse of snapshots with earlier
// results.
const RAW_DIGESTS = {
"6502": "6f904539c00521b04f76a3bfd1ca88275e080b5751f9e2305de66c20d0bd2fa0",
"6502": "9f1632d3209c7cb3b17be1de28311aedf3a1b23d66ac45ac0006a4cd9a956ac1",
"x86-disabled-analyzer":
"5a534d959baf34abdafaea64ae9fcb9126553802d42ffa5e4bc0426de5f0af8c",
"ab9dd0676e0e647654b70c4dca25e1efeae3867c52623af0cd4a842f3f7aab68",
} as const;
const configFor = (environment: NodeJS.ProcessEnv) => {
@@ -50,6 +50,7 @@ describe("raw-binary analysis profile digest", () => {
"source-mappings-entry-context-relocation-symbol-v3",
instruction_status: "unlisted-address-without-decode-v1",
byte_search_boundary: "contiguous-eligible-initialized-blocks-v1",
byte_search_coverage: "separate-loaded-overlays-v1",
data_type_observations: "bit-field-base-and-lookup-v1",
analyzer_preset: "ghidra-default",
});
+5
View File
@@ -307,6 +307,11 @@ describe("Ghidra relationship coverage replay", () => {
describe("Ghidra snapshot profile admission", () => {
it.each([
{
parameter: "byte_search_coverage",
current: "separate-loaded-overlays-v1",
legacy: undefined,
},
{
parameter: "data_type_observations",
current: "bit-field-base-and-lookup-v1",
@@ -255,11 +255,11 @@ cliTest(
code: "invalid_request",
details: {
issues: [
{
expect.objectContaining({
path: ["executable_path"],
reason: "invalid_format",
message: expect.stringContaining("absolute local filesystem path"),
},
expected: "regex",
}),
],
},
});
@@ -4,6 +4,7 @@ import { join } from "node:path";
import { fileURLToPath } from "node:url";
import { promisify } from "node:util";
import { expect, it } from "vitest";
import { z } from "zod";
import { toolContract } from "../../../src/contracts/toolContracts.js";
import { createEvidence, parseEvidence } from "../../../src/domain/evidence.js";
@@ -329,21 +330,84 @@ const pathRequests: readonly [string, Record<string, unknown>][] = [
["start_gdb_session", { inferior_tty: nul }],
];
const nulIssuePaths = (
value: unknown,
path: readonly (string | number)[] = [],
): (string | number)[][] => {
if (typeof value === "string") return value.includes("\0") ? [[...path]] : [];
if (Array.isArray(value))
return value.flatMap((item, index) =>
nulIssuePaths(item, [...path, index]),
);
if (typeof value === "object" && value !== null)
return Object.entries(value).flatMap(([key, item]) =>
nulIssuePaths(item, [...path, key]),
);
return [];
};
it("rejects OS-invalid local inputs through the real advertised MCP contracts", async () => {
const { call } = await connectLocalToolsMcp();
const checked: string[] = [];
for (const [name, arguments_] of pathRequests) {
const response = await call(name, arguments_);
expect(response.isError, name).toBe(true);
expect(JSON.stringify(response.content), name).toContain("NUL");
const paths = nulIssuePaths(arguments_);
expect(paths.length, name).toBeGreaterThan(0);
const error = parseMcpToolError(await call(name, arguments_)).error;
expect(error, name).toMatchObject({
code: "invalid_request",
details: { operation: name },
});
const issues = z
.object({
issues: z.array(
z.object({
path: z.array(z.union([z.string(), z.number()])),
reason: z.string(),
message: z.string(),
expected: z.unknown().optional(),
}),
),
})
.parse(error.details).issues;
expect(issues, name).toEqual(
expect.arrayContaining(
paths.map((path) =>
expect.objectContaining({
path,
reason: "invalid_format",
expected: "regex",
message: expect.stringMatching(/\bNUL\b|null byte|zero byte/i),
}),
),
),
);
checked.push(name);
}
expect(checked).toEqual(
expect.arrayContaining([
"extract_firmware",
"inspect_wasm_artifact",
"demangle_swift",
"capture_electron_scenario",
]),
);
const relative = await call("extract_firmware", {
path: "/tmp/firmware.bin",
output_directory: "relative-output",
});
expect(relative.isError).toBe(true);
expect(JSON.stringify(relative.content)).toContain(
"absolute local filesystem path",
);
expect(parseMcpToolError(relative).error).toMatchObject({
code: "invalid_request",
details: {
operation: "extract_firmware",
issues: [
expect.objectContaining({
path: ["output_directory"],
reason: "invalid_format",
expected: "regex",
}),
],
},
});
}, 60_000);
const ignoredOptions = [
@@ -55,10 +55,7 @@ const singleDefaultOptions: readonly {
{ command: "inspect-signal-decoders", options: ["--offset"] },
];
const optionLine = (
help: string,
option: string,
): string | undefined => {
const optionLine = (help: string, option: string): string | undefined => {
const line = help
.split(/\r?\n/u)
.find((candidate) => candidate.trimStart().startsWith(`${option} `));
@@ -80,7 +77,9 @@ describe("CLI help default annotations", () => {
}
const statements = line.match(/default/giu)?.length ?? 0;
if (statements !== 1)
repeated.push(`${command} ${option}: ${statements} default statements`);
repeated.push(
`${command} ${option}: ${statements} default statements`,
);
}
}
expect(repeated).toEqual([]);
@@ -13,7 +13,7 @@ interface PathInputCase {
readonly field: string;
readonly valid: Readonly<Record<string, unknown>>;
readonly invalid: Readonly<Record<string, unknown>>;
readonly issuePath?: readonly string[];
readonly issuePath: readonly string[];
}
const createMcp = async (
@@ -104,7 +104,13 @@ const inputCases = (
field: string,
valid: Readonly<Record<string, unknown>>,
invalid: Readonly<Record<string, unknown>>,
): PathInputCase => ({ name, field, valid, invalid });
): PathInputCase => ({
name,
field,
valid,
invalid,
issuePath: field.split("."),
});
return [
pathCase("open_binary", "path", { path: absolute }, { path: relative }),
@@ -246,8 +252,7 @@ it("advertises full host-absolute path constraints after SDK schema conversion",
const ajv = new Ajv2020({ strict: false, validateFormats: false });
const absolute = resolve("rea-absolute-input");
const relative = "relative-input";
for (const { name, field, valid, invalid } of [
const schemaCases = [
...inputCases(absolute, relative),
{
name: "inspect_go_binary",
@@ -276,7 +281,9 @@ it("advertises full host-absolute path constraints after SDK schema conversion",
right_bundle_path: absolute,
},
},
]) {
];
const checked: string[] = [];
for (const { name, field, valid, invalid } of schemaCases) {
const schema = listed.tools.find(
(tool) => tool.name === name,
)?.inputSchema;
@@ -288,7 +295,16 @@ it("advertises full host-absolute path constraints after SDK schema conversion",
validate(valid),
`${name}.${field}: ${JSON.stringify(validate.errors)}`,
).toBe(true);
checked.push(`${name}.${field}`);
}
expect(checked).toEqual(
expect.arrayContaining([
"open_binary.path",
"extract_firmware.output_directory",
"inspect_go_binary.path NUL",
"compare_bundles.left_bundle_path empty",
]),
);
expect(firmwareCalls).toEqual([]);
} finally {
await close();
@@ -299,19 +315,39 @@ it("rejects relative paths through in-memory MCP handlers", async () => {
const firmwareCalls: string[] = [];
const { client, close } = await createMcp(firmwareCalls);
try {
for (const { name, field, invalid, issuePath } of inputCases(
resolve("rea-absolute-input"),
"relative-input",
)) {
const cases = inputCases(resolve("rea-absolute-input"), "relative-input");
const checked: string[] = [];
for (const { name, field, invalid, issuePath } of cases) {
const result = await client.callTool({ name, arguments: invalid });
expect(result.isError, `${name}.${field}`).toBe(true);
expect(JSON.stringify(result), `${name}.${field}`).toContain("absolute");
if (issuePath !== undefined)
expect(parseMcpToolError(result).error.details).toMatchObject({
operation: name,
issues: [{ path: issuePath, reason: "invalid_format" }],
});
expect(parseMcpToolError(result).error, `${name}.${field}`).toMatchObject(
{
code: "invalid_request",
details: {
operation: name,
issues: [
expect.objectContaining({
path: issuePath,
reason: "invalid_format",
expected: "regex",
}),
],
},
},
);
checked.push(`${name}.${field}`);
}
expect(checked).toEqual(
expect.arrayContaining([
"open_binary.path",
"open_binary.snapshot_path",
"extract_firmware.path",
"extract_firmware.output_directory",
"inspect_analysis_view.source.path",
"capture_browser_scenario.browser.executable_path",
"capture_electron_scenario.application_root",
]),
);
expect(firmwareCalls).toEqual([]);
} finally {
await close();
@@ -75,6 +75,14 @@ it("retains byte-search coverage and rejects malformed pages before creating Evi
skipped_not_loaded: [
{ segment: ".symtab", start: "0x0", end: "0x17f", bytes: 384 },
],
skipped_overlays: [
{
segment: "overlay",
start: "overlay:0x401000",
end: "overlay:0x40100f",
bytes: 16,
},
],
},
complete: false,
next_cursor: "continuation",
@@ -116,6 +124,27 @@ it("retains byte-search coverage and rejects malformed pages before creating Evi
expect(parseMcpToolError(malformed)).toMatchObject({
error: { code: "unreadable_output" },
});
output = jsonValueSchema.parse({
...page,
searched: {
...page.searched,
skipped_overlays: [
{
segment: "overlay",
start: "overlay:0x401000",
end: "overlay:0x40100f",
bytes: 15,
},
],
},
});
const invalidOverlay = await harness.mcp.callTool({
name: "search_bytes",
arguments: { pattern: "48 ??" },
});
expect(parseMcpToolError(invalidOverlay)).toMatchObject({
error: { code: "unreadable_output" },
});
for (const block of [
{ segment: ".text", start: "0x1000", end: "0x1009", bytes: 1 },
{ segment: ".text", start: "0x1009", end: "0x1000", bytes: 1 },
@@ -101,6 +101,7 @@ describe("official operation arguments", () => {
bytes: 0,
skipped_uninitialized: [],
skipped_not_loaded: [],
skipped_overlays: [],
},
complete: true,
next_cursor: null,
+26 -4
View File
@@ -709,8 +709,19 @@ describe("session filesystem path boundaries over MCP", () => {
name: "export_evidence_bundle",
arguments: { path: "relative-bundle.json" },
});
expect(exported.isError, JSON.stringify(exported.content)).toBe(true);
expect(JSON.stringify(exported.content)).toContain("absolute");
expect(parseMcpToolError(exported).error).toMatchObject({
code: "invalid_request",
details: {
operation: "export_evidence_bundle",
issues: [
expect.objectContaining({
path: ["path"],
reason: "invalid_format",
expected: "regex",
}),
],
},
});
const opened = await mcp.callTool({
name: "open_binary",
@@ -721,8 +732,19 @@ describe("session filesystem path boundaries over MCP", () => {
name: "close_binary",
arguments: { snapshot_path: "relative-analysis.json" },
});
expect(closed.isError, JSON.stringify(closed.content)).toBe(true);
expect(JSON.stringify(closed.content)).toContain("absolute");
expect(parseMcpToolError(closed).error).toMatchObject({
code: "invalid_request",
details: {
operation: "close_binary",
issues: [
expect.objectContaining({
path: ["snapshot_path"],
reason: "invalid_format",
expected: "regex",
}),
],
},
});
await mcp.callTool({ name: "close_binary", arguments: {} });
}, 10_000);
});
+22 -14
View File
@@ -383,7 +383,7 @@ it("attributes rejected WASM paths to their selected fields before provider work
},
});
onTestFinished(harness.close);
for (const [input, paths] of [
const relativeCases = [
[{ path: "relative.wasm" }, [["path"]]],
[
{ path: value.artifact.path, glue_paths: ["relative.js"] },
@@ -412,7 +412,9 @@ it("attributes rejected WASM paths to their selected fields before provider work
},
[["path"], ["glue_paths", 0], ["candidate_paths", 0]],
],
] as const) {
] as const;
const checked: string[] = [];
for (const [input, paths] of relativeCases) {
const response = await harness.client.callTool({
name: "inspect_wasm_artifact",
arguments: input,
@@ -423,14 +425,19 @@ it("attributes rejected WASM paths to their selected fields before provider work
.object({ issues: z.array(z.unknown()) })
.parse(failure.details);
expect(issues).toEqual(
paths.map((path) => ({
path,
reason: "invalid_format",
expected: "regex",
message: "must be an absolute local filesystem path",
})),
paths.map((path) =>
expect.objectContaining({
path,
reason: "invalid_format",
expected: "regex",
}),
),
);
checked.push(paths.map((path) => path.join(".")).join("+"));
}
expect(checked).toEqual(
expect.arrayContaining(["path", "glue_paths.0", "candidate_paths.0"]),
);
expect(inspect).not.toHaveBeenCalled();
const valid = await harness.client.callTool({
name: "inspect_wasm_artifact",
@@ -760,12 +767,13 @@ it("rejects malformed paths, refuses WASM summary delivery before provider work
expect(
z.object({ issues: z.array(z.unknown()) }).parse(rejected.details).issues,
).toEqual(
[["path"], ["glue_paths", 0]].map((path) => ({
path,
reason: "invalid_format",
expected: "regex",
message: "must be an absolute local filesystem path",
})),
[["path"], ["glue_paths", 0]].map((path) =>
expect.objectContaining({
path,
reason: "invalid_format",
expected: "regex",
}),
),
);
expect(inspect).not.toHaveBeenCalled();
for (const input of [
@@ -0,0 +1,205 @@
import { execFileSync } from "node:child_process";
import { writeFile } from "node:fs/promises";
import { join } from "node:path";
import { expect, it } from "vitest";
import {
PYTHON_HELPERS,
PYTHON_IMPORTS,
} from "../../../src/domain/desktop/appExportPython.js";
import {
detectStateFormat,
textualView,
} from "../../../src/domain/desktop/stateFormats.js";
import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js";
const arrays = "[".repeat(1500) + "1" + "]".repeat(1500);
const objects = '{"a":'.repeat(1500) + "1" + "}".repeat(1500);
const cases = [
["discarded arrays", `{"value":${arrays},"value":"done"}`, true],
["discarded objects", `{"value":${objects},"value":"done"}`, true],
["escaped duplicate key", `{"val\\u0075e":${arrays},"value":"done"}`, true],
["retained deep value", `{"value":${arrays}}`, false],
[
"retained deep inexact number",
`{"value":${"[".repeat(600)}9007199254740993${"]".repeat(600)}}`,
false,
],
[
"deep overwritten inexact number",
`{"value":${"[".repeat(600)}{"x":9007199254740993,"x":1}${"]".repeat(600)}}`,
false,
],
[
"deep empty-key overwrite",
`{"value":${"[".repeat(600)}{"":9007199254740993,"":1}${"]".repeat(600)}}`,
false,
],
[
"deep multiple inexact keys with one overwritten",
`{"value":${"[".repeat(600)}{"a":9007199254740993,"b":9007199254740993,"a":1}${"]".repeat(600)}}`,
false,
],
[
"deep multiple inexact keys all overwritten",
`{"value":${"[".repeat(600)}{"a":9007199254740993,"b":9007199254740993,"a":1,"b":2}${"]".repeat(600)}}`,
false,
],
[
"deep inexact key replaced by inexact",
`{"value":${"[".repeat(600)}{"a":9007199254740993,"a":9007199254740995}${"]".repeat(600)}}`,
false,
],
[
"deep escaped duplicate overwritten",
`{"value":${"[".repeat(600)}{"\\u0061":9007199254740993,"a":1}${"]".repeat(600)}}`,
false,
],
[
"discarded deep inexact subtree",
`{"value":${"[".repeat(600)}9007199254740993${"]".repeat(600)},"value":"done"}`,
true,
],
["last admitted depth", "[".repeat(500) + "1" + "]".repeat(500), true],
["empty container at limit", "[".repeat(501) + "]".repeat(501), true],
["empty container over limit", "[".repeat(502) + "]".repeat(502), false],
["trailing comma", `{"value":${arrays},"value":"done",}`, false],
["missing value", `{"value":${arrays},"value":}`, false],
["wrong closing bracket", `{"value":${arrays},"value":"done"]`, false],
] as const;
it.each(cases)("reads JSON with %s", async (_name, text, readable) => {
const root = await createTestTempDirectory("desktop-json-depth-");
const script = join(root, "read.py");
const imports = PYTHON_IMPORTS.split("\n")
.filter((line) => !line.includes("agentenv_protocol"))
.join("\n");
await writeFile(
script,
`${imports}${PYTHON_HELPERS}
text = json.load(sys.stdin)
print(json.dumps(_text_view("state.json", text, len(text.encode("utf-8")))))
`,
);
const output = JSON.parse(
execFileSync(
process.platform === "win32" ? "python" : "python3",
["-I", "-B", "-X", "utf8", script],
{ input: JSON.stringify(text), encoding: "utf8" },
),
) as { format: string; view: unknown; error?: string };
const bytes = Buffer.from(text);
const format = detectStateFormat("state.json", bytes);
if (format !== "json" && format !== "text")
throw new Error(`Unexpected native format ${format}`);
const native = textualView(format, bytes);
expect(format === "json" && native.ok).toBe(readable);
expect(output.format).toBe(format);
if (native.ok) {
expect(output.error).toBeUndefined();
// Compare JSON values on their wire form, independent of object prototypes.
expect(JSON.stringify(output.view)).toBe(JSON.stringify(native.value));
if (
readable &&
(_name.startsWith("discarded") || _name === "escaped duplicate key")
)
expect(output.view).toEqual({ value: "done" });
} else {
expect(output.view).toBeNull();
expect(output.error).toBe(native.error);
}
});
it("bounds construction before refusing retained excessive JSON depth", async () => {
const root = await createTestTempDirectory("desktop-json-construction-");
const script = join(root, "bounded.py");
const imports = PYTHON_IMPORTS.split("\n")
.filter((line) => !line.includes("agentenv_protocol"))
.join("\n");
await writeFile(
script,
`${imports}${PYTHON_HELPERS}
import tracemalloc
text = "[" * 20_000 + "0" + "]" * 20_000
recursion_limit = sys.getrecursionlimit()
tracemalloc.start()
entry = _json_view(text, len(text))
peak = tracemalloc.get_traced_memory()[1]
tracemalloc.stop()
print(json.dumps({"entry": entry, "peak": peak, "recursion_unchanged": sys.getrecursionlimit() == recursion_limit}))
`,
);
const output = JSON.parse(
execFileSync(
process.platform === "win32" ? "python" : "python3",
["-I", "-B", "-X", "utf8", script],
{ encoding: "utf8" },
),
) as { entry: unknown; peak: number; recursion_unchanged: boolean };
expect(output.entry).toEqual({
format: "json",
view: null,
error: "nested deeper than the 500 levels a JSON view keeps",
});
expect(output.recursion_unchanged).toBe(true);
// Python heap only, excluding source construction and interpreter/OS memory.
// A value retained only to refuse it must not allocate every input container.
expect(output.peak).toBeLessThan(1024 * 1024);
});
it("bounds allocations when a very deep value is overwritten", async () => {
const root = await createTestTempDirectory("desktop-json-memory-");
const script = join(root, "read.py");
const imports = PYTHON_IMPORTS.split("\n")
.filter((line) => !line.includes("agentenv_protocol"))
.join("\n");
await writeFile(
script,
`${imports}\nimport tracemalloc\n${PYTHON_HELPERS}
text = '{"value":' + '[' * 200000 + '1' + ']' * 200000 + ',"value":"done"}'
tracemalloc.start()
result = _json_view(text, len(text))
_, peak = tracemalloc.get_traced_memory()
print(json.dumps({"view": result["view"], "peak": peak}))
`,
);
const measured = JSON.parse(
execFileSync(
process.platform === "win32" ? "python" : "python3",
["-I", "-B", script],
{ encoding: "utf8", timeout: 30000 },
),
) as { view: unknown; peak: number };
expect(measured.view).toEqual({ value: "done" });
expect(measured.peak).toBeLessThan(8 * 1024 * 1024);
}, 40000);
it("bounds a wide container already beyond the retained view depth", async () => {
const root = await createTestTempDirectory("desktop-json-wide-depth-");
const script = join(root, "read.py");
const imports = PYTHON_IMPORTS.split("\n")
.filter((line) => !line.includes("agentenv_protocol"))
.join("\n");
await writeFile(
script,
`${imports}\nimport tracemalloc\n${PYTHON_HELPERS}
text = '[' * 501 + ','.join(['"abc"'] * 600000) + ']' * 501
tracemalloc.start()
result = _json_view(text, len(text))
_, peak = tracemalloc.get_traced_memory()
print(json.dumps({"error": result.get("error"), "peak": peak}))
`,
);
const measured = JSON.parse(
execFileSync(
process.platform === "win32" ? "python" : "python3",
["-I", "-B", script],
{ encoding: "utf8", timeout: 30000 },
),
) as { error: string; peak: number };
expect(measured.error).toBe(
"nested deeper than the 500 levels a JSON view keeps",
);
expect(measured.peak).toBeLessThan(8 * 1024 * 1024);
}, 40000);
@@ -34,7 +34,7 @@ const execPython = (
try {
writeFileSync(script, args[codeIndex + 1]!, "utf8");
return execFileSync(
"python3",
process.platform === "win32" ? "python" : "python3",
[
"-X",
"utf8",
@@ -375,15 +375,16 @@ it("checks text for JSON the way REA does, token by token, without parsing it",
"-I",
"-B",
"-c",
`${imports}${PYTHON_HELPERS}\nprint(json.dumps([_is_json(text) for text in json.load(sys.stdin)]))`,
`${imports}${PYTHON_HELPERS}\ntexts = json.load(sys.stdin)\nprint(json.dumps({"syntax": [_read_json(text, build=False) is not _INVALID_JSON for text in texts], "built": [_read_json(text) is not _INVALID_JSON for text in texts]}))`,
],
{ input: JSON.stringify(texts), encoding: "utf8" },
),
) as boolean[];
) as { syntax: boolean[]; built: boolean[] };
const parsed = texts.map((text) => safeParseJson(text).ok);
expect(texts.map((text) => isJsonText(text))).toEqual(parsed);
expect(python).toEqual(parsed);
expect(python.syntax).toEqual(parsed);
expect(python.built).toEqual(parsed);
expect(parsed.filter(Boolean)).toHaveLength(10);
});
@@ -445,9 +446,13 @@ print(json.dumps([_text_view("settings.ini", text, len(text.encode()))["view"]["
const root = await createTestTempDirectory("rea-keyfile-python-");
const script = join(root, "fixture.py");
await writeFile(script, program, "utf8");
const output = execFileSync("python3", ["-I", "-B", "-X", "utf8", script], {
input: JSON.stringify(inputs),
encoding: "utf8",
});
const output = execFileSync(
process.platform === "win32" ? "python" : "python3",
["-I", "-B", "-X", "utf8", script],
{
input: JSON.stringify(inputs),
encoding: "utf8",
},
);
expect(JSON.parse(output)).toEqual(inputs.map(keyfileView));
});
+1
View File
@@ -636,6 +636,7 @@ const inventoryResultFor = (method, params) => {
},
],
skipped_not_loaded: [],
skipped_overlays: [],
},
complete: !paged,
next_cursor: paged ? "fixture-second-page" : null,