Compare commits

..
Author SHA1 Message Date
Kaoru0822 54906fdb1f fix(verify): check Android readiness repair guidance (#10242) 2026-11-12 20:40:42 -08:00
Kaoru0822 0084810ef7 fix(browser): bound cached responses with zero-padded lengths (#10237) 2026-11-12 20:40:26 -08:00
Kaoru0822 feca30c72c fix(go): validate string extents before budget admission (#10231) 2026-11-12 20:40:05 -08:00
Kaoru0822 913b029c96 fix(desktop): retain dconf failures without stderr (#10230) 2026-11-12 20:39:50 -08:00
Kaoru0822 bfecbb1d48 test(native): cover deadlines throughout signature inspection (#10217) 2026-11-12 20:39:31 -08:00
Kaoru0822 070cfce443 fix(ci): settle cancelled test-group descendants (#10229) 2026-11-12 20:38:32 -08:00
Kaoru0822 0c01517cc0 fix(pcap): disclose truncated DNS responses as partial (#10214)
* fix(pcap): disclose truncated DNS responses as partial

* fix(pcap): explain DNS truncation in coverage limitations

* fix(pcap): preserve mDNS TC semantics
2026-11-12 20:34:34 -08:00
Kaoru0822 7c9d65fe97 fix(test): resolve filesystem aliases before planning prerequisites (#10213) 2026-11-12 20:29:42 -08:00
ShuaixinHuangandShuaixinHuang 9c86465d47 fix(captures): retain preparation cleanup owners (#10288)
Co-authored-by: ShuaixinHuang <[email protected]>
2026-11-12 22:08:37 -05:00
morluto c836f57b7d refactor: consolidate shared analysis boundaries (#10293) 2026-11-12 22:05:32 -05:00
morluto 6878a84796 fix(tests): restore worker fixture and allow setup completion 2026-11-12 20:57:43 -05:00
Kaoru0822 937d10df1c fix(pe): enforce DER extension critical encoding (#10212) 2026-11-12 19:33:31 -05:00
Kaoru0822 c19d319929 fix(desktop): preserve literal directories in ELF origin expansion (#10215)
fix(desktop): expand ELF origin paths as literal directory names
2026-11-12 19:33:16 -05:00
Kaoru0822 966bde3780 fix(android): enforce DEX array dimension limit (#10227) 2026-11-12 19:29:53 -05:00
Kaoru0822 4d15b7f509 fix(android): preserve SDK-gated manifest permissions (#10236) 2026-11-12 19:29:40 -05:00
Kaoru0822 50d4f7bea4 fix(desktop): validate guest file replies at their shared boundary (#10224)
* fix(desktop): reject malformed guest file base64

* fix(desktop): scope guest file metadata validation to its namespace
2026-11-12 19:29:18 -05:00
Kaoru0822 594d73985c fix(desktop): preserve Unicode separators in entry values (#10210) 2026-11-12 19:29:05 -05:00
Kaoru0822 418e3f5e91 fix(release): honor selected npm in update verification (#10240) 2026-11-12 19:28:29 -05:00
Kaoru0822 0f5ac9d5f3 fix(desktop): preserve multiline apt-config locations (#10241) 2026-11-12 19:28:11 -05:00
2SinceJulyand2SinceJuly 7559e06740 fix(contracts): drop nested defaults from a root union sibling (#10254)
* fix(contracts): drop nested defaults from a root union sibling

Flattening a union of strict object groups lifts each group's properties
into one object. A default the scenario group carried inside
normalization.rules survived that lift even though the passive group
declares no normalization, so a client that fills nested defaults sent
{normalization: {rules: []}} and the runtime parser refused the call
with "Unrecognized key: normalization".

Remove defaults below a property that any group omits, not just the one
on the property itself. A property every group owns keeps its own
nested guidance.

* fix(contracts): keep a nested default a reference still targets

Dropping the nested default of a property a sibling group omits also removed defaults that references point into. A resource whose validation $ref targets its own default was advertised with that reference intact but its target gone, so the advertised schema no longer compiled in AJV or client projection.

Collect the paths into a default that references inside the branch target and keep those entries while stripping the rest. Other nested defaults are still removed, so the strict sibling group keeps refusing the object.

* test(contracts): type the root union default regression fixture

The fixture spread the canonical branch through a widened cast, so the repository typecheck rejected the spread and indexed access. Narrow the branch shape and guard the optional schema before collecting its referenced defaults.

---------

Co-authored-by: 2SinceJuly <[email protected]>
2026-11-12 19:27:58 -05:00
ShuaixinHuangandShuaixinHuang 9d32836d37 fix(apple): expand loaded siblings along the first dyld chain (#10248)
* fix(apple): expand loaded siblings along the first dyld chain

* fix(apple): retain conditional incoming dyld expansion edges

* fix(apple): qualify findings by their expansion chain

---------

Co-authored-by: ShuaixinHuang <[email protected]>
2026-11-12 19:27:40 -05:00
morluto 02ebd3e02d refactor: prune test-only seams and obsolete helpers (#10275) 2026-11-12 19:24:54 -05:00
morluto a8527d43b8 fix(setup): supply registration profile in catalog test 2026-11-12 11:47:18 -05:00
409 changed files with 5244 additions and 2744 deletions

No files matched your search

+3 -1
View File
@@ -464,7 +464,9 @@ class ReaJadxBridge {
if (element.getTagName().equals("uses-sdk")) {
minSdk = element.getAttributeNS(android, "minSdkVersion");
targetSdk = element.getAttributeNS(android, "targetSdkVersion");
} else if (element.getTagName().equals("uses-permission")) {
} else if (element.getTagName().equals("uses-permission")
|| element.getTagName().equals("uses-permission-sdk-23")
|| element.getTagName().equals("uses-permission-sdk-m")) {
String permission = element.getAttributeNS(android, "name");
if (!permission.isEmpty()) permissions.add(permission);
}
-9
View File
@@ -958,15 +958,6 @@ class _BridgeServer(threading.Thread):
proof.update(fragment.encode("utf-8"))
return proof.hexdigest()
def _encode_authenticated_response(
self, value: dict[str, Any], client_nonce: str, server_nonce: str
) -> bytes | None:
"""Return one exact frame. The socket path writes the same pieces."""
proof = self._authenticated_proof(value, client_nonce, server_nonce)
if proof is None:
return None
return b"".join(self._iter_authenticated_frame(value, proof))
def _iter_authenticated_frame(self, value: dict[str, Any], proof: str):
yield _AUTHENTICATED_PREFIX
for fragment in _response_fragments(value):
-7
View File
@@ -154,13 +154,6 @@ class HopperApiFacade:
)
def _configure_hopper_api(document_provider):
"""Inject Hopper's document provider or a standalone test facade."""
global _hopper_api, _rea_session_document
_hopper_api = HopperApiFacade(document_provider)
_rea_session_document = None
def _api():
"""Resolve Hopper globals lazily so importing this module needs no Hopper."""
global _hopper_api
@@ -215,7 +215,10 @@ Resolution follows dyld:
image that loaded it, up to the process root.
- Dependencies load dependents-first in load-command order, and each image
loads once per process. A later @rpath request whose install name matches an
already loaded image reuses it.
already loaded image reuses it. Loading an image and expanding its dependencies
are separate: an already loaded sibling can first be expanded through another
dependency, inheriting that traversal's rpath stack. Cycles do not expand an
image twice, and lazy edges do not establish a launch-time traversal chain.
- Symlinks such as `Versions/Current` are followed inside the analyzed root.
An escape through `..` or an absolute link is `escapes-target`.
- Dyld path components are POSIX names. On Windows hosts, link text uses `\`
@@ -252,8 +255,12 @@ one of:
A `conditional` image is still traced, because it is an in-root candidate
that may load. Every edge below it has `loader_conditional: true`, and a
request that reuses it by install name is `conditional` too. An in-root dylib
with no platform load command remains unknown unless the root is an executable
request that reuses it by install name is `conditional` too. A definitely loaded
image can also first expand through a conditional chain: its dependency edges
then have `loader_conditional: true`, while later reuse of the image retains its
definite first-load result.
An in-root dylib with no platform load command remains unknown unless the root is an executable
with observed macOS platform and the candidate's CPU type is x86_64, i386, or arm64. In that macOS context,
[Apple's dyld](https://github.com/apple-oss-distributions/dyld/blob/fd8d0c4d52320ebf64db34f3cb280310d905c5ae/mach_o/UnsafeHeader.cpp#L1137-L1145) supplies an implicit macOS platform to legacy dylibs on those CPU
types. The candidate records `platform_basis: "macos-dyld-implicit"`; the
+3 -8
View File
@@ -222,7 +222,7 @@ cannot answer, such as an address without one exact file offset, omits this
hint; follow its `remediation.action`. `recovery.kind=inspect_timeout_state` names
the operation and the REA server/provider deadline. Inspect partial Evidence
and reported cleanup state, then follow the returned remediation. Change a
request field or REA_* setting only when the returned remediation explicitly
request field or REA\_\* setting only when the returned remediation explicitly
recommends that change. Restart REA when that text says the value is read at
startup. A mention that a control does not raise the deadline is not a
recommendation to change it. When it names no setting, reduce supported work;
@@ -1911,13 +1911,10 @@ support for omitted historical values.
known and differ. If only one value is known, storage comparison is unknown;
when both are omitted, storage is not compared.
<!-- BEGIN:compare-artifacts:compared-format-metadata-kinds -->
`compare_artifacts` also compares the declarations in the `format_metadata`
of UnityFS members (`unityfs-entry`), PyInstaller entries
`compare_artifacts` also compares the declarations in the `format_metadata` of <!-- BEGIN:compare-artifacts:compared-format-metadata-kinds -->UnityFS members (`unityfs-entry`), PyInstaller entries
(`pyinstaller-entry`), Node SEA members (`node-sea-entry`),
ESP image segments (`esp-image-segment`), DfuSe elements (`dfuse-element`),
ESP flash partitions (`esp-flash-partition`) and FIT images (`fit-image`);
ESP flash partitions (`esp-flash-partition`) and FIT images (`fit-image`)<!-- END:compare-artifacts:compared-format-metadata-kinds -->;
each format's guide names the compared fields. A changed declaration is a
metadata change, including when content is unavailable. A declaration that
only one side records, or that the other side records as another kind, is
@@ -1928,8 +1925,6 @@ container or from caller options that identify the target, or repeat
occurrence fields, except Godot PCK source paths, which
[Godot PCK archives](../godot-pck.md) explains.
<!-- END:compare-artifacts:compared-format-metadata-kinds -->
Every occurrence records `entry_kind`, `executable` and `encrypted`. A
different entry kind or `encrypted` value is a `metadata` difference, and a
different `executable` value is an `executable` difference. Occurrence and node
@@ -10,8 +10,8 @@
4. Verifier: `appTaskPython.ts` renders `READER_RUNNER`, `READERS`,
`_reader_views` and `evaluate_with_readers` only for a task with readers,
so a task without renders as before.
5. Application: `readTaskReaderViews` and `readerViewsOfBytes` in
`DesktopTaskReaders.ts`; `runReaderHere` in `src/desktop/TaskReaderProcess.ts`;
5. Application: `readerViewsOfBytes` in `DesktopTaskReaders.ts`;
`runReaderHere` in `src/desktop/TaskReaderProcess.ts`;
the witness runs readers through `AgentEnvComputer.exec`; the export runs
them on `shippedSeedBytes` and the end's retained content.
6. Docs: the tasks reference, the witness page, the build skill's recipes for
+5
View File
@@ -19,6 +19,11 @@ if (
const child = spawn(
process.execPath,
[
// Reuse the development runner's POSIX process-group owner. Windows
// source tests must remain usable without the built native Job Object.
...(process.platform === "win32"
? []
: [resolve("scripts/run-exclusive.mjs"), "--unlocked", process.execPath]),
resolve("node_modules/vitest/vitest.mjs"),
"run",
...projects.flatMap(({ name }) => ["--project", name]),
+1
View File
@@ -15,6 +15,7 @@ export function parseDevelopmentTestRequest(
export function developmentTestPlan(
request: DevelopmentTestRequest,
baseCommit?: string,
resolvedPaths?: readonly string[],
): {
readonly needsBuild: boolean;
readonly artifactTasks: readonly string[];
+7 -3
View File
@@ -58,7 +58,11 @@ const artifactConsumers = {
};
/** Plan source feedback or explicit tests with their runtime/artifact prerequisites. */
export const developmentTestPlan = (request, baseCommit) => {
export const developmentTestPlan = (
request,
baseCommit,
resolvedPaths = request.paths,
) => {
const explicit = request.paths.length > 0;
if (
request.mode === "changed" &&
@@ -70,12 +74,12 @@ export const developmentTestPlan = (request, baseCommit) => {
? []
: SOURCE_TEST_PROJECTS.map(({ name }) => name);
return {
needsBuild: request.paths.some(
needsBuild: resolvedPaths.some(
(path) => projectForTest(path).group === "built",
),
artifactTasks: Object.entries(artifactConsumers).flatMap(
([task, consumers]) =>
request.paths.some((path) => consumers.includes(path)) ? [task] : [],
resolvedPaths.some((path) => consumers.includes(path)) ? [task] : [],
),
vitestArguments: [
"run",
+5 -2
View File
@@ -10,7 +10,6 @@ import { access, readFile, readdir, stat } from "node:fs/promises";
import { dirname, join } from "node:path";
import { setTimeout as delay } from "node:timers/promises";
import { promisify } from "node:util";
import { GHIDRA_ANNOTATIONS_RETAINED } from "../../dist/ghidra/GhidraProviderClient.js";
import Ajv from "ajv/dist/2020.js";
import addFormats from "ajv-formats";
import { OFFICIAL_TOOL_CONTRACTS } from "../../dist/contracts/officialToolContracts.js";
@@ -571,7 +570,11 @@ export async function verifyGhidraBoundaries(
// The producer reason describes the failure; continuing-session guidance
// is projected separately and the following live calls verify it.
assert.match(regexError.details.reason, /stack while matching pattern/u);
assert.ok(regexError.message.includes(GHIDRA_ANNOTATIONS_RETAINED));
assert.ok(
regexError.message.includes(
"The active analysis session and annotations remain available.",
),
);
rejectedCalls++;
const nestedPattern = "(".repeat(12000) + "a" + ")".repeat(12000);
const compileFailure = await client.callTool(
+17 -3
View File
@@ -1,12 +1,13 @@
import { spawn } from "node:child_process";
import { stat } from "node:fs/promises";
import { resolve } from "node:path";
import { realpath, stat } from "node:fs/promises";
import { relative, resolve, sep } from "node:path";
import { fileURLToPath } from "node:url";
import {
developmentTestPlan,
parseDevelopmentTestRequest,
} from "./lib/development-tests.mjs";
import { projectForTest } from "./lib/test-projects.mjs";
let child;
let forceTimer;
@@ -29,9 +30,22 @@ for (const signal of Object.keys(signalExitCodes))
try {
const [mode, ...arguments_] = process.argv.slice(2);
const request = parseDevelopmentTestRequest(mode, arguments_);
const resolvedPaths = [];
const repository = await realpath(".");
for (const path of request.paths) {
if (!(await stat(path)).isFile())
throw new Error(`Test path is not a file: ${path}`);
// Ownership follows the filesystem's canonical spelling. Execution keeps
// the caller path so exact discovery still rejects excluded symlink aliases.
const resolvedPath = relative(repository, await realpath(path))
.split(sep)
.join("/");
if (
request.mode !== "focused" &&
projectForTest(resolvedPath).group !== "source"
)
throw new Error(`Use test:focused for built test projects: ${path}`);
resolvedPaths.push(resolvedPath);
}
let baseCommit;
if (request.mode === "changed") {
@@ -42,7 +56,7 @@ try {
);
baseCommit = base.output.trim();
}
const plan = developmentTestPlan(request, baseCommit);
const plan = developmentTestPlan(request, baseCommit, resolvedPaths);
if (request.dryRun) {
console.log(JSON.stringify({ ...request, baseCommit, ...plan }, null, 2));
} else {
+3 -5
View File
@@ -99,8 +99,7 @@ async function verifyPublishedUpdate({
environment,
);
await commands.exec(
"npm",
await commands.execNpm(
[
"install",
"--global",
@@ -118,7 +117,7 @@ async function verifyPublishedUpdate({
);
const npmRoot = (
await commands.exec("npm", ["root", "--global"], {
await commands.execNpm(["root", "--global"], {
cwd: canaryRoot,
env: environment,
timeout: 30000,
@@ -186,8 +185,7 @@ if (result.status === "failed") process.exitCode = 1;
}
async function findPreviousPublishedVersion(targetVersion, cwd, environment) {
const { stdout } = await commands.exec(
"npm",
const { stdout } = await commands.execNpm(
["view", "rea-agents", "versions", "--json"],
{ cwd, env: environment, timeout: 30000, maxBuffer: 16 * 1024 * 1024 },
);
+2 -7
View File
@@ -151,7 +151,7 @@ export async function verifyAndroidFailureRecovery({
failure_code: "runtime_unavailable",
},
availabilityReason: "provider_unavailable",
availabilityRemediation: /selected Java runtime/u,
availabilityRemediation: /selected Java launcher/u,
message: /selected Java runtime/u,
remediation: /selected launcher/u,
});
@@ -194,12 +194,7 @@ export async function verifyAndroidFailureRecovery({
repository,
available: false,
unavailableReason: scenario.availabilityReason ?? "provider_missing",
remediation:
scenario.availabilityRemediation ??
new RegExp(
scenario.observation.replace(/[.*+?^${}()|[\]\\]/gu, "\\$&"),
"u",
),
remediation: scenario.availabilityRemediation ?? scenario.remediation,
});
console.log(`PASS CLI applicable recovery: ${scenario.label}`);
}
+12 -16
View File
@@ -58,7 +58,7 @@ export const androidSnapshotCopy = (
: undefined;
/** Adopt a snapshot descriptor whose close rejected, keyed by its exact path. */
export type AndroidSnapshotCloseRetention = (
type AndroidSnapshotCloseRetention = (
path: string,
owner: OwnedFileHandle,
) => void;
@@ -69,8 +69,8 @@ export const snapshotAndroidTarget = async (
sha256: string,
root: string,
operation: AndroidOperation,
signal?: AbortSignal,
retainFailedClose?: AndroidSnapshotCloseRetention,
signal: AbortSignal | undefined,
retainFailedClose: AndroidSnapshotCloseRetention,
): Promise<string> => {
const snapshot = join(root, "target.apk");
try {
@@ -105,8 +105,8 @@ export const snapshotAndroidEngine = async (
sha256: string,
root: string,
operation: AndroidOperation,
signal?: AbortSignal,
retainFailedClose?: AndroidSnapshotCloseRetention,
signal: AbortSignal | undefined,
retainFailedClose: AndroidSnapshotCloseRetention,
): Promise<{ path: string; sha256: string }> => {
const path = join(root, "engine.jar");
try {
@@ -159,17 +159,14 @@ const copyAndroidSnapshot = async (
snapshotPath: string,
expectedSha256: string,
signal: AbortSignal | undefined,
retainFailedClose: AndroidSnapshotCloseRetention | undefined,
retainFailedClose: AndroidSnapshotCloseRetention,
): Promise<AndroidSnapshotCopy> => {
assertRegularFileActive(signal);
let removeSnapshot = false;
const retain = (path: string) =>
retainFailedClose === undefined
? {}
: {
retainFailedClose: (owner: OwnedFileHandle) =>
retainFailedClose(path, owner),
};
const retain = (path: string) => ({
retainFailedClose: (owner: OwnedFileHandle) =>
retainFailedClose(path, owner),
});
try {
// Close each descriptor through its owner. A rejected close keeps the
// copy outcome and stays retained until that owner is retried; the
@@ -287,7 +284,7 @@ const openSnapshotSource = async (
path: string,
signal: AbortSignal | undefined,
retention: {
readonly retainFailedClose?: (owner: OwnedFileHandle) => void;
readonly retainFailedClose: (owner: OwnedFileHandle) => void;
},
): Promise<StableRegularFileDescriptor> => {
try {
@@ -305,10 +302,9 @@ const openSnapshotSource = async (
};
const retainAdmissionOwner = (
retain: ((owner: OwnedFileHandle) => void) | undefined,
retain: (owner: OwnedFileHandle) => void,
owner: OwnedFileHandle,
): RegularFileCleanupFailure["retention"] => {
if (retain === undefined) return { kind: "not_requested" };
try {
retain(owner);
return { kind: "callback_returned" };
-9
View File
@@ -6,7 +6,6 @@ import {
resolveWindowsChangeTimeSettle,
} from "../config/windowsChangeTimeSettle.js";
import { pwntoolsInputBytes } from "../native/pwntools/PwntoolsInputLimit.js";
import { resolveBridgeMaxRequestBytes } from "../config/bridgeMaxRequestBytes.js";
import { resolveCaptureDecoderHeap } from "../config/captureDecoderHeap.js";
import { resolveRegexMatchingDeadline } from "../config/regexMatchingDeadline.js";
import { resolveRegexWorkerHeapRequest } from "../config/regexWorkerHeapRequest.js";
@@ -366,7 +365,6 @@ it("reads an accepted override before an invalid launch seed", () => {
{
revision: 1,
overrides: {
"bridge.max_request_bytes": 8192,
"capture.decoder_old_generation_mib": 64,
"regex.matching_deadline_ms": 250,
"regex.worker_heap_request_mib": 128,
@@ -375,9 +373,6 @@ it("reads an accepted override before an invalid launch seed", () => {
},
},
() => ({
bridge: resolveBridgeMaxRequestBytes({
REA_BRIDGE_MAX_REQUEST_BYTES: "nope",
}),
capture: resolveCaptureDecoderHeap({
REA_CAPTURE_DECODER_OLD_GENERATION_MIB: "0",
}),
@@ -393,10 +388,6 @@ it("reads an accepted override before an invalid launch seed", () => {
}),
}),
);
expect(resolved.bridge).toMatchObject({
maximumBytes: 8192,
source: "runtime",
});
expect(resolved.capture).toMatchObject({
oldGenerationMiB: 64,
source: "runtime",
+5 -9
View File
@@ -86,17 +86,13 @@ describe("app bundle Info.plist magic", () => {
const plist = Buffer.from("bplist00");
plist[0] = (plist[0] ?? 0) | 0x80;
await writeFile(plistPath, plist);
let decoded = false;
const result = await resolveAppBundleExecutable(bundle, {
readFile: async () => plist,
realpath: async (path) => path,
decodeBinaryPlist: async () => {
decoded = true;
return "Fixture";
const result = await resolveAppBundleExecutable(bundle);
expect(result).toMatchObject({
ok: false,
error: {
cause: { name: "XmlPlistTextDecodeError", reason: "invalid-bytes" },
},
});
expect(decoded).toBe(false);
expect(result.ok).toBe(false);
} finally {
await rm(root, { recursive: true, force: true });
}
+2 -20
View File
@@ -91,7 +91,6 @@ export interface ResolvedAppBundleResourceRoot {
export interface AppBundleFileSystem {
readonly readFile: (path: string) => Promise<Buffer>;
readonly realpath: (path: string) => Promise<string>;
readonly decodeBinaryPlist?: (path: string) => Promise<string>;
}
const defaultFileSystem: AppBundleFileSystem = { readFile, realpath };
@@ -396,15 +395,6 @@ const codeDescription = (cause: unknown): string => {
: "";
};
const isSystemErrorCode = (cause: unknown): boolean => {
const code = errorCode(cause);
return (
typeof code === "string" &&
/^[A-Z][A-Z0-9_]+$/u.test(code) &&
!code.startsWith("ERR_")
);
};
const isPermissionDenied = (cause: unknown): boolean =>
errorCode(cause) === "EACCES" || errorCode(cause) === "EPERM";
@@ -461,20 +451,12 @@ const resolveLayoutExecutable = async (
let name: string;
try {
name =
hasBinaryPlistSignature(plist) &&
fileSystem.decodeBinaryPlist !== undefined
? await fileSystem.decodeBinaryPlist(plistPath)
: executableName(decodeInfoPlistValue(plist));
name = executableName(decodeInfoPlistValue(plist));
} catch (cause: unknown) {
return err(
new BinaryTargetError(
plistPath,
isPermissionDenied(cause)
? `permission denied decoding ${noun} Info.plist ${plistPath}${codeDescription(cause)}`
: isSystemErrorCode(cause)
? `could not decode ${noun} Info.plist ${plistPath}${codeDescription(cause)}`
: `${noun} Info.plist is malformed, unsupported, or lacks CFBundleExecutable: ${plistPath}${cause instanceof Error ? ` (${cause.message})` : ""}`,
`${noun} Info.plist is malformed, unsupported, or lacks CFBundleExecutable: ${plistPath}${cause instanceof Error ? ` (${cause.message})` : ""}`,
{
cause,
// A bundle without code, such as a resource bundle, declares no
+2 -2
View File
@@ -747,8 +747,8 @@ const throwIfTargetResolutionCancelled = (signal?: AbortSignal): void => {
const detectArtifactFormat = async (
path: string,
source: StableRegularFileDescriptor,
signal?: AbortSignal,
sourcePath = path,
signal: AbortSignal | undefined,
sourcePath: string,
): Promise<
| Exclude<
BinaryTarget["format"],
+3 -3
View File
@@ -23,14 +23,14 @@ export interface ClientConfigurationSource extends ClientConfigurationDocument {
const invalidEnvironment = (
path: string,
error: ConfigurationError,
overrides: Readonly<Record<string, string>> = {},
platform: NodeJS.Platform = process.platform,
overrides: Readonly<Record<string, string>> | undefined,
platform: NodeJS.Platform,
): ClientConfigurationFileError => {
const constraints = error.settings
.map(({ setting, constraint }) => `${setting}: ${constraint}`)
.join("; ");
const overrideNames = new Set(
Object.keys(overrides).map((name) =>
Object.keys(overrides ?? {}).map((name) =>
platform === "win32" ? name.toUpperCase() : name,
),
);
@@ -1,15 +1,9 @@
import { validateRegistrationEnvironment } from "./RegistrationEnvironmentValidation.js";
import { clientRegistrationSource } from "./ClientRegistrationSource.js";
import type { SetupClient } from "./SupportedClients.js";
import {
inspectAnalysisSettingSeeds,
launchAnalysisSetting,
} from "../config/analysisSettings.js";
import {
parseEnvironment,
parseEnvironmentValues,
type Environment,
} from "../config/environment.js";
import { launchAnalysisSetting } from "../config/analysisSettings.js";
import type { Environment } from "../config/environment.js";
import {
DEFAULT_GHIDRA_STARTUP_TIMEOUT_MS,
DEFAULT_HOPPER_STARTUP_TIMEOUT_MS,
@@ -314,34 +308,24 @@ export const clientRegistrationEnvironmentDetailsWithBudget = async (
const validatedDetails = (
environment: Readonly<Record<string, string>>,
validation: "fragment" | "effective",
platform: NodeJS.Platform = process.platform,
expressionSettings: ReadonlySet<string> = new Set(),
platform: NodeJS.Platform,
expressionSettings: ReadonlySet<string>,
): Result<ClientRegistrationEnvironmentDetails, ConfigurationError> => {
const runtimeEnvironment = snapshotEnvironment(environment, platform);
const unresolved = new Set(
Object.keys(
snapshotEnvironment(
Object.fromEntries([...expressionSettings].map((name) => [name, ""])),
platform,
),
),
const validated = validateRegistrationEnvironment(
environment,
expressionSettings,
validation,
platform,
);
const validated =
validation === "fragment"
? parseEnvironmentValues(runtimeEnvironment, unresolved)
: parseEnvironment(runtimeEnvironment, unresolved);
if (!validated.ok) return validated;
const seedIssues = inspectAnalysisSettingSeeds(
runtimeEnvironment,
unresolved,
);
if (seedIssues.length > 0)
return err(
new ConfigurationError("Invalid REA environment configuration", {
settings: seedIssues,
}),
);
return ok(environmentDetails(environment, validated.value, unresolved));
return validated.ok
? ok(
environmentDetails(
environment,
validated.value.configuration,
validated.value.unresolvedSettings,
),
)
: validated;
};
const admittedTimeout = (
@@ -397,7 +381,7 @@ const registrationTimeoutInputs = (
const environmentDetails = (
environment: Readonly<Record<string, string>>,
configuration: Environment,
unresolvedSettings: ReadonlySet<string> = new Set(),
unresolvedSettings: ReadonlySet<string>,
): ClientRegistrationEnvironmentDetails => ({
values: Object.fromEntries(
Object.entries(environment).sort(([left], [right]) =>
+1 -1
View File
@@ -649,7 +649,7 @@ const packageVersionRemediation = (
const unavailableStatus = (
client: SetupClient,
state: "missing" | "invalid",
configPath: string = client.configPath,
configPath: string,
): ClientRegistrationStatus => ({
client: client.name,
config_path: configPath,
+11 -38
View File
@@ -1,18 +1,11 @@
import { inspectAnalysisSettingSeeds } from "../config/analysisSettings.js";
import { validateRegistrationEnvironment } from "./RegistrationEnvironmentValidation.js";
import { isMap, isSeq, type Node, type YAMLMap } from "yaml";
import {
parseEnvironment,
parseEnvironmentValues,
type Environment,
} from "../config/environment.js";
import type { Environment } from "../config/environment.js";
import { ConfigurationError } from "../domain/configurationErrors.js";
import { err, ok, type Result } from "../domain/result.js";
import type { ClientConfigurationDocument } from "./ClientConfigurationDocument.js";
import { CordisExpression, DshCordisYaml } from "./DshCordisYaml.js";
import {
mergeEnvironmentOverrides,
snapshotEnvironment,
} from "../process/snapshotEnvironment.js";
import { mergeEnvironmentOverrides } from "../process/snapshotEnvironment.js";
export const DSH_ENTRY_ID = "rea-mcp";
export const DSH_CLIENT_PACKAGE = "@deepseek-ai/dsh-mcp-client";
@@ -255,35 +248,15 @@ export class DshCordisDocument {
);
literals[key] = value;
}
const runtimeEnvironment = snapshotEnvironment(literals, platform);
const unresolved = new Set(
Object.keys(
snapshotEnvironment(
Object.fromEntries([...unresolvedSettings].map((key) => [key, ""])),
platform,
),
),
const validated = validateRegistrationEnvironment(
literals,
unresolvedSettings,
validation,
platform,
);
const validated =
validation === "fragment"
? parseEnvironmentValues(runtimeEnvironment, unresolved)
: parseEnvironment(runtimeEnvironment, unresolved);
if (!validated.ok) return validated;
const seedIssues = inspectAnalysisSettingSeeds(
runtimeEnvironment,
unresolved,
);
if (seedIssues.length > 0)
return err(
new ConfigurationError("Invalid REA environment configuration", {
settings: seedIssues,
}),
);
return ok({
values: literals,
configuration: validated.value,
unresolvedSettings: unresolved,
});
return validated.ok
? ok({ values: literals, ...validated.value })
: validated;
}
desired(
+4 -18
View File
@@ -1,3 +1,4 @@
import { readElfAttributeUleb } from "../domain/elfAttributeUleb.js";
import type { MipsGnuAttributes } from "../domain/binaryTargetTypes.js";
import { err, ok, type Result } from "../domain/result.js";
@@ -22,21 +23,6 @@ export const parseMipsGnuAttributes = (
const readU32 = (offset: number): number =>
littleEndian ? bytes.readUInt32LE(offset) : bytes.readUInt32BE(offset);
const readUleb = (
offset: number,
end: number,
): { readonly value: number; readonly next: number } | undefined => {
let value = 0;
for (let index = 0; index < 5 && offset + index < end; index += 1) {
const byte = bytes[offset + index]!;
value += (byte & 0x7f) * 2 ** (7 * index);
if ((byte & 0x80) === 0)
return value <= 0xffffffff
? { value, next: offset + index + 1 }
: undefined;
}
return undefined;
};
let fpAbi: number | null = null;
let msa: number | null = null;
@@ -65,7 +51,7 @@ export const parseMipsGnuAttributes = (
let cursor = vendorEnd + 1;
while (cursor < end) {
const scopeStart = cursor;
const scope = readUleb(cursor, end);
const scope = readElfAttributeUleb(bytes, cursor, end);
if (scope === undefined)
return err("malformed MIPS GNU attribute scope tag");
if (scope.value !== TAG_FILE)
@@ -85,7 +71,7 @@ export const parseMipsGnuAttributes = (
const scopeEnd = scopeStart + scopeLength;
cursor = scope.next + 4;
while (cursor < scopeEnd) {
const tag = readUleb(cursor, scopeEnd);
const tag = readElfAttributeUleb(bytes, cursor, scopeEnd);
if (tag === undefined) return err("malformed MIPS GNU attribute tag");
if (
tag.value !== TAG_GNU_MIPS_ABI_FP &&
@@ -94,7 +80,7 @@ export const parseMipsGnuAttributes = (
return err(
`MIPS ELF declares GNU attribute tag ${tag.value}, which REA does not inspect`,
);
const value = readUleb(tag.next, scopeEnd);
const value = readElfAttributeUleb(bytes, tag.next, scopeEnd);
if (value === undefined)
return err("malformed MIPS GNU attribute value");
if (tag.value === TAG_GNU_MIPS_ABI_FP) {
@@ -34,7 +34,7 @@ export interface ParsedReferenceSourceEntries {
}
/** Preserve the observed reader failure and add entry-specific recovery guidance. */
export const projectReferenceSourceEntryFailure = (
const projectReferenceSourceEntryFailure = (
entry: Extract<ReferenceSourceEntry, { status: "failed" }>,
): string => `${entry.message} ${entryRecoveryGuidance(entry)}`;
@@ -1,7 +1,6 @@
import { describe, expect, it } from "vitest";
import {
projectReferenceSourceImportError,
referenceSourceImportAnalysisError,
type ReferenceSourceImportError,
} from "./ReferenceSourceImportTypes.js";
@@ -20,11 +19,13 @@ describe("reference-source import error projection", () => {
parse: "execution_failure",
} as const;
for (const [code, category] of Object.entries(expectedCategories)) {
const projected = projectReferenceSourceImportError({
tag: "reference-source-import",
code: code as ReferenceSourceImportError["code"],
message: "EIO while reading /selected/source/src/main.ts",
});
const projected = projectAnalysisError(
referenceSourceImportAnalysisError({
tag: "reference-source-import",
code: code as ReferenceSourceImportError["code"],
message: "EIO while reading /selected/source/src/main.ts",
}),
);
expect(analysisErrorProjectionSchema.safeParse(projected).success).toBe(
true,
);
@@ -5,10 +5,6 @@ import {
AnalysisOutputError,
} from "../domain/analysisErrorCore.js";
import { analysisErrorWithCancellation } from "../domain/analysisErrorCancellation.js";
import {
projectAnalysisError,
type AnalysisErrorProjection,
} from "../domain/analysisErrorProjection.js";
import type {
AnalysisError,
AnalysisErrorOptions,
@@ -125,12 +121,6 @@ export const referenceSourceImportAnalysisError = (
);
};
/** Canonical CLI/MCP projection, retaining source facts under diagnostic details. */
export const projectReferenceSourceImportError = (
error: ReferenceSourceImportError,
): AnalysisErrorProjection =>
projectAnalysisError(referenceSourceImportAnalysisError(error));
/** Import a caller-selected local source tree as historical reference. */
export interface ReferenceSourceImportOptions {
readonly root: string;
@@ -0,0 +1,49 @@
import { inspectAnalysisSettingSeeds } from "../config/analysisSettings.js";
import {
parseEnvironment,
parseEnvironmentValues,
type Environment,
} from "../config/environment.js";
import { ConfigurationError } from "../domain/configurationErrors.js";
import { err, ok, type Result } from "../domain/result.js";
import { snapshotEnvironment } from "../process/snapshotEnvironment.js";
/** Validate literal registration settings while retaining expression-owned unknowns. */
export const validateRegistrationEnvironment = (
values: Readonly<Record<string, string>>,
expressionSettings: ReadonlySet<string>,
validation: "fragment" | "effective",
platform: NodeJS.Platform,
): Result<
{
readonly configuration: Environment;
readonly unresolvedSettings: ReadonlySet<string>;
},
ConfigurationError
> => {
const runtimeEnvironment = snapshotEnvironment(values, platform);
const unresolvedSettings = new Set(
Object.keys(
snapshotEnvironment(
Object.fromEntries([...expressionSettings].map((name) => [name, ""])),
platform,
),
),
);
const validated =
validation === "fragment"
? parseEnvironmentValues(runtimeEnvironment, unresolvedSettings)
: parseEnvironment(runtimeEnvironment, unresolvedSettings);
if (!validated.ok) return validated;
const seedIssues = inspectAnalysisSettingSeeds(
runtimeEnvironment,
unresolvedSettings,
);
if (seedIssues.length > 0)
return err(
new ConfigurationError("Invalid REA environment configuration", {
settings: seedIssues,
}),
);
return ok({ configuration: validated.value, unresolvedSettings });
};
+2 -2
View File
@@ -420,8 +420,8 @@ export const validateCanonicalSkillDirectories = async (
const canonicalSkillFiles = async (
home: string,
clientIds: readonly string[],
environment: Readonly<NodeJS.ProcessEnv> = {},
platform: NodeJS.Platform = process.platform,
environment: Readonly<NodeJS.ProcessEnv>,
platform: NodeJS.Platform,
resources?: ArtifactResourceScope,
): Promise<readonly CanonicalSkillFile[]> => {
const destinations = skillDestinations(
+1 -1
View File
@@ -872,7 +872,7 @@ const readClientConfiguration = async (
client: UninstallClient,
fileSystem: UninstallFileSystem,
resources: ArtifactResourceScope,
ownershipEstablished = false,
ownershipEstablished: boolean,
): Promise<ClientConfigurationRead> => {
const applicabilityUnverified =
client.applicability === "unverified" && !ownershipEstablished;
@@ -17,7 +17,7 @@ import {
AnalysisCapabilityUnavailableError,
} from "../../domain/analysisErrorCore.js";
import { analysisInputErrorFromIssues } from "../../domain/inputIssueProjection.js";
import { EvidenceIntegrityError } from "../../domain/evidenceErrors.js";
import { validateAnalysisViewSubject } from "../../domain/analysisView/analysisViewParent.js";
import { safeParseEvidence } from "../../domain/evidenceValidation.js";
import {
createEvidence,
@@ -249,15 +249,11 @@ const projectResolvedAnalysisView = (
},
});
if (!projected.ok) return projected;
if (
parent.subject !== null &&
parent.subject.digest.sha256 !== projected.value.artifact?.sha256
)
return err(
new EvidenceIntegrityError(
`Parent subject SHA-256 ${parent.subject.digest.sha256} differs from analysis artifact SHA-256 ${projected.value.artifact?.sha256 ?? "unknown"}.`,
),
);
const identity = validateAnalysisViewSubject(
parent.subject,
projected.value.artifact?.sha256,
);
if (!identity.ok) return identity;
return ok(createAnalysisViewEvidence(input, presentation, projected.value));
};
@@ -11,7 +11,7 @@ import {
inspectAnalysisView,
projectRetainedAnalysis,
} from "./AnalysisViewService.js";
import { analysisViewRequestSchema } from "../../domain/analysisView/analysisView.js";
import { analysisViewRequestSchema } from "../../domain/analysisView/analysisViewRequest.js";
import type { Evidence } from "../../domain/evidence.js";
const trace = async (
@@ -58,9 +58,9 @@ interface PrepareSessionExecutionInput {
}
/** Recovery for an operation absent from the selected binary route. */
export const undeclaredOperationGuidance = (
const undeclaredOperationGuidance = (
operation: string,
entryCommentAvailable = false,
entryCommentAvailable: boolean,
): string => {
const entryComment =
entryCommentAvailable &&
@@ -31,9 +31,7 @@ type RoutedOperation = Pick<
>;
/** Reason recorded for an operation's own refusal or provider-wide failure. */
export const runtimeFailureReason = (
failure: AnalysisError,
): string | undefined => {
const runtimeFailureReason = (failure: AnalysisError): string | undefined => {
if (failure._tag === "AnalysisCapabilityUnavailableError")
return failure.userMessage ?? failure.message;
return PROVIDER_WIDE_FAILURES.has(failure._tag)
@@ -42,7 +40,7 @@ export const runtimeFailureReason = (
};
/** Explain unavailability inferred for an operation no request failed for. */
export const inferredUnavailabilityReason = (
const inferredUnavailabilityReason = (
providerId: string,
failure: Pick<ProviderFailure, "operation" | "reason">,
): string =>
@@ -29,16 +29,7 @@ import type {
} from "../../domain/desktop/desktopCapture.js";
import { renderBuildBundle } from "../../domain/desktop/appBuild.js";
import { readDownloadInstall } from "./DesktopExportInstall.js";
import {
NO_LAUNCH_DOCUMENT,
launchDocumentNote,
seedWording,
} from "../../domain/desktop/appExport.js";
import { exportDesktopApplication } from "./DesktopApplicationExport.js";
import {
seedLimitation,
trashedFilesLimitation,
} from "./DesktopExportLimitations.js";
import {
EDITED_NOTE,
HOME,
@@ -529,13 +520,9 @@ describe("exportDesktopApplication and the desktop Trash", () => {
"Documents/today.txt",
],
});
const trashPaths = [
".local/share/Trash/files/today.txt",
".local/share/Trash/info/today.txt.trashinfo",
];
const notice = trashedFilesLimitation(trashPaths);
expect(result.limitations).toContain(notice);
for (const path of trashPaths) expect(notice).toContain(`~/${path}`);
expect(result.limitations).toContain(
"The desktop Trash held 2 file(s) at the seed checkpoint, which are not seeded: ~/.local/share/Trash/files/today.txt, ~/.local/share/Trash/info/today.txt.trashinfo. Restore one the task needs and capture again.",
);
});
});
@@ -1936,7 +1923,9 @@ const exportSeeded = async (
/** The limitations on what the launch opens, found by the lead their formatter gives them. */
const launchLimitations = (limitations: readonly string[]) =>
limitations.filter((line) => line.includes(NO_LAUNCH_DOCUMENT));
limitations.filter((line) =>
line.includes("The app opens no document at launch"),
);
/**
* The seed-then-end capture of an app whose titles name no file, given the
@@ -2037,9 +2026,7 @@ describe("exportDesktopApplication launch document", () => {
`command=["sh", "-c", "cd \\"$HOME\\" 2>/dev/null; exec 'notes' \\"$HOME\\"/'Documents/today.txt'"]`,
);
expect(readme).toContain(
launchDocumentNote({
opens: { kind: "seed", document: "Documents/today.txt" },
}),
"At launch the app opens `~/Documents/today.txt`, the seed's one document.",
);
expect(readme).toContain("`~/Documents/today.txt`");
expect(launchLimitations(result.limitations)).toEqual([]);
@@ -2151,8 +2138,13 @@ describe("exportDesktopApplication launch document", () => {
const made = await exportSeeded("notes %U", { syntheticSeed: true });
const captured = await exportSeeded("notes %U");
const notice = (syntheticSeed: boolean) =>
seedWording({ name: "notes", seedCheckpoint: "seed", syntheticSeed })
.notice;
syntheticSeed
? "The seed was made for this task, not taken from a user's own data; read `envs/notes/seed/` before publishing the bundle."
: "The seed is the user's real data. Keep this bundle local; do not publish it.";
const limitation = (syntheticSeed: boolean) =>
syntheticSeed
? "The seed was made for the task, not taken from a user's own data; read it before publishing the bundle."
: "The seed is the user's real state at the seed checkpoint; keep the bundle local.";
expect(made.result.parameters).toMatchObject({ synthetic_seed: true });
expect(captured.result.parameters).toMatchObject({ synthetic_seed: false });
@@ -2160,10 +2152,8 @@ describe("exportDesktopApplication launch document", () => {
[made, true],
[captured, false],
] as const) {
expect(exported.result.limitations).toContain(seedLimitation(synthetic));
expect(exported.result.limitations).not.toContain(
seedLimitation(!synthetic),
);
expect(exported.result.limitations).toContain(limitation(synthetic));
expect(exported.result.limitations).not.toContain(limitation(!synthetic));
expect(exported.readme).toContain(notice(synthetic));
expect(exported.readme).not.toContain(notice(!synthetic));
}
@@ -235,10 +235,6 @@ type CheckpointTaken = Omit<
readonly more_changed_files: number | null;
};
/** The capture's progress event in a line the controller waited for. */
const eventOf = (line: string): DesktopCaptureEvent | undefined =>
captureProgressEvent(line);
/**
* One capture in the computer. `unreleased` is a capture that may be
* running although its start failed, kept until a stop is confirmed;
@@ -441,7 +437,8 @@ export class DesktopBuildCapture {
),
answer,
});
const event = answer.line === undefined ? undefined : eventOf(answer.line);
const event =
answer.line === undefined ? undefined : captureProgressEvent(answer.line);
if (event?.kind !== "window_open")
return err({
failure: await endedCapture(
@@ -501,7 +498,9 @@ export class DesktopBuildCapture {
{ timeoutS: 600, signal },
);
const event =
answer.line === undefined ? undefined : eventOf(answer.line);
answer.line === undefined
? undefined
: captureProgressEvent(answer.line);
if (event?.kind === "checkpoint_refused")
return err(refused(MARK, event.reason, "Use another checkpoint name."));
if (event?.kind !== "checkpoint")
@@ -328,7 +328,7 @@ const liveComputer = (
* message.
*/
/** The start failure's notice that the bundle seed still holds lock files. */
export const seededLockStartNotice = (locks: readonly string[]): string =>
const seededLockStartNotice = (locks: readonly string[]): string =>
`The bundle's seed holds lock files, which can keep the app from starting when it finds its profile or document in use: ${counted(locks)}. Delete them from the bundle, or export it again, which leaves them out, and start again.`;
const withSeededLocks = async (
@@ -0,0 +1,64 @@
import { writeFile } from "node:fs/promises";
import { join } from "node:path";
import { expect, it } from "vitest";
import { createTestTempDirectory } from "../../../tests/fixtures/temporaryDirectory.js";
import { dumpDconfDatabase } from "../../desktop/DconfDatabaseReader.js";
import { desktopApplicationStateSchema } from "../../domain/desktop/desktopState.js";
import { inspectDesktopApplicationState } from "./DesktopApplicationState.js";
import {
HOME,
capture,
checkpoint,
readers,
retained,
writeCaptureFolder,
} from "./desktopCapture.fixture.js";
it.skipIf(process.platform === "win32")(
"retains a dconf command failure in state Evidence",
async () => {
const tools = await createTestTempDirectory("dconf-state-diagnostics-");
const command = join(tools, "dconf");
await writeFile(command, "#!/bin/sh\nexit 17\n", { mode: 0o600 });
const base = capture();
const path = `${HOME}/.config/dconf/user`;
const bytes = Buffer.from("GVariant\0\0\0\0");
const folder = await writeCaptureFolder(
{
...base,
state: {
...base.state,
content_roots: [path],
checkpoints: [
checkpoint("baseline", "baseline", [], [retained(path, bytes)]),
],
},
},
undefined,
[bytes],
);
const evidence = await inspectDesktopApplicationState(folder, {
...readers,
dconf: (database, signal) =>
dumpDconfDatabase(database, { REA_DCONF_COMMAND: command }, signal),
});
if (!("normalized_result" in evidence)) throw evidence;
const state = desktopApplicationStateSchema.parse(
evidence.normalized_result,
);
expect(state.paths).toEqual([
expect.objectContaining({
home_path: ".config/dconf/user",
versions: [
expect.objectContaining({
format: "dconf",
view: null,
view_error: expect.stringContaining("EACCES"),
}),
],
}),
]);
expect(state.paths[0]?.versions[0]?.view_error).toContain(command);
},
);
@@ -15,7 +15,7 @@ import type { HostIdentity } from "../../domain/desktop/hostIdentity.js";
import { recoveryListLimitation } from "../../domain/desktop/seedRecoveryList.js";
/** The limitation naming files the desktop Trash held at the seed; none when it held none. */
export const trashedFilesLimitation = (
const trashedFilesLimitation = (
files: readonly string[],
): string | undefined =>
files.length === 0
@@ -23,7 +23,7 @@ export const trashedFilesLimitation = (
: `The desktop Trash held ${String(files.length)} file(s) at the seed checkpoint, which are not seeded: ${files.map((home) => `~/${home}`).join(", ")}. Restore one the task needs and capture again.`;
/** The limitation on where the seed came from: made for the task, or the user's real state, kept local. */
export const seedLimitation = (syntheticSeed: boolean): string =>
const seedLimitation = (syntheticSeed: boolean): string =>
syntheticSeed
? "The seed was made for the task, not taken from a user's own data; read it before publishing the bundle."
: "The seed is the user's real state at the seed checkpoint; keep the bundle local.";
+11 -30
View File
@@ -30,33 +30,6 @@ export type ReaderFileBytes =
| { readonly kind: "error"; readonly error: string }
| { readonly kind: "bytes"; readonly bytes: Buffer };
/**
* Each reader's view of its file by reader id, read by REA's reader runner;
* an absent file has none. `pathOf` names bytes to read in place of the
* home's file. A runner that does not answer throws.
*/
export const readTaskReaderViews = async (
readers: readonly DesktopTaskReader[],
run: ReaderCommandRun,
pathOf: (reader: DesktopTaskReader) => string | undefined = () => undefined,
work?: string,
): Promise<ReadonlyMap<string, TaskFileView>> => {
const views = new Map<string, TaskFileView>();
for (const reader of readers) {
const result = await run(
readerCommand(reader, pathOf(reader), work),
readerTimeout(reader) + READER_RUNNER_SLACK_S,
);
if (result.exit_code !== 0 || result.timed_out)
throw new Error(
`REA's reader runner for ${reader.id} ${result.timed_out ? "timed out" : `exited ${String(result.exit_code)}`}: ${result.stderr.slice(-1024) || result.stdout.slice(-1024)}`,
);
const view = readerAnswerView(result.stdout);
if (view !== undefined) views.set(reader.id, view);
}
return views;
};
/** Recover only a complete, valid answer emitted before later finalization failed. */
const completedReaderAnswer = (
cause: unknown,
@@ -112,9 +85,17 @@ export const readerViewsOfBytes = async (
const path = join(folder, reader.file.split("/").at(-1) ?? "file");
await writeFile(path, file.bytes);
// The runner works inside this folder, so removing it cleans up after a runner that was killed.
const read = await readTaskReaderViews([reader], run, () => path, folder);
for (const [id, view] of read) views.set(id, view);
if (!read.has(reader.id)) absent.push(reader.id);
const result = await run(
readerCommand(reader, path, folder),
readerTimeout(reader) + READER_RUNNER_SLACK_S,
);
if (result.exit_code !== 0 || result.timed_out)
throw new Error(
`REA's reader runner for ${reader.id} ${result.timed_out ? "timed out" : `exited ${String(result.exit_code)}`}: ${result.stderr.slice(-1024) || result.stdout.slice(-1024)}`,
);
const view = readerAnswerView(result.stdout);
if (view === undefined) absent.push(reader.id);
else views.set(reader.id, view);
} catch (cause: unknown) {
failure = { cause };
const completed = completedReaderAnswer(cause);
@@ -26,7 +26,7 @@ import { createTestTempDirectory } from "../../../tests/fixtures/temporaryDirect
import { ArtifactFileCleanupFailure } from "../../artifacts/ArtifactFileOperation.js";
import { writeJsonOutput } from "../../cli/jsonOutput.js";
import { projectAnalysisError } from "../../domain/analysisErrorProjection.js";
import { analysisViewRequestSchema } from "../../domain/analysisView/analysisView.js";
import { analysisViewRequestSchema } from "../../domain/analysisView/analysisViewRequest.js";
import { summarizeEnvironmentCoverage } from "../../domain/analysisView/environmentCoverageView.js";
import type { Evidence } from "../../domain/evidence.js";
import { inspectAnalysisViewValidated } from "../analysisView/AnalysisViewService.js";
@@ -26,11 +26,14 @@ const withTools = (tools: readonly EnvironmentManifestTool[]) => ({
manifest: { ...source.manifest, tools: [...tools] },
});
const environmentToolInputSchema = (tool: EnvironmentManifestTool) => {
const [definition] = environmentToolDefinitions(withTools([tool]));
const [definition] = environmentToolDefinitions(
withTools([tool]),
source.manifest.name,
);
if (definition === undefined) throw new Error("Missing tool definition");
return definition.inputSchema;
};
const definitions = environmentToolDefinitions(source);
const definitions = environmentToolDefinitions(source, source.manifest.name);
const tool = (name: string): EnvironmentManifestTool => {
const found = source.manifest.tools.find((entry) => entry.name === name);
@@ -47,9 +50,9 @@ describe("environmentToolDefinitions", () => {
expect(definition.description).toContain(
"state.source, not replay.exact_match, identifies state vs recording",
);
expect(environmentToolDefinitions(source)[0]?.description).not.toContain(
"state.source",
);
expect(
environmentToolDefinitions(source, source.manifest.name)[0]?.description,
).not.toContain("state.source");
});
it("maps every manifest tool in order with a replay description naming its records and evidence", () => {
@@ -154,10 +157,13 @@ const usersAnswering = (
record_ordinals: [...exchanges.keys()],
statuses: [...statuses],
};
const [definition] = environmentToolDefinitions({
...withTools([users]),
exchanges,
});
const [definition] = environmentToolDefinitions(
{
...withTools([users]),
exchanges,
},
source.manifest.name,
);
return definition?.description;
};
const returnsOf = (bodyText: string): string | undefined =>
@@ -314,28 +320,31 @@ describe("environment tool response summaries", () => {
const users = tool("get_users_by_id");
const ordinal = users.record_ordinals[0] ?? 0;
const catalog = (contentType: string) =>
prepareDefinitions({
...withTools(
Array.from({ length: 200 }, (_, index) => ({
...users,
name: `get_users_${String(index)}`,
record_ordinals: [ordinal],
})),
),
exchanges: new Map([
[
ordinal,
{
prepareDefinitions(
{
...withTools(
Array.from({ length: 200 }, (_, index) => ({
...users,
name: `get_users_${String(index)}`,
record_ordinals: [ordinal],
})),
),
exchanges: new Map([
[
ordinal,
path: "/api/users/2",
query: null,
status: 200,
contentType,
bodyText: JSON.stringify(fields),
},
],
]),
});
{
ordinal,
path: "/api/users/2",
query: null,
status: 200,
contentType,
bodyText: JSON.stringify(fields),
},
],
]),
},
source.manifest.name,
);
expect(catalog("text/html").ok).toBe(true);
const bounded = catalog("application/json");
expect(bounded).toMatchObject({
@@ -346,18 +355,21 @@ describe("environment tool response summaries", () => {
reportedLimits: { max_catalog_bytes: ENVIRONMENT_CATALOG_LIMIT_BYTES },
},
});
const [single] = environmentToolDefinitions({
...withTools([users]),
exchanges: new Map([
[
ordinal,
{
...source.exchanges.get(ordinal)!,
bodyText: JSON.stringify(fields),
},
],
]),
});
const [single] = environmentToolDefinitions(
{
...withTools([users]),
exchanges: new Map([
[
ordinal,
{
...source.exchanges.get(ordinal)!,
bodyText: JSON.stringify(fields),
},
],
]),
},
source.manifest.name,
);
expect(Buffer.byteLength(single?.description ?? "")).toBeLessThan(1700);
expect(single?.description).toContain(
"/300 shown; full list in capture Evidence)",
@@ -374,7 +386,10 @@ describe("environment catalog description budgets", () => {
path_template: `/${"😀".repeat(1000)}`,
query_parameter_names: [query],
};
const [definition] = environmentToolDefinitions(withTools([expanded]));
const [definition] = environmentToolDefinitions(
withTools([expanded]),
source.manifest.name,
);
expect(definition?.tool).toBe(expanded);
expect(Buffer.byteLength(definition?.description ?? "")).toBeLessThan(3500);
expect(definition?.description).toContain(
@@ -406,7 +421,10 @@ describe("environment catalog description budgets", () => {
})),
record_ordinals: ordinals,
};
const [definition] = environmentToolDefinitions(withTools([expanded]));
const [definition] = environmentToolDefinitions(
withTools([expanded]),
source.manifest.name,
);
expect(definition?.tool).toBe(expanded);
expect(expanded.params[0]?.observed_values).toBe(observations);
expect(expanded.record_ordinals).toBe(ordinals);
@@ -431,7 +449,10 @@ describe("environment catalog description budgets", () => {
...users,
path_template: `/api/${"segment/".repeat(200_000)}{id}`,
};
const [definition] = environmentToolDefinitions(withTools([long]));
const [definition] = environmentToolDefinitions(
withTools([long]),
source.manifest.name,
);
expect(Buffer.byteLength(definition?.description ?? "")).toBeLessThan(2048);
expect(definition?.description).toContain(
"path display truncated; full value in manifest Evidence",
@@ -595,6 +616,7 @@ describe("environment tool annotations", () => {
method,
})),
),
source.manifest.name,
).map((definition) => definition.annotations);
expect(hints).toEqual([
{ readOnlyHint: true, idempotentHint: true, openWorldHint: false },
@@ -432,7 +432,7 @@ const catalogBody = (
/** Admit complete discovery before binding; six bytes per UTF-16 unit covers JSON escaping and UTF-8 expansion. */
export const environmentToolDefinitions = (
source: EnvironmentSource,
name: string = source.manifest.name,
name: string,
{
operation = ENVIRONMENT_SERVE_OPERATION,
dataPlane = "none",
@@ -5,7 +5,7 @@ import {
} from "./JavaScriptArtifactAnalysis.js";
import { createJavaScriptSemanticGraphProjection } from "./JavaScriptSemanticGraphBuilder.js";
import { AnalysisResourceConstraintError } from "../../domain/analysisErrorCore.js";
import { analyzeJavaScriptStaticSource } from "../../domain/javascript/javascriptStaticAnalysis.js";
import { analyzeJavaScriptStaticSource } from "../../../tests/fixtures/javascriptStaticAnalysis.js";
import type { JavaScriptArtifactFile } from "../../domain/javascript/javascriptArtifactFiles.js";
import { err, ok } from "../../domain/result.js";
@@ -1,7 +1,7 @@
import { expect, it } from "vitest";
import { analyzeAndAdoptJavaScriptArtifactFiles } from "./JavaScriptArtifactAnalysis.js";
import { createJavaScriptSemanticGraphProjection } from "./JavaScriptSemanticGraphBuilder.js";
import { analyzeJavaScriptStaticSource } from "../../domain/javascript/javascriptStaticAnalysis.js";
import { analyzeJavaScriptStaticSource } from "../../../tests/fixtures/javascriptStaticAnalysis.js";
import type {
JavaScriptArtifactFile,
JavaScriptUnanalyzedArtifactFile,
@@ -5,7 +5,7 @@ import {
AnalysisCancelledError,
AnalysisResourceConstraintError,
} from "../../domain/analysisErrorCore.js";
import { analyzeJavaScriptStaticSource } from "../../domain/javascript/javascriptStaticAnalysis.js";
import { analyzeJavaScriptStaticSource } from "../../../tests/fixtures/javascriptStaticAnalysis.js";
import type { JavaScriptArtifactFile } from "../../domain/javascript/javascriptArtifactFiles.js";
import { err, ok } from "../../domain/result.js";
@@ -8,7 +8,7 @@ import type { JavaScriptArtifactFile } from "../../domain/javascript/javascriptA
import {
analyzeJavaScriptStaticSource,
analyzeParsedJavaScriptStaticSource,
} from "../../domain/javascript/javascriptStaticAnalysis.js";
} from "../../../tests/fixtures/javascriptStaticAnalysis.js";
import { parseJavaScriptSource } from "../../domain/javascript/javascriptSourceParser.js";
import type {
AnalyzedJavaScriptArtifactFile,
@@ -8,7 +8,7 @@ import { JavaScriptJsonDocuments } from "./JavaScriptJsonDocuments.js";
import type { JavaScriptArtifactFile } from "../../domain/javascript/javascriptArtifactFiles.js";
import { artifactLocalIdentity } from "./JavaScriptArtifactGraphContext.js";
import { applicationNodeIdentitySchema } from "../../domain/javascript/javascriptApplicationEvidenceSchemas.js";
import { analyzeJavaScriptStaticSource } from "../../domain/javascript/javascriptStaticAnalysis.js";
import { analyzeJavaScriptStaticSource } from "../../../tests/fixtures/javascriptStaticAnalysis.js";
const completeInventoryCoverage = {
scope: "container-entries",
@@ -1,5 +1,5 @@
import { expect, it } from "vitest";
import { analyzeJavaScriptStaticSource } from "../../domain/javascript/javascriptStaticAnalysis.js";
import { analyzeJavaScriptStaticSource } from "../../../tests/fixtures/javascriptStaticAnalysis.js";
import { analyzeJavaScriptSemantics } from "../../domain/javascript/javascriptSemanticAnalysis.js";
import type { JavaScriptArtifactFile } from "../../domain/javascript/javascriptArtifactFiles.js";
import { projectJavaScriptSemanticFileSteps } from "./JavaScriptSemanticGraphBuilder.js";
@@ -8,6 +8,12 @@ import { jsonValueSchema } from "../../domain/jsonValue.js";
import type { AnalysisError } from "../../domain/analysisErrorBase.js";
import { err, ok } from "../../domain/result.js";
/** Expected user-facing unknown reasons, independent of the trace producer. */
export const EXPECTED_CLEANUP_STOPPED_TRACE =
"Trace stopped because the provider did not confirm cleanup of resources it owns";
export const EXPECTED_UNRECORDED_VALUE_FLOW =
"Native def-use was not recorded for this procedure.";
/** Simulated provider seam for value traces; not a real decompiler observation. */
export const valueTraceSubject = {
path: "/fixture",
@@ -1,9 +1,6 @@
import { createAnalysisExecutionEvidence } from "../../domain/analysisObservation.js";
import { describe, expect, it } from "vitest";
import {
traceNativeValues as traceNativeValuesCore,
UNRECORDED_NATIVE_VALUE_FLOW,
} from "./NativeValueTrace.js";
import { traceNativeValues as traceNativeValuesCore } from "./NativeValueTrace.js";
import { executeComposedWorkflow } from "../WorkflowEvidence.js";
import {
createAnalysisExecution,
@@ -22,6 +19,7 @@ import {
import { NoBinaryOpenError } from "../../domain/configurationErrors.js";
import { ok, err } from "../../domain/result.js";
import {
EXPECTED_UNRECORDED_VALUE_FLOW,
nativeFlowDossier,
pcodeOperation as node,
pcodeScalar as scalar,
@@ -196,7 +194,7 @@ describe("native value trace seed admission", () => {
{
procedure: base.procedure.address,
address: null,
reason: UNRECORDED_NATIVE_VALUE_FLOW,
reason: EXPECTED_UNRECORDED_VALUE_FLOW,
},
],
});
+2 -2
View File
@@ -29,11 +29,11 @@ import {
import { createValueTraceBudget } from "./NativeValueTraceBudget.js";
/** The located unknown where unconfirmed provider cleanup stopped a trace. */
export const CLEANUP_STOPPED_VALUE_TRACE =
const CLEANUP_STOPPED_VALUE_TRACE =
"Trace stopped because the provider did not confirm cleanup of resources it owns";
/** Fallback when a dossier has no native value-flow facet for this procedure. */
export const UNRECORDED_NATIVE_VALUE_FLOW =
const UNRECORDED_NATIVE_VALUE_FLOW =
"Native def-use was not recorded for this procedure.";
/** Compose bounded high-p-code def-use graphs across exactly referenced call destinations. */
@@ -6,11 +6,9 @@ import {
AnalysisCapabilityUnavailableError,
} from "../../domain/analysisErrorCore.js";
import { nativeValueTraceSchema } from "../../domain/native/nativeValueTrace.js";
import { traceNativeValues } from "./NativeValueTrace.js";
import {
CLEANUP_STOPPED_VALUE_TRACE,
traceNativeValues,
} from "./NativeValueTrace.js";
import {
EXPECTED_CLEANUP_STOPPED_TRACE,
failingValueTraceReads,
valueTraceDiamond,
valueTraceProvider as provider,
@@ -149,7 +147,7 @@ describe("native value trace child stops", () => {
{
procedure: "0x1000",
address: "0x1008",
reason: CLEANUP_STOPPED_VALUE_TRACE,
reason: EXPECTED_CLEANUP_STOPPED_TRACE,
},
],
);
+3 -5
View File
@@ -4,14 +4,12 @@ import { jsonValueSchema } from "../domain/jsonValue.js";
import { DarwinProcessOwnershipInspectionError } from "../process/DarwinProcessRunTokenReader.js";
import { OwnedCommandFailure } from "../process/OwnedCommand.js";
import {
DARWIN_PROCESS_OWNERSHIP_REMEDIATION,
LINUX_PROCESS_OWNERSHIP_REMEDIATION,
LinuxOwnedProcessPrerequisiteError,
} from "../process/ProcessOwnershipObservation.js";
import type { ProviderAvailability } from "./AnalysisProvider.js";
const DARWIN_REMEDIATION =
"Check the selected Xcode or Command Line Tools and Swift compiler configuration, then restart REA and retry.";
// Provider adapters re-attribute capability errors and keep the original as cause.
const prerequisiteRefusal = (
cause: unknown,
@@ -94,8 +92,8 @@ export const ownedProcessPrerequisiteAvailability = (
return {
status: "unavailable",
code: "runtime_unavailable",
reason: `${darwin.message}. ${DARWIN_REMEDIATION}`,
remediation: DARWIN_REMEDIATION,
reason: `${darwin.message}. ${DARWIN_PROCESS_OWNERSHIP_REMEDIATION}`,
remediation: DARWIN_PROCESS_OWNERSHIP_REMEDIATION,
diagnostics: {
stage: "ownership_preparation",
...(darwin.compilerOutcome === undefined
+2 -7
View File
@@ -20,6 +20,7 @@ import { processScenarioSchema } from "../../domain/process/processScenario.js";
import {
compareParsedProcessEvidence,
readProcessComparisonSource,
unsupportedProcessCaptureProvider,
} from "./ProcessComparisonService.js";
import { createProgressReporter } from "../ProgressReporter.js";
import { captureProcessScenario } from "../../process/capture/ProcessHarness.js";
@@ -287,13 +288,7 @@ const projectCaptureFileFailure = (
"invalid_value",
);
case "unsupported-provider":
return new AnalysisInputError("compare_process_captures", undefined, [
{
path: [side, "provider"],
reason: "invalid_value",
message: `The ${side} Evidence provider ${failure.provider.id} v${failure.provider.version} (${failure.provider.name}) is not the current or supported legacy ${PROCESS_PROVIDER.id} v${PROCESS_PROVIDER.version} process capture provider.`,
},
]);
return unsupportedProcessCaptureProvider(side, failure);
case "invalid-capture":
return invalidCaptureEvidence(failure.cause, side);
}
@@ -34,7 +34,6 @@ export const compareProcessEvidence = (
lookup?: EvidenceLookup,
options: {
readonly signal?: AbortSignal;
readonly now?: () => number;
} = {},
): Result<ProcessComparisonExecution, AnalysisError> => {
const selected = resolveEvidenceRequest(
@@ -57,7 +56,6 @@ export const compareProcessEvidence = (
...(request.trace_spec === undefined
? {}
: { traceSpecification: request.trace_spec }),
...(options.now === undefined ? {} : { now: options.now }),
});
};
@@ -126,15 +124,7 @@ export const readProcessComparisonSource = (
]),
);
case "unsupported-provider":
return err(
new AnalysisInputError(operation, undefined, [
{
path: [side, "provider"],
reason: "invalid_value",
message: `The ${side} Evidence provider ${failure.provider.id} v${failure.provider.version} (${failure.provider.name}) is not the current or supported legacy ${PROCESS_PROVIDER.id} v${PROCESS_PROVIDER.version} process capture provider.`,
},
]),
);
return err(unsupportedProcessCaptureProvider(side, failure));
case "invalid-capture":
return err(
new EvidenceIntegrityError(
@@ -148,6 +138,22 @@ export const readProcessComparisonSource = (
}
};
/** Report the same provider admission failure at both public adapters. */
export const unsupportedProcessCaptureProvider = (
side: "left" | "right",
failure: Extract<
ProcessCaptureEvidenceFailure,
{ kind: "unsupported-provider" }
>,
): AnalysisInputError =>
new AnalysisInputError("compare_process_captures", undefined, [
{
path: [side, "provider"],
reason: "invalid_value",
message: `The ${side} Evidence provider ${failure.provider.id} v${failure.provider.version} (${failure.provider.name}) is not the current or supported legacy ${PROCESS_PROVIDER.id} v${PROCESS_PROVIDER.version} process capture provider.`,
},
]);
const comparisonUnknownInput = (
parsed: {
readonly left_evidence_id: string;
+55
View File
@@ -0,0 +1,55 @@
import type { FileHandle } from "node:fs/promises";
import { createHash } from "node:crypto";
import { readFileHandleRange } from "../filesystem/readFileHandleRange.js";
import { abortIfNeeded } from "./ArtifactHash.js";
interface ArtifactRangeCarrier {
readonly handle: FileHandle;
readonly fileSize: number;
readonly signal: AbortSignal | undefined;
}
/** Read exactly one bounded carrier range, returning undefined for a short read. */
export const readArtifactRange = async (
carrier: ArtifactRangeCarrier,
offset: number,
length: number,
): Promise<Buffer | undefined> => {
abortIfNeeded(carrier.signal);
if (
!Number.isSafeInteger(offset) ||
!Number.isSafeInteger(length) ||
offset < 0 ||
length < 0 ||
offset > carrier.fileSize ||
length > carrier.fileSize - offset
)
return undefined;
const bytes = await readFileHandleRange(carrier.handle, offset, length, () =>
abortIfNeeded(carrier.signal),
);
// The shared reader reports EOF before a late cancellation. These carrier
// readers have always observed cancellation after every read, including EOF.
abortIfNeeded(carrier.signal);
return bytes.length === length ? bytes : undefined;
};
/** Hash one carrier range in bounded chunks; undefined means it was truncated. */
export const hashArtifactRange = async (
carrier: ArtifactRangeCarrier,
offset: number,
length: number,
): Promise<string | undefined> => {
const hash = createHash("sha256");
let remaining = length;
let position = offset;
while (remaining > 0) {
const chunkLength = Math.min(64 * 1024, remaining);
const chunk = await readArtifactRange(carrier, position, chunkLength);
if (chunk === undefined) return undefined;
hash.update(chunk);
position += chunkLength;
remaining -= chunkLength;
}
return hash.digest("hex");
};
+81
View File
@@ -0,0 +1,81 @@
import { expect, it, vi } from "vitest";
import { ArtifactPreparation } from "./ArtifactPreparation.js";
import { ArtifactReaderFailure } from "./ArtifactReader.js";
it.each(["physical", "cleanup", "waiter-cancelled"] as const)(
"does not retry another caller's %s failure",
async (kind) => {
const owner = new ArtifactPreparation();
const first = new AbortController();
const waiter = new AbortController();
const reason = new Error("first caller cancelled");
let reject!: (cause: unknown) => void;
const pending = new Promise<void>((_resolve, rejectPromise) => {
reject = rejectPromise;
});
const prepare = vi.fn(() => pending);
const assertOpen = vi.fn();
const initial = owner.run(first.signal, prepare, assertOpen);
const peer = owner.run(waiter.signal, prepare, assertOpen);
first.abort(reason);
if (kind === "waiter-cancelled") waiter.abort("waiter stopped");
const failure =
kind === "physical"
? new Error("read failed")
: new ArtifactReaderFailure("cancelled", "cancelled", {
cause: reason,
...(kind === "cleanup"
? { cleanup: { reason: "close failed", resources: ["file"] } }
: {}),
});
reject(failure);
await expect(initial).rejects.toBe(failure);
await expect(peer).rejects.toBe(failure);
expect(prepare).toHaveBeenCalledTimes(1);
expect(assertOpen).not.toHaveBeenCalled();
// A later call can try again after the failed shared preparation settles.
await expect(
owner.run(
undefined,
async () => {},
() => {},
),
).resolves.toBeUndefined();
},
);
it("shares one replacement when multiple callers survive a peer cancellation", async () => {
const owner = new ArtifactPreparation();
const first = new AbortController();
const reason = new Error("first caller cancelled");
let reject!: (cause: unknown) => void;
const initialWork = new Promise<void>((_resolve, rejectPromise) => {
reject = rejectPromise;
});
let complete!: () => void;
const replacement = new Promise<void>((resolve) => {
complete = resolve;
});
const retry = vi.fn(() => replacement);
const initial = owner.run(
first.signal,
() => initialWork,
() => {},
);
const left = owner.run(undefined, retry, () => {});
const right = owner.run(undefined, retry, () => {});
first.abort(reason);
const failure = new ArtifactReaderFailure("cancelled", "cancelled", {
cause: reason,
});
reject(failure);
await expect(initial).rejects.toBe(failure);
expect(retry).toHaveBeenCalledTimes(1);
complete();
await expect(Promise.all([left, right])).resolves.toEqual([
undefined,
undefined,
]);
expect(retry).toHaveBeenCalledTimes(1);
});
+40
View File
@@ -0,0 +1,40 @@
import { artifactReadCancelledBy } from "./ArtifactFileOperation.js";
/** Share one preparation while allowing another caller to retry its cancellation. */
export class ArtifactPreparation {
#pending:
| {
readonly signal: AbortSignal | undefined;
readonly promise: Promise<void>;
}
| undefined;
async run(
signal: AbortSignal | undefined,
prepare: () => Promise<void>,
assertOpen: () => void,
): Promise<void> {
for (;;) {
const pending = (this.#pending ??= { signal, promise: prepare() });
try {
await pending.promise;
return;
} catch (cause: unknown) {
if (this.#pending === pending) this.#pending = undefined;
if (
pending.signal !== signal &&
signal?.aborted !== true &&
artifactReadCancelledBy(cause, pending.signal)
) {
assertOpen();
continue;
}
throw cause;
}
}
}
clear(): void {
this.#pending = undefined;
}
}
+9 -32
View File
@@ -1,3 +1,4 @@
import { ArtifactPreparation } from "./ArtifactPreparation.js";
import {
DEFAULT_BUN_READER_LIMITS,
type BunReaderLimits,
@@ -16,10 +17,7 @@ import type {
import { jsonValueByteLength } from "../domain/jsonValueByteLength.js";
import type { JsonValue } from "../domain/jsonValue.js";
import { readFileHandleChunks } from "../filesystem/readFileHandleChunks.js";
import {
artifactReadCancelledBy,
assertArtifactReadActive,
} from "./ArtifactFileOperation.js";
import { assertArtifactReadActive } from "./ArtifactFileOperation.js";
import {
normalizeArtifactPath,
ArtifactPathRegistry,
@@ -62,12 +60,7 @@ export class BunStandaloneArtifactReader implements ArtifactReader {
readonly #streams = new Set<Readable>();
readonly #source: ArtifactFileSource;
#admission: BunStandaloneAdmission | undefined;
#prepared:
| {
readonly signal: AbortSignal | undefined;
readonly promise: Promise<void>;
}
| undefined;
readonly #preparation = new ArtifactPreparation();
#header: BunStandaloneMetadata | undefined;
#closing = false;
@@ -108,27 +101,11 @@ export class BunStandaloneArtifactReader implements ArtifactReader {
): AsyncIterable<ArtifactEntry> {
await this.#scope.run(async () => {
this.#assertOpen(signal);
for (;;) {
const pending = (this.#prepared ??= {
signal,
promise: this.#prepare(signal),
});
try {
await pending.promise;
break;
} catch (cause: unknown) {
if (this.#prepared === pending) this.#prepared = undefined;
if (
pending.signal !== signal &&
signal?.aborted !== true &&
artifactReadCancelledBy(cause, pending.signal)
) {
this.#assertOpen(signal);
continue;
}
throw cause;
}
}
await this.#preparation.run(
signal,
() => this.#prepare(signal),
() => this.#assertOpen(signal),
);
await this.#source.readSelectedPath(signal);
});
let index = 0;
@@ -209,7 +186,7 @@ export class BunStandaloneArtifactReader implements ArtifactReader {
this.#entries.clear();
this.#header = undefined;
this.#admission = undefined;
this.#prepared = undefined;
this.#preparation.clear();
}
provenance(): readonly [] {
+8 -28
View File
@@ -1,3 +1,4 @@
import { ArtifactPreparation } from "./ArtifactPreparation.js";
import { Readable } from "node:stream";
import { setImmediate } from "node:timers/promises";
@@ -18,10 +19,7 @@ import {
type ArtifactTraversalOptions,
} from "./ArtifactReader.js";
import { ArtifactResourceScope } from "./ArtifactResourceScope.js";
import {
artifactReadCancelledBy,
assertArtifactReadActive,
} from "./ArtifactFileOperation.js";
import { assertArtifactReadActive } from "./ArtifactFileOperation.js";
import { ArtifactFileSource } from "./ArtifactFileSource.js";
import { readFileHandleChunks } from "../filesystem/readFileHandleChunks.js";
import type { StableRegularFileDescriptor } from "../filesystem/RegularFile.js";
@@ -72,9 +70,7 @@ export class DecodedRegionArtifactReader<Image> implements ArtifactReader {
readonly #streams = new Set<Readable>();
readonly #source: ArtifactFileSource;
readonly #decoding: RegionDecoding<Image>;
#prepared:
| { signal: AbortSignal | undefined; promise: Promise<void> }
| undefined;
readonly #preparation = new ArtifactPreparation();
/** Whole-file facts only; the region map is the sole owner of region bytes. */
#described: Omit<DecodedRegions, "regions"> | undefined;
#closing = false;
@@ -115,27 +111,11 @@ export class DecodedRegionArtifactReader<Image> implements ArtifactReader {
): AsyncIterable<ArtifactEntry> {
await this.#scope.run(async () => {
this.#assertOpen(signal);
for (;;) {
const pending = (this.#prepared ??= {
signal,
promise: this.#prepare(signal),
});
try {
await pending.promise;
break;
} catch (cause: unknown) {
if (this.#prepared === pending) this.#prepared = undefined;
if (
pending.signal !== signal &&
signal?.aborted !== true &&
artifactReadCancelledBy(cause, pending.signal)
) {
this.#assertOpen(signal);
continue;
}
throw cause;
}
}
await this.#preparation.run(
signal,
() => this.#prepare(signal),
() => this.#assertOpen(signal),
);
await this.#source.read(signal);
});
const coverage = this.#described?.coverage;
+13 -38
View File
@@ -1,3 +1,4 @@
import { ArtifactPreparation } from "./ArtifactPreparation.js";
import { Readable, Transform, pipeline } from "node:stream";
import { setImmediate } from "node:timers/promises";
@@ -11,10 +12,7 @@ import type {
} from "../domain/artifactFormatMetadata.js";
import type { JsonValue } from "../domain/jsonValue.js";
import { jsonValueByteLength } from "../domain/jsonValueByteLength.js";
import {
assertArtifactReadActive,
artifactReadCancelledBy,
} from "./ArtifactFileOperation.js";
import { assertArtifactReadActive } from "./ArtifactFileOperation.js";
import { ArtifactFileSource } from "./ArtifactFileSource.js";
import {
ArtifactPathRegistry,
@@ -42,6 +40,8 @@ import { ArtifactEmbeddedFormatProbes } from "./ArtifactEmbeddedFormatProbes.js"
import {
DENO_CHILD_SUFFIXES,
DENO_PATH_LENGTH_LIMITATION,
DENO_PATH_PARTS_LIMITATION,
DENO_UNSAFE_PATH_LIMITATION,
denoLogicalPathFits,
MAX_DENO_LOGICAL_PATH_PARTS,
type DenoVfsMember,
@@ -73,12 +73,7 @@ export class DenoStandaloneArtifactReader implements ArtifactReader {
readonly #streams = new Set<Readable>();
readonly #source: ArtifactFileSource;
#admission: DenoStandaloneAdmission | undefined;
#prepared:
| {
readonly signal: AbortSignal | undefined;
readonly promise: Promise<void>;
}
| undefined;
readonly #preparation = new ArtifactPreparation();
#header: DenoStandaloneMetadata | undefined;
#closing = false;
@@ -117,27 +112,11 @@ export class DenoStandaloneArtifactReader implements ArtifactReader {
): AsyncIterable<ArtifactEntry> {
await this.#scope.run(async () => {
this.#assertOpen(signal);
for (;;) {
const pending = (this.#prepared ??= {
signal,
promise: this.#prepare(signal),
});
try {
await pending.promise;
break;
} catch (cause: unknown) {
if (this.#prepared === pending) this.#prepared = undefined;
if (
pending.signal !== signal &&
signal?.aborted !== true &&
artifactReadCancelledBy(cause, pending.signal)
) {
this.#assertOpen(signal);
continue;
}
throw cause;
}
}
await this.#preparation.run(
signal,
() => this.#prepare(signal),
() => this.#assertOpen(signal),
);
await this.#source.readSelectedPath(signal);
});
if (this.#header?.vfs_listing === "unsupported")
@@ -247,7 +226,7 @@ export class DenoStandaloneArtifactReader implements ArtifactReader {
this.#entries.clear();
this.#header = undefined;
this.#admission = undefined;
this.#prepared = undefined;
this.#preparation.clear();
}
provenance(): readonly [] {
@@ -612,9 +591,7 @@ const remoteDestination = (
return {
path: fallback,
sourcePath: text,
limitations: [
`The virtual name exceeds the fixed ${String(MAX_DENO_LOGICAL_PATH_PARTS)}-part logical path allowance; an indexed path is used and the original name is retained.`,
],
limitations: [DENO_PATH_PARTS_LIMITATION],
};
let normalized: string;
try {
@@ -625,9 +602,7 @@ const remoteDestination = (
return {
path: fallback,
sourcePath: text,
limitations: [
"The virtual name is not a safe relative path; an indexed path is used and the original name is retained.",
],
limitations: [DENO_UNSAFE_PATH_LIMITATION],
};
}
// A specifier is generated into a path, so its spelling can follow every
+8 -28
View File
@@ -1,3 +1,4 @@
import { ArtifactPreparation } from "./ArtifactPreparation.js";
import { Readable, Transform, pipeline } from "node:stream";
import { setImmediate } from "node:timers/promises";
@@ -13,10 +14,7 @@ import {
type ArtifactTraversalOptions,
} from "./ArtifactReader.js";
import { ArtifactResourceScope } from "./ArtifactResourceScope.js";
import {
artifactReadCancelledBy,
assertArtifactReadActive,
} from "./ArtifactFileOperation.js";
import { assertArtifactReadActive } from "./ArtifactFileOperation.js";
import { ArtifactFileSource } from "./ArtifactFileSource.js";
import {
createFitDigest,
@@ -37,9 +35,7 @@ export class FitArtifactReader implements ArtifactReader {
readonly #streams = new Set<Readable>();
readonly #path: string;
readonly #source: ArtifactFileSource;
#prepared:
| { signal: AbortSignal | undefined; promise: Promise<void> }
| undefined;
readonly #preparation = new ArtifactPreparation();
#metadata: FitMetadata | undefined;
#coverage: ArtifactReaderTraversal | undefined;
#closing = false;
@@ -71,27 +67,11 @@ export class FitArtifactReader implements ArtifactReader {
): AsyncIterable<ArtifactEntry> {
await this.#scope.run(async () => {
this.#assertOpen(signal);
for (;;) {
const pending = (this.#prepared ??= {
signal,
promise: this.#prepare(signal),
});
try {
await pending.promise;
break;
} catch (cause: unknown) {
if (this.#prepared === pending) this.#prepared = undefined;
if (
pending.signal !== signal &&
signal?.aborted !== true &&
artifactReadCancelledBy(cause, pending.signal)
) {
this.#assertOpen(signal);
continue;
}
throw cause;
}
}
await this.#preparation.run(
signal,
() => this.#prepare(signal),
() => this.#assertOpen(signal),
);
await this.#source.read(signal);
});
if (this.#coverage !== undefined)
+3 -3
View File
@@ -22,9 +22,9 @@ import type { ArtifactEntry } from "./ArtifactReader.js";
import { artifactFormatDescriptor } from "./artifactFormats.js";
import { ArtifactResourceScope } from "./ArtifactResourceScope.js";
import { extractArtifact } from "./extraction/ArtifactExtraction.js";
import { NODE_SEA_MAGIC } from "./nodeSea/NodeSeaBlob.js";
import {
NODE_SEA_FUSE_HASH,
NODE_SEA_FIXTURE_MAGIC,
buildNodeSeaBlob,
nodeSeaElf,
nodeSeaMachO,
@@ -206,7 +206,7 @@ it.each([
expect(root).toMatchObject({ artifact_kind: "container" });
expect(root?.format_metadata).toMatchObject({
kind: "node-sea",
magic: NODE_SEA_MAGIC,
magic: NODE_SEA_FIXTURE_MAGIC,
carrier,
code_path: "app/main.js",
disable_experimental_sea_warning: true,
@@ -503,7 +503,7 @@ it("fails closed when the located blob is truncated, has a bad magic, or unknown
message: expect.stringMatching(/magic.*not.*0x143da20/iu),
});
const unknown = Buffer.alloc(9);
unknown.writeUInt32LE(NODE_SEA_MAGIC, 0);
unknown.writeUInt32LE(NODE_SEA_FIXTURE_MAGIC, 0);
unknown.writeUInt32LE(1 << 5, 4);
unknown[8] = 1;
await expect(
@@ -273,3 +273,45 @@ it("delivers acquired extension metadata through the canonical inventory schema"
},
});
});
it("does not finish a traversal when close begins during final source verification", async () => {
const reader = new NtfsMftArtifactReader(
await writeMft("$MFT", recordBytes({})),
);
const read = ArtifactFileSource.prototype.read;
let calls = 0;
let entered!: () => void;
let resume!: () => void;
const started = new Promise<void>((resolve) => {
entered = resolve;
});
const gate = new Promise<void>((resolve) => {
resume = resolve;
});
const spy = vi
.spyOn(ArtifactFileSource.prototype, "read")
.mockImplementation(async function (this: ArtifactFileSource, signal) {
const result = await read.call(this, signal);
if (++calls === 2) {
entered();
await gate;
}
return result;
});
try {
const traversal = reader.entries()[Symbol.asyncIterator]().next();
const rejected = expect(traversal).rejects.toMatchObject({
reason: "unavailable",
});
await started;
const closing = reader.close();
resume();
await rejected;
await closing;
expect(calls).toBe(2);
} finally {
resume();
spy.mockRestore();
await reader.close();
}
});
+8 -28
View File
@@ -1,3 +1,4 @@
import { ArtifactPreparation } from "./ArtifactPreparation.js";
import type { Readable } from "node:stream";
import {
@@ -17,10 +18,7 @@ import {
type ArtifactTraversalOptions,
} from "./ArtifactReader.js";
import { ArtifactResourceScope } from "./ArtifactResourceScope.js";
import {
artifactReadCancelledBy,
assertArtifactReadActive,
} from "./ArtifactFileOperation.js";
import { assertArtifactReadActive } from "./ArtifactFileOperation.js";
import { ArtifactFileSource } from "./ArtifactFileSource.js";
import { readFileHandleChunks } from "../filesystem/readFileHandleChunks.js";
import type { StableRegularFileDescriptor } from "../filesystem/RegularFile.js";
@@ -34,9 +32,7 @@ export class NtfsMftArtifactReader implements ArtifactReader {
readonly format = "ntfs-mft" as const;
readonly #scope = new ArtifactResourceScope();
readonly #source: ArtifactFileSource;
#prepared:
| { signal: AbortSignal | undefined; promise: Promise<void> }
| undefined;
readonly #preparation = new ArtifactPreparation();
#metadata: ArtifactFormatMetadata | undefined;
#limitations: readonly string[] = [];
#closing = false;
@@ -67,27 +63,11 @@ export class NtfsMftArtifactReader implements ArtifactReader {
): AsyncIterable<ArtifactEntry> {
await this.#scope.run(async () => {
this.#assertOpen(signal);
for (;;) {
const pending = (this.#prepared ??= {
signal,
promise: this.#prepare(signal),
});
try {
await pending.promise;
break;
} catch (cause: unknown) {
if (this.#prepared === pending) this.#prepared = undefined;
if (
pending.signal !== signal &&
signal?.aborted !== true &&
artifactReadCancelledBy(cause, pending.signal)
) {
this.#assertOpen(signal);
continue;
}
throw cause;
}
}
await this.#preparation.run(
signal,
() => this.#prepare(signal),
() => this.#assertOpen(signal),
);
this.#assertOpen(signal);
await this.#source.read(signal);
});
+8 -31
View File
@@ -1,3 +1,4 @@
import { ArtifactPreparation } from "./ArtifactPreparation.js";
import { Readable, Transform, pipeline } from "node:stream";
import { setImmediate } from "node:timers/promises";
@@ -11,10 +12,7 @@ import {
type ArtifactTraversalOptions,
} from "./ArtifactReader.js";
import { ArtifactResourceScope } from "./ArtifactResourceScope.js";
import {
artifactReadCancelledBy,
assertArtifactReadActive,
} from "./ArtifactFileOperation.js";
import { assertArtifactReadActive } from "./ArtifactFileOperation.js";
import { ArtifactFileSource } from "./ArtifactFileSource.js";
import { readFileHandleChunks } from "../filesystem/readFileHandleChunks.js";
import type { StableRegularFileDescriptor } from "../filesystem/RegularFile.js";
@@ -45,12 +43,7 @@ export abstract class PositionalArtifactReader implements ArtifactReader {
readonly #streams = new Set<Readable>();
readonly #label: string;
readonly #source: ArtifactFileSource;
#prepared:
| {
readonly signal: AbortSignal | undefined;
readonly promise: Promise<void>;
}
| undefined;
readonly #preparation = new ArtifactPreparation();
#coverage: ArtifactReaderTraversal | undefined;
#closing = false;
@@ -84,27 +77,11 @@ export abstract class PositionalArtifactReader implements ArtifactReader {
): AsyncIterable<ArtifactEntry> {
await this.#scope.run(async () => {
this.assertOpen(signal);
for (;;) {
const pending = (this.#prepared ??= {
signal,
promise: this.#prepare(signal),
});
try {
await pending.promise;
break;
} catch (cause: unknown) {
if (this.#prepared === pending) this.#prepared = undefined;
if (
pending.signal !== signal &&
signal?.aborted !== true &&
artifactReadCancelledBy(cause, pending.signal)
) {
this.assertOpen(signal);
continue;
}
throw cause;
}
}
await this.#preparation.run(
signal,
() => this.#prepare(signal),
() => this.assertOpen(signal),
);
await this.#source.read(signal);
});
this.assertOpen(signal);
+1 -1
View File
@@ -2551,7 +2551,7 @@ const embeddedPackageFormat = async (
nativeFormat: NativeFormat,
state: ScanState,
context: ScanContext,
location = filePath,
location: string,
expectedSha256?: string,
): Promise<EmbeddedPackageProbe> => {
try {
+8 -28
View File
@@ -1,3 +1,4 @@
import { ArtifactPreparation } from "./ArtifactPreparation.js";
import { Readable, Transform, pipeline } from "node:stream";
import { crc32 } from "node:zlib";
import { readArtifactFileBytes } from "./ArtifactFileRange.js";
@@ -8,10 +9,7 @@ import type {
UImageMetadata,
} from "../domain/artifactFormatMetadata.js";
import { ArtifactFileSource } from "./ArtifactFileSource.js";
import {
artifactReadCancelledBy,
assertArtifactReadActive,
} from "./ArtifactFileOperation.js";
import { assertArtifactReadActive } from "./ArtifactFileOperation.js";
import {
ArtifactReaderFailure,
copyArtifactEntry,
@@ -50,9 +48,7 @@ export class UImageArtifactReader implements ArtifactReader {
readonly #entries = new Map<string, UImageEntry>();
readonly #streams = new Set<Readable>();
readonly #source: ArtifactFileSource;
#prepared:
| { signal: AbortSignal | undefined; promise: Promise<void> }
| undefined;
readonly #preparation = new ArtifactPreparation();
#header: UImageMetadata | undefined;
#closing = false;
@@ -80,27 +76,11 @@ export class UImageArtifactReader implements ArtifactReader {
): AsyncIterable<ArtifactEntry> {
await this.#scope.run(async () => {
this.#assertOpen(signal);
for (;;) {
const pending = (this.#prepared ??= {
signal,
promise: this.#prepare(signal),
});
try {
await pending.promise;
break;
} catch (cause: unknown) {
if (this.#prepared === pending) this.#prepared = undefined;
if (
pending.signal !== signal &&
signal?.aborted !== true &&
artifactReadCancelledBy(cause, pending.signal)
) {
this.#assertOpen(signal);
continue;
}
throw cause;
}
}
await this.#preparation.run(
signal,
() => this.#prepare(signal),
() => this.#assertOpen(signal),
);
await this.#source.read(signal);
});
this.#assertOpen(signal);
+8 -28
View File
@@ -1,3 +1,4 @@
import { ArtifactPreparation } from "./ArtifactPreparation.js";
import { artifactExpansionBudgetBytes } from "../domain/artifactExpansionBudget.js";
import type { FileHandle } from "node:fs/promises";
import { setImmediate } from "node:timers/promises";
@@ -12,10 +13,7 @@ import {
type ArtifactTraversalOptions,
} from "./ArtifactReader.js";
import { ArtifactResourceScope } from "./ArtifactResourceScope.js";
import {
artifactReadCancelledBy,
assertArtifactReadActive,
} from "./ArtifactFileOperation.js";
import { assertArtifactReadActive } from "./ArtifactFileOperation.js";
import { readArtifactFileBytes } from "./ArtifactFileRange.js";
import type {
ArtifactFormatMetadata,
@@ -58,9 +56,7 @@ export class UnityFsArtifactReader implements ArtifactReader {
#entries = new Map<string, UnityFsEntryState>();
readonly #streams = new Map<Readable, AbortController>();
readonly #source: ArtifactFileSource;
#prepared:
| { signal: AbortSignal | undefined; promise: Promise<void> }
| undefined;
readonly #preparation = new ArtifactPreparation();
#header: UnityFsMetadata | undefined;
#blocks: readonly UnityFsBlock[] = [];
#dataStart = 0;
@@ -93,27 +89,11 @@ export class UnityFsArtifactReader implements ArtifactReader {
): AsyncIterable<ArtifactEntry> {
await this.#scope.run(async () => {
this.#assertOpen(signal);
for (;;) {
const pending = (this.#prepared ??= {
signal,
promise: this.#prepare(signal),
});
try {
await pending.promise;
break;
} catch (cause: unknown) {
if (this.#prepared === pending) this.#prepared = undefined;
if (
pending.signal !== signal &&
signal?.aborted !== true &&
artifactReadCancelledBy(cause, pending.signal)
) {
this.#assertOpen(signal);
continue;
}
throw cause;
}
}
await this.#preparation.run(
signal,
() => this.#prepare(signal),
() => this.#assertOpen(signal),
);
await this.#source.read(signal);
});
this.#assertOpen(signal);
+8 -28
View File
@@ -1,3 +1,4 @@
import { ArtifactPreparation } from "./ArtifactPreparation.js";
import { artifactExpansionBudgetBytes } from "../domain/artifactExpansionBudget.js";
import type { FileHandle } from "node:fs/promises";
import { setImmediate } from "node:timers/promises";
@@ -12,10 +13,7 @@ import {
type ArtifactTraversalOptions,
} from "./ArtifactReader.js";
import { ArtifactResourceScope } from "./ArtifactResourceScope.js";
import {
artifactReadCancelledBy,
assertArtifactReadActive,
} from "./ArtifactFileOperation.js";
import { assertArtifactReadActive } from "./ArtifactFileOperation.js";
import { readArtifactFileBytes } from "./ArtifactFileRange.js";
import type {
ArtifactFormatMetadata,
@@ -54,9 +52,7 @@ export class WxapkgArtifactReader implements ArtifactReader {
#entries = new Map<string, WxapkgEntryState>();
readonly #streams = new Map<Readable, AbortController>();
readonly #source: ArtifactFileSource;
#prepared:
| { signal: AbortSignal | undefined; promise: Promise<void> }
| undefined;
readonly #preparation = new ArtifactPreparation();
#header: WxapkgMetadata | undefined;
#remainingReadOperations = 0;
#admittedReadOperations = 0;
@@ -86,27 +82,11 @@ export class WxapkgArtifactReader implements ArtifactReader {
): AsyncIterable<ArtifactEntry> {
await this.#scope.run(async () => {
this.#assertOpen(signal);
for (;;) {
const pending = (this.#prepared ??= {
signal,
promise: this.#prepare(signal),
});
try {
await pending.promise;
break;
} catch (cause: unknown) {
if (this.#prepared === pending) this.#prepared = undefined;
if (
pending.signal !== signal &&
signal?.aborted !== true &&
artifactReadCancelledBy(cause, pending.signal)
) {
this.#assertOpen(signal);
continue;
}
throw cause;
}
}
await this.#preparation.run(
signal,
() => this.#prepare(signal),
() => this.#assertOpen(signal),
);
await this.#source.read(signal);
});
this.#assertOpen(signal);
@@ -1014,8 +1014,8 @@ const isInterfaceBuilderArchive = (path: string): boolean => {
const readEntry = async (
reader: DirectoryArtifactReader,
entry: ArtifactEntry,
signal?: AbortSignal,
maxInputBytes = MAX_AGGREGATE_INPUT_BYTES,
signal: AbortSignal | undefined,
maxInputBytes: number,
): Promise<Buffer> => {
if (entry.declaredSize !== null && entry.declaredSize > maxInputBytes)
throw new InterfaceBuilderDecodeBudgetExceeded(
+49 -14
View File
@@ -1,6 +1,34 @@
import { describe, expect, it } from "vitest";
import { projectNibViewHierarchy } from "./NibViewHierarchy.js";
import { InterfaceBuilderDecodeBudget } from "./InterfaceBuilderDecodeBudget.js";
// Use the same budget and diagnostic accounting as Interface Builder's NIB owner.
const projectFixture = (
objects: Parameters<typeof projectNibViewHierarchy>[0],
retainedIds: ReadonlySet<number>,
expectedDiagnostics: readonly string[] = [],
maximumBytes = 4_000_000,
) => {
const budget = new InterfaceBuilderDecodeBudget(maximumBytes);
const diagnostics: string[] = [];
const projection = projectNibViewHierarchy(
objects,
retainedIds,
() => budget.tryReserve(128),
20_000,
(description) => {
budget.reserve(
256,
"NIB registry hierarchy diagnostics exceed the aggregate Interface Builder decode budget",
);
budget.reserveJsonStringOccurrence(description);
diagnostics.push(description);
},
);
expect(diagnostics).toEqual(expectedDiagnostics);
return projection;
};
const object = (id: number, parent?: number) => ({
id,
@@ -11,7 +39,7 @@ const object = (id: number, parent?: number) => ({
describe("serialized NIB view hierarchy bounds", () => {
it("does not invent hierarchy for objects without serialized relationships", () => {
expect(projectNibViewHierarchy([object(0)], new Set([0]))).toEqual({
expect(projectFixture([object(0)], new Set([0]))).toEqual({
hierarchy: null,
omitted: 0,
});
@@ -19,7 +47,7 @@ describe("serialized NIB view hierarchy bounds", () => {
it("uses the serialized parent direction and retains all sibling links", () => {
expect(
projectNibViewHierarchy(
projectFixture(
[object(0), object(1, 0), object(2, 0)],
new Set([0, 1, 2]),
),
@@ -38,7 +66,7 @@ describe("serialized NIB view hierarchy bounds", () => {
});
it("reports missing parents without inventing parentage", () => {
expect(projectNibViewHierarchy([object(0, 99)], new Set([0]))).toEqual({
expect(projectFixture([object(0, 99)], new Set([0]))).toEqual({
hierarchy: [],
omitted: 1,
});
@@ -46,7 +74,7 @@ describe("serialized NIB view hierarchy bounds", () => {
it("reports a missing encoded window content view as incomplete", () => {
expect(
projectNibViewHierarchy(
projectFixture(
[
{
id: 0,
@@ -61,7 +89,7 @@ describe("serialized NIB view hierarchy bounds", () => {
it("marks unsupported serialized parent values as incomplete", () => {
expect(
projectNibViewHierarchy(
projectFixture(
[{ id: 0, class_name: "NSView", values: { NSSuperview: "unknown" } }],
new Set([0]),
),
@@ -70,7 +98,7 @@ describe("serialized NIB view hierarchy bounds", () => {
it("reports cyclic and self-referential parentage as incomplete", () => {
expect(
projectNibViewHierarchy(
projectFixture(
[object(0, 1), object(1, 0), object(2, 2)],
new Set([0, 1, 2]),
),
@@ -81,9 +109,7 @@ describe("serialized NIB view hierarchy bounds", () => {
});
it("does not project children whose parent was not retained", () => {
expect(
projectNibViewHierarchy([object(0), object(1, 0)], new Set([1])),
).toEqual({
expect(projectFixture([object(0), object(1, 0)], new Set([1]))).toEqual({
hierarchy: [],
omitted: 1,
});
@@ -94,8 +120,7 @@ describe("serialized NIB view hierarchy bounds", () => {
object(id, id === 0 ? undefined : id - 1),
);
expect(
projectNibViewHierarchy(objects, new Set(objects.map(({ id }) => id)))
.omitted,
projectFixture(objects, new Set(objects.map(({ id }) => id))).omitted,
).toBe(3);
});
@@ -104,14 +129,21 @@ describe("serialized NIB view hierarchy bounds", () => {
object(id, id === 0 ? undefined : 0),
);
expect(
projectNibViewHierarchy(objects, new Set(objects.map(({ id }) => id)))
.omitted,
projectFixture(objects, new Set(objects.map(({ id }) => id))).omitted,
).toBe(2);
});
it("reports nodes the shared representation budget cannot reserve", () => {
const objects = [object(0), object(1, 0)];
expect(projectFixture(objects, new Set([0, 1]), [], 128)).toEqual({
hierarchy: [{ objectID: "0", children: [] }],
omitted: 1,
});
});
});
it("does not treat a dangling registry owner as top-level authority", () => {
const result = projectNibViewHierarchy(
const result = projectFixture(
[
{
id: 0,
@@ -135,6 +167,9 @@ it("does not treat a dangling registry owner as top-level authority", () => {
},
],
new Set([1]),
[
"NSIBObjectData object 0: NSObjectsKeys/NSObjectsValues pair child 1 with registry value 99; NSRoot is 99 (unresolved). This decoded registry association does not establish visual parentage.",
],
);
expect(result.hierarchy).toEqual([]);
expect(result.omitted).toBeGreaterThan(0);
+3 -3
View File
@@ -56,9 +56,9 @@ const isNonViewCell = (
export const projectNibViewHierarchy = (
objects: NibArchiveDocument["objects"],
retainedIds: ReadonlySet<number>,
reserveNode: () => boolean = () => true,
maxNodes = 20_000,
reportUnknownRegistry: (description: string) => void = () => {},
reserveNode: () => boolean,
maxNodes: number,
reportUnknownRegistry: (description: string) => void,
) => {
const byId = new Map(objects.map((object) => [object.id, object]));
const parents = new Map<number, number | null>();
@@ -74,13 +74,10 @@ const XML_REFERENCE_BYTES = 64;
const estimateXmlPlistExpansion = (
bytes: Buffer,
maxBytes: number,
xmlText?: string,
numberProjection = false,
budget = "the property list decode budget",
fail: (
kind: BinaryPlistExpansionFailure,
message: string,
) => Error = propertyListDecodeBudgetFailure,
xmlText: string | undefined,
numberProjection: boolean,
budget: string,
fail: (kind: BinaryPlistExpansionFailure, message: string) => Error,
): number => {
const xml = xmlText ?? decodeXmlPlistText(bytes);
// The decoded text and the DOM's copies of its character data.
+10 -5
View File
@@ -5,10 +5,7 @@ import {
ZIP_PACKAGE_FORMATS,
type ZipPackageFormat,
} from "../domain/zipPackageFormat.js";
import {
ARTIFACT_FORMATS,
ZIP_FAMILY_CAPABILITIES,
} from "./artifactFormats.js";
import { ARTIFACT_FORMATS } from "./artifactFormats.js";
import {
createArtifactReader,
type ArtifactReaderFormat,
@@ -76,7 +73,15 @@ describe("artifact format descriptors", () => {
it("shares one capability row across zip package families", () => {
for (const id of ZIP_PACKAGE_FORMATS)
expect(ARTIFACT_FORMATS[id]).toMatchObject(ZIP_FAMILY_CAPABILITIES);
expect(ARTIFACT_FORMATS[id]).toMatchObject({
zipFamily: true,
occurrenceContainer: true,
roleContainer: false,
retainClassifiedSource: true,
extractable: true,
requiresDarwin: false,
absentInventory: "unavailable",
});
const zipIds = new Set<ZipPackageFormat>(ZIP_PACKAGE_FORMATS);
for (const id of ARTIFACT_FORMAT_IDS)
if (!zipIds.has(id as ZipPackageFormat))
+1 -1
View File
@@ -134,7 +134,7 @@ function admittedAs(
}
/** One capability row for every ZIP-backed package family. */
export const ZIP_FAMILY_CAPABILITIES = {
const ZIP_FAMILY_CAPABILITIES = {
zipFamily: true,
occurrenceContainer: true,
roleContainer: false,
+7 -5
View File
@@ -6,13 +6,11 @@ import { open, writeFile } from "node:fs/promises";
import { join } from "node:path";
import { expect, it, vi } from "vitest";
import { createTestTempDirectory } from "../../../tests/fixtures/temporaryDirectory.js";
import {
probeBunStandaloneGraph,
BUN_GRAPH_SUFFIX_BYTES,
} from "./BunModuleGraphLayout.js";
import { probeBunStandaloneGraph } from "./BunModuleGraphLayout.js";
import { decodeBunModuleGraph } from "./BunModuleGraphMembers.js";
import {
locateBunGraphFixture,
BUN_FIXTURE_GRAPH_SUFFIX_BYTES,
bunModuleGraphPayload,
bunPrefixedPayload,
machoBunSection,
@@ -167,7 +165,11 @@ it.each(["trailer", "byte_count"])(
modules: [{ name: "/$bunfs/root/main.js" }],
});
if (field === "trailer") payload[payload.length - 1] = 0;
else payload.writeBigUInt64LE(0n, payload.length - BUN_GRAPH_SUFFIX_BYTES);
else
payload.writeBigUInt64LE(
0n,
payload.length - BUN_FIXTURE_GRAPH_SUFFIX_BYTES,
);
const prefix = Buffer.alloc(4);
prefix.writeUInt32LE(payload.length);
expect(
+4 -4
View File
@@ -17,10 +17,10 @@ import { abortIfNeeded } from "../ArtifactHash.js";
import { ArtifactReaderFailure } from "../ArtifactReader.js";
/** Exact trailer written by Bun's standalone module graph serializer. */
export const BUN_TRAILER = Buffer.from("\n---- Bun! ----\n");
export const BUN_TRAILER_BYTES = BUN_TRAILER.length;
export const BUN_OFFSETS_BYTES = 32;
export const BUN_GRAPH_SUFFIX_BYTES = BUN_OFFSETS_BYTES + BUN_TRAILER_BYTES;
const BUN_TRAILER = Buffer.from("\n---- Bun! ----\n");
const BUN_TRAILER_BYTES = BUN_TRAILER.length;
const BUN_OFFSETS_BYTES = 32;
const BUN_GRAPH_SUFFIX_BYTES = BUN_OFFSETS_BYTES + BUN_TRAILER_BYTES;
/** Current compilers prefix a section with this little-endian payload length. */
export const BUN_LENGTH_PREFIX_BYTES = 8;
/** Bun 1.3.3 through 1.3.8 module records. */
+14 -9
View File
@@ -1,9 +1,5 @@
import type { FileHandle } from "node:fs/promises";
import {
BUN_OFFSETS_BYTES,
BUN_RECORD_BYTES_1_3_3,
BUN_RECORD_BYTES_1_3_9,
BUN_TRAILER,
loadBunStandaloneGraph,
probeBunStandaloneGraph,
type BunLayoutRevision,
@@ -11,6 +7,15 @@ import {
type BunStringPointer,
} from "./BunModuleGraphLayout.js";
// https://github.com/oven-sh/bun/blob/bun-v1.3.9/src/StandaloneModuleGraph.zig
// and the 1.3.3 serializer define the two supported layouts.
const OFFSETS_BYTES = 32;
const RECORD_BYTES_1_3_3 = 36;
const RECORD_BYTES_1_3_9 = 52;
export const BUN_FIXTURE_TRAILER = Buffer.from("\n---- Bun! ----\n");
export const BUN_FIXTURE_GRAPH_SUFFIX_BYTES =
OFFSETS_BYTES + BUN_FIXTURE_TRAILER.length;
/** Exercise the same two-stage probe and payload read as the artifact reader. */
export const locateBunGraphFixture = async (
handle: FileHandle,
@@ -46,7 +51,7 @@ export const bunModuleGraphPayload = (input: {
}): Buffer => {
const revision = input.revision ?? "bun-1.3.9";
const recordBytes =
revision === "bun-1.3.3" ? BUN_RECORD_BYTES_1_3_3 : BUN_RECORD_BYTES_1_3_9;
revision === "bun-1.3.3" ? RECORD_BYTES_1_3_3 : RECORD_BYTES_1_3_9;
const modules = input.modules ?? [];
const chunks: Buffer[] = [];
let cursor = 0;
@@ -87,8 +92,8 @@ export const bunModuleGraphPayload = (input: {
writePointer(record, 8, contents);
writePointer(record, 16, row.sourcemap);
writePointer(record, 24, row.bytecode);
const metadataAt = recordBytes === BUN_RECORD_BYTES_1_3_9 ? 48 : 32;
if (recordBytes === BUN_RECORD_BYTES_1_3_9) {
const metadataAt = recordBytes === RECORD_BYTES_1_3_9 ? 48 : 32;
if (recordBytes === RECORD_BYTES_1_3_9) {
writePointer(record, 32, row.moduleInfo);
writePointer(record, 40, row.origin);
}
@@ -100,13 +105,13 @@ export const bunModuleGraphPayload = (input: {
chunks.push(records.subarray(0, tableLength));
cursor += tableLength;
const data = Buffer.concat(chunks);
const offsets = Buffer.alloc(BUN_OFFSETS_BYTES);
const offsets = Buffer.alloc(OFFSETS_BYTES);
offsets.writeBigUInt64LE(BigInt(data.length), 0);
writePointer(offsets, 8, { offset: tableOffset, length: tableLength });
offsets.writeUInt32LE(input.entryPointId ?? 0, 16);
writePointer(offsets, 20, argv);
offsets.writeUInt32LE(input.flags ?? 0, 28);
return Buffer.concat([data, offsets, BUN_TRAILER]);
return Buffer.concat([data, offsets, BUN_FIXTURE_TRAILER]);
};
/** Length prefix used by current Mach-O, PE, and ELF section writers. */
+13 -5
View File
@@ -1,18 +1,26 @@
import { expect, it } from "vitest";
import {
DENO_SECTION_MAGIC,
MAX_DENO_METADATA_JSON_BYTES,
MAX_DENO_TABLE_COUNT,
parseDenoSectionBytes,
parseDenoSection,
} from "./DenoSectionLayout.js";
import {
buildDenoSection,
defaultDenoMetadata,
DENO_FIXTURE_SECTION_MAGIC,
} from "./DenoStandalone.fixture.js";
const secret = "super-secret-value";
const parseDenoSectionBytes = (bytes: Buffer, signal?: AbortSignal) =>
parseDenoSection(
bytes.length,
(offset, length) =>
Promise.resolve(bytes.subarray(offset, offset + length)),
signal,
);
it("parses metadata, tables, remote regions, and embedded configuration", async () => {
const original = Buffer.from("export const value = 1;\n");
const transpiled = Buffer.from("transpiled\n");
@@ -123,9 +131,9 @@ it("treats a missing opening marker as an unsupported layout", async () => {
it("refuses metadata JSON above the named allowance before reading it as JSON", async () => {
const claimed = MAX_DENO_METADATA_JSON_BYTES + 1;
const bytes = Buffer.alloc(DENO_SECTION_MAGIC.length + 8 + claimed);
DENO_SECTION_MAGIC.copy(bytes, 0);
bytes.writeBigUInt64LE(BigInt(claimed), DENO_SECTION_MAGIC.length);
const bytes = Buffer.alloc(DENO_FIXTURE_SECTION_MAGIC.length + 8 + claimed);
DENO_FIXTURE_SECTION_MAGIC.copy(bytes, 0);
bytes.writeBigUInt64LE(BigInt(claimed), DENO_FIXTURE_SECTION_MAGIC.length);
await expect(parseDenoSectionBytes(bytes)).rejects.toMatchObject({
reason: "limit",
message: expect.stringContaining("metadata JSON"),
-12
View File
@@ -366,18 +366,6 @@ export const parseDenoSection = async (
};
};
/** Parse a section that is already in memory. Tests and small fixtures use this. */
export const parseDenoSectionBytes = (
bytes: Buffer,
signal?: AbortSignal,
): Promise<DenoParsedSection> =>
parseDenoSection(
bytes.length,
(offset, length) =>
Promise.resolve(bytes.subarray(offset, offset + length)),
signal,
);
const addRemoteBody = (total: bigint, length: number): bigint => {
const next = total + BigInt(length);
if (next > BigInt(MAX_DENO_REMOTE_BODY_BYTES))
+5 -3
View File
@@ -1,7 +1,9 @@
import { elfNote, elfNoteFixture } from "../../native/elf/ElfNotes.fixture.js";
import { peResourceFixture } from "../../native/pe/PeResources.fixture.js";
import { DENO_SECTION_MAGIC } from "./DenoSectionLayout.js";
// https://github.com/denoland/deno/blob/v2.9.8/cli/lib/standalone/binary.rs
// declares this section marker.
export const DENO_FIXTURE_SECTION_MAGIC = Buffer.from("d3n0l4nd", "latin1");
export interface DenoSpecifierFixture {
readonly text?: string;
@@ -93,7 +95,7 @@ export const buildDenoSection = (options: DenoSectionFixture = {}): Buffer => {
const vfsBytes = Buffer.from(JSON.stringify(vfs), "utf8");
const parts: Buffer[] = [];
if (options.openingMagic !== null)
parts.push(options.openingMagic ?? DENO_SECTION_MAGIC);
parts.push(options.openingMagic ?? DENO_FIXTURE_SECTION_MAGIC);
parts.push(
u64(options.metadataLength ?? metadataBytes.length),
metadataBytes,
@@ -138,7 +140,7 @@ export const buildDenoSection = (options: DenoSectionFixture = {}): Buffer => {
}
parts.push(u64(vfsBytes.length), vfsBytes, u64(contents.length), contents);
if (options.closingMagic !== null)
parts.push(options.closingMagic ?? DENO_SECTION_MAGIC);
parts.push(options.closingMagic ?? DENO_FIXTURE_SECTION_MAGIC);
if (options.trailing !== undefined) parts.push(options.trailing);
return Buffer.concat(parts);
};
@@ -19,13 +19,13 @@ import { artifactFormatDescriptor } from "../artifactFormats.js";
import { ArtifactPathRegistry } from "../ArtifactPaths.js";
import { DenoStandaloneArtifactReader } from "../DenoStandaloneArtifactReader.js";
import { cArchiveExecutable } from "../pyinstaller/CArchive.fixture.js";
import { DENO_SECTION_MAGIC } from "./DenoSectionLayout.js";
import {
locateDenoStandalone,
MAX_DENO_INTEL_TRAILER_SENTINELS,
} from "./DenoStandaloneLocation.js";
import {
buildDenoSection,
DENO_FIXTURE_SECTION_MAGIC,
defaultDenoMetadata,
denoElf,
denoIntelTrailer,
@@ -249,7 +249,7 @@ it("fails closed when a named section is truncated or lacks the opening marker",
await expect(
inventoryOf(
"truncated",
denoMachO(Buffer.concat([DENO_SECTION_MAGIC, Buffer.alloc(4)])),
denoMachO(Buffer.concat([DENO_FIXTURE_SECTION_MAGIC, Buffer.alloc(4)])),
),
).rejects.toMatchObject({ reason: "format" });
await expect(
+7 -55
View File
@@ -1,4 +1,7 @@
import { createHash } from "node:crypto";
import {
hashArtifactRange,
readArtifactRange as readExact,
} from "../ArtifactCarrierRange.js";
import type { FileHandle } from "node:fs/promises";
import {
@@ -895,7 +898,9 @@ const collapseRanges = async (
"capacity",
);
remainingBytes -= range.length;
const hash = await digestRange(carrier, range.offset, range.length);
const hash = await hashArtifactRange(carrier, range.offset, range.length);
if (hash === undefined)
throw denoCarrierFailure("Deno compile section is truncated");
if (chosenHash === undefined) chosenHash = hash;
else if (hash !== chosenHash) throw conflict();
}
@@ -908,59 +913,6 @@ const collapseRanges = async (
};
};
const digestRange = async (
carrier: Carrier,
offset: number,
length: number,
): Promise<string> => {
const hash = createHash("sha256");
let remaining = length;
let position = offset;
while (remaining > 0) {
const chunkLength = Math.min(64 * 1024, remaining);
const chunk = await readExact(carrier, position, chunkLength);
if (chunk === undefined)
throw denoCarrierFailure("Deno compile section is truncated");
hash.update(chunk);
position += chunkLength;
remaining -= chunkLength;
}
return hash.digest("hex");
};
const readExact = async (
carrier: Carrier,
offset: number,
length: number,
): Promise<Buffer | undefined> => {
abortIfNeeded(carrier.signal);
if (
!Number.isSafeInteger(offset) ||
!Number.isSafeInteger(length) ||
offset < 0 ||
length < 0 ||
offset > carrier.fileSize ||
length > carrier.fileSize - offset
)
return undefined;
if (length === 0) return Buffer.alloc(0);
const bytes = Buffer.alloc(length);
let filled = 0;
while (filled < length) {
abortIfNeeded(carrier.signal);
const read = await carrier.handle.read(
bytes,
filled,
length - filled,
offset + filled,
);
abortIfNeeded(carrier.signal);
if (read.bytesRead === 0) return undefined;
filled += read.bytesRead;
}
return bytes;
};
const fixedCString = (bytes: Uint8Array): string => {
const end = bytes.indexOf(0);
return Buffer.from(end < 0 ? bytes : bytes.subarray(0, end)).toString(
+5 -5
View File
@@ -37,10 +37,10 @@ export const DENO_CHILD_SUFFIXES = {
sourceMap: ".deno-sourcemap",
} as const;
const INDEXED_PATH_LIMITATION = `The virtual name exceeds the fixed ${String(MAX_DENO_LOGICAL_PATH_PARTS)}-part logical path allowance; an indexed path is used and the original name is retained.`;
export const DENO_PATH_PARTS_LIMITATION = `The virtual name exceeds the fixed ${String(MAX_DENO_LOGICAL_PATH_PARTS)}-part logical path allowance; an indexed path is used and the original name is retained.`;
const NAME_LIMITATION = `The virtual name exceeds the fixed ${String(MAX_DENO_VFS_NAME_UNITS)}-unit name allowance; an indexed path is used and the original name is not retained.`;
export const DENO_PATH_LENGTH_LIMITATION = `The virtual name exceeds the fixed ${String(MAX_DENO_PATH_COMPONENT_BYTES)}-byte path component or ${String(MAX_DENO_LOGICAL_PATH_BYTES)}-byte logical path allowance; an indexed path is used and the original name is retained.`;
const UNSAFE_PATH_LIMITATION =
export const DENO_UNSAFE_PATH_LIMITATION =
"The virtual name is not a safe relative path; an indexed path is used and the original name is retained.";
const MTIME_LIMITATION = "The file mtime is not a safe integer; it is omitted.";
const SYMLINK_PART_LIMITATION = `A symlink target part exceeds the fixed ${String(MAX_DENO_VFS_NAME_UNITS)}-unit name allowance; the target parts are not retained.`;
@@ -384,10 +384,10 @@ const logicalDestination = (
if (parent.limitation !== null) return indexed(parent.limitation, source);
if (name.length > MAX_DENO_VFS_NAME_UNITS)
return indexed(NAME_LIMITATION, null);
if (unsafeName(name)) return indexed(UNSAFE_PATH_LIMITATION, source);
if (unsafeName(name)) return indexed(DENO_UNSAFE_PATH_LIMITATION, source);
const parts = parent.parts + 1;
if (parts > MAX_DENO_LOGICAL_PATH_PARTS)
return indexed(INDEXED_PATH_LIMITATION, source);
return indexed(DENO_PATH_PARTS_LIMITATION, source);
const joined = parent.path === null ? name : `${parent.path}/${name}`;
try {
const path = normalizeArtifactPath(joined);
@@ -402,7 +402,7 @@ const logicalDestination = (
};
} catch (cause) {
if (cause instanceof ArtifactReaderFailure && cause.reason === "path")
return indexed(UNSAFE_PATH_LIMITATION, source);
return indexed(DENO_UNSAFE_PATH_LIMITATION, source);
throw cause;
}
};
@@ -560,7 +560,7 @@ export const classifyArtifactPath = (
export const classifyArtifactContent = (
path: string,
evidence: ArtifactClassificationEvidence,
fileSize = evidence.prefix.length,
fileSize: number,
): {
readonly kind: ArtifactOccurrence["artifact_kind"];
readonly format: ArtifactOccurrence["artifact_format"];
@@ -620,7 +620,7 @@ const artifactRoleForFormat = (
/** Classify storage bytes independently of path names, roles, and permissions. */
export const classifyArtifactBytes = (
evidence: ArtifactClassificationEvidence,
fileSize = evidence.prefix.length,
fileSize: number,
): ArtifactNode["format"] => {
const { prefix, suffix, mzWindowsHeaderSignature } = evidence;
// A header-only region has the signature but not the declared cartridge
+3 -2
View File
@@ -1,7 +1,8 @@
import { elfNote, elfNoteFixture } from "../../native/elf/ElfNotes.fixture.js";
import { peResourceFixture } from "../../native/pe/PeResources.fixture.js";
import { NODE_SEA_MAGIC } from "./NodeSeaBlob.js";
/** Node 24 SEA blob magic: https://github.com/nodejs/node/blob/v24.12.0/src/node_sea.h */
export const NODE_SEA_FIXTURE_MAGIC = 0x143da20;
/** Official Node `NODE_SEA_FUSE_fce680ab2cc467b6e072b8b5df1996b2` hash. */
export const NODE_SEA_FUSE_HASH = "fce680ab2cc467b6e072b8b5df1996b2";
@@ -62,7 +63,7 @@ export const buildNodeSeaBlob = (
if (assets.length > 0) flags |= 1 << 3;
if (execArgv.length > 0) flags |= 1 << 4;
const header = Buffer.alloc(9);
header.writeUInt32LE(NODE_SEA_MAGIC, 0);
header.writeUInt32LE(NODE_SEA_FIXTURE_MAGIC, 0);
header.writeUInt32LE(flags, 4);
header[8] = options.execArgvExtension ?? 1;
const parts: Buffer[] = [header, sized(codePath), sized(body)];
+3 -7
View File
@@ -3,16 +3,12 @@ import { setImmediate } from "node:timers/promises";
import { expect, it } from "vitest";
import { DEFAULT_NODE_SEA_READER_LIMITS } from "../../domain/artifactReaderLimits.js";
import {
NODE_SEA_MAGIC,
decodeNodeSeaBlob,
observeNodeSeaFuseMatches,
} from "./NodeSeaBlob.js";
import { buildNodeSeaBlob } from "./NodeSea.fixture.js";
import { decodeNodeSeaBlob, observeNodeSeaFuseMatches } from "./NodeSeaBlob.js";
import { buildNodeSeaBlob, NODE_SEA_FIXTURE_MAGIC } from "./NodeSea.fixture.js";
const header = (flags: number, extension = 1): Buffer => {
const bytes = Buffer.alloc(9);
bytes.writeUInt32LE(NODE_SEA_MAGIC, 0);
bytes.writeUInt32LE(NODE_SEA_FIXTURE_MAGIC, 0);
bytes.writeUInt32LE(flags, 4);
bytes[8] = extension;
return bytes;
+1 -1
View File
@@ -9,7 +9,7 @@ import {
import { artifactReaderLimitFailure } from "../ArtifactReaderLimits.js";
/** Node `SeaBlob::kMagic` from `src/node_sea.h` (v24). */
export const NODE_SEA_MAGIC = 0x143da20;
const NODE_SEA_MAGIC = 0x143da20;
const FLAG_DISABLE_EXPERIMENTAL_SEA_WARNING = 1 << 0;
const FLAG_USE_SNAPSHOT = 1 << 1;
+9 -54
View File
@@ -1,3 +1,7 @@
import {
hashArtifactRange,
readArtifactRange as readExact,
} from "../ArtifactCarrierRange.js";
import { readMachoSegmentHeader } from "../../domain/apple/machoSegmentHeader.js";
import { readMachoSectionHeader } from "../../domain/apple/machoSectionHeader.js";
import { readElfSectionHeader } from "../../domain/elfSectionHeader.js";
@@ -214,8 +218,11 @@ const collapseRanges = async (
if (chosen === undefined) return undefined;
if (unique.length > 1) {
const hashes = new Set<string>();
for (const range of unique)
hashes.add(await digestRange(carrier, range.offset, range.length));
for (const range of unique) {
const hash = await hashArtifactRange(carrier, range.offset, range.length);
if (hash === undefined) throw formatFailure(NODE_SEA_TRUNCATED);
hashes.add(hash);
}
if (hashes.size !== 1) throw formatFailure(NODE_SEA_DUPLICATE_BLOB);
}
return {
@@ -694,58 +701,6 @@ const locatePeResources = async (carrier: Carrier): Promise<BlobRange[]> => {
});
};
const digestRange = async (
carrier: Carrier,
offset: number,
length: number,
): Promise<string> => {
const hash = createHash("sha256");
let remaining = length;
let position = offset;
while (remaining > 0) {
const chunkLength = Math.min(64 * 1024, remaining);
const chunk = await readExact(carrier, position, chunkLength);
if (chunk === undefined) throw formatFailure(NODE_SEA_TRUNCATED);
hash.update(chunk);
position += chunkLength;
remaining -= chunkLength;
}
return hash.digest("hex");
};
const readExact = async (
carrier: Carrier,
offset: number,
length: number,
): Promise<Buffer | undefined> => {
abortIfNeeded(carrier.signal);
if (
!Number.isSafeInteger(offset) ||
!Number.isSafeInteger(length) ||
offset < 0 ||
length < 0 ||
offset > carrier.fileSize ||
length > carrier.fileSize - offset
)
return undefined;
if (length === 0) return Buffer.alloc(0);
const bytes = Buffer.alloc(length);
let filled = 0;
while (filled < length) {
abortIfNeeded(carrier.signal);
const read = await carrier.handle.read(
bytes,
filled,
length - filled,
offset + filled,
);
abortIfNeeded(carrier.signal);
if (read.bytesRead === 0) return undefined;
filled += read.bytesRead;
}
return bytes;
};
const cString = (bytes: Buffer, offset: number, length: number): string => {
let end = offset;
const limit = Math.min(bytes.length, offset + length);
@@ -6,7 +6,7 @@ import {
} from "../../python/PythonPyc.fixture.js";
// PyInstaller CArchiveWriter's on-disk cookie signature, independent of REA's reader.
const CARCHIVE_COOKIE_MAGIC = Buffer.from("4d45490c0b0a0b0e", "hex");
export const CARCHIVE_COOKIE_MAGIC = Buffer.from("4d45490c0b0a0b0e", "hex");
/** Final CPython 3.9 bytecode magic, little-endian with the CR LF trailer. */
export const PYTHON_39_MAGIC = Buffer.from("610d0d0a", "hex");
+1 -1
View File
@@ -8,7 +8,7 @@ import {
} from "./PyInstallerFailure.js";
/** Magic at the start of a CArchive cookie. */
export const PYINSTALLER_COOKIE_MAGIC = Buffer.from("MEI\x0c\x0b\x0a\x0b\x0e");
const PYINSTALLER_COOKIE_MAGIC = Buffer.from("MEI\x0c\x0b\x0a\x0b\x0e");
/** PyInstaller 2.1+ cookie size, including the 64-byte library name. */
export const PYINSTALLER_COOKIE_V21_BYTES = 88;
@@ -9,7 +9,10 @@ import { ArtifactResourceScope } from "../ArtifactResourceScope.js";
import { inventoryArtifact } from "../inventory/ArtifactInventory.js";
import { classifyAndHashRootForInventory } from "../inventory/classify.js";
import { PyInstallerArtifactReader } from "../PyInstallerArtifactReader.js";
import { cArchiveExecutable } from "./CArchive.fixture.js";
import {
CARCHIVE_COOKIE_MAGIC,
cArchiveExecutable,
} from "./CArchive.fixture.js";
import * as layout from "./CArchiveLayout.js";
const fixture = () =>
@@ -248,7 +251,7 @@ it("retains rejected magic candidates without classifying their native carrier a
path,
Buffer.concat([
fixture().subarray(0, 64),
layout.PYINSTALLER_COOKIE_MAGIC,
CARCHIVE_COOKIE_MAGIC,
Buffer.alloc(88),
]),
);
@@ -288,11 +291,7 @@ it("continues past an invalid trailing magic copy to inventory the valid cookie"
const path = join(root, "app");
await writeFile(
path,
Buffer.concat([
fixture(),
layout.PYINSTALLER_COOKIE_MAGIC,
Buffer.alloc(88),
]),
Buffer.concat([fixture(), CARCHIVE_COOKIE_MAGIC, Buffer.alloc(88)]),
);
const resourceScope = new ArtifactResourceScope();
try {
+2 -1
View File
@@ -131,10 +131,11 @@ export const decodedBodyBound = (
headers: Readonly<Record<string, string>>,
): number | undefined => {
const declared = headers["content-length"]?.trim();
// Content-Length allows leading zeros; only significant digits bound precision.
const bytes =
received !== undefined && Number.isSafeInteger(received) && received > 0
? received
: declared !== undefined && /^[1-9]\d{0,14}$/u.test(declared)
: declared !== undefined && /^0*[1-9]\d{0,14}$/u.test(declared)
? Number(declared)
: undefined;
if (bytes === undefined) return undefined;
+3 -3
View File
@@ -614,8 +614,8 @@ const isRecord = (value: unknown): value is Record<string, unknown> =>
const waitForOpen = async (
socket: WebSocket,
operation: BrowserObservationOperation,
signal?: AbortSignal,
commandTimeoutMs: number = DEFAULT_CDP_COMMAND_TIMEOUT_MS,
signal: AbortSignal | undefined,
handshakeTimeoutMs: number,
): Promise<void> =>
await new Promise((resolve, reject) => {
const onOpen = (): void => {
@@ -651,7 +651,7 @@ const waitForOpen = async (
};
const boundedTimeoutMs = Math.min(
MAX_BROWSER_DEADLINE_MS,
commandTimeoutMs,
handshakeTimeoutMs,
);
const timer = setTimeout(() => {
terminateSocket(socket);
+1 -1
View File
@@ -43,7 +43,7 @@ export interface CdpResponseFacts {
readonly encodedDataLength: number | undefined;
readonly headers: unknown;
}
export const readCdpResponseFacts = (
const readCdpResponseFacts = (
reported: unknown,
): CdpResponseFacts | undefined => {
const source = recordValue(reported);
+2 -2
View File
@@ -2,7 +2,7 @@ import { MIMEType } from "node:util";
import { describe, expect, it } from "vitest";
import { safeResponseMetadata as projectResponseMetadata } from "./CdpSafeMetadata.js";
import { readCdpResponseFacts } from "./CdpNetworkEventFacts.js";
import { readCdpResponseReceived } from "./CdpNetworkEventFacts.js";
import type { UnknownRecord } from "./CdpCaptureValues.js";
/** Admit each raw response fixture through the production field owner. */
@@ -12,7 +12,7 @@ const safeResponseMetadata = (
response: UnknownRecord,
allowedOrigins: ReadonlySet<string>,
) => {
const facts = readCdpResponseFacts(response);
const facts = readCdpResponseReceived({ requestId, response }).response;
if (facts === undefined)
throw new Error("Expected a response fixture record");
return projectResponseMetadata(requestId, url, facts, allowedOrigins);
+4 -3
View File
@@ -15,7 +15,6 @@ export interface InspectArgumentsPreflightOptions {
readonly signal?: AbortSignal;
readonly resources?: ArtifactResourceScope;
readonly readFuseImage?: typeof readElectronFuseImage;
readonly resolveFuseTarget?: typeof resolveElectronFuseTarget;
}
const appBundleDirectoryOf = (executablePath: string): string | null => {
@@ -47,13 +46,15 @@ export const assertNodeCliInspectArgumentsSupported = async (
): Promise<void> => {
assertArtifactReadActive(options?.signal);
const reader = options?.readFuseImage ?? readElectronFuseImage;
const resolver = options?.resolveFuseTarget ?? resolveElectronFuseTarget;
try {
let targetPath = executablePath;
const bundleDir = appBundleDirectoryOf(executablePath);
if (bundleDir !== null) {
try {
const resolved = await resolver(bundleDir, options?.signal);
const resolved = await resolveElectronFuseTarget(
bundleDir,
options?.signal,
);
targetPath = resolved.file.path;
} catch (cause: unknown) {
if (options?.signal?.aborted) throw cause;
+34 -17
View File
@@ -1699,6 +1699,14 @@ describe("saving from a UI step's response", () => {
{ encoding: "identity", size: 0 },
{ encoding: "gzip", size: 0 },
{ size: 0, length: "0" },
{ size: 0, length: "0000" },
{ size: 0, length: "+17" },
{ size: 0, length: "17.0" },
{ size: 0, length: "1.7e1" },
{ size: 0, length: "17, 17" },
{ size: 0, length: "0001000000000000000" },
{ size: 0, length: "9007199254740993" },
{ size: 0, length: "0009007199254740993" },
{ size: -1 },
{ size: Number.NaN },
])(
@@ -1831,7 +1839,10 @@ describe("saving from a compressed or cached UI response", () => {
{ encoding: "deflate", size: 32 },
{ encoding: "x-gzip", size: 32 },
{ size: 0, length: "17" },
{ size: 0, length: "0017" },
{ size: 0, length: "00000000000000000017" },
{ encoding: "gzip", size: 0, length: "32" },
{ encoding: "gzip", size: 0, length: "0032" },
])(
"saves from a response whose received size or declared length bounds its decoded size: %j",
async (bounds) => {
@@ -1852,23 +1863,29 @@ describe("saving from a compressed or cached UI response", () => {
},
);
it("refuses a compressed response that could decode past the limit without reading it", async () => {
const parsed = clickSaving("/api/items");
const { page, reads } = uiPage([
{
method: "POST",
url: `${ORIGIN}/api/items`,
status: 201,
body: '{"id":"x"}',
encoding: "gzip",
size: 12_000,
},
]);
await expect(
perform(page, parsed.actions[0]!, resolved(parsed)),
).rejects.toThrow("could decode past the size limit");
expect(reads.count).toBe(0);
});
it.each([
{ encoding: "gzip", size: 12_000 },
{ size: 0, length: "00012000001" },
{ encoding: "gzip", size: 0, length: "00012000" },
])(
"refuses a response that could decode past the limit without reading it: %j",
async (bounds) => {
const parsed = clickSaving("/api/items");
const { page, reads } = uiPage([
{
method: "POST",
url: `${ORIGIN}/api/items`,
status: 201,
body: '{"id":"x"}',
...bounds,
},
]);
await expect(
perform(page, parsed.actions[0]!, resolved(parsed)),
).rejects.toThrow("could decode past the size limit");
expect(reads.count).toBe(0);
},
);
});
describe("saved response limits and deadlines", () => {
+36 -14
View File
@@ -191,9 +191,13 @@ it("retains a fetch host that ignores an item deadline until it settles", async
});
const pending = fetchWebSourceMaps(
[request],
{ ...input, allowed_origins: [new URL(request.fetchUrl).origin] },
{
...input,
allowed_origins: [new URL(request.fetchUrl).origin],
source_map_fetch_timeout_ms: 50,
},
undefined,
{ fetch: async () => fetching, timeoutMs: 50 },
{ fetch: async () => fetching },
);
const result = await pending;
if (result.ok) throw new Error("Expected pending-fetch cleanup");
@@ -365,9 +369,10 @@ it.each([
{
...input,
allowed_origins: [new URL(request.fetchUrl).origin],
source_map_limits: { fetch_bytes: 1 },
},
controller.signal,
{ fetch: async () => new Response(body), maxResponseBytes: 1 },
{ fetch: async () => new Response(body) },
);
try {
await reading;
@@ -534,7 +539,11 @@ it.each([null, false])(
try {
const pending = fetchWebSourceMaps(
[{ ...request, declaredUrl: url, fetchUrl: url }],
{ ...input, allowed_origins: [new URL(url).origin] },
{
...input,
allowed_origins: [new URL(url).origin],
source_map_fetch_timeout_ms: 10_000,
},
controller.signal,
{
fetch: async (...args) => {
@@ -542,7 +551,6 @@ it.each([null, false])(
delivered();
return response;
},
timeoutMs: 10_000,
},
);
await responseReceived;
@@ -604,9 +612,13 @@ it("keeps an actual stalled HTTP header deadline local to its map", async () =>
try {
const result = await fetchWebSourceMaps(
[{ ...request, declaredUrl: url, fetchUrl: url }],
{ ...input, allowed_origins: [new URL(url).origin] },
{
...input,
allowed_origins: [new URL(url).origin],
source_map_fetch_timeout_ms: 1000,
},
undefined,
{ fetch, timeoutMs: 1000 },
{ fetch },
);
expect(result).toMatchObject({
ok: true,
@@ -641,9 +653,13 @@ it("keeps an actual stalled HTTP response deadline local to its map", async () =
try {
const result = await fetchWebSourceMaps(
[{ ...request, declaredUrl: url, fetchUrl: url }],
{ ...input, allowed_origins: [new URL(url).origin] },
{
...input,
allowed_origins: [new URL(url).origin],
source_map_fetch_timeout_ms: 1000,
},
undefined,
{ fetch, timeoutMs: 1000 },
{ fetch },
);
expect(result).toMatchObject({
ok: true,
@@ -674,7 +690,11 @@ it.each([302, 503])(
}).tee();
const pending = fetchWebSourceMaps(
[request],
{ ...input, allowed_origins: [new URL(request.fetchUrl).origin] },
{
...input,
allowed_origins: [new URL(request.fetchUrl).origin],
source_map_fetch_timeout_ms: 100,
},
undefined,
{
fetch: async () =>
@@ -682,7 +702,6 @@ it.each([302, 503])(
status,
...(status === 302 ? { headers: { location: "/next" } } : {}),
}),
timeoutMs: 100,
},
);
let timer: ReturnType<typeof setTimeout> | undefined;
@@ -719,7 +738,11 @@ it("retains an admitted response when the deadline expires before body reading",
}).tee();
const pending = fetchWebSourceMaps(
[request],
{ ...input, allowed_origins: [new URL(request.fetchUrl).origin] },
{
...input,
allowed_origins: [new URL(request.fetchUrl).origin],
source_map_fetch_timeout_ms: 50,
},
undefined,
{
fetch: async () => {
@@ -727,7 +750,6 @@ it("retains an admitted response when the deadline expires before body reading",
Atomics.wait(new Int32Array(new SharedArrayBuffer(4)), 0, 0, 150);
return new Response(body);
},
timeoutMs: 50,
},
);
try {
@@ -757,6 +779,7 @@ it("retains an oversized redirect identity's pending response cleanup", async ()
...input,
allowed_origins: [new URL(request.fetchUrl).origin],
source_map_limits: { output_bytes: 4000 },
source_map_fetch_timeout_ms: 100,
},
undefined,
{
@@ -764,7 +787,6 @@ it("retains an oversized redirect identity's pending response cleanup", async ()
String(url) === request.fetchUrl
? new Response(null, { status: 302, headers: { location: finalUrl } })
: new Response(body),
timeoutMs: 100,
},
);
try {
Loaded 100 of 409 files, more files were not shown because too many files have changed in this diff. Show more