mirror of
https://github.com/morluto/rea.git
synced 2026-10-12 02:19:24 +00:00
fix(javascript): isolate source analysis and retain bounded partial evidence (#1651)
* refactor(javascript): validate transferable file projections * test(javascript): consume decoded module and static facts * fix(javascript): isolate source analysis and retain bounded partial evidence * test(javascript): verify cleanup retention and repeated source paths
This commit is contained in:
@@ -9,6 +9,22 @@ that static analysis. Read findings, coverage, limitations and unknowns, then
|
|||||||
follow a specific unanswered question. Older servers without `detail` return
|
follow a specific unanswered question. Older servers without `detail` return
|
||||||
the complete application Evidence inline when it fits.
|
the complete application Evidence inline when it fits.
|
||||||
|
|
||||||
|
When the connected schema advertises `max_heap_mb` and `analysis_timeout_ms`,
|
||||||
|
these select the isolated source analyzer's old-space heap and per-source
|
||||||
|
deadline. Defaults are 1024 MiB and 300000 ms. They do not change the MCP
|
||||||
|
process heap. Resource-limited analyses preserve completed static facts and
|
||||||
|
report partial semantic coverage with specific unknown scopes; this does not
|
||||||
|
mean the source is malformed. Inspect those limits before retrying a source.
|
||||||
|
When expanded relationships exceed the parent's available memory, inspect the
|
||||||
|
file's modules item view for retained `static_analysis` and
|
||||||
|
`semantic_module_analysis`. Their facts remain available even when graph links
|
||||||
|
are unknown. Cumulative result exhaustion can stop analysis before every file
|
||||||
|
is examined; the inventory and remaining unknown scopes distinguish that case.
|
||||||
|
Timeout or cancellation after source analysis begins retains partial Evidence
|
||||||
|
in the same session. A returned `error.details.partial_observation` reference
|
||||||
|
can be supplied directly to `inspect_analysis_view`. If the SDK cancels locally, retrieve its recorded ID from
|
||||||
|
the Evidence bundle before inspecting the partial analysis.
|
||||||
|
|
||||||
## Read a module without repeating analysis
|
## Read a module without repeating analysis
|
||||||
|
|
||||||
Use the actual returned parent ID, on the same connection, as `source.evidence_id`.
|
Use the actual returned parent ID, on the same connection, as `source.evidence_id`.
|
||||||
|
|||||||
@@ -89,9 +89,9 @@ queries use per-file range indexes, while call sites use exact-range lookups.
|
|||||||
Existing node budgets and coverage reporting remain in effect.
|
Existing node budgets and coverage reporting remain in effect.
|
||||||
|
|
||||||
Progress identifies the source file being parsed and projected. Cancellation is
|
Progress identifies the source file being parsed and projected. Cancellation is
|
||||||
checked between files, with event-loop yields that allow notifications and
|
checked between files and throughout result transfer. The owned analyzer can
|
||||||
completed file-local allocations to be released. A single file's synchronous
|
be stopped while its synchronous parser or semantic extractor is active;
|
||||||
parser and semantic extraction still need to finish before that boundary.
|
the CLI/MCP process remains available for control requests.
|
||||||
|
|
||||||
Graph and Evidence identifiers hash canonical JSON incrementally, without
|
Graph and Evidence identifiers hash canonical JSON incrementally, without
|
||||||
assembling a single string for the whole graph. The canonical bytes and existing
|
assembling a single string for the whole graph. The canonical bytes and existing
|
||||||
@@ -113,6 +113,46 @@ smaller view, for example `--format json --filter-output
|
|||||||
evidence_id,normalized_result.statistics`. Streaming output does not bound the
|
evidence_id,normalized_result.statistics`. Streaming output does not bound the
|
||||||
memory needed to construct the analysis graph itself.
|
memory needed to construct the analysis graph itself.
|
||||||
|
|
||||||
|
JavaScript source parsing, static analysis and semantic projection run in one
|
||||||
|
owned analysis process, reused sequentially across files. Target code is never
|
||||||
|
executed. The analyzer defaults to 1024 MiB of V8 old-space and a 300000 ms
|
||||||
|
deadline per source, including startup and fact transfer. Select
|
||||||
|
`max_heap_mb` and `analysis_timeout_ms` through MCP, or `--max-heap-mb` and
|
||||||
|
`--analysis-timeout-ms` on the dedicated CLI command. These choices control the
|
||||||
|
analyzer; the caller's `NODE_OPTIONS` still controls the CLI/MCP process heap.
|
||||||
|
V8 old-space limits do not impose a process RSS limit.
|
||||||
|
|
||||||
|
A source that exhausts the analyzer heap does not terminate the CLI/MCP
|
||||||
|
process. Completed static facts are retained, other files can still be analyzed,
|
||||||
|
and unexamined semantic relationships remain explicit unknowns. The parent
|
||||||
|
also checks transfer expansion and accumulated results against its remaining
|
||||||
|
heap before accepting semantic facts. A resource-limited result has partial
|
||||||
|
coverage, the observed limits and an `analyze-javascript-source` unknown scope
|
||||||
|
with the specific failure, including captured diagnostics when available.
|
||||||
|
Resource failures do not increment syntax parse failures. There is no fixed
|
||||||
|
source-length cutoff.
|
||||||
|
|
||||||
|
Static observations also need memory when expanded into nodes, relationships
|
||||||
|
and Evidence. If that expansion does not fit, the file node retains
|
||||||
|
`static_analysis` and any admitted `semantic_module_analysis` observations,
|
||||||
|
including their exact values and source locations. Use the public modules item
|
||||||
|
view to read them. The graph marks those unexpanded relationships as unknown.
|
||||||
|
If cumulative result capacity is exhausted, analysis stops; inventoried but
|
||||||
|
unexamined files remain present with explicit unknown scopes.
|
||||||
|
|
||||||
|
A timeout stops the remaining analysis and returns typed failure details with
|
||||||
|
completed partial Evidence. MCP failure details return
|
||||||
|
`partial_observation: { kind: "retained-evidence", evidence_id: "..." }`
|
||||||
|
after recording succeeds, so `inspect_analysis_view` can open that analysis
|
||||||
|
directly without serializing the entire partial graph into the error response.
|
||||||
|
CLI failure output retains the complete partial Evidence.
|
||||||
|
Cancellation after source analysis begins likewise
|
||||||
|
retains completed facts; an SDK client may reject its cancelled request before
|
||||||
|
receiving that reply. On the same MCP connection, inspect the Evidence bundle
|
||||||
|
and use the retained analysis ID with `inspect_analysis_view`. Owned analyzer
|
||||||
|
cleanup is verified before its private files are released. Cleanup uncertainty
|
||||||
|
is reported separately and retains ownership for retry.
|
||||||
|
|
||||||
CLI workflows parse larger JSON input files incrementally from one verified
|
CLI workflows parse larger JSON input files incrementally from one verified
|
||||||
regular-file handle, without constructing a document-sized string. Smaller
|
regular-file handle, without constructing a document-sized string. Smaller
|
||||||
files retain native JSON parsing. Both paths require strict JSON and valid
|
files retain native JSON parsing. Both paths require strict JSON and valid
|
||||||
|
|||||||
@@ -3,6 +3,7 @@
|
|||||||
"entry": [
|
"entry": [
|
||||||
"src/main.ts",
|
"src/main.ts",
|
||||||
"src/cli.ts",
|
"src/cli.ts",
|
||||||
|
"src/composition/javascriptAnalysisWorkerProcess.ts",
|
||||||
"src/contracts/errorSchemas.ts",
|
"src/contracts/errorSchemas.ts",
|
||||||
"src/evaluation/AgentEvaluationFixtures.ts",
|
"src/evaluation/AgentEvaluationFixtures.ts",
|
||||||
"src/application/RealHopperAssertions.ts",
|
"src/application/RealHopperAssertions.ts",
|
||||||
|
|||||||
@@ -30,6 +30,9 @@
|
|||||||
"access": "public"
|
"access": "public"
|
||||||
},
|
},
|
||||||
"type": "module",
|
"type": "module",
|
||||||
|
"imports": {
|
||||||
|
"#javascript-analysis-worker": "./dist/composition/javascriptAnalysisWorkerProcess.js"
|
||||||
|
},
|
||||||
"bin": {
|
"bin": {
|
||||||
"rea": "scripts/rea.mjs",
|
"rea": "scripts/rea.mjs",
|
||||||
"rea-agents": "scripts/rea.mjs"
|
"rea-agents": "scripts/rea.mjs"
|
||||||
|
|||||||
@@ -6,14 +6,9 @@ import {
|
|||||||
analyzeJavaScriptApplicationInputSchema,
|
analyzeJavaScriptApplicationInputSchema,
|
||||||
parseOwnedJavaScriptApplicationAnalysisSteps,
|
parseOwnedJavaScriptApplicationAnalysisSteps,
|
||||||
} from "../../domain/javascript/javascriptApplicationAnalysis.js";
|
} from "../../domain/javascript/javascriptApplicationAnalysis.js";
|
||||||
import {
|
import { AnalysisOutputError } from "../../domain/analysisErrorCore.js";
|
||||||
AnalysisAccessDeniedError,
|
|
||||||
AnalysisInputError,
|
|
||||||
AnalysisOutputError,
|
|
||||||
AnalysisUnsupportedTargetError,
|
|
||||||
} from "../../domain/analysisErrorCore.js";
|
|
||||||
import { ArtifactOperationError } from "../../domain/artifactOperationError.js";
|
import { ArtifactOperationError } from "../../domain/artifactOperationError.js";
|
||||||
import { type AnalysisError } from "../../domain/analysisErrorBase.js";
|
import { AnalysisError } from "../../domain/analysisErrorBase.js";
|
||||||
import type { Evidence } from "../../domain/evidence.js";
|
import type { Evidence } from "../../domain/evidence.js";
|
||||||
import { analysisInputErrorFromIssues } from "../../domain/inputIssueProjection.js";
|
import { analysisInputErrorFromIssues } from "../../domain/inputIssueProjection.js";
|
||||||
import { err, ok, type Result } from "../../domain/result.js";
|
import { err, ok, type Result } from "../../domain/result.js";
|
||||||
@@ -64,6 +59,12 @@ export const analyzeJavaScriptApplicationValidated = async (
|
|||||||
input_path: input.input_path,
|
input_path: input.input_path,
|
||||||
format: input.format,
|
format: input.format,
|
||||||
integrity_policy: input.integrity_policy,
|
integrity_policy: input.integrity_policy,
|
||||||
|
...(input.max_heap_mb === undefined
|
||||||
|
? {}
|
||||||
|
: { max_heap_mb: input.max_heap_mb }),
|
||||||
|
...(input.analysis_timeout_ms === undefined
|
||||||
|
? {}
|
||||||
|
: { analysis_timeout_ms: input.analysis_timeout_ms }),
|
||||||
},
|
},
|
||||||
scope,
|
scope,
|
||||||
options.signal,
|
options.signal,
|
||||||
@@ -110,13 +111,8 @@ export const analyzeJavaScriptApplicationValidated = async (
|
|||||||
assertJavaScriptAnalysisActive(options.signal);
|
assertJavaScriptAnalysisActive(options.signal);
|
||||||
return ok(evidence);
|
return ok(evidence);
|
||||||
} catch (cause: unknown) {
|
} catch (cause: unknown) {
|
||||||
// Input selection failures are already typed by their cause.
|
// Preserve execution constraints and their completed partial Evidence.
|
||||||
if (
|
if (cause instanceof AnalysisError) return err(cause);
|
||||||
cause instanceof AnalysisInputError ||
|
|
||||||
cause instanceof AnalysisUnsupportedTargetError ||
|
|
||||||
cause instanceof AnalysisAccessDeniedError
|
|
||||||
)
|
|
||||||
return err(cause);
|
|
||||||
if (cause instanceof ArtifactReaderFailure)
|
if (cause instanceof ArtifactReaderFailure)
|
||||||
return err(artifactFailureToOperationError(cause));
|
return err(artifactFailureToOperationError(cause));
|
||||||
if (cause instanceof z.ZodError)
|
if (cause instanceof z.ZodError)
|
||||||
|
|||||||
@@ -8,6 +8,7 @@ import {
|
|||||||
classifyParsedJavaScriptOpenReceiversSteps,
|
classifyParsedJavaScriptOpenReceiversSteps,
|
||||||
} from "../../domain/javascript/javascriptSemanticAnalysis.js";
|
} from "../../domain/javascript/javascriptSemanticAnalysis.js";
|
||||||
import type { JavaScriptSemanticIr } from "../../domain/javascript/javascriptSemanticIr.js";
|
import type { JavaScriptSemanticIr } from "../../domain/javascript/javascriptSemanticIr.js";
|
||||||
|
import { projectJavaScriptModuleSemantics } from "../../domain/javascript/javascriptModuleSemanticIr.js";
|
||||||
import { parseJavaScriptSource } from "../../domain/javascript/javascriptSourceParser.js";
|
import { parseJavaScriptSource } from "../../domain/javascript/javascriptSourceParser.js";
|
||||||
import { hasValidSourceMapContents } from "../../domain/sourceMapContents.js";
|
import { hasValidSourceMapContents } from "../../domain/sourceMapContents.js";
|
||||||
import { flattenSourceMapLeaves } from "../../domain/sourceMapEnvelope.js";
|
import { flattenSourceMapLeaves } from "../../domain/sourceMapEnvelope.js";
|
||||||
@@ -35,6 +36,15 @@ import type {
|
|||||||
import { analyzeJavaScriptJsonModule } from "./JavaScriptJsonModules.js";
|
import { analyzeJavaScriptJsonModule } from "./JavaScriptJsonModules.js";
|
||||||
import { completeJavaScriptAnalysisSteps } from "./JavaScriptAnalysisControl.js";
|
import { completeJavaScriptAnalysisSteps } from "./JavaScriptAnalysisControl.js";
|
||||||
import { htmlArtifactReferences } from "../../domain/javascript/htmlArtifactReferences.js";
|
import { htmlArtifactReferences } from "../../domain/javascript/htmlArtifactReferences.js";
|
||||||
|
import type { JavaScriptSourceAnalysisPort } from "../../domain/javascript/javascriptSourceAnalysis.js";
|
||||||
|
import type { JavaScriptSemanticGraphProjection } from "./JavaScriptSemanticGraphBuilder.js";
|
||||||
|
import { AnalysisError } from "../../domain/analysisErrorBase.js";
|
||||||
|
import { projectAnalysisError } from "../../domain/analysisErrorProjection.js";
|
||||||
|
import {
|
||||||
|
AnalysisCancelledError,
|
||||||
|
AnalysisResourceConstraintError,
|
||||||
|
AnalysisTimeoutError,
|
||||||
|
} from "../../domain/analysisErrorCore.js";
|
||||||
|
|
||||||
interface MutableArtifactAnalysis<
|
interface MutableArtifactAnalysis<
|
||||||
SemanticIr extends JavaScriptModuleSemanticIr,
|
SemanticIr extends JavaScriptModuleSemanticIr,
|
||||||
@@ -95,27 +105,7 @@ export const analyzeAndProjectJavaScriptArtifactFiles = async (
|
|||||||
): Generator<void, JavaScriptModuleSemanticIr> {
|
): Generator<void, JavaScriptModuleSemanticIr> {
|
||||||
const projection = projectSemantics(file, ir);
|
const projection = projectSemantics(file, ir);
|
||||||
if (projection !== undefined) while (projection.next().done !== true) yield;
|
if (projection !== undefined) while (projection.next().done !== true) yield;
|
||||||
const programScopes = ir.scopes.filter(({ kind }) => kind === "program");
|
return projectJavaScriptModuleSemantics(ir);
|
||||||
const programScopeId = programScopes[0]?.scopeId;
|
|
||||||
const localNames = new Set(
|
|
||||||
ir.moduleLinks.map(({ localName }) => localName),
|
|
||||||
);
|
|
||||||
const callableIds = new Set(
|
|
||||||
ir.moduleLinks.map(({ callableId }) => callableId),
|
|
||||||
);
|
|
||||||
return {
|
|
||||||
scopes: programScopes,
|
|
||||||
bindings: ir.bindings.filter(
|
|
||||||
({ name, scopeId }) =>
|
|
||||||
scopeId === programScopeId && localNames.has(name),
|
|
||||||
),
|
|
||||||
callables: ir.callables.filter(({ callableId }) =>
|
|
||||||
callableIds.has(callableId),
|
|
||||||
),
|
|
||||||
moduleLinks: ir.moduleLinks,
|
|
||||||
coverage: ir.coverage,
|
|
||||||
limitations: ir.limitations,
|
|
||||||
};
|
|
||||||
};
|
};
|
||||||
const context = { state, projectSemantics: projectFileSemantics };
|
const context = { state, projectSemantics: projectFileSemantics };
|
||||||
for (const [index, file] of fileSet.files.entries()) {
|
for (const [index, file] of fileSet.files.entries()) {
|
||||||
@@ -130,6 +120,174 @@ export const analyzeAndProjectJavaScriptArtifactFiles = async (
|
|||||||
return finalizeArtifactAnalysis(state);
|
return finalizeArtifactAnalysis(state);
|
||||||
};
|
};
|
||||||
|
|
||||||
|
/** Analyze sources behind a hard heap boundary, retaining collected facts on interruption. */
|
||||||
|
export const analyzeAndAdoptJavaScriptArtifactFiles = async (
|
||||||
|
fileSet: JavaScriptArtifactFileSet,
|
||||||
|
worker: Pick<JavaScriptSourceAnalysisPort, "analyze">,
|
||||||
|
projection: JavaScriptSemanticGraphProjection,
|
||||||
|
beforeFile: (
|
||||||
|
file: JavaScriptArtifactFile,
|
||||||
|
completed: number,
|
||||||
|
total: number,
|
||||||
|
) => Promise<void>,
|
||||||
|
signal?: AbortSignal,
|
||||||
|
): Promise<{
|
||||||
|
readonly analysis: JavaScriptModuleArtifactAnalysis;
|
||||||
|
readonly interruption: AnalysisError | null;
|
||||||
|
}> => {
|
||||||
|
const state = emptyArtifactAnalysis<JavaScriptModuleSemanticIr>();
|
||||||
|
let interruption: AnalysisError | null = null;
|
||||||
|
for (const [index, file] of fileSet.files.entries()) {
|
||||||
|
try {
|
||||||
|
await setImmediate();
|
||||||
|
await beforeFile(file, index, fileSet.files.length);
|
||||||
|
if (signal?.aborted === true)
|
||||||
|
throw new AnalysisCancelledError("analyze_javascript_application");
|
||||||
|
if (file.kind !== "javascript" || !file.text.included) {
|
||||||
|
const steps = analyzeArtifactFileSteps(file, {
|
||||||
|
state,
|
||||||
|
projectSemantics: function* (_file, ir) {
|
||||||
|
yield;
|
||||||
|
return projectJavaScriptModuleSemantics(ir);
|
||||||
|
},
|
||||||
|
});
|
||||||
|
await completeJavaScriptAnalysisSteps(steps, signal);
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
const result = await worker.analyze(
|
||||||
|
file,
|
||||||
|
projection.remainingFileNodeBudget(),
|
||||||
|
signal,
|
||||||
|
);
|
||||||
|
const facts = result.ok ? result.value : result.error;
|
||||||
|
const applicationProjectionFailure = facts.applicationProjectionFailure;
|
||||||
|
if (facts.projection !== null)
|
||||||
|
projection.adoptFileProjection(facts.projection);
|
||||||
|
const error = result.ok ? null : result.error.error;
|
||||||
|
const limits =
|
||||||
|
error instanceof AnalysisResourceConstraintError
|
||||||
|
? error.reportedLimits
|
||||||
|
: error instanceof AnalysisTimeoutError
|
||||||
|
? { worker_timeout_ms: error.timeoutMs }
|
||||||
|
: null;
|
||||||
|
const failure =
|
||||||
|
error === null
|
||||||
|
? undefined
|
||||||
|
: {
|
||||||
|
reason: `${file.path}: ${error.message}`,
|
||||||
|
error: projectAnalysisError(error),
|
||||||
|
limits,
|
||||||
|
};
|
||||||
|
state.files.push({
|
||||||
|
file,
|
||||||
|
javascript: facts.javascript,
|
||||||
|
semantic: facts.module === null ? null : { ir: facts.module },
|
||||||
|
...(failure === undefined ? {} : { analysis_failure: failure }),
|
||||||
|
...(applicationProjectionFailure === undefined
|
||||||
|
? {}
|
||||||
|
: {
|
||||||
|
application_projection_failure: {
|
||||||
|
reason: applicationProjectionFailure.message,
|
||||||
|
error: projectAnalysisError(applicationProjectionFailure),
|
||||||
|
limits: applicationProjectionFailure.reportedLimits,
|
||||||
|
},
|
||||||
|
}),
|
||||||
|
});
|
||||||
|
if (failure !== undefined && facts.projection === null)
|
||||||
|
projection.recordUnavailableFile(
|
||||||
|
file,
|
||||||
|
failure.reason,
|
||||||
|
failure.limits,
|
||||||
|
error instanceof AnalysisResourceConstraintError ||
|
||||||
|
error instanceof AnalysisTimeoutError
|
||||||
|
? "resource-limit"
|
||||||
|
: "incomplete-module",
|
||||||
|
);
|
||||||
|
if (facts.javascript !== null) {
|
||||||
|
state.visitedNodes += facts.javascript.visited_ast_nodes;
|
||||||
|
state.findings +=
|
||||||
|
findingCount(facts.javascript) +
|
||||||
|
(facts.module?.moduleLinks.length ?? 0);
|
||||||
|
state.modules += facts.javascript.bundler_registrations.reduce(
|
||||||
|
(count, item) => count + item.modules.length,
|
||||||
|
0,
|
||||||
|
);
|
||||||
|
if (facts.javascript.parse_status === "failed")
|
||||||
|
state.parseFailures += 1;
|
||||||
|
}
|
||||||
|
if (
|
||||||
|
error !== null &&
|
||||||
|
(!(error instanceof AnalysisResourceConstraintError) ||
|
||||||
|
error.reportedLimits?.parent_result_budget_exhausted === true)
|
||||||
|
) {
|
||||||
|
interruption = error;
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
} catch (cause: unknown) {
|
||||||
|
if (signal?.aborted !== true && !(cause instanceof AnalysisError))
|
||||||
|
throw cause;
|
||||||
|
interruption =
|
||||||
|
signal?.aborted === true
|
||||||
|
? new AnalysisCancelledError("analyze_javascript_application")
|
||||||
|
: cause instanceof AnalysisError
|
||||||
|
? cause
|
||||||
|
: null;
|
||||||
|
if (interruption !== null) {
|
||||||
|
state.files.push({
|
||||||
|
file,
|
||||||
|
javascript: null,
|
||||||
|
semantic: null,
|
||||||
|
analysis_failure: {
|
||||||
|
reason: `${file.path}: ${interruption.message}`,
|
||||||
|
error: projectAnalysisError(interruption),
|
||||||
|
limits: null,
|
||||||
|
},
|
||||||
|
});
|
||||||
|
projection.recordUnavailableFile(
|
||||||
|
file,
|
||||||
|
interruption.message,
|
||||||
|
null,
|
||||||
|
"incomplete-module",
|
||||||
|
);
|
||||||
|
}
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if (interruption !== null) {
|
||||||
|
const analyzedPaths = new Set(state.files.map(({ file }) => file.path));
|
||||||
|
const failure = {
|
||||||
|
reason: `Source analysis was interrupted: ${interruption.message}`,
|
||||||
|
error: projectAnalysisError(interruption),
|
||||||
|
limits:
|
||||||
|
interruption instanceof AnalysisResourceConstraintError
|
||||||
|
? interruption.reportedLimits
|
||||||
|
: interruption instanceof AnalysisTimeoutError
|
||||||
|
? { worker_timeout_ms: interruption.timeoutMs }
|
||||||
|
: null,
|
||||||
|
};
|
||||||
|
for (const file of fileSet.files) {
|
||||||
|
if (analyzedPaths.has(file.path)) continue;
|
||||||
|
state.files.push({
|
||||||
|
file,
|
||||||
|
javascript: null,
|
||||||
|
semantic: null,
|
||||||
|
analysis_failure: failure,
|
||||||
|
});
|
||||||
|
if (file.kind === "javascript")
|
||||||
|
projection.recordUnavailableFile(
|
||||||
|
file,
|
||||||
|
failure.reason,
|
||||||
|
failure.limits,
|
||||||
|
interruption instanceof AnalysisResourceConstraintError ||
|
||||||
|
interruption instanceof AnalysisTimeoutError
|
||||||
|
? "resource-limit"
|
||||||
|
: "incomplete-module",
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return { analysis: finalizeArtifactAnalysis(state), interruption };
|
||||||
|
};
|
||||||
|
|
||||||
const analyzeArtifactFiles = <SemanticIr extends JavaScriptModuleSemanticIr>(
|
const analyzeArtifactFiles = <SemanticIr extends JavaScriptModuleSemanticIr>(
|
||||||
fileSet: JavaScriptArtifactFileSet,
|
fileSet: JavaScriptArtifactFileSet,
|
||||||
projectSemantics: ArtifactAnalysisContext<SemanticIr>["projectSemantics"],
|
projectSemantics: ArtifactAnalysisContext<SemanticIr>["projectSemantics"],
|
||||||
@@ -161,6 +319,12 @@ const finalizeArtifactAnalysis = <
|
|||||||
limitations: [
|
limitations: [
|
||||||
"JavaScript and HTML were parsed as inert text; bundle bootstrap code was never executed.",
|
"JavaScript and HTML were parsed as inert text; bundle bootstrap code was never executed.",
|
||||||
"Static paths and relationships may remain unresolved when expressions are dynamic or obfuscated.",
|
"Static paths and relationships may remain unresolved when expressions are dynamic or obfuscated.",
|
||||||
|
...state.files.flatMap(
|
||||||
|
({ analysis_failure, application_projection_failure }) =>
|
||||||
|
[analysis_failure, application_projection_failure].flatMap(
|
||||||
|
(failure) => (failure === undefined ? [] : [failure.reason]),
|
||||||
|
),
|
||||||
|
),
|
||||||
],
|
],
|
||||||
};
|
};
|
||||||
};
|
};
|
||||||
|
|||||||
@@ -4,7 +4,9 @@ import type {
|
|||||||
JavaScriptStaticAnalysis,
|
JavaScriptStaticAnalysis,
|
||||||
} from "../../domain/javascript/javascriptStaticAnalysisTypes.js";
|
} from "../../domain/javascript/javascriptStaticAnalysisTypes.js";
|
||||||
import type { JavaScriptSemanticIr } from "../../domain/javascript/javascriptSemanticIr.js";
|
import type { JavaScriptSemanticIr } from "../../domain/javascript/javascriptSemanticIr.js";
|
||||||
|
import type { JavaScriptModuleSemanticIr } from "../../domain/javascript/javascriptModuleSemanticIr.js";
|
||||||
import type { JavaScriptArtifactFile } from "../../domain/javascript/javascriptArtifactFiles.js";
|
import type { JavaScriptArtifactFile } from "../../domain/javascript/javascriptArtifactFiles.js";
|
||||||
|
import type { JsonValue } from "../../domain/jsonValue.js";
|
||||||
|
|
||||||
interface JavaScriptStructuredObservation {
|
interface JavaScriptStructuredObservation {
|
||||||
readonly path: string;
|
readonly path: string;
|
||||||
@@ -89,15 +91,7 @@ export type JavaScriptJsonModuleObservation = JavaScriptStructuredObservation &
|
|||||||
);
|
);
|
||||||
|
|
||||||
/** Module facts needed after file-local semantic projection has finished. */
|
/** Module facts needed after file-local semantic projection has finished. */
|
||||||
export type JavaScriptModuleSemanticIr = Pick<
|
export type { JavaScriptModuleSemanticIr } from "../../domain/javascript/javascriptModuleSemanticIr.js";
|
||||||
JavaScriptSemanticIr,
|
|
||||||
| "scopes"
|
|
||||||
| "bindings"
|
|
||||||
| "callables"
|
|
||||||
| "moduleLinks"
|
|
||||||
| "coverage"
|
|
||||||
| "limitations"
|
|
||||||
>;
|
|
||||||
|
|
||||||
/** One relevant file plus optional AST-only JavaScript facts. */
|
/** One relevant file plus optional AST-only JavaScript facts. */
|
||||||
export interface AnalyzedJavaScriptArtifactFile<
|
export interface AnalyzedJavaScriptArtifactFile<
|
||||||
@@ -108,6 +102,18 @@ export interface AnalyzedJavaScriptArtifactFile<
|
|||||||
readonly semantic: {
|
readonly semantic: {
|
||||||
readonly ir: SemanticIr;
|
readonly ir: SemanticIr;
|
||||||
} | null;
|
} | null;
|
||||||
|
/** Operational incompleteness, distinct from a malformed source file. */
|
||||||
|
readonly analysis_failure?: {
|
||||||
|
readonly reason: string;
|
||||||
|
readonly error: JsonValue;
|
||||||
|
readonly limits: Readonly<Record<string, JsonValue>> | null;
|
||||||
|
};
|
||||||
|
/** Facts retained inline when their expanded graph exceeds available memory. */
|
||||||
|
readonly application_projection_failure?: {
|
||||||
|
readonly reason: string;
|
||||||
|
readonly error: JsonValue;
|
||||||
|
readonly limits: Readonly<Record<string, JsonValue>> | null;
|
||||||
|
};
|
||||||
}
|
}
|
||||||
|
|
||||||
/** Complete static-analysis projection before graph construction. */
|
/** Complete static-analysis projection before graph construction. */
|
||||||
|
|||||||
@@ -12,7 +12,7 @@ export class JavaScriptArtifactGraphAccumulator {
|
|||||||
readonly #nodes = new Map<string, ApplicationNode>();
|
readonly #nodes = new Map<string, ApplicationNode>();
|
||||||
readonly #edges = new Map<string, ApplicationEdge>();
|
readonly #edges = new Map<string, ApplicationEdge>();
|
||||||
|
|
||||||
/** Create or merge one canonical node. */
|
/** Create or merge one canonical node, retaining a stable live construction reference. */
|
||||||
addNode<Input extends DisplayableNodeInput>(input: Input): ApplicationNode {
|
addNode<Input extends DisplayableNodeInput>(input: Input): ApplicationNode {
|
||||||
const created = createJavaScriptApplicationNode(displayableNode(input));
|
const created = createJavaScriptApplicationNode(displayableNode(input));
|
||||||
const existing = this.#nodes.get(created.node_id);
|
const existing = this.#nodes.get(created.node_id);
|
||||||
@@ -23,19 +23,23 @@ export class JavaScriptArtifactGraphAccumulator {
|
|||||||
const bySemanticId = new Map(
|
const bySemanticId = new Map(
|
||||||
[...existing.observations, ...created.observations].map((observation) => [
|
[...existing.observations, ...created.observations].map((observation) => [
|
||||||
observation.observation_id,
|
observation.observation_id,
|
||||||
observationInput(observation),
|
observation,
|
||||||
]),
|
]),
|
||||||
);
|
);
|
||||||
const observations = [...bySemanticId.entries()]
|
const observations = [...bySemanticId.values()].sort((left, right) =>
|
||||||
.sort(([left], [right]) => (left < right ? -1 : left > right ? 1 : 0))
|
left.observation_id < right.observation_id
|
||||||
.map(([, observation]) => observation);
|
? -1
|
||||||
const merged = createJavaScriptApplicationNode({
|
: left.observation_id > right.observation_id
|
||||||
kind: existing.kind,
|
? 1
|
||||||
identity: existing.identity,
|
: 0,
|
||||||
observations,
|
);
|
||||||
});
|
// Both factories already validated and identified these observations.
|
||||||
this.#nodes.set(merged.node_id, merged);
|
// Replacing the node retained every previous observation array through
|
||||||
return merged;
|
// path/chunk/IPC indexes when multiple paths shared an entity identity.
|
||||||
|
existing.observations.length = 0;
|
||||||
|
for (const observation of observations)
|
||||||
|
existing.observations.push(observation);
|
||||||
|
return existing;
|
||||||
}
|
}
|
||||||
|
|
||||||
/** Create or replace one canonical edge by semantic identifier. */
|
/** Create or replace one canonical edge by semantic identifier. */
|
||||||
@@ -56,17 +60,6 @@ export class JavaScriptArtifactGraphAccumulator {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
const observationInput = (
|
|
||||||
observation: ApplicationNode["observations"][number],
|
|
||||||
) => {
|
|
||||||
const {
|
|
||||||
observation_id: _observationId,
|
|
||||||
identifier_strategy: _strategy,
|
|
||||||
...input
|
|
||||||
} = observation;
|
|
||||||
return input;
|
|
||||||
};
|
|
||||||
|
|
||||||
/** Node fields whose display form is normalized before validation. */
|
/** Node fields whose display form is normalized before validation. */
|
||||||
interface DisplayableNodeInput {
|
interface DisplayableNodeInput {
|
||||||
readonly observations: readonly { readonly label: string | null }[];
|
readonly observations: readonly { readonly label: string | null }[];
|
||||||
|
|||||||
@@ -107,19 +107,33 @@ function* buildJavaScriptArtifactGraphInputSteps(
|
|||||||
yield;
|
yield;
|
||||||
addJavaScriptArtifactFiles(context);
|
addJavaScriptArtifactFiles(context);
|
||||||
yield;
|
yield;
|
||||||
const packageRoots = addJavaScriptPackageNodes(context);
|
// Raw observations remain on the file nodes. Only admitted expansions feed
|
||||||
|
// the relationship builders, including their Electron and bundler callers.
|
||||||
|
const projectionContext = {
|
||||||
|
...context,
|
||||||
|
analysis: {
|
||||||
|
...analysis,
|
||||||
|
files: analysis.files.map((file) =>
|
||||||
|
file.application_projection_failure === undefined
|
||||||
|
? file
|
||||||
|
: { ...file, javascript: null, semantic: null },
|
||||||
|
),
|
||||||
|
},
|
||||||
|
};
|
||||||
|
const packageRoots = addJavaScriptPackageNodes(projectionContext);
|
||||||
yield;
|
yield;
|
||||||
addJavaScriptSourceModules(context);
|
addJavaScriptSourceModules(projectionContext);
|
||||||
yield;
|
yield;
|
||||||
const bundlerLimitations = addJavaScriptBundlerNodes(context);
|
const bundlerLimitations = addJavaScriptBundlerNodes(projectionContext);
|
||||||
const relationshipOmissions =
|
const relationshipOmissions =
|
||||||
yield* addJavaScriptModuleRelationshipsSteps(context);
|
yield* addJavaScriptModuleRelationshipsSteps(projectionContext);
|
||||||
const findingLimitations = yield* addJavaScriptStaticFindingsSteps(context);
|
const findingLimitations =
|
||||||
addElectronBoundaries(context);
|
yield* addJavaScriptStaticFindingsSteps(projectionContext);
|
||||||
|
addElectronBoundaries(projectionContext);
|
||||||
yield;
|
yield;
|
||||||
addJavaScriptHtmlRoles(context);
|
addJavaScriptHtmlRoles(projectionContext);
|
||||||
yield;
|
yield;
|
||||||
addJavaScriptSourceMapOriginals(context);
|
addJavaScriptSourceMapOriginals(projectionContext);
|
||||||
yield;
|
yield;
|
||||||
const coverage = graphCoverage(context);
|
const coverage = graphCoverage(context);
|
||||||
return {
|
return {
|
||||||
@@ -153,14 +167,34 @@ const graphCoverage = (context: JavaScriptArtifactGraphContext) => {
|
|||||||
javascript !== null && javascript.parse_status === "partial",
|
javascript !== null && javascript.parse_status === "partial",
|
||||||
);
|
);
|
||||||
const unknownGap =
|
const unknownGap =
|
||||||
|
context.analysis.files.some(
|
||||||
|
({ analysis_failure, application_projection_failure }) =>
|
||||||
|
analysis_failure !== undefined ||
|
||||||
|
application_projection_failure !== undefined,
|
||||||
|
) ||
|
||||||
context.analysis.parse_failures > 0 ||
|
context.analysis.parse_failures > 0 ||
|
||||||
context.fileSet.invalid_utf8_files > 0 ||
|
context.fileSet.invalid_utf8_files > 0 ||
|
||||||
sourceMapPolicyGap ||
|
sourceMapPolicyGap ||
|
||||||
malformedStructuredData ||
|
malformedStructuredData ||
|
||||||
partialJavaScript;
|
partialJavaScript;
|
||||||
if (resourceLimits.length > 0)
|
const executionLimits = context.analysis.files.flatMap(
|
||||||
|
({ analysis_failure, application_projection_failure }) =>
|
||||||
|
[analysis_failure, application_projection_failure].flatMap((failure) =>
|
||||||
|
failure === undefined
|
||||||
|
? []
|
||||||
|
: javaScriptAnalysisCoverageLimits(failure.limits),
|
||||||
|
),
|
||||||
|
);
|
||||||
|
if (resourceLimits.length > 0 || executionLimits.length > 0)
|
||||||
return partialApplicationCoverage(
|
return partialApplicationCoverage(
|
||||||
semanticResourceLimitCoverage(resourceLimits),
|
[
|
||||||
|
...new Map(
|
||||||
|
[
|
||||||
|
...semanticResourceLimitCoverage(resourceLimits),
|
||||||
|
...executionLimits,
|
||||||
|
].map((limit) => [JSON.stringify(limit), limit]),
|
||||||
|
).values(),
|
||||||
|
],
|
||||||
null,
|
null,
|
||||||
);
|
);
|
||||||
if (unknownGap) return partialApplicationCoverage([], null);
|
if (unknownGap) return partialApplicationCoverage([], null);
|
||||||
@@ -260,3 +294,4 @@ const graphLimitations = (
|
|||||||
]),
|
]),
|
||||||
];
|
];
|
||||||
};
|
};
|
||||||
|
import { javaScriptAnalysisCoverageLimits } from "../../domain/javascript/javascriptAnalysisResourceControls.js";
|
||||||
|
|||||||
@@ -1,4 +1,5 @@
|
|||||||
import type { ArtifactInventorySnapshot } from "../../domain/artifactInventorySnapshot.js";
|
import type { ArtifactInventorySnapshot } from "../../domain/artifactInventorySnapshot.js";
|
||||||
|
import type { JsonValue } from "../../domain/jsonValue.js";
|
||||||
import { compositeKey } from "../../domain/unicodeCodePointOrder.js";
|
import { compositeKey } from "../../domain/unicodeCodePointOrder.js";
|
||||||
import {
|
import {
|
||||||
completeApplicationCoverage,
|
completeApplicationCoverage,
|
||||||
@@ -99,6 +100,8 @@ export interface UnavailableParseInput {
|
|||||||
readonly asset: ApplicationNode;
|
readonly asset: ApplicationNode;
|
||||||
readonly operation: string;
|
readonly operation: string;
|
||||||
readonly limitation: string;
|
readonly limitation: string;
|
||||||
|
readonly failure?: JsonValue;
|
||||||
|
readonly limits?: JavaScriptArtifactGraphCoverage["limits"];
|
||||||
}
|
}
|
||||||
|
|
||||||
/** Add a direct artifact-containment edge. */
|
/** Add a direct artifact-containment edge. */
|
||||||
@@ -182,12 +185,15 @@ export const addUnavailableStaticParseScope = (
|
|||||||
path: input.file.path,
|
path: input.file.path,
|
||||||
operation: input.operation,
|
operation: input.operation,
|
||||||
limitation: input.limitation,
|
limitation: input.limitation,
|
||||||
|
...(input.failure === undefined
|
||||||
|
? {}
|
||||||
|
: { analysis_failure: input.failure }),
|
||||||
},
|
},
|
||||||
evidence: unavailableAstEvidence({
|
evidence: unavailableAstEvidence({
|
||||||
sha256: input.file.sha256,
|
sha256: input.file.sha256,
|
||||||
path: input.file.path,
|
path: input.file.path,
|
||||||
operation: input.operation,
|
operation: input.operation,
|
||||||
coverage: partialApplicationCoverage([], null),
|
coverage: partialApplicationCoverage(input.limits ?? [], null),
|
||||||
limitation: input.limitation,
|
limitation: input.limitation,
|
||||||
}),
|
}),
|
||||||
},
|
},
|
||||||
|
|||||||
@@ -2,6 +2,9 @@ import type { ArtifactInventorySnapshot } from "../../domain/artifactInventorySn
|
|||||||
import type { ApplicationNode } from "../../domain/javascript/javascriptApplicationGraphSchemas.js";
|
import type { ApplicationNode } from "../../domain/javascript/javascriptApplicationGraphSchemas.js";
|
||||||
import { completeApplicationCoverage } from "../../domain/javascript/javascriptApplicationEvidenceSchemas.js";
|
import { completeApplicationCoverage } from "../../domain/javascript/javascriptApplicationEvidenceSchemas.js";
|
||||||
import type { JavaScriptModuleArtifactAnalysis } from "./JavaScriptArtifactAnalysisTypes.js";
|
import type { JavaScriptModuleArtifactAnalysis } from "./JavaScriptArtifactAnalysisTypes.js";
|
||||||
|
import { jsonValueSchema } from "../../domain/jsonValue.js";
|
||||||
|
import { freezeJsonSnapshot } from "../../domain/immutableJson.js";
|
||||||
|
import type { JavaScriptSourceRange } from "../../domain/javascript/javascriptStaticAnalysisTypes.js";
|
||||||
import type {
|
import type {
|
||||||
JavaScriptArtifactContainer,
|
JavaScriptArtifactContainer,
|
||||||
JavaScriptArtifactFile,
|
JavaScriptArtifactFile,
|
||||||
@@ -12,12 +15,14 @@ import {
|
|||||||
addUnavailableStaticParseScope,
|
addUnavailableStaticParseScope,
|
||||||
artifactFileNodeKind,
|
artifactFileNodeKind,
|
||||||
artifactLocalIdentity,
|
artifactLocalIdentity,
|
||||||
|
javascriptAnalysisCoverage,
|
||||||
createElectronRoleNode,
|
createElectronRoleNode,
|
||||||
linkElectronRoleToAsset,
|
linkElectronRoleToAsset,
|
||||||
type JavaScriptArtifactGraphContext,
|
type JavaScriptArtifactGraphContext,
|
||||||
} from "./JavaScriptArtifactGraphContext.js";
|
} from "./JavaScriptArtifactGraphContext.js";
|
||||||
import {
|
import {
|
||||||
artifactObservationEvidence,
|
artifactObservationEvidence,
|
||||||
|
astObservationEvidence,
|
||||||
staticInferenceEvidence,
|
staticInferenceEvidence,
|
||||||
} from "./JavaScriptArtifactGraphEvidence.js";
|
} from "./JavaScriptArtifactGraphEvidence.js";
|
||||||
import { resolveArtifactPathByContext } from "./JavaScriptArtifactPathResolution.js";
|
import { resolveArtifactPathByContext } from "./JavaScriptArtifactPathResolution.js";
|
||||||
@@ -127,6 +132,40 @@ export const addJavaScriptArtifactFiles = (
|
|||||||
analyzed.javascript,
|
analyzed.javascript,
|
||||||
jsonValue,
|
jsonValue,
|
||||||
);
|
);
|
||||||
|
if (
|
||||||
|
analyzed.application_projection_failure !== undefined &&
|
||||||
|
analyzed.javascript !== null
|
||||||
|
)
|
||||||
|
context.accumulator.addNode({
|
||||||
|
kind: target.kind,
|
||||||
|
identity: target.identity,
|
||||||
|
observations: [
|
||||||
|
{
|
||||||
|
label: file.path,
|
||||||
|
properties: {
|
||||||
|
static_analysis: freezeJsonSnapshot(
|
||||||
|
jsonValueSchema.parse(analyzed.javascript),
|
||||||
|
),
|
||||||
|
semantic_module_analysis:
|
||||||
|
analyzed.semantic === null
|
||||||
|
? null
|
||||||
|
: freezeJsonSnapshot(
|
||||||
|
jsonValueSchema.parse(analyzed.semantic.ir),
|
||||||
|
),
|
||||||
|
application_projection_failure:
|
||||||
|
analyzed.application_projection_failure.error,
|
||||||
|
},
|
||||||
|
evidence: astObservationEvidence({
|
||||||
|
sha256: file.sha256,
|
||||||
|
path: file.path,
|
||||||
|
operation: "retain-unexpanded-static-findings",
|
||||||
|
range: sourceFileRange(file),
|
||||||
|
coverage: javascriptAnalysisCoverage(analyzed.javascript),
|
||||||
|
limitations: [analyzed.application_projection_failure.reason],
|
||||||
|
}),
|
||||||
|
},
|
||||||
|
],
|
||||||
|
});
|
||||||
context.fileNodes.set(file.path, target);
|
context.fileNodes.set(file.path, target);
|
||||||
if (file.kind === "javascript") context.assetNodes.set(file.path, target);
|
if (file.kind === "javascript") context.assetNodes.set(file.path, target);
|
||||||
const entry = createAsarEntry(context, file);
|
const entry = createAsarEntry(context, file);
|
||||||
@@ -159,7 +198,18 @@ export const addJavaScriptArtifactFiles = (
|
|||||||
}),
|
}),
|
||||||
});
|
});
|
||||||
}
|
}
|
||||||
if (
|
if (analyzed.analysis_failure !== undefined)
|
||||||
|
addUnavailableStaticParseScope(context, {
|
||||||
|
file,
|
||||||
|
asset: target,
|
||||||
|
operation: "analyze-javascript-source",
|
||||||
|
limitation: analyzed.analysis_failure.reason,
|
||||||
|
failure: analyzed.analysis_failure.error,
|
||||||
|
limits: javaScriptAnalysisCoverageLimits(
|
||||||
|
analyzed.analysis_failure.limits,
|
||||||
|
),
|
||||||
|
});
|
||||||
|
else if (
|
||||||
file.kind === "javascript" &&
|
file.kind === "javascript" &&
|
||||||
(analyzed.javascript === null ||
|
(analyzed.javascript === null ||
|
||||||
analyzed.javascript.parse_status === "failed")
|
analyzed.javascript.parse_status === "failed")
|
||||||
@@ -172,6 +222,17 @@ export const addJavaScriptArtifactFiles = (
|
|||||||
? "JavaScript syntax could not be parsed."
|
? "JavaScript syntax could not be parsed."
|
||||||
: `JavaScript text was unavailable: ${file.text.reason}.`,
|
: `JavaScript text was unavailable: ${file.text.reason}.`,
|
||||||
});
|
});
|
||||||
|
if (analyzed.application_projection_failure !== undefined)
|
||||||
|
addUnavailableStaticParseScope(context, {
|
||||||
|
file,
|
||||||
|
asset: target,
|
||||||
|
operation: "project-javascript-application",
|
||||||
|
limitation: analyzed.application_projection_failure.reason,
|
||||||
|
failure: analyzed.application_projection_failure.error,
|
||||||
|
limits: javaScriptAnalysisCoverageLimits(
|
||||||
|
analyzed.application_projection_failure.limits,
|
||||||
|
),
|
||||||
|
});
|
||||||
const packageValue = packagesByPath.get(file.path);
|
const packageValue = packagesByPath.get(file.path);
|
||||||
if (packageValue !== undefined && packageValue.status !== "included")
|
if (packageValue !== undefined && packageValue.status !== "included")
|
||||||
addUnavailableStaticParseScope(context, {
|
addUnavailableStaticParseScope(context, {
|
||||||
@@ -198,6 +259,23 @@ export const addJavaScriptArtifactFiles = (
|
|||||||
}
|
}
|
||||||
};
|
};
|
||||||
|
|
||||||
|
const sourceFileRange = (
|
||||||
|
file: JavaScriptArtifactFile,
|
||||||
|
): JavaScriptSourceRange => {
|
||||||
|
if (!file.text.included)
|
||||||
|
throw new TypeError("Retained static facts require their source text");
|
||||||
|
let line = 1;
|
||||||
|
let lastLineStart = 0;
|
||||||
|
for (const match of file.text.value.matchAll(/\r\n|[\n\r\u2028\u2029]/gu)) {
|
||||||
|
line += 1;
|
||||||
|
lastLineStart = match.index + match[0].length;
|
||||||
|
}
|
||||||
|
return {
|
||||||
|
start: { line: 1, column: 0 },
|
||||||
|
end: { line, column: file.text.value.length - lastLineStart },
|
||||||
|
};
|
||||||
|
};
|
||||||
|
|
||||||
/** Project package metadata and its declared Electron roles. */
|
/** Project package metadata and its declared Electron roles. */
|
||||||
export const addJavaScriptPackageNodes = (
|
export const addJavaScriptPackageNodes = (
|
||||||
context: JavaScriptArtifactGraphContext,
|
context: JavaScriptArtifactGraphContext,
|
||||||
@@ -407,3 +485,4 @@ const containerProperties = (container: JavaScriptArtifactContainer) => ({
|
|||||||
bytes: container.bytes,
|
bytes: container.bytes,
|
||||||
inventory_artifact_id: container.inventory_artifact_id,
|
inventory_artifact_id: container.inventory_artifact_id,
|
||||||
});
|
});
|
||||||
|
import { javaScriptAnalysisCoverageLimits } from "../../domain/javascript/javascriptAnalysisResourceControls.js";
|
||||||
|
|||||||
@@ -11,9 +11,15 @@ import { createJavaScriptArtifactReader as createReader } from "../../artifacts/
|
|||||||
import type { JavaScriptApplicationGraph } from "../../domain/javascript/javascriptApplicationGraph.js";
|
import type { JavaScriptApplicationGraph } from "../../domain/javascript/javascriptApplicationGraph.js";
|
||||||
import type { JavaScriptSemanticGraph } from "../../domain/javascript/javascriptSemanticGraph.js";
|
import type { JavaScriptSemanticGraph } from "../../domain/javascript/javascriptSemanticGraph.js";
|
||||||
import type { ElectronBoundarySummary } from "../../domain/javascript/javascriptApplicationAnalysis.js";
|
import type { ElectronBoundarySummary } from "../../domain/javascript/javascriptApplicationAnalysis.js";
|
||||||
|
import { parseOwnedJavaScriptApplicationAnalysisSteps } from "../../domain/javascript/javascriptApplicationAnalysis.js";
|
||||||
|
import { AnalysisError } from "../../domain/analysisErrorBase.js";
|
||||||
import type { ArtifactInventorySnapshot } from "../../domain/artifactInventorySnapshot.js";
|
import type { ArtifactInventorySnapshot } from "../../domain/artifactInventorySnapshot.js";
|
||||||
import type { ArtifactInventoryPartialObservation } from "../../domain/artifactPartialObservation.js";
|
import type { ArtifactInventoryPartialObservation } from "../../domain/artifactPartialObservation.js";
|
||||||
import { analyzeAndProjectJavaScriptArtifactFiles } from "./JavaScriptArtifactAnalysis.js";
|
import type { ArtifactResourceScope } from "../../artifacts/ArtifactResourceScope.js";
|
||||||
|
import { analyzeAndAdoptJavaScriptArtifactFiles } from "./JavaScriptArtifactAnalysis.js";
|
||||||
|
import { createJavaScriptSourceAnalysis } from "../../artifacts/javascript/JavaScriptSourceAnalysis.js";
|
||||||
|
import { JAVASCRIPT_ANALYSIS_RESOURCE_DEFAULTS } from "../../domain/javascript/javascriptAnalysisResourceControls.js";
|
||||||
|
import { createOwnedJavaScriptApplicationEvidenceCooperatively } from "./JavaScriptApplicationEvidence.js";
|
||||||
import { readJavaScriptArtifactFiles } from "../../artifacts/javascript/JavaScriptArtifactFiles.js";
|
import { readJavaScriptArtifactFiles } from "../../artifacts/javascript/JavaScriptArtifactFiles.js";
|
||||||
import { buildImmutableJavaScriptArtifactGraphSteps } from "./JavaScriptArtifactGraphBuilder.js";
|
import { buildImmutableJavaScriptArtifactGraphSteps } from "./JavaScriptArtifactGraphBuilder.js";
|
||||||
import {
|
import {
|
||||||
@@ -21,7 +27,6 @@ import {
|
|||||||
type JavaScriptArtifactReconstructionInput,
|
type JavaScriptArtifactReconstructionInput,
|
||||||
} from "./JavaScriptArtifactReconstructionInput.js";
|
} from "./JavaScriptArtifactReconstructionInput.js";
|
||||||
import { scanCanonicalArtifactInventoryInScope } from "../../artifacts/inventory/scanCanonical.js";
|
import { scanCanonicalArtifactInventoryInScope } from "../../artifacts/inventory/scanCanonical.js";
|
||||||
import type { ArtifactResourceScope } from "../../artifacts/ArtifactResourceScope.js";
|
|
||||||
import { summarizeElectronBoundaries } from "./ElectronBoundaryAnalysis.js";
|
import { summarizeElectronBoundaries } from "./ElectronBoundaryAnalysis.js";
|
||||||
import { createJavaScriptSemanticGraphProjection } from "./JavaScriptSemanticGraphBuilder.js";
|
import { createJavaScriptSemanticGraphProjection } from "./JavaScriptSemanticGraphBuilder.js";
|
||||||
import type { ProgressReporter } from "../ProgressReporter.js";
|
import type { ProgressReporter } from "../ProgressReporter.js";
|
||||||
@@ -63,12 +68,19 @@ export const reconstructJavaScriptArtifact = async (
|
|||||||
progress?: ProgressReporter,
|
progress?: ProgressReporter,
|
||||||
): Promise<JavaScriptArtifactReconstructionResult> =>
|
): Promise<JavaScriptArtifactReconstructionResult> =>
|
||||||
scope.run(async () => {
|
scope.run(async () => {
|
||||||
|
let effectiveSignal = signal;
|
||||||
|
let effectiveProgress = progress;
|
||||||
const reportPhase = async (
|
const reportPhase = async (
|
||||||
phase: string,
|
phase: string,
|
||||||
message: string,
|
message: string,
|
||||||
): Promise<void> => {
|
): Promise<void> => {
|
||||||
await progress?.report({ phase, completed: 0, total: 1, message });
|
await effectiveProgress?.report({
|
||||||
await checkpointJavaScriptAnalysis(signal);
|
phase,
|
||||||
|
completed: 0,
|
||||||
|
total: 1,
|
||||||
|
message,
|
||||||
|
});
|
||||||
|
await checkpointJavaScriptAnalysis(effectiveSignal);
|
||||||
};
|
};
|
||||||
const input = javascriptArtifactReconstructionInputSchema.parse(rawInput);
|
const input = javascriptArtifactReconstructionInputSchema.parse(rawInput);
|
||||||
abortIfNeeded(signal);
|
abortIfNeeded(signal);
|
||||||
@@ -85,7 +97,13 @@ export const reconstructJavaScriptArtifact = async (
|
|||||||
`Artifact inventory classified ${path} as ${snapshot.manifest.root_format}, not ${format}`,
|
`Artifact inventory classified ${path} as ${snapshot.manifest.root_format}, not ${format}`,
|
||||||
);
|
);
|
||||||
const reader = createReader(path, format);
|
const reader = createReader(path, format);
|
||||||
let outcome:
|
const readerOwner = {
|
||||||
|
kind: "reader" as const,
|
||||||
|
reader,
|
||||||
|
resource: `JavaScript artifact reader for ${path}`,
|
||||||
|
};
|
||||||
|
let readerOpen = true;
|
||||||
|
let reconstructionOutcome:
|
||||||
| {
|
| {
|
||||||
readonly kind: "completed";
|
readonly kind: "completed";
|
||||||
readonly result: JavaScriptArtifactReconstructionResult;
|
readonly result: JavaScriptArtifactReconstructionResult;
|
||||||
@@ -102,28 +120,75 @@ export const reconstructJavaScriptArtifact = async (
|
|||||||
scope,
|
scope,
|
||||||
signal,
|
signal,
|
||||||
);
|
);
|
||||||
|
// Release acquisition before analysis consumes these owned file facts.
|
||||||
|
const earlyCleanup = await scope.release(readerOwner);
|
||||||
|
readerOpen = false;
|
||||||
|
if (earlyCleanup.kind === "failed")
|
||||||
|
throw ArtifactReaderFailure.withCleanup(
|
||||||
|
earlyCleanup.cause,
|
||||||
|
ArtifactReaderFailure.cleanupObservation(
|
||||||
|
earlyCleanup.cause,
|
||||||
|
readerOwner.resource,
|
||||||
|
),
|
||||||
|
{ kind: "artifact-inventory", inventory: snapshot },
|
||||||
|
);
|
||||||
await reportPhase(
|
await reportPhase(
|
||||||
"parse_javascript_sources",
|
"parse_javascript_sources",
|
||||||
`Parsing and projecting ${String(files.files.length)} application source files`,
|
`Parsing and projecting ${String(files.files.length)} application source files`,
|
||||||
);
|
);
|
||||||
const semanticProjection = createJavaScriptSemanticGraphProjection();
|
const semanticProjection = createJavaScriptSemanticGraphProjection();
|
||||||
const analysis = await analyzeAndProjectJavaScriptArtifactFiles(
|
const worker = createJavaScriptSourceAnalysis(
|
||||||
files,
|
{
|
||||||
semanticProjection.projectFileSteps,
|
heapMb:
|
||||||
async (file, completed, total) => {
|
input.max_heap_mb ?? JAVASCRIPT_ANALYSIS_RESOURCE_DEFAULTS.heapMb,
|
||||||
abortIfNeeded(signal);
|
timeoutMs:
|
||||||
await progress?.report({
|
input.analysis_timeout_ms ??
|
||||||
phase: "parse_javascript_source",
|
JAVASCRIPT_ANALYSIS_RESOURCE_DEFAULTS.timeoutMs,
|
||||||
completed: 0,
|
|
||||||
total: 1,
|
|
||||||
message: `Parsing and projecting ${file.path} (${String(completed + 1)}/${String(total)})`,
|
|
||||||
});
|
|
||||||
abortIfNeeded(signal);
|
|
||||||
},
|
},
|
||||||
signal,
|
files.files.filter(
|
||||||
|
(file) => file.kind === "javascript" && file.text.included,
|
||||||
|
).length,
|
||||||
);
|
);
|
||||||
await checkpointJavaScriptAnalysis(signal);
|
let outcome;
|
||||||
abortIfNeeded(signal);
|
let cleanupFailure: AnalysisError | undefined;
|
||||||
|
try {
|
||||||
|
outcome = await analyzeAndAdoptJavaScriptArtifactFiles(
|
||||||
|
files,
|
||||||
|
worker,
|
||||||
|
semanticProjection,
|
||||||
|
async (file, completed, total) => {
|
||||||
|
abortIfNeeded(signal);
|
||||||
|
await progress?.report({
|
||||||
|
phase: "parse_javascript_source",
|
||||||
|
completed: 0,
|
||||||
|
total: 1,
|
||||||
|
message: `Parsing and projecting ${file.path} (${String(completed + 1)}/${String(total)})`,
|
||||||
|
});
|
||||||
|
abortIfNeeded(signal);
|
||||||
|
},
|
||||||
|
signal,
|
||||||
|
);
|
||||||
|
} finally {
|
||||||
|
try {
|
||||||
|
const cleanup = await scope.release({
|
||||||
|
kind: "javascript-source-analysis",
|
||||||
|
analysis: worker,
|
||||||
|
resource: `JavaScript source analyzer for ${path}`,
|
||||||
|
});
|
||||||
|
if (cleanup.kind === "failed") throw cleanup.cause;
|
||||||
|
} catch (cause: unknown) {
|
||||||
|
if (!(cause instanceof AnalysisError)) throw cause;
|
||||||
|
cleanupFailure = cause;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
const analysis = outcome.analysis;
|
||||||
|
const interruption = cleanupFailure ?? outcome.interruption;
|
||||||
|
if (interruption !== null && interruption !== undefined) {
|
||||||
|
effectiveSignal = undefined;
|
||||||
|
effectiveProgress = undefined;
|
||||||
|
}
|
||||||
|
await checkpointJavaScriptAnalysis(effectiveSignal);
|
||||||
|
abortIfNeeded(effectiveSignal);
|
||||||
await reportPhase(
|
await reportPhase(
|
||||||
"build_javascript_application_graph",
|
"build_javascript_application_graph",
|
||||||
"Constructing application and Electron boundary relationships",
|
"Constructing application and Electron boundary relationships",
|
||||||
@@ -139,7 +204,7 @@ export const reconstructJavaScriptArtifact = async (
|
|||||||
);
|
);
|
||||||
const graph = await completeJavaScriptAnalysisSteps(
|
const graph = await completeJavaScriptAnalysisSteps(
|
||||||
applicationGraphSteps,
|
applicationGraphSteps,
|
||||||
signal,
|
effectiveSignal,
|
||||||
);
|
);
|
||||||
await reportPhase(
|
await reportPhase(
|
||||||
"build_javascript_semantic_graph",
|
"build_javascript_semantic_graph",
|
||||||
@@ -155,63 +220,83 @@ export const reconstructJavaScriptArtifact = async (
|
|||||||
);
|
);
|
||||||
const semanticGraph = await completeJavaScriptAnalysisSteps(
|
const semanticGraph = await completeJavaScriptAnalysisSteps(
|
||||||
semanticGraphSteps,
|
semanticGraphSteps,
|
||||||
signal,
|
effectiveSignal,
|
||||||
);
|
);
|
||||||
await checkpointJavaScriptAnalysis(signal);
|
await checkpointJavaScriptAnalysis(effectiveSignal);
|
||||||
outcome = {
|
const completed = {
|
||||||
kind: "completed",
|
input_path: path,
|
||||||
result: {
|
format,
|
||||||
input_path: path,
|
root_artifact_sha256: snapshot.manifest.root_sha256,
|
||||||
format,
|
inventory_manifest_id: snapshot.manifest.manifest_id,
|
||||||
root_artifact_sha256: snapshot.manifest.root_sha256,
|
inventory_graph_sha256: snapshot.manifest.graph_sha256,
|
||||||
inventory_manifest_id: snapshot.manifest.manifest_id,
|
integrity_contradictions: snapshot.integrity_contradictions,
|
||||||
inventory_graph_sha256: snapshot.manifest.graph_sha256,
|
graph,
|
||||||
integrity_contradictions: snapshot.integrity_contradictions,
|
semantic_graph: semanticGraph,
|
||||||
graph,
|
electron_summary: summarizeElectronBoundaries(analysis),
|
||||||
semantic_graph: semanticGraph,
|
statistics: {
|
||||||
electron_summary: summarizeElectronBoundaries(analysis),
|
relevant_files: files.files.length,
|
||||||
statistics: {
|
nested_asar_containers: files.containers.length,
|
||||||
relevant_files: files.files.length,
|
text_bytes_read: files.text_bytes_read,
|
||||||
nested_asar_containers: files.containers.length,
|
invalid_utf8_files: files.invalid_utf8_files,
|
||||||
text_bytes_read: files.text_bytes_read,
|
parsed_javascript_files: analysis.files.filter(
|
||||||
invalid_utf8_files: files.invalid_utf8_files,
|
({ javascript }) => javascript !== null,
|
||||||
parsed_javascript_files: analysis.files.filter(
|
).length,
|
||||||
({ javascript }) => javascript !== null,
|
visited_ast_nodes: analysis.visited_ast_nodes,
|
||||||
).length,
|
findings: analysis.findings,
|
||||||
visited_ast_nodes: analysis.visited_ast_nodes,
|
modules: analysis.modules,
|
||||||
findings: analysis.findings,
|
parse_failures: analysis.parse_failures,
|
||||||
modules: analysis.modules,
|
|
||||||
parse_failures: analysis.parse_failures,
|
|
||||||
},
|
|
||||||
limitations: analysis.limitations,
|
|
||||||
},
|
},
|
||||||
|
limitations: analysis.limitations,
|
||||||
};
|
};
|
||||||
|
if (interruption !== null && interruption !== undefined) {
|
||||||
|
const { electron_summary: summary, ...application } = completed;
|
||||||
|
const partial = await completeJavaScriptAnalysisSteps(
|
||||||
|
parseOwnedJavaScriptApplicationAnalysisSteps({
|
||||||
|
...application,
|
||||||
|
summary,
|
||||||
|
limitations: completed.graph.limitations,
|
||||||
|
}),
|
||||||
|
);
|
||||||
|
const { max_heap_mb, analysis_timeout_ms, ...coreInput } = input;
|
||||||
|
const parameters = {
|
||||||
|
...coreInput,
|
||||||
|
...(max_heap_mb === undefined ? {} : { max_heap_mb }),
|
||||||
|
...(analysis_timeout_ms === undefined ? {} : { analysis_timeout_ms }),
|
||||||
|
};
|
||||||
|
const evidence =
|
||||||
|
await createOwnedJavaScriptApplicationEvidenceCooperatively(
|
||||||
|
parameters,
|
||||||
|
partial,
|
||||||
|
);
|
||||||
|
throw interruption.retainPartialObservation(evidence);
|
||||||
|
}
|
||||||
|
reconstructionOutcome = { kind: "completed", result: completed };
|
||||||
} catch (cause: unknown) {
|
} catch (cause: unknown) {
|
||||||
outcome = { kind: "failed", cause };
|
reconstructionOutcome = { kind: "failed", cause };
|
||||||
}
|
}
|
||||||
const owner = {
|
if (readerOpen) {
|
||||||
kind: "reader" as const,
|
const cleanupAttempt = await scope.release(readerOwner);
|
||||||
reader,
|
if (cleanupAttempt.kind === "failed") {
|
||||||
resource: `JavaScript artifact reader for ${path}`,
|
const cleanup = ArtifactReaderFailure.cleanupObservation(
|
||||||
};
|
cleanupAttempt.cause,
|
||||||
const cleanupAttempt = await scope.release(owner);
|
readerOwner.resource,
|
||||||
if (cleanupAttempt.kind === "failed") {
|
);
|
||||||
const cleanup = ArtifactReaderFailure.cleanupObservation(
|
const partialObservation: ArtifactInventoryPartialObservation = {
|
||||||
cleanupAttempt.cause,
|
kind: "artifact-inventory",
|
||||||
owner.resource,
|
inventory: snapshot,
|
||||||
);
|
};
|
||||||
const partialObservation: ArtifactInventoryPartialObservation = {
|
throw ArtifactReaderFailure.withCleanup(
|
||||||
kind: "artifact-inventory",
|
reconstructionOutcome.kind === "failed"
|
||||||
inventory: snapshot,
|
? reconstructionOutcome.cause
|
||||||
};
|
: cleanupAttempt.cause,
|
||||||
throw ArtifactReaderFailure.withCleanup(
|
cleanup,
|
||||||
outcome.kind === "failed" ? outcome.cause : cleanupAttempt.cause,
|
partialObservation,
|
||||||
cleanup,
|
);
|
||||||
partialObservation,
|
}
|
||||||
);
|
|
||||||
}
|
}
|
||||||
if (outcome.kind === "failed") throw outcome.cause;
|
if (reconstructionOutcome.kind === "failed")
|
||||||
return outcome.result;
|
throw reconstructionOutcome.cause;
|
||||||
|
return reconstructionOutcome.result;
|
||||||
});
|
});
|
||||||
|
|
||||||
const OPERATION = "analyze_javascript_application";
|
const OPERATION = "analyze_javascript_application";
|
||||||
|
|||||||
@@ -1,11 +1,13 @@
|
|||||||
import { z } from "zod";
|
import { z } from "zod";
|
||||||
import { artifactIntegrityPolicySchema } from "../../domain/artifactIntegrityPolicy.js";
|
import { artifactIntegrityPolicySchema } from "../../domain/artifactIntegrityPolicy.js";
|
||||||
|
import { javaScriptAnalysisResourceControlsSchema } from "../../domain/javascript/javascriptAnalysisResourceControls.js";
|
||||||
|
|
||||||
/** Local ASAR/directory reconstruction request. */
|
/** Local ASAR/directory reconstruction request. */
|
||||||
export const javascriptArtifactReconstructionInputSchema = z.strictObject({
|
export const javascriptArtifactReconstructionInputSchema = z.strictObject({
|
||||||
input_path: z.string().min(1),
|
input_path: z.string().min(1),
|
||||||
format: z.enum(["auto", "asar", "directory"]).default("auto"),
|
format: z.enum(["auto", "asar", "directory"]).default("auto"),
|
||||||
integrity_policy: artifactIntegrityPolicySchema,
|
integrity_policy: artifactIntegrityPolicySchema,
|
||||||
|
...javaScriptAnalysisResourceControlsSchema.shape,
|
||||||
});
|
});
|
||||||
|
|
||||||
/** Parsed local reconstruction request. */
|
/** Parsed local reconstruction request. */
|
||||||
|
|||||||
@@ -1,5 +1,6 @@
|
|||||||
import {
|
import {
|
||||||
createJavaScriptSemanticGraph,
|
createJavaScriptSemanticGraph,
|
||||||
|
createJavaScriptSemanticGraphUnknown,
|
||||||
sealTransferredJavaScriptSemanticGraphSteps,
|
sealTransferredJavaScriptSemanticGraphSteps,
|
||||||
type JavaScriptSemanticGraph,
|
type JavaScriptSemanticGraph,
|
||||||
} from "../../domain/javascript/javascriptSemanticGraph.js";
|
} from "../../domain/javascript/javascriptSemanticGraph.js";
|
||||||
@@ -10,6 +11,7 @@ import type {
|
|||||||
import {
|
import {
|
||||||
JAVASCRIPT_SEMANTIC_RELATION_FAMILIES,
|
JAVASCRIPT_SEMANTIC_RELATION_FAMILIES,
|
||||||
JAVASCRIPT_SEMANTIC_RELATION_FAMILY,
|
JAVASCRIPT_SEMANTIC_RELATION_FAMILY,
|
||||||
|
JAVASCRIPT_SEMANTIC_RELATIONS,
|
||||||
} from "../../domain/javascript/javascriptSemanticGraphSchemas.js";
|
} from "../../domain/javascript/javascriptSemanticGraphSchemas.js";
|
||||||
import type {
|
import type {
|
||||||
JavaScriptSemanticCallArgument,
|
JavaScriptSemanticCallArgument,
|
||||||
@@ -33,6 +35,13 @@ import { projectSemanticResources } from "./JavaScriptSemanticGraphResourceProje
|
|||||||
import { projectSemanticObjects } from "./JavaScriptSemanticGraphObjectProjection.js";
|
import { projectSemanticObjects } from "./JavaScriptSemanticGraphObjectProjection.js";
|
||||||
import { projectSemanticValues } from "./JavaScriptSemanticGraphValueProjection.js";
|
import { projectSemanticValues } from "./JavaScriptSemanticGraphValueProjection.js";
|
||||||
import { inferredSemanticEvidenceAt } from "./JavaScriptSemanticGraphEvidence.js";
|
import { inferredSemanticEvidenceAt } from "./JavaScriptSemanticGraphEvidence.js";
|
||||||
|
import { observedSemanticEvidence } from "./JavaScriptSemanticGraphEvidence.js";
|
||||||
|
import {
|
||||||
|
partialApplicationCoverage,
|
||||||
|
type ApplicationCoverage,
|
||||||
|
} from "../../domain/javascript/javascriptApplicationEvidenceSchemas.js";
|
||||||
|
import type { JsonValue } from "../../domain/jsonValue.js";
|
||||||
|
import { javaScriptAnalysisCoverageLimits } from "../../domain/javascript/javascriptAnalysisResourceControls.js";
|
||||||
import { projectSemanticFunctionFingerprints } from "./JavaScriptSemanticGraphFingerprintProjection.js";
|
import { projectSemanticFunctionFingerprints } from "./JavaScriptSemanticGraphFingerprintProjection.js";
|
||||||
import {
|
import {
|
||||||
projectSemanticClosureCaptures,
|
projectSemanticClosureCaptures,
|
||||||
@@ -55,6 +64,7 @@ import {
|
|||||||
bindSemanticGraphApplicationNodes,
|
bindSemanticGraphApplicationNodes,
|
||||||
type SemanticGraphProjectionState as BuilderState,
|
type SemanticGraphProjectionState as BuilderState,
|
||||||
} from "./JavaScriptSemanticGraphConstruction.js";
|
} from "./JavaScriptSemanticGraphConstruction.js";
|
||||||
|
import type { JavaScriptSemanticFileProjection } from "../../domain/javascript/javascriptSemanticFileProjection.js";
|
||||||
|
|
||||||
interface BuilderInput {
|
interface BuilderInput {
|
||||||
readonly rootArtifactSha256: string;
|
readonly rootArtifactSha256: string;
|
||||||
@@ -126,6 +136,19 @@ export const buildJavaScriptSemanticGraph = ({
|
|||||||
|
|
||||||
/** File-local semantic projection that does not retain consumed source IR. */
|
/** File-local semantic projection that does not retain consumed source IR. */
|
||||||
export interface JavaScriptSemanticGraphProjection {
|
export interface JavaScriptSemanticGraphProjection {
|
||||||
|
/** Node capacity offered to the next independently analyzed source file. */
|
||||||
|
readonly remainingFileNodeBudget: () => number;
|
||||||
|
/** Adopt decoded file-local facts; final validation and proof remain in this realm. */
|
||||||
|
readonly adoptFileProjection: (
|
||||||
|
projection: JavaScriptSemanticFileProjection,
|
||||||
|
) => void;
|
||||||
|
/** Preserve source-bound unknowns for an execution or result-retention failure. */
|
||||||
|
readonly recordUnavailableFile: (
|
||||||
|
file: JavaScriptArtifactFile,
|
||||||
|
reason: string,
|
||||||
|
limits: Readonly<Record<string, JsonValue>> | null,
|
||||||
|
unknownReason: "resource-limit" | "incomplete-module",
|
||||||
|
) => void;
|
||||||
readonly projectFile: (
|
readonly projectFile: (
|
||||||
file: JavaScriptArtifactFile,
|
file: JavaScriptArtifactFile,
|
||||||
ir: JavaScriptSemanticIr,
|
ir: JavaScriptSemanticIr,
|
||||||
@@ -151,6 +174,12 @@ export const createJavaScriptSemanticGraphProjection =
|
|||||||
const state = emptyState();
|
const state = emptyState();
|
||||||
const fingerprints: JavaScriptSemanticFingerprint[] = [];
|
const fingerprints: JavaScriptSemanticFingerprint[] = [];
|
||||||
let truncatedFiles = 0;
|
let truncatedFiles = 0;
|
||||||
|
let unavailableFiles = 0;
|
||||||
|
let resourceTruncatedFiles = 0;
|
||||||
|
const executionLimits = new Map<
|
||||||
|
string,
|
||||||
|
ApplicationCoverage["limits"][number]
|
||||||
|
>();
|
||||||
const projectSourceSteps = function* (
|
const projectSourceSteps = function* (
|
||||||
file: JavaScriptArtifactFile,
|
file: JavaScriptArtifactFile,
|
||||||
ir: JavaScriptSemanticIr,
|
ir: JavaScriptSemanticIr,
|
||||||
@@ -207,12 +236,15 @@ export const createJavaScriptSemanticGraphProjection =
|
|||||||
fingerprints: [...fingerprints],
|
fingerprints: [...fingerprints],
|
||||||
unknowns,
|
unknowns,
|
||||||
coverage: {
|
coverage: {
|
||||||
status: truncatedFiles > 0 ? "partial" : "unknown",
|
status:
|
||||||
truncated: truncatedFiles > 0,
|
truncatedFiles > 0 || unavailableFiles > 0 ? "partial" : "unknown",
|
||||||
omitted_nodes: truncatedFiles > 0 ? null : 0,
|
truncated: truncatedFiles > 0 || resourceTruncatedFiles > 0,
|
||||||
omitted_relations: truncatedFiles > 0 ? null : 0,
|
omitted_nodes: truncatedFiles > 0 || unavailableFiles > 0 ? null : 0,
|
||||||
limits:
|
omitted_relations:
|
||||||
truncatedFiles > 0
|
truncatedFiles > 0 || unavailableFiles > 0 ? null : 0,
|
||||||
|
limits: [
|
||||||
|
...executionLimits.values(),
|
||||||
|
...(truncatedFiles > 0
|
||||||
? [
|
? [
|
||||||
{
|
{
|
||||||
name: "semantic_graph_node_ceiling",
|
name: "semantic_graph_node_ceiling",
|
||||||
@@ -220,12 +252,14 @@ export const createJavaScriptSemanticGraphProjection =
|
|||||||
unit: "items" as const,
|
unit: "items" as const,
|
||||||
},
|
},
|
||||||
]
|
]
|
||||||
: [],
|
: []),
|
||||||
|
],
|
||||||
families: JAVASCRIPT_SEMANTIC_RELATION_FAMILIES.map((family) => ({
|
families: JAVASCRIPT_SEMANTIC_RELATION_FAMILIES.map((family) => ({
|
||||||
family,
|
family,
|
||||||
status: semanticFamilyStatus(family),
|
status: semanticFamilyStatus(family),
|
||||||
retained_relations: retainedByFamily.get(family) ?? 0,
|
retained_relations: retainedByFamily.get(family) ?? 0,
|
||||||
omitted_relations: truncatedFiles > 0 ? null : 0,
|
omitted_relations:
|
||||||
|
truncatedFiles > 0 || unavailableFiles > 0 ? null : 0,
|
||||||
unknown_ids: unknownIdsByFamily.get(family) ?? [],
|
unknown_ids: unknownIdsByFamily.get(family) ?? [],
|
||||||
})),
|
})),
|
||||||
},
|
},
|
||||||
@@ -252,6 +286,87 @@ export const createJavaScriptSemanticGraphProjection =
|
|||||||
return graph;
|
return graph;
|
||||||
};
|
};
|
||||||
return {
|
return {
|
||||||
|
remainingFileNodeBudget: () =>
|
||||||
|
Math.min(
|
||||||
|
SEMANTIC_GRAPH_FILE_NODE_CEILING,
|
||||||
|
Math.max(0, SEMANTIC_GRAPH_NODE_CEILING - state.nodes.size),
|
||||||
|
),
|
||||||
|
adoptFileProjection: (projection) => {
|
||||||
|
const newNodes = projection.nodes.filter(
|
||||||
|
({ node_id }) => !state.nodes.has(node_id),
|
||||||
|
);
|
||||||
|
if (
|
||||||
|
newNodes.length >
|
||||||
|
Math.min(
|
||||||
|
SEMANTIC_GRAPH_FILE_NODE_CEILING,
|
||||||
|
SEMANTIC_GRAPH_NODE_CEILING - state.nodes.size,
|
||||||
|
)
|
||||||
|
)
|
||||||
|
throw new RangeError(
|
||||||
|
"Transferred semantic nodes exceed the offered tree capacity",
|
||||||
|
);
|
||||||
|
for (const context of projection.evidenceContexts)
|
||||||
|
state.evidenceContexts.adopt(context);
|
||||||
|
for (const node of newNodes) state.nodes.set(node.node_id, node);
|
||||||
|
for (const relation of projection.relations)
|
||||||
|
state.relations.set(relation.relation_id, relation);
|
||||||
|
for (const unknown of projection.unknowns)
|
||||||
|
state.unknowns.set(unknown.unknown_id, unknown);
|
||||||
|
for (const root of projection.roots) state.roots.add(root);
|
||||||
|
fingerprints.push(...projection.fingerprints);
|
||||||
|
if (projection.truncated) truncatedFiles += 1;
|
||||||
|
},
|
||||||
|
recordUnavailableFile: (file, reason, limits, unknownReason) => {
|
||||||
|
const retainedLimits = javaScriptAnalysisCoverageLimits(limits);
|
||||||
|
for (const limit of retainedLimits) {
|
||||||
|
executionLimits.set(
|
||||||
|
JSON.stringify([limit.name, limit.value, limit.unit]),
|
||||||
|
limit,
|
||||||
|
);
|
||||||
|
}
|
||||||
|
state.fileNodeBudget = Math.min(
|
||||||
|
SEMANTIC_GRAPH_FILE_NODE_CEILING,
|
||||||
|
Math.max(0, SEMANTIC_GRAPH_NODE_CEILING - state.nodes.size),
|
||||||
|
);
|
||||||
|
const module = retainNode(state, file, {
|
||||||
|
kind: "module",
|
||||||
|
roleKey: "module",
|
||||||
|
location: null,
|
||||||
|
label: file.path,
|
||||||
|
functionNodeId: null,
|
||||||
|
});
|
||||||
|
state.fileNodeBudget = null;
|
||||||
|
if (module !== null) state.roots.add(module.node_id);
|
||||||
|
const evidence = {
|
||||||
|
...observedSemanticEvidence(file, null),
|
||||||
|
authority: "unknown" as const,
|
||||||
|
state: "unknown" as const,
|
||||||
|
confidence: "unknown" as const,
|
||||||
|
coverage: partialApplicationCoverage(retainedLimits, null),
|
||||||
|
limitations: [reason],
|
||||||
|
};
|
||||||
|
for (const family of JAVASCRIPT_SEMANTIC_RELATION_FAMILIES) {
|
||||||
|
const unknown = createJavaScriptSemanticGraphUnknown(
|
||||||
|
{
|
||||||
|
node_id: module?.node_id ?? null,
|
||||||
|
family,
|
||||||
|
relation_kinds: JAVASCRIPT_SEMANTIC_RELATIONS.filter(
|
||||||
|
(relation) =>
|
||||||
|
JAVASCRIPT_SEMANTIC_RELATION_FAMILY[relation] === family,
|
||||||
|
),
|
||||||
|
reason: unknownReason,
|
||||||
|
detail: reason,
|
||||||
|
candidate_node_ids: [],
|
||||||
|
evidence,
|
||||||
|
},
|
||||||
|
state.evidenceContexts,
|
||||||
|
);
|
||||||
|
state.unknowns.set(unknown.unknown_id, unknown);
|
||||||
|
}
|
||||||
|
unavailableFiles += 1;
|
||||||
|
if (unknownReason === "resource-limit" && retainedLimits.length > 0)
|
||||||
|
resourceTruncatedFiles += 1;
|
||||||
|
},
|
||||||
projectFile: (file, ir) => {
|
projectFile: (file, ir) => {
|
||||||
const steps = projectSourceSteps(file, ir);
|
const steps = projectSourceSteps(file, ir);
|
||||||
while (steps.next().done !== true);
|
while (steps.next().done !== true);
|
||||||
@@ -272,6 +387,35 @@ export const createJavaScriptSemanticGraphProjection =
|
|||||||
};
|
};
|
||||||
};
|
};
|
||||||
|
|
||||||
|
/** Project one source with the parent's remaining node capacity, retaining no full IR. */
|
||||||
|
export function* projectJavaScriptSemanticFileSteps(
|
||||||
|
file: JavaScriptArtifactFile,
|
||||||
|
ir: JavaScriptSemanticIr,
|
||||||
|
nodeBudget: number,
|
||||||
|
): Generator<void, JavaScriptSemanticFileProjection> {
|
||||||
|
if (
|
||||||
|
!Number.isSafeInteger(nodeBudget) ||
|
||||||
|
nodeBudget < 0 ||
|
||||||
|
nodeBudget > SEMANTIC_GRAPH_FILE_NODE_CEILING
|
||||||
|
)
|
||||||
|
throw new RangeError(
|
||||||
|
"File semantic projection requires a valid offered node capacity",
|
||||||
|
);
|
||||||
|
const state = emptyState();
|
||||||
|
state.fileNodeBudget = nodeBudget;
|
||||||
|
state.fileNodesDropped = false;
|
||||||
|
const fingerprints = yield* projectFileSteps(file, ir, state);
|
||||||
|
return {
|
||||||
|
roots: [...state.roots],
|
||||||
|
evidenceContexts: state.evidenceContexts.contexts,
|
||||||
|
nodes: [...state.nodes.values()],
|
||||||
|
relations: [...state.relations.values()],
|
||||||
|
unknowns: [...state.unknowns.values()],
|
||||||
|
fingerprints,
|
||||||
|
truncated: Boolean(state.fileNodesDropped),
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
function* projectFileSteps(
|
function* projectFileSteps(
|
||||||
file: JavaScriptArtifactFile,
|
file: JavaScriptArtifactFile,
|
||||||
ir: JavaScriptSemanticIr,
|
ir: JavaScriptSemanticIr,
|
||||||
|
|||||||
@@ -1,5 +1,7 @@
|
|||||||
import { expect, it } from "vitest";
|
import { expect, it } from "vitest";
|
||||||
|
|
||||||
|
import { err } from "../domain/result.js";
|
||||||
|
import { AnalysisOutputError } from "../domain/analysisErrorCore.js";
|
||||||
import type { ArtifactReader } from "./ArtifactReader.js";
|
import type { ArtifactReader } from "./ArtifactReader.js";
|
||||||
import { ArtifactResourceScope } from "./ArtifactResourceScope.js";
|
import { ArtifactResourceScope } from "./ArtifactResourceScope.js";
|
||||||
|
|
||||||
@@ -66,3 +68,47 @@ it("retains the same failed reader owner and retries before admitting work", asy
|
|||||||
await scope.close();
|
await scope.close();
|
||||||
expect(closeAttempts).toBe(4);
|
expect(closeAttempts).toBe(4);
|
||||||
});
|
});
|
||||||
|
|
||||||
|
it("retains a failed source analyzer until shutdown can verify its release", async () => {
|
||||||
|
const scope = new ArtifactResourceScope();
|
||||||
|
let releaseAllowed = false;
|
||||||
|
const failure = new AnalysisOutputError(
|
||||||
|
"analyze_javascript_application",
|
||||||
|
"Owned worker is still running",
|
||||||
|
);
|
||||||
|
const owner = {
|
||||||
|
kind: "javascript-source-analysis" as const,
|
||||||
|
resource: "fixture source analyzer",
|
||||||
|
analysis: {
|
||||||
|
async analyze() {
|
||||||
|
return err({
|
||||||
|
error: failure,
|
||||||
|
javascript: null,
|
||||||
|
module: null,
|
||||||
|
projection: null,
|
||||||
|
});
|
||||||
|
},
|
||||||
|
async close() {
|
||||||
|
if (!releaseAllowed) throw failure;
|
||||||
|
},
|
||||||
|
},
|
||||||
|
};
|
||||||
|
expect(await scope.release(owner)).toEqual({
|
||||||
|
kind: "failed",
|
||||||
|
cause: failure,
|
||||||
|
});
|
||||||
|
let admitted = false;
|
||||||
|
await expect(
|
||||||
|
scope.run(async () => {
|
||||||
|
admitted = true;
|
||||||
|
}),
|
||||||
|
).rejects.toMatchObject({
|
||||||
|
cleanup: { resources: ["fixture source analyzer"] },
|
||||||
|
});
|
||||||
|
expect(admitted).toBe(false);
|
||||||
|
await expect(scope.close()).rejects.toMatchObject({
|
||||||
|
cleanup: { resources: ["fixture source analyzer"] },
|
||||||
|
});
|
||||||
|
releaseAllowed = true;
|
||||||
|
await scope.close();
|
||||||
|
});
|
||||||
|
|||||||
@@ -1,3 +1,4 @@
|
|||||||
|
import type { JavaScriptSourceAnalysisPort } from "../domain/javascript/javascriptSourceAnalysis.js";
|
||||||
import type { OwnedFileHandle } from "../filesystem/OwnedFileHandle.js";
|
import type { OwnedFileHandle } from "../filesystem/OwnedFileHandle.js";
|
||||||
import type { OwnedDirectoryHandle } from "../filesystem/OwnedDirectoryHandle.js";
|
import type { OwnedDirectoryHandle } from "../filesystem/OwnedDirectoryHandle.js";
|
||||||
|
|
||||||
@@ -7,6 +8,11 @@ import type { SafeOutputTree } from "./SafeOutputTree.js";
|
|||||||
|
|
||||||
/** One artifact resource and its format-specific cleanup owner. */
|
/** One artifact resource and its format-specific cleanup owner. */
|
||||||
export type ArtifactResourceOwner =
|
export type ArtifactResourceOwner =
|
||||||
|
| {
|
||||||
|
readonly kind: "javascript-source-analysis";
|
||||||
|
readonly resource: string;
|
||||||
|
readonly analysis: JavaScriptSourceAnalysisPort;
|
||||||
|
}
|
||||||
| {
|
| {
|
||||||
readonly kind: "directory-handle";
|
readonly kind: "directory-handle";
|
||||||
readonly resource: string;
|
readonly resource: string;
|
||||||
@@ -134,6 +140,9 @@ export class ArtifactResourceScope {
|
|||||||
|
|
||||||
const cleanup = async (owner: ArtifactResourceOwner): Promise<void> => {
|
const cleanup = async (owner: ArtifactResourceOwner): Promise<void> => {
|
||||||
switch (owner.kind) {
|
switch (owner.kind) {
|
||||||
|
case "javascript-source-analysis":
|
||||||
|
await owner.analysis.close();
|
||||||
|
return;
|
||||||
case "reader":
|
case "reader":
|
||||||
await owner.reader.close();
|
await owner.reader.close();
|
||||||
return;
|
return;
|
||||||
|
|||||||
@@ -0,0 +1,12 @@
|
|||||||
|
import type {
|
||||||
|
JavaScriptSourceAnalysisLimits,
|
||||||
|
JavaScriptSourceAnalysisPort,
|
||||||
|
} from "../../domain/javascript/javascriptSourceAnalysis.js";
|
||||||
|
import { JavaScriptAnalysisWorker } from "../../javascript/analysis/JavaScriptAnalysisWorker.js";
|
||||||
|
|
||||||
|
/** Construct the inert artifact analyzer; resources are acquired on the first source. */
|
||||||
|
export const createJavaScriptSourceAnalysis = (
|
||||||
|
limits: JavaScriptSourceAnalysisLimits,
|
||||||
|
sourceCount: number,
|
||||||
|
): JavaScriptSourceAnalysisPort =>
|
||||||
|
new JavaScriptAnalysisWorker(limits, undefined, sourceCount);
|
||||||
@@ -198,6 +198,12 @@ const registerJavaScriptApplicationCommand = (
|
|||||||
input_path: args.path,
|
input_path: args.path,
|
||||||
format: options.artifactFormat,
|
format: options.artifactFormat,
|
||||||
integrity_policy: options.integrityPolicy,
|
integrity_policy: options.integrityPolicy,
|
||||||
|
...(options.maxHeapMb === undefined
|
||||||
|
? {}
|
||||||
|
: { max_heap_mb: options.maxHeapMb }),
|
||||||
|
...(options.analysisTimeoutMs === undefined
|
||||||
|
? {}
|
||||||
|
: { analysis_timeout_ms: options.analysisTimeoutMs }),
|
||||||
},
|
},
|
||||||
signal,
|
signal,
|
||||||
),
|
),
|
||||||
|
|||||||
@@ -1,4 +1,5 @@
|
|||||||
import { z } from "incur";
|
import { z } from "incur";
|
||||||
|
import { javaScriptAnalysisResourceControlsSchema } from "./domain/javascript/javascriptAnalysisResourceControls.js";
|
||||||
|
|
||||||
/** Accept an agent-selected browser observation duration without an artificial ceiling. */
|
/** Accept an agent-selected browser observation duration without an artificial ceiling. */
|
||||||
export const observationDuration = (fallback: number, minimum = 0) =>
|
export const observationDuration = (fallback: number, minimum = 0) =>
|
||||||
@@ -59,6 +60,9 @@ export const electronPageInspectionOptions = z.object({
|
|||||||
});
|
});
|
||||||
|
|
||||||
export const javascriptApplicationOptions = z.object({
|
export const javascriptApplicationOptions = z.object({
|
||||||
|
maxHeapMb: javaScriptAnalysisResourceControlsSchema.shape.max_heap_mb,
|
||||||
|
analysisTimeoutMs:
|
||||||
|
javaScriptAnalysisResourceControlsSchema.shape.analysis_timeout_ms,
|
||||||
artifactFormat: z
|
artifactFormat: z
|
||||||
.enum(["auto", "asar", "directory"])
|
.enum(["auto", "asar", "directory"])
|
||||||
.default("auto")
|
.default("auto")
|
||||||
|
|||||||
@@ -0,0 +1,82 @@
|
|||||||
|
import { parseJavaScriptSource } from "../domain/javascript/javascriptSourceParser.js";
|
||||||
|
import { analyzeParsedJavaScriptStaticSourceSteps } from "../domain/javascript/javascriptStaticAnalysis.js";
|
||||||
|
import {
|
||||||
|
analyzeParsedJavaScriptSemanticsSteps,
|
||||||
|
classifyParsedJavaScriptOpenReceiversSteps,
|
||||||
|
} from "../domain/javascript/javascriptSemanticAnalysis.js";
|
||||||
|
import { failedJavaScriptStaticAnalysis } from "../domain/javascript/javascriptStaticAnalysisHelpers.js";
|
||||||
|
import { projectJavaScriptModuleSemantics } from "../domain/javascript/javascriptModuleSemanticIr.js";
|
||||||
|
import { measureJavaScriptModuleTransfer } from "../javascript/analysis/JavaScriptAnalysisTransfer.js";
|
||||||
|
import { AnalysisResourceConstraintError } from "../domain/analysisErrorCore.js";
|
||||||
|
import { projectJavaScriptSemanticFileSteps } from "../application/javascript/JavaScriptSemanticGraphBuilder.js";
|
||||||
|
import { completeJavaScriptAnalysisSteps } from "../application/javascript/JavaScriptAnalysisControl.js";
|
||||||
|
import {
|
||||||
|
serveJavaScriptAnalysisWorker,
|
||||||
|
type JavaScriptSourceWorkerAnalyzer,
|
||||||
|
} from "../javascript/analysis/JavaScriptAnalysisWorkerServer.js";
|
||||||
|
|
||||||
|
const analyze: JavaScriptSourceWorkerAnalyzer = async (
|
||||||
|
file,
|
||||||
|
nodeBudget,
|
||||||
|
callbacks,
|
||||||
|
) => {
|
||||||
|
if (!file.text.included)
|
||||||
|
throw new TypeError("JavaScript worker requires decoded source text");
|
||||||
|
await callbacks.phase("parse");
|
||||||
|
let parsed = parseJavaScriptSource(file.text.value, file.path);
|
||||||
|
if (parsed === null) {
|
||||||
|
await callbacks.checkpoint(failedJavaScriptStaticAnalysis());
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
await callbacks.phase("static");
|
||||||
|
const receivers = await completeJavaScriptAnalysisSteps(
|
||||||
|
classifyParsedJavaScriptOpenReceiversSteps(parsed),
|
||||||
|
);
|
||||||
|
const facts = await completeJavaScriptAnalysisSteps(
|
||||||
|
analyzeParsedJavaScriptStaticSourceSteps(
|
||||||
|
file.text.value,
|
||||||
|
parsed,
|
||||||
|
receivers,
|
||||||
|
),
|
||||||
|
);
|
||||||
|
const resultBudgetBytes = await callbacks.checkpoint(facts);
|
||||||
|
if (facts.parse_status === "failed") return null;
|
||||||
|
if (resultBudgetBytes === 0)
|
||||||
|
throw new AnalysisResourceConstraintError(
|
||||||
|
"analyze_javascript_application",
|
||||||
|
"memory",
|
||||||
|
`The parent reserved its remaining result capacity for static observations; semantic analysis of ${file.path} was not started`,
|
||||||
|
{ semantic_result_budget_bytes: 0 },
|
||||||
|
);
|
||||||
|
await callbacks.phase("semantic");
|
||||||
|
const ir = await completeJavaScriptAnalysisSteps(
|
||||||
|
analyzeParsedJavaScriptSemanticsSteps(parsed),
|
||||||
|
);
|
||||||
|
parsed = null;
|
||||||
|
const module = projectJavaScriptModuleSemantics(ir);
|
||||||
|
const measured = measureJavaScriptModuleTransfer(module);
|
||||||
|
if (measured.retentionBytes > resultBudgetBytes)
|
||||||
|
throw new AnalysisResourceConstraintError(
|
||||||
|
"analyze_javascript_application",
|
||||||
|
"memory",
|
||||||
|
`Completed module facts for ${file.path} exceed the parent's offered result budget; static facts were retained`,
|
||||||
|
{
|
||||||
|
semantic_result_budget_bytes: resultBudgetBytes,
|
||||||
|
module_transfer_bytes: measured.bytes,
|
||||||
|
},
|
||||||
|
);
|
||||||
|
await callbacks.phase("projection");
|
||||||
|
const projection = await completeJavaScriptAnalysisSteps(
|
||||||
|
projectJavaScriptSemanticFileSteps(file, ir, nodeBudget),
|
||||||
|
);
|
||||||
|
return { module, projection };
|
||||||
|
};
|
||||||
|
|
||||||
|
try {
|
||||||
|
await serveJavaScriptAnalysisWorker(analyze);
|
||||||
|
} catch (cause: unknown) {
|
||||||
|
process.stderr.write(
|
||||||
|
`${cause instanceof Error ? cause.message : String(cause)}\n`,
|
||||||
|
);
|
||||||
|
process.exitCode = 1;
|
||||||
|
}
|
||||||
@@ -1,4 +1,5 @@
|
|||||||
import { jsonValueSchema, type JsonValue } from "./jsonValue.js";
|
import { jsonValueSchema, type JsonValue } from "./jsonValue.js";
|
||||||
|
import { isImmutableJsonSnapshot } from "./immutableJson.js";
|
||||||
|
|
||||||
import {
|
import {
|
||||||
analysisErrorCategory,
|
analysisErrorCategory,
|
||||||
@@ -71,7 +72,9 @@ export const projectAnalysisError = (
|
|||||||
...(error.partialObservation === undefined
|
...(error.partialObservation === undefined
|
||||||
? {}
|
? {}
|
||||||
: {
|
: {
|
||||||
partial_observation: jsonValueSchema.parse(error.partialObservation),
|
partial_observation: isImmutableJsonSnapshot(error.partialObservation)
|
||||||
|
? error.partialObservation
|
||||||
|
: jsonValueSchema.parse(error.partialObservation),
|
||||||
}),
|
}),
|
||||||
};
|
};
|
||||||
return {
|
return {
|
||||||
|
|||||||
@@ -0,0 +1,56 @@
|
|||||||
|
import { z } from "zod";
|
||||||
|
import type { JsonValue } from "../jsonValue.js";
|
||||||
|
import type { ApplicationCoverage } from "./javascriptApplicationEvidenceSchemas.js";
|
||||||
|
|
||||||
|
/** Shared operational defaults for the isolated source-analysis boundary. */
|
||||||
|
export const JAVASCRIPT_ANALYSIS_RESOURCE_DEFAULTS = {
|
||||||
|
heapMb: 1024,
|
||||||
|
timeoutMs: 300_000,
|
||||||
|
} as const;
|
||||||
|
|
||||||
|
/** Project observed execution limits without inventing unmeasured values. */
|
||||||
|
export const javaScriptAnalysisCoverageLimits = (
|
||||||
|
limits: Readonly<Record<string, JsonValue>> | null,
|
||||||
|
): ApplicationCoverage["limits"] => {
|
||||||
|
const result: ApplicationCoverage["limits"] = [];
|
||||||
|
for (const key of [
|
||||||
|
"worker_heap_limit_bytes",
|
||||||
|
"worker_timeout_ms",
|
||||||
|
"parent_result_budget_bytes",
|
||||||
|
"semantic_result_budget_bytes",
|
||||||
|
"transfer_record_bytes",
|
||||||
|
"transfer_file_bytes",
|
||||||
|
]) {
|
||||||
|
const value = limits?.[key];
|
||||||
|
if (typeof value !== "number" || !Number.isSafeInteger(value) || value < 0)
|
||||||
|
continue;
|
||||||
|
result.push({
|
||||||
|
name: `javascript_analysis_${key}`,
|
||||||
|
value,
|
||||||
|
unit: key.endsWith("_ms") ? "milliseconds" : "bytes",
|
||||||
|
});
|
||||||
|
}
|
||||||
|
return result;
|
||||||
|
};
|
||||||
|
|
||||||
|
/** Optional operational choices; omission keeps them out of Evidence parameters. */
|
||||||
|
export const javaScriptAnalysisResourceControlsSchema = z.strictObject({
|
||||||
|
max_heap_mb: z
|
||||||
|
.number()
|
||||||
|
.int()
|
||||||
|
.min(128)
|
||||||
|
.max(16_384)
|
||||||
|
.optional()
|
||||||
|
.describe(
|
||||||
|
"Maximum V8 old-space heap in MiB for the isolated JavaScript source analyzer; defaults to 1024. This does not change the CLI/MCP process heap.",
|
||||||
|
),
|
||||||
|
analysis_timeout_ms: z
|
||||||
|
.number()
|
||||||
|
.int()
|
||||||
|
.min(1)
|
||||||
|
.max(2_147_483_647)
|
||||||
|
.optional()
|
||||||
|
.describe(
|
||||||
|
"Maximum startup, analysis and transfer time per JavaScript source, in milliseconds; defaults to 300000. On timeout, return collected partial Evidence and stop the analysis.",
|
||||||
|
),
|
||||||
|
});
|
||||||
@@ -15,6 +15,7 @@ import { prefixedDigestSchema } from "../digests.js";
|
|||||||
import { integrityContradictionSchema } from "../artifactGraph.js";
|
import { integrityContradictionSchema } from "../artifactGraph.js";
|
||||||
import { artifactIntegrityPolicySchema } from "../artifactIntegrityPolicy.js";
|
import { artifactIntegrityPolicySchema } from "../artifactIntegrityPolicy.js";
|
||||||
import { localPathStringSchema } from "../localPath.js";
|
import { localPathStringSchema } from "../localPath.js";
|
||||||
|
import { javaScriptAnalysisResourceControlsSchema } from "./javascriptAnalysisResourceControls.js";
|
||||||
|
|
||||||
const countSchema = z.number().int().min(0);
|
const countSchema = z.number().int().min(0);
|
||||||
|
|
||||||
@@ -23,6 +24,7 @@ export const analyzeJavaScriptApplicationInputSchema = z.strictObject({
|
|||||||
input_path: localPathStringSchema,
|
input_path: localPathStringSchema,
|
||||||
format: z.enum(["auto", "asar", "directory"]).default("auto"),
|
format: z.enum(["auto", "asar", "directory"]).default("auto"),
|
||||||
integrity_policy: artifactIntegrityPolicySchema,
|
integrity_policy: artifactIntegrityPolicySchema,
|
||||||
|
...javaScriptAnalysisResourceControlsSchema.shape,
|
||||||
});
|
});
|
||||||
|
|
||||||
/** Compact counts for the high-level Electron architecture/security surface. */
|
/** Compact counts for the high-level Electron architecture/security surface. */
|
||||||
|
|||||||
@@ -8,7 +8,10 @@ import { canonicalJson } from "../comparisonSemantics.js";
|
|||||||
import { digestCanonicalValue } from "../canonicalDigest.js";
|
import { digestCanonicalValue } from "../canonicalDigest.js";
|
||||||
import { uniqueSorted } from "../canonicalOrdering.js";
|
import { uniqueSorted } from "../canonicalOrdering.js";
|
||||||
import { compareUnicodeCodePoints } from "../unicodeCodePointOrder.js";
|
import { compareUnicodeCodePoints } from "../unicodeCodePointOrder.js";
|
||||||
import { freezeOwnedJsonSnapshotSteps } from "../immutableJson.js";
|
import {
|
||||||
|
freezeJsonSnapshot,
|
||||||
|
freezeOwnedJsonSnapshotSteps,
|
||||||
|
} from "../immutableJson.js";
|
||||||
|
|
||||||
import type {
|
import type {
|
||||||
ApplicationGraphEvidence,
|
ApplicationGraphEvidence,
|
||||||
@@ -449,6 +452,10 @@ export const createJavaScriptApplicationNode = (
|
|||||||
input: unknown,
|
input: unknown,
|
||||||
): ApplicationNode => {
|
): ApplicationNode => {
|
||||||
const parsed = applicationNodeInputSchema.parse(input);
|
const parsed = applicationNodeInputSchema.parse(input);
|
||||||
|
// These properties are newly parsed and exclusively owned. Authenticate
|
||||||
|
// them before the observation/node schemas and final graph validate again.
|
||||||
|
for (const observation of parsed.observations)
|
||||||
|
freezeJsonSnapshot(observation.properties);
|
||||||
const identity = normalizeIdentity(parsed.identity);
|
const identity = normalizeIdentity(parsed.identity);
|
||||||
const identifier = nodeId({ kind: parsed.kind, identity });
|
const identifier = nodeId({ kind: parsed.kind, identity });
|
||||||
const observations = parsed.observations
|
const observations = parsed.observations
|
||||||
@@ -479,6 +486,7 @@ export const createJavaScriptApplicationEdge = (
|
|||||||
input: unknown,
|
input: unknown,
|
||||||
): ApplicationEdge => {
|
): ApplicationEdge => {
|
||||||
const parsed = applicationEdgeInputSchema.parse(input);
|
const parsed = applicationEdgeInputSchema.parse(input);
|
||||||
|
freezeJsonSnapshot(parsed.properties);
|
||||||
const semantic = {
|
const semantic = {
|
||||||
...parsed,
|
...parsed,
|
||||||
evidence: normalizeEvidence(parsed.evidence),
|
evidence: normalizeEvidence(parsed.evidence),
|
||||||
|
|||||||
@@ -0,0 +1,42 @@
|
|||||||
|
import type { JavaScriptStaticAnalysis } from "./javascriptStaticAnalysisTypes.js";
|
||||||
|
|
||||||
|
/** Forecast static-fact expansion into observations, nodes and relationships. */
|
||||||
|
export const estimateJavaScriptStaticApplicationProjection = (
|
||||||
|
facts: JavaScriptStaticAnalysis,
|
||||||
|
checkpointBytes: number,
|
||||||
|
sourcePath: string,
|
||||||
|
): number => {
|
||||||
|
// A finding can produce a node, a separate observation and several edges.
|
||||||
|
// Allow for identifiers, locations, authority, coverage and schema copies,
|
||||||
|
// independently of the producer's compact checkpoint representation.
|
||||||
|
const findings =
|
||||||
|
facts.references.length +
|
||||||
|
facts.endpoints.length +
|
||||||
|
facts.storage.length +
|
||||||
|
facts.role_paths.length +
|
||||||
|
facts.source_map_urls.length +
|
||||||
|
facts.electron.browser_windows.length +
|
||||||
|
facts.electron.context_bridge_apis.length +
|
||||||
|
facts.electron.ipc.length +
|
||||||
|
facts.electron.sender_validations.length +
|
||||||
|
facts.electron.utility_processes.length +
|
||||||
|
facts.electron.native_addon_bindings.length;
|
||||||
|
let records = findings * 8;
|
||||||
|
let repeatedBytes = 0;
|
||||||
|
for (const registration of facts.bundler_registrations) {
|
||||||
|
records +=
|
||||||
|
3 +
|
||||||
|
registration.modules.length * 3 +
|
||||||
|
registration.entry_module_keys.length * 3 +
|
||||||
|
registration.async_chunk_keys.length * 3;
|
||||||
|
// Every module observation and containment edge repeats its chunk keys.
|
||||||
|
repeatedBytes +=
|
||||||
|
Buffer.byteLength(JSON.stringify(registration.chunk_keys)) *
|
||||||
|
registration.modules.length *
|
||||||
|
2;
|
||||||
|
}
|
||||||
|
for (const binding of facts.electron.native_addon_bindings)
|
||||||
|
records += binding.members.length * 3;
|
||||||
|
const metadataBytes = 2048 + Buffer.byteLength(sourcePath) * 4;
|
||||||
|
return (checkpointBytes * 4 + repeatedBytes + records * metadataBytes) * 8;
|
||||||
|
};
|
||||||
@@ -0,0 +1,36 @@
|
|||||||
|
import type { JavaScriptSemanticIr } from "./javascriptSemanticIr.js";
|
||||||
|
|
||||||
|
/** Module facts needed after file-local semantic projection has finished. */
|
||||||
|
export type JavaScriptModuleSemanticIr = Pick<
|
||||||
|
JavaScriptSemanticIr,
|
||||||
|
| "scopes"
|
||||||
|
| "bindings"
|
||||||
|
| "callables"
|
||||||
|
| "moduleLinks"
|
||||||
|
| "coverage"
|
||||||
|
| "limitations"
|
||||||
|
>;
|
||||||
|
|
||||||
|
/** Retain facts consumed by source-module and export-return composition. */
|
||||||
|
export const projectJavaScriptModuleSemantics = (
|
||||||
|
ir: JavaScriptSemanticIr,
|
||||||
|
): JavaScriptModuleSemanticIr => {
|
||||||
|
const scopes = ir.scopes.filter(({ kind }) => kind === "program");
|
||||||
|
const programScopeId = scopes[0]?.scopeId;
|
||||||
|
const localNames = new Set(ir.moduleLinks.map(({ localName }) => localName));
|
||||||
|
const callableIds = new Set(
|
||||||
|
ir.moduleLinks.map(({ callableId }) => callableId),
|
||||||
|
);
|
||||||
|
return {
|
||||||
|
scopes,
|
||||||
|
bindings: ir.bindings.filter(
|
||||||
|
({ name, scopeId }) => scopeId === programScopeId && localNames.has(name),
|
||||||
|
),
|
||||||
|
callables: ir.callables.filter(({ callableId }) =>
|
||||||
|
callableIds.has(callableId),
|
||||||
|
),
|
||||||
|
moduleLinks: ir.moduleLinks,
|
||||||
|
coverage: ir.coverage,
|
||||||
|
limitations: ir.limitations,
|
||||||
|
};
|
||||||
|
};
|
||||||
@@ -0,0 +1,18 @@
|
|||||||
|
import type {
|
||||||
|
JavaScriptSemanticEvidenceContext,
|
||||||
|
JavaScriptSemanticFingerprint,
|
||||||
|
JavaScriptSemanticGraphNode,
|
||||||
|
JavaScriptSemanticGraphRelation,
|
||||||
|
JavaScriptSemanticGraphUnknown,
|
||||||
|
} from "./javascriptSemanticGraphSchemas.js";
|
||||||
|
|
||||||
|
/** File-local facts transported before binding them to the containing application graph. */
|
||||||
|
export interface JavaScriptSemanticFileProjection {
|
||||||
|
readonly roots: readonly string[];
|
||||||
|
readonly evidenceContexts: readonly JavaScriptSemanticEvidenceContext[];
|
||||||
|
readonly nodes: readonly JavaScriptSemanticGraphNode[];
|
||||||
|
readonly relations: readonly JavaScriptSemanticGraphRelation[];
|
||||||
|
readonly unknowns: readonly JavaScriptSemanticGraphUnknown[];
|
||||||
|
readonly fingerprints: readonly JavaScriptSemanticFingerprint[];
|
||||||
|
readonly truncated: boolean;
|
||||||
|
}
|
||||||
@@ -19,6 +19,7 @@ import {
|
|||||||
JAVASCRIPT_SEMANTIC_RELATION_FAMILIES,
|
JAVASCRIPT_SEMANTIC_RELATION_FAMILIES,
|
||||||
JAVASCRIPT_SEMANTIC_RELATION_FAMILY,
|
JAVASCRIPT_SEMANTIC_RELATION_FAMILY,
|
||||||
javaScriptSemanticFingerprintInputSchema,
|
javaScriptSemanticFingerprintInputSchema,
|
||||||
|
javaScriptSemanticEvidenceContextSchema,
|
||||||
javaScriptSemanticGraphInputSchema,
|
javaScriptSemanticGraphInputSchema,
|
||||||
javaScriptSemanticGraphRecordSchema,
|
javaScriptSemanticGraphRecordSchema,
|
||||||
javaScriptSemanticNodeInputSchema,
|
javaScriptSemanticNodeInputSchema,
|
||||||
@@ -109,6 +110,33 @@ export class JavaScriptSemanticEvidenceContextRegistry {
|
|||||||
this.#contexts.clear();
|
this.#contexts.clear();
|
||||||
this.#idsByCanonicalContext.clear();
|
this.#idsByCanonicalContext.clear();
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/** Validate and intern provenance decoded in this realm from an analysis worker. */
|
||||||
|
adopt(input: unknown): void {
|
||||||
|
const context = javaScriptSemanticEvidenceContextSchema.parse(input);
|
||||||
|
const { context_id: identifier, ...value } = context;
|
||||||
|
if (identifier !== evidenceContextId(value))
|
||||||
|
throw new TypeError(
|
||||||
|
"Transferred semantic evidence context identifier is stale",
|
||||||
|
);
|
||||||
|
const canonical = canonicalJson(
|
||||||
|
value,
|
||||||
|
"JavaScript semantic evidence context",
|
||||||
|
);
|
||||||
|
const existing = this.#contexts.get(identifier);
|
||||||
|
if (existing !== undefined) {
|
||||||
|
const { context_id: _existingIdentifier, ...existingValue } = existing;
|
||||||
|
if (
|
||||||
|
canonical !==
|
||||||
|
canonicalJson(existingValue, "JavaScript semantic evidence context")
|
||||||
|
)
|
||||||
|
throw new TypeError(
|
||||||
|
"Transferred semantic evidence context digest collision",
|
||||||
|
);
|
||||||
|
}
|
||||||
|
this.#contexts.set(identifier, context);
|
||||||
|
this.#idsByCanonicalContext.set(canonical, identifier);
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
/** Derive a semantic node ID from its kind and exact artifact identity. */
|
/** Derive a semantic node ID from its kind and exact artifact identity. */
|
||||||
|
|||||||
@@ -0,0 +1,40 @@
|
|||||||
|
import type { AnalysisError } from "../analysisErrorBase.js";
|
||||||
|
import type { AnalysisResourceConstraintError } from "../analysisErrorCore.js";
|
||||||
|
import type { Result } from "../result.js";
|
||||||
|
import type { JavaScriptArtifactFile } from "./javascriptArtifactFiles.js";
|
||||||
|
import type { JavaScriptStaticAnalysis } from "./javascriptStaticAnalysisTypes.js";
|
||||||
|
import type { JavaScriptModuleSemanticIr } from "./javascriptModuleSemanticIr.js";
|
||||||
|
import type { JavaScriptSemanticFileProjection } from "./javascriptSemanticFileProjection.js";
|
||||||
|
|
||||||
|
/** Portable facts collected from one inert source. */
|
||||||
|
export interface JavaScriptAnalyzedSource {
|
||||||
|
readonly javascript: JavaScriptStaticAnalysis;
|
||||||
|
readonly module: JavaScriptModuleSemanticIr | null;
|
||||||
|
readonly projection: JavaScriptSemanticFileProjection | null;
|
||||||
|
readonly applicationProjectionFailure?: AnalysisResourceConstraintError;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Execution failure alongside facts already collected by the consumer. */
|
||||||
|
export interface JavaScriptSourceAnalysisFailure {
|
||||||
|
readonly error: AnalysisError;
|
||||||
|
readonly javascript: JavaScriptStaticAnalysis | null;
|
||||||
|
readonly module: JavaScriptModuleSemanticIr | null;
|
||||||
|
readonly projection: JavaScriptSemanticFileProjection | null;
|
||||||
|
readonly applicationProjectionFailure?: AnalysisResourceConstraintError;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Operational choices for source analysis, independent of transport framing. */
|
||||||
|
export interface JavaScriptSourceAnalysisLimits {
|
||||||
|
readonly heapMb: number;
|
||||||
|
readonly timeoutMs: number;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Sequential source-analysis authority with independently verified cleanup. */
|
||||||
|
export interface JavaScriptSourceAnalysisPort {
|
||||||
|
analyze(
|
||||||
|
file: JavaScriptArtifactFile,
|
||||||
|
nodeBudget: number,
|
||||||
|
signal?: AbortSignal,
|
||||||
|
): Promise<Result<JavaScriptAnalyzedSource, JavaScriptSourceAnalysisFailure>>;
|
||||||
|
close(): Promise<void>;
|
||||||
|
}
|
||||||
@@ -78,6 +78,8 @@ export function* jsonValueValidationSteps(
|
|||||||
}
|
}
|
||||||
|
|
||||||
const preserveJsonValue = (value: JsonValue): JsonValue => {
|
const preserveJsonValue = (value: JsonValue): JsonValue => {
|
||||||
|
const immutableCandidate: unknown = value;
|
||||||
|
if (isImmutableJsonSnapshot(immutableCandidate)) return value;
|
||||||
if (Array.isArray(value)) return value.map(preserveJsonValue);
|
if (Array.isArray(value)) return value.map(preserveJsonValue);
|
||||||
if (value !== null && typeof value === "object")
|
if (value !== null && typeof value === "object")
|
||||||
return Object.fromEntries<JsonValue>(
|
return Object.fromEntries<JsonValue>(
|
||||||
|
|||||||
@@ -0,0 +1,414 @@
|
|||||||
|
import { createHash } from "node:crypto";
|
||||||
|
import { createReadStream } from "node:fs";
|
||||||
|
import { open, stat } from "node:fs/promises";
|
||||||
|
import { z } from "zod";
|
||||||
|
import { AnalysisResourceConstraintError } from "../../domain/analysisErrorCore.js";
|
||||||
|
|
||||||
|
import type { JavaScriptArtifactFile } from "../../domain/javascript/javascriptArtifactFiles.js";
|
||||||
|
import type { JavaScriptModuleSemanticIr } from "../../domain/javascript/javascriptModuleSemanticIr.js";
|
||||||
|
import type { JavaScriptSemanticFileProjection } from "../../domain/javascript/javascriptSemanticFileProjection.js";
|
||||||
|
import { digestSchema } from "../../domain/digests.js";
|
||||||
|
import {
|
||||||
|
javaScriptWorkerGraphRecordSchema,
|
||||||
|
javaScriptWorkerModuleHeaderSchema,
|
||||||
|
javaScriptWorkerModuleRecordSchema,
|
||||||
|
javaScriptWorkerProjectionHeaderSchema,
|
||||||
|
} from "./JavaScriptAnalysisWorkerSchemas.js";
|
||||||
|
|
||||||
|
// Budgets bound transfer representation, independently of source length.
|
||||||
|
/** Record bound whose transient decoding expansion is reserved by the consumer. */
|
||||||
|
export const JAVASCRIPT_ANALYSIS_TRANSFER_RECORD_BYTES = 8 * 1024 * 1024;
|
||||||
|
const FRAME_BYTES = JAVASCRIPT_ANALYSIS_TRANSFER_RECORD_BYTES;
|
||||||
|
const TRANSFER_BYTES = 512 * 1024 * 1024;
|
||||||
|
const BATCH_BYTES = 1024 * 1024;
|
||||||
|
const recordSchema = z.discriminatedUnion("kind", [
|
||||||
|
z.strictObject({
|
||||||
|
kind: z.literal("projection-header"),
|
||||||
|
value: javaScriptWorkerProjectionHeaderSchema,
|
||||||
|
}),
|
||||||
|
z.strictObject({
|
||||||
|
kind: z.literal("module-header"),
|
||||||
|
value: javaScriptWorkerModuleHeaderSchema,
|
||||||
|
}),
|
||||||
|
...javaScriptWorkerGraphRecordSchema.options,
|
||||||
|
...javaScriptWorkerModuleRecordSchema.options,
|
||||||
|
]);
|
||||||
|
|
||||||
|
/** Complete-file coordinates authenticated before adopting any worker facts. */
|
||||||
|
export const javaScriptAnalysisTransferDescriptorSchema = z.strictObject({
|
||||||
|
bytes: z.number().int().nonnegative().max(TRANSFER_BYTES),
|
||||||
|
sha256: digestSchema,
|
||||||
|
records: z.number().int().nonnegative(),
|
||||||
|
module_bytes: z.number().int().nonnegative(),
|
||||||
|
module_records: z.number().int().nonnegative(),
|
||||||
|
graph_bytes: z.number().int().nonnegative(),
|
||||||
|
graph_records: z.number().int().nonnegative(),
|
||||||
|
});
|
||||||
|
type Descriptor = z.output<typeof javaScriptAnalysisTransferDescriptorSchema>;
|
||||||
|
|
||||||
|
/** Conservative retained-object and sealing expansion for validated transfer rows. */
|
||||||
|
export const javaScriptTransferRetentionBytes = (
|
||||||
|
bytes: number,
|
||||||
|
records: number,
|
||||||
|
): number => bytes * 8 + records * 64;
|
||||||
|
|
||||||
|
const encodeTransferRecord = (input: unknown, previousBytes: number) => {
|
||||||
|
const record = recordSchema.parse(input);
|
||||||
|
const line = `${JSON.stringify(record)}\n`;
|
||||||
|
const size = Buffer.byteLength(line);
|
||||||
|
if (size > FRAME_BYTES || previousBytes + size > TRANSFER_BYTES)
|
||||||
|
throw new AnalysisResourceConstraintError(
|
||||||
|
"analyze_javascript_application",
|
||||||
|
"transport",
|
||||||
|
size > FRAME_BYTES
|
||||||
|
? "A complete JavaScript fact exceeds the transfer record byte budget"
|
||||||
|
: "Complete JavaScript file facts exceed the transfer byte budget",
|
||||||
|
{
|
||||||
|
transfer_record_bytes: FRAME_BYTES,
|
||||||
|
transfer_file_bytes: TRANSFER_BYTES,
|
||||||
|
record_bytes: size,
|
||||||
|
transfer_bytes_at_least: previousBytes + size,
|
||||||
|
},
|
||||||
|
);
|
||||||
|
return { line, size, kind: record.kind };
|
||||||
|
};
|
||||||
|
|
||||||
|
function* moduleTransferRecords(
|
||||||
|
module: JavaScriptModuleSemanticIr,
|
||||||
|
): Generator<unknown> {
|
||||||
|
yield {
|
||||||
|
kind: "module-header",
|
||||||
|
value: { coverage: module.coverage, limitations: module.limitations },
|
||||||
|
};
|
||||||
|
for (const value of module.scopes) yield { kind: "scope", value };
|
||||||
|
for (const value of module.bindings) yield { kind: "binding", value };
|
||||||
|
for (const value of module.callables) yield { kind: "callable", value };
|
||||||
|
for (const value of module.moduleLinks) yield { kind: "module-link", value };
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Measure actual validated module rows before allocating an unadmittable graph. */
|
||||||
|
export const measureJavaScriptModuleTransfer = (
|
||||||
|
module: JavaScriptModuleSemanticIr,
|
||||||
|
) => {
|
||||||
|
let bytes = 0;
|
||||||
|
let records = 0;
|
||||||
|
for (const record of moduleTransferRecords(module)) {
|
||||||
|
bytes += encodeTransferRecord(record, bytes).size;
|
||||||
|
records += 1;
|
||||||
|
}
|
||||||
|
return {
|
||||||
|
bytes,
|
||||||
|
records,
|
||||||
|
retentionBytes: javaScriptTransferRetentionBytes(bytes, records),
|
||||||
|
};
|
||||||
|
};
|
||||||
|
|
||||||
|
/** Encode validated facts in bounded records, without serializing a whole graph or IR. */
|
||||||
|
export const writeJavaScriptAnalysisTransfer = async (
|
||||||
|
path: string,
|
||||||
|
module: JavaScriptModuleSemanticIr,
|
||||||
|
projection: JavaScriptSemanticFileProjection,
|
||||||
|
): Promise<Descriptor> => {
|
||||||
|
const output = await open(path, "wx", 0o600);
|
||||||
|
const digest = createHash("sha256");
|
||||||
|
let bytes = 0;
|
||||||
|
let records = 0;
|
||||||
|
let moduleBytes = 0;
|
||||||
|
let moduleRecords = 0;
|
||||||
|
let graphBytes = 0;
|
||||||
|
let graphRecords = 0;
|
||||||
|
let batch: string[] = [];
|
||||||
|
let batchBytes = 0;
|
||||||
|
const flush = async (): Promise<void> => {
|
||||||
|
if (batch.length === 0) return;
|
||||||
|
const buffer = Buffer.from(batch.join(""));
|
||||||
|
// FileHandle.write can be short; writeFile completes the buffer.
|
||||||
|
await output.writeFile(buffer);
|
||||||
|
digest.update(buffer);
|
||||||
|
batch = [];
|
||||||
|
batchBytes = 0;
|
||||||
|
};
|
||||||
|
const append = async (input: unknown): Promise<void> => {
|
||||||
|
const { line, size, kind } = encodeTransferRecord(input, bytes);
|
||||||
|
bytes += size;
|
||||||
|
records += 1;
|
||||||
|
if (
|
||||||
|
["module-header", "scope", "binding", "callable", "module-link"].includes(
|
||||||
|
kind,
|
||||||
|
)
|
||||||
|
) {
|
||||||
|
moduleBytes += size;
|
||||||
|
moduleRecords += 1;
|
||||||
|
} else {
|
||||||
|
graphBytes += size;
|
||||||
|
graphRecords += 1;
|
||||||
|
}
|
||||||
|
batch.push(line);
|
||||||
|
batchBytes += size;
|
||||||
|
if (batchBytes >= BATCH_BYTES) await flush();
|
||||||
|
};
|
||||||
|
try {
|
||||||
|
await append({
|
||||||
|
kind: "projection-header",
|
||||||
|
value: { roots: projection.roots, truncated: projection.truncated },
|
||||||
|
});
|
||||||
|
for (const record of moduleTransferRecords(module)) await append(record);
|
||||||
|
for (const value of projection.evidenceContexts)
|
||||||
|
await append({ kind: "evidence-context", value });
|
||||||
|
for (const value of projection.nodes) await append({ kind: "node", value });
|
||||||
|
for (const value of projection.relations)
|
||||||
|
await append({ kind: "relation", value });
|
||||||
|
for (const value of projection.unknowns)
|
||||||
|
await append({ kind: "unknown", value });
|
||||||
|
for (const value of projection.fingerprints)
|
||||||
|
await append({ kind: "fingerprint", value });
|
||||||
|
await flush();
|
||||||
|
return {
|
||||||
|
bytes,
|
||||||
|
sha256: digest.digest("hex"),
|
||||||
|
records,
|
||||||
|
module_bytes: moduleBytes,
|
||||||
|
module_records: moduleRecords,
|
||||||
|
graph_bytes: graphBytes,
|
||||||
|
graph_records: graphRecords,
|
||||||
|
};
|
||||||
|
} finally {
|
||||||
|
await output.close();
|
||||||
|
}
|
||||||
|
};
|
||||||
|
|
||||||
|
/** Decode producer records into realm-local, source-bound facts; sealing follows in the consumer. */
|
||||||
|
export function readJavaScriptAnalysisTransfer(
|
||||||
|
path: string,
|
||||||
|
descriptor: Descriptor,
|
||||||
|
source: Pick<JavaScriptArtifactFile, "path" | "sha256">,
|
||||||
|
signal?: AbortSignal,
|
||||||
|
): Promise<{
|
||||||
|
readonly module: JavaScriptModuleSemanticIr;
|
||||||
|
readonly projection: JavaScriptSemanticFileProjection;
|
||||||
|
}>;
|
||||||
|
/** Validate the transfer while retaining only module facts when graph admission fails. */
|
||||||
|
export function readJavaScriptAnalysisTransfer(
|
||||||
|
path: string,
|
||||||
|
descriptor: Descriptor,
|
||||||
|
source: Pick<JavaScriptArtifactFile, "path" | "sha256">,
|
||||||
|
signal: AbortSignal | undefined,
|
||||||
|
retainGraph: boolean,
|
||||||
|
): Promise<{
|
||||||
|
readonly module: JavaScriptModuleSemanticIr;
|
||||||
|
readonly projection: JavaScriptSemanticFileProjection | null;
|
||||||
|
}>;
|
||||||
|
export async function readJavaScriptAnalysisTransfer(
|
||||||
|
path: string,
|
||||||
|
descriptor: Descriptor,
|
||||||
|
source: Pick<JavaScriptArtifactFile, "path" | "sha256">,
|
||||||
|
signal?: AbortSignal,
|
||||||
|
retainGraph = true,
|
||||||
|
): Promise<{
|
||||||
|
readonly module: JavaScriptModuleSemanticIr;
|
||||||
|
readonly projection: JavaScriptSemanticFileProjection | null;
|
||||||
|
}> {
|
||||||
|
const expected = javaScriptAnalysisTransferDescriptorSchema.parse(descriptor);
|
||||||
|
if ((await stat(path)).size !== expected.bytes)
|
||||||
|
throw new TypeError(
|
||||||
|
"JavaScript analysis transfer byte count does not match",
|
||||||
|
);
|
||||||
|
const scopes: JavaScriptModuleSemanticIr["scopes"][number][] = [];
|
||||||
|
const bindings: JavaScriptModuleSemanticIr["bindings"][number][] = [];
|
||||||
|
const callables: JavaScriptModuleSemanticIr["callables"][number][] = [];
|
||||||
|
const moduleLinks: JavaScriptModuleSemanticIr["moduleLinks"][number][] = [];
|
||||||
|
const evidenceContexts: JavaScriptSemanticFileProjection["evidenceContexts"][number][] =
|
||||||
|
[];
|
||||||
|
const nodes: JavaScriptSemanticFileProjection["nodes"][number][] = [];
|
||||||
|
const relations: JavaScriptSemanticFileProjection["relations"][number][] = [];
|
||||||
|
const unknowns: JavaScriptSemanticFileProjection["unknowns"][number][] = [];
|
||||||
|
const fingerprints: JavaScriptSemanticFileProjection["fingerprints"][number][] =
|
||||||
|
[];
|
||||||
|
let projectionHeader:
|
||||||
|
| z.output<typeof javaScriptWorkerProjectionHeaderSchema>
|
||||||
|
| undefined;
|
||||||
|
let moduleHeader:
|
||||||
|
| z.output<typeof javaScriptWorkerModuleHeaderSchema>
|
||||||
|
| undefined;
|
||||||
|
const identities = new Map<string, Set<string>>();
|
||||||
|
const retainIdentity = (kind: string, identifier: string): void => {
|
||||||
|
let seen = identities.get(kind);
|
||||||
|
if (seen === undefined) {
|
||||||
|
seen = new Set();
|
||||||
|
identities.set(kind, seen);
|
||||||
|
}
|
||||||
|
if (seen.has(identifier))
|
||||||
|
throw new TypeError(
|
||||||
|
`Duplicate JavaScript transfer ${kind} identity: ${identifier}`,
|
||||||
|
);
|
||||||
|
seen.add(identifier);
|
||||||
|
};
|
||||||
|
let records = 0;
|
||||||
|
let moduleBytes = 0;
|
||||||
|
let moduleRecords = 0;
|
||||||
|
let graphBytes = 0;
|
||||||
|
let graphRecords = 0;
|
||||||
|
const digest = createHash("sha256");
|
||||||
|
const stream = createReadStream(path, {
|
||||||
|
highWaterMark: 64 * 1024,
|
||||||
|
...(signal === undefined ? {} : { signal }),
|
||||||
|
});
|
||||||
|
let bytes = 0;
|
||||||
|
let pending = "";
|
||||||
|
stream.setEncoding("utf8");
|
||||||
|
try {
|
||||||
|
for await (const chunk of stream) {
|
||||||
|
signal?.throwIfAborted();
|
||||||
|
if (typeof chunk !== "string")
|
||||||
|
throw new TypeError("JavaScript transfer stream is not UTF-8 text");
|
||||||
|
bytes += Buffer.byteLength(chunk);
|
||||||
|
if (bytes > expected.bytes)
|
||||||
|
throw new RangeError("JavaScript transfer grew while being read");
|
||||||
|
digest.update(chunk);
|
||||||
|
pending += chunk;
|
||||||
|
let newline: number;
|
||||||
|
while ((newline = pending.indexOf("\n")) >= 0) {
|
||||||
|
const line = pending.slice(0, newline);
|
||||||
|
pending = pending.slice(newline + 1);
|
||||||
|
if (Buffer.byteLength(line) + 1 > FRAME_BYTES)
|
||||||
|
throw new RangeError(
|
||||||
|
"JavaScript transfer record exceeds its byte budget",
|
||||||
|
);
|
||||||
|
const input: unknown = JSON.parse(line);
|
||||||
|
const record = recordSchema.parse(input);
|
||||||
|
if (
|
||||||
|
[
|
||||||
|
"module-header",
|
||||||
|
"scope",
|
||||||
|
"binding",
|
||||||
|
"callable",
|
||||||
|
"module-link",
|
||||||
|
].includes(record.kind)
|
||||||
|
) {
|
||||||
|
moduleBytes += Buffer.byteLength(line) + 1;
|
||||||
|
moduleRecords += 1;
|
||||||
|
} else {
|
||||||
|
graphBytes += Buffer.byteLength(line) + 1;
|
||||||
|
graphRecords += 1;
|
||||||
|
}
|
||||||
|
if (record.kind === "projection-header") {
|
||||||
|
if (records !== 0 || projectionHeader !== undefined)
|
||||||
|
throw new TypeError(
|
||||||
|
"Duplicate or late JavaScript projection header",
|
||||||
|
);
|
||||||
|
projectionHeader = record.value;
|
||||||
|
} else if (record.kind === "module-header") {
|
||||||
|
if (records !== 1 || moduleHeader !== undefined)
|
||||||
|
throw new TypeError("Duplicate or late JavaScript module header");
|
||||||
|
moduleHeader = record.value;
|
||||||
|
} else {
|
||||||
|
if (projectionHeader === undefined || moduleHeader === undefined)
|
||||||
|
throw new TypeError(
|
||||||
|
"JavaScript transfer facts precede their headers",
|
||||||
|
);
|
||||||
|
if (
|
||||||
|
!retainGraph &&
|
||||||
|
[
|
||||||
|
"evidence-context",
|
||||||
|
"node",
|
||||||
|
"relation",
|
||||||
|
"unknown",
|
||||||
|
"fingerprint",
|
||||||
|
].includes(record.kind)
|
||||||
|
) {
|
||||||
|
if (
|
||||||
|
record.kind === "node" &&
|
||||||
|
(record.value.identity.artifact_sha256 !== source.sha256 ||
|
||||||
|
record.value.identity.module_path !== source.path)
|
||||||
|
)
|
||||||
|
throw new TypeError(
|
||||||
|
"JavaScript transfer node belongs to a different artifact source",
|
||||||
|
);
|
||||||
|
records += 1;
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
switch (record.kind) {
|
||||||
|
case "scope":
|
||||||
|
retainIdentity(record.kind, record.value.scopeId);
|
||||||
|
scopes.push(record.value);
|
||||||
|
break;
|
||||||
|
case "binding":
|
||||||
|
retainIdentity(record.kind, record.value.bindingId);
|
||||||
|
bindings.push(record.value);
|
||||||
|
break;
|
||||||
|
case "callable":
|
||||||
|
retainIdentity(record.kind, record.value.callableId);
|
||||||
|
callables.push(record.value);
|
||||||
|
break;
|
||||||
|
case "module-link":
|
||||||
|
moduleLinks.push(record.value);
|
||||||
|
break;
|
||||||
|
case "evidence-context":
|
||||||
|
retainIdentity(record.kind, record.value.context_id);
|
||||||
|
evidenceContexts.push(record.value);
|
||||||
|
break;
|
||||||
|
case "node": {
|
||||||
|
retainIdentity(record.kind, record.value.node_id);
|
||||||
|
if (
|
||||||
|
record.value.identity.artifact_sha256 !== source.sha256 ||
|
||||||
|
record.value.identity.module_path !== source.path
|
||||||
|
)
|
||||||
|
throw new TypeError(
|
||||||
|
"JavaScript transfer node belongs to a different artifact source",
|
||||||
|
);
|
||||||
|
nodes.push(record.value);
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
case "relation":
|
||||||
|
retainIdentity(record.kind, record.value.relation_id);
|
||||||
|
relations.push(record.value);
|
||||||
|
break;
|
||||||
|
case "unknown":
|
||||||
|
retainIdentity(record.kind, record.value.unknown_id);
|
||||||
|
unknowns.push(record.value);
|
||||||
|
break;
|
||||||
|
case "fingerprint":
|
||||||
|
retainIdentity(record.kind, record.value.fingerprint_id);
|
||||||
|
fingerprints.push(record.value);
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
records += 1;
|
||||||
|
}
|
||||||
|
if (Buffer.byteLength(pending) + 1 > FRAME_BYTES)
|
||||||
|
throw new RangeError(
|
||||||
|
"Unterminated JavaScript transfer record exceeds its byte budget",
|
||||||
|
);
|
||||||
|
}
|
||||||
|
} finally {
|
||||||
|
stream.destroy();
|
||||||
|
}
|
||||||
|
if (
|
||||||
|
pending !== "" ||
|
||||||
|
records !== expected.records ||
|
||||||
|
bytes !== expected.bytes ||
|
||||||
|
moduleBytes !== expected.module_bytes ||
|
||||||
|
moduleRecords !== expected.module_records ||
|
||||||
|
graphBytes !== expected.graph_bytes ||
|
||||||
|
graphRecords !== expected.graph_records ||
|
||||||
|
digest.digest("hex") !== expected.sha256
|
||||||
|
)
|
||||||
|
throw new TypeError(
|
||||||
|
"JavaScript analysis transfer is incomplete or its digest does not match",
|
||||||
|
);
|
||||||
|
if (projectionHeader === undefined || moduleHeader === undefined)
|
||||||
|
throw new TypeError("JavaScript analysis transfer is missing its headers");
|
||||||
|
return {
|
||||||
|
module: { ...moduleHeader, scopes, bindings, callables, moduleLinks },
|
||||||
|
projection: retainGraph
|
||||||
|
? {
|
||||||
|
...projectionHeader,
|
||||||
|
evidenceContexts,
|
||||||
|
nodes,
|
||||||
|
relations,
|
||||||
|
unknowns,
|
||||||
|
fingerprints,
|
||||||
|
}
|
||||||
|
: null,
|
||||||
|
};
|
||||||
|
}
|
||||||
@@ -0,0 +1,726 @@
|
|||||||
|
import { createHash, randomUUID } from "node:crypto";
|
||||||
|
import { readFile, rm, stat, writeFile } from "node:fs/promises";
|
||||||
|
import { join } from "node:path";
|
||||||
|
import { StringDecoder } from "node:string_decoder";
|
||||||
|
import { fileURLToPath } from "node:url";
|
||||||
|
import { getHeapStatistics } from "node:v8";
|
||||||
|
import { z } from "zod";
|
||||||
|
import type { JavaScriptArtifactFile } from "../../domain/javascript/javascriptArtifactFiles.js";
|
||||||
|
import type { JavaScriptStaticAnalysis } from "../../domain/javascript/javascriptStaticAnalysisTypes.js";
|
||||||
|
import { estimateJavaScriptStaticApplicationProjection } from "../../domain/javascript/javascriptApplicationProjectionResources.js";
|
||||||
|
import type { JavaScriptModuleSemanticIr } from "../../domain/javascript/javascriptModuleSemanticIr.js";
|
||||||
|
import type { JavaScriptSemanticFileProjection } from "../../domain/javascript/javascriptSemanticFileProjection.js";
|
||||||
|
import type {
|
||||||
|
JavaScriptAnalyzedSource,
|
||||||
|
JavaScriptSourceAnalysisFailure,
|
||||||
|
JavaScriptSourceAnalysisLimits,
|
||||||
|
} from "../../domain/javascript/javascriptSourceAnalysis.js";
|
||||||
|
import { AnalysisError } from "../../domain/analysisErrorBase.js";
|
||||||
|
import {
|
||||||
|
AnalysisCancelledError,
|
||||||
|
AnalysisTimeoutError,
|
||||||
|
AnalysisOutputError,
|
||||||
|
AnalysisResourceConstraintError,
|
||||||
|
AnalysisCapabilityUnavailableError,
|
||||||
|
} from "../../domain/analysisErrorCore.js";
|
||||||
|
import { ProviderAdapterError } from "../../domain/providerAdapterError.js";
|
||||||
|
import { ProviderCleanupError } from "../../domain/providerCleanupError.js";
|
||||||
|
import { ok, err, type Result } from "../../domain/result.js";
|
||||||
|
import {
|
||||||
|
PrivateRuntimeRoot,
|
||||||
|
PrivateRuntimeRootUnavailableError,
|
||||||
|
} from "../../process/PrivateRuntimeRoot.js";
|
||||||
|
import {
|
||||||
|
ProviderProcessSupervisor,
|
||||||
|
spawnOwnedProviderProcess,
|
||||||
|
} from "../../process/ProviderProcess.js";
|
||||||
|
import { cleanupOwnedProcessGroup } from "../../process/ProcessOwnership.js";
|
||||||
|
import { prepareProcessOwnershipInspection } from "../../process/ProcessOwnershipObservation.js";
|
||||||
|
import {
|
||||||
|
ProviderStartupDeadline,
|
||||||
|
waitForAbortableDelay,
|
||||||
|
} from "../../process/ProviderDeadline.js";
|
||||||
|
import {
|
||||||
|
readJavaScriptAnalysisTransfer,
|
||||||
|
JAVASCRIPT_ANALYSIS_TRANSFER_RECORD_BYTES,
|
||||||
|
javaScriptTransferRetentionBytes,
|
||||||
|
} from "./JavaScriptAnalysisTransfer.js";
|
||||||
|
import { javaScriptWorkerStaticAnalysisSchema } from "./JavaScriptAnalysisWorkerSchemas.js";
|
||||||
|
import {
|
||||||
|
JAVASCRIPT_ANALYSIS_WORKER_DEFAULTS as DEFAULTS,
|
||||||
|
javaScriptWorkerMessageSchema,
|
||||||
|
javaScriptWorkerRequestSchema,
|
||||||
|
javaScriptWorkerAdmissionSchema,
|
||||||
|
type JavaScriptWorkerMessage,
|
||||||
|
} from "./JavaScriptAnalysisWorkerProtocol.js";
|
||||||
|
|
||||||
|
const OPERATION = "analyze_javascript_application";
|
||||||
|
const PROVIDER = "javascript-static-analysis";
|
||||||
|
const MIB = 1024 * 1024;
|
||||||
|
const pendingCleanup = new Set<JavaScriptAnalysisWorker>();
|
||||||
|
|
||||||
|
/** One sequential owned worker, retained between source files and closed after analysis. */
|
||||||
|
export class JavaScriptAnalysisWorker {
|
||||||
|
readonly #runId = `rea-javascript-analysis-${randomUUID()}`;
|
||||||
|
#root: PrivateRuntimeRoot | undefined;
|
||||||
|
#supervisor: ProviderProcessSupervisor | undefined;
|
||||||
|
#ready = false;
|
||||||
|
#heapLimitBytes = 0;
|
||||||
|
#frames: JavaScriptWorkerMessage[] = [];
|
||||||
|
#pending = "";
|
||||||
|
#decoder = new StringDecoder("utf8");
|
||||||
|
#protocolFailure: string | undefined;
|
||||||
|
#phase = "startup";
|
||||||
|
#lastFailure: AnalysisError | undefined;
|
||||||
|
#estimatedRetainedBytes = 0;
|
||||||
|
#applicationProjectionFailure: AnalysisResourceConstraintError | undefined;
|
||||||
|
#parentCapacityExhausted = false;
|
||||||
|
#retainedStaticBytes = 0;
|
||||||
|
#sourcesRemaining: number | undefined;
|
||||||
|
readonly #resultBudgetBytes: number;
|
||||||
|
readonly #onStdout = (chunk: Buffer | string): void => {
|
||||||
|
try {
|
||||||
|
this.#pending +=
|
||||||
|
typeof chunk === "string" ? chunk : this.#decoder.write(chunk);
|
||||||
|
let newline: number;
|
||||||
|
while ((newline = this.#pending.indexOf("\n")) >= 0) {
|
||||||
|
const line = this.#pending.slice(0, newline);
|
||||||
|
this.#pending = this.#pending.slice(newline + 1);
|
||||||
|
if (
|
||||||
|
Buffer.byteLength(line) > DEFAULTS.protocolBytes ||
|
||||||
|
this.#frames.length >= 16
|
||||||
|
)
|
||||||
|
throw new RangeError(
|
||||||
|
"JavaScript worker exceeded its control-frame budget",
|
||||||
|
);
|
||||||
|
const raw: unknown = JSON.parse(line);
|
||||||
|
this.#frames.push(javaScriptWorkerMessageSchema.parse(raw));
|
||||||
|
}
|
||||||
|
if (Buffer.byteLength(this.#pending) > DEFAULTS.protocolBytes)
|
||||||
|
throw new RangeError(
|
||||||
|
"JavaScript worker control frame is unterminated or oversized",
|
||||||
|
);
|
||||||
|
} catch (cause: unknown) {
|
||||||
|
this.#protocolFailure = message(cause);
|
||||||
|
}
|
||||||
|
};
|
||||||
|
|
||||||
|
constructor(
|
||||||
|
readonly limits: JavaScriptSourceAnalysisLimits = DEFAULTS,
|
||||||
|
readonly launcher: typeof spawnOwnedProviderProcess = spawnOwnedProviderProcess,
|
||||||
|
sourceCount?: number,
|
||||||
|
) {
|
||||||
|
if (
|
||||||
|
!Number.isSafeInteger(limits.heapMb) ||
|
||||||
|
limits.heapMb < 128 ||
|
||||||
|
limits.heapMb > 16_384 ||
|
||||||
|
!Number.isSafeInteger(limits.timeoutMs) ||
|
||||||
|
limits.timeoutMs <= 0 ||
|
||||||
|
limits.timeoutMs > 2_147_483_647
|
||||||
|
)
|
||||||
|
throw new RangeError(
|
||||||
|
"JavaScript worker requires valid heap and execution limits",
|
||||||
|
);
|
||||||
|
if (
|
||||||
|
sourceCount !== undefined &&
|
||||||
|
(!Number.isSafeInteger(sourceCount) || sourceCount < 0)
|
||||||
|
)
|
||||||
|
throw new RangeError(
|
||||||
|
"JavaScript worker source count must be a nonnegative safe integer",
|
||||||
|
);
|
||||||
|
this.#sourcesRemaining = sourceCount;
|
||||||
|
// Reserve host-side control, schema/Evidence work and a maximum record's
|
||||||
|
// transient copies. Admission also accounts for retained earlier files.
|
||||||
|
this.#resultBudgetBytes = Math.max(
|
||||||
|
0,
|
||||||
|
Math.floor(
|
||||||
|
(getHeapStatistics().heap_size_limit -
|
||||||
|
process.memoryUsage().heapUsed -
|
||||||
|
24 * JAVASCRIPT_ANALYSIS_TRANSFER_RECORD_BYTES) /
|
||||||
|
2,
|
||||||
|
),
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Analyze inert source; hard worker failure does not terminate this process. */
|
||||||
|
async analyze(
|
||||||
|
file: JavaScriptArtifactFile,
|
||||||
|
nodeBudget: number,
|
||||||
|
signal?: AbortSignal,
|
||||||
|
): Promise<
|
||||||
|
Result<JavaScriptAnalyzedSource, JavaScriptSourceAnalysisFailure>
|
||||||
|
> {
|
||||||
|
let javascript: JavaScriptStaticAnalysis | null = null;
|
||||||
|
let module: JavaScriptModuleSemanticIr | null = null;
|
||||||
|
let projection: JavaScriptSemanticFileProjection | null = null;
|
||||||
|
const id = randomUUID();
|
||||||
|
this.#applicationProjectionFailure = undefined;
|
||||||
|
this.#parentCapacityExhausted = false;
|
||||||
|
if (this.#sourcesRemaining !== undefined)
|
||||||
|
this.#sourcesRemaining = Math.max(0, this.#sourcesRemaining - 1);
|
||||||
|
let deadline: ProviderStartupDeadline | undefined;
|
||||||
|
let result: Result<
|
||||||
|
JavaScriptAnalyzedSource,
|
||||||
|
JavaScriptSourceAnalysisFailure
|
||||||
|
>;
|
||||||
|
try {
|
||||||
|
if (signal?.aborted === true) throw new AnalysisCancelledError(OPERATION);
|
||||||
|
if (!file.text.included || file.kind !== "javascript")
|
||||||
|
throw new TypeError(
|
||||||
|
"JavaScript analysis worker requires included JavaScript text",
|
||||||
|
);
|
||||||
|
// Host ownership preparation is outside the parser execution deadline.
|
||||||
|
await prepareProcessOwnershipInspection(signal);
|
||||||
|
for (const owner of pendingCleanup) await owner.close();
|
||||||
|
deadline = new ProviderStartupDeadline(this.limits.timeoutMs, signal);
|
||||||
|
await this.#start(deadline);
|
||||||
|
const root = this.#root;
|
||||||
|
const supervisor = this.#supervisor;
|
||||||
|
if (root === undefined || supervisor === undefined)
|
||||||
|
throw new TypeError("JavaScript worker resources were not acquired");
|
||||||
|
if (this.#frames.length !== 0 || this.#pending !== "")
|
||||||
|
throw new AnalysisOutputError(
|
||||||
|
OPERATION,
|
||||||
|
"JavaScript worker has unsolicited or incomplete control frames",
|
||||||
|
);
|
||||||
|
supervisor.resetOutput();
|
||||||
|
this.#phase = "source-transfer";
|
||||||
|
await writeFile(join(root.path, `${id}.source`), file.text.value, {
|
||||||
|
flag: "wx",
|
||||||
|
mode: 0o600,
|
||||||
|
signal: deadline.signal,
|
||||||
|
});
|
||||||
|
const { text: _text, ...metadata } = file;
|
||||||
|
let offeredResultBudget = 0;
|
||||||
|
const request = javaScriptWorkerRequestSchema.parse({
|
||||||
|
id,
|
||||||
|
node_budget: nodeBudget,
|
||||||
|
text_sha256: createHash("sha256").update(file.text.value).digest("hex"),
|
||||||
|
file: metadata,
|
||||||
|
});
|
||||||
|
const frame = `${JSON.stringify(request)}\n`;
|
||||||
|
if (Buffer.byteLength(frame) > DEFAULTS.protocolBytes)
|
||||||
|
throw new AnalysisResourceConstraintError(
|
||||||
|
OPERATION,
|
||||||
|
"transport",
|
||||||
|
`Source identity for ${file.path} exceeds the worker request byte budget`,
|
||||||
|
{ worker_request_bytes: DEFAULTS.protocolBytes },
|
||||||
|
);
|
||||||
|
const stdin = supervisor.launch.process.stdin;
|
||||||
|
if (stdin === undefined || stdin === null)
|
||||||
|
throw new TypeError("JavaScript worker has no writable request stream");
|
||||||
|
await new Promise<void>((resolve, reject) =>
|
||||||
|
stdin.write(frame, (error) =>
|
||||||
|
error === null || error === undefined ? resolve() : reject(error),
|
||||||
|
),
|
||||||
|
);
|
||||||
|
for (;;) {
|
||||||
|
const response = await this.#next(deadline, file.path);
|
||||||
|
if (response.kind === "ready" || response.id !== id)
|
||||||
|
throw new AnalysisOutputError(
|
||||||
|
OPERATION,
|
||||||
|
"JavaScript worker response names a different source job",
|
||||||
|
);
|
||||||
|
if (response.kind === "phase") {
|
||||||
|
this.#phase = response.phase;
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
if (response.kind === "static") {
|
||||||
|
if (javascript !== null)
|
||||||
|
throw new AnalysisOutputError(
|
||||||
|
OPERATION,
|
||||||
|
"JavaScript worker emitted a duplicate static checkpoint",
|
||||||
|
);
|
||||||
|
javascript = await this.#readCheckpoint(id, file.path, response);
|
||||||
|
if (this.#parentCapacityExhausted)
|
||||||
|
throw new AnalysisResourceConstraintError(
|
||||||
|
OPERATION,
|
||||||
|
"memory",
|
||||||
|
`The parent's result capacity was exhausted after retaining static facts for ${file.path}; remaining sources were not analyzed`,
|
||||||
|
{
|
||||||
|
...this.#limits(),
|
||||||
|
parent_result_budget_bytes: this.#resultBudgetBytes,
|
||||||
|
parent_result_budget_exhausted: true,
|
||||||
|
},
|
||||||
|
);
|
||||||
|
offeredResultBudget =
|
||||||
|
this.#applicationProjectionFailure !== undefined &&
|
||||||
|
this.#sourcesRemaining !== undefined &&
|
||||||
|
this.#sourcesRemaining > 0
|
||||||
|
? 0
|
||||||
|
: this.#remainingResultBudget();
|
||||||
|
const admission = javaScriptWorkerAdmissionSchema.parse({
|
||||||
|
id,
|
||||||
|
semantic_result_budget_bytes: offeredResultBudget,
|
||||||
|
});
|
||||||
|
await new Promise<void>((resolve, reject) =>
|
||||||
|
stdin.write(`${JSON.stringify(admission)}\n`, (error) =>
|
||||||
|
error === null || error === undefined ? resolve() : reject(error),
|
||||||
|
),
|
||||||
|
);
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
if (response.kind === "failure") {
|
||||||
|
if (response.reason === "resource") {
|
||||||
|
result = err({
|
||||||
|
javascript,
|
||||||
|
module,
|
||||||
|
projection,
|
||||||
|
error: new AnalysisResourceConstraintError(
|
||||||
|
OPERATION,
|
||||||
|
response.resource ?? "memory",
|
||||||
|
`${file.path}: ${response.message}`,
|
||||||
|
{ ...this.#limits(), ...response.limits },
|
||||||
|
),
|
||||||
|
});
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
throw new AnalysisOutputError(
|
||||||
|
OPERATION,
|
||||||
|
`${file.path} during ${this.#phase}: ${response.message}`,
|
||||||
|
);
|
||||||
|
}
|
||||||
|
if (javascript === null)
|
||||||
|
throw new AnalysisOutputError(
|
||||||
|
OPERATION,
|
||||||
|
"JavaScript worker completed before a static checkpoint",
|
||||||
|
);
|
||||||
|
if (response.descriptor === null) {
|
||||||
|
result = ok({ javascript, module: null, projection: null });
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
const descriptor = response.descriptor;
|
||||||
|
// JSON bytes alone omit object/array overhead and the later graph/Evidence
|
||||||
|
// sealing/serialization copies. Reserve a conservative expansion budget.
|
||||||
|
const moduleEstimate = javaScriptTransferRetentionBytes(
|
||||||
|
descriptor.module_bytes,
|
||||||
|
descriptor.module_records,
|
||||||
|
);
|
||||||
|
const graphEstimate = javaScriptTransferRetentionBytes(
|
||||||
|
descriptor.graph_bytes,
|
||||||
|
descriptor.graph_records,
|
||||||
|
);
|
||||||
|
const remaining = Math.min(
|
||||||
|
offeredResultBudget,
|
||||||
|
this.#remainingResultBudget(),
|
||||||
|
);
|
||||||
|
if (moduleEstimate > remaining) {
|
||||||
|
result = err({
|
||||||
|
javascript,
|
||||||
|
module: null,
|
||||||
|
projection: null,
|
||||||
|
error: new AnalysisResourceConstraintError(
|
||||||
|
OPERATION,
|
||||||
|
"memory",
|
||||||
|
`Completed module facts for ${file.path} exceed the parent's remaining result budget; static facts were retained`,
|
||||||
|
{
|
||||||
|
...this.#limits(),
|
||||||
|
parent_result_budget_bytes: this.#resultBudgetBytes,
|
||||||
|
module_transfer_bytes: descriptor.module_bytes,
|
||||||
|
},
|
||||||
|
),
|
||||||
|
});
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
const retainGraph = moduleEstimate + graphEstimate <= remaining;
|
||||||
|
this.#phase = "result-admission";
|
||||||
|
const decoded = await readJavaScriptAnalysisTransfer(
|
||||||
|
join(root.path, `${id}.records`),
|
||||||
|
descriptor,
|
||||||
|
file,
|
||||||
|
deadline.signal,
|
||||||
|
retainGraph,
|
||||||
|
);
|
||||||
|
module = decoded.module;
|
||||||
|
projection = decoded.projection;
|
||||||
|
this.#estimatedRetainedBytes +=
|
||||||
|
moduleEstimate + (retainGraph ? graphEstimate : 0);
|
||||||
|
result = retainGraph
|
||||||
|
? ok({ javascript, module, projection: decoded.projection })
|
||||||
|
: err({
|
||||||
|
javascript,
|
||||||
|
module,
|
||||||
|
projection,
|
||||||
|
error: new AnalysisResourceConstraintError(
|
||||||
|
OPERATION,
|
||||||
|
"memory",
|
||||||
|
`Semantic graph for ${file.path} exceeds the parent's remaining result budget; static and module facts were retained`,
|
||||||
|
{
|
||||||
|
...this.#limits(),
|
||||||
|
parent_result_budget_bytes: this.#resultBudgetBytes,
|
||||||
|
graph_transfer_bytes: descriptor.graph_bytes,
|
||||||
|
graph_records: descriptor.graph_records,
|
||||||
|
},
|
||||||
|
),
|
||||||
|
});
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
} catch (cause: unknown) {
|
||||||
|
const error = this.#failure(cause, file.path, deadline);
|
||||||
|
this.#lastFailure = error;
|
||||||
|
// Complete checkpoints remain readable after a child crash or cancellation.
|
||||||
|
const cleanup = await this.#stop(error);
|
||||||
|
if (
|
||||||
|
cleanup === undefined &&
|
||||||
|
javascript === null &&
|
||||||
|
this.#root !== undefined
|
||||||
|
) {
|
||||||
|
try {
|
||||||
|
javascript = await this.#readCheckpoint(id, file.path);
|
||||||
|
} catch (checkpointCause: unknown) {
|
||||||
|
void checkpointCause;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
result = err({ error: cleanup ?? error, javascript, module, projection });
|
||||||
|
} finally {
|
||||||
|
deadline?.dispose();
|
||||||
|
}
|
||||||
|
if (result.ok) this.#lastFailure = undefined;
|
||||||
|
else if (!(result.error.error instanceof ProviderCleanupError))
|
||||||
|
this.#lastFailure = result.error.error;
|
||||||
|
// Child stop must be verified before temporary files are released. On
|
||||||
|
// cleanup uncertainty, close retains this owner for a later retry.
|
||||||
|
if (this.#root !== undefined && !pendingCleanup.has(this)) {
|
||||||
|
try {
|
||||||
|
for (const suffix of ["source", "static.json", "records"])
|
||||||
|
await rm(join(this.#root.path, `${id}.${suffix}`), { force: true });
|
||||||
|
} catch (cause: unknown) {
|
||||||
|
pendingCleanup.add(this);
|
||||||
|
return err({
|
||||||
|
javascript,
|
||||||
|
module,
|
||||||
|
projection,
|
||||||
|
error: this.#cleanupError(message(cause)),
|
||||||
|
});
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if (this.#applicationProjectionFailure !== undefined)
|
||||||
|
return result.ok
|
||||||
|
? ok({
|
||||||
|
...result.value,
|
||||||
|
applicationProjectionFailure: this.#applicationProjectionFailure,
|
||||||
|
})
|
||||||
|
: err({
|
||||||
|
...result.error,
|
||||||
|
applicationProjectionFailure: this.#applicationProjectionFailure,
|
||||||
|
});
|
||||||
|
return result;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Verify process cleanup, then release the private runtime; failed close is retryable. */
|
||||||
|
async close(): Promise<void> {
|
||||||
|
const failure = await this.#stop();
|
||||||
|
if (failure !== undefined) throw failure;
|
||||||
|
if (this.#root !== undefined) {
|
||||||
|
try {
|
||||||
|
await this.#root.close();
|
||||||
|
this.#root = undefined;
|
||||||
|
} catch (cause: unknown) {
|
||||||
|
pendingCleanup.add(this);
|
||||||
|
throw this.#cleanupError(message(cause));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
pendingCleanup.delete(this);
|
||||||
|
}
|
||||||
|
|
||||||
|
async #start(deadline: ProviderStartupDeadline): Promise<void> {
|
||||||
|
if (this.#supervisor !== undefined && this.#ready) return;
|
||||||
|
this.#phase = "startup";
|
||||||
|
this.#heapLimitBytes = 0;
|
||||||
|
this.#root ??= await PrivateRuntimeRoot.create({
|
||||||
|
prefix: "rea-javascript-analysis-",
|
||||||
|
});
|
||||||
|
const environment: NodeJS.ProcessEnv = {
|
||||||
|
...process.env,
|
||||||
|
UV_THREADPOOL_SIZE: "1",
|
||||||
|
};
|
||||||
|
delete environment.NODE_OPTIONS;
|
||||||
|
const spawned = await this.launcher({
|
||||||
|
command: process.execPath,
|
||||||
|
arguments: [
|
||||||
|
`--max-old-space-size=${String(this.limits.heapMb)}`,
|
||||||
|
"--max-semi-space-size=8",
|
||||||
|
"--v8-pool-size=1",
|
||||||
|
"--expose-gc",
|
||||||
|
"--stack_size=2048",
|
||||||
|
fileURLToPath(import.meta.resolve("#javascript-analysis-worker")),
|
||||||
|
],
|
||||||
|
runId: `${this.#runId}-${randomUUID()}`,
|
||||||
|
cwd: this.#root.path,
|
||||||
|
hostEnvironment: environment,
|
||||||
|
stdin: "pipe",
|
||||||
|
signal: deadline.signal,
|
||||||
|
});
|
||||||
|
this.#supervisor = new ProviderProcessSupervisor(
|
||||||
|
{
|
||||||
|
...spawned,
|
||||||
|
ownsProcessLifetime: true,
|
||||||
|
cleanup:
|
||||||
|
spawned.cleanup ??
|
||||||
|
(() => cleanupOwnedProcessGroup(spawned.ownership)),
|
||||||
|
},
|
||||||
|
{ captureStdout: false, maxDiagnosticBytes: 256 * 1024 },
|
||||||
|
);
|
||||||
|
spawned.process.stdout?.on("data", this.#onStdout);
|
||||||
|
const ready = await this.#next(deadline, "worker startup");
|
||||||
|
if (ready.kind !== "ready")
|
||||||
|
throw new AnalysisOutputError(
|
||||||
|
OPERATION,
|
||||||
|
"JavaScript worker emitted facts before its startup handshake",
|
||||||
|
);
|
||||||
|
this.#heapLimitBytes = ready.heap_limit_bytes;
|
||||||
|
this.#ready = true;
|
||||||
|
}
|
||||||
|
|
||||||
|
async #next(
|
||||||
|
deadline: ProviderStartupDeadline,
|
||||||
|
path: string,
|
||||||
|
): Promise<JavaScriptWorkerMessage> {
|
||||||
|
for (;;) {
|
||||||
|
if (deadline.interruption === "cancelled")
|
||||||
|
throw new AnalysisCancelledError(OPERATION);
|
||||||
|
if (deadline.interruption === "timeout")
|
||||||
|
throw new AnalysisTimeoutError(OPERATION, this.limits.timeoutMs);
|
||||||
|
if (this.#protocolFailure !== undefined)
|
||||||
|
throw new AnalysisOutputError(OPERATION, this.#protocolFailure);
|
||||||
|
const frame = this.#frames.shift();
|
||||||
|
if (frame !== undefined) return frame;
|
||||||
|
const snapshot = this.#supervisor?.snapshot();
|
||||||
|
if (
|
||||||
|
snapshot !== undefined &&
|
||||||
|
(snapshot.exitCode !== undefined || snapshot.signal !== undefined)
|
||||||
|
) {
|
||||||
|
// Exit is not complete output: let any final control frame drain first.
|
||||||
|
if (!(await this.#supervisor?.waitForOutputClose(10))) continue;
|
||||||
|
if (this.#frames.length !== 0) continue;
|
||||||
|
const drained = this.#supervisor?.snapshot() ?? snapshot;
|
||||||
|
throw /heap out of memory|heap limit|allocation failed/iu.test(
|
||||||
|
drained.stderr.text,
|
||||||
|
)
|
||||||
|
? new AnalysisResourceConstraintError(
|
||||||
|
OPERATION,
|
||||||
|
"memory",
|
||||||
|
`Owned JavaScript worker exhausted its heap while analyzing ${path} during ${this.#phase}`,
|
||||||
|
this.#limits(),
|
||||||
|
{ capturedOutput: this.#captured() },
|
||||||
|
)
|
||||||
|
: new ProviderAdapterError(PROVIDER, OPERATION, {
|
||||||
|
capturedOutput: this.#captured(),
|
||||||
|
diagnostics: {
|
||||||
|
source_path: path,
|
||||||
|
phase: this.#phase,
|
||||||
|
exit_code: drained.exitCode ?? null,
|
||||||
|
signal: drained.signal ?? null,
|
||||||
|
},
|
||||||
|
});
|
||||||
|
}
|
||||||
|
await waitForAbortableDelay(10, deadline.signal);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
async #readCheckpoint(
|
||||||
|
id: string,
|
||||||
|
sourcePath: string,
|
||||||
|
expected?: { readonly bytes: number; readonly sha256: string },
|
||||||
|
): Promise<JavaScriptStaticAnalysis> {
|
||||||
|
if (this.#root === undefined)
|
||||||
|
throw new TypeError("JavaScript checkpoint has no owned runtime");
|
||||||
|
const path = join(this.#root.path, `${id}.static.json`);
|
||||||
|
const bytes = (await stat(path)).size;
|
||||||
|
if (
|
||||||
|
bytes > DEFAULTS.checkpointBytes ||
|
||||||
|
(expected !== undefined && expected.bytes !== bytes)
|
||||||
|
)
|
||||||
|
throw new AnalysisOutputError(
|
||||||
|
OPERATION,
|
||||||
|
"JavaScript static checkpoint byte count exceeds or differs from its declared budget",
|
||||||
|
);
|
||||||
|
if (
|
||||||
|
bytes * 16 + 4 * MIB >
|
||||||
|
getHeapStatistics().heap_size_limit -
|
||||||
|
process.memoryUsage().heapUsed -
|
||||||
|
64 * MIB
|
||||||
|
)
|
||||||
|
throw new AnalysisResourceConstraintError(
|
||||||
|
OPERATION,
|
||||||
|
"memory",
|
||||||
|
"The parent's remaining heap cannot safely decode the completed JavaScript static checkpoint",
|
||||||
|
{ ...this.#limits(), checkpoint_bytes: bytes },
|
||||||
|
);
|
||||||
|
const text = await readFile(path, "utf8");
|
||||||
|
if (
|
||||||
|
expected !== undefined &&
|
||||||
|
createHash("sha256").update(text).digest("hex") !== expected.sha256
|
||||||
|
)
|
||||||
|
throw new AnalysisOutputError(
|
||||||
|
OPERATION,
|
||||||
|
"JavaScript static checkpoint digest does not match",
|
||||||
|
);
|
||||||
|
const value: unknown = JSON.parse(text);
|
||||||
|
const facts = javaScriptWorkerStaticAnalysisSchema.parse(value);
|
||||||
|
const remaining = this.#remainingTotalResultBudget();
|
||||||
|
const retainedBytes = bytes * 6;
|
||||||
|
this.#parentCapacityExhausted = retainedBytes > remaining;
|
||||||
|
this.#estimatedRetainedBytes += retainedBytes;
|
||||||
|
this.#retainedStaticBytes += retainedBytes;
|
||||||
|
const expandedBytes = estimateJavaScriptStaticApplicationProjection(
|
||||||
|
facts,
|
||||||
|
bytes,
|
||||||
|
sourcePath,
|
||||||
|
);
|
||||||
|
if (expandedBytes <= this.#remainingResultBudget())
|
||||||
|
this.#estimatedRetainedBytes += expandedBytes;
|
||||||
|
else
|
||||||
|
this.#applicationProjectionFailure = new AnalysisResourceConstraintError(
|
||||||
|
OPERATION,
|
||||||
|
"memory",
|
||||||
|
`Expanded application relationships for ${sourcePath} exceed the parent's result budget; complete static facts were retained on the file node`,
|
||||||
|
{
|
||||||
|
parent_result_budget_bytes: this.#resultBudgetBytes,
|
||||||
|
application_projection_required_bytes: expandedBytes,
|
||||||
|
},
|
||||||
|
);
|
||||||
|
return facts;
|
||||||
|
}
|
||||||
|
|
||||||
|
#limits() {
|
||||||
|
return {
|
||||||
|
worker_old_space_mb: this.limits.heapMb,
|
||||||
|
...(this.#heapLimitBytes === 0
|
||||||
|
? {}
|
||||||
|
: { worker_heap_limit_bytes: this.#heapLimitBytes }),
|
||||||
|
worker_timeout_ms: this.limits.timeoutMs,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
#remainingResultBudget(): number {
|
||||||
|
// Future sources need their compact observations before current modules
|
||||||
|
// consume capacity with expanded relationships. A final or sole source
|
||||||
|
// can use the unused reserve as well.
|
||||||
|
const staticReserve =
|
||||||
|
this.#sourcesRemaining !== undefined && this.#sourcesRemaining > 0
|
||||||
|
? Math.max(
|
||||||
|
0,
|
||||||
|
Math.floor((this.#resultBudgetBytes * 4) / 5) -
|
||||||
|
this.#retainedStaticBytes,
|
||||||
|
)
|
||||||
|
: 0;
|
||||||
|
return Math.max(0, this.#remainingTotalResultBudget() - staticReserve);
|
||||||
|
}
|
||||||
|
#remainingTotalResultBudget(): number {
|
||||||
|
return Math.max(
|
||||||
|
0,
|
||||||
|
Math.min(
|
||||||
|
this.#resultBudgetBytes - this.#estimatedRetainedBytes,
|
||||||
|
getHeapStatistics().heap_size_limit -
|
||||||
|
process.memoryUsage().heapUsed -
|
||||||
|
24 * JAVASCRIPT_ANALYSIS_TRANSFER_RECORD_BYTES,
|
||||||
|
),
|
||||||
|
);
|
||||||
|
}
|
||||||
|
#captured() {
|
||||||
|
const snapshot = this.#supervisor?.snapshot();
|
||||||
|
return {
|
||||||
|
stdout: "",
|
||||||
|
stderr: snapshot?.stderr.text ?? "",
|
||||||
|
truncated: snapshot?.diagnosticTruncated ?? false,
|
||||||
|
stderr_bytes: snapshot?.stderr.observedBytes ?? 0,
|
||||||
|
exit_code: snapshot?.exitCode ?? null,
|
||||||
|
signal: snapshot?.signal ?? null,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
#failure(
|
||||||
|
cause: unknown,
|
||||||
|
path: string,
|
||||||
|
deadline: ProviderStartupDeadline | undefined,
|
||||||
|
): AnalysisError {
|
||||||
|
if (deadline?.interruption === "cancelled")
|
||||||
|
return new AnalysisCancelledError(OPERATION, {
|
||||||
|
capturedOutput: this.#captured(),
|
||||||
|
});
|
||||||
|
if (deadline?.interruption === "timeout")
|
||||||
|
return new AnalysisTimeoutError(OPERATION, this.limits.timeoutMs, {
|
||||||
|
capturedOutput: this.#captured(),
|
||||||
|
});
|
||||||
|
if (cause instanceof AnalysisError) return cause;
|
||||||
|
if (
|
||||||
|
cause instanceof z.ZodError ||
|
||||||
|
cause instanceof TypeError ||
|
||||||
|
cause instanceof SyntaxError
|
||||||
|
)
|
||||||
|
return new AnalysisOutputError(
|
||||||
|
OPERATION,
|
||||||
|
`${path} during ${this.#phase}: ${message(cause)}`,
|
||||||
|
{ cause, capturedOutput: this.#captured() },
|
||||||
|
);
|
||||||
|
return cause instanceof PrivateRuntimeRootUnavailableError
|
||||||
|
? new AnalysisCapabilityUnavailableError(
|
||||||
|
PROVIDER,
|
||||||
|
OPERATION,
|
||||||
|
cause.message,
|
||||||
|
{ cause },
|
||||||
|
)
|
||||||
|
: new ProviderAdapterError(PROVIDER, OPERATION, {
|
||||||
|
cause,
|
||||||
|
capturedOutput: this.#captured(),
|
||||||
|
diagnostics: {
|
||||||
|
source_path: path,
|
||||||
|
phase: this.#phase,
|
||||||
|
message: message(cause),
|
||||||
|
},
|
||||||
|
});
|
||||||
|
}
|
||||||
|
#cleanupError(
|
||||||
|
reason: string,
|
||||||
|
original: AnalysisError | undefined = this.#lastFailure,
|
||||||
|
): ProviderCleanupError {
|
||||||
|
return new ProviderCleanupError(
|
||||||
|
PROVIDER,
|
||||||
|
[this.#runId, ...(this.#root === undefined ? [] : [this.#root.path])],
|
||||||
|
{
|
||||||
|
reason,
|
||||||
|
worker_pid: this.#supervisor?.launch.process.pid ?? null,
|
||||||
|
...(original === undefined
|
||||||
|
? {}
|
||||||
|
: {
|
||||||
|
execution_failure: {
|
||||||
|
tag: original._tag,
|
||||||
|
message: original.message,
|
||||||
|
captured_output: {
|
||||||
|
stdout: original.capturedOutput?.stdout ?? "",
|
||||||
|
stderr: original.capturedOutput?.stderr ?? "",
|
||||||
|
truncated: original.capturedOutput?.truncated ?? false,
|
||||||
|
},
|
||||||
|
},
|
||||||
|
}),
|
||||||
|
},
|
||||||
|
{
|
||||||
|
operation: OPERATION,
|
||||||
|
...(original === undefined ? {} : { cause: original }),
|
||||||
|
},
|
||||||
|
);
|
||||||
|
}
|
||||||
|
async #stop(
|
||||||
|
original: AnalysisError | undefined = this.#lastFailure,
|
||||||
|
): Promise<ProviderCleanupError | undefined> {
|
||||||
|
if (this.#supervisor !== undefined) {
|
||||||
|
const supervisor = this.#supervisor;
|
||||||
|
const stopped = await supervisor.stop();
|
||||||
|
if (stopped.status === "incomplete") {
|
||||||
|
pendingCleanup.add(this);
|
||||||
|
return this.#cleanupError(stopped.reason, original);
|
||||||
|
}
|
||||||
|
supervisor.launch.process.stdout?.off("data", this.#onStdout);
|
||||||
|
supervisor.dispose();
|
||||||
|
this.#supervisor = undefined;
|
||||||
|
}
|
||||||
|
this.#ready = false;
|
||||||
|
this.#frames = [];
|
||||||
|
this.#pending = "";
|
||||||
|
this.#decoder = new StringDecoder("utf8");
|
||||||
|
this.#protocolFailure = undefined;
|
||||||
|
return undefined;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
const message = (cause: unknown): string =>
|
||||||
|
cause instanceof Error ? cause.message : String(cause);
|
||||||
@@ -0,0 +1,73 @@
|
|||||||
|
import { z } from "zod";
|
||||||
|
import { jsonValueSchema } from "../../domain/jsonValue.js";
|
||||||
|
import { digestSchema } from "../../domain/digests.js";
|
||||||
|
import { JAVASCRIPT_ANALYSIS_RESOURCE_DEFAULTS } from "../../domain/javascript/javascriptAnalysisResourceControls.js";
|
||||||
|
import { javaScriptAnalysisTransferDescriptorSchema } from "./JavaScriptAnalysisTransfer.js";
|
||||||
|
|
||||||
|
/** Independent worker execution budget; source length is not an admission rule. */
|
||||||
|
export const JAVASCRIPT_ANALYSIS_WORKER_DEFAULTS = {
|
||||||
|
...JAVASCRIPT_ANALYSIS_RESOURCE_DEFAULTS,
|
||||||
|
checkpointBytes: 8 * 1024 * 1024,
|
||||||
|
protocolBytes: 64 * 1024,
|
||||||
|
} as const;
|
||||||
|
|
||||||
|
/** Caller-selected artifact identity, separate from the decoded source-text digest. */
|
||||||
|
export const javaScriptWorkerRequestSchema = z.strictObject({
|
||||||
|
id: z.uuid(),
|
||||||
|
node_budget: z.number().int().min(0).max(20_000),
|
||||||
|
text_sha256: digestSchema,
|
||||||
|
file: z.strictObject({
|
||||||
|
path: z.string(),
|
||||||
|
sha256: digestSchema,
|
||||||
|
container_sha256: digestSchema,
|
||||||
|
bytes: z.number().int().nonnegative(),
|
||||||
|
inventory_artifact_id: z.string(),
|
||||||
|
kind: z.literal("javascript"),
|
||||||
|
unpacked: z.boolean(),
|
||||||
|
}),
|
||||||
|
});
|
||||||
|
|
||||||
|
/** Parent admission after validated static observations have been retained. */
|
||||||
|
export const javaScriptWorkerAdmissionSchema = z.strictObject({
|
||||||
|
id: z.uuid(),
|
||||||
|
semantic_result_budget_bytes: z.number().int().nonnegative(),
|
||||||
|
});
|
||||||
|
|
||||||
|
const identity = { id: z.uuid() };
|
||||||
|
|
||||||
|
/** Control frames never contain AST, full IR, graph rows or source text. */
|
||||||
|
export const javaScriptWorkerMessageSchema = z.discriminatedUnion("kind", [
|
||||||
|
z.strictObject({
|
||||||
|
kind: z.literal("ready"),
|
||||||
|
heap_limit_bytes: z.number().int().positive(),
|
||||||
|
}),
|
||||||
|
z.strictObject({
|
||||||
|
kind: z.literal("phase"),
|
||||||
|
...identity,
|
||||||
|
phase: z.enum(["parse", "static", "semantic", "projection", "transfer"]),
|
||||||
|
}),
|
||||||
|
z.strictObject({
|
||||||
|
kind: z.literal("static"),
|
||||||
|
...identity,
|
||||||
|
bytes: z.number().int().nonnegative(),
|
||||||
|
sha256: digestSchema,
|
||||||
|
}),
|
||||||
|
z.strictObject({
|
||||||
|
kind: z.literal("complete"),
|
||||||
|
...identity,
|
||||||
|
descriptor: javaScriptAnalysisTransferDescriptorSchema.nullable(),
|
||||||
|
}),
|
||||||
|
z.strictObject({
|
||||||
|
kind: z.literal("failure"),
|
||||||
|
...identity,
|
||||||
|
reason: z.enum(["resource", "output", "io"]),
|
||||||
|
resource: z.enum(["memory", "cpu", "file-size", "transport"]).nullable(),
|
||||||
|
limits: z.record(z.string(), jsonValueSchema).nullable(),
|
||||||
|
message: z.string(),
|
||||||
|
}),
|
||||||
|
]);
|
||||||
|
|
||||||
|
/** One decoded control frame from the independently owned worker. */
|
||||||
|
export type JavaScriptWorkerMessage = z.output<
|
||||||
|
typeof javaScriptWorkerMessageSchema
|
||||||
|
>;
|
||||||
@@ -0,0 +1,582 @@
|
|||||||
|
import { z } from "zod";
|
||||||
|
import { digestSchema } from "../../domain/digests.js";
|
||||||
|
import type { JavaScriptStaticAnalysis } from "../../domain/javascript/javascriptStaticAnalysisTypes.js";
|
||||||
|
import type {
|
||||||
|
JavaScriptSemanticProperty,
|
||||||
|
JavaScriptSemanticValue,
|
||||||
|
} from "../../domain/javascript/javascriptSemanticValueTypes.js";
|
||||||
|
import {
|
||||||
|
javaScriptSemanticNodeSchema,
|
||||||
|
javaScriptSemanticRelationSchema,
|
||||||
|
javaScriptSemanticUnknownSchema,
|
||||||
|
javaScriptSemanticFingerprintSchema,
|
||||||
|
javaScriptSemanticEvidenceContextSchema,
|
||||||
|
} from "../../domain/javascript/javascriptSemanticGraphSchemas.js";
|
||||||
|
|
||||||
|
const range =
|
||||||
|
javaScriptSemanticNodeSchema.shape.identity.shape.source_range.unwrap();
|
||||||
|
const count = z.number().int().nonnegative();
|
||||||
|
const strings = z.array(z.string());
|
||||||
|
const pathContext = z.enum(["module-specifier", "filesystem-expression"]);
|
||||||
|
const findingContext = {
|
||||||
|
module_key: z.string().nullable(),
|
||||||
|
module_runtime: z.string().optional(),
|
||||||
|
location: range,
|
||||||
|
};
|
||||||
|
const withRuntime = <Value extends { module_runtime?: string | undefined }>(
|
||||||
|
value: Value,
|
||||||
|
) => {
|
||||||
|
const { module_runtime, ...rest } = value;
|
||||||
|
return {
|
||||||
|
...rest,
|
||||||
|
...(module_runtime === undefined ? {} : { module_runtime }),
|
||||||
|
};
|
||||||
|
};
|
||||||
|
const staticValue = z.discriminatedUnion("status", [
|
||||||
|
z.strictObject({
|
||||||
|
status: z.literal("literal"),
|
||||||
|
value: z.union([z.string(), z.number(), z.boolean(), z.null()]),
|
||||||
|
expression: z.null(),
|
||||||
|
}),
|
||||||
|
z.strictObject({
|
||||||
|
status: z.literal("dynamic"),
|
||||||
|
value: z.null(),
|
||||||
|
expression: z.string(),
|
||||||
|
}),
|
||||||
|
]);
|
||||||
|
const nullHandler = { handler_kind: z.null(), handler_location: z.null() };
|
||||||
|
const missingHandler = {
|
||||||
|
handler_kind: z.literal("missing"),
|
||||||
|
handler_location: z.null(),
|
||||||
|
};
|
||||||
|
const locatedHandler = {
|
||||||
|
handler_kind: z.enum([
|
||||||
|
"inline-function",
|
||||||
|
"identifier",
|
||||||
|
"member-expression",
|
||||||
|
"dynamic-expression",
|
||||||
|
]),
|
||||||
|
handler_location: range,
|
||||||
|
};
|
||||||
|
const ipcChannel = <Shape extends z.ZodRawShape>(shape: Shape) =>
|
||||||
|
z.union([
|
||||||
|
z.strictObject({
|
||||||
|
...findingContext,
|
||||||
|
...shape,
|
||||||
|
channel: z.string(),
|
||||||
|
channel_expression: z.null(),
|
||||||
|
}),
|
||||||
|
z.strictObject({
|
||||||
|
...findingContext,
|
||||||
|
...shape,
|
||||||
|
channel: z.null(),
|
||||||
|
channel_expression: z.string(),
|
||||||
|
}),
|
||||||
|
]);
|
||||||
|
const ipc = z
|
||||||
|
.union([
|
||||||
|
ipcChannel({
|
||||||
|
side: z.literal("renderer"),
|
||||||
|
operation: z.enum(["send", "send-sync", "post-message", "send-to-host"]),
|
||||||
|
mode: z.literal("send"),
|
||||||
|
...nullHandler,
|
||||||
|
}),
|
||||||
|
ipcChannel({
|
||||||
|
side: z.literal("renderer"),
|
||||||
|
operation: z.literal("invoke"),
|
||||||
|
mode: z.literal("invoke"),
|
||||||
|
...nullHandler,
|
||||||
|
}),
|
||||||
|
ipcChannel({
|
||||||
|
side: z.enum(["renderer", "main"]),
|
||||||
|
operation: z.enum(["on", "once"]),
|
||||||
|
mode: z.literal("listen"),
|
||||||
|
...missingHandler,
|
||||||
|
}),
|
||||||
|
ipcChannel({
|
||||||
|
side: z.enum(["renderer", "main"]),
|
||||||
|
operation: z.enum(["on", "once"]),
|
||||||
|
mode: z.literal("listen"),
|
||||||
|
...locatedHandler,
|
||||||
|
}),
|
||||||
|
ipcChannel({
|
||||||
|
side: z.literal("main"),
|
||||||
|
operation: z.enum(["handle", "handle-once"]),
|
||||||
|
mode: z.literal("handle"),
|
||||||
|
...missingHandler,
|
||||||
|
}),
|
||||||
|
ipcChannel({
|
||||||
|
side: z.literal("main"),
|
||||||
|
operation: z.enum(["handle", "handle-once"]),
|
||||||
|
mode: z.literal("handle"),
|
||||||
|
...locatedHandler,
|
||||||
|
}),
|
||||||
|
])
|
||||||
|
.transform((value) => {
|
||||||
|
const { module_runtime, ...rest } = value;
|
||||||
|
return {
|
||||||
|
...rest,
|
||||||
|
...(module_runtime === undefined ? {} : { module_runtime }),
|
||||||
|
};
|
||||||
|
});
|
||||||
|
const windowShape = {
|
||||||
|
...findingContext,
|
||||||
|
options_status: z.enum(["object-literal", "dynamic", "missing"]),
|
||||||
|
web_preferences_status: z.enum(["object-literal", "dynamic", "missing"]),
|
||||||
|
web_preferences: z.array(
|
||||||
|
z.strictObject({ name: z.string(), value: staticValue }),
|
||||||
|
),
|
||||||
|
};
|
||||||
|
const bridgeShape = {
|
||||||
|
...findingContext,
|
||||||
|
world: z.enum(["main", "isolated"]),
|
||||||
|
world_id: staticValue.nullable(),
|
||||||
|
api_status: z.enum(["object-literal", "dynamic", "missing"]),
|
||||||
|
members: strings,
|
||||||
|
unknown_members: count,
|
||||||
|
};
|
||||||
|
const utilityShape = { ...findingContext, service_name: z.string().nullable() };
|
||||||
|
const electron = z.strictObject({
|
||||||
|
browser_windows: z.array(
|
||||||
|
z.union([
|
||||||
|
z
|
||||||
|
.strictObject({
|
||||||
|
...windowShape,
|
||||||
|
preload_path: z.string(),
|
||||||
|
preload_resolution_context: pathContext,
|
||||||
|
})
|
||||||
|
.transform(withRuntime),
|
||||||
|
z
|
||||||
|
.strictObject({
|
||||||
|
...windowShape,
|
||||||
|
preload_path: z.null(),
|
||||||
|
preload_resolution_context: z.null(),
|
||||||
|
})
|
||||||
|
.transform(withRuntime),
|
||||||
|
]),
|
||||||
|
),
|
||||||
|
context_bridge_apis: z.array(
|
||||||
|
z.union([
|
||||||
|
z
|
||||||
|
.strictObject({
|
||||||
|
...bridgeShape,
|
||||||
|
api_key: z.string(),
|
||||||
|
api_key_expression: z.null(),
|
||||||
|
})
|
||||||
|
.transform(withRuntime),
|
||||||
|
z
|
||||||
|
.strictObject({
|
||||||
|
...bridgeShape,
|
||||||
|
api_key: z.null(),
|
||||||
|
api_key_expression: z.string(),
|
||||||
|
})
|
||||||
|
.transform(withRuntime),
|
||||||
|
z
|
||||||
|
.strictObject({
|
||||||
|
...bridgeShape,
|
||||||
|
api_key: z.null(),
|
||||||
|
api_key_expression: z.null(),
|
||||||
|
})
|
||||||
|
.transform(withRuntime),
|
||||||
|
]),
|
||||||
|
),
|
||||||
|
ipc: z.array(ipc),
|
||||||
|
sender_validations: z.array(
|
||||||
|
z.strictObject({
|
||||||
|
...findingContext,
|
||||||
|
subject: z.enum([
|
||||||
|
"sender-url",
|
||||||
|
"sender-origin",
|
||||||
|
"sender-frame",
|
||||||
|
"sender-id",
|
||||||
|
"frame-id",
|
||||||
|
"process-id",
|
||||||
|
]),
|
||||||
|
mechanism: z.string(),
|
||||||
|
expected: staticValue,
|
||||||
|
enforcement: z.literal("unknown"),
|
||||||
|
}),
|
||||||
|
),
|
||||||
|
utility_processes: z.array(
|
||||||
|
z.union([
|
||||||
|
z
|
||||||
|
.strictObject({
|
||||||
|
...utilityShape,
|
||||||
|
module_path: z.string(),
|
||||||
|
module_resolution_context: pathContext,
|
||||||
|
module_expression: z.null(),
|
||||||
|
})
|
||||||
|
.transform(withRuntime),
|
||||||
|
z
|
||||||
|
.strictObject({
|
||||||
|
...utilityShape,
|
||||||
|
module_path: z.null(),
|
||||||
|
module_resolution_context: z.null(),
|
||||||
|
module_expression: z.string(),
|
||||||
|
})
|
||||||
|
.transform(withRuntime),
|
||||||
|
]),
|
||||||
|
),
|
||||||
|
native_addon_bindings: z.array(
|
||||||
|
z.strictObject({
|
||||||
|
...findingContext,
|
||||||
|
specifier: z.string(),
|
||||||
|
binding_kind: z.enum(["import", "require", "re-export"]),
|
||||||
|
module_kind: z.enum(["import", "require"]),
|
||||||
|
members: strings,
|
||||||
|
namespace_access: z.boolean().optional(),
|
||||||
|
dynamic_member_access: z.boolean().optional(),
|
||||||
|
}),
|
||||||
|
),
|
||||||
|
});
|
||||||
|
const referenceShape = {
|
||||||
|
...findingContext,
|
||||||
|
kind: z.enum([
|
||||||
|
"static-import",
|
||||||
|
"dynamic-import",
|
||||||
|
"require",
|
||||||
|
"worker",
|
||||||
|
"service-worker",
|
||||||
|
]),
|
||||||
|
};
|
||||||
|
|
||||||
|
/** Exact producer facts recoverable before a later semantic worker failure. */
|
||||||
|
export const javaScriptWorkerStaticAnalysisSchema = z
|
||||||
|
.strictObject({
|
||||||
|
parse_status: z.enum(["complete", "partial", "failed"]),
|
||||||
|
parse_error_count: count,
|
||||||
|
visited_ast_nodes: count,
|
||||||
|
references: z.array(
|
||||||
|
z.union([
|
||||||
|
z
|
||||||
|
.strictObject({
|
||||||
|
...referenceShape,
|
||||||
|
specifier: z.string(),
|
||||||
|
expression: z.null(),
|
||||||
|
})
|
||||||
|
.transform(withRuntime),
|
||||||
|
z
|
||||||
|
.strictObject({
|
||||||
|
...referenceShape,
|
||||||
|
specifier: z.null(),
|
||||||
|
expression: z.string(),
|
||||||
|
})
|
||||||
|
.transform(withRuntime),
|
||||||
|
]),
|
||||||
|
),
|
||||||
|
endpoints: z.array(
|
||||||
|
z.strictObject({
|
||||||
|
...findingContext,
|
||||||
|
kind: z.enum(["route", "network"]),
|
||||||
|
value: z.string(),
|
||||||
|
mechanism: z.string(),
|
||||||
|
}),
|
||||||
|
),
|
||||||
|
storage: z.array(
|
||||||
|
z.strictObject({
|
||||||
|
...findingContext,
|
||||||
|
kind: z.enum([
|
||||||
|
"local-storage",
|
||||||
|
"session-storage",
|
||||||
|
"indexed-db",
|
||||||
|
"cache-storage",
|
||||||
|
"sqlite",
|
||||||
|
]),
|
||||||
|
name: z.string().nullable(),
|
||||||
|
mechanism: z.string(),
|
||||||
|
}),
|
||||||
|
),
|
||||||
|
bundler_registrations: z.array(
|
||||||
|
z.strictObject({
|
||||||
|
bundler: z.enum(["webpack", "rspack", "esbuild"]),
|
||||||
|
runtime: z.string(),
|
||||||
|
chunk_keys: strings,
|
||||||
|
unknown_chunk_keys: count,
|
||||||
|
runtime_require_name: z.string().nullable(),
|
||||||
|
runtime_module_cache_status: z.enum(["observed", "not-observed"]),
|
||||||
|
entry_module_keys: strings,
|
||||||
|
unknown_entry_module_keys: count,
|
||||||
|
async_chunk_keys: strings,
|
||||||
|
unknown_async_chunk_keys: count,
|
||||||
|
modules: z.array(
|
||||||
|
z.strictObject({
|
||||||
|
module_key: z.string(),
|
||||||
|
factory_require_name: z.string().nullable(),
|
||||||
|
source_sha256: digestSchema,
|
||||||
|
exports: strings,
|
||||||
|
location: range,
|
||||||
|
structural_fingerprint_sha256: digestSchema,
|
||||||
|
structural_fingerprint_algorithm: z.literal("babel-ast-v1"),
|
||||||
|
}),
|
||||||
|
),
|
||||||
|
location: range,
|
||||||
|
}),
|
||||||
|
),
|
||||||
|
role_paths: z.array(
|
||||||
|
z.strictObject({
|
||||||
|
...findingContext,
|
||||||
|
role: z.enum(["preload", "renderer"]),
|
||||||
|
path: z.string(),
|
||||||
|
resolution_context: pathContext,
|
||||||
|
mechanism: z.string(),
|
||||||
|
}),
|
||||||
|
),
|
||||||
|
source_map_urls: z.array(
|
||||||
|
z.strictObject({ declared_url: z.string(), location: range }),
|
||||||
|
),
|
||||||
|
vendors: strings,
|
||||||
|
electron,
|
||||||
|
limitations: strings,
|
||||||
|
})
|
||||||
|
.transform((analysis) => ({
|
||||||
|
...analysis,
|
||||||
|
endpoints: analysis.endpoints.map(withRuntime),
|
||||||
|
storage: analysis.storage.map(withRuntime),
|
||||||
|
role_paths: analysis.role_paths.map(withRuntime),
|
||||||
|
electron: {
|
||||||
|
...analysis.electron,
|
||||||
|
sender_validations: analysis.electron.sender_validations.map(withRuntime),
|
||||||
|
native_addon_bindings: analysis.electron.native_addon_bindings.map(
|
||||||
|
(value) => {
|
||||||
|
const { namespace_access, dynamic_member_access, ...rest } =
|
||||||
|
withRuntime(value);
|
||||||
|
return {
|
||||||
|
...rest,
|
||||||
|
...(namespace_access === undefined ? {} : { namespace_access }),
|
||||||
|
...(dynamic_member_access === undefined
|
||||||
|
? {}
|
||||||
|
: { dynamic_member_access }),
|
||||||
|
};
|
||||||
|
},
|
||||||
|
),
|
||||||
|
},
|
||||||
|
})) satisfies z.ZodType<JavaScriptStaticAnalysis>;
|
||||||
|
|
||||||
|
const resourceLimit = z.enum([
|
||||||
|
"primitive-candidates",
|
||||||
|
"primitive-bytes",
|
||||||
|
"expression-depth",
|
||||||
|
]);
|
||||||
|
const primitive = z.union([z.string(), z.number(), z.boolean(), z.null()]);
|
||||||
|
const property: z.ZodType<JavaScriptSemanticProperty> = z.lazy(() =>
|
||||||
|
z.strictObject({
|
||||||
|
name: z.string(),
|
||||||
|
value: semanticValue,
|
||||||
|
presence: z.enum(["present", "absent", "unknown-coverage"]),
|
||||||
|
}),
|
||||||
|
);
|
||||||
|
const semanticValue: z.ZodType<JavaScriptSemanticValue> = z.lazy(() =>
|
||||||
|
z
|
||||||
|
.discriminatedUnion("status", [
|
||||||
|
z.strictObject({ status: z.literal("literal"), value: primitive }),
|
||||||
|
z.strictObject({
|
||||||
|
status: z.literal("union"),
|
||||||
|
values: z.array(primitive),
|
||||||
|
}),
|
||||||
|
z.discriminatedUnion("unknownProperties", [
|
||||||
|
z.strictObject({
|
||||||
|
status: z.literal("object"),
|
||||||
|
properties: z.array(property),
|
||||||
|
unknownProperties: z.literal(false),
|
||||||
|
omittedProperties: z.literal(0),
|
||||||
|
}),
|
||||||
|
z.strictObject({
|
||||||
|
status: z.literal("object"),
|
||||||
|
properties: z.array(property),
|
||||||
|
unknownProperties: z.literal(true),
|
||||||
|
omittedProperties: count.nullable(),
|
||||||
|
}),
|
||||||
|
]),
|
||||||
|
z.discriminatedUnion("unknownItems", [
|
||||||
|
z.strictObject({
|
||||||
|
status: z.literal("array"),
|
||||||
|
items: z.array(property),
|
||||||
|
unknownItems: z.literal(false),
|
||||||
|
omittedItems: z.literal(0),
|
||||||
|
}),
|
||||||
|
z.strictObject({
|
||||||
|
status: z.literal("array"),
|
||||||
|
items: z.array(property),
|
||||||
|
unknownItems: z.literal(true),
|
||||||
|
omittedItems: count.nullable(),
|
||||||
|
}),
|
||||||
|
]),
|
||||||
|
z.strictObject({
|
||||||
|
status: z.enum(["unknown", "ambiguous", "cycle"]),
|
||||||
|
reason: z.string(),
|
||||||
|
resourceLimit: resourceLimit.optional(),
|
||||||
|
}),
|
||||||
|
])
|
||||||
|
.transform((value) => {
|
||||||
|
if (
|
||||||
|
value.status === "literal" ||
|
||||||
|
value.status === "union" ||
|
||||||
|
value.status === "object" ||
|
||||||
|
value.status === "array"
|
||||||
|
)
|
||||||
|
return value;
|
||||||
|
const { resourceLimit: limit, ...rest } = value;
|
||||||
|
return {
|
||||||
|
...rest,
|
||||||
|
...(limit === undefined ? {} : { resourceLimit: limit }),
|
||||||
|
};
|
||||||
|
}),
|
||||||
|
);
|
||||||
|
const origin = z.strictObject({ specifier: z.string(), importedPath: strings });
|
||||||
|
const provenance = z.discriminatedUnion("status", [
|
||||||
|
z.strictObject({
|
||||||
|
status: z.literal("module"),
|
||||||
|
origins: z.tuple([origin]),
|
||||||
|
reason: z.null(),
|
||||||
|
}),
|
||||||
|
z.strictObject({
|
||||||
|
status: z.literal("local"),
|
||||||
|
origins: z.tuple([]),
|
||||||
|
reason: z.null(),
|
||||||
|
}),
|
||||||
|
z.strictObject({
|
||||||
|
status: z.literal("ambiguous"),
|
||||||
|
origins: z.array(origin),
|
||||||
|
reason: z.string(),
|
||||||
|
}),
|
||||||
|
z.strictObject({
|
||||||
|
status: z.enum(["unknown", "cycle"]),
|
||||||
|
origins: z.tuple([]),
|
||||||
|
reason: z.string(),
|
||||||
|
}),
|
||||||
|
]);
|
||||||
|
const coverage = z
|
||||||
|
.discriminatedUnion("status", [
|
||||||
|
z.strictObject({
|
||||||
|
status: z.literal("complete"),
|
||||||
|
omittedCount: z.literal(0),
|
||||||
|
resourceLimits: z.array(resourceLimit).optional(),
|
||||||
|
}),
|
||||||
|
z.strictObject({
|
||||||
|
status: z.literal("partial"),
|
||||||
|
omittedCount: z.union([z.literal(0), z.null()]),
|
||||||
|
resourceLimits: z.array(resourceLimit).optional(),
|
||||||
|
}),
|
||||||
|
z.strictObject({
|
||||||
|
status: z.literal("failed"),
|
||||||
|
omittedCount: z.null(),
|
||||||
|
resourceLimits: z.array(resourceLimit).optional(),
|
||||||
|
}),
|
||||||
|
])
|
||||||
|
.transform((value) => {
|
||||||
|
const { resourceLimits, ...rest } = value;
|
||||||
|
return {
|
||||||
|
...rest,
|
||||||
|
...(resourceLimits === undefined ? {} : { resourceLimits }),
|
||||||
|
};
|
||||||
|
});
|
||||||
|
const moduleLink = z.strictObject({
|
||||||
|
kind: z.enum(["import", "require", "export", "re-export", "commonjs-export"]),
|
||||||
|
specifier: z.string().nullable(),
|
||||||
|
importedName: z.string().nullable(),
|
||||||
|
localName: z.string().nullable(),
|
||||||
|
exportedName: z.string().nullable(),
|
||||||
|
callableId: z.string().nullable(),
|
||||||
|
location: range,
|
||||||
|
});
|
||||||
|
const definitionKind = z.enum([
|
||||||
|
"import",
|
||||||
|
"variable",
|
||||||
|
"parameter",
|
||||||
|
"function",
|
||||||
|
"class",
|
||||||
|
"catch",
|
||||||
|
"assignment",
|
||||||
|
]);
|
||||||
|
const scope = z.strictObject({
|
||||||
|
scopeId: z.string(),
|
||||||
|
parentScopeId: z.string().nullable(),
|
||||||
|
kind: z.enum([
|
||||||
|
"program",
|
||||||
|
"function",
|
||||||
|
"block",
|
||||||
|
"static-block",
|
||||||
|
"class",
|
||||||
|
"catch",
|
||||||
|
]),
|
||||||
|
location: range,
|
||||||
|
bindingsComplete: z.boolean(),
|
||||||
|
bindingIds: strings,
|
||||||
|
});
|
||||||
|
const binding = z.strictObject({
|
||||||
|
bindingId: z.string(),
|
||||||
|
scopeId: z.string(),
|
||||||
|
name: z.string(),
|
||||||
|
kind: definitionKind,
|
||||||
|
mutable: z.boolean(),
|
||||||
|
definitions: z.array(
|
||||||
|
z.strictObject({ kind: definitionKind, location: range }),
|
||||||
|
),
|
||||||
|
value: semanticValue,
|
||||||
|
provenance,
|
||||||
|
});
|
||||||
|
const callable = z.strictObject({
|
||||||
|
callableId: z.string(),
|
||||||
|
kind: z.enum(["function", "class", "method"]),
|
||||||
|
name: z.string().nullable(),
|
||||||
|
containerScopeId: z.string(),
|
||||||
|
bodyScopeId: z.string().nullable(),
|
||||||
|
location: range,
|
||||||
|
returnSites: z.array(
|
||||||
|
z.strictObject({
|
||||||
|
returnSiteId: z.string(),
|
||||||
|
location: range,
|
||||||
|
identityReferenceLocation: range.nullable(),
|
||||||
|
value: semanticValue,
|
||||||
|
}),
|
||||||
|
),
|
||||||
|
returnCoverage: z.strictObject({
|
||||||
|
retainedCount: count,
|
||||||
|
status: z.enum(["complete", "partial"]),
|
||||||
|
omittedCount: z.union([z.literal(0), z.null()]),
|
||||||
|
}),
|
||||||
|
});
|
||||||
|
|
||||||
|
/** Validate each transferred graph fact independently of its emitting realm. */
|
||||||
|
export const javaScriptWorkerGraphRecordSchema = z.discriminatedUnion("kind", [
|
||||||
|
z.strictObject({
|
||||||
|
kind: z.literal("evidence-context"),
|
||||||
|
value: javaScriptSemanticEvidenceContextSchema,
|
||||||
|
}),
|
||||||
|
z.strictObject({
|
||||||
|
kind: z.literal("node"),
|
||||||
|
value: javaScriptSemanticNodeSchema,
|
||||||
|
}),
|
||||||
|
z.strictObject({
|
||||||
|
kind: z.literal("relation"),
|
||||||
|
value: javaScriptSemanticRelationSchema,
|
||||||
|
}),
|
||||||
|
z.strictObject({
|
||||||
|
kind: z.literal("unknown"),
|
||||||
|
value: javaScriptSemanticUnknownSchema,
|
||||||
|
}),
|
||||||
|
z.strictObject({
|
||||||
|
kind: z.literal("fingerprint"),
|
||||||
|
value: javaScriptSemanticFingerprintSchema,
|
||||||
|
}),
|
||||||
|
]);
|
||||||
|
|
||||||
|
/** Portable per-file header, before any graph binding or immutable proof. */
|
||||||
|
export const javaScriptWorkerProjectionHeaderSchema = z.strictObject({
|
||||||
|
roots: z.array(javaScriptSemanticNodeSchema.shape.node_id),
|
||||||
|
truncated: z.boolean(),
|
||||||
|
});
|
||||||
|
|
||||||
|
/** Module facts are transferred individually to avoid a whole-IR JSON string. */
|
||||||
|
export const javaScriptWorkerModuleRecordSchema = z.discriminatedUnion("kind", [
|
||||||
|
z.strictObject({ kind: z.literal("scope"), value: scope }),
|
||||||
|
z.strictObject({ kind: z.literal("binding"), value: binding }),
|
||||||
|
z.strictObject({ kind: z.literal("callable"), value: callable }),
|
||||||
|
z.strictObject({ kind: z.literal("module-link"), value: moduleLink }),
|
||||||
|
]);
|
||||||
|
|
||||||
|
/** Coverage and limitations are independent of module collection sizes. */
|
||||||
|
export const javaScriptWorkerModuleHeaderSchema = z.strictObject({
|
||||||
|
coverage,
|
||||||
|
limitations: strings,
|
||||||
|
});
|
||||||
@@ -0,0 +1,185 @@
|
|||||||
|
import { createHash } from "node:crypto";
|
||||||
|
import { readFile, writeFile } from "node:fs/promises";
|
||||||
|
import { join } from "node:path";
|
||||||
|
import { createInterface } from "node:readline";
|
||||||
|
import { getHeapStatistics } from "node:v8";
|
||||||
|
import { AnalysisResourceConstraintError } from "../../domain/analysisErrorCore.js";
|
||||||
|
import type { JavaScriptArtifactFile } from "../../domain/javascript/javascriptArtifactFiles.js";
|
||||||
|
import type { JavaScriptStaticAnalysis } from "../../domain/javascript/javascriptStaticAnalysisTypes.js";
|
||||||
|
import type { JavaScriptModuleSemanticIr } from "../../domain/javascript/javascriptModuleSemanticIr.js";
|
||||||
|
import type { JavaScriptSemanticFileProjection } from "../../domain/javascript/javascriptSemanticFileProjection.js";
|
||||||
|
import { writeJavaScriptAnalysisTransfer } from "./JavaScriptAnalysisTransfer.js";
|
||||||
|
import { javaScriptWorkerStaticAnalysisSchema } from "./JavaScriptAnalysisWorkerSchemas.js";
|
||||||
|
import {
|
||||||
|
JAVASCRIPT_ANALYSIS_WORKER_DEFAULTS as DEFAULTS,
|
||||||
|
javaScriptWorkerRequestSchema,
|
||||||
|
javaScriptWorkerAdmissionSchema,
|
||||||
|
javaScriptWorkerMessageSchema,
|
||||||
|
type JavaScriptWorkerMessage,
|
||||||
|
} from "./JavaScriptAnalysisWorkerProtocol.js";
|
||||||
|
|
||||||
|
/** Trusted composition callback; this provider owns framing and temporary I/O. */
|
||||||
|
export interface JavaScriptSourceWorkerAnalyzer {
|
||||||
|
(
|
||||||
|
file: JavaScriptArtifactFile,
|
||||||
|
nodeBudget: number,
|
||||||
|
callbacks: {
|
||||||
|
readonly phase: (
|
||||||
|
phase: "parse" | "static" | "semantic" | "projection",
|
||||||
|
) => Promise<void>;
|
||||||
|
readonly checkpoint: (facts: JavaScriptStaticAnalysis) => Promise<number>;
|
||||||
|
},
|
||||||
|
): Promise<{
|
||||||
|
readonly module: JavaScriptModuleSemanticIr;
|
||||||
|
readonly projection: JavaScriptSemanticFileProjection;
|
||||||
|
} | null>;
|
||||||
|
}
|
||||||
|
|
||||||
|
const send = async (input: JavaScriptWorkerMessage): Promise<void> => {
|
||||||
|
const value = javaScriptWorkerMessageSchema.parse(input);
|
||||||
|
const frame = `${JSON.stringify(value)}\n`;
|
||||||
|
if (Buffer.byteLength(frame) > DEFAULTS.protocolBytes)
|
||||||
|
throw new RangeError(
|
||||||
|
"JavaScript worker control frame exceeds its byte budget",
|
||||||
|
);
|
||||||
|
await new Promise<void>((resolve, reject) => {
|
||||||
|
process.stdout.write(frame, (error) =>
|
||||||
|
error === null || error === undefined ? resolve() : reject(error),
|
||||||
|
);
|
||||||
|
});
|
||||||
|
};
|
||||||
|
|
||||||
|
/** Serve sequential inert-source requests from a parent-owned private runtime root. */
|
||||||
|
export const serveJavaScriptAnalysisWorker = async (
|
||||||
|
analyze: JavaScriptSourceWorkerAnalyzer,
|
||||||
|
): Promise<void> => {
|
||||||
|
const root = process.cwd();
|
||||||
|
const input = createInterface({ input: process.stdin, crlfDelay: Infinity });
|
||||||
|
const requests = input[Symbol.asyncIterator]();
|
||||||
|
await send({
|
||||||
|
kind: "ready",
|
||||||
|
heap_limit_bytes: getHeapStatistics().heap_size_limit,
|
||||||
|
});
|
||||||
|
try {
|
||||||
|
for (;;) {
|
||||||
|
const next = await requests.next();
|
||||||
|
if (next.done) break;
|
||||||
|
const line = next.value;
|
||||||
|
if (Buffer.byteLength(line) > DEFAULTS.protocolBytes)
|
||||||
|
throw new RangeError(
|
||||||
|
"JavaScript worker request exceeds its byte budget",
|
||||||
|
);
|
||||||
|
const raw: unknown = JSON.parse(line);
|
||||||
|
const request = javaScriptWorkerRequestSchema.parse(raw);
|
||||||
|
// Function scope releases source AST/IR/results before accepting another job.
|
||||||
|
await analyzeJob(root, request, analyze, requests);
|
||||||
|
if (
|
||||||
|
process.memoryUsage().heapUsed >
|
||||||
|
getHeapStatistics().heap_size_limit / 2
|
||||||
|
)
|
||||||
|
global.gc?.();
|
||||||
|
}
|
||||||
|
} finally {
|
||||||
|
input.close();
|
||||||
|
}
|
||||||
|
};
|
||||||
|
|
||||||
|
const analyzeJob = async (
|
||||||
|
root: string,
|
||||||
|
request: ReturnType<typeof javaScriptWorkerRequestSchema.parse>,
|
||||||
|
analyze: JavaScriptSourceWorkerAnalyzer,
|
||||||
|
requests: AsyncIterator<string>,
|
||||||
|
): Promise<void> => {
|
||||||
|
try {
|
||||||
|
const source = await readFile(join(root, `${request.id}.source`), "utf8");
|
||||||
|
if (
|
||||||
|
createHash("sha256").update(source).digest("hex") !== request.text_sha256
|
||||||
|
)
|
||||||
|
throw new TypeError(
|
||||||
|
"JavaScript worker source text digest does not match its request",
|
||||||
|
);
|
||||||
|
const result = await analyze(
|
||||||
|
{
|
||||||
|
...request.file,
|
||||||
|
text: { included: true, value: source },
|
||||||
|
},
|
||||||
|
request.node_budget,
|
||||||
|
{
|
||||||
|
phase: async (phase) => send({ kind: "phase", id: request.id, phase }),
|
||||||
|
checkpoint: async (facts) => {
|
||||||
|
const value = javaScriptWorkerStaticAnalysisSchema.parse(facts);
|
||||||
|
const text = JSON.stringify(value);
|
||||||
|
const bytes = Buffer.byteLength(text);
|
||||||
|
if (bytes > DEFAULTS.checkpointBytes)
|
||||||
|
throw new AnalysisResourceConstraintError(
|
||||||
|
"analyze_javascript_application",
|
||||||
|
"transport",
|
||||||
|
"JavaScript static checkpoint exceeds its complete-fact byte budget",
|
||||||
|
{
|
||||||
|
checkpoint_bytes: bytes,
|
||||||
|
checkpoint_limit_bytes: DEFAULTS.checkpointBytes,
|
||||||
|
},
|
||||||
|
);
|
||||||
|
await writeFile(join(root, `${request.id}.static.json`), text, {
|
||||||
|
flag: "wx",
|
||||||
|
mode: 0o600,
|
||||||
|
});
|
||||||
|
await send({
|
||||||
|
kind: "static",
|
||||||
|
id: request.id,
|
||||||
|
bytes,
|
||||||
|
sha256: createHash("sha256").update(text).digest("hex"),
|
||||||
|
});
|
||||||
|
const next = await requests.next();
|
||||||
|
if (
|
||||||
|
next.done ||
|
||||||
|
Buffer.byteLength(next.value) > DEFAULTS.protocolBytes
|
||||||
|
)
|
||||||
|
throw new TypeError(
|
||||||
|
"JavaScript worker admission frame is missing or oversized",
|
||||||
|
);
|
||||||
|
const raw: unknown = JSON.parse(next.value);
|
||||||
|
const admission = javaScriptWorkerAdmissionSchema.parse(raw);
|
||||||
|
if (admission.id !== request.id)
|
||||||
|
throw new TypeError(
|
||||||
|
"JavaScript worker admission names a different source job",
|
||||||
|
);
|
||||||
|
return admission.semantic_result_budget_bytes;
|
||||||
|
},
|
||||||
|
},
|
||||||
|
);
|
||||||
|
await send({ kind: "phase", id: request.id, phase: "transfer" });
|
||||||
|
const descriptor =
|
||||||
|
result === null
|
||||||
|
? null
|
||||||
|
: await writeJavaScriptAnalysisTransfer(
|
||||||
|
join(root, `${request.id}.records`),
|
||||||
|
result.module,
|
||||||
|
result.projection,
|
||||||
|
);
|
||||||
|
await send({ kind: "complete", id: request.id, descriptor });
|
||||||
|
} catch (cause: unknown) {
|
||||||
|
await send({
|
||||||
|
kind: "failure",
|
||||||
|
id: request.id,
|
||||||
|
reason:
|
||||||
|
cause instanceof AnalysisResourceConstraintError ||
|
||||||
|
cause instanceof RangeError
|
||||||
|
? "resource"
|
||||||
|
: cause instanceof TypeError
|
||||||
|
? "output"
|
||||||
|
: "io",
|
||||||
|
resource:
|
||||||
|
cause instanceof AnalysisResourceConstraintError
|
||||||
|
? cause.resource
|
||||||
|
: cause instanceof RangeError
|
||||||
|
? "memory"
|
||||||
|
: null,
|
||||||
|
limits:
|
||||||
|
cause instanceof AnalysisResourceConstraintError
|
||||||
|
? cause.reportedLimits
|
||||||
|
: null,
|
||||||
|
message: cause instanceof Error ? cause.message : String(cause),
|
||||||
|
});
|
||||||
|
}
|
||||||
|
};
|
||||||
@@ -184,8 +184,23 @@ const runElectronTool = async <Input>(
|
|||||||
progress: mcpProgressReporter(context),
|
progress: mcpProgressReporter(context),
|
||||||
}),
|
}),
|
||||||
);
|
);
|
||||||
if (!result.ok)
|
if (!result.ok) {
|
||||||
|
const partial = result.error.partialObservation;
|
||||||
|
if (
|
||||||
|
partial !== undefined &&
|
||||||
|
"evidence_id" in partial &&
|
||||||
|
typeof partial.evidence_id === "string"
|
||||||
|
) {
|
||||||
|
const recorded = options.recordEvidence?.(partial);
|
||||||
|
if (recorded !== undefined)
|
||||||
|
return options.delivery.toRecordedPartialErrorToolResult(
|
||||||
|
result.error,
|
||||||
|
partial.evidence_id,
|
||||||
|
recorded,
|
||||||
|
);
|
||||||
|
}
|
||||||
return options.delivery.toCallToolResult(result, contract);
|
return options.delivery.toCallToolResult(result, contract);
|
||||||
|
}
|
||||||
const recorded = options.recordEvidence?.(result.value);
|
const recorded = options.recordEvidence?.(result.value);
|
||||||
return options.delivery.toEvidenceToolResult(
|
return options.delivery.toEvidenceToolResult(
|
||||||
result.value,
|
result.value,
|
||||||
|
|||||||
@@ -3,7 +3,10 @@ import { STDIO_DEFAULT_MAX_BUFFER_SIZE } from "@modelcontextprotocol/server";
|
|||||||
|
|
||||||
import type { ToolContract } from "../contracts/toolContractTypes.js";
|
import type { ToolContract } from "../contracts/toolContractTypes.js";
|
||||||
import type { Evidence } from "../domain/evidence.js";
|
import type { Evidence } from "../domain/evidence.js";
|
||||||
import { projectAnalysisError } from "../domain/analysisErrorProjection.js";
|
import {
|
||||||
|
projectAnalysisError,
|
||||||
|
type AnalysisErrorProjection,
|
||||||
|
} from "../domain/analysisErrorProjection.js";
|
||||||
import { type AnalysisError } from "../domain/analysisErrorBase.js";
|
import { type AnalysisError } from "../domain/analysisErrorBase.js";
|
||||||
import type { JsonValue } from "../domain/jsonValue.js";
|
import type { JsonValue } from "../domain/jsonValue.js";
|
||||||
import type { Result } from "../domain/result.js";
|
import type { Result } from "../domain/result.js";
|
||||||
@@ -64,7 +67,33 @@ export class ToolResultDelivery {
|
|||||||
|
|
||||||
/** Project an error without allocating oversized repeated MCP text. */
|
/** Project an error without allocating oversized repeated MCP text. */
|
||||||
toErrorToolResult(error: AnalysisError): CallToolResult {
|
toErrorToolResult(error: AnalysisError): CallToolResult {
|
||||||
const structuredContent = { error: projectAnalysisError(error) };
|
return this.projectedErrorResult(projectAnalysisError(error));
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Deliver the original failure with the producer-acknowledged partial Evidence reference. */
|
||||||
|
toRecordedPartialErrorToolResult(
|
||||||
|
error: AnalysisError,
|
||||||
|
evidenceId: string,
|
||||||
|
recorded: Result<unknown, AnalysisError>,
|
||||||
|
): CallToolResult {
|
||||||
|
if (!recorded.ok) return this.toErrorToolResult(error);
|
||||||
|
const projected = projectAnalysisError(error);
|
||||||
|
return this.projectedErrorResult({
|
||||||
|
...projected,
|
||||||
|
details: {
|
||||||
|
...projected.details,
|
||||||
|
partial_observation: {
|
||||||
|
kind: "retained-evidence",
|
||||||
|
evidence_id: evidenceId,
|
||||||
|
},
|
||||||
|
},
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
private projectedErrorResult(
|
||||||
|
projected: AnalysisErrorProjection,
|
||||||
|
): CallToolResult {
|
||||||
|
const structuredContent = { error: projected };
|
||||||
// Errors reach the wire as text only; the structured copy is a private
|
// Errors reach the wire as text only; the structured copy is a private
|
||||||
// carrier the transport removes after its own budget check.
|
// carrier the transport removes after its own budget check.
|
||||||
const encoded = encodeToolResult(
|
const encoded = encodeToolResult(
|
||||||
|
|||||||
@@ -0,0 +1,93 @@
|
|||||||
|
import { createHash } from "node:crypto";
|
||||||
|
import { expect, it } from "vitest";
|
||||||
|
import { JavaScriptAnalysisWorker } from "../../../src/javascript/analysis/JavaScriptAnalysisWorker.js";
|
||||||
|
import { spawnOwnedProviderProcess } from "../../../src/process/ProviderProcess.js";
|
||||||
|
import type { JavaScriptArtifactFile } from "../../../src/domain/javascript/javascriptArtifactFiles.js";
|
||||||
|
|
||||||
|
const source = (path: string, value: string): JavaScriptArtifactFile => ({
|
||||||
|
path,
|
||||||
|
sha256: createHash("sha256").update(value).digest("hex"),
|
||||||
|
container_sha256: "1".repeat(64),
|
||||||
|
bytes: Buffer.byteLength(value),
|
||||||
|
inventory_artifact_id: "fixture-source",
|
||||||
|
kind: "javascript",
|
||||||
|
unpacked: false,
|
||||||
|
text: { included: true, value },
|
||||||
|
});
|
||||||
|
|
||||||
|
it("reuses one real owned analyzer while retaining distinct source identities and exported returns", async () => {
|
||||||
|
const launches: number[] = [];
|
||||||
|
const worker = new JavaScriptAnalysisWorker(
|
||||||
|
{ heapMb: 128, timeoutMs: 30_000 },
|
||||||
|
async (request) => {
|
||||||
|
const spawned = await spawnOwnedProviderProcess(request);
|
||||||
|
const pid = spawned.process.pid;
|
||||||
|
if (pid === undefined)
|
||||||
|
throw new Error("Owned analyzer has no observed PID");
|
||||||
|
launches.push(pid);
|
||||||
|
return spawned;
|
||||||
|
},
|
||||||
|
);
|
||||||
|
try {
|
||||||
|
const first = source(
|
||||||
|
"first.mjs",
|
||||||
|
"export function create() { return { observed: 7 }; }",
|
||||||
|
);
|
||||||
|
const second = source(
|
||||||
|
"second.mjs",
|
||||||
|
"export function next() { return { observed: 8 }; }",
|
||||||
|
);
|
||||||
|
for (const [index, file] of [first, second].entries()) {
|
||||||
|
const result = await worker.analyze(file, 100);
|
||||||
|
if (!result.ok) throw result.error.error;
|
||||||
|
expect(result.value.javascript.parse_status).toBe("complete");
|
||||||
|
expect(
|
||||||
|
result.value.module?.callables[0]?.returnSites[0]?.value,
|
||||||
|
).toMatchObject({
|
||||||
|
status: "object",
|
||||||
|
properties: [
|
||||||
|
{ name: "observed", value: { status: "literal", value: index + 7 } },
|
||||||
|
],
|
||||||
|
});
|
||||||
|
expect(
|
||||||
|
result.value.projection?.nodes.every(
|
||||||
|
({ identity }) => identity.artifact_sha256 === file.sha256,
|
||||||
|
),
|
||||||
|
).toBe(true);
|
||||||
|
expect(
|
||||||
|
result.value.projection?.nodes.some(
|
||||||
|
({ identity }) => identity.module_path === file.path,
|
||||||
|
),
|
||||||
|
).toBe(true);
|
||||||
|
}
|
||||||
|
expect(launches).toHaveLength(1);
|
||||||
|
} finally {
|
||||||
|
await worker.close();
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
it("reports an actual startup deadline and closes its owned resources before subsequent analysis", async () => {
|
||||||
|
const file = source("app.mjs", "export const observed = 7;");
|
||||||
|
const expired = new JavaScriptAnalysisWorker({ heapMb: 128, timeoutMs: 1 });
|
||||||
|
try {
|
||||||
|
const result = await expired.analyze(file, 100);
|
||||||
|
if (result.ok)
|
||||||
|
throw new Error(
|
||||||
|
"A one-millisecond fresh process deadline unexpectedly completed",
|
||||||
|
);
|
||||||
|
expect(result.error.error).toMatchObject({
|
||||||
|
_tag: "AnalysisTimeoutError",
|
||||||
|
timeoutMs: 1,
|
||||||
|
});
|
||||||
|
} finally {
|
||||||
|
await expired.close();
|
||||||
|
}
|
||||||
|
const next = new JavaScriptAnalysisWorker({ heapMb: 128, timeoutMs: 30_000 });
|
||||||
|
try {
|
||||||
|
const result = await next.analyze(file, 100);
|
||||||
|
if (!result.ok) throw result.error.error;
|
||||||
|
expect(result.value.javascript.parse_status).toBe("complete");
|
||||||
|
} finally {
|
||||||
|
await next.close();
|
||||||
|
}
|
||||||
|
});
|
||||||
@@ -12,9 +12,10 @@ import {
|
|||||||
} from "../../fixtures/binarySession.js";
|
} from "../../fixtures/binarySession.js";
|
||||||
import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js";
|
import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js";
|
||||||
|
|
||||||
it("stops a cancelled SDK request on the server, retains prior Evidence, and accepts subsequent work", async () => {
|
it("retains completed source facts and prior Evidence after SDK cancellation, then accepts subsequent work", async () => {
|
||||||
const root = await createTestTempDirectory("rea-mcp-js-cancellation-");
|
const root = await createTestTempDirectory("rea-mcp-js-cancellation-");
|
||||||
await writeFile(join(root, "main.js"), "export const observed = 1;\n");
|
await writeFile(join(root, "main.js"), "export const observed = 1;\n");
|
||||||
|
await writeFile(join(root, "second.js"), "export const pending = 2;\n");
|
||||||
const session = createTestBinarySession(() => {
|
const session = createTestBinarySession(() => {
|
||||||
throw new Error(
|
throw new Error(
|
||||||
"Static JavaScript analysis must not start a binary provider",
|
"Static JavaScript analysis must not start a binary provider",
|
||||||
@@ -74,7 +75,12 @@ it("stops a cancelled SDK request on the server, retains prior Evidence, and acc
|
|||||||
},
|
},
|
||||||
{
|
{
|
||||||
signal: controller.signal,
|
signal: controller.signal,
|
||||||
onprogress() {
|
onprogress(update) {
|
||||||
|
if (
|
||||||
|
!update.message?.startsWith("parse_javascript_source:") ||
|
||||||
|
!update.message.includes("second.js")
|
||||||
|
)
|
||||||
|
return;
|
||||||
requested = true;
|
requested = true;
|
||||||
controller.abort(new Error("SDK cancellation verification"));
|
controller.abort(new Error("SDK cancellation verification"));
|
||||||
},
|
},
|
||||||
@@ -89,15 +95,38 @@ it("stops a cancelled SDK request on the server, retains prior Evidence, and acc
|
|||||||
name: "get_evidence_bundle",
|
name: "get_evidence_bundle",
|
||||||
arguments: {},
|
arguments: {},
|
||||||
});
|
});
|
||||||
|
const records = z
|
||||||
|
.object({
|
||||||
|
result: z.object({
|
||||||
|
records: z.array(z.object({ evidence_id: z.string() })),
|
||||||
|
}),
|
||||||
|
})
|
||||||
|
.parse(bundle.structuredContent).result.records;
|
||||||
|
expect(records).toContainEqual({ evidence_id: prior.evidence_id });
|
||||||
|
const partial = records.find(
|
||||||
|
({ evidence_id }) => evidence_id !== prior.evidence_id,
|
||||||
|
);
|
||||||
|
if (partial === undefined)
|
||||||
|
throw new Error("Cancelled analysis discarded its completed source facts");
|
||||||
expect(
|
expect(
|
||||||
z
|
z
|
||||||
.object({
|
.object({
|
||||||
result: z.object({
|
statistics: z.object({
|
||||||
records: z.array(z.object({ evidence_id: z.string() })),
|
parsed_javascript_files: z.number(),
|
||||||
|
parse_failures: z.number(),
|
||||||
}),
|
}),
|
||||||
})
|
})
|
||||||
.parse(bundle.structuredContent).result.records,
|
.parse(session.evidenceById(partial.evidence_id)?.normalized_result)
|
||||||
).toEqual([{ evidence_id: prior.evidence_id }]);
|
.statistics,
|
||||||
|
).toEqual({ parsed_javascript_files: 1, parse_failures: 0 });
|
||||||
|
const view = await client.callTool({
|
||||||
|
name: "inspect_analysis_view",
|
||||||
|
arguments: {
|
||||||
|
source: { kind: "retained-evidence", evidence_id: partial.evidence_id },
|
||||||
|
view: { kind: "summary" },
|
||||||
|
},
|
||||||
|
});
|
||||||
|
expect(view.isError, JSON.stringify(view)).not.toBe(true);
|
||||||
const next = await client.callTool({
|
const next = await client.callTool({
|
||||||
name: "analyze_javascript_application",
|
name: "analyze_javascript_application",
|
||||||
arguments: {
|
arguments: {
|
||||||
|
|||||||
@@ -0,0 +1,331 @@
|
|||||||
|
import { Client } from "@modelcontextprotocol/client";
|
||||||
|
import { StdioClientTransport } from "@modelcontextprotocol/client/stdio";
|
||||||
|
import { createHash } from "node:crypto";
|
||||||
|
import { readFile, writeFile } from "node:fs/promises";
|
||||||
|
import { join, resolve } from "node:path";
|
||||||
|
import { expect, it } from "vitest";
|
||||||
|
import { z } from "zod";
|
||||||
|
|
||||||
|
import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js";
|
||||||
|
|
||||||
|
it("retains every static finding through public module views when application expansion exceeds memory, then accepts more work", async () => {
|
||||||
|
const root = await createTestTempDirectory("rea-js-resource-boundary-");
|
||||||
|
const nextRoot = await createTestTempDirectory("rea-js-resource-recovery-");
|
||||||
|
await writeFile(join(nextRoot, "main.js"), "export const recovered = 9;\n");
|
||||||
|
const source = Array.from(
|
||||||
|
{ length: 4000 },
|
||||||
|
(_, index) =>
|
||||||
|
`import { observed as binding${String(index)} } from "./target.js";`,
|
||||||
|
).join("\n");
|
||||||
|
const path = join(root, "main.js");
|
||||||
|
await writeFile(path, source);
|
||||||
|
await writeFile(join(root, "target.js"), "export const observed = 7;\n");
|
||||||
|
const digest = createHash("sha256").update(source).digest("hex");
|
||||||
|
const { client, transport, diagnostics } = createResourceClient();
|
||||||
|
try {
|
||||||
|
await client.connect(transport);
|
||||||
|
const response = await client.callTool(
|
||||||
|
{
|
||||||
|
name: "analyze_javascript_application",
|
||||||
|
arguments: {
|
||||||
|
input_path: root,
|
||||||
|
format: "directory",
|
||||||
|
max_heap_mb: 128,
|
||||||
|
detail: "summary",
|
||||||
|
},
|
||||||
|
},
|
||||||
|
{ timeout: 60000 },
|
||||||
|
);
|
||||||
|
expect(response.isError).not.toBe(true);
|
||||||
|
const summary = z
|
||||||
|
.object({
|
||||||
|
normalized_result: z.object({
|
||||||
|
parent_evidence_id: z.string(),
|
||||||
|
summary: z.object({
|
||||||
|
coverage: z.object({
|
||||||
|
application: z.object({ status: z.string() }),
|
||||||
|
}),
|
||||||
|
}),
|
||||||
|
}),
|
||||||
|
})
|
||||||
|
.parse(response.structuredContent).normalized_result;
|
||||||
|
expect(summary.summary.coverage.application.status).toBe("partial");
|
||||||
|
const retained = {
|
||||||
|
kind: "retained-evidence",
|
||||||
|
evidence_id: summary.parent_evidence_id,
|
||||||
|
};
|
||||||
|
const pageResponse = await client.callTool({
|
||||||
|
name: "inspect_analysis_view",
|
||||||
|
arguments: {
|
||||||
|
source: retained,
|
||||||
|
view: { kind: "page", collection: "modules", offset: 0, limit: 100 },
|
||||||
|
},
|
||||||
|
});
|
||||||
|
expect(pageResponse.isError).not.toBe(true);
|
||||||
|
const page = z
|
||||||
|
.object({
|
||||||
|
normalized_result: z.object({
|
||||||
|
items: z.array(
|
||||||
|
z.object({
|
||||||
|
node_id: z.string(),
|
||||||
|
kind: z.string(),
|
||||||
|
path: z.string().nullable(),
|
||||||
|
}),
|
||||||
|
),
|
||||||
|
}),
|
||||||
|
})
|
||||||
|
.parse(pageResponse.structuredContent).normalized_result;
|
||||||
|
const module = page.items.find(
|
||||||
|
(item) => item.kind === "javascript-asset" && item.path === "main.js",
|
||||||
|
);
|
||||||
|
if (module === undefined)
|
||||||
|
throw new Error("The analyzed file disappeared from public module views");
|
||||||
|
const itemResponse = await client.callTool({
|
||||||
|
name: "inspect_analysis_view",
|
||||||
|
arguments: {
|
||||||
|
source: retained,
|
||||||
|
view: {
|
||||||
|
kind: "item",
|
||||||
|
collection: "modules",
|
||||||
|
selector: { node_id: module.node_id },
|
||||||
|
},
|
||||||
|
},
|
||||||
|
});
|
||||||
|
expect(itemResponse.isError).not.toBe(true);
|
||||||
|
const item = z
|
||||||
|
.object({
|
||||||
|
normalized_result: z.object({
|
||||||
|
item: z.object({
|
||||||
|
identity: z.object({ sha256: z.string() }),
|
||||||
|
observations: z.array(
|
||||||
|
z.object({
|
||||||
|
properties: z.object({
|
||||||
|
static_analysis: z
|
||||||
|
.object({
|
||||||
|
parse_status: z.string(),
|
||||||
|
references: z.array(
|
||||||
|
z.object({
|
||||||
|
specifier: z.string(),
|
||||||
|
location: z.object({
|
||||||
|
start: z.object({ line: z.number() }),
|
||||||
|
}),
|
||||||
|
}),
|
||||||
|
),
|
||||||
|
})
|
||||||
|
.optional(),
|
||||||
|
}),
|
||||||
|
}),
|
||||||
|
),
|
||||||
|
}),
|
||||||
|
}),
|
||||||
|
})
|
||||||
|
.parse(itemResponse.structuredContent).normalized_result.item;
|
||||||
|
expect(item.identity.sha256).toBe(digest);
|
||||||
|
const facts = item.observations.find(
|
||||||
|
(observation) => observation.properties.static_analysis !== undefined,
|
||||||
|
)?.properties.static_analysis;
|
||||||
|
if (facts === undefined)
|
||||||
|
throw new Error("Completed static observations were discarded");
|
||||||
|
expect(facts.parse_status).toBe("complete");
|
||||||
|
expect(
|
||||||
|
facts.references.map((reference) => [
|
||||||
|
reference.specifier,
|
||||||
|
reference.location.start.line,
|
||||||
|
]),
|
||||||
|
).toEqual(
|
||||||
|
Array.from({ length: 4000 }, (_, index) => ["./target.js", index + 1]),
|
||||||
|
);
|
||||||
|
await client.ping();
|
||||||
|
const next = await client.callTool({
|
||||||
|
name: "analyze_javascript_application",
|
||||||
|
arguments: {
|
||||||
|
input_path: nextRoot,
|
||||||
|
format: "directory",
|
||||||
|
max_heap_mb: 128,
|
||||||
|
detail: "summary",
|
||||||
|
},
|
||||||
|
});
|
||||||
|
expect(next.isError).not.toBe(true);
|
||||||
|
await client.ping();
|
||||||
|
expect(
|
||||||
|
createHash("sha256")
|
||||||
|
.update(await readFile(path))
|
||||||
|
.digest("hex"),
|
||||||
|
).toBe(digest);
|
||||||
|
expect(diagnostics()).not.toMatch(/FATAL ERROR:.*heap/iu);
|
||||||
|
} finally {
|
||||||
|
await client.close();
|
||||||
|
await transport.close();
|
||||||
|
}
|
||||||
|
}, 90000);
|
||||||
|
|
||||||
|
it("returns a directly usable partial analysis reference for an oversized timeout observation", async () => {
|
||||||
|
const root = await createTestTempDirectory("rea-js-large-timeout-");
|
||||||
|
for (let offset = 0; offset < 1500; offset += 50)
|
||||||
|
await Promise.all(
|
||||||
|
Array.from({ length: 50 }, (_, index) =>
|
||||||
|
writeFile(
|
||||||
|
join(root, `file-${String(offset + index)}.js`),
|
||||||
|
"export const observed = 7;\n",
|
||||||
|
),
|
||||||
|
),
|
||||||
|
);
|
||||||
|
const nextRoot = await createTestTempDirectory("rea-js-timeout-recovery-");
|
||||||
|
await writeFile(join(nextRoot, "main.js"), "export const recovered = 9;\n");
|
||||||
|
const { client, transport, diagnostics } = createResourceClient();
|
||||||
|
try {
|
||||||
|
await client.connect(transport);
|
||||||
|
const response = await client.callTool(
|
||||||
|
{
|
||||||
|
name: "analyze_javascript_application",
|
||||||
|
arguments: {
|
||||||
|
input_path: root,
|
||||||
|
format: "directory",
|
||||||
|
max_heap_mb: 128,
|
||||||
|
analysis_timeout_ms: 1,
|
||||||
|
},
|
||||||
|
},
|
||||||
|
{ timeout: 60000 },
|
||||||
|
);
|
||||||
|
expect(response.isError).toBe(true);
|
||||||
|
const text = response.content.find((item) => item.type === "text");
|
||||||
|
if (text?.type !== "text")
|
||||||
|
throw new Error("Missing actionable timeout response");
|
||||||
|
const projected = z
|
||||||
|
.object({
|
||||||
|
error: z.object({
|
||||||
|
code: z.string(),
|
||||||
|
details: z.object({
|
||||||
|
partial_observation: z.object({
|
||||||
|
kind: z.literal("retained-evidence"),
|
||||||
|
evidence_id: z.string(),
|
||||||
|
}),
|
||||||
|
}),
|
||||||
|
}),
|
||||||
|
})
|
||||||
|
.parse(JSON.parse(text.text));
|
||||||
|
expect(projected.error.code).toBe("provider_timeout");
|
||||||
|
const summaryResponse = await client.callTool({
|
||||||
|
name: "inspect_analysis_view",
|
||||||
|
arguments: {
|
||||||
|
source: projected.error.details.partial_observation,
|
||||||
|
view: { kind: "summary" },
|
||||||
|
},
|
||||||
|
});
|
||||||
|
expect(summaryResponse.isError).not.toBe(true);
|
||||||
|
const summary = z
|
||||||
|
.object({
|
||||||
|
normalized_result: z.object({
|
||||||
|
summary: z.object({
|
||||||
|
statistics: z.object({
|
||||||
|
relevant_files: z.number(),
|
||||||
|
parsed_javascript_files: z.number(),
|
||||||
|
parse_failures: z.number(),
|
||||||
|
}),
|
||||||
|
coverage: z.object({ semantic_unknowns: z.number() }),
|
||||||
|
}),
|
||||||
|
}),
|
||||||
|
})
|
||||||
|
.parse(summaryResponse.structuredContent).normalized_result.summary;
|
||||||
|
expect(summary.statistics).toEqual({
|
||||||
|
relevant_files: 1500,
|
||||||
|
parsed_javascript_files: 0,
|
||||||
|
parse_failures: 0,
|
||||||
|
});
|
||||||
|
expect(summary.coverage.semantic_unknowns).toBeGreaterThan(1500);
|
||||||
|
await verifyRepeatedSourcePaths(
|
||||||
|
client,
|
||||||
|
projected.error.details.partial_observation,
|
||||||
|
);
|
||||||
|
await client.ping();
|
||||||
|
const next = await client.callTool({
|
||||||
|
name: "analyze_javascript_application",
|
||||||
|
arguments: {
|
||||||
|
input_path: nextRoot,
|
||||||
|
format: "directory",
|
||||||
|
max_heap_mb: 128,
|
||||||
|
detail: "summary",
|
||||||
|
},
|
||||||
|
});
|
||||||
|
expect(next.isError).not.toBe(true);
|
||||||
|
await client.ping();
|
||||||
|
expect(diagnostics()).not.toMatch(/FATAL ERROR:.*heap/iu);
|
||||||
|
} finally {
|
||||||
|
await client.close();
|
||||||
|
await transport.close();
|
||||||
|
}
|
||||||
|
}, 90000);
|
||||||
|
|
||||||
|
const createResourceClient = () => {
|
||||||
|
const client = new Client({ name: "javascript-resources", version: "1" });
|
||||||
|
const transport = new StdioClientTransport({
|
||||||
|
command: process.execPath,
|
||||||
|
args: [resolve("scripts/rea.mjs"), "mcp"],
|
||||||
|
cwd: process.cwd(),
|
||||||
|
env: {
|
||||||
|
PATH: process.env.PATH ?? "",
|
||||||
|
NODE_OPTIONS: "--max-old-space-size=512 --max-semi-space-size=8",
|
||||||
|
},
|
||||||
|
stderr: "pipe",
|
||||||
|
});
|
||||||
|
let diagnostics = "";
|
||||||
|
transport.stderr?.on("data", (chunk: Buffer) => {
|
||||||
|
if (diagnostics.length < 262144) diagnostics += chunk.toString();
|
||||||
|
});
|
||||||
|
return { client, transport, diagnostics: () => diagnostics };
|
||||||
|
};
|
||||||
|
|
||||||
|
const verifyRepeatedSourcePaths = async (
|
||||||
|
client: Client,
|
||||||
|
source: { readonly kind: "retained-evidence"; readonly evidence_id: string },
|
||||||
|
): Promise<void> => {
|
||||||
|
const page = await client.callTool({
|
||||||
|
name: "inspect_analysis_view",
|
||||||
|
arguments: {
|
||||||
|
source,
|
||||||
|
view: { kind: "page", collection: "modules", offset: 0, limit: 4 },
|
||||||
|
},
|
||||||
|
});
|
||||||
|
expect(page.isError).not.toBe(true);
|
||||||
|
const modules = z
|
||||||
|
.object({
|
||||||
|
normalized_result: z.object({
|
||||||
|
items: z.array(z.object({ kind: z.string(), node_id: z.string() })),
|
||||||
|
}),
|
||||||
|
})
|
||||||
|
.parse(page.structuredContent);
|
||||||
|
const asset = modules.normalized_result.items.find(
|
||||||
|
({ kind }) => kind === "javascript-asset",
|
||||||
|
);
|
||||||
|
if (asset === undefined)
|
||||||
|
throw new Error(
|
||||||
|
"Identical source files disappeared from the retained graph",
|
||||||
|
);
|
||||||
|
const item = await client.callTool({
|
||||||
|
name: "inspect_analysis_view",
|
||||||
|
arguments: {
|
||||||
|
source,
|
||||||
|
view: {
|
||||||
|
kind: "item",
|
||||||
|
collection: "modules",
|
||||||
|
selector: { node_id: asset.node_id },
|
||||||
|
},
|
||||||
|
},
|
||||||
|
});
|
||||||
|
expect(item.isError).not.toBe(true);
|
||||||
|
const observations = z
|
||||||
|
.object({
|
||||||
|
normalized_result: z.object({
|
||||||
|
item: z.object({
|
||||||
|
observations: z.array(z.object({ label: z.string() })),
|
||||||
|
}),
|
||||||
|
}),
|
||||||
|
})
|
||||||
|
.parse(item.structuredContent).normalized_result.item.observations;
|
||||||
|
expect(observations.map(({ label }) => label).sort()).toEqual(
|
||||||
|
Array.from(
|
||||||
|
{ length: 1500 },
|
||||||
|
(_, index) => `file-${String(index)}.js`,
|
||||||
|
).sort(),
|
||||||
|
);
|
||||||
|
};
|
||||||
+134
-3
@@ -3,7 +3,7 @@ import {
|
|||||||
primitiveByteExpansionSource,
|
primitiveByteExpansionSource,
|
||||||
} from "../../fixtures/javascriptPrimitiveExpansion.js";
|
} from "../../fixtures/javascriptPrimitiveExpansion.js";
|
||||||
import { createHash } from "node:crypto";
|
import { createHash } from "node:crypto";
|
||||||
import { mkdir, rm, writeFile } from "node:fs/promises";
|
import { mkdir, readFile, rm, writeFile } from "node:fs/promises";
|
||||||
import { join } from "node:path";
|
import { join } from "node:path";
|
||||||
|
|
||||||
import { createPackageWithOptions } from "@electron/asar";
|
import { createPackageWithOptions } from "@electron/asar";
|
||||||
@@ -23,7 +23,20 @@ import { buildJavaScriptArtifactGraph } from "../../../src/application/javascrip
|
|||||||
import {
|
import {
|
||||||
buildJavaScriptSemanticGraph,
|
buildJavaScriptSemanticGraph,
|
||||||
createJavaScriptSemanticGraphProjection,
|
createJavaScriptSemanticGraphProjection,
|
||||||
|
projectJavaScriptSemanticFileSteps,
|
||||||
} from "../../../src/application/javascript/JavaScriptSemanticGraphBuilder.js";
|
} from "../../../src/application/javascript/JavaScriptSemanticGraphBuilder.js";
|
||||||
|
import {
|
||||||
|
projectJavaScriptModuleSemantics,
|
||||||
|
type JavaScriptModuleSemanticIr,
|
||||||
|
} from "../../../src/domain/javascript/javascriptModuleSemanticIr.js";
|
||||||
|
import type { JavaScriptStaticAnalysis } from "../../../src/domain/javascript/javascriptStaticAnalysisTypes.js";
|
||||||
|
import type { JavaScriptArtifactFile } from "../../../src/domain/javascript/javascriptArtifactFiles.js";
|
||||||
|
import { completeJavaScriptAnalysisSteps } from "../../../src/application/javascript/JavaScriptAnalysisControl.js";
|
||||||
|
import {
|
||||||
|
writeJavaScriptAnalysisTransfer,
|
||||||
|
readJavaScriptAnalysisTransfer,
|
||||||
|
} from "../../../src/javascript/analysis/JavaScriptAnalysisTransfer.js";
|
||||||
|
import { javaScriptWorkerStaticAnalysisSchema } from "../../../src/javascript/analysis/JavaScriptAnalysisWorkerSchemas.js";
|
||||||
import { createJavaScriptArtifactReader } from "../../../src/artifacts/javascript/JavaScriptArtifactReader.js";
|
import { createJavaScriptArtifactReader } from "../../../src/artifacts/javascript/JavaScriptArtifactReader.js";
|
||||||
import {
|
import {
|
||||||
scanCanonicalArtifactInventory,
|
scanCanonicalArtifactInventory,
|
||||||
@@ -31,7 +44,10 @@ import {
|
|||||||
} from "../../fixtures/artifactInventory.js";
|
} from "../../fixtures/artifactInventory.js";
|
||||||
import { parseJavaScriptApplicationGraph } from "../../../src/domain/javascript/javascriptApplicationGraph.js";
|
import { parseJavaScriptApplicationGraph } from "../../../src/domain/javascript/javascriptApplicationGraph.js";
|
||||||
import { javascriptApplicationAnalysisResultSchema } from "../../../src/domain/javascript/javascriptApplicationAnalysis.js";
|
import { javascriptApplicationAnalysisResultSchema } from "../../../src/domain/javascript/javascriptApplicationAnalysis.js";
|
||||||
import { createJavaScriptSemanticGraph } from "../../../src/domain/javascript/javascriptSemanticGraph.js";
|
import {
|
||||||
|
createJavaScriptSemanticGraph,
|
||||||
|
isValidatedImmutableJavaScriptSemanticGraph,
|
||||||
|
} from "../../../src/domain/javascript/javascriptSemanticGraph.js";
|
||||||
import { parseJavaScriptSemanticGraph } from "../../../src/domain/javascript/javascriptSemanticGraphSerialization.js";
|
import { parseJavaScriptSemanticGraph } from "../../../src/domain/javascript/javascriptSemanticGraphSerialization.js";
|
||||||
import { writeJavaScriptArtifactFixture } from "../../fixtures/javascriptArtifactApplication.js";
|
import { writeJavaScriptArtifactFixture } from "../../fixtures/javascriptArtifactApplication.js";
|
||||||
|
|
||||||
@@ -48,7 +64,10 @@ it("reports file progress and honors cancellation before the next source", async
|
|||||||
}
|
}
|
||||||
},
|
},
|
||||||
}),
|
}),
|
||||||
).rejects.toMatchObject({ reason: "cancelled" });
|
).rejects.toMatchObject({
|
||||||
|
_tag: "AnalysisCancelledError",
|
||||||
|
partialObservation: { operation: "analyze_javascript_application" },
|
||||||
|
});
|
||||||
expect(progress).toHaveLength(2);
|
expect(progress).toHaveLength(2);
|
||||||
expect(progress[0]).not.toBe(progress[1]);
|
expect(progress[0]).not.toBe(progress[1]);
|
||||||
});
|
});
|
||||||
@@ -61,6 +80,7 @@ it("preserves graph commitments and export shapes when consuming file-local IR",
|
|||||||
import { readFile } from "node:fs";
|
import { readFile } from "node:fs";
|
||||||
export { readFile as read };
|
export { readFile as read };
|
||||||
export function create(value) { return { value, nested: { enabled: true } }; }
|
export function create(value) { return { value, nested: { enabled: true } }; }
|
||||||
|
export function configured(condition) { return { values: [1, null, undefined], choice: condition ? 1 : 2, ...external }; }
|
||||||
function unexported() { return "private"; }
|
function unexported() { return "private"; }
|
||||||
`,
|
`,
|
||||||
);
|
);
|
||||||
@@ -96,6 +116,7 @@ it("preserves graph commitments and export shapes when consuming file-local IR",
|
|||||||
);
|
);
|
||||||
expect(exported?.semantic?.ir.callables.map(({ name }) => name)).toEqual([
|
expect(exported?.semantic?.ir.callables.map(({ name }) => name)).toEqual([
|
||||||
"create",
|
"create",
|
||||||
|
"configured",
|
||||||
]);
|
]);
|
||||||
expect(
|
expect(
|
||||||
full.files
|
full.files
|
||||||
@@ -103,11 +124,121 @@ it("preserves graph commitments and export shapes when consuming file-local IR",
|
|||||||
?.semantic?.ir.callables.map(({ name }) => name),
|
?.semantic?.ir.callables.map(({ name }) => name),
|
||||||
).toContain("unexported");
|
).toContain("unexported");
|
||||||
expect(parseJavaScriptSemanticGraph(semantic)).toEqual(semantic);
|
expect(parseJavaScriptSemanticGraph(semantic)).toEqual(semantic);
|
||||||
|
const transferred = createJavaScriptSemanticGraphProjection();
|
||||||
|
const wireRoot = await createTestTempDirectory("rea-javascript-transfer-");
|
||||||
|
const decodedModules = new Map<string, JavaScriptModuleSemanticIr>();
|
||||||
|
const decodedStatic = new Map<string, JavaScriptStaticAnalysis>();
|
||||||
|
let rejectedInvalidTransfer = false;
|
||||||
|
for (const [index, analyzed] of full.files.entries()) {
|
||||||
|
if (analyzed.javascript !== null) {
|
||||||
|
const raw: unknown = JSON.parse(JSON.stringify(analyzed.javascript));
|
||||||
|
const decodedFacts = javaScriptWorkerStaticAnalysisSchema.parse(raw);
|
||||||
|
expect(decodedFacts).toEqual(analyzed.javascript);
|
||||||
|
decodedStatic.set(analyzed.file.path, decodedFacts);
|
||||||
|
}
|
||||||
|
if (analyzed.semantic === null) continue;
|
||||||
|
const fragment = await completeJavaScriptAnalysisSteps(
|
||||||
|
projectJavaScriptSemanticFileSteps(
|
||||||
|
analyzed.file,
|
||||||
|
analyzed.semantic.ir,
|
||||||
|
transferred.remainingFileNodeBudget(),
|
||||||
|
),
|
||||||
|
);
|
||||||
|
const module = projectJavaScriptModuleSemantics(analyzed.semantic.ir);
|
||||||
|
const wirePath = join(wireRoot, `${String(index)}.jsonl`);
|
||||||
|
const descriptor = await writeJavaScriptAnalysisTransfer(
|
||||||
|
wirePath,
|
||||||
|
module,
|
||||||
|
fragment,
|
||||||
|
);
|
||||||
|
const decoded = await readJavaScriptAnalysisTransfer(
|
||||||
|
wirePath,
|
||||||
|
descriptor,
|
||||||
|
analyzed.file,
|
||||||
|
);
|
||||||
|
expect(decoded.module).toEqual(module);
|
||||||
|
decodedModules.set(analyzed.file.path, decoded.module);
|
||||||
|
transferred.adoptFileProjection(decoded.projection);
|
||||||
|
if (!rejectedInvalidTransfer) {
|
||||||
|
await rejectInvalidTransfer(
|
||||||
|
wirePath,
|
||||||
|
descriptor,
|
||||||
|
analyzed.file,
|
||||||
|
wireRoot,
|
||||||
|
);
|
||||||
|
rejectedInvalidTransfer = true;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
const decodedGraph = buildJavaScriptArtifactGraph(snapshot, files, {
|
||||||
|
...compact,
|
||||||
|
files: compact.files.map((analyzed) => {
|
||||||
|
const module = decodedModules.get(analyzed.file.path);
|
||||||
|
return {
|
||||||
|
...analyzed,
|
||||||
|
javascript:
|
||||||
|
decodedStatic.get(analyzed.file.path) ?? analyzed.javascript,
|
||||||
|
semantic: module === undefined ? analyzed.semantic : { ir: module },
|
||||||
|
};
|
||||||
|
}),
|
||||||
|
});
|
||||||
|
expect(decodedGraph).toEqual(graph);
|
||||||
|
const resealed = await completeJavaScriptAnalysisSteps(
|
||||||
|
transferred.finishImmutableSteps(
|
||||||
|
snapshot.manifest.root_sha256,
|
||||||
|
decodedGraph,
|
||||||
|
),
|
||||||
|
);
|
||||||
|
expect(resealed).toEqual(semantic);
|
||||||
|
expect(isValidatedImmutableJavaScriptSemanticGraph(resealed)).toBe(true);
|
||||||
} finally {
|
} finally {
|
||||||
await reader.close();
|
await reader.close();
|
||||||
}
|
}
|
||||||
});
|
});
|
||||||
|
|
||||||
|
const rejectInvalidTransfer = async (
|
||||||
|
wirePath: string,
|
||||||
|
descriptor: Awaited<ReturnType<typeof writeJavaScriptAnalysisTransfer>>,
|
||||||
|
file: JavaScriptArtifactFile,
|
||||||
|
wireRoot: string,
|
||||||
|
): Promise<void> => {
|
||||||
|
await expect(
|
||||||
|
readJavaScriptAnalysisTransfer(wirePath, descriptor, {
|
||||||
|
...file,
|
||||||
|
sha256: "0".repeat(64),
|
||||||
|
}),
|
||||||
|
).rejects.toThrow("different artifact source");
|
||||||
|
await expect(
|
||||||
|
readJavaScriptAnalysisTransfer(
|
||||||
|
wirePath,
|
||||||
|
{ ...descriptor, sha256: "0".repeat(64) },
|
||||||
|
file,
|
||||||
|
),
|
||||||
|
).rejects.toThrow("digest does not match");
|
||||||
|
const wire = await readFile(wirePath, "utf8");
|
||||||
|
const duplicate = wire
|
||||||
|
.split("\n")
|
||||||
|
.find((line) => line.startsWith('{"kind":"node"'));
|
||||||
|
if (duplicate === undefined)
|
||||||
|
throw new Error("Fixture emitted no semantic node record");
|
||||||
|
const invalid = `${wire}${duplicate}\n`;
|
||||||
|
const invalidPath = join(wireRoot, "duplicate.jsonl");
|
||||||
|
await writeFile(invalidPath, invalid);
|
||||||
|
await expect(
|
||||||
|
readJavaScriptAnalysisTransfer(
|
||||||
|
invalidPath,
|
||||||
|
{
|
||||||
|
...descriptor,
|
||||||
|
bytes: Buffer.byteLength(invalid),
|
||||||
|
records: descriptor.records + 1,
|
||||||
|
graph_bytes: descriptor.graph_bytes + Buffer.byteLength(duplicate) + 1,
|
||||||
|
graph_records: descriptor.graph_records + 1,
|
||||||
|
sha256: createHash("sha256").update(invalid).digest("hex"),
|
||||||
|
},
|
||||||
|
file,
|
||||||
|
),
|
||||||
|
).rejects.toThrow("Duplicate JavaScript transfer node identity");
|
||||||
|
};
|
||||||
|
|
||||||
it("reports semantic value resource limits in application graph coverage", async () => {
|
it("reports semantic value resource limits in application graph coverage", async () => {
|
||||||
const root = await createTestTempDirectory("rea-javascript-semantic-limit-");
|
const root = await createTestTempDirectory("rea-javascript-semantic-limit-");
|
||||||
await writeFile(join(root, "app.js"), primitiveCandidateExpansionSource());
|
await writeFile(join(root, "app.js"), primitiveCandidateExpansionSource());
|
||||||
|
|||||||
Reference in New Issue
Block a user