fix(desktop): charge retained rows at their measured representation (#8105)

* fix(desktop): charge retained rows at their measured representation

The desktop scan reserved 1024 + 24 bytes per path character for every
row, omission and the root against its 32 MiB representation budget. Real
rows cost 800-940 heap and 490-640 JSON bytes, so the charge overstated
them 6-13 times and large macOS bundles stopped at 3,700-7,200 entries,
far below max_entries.

Charge each path at the larger of its UTF-16 heap size and its escaped
JSON size, plus the existing fixed row and omission overheads, in one
helper. The bound still covers the retained heap and the serialized
report, including control-character paths that expand six times in JSON.

Fixes #8104

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* fix(desktop): charge every retained copy of a path

The tighter row charge counted each path once, but rows repeat it in
convention-fact values and the next probes those facts offer, and host
failure issues name the absolute path. Control-character paths under a
convention such as Content/Paks or resources/app.asar therefore serialized
47-80 MiB of reports within the 32 MiB budget.

Reserve each copy through the same text helper where it is stored: path
convention facts with their row, the probes any fact offers (at most two,
pinned by a projection test) with that fact, retained read ranges before
their read, and probe failure issues before they are recorded.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

---------

Co-authored-by: Claude Opus 5.5 <[email protected]>

docs/desktop-applications.md

@@ -135,11 +135,13 @@ observations, not an atomic directory snapshot. Unread bytes remain unknown.
 ## Resource bounds and reuse
 
 Defaults are 20,000 examined entries and 8 MiB of total probe reads. Callers may
-select up to 100,000 entries and 64 MiB. A separate 32 MiB conservative
-representation budget bounds retained paths and facts before growth. Limits
-and failed probes produce partial coverage. Traversal and reads are serial.
-Each reader admits one inspection at a time; overlapping requests receive a
-retryable capacity refusal without cancelling the admitted caller.
+select up to 100,000 entries and 64 MiB. A separate 32 MiB representation
+budget bounds retained rows, paths and facts before growth. Each retained path
+is charged at the larger of its UTF-16 heap size and its escaped JSON size, so
+typical macOS bundle paths fit about 20,000 rows, and control-character paths
+fewer. Limits and failed probes produce partial coverage. Traversal and reads
+are serial. Each reader admits one inspection at a time; overlapping requests
+receive a retryable capacity refusal without cancelling the admitted caller.
 
 Unity reference decoding stops after a witness in each `Data/Managed` directory;
 other assemblies' references remain unexamined, while their headers are still

src/artifacts/DesktopApplicationReader.ts

@@ -16,12 +16,16 @@ import { abortIfNeeded } from "./ArtifactHash.js";
 import { translateArtifactFailure } from "./ArtifactOperationFailure.js";
 import {
   inspectDesktopFile,
+  RETAINED_FACT_BYTES,
+  retainedFactReferenceBytes,
+  retainedTextBytes,
   type DesktopReadBudget,
 } from "./DesktopFileInspection.js";
 import type { DesktopApplicationPort } from "../application/artifacts/DesktopApplicationPort.js";
 import type { ExecutionOptions } from "../application/AnalysisProvider.js";
 import type {
   DesktopApplication,
+  DesktopFact,
   DesktopFile,
   DesktopOmission,
   InspectDesktopApplicationInput,
@@ -42,6 +46,16 @@ import type {
 import { BinaryTargetError } from "../domain/configurationErrors.js";
 
 const RETAINED_METADATA_BYTES = 32 * 1024 * 1024;
+/**
+ * Upper bound for one file row besides its texts, facts and ranges: the row
+ * and identity objects, file_id, identity strings, arrays, candidate and
+ * component basis IDs and short constant issues. Whole rows, paths included,
+ * measured 800–940 heap and 490–640 JSON bytes on real app bundles (64-bit
+ * Node; #8104).
+ */
+const RETAINED_ROW_BYTES = 1024;
+const RETAINED_OMISSION_BYTES = 512;
+const RETAINED_ISSUE_BYTES = 64;
 
 /** Stop traversal without discarding the preceding bounded observations. */
 class DesktopRepresentationLimit extends ArtifactReaderFailure {
@@ -225,13 +239,20 @@ const scanDirectory = async (
   const budget: DesktopReadBudget = {
     bytes: 0,
     maximum: input.max_probe_bytes,
-    retainedBytes: 1024 + (input.path.length + root.canonical_path.length) * 24,
+    retainedBytes:
+      RETAINED_ROW_BYTES +
+      retainedTextBytes(input.path) +
+      retainedTextBytes(root.canonical_path),
     maximumRetainedBytes: RETAINED_METADATA_BYTES,
   };
-  const reserve = (bytes: number) => {
+  const fits = (bytes: number): boolean => {
     if (bytes > budget.maximumRetainedBytes - budget.retainedBytes)
-      throw new DesktopRepresentationLimit();
+      return false;
     budget.retainedBytes += bytes;
+    return true;
+  };
+  const reserve = (bytes: number) => {
+    if (!fits(bytes)) throw new DesktopRepresentationLimit();
   };
   let entries = 0;
   let probed = 0;
@@ -256,7 +277,7 @@ const scanDirectory = async (
           entries++;
         }
         partial = true;
-        reserve(512 + omission.path.length * 24);
+        reserve(RETAINED_OMISSION_BYTES + retainedTextBytes(omission.path));
         const semantic = {
           root_namespace_id: namespaceId,
           path: omission.path,
@@ -277,18 +298,28 @@ const scanDirectory = async (
     })) {
       entries++;
       abortIfNeeded(signal);
-      // Reserve a conservative representation estimate before retaining path-dependent objects.
-      const reservation =
-        1024 + (entry.path.length + entry.adapterKey.length) * 24;
-      reserve(reservation);
+      const conventions =
+        entry.kind === "file" ? pathConventions(entry.path) : undefined;
+      // Reserve the row and every repeated path before retaining them: each
+      // convention fact copies the path and its probes copy the local path.
+      reserve(
+        RETAINED_ROW_BYTES +
+          retainedTextBytes(entry.path) +
+          retainedTextBytes(entry.adapterKey) +
+          (conventions?.facts.length ?? 0) *
+            (RETAINED_FACT_BYTES +
+              retainedTextBytes(entry.path) +
+              retainedFactReferenceBytes(entry.adapterKey)),
+      );
       const row = fileRow(entry, namespaceId);
       files.push(row);
       if (entry.kind === "symlink") {
         symlinks++;
         continue;
       }
-      if (entry.kind !== "file") continue;
-      const managedRoot = addPathFacts(row);
+      if (conventions === undefined) continue;
+      row.facts.push(...conventions.facts);
+      const managedRoot = conventions.unityManagedDirectory;
       if (!probeSelected(entry)) continue;
       if (budget.bytes >= budget.maximum) {
         row.read_status = "unavailable";
@@ -325,7 +356,11 @@ const scanDirectory = async (
         if (!(cause instanceof ArtifactReaderFailure)) throw cause;
         const probeIssueCount = row.issues.length;
         row.read_status = "unavailable";
-        row.issues.push(cause.message);
+        // Host failures name the absolute path; charge that copy too.
+        const issueRetained = fits(
+          RETAINED_ISSUE_BYTES + retainedTextBytes(cause.message),
+        );
+        if (issueRetained) row.issues.push(cause.message);
         partial = true;
         const completedRead = artifactFileCompletion(cause) !== undefined;
         if (completedRead)
@@ -341,6 +376,7 @@ const scanDirectory = async (
         }
         if (cause.cleanup !== undefined || cause.reason === "cancelled")
           throw cause;
+        if (!issueRetained) throw new DesktopRepresentationLimit();
       }
     }
   } catch (cause: unknown) {
@@ -451,27 +487,28 @@ const fileRow = (
   };
 };
 
-const addPathFacts = (row: DesktopFile): string | null => {
-  const conventions = desktopPathConventions(row.path);
+/** Path-convention facts of one file; each repeats the path as its value. */
+const pathConventions = (path: string) => {
+  const conventions = desktopPathConventions(path);
+  const facts: DesktopFact[] = [];
   if (conventions.electronArchive)
-    row.facts.push({
-      kind: "electron-asar-convention",
-      offset: null,
-      value: row.path,
-    });
+    facts.push({ kind: "electron-asar-convention", offset: null, value: path });
   if (conventions.unrealContainer)
-    row.facts.push({
+    facts.push({
       kind: "unreal-container-convention",
       offset: null,
-      value: row.path,
+      value: path,
     });
   if (conventions.il2cppLibrary)
-    row.facts.push({
+    facts.push({
       kind: "il2cpp-library-convention",
       offset: null,
-      value: row.path,
+      value: path,
     });
-  return conventions.unityManagedDirectory;
+  return {
+    facts,
+    unityManagedDirectory: conventions.unityManagedDirectory,
+  };
 };
 
 const probeSelected = (entry: DirectoryArtifactEntry): boolean =>

src/artifacts/DesktopFileInspection.ts

@@ -10,6 +10,8 @@ import { createHash } from "node:crypto";
 import { setImmediate } from "node:timers/promises";
 import type { DesktopFile, DesktopFact } from "../domain/desktopApplication.js";
 import { jsonValueByteLength } from "../domain/jsonValueByteLength.js";
+import { jsonStringByteLength } from "../domain/jsonStringByteLength.js";
+import { DESKTOP_PROBES_PER_FACT } from "../domain/desktopApplicationProjection.js";
 import { observeIl2cppMetadataHeaderPrefix } from "../domain/il2cpp/il2cppMetadataHeaderPrefix.js";
 import {
   parsePeRecordFacts,
@@ -34,6 +36,31 @@ export interface DesktopReadBudget {
   readonly maximumRetainedBytes: number;
 }
 
+/**
+ * A retained string's larger representation: two-byte heap code units, or its
+ * escaped UTF-8 JSON (up to six bytes for a control character).
+ */
+export const retainedTextBytes = (text: string): number =>
+  Math.max(2 * text.length, jsonStringByteLength(text));
+
+/** One retained fact besides its value and probes: kind, offset and keys. */
+export const RETAINED_FACT_BYTES = 512;
+/** One retained read range: offset, length and SHA-256. */
+const RETAINED_RANGE_BYTES = 256;
+/** One next probe besides its local_path: IDs, operation and rationale. */
+const RETAINED_PROBE_BYTES = 512;
+/** Charge the next probes that one fact can make repeat a row's local_path. */
+export const retainedFactReferenceBytes = (localPath: string): number =>
+  DESKTOP_PROBES_PER_FACT *
+  (RETAINED_PROBE_BYTES + retainedTextBytes(localPath));
+
+const retainedFactsLimit = () =>
+  new ArtifactReaderFailure(
+    "limit",
+    "Directory facts reached the retained metadata budget",
+    { boundary: "memory" },
+  );
+
 /** A resource refusal is separate from malformed format declarations. */
 class DesktopProbeLimit extends ArtifactReaderFailure {
   constructor() {
@@ -73,6 +100,13 @@ export const inspectDesktopFile = async (
         "Declared byte range lies outside the file",
       );
     if (bytes > budget.maximum - budget.bytes) throw new DesktopProbeLimit();
+    // Reserve the retained range before reading its bytes.
+    if (
+      RETAINED_RANGE_BYTES >
+      budget.maximumRetainedBytes - budget.retainedBytes
+    )
+      throw retainedFactsLimit();
+    budget.retainedBytes += RETAINED_RANGE_BYTES;
     const result = Buffer.alloc(bytes);
     let consumed = 0;
     while (consumed < bytes) {
@@ -101,14 +135,12 @@ export const inspectDesktopFile = async (
   };
   const fact = (observation: DesktopFact) => {
     const remaining = budget.maximumRetainedBytes - budget.retainedBytes;
+    const fixed =
+      RETAINED_FACT_BYTES + retainedFactReferenceBytes(row.local_path);
     const reservation =
-      512 + jsonValueByteLength(observation.value, (remaining - 512) / 12) * 12;
-    if (reservation > budget.maximumRetainedBytes - budget.retainedBytes)
-      throw new ArtifactReaderFailure(
-        "limit",
-        "Directory facts reached the retained metadata budget",
-        { boundary: "memory" },
-      );
+      fixed +
+      jsonValueByteLength(observation.value, (remaining - fixed) / 12) * 12;
+    if (reservation > remaining) throw retainedFactsLimit();
     budget.retainedBytes += reservation;
     row.facts.push(observation);
   };

src/domain/desktopApplicationProjection.test.ts

@@ -0,0 +1,86 @@
+import { describe, expect, it } from "vitest";
+
+import type { DesktopFact, DesktopFile } from "./desktopApplication.js";
+import {
+  DESKTOP_PROBES_PER_FACT,
+  projectDesktopApplication,
+} from "./desktopApplicationProjection.js";
+
+const row = (path: string, facts: DesktopFact[]): DesktopFile => ({
+  file_id: `dsf_${"a".repeat(64)}`,
+  path,
+  local_path: `/fixture/${path}`,
+  kind: "file",
+  bytes: 1,
+  identity: null,
+  read_status: "examined",
+  ranges: [],
+  facts,
+  issues: [],
+});
+
+// Every fact kind, including the IL2CPP pair that offers a native probe.
+const facts: DesktopFact[] = [
+  { kind: "pe-cli-directory", offset: 0, value: { rva: 1, bytes: 72 } },
+  {
+    kind: "unity-assembly-reference",
+    offset: 0,
+    value: { name: "UnityEngine", token: "0x23000001", managed_directory: "M" },
+  },
+  {
+    kind: "native-image-header",
+    offset: 0,
+    value: { format: "mach-o", signature: "cffaedfe" },
+  },
+  {
+    kind: "il2cpp-metadata-header",
+    offset: 0,
+    value: { magic: "0xfab11baf", raw_version: 29 },
+  },
+  {
+    kind: "godot-pck-header",
+    offset: 0,
+    value: {
+      pack_version: 2,
+      engine_version: [4, 2, 0],
+      placement: "standalone",
+    },
+  },
+  {
+    kind: "unityfs-header",
+    offset: 0,
+    value: {
+      format_version: 8,
+      engine_version: "2022.3.0f1",
+      flags: 0,
+      declared_size: 1,
+    },
+  },
+  {
+    kind: "gamemaker-form-header",
+    offset: 0,
+    value: { container: "FORM", first_chunk: "GEN8" },
+  },
+  { kind: "il2cpp-library-convention", offset: null, value: "p" },
+  { kind: "electron-asar-convention", offset: null, value: "p" },
+  { kind: "unreal-container-convention", offset: null, value: "p" },
+];
+
+describe("desktop next probe fan-out", () => {
+  it.each(facts)("derives at most the reserved probes from $kind", (fact) => {
+    const { next_probes } = projectDesktopApplication([
+      row("resources/app.asar", [fact]),
+    ]);
+    expect(next_probes.length).toBeLessThanOrEqual(DESKTOP_PROBES_PER_FACT);
+  });
+
+  it("derives at most the reserved probes per fact on one file", () => {
+    const { next_probes } = projectDesktopApplication([
+      row("resources/app.asar", facts),
+    ]);
+    expect(next_probes.length).toBeGreaterThan(0);
+    expect(next_probes.length).toBeLessThanOrEqual(
+      DESKTOP_PROBES_PER_FACT * facts.length,
+    );
+  });
+});

src/domain/desktopApplicationProjection.ts

@@ -2,6 +2,9 @@ import { IL2CPP_METADATA_VERSIONS } from "./il2cpp/il2cppMetadata.js";
 import { digestCanonicalValue } from "./canonicalDigest.js";
 import type { DesktopApplication, DesktopFile } from "./desktopApplication.js";
 
+/** The most next probes this projection derives from any one fact. */
+export const DESKTOP_PROBES_PER_FACT = 2;
+
 /** Interpret only the facts admitted by the directory reader, never target bytes. */
 export const projectDesktopApplication = (
   files: readonly DesktopFile[],

tests/boundary/filesystem/desktopApplication.test.ts

@@ -13,7 +13,10 @@ import { createServer as createNetServer } from "node:net";
 import { expect, it, onTestFinished, vi } from "vitest";
 import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js";
 import { createDesktopApplicationService } from "../../../src/composition/desktop.js";
-import { desktopApplicationSchema } from "../../../src/domain/desktopApplication.js";
+import {
+  desktopApplicationSchema,
+  type DesktopFact,
+} from "../../../src/domain/desktopApplication.js";
 import { createDesktopApplicationEvidence } from "../../../src/domain/desktopApplicationEvidence.js";
 import { parseEvidence } from "../../../src/domain/evidence.js";
 import { inspectAnalysisView } from "../../../src/application/analysisView/AnalysisViewService.js";
@@ -35,6 +38,7 @@ import { DirectoryArtifactReader } from "../../../src/artifacts/DirectoryArtifac
 import { ArtifactReaderFailure } from "../../../src/artifacts/ArtifactReader.js";
 import { artifactFilePrimaryFacts } from "../../../src/artifacts/ArtifactFileOperation.js";
 import { DesktopApplicationReader } from "../../../src/artifacts/DesktopApplicationReader.js";
+import { inspectDesktopFile } from "../../../src/artifacts/DesktopFileInspection.js";
 import { DesktopApplicationService } from "../../../src/application/artifacts/DesktopApplicationService.js";
 
 const inspect = async (path: string, extra: Record<string, number> = {}) => {
@@ -701,7 +705,7 @@ it("bounds omitted-entry retention before allocating rows or serializing identit
       reader.entries = async function* (_signal, options) {
         if (typeof options === "function")
           throw new Error("Expected traversal options");
-        for (let index = 0; index < 400; index++) {
+        for (let index = 0; index < 5000; index++) {
           examined++;
           options?.onOmission?.({
             path: `${prefix}/${index}/entry`,
@@ -716,7 +720,7 @@ it("bounds omitted-entry retention before allocating rows or serializing identit
   const result = await service.inspect({ path: root });
   if (!result.ok) throw result.error;
   const report = desktopApplicationSchema.parse(result.value.normalized_result);
-  expect(examined).toBeLessThan(400);
+  expect(examined).toBeLessThan(5000);
   expect(report.omissions).toHaveLength(examined - 1);
   expect(report.coverage).toMatchObject({
     status: "partial",
@@ -728,6 +732,155 @@ it("bounds omitted-entry retention before allocating rows or serializing identit
   );
 });
 
+/** Inspect simulated directory rows without creating their filesystem tree. */
+const inspectSimulatedRows = async (
+  count: number,
+  path: (index: number) => string,
+  kind: "file" | "directory" = "directory",
+) => {
+  const root = await createTestTempDirectory("rea-desktop-row-capacity-");
+  let yielded = 0;
+  const service = new DesktopApplicationService(
+    new DesktopApplicationReader((selected) => {
+      const reader = new DirectoryArtifactReader(selected);
+      reader.entries = async function* () {
+        for (let index = 0; index < count; index++) {
+          yielded++;
+          yield {
+            path: path(index),
+            kind,
+            declaredSize: kind === "file" ? 1 : null,
+            compressedSize: null,
+            executable: false,
+            encrypted: false,
+            byteOffset: null,
+            declaredIntegrity: null,
+            unpacked: false,
+            limitations: [],
+            adapterKey: `${root}/${path(index)}`,
+          };
+        }
+      };
+      return reader;
+    }),
+  );
+  onTestFinished(() => service.close());
+  const result = await service.inspect({ path: root, max_entries: count });
+  if (!result.ok) throw result.error;
+  return {
+    yielded,
+    report: desktopApplicationSchema.parse(result.value.normalized_result),
+  };
+};
+
+it("admits bundle-sized rows up to max_entries within the representation budget", async () => {
+  // Localized resources dominate large macOS bundles such as iWork apps.
+  const { report } = await inspectSimulatedRows(
+    20_000,
+    (index) =>
+      `Contents/Resources/${String(index % 40).padStart(2, "0")}.lproj/Localizable${index}.strings`,
+  );
+  expect(report.files).toHaveLength(20_000);
+  expect(report.coverage).toMatchObject({
+    status: "complete-within-scan",
+    entries_examined: 20_000,
+  });
+  expect(report.limitations).not.toContainEqual(
+    expect.stringContaining("representation budget"),
+  );
+});
+
+it("charges escaped JSON expansion before retaining control-character paths", async () => {
+  const { yielded, report } = await inspectSimulatedRows(
+    6000,
+    (index) => `${"\u0001".repeat(1000)}/${index}`,
+  );
+  expect(report.files.length).toBeLessThan(6000);
+  expect(report.files).toHaveLength(yielded - 1);
+  expect(report.coverage).toMatchObject({
+    status: "partial",
+    entries_examined: yielded,
+  });
+  expect(report.limitations).toContainEqual(
+    expect.stringContaining("representation budget"),
+  );
+  // Escaping makes each path six times longer in JSON than in memory.
+  expect(Buffer.byteLength(JSON.stringify(report.files))).toBeLessThan(
+    32 * 1024 * 1024,
+  );
+});
+
+it.each([
+  ["Unreal pack", "Content/Paks/pakchunk0.pak", 0],
+  ["Electron archive", "resources/app.asar", 2],
+])(
+  "charges %s convention facts and probes that repeat control-character paths",
+  async (_name, suffix, probes) => {
+    const { yielded, report } = await inspectSimulatedRows(
+      3000,
+      (index) => `${"\u0001".repeat(1000)}/${index}/${suffix}`,
+      "file",
+    );
+    expect(report.files.length).toBeLessThan(3000);
+    expect(report.files).toHaveLength(yielded - 1);
+    // The fact value and each next probe repeat the escaped path.
+    expect(report.files[0]?.facts).toHaveLength(1);
+    expect(report.next_probes).toHaveLength(probes * report.files.length);
+    expect(report.coverage.status).toBe("partial");
+    expect(report.limitations).toContainEqual(
+      expect.stringContaining("representation budget"),
+    );
+    expect(Buffer.byteLength(JSON.stringify(report))).toBeLessThan(
+      32 * 1024 * 1024,
+    );
+  },
+);
+
+it("charges the next probes a header fact offers before retaining the fact", async () => {
+  const root = await createTestTempDirectory("rea-desktop-fact-probes-");
+  const bytes = buildManagedPeFixture();
+  await put(root, "Managed.dll", bytes);
+  // Each managed fact offers a probe that repeats this escaped local path.
+  const localPath = "\u0001".repeat(100_000);
+  const inspectWithin = async (maximumRetainedBytes: number) => {
+    const row = {
+      file_id: `dsf_${"a".repeat(64)}`,
+      path: "Managed.dll",
+      local_path: localPath,
+      kind: "file" as const,
+      bytes: bytes.length,
+      identity: null,
+      read_status: "not-selected" as const,
+      ranges: [],
+      facts: [] as DesktopFact[],
+      issues: [] as string[],
+    };
+    const file = await open(join(root, "Managed.dll"), "r");
+    try {
+      await inspectDesktopFile(file, row, {
+        bytes: 0,
+        maximum: 1024 * 1024,
+        retainedBytes: 0,
+        maximumRetainedBytes,
+      });
+    } finally {
+      await file.close();
+    }
+    return row;
+  };
+  const refused = await inspectWithin(1024 * 1024);
+  expect(refused.facts).toEqual([]);
+  expect(refused.issues).toEqual([
+    "Directory facts reached the retained metadata budget",
+  ]);
+  const admitted = await inspectWithin(32 * 1024 * 1024);
+  expect(admitted.facts.map((fact) => fact.kind)).toEqual([
+    "native-image-header",
+    "pe-cli-directory",
+  ]);
+  expect(admitted.issues).toEqual([]);
+});
+
 it("preserves the selected root alias and directs follow-ups to the admitted physical root", async () => {
   const root = await createTestTempDirectory("rea-desktop-root-alias-");
   const admitted = join(root, "admitted");