Compare commits

...
Author SHA1 Message Date
morluto 41bfee3b5d fix(test): reconcile Kuna target selection with main 2026-11-07 23:09:58 +08:00
Heo Hao f8bb792f15 fix(cli): stop restating schema defaults in help descriptions (#9278)
The help renderer appends "(default: X)" whenever the option field declares a
schema default, so descriptions that also spell the default out print it twice:

  --block-size <value>  Power-of-two block size from 256 to 65536 bytes
                        (default 4096) (default: 4096)

Remove the hand-written default from the eleven options that own a schema
default across inspect-byte-entropy, inspect-devicetree,
inspect-packet-capture, inspect-packet-fields, inspect-signal-decoders and
decode-signal-recording, matching the rule in cliDecimalField's contract that
the domain field owns bounds, defaults and help.

Options without a schema default keep their prose: --match_limit, --line_limit,
--limit and --packet_limit are nullable or optional, so the renderer emits no
annotation and the description is the only place the effective value appears.

Add a boundary test asserting each such option states its default exactly once,
so a restated default cannot come back unnoticed.
2026-11-07 22:38:03 +08:00
Cursor Agent 947ffcd64c merge(main): record caller-target selection in the Kuna lane
testing.md now points at the moved lane. The selection report still
names the directory, requested functions, nm matches, and a partial
scan when symbol inspection fails.
2026-11-07 11:57:12 +00:00
morluto 8c5e7511c8 test(kuna): cover passing caller target selection 2026-11-01 08:54:13 +08:00
morluto 8c9bf74145 fix(test): report caller target selection coverage 2026-11-01 08:48:07 +08:00
10 changed files with 424 additions and 36 deletions

No files matched your search

+1 -1
View File
@@ -183,7 +183,7 @@ REA_KUNA_COMMAND=/absolute/path/to/kuna GHIDRA_INSTALL_DIR=/absolute/path/to/ghi
npm run verify:kuna -- --targets DIR --functions inflate,crc32 --compare-ghidra
```
The lane compiles a small source-owned fixture with the host C compiler and verifies the MCP workflow end to end. `--targets DIR --functions a,b` adds caller-built binaries. `--compare-ghidra` (with `GHIDRA_INSTALL_DIR`) runs those same functions through Ghidra; it requires `--targets` and `--functions` and refuses empty matrices or requested function names missing from every target. See [testing](testing.md#kuna-lane) for coverage.
The lane compiles a small source-owned fixture with the host C compiler and verifies the MCP workflow end to end. `--targets DIR --functions a,b` adds caller-built binaries. `--compare-ghidra` (with `GHIDRA_INSTALL_DIR`) runs those same functions through Ghidra; it requires `--targets` and `--functions` and refuses empty matrices or requested function names missing from every target. The report lists examined caller binaries and their matching requested symbols, including inputs omitted from analysis. See [testing](testing.md#kuna-lane) for coverage.
Failed runs also report process-lineage evidence and exit unsuccessfully. If provider or transport cleanup is unconfirmed, the report says `cleanup: "incomplete"` and records `retained_workspace`; that directory is preserved for investigation and cleanup retries.
+10 -1
View File
@@ -448,7 +448,16 @@ caller-built functions through Ghidra (it refuses to run without `--targets` and
reports latency, sampled process-tree CPU and peak memory, UTF-8 response size,
and goto and warning-marker counts. Those counts are not a correctness judgement,
and model tokens are not measured. Reference timings and response sizes are
reported separately; reference caches expire when the server closes. `--report PATH` writes the JSON report.
reported separately; reference caches expire when the server closes. The report's
`caller_target_selection` retains the selected directory, the original functions
argument, and each examined binary's requested names matched against `nm` text
symbols. `symbol_scan_complete` means the scan of non-`.stripped` entries in
the immediate flavor directories finished; it does not cover other directory
entries. An `inspection_failed` entry and `symbol_scan_complete: false` retain
partial discovery when symbol inspection fails. A missing symbol is not proof
that code is absent, and selection does not establish that a stripped counterpart
was verified; consult `results` for completed variants. `--report PATH` writes
the JSON report.
The first matrix run used Kuna 1.833 and Ghidra 12.1.4 with JDK 21 on macOS
arm64. It covered zlib 1.3.2 `minigzip`, coreutils 9.9 `sort` and the sqlite
+49 -13
View File
@@ -595,15 +595,37 @@ const measureGhidra = async (path, functions) => {
return answers;
};
/** Symbol-bearing caller binaries with the requested functions, beside their stripped copies. */
const targetPairs = async (root, names) => {
const describeFailure = (cause) => ({
message: cause instanceof Error ? cause.message : String(cause),
...(cause instanceof AggregateError
? { causes: cause.errors.map(describeFailure) }
: {}),
});
/** Inspect symbol-bearing caller binaries, retaining both selected and unmatched inputs. */
const targetPairs = async (root, names, selection) => {
const pairs = [];
for (const flavor of (await readdir(root)).sort()) {
const directory = join(root, flavor);
if (!(await stat(directory)).isDirectory()) continue;
for (const file of (await readdir(directory)).sort()) {
if (file.endsWith(".stripped")) continue;
const symbols = await symbolAddresses(join(directory, file), names);
let symbols;
try {
symbols = await symbolAddresses(join(directory, file), names);
} catch (cause) {
selection.examined.push({
binary: `${flavor}/${file}`,
status: "inspection_failed",
error: describeFailure(cause),
});
throw cause;
}
selection.examined.push({
binary: `${flavor}/${file}`,
status: symbols.size > 0 ? "selected" : "no_matching_text_symbols",
matched_functions: [...symbols.keys()],
});
if (symbols.size > 0) pairs.push({ flavor, directory, file, symbols });
}
}
@@ -624,6 +646,18 @@ const verifyVariant = async ({ flavor, directory, symbols }, variant) => {
const results = [];
let timeoutRecovery;
const failures = [];
const callerTargetSelection =
values.targets === undefined
? undefined
: {
directory: values.targets,
functions_argument: values.functions ?? null,
requested_functions: (values.functions ?? "")
.split(",")
.filter(Boolean),
symbol_scan_complete: false,
examined: [],
};
let inspectionResourcesClean = true;
try {
const installation = await inspectKunaInstallation(
@@ -636,13 +670,18 @@ try {
);
kunaVersion = `kuna ${installation.value.version}`;
let callerPairs = [];
if (values.targets !== undefined) {
const names = (values.functions ?? "").split(",").filter(Boolean);
if (callerTargetSelection !== undefined) {
const names = callerTargetSelection.requested_functions;
assert.ok(names.length > 0, "--targets requires --functions");
callerPairs = await targetPairs(values.targets, names);
callerPairs = await targetPairs(
callerTargetSelection.directory,
names,
callerTargetSelection,
);
callerTargetSelection.symbol_scan_complete = true;
assert.ok(
callerPairs.length > 0,
`No caller-built targets contain the requested functions in ${values.targets}`,
`No caller-built targets contain the requested functions in ${callerTargetSelection.directory}`,
);
const found = new Set(
callerPairs.flatMap(({ symbols }) => [...symbols.keys()]),
@@ -730,18 +769,15 @@ if (cleanupComplete)
cleanupComplete = false;
failures.push(cause);
}
const describeFailure = (cause) => ({
message: cause instanceof Error ? cause.message : String(cause),
...(cause instanceof AggregateError
? { causes: cause.errors.map(describeFailure) }
: {}),
});
const report = {
lane: "kuna",
status: failures.length === 0 ? "passed" : "failed",
kuna_version: kunaVersion,
platform: `${process.platform}/${process.arch}`,
verifier_run: lifecycle,
...(callerTargetSelection === undefined
? {}
: { caller_target_selection: callerTargetSelection }),
results,
...(timeoutRecovery === undefined
? {}
+6 -8
View File
@@ -266,12 +266,10 @@ export const registerBinaryDiagnosticsCommands = (
// "0x10" or "1e2".
blockSize: cliDecimalField(
inspectByteEntropyInputSchema.shape.block_size,
).describe(
"Power-of-two block size from 256 to 65536 bytes (default 4096)",
),
).describe("Power-of-two block size from 256 to 65536 bytes"),
offset: cliDecimalField(
inspectByteEntropyInputSchema.shape.offset,
).describe("Window start as a decimal file byte offset (default 0)"),
).describe("Window start as a decimal file byte offset"),
length: cliDecimalField(
inspectByteEntropyInputSchema.shape.length,
).describe("Window length in bytes; omit to extend through EOF"),
@@ -429,16 +427,16 @@ export const registerBinaryDiagnosticsCommands = (
options: z.object({
maxFileBytes: cliDecimalField(
inspectDevicetreeInputSchema.shape.max_file_bytes,
).describe("Maximum FDT file size in bytes (default 4194304)"),
).describe("Maximum FDT file size in bytes"),
maxNodes: cliDecimalField(
inspectDevicetreeInputSchema.shape.max_nodes,
).describe("Maximum number of device-tree nodes (default 8192)"),
).describe("Maximum number of device-tree nodes"),
maxProperties: cliDecimalField(
inspectDevicetreeInputSchema.shape.max_properties,
).describe("Maximum number of device-tree properties (default 32768)"),
).describe("Maximum number of device-tree properties"),
maxDepth: cliDecimalField(
inspectDevicetreeInputSchema.shape.max_depth,
).describe("Maximum device-tree nesting depth (default 64)"),
).describe("Maximum device-tree nesting depth"),
...analysisEvidenceDeliveryOptions,
}),
run: ({ args, options }) =>
+1 -1
View File
@@ -51,7 +51,7 @@ const captureInputShape = z.strictObject({
.int()
.nonnegative()
.default(0)
.describe("Skip this many matching packets after filtering; default 0"),
.describe("Skip this many matching packets after filtering"),
match_limit: z
.number()
.int()
+2 -4
View File
@@ -5,9 +5,7 @@ export const inspectPacketFieldsInputSchema = z.strictObject({
kind: z
.enum(["protocols", "fields"])
.default("protocols")
.describe(
"Return protocol registrations or field definitions; defaults to protocols",
),
.describe("Return protocol registrations or field definitions"),
protocol: z
.string()
.min(1)
@@ -30,7 +28,7 @@ export const inspectPacketFieldsInputSchema = z.strictObject({
.nonnegative()
.max(Number.MAX_SAFE_INTEGER)
.default(0)
.describe("Number of matching records to skip; defaults to 0"),
.describe("Number of matching records to skip"),
limit: z
.number()
.int()
+1 -3
View File
@@ -24,9 +24,7 @@ export const inspectSignalDecodersInputSchema = z.strictObject({
.nonnegative()
.max(Number.MAX_SAFE_INTEGER)
.default(0)
.describe(
"Number of matching decoders or help lines to skip; defaults to 0",
),
.describe("Number of matching decoders or help lines to skip"),
limit: z
.number()
.int()
+1 -1
View File
@@ -31,7 +31,7 @@ const signalInputShape = z.strictObject({
.int()
.nonnegative()
.default(0)
.describe("Skip this many literal LF-separated output lines; default 0"),
.describe("Skip this many literal LF-separated output lines"),
line_limit: z
.number()
.int()
@@ -0,0 +1,88 @@
import { describe, expect, it } from "vitest";
import { createCli } from "../../../src/cli.js";
type CliInstance = ReturnType<typeof createCli>;
const serve = async (
cli: CliInstance,
argv: readonly string[],
): Promise<{ readonly stdout: string; readonly exitCode: number }> => {
let stdout = "";
let exitCode = 0;
await cli.serve([...argv], {
env: {},
exit: (code) => {
exitCode = code;
},
stdout: (text) => {
stdout += text;
},
});
return { stdout, exitCode };
};
/**
* Options whose owning field already declares a schema default. The help
* renderer appends that default, so the description must not restate it.
*/
const singleDefaultOptions: readonly {
readonly command: string;
readonly options: readonly string[];
}[] = [
{ command: "inspect-byte-entropy", options: ["--block-size", "--offset"] },
{
command: "inspect-devicetree",
options: [
"--max-file-bytes",
"--max-nodes",
"--max-properties",
"--max-depth",
],
},
{
command: "decode-signal-recording",
options: ["--line_offset", "--line_limit"],
},
{
command: "inspect-packet-capture",
options: ["--match_offset", "--match_limit"],
},
{
command: "inspect-packet-fields",
options: ["--kind", "--offset"],
},
{ command: "inspect-signal-decoders", options: ["--offset"] },
];
const optionLine = (
help: string,
option: string,
): string | undefined => {
const line = help
.split(/\r?\n/u)
.find((candidate) => candidate.trimStart().startsWith(`${option} `));
return line?.trim();
};
describe("CLI help default annotations", () => {
it("states each schema-owned option default exactly once", async () => {
const cli = createCli({});
const repeated: string[] = [];
for (const { command, options } of singleDefaultOptions) {
const { stdout, exitCode } = await serve(cli, [command, "--help"]);
expect(exitCode, command).toBe(0);
for (const option of options) {
const line = optionLine(stdout, option);
if (line === undefined) {
repeated.push(`${command} ${option}: option line missing`);
continue;
}
const statements = line.match(/default/giu)?.length ?? 0;
if (statements !== 1)
repeated.push(`${command} ${option}: ${statements} default statements`);
}
}
expect(repeated).toEqual([]);
});
});
@@ -1,5 +1,6 @@
import { spawnSync } from "node:child_process";
import { mkdir, readFile, rm, writeFile } from "node:fs/promises";
import { createHash } from "node:crypto";
import { copyFile, mkdir, readFile, rm, writeFile } from "node:fs/promises";
import { join, resolve } from "node:path";
import { pathToFileURL } from "node:url";
@@ -10,6 +11,164 @@ import { fakeKunaCalls, writeFakeKuna } from "./fakeKuna.js";
const it = test.skipIf(process.platform === "win32");
const createCallerMatrix = async (root: string) => {
const targets = join(root, "targets");
const binaries: string[] = [];
for (const [flavor, name, multiplier] of [
["left", "audit_left", 3],
["right", "audit_right", 7],
["unmatched", "audit_other", 11],
] as const) {
const directory = join(targets, flavor);
await mkdir(directory, { recursive: true });
const source = join(root, `${flavor}.c`);
const binary = join(directory, "program");
await writeFile(
source,
`__attribute__((noinline,used)) int ${name}(int x) { return x * ${multiplier} + 1; }\nint main(void) { return ${name}(2); }\n`,
);
const compiled = spawnSync(
process.env.REA_CC ?? "cc",
["-O1", "-o", binary, source],
{ encoding: "utf8", timeout: 30_000 },
);
expect(compiled.error).toBeUndefined();
expect(compiled.status, compiled.stderr).toBe(0);
const stripped = `${binary}.stripped`;
await copyFile(binary, stripped);
const strip = spawnSync(process.env.REA_STRIP ?? "strip", [stripped], {
encoding: "utf8",
timeout: 30_000,
});
expect(strip.error).toBeUndefined();
expect(strip.status, strip.stderr).toBe(0);
binaries.push(binary, stripped);
}
return { targets, binaries };
};
const expectedCallerSelection = (targets: string) => ({
directory: targets,
functions_argument: "audit_left,audit_right",
requested_functions: ["audit_left", "audit_right"],
symbol_scan_complete: true,
examined: [
{
binary: "left/program",
status: "selected",
matched_functions: ["audit_left"],
},
{
binary: "right/program",
status: "selected",
matched_functions: ["audit_right"],
},
{
binary: "unmatched/program",
status: "no_matching_text_symbols",
matched_functions: [],
},
],
});
it("accepts requested functions distributed across caller targets", async () => {
const root = await createTestTempDirectory(
"rea-kuna-verifier-matrix-boundary-",
);
const { targets } = await createCallerMatrix(root);
const kuna = await writeFakeKuna(root);
const result = spawnSync(
process.execPath,
[
"scripts/verify-real-kuna.mjs",
"--targets",
targets,
"--functions",
"audit_left,audit_right",
],
{
encoding: "utf8",
timeout: 10_000,
env: {
PATH: process.env.PATH,
HOME: root,
REA_KUNA_COMMAND: kuna,
REA_CC: join(root, "compiler-must-not-run"),
},
},
);
expect(result.error).toBeUndefined();
expect(result.status).toBe(1);
expect(result.stderr).toContain("compiler-must-not-run");
expect(JSON.parse(result.stdout)).toMatchObject({
status: "failed",
cleanup: "complete",
caller_target_selection: expectedCallerSelection(targets),
});
});
test.skipIf(
process.platform === "win32" || process.env.REA_KUNA_COMMAND === undefined,
)(
"reports the complete selection of a passing real-provider matrix",
async () => {
const root = await createTestTempDirectory("rea-kuna-verifier-matrix-");
const { targets, binaries } = await createCallerMatrix(root);
const hashes = async () =>
Promise.all(
binaries.map(async (path) =>
createHash("sha256")
.update(await readFile(path))
.digest("hex"),
),
);
const before = await hashes();
const reportPath = join(root, "report.json");
const result = spawnSync(
process.execPath,
[
"scripts/verify-real-kuna.mjs",
"--targets",
targets,
"--functions",
"audit_left,audit_right",
"--report",
reportPath,
],
{
encoding: "utf8",
timeout: 100_000,
maxBuffer: 10 * 1024 * 1024,
env: process.env,
},
);
expect(result.error).toBeUndefined();
expect(result.status, result.stderr).toBe(0);
const report = JSON.parse(await readFile(reportPath, "utf8"));
expect(JSON.parse(result.stdout)).toEqual(report);
expect(report).toMatchObject({
status: "passed",
cleanup: "complete",
verifier_run: {
process_lineage: { status: "verified", descendants: [] },
},
caller_target_selection: expectedCallerSelection(targets),
});
expect(
report.results.map(({ binary }: { binary: string }) => binary),
).toEqual([
"fixture/rea-kuna-fixture",
"fixture/rea-kuna-fixture.stripped",
"left/program",
"left/program.stripped",
"right/program",
"right/program.stripped",
]);
expect(await hashes()).toEqual(before);
},
120_000,
);
it("accepts empty caller inventories but requires source-owned fixture literals", async () => {
const root = await createTestTempDirectory("rea-kuna-verifier-empty-");
const result = spawnSync(
@@ -48,16 +207,39 @@ await assert.rejects(() => verifyKunaStrings({ ...options, expectedStrings: ["re
});
it.each([
{ populated: false, names: "main", reason: "No caller-built targets" },
{ populated: true, names: "typo", reason: "No caller-built targets" },
{
populated: false,
names: "main",
reason: "No caller-built targets",
examined: [],
},
{
populated: true,
names: "typo",
reason: "No caller-built targets",
examined: [
{
binary: "linux/program",
status: "no_matching_text_symbols",
matched_functions: [],
},
],
},
{
populated: true,
names: "main,typo",
reason: "Requested functions were not found",
examined: [
{
binary: "linux/program",
status: "selected",
matched_functions: ["main"],
},
],
},
])(
"rejects an unverified caller matrix: $names, populated=$populated",
async ({ populated, names, reason }) => {
async ({ populated, names, reason, examined }) => {
const root = await createTestTempDirectory("rea-kuna-verifier-selection-");
const kuna = await writeFakeKuna(root);
const targets = join(root, "targets");
@@ -104,12 +286,91 @@ it.each([
expect(result.status).toBe(1);
expect(result.stderr).toContain(reason);
expect(result.stderr).not.toContain("compiler-must-not-run");
const report = JSON.parse(result.stdout);
expect(report.caller_target_selection).toEqual({
directory: targets,
functions_argument: names,
requested_functions: names.split(","),
symbol_scan_complete: true,
examined,
});
expect(report.cleanup).toBe("complete");
expect((await fakeKunaCalls(kuna)).map(({ argv }) => argv)).toEqual([
["--version"],
]);
},
);
it("retains examined caller targets when a later symbol inspection fails", async () => {
const root = await createTestTempDirectory("rea-kuna-verifier-inspection-");
const kuna = await writeFakeKuna(root);
const targets = join(root, "targets");
await mkdir(join(targets, "first"), { recursive: true });
await mkdir(join(targets, "second"), { recursive: true });
await writeFile(join(targets, "first", "program"), "not analyzed");
await writeFile(join(targets, "second", "broken"), "not analyzed");
const nm = join(root, "nm");
await writeFile(
nm,
`#!${process.execPath}\nif (process.argv.at(-1).endsWith("broken")) { process.stderr.write("inspection failed"); process.exit(9); }\nprocess.stdout.write("00001000 T main\\n");\n`,
{ mode: 0o755 },
);
const reportPath = join(root, "report.json");
const result = spawnSync(
process.execPath,
[
"scripts/verify-real-kuna.mjs",
"--targets",
targets,
"--functions",
"main",
"--report",
reportPath,
],
{
encoding: "utf8",
timeout: 10_000,
env: {
PATH: process.env.PATH,
HOME: root,
REA_KUNA_COMMAND: kuna,
REA_NM: nm,
REA_CC: join(root, "compiler-must-not-run"),
},
},
);
expect(result.error).toBeUndefined();
expect(result.status).toBe(1);
const report = JSON.parse(await readFile(reportPath, "utf8"));
expect(JSON.parse(result.stdout)).toEqual(report);
expect(report).toMatchObject({
status: "failed",
cleanup: "complete",
caller_target_selection: {
directory: targets,
functions_argument: "main",
requested_functions: ["main"],
symbol_scan_complete: false,
examined: [
{
binary: "first/program",
status: "selected",
matched_functions: ["main"],
},
{
binary: "second/broken",
status: "inspection_failed",
error: { message: expect.stringContaining("inspection failed") },
},
],
},
});
expect(result.stderr).not.toContain("compiler-must-not-run");
expect((await fakeKunaCalls(kuna)).map(({ argv }) => argv)).toEqual([
["--version"],
]);
});
it.each([
{
rejectClose: false,