mirror of
https://github.com/morluto/rea.git
synced 2026-11-07 05:56:19 -10:00
fix(docs): preserve verification reports after experiment archival (#7462)
Move the retained allocation report and unchanged receipt JSON into verification documentation. Keep one report producer, repair current and archived links, document test-support ownership, and remove retired lint inputs. Fixes #7458. Validated focused/Python checks, report consistency, docs rendering, and real Bash CLI/MCP facts against an independent oracle. Greptile reviewed all seven files without comments. CI source/Windows/shard-8 failures match clean-main or existing-main controls; the requester authorized merging with those established failures. Codex review quota was unavailable.
This commit is contained in:
1 parent
0676ad254c
commit
2817d30ab2
7 files changed
+44
-11
No files matched your search
+3
-2
@@ -7,8 +7,9 @@ installed-client and agent runs.
|
|||||||
|
|
||||||
Historical verification records may refer to `experiments/analysis-efficiency`.
|
Historical verification records may refer to `experiments/analysis-efficiency`.
|
||||||
Those experiment scripts are available in the [archived source tree](https://github.com/morluto/rea/tree/a2e2c29b59e8775b0e6aedb09a9eb16adf32646b/experiments/analysis-efficiency);
|
Those experiment scripts are available in the [archived source tree](https://github.com/morluto/rea/tree/a2e2c29b59e8775b0e6aedb09a9eb16adf32646b/experiments/analysis-efficiency);
|
||||||
they are not included in current checkouts. Test support remains under
|
they are not included in current checkouts. The retained [test support](https://github.com/morluto/rea/blob/main/tests/support/analysis-efficiency/README.md)
|
||||||
`tests/support/analysis-efficiency`.
|
contains deterministic checkers and fixtures. The [allocation report](verification/analysis-efficiency/mcp-integer-allocation.md)
|
||||||
|
and its original receipt JSON remain documentation, checked by the report producer.
|
||||||
|
|
||||||
## Developer commands
|
## Developer commands
|
||||||
|
|
||||||
|
|||||||
File renamed without changes.
+11
-4
@@ -22,7 +22,7 @@ Seven dependency-boundary variants match parsed values and complete error issues
|
|||||||
|
|
||||||
## Tasks, artifacts and bounds
|
## Tasks, artifacts and bounds
|
||||||
|
|
||||||
Reuse the shared SDK task driver and independent ELF/section/zlib/JSON oracles from [the catalog study](mcp-input-catalog.md): official Alpine v3.21 x86_64 ripgrep 14.1.1-r0 and Zellij 0.40.0-r0. All 42/422 packages and 88/1,063 edges match the independent oracles. Select serde name/version/kind, repeat exact retained-parent reads twice, verify a canonical negative-offset refusal, export/check complete Evidence and confirm the unchanged directory. `kind: null` stays unknown. Targets are not executed; dependency presence does not establish runtime use. Acquisition URLs, archive/executable/oracle hashes and complete answers are in [the receipts](mcp-integer-allocation.json). Previously acquired public bytes are rehashed, not redownloaded.
|
Reuse the shared SDK task driver and independent ELF/section/zlib/JSON oracles from [the archived catalog study](https://github.com/morluto/rea/blob/a2e2c29b59e8775b0e6aedb09a9eb16adf32646b/experiments/analysis-efficiency/mcp-input-catalog.md): official Alpine v3.21 x86_64 ripgrep 14.1.1-r0 and Zellij 0.40.0-r0. All 42/422 packages and 88/1,063 edges match the independent oracles. Select serde name/version/kind, repeat exact retained-parent reads twice, verify a canonical negative-offset refusal, export/check complete Evidence and confirm the unchanged directory. `kind: null` stays unknown. Targets are not executed; dependency presence does not establish runtime use. Acquisition URLs, archive/executable/oracle hashes and complete answers are in [the receipts](https://github.com/morluto/rea/blob/main/docs/verification/analysis-efficiency/mcp-integer-allocation.json). Previously acquired public bytes are rehashed, not redownloaded.
|
||||||
|
|
||||||
Normal cohort: three alternating baseline/candidate pairs for each task × cold/warm server compile cache × default compact/full inputs: 72 trials, plus four excluded full-task warm priming runs. Separate diagnostic cohort: three alternating pairs per profile, cold Zellij tasks, 12 trials. All pass, with identical complete parent IDs/hashes and answers.
|
Normal cohort: three alternating baseline/candidate pairs for each task × cold/warm server compile cache × default compact/full inputs: 72 trials, plus four excluded full-task warm priming runs. Separate diagnostic cohort: three alternating pairs per profile, cold Zellij tasks, 12 trials. All pass, with identical complete parent IDs/hashes and answers.
|
||||||
|
|
||||||
@@ -115,6 +115,12 @@ Question, explicit recipe, server instructions, one complete name/description/in
|
|||||||
- Both runtime builds and all six static checks plus architecture guards pass. Before the final main synchronization, 5,972 source/contract/public CLI/MCP tests pass across 323 files; four skip for unavailable/guarded coverage. After synchronization, both builds/static checks repeat successfully, 77 affected producer and CLI/MCP tests pass, and all 974 registered numeric-parameter parser cases match again. Includes real SQLite CLI/MCP record/column selection and JavaScript heap/deadline preservation. A first direct schema-validity invocation lacked generated catalog metadata; the supported focused entry generates it and all 15 cases pass. A later affected-test invocation selected two nonexistent paths; the corrected three-file producer/consumer run is the 77-case result above. Neither setup failure was counted as a passing test, and no production or timeout change was needed.
|
- Both runtime builds and all six static checks plus architecture guards pass. Before the final main synchronization, 5,972 source/contract/public CLI/MCP tests pass across 323 files; four skip for unavailable/guarded coverage. After synchronization, both builds/static checks repeat successfully, 77 affected producer and CLI/MCP tests pass, and all 974 registered numeric-parameter parser cases match again. Includes real SQLite CLI/MCP record/column selection and JavaScript heap/deadline preservation. A first direct schema-validity invocation lacked generated catalog metadata; the supported focused entry generates it and all 15 cases pass. A later affected-test invocation selected two nonexistent paths; the corrected three-file producer/consumer run is the 77-case result above. Neither setup failure was counted as a passing test, and no production or timeout change was needed.
|
||||||
- All 84 measured tasks and four warm priming tasks pass independent oracles, canonical refusal, parent/directory identity and owned cleanup checks. This does not verify external Hopper/Ghidra/IDA, Android devices, all providers or foreign OS behavior. The earlier macOS numbers in #4360 describe a different revision/platform and are not combined with these results.
|
- All 84 measured tasks and four warm priming tasks pass independent oracles, canonical refusal, parent/directory identity and owned cleanup checks. This does not verify external Hopper/Ghidra/IDA, Android devices, all providers or foreign OS behavior. The earlier macOS numbers in #4360 describe a different revision/platform and are not combined with these results.
|
||||||
|
|
||||||
|
The historical cohort commands below use the retired experiment layout. Run them
|
||||||
|
from the matching recorded producer checkout; the [archived source tree](https://github.com/morluto/rea/tree/a2e2c29b59e8775b0e6aedb09a9eb16adf32646b/experiments/analysis-efficiency)
|
||||||
|
provides the former files for reference. Current checkouts retain the report and
|
||||||
|
its conformance producer, rather than the full measurement suite. Moving these
|
||||||
|
files does not requalify the historical measurements against current REA.
|
||||||
|
|
||||||
Build the pinned baseline/candidate in separate clean worktrees with the locked toolchain. Use the candidate’s producer scripts for both roles. `measurement_inputs` in the JSON records the complete matrix, questions/recipes, runtime bindings and producer hashes; extract it unchanged as the bindings file. Historical receipts require their matching producer revision. One normal task:
|
Build the pinned baseline/candidate in separate clean worktrees with the locked toolchain. Use the candidate’s producer scripts for both roles. `measurement_inputs` in the JSON records the complete matrix, questions/recipes, runtime bindings and producer hashes; extract it unchanged as the bindings file. Historical receipts require their matching producer revision. One normal task:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
@@ -133,11 +139,12 @@ python experiments/analysis-efficiency/mcp-startup-costs.py "$RECEIPT_ROOT" \
|
|||||||
|
|
||||||
The aggregator requires the recorded tiktoken version, exact catalog/parent/runtime/producer/guard identities, clean source, successful payload cleanup and matching matrix coverage. Default driver behavior continues without an inspector; the heap flag is confined to the experiment.
|
The aggregator requires the recorded tiktoken version, exact catalog/parent/runtime/producer/guard identities, clean source, successful payload cleanup and matching matrix coverage. Default driver behavior continues without an inspector; the heap flag is confined to the experiment.
|
||||||
|
|
||||||
Regenerate or verify every numerical section directly from the committed JSON:
|
In a current checkout, regenerate or verify every numerical section directly
|
||||||
|
from the committed JSON:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
node experiments/analysis-efficiency/mcp-integer-report.mjs
|
node tests/support/analysis-efficiency/mcp-integer-report.mjs
|
||||||
node experiments/analysis-efficiency/mcp-integer-report.mjs --check
|
node tests/support/analysis-efficiency/mcp-integer-report.mjs --check
|
||||||
```
|
```
|
||||||
|
|
||||||
The documentation conformance gate checks current receipts, rejects stale normal/heap tables, and refuses missing producer markers without writing.
|
The documentation conformance gate checks current receipts, rejects stale normal/heap tables, and refuses missing producer markers without writing.
|
||||||
@@ -89,7 +89,7 @@ describe("published performance evidence", () => {
|
|||||||
"tests/support/analysis-efficiency/mcp-integer-report.mjs",
|
"tests/support/analysis-efficiency/mcp-integer-report.mjs",
|
||||||
);
|
);
|
||||||
const report = resolve(
|
const report = resolve(
|
||||||
"tests/support/analysis-efficiency/mcp-integer-allocation.md",
|
"docs/verification/analysis-efficiency/mcp-integer-allocation.md",
|
||||||
);
|
);
|
||||||
processTest(
|
processTest(
|
||||||
"keeps generated report facts aligned with committed receipts",
|
"keeps generated report facts aligned with committed receipts",
|
||||||
|
|||||||
@@ -0,0 +1,23 @@
|
|||||||
|
# Analysis verification test support
|
||||||
|
|
||||||
|
These files support the existing comparison, independent-oracle, receipt and
|
||||||
|
cleanup regressions. The TypeScript consumers live in `tests/boundary/`,
|
||||||
|
`tests/evaluation/` and `tests/conformance/`; `npm run test:python` discovers
|
||||||
|
the Python checks here. Build first for checks that use the compiled runtime.
|
||||||
|
|
||||||
|
The agent-harness tests inject command outcomes and own temporary workspaces.
|
||||||
|
They do not invoke a real model or establish real-provider behavior. The
|
||||||
|
retained resource observer keeps its process-family CPU, memory and cleanup
|
||||||
|
guards. Public corpus identities and original receipt provenance stay intact.
|
||||||
|
|
||||||
|
The historical [allocation report and receipt JSON](../../../docs/verification/analysis-efficiency/mcp-integer-allocation.md)
|
||||||
|
belong to documentation. Their numerical sections are produced and checked by:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
node tests/support/analysis-efficiency/mcp-integer-report.mjs --check
|
||||||
|
```
|
||||||
|
|
||||||
|
The full experiment suite was retired. For historical reproduction, use the
|
||||||
|
recorded runtime and producer revisions; the [archived source tree](https://github.com/morluto/rea/tree/a2e2c29b59e8775b0e6aedb09a9eb16adf32646b/experiments/analysis-efficiency)
|
||||||
|
preserves the former layout. Current deterministic checks do not requalify
|
||||||
|
those historical measurements.
|
||||||
@@ -10,10 +10,14 @@ const { values } = parseArgs({
|
|||||||
check: { type: "boolean", default: false },
|
check: { type: "boolean", default: false },
|
||||||
},
|
},
|
||||||
});
|
});
|
||||||
|
const reportDirectory = new URL(
|
||||||
|
"../../../docs/verification/analysis-efficiency/",
|
||||||
|
import.meta.url,
|
||||||
|
);
|
||||||
const dataPath =
|
const dataPath =
|
||||||
values.data ?? new URL("mcp-integer-allocation.json", import.meta.url);
|
values.data ?? new URL("mcp-integer-allocation.json", reportDirectory);
|
||||||
const markdownPath =
|
const markdownPath =
|
||||||
values.markdown ?? new URL("mcp-integer-allocation.md", import.meta.url);
|
values.markdown ?? new URL("mcp-integer-allocation.md", reportDirectory);
|
||||||
const data = JSON.parse(await readFile(dataPath, "utf8"));
|
const data = JSON.parse(await readFile(dataPath, "utf8"));
|
||||||
assert.equal(data.status, "passed");
|
assert.equal(data.status, "passed");
|
||||||
const original = await readFile(markdownPath, "utf8");
|
const original = await readFile(markdownPath, "utf8");
|
||||||
|
|||||||
@@ -84,11 +84,9 @@
|
|||||||
"docs/**/*.{cjs,cts,js,jsx,mjs,mts,ts,tsx}",
|
"docs/**/*.{cjs,cts,js,jsx,mjs,mts,ts,tsx}",
|
||||||
".github/actions/**/*.{cjs,cts,js,jsx,mjs,mts,ts,tsx}",
|
".github/actions/**/*.{cjs,cts,js,jsx,mjs,mts,ts,tsx}",
|
||||||
"website/**/*.{cjs,cts,js,jsx,mjs,mts,ts,tsx}",
|
"website/**/*.{cjs,cts,js,jsx,mjs,mts,ts,tsx}",
|
||||||
"experiments/**/*.{cjs,cts,js,jsx,mjs,mts,ts,tsx}",
|
|
||||||
"!docs/.vitepress/cache/**",
|
"!docs/.vitepress/cache/**",
|
||||||
"!docs/.vitepress/dist/**",
|
"!docs/.vitepress/dist/**",
|
||||||
"!website/_site/**",
|
"!website/_site/**",
|
||||||
"!experiments/analysis-efficiency/results/**",
|
|
||||||
"*.{cjs,cts,js,jsx,mjs,mts,ts,tsx}"
|
"*.{cjs,cts,js,jsx,mjs,mts,ts,tsx}"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
|
|||||||
Reference in new issue
Block a user