Files
qqqysandqqqys 6775f24927 feat(extensions): add test failure report explorer (#13854)
* feat(extensions): add test failure report explorer

* fix(extensions): bound report snapshots before serialization

* fix(extensions): bound test report validation

---------

Co-authored-by: qqqys <[email protected]>
2026-10-11 12:58:55 +00:00
..

Vitest report fixtures

These reports were generated by the repository's locked Vitest 3.2.7 in an isolated temporary micro-project. They are not hand-written approximations of the reporter schema. provenance.json records the actual Node version, commands, process exits, source hashes, and raw and sanitized report hashes.

From the repository root, after installing the locked dependencies:

node integrations/test-failure-explorer/scripts/generate-fixtures.mjs

The generator rejects any Vitest version other than 3.2.7. It copies runner/ to a temporary directory, links the existing locked node_modules, and runs each case serially using the dedicated vitest.config.mjs. Each command uses an absolute --root, absolute --config, --reporter=default --reporter=json, and an absolute --outputFile.json. The two no-tests cases use the same empty directory; only the successful case adds --passWithNoTests.

Intentional test sources use .fixture.ts, rather than .test.ts, and are collected only by the dedicated generation config. They must remain excluded from ordinary package tests. The generator checks each expected process exit before recording a report and removes its temporary project after completion. The expected failures below are fixture inputs, not failing product tests.

Independently checkable outcomes

The expected assertions in this table come from the committed tiny source files, rather than the report parser under test. Reporter suite counters are also recorded in provenance: suites include nested describe nodes and file entries, so numTotalTestSuites must not be labeled as a file count.

Report Expected process exit Expected report facts
mixed.json 1 Two files; four assertions: three passed, one failed. Nested suites make the reporter suite count four.
file-error.json 1 One failed file with a collection-error message and zero assertions; zero failed assertions does not imply success.
statuses.json 0 One passed, two skipped (explicit skip and runtime skip), one todo. numPendingTests is two, while the actual row status is skipped.
no-tests-fail.json 1 Empty testResults, zero assertions, success: false.
no-tests-pass.json 0 Empty testResults, zero assertions, success: true.
unhandled.json 1 One passed assertion and success: true, despite a real unhandled rejection and failing process exit.
duplicate-titles.json 1 Two failed assertions with the exact same suite/title/fullName; both rows must remain addressable.
hostile-text.json 1 One failure with Unicode, ANSI, literal HTML/script/instruction text, and 160 diagnostic lines; diagnostic data must stay inert.
duplicate-paths.derived.json No execution claimed Explicitly derived from duplicate-titles.json: duplicate its file entry and double the total/failed suite and test counters. Two identical file paths and four rows must remain addressable.

Vitest's reporter does not emit the unhandled rejection in unhandled.json: its file has no collection-error message and its assertion has no failure message. Provenance records that the default reporter observed both Unhandled Rejection and the source's Fixture unhandled rejection marker. This sample proves that report success cannot establish process success. The product imports the report only; provenance is test evidence, not an additional runtime field from which to manufacture an execution status.

This run produces passed, failed, skipped, and todo assertion statuses. Any pending/disabled row-status mutation or other schema boundary created by a product test must be labeled derived. No claim is made that this runner execution produced those row statuses.

Sanitization and replay

Only machine-specific absolute path prefixes inside JSON string values are replaced: the temporary source root becomes /fixture-workspace, the repository becomes /fixture-repository, temporary output becomes /fixture-output, and the Node executable in recorded commands becomes /fixture-node. These are deterministic safe placeholders; report filenames and stack locations retain their source-relative suffixes. Counts, statuses, messages, durations, timestamps, locations and stacks are otherwise preserved. Paths in imported fixture reports are data, not paths to open on this machine.

rawSha256 hashes the actual runner bytes before sanitation; sanitizedSha256 hashes the exact committed JSON bytes. consoleSha256 records the default reporter output without retaining its machine-specific raw text. Every source file and the dedicated config are hashed in sources; the generation script is hashed in generatorSha256. Derived reports name their source hash and transformation and have no invented exit code.

Rerunning changes wall-clock timestamps, durations and stack timing, so it also updates report hashes and provenance. Replay must reproduce the semantic outcomes in the table, rather than identical timing-dependent JSON bytes. Large-array, oversized-field, count-mismatch and malformed-JSON cases should be constructed explicitly in tests from these reports; do not commit large synthetic reports as if they were measurements of a real repository.