morluto/rea
1
0 Fork 0
mirror of https://github.com/morluto/rea.git synced 2026-11-07 05:56:19 -10:00
Code Issues Packages Projects Releases 1 Wiki Activity

feat(environment): answer list filters, search, sorting and paging from the state

Browse Source 
Learn supported filters, plain search, sorting and paging from retained list recordings and evaluate them through shared local-state workflows.

Preserve whole-membership admission, Evidence, query normalization, counter representation and omission. Use successful retry coverage consistently for confirmation and pending-write cleanup. Unsupported semantics remain replayed or unknown.

Validated 623 adjacent source tests, exact-head CI, repeated regressions, real retained GitHub CLI/MCP pagination, actual owned HTTP fixture sessions and Codex task/review runs. Third-party application-write fidelity and exhaustive scale/provider qualification remain unverified; details and measurements are recorded in the PR.

Refs #4933, #6262.

Assisted-by: Kogasa (gensokyo-skills:kogasa-surprise-testing)
This commit is contained in:
Edgar Arakelyan authored and GitHub committed 2026-10-22 12:41:32 +00:00
1 parent e93e3b1e7d
commit 293832bafc
28 files changed
+2809 -368
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

docs/agent-environments.md
+10 -10
View File
@@ -422,12 +422,12 @@ member reports which applied:
Each read tool answers according to its view in the state:
| View | Answer |
| ---------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `item` | The entity the path arguments name, rendered into the recorded envelope of `envelope_ordinal`. Arguments match as replay matches them: each collection segment equals the held collection's segment as written or after percent-decoding, and the identity is the last argument percent-decoded, then as written; values are never re-split on `/`. Any entity the state holds answers, including one `data/add` loaded; query arguments only decide `replay.exact_match`. An entity the state does not hold answers the tool's latest recorded 400, 404 or 410 response with `state.view` `missing`, or is a tool error when none was recorded; one lacking a field every recording carried is a tool error; a body is never invented. |
| `list` | When the arguments name one of the lines the view checked exactly (its latest recordings before the first related write; every path and query value equal to the recording's, and every recorded query parameter supplied), that recorded envelope with its array rendered from the state: every entity of the collection in state order for `membership: "all"` or a line `all_record_ordinals` names, or the recorded members still held. A call with no query argument answers from such a line of the collection its path arguments name that only pages its list, with `replay.exact_match` false. A member lacking a field every recording carried makes the call a tool error rather than a partial list. |
| `document` | When the arguments name the document's checked line exactly, the state's value for it; a tool error when the state holds none. |
| `recorded` | The recording, as replay answers it. |
| View | Answer |
| ---------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `item` | The entity the path arguments name, rendered into the recorded envelope of `envelope_ordinal`. Arguments match as replay matches them: each collection segment equals the held collection's segment as written or after percent-decoding, and the identity is the last argument percent-decoded, then as written; values are never re-split on `/`. Any entity the state holds answers, including one `data/add` loaded; query arguments only decide `replay.exact_match`. An entity the state does not hold answers the tool's latest recorded 400, 404 or 410 response with `state.view` `missing`, or is a tool error when none was recorded; one lacking a field every recording carried is a tool error; a body is never invented. |
| `list` | When the arguments name one of the lines the view checked exactly (its latest recordings before the first related write; every path and query value equal to the recording's, and every recorded query parameter supplied), that recorded envelope with its array rendered from the state: every entity of the collection in state order for `membership: "all"` or a line `all_record_ordinals` names, or the recorded members still held. A call with no query argument answers from such a line of the collection its path arguments name that only pages its list, with `replay.exact_match` false. On a line the view's `query` answers, and for a call whose every query argument has a role there (an argument-less call aside), the answer is the query evaluated over the state: filtered, searched, sorted and paged, with counters at the number listed and `query.totals` at the number matched, and `state.membership: "query"`; such a call answers on a query or whole line of its collection with `replay.exact_match` false. A value an `inert` parameter never recorded, or a parameter without a role, replays its recording. A member lacking a field every recording carried makes the call a tool error rather than a partial list. |
| `document` | When the arguments name the document's checked line exactly, the state's value for it; a tool error when the state holds none. |
| `recorded` | The recording, as replay answers it. |
A list or document call that names no checked line, including a line recorded
after a write or with other query values, answers from the recording. Every
@@ -442,10 +442,10 @@ recorded (`rendered-oversize`) and replays the captured body. Query `POST`s
stay served and answer from their recordings with the reason `query-body`.
A line establishes whole-collection membership only from the beginning:
`offset`, `start`, `_start` and `page` must be zero when present. Positive
page numbers have an unknown zero-/one-based convention and stay recorded,
as do malformed paging values. An empty later page therefore cannot start
returning earlier entities after a write or `data/add`.
`offset`, `start`, `_start` and `page` must be zero when present. Positive or
malformed positions do not by themselves establish whole membership. A separate
query model may describe supported paging; an empty later page cannot start
returning earlier entities merely because it was empty in the capture.
`POST`, `PUT`, `PATCH` and `DELETE` tools are served only when the state's
[write plan](web-network-captures.md#environment-state) for them is served:
docs/specs/agent-environments/list-queries/plan.md
+14
View File
@@ -0,0 +1,14 @@
# Implementation plan
1. **Query module.** A domain module holds the roles, explains one line (trying roles by parameter name, filters and
inert for any parameter, at most 256 combinations), merges lines, evaluates a query over entities, and renders an
answer with counters and totals.
2. **Gate.** The list plan learns the query from the lines that are not whole, reading each body once for its listed
identities, role choices and counts; demotes a line whose envelope or fields do not reproduce; and adds `query` to
the view.
3. **Write gate.** A read on a query line is checked against the query over the replayed state, covering the
collection only for sort- and page-only answers that hold every match.
4. **Serving.** Line selection moves to its own module and gains the query line; list answers choose whole,
query or recorded members and render totals.
5. **Tests and docs.** Unit, domain and serving tests; the environment-state and agent-environment docs and the
analysis-view tool description.
docs/specs/agent-environments/list-queries/spec.md
+86
View File
@@ -0,0 +1,86 @@
# Answer list filters, search, sorting and paging from the state
Issue: https://github.com/morluto/rea/issues/4933
## Outcome
Agents list, filter, search and sort constantly. A list view answered a recorded request line from the state only when
that line listed the whole collection; any other line, such as `?shelf=a`, `?q=tide` or `?orderBy=name`, kept the
members it recorded, and a call with a value no line recorded replayed whichever recording was closest. A filtered
list therefore never showed what a write created or changed, and a search for anything new answered with an
unrelated recording.
A list view now learns what each of its query parameters does, from the lines it recorded, and answers those lines,
and calls with new values, by evaluating the query over the live state.
## Learning
For each checked line that does not list the whole collection, the first combination of roles that lists exactly what
the line recorded, from the collection's initial entities, explains it:
- `filter` on a field every listed item holds the value in: equal as text, or in an array directly or as an object's
`id`, `slug` or `name`;
- `search` for plain substring terms over the string fields the term matched in the listed items, ignoring case (URL fields only for an item
nothing else matched); an empty term matches everything, and a term that matched no field leaves the parameter inert
until another line's term does;
- `sort` by the field the value names, with `-` or an `asc`/`desc` suffix, and `direction` for a separate `asc`/`desc`
parameter: numbers by value, text ignoring case, null last ascending, ties (case-only differences included) in state
order; an empty selection needs no sort;
- `limit`, `offset` and `page` (counted from 1 or 0 only when the line distinguishes them) for paging; zero or a negative limit
lists every match;
- `inert` for a parameter that changes nothing, valid only for the values it recorded.
Search values resembling operators, field syntax, tags, quotes or wildcards replay rather than asserting a literal-substring result.
Paging values must be safe integers; malformed positions are not learned as inert flags. Identical decoded repeats
of a query value impose one constraint; conflicting repeated values remain unknown. Counter presence belongs to
each response envelope: page counters update where observed, and serving never adds a query-only total to another envelope.
Query totals retain their numeric or canonical decimal-string type. Unknown or differently formatted strings
(such as padded `"03"`) are not assigned an unverified formatting rule and stay as recorded.
`state.source` distinguishes a computed response from recording replay; `replay.exact_match` describes only the
retained source envelope and can be false for a correctly computed new query.
Parameter names decide which roles are tried (`sort` for `orderBy`, `search` for `q`), and any parameter may be a
filter or inert. No two parameters of one call take the same `sort`, `direction` or paging role: the answer would
depend on which applied last, and serving supplies a call's values in name order rather than as recorded, so such a
combination explains no line and such a call replays its recording. A line with more than 12 query parameters is left
unexplained, which bounds the search; names such as `__proto__` are kept as the query's own members. The lines' roles
merge: lines that disagree drop the parameter, inert values gather, search fields unite. The merged query keeps the
lines it reproduces, envelope and fields included; any other line keeps its recorded members. The view's `query` names
the roles, those lines, and the envelope members that count every match before paging (`totals`); a `number`-of-items
member such as `numberOfElements` counts the page.
## Serving and the write gate
- A call that names a query line exactly, or whose every query argument has a role (an argument-less call keeps
answering on a whole line), is answered by evaluating the query over the state, in the envelope of a query or whole
line of its collection, with counters at the number listed and totals at the number matched. The answer's state
provenance says `membership: "query"`.
- A value an inert parameter never recorded, or a parameter without a role, replays its recording as before.
- The write gate checks a read on a query line, or one whose every query parameter has a role on a line it did not
name, against the query over its copy of the state, as serving answers it: on the line replay ranks first, by the
most verbatim path values, then the most query values the read shares, then recorded order. Only a line whose
parameters all sort or page, and whose answer holds every match, covers the collection; any other covers the
entities it lists and those the line or the read listed that the state no longer holds, so a filtered list shows
whether a delete or create took effect without speaking for the rest of the collection. A failed read also covers
the held entities it listed that the answer leaves out, as a write may have moved them out of it. A read on a line
it did not name counts as gone only what it listed itself, since the line lists another call's members. A read with
no query answered on a query line, as serving does when no whole line fits, is checked against that line's recorded
members, as serving lists them. A read on a line that keeps its recorded members is compared with the totals it
recorded, as serving answers it. A read on a line it did not name is compared in its own envelope: that line's
envelope, such as a `next` link naming the line's own values, comes from another call, so it says nothing of a
write. A read whose query applies a sort shows where a create put its entity only when the sort leaves that entity
tied with another, and an unsorted one only when more than its entity matches, so only then does it fix the create's
insert position.
The roles are inferences from recorded behaviour and are stated as such. Nothing changes for a list whose lines all
list the whole collection, or that no role explains.
## Acceptance
- Unit tests: each role found from a line, including filters on arrays and object ids, search ignoring URLs, sort
spellings and nulls, paging by offset and by page, inert values; merging lines; values a role cannot apply.
- Domain: a tool recorded whole, filtered, searched, sorted and with a flag learns all four, and a create the filter
later lists is confirmed.
- Serving: after a served create, new filter, search and sort values list it; an unrecorded inert value replays.
- Measured: the fifty published captures' learned queries and write verdicts, and the six environments compared with
their applications, before and after.
docs/specs/agent-environments/list-queries/tasks.md
+11
View File
@@ -0,0 +1,11 @@
# Tasks
- [x] Explain recorded list lines with filter, search, sort, direction, paging and inert roles.
- [x] Merge roles across lines and keep the lines the query reproduces.
- [x] Count matches before paging in the envelope's totals.
- [x] Check reads on query lines in the write gate.
- [x] Serve query lines and calls with new values from the state.
- [x] Add unit, domain and serving tests.
- [x] Document list queries.
- [x] Measure the published captures and the six environments before and after.
- [x] Run static, formatting, duplication, documentation and test gates.
docs/specs/agent-environments/list-queries/verification.md
+233
View File
@@ -0,0 +1,233 @@
# Verification
## Maintainer integration, 2026-10-22
Review of `62a9adf6c` found one attributable retry-coverage defect: a successful
insertion-order retry returned only a boolean, so confirmation and pending-write
cleanup still used the failed query page's covered entities. Two deterministic
guards failed (2 failures/82 passes): an unseen update was falsely confirmed,
and a later actual item read could no longer confirm it. The retry now returns
its successful `CheckOutcome`; guards, targets, exercised writes, interference,
placement and cleanup use that result. No duplicate post-retry analysis is added.
All 623 adjacent source tests, three 161-test stability runs, 8 compiled served
workflow boundaries, pinned static/build/docs/9 README checks and native public
pagination pass. Independent targeted Codex source review reports no definite
remaining defect. Public sample: 20.19s wall/28.63s CPU/490.31MiB sampled family
RSS, no observed descendants left. Earlier CI `39776472358` passed on `62a9adf6c`,
but cannot replace CI for the subsequent correction. Live application-write
fidelity remains unverified; the failed-candidate policy already in main is
unchanged, not a newly qualified rollback guarantee.
Representation cross-check found and reproduced one further stale counter on
`1c9bb2e7`: canonical string total `"3"` remained `"3"` after removing an entity
(one failure, 42 passes). Query-total interpretation now has one shared owner for
inference and rendering, preserving number versus canonical decimal-string type.
Unknown, padded, nonintegral or unsafe representations stay unmodeled/recorded;
no display convention is guessed. The updated 621-test source batch and pinned
typecheck/lint pass. The 102 query/write tests pass three consecutive runs, as do
build, docs/9 README, format and native held-out pagination. Independent targeted
Codex source review found no definite remaining format defect. This later public
run took 20.13s wall/28.72s CPU/492.53MiB sampled family RSS, all observed children
exited; it must not be hidden by the earlier faster runs or called a speedup.
Historical counts below are pinned to their stated heads.
Latest-main integration: main advanced to `e93e3b1e7` (APK signing declarations).
Merged it without dropping that feature; the sole conflict was the shared tool
description, resolved from main with only the existing list-query clause added.
The integrated tree passes all 619 adjacent source tests, pinned typecheck/lint,
build, docs/9 README checks and format validation. All 80 CLI/MCP/schema boundaries
also passed on `02ff6304d` before this main merge. An interleaved pre-merge docs
attempt saw stale generated skill output, and format saw the temporary conflict;
both attempts are retained and the completed integration checks pass. They are
not attributed to a defect in main. Fresh-head CI and compiled schema verification
remain necessary. Earlier CI run `39773075218` succeeded on `665ec74c6`, not on
the subsequent corrections or integration.
Independent read-only Codex review of `665ec74c6` found one additional omission
case: intersecting query-total names discarded a valid total when another envelope
omitted it. A new service guard failed on that head (one failure, 41 passes).
The producer now gathers supported totals and rejects actual contradictory counts,
not absent members; the existing projection guard preserves omission. The updated
527-test adjacent batch passes, including deleting a locally loaded entity and
observing the remaining total, plus an explicit contradictory-count control.
This review was source-derived until the regression was run; no live application
mutation was used. The original 618-test/80-boundary counts below describe the
prior reviewed head, not a newly invented final-head run.
The historical contributor runs below are not newly executed checks. This integration
uses main `2c3dd946d` and contributor head `30e90fc64`, retaining main's zero-position
whole-list guard and shared path decoder. New deterministic guards reproduced five
failures before correction: unsupported search syntax, identical repeated constraints,
stale query-only counts, and ambiguous empty-page membership. They now pass. Related
cleanup gives decoded single-valued queries one owner, preserves per-envelope counter
omission, and uses safe numeric precision instead of a nine-digit paging limit.
Executed with pinned Node 24.18.0/npm 11.16.0 on Linux: 526 adjacent source tests,
92 catalog/schema/server source tests, 80 public CLI/MCP/schema boundary tests, and 9 README
checks; typecheck/lint, runtime build, docs and format checks passed. Compilation is
not evidence of live deep-provider behavior. The 99 query/write regressions also
passed three consecutive runs; reordered constraints and equivalent encodings are
covered without weakening conflicting-value or whole-page assertions.
Real-data oracle: five read-only GitHub labels observations, including a repeated
complete collection to check snapshot stability. Only three requests enter the HAR;
the native `per_page=3&page=2` response is held out. CLI and MCP views agree; serving
returns the exact three native label facts with Evidence, and reordered query names
and three warm reads agree. Main needs replay plus a full-list read/client slice;
the query model needs one read. Structured replies total 13,282 versus 1,383 bytes;
catalog discovery is 1,057 versus 1,090 bytes after provenance clarification.
| Completed task | Wall seconds | CPU seconds | Sampled family RSS MiB |
| ------------------------------------------------------- | -----------: | ----------: | ---------------------: |
| Main public workflow | 10.01 | 15.46 | 490.16 |
| Candidate public workflow, before catalog clarification | 10.33 | 16.19 | 495.90 |
| Main plus actual Codex task | 31.55 | 22.56 | 958.68 |
| Candidate plus Codex, original catalog | 31.09 | 18.77 | 1,006.82 |
| Candidate plus Codex, clarified catalog | 27.65 | 19.56 | 952.25 |
| 526-test source batch | 8.75 | 20.64 | 1,513.68 |
RSS is a 500 ms PID/start-identity process-family sample, not a kernel peak or hard
cap. Fresh runtime processes are separate from three later warm calls; OS/dependency
caches and host load are uncontrolled. No speedup or cache claim follows from these
single runs. All observed descendants exited. Node heap allowance was 2 GiB, 1 GiB
for runtime/model children; no ambient environment was persisted.
Actual Codex CLI (`gpt-6.1-sol`) returned correct IDs, names, Evidence and local/not-live
semantics in all three sessions. Main used two calls and 94,649 input tokens
(66,816 cached), 246 output. Candidate initially still made two calls: 79,752 input
(59,904 cached), 284 output. After clarifying that `state.source`, not
`replay.exact_match`, identifies computed answers, the unchanged prompt used one
call: 53,308 input (46,848 cached), 205 output. These are whole-session usage,
including common runner context, not reply-byte token estimates or a universal
agent-efficiency guarantee. Raw prompts, traces, exits and measurements were retained
locally, including intermediate failed test/typecheck attempts.
No GitHub mutation, live-application write fidelity, full fifty-capture census,
exhaustive search-language support or real deep-provider qualification was performed
in this integration. Unchanged-main create-response and ignored-write-query findings
are tracked separately in #6262 and are not claimed repaired by this PR.
Verified commit: 6db41d29 (this branch before this file, with #5787 and the write-plan branch's head `fe8ab584` merged in). Executed on macOS with Node v22.23.1 (the repository pins 24.18.0 in `.nvmrc`, which was
not installed); the probes were recorded against the live applications on six disposable Linux VMs.
## Gates
- `npm run build:cached`, `npm run check:fast` (typecheck and lint), `npm run format:check`, `npx knip`,
`npm run scan:todos`, `npm run docs:check`, `npm run verify:test-discovery` and `npm run mcp-catalog:check`: passed.
- `npm run jscpd`: 3.44% duplicated lines over 3650 files, under the 3.5% gate.
- `npx vitest run --project domain --project services`: 7630 passed, 5 pre-existing skips.
- `npm run test:prepare`, then `npm run test:focused --` the environment serve and export CLI boundaries, the serve
and served-writes MCP boundaries, the analysis-view CLI and MCP boundaries, the compact input schema boundary and
tool schema validity: 8 files, 78 passed.
New tests: `environmentStateListQuery` (9 unit tests: each role found from a line, including filters on arrays and
object ids, search that leaves out URLs, sort spellings and nulls, paging by offset and by page, inert values, a filter
that excludes nothing kept inert, merging lines and refusals), one domain test in `environmentStateWrites` (a tool
recorded whole, filtered, searched, sorted and with a flag learns all four, and a create the filter later lists is
confirmed) and one serving test in `environmentStateWriteAnswer` (after a served create, new filter, search and sort
values list it; an unrecorded inert value replays).
## Fifty published captures
The environment-state view ran over the 50 published captures with each manifest's template rules, on builds with and
without this change (both with the prefix-variance branch merged, which some manifests need).
- 161 list views in 37 environments learn a query, and 223 checked lines are now answered by evaluating it. Roles
learned: 27 filters, 37 searches, 35 sorts, 26 directions, 66 limits, 32 offsets, 25 page parameters and 220 inert
parameters (each valid only for its recorded values).
- No other view changed shape and every write verdict is unchanged (31 served write tools).
## Six environments against their applications
The six environments of the fidelity comparison (NetBox, Komga, Mealie, n8n, Shlink, self-hosted Sentry) were compared
with their applications by `environment-compare`, with and without this change, on the comparison's probes and the
held-out sessions recorded since: NetBox 98.5%, Komga 91.5%, Mealie 94.0%, n8n 71.6%, Shlink 87.1% and Sentry 24.5%,
unchanged. Those probes repeat the capture's own query values, which recorded members already answer, so they cannot
show this change.
New held-out probes were therefore recorded against Komga and Mealie, the two applications whose list views learned
queries a new value can exercise, each restored from its snapshot first. After the capture's read-only prefix (so the
driver learns the application's ids), each learned query was called with values no recording used: other search terms
and filter values drawn from the seeded entities, the other sort direction, a page size of 1 and the second page.
| Application | Query calls | Exact before | Exact after | Right entities in the right order, before | After |
| ----------- | ----------: | -----------: | ----------: | ----------------------------------------: | ----: |
| Mealie | 38 | 2 | 17 | 2 | 38 |
| Komga | 9 | 0 | 0 | 0 | 8 |
Every Mealie call now lists the entities the application lists, in its order, where 36 of them listed the recorded
members before: sorts in the other direction (9) and new recipe searches and category filters (6) now match exactly.
The calls that still differ, and 8 of Komga's 9, differ only in paging members of the envelope: `per_page`, `page`,
`total_pages` and `next`/`previous` (Komga's `pageable` and `totalPages`) keep the base line's values when a call asks
for another page size or page. One Komga search term matched more series than the learned `name`/`url` fields, since
Komga also searches fields the list does not show. Over the whole probe Mealie went from 133 to 148 of 171 steps.
## What a run found
The first census learned Komga's `unpaged=false` as a filter on a field that is false for every entity, so `true` would
have emptied the list. A filter that excludes nothing is now inert, and a parameter filters only where a line shows it
excluding something: 37 of the census's 64 filters became inert values this way. A `-name` sort value matched the field loosely before its sign was read, so it sorted ascending;
the sign is now read first.
## Review round
The branch merged the list-lines branch at de13b0c5, which carries the write-plan branch through 43d98ea9 and main
d8a9673f. The gate's line selection, moved here to `environmentStateCheckedLines.ts`, takes the newer rules there, and
serving answers a whole line's outgrown page from its recording before the query selection. It also fixed seven
findings Greptile raised on this branch:
- a sort of an emptied collection answers no entities;
- an empty or unmatched search term stays inert;
- the gate checks a read with new query values as serving answers it;
- a recorded-member line keeps its recorded totals in the gate;
- a line with more than 12 query parameters is left unexplained, bounding the search;
- `__proto__` and similar names stay the query's own members;
- case-only ties sort in state order.
Each new test fails on the previous head. The partial-page counter test from the list-lines branch now expects the
page, a query line here, to answer from the state with the state's total.
The census on this head, with the fidelity branch's compare command merged in as before, at first served 29 of 183
writes, two fewer than at d5a07e7a. Airbyte's `GET /sources?workspaceIds=…` names a workspace no line recorded, so the
gate now checked it on the other workspace's line, whose `next` link names that workspace. The read listed the right
sources, but with no write exercised the failure marked the create and patch before it as side effects, and tainted
the created source so the read after its delete went unchecked. A read on a line it did not name is now compared in
its own envelope, which a test that fails on the previous head covers. The census again serves 31 of 183, every
verdict as at d5a07e7a. Its 160 query views (161 before) differ from d5a07e7a only as the findings intend:
- empty search terms are inert in Keycloak, PhotoPrism's albums and labels, Tandoor's keywords and Umami;
- Gitea, Grafana and Tandoor's units drop a search term that matched no shown field, with the line that carried it
(Tandoor's units keep no query);
- PhotoPrism's photo query leaves line 199 and its 17 parameters unexplained, and learns lines 225 and 226 instead;
- the list-lines merge adds Bookstack's `total` and Linkding's `count` counters.
The six environments compare as before: NetBox 472 of 479 steps, Komga 549/600, Mealie 422/449, n8n 199/278, Shlink
176/202 and Sentry 91/372.
A second review round merged the list-lines branch at c68c63be, which carries the write-plan branch through 6257de93:
an update's identity member never takes a rule and a page size of zero bounds nothing, fixing two findings Greptile
raised here against that branch's code, and the paging rules move to `environmentStatePaging.ts`, which this branch's
line selection now imports. It also fixed three findings in this branch's gate:
- a read with no query answered on a query line is checked against that line's recorded members, as serving lists
them, rather than against every entity;
- a query read on a line it did not name counts as gone only what it listed, so it cannot confirm a delete of a book
that line listed;
- a read whose query applies a sort no longer fixes a create's insert position, so a later whole list can still try
another rule.
Each new test fails on the previous head. Over the 50 captures every view and write verdict is as on the previous head,
and the six environments compare exactly as before.
It then merged the list-lines branch at ed4b6dbb, which carries the write-plan branch through 466ed8b4: entities named
as serving names them, interference weighed per write, a zero page size bounding nothing only where a recording listed
items under it, and the other fixes of that branch's fifteenth round. Over the 50 captures every view and write verdict
is again as before, and the six environments compare exactly as before.
A third review round merged the list-lines branch at 8eea9f6d, which carries the write-plan branch through 7cef2fb2,
and fixed Greptile's finding on the second round's sort rule: a sorted read whose values tie shows the state's order
among them, so it still fixes a create's insert position, and a later whole list can no longer move the create where
that read showed otherwise. Its test fails on the previous head. Over the 50 captures every view and write verdict is
as before, and the six environments compare exactly as before. A follow-up narrowed it per entity: a sorted read fixes a create's position only when the
sort leaves the created entity itself tied, so a tie between other books no longer locks it. It also merged
the list-lines branch at 27dd6062, carrying the write-plan branch through 45a5c8b9.
docs/web-network-captures.md
+28
View File
@@ -618,6 +618,29 @@ when its content type is
JSON-like, it is UTF-8 text of at most 256 KiB, every number survives native
conversion exactly, and no member is named `__proto__`.
A list view also learns, from its checked lines that do not list the whole
collection, what each query parameter does, and names the result in `query`.
Each such line is explained by the first combination of roles that lists
exactly what it recorded from the collection's initial entities:
| Role | Parameter | Effect |
| ----------- | ------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `filter` | any, on a `field` every listed item holds the value in | entities whose field equals the value as text, or whose array field holds it directly or as an object's `id`, `slug` or `name` |
| `search` | `q`, `query`, `search`, `searchTerm`, `term`, `keyword`(s) | entities with one of `fields` containing the value, ignoring case; the fields are those it matched in the listed items, URLs only where nothing else matched |
| `sort` | `order`, `orderBy`, `ordering`, `sort`, `sortBy` (and `_by` spellings) | ordered by the field the value names (`name`, `-name`, `name:desc`, `name-ASC`): numbers by value, text ignoring case, null last ascending, ties in state order |
| `direction` | `dir`, `direction`, `order`, `orderDirection`, `sortDir`, `sortOrder` with `asc` or `desc` | the sort's direction |
| `limit` | `limit`, `per_page`, `perPage`, `page_size`, `size`, `count`, `take`, `itemsPerPage` | page size; zero or a negative value lists every match |
| `offset` | `offset`, `start`, `_start`, `skip` | matches skipped before the page |
| `page` | `page` | page number, counted from `first` (1 or 0, whichever explains the line) |
| `inert` | any | changes nothing for its recorded `values`; other values are not computed |
Lines that disagree on a parameter's role drop it, inert values gather and
search fields unite. `query.record_ordinals` are the lines the merged roles
reproduce, envelope and fields included, and `query.totals` the envelope
members that count every match before paging on each of them (a
`number`-of-items member counts the page). The roles are inferences from the
recorded lines; a line they do not reproduce keeps its recorded members.
`writes` lists one plan per tool whose request may change state, in tool
order. Its `effect` is `create` for a `POST` to a type's
`collection_template`, `update` for a `PUT`, `PATCH` or `POST` to that template
@@ -641,6 +664,11 @@ sending must be a clock, an increment or a slug, or the plan is
tool. Every member the first call sent is required, except by `PATCH` and
`POST` updates.
A read on a line `query` answers is checked against the query evaluated over
the replayed state. It covers the collection only when every parameter sorts or
pages and the answer holds every match; otherwise it covers the entities it
lists, and those the line or the read listed that the state no longer holds.
The view then replays the timeline on a copy of the initial state. Modelled
writes apply their effect; only a `2xx` write is modelled or an example, and
any other write (a redirect, an unanswered or unmodelled one) leaves the entity
src/application/environment/environmentStateAnswer.test.ts
+14
View File
@@ -288,6 +288,20 @@ describe("list answers", () => {
});
});
it("answers a recorded call that pages by two parameters as it recorded, whatever order they are supplied in", () => {
const books = servingFrom([
["GET", "/api/books", [{ id: 1 }, { id: 2 }, { id: 3 }]],
["GET", "/api/books?size=2&limit=1", [{ id: 1 }]],
]);
expect(answer(books, "get_books", { size: "2", limit: "1" })).toMatchObject(
{
body: [{ id: 1 }],
replay: { record_ordinal: 1, exact_match: true },
state: { source: "state", membership: "query", listed: 1 },
},
);
});
it("refuses a list whose member lacks a field every recording carried, never answering a partial body", () => {
const serving = servingOf();
load(serving, (entities) =>
src/application/environment/environmentStateAnswer.ts
+96 -132
View File
@@ -10,7 +10,6 @@ import {
renderStateList,
replaceAt,
scalarText,
segmentsOf,
type JsonObject,
type RecordedViewReason,
type StateEntity,
@@ -18,11 +17,12 @@ import {
type StateListView,
type StateView,
} from "../../domain/analysisView/environmentStateRules.js";
import {
pageBound,
pagesOnly,
} from "../../domain/analysisView/environmentStatePaging.js";
import { pageBound } from "../../domain/analysisView/environmentStatePaging.js";
import { isAbsence } from "../../domain/analysisView/environmentStateWriteChecks.js";
import {
evaluateQuery,
renderQueryList,
} from "../../domain/analysisView/environmentStateListQuery.js";
import type { AnalysisError } from "../../domain/analysisErrorBase.js";
import {
AnalysisInputError,
@@ -31,11 +31,9 @@ import {
import type { JsonValue } from "../../domain/jsonValue.js";
import { err, ok, type Result } from "../../domain/result.js";
import {
pathValueMatches,
replayAnswer,
replayEnvironmentTool,
selectReplayExchange,
selectsRecordedLine,
type ReplayArguments,
type ReplayResult,
} from "./environmentReplay.js";
@@ -46,8 +44,16 @@ import {
type EnvironmentSource,
type RecordedExchange,
} from "./environmentSource.js";
import {
callQuery,
callSegments,
checkedLine,
collectionMatch,
queryLine,
unfilteredLine,
type CallSegment,
} from "./environmentStateCallLines.js";
import type { EnvironmentStateStore } from "./environmentStateStore.js";
import { environmentToolArguments } from "./environmentTools.js";
/** Where a served answer came from: the live state, or a recording and why. */
export type StateProvenance =
@@ -71,7 +77,7 @@ export type StateProvenance =
readonly view: "list";
readonly type: string;
readonly collection: string;
readonly membership: "all" | "recorded";
readonly membership: "all" | "query" | "recorded";
readonly listed: number;
}
| {
@@ -174,42 +180,12 @@ export const stateAnswer = (
});
};
/** One collection segment an item call names: a template literal, or a supplied path value. */
export interface CallSegment {
readonly text: string;
readonly supplied: boolean;
}
/** An item call's collection segments and identity value, kept as supplied rather than joined and re-split. */
export interface ItemCall {
readonly segments: readonly CallSegment[];
readonly id: string;
}
/** A call's path segments: template literals and supplied values, or undefined when a path argument is missing. */
export const callSegments = (
tool: EnvironmentManifestTool,
args: ReplayArguments,
): CallSegment[] | undefined => {
const names = environmentToolArguments(tool).path;
const params = new Map(tool.params.map((param) => [param.index, param]));
const segments: CallSegment[] = [];
const template = tool.path_template
.slice(tool.path_template.indexOf("/") + 1)
.split("/");
for (const [index, text] of template.entries()) {
const param = params.get(index);
if (param === undefined) {
if (text !== "") segments.push({ text, supplied: false });
continue;
}
const value = args[names.get(param.name) ?? param.name];
if (typeof value !== "string") return undefined;
segments.push({ text: value, supplied: true });
}
return segments;
};
/** What an item call names, or undefined when a path argument is missing. */
export const itemCall = (
tool: EnvironmentManifestTool,
@@ -222,23 +198,6 @@ export const itemCall = (
: undefined;
};
/** How many supplied segments a held collection matches verbatim, when it matches the call under replay's path rule. */
export const collectionMatch = (
collection: string,
segments: readonly CallSegment[],
): number | undefined => {
const held = segmentsOf(collection);
if (held.length !== segments.length) return undefined;
let verbatim = 0;
for (const [index, { text, supplied }] of segments.entries()) {
const segment = held[index]!;
if (supplied ? !pathValueMatches(segment, text) : segment !== text)
return undefined;
if (supplied && segment === text) verbatim += 1;
}
return verbatim;
};
/** The entity an item call names, preferring verbatim collection spellings as replay does; the id is tried percent-decoded, then as written. */
export const namedEntity = (
store: EnvironmentStateStore,
@@ -362,49 +321,6 @@ const answerItem = (
);
};
/** The checked recorded line a list or document call names exactly, chosen among them as replay chooses; later or filtered recordings never qualify. */
const checkedLine = (
source: EnvironmentSource,
tool: EnvironmentManifestTool,
checked: readonly number[],
args: ReplayArguments,
): RecordedExchange | undefined => {
const lines = new Map<number, RecordedExchange>();
for (const ordinal of checked) {
const exchange = source.exchanges.get(ordinal);
if (exchange !== undefined && selectsRecordedLine(tool, exchange, args))
lines.set(ordinal, exchange);
}
return selectReplayExchange(tool, lines, args)?.exchange;
};
/** For a list of a whole collection, a call that supplies no query argument answers from a checked line of the collection its path arguments name that only pages it, chosen as replay chooses; a requested page answers from its recording. */
const unfilteredLine = (
source: EnvironmentSource,
tool: EnvironmentManifestTool,
checked: readonly number[],
args: ReplayArguments,
): RecordedExchange | undefined => {
const query = environmentToolArguments(tool).query;
const segments = callSegments(tool, args);
if (
segments === undefined ||
[...query.values()].some((argument) => args[argument] !== undefined)
)
return undefined;
const lines = new Map<number, RecordedExchange>();
for (const ordinal of checked) {
const exchange = source.exchanges.get(ordinal);
if (
exchange !== undefined &&
pagesOnly(exchange.query ?? undefined) &&
collectionMatch(collectionPathOf(exchange, false), segments) !== undefined
)
lines.set(ordinal, exchange);
}
return selectReplayExchange(tool, lines, args)?.exchange;
};
type StateDocumentView = Extract<StateView, { shape: "document" }>;
/** One tool call: the tool, the arguments it was called with, and whether they name the answering line exactly. */
@@ -414,62 +330,109 @@ interface ToolCall {
readonly exact: boolean;
}
/** A list call renders the members of the checked line it names from the state. */
/**
* The entities a list call shows on its line: every entity of the collection on a whole line, the query's matches on
* a line the query answers or for a call it computes, else the line's recorded members still held; undefined when the
* query cannot apply the call's values.
*/
const listSelection = (
store: EnvironmentStateStore,
{ tool, args, exact }: ToolCall,
view: StateListView,
line: {
readonly base: RecordedExchange;
readonly recorded: readonly JsonObject[];
},
):
| {
readonly members: readonly StateEntity[];
readonly membership: "all" | "query" | "recorded";
readonly total: number;
}
| undefined => {
const collection = collectionPathOf(line.base, false);
const values = callQuery(tool, args);
// A named query line, or a call with query arguments answered on a line it did not name, evaluates the query.
const queried =
view.query !== undefined &&
(exact
? view.query.record_ordinals.includes(line.base.ordinal)
: values.size > 0);
if (queried) {
const computed = evaluateQuery(
view.query!.parameters,
values,
store.inCollection(view.type, collection),
);
return computed === undefined
? undefined
: { members: computed.page, membership: "query", total: computed.total };
}
if (listsAll(view, line.base.ordinal)) {
const members = store.inCollection(view.type, collection);
return { members, membership: "all", total: members.length };
}
const idField = store.type(view.type)?.id_field ?? "";
const members = line.recorded.flatMap((item) => {
const id = scalarText(item[idField]);
const entity =
id === undefined ? undefined : store.entity(view.type, collection, id);
return entity === undefined ? [] : [entity];
});
return { members, membership: "recorded", total: members.length };
};
/** A list call renders the entities its line selects from the state into that line's recorded envelope. */
const answerList = (
serving: StateServing,
{ tool, args, exact }: ToolCall,
call: ToolCall,
view: StateListView,
base: RecordedExchange,
): Result<StateAnswer, AnalysisError> => {
const { source, store } = serving;
const { tool, args, exact } = call;
const envelope = recordedJson(base);
const recorded =
envelope === undefined ? undefined : listItems(envelope, view.pointer);
if (envelope === undefined || recorded === undefined)
return fromRecording(source, tool, args, "body-unusable");
const collection = collectionPathOf(base, false);
const type = store.type(view.type);
const all = listsAll(view, base.ordinal);
// A requested page the collection has outgrown is answered as recorded, never with more items than it asked for.
const size = exact
? pageBound(base.query ?? undefined, recorded.length)
: undefined;
if (
all &&
listsAll(view, base.ordinal) &&
size !== undefined &&
store.inCollection(view.type, collection).length > size
store.inCollection(view.type, collectionPathOf(base, false)).length > size
)
return fromRecording(source, tool, args, "page-exceeded");
const members: readonly StateEntity[] = all
? store.inCollection(view.type, collection)
: recorded.flatMap((item) => {
const id = scalarText(item[type?.id_field ?? ""]);
const entity =
id === undefined
? undefined
: store.entity(view.type, collection, id);
return entity === undefined ? [] : [entity];
});
const volatile = new Set(type?.volatile_fields);
const selection = listSelection(store, call, view, { base, recorded });
if (selection === undefined)
return fromRecording(source, tool, args, "unrecorded-request");
const volatile = new Set(store.type(view.type)?.volatile_fields);
const items: JsonObject[] = [];
for (const entity of members) {
for (const entity of selection.members) {
const shown = shownEntity(tool, view, entity, volatile);
if (!shown.ok) return shown;
items.push(shown.value);
}
return stateAnswer(
serving,
{ exchange: base, exact },
renderStateList(view, envelope, items),
{
source: "state",
view: "list",
type: view.type,
collection,
membership: all ? "all" : "recorded",
listed: items.length,
},
);
const body =
view.query === undefined || selection.membership === "recorded"
? renderStateList(view, envelope, items)
: renderQueryList(
{ ...view, query: view.query },
envelope,
items,
selection.total,
);
return stateAnswer(serving, { exchange: base, exact }, body, {
source: "state",
view: "list",
type: view.type,
collection: collectionPathOf(base, false),
membership: selection.membership,
listed: items.length,
});
};
/** A document call answers the state's value for the document on its checked line's status and provenance. */
@@ -504,7 +467,8 @@ const answerCheckedLine = (
const base =
named ??
(view.shape === "list"
? unfilteredLine(serving.source, tool, allLines(view), args)
? (unfilteredLine(serving.source, tool, allLines(view), args) ??
queryLine(serving.source, tool, view, args))
: undefined);
if (base === undefined)
return fromRecording(serving.source, tool, args, "unrecorded-request");
src/application/environment/environmentStateCallLines.ts
+152
View File
@@ -0,0 +1,152 @@
import {
allLines,
collectionPathOf,
segmentsOf,
type StateListView,
} from "../../domain/analysisView/environmentStateRules.js";
import { pagesOnly } from "../../domain/analysisView/environmentStatePaging.js";
import {
pathValueMatches,
selectReplayExchange,
selectsRecordedLine,
type ReplayArguments,
} from "./environmentReplay.js";
import type {
EnvironmentManifestTool,
EnvironmentSource,
RecordedExchange,
} from "./environmentSource.js";
import { environmentToolArguments } from "./environmentTools.js";
/** One collection segment an item call names: a template literal, or a supplied path value. */
export interface CallSegment {
readonly text: string;
readonly supplied: boolean;
}
/** A call's path segments: template literals and supplied values, or undefined when a path argument is missing. */
export const callSegments = (
tool: EnvironmentManifestTool,
args: ReplayArguments,
): CallSegment[] | undefined => {
const names = environmentToolArguments(tool).path;
const params = new Map(tool.params.map((param) => [param.index, param]));
const segments: CallSegment[] = [];
const template = tool.path_template
.slice(tool.path_template.indexOf("/") + 1)
.split("/");
for (const [index, text] of template.entries()) {
const param = params.get(index);
if (param === undefined) {
if (text !== "") segments.push({ text, supplied: false });
continue;
}
const value = args[names.get(param.name) ?? param.name];
if (typeof value !== "string") return undefined;
segments.push({ text: value, supplied: true });
}
return segments;
};
/** How many supplied segments a held collection matches verbatim, when it matches the call under replay's path rule. */
export const collectionMatch = (
collection: string,
segments: readonly CallSegment[],
): number | undefined => {
const held = segmentsOf(collection);
if (held.length !== segments.length) return undefined;
let verbatim = 0;
for (const [index, { text, supplied }] of segments.entries()) {
const segment = held[index]!;
if (supplied ? !pathValueMatches(segment, text) : segment !== text)
return undefined;
if (supplied && segment === text) verbatim += 1;
}
return verbatim;
};
/** The checked recorded line a list or document call names exactly, chosen among them as replay chooses; later or filtered recordings never qualify. */
export const checkedLine = (
source: EnvironmentSource,
tool: EnvironmentManifestTool,
checked: readonly number[],
args: ReplayArguments,
): RecordedExchange | undefined => {
const lines = new Map<number, RecordedExchange>();
for (const ordinal of checked) {
const exchange = source.exchanges.get(ordinal);
if (exchange !== undefined && selectsRecordedLine(tool, exchange, args))
lines.set(ordinal, exchange);
}
return selectReplayExchange(tool, lines, args)?.exchange;
};
/** For a list of a whole collection, a call that supplies no query argument answers from a checked line of the collection its path arguments name that only pages it, chosen as replay chooses; a requested page answers from its recording. */
export const unfilteredLine = (
source: EnvironmentSource,
tool: EnvironmentManifestTool,
checked: readonly number[],
args: ReplayArguments,
): RecordedExchange | undefined => {
const query = environmentToolArguments(tool).query;
const segments = callSegments(tool, args);
if (
segments === undefined ||
[...query.values()].some((argument) => args[argument] !== undefined)
)
return undefined;
const lines = new Map<number, RecordedExchange>();
for (const ordinal of checked) {
const exchange = source.exchanges.get(ordinal);
if (
exchange !== undefined &&
pagesOnly(exchange.query ?? undefined) &&
collectionMatch(collectionPathOf(exchange, false), segments) !== undefined
)
lines.set(ordinal, exchange);
}
return selectReplayExchange(tool, lines, args)?.exchange;
};
/** A call's query arguments by parameter name. */
export const callQuery = (
tool: EnvironmentManifestTool,
args: ReplayArguments,
): Map<string, string> =>
new Map(
[...environmentToolArguments(tool).query].flatMap(
([name, argument]): [string, string][] => {
const value = args[argument];
return value === undefined ? [] : [[name, String(value)]];
},
),
);
/** For a list with a query, a call whose every query argument has a role answers on a query or whole line of the collection its path arguments name. */
export const queryLine = (
source: EnvironmentSource,
tool: EnvironmentManifestTool,
view: StateListView,
args: ReplayArguments,
): RecordedExchange | undefined => {
const query = view.query;
const segments = callSegments(tool, args);
if (
query === undefined ||
segments === undefined ||
[...callQuery(tool, args).keys()].some(
(name) => !Object.hasOwn(query.parameters, name),
)
)
return undefined;
const lines = new Map<number, RecordedExchange>();
for (const ordinal of [...query.record_ordinals, ...allLines(view)]) {
const exchange = source.exchanges.get(ordinal);
if (
exchange !== undefined &&
collectionMatch(collectionPathOf(exchange, false), segments) !== undefined
)
lines.set(ordinal, exchange);
}
return selectReplayExchange(tool, lines, args)?.exchange;
};
src/application/environment/environmentStateCollections.ts
+2 -5
View File
@@ -1,9 +1,6 @@
import { collectionPathOf } from "../../domain/analysisView/environmentStateRules.js";
import {
callSegments,
collectionMatch,
type StateServing,
} from "./environmentStateAnswer.js";
import type { StateServing } from "./environmentStateAnswer.js";
import { callSegments, collectionMatch } from "./environmentStateCallLines.js";
import type { ReplayArguments } from "./environmentReplay.js";
import type { EnvironmentManifestTool } from "./environmentSource.js";
src/application/environment/environmentStateWriteAnswer.test.ts
+145 -17
View File
@@ -11,6 +11,7 @@ import {
DELETE_NOT_FOUND,
FIRST_BOOK,
LIST_LINE_ROWS,
QUERY_ROWS,
SECOND_BOOK,
THIRD_BOOK,
book,
@@ -581,24 +582,26 @@ describe("list lines", () => {
},
});
if (!loaded.ok) throw loaded.error;
expect(
books.answer("get_orgs_books_by_id", {
id: "2",
...Object.fromEntries(new URLSearchParams(query)),
limit: "50",
}),
).toMatchObject({
body: whole ? [beta] : [],
state: {
source: "state",
membership: whole ? "all" : "recorded",
listed: whole ? 1 : 0,
},
const answer = books.answer("get_orgs_books_by_id", {
id: "2",
...Object.fromEntries(new URLSearchParams(query)),
limit: "50",
});
expect(answer.body, query).toEqual(whole ? [beta] : []);
if (whole)
expect(answer).toMatchObject({
state: { source: "state", membership: "all", listed: 1 },
});
else {
expect(answer).toMatchObject({
state: { source: expect.stringMatching(/^(state|recording)$/u) },
});
expect(answer).not.toMatchObject({ state: { membership: "all" } });
}
}
});
it("lists a created book on the line that lists every book, while a filtered line keeps its recorded members", () => {
it("lists a created book on the line that lists every book, and a filtered line by its filter", () => {
const books = servedFrom(LIST_LINE_ROWS);
const created = books.answer("post_books", CREATE_NEW_BOOK).body as Book;
expect(books.answer("get_books")).toMatchObject({
@@ -607,11 +610,134 @@ describe("list lines", () => {
});
expect(books.answer("get_books", { name: "First Book" })).toMatchObject({
body: bookPage([FIRST_BOOK]),
state: { source: "state", membership: "recorded", listed: 1 },
state: { source: "state", membership: "query", listed: 1 },
});
});
});
describe("list queries", () => {
it.each([3, "3"])(
"preserves query total %j across omission, retaining its type and rejecting contradictions",
(initialTotal) => {
const alpha = { id: 1, name: "Alpha" },
beta = { id: 2, name: "Beta" },
gamma = { id: 3, name: "Gamma" };
for (const contradicts of [false, true]) {
const books = servedFrom([
["GET", "/api/books", { items: [alpha, beta, gamma] }],
[
"GET",
"/api/books?limit=1",
{ items: [alpha], total: initialTotal },
],
[
"GET",
"/api/books?limit=2",
{ items: [alpha, beta], ...(contradicts ? { total: 20 } : {}) },
],
]);
expect(books.state.views.get("get_books")).toMatchObject({
query: { totals: contradicts ? [] : ["total"] },
});
const snapshot = books.store.snapshot();
const loaded = books.store.replace({
...snapshot,
rea_state: {
...snapshot.rea_state,
entities: snapshot.rea_state.entities.filter(
(entity) => entity.id !== "3",
),
},
});
if (!loaded.ok) throw loaded.error;
expect(books.answer("get_books", { limit: "1" })).toMatchObject({
body: {
items: [alpha],
total: contradicts
? initialTotal
: typeof initialTotal === "string"
? "2"
: 2,
},
state: { source: "state", membership: "query" },
});
if (!contradicts)
expect(
books.answer("get_books", { limit: "2" }).body,
).not.toHaveProperty("total");
}
},
);
it("replays unsupported search syntax instead of inventing an empty state-backed result", () => {
const books = servedFrom(QUERY_ROWS);
for (const q of ["title:Gamma", "#sea", '"First Book"']) {
expect(books.answer("get_books", { q }), q).toMatchObject({
state: { source: "recording", reason: "unrecorded-request" },
});
}
expect(books.answer("get_books", { q: "SECOND" })).toMatchObject({
state: { source: "state", membership: "query", listed: 1 },
});
});
it("updates on each line only the counters its recording counted, so a partial page keeps its total", () => {
it("updates a query-only page counter without adding it to an envelope that never recorded it", () => {
const alpha = { id: 1, name: "Alpha", shelf: "a" };
const beta = { id: 2, name: "Beta", shelf: "b" };
const gamma = { id: 3, name: "Gamma", shelf: "a" };
const books = servedFrom([
["GET", "/api/books", { items: [alpha, beta], total: 2 }],
[
"GET",
"/api/books?shelf=a",
{ items: [alpha], total: 1, numberOfElements: 1 },
],
["GET", "/api/books/1", alpha],
["GET", "/api/books/2", beta],
["POST", "/api/books", gamma, 201, { name: "Gamma", shelf: "a" }],
["GET", "/api/books/3", gamma],
["GET", "/api/books", { items: [alpha, beta, gamma], total: 3 }],
]);
const created = books.answer("post_books", {
body: { name: "Delta", shelf: "a" },
}).body;
expect(books.answer("get_books", { shelf: "a" })).toMatchObject({
body: { items: [alpha, created], total: 2, numberOfElements: 2 },
state: { source: "state", membership: "query", listed: 2 },
});
expect(books.answer("get_books").body).toEqual({
items: [alpha, beta, created],
total: 3,
});
});
it("filters, searches and sorts the state for values no line recorded, created books included", () => {
const books = servedFrom(QUERY_ROWS);
const created = books.answer("post_books", {
body: { name: "New Book", shelf: "b" },
}).body as Book;
const listed = (args: Record<string, string>) => {
const answer = books.answer("get_books", args);
return {
ids: (answer.body as Book[]).map((entry) => entry["id"]),
state: (answer as { readonly state?: unknown }).state,
};
};
expect(listed({ shelf: "b" })).toMatchObject({
ids: [2, created["id"]],
state: { source: "state", membership: "query", listed: 2 },
});
expect(listed({ q: "NEW" }).ids).toEqual([created["id"]]);
expect(listed({ sort: "-id" }).ids).toEqual([created["id"], 2, 1]);
expect(listed({ sort: "-id", shelf: "a" }).ids).toEqual([1]);
// A value the flag never recorded, or a parameter the query has no role for, replays as before.
expect(listed({ include: "none" }).state).toMatchObject({
source: "recording",
reason: "unrecorded-request",
});
});
it("keeps a whole line's total following the state beside a partial page, which its query now answers", () => {
const partial = {
content: [FIRST_BOOK],
totalElements: 2,
@@ -626,8 +752,10 @@ describe("list lines", () => {
expect(books.answer("get_books")).toMatchObject({
body: bookPage([FIRST_BOOK, SECOND_BOOK, created]),
});
// The partial page is a query line here: its page and total come from the state, which now holds three books.
expect(books.answer("get_books", { page: "1", size: "1" })).toMatchObject({
body: partial,
body: { ...partial, totalElements: 3 },
state: { membership: "query" },
});
});
});
src/application/environment/environmentStateWriteAnswer.ts
+1 -2
View File
@@ -40,9 +40,8 @@ import {
type ReplayArguments,
} from "./environmentReplay.js";
import type { EnvironmentManifestTool } from "./environmentSource.js";
import { callSegments, collectionMatch } from "./environmentStateCallLines.js";
import {
callSegments,
collectionMatch,
itemCall,
missingAnswer,
namedEntity,
src/application/environment/environmentTools.test.ts
+13
View File
@@ -39,6 +39,19 @@ const tool = (name: string): EnvironmentManifestTool => {
};
describe("environmentToolDefinitions", () => {
it("distinguishes computed answers from their retained envelope's match metadata", () => {
const definitions = environmentToolDefinitions(source, "query-provenance", {
dataPlane: "state",
});
for (const definition of definitions)
expect(definition.description).toContain(
"state.source, not replay.exact_match, identifies state vs recording",
);
expect(environmentToolDefinitions(source)[0]?.description).not.toContain(
"state.source",
);
});
it("maps every manifest tool in order with a replay description naming its records and evidence", () => {
expect(definitions.map(({ name }) => name)).toEqual(
source.manifest.tools.map(({ name }) => name),
src/application/environment/environmentTools.ts
+1 -1
View File
@@ -408,7 +408,7 @@ export const environmentToolDefinitions = (
return refused();
const answers =
dataPlane === "state"
? "answers from the served environment state where the state backs this tool, otherwise replays retained records"
? "answers from the served environment state where the state backs this tool; state.source, not replay.exact_match, identifies state vs recording"
: "replays retained records";
const definitions: EnvironmentToolDefinition[] = [];
for (const tool of source.manifest.tools) {
src/contracts/analysisViewToolContracts.ts
+1 -1
View File
@@ -17,7 +17,7 @@ export const ANALYSIS_VIEW_TOOL_CONTRACTS = [
...toolContractMetadata("inspect_analysis_view"),
kind: "application",
description:
'Use javascript-routes collections to query route patterns, source-linked occurrences, source status, unknowns or summary counts from retained JavaScript application or web-bundle Evidence. Inspect selected views of completed Frida session status, GDB/objdump command observations, Go build metadata, Rust embedded dependencies, binary-layout, JavaScript application, native analyze_function, inspect_artifact, inventory_artifact, inspect_wasm_artifact, inspect_macho, decode_interface_builder, inspect_evm_interface, inspect_managed_artifact, inspect_managed_members, inspect_android_package, inspect_dart_aot, inspect_il2cpp_metadata, inspect_model_artifact, inspect_gpu_code_objects or inspect_web_network_capture Evidence without repeating analysis. Command summary views preserve ordered GDB records or opaque objdump stdout/stderr once, source parameters, completion facts and parent identity. They do not establish current target state. Historical capture summaries return identity and counts without bodies or archive inventories. Capture views page retained records: view {kind: "capture", detail: "preview" | "complete", offset: 0, limit: 20}; preview selects recorded headline fields and binary descriptors, complete returns exact records with all sidecars. Offsets follow the selected parent record order, preserving original producer ordinals. Paging never reopens the source or reruns the decoder. Go metadata uses kind go, collection dependencies/settings, exact paths/keys or indices, fields and pagination. Row index refers to the original collection; empty selections return no rows, and dependency coverage remains unknown. Rust metadata uses kind rust, collection packages/edges, exact names/versions or original indices, selected fields, edge from_indices/to_indices and matching-row pagination. Embedded-record coverage is separate from unknown runtime dependencies. For inspect_pcap_capture parents, kind pcap pages endpoints, dns-queries, tls-sni, protocols, http-uris or unattributed-addresses. Select an exact query for dns-addresses, or a tshark/capinfos stream for UTF-16 diagnostics text. Page totals count retained facts; source coverage remains explicit. Qlog summaries keep profile/scope and counts; kind qlog pages traces, streams, loss-declarations, incomplete-records, contributions, ranges or final-sizes. Except traces, select trace_index; stream-specific facets also require exact stream_id and direction. Facts retain all-recorded-events-before-selection scope, and interval overlap or endpoint loss declarations do not prove retransmission or network loss. The api-surface view groups retained HAR requests by method, origin and format-rule path template; templated segments are inferences. It reads no headers or bodies and reports no query values. The environment-manifest view projects the same HAR requests on the primary origin (most JSON-like records) into a replayable tool surface: view {kind: "environment-manifest", template_rules: ["after-uid-literal", "opaque-token", "sibling-variance"], include_query_parameter_names: true}. Each tool has a deterministic name, a path template and parameters from the four format rules plus the selected opt-in rules, observed query parameter names, a value-free body shape for POST/PUT/PATCH, status and content-type sets and record ordinals; records are pointers into the parent, excluded groups carry a reason and three fixed unknowns are always stated. Templates, parameters and names are inferences; the view copies no header, cookie or body values and refuses parents inspected with sensitive values. A capture_browser_scenario parent that selected response bodies or header values projects the same way, one record per network transaction in request order; it relies on capture-time redaction of declared secrets instead of that refusal, and a transaction whose URL was redacted is left unrepresented. The environment-state view extracts, from the same parents, the initial state the recorded reads observed: view {kind: "environment-state", template_rules: [...]} with the rules of the manifest it accompanies. It reads response bodies recorded before the first related write and reports entity types with an identity field (proved by a path parameter or inferred from a field name), entities in collection order with per-view values where views differ, documents for non-parameterized reads that are not entities and have one initial request line (item-shaped paths stay recorded), and a render plan per read tool: item and list views reproduce their latest recordings from the state, and every other read tool is a recorded view with a reason. It also reports a write plan per state-changing tool (create, update, delete or document, built from its first successful call) and write_clock; a plan is served only when replaying the capture shows a later read confirming its effect and none contradicting it. Types and identity are inferences; values are recorded data, so it refuses parents inspectLine truncated
'Use javascript-routes collections to query route patterns, source-linked occurrences, source status, unknowns or summary counts from retained JavaScript application or web-bundle Evidence. Inspect selected views of completed Frida session status, GDB/objdump command observations, Go build metadata, Rust embedded dependencies, binary-layout, JavaScript application, native analyze_function, inspect_artifact, inventory_artifact, inspect_wasm_artifact, inspect_macho, decode_interface_builder, inspect_evm_interface, inspect_managed_artifact, inspect_managed_members, inspect_android_package, inspect_dart_aot, inspect_il2cpp_metadata, inspect_model_artifact, inspect_gpu_code_objects or inspect_web_network_capture Evidence without repeating analysis. Command summary views preserve ordered GDB records or opaque objdump stdout/stderr once, source parameters, completion facts and parent identity. They do not establish current target state. Historical capture summaries return identity and counts without bodies or archive inventories. Capture views page retained records: view {kind: "capture", detail: "preview" | "complete", offset: 0, limit: 20}; preview selects recorded headline fields and binary descriptors, complete returns exact records with all sidecars. Offsets follow the selected parent record order, preserving original producer ordinals. Paging never reopens the source or reruns the decoder. Go metadata uses kind go, collection dependencies/settings, exact paths/keys or indices, fields and pagination. Row index refers to the original collection; empty selections return no rows, and dependency coverage remains unknown. Rust metadata uses kind rust, collection packages/edges, exact names/versions or original indices, selected fields, edge from_indices/to_indices and matching-row pagination. Embedded-record coverage is separate from unknown runtime dependencies. For inspect_pcap_capture parents, kind pcap pages endpoints, dns-queries, tls-sni, protocols, http-uris or unattributed-addresses. Select an exact query for dns-addresses, or a tshark/capinfos stream for UTF-16 diagnostics text. Page totals count retained facts; source coverage remains explicit. Qlog summaries keep profile/scope and counts; kind qlog pages traces, streams, loss-declarations, incomplete-records, contributions, ranges or final-sizes. Except traces, select trace_index; stream-specific facets also require exact stream_id and direction. Facts retain all-recorded-events-before-selection scope, and interval overlap or endpoint loss declarations do not prove retransmission or network loss. The api-surface view groups retained HAR requests by method, origin and format-rule path template; templated segments are inferences. It reads no headers or bodies and reports no query values. The environment-manifest view projects the same HAR requests on the primary origin (most JSON-like records) into a replayable tool surface: view {kind: "environment-manifest", template_rules: ["after-uid-literal", "opaque-token", "sibling-variance"], include_query_parameter_names: true}. Each tool has a deterministic name, a path template and parameters from the four format rules plus the selected opt-in rules, observed query parameter names, a value-free body shape for POST/PUT/PATCH, status and content-type sets and record ordinals; records are pointers into the parent, excluded groups carry a reason and three fixed unknowns are always stated. Templates, parameters and names are inferences; the view copies no header, cookie or body values and refuses parents inspected with sensitive values. A capture_browser_scenario parent that selected response bodies or header values projects the same way, one record per network transaction in request order; it relies on capture-time redaction of declared secrets instead of that refusal, and a transaction whose URL was redacted is left unrepresented. The environment-state view extracts, from the same parents, the initial state the recorded reads observed: view {kind: "environment-state", template_rules: [...]} with the rules of the manifest it accompanies. It reads response bodies recorded before the first related write and reports entity types with an identity field (proved by a path parameter or inferred from a field name), entities in collection order with per-view values where views differ, documents for non-parameterized reads that are not entities and have one initial request line (item-shaped paths stay recorded), and a render plan per read tool: item and list views reproduce their latest recordings from the state (a list also learns the filters, search, sort and paging its recordings verify, and answers new values of them from the state), and every other read tool is a recorded view with a reason. It also reports a write plan per state-changing tool (create, update, delete or document, built from its first successful call) and write_clock; a plan is served only when replaying the capture shows a later read confirming iLine truncated
inputSchema: inspectAnalysisViewInputSchema,
outputSchema: evidenceResultOf(analysisViewResultSchema),
examples: [
src/domain/analysisView/EnvironmentStateWriteSimulation.test.ts
+97
View File
@@ -439,9 +439,106 @@ describe("checked lines", () => {
checked_record_ordinals: [6, 8, 9],
});
});
const TWICE = "/api/orgs/%61%63me1/books";
const shelved = (entry: Book, shelf: string): Book => ({ ...entry, shelf });
const ENCODED_LINE: TimedCaptureRow = [
"GET",
ENCODED,
[shelved(FIRST_BOOK, "a"), shelved(SECOND_BOOK, "b")],
];
const TWICE_LINE: TimedCaptureRow = [
"GET",
TWICE,
[shelved(THIRD_BOOK, "a"), shelved(RENAMED, "b")],
];
const SHELF_LINE: TimedCaptureRow = [
"GET",
`${ENCODED}?shelf=a`,
[shelved(FIRST_BOOK, "a")],
];
/** Lines of one organisation spelled two encoded ways, deletes each whole list confirms, then a plain read of new query values still listing the deleted book. */
const deletedThenQueried = (
lines: readonly TimedCaptureRow[],
read: string,
): TimedCaptureRow[] => [
...lines,
["GET", OTHER, [shelved(SECOND_BOOK, "a")]],
["DELETE", `${TWICE}/1`, "", 204],
["DELETE", `${OTHER}/2`, "", 204],
["GET", TWICE, [shelved(THIRD_BOOK, "a")]],
["GET", OTHER, []],
["GET", read, [shelved(RENAMED, "b")]],
];
const deleteOf = (rows: readonly TimedCaptureRow[]) =>
environmentStateViewOf(timedCaptureEvidence(rows), [
"sibling-variance",
]).writes.find((candidate) => candidate.effect === "delete");
it("checks a read on new query values against the line serving ranks first by the values it shares", () => {
const rows = deletedThenQueried(
[
ENCODED_LINE,
TWICE_LINE,
SHELF_LINE,
["GET", `${TWICE}?shelf=b`, [shelved(RENAMED, "b")]],
["GET", `${ENCODED}?limit=1`, [shelved(FIRST_BOOK, "a")]],
],
`${PLAIN}?shelf=b&limit=5`,
);
// Serving lists the twice-encoded collection, whose `shelf=b` line the read shares a value with.
expect(deleteOf(rows)).toMatchObject({
served: false,
contradicted_by: 10,
});
});
it("checks a read on new query values against the line serving takes first in record order", () => {
const rows = deletedThenQueried(
[TWICE_LINE, ENCODED_LINE, SHELF_LINE],
`${PLAIN}?shelf=b`,
);
// No line shares a value, so serving lists the collection of the line recorded first.
expect(deleteOf(rows)).toMatchObject({
served: false,
contradicted_by: 8,
});
});
});
describe("insert rules", () => {
it.each([false, true])(
"uses the successful retry's page coverage, keeping an unseen update pending (later read: %s)",
(laterRead) => {
const beta = { id: 1, name: "Beta", note: "first" };
const zulu = { id: 2, name: "Zulu", note: "original" };
const patched = { ...zulu, note: "updated" };
const alpha = { id: 3, name: "Alpha", note: "created" };
const rows: TimedCaptureRow[] = [
["GET", "/api/books?limit=5", [beta, zulu]],
["GET", "/api/books?limit=1", [beta]],
["GET", "/api/books/1", beta],
["GET", "/api/books/2", zulu],
["PATCH", "/api/books/2", patched, 200, { note: "updated" }],
["POST", "/api/books", alpha, 201, { name: "Alpha", note: "created" }],
// The initial page includes Zulu. Retrying the create in name order
// replaces it with Alpha, so the passing page proves nothing of Zulu.
["GET", "/api/books?limit=2", [alpha, beta]],
];
if (laterRead) rows.push(["GET", "/api/books/2", patched]);
expect(gated(rows, "post_books")).toMatchObject({
served: true,
insert: { rule: "sorted", field: "name", direction: "ascending" },
checked_record_ordinals: [6],
});
expect(gated(rows, "patch_books_by_id")).toMatchObject({
served: laterRead,
...(laterRead ? {} : { reason: "unconfirmed" }),
checked_record_ordinals: laterRead ? [7] : [],
});
},
);
it("still learns where a list places a create after an item read confirmed it", () => {
const alpha = book(3, "Alpha", bookClock(3));
const rows: TimedCaptureRow[] = [
src/domain/analysisView/EnvironmentStateWriteSimulation.ts
+48 -34
View File
@@ -17,19 +17,22 @@ import {
checkDocumentRead,
checkItemRead,
checkListRead,
checkedLines,
collectionTarget,
entityTarget,
isAbsence,
itemTarget,
servedLine,
succeeded,
withBodies,
showsPlacement,
SimulatedState,
type CheckOutcome,
} from "./environmentStateWriteChecks.js";
import {
checkedLines,
servedLine,
type CheckedLines,
type ServedLine,
} from "./environmentStateWriteChecks.js";
} from "./environmentStateCheckedLines.js";
import {
exercisedBy,
interference,
@@ -336,10 +339,7 @@ export class EnvironmentStateWriteSimulation {
};
return item
? checkItemRead(this.#state, view, record, check)
: checkListRead(this.#state, view, record, {
...check,
all: listsAll(view, base),
});
: checkListRead(this.#state, view, record, { ...check, line: line! });
});
}
@@ -353,30 +353,42 @@ export class EnvironmentStateWriteSimulation {
const borrowers = this.#observations.get(record.ordinal);
if (view === undefined || (!borrowers && this.#resynced(view, record)))
return;
const outcome = this.#check(view, record);
let outcome = this.#check(view, record);
if (outcome === undefined) return;
const keys = [...outcome.targets, ...outcome.unheld];
if (
keys.some(
let keys = [...outcome.targets, ...outcome.unheld];
const blocked = (targets: readonly string[]) =>
targets.some(
(key) =>
this.#taint.tainted(key) ||
runningAt(this.#touchedSince, key, record.start),
)
)
return;
const exercised = exercisedBy(this.#touched, keys, borrowers);
const interferes = interference(this.#history, outcome);
);
if (blocked(keys)) return;
let exercised = exercisedBy(this.#touched, keys, borrowers);
if (borrowers !== undefined) {
if (!outcome.pass) this.#contradictOthers(exercised, record, interferes);
if (!outcome.pass)
this.#contradictOthers(
exercised,
record,
interference(this.#history, outcome),
);
return;
}
const pass =
outcome.pass ||
(view.shape === "list" && this.#retryInsert(view, record, exercised));
const unseen = pass
? this.#confirm(view, record, keys, exercised)
if (!outcome.pass && view.shape === "list") {
const retried = this.#retryInsert(view, record, exercised);
if (retried !== undefined) {
// Reordering can replace the entities on a query page. Confirmation
// and pending-write cleanup must use the successful check's coverage.
outcome = retried;
keys = [...outcome.targets, ...outcome.unheld];
if (blocked(keys)) return;
exercised = exercisedBy(this.#touched, keys, borrowers);
}
}
const interferes = interference(this.#history, outcome);
const unseen = outcome.pass
? this.#confirm(view, record, outcome, exercised)
: undefined;
if (!pass) {
if (!outcome.pass) {
const interfered = new Set([...exercised].filter(interferes));
if (interfered.size === 0)
this.#mismatch({ record, index, keys, exercised, view });
@@ -414,17 +426,18 @@ export class EnvironmentStateWriteSimulation {
#confirm(
view: StateView,
record: StateRecordFacts,
keys: readonly string[],
outcome: CheckOutcome,
exercised: ReadonlySet<string>,
): Map<string, Set<string>> {
const shown = shownWrites(this.#touched, this.#effects, {
keys,
keys: [...outcome.targets, ...outcome.unheld],
exercised,
fields: shownFields(view, this.#input.volatileOf),
});
for (const name of shown.confirmed) {
this.#drafts.get(name)!.confirmed.push(record.ordinal);
if (view.shape === "list") this.#placed.add(name);
if (showsPlacement(view, outcome, this.#inserted, name))
this.#placed.add(name);
}
return shown.unseen;
}
@@ -524,14 +537,14 @@ export class EnvironmentStateWriteSimulation {
for (const key of failed.keys) this.#taint.fail(key);
}
/** A list that fails only for where one create placed its entities passes under another of that create's insert candidates. */
/** The successful check under another create insertion candidate, including its possibly different query-page coverage. */
#retryInsert(
view: Extract<StateView, { shape: "list" }>,
record: StateRecordFacts,
exercised: ReadonlySet<string>,
): boolean {
): CheckOutcome | undefined {
const line = this.#line(view, record);
if (line === undefined || !listsAll(view, line.ordinal)) return false;
if (line === undefined || !listsAll(view, line.ordinal)) return undefined;
const members = this.#state.members(view.type, line.collection);
const key = (entity: StateEntity) =>
entityTarget(entity.type, entity.collection, entity.id);
@@ -541,18 +554,19 @@ export class EnvironmentStateWriteSimulation {
const creates = new Set(
inserted.map((entity) => this.#inserted.get(key(entity))!),
);
if (creates.size !== 1) return false;
if (creates.size !== 1) return undefined;
const [name] = creates;
const draft = this.#drafts.get(name!)!;
// An item read confirms the entity, not where a list places it.
if (!exercised.has(name!) || this.#placed.has(name!)) return false;
if (!exercised.has(name!) || this.#placed.has(name!)) return undefined;
for (const candidate of draft.candidates.slice(1)) {
this.#state.reinsert(view.type, line.collection, inserted, candidate);
if (this.#check(view, record)?.pass === true) {
const outcome = this.#check(view, record);
if (outcome?.pass === true) {
draft.plan.insert = candidate;
return true;
return outcome;
}
}
return false;
return undefined;
}
}
src/domain/analysisView/environmentStateCheckedLines.ts
+205
View File
@@ -0,0 +1,205 @@
import {
compareUnicodeCodePoints as compare,
compositeKey,
} from "../unicodeCodePointOrder.js";
import {
iterateUrlQueryParameters,
singleQueryValues,
} from "../recordedTransportUrl.js";
import {
allLines,
decodedSegment,
isParam,
listsAll,
segmentsOf,
type StateRecordFacts,
type StateToolFacts,
type StateView,
} from "./environmentStateRules.js";
import { pagesOnly } from "./environmentStatePaging.js";
/** A query as replay compares a call's: decoded values by name, in any order; undefined when it repeats a name with another value, which no call's arguments can carry. */
const queryKey = (query: string | undefined): string | undefined => {
const values = singleQueryValues(query);
if (values === undefined) return undefined;
return compositeKey(
[...values].sort(([left], [right]) => compare(left, right)),
);
};
interface CheckedLine {
readonly ordinal: number;
readonly segments: readonly string[];
readonly query: string | undefined;
}
/** A list or document view's checked lines, by query, in the tool's recorded order, and which path segments are the tool's parameters. */
export interface CheckedLines {
readonly params: ReadonlySet<number>;
readonly byQuery: ReadonlyMap<string, readonly CheckedLine[]>;
/** The lines listing their whole collection that only page it, which serving answers a call supplying no query from. */
readonly unfiltered: readonly CheckedLine[];
/** For a list with a query: the names with a learned role, and the query and whole lines serving evaluates new values on. */
readonly query?: {
readonly roles: ReadonlySet<string>;
readonly lines: readonly CheckedLine[];
};
}
/** The line a read's call is answered from, and the collection serving lists for it. */
export interface ServedLine {
readonly ordinal: number;
readonly collection: string;
/** Whether the read's query names the line; serving answers a read on any other line in that line's envelope. */
readonly named: boolean;
/** Set when serving evaluates the read's own query values on a line it did not name. */
readonly queried?: boolean;
}
type LineMatch = Pick<ServedLine, "ordinal" | "collection">;
const lineOf = (record: StateRecordFacts): CheckedLine => ({
ordinal: record.ordinal,
segments: segmentsOf(record.path),
query: record.query,
});
/** A query list's roles, and its query and whole lines in the tool's recorded order, which replay keeps among equal matches. */
const queryLines = (
view: StateView,
tool: StateToolFacts,
byOrdinal: ReadonlyMap<number, StateRecordFacts>,
): CheckedLines["query"] => {
if (view.shape !== "list" || view.query === undefined) return undefined;
const answered = new Set([...view.query.record_ordinals, ...allLines(view)]);
return {
roles: new Set(Object.keys(view.query.parameters)),
lines: tool.record_ordinals.flatMap((ordinal) => {
const record = byOrdinal.get(ordinal);
return record === undefined || !answered.has(ordinal)
? []
: [lineOf(record)];
}),
};
};
/** Each list or document view's checked lines. */
export const checkedLines = (input: {
readonly tools: readonly StateToolFacts[];
readonly records: readonly StateRecordFacts[];
readonly views: ReadonlyMap<string, StateView>;
}): Map<string, CheckedLines> => {
const byOrdinal = new Map(
input.records.map((record) => [record.ordinal, record]),
);
const tools = new Map(input.tools.map((tool) => [tool.name, tool]));
const byView = new Map<string, CheckedLines>();
for (const view of input.views.values()) {
const tool = tools.get(view.tool);
if (view.shape === "recorded" || view.shape === "item" || !tool) continue;
const checked = new Set(view.checked_record_ordinals);
const byQuery = new Map<string, CheckedLine[]>();
const unfiltered: CheckedLine[] = [];
for (const ordinal of tool.record_ordinals) {
const record = byOrdinal.get(ordinal);
const key = record === undefined ? undefined : queryKey(record.query);
if (!checked.has(ordinal) || key === undefined) continue;
const line = lineOf(record!);
byQuery.set(key, [...(byQuery.get(key) ?? []), line]);
if (
view.shape === "list" &&
listsAll(view, ordinal) &&
pagesOnly(record!.query)
)
unfiltered.push(line);
}
const params = new Set(
segmentsOf(tool.path_template).flatMap((segment, index) =>
isParam(segment) ? [index] : [],
),
);
const query = queryLines(view, tool, byOrdinal);
byView.set(view.tool, {
params,
byQuery,
unfiltered,
...(query === undefined ? {} : { query }),
});
}
return byView;
};
/**
* The checked line serving answers a read's call from, ranked as replay ranks lines: every
* parameter the read's value verbatim or once percent-decoded and the query the read's, then the
* most verbatim parameters, then the most query values the read shares, then recorded order. A
* read with no query falls back, as serving does, on a whole list's lines that only page it.
*/
export const servedLine = (
lines: CheckedLines | undefined,
record: StateRecordFacts,
): ServedLine | undefined => {
const key = queryKey(record.query);
if (lines === undefined || key === undefined) return undefined;
const names = [...iterateUrlQueryParameters(record.query ?? "")].map(
([name]) => name,
);
const named = bestLine(lines, lines.byQuery.get(key) ?? [], record);
if (named !== undefined) return { ...named, named: true };
const unfiltered =
names.length === 0 ? bestLine(lines, lines.unfiltered, record) : undefined;
if (unfiltered !== undefined) return { ...unfiltered, named: false };
if (lines.query === undefined) return undefined;
// As serving does, a read whose every query name has a role answers on a query or whole line, on its own values.
const { roles } = lines.query;
if (!names.every((name) => roles.has(name))) return undefined;
const line = bestLine(lines, lines.query.lines, record);
return line === undefined
? undefined
: { ...line, named: false, queried: names.length > 0 };
};
/** How many of the read's query names a line recorded with the read's value, as replay counts them. */
const queryMatches = (
read: ReadonlyMap<string, string>,
query: string | undefined,
): number => {
const matched = new Set<string>();
for (const [name, value] of iterateUrlQueryParameters(query ?? ""))
if (read.get(name) === value) matched.add(name);
return matched.size;
};
const bestLine = (
lines: CheckedLines,
candidates: readonly CheckedLine[],
record: StateRecordFacts,
): LineMatch | undefined => {
const read = segmentsOf(record.path);
const values = new Map(iterateUrlQueryParameters(record.query ?? ""));
let best: { line: LineMatch; verbatim: number; query: number } | undefined;
for (const { ordinal, segments, query } of candidates) {
if (segments.length !== read.length) continue;
let verbatim = 0;
const matches = segments.every((segment, index) => {
if (segment === read[index]) {
if (lines.params.has(index)) verbatim += 1;
return true;
}
return lines.params.has(index) && decodedSegment(segment) === read[index];
});
if (!matches) continue;
const matched = queryMatches(values, query);
if (
best === undefined ||
verbatim > best.verbatim ||
(verbatim === best.verbatim && matched > best.query)
)
best = {
line: { ordinal, collection: `/${segments.join("/")}` },
verbatim,
query: matched,
};
}
return best?.line;
};
src/domain/analysisView/environmentStateGate.ts
+127 -19
View File
@@ -19,6 +19,16 @@ import {
type StateToolFacts,
type StateView,
} from "./environmentStateRules.js";
import {
evaluateQuery,
learnQuery,
lineChoices,
queryTotalCount,
renderQueryList,
type ListQuery,
type QueryLine,
} from "./environmentStateListQuery.js";
import { singleQueryValues } from "../recordedTransportUrl.js";
import { startsAtBeginning } from "./environmentStatePaging.js";
import { ENTRY_BYTES, textBytes } from "./webNetworkRouteTemplates.js";
@@ -182,8 +192,8 @@ const listFacts = (
record.ordinal,
counted.filter((name) => parent[name] === items.length),
);
// A later (or ambiguously positioned) page cannot establish whole membership,
// even when its collection is empty. Totals and continuation also disprove it.
// Only a line that pages at most can hold its whole collection; a total that differs from
// the number listed, or a next page, shows it is partial.
if (
!startsAtBeginning(record.query) ||
counted.some((name) => countOf(parent[name]) !== items.length) ||
@@ -196,16 +206,10 @@ const listFacts = (
.map((entity) => entity.id);
if (same(ids, expected)) whole.add(record.ordinal);
});
// A whole line's counters count the collection, which a partial page's total does not, so the whole lines decide.
const lines = [...counting].filter(
([ordinal]) => whole.size === 0 || whole.has(ordinal),
);
const counters = lines.reduce<string[] | undefined>(
(kept, [, names]) =>
kept === undefined ? names : kept.filter((name) => names.includes(name)),
undefined,
);
return { whole, counters: (counters ?? []).sort(compare) };
// Counter presence is per envelope. Keep every observed page counter;
// renderStateList updates only one the selected base actually counted.
const counters = [...new Set([...counting.values()].flat())].sort(compare);
return { whole, counters };
};
const nameTokens = (name: string): string[] =>
@@ -259,7 +263,76 @@ const continues = (name: string, value: JsonValue | undefined): boolean =>
value !== false &&
value !== "";
/** Whether one latest list recording renders from its own envelope and the listed entities' state. */
/** The query that lines which are not whole follow, with the item totals that count its matches before paging. */
const listQuery = (
context: StateGateContext,
plan: StatePlan,
whole: ReadonlySet<number>,
): ListQuery | undefined => {
const field = context.idField(plan.type);
const lines: QueryLine[] = [];
const counts = new Map<number, [string, number][]>();
for (const record of plan.latest) {
const values = whole.has(record.ordinal)
? undefined
: singleQueryValues(record.query);
if (values === undefined || values.size === 0) continue;
context.withBody(record.ordinal, (body) => {
const items =
body === undefined ? undefined : listItems(body, plan.pointer);
if (body === undefined || items === undefined) return;
const choices = lineChoices(values, items);
context.retain(ENTRY_BYTES * (items.length + choices.length * 4 + 2));
lines.push({
ordinal: record.ordinal,
values,
choices,
listed: items.map((item) => scalarText(item[field]) ?? ""),
entities: context.inCollection(
plan.type,
collectionPathOf(record, false),
),
});
const parent = plan.pointer === "" || !isObject(body) ? {} : body;
counts.set(
record.ordinal,
Object.keys(parent).flatMap((name): [string, number][] => {
const count = isItemTotal(name)
? queryTotalCount(parent[name])
: undefined;
return count === undefined ? [] : [[name, count]];
}),
);
});
}
const learned = learnQuery(lines);
if (learned === undefined) return undefined;
// Totals count every match on each line the query answers; a `number`-of-items member counts the page.
const totals = new Set<string>();
const contradicted = new Set<string>();
const answered = new Set(learned.record_ordinals);
for (const line of lines) {
if (!answered.has(line.ordinal)) continue;
const { total } = evaluateQuery(
learned.parameters,
line.values,
line.entities,
)!;
// Absence says nothing about another envelope's counter. Only a present
// count that disagrees invalidates the shared total interpretation.
for (const [name, count] of counts.get(line.ordinal) ?? []) {
if (nameTokens(name).includes("number")) continue;
if (count === total) totals.add(name);
else contradicted.add(name);
}
}
return {
...learned,
totals: [...totals].filter((name) => !contradicted.has(name)).sort(compare),
};
};
/** Whether one latest list recording renders from its own envelope and the state: the whole collection, the query's matches, or the listed entities. */
const listReproduces = (
context: StateGateContext,
plan: StatePlan,
@@ -269,10 +342,12 @@ const listReproduces = (
readonly pointer: string;
readonly counters: string[];
readonly whole: ReadonlySet<number>;
readonly query: ListQuery | undefined;
},
record: StateRecordFacts,
): boolean => {
const field = context.idField(plan.type);
const collection = collectionPathOf(record, false);
return context.withBody(record.ordinal, (body) => {
const items =
body === undefined ? undefined : listItems(body, plan.pointer);
@@ -280,16 +355,24 @@ const listReproduces = (
const observed = new Map(
items.map((item) => [scalarText(item[field]) ?? "", item]),
);
const computed = view.query?.record_ordinals.includes(record.ordinal)
? evaluateQuery(
view.query.parameters,
singleQueryValues(record.query) ?? new Map(),
context.inCollection(plan.type, collection),
)
: undefined;
const members = view.whole.has(record.ordinal)
? context.inCollection(plan.type, collectionPathOf(record, false))
: items.flatMap((item) => {
? context.inCollection(plan.type, collection)
: (computed?.page ??
items.flatMap((item) => {
const entity = context.entity(
plan.type,
collectionPathOf(record, false),
collection,
scalarText(item[field]) ?? "",
);
return entity === undefined ? [] : [entity];
});
}));
const rendered = members.map((entity) =>
renderStateEntity(
entity,
@@ -298,7 +381,17 @@ const listReproduces = (
observed.get(entity.id),
),
);
return same(renderStateList(view, body, rendered), body);
const answer =
view.query === undefined ||
(computed === undefined && !view.whole.has(record.ordinal))
? renderStateList(view, body, rendered)
: renderQueryList(
{ ...view, query: view.query },
body,
rendered,
computed?.total ?? rendered.length,
);
return same(answer, body);
});
};
@@ -310,13 +403,27 @@ export const checkListPlan = (
const union = new FieldUnion();
const { whole, counters } = listFacts(context, plan, union);
context.retain(union.bytes);
const view = {
const base = {
tool: plan.tool.name,
fields: union.fields,
pointer: plan.pointer,
counters,
whole,
};
let query = listQuery(context, plan, whole);
// A line the query explains but whose envelope or fields the state does not reproduce keeps its recorded members.
if (query !== undefined) {
const failing = plan.latest.filter(
(record) =>
query!.record_ordinals.includes(record.ordinal) &&
!listReproduces(context, plan, { ...base, query }, record),
);
const kept = query.record_ordinals.filter(
(ordinal) => !failing.some((record) => record.ordinal === ordinal),
);
query = kept.length === 0 ? undefined : { ...query, record_ordinals: kept };
}
const view = { ...base, query };
const all = plan.latest.every((record) => whole.has(record.ordinal));
if (
!plan.latest.every((record) => listReproduces(context, plan, view, record))
@@ -338,6 +445,7 @@ export const checkListPlan = (
.filter((ordinal) => whole.has(ordinal)),
}),
counters,
...(query === undefined ? {} : { query }),
checked_record_ordinals: plan.latest.map((record) => record.ordinal),
};
};
src/domain/analysisView/environmentStateListQuery.test.ts
+344
View File
@@ -0,0 +1,344 @@
import { describe, expect, it } from "vitest";
import {
evaluateQuery,
explainLine,
learnQuery,
lineChoices,
queryTotalCount,
renderQueryList,
type QueryLine,
} from "./environmentStateListQuery.js";
import { singleQueryValues } from "../recordedTransportUrl.js";
import type { JsonObject, StateEntity } from "./environmentStateRules.js";
const entity = (fields: JsonObject): StateEntity => ({
type: "/api/books",
collection: "/api/books",
id: String(fields["id"]),
fields,
view_fields: [],
record_ordinals: [],
});
const BOOKS = [
{
id: 1,
name: "Tide Tables",
shelf: "a",
tags: ["sea"],
url: "https://x/tide",
},
{ id: 2, name: "Lamp Oil", shelf: "b", tags: [{ id: "t1", name: "oil" }] },
{ id: 3, name: "tidewater", shelf: "a", tags: [], url: "https://x/other" },
{
id: 4,
name: "Kelp",
shelf: null,
tags: ["sea"],
url: "https://x/kelp-tide",
},
].map((book) => entity({ ...book, active: true }));
/** A line as the gate reads it: its query, and the books it listed in order. */
const line = (
query: string,
listed: readonly number[],
ordinal = 0,
): QueryLine => {
const values = singleQueryValues(query)!;
const items = listed.map(
(id) => BOOKS.find((book) => book.id === String(id))!.fields,
);
return {
ordinal,
values,
listed: listed.map(String),
choices: lineChoices(values, items),
entities: BOOKS,
};
};
const ids = (query: string, parameters: Parameters<typeof evaluateQuery>[0]) =>
evaluateQuery(parameters, singleQueryValues(query)!, BOOKS)?.page.map(
(book) => book.id,
);
describe("explaining a recorded line", () => {
it("finds a filter on the field every listed item holds, scalar or in an array", () => {
expect(explainLine(line("?shelf=a", [1, 3]))).toEqual({
shelf: { role: "filter", field: "shelf" },
});
expect(explainLine(line("?tag=sea", [1, 4]))).toEqual({
tag: { role: "filter", field: "tags" },
});
expect(explainLine(line("?tag=t1", [2]))).toEqual({
tag: { role: "filter", field: "tags" },
});
});
it("searches the fields the term matched, ignoring case, and leaves out URLs that merely repeat it", () => {
expect(explainLine(line("?q=TIDE", [1, 3]))).toEqual({
q: { role: "search", fields: ["name"] },
});
});
it("sorts by the field a value names, with a direction parameter, prefix or suffix", () => {
expect(
explainLine(line("?orderBy=name&orderDirection=desc", [3, 1, 2, 4])),
).toEqual({
orderBy: { role: "sort" },
orderDirection: { role: "direction" },
});
expect(ids("?sort=-name", { sort: { role: "sort" } })).toEqual([
"3",
"1",
"2",
"4",
]);
expect(ids("?sort=NAME-asc", { sort: { role: "sort" } })).toEqual([
"4",
"2",
"1",
"3",
]);
// Null values sort last ascending.
expect(ids("?sort=shelf", { sort: { role: "sort" } })).toEqual([
"1",
"3",
"2",
"4",
]);
});
it("pages by limit and offset, or by page number from the first page it explains", () => {
expect(explainLine(line("?limit=2&offset=1", [2, 3]))).toEqual({
limit: { role: "limit" },
offset: { role: "offset" },
});
expect(explainLine(line("?page=2&per_page=2", [3, 4]))).toEqual({
page: { role: "page", first: 1 },
per_page: { role: "limit" },
});
expect(ids("?per_page=-1", { per_page: { role: "limit" } })).toEqual([
"1",
"2",
"3",
"4",
]);
});
it("takes a parameter that changes nothing as inert, for its recorded values only", () => {
const explained = explainLine(line("?includeDeleted=true", [1, 2, 3, 4]));
expect(explained).toEqual({
includeDeleted: { role: "inert", values: ["true"] },
});
expect(ids("?includeDeleted=false", explained!)).toBeUndefined();
});
it("leaves ambiguous page conventions unknown rather than preferring the first candidate", () => {
expect(explainLine(line("?page=100&limit=1", []))).toBeUndefined();
expect(explainLine(line("?page=0&limit=2", [1, 2]))).toEqual({
page: { role: "page", first: 0 },
limit: { role: "limit" },
});
expect(explainLine(line("?limit=2&page=2", [3, 4]))).toEqual({
limit: { role: "limit" },
page: { role: "page", first: 1 },
});
});
it.each(["title:Tide", "#sea", '"Tide Tables"', "Tide OR Oil", "+Tide"])(
"does not infer a literal substring model for search syntax: %s",
(term) => {
expect(
explainLine(line(`?q=${encodeURIComponent(term)}`, [1, 2, 3, 4])),
).toBeUndefined();
expect(
ids(`?q=${encodeURIComponent(term)}`, {
q: { role: "search", fields: ["name"] },
}),
).toBeUndefined();
},
);
it("uses safe numeric precision rather than an arbitrary digit-count limit", () => {
expect(ids("?offset=1000000000", { offset: { role: "offset" } })).toEqual(
[],
);
expect(
ids("?limit=9007199254740991", { limit: { role: "limit" } }),
).toEqual(["1", "2", "3", "4"]);
expect(
ids("?offset=9007199254740992", { offset: { role: "offset" } }),
).toBeUndefined();
expect(explainLine(line("?offset=unknown", [1, 2, 3, 4]))).toBeUndefined();
});
it("keeps a filter that excludes nothing inert, as a flag", () => {
expect(explainLine(line("?active=true", [1, 2, 3, 4]))).toEqual({
active: { role: "inert", values: ["true"] },
});
});
it("explains nothing that the collection cannot reproduce", () => {
expect(explainLine(line("?shelf=a", [3, 1]))).toBeUndefined();
});
});
describe("learning a list's query", () => {
it("joins lines into one role per parameter and answers the lines it reproduces", () => {
const learned = learnQuery([
line("?shelf=a", [1, 3], 10),
line("?shelf=b", [2], 11),
line("?q=kelp", [4], 12),
line("?q=oil", [2], 13),
]);
expect(learned).toEqual({
parameters: {
shelf: { role: "filter", field: "shelf" },
q: { role: "search", fields: ["name"] },
},
record_ordinals: [10, 11, 12, 13],
});
// A value no line recorded is evaluated the same way.
expect(ids("?q=TIDE&shelf=a", learned!.parameters)).toEqual(["1", "3"]);
});
it("refuses a value a parameter cannot apply, or a parameter it never learned", () => {
const parameters = { limit: { role: "limit" } } as const;
expect(ids("?limit=two", parameters)).toBeUndefined();
expect(ids("?shelf=a", parameters)).toBeUndefined();
expect(singleQueryValues("?a=1&a=2")).toBeUndefined();
});
it("normalizes identical repeats as serving does, retaining conflicting values as unknown", () => {
expect(singleQueryValues("?shelf=a&shelf=%61")).toEqual(
new Map([["shelf", "a"]]),
);
expect(singleQueryValues("?shelf=a&shelf=b")).toBeUndefined();
});
it("gives no two parameters one paging role, which would answer differently in serving's name order", () => {
const both = { size: { role: "limit" }, limit: { role: "limit" } } as const;
expect(ids("?size=2&limit=1", both)).toBeUndefined();
const learned = learnQuery([line("?size=2&limit=1", [1], 40)]);
expect(learned?.parameters).toEqual({
size: { role: "inert", values: ["2"] },
limit: { role: "limit" },
});
// Serving supplies a call's values in name order.
expect(ids("?limit=1&size=2", learned!.parameters)).toEqual(["1"]);
});
});
describe("queries the state can always answer", () => {
it("models only reproducible total representations, without losing their source type", () => {
for (const value of [
0,
3,
"0",
"3",
Number.MAX_SAFE_INTEGER,
String(Number.MAX_SAFE_INTEGER),
])
expect(queryTotalCount(value)).toBe(Number(value));
for (const value of [
null,
"unknown",
"03",
"3.0",
" 3",
"-1",
-1,
1.5,
"9007199254740992",
])
expect(queryTotalCount(value)).toBeUndefined();
const view = {
pointer: "/items",
counters: [],
query: { totals: ["total"] },
};
expect(renderQueryList(view, { items: [], total: "3" }, [], 2)).toEqual({
items: [],
total: "2",
});
expect(renderQueryList(view, { items: [], total: "03" }, [], 2)).toEqual({
items: [],
total: "03",
});
});
it("projects query totals only into envelopes that actually contain a known count", () => {
const view = {
pointer: "/items",
counters: ["numberOfElements"],
query: { totals: ["total"] },
};
const selected = [{ id: 1 }];
expect(
renderQueryList(view, { items: [], numberOfElements: 0 }, selected, 7),
).toEqual({ items: selected, numberOfElements: 1 });
expect(
renderQueryList(view, { items: [], total: null }, selected, 7),
).toEqual({ items: selected, total: null });
expect(renderQueryList(view, { items: [], total: 3 }, selected, 7)).toEqual(
{ items: selected, total: 7 },
);
});
it("sorts case-only ties in state order, ignoring case", () => {
const named = [
entity({ id: 10, name: "alpha" }),
entity({ id: 11, name: "Alpha" }),
];
const order = (entities: readonly StateEntity[]) =>
evaluateQuery(
{ sort: { role: "sort" } },
singleQueryValues("?sort=name")!,
entities,
)?.page.map((entry) => entry.id);
expect(order(named)).toEqual(["10", "11"]);
expect(order([...named].reverse())).toEqual(["11", "10"]);
});
it("answers a sort of an emptied collection with no entities rather than giving up", () => {
expect(
evaluateQuery(
{ sort: { role: "sort" } },
singleQueryValues("?sort=-id")!,
[],
),
).toEqual({ page: [], total: 0, positional: "all" });
});
it("learns no search from a term that matched no field, keeping it inert", () => {
const learned = learnQuery([line("?q=", [1, 2, 3, 4], 20)]);
expect(learned?.parameters["q"]).toEqual({ role: "inert", values: [""] });
});
it("explains no line with more query names than it can search, without exhausting the stack", () => {
const values = new Map(
Array.from({ length: 5_000 }, (_, index) => [`p${String(index)}`, "x"]),
);
const choices = lineChoices(
values,
BOOKS.map((entry) => entry.fields),
);
expect(choices).toEqual([]);
expect(
explainLine({ ordinal: 0, values, listed: [], choices, entities: BOOKS }),
).toBeUndefined();
});
it("keeps a parameter named like an object's prototype as its own role", () => {
const learned = learnQuery([line("?__proto__=x&shelf=a", [1, 3], 30)]);
expect(Object.keys(learned!.parameters)).toContain("__proto__");
const round = JSON.parse(JSON.stringify(learned!.parameters)) as object;
expect(Object.getOwnPropertyDescriptor(round, "__proto__")?.value).toEqual({
role: "inert",
values: ["x"],
});
});
});
src/domain/analysisView/environmentStateListQuery.ts
+607
View File
@@ -0,0 +1,607 @@
import type { JsonValue } from "../jsonValue.js";
import { compareUnicodeCodePoints as compare } from "../unicodeCodePointOrder.js";
import {
isObject,
pointerTo,
renderStateList,
replaceAt,
scalarText,
type JsonObject,
type StateEntity,
type StateListView,
} from "./environmentStateRules.js";
/** What one list query parameter does to a list, learned from the recorded lines it explains. */
export type QueryParameter =
| { readonly role: "filter"; readonly field: string }
| { readonly role: "search"; readonly fields: string[] }
| { readonly role: "sort" }
| { readonly role: "direction" }
| { readonly role: "limit" }
| { readonly role: "offset" }
| { readonly role: "page"; readonly first: number }
| { readonly role: "inert"; readonly values: string[] };
/** A list view's query model: each parameter's role, the checked lines it answers, and the members that count matches before paging. */
export interface ListQuery {
readonly parameters: Record<string, QueryParameter>;
readonly record_ordinals: number[];
readonly totals: string[];
}
/** One recorded line a query may explain: its query values, the identities it listed in order, the roles each parameter may play and the collection's entities. */
export interface QueryLine {
readonly ordinal: number;
readonly values: ReadonlyMap<string, string>;
readonly listed: readonly string[];
readonly choices: readonly (readonly [string, readonly QueryParameter[]])[];
readonly entities: readonly StateEntity[];
}
const SORT = new Set([
"order",
"orderby",
"order_by",
"ordering",
"sort",
"sortby",
"sort_by",
]);
const DIRECTION = new Set([
"dir",
"direction",
"order",
"orderdirection",
"order_direction",
"sortdir",
"sort_dir",
"sortorder",
"sort_order",
]);
const LIMIT = new Set([
"count",
"itemsperpage",
"limit",
"page_size",
"pagesize",
"per_page",
"perpage",
"size",
"take",
]);
const OFFSET = new Set(["_start", "offset", "skip", "start"]);
const SEARCH = new Set([
"keyword",
"keywords",
"q",
"query",
"search",
"search_term",
"searchterm",
"term",
]);
const DIRECTIONS = new Set(["asc", "desc"]);
const IDENTIFYING = ["id", "slug", "name"];
/** At most this many role combinations are tried for one line. */
const MAX_COMBINATIONS = 256;
/** A line with more query parameters than this is left unexplained, which bounds the search's depth and work. */
const MAX_QUERY_NAMES = 12;
/** Only plain substring terms are modeled; syntax-like operators require recorded replay. */
const plainSearch = (value: string): boolean =>
!/[:#"()[\]{}<>|*?\\]/u.test(value) &&
!/(?:^|\s)(?:AND|OR|NOT)(?:\s|$)/u.test(value) &&
!/(?:^|\s)[+-]\S/u.test(value);
const textOf = (value: JsonValue | undefined): string | undefined =>
typeof value === "boolean" ? String(value) : scalarText(value);
/** Whether a field holds a value: equal as text, or in an array directly or as an object's id, slug or name. */
const holds = (field: JsonValue | undefined, value: string): boolean => {
const names = (item: JsonValue | undefined) =>
isObject(item) && IDENTIFYING.some((key) => textOf(item[key]) === value);
return Array.isArray(field)
? field.some((item) => textOf(item) === value || names(item))
: textOf(field) === value || names(field);
};
const contains = (
entity: StateEntity,
fields: readonly string[],
term: string,
): boolean => {
const needle = term.toLowerCase();
return (
needle === "" ||
fields.some((field) => {
const value = entity.fields[field];
return typeof value === "string" && value.toLowerCase().includes(needle);
})
);
};
const normalized = (name: string): string =>
name.toLowerCase().replace(/[^a-z0-9]/gu, "");
/** The field a sort value names and its direction: `name`, `-name`, `name:desc`, `name-ASC`; names match ignoring case and separators. */
const sortKey = (
value: string,
names: readonly string[],
): { readonly field: string; readonly descending: boolean } | undefined => {
const field = (spelled: string) => {
if (names.includes(spelled)) return spelled;
const found = names.filter(
(name) => normalized(name) === normalized(spelled),
);
return found.length === 1 ? found[0] : undefined;
};
if (names.includes(value)) return { field: value, descending: false };
// A leading `-` descends; matching ignores separators, so it is read before the name is matched loosely.
if (value.startsWith("-")) {
const named = field(value.slice(1));
return named === undefined ? undefined : { field: named, descending: true };
}
const whole = field(value);
if (whole !== undefined) return { field: whole, descending: false };
const suffixed = /^(.+?)[\s:_-](asc|desc)$/iu.exec(value);
const named = suffixed === null ? undefined : field(suffixed[1]!);
return named === undefined
? undefined
: { field: named, descending: suffixed![2]!.toLowerCase() === "desc" };
};
/** Ascending order: numbers by value, text ignoring case, missing and null values last. */
const ascending = (
left: JsonValue | undefined,
right: JsonValue | undefined,
): number => {
const absent = (value: JsonValue | undefined) =>
value === undefined || value === null;
if (absent(left) || absent(right))
return absent(left) ? (absent(right) ? 0 : 1) : -1;
if (typeof left === "number" && typeof right === "number")
return left - right;
const a = textOf(left) ?? JSON.stringify(left);
const b = textOf(right) ?? JSON.stringify(right);
return compare(a.toLowerCase(), b.toLowerCase());
};
const integer = (value: string): number | undefined => {
const number = Number(value);
return /^-?\d+$/u.test(value) && Number.isSafeInteger(number)
? number
: undefined;
};
/** What a call's values ask of the collection, or undefined when one has no role or cannot apply. */
interface QueryCall {
readonly selections: ((entity: StateEntity) => boolean)[];
sort?: string;
direction?: string;
limit?: number | undefined;
offset: number;
page?: { readonly index: number; readonly first: number };
}
/** Roles that only sort or page an answer; one value sets each for a call, as two parameters in one would answer by whichever applied last. */
const PAGING_ROLES = new Set(["sort", "direction", "limit", "offset", "page"]);
const readCall = (
parameters: Readonly<Record<string, QueryParameter>>,
values: ReadonlyMap<string, string>,
): QueryCall | undefined => {
const call: QueryCall = { selections: [], offset: 0 };
const taken = new Set<string>();
for (const [name, value] of values) {
const parameter = Object.hasOwn(parameters, name)
? parameters[name]
: undefined;
const number = integer(value);
if (parameter === undefined || taken.has(parameter.role)) return undefined;
if (PAGING_ROLES.has(parameter.role)) taken.add(parameter.role);
if (parameter.role === "filter")
call.selections.push((entity) =>
holds(entity.fields[parameter.field], value),
);
else if (parameter.role === "search") {
if (!plainSearch(value)) return undefined;
call.selections.push((entity) =>
contains(entity, parameter.fields, value),
);
} else if (parameter.role === "sort") call.sort = value;
else if (parameter.role === "direction") {
if (!DIRECTIONS.has(value.toLowerCase())) return undefined;
call.direction = value.toLowerCase();
} else if (parameter.role === "inert") {
if (!parameter.values.includes(value)) return undefined;
} else if (number === undefined) return undefined;
else if (parameter.role === "limit")
call.limit = number > 0 ? number : undefined;
else if (parameter.role === "offset" && number >= 0) call.offset = number;
else if (parameter.role === "page" && number >= parameter.first)
call.page = { index: number, first: parameter.first };
else return undefined;
}
return call;
};
/** Entities in the order a sort value names, ties in state order, and those whose place state order decides: all without a sort, else those tied with a neighbour; undefined when it names no field. */
const sorted = (
matched: readonly StateEntity[],
entities: readonly StateEntity[],
call: QueryCall,
):
| {
readonly entities: StateEntity[];
readonly positional: readonly StateEntity[] | "all";
}
| undefined => {
// Nothing to order: an emptied collection answers no entities rather than a recording.
if (call.sort === undefined || matched.length === 0)
return { entities: [...matched], positional: "all" };
const names = [
...new Set(entities.flatMap((entity) => Object.keys(entity.fields))),
];
const key = sortKey(call.sort, names);
if (key === undefined) return undefined;
const sign = (
call.direction === undefined ? key.descending : call.direction === "desc"
)
? -1
: 1;
const ordered = matched
.map((entity, index) => ({ entity, index }))
.sort(
(left, right) =>
sign *
ascending(
left.entity.fields[key.field],
right.entity.fields[key.field],
) || left.index - right.index,
)
.map(({ entity }) => entity);
const tied = (
left: StateEntity | undefined,
right: StateEntity | undefined,
) =>
left !== undefined &&
right !== undefined &&
ascending(left.fields[key.field], right.fields[key.field]) === 0;
return {
entities: ordered,
positional: ordered.filter(
(entity, index) =>
tied(ordered[index - 1], entity) || tied(entity, ordered[index + 1]),
),
};
};
/** The entities a query selects, in its order, and how many matched before paging; undefined when a value cannot be applied. */
export const evaluateQuery = (
parameters: Readonly<Record<string, QueryParameter>>,
values: ReadonlyMap<string, string>,
entities: readonly StateEntity[],
):
| {
readonly page: StateEntity[];
readonly total: number;
/** The matches whose place in the answer state order decides: all without a sort, else those the sort leaves tied. */
readonly positional: readonly StateEntity[] | "all";
}
| undefined => {
const call = readCall(parameters, values);
if (call === undefined) return undefined;
const selection = sorted(
entities.filter((entity) =>
call.selections.every((selects) => selects(entity)),
),
entities,
call,
);
if (selection === undefined) return undefined;
const matched = selection.entities;
let { offset } = call;
const { limit, page } = call;
if (page !== undefined && page.index > page.first) {
if (limit === undefined) return undefined;
offset += (page.index - page.first) * limit;
}
return {
page: matched.slice(
offset,
limit === undefined ? undefined : offset + limit,
),
total: matched.length,
positional: selection.positional,
};
};
/** A query's answer and whether it shows the whole collection: every parameter only sorts or pages, and the page holds every match. */
export const answerQuery = (
query: Pick<ListQuery, "parameters">,
values: ReadonlyMap<string, string>,
entities: readonly StateEntity[],
):
| {
readonly page: StateEntity[];
readonly total: number;
readonly positional: readonly StateEntity[] | "all";
readonly whole: boolean;
}
| undefined => {
const result = evaluateQuery(query.parameters, values, entities);
return result === undefined
? undefined
: {
...result,
whole:
result.page.length === result.total &&
// Every name has its own role, or evaluating the call would have refused it.
[...values.keys()].every((name) =>
PAGING_ROLES.has(query.parameters[name]!.role),
),
};
};
/** Fields every listed item holds the value in, those named like the parameter first. */
const filterFields = (
items: readonly JsonObject[],
name: string,
value: string,
): string[] =>
items.length === 0
? []
: Object.keys(items[0]!)
.filter((field) => items.every((item) => holds(item[field], value)))
.sort(
(left, right) =>
Number(normalized(right) === normalized(name)) -
Number(normalized(left) === normalized(name)),
)
.slice(0, 4);
const isUrl = (value: string): boolean =>
/^[a-z][a-z0-9+.-]*:\/\//iu.test(value);
/**
* The string fields the term matched in the listed items: for each item the fields that contain it, ignoring case,
* other than URLs, which a name or slug often repeats; an item the term matched only in URLs adds those.
*/
const searchFields = (items: readonly JsonObject[], term: string): string[] => {
const needle = term.toLowerCase();
if (needle === "") return [];
const fields = new Set<string>();
for (const item of items) {
const matching = Object.keys(item).filter((field) => {
const value = item[field];
return typeof value === "string" && value.toLowerCase().includes(needle);
});
const plain = matching.filter((field) => !isUrl(item[field] as string));
for (const field of plain.length > 0 ? plain : matching) fields.add(field);
}
return [...fields].sort(compare);
};
/** The roles a parameter may play on a line that listed `items`, most specific first; inert last. */
const choicesFor = (
name: string,
value: string,
items: readonly JsonObject[],
): QueryParameter[] => {
const key = name.toLowerCase();
const choices: QueryParameter[] = [];
const number = integer(value);
if (
((LIMIT.has(key) || OFFSET.has(key) || key === "page") &&
(number === undefined ||
((OFFSET.has(key) || key === "page") && number < 0))) ||
(SEARCH.has(key) && !plainSearch(value))
)
return [];
if (DIRECTION.has(key) && DIRECTIONS.has(value.toLowerCase()))
choices.push({ role: "direction" });
else if (SORT.has(key)) choices.push({ role: "sort" });
else if (LIMIT.has(key)) choices.push({ role: "limit" });
else if (OFFSET.has(key)) choices.push({ role: "offset" });
else if (key === "page")
choices.push({ role: "page", first: 1 }, { role: "page", first: 0 });
else if (SEARCH.has(key)) {
// An empty or unmatched term shows no searchable field, so it stays inert until a term matches one.
const fields = searchFields(items, value);
if (fields.length > 0) choices.push({ role: "search", fields });
}
if (choices.length === 0 || SEARCH.has(key))
for (const field of filterFields(items, name, value))
choices.push({ role: "filter", field });
choices.push({ role: "inert", values: [value] });
return choices;
};
/** The roles each of a line's parameters may play, read from the items it listed. */
export const lineChoices = (
values: ReadonlyMap<string, string>,
items: readonly JsonObject[],
): [string, QueryParameter[]][] =>
values.size > MAX_QUERY_NAMES
? []
: [...values].map(([name, value]) => [
name,
choicesFor(name, value, items),
]);
const listsAsRecorded = (
line: QueryLine,
parameters: Readonly<Record<string, QueryParameter>>,
): boolean => {
const result = evaluateQuery(parameters, line.values, line.entities);
return (
result !== undefined &&
result.page.length === line.listed.length &&
result.page.every((entity, index) => entity.id === line.listed[index])
);
};
/** A filter that keeps every entity of the line's collection shows nothing a flag would not: it stays inert for that value. */
const withoutIdleFilters = (
line: QueryLine,
chosen: Record<string, QueryParameter>,
): Record<string, QueryParameter> =>
Object.fromEntries(
Object.entries(chosen).map(([name, role]) => {
const value = line.values.get(name)!;
return role.role === "filter" &&
line.entities.every((entity) => holds(entity.fields[role.field], value))
? [name, { role: "inert", values: [value] }]
: [name, role];
}),
);
/** The first combination of roles, most specific first, that lists exactly what the line recorded. */
export const explainLine = (
line: QueryLine,
): Record<string, QueryParameter> | undefined => {
const options = line.choices;
let tried = 0;
const search = (
index: number,
chosen: Record<string, QueryParameter>,
): Record<string, QueryParameter> | undefined => {
if (index === options.length) {
tried += 1;
return listsAsRecorded(line, chosen)
? withoutIdleFilters(line, chosen)
: undefined;
}
const [name, choices] = options[index]!;
for (const choice of choices) {
if (tried >= MAX_COMBINATIONS) return undefined;
const found = search(index + 1, { ...chosen, [name]: choice });
if (found !== undefined) return found;
}
return undefined;
};
const found = search(0, {});
if (found === undefined) return undefined;
// An empty/out-of-range page can fit both conventions. Do not turn a
// preferred candidate's position into evidence of a zero-/one-based API.
for (const [name, role] of Object.entries(found)) {
if (role.role !== "page") continue;
for (const candidate of line.choices.find(([key]) => key === name)?.[1] ??
[]) {
if (candidate.role !== "page" || candidate.first === role.first) continue;
if (tried >= MAX_COMBINATIONS) return undefined;
tried++;
if (listsAsRecorded(line, { ...found, [name]: candidate }))
return undefined;
}
}
return found;
};
/** One role per parameter across the lines that explained it: inert values gather, search fields unite, and any other disagreement drops the parameter. */
const mergeRoles = (
explained: readonly Record<string, QueryParameter>[],
): Record<string, QueryParameter> => {
const roles = new Map<string, QueryParameter[]>();
for (const parameters of explained)
for (const [name, role] of Object.entries(parameters))
roles.set(name, [...(roles.get(name) ?? []), role]);
// Entries become own members even for names such as `__proto__`, which assignment would treat as the prototype.
const merged: [string, QueryParameter][] = [];
for (const [name, seen] of roles) {
const active = seen.filter((role) => role.role !== "inert");
if (active.length === 0) {
merged.push([
name,
{
role: "inert",
values: [
...new Set(
seen.flatMap((role) =>
role.role === "inert" ? role.values : [],
),
),
].sort(compare),
},
]);
continue;
}
const [first] = active;
if (first!.role === "search") {
if (active.every((role) => role.role === "search"))
merged.push([
name,
{
role: "search",
fields: [
...new Set(
active.flatMap((role) =>
role.role === "search" ? role.fields : [],
),
),
].sort(compare),
},
]);
continue;
}
if (active.every((role) => JSON.stringify(role) === JSON.stringify(first)))
merged.push([name, first!]);
}
return Object.fromEntries(merged);
};
/** The query model that lists exactly what each explained line recorded, or undefined when none does. */
export const learnQuery = (
lines: readonly QueryLine[],
): Pick<ListQuery, "parameters" | "record_ordinals"> | undefined => {
const explained = lines.flatMap((line) => {
const parameters = explainLine(line);
return parameters === undefined ? [] : [parameters];
});
if (explained.length === 0) return undefined;
const parameters = mergeRoles(explained);
const record_ordinals = lines
.filter((line) => listsAsRecorded(line, parameters))
.map((line) => line.ordinal);
return record_ordinals.length === 0
? undefined
: { parameters, record_ordinals };
};
/** A count whose numeric or canonical decimal-string representation can be reproduced without guessing formatting. */
export const queryTotalCount = (
value: JsonValue | undefined,
): number | undefined => {
const count =
typeof value === "number"
? value
: typeof value === "string" && /^(?:0|[1-9]\d*)$/u.test(value)
? Number(value)
: undefined;
return count !== undefined && Number.isSafeInteger(count) && count >= 0
? count
: undefined;
};
/** A query answer: the base line's envelope with the selected entities, counters at the number listed and totals at the number matched. */
export const renderQueryList = (
view: Pick<StateListView, "pointer" | "counters"> & {
readonly query: Pick<ListQuery, "totals">;
},
base: JsonValue,
items: readonly JsonObject[],
total: number,
): JsonValue =>
view.query.totals.reduce(
(body, name) =>
isObject(base) &&
Object.hasOwn(base, name) &&
queryTotalCount(base[name]) !== undefined
? replaceAt(
body,
pointerTo(name),
typeof base[name] === "string" ? String(total) : total,
)
: body,
renderStateList(view, base, items),
);
src/domain/analysisView/environmentStateRules.ts
+3
View File
@@ -1,3 +1,4 @@
import type { ListQuery } from "./environmentStateListQuery.js";
import { canonicalJson } from "../comparisonSemantics.js";
import type { JsonValue } from "../jsonValue.js";
import type { StateWritePlan } from "./environmentStateWriteRules.js";
@@ -111,6 +112,8 @@ export type StateListView = EntityViewPlan & {
/** With recorded membership, the checked lines that each list their whole collection. */
readonly all_record_ordinals?: number[] | undefined;
readonly counters: string[];
/** The query the view's other lines follow, when one explains any of them. */
readonly query?: ListQuery | undefined;
};
/** Whether a list view answers on a checked line with every entity of the collection. */
src/domain/analysisView/environmentStateView.ts
+69
View File
@@ -21,6 +21,70 @@ export const environmentStateViewRequestSchema = z.strictObject({
});
const jsonObjectFieldsSchema = z.record(z.string(), jsonValueSchema);
const queryParameterSchema = z.discriminatedUnion("role", [
z.strictObject({
role: z.literal("filter"),
field: z
.string()
.describe(
"Entities whose field equals the value as text, or whose array field holds it directly or as an object's id, slug or name.",
),
}),
z.strictObject({
role: z.literal("search"),
fields: z
.array(z.string())
.describe(
"Entities with a string field containing a plain substring value, ignoring case; an empty value matches every entity. Syntax-like operators replay instead.",
),
}),
z
.strictObject({ role: z.literal("sort") })
.describe(
"Orders by the field the value names (`name`, `-name`, `name:desc`, `name-ASC`): numbers by value, text ignoring case, null last ascending.",
),
z
.strictObject({ role: z.literal("direction") })
.describe("asc or desc for the sort parameter."),
z
.strictObject({ role: z.literal("limit") })
.describe("Page size; zero or a negative value lists every match."),
z
.strictObject({ role: z.literal("offset") })
.describe("Matches skipped before the page."),
z.strictObject({
role: z.literal("page"),
first: z
.number()
.int()
.describe(
"The first page, 0 or 1, when recordings distinguish the convention.",
),
}),
z.strictObject({
role: z.literal("inert"),
values: z
.array(z.string())
.describe(
"Recorded values that change nothing the view lists; other values are not computed.",
),
}),
]);
const listQuerySchema = z.strictObject({
parameters: z
.record(z.string(), queryParameterSchema)
.describe(
"What each query parameter does. A call whose every parameter has a role is answered by evaluating them over the state.",
),
record_ordinals: z
.array(z.number().int().nonnegative())
.describe("Checked lines the query reproduces, answered by evaluating it."),
totals: z
.array(z.string())
.describe(
"Observed envelope members with reproducible counts of every match before paging; absent or unknown representations stay recorded.",
),
});
const checkedSchema = z
.array(z.number().int().nonnegative())
.describe(
@@ -69,6 +133,11 @@ const viewSchema = z.discriminatedUnion("shape", [
counters: z
.array(z.string())
.describe("Envelope members that count the listed entities."),
query: listQuerySchema
.optional()
.describe(
"How the view's other checked lines select and order entities, when filters, a search, a sort or paging explain them; absent when none does.",
),
}),
z.strictObject({
tool: z.string(),
src/domain/analysisView/environmentStateWriteChecks.ts
+143 -147
View File
@@ -1,153 +1,33 @@
import type { JsonValue } from "../jsonValue.js";
import { iterateUrlQueryParameters } from "../recordedTransportUrl.js";
import {
compareUnicodeCodePoints as compare,
compositeKey,
} from "../unicodeCodePointOrder.js";
import { compositeKey } from "../unicodeCodePointOrder.js";
import {
at,
decodedSegment,
isObject,
isParam,
listsAll,
listItems,
listsAll,
missingRequiredFields,
renderStateEntity,
renderStateList,
replaceAt,
same,
scalarText,
segmentsOf,
type JsonObject,
type StateDocument,
type StateEntity,
type StateItemView,
type StateListView,
type StateRecordFacts,
type StateToolFacts,
type StateView,
segmentsOf,
} from "./environmentStateRules.js";
import { pageBound, pagesOnly } from "./environmentStatePaging.js";
import { pageBound } from "./environmentStatePaging.js";
import { answerQuery, renderQueryList } from "./environmentStateListQuery.js";
import { singleQueryValues } from "../recordedTransportUrl.js";
import type { ServedLine } from "./environmentStateCheckedLines.js";
import { insertIndex } from "./environmentStateWriteExplain.js";
import type { InsertRule } from "./environmentStateWriteRules.js";
/** A query as replay compares a call's: decoded values by name, in any order; undefined when it repeats a name with another value, which no call's arguments can carry. */
const queryKey = (query: string | undefined): string | undefined => {
const values = new Map<string, string>();
for (const [name, value] of iterateUrlQueryParameters(query ?? "")) {
if (values.has(name) && values.get(name) !== value) return undefined;
values.set(name, value);
}
return compositeKey(
[...values].sort(([left], [right]) => compare(left, right)),
);
};
interface CheckedLine {
readonly ordinal: number;
readonly segments: readonly string[];
}
/** A list or document view's checked lines, by query, in the tool's recorded order, and which path segments are the tool's parameters. */
export interface CheckedLines {
readonly params: ReadonlySet<number>;
readonly byQuery: ReadonlyMap<string, readonly CheckedLine[]>;
/** The lines listing their whole collection that only page it, which serving answers a call supplying no query from. */
readonly unfiltered: readonly CheckedLine[];
}
/** The line a read's call is answered from, and the collection serving lists for it. */
export interface ServedLine {
readonly ordinal: number;
readonly collection: string;
}
/** Each list or document view's checked lines. */
export const checkedLines = (input: {
readonly tools: readonly StateToolFacts[];
readonly records: readonly StateRecordFacts[];
readonly views: ReadonlyMap<string, StateView>;
}): Map<string, CheckedLines> => {
const byOrdinal = new Map(
input.records.map((record) => [record.ordinal, record]),
);
const tools = new Map(input.tools.map((tool) => [tool.name, tool]));
const byView = new Map<string, CheckedLines>();
for (const view of input.views.values()) {
const tool = tools.get(view.tool);
if (view.shape === "recorded" || view.shape === "item" || !tool) continue;
const checked = new Set(view.checked_record_ordinals);
const byQuery = new Map<string, CheckedLine[]>();
const unfiltered: CheckedLine[] = [];
for (const ordinal of tool.record_ordinals) {
const record = byOrdinal.get(ordinal);
const key = record === undefined ? undefined : queryKey(record.query);
if (!checked.has(ordinal) || key === undefined) continue;
const line = { ordinal, segments: segmentsOf(record!.path) };
byQuery.set(key, [...(byQuery.get(key) ?? []), line]);
if (
view.shape === "list" &&
listsAll(view, ordinal) &&
pagesOnly(record!.query)
)
unfiltered.push(line);
}
const params = new Set(
segmentsOf(tool.path_template).flatMap((segment, index) =>
isParam(segment) ? [index] : [],
),
);
byView.set(view.tool, { params, byQuery, unfiltered });
}
return byView;
};
/**
* The checked line serving answers a read's call from, ranked as replay ranks lines: every
* parameter the read's value verbatim or once percent-decoded and the query the read's, then the
* most verbatim parameters, then recorded order. A read with no query falls back, as serving does,
* on a whole list's lines that only page it.
*/
export const servedLine = (
lines: CheckedLines | undefined,
record: StateRecordFacts,
): ServedLine | undefined => {
const key = queryKey(record.query);
if (lines === undefined || key === undefined) return undefined;
return (
bestLine(lines, lines.byQuery.get(key) ?? [], record) ??
([...iterateUrlQueryParameters(record.query ?? "")].length === 0
? bestLine(lines, lines.unfiltered, record)
: undefined)
);
};
const bestLine = (
lines: CheckedLines,
candidates: readonly CheckedLine[],
record: StateRecordFacts,
): ServedLine | undefined => {
const read = segmentsOf(record.path);
let best: { line: ServedLine; verbatim: number } | undefined;
for (const { ordinal, segments } of candidates) {
if (segments.length !== read.length) continue;
let verbatim = 0;
const matches = segments.every((segment, index) => {
if (segment === read[index]) {
if (lines.params.has(index)) verbatim += 1;
return true;
}
return lines.params.has(index) && decodedSegment(segment) === read[index];
});
if (matches && (best === undefined || verbatim > best.verbatim))
best = {
line: { ordinal, collection: `/${segments.join("/")}` },
verbatim,
};
}
return best?.line;
};
export const entityTarget = (type: string, collection: string, id: string) =>
`e${compositeKey([type, collection, id])}`;
export const collectionTarget = (type: string, collection: string) =>
@@ -289,6 +169,8 @@ export const withBodies = <T>(
/** A check's outcome and the targets it covered. */
export interface CheckOutcome {
readonly pass: boolean;
/** The entities, by target, whose place a list read shows state order decides; absent, every listed entity's. */
readonly positional?: ReadonlySet<string>;
readonly targets: readonly string[];
/** Entities the recording, or the served base of a recorded-membership list, lists that the state does not hold, by target. */
readonly unheld: readonly string[];
@@ -296,6 +178,19 @@ export interface CheckOutcome {
readonly differing?: readonly string[];
}
/** Whether a passing check shows where a create placed its entity: a list's does unless a sort orders that entity, leaving it tied with none, or it matched alone. */
export const showsPlacement = (
view: StateView,
outcome: Pick<CheckOutcome, "positional">,
inserted: ReadonlyMap<string, string>,
tool: string,
): boolean =>
view.shape === "list" &&
(outcome.positional === undefined ||
[...inserted].some(
([target, create]) => create === tool && outcome.positional!.has(target),
));
/** What a check compares: the read's recorded body and the served base recording it would answer on. */
interface ReadBodies {
readonly body: JsonValue | undefined;
@@ -446,6 +341,8 @@ const listCheck = (
view: StateListView,
check: ListBodies,
members: readonly StateEntity[],
/** The matches a whole or query answer counts; undefined for recorded members, whose totals stay as recorded. */
total: number | undefined,
): Pick<CheckOutcome, "pass" | "differing"> => {
if (
members.some(
@@ -460,8 +357,16 @@ const listCheck = (
const rendered = members.map((entity) =>
renderStateEntity(entity, view, check.volatile, observed.get(entity.id)),
);
if (same(renderStateList(view, check.base, rendered), check.body))
return { pass: true };
const render = (items: readonly JsonObject[]): JsonValue =>
view.query === undefined || total === undefined
? renderStateList(view, check.base, items)
: renderQueryList(
{ ...view, query: view.query },
check.base,
items,
total,
);
if (same(render(rendered), check.body)) return { pass: true };
// When the recorded members alone account for the difference, only they differ.
const differs = members.filter((entity, index) => {
const item = observed.get(entity.id);
@@ -470,8 +375,7 @@ const listCheck = (
const patched = rendered.map(
(item, index) => observed.get(members[index]!.id) ?? item,
);
return differs.length > 0 &&
same(renderStateList(view, check.base, patched), check.body)
return differs.length > 0 && same(render(patched), check.body)
? {
pass: false,
differing: differs.map((entity) =>
@@ -481,6 +385,72 @@ const listCheck = (
: { pass: false };
};
/** The base line's and the read's listed entities by target: those the state holds, and those it no longer does. */
const listedTargets = (
state: SimulatedState,
view: StateListView,
check: ListBodies,
): { readonly targets: string[]; readonly unheld: string[] } => {
const targets: string[] = [];
const unheld: string[] = [];
const ids = [...check.baseItems, ...check.items].map(
(item) => scalarText(item[check.idField]) ?? "",
);
for (const id of new Set(ids))
(state.entity(view.type, check.collection, id) === undefined
? unheld
: targets
).push(entityTarget(view.type, check.collection, id));
return { targets, unheld };
};
/**
* A list read on a line the query answers: the query's matches in the state, in its order and page. Only a line whose
* parameters all sort or page, and that shows every match, covers the collection; any other covers the entities it lists.
*/
const checkQueryList = (
state: SimulatedState,
view: StateListView,
check: ListBodies,
values: ReadonlyMap<string, string>,
): CheckOutcome | undefined => {
const { collection } = check;
const result = answerQuery(
view.query!,
values,
state.members(view.type, collection),
);
if (result === undefined) return undefined;
const outcome = listCheck(view, check, result.page, result.total);
return {
...outcome,
// A lone match has no neighbour whose order it shows, so it shows nothing of where a create put it.
...(result.positional === "all" && result.total > 1
? {}
: {
positional: new Set(
(result.positional === "all" ? [] : result.positional).map(
(entity) => entityTarget(view.type, collection, entity.id),
),
),
}),
targets: [
...new Set([
...(result.whole ? [collectionTarget(view.type, collection)] : []),
...result.page.map((entity) =>
entityTarget(view.type, collection, entity.id),
),
// A failed read also covers the held entities it lists, as a write may have moved them out of the answer.
...(outcome.pass
? []
: listedTargets(state, view, { ...check, baseItems: [] }).targets),
]),
],
// What the base line or the read listed that the state no longer holds: the read shows whether it went.
unheld: listedTargets(state, view, check).unheld,
};
};
/** A list read of every entity of a collection: membership, order, fields and counters must render from the state. */
const checkAllList = (
state: SimulatedState,
@@ -491,7 +461,7 @@ const checkAllList = (
const members = state.members(view.type, collection);
const held = new Set(members.map((entity) => entity.id));
return {
...listCheck(view, check, members),
...listCheck(view, check, members, members.length),
targets: [
collectionTarget(view.type, collection),
...members.map((entity) =>
@@ -517,16 +487,26 @@ const checkRecordedList = (
const entity = state.entity(view.type, collection, idOf(item));
return entity === undefined ? [] : [entity];
});
const targets: string[] = [];
const unheld: string[] = [];
for (const id of new Set([...check.baseItems, ...check.items].map(idOf)))
(state.entity(view.type, collection, id) === undefined
? unheld
: targets
).push(entityTarget(view.type, collection, id));
return { ...listCheck(view, check, members), targets, unheld };
return {
...listCheck(view, check, members, undefined),
...listedTargets(state, view, check),
};
};
/** The read's query values when serving evaluates the query for it: on a named query line, or with query values on a line it did not name. */
const evaluatedQuery = (
view: StateListView,
record: StateRecordFacts,
line: ServedLine,
): ReadonlyMap<string, string> | undefined =>
(
line.named
? view.query?.record_ordinals.includes(line.ordinal) === true
: line.queried === true
)
? singleQueryValues(record.query)
: undefined;
/** A list read on a checked line, compared with the answer the served environment renders on that line's base recording from the collection it names: every entity when that line lists them all. */
export const checkListRead = (
state: SimulatedState,
@@ -535,11 +515,14 @@ export const checkListRead = (
check: ReadBodies & {
readonly idField: string;
readonly collection: string;
/** Whether the base line lists every entity of its collection. */
readonly all: boolean;
/** The checked line serving answers the read on. */
readonly line: ServedLine;
},
): CheckOutcome | undefined => {
if (!succeeded(record)) return undefined;
const { line } = check;
const query = evaluatedQuery(view, record, line);
const all = query === undefined && listsAll(view, line.ordinal);
const { body, base } = check;
const baseItems =
base === undefined ? undefined : listItems(base, view.pointer);
@@ -548,17 +531,30 @@ export const checkListRead = (
// Serving answers a page too small for the collection from its recording, so such a read checks nothing.
const size = pageBound(record.query, baseItems.length);
if (
check.all &&
all &&
size !== undefined &&
state.members(view.type, check.collection).length > size
)
return undefined;
// A usable body that no longer holds the list is not what serving answers on this line.
const items = listItems(body, view.pointer);
const bodies = { ...check, body, base, items: items ?? [], baseItems };
const outcome = check.all
? checkAllList(state, view, bodies)
: checkRecordedList(state, view, bodies);
// A line the read did not name keeps its own call's envelope, such as a link naming that call's values, which no write changes.
const envelope = line.named || items === undefined ? base : body;
// A query answered on a line the read did not name lists none of that line's members, so only the read's show what the state no longer holds.
const bodies = {
...check,
body,
base: envelope,
items: items ?? [],
baseItems: line.named || query === undefined ? baseItems : [],
};
const outcome =
query !== undefined
? checkQueryList(state, view, bodies, query)
: all
? checkAllList(state, view, bodies)
: checkRecordedList(state, view, bodies);
if (outcome === undefined) return undefined;
return items === undefined ? { ...outcome, pass: false } : outcome;
};
src/domain/analysisView/environmentStateWrites.test.ts
+302
View File
@@ -12,6 +12,7 @@ import {
BOOK_WRITE_CLOCK,
FIRST_BOOK,
LIST_LINE_ROWS,
QUERY_ROWS,
SECOND_BOOK,
THIRD_BOOK,
book,
@@ -544,6 +545,24 @@ describe("observation from a later read", () => {
});
describe("list lines", () => {
it("confirms the same create when a learned filter repeats identical decoded query values", () => {
for (const query of [
"shelf=a&shelf=a",
"shelf=%61&shelf=a",
"shelf=a&include=all&shelf=a",
]) {
const rows = QUERY_ROWS.map((row, index): TimedCaptureRow => {
if (index !== QUERY_ROWS.length - 1) return row;
const [method, , ...response] = row;
return [method, `/api/books?${query}`, ...response];
});
expect(planOf(stateOf(rows), "post_books"), query).toMatchObject({
served: true,
contradicted_by: null,
});
}
});
it("lists every entity on a line that lists its whole collection, though another line of the tool is filtered", () => {
const state = stateOf(LIST_LINE_ROWS);
expect(state.views.find((view) => view.tool === "get_books")).toMatchObject(
@@ -604,3 +623,286 @@ describe("list lines", () => {
);
});
});
describe("list queries", () => {
it("learns a filter, a search, a sort and an inert flag from the lines the whole collection does not explain", () => {
const state = stateOf(QUERY_ROWS);
expect(state.views.find((view) => view.tool === "get_books")).toMatchObject(
{
shape: "list",
membership: "recorded",
all_record_ordinals: [0],
query: {
parameters: {
shelf: { role: "filter", field: "shelf" },
q: { role: "search", fields: ["name", "slug"] },
sort: { role: "sort" },
include: { role: "inert", values: ["all"] },
},
record_ordinals: [1, 2, 3, 4],
totals: [],
},
checked_record_ordinals: [0, 1, 2, 3, 4],
},
);
// The shelf filter after the create lists the created book, which confirms it.
expect(planOf(state, "post_books")).toMatchObject({
served: true,
checked_record_ordinals: [8],
});
});
});
describe("list reads with query values no line recorded", () => {
it("serves no create that a filtered read on a new value omits, as serving would list it", () => {
const third = { ...book(3, "Third Book", bookClock(7)), shelf: "b" };
const rows: TimedCaptureRow[] = [
...QUERY_ROWS.slice(0, 7),
["POST", "/api/books", third, 201, { name: "Third Book", shelf: "b" }],
["GET", "/api/books/3", third],
// Serving evaluates `shelf=b` over the state, which lists the created book; the recording does not.
["GET", "/api/books?shelf=b", [QUERY_ROWS[6]![2] as Book]],
];
// The item read confirmed the book, so the failing filtered read blames the related create as a side effect.
expect(planOf(stateOf(rows), "post_books")).toMatchObject({
served: false,
reason: "side-effect",
});
});
it("checks what the state answers on a line the read did not name, not the envelope that line's own call recorded", () => {
const first = { ...FIRST_BOOK, shelf: "a" };
const second = { ...SECOND_BOOK, shelf: "b" };
const third = { ...book(3, "Third Book", bookClock(4)), shelf: "c" };
// Each page links the next under its own query.
const page = (query: string, books: readonly Book[]) => ({
data: [...books],
next: `/api/books?${query}offset=20`,
});
const rows: TimedCaptureRow[] = [
["GET", "/api/books", page("", [first, second])],
["GET", "/api/books?shelf=a", page("shelf=a&", [first])],
["GET", "/api/books/1", first],
["GET", "/api/books/2", second],
["POST", "/api/books", third, 201, { name: "Third Book", shelf: "c" }],
// Serving lists `shelf=c` from the state in the `shelf=a` line's envelope, whose link names shelf a.
["GET", "/api/books?shelf=c", page("shelf=c&", [third])],
];
expect(planOf(stateOf(rows), "post_books")).toMatchObject({
served: true,
checked_record_ordinals: [5],
});
});
it("checks a recorded-member line with the total it recorded, as serving answers it", () => {
const envelope = (books: readonly Book[]) => ({
items: [...books],
total: 3,
});
const renamed = renamedBook(SECOND_BOOK, "Renamed Second", 6);
const rows: TimedCaptureRow[] = [
["GET", "/api/books", envelope([FIRST_BOOK, SECOND_BOOK, THIRD_BOOK])],
["GET", "/api/books?limit=1", envelope([FIRST_BOOK])],
// A cursor no role explains keeps its recorded member and total.
["GET", "/api/books?cursor=abc", envelope([FIRST_BOOK])],
["GET", "/api/books/1", FIRST_BOOK],
["GET", "/api/books/2", SECOND_BOOK],
["GET", "/api/books/3", THIRD_BOOK],
["PATCH", "/api/books/2", renamed, 200, { name: "Renamed Second" }],
["GET", "/api/books/2", renamed],
["GET", "/api/books?cursor=abc", envelope([FIRST_BOOK])],
// Failing that read would taint book 1 and leave its confirmed delete unserved.
["DELETE", "/api/books/1", "", 204],
["GET", "/api/books/1", BOOK_NOT_FOUND, 404],
];
const state = stateOf(rows);
expect(planOf(state, "patch_books_by_id")).toMatchObject({ served: true });
expect(planOf(state, "delete_books_by_id")).toMatchObject({
served: true,
checked_record_ordinals: [10],
});
});
});
describe("list reads on a line they did not name", () => {
const first = { ...FIRST_BOOK, shelf: "a" };
const second = { ...SECOND_BOOK, shelf: "b" };
it("checks a read with no query on a query line by that line's recorded members, as serving lists them", () => {
const third = { ...book(3, "Third Book", bookClock(3)), shelf: "c" };
const rows: TimedCaptureRow[] = [
["GET", "/api/books?shelf=a", [first]],
["GET", "/api/books/1", first],
["GET", "/api/books/2", second],
["POST", "/api/books", third, 201, { name: "Third Book", shelf: "c" }],
// Serving answers this call on the `shelf=a` line with its recorded book, not the created one.
["GET", "/api/books", [first, second, third]],
];
expect(planOf(stateOf(rows), "post_books")).toMatchObject({
served: false,
checked_record_ordinals: [],
});
});
it("confirms no delete by a filtered read on another value, whose answer never listed the deleted book", () => {
const rows: TimedCaptureRow[] = [
["GET", "/api/books", [first, second]],
["GET", "/api/books?shelf=a", [first]],
["GET", "/api/books/1", first],
["GET", "/api/books/2", second],
["DELETE", "/api/books/1", "", 204],
["GET", "/api/books?shelf=b", [second]],
];
expect(planOf(stateOf(rows), "delete_books_by_id")).toMatchObject({
served: false,
reason: "unconfirmed",
checked_record_ordinals: [],
});
});
it("lets a sorted read whose values tie fix a create's position, as its order shows where the create put it", () => {
const shelved = (id: number, name: string, shelf: string, at: number) => ({
...book(id, name, bookClock(at)),
shelf,
});
const beta = shelved(1, "Beta", "a", 0);
const mike = shelved(2, "Mike", "b", 1);
const zulu = shelved(3, "Zulu", "a", 2);
const alpha = shelved(4, "Alpha", "a", 6);
const rows: TimedCaptureRow[] = [
["GET", "/api/books", [beta, mike, zulu]],
["GET", "/api/books?shelf=a", [beta, zulu]],
["GET", "/api/books?sort=-id", [zulu, mike, beta]],
["GET", "/api/books/1", beta],
["GET", "/api/books/2", mike],
["GET", "/api/books/3", zulu],
["POST", "/api/books", alpha, 201, { name: "Alpha", shelf: "a" }],
// Every match shares the shelf, so the order is the state's: the create went last.
["GET", "/api/books?shelf=a&sort=shelf", [beta, zulu, alpha]],
// A name order would put it first, which the read before contradicts.
["GET", "/api/books", [alpha, beta, mike, zulu]],
];
expect(planOf(stateOf(rows), "post_books")).toMatchObject({
served: false,
contradicted_by: 8,
});
});
it("leaves a create's position open after a sorted read that ties other books but not the created one", () => {
const shelved = (id: number, name: string, shelf: string, at: number) => ({
...book(id, name, bookClock(at)),
category: "x",
shelf,
});
const beta = shelved(1, "Beta", "a", 0);
const mike = { ...shelved(2, "Mike", "b", 1), category: "y" };
const zulu = shelved(3, "Zulu", "a", 2);
const alpha = shelved(4, "Alpha", "c", 6);
const rows: TimedCaptureRow[] = [
["GET", "/api/books", [beta, mike, zulu]],
["GET", "/api/books?category=x", [beta, zulu]],
["GET", "/api/books?sort=-id", [zulu, mike, beta]],
["GET", "/api/books/1", beta],
["GET", "/api/books/2", mike],
["GET", "/api/books/3", zulu],
[
"POST",
"/api/books",
alpha,
201,
{ name: "Alpha", category: "x", shelf: "c" },
],
// Beta and Zulu tie on shelf a, but the created book's shelf c sorts it last wherever it was put.
["GET", "/api/books?category=x&sort=shelf", [beta, zulu, alpha]],
["GET", "/api/books", [alpha, beta, mike, zulu]],
];
expect(planOf(stateOf(rows), "post_books")).toMatchObject({
served: true,
checked_record_ordinals: [7, 8],
insert: { rule: "sorted", field: "name", direction: "ascending" },
});
});
it("leaves a create's position open after a sorted read, for a later whole list to settle", () => {
const shelved = (id: number, name: string, shelf: string, at: number) => ({
...book(id, name, bookClock(at)),
shelf,
});
const beta = shelved(1, "Beta", "a", 0);
const mike = shelved(2, "Mike", "b", 1);
const zulu = shelved(3, "Zulu", "a", 2);
const alpha = shelved(4, "Alpha", "a", 6);
const rows: TimedCaptureRow[] = [
["GET", "/api/books", [beta, mike, zulu]],
["GET", "/api/books?shelf=a", [beta, zulu]],
["GET", "/api/books?sort=-id", [zulu, mike, beta]],
["GET", "/api/books/1", beta],
["GET", "/api/books/2", mike],
["GET", "/api/books/3", zulu],
["POST", "/api/books", alpha, 201, { name: "Alpha", shelf: "a" }],
// The sorted shelf passes wherever the create placed its book; the whole list orders by name.
["GET", "/api/books?shelf=a&sort=-id", [alpha, zulu, beta]],
["GET", "/api/books", [alpha, beta, mike, zulu]],
];
expect(planOf(stateOf(rows), "post_books")).toMatchObject({
served: true,
checked_record_ordinals: [7, 8],
insert: { rule: "sorted", field: "name", direction: "ascending" },
});
});
});
describe("what a list query read shows of writes", () => {
const shelved = (id: number, name: string, shelf: string, at: number) => ({
...book(id, name, bookClock(at)),
shelf,
});
const beta = shelved(1, "Beta", "a", 0);
const mike = shelved(2, "Mike", "b", 1);
const zulu = shelved(3, "Zulu", "a", 2);
const INIT: TimedCaptureRow[] = [
["GET", "/api/books", [beta, mike, zulu]],
["GET", "/api/books?shelf=a", [beta, zulu]],
["GET", "/api/books/1", beta],
["GET", "/api/books/2", mike],
["GET", "/api/books/3", zulu],
];
it("blames a write that moved a book off a filter whose failed read still lists it", () => {
const moved = { ...beta, shelf: "b", updated_at: bookClock(5) };
const renamed = renamedBook(zulu, "Zulu Two", 7);
const rows: TimedCaptureRow[] = [
...INIT,
["PATCH", "/api/books/1", moved, 200, { shelf: "b" }],
["PUT", "/api/books/3", renamed, 200, { name: "Zulu Two", shelf: "a" }],
// Serving lists Zulu alone on shelf a; the recording still lists the moved book there.
["GET", "/api/books?shelf=a", [beta, renamed]],
["GET", "/api/books/1", moved],
];
const state = stateOf(rows);
expect(planOf(state, "patch_books_by_id")).toMatchObject({
served: false,
contradicted_by: 7,
});
expect(planOf(state, "put_books_by_id")).toMatchObject({
served: false,
contradicted_by: 7,
});
});
it("leaves a create's position open after a filter that lists only its book, for a later whole list to settle", () => {
const alpha = shelved(4, "Alpha", "c", 5);
const rows: TimedCaptureRow[] = [
...INIT,
["POST", "/api/books", alpha, 201, { name: "Alpha", shelf: "c" }],
// A lone match shows nothing of where the create put it among the other books.
["GET", "/api/books?shelf=c", [alpha]],
["GET", "/api/books", [alpha, beta, mike, zulu]],
];
expect(planOf(stateOf(rows), "post_books")).toMatchObject({
served: true,
checked_record_ordinals: [6, 7],
insert: { rule: "sorted", field: "name", direction: "ascending" },
});
});
});
src/domain/recordedTransportUrl.test.ts
+17
View File
@@ -5,6 +5,7 @@ import {
parseUrlQueryParameterNames,
iterateUrlQueryParameters,
recordedPathSegments,
singleQueryValues,
} from "./recordedTransportUrl.js";
it("recognizes an empty userinfo marker at the authority start", () => {
@@ -53,6 +54,22 @@ it.each([
);
});
it.each([
"?shelf=a&shelf=%61",
"?shelf=%61&include=all&shelf=a",
"?q=two+words&q=two%20words",
"?%73helf=a&shelf=a",
"?=a&=a",
])("normalizes identical decoded query constraints: %s", (query) => {
expect(singleQueryValues(query)).toEqual(new Map(new URLSearchParams(query)));
});
it("retains conflicting duplicate constraints as unknown, including empty values", () => {
expect(singleQueryValues("?shelf=a&shelf=b")).toBeUndefined();
expect(singleQueryValues("?flag&flag=false")).toBeUndefined();
expect(singleQueryValues(undefined)).toEqual(new Map());
});
it("reads only requested path coordinates, retaining empty and raw segments", () => {
for (const path of ["", "/", "//a/", "/a/%2F/b", "relative/first/last"])
for (const index of [-1, 0, 1, 2, 3, 8])
src/domain/recordedTransportUrl.ts
+12
View File
@@ -134,6 +134,18 @@ export function* iterateUrlQueryParameters(
}
}
/** A single-valued form query: identical decoded repeats agree; conflicting values remain unknown. */
export const singleQueryValues = (
query: string | undefined,
): ReadonlyMap<string, string> | undefined => {
const values = new Map<string, string>();
for (const [name, value] of iterateUrlQueryParameters(query ?? "")) {
if (values.has(name) && values.get(name) !== value) return undefined;
values.set(name, value);
}
return values;
};
/** WHATWG form-query names, preserving order, repeats and empty names without decoding values. */
export function* parseUrlQueryParameterNames(
query: string,
tests/fixtures/environmentStateWrites.ts
Vendored
+28
View File
@@ -114,6 +114,34 @@ export const LIST_LINE_ROWS: readonly TimedCaptureRow[] = [
["GET", "/api/books", bookPage([FIRST_BOOK, SECOND_BOOK, THIRD_BOOK_AT_4])],
];
const shelved = (entry: Book, shelf: string): Book => ({ ...entry, shelf });
const SHELVED_FIRST = shelved(FIRST_BOOK, "a");
const SHELVED_SECOND = shelved(SECOND_BOOK, "b");
/** The book the create at exchange 7 of QUERY_ROWS made, on shelf a. */
export const SHELVED_THIRD = shelved(book(3, "Third Book", bookClock(7)), "a");
/**
* One list tool recorded whole, filtered by shelf, searched, sorted and with a flag that changes nothing; item reads;
* then a create the shelf filter later lists.
*/
export const QUERY_ROWS: readonly TimedCaptureRow[] = [
["GET", "/api/books", [SHELVED_FIRST, SHELVED_SECOND]],
["GET", "/api/books?shelf=a", [SHELVED_FIRST]],
["GET", "/api/books?q=second", [SHELVED_SECOND]],
["GET", "/api/books?sort=-id", [SHELVED_SECOND, SHELVED_FIRST]],
["GET", "/api/books?include=all", [SHELVED_FIRST, SHELVED_SECOND]],
["GET", "/api/books/1", SHELVED_FIRST],
["GET", "/api/books/2", SHELVED_SECOND],
[
"POST",
"/api/books",
SHELVED_THIRD,
201,
{ name: "Third Book", shelf: "a" },
],
["GET", "/api/books?shelf=a", [SHELVED_FIRST, SHELVED_THIRD]],
];
/** The end of the last exchange of BOOK_ROWS, to the next whole second: where the served clock starts. */
export const BOOK_WRITE_CLOCK = "2026-10-17T18:00:12.000Z";