mirror of
https://github.com/milvus-io/milvus.git
synced 2026-10-12 02:19:10 +00:00
6648d9371b
## Overview Adds a unified membership-filter surface syntax and merges the previously separate `bloom_match` / `roaring_match` chains into one parameterized execution path. issue: #52777 ## What's in this PR ### 1. Unified surface syntax ``` membership_match(<field>, {<blob-bytes-template>}) membership_match(<field>, {<blob-bytes-template>}, type=bloom|roaring) not membership_match(<field>, {<blob-bytes-template>}) ``` The filter kind is derived from the blob's self-describing magic header — `MBF1` lowers to the existing `BloomFilterExpr` plan node, `MRB1` to `RoaringFilterExpr`, and unknown headers fail closed. The optional `type=` argument is a readability aid and a consistency pin (it must agree with the magic). The unreleased `bloom_match` / `roaring_match` surface names are intentionally not retained — neither has shipped in a release. The wire plan bytes are unchanged from the two predecessor implementations, so the plan proto stays frozen and rolling-upgrade behavior for clients using `membership_match` matches what the predecessor docs specified. ### 2. One control flow, per-kind data planes - **Go parser/proxy**: one visitor/fill skeleton (`visitMembershipCall` / `fillMembershipMatchExpressionValue`), one spec table for behavior switches (delete safety), and unified tree walkers serving delete guards, element-level guards, plan-size accounting, and log redaction. - **segcore**: a single `PhyMembershipFilterExpr<LogicalExpr, ProbePolicy>` template carries the shared control flow; `BloomMembershipProbe` (zero-copy SBBF view) and `RoaringMembershipProbe` (decode-once Roaring64) remain separate data planes. Historical class names remain aliases, so factory construction and logs are unchanged. - Kind-specific admission gates (MBF1 envelope + domain check, MRB1 body walk + decoded-memory estimate) remain in their own files. - Client builders move under `client/membership/` (`sbbf`, `roaringfilter`); `bloom_match`/`roaring_match` builder packages are removed with the surface names. ### 3. Preserve existing mask/null semantics The unified executor preserves current `master` behavior on both raw-data and index-only paths: candidate-mask-excluded rows are left at their initialized `(false, valid)` value, including excluded NULL rows; probed NULL rows produce `(false, invalid)`. This PR does not change the framework reverse-lookup helper. ### 4. Two independent proxy budgets - `proxy.maxMembershipFilterSize` (default 64 MiB) limits one MBF1 or MRB1 body. It falls back to the released `proxy.maxBloomFilterSize` key and the development-only `proxy.maxRoaringFilterSize` key. - `proxy.maxMembershipFilterPlanSize` (default 128 MiB) remains the separate request-wide aggregate serialized-plan limit. It falls back to `proxy.maxBloomFilterPlanSize`. - Main predicates, scorer filters, and HybridSearch sub-requests share the same preflight budget. The preflight charges body bytes, matching the per-blob gate that allows the fixed 32-byte envelope header on top; final `proto.Size` accounting remains authoritative. Keeping the plan budget (128 MiB) well above the per-body limit (64 MiB) means a maximum-size body plus its 32-byte envelope and plan overhead always fits — no silent halving of the usable tier. ## Deliberate behavior changes 1. Adds `membership_match` with magic-based MBF1/MRB1 dispatch, an optional `type=` consistency pin, and fail-closed handling for unknown headers. 2. Removes the unreleased `bloom_match`/`roaring_match` surface names (and their client builder packages); the wire plan bytes for equivalent inputs are unchanged. 3. Updates error wording in bytes-template, element-level, MATCH_* and delete guards. 4. MATCH_* element-predicate guards now reject every membership-filter surface name. 5. Preflight aggregate accounting charges blob bodies rather than whole envelopes, matching the per-blob admission rule. ## Non-goals - No change to MBF1/MRB1 formats, golden vectors, or SDK builders. - No change to the plan proto or rolling-upgrade contract: fields 22/23 remain frozen. - No repo-wide NULL/candidate-mask semantic change. ## Design document [20260822-membership-match-expression.md](https://github.com/xiaofan-luan/milvus/blob/unify-membership-match/docs/design-docs/design_docs/20260822-membership-match-expression.md) ## Test plan - [x] `internal/parser/planparserv2`: full suite green, including magic dispatch, alias byte identity, fail-closed cases, delete safety, redaction, and shared preflight budgeting. - [x] `pkg/util/paramtable`: fallback-key precedence tests. - [x] `internal/proxy`: membership size/error/search/query focused tests green; the full local suite is green except `TestProxy`, which requires unavailable integration infrastructure and fails identically on clean HEAD. - [x] segcore: all 73 bloom/roaring/membership expression tests green; full `all_tests` has only failures reproduced on clean HEAD (6 ConjunctExprTest + 1 ExprJsonIndexTest abort). - [x] Local segcore build and unit-test binary link with llvm@19. --------- Signed-off-by: xiaofanluan <[email protected]> Signed-off-by: xiaofanluan <[email protected]> Co-authored-by: xiaofanluan <[email protected]> Co-authored-by: Claude Opus 4.6 <[email protected]>