3 people 6648d9371b enhance: unify membership filter expressions into one chain with membership_match syntax (#52778) ...

## Overview

Adds a unified membership-filter surface syntax and merges the
previously separate `bloom_match` / `roaring_match` chains into one
parameterized execution path.

issue: #52777

## What's in this PR

### 1. Unified surface syntax

```
membership_match(<field>, {<blob-bytes-template>})
membership_match(<field>, {<blob-bytes-template>}, type=bloom|roaring)
not membership_match(<field>, {<blob-bytes-template>})
```

The filter kind is derived from the blob's self-describing magic header
— `MBF1` lowers to the existing `BloomFilterExpr` plan node, `MRB1` to
`RoaringFilterExpr`, and unknown headers fail closed. The optional
`type=` argument is a readability aid and a consistency pin (it must
agree with the magic). The unreleased `bloom_match` / `roaring_match`
surface names are intentionally not retained — neither has shipped in a
release. The wire plan bytes are unchanged from the two predecessor
implementations, so the plan proto stays frozen and rolling-upgrade
behavior for clients using `membership_match` matches what the
predecessor docs specified.

### 2. One control flow, per-kind data planes

- **Go parser/proxy**: one visitor/fill skeleton (`visitMembershipCall`
/ `fillMembershipMatchExpressionValue`), one spec table for behavior
switches (delete safety), and unified tree walkers serving delete
guards, element-level guards, plan-size accounting, and log redaction.
- **segcore**: a single `PhyMembershipFilterExpr<LogicalExpr,
ProbePolicy>` template carries the shared control flow;
`BloomMembershipProbe` (zero-copy SBBF view) and
`RoaringMembershipProbe` (decode-once Roaring64) remain separate data
planes. Historical class names remain aliases, so factory construction
and logs are unchanged.
- Kind-specific admission gates (MBF1 envelope + domain check, MRB1 body
walk + decoded-memory estimate) remain in their own files.
- Client builders move under `client/membership/` (`sbbf`,
`roaringfilter`); `bloom_match`/`roaring_match` builder packages are
removed with the surface names.

### 3. Preserve existing mask/null semantics

The unified executor preserves current `master` behavior on both
raw-data and index-only paths: candidate-mask-excluded rows are left at
their initialized `(false, valid)` value, including excluded NULL rows;
probed NULL rows produce `(false, invalid)`. This PR does not change the
framework reverse-lookup helper.

### 4. Two independent proxy budgets

- `proxy.maxMembershipFilterSize` (default 64 MiB) limits one MBF1 or
MRB1 body. It falls back to the released `proxy.maxBloomFilterSize` key
and the development-only `proxy.maxRoaringFilterSize` key.
- `proxy.maxMembershipFilterPlanSize` (default 128 MiB) remains the
separate request-wide aggregate serialized-plan limit. It falls back to
`proxy.maxBloomFilterPlanSize`.
- Main predicates, scorer filters, and HybridSearch sub-requests share
the same preflight budget. The preflight charges body bytes, matching
the per-blob gate that allows the fixed 32-byte envelope header on top;
final `proto.Size` accounting remains authoritative. Keeping the plan
budget (128 MiB) well above the per-body limit (64 MiB) means a
maximum-size body plus its 32-byte envelope and plan overhead always
fits — no silent halving of the usable tier.

## Deliberate behavior changes

1. Adds `membership_match` with magic-based MBF1/MRB1 dispatch, an
optional `type=` consistency pin, and fail-closed handling for unknown
headers.
2. Removes the unreleased `bloom_match`/`roaring_match` surface names
(and their client builder packages); the wire plan bytes for equivalent
inputs are unchanged.
3. Updates error wording in bytes-template, element-level, MATCH_* and
delete guards.
4. MATCH_* element-predicate guards now reject every membership-filter
surface name.
5. Preflight aggregate accounting charges blob bodies rather than whole
envelopes, matching the per-blob admission rule.

## Non-goals

- No change to MBF1/MRB1 formats, golden vectors, or SDK builders.
- No change to the plan proto or rolling-upgrade contract: fields 22/23
remain frozen.
- No repo-wide NULL/candidate-mask semantic change.

## Design document


[20260822-membership-match-expression.md](https://github.com/xiaofan-luan/milvus/blob/unify-membership-match/docs/design-docs/design_docs/20260822-membership-match-expression.md)

## Test plan

- [x] `internal/parser/planparserv2`: full suite green, including magic
dispatch, alias byte identity, fail-closed cases, delete safety,
redaction, and shared preflight budgeting.
- [x] `pkg/util/paramtable`: fallback-key precedence tests.
- [x] `internal/proxy`: membership size/error/search/query focused tests
green; the full local suite is green except `TestProxy`, which requires
unavailable integration infrastructure and fails identically on clean
HEAD.
- [x] segcore: all 73 bloom/roaring/membership expression tests green;
full `all_tests` has only failures reproduced on clean HEAD (6
ConjunctExprTest + 1 ExprJsonIndexTest abort).
- [x] Local segcore build and unit-test binary link with llvm@19.

---------

Signed-off-by: xiaofanluan <[email protected]>
Signed-off-by: xiaofanluan <[email protected]>
Co-authored-by: xiaofanluan <[email protected]>
Co-authored-by: Claude Opus 4.6 <[email protected]>

2026-08-30 22:47:34 +08:00

..

roaring_match_test.go

enhance: unify membership filter expressions into one chain with membership_match syntax (#52778)

2026-08-30 22:47:34 +08:00