mirror of
https://github.com/milvus-io/milvus.git
synced 2026-10-12 02:19:10 +00:00
master
25727
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Jamesandxiaofanluan
|
ca31ef508b |
fix: update array equality test for renamed offsets API (#54130)
The ARRAY equality regression test still calls `GetArrayOffsets`, which was renamed to `GetStructElementOffsets` in #54045. Use the current accessor so the C++ coverage target compiles. This only updates the test fixture; production behavior is unchanged. Related: [#53411](https://github.com/milvus-io/milvus/pull/53411), [#54045](https://github.com/milvus-io/milvus/pull/54045). Signed-off-by: xiaofanluan <[email protected]> Co-authored-by: xiaofanluan <[email protected]> |
||
|
yihao.daiandClaude Opus 5.5
|
e2d836c50e |
test: fix ExprArrayTest compile error from GetArrayOffsets rename (#54119)
**Master build blocker.** The C++ unittest build on master fails. Every
PR that runs `ci-v2/build-ut-cov` fails at the UT-CPP-Cov step:
```
ExprArrayTest.cpp:1117:39: error: 'class {anonymous}::ArrayEqualityCountingSegment' has no member named 'GetArrayOffsets'
```
### Cause
- #54045 (merged 10-09) renamed `GetArrayOffsets` to
`GetStructElementOffsets`.
- #53411 (merged 10-10) added
`Expr.TestArrayIndexEqualityReadsOnlyCandidates`, which calls the old
name. Its last CI run (10-08) was before the rename.
### Change
Call `GetStructElementOffsets` in the test. The assertion still checks
that the nested-array offset map is not null. No production code
changes.
### Verification
- With the local build's compile flags, the fixed `ExprArrayTest.cpp`
compiles with no errors. The unfixed file fails with the same error as
CI.
- Not run locally: the full `all_tests` build. This PR's CI covers it.
#54118 contains the same one-line change in a larger AWS CRT runtime
fix. This PR unblocks master independently. #54118 can drop that commit
after this PR merges.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Signed-off-by: bigsheeper <[email protected]>
Co-authored-by: Claude Opus 5.5 <[email protected]>
|
||
|
Jamesandxiaofanluan
|
f7727377fd |
fix: support contextual keywords as field names (#53968)
Fields named `iso` or `interval` can be created, but filters such as `iso > 1` fail because the lexer emits a keyword token where the parser expects an identifier. Accept 20 contextual keyword families through a shared `fieldName` rule in expression field positions while preserving their function, option, and timestamp syntax. Update the visitor and regenerate the parser with ANTLR 4.13.2. Reject `LIKE`, `AND`, `OR`, `NOT`, and `IN` as field names in every casing, and retain the existing case-insensitive `NULL` policy. Validate struct-array parent names on both Create and Add paths, alongside child names. Classify `ErrFieldInvalidName` (1701) as `InputError` at its definition so ordinary names, reserved names, and RootCoord's add-struct-field validator report the same classification. Remove the redundant Proxy error markers and validate each struct parent name once while preserving the existing validation order, codes, reasons, identity, and non-retryability. Compatibility: mixed-case names such as `And`, `In`, and `Like` previously lexed as ordinary identifiers and could be created and filtered. New Create/Add requests reject these names. Existing collections are not revalidated, but backup restoration or cross-cluster schema recreation containing these names will require renaming the affected fields. This tightening is intentional; contextual keyword field names remain supported. Regression coverage includes contextual keywords and their dedicated syntax, field identity/casing, SLL/LL parsing, core keyword rejection, ordinary and struct-array Create/Add paths, reserved field names, and InputError status/metric round trips. RootCoord's name validator now also has classification and status round-trip coverage. Validation: - Current review follow-up: all tests in `pkg/util/merr`, `pkg/util/requestutil`, and `pkg/common` passed with `-tags dynamic,test -gcflags='all=-N -l' -count=1`; `git diff --check` passed. - Current focused Proxy/RootCoord tests were blocked before execution by older local native libraries missing required APIs. The development host was inaccessible under the current network restrictions; native CI validation is pending. - Before this follow-up, the unchanged parser/rewriter implementation passed 1,182 tests/subtests, focused Proxy regressions passed 248 tests/subtests with race detection and coverage, and `merr`/`requestutil` guards passed 143 tests/subtests with race detection and coverage. - Generated parser output was reproduced with ANTLR 4.13.2. - A previous full `make -o build-cpp-with-unittest test-go` attempt timed out in `TestProxy/create_collection` while waiting for streaming assignments and metadata-cache initialization. Later groups were not reached; no fresh C++ build was performed. issue: #53925 Fixes #53925 --------- Signed-off-by: xiaofanluan <[email protected]> Co-authored-by: xiaofanluan <[email protected]> |
||
|
Jamesandxiaofanluan
|
f0f197ec04 |
enhance: add fastpb runtime switch and optimize decoding (#54017)
issue: #50742 Add `common.enableFastPB` (default `true`) to select fastPB or official protobuf decoding on the supported RPC and direct search-result paths. The selected value is cached atomically so the RPC hot path avoids the config-manager read lock. Codec use before parameter initialization preserves the default enabled behavior without initializing configuration itself. The switch covers releaseCodec's RetrieveResults/InsertRequest/UpsertRequest dispatch and the direct SearchResultData decode sites in Proxy, QueryNode, LocalWorker, and late output-field materialization. The pure fastpb package remains configuration-independent. WAL encoding/repacking and codec buffer pooling are separate. Client Insert/Upsert strings retain proto3 UTF-8 validation. Enabled internal-result fast string paths assume trusted Milvus/segcore output and skip validation; disabling fastPB uses the official decoder and restores its UTF-8 validation. Packed scalar arrays (`LongArray`, `IntArray`, `BoolArray`) use protobuf-go's specialized decoder, avoiding the allocation and CPU regression from per-element growth. Raw packed fields still preallocate once. The shared wire primitive now uses `protowire.ConsumeVarint`, which has unrolled normal-value paths and rejects an overflowing varint after at most ten bytes. This covers long `HashKeys` and sealed-segment IDs as well as tags, length prefixes, and scalar values. Serialization and peer compatibility are unchanged. Validation for the latest update: - `go test ./util/fastpb ./util/resource -count=1` passed in the nested `pkg` module on master, 3.0, and 2.6. - 48 differential boundary cases cover max uint64, nonminimal ten-byte zero, truncation, overflow, and a 1 MiB continuation sequence through scalar, packed, length-prefix, and unknown-field paths. - Added reproducible `BenchmarkInsert_HashKeys` and `BenchmarkRetrieve_SealedSegmentIDs` cases. - Paired local master benchmarks (Go 1.26.6, Apple M4 Pro, `GOMAXPROCS=1`, 11 alternating rounds) reduced 1000 five-byte HashKeys decoding from 5.377 to 4.561 us and 1000 nine-byte sealed IDs from 8.465 to 7.157 us. Allocations were unchanged, and both cases were near the official decoder. These are synthetic decode benchmarks, not end-to-end QPS results. - For a malformed NumRows value, increasing a continuation sequence from ten bytes to 1 MiB no longer increases rejection time. Packed fields still scan their payload to count elements before decoding. Native internal integration paths and end-to-end QPS/P99 were not rerun locally. CI results must be assessed on the latest PR head. Runtime configuration refresh now registers an exact-key handler through the config dispatcher, which matches normalized aliases including etcd's separator-free key. CREATE, UPDATE, and DELETE events re-read the uncached effective value, preserving runtime overrides and restoring the default after deletion or Reset. The refresh read and atomic publication are serialized on the update path; RPC decoding still only loads the atomic snapshot. Callback regression validation: - The normalized CREATE test failed before the fix: the effective ParamItem was false while the atomic switch remained true. - The new source-event test covers normalized CREATE/UPDATE/DELETE, a prewarmed typed cache, runtime overrides, Reset to the default, and runtime key aliases. It checks actual gRPC RetrieveResults and direct SearchResultData decoder selection without mocking. - `go test ./util/fastpb ./util/resource -count=1` passed on master, 3.0, and 2.6 after the callback fix. - `go test -race ./util/resource -run '^TestFastPBConfigSourceEvents$' -count=3` passed on all three branches. - Focused `go test ./util/paramtable -run FastPB -count=1` passed on master and 3.0. The pre-existing 2.6 paramtable test-package compilation issue involving `log.CaptureGlobalLogs` was not changed; the production paramtable package compiled through the resource tests. ```yaml common: enableFastPB: true ``` Related decoder backports: #54002 (2.6), #54003 (3.0). --------- Signed-off-by: xiaofanluan <[email protected]> Co-authored-by: xiaofanluan <[email protected]> |
||
|
congqixia
|
fb7d370050 |
fix: [GoSDK] support Timestamptz row insert (#53775)
getColumnCreators has no cases for FieldTypeTimestamptz or FieldTypeGeometry, so a row-based insert into a collection containing either field silently produced a nil column and panicked later in AnyToColumns' AppendValue call (nil interface method dispatch). Add column creators for both field types: - Timestamptz maps to NewColumnTimestamptz; ColumnTimestamptz now overrides AppendValue to accept time.Time, *time.Time and ISO 8601 strings (RFC3339Nano), matching the wire format the server expects. - Geometry maps to NewColumnGeometryWKT. Also add an exhaustive regression test enumerating all supported scalar field types, asserting getColumnCreators yields a non-nil column, so a future field type missing a creator is caught by CI instead of panicking at runtime. Related to #53764 --------- Signed-off-by: Congqi Xia <[email protected]> |
||
|
Li Liu
|
1c831e2b03 |
enhance: bound raw array reads during indexed equality checks (#53411)
Indexed ARRAY equality and inequality currently reconstruct and copy a full chunk of ArrayViews for every candidate requiring exact comparison. Collect the candidate offsets once, sort them to reuse Raw cell pins, and use one borrowed Take result for the exact post-filter. Retain index candidate selection, full-array order/length/duplicate comparison, nullable results, nested row mapping, growing-segment behavior, and cached result batches. issue: #53407 Sorting adds O(K log K) work and an O(K) offset vector for K candidates. No production latency or allocated-byte reduction has been measured. Validation: - CI fixture repair, current head `c4503f6aef1b5ff64749cbb69fdd952bc611691c`: the counting fixture now returns `FieldSkipMetricsView` and directly includes its defining header, matching the upstream scan-resource interface. Authenticated Jenkins coverage-build logs reproduced the original compiler error. Source-extracted type/call checks failed before and passed after with Clang and Linux GCC; independent Spec and Standards re-reviews found zero remaining findings. All existing assertions and production code are retained. Full local test translation-unit attempts are blocked by missing Boost/Arrow headers, so these checks do not establish full compilation or gtest success. New-head CI is pending. - Rebased on 2026-09-22 onto master `8c359cbaf6886de048860fee65514e5a3c90da84`; rebased head `7dfdc9bf87ee307ac5f3942b551568d36319050c`. Resolved the overlap with #53455 by retaining master's cached const ARRAY reference and this PR's candidate-only Take post-filter. Range-diff review, changed-line clang-format 15.0.7, `git diff --check`, and commit sign-off checks passed. The focused C++ test command could not execute locally because `internal/core/output/unittest` is absent; current-head build/runtime CI is pending. - Historical head `9562276f48caf0f8365a35d2dec0dab67222f79f` passed the main build/code-check/Go/integration/C++ pipeline (9,107 C++ tests, zero failures), default E2E, Go SDK and DCO. [CI result](https://github.com/milvus-io/milvus/pull/53411#issuecomment-5645886347). - Added a regression using real expression evaluation, inverted indexes, and loaded columns: `Expr.TestArrayIndexEqualityReadsOnlyCandidates`. It covers nullable INT64 arrays, regular/nested indexes, memory/mmap loading, unequal chunks, sparse/dense candidates, empty arrays, and repeated batches. Authenticated CI shard logs confirm this committed gtest executed and passed. - The production UnaryExpr.cpp and final regression translation unit passed C++20 syntax compilation. - An ASAN/UBSan harness containing the complete production function body passed 32 scenarios on both baseline and changed code. Its baseline constructed 1,252,352 full-chunk ArrayViews; the change constructed zero. The same construction-count guard fails on baseline and passes on the change. - The harness substitutes the index, segment, Take, and storage infrastructure. A synthetic exception injected at its Take boundary escapes the function; this does **not** validate real storage error categories, retries, cancellation, cold loading, or eviction. Concurrent snapshot replacement and Vortex execution are also unverified. The historical CI execution confirmed the regression ran on the pre-rebase head. It does not verify the current rebased head or establish the unmeasured performance or storage-failure benefits listed above. --------- Signed-off-by: Li Liu <[email protected]> |
||
|
|
55ba255cc7 |
enhance: convert ARRAY_CONTAINS terms only on an index-result cache miss (#54052)
issue: #54051 ## What `ExecArrayContainsForIndexSegmentImpl` rebuilt the `unordered_set` of query terms, and the vector handed to the index, on every call. The index lookup is cached per index chunk, but the call runs once per 8192-row batch, so a 5.3M-row segment paid for 644 rebuilds per query. With hundreds of terms that dominated the cost of an indexed `ARRAY_CONTAINS_ANY`. #48049 hides this when every expression in the plan is index-served; any raw-data expression in the plan (for example `ARRAY_LENGTH(f) == 0`) brings it back. Build the term list inside the index callback, which `ProcessIndexChunksImpl` invokes only on a cache miss, as `ExecArrayEqualForIndex` already does. An ExprResCache hit skips the conversion as well. Also drops an unused `unordered_set` in `ExecJsonContainsByStats`. ## Measurements 8-core box, 5.3M rows, VARCHAR array with an inverted index, 16 concurrent clients, filter-only CPU per query (core-ms): | filter | before | after | |---|---|---| | `ARRAY_CONTAINS_ANY(arr, [965 terms])` | 92.1 | 58.4 | | same, OR `ARRAY_LENGTH(arr) == 0` (customer filter) | 138.8 | 99.3 | | 966 terms on a BITMAP-indexed field | 63.4 | 24.2 | Results identical on 11 correctness filters covering nested AND/OR/NOT. Measured on a v2.6.22 build of the same change; on master the same cost shows up whenever the plan is not fully index-served (965-term clause: 19.9 core-ms alone, 94.8 when AND'ed with an `ARRAY_LENGTH` predicate that costs 32.4 alone). ## Test New `Expr.TestArrayContainsIndexPathAcrossBatches`: a 20,000-row sealed segment with a BITMAP index on a VARCHAR array (built, serialized and reloaded), evaluated with `EvalExprInBatches` so it runs as three batches without the single-pass optimization; `ContainsAny` with a few hundred terms each listed twice and `ContainsAll`, checked row by row against a brute-force evaluation. `all_tests --gtest_filter='Expr.TestArrayContains*:Expr.TestJsonContains*:Expr.*ContainsAny*:*JsonContains*'`: 604 passed. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Signed-off-by: Zack <[email protected]> Signed-off-by: Li Liu <[email protected]> Co-authored-by: Zack <[email protected]> Co-authored-by: Claude Opus 5.5 <[email protected]> |
||
|
Buqian Zheng
|
fc880d2945 |
build(deps): update Tantivy for phrase-match optimizations (#54023)
Advance the locked Tantivy 0.23 revision to `fef3c2a5748bdb2b2c2f8ca1d9fc125a0d1e8b71`, including the no-scoring phrase optimizations from [#30](https://github.com/zilliztech/tantivy/pull/30), [#33](https://github.com/zilliztech/tantivy/pull/33), and the repeated-term correctness fix from [#34](https://github.com/zilliztech/tantivy/pull/34). For example, `a a` with slop 1 must not match a document containing only one `a`. Milvus's phrase-match collectors disable scoring and reach the corrected path. ### Revision and scope - Verified #34 merged at 2026-10-10 03:14:27 UTC. Its merge commit is the selected revision and was the upstream default branch `main` HEAD when this update began. - Its sole parent is the previous PR pin `596a0f65ba24dd94b15ef97b9b9a909b610b8f22` (#33), whose parent is `9ce00de9a822b2a1cfa8d2b91454d7ae017969b9` (#30). GitHub state and local Git ancestry agree. - This follow-up changes only nine Tantivy workspace source hashes in `internal/core/thirdparty/tantivy/tantivy-binding/Cargo.lock` (+9/-9). The upstream increment changes four phrase-query source files, with no manifest changes. - The complete PR remains lockfile-only (+20/-11 from the branch base), retaining the original rust-stemmers 1.3.0 addition required by modern Tantivy. Legacy Tantivy 0.21.1, its rust-stemmers 1.2.0 pin, all registry packages, and Cargo.toml remain unchanged. ### Current local validation (2026-10-10, fef3c2a57) - Normal retained `build-server.sh`: passed; release Rust bindings, Milvus core, C++ test targets and Go server rebuilt with dependency checks. FFI header and artifact/shared-library checks passed. - Binding release text/phrase tests: 4 passed. Analyzer tests: 6 passed. - Upstream `cargo +1.89 test --locked --lib`: 919 passed, 5 ignored. Phrase subset: 39 passed, 1 ignored. Includes repeated-term indexed queries, exhaustive and randomized injective-assignment regressions, clone sharing, and explain tests. - Freshly linked C++ `TextMatch.*:PackedTextMatchAsyncLoadTest.*`: 33 passed, 2 failed because existing LOB fixtures write `/sealed_text_lob_index_*` and `/growing_text_lob_index_*` on a read-only filesystem. No fixture edits were applied this round; no #53986 changes are included. - `git diff --check` and parsed lockfile comparison: passed; only the nine source hashes changed in this follow-up. The earlier validation of 596a0f65 is historical: its two LOB fixture tests passed with temporary #53986 path adjustments, subsequently reverted. That result is not claimed as a test of this revision. No live-cluster E2E or local performance benchmark was run. Upstream benchmark sources are not checked in; upstream #34 reports some regressions as well as improvements, so this update makes no universal speedup or measured Milvus performance claim. The fix covers unscored literal PhraseQuery; scored, regex and prefix phrase semantics are outside its scope. --------- Signed-off-by: Buqian Zheng <[email protected]> |
||
|
aoiasd
|
87bdd62402 |
feat: [RLS8] decouple RLS metadata from enforcement (#54059)
relate: #50263 design doc: docs/design-docs/design_docs/20250610-rls_design.md design doc PR: #53173 ## Summary Allow changing `rls.enabled` independently of policy/tag management. Preserve metadata across toggles and permit configuration before enabling enforcement; keep schema dependency checks and explicit drop cleanup. Recover all saved policies into resident Coord metadata before readiness, including disabled collections, and maintain the map after successful persistence. Principal tags remain persisted in etcd and point-read by Coord without startup preload; Proxy execution caches are unchanged. Reuse existing collection-cache invalidation, continue invalidation after an ignored missing-resource-group result, and simplify redundant request/enum adapters without removing supported protocol modes. ## Validation Two fresh independent static reviewers completed the full eligible production diff at `96688e8484`; primary reconciliation found no new actionable finding. Static failure-path checks cover recovery failure, persisted mutation failure, metadata preservation across toggles and cache invalidation retries. Regression coverage was updated but has not been executed for this revision. No new local build or test run has been performed for this revision. Earlier build and test results apply to earlier commits only; current runtime validation remains with PR CI. Static review excludes tests/generated contents and does not verify distributed execution. Distributed cutover is not claimed to be atomic. --------- Signed-off-by: aoiasd <[email protected]> |
||
|
zhenshan.caoandZack
|
cd6c63d85f |
fix: return the replicate configuration of an unconfigured cluster instead of blocking (#52687)
issue: #52686 ## Problem `GetReplicateConfiguration` never returns on a cluster that has never been given a replicate configuration — it blocks until the caller's deadline and surfaces as `DEADLINE_EXCEEDED`, which reads as an unhealthy cluster. Every freshly installed cluster is in that state, so reading the topology before configuring replication appears to fail. A cluster with no replication has no configuration to report, and that is nil all the way down: the balancer pushes a nil `ReplicateConfiguration` when `cm.replicateConfig` is nil, `NewConfigHelper` maps a nil config to a nil helper, and the watcher's wait condition treated that nil helper as "no assignment received yet": ```go for (…Version.Global == -1 && …Version.Local == -1) || w.lastVersionedAssignment.ReplicateConfigHelper == nil { ``` The second term can never be satisfied there. The sibling accessors on the same watcher, `GetLatestStreamingVersion` and `GetLatestDiscover`, wait only on the version. Sending an empty configuration from the balancer instead is not viable: `NewConfigHelper` rejects a configuration with no clusters (`primary count is not 1`), so `MustNewConfigHelper` would panic in every client. A nil helper *is* the representation of "no replication configured". #47517 made this deterministic rather than intermittent: the public API now always takes the fresh-read path, which builds a throwaway watcher and waits for one assignment push, so a genuinely fresh cluster fails every time while a previously-configured one still answers from its long-lived watcher. ## Change - `watcher.GetLatestReplicateConfiguration` waits only until an assignment has been received, and returns the possibly-nil helper. - `ConfigHelper.GetReplicateConfiguration` and `GetCluster` are nil-receiver safe, so callers can use the result directly. - `overwriteReplicateMessage` rejects a replicate message as a replicate violation when the cluster has no configuration. This state is reached in the normal bootstrap order, when the primary has been given the replicate configuration and the secondary has not yet; it previously blocked on the configuration query and is now rejected explicitly. - `replicateStreamClient` no longer resets its reconnect backoff as soon as `CreateReplicateStream` succeeds. Against a secondary that accepts the stream but rejects every message, the retry interval never left its 100 ms initial value and each round re-sent the pending messages from the head. The backoff is now reset only once a connection has proven healthy: the target confirmed at least one message, or the stream was closed by the idle timeout because there was nothing to send. Rejected connections back off from 100 ms up to 10 s. ## Tests - `blocks_when_version_set_but_config_nil` asserted the old behaviour (deadline). It is replaced by `returns_nil_when_version_set_but_config_nil`, which asserts a prompt nil result and that the read accessors stay usable on it. - New `blocks_until_first_assignment_arrives` keeps the genuine "not known yet" case covered: with no assignment at all the call still blocks, and returns once one arrives. - New `TestReplicateServiceNoConfig`: the configuration query answers `(nil, nil)` and `Append` returns `STREAMING_CODE_REPLICATE_VIOLATION`. - New `TestReplicateStreamClient_BackoffNotResetWhileTargetRejects`: the stream opens but `Recv` fails with `FailedPrecondition`; after five reconnects the backoff was never reset. `TestReplicateStreamClient_BackoffResetAfterIdleTimeout`: `Recv` fails with the stream idle timeout; the backoff is reset on every round. - `go test ./internal/cdc/replication/replicatestream/ ./internal/distributed/streaming/ ./internal/streamingcoord/client/assignment/` and `golangci-lint` on the same packages pass on current master. Signed-off-by: zhenshan.cao <[email protected]> Signed-off-by: Zack <[email protected]> Co-authored-by: Zack <[email protected]> |
||
|
Li LiuandCodex
|
6b1ba4f71c |
fix: respect metric direction in aggregation top hits (#54083)
issue: #54079 Pass the reducer's resolved metric to search aggregation so default `top_hits`, omitted `_score` directions, and scalar sort ties keep the most similar candidates before truncation. Preserve explicit numeric `asc`/`desc`, returned scores, and `_score` metric values. Correct the design document's score-direction contract. Validation: - Reproduced incorrect L2/HAMMING parent hits through the cross-shard reduce → aggregation pipeline on the baseline; the fixed pipeline passes with both serialized and in-process shard results. - Full `internal/proxy/search_agg` tests, including distance/similarity metrics, explicit sorting, PK ties, and parent/leaf truncation. - Full `TestSearchPipeline` suite and `TestReduceMultiGroupBy*` in `internal/proxy/dql`. - Tests used `-tags dynamic,test -gcflags='all=-N -l'`; no server-level e2e run. CI follow-up: update the existing RLS nested-NOT regression to account for parser normalization already introduced on master by #53624. The test still constructs and rejects an unsupported nested AST, and now checks double-negation TRUE/FALSE/UNKNOWN semantics with optimization enabled and disabled. Parser, validator, and runtime production code are unchanged. The full `rlsutil` package passed three runs, and an independent review verified the retained rejection coverage. --------- Signed-off-by: Li Liu <[email protected]> Co-authored-by: Codex <[email protected]> |
||
|
pymilvus-bot
|
1e943dc23b |
test: Increase PyMilvus version to 3.1.0rc92 from master for master branch (#54068)
Automated bump from pymilvus master branch to milvus master branch. Updates tests/python_client/requirements.txt. Signed-off-by: pymilvus-bot <[email protected]> |
||
|
Li Liu
|
d78a89dbe1 |
fix: release Rust token strings after word-level MinHash computation (#54082)
Word-level MinHash kept the Rust tokenizer's allocated strings in a raw-pointer vector and never released them. Own each token with `unique_ptr` and `free_rust_string` before inserting it, so normal completion and allocation exceptions release every token after its last use. Add word-level regression coverage for both hash functions, normalized/UTF-8 text, empty and short inputs, overlapping shingles, vector growth and repeated calls. Hash values remain unchanged, including the existing SHA1 conversion behavior. Validation: a focused ASan/LSan native harness runs the actual MinHash source and these tests against real Rust token FFI, and separately against the official `master-20261009-e335524a-arm64` native library. Before: 16 behavioral tests pass, but 5,134 allocations leak 35,928 bytes. After: all 18 tests pass, including allocation-failure injection during token-vector and shingle-buffer growth; all 5,391 token allocations are freed, with no sanitizer errors. The harness uses native MinHash dispatch and small adapters for unrelated logging/assertion dependencies; a full Milvus service e2e was not run. Ordinary builds run the committed hash assertions; leak detection and the two injected-failure cases were checked in the local sanitizer harness. Fixes #54081 issue: #54081 CI follow-up: update the existing RLS nested-NOT regression to account for parser normalization already introduced on master by #53624. The test still constructs and rejects an unsupported nested AST, and now checks double-negation TRUE/FALSE/UNKNOWN semantics with optimization enabled and disabled. Parser, validator, and runtime production code are unchanged. The full `rlsutil` package passed three runs, and an independent review verified the retained rejection coverage. --------- Signed-off-by: Li Liu <[email protected]> |
||
|
Li Liu
|
89aa9e31c4 |
fix: preserve omitted timestamptz fields in partial upserts (#54087)
Partial upserts that omit a nullable or default-valued TIMESTAMPTZ field can panic or reject valid requests when they insert new PKs. The generated integer column replaces the string-form query results during merge, leaving the validity mask and payload out of alignment. Generate omitted TIMESTAMPTZ values as strings, matching retrieved rows until insert validation converts the complete column to UTC microseconds and fills nulls/defaults. Existing timestamps and their row association are preserved. issue: #54084 Validation: - Added 30 regression cases through production upsert preparation and insert validation, with mocked query/allocator/WAL dependencies. Covers nullable/default schemas, all-new and mixed PK batches, null ordering, non-UTC collection timezone, and RLS USING on/off. - Before the fix: 8 panic failures and 22 timestamp-type validation failures. All pass after the fix, with exact timestamp/default/null assertions. - Full `internal/proxy/dml`, `internal/proxy/fieldvalidator`, and `internal/proxy/dql` tests passed using `-tags dynamic,test -gcflags="all=-N -l" -count=1` and the official native libraries. No live-cluster reproduction was performed locally. CI follow-up: update the existing RLS nested-NOT regression to account for parser normalization already introduced on master by #53624. The test still constructs and rejects an unsupported nested AST, and now checks double-negation TRUE/FALSE/UNKNOWN semantics with optimization enabled and disabled. Parser, validator, and runtime production code are unchanged. The full `rlsutil` package passed three runs, and an independent review verified the retained rejection coverage. Additional CI follow-up: fix `TestMaybeForcePersistedSync` waiting for a notification sent before `asyncSync` released its working slot. The test now waits for actual slot cleanup, checks the scheduling decisions, and retains the two persisted-sync calls. This changes only the test. The original test hung naturally 3 times in 13,904 race-enabled runs at the same receive as CI; the corrected test passed 13,904 race-enabled runs, 1,000 non-race runs, and the full inspector package with `-race`. Independent review confirmed the synchronization fix and retained assertions. --------- Signed-off-by: Li Liu <[email protected]> |
||
|
Jamesandxiaofanluan
|
f4ae66a2fd |
enhance: add startup flag for existing native profiling (#53997)
Milvus already provides jemalloc native heap profiling and `/debug/jemalloc/pprof/heap`. This PR only adds an opt-in startup flag, using the existing subprocess path so jemalloc receives its configuration before allocator initialization. ```sh milvus run querynode --native-profile curl -f http://localhost:9091/debug/jemalloc/pprof/heap -o native.heap ``` The flag enables profiling and allocation sampling while preserving other `MALLOC_CONF` options, including the sampling interval. Existing builds, HTTP routing, dump behavior, and dependencies are reused. The legacy HTTP endpoint requires a Linux `with_jemalloc` build with profiling support and remains subject to existing pprof/management routing; enabling management authentication hides legacy DefaultServeMux routes. Fixes #53985 Validation: focused CLI tests and Go vet passed; the startup-only revision also passed a Linux `go build -mod=readonly -tags dynamic,with_jemalloc ./cmd` and gosec on `cmd/milvus`. Formatting and diff checks passed. No service or profiling workload was started. Signed-off-by: xiaofanluan <[email protected]> Co-authored-by: xiaofanluan <[email protected]> |
||
|
|
5ce06431c6 |
fix: prevent stack overflow and O(N^2) regression on large OR/AND filter expressions (#53624)
## What this PR fixes Fixes #53623. Large, flat OR/AND filters produced left-deep parse and plan trees. Recursive visiting could exhaust the stack, while the rewriter repeatedly flattened the same subtrees, approaching O(N²). This PR collects each same-operator chain once and rebuilds balanced plans while preserving operand order and mixed AND/OR boundaries. ## Changes - Share iterative AND/OR operand collection and pairwise folding. Validate operands before constant folding and preserve template metadata and terminal `random_sample` / `element_filter` behavior. - Collapse consecutive NOT operators by parity after validating the underlying operand. - Preserve existing document predicates in nested wrappers, including `a AND (b AND random_sample(...))` and `element_filter` combinations. - Reject document-predicate OR combinations with `element_filter` in both `ParseExpr` and `ParseExprTemplate`. Keep Boolean-literal folding and template-value validation; retain the post-fill top-level placement check. - Preserve all remaining IN constraints when nullable or missing-path semantics prevent an empty intersection from becoming a constant. - Retain IN/`!=` cross-rewrites. Build a typed exclusion set once and scan the IN list once, giving expected O(M+K) membership work for M IN values and K inequalities. A single `!=` uses direct comparisons. Preserve NULL/missing-path guards and conservatively skip float32 narrowing, NaN, reverse membership, mixed literal kinds and multiple remaining IN constraints. ## Validation - Passed the complete Go parser/rewriter suite on the final local implementation: ```bash LOCAL_STORAGE_PATH=/tmp go test -tags dynamic,test -gcflags='all=-N -l' -count=1 ./internal/parser/planparserv2/... ``` - Coverage includes 6,000-term chains and protobuf round trips, operand order and mixed-operator boundaries, templates and deferred constants, wrapper predicate preservation, nullable/missing-path constraints, outer NOT, float32/NaN/reverse-membership guards, and typed-key equality including signed zero and large integers. The existing deterministic randomized three-valued tests also pass. - Local Apple M4 Pro microbenchmarks with the same disabled-optimization flags, 50,000 IN values and 100 disjoint `!=` values: membership lookup 79.35 ms with nested scanning versus 1.77 ms with the set-based approach; complete cross-rule AND/OR execution 3.53/3.08 ms. These measurements exclude parsing, sorting, other rewrite rules and vector search. - `gofmt` and `git diff --check` passed; read-only review found no blocking issue. ## Scope and limitations - Go parser/rewriter changes only; no C++ or plan-protocol changes. - No universal limit is added for arbitrary syntactic nesting, and FLOAT/ARRAY<FLOAT> literal-precision behavior is not changed. - Native end-to-end execution and JSON raw/stats/index equivalence were not run locally. Check CI for the new head; results from the previous head do not validate this update. - A non-blocking allocation improvement remains: AND allocates its filtered IN buffer even when no value is excluded. issue: #53623 Signed-off-by: xiaofanluan <[email protected]> Co-authored-by: xiaofanluan <[email protected]> Co-authored-by: Codex <[email protected]> |
||
|
Buqian Zheng
|
65761bd3cb |
enhance: rename the struct ArrayOffsets abstraction to StructElementOffsets (#54045)
issue: #51162 Rename the segcore struct offsets abstraction from `IArrayOffsets` / `ArrayOffsetsSealed` / `ArrayOffsetsGrowing` to `IStructElementOffsets` / `StructElementOffsetsSealed` / `StructElementOffsetsGrowing`. The mapping is built only for struct sub-fields: one per struct, shared by all sibling sub-fields, from each row to its struct elements, with element IDs contiguous across the segment. The old name suggested a per-field or per-nesting-level offsets array, which is what each column's own chunk (`ArrayChunk`, `ColumnarArrayChunk`, `VectorArrayChunk`) already stores. - Rename the interface and both implementations, `GetArrayOffsets` to `GetStructElementOffsets`, the related segment methods, members, messages and test names, and `common/ArrayOffsets.{h,cpp}` to `common/StructElementOffsets.{h,cpp}`. - Document the abstraction on the interface. - The chunk-local `ArrayOffsets` alias in `ColumnarArrayChunk.h` is a different concept and stays unchanged. No behavior change. The mapping is an in-memory structure only: it does not appear in any proto, C API, config, metric or on-disk name. Tests: `all_tests` groups for struct offsets, growing, sealed, default-value translator, reopen, struct, MATCH, element filter, search group-by and array values pass locally (1134 passed; 2 TEXT LOB cases fail on local directory permissions, unrelated). Signed-off-by: Buqian Zheng <[email protected]> |
||
|
Jamesandxiaofanluan
|
ed1ea38f49 |
enhance: avoid redundant clearing in the RPC codec (#53993)
issue: #53990 The RPC release codec obtains pooled output and receive scratch buffers that gRPC clears before the codec overwrites their visible bytes. Use a shared codec-owned non-zeroing pool for pooled MarshalAppend and fragmented Unmarshal materialization. This preserves protobuf output lengths, single-buffer references, and MsgPins cleanup while leaving the gRPC default pool and dependency unchanged. The private pool retains the current default size classes (256B, 4KiB, 16KiB, 32KiB, 1MiB) and page-rounded oversized fallback policy. No configuration, wire-format, or borrowed-decoder change is introduced. New allocations still receive Go's zero initialization; the optimization applies to reused buffers. Validation: - Full resource tests with `-tags dynamic,test -gcflags='all=-N -l'` passed. - Targeted codec/coalescing race tests and resource vet passed. - Dirty-data tests verify exact wire lengths, hidden capacity tails, decoder output ownership, and held Marshal buffers. Error cases verify buffer returns and MsgPins release; tests also cover concurrent fastPB/official decoding, singleton references, and varying oversized payloads. Same-size-class, real-codec microbenchmarks on Apple M4 Pro / darwin-arm64, optimized compilation, `-cpu=8`, three runs (median): | Payload/path | Default pool | Codec-owned pool | Time reduction | | --- | ---: | ---: | ---: | | RetrieveResults varchar 64KiB Marshal | 6.852 us | 2.348 us | 65.7% | | RetrieveResults varchar 64KiB fragmented Unmarshal | 12.729 us | 7.215 us | 43.3% | | RetrieveResults varchar 4MiB Marshal | 117.774 us | 101.595 us | 13.7% | | RetrieveResults varchar 4MiB fragmented Unmarshal | 242.797 us | 209.939 us | 13.5% | | SearchResults blob 4MiB Marshal | 68.901 us | 51.957 us | 24.6% | | SearchResults blob 4MiB fragmented Unmarshal | 204.167 us | 183.050 us | 10.3% | Allocations/op were unchanged. Single-buffer Unmarshal has no systematic improvement because it retains the existing reference-only path. Reproduce from `pkg/` with optimized benchmark builds: ```sh go test -tags dynamic,test ./util/resource -run '^$' -bench BenchmarkReleaseCodecDirtyPool -benchmem -cpu=8 -count=3 -benchtime=100ms ``` These are local codec measurements, not end-to-end QPS/P99 results. In the motivating downstream CPU profile, directly removable pool clearing under this codec accounts for 7.65 of 250.55 sampled CPU seconds (3.05%); the deployment's exact gRPC version was not established from the profile. Production cache/RSS and full-workload throughput remain unmeasured. The private pool adds a separate cache group; it follows the existing retention policy rather than introducing a new memory limit. Fixes #53990. Backports: #53994 (2.6), #53995 (3.0). --------- Signed-off-by: xiaofanluan <[email protected]> Co-authored-by: xiaofanluan <[email protected]> |
||
|
congqixia
|
e335524ae6 |
enhance: pin sealed snapshot for result fill (#54042)
Attach the request-pinned SegmentReadSnapshot to SearchResult and reuse it across result fill instead of re-capturing the immutable published state once per output field. Result fill (FillTargetEntry / bulk_subscript / take) and both export paths (proto + Arrow record batch) now read through the pinned snapshot, so every filled field comes from the same generation as the search request. Growing segments and non-pinned paths keep the per-call fallback with identical semantics. The read facade gains a type-erased GetState() accessor; the segment rebinds it to the concrete published state via an alias shared_ptr (no ref-count churn), guarded by a dynamic_cast in ToPublishedState. Cross-segment reuse is rejected by a segment_ == this assertion in FillTargetEntry. Deeper column/index helpers (get_raw_data, PinIndex, HasFieldData) still self-capture; migrating them is a Phase 4 follow-up. Tests: pinned/unpinned parity for FillTargetEntry and the proto export; AttachSealedRequestLease now pins the snapshot so existing sealed export tests exercise the snapshot-backed branch. Related to #53247 --------- Signed-off-by: Congqi Xia <[email protected]> |
||
|
Jamesandxiaofanluan
|
addade7294 |
fix: use committed data and queryable coverage for index readiness (#53963)
Index readiness could skip committed imports because allocation expiry is not a data timestamp. It could also report Finished during ordinary flush-to-sort transitions when every invisible segment was excluded. Use GetEarliestTs() for timestamp-filtered readiness: commit timestamp when present, otherwise the earliest insert timestamp. Include unknown timestamps conservatively. Keep uncommitted imports excluded, and exclude only invisible segments created by compaction. Ordinary flushed segments waiting for sort remain pending; after sort replaces the input with a visible sorted output, readiness follows that output's requested index. For each requested IndexID, accept its own finished index or complete ancestor coverage in a recoverable query frontier. Share query metadata admission and the existing branch-specific frontier selector, so ancestors must be selected together as a usable replacement, have the same finished IndexID, and match collection/channel/partition. Reject known cycles; missing historical ancestors do not invalidate a selected segment's own finished index. Whole-schema versions do not gate coverage, and collections need not already be loaded. Persist existing SegmentIndex.IsDeleted markers before dropped-segment artifact deletion, after snapshot/manifest protection checks. If marking fails, keep all files. Index state and query index lookups ignore marked tasks; GC retains their records and file keys for idempotent retry and restart recovery, including file or catalog deletion failures. Keep collection-wide row-statistics scope and current-index version reporting. IndexedRows and PendingIndexRows continue to describe their existing accounting views; ancestor coverage does not relabel an unfinished physical output task as built. Deprecated last_expire_time retains its wire field and legacy allocation/recovery/cleanup usage. The handoff policy and GC eligibility policy are preserved. This does not change them to wait for all indexes or redesign the GC descendant graph. Validation: - Branch-specific native-free source-extraction validation passed all 178 leaf cases. It executes the actual collector, query admission/frontier selector, indexMeta accessors, dropped GC ordering, sort metadata producer, and all four status APIs with real branch protobufs and fixture storage/server adapters. This is not a full DataCoord package UT. - The pre-narrowing local control reproduced 21 failures in the same 178 tests, isolating ordinary invisible flushes and their sort transition. Twelve cases execute the actual sort metadata producer and verify the input becomes Dropped and the sorted output becomes visible before its index work controls readiness. - Separate query/GC validation against the previous published head reproduced 23 failures in 164 cases; this established the recoverability/frontier/deletion-marker regressions before the High narrowing. - Independent source and test review, producer tracing, gofmt, and git diff --check passed. The generated descriptor differs only in last_expire_time's deprecated option. - Full native package tests remain blocked by the previously confirmed local header/library mismatch (missing JSON stats executor functions and a NewPackedReaderWithProperties signature mismatch). Worker-side sorting/file generation and asynchronous end-to-end index execution were not run. The dedicated manifest-tombstone hydration test is checked in but was not executed locally. Addresses the ordinary flush/sort High finding: https://github.com/milvus-io/milvus/pull/53963#issuecomment-6060052156 Targets master. 3.0 backport: #54037. Fixes #53920 Signed-off-by: xiaofanluan <[email protected]> Co-authored-by: xiaofanluan <[email protected]> |
||
|
yihao.daiandClaude Opus 5.5
|
2b6d94c82d |
fix: take the global WAL lock for force promote on the control channel pchannel (#54018)
issue: #54016 ## Summary `handleForcePromote` built its broadcast channel list by hand. It put the control channel in place of its pchannel, but it did not mark the message as pchannel-level. On the pchannel that hosts the control channel, the lock interceptor therefore took only the global read lock for the `AlterReplicateConfig` copy, not the global write lock. The replicate interceptor expects the global write lock around `SwitchReplicateMode`, the append and `RollbackAllInFlightTransactions`. This PR builds the force-promote message with `WithClusterLevelBroadcast`, the same as the normal `UpdateReplicateConfiguration` path. Each copy is now pchannel-level and takes the global write lock on its pchannel. ## Changes - `internal/streamingcoord/server/service/assignment.go`: use `WithClusterLevelBroadcast(message.ClusterChannels{...}, OptBuildBroadcastAckSyncUp())` and remove the hand-written channel swap. - `internal/streamingcoord/server/service/assignment_test.go`: `TestForcePromoteSuccess` captures the broadcast message and checks that the channel list is the control channel and that each split copy is pchannel-level. ## Behavior difference `WithClusterLevelBroadcast` panics if the control channel is not on any of the listed pchannels, or if the list has a name that is not a pchannel. The old code did not check this. The normal `UpdateReplicateConfiguration` path already depends on the same condition. ## Verification - [x] `go test ./internal/streamingcoord/server/service/...` pass - [x] The new assertion in `TestForcePromoteSuccess` fails without the fix and passes with it - [x] `make static-check` pass - [x] No other hand-written control-channel swap in the repo (`IsOnPhysicalChannel(` is used only in `WithClusterLevelBroadcast`) - [ ] Runtime force promote with concurrent replication is not tested. This PR makes sure that the lock is correct. The effect of the old lock gap on data is not reproduced. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Signed-off-by: bigsheeper <[email protected]> Co-authored-by: Claude Opus 5.5 <[email protected]> |
||
|
junjiejiangjjj
|
8cfeb14750 |
test: Enable hybrid rerank chain test (#54041)
issue: https://github.com/milvus-io/milvus/issues/46565 Signed-off-by: junjie.jiang <[email protected]> |
||
|
XuanYang-cn
|
2ca0398246 |
fix: batch clustering compaction output across input records (#53678)
Clustering compaction converts each input row to a Value and flushes each bucket at input Record boundaries. When rows are spread across buckets, this produces many small Arrow Records and amplifies allocation and retained-object overhead in packed writers. Accumulate surviving rows in a RecordBuilder owned by each bucket across input boundaries. Batch at half the binlog budget using allocated Arrow buffer bytes, including null slots, validity/offset buffers and spare capacity; use the same accounting for pending-bucket pressure. Keep payload-size accounting for existing sorting callers and halve the V2 writer buffer option. Preserve filtering, routing, nullable/default values and output statistics, including the Double and Timestamptz default fixes required by the RecordBuilder path. On an append or writer failure, discard the bucket batch under the bucket lock, reject further writes to that bucket, cancel sibling mapping workers and join them before cleanup while preserving the original error. Reject malformed timestamp column types with a typed internal error instead of panicking. Always close writers after flush failures and during task cleanup. Keep batching for V1/V2/V3; project records onto the writer schema, normalize all-null TEXT columns to its binary/UTF8 representation, and return an error for incompatible non-null columns before constructing the Arrow Record. V3 clustering batches use the writer’s physical field names, including ExternalField overrides, so matching batches reuse their Arrow Record. The previous projection path did not copy column payloads; this only removes a record wrapper and column retain/release work per matching batch. Validation: Targeted failing tests reproduced a panic for malformed or absent timestamp columns, continued writes after an injected writer failure, and the V3 builder/writer schema mismatch. The focused internal/storage and internal/datanode/compactor regressions pass after the fixes, including TEXT normalization, V1/V2 persisted readback, the V3 identity fast path, lazy-writer close, and concurrent cancellation. A controlled rebuild of the Milvus native libraries and Go binary passed against the pinned native storage dependency. The full repository static check passed. Memory scope: RecordBuilder.GetMemorySize() counts only the pending batch and resets after Build; V2/V3 writer-retained Arrow/native memory remains outside that counter. Their GetBufferUncompressed() reports the configured multipart buffer size rather than actual retained bytes, and FlushChunk does not force native release. As a result, getBufferTotalUsedMemorySize() and flushLargestBuffers() can undercount or mis-rank writer memory; the high-watermark is not a bound on process RSS. A focused V2 test forces an 8 MiB writer buffer to verify FlushChunk retention under that fixture, not production peak memory. Accurate native-writer accounting and release remain separate follow-up work; native writer implementation and file layout are unchanged. issue: #53432 --------- Signed-off-by: yangxuan <[email protected]> |
||
|
|
5f0342f680 |
fix: restore legacy segment schema versions during WAL migration (#53966)
Legacy 2.6 segment metadata can decode an absent `schema_version` as 0 even when the segment was written after fields were added. Restoring the initial schema in that case can disagree with persisted column-group `ChildFields`, including field 121 reported in #53919. An unchecked field lookup then selects column 0, and the StorageV2 null-count lookup can panic after writing. Resolve these versions during legacy recovery migration: use the original allocation timetick, or the first-data timestamp already supplied by reconstruction, to select retained schema history; fall back to the latest schema when that history is unavailable. Persist the selected encoding version before publishing the V2 recovery checkpoint. Known nonzero versions are preserved; if the timestamp lookup resolves V0, the stored version remains 0. - Restore growing segments using their persisted encoding version while retaining collection and partition lifetime filters. - Record the timestamp-selected version when replaying an old CreateSegment message with a missing version, so the next native restart keeps it. - Validate every persisted column-group batch before creating the writer: reject unknown fields, duplicate fields, and incorrect column mappings. Older groups may omit nullable/defaulted fields added later. Validation: ```sh go test -v -tags dynamic,test -gcflags='all=-N -l' -count=1 -timeout 10m \ ./internal/streamingnode/server/wal/recovery \ ./internal/streamingnode/server/wal/vchannel \ ./internal/streamingnode/server/wal/vchannel/segment ``` - 187 top-level tests / 267 including subtests passed, with no failures or skips. - A regression test performs legacy migration, a protobuf round-trip of the saved metadata, restoration from a Magic=2 checkpoint, WAL-tail replay, and actual StorageV2 writes/reads; field 121 reads back as `7, 8, NULL`. - A latest-schema case adds nullable field 122 and reads the old physical groups with the complete latest schema, verifying NULL filling. Incompatible groups fail before the object write callback. - The non-nullable default-field case verifies preservation of the old physical columns; it does not validate materialization of the new default value. Native IO tests reuse existing C++ libraries rather than a fresh native build. This change applies to legacy migration and subsequent WAL replay. It does not rewrite an already-published Magic=2 snapshot containing an incorrect zero version. The original CI metadata snapshot was not retained, so this verifies the compatibility mechanism rather than claiming an exact reproduction of the original rolling-upgrade run. Fixes #53919 Signed-off-by: xiaofanluan <[email protected]> Co-authored-by: xiaofanluan <[email protected]> Co-authored-by: Codex <[email protected]> |
||
|
Buqian Zheng
|
ea577911a1 |
test: isolate local storage fixtures and fix array bitmap cleanup (#53986)
### What this PR does Fix local filesystem ownership in C++ test fixtures: - Keep the original writable paths and matching cleanup for sealed/growing TEXT LOB index decoding. - Use complete `TestLocalPath` keys for all ten storage-v2 SkipIndexPr51441 filesystem fixtures, including nine additional fixtures found during the full UT run. Relative keys resolved under / and caused read-only-root errors followed by Arrow `ValueOrDie()` aborts. - Give each parameterized ArrayBitmap fixture an exclusively created directory. Release the loaded index/reader and filesystem handles in `TearDown()` before removing only that fixture's directory. The original ArrayBitmap destructor removed the shared `TestLocalPath` while `index_` was still alive. A controlled sibling marker was deleted even though the selected tests passed; the fixed fixture preserves that marker. No production code or assertions change. ### Validation Validated the exact final three-file patch at base HEAD `0f34d4d512d0a08e7ad297e0d4d5396943420048` in the verified milvus-compile-base snapshot. Tests used repository `scripts/setenv.sh`, jemalloc, one fresh writable `MILVUS_TEST_ROOT_DIR`, and `MILVUS_CONF_LOCALSTORAGE_PATH` under that root. - Ordinary default `bash /workspace/milvus-dev/build-server.sh` incremental build: exit 0, 121.65 seconds. Only the two newly modified test translation units compiled before relinking `all_tests`. No inline offline/refresh parameters, manual cache clearing or timestamp normalization. - The ten previously aborted registrations (nine SkipIndexPr51441 tests and `ArrayBitmapE2ECheckV1/ArrayBitmapIndexTestV1/2.CountFuncTest`) each passed 20 focused repetitions: **200/200** observations. All ten also passed in the complete suite. - ArrayBitmap selection: **24 tests × 30 repetitions = 720/720** passes, both before and after the fixture changes. - Exact original 46-test predecessor/target order: **46 × 100 = 4,600/4,600** passes, both before and after. - Controlled directory ownership probe: sibling marker deleted before; preserved after. Both probe subprocesses returned 0. - Fresh complete enumeration and XML/log reconciliation preserved duplicate registrations. Compared with the previous complete coverage, exactly ten statuses changed from aborted to passed; no other registration changed status. - `git diff --check` passed. The original +7/-7 patch remains intact. The appended commit changes two test files +27/-14; the combined PR changes three test files +34/-21. | Complete C++ UT executable | Registered instances | Passed | Failed | Runtime skipped | Disabled | Enabled unrun | Aborted | Exit | Time | | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | | all_tests | 9,305 | 9,262 | 7 | 12 | 24 | 0 | 0 | 1 | 992.66 s | | bitset_test | 332 | 332 | 0 | 0 | 0 | 0 | 0 | 0 | 0.96 s | | json_stats_test | 118 | 118 | 0 | 0 | 0 | 0 | 0 | 0 | 6.78 s | | test_json_uint64 | 17 | 17 | 0 | 0 | 0 | 0 | 0 | 0 | 0.10 s | | Total | 9,772 | 9,729 | 7 | 12 | 24 | 0 | 0 | — | — | ### Environment failures and limits **The complete run is not all green.** All seven failures are `AzureChunkManagerTest`: `BucketPositive`, `BucketNegtive`, `ObjectExist`, `WritePositive`, `ReadPositive`, `RemovePositive`, and `ListWithPrefixPositive`. These fixtures use the local Azurite blob endpoint `127.0.0.1:10000`; no listener was present. The recorded failures report requests cancelled by context. Azure integration behavior remains unverified and needs a rerun in an environment with Azurite. No credentials or network settings were added or changed. The historical ArrayBitmap `Directory not empty` SIGABRT **did not reproduce stably** in the bounded pre-change ordering/pressure runs. The shared-root ownership violation was directly demonstrated, and the corrected ownership/release order passed the post-change stress and complete run. These results do not prove that every possible underlying race has been eliminated. The 12 runtime skips and 24 existing disabled registrations remain separately accounted for and are not counted as passes. Complete enumeration, commands, UTC timestamps, exit codes, skip/failure reasons and logs are retained locally. Go/runtime suites and a coverage-instrumented rebuild were not run. --------- Signed-off-by: Buqian Zheng <[email protected]> |
||
|
marcelo-cjl
|
33a10e9ded |
enhance: batch vector validity checks for null expressions (#53981)
## Summary - consume Knowhere IdMap batch validity APIs for vector null predicates - add a contiguous segment-level validity path so sequential scans no longer materialize offset arrays - batch arbitrary-offset validity checks through one IdMap snapshot - preserve field-data fallbacks and non-nullable vector semantics - cover range writes, bitmap-view offsets, arbitrary offsets, and out-of-range rows Related to #52239 Knowhere dependency: zilliztech/knowhere#1858 (merged) ## Test plan - [x] `git diff upstream/master..HEAD --check` - [x] `git clang-format --binary clang-format-15 --diff HEAD^` - [x] `bash scripts/check_segcore_error_boundaries.sh` - [x] Managed Release+ASan integration build completed on the pre-rebase base with Knowhere `16e873b` - [ ] Full build and unit tests on the rebased head are delegated to PR CI; the previous local all-tests attempt was stopped after its unit-test build profile invalidated the production object cache and exhausted the shared host Signed-off-by: marcelo-cjl <[email protected]> |
||
|
jiaqizho
|
cab9d65256 |
enhance: configure Talon read routing for remote filesystems (#53817)
issue: #53875 Add common.storage.talon settings and propagate them through the existing StorageConfig and filesystem properties pipeline. Support routing all reads or only small reads through Talon, with a default small-read threshold of 512 KiB. Talon remains disabled by default; External Tables require enableForExternalTable to opt in. Also bump storage to e5ac4d8. --------- Signed-off-by: jiaqizho <[email protected]> |
||
|
congqixia
|
4544855911 |
enhance: add atomic batch publication primitives (#53892)
Related to #53321 ## What - `dataview.Manager` gains `PublishChange`: builds the post-batch-publish DataView snapshot under the Collection lock, symmetric with `PrepareFlush`. New members are added and superseded parents are removed in one snapshot, advancing only `compact_version`; an idempotent replay returns the current snapshot without burning a version, and the commit/abort callbacks match the flush contract. - `meta.UpdateSegmentsInfoAndChangeGroupsAndDataView` composes SegmentMeta operators, group actions, and the DataView snapshot in ONE catalog txn, so a group's members flip visible, its superseded parents retire, the group record reaches COMMITTED, and the snapshot (compact_version +1) land together. The DataView action is commit-marked alongside the terminal group actions, so the over-limit fallback lands both in the final guarded txn. - The dataview-less `UpdateSegmentsInfoAndChangeGroups` is kept as a delegate so the same publish core works without a DataView manager (3.0 cherry-pick, dual-mode per the implementation plan). ## Why Batch segment changes (compaction/import/copy) must become visible atomically. Staging requires one committed DataView snapshot per batch publish; otherwise a half-published batch would let queries observe a subset of the members (see docs/design-docs/design_docs/20260909-datacoord-segment-change-staging-design.md, sections 5.4 and 5.5). ## Test - `go test -tags dynamic,test -gcflags="all=-N -l" -count=1 ./internal/dataview/...` - `go test -tags dynamic,test -gcflags="all=-N -l" -count=1 ./internal/datacoord/ -run TestMeta_UpdateSegmentsInfoAndChangeGroups` - `go test -tags dynamic,test -gcflags="all=-N -l" -count=1 ./internal/metastore/kv/datacoord/...` --------- Signed-off-by: Congqi Xia <[email protected]> |
||
|
aoiasdandClaude Opus 4.6
|
79a0f6b62b |
feat: support RLS array tags and NOT expressions (#53944)
relate: #50263 design doc: docs/design-docs/design_docs/20250610-rls_design.md design doc PR: #53173 ## Summary - Support homogeneous one-dimensional array tags, promoting mixed integer/double arrays to double, and unary NOT over the existing simple-predicate subset. - Keep missing/incompatible tags and null/type-invalid data fail-closed; bound tag metadata and materialized templates and account for aggregate RLS HybridSearch plan size. - Capture a locally coherent policy/tag snapshot per request, shared by upsert USING/CHECK, and retry acquisition if policies change during tag loading. - Reuse RLS expression-length and field-dependency helpers, move test-only helpers out of production, and fail fast on duplicate policy names during recovery. ## Verification - Static call-path and invalidation-interleaving review; gofumpt and git diff --check passed. - Added numeric promotion, type/null integrity, resource-bound, recovery, and policy-change-during-tag-load regression tests. - Local build/tests have not been run for this revision; PR CI will provide execution results. ## Scope Authorization remains request-snapshot-based, not mutation-CAS or distributed-linearizable. Previously deferred dynamic enablement, process-wide cache budgeting, and pagination are unchanged. --------- Signed-off-by: aoiasd <[email protected]> Co-authored-by: Claude Opus 4.6 <[email protected]> |
||
|
yihao.daiandClaude Opus 5.5
|
8f3784fbba |
doc: add header vs body placement rules for WAL messages (#53823)
## What this PR does Adds a "Header vs Body" section to `docs/agent_guides/streaming-system/message/message.md`. The section gives five rules for where a new message field goes: 1. Put a field in the header only if the WAL infrastructure (interceptors, RecoveryStorage, flusher, CDC, broadcast ack callback dispatch) must read or write it without a body decode. 2. The header size must not grow with user data volume. 3. Put secrets and user data only in the body. Cipher encrypts only the payload, so the header stays plaintext in the WAL backend and in logs. 4. Put a field that the WAL writes during append in the header (`OverwriteHeader` is cheap, `OverwriteBody` marshals and encrypts again). 5. Put metadata that applies to many message types in a message property, not in a specialized header. Each rule cites an example from the current code. The header definition comment in `pkg/proto/messages.proto` now points to this section. The comment is detached, so generated code does not change. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Signed-off-by: bigsheeper <[email protected]> Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]> |
||
|
zhenshan.cao
|
6e6b5c462d |
fix: treat an HTTP 200 DeleteObject response as a successful delete (#53977)
issue: #53976 ## Problem Some S3-compatible object stores answer `DeleteObject` with HTTP 200 instead of the 204 the S3 API specifies. minio-go reports every non-204 response as an error (`api-remove.go`), and `MinioObjectStorage.RemoveObject` passes that error through `mapObjectStorageError`, where nothing matches it, so it surfaces as `ErrIoFailed`. The object is gone on the store; only the response code is non-standard. On such a store DataCoord GC can never finish a segment (`GC segment remove logs failed ... IO failed[key=...]: No message` on every file), and the dropped-segment count only grows. ## Change `RemoveObject` treats an error whose `ErrorResponse.StatusCode` is 200 as a successful delete and returns nil. Everything else still goes through `mapObjectStorageError`, so a 404 keeps surfacing as `ErrIoKeyNotFound` for the callers that handle it (GC ignores it as removed by a previous run). A conforming store never answers 200 here, so S3, MinIO, GCS and the other providers are unaffected. The same handling is on `hotfix-2.5.13` (#47507) and in `milvus-backup` main (zilliztech/milvus-backup#931); the earlier master PR #47508 was closed by the stale bot without review. This PR carries only the 200 case: #47507 also returned nil on 404, which would hide "already gone" from the callers that currently log and count it. ## Test `TestIsDeleteSucceededWithOK` covers nil, a 200 response, a 404 `NoSuchKey` response (still mapped to `ErrIoKeyNotFound`), and a plain error. The existing `TestMinioObjectStorage` suite passes against a local MinIO. Signed-off-by: zhenshan.cao <[email protected]> |
||
|
sijie-ni-0214
|
0f34d4d512 |
enhance: Reduce verbose QueryNode loading logs (#53439)
issue: #53434 ## What this PR does - Demotes routine per-segment, index, bloom-filter, manifest, and storage-translator messages from Info to Debug. - Removes redundant collection schema and index metadata dumps from normal logs. - Uses request contexts for affected Go log calls when a context is available. - Skips log-only field summary and manifest/stat preparation when Debug logging is disabled. The change preserves failure logs and higher-level segment loading lifecycle messages. ## Test Plan - [x] `make verifiers` - [x] QueryNode segment bloom-filter loading tests - [x] QueryNode segments package compilation with `dynamic,test` build tags Signed-off-by: sijie-ni-0214 <[email protected]> |
||
|
Kuroda Kayn
|
6140f5f55c |
enhance: optimize sorted scalar membership lookups (#53901)
issue: #53853 design doc: https://github.com/KurodaKayn/milvus/blob/enhance/batch-sorted-int64-membership/docs/design-docs/design_docs/20260929-sorted-int64-membership.md ## Summary `STL_SORT` `IN` / `NOT IN` previously performed one independent lookup per query term. Duplicate terms repeated index searches and bitmap writes. This PR batches membership lookup by sorting and deduplicating query terms, then reusing lookup progress through the sorted index. Supported types: - Int8/Int16/Int32/Int64 - Bool - Float/Double - String/VarChar Existing NULL, NaN, signed-zero, infinity, row-offset, and index-format semantics remain unchanged. ## Benchmark Standalone Int64 benchmark against the previous binary-search path. Includes query preparation and bitmap writes. | Query terms | Average speedup | | ---: | ---: | | 1 | 1.03x | | 8 | 2.27x | | 64 | 3.87x | | 128 | 4.86x | | 512 | 9.24x | | 4,096 | 33.74x | Small queries and small indexes can regress. Per-type and small-list strategy tuning remains follow-up work. ## Validation - Scan-oracle checks for all supported scalar types - Duplicate, empty, missing, extreme, and NULL queries - Float/Double NaN, signed-zero, and infinity cases - String embedded-NUL, UTF-8, long-prefix, and repeated-value cases - Heap/mmap, legacy/packed reload, sync/async load paths - Query input immutability checks --------- Signed-off-by: Kuroda Kayn <[email protected]> |
||
|
Alexander Guzhva
|
243ebd872b |
enhance: bump simdjson from v3.12.2 to v5.0.2 (#53941)
issue #53940 The new version is slightly (up to several percents) faster on benchmarks. Tested on an ARM machine Signed-off-by: Alexandr Guzhva <[email protected]> |
||
|
Jamesandxiaofanluan
|
d1cb277f0c |
test: wait for compactable segments in struct partial-update lifecycle (#53964)
issue: #53962 The Struct Array partial-update lifecycle test can request manual compaction while its freshly flushed segments are still being sorted. DataCoord then returns the valid no-plan result `-1`, and the immediate positive-ID assertion fails before the compaction/reload checks run. Wait for vector-index readiness, then retry only `-1` against a monotonic 120-second deadline, passing the remaining budget to each compact RPC. Require an accepted compaction to complete and check the baseline both immediately afterward and after release/load. RPC failures, unexpected IDs, timeouts, and data mismatches remain failures. This follows the existing Struct nullable lifecycle test's approach from #52108. The observed failure is in [PR #53941's E2E run](https://jenkins-milvus-ci.milvus.io/job/MILVUS-CI-V2-PR-PIPELINES/job/milvus-e2e-pipeline-gcp/61102/console); this change does not modify server compaction behavior. Validation: - Ruff lint and format checks, Python syntax compilation, and `git diff --check`. - Executed the changed statements extracted from the actual file with simulated time/RPC responses: retry then success, persistent no-plan timeout, invalid IDs, RPC failure, index/compaction wait failures, and both post-compaction and post-reload data mismatches. - Live E2E has not been run against this fix locally; CI validation is pending. The separately requested rerun of #53941 uses its original head and does not validate this patch. Signed-off-by: xiaofanluan <[email protected]> Co-authored-by: xiaofanluan <[email protected]> |
||
|
Alexander Guzhva
|
81f3209ceb |
fix: fix a Regex-related ASAN violation on ARM machine (#53943)
related to #53942 The baseline code: ```C++ bool is_special(char c) { // initial special_bytes_bitmap only once. static std::once_flag _initialized; static std::string special_bytes(R"(\.+*?()|[]{}^$)"); static std::vector<bool> special_bytes_bitmap; std::call_once(_initialized, []() -> void { special_bytes_bitmap.resize(256); for (char b : special_bytes) { special_bytes_bitmap[b + 128] = true; } }); return special_bytes_bitmap[c + 128]; } ``` The `c + 128` triggers a problem, because of goes outside of `[0..255]` buffer. The root reason is that `char` is an unsigned type on ARM. The baseline code leads to an overflow. Signed-off-by: Alexandr Guzhva <[email protected]> |
||
|
2sumtechandClaude Fable 5.1
|
b2a20c810b |
fix: send VoyageAI embedding truncation=false instead of dropping it (#53867)
/kind bug issue: #53866 ### What - `models/voyageai/voyageai_client.go`: remove `omitempty` from `EmbeddingRequest.Truncation`, so the value is always sent. - `embedding/voyageai_embedding_provider.go`: default `truncate` to `true` instead of `false`. ### Why VoyageAI treats a missing `truncation` as `true`. With `omitempty` on a `bool`, an explicit `truncation: "false"` was never sent, so VoyageAI truncated over-length text and Milvus stored/searched with a vector of the truncated text instead of returning the error the user asked for. Both VoyageAI paths dropped an explicit `truncation=false`. On the embedding path, the request struct tagged `Truncation` with `omitempty`, so false was omitted from the JSON body. On the rerank path, the provider parsed `truncation` into its params but passed `nil` to the client, so the value never reached the request. In both cases VoyageAI then applied its default (true) and silently truncated over-length input. This PR fixes both: the embedding request always sends the field (the provider defaults to true when the param is unset, preserving existing behavior), and the rerank provider forwards its params like the other rerank providers. Tests capture the outgoing request body for both paths. The two changes go together. Dropping `omitempty` alone would start sending the provider's old default `false` for every function that never set the param, so those collections would suddenly reject long text. Setting the default to `true` keeps them on the behavior they get today, which also matches the VoyageAI API default and the Milvus VoyageAI docs ("Defaults to true"). | function param | sent before | sent after | VoyageAI behavior after | | --- | --- | --- | --- | | unset | (absent → true) | `true` | truncate (unchanged) | | `"true"` | `true` | `true` | truncate (unchanged) | | `"false"` | (absent → true) | `false` | error on over-length input (**fixed**) | ### How tested New table-driven test `TestEmbeddingSendsTruncation` decodes the request body at an `httptest` stub and asserts `truncation` is present with the caller's value. <details> <summary>Before the fix — false case fails, field absent from the body</summary> ``` $ go test -tags dynamic,test -gcflags="all=-N -l" -count=1 -v \ -run TestEmbeddingSendsTruncation ./internal/util/function/models/voyageai/ === RUN TestEmbeddingSendsTruncation === RUN TestEmbeddingSendsTruncation/truncation_true === RUN TestEmbeddingSendsTruncation/truncation_false voyageai_client_test.go:235: Error Trace: internal/util/function/models/voyageai/voyageai_client_test.go:235 Error: Should be true Test: TestEmbeddingSendsTruncation/truncation_false Messages: request body must contain truncation, got map[input:[sentence] input_type:query model:voyage-3 output_dtype:float] voyageai_client_test.go:236: Error Trace: internal/util/function/models/voyageai/voyageai_client_test.go:236 Error: Not equal: expected: bool(false) actual : <nil>(<nil>) Test: TestEmbeddingSendsTruncation/truncation_false --- FAIL: TestEmbeddingSendsTruncation (0.00s) --- PASS: TestEmbeddingSendsTruncation/truncation_true (0.00s) --- FAIL: TestEmbeddingSendsTruncation/truncation_false (0.00s) FAIL FAIL github.com/milvus-io/milvus/internal/util/function/models/voyageai 0.608s FAIL ``` </details> <details> <summary>After the fix, plus package tree, -race, gofmt, vet</summary> ``` $ go test -tags dynamic,test -gcflags="all=-N -l" -count=1 -v \ -run TestEmbeddingSendsTruncation ./internal/util/function/models/voyageai/ === RUN TestEmbeddingSendsTruncation === RUN TestEmbeddingSendsTruncation/truncation_true === RUN TestEmbeddingSendsTruncation/truncation_false --- PASS: TestEmbeddingSendsTruncation (0.00s) --- PASS: TestEmbeddingSendsTruncation/truncation_true (0.00s) --- PASS: TestEmbeddingSendsTruncation/truncation_false (0.00s) PASS ok github.com/milvus-io/milvus/internal/util/function/models/voyageai 0.653s $ go test -tags dynamic,test -gcflags="all=-N -l" -count=1 \ ./internal/util/function/models/... ./internal/util/function/rerank/... ok github.com/milvus-io/milvus/internal/util/function/models 5.732s ok github.com/milvus-io/milvus/internal/util/function/models/ali 7.636s ok github.com/milvus-io/milvus/internal/util/function/models/cohere 10.580s ? github.com/milvus-io/milvus/internal/util/function/models/gemini [no test files] ok github.com/milvus-io/milvus/internal/util/function/models/huggingface 2.953s ok github.com/milvus-io/milvus/internal/util/function/models/openai 24.660s ok github.com/milvus-io/milvus/internal/util/function/models/siliconflow 9.419s ok github.com/milvus-io/milvus/internal/util/function/models/tei 7.416s ok github.com/milvus-io/milvus/internal/util/function/models/vertexai 8.054s ok github.com/milvus-io/milvus/internal/util/function/models/vllm 12.400s ok github.com/milvus-io/milvus/internal/util/function/models/voyageai 12.575s ok github.com/milvus-io/milvus/internal/util/function/models/zilliz 2.738s ok github.com/milvus-io/milvus/internal/util/function/rerank 9.310s $ go test -race -tags dynamic,test -gcflags="all=-N -l" -count=1 \ ./internal/util/function/models/voyageai/ ./internal/util/function/rerank/ ok github.com/milvus-io/milvus/internal/util/function/models/voyageai 9.004s ok github.com/milvus-io/milvus/internal/util/function/rerank 11.292s $ gofmt -l internal/util/function/models/voyageai/ internal/util/function/embedding/voyageai_embedding_provider.go (no output) $ go vet -tags dynamic,test ./internal/util/function/models/voyageai/ (exit 0) ``` </details> Not verified locally: `internal/util/function/embedding` needs the C++ core (`internal/util/cgo` via pkg-config) and does not build on the machine used, so the one-line default change in `voyageai_embedding_provider.go` is left to CI (its existing tests only parse `"true"` / `"false"` / invalid values). No call was made against the live VoyageAI endpoint; the absent-means-true default is taken from the VoyageAI API reference. Disclosure: prepared with AI assistance (Claude Code); I reviewed the change and take responsibility for it. 🤖 Generated with [Claude Code](https://claude.com/claude-code) ```` --- --------- Signed-off-by: 2sumtech <[email protected]> Signed-off-by: xiaofanluan <[email protected]> Co-authored-by: Claude Fable 5.1 <[email protected]> |
||
|
2sumtechandClaude Fable 5.1
|
8247e529e7 |
fix: return the failure a zilliz model service reports in its response status (#53872)
/kind bug issue: #53871 ### What `internal/util/function/models/zilliz/zilliz_client.go`: `Embedding`, `Rerank` and `Highlight` now check the `Status` of the model service response through one helper, `checkResponseStatus`, and return `merr.WrapErrFunctionFailedMsg("model service returned error, code: %d, msg: %s")` when the code is non-zero. A nil `Status` is still treated as success. ### Why Each model service response (`TextEmbeddingResponse`, `TextRerankResponse`, `HighlightResponse`) carries a `Status` next to the payload, and the client only checked the gRPC error. A failure reported in `Status` on an RPC that returned normally came back as an empty successful result, and the service's code and message were dropped. Downstream this turned into a different, less useful error: rerank reports `ServiceInternal: rerank service returned 0 scores for N docs`, highlight reports a size mismatch, and the embedding path passes the empty result on (the embedding function's `Check` indexes `embds[0]`). `FunctionFailed` is the class the HTTP providers already use for a non-transient failure reported by the model service (`models/common.go:send`). The status codes of the model service are not defined in this repository, so the change does not try to map any of them to a retriable class. ### How tested New table-driven `TestZillizClient_ResponseStatus`: an in-process gRPC server (bufconn, same setup as the existing tests) answers each of the three methods with a failed, a successful, and a missing `Status`. <details> <summary>Before the fix — the three failed-status cases return no error</summary> ``` $ go test -tags dynamic,test -gcflags="all=-N -l" -count=1 -v \ -run TestZillizClient_ResponseStatus ./internal/util/function/models/zilliz/ zilliz_client_test.go:898: Error: An error is expected but got nil. Test: TestZillizClient_ResponseStatus/embedding_failed_status ... --- FAIL: TestZillizClient_ResponseStatus (0.01s) --- FAIL: TestZillizClient_ResponseStatus/embedding_failed_status (0.00s) --- PASS: TestZillizClient_ResponseStatus/embedding_ok_status (0.00s) --- PASS: TestZillizClient_ResponseStatus/embedding_nil_status (0.00s) --- FAIL: TestZillizClient_ResponseStatus/rerank_failed_status (0.00s) --- PASS: TestZillizClient_ResponseStatus/rerank_ok_status (0.00s) --- PASS: TestZillizClient_ResponseStatus/rerank_nil_status (0.00s) --- FAIL: TestZillizClient_ResponseStatus/highlight_failed_status (0.00s) --- PASS: TestZillizClient_ResponseStatus/highlight_ok_status (0.00s) --- PASS: TestZillizClient_ResponseStatus/highlight_nil_status (0.00s) FAIL FAIL github.com/milvus-io/milvus/internal/util/function/models/zilliz 0.503s ``` </details> <details> <summary>After the fix — all cases pass; package suites, race, gofmt, vet</summary> ``` $ go test -tags dynamic,test -gcflags="all=-N -l" -count=1 -v \ -run TestZillizClient_ResponseStatus ./internal/util/function/models/zilliz/ --- PASS: TestZillizClient_ResponseStatus (0.01s) --- PASS: TestZillizClient_ResponseStatus/embedding_failed_status (0.00s) --- PASS: TestZillizClient_ResponseStatus/embedding_ok_status (0.00s) --- PASS: TestZillizClient_ResponseStatus/embedding_nil_status (0.00s) --- PASS: TestZillizClient_ResponseStatus/rerank_failed_status (0.00s) --- PASS: TestZillizClient_ResponseStatus/rerank_ok_status (0.00s) --- PASS: TestZillizClient_ResponseStatus/rerank_nil_status (0.00s) --- PASS: TestZillizClient_ResponseStatus/highlight_failed_status (0.00s) --- PASS: TestZillizClient_ResponseStatus/highlight_ok_status (0.00s) --- PASS: TestZillizClient_ResponseStatus/highlight_nil_status (0.00s) PASS ok github.com/milvus-io/milvus/internal/util/function/models/zilliz 0.561s $ go test -tags dynamic,test -gcflags="all=-N -l" -count=1 \ ./internal/util/function/models/zilliz/ ./internal/util/function/rerank/ ./internal/util/function/highlight/ ok github.com/milvus-io/milvus/internal/util/function/models/zilliz 0.470s ok github.com/milvus-io/milvus/internal/util/function/rerank 9.230s ok github.com/milvus-io/milvus/internal/util/function/highlight 6.397s $ go test -race -tags dynamic,test -gcflags="all=-N -l" -count=1 \ ./internal/util/function/models/zilliz/ ./internal/util/function/rerank/ ./internal/util/function/highlight/ ok github.com/milvus-io/milvus/internal/util/function/models/zilliz 1.674s ok github.com/milvus-io/milvus/internal/util/function/rerank 10.644s ok github.com/milvus-io/milvus/internal/util/function/highlight 7.932s $ gofmt -l internal/util/function/models/zilliz/ (no output) $ go vet -tags dynamic,test ./internal/util/function/models/zilliz/ (exit 0) ``` </details> Not verified locally: `internal/util/function/embedding` and `internal/util/function/chain/expr` need the C++ core and do not build on the machine used, so the downstream effects listed under Why are from reading the code, not from a run; they consume this client only through the three methods changed here. No call was made against a real zilliz model service. Disclosure: prepared with AI assistance (Claude Code); I reviewed the change and take responsibility for it. 🤖 Generated with [Claude Code](https://claude.com/claude-code) ```` --- --------- Signed-off-by: 2sumtech <[email protected]> Co-authored-by: Claude Fable 5.1 <[email protected]> |
||
|
Zhen Ye
|
0d166dc4df |
fix: preserve GC checkpoint protection on channel lookup errors (#53924)
issue: #53916 - DataCoord GC: check the channel checkpoint first, and retain dropped segments for retry when a required channel lookup fails - Metastore: distinguish channel lookup errors from missing markers while preserving successful-read behavior Signed-off-by: chyezh <[email protected]> |
||
|
congqixia
|
2c1cef7cff |
enhance: pin sealed read-snapshot view reads through frozen column (#53913)
Related to #53247 Perchunk chunk_data/chunk_view reads in the expression and chunk-reader hot loop still call segment accessors that re-capture the immutable PublishedSegmentState on every access. Phase 1 routed the metadata hot loop (chunk_size, num_rows_until_chunk, get_chunk_by_offset, num_chunk_data, get_row_count) through the request-scoped SegmentReadSnapshot, but the actual data and view reads kept paying one atomic_load plus two ref-count RMWs per chunk on sealed segments. Route the view family through the already-pinned column obtained from GetDataScanResources so every data read derives from the same frozen generation as the chunk boundaries, with zero atomics and zero ref-count churn: - SegmentChunkReader::ChunkData<T> / ChunkStringView - SegmentExpr::GetChunkData / GetChunkView / GetChunkViewsByOffsets / GetBatchViews / GetViewsByOffsets (including the Json conversion branch) Migrate the sealed hot-loop call sites: SegmentChunkReader.cpp, Expr.h, CompareExpr.h, UnaryExpr.cpp, and the group-by path (SearchGroupByOperator + StrictGroupFilteredSearch). PhySearchGroupByNode captures the request snapshot once in its constructor and threads it into SealedDataGetter, mirroring how segment_ and search_info_ are bound. Growing segments and non-pinned paths keep the existing per-call segment access through the same fallback helpers, so behavior is bit-for-bit identical; sealed segments now read the view family from the pinned snapshot with no per-chunk capture. Verified with the segcore unittest binary: SegmentChunkReader, group-by, sealed read-snapshot, expression, and chunked-sealed suites all pass. --------- Signed-off-by: Congqi Xia <[email protected]> |
||
|
Li LiuandCodex
|
0acf11fde8 |
fix: authorize collection metadata RPCs (#53722)
Authenticated callers can reach segment and replica metadata without a target-collection permission check. Require the existing `GetStatistics` privilege for `GetPersistentSegmentInfo` / `GetQuerySegmentInfo`, and `GetLoadState` for `GetReplicas`. Both permissions remain compatible with the existing ReadOnly group. Apply the checks at gRPC dispatch and at the public Proxy methods used by direct callers. Normalize each request's database before authorization and execution. ID-only replica requests are authorized against the resolved collection's actual database and name; incomplete identity fails closed. Preserve name-selector precedence, configured alias behavior, root role binding, and other public/result-filtered RPC contracts. For ID-only `GetReplicas`, missing collection/database identities and existing but unauthorized identities now return the same `PermissionDenied` code and sanitized message. The response does not disclose the resolved database or collection name. Direct Proxy calls likewise return matching response status fields. Only lookup not-found errors are concealed; cancellation, timeout, not-ready, and unavailable errors retain their existing semantics. Shared merr sentinels and internal retry policies are unchanged. Scope: master PR #53753 has removed all 47 non-underscore legacy `/api/v1/*` routes from the metrics port, addressing the broader legacy-router exposure raised in review. #53416 tracks RBAC checks for retaining those handlers. This PR covers the three public metadata RPCs and their direct-call safeguards. Its retained legacy HTTP handler changes forward middleware-verified identity while preserving request metadata, deadlines and cancellation; those handlers are no longer registered by current master. issue: #53721 Validation: - Regression coverage includes denied/allowed users, same-name collections in different databases, request/context/default database precedence, cold/hot ID lookups, grant revocation, ReadOnly/root behavior, aliases, downstream collection IDs, and legacy HTTP identity propagation. - New regression cases compare the complete gRPC error status and direct Proxy response for existing versus missing IDs, including database-name redaction. They cover users without roles and users granted the same collection name in another database. Error cases also verify wrapped not-found handling and preservation of cancellation, timeout, not-ready and unavailable errors. - Observed the new tests fail before this fix: missing IDs returned raw not-found errors while existing IDs returned permission errors exposing the actual database. The corrected tests and full supplementary harness pass with `-race -tags dynamic,test -gcflags='all=-N -l' -count=1`. - The supplementary harness extracts production authorization and handler functions and uses real Casbin, privilege cache and MetaCache with coordinator doubles. It does not establish that the full Proxy package builds or that deployed gRPC/HTTP entry points have been exercised. - Passed the real `pkg/util/merr`, `pkg/util/funcutil` and `pkg/util/contextutil` suites with the required tags/flags. `go vet` on the supplementary harness, repository formatter checks and `git diff --check` passed. - Audited the latest-ID source path: RootCoord constructs typed collection/database not-found errors, and task/status/MetaCache propagation preserves their codes through the authorization boundary. The historical catalog path that can relabel storage errors is not traversed by this lookup. - Retried actual Proxy and HTTP package tests; compilation remains blocked by missing `rocksdb.pc`, `milvus_core.pc` and `milvus-storage.pc`. Invoked `make test-go`, then stopped its native dependency preparation after Conan found numerous unavailable local binaries requiring a native rebuild; the full Go test stage was not reached. Native validation of this update remains pending CI. Follow-up and limitations: identity lookup still precedes the permission decision. This change does not reduce lookup RPCs, add rate limiting or negative caching, or make response timing indistinguishable. Deployed-cluster, concurrent-DDL and mixed-version behavior have not been exercised. --------- Signed-off-by: Li Liu <[email protected]> Co-authored-by: Codex <[email protected]> |
||
|
aoiasdandClaude Opus 4.6
|
1211f93389 |
feat: [RLS6] enforce row-level security for bulk import (#53911)
relate: #50263 design doc: docs/design-docs/design_docs/20250610-rls_design.md design doc PR: #53173 ## Summary Enforces row-level security for bulk import by authorizing import options at Proxy, persisting a trusted policy snapshot with the import job, and validating imported rows before write. --------- Signed-off-by: aoiasd <[email protected]> Co-authored-by: Claude Opus 4.6 <[email protected]> |
||
|
congqixia
|
348c579977 |
enhance: extract proxy DML tasks into dml package (#53806)
Related to #53685 Move the DML task implementations (insert, upsert, delete, batch update manifest) out of the root proxy package into a dedicated dml package, continuing the proxy task package split begun with the dql extraction (#53685). The dml package now owns the DML task lifecycle and exposes tasks via exported constructors (NewInsertTask, NewUpsertTask, NewDeleteRunner, NewBatchUpdateManifestTask) plus result and routing getters, so the root package no longer reaches into task internals. Shared helpers used by DML are exported from dql; DML-only helpers live in dml/util_dml.go. Migrate the DML test suite into the dml package and rework the previously build-ignored integration tests with mockey/mockery doubles: partition-key routing runs against a mocked meta cache, the upsert lifecycle runs against a mock query-runner node, and tests that exercise the full streaming write path stay as documented skips until a WAL test double exists. Compile, vet, and test for proxy, dml, and dql all pass; neither dml nor dql imports the root proxy package. --------- Signed-off-by: Congqi Xia <[email protected]> |
||
|
wei liu
|
a46f0acfa9 |
fix: Keep manifest-only segments in query recovery targets (#53781)
issue: #53778 Recovered V3 segments can have a committed manifest but no fake binlogs or segment positions. The empty-segment guard previously discarded them, leaving channel-only targets that could report load completion without loading data. Validate manifests before the empty-segment guard. Malformed V3 manifests produce an individual `mlog.Warn` with the segment ID and parse error and are skipped, including segments with binlogs or positions. Warnings are not rate-limited, so a burst of invalid manifests does not suppress affected segment IDs. When positions and binlogs are absent, only a committed V3 manifest (version > 0) may pass; earliest/latest placeholders remain excluded. The recovery RPC interface is unchanged. Validation before the latest rebase and logging-only follow-up (`f8e7446`): - Reused this worktree's validated master C++ build; native build inputs are unchanged. - 17 recovery cases and 6 committed-manifest helper cases pass using `go test -tags dynamic,test -gcflags="all=-N -l" -count=1 -coverprofile=<branch>-coverage.out github.com/milvus-io/milvus/internal/datacoord -run '^(TestGetRecoveryInfoV2_ManifestOnlySegment|TestHasCommittedManifest)$' -v`, with the local RPATH linker setting. - All changed guard statements are covered. The 3 new cases for malformed manifests with binlogs or positions fail before moving the check, and pass after the change on master. - `git diff --check`; master and 3.0 have identical stable patch IDs. - The full external refresh → DataCoord restart → load workflow has not been rerun locally. Latest logging-only follow-up: - Rebased onto the current target branch; the existing patch is unchanged by stable patch-id comparison. - `git diff --check` and `gofmt -l internal/datacoord/handler.go` pass; inspected the `mlog.Warn` signature and non-rated logging path. - Tests and builds were not rerun for this one-line logging change. The test results above belong to the earlier commits. --------- Signed-off-by: Wei Liu <[email protected]> |
||
|
|
15fcd798cf |
test: add continuous streaming data integrity draft (#53307) (#53868)
issue: #53307 ## What changed - Add a Phase 2 draft continuous-streaming DataIntegrity L3 workload with concurrent MutationIngress and ImportIngress lanes, no manual flush, and no manual compaction during streaming. - Extend the Phase 1 framework through opt-in sequence, level, and filtered-verification helpers while keeping existing cases unchanged. - Add the Phase 2 design contract and include the draft workload in the existing dedicated serial DataIntegrity job. - After automatic coverage and final ingress verification pass, request one post-ingress compaction and require Phase 1 lineage/serving checkpoints, fenced full-data verification, and release/load verification; this does not establish per-Delete L0 attribution. - Keep fixed epochs without producer/validator coordination; missing suffix progress fails with explicit coverage diagnostics for CI failure-rate measurement. ## Validation - Related DataIntegrity unit tests: 202 passed. - Ruff, format checks, and scoped git diff checks passed. - This review update was verified in medium mode; the new post-ingress compaction phase has not yet been rerun against a live cluster. Signed-off-by: MrPresent-Han <[email protected]> Co-authored-by: MrPresent-Han <[email protected]> Co-authored-by: GPT-5 <[email protected]> |
||
|
foxspy
|
1bcc8cb1d4 |
enhance: update Knowhere to 16e873b1 (#53914)
Update Knowhere from `faff72c` to `16e873b1` and align `milvus-common` with the version used by Knowhere (`45fca32`). Handle Knowhere's new `cancelled` status as the existing `FollyCancel` code so the exhaustive status mapper continues to compile. Add a regression test covering cancellation through `GuardedCall` and both search/build status mappers. issue: #42937 Validation: - Compiled and ran all 6 `KnowhereStatusMapping` tests as a standalone executable against the new Knowhere headers and milvus-common headers, reusing existing generated protobuf headers. - Segcore error-code Go tests passed. - Both dependency recipes resolve without version conflicts; Python parsing and `git diff --check` passed. - Full Milvus build and UT have not been run. The local Conan cache lacks the milvus-common binary for the complete Milvus dependency configuration. Signed-off-by: xianliang.li <[email protected]> |
||
|
zhuwenxing
|
93af23b066 |
test: declare py dependency for REST parallel tests (#53906)
Explicitly pin py==1.11.0 in the REST test requirements. pytest-xdist 2.5.0 imports py.log when starting parallel workers; without py installed, pytest fails before executing test cases. Validation: git diff --check passed. Parallel E2E has not been rerun. Signed-off-by: zhuwenxing <[email protected]> |
||
|
aoiasdandClaude Opus 4.6
|
03516008f7 |
feat: [RLS4] enforce row-level security (#52075)
relate: #50263 design doc: docs/design-docs/design_docs/20250610-rls_design.md design doc PR: #53173 ## Summary Enforces RLS across query, search, delete, insert, and upsert, including privileged `skip_rls`. --------- Signed-off-by: aoiasd <[email protected]> Co-authored-by: Claude Opus 4.6 <[email protected]> |
||
|
aoiasdandClaude Opus 4.6
|
92f9a74f06 |
feat: [RLS5] expose row policy and principal tag APIs (#53873)
relate: #50263 design doc: docs/design-docs/design_docs/20250610-rls_design.md design doc PR: #53173 ## Summary Exposes row policy and principal tag management through Proxy and the Go client. --------- Signed-off-by: aoiasd <[email protected]> Co-authored-by: Claude Opus 4.6 <[email protected]> |
||
|
wei liu
|
62569cb333 |
fix: Use multipart copy above a configurable 1 GiB threshold (#53787)
## Problem issue: #53786 An approximately 2 GB cross-bucket OSS snapshot copy currently uses one CopyObject request and can fail with `net/http: timeout awaiting response headers`. ## Change Use the existing server-side multipart copy path above 1 GiB by default. Add the refreshable `minio.multipartCopyThreshold` setting (bytes, version 3.0.3), read for each subsequent copy and exported in the default YAML. Positive int64 values are accepted; invalid values fall back to 1073741824. The effective single-copy limit remains capped at 5 GiB. Explicit and endpoint-inferred GCP configurations retain the single-copy path through shared provider resolution without mutating caller configuration. The separate Azure implementation is retained. ## Validation On the rebased master commit, the targeted storage and paramtable tests pass with `-tags dynamic,test -gcflags='all=-N -l' -count=1`. The real SDK HTTP test verifies that a 2,000,000,000-byte object emits four server-side part-copy requests and completes, and that a rejected second part returns the storage permission error without completing the upload. Routing tests cover threshold boundaries, runtime changes, the protocol cap, explicit and endpoint-inferred GCP configurations (including IAM), provider precedence, and caller configuration immutability. Configuration tests cover valid and invalid values; the generated YAML consistency test also passes. ```sh source ~/.profile && source scripts/setenv.sh go test -tags dynamic,test -gcflags='all=-N -l' -count=1 -timeout=5m \ -ldflags="-r ${RPATH}" \ github.com/milvus-io/milvus/internal/storage \ github.com/milvus-io/milvus/pkg/v3/util/paramtable \ -run '^(TestMinioObjectStorageCopyObjectCrossBucket.*|TestMinioConfig_MultipartCopyThreshold)$' -v ``` `git diff --check` passes. No live OSS timeout reproduction was run. ## Known limitations - minio-go v7.0.73 still uses a background context for its internal source HEAD and does not abort an incomplete multipart upload on failure. Those existing SDK limitations remain out of scope. ## CI follow-up The multipart HTTP fixture now generates ETags from its internal part counter instead of reflecting a query parameter, resolving gosec G705. The unrelated hookutil test-isolation change has been removed from the final diff. Its pre-existing race is outside this PR. Provider-resolution and inferred-client tests pass in `pkg/objectstorage`. No live GCS integration test was run. --------- Signed-off-by: Wei Liu <[email protected]> |