571 Commits

Include renames

Author	SHA1	Message	Date
zhagnluandluzhang	48faae3b3a	enhance: parallelize json stats build (#52120) ... issue: #53765 JSON stats build benchmark — 500K rows per segment Concurrent segments || Non-parallel || 8 workers || Speedup 1 || 6.810 s || 1.641 s || 4.15× faster 4 || 7.140 s || 4.806 s || 1.49× faster 8 || 10.878 s || 9.658 s || 1.13× faster Setup: Release build, 12 CPUs, local filesystem, one warm-up followed by five measured runs; median values are reported. Data generation is excluded from the timing. Note: “Non-parallel” means each segment is built serially internally. Multiple segment builds are still launched concurrently in 8-segment cases. Signed-off-by: luzhang <[email protected]> Co-authored-by: luzhang <[email protected]>	2026-09-24 19:07:34 +08:00
zhikunyao	0df04194a6	fix: safely reuse Cargo build artifacts (#53522) ... ### What this PR does - invokes the Tantivy Cargo build target on every CMake build so Cargo fingerprinting decides whether Rust artifacts are fresh - adds the optional `MILVUS_CARGO_TARGET_ROOT` Make variable for persistent Rust build artifacts - assigns separate `tantivy` and `milvus-storage` subdirectories below that root - redirects Corrosion's existing `cmake_build/cargo/build` target through a symbolic link, without modifying the generated Cargo command - migrates an existing generated `cmake_build/cargo` directory when the persistent root is enabled for the first time - preserves the current workspace-local build directories when the new variable is unset - keeps the existing direct `CARGO_TARGET_DIR` behavior as a compatibility fallback for current CI callers ### Motivation The previous `add_custom_command(OUTPUT libtantivy_binding.a)` did not declare Rust sources, `Cargo.toml`, or `Cargo.lock` as dependencies. Once the archive existed, CMake/Ninja could skip Cargo entirely and silently reuse a stale library after Rust inputs changed. Both Tantivy and milvus-storage's Rust bridge currently place Cargo build artifacts below `cmake_build`, which CI workspace cleanup removes. A single Milvus-owned cache root lets CI mount and manage one persistent directory without knowing Rust component names, while Cargo remains responsible for freshness within each workspace target. ### Cache layout When `MILVUS_CARGO_TARGET_ROOT=/path/to/cache` is set: ```text /path/to/cache/ ├── tantivy/ └── milvus-storage/ └── build/ ``` The `milvus-storage/build` suffix is retained because Corrosion already passes `--target-dir=<cmake_build>/cargo/build`. ### Validation - configured the updated Tantivy CMake file in an isolated CMake/Ninja harness - built `install_tantivy_target` repeatedly and verified Cargo was invoked on every build - verified an external Cargo target directory was used for the generated archive - verified `copy_if_different` preserved the installed archive timestamp when its content was unchanged - verified an existing generated `cmake_build/cargo` directory is replaced by the persistent-root symlink - verified a non-directory path at `cmake_build/cargo` is rejected and preserved - `bash -n scripts/core_build.sh` - `git diff --check` Full Milvus compilation with the new shared root has not been run locally. The follow-up Jenkins change will mount a PR-scoped persistent root and validate cold/warm builds for all three build-ut-cov variants. Jenkins integration PR: zilliztech/vdc-jenkins#636. Previous CI integration experiments: zilliztech/vdc-jenkins#631 and zilliztech/vdc-jenkins#632. --------- Signed-off-by: Zhikun Yao <[email protected]>	2026-09-23 19:31:34 +08:00
sijie-ni-0214	8c359cbaf6	enhance: Defer JSON stats initialization until first use (#53708) ... ## Purpose With JSON stats enabled, loading a sealed segment currently calls `JsonKeyStats::Load()` even when cache warmup is disabled. That reads JSON metadata and opens shredding Parquet readers before any query needs those stats. Collections with many sealed segments therefore pay storage-access and initialization costs for stats that may never be used. This change follows the deferred initialization model of file-backed text stats: with warmup disabled, segment loading registers the stats, and the first actual stats access performs initialization. The goal is to remove unused JSON stats initialization from the segment-load path. Fixes #53707 ## Implementation - Store a single-cell `CacheSlot<JsonKeyStats>` with a `JsonStatsTranslator`. `HasJsonStats()` checks registration without loading; `GetJsonStats()` pins the cell and returns an aliasing shared pointer that keeps the accessor alive. - Keep initialized metadata, path/type mappings, and column-management structures resident. Existing shredding-column and BSON-index slots continue to own and evict their payloads independently. The outer slot does not duplicate their payload accounting; its metadata remains segment-management overhead. - Aggregate the existing warmup inputs for the outer slot (`sync > async > disable`) and continue passing the existing configuration into `JsonKeyStats::Load()`. - Share the immutable load proto and create temporary loading configuration on demand. Register different fields through the existing thread pool, draining submitted work before unwinding on failure. ## Behavior and scope Cold first access pays the deferred initialization cost, and file-reading errors can surface at that access. Stats file formats and inner payload eviction granularity are unchanged. This PR does not claim a measured end-to-end latency or memory improvement. ## Validation - `CLANG_FORMAT=clang-format make cppcheck` passed on the rebased tree (clang-format 15.0.7); the working tree is clean. - Before rebasing from `56c674fabd`: C++ build with unit tests passed; all four focused cases passed after the final test cleanup (initial load, concurrent first access, expression routing/first-use/reuse, and warmup-policy aggregation). - An earlier broader JSON stats run passed 13/14 cases. The existing `JsonStatsBinaryRangeTest.ShreddingMatchesRawData` failed at the macOS bitmap cast in `ExprBatchTestUtils.h`; debugger inspection located the failure in the raw-data segment evaluation, before the stats-segment evaluation. - Rebased without conflicts onto `5ddad821c9`. `make verifiers` was attempted, but its C++ build is blocked on macOS by unchanged upstream code from #53483: `POSIX_FADV_DONTNEED` is undeclared in `BitmapIndex.cpp`, and `SaturatingAdd`/`SaturatingMultiply` have conflicting `size_t`/`uint64_t` template arguments in `IndexFactory.cpp`. Tests have not been rerun against the rebased tree; CI validation is pending. - No remote-S3 failure injection or end-to-end performance benchmark was run. --------- Signed-off-by: sijie-ni-0214 <[email protected]>	2026-09-23 09:09:35 +08:00
cai.zhangandClaude Fable 5.1	2cbbb72041	feat: add an in-tree extension mechanism for distributions (#52981) ... issue: #52979 design doc: docs/design-docs/design_docs/20260831-in_tree_extension_mechanism.md A distribution that compiles its own behavior into the milvus binary needs little from milvus: a way to install its request hook, the QueryNode's tuning hook and the cipher without a `.so`, a callback that starts its control-plane engine when the coordinator becomes active and stops it on shutdown, and a declaration that it runs the deployment shape its coordinator behaviors are written for. This PR is that seam and nothing else. A stock binary installs nothing and behaves exactly as before. An earlier revision of this PR shipped a table of typed capabilities consulted from many places in the tree. Reviewing it against what the distribution actually needs showed most entries were constants per deployment (configuration), duplicates of what the request hook already does, or answers that belonged on the wire. The table is gone; what remains is small enough to state here. ## Public surface (`pkg/extension`) ```go func SetHook(h hook.Hook) // hookutil prefers it over proxy.soPath func InstalledHook() hook.Hook type QueryHook interface { ... } // the queryNode.soPath plug-in's method set func SetQueryHook(h QueryHook) // the QueryNode prefers it over queryNode.soPath func InstalledQueryHook() QueryHook func SetCipher(c hook.Cipher) // hookutil prefers it over cipherPlugin.soPathGo func InstalledCipher() hook.Cipher type Coordinator interface { // the coordinator as its own clients see it rootcoordpb.RootCoordClient querypb.QueryCoordClient datapb.DataCoordClient } type CoordinatorEngine interface { Start(ctx context.Context, coord Coordinator) error Stop() error } func SetCoordinatorEngine(e CoordinatorEngine) func InstalledCoordinatorEngine() CoordinatorEngine func SetForm() // marks a deployment form installed, in every role func FormInstalled() bool // the coordinators' form-gated behaviors read it func WithQueryResourceGroup(ctx, rg) context.Context // set by a hook's Before func QueryResourceGroupFromContext(ctx) string ``` A distribution calls the setters it needs, then `cmd/milvus.Main(os.Args)` (exported so a distribution can ship its own `main`). - **Request hook.** `hook.Hook` (milvus-proto) is consulted by the proxy's unary interceptor for every RPC on the gRPC and REST surfaces, and by hand by the service's two streams, `CreateReplicateStream` and `DumpMessages`, in the same order and with the handler under the context `Before` returns; a test fails if the service declares a stream that does not. A compiled-in hook is used in preference to `proxy.soPath`; configuring both is refused at start-up. The compiled-in hook is initialized as a plug-in is: `Init` with the `hook.*` configuration before it is stored, and the `hook.*` reload watcher re-inits it on a config change. A failing `Init` is fatal at start-up when a form is installed, whatever `common.panicWhenPluginFail` says; a hook installed without a form follows the setting like any plug-in. - **Query hook and cipher.** The QueryNode's tuning hook (`SetQueryHook`) and the Go half of the cipher pair (`SetCipher`) can be compiled in the same way. Each is used in preference to its so path (`queryNode.soPath`, `cipherPlugin.soPathGo`), is refused beside it, and gets the same `Init` calls and watchers a plug-in gets. `cipherPlugin.soPathCpp` still declares that encryption is on, so a compiled-in cipher in a deployment without it stays idle. `optimizers.QueryHook` is now an alias of `extension.QueryHook`. None of the three slots implies another or `FormInstalled`. - **Coordinator engine.** `mixcoord` starts the installed engine once the replica is ACTIVE (a standby never starts it), on the coordinator client it uses itself, and stops it on shutdown. A start failure on activation is fatal. The engine reaches the coordinator only through `Coordinator`, so it can do exactly what a proxy can do. - **Query resource-group mark.** `WithQueryResourceGroup` pins a query to one resource group: the shard client routes it to the leaders whose replica lives in that group (`ShardLeadersList.resource_groups`, added on master by #52716) and the proxy attributes its latency to that group. Nothing in a stock binary sets it. ## Behaviors a form turns on by declaring itself These apply only when a form is declared (`extension.SetForm()`, which a distribution does in every role); a stock binary keeps master's semantics unchanged. `FormInstalled` is independent of `SetHook`: a distribution may install a request hook without switching any coordinator behavior on. - **Delegators on regular query nodes.** Under a form, a replica's shard delegators live on its regular RW query nodes, which watch the channel and read the WAL remotely, instead of on the streaming node's embedded query node; the streaming node is kept for DDL and the write ahead log. This lets a form load the same collection into several resource groups, one streaming node total. The trade is stated in the design doc: a Strong-consistency search no longer takes the WAL-MVCC shortcut, and the manual-flush prepare step is skipped with a warning. This shape has been exercised on a real one-streaming-node / multi-RG cluster, not only in unit tests: `scripts/walkthrough.py` (create, bulk-import ingest, index, load into a query cluster, cold and warm query, drop) and the e2e suite (55 tests, including two resource groups loading the same collection) run against a local k8s lake-form instance, and five concurrent rounds of two resource groups querying one collection return correct results with both replicas intact. - **A load request speaks only for the resource groups it names.** Loading a collection into `rg_b` does not restate the whole placement and evict it from `rg_a`; the groups the request does not name keep their replica count. A request naming no group states the whole placement. Per-group load progress is readable through `ShowLoadCollections` with `resource_group` set (appended proto field; `-1` means the group holds no replica of the collection). A load that only adds resource groups or replicas to an already loaded collection keeps the collection serving (an incremental expansion) instead of resetting it to loading, whether the added replica lands in a group the request newly names or one that already serves it; the checkers load the added replicas, as they do on every other path. An expansion replayed after a crash stays idempotent. - **Index engine version with no QueryNode session.** With no QueryNode registered, datacoord answers the index engine version compiled into the coordinator itself instead of `0`, and bounds an explicit target version by that same value, on the assumption that a QueryNode started later runs the same image. ## Removed from the earlier revision `common.security.requireAPIKey`, the `proxy.internalDomain.*` listeners, `queryCoord.resourceGroupScopedLoad`, `queryCoord.reloadSegmentOnIndexDrop`, `dataCoord.index.allowVectorIndexDropOnLoadedCollection`, `dataCoord.index.queryNodesScaleToZero`, the `RefusalResponse` hook extension, the capability table and `Provider`, and the link-time primary configuration file name. Each is either a deployment constant (configuration a distribution ships in its own files), something the request hook already does, or a default this PR now makes unconditional. ## Compatibility - `ShowCollectionsRequest.resource_group` is the only proto field this PR appends; an old peer leaves it empty and both sides read empty as "no scope". `ShardLeadersList.resource_groups`, which the query mark also reads, was added on master by #52716, not here; an old peer leaves it empty too and empty is read as "unknown". - `hook.Hook` is milvus-proto's and unchanged. No new configuration key. - `ResetForTest` is built only under the `test` tag. ## Testing - `pkg/extension`: setters, getters, the context mark, and the split between `SetHook` and `SetForm` (each answers without the other). - hookutil: compiled-in hook used, refused beside a plug-in, absent by default, initialized with the hook configuration, re-initialized by the reload watcher, a failing `Init` fails start-up when a form is installed even with `common.panicWhenPluginFail` off but follows the setting without one; compiled-in cipher used when `soPathCpp` is set, refused beside `soPathGo`, idle without `soPathCpp`, re-initialized on a `cipherPlugin.*` edit. - querynode: compiled-in query hook used, refused beside `queryNode.soPath`, initialized with the `autoIndex.params.*` configuration, not installed when it cannot initialize. - proxy: a hook-pinned resource group reaches the search task; per-group latency series exist only for pinned requests; both streams consult the hook, and every stream the service declares is one of them. - mixcoord: the engine starts on activation only, receives the coordinator client, is stopped once. - querycoordv2: scoped `ShowLoadCollections` (100 / -1 / unknown group refused); scoped load placement with and without a form declared; the expansion predicate's legs, a request that only adds a replica to an already-loaded group, and a replayed expansion; delegators placed on a regular query node under a form and on a streaming query node without one. - datacoord: zero-session engine version and its upper bound with a form declared; `0` without. - `golangci-lint` with the repo config on every touched package. One deliberate difference from the "stock binary is untouched" claim, listed rather than hidden: a stock binary with the streaming service on bypasses the channel checker's version cache for a tick when a replica's read-write node set is empty at that tick (a streaming node mid-restart). The cost is one extra `checkReplica` per tick; the case it exists for, a deployment that starts a replica's nodes on demand, is the form's. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_015JwRUhmz28AXHyuAJTSs3E --------- Signed-off-by: Cai Zhang <[email protected]> Signed-off-by: cai.zhang <[email protected]> Co-authored-by: Claude Fable 5.1 <[email protected]>	2026-09-23 05:45:35 +08:00
sparknack	d510696ac2	enhance: load packed scalar indexes asynchronously (#53483) ... issue: #51245 design doc: [Async packed scalar index loading](https://github.com/sparknack/milvus/blob/feat/async-packed-scalar-index/docs/design-docs/design_docs/20260907-async-packed-scalar-index-loading.md) Load packed scalar indexes through the shared async executor when `queryNode.segcore.storageV2.enableAsyncLoad` is enabled. Stream open and range reads use the milvus-common async interfaces. `PlanLoad` selects entry destinations; `ReadEntriesAsync` interleaves slices under global byte/slot admission and verifies CRCs; `FinishLoadAsync` initializes query state from the completed targets. File preparation, explicit writes, and cleanup use `LocalFileIOPool`. StringSort and Bitmap validity entries, and FMIndex null bitmap entries, read directly into their final `TargetBitmap` allocations. Preserve tail-bit handling and synchronous-path staging reservations. Bitmap mmap loading converts postings to frozen format in 16 MiB write batches during `FinishLoadAsync`. Includes packed scalar indexes and TextMatch `.v3`, with cancellation and failure cleanup. Small magic/footer and directory reads bypass admission; `_meta` payloads use slice admission. The design document includes architecture and sequence diagrams. Legacy index streaming, Knowhere, BSON, and JSON stats remain separate follow-ups. Dependency: [milvus-common #132](https://github.com/zilliztech/milvus-common/pull/132) and [Conan recipe #189](https://github.com/milvus-io/conanfiles/pull/189) are merged. The pinned version is `v1.0.0-fe62001`. The production Conan remote lists the recipe but returned no binary packages in the latest check. Local validation used a staged build matching the merged milvus-common source; installation from the production package remains unverified. --------- Signed-off-by: Shawn Wang <[email protected]>	2026-09-20 21:55:37 +08:00
zhikunyao	995429eb5e	enhance: update master deployment image references to v3.0.2 (#53616) ... Release v3.0.2: update the six deployment and standalone image references from v3.0.1 to v3.0.2 (GPU: v3.0.2-gpu). After the bind-mount ownership changes, both Compose services share the updated image anchor and standalone_embed.sh uses the updated MILVUS_IMAGE variable. Source tag: https://github.com/milvus-io/milvus/releases/tag/v3.0.2 Validation: git diff --check; bash -n scripts/standalone_embed.sh; docker compose config --quiet for CPU and GPU. No source-code or image-build changes. Signed-off-by: Zhikun Yao <[email protected]>	2026-09-20 12:07:37 +08:00
zhikunyao	ed7c7ff0d6	fix: initialize Docker bind mount ownership (#53549) ... issue: #51221 related: #49775 ## What this changes - Add a one-shot volume initializer to the CPU and GPU standalone Compose files. - Resolve the numeric UID/GID from the same Milvus image and migrate `/var/lib/milvus` before the non-root server starts. - Write a UID/GID-specific marker only after a successful recursive migration, so large existing volumes are not scanned on every start and interrupted migrations remain retryable. - Prepare the bind mount in `standalone_embed.sh`, including when a previously created container is stopped after a permission failure. - Teach Docker deployment health checks to exclude one-shot services. - Document the ownership and restore contract. The initializer has no network, uses a read-only root filesystem, and mounts only the Milvus data directory. The Milvus server continues to run as `milvus:milvus`. This fixes both fresh rootful-Docker installations, where Docker creates the bind source as `root:root`, and upgrades from root-running images with root-owned data or cache files. It avoids the ineffective image-layer `chmod` approach from #51226 because a bind mount hides image-layer ownership and mode. ## Compatibility - The recursive ownership migration runs once for each image UID/GID. - A failed or interrupted migration does not create the marker and is retried. - Root-running older images can still read and write the migrated files on rollback. - After restoring files owned by another user, remove `.milvus-volume-owner-*` before starting Milvus to rerun migration. ## Validation - `git diff --check` - `bash -n scripts/standalone_embed.sh tests/python_client/deploy/utils.sh tests/python_client/deploy/check_healthy.sh` - `docker compose -f deployments/docker/standalone/docker-compose.yml config --quiet` - `docker compose -f deployments/docker/gpu/standalone/docker-compose.yml config --quiet` - Verified the deployment health-count expressions return the three long-running services. - Simulated the existing release bump regexes and confirmed both init/server image references and `MILVUS_IMAGE` are updated. Container-level startup was not run locally because the Docker daemon is unavailable; the Linux deployment CI must exercise the fresh-install and root-owned upgrade paths. --------- Signed-off-by: Zhikun Yao <[email protected]>	2026-09-20 10:41:32 +08:00
sre-ci-robotandsre-ci-robot	ed018a736f	[automated] Bump milvus version to v3.0.1 (#53142) ... Bump milvus version to v3.0.1 on master Signed-off-by: sre-ci-robot <[email protected]> Validation: - exactly six release files changed - deployment and standalone embed CPU image tags validated - GPU image tag validated - git diff --check Signed-off-by: sre-ci-robot <[email protected]> Co-authored-by: sre-ci-robot <[email protected]>	2026-09-17 17:23:36 +08:00
Jamesandxiaofanluan	37f3a829e7	fix: make local storage keys complete paths at every layer (#53145) ... issue: #53051, #53052, #53152 ## Summary Align local storage around one path contract: a key is a complete absolute filesystem path, and the local loon/Arrow filesystem is rooted at `/`. Previously, some writers used `minio.rootPath` while others supplied complete local paths to an already-prefixed filesystem, placing files in different physical locations from those expected by readers and GC. Remote keys remain complete bucket-relative object keys; their filesystem-root convention is unchanged. ## Changes - Unify Go/C++ local filesystem roots and StorageV3 manifest-base construction across writers. Preserve the existing base/version pair when continuing a growing-source manifest. - Fix local JSON/TEXT stats reads, rejected-stats cleanup, and explore temporary paths. Exercise resolved V3 stats paths through the actual `LocalChunkManager.MultiRead` interface. - Resolve legacy local manifest bases in loaded metadata and locally referenced snapshots, without eager catalog or manifest-file writes. Preserve manifest CAS compatibility and pathless V3 summaries; copy/restore writes use the target storage root. - Require an absolute, nonempty `localStorage.path` at initialization, including on remote deployments where it is a cache directory. Freeze forbidden parameters against configuration-source refresh while retaining explicit runtime/test overrides, and derive mmap, pprof, and disk-capacity defaults from the same canonical startup value. - Make segment deletion prefixes segment-boundary aware, retain root/identity and snapshot checks, and add BM25 orphan scanning. These shared GC changes also apply to remote storage; this PR does not claim every remote code path is untouched. - Remove the local test-only root override and unused local-only property initialization chain; use the production storage-config initialization path. ## Automatic local upgrade Let `R = localStorage.path` and `M = minio.rootPath`. Before standalone components start, local deployments check the known legacy locations: | Existing data | Upgrade handling | | --- | --- | | Canonical files under `R/<layout>` | Leave in place. | | Double-root files under `R/<R-without-leading-slash>/<layout>` | Move into `R/<layout>` for `insert_log`, `text_log`, `json_stats`, `index_files`, and `index_v1`. | | Recognized late-2.6 unified packed indexes under the process CWD | Move into the corresponding directory under `R`, if visible to the new process. | | Legacy manifest base `M/insert_log/<c>/<p>/<s>`, with files under `R/M/insert_log/<c>/<p>/<s>` | **Do not move the files.** Resolve the loaded manifest base to that existing absolute location. New segment layouts use the canonical root; updates to an existing legacy manifest can retain its base. | Migration prefers directory/subtree rename and recursively merges overlapping directories. Cross-filesystem moves fall back to copy-then-remove. Existing same-name files cause a conflict rather than content comparison or overwrite, except a published copy recognized as this migration's own interrupted work. The copy path synchronizes the file and target directory chain before deleting the source. Remaining sources are rediscovered on retry; empty legacy directories are pruned. Once migrated, restarts probe the known old locations without rescanning canonical data. The initial pass walks legacy file metadata but does not hash file contents. There is no configured migration timeout. SIGTERM/SIGINT cancels the migration and prevents etcd or components from starting; directory walks and cross-filesystem copies observe cancellation between filesystem reads. A canceled partial copy is never published and is recopied from the intact source on the next start. No user migration script, offline command, reverse migration, journal, or backup is provided. Successfully moved sources are removed. ## Compatibility and limits - Preserve the absolute local root, data mounts, and legacy `minio.rootPath` when upgrading. Canonical 2.6 data stays in place; misplaced late-2.6 indexes and affected 3.0.0/3.0.1 layouts are handled as above. Relative roots from older versions are unsupported. - Automatic migration is standalone/local-only. If old CWD indexes disappeared with an old container, they cannot be discovered: preserve/make those files visible or rebuild the derived indexes. Cross-filesystem copy has no configured timeout and can extend the first startup, but a shutdown signal cancels it and the next start resumes from the remaining source. - Downgrade to the affected old layout is unsupported after migration. Symlink layouts require manual resolution. - Known legacy LOB/orphan GC gaps under `R/M/insert_log`, pending old external-refresh paths, and rare interrupted-copy publication residue remain outside scope. This is not a claim that every historical file is recovered or reclaimable. ## Validation At head `898dba4b41`, `ci-v2/build-ut-cov` passed Build, Code Check, Integration UT, Go UT, and C++ UT (9,145/9,145). Diff coverage was 100% for C++ and 81.2% for Go; total patch coverage was 82.8%. Focused local tests for canonical/frozen `localStorage.path` derivation and migration cancellation/deadline/copy recovery passed with `-tags dynamic,test -gcflags='all=-N -l' -count=1`, and `git diff --check` passed before publication. Full paramtable execution remains locally blocked by tests attempting to create `/var/lib/milvus/data`; roles/native-dependent Go tests remain build-blocked by local C++ header/library mismatches. A real 3.0.0/3.0.1 version-to-version upgrade was not executed locally. `ci-v2/e2e-default` and `ci-v2/go-sdk` are still pending for this head. Signed-off-by: xiaofanluan <[email protected]> Co-authored-by: xiaofanluan <[email protected]>	2026-09-17 09:43:08 +08:00
3 people	10a6bd999d	enhance: make configuration projections safe by default (#52579) ... issue: #49846 Design: [Safe Configuration Projections](https://github.com/liliu-z/milvus/blob/enhance/secure-config-projections/docs/design-docs/design_docs/20260826-safe-configuration-projections.md) Configuration diagnostics and management endpoints previously shared raw manager views with runtime consumers. Those views can expose credentials, connection topology, process-environment entries, and deletion tombstones. This change adds declared scalar/group metadata and safe projections while preserving the management configuration API's existing write contract. ## Behavior - Safe configuration views omit undeclared entries and mask sensitive values. Management point reads return `*****` for sensitive keys and reject undeclared keys. Registration and sensitivity decisions account for aliases and dynamic group namespaces. - `ParamItem.Sensitivity` uses `Auto` (zero value), `Sensitive`, and `NonSensitive`; explicit states override inference. Sensitivity controls projections and logging only. Generic management SET/DELETE retains the existing immutable, `mq.type`, and `common.security.adminAuthEnabled` restrictions, original request formats, caller-key handling, duplicate validation, and etcd transaction behavior. Sensitive, unregistered, and security-related settings remain writable unless an existing restriction applies. A successful write does not make its raw value visible through GET. - Built-in file/etcd publication and safe reads share a snapshot lock, keeping source values and their spelling classification together. Collisions and first-time spelling changes retain ambiguity across deferred ownership events and removal. Initial local file spellings are established before environment overrides. Raw manager getters retain their previous locking behavior and source priority. - Effective group reads omit tombstones; deleting or resetting an overlay through an alias addresses equivalent stored overlay spellings. - Sensitivity metadata is installed before scalar/group declarations become visible to concurrent refreshes, including fallback keys and the hook namespace. Positive inventories check actual declarations for reviewed credentials, connection targets, transport, authentication, trust, and routing settings, including the Pulsar tenant and namespace consumed by topic construction. Registration classification stays internal; audit tests use public reads and projections. - WAL configuration maps, management mutations, and dynamic config events omit request names and values from logs. Configuration parser, remote-source TLS initialization, and opaque plugin/callback diagnostics avoid echoing sensitive input. Immutable startup writes mask composite keys containing the protected root path. - Native plugin initialization reports the failed stage without the configured path, plugin-provided name, or opaque exception text. Sanitization precedes native exception observers and Go startup logs while retaining typed codes, allocation-failure classification, and untyped-exception counting. Go plugin wrappers preserve returned errors while suppressing their diagnostic text. - Kafka diagnostic strings mask credentials. `FieldBroadcastID` uses the native `uint64` type and preserves its full range through RPC metadata. Projections remain uncached O(N) diagnostic/management operations. Management authentication, authorization, and additional write restrictions are separate work; endpoint authentication is supplied by #52580, now included through master. This PR does not encrypt stored configuration or audit application-wide backend diagnostics and logs emitted inside out-of-tree plugins. ## Validation ### Merge with current master Validated merge commit `24a8c123fd` with master `2e0677c1d2`. All Go tests used `-tags dynamic,test -gcflags="all=-N -l" -count=1`. - Full `pkg/config` passed with an isolated embedded etcd server. - Focused ParamTable sensitivity, declaration ordering, projection, and lifecycle tests passed. - Config race tests passed for all normalized key spellings, cache entry/byte/concurrency bounds, owned request strings, sensitivity aliases, `knowhere` identity, and refresh consistency. - The actual config GET/ALTER handlers, extracted into an isolated adapter, passed retained coordinator regressions against real embedded etcd. Coverage includes ordinary and sensitive writes, redacted GET, admin-auth SET/DELETE rejection under five spellings, no partial writes in rejected batches, `mq.type` aliases, transaction failures, and omission of payload names/values from logs. - Full `internal/http` and `internal/util/adminauth` passed. `go vet` passed for config, ParamTable, and the extracted handler adapter; `git diff --check` passed. The real coordinator package could not compile locally because RocksDB and `milvus_core` native dependencies are unavailable. The extracted-handler checks do not replace native coordinator CI. ### Feature validation before the master merge Prior local validation used `-tags dynamic,test -gcflags="all=-N -l" -count=1`: - Full `pkg/config` suite passed with an isolated embedded etcd server. - Focused ParamTable tests passed for the sensitivity enum, scalar/fallback overrides, Pulsar resource selectors under dotted/slash/uppercase-underscore/collapsed spellings, raw runtime values, projections, callback logs, declaration audits, and credential/immutability rules. The Pulsar point-read regression failed on the prior head and passed after the fix. - Focused ParamTable `-race` tests passed for enum behavior, Pulsar resource visibility, and sensitivity registration before declaration visibility. - The retained coordinator regressions passed in an isolated adapter that extracts the actual configuration HTTP handlers and connects them to real embedded etcd. The same write-behavior checks passed against the pre-PR write handler. Coverage includes sensitive/unregistered/security settings, Pulsar tenant/namespace SET and DELETE across four key spellings, single and batch requests, empty values, omitted/null resets, raw storage values, redacted GET responses, original key spellings, immutable/WAL rejection, duplicate error behavior, and transaction atomicity. Revised-handler logs omit request canaries on successful writes and transaction failure. - `go vet -tags dynamic,test` passed for config, ParamTable, and the extracted current-handler adapter; `git diff --check` passed. Full ParamTable testing was attempted but stopped because existing tests require creating `/var/lib/milvus/data`, which this host cannot write. Full coordinator tests were attempted but could not compile on this host because RocksDB and `milvus_core` native libraries are unavailable. The extracted-handler checks do not replace the full coordinator/native CI run. Broader backend RPC error descriptions, third-party network-client diagnostics, and logs emitted inside external plugins remain outside the verified log-sanitization coverage. --------- Signed-off-by: Li Liu <[email protected]> Co-authored-by: OpenAI Codex <[email protected]> Co-authored-by: Claude Opus 5 (1M context) <[email protected]>	2026-09-15 20:01:36 +08:00
Li LiuandClaude Opus 5	e008a8429d	enhance: add optional root authentication for the management plane (#52580) ... issue: #49846 design doc: docs/design-docs/design_docs/20260912-management-plane-authentication.md Merge dependency: #52579 supplies safe configuration projections. Both changes are needed for the full configuration-redaction requirement in #49846. ## Problem and behavior Port 9091 exposes process shutdown, coordinator management, profiling, event logs, the web console, and legacy REST data operations. Add `common.security.adminAuthEnabled`, default **false**, to authenticate this surface independently of the main service port's data-plane authorization. When enabled: | Surface | Policy | | --- | --- | | `/management/*` except readiness; `/debug/pprof/*`; `/log/level`; `/eventlog`; `/webui/`; `/telemetry`; `/api/v1/_*` | Root HTTP Basic authentication | | Legacy REST data operations under `/api/v1` | Existing valid-user/API-key authentication if `authorizationEnabled=true`; otherwise root Basic authentication | | `/healthz`, `/livez`, `/metrics`, `/metrics_default`, `/management/check/ready` | Open | | `/api/v1/health` | Exempt from the new gate; existing data-plane authentication still applies | After the request-origin checks below, missing/incorrect credentials return 401; non-root users are rejected before password lookup with 403; unavailable credential storage and saturated verification return 503. Dependency details and stack traces remain out of unauthenticated response bodies. HTTP and Gin share one decision and carry a verified root principal through a typed request context for in-process telemetry calls. ## Browser and client requirements The gate rejects requests identified as cross-site or same-site. An unknown Fetch Metadata value and `Origin: null` are rejected as well. Top-level GET document navigation is allowed only on document surfaces, not operator actions or console APIs. **Credential-bearing requests without Origin or Fetch Metadata must include `X-Milvus-Admin-Request: true`.** This is required because missing metadata does not establish that a request is from a non-browser client; ordinary remote HTTP is outside the Fetch Metadata trustworthy-URL guarantee. An explicit header does not override a reported cross-site origin. The metrics-port CORS preflight must not grant this header to cross-origin callers. Browser console access requires HTTPS with Fetch Metadata preserved by the reverse proxy. A document request without sufficient request context is refused without a Basic challenge. Non-browser callers add the explicit header, for example: ```sh curl --user root --header 'X-Milvus-Admin-Request: true' \ 'https://milvus-admin.example/management/querycoord/balance/status' ``` The flag remains opt-in. Existing clients with the flag disabled keep their prior behavior. ## Credential lifecycle and resource bounds Verifier ownership is deterministic in standalone: Proxy, then MixCoord, then worker. Workers create a MixCoord client lazily. Each lookup caller has its own five-second wait bound, so a service-discovery call that ignores the backend context cannot block the HTTP request indefinitely. Its accounting slot remains occupied until the shared backend result completes, preserving the resource bound even in that case. Root hashes are cached for 10 seconds, reused for up to 10 minutes after their last successful fetch if refresh fails, and lookup failures are cached for 2 seconds. The stale window preserves management access during a coordinator outage but can also retain acceptance of a password rotated during that outage. Concurrent hash refreshes share work and admit at most 32 callers, including canceled callers whose singleflight result channels remain retained. Worker client construction uses a single shared completion signal; repeated lookup timeout windows no longer append a retained channel to a stuck constructor. Construction failure can be retried, and a client completed after shutdown is closed instead of published. bcrypt concurrency is bounded to a quarter of GOMAXPROCS, floored at two, with a bounded queue. This is a concurrency cap, not an attempt rate limit; on small pods sustained incorrect passwords can still consume substantial CPU. ## Registration, configuration, and event logs - `http.Register` refuses an unmarked operator route, including method- and host-qualified ServeMux patterns. Assembled-router tests and a console-prefix test cover Gin registration. - Milvus-owned handlers use a private mux. Flag-off compatibility with more-specific legacy DefaultServeMux routes is retained when pprof is enabled; enabling the gate removes that fallback. - `/management/config/alter` always rejects this flag, using the same `config.FormatKey` identity as config storage, so an anonymous caller cannot plant a disabling etcd value before the operator enables the gate. - Key-normalization caching admits at most 4096 entries and only keys whose input and normalized forms fit 1024 bytes, using owned string storage so short substrings cannot pin large request buffers. Larger keys still normalize correctly without retained memo entries. This bounds cache retention, not individual HTTP body size. - The eventlog gRPC stream switches to loopback with authentication enabled. A failed security upgrade closes the wildcard listener. Event-log shutdown uses a separate close signal and stops the gRPC server, preserving safe concurrent recording. - `milvus_admin_auth_total{endpoint,result}` records root-gate and origin decisions using registered route patterns; failure logs are rate limited. - The standard Go runner now includes HTTP, adminauth, coordinator, eventlog, and metrics regressions. It restores fail-fast behavior after sourcing the environment so a failed test command returns a failure from the runner. ## Compatibility and deployment Port 9091 remains plaintext. Use a trusted network or TLS-terminating proxy, rotate root using `UpdateCredential` / `db.update_password`, and update graceful-shutdown and profiling clients before enabling. Editing `defaultRootPassword` does not rotate an existing credential. Official compose deployments and embedded standalone scripts publish 9091, so review that exposure. Browser access needs the HTTPS deployment described above. Regardless of the flag, downstream operator routes without `AdminAuth` now fail registration; the admin-auth key cannot be altered through the management API; configuration normalization is bounded; pprof registration is explicit; and the data-plane stored-password comparison shares the credential-format helper. ## Verification and limits CI observed on head `c8992fc7b4d3fcf9a28bf73ad4700283922367f2` at **2026-09-12 14:30 UTC**: `ci-passed` and `dco-passed` are present. | Check | Result | | --- | --- | | [Build / unit / integration / code checks](https://jenkins-milvus-ci.milvus.io/job/MILVUS-CI-V2-PR-PIPELINES/job/milvus-build-ut-ciloop-pipeline/30883/pipeline-overview/) | SUCCESS; status reports build, code checks, integration, Go, and 9,106 C++ tests passed | | [Go SDK E2E](https://jenkins-milvus-ci.milvus.io/job/MILVUS-CI-V2-PR-PIPELINES/job/milvus-go-sdk-pipeline/15021/pipeline-overview/) | SUCCESS | | [Default E2E](https://jenkins-milvus-ci.milvus.io/job/MILVUS-CI-V2-PR-PIPELINES/job/milvus-e2e-pipeline-dispatcher/19796/pipeline-overview/) | SUCCESS | | Main CI trigger / DCO | SUCCESS | The non-gating Tencent Pulsar gray run reports `ERROR` because its downstream job could not be triggered. The previous PR head had the identical start failure. No Pulsar test result is claimed, and the precise trigger failure is unknown because Jenkins logs are inaccessible here. The TC-US shadow build and legacy Jenkins status still show pending; they are outside the master `ci-passed` requirements. No label was added manually. Local verification and remaining limits: All Go tests use `-tags dynamic,test -gcflags="all=-N -l"`. - Full `internal/util/adminauth`, including race detection, passed. A blocked-constructor regression retained 20,011 extra heap objects after 10,000 canceled waits before the fix and no retained growth after it. Cancellation, retry, panic, and a client completing after close are covered. Four blocked-backend scenarios cover cold cache, expired stale state, and correct/incorrect stale passwords; race runs repeated three times passed, including retained slots and recovery after release. - Full `internal/http` and its race run passed with the explicit-request-header changes, including 34 new net/http/Gin cases. `pkg/eventlog`, `pkg/metrics`, and `pkg/util/merr` also passed. - Configuration-normalization tests passed, including oversized inputs, Unicode expansion, and request-buffer ownership. The full config race suite passed excluding `TestEtcdSource`, whose two tests require a reachable external etcd and failed against this host's deliberately unavailable test endpoint. The config comment and ParamItem documentation match exactly. - The newly scheduled HTTP, adminauth, eventlog, and metrics commands passed with the runner's race and coverage flags. A dispatch harness checked all five package additions through the full and selected-group entry points. Ten injected command failures were swallowed before restoring fail-fast and now stop the runner immediately with the original nonzero status. Coordinator execution was unavailable locally; the subsequent native CI aggregate reports Go success. - Legacy Basic challenges are preserved on the main port, gate-off metrics routes, and the health exemption. A regression using the actual extracted parser with real Gin failed before the repair and passed after it; assembled-route and dynamic-gate compatibility tests were added to the native package suites. The subsequent CI Go check reports success; individual native test logs were not accessible here. - `internal/proxy`, `internal/distributed/proxy`, `internal/coordinator`, and `cmd/roles` cannot compile on this host because `milvus_core.pc`, `milvus-storage.pc`, and `rocksdb.pc` are unavailable. The subsequent native CI aggregate reports build and Go success; this does not change the local compilation limitation. - Full `make test-go` was attempted; native dependency construction failed in the host SDK linker (`library 'System' not found`) before the Go suite. This is not recorded as a passing full test run. - golangci-lint v2.11.3 built with Go 1.26.6 reported zero new issues in `internal/http`, `internal/util/adminauth`, and the changed config/eventlog/metrics/paramtable packages. Native-linked packages could not be linted locally; the subsequent CI code-check status reports success. - A real coordinator failover and a real browser console flow remain unverified. Unit and source-level failure tracing do not establish full browser credential replay or transport behavior. --------- Signed-off-by: Li Liu <[email protected]> Co-authored-by: Claude Opus 5 (1M context) <[email protected]>	2026-09-15 14:27:33 +08:00
Li LiuandClaude Opus 5	a7941e8b6f	fix: add MinioChunkManager.cpp to segcore error boundary baseline (#53458) ... issue: #53283 ## What Add `internal/core/src/storage/minio/MinioChunkManager.cpp` to the `milvus_storage` section of `scripts/segcore_error_boundary_baseline.txt`. ## Why master's `make static-check` is currently broken: ``` Start check segcore error boundaries ERROR: new file(s) reference 'milvus_storage' (ratcheted library). internal/core/src/storage/minio/MinioChunkManager.cpp ``` - #50768 (merged 09-11) introduced the ratcheted segcore error boundary check. - #53284 (merged 09-14 09:13Z) added `milvus_storage::` references to `MinioChunkManager.cpp`. Its CI ran without the check in scope, so the baseline was never updated. Evidence that this breaks things: - master post-merge `build-ut-cov` on `e3654ef` fails in Code-Check with this error ([#31153](https://jenkins-milvus-ci.milvus.io/job/MILVUS-CI-V2-PR-PIPELINES/job/milvus-build-ut-ciloop-pipeline/31153/)). - PRs whose head includes #53284 fail the same way, e.g. #53377 ([#31143](https://jenkins-milvus-ci.milvus.io/job/MILVUS-CI-V2-PR-PIPELINES/job/milvus-build-ut-ciloop-pipeline/31143/)) and #53456 ([#31158](https://jenkins-milvus-ci.milvus.io/job/MILVUS-CI-V2-PR-PIPELINES/job/milvus-build-ut-ciloop-pipeline/31158/)). - PRs merged after #53284 were not affected only because their CI ran before it landed. ## Why a baseline entry is acceptable The check requires a new consumer to classify errors on every path. In the new code, the only failing `milvus_storage` call (`BuildGcpProviderFromConfig`) is already mapped with `throw milvus_storage::ToSegcoreError(provider.status())`. That form is explicitly allowlisted by the script. ## Notes - The line was added by hand. Regenerating with `UPDATE_SEGCORE_ERROR_BASELINE=1` on macOS reorders entries (locale-dependent sort) and drops `pb/index_cgo_msg.pb.h`, which only exists after proto generation. - #53145 also adds this baseline line, but at a different position. A three-way merge applies without conflict but leaves a duplicate line. The duplicate is harmless to the script (it compares sorted sets), but whichever PR lands second should drop its copy. ## Verification `bash scripts/check_segcore_error_boundaries.sh` on upstream master `5c86e948be`: - before: exit 1 with the error above - after: `segcore error boundaries OK`, exit 0 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01PGMeYPnXk9FbvEB2uCJLj9 Signed-off-by: Li Liu <[email protected]> Co-authored-by: Claude Opus 5 (1M context) <[email protected]>	2026-09-14 15:25:16 -07:00
4 people	82bcce8867	enhance: classify segcore errors across producers and enforce classification end-to-end (#50768) ... ## What Consume the producer-owned error classification at the segcore boundary and make the whole C++→Go classification drift-proof, so a segcore error is classified as **input** (caller's fault, non-retriable), **transient** (retriable) or **permanent** (non-retriable) instead of flattening to `UnexpectedError(2001)` or carrying the wrong retry default. Design + tracking: #50903. ## Changes - **T1** — register the storage fallback pair in `pkg/util/merr/segcore.go`: `StorageError(2044)` non-retriable, `StorageTransientError(2045)` retriable. - **T2** — `KnowhereStatusToErrorCode` → a switch with **no `default` + `-Werror=switch`** over the full `knowhere::Status`; add build-path variant `KnowhereBuildStatusToErrorCode` so a build-time OOM / disk read stays **retriable** instead of collapsing into a permanent `IndexBuildError`. - **T3/T4** — `ArrowStatusToErrorCode` delegates to the producer's `milvus_storage::ToSegcoreError` (retires milvus's duplicate mapper); audited and routed **25 storage arrow-status sites** that were collapsing to `2001` through the single mapper (extracted to `storage/StatusToErrorCode.h`), always preserving the arrow sub-code in the message. - **T5** — unmapped-code observability: `UnmappedSegcoreCodeTotal{code}` counter + rate-limited WARN via an observer hook (merr is a leaf package); registered on QueryNode and DataNode. Unknown code degrades to non-retriable, never panics. - **T6** — codegen + compile-time enforcement: a generated `SegcoreCode` type (from milvus-common's `EasyAssert.h`) + an exhaustive `classForCode` switch marked `//exhaustive:enforce`, with the `exhaustive` golangci-lint enabled opt-in — a new C++ code that is not classified fails lint (the C++→Go analog of `-Werror=switch`). - **§3 B-tier** — classify `marisa` and `simdjson` errors (build/load/parse) instead of collapsing to `2001`, sub-code in the message; simdjson optional-access (`NO_SUCH_FIELD`/`INCORRECT_TYPE`) stays a benign skip; the `loon_ffi` FFI boundary is untouched. - **Boundary hardening (adversarial self-review of this PR's own diff)** — closed the escapes that would defeat the mapping above: a `throw e;` slicing rethrow in `LoadWithStrategy` that destroyed the very codes the columnar-read mapping attaches (bare `throw;` now), the same slice in `MinioChunkManager::PreCheck`; `GetCoreMetrics` / `EstimateLoadIndexResource` / init-and-config entry points that could let an exception cross the C ABI and terminate the process; and every remaining extern-C entry that caught only `std::exception` now ends in `catch(...)` via the shared `CGoCatch.h` macros. - **Pin + semantics** — bump `milvus-storage_VERSION` to `11f8a36` (the milvus-io/milvus-storage#574 merge, which also contains #575) and align the no-detail `IOError` expectation with the settled semantics: the producer tags every known-transient failure with a retryable `ExtendStatusDetail`, so a bare `IOError` with no detail is unclassified and deliberately falls back to permanent `StorageError(2044)` — a stripped-detail NotFound now degrades to non-retriable (safe) instead of retriable (retry storm on a permanent 404). - **Wire pass-through (client-visible)** — a segcore error now reaches the client with its ORIGINAL code (2009 stays 2009, 2024 stays 2024) instead of collapsing to the `ErrSegcore(2000)` umbrella with the real code buried in the message. Family identity for `errors.Is` is preserved via inner/Unwrap; input/system/retriable classification unchanged. Guardrails: only in-band (2000-2099) codes pass through (garbage still collapses to 2000); cross-family mappings (2046 → wire 110) keep their sentinel's code. `ErrSegcoreUnsupported`/`ErrSegcorePretendFinished` move to the C++ values they represent (2001→2003, 2002→2033) — their old numbers squatted on C++ UnexpectedError/NotImplemented and would false-match under code-based `errors.Is`. Verified end-to-end on a live standalone (ef<k reaches the client as 2042, unsupported tokenizer as 2001); the three e2e assertions pinning the old 2000 updated. - **Remaining code-destroying sites** — the three classes that still swallowed a producer's classification before the cgo boundary are now gone from `internal/core/src` and `internal/core/thirdparty`: status-consuming `AssertInfo` (104 → 0, incl. ~47 arrow builder paths whose commonest failure is OOM, now retriable `MemAllocateFailed` instead of a permanent 2001), bare `throw std::runtime_error/logic_error/bad_alloc` (68 → 0 — these were not `SegcoreError`, so they collapsed to 2001 *and* falsely fired the untyped-exception observer), and `throw fmt::format(...)` (12 → 0 — it throws a `std::string`, which `catch (std::exception&)` cannot see at all). tantivy's 73 `AssertInfo(res.result_->success, ...)` (plus 10 raw-`RustResult` stragglers found later) now classify the rust error — originally by its Display prefix, since replaced by a proper `#[repr(i32)]` discriminant carried in `RustResult.error_code` (see the Aug-10 update below). Typed `ThrowInfo` sites: 894 → 1081. The ~1500 genuine invariant asserts are untouched — 2001 is correct for them. The long-standing FIXME about `err_code` not surviving the nested LOON FFI boundary is also resolved, delegating to `milvus_storage::ToSegcoreErrorCode` rather than duplicating its table. ## Verification **Verified in this PR:** - **Mapping correctness (unit-tested, in-process):** `test_knowhere_status_mapping.cpp` / `test_storage_error_code.cpp` / `test_exec.cpp` cover every mapper branch (knowhere Status incl. the build variant, arrow/extend status incl. `AwsErrorNotFound→ObjectNotExist(2017)`, permanent-S3 vs transient), plus `FailureCStatus` code preservation and both observer hooks firing. - **Code projection to Go (one hop, unit-tested):** `segcore_test.go` pins `classForCode` for every generated code and asserts `merr.Status(err).GetRetriable()` for transient codes; the T6 generator is idempotent and the `exhaustive` lint fails on an unclassified code. - **Full C++ suite:** 8213/8223 unit tests pass locally (10 skipped; Azure connectivity tests excluded), 8648 in CI, rebased on current master (one pre-existing, unrelated concurrency test excluded: `GrowingConcurrentReopenTest` deadlocks deterministically on current master with or without this PR — rwlock writer starvation in growing-segment reopen code this PR does not touch; reported separately). - **Static audit (grep-verifiable):** every storage arrow-status consumption site on the read path routes through `ArrowStatusToErrorCode`, and every extern-C boundary ends in a `catch(...)` tail. **Explicitly NOT verified here (follow-up):** - **Runtime fault injection.** No S3 throttle / 404 / OOM / corrupt-file failure has been triggered end-to-end in a running cluster. Transient codes reach Go with `retriable=true` (unit-tested projection), but the downstream consumption — `lb_policy` replica reroute on `merr.IsRetryableErr`, index/analyze scheduler retry — is pre-existing logic from #50221 and has **not** been driven by a real segcore transient error in this PR. This PR preserves classification for observability and correct retry defaults; the retry behavior itself is exercised only by its own pre-existing tests. ## Dependencies - ~~milvus-common `StorageTransientError(2045)` — zilliztech/milvus-common#102~~ **merged**. - ~~milvus-storage `ToSegcoreError` / packed `ExtendStatusCode` — milvus-io/milvus-storage#575 + #574~~ **merged; pin bumped in-tree to `11f8a36`**. - ~~knowhere three-way classification — zilliztech/knowhere#1704~~ **merged** (the milvus-side `KnowhereStatusToErrorCode` → thin delegate to knowhere's own `ToSegcoreErrorCode` is a follow-up, gated on a knowhere version bump). - ~~milvus-common untyped-cgo-exception observer — zilliztech/milvus-common#112~~ **merged and released as `1.0.0-1fd1160`; the pin now points at the published package.** All dependencies are in. ## Update (Aug 10) — full-population audit, LOON path, runtime observability The originally deferred FFI/LOON path is now **done on the milvus side**, and the audit was extended from the three grep-able classes to the *entire* 2001-producing population: - **Every remaining 2001 site read.** All 1,517 `AssertInfo` (four sweeps: errno fingerprint, failure-keyword messages, condition morphology, and finally **data provenance** — does the guarded value come from disk/network?) and all 198 explicit `ThrowInfo(UnexpectedError)` sites. ~290 were externally-triggerable and now carry typed codes: file/remote IO -> `FileOpen/Create/Read/WriteFailed` (retriable), mmap/allocation -> `MmapError`/`MemAllocateFailed` (retriable), persisted-format damage (CRC/magic/parquet meta/index-meta keys) -> `DataFormatBroken`, deployment config -> `ConfigInvalid`, request content -> `InvalidParameter`, a cancel-race -> `FollyCancel`. The ~1,400 kept sites are genuine invariants or cgo contracts where 2001 is the correct report. - **Two infinite-retry bugs.** Statically-impossible conditions (index_type x metric blacklist, per-type metric allowlists, json/geometry index gates) threw 2001 -> generic retry -> the build task spun forever; they now throw `Unsupported`, which `getStateFromError` maps to a terminal `JobStateFailed`. Missing `index_type`/`metric_type`/`min_gram`/`max_gram` keys in persisted index meta had the same loop on the load path; they are `DataFormatBroken` now. - **knowhere `expected<>` bypasses closed** (8 sites in `QueryResult.h`/`CachedSearchIterator`): iterator failures went through `AssertInfo` and discarded the Status knowhere had already classified; they now route through `KnowhereStatusToErrorCode`, so an OOM/disk failure during search iteration stays retriable. Preflight rewraps in `segment_c`/`boost_score` similarly preserved the original `SegcoreError` code instead of flattening to 2001+string. - **tantivy discriminant over the FFI.** `RustResult` now carries `error_code` (`#[repr(i32)] TantivyBindingErrorCode`, cbindgen-exported); the C++ mapper switches on the enum instead of parsing the Display text, and the inner `tantivy::TantivyError` is discriminated too (`IoError/Open*Error` -> Io/retriable, `DataCorruption/IncompatibleIndex` -> DataCorruption). Wording changes on the rust side can no longer silently degrade classification. - **LOON / FFI path (the deferred item), milvus side complete.** The Go funnel `HandleLoonFFIResult` dropped `err_code` entirely and wrapped every failure as `ErrLoonTransient` — a 404/access-denied/corrupt-data retried as transient. It now classifies by the producer's own `loon_ffi_is_retryable_errcode`; permanent failures carry the new `ErrLoonPermanent` and terminate retry loops (`pack_writer_v3` via `retry.Unrecoverable`; the external-refresh manager guard extended so behavior does not invert). On the C++ side `LoonErrCodeToErrorCode` is the single classification entry (low band -> hand table, extend band -> producer's `ToSegcoreErrorCode`, unknown -> producer's retryable probe), unifying the two previously-divergent `ThrowIfFFIError` helpers — `LOON_FILE_NOT_FOUND(12)` now converges to `ObjectNotExist(2017)` on both integration paths. Remaining LOON items (e.g. promoting FileNotFound into `ExtendStatusCode`) live in the milvus-storage repo. - **Regression guards.** `scripts/check_segcore_error_boundaries.sh` wired into `make static-check`: every `throw` in `internal/core/src` must carry a milvus ErrorCode (zero-tolerance; currently 0 violations); vendored `fmindex::` is confined to its boundary files; knowhere/arrow/milvus_storage/tantivy are ratcheted by a checked-in file-set baseline (new consumer files fail the check; shrinking is free). - **Runtime observability for what is left.** `milvus_cgo_unexpected_segcore_origin_total{origin="<file>:<line>"}` counts every 2001 crossing the cgo boundary by its C++ source location (parsed from the ` at file:line` suffix `AssertInfo` already emits, build paths collapsed to repo-relative). A site that fires in production names itself — reclassification becomes evidence-driven instead of re-reading ~1,400 asserts. Site count for the 2001 family: 1,955 on master -> 1,525 on this branch; the delta is reclassification into actionable codes, not deletion of checks. ## Deferred - milvus-storage-side LOON improvements: promote `LOON_FILE_NOT_FOUND` into `ExtendStatusCode`, category byte (design §4.7) — tracked in the storage repo. - knowhere-side: thin-delegate `KnowhereStatusToErrorCode` to knowhere's own `ToSegcoreErrorCode`, gated on a knowhere version bump. issue: #50903 --------- Signed-off-by: Zack <[email protected]> Co-authored-by: Zack <[email protected]> Co-authored-by: Claude Fable 5 <[email protected]> Co-authored-by: xiaofanluan <[email protected]>	2026-09-11 14:18:26 -07:00
zhikunyao	73e495d7d5	fix: keep installed C++ debug info self-contained (#53214) ... ## What this fixes Linux builds currently enable split DWARF by default, but Milvus installation and published debug images do not include the generated `.dwo` files. The installed ELF files therefore contain dangling DWO references and GDB cannot load the expected debug information. ## Changes - make split DWARF an explicit CMake opt-in, defaulting to OFF - expose `USE_SPLIT_DWARF=ON` through `scripts/core_build.sh` for build-only workflows that preserve `.dwo` files - keep the existing release image, tag, multi-arch manifest, and strip pipeline unchanged With the default build, debug information remains self-contained in the installed ELF files. The existing release pipeline can continue publishing the unstripped build as `-debug` and stripping the same build for the regular image without copying sidecar debug files into production images. ## Release impact - regular stripped images should retain their current layout and strip behavior - debug images can grow because debug sections are embedded instead of being left behind in the build tree - Release builds still use the existing `-g1`; changing the debug level or publishing `.dwp` artifacts is intentionally out of scope - the first C++ build after this flag change may have lower cache reuse ## Validation - `bash -n scripts/core_build.sh` - parsed `DefineOptions.cmake` with CMake - verified `MILVUS_USE_SPLIT_DWARF` defaults to OFF and accepts explicit ON - `git diff --check` Supersedes #53188, which was recreated from the contributor fork to follow the repository contribution workflow. Fixes #53182 Signed-off-by: Zhikun Yao <[email protected]>	2026-09-11 15:31:33 +08:00
sparknackandClaude Opus 4.6	5f7ddf23d2	enhance: add async storage v3 field data loading (#51246) ... issue: #51245 - Add an async executor and Storage V3 field-data load pipeline for manifest column groups and JSON key stats. - Group requested cells into contiguous, byte-bounded row-group read windows and admit each window through the shared transient-memory budget. - Preserve load priority, requested result order, cancellation, and the first storage or finalization failure across coroutine boundaries. - Route mmap finalization to the optional local-file I/O executor while keeping the legacy synchronous path as the default. - Add the temporary disabled-by-default rollout switch `queryNode.segcore.storageV2.enableAsyncLoad`; `storageV2` is retained only as the existing internal configuration name. --------- Signed-off-by: Shawn Wang <[email protected]> Co-authored-by: Claude Opus 4.6 <[email protected]>	2026-09-09 14:35:28 +08:00
XuanYang-cn	720983fe5d	test: add Scalar Index CMEK positive matrix (#52816) ... ## Summary - Add the positive Scalar Index CMEK matrix for engines 2, 3, and 5 across V2/V3 artifact formats. - Cover 17 cells with repository-owned Go/C++ CipherPlugin fixtures, authoritative object locators, independent V2/V3 envelope inspectors, release/load checks, and type-specific query oracles. - Verify database EZ ID inheritance and configured root-key material propagation: the strict fixtures now fail when an EZ key was not delivered, and the C++ fixture derives encryption keys from the exact forwarded root key. - Keep vector-index setup in every collection with an explicit FLAT/L2 index. - Keep `integration-test` as the aggregate CI gate: it depends on the existing non-CMEK scope and the dedicated CMEK scope, while only the CMEK target builds its fixture plugins. - Group the campaign tests by suite in the V2, V3, and FMINDEX suite files. ## Review follow-up - Complete the ASF license headers in all new source and CMake files. - Remove the unused text-log inspector wrapper. - Link the independent V3 inspector to the format design and production reader/writer sources to make drift reviewable without coupling the oracle to production decoding. - Keep large V2 EZ ID parsing local to the inspector with json.Number; this PR does not modify production descriptor decoding. - Split the default integration gate into `integration-test-base` and `integration-test-cmek`, exclude only the top-level CMEK package from the base run, and append CMEK coverage to the same `it_coverage.txt` only when invoked through the aggregate target. ## Verification - Controlled full Milvus build — passed after merging current master. - `make integration-test-cmek` — 23 tests passed in 3m02s. - CMEK fixture build regression — passed. - Target graph dry-run — the aggregate runs base before CMEK with append enabled, while standalone CMEK resets the single coverage file. - Strict Go fixture unit test — passed after reproducing the fallback gap. - Scoped V2/V3 inspector tests — passed. - `make fmt` — passed. - Controlled static-check — 0 issues in core, pkg, client, and go_client. - All commits are signed off. issue: #52815 --------- Signed-off-by: yangxuan <[email protected]>	2026-09-03 01:25:29 +08:00
Li Liu	e36fbfdc2d	fix: migrate archived hamba/avro to maintained fork and suppress plan-parser pseudo-version (#52782) ... ## Summary Address the remaining master image CVEs left after #52685 (which only bumped x/net, x/text, grpc). | CVE | Package | Severity | Remediation | |-----|---------|----------|-------------| | CVE-2026-46385 | github.com/hamba/avro/v2 v2.29.0 | High | Migrate to maintained fork `github.com/iskorotkov/avro/v2` v2.33.1 (direct imports) | | CVE-2026-46384 | github.com/hamba/avro/v2 v2.29.0 | High | Same; plus `replace github.com/hamba/avro/v2 => github.com/iskorotkov/avro/v2 v2.33.0` for transitive (Pulsar) consumers | | CVE-2026-26190 | github.com/milvus-io/milvus self-reference in libmilvus-planparser.so | Critical (scanner finding) | Build plan-parser with `-buildvcs=false` so the local module stays `(devel)` instead of stamping a misleading affected pseudo-version | The archived `hamba/avro/v2` has no fixed release (project archived Jan 2026); the published mitigation is to migrate to the maintained `iskorotkov/avro/v2` fork. The actual endpoint-auth fix for CVE-2026-26190 is already in master (PR #47277); the scanner flags the stamped pseudo-version, so this change suppresses the false positive. ## Change scope - `internal/snapshotio/snapshot.go`, `snapshot_test.go`, `storage/snapshot_data_test.go`: import `github.com/iskorotkov/avro/v2` - `go.mod` / `go.sum`: replace hamba direct require with iskorotkov v2.33.1; keep hamba v2.29.0 as indirect; add `replace` directive - `pkg/go.mod` / `pkg/go.sum`: add iskorotkov indirect + `replace` directive - `scripts/build_plan_parser.sh`: add `-buildvcs=false` ## Runtime image coverage Go-dependency and build-metadata changes only. All six published runtime Dockerfiles copy the same root-module `bin/milvus` and `lib/` outputs, so one dependency-graph change covers every image: - `build/docker/milvus/ubuntu22.04` - `build/docker/milvus/ubuntu20.04` - `build/docker/milvus/amazonlinux2023` - `build/docker/milvus/rockylinux9` - `build/docker/milvus/gpu/ubuntu22.04` - `build/docker/milvus/gpu/ubuntu20.04` No image variant is skipped; no per-Dockerfile package-manager change is required. ## Validation - `go mod verify` passed in root and `pkg`. - `go mod tidy` clean in `tests/go_client`. - `go build -tags dynamic,test ./internal/snapshotio/...` resolves the new module graph (fails only at the `milvus_core` pkg-config stage, which this checkout does not provide; CI rebuilds native deps). Auto-generated CVE fix (autonomous run). issue: #50763 Signed-off-by: Li Liu <[email protected]>	2026-09-01 07:03:38 +08:00
David Dallakyanandxiaofanluan	3dfae89e93	feat: route general LIKE through FMINDEX with candidate recheck (#52697) ... General LIKE with interior `%` or `_` was declined by FMINDEX even though PrefixMatch, PostfixMatch, and InnerMatch already use it. This routes VARCHAR Match through `ShouldUseOp`: split the pattern on wildcards, take candidates from the rarest literal fragment, then recheck those rows on sealed VARCHAR with `LikePatternMatcher` via `ProcessDataByOffsets`. FMINDEX load no longer materializes ISA. Match phase 2 rechecks candidate rows on sealed VARCHAR via ProcessDataByOffsets. The Match guard is locate-only again, the same occ times sa_sample_rate bound as Prefix, Inner, and Postfix. Empty-fragment patterns still decline. RegexMatch, equality, IN, TEXT, JSON, ARRAY, and growing segments are unchanged. FMINDEX Match is never served as unrechecked PatternMatch candidates through the ScalarIndex path. Design: `docs/design-docs/design_docs/20260708-fm_index_scalar_index.md` issue: #52683 ## How to test `LOCAL_STORAGE_SIZE=10` is required on this host. Without the VARCHAR recheck (product from `47f5cf5791`, tests kept): 5 FAILED (`ShouldUseOp(Match)` still prices Extract, so `%RARE%` / `%ZEBRA%` stay declined). With `7a3e274a36`: ``` source scripts/setenv.sh export DYLD_LIBRARY_PATH="$PWD/cmake_build/src:$DYLD_LIBRARY_PATH" LOCAL_STORAGE_SIZE=10 cmake_build/unittest/all_tests --gtest_filter='FMIndex.*' [==========] 23 tests from 1 test suite ran. [ PASSED ] 23 tests. ``` Includes `FMIndex.ExecutorPathMatchRechecksVarchar` and `FMIndex.MatchOracleEqualsBruteForce`. Whole C++ module (three Darwin crashers excluded): ``` export LOCAL_STORAGE_SIZE=10 internal/core/output/unittest/all_tests --gtest_filter='-JsonIndexTest.LargeInt64LiteralDoesNotAliasInDoublePathIndex:JsonFlatIndexExprTest.PreservesLargeInt64LiteralPrecision:SortBufferTest.*' [==========] Running 8326 tests from 473 test suites. [ PASSED ] 8167 tests. ``` None of the 138 failures are `FMIndex.*`. Reverse deps: Ngram 10/10, scalar/hybrid/inverted 26/26, json_stats_test 79/79, LikeConjunctExpr 1/1. /cc @xiaofan-luan --------- Signed-off-by: David Dallakyan <[email protected]> Signed-off-by: xiaofanluan <[email protected]> Co-authored-by: xiaofanluan <[email protected]>	2026-08-30 22:40:06 +08:00
Zhen Ye	5097a4246b	feat: add query view state machines (#51887) ... issue: #40451 - QueryView protocol: add lifecycle states, per-node view payloads, bidirectional sync messages, and generated Go bindings - state machines: add shared QueryView value objects and focused Coord, StreamingNode, and QueryNode lifecycle implementations with unit tests - documentation: describe state transitions, recovery of versioned Up views, StreamingNode query-lease ownership, and TODO integration boundaries --------- Signed-off-by: chyezh <[email protected]>	2026-08-11 17:00:23 +08:00
zhikunyao	d3793e4463	auto: bump Milvus version to v3.0.0 (#52024) ... Update the default Milvus release image references on master from v3.0-beta to v3.0.0. This is the master counterpart of #52016. Changes: - bump CPU deployment and binary examples to v3.0.0 - bump GPU standalone deployment to v3.0.0-gpu - bump standalone embed shell and batch scripts to v3.0.0 Validation: - exactly six release files changed - no targeted v3.0-beta image references remain - bash -n scripts/standalone_embed.sh - docker compose config --quiet for CPU and GPU standalone files - git diff --check Signed-off-by: Zhikun Yao <[email protected]>	2026-07-30 19:12:08 +08:00
Bingyi Sun	70f144a91c	enhance: add delete/shard by namespace (#50153) ... Shard data by namespace. issue: #50154 --------- Signed-off-by: sunby <[email protected]>	2026-07-02 15:36:35 +08:00
3 people	ef166a673a	enhance: support building core on macOS (Apple Silicon) with clang 19 (#50616) ... issue: #50615 ## What this PR does Makes the C++ core build cleanly on **macOS (Apple Silicon) with Homebrew LLVM/clang 19**. > [!IMPORTANT] > **Scope / impact is NOT macOS-only.** Although this PR is framed as macOS/clang-19 enablement, two of the changes are **top-level `requires(...)` in `internal/core/conanfile.py`, so they apply to every platform, including Linux production builds:** > - **`azure-sdk-for-cpp` `1.11.3@milvus/dev` → `1.16.0@milvus/dev`** (5 minor versions, `force=True`). This is the SDK Arrow uses for the **Azure Blob remote-storage path**, so this is effectively a major Azure SDK bump on the **production Azure object-storage path for Linux deployments** too. > - **`arrow` recipe `17.0.0@milvus/dev-2.6#c743ea7a…` → `17.0.0@milvus/dev#f411aa73…`** — same Arrow version, but **rebuilt against azure 1.16.0** with a patched `FindAzure.cmake`. The Arrow **`package_id` (recipe revision) DID change**; it is not the same binary as before. > - **`libbson/1.30.6@milvus/dev`** added (also global, but new — no pre-existing behavior to regress). > > **Testing blind spot:** CI green proves compile/link/UT/e2e pass, but the UT and integration suites **do not exercise real Azure Blob read/write**, so the Azure object-storage path is **not covered** by this PR's CI. A real Azure Blob remote-storage regression is recommended before merge (see Verification). ### Why the dependency bumps are required for clang 19 - **azure 1.16.0**: azure 1.11.3 bundles an older `nlohmann/json` that uses `std::char_traits<unsigned char>`, which was **removed in libc++ 19** → 1.11.3 fails to compile on clang 19. 1.16.0 fixes this. - **arrow rebuilt against azure 1.16.0**: when Arrow is built from source on clang 19 it must link the same azure 1.16.0; the recipe also patches `FindAzure.cmake` to the monolithic `Azure` config so `find_package(Azure)` resolves. Hence a new Arrow recipe revision. ### Toolchain / build enablement - **`scripts/setenv.sh`**: probe LLVM 19/20/21 first (libc++ 17/18 `chrono operator<<` clashes with Arrow's vendored `date`; libc++ 19 fixes it). - **`scripts/3rdparty_build.sh`**: arm64 `-march=armv8-a+crypto+crc` so folly's F14 (SimdAndCrc on Apple Silicon) matches core/knowhere — fixes `F14LinkCheck` undefined-symbol link errors. - **`InvertedIndexTantivy.cpp`**: drop spurious `->template` on non-template members (clang 19 `-Wmissing-template-arg-list-after-template-kw`). ### Drop bsoncxx / libmongoc → libbson only The BSON layer used the **bsoncxx** C++ driver, whose CMake unconditionally builds mongo-cxx-driver → **libmongoc** → bundled **utf8proc**, which fails to configure under newer CMake/clang on macOS. Milvus only needs BSON (de)serialization. - New Conan package **`libbson/1.30.6@milvus/dev`** (libbson-only, `ENABLE_MONGOC=OFF`; recipe published to milvus conanfiles & production). - New **`src/common/bson_shim.h`** — a thin `milvus::bson` C++ layer over libbson's C API. - Migrated `bson_view.h`, `bson_builder.{h,cpp}`, json_stats consumers and unit tests off bsoncxx. - CMake: `find_package(bson-1.0)` + link `mongo::bson_shared`; removed FetchContent `thirdparty/bsoncxx`. All three global Conan requires (arrow / libbson / azure) pin an explicit recipe `#revision` for reproducibility. ## Verification - Full clean rebuild on macOS arm64 / clang 19: `bin/milvus` builds; `libmilvus_core` links **only `libbson-1.0.dylib`** (no libmongoc/mongocxx/utf8proc). - libbson / arrow / azure resolve from production Conan at the pinned revisions. - All 21 BSON unit tests (BsonView / BsonBuilder / DomNode) pass. - Full ci-v2 suite green on Linux (build / build-ut-cov / ut-cpp / ut-go / integration / e2e / go-sdk). - **TODO before merge:** real Azure Blob remote-storage read/write regression to cover the azure-1.16.0 / arrow-rebuild blast radius (not exercised by UT/integration). 🤖 Generated with [Claude Code](https://claude.com/claude-code) Signed-off-by: xiaofanluan <[email protected]> Co-authored-by: xiaofanluan <[email protected]> Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>	2026-06-19 20:27:10 -07:00
3 people	6492fa011b	enhance: remove unused C++ dependencies (rapidjson, OpenDAL backend) (#50552) ... ## What this PR does Removes two unused C++ third-party dependencies from `internal/core` to reduce dependency count, comprehension cost, and link/build surface. ### rapidjson Linked into `milvus_core` but has **zero `#include` usage** in the C++ sources (only a stale comment in `common/Json.h`). knowhere v3.0.3 and milvus-storage do not depend on it. Removes the conan requirement, `find_package(RapidJSON)`, and the link target. ### OpenDAL backend Experimental `ChunkManager` backend gated behind `USE_OPENDAL` (**default OFF**) — released binaries never built it. Removes `OpenDALChunkManager`, the vendored OpenDAL Rust dependency, and all `USE_OPENDAL` build plumbing (Makefile / core_build.sh / 3rdparty_build.sh / CMake). The Go-side `"opendal"` storageType value remains a backward-compatible alias to the standard remote chunk manager. ### Kept `simde` is **not** removed — knowhere v3.0.3 requires it (`find_package(simde REQUIRED)`, used in `src/index/sparse/codec/varintdecode.c`). issue: #50551 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: xiaofanluan <[email protected]> Co-authored-by: xiaofanluan <[email protected]> Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>	2026-06-16 15:02:23 +08:00
zhenshan.caoandClaude Opus 4.7	e2787d3981	enhance: standardize error handling on merr + Sys/Input classification (#50221) ... issue: #47420 ## What this PR does Project-wide migration of raw `fmt.Errorf` / `errors.New` in function bodies onto the `merr` framework, plus the Sys-vs-Input error classification and the machinery it drives (retriability, fine-grained metrics, segcore unification), plus the convention docs and a linter that keeps it from regressing. Scope: storage, proxy, coordinators (root/data/query), query node, data node, `pkg/util` & `internal/util`, expression parser, message queue, streaming, and misc packages. Bare raw-error usages went from ~3000 to a ~340 allowlist (package-level sentinels / build-tag / test sites). --- ## How to review this PR It is large but the vast majority is mechanical. Changes fall into three tiers; spend review budget on Part 2 and Part 3. ### Part 1 — Mechanical standardization (low risk, verify by rule) Each converted call follows one of a small fixed set of rules. To review, check that each site obeys the matching rule rather than reading every line: | Pattern | Rule | |---|---| | `fmt.Errorf("...")` originating a new error | → `merr.WrapErrXxxMsg("...")` with a code matching the failure's meaning | | Adding context to an existing typed error | → `merr.Wrap(err, "...")` / `merr.Wrapf(...)` — **preserves** the inner code (never `WrapErr*Err`, which overwrites it) | | Errors inside the streaming subsystem | → `status.New*` factories (StreamingError), **not** merr — this is the component-internal dialect (see `docs/dev/error_handling_guide.md`) | | Low-level / control-flow signal caught by `errors.Is` | → kept as a package-level `errors.New` sentinel (lowercase, same-package) | Conventions are documented in `docs/dev/error_handling_guide.md` (how-to) and `docs/dev/error_sentinel_convention.md` (rules + audit). A `gocritic`/`ruleguard` rule (`rawmerrerror`, in `rules.go`) enforces "no raw `return errors.New/fmt.Errorf`" under `make verifiers`. ### Part 2 — Behavior changes (review these closely) These are the sites where the wire contract or runtime behavior changes, not just the source text. Listed by category; representative locations given, full set in the diff. **A. gRPC wire-code shifts: `UnexpectedError(1)/Code 65535` → typed code.** Where a handler previously returned a raw error (collapsed to `Code=65535` on the wire), it now returns a typed merr, so the client sees a real code. The most common shift is to `IllegalArgument(5)/Code 1100` (ParameterInvalid). Touch points include datanode task handlers (CreateTask/Query/Drop), proxy Upsert, querynode GetMetrics, datacoord CreateIndex, httpserver query-response builder, and typeutil schema validation. One code refinement: an index-param validation moved `1100` → `1101` (ParameterMissing). **Client/SDK assertions and any code that switched on `Code=65535` for these paths must be re-checked** (the go_client e2e assertions were already aligned in this PR). **B. Prometheus `status` label contract change (externally visible).** The proxy metric's coarse `fail` / `rejected` values are split into `fail_input` / `fail_system` and `rejected_user` / `rejected_system` (in `requestutil.ParseMetricLabel`; auth/privilege rejections count as `rejected_user`), so dashboards can attribute a failure to caller vs operator. **Dashboards/alerts querying `status="fail"` must migrate to `status=~"fail_.*"`, and `status="rejected"` to `status=~"rejected_.*"`.** The in-repo Grafana dashboard is already migrated; external dashboards built on the old values silently go empty after upgrade. This is the one change that requires an ops-side migration. **C. Retriability semantics.** - C1: `merr.Status(err)` now forces `Retriable=false` when the error is an `InputError` — a malformed request can never succeed on blind retry, so clients never get the self-contradictory "your input is wrong but you may retry". - C2: `retry.Do` short-circuits an `InputError` (non-retriable) — **but only when the caller did not pass a `RetryErr` predicate**. The check is an `if c.isRetryErr != nil { ... } else if InputError { ... }` *mutually exclusive* branch (`pkg/util/retry/retry.go`): an explicit `RetryErr` takes precedence and bypasses the InputError abort. `retry.Handle` deliberately does **not** apply the InputError abort (its callers signal abort via `shouldRetry=false`). Four flusher startup callsites that must retry through transient "not ready" errors were given explicit `RetryErr` escape hatches. **D. segcore (C++→Go) error classification.** A single shared Go-side table (`pkg/util/merr/segcore.go`) maps each segcore code to a merr sentinel + InputError/signal category, replacing scattered hand-written `if errorCode == ...` switches in the cgo wrappers. **Wire `Code` values change for every segcore pass-through error, not just the remapped ones.** Named sentinels remap (C++ `2003` → merr `2001`, `2033` → `2002`, Folly/Knowhere codes likewise); **all remaining pass-through codes (`2004`–`2043`, previously surfaced to clients as raw C++ enum values) now serialize as `2000`** (`ErrSegcore`), with the original C++ code preserved in the `Reason` text (`segcoreCode=...`); unknown/future codes collapse to `2000` as well (pinned by the `wire_code_projection` test). Transient segcore classes (object storage / file IO / OOM / mmap / FieldNotLoaded — 11 codes) now report `Retriable=true`. **Any client switching on raw segcore codes in the `2004`–`2043` range must be re-checked**; the in-Reason code remains available for diagnostics. Signal codes (PretendFinished / FollyCancel) are recognized centrally. `errors.Is`-based control flow on these (e.g. scheduler skip/retry) is preserved. **E. InputError classification (25 sentinels + dynamic marks).** 25 sentinels in `errors.go` carry `WithErrorType(InputError)` (the Collection / ResourceGroup / Database families, `ErrIndexDuplicate`, `ErrParameterInvalid`, `ErrPrivilegeNotAuthenticated`, `ErrImportFailed`, `ErrQueryPlan`, ...), plus dynamic marks for the 8 segcore input codes (ExprInvalid, DimNotMatch, MetricTypeInvalid, FieldIDInvalid, ...) and `WrapErrAsInputError`. The widest blast radius is `ErrParameterInvalid` (1100): ~2335 `WrapErrParameterInvalid*` callsites now classify as input / non-retriable. Because of C1/C2 this changes retriability for any path that returns these. **The audit to confirm no transient path was mis-marked is the single most important review item** (see Part 3). One reverse correction: storage field-stats parsing moved from `ErrParameterInvalid` (input) to `ErrDataIntegrity` — a corrupted stored stat is data corruption, not user input. ### Part 3 — Known risks & traps (called out proactively) 1. **`merr.Wrap` vs `WrapErr*Err` (code-masking).** `WrapErr*Err` builds a `wrappedMilvusError{sentinel: ErrServiceInternal}` whose `code()` returns the *outer* sentinel — it overwrites the inner typed code and hides the `errors.Is` chain. This is intentional (use it to *deliberately* downgrade), but it was a recurring conversion defect; the rule "add context with `merr.Wrap`, downgrade with `WrapErr*Err`" is enforced by convention and reviewed across the diff. 2. **InputError × `retry.Do` blast radius.** Marking a sentinel `InputError` makes any `retry.Do(...)` without a `RetryErr` predicate stop retrying it. Reviewers should sanity-check that no transient use of the 19 newly-marked sentinels (especially `ErrParameterInvalid`) sits inside a retry loop that needed to keep spinning. The known flusher cases were handled (see C2). 3. **The ~340 raw-error allowlist.** What remains as bare `errors.New` is, by design: package-level sentinels (caught by `errors.Is`), `//go:build test` sites, and out-of-band trees (`cmd/`, `tests/`, codegen, walimpls). The linter only bans the *direct-return* form; assignment-then-return escapes and the full no-exceptions ban are deferred to an AST-based linter (Tier 2, documented). 4. **segcore C++ second step deferred.** This PR unifies classification on the Go side; splitting the dual-semantic C++ codes at the source is a follow-up. --- ## Validation - `make verifiers`: Go side clean (gofmt + static-check across modules, including the new `rawmerrerror` rule with a 0-hit baseline repo-wide). - `make test-go`: passing; the one real regression introduced (a datanode `invalid_task_type` assertion shifting `1` → `5` from a ParameterInvalid conversion) was fixed in-tree. - go_client e2e CreateIndex assertions aligned to the new merr messages. --------- Signed-off-by: zhenshan.cao <[email protected]> Co-authored-by: Claude Opus 4.7 (1M context) <[email protected]>	2026-06-12 15:04:51 -07:00
jiaqizhoandEC2 Default User	385caab437	fix: use C++20 for Conan build profile (#50294) ... issue: https://github.com/milvus-io/milvus/issues/47425 Milvus already asks Conan to build the C++ third-party stack with C++20 for the host profile, but Conan keeps the build profile separate. That means packages used during the build can still fall back to the default C++ standard, which caused protobuf and abseil to be compiled as gnu17 in some environments and fail during the dependency build. This change passes C++20 to the Conan build profile as well, so both the host context and the build context use the same language level. It applies the setting to both Linux Conan install branches in the third-party build script, including the gcc4 ABI path and the normal libstdc++11 path. Signed-off-by: jiaqizho <[email protected]> Co-authored-by: EC2 Default User <[email protected]>	2026-06-09 11:24:19 +08:00
groot	965e4b1564	build(deps): Upgrade knowhere and manage milvus-common via conan (#49912) ... issue: https://github.com/milvus-io/milvus/issues/47425 # Plan: upgrade Milvus to Knowhere with Conan `milvus-common` and Conan OpenBLAS ## Context Milvus currently fetches both `knowhere` and `milvus-common` from source with CMake `FetchContent`. The requested Knowhere commit `0e53e57cd1f67f74decbdf9af42e74d0e3850155` expects `milvus-common` and Linux OpenBLAS from Conan 2.x instead: - `milvus-common/1.0.0-9ca5ea6@milvus/dev#274d428d85f1d3d996e1092f0c9c7144` - `find_package(milvus-common REQUIRED)` and target `milvus-common::milvus-common` - Linux `openblas/0.3.30` with CMake target `OpenBLAS::OpenBLAS` - macOS Apple BLAS/LAPACK via `BLA_VENDOR Apple`, not Conan OpenBLAS Milvus also currently installs system OpenBLAS in builder Dockerfiles, runtime Dockerfiles, dependency scripts, and packaging scripts. The intended outcome is one build-time OpenBLAS source on Linux: Conan. Runtime packaging should use the Conan-built shared libraries instead of relying on distro `libopenblas-dev`/`openblas-devel` where possible. ## Implementation 1. **Update Conan dependency pins in `internal/core/conanfile.py`.** - Add direct requirement `milvus-common/1.0.0-9ca5ea6@milvus/dev#274d428d85f1d3d996e1092f0c9c7144`. - Align dependencies required by Knowhere `0e53e57c`: - `folly/2026.04.20.00@milvus/dev#06852bea5b6449f0c4eb0df002b5779c` - `fmt/11.2.0#eb98daa559c7c59d591f4720dde4cd5c` - `lz4/1.10.0#982d9b673900f665a1da109e09c17cab` - `fast_float/8.0.0@milvus/dev#c7802833c74c5a86ffed70e4af1a795e` - Linux-only `openblas/0.3.30` - Add `openblas/*:dynamic_arch = True` to default options to match Knowhere. 2. **Move Milvus core build settings to C++20.** - Change `internal/core/CMakeLists.txt` from `CMAKE_CXX_STANDARD 17` to `20`. - Keep `OPENTELEMETRY_STL_VERSION=2017` unchanged; it controls OpenTelemetry ABI mode, not the language standard. - Change all `compiler.cppstd=17` Conan install settings in `scripts/3rdparty_build.sh` to `compiler.cppstd=20`. - Review `internal/core/src/storage/gcp-native-storage/CMakeLists.txt`; if it inherits incompatible flags or includes C++20-dependent headers, change its local `CMAKE_CXX_STANDARD 17` to `20` too. 3. **Expose Conan packages to Milvus CMake.** - Add `find_package(milvus-common REQUIRED)` in `internal/core/CMakeLists.txt` near the existing Conan `find_package(...)` block. - Add Linux-only `find_package(OpenBLAS CONFIG REQUIRED)` in `internal/core/CMakeLists.txt` so missing Conan OpenBLAS fails before Knowhere configures. - Remove `-DOpenBLAS_SOURCE=AUTO` from `internal/core/build.sh`; the requested Knowhere commit no longer consumes that flag and directly calls `find_package(OpenBLAS CONFIG REQUIRED)` on non-Apple platforms. 4. **Stop executing source FetchContent for `milvus-common`.** - Remove `add_subdirectory(milvus-common)` from `internal/core/thirdparty/CMakeLists.txt`. - Keep `add_subdirectory(knowhere)`. - Do not keep a compatibility alias to the old source target; all consumers should use the Conan target. - Leave `internal/core/thirdparty/milvus-common/` files in place for the first pass unless cleanup is explicitly desired; they will no longer be active once the subdirectory is removed. 5. **Pin Knowhere to the requested commit.** - Update `internal/core/thirdparty/knowhere/CMakeLists.txt`: - `KNOWHERE_VERSION` -> `0e53e57cd1f67f74decbdf9af42e74d0e3850155`. - Use the full SHA for deterministic FetchContent checkout. 6. **Migrate CMake links/includes from source target to Conan target.** - In `internal/core/src/CMakeLists.txt`: - Remove `${MILVUS_COMMON_INCLUDE_DIR}` from include directories. - Add `milvus-common::milvus-common` to `CONAN_TARGETS` so `milvus_conan_deps` propagates include directories to object libraries. - Remove plain `milvus-common` from `LINK_TARGETS`. - In `internal/core/unittest/CMakeLists.txt`: - Remove `${MILVUS_COMMON_INCLUDE_DIR}` from include directories. - Remove plain `milvus-common` from `all_tests` and `test_json_uint64` link lists; both already link `milvus_conan_deps`, which will carry `milvus-common::milvus-common`. 7. **Adjust build-time system OpenBLAS installs.** - Remove Linux build-time OpenBLAS package installs that can shadow or duplicate Conan OpenBLAS: - `scripts/install_deps.sh`: remove Ubuntu `libopenblas-dev`; remove Rocky/Amazon/CentOS `openblas-devel` where only needed for C++ core build. - `build/docker/builder/cpu/ubuntu20.04/Dockerfile`, `ubuntu22.04/Dockerfile`, `ubuntu24.04/Dockerfile`: remove `libopenblas-dev`. - `build/docker/builder/gpu/ubuntu20.04/Dockerfile`, `gpu/ubuntu22.04/Dockerfile`: remove `libopenblas-dev`. - `build/docker/builder/cpu/amazonlinux2023/Dockerfile`, `rockylinux9/Dockerfile`: remove `openblas-devel` and any OpenBLAS header symlink workaround that only supported system OpenBLAS. - `build/deb/build_deb.sh`: remove build-time `libopenblas-dev` install if no longer needed after Conan generation. - Keep Fortran/toolchain packages needed to build Conan OpenBLAS from source when binary packages are missing. - Do not change `scripts/install_deps_embd.sh` or `scripts/install_deps_msys.sh` in the first pass unless the target build path uses them; they are separate embedded/MSYS dependency paths and may still require manual/system OpenBLAS. - On macOS, remove `openblas` from `.github/workflows/mac.yaml` only after confirming the new Knowhere path uses Apple BLAS/LAPACK and Milvus has no other macOS OpenBLAS consumer. 8. **Adjust runtime/package OpenBLAS handling carefully.** - The runtime should still include `libopenblas.so*` if the Milvus binary or Conan-built Knowhere/Faiss links it dynamically. - Prefer copying the Conan-provided OpenBLAS shared library from `cmake_build/lib` or the Conan package output instead of installing distro development packages in runtime images. - Update packaging sites that currently copy or depend on system OpenBLAS: - `build/deb/build_deb.sh`: change the copied source from `/usr/lib/x86_64-linux-gnu/libopenblas.so.0` to the Conan/build output path if present. - `build/rpm/milvus.spec`: stop assuming `/usr/lib/libopenblas-r0.3.9.so`; use the packaged Conan/build output library. - Runtime Dockerfiles under `build/docker/milvus/**`: remove `libopenblas-dev`/`openblas-devel` only after confirming the image gets the Conan OpenBLAS `.so` with Milvus libs. - `build/docker/milvus/gpu/ubuntu20.04/Dockerfile.base`: review the explicit `milvusdb/openblas` stage; replace it with the Conan-produced library if that image path is still active. - Keep runtime `libgomp1`/`libgomp` and `libaio` if required by OpenMP/libaio-linked dependencies. 9. **Clean stale build state before verification.** - Use a fresh `cmake_build` or remove the old FetchContent checkout/cache before rebuilding; stale `MILVUS_COMMON_INCLUDE_DIR`, old `knowhere-src`, or system OpenBLAS paths can hide integration errors. Signed-off-by: yhmo <[email protected]>	2026-06-03 14:48:19 +08:00
yihao.daiandClaude Opus 4.6	0ca60c5b47	feat: add commit_timestamp to SegmentInfo for correct MVCC/TTL/GC on import/CDC segments (#48472) ... ## What this PR does Adds `commit_timestamp` (uint64) to `SegmentInfo` and `SegmentLoadInfo` so that import/CDC segments use their logical commit time — not raw row timestamps — for all temporal decisions. **Problem:** When bulk-insert imports rows, the row timestamps generated during the import process may be slightly older than the actual commit time. Every time-based check in the system (MVCC snapshot visibility, GC eligibility, collection-level TTL compaction triggering, delete-buffer anchoring) sees these outdated timestamps instead of the actual commit time, producing correctness bugs. **Solution:** Two-layer approach: 1. **C++ segcore (load-time overwrite):** During `LoadFieldData`, the in-memory timestamp column is overwritten to `commit_ts_` when `commit_ts_ != 0`. This makes existing MVCC/delete logic work correctly with zero hot-path changes. 2. **Compaction normalization:** During compaction, all compaction paths (mix/sort/clustering) rewrite row timestamps to `commit_ts` in output binlogs, then set `CommitTimestamp = 0` on the output segment. After first compaction, the segment becomes a normal segment with no special handling needed. `commit_ts` is a temporary state that only exists between import and first compaction. **Key design decisions:** - **TTL field (per-row):** Not affected by commit_ts. TTL field values represent user-specified expiration intent and are honored as-is. - **Collection-level TTL:** Uses `max(row_ts, commit_ts)` as the effective row age to prevent premature expiration. - **Delete/Upsert before commit_ts:** A delete/upsert with `ts < commit_ts` does NOT take effect, because the row did not exist at that time. The original `search_pk(pk, delete_ts)` logic handles this correctly since `row_ts` is overwritten to `commit_ts`. - **CommitTimestamp assignment:** Currently TODO in `import_checker.go`, pending companion PR for 2PC commit flow. ## Changes - `pkg/proto/data_coord.proto`, `query_coord.proto`, `segcore.proto`: add `commit_timestamp` field - `internal/datacoord/segment_info.go`: `segmentEffectiveTs`/`segmentEffectiveDmlTs`/`effectiveTimestamp` helpers - `internal/datacoord/meta.go`: compaction completion mutations set `CommitTimestamp=0` and normalize position timestamps via `normalizePositionTimestamp` helper - `internal/datacoord/compaction_trigger.go`: use `effectiveTimestamp` for TTL compaction trigger - `internal/datacoord/handler.go`, `garbage_collector.go`, `compaction_task_l0.go`: use effective timestamp helpers - `internal/querycoordv2`: propagate `CommitTimestamp` in `PackSegmentLoadInfo` - `internal/querynodev2/delegator`: use `segmentEffectiveTs` for delete-buffer pin/list anchoring - `internal/core/src/segcore/ChunkedSegmentSealedImpl`: overwrite timestamp column at load time when `commit_ts_ != 0` - `internal/datanode/compactor/timestamp_overwrite.go`: Record/Reader wrappers for compaction timestamp normalization - `internal/datanode/compactor/{mix,merge_sort,sort,clustering}_compactor.go`: apply timestamp overwrite during compaction ## Tests - Unit tests: DataCoord helpers, `UpdateCommitTimestamp` operator, `GenSnapshot` filter, GC eligibility, TTL compaction trigger - Unit tests: QueryNode `segmentEffectiveTs`, delete-buffer pin at commit_ts - Unit tests: `timestamp_overwrite.go` Record/Reader wrappers - C++ tests: MVCC gate, TTL gate, pre-commit delete not applied, normal segment unchanged - Integration tests: MVCC visibility, delete/upsert on import segment, delete/upsert before commit_ts (should not apply), compaction normalizes commit_ts to 0 ## issue issue: #48471 design doc: https://github.com/milvus-io/milvus-design-docs/blob/main/design_docs/20260324-commit-timestamp.md --------- Signed-off-by: Yihao Dai <[email protected]> Signed-off-by: bigsheeper <[email protected]> Co-authored-by: Claude Opus 4.6 (1M context) <[email protected]>	2026-05-26 10:50:33 +08:00
cai.zhangandClaude Opus 4.7	73eec01a0c	fix: disable SDK default checksum to fix non-AWS S3 uploads (#49978) ... issue: #49977 ## Summary AWS SDK C++ 1.11.692 (bumped from 1.11.352 in #47810) changed two defaults that together break PutObject against non-AWS S3-compatible storage: 1. `PutObjectRequest::GetChecksumAlgorithmName()` returns `"crc64nvme"` by default 2. `ClientConfiguration::checksumConfig::requestChecksumCalculation` defaults to `WHEN_SUPPORTED` `ChecksumInterceptor` then attaches a request hash to every PutObject, and the V4 signer switches the request to `Transfer-Encoding: aws-chunked` + `x-amz-content-sha256: STREAMING-UNSIGNED-PAYLOAD-TRAILER`. Aliyun OSS, Tencent COS, Huawei OBS, and GCP all reject this streaming form. This makes any `MinioChunkManager::PutObjectBuffer` call fail in an infinite `JobStateRetry` loop — json_stats / bson_inverted shared_key_index builds and any non-DiskANN-style index upload (IVF_*, HNSW, FLAT) are affected. ## Fix Restrict `requestChecksumCalculation` to `WHEN_REQUIRED` on non-AWS providers so the SDK only adds checksums when the operation model demands them (PutObject does not). Same approach as milvus-io/milvus-storage#500. Patched in both client construction paths: - `MinioChunkManager::MinioChunkManager` (storage_type=minio, address-based dispatch) — gated on `storageType != S3` - `ChunkManager.cpp` subclasses (storage_type=remote, cloud_provider-based dispatch) — applies in `GcpChunkManager`, `AliyunChunkManager`, `TencentCloudChunkManager`, `HuaweiCloudChunkManager`. `AwsChunkManager` keeps the default `WHEN_SUPPORTED`. CARDINAL_TIERED / DiskANN indexes already work because they upload via Arrow's `S3FileSystem` provided by milvus-storage, which was fixed in #500. ## Test plan - [x] Added `TEST(MinioChecksumConfig, OverridesAreWhenRequired)` in `test_storage.cpp` — asserts the override flips both directions from `WHEN_SUPPORTED` to `WHEN_REQUIRED` - [ ] Build / CI green - [ ] Manual smoke test on Aliyun OSS endpoint — index build (IVF_SQ8) and json_stats no longer fail with `x-oss-ec=0017-00000804`. CI cannot regression-detect this since MinIO doesn't reject `aws-chunked`; a real Aliyun OSS endpoint is required. ## Workaround (no code change) Setting `AWS_REQUEST_CHECKSUM_CALCULATION=when_required` and `AWS_RESPONSE_CHECKSUM_VALIDATION=when_required` on milvus pods also fixes this — the AWS SDK reads them at `ClientConfiguration` default-construct time, so it works across all bundled SDK copies without rebuilding. Useful for hotfix on running clusters until this PR ships. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: cai.zhang <[email protected]> Co-authored-by: Claude Opus 4.7 (1M context) <[email protected]>	2026-05-26 05:34:32 +08:00
yihao.daiandClaude Opus 4.7	66fa592752	enhance: revert index_store_path full-path plumbing to montage assembly (#49805) ... ## Summary PR #49325 introduced collection-rooted index storage and also added a precomputed `index_store_path` string flowing through the build path. This PR scopes that back down: the collection-rooted layout stays, but the precomputed `index_store_path` string is removed from the write/build path. Build/write path: - DataCoord sends `IndexStorePathVersion` with the build job. - DataNode forwards the version to C++ indexbuilder. - C++ `FileManager` assembles the index object prefix locally from `(path_version, collID, partID, segID, buildID, indexVersion)`. Read/load path: - Go keeps passing full `IndexFilePaths` through DataCoord -> QueryCoord -> QueryNode -> cgo/segcore. - Load-side protos stay on full paths; no `index_file_keys` / `index_store_path_version` split is added there. - C++ load keeps PR #49325's direct full-path open behavior for `index_files/...` and `index_v1/...` paths. The v1 root is intentionally named `index_v1` before release. v0 remains `index_files`. issue: #49086 ## Compatibility note This PR is wire-incompatible with the short-lived #49325-era internal schema that carried `index_store_path`. That is intentional and acceptable because no release has shipped between the merge of #49325 and this rework. In other words, there are no released binaries or released persisted wire data that rely on the #49325-era `index_store_path` field semantics. This PR treats the proto changes as a pre-release schema correction rather than a compatibility migration. Because of that: - `BuildIndexInfo.index_store_path = 18` is replaced by `index_store_path_version = 18`. - `CreateJobRequest.index_store_path = 21` is removed. - `CreateJobRequest.index_store_path_version = 36` is kept. - We do not reserve the removed field numbers/names in this PR; if this schema had crossed a release boundary, we would reserve them and use fresh field numbers instead. ## What changes **Protos** - `worker.proto`: drop `CreateJobRequest.index_store_path`; keep `CreateJobRequest.index_store_path_version`. - `index_cgo_msg.proto`: replace `BuildIndexInfo.index_store_path` with `BuildIndexInfo.index_store_path_version`. - Read/load-side protos keep their full-path shape: no `IndexFilePaths -> IndexFileKeys` rename and no load-side `index_store_path_version` propagation. **C++** - `storage::IndexMeta`: replace `std::string index_store_path` with `IndexStorePathVersion index_store_path_version`. - `FileManager::GetRemoteIndexObjectPrefix()`: assemble v0/v1 prefixes from path version and IDs. - `FileManager::OpenInputStream()`: keep direct full-path open support for read/load paths. - `indexbuilder/index_c.cpp`: pass `index_store_path_version` into `IndexMeta`. - `DiskFileManagerTest`: cover v0 and v1 prefix assembly. **Go** - DataCoord `task_index.go`: stop building/sending `IndexStorePath`; continue sending `IndexStorePathVersion`. - DataNode `index/task_index.go`: forward `IndexStorePathVersion` to C++ build info. - DataNode `index_services.go`: drop logs for the removed `index_store_path` field. - DataCoord FinishTask validation and `SegmentIndex.IndexStorePathVersion` metadata remain unchanged. - GC/copy/import/snapshot/read path keep full-path behavior; only expected v1 root strings are updated from `index_files_v1` to `index_v1`. ## Notes for reviewers - The read/load path intentionally remains full-path based. QueryNode should not convert full paths to basenames or parse path version for load. - `IndexFilePathInfo.IndexStorePathVersion` remains as DataCoord metadata, but QueryCoord does not forward it for loading. - `metautil.IndexPathBuilder` remains needed for Go-side object storage operations such as GC, snapshot, copy-segment, and DataCoord `GetIndexInfos` path construction. - Mixed-version concerns are limited to unreleased #49325-era binaries, which is why this PR can remove/retype `index_store_path` without compatibility scaffolding. ## Test plan - [x] C++ build: `ninja milvus_core` clean. - [x] `DiskAnnFileManagerTest.GetRemoteIndexObjectPrefix_{V0BuildRooted,V1CollectionRooted}` covers v0/v1 prefix assembly. - [x] Updated Go/C++ tests for the `index_v1` root rename. - [ ] CI on this PR: code-check, UT, E2E. Signed-off-by: bigsheeper <[email protected]> Co-authored-by: Claude Opus 4.7 (1M context) <[email protected]>	2026-05-16 00:02:29 +08:00
sre-ci-robotandgithub-actions[bot]	b664969c3b	[automated] Bump milvus version to v3.0-beta (#49681) ... Bump milvus version to v3.0-beta Signed-off-by: sre-ci-robot [email protected] Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>	2026-05-11 15:44:09 +08:00
Buqian Zheng	99097abea8	feat: support Sort/Bitmap/Hybrid index types for JSON Path Index (#48953) ... this PR allows to create Sort/Bitmap/Hybrid index for json path index, and also removed the `JsonInvertedIndex` class, to use `InvertedIndexTantivy` directly for Inverted index type. C++ changes: - Add ConvertJsonToTypedFieldData<T>() to extract typed values from JSON field data with separate tracking of non_exist_offsets for EXISTS semantics - Add JsonScalarIndexWrapper<T, BaseIndex> template for Sort/Bitmap with dual file-manager pattern (original JSON schema for reading, cast-type schema for base index dispatching) - Add JsonHybridScalarIndex<T> with validity-aware cardinality counting - Add IndexBase::Exists() virtual method; override in Sort/Bitmap/Hybrid wrappers using non_exist_offsets (serialized via WriteEntries/LoadEntries) - Simplify ExistsExpr to use index->Exists() uniformly - Extend IndexFactory::CreateJsonIndex() to route STL_SORT/BITMAP/HYBRID Go changes: - Update STL_SORT/Bitmap/Hybrid checkers to accept JSON with cast_type and json_path validation - Change AUTOINDEX default for JSON from INVERTED to HYBRID - Bump ScalarIndexEngineVersion to 4 - Add version gate in DataCoord CreateIndex and snapshot RestoreIndexes - Update test fixtures to cover new (index_type, cast_type) combinations <img width="2380" height="708" alt="image" src="https://github.com/user-attachments/assets/8b3923a0-2cb3-4af7-b73a-b76d1d1ec2d0" /> issue: https://github.com/milvus-io/milvus/issues/48954 design-doc: https://github.com/milvus-io/milvus-design-docs/blob/main/design_docs/20260410-json_path_index_multi_type.md --------- Signed-off-by: Buqian Zheng <[email protected]>	2026-05-08 17:36:12 +08:00
Konstantin VikhlyantsevandClaude Sonnet 4.6	34c9f57ad5	enhance: support Conan installation on PEP 668 systems (Ubuntu 24.04+) (#48146) ... Related issue: #48145 scripts/install_deps.sh fails to install Conan on Ubuntu 24.04 in two scenarios: 1. System Python (PEP 668): `pip install --user conan` fails with `error: externally-managed-environment` 2. Active virtual environment: `pip install --user conan` fails with `ERROR: Can not perform a '--user' install. User site-packages are not visible in this virtualenv.` Fall back to an isolated venv (~/.venv/conan) with a ~/.local/bin/conan symlink when --user install is unavailable. Behavior on Ubuntu 20.04/22.04, macOS, and Rocky Linux is unchanged. Signed-off-by: Vikhlyantsev Konstantin <[email protected]> Co-authored-by: Claude Sonnet 4.6 <[email protected]>	2026-05-05 07:26:06 +08:00
congqixia	7311d450a0	enhance: bump Go dependencies to v3 modules (#49485) ... Related to #49398 Bump Go module references from pkg/v2 and milvus-proto/go-api/v2 to pkg/v3 and milvus-proto/go-api/v3 so the client tracks the Milvus 3.x release line. This prepares the repository for the upcoming 3.x.y release by aligning imports, module dependencies, and proto API references with the new major-version module paths. --------- Signed-off-by: Congqi Xia <[email protected]>	2026-05-01 10:30:13 +08:00
zhikunyaoandClaude Opus 4.6	10a6c5abba	doc: add scripts/milvus-debug.sh for coredump analysis with debug image (#49084) ... ## Summary Add `scripts/milvus-debug.sh` to automate GDB coredump analysis using the `-debug` image that accompanies each Milvus release starting from v2.6.14. ## Background Starting with v2.6.14, each Milvus release publishes two image variants: | Tag | Content | Size | Use | |-----|---------|------|-----| | `milvusdb/milvus:<tag>` | **stripped** (production) | ~1/3 | Production deployment | | `milvusdb/milvus:<tag>-debug` | unstripped (full symbols) | full | GDB coredump analysis | The stripped production image is **byte-identical** to the debug image in its code sections (`strip --strip-debug` removes only `.debug_*` sections and symbol tables). So GDB can use the debug image's symbols to resolve addresses in a coredump produced by the stripped image. ## What the script does 1. Pulls the debug image 2. Mounts the coredump file read-only into a container 3. Installs `gdb` on demand (not present in the runtime image by default) 4. Launches GDB with `solib-search-path=/milvus/lib` so symbols from all shared libraries (`libknowhere.so`, `libtantivy-binding.so`, etc.) resolve correctly 5. Auto-prints the initial backtrace ## Typical usage after a production crash ```bash # 1. Copy the coredump out of the crashed pod kubectl cp <ns>/<pod>:/tmp/cores/core.<pid> ./core.<pid> # 2. Run this script (GDB starts automatically with backtrace) ./scripts/milvus-debug.sh ./core.<pid> \ milvusdb/milvus:v2.6.14 milvusdb/milvus:v2.6.14-debug ``` ## Test plan - [x] `bash -n scripts/milvus-debug.sh` (syntax check passes) - [x] Mirrors the documented procedure in the strip-image-coredump-debug-guide Feishu doc - [ ] Manual validation on a real coredump (waiting for v2.6.14 production crash; can be exercised with any coredump for now) 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Zhikun Yao <[email protected]> Co-authored-by: Claude Opus 4.6 <[email protected]>	2026-04-29 17:31:51 +08:00
marcelo-cjlandClaude Opus 4.7	9fe5cddd02	enhance: preserve unchanged proto output timestamps (#49407) ... ## Summary - Preserve generated proto output mtimes when regenerated content is unchanged. - Back up generated files under `cmake_build/.proto-backup.*` during `generate_proto.sh`. - Avoid unnecessary C++ incremental rebuilds caused by unchanged `.pb.h` / `.pb.cc` timestamp churn. issue: #49406 Closes #49406 ## Test plan - [x] `bash -n scripts/generate_proto.sh` - [x] `git diff --check master...HEAD` - [ ] `make milvus-gpu` twice and confirm unchanged proto outputs no longer trigger proto-dependent C++ rebuilds 🤖 Generated with [Claude Code](https://claude.com/claude-code) Signed-off-by: marcelo-cjl <[email protected]> Co-authored-by: Claude Opus 4.7 (1M context) <[email protected]>	2026-04-29 11:13:51 +08:00
zhagnluandluzhang	2a09964041	fix: use configured parallel jobs for ninja install (#49405) ... #48783 Signed-off-by: luzhang <[email protected]> Co-authored-by: luzhang <[email protected]>	2026-04-28 20:39:51 +08:00
wei liu	7fc234d1fe	build(deps): bump rust to 1.92 in builder images and install scripts (#49362) ... issue: [#45691](https://github.com/milvus-io/milvus/issues/45691) ## Summary Bump rust toolchain `1.89 → 1.92` only in installation paths (Dockerfiles + install scripts). ## Why milvus-storage HEAD upgrades `iceberg` crate `0.8 → 0.9`, which declares `edition = "2024"` and `rust-version = "1.92"`. The new MSRV requires the builder image to ship rust 1.92 before milvus can take that milvus-storage commit. This PR is the **first step**: 1. Bump rust toolchain in installation scripts and Dockerfiles. 2. Once merged, builder image is republished with rust 1.92. 3. Follow-up PR will bump `.env` `DATE_VERSION` to the new tag, switch `cargo +1.89` → `cargo +1.92` in `internal/core/thirdparty/{tantivy,opendal}/CMakeLists.txt`, and bump milvus-storage to the iceberg-0.9 commit. This split avoids the chicken-egg problem: changing the cargo pin or milvus-storage version in this PR would fail CI, since the currently pinned builder image only ships rust 1.89. ## Files changed | File | Change | |---|---| | `scripts/install_deps.sh` | `RUST_VERSION="1.89"` → `1.92` | | `scripts/3rdparty_build.sh` | rust toolchain `1.89` → `1.92` (mac + linux paths) | | `build/docker/builder/{cpu,gpu}/*/Dockerfile` (7 files) | rustup `--default-toolchain=1.89` → `1.92` | | `DEVELOPMENT.md` | doc table rust version | ## Test plan - [ ] CI passes on existing builder image (no behaviour change for current build path) - [ ] After merge, builder image is republished with rust 1.92 - [ ] Follow-up PR can then bump `.env` and milvus-storage Signed-off-by: Wei Liu <[email protected]>	2026-04-26 20:55:46 +08:00
zhagnluandluzhang	5409a81f40	feat: support TEXT column with LOB storage for large text fields (#47567) ... issue: #48783 design doc: https://github.com/milvus-io/milvus-design-docs/blob/main/design_docs/20260407-text_lob_storage.md Signed-off-by: luzhang <[email protected]> Co-authored-by: luzhang <[email protected]>	2026-04-24 14:25:45 +08:00
zhenshan.caoandClaude Opus 4.7	d77009d654	enhance: support CONAN_CMD override in 3rdparty_build.sh (#49107) ... Fixes #49106. Follow-up to #48165. Adds an opt-in `CONAN_CMD` environment variable so developers working on both master (Conan 2.x) and release-2.5/2.6 (Conan 1.x) on the same machine can select a Conan binary per-build without switching their default `conan`. See commit message for details. ## Test plan - [x] `make` with default conan (2.x) — unchanged behavior - [x] `CONAN_CMD=conan-1 make` — fails with clear version mismatch error - [x] `CONAN_CMD=/nonexistent make` — fails with clear error Signed-off-by: zhenshan.cao <[email protected]> Co-authored-by: Claude Opus 4.7 (1M context) <[email protected]>	2026-04-17 11:27:41 +08:00
groot	499619033f	enhance: Migrate to Conan 2.x (#48165) ... new JFrog artifactory for conan 2.x: https://milvus01.jfrog.io/ui/repos/tree/General/default-conan-local2 issue: https://github.com/milvus-io/milvus/issues/47425 conanfiles pr: https://github.com/milvus-io/conanfiles/pull/92 milvus-common: https://github.com/zilliztech/milvus-common/pull/69 milvus-storage: https://github.com/milvus-io/milvus-storage/pull/445 knowhere: https://github.com/zilliztech/knowhere/pull/1495 Signed-off-by: yhmo <[email protected]>	2026-04-16 18:31:42 +08:00
Alexander Guzhva	98245e12a4	enhance: Introduce the support of Intel SVS (#48799) ... related PRs: * https://github.com/zilliztech/VectorDBBench/pull/749 for vectordbbench * https://github.com/zilliztech/knowhere/pull/1537 for knowhere * https://github.com/milvus-io/milvus/pull/48798 for Milvus 3.0 Signed-off-by: Alexandr Guzhva <[email protected]>	2026-04-08 06:39:41 +08:00
3 people	0449ea35a1	test: fix C++ UT portability on macOS (#48741) ... ## Summary - Skip `AzureChunkManagerTest` via TCP probe when Azurite emulator is not running locally - Remove Linux-specific `read()` single-call size limit assertion from `UtilIndex.ReadFromFD` - Fix `IterativeFilter.SealedIndex` platform-dependent RNG: explicitly construct int16 column with guaranteed coverage then shuffle - Skip faiss mmap tests on macOS with `#ifdef __APPLE__` — `MmappedFileMappingOwner` is not implemented in faiss upstream for Apple platforms (`mapped_io.cpp` only covers Linux/FreeBSD and Windows) After these fixes: **6745 passed, 25 skipped (all with documented reasons), 0 failed** on macOS ARM64. issue: #48740 ## Test plan - [ ] Run `internal/core/output/unittest/all_tests` on macOS ARM64 — expect 0 FAILED - [ ] Run `internal/core/output/unittest/all_tests` on Linux — expect no regressions (Azure/mmap skips are guarded by `__APPLE__` or runtime checks) 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: james <[email protected]> Co-authored-by: james <[email protected]> Co-authored-by: Claude Sonnet 4.6 <[email protected]>	2026-04-04 14:10:39 +08:00
sre-ci-robotandgithub-actions[bot]	6da09b8b02	[automated] Bump milvus version to v2.6.14 (#48744) ... Bump milvus version to v2.6.14 Signed-off-by: sre-ci-robot [email protected] Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>	2026-04-04 10:21:35 +08:00
XiaofanandClaude Opus 4.6	79672130af	enhance: optimize C++ build speed by ~25% (#48696) ... ## Summary - Optimize C++ compilation speed with minimal code changes and zero runtime impact (except tantivy codegen-units) - Measured **25% faster** on full clean build (j8, cold ccache): 17m39s → 13m13s ## Changes ### Compiler & Linker (`internal/core/CMakeLists.txt`) - `-g1` instead of `-g` for Release/MinSizeRel (smaller debug info) - `-pipe` for faster compiler inter-stage I/O - `-gsplit-dwarf` on Linux (smaller .o files, faster linking) - Auto-detect faster linker: mold > lld > gold > bfd ### Precompiled Headers (`src/CMakeLists.txt` + `cmake/milvus_pch.hxx`) - Apply PCH (STL + glog + fmt) to 15 OBJECT targets - Controllable via `USE_PCH=ON/OFF` (default ON) ### Tantivy (`Cargo.toml`) - `codegen-units=4` (was 1) — ~1-2% runtime cost for text search only ### Conan (`conanfile.py`) - Disable 8 unused Boost compiled modules ### Build Scripts - Pass PCH/Unity options through `core_build.sh` - `CONAN_CPU_COUNT` for parallel downloads ## Benchmark (j8, cold ccache, full clean build) | Metric | Before | After | Improvement | |--------|--------|-------|-------------| | **Total** | 17m 39s | 13m 13s | **-25%** | | Milvus src | 11m 36s | 8m 42s | -25% | | Thirdparty | 6m 03s | 4m 31s | -25% | ## Runtime Impact - `codegen-units=4` (Tantivy): ~1-2% slower for full-text search tokenization - All other changes: **zero runtime impact** ## Test plan - [x] Full clean build passes (`libmilvus_core.so` generated) - [x] Cross-platform: Linux flags guarded, macOS paths preserved - [ ] CI build verification - [ ] Run full test suite to verify no regressions issue: #48695 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: xiaofanluan <[email protected]> Co-authored-by: Claude Opus 4.6 <[email protected]>	2026-04-03 23:57:36 +08:00
XiaofanandClaude Opus 4.6	c48aadd289	enhance: optimize LIKE pattern matching with SmartPatternMatcher replacing regex (#47310) ... issue: #47307 Signed-off-by: xiaofanluan <[email protected]> Co-authored-by: Claude Opus 4.6 (1M context) <[email protected]>	2026-03-27 17:09:30 +08:00
Li LiuandClaude Opus 4.6	b0ec4a2dd8	fix: upgrade Go to 1.25.8 for CVE-2025-68121, CVE-2026-27142, CVE-2026-25679 (#48286) ... issue: #48574 ## Summary - Upgrade Go from 1.24.12 to 1.25.8 across all go.mod files and Dockerfiles - Fixes CVE-2025-68121 (CRITICAL), CVE-2026-27142 (HIGH), CVE-2026-25679 (HIGH) in Go stdlib - All three CVEs affect the Go standard library and are resolved by upgrading to Go 1.25.8 ## Changes - Updated `go` directive in go.mod files (root, pkg/, client/, tests/go_client/, examples/) - Updated Go download URLs in Dockerfiles (build/docker/builder/) - Updated `toolchain` directives where present issue: https://github.com/milvus-io/milvus/issues/TBD ## Test plan - [x] `make milvus` builds successfully with Go 1.25.8 - [ ] CI passes 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Li Liu <[email protected]> Co-authored-by: Claude Opus 4.6 (1M context) <[email protected]>	2026-03-27 14:23:30 +08:00
XuanYang-cnandClaude Opus 4.6	c54b34a880	enhance: enable OpenSSL FIPS mode for Milvus (#48331) ... Layer 2 (OpenSSL FIPS) changes: - Add programmatic FIPS activation via OSSL_LIB_CTX_load_config in boring_enabled.go (gated by //go:build boringcrypto) - Add openssl-fips.cnf with fips + default providers and default_properties = fips=yes - Use absolute .include path for fipsmodule.cnf — OpenSSL resolves relative .include from the process working directory, not the config file's directory, causing silent FIPS provider load failure - Add RAND_bytes probe after config load to verify the FIPS provider is truly functional (EVP_default_properties_is_fips_enabled only checks the property string, not whether the provider loaded) - Dockerfiles: add openssl fipsinstall + OPENSSL_MODULES env var - Log OpenSSL FIPS status from C++ via EVP_default_properties_is_fips_enabled Layer 1 (Go BoringCrypto) changes: - Add GOEXPERIMENT=boringcrypto build flag (conditional on MILVUS_FIPS_ENABLED=ON) - Add boringEnabled() build-tagged functions for startup logging s2n-tls upgrade: - Override s2n 1.4.1 (from aws-c-io) to 1.6.0 in conanfile.py. s2n 1.4.1 only detects FIPS via the legacy OPENSSL_FIPS define (not set by OpenSSL 3.x). s2n 1.6.0 adds EVP_default_properties_is_fips_enabled() detection so s2n enters FIPS mode and uses RAND_bytes() through the FIPS provider. See also: #48202, #48301 Signed-off-by: yangxuan <[email protected]> Co-authored-by: Claude Opus 4.6 <[email protected]>	2026-03-23 15:09:28 +08:00
sre-ci-robotandgithub-actions[bot]	4c5ea64714	[automated] Bump milvus version to v2.6.13 (#48431) ... Bump milvus version to v2.6.13 Signed-off-by: sre-ci-robot [email protected] Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>	2026-03-23 11:07:28 +08:00
XiaofanandClaude Opus 4.6	7dbe7e89d1	enhance: use RelWithDebInfo build type for CI to reduce peak compiler memory (#48154) ... ## Summary Switch CI pipelines from Release (`-O3`) to RelWithDebInfo (`-O2 -g -DNDEBUG`), reducing peak compiler memory by ~15%. RelWithDebInfo is a standard CMake build type that uses `-O2` natively and includes debug symbols for better crash diagnostics in CI. Production image builds remain at Release (`-O3`). ## Changes - Update Jenkins (PR, PR-Arm, PR-for-go-sdk, UT-CPP) to use `mode=RelWithDebInfo` - Update GitHub Actions (`main.yaml`) to use `mode=RelWithDebInfo` - Allow all standard CMake build types in `3rdparty_build.sh` validation - Add gcc>=13 implicit include workaround in `CMakeLists.txt` ## Test plan - [ ] CI builds with `mode=RelWithDebInfo` use `-O2` instead of `-O3` - [ ] Default builds still use Release with `-O3` - [ ] ASAN CI jobs pass with reduced memory usage issue: #48400 Signed-off-by: xiaofan-luan <[email protected]> Signed-off-by: xiaofanluan <[email protected]> Co-authored-by: Claude Opus 4.6 (1M context) <[email protected]>	2026-03-20 13:17:26 +08:00
sijie-ni-0214	8900b46f45	fix: split macOS RPATH into separate -r flags for linker compatibility (#48241) ... ## Summary - macOS `ld` treats each `-r` flag as a single directory path, unlike Linux `ld` which accepts colon-separated paths - Since d69bdd288c added `cmake_build/lib` to `DYLD_LIBRARY_PATH`, `RPATH=$DYLD_LIBRARY_PATH` produces a colon-separated value like `path1:path2` - The Makefile's `-r $RPATH` embeds this as a single invalid path in the binary, causing `Library not loaded: @rpath/libmilvus-storage.dylib` at runtime - Fix: emit two separate `-r` flags so the Makefile expands to `-r path1 -r path2` ## Test plan - [x] Verified `RPATH` variable output: `path1 -r path2` (correct) - [x] Verified binary loads correctly after fix (`otool -l` shows two separate `LC_RPATH` entries) - [x] Verified Milvus standalone starts and serves requests successfully issue: #47809 Signed-off-by: sijie-ni-0214 <[email protected]>	2026-03-18 00:43:30 +08:00