mirror of
https://github.com/milvus-io/milvus.git
synced 2026-10-12 02:19:10 +00:00
master
492
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Jamesandxiaofanluan
|
f4ae66a2fd |
enhance: add startup flag for existing native profiling (#53997)
Milvus already provides jemalloc native heap profiling and `/debug/jemalloc/pprof/heap`. This PR only adds an opt-in startup flag, using the existing subprocess path so jemalloc receives its configuration before allocator initialization. ```sh milvus run querynode --native-profile curl -f http://localhost:9091/debug/jemalloc/pprof/heap -o native.heap ``` The flag enables profiling and allocation sampling while preserving other `MALLOC_CONF` options, including the sampling interval. Existing builds, HTTP routing, dump behavior, and dependencies are reused. The legacy HTTP endpoint requires a Linux `with_jemalloc` build with profiling support and remains subject to existing pprof/management routing; enabling management authentication hides legacy DefaultServeMux routes. Fixes #53985 Validation: focused CLI tests and Go vet passed; the startup-only revision also passed a Linux `go build -mod=readonly -tags dynamic,with_jemalloc ./cmd` and gosec on `cmd/milvus`. Formatting and diff checks passed. No service or profiling workload was started. Signed-off-by: xiaofanluan <[email protected]> Co-authored-by: xiaofanluan <[email protected]> |
||
|
Zhen Ye
|
5f5d656062 |
fix: wait for component registration before reporting healthz readiness (#53734)
issue: #53731 - health checks: require complete component registration before reporting readiness, synchronize registration state, and preserve Healthy, StandBy, and explicit unregistration behavior. - role startup: count actual components, treating legacy coordinator flags as a single MixCoord. Signed-off-by: chyezh <[email protected]> |
||
|
cai.zhangandClaude Fable 5.1
|
2cbbb72041 |
feat: add an in-tree extension mechanism for distributions (#52981)
issue: #52979 design doc: docs/design-docs/design_docs/20260831-in_tree_extension_mechanism.md A distribution that compiles its own behavior into the milvus binary needs little from milvus: a way to install its request hook, the QueryNode's tuning hook and the cipher without a `.so`, a callback that starts its control-plane engine when the coordinator becomes active and stops it on shutdown, and a declaration that it runs the deployment shape its coordinator behaviors are written for. This PR is that seam and nothing else. A stock binary installs nothing and behaves exactly as before. An earlier revision of this PR shipped a table of typed capabilities consulted from many places in the tree. Reviewing it against what the distribution actually needs showed most entries were constants per deployment (configuration), duplicates of what the request hook already does, or answers that belonged on the wire. The table is gone; what remains is small enough to state here. ## Public surface (`pkg/extension`) ```go func SetHook(h hook.Hook) // hookutil prefers it over proxy.soPath func InstalledHook() hook.Hook type QueryHook interface { ... } // the queryNode.soPath plug-in's method set func SetQueryHook(h QueryHook) // the QueryNode prefers it over queryNode.soPath func InstalledQueryHook() QueryHook func SetCipher(c hook.Cipher) // hookutil prefers it over cipherPlugin.soPathGo func InstalledCipher() hook.Cipher type Coordinator interface { // the coordinator as its own clients see it rootcoordpb.RootCoordClient querypb.QueryCoordClient datapb.DataCoordClient } type CoordinatorEngine interface { Start(ctx context.Context, coord Coordinator) error Stop() error } func SetCoordinatorEngine(e CoordinatorEngine) func InstalledCoordinatorEngine() CoordinatorEngine func SetForm() // marks a deployment form installed, in every role func FormInstalled() bool // the coordinators' form-gated behaviors read it func WithQueryResourceGroup(ctx, rg) context.Context // set by a hook's Before func QueryResourceGroupFromContext(ctx) string ``` A distribution calls the setters it needs, then `cmd/milvus.Main(os.Args)` (exported so a distribution can ship its own `main`). - **Request hook.** `hook.Hook` (milvus-proto) is consulted by the proxy's unary interceptor for every RPC on the gRPC and REST surfaces, and by hand by the service's two streams, `CreateReplicateStream` and `DumpMessages`, in the same order and with the handler under the context `Before` returns; a test fails if the service declares a stream that does not. A compiled-in hook is used in preference to `proxy.soPath`; configuring both is refused at start-up. The compiled-in hook is initialized as a plug-in is: `Init` with the `hook.*` configuration before it is stored, and the `hook.*` reload watcher re-inits it on a config change. A failing `Init` is fatal at start-up when a form is installed, whatever `common.panicWhenPluginFail` says; a hook installed without a form follows the setting like any plug-in. - **Query hook and cipher.** The QueryNode's tuning hook (`SetQueryHook`) and the Go half of the cipher pair (`SetCipher`) can be compiled in the same way. Each is used in preference to its so path (`queryNode.soPath`, `cipherPlugin.soPathGo`), is refused beside it, and gets the same `Init` calls and watchers a plug-in gets. `cipherPlugin.soPathCpp` still declares that encryption is on, so a compiled-in cipher in a deployment without it stays idle. `optimizers.QueryHook` is now an alias of `extension.QueryHook`. None of the three slots implies another or `FormInstalled`. - **Coordinator engine.** `mixcoord` starts the installed engine once the replica is ACTIVE (a standby never starts it), on the coordinator client it uses itself, and stops it on shutdown. A start failure on activation is fatal. The engine reaches the coordinator only through `Coordinator`, so it can do exactly what a proxy can do. - **Query resource-group mark.** `WithQueryResourceGroup` pins a query to one resource group: the shard client routes it to the leaders whose replica lives in that group (`ShardLeadersList.resource_groups`, added on master by #52716) and the proxy attributes its latency to that group. Nothing in a stock binary sets it. ## Behaviors a form turns on by declaring itself These apply only when a form is declared (`extension.SetForm()`, which a distribution does in every role); a stock binary keeps master's semantics unchanged. `FormInstalled` is independent of `SetHook`: a distribution may install a request hook without switching any coordinator behavior on. - **Delegators on regular query nodes.** Under a form, a replica's shard delegators live on its regular RW query nodes, which watch the channel and read the WAL remotely, instead of on the streaming node's embedded query node; the streaming node is kept for DDL and the write ahead log. This lets a form load the same collection into several resource groups, one streaming node total. The trade is stated in the design doc: a Strong-consistency search no longer takes the WAL-MVCC shortcut, and the manual-flush prepare step is skipped with a warning. This shape has been exercised on a real one-streaming-node / multi-RG cluster, not only in unit tests: `scripts/walkthrough.py` (create, bulk-import ingest, index, load into a query cluster, cold and warm query, drop) and the e2e suite (55 tests, including two resource groups loading the same collection) run against a local k8s lake-form instance, and five concurrent rounds of two resource groups querying one collection return correct results with both replicas intact. - **A load request speaks only for the resource groups it names.** Loading a collection into `rg_b` does not restate the whole placement and evict it from `rg_a`; the groups the request does not name keep their replica count. A request naming no group states the whole placement. Per-group load progress is readable through `ShowLoadCollections` with `resource_group` set (appended proto field; `-1` means the group holds no replica of the collection). A load that only adds resource groups or replicas to an already loaded collection keeps the collection serving (an incremental expansion) instead of resetting it to loading, whether the added replica lands in a group the request newly names or one that already serves it; the checkers load the added replicas, as they do on every other path. An expansion replayed after a crash stays idempotent. - **Index engine version with no QueryNode session.** With no QueryNode registered, datacoord answers the index engine version compiled into the coordinator itself instead of `0`, and bounds an explicit target version by that same value, on the assumption that a QueryNode started later runs the same image. ## Removed from the earlier revision `common.security.requireAPIKey`, the `proxy.internalDomain.*` listeners, `queryCoord.resourceGroupScopedLoad`, `queryCoord.reloadSegmentOnIndexDrop`, `dataCoord.index.allowVectorIndexDropOnLoadedCollection`, `dataCoord.index.queryNodesScaleToZero`, the `RefusalResponse` hook extension, the capability table and `Provider`, and the link-time primary configuration file name. Each is either a deployment constant (configuration a distribution ships in its own files), something the request hook already does, or a default this PR now makes unconditional. ## Compatibility - `ShowCollectionsRequest.resource_group` is the only proto field this PR appends; an old peer leaves it empty and both sides read empty as "no scope". `ShardLeadersList.resource_groups`, which the query mark also reads, was added on master by #52716, not here; an old peer leaves it empty too and empty is read as "unknown". - `hook.Hook` is milvus-proto's and unchanged. No new configuration key. - `ResetForTest` is built only under the `test` tag. ## Testing - `pkg/extension`: setters, getters, the context mark, and the split between `SetHook` and `SetForm` (each answers without the other). - hookutil: compiled-in hook used, refused beside a plug-in, absent by default, initialized with the hook configuration, re-initialized by the reload watcher, a failing `Init` fails start-up when a form is installed even with `common.panicWhenPluginFail` off but follows the setting without one; compiled-in cipher used when `soPathCpp` is set, refused beside `soPathGo`, idle without `soPathCpp`, re-initialized on a `cipherPlugin.*` edit. - querynode: compiled-in query hook used, refused beside `queryNode.soPath`, initialized with the `autoIndex.params.*` configuration, not installed when it cannot initialize. - proxy: a hook-pinned resource group reaches the search task; per-group latency series exist only for pinned requests; both streams consult the hook, and every stream the service declares is one of them. - mixcoord: the engine starts on activation only, receives the coordinator client, is stopped once. - querycoordv2: scoped `ShowLoadCollections` (100 / -1 / unknown group refused); scoped load placement with and without a form declared; the expansion predicate's legs, a request that only adds a replica to an already-loaded group, and a replayed expansion; delegators placed on a regular query node under a form and on a streaming query node without one. - datacoord: zero-session engine version and its upper bound with a form declared; `0` without. - `golangci-lint` with the repo config on every touched package. One deliberate difference from the "stock binary is untouched" claim, listed rather than hidden: a stock binary with the streaming service on bypasses the channel checker's version cache for a tick when a replica's read-write node set is empty at that tick (a streaming node mid-restart). The cost is one extra `checkReplica` per tick; the case it exists for, a deployment that starts a replica's nodes on demand, is the form's. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_015JwRUhmz28AXHyuAJTSs3E --------- Signed-off-by: Cai Zhang <[email protected]> Signed-off-by: cai.zhang <[email protected]> Co-authored-by: Claude Fable 5.1 <[email protected]> |
||
|
Jamesandxiaofanluan
|
37f3a829e7 |
fix: make local storage keys complete paths at every layer (#53145)
issue: #53051, #53052, #53152 ## Summary Align local storage around one path contract: a key is a complete absolute filesystem path, and the local loon/Arrow filesystem is rooted at `/`. Previously, some writers used `minio.rootPath` while others supplied complete local paths to an already-prefixed filesystem, placing files in different physical locations from those expected by readers and GC. Remote keys remain complete bucket-relative object keys; their filesystem-root convention is unchanged. ## Changes - Unify Go/C++ local filesystem roots and StorageV3 manifest-base construction across writers. Preserve the existing base/version pair when continuing a growing-source manifest. - Fix local JSON/TEXT stats reads, rejected-stats cleanup, and explore temporary paths. Exercise resolved V3 stats paths through the actual `LocalChunkManager.MultiRead` interface. - Resolve legacy local manifest bases in loaded metadata and locally referenced snapshots, without eager catalog or manifest-file writes. Preserve manifest CAS compatibility and pathless V3 summaries; copy/restore writes use the target storage root. - Require an absolute, nonempty `localStorage.path` at initialization, including on remote deployments where it is a cache directory. Freeze forbidden parameters against configuration-source refresh while retaining explicit runtime/test overrides, and derive mmap, pprof, and disk-capacity defaults from the same canonical startup value. - Make segment deletion prefixes segment-boundary aware, retain root/identity and snapshot checks, and add BM25 orphan scanning. These shared GC changes also apply to remote storage; this PR does not claim every remote code path is untouched. - Remove the local test-only root override and unused local-only property initialization chain; use the production storage-config initialization path. ## Automatic local upgrade Let `R = localStorage.path` and `M = minio.rootPath`. Before standalone components start, local deployments check the known legacy locations: | Existing data | Upgrade handling | | --- | --- | | Canonical files under `R/<layout>` | Leave in place. | | Double-root files under `R/<R-without-leading-slash>/<layout>` | Move into `R/<layout>` for `insert_log`, `text_log`, `json_stats`, `index_files`, and `index_v1`. | | Recognized late-2.6 unified packed indexes under the process CWD | Move into the corresponding directory under `R`, if visible to the new process. | | Legacy manifest base `M/insert_log/<c>/<p>/<s>`, with files under `R/M/insert_log/<c>/<p>/<s>` | **Do not move the files.** Resolve the loaded manifest base to that existing absolute location. New segment layouts use the canonical root; updates to an existing legacy manifest can retain its base. | Migration prefers directory/subtree rename and recursively merges overlapping directories. Cross-filesystem moves fall back to copy-then-remove. Existing same-name files cause a conflict rather than content comparison or overwrite, except a published copy recognized as this migration's own interrupted work. The copy path synchronizes the file and target directory chain before deleting the source. Remaining sources are rediscovered on retry; empty legacy directories are pruned. Once migrated, restarts probe the known old locations without rescanning canonical data. The initial pass walks legacy file metadata but does not hash file contents. There is no configured migration timeout. SIGTERM/SIGINT cancels the migration and prevents etcd or components from starting; directory walks and cross-filesystem copies observe cancellation between filesystem reads. A canceled partial copy is never published and is recopied from the intact source on the next start. No user migration script, offline command, reverse migration, journal, or backup is provided. Successfully moved sources are removed. ## Compatibility and limits - Preserve the absolute local root, data mounts, and legacy `minio.rootPath` when upgrading. Canonical 2.6 data stays in place; misplaced late-2.6 indexes and affected 3.0.0/3.0.1 layouts are handled as above. Relative roots from older versions are unsupported. - Automatic migration is standalone/local-only. If old CWD indexes disappeared with an old container, they cannot be discovered: preserve/make those files visible or rebuild the derived indexes. Cross-filesystem copy has no configured timeout and can extend the first startup, but a shutdown signal cancels it and the next start resumes from the remaining source. - Downgrade to the affected old layout is unsupported after migration. Symlink layouts require manual resolution. - Known legacy LOB/orphan GC gaps under `R/M/insert_log`, pending old external-refresh paths, and rare interrupted-copy publication residue remain outside scope. This is not a claim that every historical file is recovered or reclaimable. ## Validation At head `898dba4b41`, `ci-v2/build-ut-cov` passed Build, Code Check, Integration UT, Go UT, and C++ UT (9,145/9,145). Diff coverage was 100% for C++ and 81.2% for Go; total patch coverage was 82.8%. Focused local tests for canonical/frozen `localStorage.path` derivation and migration cancellation/deadline/copy recovery passed with `-tags dynamic,test -gcflags='all=-N -l' -count=1`, and `git diff --check` passed before publication. Full paramtable execution remains locally blocked by tests attempting to create `/var/lib/milvus/data`; roles/native-dependent Go tests remain build-blocked by local C++ header/library mismatches. A real 3.0.0/3.0.1 version-to-version upgrade was not executed locally. `ci-v2/e2e-default` and `ci-v2/go-sdk` are still pending for this head. Signed-off-by: xiaofanluan <[email protected]> Co-authored-by: xiaofanluan <[email protected]> |
||
|
Li LiuandClaude Opus 5
|
e008a8429d |
enhance: add optional root authentication for the management plane (#52580)
issue: #49846 design doc: docs/design-docs/design_docs/20260912-management-plane-authentication.md Merge dependency: #52579 supplies safe configuration projections. Both changes are needed for the full configuration-redaction requirement in #49846. ## Problem and behavior Port 9091 exposes process shutdown, coordinator management, profiling, event logs, the web console, and legacy REST data operations. Add `common.security.adminAuthEnabled`, default **false**, to authenticate this surface independently of the main service port's data-plane authorization. When enabled: | Surface | Policy | | --- | --- | | `/management/*` except readiness; `/debug/pprof/*`; `/log/level`; `/eventlog`; `/webui/`; `/telemetry`; `/api/v1/_*` | Root HTTP Basic authentication | | Legacy REST data operations under `/api/v1` | Existing valid-user/API-key authentication if `authorizationEnabled=true`; otherwise root Basic authentication | | `/healthz`, `/livez`, `/metrics`, `/metrics_default`, `/management/check/ready` | Open | | `/api/v1/health` | Exempt from the new gate; existing data-plane authentication still applies | After the request-origin checks below, missing/incorrect credentials return 401; non-root users are rejected before password lookup with 403; unavailable credential storage and saturated verification return 503. Dependency details and stack traces remain out of unauthenticated response bodies. HTTP and Gin share one decision and carry a verified root principal through a typed request context for in-process telemetry calls. ## Browser and client requirements The gate rejects requests identified as cross-site or same-site. An unknown Fetch Metadata value and `Origin: null` are rejected as well. Top-level GET document navigation is allowed only on document surfaces, not operator actions or console APIs. **Credential-bearing requests without Origin or Fetch Metadata must include `X-Milvus-Admin-Request: true`.** This is required because missing metadata does not establish that a request is from a non-browser client; ordinary remote HTTP is outside the Fetch Metadata trustworthy-URL guarantee. An explicit header does not override a reported cross-site origin. The metrics-port CORS preflight must not grant this header to cross-origin callers. Browser console access requires HTTPS with Fetch Metadata preserved by the reverse proxy. A document request without sufficient request context is refused without a Basic challenge. Non-browser callers add the explicit header, for example: ```sh curl --user root --header 'X-Milvus-Admin-Request: true' \ 'https://milvus-admin.example/management/querycoord/balance/status' ``` The flag remains opt-in. Existing clients with the flag disabled keep their prior behavior. ## Credential lifecycle and resource bounds Verifier ownership is deterministic in standalone: Proxy, then MixCoord, then worker. Workers create a MixCoord client lazily. Each lookup caller has its own five-second wait bound, so a service-discovery call that ignores the backend context cannot block the HTTP request indefinitely. Its accounting slot remains occupied until the shared backend result completes, preserving the resource bound even in that case. Root hashes are cached for 10 seconds, reused for up to 10 minutes after their last successful fetch if refresh fails, and lookup failures are cached for 2 seconds. The stale window preserves management access during a coordinator outage but can also retain acceptance of a password rotated during that outage. Concurrent hash refreshes share work and admit at most 32 callers, including canceled callers whose singleflight result channels remain retained. Worker client construction uses a single shared completion signal; repeated lookup timeout windows no longer append a retained channel to a stuck constructor. Construction failure can be retried, and a client completed after shutdown is closed instead of published. bcrypt concurrency is bounded to a quarter of GOMAXPROCS, floored at two, with a bounded queue. This is a concurrency cap, not an attempt rate limit; on small pods sustained incorrect passwords can still consume substantial CPU. ## Registration, configuration, and event logs - `http.Register` refuses an unmarked operator route, including method- and host-qualified ServeMux patterns. Assembled-router tests and a console-prefix test cover Gin registration. - Milvus-owned handlers use a private mux. Flag-off compatibility with more-specific legacy DefaultServeMux routes is retained when pprof is enabled; enabling the gate removes that fallback. - `/management/config/alter` always rejects this flag, using the same `config.FormatKey` identity as config storage, so an anonymous caller cannot plant a disabling etcd value before the operator enables the gate. - Key-normalization caching admits at most 4096 entries and only keys whose input and normalized forms fit 1024 bytes, using owned string storage so short substrings cannot pin large request buffers. Larger keys still normalize correctly without retained memo entries. This bounds cache retention, not individual HTTP body size. - The eventlog gRPC stream switches to loopback with authentication enabled. A failed security upgrade closes the wildcard listener. Event-log shutdown uses a separate close signal and stops the gRPC server, preserving safe concurrent recording. - `milvus_admin_auth_total{endpoint,result}` records root-gate and origin decisions using registered route patterns; failure logs are rate limited. - The standard Go runner now includes HTTP, adminauth, coordinator, eventlog, and metrics regressions. It restores fail-fast behavior after sourcing the environment so a failed test command returns a failure from the runner. ## Compatibility and deployment Port 9091 remains plaintext. Use a trusted network or TLS-terminating proxy, rotate root using `UpdateCredential` / `db.update_password`, and update graceful-shutdown and profiling clients before enabling. Editing `defaultRootPassword` does not rotate an existing credential. Official compose deployments and embedded standalone scripts publish 9091, so review that exposure. Browser access needs the HTTPS deployment described above. Regardless of the flag, downstream operator routes without `AdminAuth` now fail registration; the admin-auth key cannot be altered through the management API; configuration normalization is bounded; pprof registration is explicit; and the data-plane stored-password comparison shares the credential-format helper. ## Verification and limits CI observed on head `c8992fc7b4d3fcf9a28bf73ad4700283922367f2` at **2026-09-12 14:30 UTC**: `ci-passed` and `dco-passed` are present. | Check | Result | | --- | --- | | [Build / unit / integration / code checks](https://jenkins-milvus-ci.milvus.io/job/MILVUS-CI-V2-PR-PIPELINES/job/milvus-build-ut-ciloop-pipeline/30883/pipeline-overview/) | SUCCESS; status reports build, code checks, integration, Go, and 9,106 C++ tests passed | | [Go SDK E2E](https://jenkins-milvus-ci.milvus.io/job/MILVUS-CI-V2-PR-PIPELINES/job/milvus-go-sdk-pipeline/15021/pipeline-overview/) | SUCCESS | | [Default E2E](https://jenkins-milvus-ci.milvus.io/job/MILVUS-CI-V2-PR-PIPELINES/job/milvus-e2e-pipeline-dispatcher/19796/pipeline-overview/) | SUCCESS | | Main CI trigger / DCO | SUCCESS | The non-gating Tencent Pulsar gray run reports `ERROR` because its downstream job could not be triggered. The previous PR head had the identical start failure. No Pulsar test result is claimed, and the precise trigger failure is unknown because Jenkins logs are inaccessible here. The TC-US shadow build and legacy Jenkins status still show pending; they are outside the master `ci-passed` requirements. No label was added manually. Local verification and remaining limits: All Go tests use `-tags dynamic,test -gcflags="all=-N -l"`. - Full `internal/util/adminauth`, including race detection, passed. A blocked-constructor regression retained 20,011 extra heap objects after 10,000 canceled waits before the fix and no retained growth after it. Cancellation, retry, panic, and a client completing after close are covered. Four blocked-backend scenarios cover cold cache, expired stale state, and correct/incorrect stale passwords; race runs repeated three times passed, including retained slots and recovery after release. - Full `internal/http` and its race run passed with the explicit-request-header changes, including 34 new net/http/Gin cases. `pkg/eventlog`, `pkg/metrics`, and `pkg/util/merr` also passed. - Configuration-normalization tests passed, including oversized inputs, Unicode expansion, and request-buffer ownership. The full config race suite passed excluding `TestEtcdSource`, whose two tests require a reachable external etcd and failed against this host's deliberately unavailable test endpoint. The config comment and ParamItem documentation match exactly. - The newly scheduled HTTP, adminauth, eventlog, and metrics commands passed with the runner's race and coverage flags. A dispatch harness checked all five package additions through the full and selected-group entry points. Ten injected command failures were swallowed before restoring fail-fast and now stop the runner immediately with the original nonzero status. Coordinator execution was unavailable locally; the subsequent native CI aggregate reports Go success. - Legacy Basic challenges are preserved on the main port, gate-off metrics routes, and the health exemption. A regression using the actual extracted parser with real Gin failed before the repair and passed after it; assembled-route and dynamic-gate compatibility tests were added to the native package suites. The subsequent CI Go check reports success; individual native test logs were not accessible here. - `internal/proxy`, `internal/distributed/proxy`, `internal/coordinator`, and `cmd/roles` cannot compile on this host because `milvus_core.pc`, `milvus-storage.pc`, and `rocksdb.pc` are unavailable. The subsequent native CI aggregate reports build and Go success; this does not change the local compilation limitation. - Full `make test-go` was attempted; native dependency construction failed in the host SDK linker (`library 'System' not found`) before the Go suite. This is not recorded as a passing full test run. - golangci-lint v2.11.3 built with Go 1.26.6 reported zero new issues in `internal/http`, `internal/util/adminauth`, and the changed config/eventlog/metrics/paramtable packages. Native-linked packages could not be linted locally; the subsequent CI code-check status reports success. - A real coordinator failover and a real browser console flow remain unverified. Unit and source-level failure tracing do not establish full browser credential replay or transport behavior. --------- Signed-off-by: Li Liu <[email protected]> Co-authored-by: Claude Opus 5 (1M context) <[email protected]> |
||
|
Jamesandxiaofanluan
|
ecdaf12d00 |
fix: tolerate transient etcd leader-election errors during startup (#53367)
## What this PR does / why we need it Milvus standalone with embedded etcd can panic during startup with `panic: etcdserver: leader changed`. A single-node embedded etcd starts serving traffic before it has elected itself leader. `Session.Init` could run inside that election window; its `getServerID` etcd `Get` then returns a transient `etcdserver: leader changed`, which `Init` treats as fatal via `panic(err)`, terminating the whole process. ## Changes - `pkg/util/etcd/etcd_server.go`: wait for `Etcd.Server.ReadyNotify()` (leader elected + member published) in `InitEtcdServer` so components only start after etcd is truly ready. - `pkg/util/etcd/etcd_util.go`: add `IsRetriableEtcdErr` to classify transient leader-election / availability errors (leader changed, no leader, request-timeout sentinels, gRPC Unavailable). - `internal/util/sessionutil/session_util.go`: retry `getServerID` on retriable transient etcd errors with bounded backoff instead of failing fast (`Register` already retries; `Init` was the gap). - `pkg/util/etcd/etcd_util_test.go`: table-driven tests for `IsRetriableEtcdErr`. issue: #53144 --------- Signed-off-by: xiaofanluan <[email protected]> Co-authored-by: xiaofanluan <[email protected]> |
||
|
Zhen Ye
|
35f180913f |
enhance: gate write-before function materialization on cluster version (#53011)
issue: #52988 - paramtable version gate: add VersionGateSwitcher (EnableAutoSwitchValue/PreSwitchValue/GateVersion/TargetValue/SwitchDelay) with EffectiveValue()/GetAsBoolEffective() and a one-shot Confirmator that watches cluster sessions, maintains the minimum online version and flips gated config items into the etcd config center once the whole cluster stays above the gate version for the SwitchDelay stability window - version gate startup: the global once `StartVersionGateSwitcher()` is called by the MixCoord role only (after the role has been set); other roles observe the flipped value through the regular config refresh. Embedded-etcd (single-process) deployments resolve gates locally and skip the confirmator; cluster deployments start it in the background. The flip re-reads the process-local config value (file/env sources) right before the write, so an explicit operator value set while the confirmator is running wins and no etcd write happens - function config: gate function.enableWriteBeforeMaterialization (auto -> flip to true, false escape hatch, true force) behind cluster version 2.6.23; non-exported - streamingnode shard interceptor: materialize function output fields (e.g. BM25 sparse vectors) before WAL append only when the gated config is effective --------- Signed-off-by: chyezh <[email protected]> |
||
|
jiaqizho
|
476c7aa283 |
enhance: collect cached filesystem metrics on scrape (#52676)
issue: https://github.com/milvus-io/milvus/issues/47520 Filesystem metrics were previously pushed from task-completion paths, which could miss cached filesystems and leave stale values. Collect them directly from the storage cache at scrape time instead. Update milvus-storage to use the cache-wide metrics API, expose cached snapshots through storagev2, and register a thread-safe Prometheus collector while preserving existing metric names and labels. Remove the per-task publishing calls and old config-based helpers so compaction, index, flush, and load paths no longer manage filesystem observability. --------- Signed-off-by: jiaqizho <[email protected]> |
||
|
|
6648d9371b |
enhance: unify membership filter expressions into one chain with membership_match syntax (#52778)
## Overview Adds a unified membership-filter surface syntax and merges the previously separate `bloom_match` / `roaring_match` chains into one parameterized execution path. issue: #52777 ## What's in this PR ### 1. Unified surface syntax ``` membership_match(<field>, {<blob-bytes-template>}) membership_match(<field>, {<blob-bytes-template>}, type=bloom|roaring) not membership_match(<field>, {<blob-bytes-template>}) ``` The filter kind is derived from the blob's self-describing magic header — `MBF1` lowers to the existing `BloomFilterExpr` plan node, `MRB1` to `RoaringFilterExpr`, and unknown headers fail closed. The optional `type=` argument is a readability aid and a consistency pin (it must agree with the magic). The unreleased `bloom_match` / `roaring_match` surface names are intentionally not retained — neither has shipped in a release. The wire plan bytes are unchanged from the two predecessor implementations, so the plan proto stays frozen and rolling-upgrade behavior for clients using `membership_match` matches what the predecessor docs specified. ### 2. One control flow, per-kind data planes - **Go parser/proxy**: one visitor/fill skeleton (`visitMembershipCall` / `fillMembershipMatchExpressionValue`), one spec table for behavior switches (delete safety), and unified tree walkers serving delete guards, element-level guards, plan-size accounting, and log redaction. - **segcore**: a single `PhyMembershipFilterExpr<LogicalExpr, ProbePolicy>` template carries the shared control flow; `BloomMembershipProbe` (zero-copy SBBF view) and `RoaringMembershipProbe` (decode-once Roaring64) remain separate data planes. Historical class names remain aliases, so factory construction and logs are unchanged. - Kind-specific admission gates (MBF1 envelope + domain check, MRB1 body walk + decoded-memory estimate) remain in their own files. - Client builders move under `client/membership/` (`sbbf`, `roaringfilter`); `bloom_match`/`roaring_match` builder packages are removed with the surface names. ### 3. Preserve existing mask/null semantics The unified executor preserves current `master` behavior on both raw-data and index-only paths: candidate-mask-excluded rows are left at their initialized `(false, valid)` value, including excluded NULL rows; probed NULL rows produce `(false, invalid)`. This PR does not change the framework reverse-lookup helper. ### 4. Two independent proxy budgets - `proxy.maxMembershipFilterSize` (default 64 MiB) limits one MBF1 or MRB1 body. It falls back to the released `proxy.maxBloomFilterSize` key and the development-only `proxy.maxRoaringFilterSize` key. - `proxy.maxMembershipFilterPlanSize` (default 128 MiB) remains the separate request-wide aggregate serialized-plan limit. It falls back to `proxy.maxBloomFilterPlanSize`. - Main predicates, scorer filters, and HybridSearch sub-requests share the same preflight budget. The preflight charges body bytes, matching the per-blob gate that allows the fixed 32-byte envelope header on top; final `proto.Size` accounting remains authoritative. Keeping the plan budget (128 MiB) well above the per-body limit (64 MiB) means a maximum-size body plus its 32-byte envelope and plan overhead always fits — no silent halving of the usable tier. ## Deliberate behavior changes 1. Adds `membership_match` with magic-based MBF1/MRB1 dispatch, an optional `type=` consistency pin, and fail-closed handling for unknown headers. 2. Removes the unreleased `bloom_match`/`roaring_match` surface names (and their client builder packages); the wire plan bytes for equivalent inputs are unchanged. 3. Updates error wording in bytes-template, element-level, MATCH_* and delete guards. 4. MATCH_* element-predicate guards now reject every membership-filter surface name. 5. Preflight aggregate accounting charges blob bodies rather than whole envelopes, matching the per-blob admission rule. ## Non-goals - No change to MBF1/MRB1 formats, golden vectors, or SDK builders. - No change to the plan proto or rolling-upgrade contract: fields 22/23 remain frozen. - No repo-wide NULL/candidate-mask semantic change. ## Design document [20260822-membership-match-expression.md](https://github.com/xiaofan-luan/milvus/blob/unify-membership-match/docs/design-docs/design_docs/20260822-membership-match-expression.md) ## Test plan - [x] `internal/parser/planparserv2`: full suite green, including magic dispatch, alias byte identity, fail-closed cases, delete safety, redaction, and shared preflight budgeting. - [x] `pkg/util/paramtable`: fallback-key precedence tests. - [x] `internal/proxy`: membership size/error/search/query focused tests green; the full local suite is green except `TestProxy`, which requires unavailable integration infrastructure and fails identically on clean HEAD. - [x] segcore: all 73 bloom/roaring/membership expression tests green; full `all_tests` has only failures reproduced on clean HEAD (6 ConjunctExprTest + 1 ExprJsonIndexTest abort). - [x] Local segcore build and unit-test binary link with llvm@19. --------- Signed-off-by: xiaofanluan <[email protected]> Signed-off-by: xiaofanluan <[email protected]> Co-authored-by: xiaofanluan <[email protected]> Co-authored-by: Claude Opus 4.6 <[email protected]> |
||
|
|
f5eef7da8a |
enhance: remove the /expr runtime-introspection endpoint (#52876)
Removes the `/expr` runtime-introspection endpoint, everything that exists only to serve it, and the forked expression library it depended on. ### Why `/expr` evaluated a caller-supplied expression against live in-process objects (`param`, `proxy`, `cache`, and the component servers) and returned the result. It was useful for reading an effective config value or unexported state without a debugger or a rebuild, but the cost is hard to justify for what it does: - It required a fork of `expr-lang/expr` whose purpose is to remove the library's sandbox. The first commit on that fork is "Support the private field and auto change the struct to the point when exec the method". Upstream deliberately refuses unexported-field access; the fork exists to allow it. - That fork sits on a personal account, is 81 commits behind upstream, and was last pushed in May 2025. `go.mod` pinned it through `replace`, making it build-critical. Every other `replace` in that block points at `milvus-io/*`. - Nothing else imports the library. `pkg/util/expr/expr.go` was the only importer, using four symbols: `expr.Compile`, `expr.Env`, `expr.WithContext` and `vm.VM.Run`. The fork's ~550 lines of product code add no API; they change the behaviour of those four calls. Swapping back to upstream would therefore still compile and still pass CI, and break only at runtime — no test in the tree exercises unexported-field access, pointer-method calls, `mapValue`, the `view` builtin, or automatic type conversion. Removing the endpoint removes the only reason the fork was there. ### What is removed - the `pkg/util/expr` package - nine `expr.Init` / `expr.Register` call sites across roles, datacoord, datanode, proxy, meta cache, querycoord, querynode and rootcoord - the `/static/` Expr Executor page (`internal/http/static/index.html`, whose only function was to call `/expr`), `static_view.go`, `StaticPath` and its route registration - the `/expr` handler in `internal/http/server.go`, `ExprPath` in `router.go`, and `CheckExprAuth` / `checkExprRootAuth` / the two auth-mode constants in `internal/http/rbac.go` - `common.security.exprEnabled` and `common.security.exprAuthMode`, and their entries in `configs/milvus.yaml` - `github.com/expr-lang/expr` and its `replace` to `github.com/SimFG/expr`, from both `go.mod` and `pkg/go.mod` ### The Expr privilege v3.0.0 shipped `Expr` as a grantable RBAC privilege for this endpoint. The name cannot simply be dropped: grant and revoke share one validation path (`ddl_callbacks_rbac_privilege.go` dispatches on request version, not on grant-vs-revoke), so an undefined name would leave grants made on v3.0.0 both unusable and unrevocable, clearable only by editing etcd by hand. The second commit rejects the grant explicitly instead, with a message that says why, and leaves every other path untouched. The name stays recognized and inert so existing grants remain listable and revocable, and can be dropped entirely once no supported release still has the endpoint. ### What is deliberately kept `CheckPrivilege`, `RegisterGetUserRoleFunc`, `RegisterPasswordVerifyFunc` and `parseHTTPAuth` in `internal/http/rbac.go`. These are general HTTP RBAC helpers rather than expr-specific, and #52580 builds on that surface. After this change only tests call `CheckPrivilege`; that is intentional, not an oversight. ### Verification Unit tests pass for `internal/http`, `internal/rootcoord` and `pkg/util`, including `TestExprPrivilegeDefinition` and a new `TestIsDeprecatedExprPrivilege` that covers both the API and metastore spellings plus negative cases. Behaviour was also checked against a running instance rather than inferred: ``` route removal /expr 404 "404 page not found" /nonexistent-route-xyz 404 "404 page not found" identical response /healthz, /metrics 200 server is healthy /expr on a build that still registers it 500 from expr.Exec privilege guard grant Expr (v2) rejected: privilege "Expr" is deprecated: the /expr grant Expr (v1) rejected endpoint it granted access to has been removed revoke Expr (v2) accepted revoke Expr (v1) accepted grant/revoke Load accepted (control) a pre-existing v3.0.0 grant wrote grantee-id/<id>/PrivilegeExpr directly into etcd, restarted the cluster describe_role shows Global/Expr alongside Global/CreateDatabase revoke Expr succeeds describe_role only CreateDatabase remains etcd the PrivilegeExpr key is gone, no orphan left behind ``` ### Scope note This does not close the unauthenticated `:9091` surface. `/metrics` and `/debug/pprof/` remain unauthenticated; that is #52580's subject. This change removes one route from that port and the supply-chain exposure behind it. --------- Signed-off-by: zhenshan.cao <[email protected]> Signed-off-by: Zack <[email protected]> Co-authored-by: Zack <[email protected]> Co-authored-by: Claude Fable 5 <[email protected]> |
||
|
Li LiuandClaude Opus 5
|
0d7e6c2216 |
enhance: cut the duplicate halves of the MRB1 roaring codec (#52586)
issue: #52094 Review follow-up to #51968. No behaviour change: the MRB1 wire format, the validation rules and the limits are all unchanged, and every surviving production function -- `Build`, `Validate`, `scanPortableRoaring64Body`, `validateEnvelope` -- is byte-identical to master. The segcore edits are comments only. The Go production changes are deletions, plus one behaviour-preserving refactor: `PlanContainsBloomFilter` and `PlanContainsRoaringFilter` now share `planContainsFilter` (see "Two smaller items"). ## The duplicate halves of the codec `client/v3/roaringfilter` and `pkg/v3/util/roaringfilter` each exported the whole codec — `Build`, `Validate`, `Parse`, `Filter` — while production reached half of each: | | production entry point | reachable only from tests | | --- | --- | --- | | `client/v3/roaringfilter` | `NewRoaringBitmapBlob` → `Build` | `Validate`, `Parse`, `Filter`, and all 284 lines of `portable_roaring64.go` | | `pkg/v3/util/roaringfilter` | `validateRoaringBitmapBlob` → `Validate` | `Build`, `Parse`, `Filter` | The two `portable_roaring64.go` files were 2 lines apart, because one was copied from the other. The two test files were 19 lines apart out of ~985. `pkg`'s copy even carried a wrapper, `validatePortableRoaring64Body`, that nothing called at all. Neither package appears in any release tag — `client/v3.0.0-beta` (2026-07-18), `pkg/v3.0-beta`, `pkg/v3.0.0` and `v3.0.0` (2026-07-29) all predate #51968, which merged 2026-08-17 — so trimming the two package APIs breaks no published surface. `git ls-tree` on each of those tags finds neither directory. The only residual exposure is a consumer pinned to a master pseudo-version inside a two-day window on an unreleased package. Each package now owns one job: **the client builds, pkg validates, segcore decodes.** The builder and decoder pieces pkg's own tests still need moved to `fixtures_test.go`, so the validator suite is unchanged and the package's API is `Validate` alone. That fixture builder is trimmed to an encoder: the SDK additionally pre-rejects member sets the proxy would reject, and none of the `buildFixture` call sites assert a rejection, so reproducing that admission logic there would be a second copy of it that nothing checks. The bytes are the same either way — a Roaring bitmap is a set, so insertion order cannot change the encoding. One side effect worth naming. `pkg` is linked into every server component, and it was the only non-test importer of `RoaringBitmap/roaring` outside the SDK, so after this change no production Go code outside `client/` imports the library. `go.mod` moves it from the indirect block to a direct require in the **root** module, because a root-module test file now imports `roaring64` directly; `pkg/go.mod` keeps it as a direct require for the same reason. So the module graph is unchanged for anyone depending on `pkg/v3` — what changed is the production import graph, not the dependency list. `TestClientAndServerRoaringBuildAgree` existed to compare two builders. With one builder there is nothing to compare, so it becomes `TestClientBuiltBlobsPassProxyValidation`: the contract that still spans two implementations is "a blob the SDK builds is one the proxy accepts". It keeps the same distribution shapes and sizes, adds a shape that reaches the bitmap-container and offset-table branches with bytes a real builder produced, and grades the validator's structural scan against counts recomputed a third time from the key set, so neither side is graded against itself. ## The C++ copy of the limits The nine MRB1 envelope and admission constants exist in three places. The Go pair was already pinned to each other. segcore consumes its own copy — `RoaringMembership.cpp` reads all nine — but **nothing pinned them to the Go values**, so editing `kMaxHighContainerCount` there alone compiled and passed CI, and the symptom only showed up in production as the proxy admitting a blob every querynode refuses, or the SDK refusing to build one the cluster would take. segcore's copy is now **generated** from the Go one rather than checked against it. `cmd/tools/genmrb1limits` writes `internal/core/src/common/MRB1Limits.generated.h` from `pkg/util/roaringfilter`'s constants, and `RoaringMembership.h` aliases the names in it: ```cpp static constexpr auto kHeaderSize = mrb1::kHeaderSize; ``` There is one definition, in Go, and segcore resolves it. `RoaringMembership.cpp` is untouched: `RoaringMembership::kFoo` still names the same value at the same width. An earlier revision did this the other way — a Go test opened the two segcore sources and parsed the declarations out of them. It worked, and fault injection said so, but it was 623 lines, and the reason it needed them is that every way of being shown the wrong declaration is a way to pass over a diverged constant: comments carrying an old value, non-literal initializers, a declared type that silently truncates (`uint16_t kX = uint64_t{1} << 18` is 0), a name mentioned rather than declared, an enum member or `#ifdef` pair hiding the live declaration behind a readable decoy, and a macro defeating the check that catches those. The cost landed on C++: those two files had to stay macro-free, one-declarator-per-statement, and evaluable, enforced by a Go test in another directory that takes the whole `internal` UT job down with it — in a tree that has 172 `#define` and five headers using include guards. Generating removes the need for all of it; segcore may spell its constants however it likes now, because nothing reads them. ### What each half guarantees - **Values and types reach segcore intact.** A standalone TU reproducing the class, the aliases and the call-site shapes from `RoaringMembership.cpp` compiles under `clang++ -std=c++20 -Wall -Wextra` and `static_assert`s that every constant still equals the literal it replaced and still has its old declared type (`const std::string_view` / `uint16_t` / `size_t` / `uint64_t`), so no comparison in the `.cpp` changes width. - **The header cannot go stale.** `TestGeneratedHeaderIsCurrent` fails when the file on disk is not what the generator would write. Injected both directions — a Go constant changed without regenerating, and a hand-edited generated header — and both fail with the regenerate instruction. - **A wrong type in the generator's table is a compile error, not a truncation.** The values are brace-initialised for exactly this, measured rather than assumed: clang accepts `constexpr uint16_t kX = 262144` without a word and gives it 0, while `constexpr uint16_t kX{262144}` is `-Wc++11-narrowing`, an error by default. Injected: narrowing one row fails at the generated header and again at every `static_assert` downstream. It also lands somewhere better to run. `internal/parser` is covered by the codecov job only; `cmd/tools` is named by both it and `scripts/run_go_unittest.sh`, so `make test-go` now covers this where it did not cover the pin. ### What this gives up - **The completeness direction.** The scan required every `k`-prefixed constant in those two files to be classified as pinned or deliberately unpinned, so a limit added to segcore alone was caught. Nothing replaces that: a new MRB1 limit has to be added to the generator's table, and adding one only to C++ is now caught by review rather than by CI. - **Use sites, which neither approach ever covered.** Changing `/ kPortableRoaring64MinEntryBytes` to `/ 16`, or a `>=` to a `>`, diverges segcore from the Go validator exactly as a changed constant would, and passes. The seven portable-Roaring wire constants in `RoaringMembership.cpp` remain deliberately **not** pinned by value. They are RoaringFormatSpec values rather than a Milvus contract, and two of them are derived rather than chosen. Each side is held to them behaviourally, at the boundary, by fixtures written independently on each side. That catches something a value check cannot — a value check reads declarations, so flipping a `>` to a `>=` passes it while the fixtures fail — but it is not strictly stronger, and injection says so: `kPortableRoaring64MinEntryBytes` 12 → 16 fails on both sides, 12 → 8 fails on neither. Loosening is benign here, since the bound is a pre-filter and the full scan behind it rejects the same input, but the two sides really can drift in that direction unnoticed. That reasoning now lives next to the constants in `RoaringMembership.cpp` rather than in a test file. ## Two smaller items - `PlanContainsBloomFilter` and `PlanContainsRoaringFilter` were line-for-line copies of each other, and both re-implemented the switch `planPredicates` already does — three copies of one switch in one package. They share `planContainsFilter` now. Its nil check is gone with them: the generated getters are nil-safe and `walkExpr` reports false for a nil expression, and both nil cases are covered by existing tests. - `PhyRoaringFilterExpr` passes a `nullptr` `skip_func`, so no chunk is ever skipped. Bloom has no choice — an SBBF exposes no range — but a Roaring bitmap has `Minimum()`/`Maximum()` in O(1). Left as a TODO against this issue, carrying the two's-complement caveat, so it doesn't read as "roaring can't". ## Verification Coverage of the deleted client-side tests was checked rather than assumed: all 18 `Test*` functions removed from `client/roaringfilter/roaring_filter_test.go` have a counterpart in `pkg/util/roaringfilter/roaring_filter_test.go`, which is where the validator they exercise now lives. 4 keep their name outright; the other 14 are `TestParse*`/`TestBuildParse*` names renamed to `TestValidate*` because the package no longer exports `Build` or `Parse`. The differential oracle the old cross-module test provided is not lost: `TestBuildMatchesLegacyByteForByte` pins the SDK's output byte-for-byte against a naive builder, which is a better oracle than comparing two copies of the same wrapper. `FuzzParse` becomes `FuzzValidate`. One direction of its equivalence became trivially true, because `parseFixture` calls `Validate` first; the comment says so rather than implying otherwise. The load-bearing comparison survives: the cardinality the hand-rolled scanner computes is cross-checked against what `roaring64.ReadFrom` decodes, so the grader is the upstream library, not the implementation under test. Passed in this checkout: - `client/v3/roaringfilter`, `client/v3/milvusclient` - `pkg/v3/util/roaringfilter` — the full validator suite, plus `FuzzValidate` - `internal/parser/planparserv2` — the whole package, including the rewritten `TestClientBuiltBlobsPassProxyValidation` - `cmd/tools/genmrb1limits` — `TestGeneratedHeaderIsCurrent` and its two table guards; the generator is idempotent - `go vet` on all three modules; `gofmt -l` clean; `clang-format-15`, the version `internal/core` pins, reports all four touched C++ files clean Running `internal/parser/planparserv2` locally needs `LOCALSTORAGEPATH` pointed somewhere writable: paramtable's `LOCAL_STORAGE_SIZE` formatter mkdirs `/var/lib/milvus/data/` and `mlog.Fatal`s if it cannot, which on a stock macOS checkout takes the package down before any assertion runs. That reproduces identically on `upstream/master`. Two things I could not check locally and am not claiming: `internal/proxy` and `internal/distributed/proxy/httpserver` need a built `milvus_core`/`rocksdb`, so the one external caller of the trimmed API — `internal/proxy/task_delete_test.go`, which uses the client's `Build`, still exported — is grep-verified only. segcore was not rebuilt: the standalone TU above is the strongest check I could run without it, and CI is the first real segcore build. Its non-comment edit is nine one-line aliases and one `#include`. `golangci-lint` crashes locally on a Go toolchain mismatch, so CI is the first real lint run. Where the new guard runs: `cmd/tools`, which both `make codecov-go-without-build` → `scripts/run_go_codecov.sh` and `scripts/run_go_unittest.sh` → `make test-go` cover. The test's own doc comment says this. ## Follow-ups - `client/sbbf` is in the same shape this PR fixes for roaring: it exports `Parse`, `Filter` and six accessors with no non-test caller, it is likewise absent from every release tag, and MBF1 has the same three-way limit contract with no guard. Left alone deliberately to keep this diff to one format; adding it now means one more table in `cmd/tools/genmrb1limits`, not a second mechanism. - `redactExprParams`' doc comment is attached to `hasExprParams` in `internal/distributed/proxy/httpserver/handler_v2.go`, leaving `redactExprParams` undocumented in godoc. That fix was in an earlier revision of this PR and was taken out: it is another subsystem, and it belongs in its own `doc:` PR. --------- Signed-off-by: Li Liu <[email protected]> Co-authored-by: Claude Opus 5 (1M context) <[email protected]> |
||
|
junjiejiangjjj
|
b01f360229 |
enhance: support file resource sync on proxy (#52174)
issue: #51205 Add Proxy as a FileResource sync target by wiring it into the existing FileResourceObserver flow. Proxy now initializes a fileresource manager, exposes SyncFileResource over internal RPC, and RootCoord injects the proxy client manager so resource updates can be pushed to Proxy nodes. Signed-off-by: junjie.jiang <[email protected]> |
||
|
tinswzyandClaude Fable 5
|
b9621d7d32 |
fix: stop runtime overlay from shadowing etcd-persisted mq.type after WAL switch (#51543)
issue: #51497 After a successful WAL switch via `POST /management/wal/alter`, `GET /management/config/get?keys=mq.type` kept returning the pre-switch value with `"source":"RuntimeSource"` until the coordinator restarted, even though the ack callback had already persisted the new mq.type into etcd (and linearizably refreshed the local EtcdSource). External operators polling that endpoint to confirm switch completion never saw the new value. ### Root cause `InitAndSelectWALName()` saved the startup-resolved WAL name into the config manager's **runtime overlay**, and `Manager.GetConfig` serves the overlay with top priority — permanently shadowing the EtcdSource for every reader in the process. The overlay write existed only to feed `ProcessImmutableConfigs`, which pins `GetConfig`'s value into etcd create-if-absent at first boot (without it, the literal `"default"` would be persisted). ### Fix Replace the overlay bridge with an explicit per-key **renderer** on the persistence path, so mq.type reads fall through to the etcd source and naturally follow a WAL switch: - `ProcessImmutableConfigs(renderers map[string]func(raw string) string)`: a renderer converts a placeholder raw value (mq.type's literal `"default"`) into the concrete value at create-if-absent persist time. It never runs when etcd already holds the key (a completed switch survives restart), and it also covers the key-absent-from-all-sources case (raw is passed as `""`). Generic: any future immutable config with a placeholder value can plug in its own renderer. - `InitAndSelectWALName()` no longer writes the runtime overlay and returns the resolved `message.WALName`; the coordinator startup passes a mq.type renderer backed by that name. - `dependency.HealthCheck` resolves the literal `"default"` to the actually-enabled MQ before probing (this gap predates the overlay Save: since #36822, deployments running with `mq.type: default` probed nothing and always reported unhealthy). This also fixes two other readers of the shadowed value: - the `HandleAlterWAL` same-type short-circuit: re-posting the completed target no longer re-broadcasts, and a rollback to the original MQ is no longer silently swallowed with "already configured"; - the proxy MQ health check (`/_cluster/dependencies`) no longer keeps probing the old MQ forever after a switch. ### Behavior note `MQCfg.Type.GetValue()` now follows etcd at runtime instead of being frozen at process start: the coordinator sees a switch immediately (linearizable refresh in the ack callback); other nodes converge within one EtcdSource poll cycle (default `refreshInterval` 5s). Audited readers: `mustSelectMQType` / `MustSelectWALName` resolve `"default"` themselves and are startup-scoped; the AlterWAL idempotency check and the proxy health check are exactly the readers this change fixes. ### Verification - New tests (all watched failing first): renderer converts placeholder before first persist / existing etcd value never overwritten nor re-rendered / no-renderer keys unchanged / key absent from all sources still pinned via renderer (`pkg/config`); `InitAndSelectWALName` leaves mq.type served from its original source, not `RuntimeSource` (`streamingutil/util`); `HealthCheck("default")` resolves to the enabled MQ (`dependency`). - Full package runs green locally: `pkg/config`, `internal/util/streamingutil/util`, `internal/util/dependency` (with `-tags dynamic,test -gcflags="all=-N -l"`). - `cmd/roles`, `internal/coordinator`, `internal/distributed/streaming` type-checked via `go vet` locally (no current-master segcore build on this machine); relying on CI for full compilation. A live-cluster alter → config/get end-to-end pass is still pending — the switch-visibility chain (etcd write → `GetConfig` returns the new value) is covered at unit level by the existing `TestAlterConfigsInEtcd`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Signed-off-by: tinswzy <[email protected]> Co-authored-by: Claude Fable 5 <[email protected]> |
||
|
Zhen Ye
|
b07c62d53c |
enhance: replace log package with context-aware mlog (#50094)
issue: #35917 - mlog package: move logger initialization, zap core, async buffered writes, field helpers, and scoped logger binding into pkg/mlog while removing pkg/log. - logging callsites: migrate Milvus logging usage to context-aware mlog APIs and simplify redundant With chains across components, utilities, tests, and tools. - trace propagation: replace logutil trace interceptors with mlog/tracer integration and add client_request_id fallback propagation for server stats handlers. --------- Signed-off-by: chyezh <[email protected]> |
||
|
jiaqizho
|
148bcc3e87 |
enhance: force CRC32C when OpenSSL FIPS is enabled (#50239)
issue: #48359 This change makes Milvus automatically use CRC32C for S3 PutObject requests when OpenSSL FIPS mode is active. In FIPS mode, checksum choices are more constrained, so relying on the configured MinIO checksum behavior can lead to requests using an algorithm that is not acceptable in that runtime. The implementation moves OpenSSL FIPS enablement into the shared `pkg/util/fips` package and makes it return whether FIPS was actually enabled. That lets startup code keep enabling OpenSSL FIPS as before, while paramtable can also reuse the same state during config initialization. When FIPS is successfully enabled and `minio.ssl.useCRC32C` is still false, Milvus now logs a warning and temporarily overrides the runtime value to true. This keeps the user-facing config unchanged, but makes the effective runtime behavior compatible with FIPS requirements. Signed-off-by: jiaqizho <[email protected]> |
||
|
Zhen Ye
|
40dbf4a3ef |
fix: bound streaming release wait (#49624)
issue: #49609 Signed-off-by: chyezh <[email protected]> |
||
|
congqixia
|
7311d450a0 |
enhance: bump Go dependencies to v3 modules (#49485)
Related to #49398 Bump Go module references from pkg/v2 and milvus-proto/go-api/v2 to pkg/v3 and milvus-proto/go-api/v3 so the client tracks the Milvus 3.x release line. This prepares the repository for the upcoming 3.x.y release by aligning imports, module dependencies, and proto API references with the new major-version module paths. --------- Signed-off-by: Congqi Xia <[email protected]> |
||
|
Li LiuandClaude Opus 4.6
|
87b04e8a63 |
fix: remove golangci-lint v2 exclusion rules and fix ~1500 lint violations (#48586)
## Summary Remove all temporary exclusion rules added during the golangci-lint v1→v2 upgrade (PR #48286), fixing ~1500 lint violations: - **QF series (~1080)**: auto-fixed with `--fix` (embedded field selectors, if/else→switch, De Morgan's law) - **ST1005 (148)**: lowercase error strings per Go conventions - **gosec (116)**: fix G118 context cancel leaks, G306 file permissions; nolint for G602/G120/G705 false positives - **revive (41)**: `Json`→`JSON`, `Url`→`URL` naming conventions - **ineffassign (14)**: remove unused assignments - **unconvert (13)**: remove unnecessary type conversions - **depguard (9)**: replace banned `errors` import with `cockroachdb/errors` - **gocritic (14)**: fix ruleguard violations - **Other staticcheck (12)**: S1034, S1008, SA3001 etc. Kept disabled: govet `printf` analyzer (known Go 1.25 + golangci-lint v2 panic bug) issue: #48574 pr: #48286 ## Test plan - [x] `golangci-lint run` passes with 0 issues locally - [ ] CI passes 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Li Liu <[email protected]> Co-authored-by: Claude Opus 4.6 (1M context) <[email protected]> |
||
|
|
e0873a65d4 |
enhance: remove SuffixSnapshot mechanism from RootCoord metadata storage (#47960)
## Summary Remove the entire SuffixSnapshot time-travel layer from RootCoord metadata storage, replacing it with plain KV operations. Replace SnapShotKV with TxnKV in Catalog (~30 call sites) Delete suffix_snapshot.go (~750 lines) and tests (~917 lines) Remove SnapShotKV interface from pkg/kv/kv.go Remove snapshot GC config params (ttl, reserveTime) Keep IsTombstone/ComposeSnapshotKey constants for migration tool compatibility Why: Investigation confirmed snapshot keys are write-only artifacts that nobody reads: No client/proxy sends non-zero timestamp for time-travel metadata queries Tombstone sweeper calls RemoveCollection(ts=0), bypassing snapshot tombstone writes entirely Snapshot keys accumulate as orphans — the old GC couldn't fully clean them because ts=0 deletion path never wrote tombstone markers Net reduction: -2180 lines related: #48697 ## Test plan - [x] Unit tests for GC batch logic (expiry, cursor progress, latest-per-group protection, tombstone cleanup) - [x] Unit tests for Save/Load with time-travel verification - [x] `go vet` passes on all changed packages - [ ] CI passes Signed-off-by: MrPresent-Han <[email protected]> Co-authored-by: MrPresent-Han <[email protected]> Co-authored-by: Claude Opus 4.6 (1M context) <[email protected]> |
||
|
XuanYang-cnandClaude Opus 4.6
|
c54b34a880 |
enhance: enable OpenSSL FIPS mode for Milvus (#48331)
Layer 2 (OpenSSL FIPS) changes: - Add programmatic FIPS activation via OSSL_LIB_CTX_load_config in boring_enabled.go (gated by //go:build boringcrypto) - Add openssl-fips.cnf with fips + default providers and default_properties = fips=yes - Use absolute .include path for fipsmodule.cnf — OpenSSL resolves relative .include from the process working directory, not the config file's directory, causing silent FIPS provider load failure - Add RAND_bytes probe after config load to verify the FIPS provider is truly functional (EVP_default_properties_is_fips_enabled only checks the property string, not whether the provider loaded) - Dockerfiles: add openssl fipsinstall + OPENSSL_MODULES env var - Log OpenSSL FIPS status from C++ via EVP_default_properties_is_fips_enabled Layer 1 (Go BoringCrypto) changes: - Add GOEXPERIMENT=boringcrypto build flag (conditional on MILVUS_FIPS_ENABLED=ON) - Add boringEnabled() build-tagged functions for startup logging s2n-tls upgrade: - Override s2n 1.4.1 (from aws-c-io) to 1.6.0 in conanfile.py. s2n 1.4.1 only detects FIPS via the legacy OPENSSL_FIPS define (not set by OpenSSL 3.x). s2n 1.6.0 adds EVP_default_properties_is_fips_enabled() detection so s2n enters FIPS mode and uses RAND_bytes() through the FIPS provider. See also: #48202, #48301 Signed-off-by: yangxuan <[email protected]> Co-authored-by: Claude Opus 4.6 <[email protected]> |
||
|
XiaofanandClaude Opus 4.6
|
d8f0811af1 |
fix: remove IsTriggerKill SIGINT from datacoord and querycoord session watchers (#48252)
## Summary Removes the `IsTriggerKill` / SIGINT block from `datacoord.stopServiceWatch()` and `querycoordv2.watchNodes()`. ### Root Cause In MixCoord mode all three coordinators share the same etcd `Session` object. During shutdown, when any coordinator calls `session.Stop()` it cancels the shared context, closing the other coordinators' etcd watches — triggering `stopServiceWatch()` / `watchNodes()` which sent SIGINT to the process during a normal, coordinated teardown. ### Why remove SIGINT? - `go s.Stop()` already handles unexpected session loss — SIGINT was just a backstop in case `Stop()` hangs - No other component (rootcoord, proxy, datanode, querynode) has this logic - The false-positive risk (killing the process during normal shutdown) outweighs the marginal benefit issue: #48242 --------- Signed-off-by: xiaofanluan <[email protected]> Co-authored-by: Claude Opus 4.6 (1M context) <[email protected]> |
||
|
congqixia
|
fdaebc4127 |
enhance: skip remote and env config sources in yaml generation (#48358)
The config gen-yaml tool initialized BaseTable with default options, which connected to etcd and loaded environment variables. This caused runtime config values (e.g. mq.type=pulsar from etcd) to override code-defined defaults, producing incorrect generated yaml files. Signed-off-by: Congqi Xia <[email protected]> |
||
|
Li LiuandClaude Opus 4.6
|
ff5b6d0a64 |
enhance: remove spf13/viper dependency (#47944)
## Summary - Replace spf13/viper with yaml.v3 in the only 2 files that used it (printer.go, server_test.go) - Remove viper + 7 transitive deps from go.mod (8 packages total) - C++ ghost dep removal (libsodium, simde) deferred to separate PR to avoid CI conan issues ### Removed packages spf13/viper, fsnotify/fsnotify, hashicorp/hcl, mitchellh/mapstructure, pelletier/go-toml, spf13/afero, spf13/jwalterweatherman, subosito/gotenv ## Test plan - [ ] CI passes (Go build + code-check) - [ ] `go mod tidy` produces no diff - [ ] querynodev2 TestInit_QueryHook passes with new yamlConfigWriter issue: https://github.com/milvus-io/milvus/issues/47919 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Li Liu <[email protected]> Co-authored-by: Claude Opus 4.6 <[email protected]> |
||
|
XiaofanandClaude Opus 4.6
|
d69bdd288c |
fix: fix macOS 15 ARM64 compilation issues (#7437) (#47810)
issue: #47809 --------- Signed-off-by: xiaofanluan <[email protected]> Co-authored-by: Claude Opus 4.6 <[email protected]> |
||
|
yanliang567andClaude Opus 4.6
|
8cffd8e892 |
enhance: use effective version in banner and metrics BuildInfo label (#47953)
## Summary
- Use `getEffectiveVersion()` instead of hardcoded `common.Version` for
startup banner and `milvus_build_info` Prometheus metric
- Dev builds now correctly display `master-20260228-add6e4c6d4` instead
of always showing `2.6.6`
- Release builds are unaffected (git tag matches semver, so
`getEffectiveVersion()` returns the same value)
## Motivation
The startup banner (`Version:` line) and
`milvus_build_info{version="..."}` metric currently always show the
hardcoded semver from `common.Version` (e.g. `2.6.6`), regardless of
whether the binary is a release build or a dev build. This makes it
impossible to identify the actual build version from logs or monitoring
dashboards.
`getEffectiveVersion()` already exists and is used by the `GetVersion`
RPC and `Connect` RPC (via `MILVUS_GIT_BUILD_TAGS` env var, introduced
in PR #47822). This PR makes the banner and metrics consistent with
those APIs.
### What changes
| Component | Before | After (release) | After (dev) |
|-----------|--------|-----------------|-------------|
| Banner `Version:` | `2.6.6` | `2.6.6` | `master-20260228-add6e4c6d4` |
| Metric `milvus_build_info{version=}` | `2.6.6` | `2.6.6` |
`master-20260228-add6e4c6d4` |
| Session etcd version | `2.6.6` | `2.6.6` (unchanged) | `2.6.6`
(unchanged) |
| `GetVersion` RPC | `master-20260228-...` | `2.6.6` (unchanged) |
`master-20260228-...` (unchanged) |
### What is NOT affected
- `common.Version` itself (still hardcoded semver, used for session
compatibility)
- Session registration in etcd (still uses `common.Version` for cluster
version negotiation)
- Coordinator version range checks (still uses `common.Version`)
## Test plan
- [x] `go vet ./cmd/milvus/...` passes
- [x] Verified `getEffectiveVersion()` fallback: when `MilvusVersion` is
empty or "unknown", returns `common.Version.String()`
- [x] Verified no Grafana dashboard on internal 4am Grafana queries
`milvus_build_info` — zero PromQL impact
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Signed-off-by: yanliang567 <[email protected]>
Co-authored-by: Claude Opus 4.6 <[email protected]>
|
||
|
Li LiuandClaude Opus 4.6
|
72918439ef |
enhance: remove unused Go dependencies (ansi, fastjson, grpc/examples, sizedwaitgroup) (#47852)
Related to #46199 ## Summary Remove 5 unused or misused Go dependencies to reduce module bloat and consolidate overlapping libraries: - **`mgutz/ansi`** → replaced with inline ANSI escape codes (only used for 3 color constants in migration console) - **`valyala/fastjson`** → replaced with `tidwall/gjson` (only 1 file used fastjson; gjson is already used in 22+ files) - **`google.golang.org/grpc/examples`** → replaced with existing `rootcoordpb` (test file pulled in entire grpc examples repo for a mock server) - **`remeh/sizedwaitgroup`** → replaced with `chan` semaphore + `sync.WaitGroup` (only 2 files, trivial pattern) - **`pkg/errors`** → replaced with `cockroachdb/errors` (the project standard; `pkg/errors` was used in 1 file) ## Behavior change: DeleteLog.Parse() fail-fast on missing fields The `fastjson` → `gjson` migration adds explicit `Exists()` validation for `ts`, `pk`, and `pkType` fields in the JSON parsing branch. Previously, both fastjson and gjson would silently return zero values for missing fields, causing `dl.Pk` to remain nil and panicking downstream. The new code fails fast with a descriptive error at parse time. This is a defensive improvement (the original code had identical silent-failure behavior). ## Performance impact | Change | Path type | Perf delta | Matters? | |--------|-----------|------------|----------| | `pkg/errors` → `cockroachdb/errors` | Cold (offline CLI tool `config-docs-generator`) | Negligible | No | | `mgutz/ansi` → inline ANSI codes | Cold (offline CLI tool `migration/console`) | Marginally faster (eliminates map lookup) | No | | `fastjson` → `gjson` (`DeleteLog.Parse`) | Warm — old-format deltalog deserialization only | **~2.5x slower** per JSON parse (143ns→361ns) | **No** — see below | | `grpc/examples` → `rootcoordpb` | Test only (`client_test.go`) | None | No | | `sizedwaitgroup` → chan+WaitGroup | Test only (`wal_test.go`, `test_framework.go`) | None | No | ### fastjson → gjson regression detail `DeleteLog.Parse()` is called per-row during deltalog deserialization, but **only for the legacy single-field format**. The new multi-field parquet format (`newDeltalogMultiFieldReader`) reads pk/ts as separate Arrow columns and bypasses `Parse()` entirely. Legacy deltalogs are rewritten to parquet format during compaction, so this is a dying code path. Additionally, deltalog loading is I/O-bound — the JSON parse cost (~361ns/row) is negligible compared to disk read and Arrow deserialization overhead. Benchmark (Go 1.24, arm64): ``` BenchmarkFastjsonSmall-4 8,315,624 143.1 ns/op 0 B/op 0 allocs/op BenchmarkGjsonOptimized-4 3,321,613 361.4 ns/op 96 B/op 1 allocs/op ``` ## Test plan - [x] CI build passes - [x] CI code-check passes - [ ] CI ut-go passes - [ ] CI e2e passes - [x] Boundary test cases added (bare number, missing pkType/ts/pk) 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Li Liu <[email protected]> Co-authored-by: Claude Opus 4.6 <[email protected]> |
||
|
yanliang567andClaude Opus 4.6
|
d81ede1818 |
enhance: make GetVersion return docker-image-tag style version string (#47822)
## Summary - Make `GetVersion` API return version strings consistent with Docker image tags, replacing the hardcoded semver - Assemble `MilvusVersion` in Makefile via `git describe --exact-match --tags` (release) or `branch-date-commit` (dev), inject via ldflags - Both `detail=False` (`GetVersion` RPC) and `detail=True` (`Connect` RPC `server_info.build_tags`) return the new format | Scenario | Before | After | |----------|--------|-------| | Release (tag v2.6.11) | `2.6.11` | `2.6.11` | | Dev (master branch) | `2.6.11` | `master-20260224-2d14975d18` | | Feature branch | `2.6.11` | `feature-xxx-20260224-2d14975d18` | Banner still displays semver via `common.Version.String()` — unchanged. issue: #47823 ## Test plan - [x] Makefile dry-run verified: `MilvusVersion` correctly injected in all 4 ldflags targets - [x] Tag scenario verified: `v2.6.11` → `2.6.11` (v prefix stripped) - [x] Dev scenario verified: `feature/getversion-docker-tag-format-20260224-2d14975d18` - [x] Go syntax verified: `gofmt` and `go vet` pass - [ ] Full `make build-go` (requires C++ env) - [ ] E2E: `client.get_server_version()` and `client.get_server_version(detail=True)` 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: yanliang567 <[email protected]> Co-authored-by: Claude Opus 4.6 <[email protected]> |
||
|
tinswzy
|
cd2d8c7f39 |
enhance: support switching of WAL implementation (#45286)
issue: #44726 Introduce an immutable option to prevent accidental modification of critical configurations. Support switching of WAL implementation. Note: This PR depends on [milvus-proto PR #503](https://github.com/milvus-io/milvus-proto/pull/503) being merged first. Signed-off-by: tinswzy <[email protected]> |
||
|
Zhen Ye
|
27525d57cc |
enhance: add glog sink to transfer cgo log into zap (#46721)
issue: #45640 - After async logging, the C log and go log has no order promise, meanwhile the C log format is not consistent with Go Log; so we close the output of glog, just forward the log result operation into Go side which will be handled by the async zap logger. - Use CGO to filter all cgo logging and promise the order between c log and go log. - Also fix the metric name, add new metric to count the logging. - TODO: after woodpecker use the logger of milvus, we can add bigger buffer for logging. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> - Core invariant: all C (glog) and Go logs must be routed through the same zap async pipeline so ordering and formatting are preserved; this PR ensures every glog emission is captured and forwarded to zap before any async buffering diverges the outputs. - Logic removed/simplified: direct glog outputs and hard stdout/stderr/log_dir settings are disabled (configs/glog.conf and flags in internal/core/src/config/ConfigKnowhere.cpp) because they are redundant once a single zap sink handles all logs; logging metrics were simplified from per-length/volatile gauges to totalized counters (pkg/metrics/logging_metrics.go & pkg/log/*), removing duplicate length-tracking and making accounting consistent. - No data loss or behavior regression (concrete code paths): Google logging now adds a GoZapSink (internal/core/src/common/logging_c.h, logging_c.cpp) that calls the exported CGO bridge goZapLogExt (internal/util/cgo/logging/logging.go). Go side uses C.GoStringN/C.GoString to capture full message and file, maps glog severities to zapcore levels, preserves caller info, and writes via the existing zap async core (same write path used by Go logs). The C++ send() trims glog's trailing newline and forwards exact buffers/lengths, so message content, file, line, and severity are preserved and serialized through the same async writer—no log entries are dropped or reordered relative to Go logs. - Capability added (where it takes effect): a CGO bridge that forwards glog into zap—new Go-exported function goZapLogExt (internal/util/cgo/logging/logging.go), a GoZapSink in C++ that forwards glog sends (internal/core/src/common/logging_c.h/.cpp), and blank imports of the cgo initializer across multiple packages (various internal/* files) to ensure the bridge is registered early so all C logs are captured. <!-- end of auto-generated comment: release notes by coderabbit.ai --> Signed-off-by: chyezh <[email protected]> |
||
|
Zhen Ye
|
2c2cbe89c2 |
fix: flush log when os exit (#46608)
issue: #45640 Signed-off-by: chyezh <[email protected]> |
||
|
Buqian Zheng
|
1a7ca339a5 |
feat: expose the Go expr parser to C++ and embed into libmilvus-core.so (#45703)
generated a library that wraps the go expr parser, and embedded that into libmilvus-core.so issue: https://github.com/milvus-io/milvus/issues/45702 see `internal/core/src/plan/milvus_plan_parser.h` for the exposed interface <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Introduced C++ API for plan parsing with schema registration and expression parsing capabilities. * Plan parser now available as shared libraries instead of a standalone binary tool. * **Refactor** * Reorganized build system to produce shared library artifacts instead of executable binaries. * Build outputs relocated to standardized library and include directories. <sub>✏️ Tip: You can customize this high-level summary in your review settings.</sub> <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Signed-off-by: Buqian Zheng <[email protected]> |
||
|
Zhen Ye
|
459425ac84 |
fix: wrong context using by session of grpc client (#46183)
issue: #46182 Signed-off-by: chyezh <[email protected]> |
||
|
Zhen Ye
|
bf76a9e8e2 |
fix: milvus fast fail if any component is not ready (#46069)
issue: #45243 Signed-off-by: chyezh <[email protected]> |
||
|
Zhen Ye
|
2ef18c5b4f |
enhance: remove watch at session liveness check (#45968)
issue: #45724 --------- Signed-off-by: chyezh <[email protected]> |
||
|
Zhen Ye
|
c3fe6473b8 |
enhance: support async write syncer for milvus logging (#45805)
issue: #45640 - log may be dropped if the underlying file system is busy. - use async write syncer to avoid the log operation block the milvus major system. - remove some log dependency from the until function to avoid dependency-loop. --------- Signed-off-by: chyezh <[email protected]> |
||
|
congqixia
|
f51fcc09ae |
fix: resolve SessionWatcher goroutine leak and unstable UT in querycoordv2 (#45627)
Related to #44620 Related to unstable ut "internal/querycoordv2 TestServer/TestNodeUp" Introduce SessionWatcher interface to fix race condition and goroutine leak that caused unstable unit test TestServer/TestNodeUp. Changes: - Add SessionWatcher interface with EventChannel() and Stop() methods - Refactor WatchServices() to return SessionWatcher instead of raw channel - Fix cleanup order in QueryCoordV2: stop watcher before session - Update DataCoord, ConnectionManager to use SessionWatcher - Add MockSessionWatcher for testing Fixes race condition between session context cancellation and internal loop exit. Eliminates goroutine leak by providing explicit lifecycle management. --------- Signed-off-by: Congqi Xia <[email protected]> |
||
|
wei liu
|
7708abd8fe |
fix: Prevent deadlock in runComponent when Prepare fails (#45609)
issue: #45068 When component.Prepare() fails (e.g., net listener creation error), the sign channel was never closed, causing runComponent to block indefinitely at <-sign. This resulted in the entire process hanging after logging the error message. Changes: - Move close(sign) to defer statement in runComponent goroutine - Ensures sign channel is always closed regardless of success/failure - Allows proper error propagation through future.Await() mechanism --------- Signed-off-by: Wei Liu <[email protected]> |
||
|
Xiaofan
|
a9895bb904 |
enhance: add robust handle etcd servercrash (#45304)
related to #45303 fix milvus pod may restart when etcd pod start Signed-off-by: xiaofanluan <[email protected]> |
||
|
Zhen Ye
|
d320ccab99 |
fix: milvus role cannot stop at initializing state (#45244)
issue: #45243 Signed-off-by: chyezh <[email protected]> |
||
|
Zhen Ye
|
576084fe86 |
enhance: support alter collection/database with WAL-based DDL framework (#45266)
issue: #43897 - Alter collection/database is implemented by WAL-based DDL framework now. - Support AlterCollection/AlterDatabase in wal now. - Alter operation can be synced by new CDC now. - Refactor some UT for alter DDL. --------- Signed-off-by: chyezh <[email protected]> |
||
|
Buqian Zheng
|
c284e8c4a8 |
enhance: some minor code cleanup, prepare for scalar benchmark (#45008)
issue: https://github.com/milvus-io/milvus/issues/44452 --------- Signed-off-by: Buqian Zheng <[email protected]> |
||
|
aoiasd
|
cfeb095ad7 |
enhance: forbid build analyzer at proxy (#44067)
relate: https://github.com/milvus-io/milvus/issues/43687 We used to run the temporary analyzer and validate analyzer on the proxy, but the proxy should not be a computation-heavy node. This PR move all analyzer calculations to the streaming node. --------- Signed-off-by: aoiasd <[email protected]> |
||
|
Zhen Ye
|
53e8f150e8 |
fix: check if qn is sqn with label and streamingnode list (#44792)
issue: #44014 - On standalone, the query node inside need to load segment and watch channel, so the querynode is not a embeded querynode in streamingnode without `LabelStreamingNodeEmbeddedQueryNode`. The channel dist manager can not confirm a standalone node is a embededStreamingNode. Bug is introduced by #44099 Signed-off-by: chyezh <[email protected]> |
||
|
yihao.dai
|
2fe244c587 |
enhance: Don't start CDC by default in standalone mode (#44533)
issue: https://github.com/milvus-io/milvus/issues/44123 Signed-off-by: bigsheeper <[email protected]> |
||
|
yihao.daiandchyezh
|
51f69f32d0 |
feat: Add CDC support (#44124)
This PR implements a new CDC service for Milvus 2.6, providing log-based cross-cluster replication. issue: https://github.com/milvus-io/milvus/issues/44123 --------- Signed-off-by: bigsheeper <[email protected]> Signed-off-by: chyezh <[email protected]> Co-authored-by: chyezh <[email protected]> |
||
|
Chun HanandMrPresent-Han
|
da156981c6 |
feat: milvus support posix-compatible mode(milvus-io#43942) (#43944)
related: #43942 Signed-off-by: MrPresent-Han <[email protected]> Co-authored-by: MrPresent-Han <[email protected]> |
||
|
zhagnluandluzhang
|
1a30012014 |
enhance: support trace log level for segcore (#44003)
#43230 Signed-off-by: luzhang <[email protected]> Co-authored-by: luzhang <[email protected]> |
||
|
congqixia
|
3647568cf3 |
fix: Delay holmes initialization preventing initialization too soon (#43773)
Related to #43760 Signed-off-by: Congqi Xia <[email protected]> |
||
|
Zhen Ye
|
5551d99425 |
enhance: remove old arch non-streaming arch code (#43651)
issue: #41609 - remove all dml dead code at proxy - remove dead code at l0_write_buffer - remove msgstream dependency at proxy - remove timetick reporter from proxy - remove replicate stream implementation --------- Signed-off-by: chyezh <[email protected]> |
||
|
sthuangandnico
|
43c3c160ff |
feat: [StorageV2] cmd binlog tool (#43648)
related: #39173 Core Features * Parquet File Analysis: Analyze Milvus binlog Parquet files with metadata extraction * MinIO Integration: Direct connection to MinIO storage for remote file analysis * Vector Data Deserialization: Specialized handling of Milvus vector data in binlog files * Interactive CLI: Command-line interface with interactive exploration Analysis Capabilities * Metadata & Vector Analysis: Extract schema info, row counts, and vector statistics * Data Export: Export data to JSON format with configurable limits * Query Functionality: Search for specific records by ID * Batch Processing: Analyze multiple Parquet files simultaneously User Experience * Verbose Output: Detailed logging for debugging * Error Handling: Robust error handling for file access and parsing * Flexible Output: Support for single file and batch analysis formats --------- Signed-off-by: shaoting-huang <[email protected]> Co-authored-by: nico <[email protected]> |