mirror of
https://github.com/milvus-io/milvus.git
synced 2026-10-12 02:19:10 +00:00
Jamesandxiaofanluan
37f3a829e7
issue: #53051, #53052, #53152 ## Summary Align local storage around one path contract: a key is a complete absolute filesystem path, and the local loon/Arrow filesystem is rooted at `/`. Previously, some writers used `minio.rootPath` while others supplied complete local paths to an already-prefixed filesystem, placing files in different physical locations from those expected by readers and GC. Remote keys remain complete bucket-relative object keys; their filesystem-root convention is unchanged. ## Changes - Unify Go/C++ local filesystem roots and StorageV3 manifest-base construction across writers. Preserve the existing base/version pair when continuing a growing-source manifest. - Fix local JSON/TEXT stats reads, rejected-stats cleanup, and explore temporary paths. Exercise resolved V3 stats paths through the actual `LocalChunkManager.MultiRead` interface. - Resolve legacy local manifest bases in loaded metadata and locally referenced snapshots, without eager catalog or manifest-file writes. Preserve manifest CAS compatibility and pathless V3 summaries; copy/restore writes use the target storage root. - Require an absolute, nonempty `localStorage.path` at initialization, including on remote deployments where it is a cache directory. Freeze forbidden parameters against configuration-source refresh while retaining explicit runtime/test overrides, and derive mmap, pprof, and disk-capacity defaults from the same canonical startup value. - Make segment deletion prefixes segment-boundary aware, retain root/identity and snapshot checks, and add BM25 orphan scanning. These shared GC changes also apply to remote storage; this PR does not claim every remote code path is untouched. - Remove the local test-only root override and unused local-only property initialization chain; use the production storage-config initialization path. ## Automatic local upgrade Let `R = localStorage.path` and `M = minio.rootPath`. Before standalone components start, local deployments check the known legacy locations: | Existing data | Upgrade handling | | --- | --- | | Canonical files under `R/<layout>` | Leave in place. | | Double-root files under `R/<R-without-leading-slash>/<layout>` | Move into `R/<layout>` for `insert_log`, `text_log`, `json_stats`, `index_files`, and `index_v1`. | | Recognized late-2.6 unified packed indexes under the process CWD | Move into the corresponding directory under `R`, if visible to the new process. | | Legacy manifest base `M/insert_log/<c>/<p>/<s>`, with files under `R/M/insert_log/<c>/<p>/<s>` | **Do not move the files.** Resolve the loaded manifest base to that existing absolute location. New segment layouts use the canonical root; updates to an existing legacy manifest can retain its base. | Migration prefers directory/subtree rename and recursively merges overlapping directories. Cross-filesystem moves fall back to copy-then-remove. Existing same-name files cause a conflict rather than content comparison or overwrite, except a published copy recognized as this migration's own interrupted work. The copy path synchronizes the file and target directory chain before deleting the source. Remaining sources are rediscovered on retry; empty legacy directories are pruned. Once migrated, restarts probe the known old locations without rescanning canonical data. The initial pass walks legacy file metadata but does not hash file contents. There is no configured migration timeout. SIGTERM/SIGINT cancels the migration and prevents etcd or components from starting; directory walks and cross-filesystem copies observe cancellation between filesystem reads. A canceled partial copy is never published and is recopied from the intact source on the next start. No user migration script, offline command, reverse migration, journal, or backup is provided. Successfully moved sources are removed. ## Compatibility and limits - Preserve the absolute local root, data mounts, and legacy `minio.rootPath` when upgrading. Canonical 2.6 data stays in place; misplaced late-2.6 indexes and affected 3.0.0/3.0.1 layouts are handled as above. Relative roots from older versions are unsupported. - Automatic migration is standalone/local-only. If old CWD indexes disappeared with an old container, they cannot be discovered: preserve/make those files visible or rebuild the derived indexes. Cross-filesystem copy has no configured timeout and can extend the first startup, but a shutdown signal cancels it and the next start resumes from the remaining source. - Downgrade to the affected old layout is unsupported after migration. Symlink layouts require manual resolution. - Known legacy LOB/orphan GC gaps under `R/M/insert_log`, pending old external-refresh paths, and rare interrupted-copy publication residue remain outside scope. This is not a claim that every historical file is recovered or reclaimable. ## Validation At head `898dba4b41`, `ci-v2/build-ut-cov` passed Build, Code Check, Integration UT, Go UT, and C++ UT (9,145/9,145). Diff coverage was 100% for C++ and 81.2% for Go; total patch coverage was 82.8%. Focused local tests for canonical/frozen `localStorage.path` derivation and migration cancellation/deadline/copy recovery passed with `-tags dynamic,test -gcflags='all=-N -l' -count=1`, and `git diff --check` passed before publication. Full paramtable execution remains locally blocked by tests attempting to create `/var/lib/milvus/data`; roles/native-dependent Go tests remain build-blocked by local C++ header/library mismatches. A real 3.0.0/3.0.1 version-to-version upgrade was not executed locally. `ci-v2/e2e-default` and `ci-v2/go-sdk` are still pending for this head. Signed-off-by: xiaofanluan <[email protected]> Co-authored-by: xiaofanluan <[email protected]>