issue: #53051, #53052, #53152
## Summary
Align local storage around one path contract: a key is a complete
absolute filesystem path, and the local loon/Arrow filesystem is rooted
at `/`. Previously, some writers used `minio.rootPath` while others
supplied complete local paths to an already-prefixed filesystem, placing
files in different physical locations from those expected by readers and
GC.
Remote keys remain complete bucket-relative object keys; their
filesystem-root convention is unchanged.
## Changes
- Unify Go/C++ local filesystem roots and StorageV3 manifest-base
construction across writers. Preserve the existing base/version pair
when continuing a growing-source manifest.
- Fix local JSON/TEXT stats reads, rejected-stats cleanup, and explore
temporary paths. Exercise resolved V3 stats paths through the actual
`LocalChunkManager.MultiRead` interface.
- Resolve legacy local manifest bases in loaded metadata and locally
referenced snapshots, without eager catalog or manifest-file writes.
Preserve manifest CAS compatibility and pathless V3 summaries;
copy/restore writes use the target storage root.
- Require an absolute, nonempty `localStorage.path` at initialization,
including on remote deployments where it is a cache directory. Freeze
forbidden parameters against configuration-source refresh while
retaining explicit runtime/test overrides, and derive mmap, pprof, and
disk-capacity defaults from the same canonical startup value.
- Make segment deletion prefixes segment-boundary aware, retain
root/identity and snapshot checks, and add BM25 orphan scanning. These
shared GC changes also apply to remote storage; this PR does not claim
every remote code path is untouched.
- Remove the local test-only root override and unused local-only
property initialization chain; use the production storage-config
initialization path.
## Automatic local upgrade
Let `R = localStorage.path` and `M = minio.rootPath`. Before standalone
components start, local deployments check the known legacy locations:
| Existing data | Upgrade handling |
| --- | --- |
| Canonical files under `R/<layout>` | Leave in place. |
| Double-root files under `R/<R-without-leading-slash>/<layout>` | Move
into `R/<layout>` for `insert_log`, `text_log`, `json_stats`,
`index_files`, and `index_v1`. |
| Recognized late-2.6 unified packed indexes under the process CWD |
Move into the corresponding directory under `R`, if visible to the new
process. |
| Legacy manifest base `M/insert_log/<c>/<p>/<s>`, with files under
`R/M/insert_log/<c>/<p>/<s>` | **Do not move the files.** Resolve the
loaded manifest base to that existing absolute location. New segment
layouts use the canonical root; updates to an existing legacy manifest
can retain its base. |
Migration prefers directory/subtree rename and recursively merges
overlapping directories. Cross-filesystem moves fall back to
copy-then-remove. Existing same-name files cause a conflict rather than
content comparison or overwrite, except a published copy recognized as
this migration's own interrupted work. The copy path synchronizes the
file and target directory chain before deleting the source.
Remaining sources are rediscovered on retry; empty legacy directories
are pruned. Once migrated, restarts probe the known old locations
without rescanning canonical data. The initial pass walks legacy file
metadata but does not hash file contents. There is no configured
migration timeout. SIGTERM/SIGINT cancels the migration and prevents
etcd or components from starting; directory walks and cross-filesystem
copies observe cancellation between filesystem reads. A canceled partial
copy is never published and is recopied from the intact source on the
next start.
No user migration script, offline command, reverse migration, journal,
or backup is provided. Successfully moved sources are removed.
## Compatibility and limits
- Preserve the absolute local root, data mounts, and legacy
`minio.rootPath` when upgrading. Canonical 2.6 data stays in place;
misplaced late-2.6 indexes and affected 3.0.0/3.0.1 layouts are handled
as above. Relative roots from older versions are unsupported.
- Automatic migration is standalone/local-only. If old CWD indexes
disappeared with an old container, they cannot be discovered:
preserve/make those files visible or rebuild the derived indexes.
Cross-filesystem copy has no configured timeout and can extend the first
startup, but a shutdown signal cancels it and the next start resumes
from the remaining source.
- Downgrade to the affected old layout is unsupported after migration.
Symlink layouts require manual resolution.
- Known legacy LOB/orphan GC gaps under `R/M/insert_log`, pending old
external-refresh paths, and rare interrupted-copy publication residue
remain outside scope. This is not a claim that every historical file is
recovered or reclaimable.
## Validation
At head `898dba4b41`, `ci-v2/build-ut-cov` passed Build, Code Check,
Integration UT, Go UT, and C++ UT (9,145/9,145). Diff coverage was 100%
for C++ and 81.2% for Go; total patch coverage was 82.8%.
Focused local tests for canonical/frozen `localStorage.path` derivation
and migration cancellation/deadline/copy recovery passed with `-tags
dynamic,test -gcflags='all=-N -l' -count=1`, and `git diff --check`
passed before publication. Full paramtable execution remains locally
blocked by tests attempting to create `/var/lib/milvus/data`;
roles/native-dependent Go tests remain build-blocked by local C++
header/library mismatches. A real 3.0.0/3.0.1 version-to-version upgrade
was not executed locally. `ci-v2/e2e-default` and `ci-v2/go-sdk` are
still pending for this head.
Signed-off-by: xiaofanluan <[email protected]>
Co-authored-by: xiaofanluan <[email protected]>
See #50406
DataCoord today derives aggregate segment metrics by iterating
SegmentInfo's FieldBinlog arrays on every scheduling decision.
For V3 segments the per-field binlog KV entries in etcd are pure
write amplification because the manifest is already the
authoritative path source.
This PR adds a Statistics message embedded on SegmentInfo so
DataCoord reads aggregates (sizes, counts, timestamp range,
null counts, quantiles) directly from a persisted field with no
branching on storage version at the read sites.
Wire formats
SaveBinlogPathsRequest.stats is populated by every flush. The
receiver consumes only stats_binlog_size, and only for V3
segments where stats live in the manifest. V2 derives the
cumulative footprint from the statslog FieldBinlog array.
CompactionSegment.stats is populated by every compactor at
write completion and copied verbatim onto the new SegmentInfo.
Insert and delta aggregates come from the FieldBinlog arrays
the compactor ships; stats blob footprint comes from a counter
the BinlogRecordWriter accumulates inside writeStats /
appendV3Stats and exposes via GetStatsBlobSize.
V3 etcd KV write skip
AlterSegments skips per-FieldBinlog KV writes for V3 segments;
the manifest is authoritative for paths and Stats carries the
aggregates DataCoord needs. Pre-existing V3 segments with binlog
KVs still load correctly via applyBinlogInfo.
Migration
Lazy. NewSegmentInfo back-fills Stats from arrays on first load
when the persisted field is nil. The back-fill lives in memory
until the next mutating operator persists the SegmentInfo proto.
Other notable changes
ShouldDoSingleCompaction reports expired_fraction one bucket
below the precise lower bound to prevent byte-size over-trigger
when binlog sizes vary.
EnsureStats is non-mutating to avoid a data race with
concurrent RLock readers; eager init happens at NewSegmentInfo
and under the meta.segMu write lock in mutating operators.
V3 L0 compaction outputs now carry TimestampFrom/To on the
resulting deltalog, matching the V1 path.
Test plan
- [x] Unit tests for storage.StatisticsCollector and
BuildStatsFromFieldBinlogs
- [x] Unit tests for UpdateSegmentStats V2/V3 differentiation
- [x] Unit tests for ShouldDoSingleCompaction under-estimate
- [x] Unit tests for CompactionSegment.Stats V3 blob size flow
- [x] Unit tests for import L0 / non-L0 timestamp plumbing
- [ ] CI integration / e2e
---------
Signed-off-by: Ted Xu <[email protected]>
issue: #51250
Keep TEXT segments on StorageV3 while making growing-source flush an
explicit allowed mode instead of a mandatory TEXT path.
Propagate create-segment storage versions through WAL flusher and write
buffer, reject storage-version mismatches in DataCoord, and keep raw
segcore chunks sticky for segments that may be flushed from growing
source.
Signed-off-by: luzhang <[email protected]>
Co-authored-by: luzhang <[email protected]>
issue: #50697
Summary:
1. Align StorageV3 column group layout for growing-source flush by
reusing the segment currentSplit/manifest layout, passing a schema-based
split pattern to C++, and using the schema_based writer policy instead
of always writing a single column group.
2. Project growing flush output to the target segment layout by passing
AllowedFieldIDs to C++, so ordinary fields, vector fields, text fields,
and BM25 output fields are all filtered to the old segment layout when
appending to an existing StorageV3 manifest.
3. Stop retrying non-retryable layout mismatches and preserve V3 layout
metadata, so column count/group mismatches do not loop forever and
recovery/balance can restore currentSplit for future layout-compatible
appends.
Signed-off-by: luzhang <[email protected]>
Co-authored-by: luzhang <[email protected]>
issue: #50750
Move sync task submission out of the writebuffer mutex. The writebuffer
still selects segments, yields buffered payload, records sync
checkpoints, and marks segment syncing while holding the lock, but
submits the already built tasks to syncmgr after releasing it.
This prevents ReleaseCollection from hanging in manual-flush preparation
when syncmgr dispatcher backpressure blocks SyncData submission, because
CheckReleaseManualFlushNeed can still acquire the writebuffer read lock.
Also move growing-source follow-up resync out of the callback critical
section and add a regression test that blocks SyncData submission while
verifying writebuffer readers are not blocked.
Signed-off-by: luzhang <[email protected]>
Co-authored-by: luzhang <[email protected]>
issue: #47420
## What this PR does
Project-wide migration of raw `fmt.Errorf` / `errors.New` in function
bodies onto
the `merr` framework, plus the Sys-vs-Input error classification and the
machinery it drives (retriability, fine-grained metrics, segcore
unification),
plus the convention docs and a linter that keeps it from regressing.
Scope: storage, proxy, coordinators (root/data/query), query node, data
node,
`pkg/util` & `internal/util`, expression parser, message queue,
streaming, and
misc packages. Bare raw-error usages went from ~3000 to a ~340 allowlist
(package-level sentinels / build-tag / test sites).
---
## How to review this PR
It is large but the vast majority is mechanical. Changes fall into three
tiers;
spend review budget on Part 2 and Part 3.
### Part 1 — Mechanical standardization (low risk, verify by rule)
Each converted call follows one of a small fixed set of rules. To
review, check
that each site obeys the matching rule rather than reading every line:
| Pattern | Rule |
|---|---|
| `fmt.Errorf("...")` originating a new error | →
`merr.WrapErrXxxMsg("...")` with a code matching the failure's meaning |
| Adding context to an existing typed error | → `merr.Wrap(err, "...")`
/ `merr.Wrapf(...)` — **preserves** the inner code (never `WrapErr*Err`,
which overwrites it) |
| Errors inside the streaming subsystem | → `status.New*` factories
(StreamingError), **not** merr — this is the component-internal dialect
(see `docs/dev/error_handling_guide.md`) |
| Low-level / control-flow signal caught by `errors.Is` | → kept as a
package-level `errors.New` sentinel (lowercase, same-package) |
Conventions are documented in `docs/dev/error_handling_guide.md`
(how-to) and
`docs/dev/error_sentinel_convention.md` (rules + audit). A
`gocritic`/`ruleguard`
rule (`rawmerrerror`, in `rules.go`) enforces "no raw `return
errors.New/fmt.Errorf`"
under `make verifiers`.
### Part 2 — Behavior changes (review these closely)
These are the sites where the wire contract or runtime behavior changes,
not just
the source text. Listed by category; representative locations given,
full set in
the diff.
**A. gRPC wire-code shifts: `UnexpectedError(1)/Code 65535` → typed
code.**
Where a handler previously returned a raw error (collapsed to
`Code=65535` on the
wire), it now returns a typed merr, so the client sees a real code. The
most
common shift is to `IllegalArgument(5)/Code 1100` (ParameterInvalid).
Touch
points include datanode task handlers (CreateTask/Query/Drop), proxy
Upsert,
querynode GetMetrics, datacoord CreateIndex, httpserver query-response
builder,
and typeutil schema validation. One code refinement: an index-param
validation
moved `1100` → `1101` (ParameterMissing). **Client/SDK assertions and
any code
that switched on `Code=65535` for these paths must be re-checked** (the
go_client
e2e assertions were already aligned in this PR).
**B. Prometheus `status` label contract change (externally visible).**
The proxy metric's coarse `fail` / `rejected` values are split into
`fail_input` / `fail_system` and `rejected_user` / `rejected_system` (in
`requestutil.ParseMetricLabel`; auth/privilege rejections count as
`rejected_user`), so dashboards can attribute a failure to caller vs
operator.
**Dashboards/alerts querying `status="fail"` must migrate to
`status=~"fail_.*"`, and `status="rejected"` to
`status=~"rejected_.*"`.** The
in-repo Grafana dashboard is already migrated; external dashboards built
on the
old values silently go empty after upgrade. This is the one change that
requires an ops-side migration.
**C. Retriability semantics.**
- C1: `merr.Status(err)` now forces `Retriable=false` when the error is
an
`InputError` — a malformed request can never succeed on blind retry, so
clients
never get the self-contradictory "your input is wrong but you may
retry".
- C2: `retry.Do` short-circuits an `InputError` (non-retriable) — **but
only when
the caller did not pass a `RetryErr` predicate**. The check is an
`if c.isRetryErr != nil { ... } else if InputError { ... }` *mutually
exclusive*
branch (`pkg/util/retry/retry.go`): an explicit `RetryErr` takes
precedence and
bypasses the InputError abort. `retry.Handle` deliberately does **not**
apply
the InputError abort (its callers signal abort via `shouldRetry=false`).
Four
flusher startup callsites that must retry through transient "not ready"
errors
were given explicit `RetryErr` escape hatches.
**D. segcore (C++→Go) error classification.**
A single shared Go-side table (`pkg/util/merr/segcore.go`) maps each
segcore code
to a merr sentinel + InputError/signal category, replacing scattered
hand-written
`if errorCode == ...` switches in the cgo wrappers. **Wire `Code` values
change
for every segcore pass-through error, not just the remapped ones.**
Named
sentinels remap (C++ `2003` → merr `2001`, `2033` → `2002`,
Folly/Knowhere codes
likewise); **all remaining pass-through codes (`2004`–`2043`, previously
surfaced to clients as raw C++ enum values) now serialize as `2000`**
(`ErrSegcore`), with the original C++ code preserved in the `Reason`
text
(`segcoreCode=...`); unknown/future codes collapse to `2000` as well
(pinned by
the `wire_code_projection` test). Transient segcore classes (object
storage /
file IO / OOM / mmap / FieldNotLoaded — 11 codes) now report
`Retriable=true`.
**Any client switching on raw segcore codes in the `2004`–`2043` range
must be
re-checked**; the in-Reason code remains available for diagnostics.
Signal
codes (PretendFinished / FollyCancel) are recognized centrally.
`errors.Is`-based
control flow on these (e.g. scheduler skip/retry) is preserved.
**E. InputError classification (25 sentinels + dynamic marks).**
25 sentinels in `errors.go` carry `WithErrorType(InputError)` (the
Collection /
ResourceGroup / Database families, `ErrIndexDuplicate`,
`ErrParameterInvalid`,
`ErrPrivilegeNotAuthenticated`, `ErrImportFailed`, `ErrQueryPlan`, ...),
plus dynamic
marks for the 8 segcore input codes (ExprInvalid, DimNotMatch,
MetricTypeInvalid, FieldIDInvalid, ...) and
`WrapErrAsInputError`. The widest blast radius is `ErrParameterInvalid`
(1100):
~2335 `WrapErrParameterInvalid*` callsites now classify as input /
non-retriable. Because of C1/C2 this changes retriability for
any path that returns these. **The audit to confirm no transient path
was
mis-marked is the single most important review item** (see Part 3). One
reverse
correction: storage field-stats parsing moved from `ErrParameterInvalid`
(input)
to `ErrDataIntegrity` — a corrupted stored stat is data corruption, not
user
input.
### Part 3 — Known risks & traps (called out proactively)
1. **`merr.Wrap` vs `WrapErr*Err` (code-masking).** `WrapErr*Err` builds
a
`wrappedMilvusError{sentinel: ErrServiceInternal}` whose `code()`
returns the
*outer* sentinel — it overwrites the inner typed code and hides the
`errors.Is`
chain. This is intentional (use it to *deliberately* downgrade), but it
was a
recurring conversion defect; the rule "add context with `merr.Wrap`,
downgrade
with `WrapErr*Err`" is enforced by convention and reviewed across the
diff.
2. **InputError × `retry.Do` blast radius.** Marking a sentinel
`InputError` makes
any `retry.Do(...)` without a `RetryErr` predicate stop retrying it.
Reviewers
should sanity-check that no transient use of the 19 newly-marked
sentinels
(especially `ErrParameterInvalid`) sits inside a retry loop that needed
to keep
spinning. The known flusher cases were handled (see C2).
3. **The ~340 raw-error allowlist.** What remains as bare `errors.New`
is, by
design: package-level sentinels (caught by `errors.Is`), `//go:build
test`
sites, and out-of-band trees (`cmd/`, `tests/`, codegen, walimpls). The
linter
only bans the *direct-return* form; assignment-then-return escapes and
the full
no-exceptions ban are deferred to an AST-based linter (Tier 2,
documented).
4. **segcore C++ second step deferred.** This PR unifies classification
on the Go
side; splitting the dual-semantic C++ codes at the source is a
follow-up.
---
## Validation
- `make verifiers`: Go side clean (gofmt + static-check across modules,
including
the new `rawmerrerror` rule with a 0-hit baseline repo-wide).
- `make test-go`: passing; the one real regression introduced (a
datanode
`invalid_task_type` assertion shifting `1` → `5` from a ParameterInvalid
conversion) was fixed in-tree.
- go_client e2e CreateIndex assertions aligned to the new merr messages.
---------
Signed-off-by: zhenshan.cao <[email protected]>
Co-authored-by: Claude Opus 4.7 (1M context) <[email protected]>
relate: #49716
## Summary
Move BM25/MinHash function execution from DN/QN pipeline embedding nodes
to the write-before path before WAL append. Add collection-scoped
function runner caching and keep pipeline-side compatibility fill for
old insert messages without generated function output fields.
Signed-off-by: aoiasd <[email protected]>
Related to #49398
Bump Go module references from pkg/v2 and milvus-proto/go-api/v2 to
pkg/v3 and milvus-proto/go-api/v3 so the client tracks the Milvus 3.x
release line.
This prepares the repository for the upcoming 3.x.y release by aligning
imports, module dependencies, and proto API references with the new
major-version module paths.
---------
Signed-off-by: Congqi Xia <[email protected]>
## Summary
- Add schema version check on insert path in streaming node shard
interceptor, rejecting inserts with mismatched schema version
- Add
`AlterCollection`/`AppendNewCollectionSchema`/`CheckIfCollectionSchemaVersionMatch`
methods to shard manager for schema lifecycle management
- Propagate schema version through segment allocation chain:
`shard_manager_segment` → `partition_manager` → `segment_alloc_worker` →
`CreateSegment` message header
- Add `SchemaVersionMismatch` streaming error type (classified as
unrecoverable to force proxy metadata refresh)
- Track collection schema in `CollectionInfo` with
`CollectionSchemaOfVChannel` for version gating
related: https://github.com/milvus-io/milvus/issues/48808
design doc:
https://github.com/milvus-io/milvus-design-docs/blob/main/design_docs/20260129-add-function-field-design.md
## Test plan
- [x] partition_manager_test.go updated for new `schemaVersion`
parameter
- [x] wal_test.go updated with CollectionSchema in CreateCollection
messages
- [ ] CI: code-check, ut-go, integration-test
🤖 Generated with [Claude Code](https://claude.ai/code)
Signed-off-by: MrPresent-Han <[email protected]>
Co-authored-by: MrPresent-Han <[email protected]>
Co-authored-by: Claude Opus 4.6 <[email protected]>
Replace all hard-coded `-1` literal used as manifest version with the
named constant `packed.ManifestEarliest` (value `0`) for new segment
creation, making the intent explicit and consistent across the codebase.
Key changes:
- Define `ManifestLatest` (-1) and `ManifestEarliest` (0) constants in
`internal/storagev2/packed/constant.go`
- Update all production code (segment_manager, write_buffer,
importv2/util, binlog_record_writer) to use `packed.ManifestEarliest`
- Update all test code (pack_writer_v3_test, task_test) to use
`packed.ManifestEarliest` instead of raw `-1`
- Fix `CreateManifestForSegment` FFI call: use version `0` (earliest)
instead of `-1` (latest) and use `LOON_TRANSACTION_RESOLVE_OVERWRITE`
resolve strategy
- Fix `FFIPackedWriter.Close` to use
`LOON_TRANSACTION_RESOLVE_OVERWRITE` resolve strategy
issue: #48543
---------
Signed-off-by: Congqi Xia <[email protected]>
issue: #47790
Replace 131 occurrences of fmt.Sprint(paramtable.GetNodeID()) with the
existing paramtable.GetStringNodeID() across 40 files. This avoids
redundant fmt.Sprint allocations on every call since GetStringNodeID()
uses the more efficient strconv.FormatInt. In the search hot path alone,
this eliminates 3+ string allocations per segment per query.
Signed-off-by: lyang24 <[email protected]>
Co-authored-by: Claude Opus 4.6 <[email protected]>
issue: #44726
Introduce an immutable option to prevent accidental modification of
critical configurations.
Support switching of WAL implementation.
Note: This PR depends on [milvus-proto PR
#503](https://github.com/milvus-io/milvus-proto/pull/503) being merged
first.
Signed-off-by: tinswzy <[email protected]>
Related to #44956
Introduce StorageV3 constant to distinguish loon manifest format from
StorageV2 packed writer format. When UseLoonFFI is enabled, segments are
now marked with StorageVersion=3 instead of 2.
The V3 format currently shares most V2 logic paths since the underlying
storage format is similar. In a future refactor, V3-specific logic will
be separated from the V2 code path for better readability.
Changes:
- Add StorageV3 constant (value=3) in Go and C++ code
- Set StorageVersion=3 when UseLoonFFI is enabled for:
- Compaction params
- Import segments allocation
- Write buffer growing segments
- Streaming node segment allocation
- Add V3 support to existing V2 code paths (read/write/sync)
- Separate V2 and V3 writer creation in NewBinlogRecordWriter
Signed-off-by: Congqi Xia <[email protected]>
Related to #46595
Remove the EnableStorageV2 config option and enforce StorageV2 format
across all write paths including compaction, import, write buffer, and
streaming segment allocation. V1 format write tests are now skipped as
writing V1 format is no longer supported.
---------
Signed-off-by: Congqi Xia <[email protected]>
Related to #44956
This change propagates the useLoonFFI configuration through the import
pipeline to enable LOON FFI usage during data import operations.
Key changes:
- Add use_loon_ffi field to ImportRequest protobuf message
- Add manifest_path field to ImportSegmentInfo for tracking manifest
- Initialize manifest path when creating segments (both import and
growing)
- Pass useLoonFFI flag through NewSyncTask in import tasks
- Simplify pack_writer_v2 by removing GetManifestInfo method and relying
on pre-initialized manifest path from segment creation
- Update segment meta with manifest path after import completion
This allows the import workflow to use the LOON FFI based packed writer
when the common.useLoonFFI configuration is enabled.
---------
Signed-off-by: Congqi Xia <[email protected]>
issue: #43072, #43289
- manage the schema version at recovery storage.
- update the schema when creating collection or alter schema.
- get schema at write buffer based on version.
- recover the schema when upgrading from 2.5.
---------
Signed-off-by: chyezh <[email protected]>
issue: #37172
- add redo interceptor to implement append context refresh. (make new
timetick)
- add create segment handler for flusher.
- make empty segment flushable and directly change it into dropped.
- add create segment message into wal when creating new growing segment.
- make the insert operation into following seq: createSegment -> insert
-> insert -> flushSegment.
- make manual flush into following seq: flushTs -> flushsegment ->
flushsegment -> manualflush.
---------
Signed-off-by: chyezh <[email protected]>
issue: #36621
1. Add API to access task runtime metrics, including:
- build index task
- compaction task
- import task
- balance (including load/release of segments/channels and some leader
tasks on querycoord)
- sync task
2. Add a debug model to the webpage by using debug=true or debug=false
in the URL query parameters to enable or disable debug mode.
Signed-off-by: jaime <[email protected]>
