mirror of
https://github.com/milvus-io/milvus.git
synced 2026-10-12 02:19:10 +00:00
Chun HanandMrPresent-Han
33d4d826f3
issue: #53307 Protocol dependency: milvus-io/milvus-proto#668 (merged). ## What changed - Reuse the existing RPCs to expose retained segment lifecycle metadata and direct compaction sources; retain the existing sealed/loaded segment reporting contract without adding Growing segments to GetQuerySegmentInfo. - Expose collection-scoped retained compaction tasks, with complete task identity, public type/state enums, source/target sets, and failure reasons, while preserving manual job-ID lookup. - Preserve failure reasons across worker cleanup without changing task cleanup or metadata GC semantics. - Add seven **L3-only** data-integrity workloads: fixed V2/V3 Insert, fixed V2/V3 Import, V2 add/drop-field evolution through MixCompaction with schema bump disabled, V3 schema evolution, and V2-to-V3-to-V2 storage transitions. - Validate deterministic full-row/full-field data against an independent oracle, fenced persistence/serving checkpoints, and actual lineage transitions, allowing either automatic or manual compaction. - Provide `tests/scripts/ci_compaction_integrity.sh` as a dedicated, single-invocation, serial entry; leave the shared `tests/scripts/ci_e2e.sh` unchanged from the PR base. - Retain successful audit records only for the dedicated workload marker, preserving ordinary tests' JSON, HTML, and worker-report formats. ## Compatibility and execution boundary - Omitted segment states preserve the existing persistent-state behavior; task and dropped-segment history are subject to existing retention/GC, not a fixed response-size or record-count limit. - These workloads are not added to per-PR E2E or ordinary Nightly pipelines. - The dedicated job must own the deployment/config namespace exclusively, install a compatible PyMilvus build, and provide connection options plus a unique `CI_LOG_PATH`. - The entry enforces `-n 0`, the L3 workload marker, and explicit opt-in, while forwarding connection options, filters, and pytest's exit status. - Insert/Import workloads use three rounds by default; insert load units are batches, not a promise of a physical segment count, and dense vector dimensions are 16. ## Observability semantics and follow-ups - **Segment listing is not an atomic snapshot.** The Proxy first calls `GetSegmentsByStates`, then `GetSegmentInfo`; returned details are filtered again against the requested states (or the legacy default: Flushing, Flushed, Sealed). A segment whose fetched state no longer matches is omitted, and a segment becoming eligible after ID enumeration may be absent. Consumers must not treat a single response as a consistent lifecycle snapshot. - **Existing lookup errors are preserved.** `IncludeUnHealthy` is enabled only when Dropped is explicitly requested. In the default healthy-only path, a selected source becoming Dropped before detail lookup still returns `SegmentNotFound`, as before; that error is not converted into a successful partial list. A history segment being GC'd between the two RPCs can also fail the lookup. - **Aggregate `Completed` means no running tasks, not universal success.** This preserves the existing aggregation semantics: all-failed/all-timeout retained tasks, or an empty task set, can report Completed. Inspect each task's `state` and `failure_reason` for its outcome; proving an actual data transition additionally requires lineage, active-set adoption, and serving-set evidence. This PR does not redefine task retry or cleanup behavior. - **History capacity is a follow-up.** Dropped-segment queries enumerate the retained history and fetch its details, while collection-scoped compaction queries return retained tasks across triggers. Neither path currently provides pagination or a fixed count/size cap, so scan, copy, and response costs grow with the retained record count despite retention/GC. Pagination and capacity controls for both paths are deferred; they must preserve the ability to enumerate complete retained history rather than silently truncate lineage or task evidence. ## Previously recorded verification The results below are earlier recorded runs, including the original six-workload suite before the V2 DDL workload was added; the documentation/review-response update did not rerun tests. - 22 server-independent CI boundary tests passed, covering ordinary/special audit output, worker report merging, scoped execution guards, and the dedicated entry's argument/exit-code behavior. - 20 differential checks against the PR-base logging plugin produced byte-identical ordinary worker JSON, report JSON, and HTML across all five result outcomes. - 14 existing DataIntegrity and etcd-helper unit tests passed. - The three original L1 DataIntegrity parameter instances passed with `-n 2 --dist loadgroup` in 59.39 seconds; their merged successful JSON records retained the original metadata-only shape. - Ruff lint/format, shell syntax, and scoped PR-base whitespace checks passed; the shared runner is byte-identical to the PR base. - The new dedicated entry ran all six workloads serially against the existing local cluster: **6 passed in 3373.52 seconds**, with three rounds for each Insert/Import workload and the storage-transition workload, plus five DDL checkpoints over 25,000 rows. - All six successful reports retained their audit evidence (403 structured events), including every expected round/DDL checkpoint, full segment/task snapshots, serving frontiers, and matching expected/retrieved PK digests. ## Design and job entry - [Data-integrity design contract](https://github.com/MrPresent-Han/milvus/blob/codex/compaction-e2e-observability/docs/design-docs/design_docs/compaction_data_integrity_e2e_design.md) - [Dedicated serial entry](https://github.com/MrPresent-Han/milvus/blob/codex/compaction-e2e-observability/tests/scripts/ci_compaction_integrity.sh) Signed-off-by: MrPresent-Han <[email protected]> Co-authored-by: MrPresent-Han <[email protected]>