mirror of
https://github.com/milvus-io/milvus.git
synced 2026-10-12 02:19:10 +00:00
Zhen YeandClaude Opus 4.6
06ba9d2298
issue: #40451 - WAL recovery: replace the legacy flusher with asynchronous VChannel/Segment persistence, retained-message completion, and owner-fenced snapshots before a single global checkpoint. Publish schemas before VChannel base metadata and use the base checkpoint to filter incomplete schema publication during recovery; delegate task concurrency to the shared NodeScheduler. Preserve transaction state through RecoveryBarrier with one scanner, reconcile legacy segment state during migration, and mark the WAL unavailable if background checkpoint persistence terminates unexpectedly. - WALSummary and idempotency: keep summaries and keyed-write deduplication always available; persist chunks/manifests independently, restore deduplication windows, provide bounded Delete reads, and manage WAL coverage, retention, and bootstrap replay boundaries. Seal on staged size or WAL tail pressure, report terminal Summary failures to the WAL owner, and perform physical GC outside reader/publication locks. Remove obsolete idempotency retention settings. - Data persistence: materialize missing function outputs before growing persistence, publish growing L1 data to DataCoord before completing Inserts, and track commits with DataVersion. Retire empty L1 segments explicitly through DataCoord without generating a manifest or publishing DataView; install the SN tombstone without a DataVersion only after retirement is confirmed, allowing ManualFlush to complete. Materialize retained Deletes into L0 using size, age, recovery pressure, or explicit WAL boundaries; retain the Summary-based materializer for future QueryView integration. Report complete recovery positions and poison work after WAL ownership loss. - Lifecycle completion: synchronize ManualFlush, FlushAll, and snapshot creation through consuming-side L1/L0 completion and broadcast acknowledgements. Let Broadcaster add CChannel centrally while callers supply the required collection VChannels. Rename the import ID-range update message to UpdateImport while preserving its payload and behavior. Preserve coordinator-owned Import visibility and legacy CommitImport completion during upgrades, and flush surviving partitions on DropPartition. Fence dropped objects in memory, publish durable tombstones only after asynchronous dependencies finish, and retire metadata and historical schemas through checkpoint- and Summary-gated GC. Cancel stale QueryCoord orphan-segment cleanup when a loaded collection reassigns the node to a replica. --------- Signed-off-by: chyezh <[email protected]> Co-authored-by: Claude Opus 4.6 <[email protected]>