mirror of
https://github.com/milvus-io/milvus.git
synced 2026-10-12 02:19:10 +00:00
Ted Xu
2fc31f825a
See #51351 Coordinator metadata mutations that are logically one change were persisted as several independent catalog calls, each its own etcd/TiKV transaction. This adds a composite primitive so those collapse into a single guarded transaction. Update(actions ...UpdateAction) is added to the DataCoord, RootCoord and QueryCoord catalogs, plus a txn.Builder-backed SaveRecoverySnapshot on StreamingNode. A shared kv/txn.Builder accumulates ordered Save/Remove ops plus commit-marker ops; txn.Commit applies them atomically in one etcd transaction when the op set fits MaxEtcdTxnNum, otherwise via a caller-ordered chunked fallback that writes the commit marker last, so a crash leaves a recoverable partial state. Each action's (Type, Entry) reuses the legacy per-model key/value encoding, so persisted keys and values are byte-for-byte unchanged. Two goals: fewer KV round-trips (a large win on TiKV), and free atomicity whenever the mutation fits one etcd transaction (128 ops), removing the hand-rolled crash-safety ordering callers used to maintain. Migrated call sites (in-memory state applied only after the write): - DataCoord: mix-compaction completion, channel drop, external collection refresh job drop and task creation, clustering-compaction partition-stats save, partition-stats + analyze-task cleanup. - RootCoord: create-collection, drop-collection. - QueryCoord: replica spawn + redundant release. - StreamingNode: WAL recovery snapshot. Notes: - Compaction retires compactFrom inputs via AlterSegment, reproducing the legacy AlterSegments encoding byte-for-byte, including the handleDroppedSegment GC-compat write; channel drop keeps the record-only UpdateSegment encoding to avoid a per-segment prefix read on an unbounded batch. - Foreign or unsupported (Entry, Type) combinations are rejected as a ServiceInternal programming error with no write. Deferred as follow-ups (not in this PR): - QueryCoord ResourceManager per-node assignment loops. Orthogonal to this primitive: SaveResourceGroup is already variadic, so the fix is a caller-side staging-overlay refactor with no interface change. - RootCoord dropRole (DropRole + DropGrant). Blocked until the txn engine can emit exact and prefix removals in one transaction. Testing: unit tests for all four catalogs (encoding equivalence, ordering, partial failure, idempotent retry) and the migrated callers; lint clean with the pinned golangci-lint. Signed-off-by: Ted Xu <[email protected]>