mirror of
https://github.com/milvus-io/milvus.git
synced 2026-10-12 02:19:10 +00:00
10a6bd999d
issue: #49846 Design: [Safe Configuration Projections](https://github.com/liliu-z/milvus/blob/enhance/secure-config-projections/docs/design-docs/design_docs/20260826-safe-configuration-projections.md) Configuration diagnostics and management endpoints previously shared raw manager views with runtime consumers. Those views can expose credentials, connection topology, process-environment entries, and deletion tombstones. This change adds declared scalar/group metadata and safe projections while preserving the management configuration API's existing write contract. ## Behavior - Safe configuration views omit undeclared entries and mask sensitive values. Management point reads return `*****` for sensitive keys and reject undeclared keys. Registration and sensitivity decisions account for aliases and dynamic group namespaces. - `ParamItem.Sensitivity` uses `Auto` (zero value), `Sensitive`, and `NonSensitive`; explicit states override inference. Sensitivity controls projections and logging only. Generic management SET/DELETE retains the existing immutable, `mq.type`, and `common.security.adminAuthEnabled` restrictions, original request formats, caller-key handling, duplicate validation, and etcd transaction behavior. Sensitive, unregistered, and security-related settings remain writable unless an existing restriction applies. A successful write does not make its raw value visible through GET. - Built-in file/etcd publication and safe reads share a snapshot lock, keeping source values and their spelling classification together. Collisions and first-time spelling changes retain ambiguity across deferred ownership events and removal. Initial local file spellings are established before environment overrides. Raw manager getters retain their previous locking behavior and source priority. - Effective group reads omit tombstones; deleting or resetting an overlay through an alias addresses equivalent stored overlay spellings. - Sensitivity metadata is installed before scalar/group declarations become visible to concurrent refreshes, including fallback keys and the hook namespace. Positive inventories check actual declarations for reviewed credentials, connection targets, transport, authentication, trust, and routing settings, including the Pulsar tenant and namespace consumed by topic construction. Registration classification stays internal; audit tests use public reads and projections. - WAL configuration maps, management mutations, and dynamic config events omit request names and values from logs. Configuration parser, remote-source TLS initialization, and opaque plugin/callback diagnostics avoid echoing sensitive input. Immutable startup writes mask composite keys containing the protected root path. - Native plugin initialization reports the failed stage without the configured path, plugin-provided name, or opaque exception text. Sanitization precedes native exception observers and Go startup logs while retaining typed codes, allocation-failure classification, and untyped-exception counting. Go plugin wrappers preserve returned errors while suppressing their diagnostic text. - Kafka diagnostic strings mask credentials. `FieldBroadcastID` uses the native `uint64` type and preserves its full range through RPC metadata. Projections remain uncached O(N) diagnostic/management operations. Management authentication, authorization, and additional write restrictions are separate work; endpoint authentication is supplied by #52580, now included through master. This PR does not encrypt stored configuration or audit application-wide backend diagnostics and logs emitted inside out-of-tree plugins. ## Validation ### Merge with current master Validated merge commit `24a8c123fd` with master `2e0677c1d2`. All Go tests used `-tags dynamic,test -gcflags="all=-N -l" -count=1`. - Full `pkg/config` passed with an isolated embedded etcd server. - Focused ParamTable sensitivity, declaration ordering, projection, and lifecycle tests passed. - Config race tests passed for all normalized key spellings, cache entry/byte/concurrency bounds, owned request strings, sensitivity aliases, `knowhere` identity, and refresh consistency. - The actual config GET/ALTER handlers, extracted into an isolated adapter, passed retained coordinator regressions against real embedded etcd. Coverage includes ordinary and sensitive writes, redacted GET, admin-auth SET/DELETE rejection under five spellings, no partial writes in rejected batches, `mq.type` aliases, transaction failures, and omission of payload names/values from logs. - Full `internal/http` and `internal/util/adminauth` passed. `go vet` passed for config, ParamTable, and the extracted handler adapter; `git diff --check` passed. The real coordinator package could not compile locally because RocksDB and `milvus_core` native dependencies are unavailable. The extracted-handler checks do not replace native coordinator CI. ### Feature validation before the master merge Prior local validation used `-tags dynamic,test -gcflags="all=-N -l" -count=1`: - Full `pkg/config` suite passed with an isolated embedded etcd server. - Focused ParamTable tests passed for the sensitivity enum, scalar/fallback overrides, Pulsar resource selectors under dotted/slash/uppercase-underscore/collapsed spellings, raw runtime values, projections, callback logs, declaration audits, and credential/immutability rules. The Pulsar point-read regression failed on the prior head and passed after the fix. - Focused ParamTable `-race` tests passed for enum behavior, Pulsar resource visibility, and sensitivity registration before declaration visibility. - The retained coordinator regressions passed in an isolated adapter that extracts the actual configuration HTTP handlers and connects them to real embedded etcd. The same write-behavior checks passed against the pre-PR write handler. Coverage includes sensitive/unregistered/security settings, Pulsar tenant/namespace SET and DELETE across four key spellings, single and batch requests, empty values, omitted/null resets, raw storage values, redacted GET responses, original key spellings, immutable/WAL rejection, duplicate error behavior, and transaction atomicity. Revised-handler logs omit request canaries on successful writes and transaction failure. - `go vet -tags dynamic,test` passed for config, ParamTable, and the extracted current-handler adapter; `git diff --check` passed. Full ParamTable testing was attempted but stopped because existing tests require creating `/var/lib/milvus/data`, which this host cannot write. Full coordinator tests were attempted but could not compile on this host because RocksDB and `milvus_core` native libraries are unavailable. The extracted-handler checks do not replace the full coordinator/native CI run. Broader backend RPC error descriptions, third-party network-client diagnostics, and logs emitted inside external plugins remain outside the verified log-sanitization coverage. --------- Signed-off-by: Li Liu <[email protected]> Co-authored-by: OpenAI Codex <[email protected]> Co-authored-by: Claude Opus 5 (1M context) <[email protected]>