mirror of
https://github.com/milvus-io/milvus.git
synced 2026-10-12 02:19:10 +00:00
issue: #49846 design doc: docs/design-docs/design_docs/20260912-management-plane-authentication.md Merge dependency: #52579 supplies safe configuration projections. Both changes are needed for the full configuration-redaction requirement in #49846. ## Problem and behavior Port 9091 exposes process shutdown, coordinator management, profiling, event logs, the web console, and legacy REST data operations. Add `common.security.adminAuthEnabled`, default **false**, to authenticate this surface independently of the main service port's data-plane authorization. When enabled: | Surface | Policy | | --- | --- | | `/management/*` except readiness; `/debug/pprof/*`; `/log/level`; `/eventlog`; `/webui/`; `/telemetry`; `/api/v1/_*` | Root HTTP Basic authentication | | Legacy REST data operations under `/api/v1` | Existing valid-user/API-key authentication if `authorizationEnabled=true`; otherwise root Basic authentication | | `/healthz`, `/livez`, `/metrics`, `/metrics_default`, `/management/check/ready` | Open | | `/api/v1/health` | Exempt from the new gate; existing data-plane authentication still applies | After the request-origin checks below, missing/incorrect credentials return 401; non-root users are rejected before password lookup with 403; unavailable credential storage and saturated verification return 503. Dependency details and stack traces remain out of unauthenticated response bodies. HTTP and Gin share one decision and carry a verified root principal through a typed request context for in-process telemetry calls. ## Browser and client requirements The gate rejects requests identified as cross-site or same-site. An unknown Fetch Metadata value and `Origin: null` are rejected as well. Top-level GET document navigation is allowed only on document surfaces, not operator actions or console APIs. **Credential-bearing requests without Origin or Fetch Metadata must include `X-Milvus-Admin-Request: true`.** This is required because missing metadata does not establish that a request is from a non-browser client; ordinary remote HTTP is outside the Fetch Metadata trustworthy-URL guarantee. An explicit header does not override a reported cross-site origin. The metrics-port CORS preflight must not grant this header to cross-origin callers. Browser console access requires HTTPS with Fetch Metadata preserved by the reverse proxy. A document request without sufficient request context is refused without a Basic challenge. Non-browser callers add the explicit header, for example: ```sh curl --user root --header 'X-Milvus-Admin-Request: true' \ 'https://milvus-admin.example/management/querycoord/balance/status' ``` The flag remains opt-in. Existing clients with the flag disabled keep their prior behavior. ## Credential lifecycle and resource bounds Verifier ownership is deterministic in standalone: Proxy, then MixCoord, then worker. Workers create a MixCoord client lazily. Each lookup caller has its own five-second wait bound, so a service-discovery call that ignores the backend context cannot block the HTTP request indefinitely. Its accounting slot remains occupied until the shared backend result completes, preserving the resource bound even in that case. Root hashes are cached for 10 seconds, reused for up to 10 minutes after their last successful fetch if refresh fails, and lookup failures are cached for 2 seconds. The stale window preserves management access during a coordinator outage but can also retain acceptance of a password rotated during that outage. Concurrent hash refreshes share work and admit at most 32 callers, including canceled callers whose singleflight result channels remain retained. Worker client construction uses a single shared completion signal; repeated lookup timeout windows no longer append a retained channel to a stuck constructor. Construction failure can be retried, and a client completed after shutdown is closed instead of published. bcrypt concurrency is bounded to a quarter of GOMAXPROCS, floored at two, with a bounded queue. This is a concurrency cap, not an attempt rate limit; on small pods sustained incorrect passwords can still consume substantial CPU. ## Registration, configuration, and event logs - `http.Register` refuses an unmarked operator route, including method- and host-qualified ServeMux patterns. Assembled-router tests and a console-prefix test cover Gin registration. - Milvus-owned handlers use a private mux. Flag-off compatibility with more-specific legacy DefaultServeMux routes is retained when pprof is enabled; enabling the gate removes that fallback. - `/management/config/alter` always rejects this flag, using the same `config.FormatKey` identity as config storage, so an anonymous caller cannot plant a disabling etcd value before the operator enables the gate. - Key-normalization caching admits at most 4096 entries and only keys whose input and normalized forms fit 1024 bytes, using owned string storage so short substrings cannot pin large request buffers. Larger keys still normalize correctly without retained memo entries. This bounds cache retention, not individual HTTP body size. - The eventlog gRPC stream switches to loopback with authentication enabled. A failed security upgrade closes the wildcard listener. Event-log shutdown uses a separate close signal and stops the gRPC server, preserving safe concurrent recording. - `milvus_admin_auth_total{endpoint,result}` records root-gate and origin decisions using registered route patterns; failure logs are rate limited. - The standard Go runner now includes HTTP, adminauth, coordinator, eventlog, and metrics regressions. It restores fail-fast behavior after sourcing the environment so a failed test command returns a failure from the runner. ## Compatibility and deployment Port 9091 remains plaintext. Use a trusted network or TLS-terminating proxy, rotate root using `UpdateCredential` / `db.update_password`, and update graceful-shutdown and profiling clients before enabling. Editing `defaultRootPassword` does not rotate an existing credential. Official compose deployments and embedded standalone scripts publish 9091, so review that exposure. Browser access needs the HTTPS deployment described above. Regardless of the flag, downstream operator routes without `AdminAuth` now fail registration; the admin-auth key cannot be altered through the management API; configuration normalization is bounded; pprof registration is explicit; and the data-plane stored-password comparison shares the credential-format helper. ## Verification and limits CI observed on head `c8992fc7b4d3fcf9a28bf73ad4700283922367f2` at **2026-09-12 14:30 UTC**: `ci-passed` and `dco-passed` are present. | Check | Result | | --- | --- | | [Build / unit / integration / code checks](https://jenkins-milvus-ci.milvus.io/job/MILVUS-CI-V2-PR-PIPELINES/job/milvus-build-ut-ciloop-pipeline/30883/pipeline-overview/) | SUCCESS; status reports build, code checks, integration, Go, and 9,106 C++ tests passed | | [Go SDK E2E](https://jenkins-milvus-ci.milvus.io/job/MILVUS-CI-V2-PR-PIPELINES/job/milvus-go-sdk-pipeline/15021/pipeline-overview/) | SUCCESS | | [Default E2E](https://jenkins-milvus-ci.milvus.io/job/MILVUS-CI-V2-PR-PIPELINES/job/milvus-e2e-pipeline-dispatcher/19796/pipeline-overview/) | SUCCESS | | Main CI trigger / DCO | SUCCESS | The non-gating Tencent Pulsar gray run reports `ERROR` because its downstream job could not be triggered. The previous PR head had the identical start failure. No Pulsar test result is claimed, and the precise trigger failure is unknown because Jenkins logs are inaccessible here. The TC-US shadow build and legacy Jenkins status still show pending; they are outside the master `ci-passed` requirements. No label was added manually. Local verification and remaining limits: All Go tests use `-tags dynamic,test -gcflags="all=-N -l"`. - Full `internal/util/adminauth`, including race detection, passed. A blocked-constructor regression retained 20,011 extra heap objects after 10,000 canceled waits before the fix and no retained growth after it. Cancellation, retry, panic, and a client completing after close are covered. Four blocked-backend scenarios cover cold cache, expired stale state, and correct/incorrect stale passwords; race runs repeated three times passed, including retained slots and recovery after release. - Full `internal/http` and its race run passed with the explicit-request-header changes, including 34 new net/http/Gin cases. `pkg/eventlog`, `pkg/metrics`, and `pkg/util/merr` also passed. - Configuration-normalization tests passed, including oversized inputs, Unicode expansion, and request-buffer ownership. The full config race suite passed excluding `TestEtcdSource`, whose two tests require a reachable external etcd and failed against this host's deliberately unavailable test endpoint. The config comment and ParamItem documentation match exactly. - The newly scheduled HTTP, adminauth, eventlog, and metrics commands passed with the runner's race and coverage flags. A dispatch harness checked all five package additions through the full and selected-group entry points. Ten injected command failures were swallowed before restoring fail-fast and now stop the runner immediately with the original nonzero status. Coordinator execution was unavailable locally; the subsequent native CI aggregate reports Go success. - Legacy Basic challenges are preserved on the main port, gate-off metrics routes, and the health exemption. A regression using the actual extracted parser with real Gin failed before the repair and passed after it; assembled-route and dynamic-gate compatibility tests were added to the native package suites. The subsequent CI Go check reports success; individual native test logs were not accessible here. - `internal/proxy`, `internal/distributed/proxy`, `internal/coordinator`, and `cmd/roles` cannot compile on this host because `milvus_core.pc`, `milvus-storage.pc`, and `rocksdb.pc` are unavailable. The subsequent native CI aggregate reports build and Go success; this does not change the local compilation limitation. - Full `make test-go` was attempted; native dependency construction failed in the host SDK linker (`library 'System' not found`) before the Go suite. This is not recorded as a passing full test run. - golangci-lint v2.11.3 built with Go 1.26.6 reported zero new issues in `internal/http`, `internal/util/adminauth`, and the changed config/eventlog/metrics/paramtable packages. Native-linked packages could not be linted locally; the subsequent CI code-check status reports success. - A real coordinator failover and a real browser console flow remain unverified. Unit and source-level failure tracing do not establish full browser credential replay or transport behavior. --------- Signed-off-by: Li Liu <[email protected]> Co-authored-by: Claude Opus 5 (1M context) <[email protected]>
79 lines
2.4 KiB
Go
79 lines
2.4 KiB
Go
// Licensed to the LF AI & Data foundation under one
|
|
// or more contributor license agreements. See the NOTICE file
|
|
// distributed with this work for additional information
|
|
// regarding copyright ownership. The ASF licenses this file
|
|
// to you under the Apache License, Version 2.0 (the
|
|
// "License"); you may not use this file except in compliance
|
|
// with the License. You may obtain a copy of the License at
|
|
//
|
|
// http://www.apache.org/licenses/LICENSE-2.0
|
|
//
|
|
// Unless required by applicable law or agreed to in writing, software
|
|
// distributed under the License is distributed on an "AS IS" BASIS,
|
|
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
// See the License for the specific language governing permissions and
|
|
// limitations under the License.
|
|
|
|
package eventlog
|
|
|
|
import (
|
|
"encoding/json"
|
|
"net/http"
|
|
|
|
"github.com/milvus-io/milvus/pkg/v3/mlog"
|
|
)
|
|
|
|
const (
|
|
// ContentTypeHeader is the health check request type header.
|
|
ContentTypeHeader = "Content-Type"
|
|
// ContentTypeText is the health check request type text.
|
|
ContentTypeText = "text/plain"
|
|
// ContentTypeJSON is another health check request type text, which response contains more info.
|
|
ContentTypeJSON = "application/json"
|
|
)
|
|
|
|
// eventLogHandler is the event log http handler.
|
|
type eventLogHandler struct{}
|
|
|
|
// Handler returns the discovery handler.
|
|
//
|
|
// It carries no listener-mode policy of its own. Whether the gRPC stream binds
|
|
// to loopback is owned entirely by EnsureListenerMode, which the embedder calls
|
|
// before serving and again on every config change. A second, per-request source
|
|
// of truth for the same decision would be one that could disagree.
|
|
func Handler() http.Handler {
|
|
return &eventLogHandler{}
|
|
}
|
|
|
|
type eventLogResponse struct {
|
|
Status int `json:"status"`
|
|
Port int `json:"port"`
|
|
}
|
|
|
|
// ServeHTTP handler which start a grpc server listen on random available port.
|
|
func (h *eventLogHandler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
|
|
resp := &eventLogResponse{
|
|
Status: http.StatusOK,
|
|
}
|
|
|
|
port, err := getGrpcLogger()
|
|
if err != nil {
|
|
resp.Status = http.StatusInternalServerError
|
|
writeJSON(w, r, resp)
|
|
return
|
|
}
|
|
|
|
resp.Port = port
|
|
|
|
writeJSON(w, r, resp)
|
|
}
|
|
|
|
func writeJSON(w http.ResponseWriter, r *http.Request, resp *eventLogResponse) {
|
|
w.Header().Set(ContentTypeHeader, ContentTypeJSON)
|
|
bs, err := json.Marshal(resp)
|
|
if err != nil {
|
|
mlog.Warn(r.Context(), "faild to send response", mlog.Err(err))
|
|
}
|
|
w.Write(bs)
|
|
}
|