Files
Li LiuandClaude Opus 5 e008a8429d enhance: add optional root authentication for the management plane (#52580)
issue: #49846

design doc:
docs/design-docs/design_docs/20260912-management-plane-authentication.md

Merge dependency: #52579 supplies safe configuration projections. Both
changes are needed for the full configuration-redaction requirement in
#49846.

## Problem and behavior

Port 9091 exposes process shutdown, coordinator management, profiling,
event logs, the web console, and legacy REST data operations. Add
`common.security.adminAuthEnabled`, default **false**, to authenticate
this surface independently of the main service port's data-plane
authorization.

When enabled:

| Surface | Policy |
| --- | --- |
| `/management/*` except readiness; `/debug/pprof/*`; `/log/level`;
`/eventlog`; `/webui/`; `/telemetry`; `/api/v1/_*` | Root HTTP Basic
authentication |
| Legacy REST data operations under `/api/v1` | Existing
valid-user/API-key authentication if `authorizationEnabled=true`;
otherwise root Basic authentication |
| `/healthz`, `/livez`, `/metrics`, `/metrics_default`,
`/management/check/ready` | Open |
| `/api/v1/health` | Exempt from the new gate; existing data-plane
authentication still applies |

After the request-origin checks below, missing/incorrect credentials
return 401; non-root users are rejected before password lookup with 403;
unavailable credential storage and saturated verification return 503.
Dependency details and stack traces remain out of unauthenticated
response bodies. HTTP and Gin share one decision and carry a verified
root principal through a typed request context for in-process telemetry
calls.

## Browser and client requirements

The gate rejects requests identified as cross-site or same-site. An
unknown Fetch Metadata value and `Origin: null` are rejected as well.
Top-level GET document navigation is allowed only on document surfaces,
not operator actions or console APIs.

**Credential-bearing requests without Origin or Fetch Metadata must
include `X-Milvus-Admin-Request: true`.** This is required because
missing metadata does not establish that a request is from a non-browser
client; ordinary remote HTTP is outside the Fetch Metadata
trustworthy-URL guarantee. An explicit header does not override a
reported cross-site origin. The metrics-port CORS preflight must not
grant this header to cross-origin callers.

Browser console access requires HTTPS with Fetch Metadata preserved by
the reverse proxy. A document request without sufficient request context
is refused without a Basic challenge. Non-browser callers add the
explicit header, for example:

```sh
curl --user root --header 'X-Milvus-Admin-Request: true' \
  'https://milvus-admin.example/management/querycoord/balance/status'
```

The flag remains opt-in. Existing clients with the flag disabled keep
their prior behavior.

## Credential lifecycle and resource bounds

Verifier ownership is deterministic in standalone: Proxy, then MixCoord,
then worker. Workers create a MixCoord client lazily. Each lookup caller
has its own five-second wait bound, so a service-discovery call that
ignores the backend context cannot block the HTTP request indefinitely.
Its accounting slot remains occupied until the shared backend result
completes, preserving the resource bound even in that case. Root hashes
are cached for 10 seconds, reused for up to 10 minutes after their last
successful fetch if refresh fails, and lookup failures are cached for 2
seconds. The stale window preserves management access during a
coordinator outage but can also retain acceptance of a password rotated
during that outage.

Concurrent hash refreshes share work and admit at most 32 callers,
including canceled callers whose singleflight result channels remain
retained. Worker client construction uses a single shared completion
signal; repeated lookup timeout windows no longer append a retained
channel to a stuck constructor. Construction failure can be retried, and
a client completed after shutdown is closed instead of published.

bcrypt concurrency is bounded to a quarter of GOMAXPROCS, floored at
two, with a bounded queue. This is a concurrency cap, not an attempt
rate limit; on small pods sustained incorrect passwords can still
consume substantial CPU.

## Registration, configuration, and event logs

- `http.Register` refuses an unmarked operator route, including method-
and host-qualified ServeMux patterns. Assembled-router tests and a
console-prefix test cover Gin registration.
- Milvus-owned handlers use a private mux. Flag-off compatibility with
more-specific legacy DefaultServeMux routes is retained when pprof is
enabled; enabling the gate removes that fallback.
- `/management/config/alter` always rejects this flag, using the same
`config.FormatKey` identity as config storage, so an anonymous caller
cannot plant a disabling etcd value before the operator enables the
gate.
- Key-normalization caching admits at most 4096 entries and only keys
whose input and normalized forms fit 1024 bytes, using owned string
storage so short substrings cannot pin large request buffers. Larger
keys still normalize correctly without retained memo entries. This
bounds cache retention, not individual HTTP body size.
- The eventlog gRPC stream switches to loopback with authentication
enabled. A failed security upgrade closes the wildcard listener.
Event-log shutdown uses a separate close signal and stops the gRPC
server, preserving safe concurrent recording.
- `milvus_admin_auth_total{endpoint,result}` records root-gate and
origin decisions using registered route patterns; failure logs are rate
limited.
- The standard Go runner now includes HTTP, adminauth, coordinator,
eventlog, and metrics regressions. It restores fail-fast behavior after
sourcing the environment so a failed test command returns a failure from
the runner.

## Compatibility and deployment

Port 9091 remains plaintext. Use a trusted network or TLS-terminating
proxy, rotate root using `UpdateCredential` / `db.update_password`, and
update graceful-shutdown and profiling clients before enabling. Editing
`defaultRootPassword` does not rotate an existing credential. Official
compose deployments and embedded standalone scripts publish 9091, so
review that exposure. Browser access needs the HTTPS deployment
described above.

Regardless of the flag, downstream operator routes without `AdminAuth`
now fail registration; the admin-auth key cannot be altered through the
management API; configuration normalization is bounded; pprof
registration is explicit; and the data-plane stored-password comparison
shares the credential-format helper.

## Verification and limits

CI observed on head `c8992fc7b4d3fcf9a28bf73ad4700283922367f2` at
**2026-09-12 14:30 UTC**: `ci-passed` and `dco-passed` are present.

| Check | Result |
| --- | --- |
| [Build / unit / integration / code
checks](https://jenkins-milvus-ci.milvus.io/job/MILVUS-CI-V2-PR-PIPELINES/job/milvus-build-ut-ciloop-pipeline/30883/pipeline-overview/)
| SUCCESS; status reports build, code checks, integration, Go, and 9,106
C++ tests passed |
| [Go SDK
E2E](https://jenkins-milvus-ci.milvus.io/job/MILVUS-CI-V2-PR-PIPELINES/job/milvus-go-sdk-pipeline/15021/pipeline-overview/)
| SUCCESS |
| [Default
E2E](https://jenkins-milvus-ci.milvus.io/job/MILVUS-CI-V2-PR-PIPELINES/job/milvus-e2e-pipeline-dispatcher/19796/pipeline-overview/)
| SUCCESS |
| Main CI trigger / DCO | SUCCESS |

The non-gating Tencent Pulsar gray run reports `ERROR` because its
downstream job could not be triggered. The previous PR head had the
identical start failure. No Pulsar test result is claimed, and the
precise trigger failure is unknown because Jenkins logs are inaccessible
here. The TC-US shadow build and legacy Jenkins status still show
pending; they are outside the master `ci-passed` requirements. No label
was added manually.

Local verification and remaining limits:

All Go tests use `-tags dynamic,test -gcflags="all=-N -l"`.

- Full `internal/util/adminauth`, including race detection, passed. A
blocked-constructor regression retained 20,011 extra heap objects after
10,000 canceled waits before the fix and no retained growth after it.
Cancellation, retry, panic, and a client completing after close are
covered. Four blocked-backend scenarios cover cold cache, expired stale
state, and correct/incorrect stale passwords; race runs repeated three
times passed, including retained slots and recovery after release.
- Full `internal/http` and its race run passed with the
explicit-request-header changes, including 34 new net/http/Gin cases.
`pkg/eventlog`, `pkg/metrics`, and `pkg/util/merr` also passed.
- Configuration-normalization tests passed, including oversized inputs,
Unicode expansion, and request-buffer ownership. The full config race
suite passed excluding `TestEtcdSource`, whose two tests require a
reachable external etcd and failed against this host's deliberately
unavailable test endpoint. The config comment and ParamItem
documentation match exactly.
- The newly scheduled HTTP, adminauth, eventlog, and metrics commands
passed with the runner's race and coverage flags. A dispatch harness
checked all five package additions through the full and selected-group
entry points. Ten injected command failures were swallowed before
restoring fail-fast and now stop the runner immediately with the
original nonzero status. Coordinator execution was unavailable locally;
the subsequent native CI aggregate reports Go success.
- Legacy Basic challenges are preserved on the main port, gate-off
metrics routes, and the health exemption. A regression using the actual
extracted parser with real Gin failed before the repair and passed after
it; assembled-route and dynamic-gate compatibility tests were added to
the native package suites. The subsequent CI Go check reports success;
individual native test logs were not accessible here.
- `internal/proxy`, `internal/distributed/proxy`,
`internal/coordinator`, and `cmd/roles` cannot compile on this host
because `milvus_core.pc`, `milvus-storage.pc`, and `rocksdb.pc` are
unavailable. The subsequent native CI aggregate reports build and Go
success; this does not change the local compilation limitation.
- Full `make test-go` was attempted; native dependency construction
failed in the host SDK linker (`library 'System' not found`) before the
Go suite. This is not recorded as a passing full test run.
- golangci-lint v2.11.3 built with Go 1.26.6 reported zero new issues in
`internal/http`, `internal/util/adminauth`, and the changed
config/eventlog/metrics/paramtable packages. Native-linked packages
could not be linted locally; the subsequent CI code-check status reports
success.
- A real coordinator failover and a real browser console flow remain
unverified. Unit and source-level failure tracing do not establish full
browser credential replay or transport behavior.

---------

Signed-off-by: Li Liu <[email protected]>
Co-authored-by: Claude Opus 5 (1M context) <[email protected]>
2026-09-15 14:27:33 +08:00

79 lines
2.4 KiB
Go

// Licensed to the LF AI & Data foundation under one
// or more contributor license agreements. See the NOTICE file
// distributed with this work for additional information
// regarding copyright ownership. The ASF licenses this file
// to you under the Apache License, Version 2.0 (the
// "License"); you may not use this file except in compliance
// with the License. You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package eventlog
import (
"encoding/json"
"net/http"
"github.com/milvus-io/milvus/pkg/v3/mlog"
)
const (
// ContentTypeHeader is the health check request type header.
ContentTypeHeader = "Content-Type"
// ContentTypeText is the health check request type text.
ContentTypeText = "text/plain"
// ContentTypeJSON is another health check request type text, which response contains more info.
ContentTypeJSON = "application/json"
)
// eventLogHandler is the event log http handler.
type eventLogHandler struct{}
// Handler returns the discovery handler.
//
// It carries no listener-mode policy of its own. Whether the gRPC stream binds
// to loopback is owned entirely by EnsureListenerMode, which the embedder calls
// before serving and again on every config change. A second, per-request source
// of truth for the same decision would be one that could disagree.
func Handler() http.Handler {
return &eventLogHandler{}
}
type eventLogResponse struct {
Status int `json:"status"`
Port int `json:"port"`
}
// ServeHTTP handler which start a grpc server listen on random available port.
func (h *eventLogHandler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
resp := &eventLogResponse{
Status: http.StatusOK,
}
port, err := getGrpcLogger()
if err != nil {
resp.Status = http.StatusInternalServerError
writeJSON(w, r, resp)
return
}
resp.Port = port
writeJSON(w, r, resp)
}
func writeJSON(w http.ResponseWriter, r *http.Request, resp *eventLogResponse) {
w.Header().Set(ContentTypeHeader, ContentTypeJSON)
bs, err := json.Marshal(resp)
if err != nil {
mlog.Warn(r.Context(), "faild to send response", mlog.Err(err))
}
w.Write(bs)
}