mirror of
https://github.com/milvus-io/milvus.git
synced 2026-10-12 02:19:10 +00:00
cai.zhangandClaude Fable 5.1
2cbbb72041
issue: #52979 design doc: docs/design-docs/design_docs/20260831-in_tree_extension_mechanism.md A distribution that compiles its own behavior into the milvus binary needs little from milvus: a way to install its request hook, the QueryNode's tuning hook and the cipher without a `.so`, a callback that starts its control-plane engine when the coordinator becomes active and stops it on shutdown, and a declaration that it runs the deployment shape its coordinator behaviors are written for. This PR is that seam and nothing else. A stock binary installs nothing and behaves exactly as before. An earlier revision of this PR shipped a table of typed capabilities consulted from many places in the tree. Reviewing it against what the distribution actually needs showed most entries were constants per deployment (configuration), duplicates of what the request hook already does, or answers that belonged on the wire. The table is gone; what remains is small enough to state here. ## Public surface (`pkg/extension`) ```go func SetHook(h hook.Hook) // hookutil prefers it over proxy.soPath func InstalledHook() hook.Hook type QueryHook interface { ... } // the queryNode.soPath plug-in's method set func SetQueryHook(h QueryHook) // the QueryNode prefers it over queryNode.soPath func InstalledQueryHook() QueryHook func SetCipher(c hook.Cipher) // hookutil prefers it over cipherPlugin.soPathGo func InstalledCipher() hook.Cipher type Coordinator interface { // the coordinator as its own clients see it rootcoordpb.RootCoordClient querypb.QueryCoordClient datapb.DataCoordClient } type CoordinatorEngine interface { Start(ctx context.Context, coord Coordinator) error Stop() error } func SetCoordinatorEngine(e CoordinatorEngine) func InstalledCoordinatorEngine() CoordinatorEngine func SetForm() // marks a deployment form installed, in every role func FormInstalled() bool // the coordinators' form-gated behaviors read it func WithQueryResourceGroup(ctx, rg) context.Context // set by a hook's Before func QueryResourceGroupFromContext(ctx) string ``` A distribution calls the setters it needs, then `cmd/milvus.Main(os.Args)` (exported so a distribution can ship its own `main`). - **Request hook.** `hook.Hook` (milvus-proto) is consulted by the proxy's unary interceptor for every RPC on the gRPC and REST surfaces, and by hand by the service's two streams, `CreateReplicateStream` and `DumpMessages`, in the same order and with the handler under the context `Before` returns; a test fails if the service declares a stream that does not. A compiled-in hook is used in preference to `proxy.soPath`; configuring both is refused at start-up. The compiled-in hook is initialized as a plug-in is: `Init` with the `hook.*` configuration before it is stored, and the `hook.*` reload watcher re-inits it on a config change. A failing `Init` is fatal at start-up when a form is installed, whatever `common.panicWhenPluginFail` says; a hook installed without a form follows the setting like any plug-in. - **Query hook and cipher.** The QueryNode's tuning hook (`SetQueryHook`) and the Go half of the cipher pair (`SetCipher`) can be compiled in the same way. Each is used in preference to its so path (`queryNode.soPath`, `cipherPlugin.soPathGo`), is refused beside it, and gets the same `Init` calls and watchers a plug-in gets. `cipherPlugin.soPathCpp` still declares that encryption is on, so a compiled-in cipher in a deployment without it stays idle. `optimizers.QueryHook` is now an alias of `extension.QueryHook`. None of the three slots implies another or `FormInstalled`. - **Coordinator engine.** `mixcoord` starts the installed engine once the replica is ACTIVE (a standby never starts it), on the coordinator client it uses itself, and stops it on shutdown. A start failure on activation is fatal. The engine reaches the coordinator only through `Coordinator`, so it can do exactly what a proxy can do. - **Query resource-group mark.** `WithQueryResourceGroup` pins a query to one resource group: the shard client routes it to the leaders whose replica lives in that group (`ShardLeadersList.resource_groups`, added on master by #52716) and the proxy attributes its latency to that group. Nothing in a stock binary sets it. ## Behaviors a form turns on by declaring itself These apply only when a form is declared (`extension.SetForm()`, which a distribution does in every role); a stock binary keeps master's semantics unchanged. `FormInstalled` is independent of `SetHook`: a distribution may install a request hook without switching any coordinator behavior on. - **Delegators on regular query nodes.** Under a form, a replica's shard delegators live on its regular RW query nodes, which watch the channel and read the WAL remotely, instead of on the streaming node's embedded query node; the streaming node is kept for DDL and the write ahead log. This lets a form load the same collection into several resource groups, one streaming node total. The trade is stated in the design doc: a Strong-consistency search no longer takes the WAL-MVCC shortcut, and the manual-flush prepare step is skipped with a warning. This shape has been exercised on a real one-streaming-node / multi-RG cluster, not only in unit tests: `scripts/walkthrough.py` (create, bulk-import ingest, index, load into a query cluster, cold and warm query, drop) and the e2e suite (55 tests, including two resource groups loading the same collection) run against a local k8s lake-form instance, and five concurrent rounds of two resource groups querying one collection return correct results with both replicas intact. - **A load request speaks only for the resource groups it names.** Loading a collection into `rg_b` does not restate the whole placement and evict it from `rg_a`; the groups the request does not name keep their replica count. A request naming no group states the whole placement. Per-group load progress is readable through `ShowLoadCollections` with `resource_group` set (appended proto field; `-1` means the group holds no replica of the collection). A load that only adds resource groups or replicas to an already loaded collection keeps the collection serving (an incremental expansion) instead of resetting it to loading, whether the added replica lands in a group the request newly names or one that already serves it; the checkers load the added replicas, as they do on every other path. An expansion replayed after a crash stays idempotent. - **Index engine version with no QueryNode session.** With no QueryNode registered, datacoord answers the index engine version compiled into the coordinator itself instead of `0`, and bounds an explicit target version by that same value, on the assumption that a QueryNode started later runs the same image. ## Removed from the earlier revision `common.security.requireAPIKey`, the `proxy.internalDomain.*` listeners, `queryCoord.resourceGroupScopedLoad`, `queryCoord.reloadSegmentOnIndexDrop`, `dataCoord.index.allowVectorIndexDropOnLoadedCollection`, `dataCoord.index.queryNodesScaleToZero`, the `RefusalResponse` hook extension, the capability table and `Provider`, and the link-time primary configuration file name. Each is either a deployment constant (configuration a distribution ships in its own files), something the request hook already does, or a default this PR now makes unconditional. ## Compatibility - `ShowCollectionsRequest.resource_group` is the only proto field this PR appends; an old peer leaves it empty and both sides read empty as "no scope". `ShardLeadersList.resource_groups`, which the query mark also reads, was added on master by #52716, not here; an old peer leaves it empty too and empty is read as "unknown". - `hook.Hook` is milvus-proto's and unchanged. No new configuration key. - `ResetForTest` is built only under the `test` tag. ## Testing - `pkg/extension`: setters, getters, the context mark, and the split between `SetHook` and `SetForm` (each answers without the other). - hookutil: compiled-in hook used, refused beside a plug-in, absent by default, initialized with the hook configuration, re-initialized by the reload watcher, a failing `Init` fails start-up when a form is installed even with `common.panicWhenPluginFail` off but follows the setting without one; compiled-in cipher used when `soPathCpp` is set, refused beside `soPathGo`, idle without `soPathCpp`, re-initialized on a `cipherPlugin.*` edit. - querynode: compiled-in query hook used, refused beside `queryNode.soPath`, initialized with the `autoIndex.params.*` configuration, not installed when it cannot initialize. - proxy: a hook-pinned resource group reaches the search task; per-group latency series exist only for pinned requests; both streams consult the hook, and every stream the service declares is one of them. - mixcoord: the engine starts on activation only, receives the coordinator client, is stopped once. - querycoordv2: scoped `ShowLoadCollections` (100 / -1 / unknown group refused); scoped load placement with and without a form declared; the expansion predicate's legs, a request that only adds a replica to an already-loaded group, and a replayed expansion; delegators placed on a regular query node under a form and on a streaming query node without one. - datacoord: zero-session engine version and its upper bound with a form declared; `0` without. - `golangci-lint` with the repo config on every touched package. One deliberate difference from the "stock binary is untouched" claim, listed rather than hidden: a stock binary with the streaming service on bypasses the channel checker's version cache for a tick when a replica's read-write node set is empty at that tick (a streaming node mid-restart). The cost is one extra `checkReplica` per tick; the case it exists for, a deployment that starts a replica's nodes on demand, is the form's. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_015JwRUhmz28AXHyuAJTSs3E --------- Signed-off-by: Cai Zhang <[email protected]> Signed-off-by: cai.zhang <[email protected]> Co-authored-by: Claude Fable 5.1 <[email protected]>
76 lines
2.3 KiB
Bash
Executable File
76 lines
2.3 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
|
|
# Licensed to the LF AI & Data foundation under one
|
|
# or more contributor license agreements. See the NOTICE file
|
|
# distributed with this work for additional information
|
|
# regarding copyright ownership. The ASF licenses this file
|
|
# to you under the Apache License, Version 2.0 (the
|
|
# "License"); you may not use this file except in compliance
|
|
# with the License. You may obtain a copy of the License at
|
|
#
|
|
# http://www.apache.org/licenses/LICENSE-2.0
|
|
#
|
|
# Unless required by applicable law or agreed to in writing, software
|
|
# distributed under the License is distributed on an "AS IS" BASIS,
|
|
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
# See the License for the specific language governing permissions and
|
|
# limitations under the License.
|
|
|
|
FILE_COVERAGE_INFO="$PWD/go_coverage.txt"
|
|
FILE_COVERAGE_HTML="$PWD/go_coverage.html"
|
|
|
|
|
|
BASEDIR=$(dirname "$0")
|
|
source $BASEDIR/setenv.sh
|
|
|
|
set -e
|
|
|
|
echo "mode: atomic" > ${FILE_COVERAGE_INFO}
|
|
|
|
TEST_CMD=$@
|
|
if [ -z "$TEST_CMD" ]; then
|
|
TEST_CMD="go test"
|
|
fi
|
|
|
|
PARALLEL=${PARALLEL:-1}
|
|
|
|
export MILVUS_UT_WITHOUT_KAFKA=1 # kafka is not available in the CI environment, so skip the kafka tests
|
|
|
|
# starting the timer
|
|
beginTime=$(date +%s)
|
|
echo -e "=== Running go unittest (parallel=$PARALLEL) ===\n\n"
|
|
|
|
for d in cmd/tools cmd/milvus internal pkg client; do
|
|
pushd "$d"
|
|
PROFILE="profile_${d//\//_}.out"
|
|
echo -e "-----------------------------------\nRunning test cases under $d with -p $PARALLEL ..."
|
|
|
|
$TEST_CMD \
|
|
"-gcflags=all=-N -l" \
|
|
-race \
|
|
-tags dynamic,test \
|
|
-v \
|
|
-failfast \
|
|
-buildvcs=false \
|
|
-coverpkg=./... \
|
|
-coverprofile="$PROFILE" \
|
|
-covermode=atomic \
|
|
-p "$PARALLEL" \
|
|
-ldflags="-r ${RPATH}" \
|
|
./...
|
|
|
|
if [ -f "$PROFILE" ]; then
|
|
sed '1{/^mode:/d}' "$PROFILE" | grep -vE '(planparserv2/generated|mocks)' >> "${FILE_COVERAGE_INFO}" || [ $? -eq 1 ]
|
|
rm "$PROFILE"
|
|
fi
|
|
echo -e "-----------------------------------\n"
|
|
popd
|
|
done
|
|
|
|
endTime=$(date +%s)
|
|
echo -e "=== Total time for go unittest: $(($endTime-$beginTime)) s ==="
|
|
|
|
# generate html report
|
|
# go tool cover -html=./${FILE_COVERAGE_INFO} -o ./${FILE_COVERAGE_HTML}
|
|
# echo "Generate go coverage report to ${FILE_COVERAGE_HTML}"
|