feat: add an in-tree extension mechanism for distributions (#52981)

issue: #52979
design doc:
docs/design-docs/design_docs/20260831-in_tree_extension_mechanism.md

A distribution that compiles its own behavior into the milvus binary
needs little from milvus: a way to install its request hook, the
QueryNode's tuning hook and the cipher without a `.so`, a callback that
starts its control-plane engine when the coordinator becomes active and
stops it on shutdown, and a declaration that it runs the deployment
shape its coordinator behaviors are written for. This PR is that seam
and nothing else. A stock binary installs nothing and behaves exactly as
before.

An earlier revision of this PR shipped a table of typed capabilities
consulted from many places in the tree. Reviewing it against what the
distribution actually needs showed most entries were constants per
deployment (configuration), duplicates of what the request hook already
does, or answers that belonged on the wire. The table is gone; what
remains is small enough to state here.

## Public surface (`pkg/extension`)

```go
func SetHook(h hook.Hook)                       // hookutil prefers it over proxy.soPath
func InstalledHook() hook.Hook

type QueryHook interface { ... }                // the queryNode.soPath plug-in's method set
func SetQueryHook(h QueryHook)                  // the QueryNode prefers it over queryNode.soPath
func InstalledQueryHook() QueryHook

func SetCipher(c hook.Cipher)                   // hookutil prefers it over cipherPlugin.soPathGo
func InstalledCipher() hook.Cipher

type Coordinator interface {                    // the coordinator as its own clients see it
    rootcoordpb.RootCoordClient
    querypb.QueryCoordClient
    datapb.DataCoordClient
}
type CoordinatorEngine interface {
    Start(ctx context.Context, coord Coordinator) error
    Stop() error
}
func SetCoordinatorEngine(e CoordinatorEngine)
func InstalledCoordinatorEngine() CoordinatorEngine

func SetForm()                                  // marks a deployment form installed, in every role
func FormInstalled() bool                       // the coordinators' form-gated behaviors read it

func WithQueryResourceGroup(ctx, rg) context.Context   // set by a hook's Before
func QueryResourceGroupFromContext(ctx) string
```

A distribution calls the setters it needs, then
`cmd/milvus.Main(os.Args)` (exported so a distribution can ship its own
`main`).

- **Request hook.** `hook.Hook` (milvus-proto) is consulted by the
proxy's unary interceptor for every RPC on the gRPC and REST surfaces,
and by hand by the service's two streams, `CreateReplicateStream` and
`DumpMessages`, in the same order and with the handler under the context
`Before` returns; a test fails if the service declares a stream that
does not. A compiled-in hook is used in preference to `proxy.soPath`;
configuring both is refused at start-up. The compiled-in hook is
initialized as a plug-in is: `Init` with the `hook.*` configuration
before it is stored, and the `hook.*` reload watcher re-inits it on a
config change. A failing `Init` is fatal at start-up when a form is
installed, whatever `common.panicWhenPluginFail` says; a hook installed
without a form follows the setting like any plug-in.
- **Query hook and cipher.** The QueryNode's tuning hook
(`SetQueryHook`) and the Go half of the cipher pair (`SetCipher`) can be
compiled in the same way. Each is used in preference to its so path
(`queryNode.soPath`, `cipherPlugin.soPathGo`), is refused beside it, and
gets the same `Init` calls and watchers a plug-in gets.
`cipherPlugin.soPathCpp` still declares that encryption is on, so a
compiled-in cipher in a deployment without it stays idle.
`optimizers.QueryHook` is now an alias of `extension.QueryHook`. None of
the three slots implies another or `FormInstalled`.
- **Coordinator engine.** `mixcoord` starts the installed engine once
the replica is ACTIVE (a standby never starts it), on the coordinator
client it uses itself, and stops it on shutdown. A start failure on
activation is fatal. The engine reaches the coordinator only through
`Coordinator`, so it can do exactly what a proxy can do.
- **Query resource-group mark.** `WithQueryResourceGroup` pins a query
to one resource group: the shard client routes it to the leaders whose
replica lives in that group (`ShardLeadersList.resource_groups`, added
on master by #52716) and the proxy attributes its latency to that group.
Nothing in a stock binary sets it.

## Behaviors a form turns on by declaring itself

These apply only when a form is declared (`extension.SetForm()`, which a
distribution does in every role); a stock binary keeps master's
semantics unchanged. `FormInstalled` is independent of `SetHook`: a
distribution may install a request hook without switching any
coordinator behavior on.

- **Delegators on regular query nodes.** Under a form, a replica's shard
delegators live on its regular RW query nodes, which watch the channel
and read the WAL remotely, instead of on the streaming node's embedded
query node; the streaming node is kept for DDL and the write ahead log.
This lets a form load the same collection into several resource groups,
one streaming node total. The trade is stated in the design doc: a
Strong-consistency search no longer takes the WAL-MVCC shortcut, and the
manual-flush prepare step is skipped with a warning. This shape has been
exercised on a real one-streaming-node / multi-RG cluster, not only in
unit tests: `scripts/walkthrough.py` (create, bulk-import ingest, index,
load into a query cluster, cold and warm query, drop) and the e2e suite
(55 tests, including two resource groups loading the same collection)
run against a local k8s lake-form instance, and five concurrent rounds
of two resource groups querying one collection return correct results
with both replicas intact.
- **A load request speaks only for the resource groups it names.**
Loading a collection into `rg_b` does not restate the whole placement
and evict it from `rg_a`; the groups the request does not name keep
their replica count. A request naming no group states the whole
placement. Per-group load progress is readable through
`ShowLoadCollections` with `resource_group` set (appended proto field;
`-1` means the group holds no replica of the collection). A load that
only adds resource groups or replicas to an already loaded collection
keeps the collection serving (an incremental expansion) instead of
resetting it to loading, whether the added replica lands in a group the
request newly names or one that already serves it; the checkers load the
added replicas, as they do on every other path. An expansion replayed
after a crash stays idempotent.
- **Index engine version with no QueryNode session.** With no QueryNode
registered, datacoord answers the index engine version compiled into the
coordinator itself instead of `0`, and bounds an explicit target version
by that same value, on the assumption that a QueryNode started later
runs the same image.

## Removed from the earlier revision

`common.security.requireAPIKey`, the `proxy.internalDomain.*` listeners,
`queryCoord.resourceGroupScopedLoad`,
`queryCoord.reloadSegmentOnIndexDrop`,
`dataCoord.index.allowVectorIndexDropOnLoadedCollection`,
`dataCoord.index.queryNodesScaleToZero`, the `RefusalResponse` hook
extension, the capability table and `Provider`, and the link-time
primary configuration file name. Each is either a deployment constant
(configuration a distribution ships in its own files), something the
request hook already does, or a default this PR now makes unconditional.

## Compatibility

- `ShowCollectionsRequest.resource_group` is the only proto field this
PR appends; an old peer leaves it empty and both sides read empty as "no
scope". `ShardLeadersList.resource_groups`, which the query mark also
reads, was added on master by #52716, not here; an old peer leaves it
empty too and empty is read as "unknown".
- `hook.Hook` is milvus-proto's and unchanged. No new configuration key.
- `ResetForTest` is built only under the `test` tag.

## Testing

- `pkg/extension`: setters, getters, the context mark, and the split
between `SetHook` and `SetForm` (each answers without the other).
- hookutil: compiled-in hook used, refused beside a plug-in, absent by
default, initialized with the hook configuration, re-initialized by the
reload watcher, a failing `Init` fails start-up when a form is installed
even with `common.panicWhenPluginFail` off but follows the setting
without one; compiled-in cipher used when `soPathCpp` is set, refused
beside `soPathGo`, idle without `soPathCpp`, re-initialized on a
`cipherPlugin.*` edit.
- querynode: compiled-in query hook used, refused beside
`queryNode.soPath`, initialized with the `autoIndex.params.*`
configuration, not installed when it cannot initialize.
- proxy: a hook-pinned resource group reaches the search task; per-group
latency series exist only for pinned requests; both streams consult the
hook, and every stream the service declares is one of them.
- mixcoord: the engine starts on activation only, receives the
coordinator client, is stopped once.
- querycoordv2: scoped `ShowLoadCollections` (100 / -1 / unknown group
refused); scoped load placement with and without a form declared; the
expansion predicate's legs, a request that only adds a replica to an
already-loaded group, and a replayed expansion; delegators placed on a
regular query node under a form and on a streaming query node without
one.
- datacoord: zero-session engine version and its upper bound with a form
declared; `0` without.
- `golangci-lint` with the repo config on every touched package.

One deliberate difference from the "stock binary is untouched" claim,
listed rather than hidden: a stock binary with the streaming service on
bypasses the channel checker's version cache for a tick when a replica's
read-write node set is empty at that tick (a streaming node
mid-restart). The cost is one extra `checkReplica` per tick; the case it
exists for, a deployment that starts a replica's nodes on demand, is the
form's.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_015JwRUhmz28AXHyuAJTSs3E

---------

Signed-off-by: Cai Zhang <[email protected]>
Signed-off-by: cai.zhang <[email protected]>
Co-authored-by: Claude Fable 5.1 <[email protected]>

scripts/run_go_codecov.sh

@@ -40,7 +40,7 @@ export MILVUS_UT_WITHOUT_KAFKA=1 # kafka is not available in the CI environment,
 beginTime=$(date +%s)
 echo -e "=== Running go unittest (parallel=$PARALLEL) ===\n\n"
 
-for d in cmd/tools internal pkg client; do
+for d in cmd/tools cmd/milvus internal pkg client; do
     pushd "$d"
     PROFILE="profile_${d//\//_}.out"
     echo -e "-----------------------------------\nRunning test cases under $d with -p $PARALLEL ..."

scripts/run_go_unittest.sh

@@ -167,6 +167,7 @@ go test -gcflags="all=-N -l" -race -cover -tags dynamic,test "${MILVUS_DIR}/meta
 function test_cmd()
 {
 go test -gcflags="all=-N -l" -race -cover -tags dynamic,test "${ROOT_DIR}/cmd/tools/..." -failfast -count=1 -ldflags="-r ${RPATH}"
+go test -gcflags="all=-N -l" -race -cover -tags dynamic,test "${ROOT_DIR}/cmd/milvus/..." -failfast -count=1 -ldflags="-r ${RPATH}"
 }
 
 function test_streaming()